# Parameter Tuning

Like most machine learning methods, you will likely need to tune parameters of optimal tree learners through validation to get the best results. This page discusses which parameters to validate and some suggested approaches for validation.

Refer to the IAIBase documentation on parameter tuning for a general description on the tuning interface.

It is **highly recommended** that you use the `GridSearch`

interface whenever you are fitting optimal tree models, as it will automatically tune the complexity parameter `cp`

using a method that is **significantly stronger** than manual tuning.

## General approach to parameter tuning

First, we outline a strategy for parameter tuning that should provide a good start for most applications. These suggestions are based on our experiences, gained through tests with synthetic and real-world datasets as well as many applications.

For most problems, the key parameters that affect the quality of the generated trees are:

`cp`

`max_depth`

`criterion`

As mentioned above, when fitting using a `GridSearch`

, the value of `cp`

will be automatically tuned with high precision. We recommend tuning `max_depth`

and trying different values of `criterion`

as appropriate for your problem. Typically values for `max_depth`

in the range of 5–10 are sufficient, but it is often valuable to keep trying deeper trees until the performance stops significantly improving.

The following code tunes an Optimal Classification Tree with `max_depth`

between 5 and 10 and with `cp`

automatically tuned:

```
grid = IAI.GridSearch(
IAI.OptimalTreeClassifier(
random_seed=1,
),
max_depth=1:10,
)
IAI.fit!(grid, X, y)
```

```
All Grid Results:
│ Row │ max_depth │ cp │ train_score │ valid_score │ rank_valid_score │
│ │ Int64 │ Float64 │ Float64 │ Float64 │ Int64 │
├─────┼───────────┼────────────┼─────────────┼─────────────┼──────────────────┤
│ 1 │ 1 │ 0.323185 │ 0.842708 │ 0.864078 │ 10 │
│ 2 │ 2 │ 0.0257611 │ 0.919792 │ 0.940049 │ 9 │
│ 3 │ 3 │ 0.00468384 │ 0.98125 │ 0.960922 │ 8 │
│ 4 │ 4 │ 0.00702576 │ 0.995833 │ 0.967961 │ 7 │
│ 5 │ 5 │ 0.00117096 │ 1.0 │ 0.99199 │ 1 │
│ 6 │ 6 │ 0.0019516 │ 1.0 │ 0.986893 │ 2 │
│ 7 │ 7 │ 0.00058548 │ 1.0 │ 0.984951 │ 3 │
│ 8 │ 8 │ 0.00058548 │ 1.0 │ 0.984709 │ 4 │
│ 9 │ 9 │ 0.00058548 │ 1.0 │ 0.983738 │ 5 │
│ 10 │ 10 │ 0.00058548 │ 1.0 │ 0.983738 │ 6 │
Best Params:
cp => 0.00117096018735363
max_depth => 5
Best Model - Fitted OptimalTreeClassifier:
1) Split: skewness < 5.161
2) Split: variance < 0.3202
3) Split: curtosis < 6.746
4) Split: curtosis < 3.064
5) Predict: 1 (100.00%), [0,305], 305 points, error 0
6) Split: skewness < -0.4759
7) Predict: 1 (100.00%), [0,68], 68 points, error 0
8) Predict: 0 (100.00%), [3,0], 3 points, error 0
9) Split: skewness < -4.8
10) Split: variance < -0.5345
11) Predict: 1 (100.00%), [0,120], 120 points, error 0
12) Predict: 0 (100.00%), [1,0], 1 points, error 0
13) Predict: 0 (100.00%), [23,0], 23 points, error 0
14) Split: curtosis < 0.1899
15) Split: variance < 2.462
16) Predict: 1 (97.33%), [2,73], 75 points, error 2
17) Predict: 0 (100.00%), [26,0], 26 points, error 0
18) Split: variance < 0.7508
19) Split: entropy < 0.7304
20) Predict: 0 (100.00%), [30,0], 30 points, error 0
21) Predict: 1 (100.00%), [0,3], 3 points, error 0
22) Predict: 0 (100.00%), [278,0], 278 points, error 0
23) Split: variance < -3.368
24) Split: entropy < -1.916
25) Predict: 1 (100.00%), [0,40], 40 points, error 0
26) Predict: 0 (100.00%), [1,0], 1 points, error 0
27) Split: curtosis < -5.079
28) Predict: 1 (100.00%), [0,1], 1 points, error 0
29) Predict: 0 (100.00%), [398,0], 398 points, error 0
```

We can see that the performance on the validation set levels out around depth 5, so we don't need to push for deeper trees with this data. Also note that for each depth, `cp`

has indeed been automatically tuned very precisely during the validation process.

Depending on your problem, you might also find it beneficial to tune the following parameters:

`minbucket`

: if you notice that you are seeing a lot of leaves with small numbers of points, you might like to increase`minbucket`

to generate trees with larger leaves`ls_num_tree_restarts`

: if you are noticing instability or fluctuation in the results, you might get more consistent and higher quality results with a larger number of trees - a larger number of trees is usually better for any problem, but at the cost of longer training times for what might only be a marginal benefit

## Optimal Regression Trees with Linear Predictions

When using Optimal Regression Trees with linear predictions in the leaves, it is crucial to tune `regression_lambda`

, the amount of regularization in the linear regression equations. We heavily recommend tuning both `max_depth`

and `regression_lambda`

to get the best results, but it can be computationally expensive to tune these simultaneously in the same grid search. Instead, we suggest the following three step process that tunes the parameters in an alternating fashion. We have found that this is much faster and typically has very similar performance to the full grid search.

#### Step 1: Get a starting estimate for `regression_lambda`

We need to choose a starting value for `regression_lambda`

. You can either use the default value, or find a good starting estimate yourself.

One approach to doing this yourself cheaply is to validate over `regression_lambda`

with `max_depth`

fixed to zero - this is effectively just fitting a linear regression to the data and allows you to find a good baseline level of regularization:

```
grid = IAI.GridSearch(
IAI.OptimalTreeRegressor(
random_seed=1,
max_depth=0,
regression_sparsity=:all,
),
regression_lambda=[0.0001, 0.001, 0.01, 0.1],
)
IAI.fit!(grid, X, y)
starting_lambda = IAI.get_best_params(grid)[:regression_lambda]
```

`0.1`

#### Step 2: Tune `max_depth`

with `regression_lambda`

fixed

Using the starting estimate from Step 1 for `regression_lambda`

, we now tune `max_depth`

:

```
grid = IAI.GridSearch(
IAI.OptimalTreeRegressor(
random_seed=1,
regression_sparsity=:all,
regression_lambda=starting_lambda,
),
max_depth=1:5,
)
IAI.fit!(grid, X, y)
best_depth = IAI.get_best_params(grid)[:max_depth]
```

`4`

#### Step 3: Fix `max_depth`

and tune `regression_lambda`

Finally, we fix `max_depth`

to the value found in Step 2, and tune `regression_lambda`

to get the final result:

```
grid = IAI.GridSearch(
IAI.OptimalTreeRegressor(
random_seed=1,
max_depth=best_depth,
regression_sparsity=:all,
),
regression_lambda=[0.0001, 0.001, 0.01, 0.1],
)
IAI.fit!(grid, X, y)
IAI.get_best_params(grid)
```

```
Dict{Symbol,Any} with 2 entries:
:regression_lambda => 0.001
:cp => 2.40921e-5
```