Quick Start Guide: Optimal Policy Trees with Categorical Treatment
In this guide we will give a demonstration of how to use Optimal Policy Trees with categoric treatment options. For this example, we will use the Credit Approval dataset, where the task is to predict the approval outcome of credit card application. Because the features have been anonymized for confidentiality reasons, we will arbitrarily select one of the categoric variables A1 to be the treatment.
Note: this case is not intended to serve as a practical application of policy trees, but rather to serve as an illustration of the training and evaluation process.
First we load in the data and drop 37 rows with missing values:
using CSV, DataFrames
df = CSV.read("crx.data", DataFrame, header=string.("A", 1:16),
missingstring="?", pool=true)
dropmissing!(df)653×16 DataFrame
Row │ A1 A2 A3 A4 A5 A6 A7 A8 A9 ⋯
│ String Float64 Float64 String String String String Float64 Stri ⋯
─────┼──────────────────────────────────────────────────────────────────────────
1 │ b 30.83 0.0 u g w v 1.25 t ⋯
2 │ a 58.67 4.46 u g q h 3.04 t
3 │ a 24.5 0.5 u g q h 1.5 t
4 │ b 27.83 1.54 u g w v 3.75 t
5 │ b 20.17 5.625 u g w v 1.71 t ⋯
6 │ b 32.08 4.0 u g m v 2.5 t
7 │ b 33.17 1.04 u g r h 6.5 t
8 │ a 22.92 11.585 u g cc v 0.04 t
⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋱
647 │ b 36.42 0.75 y p d v 0.585 f ⋯
648 │ b 40.58 3.29 u g m v 3.5 f
649 │ b 21.08 10.085 y p e h 1.25 f
650 │ a 22.67 0.75 u g c v 2.0 f
651 │ a 25.25 13.5 y p ff ff 2.0 f ⋯
652 │ b 17.92 0.205 u g aa v 0.04 f
653 │ b 35.0 3.375 u g c h 8.29 f
8 columns and 638 rows omittedPolicy trees are trained using a features matrix/dataframe X as usual and a rewards matrix that has one column for each potential treatment that contains the outcome for each sample under that treatment.
There are two ways to get this rewards matrix:
- in rare cases, the problem may have full information about the outcome associated with each treatment for each sample
- more commonly, we have observational data, and use this partial data to train models to estimate the outcome associated with each treatment
Refer to the documentation on data preparation for more information on the data format.
In this case, the dataset is observational, and so we will use RewardEstimation to estimate our rewards matrix.
Reward Estimation
First, we split into training and testing:
X = select(df, Not([:A16, :A1]))
treatments = df.A1
outcomes = df.A16 .== "+"
(train_X, train_treatments, train_outcomes), (test_X, test_treatments, test_outcomes) =
IAI.split_data(:policy_maximize, X, treatments, outcomes, seed=1, train_proportion=0.6)Note that we have used a training/test split of 60%/40% split, so that we save more data for testing to ensure high-quality reward estimation on the test set.
The treatment, A1, is a categoric variable, so we follow the process for estimating rewards with categorical treatments.
We use a CategoricalRewardEstimator to estimate the approval outcome under each option with a doubly-robust reward estimation method using random forests to estimate both propensity scores and outcomes:
reward_lnr = IAI.CategoricalRewardEstimator(
propensity_estimator=IAI.RandomForestClassifier(),
outcome_estimator=IAI.RandomForestClassifier(),
reward_estimator=:doubly_robust,
random_seed=1,
)
train_rewards, train_reward_score = IAI.fit_predict!(
reward_lnr, train_X, train_treatments, train_outcomes,
propensity_score_criterion=:auc, outcome_score_criterion=:auc)
train_rewards392×2 DataFrame
Row │ a b
│ Float64 Float64
─────┼─────────────────────────
1 │ 0.507297 1.12704
2 │ 1.4303 0.717781
3 │ 0.9175 1.23799
4 │ 0.438333 1.2739
5 │ 0.667778 1.04595
6 │ 2.09962 0.712232
7 │ 0.393333 1.88344
8 │ 1.08413 0.894286
⋮ │ ⋮ ⋮
386 │ -0.0277255 0.121492
387 │ -0.189727 0.161638
388 │ 0.217105 -0.0299319
389 │ 0.0811905 -0.0858765
390 │ 0.148122 -0.00636079
391 │ -0.215015 0.127157
392 │ 0.400429 -0.0418236
377 rows omittedtrain_reward_scoreDict{Symbol,Any} with 2 entries:
:propensity => 0.616021
:outcome => Dict("b"=>0.913451,"a"=>0.943185)We can see that the internal outcome estimation models have AUCs of 0.91 and 0.94, which gives us confidence that the reward estimates are of decent quality, and good to base our training on. The AUC for the propensity model is lower at 0.62, which is not particularly high, and suggests difficulty in reliably estimating the propensity. The doubly-robust estimation method should help to alleviate this problem, as it is designed to deliver good results if either propensity scores or outcomes are estimated well. However, we should always pay attention to these scores and proceed with caution if the estimation quality is low.
Optimal Policy Trees
Now that we have a complete rewards matrix, we can train a tree to learn an optimal prescription policy that maximizes credit approval outcome. We will use a GridSearch to fit an OptimalTreePolicyMaximizer (note that if we were trying to minimize the outcomes, we would use OptimalTreePolicyMinimizer):
grid = IAI.GridSearch(
IAI.OptimalTreePolicyMaximizer(
random_seed=123,
max_categoric_levels_before_warning=20,
),
max_depth=1:5,
)
IAI.fit!(grid, train_X, train_rewards)Fitted OptimalTreePolicyMaximizer:
1) Split: A6 in [aa,e,i,k,m,q,r,x] or is missing
2) Split: A14 < 162
3) Prescribe: a, 101 points, error 3.475
4) Prescribe: b, 89 points, error 3.721
5) Split: A2 < 28.63
6) Prescribe: b, 95 points, error 3.813
7) Prescribe: a, 107 points, error 3.706The resulting tree recommends different values for A1 based on other characteristics: namely, the values of A2, A6, and A14. The intensity of the color in each leaf shows the difference in quality between the best and second-best values. We can see that both a and b values are prescribed by the tree, implying each treatment is better suited to certain subgroups.
We can make treatment prescriptions using predict:
IAI.predict(grid, train_X)392-element Array{String,1}:
"a"
"b"
"b"
"b"
"a"
"a"
"a"
"a"
"b"
"b"
⋮
"b"
"b"
"b"
"b"
"b"
"a"
"b"
"b"
"a"If we want more information about the relative performance of treatments for these points, we can predict the full treatment ranking with predict_treatment_rank:
IAI.predict_treatment_rank(grid, train_X)392×2 Array{String,2}:
"a" "b"
"b" "a"
"b" "a"
"b" "a"
"a" "b"
"a" "b"
"a" "b"
"a" "b"
"b" "a"
"b" "a"
⋮
"b" "a"
"b" "a"
"b" "a"
"b" "a"
"b" "a"
"a" "b"
"b" "a"
"b" "a"
"a" "b"To quantify the difference in performance behind the treatment rankings, we can use predict_treatment_outcome to extract the estimated quality of each treatment for each point:
IAI.predict_treatment_outcome(grid, train_X)392×2 DataFrame
Row │ a b
│ Float64 Float64
─────┼────────────────────
1 │ 0.510482 0.352249
2 │ 0.317545 0.495565
3 │ 0.317545 0.495565
4 │ 0.317545 0.495565
5 │ 0.510482 0.352249
6 │ 0.741216 0.561335
7 │ 0.741216 0.561335
8 │ 0.741216 0.561335
⋮ │ ⋮ ⋮
386 │ 0.251901 0.402943
387 │ 0.251901 0.402943
388 │ 0.317545 0.495565
389 │ 0.741216 0.561335
390 │ 0.317545 0.495565
391 │ 0.251901 0.402943
392 │ 0.510482 0.352249
377 rows omittedEvaluating Optimal Policy Trees
It is critical for a fair evaluation that we do not evaluate the quality of the policy using rewards from our existing reward estimator trained on the training set. This is to avoid any information from the training set leaking through to the out-of-sample evaluation.
Instead, what we need to do is to estimate a new set of rewards using only the test set, and evaluate the policy against these rewards:
test_rewards, test_reward_score = IAI.fit_predict!(
reward_lnr, test_X, test_treatments, test_outcomes,
propensity_score_criterion=:auc, outcome_score_criterion=:auc)
test_rewards261×2 DataFrame
Row │ a b
│ Float64 Float64
─────┼─────────────────────────
1 │ 1.05874 0.958918
2 │ 0.69471 1.04043
3 │ 0.538095 1.4718
4 │ 0.499545 1.29655
5 │ 2.21112 0.588636
6 │ 0.23 1.27902
7 │ 0.324085 1.08234
8 │ 1.50441 0.921468
⋮ │ ⋮ ⋮
255 │ -1.2451 0.3688
256 │ 0.203788 -0.0312968
257 │ 0.3275 -0.0280927
258 │ 0.0683333 -0.00856246
259 │ 0.2025 -0.0245124
260 │ -1.53126 0.369921
261 │ 0.112667 -0.215225
246 rows omittedtest_reward_scoreDict{Symbol,Any} with 2 entries:
:propensity => 0.575921
:outcome => Dict("b"=>0.940979,"a"=>0.91585)We see the scores are similar to those on the training set, giving us confidence that the estimated rewards are a fair reflection of reality, and will serve as a good basis for evaluation. The AUC for the propensity model is again on the low side, but should be compensated for by means of the doubly-robust estimator.
We can now evaluate the quality using these new estimated rewards. First, we will calculate the average predicted credict approval probability under the treatments prescribed by the tree for the test set. To do this, we use predict_outcomes which uses the model to make prescriptions and looks up the predicted outcomes under these prescriptions:
policy_outcomes = IAI.predict_outcomes(grid, test_X, test_rewards)261-element Array{Float64,1}:
1.0587409115392878
1.0404290934416511
1.4717999433572835
1.296552772894011
0.5886363636363636
1.2790227181522082
0.3240854058222479
1.504410672932523
1.0825273228278531
1.1061303560636178
⋮
0.06972222222222224
0.06000000000000005
-1.2450950376323506
0.20378787878787885
-0.028092657153395262
0.06833333333333336
-0.024512384156882172
0.369920634920635
-0.21522544999798754We can then get the average estimated approval probability under our treatments:
using Statistics
mean(policy_outcomes)0.4443510311691845We can compare this number to the average approval probability under the treatment assignments that were actually observed:
mean([test_rewards[i, test_treatments[i]] for i in 1:length(test_treatments)])0.42341831736732We see a small improvement over the real treatments.