Classification

Quick Start Guide: Optimal Feature Selection for Classification

In this example we will use optimal feature selection on the Mushroom dataset, where the goal is to distinguish poisonous from edible mushrooms.

First we load in the data and split into training and test datasets:

using CSV
df = CSV.read("agaricus-lepiota.data", categorical=true, copycols=true,
              header=[:target, :cap_shape, :cap_surface, :cap_color, :bruises,
                      :odor, :gill_attachment, :gill_spacing, :gill_size,
                      :gill_color, :stalk_shape, :stalk_root,
                      :stalk_surface_above, :stalk_surface_below,
                      :stalk_color_above, :stalk_color_below, :veil_type,
                      :veil_color, :ring_number, :ring_type, :spore_color,
                      :population, :habitat])
8124×23 DataFrames.DataFrame. Omitted printing of 19 columns
│ Row  │ target       │ cap_shape    │ cap_surface  │ cap_color    │
│      │ Categorical… │ Categorical… │ Categorical… │ Categorical… │
├──────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ 1    │ p            │ x            │ s            │ n            │
│ 2    │ e            │ x            │ s            │ y            │
│ 3    │ e            │ b            │ s            │ w            │
│ 4    │ p            │ x            │ y            │ w            │
│ 5    │ e            │ x            │ s            │ g            │
│ 6    │ e            │ x            │ y            │ y            │
│ 7    │ e            │ b            │ s            │ w            │
⋮
│ 8117 │ p            │ k            │ y            │ n            │
│ 8118 │ p            │ k            │ s            │ e            │
│ 8119 │ p            │ k            │ y            │ n            │
│ 8120 │ e            │ k            │ s            │ n            │
│ 8121 │ e            │ x            │ s            │ n            │
│ 8122 │ e            │ f            │ s            │ n            │
│ 8123 │ p            │ k            │ y            │ n            │
│ 8124 │ e            │ x            │ s            │ n            │
X = df[:, 2:end]
y = df.target
(train_X, train_y), (test_X, test_y) = IAI.split_data(:classification, X, y,
                                                      seed=1)

Model Fitting

We will use a GridSearch to fit an OptimalFeatureSelectionClassifier:

grid = IAI.GridSearch(
    IAI.OptimalFeatureSelectionClassifier(
        random_seed=1,
    ),
    sparsity=1:10,
)
IAI.fit!(grid, train_X, train_y, validation_criterion=:auc)
All Grid Results:

│ Row │ sparsity │ train_score │ valid_score │ rank_valid_score │
│     │ Int64    │ Float64     │ Float64     │ Int64            │
├─────┼──────────┼─────────────┼─────────────┼──────────────────┤
│ 1   │ 1        │ 0.507063    │ 0.892471    │ 10               │
│ 2   │ 2        │ 0.616228    │ 0.945726    │ 9                │
│ 3   │ 3        │ 0.711851    │ 0.969207    │ 8                │
│ 4   │ 4        │ 0.764492    │ 0.977629    │ 7                │
│ 5   │ 5        │ 0.802974    │ 0.981979    │ 6                │
│ 6   │ 6        │ 0.82443     │ 0.984656    │ 5                │
│ 7   │ 7        │ 0.877074    │ 0.997937    │ 3                │
│ 8   │ 8        │ 0.916796    │ 0.999686    │ 1                │
│ 9   │ 9        │ 0.874916    │ 0.995357    │ 4                │
│ 10  │ 10       │ 0.907388    │ 0.99862     │ 2                │

Best Params:
  sparsity => 8

Best Model - Fitted OptimalFeatureSelectionClassifier:
  Constant: 1.35677
  Weights:
    gill_color==b:   1.24686
    gill_size==b:   -1.20474
    gill_size==n:    1.20474
    odor==a:        -3.81151
    odor==f:         2.98149
    odor==l:        -3.81851
    odor==n:        -3.70758
    spore_color==r:  5.98771

The model selected a sparsity of 8 as the best parameter, but we observe that the validation scores are close for many of the parameters. We can use the results of the grid search to explore the tradeoff between the complexity of the regression and the quality of predictions:

using Plots
results = IAI.get_grid_results(grid)
plot(results.sparsity, results.valid_score, xlabel="Sparsity",
     ylabel="Validation AUC", legend=nothing)

We see that the quality of the model quickly increases with additional terms to reach AUC 0.98 with four terms. After this, additional terms increase the quality more slowly, reaching AUC close to 1 with 7 terms. Depending on the application, we might decide to choose a lower sparsity for the final model than the value chosen by the grid search.

We can make predictions on new data using predict:

IAI.predict(grid, test_X)
2437-element Array{String,1}:
 "e"
 "e"
 "e"
 "p"
 "e"
 "e"
 "e"
 "e"
 "e"
 "e"
 ⋮
 "p"
 "p"
 "p"
 "e"
 "e"
 "e"
 "e"
 "p"
 "e"

We can evaluate the quality of the model using score with any of the supported loss functions. For example, the misclassification on the training set:

IAI.score(grid, train_X, train_y, criterion=:misclassification)
0.9934939335326183

Or the AUC on the test set:

IAI.score(grid, test_X, test_y, criterion=:auc)
0.9997997100178709