Scoring Criteria

Scoring Criteria

There are various scoring criteria that you can use for training, tuning and evaluating the models. The criterion to use in the relevant functions is specified by passing the indicated Symbol as the criterion argument.

When evaluating the performance of a learner using the score function, the results are scaled so that a larger score is better, the baseline model receives a score of 0, and a perfect model would receive a score of 1. In this sense, the results from score are similar to $R^2$ for regression problems, where 0 indicates the model is no better than predicting the mean, and 1 is a perfect prediction for each point.

In the definitions below, $w_i$ is the sample weight assigned to the $i$th point (where $i = 1, \ldots, n$).

The available criteria depend on the problem type.

Classification

The classification criteria use the following notation:

Misclassification

Argument name: :misclassification

Misclassification calculates the (weighted) proportion of points that are misclassified:

\[\frac{\sum_{i = 1}^n w_i \cdot I(f_i \neq y_i)}{\sum_{i = 1}^n w_i}\]

where $I(x)$ denotes the indicator function (1 if $x$ is true and 0 otherwise).

Gini Impurity

Argument name: :gini

The gini impurity measures how well the fitted probabilities match the empirical probability distribution:

\[\sum_{i = 1}^n w_i (1 - p_{if_i})\]

Note: the gini impurity is equivalent to treating the output as a continuous value and calculating the mean-squared error. It is also equivalent to the Brier score. In particular, the Brier score decomposes into refinement and calibration components:

This means the gini impurity can often be a better measure of performance than AUC, as AUC only evaluates the ordering of the predictions. Additionally, many methods cannot train with AUC as the objective function, but can use the gini impurity, thus if AUC is the end metric, it is often the case that using gini as the criterion can improve results.

Entropy

Argument name: :entropy

The entropy criterion measures the goodness of the predictions using the concept of entropy from information theory:

\[- \sum_{i = 1}^n w_i \log (p_{if_i})\]
Warning

it is not recommended to use the entropy criterion for validation or out-of-sample evaluation as the score can be extremely negative if even a single point in the new data has a predicted probability close to zero (due to the logarithm in the calculation).

Note: entropy is equivalent to the log loss and other related loss functions.

AUC

Info

Can only be applied to classification problems with $K=2$ classes.

Argument name: :auc

The area under the curve is a holistic measure of quality that gives the probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative one.

Threshold-based

Info

Can only be applied to classification problems with $K=2$ classes.

Argument names:

The threshold-based criteria are a family of values based on the confusion matrix and are calculated based on the number of true/false positives and negatives in the prediction. The label to treat as the positive class must be supplied using the positive_label keyword argument. The threshold keyword argument allows you to specify the threshold that determines the prediction cutoff (if not specified it defaults to 0.5, i.e. predict the class with the highest probability).

Hinge Loss (Classification)

Info

Can only be applied to classification problems with $K=2$ classes.

Argument names:

The L1 hinge loss criterion scores the quality of predictions using a linear function that goes to zero once the classification of a sample is "sufficiently correct":

\[\sum_{i = 1}^n \max(0, 1 - \hat{y_i} \cdot d_i)\]

where $\hat{y_i}$ is -1 if $y_i$ is the first class, and +1 if $y_i$ is the second class, and $d_i$ is the raw output of the model's decision function, such that

\[P(\hat{y_i} = -1) = \frac{1}{1 + e ^ {d_i}}\]

The L2 hinge loss criterion is a variant of the L1 hinge loss that uses a squared loss instead of linear:

\[\sum_{i = 1}^n \frac{1}{2} \max(0, 1 - \hat{y_i} \cdot d_i) ^ 2\]

Regression

The regression criteria use the following notation:

Mean-squared error

Argument name: :mse

The mean-squared error measures the (weighted) mean of squared errors between the fitted and true values:

\[\frac{\sum_{i = 1}^n w_i \cdot (f_i - y_i)^2}{\sum_{i = 1}^n w_i}\]

Note that the result returned by score is actually the $R^2$ due to the way scores are scaled as described at the top of this page.

Tweedie

Argument name: :tweedie

The Tweedie distribution is often used in insurance applications, where the total claim amount for a covered risk usually has a continuous distribution on positive values, except for the possibility of being exactly zero when the claim does not occur. The Tweedie distribution models the total claim amount as the sum of a Poisson number of independent Gamma variables. The Tweedie criterion measures the log-likelihood of the predicted values against this distribution:

\[\sum_{i = 1}^n w_i \left( - \frac{y_i e^{(1 - \rho) \log(f_i)}}{1 - \rho} + \frac{e^{(2 - \rho) \log(f_i)}}{2 - \rho} \right)\]

where $\rho$ is given by the tweedie_variance_power parameter (between 1 and 2) which characterizes the distribution of the responses. Set closer to 2 to shift towards a gamma distribution and set closer to 1 to shift towards a Poisson distribution.

Hinge Loss (Regression)

Argument names:

The hinge loss criteria for regression problems score the quality of predictions using an adjusted absolute loss that goes to zero if the error of the prediction is lower than $\epsilon$ (given by the hinge_epsilon parameter).

The L1 hinge loss simply uses this adjusted absolute error:

\[\sum_{i = 1}^n \max(0, |f_i - y_i| - \epsilon)\]

The L2 hinge loss criterion squares the adjusted absolute error:

\[\sum_{i = 1}^n \frac{1}{2} \max(0, |f_i - y_i| - \epsilon) ^ 2\]

Prescription

The prescription criteria use the following notation:

Combined Performance

Argument name: :combined_performance

It is shown empirically in the Optimal Prescriptive Trees paper that focusing only on prescription quality may limit the quality of results in prescription problems. To address this, the combined performance criterion balances the quality of the prescriptions being made (the first term below) against the accuracy of the predicted outcomes against those outcomes that were actually observed (the second term below):

\[\mu \left( \sum_{i = 1}^n w_i f_i(z_i) \right) + (1 - \mu) \left( \sum_{i = 1}^n w_i (f_i(T_i) - y_i) ^ 2 \right)\]

where $\mu$ is given by the prescription_factor parameter (between 0 and 1) and controls the tradeoff between the prescriptive and predictive components of the score.

Prescription Outcome

Argument name: :prescription_outcome

Identical to Combined Performance with prescription_factor set to 1, so only considers optimizing the expected outcome of the prescriptions.

Prediction Accuracy

Argument name: :prediction_accuracy

Identical to Combined Performance with prescription_factor set to 0, so only considers matching the predicted outcomes under the observed treatments with the observed outcomes.

Survival

The survival criteria use the following notation:

Full Local Likelihood

Argument name: :fulllocallikelihood

The full local likelihood is the objective function used by LeBlanc and Crowley and assumes that the survival distribution for each observation is a function of the cumulative hazard function $\Lambda$ and a point-wise adjustment $\theta_i$:

\[\mathbb{P}(\text{survival time} \leq t) = 1 - e^{-\theta_i \Lambda(t)}\]

We then measure the log-likelihood of this survival function against the data:

\[\sum_{i = 1}^n w_i \biggl( \Lambda(t_i) \theta_i - \delta_i \left[ \log (\Lambda(t_i)) + \log (\theta_i) + 1 \right] \biggr)\]

Log Likelihood

Argument name: :loglikelihood

The log likelihood criterion uses the fitted survival curve for each point to calculate the log likelihood of the data:

\[\sum_{i = 1}^n -w_i \log(1 - f_i(t_i))\]

Integrated Brier Score

Argument name: :integratedbrier

The integrated Brier score is a variant of the Brier score for classification (see Gini Impurity) that was adapted by Graf et al. to deal with censored data:

\[\sum_{i = 1}^n w_i \biggl( \int_{0}^{t_i} \frac{(1 - f_i(t))^2}{g_i(t)} dt + \delta_i \int_{t_i}^{t_\max} \frac{f_i(t)^2}{g_i(t_i)} dt \biggr)\]

Harrell's c-statistic

Argument name: :harrell_c_statistic

Harrell's c-statistic is the most widely-used measure of goodness-of-fit for survival models, and is analogous to the AUC for classification problems.

Mathematically, the score gives the probability that the model assigns a higher probability of survival to the patient that actually lived longer (i.e. the patient with better survival outlook is correctly identified by the model) when presented with a random pair of patients from the dataset in question. Therefore, like AUC, a simple baseline of random guessing will get a score of 0.5, while a perfect model will achieve a score of 1.0.

For more information on concordance statistics and evaluation for survival problems, refer to the vignette on concordance from the R survival package.

Imputation

As an unsupervised task, in imputation there is no concept of the predictions matching the target from a scoring perspective. Instead, we use criteria to measure the similarity between datasets after imputation:

We use the following notation:

Euclidean/L2 Distance

Argument name: :l2

The Euclidean distance calculates the squared distance between all imputed values and their known-true values:

\[\sum_{(i, j) \in \mathcal M} w_i (f_{ij} - x_{ij}) ^ 2\]

Manhattan/L1 Distance

Argument name: :l1

The Manhattan distance is the same as the euclidean criterion, except it uses the absolute distance rather than squared distance:

\[\sum_{(i, j) \in \mathcal M} w_i |f_{ij} - x_{ij}|\]