|
|
|
|
|
Accuracy is the proportion
of the total number of predictions that were correct.
|
|
|
|
AR is the summary index
of Cumulative Accuracy Profile (CAP) and is also known as Gini coefficient.
It shows the performance of the model that is being evaluated by depicting
the percentage of defaulted accounts that are captured by the model
across different scores.
|
|
|
|
AUC can be interpreted
as the average ability of the rating model to accurately classify
non-default accounts and default accounts. It represents the discrimination
between the two populations. A higher area denotes higher discrimination.
When AUC is 0.5, it means that non-default accounts and default accounts
are randomly classified, and when AUC is 1, it means that the scoring
model accurately classifies non-default accounts and default accounts.
Thus, the AUC ranges between 0.5 and 1.
|
|
|
Bayesian Error Rate
(BER)
|
BER is the proportion
of the whole sample that is misclassified when the rating system is
in optimal use. For a perfect rating model, the BER has a value of
zero. A model's BER depends on the probability of default. The lower
the BER, and the lower the classification error, the better the model.
|
|
|
|
The D Statistic is the
mean difference of scores between default accounts and non-default
accounts, weighted by the relative distribution of those scores.
|
|
|
|
The Error Rate is the
proportion of the total number of incorrect predictions.
|
|
|
Information Statistic
(I)
|
The Information Statistic
value is a weighted sum of the difference between conditional default
and conditional non-default rates. The higher the value, the more
likely a model can predict a default account.
|
|
|
|
Kendall's tau-b is a
nonparametric measure of association based on the number of concordances
and discordances in paired observations. Kendall's tau values range
between -1 and +1, with a positive correlation indicating that the
ranks of both variables increase together. A negative association
indicates that as the rank of one variable increases, the rank of
the other variable decreases.
|
|
|
Kullback-Leibler Statistic
(KL)
|
KL is a non-symmetric
measure of the difference between the distributions of default accounts
and non-default accounts. This score has similar properties to the
information value.
|
|
|
Kolmogorov-Smirnov Statistic
(KS)
|
KS is the maximum distance
between two population distributions. This statistic helps discriminate
default accounts from non-default accounts. It is also used to determine
the best cutoff in application scoring. The best cutoff maximizes
KS, which becomes the best differentiator between the two populations.
The KS value can range between 0 and 1, where 1 implies that the model
is perfectly accurate in predicting default accounts or separating
the two populations. A higher KS denotes a better model.
|
|
|
|
1-PH is the percentage
of cumulative non-default accounts for the cumulative 50% of the default
accounts.
|
|
|
Mean Square Error (MSE),
Mean Absolute Deviation (MAD), and Mean Absolute Percent Error (MAPE)
|
MSE, MAD, and MAPE are
generated for LGD reports. These statistics measure the differences
between the actual LGD and predicted LGD.
|
|
|
|
The Pietra Index is
a summary index of Receiver Operating Characteristic (ROC) statistics
because the Pietra Index is defined as the maximum area of a triangle
that can be inscribed between the ROC curve and the diagonal of the
unit square.
The Pietra Index can
take values between 0 and 0.353. As a rating model's performance improves,
the value is closer to 0.353. This expression is interpreted as the
maximum difference between the cumulative frequency distributions
of default accounts and non-default accounts.
|
|
|
|
Precision is the proportion
of the actual default accounts among the predicted default accounts.
|
|
|
|
Sensitivity is the ability
to correctly classify default accounts that have actually defaulted.
|
|
|
|
Somers' D is a nonparametric
measure of association that is based on the number of concordances
and discordances in paired observations. It is an asymmetric modification
of Kendall's tau. Somers' D differs from Kendall’s tau in that
it uses a correction only for pairs that are tied on the independent
variable. Values range between -1 and +1. A positive association indicates
that the ranks for both variables increase together. A negative association
indicates that as the rank of one variable increases, the rank of
the other variable decreases.
|
|
|
|
Specificity is the ability
to correctly classify non-default accounts that have not defaulted.
|
|
|
|
The Validation Score
is the average scaled value of seven distance measures, anchored to
a scale of 1 to 13, lowest to highest. The seven measures are the
mean difference (D), the percentage of cumulative non-default accounts
for the cumulative 50% of the default accounts (1-PH), the maximum
deviation (KS), the Gini coefficient (G), the Information Statistic
(I), the Area Under the Curve (AUC), or Receiver Operating Characteristic
(ROC) statistic, and the Kullback-Leibler statistic (KL).
|
|
|