After
you execute a performance definition for the champion model, you can
execute a performance definition for the challenger model using the
same performance data sets. SAS Model Manager updates
the output data sets with the performance data for the challenger
model. You can create a Champion and Challenger Performance report
that compares the performance of the two models.
The Champion and Challenger
Performance report contains these charts:
Number of Predictors Exceeding Deviation Threshold
This characteristic
report creates a chart for each index that exceeds a deviation threshold
(either 0.1 or 0.25) as indicated in the define performance definition.
The characteristic report detects shifts in the distribution of input
variables over time.
Lift Trend Chart
A Lift Trend chart
displays the cumulative lift of the champion model over time.
Gini - Trend
When the Gini - ROC
Chart is created, the Gini index for each ROC curve is also created.
The Gini coefficient represents the area under the ROC curve and is
a benchmark statistic that can be used to summarize the predictive
accuracy of a model. The Gini - Trend Chart plots a model's Gini
index scores over time, and these are used to monitor model degradation
over time.
Gini - ROC Chart
Sensitivity is the
proportion of true positive events, and specificity is the proportion
of true negative events. The Gini - ROC Chart plots Sensitivity on
the Y axis and 1 - Specificity on the X axis.
KS Trend Chart
When you create a Kolmogorov-Smirnov
report, the KS statistic and the corresponding probability cutoff
are computed for each Kolmogorov-Smirnov table. The KS Trend Chart
uses a summary data set that plots the KS Statistic and the probability
cutoff values over time. The KS Trend Chart is used to monitor model
degradation over time.
KS Chart
The KS Chart uses the
Kolmogorov-Smirnov statistic to measure the maximum vertical separation,
or deviation between the cumulative distributions of events and non-events.
Score Histogram
The Score Histogram
compares the scoring result distribution at different time periods
using a histogram.
Score Distribution Line Plot
The Score Distribution
Line Plot compares the scoring result distribution at different time
periods using a line plot.
Before you create a
Champion and Challenger Performance report, verify the performance
data and model status.