Format a Champion and Challenger Performance Report

About the Champion and Challenger Performance Report

After you execute a performance task for the champion model, you can execute a performance task for the challenger model using the same performance data sets. SAS Model Manager updates the output data sets in the Resources folder with the performance data for the challenger model. You can create a Champion and Challenger Performance report that compares the performance of the two models.
The Champion and Challenger Performance report contains these charts:
Number of Predictors Exceeding Deviation Threshold
This characteristic report creates a chart for each index exceeding a deviation threshold (either 0.1 or 0.25) as indicated in the define performance task. The characteristic report detects shifts in the distribution of input variables over time.
Lift Trend Chart
A Lift Trend chart displays the cumulative lift of the champion model over time.
Gini - Trend
When the Gini - ROC Chart is created, the Gini index for each ROC curve is also created. The Gini coefficient represents the area under the ROC curve and is a benchmark statistic that can be used to summarize the predictive accuracy of a model. The Gini - Trend Chart plots a model's Gini index scores over time, and these are used to monitor model degradation over time.
Gini - ROC Chart
Sensitivity is the proportion of true positive events and specificity is the proportion of true negative events. The Gini - ROC Chart plots Sensitivity on the Y axis and 1 - Specificity on the X axis.
KS Trend Chart
When you create a Kolmogorov-Smirnov report, the KS statistic and the corresponding probability cutoff are computed for each Kolmogorov-Smirnov table. The KS Trend Chart uses a summary data set that plots the KS Statistic and the probability cutoff values over time. The KS Trend Chart is used to monitor model degradation over time.
KS Chart
The KS Chart uses the Kolmogorov-Smirnov statistic to measure the maximum vertical separation, or deviation between the cumulative distributions of events and non-events.
Score Histogram
The Score Histogram compares the scoring result distribution at different time periods using a histogram.
Score Distribution Line Plot
The Score Distribution Line Plot compares the scoring result distribution at different time periods using a line plot
Before you create a Champion and Challenger Performance report, verify the performance data and model status.

Verify Performance Data and Model Status

Before you can create a Champion and Challenger Performance report:
  1. Click the version folder Version folder and verify that the champion model has been set. The champion model has the check mark Champion model checkmark. If it does not, right-click the model and select Set as Champion.
  2. Ensure that the challenger model is flagged Challenger flag. If it is not, right-click the model and select Flag as Challenger.
  3. Verify that performance monitoring data is available for the champion model and the challenger model. Performance monitoring results must exist for the same performance data using the same time periods and data labels. Click the Resources node and select the file jobstatus.sas7bdat. The Content tab displays performance monitoring status data.
    1. Verify that the UUIDs for the champion and challenger models are in the Model UUID column.
    2. Using the name column and the time column, verify that matching date labels exist for the champion and challenger models for each type of report. If there are multiple date labels for a model for any given report, SAS Model Manager uses the most recent job.

Create a Champion and Challenger Performance Report

To create a monitoring report:
  1. Expand the version folder Version Folder Icon.
  2. Right-click the Reports node and select Reportsthen selectNew Report. The New Report window appears.
    New Report Window Champion and Challenger Performance Report
  3. In the Format list box, select the type of output that you want to create. The default is PDF. Other options are HTML, EXCEL, and RTF.
  4. In the Style list box, select a style for the output. The default is SAS default. Other options are Seaside, Meadow, and Harvest.
  5. In the Report Properties table, complete the Name and Description properties if you do not want to use the default values. The default value for the Name property uses the form ChampionChallenger_DdateTtime. The name can contain letters, spaces, the underscore ( _ ), the hyphen ( - ), and the period ( . ).
  6. Click OK. A message box confirms that the report was created successfully.

See Also

View Reports
Performance Report Output Files
Copyright © SAS Institute Inc. All rights reserved.