Formatting Performance Monitoring Reports
About Monitoring Reports
After you execute a
performance task from the SAS Model Manager window or run the %MM_RunReports() macro in production mode, as
a batch job, SAS Model Manager stores the output data sets in the
default version Resources folder. You can
use the New Reports wizard to format the
performance monitoring results in PDF, HTML, RTF, or Excel output
formats, or you can view the performance monitoring results by selecting
the default version Performance node. When
you create monitoring reports using the New Reports Wizard, the report creates the following charts:
Assessment charts
summarize the utility that one can expect by using the respective
models, as compared to using only baseline information. Assessment
charts can present a model's lift at a given point in time or the
sequential lift performance of a model's lift over time. A monitoring
report creates the following assessment charts:
Sensitivity is the
proportion of true positive events and specificity is the proportion
of true negative events. The Gini - ROC Chart plots Sensitivity on
the Y axis and 1 - Specificity on the X axis.
When the Gini - ROC
Chart is created, the Gini index for each ROC curve is also created.
The Gini coefficient represents the area under the ROC curve and is
a benchmark statistic that can be used to summarize the predictive
accuracy of a model. The Gini - Trend Chart plots a model's Gini index
scores over time, and these are used to monitor model degradation
over time.
The KS Chart uses the
Kolmogorov-Smirnov statistic to measure the maximum vertical separation,
or deviation between the cumulative distributions of events and non-events.
When you create a Kolmogorov-Smirnov
report, the KS statistic and the corresponding probability cutoff
are computed for each Kolmogorov-Smirnov table. The KS Trend Chart
uses a summary data set that plots the KS Statistic and the probability
cutoff values over time. The KS Trend Chart is used to monitor model
degradation over time.
Before you create a
monitoring report, you must ensure that certain project and model
properties are set. For more information, see Verify Project and Model Property Settings.
Monitoring Report Output Files
The New
Reports Wizard stores the monitoring report output files
in a report node under the Reports folder.
The name of the report node is the value of the Name field that you specified in the New Report Wizard Report
Properties table.