To ensure that a champion
model in a production environment is performing efficiently, you can
collect performance data that has been created by the model at intervals
that are determined by your organization. A performance data set is
used to assess model prediction accuracy. It includes all of the required
input variables as well as one or more actual target variables. For
example, you might want to create performance data sets monthly or
quarterly and then use SAS Model Manager to create performance monitoring
tasks for each time interval. After you create and execute the performance
monitory tasks, you can view the performance data through report charts
in SAS Model Manager that give a graphical representation of the model's
performance. SAS Model Manager enables you to create performance monitoring
reports in PDF, HTML, RTF, and Excel output formats from the
Reports node.
Note: Performance monitoring is
designed to work only with a project that is associated with a classification
model function and has a binary target, or a prediction model function
and has an interval target. Only models that are associated with the
classification and prediction model types and are set as champion
models can be monitored for performance.
SAS Model Manager provides
the following types of output for performance monitoring:
-
Summaries of the types of information
in project folders such as the number of models, model age distribution,
input variables, and target variables.
-
Reports that detect and quantify
shifts in the distribution of variable values over time that occur
in input data and scored output data.
-
Performance monitoring reports
that evaluate the predicted and actual target values for a champion
model at multiple points in time.
You can create the performance
monitoring output, except for summaries, using either of the following
methods:
-
In the
SAS Model Manager window,
use the
Define Performance Task wizard to
generate the SAS code that creates the performance output and then
execute the generated code.
-
Write your own SAS program using
the report creation macros that are provided with SAS Model Manager
and submit your program as a batch job. You can run your SAS program
in any SAS session as long as the SAS session can access the SAS Content
Server.
After you create and
execute the performance task, you view the report charts in the
SAS
Model Manager window by selecting the
Performance node
in the default version. The report charts are interactive charts in
which you modify charts to help you assess the champion model performance.
For example, you can select different variables for the x-axis and
y-axis, filter observations, and change chart types.
If you have flagged
a challenger model to compare with the champion model, you can use
the performance data that you collected for the champion model to
create reports for the challenger model. After all of the performance
monitoring tasks have been run, you can use the New Report window
to create a Champion and Challenger Performance report that compares
the champion model to the challenger model.