To ensure that a champion
model in a production environment is performing efficiently, you can
collect performance data that has been created by the model at intervals
that are determined by your organization. A performance data set is
used to assess model prediction accuracy. It includes all of the required
input variables as well as one or more actual target variables. For
example, you might want to create performance data sets monthly or
quarterly and then use SAS Model Manager to create performance monitoring
reports for each time interval. After you create the reports, you
can view the report charts in SAS Model Manager that give a graphical
representation of the model's performance.
Note: The performance monitoring
reports are designed to work only with classification models that
contain a binary target.
SAS Model Manager provides
the following types of performance monitoring reports:
-
Summaries of the types of information
in project folders such as the number of models, model age distribution,
input variables, and target variables.
-
Reports that detect and quantify
shifts in the distribution of variable values over time that occur
in input data and scored output data.
-
Performance monitoring reports
that evaluate the predicted and actual target values for a champion
model at multiple points in time.
You can create the performance
monitoring reports, except for summaries, using either of the following
methods:
-
In the
SAS Model Manager window, use the
Define Performance Task wizard to generate the SAS code that creates the reports and then
execute the generated code.
-
Write your own SAS program using
the report creation macros that are provided with SAS Model Manager
and submit your program as a batch job. You can run your SAS program
in any SAS session as long as the SAS session can access the SAS Content
Server.
After you create the
reports, you view the report charts in the
SAS Model Manager window by selecting the
Performance node
in the default version. The report charts are interactive charts in
which you modify charts to help you assess the champion model performance.
For example, you can select different variables for the x-axis and
y-axis, filter observations, and change chart types.