Types of Performance Monitoring
Overview of the Types of Performance Monitoring
After a champion model
is in production, you can monitor the performance of the model by
analyzing the performance results. You can create the performance
output interactively using the Edit Performance Definition wizard
on the Performance page of a project or you
can submit batch programs within SAS.
You can create the following
types of performance output:
Summary Results
The Summary results
summarize the number of models, the number of versions, the number
of scoring tests, and the number of reports. The summary information
enables you to compare the contents of folders, projects, and versions.
You view the Summary results by selecting Actions
View Summary.
View Summary.
Data Composition Reports
The
Variable Distribution report shows you the distributions for a variable
in one or more time periods, which enables you to see the differences
and changes over time. The Characteristic and Stability reports detect
and quantify shifts in the distribution of variable values that occur
in input data and scored output data over time. By analyzing these
shifts, you can gain insights on scoring input and output variables.
Model Monitoring Reports
The model monitoring
reports are a collection of performance assessment reports that evaluate
the predicted and actual target values. The model monitoring reports
create several charts:
-
Lift
-
Gini - ROC (Receiver Operating Characteristic)
-
Gini - Trend
-
KS
-
MSE (Mean Squared Error) for prediction models
When you create Data
Composition reports and Model Monitoring reports, you can set performance
index warnings and alerts. When certain thresholds are met, SAS Decision Manager
can send a warning and alert notification to email addresses that
you configure either in the Edit Performance Definition wizard
or in a SAS program.
You view the Data Composition
reports and the Model Monitoring reports on the Results tab
on the Performance page.
Summary Results
The Summary results
summarizes the contents of different folders and projects.
The contents of the
Summary results is dynamic and is updated according to the selected
project. The scope of the information that is reported is defined
by the collection of folders and objects that exist beneath the folder
that is selected.
To view the Summary
results, select ActionsView
Summary.
View
Summary.
Use the following sections
to evaluate and compare the contents of the project:
General
Use the General section
to browse the number of models, the number of versions, and the number
of scoring tests.
Summary of Reports
Use the Summary
of Reports section to browse the number of reports that
are available on the Reports page for the
selected object.
Model Target Variable Report
Use the Model
Target Variable Report to see the frequency with which
target variables are used in the models that exist for the selected
object. Each unique model target variable is reported, listing the
number of models that use that variable as a target variable.
Model Input Variable Report
Use the Model
Input Variable Report to see the frequency with which
input variables are used in the models for a folder or project. Each
unique model input variable is reported, listing the number of models
that use that variable as an input variable.
Data Composition Reports
Variable Distribution Report
Select the Results tab
on the Performance page to view the Variable
Distribution report. The
variable distribution chart is a graphical representation of distributions
over a period of time for the selected variable. Each line plot represents
the data for a specific period of time. The Y-axis is the percentage
of observations in a bin that is proportional to the total count.
To change the variable
that appears in the chart, select a variable from the drop-down list.
Here is an example of
a Variable Distribution report. By placing the cursor over a point
in the chart, you can view the data for that point.
Characteristic and Stability Reports
Together, the Characteristic
and Stability reports detect and quantify shifts that can occur in
the distribution of model performance data, scoring input data, and
the scored output data that a model produces.
Note: For each time period that
you execute a performance definition, SAS Decision Manager
creates a new point on the charts. Line segments between points in
time do not appear on the charts unless you specify at least three
data sources and collection dates as part of the performance definition.
Characteristic Report
The Characteristic
report detects and quantifies the shifts in the distribution of variable
values in the input data over time. These shifts can point to significant
changes in customer behavior that are due to new technology, competition,
marketing promotions, new laws, or other influences.
To find shifts, the
Characteristic report compares the distributions of the variables
in these two data sets:
-
the training data set that was used to develop the model
-
a current data set
If large enough shifts
occur in the distribution of variable values over time, the original
model might not be the best predictive or classification tool to use
with the current data.
The Characteristic
report uses a deviation index to quantify the shifts in a variable's
values distribution that can occur between the training data set and
the current data set. The deviation index is computed for each predictor
variable in the data set, using this equation:
Numeric predictor variable
values are placed into bins for frequency analysis. Outlier values
are removed to facilitate better placement of values and to avoid
scenarios that can aggregate most observations into a single bin.
If the training data
set and the current data set have identical distributions for a variable,
the variable's deviation index is equal to 0. A variable with
a deviation index value that is P1>2 is classified as having a
mild deviation. The Characteristic report uses the performance measure
P1 to count the number of variables that receive a deviation index
value that is greater than 0.1.
A variable that has
a deviation index value that is P1>5 or P25>0 is classified
as having a significant deviation. A performance measure P25 is used
to count the number of variables that have significant deviations,
or the number of input variables that receive a deviation index score
value that is greater than or equal to 0.25.
Stability Report
The Stability report
evaluates changes in the distribution of scored output variable values
as models score data over time, and detects and quantifies shifts
in the distribution of output variable values in the data that is
produced by the models. If an output variable from the training data
set and the output variable from the current data set have identical
distributions, then that output variable's deviation index is
equal to 0. An output variable with a deviation index value that is
greater than 0.10 and less than 0.25 is classified as having a mild
deviation. A variable that has a deviation index value that is greater
than 0.30 is classified as having a significant deviation. Too much
deviation in predictive variable output can indicate that model tuning,
retraining, or replacement might be necessary.
Here is an example of
Characteristic and Stability reports. By placing the cursor over a
point in the chart, you can view the data for that point.
Model Monitoring Reports
Lift Report
The Lift report provides
a visual summary of the usefulness of the information that is provided
by a model for predicting a binary outcome variable. Specifically,
the report summarizes the utility that you can expect by using the
champion model as compared to using baseline information only. Baseline
information is the prediction accuracy performance of the initial
performance monitoring definition or batch program using operational
data.
A monitoring Lift report
can show a model's cumulative lift at a given point in time or
the sequential lift performance of a model's lift over time.
To detect model performance degradation, you can set the Lift report
performance indexes Lift5Decay, Lift10Decay, Lift15Decay, and Lift20Decay.
The data that underlies the Lift report is contained in the report
file mm_lift.sas7bdat. This file is available on the Attachments page.
Here is an example of
a monitoring Lift report. By placing the cursor over a point in the
report, you can view the data for that point.
Gini (ROC and Trend) Report
The Gini
(ROC and Trend)
reports show you the predictive accuracy of a model that has a binary
target. The plot displays sensitivity information about the y-axis
and 1-Specificity information about the x-axis. Sensitivity is the
proportion of true positive events. Specificity is the proportion
of true negative events. The Gini index is calculated for each ROC
curve. The Gini coefficient, which represents the area under the ROC
curve, is a benchmark statistic that can be used to summarize the
predictive accuracy of a model.
Use the monitoring Gini
(ROC and Trend) report to detect degradations in the
predictive power of a model.
The data that underlies
the monitoring Gini (ROC and Trend) report
is contained in the report component file mm_roc.sas7bdat.
The following chart
is an example of a monitoring Gini (ROC and Trend) report.
By placing the cursor over a point in the chart, you can view the
data for that point.
KS Report
The KS report contains
the Kolmogorov-Smirnov (KS) test plots for models with a binary target.
The KS statistic measures the maximum vertical separation, or deviation
between the cumulative distributions of events and non-events. This
trend report uses a summary data set that plots the KS statistic and
the KS probability cutoff values over time.
Use the KS report to
detect degradations in the predictive power of a model. To scroll
through a successive series of KS performance depictions, select a
time interval from the Time Interval list
box. If model performance is declining, it is indicated by the decreasing
distances between the KS plot lines.
To detect model performance
degradation, you can set the ksDecay performance index in the KS report.
The data that underlies
the KS chart is contained in the report component file mm_ks.sas7bdat.
The following report
is an example of a KS report. By placing the cursor over a point in
the chart, you can view the data for that point.
Mean Squared Error Report
The Mean Squared Error
(MSE) report checks the accuracy of a prediction model with an interval
target by comparing the estimation derived from the test data and
the actual outcomes that are associated with the test data for different
time periods.
The following report
is an example of an MSE report.
Copyright © SAS Institute Inc. All Rights Reserved.
Last updated: February 22, 2017