The production
model performance reports use performance measurement thresholds to
benchmark and gauge the performance of a
predictive model. When one of the performance measurements exceeds one or more specified indexes or
thresholds, warning and alert events occur. When warning or alert events occur, warning
and alert notifications are automatically sent by email to
recipients whose email address is configured either in the
Edit
Performance Definition wizard or in the batch program that runs the reports.
Use the following assignment statements to set warning and
alert conditions:
alertCondition='alert-condition';
warningCondition='warning-condition';
Note: The condition must be enclosed
in quotation marks if you use SAS code to create the report. An error
occurs if you enclose the condition in quotation marks in the Edit
Performance Definition wizard.
The following indexes
and thresholds can be configured in either the
Edit Performance
Definition wizard or in a batch program that creates the report specifications:
Characteristic report
You can configure the thresholds for the performance indexes P1 and P25. The P1 and
P25 indexes represent the count of
input variables with
deviation index scores exceeding 0.1 and 0.25, respectively. Here is an example of alert and warning
thresholds:
alertCondition='p1>5 or p25>0';
warningCondition='p1>2';
Stability report
You can configure output deviation index scores for a model's
output variable. The output deviation index scores represent the deviation levels in the distribution
of the model's scored
output variables. Here is an example of alert and warning thresholds:
alertCondition='outputDeviation>0.03';
warningCondition='outputDeviation>0.01';
Model Assessment reports
For the Lift, Gini (ROC and Trend), and KS reports, you can configure
threshold values for the following decay statistics.
| lift5Decay |
is the lift performance decay based on the top 5%
of the target population of interest from time A to time B.
|
| lift10Decay |
is the lift performance decay based on the top 10%
of the target population of interest from time A to time B.
|
| lift15Decay |
is the lift performance decay based on the top 15%
of the target population of interest from time A to time B.
|
| lift20Decay |
is the lift performance decay based on the top 20%
of the target population of interest from time A to time B.
|
| giniDecay |
is the performance decay of the Gini index from
time A to time B.
|
| ksDecay |
is the performance decay of the KS statistic from
time A to time B.
|
For the
prediction model MSE report, you can configure the mseDecay statistic threshold values. The mseDecay
statistic is the
performance decay of the MSE statistic from time A to time B.
Here is an example
of alert and warning thresholds:
alertCondition='(lift5Decay>0.15 and lift10Decay>0.12)
or giniDecay>0.1 or ksDecay>0.1';
warningCondition='lift5Decay>0.05';