The production model
performance reports use performance measurement thresholds to benchmark
and gauge the performance of a predictive model. When one of the performance
measurements exceeds one or more specified indexes or thresholds,
warning and alert events occur. When warning or alert events occur,
warning and alert notifications are automatically sent by email to
recipients whose email address is configured either in the Edit
Performance Definition wizard or in the batch program
that runs the reports.
Use the following assignment
statements to set warning and alert conditions:
alertCondition='alert-condition';
warningCondition='warning-condition';
Note: The condition must be enclosed
in quotation marks if you use SAS code to create the report. An error
occurs if you enclose the condition in quotation marks in the Edit
Performance Definition wizard.
The following indexes
and thresholds can be configured in either the
Edit Performance
Definition wizard or in a batch program that creates
the report specifications:
Characteristic report
You can configure the
thresholds for the performance indexes P1 and P25. The P1 and P25
indexes represent the count of input variables with deviation index
scores exceeding 0.1 and 0.25, respectively. Here is an example of
alert and warning thresholds:
alertCondition='p1>5 or p25>0';
warningCondition='p1>2';
Stability report
You can configure output
deviation index scores for a model's output variable. The output deviation
index scores represent the deviation levels in the distribution of
the model's scored output variables. Here is an example of alert and
warning thresholds:
alertCondition='outputDeviation>0.03';
warningCondition='outputDeviation>0.01';
Model Assessment reports
For the Lift, Gini
(ROC and Trend), and KS reports, you can configure threshold values
for the following decay statistics.
| lift5Decay |
is the lift performance decay based on the top 5%
of the target population of interest from time A to time B.
|
| lift10Decay |
is the lift performance decay based on the top 10%
of the target population of interest from time A to time B.
|
| lift15Decay |
is the lift performance decay based on the top 15%
of the target population of interest from time A to time B.
|
| lift20Decay |
is the lift performance decay based on the top 20%
of the target population of interest from time A to time B.
|
| giniDecay |
is the performance decay of the Gini index from
time A to time B.
|
| ksDecay |
is the performance decay of the KS statistic from
time A to time B.
|
For the prediction
model MSE report, you can configure the mseDecay statistic threshold
values. The mseDecay statistic is the performance decay of the MSE
statistic from time A to time B.
Here is an example
of alert and warning thresholds:
alertCondition='(lift5Decay>0.15 and lift10Decay>0.12)
or giniDecay>0.1 or ksDecay>0.1';
warningCondition='lift5Decay>0.05';