Performance Index Warnings and Alerts

The production model performance reports use performance measurement thresholds to benchmark and gauge the performance of a predictive model. When one of the performance measurements exceeds one or more specified indexes or thresholds, warning and alert events occur. When warning or alert events occur, warning and alert notifications are automatically sent by email to recipients whose email address is configured either in the Edit Performance Definition wizard or in the batch program that runs the reports.
Use the following assignment statements to set warning and alert conditions:
alertCondition='alert-condition';
warningCondition='warning-condition';
Note: The condition must be enclosed in quotation marks if you use SAS code to create the report. An error occurs if you enclose the condition in quotation marks in the Edit Performance Definition wizard.
The following indexes and thresholds can be configured in either the Edit Performance Definition wizard or in a batch program that creates the report specifications:
Characteristic report
You can configure the thresholds for the performance indexes P1 and P25. The P1 and P25 indexes represent the count of input variables with deviation index scores exceeding 0.1 and 0.25, respectively. Here is an example of alert and warning thresholds:
alertCondition='p1>5 or p25>0';
warningCondition='p1>2';
Stability report
You can configure output deviation index scores for a model's output variable. The output deviation index scores represent the deviation levels in the distribution of the model's scored output variables. Here is an example of alert and warning thresholds:
alertCondition='outputDeviation>0.03';
warningCondition='outputDeviation>0.01';
Model Assessment reports
For the Lift, Gini (ROC and Trend), and KS reports, you can configure threshold values for the following decay statistics.
lift5Decay is the lift performance decay based on the top 5% of the target population of interest from time A to time B.
lift10Decay is the lift performance decay based on the top 10% of the target population of interest from time A to time B.
lift15Decay is the lift performance decay based on the top 15% of the target population of interest from time A to time B.
lift20Decay is the lift performance decay based on the top 20% of the target population of interest from time A to time B.
giniDecay is the performance decay of the Gini index from time A to time B.
ksDecay is the performance decay of the KS statistic from time A to time B.
For the prediction model MSE report, you can configure the mseDecay statistic threshold values. The mseDecay statistic is the performance decay of the MSE statistic from time A to time B.
Here is an example of alert and warning thresholds:
alertCondition='(lift5Decay>0.15 and lift10Decay>0.12) 
                            or giniDecay>0.1 or ksDecay>0.1';
warningCondition='lift5Decay>0.05';
Last updated: February 22, 2017