Performance Index Warnings and Alerts :: SAS(R) Model Manager 12.3: User's Guide

The production model performance reports use performance measurement thresholds to benchmark and gauge the performance of a predictive model. When one of the performance measurements exceeds one or more specified indexes or thresholds, warning and alert events occur. When warning or alert events occur, warning and alert notifications are automatically sent by e-mail to recipients whose e-mail address is configured either in the Define Performance Task wizard or in the batch program that runs the reports.

Use the following assignment statements to set warning and alert conditions:

alertConditon='alert-condition';
warningCondition='warning-conditon';

The condition must be enclosed in quotation marks if you use SAS code to create the report. An error occurs if you enclose the condition in quotation marks in the Define Performance Task wizard.

The following indexes and thresholds can be configured in either the Define Performance Task wizard or in a batch program that creates the report specifications:

Characteristic report

You can configure the thresholds for the performance indexes P1 and P25. The P1 and P25 indexes represent the count of input variables with deviation index scores exceeding 0.1 and 0.25, respectively. Here is an example of alert and warning thresholds:

alertCondition='p1>5 or p25>0';
warningCondition='p1>2';

Stability report

You can configure output deviation index scores for a model's output variable. The output deviation index scores represent the deviation levels in the distribution of the model's scored output variables. Here is an example of alert and warning thresholds:

alertCondition='outputDeviation>0.03';
warningCondition='outputDeviation>0.01';

Model Assessment reports

For the Lift, Gini (ROC and Trend), and KS reports, you can configure threshold values for the following decay statistics.

lift5Decay	is the lift performance decay based on the top 5% of the target population of interest from time A to time B.
lift10Decay	is the lift performance decay based on the top 10% of the target population of interest from time A to time B.
lift15Decay	is the lift performance decay based on the top 15% of the target population of interest from time A to time B.
lift20Decay	is the lift performance decay based on the top 20% of the target population of interest from time A to time B.
giniDecay	is the performance decay of the Gini index from time A to time B.
ksDecay	is the performance decay of the KS statistic from time A to time B.

For the prediction model MSE report, you can configure the mseDecay statistic threshold values. The mseDecay statistic is the performance decay of the MSE statistic from time A to time B.

Here is an example of alert and warning thresholds:

alertCondition='(lift5Decay>0.15 and lift10Decay>0.12) 
                            or giniDecay>0.1 or ksDecay>0.1';
warningCondition='lift5Decay>0.05';

The following table is an example of a warnings and alerts notification table:

The warnings and alerts notification table displays the computed performance indexes and performance decay statistics that were calculated for the model, as well as summaries of the alert and warning threshold settings that were specified for the model. The calculated statistics are compared with the alert and warning threshold settings.

When an alertEvaluation or warningEvaluation column displays a True value, the warnings and alerts table is e-mailed to the configured recipients. When the value is False, no e-mail notification is sent.

Performance Index Warnings and Alerts

See Also