Create the Champion Model Performance Data Sets for a Classification Project

In this exercise, you run the Define Performance Task wizard for the Loan classification modeling project to create a performance monitoring task for the champion model, Tree 1. The performance monitoring task uses the information that you supply in the Define Performance Task wizard to create SAS programs. You then execute the SAS programs to create the performance monitoring data sets.

Ensure the Project and Model Properties Are Set

The Define Performance Task wizard requires that specific project properties be set before you can run the wizard.

Expand the Tutorial3 folder.

Select the Loan project and ensure that the following project properties are set:

Project Property	Value
Training Target Variable	`BAD`
Target Event Value	`1`
Class Target Level	Binary
Output Event Probability Variable	score

Expand the 2013 version and Models folder. Select the champion model and verify that the value of the Score Code Type property is set to DATA step.

Run the Define Performance Task Wizard

To run the Define Performance Task wizard:

Expand the Tutorial3 organizational folder, right-click Loan, and select Define Performance Task from the pop-up menu. The Define Performance Task wizard appears.
In the Output Variables for Stability Analysis table, select the box for the score variable.
In the Input Variables for Characteristic Analysis table, click Select All. Click Next.
On the Warning and Alert Conditions page, accept the default alert and warning conditions by clicking Next. The Data and Model Specifications page appears.
Accept the default process method of Standard configuration with the Run model score code option selected.
Here are the available data processing method options:
- To run a standard environment, select Standard configuration. When this data processing method is selected, you can select Run model score code to run the scoring task code in the performance monitor job. If Run model score code is not selected, then the performance data source must contain the project output variables and model scoring results.
- To run the performance monitoring task in a SAS High-Performance Analytics environment, select High performance configuration. When this option is selected, the Run model score code check box is not available. The performance data source must contain the project output variables and model scoring results.
  
  Note: To use the high-performance configuration, you must license SAS High-Performance Analytics Server. Teradata and Greenplum currently support the high-performance configuration.
Select the Tree 1 champion model from the Models list. If challenger models have been flagged, the challenger modes are listed in the Models table.
Click , click the empty cell in the Data Source column, and click Browse. Navigate to the location of the Tutorial5 library folder and select HMEQ_2012Q2. Click OK.

Note: When you add multiple tables, the baseline performance table is the table with the earliest collection date.
Click the empty cell in the Collection Date column and click . Select June 30, 2013 and click OK.

Note: The date that you select is used by SAS Model Manager for sequencing data and does not appear in any charts. If the performance data is for the first quarter of 2013, the date could be any date between January 1 and March 31, 2013.
To add a label for the date, enter 2012Q2 in the Report Label column. The report label represents the time point of the performance data source. Because the report label appears in the performance charts, use a label that has not been used in another report, is short, and is understandable (for example, 2013Q1 or 2013).

Note: Duplicate report labels result in previous performance results being overwritten.

When the performance monitoring report is for a challenger model and when the data will be used in a Champion and Challenger Performance report, some requirements apply. Namely, the value of the Report Label field must be the same report label that was used for the same time period when the performance monitoring report was run for the champion model. For example, if the report label for the champion model’s data from the first quarter of 2013 is 2013Q1, the report label for the challenger model’s data from the first quarter of 2013 must be 2013Q1.
Click Validate to verify that the selected input variables and target variables are included in each performance data source. Click Close when the successful validation message is displayed.

Define a performance task and execute the SAS program for the remaining three Tutorial 5 performance data sources. Complete steps 7 through 10 for each performance data source.

On the Define Performance Task wizard, page 1, select all input and output variables if they are not already selected.

On page 2, use the default alert and warning conditions. No changes are necessary.

On page 3, use these values for the Data Source, Collection Date, and Report Label boxes:

Data Source	Collection Date	Report Label
HMEQ_2012Q3	September 30, 2012	2012Q3
HMEQ_2012Q4	December 31, 2012	2012Q4
HMEQ_2013Q1	March 31, 2013	2013Q1

Click the Default server list and select a SAS Application Server where the performance task is to execute.
Click Next. The Optional E-mail Notifications page of the wizard appears.
Click Add. The Add Contact dialog box appears.

Enter your e-mail address, and click OK.
Click Finish. The wizard creates the SAS code that can be run to create the performance monitoring data sets.
Execute the SAS program. Under the Loan project, right-click PerformanceMonitor and select Execute. SAS Model Manager executes the performance monitoring program. When the program execution is complete, an information message indicates whether the program ran successfully. Click Close.

Note: To create a schedule to execute the task to run at a specific time, see Schedule Performance Monitoring Tasks.
Expand Performance Monitor. Here you can see the SAS program that created the performance monitoring data sets and the resulting SAS log. Click both files to see the file contents in the Content pane.
Expand the Resources folder under the default version 2013. The Resources folder contains the data sets that are created by executing the performance task. When a performance task is executed the first time for a given champion model, the performance task creates the initial data sets that are used for plotting the model performance charts. In executing subsequent performance tasks that use new performance data for the given champion model, SAS Model Manager appends the resulting data sets to the existing data sets. All of the data in the model performance data sets for a given champion model is used to plot the model performance charts.

Click on any file to see the contents of that file in the Content pane.
Select Performance. The Performance node displays the champion model performance data as a graph and as a data set.

Note: To view at least one line segment in Characteristic and Stability graphs, SAS Model Manager requires performance data sets from three performance task executions, at a minimum.