Create the Challenger Model Performance Data Sets

In this exercise, you run the Define Performance Task wizard to create a performance monitoring task for the challenger model, Reg 1. The performance monitoring task uses the information that you supply in the Define Performance Task wizard to create SAS programs. You then execute the SAS programs to create the performance monitoring data sets that are then used to create the Champion and Challenger Performance report.

Ensure the Project and Model Properties Are Set

The Define Performance Task wizard requires that specific project properties be set before you can run the wizard.
Note: You can skip these steps if you already performed them when creating the performance data sets for the champion model.
  1. Expand the Tutorial3 folder.
  2. Select the Loan project and ensure that the following project properties are set:
    Project Property
    Value
    Training Target Variable
    BAD
    Target Event Value
    1
    Class Target Level
    Binary
    Output Event Probability Variable
    score
  3. Expand the 2012 version and Models folder. Select the champion model and verify that the value of the Score Code Type property is set to DATA step.

Run the Define Performance Task Wizard

To run the Define Performance Task wizard, follow these steps:
  1. Expand the Tutorial3 organizational folder, right-click Loan, and select Define Performance Task from the pop-up menu. The Define Performance Task wizard appears.
    Define Performance Task – Step 1 of 4
  2. In the Output Variables for Stability Analysis table, select the box for the score variable.
  3. In the Input Variables for Characteristic Analysis table, click Select All. Click Next.
  4. On the Warning and Alert Conditions page, accept the default alert and warning conditions by clicking Next. The Data and Model Specifications page appears.
    The performance data options are populated with the values from the last performance task that was defined for the champion model.
    Define Performance Task – Step 3 of 4
  5. Accept the default process method of Standard configuration with the Run scoring task option selected.
    Here are the available data processing method options:
    • To run a standard environment, select Standard configuration. When this data processing method is selected, you can select Run scoring task to run the scoring task code in the performance monitor job. If Run scoring task is not selected, then the performance data source must contain the project output variables and model scoring results.
    • To run the performance monitoring task in a High Performance Analytic environment, select High performance configuration. When this option is selected, the Run scoring task check box is not available. The performance data source must contain the project output variables and model scoring results.
      Note: To use the high performance configuration, you must license the High Performance Analytics server product. Teradata and Greenplum currently support the High-Performance Analytics configuration.
  6. Click Browse to select a Performance data source. Navigate to the location of the Tutorial5 library folder and select HMEQ_2011Q2. Click OK.
  7. Click Validate to validate the performance data set. Click Close when the successful validation message is displayed.
  8. Click the calendar button to select a Data collection date and select June 30, 2011, using the calendar. Click OK.
    Note: The date can be any date within the second quarter of 2011, but must be the same as the date used when creating a performance task for the champion model.
  9. In the Date label box, enter 2011Q2.
    Note: The label is used to identify the performance data in the performance monitoring charts. When you view the charts by using the Performance node in the Project Tree, SAS Model Manager uses the label baseline for the first set of performance data that is created for a champion model. SAS Model Manager does not use the text that you entered in the Date label box. The date label should be the same as the date label that was used when creating a performance task for the champion model.
  10. Select the Reg 1 challenger model from the Models list.
    Data and Model Specifications - Step 3 of 4
  11. Click Next. The Optional E-mail Notifications page of the wizard appears.
    Optional E-mail Notifications – Step 4 of 4
  12. Click Add. The Add Contact dialog box appears.
    Add Contact window
    Enter your e-mail address, and click OK.
  13. Click Finish. The wizard creates the SAS code that can be run to create the performance monitoring data sets.
  14. Execute the SAS program. Under the Loan project, right-click PerformanceMonitor and select Execute. SAS Model Manager executes the performance monitoring program. When the program execution is complete, an information message indicates whether the program ran successfully. Click Close.
  15. Define a performance task and execute the SAS program for the remaining three Tutorial 5 performance data sources. Complete steps 1 through 14 for each performance data source.
    In the Define Performance Task wizard, page 1, select all input and output variables if they are not already selected.
    On page 2, use the default alert and warning conditions. No changes are necessary.
    On page 3, use these values for the Performance data source, Data collection date, and Date label boxes:
    Performance data source
    Data collection date
    Date label
    HMEQ_2011Q3
    September 30, 2011
    2011Q3
    HMEQ_2011Q4
    December 31, 2011
    2011Q4
    HMEQ_2012Q1
    March 31, 2012
    2012Q1
Copyright © SAS Institute Inc. All rights reserved.