Create the Challenger Model Performance Data Sets

In this exercise, you run the Define Performance Task wizard to create a performance monitoring task for the challenger model, Reg 1. The performance monitoring task uses the information that you supply in the Define Performance Task wizard to create SAS programs. You then execute the SAS programs to create the performance monitoring data sets that are then used to create the Champion and Challenger Performance report.

Ensure the Project and Model Properties Are Set

The Define Performance Task wizard requires that specific project properties be set before you can run the wizard.
Note: You can skip these steps if you already performed them when creating the performance data sets for the champion model.
  1. Expand the Tutorial3 folder.
  2. Select the Loan project and ensure that the following project properties are set:
    Project Property
    Value
    Training Target Variable
    BAD
    Target Event Value
    1
    Class Target Level
    Binary
    Output Event Probability Variable
    score
  3. Expand the 2013 version and Models folder. Select the champion model and verify that the value of the Score Code Type property is set to DATA step.

Run the Define Performance Task Wizard

To run the Define Performance Task wizard:
  1. Expand the Tutorial3 organizational folder, right-click Loan, and select Define Performance Task from the pop-up menu. The Define Performance Task wizard appears.
    Define Performance Task – Step 1 of 4
  2. In the Output Variables for Stability Analysis table, select the box for the score variable.
  3. In the Input Variables for Characteristic Analysis table, click Select All. Click Next.
  4. On the Warning and Alert Conditions page, accept the default alert and warning conditions by clicking Next. The Data and Model Specifications page appears.
    The performance data options are populated with the values from the last performance task that was defined for the champion model.
    Define Performance Task – Step 3 of 4
  5. Accept the default process method of Standard configuration with the Run model score code option selected.
  6. Select the Reg 1 challenger model from the Models list.
  7. Click Add Data Source Button, click the empty cell in the Data Source column, and click Browse. Navigate to the location of the Tutorial5 library folder and select HMEQ_2012Q2. Click OK.
  8. Click the empty cell in the Collection Date column and click Calendar button. Select June 30, 2012 and click OK.
    Note: The date that you select is used by SAS Model Manager for sequencing data and does not appear in any charts. If the performance data is for the first quarter of 2013, the date could be any date between January 1 and March 31, 2013.
  9. To add a label for the date, enter 2012Q2 in the Report Label column. The report label represents the time point of the performance data source. Because the report label appears in the performance charts, use a label that has not been used in another report, is short, and is understandable (for example, 2013Q1 or 2013).
    Note: Duplicate report labels result in previous performance results being overwritten.
    When the performance monitoring report is for a challenger model and when the data will be used in a Champion and Challenger Performance report, some requirements apply. Namely, the value of the Report Label field must be the same report label that was used for the same time period when the performance monitoring report was run for the champion model. For example, if the report label for the champion model’s data from the first quarter of 2013 is 2013Q1, the date label for the challenger model’s data from the first quarter of 2013 must be 2013Q1.
  10. Click Validate to verify that the selected input variables and target variables are included in each performance data source. Click Close when the successful validation message is displayed.
  11. Define a performance task and execute the SAS program for the remaining three Tutorial 5 performance data sources. Complete steps 7 through 10 for each performance data source.
    On the Define Performance Task wizard, page 1, select all input and output variables if they are not already selected.
    On page 2, use the default alert and warning conditions. No changes are necessary.
    On page 3, use these values for the Data Source, Collection Date, and Report Label boxes:
    Data Source
    Collection Date
    Report Label
    HMEQ_2012Q3
    September 30, 2012
    2012Q3
    HMEQ_2012Q4
    December 31, 2012
    2012Q4
    HMEQ_2013Q1
    March 31, 2013
    2013Q1
  12. Click the Default server list and select a SAS Application Server where the performance task is to execute.
  13. Click Next. The Optional E-mail Notifications page of the wizard appears.
    Optional E-mail Notifications – Step 4 of 4
  14. Click Add. The Add Contact dialog box appears.
    Add Contact window
    Enter your e-mail address, and click OK.
  15. Click Finish. The wizard creates the SAS code that can be run to create the performance monitoring data sets.
  16. Execute the SAS program. Under the Loan project, right-click Performance Monitor and select Execute. SAS Model Manager executes the performance monitoring program. When the program execution is complete, an information message indicates whether the program ran successfully. Click Close.
    Note: To create a schedule to execute the task to run at a specific time, see Schedule Performance Monitoring Tasks.
Copyright © SAS Institute Inc. All rights reserved.