Creating a Performance Table

About Performance Tables

Here are the requirements for a performance table:

the input variables that you want reported in a Characteristic report
if you have score code:
- all input variables that are used by the champion model or challenger models
- all output variables that are used by the champion model or challenger models
if you have no score code:
- the actual value of the dependent variable and the predicted score variable
- all output variables that you want reported in a Stability Report

You create a performance table by taking a sampling of data from an operational data mart. Make sure that your sampling of data includes the target or response variables. The data that you sample must be prepared by using your extract, transform, and load business processes. When this step is complete, you can then use that data to create your performance table.

As part of the planning phase, you can determine how often you want to sample operational data to monitor the champion model performance. Ensure that the operational data that you sample and prepare represents the period that you want to monitor. For example, to monitor a model that determines whether a home equity loan could be bad, you might want to monitor the model every six months. To do this, you would have two performance tables a year. The first table might represent the data from January through June, and the second table might represent the data from July through December.

Here is another example. You might want to monitor the performance of a champion model that predicts the delinquency of credit card holders. In this case, you might want to monitor the champion model more frequently, possibly monthly. You would need to prepare a performance table for each month in order to monitor this champion model.

In addition to planning how often you sample the operational data, you can also plan how much data to sample and how to sample the data. Examples in this section show you two methods of sampling data and naming the performance tables. You can examine the sampling methods to determine which might be best for your organization.

Naming a Performance Table for Use with the Define Performance Task Wizard

The Define Performance Task wizard is a graphical interface to assist you in creating a performance task to monitor the champion model performance. When you run the Define Performance Task wizard, you specify a performance table that has been registered using SAS Management Console, or you specify a performance table on a local or network drive. When you create a performance table, you can collect and name the performance table using a method that is most suitable for your business process. Here are two methods of collecting performance data:

Method 1: You periodically take a snapshot of an operational data set to create a performance data set. Each time you take a snapshot, you give the performance data set a new name. Each performance data set must be registered in SAS Management Console or a libref must be defined for the local or network drive where the data set resides. For each time interval, you name a new performance data source when you run the Define Performance Task wizard.
Method 2: You take a snapshot of the operational data set to create a performance data set over time, and you reuse the same name for each performance data set every time you take a snapshot. If the data set resides in the SAS Metadata Repository, you register the performance data set with SAS Management Console only once. Each time you take a snapshot, you replace the performance data set at the location where the performance data set is registered in SAS Management Console or where the data set is saved on the local or network drive.

When you run the Define Performance Task wizard, the name of the performance data source does not change. Because you used the performance data source static name as the Default Performance Table in the project properties, the Performance data source box in the wizard is completed by SAS Model Manager.

Create a Performance Table

You can use the following DATA steps as examples to create your performance tables.

This DATA step creates a performance table using 5,000 sequential observations from the operational data:

data hmeqtabl.perform;
   set hmeqop.JulDec (firstobs=12001 obs=17000);
run;

This DATA step creates a performance table from operational data for the past six months of the year. The IF statement creates a random sampling of approximately 10% of the operational data:

data hmeqtabl.perform;
  set hmeqop.JulDec;
  if ranuni(1234) < 0.1;
run;

Creating a Performance Table

About Performance Tables

Naming a Performance Table for Use with the Define Performance Task Wizard

See Also

Create a Performance Table