Create the Champion Model Performance Data Sets for a Classification
Project
In this exercise, you run the Define Performance
Task wizard for the Loan classification
modeling project to create performance monitoring task for the champion
model, Tree 1. The performance monitoring
task uses the information that you supply in the Define
Performance Task wizard to create SAS programs. You then
execute the SAS programs to create the performance monitoring data
sets.
Ensure the Project and Model Properties Are Set
The
Define
Performance Task wizard requires that specific project
properties be set before you can run the wizard.
-
Expand the
Tutorial3 folder.
-
Select the
Loan project
and ensure that the following project properties are set:
|
|
|
|
|
|
|
|
|
|
|
|
Output Event
Probability Variable
|
|
-
Expand the
2012 version
and
Models folder. Select the champion model
and verify that the value of the
Score Code Type property
is set to
DATA step.
Run the Define Performance Task Wizard
To run the
Define
Performance Task wizard, follow these steps:
-
Expand the
Tutorial3 organizational
folder, right-click
Loan, and select
Define
Performance Task from the pop-up menu. The
Define
Performance Task wizard appears.
-
In the
Output
Variables for Stability Analysis table, select the box
for the
score variable.
-
In the
Input
Variables for Characteristic Analysis table, click
Select
All. Click
Next.
-
On the
Warning
and Alert Conditions page, accept the default alert and
warning conditions by clicking
Next. The
Data
and Model Specifications page appears.
-
Accept the default process
method of
Standard configuration with the
Run
scoring task option selected.
Here are the available
data processing method options:
-
To run a standard environment,
select
Standard configuration. When this
data processing method is selected you can select
Run
scoring task to run the scoring task code in the performance
monitor job. If
Run scoring task is not selected,
then the performance data source must contain the project output variables
and model scoring results.
-
To run the performance monitoring
task in a High Performance Analytic environment, select
High
performance configuration. When this option is selected,
the
Run scoring task check box is not available.
The performance data source must contain the project output variables
and model scoring results.
Note: To use the high performance
configuration, you must license the High Performance Analytics server
product. Teradata and Greenplum currently support the High-Performance
Analytics configuration.
-
Click
Browse to
select a
Performance data source. Navigate
to the location of the
Tutorial5 library
folder and select
HMEQ_2011Q2. Click
OK.
-
Click
Validate to
validate the performance data set. Click
Close when
the successful validation message is displayed.
-
Click the calendar button
to select a
Data collection date and select
June
30, 2011 using the calendar. Click
OK.
Note: The date can be any date
within the second quarter of 2011.
-
In the
Date
label box, enter
2011Q2.
Note: The label is used to identify
the performance data in the performance monitoring charts. When you
view the charts by using the Performance node in the Project Tree,
SAS Model Manager uses the label
baseline for
the first set of performance data that is created for a champion model.
SAS Model Manager does not use the text that you entered in the
Date
label box.
-
Select the
Tree
1 champion model from the
Models list.
If challenger models have been flagged, the challenger modes are listed
in the
Models table.
-
Click
Next.
The
Optional E-mail Notifications page of
the wizard appears.
-
Click
Add.
The
Add Contact dialog box appears.
Enter your e-mail address,
and click
OK.
-
Click
Finish.
The wizard creates the SAS code that can be run to create the performance
monitoring data sets.
-
Execute the SAS program.
Under the
Loan project, right-click
PerformanceMonitor and
select
Execute. SAS Model Manager executes
the performance monitoring program. When the program execution is
complete, an information message indicates whether the program ran
successfully. Click
Close.
-
Expand
PerformanceMonitor.
Here you can see the SAS program that created the performance monitoring
data sets and the resulting SAS log. Click both files to see the file
contents in the
Content pane.
-
Expand the
Resources folder
under the default version
2012. The
Resources folder
contains the data sets that are created by executing the performance
task. When a performance task is executed the first time for a given
champion model, the performance task creates the initial data sets
that are used for plotting the model performance charts. In executing
subsequent performance tasks that use new performance data for the
given champion model, SAS Model Manager appends the resulting data
sets to the existing data sets. All of the data in the model performance
data sets for a given champion model is used to plot the model performance
charts.
Click on any file to
see the contents of that file in the
Content pane.
-
Select
Performance.
The Performance node displays the champion model performance data
as a graph and as a data set.
Note: To view at least one line
segment in Characteristic and Stability graphs, SAS Model Manager
requires performance data sets from three performance task executions,
at a minimum.
-
Define a performance
task and execute the SAS program for the remaining three Tutorial
5 performance data sources. Complete steps 1 through 14 for each performance
data source.
On the
Define
Performance Task wizard, page 1, select all input and
output variables if they are not already selected.
On page 2, use the default
alert and warning conditions. No changes are necessary.
On page 3, use these
values for the
Performance data source,
Data
collection date, and
Date label boxes:
Copyright © SAS Institute Inc. All rights reserved.
