Your project plan might
include a schedule to monitor the champion model performance, or your
plan might require that you monitor the performance at any time. For
each time period that you monitor the champion model, you take a snapshot
of the data for that time period and use that data as the performance
data source for creating the monitoring reports.
You can create the monitoring
reports from the Project Tree in the
SAS Model Manager window
by creating and executing a performance task, or you can submit batch
programs to create the reports. Both methods require the same information.
Both methods can process one or more performance data sources. When
you create a performance task, you can specify one or more data sources
to process. When you use a batch program, you use a separate DATA
step to process each data source.
If you run batch programs,
you can find example programs in the sashelp.modelmgr.source catalog.
These reports’ filenames are
reportexamplex,
where
x is a number from 1
to 4.
The following table
lists the tasks that are required to create performance reports:
|
|
Reports Created by Using
the Define Performance Task Wizard
|
Reports Created Using
SAS Programs That Run in Batch
|
Create a folder structure
for report files
|
The folder structure
is inherent in the Project Tree. No action is necessary.
|
Create a folder structure
on a local computer.
|
|
|
The performance data
is one or more SAS data sets that are a snapshot of model output.
They can be registered in SAS Management Console or they can be accessed
by using a libref that has been defined by using the Edit
Start-up Code window.
|
The performance data
is used to assess model prediction accuracy. It includes all of the
required scoring input variables as well as one or more actual target
variables. You can store performance data sets anywhere as long as
they can be accessed by the SAS session that runs the batch program.
The data sets do not need to be registered with SAS Management Console.
|
Ensure access to the
champion or challenger model
|
This process is performed
by the Define Performance Task wizard. No
action is necessary.
|
Run the %MM_GetModels()
macro to extract the champion model in a channel to the local computer.
|
Map model and project
output variables.
|
Map the model and project
output variables using the Project Tree.
|
Map the model and project
output variables using the Project Tree.
|
Define report specifications
|
The report specification
are derived from project data and input that you specify in the Define
Performance Task wizard. The wizard generates the SAS
code to create the performance reports.
|
Write the following
DATA steps:
|
Specify the report execution
operational environment
|
The operational environment
is known to SAS Model Manager. No action is necessary.
|
Define the required
macro variables that are used by the %MM_RunReports() macro.
|
|
|
Execute the code from
the Performance Monitor node that was generated
by the Define Performance Task wizard or
schedule the performance task from the Performance Monitor pop-up
menu. The data sets that underlie the monitoring reports are stored
in the Resources folder.
|
Create a DATA step that
points to the performance data sets and execute the %MM_RunReports()
macro. The data sets that underlie the monitoring reports are stored
in the Resources folder when the reports
are created in production mode. In Test mode, the monitoring reports
data sets reside in the location specified in the mm_jobs.project
data set.
|
|
|
Select the Performance folder
for the champion model’s version to view the reports.
|
Select the Performance folder
for the champion model’s version to view the reports.
|