Monitoring Champion Models

Your project plan might include a schedule to monitor the champion model performance, or your plan might require that you monitor the performance at any time. For each time period that you monitor the champion model, you take a snapshot of the data for that time period and use that data as the performance data source for creating the monitoring reports.

You can create monitoring reports by creating and executing a performance definition, or you can submit batch programs to create the reports. Both methods require the same information. Both methods can process one or more performance data sources. When you create a performance definition, you can specify one or more data sources to process. When you use a batch program, you use a separate DATA step to process each data source.

If you run batch programs, you can find example programs in the sashelp.modelmgr.source catalog. These reports’ filenames are reportexamplex, where x is a number from 1 to 4.

The following table lists the definitions that are required to create performance reports:

Definition	Reports Created by Using the Edit Performance Definition Wizard	Reports Created Using SAS Programs That Run in Batch
Create a folder structure for report files	The folder structure is inherent in the Project. No action is necessary.	Create a folder structure on a local drive of the SAS Model Manager server.
Obtain performance data	The performance data is one or more SAS data sets that are a snapshot of model output. They can be registered in SAS Management Console or they can be accessed by using a libref that has been defined by using the Edit Start-up Code window.	The performance data is used to assess model prediction accuracy. It includes all of the required scoring input variables as well as one or more actual target variables. You can store performance data sets anywhere as long as they can be accessed by the SAS session that runs the batch program. The data sets do not need to be registered with SAS Management Console.
Ensure access to the champion or challenger model	This process is performed by the Edit Performance Definition wizard. No action is necessary.	Run the %MM_GetModels() macro to extract the champion model in a channel to a local drive of the SAS Model Manager server.
Map model and project output variables.	Map the model and project output variables using the Project Tree.	Map the model and project output variables using the Project Tree.
Define report specifications	The report specification are derived from project data and input that you specify in the Edit Performance Definition wizard. The wizard generates the SAS code to create the performance reports.	Write the following DATA steps: mm_jobs.project mm_jobs.emailaddr mm_jobs.reportdef mm_jobs.jobtime
Specify the report execution operational environment	The operational environment is known to SAS Model Manager. No action is necessary.	Define the required macro variables that are used by the %MM_RunReports() macro.
Run the reports	Execute the code from the Performance page that was generated by the Edit Performance Definition wizard or schedule the performance definition from the Performance page. The data sets that underlie the monitoring reports are stored in the Results tab.	Create a DATA step that points to the performance data sets and execute the %MM_RunReports() macro. The data sets that underlie the monitoring reports are stored in the Results tab when the reports are created in production mode. In Test mode, the monitoring reports data sets reside in the location specified in the mm_jobs.project data set.
View the reports	Select the Performance page for the champion model to view the reports.	Select the Performance page for the champion model to view the reports.

Last updated: June 12, 2017