Define the Report Specifications

Overview of Code to Define Report Specifications

Before you can create a monitoring report for a project, you must create several data sets that define the report specifications:
mm_jobs.project defines the project information, such as the project UUID, project variables, and the model repository URL for the project. It is recommended that you create only one observation in this data set.
mm_jobs.emailaddr defines the e-mail addresses for the recipients of job status and the notification flags for alert and warning notifications.
mm_jobs.reportdef defines the types of reports to create and the warning and alert conditions that are associated with those reports.
mm_jobs.jobtime defines the date and time to run the reports and a label that describes the time performance data set period.
The code that you write to create the report specifications might need to be run only after it is created and only whenever it is modified. These data sets might not need to be created every time you want to create reports.

Required Libref

To create the report specifications, you need to define the following libref:
mm_jobs
defines the local path to the folder that contains the report job files.
Example: libname mm_jobs "c:\mmReports\HMEQ";

Project Specifications

DATA Step mm_jobs.project

This DATA step defines the project specifications.
/**********************************************/
/* DATA step mm_jobs.project                  */
/*                                            */
/* Create a data set to initialize the        */
/* performance monitoring batch program       */
/* project specifications                     */
/**********************************************/

DATA mm_jobs.project;
   length testDestination %150
      projectuuid $36
      projectpath $2000
      projectAlias $50
      precode $32000
      isActive $1
      notes $500;

   isActive='Y';

/**********************************************/
/* Specify the destination for the report     */
/* output when the report is run in TEST mode */
/**********************************************/

testDestination='reportOutputPath';

/***********************************************/
/* Specify the project UUID                    */
/***********************************************/

projectuuid='projectuuid';

/************************************************/
/* Map project specification variables to       */
/* macro variables                              */
/************************************************/

precode='
   %let _MM_EventProbVar=eventProbabilityVariable;
   %let _MM_TargetVar=targetVariable;
   %let _MM_TargetEvent=targeEventValue;
   %let _MM_ReportDatasrc=scoreIn.dataSetName;
   %let _MM_KeepVars=variablesToKeep;
   %let _MM_DropVars=variableToDrop;';

/*************************************************/
/* Specify the URL to the project in the model   */
/* repository and a description of the project   */
/*************************************************/

projectPath='projectURL';
projectAlias='alternateProjectName';

run;

Variable Descriptions for mm_jobs.project

The following variables are used in the mm_jobs.project DATA step:
isActive='Y | N'
specifies whether to enable the project definitions. Valid values are Y (yes) and N (no). Specifying N means that project files do not need to be removed from the local computer to deactivate a project entry. Enclose the value of isActive in quotation marks.
Interaction: Always set isActive='Y' when the data set mm_jobs.project has only one observation.
testDestination='reportOutputPath';
specifies the local path that contains the output files that are created when the %MM_RunReports() macro report mode macro variable _MM_ReportMode is set to TEST. Enclose the value of testDestination in quotation marks.
Example: testDestination='c:\mmReports\HMEQ\testOutput';
projectuuid
specifies the universally unique identifier for a SAS Model Manager modeling project. To obtain the project UUID, in the SAS Model Manager, open a project and select System on the Properties page. You can copy the UUID from the UUID property. projectuuid is used to redirect reporting job output data sets to the appropriate project folders in the model repository when the %MM_RunReports() macro is run in Production mode.
Note: If you copy the UUID from SAS Model Manager, you might need to remove leading text and spaces that are copied with the UUID.
precode='macroVariableDefinitions'
specifies the macro variables that are used by the %MM_RunReports() macro. Enclose the value of the precode variable in quotation marks.
%let _MM_EventProbVar=outputEventProbabilityVariable;
specifies the output event probability variable name. To obtain the name, select the project in the Project Tree and expand Specific Properties. Use one of the values for the Output Event Probability Variable property list box.
%let _MM_TargetVar=targetVariable;
specifies the target variable name. To obtain the name, select the project in the Project Tree and expand Specific Properties. The target event variable is found in the property Training Target Variable. If a target variable is not specified, see your performance data set or the model for the name of the target variable.
%let _MM_TargetEvent=targetEventValue;
specifies the target event value. To obtain the name, select the project in the Project Tree and expand Specific Properties. The value is found in the property Target Event Value. If a target event value is not specified, see your performance data set or the model to determine the value.
Requirement: The value of _MM_TargetEvent must be an unformatted, raw value even if the original target variable has a SAS format applied to it.
%let _MM_ReportDatasrc=scoreIn.dataSetName;
specifies the libref and the data set name for the performance data set that is being analyzed.
If you process multiple data sets at one time, you can specify a generic data set name in this macro definition. The generic data set name is used to process all performance data sets. Before you run the %MM_RunReports() macro, you should create a DATA step with the name scoreIn.genericDataSetName, where the only statement in the DATA step is the SET statement that specifies the performance data set to process.
%let _MM_KeepVars=variablesToKeep;
specifies one or more output variables, separated by a space, that are kept in the performance data source to create the Stability report data set.
%let _MM_DropVars=variablesToDrop;
specifies one or more input variables, separated by a space, that are dropped from the performance data source to create the Characteristic report data set.
projectPath='projectURL'
specifies the project URL. To obtain the project URL, select the project in the Project Tree and expand System Properties. You can copy the URL from the URL property. The project URL is used for information purposes only; it is not used to access project resources. projectURL is dynamically retrieved when the %MM_RunReports() macro runs. Enclose the value of projectPath in quotation marks.
Note: If you copy the URL from the window, you might need to remove leading text and spaces that are copied with the URL.
projectAlias='alternateProjectName'
specifies an alternate project name. The alternate project name can be used to help identify the project when the projectPath is long. If you do not have an alternate project name, you can leave this variable blank. Enclose the value of projectAlias in quotation marks.
notes='userNotes'
specifies any notes that the user might want to add to the project specifications. Enclose the value of notes in quotation marks.

E-mail Recipient Specifications

DATA Step mm_jobs.emailaddr

This DATA step defines the e-mail recipient specifications:
/****************************************************/
/* DATA mm_jobs.emailaddr                           */
/*                                                  */
/* Create a data set that specifies the e-mail      */
/* addresses of the users who will receive job      */
/* status notification as well as warnings and      */
/* alerts.                                          */
/****************************************************/

DATA mm_jobs.emailaddr;
   length address $50 sendAlertWarning sendJobStatus $1;
   address='e-mailAddress';
      sendAlertWarning='Y';
      sendJobStatus='N';
      output;
   address='e-mailAddress';
      sendAlertWarning='Y';
      sendJobStatus='Y';
      output;
run;
    

Variable Descriptions for mm_jobs.emailaddr

The following variables are used in the mm_jobs.emailaddr DATA step:
address='e-mailAddress'
specifies the e-mail address of the user to receive job, alert, and warning notices. Enclose the value of address in quotation marks.
sendAlertWarning='Y | N'
specifies whether alert warning notifications are sent to the e-mail address specified in address. Valid values are Y (yes) and N (no). Enclose the value of sendAlertWarning in quotation marks.
sendJobStatus='Y | N'
specifies whether the job status report is sent to the e-mail address specified in address. Valid values are Y (yes) and N (no). Enclose Y or N in quotation marks.

Report Specifications

DATA Step mm_jobs.reportdef

This DATA step defines the type of reports to create, provides the macro syntax for the report type, and defines alert and warning specifications. You can specify one, two, or three report types in the DATA step. The %MM_RunReports() macro runs the reports that are defined in the mm_jobs.reportdef data set. For each type of report, assign the reportName, the macro, and alert and warning conditions.
/***************************************************/
/* DATA set mm_jobs.reportdef                      */
/*                                                 */
/* Create a data set that defines the report       */
/* metadata and alarm thresholds for the           */
/* Characteristic, Stability, and Model Assessment */
/* reporting jobs.                                 */
/***************************************************/

DATA mm_jobs.reportdef;
   length reportName $20 
   macro $1000
   alertCondition $200 
   warningCondition $200 
   isActive $1 
   notes $500;

   isActive='Y';

  /*************************/
  /* Characteristic Report */
  /*************************/
  
   reportName='Characteristic';
   macro='
     %MM_UpdateCharacteristicTable(
        datasrc=&_MM_ReportDatasrc,
        dropVars=&_MM_DropVars;';

   alertCondition='alertConditions';
   warningCondition='warningConditions';
   output;

  /********************/
  /* Stability Report */
  /********************/
  
   reportName='Stability';
   macro='
     %MM_UpdateStabilityTable(
        datasrc=&_MM_ReportDatasrc,
        keepVars=&_MM_KeepVars;';
  
   alertCondition='alertConditions';
   warningCondition='warningConditions';
   output;
  
  /***************************/
  /* Model Assessment Report */
  /***************************/
  
   reportName='Model Assessment';
   macro='
     %MM_UpdateAssessmentTable(
        datasrc=&_MM_ReportDatasrc);';
  
   alertCondition='alertConditions';
   warningCondition='warningConditions';
   output;
run;

Variable Descriptions for mm_job.reportdef

The following variable definitions are used in the mm_jobs.reportdef DATA step:
isActive
specifies whether to enable the report definitions. Valid values are Y (yes) and N (no). Specifying N means that a report definition file does not need to be removed from the local computer to deactivate a report definition entry.
Interaction: Always set isActive='Y' when the data set mm_jobs.project has only one observation.
reportName='reportName'
specifies the name of the report. The following are valid report types:
  • Characteristic
  • Stability
  • Model Assessment
Enclose reportName in quotation marks. This argument is required.
macro='macroDefinition';
specifies the report macro that is executed when the %MM_RunReports() macro is executed. This argument is required.
alertConditions='alertConditions';
specifies an alert condition for the type of report. Enclose alertConditions in quotation marks. Here are example alert conditions for each type of report:
Report Type
Example Alert Condition
Characteristic
alertCondition='p1>5 or p25>0';
Stability
alertCondition='outputDeviation > 0.03';
Model Assessment
alertCondition='lift5Decay>0.15 and lift10Decay>0.12) or giniDecay>0.1 or ksDecay>0.1';
alertConditiion=’msedecay > 20’;
warningConditions='warningConditions';
specifies a warning condition for the type of report. Enclose warningConditions in quotation marks.
Report Type
Example Warning Condition
Characteristic
warningCondition='p1>2';
Stability
alertCondition='outputDeviation > 0.01';
Model Assessment
warningCondition='lift5Decay>0.05';
warningCondition==’msedecay >10’;
notes='userNotes';
specifies a note to add to the report definition data set. Enclose userNotes in quotation marks.

%MM_UpdateCharacteristicTable() Macro

Here is the syntax for the %MM_UpdateCharacteristicTable() macro:
%MM_UpdateCharacteristicTable(datasrc=&_MM_ReportDatasrc,
<dropvars=&_MM_DropVars>);
datasrc=&_MM_ReportDatasrc
specifies the macro variable that defines the performance data set that is used to create the Characteristic report.
dropvars=&_MM_DropVars
specifies the macro variable that defines the input variables to drop from the performance data set. Consider dropping variables from the performance data set whose values do not need to be monitored.

%MM_UpdateStabilityTable() Macro

Here is the syntax for the %MM_UpdateStabilityTable() macro:
%MM_UpdateStabilityTable(datasrc=&_MM_ReportDatasrc,
<keepvars=&_MM_KeepVars>);
datasrc=&_MM_ReportDatasrc
specifies the macro variable that defines the performance data set that is used to create the Stability report.
keepvars=&_MM_KeepVars
specifies the macro variable that defines the output variables to keep in the performance data set. Consider keeping only the variables in the performance data set whose values are to be monitored.

%MM_UpdateAssessmentTable() Macro

Here is the syntax for the %MM_UpdateAssessmentTable() macro:
%MM_UpdateAssessmentTable(datasrc=&_MM_ReportDatasrc);
datasrc=&_MM_ReportDatasrc
specifies the macro variable that defines the performance data set that is used to create the Model Assessment reports.

Job Scheduling Specifications

DATA Step mm_jobs.jobtime

This DATA step defines the dates and times that the data sets that underlie the performance monitoring reports are to be created or updated.
/**************************************************/
/* DATA step mm_jobs.jobtime                      */
/*                                                */
/* Define the report schedule by specifying the   */
/* dates and times for each incremental reporting */
/* interval. You can schedule as many jobs as you */
/* would like. The following jobs are scheduled to*/
/* run one second before midnight on the dates    */
/* listed below.                                  */
/**************************************************/
  
DATA mm_jobs.jobtime;
  length scheduledTime $18 time $10;
  scheduledTime='dateTime';time='timePeriodLabel';output;
run;

Variable Descriptions for mm_jobs.jobtime

Here are the variables that are used in the DATA step mm_jobs.jobtime:
scheduledTime='dateTime'
specifies the date and time to run the report. The value of scheduledTime must be in the form ddmmmyyyy:hh:mm:ss where dd is a two-digit year, mmm is the first three letters of the month, yyyy is a four-digit year, hh is a two-digit hour, mm is a two-digit minute, and ss is a two-digit second. Enclose dateTime in quotation marks.
The values of scheduledTime are used by the %MM_RunReports() macro, rather than by your job scheduler. Each time that the %MM_RunReports() macro runs, it checks the values of the scheduleTime variable. If the scheduled time has passed, the report runs. If it has not passed, the performance data sets are not created.
Example: scheduledTime='03Jun2012:23:59:00';
time='ttimePeriodLabel'
specifies a label that represents the time period for which the performance data was collected. Enclose timePerodLabel in quotation marks. Use short and clear labels to create charts that can be easily read.
Example: time='2012Q4';

Example Code to Create the Report Specifications

This example creates a single SAS program to create the report specification data sets. After you copy the example code from the sashelp.modelmgr library, you providing values for the required variables and macros. The variable and macro names are highlighted in the example code to identify the values that you would modify to create the report specifications.
/* Source file name: sashelp.modelmgr.reportExample3.source */

LIBNAME mm_jobs 'c:\mm.test\report.auto';

/*********************************************/
/* DATA step mm_jobs.project                 */
/*                                           */
/* Create a data set to initialize the       */
/* performance monitoring report batch       */
/* job project specification metadata and    */
/* report precode metadata.                  */
/*********************************************/
  
DATA mm_jobs.project;
  length testDestination $50 
    projectuuid $36 
    projectpath $200 
    projectAlias $50
    precode $32000 
    isActive $1 
    notes $500;
  
  isActive='Y';
  
/***********************************************/
/* Specify the destination path for the report */
/* and the universal unique ID for the project */
/***********************************************/
  
testDestination=
    'c:\mm.test\report.test.output\project_123';
  projectuuid=
    '8817ea06-0a28-0c10-0034-68f4ba396538';
  
/************************************************/
/* The precode section uses macro variables to  */
/* map individual model metadata components     */
/* to their respective variables, target event  */
/* values, and data used to create the report.  */
/************************************************/
  
precode='
    %let _MM_EventProbVar=p_bad1;
    %let _MM_TargetVar=bad;
    %let _MM_TargetEvent=1;
    %let _MM_ReportDatasrc=scoreIn.hmeq0;
    %let _MM_KeepVars=p_bad1; 
    %let _MM_DropVars=bad job;
  ';
   
/************************************************/
/* Specify the path to the project and provide  */
/* an Alias name for the project reports.       */
/************************************************/
   
  projectPath=
    'http://myserver:8080/ModelManager/MMRoot/demo/Creditcardpromotion';
  projectAlias=
    'credit risk for younger customers';
run;

/*************************************************/
/* DATA set mm_jobs.emailaddr                    */
/*                                               */
/* Create a data set that specifies the e-mail   */
/* recipient notification list, and whether to   */
/* send the alert, warning, and job status       */
/* notifications.                                */
/*************************************************/
   
DATA mm_jobs.emailaddr;
  length address $50 sendAlertWarning sendJobStatus $1;
  address='recipient1@mail.com'; 
    sendAlertWarning='Y'; 
    sendJobStatus='N'; 
    output;
  address='recipient2@mail.com'; 
    sendAlertWarning='Y'; 
    sendJobStatus='Y'; 
    output;
run;
  
/***************************************************/
/* DATA set mm_jobs.reportdef                      */
/*                                                 */
/* Create a data set that defines the report       */
/* metadata and alarm thresholds for the           */
/* Characteristic, Stability, and Model Assessment */
/* reporting jobs.                                 */
/***************************************************/

DATA mm_jobs.reportdef;
  length reportName $20 
    macro $1000
    alertCondition $200 
    warningCondition $200 
    isActive $1 
    notes $500;

  isActive='Y';

  /*************************/
  /* Characteristic Report */
  /*************************/
  
  reportName='Characteristic';
  macro='
    %MM_UpdateCharacteristicTable(
      datasrc=&_MM_ReportDatasrc,
      dropVars=&_MM_DropVars;';

  alertCondition='p1>5 or p25>0';
  warningCondition='p1>2';
  output;

  /********************/
  /* Stability Report */
  /********************/
  
  reportName='Stability';
  macro='
    %MM_UpdateStabilityTable(
      datasrc=&_MM_ReportDatasrc,
      keepVars=&_MM_KeepVars;';
  
  alertCondition='outputDeviation > 0.03';
  warningCondition='outputDeviation > 0.01';
  output;
  
  /***************************/
  /* Model Assessment Report */
  /***************************/
  
  reportName='Model Assessment';
  macro='
    %MM_UpdateAssessmentTable(
    datasrc=&_MM_ReportDatasrc);';
  
  alertCondition='
    (lift5Decay>0.15 and lift10Decay>0.12) 
     or giniDecay>0.1 
     or ksDecay>0.1';
  warningCondition='lift5Decay>0.05';
  output;
run;

/**************************************************/
/* DATA step mm_jobs.jobtime                      */
/*                                                */
/* Define the report schedule by specifying the   */
/* dates and times for each incremental reporting */
/* interval. The jobs below are scheduled to run  */
/* one second before midnight on the dates listed */
/* below.                                         */
/*                                                */
/* For each scheduledTime variable you need a     */
/* separate DATA step to execute whose SET        */
/* statement names the appropriate performance    */
/* data source.                                   */
/**************************************************/
  
DATA mm_jobs.jobtime;
  length scheduledTime $18 Time $10;
  scheduledTime='01OCT2012:23:59:59';time='2012Q3';output;
  scheduledTime='01JAN2013:23:59:59';time='2012Q4';output;
  scheduledTime='01APR2013:23:59:59';time='2013Q1';output;
  scheduledTime='01JUL2013:23:59:59';time='2013Q2';output;
  scheduledTime='01OCT2013:23:59:59';time='2013Q3';output;
  
run;