Overview of Model Comparison, Validation, and Summary Reports

The SAS Model Manager model comparison, validation, and summary reports are tools that you can use to evaluate and compare the candidate models in a version or across versions to help you select and approve the champion model that moves to production status. The model comparison reports are analytical tools that project managers, statisticians, and analysts can use to assess the structure, performance, and resilience of candidate models. The model validation reports use statistical measures to validate the stability, performance, and calibration of risk models and parameters. The training summary data set report creates frequency and distribution charts that summarize the train table variables.

The reports present information about a number of attributes that can affect model performance. Together, the reports provide qualified information that can serve as the analytical basis for choosing and monitoring a champion model.

Here is a description of the comparison reports:

The following are the Basel III model validation reports:

The model validation reports use statistical measures that report on these model validation measures:

This is the train table data set summary report:

After you execute a performance definition, you can generate performance monitoring results and compare the champion and challenger models:

You create the reports using the New Report window that you start from a project’s Reports page.

SAS Model Manager uses a test table as the input table for the Dynamic Lift report and the Interval Target Variable report.

Before you can create a Dynamic Lift report or the Interval Target Variable report, make sure that a test table has been added to the SAS Metadata Repository and registered in the Data Tables category or SAS Management Console. The test table can be viewed in the Data Tables category view. Then, specify the test table in the project property Default test table.

You specify the input table for validation reports in the New Report window. The input file for the validation reports can contain only input variables or it can contain input and output variables. If the input table contains input and output variables, the report generation does not need to run a scoring test to obtain the output variables.

When you create a train table summary report, the train table or specified input table is used to create the training summary data sets. The train table must be available in the SAS Metadata Repository. The train table must then be specified in the project property for the Default train table.

The Reports page stores the model comparison, validation, and summary report output files in the Model Evaluation tab. The name of the report is the value of the Name box that you specified in the New Report window.

Each time you create a report, these files are generated:

Here is a description of the model comparison output files:

After you create a report, you can view the report from the Reports page.