Overview of Model Comparison, Validation, and Summary Reports

The SAS Model Manager model comparison, validation, and summary reports are tools that you can use to evaluate and compare the candidate models in a version or across versions to help you select and approve the champion model that moves to production status. The SAS Model Manager model comparison reports are analytical tools that project managers, statisticians, and analysts can use to assess the structure, performance, and resilience of candidate models. The model validation reports use statistical measures to validate the stability, performance, and calibration of Basel II risk models and parameters. The training summary data set report creates frequency and distribution charts that summarize the train table variables.

The reports present information about a number of attributes that can affect model performance. Together, the reports provide qualified information that can serve as the analytical basis for choosing and monitoring a champion model.

Here is a description of the comparison reports:

These are the Basel II model validation reports:

The model validation reports use statistical measures that report on these model validation measures:

This is the train table data set summary report:

You create the model comparison, validation, and summary reports using the New Report window that you start from a version's Reports node:

SAS Model Manager uses a test table as the input table for the Dynamic Lift report and the Interval Target Variable report.

Before you can create a Dynamic Lift report or the Interval Target Variable report, make sure that a test table has been added to the SAS Metadata Repository using SAS Management Console or that a libref has been defined in the Edit Start-up Code window for the SAS library where the test table resides. The test table can be viewed in the Data Sources category view. Then, specify the test table in the project property Default Test Table.

You specify the input table for Basel II validation reports in the New Report window. The input file for the validation reports can contain only input variables or it can contain input and output variables. If the input table contains input and output variables, the report generation does not need to run a scoring task to obtain the output variables.

To create a train table summary report, use a train table to create training summary data sets. The training summary data sets are used to create the train table summary report. Either the train table must be added to the SAS Metadata Repository or a libref must be defined in the Edit Start-up Code window for the train table library. The train table must then be specified in the project or version property for Default Train Table. You create the training summary data sets by using the Generate Training Summary Data Set feature of SAS Model Manager from the version node in the Project Tree.

The New Reports window stores the model comparison, validation, and summary report output files in a report node under the Reports node. The name of the report node is the value of the Name box that you specified in the New Report window Report Properties table.

Each time you create a report, these files are generated:

Here is a description of the model comparison output files:

After you create a report, you can view the report from the Reports node: