Many organizations have some combination of development, test, and production environments. Below is a best practice to help guide you. Two different configurations are discussed. These configurations are valid for SAS Model Manager 2.2, 2.3, 3.1, and 3.1 M1. The first proposal is for SAS Model Manager deployed only in a development environment. The second proposal shows SAS Model Manager in both development and production environments.

 

SAS Model Manager Installed only in a development environment:

 

Development environment:

   Development Cycle

• Start with the training data and develop your models via SAS Enterprise Miner.
• Register your candidate SAS Enterprise Miner model packages into SAS^® Metadata Server.
• Use SAS Model Manager to import the candidate SAS Enterprise Miner models from SAS Metadata Server.
• View model properties or create reports to determine which model you want to promote as champion.
• Export the champion model to SAS^® Metadata Repository.
• (Optional) Use SAS^® Data Integration Studio or SAS^® Enterprise Guide^® to create a job that is based on the MiningResult object.
• Use SAS Management Console to export the MiningResult object and corresponding SAS Data Integration Studio or SAS Enterprise Guide jobs from SAS Metadata Server. Create a SAS Package file on the file system so that the SAS Package file is accessible to the test system.
• Use the training data set to run the performance task once to establish the baseline for model monitoring.

 

    Monitoring Cycle

• Import the scored data set from the production environment into the development environment. The data set can be one of the following formats:
  • A data set that contains only the model’s predicted values and the actual outcome. Use this data set if you only want SAS Model Manager to monitor the model’s performance (Lift, ROC, Gini, and so on) and stability report.
  • A data set that contains the actual outcome (or target) and the model’s input variables. Use this data set if you want SAS Model Manager to monitor characteristic, model performance, and stability. No customer ID is needed.

Alternatively, if you can define a library that points to the production environment, the above step is not necessary. 

• Define a metadata library in SAS Metadata Server in the development environment to contain the performance data set imported from the production environment.
• Create a performance report task in SAS Model Manager to generate the model monitoring results based on the data that is imported from the production environment.
• Repeat the production scoring process and development monitoring cycle until a new model is needed.

    Test Environment:

• Import the SAS Package file exported from the Development environment into SAS Metadata Server.
• Use SAS Data Integration Studio or SAS Enterprise Guide to create a custom scoring flow.
• Test your scoring job.
• Use SAS Management Console to export the MiningResult object and corresponding SAS Data Integration Studio or SAS Enterprise Guide jobs from SAS Metadata Server. Create a SAS Package file on the file system so that the SAS Package file is accessible to the test system.

    Production Environment:

• Import the SAS Package file or SAS Data Integration Studio or SAS Enterprise Guide job from the test environment into the SAS Metadata Server.
• Score the data.
• Take either the entire data or a sample of the data to be exported back to the development environment for monitoring. This data does not need to contain any customer information. It is similar to the training data in that it needs to have the model inputs, the model predicted results, and the actual customer response. Move this SAS data set to the development environment for monitoring, or just make the data available to the test system. For more information, look under the Monitoring Cycle bullet under the Development Environment section.

SAS Model Manager  Installed in the Development and Production Environments

 

There are some scenarios in which you want to have SAS Model Manager in both Development and Production environments. For example, you might not want to move sensitive or large amounts of data across environments. That scenario is currently under testing, so this note might be updated based on the testing results.

 

    Development environment:

 

Development Cycle

• Start with the training data and develop your models via SAS Enterprise Miner.
• Register your candidate SAS Enterprise Miner model packages into SAS Metadata Server.
• Use SAS Model Manager to import the candidate SAS Enterprise Miner models from SAS Metadata Server.
• View model properties or create reports to determine which model you want to promote as champion.
• Export the champion model to the SAS Metadata Repository.
• Use the WebDAVDump utility to dump the current SAS Model Manager project into a zip file. This zip file is used to import the project into the Production environment.
• (Optional) Use SAS Data Integration Studio or SAS Enterprise Guide to create a job that is based on the MiningResult object.
• Use SAS Management Console to export the MiningResult object and corresponding SAS Data Integration Studio or SAS Enterprise Guide jobs from SAS Metadata Server. Create a SAS Package file on the file system so that the SAS Package file is accessible to the test system.

     Test Environment:

• Import the SAS Package file exported from the Development environment into SAS Metadata Server.
• Use SAS Data Integration Studio or SAS Enterprise Guide to create a custom scoring flow.
• Test the scoring job.
• Use SAS Management Console to export the MiningResult object and corresponding SAS Data Integration Studio or SAS Enterprise Guide jobs from SAS Metadata Server. Create a SAS Package file on the file system so that the SAS Package file is accessible to the test system.

 

Production Environment:

• Import the SAS Package file, SAS Data Integration Studio job, or SAS Enterprise Guide job from the test environment into SAS Metadata Server.
• Use the WebDAVRestore utility to import the zip file from the Development environment that contains the current Project.
• Score the data.

     Monitoring Cycle

• Create a metadata library to the scored data set in metadata.
• Create a performance report task in SAS Model Manager to generate the model monitoring reports on the scored production data.
• Repeat the production scoring process and SAS Model Manager monitoring cycle until a new model is needed.

 

Here are a few important points to note with these two SAS Model Manager scenarios:

1. When you import the dump file on the production environment, you need to import the project only one time. The next time you run through the model cycle, you need to only import the new version that contains the new champion model. The assumption is that once a development environment signals that a model is ready to be deployed, everything in that version does not change. All the information about that model is promoted to the production environment for model monitoring, and you should dump the containing version from that point forward.
2. When the data is imported via the WebDAVRestore utility, the creation and modification dates are not preserved. All of the objects that are imported have a date corresponding to the time when Restore was performed.
3. The above steps assume that users in the development and production environments work sequentially.
4. For more information about the WebDAVDump and WebDAVRestore utilities, refer to this SAS Note: 38667 "Using the WebDAVDump and WebDAVRestore utilities".

Operating System and Release Information

Product Family	Product	System	Product Release		SAS Release
			Reported	Fixed*	Reported	Fixed*
SAS System	SAS Model Manager	Microsoft® Windows® for x64	2.3		9.2 TS2M3
		Microsoft Windows Server 2003 Datacenter Edition	2.3		9.2 TS2M3
		Microsoft Windows Server 2003 Enterprise Edition	2.3		9.2 TS2M3
		Microsoft Windows Server 2003 Standard Edition	2.3		9.2 TS2M3
		Microsoft Windows Server 2003 for x64	2.3		9.2 TS2M3
		Microsoft Windows Server 2008	2.3		9.2 TS2M3
		Microsoft Windows Server 2008 for x64	2.3		9.2 TS2M3
		Microsoft Windows XP Professional	2.3		9.2 TS2M3
		Windows 7 Enterprise 32 bit	2.3		9.2 TS2M3
		Windows 7 Enterprise x64	2.3		9.2 TS2M3
		Windows 7 Home Premium 32 bit	2.3		9.2 TS2M3
		Windows 7 Home Premium x64	2.3		9.2 TS2M3
		Windows 7 Professional 32 bit	2.3		9.2 TS2M3
		Windows 7 Professional x64	2.3		9.2 TS2M3
		Windows 7 Ultimate 32 bit	2.3		9.2 TS2M3
		Windows 7 Ultimate x64	2.3		9.2 TS2M3
		Windows Vista	2.3		9.2 TS2M3
		Windows Vista for x64	2.3		9.2 TS2M3
		64-bit Enabled AIX	2.3		9.2 TS2M3
		64-bit Enabled Solaris	2.3		9.2 TS2M3
		HP-UX IPF	2.3		9.2 TS2M3
		Linux for x64	2.3		9.2 TS2M3
		Solaris for x64	2.3		9.2 TS2M3

* For software releases that are not yet generally available, the Fixed Release is the software release in which the problem is planned to be fixed.

---

Type:	Usage Note
Priority:
Topic:	Analytics ==> Model Management

Date Modified:	2012-05-09 17:30:32
Date Created:	2012-03-14 14:10:54