Aggregation Transformation - Background

Note: SAS^® IT Resource Management sites that have hot fix 311IS04 applied should follow the steps described in SAS Note 34284. We recommend that you apply hotfix 311IS04 due to a simpler backup process, improved performance, and a smaller footprint for data marts in traditional data sets.

In SAS IT Resource Management, an aggregation transformation summarizes data and prepares it for reporting. In this process, the user can request multiple output tables from any given input table. Each of these output tables summarizes and presents data from the input table in a distinct manner.

The Adapter Setup Wizard creates separate component jobs for each data transformation. When you run these jobs, they execute transformations that stage data, create information maps, and prepare raw data for IT performance data reporting. This wizard also creates an overall job that contains all of the component jobs.

Note: It is recommended that you NOT submit the overall job. Instead, you should use your local job scheduling facilities to schedule the staging job first, and then schedule the various aggregation jobs to run in parallel.

However, regardless of whether you execute aggregation jobs through SAS Data Integration Studio or by deploying the jobs on the server, there may be circumstances (such as running out of disk space) that might cause an aggregation job to fail and not complete successfully. When an aggregation job fails, you should be aware that it is possible that some of the output tables for the aggregation transformation might not be updated.

Aggregation – Use of Libraries, Tables, and Table Data

Aggregation seldom uses the SAS WORK library, instead all intermediate data is written to the same library as the aggregation output table. Nomenclature of these intermediate output tables follows the file type and metadata identifier of the output table. Hence, in the SAS Log, the output tables are assigned similar names, but with suffixes such as _STAT, _FLT, _PRANK, _DRANK, _CLASSIDVARS, _INDEXES, etc.

At the successful completion of each aggregation in the transformation, the system deletes the intermediate output tables. Thus, it is critical to remember that if there are any intermediate tables that remain after completion of an aggregation job, their presence indicates the failure of one or more aggregations.

Aggregation jobs access information from the metadata server to control the aging of the aggregation table data and duration for which it is retained. Such information is stored in the metadata server in the form of extended attributes of the output aggregation tables.

Backup and Recovery

Aggregation jobs are very dependent on the data repositories and tables that are on the metadata server. Therefore, it is essential that you back up the ITMS metadata repository and the aggregation table data before you run the aggregation job.

Backup: Limited Versus Full

While it is possible to back up the metadata repository and the data libraries for a specific aggregation job, the amount of work that is required to perform such a limited backup usually outweighs the benefits. Therefore, in most circumstances it is advisable to back up and restore the entire IT Data Mart.

Note that the techniques explained in this note assume that ANY failure in processing of the aggregation job makes it necessary for you to restore the entire IT Data Mart.

Backup: Metadata Repository

See SAS Note 14450 for instructions and information on making a backup of a metadata repository.

Backup: SAS Libraries and Aggregation Tables

Use the following methods to back up the SAS libraries that contain your aggregation tables:

• COPY Procedure
• external backup systems
• operating system commands at your site
  
  Note: It is critical to restore these SAS libraries to their original library structure, in accordance with their Data Control Block (DCB) attributes. It is also critical to back up the ADMIN library of the IT Data Mart to which the aggregation tables belong.

It is a tedious task to determine the associations and relationships between data libraries and aggregation jobs and tables. Therefore, it is very highly recommended that you back up the entire IT Data Mart and at the same time that you back up the metadata repository.

By examining the header block comments of the aggregation job and the libname statements in the job, you can determine which libraries are used by a given aggregation job.

Your Course of Action After the Failure of Aggregation Jobs

If an aggregation job fails, you should perform the following steps:

1. Resolve the root cause of the failure (such as insufficient disk space).
2. Restore the ITMS metadata repository and the SAS IT Resource Data Mart.
3. Restart the aggregation transformation process, starting with the staging jobs.

Operating System and Release Information

A fix for this issue for SAS IT Resource Management 3.1.1 is available at: