Working with the APM ETL :: SAS(R) Environment Manager 2.5: User's Guide, Second Edition

Overview

The APM ETL process extracts performance metric information from various SAS server logs, HTTP access logs, SAS job logs, and SAS metadata audit data and loads that information into the data mart. Custom parsers can be added to enable the ETL to process information specific to a SAS solution or application. As with other information in the data mart, you can use this information to produce predefined reports or to conduct custom analysis and reporting.

Note: Enabling the APM ETL process causes a separate log to be created for each spawned SAS Workspace Server. You must plan for the large number of log files that this process could create. A best practice is to create a daily archive file of the day’s log files, and then to copy the file to archive storage.

Log Collection Process

APM data is processed and loaded into the data mart in these steps:

The logs in your SAS deployment are discovered. The log discovery process runs for the first time when the APM ETL is initialized. This process scans the components in your SAS system for log files and includes the logs that it finds that are supported by the APM ETL. By default, the log discovery process runs every 15 minutes throughout each day, so any new logs created by new components in your SAS environment are discovered and included in the log collection process. You can also choose to run the log discovery process manually though a control action, which enables you to start collecting log data sooner than if you waited for the scheduled process. This discovery process improves the accuracy of the collected log data, because it reflects the SAS applications that are actually installed. The discovery process runs when the APM ETL is initialized, and runs at regular intervals after initialization is complete. You can also run the log discovery process manually at any time.

See Manually Discovering and Collecting Logs for information about manual processes.
The discovered logs are collected locally on the machine where they are created. The logs are stored in the landing zone directory, which is [LevelRoot]/Web/SASEnvironmentManager/emi-client/LandingZone. By default, the logs are collected nightly, but they can be collected manually as often as every 30 minutes in order to obtain an update view of the log information.
The locally collected logs are collected from the local landing zone directories to a central landing zone directory, which is located on the SAS Environment Manager Enablement Kit Server. This machine is the machine containing the alphabetically first SAS Application Server context that contains a SAS Workspace Server. Beginning with the third maintenance release after SAS 9.4, you can use SAS Deployment Agent to automatically copy the log files from the local landing zone directories to the central landing zone directory. You can also set up file mounts or NFS shares to the local landing zone directories so that the central landing zone directory has access to the log files whenever they are saved to the local landing zone directories. After the logs are collected in the central landing zone directory, they are deleted from the local landing zone directories.
The ETL process parses the logs in the central landing zone directory, puts the information into a standard format, and archives the original log files. The data is then put into the appropriate tables in the data mart.
APM data in the data mart is available for analysis and reporting.

The following figure illustrates the log collection process.

Manually Discovering and Collecting Logs

Follow these steps to manually run the APM ETL process to discover and collect logs:

Select ResourcesBrowseServices. Select the APM ETL Processing entry in the table.
On the page for the APM ETL Processing service, select Control.
In the Quick Control area, select Run in the Control Action field and then click the Run icon .

You can also start the log collection process from a command line by using the master_apm_etl script. You can automate this process by using a job scheduler to run the script. See Running the APM ETL Processes By Using master_apm_etl for more information and syntax.

Generating Reports

Reports that are produced by APM display the following types of information:

the most heavily used SAS procedures
the top ten users of the SAS Workspace Server
the number of times per day each user ID has accessed the SAS Metadata Server
each instance of an authentication error or of an unauthorized login attempt
the response time and run time for SAS stored processes