The SAS Environment Manager Service Architecture is an optional tool that extends the application’s auditing and user monitoring capabilities. To learn more about the Service Architecture and the nightly processes, review Understanding SAS Environment Manager Service Architecture

You can verify that the ETL processes are initialized and enabled by running the following script on the compute (or grid controller) machine:

If the Service Architecture is enabled, the output from the above script should look similar to this:

Here are the steps in the process in the order in which they automatically run:

1. Log Discovery
2. Log Collect
3. Log Centralize
4. ACM ETL
5. APM ETL

Log Discovery

Location: This script is located on each machine in the deployment:

Linux: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-client/bin/log_discovery.sh

Windows: SAS-configuration-directory\LevN\Web\SASEnvironmentManager\emi-client\bin\log_discovery.bat

Description: This script scans the paths and creates/updates the log_locations.json file with the locations of the discovered log directories.

Process: This script automatically runs every four hours (invoked by agent code that initiates metric collection).

Log Collect

Location: This script is located in the emi-client on each machine in the deployment:

Linux: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-client/bin/log_collect.sh

Windows: SAS-configuration-directory\LevN\Web\SASEnvironmentManager\emi-client\bin\log_collect.bat

Description: This script collects the logs that are located in the paths specified in the log_locations.json file and moves those logs to a local landing zone for each machine.

Process: This script runs nightly (by default at 12:00 AM).

1. It moves the logs from their log location to the Local Landing Zone. ​
2. It purges logs from the log location if these three conditions are met:

• The log has been collected (and therefore archived and not lost).
• The log file has not been modified for 30 days (default).
• The log file is not the most recently modified processed file regardless of age.

*Note: Files that do not match the pattern in the logDiscovery.groovy script are not purged.

Log Centralize

Location: This script is located in the emi-framework on the compute (or grid controller) machine:

Linux: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-framework/bin/log_centralize.sh

Windows: SAS-configuration-directory\LevN\Web\SASEnvironmentManager\emi-framework\bin\log_centralize.bat

Description: This script calls on the Deployment Agents to zip and move the logs from each machine's local landing zone (emi-client) to the central landing zone (emi-framework).

Process: This script runs nightly (by default at 1:00 AM).

• If necessary, log centralize attempts to start the remote Deployment Agent for each host.

ACM ETL

Location: This script is located in the emi-framework on the compute (or grid controller) machine:

Linux: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-framework/bin/master_acm_etl.sh

Windows: SAS-configuration-directory\LevN\Web\SASEnvironmentManager\emi-framework\bin\master_acm_etl.bat

Description: This script extracts and stores metric data that is collected by SAS Environment Manager agents and standardized into data sets.

Process: This script runs nightly (by default at 2:00 AM).

• ACM ETL gets metric data from the SAS Environment Manager database (not logs); therefore it does not depend on the log_collect.sh and log_centralize.sh files to retrieve updated information.

APM ETL

Location: This script is located in the emi-framework on the compute (or grid controller) machine:

Linux: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-framework/bin/master_apm_etl.sh

Windows: SAS-configuration-directory\LevN\Web\SASEnvironmentManager\emi-frameworkbin\master_apm_etl.bat

Description: This script moves the logs from the central landing zone to a temporary staging area and then parses and analyzes the logs, puts the information into a standard format (data sets), and archives the original log files.

Process: This script runs nightly (by default at 3:00 AM).

1. The APM script moves the zipped logs from the Central Landing Zone to a temporary .staging directory where the logs are processed. 
2. APM runs a series of Groovy scripts to parse the logs into data sets (located in the Datamart directory).
3. Once the logs are processed and analyzed (the APM completed successfully), they are moved from the temporary area directory to the Archive directory; the .staging directory is automatically removed when this process is complete.

Configure the Nightly Process:

To update the nightly process of the Data Mart Services (i.e., disable automatic execution of the process or change the hour that the process is run):

On the SAS Environment Manager Web Console, select Resources ► Browse ► Services ► select the resource {Log Collection, Log Centralization, APM ETL Processing, ACM ETL Processing} ► expand Tools Menu ► Configure Service.

Example: APM ETL Processing resource configuration:

Important directories:

Log locations: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-client/Conf/log_locations.json

Log Inventory: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-client/Inventory/log_inventory.json

• Used (only) by log_collect to keep track of logs that it has processed so that it does not pull that data multiple times.

• Keeps track of positions within each given log file so that it can handle collecting data from partially-complete logs.

Local Landing Zone:  SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-client/LandingZone

• Each machine in the deployment has a local landing zone where logs are stored by the log collect script.
• The logs are removed from the local landing zone once the log centralize script has completed successfully.

Central Landing Zone: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-framework/LandingZone

• The central landing zone is on only the machine where the emi-framework is configured (typically the compute/grid controller).
• The logs are moved from the central landing zone to .staging when the APM script runs.
• The logsProcessed_<timestamp>.json files do not get removed automatically from this area.

Staging directory: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-framework/LandingZone/.staging

• The .staging directory is a temporary location where the logs are processed and analyzed, and then archived.
• This directory is automatically removed once the APM process has completed successfully.

Datamart: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-framework/Datamart

• ​​​​The SAS Environment Manager data mart is a collection of tables that store monitored data in a data infrastructure that is provided by the SAS Environment Manager Extended Monitoring.
• By default, data is retained in the data mart for 60 days.
• The 60-day retention period for data in the data mart can be changed in the SAS^® Management Console Configuration Manager plug-in:

1. In SAS Management Console, select Application Management ► Configuration Manager.
2. Right-click EnvMgr Enablement Kit 2.x and select Properties.

Archive directory: SAS-configuration-directory/LevN/Web/SASEnvironmentManager/emi-framework/Archive

• This directory contains the archive of all logs gathered during the masterAPMETL process.
• These logs are not automatically removed by any SAS script or process.
• Best practices would suggest the copying of the zipped logs to some form of shared network storage or tape media depending on your IT organization's requirements for protection of access and activity historical records.

Operating System and Release Information