Amazon CloudWatch — Accessing Raw Data :: SAS(R) IT Resource Management 3.7: Administrator's Guide

Preparing Amazon CloudWatch Data for SAS IT Resource Management

Amazon CloudWatch is a monitoring service for Amazon Web Services (AWS) cloud resources and the applications that you run on AWS. It collects and tracks metrics about your resources and applications and sends notifications when alarms are triggered. This information is stored in a repository from which the metrics can be retrieved.

The Amazon CloudWatch adapter that SAS IT Resource Management provides consists of these components:

The AWS batch script fetches performance metrics data from the AWS CloudWatch repository and stores the data in text files. A separate text file is created for each metric.
The Amazon CloudWatch staging transformation code reads data from the text files and writes it to a SAS data set. Supplied templates create the staged tables and variables that are needed by SAS IT Resource Management.

Note: The Adapter Setup wizard can be used to specify the Amazon CloudWatch adapter. The wizard produces a detail aggregation table in addition to the Amazon CloudWatch staging transformation.

Extracting Data from Amazon CloudWatch

Use a Windows batch script to fetch the data from the AWS CloudWatch repository. Here are the prerequisites for using the batch script:

Windows PowerShell must be installed on your machine. (Windows PowerShell is a task automation and configuration management framework from Microsoft. It consists of a command-line shell and associated scripting language built on the .NET Framework.)
AWS CLI software must be installed on a Windows machine (for example, the SAS IT Resource Management client) where the script runs. This installation requirement is necessary because the batch script uses AWS CLI commands to fetch the data from the AWS CloudWatch repository. The latest version of the AWS CLI software can be found here: http://aws.amazon.com/cli/. (AWS Command Line Interface (CLI) is a unified tool for managing your AWS services.)

Note: The machine that hosts the batch script must be available when the batch script is scheduled to run.
The Windows environment variable, AWS_CLI_HOME, must be configured so that it points to the AWS CLI installation directory.
The IAM user must be specified. This user fetches the data from the AWS CloudWatch repository. Therefore, it should have the access and permissions necessary to use the AWS services and to fetch Amazon CloudWatch metrics.
The AWS Access Key ID and AWS Secret Access Key must be configured as the default profile. These fields are credentials for the account of the IAM user. They should be saved at this location : %USERPROFILE%/.aws/credentials.

Credentials for IAM User
The raw data files are generated at this location: %USERPROFILE%/ raw_data_files.
The batch script can be found at this location: <SAS_INSTALL_DIR>/SASFoundation/9.4/itmsmvadata/sasmisc/AWSDATA.bat.
The batch script, AWSDATA.bat, can be run as follows:
- For day-to-day operations, without parameters, to fetch data from yesterday’s data, that is (today -1).
- Interactively, with one parameter (–h) to fetch data from the prior hour.
- Interactively, with start_date and end_date parameters to fetch data from the start date through the end date. (The format for these dates is yyyy-mm-dd.)
  
  Tip
  AWS CloudWatch stores data for only the past 14 days. Therefore, the start date cannot be older than 14 days from the current date.
By default, AWS CloudWatch data is available in 5-minute periods. In the following example, AWS data from October 15, 2015 through October 19, 2015 is collected:
```
C:\users\AWSDATA.bat 2015-10-15 2015-10-19
```
The maximum number of data points (observations) that can be returned from a single AWS CloudWatch GetMetricStatistics request is 1,440. Therefore, using the default settings of 5-minute intervals, 288 observations can be returned from a request that is run once a day.

Note: If you choose 1–minute intervals for your data, more than 1,440 observations could be returned. In this case, AWS CloudWatch posts an error message. To resolve this problem, schedule your batch script to run more frequently during the day. Make sure that the output files from these runs are specified so that they do not overwrite one other.

How to Run the Batch Script Interactively

To support backloading data or testing, you might want to run the batch script interactively.

Note: The batch script can run only on a Windows machine. Therefore, if the SAS IT Resource Management server is on Windows, the script can run on the server itself. However, if the SAS IT Resource Management server is on a different platform, copy the batch script to a Windows machine (for example, the SAS IT Resource Management client machine).

Install the AWS CLI on that machine and set the credential file with the AWS access parameters for the AWS account. You can make the following changes to the batch script:

Gather Data for a Specific Period of Time

Specify one date to fetch data for that specified day.
Specify two dates to fetch data for the specified start date through the specified end date.

Note: Use this format for dates: yyyy-mm-dd.

Change the Location Where the Raw Data Files Are Generated

By default, raw data files are generated at this location:

%USERPROFILE%/
                           raw_data_files

on the Windows machine from which the batch script is run. To change the location, perform the following steps:

Locate this code in the batch script:
```
set datapath=%USERPROFILE%
```
Change the value of the datapath field to the desired location (for example, set datapath=\\networkpath\shared_location).

The raw data files are generated at this location: \\ networkpath\shared_location\raw_data_files.

Change the Value of the -h Parameter

By default, the –h parameter fetches data for last 3,600 seconds, the prior hour.

Locate this code in the batch script:

powershell get-date((get-date^).ToUniversalTime(^).addSeconds(-3600^)^)

Change the value -3600 to the new value. This value is the number of seconds between the current datetime and the required datetime. The negative sign means that the system needs the previous datetime. For example, -18000 fetches data for the previous five hours; -86400 fetches data for the previous day.

Notes about Amazon CloudWatch

The AWS Management Pack enables Microsoft System Center Operations Manager (MS SCOM) to access your AWS resources (such as instances and volumes), so that it can collect performance data and monitor your AWS resources. The AWS Management Pack is an extension to SCOM.

You can download and deploy AWS Management Pack for Microsoft System Center from http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/AWSManagementPack.html for both 2012 and 2007. That site also provides documentation about the AWS Management Pack. To access the AWS CloudWatch data from the Microsoft System Center Operations Manager System, use the SAS IT Resource Management %RMMSSCOM macro. For information about the %RMMSSCOM macro, see %RMMSSCOM.