The SAS audit tables in the SharedServices database increase in size over time and can negatively affect the performance and availability of the SAS^® Web Application Server, SASServer1_1.

Symptoms

When the size of the SAS audit tables affects your environment negatively, the problem can manifest in a variety of ways, as shown in these examples:

• SASServer1_1 might stop responding periodically or it might fail periodically.
• SASServer1_1 does not start.
• SASServer1_1 takes increasingly longer amounts of time to start.
• In Windows operating environments, SASServer1_1 spontaneously restarts.
• SASServer1_1 fails to start, and the following out-of-memory exception is generated in the server.log file that resides in the SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/logs/ directory.
  java.lang.OutOfMemoryError: Java heap space
• SASServer1_1 uses unusually high amounts of CPU on your system.

In addition, any of the symptoms listed above might present as an outage in your environment in which you cannot access SAS^® Visual Analytics or other SAS web applications.

To confirm that the symptoms are caused by the SAS audit-table issue:

• Obtain the size of the SAS audit tables. You can use this script to accomplish this task. The script displays the number of records in the SAS audit tables, which you can then send to SAS Technical Support for further analysis.
• Monitor the heap growth of the SASServer1_1 JVM during start-up. When the audit-table issue affects your system, you see unusually large heap growth during the start-up period for SASServer1_1. That growth normalizes after the start-up process is complete. If your JVM cannot survive this heap growth, it will fail during the start-up process. To monitor heap growth, you need to enable garbage-collection logging for your SASServer1_1. If you do not already have garbage-collection logging enabled, see the instructions in SAS Note 50305: "Enabling verbose garbage collection (verboseGC) in your web application server (JBoss, Oracle WebLogic, IBM WebSphere, and Apache Tomcat)."

Problem Source

Two of the tables that are related to the auditing feature (SAS_AUDIT and SAS_AUDIT_ENTRY) are read into memory upon initialization of the SASServer1_1. Therefore, when the records for these tables increase, they might reach an unmanageable size that your SASServer1_1 JVM cannot handle. When the size of these tables reaches that unmanageable point, the SASServer1_1 JVM might fail when you attempt to read in these tables. Determining when the size of these tables reaches a problem point is very much site specific. It depends on the specifications of your machine and of your JVM. Sometimes, this problem occurs seemingly randomly and affects a system that was otherwise working without issues. This situation occurs because this problem affects only the SASServer1_1 JVM at start-up. For example, if your system runs for a period of two weeks without a restart, your audit tables might reach an unmanageable size during that two-week period. However, the size issue does not affect your system until the next time you attempt to restart SASServer1_1. This restart attempt will be the first time you notice any symptoms (for example, SASServer1_1 fails or it does not start). Under Windows, the problem might present with SASServer1_1 restarting automatically.

Problem Resolution

To resolve this problem, use either of the following methods:

• Delete old audit records from the database using this script.
• Increase the maximum heap size for the SASServer1_1 JVM. To do this, you need to increase the -Xmx value for the SASServer1_1 JVM. That value is located in the following files:
  • Windows: SAS-configuration-directory\Lev1\Web\WebAppServer\SASServer1_1\conf\wrapper.conf
  • Linux: SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/bin/setenv.sh

  This change takes effect when you restart SASServer1_1.

SAS Technical Support recommends that you purge records in the SAS audit tables periodically to keep those tables at a manageable size. However, if you have a business need to maintain a certain amount of data in those tables, you can increase that maximum heap value accordingly to avoid any problems.

Hot Fixes

The Hot Fix tab of this SAS Note contains fixes for the following releases:

• SAS^® 9.4M3 (TS1M3), SAS^® 9.4M4 (TS1M4), and SAS^® 9.4M5 (TS1M5): These hot fixes improve the way that the SAS_AUDIT and SAS_AUDIT_ENTRY tables are read into memory and reduce the likelihood that this problem will occur. Note that even if you install these hot fixes, you might still encounter this problem and need to follow the guidance in this SAS Note if you have a very high volume of records in your SAS audit tables. It is recommended to upgrade to SAS^® ​​​​​​9.4M6 (TS1M6) or SAS^® 9.4M7 (TS1M7) or later if this behavior continues to impact your environment after you apply the hot fix.
• SAS 9.4M6: This fix further limits the number of records that are archived per transaction and should mitigate the risk of this problem occurring, even in cases where you have a high volume of records in your SAS audit tables. This fix was included in SAS 9.4M7, so you do not have to install a hot fix if you use SAS 9.4M7 or later. 

See Also

For more information about the SAS Audit tables, please see the following documentation:

• "Configuring Auditing for SAS Web Applications" ("Chapter 7: Administering the SAS Web Infrastructure Platform") in SAS® 9.4 Intelligence Platform: Middle-Tier Administration Guide - Configuring Auditing for SAS Web Applications
• "Key Actions Auditing" ("Chapter 3: Security") in the SAS Visual Analytics Administration Guide
• SAS Note 58599: "Managing SAS^® audit tables"

Operating System and Release Information

A fix for this issue for SAS Middle Tier 9.4_M6 is available at:

A fix for this issue for SAS Middle Tier 9.4_M5 is available at:

A fix for this issue for SAS Middle Tier 9.4_M4 is available at:

A fix for this issue for SAS Middle Tier 9.4_M3 is available at: