Part 1: Restoring Unresponsive SAS^® Stored Process Servers with SAS^® 9.2

SAS Technical Support has received reports of previously working SAS Stored Process Servers becoming unresponsive over time for unknown reasons. Unresponsive means that the SAS Stored Process Servers are running, but no requests from client applications are getting through to the server. These servers might also be referred to as hung or orphaned SAS processes.

You might have encountered this problem in one or more ways. You might be the end user working with one of the SAS^® Business Intelligence client applications, such as SAS^® Enterprise Guide^®, SAS^® Web Report Studio, the SAS^® Add-in for Microsoft Office, the SAS^® Information Delivery Portal, SAS Stored Processes Web application, or another Web-based application. You click on a button expecting a report to be returned, but instead you receive a generic error or Java dump. You might be the systems administrator, who gets a call from the end user and determines that there is no SAS Stored Process Server responding, or has at least narrowed the problem down to involve a SAS server rather than a client.

What do you do?

1. Initially with all customers, troubleshooting starts as a "Put out the Fire" situation where Technical Support offers suggestions to confirm that the servers are down or unresponsive and clean up and recover from the problem immediately. This document provides tips to evaluate and restore your SAS Stored Process Servers.
2. Then, a long-term strategy is needed to gather information and determine why the problem occurs. See SAS Note 43163 "Tips for addressing unresponsive SAS^® 9.2 Stored Process Servers, Part 2" for a suggested approach.

Conducting Short-Term Troubleshooting to "put out the fire"

Use the following five–step approach to evaluate the status of and restore your SAS Stored Process Servers:

1. Test the connection to the stored process server from the SAS^® Management Console.
2. Check the status of the stored process server ports by running the NETSTAT system command.
3. Stop the object spawner.
4. Check applicable server log files.
5. Stop existing stored process servers and restart the object spawner.

Step 1: Test the Connection to the SAS Stored Process Server from the SAS Management Console

A basic test of stored process server functionality is available in the SAS Management Console. To conduct this test, follow the steps below:

1. Open SAS Management Console. Select the Plug–ins tab. The hierarchy appears in the left pane.
2. Click + to expand the Server Manager hierarchy.
3. Click + to expand the SASApp server group hierarchy.
4. Highlight+ the SASApp – Logical Stored Process Server object.
5. Right-click and select Validate the server.

A box with the message Test Connection Successful! appears if the test was successful.

Customers usually receive the following error in SAS Management Console when their stored process servers are unresponsive:

Clicking on the Details button in the error box might display messages similar to the following:

Step 2: Check the Status of the Stored Process Server Ports

The command line tool NETSTAT (network statistics) displays incoming and outgoing network connections, routing tables, and various network interface statistics. This tool is available on UNIX and Windows operating systems. Use this tool to provide information about the status of the ports on which a particular stored process server runs, which by default are ports 8611, 8621, and 8631.

Follow these steps:

1. Execute the following command in a Command prompt window:

• Windows

Note: substitute 8621 or 8631 in order to check the other ports.

• UNIX

prompt> netstat –an | grep "8611"

Note: substitute 8621 or 8631 in order to check the other ports.

2. Evaluate the port status.

• LISTENING and ESTABLISHED are normal states.
• TIME_WAIT and CLOSE_WAIT are normal states if the connection is shutting down.

Note: A state of CLOSE_WAIT that persists for longer than 2-5 minutes might indicate that the server is unresponsive.

See RFC793 (pages 20 and 21) for details about the progression of states for a TCP/IP connection.

The appearance of NETSTAT output when port 8611 is unresponsive is shown in the example below.

tcp4    1312      0  myserver.na.sas.8611  otherserver.na.sas.56480 CLOSE_WAIT
tcp4    1254      0  myserver.na.sas.8611  otherserver.na.sas.56487 CLOSE_WAIT
tcp4    1300      0  myserver.na.sas.8611  otherserver.na.sas.56498 CLOSE_WAIT
tcp4       0         0  *.8611                 *.*                    LISTEN
tcp4    1280      0  myserver.na.sas.8611  otherserver.na.sas.56805 CLOSE_WAIT
tcp4    1011      0  myserver.na.sas.8611  otherserver.na.sas.56816 CLOSE_WAIT
tcp4    1267      0  myserver.na.sas.8611  otherserver.na.sas.56822 CLOSE_WAIT
tcp4    1234      0  myserver.na.sas.8611  otherserver.na.sas.56825 CLOSE_WAIT
tcp4    1260      0  myserver.na.sas.8611  otherserver.na.sas.56828 CLOSE_WAIT
tcp4       0         0  *.8621                 *.*                    LISTEN
tcp4       0         0  *.8631                 *.*                    LISTEN
tcp4    1299      0  myserver.na.sas.8611  otherserver.na.sas.56850 CLOSE_WAIT
tcp4    1280      0  myserver.na.sas.8611  otherserver.na.sas.56854 CLOSE_WAIT
tcp4    1311      0  myserver.na.sas.8611  otherserver.na.sas.56865 CLOSE_WAIT
tcp4    1269      0  myserver.na.sas.8611  otherserver.na.sas.32775 ESTABLISHED
tcp4    1273      0  myserver.na.sas.8611  otherserver.na.sas.32781 ESTABLISHED
tcp4    1281      0  myserver.na.sas.8611  otherserver.na.sas.57282 CLOSE_WAIT
tcp4    1272      0  myserver.na.sas.8611  otherserver.na.sas.32786 ESTABLISHED
tcp4    1322      0  myserver.na.sas.8611  otherserver.na.sas.57288 CLOSE_WAIT
tcp4    1284      0  myserver.na.sas.8611  otherserver.na.sas.32795 ESTABLISHED
tcp4    1302      0  myserver.na.sas.8611  otherserver.na.sas.57298 CLOSE_WAIT
tcp4    1296      0  myserver.na.sas.8611  otherserver.na.sas.32804 ESTABLISHED               

Step 3: Evaluate SAS Processes after Terminating the Object Spawner

By default, the SAS Stored Process Server is configured to execute using the sassrv user account.

If executing the NETSTAT command reveals unresponsive servers, you should terminate the object spawner and search for remaining SAS processes that are owned by the sassrv user account. Any SAS processes owned by the sassrv account that persist after the Object Spawner shuts down are likely to be unresponsive SAS Stored Process Servers (although other possible explanations exist).

Follow these steps:

1. Stop the object spawner.

    

   • Windows

     Stop the object spawner service through the Windows Services Manager.

   • UNIX

     Locate the ObjectSpawner.sh file in your SAS Configuration directory. For example:
     !SASROOT/configuration-directory/Lev1/ObjectSpawner/ObjectSpawner.sh
     
     From a system prompt, change directories to where the ObjectSpawner.sh script is located and submit the following:

     prompt> ./ObjectSpawner.sh stop

      

2. Search for remaining SAS processes that still persist after you terminate the object spawner and are owned by the sassrv user account (or equivalent at your site):

    

    

   • Windows

     Use the Windows Task Manager.

     From the Processes tab, sort the process list by the User Name column and look for processes with an associated user name of sassrv.

   • UNIX

     Use the PS command.

     prompt> ps-ef | grep "sassrv"
     

      

   • AIX

     prompt>  ps –ef | grep "sassrv" | grep "8611"
     prompt>  ps –ef | grep "sassrv" | grep "8621"
     prompt>  ps –ef | grep "sassrv" | grep "8631"
     

   • HP-UX or Solaris

     prompt> ps –ef | grep "sassrv" | grep "sasexe/sas"
     prompt> ps –ef | grep "sassrv"
     

Step 4: Check Applicable Server Log Files

When stored process servers become unresponsive, error and warning messages might appear in log files for the object spawner, stored process servers, metadata servers, and (for Windows systems only) the Windows Event Viewer.

You should review all the log file types noted below.

Checking Object Spawner Logs

Go to the appropriate folder and view the object spawner log file.

• Windows

  Example Windows path: C:\SAS\configuration-directory/Lev1\ObjectSpawner\Logs

  Windows filename: ObjectSpawner_yyyy-mm-dd_pid.log

• UNIX

  Example UNIX path: !SASROOT/configuration-directory/Lev1/Logs

  UNIX filename: ObjectSpawner_console.log

  UNIX filename: ObjectSpawner_yyyy-mm-dd_pid.log

   

   

  Caution: In order to preserve these log files and document specific incidents, it is critical that you copy the object spawner console logs to backup files before restarting the object spawner. The existing object spawner console log files will be overwritten when object spawner starts.

   

  Note: After a stored process server becomes unresponsive, you will find one or more of the following messages written to the object spawner log file named ObjectSpawner_yyyy-mm-dd_pid.log:

  WARN  Lost contact with the server when calling an IOM interface.
  ERROR The launch of server SASApp - Stored Process Server for user sassrv failed. 
  WARN  Load Balancing interface call failed with exception 
  ERROR An error occurred while server (A5JHY10U.AW00000A_!A5JHY10U.AV00000J_@hostname.com) 
        was starting.  Now attempting a different server.
  ERROR Unable to find an available server to redirect client.
  INFO  Failed to redirect client to available server in cluster SASApp - Logical Stored 
        Process Server (A5JHY10U.AU000005).
  

Checking Stored Process Server Logs

Go to the appropriate folder and view the stored process server log file.

• Windows

  Example Windows path: C:\SAS\project-directory\Lev1\SASApp\StoredProcessServer\Logs

  Windows filename: SASApp_STPServer_yyyy-mm-dd_pid.log

• UNIX

  Example UNIX path: !SASROOT/configuration-directory/Lev1/Logs

  UNIX filename: SASApp_STPServer_yyyy-mm-dd_pid.log

It is possible that the log from an unresponsive stored process server will contain no errors. In this case, you should note the last step or program that executed successfully and the last entry that was written in the log.

Often the last entry in an unresponsive stored process server log file will note that a request has started executing, similar to the example below:

INFO  [00000417] 4:sasdemo - STP: 1: Executing c:\mycode stp_report.sas

If servers are unresponsive and you restart object spawner, log entries similar to those below will appear in the log file when the stored process server attempts to restart. See example below:

ERROR [00000007] :sassrv - The TCP/IP tcpSockBind support routine failed with error 10048   
      (The specified address is already in use.).
ERROR [00000007] :sassrv - Bridge Protocol Engine Socket Access Method was unable to bind 
      the listen socket to port 8611.
ERROR [00000007] :sassrv - The Bridge Protocol Engine Socket Access Method listen thread 
      failed during the definition of the server listen.
INFO  [00000007] :sassrv - Bridge protocol engine is unloading.

Note: These errors will occur because the server is still bound to the designated port that is trying to start. If the stored process server log contains the statements above, you will need to stop the existing stored process servers and restart the object spawner.

For instructions, see the section "Step 5: Stop Existing Stored Process Servers and Restart the Object Spawner" below.

Checking the Metadata Server Log

Go to the appropriate folder and view the metadata server log file.

• Windows

  Example Windows path: C:\SAS\project-directory\Lev1\SASMeta\MetadataServer\Logs

  Windows filename: SASMeta_MetadataServer_<yyyy-mm-dd_pid>.log

• UNIX

  Example UNIX path: project-directory/!SASROOT/configuration-directory/Lev1/Logs

  UNIX filename: SASMeta_MetadataServer_yyyy-mm-dd_pid.log

  Note: Normally, there are no errors logged by the metadata server when stored process servers become unresponsive. However, you should check this log as a precautionary measure.

Checking the Windows Event Viewer for Errors or Clues (Windows operating system only)

The Windows Event Viewer is a Windows System Tool found in the Microsoft Management Console. Follow these tasks to check the Windows Event Viewer for errors:

1. From the Windows Control Panel open Administrative Tools ► Computer Management.
2. Expand System Tools then expand the Event Viewer tool.
3. From Event Viewer, click on each log file and in the right-hand pane look for any SAS event or other activity that occurred during the same date and time as the unresponsive condition of the stored process servers.

Step 5: Stop Existing SAS Stored Process Servers and Restart the SAS Object Spawner

1. Retrieve the listing of process IDs for unresponsive SAS Stored Process servers that you noted after terminating the SAS Object Spawner.
2. Stop the unresponsive processes.

    

   • Windows

     Use Task Manager or the kill command to terminate the unresponsive SAS Stored Process servers.

   • UNIX

     Use the kill command. See the example below:

     prompt> kill <process ID#>
3. Restart the object spawner.

    

   • Windows

     Start the Object Spawner service through the Windows Services manager.

   • UNIX

     Locate your ObjectSpawner.sh file in your SAS Configuration directory. For example:

     !SASROOT/configuration-directory/Lev1/ObjectSpawner/ObjectSpawner.sh

     From a system prompt, change directories to where the ObjectSpawner.sh script is located and submit

     prompt> ./ObjectSpawner.sh start
4. Using SAS Management Console, test the connection to the SAS Stored Process server to ensure that the servers were successfully restored.
5. Clean up leftover WORK library files. These files accumulate as a result of abnormal termination of the stored process server sessions.
   • Windows

     Use the SAS Disk Cleanwork Handler. For further information about this program see SAS Note 8786 "SAS Disk Cleanup Handler replaces a SAS 9.1 program called cleanworkpc.sas to clean work directories on Windows systems."

   • UNIX

     Use the cleanwork tool. For further information, see the cleanwork Command chapter in the SAS^® 9.2 Companion for UNIX Environments.

If the above steps do not restore the SAS Stored Process Servers to normal functionality, please compile results from the tests and the log files referenced in these steps and contact SAS Technical Support for further assistance.

If you are experiencing unresponsive stored process servers using SAS 9.1.3, please refer to the following notes:

Usage Note 30952 "Tips for addressing unresponsive SAS® 9.1.3 Stored Process Servers, Part 1"

Usage Note 30716 "Tips for addressing unresponsive SAS® 9.1.3 Stored Process Servers, Part 2"

Operating System and Release Information