In a SAS Real-Time Decision Manager run-time environment, the SAS Federation Server process might become unresponsive under a heavy load.  There is not one specific warning or error message in any of the logs. You might encounter one or more of the following messages, which are symptomatic of a resource-constraint violation.

• In the SAS^® Decision Services engine server log:

  yyyy-mm-dd hh:mm:ss,sss TRACE 24d566a3-7eda-4824-bfd9-fbbc855352a3 com.sas.rtdm.implementation.activity.sasactivity.DS2Activity - DS2Activity.execute() preparing statement using SQL {call _SAS_CONTACT_RULE_DS2_ACTIVITY.execute(CAST(? AS varchar), CAST(? AS varchar), CAST(? AS varchar), CAST(? AS varchar), CAST(? AS varchar), CAST(? AS VARCHAR), CAST(? AS VARCHAR), CAST(? AS DOUBLE), CAST(? AS VARCHAR))}

  yyyy-mm-dd hh:mm:ss,sss ERROR 24d566a3-7eda-4824-bfd9-fbbc855352a3 com.sas.rtdm.implementation.activity.sasactivity.DS2Activity - Unable to execute jdbc statement for resource: $SAS_Activity_Resource(jdbc:sastkts://localhost:24142). Enable debug logging to see details.

  yyyy-mm-dd hh:mm:ss,sss DEBUG 24d566a3-7eda-4824-bfd9-fbbc855352a3 com.sas.rtdm.implementation.activity.sasactivity.DS2Activity - JDBC exception: $SAS_Activity_Resource(jdbc:sastkts://localhost:24142) SQLState: 42000 Err: -2130708373 Msg: Server error occurred while processing invokeExecute2. Error message: [com.sas.tkts.sql.FSException: T_EXECUTE2: Table Service: Function failed.] SQLSTATE:[42000] MESSAGE:[Syntax error or access violation] NATIVEERROR:[-2130708373]

  yyyy-mm-dd hh:mm:ss,sss DEBUG 24d566a3-7eda-4824-bfd9-fbbc855352a3 com.sas.rtdm.implementation.activity.sasactivity.DS2Activity - JDBC exception: $SAS_Activity_Resource(jdbc:sastkts://localhost:24142) SQLState: HY000 Err: 1007669087 Msg: Unrecoverable error during execution of DS2 method. Prepared statement is no longer valid.

• In the SAS® Federation Server log, you might receive one or more of the following messages:

  Message 1

  yyyy-mm-ddThh:mm:ss,sss ERROR [04396535] App.Statement.Statement.Execute2 104:sasfedadm@saspw - 54; sasfedadm; 1.1.9; T_EXECUTE2: Table Service: Function failed.  Diagnostics: SQLState=42000, NativeRC=80FFF86B, Text: Syntax error or access violation.  Diagnostics: SQLState=HY000, NativeRC=3C0FCF5F, Text: Unrecoverable error during execution of DS2 method. Prepared statement is no longer valid.

  yyyy-mm-ddThh:mm:ss,sss DEBUG [04396535] IOM 104:sasfedadm@saspw - IOM LOGIC MapTKStatusTOWoiStatus() Mapping tkstatus: 80fff802

  yyyy-mm-ddThh:mm:ss,sss TRACE [04396535] IOM 104:sasfedadm@saspw - IOM LOGIC MapTKStatusTOWoiStatus() Found exception mapping: exception=exception=3c663aff-0fc8-4b61-b2-bf-be2ee7c2e826

  yyyy-mm-ddThh:mm:ss,sss DEBUG [04396535] App.Statement.Statement.Execute2 104:sasfedadm@saspw - IOM RETURN -1={compRef:7f960b1952e0}->Statement::Execute2(): text=<?xml version="1.0" ?><Exceptions><Exception><SASMessage severity="Error">T_EXECUTE2: Table Service: Function failed.</SASMessage></Exception></Exceptions>

  Message 2

  yyyy-mm-ddThh:mm:ss,sss DEBUG [00000010] IOM.PE.Bridge.Dispatch :sas - Bridge PE [7fbc4d1c5b60] Thread puddle is saturated

  yyyy-mm-ddThh:mm:ss,sss DEBUG [00000010] IOM.PE.Bridge.Dispatch :sas - Bridge PE [7fbc4d1c5b60] Saturated thread puddle request type is OTHER, sending to overflow puddle.

  yyyy-mm-ddThh:mm:ss,sss ERROR [00000010] App.Program :sas - Creating thread.  OS Error: 11 (Resource temporarily unavailable)

  yyyy-mm-ddThh:mm:ss,sss ERROR [00000010] App.Program :sas - Critical TK KERNEL error

  yyyy-mm-ddThh:mm:ss,sss ERROR [00000010] App.Program :sas - Message file is not loaded.

  yyyy-mm-ddThh:mm:ss,sss ERROR [00000010] Logging :sas - tkzCapture() failed

  Message 3

  yyyy-mm-ddThh:mm:ss,sss ERROR [04413911] App.Connection.Connection.DriverConnect 334:sasfedadm@saspw - 181; sasfedadm; 1.1.0; C_DRV_CONNECT: Table Service: Function failed.  Diagnostics: SQLState=HY000, NativeRC=803FC193, Text: Unable to load extension: (tktsqdb).  Diagnostics: SQLState=HY000, NativeRC=803FC194, Text: Extension Load Failure: OS Error: -1 (/opt/sasbin/SASFederationServer/4.2/fedserver/lib/tktsqdb.so: failed to map segment from shared object: Cannot allocate memory).  Diagnostics: SQLState=IM003, NativeRC=80FFF8B3, Text: Could not load tktsqdb Extension..  Diagnostics: SQLState=00000, NativeRC=80FFF9E7, Text: DRIVER keyword missing. Defaulting to DRIVER=ORACLE for data service "CDM"..  Diagnostics: SQLState=00000, NativeRC=80FFF9E7, Text: DRIVER keyword missing. Defaulting to DRIVER=MDS for data service "SASDS_MDS".

If you encounter this issue, your SAS Federation Server process might be exceeding the MAX_MAP_COUNT system limit because of a software issue with memory protection in the code generation process. MAX_MAP_COUNT is the maximum number of memory map areas a process can have. To check this limit on your system, issue the following command:

To verify that your SAS Federation Server process is reaching the limit, determine how many maps the process is currently using by issuing the following command (before killing an unresponsive process):

In this command, fs-pid specifies the process ID of the SAS Federation Server process.

If your process reaches the limit at the time the process becomes unresponsive, you can work around the issue by having your system administrator increase the value of the MAX_MAP_COUNT system limit by at least 50%.

Click the Hot Fix tab in this note to access the hot fix for this issue.

Operating System and Release Information

A fix for this issue for SAS Federation Server 4.2 is available at: