You might encounter a problem in which load-balanced object spawners or metadata server cluster nodes do not successfully reconnect after quorum loss. See the section below that describes your issue.

Metadata Server Cluster

Metadata server cluster nodes do not successfully reconnect with each other to re-establish the quorum after quorum loss. This problem occurs with a SAS^® 9.4 deployment that contains a metadata server cluster configured with three or more cluster nodes.  

The following sequence of events occurs to cause the issue:

1. The metadata server cluster is in quorum and running normally. The master node is redirecting client requests to the two or more slave nodes in the cluster. 
2. Then, a connectivity problem occurs that results in two or more nodes losing their connections to each other. This connection issue results in a loss of quorum. Here are some messages that might be logged:
   2017-09-13T06:37:42,520 WARN  [04350166] 168:sas - Load Balancing interface call failed with exception <?xml version="1.0"?><Exceptions>
   <Exception><SASMessage severity="Error">The outcall request did not complete in the time allotted.</SASMessage></Exception></Exceptions>.
   2017-09-13T06:37:42,520 WARN  [04350166] 168:sas - <?xml version="1.0" ?><Exceptions></Exceptions>
   2017-09-13T06:37:42,520 INFO  [04350166] 168:sas - Disconnecting server SASMeta - Metadata Server from cluster SASMeta - Logical Metadata Server.
   2017-09-13T06:37:42,520 WARN  [04350166] 168:sas - Lost contact with the server when calling an IOM interface.
   2017-09-13T06:37:42,520 INFO  [04350166] 168:sas - Setting the master node to SASMeta - Metadata Server.
   2017-09-13T06:37:42,520 INFO  [04350171] :sas - The cluster has lost quorum and is now OFFLINE.
3. When the nodes attempt to reconnect to each other, the following messages might be logged in some cases:
   2017-09-13T06:37:50,191 ERROR [00000014] :sas - INTERNAL ERROR: Connecting node SASMeta - Metadata Server has no pIface.
   2017-09-13T06:37:50,191 WARN  [04350203] :sas - Internal error: The Cluster Manager node connection was not initialized.
   2017-09-13T06:37:50,191 WARN  [04350203] :sas - Internal error: The Cluster Manager node connection was not initialized.
   It is important to note that you can encounter this defect even if these ERROR and WARN messages are not logged. It is possible that the logs might show only connecting messages that appear normal. However, the defect is still occurring. 

In this scenario, the metadata server cluster is not able to repair itself. You must perform a full restart of the metadata server cluster to restore service. 

Load-Balanced Object Spawners

You might also encounter an issue in which the load-balanced object spawners lose connection and do not successfully reconnect with one another.

In this scenario, these are the messages that you would see in the object spawner log:

Click the Hot Fix tab in this note to access the hot fix for this issue.

Note: This issue is fixed in Rev. 940_18w25.

Operating System and Release Information

A fix for this issue for Threaded Kernel IOM 9.4_M5 is available at:

A fix for this issue for Threaded Kernel IOM 9.4_M4 is available at:

A fix for this issue for Base SAS 9.4_M3 is available at: