SUPPORT / SAMPLES & SAS NOTES
 

Support

Problem Note 57428: In a SAS® Metadata Server cluster, the master node does not send redirection requests to a slave node that is a part of the quorum

DetailsHotfixAboutRate It

In a SAS Metadata Server cluster, you might encounter a situation where a slave node is reported to be a part of the quorum, but it does not receive redirection requests from the master node.

This situation is most likely to occur when Enterprise Grid Orchestrator (EGO) is used to start the metadata server cluster nodes or if the metadata servers on each node are all started at approximately the same time. This problem occurs on start-up of the metadata servers when they are trying to connect to each other in order to determine which server will become the master and which server will be the slave.

The issue might also occur when a quorum loss occurs during normal run time. After the quorum loss, if two cluster members both connect to a third cluster member at almost the same time, this situation might occur where one slave joins the quorum without receiving future redirection requests from the master node.

In either situation, the following messages will be seen in the metadata server log of one of the nodes in the metadata server cluster:

2015-10-24T08:58:32,621 WARN [00000012] :sas - <?xml version="1.0" ?><Exceptions></Exceptions> 2015-10-24T08:58:32,622 INFO [00000012] :sas - Connecting server SASMeta - Metadata Server to cluster SASMeta - Logical Metadata Server. 2015-10-24T08:58:32,622 ERROR [00000012] :sas - INTERNAL ERROR: Connecting node SASMeta - Metadata Server has no pIface. 2015-10-24T08:58:32,622 INFO [00000012] :sas - Setting the master node to SASMeta - Metadata Server Node 2. 2015-10-24T08:58:32,622 WARN [00000096] :sas - Internal error: The Cluster Manager node connection was not initialized. 2015-10-24T08:58:32,622 INFO [00000096] :sas - The connecting node is running with a lower model version (0) than the master node (16.01). We assume this is an upgrade and the new node has not yet been upgraded. 2015-10-24T08:58:32,622 WARN [00000096] :sas - The server connecting to the master node has a lower model version and will be shut down since it cannot participate in this cluster until it is upgraded. 2015-10-24T08:58:32,622 WARN [00000096] :sas - Internal error: The Cluster Manager node connection was not initialized. 2015-10-24T08:58:32,622 WARN [00000096] :sas - The server connecting to the master node has a lower model version and will be shut down since it cannot participate in this cluster until it is upgraded. 2015-10-24T08:58:32,622 WARN [00000080] 4:sas - <?xml version="1.0" ?><Exceptions></Exceptions> 2015-10-24T08:58:32,630 INFO [00000080] 4:sasi - Connecting server SASMeta - Metadata Server to cluster SASMeta - Logical Metadata Server. 2015-10-24T08:58:33,262 INFO [00000011] :sas - The cluster has achieved quorum and is now ONLINE.

This problem might be hard to detect at run time, because it requires a review of the metadata server logs. To determine whether you are encountering this problem, you must check the metadata server master node log to see whether redirect requests are going to all slaves or to only a subset of slave nodes.

When this problem occurs, the following message is logged to the metadata server master node log file (SASMeta_MetadataServer_date_host_pid.log) when the connection request is redirected to one of the slave nodes. In this case, the connection request is redirected to server "SASMeta - Metadata Server Node 3":

2015-11-14T10:51:14,620 INFO [00005585] 4:sas - Redirect client in cluster SASMeta - Logical Metadata Server (A4BN3LB9.AX000001) to server SASMeta - Metadata Server Node 3 (A5BN3KB8.AY000003) at myhost.mycompany.com:8564.

Under normal operating conditions, you should also see redirect requests go to the other slave node. Based on the log message above, the other slave node that receives redirection requests might be "SASMeta - Metadata Server Node 2" or "SASMeta - Metadata Server" depending on who the master is.

If you see that ALL of the redirection requests are going to only ONE slave node and not to the other, it is possible that you are encountering the problem described in this note.

To work around this problem, allow more time in between start-up of metadata server cluster nodes. This extra time prevents the condition where two nodes could encounter the error during start-up.

Click the Hot Fix tab in this note to access the hot fix for this issue.



Operating System and Release Information

Product FamilyProductSystemProduct ReleaseSAS Release
ReportedFixed*ReportedFixed*
SAS SystemSAS Metadata ServerMicrosoft® Windows® for x649.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 8 Enterprise 32-bit9.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 8 Enterprise x649.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 8 Pro 32-bit9.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 8 Pro x649.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 8.1 Enterprise 32-bit9.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 8.1 Enterprise x649.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 8.1 Pro9.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 8.1 Pro 32-bit9.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows 109.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows Server 20089.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows Server 2008 R29.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows Server 2008 for x649.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows Server 2012 Datacenter9.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows Server 2012 R2 Datacenter9.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows Server 2012 R2 Std9.49.4_M49.4 TS1M29.4 TS1M4
Microsoft Windows Server 2012 Std9.49.4_M49.4 TS1M29.4 TS1M4
Windows 7 Enterprise 32 bit9.49.4_M49.4 TS1M29.4 TS1M4
Windows 7 Enterprise x649.49.4_M49.4 TS1M29.4 TS1M4
Windows 7 Home Premium 32 bit9.49.4_M49.4 TS1M29.4 TS1M4
Windows 7 Home Premium x649.49.4_M49.4 TS1M29.4 TS1M4
Windows 7 Professional 32 bit9.49.4_M49.4 TS1M29.4 TS1M4
Windows 7 Professional x649.49.4_M49.4 TS1M29.4 TS1M4
Windows 7 Ultimate 32 bit9.49.4_M49.4 TS1M29.4 TS1M4
Windows 7 Ultimate x649.49.4_M49.4 TS1M29.4 TS1M4
64-bit Enabled AIX9.49.4_M49.4 TS1M29.4 TS1M4
64-bit Enabled Solaris9.49.4_M49.4 TS1M29.4 TS1M4
HP-UX IPF9.49.4_M49.4 TS1M29.4 TS1M4
Linux for x649.49.4_M49.4 TS1M29.4 TS1M4
Solaris for x649.49.4_M49.4 TS1M29.4 TS1M4
* For software releases that are not yet generally available, the Fixed Release is the software release in which the problem is planned to be fixed.