The information in this SAS Note applies only to SAS 9.4 single-server and clustered-server metadata server environments. For information about resolving this issue for SAS^® 9.3 environments, see SAS Note 45508, "SAS^® Metadata Server in SAS^® 9.3 fails to start if MetadataServerBackupManifest.xml has been moved or deleted."

SAS Metadata Server fails to start if the MetadataServerBackupManifest.xml file has been modified or if the file has been moved or deleted from its default location of SAS-configuration-directory/Lev1/SASMeta/MetadataServer.  

When SAS Metadata Server starts, the MetadataServerBackupManifest.xml file is opened and read. If the file has been moved, deleted, or modified, SAS Metadata Server creates a new file that contains a new, unique GUID. SAS Metadata Server might fail to start because the GUID in the MetadataServerBackupManifest.xml file does not match the GUID in the MetadataServerBackupHistory.xml and MetadataServerBackupConfiguration.xml files. In this situation, the following message is written to the server log:

In addition, the log might contain other similar messages. The type of message is based on the actual underlying symptom that is occurring. Here is an example:

In all cases, you must resolve the issue regarding the incorrect GUID match before SAS Metadata Server can start correctly. Choose the correct section below based on whether you have a single-server configuration or a clustered-server configuration.  

Configuration with a Single Metadata Server 

Recovery Steps

In this example, for the command syntax, sas in sas@ is a placeholder for the SAS installer user ID. myhost in the command syntax is the host name of the machine that is failing and where the metadata server runs.

1. Confirm that the metadata server process is not running by checking the output of the ./MetadataServer.sh -status command and the output of ps –ef | grep SAS-installer-user.

2. Stop all other SAS processes that are running in the environment.  



3. Rename the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Backups directory:
      [sas@myhost MetadataServer]$ mv Backups Backups.bak


4. Create a new empty SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Backups directory:
      [sas@myhost MetadataServer]$ mkdir Backups


5. Rename the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhost MetadataServer]$ mv Journal Journal.bak


6. Create a new empty SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhost MetadataServer]$ mkdir Journal


7. Rename the MetadataServerBackupManifest.xml file:
      [sas@myhost MetadataServer]$ mv MetadataServerBackupManifest.xml MetadataServerBackupManifest.xml.bak


8. Start only the metadata server. Do not use the sas.servers script to start all of the SAS processes on the machine.  
      [sas@myhost MetadataServer]$ ./MetadataServer.sh start
   
   After you submit the command, you see the following output:
      Waiting up to 600 seconds for Metadata Server to listen to port 8561.
      Server is started (pid 3418)...
      Metadata server is responding.


9. When the server starts up, a new MetadataServerBackupManifest.xml file is generated. This new file contains a different server GUID for the metadata server when compared to the one seen in the previous MetadataServerBackupManifest.xml file.  
   If you enter this command, you can see the value of the GUID for any given metadata server:
      [sas@myhost MetadataServer]$ xmllint --format MetadataServerBackupManifest.xml | head
   
   After you submit the command, you see output that is similar to the following:
      <?xml version="1.0" encoding="UTF-8"?>
      <MetadataServerBackupManifest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="MetadataServerBackupManifest.xsd" ClusterManifestRevisionNumber="7">
     <MetadataServer GUID="C45505D9-A922-CA49-B36A-1690A0EF6838" BackupLocation="Backups" JournalType="ROLL_FORWARD" SASVersion="9.04.01M5P091317" Pathname="/usr/local/SASConfig/Lev1/SASMeta/MetadataServer">


10. When the server starts up, a new metadata backup is created and put into the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Backups directory.


11. The last step is to log on to the metadata server using SAS^® Management Console. Reset the metadata backup schedule and the number of retention days to the desired values. In order to view the previous values that had been in place before the issue occurred, look at the MetadataServerBackupConfiguration.xml file located in the following directory: SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Backups.bak.

• In SAS Management Console, navigate to the following section and make the appropriate configuration changes: Environment Management ► Metadata Manager ► Metadata Utilities ► Server Backup.
• Under Server Backup, right-click Backup Schedule and enter the desired value.
• Also, under Server Backup, right-click Backup Configuration and select Number of days to retain backups. Then, enter the desired value.

Configuration with Clustered Metadata Servers

Recovery Steps

Stop all metadata server nodes and all other SAS processes that are running in the deployment using one of the following four options.

Option 1  

Back up the Journal and Backups directory and rename the MetadataServerBackupManifest.xml file on the failing node and start the metadata server to generate a new one. In the example below, the failing node is myhostZ. For the command syntax, sas in sas@ is a placeholder for the SAS installer user ID. Make sure to note on which host the commands listed below should be invoked. 

1. Rename the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhostZ MetadataServer]$ mv Journal Journal.bak
    
2. Create a new empty SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhostZ MetadataServer]$ mkdir Journal
    
3. Rename the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Backups directory:
      [sas@myhostZ MetadataServer]$ mv Backups Backups.bak
4. Create a new empty SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Backups directory:
      [sas@myhostZ MetadataServer]$ mkdir Backups
    
5. Rename the MetadataServerBackupManifest.xml file:
      [sas@myhostZ  MetadataServer] $ mv MetadataServerBackupManifest.xml MetadataServerBackupManifest.xml.bak
6. Start the metadata servers on the other nodes that had not failed to start.  
      [sas@myhostX  MetadataServer] $ ./MetadataServer.sh start
      [sas@myhostY  MetadataServer] $ ./MetadataServer.sh start
   
   Ensure that a quorum is established by looking at the Active Server properties in SAS Management Console. Navigate to Environment Management ► Metadata Manager ► Active Server. Right-click Properties and select the Cluster tab to view the properties.


7. Start the metadata server on the original failing node.
      [sas@myhostZ  MetadataServer] $  ./MetadataServer.sh start


8. Check the Active Server properties again to see that the node has now joined the cluster and that all nodes are part of the quorum.

Option 2

Determine whether the failing node performed a backup that has not yet been purged via the retention policy. In the example below, the failing node is myhostZ. For the command syntax, sas in sas@ is a placeholder for the SAS installer user ID. Make sure to note on which host the commands listed below should be invoked.  

1.  Perform a grep search on the SAS-configuration-directory/SASMeta/MetadataServer/Logs (or equivalent) directory for the failing node:
      [sas@myhostZ  Logs]$    grep "The Backup has completed successfully" *myhostZ*.log
   
   After you submit the command, you see output that is similar to the following:
      SASMeta_MetadataServer_2018-09-17_myhost_4598.log:2018-09-17T13:06:23,097 INFO
      [00000217] :sas - The Backup has completed successfully.
    
2. If the backup was taken at the date and time indicated by the log message and is still available in the shared metadata backup location (the shared-backup-location directory), then continue with the next steps. 
   
   In the example above, you need to check to see whether the backup file shared-backup-location/2018-09-17T13_06_23-05_00 still exists.  
   If not, then move to the "Option 3" section below.  
    
3. Make a file system backup of both the shared-backup-location directory and the SAS-configuration-directory/SASMeta/MetadataServer/ directory. For example, to back up the MetadataServer directory, run this command:
      [sas@myhostZ SASMeta] $   tar cvf MetadataServer.tar MetadataServer 
    
4. Rename the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhostZ MetadataServer]$ mv Journal Journal.bak
5. Create a new empty SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhostZ MetadataServer]$ mkdir Journal
    
6. Rename the MetadataServerBackupManifest.xml file:
      [sas@myhostZ  MetadataServer] $ mv MetadataServerBackupManifest.xml MetadataServerBackupManifest.xml.bak
   ​​​​​​​
7. Copy the MetadataServerBackupManifest.xml from the backup directory identified in steps 1 and 2 to the current SAS-configuration-directory/Lev1/SASMeta/MetadataServer directory.
   
   Note: It is important to make sure that the failing node is the one that took the backup from which you are copying the MetadataServerBackupManifest.xml file.  
      [sas@myhostZ  MetadataServer] $  cp -p shared-backup-location/date-and-time/MetadataServerBackupManifest.xml 
    
8. Start the metadata servers on the other nodes that had not failed to start.  
      [sas@myhostX  MetadataServer] $ ./MetadataServer.sh start
      [sas@myhostY  MetadataServer] $ ./MetadataServer.sh start
   
   Ensure that a quorum is established by looking at the Active Server properties in SAS Management Console. Navigate to Environment Management ► Metadata Manager ► Active Server. Right-click Properties and select the Cluster tab to view the properties.
    
9. Start the metadata server on the original failing node.
      [sas@myhostZ  MetadataServer] $  ./MetadataServer.sh start
   ​​​​​​​
10. Check the Active Server properties again to see that the node has now joined the cluster and that all nodes are part of the quorum.

Option 3

If the failing node did not perform a backup that is still available in the shared-backup-location directory, then you need to check to see whether the SAS-configuration-directory/SASMeta/MetadataServer/Backups directory exists locally on the failing node. In this example, myhostZ is the failing node. For the command syntax, sas in sas@ is a placeholder for the SAS installer user ID.

1. Check the SAS-configuration-directory/SASMeta/MetadataServer/Backups directory on the failing node's local file system. If the failing node is not the "first node" of the metadata cluster, then it is likely that there are two backups in this directory with timestamps that correspond to the time at which the node was initially deployed. If there is at least one backup directory present here, then proceed with the next steps. If not, then move to the "Option 4" section below.  


2. Make a file system backup of both the shared-backup-location directory and the SAS-configuration-directory/SASMeta/MetadataServer/ directory. For example, to back up the MetadataServer directory, run this command:
      [sas@myhostZ SASMeta] $   tar cvf MetadataServer.tar MetadataServer


3. Rename the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhostZ MetadataServer]$ mv Journal Journal.bak


4. Create a new empty SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhostZ MetadataServer]$ mkdir Journal


5. Rename the MetadataServerBackupManifest.xml file:
      [sas@myhostZ  MetadataServer] $ mv MetadataServerBackupManifest.xml MetadataServerBackupManifest.xml.bak


6. Copy the MetadataServerBackupManifest.xml file from the dated backup directory identified in step 1 to the current SAS-configuration-directory/Lev1/SASMeta/MetadataServer directory: 
      [sas@myhostZ  MetadataServer] $  cp -p Backups/date-and-time/MetadataServerBackupManifest.xml .


7. Start the metadata servers on the other nodes that had not failed to start: 
      [sas@myhostX  MetadataServer] $ ./MetadataServer.sh start
      [sas@myhostY  MetadataServer] $ ./MetadataServer.sh start
   
   Ensure that a quorum is established by looking at the Active Server properties in SAS Management Console. Navigate to Environment Management ► Metadata Manager ► Active Server. Right-click Properties and then select the Cluster tab to view the properties.


8. Start the metadata server on the original failing node:
      [sas@myhostZ  MetadataServer] $  ./MetadataServer.sh start


9. Check the Active Server properties again to see that the node has now joined the cluster and that all nodes are part of the quorum.

Option 4

If the failing node has not performed a backup that is available in either the shared-backup-location directory or the SAS-configuration-directory/SASMeta/MetadataServer/Backups directory, then use the following steps on the failing node. In this example, the failing node is myhostZ. For the command syntax, sas in sas@ is a placeholder for the SAS installer user ID.

1. Ensure that all SAS services that are running are stopped on all hosts in the deployment. 


2. Make a file system backup of both the shared-backup-location directory and the SAS-configuration-directory/SASMeta/MetadataServer/ directory. For example, to back up the MetadataServer directory, run this command:
      [sas@myhostZ SASMeta] $   tar cvf MetadataServer.tar MetadataServer


3. Rename the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhostZ MetadataServer]$ mv Journal Journal.bak


4. Create a new empty SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Journal directory:
      [sas@myhostZ MetadataServer]$ mkdir Journal


5. Rename the SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Backups directory:
      [sas@myhostZ  MetadataServer] $ mv Backups Backups.bak


6. Create a new empty SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Backups directory:
      [sas@myhostZ  MetadataServer] $ mkdir Backups


7. Rename the MetadataServerBackupManifest.xml file:
      [sas@myhostZ  MetadataServer] $ mv MetadataServerBackupManifest.xml
      MetadataServerBackupManifest.xml.bak


8. Start only the metadata server on the failing node, myhostZ:
      [sas@myhostZ  MetadataServer] $ ./MetadataServer.sh start
   
   This step creates a new MetadataServerBackupManifest.xml file for the failing node.  
   The MetadataServerBackupManifest.xml file now contains the default backup location of BackupLocation="Backups" and does not contain the ClusterGUID="<value>" attribute on the "MetadataServer" element.  


9. Stop the metadata server on the failing node:
      [sas@myhostZ  MetadataServer] $ ./MetadataServer.sh stop


10. Start the metadata servers on the non-failing nodes and wait for them to achieve a quorum:
       [sas@myhostX  MetadataServer] $ ./MetadataServer.sh start
       [sas@myhostY  MetadataServer] $ ./MetadataServer.sh start
    
    Ensure that a quorum is established by looking at the Active Server properties in SAS Management Console. Navigate to Environment Management ► Metadata Manager ► Active Server. Right-click Properties and then select the Cluster tab to view the properties.


11. Start the metadata server on the original failing node:
       [sas@myhostZ  MetadataServer] $  ./MetadataServer.sh start
    
    The MetadataServerBackupManifest.xml file on the original failing node now contains the proper values for BackupLocation="shared-backup-location" and ClusterGUID="<value>". The master node in the cluster instructed the joining slave (failing node) to point to the correct shared backup location and to adopt the correct cluster GUID that represents the cluster.


12. Check the Active Server properties again to see that the node has now joined the cluster and that all nodes are part of the quorum.
    Note: The previously failing node (myhostZ) now has a different server GUID than before. When the server started up, a new MetadataServerBackupManifest.xml file is generated. This new file contains a different server GUID for the metadata server when compared to the one seen in the previous MetadataServerBackupManifest.xml file.  
    
    If you enter the following command, you can see the value of the GUID for any given metadata server:
    [sas@myhost MetadataServer] xmllint --format MetadataServerBackupManifest.xml | ​​​​​head 
    
    After you submit the command, you see output that is similar to the following:  
       <?xmlversion="1.0" encoding="UTF-8"?> <MetadataServerBackupManifest
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
       xsi:noNamespaceSchemaLocation="MetadataServerBackupManifest.xsd"  
       ClusterManifestRevisionNumber="7">
       <MetadataServer GUID="C45505D9-A922-CA49-B36A-1690A0EF6838" BackupLocation="Backups"     
       JournalType="ROLL_FORWARD" SASVersion="9.04.01M5P091317"  
       Pathname="/usr/local/SASConfig/Lev1/SASMeta/MetadataServer">

Operating System and Release Information