Your Consul pods fail to start if they fail to elect a cluster leader. You might see messages like the following in the logs of the pods:

The first step in recovering the Consul pods in this scenario is usually to scale the Consul StatefulSet down to zero replicas, then to just one replica, and finally back to the original three replicas. Take the following steps. Note: Replace Viya-namespace in each command below with the namespace where your SAS Viya environment is deployed:

1. Scale the sas-consul-server StatefulSet to zero replicas with the following command:
   
      kubectl -n Viya-namespace scale statefulset sas-consul-server –replicas=0
    
2. Wait until all three Consul pods are terminated completely. To confirm that, run the following command and wait until it returns "condition met" messages:
   
      kubectl -n Viya-namespace wait --for=delete pod sas-consul-server-{0..2}
    
3. After you confirm that there are no Consul pods running, scale the sas-consul-server StatefulSet to just one replica with the following command:
   
      kubectl –n Viya-namespace scale statefulset sas-consul-server –replicas=1
   
   That should bring up a single sas-consul-server-0 pod. 
    
4. Wait for a few minutes and scale the sas-consul-server StatefulSet back to three replicas with the following command:
   
      kubectl -n Viya-namespace scale statefulset sas-consul-server –replicas=3
   
   That should bring up an additional two Consul pods, sas-consul-server-1 and sas-consul-server-2. When you run the following command, you should see three pods- sas-consul-server-0, sas-consul-server-1, and sas-consul-server-2:
   
         kubectl -n Viya-namespace get pods -l app.kubernetes.io/name=sas-consul-server
    
5. Check whether all three pods start successfully after a few minutes.
    
6. If the pods still fail to start, there might be some additional manual steps required. Follow the steps from the Error When Consul Pods Move to Different Nodes section of the SAS^® Viya^® Administration guide. Make sure that you select the correct version of the guide that matches the cadence and version of your SAS Viya software. 

If these steps do not help, contact SAS Technical Support with the following information and logs:
 

• Your SAS Viya cadence (LTS/stable) version
• Describe outputs from each of the three Consul pods by using the commands below:
   
     kubectl -n Viya-namespace describe pod sas-consul-server-0 > sas-consul-server-0-describe.txt
     kubectl -n Viya-namespace describe pod sas-consul-server-1 > sas-consul-server-1-describe.txt
     kubectl -n Viya-namespace describe pod sas-consul-server-2 > sas-consul-server-2-describe.txt
• Provide the logs from each of the three Consul pods by using the commands below:
  
     kubectl -n Viya-namespace logs sas-consul-server-0 > sas-consul-server-0.log
     kubectl -n Viya-namespace logs sas-consul-server-1 > sas-consul-server-1.log
     kubectl -n Viya-namespace logs sas-consul-server-2 > sas-consul-server-2.log
   

• Provide the output that is returned by one of the commands below:
  
  If you have full-stack Transport Layer Security (TLS) enabled: 
  
  kubectl -n Viya-namespace exec -it sas-consul-server-0 -c sas-consul-server -- bash -c "export CONSUL_HTTP_SSL=true; /opt/sas/viya/home/bin/consul operator raft list-peers"
  
  If you have just front-door TLS enabled or no TLS enabled:
  
  kubectl -n Viya-namespace exec -it sas-consul-server-0 -c sas-consul-server -- bash -c "/opt/sas/viya/home/bin/consul operator raft list-peers"

Operating System and Release Information