Problem Note 67985: The PostgreSQL cluster fails to restart after you apply the Crunchy Data patch update
Applying updates to the SAS® Infrastructure Data Server (Crunchy Data PostgreSQL) results in a pod that stops responding. Messages similar to the following might appear in the log for the affected pod:
- Mon Apr 26 01:57:25 UTC 2021 INFO: PGHA_INIT is 'true', waiting to initialize as primary
2021-04-26 01:57:25,942 INFO: waiting for end of recovery after bootstrap
- Mon Apr 26 02:06:31 UTC 2021 WARN: Detected an earlier failed attempt to initialize
Mon Apr 26 02:06:31 UTC 2021 INFO: Correct the issue, remove '/pgdata/sas-crunchy-data-postgres.initializing', and try again
Mon Apr 26 02:06:31 UTC 2021 INFO: Your data might be in: /pgdata/sas-crunchy-data-postgres_...
- 2021-04-26 02:14:04,036 INFO: Lock owner: None; I am sas-crunchy-data-postgres-7568cc98f4-klhvf
2021-04-26 02:14:04,086 INFO: waiting for leader to bootstrap
Update Procedure
To successfully apply updates, use the following procedure:
Important Note:
- You must be an administrator with namespace permissions in order to perform the following steps.
- In the following steps, you must replace <cluster-name> with the cluster that you are working on: sas-crunchy-data-postgres, sas-crunchy-data-cpspostgres, or sas-crunchy-data-cdspostgres
- To make it easier to perform these commands, you might want to make an alias:
- alias kc='kubectl -n <namespace>'
- Execute into the hanging primary pod:
- kc exec -it <primary hanging pod> -- bash
- Once in the pod, change directory to the data directory:
- If the <cluster-name>.initializing file exists, then remove it. Note: The file might not exist.
- Here is an example:
- rm sas-crunchy-data-postgres.initializing
- If this directory exists, <cluster-name>_[tmp or timestamp], then move it to make it the primary directory:
- mv sas-crunchy-data-postgres_######## sas-crunchy-data-postgres
- Change directory into the <cluster-name> now:
- cd sas-crunchy-data-postgres
- If any of the following files exist in the cluster directory, remove them:
- rm standby.signal
- rm recovery.signal
- rm recovery.conf
- Exit from the primary pod:
- Edit the clusters configuration map to make sure the 'init' value is set to 'false':
- kc edit configmap <cluster-name>-pgha-config
- Make sure that this is the value for init (and, if not, change it):
- Scale down the cluster <cluster-name>:
- kc scale --replicas=0 deploy sas-crunchy-data-postgres
- Make sure the <cluster-name> secret still exists:
- kc get secret sas-crunchy-data-postgres-tls-secret
- If the <cluster name>-tls-secret does not exist, then download the cluster-name-postgres-tls.zip file to create a new one:
- Download the cluster-name-postgres-tls.zip file to anywhere on your system and unzip it.
- Edit cluster-name-postgres-tls.yaml to rename <cluster-name> in the file to the appropriate cluster name:
- vi cluster-name-postres-tls-yaml
- Change any reference in the file for <cluster-name> to sas-crunchy-data-postgres, sas-crunchy-data-cpspostgres, or sas-crunchy-data-cdspostgres.
- Apply the file:
- kc apply -f cluster-name-postgres-tls.yaml
- Scale the <cluster-name> backup:
- kc scale --replicas=1 deploy sas-crunchy-data-postgres
- Ensure that the cluster comes back up:
- kc get pods | grep -i postgres
For more information about SAS Viya software updates, see Getting Started with Updating Software in SAS® Viya® Operations.
Operating System and Release Information
| SAS System | SAS Viya | Linux for x64 | 2020.0.6 | 2021.1.1 | Viya | Viya |
*
For software releases that are not yet generally available, the Fixed
Release is the software release in which the problem is planned to be
fixed.
| Type: | Problem Note |
| Priority: | high |
| Date Modified: | 2022-12-22 11:21:17 |
| Date Created: | 2021-06-01 13:43:26 |