Skip to content

Instantly share code, notes, and snippets.

@senthilcodr
Created September 13, 2019 07:38
Show Gist options
  • Save senthilcodr/55f26072c0e33dad985f5311ea13af5a to your computer and use it in GitHub Desktop.
Save senthilcodr/55f26072c0e33dad985f5311ea13af5a to your computer and use it in GitHub Desktop.
Zalando Standby Cluster using MinIO

To set-up wal shipping in primary cluster:

  1. In the file manifests/configmap.yaml, uncomment the line containing wal_s3_bucket and set it to the bucket name created in MinIO.
  2. And then, add the following lines to the same file:
  pod_environment_configmap: postgres-pod-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-pod-config
  namespace: default
data:
  AWS_ACCESS_KEY_ID: <MinIO access key>
  AWS_SECRET_ACCESS_KEY: <MinIO secret key>
  AWS_S3_FORCE_PATH_STYLE: "true"
  AWS_REGION: us-east-1
  AWS_ENDPOINT: <MinIO endpoint (e.g. http://<MinIO server IP>:9000)>
  USE_WALG_BACKUP: "true"
  WALG_DISABLE_S3_SSE: "true"

To set-up replication in standby cluster:

  1. Add the following to the file manifests/configmap.yaml
  pod_environment_configmap: postgres-pod-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-pod-config
  namespace: default
data:
  STANDBY_AWS_ACCESS_KEY_ID: <MinIO access key>
  STANDBY_AWS_SECRET_ACCESS_KEY: <MinIO secret key>
  STANDBY_WALE_ENV_DIR: /home/postgres/etc/wal-e.d/env-standby
  STANDBY_USE_WALG_RESTORE: "true"
  STANDBY_AWS_S3_FORCE_PATH_STYLE: "true"
  STANDBY_AWS_REGION: us-east-1
  STANDBY_AWS_ENDPOINT: <MinIO endpoint (e.g. http://<MinIO Server IP>:9000)>
  1. In the file manifests/standby-manifest.yaml, edit s3_wal_path to use the bucket/path name from the MinIO. For example,
s3_wal_path: "s3://<bucket_name>/spilo/acid-minimal-cluster/1a793032-ba83-11e9-8bc8-080027377c7c/wal/"
  1. Finally start the operator/pods with this manifest (standby-manifest.yaml).
@anshudutta
Copy link

Hi,
I am trying to it working for GCP. I have a bucket wit WAL streaming in GCS bucket.
What is the 1a793032-ba83-11e9-8bc8-080027377c7c in your example

@senthilcodr
Copy link
Author

It is the uid of the primary cluster (not standby cluster). You can get it by running the following command:
kubectl get postgresql <primary_cluster_name> -o=jsonpath='{.metadata.uid}'

@memogarcia
Copy link

memogarcia commented Nov 18, 2021

Have you tried this with this version? spilo-14:2.1-p3

In this version it seems that there is a typo somewhere and the wal-fetch command keeps failing with the error/warning

Usage: /scripts/wal-e-wal-fetch.sh wal-fetch [--prefetch PREFETCH] WAL_SEGMENT WAL_DESTINATION

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment