1.6 Backup

In this lab, you will create scheduled backups for the most important cluster components.

Task 1.6.1: Create user workload backups

First, check the backupStorageLocation’s health and name:

oc -n openshift-adp get backupstoragelocations.velero.io

The output should show you that you’ve got backupStorageLocation resource named default with PHASE Available. If the PHASE shows something else, there’s most probably a permissions or typo problem in your dataProtectionApplication resource.

NAME      PHASE       LAST VALIDATED   AGE   DEFAULT
default   Available   4s               3m    true

Now, create a daily backup of the resources in namespace uptime-app-prod with a lifetime of 5 days by creating the following Schedule manifest:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup-uptime-app-prod
  namespace: openshift-adp
spec:
  schedule: '@every 24h'
  template:
    hooks: {}
    includedNamespaces:
      - uptime-app-prod
    ttl: 120h0m0s

Note

This resource file is also available at https://raw.githubusercontent.com/acend/openshift-operations-training/main/content/en/docs/01/resources/schedule_daily-backup-uptime-app-prod.yaml .

oc apply -f https://raw.githubusercontent.com/acend/openshift-operations-training/main/content/en/docs/01/resources/schedule_daily-backup-uptime-app-prod.yaml

In order to test it, let’s trigger a manual backup. We are using the velero cli for this which is pre-installed on your bastion host:

velero -n openshift-adp backup create --from-schedule daily-backup-uptime-app-prod

When the backup is completed, have a look at its status part:

oc -n openshift-adp describe backup | grep -A 9 Status

The output should look similar to this:

Status:
  Completion Timestamp:  2023-04-18T10:04:32Z
  Expiration:            2023-04-28T10:03:59Z
  Format Version:        1.1.0
  Phase:                 Completed
  Progress:
    Items Backed Up:  55
    Total Items:      55
  Start Timestamp:    2023-04-18T10:03:59Z
  Version:            1

Task 1.6.2 Create etcd backup

To be able to restore the cluster in case of a disaster, you will create a scheduled etcd backup.

Create a new project named training-infra-etcd-backup.

Hints

oc new-project training-infra-etcd-backup

Since etcd is running on the control plane nodes, we need to make sure the cronjob’s pods that are creating the etcd snapshots are running on one as well. We can do this by annotating the namespace with the corresponding node selector:

oc patch namespace training-infra-etcd-backup -p \
    '{"metadata":{"annotations":{"openshift.io/node-selector": "node-role.kubernetes.io/master="}}}'

For the cronjob pod to be able to access the etcd files, we need to create a service account with additional permissions by attaching a custom SecurityContextConstraints (SCC) policy:

Service account:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: etcd-backup

SCC:

kind: SecurityContextConstraints
metadata:
  name: privileged-etcd-backup
  annotations:
    kubernetes.io/description: 'privileged allows access to all privileged and host
      features and the ability to run as any user, any group, any fsGroup, and with
      any SELinux context.  WARNING: this is the most relaxed SCC and should be used
      only for cluster administration. Grant with caution.'
priority: null
allowHostDirVolumePlugin: true
allowHostIPC: true
allowHostNetwork: true
allowHostPID: true
allowHostPorts: true
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities:
  - '*'
allowedUnsafeSysctls:
  - '*'
apiVersion: security.openshift.io/v1
defaultAddCapabilities: null
fsGroup:
  type: RunAsAny
groups:
  - system:cluster-admins
  - system:nodes
  - system:masters
readOnlyRootFilesystem: false
requiredDropCapabilities: null
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
seccompProfiles:
  - '*'
supplementalGroups:
  type: RunAsAny
users:
  - "system:serviceaccount:training-infra-etcd-backup:etcd-backup"
volumes:
  - '*'

oc -n training-infra-etcd-backup apply -f https://raw.githubusercontent.com/acend/openshift-operations-training/main/content/en/docs/01/resources/etcd-backup/sa_etcd-backup.yaml
oc -n training-infra-etcd-backup apply -f https://raw.githubusercontent.com/acend/openshift-operations-training/main/content/en/docs/01/resources/etcd-backup/scc_privileged-etcd-backup.yaml

Edit the file ~/ocp4-ops/resources/etcd-backup/secret_etcd-backup-s3-bucket.yaml and add your S3 bucket name for etcd backups:

[...]
AWS_S3_BUCKET: +username+-ops-training-backup-etcd
type: Opaque

Now we can create the secret:

oc -n training-infra-etcd-backup apply -f ~/ocp4-ops/resources/etcd-backup/secret_etcd-backup-s3-bucket.yaml

Finally we can create the ConfigMap containing the backup script and the CronJob resources. Here’s what they look like:

kind: ConfigMap
apiVersion: v1
metadata:
  name: backup-script
data:
  backup: |-
    #!/bin/sh

    set -e
    set -u

    chroot /host /usr/local/bin/cluster-backup.sh /home/core/assets
    aws s3 sync /host/home/core/assets/ "s3://${AWS_S3_BUCKET}/etcd-backup"

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
spec:
  schedule: "5 0,6,12,18 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          nodeSelector:
            node-role.kubernetes.io/master: ''
          hostNetwork: true
          restartPolicy: OnFailure
          serviceAccountName: etcd-backup
          containers:
          - name: etcd-backup
            image: 'docker.io/amazon/aws-cli:latest'
            imagePullPolicy: Always
            resources:
              limits:
                cpu: 500m
                memory: 512Mi
              requests:
                cpu: 10m
                memory: 32Mi
            command:
              - /usr/local/bin/scripts/backup
            envFrom:
              - secretRef:
                  name: etcd-backup-s3-bucket
            securityContext:
              privileged: true
              runAsUser: 0
            volumeMounts:
              - name: host
                mountPath: /host
              - name: backup-script
                mountPath: /usr/local/bin/scripts
          volumes:
            - name: host
              hostPath:
                path: /
                type: Directory
            - name: backup-script
              configMap:
                name: backup-script
                defaultMode: 493
          dnsPolicy: ClusterFirst
          tolerations:
            - operator: Exists

Create them with:

oc -n training-infra-etcd-backup apply -f https://raw.githubusercontent.com/acend/openshift-operations-training/main/content/en/docs/01/resources/etcd-backup/cm_backup-script.yaml
oc -n training-infra-etcd-backup apply -f https://raw.githubusercontent.com/acend/openshift-operations-training/main/content/en/docs/01/resources/etcd-backup/cronjob_etcd-backup.yaml

Note

In our example of the CronJob the backup job is started every six hours. If you do not want to wait that long, trigger the job manually with
oc create job --from=cronjob/etcd-backup -n training-infra-etcd-backup etcd-test-job

You can verify the backup job by looking at the result or by checking the logs of the pod:

Job status:

oc -n training-infra-etcd-backup get jobs

Check the logs:

oc -n training-infra-etcd-backup logs jobs/<job-name>

The logs should not contain any errors and should show the successful upload of the backup to the S3 bucket:

Certificate /etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt is missing. Checking in different directory
Certificate /etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-serving-ca/ca-bundle.crt found!
found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-13
found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-8
found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-7
5c933a1f3ca2a665cbbf37f5155359592eede1e9cdb4f25cebd10cdc9dad7d0b
etcdctl version: 3.5.11
API version: 3.5
{"level":"info","ts":"2024-02-18T12:02:53.735252Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/home/core/assets/snapshot_2024-02-18_120252.db.part"}
{"level":"info","ts":"2024-02-18T12:02:53.741068Z","logger":"client","caller":"v3@v3.5.11/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-02-18T12:02:53.741097Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://10.0.89.148:2379"}
{"level":"info","ts":"2024-02-18T12:02:54.940107Z","logger":"client","caller":"v3@v3.5.11/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2024-02-18T12:02:55.569259Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://10.0.89.148:2379","size":"113 MB","took":"1 second ago"}
{"level":"info","ts":"2024-02-18T12:02:55.569335Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/home/core/assets/snapshot_2024-02-18_120252.db"}
Snapshot saved at /home/core/assets/snapshot_2024-02-18_120252.db
{"hash":2933310916,"revision":480301,"totalKey":10630,"totalSize":112947200}
snapshot db and kube resources are successfully saved to /home/core/assets
upload: ../host/home/core/assets/snapshot_2024-02-18_120022.db to s3://<redacted-bucket-name>/etcd-backup/snapshot_2024-02-18_120022.db
upload: ../host/home/core/assets/static_kuberesources_2024-02-18_120015.tar.gz to s3://<redacted-bucket-name>/etcd-backup/static_kuberesources_2024-02-18_120015.tar.gz
upload: ../host/home/core/assets/static_kuberesources_2024-02-18_120022.tar.gz to s3://<redacted-bucket-name>/etcd-backup/static_kuberesources_2024-02-18_120022.tar.gz
upload: ../host/home/core/assets/static_kuberesources_2024-02-18_120038.tar.gz to s3://<redacted-bucket-name>/etcd-backup/static_kuberesources_2024-02-18_120038.tar.gz
upload: ../host/home/core/assets/snapshot_2024-02-18_120252.db to s3://<redacted-bucket-name>/etcd-backup/snapshot_2024-02-18_120252.db
upload: ../host/home/core/assets/static_kuberesources_2024-02-18_120109.tar.gz to s3://<redacted-bucket-name>/etcd-backup/static_kuberesources_2024-02-18_120109.tar.gz
upload: ../host/home/core/assets/snapshot_2024-02-18_120109.db to s3://<redacted-bucket-name>/etcd-backup/snapshot_2024-02-18_120109.db
upload: ../host/home/core/assets/static_kuberesources_2024-02-18_120252.tar.gz to s3://<redacted-bucket-name>/etcd-backup/static_kuberesources_2024-02-18_120252.tar.gz
upload: ../host/home/core/assets/static_kuberesources_2024-02-18_120155.tar.gz to s3://<redacted-bucket-name>/etcd-backup/static_kuberesources_2024-02-18_120155.tar.gz
upload: ../host/home/core/assets/snapshot_2024-02-18_120015.db to s3://<redacted-bucket-name>/etcd-backup/snapshot_2024-02-18_120015.db
upload: ../host/home/core/assets/snapshot_2024-02-18_120038.db to s3://<redacted-bucket-name>/etcd-backup/snapshot_2024-02-18_120038.db
upload: ../host/home/core/assets/snapshot_2024-02-18_120155.db to s3://<redacted-bucket-name>/etcd-backup/snapshot_2024-02-18_120155.db

Last modified May 1, 2026: Merge 13c5f04a7602b63584a607bb4f840408b1830cbd into 51f8206716f9645f6b09fc31ff5c56df56f7109b (df66222)