1. Introduction
Periodic backup of data is an important part of any production database deployment, which helps ensure data recovery in the event of a disaster and it also minimizes data inconsistency when a restore is required.
Couchbase provides cbbackupmgr utility that has been improved over the years to become an enterprise-grade Backup and Restore tool to backup large data sets with much higher performance therefore we recommend this tool to be used in the production. It is worth mentioning that in Couchbase Server 6.5 we have overhauled the backup-storage engine completely, and introduced higher compression ratio, which has resulted in much improved backup-restore performance and reduced storage requirements for each backup snapshot, resulting in cost saving.
2. Best Practice
Although cbbackupmgr exists under Couchbase_HOME, it is not recommended to run this utility from any of the active nodes in the cluster. As it would be competing for resources of active requests and could potentially hamper the performance of your database system.
It is, therefore, a best practice to provide a separate instance (for backup and restore needs) with only the Couchbase binaries installed but no Couchbase services running, so resources can be better managed for both the database cluster and the backup node.
As can be seen from the above figure, a separate backup/restore node is provisioned in addition to a five node Couchbase cluster. Another best practice is to allocate sufficient storage to hold at least 5x the Couchbase data set size so there is enough space to store required snapshots of the database to meet the Recovery Point Objective (RPO) of the business.
3. Backup Strategy
cbbackupmgr provides a suite of commands which enables DBAs to implement a backup strategy that best suite their business needs. Here are some of the commands:
- cbbackupmgr backup – Backs up data from a Couchbase cluster.
- cbbackupmgr compact – Compacts a backup
- cbbackupmgr merge – Merges backups together
- cbbackupmgr config – Creates a new backup repository
- cbbackupmgr list – Lists backups in the archive
Using these commands one can implement any of the three backup strategies as mentioned in the documentation. In the example below, we will describe Periodic Merge strategy, in context of Couchbase Cluster running within Kubernetes environment.
4. Periodic Merge
This backup strategy is intended to have the lowest database overhead as it requires the least amount of time to backup the changes and practically no resources consumption from the database cluster to consolidate the data during the compaction and merge process (as it happens on the backup node).
On a high level here is how Periodic Merge strategy works:
- Setup backup repository using cbbackupmgr config
- Take an incremental backup of the database (in the repository) using cbbackupmgr backup
- Perform backup compaction using cbbackupmgr compact so that disk space can be efficiently used.
- Merge ‘n’ oldest backups using cbbackupmgr merge so that the number of backups in the repository doesn’t grow infinitely and space requirements remain under check.
Note: The above steps are captured in the backup-with-periodic-merge.sh script, which we will later use in our Kubernetes setup to take periodic backups.
5. Backup Couchbase Data
In my last blog on Couchbase Autonomous Operator, I have described step-by-step on how to deploy self-healing, highly-available Couchbase cluster using Persistent Volumes. Assuming you have followed those steps and deployed the cluster already, steps below will describe how you can setup automatic backup capability using cronjob. It is considered best practice to regularly backup your data, and also test restoring backups to confirm the restore process before disaster recovery is actually required.
This functionality is not provided by the Operator and left to the cluster administrator to define backup policies and test data restoration. This section describes some common patterns that may be employed to perform the required functions.
5.1. Create Storage Class
The Kubernetes resource definitions below illustrate a typical arrangement for backup that saves the state of the entire cluster. We would need to first define the StorageClass which we will format using xfs for the most optimal performance.
# Create storage class for backup/restore operations
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
labels:
k8s-addon: storage-aws.addons.k8s.io
name: gp2-backup-storage
parameters:
type: gp2
fsType: xfs
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumerUsing above definition in backup-sc.yaml file, we can create storage class like this:
$ kubectl create -f backup-sc.yaml -n emart5.2. Create Persistent Volume
A persistent volume is claimed to keep data safe in the event of an outage. You will need to plan the claim size based on your expected data set size, the number of days data retention and whether incremental backups are used at all.
# Define backup storage volume
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: backup-pvc
spec:
storageClassName: gp2-backup-storage
resources:
requests:
storage: 50Gi
accessModes:
- ReadWriteOnceSave above definition in backup-pvc.yaml and create the claim:
$ kubectl create -f backup-pvc.yaml -n emart5.3. Configure Backup repository
Before we can begin taking snapshots of our data periodically, we need to configure the backup archive location. A job is created to mount the persistent volume and initialize a backup repository. The repository is named couchbase which will map to the cluster name in later specifications.
# Create a backup repository
kind: Job
apiVersion: batch/v1
metadata:
name: couchbase-cluster-backup-config
spec:
template:
spec:
containers:
- name: backup-config
image: couchbase/server:enterprise-6.5.0
command: ["cbbackupmgr", "config", "--archive", "/backups", "--repo", "couchbase"]
volumeMounts:
- name: "couchbase-cluster-backup-volume"
mountPath: "/backups"
volumes:
- name: couchbase-cluster-backup-volume
persistentVolumeClaim:
claimName: backup-pvc
restartPolicy: NeverSave above definition in config.yaml and create backup repository:
$ kubectl create -f config.yaml -n emart5.3. Run Backup as CronJob
Create a cronjob as described in the periodic-backup.yaml file, which takes a backup of the Couchbase cluster by a) downloading the backup script in the pod b) running the script and taking backup of the cluster data using the persistent storage volume.
kind: CronJob
apiVersion: batch/v1beta1
metadata:
name: couchbase-cluster-backup-create
spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
#Delete backup-with-periodic-merge script so that new one can be pulled with each run
- name: delete-script
image: couchbase/server:enterprise-6.5.0
command: ["rm", "/backups/backup-with-periodic-merge.sh"]
volumeMounts:
- name: "couchbase-cluster-backup-volume"
mountPath: "/backups"
initContainers:
#Download the backup script from the git repo
- name: wget-backup-script
image: couchbase/server:enterprise-6.5.0
command: ["wget", "https://raw.githubusercontent.com/couchbaselabs/cboperator-hol/master/eks/cb-operator-guide/files/sh/backup-with-periodic-merge.sh", "-P", "/backups/."]
volumeMounts:
- name: "couchbase-cluster-backup-volume"
mountPath: "/backups"
#Change the mod of the backup script to execution
- name: chmod-script
image: couchbase/server:enterprise-6.5.0
command: ["chmod", "700", "/backups/backup-with-periodic-merge.sh"]
volumeMounts:
- name: "couchbase-cluster-backup-volume"
mountPath: "/backups"
#Run the script so it can do a) Backup b) Compaction c) Merge with each snapshot
- name: periodic-merge
image: couchbase/server:enterprise-6.5.0
command: ["sh", "-c" ,"/backups/backup-with-periodic-merge.sh --cluster cbdemo-srv.emart.svc"]
volumeMounts:
- name: "couchbase-cluster-backup-volume"
mountPath: "/backups"
volumes:
- name: couchbase-cluster-backup-volume
persistentVolumeClaim:
claimName: backup-pvc
restartPolicy: NeverIn the above YAML we are running backup every 5 mins but you can change the frequency so it can meet your business RPO. As our Couchbase cluster is deployed within namespace emart so we will deploy the backup cronjob under the same namespace:
$ kubectl apply -f periodic-backup.yaml -n emart
cronjob.batch/couchbase-cluster-backup-create created5.4 Validate Periodic Backup Job
At this point, you can begin to watch the cronjob kicking in at every 5 minutes. And once it becomes active it will execute three initContainers (wget-backup-script, chmod-script, periodic-merge) in sequential order followed by the cointainers (delete-script):
$ kubectl get pods -n emart -w
NAME READY STATUS RESTARTS AGE
backup-node 1/1 Running 0 1d
cbdemo-0000 1/1 Running 0 5d
cbdemo-0001 1/1 Running 0 5d
cbdemo-0002 1/1 Running 0 5d
cbdemo-0003 1/1 Running 0 5d
cbdemo-0004 1/1 Running 0 5d
couchbase-operator-7654d844cb-gn4bw 1/1 Running 0 5d
couchbase-operator-admission-7ff868f54c-5pklx 1/1 Running 0 5d
couchbase-cluster-backup-create-1580357820-tz2hg 0/1 Pending 0 2s
couchbase-cluster-backup-create-1580357820-tz2hg 0/1 Pending 0 2s
couchbase-cluster-backup-create-1580357820-tz2hg 0/1 Init:0/3 0 2s
couchbase-cluster-backup-create-1580357820-tz2hg 0/1 Init:1/3 0 3s
couchbase-cluster-backup-create-1580357820-tz2hg 0/1 Init:2/3 0 4s
couchbase-cluster-backup-create-1580357820-tz2hg 0/1 Init:2/3 0 6s
couchbase-cluster-backup-create-1580357820-tz2hg 0/1 PodInitializing 0 27s
couchbase-cluster-backup-create-1580357820-tz2hg 0/1 Completed 0 30sYou can view the logs of each initContainers after the pod displays status Completed. The initContainers we are interested in is called periodic-merge:
$ kubectl logs couchbase-cluster-backup-create-1580357820-tz2hg -n emart -c periodic-merge
---------------------------------------------------------
BEGIN STEP 1: BACKUP : Thu Jan 30 04:17:12 UTC 2020
Running backup...
Command: cbbackupmgr backup --archive /backups --repo couchbase --cluster couchbase://cbdemo-srv.emart.svc --username Administrator --password password --threads 2
Warning: Progress bar disabled because terminal width is less than 80 characters
Backup successfully completed
Backed up bucket "gamesim-sample" succeeded
Mutations backedup; 586, Mutations failed to backup: 0
Deletions backedup: 0, Deletions failed to backup: 0
Backed up bucket "travel-sample" succeeded
Mutations backedup; 0, Mutations failed to backup: 0
Deletions backedup: 0, Deletions failed to backup: 0
---------------------------------------------------------
BEGIN STEP 2: COMPACTION : Thu Jan 30 04:17:20 UTC 2020
List of backup snapshots ...
2020-01-28T23_01_37.592188562Z
2020-01-28T23_03_34.160387835Z
2020-01-28T23_05_08.103740281Z
2020-01-30T04_17_12.702824188Z
Last backup name is: 2020-01-30T04_17_12.702824188Z
Compacting the backup...
Command: cbbackupmgr compact --archive /backups --repo couchbase --backup 2020-01-30T04_17_12.702824188Z
Compaction succeeded, 0 bytes freed
---------------------------------------------------------
BEGIN STEP 3: Merging old backup : Thu Jan 30 04:17:24 UTC 2020
Size Items Name
604.93MB - + couchbase
192.00MB - + 2020-01-28T23_01_37.592188562Z
192.00MB - + beer-sample
37B 0 analytics.json
414B 0 bucket-config.json
192.00MB 7303 + data
192.00MB 7303 1024 Shards
2B 0 full-text.json
1.94KB 1 gsi.json
784B 1 views.json
192.02MB - + 2020-01-28T23_03_34.160387835Z
192.02MB - + travel-sample
0B 0 analytics.json
416B 0 bucket-config.json
192.00MB 31591 + data
192.00MB 31591 1024 Shards
2B 0 full-text.json
15.57KB 10 gsi.json
2B 0 views.json
64.02MB - + 2020-01-28T23_05_08.103740281Z
64.02MB - + travel-sample
0B 0 analytics.json
416B 0 bucket-config.json
64.00MB 0 + data
64.00MB 0 1024 Shards
2B 0 full-text.json
15.57KB 10 gsi.json
2B 0 views.json
156.89MB - + 2020-01-30T04_17_12.702824188Z
92.88MB - + gamesim-sample
0B 0 analytics.json
417B 0 bucket-config.json
92.88MB 586 + data
92.88MB 586 1024 Shards
2B 0 full-text.json
1.95KB 1 gsi.json
501B 1 views.json
64.02MB - + travel-sample
0B 0 analytics.json
416B 0 bucket-config.json
64.00MB 0 + data
64.00MB 0 1024 Shards
2B 0 full-text.json
15.57KB 10 gsi.json
2B 0 views.json
Start 2020-01-28T23_01_37.592188562Z, END 2020-01-28T23_03_34.160387835Z
Merging old backups...
Command: cbbackupmgr merge --archive /backups --repo couchbase --start 2020-01-28T23_01_37.592188562Z --end 2020-01-28T23_03_34.160387835Z
Merge completed successfully
Size Items Name
412.92MB - + couchbase
192.02MB - + 2020-01-28T23_03_34.160387835Z
192.02MB - + travel-sample
37B 0 analytics.json
416B 0 bucket-config.json
192.00MB 31591 + data
192.00MB 31591 1024 Shards
2B 0 full-text.json
15.57KB 10 gsi.json
2B 0 views.json
64.02MB - + 2020-01-28T23_05_08.103740281Z
64.02MB - + travel-sample
0B 0 analytics.json
416B 0 bucket-config.json
64.00MB 0 + data
64.00MB 0 1024 Shards
2B 0 full-text.json
15.57KB 10 gsi.json
2B 0 views.json
156.89MB - + 2020-01-30T04_17_12.702824188Z
92.88MB - + gamesim-sample
0B 0 analytics.json
417B 0 bucket-config.json
92.88MB 586 + data
92.88MB 586 1024 Shards
2B 0 full-text.json
1.95KB 1 gsi.json
501B 1 views.json
64.02MB - + travel-sample
0B 0 analytics.json
416B 0 bucket-config.json
64.00MB 0 + data
64.00MB 0 1024 Shards
2B 0 full-text.json
15.57KB 10 gsi.json
2B 0 views.jsonNote: As can be seen from the logs above, before the merge step there were four backups available and after merge there are three backup snapshots that are referred to as RESTOREPOINTS in backup-with-periodic-merge.sh script.
This concludes the backup section.
6. Restoring
Much like a backup, we can restore data to a new Couchbase cluster with a Kubernetes Job.
kind: Job
apiVersion: batch/v1
metadata:
name: couchbase-cluster-restore
spec:
template:
spec:
containers:
- name: couchbase-cluster-restore
image: couchbase/server:enterprise-6.0.2
command: ["cbbackupmgr", "restore", "--archive", "/backups", "--repo", "couchbase", "--cluster", "couchbase://cbdemo-srv.emart.svc", "--username", "Administrator", "--password", "password"]
volumeMounts:
- name: "couchbase-cluster-backup-volume"
mountPath: "/backups"
volumes:
- name: couchbase-cluster-backup-volume
persistentVolumeClaim:
claimName: backup-pvc
restartPolicy: NeverIf you would rather like to create a temporary backup-restore pod to see what backups are available or to troubleshoot an issue, you can mount the same persistentVolumeClaim to a new pod. Here is the definition of the pod which can be stored in backup-pod.yaml:
apiVersion: v1
kind: Pod
metadata:
name: backup-node
spec: # specification of the pod's contents
containers:
- name: backup-pod
image: couchbase/server:enterprise-6.5.0
# Just spin & wait forever
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
- name: "couchbase-cluster-backup-volume"
mountPath: "/backups"
volumes:
- name: couchbase-cluster-backup-volume
persistentVolumeClaim:
claimName: backup-pvc
restartPolicy: NeverRun kubectl to bring up the pod temporarily:
$ kubectl apply -f br/backup-pod.yaml -n emart
$ kubectl get pods -n emart
NAME READY STATUS RESTARTS AGE
backup-node 1/1 Running 0 3d1h
cbdemo-0000 1/1 Running 0 7d1h
cbdemo-0001 1/1 Running 0 7d1h
cbdemo-0002 1/1 Running 0 7d1h
cbdemo-0003 1/1 Running 0 7d1h
cbdemo-0004 1/1 Running 0 7d1h
couchbase-operator-7654d844cb-gn4bw 1/1 Running 0 7d2h
couchbase-operator-admission-7ff868f54c-5pklx 1/1 Running 0 7d2hOnce backup-node is Running, we can login to that pod:
$ kubectl exec -it backup-node -n emart -- /bin/bash
root@backup-node:/And execute cbbackupmgr list command to view existing backups:
# cbbackupmgr list --repo couchbase --archive /backups
Size Items Name
256.04MB - + couchbase
0B - + 2020-01-30T04_17_12.702824188Z
0B - + gamesim-sample
0B 0 analytics.json
0B 0 + data
0B 0 Error: no data shards were found
0B 0 full-text.json
0B 0 gsi.json
0B 0 views.json
128.02MB - + 2020-01-30T04_18_13.021340423Z
....And you can also run cbbackupmgr restore command manually:
# cbbackupmgr restore --archive /backups --repo couchbase --cluster couchbase://cbdemo-srv.emart.svc --username Administrator --password passwordOnce you are done restoring just delete the pod:
$ kubectl delete -f backup-pod.yaml -n emart7. Conclusion
We walked through step-by-step on how one can configure a backup cronjob, which automates the process of taking the periodic backup at a predefined interval. We used a backup-with-periodic-merge.sh script, that executes a) backup, b) compaction and c) merge within a single script. This script was then used in the periodic-backup.yaml file, which automated the process of taking backup within the Kubernetes environment. We hope you would use the best practices described in this blog and plan on taking regular backups as well as validate those backups using restore command regularly.
