Support for Public Kubernetes Services on GKE, AKS and EKS. Kubernetes running on public cloud was already working from day 1, but with Autonomous Operator 1.2, we are supporting it in an official capacity. For this blog’s perspective, we will be using GKE to setup kubernetes cluster on GKE with version 1.12, then deploying Autonomous Operator and then eventually deploying Couchbase Cluster with Server Groups, with persistent volumes, and with x509 TLS certificates.<\/p>\n<\/li>\n<\/ul>\n
Overall steps that we will be doing in this blog are as follows:<\/p>\n
- \n
- \n
Initialize gcloud utils<\/h5>\n<\/li>\n
- \n
Deploy kubernetes cluster (v1.12+) with 2 nodes in each availability zones<\/h5>\n<\/li>\n
- \n
Deploy Autonomous Operator 1.2<\/h5>\n<\/li>\n
- \n
Deploy Couchbase Cluster<\/h5>\n<\/li>\n
- \n
Perform Server Group Autofailover<\/h5>\n<\/li>\n<\/ol>\n
Pre-requisites<\/p>\n
- \n
- kubectl (gcloud components install kubect<\/strong>l<\/em>)<\/li>\n
- GCP account with right credentials<\/li>\n<\/ul>\n
Initialize gcloud utils<\/h5>\n
Download gcloud sdk for the OS version of your choice from this URL.<\/a><\/p>\n
One would need google cloud credentials to initialize the gcloud cli<\/p>\n
cd google-cloud-sdk\r\n.\/install.sh\r\n.\/bin\/gcloud init<\/pre>\n
<\/h5>\n
Deploy kubernetes cluster (v1.12) with 2 nodes in each availability zones<\/h5>\n
Deploying kubernetes cluster on GKE is fairly straightforward job. To deploy resilient kubernetes clusters, its good idea to deploy nodes in all available zones within a given region. Doing it in such way we can make use of Server Groups or Rack Zone or Availability Zone(AZ) awareness feature within Couchbase server, means if we lose entire AZ, couchbase can failover entire AZ and Application will be active, as it still has the working dataset.<\/p>\n
gcloud container clusters create rd-k8s-gke --region us-east1 --machine-type n1-standard-16 --num-nodes 2<\/pre>\n
Details about above command\r\nK8s cluster name :\u00a0 rd-k8s-gke\r\nmachine-type: n1-standard-16 (16 vCPUs and 60GB RAM)\r\nnum-nodes\/AZ : 2<\/pre>\n
More Machines types can be here<\/a><\/p>\n
At this point, k8s cluster with required number of nodes should be up and running<\/p>\n
$ gcloud container clusters list\r\nNAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES<\/strong> STATUS\r\nrd-k8s-gke us-east1 1.12.6-gke.10 35.229.24.36 n1-standard-16 1.12.6-gke.10 6<\/strong> RUNNING<\/pre>\n
Details of the k8s cluster can be found like below<\/p>\n
$ kubectl cluster-info\r\nKubernetes master is running at https:\/\/55.229.24.36\r\nGLBCDefaultBackend is running at https:\/\/55.229.24.36\/api\/v1\/namespaces\/kube-system\/services\/default-http-backend:http\/proxy\r\nHeapster is running at https:\/\/55.229.24.36\/api\/v1\/namespaces\/kube-system\/services\/heapster\/proxy\r\nKubeDNS is running at https:\/\/55.229.24.36\/api\/v1\/namespaces\/kube-system\/services\/kube-dns:dns\/proxy\r\nMetrics-server is running at https:\/\/55.229.24.36\/api\/v1\/namespaces\/kube-system\/services\/https:metrics-server:\/proxy<\/pre>\n
<\/h5>\n
Deploy Autonomous Operator 1.2<\/h5>\n
GKE supports RBAC in order to limit permissions. Since the Couchbase Operator creates resources in our GKE cluster, we will need to grant it the permission to do so.<\/p>\n
$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)<\/pre>\n
Download<\/a> the appropriate package for deploying kubernetes in your environment. Untar the package and deploy the admission controller.<\/p>\n
$ kubectl create -f admission.yaml<\/pre>\n
Check the status of admission controller<\/p>\n
$ kubectl get deployments\r\nNAME\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 DESIRED\u00a0 CURRENT UP-TO-DATE \u00a0AVAILABLE AGE\r\ncouchbase-operator-admission \u00a0\u00a01\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 1\u00a0 \u00a0 \u00a01\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 1 \u00a0 \u00a0 \u00a07s<\/pre>\n
In order to visualize how admission controller works in sync with operator and couchbase cluster, it can be illustrated better with the following diagram<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2019\/05\/image001.png\")
<\/p>\nNext steps are to create crd, operator role and operator 1.2<\/p>\n
$ kubectl create -f crd.yaml\r\n$ kubectl create -f operator-role.yaml\r\n$ kubectl create -f operator-deployment.yaml<\/pre>\n
Once the operator is deployed, it gets ready and available within secs<\/p>\n
$ kubectl get deployments \r\nNAME \u00a0\u00a0 DESIRED CURRENT UP-TO-DATE \u00a0AVAILABLE AGE\r\ncouchbase-operator-admission \u00a0\u00a01 1 1 \u00a0 1 11m\r\ncouchbase-operator \u00a0 \u00a01 1 1 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1 25s<\/pre>\n
Deploy Couchbase Cluster<\/h5>\n
Couchbase cluster will be deployed with following features<\/p>\n
- \n
- \n
TLS certificates<\/h6>\n<\/li>\n
- \n
Server Groups (each server group in one AZ)<\/h6>\n<\/li>\n
- \n
Persistent Volumes (which are AZ aware)<\/h6>\n<\/li>\n
- \n
Server Group auto-failover<\/h6>\n<\/li>\n<\/ul>\n
TLS certificates<\/h6>\n
Its fairly easy to generate tls certificates, details steps are be found here<\/a><\/p>\n
Once deployed, tls secrets can be found with kubectl secret command like below<\/p>\n
$ kubectl get secrets\r\n NAME TYPE DATA AGE\r\ncouchbase-operator-tls Opaque 1 1d\r\ncouchbase-server-tls Opaque 2 1d<\/pre>\n
Server Groups<\/h6>\n
Setting up server groups is also straightforward, which will be discussed in the following sections when we deploy the couchbase cluster yaml file.<\/p>\n
Persistent Volumes<\/h6>\n
Persistent Volumes provide way for a reliable way to run stateful applications. Creating them on public cloud is one click operation.<\/p>\n
First we can check what storageclass is available for use<\/p>\n
$ kubectl get storageclass\r\nNAME PROVISIONER AGE\r\nstandard (default) kubernetes.io\/gce-pd 1d<\/pre>\n
All the worker nodes available in the k8s cluster should failure domain labels like below<\/p>\n
$ kubectl get nodes -o yaml | grep zone\r\nfailure-domain.beta.kubernetes.io\/zone: us-east1-b\r\nfailure-domain.beta.kubernetes.io\/zone: us-east1-b\r\nfailure-domain.beta.kubernetes.io\/zone: us-east1-d\r\nfailure-domain.beta.kubernetes.io\/zone: us-east1-d\r\nfailure-domain.beta.kubernetes.io\/zone: us-east1-c\r\nfailure-domain.beta.kubernetes.io\/zone: us-east1-c<\/pre>\n
NOTE: I don’t have to add any failure domain labels, GKE added automatically.<\/p>\n
Create PV for each AZ<\/p>\n
$ kubectl apply -f svrgp-pv.yaml<\/pre>\n
yaml file svrgp-pv.yaml, can be found here<\/a>.<\/p>\n
Create secret for accessing couchbase UI<\/p>\n
$ kubectl apply -f secret.yaml<\/pre>\n
Finally deploy couchbase cluster with TLS support, along with Server Groups(which are Az aware) and on persistent volumes (which are also AZ aware).<\/p>\n
$ kubectl apply -f couchbase-persistent-tls-svrgps.yaml<\/pre>\n
yaml file couchbase-persistent-tls-svrgps.yaml, can be found here<\/a><\/p>\n
Give a few mins, and couchbase cluster will come up, and it should look like this<\/p>\n
$ kubectl get pods\r\nNAME READY STATUS RESTARTS AGE\r\ncb-gke-demo-0000 1\/1 Running 0 1d\r\ncb-gke-demo-0001 1\/1 Running 0 1d\r\ncb-gke-demo-0002 1\/1 Running 0 1d\r\ncb-gke-demo-0003 1\/1 Running 0 1d\r\ncb-gke-demo-0004 1\/1 Running 0 1d\r\ncb-gke-demo-0005 1\/1 Running 0 1d\r\ncb-gke-demo-0006 1\/1 Running 0 1d\r\ncb-gke-demo-0007 1\/1 Running 0 1d\r\ncouchbase-operator-6cbc476d4d-mjhx5 1\/1 Running 0 1d\r\ncouchbase-operator-admission-6f97998f8c-cp2mp 1\/1 Running 0 1d<\/pre>\n
Quick check on persistent volumes claims can be done like below<\/p>\n
$ kubectl get pvc<\/pre>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2019\/05\/Screen-Shot-2019-05-02-at-2.01.12-PM.png\")
<\/p>\nIn order to access the Couchbase Cluster UI, either we can port-foward port 8091 of any pod or service itself, on local laptop, or local machine, or it can be exposed via lb.<\/p>\n
$ kubectl port-forward service\/cb-gke-demo-ui 8091:8091<\/pre>\n
port-forward any pod like below<\/p>\n
$ kubectl port-forward cb-gke-demo-0002 8091:8091<\/pre>\n
At this point couchbase server is up and running and we have way to access it.<\/p>\n
Perform Server Group Autofailover<\/h5>\n
Server Group auto-failover<\/h6>\n
When a couchbase cluster node fails, then it can auto-failover and without any user intervention ALL the working set is available, no user intervention is needed and Application won’t see downtime.<\/p>\n
If Couchbase cluster is setup to be Server Group(SG) or AZ or Rack Zone(RZ) aware, then even if we lose entire SG then entire server groups fails over and working set is available, no user intervention is needed and Application won’t see downtime.<\/p>\n
In order to have Disaster Recovery, XDCR can be used to replicate Couchbase data to other Couchbase Cluster. This helps in the event if entire source Data Center or Region is lost, Applications can cut over to Remote site and application won’t see downtime.<\/p>\n
Lets take down the Server Group. Before that, lets see how the cluster looks like<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2019\/05\/Screen-Shot-2019-05-03-at-11.48.10-AM.png\")
<\/p>\nDelete all pods in group us-east1-b, once the pods are deleted, Couchbase cluster will see that nodes are
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2019\/05\/Screen-Shot-2019-05-03-at-11.53.53-AM.png\")
<\/p>\nOperator is constantly watching the cluster definition and it will see that server group is lost, and it spins the 3 pods, re-establishes the claims on the PVs and performs delta-node recovery, and then eventually performs rebalance operation and cluster is healthy again. All with no user-intervention whatsoever.<\/p>\n
After sometime, cluster is back and up and running.<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2019\/05\/Screen-Shot-2019-05-03-at-1.29.58-PM.png\")
<\/p>\nFrom the operator logs,<\/p>\n
$ kubectl logs -f couchbase-operator-6cbc476d4d-mjhx5<\/pre>\n
we can see that cluster is automatically rebalanced.<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2019\/05\/Screen-Shot-2019-05-03-at-2.10.00-PM.png\")
<\/p>\nEpilogue<\/h6>\n
Sustained differentiation is key to our technology. We have added quite a number of new and supportability features. With all these enterprise grade supportability features, they enable end user to find the issues faster and help operationalize the Couchbase Operator in their environments in a efficient faster and way. We are very excited about the release, feel free to give a try!<\/p>\n
<\/p>\n
References:<\/h6>\n
https:\/\/docs.couchbase.com\/operator\/1.2\/whats-new.html<\/a><\/p>\n
https:\/\/www.couchbase.com\/downloads<\/a><\/p>\n
https:\/\/docs.couchbase.com\/server\/6.0\/manage\/manage-groups\/manage-groups.html<\/a><\/p>\n
K8s Autonomous Operator Book from @AnilKumar<\/a><\/p>\n
https:\/\/info.couchbase.com\/rs\/302-GJY-034\/images\/Kubernetes_ebook.pdf<\/a><\/p>\n
https:\/\/docs.couchbase.com\/operator\/1.2\/tls.html<\/a><\/p>\n
- GCP account with right credentials<\/li>\n<\/ul>\n
- kubectl (gcloud components install kubect<\/strong>l<\/em>)<\/li>\n