Put in simple terms, kops is kubectl for kubernetes (k8s) clusters. The power of kubernetes is that it removes vendor lock-in, provides portability for your apps, databases incase decision gets made to different cloud vendor or if the cloud workflows need to go on-prem. Whatever the scenario may be, kubernetes give that freedom.

KOPS Architecture

KOPS Architecture

In the spirit of doing full disclosure, It is easier to use tools provided by cloud vendors. Opensource tools like kops work equally well and are very efficient.

The Couchbase Autonomous Operator extends the Kubernetes API by creating a Custom Resource Definition(CRD) and registering a Couchbase specific controller (the Autonomous Operator) to manage Couchbase clusters.

For this blog we will be using AWS as IaaS and will demonstrate the kops usage, capabilities and Couchbase Autonomous Operator running Couchbase Cluster at scale.

What can KOPS do?

“kops helps you create, destroy, upgrade and maintain production-grade, highly available, Kubernetes clusters from the command line. AWS (Amazon Web Services) is currently officially supported, with GCE in beta support, and VMware vSphere in alpha, and other platforms planned.” [1]

Let’s get down to nitty gritty and see how it works?

  • Install kops
  • Create k8s cluster on AWS
  • Configure k8s Dashboard
  • Verify and validate k8s cluster
  • Deploy Couchbase Autonomous Operator in K8s
  • Deploy couchbase cluster
  • Service resiliency
  • Next steps

Install KOPS

OSX From Homebrew

Linux

 

Create k8s cluster on AWS

Before we can use kops to create k8s cluster, we need to create IAM policies for kops and so that kops can provision the required resources needed to create k8s cluster. This is one time operation, needs to done only once

Prerequisites for kops

Install AWS CLI

For MacOS

$ pip3 install awscli --upgrade --user

For other platforms, click here

NOTE:  one-time op only

Enable AWS environment by running ‘aws configure’ and run the commands below

Create S3 bucket to store kops config/data

In my case, its named as rd-k8s-aws and for region us-west-1

Define env var’s for kops state store and cluster name

Pick any cluster name for your cluster, I have chosen rdc.k8s.local

Create k8s cluster with kops

Choose the AWS region, and instance size and node count for the k8s cluster.

NOTE: takes around 5-8mins, good time to get coffee!

Sample output looks something like this

Get the nodes and validate the k8s cluster

At this point kubectl context or k8s context should point to AWS k8s cluster

Configure k8s Dashboard

On a different tab/terminal run the command below to proxy k8s api server onto local laptop

Get the token to access the k8s dashboard

Choose option to use token and while grabbing token from above, skip % in the end.

Login to k8s dashboard at http://127.0.0.1:8001/ui [2]

Verify and validate k8s cluster

Validate the k8s cluster using command

# check that number of nodes mentioned in the cluster create matches here

Deploy Couchbase Autonomous Operator in K8s

Now that we have deployed the k8s cluster, now it’s time to deploy the couchbase operator

First we deploy the operator, we need to enable RBAC for the operator, we need to create ClusterRole for the Couchbase Operator

Note: This role only needs to be created once.

After the ClusterRole is created, you need to create a ServiceAccount in the namespace where you are installing the Couchbase Operator, and then assign the ClusterRole to that ServiceAccount using a ClusterRoleBinding. In this guide we will use the default namespace to create the ServiceAccount.

Deploy the couchbase operator

Running this command downloads the Couchbase Operator Docker image that is specified in the operator.yaml file and creates a deployment which manages a single instance of the Couchbase Operator. The Couchbase Operator uses a deployment so that it can restart if the pod it’s running in dies.

You should continue to poll the status of the Operator until your output looks similar to the following output:

When the couchbase operator is up and running it will look something like this

We need to deploy secret for authenticating couchbase server GUI

Deploy the 10 node Couchbase cluster now

Get the listing of the couchbase pods

To access couchbase GUI, we will access at port 8091 and for that we can perform port forwarding like below

Couchbase server GUI can accessed at http://localhost:8091

Service resiliency

In order to show service resiliency, we can kill a pod, simulating losing a couchbase node.

Biggest feature kubernetes gives us is service resiliency, if we lost a pod, then kubernetes checks for the cluster definition, and it understands that it need to have 3 pods, and it will spin up a new one, add to the cluster, and then perform rebalance on the cluster. Thus eliminating crucial administration task of monitoring the cluster for node failures, deciding what to do when it happens and then performing rebalance operation. All of this happens behind the cover, and applications will perform without any interruptions.

If we list pods again, then should see something like this

Installing cbc-pillowfight to generate load KV load, gives us an idea what are the limits we can push on this cluster

From single app server, running pillowfight with following params

KV performance with pillowfight

KV performance with pillowfight

From single app server we are able to get ~34K Ops/Sec

Cranking up a notch, using other app server to run pillowfight, we can get more performance in the order of ~51k Ops/Sec

KV performance with pillowfight

KV performance with pillowfight

Conclusion

What we learned here is using kops it becomes quite easy to deploy Couchbase clusters at scale and Autonomous operator provides benefits like service resiliency, auto-rebalance, automatic bucket creation, easy integration with CI/CD pipeline. yaml files can be standardized for Dev/QA/UAT and prod.

References:

[1] https://github.com/kubernetes/kops

[2] k8s dashboard troubleshooting

 

Author

Posted by Ram Dhakne

Ram Dhakne is Solutions Consultant - US West at Couchbase. He currently helps Enterprise customers with their digital innovations journey and helping them adopt NoSQL technologies. His current interests are running persistent applications like Couchbase NoSQL server on Kubernetes clusters running on AKS, GKE, ACS and OpenShift, securing end-to-end on kubernetes. In his past life has worked on IaaS platforms (AWS, GCP, Azure & Private Clouds), Enterprise Backup Target Products & Backup Applications.

Leave a reply