![\"KOPS](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2018\/07\/Screen-Shot-2018-07-13-at-2.57.48-PM-1024x526.png\")
KOPS Architecture<\/p><\/div>\n
In the spirit of doing full disclosure, It is easier to use tools provided by cloud vendors. Opensource tools like kops work equally well and are very efficient.<\/p>\n
The Couchbase Autonomous Operator extends the Kubernetes API by creating a Custom Resource Definition(CRD) and registering a Couchbase specific controller (the Autonomous Operator) to manage Couchbase clusters.<\/p>\n
For this blog we will be using AWS as IaaS and will demonstrate the kops usage, capabilities and Couchbase Autonomous Operator running Couchbase Cluster at scale.<\/p>\n
What can KOPS do?<\/h2>\n
\u201ckops\u00a0helps you create, destroy, upgrade and maintain production-grade, highly available, Kubernetes clusters from the command line. AWS (Amazon Web Services) is currently officially supported, with GCE in beta support, and VMware vSphere in alpha, and other platforms planned.\u201d [1]<\/p>\n
Let\u2019s get down to nitty gritty and see how it works?<\/p>\n
- \n
- Install kops<\/li>\n
- Create k8s cluster on AWS<\/li>\n
- Configure k8s Dashboard<\/li>\n
- Verify and validate k8s cluster<\/li>\n
- Deploy Couchbase Autonomous Operator in K8s<\/li>\n
- Deploy couchbase cluster<\/li>\n
- Service resiliency<\/li>\n
- Next steps<\/li>\n<\/ul>\n
Install KOPS<\/strong><\/h3>\n
OSX From Homebrew<\/strong><\/h4>\n
$ brew update\r\n$ brew install kops\r\n<\/pre>\n
Linux<\/h4>\n
$ curl -LO https:\/\/github.com\/kubernetes\/kops\/releases\/download\/$(curl -s https:\/\/api.github.com\/repos\/kubernetes\/kops\/releases\/latest | grep tag_name | cut -d '\"' -f 4)\/kops-linux-amd64chmod +x kops-linux-amd64sudo mv kops-linux-amd64 \/usr\/local\/bin\/kops<\/pre>\n
<\/p>\n
Create k8s cluster on AWS<\/strong><\/h3>\n
Before we can use kops to create k8s cluster, we need to create IAM policies for kops and so that kops can provision the required resources needed to create k8s cluster. This is one time operation, needs to done only once<\/p>\n
Prerequisites for kops<\/p>\n
Install AWS CLI<\/h4>\n
For MacOS<\/p>\n
$ pip3 install<\/span> awscli --upgrade --user<\/span><\/code><\/strong><\/p>\nFor other platforms, click here<\/a><\/p>\n
NOTE:\u00a0 one-time op only<\/em><\/p>\n
Enable AWS environment by running \u2018aws configure\u2019 and run the commands below<\/h4>\n
$ aws configure # enter appropriate credentials with user with perms to create user\/groups\r\n$ aws iam create-group --group-name kops\r\n$ aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy\/AmazonEC2FullAccess --group-name kops\r\n$ aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy\/AmazonRoute53FullAccess --group-name kops\r\n$ aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy\/AmazonS3FullAccess --group-name kops\r\n$ aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy\/IAMFullAccess --group-name kops\r\n$ aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy\/AmazonVPCFullAccess --group-name kops\r\n$ aws iam create-user --user-name kops\r\n$ aws iam add-user-to-group --user-name kops --group-name kops\r\n$ aws iam create-access-key --user-name kops<\/pre>\n
Create S3 bucket to store kops config\/data<\/h4>\n
In my case, its named as\u00a0rd-k8s-aws\u00a0<\/strong><\/em>and for region us-west-1<\/strong><\/p>\n
$ aws s3api create-bucket --bucket rd-k8s-aws --region us-west-1 --create-bucket-configuration LocationConstraint=us-west-1\r\n\r\n$ aws s3api put-bucket-versioning --bucket rd-k8s-aws --versioning-configuration Status=Enabled<\/pre>\n
Define env var\u2019s for kops state store and cluster name<\/strong><\/h4>\n
Pick any cluster name for your cluster, I have chosen\u00a0rdc.k8s.local<\/strong><\/em><\/p>\n
$ export KOPS_STATE_STORE=s3:\/\/rd-k8s-aws\r\n\r\n$ export KOPS_CLUSTER_NAME=rdc.k8s.local<\/pre>\n
Create k8s cluster with kops<\/h4>\n
Choose the AWS region, and instance size and node count for the k8s cluster.<\/p>\n
$ kops create cluster --node-count=7 --node-size=m4.4xlarge --zones=us-west-2a --name ${KOPS_CLUSTER_NAME} --state=${KOPS_STATE_STORE} --yes<\/pre>\nNOTE:<\/strong> takes around 5-8mins, good time to get coffee!<\/p>\n
Sample output looks something like this<\/h6>\n
$ kops validate cluster\r\nValidating cluster rdc.k8s.local\r\n\r\nINSTANCE GROUPS\r\nNAME ROLE MACHINETYPE MIN MAX SUBNETS\r\nmaster-us-west-1a Master m3.medium 1 1 us-west-1a\r\nnodes Node m4.4xlarge 7 7 us-west-1a\r\n\r\nNODE STATUS\r\nNAME ROLE READY\r\nip-172-20-42-242.us-west-1.compute.internal node True\r\nip-172-20-43-56.us-west-1.compute.internal node True\r\nip-172-20-44-17.us-west-1.compute.internal node True\r\nip-172-20-44-80.us-west-1.compute.internal node True\r\nip-172-20-48-254.us-west-1.compute.internal node True\r\nip-172-20-56-254.us-west-1.compute.internal node True\r\nip-172-20-60-79.us-west-1.compute.internal master True\r\nip-172-20-63-146.us-west-1.compute.internal node True\r\n\r\nYour cluster rdc.k8s.local is ready\r\n<\/pre>\n
Get the nodes and validate the k8s cluster<\/h4>\n
$ kubectl get nodes --show-labels\r\n\r\n$ kubectl cluster-info<\/pre>\n
At this point kubectl context or k8s context should point to AWS k8s cluster<\/p>\n
Configure k8s Dashboard<\/h3>\n
$ kubectl apply -f https:\/\/raw.githubusercontent.com\/kubernetes\/dashboard\/master\/src\/deploy\/recommended\/kubernetes-dashboard.yaml<\/pre>\n
$ kubectl create serviceaccount dashboard -n default\r\n$ kubectl create clusterrolebinding dashboard-admin -n default --clusterrole=cluster-admin --serviceaccount=default:dashboard<\/pre>\n
On a different tab\/terminal run the command below to proxy k8s api server onto local laptop<\/em><\/strong><\/p>\n
$ kubectl proxy<\/pre>\n
Get the token to access the k8s dashboard<\/p>\n
$ kubectl get secret $(kubectl get serviceaccount dashboard -o jsonpath=\"{.secrets[0].name}\") -o jsonpath=\"{.data.token}\" | base64 --decode<\/pre>\nChoose option to use token and while grabbing token from above, skip %<\/strong> in the end.<\/p>\n
Login to k8s dashboard at\u00a0https:\/\/127.0.0.1:8001\/ui [2]<\/p>\n
Verify and validate k8s cluster<\/strong><\/h3>\n
Validate the k8s cluster using command<\/p>\n
$ kops validate cluster<\/pre>\n
# check that number of nodes mentioned in the cluster create matches here<\/p>\n
$ kubectl get nodes<\/pre>\n
Deploy Couchbase Autonomous Operator in K8s <\/strong><\/h3>\n
Now that we have deployed the k8s cluster, now it\u2019s time to deploy the couchbase operator<\/p>\n
First we deploy the operator, we need to enable RBAC for the operator, we need to create ClusterRole for the Couchbase Operator<\/p>\n
$ kubectl create -f https:\/\/raw.githubusercontent.com\/ramdhakne\/blogs\/master\/kops\/cbyaml\/cluster-role.yaml<\/pre>\n
Note<\/em>: This role only needs to be created once.<\/p>\n
After the ClusterRole is created, you need to create a ServiceAccount in the namespace where you are installing the Couchbase Operator, and then assign the ClusterRole to that ServiceAccount using a\u00a0ClusterRoleBinding<\/em>. In this guide we will use the\u00a0
default<\/code>\u00a0namespace to create the ServiceAccount.<\/p>\n$ kubectl create serviceaccount couchbase-operator --namespace default\r\n\r\n$ kubectl create clusterrolebinding couchbase-operator --clusterrole couchbase-operator --serviceaccount default:couchbase-operator<\/pre>\n
Deploy the couchbase operator<\/h3>\n
$ kubectl create -f https:\/\/raw.githubusercontent.com\/ramdhakne\/blogs\/master\/kops\/cbyaml\/operator.yaml<\/pre>\n
Running this command downloads the Couchbase Operator Docker image that is specified in the\u00a0
operator.yaml<\/code>\u00a0file and creates a\u00a0deployment<\/em>\u00a0which manages a single instance of the Couchbase Operator. The Couchbase Operator uses a deployment so that it can restart if the pod it\u2019s running in dies.<\/p>\n$ kubectl get deployments -l app=couchbase-operator<\/pre>\n
\nYou should continue to poll the status of the Operator until your output looks similar to the following output:<\/p>\n<\/div>\n
\n\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\r\ncouchbase-operator 1 1 1 1 47s<\/pre>\n<\/div>\n<\/div>\n
When the couchbase operator is up and running it will look something like this<\/p>\n
$ kubectl get pods\r\nNAME \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0READY STATUS RESTARTS AGE\r\ncouchbase-operator-7676b7cf47-99b5v \u00a0\u00a0 1\/1 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Running \u00a00 1m<\/pre>\n
We need to deploy secret for authenticating couchbase server GUI<\/p>\n
$ kubectl create -f https:\/\/raw.githubusercontent.com\/ramdhakne\/blogs\/master\/kops\/cbyaml\/secret.yaml<\/pre>\n
Deploy the 10 node Couchbase cluster now<\/p>\n
$ kubectl create -f\u00a0https:\/\/raw.githubusercontent.com\/ramdhakne\/blogs\/master\/kops\/cbyaml\/couchbase-cluster-multi.yaml<\/pre>\n
Get the listing of the couchbase pods<\/p>\n
$ kubectl get pods\r\nNAME READY STATUS RESTARTS AGE\r\ncb-example-aws-0000 1\/1 Running 0 7m\r\ncb-example-aws-0001 1\/1 Running 0 6m\r\ncb-example-aws-0002 1\/1 Running 0 5m\r\ncb-example-aws-0003 1\/1 Running 0 4m\r\ncb-example-aws-0004 1\/1 Running 0 3m\r\ncb-example-aws-0005 1\/1 Running 0 2m\r\ncb-example-aws-0006 1\/1 Running 0 2m\r\ncb-example-aws-0007 1\/1 Running 0 1m\r\ncb-example-aws-0008 1\/1 Running 0 1m\r\ncb-example-aws-0009 1\/1 Running 0 50s\r\ncouchbase-operator-7676b7cf47-99b5v 1\/1 Running 0 17m<\/pre>\n
To access couchbase GUI, we will access at port 8091 and for that we can perform port forwarding like below<\/p>\n
$ kubectl port-forward cb-example-aws-0000 8091:8091<\/pre>\n
Couchbase server GUI can accessed at https:\/\/localhost:8091<\/p>\n
Service resiliency<\/h3>\n
In order to show service resiliency, we can kill a pod, simulating losing a couchbase node.<\/p>\n
\u00a0$ kubectl delete pod cb-example-aws-0001<\/pre>\n
Biggest feature kubernetes gives us is service resiliency, if we lost a pod, then kubernetes checks for the cluster definition, and it understands that it need to have 3 pods, and it will spin up a new one, add to the cluster, and then perform rebalance on the cluster. Thus eliminating crucial administration task of monitoring the cluster for node failures, deciding what to do when it happens and then performing rebalance operation. All of this happens behind the cover, and applications will perform without any interruptions.<\/p>\n
If we list pods again, then should see something like this<\/p>\n
$ kubectl get pods --watch\r\n...\r\n...\r\ncb-example-aws-0010 0\/1 Pending 0 0s\r\ncb-example-aws-0010 0\/1 Pending 0 0s\r\ncb-example-aws-0010 0\/1 ContainerCreating 0 0s\r\ncb-example-aws-0010 1\/1 Running 0 1s<\/pre>\n
Installing cbc-pillowfight to generate load KV load, gives us an idea what are the limits we can push on this cluster<\/p>\n
From single app server, running pillowfight with following params<\/p>\n
# cbc-pillowfight -U couchbase:\/\/localhost\/kopsClusterBucket -u Administrator -P password -J -t 4 -I 10000 -p `hostname`<\/pre>\n
![\"KV](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2018\/07\/Screen-Shot-2018-07-11-at-3.34.28-PM-1024x517.png\")
KV performance with pillowfight<\/p><\/div>\n
From single app server we are able to get ~34K Ops\/Sec<\/p>\n
Cranking up a notch, using other app server to run pillowfight, we can get more performance in the order of ~51k Ops\/Sec<\/p>\n
![\"KV](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2018\/07\/Screen-Shot-2018-07-11-at-3.41.15-PM-1024x506.png\")
KV performance with pillowfight<\/p><\/div>\n
Conclusion<\/p>\n
What we learned here is using kops it becomes quite easy to deploy Couchbase clusters at scale and Autonomous operator provides benefits like service resiliency, auto-rebalance, automatic bucket creation, easy integration with CI\/CD pipeline. yaml files can be standardized for Dev\/QA\/UAT and prod.<\/p>\n
References:<\/p>\n