Operador autônomo 2.0 do Couchbase com o Prometheus - Parte 1

We recently announced the latest preview of the Couchbase Autonomous Operator (CAO) 2.0 beta. This release is a significant update to the Couchbase Autonomous Operator. Couchbase Autonomous Operator 2.0 introduces several new enterprise-grade features with fully autonomous capabilities – security, monitoring, high-availability, and manageability. In this blog, we will examine in-depth how one of those works.

Coleção de métricas da Prometheus

O operador mais recente oferece integração nativa com o Couchbase Prometheus Exporter para coletar e expor as métricas do Couchbase Server. Essas métricas exportadas podem ser extraídas pelo Prometheus e, em seguida, visualizadas em ferramentas como o Grafana.

We will describe the steps to deploy the cluster with the Couchbase Prometheus Exporter and look at some of the metrics through Grafana. This will be a simple single cluster test deployment and will not detail all the other necessary steps for a production level deployment.

We will be closely following the Couchbase Autonomous Operator 2.0 Beta tutorial on installing on Amazon EKS.

Pré-requisitos

I assume you already have an Amazon Virtual Private Cloud (VPC) to use. Follow the documentation on Getting Started with Amazon EKS and install the following:

• kubectl
• aws-iam-authenticator
• eksctl
• Amazon EC2 key pairs
• Operador autônomo do Couchbase 2.0 Beta

Arquitetura de implantação

A quick overview of the architecture of our deployment.

Referring the above diagram:

1: Create the Kubernetes cluster on Amazon EKS. The cluster manages Kubernetes Resources and Services.
2: Add Couchbase Resources by installing the Couchbase Custom Resource Definitions.
3: Install the Couchbase Autonomous Operator. This creates 2 Pods, the Operator and the Admission Controller in the Default namespace.
4: Deploy a 3 node Couchbase Cluster in the Default namespace. This creates 3 pods, each pod has a Couchbase 6.5.0 container and a Couchbase Metrics Exporter container.
5: Create a ServiceMonitor which tells Prometheus to monitor a Service resource that defines the endpoints Prometheus scrapes.
6: Create a  Service will define the port that we described in our ServiceMonitor at earlier in the Default namespace.
7:  Add Prometheus Resources by installing the Prometheus Custom Resource Definitions.
8: Create the Prometheus/Grafana Pods in the Monitoring namespace. 

Create the Cluster and Configure kubectl

Configure kubectl

This command is vital as it sets the relevant Amazon Resource Name (ARN) variables in ~/.kube/config. Optionally, you can add --region regionName to specify a cluster in a region that is different than the default. (Your default region should have been specified when you first setup the AWS CLI through aws configurecommand.)

Install the Custom Resource Definitions (CRD)

Note: I downloaded the Operator for MacOS, renamed the package from  couchbase-autonomous-operator-kubernetes_2.0.0-macos-x86_64 para cao-2 and cd’d into this directory. 

The first step in installing the Operator is to install the custom resource definitions (CRD) that describe the Couchbase resource types.

Install the Autonomous Operator 2.0

Check the status of the Operator

The Operator is ready to deploy CouchbaseCluster resources when both the Dynamic Admission Controller (couchbase-operator-admission) and Operator (couchbase-operator) deployments are fully ready and available.

Prepare the Couchbase Cluster Configuration

I’ll deploy a 3 node Couchbase Server 6.5.0 Cluster with Prometheus Couchbase Exporter. For this, I created my-cluster.yaml file in the current directory. This is just my sample. Here is the file:

Observações:

• I have used only a minimum set of configuration parameters. Please refer to the Couchbase Cluster Resource documentation for a complete list.
• Included the secrets section in the same file to keep things simple. 
• Used only the Data, Query, Index and Search services. 
• Managing my own buckets instead of leaving it to the Operator.
• Make a note of the cluster label cb-exemplo as this will be used by Prometheus to discover the service later.

Dica: Make sure that buckets.managed is set to false. Otherwise, if you create a bucket manually once the cluster is up and running, Kubernetes will automatically drop it.

Implantar o cluster do Couchbase

Both the secret as well as the cluster are created. This does not mean they are up and running yet, for that you will have to verify as described in the next step.

Verify the deployment

Make sure that all the pods are Pronto e Em execução. In case there is some problem, you can get the logs from the Operator.

Optional: Get the Logs

If you encounter any problems in the previous step, you can check the logs as shown below.

Here, the Couchbase cluster deployed without any errors.

Optional: Examine a Couchbase pod.

Let’s describe a Couchbase pod to check what its running.

From the output above, we see that each Couchbase pod is running 2 containers. The first is running Couchbase Server 6.5.0 and and the other is running the Couchbase Prometheus Exporter which is using Port 9091.

Access the Couchbase Administration UI

In an actual production environment, you would typically deploy using DNS and a LoadBalancer acting as the proxy and would access Couchbase UI securely, with SSL using DNS SRV records. Since we are in a test environment, we will access the Couchbase UI directly from Port 8091. We do need one more step to achieve this and that is Port Forwarding.

Port Forwarding

We now have deployed three pods, however, it’s enough to port forward from one pod to access the Couchbase Admin UI.

Access the UI

http://localhost:8091

Create the buckets

Add sample bucket and create pillow bucket

Run a workload to generate some metrics

Usaremos cbc-pillowfight to generate the workload. Luckily, this is bundled along with the Operator and lets deploy this. Let’s make a minor modification to the YAML file first, so that it doesn’t stop with loading the data but performs operations on the bucket. We will use the pillow bucket we just created.

Change the bucket from default to pillow and changed the -c (number of loops) option from 10 to 10,000.

Then:

Testing Prometheus and Grafana locally

We now have a three node Couchbase Cluster with the Prometheus Couchbase Exporter. The Exporter is scraping Couchbase metrics to Port 9091. Now, we could forward that port just like we forwarded port 8091 to access the Couchbase Web Console UI from our desktop. With that forwarded port, we could then have Prometheus and Grafana running in Docker containers on the desktop and use the forwarded 9091 port to get the metrics into Prometheus and visualize it in Grafana.

With the above approach, there is a limitation. First is, we would have to forward port 9091 from all 3 nodes and those node names would be hardcoded. Hardcoding node names is a huge problem in a Kubernetes environment. Moreover, you would really not be doing port forwarding in a Production environment, where you would be typically deploying with DNS and would use the DNS SRV to connect to the cluster. Finally, its best practice to run Prometheus and Grafana in Kubernetes itself, aligning with the Cloud Native paradigm.

Próximas etapas

Em Parte 2, we will be doing just that, apart from DNS since we still want to keep this as simple as possible for quick testing.

Recursos:

• Faça o download do Couchbase Autonomous Operator 2.0 Beta para Kubernetes
• Introdução ao Couchbase Autonomous Operator 2.0 Beta
• Tutorial – Operador autônomo do Couchbase no EKS
• Compartilhe suas ideias sobre o Fóruns do Couchbase