{"id":8442,"date":"2020-04-21T06:15:14","date_gmt":"2020-04-21T13:15:14","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=8442"},"modified":"2025-06-13T22:08:23","modified_gmt":"2025-06-14T05:08:23","slug":"step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/","title":{"rendered":"Couchbase Autonomous Operator 2.0 with Prometheus &#8211; Part 2"},"content":{"rendered":"<h3><span style=\"font-weight: 400\">Prerequisites<\/span><\/h3>\n<p><span style=\"font-weight: 400\">As mentioned in <a href=\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-1\/\">Part 1<\/a> of the blog, we need to run Prometheus and Grafana in the Kubernetes environment on our Amazon EKS. The recommended way is to use <\/span><a href=\"https:\/\/github.com\/coreos\/kube-prometheus\"><span style=\"font-weight: 400\">Kube-Prometheus<\/span><\/a><span style=\"font-weight: 400\">, an Open Source project. Not only will this simplify the deployment, but adds a lot more components, like the <\/span><a href=\"https:\/\/prometheus.io\/docs\/guides\/node-exporter\/\"><span style=\"font-weight: 400\">Prometheus Node Exporter<\/span><\/a><span style=\"font-weight: 400\"> which monitors Linux host metrics and is typically used in a Kubernetes environment.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400\">Clone the <a href=\"https:\/\/github.com\/coreos\/kube-prometheus\">https:\/\/github.com\/coreos\/kube-prometheus<\/a> repository from Github, but do not create any manifests just yet.<\/span><\/p>\n<p><span style=\"font-weight: 400\"><strong>Components included in this package<\/strong>:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The <\/span><a href=\"https:\/\/github.com\/coreos\/prometheus-operator\"><span style=\"font-weight: 400\">Prometheus Operator<\/span><\/a><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Highly available <\/span><a href=\"https:\/\/prometheus.io\/\"><span style=\"font-weight: 400\">Prometheus<\/span><\/a><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Highly available <\/span><a href=\"https:\/\/github.com\/prometheus\/alertmanager\"><span style=\"font-weight: 400\">Alertmanager<\/span><\/a><\/li>\n<li style=\"font-weight: 400\"><a href=\"https:\/\/github.com\/prometheus\/node_exporter\"><span style=\"font-weight: 400\">Prometheus node-exporter<\/span><\/a><\/li>\n<li style=\"font-weight: 400\"><a href=\"https:\/\/github.com\/DirectXMan12\/k8s-prometheus-adapter\"><span style=\"font-weight: 400\">Prometheus Adapter for Kubernetes Metrics APIs<\/span><\/a><\/li>\n<li style=\"font-weight: 400\"><a href=\"https:\/\/github.com\/kubernetes\/kube-state-metrics\"><span style=\"font-weight: 400\">kube-state-metrics<\/span><\/a><\/li>\n<li style=\"font-weight: 400\"><a href=\"https:\/\/grafana.com\/\"><span style=\"font-weight: 400\">Grafana<\/span><\/a><\/li>\n<\/ul>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:default decode:true\" title=\"Listing kube-prometheus\">\u279c  kube-prometheus git:(master) \u2717 ls\r\nDCO    README.md examples jsonnet scripts\r\nLICENSE build.sh experimental jsonnetfile.json sync-to-internal-registry.jsonnet\r\nMakefile code-of-conduct.md go.mod jsonnetfile.lock.json test.sh\r\nNOTICE docs go.sum kustomization.yaml tests\r\nOWNERS example.jsonnet hack manifests<\/pre>\n<p><span style=\"font-weight: 400\"><strong>Note<\/strong>:<\/span><\/p>\n<p><span style=\"font-weight: 400\">This tutorial works on the basis that the manifests which bring up the relevant resources for Prometheus Operator are still located in the folder <\/span><b>manifests<\/b><span style=\"font-weight: 400\">.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Please adjust accordingly if changes have been made as the repository is experimental and subject to change.<\/span><\/p>\n<p>&nbsp;<\/p>\n<h3><span style=\"font-weight: 400\">Create the Couchbase ServiceMonitor<\/span><\/h3>\n<p><span style=\"font-weight: 400\">The ServiceMonitor tells Prometheus to monitor a Service resource that defines the endpoints Prometheus scrapes for incoming metrics provided by the couchbase-exporter. This file,<strong>couchbase-serviceMonitor.yaml<\/strong>, should be\u00a0<strong>kube-prometheus\/manifests<\/strong> directory.\u00a0<\/span><\/p>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:yaml decode:true\" title=\"kube-prometheus\/manifests\/couchbase-serviceMonitor.yaml\">apiVersion: monitoring.coreos.com\/v1\r\nkind: ServiceMonitor\r\nmetadata:\r\n\u00a0\u00a0name: couchbase\r\n\u00a0\u00a0namespace: default # &lt;1&gt;\r\n\u00a0\u00a0labels:\r\n\u00a0\u00a0\u00a0\u00a0app: couchbase\r\nspec:\r\n\u00a0\u00a0endpoints:\r\n\u00a0\u00a0- port: metrics \u00a0 \u00a0 \u00a0 # &lt;2&gt;\r\n\u00a0\u00a0\u00a0\u00a0interval: 5s\u00a0 \u00a0 \u00a0 \u00a0 # &lt;3&gt;\r\n\u00a0\u00a0namespaceSelector:\r\n\u00a0\u00a0\u00a0\u00a0matchNames:\r\n\u00a0\u00a0\u00a0\u00a0- default # &lt;4&gt;\r\n\u00a0\u00a0selector:\r\n\u00a0\u00a0\u00a0\u00a0matchLabels:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0app: couchbase # &lt;5&gt;\r\n<\/pre>\n<p><span style=\"font-weight: 400\"><strong>Legend<\/strong>:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">You may wish to include our Couchbase `ServiceMonitor` in the `monitoring` namespace along with the other `ServiceMonitors`. For examples of this tutorial we have just left it in the `default` namespace for ease of use.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The `port` can be a string value and will work for different port numbers of the service as long as the name matches.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">`interval` tells Prometheus how often to scrape the endpoint. Here we want to match the namespace of the `Service` will we be creating in the next step,<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">note that the namespace our `Service` will be running in must be the same namespace of the Couchbase Cluster we wish to scrape metrics from.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Similar to the `namespaceSelector`, this is a simple `labelSelector` that will select the service we will be creating.<\/span><\/li>\n<\/ol>\n<h3><span style=\"font-weight: 400\">Create the Couchbase Metrics Service<\/span><\/h3>\n<p><span style=\"font-weight: 400\">The `Service` will define the port that we described in our ServiceMonitor at `spec.endpoint[0].port` earlier. his file,<strong>couchbase-service.yaml<\/strong>, should be\u00a0<strong>kube-prometheus\/manifests<\/strong> directory.\u00a0<\/span><\/p>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:yaml decode:true\" title=\"kube-prometheus\/manifests\/couchbase-service.yaml\">apiVersion: v1\r\nkind: Service\r\nmetadata:\r\n\u00a0\u00a0name: couchbase-metrics\r\n\u00a0\u00a0namespace: default # &lt;1&gt;\r\n\u00a0\u00a0labels:\r\n\u00a0\u00a0\u00a0\u00a0app: couchbase\r\nspec:\r\n\u00a0\u00a0ports:\r\n\u00a0\u00a0- name: metrics\r\n\u00a0\u00a0\u00a0\u00a0port: 9091 # &lt;2&gt;\r\n\u00a0\u00a0\u00a0\u00a0protocol: TCP\r\n\u00a0\u00a0selector:\r\n\u00a0\u00a0\u00a0\u00a0app: couchbase\r\n\u00a0\u00a0\u00a0\u00a0couchbase_cluster: cb-example # &lt;3&gt;<\/pre>\n<p><span style=\"font-weight: 400\"><strong>Legend<\/strong>:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">As mentioned previously, make sure that the `Service` is in the same namespace as the Couchbase cluster that you wish to scrape metrics from, otherwise no pods will be selected and no endpoints will be displayed in Prometheus Targets. Also make sure this value matches up with `spec.namespaceSelector` in the `ServiceMonitor`.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Keep this port as its default value of 9091 as this is the port the Couchbase Exporter will be exporting to.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">A further level of granularity to your selector can be added in the scenario you have more than one Couchbase Cluster running in the same namespace.<\/span><\/li>\n<\/ol>\n<h3>Prometheus Dynamic Service discovery<\/h3>\n<p>Prometheus discovers the monitoring end points, dynamically, by matching the labels on the ServiceMonitor to the Services which specify the cluster and end points, Port 9091 in our case.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8458\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog-2-services-300x46.png\" alt=\"\" width=\"848\" height=\"130\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-services-300x46.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-services-1024x156.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-services-768x117.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-services-20x3.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-services.png 1312w\" sizes=\"auto, (max-width: 848px) 100vw, 848px\" \/><\/p>\n<p><span style=\"color: #343e47;font-family: Lato, 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size: 40px\">Create the Manifests<\/span><\/p>\n<p><span style=\"font-weight: 400\">Follow the specific command given in the <\/span><a href=\"https:\/\/github.com\/coreos\/kube-prometheus\/blob\/master\/README.md\"><span style=\"font-weight: 400\">Github README<\/span><\/a><span style=\"font-weight: 400\"> to bring up our created resources along with the other provided default manifests.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Components such as Prometheus, AlertManager, NodeExporter and Grafana should then startup and we can confirm this by inspecting the pods in the namespace `monitoring`.<br \/>\n<\/span><br \/>\nLet&#8217;s begin.<\/p>\n<h4><span style=\"font-weight: 400\">Create the Kubernetes namespace and CRDs<\/span><\/h4>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:zsh decode:true\" title=\"Kubernetes namespaces and CRDs\">kube-prometheus git:(master) $ kubectl create -f manifests\/setup\r\n\r\nnamespace\/monitoring created\r\ncustomresourcedefinition.apiextensions.k8s.io\/alertmanagers.monitoring.coreos.com created\r\ncustomresourcedefinition.apiextensions.k8s.io\/podmonitors.monitoring.coreos.com created\r\ncustomresourcedefinition.apiextensions.k8s.io\/prometheuses.monitoring.coreos.com created\r\ncustomresourcedefinition.apiextensions.k8s.io\/prometheusrules.monitoring.coreos.com created\r\ncustomresourcedefinition.apiextensions.k8s.io\/servicemonitors.monitoring.coreos.com created\r\ncustomresourcedefinition.apiextensions.k8s.io\/thanosrulers.monitoring.coreos.com created\r\nclusterrole.rbac.authorization.k8s.io\/prometheus-operator created\r\nclusterrolebinding.rbac.authorization.k8s.io\/prometheus-operator created\r\ndeployment.apps\/prometheus-operator created\r\nservice\/prometheus-operator created\r\nserviceaccount\/prometheus-operator created<\/pre>\n<p><span style=\"font-weight: 400\">Wait for a few minutes before the next step, but it may be necessary to run the command multiple times for all components to be created successfully.<\/span><\/p>\n<h4><span style=\"font-weight: 400\">Create the remaining resources<\/span><\/h4>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:zsh decode:true \" title=\"Remaining resources\">kube-prometheus git:(master) $ kubectl create -f manifests\/\r\n\r\nalertmanager.monitoring.coreos.com\/main created\r\nsecret\/alertmanager-main created\r\nservice\/alertmanager-main created\r\nserviceaccount\/alertmanager-main created\r\nservicemonitor.monitoring.coreos.com\/alertmanager created\r\nservice\/couchbase-metrics created\r\nservicemonitor.monitoring.coreos.com\/couchbase created\r\n...\r\nservicemonitor.monitoring.coreos.com\/kubelet created<\/pre>\n<h3><span style=\"font-weight: 400\">Check monitoring namespaces<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Components such as Prometheus, AlertManager, NodeExporter and Grafana should then startup and we can confirm this by inspecting the pods in the namespace `monitoring`.<\/span><\/p>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:default decode:true\" title=\"Monitoring namespaces\">$ kubectl get pods -n monitoring\r\n\r\nNAME \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 READY STATUS RESTARTS AGE\r\nalertmanager-main-0\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2\/2 Running 0 69m\r\nalertmanager-main-1\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2\/2 Running 0 69m\r\nalertmanager-main-2\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2\/2 Running 0 69m\r\ngrafana-75d8c76bdd-4l284 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 1\/1 Running 0 69m\r\nkube-state-metrics-54dc88ccd8-nntts\u00a0 \u00a0 3\/3 Running 0 69m\r\nnode-exporter-pk65z\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2\/2 Running 0 69m\r\nnode-exporter-s9k9n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2\/2 Running 0 69m\r\nnode-exporter-vhjpw\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2\/2 Running 0 69m\r\nprometheus-adapter-8667948d79-vfcbv\u00a0 \u00a0 1\/1 Running 0 69m\r\nprometheus-k8s-0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 3\/3 Running 1 69m\r\nprometheus-k8s-1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 3\/3 Running 0 69m\r\nprometheus-operator-696554666f-9cnnv \u00a0 2\/2 Running 0 89m<\/pre>\n<h3><span style=\"font-weight: 400\">Check that our ServiceMonitor has been created.<\/span><\/h3>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:zsh decode:true\" title=\"Check ServiceMonitors\">$ kubectl get servicemonitors --all-namespaces\r\n\r\nNAMESPACE\u00a0 \u00a0 NAME \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 AGE\r\ndefault\u00a0 \u00a0 \u00a0 couchbase \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m33s\r\nmonitoring \u00a0 alertmanager \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m33s\r\nmonitoring \u00a0 coredns \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m22s\r\nmonitoring \u00a0 grafana \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m26s\r\nmonitoring \u00a0 kube-apiserver\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m22s\r\nmonitoring \u00a0 kube-controller-manager \u00a0 2m22s\r\nmonitoring \u00a0 kube-scheduler\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m21s\r\nmonitoring \u00a0 kube-state-metrics\u00a0 \u00a0 \u00a0 \u00a0 2m25s\r\nmonitoring \u00a0 kubelet \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m21s\r\nmonitoring \u00a0 node-exporter \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m25s\r\nmonitoring \u00a0 prometheus \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 2m22s\r\nmonitoring \u00a0 prometheus-operator \u00a0 \u00a0 \u00a0 2m23s<\/pre>\n<h3><span style=\"font-weight: 400\">Check that our Service has been created.<\/span><\/h3>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:default decode:true\" title=\"Check Services\">$ kubectl get svc --all-namespaces\r\n\r\nNAMESPACE \u00a0 \u00a0 NAME \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 PORT(S)\r\ndefault \u00a0 \u00a0 \u00a0 cb-example \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 8091\/TCP,8092\/TCP,8093\/TCP,\r\ndefault \u00a0 \u00a0 \u00a0 cb-example-srv \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 11210\/TCP,11207\/TCP\r\ndefault \u00a0 \u00a0 \u00a0 couchbase-metrics\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9091\/TCP\r\ndefault \u00a0 \u00a0 \u00a0 couchbase-operator \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 8080\/TCP,8383\/TCP\r\ndefault \u00a0 \u00a0 \u00a0 couchbase-operator-admission \u00a0 443\/TCP\r\ndefault \u00a0 \u00a0 \u00a0 kubernetes \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 443\/TCP\r\nkube-system \u00a0 kube-dns \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 53\/UDP,53\/TCP\r\nkube-system \u00a0 kubelet \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 10250\/TCP,10255\/TCP,4194\/TCP,...\r\nmonitoring\u00a0 \u00a0 alertmanager-main\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9093\/TCP\r\nmonitoring\u00a0 \u00a0 alertmanager-operated\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9093\/TCP,9094\/TCP,9094\/UDP\r\nmonitoring\u00a0 \u00a0 grafana \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 3000\/TCP\r\nmonitoring\u00a0 \u00a0 kube-state-metrics \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 8443\/TCP,9443\/TCP\u00a0\r\nmonitoring\u00a0 \u00a0 node-exporter \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9100\/TCP\r\nmonitoring\u00a0 \u00a0 prometheus-adapter \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 443\/TCP\r\nmonitoring\u00a0 \u00a0 prometheus-k8s \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9090\/TCP\r\nmonitoring\u00a0 \u00a0 prometheus-operated\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 9090\/TCP\r\nmonitoring\u00a0 \u00a0 prometheus-operator\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 8443\/TCP<\/pre>\n<p><span style=\"font-weight: 400\">In the above output, we not only see the services, but also the ports. We will use this information to forward these ports, like we did with the Couchbase Administration UI, in order to access these services.<\/span><\/p>\n<p><span style=\"font-weight: 400\">To check that all is working correctly with the Prometheus Operator deployment, run the following command to view the logs:<\/span><\/p>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:zsh decode:true \">$ kubectl logs -f deployments\/prometheus-operator -n monitoring prometheus-operator<\/pre>\n<h3><span style=\"font-weight: 400\">Port forwarding<\/span><\/h3>\n<p><span style=\"font-weight: 400\">We have already forwarded the Couchbase Admin UI port 8091 from one Couchbase node previously, but I\u2019m giving this again, this time from the service point of view.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400\">In addition to that port, we actually need only the Grafana service access, Port 3000. However, let\u2019s access Prometheus service Port 9090 as well. Then we can take a look at all the metrics from the different exporters and try a little PromQL, the Prometheus Query Language as well.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400\">Now, the 3 above should be sufficient. However, there\u2019s some additional advantage of taking a look at the metrics from each individual service as well. The Couchbase exporter exposes the Couchbase metrics on Port 9091. So, we can forward those ports as well. Note that you really need only access to Grafana.<\/span><\/p>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:default decode:true\" title=\"Port forwarding\">kubectl --namespace default port-forward svc\/cb-example 8091 &amp;\r\nkubectl --namespace monitoring port-forward svc\/prometheus-k8s 9090 &amp;\r\nkubectl --namespace monitoring port-forward svc\/grafana 3000 &amp;\r\nkubectl --namespace monitoring port-forward svc\/alertmanager-main 9093 &amp;\r\nkubectl --namespace monitoring port-forward svc\/node-exporter 9100 &amp;\r\nkubectl --namespace default port-forward svc\/couchbase-metrics 9091 &amp;<\/pre>\n<h3><span style=\"font-weight: 400\">Check out Prometheus Targets<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Access: <\/span><span style=\"font-weight: 400\">https:\/\/localhost:9090\/targets<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8443\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog-2-targets-300x189.png\" alt=\"\" width=\"911\" height=\"574\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-targets-300x189.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-targets-1024x646.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-targets-768x484.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-targets-1536x969.png 1536w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-targets-20x13.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-targets-1320x833.png 1320w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-targets.png 1560w\" sizes=\"auto, (max-width: 911px) 100vw, 911px\" \/><\/p>\n<p><span style=\"font-weight: 400\">All Prometheus targets should be UP. There are quite a few of these since Kube-Prometheus deployed a bunch of exporters.<\/span><\/p>\n<h3><span style=\"font-weight: 400\">Check out the raw Couchbase Metrics<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Access: <\/span><span style=\"font-weight: 400\">https:\/\/localhost:9091\/metrics<\/span><\/p>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:zsh decode:true\" title=\"Couchbase metrics\"># HELP cbbucketinfo_basic_dataused_bytes basic_dataused\r\n# TYPE cbbucketinfo_basic_dataused_bytes gauge\r\ncbbucketinfo_basic_dataused_bytes{bucket=\"pillow\"} 1.84784896e+08\r\ncbbucketinfo_basic_dataused_bytes{bucket=\"travel-sample\"} 1.51648256e+08\r\n# HELP cbbucketinfo_basic_diskfetches basic_diskfetches\r\n# TYPE cbbucketinfo_basic_diskfetches gauge\r\ncbbucketinfo_basic_diskfetches{bucket=\"pillow\"} 0\r\ncbbucketinfo_basic_diskfetches{bucket=\"travel-sample\"} 0\r\n# HELP cbbucketinfo_basic_diskused_bytes basic_diskused\r\n# TYPE cbbucketinfo_basic_diskused_bytes gauge\r\ncbbucketinfo_basic_diskused_bytes{bucket=\"pillow\"} 1.98967788e+08\r\ncbbucketinfo_basic_diskused_bytes{bucket=\"travel-sample\"} 1.91734038e+08<\/pre>\n<p><span style=\"font-weight: 400\">This output is useful as you can rapidly search through the list.<\/span><\/p>\n<h3><span style=\"font-weight: 400\">Try a basic PromQL query<\/span><\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8444\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog-2-PromQL-300x127.png\" alt=\"\" width=\"907\" height=\"384\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-PromQL-300x127.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-PromQL-1024x435.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-PromQL-768x326.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-PromQL-1536x652.png 1536w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-PromQL-20x8.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-PromQL-1320x560.png 1320w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-PromQL.png 1560w\" sizes=\"auto, (max-width: 907px) 100vw, 907px\" \/><\/p>\n<p><span style=\"font-weight: 400\">In the above UI, click on <\/span><b>Graph<\/b><span style=\"font-weight: 400\"> first.<\/span><\/p>\n<p><span style=\"font-weight: 400\">The drop box gives you the list of metrics scraped. This is the complete list of <\/span><b>all<\/b><span style=\"font-weight: 400\"> the metrics scraped by <\/span><b>all<\/b><span style=\"font-weight: 400\"> the exporters and that&#8217;s a pretty daunting list.\u00a0 One method to narrow down the list to just Couchbase metrics, is of course, to access the 9091 endpoint as previously described.<\/span><\/p>\n<h3><span style=\"font-weight: 400\">Check out Grafana<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Access: <\/span><span style=\"font-weight: 400\">https:\/\/localhost:<\/span><span style=\"font-weight: 400\">3000<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8445\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog-2-Grafana-300x127.png\" alt=\"\" width=\"909\" height=\"385\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Grafana-300x127.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Grafana-1024x435.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Grafana-768x326.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Grafana-1536x652.png 1536w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Grafana-20x8.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Grafana-1320x560.png 1320w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Grafana.png 1560w\" sizes=\"auto, (max-width: 909px) 100vw, 909px\" \/><\/p>\n<p><span style=\"font-weight: 400\">Userid and password are: admin\/admin<\/span><\/p>\n<p><span style=\"font-weight: 400\">The kube-prometheus deployment of Grafana <\/span><b>already<\/b><span style=\"font-weight: 400\"> has the Prometheus Data Source defined and a large set of <\/span><b>Default Dashboards<\/b><span style=\"font-weight: 400\">. Let&#8217;s check out the <\/span><b>Default Node Dashboard<\/b><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8446\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog-2-default-node-grafana-300x146.png\" alt=\"\" width=\"902\" height=\"439\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-default-node-grafana-300x146.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-default-node-grafana-1024x499.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-default-node-grafana-768x374.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-default-node-grafana-1536x748.png 1536w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-default-node-grafana-20x10.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-default-node-grafana-1320x643.png 1320w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-default-node-grafana.png 1560w\" sizes=\"auto, (max-width: 902px) 100vw, 902px\" \/><\/p>\n<h3><span style=\"font-weight: 400\">Build a Sample Grafana Dashboard to Monitor Couchbase Metrics<\/span><\/h3>\n<p><span style=\"font-weight: 400\">We will not be building a complete Dashboard, but a small sample with a few panels to show how it&#8217;s done. This dashboard will monitor the number of items in a bucket and the number of GET and SET operations.<\/span><\/p>\n<p><span style=\"font-weight: 400\"><strong>Note<\/strong>: Please have the pillow-fight application running as described in Part 1. This will generate the operations which we are interested in monitoring.<\/span><\/p>\n<h4><span style=\"font-weight: 400\">Prometheus Metrics<\/span><\/h4>\n<p><span style=\"font-weight: 400\">Access: <\/span><span style=\"font-weight: 400\">https:\/\/localhost:9091\/<\/span><span style=\"font-weight: 400\">graph<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8447\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog-2-Prom-CB-metrics-300x137.png\" alt=\"\" width=\"909\" height=\"415\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Prom-CB-metrics-300x137.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Prom-CB-metrics-1024x467.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Prom-CB-metrics-768x351.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Prom-CB-metrics-1536x701.png 1536w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Prom-CB-metrics-20x9.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Prom-CB-metrics-1320x602.png 1320w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-Prom-CB-metrics.png 1560w\" sizes=\"auto, (max-width: 909px) 100vw, 909px\" \/><\/p>\n<p><span style=\"font-weight: 400\">We are interested in the current items in a bucket. There are a couple of metrics which supply that, cluster wide and per node. Let&#8217;s use the per node metric. We will then allow Prometheus to handle all aggregations, as per best practices. Another advantage is that we can even show the current items in the bucket, per node basis just to check if our data set is skewed.<\/span><\/p>\n<h4><span style=\"font-weight: 400\">Let&#8217;s take a look at one Element:<\/span><\/h4>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:zsh decode:true\" title=\"Sample Couchase metric\">cbpernodebucket_curr_items{bucket=\"pillow\",endpoint=\"metrics\",instance=\"192.168.2.93:9091\",job=\"couchbase-metrics\",namespace=\"default\",node=\"cb-example-0000.cb-example.default.svc:8091\",pod=\"cb-example-0000\",service=\"couchbase-metrics\"}<\/pre>\n<p><span style=\"font-weight: 400\">In the above example, we are interested in these labels: <\/span><span style=\"font-weight: 400\">bucket<\/span><span style=\"font-weight: 400\">, part of <\/span><span style=\"font-weight: 400\">node<\/span><span style=\"font-weight: 400\"> (the middle part, <\/span><i><span style=\"font-weight: 400\">cb-example<\/span><\/i><span style=\"font-weight: 400\"> which is the cluster name, and <\/span><span style=\"font-weight: 400\">pod<\/span><span style=\"font-weight: 400\">. We are also interested in <\/span><span style=\"font-weight: 400\">service<\/span><span style=\"font-weight: 400\"> in order to filter. These will help us design a dashboard where we can view the metrics by bucket, node or cluster.<\/span><\/p>\n<h3><span style=\"font-weight: 400\">The Sample Dashboard<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Lets create a new blank sample dashboard.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8448\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog2-grafana-sample-300x177.png\" alt=\"\" width=\"912\" height=\"538\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-grafana-sample-300x177.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-grafana-sample-1024x605.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-grafana-sample-768x454.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-grafana-sample-1536x908.png 1536w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-grafana-sample-20x12.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-grafana-sample-1320x780.png 1320w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-grafana-sample.png 1560w\" sizes=\"auto, (max-width: 912px) 100vw, 912px\" \/><\/p>\n<h4><span style=\"font-weight: 400\">Adding variables<\/span><\/h4>\n<p><span style=\"font-weight: 400\">Since we want the metrics per bucket, node and cluster, let&#8217;s add these variables so that they can be selected in a drop box.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8449\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog2-var-1-300x292.png\" alt=\"\" width=\"892\" height=\"868\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-var-1-300x292.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-var-1-1024x997.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-var-1-768x748.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-var-1-50x50.png 50w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-var-1-20x20.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-var-1.png 1132w\" sizes=\"auto, (max-width: 892px) 100vw, 892px\" \/><\/p>\n<p><span style=\"font-weight: 400\">The above example creates the variable bucket. Note the Query and Regex expression.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400\">Let&#8217;s create 2 more so that we have 3 variables:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8450\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog-2-all-vars-300x194.png\" alt=\"\" width=\"892\" height=\"577\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-all-vars-300x194.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-all-vars-1024x661.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-all-vars-768x496.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-all-vars-20x13.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-all-vars.png 1094w\" sizes=\"auto, (max-width: 892px) 100vw, 892px\" \/><\/p>\n<p><span style=\"font-weight: 400\">The Query does not change for these 3, but here are the Regex expressions:<\/span><\/p>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:default decode:true\">Query: {service=\"couchbase-metrics\"}\r\n\r\n$node: Regex= .*pod=\"(.*?)\".*\r\n$bucket: Regex= .*bucket=\"(.*?)\".*\r\n$cluster: Regex=.*node=\\\".*\\.(.*)\\..*\\..*:8091\\\".*<\/pre>\n<h3><span style=\"font-weight: 400\">Creating a Panel<\/span><\/h3>\n<h4><span style=\"font-weight: 400\">Create 3 Panels for Current Items, GETs and SETs<\/span><\/h4>\n<p><span style=\"font-weight: 400\">You can duplicate each panel and edit them. These are the queries:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8451\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog-2-GETs-300x239.png\" alt=\"\" width=\"890\" height=\"709\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-GETs-300x239.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-GETs-1024x817.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-GETs-768x613.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-GETs-20x16.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog-2-GETs.png 1050w\" sizes=\"auto, (max-width: 890px) 100vw, 890px\" \/><\/p>\n<p><span style=\"font-weight: 400\">Items Panel: sum(cbpernodebucket_curr_items{bucket=~&#8221;$bucket&#8221;,pod=~&#8221;$node&#8221;}) by (bucket)<\/span><\/p>\n<p><span style=\"font-weight: 400\">GETs Panel: sum(cbpernodebucket_cmd_get{bucket=~&#8221;$bucket&#8221;,pod=~&#8221;$node&#8221;}) by (bucket)<\/span><\/p>\n<p><span style=\"font-weight: 400\">SETs Panel: sum(cbpernodebucket_cmd_set{bucket=~&#8221;$bucket&#8221;,pod=~&#8221;$node&#8221;}) by (bucket)<\/span><\/p>\n<h3><span style=\"font-weight: 400\">The completed sample Grafana Dashboard<\/span><\/h3>\n<p><span style=\"font-weight: 400\">This is how our final sample dashboard looks like.<\/span><\/p>\n<h1><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-8433\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/Blog2-Grafana-Final-300x81.png\" alt=\"\" width=\"885\" height=\"239\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-Grafana-Final-300x81.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-Grafana-Final-1024x275.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-Grafana-Final-768x206.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-Grafana-Final-20x5.png 20w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/Blog2-Grafana-Final.png 1594w\" sizes=\"auto, (max-width: 885px) 100vw, 885px\" \/><\/h1>\n<h3><span style=\"font-weight: 400\">Clean Up<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Finally clean up your deployment:<\/span><\/p>\n<pre class=\"theme:dark-terminal toolbar:1 show-lang:2 show-plain:3 lang:default decode:true \" title=\"Cleanup\">kube-prometheus git:(master) $ kubectl delete --ignore-not-found=true -f manifests\/ -f manifests\/setup\r\n\r\ncao-2 $ kubectl delete -f pillowfight-data-loader.yaml\r\ncao-2 $ kubectl delete -f my-cluster.yaml\r\ncao-2 $ bin\/cbopcfg | kubectl delete -f -\r\ncao-2 $ kubectl delete -f crd.yaml\r\ncao-2 $ eksctl delete cluster --region=us-east-1 --name=prasadCAO2<\/pre>\n<h4>Resources:<\/h4>\n<ul>\n<li><a href=\"https:\/\/www.couchbase.com\/downloads?family=kubernetes&amp;product=open-source-kubernetes-developer\">Download Couchbase Autonomous Operator 2.0 Beta for Kubernetes<\/a><\/li>\n<li><a href=\"https:\/\/docs.couchbase.com\/operator\/2.0\/overview.html\">Getting started Couchbase Autonomous Operator 2.0 Beta<\/a><\/li>\n<li>Tutorial &#8211; <a href=\"https:\/\/docs.couchbase.com\/operator\/2.0\/tutorial-eks.html\">Couchbase Autonomous Operator on EKS<\/a><\/li>\n<li>Share your thoughts on the <a href=\"https:\/\/www.couchbase.com\/forums\/c\/couchbase-server\/Kubernetes\/\">Couchbase Forums<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Prerequisites As mentioned in Part 1 of the blog, we need to run Prometheus and Grafana in the Kubernetes environment on our Amazon EKS. The recommended way is to use Kube-Prometheus, an Open Source project. Not only will this simplify [&hellip;]<\/p>\n","protected":false},"author":57747,"featured_media":10442,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1815,1821,9284,2225,1816,2322,2334],"tags":[9243,2384,1545,1335,2383],"ppma_author":[9106],"class_list":["post-8442","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-best-practices-and-tutorials","category-couchbase-architecture","category-couchbase-autonomous-operator","category-cloud","category-couchbase-server","category-kubernetes","category-monitoring","tag-cloud-native-database","tag-grafana","tag-kubernetes","tag-monitoring","tag-prometheus"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.7.1 (Yoast SEO v25.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Couchbase Autonomous Operator 2.0 with Prometheus - Part 2 - The Couchbase Blog<\/title>\n<meta name=\"description\" content=\"A step by step guide for running the Couchbase Autonomous Operator with Prometheus on Amazon EKS and visualizing with Grafana\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Couchbase Autonomous Operator 2.0 with Prometheus - Part 2\" \/>\n<meta property=\"og:description\" content=\"A step by step guide for running the Couchbase Autonomous Operator with Prometheus on Amazon EKS and visualizing with Grafana\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/\" \/>\n<meta property=\"og:site_name\" content=\"The Couchbase Blog\" \/>\n<meta property=\"article:published_time\" content=\"2020-04-21T13:15:14+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-14T05:08:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"667\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Prasad Doddi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prasad Doddi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"10 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/\"},\"author\":{\"name\":\"Prasad Doddi\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/7870a85b21341a1cdbdd737ba6e6e077\"},\"headline\":\"Couchbase Autonomous Operator 2.0 with Prometheus &#8211; Part 2\",\"datePublished\":\"2020-04-21T13:15:14+00:00\",\"dateModified\":\"2025-06-14T05:08:23+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/\"},\"wordCount\":1343,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png\",\"keywords\":[\"Cloud-Native Database\",\"Grafana\",\"kubernetes\",\"monitoring\",\"Prometheus\"],\"articleSection\":[\"Best Practices and Tutorials\",\"Couchbase Architecture\",\"Couchbase Autonomous Operator\",\"Couchbase Capella\",\"Couchbase Server\",\"Kubernetes\",\"Monitoring\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/\",\"name\":\"Couchbase Autonomous Operator 2.0 with Prometheus - Part 2 - The Couchbase Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png\",\"datePublished\":\"2020-04-21T13:15:14+00:00\",\"dateModified\":\"2025-06-14T05:08:23+00:00\",\"description\":\"A step by step guide for running the Couchbase Autonomous Operator with Prometheus on Amazon EKS and visualizing with Grafana\",\"breadcrumb\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#primaryimage\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png\",\"width\":1200,\"height\":667},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.couchbase.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Couchbase Autonomous Operator 2.0 with Prometheus &#8211; Part 2\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"name\":\"The Couchbase Blog\",\"description\":\"Couchbase, the NoSQL Database\",\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\",\"name\":\"The Couchbase Blog\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"width\":218,\"height\":34,\"caption\":\"The Couchbase Blog\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/7870a85b21341a1cdbdd737ba6e6e077\",\"name\":\"Prasad Doddi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/eefad0ed7be820b285621aa4d67f7578\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a9ce547feba43afcbcf1425142725c663678810966eaa0ddc7d38702e647ee63?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a9ce547feba43afcbcf1425142725c663678810966eaa0ddc7d38702e647ee63?s=96&d=mm&r=g\",\"caption\":\"Prasad Doddi\"},\"description\":\"Prasad is a Senior Product Manager in Couchbase Cloud. Prior to Couchbase, he worked at IBM in various departments including Development, QA, Support and Technical Sales. Prasad holds a master\u2019s degree in Chem. Engg. from Clarkson University, NY.\",\"sameAs\":[\"www.linkedin.com\/in\/krishna-prasad-doddi\"],\"url\":\"https:\/\/www.couchbase.com\/blog\/author\/prasad-doddi\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Couchbase Autonomous Operator 2.0 with Prometheus - Part 2 - The Couchbase Blog","description":"A step by step guide for running the Couchbase Autonomous Operator with Prometheus on Amazon EKS and visualizing with Grafana","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/","og_locale":"en_US","og_type":"article","og_title":"Couchbase Autonomous Operator 2.0 with Prometheus - Part 2","og_description":"A step by step guide for running the Couchbase Autonomous Operator with Prometheus on Amazon EKS and visualizing with Grafana","og_url":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/","og_site_name":"The Couchbase Blog","article_published_time":"2020-04-21T13:15:14+00:00","article_modified_time":"2025-06-14T05:08:23+00:00","og_image":[{"width":1200,"height":667,"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png","type":"image\/png"}],"author":"Prasad Doddi","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Prasad Doddi","Est. reading time":"10 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#article","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/"},"author":{"name":"Prasad Doddi","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/7870a85b21341a1cdbdd737ba6e6e077"},"headline":"Couchbase Autonomous Operator 2.0 with Prometheus &#8211; Part 2","datePublished":"2020-04-21T13:15:14+00:00","dateModified":"2025-06-14T05:08:23+00:00","mainEntityOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/"},"wordCount":1343,"commentCount":0,"publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png","keywords":["Cloud-Native Database","Grafana","kubernetes","monitoring","Prometheus"],"articleSection":["Best Practices and Tutorials","Couchbase Architecture","Couchbase Autonomous Operator","Couchbase Capella","Couchbase Server","Kubernetes","Monitoring"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/","url":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/","name":"Couchbase Autonomous Operator 2.0 with Prometheus - Part 2 - The Couchbase Blog","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#primaryimage"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png","datePublished":"2020-04-21T13:15:14+00:00","dateModified":"2025-06-14T05:08:23+00:00","description":"A step by step guide for running the Couchbase Autonomous Operator with Prometheus on Amazon EKS and visualizing with Grafana","breadcrumb":{"@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#primaryimage","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2020\/04\/background-1.png","width":1200,"height":667},{"@type":"BreadcrumbList","@id":"https:\/\/www.couchbase.com\/blog\/step-by-step-guide-for-running-couchbase-autonomous-operator-2-0-with-prometheus-part-2\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.couchbase.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Couchbase Autonomous Operator 2.0 with Prometheus &#8211; Part 2"}]},{"@type":"WebSite","@id":"https:\/\/www.couchbase.com\/blog\/#website","url":"https:\/\/www.couchbase.com\/blog\/","name":"The Couchbase Blog","description":"Couchbase, the NoSQL Database","publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.couchbase.com\/blog\/#organization","name":"The Couchbase Blog","url":"https:\/\/www.couchbase.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","width":218,"height":34,"caption":"The Couchbase Blog"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/7870a85b21341a1cdbdd737ba6e6e077","name":"Prasad Doddi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/eefad0ed7be820b285621aa4d67f7578","url":"https:\/\/secure.gravatar.com\/avatar\/a9ce547feba43afcbcf1425142725c663678810966eaa0ddc7d38702e647ee63?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a9ce547feba43afcbcf1425142725c663678810966eaa0ddc7d38702e647ee63?s=96&d=mm&r=g","caption":"Prasad Doddi"},"description":"Prasad is a Senior Product Manager in Couchbase Cloud. Prior to Couchbase, he worked at IBM in various departments including Development, QA, Support and Technical Sales. Prasad holds a master\u2019s degree in Chem. Engg. from Clarkson University, NY.","sameAs":["www.linkedin.com\/in\/krishna-prasad-doddi"],"url":"https:\/\/www.couchbase.com\/blog\/author\/prasad-doddi\/"}]}},"authors":[{"term_id":9106,"user_id":57747,"is_guest":0,"slug":"prasad-doddi","display_name":"Prasad Doddi","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/a9ce547feba43afcbcf1425142725c663678810966eaa0ddc7d38702e647ee63?s=96&d=mm&r=g","author_category":"","last_name":"Doddi","first_name":"Prasad","job_title":"","user_url":"","description":"Prasad is a Senior Product Manager for Couchbase Supportability, Manageability and Tools. Prior to Couchbase, he worked at IBM in various departments including Development, QA, Support and Technical Sales. Prasad holds a master\u2019s degree in Chem. Engg. from Clarkson University, NY."}],"_links":{"self":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/8442","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/users\/57747"}],"replies":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/comments?post=8442"}],"version-history":[{"count":0,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/8442\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media\/10442"}],"wp:attachment":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media?parent=8442"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/categories?post=8442"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/tags?post=8442"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=8442"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}