{"id":11527,"date":"2021-09-23T00:00:10","date_gmt":"2021-09-23T07:00:10","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=11527&#038;preview=true&#038;preview_id=11527"},"modified":"2025-06-13T21:25:14","modified_gmt":"2025-06-14T04:25:14","slug":"how-to-build-observability-dashboards-prometheus-grafana-couchbase","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/","title":{"rendered":"How to Build Observability Dashboards with Prometheus, Grafana &amp; Couchbase"},"content":{"rendered":"<p><strong>You&#8217;ve certainly heard it before:<\/strong> &#8220;What gets measured gets done.&#8221;<\/p>\n<p>It&#8217;s true: what you observe and measure is what you can improve.<\/p>\n<p>The key to any improvement is to first identify <em>what<\/em> to measure and then collect the related metrics. Using those metrics, you can tune the underlying work and analyze the effectiveness of any changes. Then repeat the cycle until you&#8217;ve sufficiently improved.<\/p>\n<p>At Couchbase, we needed to improve some of our day-to-day operations, so we created observability dashboards to help us identify issues and track improvement. We used a combination of <a href=\"https:\/\/prometheus.io\/?ref=hello-from-couchbase\" target=\"_blank\" rel=\"noopener\">Prometheus<\/a>, which simplifies storing and querying time-series data, and <a href=\"https:\/\/grafana.com\/?ref=hello-from-couchbase\" target=\"_blank\" rel=\"noopener\">Grafana<\/a>, which can be used to make stunning data visualizations. In addition, we used <a href=\"https:\/\/developer.couchbase.com\/new-to-couchbase\/?ref=blog\" target=\"_blank\" rel=\"noopener\">Couchbase<\/a> to store historical data for later use with its <a href=\"https:\/\/www.couchbase.com\/products\/full-text-search\/?ref=blog\" target=\"_blank\" rel=\"noopener\">Full-Text Search<\/a> and <a href=\"https:\/\/www.couchbase.com\/products\/analytics\/?ref=blog\" target=\"_blank\" rel=\"noopener\">Analytics<\/a> tools.<\/p>\n<p>In this article, we&#8217;ll walk you through how to build your own observability dashboard using Prometheus, Grafana and Couchbase.<\/p>\n<p>Your in-house data source pipelines may vary \u2013 as might your data visualization software. However, the steps we&#8217;ll show you today should be applicable across a number of tools and deployments.<\/p>\n<h2>Generic Observability Dashboard: Design &amp; Architecture<\/h2>\n<p>In order to build a reusable and scalable tool, it&#8217;s better to work from common designs and templates as a first step. From there, you can customize as needed. With this approach, it&#8217;s quick and easy to develop future dashboards.<\/p>\n<p>The diagram below shows the generic architecture of the observability dashboards we&#8217;ll build together:<\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/architecture-diagram-couchbase-grafana-prometheus-dashboard.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12097\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/architecture-diagram-couchbase-grafana-prometheus-dashboard.png\" alt=\"An observability dashboard architecture using Prometheus, Couchbase and Grafana\" width=\"1000\" height=\"526\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/architecture-diagram-couchbase-grafana-prometheus-dashboard.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/architecture-diagram-couchbase-grafana-prometheus-dashboard-300x158.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/architecture-diagram-couchbase-grafana-prometheus-dashboard-768x404.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/architecture-diagram-couchbase-grafana-prometheus-dashboard-20x11.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>In this architecture, two different data inputs form an interface to the dashboard service. Let&#8217;s take a closer look at each of these below.<\/p>\n<ol>\n<li><strong>JSON metadata about the dashboard<\/strong>\n<ul>\n<li>Data source definitions, including information about the data sources (like DB URL, SQL, credentials), the file paths, and Jenkins artifacts URLs.<\/li>\n<li>The Grafana layout template (or visual dashboard view), which we&#8217;ll design first and then use as templates for panels in our later dashboards.<\/li>\n<\/ul>\n<\/li>\n<li><strong>The actual data source files from `.json` and `.csv` files and from Couchbase<\/strong>.\n<ul>\n<li>The design of these observability dashboards supports various data sources like Couchbase Server, and direct files like <a href=\"https:\/\/www.couchbase.com\/blog\/json-database\/?ref=blog\" target=\"_blank\" rel=\"noopener\">JSON documents<\/a> and CSV (Comma Separated Values) files. You can extend the databoard proxy service code (in `dashboard.py`) to parse other data formats as needed.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>The expected output will be a Grafana dashboard UI and Prometheus time-series collected metrics from the two inputs listed above. The central part of the above diagram shows the different services in the collection that support the creation of the dashboards.<\/p>\n<p>Let&#8217;s take a closer look at the different facets and services included in the architecture diagram:<\/p>\n<ul>\n<li><strong>Dashboard proxy service:<\/strong>\n<ul>\n<li>This is a generic Python Flask web app service (`dashboard.py`) that interacts with the Grafana service to serve the tabular data and other APIs like `\/query`, `\/add`, `\/import` and `\/export` endpoints. You can develop a similar one to have a generic template (JSON) for the panels on Grafana and attach the graph data points and tabular-data points as target JSON to display on your Grafana dashboard.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Prometheus export service:<\/strong>\n<ul>\n<li>This is a custom Prometheus exporter (say `prometheus.py`) Flask web app service that connects to the data sources and serves the requests from Prometheus itself. At a high level, this acts as a bridge between the Prometheus and datasources. Note that this service is needed only when the data source is to be maintained for time series (many trends need this).<\/li>\n<\/ul>\n<\/li>\n<li><strong>Grafana service:<\/strong>\n<ul>\n<li>This is the regular Grafana tool itself that you use to create panels and display as dashboards.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Prometheus service:<\/strong>\n<ul>\n<li>This is the regular Prometheus tool itself that holds your metrics as time-series data.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Alert Manager:<\/strong>\n<ul>\n<li>The Alert Manager has custom alert rules that receive alerts when certain thresholds are met.<\/li>\n<\/ul>\n<\/li>\n<li>Other services:\n<ul>\n<li><strong>Couchbase:<\/strong> You might already be using this <a href=\"https:\/\/www.couchbase.com\/resources\/why-nosql\/?ref=blog\" target=\"_blank\" rel=\"noopener\">NoSQL<\/a> document database, but if not, you can install it through a container or directly on a different host. Couchbase stores your data as JSON documents, or you can have it store required fields as separate documents for historic trends while preparing your health or trend data.<\/li>\n<li><strong>Docker:<\/strong> You&#8217;ll need to install the docker agent software on the host in order to use this containerized service deployment.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h3>Sample Dashboard JSON Structure<\/h3>\n<p>In the table below, you&#8217;ll see a sample of the structure of both the input metadata and the input data source.<\/p>\n<table>\n<tbody>\n<tr>\n<td>Input metadata JSON structure:<\/td>\n<td>Input data sources structure:<\/td>\n<\/tr>\n<tr>\n<td><code>{<br \/>\n\"dashboard_title\": \"\",<br \/>\n\"data\": [<br \/>\n{<br \/>\n\"source\": \"couchbase|json|csv\",<br \/>\n\"type\": \"timeseries|table\",<br \/>\n\"name\": \"&lt;unique name for this data source&gt;\",<br \/>\n\"refresh\": \"&lt;how often the data should be refreshed (seconds)&gt;\"<br \/>\n}<br \/>\n],<br \/>\n\"grafana\": [<br \/>\n{ \u201ctemplate\u201d: \u201clink to template json\u201d },<br \/>\n{<br \/>\n\"title\": \"Links\",<br \/>\n\"grid_position\": {},<br \/>\n\"type\": \"text\",<br \/>\n\"links\": []<br \/>\n},<br \/>\n]<br \/>\n}<br \/>\n<\/code><\/td>\n<td><code>\/\/Couchbase source<br \/>\n{<br \/>\n\"host\": \"&lt;couchbase host&gt;\",<br \/>\n\"username\": \"&lt;couchbase username&gt;\",<br \/>\n\"password\": \"&lt;couchbase password&gt;\",<br \/>\n\"query\": \"&lt;couchbase query&gt;\"<br \/>\n}<br \/>\n\/\/JSON or CSV source<br \/>\n{<br \/>\n\"file\": \"&lt;link to file served via http&gt;\"<br \/>\n}<br \/>\n\/\/CSV source<br \/>\n{<br \/>\n\"delimiter\": \"&lt;comma, space, tab or custom character&gt;\"<br \/>\n}<\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h3>Deploying the Observability Dashboard Services<\/h3>\n<p>Use the `docker-compose` file below to bring up all of the required services \u2013 e.g., Dashboard proxy, Grafana, Prometheus, Exporter, Alert manager \u2013 that appear in the architecture diagram above for our observability dashboards. You can install Couchbase on a different host to store your growing high-volume data.<\/p>\n<p>To bring up: `docker-compose up`<\/p>\n<p>Next, visit `https:\/\/host:3000` for the Grafana page.<\/p>\n<p>To bring down: `docker-compose down`<\/p>\n<pre>version: \"2\"\r\nservices:\r\n dashboard:\r\n   restart: unless-stopped\r\n   build: ..\/..\/\r\n   ports:\r\n     - 5001:5000\r\n   environment:\r\n     - GRAFANA_HOST=https:\/\/admin:password@grafana:3000\r\n   volumes:\r\n     - .\/config\/targets.json:\/app\/targets.json\r\n \r\n grafana:\r\n   image: grafana\/grafana:8.0.1\r\n   restart: unless-stopped\r\n   volumes:\r\n     - .\/config\/grafana:\/var\/lib\/grafana\r\n   environment:\r\n     GF_INSTALL_PLUGINS: \"simpod-json-datasource,marcusolsson-csv-datasource,ae3e-plotly-panel\"\r\n     GF_AUTH_ANONYMOUS_ENABLED: \"true\"\r\n     GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: \"ae3e-plotly-panel\"\r\n     GF_RENDERING_SERVER_URL: https:\/\/renderer:8081\/render\r\n     GF_RENDERING_CALLBACK_URL: https:\/\/grafana:3000\/\r\n   ports:\r\n     - 4000:3000\r\n \r\n renderer:\r\n   image: grafana\/grafana-image-renderer:latest\r\n \r\n prometheus:\r\n   restart: unless-stopped\r\n   image: prom\/prometheus\r\n   volumes:\r\n     - .\/config\/prometheus.yml:\/etc\/prometheus\/prometheus.yml\r\n     - .\/config\/alert.rules.yml:\/etc\/prometheus\/alert.rules.yml\r\n \r\n exporter:\r\n   restart: unless-stopped\r\n   build: ..\/..\/exporter\r\n   volumes:\r\n     - .\/config\/queries.json:\/app\/queries.json\r\n \r\n alertmanager:\r\n   restart: unless-stopped\r\n   image: prom\/alertmanager\r\n   ports:\r\n     - 9093:9093\r\n   volumes:\r\n     - .\/config\/alertmanager.yml:\/etc\/alertmanager\/alertmanager.yml\r\n     - .\/config\/alert_templates:\/etc\/alertmanager\/templates\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>The above service reference files content \u2013 or snippets for brevity \u2013 can be found in <a href=\"#implementation\">the implementation section below<\/a>.<\/p>\n<p>Using these tools, you can create a wide variety of dashboards to suit your requirements. We&#8217;ll walk through three types of example dashboards to give you an idea of what&#8217;s possible.<\/p>\n<h2>Example Dashboards: Overview<\/h2>\n<table>\n<tbody>\n<tr>\n<td><strong>#<\/strong><\/td>\n<td><strong>Dashboard<\/strong><\/td>\n<td><strong>Measurements<\/strong><\/td>\n<td><strong>Metrics<\/strong><\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>Functional Regression Testing Cycles dashboards<\/td>\n<td>Trends among functional regression testing cycles at both the build level &amp; component level<\/td>\n<td>total tests, passed, failed, aborts, total time, fresh run time, etc.<\/td>\n<\/tr>\n<tr>\n<td>2<\/td>\n<td>Infra VMs usage dashboards, including Static VMs &amp; Dynamic VMs<\/td>\n<td>Resources utilization &amp; history<\/td>\n<td>active count, available count, compute hours\/max\/created per day, week, month<\/td>\n<\/tr>\n<tr>\n<td>3<\/td>\n<td>Infra VMs Health dashboards, Static Servers, Jenkins Slaves VMs<\/td>\n<td>VM health monitoring, alerts &amp; history tracking of VMs<\/td>\n<td>ssh_fail, pool_os vs real_os, cpu-memory-disk-swap usages, file descriptors, firewall rules, pool_mac_address vs real_mac_address, booted days, total and product processes, installed app versions and services etc.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Dashboard #1: Functional Regression Testing Cycles Dashboard<\/h2>\n<p><strong>Problem<\/strong>: Before we created this dashboard for ourselves, there were no trend graphs on the regression test cycles with metrics like total time taken, pass rate, fresh vs. reruns (e.g., due to infrastructure issues), inconsistent number of aborts and failures, and also no separate component- or module-level trends.<\/p>\n<p><strong>Solution<\/strong>: The plan was to create a run analyzer script that analyzes the test data that is already stored in the Couchbase bucket. After that, we get the time-series data for the last <em>n<\/em> number of builds and targeted metrics for each build.<\/p>\n<p><strong>Dashboard snapshots:<\/strong><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12098\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-1.png\" alt=\"Weekly functional regression testing cycles Grafana dashboard, part 1\" width=\"1000\" height=\"484\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-1.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-1-300x145.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-1-768x372.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-1-20x10.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12099\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-2.png\" alt=\"Weekly functional regression testing cycles Grafana dashboard, part 2\" width=\"1000\" height=\"511\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-2.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-2-300x153.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-2-768x392.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-2-20x10.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-3.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12100\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-3.png\" alt=\"Weekly functional regression testing cycles Grafana dashboard, part 3\" width=\"1000\" height=\"456\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-3.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-3-300x137.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-3-768x350.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-3-20x9.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-4.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12101\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-4.png\" alt=\"Weekly functional regression testing cycles Grafana dashboard, part 4\" width=\"1000\" height=\"453\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-4.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-4-300x136.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-4-768x348.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-4-20x9.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-5.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12102\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-5.png\" alt=\"Weekly functional regression testing cycles Grafana dashboard, part 5\" width=\"1000\" height=\"429\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-5.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-5-300x129.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-5-768x329.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/functional-regression-testing-cycles-Grafana-dashboard-5-20x9.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h2>Dashboard #2: Infrastructure Resources \/ VMs Usage Dashboard<\/h2>\n<p><strong>Problem:<\/strong> Prior to building this dashboard, we had a large number of static and dynamic virtual machines but there was no tracking of how the hardware resources were utilized. We had no insight into metrics such as active VMs used at the time, available count, machine time used, or compute hours on a daily, weekly or monthly basis.<\/p>\n<p><strong>Solution:<\/strong> Our plan was to first collect the data for all the VMs such as dynamically allocating and releasing IPs, exact time creation, and release times, as well as any groupings such as pools, etc. Most of this data already existed in <a href=\"https:\/\/www.couchbase.com\/products\/server\/?ref=blog\" target=\"_blank\" rel=\"noopener\">Couchbase Server<\/a> (managed by the respective service managers). Using the flexibility of <a href=\"https:\/\/query-tutorial.couchbase.com\/tutorial\/?ref=blog\/#1\" target=\"_blank\" rel=\"noopener\">the SQL++ query language<\/a> (aka N1QL), we were able to extract that data into a format suitable for the graphs we wanted to show in this observability dashboard.<\/p>\n<p><strong>Dashboard snapshots<\/strong>:<\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/static-pool-VMs-Grafana-dashboard-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12104\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/static-pool-VMs-Grafana-dashboard-1.png\" alt=\"Static pool VMs Grafana dashboard, part 1\" width=\"1000\" height=\"534\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/static-pool-VMs-Grafana-dashboard-1.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/static-pool-VMs-Grafana-dashboard-1-300x160.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/static-pool-VMs-Grafana-dashboard-1-768x410.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/static-pool-VMs-Grafana-dashboard-1-20x11.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/static-pool-VMs-Grafana-dashboard-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12105\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/static-pool-VMs-Grafana-dashboard-2.png\" alt=\"Static pool VMs Grafana dashboard, part 2\" width=\"1000\" height=\"378\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/static-pool-VMs-Grafana-dashboard-2.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/static-pool-VMs-Grafana-dashboard-2-300x113.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/static-pool-VMs-Grafana-dashboard-2-768x290.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/static-pool-VMs-Grafana-dashboard-2-20x8.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/server-dynamic-VMs-Grafana-dashboard.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12103\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/server-dynamic-VMs-Grafana-dashboard.png\" alt=\"Server dynamic VMs Grafana dashboard\" width=\"1000\" height=\"534\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/server-dynamic-VMs-Grafana-dashboard.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/server-dynamic-VMs-Grafana-dashboard-300x160.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/server-dynamic-VMs-Grafana-dashboard-768x410.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/server-dynamic-VMs-Grafana-dashboard-20x11.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h2>Dashboard #3: Infrastructure VMs Health Dashboard<\/h2>\n<p><strong>Problem:<\/strong> Before we had this dashboard, regression test runs were failing inconsistently, and there were low-hanging issues with the VMs. Some of the issues included SSH Failures, OS mismatches, VM IP switches, too many open files, swap issues, need reboots, duplicate IPs among multiple runs, high memory usage, disk full (`\/` or `\/data`), firewall rules stopping the endpoint connection, slave issues due to high memory, and disk usage were all common. There was no observability dashboard to look at and observe these metrics and also no checks and alerts for the test infrastructure health.<\/p>\n<p><strong>Solution:<\/strong> We decided to create an automatic periodic health check that captures metrics data for the targeted VMs such as `ssh_fail`, `pool_os` vs `real_os`, `cpu-memory-disk-swap` usages, file descriptors, firewall rules, `pool_mac_address` vs `real_mac_address`, booted days, total and Couchbase processes, installed Couchbase version and services. (In sum, we captured ~50 metrics). These metrics are exposed as a Prometheus endpoint that is displayed in Grafana, and the information is also saved in Couchbase for future data analysis. Alerts were also created to monitor the key health metrics for issues to allow for quick intervention and finally achieve increased stability of the test runs.<\/p>\n<p><strong>Dashboard snapshots:<\/strong><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/VM-health-Grafana-dashboard-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12106\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/VM-health-Grafana-dashboard-1.png\" alt=\"VM health Grafana dashboard, part 1\" width=\"1000\" height=\"536\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/VM-health-Grafana-dashboard-1.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/VM-health-Grafana-dashboard-1-300x161.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/VM-health-Grafana-dashboard-1-768x412.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/VM-health-Grafana-dashboard-1-20x11.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/VM-health-Grafana-dashboard-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-12107\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/09\/VM-health-Grafana-dashboard-2.png\" alt=\"VM health Grafana dashboard, part 2\" width=\"1000\" height=\"526\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/VM-health-Grafana-dashboard-2.png 1000w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/VM-health-Grafana-dashboard-2-300x158.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/VM-health-Grafana-dashboard-2-768x404.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/VM-health-Grafana-dashboard-2-20x11.png 20w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h2 id=\"implementation\">Implementation<\/h2>\n<p>So far, you&#8217;ve seen the high-level architecture of the observability dashboards, what services are required, what kind of dashboards you might need, and also how to deploy these services. Now, it&#8217;s time to look at some implementation details.<\/p>\n<p>Our first stop is the collection and storage of metrics and the data visualization of the dashboards. Most of the data storage and display steps are similar for all use cases, but the metrics data collection depends on which metrics you choose to target.<\/p>\n<h3>How to Get Health Data for Your Dashboards<\/h3>\n<p>For the infrastructure monitoring use case, you have to collect various health metrics from hundreds of VMs to create a stable infrastructure.<\/p>\n<p>For this step, we created a Python script that does the SSH connection to the VMs in parallel (multiprocessing pool) and collects the required data. In our case, we also have a Jenkins job that periodically runs this script and collects the health data (CSV), and then saves it to the Couchbase database.<\/p>\n<p>The reason we created this custom script \u2013 rather than the readily available node exporter provided by Prometheus \u2013 is that some of the required metrics were not supported. In addition, this solution was simpler than deploying and maintaining the new software on 1000+ servers. The code snippet below shows some of the checks being done at the VM level.<\/p>\n<pre>def check_vm(os_name, host):    \r\n        client = SSHClient()\r\n        client.set_missing_host_key_policy(AutoAddPolicy())\r\n        ...\r\n        cpus = get_cpuinfo(client)\r\n        meminfo = get_meminfo(client)\r\n        diskinfo = get_diskinfo(client)\r\n        uptime = get_uptime(client)\r\n        ...\r\n    return ssh_status, '', ssh_resp_time, real_os_version, cpus, meminfo, diskinfo, uptime, uptime_days, systime, cpu_load, cpu_total_processes, fdinfo, \\\r\n        iptables_rules_count, mac_address, swapinfo, cb_processes, cb_version, cb_running_serv, cb_ind_serv\r\n\r\ndef get_cpuinfo(ssh_client):\r\n    return ssh_command(ssh_client,\"cat \/proc\/cpuinfo  |egrep processor |wc -l\")\r\n\r\ndef get_meminfo(ssh_client):\r\n    return ssh_command(ssh_client,\"cat \/proc\/meminfo |egrep Mem |cut -f2- -d':'|sed 's\/ \/\/g'|xargs|sed 's\/ \/,\/g'|sed 's\/kB\/\/g'\")\r\n\r\ndef get_diskinfo(ssh_client):\r\n    return ssh_command(ssh_client,\"df -ml --output=size,used,avail,pcent \/ |tail -1 |sed 's\/ \\+\/,\/g'|cut -f2- -d','|sed 's\/%\/\/g'\")\r\n\r\ndef get_uptime(ssh_client):\r\n    return ssh_command(ssh_client, \"uptime -s\")\r\n\r\ndef get_cpu_users_load_avg(ssh_client):\r\n    return ssh_command(ssh_client, \"uptime |rev|cut -f1-4 -d','|rev|sed 's\/load average:\/\/g'|sed 's\/ \\+\/\/g'|sed 's\/users,\/,\/g'|sed 's\/user,\/,\/g'\")\r\n\r\ndef get_file_descriptors(ssh_client):\r\n    return ssh_command(ssh_client, \"echo $(cat \/proc\/sys\/fs\/file-nr;ulimit -n)|sed 's\/ \/,\/g'\")\r\n\r\ndef get_mac_address(ssh_client):\r\n    return ssh_command(ssh_client, \"ifconfig `ip link show | egrep eth[0-9]: -A 1 |tail -2 |xargs|cut -f2 -d' '|sed 's\/:\/\/g'`|egrep ether |xargs|cut -f2 -d' '\")\r\n\r\ndef get_mac_address_ip(ssh_client):\r\n    return ssh_command(ssh_client, \"ip a show `ip link show | egrep eth[0-9]: -A 1 |tail -2 |xargs|cut -f2 -d' '|sed 's\/:\/\/g'`|egrep ether |xargs|cut -f2 -d' '\")\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>The below code snippet shows you how to connect to Couchbase using Python SDK 3.x with key-value operations, getting a document, or saving a document in the database.<\/p>\n<pre>        try:\r\n            self.cb_cluster = Cluster(\"couchbase:\/\/\"+self.cb_host, ClusterOptions(PasswordAuthenticator(self.cb_username, self.cb_userpassword), \\\r\ntimeout_options=ClusterTimeoutOptions(kv_timeout=timedelta(seconds=10))))\r\n            self.cb_b = self.cb_cluster.bucket(self.cb_bucket)\r\n            self.cb = self.cb_b.default_collection()\r\n        except Exception as e:\r\n            print('Connection Failed: %s ' % self.cb_host)\r\n            print(e)\r\n\r\n    def get_doc(self, doc_key, retries=3):\r\n        # ..\r\n         return self.cb.get(doc_key)\r\n            \r\n    def save_doc(self, doc_key, doc_value, retries=3):\r\n        #...\r\n        self.cb.upsert(doc_key, doc_value)\r\n        #...        \r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h3>Implementing the Dashboard Proxy Service<\/h3>\n<p>For the tests observability use cases, the data is in a Jenkins artifact URL and also in Couchbase Server. To bridge these multiple data sources together (CSV, DB), we created a proxy API service that would accept requests from Grafana and return the data format understood by Grafana.<\/p>\n<p>The below code snippets give the implementation and service preparation details.<\/p>\n<p>`dashboard.py`<\/p>\n<pre># Dashboard API service\r\n@app.route(\"\/query\", methods=['POST'])\r\ndef query():\r\n\t\"\"\"\r\n\t\/query responds to a Grafana data request and is \r\nformatted as either data points for time series data\r\n\tor rows and columns for tabular data\r\n\t\"\"\"\r\n\tfor target in request.json['targets']:\r\n   \t \tdata_type = target['type']\r\n   \t \r\n    \tif data_type == \"timeseries\":\r\n   \t\t datapoints = calculate_datapoints(target)\r\n    \telif data_type == \"table\":\r\n   \t\t datapoints = calculate_rows_and_columns(target)   \t \r\n\t...\r\n    \r\ndef calculate_datapoints(target):\r\n\t\"\"\"\r\n\tReturns data in a time series format\r\n\tdatapoints is formatted as a list of 2 item tuples in the format [value, timestamp]\r\n\t\"\"\"\r\n\t...\r\n    \r\n\tif target['source'] == \"couchbase\":\r\n   \t\t...\r\n\telif target['source'] == \"json\":\r\n \t\t ...\r\n\telif target['source'] == \"csv\":\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>`Dockerfile`<\/p>\n<pre>FROM ubuntu:latest\r\nENV DEBIAN_FRONTEND \"noninteractive\"\r\nRUN apt-get update -y &amp;&amp; apt-get install -y python3-dev python3-pip python3-setuptools cmake build-essential\r\nRUN mkdir \/app\r\nCOPY .\/requirements.txt \/app\r\nWORKDIR \/app\r\nRUN pip3 install -r requirements.txt\r\nCOPY .\/dashboard.py \/app\r\nCOPY .\/entrypoint.sh \/app\r\n\r\nENTRYPOINT [\".\/entrypoint.sh\"]\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>`entrypoint.sh`<\/p>\n<pre>#!\/bin\/bash\r\npython3 dashboard.py $GRAFANA_HOST\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>`requirements.txt`<\/p>\n<pre>couchbase==3.0.7\r\nFlask==1.1.2\r\nrequests==2.24.0\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h3>How to Get the Tabular Data in Grafana<\/h3>\n<p>Grafana is a great tool for viewing time-series data. However, sometimes you want to show some non-time-series data in the same interface.<\/p>\n<p>We achieved this goal using the <a href=\"https:\/\/grafana.com\/grafana\/plugins\/ae3e-plotly-panel\/?ref=hello-from-couchbase\" target=\"_blank\" rel=\"noopener\">Plotly plugin<\/a> which is a JavaScript graphing library. Our main use case was to illustrate trends across a variety of important metrics for our weekly regression testing runs. Some of the most important metrics we wanted to track were pass rate, the number of tests, aborted jobs, and total time is taken. Since the release of Grafana 8, there is limited support for bar graphs. At the time of writing, the bar graph functionality is still in beta and doesn\u2019t offer all of the features we require, such as stacking.<\/p>\n<p>Our goal was to support generic CSV\/JSON files or a Couchbase SQL++ query and view the data as a table in Grafana. For maximum portability, we wanted to have a single file that would define both the data sources and Grafana template layout together.<\/p>\n<p>For the tabular data to be displayed, below are the two viable options.<\/p>\n<ol>\n<li>Write a UI plugin for Grafana<\/li>\n<li>Provide a JSON proxy using the <a href=\"https:\/\/grafana.com\/grafana\/plugins\/grafana-simple-json-datasource\/?ref=hello-from-couchbase\" target=\"_blank\" rel=\"noopener\">JSON datasource plugin<\/a><\/li>\n<\/ol>\n<p>We chose option 2 for our implementation, since it seemed simpler than trying to learn the Grafana plugin tools and creating a separate UI plugin for the configuration.<\/p>\n<p>Note that since finishing this project, <a href=\"https:\/\/grafana.com\/grafana\/plugins\/marcusolsson-csv-datasource\/?ref=hello-from-couchbase\" target=\"_blank\" rel=\"noopener\">a new plugin<\/a> has been released that allows you to add CSV data to Grafana directly. If viewing tabular data from a CSV is your only requirement, then this plugin is a good solution.<\/p>\n<h3>Implementing the Prometheus Service<\/h3>\n<p>`prometheus.yml`<\/p>\n<pre># Prometheus global config\r\nglobal:\r\n  scrape_interval: 1m # Set the scrape interval to every 15 seconds. Default is every 1 minute.\r\n  scrape_timeout: 30s\r\n# Alertmanager configuration\r\nalerting:\r\n  alertmanagers:\r\n    - static_configs:\r\n        - targets:\r\n            - alertmanager:9093\r\nrule_files:\r\n  - \"alert.rules.yml\"\r\n  - job_name: \"prometheus\"\r\n    static_configs:\r\n      - targets: [\"localhost:9090\"]\r\n  - job_name: \"automation_exporter\"\r\n    static_configs:\r\n      - targets: [\"exporter:8000\"]\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>`alert.rules.yml`<\/p>\n<pre>groups:\r\n  - name: alert.rules\r\n    rules:\r\n      - alert: PoolVMDown\r\n        expr: vm_health_ssh_status == 0\r\n        for: 1m\r\n        annotations:\r\n          title: \"Server Pool VM {{ $labels.ipaddr }} SSH Failed\"\r\n          description: \"{{ $labels.ipaddr }} SSH failed with error: {{ $labels.ssh_error }}.\"\r\n        labels:\r\n          severity: \"critical\"\r\n      - alert: PoolVMHighDiskUsage\r\n        expr: disk_usage &gt;= 95\r\n        for: 1m\r\n        annotations:\r\n          title: \"Server Pool VM {{ $labels.ipaddr }} high disk usage\"\r\n          description: \"{{ $labels.ipaddr }} has disk usage of {{ $value }}%\"\r\n        labels:\r\n          severity: \"critical\"\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h3>How to Get Custom Metrics through the Prometheus Exporter<\/h3>\n<p>Many cloud-native services integrate directly with Prometheus to allow centralized metrics collection for all of your services.<\/p>\n<p>We wanted to see how we could utilize this technology to monitor our existing infrastructure. If you have services that don\u2019t directly expose a Prometheus metrics endpoint, the way to solve it is to use an exporter. In fact, there is even a Couchbase exporter to expose all of the important metrics from your cluster. (Note: In <a href=\"https:\/\/www.couchbase.com\/blog\/couchbase-server-7-0-release\/?ref=blog\" target=\"_blank\" rel=\"noopener\">Couchbase Server 7.0<\/a>, a Prometheus endpoint is directly available, and internally, Couchbase 7 uses Prometheus for server stats collection and management to service <a href=\"https:\/\/docs.couchbase.com\/server\/current\/manage\/manage-ui\/manage-ui.html?ref=blog\" target=\"_blank\" rel=\"noopener\">the web UI<\/a>).<\/p>\n<p>While creating our observability dashboards, we had various data stored in JSON files, in CSV files, and in Couchbase buckets. We wanted a way to expose all of this data and show it in Grafana both in tabular format and as time-series data using Prometheus.<\/p>\n<p>Prometheus expects a simple line-based text output. Here&#8217;s an example from our server pool monitoring:<\/p>\n<pre>available_vms{pool=\"12hrreg\"} 1\r\navailable_vms{pool=\"regression\"} 16\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>Let&#8217;s take a closer look at how to implement data sources from both CSV files and from Couchbase directly.<\/p>\n<h4>CSV Files as Your Data Source<\/h4>\n<p>Each time Prometheus polls the endpoint, we fetch the CSV, and for each column, we expose a metric, appending labels for multiple rows if a label is supplied in the config.<\/p>\n<p>For the above example, the CSV looks like:<\/p>\n<pre>pool,available_count\r\n12hrreg,1\r\nregression,16\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h4>Couchbase as Your Data Source<\/h4>\n<p>Each time Prometheus polls the endpoint, we execute the SQL++ queries defined in the config, and for each query, we expose a metric, appending labels for multiple rows if a label is supplied in the config.<\/p>\n<p>Below is an example SQL++ response that produces the above metrics:<\/p>\n<pre class=\"\">[{\r\n\t\u201cpool\u201d, \u201c12hrreg\u201d,\r\n\t\u201ccount\u201d: 1\r\n },\r\n {\r\n\t\u201cpool\u201d, \u201cregression\u201d,\r\n\t\u201ccount\u201d: 16\r\n }]\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>This exporter Python service exposes a `\/metrics` endpoint to be used in Prometheus. These metrics are defined in `queries.json` and define which queries and CSV columns should be exposed as metrics. See the below JSON snippet (reduced for brevity) as an example.<\/p>\n<p>`queries.json`<\/p>\n<pre>{\r\n  \"clusters\": {\r\n    \"static_vms\": {\r\n      \"host\": \"&lt;ip-address&gt;\",\r\n      \"username\": \"Administrator\",\r\n      \"password\": \"xxxx\"\r\n    },\r\n    ...\r\n  },\r\n  \"queries\": [\r\n    {\r\n      \"name\": \"available_vms\",\r\n      \"cluster\": \"static_vms\",\r\n      \"query\": \"SELECT poolId as `pool`, COUNT(*) AS count FROM (SELECT poolId FROM `QE-server-pool` WHERE IS_ARRAY(poolId)=FALSE and state='available' UNION ALL SELECT poolId FROM `QE-server-pool` UNNEST poolId where `QE-server-pool`.state = 'available'  ) AS pools group by poolId\",\r\n      \"description\": \"Available VMs for each server pool\",\r\n      \"value_key\": \"count\",\r\n      \"labels\": [\"pool\"]\r\n    },\r\n    ...\r\n  ],\r\n  \"csvs\": {\r\n    \"vm_health\": \"https:\/\/&lt;jenkins-host-job-url&gt;\/lastSuccessfulBuild\/artifact\/vm_health_info.csv\/\",\r\n   ...\r\n  },\r\n  \"columns\": [\r\n    {\r\n      \"name\": \"memory_usage\",\r\n      \"csv\": \"vm_health\",\r\n      \"description\": \"Memory usage\",\r\n      \"column\": \"memory_use(%)\",\r\n      \"labels\": [\"ipaddr\"]\r\n    },\r\n    {\r\n      \"name\": \"disk_usage\",\r\n      \"csv\": \"vm_health\",\r\n      \"description\": \"Disk usage\",\r\n      \"column\": \"disk_use%\",\r\n      \"labels\": [\"ipaddr\"]\r\n    },\r\n    {\r\n      \"name\": \"cpu_load_avg_5mins\",\r\n      \"csv\": \"vm_health\",\r\n      \"description\": \"CPU load average (5mins)\",\r\n      \"column\": \"cpu_load_avg_5mins\",\r\n      \"labels\": [\"ipaddr\"]\r\n    },\r\n    {\r\n      \"name\": \"vm_health_ssh_status\",\r\n      \"csv\": \"vm_health\",\r\n      \"description\": \"SSH Status\",\r\n      \"column\": \"ssh_status\",\r\n      \"labels\": [\"ipaddr\", \"ssh_error\", \"pool_state\", \"couchbase_version\", \"pool_ids\"]\r\n    },\r\n    ...\r\n  ]\r\n}\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>`exporter.py`<\/p>\n<pre>for options in settings['queries'] + settings[\"columns\"]:\r\n    log.info(\"Registered metrics collection for {}\".format(options['name']))\r\n\r\ndef get_labels(row, options):\r\n    rename_map = options.get(\"rename\", {})\r\n    return [\"{}=\\\"{}\\\"\".format(rename_map[label] if label in rename_map else label, row[label]) for label in options[\"labels\"]]\r\n\r\ndef collect_cb(clusters, metrics, options):\r\n    rows = clusters[options[\"cluster\"]].query(options[\"query\"]).rows()\r\n    for row in rows:\r\n        if len(options[\"labels\"]) &gt; 0:\r\n            labels = get_labels(row, options)\r\n            metrics.append(\"{}{{{}}} {}\".format(\r\n                options[\"name\"], \",\".join(labels), row[options[\"value_key\"]]))\r\n        else:\r\n            metrics.append(\"{} {}\".format(\r\n                options[\"name\"], row[options[\"value_key\"]]))\r\n\r\ndef collect_csv(metrics, options)\r\n    csvfile = requests.get(csvs[options[\"csv\"]]).text.splitlines()\r\n    reader = DictReader(csvfile)\r\n    for row in reader:\r\n        if options[\"column\"] not in row or row[options[\"column\"]] == \"\":\r\n            continue\r\n        if len(options[\"labels\"]) &gt; 0:\r\n            labels = get_labels(row, options)\r\n            metrics.append(\"{}{{{}}} {}\".format(\r\n                options[\"name\"], \",\".join(labels), row[options[\"column\"]]))\r\n        else:\r\n            metrics.append(\"{} {}\".format(\r\n                options[\"name\"], row[options[\"column\"]]))\r\n\r\n@app.route(\"\/metrics\")\r\ndef metrics():\r\n    metrics = []\r\n    clusters = {}\r\n    for [cluster_name, options] in settings['clusters'].items():\r\n        if cluster_name not in clusters:\r\n            try:\r\n                clusters[cluster_name] = Cluster('couchbase:\/\/'+options['host'],\r\n                                                ClusterOptions(\r\n                    PasswordAuthenticator(options['username'], options['password'])))\r\n            except Exception as e:\r\n                log.warning(\"Couldn't connect to cluster {}\".format(e))\r\n            log.debug(\"Connected to {}\".format(options['host']))\r\n    for options in settings[\"queries\"] + settings[\"columns\"]:\r\n        log.debug(\"Collecting metrics for {}\".format(options[\"name\"]))\r\n        try:\r\n            if \"cluster\" in options:\r\n                collect_cb(clusters, metrics, options)\r\n            elif \"csv\" in options:\r\n                collect_csv(metrics, options)\r\n            else:\r\n                raise Exception(\"Invalid type\")\r\n        except Exception as e:\r\n            log.warning(\"Error while collecting {}: {}\".format(\r\n                options[\"name\"], e))\r\n    return Response(\"\\n\".join(metrics), mimetype=\"text\/plain\")\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h3>Implementing the Alert Manager Service<\/h3>\n<p>Prometheus also supports alerting where it tracks specific metrics for you over time. If that metric starts returning results, it will trigger an alert.<\/p>\n<p>For the example above you could add an alert for when the regression pool has no servers available. If you specify the query as `available_vms{pool=&#8221;regression&#8221;} == 0` that will return a series when there are `0` available. Once added, Prometheus tracks this for you (default is every minute). If that is all you do, you can visit the Prometheus UI and the alerts tab will show you which alerts are firing.<\/p>\n<p>With the Alert Manager, you can take this a step further and connect communications services so that Prometheus can alert you via email or a Slack channel, for example, when an alert fires. This means you can be informed immediately via your preferred method when something goes wrong. At Couchbase, we set up alerts to be notified of high disk usage on servers as well as when servers could not be reached via SSH. See the example below:<\/p>\n<p>`alertmanager.yml`<\/p>\n<pre>global:\r\n  resolve_timeout: 1m\r\n  smtp_from: qa@couchbase.com\r\n  smtp_smarthost: mail-com.mail.protection.outlook.com:25\r\n\r\nroute:\r\n  group_by: [\"alertname\"]\r\n  group_wait: 10s\r\n  group_interval: 10s\r\n  repeat_interval: 24h\r\n  receiver: \"infra-email\"\r\n  matchers:\r\n   - alertname =~ PoolVMDown|PoolVMOSMismatch|PoolVMHighDiskUsage|SlaveVMHighDiskUsage|SlaveVMHighDiskUsageData\r\n\r\nreceivers:\r\n  - name: \"infra-email\"\r\n    email_configs:\r\n      - to: jake.rawsthorne@couchbase.com,jagadesh.munta@couchbase.com\r\n<\/pre>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<h2>Conclusion<\/h2>\n<p>In conclusion, we hope you can learn from our experience of creating observability dashboards that help you hone in on the metrics that matter most in your implementation or use case with the power of data visualization.<\/p>\n<p>In our case, this effort allowed us to find server infrastructure and test stability issues. Building dashboards also reduced the number of failed tests as well as the total regression time for multiple product releases.<\/p>\n<p>We hope this walkthrough helps you build better observability dashboards in the future.<\/p>\n<p><em>Also, we&#8217;d like to extend special thanks to Raju and the QE team for their feedback on improving the targeted metrics.<\/em><\/p>\n<div class=\"wp-block-spacer\" style=\"height: 30px\" aria-hidden=\"true\"><\/div>\n<div style=\"text-align: center\"><strong>Building something awesome?&lt;br\/ &gt;<a href=\"https:\/\/www.couchbase.com\/downloads\/?ref=blog\" target=\"_blank\" rel=\"noopener\">Why not build it on Couchbase?<\/a><\/strong><\/div>\n<div class=\"wp-block-spacer\" style=\"height: 15px\" aria-hidden=\"true\"><\/div>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>You&#8217;ve certainly heard it before: &#8220;What gets measured gets done.&#8221; It&#8217;s true: what you observe and measure is what you can improve. The key to any improvement is to first identify what to measure and then collect the related metrics. [&hellip;]<\/p>\n","protected":false},"author":78989,"featured_media":12083,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1815,2381,1816,9327,2334,9139,1812,2201],"tags":[2250,2384,1261,2313,2383],"ppma_author":[9435],"class_list":["post-11527","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-best-practices-and-tutorials","category-community","category-couchbase-server","category-javascript","category-monitoring","category-python","category-n1ql-query","category-tools-sdks","tag-data-visualization","tag-grafana","tag-json","tag-key-value","tag-prometheus"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.8 (Yoast SEO v25.8) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Observability Dashboards: Prometheus, Grafana + Couchbase<\/title>\n<meta name=\"description\" content=\"In this blog, we take you through the process you need to follow to build your own observability dashboard using Prometheus, Grafana and Couchbase.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Build Observability Dashboards with Prometheus, Grafana &amp; Couchbase\" \/>\n<meta property=\"og:description\" content=\"In this blog, we take you through the process you need to follow to build your own observability dashboard using Prometheus, Grafana and Couchbase.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/\" \/>\n<meta property=\"og:site_name\" content=\"The Couchbase Blog\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-23T07:00:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-14T04:25:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus-social.png\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"418\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Jake Rawsthorne &amp; Jagadesh Munta\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus-social.png\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jake Rawsthorne &amp; Jagadesh Munta\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"14 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"TechArticle\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/\"},\"author\":{\"name\":\"Jake &amp; Jagadesh Shared byline\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/777a6aa1a9bad7851380e8095da38ce1\"},\"headline\":\"How to Build Observability Dashboards with Prometheus, Grafana &amp; Couchbase\",\"datePublished\":\"2021-09-23T07:00:10+00:00\",\"dateModified\":\"2025-06-14T04:25:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/\"},\"wordCount\":2667,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus.png\",\"keywords\":[\"data visualization\",\"Grafana\",\"JSON\",\"key value\",\"Prometheus\"],\"articleSection\":[\"Best Practices and Tutorials\",\"Community\",\"Couchbase Server\",\"JavaScript\",\"Monitoring\",\"Python\",\"SQL++ \/ N1QL Query\",\"Tools &amp; SDKs\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/\",\"name\":\"Observability Dashboards: Prometheus, Grafana + Couchbase\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus.png\",\"datePublished\":\"2021-09-23T07:00:10+00:00\",\"dateModified\":\"2025-06-14T04:25:14+00:00\",\"description\":\"In this blog, we take you through the process you need to follow to build your own observability dashboard using Prometheus, Grafana and Couchbase.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#primaryimage\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus.png\",\"width\":1200,\"height\":628,\"caption\":\"Learn how to make observability dashboards with Couchbase, Grafana & Prometheus for time-series data\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.couchbase.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Build Observability Dashboards with Prometheus, Grafana &amp; Couchbase\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"name\":\"The Couchbase Blog\",\"description\":\"Couchbase, the NoSQL Database\",\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\",\"name\":\"The Couchbase Blog\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"width\":218,\"height\":34,\"caption\":\"The Couchbase Blog\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/777a6aa1a9bad7851380e8095da38ce1\",\"name\":\"Jake &amp; Jagadesh Shared byline\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/6b5c82568cb218e8a06486a2bd0cc6e7\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/jake-rawsthorne-jagadesh-munta-couchbase.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/jake-rawsthorne-jagadesh-munta-couchbase.png\",\"caption\":\"Jake &amp; Jagadesh Shared byline\"},\"url\":\"https:\/\/www.couchbase.com\/blog\/author\/jake-and-jagadesh\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Observability Dashboards: Prometheus, Grafana + Couchbase","description":"In this blog, we take you through the process you need to follow to build your own observability dashboard using Prometheus, Grafana and Couchbase.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/","og_locale":"en_US","og_type":"article","og_title":"How to Build Observability Dashboards with Prometheus, Grafana &amp; Couchbase","og_description":"In this blog, we take you through the process you need to follow to build your own observability dashboard using Prometheus, Grafana and Couchbase.","og_url":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/","og_site_name":"The Couchbase Blog","article_published_time":"2021-09-23T07:00:10+00:00","article_modified_time":"2025-06-14T04:25:14+00:00","og_image":[{"width":800,"height":418,"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus-social.png","type":"image\/png"}],"author":"Jake Rawsthorne &amp; Jagadesh Munta","twitter_card":"summary_large_image","twitter_image":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus-social.png","twitter_misc":{"Written by":"Jake Rawsthorne &amp; Jagadesh Munta","Est. reading time":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"TechArticle","@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#article","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/"},"author":{"name":"Jake &amp; Jagadesh Shared byline","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/777a6aa1a9bad7851380e8095da38ce1"},"headline":"How to Build Observability Dashboards with Prometheus, Grafana &amp; Couchbase","datePublished":"2021-09-23T07:00:10+00:00","dateModified":"2025-06-14T04:25:14+00:00","mainEntityOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/"},"wordCount":2667,"commentCount":1,"publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus.png","keywords":["data visualization","Grafana","JSON","key value","Prometheus"],"articleSection":["Best Practices and Tutorials","Community","Couchbase Server","JavaScript","Monitoring","Python","SQL++ \/ N1QL Query","Tools &amp; SDKs"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/","url":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/","name":"Observability Dashboards: Prometheus, Grafana + Couchbase","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#primaryimage"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus.png","datePublished":"2021-09-23T07:00:10+00:00","dateModified":"2025-06-14T04:25:14+00:00","description":"In this blog, we take you through the process you need to follow to build your own observability dashboard using Prometheus, Grafana and Couchbase.","breadcrumb":{"@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#primaryimage","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/how-to-make-observability-dashboard-couchbase-grafana-prometheus.png","width":1200,"height":628,"caption":"Learn how to make observability dashboards with Couchbase, Grafana & Prometheus for time-series data"},{"@type":"BreadcrumbList","@id":"https:\/\/www.couchbase.com\/blog\/how-to-build-observability-dashboards-prometheus-grafana-couchbase\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.couchbase.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How to Build Observability Dashboards with Prometheus, Grafana &amp; Couchbase"}]},{"@type":"WebSite","@id":"https:\/\/www.couchbase.com\/blog\/#website","url":"https:\/\/www.couchbase.com\/blog\/","name":"The Couchbase Blog","description":"Couchbase, the NoSQL Database","publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.couchbase.com\/blog\/#organization","name":"The Couchbase Blog","url":"https:\/\/www.couchbase.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","width":218,"height":34,"caption":"The Couchbase Blog"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/777a6aa1a9bad7851380e8095da38ce1","name":"Jake &amp; Jagadesh Shared byline","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/6b5c82568cb218e8a06486a2bd0cc6e7","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/jake-rawsthorne-jagadesh-munta-couchbase.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/jake-rawsthorne-jagadesh-munta-couchbase.png","caption":"Jake &amp; Jagadesh Shared byline"},"url":"https:\/\/www.couchbase.com\/blog\/author\/jake-and-jagadesh\/"}]}},"authors":[{"term_id":9435,"user_id":78989,"is_guest":0,"slug":"jake-and-jagadesh","display_name":"Jake Rawsthorne &amp; Jagadesh Munta","avatar_url":{"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/jake-rawsthorne-jagadesh-munta-couchbase.png","url2x":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2021\/09\/jake-rawsthorne-jagadesh-munta-couchbase.png"},"author_category":"","last_name":"& Jagadesh Munta","first_name":"Jake Rawsthorne","job_title":"","user_url":"","description":"Jake Rawsthorne and Jagadesh Munta are software engineers at Couchbase."}],"_links":{"self":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/11527","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/users\/78989"}],"replies":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/comments?post=11527"}],"version-history":[{"count":0,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/11527\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media\/12083"}],"wp:attachment":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media?parent=11527"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/categories?post=11527"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/tags?post=11527"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=11527"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}