{"id":14996,"date":"2023-10-30T12:43:23","date_gmt":"2023-10-30T19:43:23","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=14996"},"modified":"2023-12-18T12:31:06","modified_gmt":"2023-12-18T20:31:06","slug":"scraping-database-metrics-from-couchbase-capella-with-prometheus","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/scraping-database-metrics-from-couchbase-capella-with-prometheus\/","title":{"rendered":"Scraping Database Metrics from Couchbase Capella with Prometheus"},"content":{"rendered":"

In this blog post, the first in a series, we’re going to show you how to set up a Prometheus server and connect it to your Couchbase Capella Database in order to collect metrics.<\/span><\/p>\n

What is Prometheus?<\/span><\/h2>\n

Prometheus<\/span><\/a> is a very popular open-source systems monitoring and alerting toolkit, with a very active developer and user community. It was originally built at <\/span>SoundCloud<\/span><\/a>, but is now a standalone open source project, maintained independently of any company, and it joined the <\/span>Cloud Native Computing Foundation<\/span><\/a> in 2016 as the second hosted project, after <\/span>Kubernetes<\/span><\/a>.<\/span><\/p>\n

What is Couchbase Capella?<\/span><\/h2>\n

Couchbase Capella<\/span><\/a> is our fully-managed Database as a Service (DBaaS) offering, the easiest and fastest way to begin with Couchbase and to eliminate ongoing database management efforts. Your Capella Database contains a native Prometheus scrape target, allowing you to connect your Prometheus (or Prometheus-compatible) monitoring systems to collect metrics.<\/span><\/p>\n

Prerequisites<\/span><\/h2>\n

To follow the steps in this guide, you’ll need a working Docker installation (which we’ll use to deploy Prometheus), a Capella Database, and some common shell utilities.<\/span><\/p>\n

Note: It is outside of the scope of this post to show how to install and configure <\/span><\/i>Docker<\/span><\/i><\/a>. There are many tutorials out there, including Docker’s own <\/span><\/i>Get Started with Docker<\/span><\/i><\/a>. We’ll assume at this point that you have successfully installed and tested it (with e.g. <\/span><\/i>Hello, World!<\/span><\/i><\/a>).<\/span><\/i><\/p>\n

At the time of writing, the latest version of Prometheus was 2.46. The steps here were carried out using Rancher Desktop 1.9.1, with the Container Engine set to dockerd (moby), and using the provided docker CLI. The expectation is that the steps should work as is on any current Docker installation, or an equivalent stack such as containerd with the nerdctl CLI.<\/span><\/i><\/p>\n

Lastly, the commands shared here are suitable for short-lived, standalone development environments – you should of course follow your local best practices for networking and security where appropriate.<\/span><\/i><\/p>\n

Running a Prometheus Server in Docker<\/span><\/h2>\n

Let’s just jump right in and run the latest Prometheus container (<\/span>prom\/prometheus<\/span><\/a>) with its default settings.<\/span><\/p>\n

The <\/span>docker run<\/span> command has many options, but for this example we’ll just use a <\/span>-p<\/em> \/ –publish<\/em><\/span> to expose the Prometheus port (which defaults to 9090<\/em>) so that we can access it from our local browser, and an <\/span>–rm<\/span><\/em> to make sure that the container is cleaned up when it exits.<\/span><\/p>\n

By default the running container will remain “attached”, streaming the logs to Standard Output (in this case, your terminal). Let’s do that for the first run so we can make sure it starts up correctly:<\/span><\/p>\n

docker run --rm --publish 9090:9090 prom\/prometheus<\/pre>\n
Unless you see any errors in the console output, you should now have a Prometheus server running at <\/span>https:\/\/localhost:9090\/<\/em><\/span><\/a>. Open it up in a browser window, and you should see something like this:<\/span><\/p>\n
\"\"<\/a><\/p>\n
Now go back to your terminal window and hit Ctrl+C<\/em> to kill the running container, then we’ll start it again, but this time using a <\/span>-d<\/em> \/ –detach<\/em><\/span> to background it and free up our terminal:<\/span><\/p>\n
docker run --detach --rm --publish 9090:9090 prom\/prometheus<\/pre>\n
On success, Docker will print out the Container ID, and you can also find it with <\/span>docker ps<\/em>.<\/span>\u00a0As you’ll need it for some of the later steps anyway, let’s filter (with <\/span>-f<\/em> \/ –filter<\/em><\/span>) the output of <\/span>docker ps<\/span><\/em> to show just the container ID that we want, based on the network ports that we’re using (in this case we’re publishing port 9090<\/em>):<\/span><\/p>\n
docker ps --filter publish=9090<\/pre>\n
We can add a <\/span>-q<\/em> \/ –quiet<\/em><\/span> to <\/span>only<\/span><\/i> output the Container ID, and use the result in another command. For example, to view the logs of the Prometheus container you just started:<\/span><\/p>\n
docker logs $(docker ps --quiet --filter publish=9090)<\/pre>\n
That command will output something similar to when you started the Prometheus server the first time, when the container was still attached to the terminal. You can use the same technique with (for example) <\/span>docker exec<\/span><\/em> (if you want to start a shell in a running container), or <\/span>docker kill<\/span><\/em> (to send a signal to the container).<\/span><\/p>\n
Let’s take a deeper look at the Prometheus server you have running on <\/span>https:\/\/localhost:9090\/<\/span><\/a><\/em>. If you navigate to Status -> Targets<\/strong>, you can see that there is already a job running:<\/span><\/p>\n
\"\"<\/a><\/span><\/p>\n
This Prometheus server has been configured to scrape <\/span>itself<\/span><\/i> as a target, which you can confirm by viewing its configuration (Status -> Configuration<\/strong>):<\/span><\/p>\n
\"\"<\/a><\/p>\n
You can see in the above that there is a job named <\/span>prometheus<\/span><\/em>, which is targeting <\/span>localhost:9090\/metrics<\/span><\/em> (<\/span>targets + metrics_path<\/span><\/em>). See <\/span>Configuring Prometheus to monitor itself<\/span><\/a> for a sample configuration.<\/span><\/p>\n
Before we go back to our terminal, let’s take a quick look at metrics being scraped in the above job. If you click on Graph, you’ll find the <\/span>Expression Browser<\/span><\/a>, which allows you to enter PromQL expressions (see <\/span>Querying Prometheus<\/span><\/a> for an overview), and view the results.<\/span><\/p>\n
Note the Expression Browser<\/strong> is only meant for adhoc usage – the recommendation is to use <\/span>Grafana<\/span><\/a> as a fully-fledged graphing solution, and we’ll look at that in a future post. In the meantime, as a quick example, this is the number of time series currently stored in its own database (<\/span>prometheus_tsdb_head_series<\/span><\/em>):<\/span><\/p>\n
\"\"<\/a><\/p>\n
If we want to add our Capella Database as a Scrape Job in our Prometheus server, this means we need to make changes to the configuration we saw above, but the configuration is baked into our container image. Once we have a final configuration we can rebuild the image, but for now it would be much more convenient if we could apply new configuration options and test without going to that much effort.<\/span><\/p>\n
Let’s make use of Docker’s<\/span> -v<\/em> \/ –volume<\/em><\/span> option, which allows you to mount a local file or directory in the container. The Prometheus configuration file in the container is <\/span>\/etc\/prometheus\/prometheus.yml<\/span><\/em> – if that was the only file we needed then we could simply mount the file directly, but we’ll need a few more than that so we’ll mount a directory instead.<\/span><\/p>\n
In your working directory, create a sub-directory that we’ll use to keep our configuration (<\/span>prometheus.yml<\/span><\/em>) and other associated files.<\/span><\/p>\n
mkdir prometheus<\/pre>\n
Now we need a copy of the configuration file that we can work with. As always, there’s more than one way to do this, but by far the simplest is to use the <\/span>docker cp<\/span> command, to copy <\/span>\/etc\/prometheus\/prometheus.yml<\/span><\/em> from your running container, out to a local file:<\/span><\/p>\n
docker cp $(docker ps --quiet --filter publish=9090):\/etc\/prometheus\/prometheus.yml prometheus\/<\/pre>\n
Now kill your currently-running container:<\/span><\/p>\n
docker kill $(docker ps --quiet --filter publish=9090)<\/pre>\n
And then start a new container, this time mounting your local <\/span>prometheus<\/span><\/em> directory as <\/span>\/etc\/prometheus\/<\/span><\/em> in the container:<\/span><\/p>\n
docker run --detach --rm --publish 9090:9090 --volume $(pwd)\/prometheus\/:\/etc\/prometheus\/ prom\/prometheus<\/pre>\n
If, like the author, you find yourself wondering how to tell if the Prometheus server is really using your copy of <\/span>prometheus.yml<\/span><\/em> or not, you can check the mounts inside the container to make sure that <\/span>\/etc\/prometheus\/ <\/span><\/em>is specifically mounted with:<\/span><\/p>\n
docker exec $(docker ps --quiet --filter publish=9090) mount | grep '\/etc\/prometheus'<\/pre>\n
You could also make a small change to the configuration (for example, the name of the existing scrape job), and this should be reflected in the console (at <\/span>https:\/\/localhost:9090\/config<\/span><\/a>).<\/span><\/p>\n
Lastly for this section, as we’ll be making a few changes to our configuration file, it would be good if we didn’t have to recreate the container to apply the changes. Prometheus does have a built-in API method for this, discussed in <\/span>Configuration<\/span><\/a>, but the <\/span>–web.enable-lifecycle<\/span><\/em> option is not enabled by default in the official container. Fortunately (for MacOS and Linux users at least), there is the option to send a SIGHUP with <\/span>docker kill<\/span><\/em> (resulting in a “Loading configuration file” message in the logs):<\/span><\/p>\n
docker kill --signal SIGHUP $(docker ps --quiet --filter publish=9090)<\/pre>\n
In summary so far then, we now have a running Prometheus server, we know how to check its logs, browse the metrics that it is scraping, and how to update and reload its configuration, making it easy for us to add in our Capella Database as a scrape target.<\/span><\/p>\n
Adding a Capella Database to your Prometheus Server<\/span><\/h2>\n
Now we have a Prometheus server, let’s take a look at adding a Couchbase Capella database.<\/span><\/p>\n
Prerequisites<\/span><\/h3>\n
For each database that you want to collect metrics from, you’re going to need:<\/span><\/p>\n