{"id":2445,"date":"2017-01-06T11:00:32","date_gmt":"2017-01-06T11:00:31","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=2445"},"modified":"2023-06-21T06:03:43","modified_gmt":"2023-06-21T13:03:43","slug":"kubernetes-monitoring-heapster-influxdb-grafana","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/kubernetes-monitoring-heapster-influxdb-grafana\/","title":{"rendered":"Kubernetes Monitoring with Heapster, InfluxDB and Grafana"},"content":{"rendered":"

Kubernetes provides detailed insights about resource usage in the cluster.\u00a0This is\u00a0enabled by using Heapster<\/a>, cAdvisor<\/a>, InfluxDB<\/a> and Grafana<\/a>.<\/p>\n

Heapster is installed as a\u00a0cluster-wide pod.\u00a0It gathers monitoring and events data for all pods on each node by\u00a0talking to the Kubelet. Kubelet itself fetches this data from cAdvisor. This data is persisted in InfluxDB and then visualized
\nusing Grafana.<\/p>\n

\"kubernetes-logging\"<\/a><\/p>\n

Resource Usage Monitoring<\/a>\u00a0provide more details about\u00a0monitoring resources in Kubernetes. Heapster, InfluxDB and Grafana\u00a0are Kubernetes addons<\/a>.\u00a0They
\nare enabled by default if you are running the cluster on Amazon Web Services or Google Cloud. But need to be explicitly enabled if the cluster is started using minikube<\/a>\u00a0or
\nkops<\/a> addons<\/a>.<\/p>\n

Start a Kubernetes cluster on Amazon Web Services as:
\nKUBERNETES_PROVIDER=aws; kube-up.sh<\/code><\/p>\n

More details about starting a Kubernetes cluster are available at Getting Started with Kubernetes 1.4<\/a>. By default, it creates a 4-node Kubernetes cluster in\u00a0us-west-2a<\/code> region.\u00a0More details about the cluster can be seen using the command\u00a0kubectl cluster-info<\/code> and it shows the results as:<\/p>\n

Kubernetes master is running at https:\/\/35.165.6.91\r\nElasticsearch is running at https:\/\/35.165.6.91\/api\/v1\/proxy\/namespaces\/kube-system\/services\/elasticsearch-logging\r\nHeapster is running at https:\/\/35.165.6.91\/api\/v1\/proxy\/namespaces\/kube-system\/services\/heapster\r\nKibana is running at https:\/\/35.165.6.91\/api\/v1\/proxy\/namespaces\/kube-system\/services\/kibana-logging\r\nKubeDNS is running at https:\/\/35.165.6.91\/api\/v1\/proxy\/namespaces\/kube-system\/services\/kube-dns\r\nkubernetes-dashboard is running at https:\/\/35.165.6.91\/api\/v1\/proxy\/namespaces\/kube-system\/services\/kubernetes-dashboard\r\nGrafana is running at https:\/\/35.165.6.91\/api\/v1\/proxy\/namespaces\/kube-system\/services\/monitoring-grafana\r\nInfluxDB is running at https:\/\/35.165.6.91\/api\/v1\/proxy\/namespaces\/kube-system\/services\/monitoring-influxdb\r\n\r\nTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.<\/pre>\n
Note the\u00a0URL for the Grafana service. Open this URL in a browser window. You’ll be prompted for an\u00a0invalid certificate warning but this can be safely ignored at this time. In production system, appropriate certificates should\u00a0be installed.
\nThen you’ll be prompted for credentials. These can be obtained using\u00a0kubectl config view<\/code> command. It will show the output as:<\/p>\n
- name: aws_kubernetes-basic-auth\r\n  user:\r\n    password: ZeH4JpQzAtGDEBdb\r\n    username: admin<\/pre>\n
Use\u00a0the value from\u00a0username<\/code> and password<\/code> fields. This shows the default dashboard:
\n\"kubernetes-grafana-empty-dashboard\"<\/p>\n
It consists of two dashboards – one for cluster and another for pods.<\/p>\n
\"kubernetes-grafana-dashboards\"<\/a><\/p>\n
For this blog, a\u00a04-node Couchbase cluster was created following the steps outlined in\u00a0Create a Couchbase Cluster using Kubernetes.<\/a> A cluster-wide
\ndashboard shows CPU, Memory, Filesystem and Network usage across all\u00a0the hosts and looks like:<\/p>\n
\"kubernetes-grafana-cluster\"<\/a><\/p>\n
CPU, memory, filesystem and network usage for all\u00a0nodes may be seen:<\/p>\n
\"\"<\/p>\n
Details for each node may be seen by selecting the node:<\/p>\n
\"kubernetes-grafana-cluster-nodelist\"<\/p>\n
CPU, memory, filesystem and network usage for each node\u00a0is\u00a0displayed:<\/p>\n
\"kubernetes-grafana-cluster-one-node\"<\/p>\n
Pods dashboard shows CPU, memory, filesystem and network usage for each pod:<\/p>\n
\"kubernetes-grafana-pods\"<\/a><\/p>\n
A different pod may be chosen:<\/p>\n
\"kubernetes-grafana-pods-list\"<\/a><\/p>\n
A complete list of all services running in the Kubernetes can be seen using kubectl get services --all-namespaces<\/code> command. It shows the output as:<\/p>\n
kubectl.sh get svc --all-namespaces\r\nNAMESPACE     NAME                       CLUSTER-IP     EXTERNAL-IP        PORT(S)             AGE\r\ndefault       couchbase-master-service   10.0.70.206    aef06961eb8f3...   8091\/TCP            1h\r\ndefault       kubernetes                 10.0.0.1                    443\/TCP             1h\r\nkube-system   elasticsearch-logging      10.0.54.112                 9200\/TCP            1h\r\nkube-system   heapster                   10.0.146.18                 80\/TCP              1h\r\nkube-system   kibana-logging             10.0.123.37                 5601\/TCP            1h\r\nkube-system   kube-dns                   10.0.0.10                   53\/UDP,53\/TCP       1h\r\nkube-system   kubernetes-dashboard       10.0.146.179                80\/TCP              1h\r\nkube-system   monitoring-grafana         10.0.33.81                  80\/TCP              1h\r\nkube-system   monitoring-influxdb        10.0.26.251                 8083\/TCP,8086\/TCP   1h<\/pre>\n
A complete list of all the pods running in the Kubernetes cluster can be seen using\u00a0kubectl get pods --all-namespaces<\/code>.<\/p>\n
Some references:<\/p>\n