= Kubernetes Cluster Monitoring :toc: :icons: :linkcss: :imagesdir: ../../resources/images == Introduction This chapter will demonstrate how to monitor a Kubernetes cluster using the following: . Kubernetes Dashboard . Heapster, InfluxDB and Grafana . Prometheus, Node exporter and Grafana http://prometheus.io/[Prometheus] is an open-source systems monitoring and alerting toolkit. Prometheus collects metrics from monitored targets by scraping metrics from HTTP endpoints on these targets. Heapster is limited to Kuberenetes container metrics, it is not general use. Heapster can be used as Prometheus scrape target. == Prerequisites In order to perform exercises in this chapter, you’ll need to deploy configurations to a Kubernetes cluster. To create an EKS-based Kubernetes cluster, use the link:../../01-path-basics/102-your-first-cluster#create-a-kubernetes-cluster-with-eks[AWS CLI] (recommended). If you wish to create a Kubernetes cluster without EKS, you can instead use link:../../01-path-basics/102-your-first-cluster#alternative-create-a-kubernetes-cluster-with-kops[kops]. All configuration files for this chapter are in the link:templates[201-cluster-monitoring/templates] directory. == Kubernetes Dashboard https://github.com/kubernetes/dashboard[Kubernetes Dashboard] is a general purpose web-based UI for Kubernetes clusters. The Dashboard uses the https://kubernetes.io/docs/admin/authorization/rbac/[RBAC API], which has been promoted in Kubernetes v1.8 to GA rather than Beta, so you'll use a different version of the dashboard depending on the version of Kubernetes you are running. Check your Kubernetes version using the following command - check the value of the `Server Version`, which is v1.7.4 in this example: kubectl version $ kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:30:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} If you are using v1.7.x, deploy the Dashboard using the following command: kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/6dc75162dce25b5a94aa500ebba923e8223e5cfd/src/deploy/recommended/kubernetes-dashboard.yaml If you are using v1.8 or above, deploy the Dashboard using the following command: kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml Dashboard can be seen using the following command: kubectl proxy --address 0.0.0.0 --accept-hosts '.*' --port 8080 Now, Dashboard is accessible via `Preview`, `Preview Running Application` as: https://ENVIRONMENT_ID.vfs.cloud9.REGION_ID.amazonaws.com/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ Where `ENVIRONMENT_ID` is your Cloud9 IDE environment id (you should see it once click the built-in browser address bar) and `REGION_ID` is AWS region id (e.g. us-east-1). Starting with Kubernetes 1.7, Dashboard supports authentication. Read more about it at https://github.com/kubernetes/dashboard/wiki/Access-control#introduction. We'll use a bearer token for authentication. Check existing secrets in the `kube-system` namespace: kubectl -n kube-system get secret It shows the output as: NAME TYPE DATA AGE attachdetach-controller-token-dhkcr kubernetes.io/service-account-token 3 3h certificate-controller-token-p131b kubernetes.io/service-account-token 3 3h daemon-set-controller-token-r4mmp kubernetes.io/service-account-token 3 3h default-token-7vh0x kubernetes.io/service-account-token 3 3h deployment-controller-token-jlzkj kubernetes.io/service-account-token 3 3h disruption-controller-token-qrx2v kubernetes.io/service-account-token 3 3h dns-controller-token-v49b6 kubernetes.io/service-account-token 3 3h endpoint-controller-token-hgkbm kubernetes.io/service-account-token 3 3h generic-garbage-collector-token-34fvc kubernetes.io/service-account-token 3 3h horizontal-pod-autoscaler-token-lhbkf kubernetes.io/service-account-token 3 3h job-controller-token-c2s8j kubernetes.io/service-account-token 3 3h kube-dns-autoscaler-token-s3svx kubernetes.io/service-account-token 3 3h kube-dns-token-92xzb kubernetes.io/service-account-token 3 3h kube-proxy-token-0ww14 kubernetes.io/service-account-token 3 3h kubernetes-dashboard-certs Opaque 2 9m kubernetes-dashboard-key-holder Opaque 2 9m kubernetes-dashboard-token-vt0fd kubernetes.io/service-account-token 3 10m namespace-controller-token-423gh kubernetes.io/service-account-token 3 3h node-controller-token-r6lsr kubernetes.io/service-account-token 3 3h persistent-volume-binder-token-xv30g kubernetes.io/service-account-token 3 3h pod-garbage-collector-token-fwmv4 kubernetes.io/service-account-token 3 3h replicaset-controller-token-0cg8r kubernetes.io/service-account-token 3 3h replication-controller-token-3fwxd kubernetes.io/service-account-token 3 3h resourcequota-controller-token-6rl9f kubernetes.io/service-account-token 3 3h route-controller-token-9brzb kubernetes.io/service-account-token 3 3h service-account-controller-token-bqlsk kubernetes.io/service-account-token 3 3h service-controller-token-1qlg6 kubernetes.io/service-account-token 3 3h statefulset-controller-token-kmgzg kubernetes.io/service-account-token 3 3h ttl-controller-token-vbnhf kubernetes.io/service-account-token 3 3h We can login using the secret with type 'kubernetes.io/namespace-controller-token'. In our case, we'll use the token from secret `namespace-controller-token-423gh` to login. Use the following command to get the token for this secret: kubectl -n kube-system describe secret namespace-controller-token-423gh Note you'll need to replace `namespace-controller-token-423gh` with the namespace-controller-token from your output list. It shows the output: ``` Name: namespace-controller-token-423gh Namespace: kube-system Labels: Annotations: kubernetes.io/service-account.name=default kubernetes.io/service-account.uid=3a3fea86-b3a1-11e7-9d90-06b1e747c654 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1046 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLTd2aDB4Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzYTNmZWE4Ni1iM2ExLTExZTctOWQ5MC0wNmIxZTc0N2M2NTQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.GHW-7rJcxmvujkClrN6heOi_RYlRivzwb4ScZZgGyaCR9tu2V0Z8PE5UR6E_3Vi9iBCjuO6L6MLP641bKoHB635T0BZymJpSeMPQ7t1F02BsnXAbyDFfal9NUSV7HoPAhlgURZWQrnWojNlVIFLqhAPO-5T493SYT56OwNPBhApWwSBBGdeF8EvAHGtDFBW1EMRWRt25dSffeyaBBes5PoJ4SPq4BprSCLXPdt-StPIB-FyMx1M-zarfqkKf7EJKetL478uWRGyGNNhSfRC-1p6qrRpbgCdf3geCLzDtbDT2SBmLv1KRjwMbW3EF4jlmkM4ZWyacKIUljEnG0oltjA ``` Copy the value of token from this output, select `Token` in the Dashboard login window, and paste the text. Click on `SIGN IN` to see the default Dashboard view: image::kubernetes-dashboard-default.png[] Click on `Nodes` to see a textual representation about the nodes running in the cluster: image::monitoring-nodes-before.png[] Install a Java application as explained in link:../../03-path-application-development/306-app-management-with-helm[Deploying applications using Kubernetes Helm charts]. Click on `Pods`, again to see a textual representation about the pods running in the cluster: image::monitoring-pods-before.png[] This will change after Heapster, InfluxDB and Grafana are installed. == Heapster, InfluxDB and Grafana https://github.com/kubernetes/heapster[Heapster] is a metrics aggregator and processor. It is installed as a cluster-wide pod. It gathers monitoring and events data for all containers on each node by talking to the Kubelet. Kubelet itself fetches this data from https://github.com/google/cadvisor[cAdvisor]. This data is persisted in a time series database https://github.com/influxdata/influxdb[InfluxDB] for storage. The data is then visualized using a http://grafana.org/[Grafana] dashboard, or it can be viewed in Kubernetes Dashboard. Heapster collects and interprets various signals like compute resource usage, lifecycle events, etc., and exports cluster metrics via REST endpoints. Heapster, InfluxDB and Grafana are http://kubernetes.io/docs/admin/addons/[Kubernetes addons]. === Installation Execute this command to install Heapster, InfluxDB and Grafana: $ kubectl apply -f templates/heapster/ deployment "monitoring-grafana" created service "monitoring-grafana" created clusterrolebinding "heapster" created serviceaccount "heapster" created deployment "heapster" created service "heapster" created deployment "monitoring-influxdb" created service "monitoring-influxdb" created Heapster is now aggregating metrics from the cAdvisor instances running on each node. This data is stored in an InfluxDB instance running in the cluster. Grafana dashboard, accessible at https://ENVIRONMENT_ID.vfs.cloud9.REGION_ID.amazonaws.com/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy/?orgId=1, now shows the information about the cluster. NOTE: Grafana dashboard will not be available if Kubernetes proxy is not running. If proxy is not running, it can be started with the command `kubectl proxy --address 0.0.0.0 --accept-hosts '.*' --port 8080`. === Grafana dashboard There are some built-in dashboards for monitoring the cluster and workloads. They are available by clicking on the upper left corner of the screen. image::monitoring-grafana-dashboards.png[] The "`Cluster`" dashboard shows all worker nodes, and their CPU and memory metrics. Type in a node name to see its collected metrics during a chosen period of time. The cluster dashboard looks like this: image::monitoring-grafana-dashboards-cluster.png[] The "`Pods`"" dashboard allows you to see the resource utilization of every pod in the cluster. As with nodes, you can select the pod by typing its name in the top filter box. image::monitoring-grafana-dashboards-pods.png[] After the deployment of Heapster, Kubernetes Dashboard now shows additional graphs such as CPU and Memory utilization for pods and nodes, and other workloads. The updated view of the cluster in Kubernetes Dashboard looks like this: image::monitoring-nodes-after.png[] The updated view of pods looks like this: image::monitoring-pods-after.png[] === Cleanup Remove all the installed components: kubectl delete -f templates/heapster/ == Prometheus, Node exporter and Grafana http://prometheus.io/[Prometheus] is an open-source systems monitoring and alerting toolkit. Prometheus collects metrics from monitored targets by scraping metrics from HTTP endpoints on these targets. Prometheus will be managed by the https://github.com/coreos/prometheus-operator/[Kubernetes Operator] - This operator uses https://kubernetes.io/docs/concepts/api-extension/custom-resources/[Custom Resources] to extend the Kubernetes API and add custom resources such as `Prometheus`, `ServiceMonitor` and `Alertmanager`. Prometheus is able to dynamically scrape new targets by adding a https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md[ServiceMonitor] - we have included a couple of them to scrape `kube-controller-manager`, `kube-scheduler`, `kube-state-metrics`, `kubelet` and `node-exporter`. https://github.com/prometheus/node_exporter[Node exporter] is a Prometheus exporter for hardware and OS metrics exposed by *NIX kernels. https://github.com/kubernetes/kube-state-metrics[kube-state-metrics] is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. === Installation First we need to deploy the Prometheus Operator which will listen for the new Custom Resources: $ kubectl apply -f templates/prometheus/prometheus-bundle.yaml namespace "monitoring" created clusterrolebinding "prometheus-operator" created clusterrole "prometheus-operator" created serviceaccount "prometheus-operator" created deployment "prometheus-operator" created Next we need to wait until the Prometheus Operator has started: $ kubectl rollout status deployment/prometheus-operator -n monitoring ... deployment "prometheus-operator" successfully rolled out As a final step we need to deploy the Prometheus Custom Resource, Service Monitors, Cluster Roles and Bindings (RBAC): $ kubectl apply -f templates/prometheus/prometheus.yaml serviceaccount "kube-state-metrics" created clusterrole "kube-state-metrics" created clusterrolebinding "kube-state-metrics" created service "kube-scheduler-prometheus-discovery" created service "kube-controller-manager-prometheus-discovery" created daemonset "node-exporter" created service "node-exporter" created deployment "kube-state-metrics" created service "kube-state-metrics" created prometheus "prometheus" created servicemonitor "prometheus-operator" created servicemonitor "kube-apiserver" created servicemonitor "kubelet" created servicemonitor "kube-controller-manager" created servicemonitor "kube-scheduler" created servicemonitor "kube-state-metrics" created servicemonitor "node-exporter" created alertmanager "main" created secret "alertmanager-main" created Lets wait for prometheus to come up: $ kubectl get po -l prometheus=prometheus -n monitoring NAME READY STATUS RESTARTS AGE prometheus-prometheus-0 2/2 Running 0 1m prometheus-prometheus-1 2/2 Running 0 1m === Prometheus Dashboard Prometheus is now scraping metrics from the different scraping targets and we forward the dashboard via: $ kubectl port-forward $(kubectl get po -l prometheus=prometheus -n monitoring -o jsonpath={.items[0].metadata.name}) 9090 -n monitoring Forwarding from 127.0.0.1:9090 -> 9090 Now open the browser at http://localhost:9090/targets and all targets should be shown as `UP` (it might take a couple of minutes until data collectors are up and running for the first time). The browser displays the output as shown: image::monitoring-grafana-prometheus-dashboard-1.png[] image::monitoring-grafana-prometheus-dashboard-2.png[] image::monitoring-grafana-prometheus-dashboard-3.png[] === Grafana Installation To install grafana we need to run: $ kubectl apply -f templates/prometheus/grafana-bundle.yaml secret "grafana-credentials" created service "grafana" created configmap "grafana-dashboards-0" created deployment "grafana" created Lets wait for grafana to come up: $ kubectl rollout status deployment/grafana -n monitoring ... deployment "grafana" successfully rolled out === Grafana Dashboard Lets forward the grafana dashboard to a local port: $ kubectl port-forward $(kubectl get pod -l app=grafana -o jsonpath={.items[0].metadata.name} -n monitoring) 3000 -n monitoring Forwarding from 127.0.0.1:3000 -> 3000 Grafana dashboard is now accessible at http://localhost:3000/. The complete list of dashboards is available using the search button at the top: image::monitoring-grafana-prometheus-dashboard-dashboard-home.png[] You can access various metrics using these dashboards: . http://localhost:3000/dashboard/db/kubernetes-control-plane-status?orgId=1[Kubernetes Cluster Control Plane] + image::monitoring-grafana-prometheus-dashboard-control-plane-status.png[] + . http://localhost:3000/dashboard/db/kubernetes-cluster-status?orgId=1[Kubernetes Cluster Status] + image::monitoring-grafana-prometheus-dashboard-cluster-status.png[] + . http://localhost:3000/dashboard/db/kubernetes-capacity-planning?orgId=1[Kubernetes Cluster Capacity Planning] + image::monitoring-grafana-prometheus-dashboard-capacity-planning.png[] + . http://localhost:3000/dashboard/db/nodes?orgId=1[Nodes in the Kubernetes cluster] + image::monitoring-grafana-prometheus-dashboard-nodes.png[] Convenient link for other dashboards are listed below: * http://localhost:3000/dashboard/db/deployment&orgId=1 * http://localhost:3000/dashboard/db/kubernetes-cluster-health?refresh=10s&orgId=1 * http://localhost:3000/dashboard/db/kubernetes-resource-requests?orgId=1 * http://localhost:3000/dashboard/db/pods?orgId=1 === Cleanup Remove all the installed components: kubectl delete -f templates/prometheus/prometheus-bundle.yaml You are now ready to continue on with the workshop! :frame: none :grid: none :valign: top [align="center", cols="2", grid="none", frame="none"] |===== |image:button-continue-standard.png[link=../../02-path-working-with-clusters/202-service-mesh] |image:button-continue-operations.png[link=../../02-path-working-with-clusters/202-service-mesh] |link:../../standard-path.adoc[Go to Standard Index] |link:../../operations-path.adoc[Go to Operations Index] |=====