Federated Prometheus Monitoring at Scale

Size: px

Start display at page:

Download "Federated Prometheus Monitoring at Scale"

Lucas George
6 years ago
Views:

1 Federated Prometheus Monitoring at Scale LungChih Tung Oath Nandhakumar Venkatachalam Oath

2 Team Core Platform Team powering all Yahoo Media Products Yahoo Media Products Homepage, News Finance Sports, Fantasy

3 Journey to Kubernetes STAGE colo1 ETCD Master Nodes Nodes Nodes Nodes CANARY ETCD Master colo1 Nodes Nodes Nodes Nodes PROD colo1 ETCD Master Nodes Nodes Nodes Nodes

4 Journey to Kubernetes Monitoring Core system monitoring Heapster InFluxDB sink Prometheus 1.x Remote write plugin Missing insight on cluster metrics

5 Growing Kubernetes Clusters 12 independent clusters STAGE 2K+ Worker Nodes CANARY CANARY CANARY CANARY CANARY CANARY Colo n 12K+ Pods 50K+ Containers 200+ Application deployments PROD PROD PROD PROD PROD PROD PROD Colo n Peak requests/sec - 400K+

6 Demand for visibility How is the cluster utilization? Who is the big consumer? How is the etcd/ control plane health? How is the pod/container cpu/mem utilization? more...

7 Monitoring Requirements Cluster Health: ETCD Controller Manager Scheduler Kubernetes API server Kubelet Kubelet CAdvisor Kube DNS Any Add-ons... Application Health: Namespace Deployment Pod Container

8 Prometheus 2.0 Huge performance improvement specifically in storage Simple syntax for aggregation and alerting rules Great documentation Scrape all prometheus metrics

9 Metrics Collection Easy configuration ex: Prometheus example yaml API server metrics/ Controller/ Scheduler ETCD Kubelet and cadvisor Kube-state-metrics kube dns/ heapster etc etc Endpoint discovery is simple. annotation prometheus.io/scrape=true Simple File based discovery for apiservers

10 Metrics Collection - API Server Prometheus example yaml Endpoint discovery is simple. annotation prometheus.io/scrape=true Simple File based discovery for apiservers Adding datacenter label

11 Metrics Proxy ETCD ports are accessible only by master machines Scheduler and controller ports bind to ETCD node Metrics Proxy forward only /metrics Port: 5443 X Port: 2379,2380

12 Metrics Proxy - ETCD

13 Metrics Proxy - Controller

14 Prometheus Federation

15 Federation Cluster 1 Cluster 2 Selective and Aggregated metrics from each cluster Federated Promo Cluster n Aggregated time series data Longer retention period Permanent storage Unified display of data

16 Federation Configuration Federate all control plane components Selective aggregated metrics

17 Aggregation Rules Ton of rules from prometheus operator team Built CPU and Memory utilization by Colo(cluster) level Namespace/ Deployment/ Pod and Container level

18 Aggregation - Colo level # colo level cpu utilization - record: colo:cpu_percentage:rate expr: 100 * CPU usage of all containers per colo sum(label_replace(irate(container_cpu_usage_seconds_total{container_name!="", container_name!="pod"}[5m]), "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo) / sum(label_replace(container_spec_cpu_shares{container_name!="", container_name!="pod"}, "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo) * 1000 CPU shares allocated to all containers per colo

19 Aggregation - Namespace level # namespace level cpu utilization - record: colo_namespace:cpu_percentage:rate expr: 100 * CPU usage of all containers per namespace sum(label_replace(irate(container_cpu_usage_seconds_total{container_name!="", container_name!="pod"}[5m]), "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo, namespace) / sum(label_replace(container_spec_cpu_shares{container_name!="", container_name!="pod"}, "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo, namespace) * 1000 CPU shares allocated to all containers per namespace

20 Aggregation - Controller level # controller level cpu utilization - record: colo_namespace_controller:cpu_percentage:rate expr: 100 * CPU usage of all containers per controller sum(label_replace(irate(container_cpu_usage_seconds_total{container_name!="", container_name!="pod"}[5m]), "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo, namespace, controller) / sum(label_replace(container_spec_cpu_shares{container_name!="", container_name!="pod"}, "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo, namespace, controller) * 1000 CPU shares allocated to all containers per controller

21 Aggregation - Pod level # pod level cpu utilization - record: colo_namespace_controller_pod:cpu_percentage:rate CPU usage of all containers per pod expr: 100 * sum(label_replace(irate(container_cpu_usage_seconds_total{container_name!="", container_name!="pod"}[5m]), "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo, namespace, controller, pod_name) / sum(label_replace(container_spec_cpu_shares{container_name!="", container_name!="pod"}, "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo, namespace, controller, pod_name) * 1000 CPU shares allocated to all containers per pod

22 Aggregation - Container level # container level cpu utilization - record: colo_namespace_controller_pod_container:cpu_percentage:rate CPU usage per container expr: 100 * sum(label_replace(irate(container_cpu_usage_seconds_total{container_name!="", container_name!="pod"}[5m]), "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo, namespace, controller, pod_name, container_name) / sum(label_replace(container_spec_cpu_shares{container_name!="", container_name!="pod"}, "controller", "$1", "pod_name", "^(.*)-[a-z0-9]+")) BY (colo, namespace, controller, pod_name, container_name) * 1000 CPU shares allocated to container

23 Alert Manager

24 Alert Manager Alert manager defines how to handle alerts ( , slack notification, etc.) Grouping alert Silences Inhibition

25 Alert Rules - alert: K8SNodeNotReady expr: kube_node_status_condition{condition="ready",status="true"} == 0 for: 1h labels: severity: warning colo: bf1 environment: production annotations: description: The Kubelet on {{ $labels.node }} has not checked in with the API, or has set itself to NotReady, for more than an hour summary: Node status is NotReady

26 Alert Rules - alert: PodCPUPercentage expr: colo_namespace_controller_pod:cpu_percentage:rate{namespace=~"kube-.*"} > 75 for: 10m labels: severity: critical colo: bf1 environment: production annotations: description: 'Pod cpu usage is above 75 for {{ $value }}. Please find out the cause of the spike and if required increase CPU allocation' summary: Pod cpu usage is above threshold

27 Alert Rules Simple alert template

28 Alerting on Prometheus Federated prometheus monitors individual prometheus A cron job monitors federated prometheus

29 Dashboards Showcase

30 Cluster View

31 Namespace

32 Deployment

33 Controller

34 Scheduler

35 API server

36 Kubelet kubenode

37 ETCD n

38 Package version

39 Volume of the metrics

40 Thank you Q&A Link to dashboards and configurations:

Operating Within Normal Parameters: Monitoring Kubernetes

Operating Within Normal Parameters: Monitoring Kubernetes Elana Hashman Two Sigma Investments, LP SREcon 2019 Americas Brooklyn, NY Disclaimer This document is being distributed for informational and educational