CHALLENGES IN A MICROSERVICES AGE: MONITORING, LOGGING AND TRACING ON OPENSHIFT. Martin Etmajer Technology May 4, 2017

Transcription

1 CHALLENGES IN A MICROSERVICES AGE: MONITORING, LOGGING AND TRACING ON OPENSHIFT Martin Etmajer Technology May 4, 2017

2 WHY A CHALLENGE?

3 Microservice A Microservice B Microservice C

4 Microservice A Microservice B Microservice C Runtime Environment Runtime Environment Runtime Environment

5 Microservice A Data Store Microservice B Data Store Microservice C Data Store Runtime Environment Runtime Environment Runtime Environment

6 Microservice A Data Store Microservice B Data Store Microservice C Data Store Runtime Environment Runtime Environment Runtime Environment

7 Microservice A Data Microservice B Data Microservice C Store Store Data Store Runtime Environment Runtime Environment Runtime Environment

8 Load Balancing 3 6 Load Balancing Load Balancing Microservice A Data Store Microservice B Data Store Microservice C Data Store Runtime Environment Runtime Environment Runtime Environment

9 Load Balancing 3 6 Load Balancing Load Balancing Microservice A Data Store Microservice B Data Store Microservice C Data Store Runtime Environment Runtime Environment Runtime Environment

10 Load Balancing 3 6 Load Balancing Load Balancing Microservice A Data Store Microservice B Data Store Microservice C Data Store Runtime Environment Runtime Environment Runtime Environment Messaging System

11 API Gateways Load Balancing 3 6 Load Balancing Load Balancing Microservice A Data Store Microservice B Data Store Microservice C Data Store Runtime Environment Runtime Environment Runtime Environment Messaging System

12 security load balancing API Gateways discovery & routing languages Load Balancing storage 3 6 Load Balancing Load Balancing Microservice A Data Store Microservice B Data Store Microservice C Data Store Runtime Environment Runtime Environment Runtime Environment messaging application servers, container runtimes, Messaging System

13 RED HAT Hello World MSA Demo

14

15

16

17 MONITORING

18 Monitoring container health checks with OpenShift Liveness Probes What? a liveness probe periodically checks if a container is able to handle requests when a condition has not been met within a given timeout, the probe fails if a probe fails, the container is killed and subjected to its restart policy

19 Monitoring container health checks with OpenShift Liveness Probes How? set in spec.containers.livenessprobe of Pod config 1 livenessprobe: 2 httpget: 3 path: /health 4 port: initialdelayseconds: 15 6 timeoutseconds: 1

20 Monitoring container health checks with OpenShift Liveness Probes How? supports HTTP Get, Container Command and TCP Socket types 1 livenessprobe: 2 httpget: 3 path: /health 4 port: initialdelayseconds: 15 6 timeoutseconds: 1

21 Monitoring container health checks with OpenShift Liveness Probes Demo How?

22 Monitoring Where are the fancy dashboards? Source: d3js.org - Data-Driven Documents

23 Monitoring container, pod and node metrics with Heapster and Hawkular Metrics Heapster Hawkular Metrics

24

25 collects resource usage data, etc.

26 exposes metrics via REST

27 supports various storage backends

28 Heapster runs in a Pod elasticsearch kafka gathers metrics exposes metrics influxdb graphite GCL GCM

29 e.g. Hawkular

30 Hawkular A collection of open source monitoring components by Red Hat

31 Hawkular A collection of open source monitoring components by Red Hat metrics storage engine

32 Hawkular A collection of open source monitoring components by Red Hat distributed tracing and application performance management

33 Hawkular A collection of open source monitoring components by Red Hat alerting

34 Hawkular A collection of open source monitoring components by Red Hat feed for Jolokia and Prometheus metrics

35 with a REST interface a metrics storage engine

36 Monitoring container, pod and node metrics with Heapster and Hawkular Metrics Demo visualizes metrics sends metrics gathers metrics!!!!!

37

38

39

40

41

42

43

44 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) Hawkular OpenShift Agent Jolokia Prometheus

45 feeds Jolokia/Prometheus metrics data into Hawkular

46 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) is deployed via a DaemonSet!!!!!

47 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) runs!!!!! HOSA HOSA HOSA HOSA HOSA

48 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) applications sends metrics gathers metrics!!!!! HOSA HOSA HOSA HOSA HOSA

49 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) gathers metrics application containers!!!!! HOSA HOSA HOSA HOSA HOSA

50

51

52

53

54 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) How? ConfigMap defines agent configuration 1 kind: ConfigMap 2 apiversion: v1 3 metadata: 4 name: my-hosa-config 5 data: 6 hawkular-openshift-agent: 7 endpoints: 8 - type: jolokia 15 - type: prometheus 9 protocol: http 16 protocol: http 10 port: port: path: /jolokia/ 18 path: /metrics/ 12 metrics 19 metrics: 13 - name: java.lang:type=memory 20 - name: process_start_time_seconds 14 type: gauge 21 type: gauge

55 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) Demo How? a volume named hawkular-openshift-agent tells HOSA to monitor this Pod 1 spec: 2 volumes: 3 - name: hawkular-openshift-agent 4 configmap: 5 name: my-hosa-config

56 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) How? 1 kind: ConfigMap 2 apiversion: v1 3 metadata: 4 name: my-hosa-config 5 data: Jolokia? 6 hawkular-openshift-agent: 7 endpoints: 8 - type: jolokia 15 - type: prometheus 9 protocol: http 16 protocol: http 10 port: port: path: /jolokia/ 18 path: /metrics/ 12 metrics 19 metrics: 13 - name: java.lang:type=memory 20 - name: process_start_time_seconds 14 type: gauge 21 type: gauge

57 Jolokia JMX on Capsaicin access JMX MBeans remotely via REST

58 Jolokia JMX on Capsaicin installs a Jolokia WAR agent into Apache Tomcat $ mv $JOLOKIA_HOME/agents/jolokia.war $TOMCAT_HOME/webapps $ $TOMCAT_HOME/bin/catalina.sh start

59 Monitoring application level metrics with Hawkular OpenShift Agent (HOSA) How? 1 kind: ConfigMap 2 apiversion: v1 3 metadata: 4 name: my-hosa-config 5 data: Prometheus? 6 hawkular-openshift-agent: 7 endpoints: 8 - type: jolokia 15 - type: prometheus 9 protocol: http 16 protocol: http 10 port: port: path: /jolokia/ 18 path: /metrics/ 12 metrics 19 metrics: 13 - name: java.lang:type=memory 20 - name: process_start_time_seconds 14 type: gauge 21 type: gauge

60 Prometheus A monitoring system for metrics data

61 Prometheus A monitoring system for metrics data metrics storage engine

62 Prometheus A monitoring system for metrics data various exporters expose metrics from nodes, database engines and cloud componentry

63 Prometheus A monitoring system for metrics data alerting

64 Prometheus A monitoring system for metrics data # build.gradle # Application.java 1 dependencies { 1 public void registermetricservlet() { 2 compile 'io.prometheus.simpleclient ' 2 // Expose Promtheus metrics. 3 compile 'io.prometheus.simpleclient_hotspot ' 3 context.addservlet(new ServletHolder (new MetricsServlet()), "/metrics"); 4 compile 'io.prometheus.simpleclient_servlet ' 4 // Add metrics about CPU, JVM memory etc. 5 } 5 DefaultExports.initialize(); 6 } exposes default JMX metrics: CPU, Memory, Garbage Collection, etc.

65 Where to start?

66 LOGGING

67 how to process all these data?

68 Logging application events with Elasticsearch, Fluentd and Kibana (EFK) Elasticsearch Fluentd Kibana

69 unifies your logging infrastructure; supports almost 200 plugins

70 Logging application events with Elasticsearch, Fluentd and Kibana (EFK) is deployed via a DaemonSet!!!!!

71 Logging application events with Elasticsearch, Fluentd and Kibana (EFK) deploys!!!!! Fluentd Fluentd Fluentd Fluentd Fluentd

72 Logging application events with Elasticsearch, Fluentd and Kibana (EFK) Demo query Kibana Elasticsearch Application ingests log data via configuration watches container logs!!!!! Fluentd Fluentd Fluentd Fluentd Fluentd

73

74

75 DISTRIBUTED TRACING

76

77 design goals

78 Google Dapper trace (transaction) initiator spans (timed operations) remote calls processes annotations, event data Source: Google Dapper Paper

79 ZipKin A distributed tracing system timescale spans

80 a standard API for reporting distributed traces with various client libraries

81 OpenTracing A vendor-neutral open standard for distributed tracing The problem? instrumentation is challenging: tracing context must propagate within and between processes unreasonable to ask all vendor and FOSS software to be intrumented by a single tracing vendor The solution? allow developers to instrument their own code using OpenTracing client libraries have the application maintainer choose the tracing technology via configuration change

82 Hawkular APM Distributed Tracing and Application Performance Management

83

84 Jaeger A distributed tracing system Source: Jaeger Architecture jaeger.readthedocs.io

85 SOLVED THE CHALLENGE?

86

87 Dynatrace All-in-one monitoring, now! dynatrace/oneagent!!!!!

88 Dynatrace All-in-one monitoring, now! deploys!!!!!

89

90

91 THANK YOU plus.google.com/+redhat facebook.com/redhatinc linkedin.com/company/red-hat twitter.com/redhatnews youtube.com/user/redhatvideos

92