Monitor your containers with the Elastic Stack. Monica Sarbu

Transcription

1

2 Monitor your containers with the Elastic Stack Monica Sarbu

3 Monica Sarbu Team lead, Beats team 3

4 Monitor your containers with the Elastic Stack

5 Elastic Stack 5

6 Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch

7 Multiple data types, one place Docker metrics Docker logs Apache logs MySQL transactions Redis logs flows diskio Redis metrics Docker metrics CPU % memory % HTTP transactions MySQL logs flows Redis transactions filesystem 7

8 Central point for your distributed infrastructure 8

9 The Beats 30+ other community Beats shipping 9

10 Filebeat 10

11 Filebeat Tails log files, without parsing them At least once guarantees, handles backpressure Extra powers: Multiline JSON logs Filtering 11

12 Parse log lines with Ingest Node I N G E S T 12

13 Parse log lines with Logstash I N G E S T 13

14 Filebeat Back pressure handling 14

15 Why back-pressure is key? 15

16 Synchronous sending registry file acked read read stream of log lines batch of messages ack 16

17 This means.. Filebeat adapts its speed automatically to as much as the next stage can process But: be aware when benchmarking 17

18 When the next stage is down.. Filebeat patiently waits Log lines are not lost It doesn t allocate memory, it doesn t buffer things on disk 18

19 Filebeat Collect container logs 19

20 Docker logging drivers 20

21 Centralize Docker logs: option 1/522 Use the Docker gelf driver and the Logstash-gelf-input Pros: No shipper to install, send directly to Logstash Cons: UDP based, no delivery guarantees, no congestion control 21

22 Centralize Docker logs: option 2/522 Use the Docker JSON driver, use Filebeat with the JSON support Pros: Simple (default driver) Easy to add container metadata (name, labels, etc.) `docker logs` works Cons: JSON driver can slow down Docker 22

23 Centralize Docker logs: option 3/522 Use the Docker syslog driver, and a local syslog server, then Filebeat for shipping Pros: Good control over the path where the files are written, rotation strategies, etc. Cons: you need to manage the syslog server metadata is serialized as string, needs to be deserialized again (opportunity for mistakes) multiline is difficult because data from containers can be mixed 23

24 Centralize Docker logs: option 4/522 Use the Docker journald driver then Filebeat for shipping Pros: journald is often already available convenient support for metadata `docker logs` works Cons: Filebeat doesn t yet support journald (a Journalbeat exists, however) 24

25 Centralize Docker logs: option 5/522 Mount a volume and have your app write logs into the volume Pros: If your app can rotate it s own logs, it s very easy to setup Scales well Cons: Difficult to pass metadata 25

26 Centralize Docker logs: conclusion json driver, syslog driver, and shared volume are pretty good options today journald driver might be better options in the future 26

27 Metricbeat new in

28 One Metricbeat module for each service + Add your own 28

29 Metricbeat system module CPU Mem diskio filesystem load network cores processes 29

30 Metricbeat Collect container metrics 30

31 Querying the Docker API in progress Dedicated Docker module Has access to container names and labels Easy to setup Offers: CPU and memory Docker container information network (in/out bytes, dropped) diskio (reads/writes) status of containers (# of stopped, running, etc) 31

32 Reading cgroup data from /proc/ Doesn t require access to the Docker API (can be a security issue) Works for any container runtime (Docker, rkt, runc, LXD, etc.) Part of the system module Automatically enhances process data with cgroup information Cannot get the container name and labels 32

33 Run as a container App1 App2 App3 Host 33

34 Elasticsearch as time series DB 34

35 Elasticsearch BKD trees Added for Geo-points faster to index #velo faster to query more disk-efficient more memory efficient 35

36 Float values On Disk Usage in kb half floats scaled floats (using a scaling factor) - great for things like percentage points float half float scaled float (factor = 4000) scaled float (factor = 100) Points disk usage (kb) docs_values disk usage (kb) 36

37 Why Elasticsearch for time series Horizontal scalability. Mature and battle tested cluster support. Flexible aggregations (incl moving averages & Holt Winters) One system for both logs and metrics #velo Timelion UI, Grafana Great ecosystem: e.g. alerting tools 37

38 Packetbeat 38

39 Supported traffic decoders Thrift DNS ICMP AMQP + Add your own 39

40 Unknown traffic, use flows Look into data for which we don t understand the application layer protocol TLS Protocols we don t yet support Get data about IP / TCP / UDP layers number of packets & bytes retransmissions inter-arrival time 40

41 Packetbeat Monitor traffic exchanged between your containers 41

42 Monitor outside containers App1 App2 App3 Packetbeat Host traffic exchanged between your containers 42

43 Demo: Metricbeat, Filebeat, Packetbeat Multiple data types, one view in Kibana 43

44 Thank you github.com/elastic/beats #elasticbeats #beats on freenode 44

45