Monitor your containers with the Elastic Stack Monica Sarbu
Monica Sarbu Team lead, Beats team monica@elastic.co 3
Monitor your containers with the Elastic Stack
Elastic Stack 5
Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
Multiple data types, one place Docker metrics Docker logs Apache logs MySQL transactions Redis logs flows diskio Redis metrics Docker metrics CPU % memory % HTTP transactions MySQL logs flows Redis transactions filesystem 7
Central point for your distributed infrastructure 8
The Beats 30+ other community Beats shipping 9
Filebeat 10
Filebeat Tails log files, without parsing them At least once guarantees, handles backpressure Extra powers: Multiline JSON logs Filtering 11
Parse log lines with Ingest Node I N G E S T 12
Parse log lines with Logstash I N G E S T 13
Filebeat Back pressure handling 14
Why back-pressure is key? 15
Synchronous sending registry file acked read read stream of log lines batch of messages ack 16
This means.. Filebeat adapts its speed automatically to as much as the next stage can process But: be aware when benchmarking 17
When the next stage is down.. Filebeat patiently waits Log lines are not lost It doesn t allocate memory, it doesn t buffer things on disk 18
Filebeat Collect container logs 19
Docker logging drivers https://docs.docker.com/engine/admin/logging/overview/ 20
Centralize Docker logs: option 1/522 Use the Docker gelf driver and the Logstash-gelf-input Pros: No shipper to install, send directly to Logstash Cons: UDP based, no delivery guarantees, no congestion control 21
Centralize Docker logs: option 2/522 Use the Docker JSON driver, use Filebeat with the JSON support Pros: Simple (default driver) Easy to add container metadata (name, labels, etc.) `docker logs` works Cons: JSON driver can slow down Docker 22
Centralize Docker logs: option 3/522 Use the Docker syslog driver, and a local syslog server, then Filebeat for shipping Pros: Good control over the path where the files are written, rotation strategies, etc. Cons: you need to manage the syslog server metadata is serialized as string, needs to be deserialized again (opportunity for mistakes) multiline is difficult because data from containers can be mixed 23
Centralize Docker logs: option 4/522 Use the Docker journald driver then Filebeat for shipping Pros: journald is often already available convenient support for metadata `docker logs` works Cons: Filebeat doesn t yet support journald (a Journalbeat exists, however) 24
Centralize Docker logs: option 5/522 Mount a volume and have your app write logs into the volume Pros: If your app can rotate it s own logs, it s very easy to setup Scales well Cons: Difficult to pass metadata 25
Centralize Docker logs: conclusion json driver, syslog driver, and shared volume are pretty good options today journald driver might be better options in the future 26
Metricbeat new in 5.0 27
One Metricbeat module for each service + Add your own 28
Metricbeat system module CPU Mem diskio filesystem load network cores processes 29
Metricbeat Collect container metrics 30
Querying the Docker API in progress Dedicated Docker module Has access to container names and labels Easy to setup Offers: CPU and memory Docker container information network (in/out bytes, dropped) diskio (reads/writes) status of containers (# of stopped, running, etc) 31
Reading cgroup data from /proc/ Doesn t require access to the Docker API (can be a security issue) Works for any container runtime (Docker, rkt, runc, LXD, etc.) Part of the system module Automatically enhances process data with cgroup information Cannot get the container name and labels 32
Run as a container App1 App2 App3 Host 33
Elasticsearch as time series DB 34
Elasticsearch BKD trees Added for Geo-points faster to index #velo faster to query more disk-efficient more memory efficient 35
Float values On Disk Usage in kb 80000 half floats 70000 60000 scaled floats (using a scaling factor) - great for 50000 things like percentage 40000 points 30000 20000 10000 0 float half float scaled float (factor = 4000) scaled float (factor = 100) Points disk usage (kb) docs_values disk usage (kb) 36
Why Elasticsearch for time series Horizontal scalability. Mature and battle tested cluster support. Flexible aggregations (incl moving averages & Holt Winters) One system for both logs and metrics #velo Timelion UI, Grafana Great ecosystem: e.g. alerting tools 37
Packetbeat 38
Supported traffic decoders http:// Thrift DNS ICMP AMQP + Add your own 39
Unknown traffic, use flows Look into data for which we don t understand the application layer protocol TLS Protocols we don t yet support Get data about IP / TCP / UDP layers number of packets & bytes retransmissions inter-arrival time 40
Packetbeat Monitor traffic exchanged between your containers 41
Monitor outside containers App1 App2 App3 Packetbeat Host traffic exchanged between your containers 42
Demo: Metricbeat, Filebeat, Packetbeat Multiple data types, one view in Kibana 43
Thank you github.com/elastic/beats discuss.elastic.co @elastic #elasticbeats #beats on freenode 44