Monitor your infrastructure with the Elastic Beats. Monica Sarbu

Transcription

1 Monitor your infrastructure with the Elastic Beats Monica Sarbu

2 Monica Sarbu Team lead, Beats team Twitter: 2

3 Monitor your servers Apache logs 3

4 Monitor your servers Apache logs memory % CPU % 4

5 Monitor your servers Apache logs memory % Apache metrics CPU % 5

6 Monitor your servers Apache logs memory % Apache metrics CPU % HTTP transactions 6

7 Multiple data types, one storage Apache logs memory % Apache metrics CPU % HTTP transactions 7

8 Scalable from day 1 8

9 Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch

10 Kibana Elastic Stack Elasticsearch Beats Logstash 10

11 The Beats 30+ other community Beats shipping 11

12 Filebeat 12

13 tail -f

14 tail -f over the Network

15 tail -f over the Network with extra powers

16 Multiline JSON logs Filtering

17 Send raw log lines { } message: GET /index.html , 17

18 Parse log lines by defining grok patterns I N G E S T or 18

19 Grok patterns { } message: GET /index.html , %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} 19

20 After parsing { } message: GET /index.html client: , method: GET, request: /index.html bytes: 15824, duration:

21 Handle back-pressure

22 Why back-pressure is key? 22

23 Synchronous sending registry file acked read read stream of log lines batch of messages ack 23

24 Filebeat adapts its speed automatically to as much as the next stage can process

25 When next stage is down Filebeat patiently waits Log lines are not lost It doesn t allocate memory It doesn t buffer log lines on disk 25

26 At-Least-Once delivery

27 No such think as exactly once #velo 27

28 batch of messages ack batch of messages #velo same batch of messages ack duplicates! 28

29 Filebeat Collect container logs 29

30 Docker logging drivers 30

31 001 Gelf driver + Logstash Pros: Cons: logs send directly to Logstash UDP based, no delivery guarantees, no congestion control 31

32 010 json-file driver + Filebeat Pros: Cons: Simple to setup as it s the default driver json-file driver can slow down Docker container Easy to add container metadata (name, labels, etc.) `docker logs` works 32

33 011 Syslog driver + Syslog server + Filebeat Pros: Cons: Good control over the path where the files are you need to manage the syslog server written, rotation strategies, etc. metadata is serialized as string, needs to be deserialized again multiline is difficult because data from containers can be mixed 33

34 100 Journald driver + Filebeat Pros: Cons: journald is often already available Filebeat doesn t yet support journald convenient support for container metadata (name, labels, etc.) You can use the community Beat, Journalbeat `docker logs` works 34

35 101 Shared volume + Filebeat Pros: Cons: If your app can rotate it s own logs, it s very easy to setup Difficult to pass container metadata (name, labels, etc.) Scales well 35

36 Conclusion At-least-once guarantees and handle No guarantees: back-pressure: Gelf driver + Logstash json-file driver + Filebeat Fluentd driver + Logstash Syslog driver + Filebeat Shared volume + Filebeat Journald driver + Filebeat (in the future) 36

37 Metricbeat new in

38 One Metricbeat module for each service + Add your own 38

39 Metricbeat system module CPU Mem diskio filesystem load network cores processes 39

40 Metricbeat Collect container metrics 40

41 Querying the Docker API CPU and memory Docker container information network (in/out bytes, dropped) diskio (reads/writes) status of containers (# of stopped, running, etc) 41

42 Docker module in Metricbeat in progress Get container metrics by querying the Docker API Has access to container names and labels Easy to setup 42

43 Reading cgroup data from /proc/ Doesn t require access to the Docker API (can be a security issue) Works for any container runtime (Docker, rkt, runc, LXD, etc.) Cannot get the container name and labels only the container ID 43

44 System module + cgroup data if cgroup option is enabled (by default is disabled) Automatically enhances process data with cgroup information 44

45 Run as a container App1 App2 App3 Host 45

46 Elasticsearch as time series DB 46

47 Elasticsearch BKD trees Added for Geo-points faster to index #velo faster to query more disk-efficient more memory efficient 47

48 Float values On Disk Usage in kb half floats scaled floats (using a scaling factor) - great for things like percentage points float half float scaled float (factor = 4000) scaled float (factor = 100) Points disk usage (kb) docs_values disk usage (kb) 48

49 Why Elasticsearch for time series Horizontal scalability. Mature and battle tested cluster support. Flexible aggregations (incl moving averages & Holt Winters) One system for both logs and metrics #velo Timelion UI, Grafana Great ecosystem: e.g. alerting tools 49

50 Packetbeat 50

51 How Packetbeat works capture network traffic decodes network traffic correlates request & response into transactions send transactions to Elasticsearch 51

52 Supported traffic decoders Thrift DNS ICMP AMQP + Add your own 52

53 Unknown traffic, use flows Look into data for which we don t understand the application layer protocol TLS Protocols we don t yet support Get data about IP / TCP / UDP layers number of packets & bytes retransmissions inter-arrival time 53

54 Monitor traffic exchanged by containers App1 App2 App3 Packetbeat Host traffic exchanged between your containers 54

55 Demo: Metricbeat, Filebeat, Packetbeat Multiple data types, one view in Kibana 55

56 Thank you github.com/elastic/beats #elasticbeats #beats on freenode 56