Unifying logs and metrics data with Elastic Beats. Monica Sarbu Team lead, Elastic Beats

Transcription

1 Unifying logs and metrics data with Elastic Beats Monica Sarbu Team lead, Elastic Beats #

2 Who am I Team lead at Elastic Beats Software engineer Joined Elastic 1 year 2

3 Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 3

4 Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 4

5 Lightweight shippers Lightweight application Written in Golang Install as agent on your servers No runtime dependencies Single purpose 5

6 Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 6

7 All kinds of operational data Filebeat collects logs Winlogbeat collects Windows event logs Packetbeat collects insides from the network packets Topbeat collects system statistics like CPU usage, disk usage, memory usage per process, etc Metricbeat not released collects metrics by interrogating periodically external services 7

8 Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch 8

9 In Elasticsearch.. you are storing the raw value You have the ability to ask and answer questions that you didn t think about when the data was stored! Felix Barnsteiner #

10 The Elastic Stack 10

11 #

12 Packetbeat Captures insights from network packets 12

13 Sniffing the network traffic Client Server sniff sniff Copy traffic at OS or hardware level ZERO latency overhead Not in the request/response path, cannot break your application 13

14 Sniffing use cases Security Intrusion Detection Systems Troubleshooting network issues Troubleshooting applications Performance analysis 14

15 Monitor the network traffic with OSS tools ssh to each of your server start a trace using tcpdump on each of your server download trace from each server to a common location merge all traces analyze it with Wireshark 15

16 The Problem you have lots of servers challenging to see the traffic exchanged between your servers Packetbeat makes it easy 16

17 Packetbeat overview It does all of this in real time directly on the target servers capture network traffic decodes network traffic correlates request & response into transactions extract measurements send data to Elasticsearch 17

18 Packetbeat: Available decoders HTTP MySQL Thrift-RPC Memcache DNS (community) AMQP (community) PostgreSQL MongoDB (community) NFS (community) Redis ICMP (community) + Add your own 18

19 Packetbeat: Configuration # Network interfaces where to sniff the data interfaces: device: any # Specify the type of your network data protocols: dns: ports: [53] http: ports: [80, 8080, 8081, 5000, 8002] mysql: ports: [3306] 19

20 20

21 21

22 22

23 23

24 24

25 25

26 26

27 27

28 28

29 29

30 30

31 Packetbeat flows flows: # network flow timeout timeout: 30s # reporting period period: 10s Look into data for which we don t understand the application layer protocol TLS Protocols we don t yet support Get data about IP / TCP / UDP layers number of packets retransmissions inter-arrival time # 31

32 32

33 33

34 34

35 Filebeat Collects log lines 35

36 Filebeat overview Simple log forwarder that sends the log lines to Elasticsearch Successor of Logstash Forwarder It remembers how far it read, so it never loses log line Reads the log files line by line It doesn t parse the log lines! 36

37 Filebeat: Parse logs with Logstash Filebeat Elasticsearch Logstash Other systems Filebeat sends out unparsed log lines Use filters like Grok, mutate, geoip to parse the log lines Combine the filters with conditionals or create custom filters in ruby Forward data to other systems using the Logstash output plugins 37

38 Filebeat: Parse logs with Ingest Node Filebeat Elasticsearch Ingest node plugin is available starting with Elasticsearch alpha1 Filebeat sends out unparsed log lines directly to Elasticsearch Use Ingest Node processors to parse the log lines Easier to setup 38

39 Filebeat: Configuration Configure prospectors to forward the log lines filebeat: # List of prospectors to fetch data. prospectors: # Type of files: log or stdin - input_type: log # Files that should be crawled and fetched. paths: - /var/log/apache2/* # File encoding: plain, utf-8, big5, gb18030, encoding: plain 39

40 40

41 41

42 Filebeat extra power Multiline multiline: # Sticks together all lines # that don t start with a [ pattern: ^\[ negate: true match: after Sticks together related log lines in a single event For all those long exceptions Can also be done by Logstash, but it s sometimes easier to configure the patterns closer to the source # 42

43 43

44 #

45 Filebeat extra power JSON logs json: keys_under_root: false message_key: message overwrite_keys: false add_error_key: false application logs in JSON format you don t have to choose what data to include in the log line don t need to use grok filters from Logstash to parse the application logs # 45

46 46

47 47

48 Filebeat extra power Basic filtering # Only send lines starting with # ERR or WARN include_lines: [ ^ERR, ^WARN ] # Exclude lines containing # a keyword exclude_lines: [ Request received ] # Exclude files all together exclude_files: [.gz$ ] Because removing stuff at the source is more efficient Flexible Whitelist + Blacklist regexp log line filtering Efficient log files filtering (excluded files are never opened) Works on multiline too # 48

49 Winlogbeat Collects Windows Event logs 49

50 Winlogbeat overview Sends out unparsed Windows event logs Remembers how far it read, so it never loses any Windows event logs Use Ingest Node or Logstash to parse the Windows event logs 50

51 Winlogbeat: Configuration Specify the event logs that you want to monitor winlogbeat: #list of event logs to monitor event_logs: - name: Application - name: Security - name: System 51

52 52

53 Topbeat Collects system statistics 53

54 Topbeat overview Like the Unix top command but instead of printing the system statistics on the screen it sends them periodically to Elasticsearch Works also on Windows 54

55 Topbeat: Exported data System wide Per process Disk usage system load total CPU usage CPU usage per core Swap, memory usage state name command line pid CPU usage memory usage available disks used, free space mounted points 55

56 Topbeat configuration Specify the system statistics that you want to monitor topbeat: # how often to send system statistics period: 10 # specify the processes to monitor procs: [".*"] # Statistics to collect (all enabled by default) stats: system: true process: true filesystem: true 56

57 57

58 #

59 #

60 #

61 #

62 #

63 #

64 #

65 #

66 Metricbeat in progress Collects periodically metrics from external systems. 66

67 Metricbeat: how it works Periodically polls monitoring APIs of various services Groups performance data into documents Ships them to Elasticsearch 67

68 Metricbeat: A module for each metric type apache module mysql module redis module system + module Metricbeat 68

69 Metricbeat: It is also a library! df module Metricbeat Use the Metricbeat infrastructure, to create a standalone Beat You can create a Beat with a single module that exports your custom data Can use the built in Metricbeat modules github.com/ruflin/df2beat 69

70 Metricbeat module vs standalone Beat Metricbeat module Standalone Beat Contributed via PR to the elastic/beats Github repository Officially supported Supports common systems Docker based integration tests In a separate Github repository Supported by the community Supports specialized systems Optional Docker based integration tests 70

71 Provide a platform to make it easier to build custom Beats on top of it 71

72 Beats platform Beat 1 Beat 2 Beat 3 + libbeat 72

73 libbeat libbeat Outputs Written in Go Provide common functionality for reading configuration files, for handling CLI arguments, for logging Makes sure reliably send the data out Provide things like encryption, authentication with certificates Has support for different outputs: Elasticsearch, Logstash, Redis, Kafka 73

74 # Community Beats

75 Community Beats Elastic Beats Community Beats libbeat Standalone projects Written in Go Use libbeat Concentrate only on collecting the data Solve a specific use case Collect, Parse & Ship 75

76 Official vs Community Beats Official Beats Community Beats In the elastic/beats Github repository In another Github repository Officially supported Synced releases with the whole stack Supported by the community Releases at any time 76

77 20 COMMUNITY BEATS Sending all sorts of data to Elasticsearch 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5 Factbeat 6 Hsbeat 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat Pingbeat 12 Redisbeat 13 Unifiedbeat 14 Uwsgibeat 15 Flowbeat 16 Lmsensorsbeat 17 Twitterbeat 18 Upbeat 19 Wmibeat 20 Packagebeat

78 Pingbeat input: # Loop every 5 seconds period: 5 # Use raw sockets for ping # Requires root! privileged: true # Whether to perform IPv4/v6 pings useipv4: true useipv6: false # List targets under the tag # you want assigned to targets: # tag: google google: - google.com.au - google.com You know, for pings Sends ICMP (v4 or v6) pings periodically to a list of hosts Can send also UDP pings (no root required) Resolves DNS Records RTT # 78

79 Pingbeat output { } "@timestamp": " T11:02:22.675Z", "beat": { "hostname": "Tudors-MBP", "name": "Tudors-MBP" }, "count": 1, "rtt": , "tag": "google", "target_addr": " ", "target_name": "google.com.au", "type": "pingbeat" 79

80 Execbeat execbeat: execs: # Each - Commands to execute. - # Cron expression # Default is every 1 minute. cron: "@every 10s" # The command to execute command: echo args: "Hello World" document_type: jolokia Run any command Accepts cron expressions Sends stdout and stderr to Elastic search Use Logstash and Grok to further parse the output fields: host: test2 # 80

81 Execbeat output { } "@timestamp": " T11:59:36.007Z", "beat": { "hostname": "Tudors-MBP", "name": "Tudors-MBP" }, "exec": { "command": "echo", "stdout": "Hello World\n" }, "fields": { "host": "test2" }, "type": "jolokia" 81

82 Dockerbeat input: # In seconds, defines how often to # read server statistics period: 5 # Define the docker socket path # By default, this will get the # unix:///var/run/docker.sock socket: Docker Monitoring Uses the Docker API Exports per container stats about: CPU Memory Disk Network IO access Log # 82

83 Dockerbeat output { "@timestamp": " T12:44:56.136Z", "containerid": "17021c571d69fe4e93ee395b129c0f073d8aed6d618c9d0d805f68e0b66b2c3f", "containername": "kibana", "memory": { "failcnt": 0, "limit": , "maxusage": , "usage": , "usage_p": }, "type": "memory" } 83

84 Nagioscheckbeat input: checks: - name: "disks" cmd: "plugins/check_disk" args: "-w 80 -c 90 -x /dev" period: "1h" - name: "load" cmd: "plugins/check_load" args: "-w 5 -c 10" period: "1m" Run Nagios checks Can execute any Nagios plugin Execution period configurable per check Sends alerts (Warning/Critical) to Elasticsearch Sends performance data to Elasticsearch # 84

85 Nagioscheckbeat output { } "@timestamp": " T18:56:33.933Z", "args": "-w 5 -c 10", "cmd": "/usr/lib64/nagios/plugins/check_load", "count": 1, "message": "OK - load average: 0.16, 0.05, 0.06", "status": "OK", "took_ms": 14, "type": "nagioscheck" 85

86 Provide a platform to make it easier to build custom Beats on top of it 86

87 Beat generator Generate the boilerplate code for you $ pip install cookiecutter $ cookiecutter project_name [Examplebeat]: Mybeat github_name [your-github-name]: monicasarbu beat [examplebeat]: mybeat beat_path [github.com/your-github-name]: github.com/ monicasarbu full_name [Firstname Lastname]: Monica Sarbu 87

88 Beats Packer Cross-compiles to all our supported platforms Produces RPMs, DEBs, Same tools that we use to build the official Elastic Beats Can be executed from Travis CI 88

89 Multiple data types, one view in Kibana metrics logs transactions logs flows system stats logs transactions metrics flows logs metrics system stats flows metrics 89

90 Monitor MySQL with Elastic Stack stats Kibana Elasticsearch queries slow queries mysql log mysql Metricbeat Filebeat Packetbeat 90

91 Monitor web server with Elastic Stack mysql & apache stats Kibana Elasticsearch queries & HTTP transactions slow queries apache logs mysql apache log mysql http Metricbeat Filebeat Packetbeat 91

92 # Thank you

93 Want to hear more about Logstash? Don t miss Ingest Logs with Style by Pere Urbon-Bayes Thursday 12:00pm - 1:00pm in MOA 05 # 93

94 Q&A Find us on: github.com/elastic/beats #elasticbeats #beats on freenode Or Here. In Real Life! #

95 Please attribute Elastic with a link to elastic.co Except where otherwise noted, this work is licensed under Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. # 95