Wrangling Logs with Logstash and ElasticSearch

Transcription

1 Wrangling Logs with Logstash and ElasticSearch Nate Jones & David Castro Media Temple OSCON 2012

2 Why are we here?

3 Size Quantity Efficiency

4 Access Locality Method Filtering

5 Grokability Noise Structure Metrics

6 Use Case: Mail Logs

7 Size 30 mail servers 2G logs / day / server 60GB / day total 1.8 TB / month 21 TB / year 1 billion log lines per week

8 Access Front-line, easy access No SSH Shareable

9 Grokability Operational Did the get delivered? Why was the message marked as SPAM? Are messages being rejected? Metrics What's the inbound/outbound message rate? How often are we seeing particular errors?

10 The Solution

11 Overview

12 Overview

13 Logstash Overview 1. Parse log line 2. Transform/extract 3. Structure and send JSON

14 Logstash Parsing Log line input T20:00: :00 mail01 spamd[2478]: spamd: clean message (-3.4/5.0) for nobody:93 in 0.0 seconds, 886 bytes. JSON output { "@timestamp" : " T06:44: Z", "@tags" : [], "@fields" : {}, "@source_path" : "/client/ :40010", "@source" : "tcp:// :6999/client/ :40010", "@source_host" : " ", "@message" : " T20:00: :00 mail01 spamd[2478]: spamd: clean message (-3.4/5.0) for nobody:93 in 0.0 seconds, 886 bytes.", "@type" : "maillog" }

15 Logstash Parsing grok { type => "maillog" pattern => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:host} %{SYSLOGPROG:service}: %{GREEDYDATA:message}" } mutate { type => "maillog" # replace the timestamp, correcting import timestamp replace => ["@timestamp", "%{timestamp}"] # replace the message sans-timestamp/host/service replace => ["@message", "%{message}"] }

16 Logstash Parsing { "@timestamp" : " T20:00: :00", "@tags" : [], "@fields" : { "pid" : [ "2478" ], "service" : [ "spamd[2478]" ], "program" : [ "spamd" ], "host" : [ "mail01" ] }, "@source_path" : "/client/ :39998", "@source" : "tcp:// :6999/client/ :39998", "@source_host" : " ", "@message" : "spamd: clean message (-3.4/5.0) for nobody:93 in 0.0 seconds, 886 bytes.", "@type" : "maillog" }

17 RabbitMQ Overview Message Queue AMQP Clustered

18 Elasticsearch Intro Index in Lucene shards Cluster-able Fault tolerant

19 Elasticsearch Head

20 Elasticsearch Browser

21 Kibana Intro User friendly front-end to elasticsearch Search log lines Graph, score, trend Streaming dashboard

22 Kibana Queries Question How many errors of a particular type are we seeing in the logs? Denied"

23 Kibana Queries

24 Kibana Queries Question Why did the mail for user X get marked as SPAM?

25 Kibana Queries

26 Kibana Queries Question How many messages are being rejected due to the sending host being listed in an RBL?

27 Kibana Queries

28 Kibana Queries Question How many log messages do we have for a specific mail host?

29 Kibana Queries

30 Report Card

31 Size Quantity Efficiency

32 Access Locality Method Filtering

33 Grokability Noise Structure Metrics

34 Next Steps Push more stats into graphite Further breaking down log messages More stuff

35 Everything you need Instructions and software Puppet code and slides Local wifi share: logwrangler (guest/guest)

36 Demo Netcat port for Logstash RabbitMQ Elasticsearch Kibana

37 Contact Info Nate David

38 Questions?