Are you visualizing your logfiles? Bastian Widmer

Transcription

1 Are you visualizing your logfiles? Bastian Widmer

2 Visualizing Logfiles with ELK Stack Bastian Widmer

3 Hola Com estàs? Bastian / bastianwidmer.ch DrupalCI: Modernizing Testbot Initiative Chief YoloOps Evangelist

4 Agenda 1 Introduction 2 ELK Stack 3 Architecture 4 Tools! 5 6 Automation P22N - Performance Optim

5 Visualizing Logfiles, why?

6 Can you check the errors from yesterday between and 15.07

7

8 Visualization > Plaintext

9

10 Patch deployed, instant feedback!

11 Visualization > Plaintext

12 VISUALIZATION > Plaintext

13 Do you log to database? dblog?

14 Okay for one site, but what if you have 70+ sites logging into your database?

15 Use Cases Audit Trail - Who changed what? Content Modules Errors - Fixing errors and getting instant feedback by easy readable graphs Billing Application Speed Deep Inspection (TOR Nodes)

16

17

18 ELK Stack!

19 ELK Stack! Elasticsearch Logstash Kibana

20 Sidenote : Things move fast! Even with minor releases

21 Elasticsearch

22 Elasticsearch Java Search and Index Distributed Copies & Shards Clustering (Zen Discovery - Multi/Unicast) API JSON / RESTful Apache Lucene Disk-Based Shard Allocation

23 Elasticsearch Index like a Database Replica Copies for Fault Tolerance Shard Lucene Instance which indexes the Data see :

24 Elasticsearch { "status" : 200, "name" : "es-03", "cluster_name" : "cluster01", "version" : { "number" : "1.7.1", "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19", "build_timestamp" : " T09:54:16Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }

25 Elasticsearch

26 ElasticSearch Plugins New Integrated Plugin System Bundles Plugins with Elasticsearch bin/plugin -install YOURPLUGIN"

27 ElasticSearch Security Speak with me: I will hereby solemnly swear not to expose my Elasticsearch Server to public, never-ever! Elastic Shield - Provides Security (Subscription Feature)

28 ElasticSearch Security - cheap Run Elasticsearch bound to localhost use an internal network ssh elasticsearch@elasticsearch.amazee.io -N -L 9200: :9200'

29 Thankmelater Security can be an issue curl -XDELETE curl -XDELETE action.destructive_requires_name: true

30 Marvel Shows Cluster Health and Real-Time Analysis Free during development product Deep insights into index creation across cluster, routing decisions and much more

31 Logstash

32 Did the Catalan Citizens invent Logstash?

33

34 Logstash Multiple Input / Multiple Output Centralize and Process Log Data Collect Parse Store / Forward

35 The life of an event Input Filters Codecs Output

36 Logstash JRuby* > FlatJAR Release is gone Instead of running java -jar logstash.jar bin/ logstash Contrib Plugins Daily Indices! * see

37 Input File Syslog Redis logstash-forwarder (former Lumberjack)

38 Filters Grok Mutate Drop Clone GeoIP (!!!)

39 Outputs Elasticsearch File / S3 Graphite StatsD

40 Logstash 1 input {! 2 stdin { }! 3 }! 4! 5 output {! 6 stdout {! 7 codec => rubydebug! 8 }! 9 }!!

41 Logstash 1 vagrant@precise64$./logstash agent -f 1_simpleconfig.cfg! 2 very important log message!! 3 {! 4 "message" => "very important log message!",! 5 "@version" => "1",! 6 "@timestamp" => " T16:18:02.952Z",! 7 "host" => "precise64"! 8 }

42 Logstash 1 input {! 2 stdin { }! 3 }! 4 output {! 5 elasticsearch{! 6 host => " "! 7 }! 8 stdout {! 9 codec => rubydebug! 10 }! 11 }

43 Logstash 1 input {! 2 file {! 3 path => "/var/log/syslog"! 4 start_position => beginning! 5 }! 6 }! 7! 8 output {! 9 stdout {! 10 codec => rubydebug! 11 }! 12 elasticsearch{! 13 host => " "! 14 }! 15 }

44 Kibana

45

46 Some history Ruby PHP Just Javascript (the crowd applauds) Node Webserver and Javascript (Kibana 4)

47 Kibana 4 D3.js - more fancyness More complex backend Much better flexibility Analytics and Aggregations

48 Architecture

49 Architecture Shipper Shipper Broker Indexer Search and Storage Shipper

50 Architecture Shipper Shipper Broker Indexer Search and Storage Shipper Syslog

51 Architecture Shipper Shipper Broker Indexer Search and Storage Shipper Syslog Logstash

52 Architecture Shipper Shipper Broker Indexer Search and Storage Shipper Syslog Logstash Elasticsearch

53 But, Bastian

54 Architecture The real deal!

55 Logstash-Forwarder Written in Go Lightweight utility to forward logs to logstash Low resource usage TLS/SSL Encrypted Transfer

56 Architecture nginx.log Shipper auth.log Shipper Shipper Broker Indexer Indexer Search and Storage drupal.log Shipper Logstash-Forwarder Logstash Redis Logstash Elasticsearch

57 Architecture nginx.log Shipper auth.log drupal.log And from here you can go crazy! Shipper Shipper Broker Indexer Indexer Shipper Search and Storage Logstash-Forwarder Logstash Redis Logstash Elasticsearch

58 Architecture High-Available nginx.log Shipper Broker Indexer auth.log Shipper Search and and Storage drupal.log Shipper Broker Indexer Logstash-Forwarder Logstash Redis Logstash Elasticsearch

59 But, Bastian!!!

60 No!

61 High Available Setup with Rocketfuel! nginx.log HAProxy Shipper Broker Indexer auth.log Forwar der KeepaliveD Search and and Storage drupal.log HAProxy Shipper Broker Indexer Logstash Forwarder Logstash Redis Logstash Elasticsearch

62 Tools! (because anyone needs a bit help)

63 Elasticsearch Head -install mobz/elasticsearch-head

64 Elasticsearch Kopf./plugin -install lmenezes/elasticsearch-kopf

65 Curator Time Series Indices? THIS IS THE TOOL! Close Indexes Delete (by space or time) Disable Bloom Filter Optimize / ForceMerge

66 Curator Time Series Indices? THIS IS THE TOOL! Close Indexes Curator Delete (by space or time) Perfect for Time Series Indexes Disable Bloom Filter Optimize / ForceMerge

67 Curator Close indices older than 14 days, delete indices older than 30 days curator --host my-elasticsearch -d 30 -c 14 Disable bloom filter for indices older than 2 days, close indices older than 14 days, delete indices older than 30 days: curator --host my-elasticsearch -b 2 -c 14 -d 30

68 Curator 1 root@precise64:/home/vagrant# curator -c 7 -b 2 -d 10! T17:57: INFO main:333 Job starting...! T17:57: INFO _new_conn:180 Starting new HTTP connection (1): localhost! T17:57: INFO log_request_success:49 GET [status:200 request:0.002s]! T17:57: INFO main:359 Deleting indices older than 10 days...! T17:57: INFO log_request_success:49 GET expand_wildcards=closed [status:200 request:0.007s]! T17:57: INFO find_expired_indices:209 logstash is 10 days, 0:00:00 above the cutoff.! T17:57: INFO index_loop:309 DELETE index operations completed.! T17:57: INFO main:364 Closing indices older than 7 days...! T17:57: INFO log_request_success:49 GET expand_wildcards=closed [status:200 request:0.001s]! T17:57: INFO find_expired_indices:209 logstash is 7 days, 0:00:00 above the cutoff.! T17:57: INFO index_loop:309 CLOSE index operations completed.! T17:57: INFO main:369 Disabling bloom filter on indices older than 2 days...! T17:57: INFO log_request_success:49 GET expand_wildcards=closed [status:200 request:0.002s]! T17:57: INFO find_expired_indices:209 logstash is 2 days, 0:00:00 above the cutoff.! T17:57: INFO index_loop:309 DISABLE BLOOM FILTER FOR index operations completed.! T17:57: INFO main:379 Done in 0:00: !

69 BigDesk bigdesk.org Elasticsearch Plugin

70 Grok Filters?! 1 grok {! 2 match => { "message" => "<% {POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} % {SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[% {POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }! 3 add_field => [ "received_at", "%{@timestamp}" ]! 4 }!

71 Grok Debugger grokdebug.herokuapp.com

72 The Logstash Book logstashbook.com

73 Elasticsearch : The Definitive Guide /en/elasticsearch/guide/current/index.html

74 Performance Optimisation or short P22N

75 Performance Remember: It s just Java File Descriptors >32k Give enough Memory (-Xms -Xmx Values) Leverage File System Cache

76 Automation!

77 Puppet Modules elasticsearch/elasticsearch (PuppetLabs Approved) elasticsearch/logstashforwarder elasticsearch/logstash

78 Puppet class { 'elasticsearch': repo_version => '1.7', manage_repo => true, java_install => true, config => { 'cluster.name' => 'cluster01' }, datadir => '/var/lib/elasticsearch/' }! elasticsearch::instance { 'es-01': config => { 'node.name' => 'es-01' } }

79 Take Home Centralized Logging saves time Is fun with the ELK Stack Gives you Graphs to Interpret can you check the errors from yesterday between and get s A LOT easier Start here tomorrow:

80 Thank you for having me here! Slides: Feedback:

81 Friday Sprints - Join us!

82 Legal (because Legal ) Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. Kibana is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. Elastic, Logstash and Marvel are trademarks of Elasticsesarch BV

83 Images Used Elk : Architecture : VideoWall : /photo/1 Tió de Nadal (CC-BY-SA 3.0)