Ingesting Logs with style. What has been cooking lately in Logstash world.

Transcription

1 Ingesting Logs with style What has been cooking lately in Logstash world. #

2 $whoami Pere Urbon-Bayes (Software Engineer since ever) Have always worked with databases, data and analytics. When not coding I enjoy my time with my wife and kid. I also enjoy movies and tv series, and used to like running, too. Basically, I m doing everything to enjoy live. 2

3 3 Life demo?

4 4 Ingredients

5 5 Metrics

6 What is the current Logstash throughput? Worried about OOM? ask for memory consumption. Broken by plugin instance. Logstash is blocked?, ask for a hot threads dump. Per instance and node. to be continued 6

7 Web API resources Current web api resources:

8 8

9 Next: Monitoring UI 9

10 10 Configuration Management

11 Step 1: Config Reloading Why? How? Previously: Any config change made to file required a process restart Feedback loop for development/ testing slow Processing pipeline must be long living File watched for changes or SIGHUP triggers reload Current Pipeline stopped Config Validated New Pipeline started - no process restart 11

12 role: frontend-logs 1 Create a role, upload config 12

13 13 The Java Event

14 The java reincarnation Logstash is based on JRuby, Ruby on top of the JVM Some core components are being rewritten in plain java. First step: the event representation. Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0. Provide the foundations for upcoming persistence queues, this change will be backwards due to the nice introp between JRuby and Java. Will enable not just ruby plugins, but using any JVM languages And not only about this, but performance.. 14

15 15 logstash-benchmarks.elastic.co

16 16 The other changes

17 17

18 Native offline plugins management support Build your plugins bundle to be used in offline environments bin/logstash-plugin pack bin/logstash-plugin unpack bin/logstash-plugin install --local Use your own private plugins server Geminabox, Gemirro, Artifactory, Gemfury, Edit the Gemfile to use the new server. Manage plugins in offline environments made easier 18

19 19 Pipeline, the next generation

20 The pipeline components I F O I F O 20

21 21 Filebeat, the new log harvester gen

22 Long life to Logstash Forwarder, welcome Filebeat 22 LSF is deprecated, next generation harvester is Filebeat. Written in Go and configured in YAML. Available as deb, rpm, linux (32 and 64bit), mac and windows binaries. Similar feature sets as LSF, including: Send data direct to Elasticsearch Send data to Logstash (logstah-forwarder protocol 2.0) Security as first class citizen HTTPS for Elasticsearch TLS for Logstash

23 Configuring Filebeat Prospectors ################### Filebeat Configuration Example ######################### filebeat: prospectors: - paths: - /Users/purbon/lib/moves/*.tcx input_type: log multiline: pattern: ^<\?xml negate: true match: after max_lines: timeout: 5s 23

24 Configuring Filebeat Outputs ################### Filebeat Configuration Example ######################### output: logstash: hosts: ["localhost:5044"] index: filebeat elasticsearch: hosts: ["localhost:9200"] 24

25 Migration to Filebeat Configuration file changed from JSON to YAML The registry file has been changed, this file track the data transfer status Recommendation: Use both in parallel until data sync is complete Command line options where moved as settings Mostly all removed, filebeat provides default values in settings. The new beats input is required in the Logstash side. Latest version enable LSF to talk to the beats input plugins, reducing migration friction. 25

26 # Plugins Community Maintainers Program

27 Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson # 27 Magnus Bäck Jurgens du Toit Philippe Weber Russell Savage

28 28

29 29 Breaking changes!!!

30 This that might break up for 5.0 New Event API proposal [#5141] bin/plugin is now bin/logstash-plugin [#4871] Require Java 8 as default [#3877] Fieldref StringInterpolation and Fixnum/Float values [#5114] Add support for x-pack s [#4872] Runner cleanup [#3148, #3860] Use settings file to complement command line arguments [#4401] 30

31 31 Demo time

32 QA thanks! Pere Urbon-Bayes Software Engineer Elastic #