Fluentd. Open Source Data Collector. Eduardo Jan 23, 2016 Scale14x, Pasadena!

Transcription

1 Fluentd Open Source Data Collector Jan 23, 2016 Scale14x, Pasadena! Eduardo

2 spread the word! #scale14x

3 About Me Eduardo Silva Github & Twitter Personal Treasure Data Open Source Engineer Fluentd / Fluent Bit Projects Monkey HTTP Server Duda I/O

4 Logging

5 Logging Matters Pros Application status Debugging General information about anomalies: errors Troubleshooting / Support Local or Remote (network)

6 Logging Matters From a business point of view Input data Analytics User interaction / behaviors Improvements

7 Assumptions

8 Logging Matters Assumptions I have enough disk space I/O operations will not block Log messages are human readable My logging mechanism scale

9 Logging Matters Assumptions Basically, yeah.. it should work.

10 Concerns

11 Logging Matters Concerns Logs increase = data increase Message format get more complex Did the Kernel flush the buffers? (sync(2)) Multi-thread application?, locking? Multiple Applications = Multiple Logs

12 Logging Matters Concerns If Multiple Applications = Multiple logs Multiple Hosts x Multiple Applications =???

13 OK, so: 1. Logging matters 2. It's really beneficial 3. but...

14 It needs to be done right.

15 Logging Common sources & inputs Application Logs Apache NginX Syslog (-ng) Custom applications / Languages C, Ruby, Python, PHP, Perl, NodeJS, Java, etc.

16 In a galaxy not so far away...

17

18 How to parse/store multiple data sources? note: performance matters!

19

20 Fluentd is an open source data collector It let's you unify the data collection for a better use and understanding of data.

21 before

22 after

23 Fluentd Highlights High Performance Built-in Reliability Structured Logs Pluggable Architecture More than 300 plugins! (input/filtering/output)

24 Fluentd Architecture

25 Fluentd Internals simplified

26 Fluentd Input plugins

27 Fluentd Output plugins

28 Fluentd Buffer plugins

29 Fluentd Buffer plugins

30

31 MxN M+N

32 Fluentd Simple Forwarding

33 Fluentd Simple Forwarding: configuration # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag backend.apache </source> # logs from client libraries <source> type forward port </source> # store logs to MongoDB <match backend.*> type mongo database fluent collection test </match>

34 Fluentd Less Simple Forwarding

35 Fluentd Lambda Architecture

36 Fluentd # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag backend.apache </source> # store logs to MongoDB <match *.*> type copy <store> type elasticsearch logstash_format true </store> # logs from client libraries <source> type forward port </source> <store> type host port path </store> </match> webhdfs 192.x.y.z /path/to/hdfs

37 Who uses Fluentd in production?

38

39

40 We collect 1M events per second!

41 Internet of Things

42 Internet of Things Facts IoT will grow to many billions of devices over the next decade. Now it's about device to device connectivity. Different frameworks and protocols are emerging. It needs Logging.

43 Internet of Things Alliances Vendors formed alliances to join forces and develop generic software layers for their products:

44 Internet of Things Solutions provided Alliance Framework

45 IoT & Big Data Analytics IoT requires a generic solution to collect events and data from different sources for further analysis. Data can come from a specific framework, radio device, sensor or other. How do we collect and unify data properly?

46 @fluentbit

47 Fluent Bit is an open source data collector It let's you collect data from IoT/Embedded devices and transport It to third party services.

48 Fluent Bit Targets Services Sensors / Signals / Radios Operating System information Automotive / Telematics

49 Fluent Bit Requirements IoT and Embedded environment requires special handling, specifically on performance and resource utilization: Lightweight Written in C Language Customizable, pluggable architecture Full integration with Fluentd

50 Fluent Bit Integration

51 Fluent Bit Direct Output

52 Fluent Bit Elastic Search support

53 Fluent Bit Elastic Search: Dashboard

54 Containers

55

56 Docker Logging driver Docker v1.6 released the concept of logging drivers Route container output Fluentd?

57 Docker

58 Docker v1.8 Fluentd Logging driver!

59 Docker Data Stream

60 Docker Data Stream

61

62 NodeJS Fluent-Logger (NPM)

63 We Love Data! Thank you!