Unifying Events & Logs into the Cloud

Transcription

1 Unifying Events & Logs into the Cloud October 05, 2015 CloudOpen/LinuxCon, Dublin Eduardo

2 About Me Eduardo Silva Github & Twitter Personal Treasure Data Open Source Engineer Fluentd / Fluent Bit Projects Monkey HTTP Server Duda I/O

3 Logging

4 Logging Matters Pros Application status Debugging General information about anomalies: errors Troubleshooting / Support Local or Remote (network)

5 Logging Matters From a business point of view Input data Analytics User interaction / behaviors Improvements

6 Assumptions

7 Logging Matters Assumptions I have enough disk space I/O operations will not block Log messages are human readable My logging mechanism scale

8 Logging Matters Assumptions Basically, yeah.. it should work.

9 Concerns

10 Logging Matters Concerns Logs increase = data increase Message format get more complex Did the Kernel flush the buffers? (sync(2)) Multi-thread application?, locking? Multiple Applications = Multiple Logs

11 Logging Matters Concerns If Multiple Applications = Multiple logs Multiple Hosts x Multiple Applications =???

12 OK, so: 1. Logging matters 2. It's really beneficial 3. but...

13 It needs to be done right.

14 Logging Common sources & inputs Application Logs Apache NginX Syslog (-ng) Custom applications / Languages C, Ruby, Python, PHP, Perl, NodeJS, Java, etc.

15 In a galaxy not so far away...

16

17 How to parse/store multiple data sources? note: performance matters!

18

19 Fluentd is an open source data collector It let's you unify the data collection for a better use and understanding of data.

20 before

21 after

22 Fluentd Highlights High Performance Built-in Reliability Structured Logs Pluggable Architecture More than 300 plugins! (input/filtering/output)

23 Fluentd Architecture

24 Fluentd Internals simplified

25 Fluentd Input plugins

26 Fluentd Output plugins

27 Fluentd Buffer plugins

28 Fluentd Buffer plugins

29

30 MxN M+N

31 Fluentd Simple Forwarding

32 Fluentd Simple Forwarding: configuration # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag backend.apache </source> # logs from client libraries <source> type forward port </source> # store logs to MongoDB <match backend.*> type mongo database fluent collection test </match>

33 Fluentd Less Simple Forwarding

34 Fluentd Lambda Architecture

35 Fluentd # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag backend.apache </source> # store logs to MongoDB <match *.*> type copy <store> type elasticsearch logstash_format true </store> # logs from client libraries <source> type forward port </source> <store> type host port path </store> </match> webhdfs 192.x.y.z /path/to/hdfs

36 Who uses Fluentd in production?

37

38

39 We collect 800k events per second!

40 Internet of Things

41 Internet of Things Facts IoT will grow to many billions of devices over the next decade. Now it's about device to device connectivity. Different frameworks and protocols are emerging. It needs Logging.

42 Internet of Things Alliances Vendors formed alliances to join forces and develop generic software layers for their products:

43 Internet of Things Solutions provided Alliance Framework

44 IoT & Big Data Analytics IoT requires a generic solution to collect events and data from different sources for further analysis. Data can come from a specific framework, radio device, sensor or other. How do we collect and unify data properly?

45 @fluentbit

46 Fluent Bit is an open source data collector It let's you collect data from IoT/Embedded devices and transport It to third party services.

47 Fluent Bit Targets Services Sensors / Signals / Radios Operating System information Automotive / Telematics

48 Fluent Bit Requirements IoT and Embedded environment requires special handling, specifically on performance and resource utilization: Lightweight Written in C Language Customizable, pluggable architecture Full integration with Fluentd

49 Fluent Bit Integration

50 Fluent Bit Direct Output

51 Containers

52

53 Docker Logging driver Docker v1.6 released the concept of logging drivers Route container output Fluentd?

54 Docker

55 Docker v1.8 Fluentd Logging driver!

56

57 NodeJS npm install fluent-logger

58 We Love Data! Thank you!