the great logfile in the

Transcription

1 the great logfile in the

2

3

4

5 Kafka

6 Kafka is a persistent publish/ subscribe messaging system designed to broker highthroughput, data streams for multiple consumers.

7

8 Kafka is a persistent, publish/subscribe messaging system designed to broker high-throughput data streams for multiple consumers. Producers Front End Front End Service Push Brokers Kafka Pull Consumers Data Warehouse Search Monitoring

9 require "kafka" producer = Kafka::Producer.new consumer = Kafka::Consumer.new message = Kafka::Message.new("Some data") producer.send(message) consumer.consume => data"...>]

10 require "kafka" producer = Kafka::Producer.new consumer = Kafka::Consumer.new message = Kafka::Message.new("Some data") producer.send(message) consumer.consume => data"...>]

11 WHY?

12 Log Aggregators

13 Message Queues

14 Low Latency Low Throughput Message Queues Log Aggregators High Throughput High Latency

15 Low Latency Low Throughput Message Queues Kafka Log Aggregators High Throughput High Latency

16 Apache Kafka is a persistent, publish/subscribe messaging system designed to broker high-throughput data streams for multiple consumers. $ ls -l /opt/kafka/logs/page_views-0/ -rw-r--r-- 1 kafka kafka Jul 25 21: kafka -rw-r--r-- 1 kafka kafka Jul 25 23: kafka -rw-r--r-- 1 kafka kafka Jul 26 01: kafka -rw-r--r-- 1 kafka kafka Jul 26 03: kafka -rw-r--r-- 1 kafka kafka Jul 26 06: kafka -rw-r--r-- 1 kafka kafka Jul 26 08: kafka $ ls -l /opt/kafka/log/analytics-5/ -rw-r--r-- 1 kafka kafka Jul 26 04: kafka -rw-r--r-- 1 kafka kafka Jul 26 22: kafka -rw-r--r-- 1 kafka kafka Jul 27 15: kafka -rw-r--r-- 1 kafka kafka Jul 28 01: kafka -rw-r--r-- 1 kafka kafka Jul 28 21: kafka

17 TOPIC

18 page_views ad_clicks service_logs

19 PARTITION

20 0..n

21 producer.send(message.new( hi )) Size "Magic" CRC Payload hi n bytes

22 %w(hi hi hi hello goodday hi).each do payload producer.send(message.new(payload)) end hi hi hi hello goodday hi

23 $ ls -l /opt/kafka/logs/page_views-0/ -rw-r--r-- 1 kafka kafka Jul 25 21: kafka { topic: page_views, partition: 0, offset: }

24 producer = Kafka::Producer.new( topic: "letters", partition: 0 ) %w(a b c d e).each do letter message = Kafka::Message.new(letter) producer.send(message) end

25 consumer = Kafka::Consumer.new( offset: 10, topic: letters, partition: 0 ) consumer.offset => 10 consumer.consume.map(&:payload) => [ b, c, d, e ] consumer.offset => 50

26 $ ls -l /opt/kafka/logs/page_views-0/ -rw-r--r-- 1 kafka kafka Jul 25 21: kafka -rw-r--r-- 1 kafka kafka Jul 25 23: kafka -rw-r--r-- 1 kafka kafka Jul 26 01: kafka -rw-r--r-- 1 kafka kafka Jul 26 03: kafka -rw-r--r-- 1 kafka kafka Jul 26 06: kafka -rw-r--r-- 1 kafka kafka Jul 26 08: kafka $ ls -l /opt/kafka/log/analytics-5/ -rw-r--r-- 1 kafka kafka Jul 26 04: kafka -rw-r--r-- 1 kafka kafka Jul 26 22: kafka -rw-r--r-- 1 kafka kafka Jul 27 15: kafka -rw-r--r-- 1 kafka kafka Jul 28 01: kafka -rw-r--r-- 1 kafka kafka Jul 28 21: kafka

27 Kafka is a persistent, publish/subscribe messaging system designed to broker high-throughput data streams for multiple consumers. { Topic, Partition } Consumer Offset: 0 Consumer Offset: 105 Consumer Offset:

28 Front End Services Postgres Redis Document Document Document Log Files Service... Document Document Document Archive Data Hadoop Search Monitoring ... Warehouse

29 Front End Services Postgres Redis Document Document Document Log Files Service... Kafka Document Document Document Archive Data Hadoop Search Monitoring ... Warehouse

30 Kafka is a persistent, publish/subscribe messaging system designed to broker high-throughput, data streams for multiple consumers.

31 API Simplicity Producer Broker Consumer Messages Messages

32 Linear Disk Access [it s] widely underappreciated: in modern systems, as demonstrated in the figure, random access to memory is typically slower than sequential access to disk. Note that random reads from disk are more than 150,000 times slower than sequential access Adam Jacobs The Pathologies of Big Data. ACM Queue, July 2009

33 Page Cache

34 $ free total used free shared buffers cached Mem: /+ buffers/cache: Swap: 0 0 0

35 Write Behind Read Ahead

36 sendfile(2)

37 pread(file, buffer, size, offset); // do something with the buffer write(socket, buffer, size);

38 User System Read 2 Buffer 1 fs Application Socket Buffer 3 4 NIC

39 sendfile(socket, file, offset, size);

40 System Read Buffer 1 fs 2 Socket Buffer 3 NIC

41 Durability Concessions

42 Stateless Broker

43 Simple Schema-less Log Format

44 RECAP

45 Kafka is a persistent, publish/ subscribe messaging system designed to broker highthroughput, data streams for multiple consumers.

46 Messaging System Cherry pick characteristics of log aggregation systems (performance) and message queues (semantics)

47 Persistent Maintain a rolling time-based window of the stream Don t fear the filesystem

48 High-Throughput Performance over features and durability Rely on operating system features Eschew user-land caching

49 Multiple Consumers Push data in, pull data out Support parallel consumers with varying rates from offline to realtime

50

51

52 Publishing content to feeds based upon events Data warehouse ETL of event data Spam flagging of user-generated content System monitoring Full text search Trigger newsletters

53 Message Requirements 1) Provide each message as a uniform JSON payload containing: Event name Timestamp of the event s occurrence Actor User ID and created_at timestamp Attributes 2) Transmit messages to Kafka asynchronously 3) Maximize producer performance by batching messages together when possible

54 Model Controller EventHandler KafkaLog Producer fire(event) fire(event) fire(event) fire(event) flush write(events) send(messages)

55 Model Controller EventHandler KafkaLog Producer fire(event) fire(event) fire(event) fire(event) flush write(events) send(messages)

56 class KafkaLog include Singleton def start(producer) Thread.new do while batch producer.batch do batch.each do message producer.send(kafka::message.new(message)) end end end end end end def = Queue.new end def end

57 Model Controller EventHandler KafkaLog Producer fire(event) fire(event) fire(event) fire(event) flush write(events) send(messages)

58 class EventHandler def = = [] end def fire(event, user, attributes={}) payload = { event: event, timestamp: Time.now.to_f, attributes: attributes, user: { id: user.id, created_at: user.created_at.to_f } end def end end

59 Model Controller EventHandler KafkaLog Producer fire(event) fire(event) fire(event) fire(event) flush write(events) send(messages)

60 class ApplicationController < ActionController::Base after_filter :flush_events_to_log def = EventHandler.new(KafkaLog.instance) end def end end

61 class PostsController < ApplicationController def = Post.new(params[:posts]) event_handler.fire("post.create", current_user, ) end end end

62 class PostsController < ApplicationController def = Post.find(params[:id]) event_handler.fire("post.show", current_user, ) end end

63 # config/initializers/kafka_log.rb producer = Kafka::Producer.new(topic: blog_log ) KafkaLog.instance.start(producer)

64 desc "Tail from the Kafka log file" task :tail, [:topic] => :environment do task, args topic = args[:topic].to_s consumer = Kafka::Consumer.new(topic: topic) puts "==> #{topic} <==" consumer.loop do messages messages.each do message json = JSON.parse(message.payload) puts JSON.pretty_generate(json), "\n" end end end end

65 THANK Slides Video of Pivotal Labs Demo Application Repo Apache Incubator: Kafka Kafka Papers & Presentations Kafka Design Kafka: A Distributed Messaging System for Log Processing IEEE Data Engineering Bulletin (July, 2012): Big Data War Stories