Real Time Processing. Karthik Ramasamy

Transcription

1 Real Time Processing Karthik Ramasamy

2 2 Information Age Real-time is key á K!

3 3 Real Time Connected World Ñ Internet of Things 30 B connected devices by 2020 Connected Vehicles Data transferred per vehicle per month 4 MB -> 5 GB + Health Care 153 Exabytes (2013) -> 2314 Exabytes (2020)! Digital Assistants (Predictive Analytics) $2B (2012) -> $6.5B (2019) [1] Siri/Cortana/Google Now Machine Data 40% of digital universe by 2020 > Augmented/Virtual Reality $150B by 2020 [2] Oculus/HoloLens/Magic Leap [1] hxp:// [2] hxp://techcrunch.com/2015/04/06/augmented-and-virtual-reality-to-hit-150-billion-by-2020/#.7q0heh:oabw

4 4 Why Real Time? real time trends real time conversations real time recommendations real time search G Ü! s Emerging break out Real time sports Real time product Real time search of trends in Twitter (in the conversations related recommendations based tweets form #hashtags) with a topic (recent goal on your behavior & or touchdown) profile

5 5 Value of Data It s contextual Time/cri8cal& Decisions& Value&of&Data&to&Decision/Making& Preven8ve/& Predic8ve& Ac8onable& Informa9on&Half%Life& In&Decision%Making& Tradi8onal& Batch &&&&&&&&&&&&&&& Business&&Intelligence& Reac8ve& Historical& Real%& Time& Seconds& Minutes& Hours& Days& Time& Months& [1] Courtesy Michael Franklin, BIRTE, 2015.

6 6 What is Real-Time? It s contextual > 1 hour 10 ms - 1 sec < 500 ms < 1 ms high throughput approximate latency sensitive low latency BATCH Real Time OLTP REAL REAL TIME adhoc queries monthly active users relevance for ads ad impressions count hash tag trends deterministic workflows fanout Tweets search for Tweets Financial Trading

7 7 Real Time Analytics C INTERACTIVE Store data and provide results instantly when a query is posed H Analyze data as it is being produced STREAMING

8 8 Real Time Use Cases I Online Services 10s of ms TransacZon log, Queues, RPCs Real Time s of ms Change propagazon, Streaming analyzcs Data for Batch Analytics secs to mins Log aggregazon, Client events

9 9 Real Time Stack Components: Many moving parts s Data a Messaging Collectors REAL TIME STACK b J Storage Compute

10 10 Scribe " Open source log aggregation Originally from Facebook. TwiXer made significant enhancements for real Zme event aggregazon { High throughput and scale Delivers 125M messages/min. Provides Zght SLAs on data reliability # Runs on every machine Simple, very reliable and efficiently uses memory and CPU

11 Event Bus & Distributed Log Next Generation Messaging "

12 12 Twitter Messaging Core Business Logic (tweets, fanouts ) Scribe Deferred RPC Gizzard Database Search Book Kestrel Kestrel Kestrel My SQL Kala Keeper HDFS

13 13 Kestrel Queue 5 e \ ( MESSAGE MULTIPLE HIGHLY LOOSELY QUEUE PROTOCOLS SCALABLE ORDERED

14 14 Kestrel Queue P # C # P # zk C

15 15 Kestrel Limitations Durability is hard to achieve # $ Adding subscribers is expensive Read-behind degrades performance Too many random I/Os! 7 Scales poorly as #queues increase " Cross DC replication

16 16 Kafka Queue V \ Ñ ( CLIENT STATE CONSUMER HIGH SIMPLE FOR MANAGEMENT SCALABILITY THROUGHPUT OPERATIONS

17 17 Kafka Limitations Relies on file system page cache # " Performance degradation when subscribers fall behind - too much random I/O Scaling to many topics! 7 Loss of data " No centralized operational stats

18 18 Rethinking Messaging Unified Stack - tradeoffs for various workloads # $ Durable writes, intra cluster and geo-replication Multi tenancy! 7 Scale resources independently Cost efficiency " Ease of Manageability

19 19 Event Bus - Pub-Sub Publisher Write Proxy Read Event Bus + Distributed Log Proxy Subscriber Distributed Log Metadata

20 20 Distributed Log BK Publisher Write Proxy BK Read Proxy Subscriber BK Distributed Log Metadata - ZK

21 21 Distributed /. -, / Manhattan Key Value Store Durable Deferred RPC Real Time Search Indexing Pub Sub System Globally Replicated Log

22 22 Distributed 400 TB/Day IN 2 Trillion Events/Day 20 PB/Day PROCESSED OUT 5-10 MS latency

23 TwiHer Heron Next Generation Streaming Engine "

24 24 Twitter Heron Better Storm \ Container Based Architecture - Separate Monitoring and Scheduling 2 Simplified ExecuQon Model % Much BeHer Performance

25 25 Twitter Heron Design: Goals Fully API compatible with Storm # Directed acyclic graph Topologies, Spouts and Bolts $ Task isolation Ease of debug-ability/isolazon/profiling Use of main stream languages C++, Java and Python! g Support for back pressure Topologies should self adjuszng Batching of tuples AmorZzing the cost of transferring tuples " G Efficiency Reduce resource consumption

26 26 Twitter Heron b \ Ñ / Guaranteed Horizontal Robust Concise Message Scalability Fault Code-Focus Passing Tolerance on Logic

27 27 Heron Terminology, Topology Directed acyclic graph verzces = computazon, and edges = streams of data tuples Spouts Sources of data tuples for the topology Examples - Kala/Kestrel/MySQL/Postgres % Bolts Process incoming tuples, and emit outgoing tuples Examples - filtering/aggregazon/join/any funczon

28 28 Heron Topology % Bolt 1 % Spout 1 Bolt 4 % Spout 2 Bolt 2 % Bolt 3 % Bolt 5

29 29 Stream Groupings /. -, Shuffle Grouping Fields Grouping All Grouping Global Grouping Random distribution of tuples Group tuples by a field or multiple fields Replicates tuples to all tasks Send the entire stream to one task

30 30 Heron Architecture: High Level Topology 1 Scheduler Topology 2 Topology Submission Topology N

31 31 Heron Architecture: Topology Topology Master Logical Plan, Physical Plan and Execution State ZK Sync Physical Plan Cluster Metrics Manager Stream Manager Stream Manager Metrics Manager I 1 I 2 I 3 I 4 I 1 I 2 I 3 I 4 CONTAINER CONTAINER

32 32 Heron Stream Manager: BackPressure % % S1 B2 B3 % B4

33 33 Stream Manager Stream Manager: BackPressure S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3

34 CONTAINER Topology Master CONTAINER CONTAINER S1 B2 S2 B1 Stream Manager Stream Manager B3 B4 B3 B4 S1 B2 S2 B1 Stream Manager Stream Manager B3 B4 B3 B5 CONTAINER CONTAINER

35 35 Heron Stream Manager: Spout BackPressure S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3

36 36 Heron Use Cases REALTIME ETL REAL TIME BI SPAM DETECTION REAL TIME TRENDS REALTIME ML REAL TIME OPS

37 37 Heron Sample Topologies

38 38 Heron has been in production for 3 years 1 stage 10 stages 3x reduction in cores and memory

39 39 Heron Performance: Settings COMPONENTS EXPT #1 EXPT #2 EXPT #3 Spout Bolt # Heron containers

40 40 Throughput Heron Performance: Atmost Once CPU usage Heron (paper) Heron (master) Heron (paper) Heron (master) x x 10, million tuples/min ,820 # cores used ,920 1, Spout Parallelism Spout Parallelism

41 41 Heron Performance: CPU Usage Heron (paper) Heron (master) 40 million tuples/min x Spout Parallelism

42 42 > 400 Real Time Jobs 500 Billions Events/Day PROCESSED MS latency

43 43 Stateful Processing in Heron OpZmisZc Approaches PessimisZc Approaches

44 Tying Together "

45 45 Lambda Architecture Combining batch and real time New Data Client

46 46 Lambda Architecture - The Good Scribe CollecZon Pipeline Event Bus Heron Compute Pipeline Results

47 47 Lambda Architecture - The Bad Have to write everything twice! # $ Have to fix everything (may be Subtle differences in semantics! 7 How much Duct Tape required? " What about Graphs, ML, SQL, etc?

48 48 Summingbird to the Rescue Heron Topology Online key value result store Message broker Summingbird Program Client HDFS Scalding/Map Reduce Batch key value result store

49 49 Curious to Learn More? Twitter Heron: Stream Processing at Scale Sanjeev Kulkarni, Nikunj Bhagat, Maosong Fu, Vikas Kedigehalli, Christopher Kellogg, Sailesh Mittal, Jignesh M. Patel *,1, Karthik Ramasamy, @staneja Twitter, Inc., *University of Wisconsin Madison Maosong Fu, Sailesh Mittal, Vikas Kedigehalli, Karthik Ramasamy, Michael Barry, Andrew Jorgensen, Christopher Kellogg, Neng Lu, Bill Graham, Jingwei Wu Twitter, Inc. Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik Ramasamy, Jignesh M. Patel*, Sanjeev Kulkarni, Jason Jackson, Krishna Gade, Maosong Fu, Jake Donham, Nikunj Bhagat, Sailesh Mittal, @challenger_nik, Twitter, Inc., *University of Wisconsin Madison

50 Curious to Learn More? 50

51 51 Interested in Heron? HERON IS OPEN SOURCED CONTRIBUTIONS ARE WELCOME! FOLLOW

52 52 Interested in Distributed Log? DISTRIBUTED LOG IS OPEN SOURCED CONTRIBUTIONS ARE WELCOME! FOLLOW

53 53 We are SOFTWARE ENGINEERS PRODUCT ENGINEERS SYSTEM ENGINEERS UI ENGINEERS/UX DESIGNERS CAREERS [AT] STREAML.IO

54 54 Any Question??? WHAT WHY WHERE WHEN WHO HOW

55 55 Get in

56 THANKS FOR ATTENDING!!!