Flying Faster with Heron

Transcription

1 Flying Faster with Heron KARTHIK #TwitterHeron

2 TALK OUTLINE BEGIN I! II ( III b OVERVIEW MOTIVATION HERON IV Z OPERATIONAL EXPERIENCES V K HERON PERFORMANCE END

3 [! OVERVIEW

4 TWITTER IS REAL TIME real time trends real time conversations real time recommendations real time search G Ü " s Emerging break out Real time sports Real time product Real time search of trends in Twitter (in the conversations related recommendations based tweets form #hashtags) with a topic (recent goal on your behavior & or touchdown) profile ANALYZING BILLIONS OF EVENTS IN REAL TIME IS A CHALLENGE!

5 TWITTER STORM Streaming platform for analyzing realtime data as they arrive, so you can react to data as it happens. b \ Ñ / GUARANTEED HORIZONTAL ROBUST CONCISE MESSAGE SCALABILITY FAULT CODE- FOCUS PROCESSING TOLERANCE ON LOGIC

6 STORM TERMINOLOGY TOPOLOGY, Directed acyclic graph Vertices=computation, and edges=streams of data tuples SPOUTS Sources of data tuples for the topology Examples - Event Bus/Kafka/Kestrel/MySQL/Postgres BOLTS % Process incoming tuples and emit outgoing tuples Examples - filtering/aggregation/join/arbitrary function

7 STORM TOPOLOGY BOLT 1 SPOUT 1 SPOUT 2 % BOLT 2 % % BOLT 4 % % BOLT 5 BOLT 3

8 WORD COUNT TOPOLOGY Live stream of Tweets % % TWEET SPOUT PARSE TWEET BOLT WORD COUNT BOLT LOGICAL PLAN

9 WORD COUNT TOPOLOGY % % TWEET SPOUT TASKS PARSE TWEET BOLT TASKS WORD COUNT BOLT TASKS When a parse tweet bolt task emits a tuple which word count bolt task should it send to?

10 STREAM GROUPINGS SHUFFLE GROUPING FIELDS GROUPING ALL GROUPING GLOBAL GROUPING /. -, Random distribution of tuples Group tuples by a field or multiple fields Replicates tuples to all tasks Sends the entire stream to one task

11 WORD COUNT TOPOLOGY SHUFFLE GROUPING FIELDS GROUPING % % TWEET SPOUT TASKS PARSE TWEET BOLT TASKS WORD COUNT BOLT TASKS

12 ( MOTIVATION

13 STORM ARCHITECTURE MASTER NODE TOPOLOGY SUBMISSION Nimbus ASSIGNMENT MAPS Multiple Functionality Scheduling/Monitoring Single point of failure ZK No resource reservation and isolation CLUSTER Storage Contention SUPERVISOR SUPERVISOR W1 W2 W3 W4 W1 W2 W3 W4 SLAVE NODE SLAVE NODE

14 STORM WORKER EXECUTOR1 EXECUTOR2 Complex hierarchy TASK1 JVM PROCESS TASK2 Hard to debug Difficult to tune TASK4 TASK5 TASK3

15 DATA FLOW IN STORM WORKERS In In In In In Queue User Logic Thread In In In Out In Queue Queue User Logic Send Thread Thread Queue Contention Global Receive Thread Outgoing Message Buffer TCP Receive Buffer Multiple Languages Global Send Thread TCP Send Buffer Kernel

16 OVERLOADED ZOOKEEPER Scaled up STORM W zk S1 W W zk S2 S3 Handled unto to 1200 workers per cluster

17 OVERLOADED ZOOKEEPER Analyzing zookeeper traffic KAFKA SPOUT 67% Offset/partition is written every 2 secs STORM RUNTIME 33% Workers write heart beats every 3 secs

18 OVERLOADED ZOOKEEPER Heart beat daemons STORM W H HH zk S1 W S2 zk W KV KVKV S workers per cluster

19 STORM - DEPLOYMENT shared pool storm cluster

20 STORM - DEPLOYMENT shared pool isolated pools joe s topology storm cluster

21 STORM - DEPLOYMENT shared pool isolated pools joe s topology storm cluster jane s topology

22 STORM - DEPLOYMENT shared pool isolated pools joe s topology storm cluster jane s topology dave s topology

23 STORM ISSUES g LACK OF BACK PRESSURE Drops tuples unpredictably EFFICIENCY G Serialization program consumes 75 cores at 30% CPU Topology consumes 600 cores at 20-30% CPU! NO BATCHING Tuple oriented system - implicit batching by 0MQ

24 EVOLUTION OR REVOLUTION? fix storm or develop a new system? FUNDAMENTAL ISSUES- REQUIRE EXTENSIVE REWRITING, Several queues for moving data Inflexible and requires longer development cycle USE EXISTING OPEN SOURCE SOLUTIONS Issues working at scale/lacks required performance Incompatible API and long migration process

25 HERONb

26 HERON DESIGN GOALS FULLY API COMPATIBLE WITH STORM " Directed acyclic graph Topologies, spouts and bolts # TASK ISOLATION Ease of debug ability/resource isolation/profiling d USE OF MAIN STREAM LANGUAGES C++/JAVA/Python

27 HERON ARCHITECTURE Scheduler Topology 1 TOPOLOGY SUBMISSION Topology 2 Topology 3 Topology N

28 TOPOLOGY ARCHITECTURE Topology Master Logical Plan, Physical Plan and Execution State ZK Sync Physical Plan CLUSTER Stream Manager Metrics Manager Stream Manager Metrics Manager I1 I2 I3 I4 I1 I2 I3 I4 CONTAINER CONTAINER

29 TOPOLOGY MASTER Solely responsible for the entire topology b \ Ñ ASSIGNS ROLE MONITORING METRICS

30 TOPOLOGY MASTER Topology Master Logical Plan, Physical Plan and Execution State ZK CLUSTER! PREVENT MULTIPLE TM BECOMING MASTERS! ALLOWS OTHER PROCESS TO DISCOVER TM

31 STREAM MANAGER Routing Engine /, Ñ ROUTES TUPLES BACK PRESSURE ACK MGMT

32 STREAM MANAGER S1 B2 B3 B4 % % %

33 STREAM MANAGER S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 O(n 2 ) O(k 2 ) S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3

34 STREAM MANAGER tcp back pressure S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 SLOWS UPSTREAM AND DOWNSTREAM INSTANCES

35 STREAM MANAGER spout back pressure S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3

36 STREAM MANAGER stage by stage back pressure S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3

37 STREAM MANAGER back pressure advantages PREDICTABILITY! Tuple failures are more deterministic SELF ADJUSTS! Topology goes as fast as the slowest component

38 HERON INSTANCE Does the real work! > > p RUNS ONE TASK EXPOSES API COLLECTS METRICS

39 HERON INSTANCE Stream Manager data-in queue Gateway Thread data-out queue Task Execution Thread Metrics Manager metrics-out queue

40 K OPERATIONAL EXPERIENCES $

41 HERON DEPLOYMENT Aurora Scheduler ZK CLUSTER Topology 1 Aurora Services Topology 2 Heron Web Topology 3 Heron Tracker Heron VIZ Topology N Observability

42 HERON SAMPLE TOPOLOGIES

43 SAMPLE TOPOLOGY DASHBOARD

44 STORM is decommissioned Large amount of data produced every day Large cluster Several topologies deployed Several billion messages every day 1 stage 10 stages 3x reduction in cores and memory

45 HERON x PERFORMANCE 9

46 HERON PERFORMANCE Settings COMPONENTS EXPT #1 EXPT #2 EXPT #3 EXPT #4 Spout Bolt # Heron containers # Storm workers

47 HERON PERFORMANCE Word count topology - Acknowledgements enabled Throughput Latency Storm Heron Storm Heron million tuples/min 700 latency (ms) Spout Parallelism Spout Parallelism 10-14x 5-15x

48 HERON PERFORMANCE Word count topology - CPU usage Storm Heron # cores used Spout Parallelism 2-3x

49 HERON PERFORMANCE Throughput and CPU usage with no acknowledgements - Word count topology Storm Heron Storm Heron million tuples/min 2500 # cores used Spout Parallelism Spout Parallelism

50 HERON EXPERIMENT RTAC topology SHUFFLE GROUPING FIELDS GROUPING % % FIELDS GROUPING % CLIENT EVENT DISTRIBUTOR USER COUNT AGGREGATOR SPOUT BOLT BOLT BOLT

51 HERON PERFORMANCE CPU usage - RTAC Topology Storm Heron Acknowledgements enabled Storm Heron No acknowledgements # cores used 200 # cores used

52 HERON PERFORMANCE Latency with acknowledgements enabled - RTAC Topology Storm Heron latency (ms)

53 CURIOUS TO LEARN MORE Twitter Heron: Stream Processing at Scale Sanjeev Kulkarni, Nikunj Bhagat, Maosong Fu, Vikas Kedigehalli, Christopher Kellogg, Sailesh Mittal, Jignesh M. Patel *,1, Karthik Ramasamy, @staneja Twitter, Inc., *University of Wisconsin Madison Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik Ramasamy, Jignesh M. Patel*, Sanjeev Kulkarni, Jason Jackson, Krishna Gade, Maosong Fu, Jake Donham, Nikunj Bhagat, Sailesh Mittal, @challenger_nik, Twitter, Inc., *University of Wisconsin Madison

54 CONCLUSION SIMPLIFIED ARCHITECTURE " Easy to debug, profile and support HIGH PERFORMANCE % 7-10x increase in throughput 5-10x improvement in latency # EFFICIENCY 3-5x decrease in resource usage

55 #ThankYou FOR LISTENING &

56 R QUESTIONS and ANSWERS ' Go ahead. Ask away.