Scalable Streaming Analytics

Transcription

1 Scalable Streaming Analytics KARTHIK

2 TALK OUTLINE BEGIN I! II ( III b Overview Storm Overview Storm Internals IV Z V K Heron Operational Experiences END

3 WHAT IS ANALYTICS? according to Wikipedia! DISCOVERY Ability to identify patterns in data!!! COMMUNICATION Provide insights in a meaningful way

4 TYPES OF ANALYTICS varieties! E CUBE ANALYTICS PREDICTIVE ANALYTICS

5 DIMENSIONS OF ANALYTICS variants STREAMING INTERACTIVE BATCH ô " Ü Ability to analyze the data immediately after it is produced Ability to provide results instantly when a query is posed Ability to provide insights after several hours/days when a query is posed

6 STREAMING VS INTERACTIVE INTERACTIVE ANALYTICS Real time alerts, Real time analytics Continuous visibility Static Batch Results/Reports Queries Bulkload Data STREAMING ANALYTICS Results Database Server Data$ Storage$ Data Stream Processing Queries Data$ Storage$

7 WHAT IS REAL TIME? msecs or secs or mins? < 500 ms latency sensitive > 1 sec approximate > 1 hour high throughput Feedback Complement OLTP REAL TIME BATCH deterministic workflows fanout Tweets search for Tweets ad impressions count hash tag trends adhoc queries monthly active users relevance for ads

8 STREAMING DATA FLOW varieties

9 STREAMING SYSTEMS first generation - SQL based NIAGARA Query Engine Stanford Stream Data Manager Aurora Stream Processing Engine Borealis Distributed Stream Processing Engine Cayuga - Stateful Event Monitoring

10 STREAMING SYSTEMS next generation - too many

11 [! STORM OVERVIEW I

12 WHAT IS STORM? Streaming platform for analyzing realtime data as they arrive, so you can react to data as it happens. b \ Ñ / GUARANTEED HORIZONTAL ROBUST CONCISE MESSAGE SCALABILITY FAULT CODE- FOCUS PROCESSING TOLERANCE ON LOGIC

13 STORM DATA MODEL TOPOLOGY, Directed acyclic graph Vertices=computation, and edges=streams of data tuples SPOUTS Sources of data tuples for the topology Examples - Kafka/Kestrel/MySQL/Postgres BOLTS % Process incoming tuples and emit outgoing tuples Examples - filtering/aggregation/join/arbitrary function

14 STORM TOPOLOGY BOLT 1 SPOUT 1 SPOUT 2 % BOLT 2 % % BOLT 4 % % BOLT 5 BOLT 3

15 WORD COUNT TOPOLOGY Live stream of Tweets % % TWEET SPOUT PARSE TWEET BOLT WORD COUNT BOLT LOGICAL PLAN

16 WORD COUNT TOPOLOGY % % TWEET SPOUT TASKS PARSE TWEET BOLT TASKS WORD COUNT BOLT TASKS When a parse tweet bolt task emits a tuple which word count bolt task should it send to?

17 STREAM GROUPINGS SHUFFLE GROUPING FIELDS GROUPING ALL GROUPING GLOBAL GROUPING /. -, Random distribution of tuples Group tuples by a field or multiple fields Replicates tuples to all tasks Sends the entire stream to one task

18 WORD COUNT TOPOLOGY SHUFFLE GROUPING FIELDS GROUPING % % TWEET SPOUT TASKS PARSE TWEET BOLT TASKS WORD COUNT BOLT TASKS

19 II ( STORM INTERNALS

20 STORM ARCHITECTURE MASTER NODE TOPOLOGY SUBMISSION Nimbus ASSIGNMENT MAPS SYNC CODE ZK CLUSTER SUPERVISOR SUPERVISOR W1 W2 W3 W4 W1 W2 W3 W4 SLAVE NODE SLAVE NODE

21 STORM WORKER EXECUTOR EXECUTOR EXECUTOR JVM PROCESS TASK TASK TASK TASK TASK TASK

22 DATA FLOW IN STORM WORKERS In In In In In Queue User Logic Thread In In In Out In Queue Queue User Logic Send Thread Thread Global Receive Thread Disruptor Queues Outgoing Message Buffer TCP Receive Buffer Global Send Thread 0mq Queues TCP Send Buffer Kernel

23 h l P b >50tb >2400 >250 >3b Large amount of data produced every day Largest storm cluster Several topologies deployed Several billion messages every day 1 stage 8 stages

24 STORM ARCHITECTURE MASTER NODE TOPOLOGY SUBMISSION Nimbus ASSIGNMENT MAPS Multiple Functionality Scheduling/Monitoring Single point of failure ZK CLUSTER Storage Contention SUPERVISOR SUPERVISOR W1 W2 W3 W4 W1 W2 W3 W4 SLAVE NODE SLAVE NODE

25 STORM WORKER EXECUTOR1 EXECUTOR2 Complex hierarchy TASK1 JVM PROCESS TASK2 Hard to debug Difficult to tune TASK4 TASK5 TASK3

26 DATA FLOW IN STORM WORKERS In In In In In Queue User Logic Thread In In In Out In Queue Queue User Logic Send Thread Thread Queue Contention Global Receive Thread Outgoing Message Buffer TCP Receive Buffer Multiple Languages Global Send Thread TCP Send Buffer Kernel

27 OVERLOADED ZOOKEEPER Scaled up STORM W zk S1 W W zk S2 S3 Handled unto to 1200 workers per cluster

28 OVERLOADED ZOOKEEPER Analyzing zookeeper traffic KAFKA SPOUT 67% Offset/partition is written every 2 secs!! STORM RUNTIME 33% Workers write heart beats every 3 secs

29 OVERLOADED ZOOKEEPER Heart beat daemons STORM W HH H W zk zk W KV KV KV 5000 workers per cluster S1 S2 S3

30 EVOLUTION OR REVOLUTION? fix storm or develop a new system? FUNDAMENTAL ISSUES- REQUIRE EXTENSIVE REWRITING, Several queues for moving data Inflexible and requires longer development cycle USE EXISTING OPEN SOURCE SOLUTIONS Issues working at scale/lacks required performance Incompatible API and long migration process

31 HERONb III

32 HERON DESIGN GOALS FULLY API COMPATIBLE WITH STORM, Directed acyclic graph Topologies, spouts and bolts USE OF WELL KNOWN LANGUAGES No Clojure C++/JAVA/Python

33 HERON ARCHITECTURE Scheduler Topology 1 TOPOLOGY SUBMISSION Topology 2 Aurora Topology 3 ECS YARN Mesos Topology N

34 TOPOLOGY ARCHITECTURE Topology Master Logical Plan, Physical Plan and Execution State ZK Sync Physical Plan CLUSTER Stream Manager Metrics Manager Stream Manager Metrics Manager I1 I2 I3 I4 I1 I2 I3 I4 CONTAINER CONTAINER

35 TOPOLOGY MASTER Solely responsible for the entire topology b \ Ñ ASSIGNS ROLE MONITORING METRICS

36 TOPOLOGY MASTER Topology Master Logical Plan, Physical Plan and Execution State ZK CLUSTER " PREVENT MULTIPLE TM BECOMING MASTERS " ALLOWS OTHER PROCESS TO DISCOVER TM

37 STREAM MANAGER Routing Engine /, Ñ ROUTES TUPLES BACKPRESSURE ACK MGMT

38 STREAM MANAGER S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 O(n 2 ) O(k 2 ) S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3

39 STREAM MANAGER tcp back pressure S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 SLOWS UPSTREAM AND DOWNSTREAM INSTANCES

40 STREAM MANAGER spout back pressure S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3 B4 S1 B2 S1 B2 Stream Manager Stream Manager B3 B4 B3

41 STREAM MANAGER back pressure advantages PREDICTABILITY " Tuple failures are more deterministic SELF ADJUSTS " Topology goes as fast as the slowest component

42 HERON INSTANCE Does the real work! > > p RUNS ONE TASK EXPOSES API COLLECTS METRICS

43 HERON INSTANCE Stream Manager data-in queue Gateway Thread data-out queue Task Execution Thread Metrics Manager metrics-out queue BOUNDED QUEUES - TRIGGERS GC IN LARGE TOPOLOGIES

44 METRICS MANAGER Optical Nerve * ò GATHERS METRICS SCRIBES ABSTRACTED

45 HERON PERFORMANCE Throughput with acknowledgements - Word count topology Storm Heron million tuples/min Spout Parallelism

46 HERON PERFORMANCE Latency with acknowledgements enabled - Word Count Topology Storm Heron latency (ms) Spout Parallelism

47 HERON PERFORMANCE CPU usage with acknowledgements enabled - Word Count Topology Storm Heron # cores used Spout Parallelism

48 HERON PERFORMANCE Throughput with no acknowledgements - Word count topology Storm Heron million tuples/min Spout Parallelism

49 HERON PERFORMANCE CPU usage with no acknowledgements - Word Count Topology Storm Heron # cores used Spout Parallelism

50 HERON PERFORMANCE CPU usage - RTAC Topology Storm Heron Acknowledgements enabled Storm Heron No acknowledgements # cores used 200 # cores used

51 HERON PERFORMANCE Latency with acknowledgements enabled - RTAC Topology Storm Heron latency (ms)

52 K IV OPERATIONAL EXPERIENCES $

53 HERON DEPLOYMENT Aurora Scheduler ZK CLUSTER Topology 1 Aurora Services Topology 2 Heron Web Topology 3 Heron Tracker Heron VIZ Topology N Observability

54 HERON SAMPLE TOPOLOGIES

55 OPERATIONAL EXPERIENCE SERVICE-LESS CLUSTER-LESS TENSION-LESS 4 \ ", All topologies run under topology owner s role Everything runs on Aurora No more 2am pages

56 DEVELOPER EXPERIENCE DEBUG TUNE DEPLOY J a G, Faster iteration Better resource utilization Devel to prod in 5min

57 MIGRATION EXPERIENCE SMALL MEDIUM LARGE J L #, Couple of hours Lots of savings Summingbird tuning takes time

58 CURRENT WORK x V 9

59 CURRENT WORK SERIALIZATION TUNING ELASTIC CONFIGURATION < " q é Use Java Reflection Determine optimal set of parameters Grow/Shrink based on data Update topology without restarting

60 R QUESTIONS and ANSWERS % Go ahead. Ask away.