Streaming OLAP Applications

Transcription

1 Streaming OLAP Applications From square one to multi-gigabit streams and beyond C. Scott Andreas HPTS

2 Roadmap Framing the problem Four phases of an architecture s evolution Code: A general-purpose lockless aggregator Demonstration Further reading

3

4

5

6 A journey of up and out Started at ~7,000 flows / second on one node Added distribution, bringing us to 7,000 flow/sec/node Implemented custom OLAP engine: 1.6 MM/sec/node Further work remains on a streaming OLAP map/reduce, demonstrated on a stream of 80 Gbps.

7 A good place to be x Single-Node Scalability Many-Node Scalability

8 A good place to be x Single-Customer Scalability Many-Customer Scalability

9 Four phases Up: Off-the-shelf CEP software Out: Distribution Up: Custom streaming OLAP engine Out: Evolution toward a streaming map/reduce

10 [1] Off-the-Shelf CEP Single customer, single node Exists, works!

11

12 A sample EPL that returns the average price per symbol for the last 100 stock ticks:! select symbol, avg(price) as avgprice from StockTickEvent.win:length(100) group by symbol; a

13

14

15

16 Single-Node Scalability you are here x 7,000 events/second one node, no HA Many-Node Scalability

17 [2] Distribution Designing an HA multi-tenant analytics engine to map M customers onto N nodes.

18 zookeeper zookeeper zookeeper zookeeper zookeeper coll01 coll02 kafka01 olap01 Client API 0 - NN Client API 0 - NN Client API 0 - NN Client API 0 - NN coll03 coll04 coll05 coll06 collectors kafkann stream buffering olapnn OLAP filtering + aggregation Storage Storage NN Storage 0 - NN Storage 0 - NN NN

19 Self-Organization github.com/boundary/ordasity

20 Self-Organization ZooKeeper broadcasts a consistently-ordered view of cluster state changes for all nodes, all active streams, and who owns what. github.com/boundary/ordasity

21 Self-Organization ZooKeeper broadcasts a consistently-ordered view of cluster state changes for all nodes, all active streams, and who owns what. Claim streams until I have at least my fair share. github.com/boundary/ordasity

22 Self-Organization ZooKeeper broadcasts a consistently-ordered view of cluster state changes for all nodes, all active streams, and who owns what. Claim streams until I have at least my fair share. If I have too much, hand off streams until I m doing my fair share. github.com/boundary/ordasity

23 Self-Organization ZooKeeper broadcasts a consistently-ordered view of cluster state changes for all nodes, all active streams, and who owns what. Claim streams until I have at least my fair share. If I have too much, hand off streams until I m doing my fair share. If I m shutting down, tell others, hand streams off, and don t claim any more. github.com/boundary/ordasity

24

25

26 Single-Node Scalability you are here x Many-Node Scalability

27 Single-Node Scalability you are here x Many-Node Scalability

28 Single-Node Scalability 7,000 flows/second any number of nodes, HA you are here x Many-Node Scalability

29 Single-Customer Scalability but you are still here x Many-Customer Scalability

30 Single-Customer Scalability but you are still here x Many-Customer Scalability

31 Single-Customer Scalability but you are still here 7,000 flows/second any number of nodes, HA x Many-Customer Scalability

32 [3] Custom Streaming OLAP Lockless aggregation of event streams

33 Timestamp Dimension Key Rollup Object

34

35 Lockless Aggregator Methodology: Launch process with thread count configuration,preload all data into memory, run for 10 minutes, and exit printing the final mean processing rate. Batch size: 10,000. Hardware: Tests run on an EC2 cc2.8xlarge (2x Xeon E5-2670; 32 vcores,16 physical) Software: Java 1.7.0_40-b43 Xmx24G CMS+Parnew. EC2 Linux amzn1.x86_64 (ami-a73758ce)

36 Lock-Striping Aggregator Methodology: Launch process with thread count configuration,preload all data into memory, run for 10 minutes, and exit printing the final mean processing rate. Batch size: 10,000. Hardware: Tests run on an EC2 cc2.8xlarge (2x Xeon E5-2670; 32 vcores,16 physical) Software: Java 1.7.0_40-b43 Xmx24G CMS+Parnew. EC2 Linux amzn1.x86_64 (ami-a73758ce)

37 Lockless Aggregator (NonBlockingHashMap) Lock-Striping Aggregator (ConcurrentHashMap) Methodology: Launch process with thread count configuration,preload all data into memory, run for 10 minutes, and exit printing the final mean processing rate. Batch size: 10,000. Hardware: Tests run on an EC2 cc2.8xlarge (2x Xeon E5-2670; 32 vcores,16 physical) Software: Java 1.7.0_40-b43 Xmx24G CMS+Parnew. EC2 Linux amzn1.x86_64 (ami-a73758ce)

38

39

40 Timestamp Dimension Key Rollup Object

41 Single-Customer Scalability moving on up! x Many-Customer Scalability

42 Single-Customer Scalability moving on up! x Many-Customer Scalability

43 Single-Customer Scalability 1.6MM flows/second/node any number of nodes, HA moving on up! x Many-Customer Scalability

44 Example Implementation

45 Example Implementation

46

47

48

49

50 demo

51 Many-Node and Large Customer Scalability x what gets us here? Many-Node and Many-Customer Scalability

52 Many-Node and Large Customer Scalability high processing rate, HA, any number of nodes, no single-node sharding limit. x what gets us here? Many-Node and Many-Customer Scalability

53 [4] Streaming OLAP Map/Reduce Incremental lockless filtering / aggregation of event streams, final rollups of total streams

54 Map Map Input Sources Map Reduce Map Map Output many, high velocity high velocity, partitioned streams top-level filtering and aggregation low velocity incremental output final aggregation

55

56

57

58 Streaming Map/Reduce

59 Streaming Map/Reduce Higher latency, but much higher velocity

60 Streaming Map/Reduce Higher latency, but much higher velocity Challenging for time-windowed aggregations (case of the slow mapper)

61 Streaming Map/Reduce Higher latency, but much higher velocity Challenging for time-windowed aggregations (case of the slow mapper) Implementations: Apache Samza atop YARN (LinkedIn), Storm (Twitter), Summingbird (Twitter)

62 Streaming Map/Reduce Higher latency, but much higher velocity Challenging for time-windowed aggregations (case of the slow mapper) Implementations: Apache Samza atop YARN (LinkedIn), Storm (Twitter), Summingbird (Twitter) Papers: MillWheel (Google at VLDB)

63

64 Parallel OLAP Aggregation

65 Parallel OLAP Aggregation Fundamental problem: contention

66 Parallel OLAP Aggregation Fundamental problem: contention Lockless data structures reduce contention but CAS is no silver bullet

67 Parallel OLAP Aggregation Fundamental problem: contention Lockless data structures reduce contention but CAS is no silver bullet One approach: thread-local aggregation with TreeMaps/HashMaps, combining operations once/sec

68 Parallel OLAP Aggregation Fundamental problem: contention Lockless data structures reduce contention but CAS is no silver bullet One approach: thread-local aggregation with TreeMaps/HashMaps, combining operations once/sec Flat Combining and the Synchronization-Parallelism Tradeoff

69

70

71 Code Streaming Aggregation: Cluster Coordination: Documentation:

72

73

74 Streaming OLAP Applications From square one to multi-gigabit streams and beyond C. Scott Andreas HPTS