MOIRA A Goal-Oriented Incremental Machine Learning Approach to Dynamic Resource Cost Estimation in Distributed Stream Processing Systems

Transcription

1 MOIRA A Goal-Oriented Incremental Machine Learning Approach to Dynamic Resource Cost Estimation in Distributed Stream Processing Systems Daniele Foroni, C. Axenie, S. Bortoli, M. Al Hajj Hassan, R. Acker, R. Tudoran, G. Brasche, Y. Velegrakis

2 Context 2

3 Context # Events Time 3

4 Context # Events Time 4

5 Context # Events Only 1 Fixed Parameter Time and NOT query specific! 5

6 Why should we consider the goal? 6

7 MOIRA Query Stream SQL MOIRA Goal Goal-centric approach Goal specified for a single query Rescheduling for improving the goal performance 7

8 MOIRA Architecture Query Stream SQL Cost Optimizer Goal Feature Extractor Monitoring System 8

9 Query Plan Map Reduce Source 1 Join Filter Join Sink Source 2 Filter 9

10 Query Plan Optimization Chain Map Reduce Source 1 Join Filter Join Sink Source 2 Filter 10

11 Query Plan Optimization Chain Map Reduce Source 1 Join Filter Filter Join Sink Source 2 Filter Filter Parallelize 11

12 Cost-Optimizer (1)!"#$ = &"'(, $#(*+&,, (h."/0h1/( c"'( + $#(*+&, + (h."/0h1/( = 100 % 12

13 Cost-Optimizer (2) window 1 Operator 1 rate Operator 1 α rate Operator 2 Operator 2 rate 1 rate 2 window 1 window 2 0 parallelism window 2 -.(/ "! " Output window #$% &' CPU usage of the window ()*+ &' RAM usage of the window, " Output Rate 13

14 Cost-Optimizer (3) Is it possible to extend the chain? Yes! Let s check if chain Source 1 chain extended Operator 1. is chainable 2. supports the parallelism of chain 3. no back-pressure 4. Cost complies the Goal )*+,,./,01)2,,h4*56h75, 14

15 Goal Moira Architecture: Before Query Stream SQL Flink 15

16 Goal Moira Architecture: Static Analysis Query Stream SQL Cost Optimizer Goal Flink Static 16

17 Goal Monitoring System Apache Flink listener through JMX CPU Network Data Rate RAM Back-pressure Window size Monitoring System Flink Static Dynamic 17

18 Goal Feature Extractor Metrics from the monitoring system Builds the operator features (input rate/window size) Invokes the cost estimator Applies the rescheduling policy Feature Extractor Flink Static Dynamic 18

19 Goal MOIRA Architecture: Dynamic Analysis Query Stream SQL Cost Optimizer Goal Feature Extractor Monitoring System Flink Static Dynamic 19

20 Latency evaluation 7 % 20 % 20 New Query Plan & Rescheduling

21 Throughput evaluation 4 % 21 New Query Plan & Rescheduling

22 Open Challenges! Machine Learning approaches to find relationships among variables ) (?!?????? ' "($)? & )! "($) then, entropy analysis to simplify the problem! ( & ' 22

23 Conclusion Goal-centric approach for each singular query MOIRA: framework for dynamic rescheduling Static and dynamic analysis Performance improvements 23

24 Thank you! Questions? 24

25 References Floratou, Avrilia, et al. "Dhalion: self-regulating stream processing in heron." Proceedings of the VLDB Endowment (2017): Han, Zheng, et al. "Elastic allocator: An adaptive task scheduler for streaming query in the cloud." Service Oriented System Engineering (SOSE), 2014 IEEE 8th International Symposium on. IEEE, Khoshkbarforoushha, Alireza, et al. "Flower: a data analytics flow elasticity manager." Proceedings of the VLDB Endowment (2017): Fu, Tom ZJ, et al. "DRS: dynamic resource scheduling for real-time analytics over fast streams." Distributed Computing Systems (ICDCS), 2015 IEEE 35th International Conference on. IEEE, Russo, Gabriele. "Towards Decentralized Auto-Scaling Policies for Data Stream Processing Applications." ZEUS 2018 (2018):

26 Experiments TPC-H queries Adapted for streaming benchmark 4k events/second Running time: 120 minutes 26

27 TPC-H Queries Query Example SELECT o_st, l_orderkey, o_shippriority, SUM(l_extendedprice) AS revenue FROM orders, lineitem WHERE l_orderkey = o_orderkey AND o_orderstatus = F AND YEAR(o_orderdate) > 1993 AND o_orderpriority LIKE '5% AND o_proctime BETWEEN l_proctime - INTERVAL '2' HOUR AND l_proctime + INTERVAL '2' HOUR GROUP BY o_st, l_orderkey, o_shippriority 27

28 TPC-H Queries Query Example SELECT o_st, l_orderkey, o_orderdate, o_shippriority, SUM(l_extendedprice*(1-l_discount)) AS revenue, FROM customer, orders, lineitem WHERE c_mktsegment = 'AUTOMOBILE AND c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate < date ' AND l_shipdate > date ' AND o_proctime BETWEEN l_proctime - INTERVAL '2' HOUR AND l_proctime + INTERVAL '2' HOUR AND o_proctime BETWEEN c_proctime - INTERVAL '2' HOUR AND c_proctime + INTERVAL '2' HOUR GROUP BY o_st, l_orderkey, o_orderdate, o_shippriority 28

29 TPC-H Queries Query Example SELECT c_custkey, c_name, c_address, n_name, c_acctbal, SUM(l_extendedprice * (1 - l_discount)) AS revenue FROM customer, orders, lineitem, nation WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND YEAR(o_orderdate) > 1990 AND l_returnflag = 'R' AND c_nationkey = n_nationkey AND o_proctime BETWEEN l_proctime - INTERVAL '2' HOUR AND l_proctime + INTERVAL '2' HOUR AND o_proctime BETWEEN c_proctime - INTERVAL '2' HOUR AND c_proctime + INTERVAL '2' HOUR AND o_proctime BETWEEN n_proctime - INTERVAL '2' HOUR AND n_proctime + INTERVAL '2' HOUR GROUP BY c_custkey, c_name, c_acctbal, n_name, c_address 29