Load Shedding in a Data Stream Manager

Transcription

1 Load Shedding in a Data Stream Manager Nesime Tatbul, Uur U Çetintemel, Stan Zdonik Brown University Mitch Cherniack Brandeis University Michael Stonebraker M.I.T.

2 The Overload Problem Push-based data sources High and unpredictable data rates Problem: Load > Capacity during spikes Load Shedding : eliminating excess load by dropping data 2

3 Aurora Data Stream Management System Aurora Query Network input data streams QoS QoS output to applications QoS Stream Query Algebra (SQuAl) Quality of Service Run-time Operation: Operator Scheduling Storage Management Performance Monitoring Load Shedding 3

4 Load Shedding by Inserting Drops QoS QoS two types of drops: Random Drop Drop k % Semantic Drop Filter P(value) 4

5 Quality of Service Loss-tolerance QoS utility Value-based QoS utility % delivery values 5

6 Assumptions Processor is the main scarce resource Operators that produce new attribute values are not considered (e.g., aggregate, map) Load Shedder operates independently from Scheduler 6

7 Problem Statement N: query network I : set of input streams C : processing capacity when Load(N(I)) > C, transform N to N such that Load(N (I)) < C Utility(N(I))-Utility(N (I)) is minimized key questions: when to shed load? where to shed load? how much load to shed? which tuples to drop? 7

8 Talk Outline Introduction Technical Overview Experimental Results Related Work Conclusions and Future Work 8

9 Algorithmic Overview evaluate the Query Network Load IF Load > Capacity insert Drops to the Query Network ELSE IF Load < Capacity AND Drops in the Query Network remove Drops from the Query Network read stats, network modify network System Catalog look up Drop Insertion Plans Queries Statistics 9

10 Load Evaluation Load Coefficients for each input stream R i Cost 1 Sel 1 Cost 2 Sel 2 Cost n Sel n n j 1 L = Sel Cost j= 1 k= 1 i k j Total Load at run time (CPU cycles per tuple) can be pre-computed! m i= 1 L i R i (CPU cycles per time unit) 10

11 Load Shedding Road Map (LSRM) Excess Load Drop Insertion Plan QoS Cursors 10% shed less! cursor 20% shed more! 300% 11

12 Constructing the LSRM 1. identify potential drop locations 2. sort the drop locations 3. apply the drop locations in order: insert one unit of drop if semantic drop, determine the filter predicate create an LSRM entry 12

13 Drop Locations utility utility % delivery 4 utility % delivery 7 8 early drops save more cycles drops before splits cause more utility loss % delivery 13

14 Best Drop Location Goal: maximize cycle gain, minimize utility loss subnetwork R Drop x % L subnetwork Cycle gain: R ( x L D) if x > 0 Gx ( ) = 0 otherwise Loss/Gain Ratio D cycles/tuple Utility loss: U(x) x du ( x) / dx negative slope of U ( x) = dg( x)/ dx R L the smaller, the better! 14

15 From Values to % Delivery when, where, how much? have the same answer for Random and Semantic Load Shedding Trick: translation between QoS functions Assumption: drop the lowest utility values first utility relative frequency utility values values %delivery 15

16 Determining the Filter Predicate utility utility relative frequency %delivery values values drop 20% lowest utility value interval drop values < 25 Filter Predicate: value 25 16

17 Talk Outline Introduction Technical Overview Experimental Results Related Work Conclusions and Future Work 17

18 Setup: Experimental Study streams: uniformly distributed integer values networks: a mix of filters and unions QoS: : value-based QoS,, range utilities chosen from a Zipf distribution Metrics used: percent loss in total average utility on loss-tolerance QoS (Tuple Utility Loss) value-based QoS (Value Utility Loss) 18

19 Random-LS beats Admission Control Input-Uniform Random-LS Load ~20% Load ~300% 19

20 Semantic-LS beats Admission Control Input-Uniform Semantic-LS Load ~20% Load ~300% 20

21 Semantic-LS beats Random-LS r=0.15 r=0.20 r=

22 Performance for Networks with Sharing theta=0.99 theta=0.5 theta=

23 Talk Outline Introduction Technical Overview Experimental Results Related Work Conclusions and Future Work 23

24 Relevance to Existing Work Congestion control in networks Multimedia streaming Approximate query answering Data stream processing sampling, shedding on aggregates STREAM [MW+03, BDM03] approximate join processing [DGR03] adjusting rates to windowed joins [KNV03] 24

25 Talk Outline Introduction Technical Overview Experimental Results Related Work Conclusions and Future Work 25

26 Highlights An end-to to-end solution for detecting and resolving overload in data stream systems Utility loss minimization for networks of queries which share resources and processing semantic utility as well as quantity-based utility Static drop insertion plans, dynamic instantiation 26

27 Future Directions Handling complex operators joins and aggregates Other resource limitations memory - windowed operators bandwidth - Aurora* power - at sensor level 27

28 More Information Aurora Web Page: Demo 28