CS 347 Parallel and Distributed Data Processing

Transcription

1 C 37 Parallel and Distributed Data Processing pring 2016 Notes : Query Optimization Query Optimization Cost estimation trategies for exploring plans Q min C 37 Notes 2 Based on estimating result sizes Like in centralized databases But # of IOs may not be the best metric E.g., transmission time may dominate work at site 1 1 work at site 2 2 answer time/$ C 37 Notes 3 C 37 Notes

2 Another reason why # of IOs is not enough: parallelism Plan A Plan B Cost metrics E.g., IOs, bytes transmitted, $, Additive ask IOs site 1 site 2 site 3 50 IOs 70 IOs 50 IOs esponse time metric Not additive Need scheduling and dependency info ask 2 ask 3 kew is important C 37 Notes 5 C 37 Notes 6 Also take into account esponse time example tart up cost Data distribution cost/time esource contention (for memory, disk, network) Cost of assembling results site 1 site 2 site 3 site start up distribution searching + sending results final processing C 37 Notes 7 C 37 Notes 8

3 earch trategies Exhaustive (with pruning) Hill climbing (greedy) Query separation Exhaustive earch Consider all query plans (given a set of techniques for operators) Prune some plans Heuristics C 37 Notes 9 C 37 Notes 10 Exhaustive earch earch trategies Example A B > > ( ) ( ) In generating plans, keep goal in mind E.g., if goal is parallelism (in system with fast network) Consider partitioning relations first 1 2 ship to semi join Prune because cross-product not necessary Prune because larger relation first ship to semi join E.g., if goal is reduction of network traffic Consider semi-joins C 37 Notes 11 C 37 Notes 12

4 Better plans 2 Better plans 1 x initial plan 1 x initial plan Worse plans Worse plans C 37 Notes 13 C 37 Notes 1 Example V A B C relation site size V 0 tuple size = 1 Goal: minimize data transmission Initial plan end relations to one site What site do we send all relations to? o site 1: cost = = 90 o site 2: cost = = 80 o site 3: cost = = 70 o site : cost = = 60 C 37 Notes 15 C 37 Notes 16

5 P 0 (1 ) (2 ) (3 ) Compute V at site Local search Consider sending each relation to neighbor C 37 Notes 17 C 37 Notes 18 Assume size = 20 size = 5 size V = 1 Option A Option B cost = 30 cost = 0 Worse off cost = 30 cost = 30 No savings C 37 Notes 19 C 37 Notes 20

6 Option C Option D cost = 50 cost = 35 Win cost = 50 cost = 25 Bigger win C 37 Notes 21 C 37 Notes 22 P 1 (2 3) α = (1 ) (3 ) Compute answer at site epeat local search reat α = as relation α 1 3 vs. α 1 3 α α 1 3 C 37 Notes 23 C 37 Notes 2

7 Hill climbing may miss best plan E.g., best plan could be P best (3 ) β = V β ( 2) β' = β β' (2 1) β'' = β' β'' (1 ) (optional) Compute answer C 37 Notes 25 β'' 3 β' V β Hill climbing may miss best plan E.g., best plan could be P best (3 ) β = V β ( 2) β' = β β' (2 1) β'' = β' β'' (1 ) (optional) Compute answer = 30 = 1 = 1 = 1 = 33 total C 37 Notes 26 β'' 3 β' V β Costs could be low because β is very selective earch trategies Exhaustive (with pruning) Hill climbing (greedy) Query separation Query eparation eparate query into 2 or more steps Optimize each step independently C 37 Notes 27 C 37 Notes 28

8 Query eparation Query eparation σc1 Example imple queries technique 1. Compute = A [ σ c2 ] = A [ σ c3 ] σc2 A σc3 σc1 A 2. Compute J = 3. Compute answer σ c1 { [ J σ c2 ] [ J σ c3 ] } σc2 σc3 C 37 Notes 29 C 37 Notes 30 Query eparation 1. Compute = A [ σ c2 ] = A [ σ c3 ] 2. Compute J = 3. Compute answer σ c1 { [ J σ c2 ] [ J σ c3 ] } Compute the A values in the answer first Query eparation imple query elations have a single attribute Output has a single attribute E.g., J = Get tuples from sites matching A and compute answer next C 37 Notes 31 C 37 Notes 32

9 Query eparation Idea 1. Decompose query Local processing imple query (or queries) Final processing Query eparation Philosophy Hard part is distributed join Do this part with only keys; get the rest of the data later impler to optimize simple queries 2. Optimize simple query C 37 Notes 33 C 37 Notes 3 ummary Cost estimation Optimization strategies Exhaustive (with pruning) Hill climbing (greedy) Query separation Words of Wisdom Optimization is like chess playing May have to make sacrifices for later gains Move data, partition relations, build indexes C 37 Notes 35 C 37 Notes 36