Query optimization. Query Optimization. Query Optimization. Cost estimation. Another reason why plain IOs not enough: Parallelism

Transcription

1 Query optimization Query Optimization It is safer to accept any chance that offers itself, and extemporize a procedure to fit it, than to get a good plan matured, and wait for a chance of using it. Thomas Hardy (187) in Far from the Madding Crowd Generate query plans Estimate size of intermediate results Estimate cost of plan ($,time, ) Q P1 P P3 Pn C1 C C3 Cn pick minimum C55 Query Optimization 1 C55 Query Optimization Query Optimization Query: 1 3 3@ite3 esult@ite1 Possible plans Plan 1: end and 3 to ite1. Perform query at ite 1 Plan : end 3 to ite; Evaluate I = 3; send I to ite1; Evaluate result = I 1 Many other plans (including types of joins, number of sites, semijoins, etc) C55 Query Optimization 3 Cost estimation As in centralized system: estimate result sizes But: # IOs may not be best metric e.g., Transmission time may dominate work work answer at site at site T1 T >>> TIME > or $ C55 Query Optimization Another reason why plain IOs not enough: Parallelism Plan A Plan B site 1 50 IOs 100 IOs site 70 IOs site 3 50 IOs Cost metrics IOs, Bytes transmitted, $, Can add together esponse time metric cannot add need scheduling and dependency info skew important Task 1 Task Task 3 C55 Query Optimization 5 C55 Query Optimization 6 1

2 Take into account: (in parallel/distributed system) tart up costs (for parallel operation) Data distribution costs/time Contention memory, disk, network, Assembling result ite 1 ite ite 3 ite Example: esponse time tartup Distri- earching Final bution +send results proc. C55 Query Optimization 7 C55 Query Optimization 8 Query Optimizer Cost model Plan space Deep tree vs bushy tree Enumeration/earch strategy Exhaustive (with pruning) Hill climbing (greedy) Query separation DD-1 (semi-join based) (1) Exhaustive - consider all query plans with a set of techniques - prune some plans (heuristics) C55 Query Optimization 9 C55 Query Optimization 10 Example: A B T > > T In generating plans, keep goal in mind: T T T T T 1 1 ( ) T (T ) ship semi ship T semi to join to join Heuristics: 1 Prune because cross-product not necessary Prune because larger relation first C55 Query Optimization 11 e.g.: Goal is parallelism in system with fast net, consider partitioning relation(s) first e.g.: Goal is reduction of net traffic, consider semi-joins C55 Query Optimization 1

3 () Hill climbing 1 Worse plans Better plans x Initial plan Hill Climbing Algorithm tep 1: Do initial processing tep : elect initial feasible solution (P 0 ).1 Determine the candidate result sites - sites where a relation referenced in the query exist. Compute the cost of transferring all the other referenced relations to each candidate site.3 P 0 = candidate site with minimum cost tep 3: Determine candidate splits of P 0 into P 1 = {P 1a, P 1b } 3.1 P 1a consists of sending one of the relations to the other relation's site 3. P 1b consists of sending the join of the relations to the final result site C55 Query Optimization 13 C55 Query Optimization 1 Hill Climbing Algorithm tep : eplace P 0 with P 1 that gives cost(p 1a ) + cost(local join) + cost(p 1b ) < cost(p 0 ) tep 5: ecursively apply steps 3 on P 1 until no such plans can be found tep 6: Check for redundant transmissions in the final plan and eliminate them. Example σ(u) V el ite ize tuple size = U 3 90 V 0 Goal: minimize data transmission C55 Query Optimization 15 C55 Query Optimization 16 tep 1: T V A B C T V el ite ize tuple size = T 3 30 V 0 T = σ(u) electivity is 1/3 tep : Initial plan send relations to one site What site do we send all relations to? To site 1: cost=0+30+0=90 To site : cost= =80 To site 3: cost=10+0+0=70 To site : cost= =60 C55 Query Optimization 17 C55 Query Optimization 18 3

4 P0: (1 ) ( ) T (3 ) Compute T V at site teps 3 & Consider sending each relation to neighbor: e.g.: 1 1 C55 Query Optimization 19 C55 Query Optimization 0 Assume: ize = 0 T = 5 T V = cost = 30 cost = 30 cost = 0 No savings Worse! 1 T T T cost = 50 cost = 35 cost = 5 A Win! A Bigger Win 5 0 T 3 C55 Query Optimization 1 C55 Query Optimization P1: P1a: ( 3) α = T P1b: (1 ) α (3 ) compute answer at site tep 5: epeat teps 3 & - Treat α = T as relation α 1 3 vs α 1 α α C55 Query Optimization 3 C55 Query Optimization

5 Hill climbing may miss best plan! Example: best plan could be: V PB: T (3 ) 30 β=t V β β T β ( ) β β = β β ( 1) 1 β = β β (1 ) Compute answer 1 33 = total [optional] T Costs could be low because is very selective C55 Query Optimization 5 (3) Query separation - separate query into or more steps - optimize each step independently C55 Query Optimization 6 Example: simple queries 1. Compute = ΠA[σc ] = ΠA[σc3 ]. Compute J = 3. Compute σc σc1 Ans = σc1{[j σc ] [J σc3 ]} A σc3 In other words: (a) Compute A values in answer (steps 1,) (b) Get tuples from sites with matching A values and compute answer (step 3) C55 Query Optimization 7 C55 Query Optimization 8 imple query - elations have a single attribute - Output has a single attribute e.g., J Idea Decompose query into Local processing imple query (or queries) Final processing Optimize simple query Philosophy Hard part is distributed join Do this part with only keys; get rest of data later impler to optimize simple queries C55 Query Optimization 9 C55 Query Optimization 30 5

6 DD-1 Algorithm tep 1: In the execution strategy (call it E), include all the local processing tep : eflect the effects of local processing on the database profile tep 3: Construct a set of beneficial semijoin operations (B) as follows : B = Ø For each semijoin J i B B J i if cost(j i ) < benefit(j i ) C55 Query Optimization 31 Consider the following query ELECT 3.C FOM 1,, 3 WHEE 1.A =.A AND.B = 3.B which has the following query graph and statistics: relation card tuple size relation size ite 1 ite ite A B attribute F size(π attribute ) 1.A A F: selectivity factor.b esult = F* C55 Query Optimization 3.B Beneficial semijoins: Assume: T MG = 0 J 1 = 1, whose benefit is 100 = (1 0.3)*3000 and cost is 36 J = 3, whose benefit is 1800 = (1 0.)*3000 and cost is 80 Nonbeneficial semijoins: J 3 = 1, whose benefit is 300 = (1 0.8)*1500 and cost is 30 Cost = transfer semijoin attribute = T MG + T T *(size) Benefit = cost of transferring irrelevant tuples (avoided by the semijoin) = (1-F)*size*T T J = 3, whose benefit is 0 and cost is 00 DD-1 Algorithm Iterative Process tep : emove the most beneficial J i from B and append it to E tep 5: Modify the database profile accordingly tep 6: Modify B appropriately compute new benefit/cost values check if any new semijoin need to be included in B tep 7: If B Ø, go back to tep. C55 Query Optimization 33 C55 Query Optimization 3 Iteration 1: emove J 1 from B and add it to E. Update statistics = size() = 900 (= ) size (.A) = 30*0.3 = 96 F (.A) = ~ = ~0. Iteration : Two beneficial semijoins: relation card tuple size relation size F attribute 1.A.A.B 3.B size(π attribute ) J = 3, whose benefit is 50 = (1 0.) 900 and cost is 80 J 3 = 1 ', whose benefit is 300=(1 0.8) 1500 and cost is 96 Add J to E NOT 110=(1-0.)*1500!! Update statistics size( ) = 360 (= 900*0.) Note: selectivity of may also change, but not important in this example. C55 Query Optimization 35 Iteration 3: No new beneficial semijoins. emove remaining beneficial semijoin J 3 from B and add it to E. Update statistics size(1) = 100 (= ) F (1.A) = ~ = 0. C55 Query Optimization 36 6

7 Assembly ite election tep 8: Find the site where the largest amount of data resides and select it as the assembly site Example: DD-1 Algorithm Amount of data stored at sites: ite 1: 100 ite : 360 ite 3: 000 Therefore, ite 3 will be chosen as the assembly site. DD-1 Algorithm Postprocessing tep 9: For each i at the assembly site, find the semijoins of the type i j where the total cost of E without this semijoin is smaller than the cost with it and remove the semijoin from E. tep 10: Permute the order of semijoins if doing so would improve the total cost of E. Example: Final strategy: end ( 1) 3 to ite 3 end (1 ) to ite 3 C55 Query Optimization 37 C55 Query Optimization 38 ummary: Query Optimization Cost/result estimation Three key components in optimizer Cost model, search space, enumeration algorithm In practice, avoid bad plans (rather than find optimal) trategies Exhaustive Hill climbing eparation DD-1 (semi-join based) C55 Query Optimization 39 7