Chapter 4 Distributed Query Processing

Transcription

1 Chapter 4 Distributed Query Processing Table of Contents Overview of Query Processing Query Decomposition and Data Localization Optimization of Distributed Queries Chapter4-1 1

2 1. Overview of Query Processing Objectives of Query Processing SQL Efficient execution strategy of low-level language : ñ Query Optimization SQL TotalCost : CPU, I/O, Communication Cost Response Time : Parallelization Chapter4-2 Layers of Query Processing SQL on Global Relations Query Decomposition Global Schema Algebraic Query on Global Relations Control Site Local Sites Data Localization Fragment Query Global Optimization Optimized Fragment Query with Communication Ops. Local Optimization Optimized Local Queries Fragment Schema Statistics on Fragments Local Schema Chapter4-3 2

3 2. Query Decomposition and Data Localization Query Decomposition Global relation ó SQL query global relation ó relational algebra Centralized DBMS ú Data Localization Global relation ó relational algebra fragment Fragment query Node communication cost. Chapter Query Decomposition Output Queries of Query Decomposition Semantically correct Good in the sense that redundant work is avoided Steps of Query Decomposition Normalization Analysis Elimination of Redundancy Rewriting Chapter4-5 3

4 Normalization Goal Query Normalized form to facilitate further processing Conjunctive normal form vs. Disjunctive normal from Example Select ENAME From E, G Where E. = G. AND G.JNO = J1 AND (DUR = 12 OR DUR = 24) CNF: E. = G. G.JNO = J1 (DUR = 12 DUR = 24) DNF: ( E. = G. G.JNO = J1 DUR = 12 ) (E. = G. G.JNO = J1 DUR = 24 ) Redundant Expression! Chapter4-6 Analysis Goal: Normalized query ó type incorrect semantically incorrect Type Incorrect Query SELECT E# FROM E WHERE ENAME > 2000; E#: E attribute ð ENAME: string type > 2000 incompatible Chapter4-7 4

5 Analysis Semantically Incorrect Query SELECT ENAME, RESP FROM E, G, J WHERE E. = G. AND DUR > 36 AND TITLE = Programmer AND JNAME = CAD/CAM ; JNAME = CAD/CAM þ.? Query Graph Node: result relation & operand relation Edge: operation(join, project) Join Graph: Subgraph of query graph Chapter4-8 Analysis Query Graph SELECT ENAME, RESP FROM E, G, J WHERE E. = G. AND G.JNO = J.JNO AND DUR > 36 AND TITLE = Programmer AND JNAME = CAD/CAM ; DUR > 36 E. = G. G G.JNO = J.JNO E. = G. G G.JNO = J.JNO TITLE = Programmer E ENAME RESP J JNAME = CAD/CAM Result E Join Graph J Query graph disconnected, semantically incorrect query! Chapter4-9 5

6 Elimination of Redundancy Idempotency rule ÿ. Example SELECT TITLE FROM E WHERE (NOT (TITLE = Programmer ) AND (TITLE = Programmer OR TITLE = Engineer ) AND NOT (TITLE = Engineer )) OR ENAME = J.Doe ; = SELECT TITLE FROM E WHERE ENAME = J.Doe ; ( p1 (p1 p2) p2) p3 = p3 Chapter4-10 Rewriting Goal: SQL relational algebra SQL relational algebra ñ relational algebra output: relational algebra tree Relational Algebra Tree Leaf node: FROM Root node: SELECT ó Projection Internal node: WHERE ó Chapter4-11 6

7 Rewriting Example Π ENAME SELECT ENAME FROM J, G, E WHERE G. = E. AND G.JNO = J.JNO AND ENAME J.Doe AND J.NAME = CAD/CAM AND (DUR = 12 OR DUR = 24); σdur = 12 DUR = 24 σjname = CAD/CAM σename J.Doe JNO J G E Chapter4-12 Rewriting Combine sequence of unary operations Π ENAME Perform selection as early as possible. Π ENAME σename J.Doe JNAME = CAD/CAM (DUR = 12 DUR = 24) JNO Π JNO,ENAME JNO Π JNO Π JNO, Π,ENAME J G E σjname = CAD/CAM σdur = 12 DUR = 24 σename J.Doe J G E Chapter4-13 7

8 2.2 Data Localization Global relation ó relational algebra tree fragment ó relational algebra tree Generic Query Relational algebra tree leaf node fragment definition Generic query ó reduction Chapter4-14 Reduction for Primary Horizontal Fragmentation Rule1 ReductionwithSelection R: R 1,R 2,,R w (R k = σ Pk (R)) σ Pi (R k )) = if x in R : (p i (x) p k (x)) R k relational algebra tree Rule2 ReductionwithJoin R i R k = if x in R i, y in R k : (p i (x) p k (x)) Join empty fragment join. Reduced query generic query ñ? Worst case: Cartesian product of two sets of fragments Partial joins can be done in parallel. Chapter4-15 8

9 Example of Rule 1 Sample Relation E E 1 = σ <= E3 (E) E 2 = σ E3 < <= E6 (E) E 3 = σ > E6 (E) Sample Query SELECT * FROM E WHERE = E5 ; Generic Query σ = E5 E 1 E 2 E 3 Reduced Query σ = E5 E 2 Chapter4-16 Example of Rule 2 Sample Relation E, G G 1 = σ <= E3 (G) G 2 = σ > E3 (G) Sample Query SELECT * FROM E, G WHERE E. = G.; Generic Query E 1 E 2 E 3 G 1 G 2 Reduced Query E 1 G 1 E 2 G 2 E 3 G 2 Chapter4-17 9

10 Reduction for Vertical Fragmentation Rule 3 R: R 1,R 2,,R w (R k = Π A (R), where A A) Π D, Key (R k ) is useless if D A. Reduction for Vertical Fragmentation Determine the useless intermediate relations. Remove the sub-trees that produce them. Chapter4-18 Example of Rule 3 Sample Relation E E 1 = Π,ENAME (E) E 2 = Π,TITLE (E) Sample Query SELECT ENAME FROM E; Generic Query Π ENAME E 1 E 2 Reduced Query Π ENAME E 1 Chapter

11 Reduction for Derived Fragmentation Transformation Distribute joins over unions. Apply rule 2. Join Reduction Reduced query generic query ñ! Sample Relation E, G E 1 = σ TITLE= Programmer (E) E 2 = σ TITLE Programmer (E) G 1 = G E1 G 2 = G E2 Sample Query SELECT * FROM E, G WHERE G. = E. AND TITLE = Engineer ; Chapter4-20 Generic Query Reduced Query σtitle= Engineer σtitle= Engineer G 1 G 2 E 1 E 2 G 2 E 2 Query after Pushing Selection Down Query after Moving Unions Up σtitle= Engineer σtitle= Engineer σtitle= Engineer G 1 G 2 E 2 E 2 G 1 E 2 G 2 Chapter

12 Reduction for Hybrid Fragmentation Sample Relation E E 1 = σ <= E4 (Π,ENAME (E)) E 2 = σ > E4 (Π,ENAME (E)) E 3 = Π,TITLE (E) Sample Query SELECT ENAME FROM E WHERE = E5 ; Generic Query E 1 E 2 Π ENAME σ= E5 E 3 Reduced Query Π ENAME σ= E5 E 2 Chapter Optimization of Distributed Queries Goal of Query Optimization Finding an optimal ordering of operations for a given query Table of Contents Inputs to Query Optimization Centralized Query Optimization Join Ordering in Fragment Queries Distributed Query Optimization Algorithms Chapter

13 3.1 Inputs to Query Optimization Cost Model Reducing the Total Cost C cpu * #insts + C I/O * #I/Os + C msg * #msgs + C tr * #bytes Major factors in WAN: Communication cost Major factors in LAN: All three components Reducing the Response Time #x: ú þ ó ó response time Response Time Total Cost (?) Chapter4-24 Database Statistics Relation R, A = {A 1,A 2,,A n }, F = {R 1,R 2,,R r } length(a i ), card(π Ai (R k )) : # of distinct values of A i on R k min(a i ), max(a i ) card(dom[a i ]) card(r k ) size(r) = card(r) * length(r) Join Selectivity Factor, SF J, of relations R and S SF J (R, S) = card(r JN S) / (card(r) * card(s)) Chapter

14 Cardinalities of Intermediate Results Selection: card(σ F (R)) = SF S (F) * card(r) SF S (A=value) = 1 / card(π A (R)) SF S (A>value) = (max(a) value) / (max(a) min(a)) Projection: card(π A (R)) = card(r), if A is a key. Cartesian Product: card(r S) = card(r) * card(s) Join: card(r JN A=B S) = card(s), if A is a key of R. card(r JN S) = SF J * card(r) * card(s) Semijoin: card(r SJ A S) = SF SJ (S.A) * card(r) SF SJ (R SJ A S) = card(π A (S)) / card(dom[a]) Union: max{card(r), card(s)} ~ card(r) + card(s) Difference: 0 ~ card(r) Chapter Centralized Query Optimization Relationships between CQO vs. DQO query local query, local query ó CQO DQO CQO DQO = CQO + Communication Effect Two Algorithms INGRES Algorithm: Dynamic Optimization System R Algorithm: Static Optimization Chapter

15 INGRES Algorithm ä Multi-variable query mono-variable query sequence Q:Q 1 Q 2 Q n (Q i :Q i 1 ) Detachment: single variable predicate ó relation Substitution: ø detachment mutivariable query ó relation query mono-variable query Chapter4-28 Example Detachment Sample Query: Q SELECT E.ENAME FROM E, G, J WHERE E. = G. AND G.JNO = J.JNO AND JNAME = CAD/CAM ; 1 ST Detachment: Q 1 SELECT J.JNO INTO JVAR FROM J WHERE JNAME = CAD/CAM ; 2 ND Detachment: Q 2 SELECT G. INTO GVAR FROM G, JVAR WHERE G.GNO = JVAR.JNO; 3 RD Detachment: Q 3 SELECT E.ENAME FROM E, GVAR WHERE E. = GVAR.; Q = Q 1 Q 2 Q 3 Chapter

16 Example Substitution 3 RD Detachment: Q 3 SELECT E.ENAME FROM E, GVAR WHERE E. = GVAR.; Assume GVAR: Consists of 2 tuples <E1> <E2> 1 ST Substitution: Q 31 SELECT E.ENAME FROM E WHERE E. = E1 ; 2 ND Substitution: Q 32 SELECT E.ENAME FROM E WHERE E. = E2 ; Chapter4-30 System R Algorithm input: QT query tree with n relations output: output the result of execution begin for each relation Ri QT do for each access path APij to Ri do determine cost(apij) best_api with minimum cost end-for for each order (Ri1, Ri2,, Rin) with i = 1,, n! do build strategy ( ((best_api1 JN Ri2) JN Ri3) JN JN Rin) compute the cost of strategy end-for output strategy with minimum cost end. Chapter

17 Example System R Algorithm Sample Query: Q SELECT E.ENAME FROM E, G, J WHERE E. = G. AND G.JNO = J.JNO AND JNAME = CAD/CAM ; Assume E has an index on. GhasanindexonJNO. J has an index on JNO and an indexonjname. Join Graph G JNO Single-Relation Access Plan E: Sequential Scan G: Sequential Scan J: Index Scan on JNAME E J Chapter4-32 Alternative Join Orders (EJNG)>(GJNE) (G JN J) > (J JN G) E G J E G E J G E G J J G J E (G E) J (J G) E Chapter

18 3.3 Join Ordering in Fragment Queries Join Ordering in a Distributed Context Communication Cost JoinOrdering Communication Cost Semijoin Chapter4-34 Join Ordering RJNS R: site 1,S:site2 If size(r) < size(s): R site 2 If size(s) < size(r): S site 1 ó Join ä:. ó & Chapter

19 Example Join Ordering J JN JNO E JN G 1. E site 2, (E JN G) site 3 2. G site 1, (E JN G) site 3 Site 2 G 3. G site 3, (G JN J) site 1 4. J site 2, (J JN G) site 1 5. E site 2, J site 2 JNO E J Site 1 Site 3 size(e), size(g), size(j) size(e JN G), size(g JN J) Chapter4-36 Semijoin Based Algorithms RJN A S(R site1,s site2 ) Π A (S) site 1 Site 1 computes R = R SJ A S R site 2 Site 2 computes R JN A S Semijoin is better if size(π A (S) + size(r SJ A S) < size(r) General Idea EJNGJNJ E JN G JN J (E = E SJ G, G = G SJ J) E = E SJ (G SJ J), ø Semijoin Program Full Reducer using Acyclic Join Graph Chapter

20 Join vs. Semijoin Semijoin is effective, If length(join attribute) < length(entire tuple) Semijoin has good selectivity. Semijoin is not effective, If Neither of the above condition is met. Communication cost is not significant. Increased local processing cost Intermediate result ó index Chapter Distributed Query Optimization Algorithm R* Algorithm Extension of System R s Optimizer LAN WAN SDD-1 Algorithm Extension of hill-climbing Algorithm Semijoin algorithm WAN Chapter

21 R* Algorithm Exhaustive Searching Permutations of join order, join methods, result site, access path to the internal relation, intersite transfer mode Heuristic search space Intersite Transfer Mode Ship-whole Fetch-as-needed External relation scan internal relation ñ ( semijoin) Chapter4-40 R* Algorithm R JN S (R: external, S: internal) Notation LC: local processing cost (I/O + CPU time) CC: communication cost s=card(s SJ A R) / card(r) Average # of S s tuple that match one tuple of R R S 4 Chapter

22 1: R S ÿ Cost = LC(retrieve card(r) tuples from R) + CC(size(R)) + LC(retrieve s tuples from S) * card(r) 2: S R ÿ Cost = LC(retrieve card(s) tuples from S) + CC(size(S)) + LC(store card(s) tuples in T) + LC(retrieve card(r) tuples from R) + LC(retrieve s tuples from T) * card(r) 3: R scan S Cost = LC(retrieve card(r) tuples from R) + CC(length(A)) * card(r) + LC(retrieve s tuples from S) * card(r) + CC(s * length(s)) * card(r) 4: R S 3 Cost = LC(retrieve card(s) tuples from S) + CC(size(S)) + LC(retrieve card(r) tuples from R) + CC(size(R)) + LC(store card(s) tuples in T) + LC(retrieve s tuples from T) * card(r) Chapter4-42 R* Algorithm size(r) >> size(s) 2 communication cost ñ, 2 S ÿ local processing cost 1 3 S size(r) join selectivity : 3 Otherwise, 1 4? 3 ÿ. Chapter

23 SDD-1 Algorithm ä Selection of Beneficial Semijons Cost(R SJ A S) = C MSG +C TR * size(π A (S)) Benefit(R SJ A S)=(1 SF SJ (S.A)) * size(r) * C TR Beneficial Semijoin: Benefit > Cost Assembly Site Selection Select site i such that i stores the largest amount of data after all local operations. Post-optimization Assembly site ÿ ó semijoin benefit Chapter4-44 Example SDD-1 Algorithm Sample Query: Q SELECT * FROM E, G, J WHERE E. = G. AND G.JNO = J.JNO; relation E G J card tuple size relation size Join Graph Site 2 attribute SF SJ size(π attribute ) E G JNO J E. G. G.JNO J.JNO Site 1 Site 3 Chapter

24 Selection of Beneficial Semijoins Step 1: Calculate the benefit and cost of semijoins SJ 1 : G SJ E, benefit = (1 0.3) * 3000, cost = 120 SJ 2 : G SJ J, benefit = (1 0.4) * 3000, cost = 200 SJ 3 : E SJ G, benefit = (1 0.8) * 1500, cost = 400 SJ 4 : J SJ G, benefit = (1 1.0) * 2000, cost = 400 Most beneficial semijoin = SJ 1 Change the size of G to 900 = 3000 * 0.3 ChangeSF SJ (G.) to 0.24 = 0.8 * 0.3 InsertSJ 1 to execution strategy Step 2: Calculate again the benefit and cost of semijoins SJ 2 : G SJ J, benefit = (1 0.4) * 900, cost = 200 SJ 3 : E SJ G, benefit = (1 0.24) * 1500, cost = 400 Change the size of E to 360 = 1500 * 0.24 Step 3: Select SJ 2 and G s size is set to 360 = 900 * 0.4 Chapter4-46 Assembly Site Selection After reduction, the amount of data stored at each site is Site1(E):360 Site2(G):360 Site 3 (J): 2000 chosen as assembly site Post-optimization J ó semijoin, post-optimization. Final Execution Strategy (G SJ E) SJ J site 3 ESJG site 3 Site 3 Chapter