Optimization of Distributed Queries

Transcription

1 Query Optimization Optimization of Distributed Queries Issues in Query Optimization Joins and Semijoins Query Optimization Algorithms Centralized query optimization: Minimize the cots function Find (the best) query execution plan in the space of equivalent query trees Gather statistics about relations Distributed query optimization brings additional issues Linear query trees are not necessarily a good choice Bushy query trees are not necessarily a bad choice What and where to ship the relations How to ship relations (ship as a whole, ship as needed) When to use semi-joins instead of joins Spring, 2006 Arturas Mazeika Page 2 Search Space (Definition) Search Space (Reduction of the Search Space) Search space is characterized by alternative execution plans Typically the main issue is to optimize the joins For N relations, there are O(N!) (equivalent join trees that can be obtained by applying commutativity and associativity rules) Example: SELECT ENAME,RESP FROM EMP, ASG, PROJ WHERE EMP.ENO=ASG.ENO AND ASG.PNO=PROJ.PNO Restrict by means of heuristics Perform unary operations before binary operations, etc Restrict the shape of the join tree Consider the type of trees (linear trees, vs. bushy ones) Linear Join Tree Bushy Join Tree Spring, 2006 Arturas Mazeika Page 3 Spring, 2006 Arturas Mazeika Page 4

2 Search Space (Scanning the Search Space) Search Space (Scanning the Space, Example) Deterministic There are two main strategies to scan the search space Deterministic Start from base relations and build plans by adding one relation at each step Dynamic programming: breadth-first Greedy: depth-first Randomized Search for optimalities around a particular starting point Iterative improvement Trades optimization time for execution time The strategy is better when more than 5-6 relations Randomized Spring, 2006 Arturas Mazeika Page 5 Cost Functions (Strategies) Spring, 2006 Arturas Mazeika Page 6 Cost Functions (Total Time) Two different strategies can be used: Reduce total time Reduce each cost (in terms of time) component individually Do as little for each cost component as possible Optimize the utilization of the resources (increase system throughput) Reduce response time Do as many things in parallel as possible May increase total time because of increased total activity Add up all individual components: Total cost = CPU cost + I/O cost + communication cost CPU cost = unit instruction cost no. of instructions I/O cost = unit disk I/O cost no. of disk I/Os communication cost = message initiation + transmission Spring, 2006 Arturas Mazeika Page 7 Spring, 2006 Arturas Mazeika Page 8

3 Cost Functions (Total Time, Network Issues) Cost Functions (Response Time) The individual components of the total cost have different weights: Wide area network Message initiation and transmission costs high Local processing cost is low (fast mainframes or minicomputers) Ratio of communication to I/O costs is 20:1 Local area networks Communication and local processing costs are more or less equal Ratio of communication to I/O costs is 1:1.6 (10MB/s network) Definition: Elapsed time between the initiation and the completion of a query Costs: Response time = CPU time + I/O time + communication time CPU time = unit instruction time no. of sequential instructions I/O time = unit I/O time no. of sequential I/Os communication time = unit msg initiation time no. of sequential msg + unit transmission time no. of sequential bytes Spring, 2006 Arturas Mazeika Page 9 Cost Functions (Response Time, Example) Spring, 2006 Arturas Mazeika Page 10 Statistics The primary cost factor is the size of intermediate relations Assume that only the communication cost is considered Total time = 2 message initialization time + unit transmission time (x+y) Response time = max{time to send x from 1 to 3, time to send y from 2 to 3} Time to send x from 1 to 3 = message initialization time + unit transmission time x Time to send y from 2 to 3 = message initialization time + unit transmission time y It is costly to compute the intermediate relations precisely. Instead global statistics of relations and fragments are computed and used to provide approximate sizes Spring, 2006 Arturas Mazeika Page 11 Spring, 2006 Arturas Mazeika Page 12

4 Statistics (Relations and Fragments) Let R[A 1, A 2,..., A k ] be a relation fragmented into R 1, R 2,..., R r. Relation statistics min and max values of each attribute min{a i }, max{a i }. length of each attribute length(a i ) number of distinct values in each fragment (cardinality) card(a i ), (card(dom(a i ))) Fragment statistics cardinality of the fragment card(r i ) cardinality of each attribute of each fragment card(π Ai (R j )) Statistics (Selectivity) size(r) = card(r) length(r) card(σ P (R)) = SF σ (P ) card(r) Selectivity factors (with assumption of uniformality): 1 SF σ (A = value) = card(π A (R)) max(a) value SF σ (A > value) = max(a) min(a) value min(a) SF σ (A < value) = max(a) min(a) Properties of the selectivity factor of the selection SF σ (p(a i ) p(a j )) = SF σ (p(a i )) SF σ (p(a j )) SF σ (p(a i ) p(a j )) = SF σ (p(a i )) + SF σ (p(a j )) (SF σ (p(a i )) SF σ (p(a j )) SF σ (A {values}) = SF σ (A {values}) card({values}) Spring, 2006 Arturas Mazeika Page 13 Statistics (Projection, Cartesian Product, Union, Set Difference) Spring, 2006 Arturas Mazeika Page 14 Statistics (Joins) Projection Cartesian Product card(π A (R)) = card(r) card(r S) = card(r) card(s) Union upper bound: card(r S) = card(r) + card(s) lower bound: card(r S) = max{card(r), card(s)} Set Difference upper bound: card(r S) = card(r) lower bound: 0 Selectivity factor for joins: General Case: SF = card(r S) card(r) card(s) card(r S) = SF card(r) card(s) Special case: R.A is a key of R and S.A is a foreign key of S; card(r R.A=S.A S) = card(s) Spring, 2006 Arturas Mazeika Page 15 Spring, 2006 Arturas Mazeika Page 16

5 Statistics (Semijoins) Join and Semijoin Ordering Selectivity factor for semijoins General case: SF < (R < A S) = SF < (S.A) = card(π A(S)) card(dom[a]) card(r < A S) = SF < (S.A) card(r) Introduction to join ordering in fragment queries Join ordering algorithms: INGRES and distributed INGRES System R and System R Semijoin ordering Hill Climbing and SDD-1 Spring, 2006 Arturas Mazeika Page 17 Introduction to Join Ordering (Two Distributed not Fragmented Relations) Spring, 2006 Arturas Mazeika Page 18 Introduction to Join Ordering (Three Distributed not Fragmented Relations 1/2) Relations involving more that two relations are substantially more complex Consider the following example Two relations located at different sites P ROJ P NO ASG ENO EMP Move the smaller relation to the other site Spring, 2006 Arturas Mazeika Page 19 Spring, 2006 Arturas Mazeika Page 20

6 Introduction to Join Ordering (Three Distributed not Fragmented Relations 2/2) Semijoin Algorithms (Expression of Joins as Semijoins) Plan 1: EMP Site 2 Site 2: EMP =EMP ASG EMP Site 3 Site 3: EMP PROJ Plan 2: ASG Site 1 Site 1: EMP =EMP ASG EMP Site 3 Site 3: EMP PROJ Plan 3: ASG Site 3 Site 3: ASG =ASG PROJ ASG Site 1 Site 1: ASG EMP Plan 4: PROJ Site 2 Site 2: PROJ =PROJ ASG PROJ Site 1 Site 1: PROJ EMP Plan 5: EMP Site 2 PROJ Site 2 Site 2: EMP PROJ ASG Semijoins can be used to implement joins For example, consider two relations: R (located at site 1) and S (located and site 2) We have two alternatives: Do the join R S Perform one of the semijoin equivalents: R S (R < S) S R (S < R) (R < S) (S < R) Spring, 2006 Arturas Mazeika Page 21 Spring, 2006 Arturas Mazeika Page 22 Semijoin Algorithms (Transfer of Data) INGRES Centralized Algorithm (Overview) To perform the join R S: send R to Site 2 Site 2 computes R S To perform the semijoins (R < S) S: S = Π A (S) S Site 1 Site 1 computes R = R < S R Site 2 Site 2 computes R S Semijoin is better if INGRES is based on three ideas: Decompose query q into q 1 q 2 q n The output or q i is send to the input of q i+1. Each q i is a mono-variable query Detachment and tuple substitution is used to achieve the decomposition There s a processor that can processes each one variable query efficiently size(π A (S)) + size(r < S) < size(r) Spring, 2006 Arturas Mazeika Page 23 Spring, 2006 Arturas Mazeika Page 24

7 INGRES Centralized Algorithm (Example, Detachment 1/2) Consider the query of names of employees working on CAD/CAM project. q 1 : SELECT EMP.ENAME FROM EMP, ASG, PROJ WHERE EMP.ENO=ASG.ENO AND ASG.PNO=PROJ.PNO AND PROJ.PNAME="CAD/CAM" Change q 1 into q 11 q : q 11 : q : SELECT PROJ.PNO INTO JVAR FROM PROJ WHERE PROJ.PNAME="CAD/CAM" SELECT EMP.ENAME FROM EMP,ASG,JVAR WHERE EMP.ENO=ASG.ENO AND ASG.PNO=JVAR.PNO Spring, 2006 Arturas Mazeika Page 25 INGRES Centralized Algorithm (Example, Tuple Substitution) q : INGRES Centralized Algorithm (Example, Detachment 2/2) Change q into q 12 q 13 : q 12 : q 13 : SELECT EMP.ENAME FROM EMP,ASG,JVAR WHERE EMP.ENO=ASG.ENO AND ASG.PNO=JVAR.PNO SELECT ASG.ENO INTO GVAR FROM ASG,JVAR WHERE ASG.PNO=JVAR.PNO SELECT EMP.ENAME FROM EMP,GVAR WHERE EMP.ENO=GVAR.ENO q 11 is a mono-variable query, q 12 and q 13 are not mono-variable queries (yet) Spring, 2006 Arturas Mazeika Page 26 Distributed INGRES Algorithm Assume GV AR consists only the tuples {E1, E2} then q 13 is rewritten with tuple substitution in the following way q 131 SELECT EMP.ENAME FROM EMP WHERE EMP.ENO="E1" q 132 : SELECT EMP.ENAME FROM EMP WHERE EMP.ENO="E2" The distributed INGRES algorithm is very similar to the centralized INGRES algorithm. In addition to the centralized INGRES the distributed one should break up each query q i into sub-queries that operation on fragments. Optimization with respect to communication cost or response time is possible Spring, 2006 Arturas Mazeika Page 27 Spring, 2006 Arturas Mazeika Page 28

8 System R Algorithm System R Algorithm (Example 1/2) Consider the query of the names of employees working on the CAD/CAM project P ROJ P NO ASG ENO EMP The algorithm is based on two ideas: Execute simple (mono-variable) queries based on the best plan (using index, scan, etc) Execute joins Determine the possible ordering of joins Determine the cost of each ordering Choose the join ordering with minimal cost Spring, 2006 Arturas Mazeika Page 29 System R Algorithm (Example 2/2) Spring, 2006 Arturas Mazeika Page 30 Distributed System R Algorithm Only the whole relations can be distributed One central site decides the ordering of joins (intra-site optimization). The local sites decides the local optimizations the local joins (inter-site optimization) Three main strategies for join R S: Join at R site Join at S site Join at the result site Data transfer possibilities: Ship-whole Fetch-as-needed Best total join order is either ((ASG EMP) PROJ) or ((PROJ ASG) EMP) Spring, 2006 Arturas Mazeika Page 31 Spring, 2006 Arturas Mazeika Page 32

9 Distributed System R Algorithm (Data Transfer) Distributed System R Algorithm (Strategy 1) Ship whole Larger data transfer Smaller number of messages Better if relations are small Fetch as needed Number of messages = O(cardinality of external relation) Data transfer per message is minimal Better if relations are large and the selectivity is good Move outer relation tuples to the site of the inner relation (a) Retrieve outer tuples (b) Send them to the inner relation site (c) Join them as they arrive Total Cost = cost(retrieving qualified outer tuples) + no. of outer tuples fetched cost(retrieving qualified inner tuples) + msg. cost (no. outer tuples fetched avg. outer tuple size)/msg. size Spring, 2006 Arturas Mazeika Page 33 Distributed System R Algorithm (Strategy 2) Spring, 2006 Arturas Mazeika Page 34 Distributed System R Algorithm (Strategy 3) Move inner relation to the site of outer relation. We cannot join as they arrive; they need to be stored Total Cost = cost(retrieving qualified outer tuples) + no. of outer tuples fetched cost(retrieving matching inner tuples from temporary storage) + cost(retrieving qualified inner tuples) + cost(storing all qualified inner tuples in temporary storage) + msg. cost (no. of inner tuples fetched avg. inner tuple size)/msg. size Move both inner and outer relations to another site Total cost = cost(retrieving qualified outer tuples) + cost(retrieving qualified inner tuples) + cost(storing inner tuples in storage) + msg. cost (no. of outer tuples fetched avg. outer tuple size)/msg. size + msg. cost (no. of inner tuples fetched avg. inner tuple size)/msg. size + no. of outer tuples fetched cost(retrieving inner tuples from temporary storage) Spring, 2006 Arturas Mazeika Page 35 Spring, 2006 Arturas Mazeika Page 36

10 Distributed System R Algorithm (Strategy 4) Hill Climbing Algorithm Fetch inner tuples as needed (a) Retrieve qualified tuples at outer relation site (b) Send request containing join column value(s) for outer tuples to inner relation site (c) Retrieve matching inner tuples at inner relation site (d) Send the matching inner tuples to outer relation site (e) Join as they arrive Total Cost = cost(retrieving qualified outer tuples) + msg. cost (no. of outer tuples fetched) + no. of outer tuples fetched (no. of inner tuples fetched avg. inner tuple size msg. cost/msg. size) + no. of outer tuples fetched cost(retrieving matching inner tuples for one outer value) 1. : Select initial feasible solution ES0 1.1 Determine the candidate result sites - sites where a relation referenced in the query exist 1.2 Compute the cost of transferring all the other referenced relations to each candidate site 1.3 ES0 = candidate site with minimum cost 2. Determine candidate splits of ES0 into ES1, ES2 2.1 ES1 consists of sending one of the relations to the other relation s site 2.2 ES2 consists of sending the join of the relations to the final result site 3. Replace ES0 with the split schedule which gives cost(es1) + cost(local join) + cost(es2) < cost(es0) 4. Recursively apply steps 2-3 on ES1 and ES2 until no such plans can be found 5. Check for redundant transmissions in the final plan and eliminate them Spring, 2006 Arturas Mazeika Page 37 Hill Climbing Algorithm (Example, Assumption) What are the salaries of engineers who work on the CAD/CAM project? Π SAL (P AY T IT LE EMP ENO (ASG P NO (σ P NAME= CAD/CAM (P ROJ)))) Spring, 2006 Arturas Mazeika Page 38 Hill Climbing Algorithm (Example, Step 1:Initial feasible solution) Alternative 1: Resulting site is Site 1 Costs Relation Size Site EMP 8 1 PAY 4 2 PROJ 4 3 ASG 10 4 Table 1: Statistics Total cost = cost(paysite 1) + cost(asgsite 1) + cost(projsite 1) = = 15 Alternative 2: Resulting site is Site 2 Alternative 3: Resulting site is Site 3 Total cost = = 19 Total cost = = 22 Assumptions: Size of relations is defined as their cardinality Minimize total cost Transmission cost between two sites is 1 Ignore local processing cost Alternative 4: Resulting site is Site 4 Total cost = = 13 Therefore ES0 = EMP Site 4; S Site 4; PROJ Site 4 Spring, 2006 Arturas Mazeika Page 39 Spring, 2006 Arturas Mazeika Page 40

11 Hill Climbing Algorithm (Example, Step 2: Candidate Splits 1/2) Hill Climbing Algorithm (Example, Step 2: Candidate Splits 2/2) Alternative 1: ES1, ES2, ES3 where ES1: EMP Site 2 ES2: (EMP PAY) Site 4 ES3: PROJ Site 4 Alternative 2: ES1, ES2, ES3 where ES1: PAY Site 1 ES2: (PAY EMP) Site 4 ES3: PROJ Site 4 Costs Alternative 1: cost(empsite 2) + cost((emp PAY) Site 4) + cost(proj Site 4) = = 17 Alternative 2: cost(paysite 1) + cost((pay EMP) Site 4) + cost(proj Site 4) = = 13 Alternative 2 is the best Spring, 2006 Arturas Mazeika Page 41 Hill Climbing Algorithm (Conclusion) Spring, 2006 Arturas Mazeika Page 42 SDD-1 Problems : Greedy algorithm determines an initial feasible solution and iteratively tries to improve it If there are local minimums, it may not find the global minima If the optimal schedule has a high initial cost, it won t find it since it won t choose it as the initial feasible solution Example : A better schedule is PROJ Site 4 ASG = (PROJ ASG) Site 1 (ASG EMP) Site 2 Total cost= = 5 Improves Hill Climbing Algorithm with the following: Semijoins More elaborate statistics Initial plan is selected better Post-optimization step is introduced Spring, 2006 Arturas Mazeika Page 43 Spring, 2006 Arturas Mazeika Page 44

12 Conclusion Distributed query optimization is more complex that centralized query processing, since (i) bushy query trees are not necessarily a bad choice, (ii) one needs to decide what, where, and how to ship the relations between the sites Query optimization searches the optimal query plan (tree) For N relations, there are O(N!) equivalent join trees. To cope with the complexity heuristics and/or restricted types of trees are considered There are two main strategies in query optimization: randomized and deterministic (Few) semi-joins can used to implement a join. The semi-joins requires more operations to perform, however reduces the data transfer rate INGRES, System R, and Hill Climbing algorithms are used to optimize quries Spring, 2006 Arturas Mazeika Page 45