CS448/648 Database Systems Implementation. Volcano: A top-down optimizer generator

Transcription

1 CS448/648 Database Systems Implementation Volcano: A top-down optimizer generator 1

2 Outline Definitions Physical and Logical expressions Equivalency of expressions Transformation rules Top-down query optimization Volcano Model Physical and logical plan generation Search algorithm 2

3 Definitions A logical operator is a function to map the operator s inputs to its outputs. (e.g. join, selection, projection) A physical operator is an algorithm that implements a logical operator (e.g. hash join, merge join) An operator expression is a hierarchy of operators. An operator expression is logical or physical if its operators are logical or physical, respectively. Logical properties are those that can be derived from logical algebra expressions (e.g., schema, expected size). 3

4 Definitions Physical properties, are the properties of physical layout of data such as being sorted or partitioned. Enforcer operators : are those physical operators that does not have a corresponding logical operator. Their goal is to enforce specific physical properties (e.g. certain order). An optimization goal is a pair of a logical expression and a vector of physical properties. A plan is an expression made up entirely of physical operators. 4

5 Logical / Physical Expressions Physical expression (ii) is equivalent to (i.e., implements) the logical expression (i). 5

6 Equivalent Expressions The following two logical expressions (A and B) are equivalent. An expression can be converted to another equivalent expression using transformation rules. Expression A is converted to B using associativity rule. Each set of equivalent expressions represents an equivalence node or class (RST in our example). 6

7 Logical / Physical rules The algebraic rules of expression equivalence, (e.g., commutativity or associativity) are specified using transformation rules. These transformation rules are used to get equivalent logical expressions. The possible mappings of operators to algorithms are specified using implementation rules. 7

8 Logical Algebraic Rules Commutative and Associative Laws R U S = S U R, R U (S U T) = (R U S) U T R S = S R, R (S T) = (R S) T R S = S R, R (S T) = (R S) T Distributive Laws R (S U T) = (R S) U (R T) 8

9 Logical Algebraic Rules Laws involving selection: σ C AND C (R) = σ C (σ C (R)) = σ C (R) σ C (R) σ C OR C (R) = σ C (R) U σ C (R) σ C (R S) = σ C (R) S (When C involves only attributes of R) σ C (R S) = σ C (R) S σ C (R U S) = σ C (R) U σ C (S) σ C (R S) = σ C (R) S 9

10 Logical Algebraic Rules Laws involving projections Π M (Π N (R)) = Π M (R) Where M is a subset of N Π M (R S) = Π M (Π P (R) Π Q (S)) Where P, Q are supersets of M and contain join keys

11 Implementation Rules Example : Join can be implemented by merge-join, hash join or nested-loop join. Each algorithm has: Required physical properties of its inputs. Physical properties of its output. Cost function. For example, merge-join requires the inputs to be sorted on the join attributes and produces the output sorted. Enforcer operators can be used when input properties do not satisfy the algorithm requirements. For example, sort operator can be used to allow merge-join of unsorted relations. 11

12 Top-down optimization The translation from a user query into a logical algebra expression is first performed by the query parser. Start with one logical expression and apply transformation rules to obtain new expressions. Keep the cheapest physical expression that implements a logical expression and physical properties. The output of the optimizer is a plan, which is a physical expression. 12

13 Volcano: optimizer generator 1. Given a specific logical algebra, physical algebra and rules, Volcano generate a top-down query optimizer. 2. A model specification is translated into optimizer source code. 3. Optimizer source code is compiled and linked with the other DBMS software such as the query execution engine and with the search engine that is part of the Volcano optimization software 4. When the DBMS is operational and a query is entered, the query is passed to the optimizer, which generates an optimized plan for it. 13

14 Model specification The optimizer implementor provides A set of logical operators and algebraic transformation rules. A set of algorithms, enforcers and implementation rules. An implementation for Abstract Data Type (ADT) named "cost" with functions for basic arithmetic and comparison. An implementation for ADT logical properties. An implementation for ADT physical property vector including comparisons functions (equality and cover), An applicability function for each algorithm and enforcer. A cost function for each algorithm and enforcer. A property function for each operator, algorithm, and enforcer. 14

15 Optimization algorithm Overview 15

16 Logical Plan Space Generation Data Structure: Logical Query Directed Acyclic Graph (LQDAG) Algorithm : Top-down exhaustive enumeration, i.e. expand LQDAG applying all possible transformation rules. 16

17 Physical Plan Space Generation Data Structure: Physical Query DAG (PQDAG) Refinement of LQDAG : each node of LQDAG and physical property gives rise to one PQDAG equivalence Node. Operation nodes: algorithms and enforcers Generation: Again, exhaustive top-down enumeration. 17

18 18

19 Searching Logical/Physical Space Top-down, depth-first search using recursive algorithm. Cache best plans for each equivalence node for future re-use (memoization). Prune choices whenever they exceed cost limit. 19

20 FindBestPlan (LogExpr, PhysProp, CostLimit) returns Plan,Cost 1. If the pair LogExpr and PhysProp is in the look-up table if its state is in progress, then ignore and return else if the cost in the look-up table < CostLimit, then return Plan and Cost else, return failure 2. Create the set of possible "moves" from: - applicable transformations - algorithms that give the required PhysProp - enforcers for required PhysProp 3. Order the set of moves by promise; and for the most promising moves: if the move uses a transformation Apply the transformation creating NewLogExpr Call FindBestPlan (NewLogExpr, PhysProp, CostLimit) else if the move uses an algorithm For each input i while TotalCost < CostLimit Determine required physical properties PP for i Cost = FindBestPlan (i, PP, CostLimit cost of algorithm) else /* move uses an enforcer */ Modify PhysProp for enforced property Call FindBestPlan for LogExpr with new PhysProp (cost less enforcing) 5. If LogExpr is not in the look-up table, then insert LogExpr into the look-up table 6. Insert PhysProp and best plan found into look-up table 7. Return best Plan and Cost 20

21 Example SELECT * FROM R, S, T WHERE R.x=S.x AND S.y=T.y ORDER BY S.y 21

22 FindBestPlan ((R S) T), S.y, 500) (R S) T R (S T) (R S) MJ T Cost=100 Sort S.y ((R S) T) Cost=200 FindBestPlan (R (S T)), S.y, 500) FindBestPlan ((R S) T),NULL,300). FindBestPlan (R S, S.y, 400) FindBestPlan (T, T.y, 400)..

23 Algorithm Details Memoization is used to prevent redundant optimization effort. To detect that two rules are inverses of each other, the current expression and physical property vector is marked as "in progress. For each combination of physical properties for which an equivalence class has already been optimized, the best plan found is kept. Branch-and-bound pruning: Once a complete plan is known for a logical expression and a physical property vector, no other plan with higher cost can be part of the optimal query evaluation plan. 23

24 Conclusion Volcano can generate top-down optimizers given a certain specifications model. Plan search is performed in logical, and physical spaces. A set of rules is used for generating equivalence expressions and for implementing logical expressions using algorithms. 24

25 References Graefe, G. and McKenna, W. J The Volcano Optimizer Generator: Extensibility and Efficient Search. In Proceedings of the Ninth international Conference on Data Engineering (April 19-23, 1993). Exploiting Upper and Lower Bounds in Top-Down Query Optimization. In Proceedings of the 2001 international Symposium on Database Engineering & Applications (July 16-18, 2001). B. Aditya Prakash and S. Sudarshan. Query Optimization (B.Tech Seminar), Department of Computer Science and Engineering, Indian Institute of Technology- Bombay. 25