DATABASE DESIGN II - 1DL400

Transcription

1 DTSE DESIGN II - 1DL400 Fall 2016 course on modern database systems Kjell Orsborn Uppsala Database Laboratory Department of Information Technology, Uppsala University, Uppsala, Sweden 06/12/16 1

2 Query processing Elmasri/Navathe ch?? Padron-McCarthy/Risch ch?? Kjell Orsborn Uppsala Database Laboratory Department of Information Technology, Uppsala University, Uppsala, Sweden 06/12/16 2

3 What is query processing? given SQL query is translated by the query processor into a low level program called an execution plan n execution plan is a program in a functional language: The language physical relational algebra, specialized for internal storage representation in the DMS. The physical relational algebra extends the relational algebra with: Primitives to search through the internal storage structures of the DMS 06/12/16 3

4 What is query optimization? SQL is a very high level language: The users specify what to search for not how the search is actually done The algorithms are chosen automatically by the DMS For a given SQL query there may be very many possible execution plans The cost of these execution plans very widely The costs vary with orders of magnitude The task of the query optimizer is to choose the cheapest plan out of the possible ones The query optimizer is the most complex (and important) part of the query processor 06/12/16 4

5 Complexity of query optimization Very many possible execution plans for a given SQL query, e.g.: J joins can be permutated with different costs O(J!) In addition there are different join algorithms to choose from Query optimization combinatorial over # of operations Q in query. With dynamic programming O( Q 2 ) in best case 06/12/16 5

6 Why does query optimization pay off? Query optimization radically improves performance of executing query The complexity of a good plan may be O(log N), while a bad one is O(N 2 ), where N is size of the database Query optimization enables scalability of declarative queries Since N is large the payoff is huge The size of the results of the query Q << N Classical query optimization can handle up to ca 12 joins Good query optimizer critical for competitive DMS! Query optimization is the key to the success of SQL 06/12/16 6

7 Query Processing Steps SQL Query PRSER (parsing and semantic checking as in any compiler) Parse tree (~ tree structure representing relational calculus expression) OPTIMIZER (very advanced) Execution plan (physical relation algebra expression) EXECUTOR (execution plan interpreter) DMS kernel Data structures 06/12/16 7

8 Query Optimizer Steps Tuple calculus View expansion Tuple calculus Query transformations Tuple calculus Cost-based query optimization Physical relational algebra 06/12/16 8

9 View expansion view is a virtual table expressed through a single SQL statement, a named query Views are textually substituted (macro expanded) by view expansion View expansion makes query larger, and thus optimization becomes even slower. View expansion allows optimizer to look inside view definitions rather than regarding views as black boxes View expansion required in order to detect indexes hidden inside views 06/12/16 9

10 View expansion CRETE TLE SUPPLIES( STORE CHR(10), ITEM CHR(10), PRICE DECIML(10,2), PRIMRY KEY(STORE, ITEM)) View defined over supplies: CRETE VIEW ICSUPPLIES S SELECT * FROM SUPPLIES WHERE STORE = IC 06/12/16 10

11 View expansion SELECT PRICE FROM ICSUPPLIES S WHERE S.ITEM = Tomatoes Translated by the view expander into: SELECT PRICE FROM SUPPLIES S WHERE S.ITEM = Tomatoes ND S.STORE = IC Query optimizer will now discover index(es) on ITEM or STORE! These indexes would NOT have been discovered if view was black box. 06/12/16 11

12 lank 06/12/16 12

13 Cost-based query optimization Elmasri/Navathe ch?? Padron-McCarthy/Risch ch?? Kjell Orsborn Uppsala Database Laboratory Department of Information Technology, Uppsala University, Uppsala, Sweden 06/12/16 13

14 Cost-based query optimization Principle: 1. Generate all possible execution plans (heuristics to avoid some unlikely ones) 2. Estimate the cost of executing each of the generated plans using a cost model based on database statistics and properties of DMS algorithms 3. Choose the cheapest one Optimization criteria # of disk blocks read (dominates) only metric for classical disk DMS CPU usage, CP important when operators are computing much Communication costs for distributed data, CO important in distributed databases. Normally weighted average of different criteria: W 1 * D + W 2 * CP + W 3 * CO, where W 1, W 2, W 3 system configured weights The costs are computed based on data statistics and cost model for operators in physical relational algebra 06/12/16 14

15 Query execution plan (physical algebra) Query execution plan is functional program with evaluation primitives such as: Segment scan operator Primary index scan operators per index kind Secondary index scan operators per index kind One or several join algorithms Sort operator Duplicate elimination operator Stop after n tuples operator, stop_after(n,s) that emits the n first tuples in scan s Other DMS-specific operators Normally pipelined execution Streams of tuples produced as intermediate results 06/12/16 15

16 Degrees of freedom for optimizer Query plan must be correct and sufficiently optimized Optimizer chooses among physical operators over streams e.g.: rute force linear table scan (most general access path) Index scan of selected part of index structure (e.g. -tree, hash table) Reverse scans traversing indexes and tables backwards Materialize stream on disk or in memory if favorable Eliminate duplicates in stream Stop scanning stream s after n tuples read, stopafter(n,s) 06/12/16 16

17 Degrees of freedom for optimizer Optimizer chooses among join strategies, e.g.: Choose order of joining tables and intermediate result streams Choice of join algorithms, e.g. Nested Loop Join, Merge Join, Hash Join Sort intermediate results if necessary lgorithm choices may depend on available amount of main memory 06/12/16 17

18 Nested-loop join Nested-loop join traverses one operand stream and for each row seeks a matching row in the second stream Cost: C NLJ = b T + (b T * b U ) + b res 06/12/16 18

19 Sort-merge join Sort-merge join traverses the two operand streams simultaneously while matching a row from the first stream with another row in the second stream. (both T and U are assumed to be sorted on the join attributes.) Cost: C SMJ = b T + b U + b res 06/12/16 19

20 Hash join Hash join the operand stream T are read into the buffer and a hash table is created on its hash field (join attribute), then for each row i U any matching row in T is retrieved by applying the same hash function on T:s hash table. (assumes the complete hash table fits in memory - otherwise partitioned hash join) Cost: C HJ = 3 * (b T + b U ) + b res or Optimal cost: (b T + b U ) + b res 06/12/16 20

21 Query Cost Model Purpose: Estimate the cost of executing a given execution plan generated by the query optimizer asic cost parameters in cost models Cost of randomly accessing disk block i.e. time for disk arm positioning Data transfer rate i.e. time to move disk arm continously forwards or backwards while transferring data to main-memory Clustering order of data tuples on disk Non-clustered index have slow index scans Sort order of data tuples on disk No sorting necessary if query specifies order by and data ordered by index Cost models for different index access methods -tree structures - LH Cost models for different join methods Cost of sorting intermediate results 06/12/16 21

22 Total Query Cost The total cost estimate of an execution plan is a formula defined in terms of the basic cost parameters. The total cost depends on how often primitive operations are invoked if they are expensive. In classical DMSs only joins and scans were expensive while selections (<,>,=,!=) were cheap Now functions doing computations may have high costs e.g. statistics or user defined computations may be expensive The execution cost of an operator depends on data volume of intermediate results. The intermediate result sizes are estimated by statistical models over data stored in database. Central factor in these statistical models: Selectivity: The percentage tuples matched by a predicate 06/12/16 22

23 Selectivity Selectivity: Percentage tuples fulfilling a predicate Selectivity used for estimating size of (intermediate) query results. Example query joining of relations T1(SSN,NME) and T2(SSN,INCOME): SELECT t2.income FROM T1 t1, T2 t2 WHERE t1.nme = Kalle ND t2.income > ND t1.ssn = t2.ssn ssume indexes on T1.SSN, T2.SSN, T1.NME, and T2.INCOME. ssume we are using nested-loop-join (NLJ) as only join algorithm 06/12/16 23

24 Evaluated plans Plan : project(t2.income, select(nlj SSN (index_scan(t2.income>95000), T1), T1.NME= Kalle )) Plan : project(t2.income,select(nlj SSN (index_scan(t1.nme= Kalle ),T2), T2.INCOME>95000)) π T2.INCOME π T2.INCOME σ T1.NME= Kalle σ T2.INCOME>95000 NLJ NLJ T1.SSN=T2.SSN T2.SSN=T1.SSN IS T2.INCOME>95000 T1 IS T1.NME= Kalle T2 If T2.INCOME > selects fewer tuples than T1.NME = Kalle then plan seems best, otherwise plan seems best 06/12/16 24

25 Selectivity calculation selectivitity(p(t)) is defined as percentage of tuples t that are selected by predicate P(t) For example: selectivity(t2.income>95000) may be The selectivity depends on value distributions in selected table column For example, the distribution of values in T2.INCOME The DMS maintains statistics, e.g. Number of rows of each table T, cardinality(t) Number of different values of each table column T.C, distinct_count(t.c). max(t.c) and min(t.c) for each table column T.C 06/12/16 25

26 Estimating selectivity ssume: max(t2.income) =100000, min(t2.income) = distinct_count(t1.nme) = 1000 ssume uniform distributions of values in all columns Modern DMSs maintain histograms for better selectivity estimates Selectivity of T2.INCOME>95000: selectivity(t2.income > 95000) = (max(t2.income) ) / (max(t2.income ) - min(t2.income)) = ( ) / ( ) = Selectivity of T1.name= Kalle : selectivity(t1.nme = Kalle ) = 1/distinct_count(T1.NME) = 1/1000 = Notice that selectivity is independent of tables size 06/12/16 26

27 Computing intermediate result sizes Computing the size of the result of a physical operator requires the size of the table T, called table cardinality, denoted cardinality(t). For example: assume cardinality(t1) = and cardinality(t2) = 8000 Number of tuples matching predicate P, matches(p), is computed by multiplying selectivity(p) with size of table. For example matches(t1.nme = Kalle ) = selectivity(t1.nme = Kalle ) * cardinality(t1) = 0.001*10000 = 10 rows matches(t2.income > 95000) = selectivity(t2.income > 95000) * cardinality(t2) = * 8000 = 500 rows Optimal plan is Since: matches(t1.nme= Kalle ) < matches(t2.income>95000) 06/12/16 27

28 Execution plans with data flows 0.5 π T2.INCOME σ T1.NME= Kalle 0.5 π T2.INCOME 500 NLJ T1.SSN=T2.SSN IS T2.INCOME>95000 T1 σ T2.INCOME> NLJ T2.SSN=T1.SSN 10 8 (since card(t2)=0.8*card(t1)) IS T1.NME= Kalle T2 The final cost furthermore takes into account the costs of the operators (join, Index Scan) This costs depends on factors like join algorithm, sort order of stream, clustering of rows on disk, etc. (no more details here) 06/12/16 28

29 Data statistics DMS maintains statistics about data in tables in order to estimate size of intermediate results, for example: Rows in tables, cardinality(t) Row length in a table Number of different values in column, count_distinct(t.c) Histograms of distributions of column values DMS contains statistical models for estimating selectivities of composite predicates. Examples of statistical models assuming independent values: selectivity(p1 ND P2) = selectivity(p1) * selectivity(p2) selectivity(p1 OR P2) = selectivity(p1) + selectivity(p2) selectivity(p1) * selectivity(p2) selectivity(not P) = 1 selectivity(p) 06/12/16 29

30 Data statistics The models are often very rough For example, assuming statistical independence and flat distributions Work rather well since models used only for comparing different execution strategies - not for getting the exact execution costs. Modern DMS have more advanced models handling statistical dependencies and different distributions as well Cost of maintaining data statistics May be cheap: e.g size of relation, depth of -tree. More expensive: e.g. distribution on non-indexed columns, histograms Need to scan tables once in a while (single pass) The statistics can be refreshed on demand by D Statistics not always up-to-date Wrong statistics => sub-optimal but still correct plans 06/12/16 30

31 lank 06/12/16 31

32 Query optimization by Dynamic programming Elmasri/Navathe ch?? Padron-McCarthy/Risch ch?? Kjell Orsborn Uppsala Database Laboratory Department of Information Technology, Uppsala University, Uppsala, Sweden 06/12/16 32

33 Optimization algorithm Variants of dynamic programing (Dijsktra, ) normally used. Find optimal join order of: C Cost: 0 Rem:,,C 06/12/16 33

34 Optimization algorithm Variants of dynamic programing (Dijsktra, ) normally used. Find optimal join order of: C Cost: 0 Rem:,,C Cost: 1 Rem:,C Cost: 10 Rem:,C C Cost: 100 Rem:, 06/12/16 34

35 Optimization algorithm Variants of dynamic programing (Dijsktra, ) normally used. Find optimal join order of: C Cost: 0 Rem:,,C Cost: 1 Rem:,C Cost: 10 Rem:,C C Cost: 100 Rem:, Cost: 19 Rem: C Cost: 15 Rem: C 06/12/16 35

36 Optimization algorithm Variants of dynamic programing (Dijsktra, ) normally used. Find optimal join order of: C Cost: 0 Rem:,,C Cost: 1 Rem:,C Cost: 10 Rem:,C C Cost: 100 Rem:, Cost: 19 Rem: C Cost: 15 Rem: C Cost: 14 Rem: C Cost: 30 Rem: C 06/12/16 36

37 Optimization algorithm Variants of dynamic programing (Dijsktra, ) normally used. Find optimal join order of: C Cost: 0 Rem:,,C Cost: 1 Rem:,C Cost: 10 Rem:,C C Cost: 100 Rem:, Cost: 19 Rem: C Cost: 15 Rem: C Cost: 14 Rem: C Cost: 30 Rem: C C Cost: 20 Rem: 06/12/16 37

38 Optimization algorithm Variants of dynamic programing (Dijsktra, ) normally used. Find optimal join order of: C Cost: 0 Rem:,,C Cost: 1 Rem:,C Cost: 10 Rem:,C C Cost: 100 Rem:, Cost: 19 Rem: C Cost: 15 Rem: C Cost: 14 Rem: C Cost: 30 Rem: C C Cost: 35 Rem: C Cost: 20 Rem: 06/12/16 38

39 Optimization algorithm Variants of dynamic programing (Dijsktra, ) normally used. Find optimal join order of: C Cost: 0 Rem:,,C Cost: 1 Rem:,C Cost: 10 Rem:,C C Cost: 100 Rem:, Cost: 19 Rem: C Cost: 15 Rem: C Cost: 14 Rem: C Cost: 30 Rem: C C Cost: 21 Rem: C Cost: 35 Rem: C Cost: 20 Rem: 06/12/16 39

40 Optimization algorithm ll costs are total costs to execute each partial join The cost of a new node is computed by using the cost model to compute the cost of the new partial join, given the previously computed cost of the node above In practice several parameters are computed in each node, i.e. CPU cost, disk access cost, communication cost. Cost: 0 Rem:,,C Cost: 1 Rem:,C Cost: 10 Rem:,C C Cost: 100 Rem:, Cost: 19 Rem: C Cost: 15 Rem: C Cost: 14 Rem: C Cost: 30 Rem: C C Cost: 21 Rem: C Cost: 35 Rem: C Cost: 20 Rem: 06/12/16 40

41 Optimizing large queries Don t optimize at all or partly, i.e. order of predicates significant (old Oracle, old MySQL) Optimize partly, i.e. up to ca 10 joins, leave rest (rem field) unoptimized (new Oracle) Heuristic methods (e.g. greedy optimization, hill climbing) Randomized (Monte Carlo) methods To speed up data access the user may sometimes manually break down very large queries into smaller optimizable queries This is often necessary for translating relational representations to complex object structures in application programs 06/12/16 41