Query Evaluation and Optimization

Transcription

1 Query Evaluation and Optimization Jan Chomicki University at Buffalo Jan Chomicki () Query Evaluation and Optimization 1 / 21

2 Evaluating σ E (R) Jan Chomicki () Query Evaluation and Optimization 2 / 21

3 Evaluating σ E (R) Pick the option of least estimated cost. Jan Chomicki () Query Evaluation and Optimization 2 / 21

4 Evaluating σ E (R) Pick the option of least estimated cost. Any E Linear (sequential) scan. Jan Chomicki () Query Evaluation and Optimization 2 / 21

5 Evaluating σ E (R) Pick the option of least estimated cost. Any E Linear (sequential) scan. E is A = c if file sorted on A, use binary search if file has an index on A, use the index if file hashed on A, use hashing Jan Chomicki () Query Evaluation and Optimization 2 / 21

6 Evaluating σ E (R) Pick the option of least estimated cost. Any E Linear (sequential) scan. E is A = c if file sorted on A, use binary search if file has an index on A, use the index if file hashed on A, use hashing E is Aθc for θ {<,, >, } if file sorted on A, then: use binary search if there is an index on A, use the index Jan Chomicki () Query Evaluation and Optimization 2 / 21

7 Complex selection conditions Jan Chomicki () Query Evaluation and Optimization 3 / 21

8 Complex selection conditions E is E 1 E 2 use E 1 or E 2 to narrow down the search use composite index (if available) use indexes for E 1 and E 2 to obtain sets of RowIds, then intersect those sets use bitmap indexes for E 1 and E 2 to obtain bitvectors, then use boolean AND use a combination of bitmap and other kinds of indexes Jan Chomicki () Query Evaluation and Optimization 3 / 21

9 Complex selection conditions E is E 1 E 2 use E 1 or E 2 to narrow down the search use composite index (if available) use indexes for E 1 and E 2 to obtain sets of RowIds, then intersect those sets use bitmap indexes for E 1 and E 2 to obtain bitvectors, then use boolean AND use a combination of bitmap and other kinds of indexes E is E 1 E 2 E 1 or E 2 cannot be used separately to narrow down the search use indexes for E 1 and E 2 to obtain sets of RowIds, then take the union of those sets use bitmap indexes for E 1 and E 2 to obtain bitvectors, then use boolean OR use a combination of bitmap and other kinds of indexes Jan Chomicki () Query Evaluation and Optimization 3 / 21

10 Evaluating R A=B S Jan Chomicki () Query Evaluation and Optimization 4 / 21

11 Evaluating R A=B S Nested loops foreach t R do foreach s S do if t[a] = s[b] then output (t, s) Jan Chomicki () Query Evaluation and Optimization 4 / 21

12 Evaluating R A=B S Nested loops foreach t R do foreach s S do if t[a] = s[b] then output (t, s) Index join (S has an index on B) foreach t R do lookup S using the index and the value t[a] Jan Chomicki () Query Evaluation and Optimization 4 / 21

13 Evaluating R A=B S Nested loops foreach t R do foreach s S do if t[a] = s[b] then output (t, s) Index join (S has an index on B) foreach t R do lookup S using the index and the value t[a] Sort-merge join (R sorted on A, S sorted on B) scan files in sorted order matching A-values in R with B-values in S Jan Chomicki () Query Evaluation and Optimization 4 / 21

14 Jan Chomicki () Query Evaluation and Optimization 5 / 21

15 Hash-join create new hashed files HR and HS; foreach t R do i := h(t[a]); store t in bucket r i of HR; foreach t S do i := h(t[a]); store t in bucket s i of HS; for each bucket r i of HR read r i and build an in-memory hash index for each tuple t in the bucket s i of S probe the hash index to find and output the tuples w such that w[a] = t[a] Jan Chomicki () Query Evaluation and Optimization 5 / 21

16 Hash-join create new hashed files HR and HS; foreach t R do i := h(t[a]); store t in bucket r i of HR; foreach t S do i := h(t[a]); store t in bucket s i of HS; for each bucket r i of HR read r i and build an in-memory hash index for each tuple t in the bucket s i of S probe the hash index to find and output the tuples w such that w[a] = t[a] the hash function used to build the hash index and to probe it has to be different than h each bucket of HR has to fit in main memory Jan Chomicki () Query Evaluation and Optimization 5 / 21

17 Oracle: selection and join Jan Chomicki () Query Evaluation and Optimization 6 / 21

18 Oracle: selection and join Equality condition A = c unique index on A: INDEX UNIQUE SCAN nonunique index on A: INDEX RANGE SCAN Jan Chomicki () Query Evaluation and Optimization 6 / 21

19 Oracle: selection and join Equality condition A = c unique index on A: INDEX UNIQUE SCAN nonunique index on A: INDEX RANGE SCAN Range condition on A any index on A: INDEX RANGE SCAN Jan Chomicki () Query Evaluation and Optimization 6 / 21

20 Oracle: selection and join Equality condition A = c unique index on A: INDEX UNIQUE SCAN nonunique index on A: INDEX RANGE SCAN Range condition on A any index on A: INDEX RANGE SCAN Conjunction both indexes: AND-EQUAL (intersection of RowId sets) Jan Chomicki () Query Evaluation and Optimization 6 / 21

21 Oracle: selection and join Equality condition A = c unique index on A: INDEX UNIQUE SCAN nonunique index on A: INDEX RANGE SCAN Range condition on A any index on A: INDEX RANGE SCAN Conjunction both indexes: AND-EQUAL (intersection of RowId sets) Join 1 SORT JOIN/MERGE JOIN (sort-merge) 2 NESTED LOOPS (nested loop or index join) 3 HASH JOIN Jan Chomicki () Query Evaluation and Optimization 6 / 21

22 Query optimization Jan Chomicki () Query Evaluation and Optimization 7 / 21

23 Query optimization Stages: 1 Parsing, validation, view expansion. 2 Logical plan selection (using algebraic laws). 3 Physical plan selection (cost-based) Jan Chomicki () Query Evaluation and Optimization 7 / 21

24 Algebraic laws Jan Chomicki () Query Evaluation and Optimization 8 / 21

25 Algebraic laws Assumption Tuples are mappings from attribute names to domain values. Jan Chomicki () Query Evaluation and Optimization 8 / 21

26 Algebraic laws Assumption Tuples are mappings from attribute names to domain values. Join (and Cartesian product) E 1 E 2 = E 2 E 1 (E 1 E 2 ) E 3 = E 1 (E 2 E 3 ). Jan Chomicki () Query Evaluation and Optimization 8 / 21

27 Algebraic laws Assumption Tuples are mappings from attribute names to domain values. Join (and Cartesian product) E 1 E 2 = E 2 E 1 (E 1 E 2 ) E 3 = E 1 (E 2 E 3 ). Cascading π A1,...,A n (π A1,...,A n,...,a n+k (E)) = π A1,...,A n (E) σ F1 (σ F2 (E)) = σ F1 F 2 (E) Jan Chomicki () Query Evaluation and Optimization 8 / 21

28 Commuting projections Jan Chomicki () Query Evaluation and Optimization 9 / 21

29 Commuting projections With selection π A1,...,A n (σ F (E)) = σ F (π A1,...,A n (E)) if F does not involve any attributes othet than A 1,..., A n. Jan Chomicki () Query Evaluation and Optimization 9 / 21

30 Commuting projections With selection π A1,...,A n (σ F (E)) = σ F (π A1,...,A n (E)) if F does not involve any attributes othet than A 1,..., A n. With Cartesian product π A1,...,A n,a n+1,...,a n+k (E 1 E 2 ) = π A1,...,A n (E 1 ) π An+1,...,A n+k (E 2 ) where A 1,..., A n come from E 1 and A n+1,..., A n+k come from E 2. Jan Chomicki () Query Evaluation and Optimization 9 / 21

31 Commuting projections With selection π A1,...,A n (σ F (E)) = σ F (π A1,...,A n (E)) if F does not involve any attributes othet than A 1,..., A n. With Cartesian product π A1,...,A n,a n+1,...,a n+k (E 1 E 2 ) = π A1,...,A n (E 1 ) π An+1,...,A n+k (E 2 ) where A 1,..., A n come from E 1 and A n+1,..., A n+k come from E 2. With union π A1,...,A n (E 1 E 2 ) = π A1,...,A n (E 1 ) π A1,...,A n (E 2 ). Jan Chomicki () Query Evaluation and Optimization 9 / 21

32 Commuting selections Jan Chomicki () Query Evaluation and Optimization 10 / 21

33 Commuting selections With Cartesian product σ F (E 1 E 2 ) = σ F (E 1 ) E 2 if F involves only the attributes of E 1. Jan Chomicki () Query Evaluation and Optimization 10 / 21

34 Commuting selections With Cartesian product σ F (E 1 E 2 ) = σ F (E 1 ) E 2 if F involves only the attributes of E 1. With union σ F (E 1 E 2 ) = σ F (E 1 ) σ F (E 2 ). Jan Chomicki () Query Evaluation and Optimization 10 / 21

35 Commuting selections With Cartesian product σ F (E 1 E 2 ) = σ F (E 1 ) E 2 if F involves only the attributes of E 1. With union σ F (E 1 E 2 ) = σ F (E 1 ) σ F (E 2 ). With set difference σ F (E 1 E 2 ) = σ F (E 1 ) σ F (E 2 ). Jan Chomicki () Query Evaluation and Optimization 10 / 21

36 Cost-based query optimization Logical query plan expression in relational algebra produced by rewrite rules that are obtained from the algebraic laws. Physical query plan operators: different implementations of relational algebra operators table-scan, index-scan, sort-scan, sort. Jan Chomicki () Query Evaluation and Optimization 11 / 21

37 Cost-based query optimization Logical query plan expression in relational algebra produced by rewrite rules that are obtained from the algebraic laws. Physical query plan operators: different implementations of relational algebra operators table-scan, index-scan, sort-scan, sort. A physical query plan of least estimated cost is produced from a logical query plan. Jan Chomicki () Query Evaluation and Optimization 11 / 21

38 Choices to be made Jan Chomicki () Query Evaluation and Optimization 12 / 21

39 Choices to be made 1 order and grouping of joins 2 choosing an algorithm for each operator occurrence 3 adding other operators like sorting 4 materialization vs. pipelining Jan Chomicki () Query Evaluation and Optimization 12 / 21

40 Choices to be made 1 order and grouping of joins 2 choosing an algorithm for each operator occurrence 3 adding other operators like sorting 4 materialization vs. pipelining Assumptions relation file cost size of intermediate relations Jan Chomicki () Query Evaluation and Optimization 12 / 21

41 Cost estimation using size estimates Jan Chomicki () Query Evaluation and Optimization 13 / 21

42 Cost estimation using size estimates Size estimates accurate easy to compute consistent Jan Chomicki () Query Evaluation and Optimization 13 / 21

43 Cost estimation using size estimates Size estimates accurate easy to compute consistent Notation B(R): number of blocks in relation R T (R): number of tuples in relation R V (R, A): number of distinct values R has in attribute A (can be generalized to sets of attributes) Jan Chomicki () Query Evaluation and Optimization 13 / 21

44 Relational algebra operators Jan Chomicki () Query Evaluation and Optimization 14 / 21

45 Relational algebra operators Projection T (π X (R)) = T (R) Jan Chomicki () Query Evaluation and Optimization 14 / 21

46 Relational algebra operators Projection T (π X (R)) = T (R) Selection T (σ A=c (R)) = T (R)/V (R, A) T (σ A<c (R) = T (R)/3 T (σ C1 C 2 (R) = T (σ C1 (σ C2 (R))) T (σ C1 C 2 (R)) = min(t (R), T (σ C1 (R)) + T (σ C2 (R))) Jan Chomicki () Query Evaluation and Optimization 14 / 21

47 Relational algebra operators Projection T (π X (R)) = T (R) Selection T (σ A=c (R)) = T (R)/V (R, A) T (σ A<c (R) = T (R)/3 T (σ C1 C 2 (R) = T (σ C1 (σ C2 (R))) T (σ C1 C 2 (R)) = min(t (R), T (σ C1 (R)) + T (σ C2 (R))) Those estimates may be inconsistent. Jan Chomicki () Query Evaluation and Optimization 14 / 21

48 Natural join R(X, Y ) S(Y, Z) Simplifying assumptions containment: V (R, Y ) V (S, Y ) implies π Y (R) π Y (S) preservation: V (R S, A) = V (R, A). Jan Chomicki () Query Evaluation and Optimization 15 / 21

49 Natural join R(X, Y ) S(Y, Z) Simplifying assumptions containment: V (R, Y ) V (S, Y ) implies π Y (R) π Y (S) preservation: V (R S, A) = V (R, A). Y consists of one attribute A T (R S) = T (R)T (S) max(v (R, A), V (S, A)). Jan Chomicki () Query Evaluation and Optimization 15 / 21

50 Natural join R(X, Y ) S(Y, Z) Simplifying assumptions containment: V (R, Y ) V (S, Y ) implies π Y (R) π Y (S) preservation: V (R S, A) = V (R, A). Y consists of one attribute A T (R S) = T (R)T (S) max(v (R, A), V (S, A)). Y consists of n attributes A 1,..., A n T (R S) = T (R)T (S) j=1,...,n max(v (R, A j), V (S, A j )). Jan Chomicki () Query Evaluation and Optimization 15 / 21

51 Multiway join S = R 1 R k Assumption A 1,... A n are the attributes that appear in more than one relation. Jan Chomicki () Query Evaluation and Optimization 16 / 21

52 Multiway join S = R 1 R k Assumption A 1,... A n are the attributes that appear in more than one relation. A-factor M(A) Product of V (R i, A) for every relation R i in which A appears, except for the smallest V (R i, A). Jan Chomicki () Query Evaluation and Optimization 16 / 21

53 Multiway join S = R 1 R k Assumption A 1,... A n are the attributes that appear in more than one relation. A-factor M(A) Product of V (R i, A) for every relation R i in which A appears, except for the smallest V (R i, A). Multiway join size T (R 1 R k ) = i=1,...,k T (R i) j=1,...,n M(A j). Jan Chomicki () Query Evaluation and Optimization 16 / 21

54 Jan Chomicki () Query Evaluation and Optimization 17 / 21

55 Properties consistency: join size estimate is independent of the order and the grouping of the joins accuracy: better estimates exist, for example based on histograms Jan Chomicki () Query Evaluation and Optimization 17 / 21

56 Properties consistency: join size estimate is independent of the order and the grouping of the joins accuracy: better estimates exist, for example based on histograms Statistics computed periodically, on-demand or incrementally do not have to be very accurate Jan Chomicki () Query Evaluation and Optimization 17 / 21

57 Computing the best plan for R 1 R k Jan Chomicki () Query Evaluation and Optimization 18 / 21

58 Computing the best plan for R 1 R k Constructing a table D(Size,LeastCost,BestPlan) Base cases: One relation R: D(T (R), 0, R) One join Ri R j : D(T (R i R j ), 0, R i R j ) if T (R i ) T (R j ) Inductive case R : 1 R = {R i1,..., R im } for m < k 2 for every j {i 1,..., i m}, retrieve the tuple D(T ( i j,r i R R i), C j, E j ) 3 find p {i 1,..., i m} such that C p + T (( i p,r i R R i)) is minimal 4 let Best = ( i p,r i R R i) 5 return D(T (Best R p), C p + T (Best), E p R p) Jan Chomicki () Query Evaluation and Optimization 18 / 21

59 Computing the best plan for R 1 R k Constructing a table D(Size,LeastCost,BestPlan) Base cases: One relation R: D(T (R), 0, R) One join Ri R j : D(T (R i R j ), 0, R i R j ) if T (R i ) T (R j ) Inductive case R : 1 R = {R i1,..., R im } for m < k 2 for every j {i 1,..., i m}, retrieve the tuple D(T ( i j,r i R R i), C j, E j ) 3 find p {i 1,..., i m} such that C p + T (( i p,r i R R i)) is minimal 4 let Best = ( i p,r i R R i) 5 return D(T (Best R p), C p + T (Best), E p R p) How many plans are considered? Jan Chomicki () Query Evaluation and Optimization 18 / 21

60 Refinements More sophisticated cost measure calculating the cost of using different algorithms for the same operation keeping multiple plans, together with their costs, to account for interesting join orders. Jan Chomicki () Query Evaluation and Optimization 19 / 21

61 Refinements More sophisticated cost measure calculating the cost of using different algorithms for the same operation keeping multiple plans, together with their costs, to account for interesting join orders. Greedy algorithm 1 start with a pair of relations whose estimated join size is minimal 2 keep adding further relations with minimal estimated join size Jan Chomicki () Query Evaluation and Optimization 19 / 21

62 Completing the evaluation plan Operations TableScan(Relation) SortScan(Relation, Attributes) IndexScan(Relation, Condition) Filter(Condition) Sort(Attributes) different algorithms for the basic operations Jan Chomicki () Query Evaluation and Optimization 20 / 21

63 Completing the evaluation plan Operations TableScan(Relation) SortScan(Relation, Attributes) IndexScan(Relation, Condition) Filter(Condition) Sort(Attributes) different algorithms for the basic operations Pipelining vs. materialization. Jan Chomicki () Query Evaluation and Optimization 20 / 21

64 Jan Chomicki () Query Evaluation and Optimization 21 / 21