Principles of Data Management. Lecture #13 (Query Optimization II)

Transcription

1 Principles of Data Management Lecture #13 (Query Optimization II) Instructor: Mike Carey Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Today s Notable News v Reminder: No lecture this coming Tuesday! I will be in Washington, DC (weather permitting). Stay home & read: Shapiro; Selinger et al; etc.! v Midterm exam update I am determined to grade them unaided My February deadline/schedule is really bad This will become my #1 priority on March 1 st You will get them back that week sometime Sorry for the horribly long delay! v Now for Part II of query optimization Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 2

2 Intermediate Result Size Estimation v Optimizers use statistics in catalog to estimate the cardinalities of operators inputs and outputs Simplifying assumptions: uniform distribution of attribute values, independence of attributes, etc. v For each relation R: Cardinality of R ( R ), avg R-tuple width, and # of pages in R ( R ) pick any 2 to know all 3 (given the page size) v For each (indexed) attribute of R: Number of distinct values π A (R) Range of values (i.e., low and high values) Number of index leaf pages Number of index levels (if B+ tree) Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 3 Simple Selection Queries (σ p ) v Equality predicate (p is A = val ) Q R / π A (R) Translation: R s cardinality divided by the number of A values Assumes all values equally likely in R.A (uniform distribution) Ex: SELECT Emp WHERE age = 23; RF (a.k.a. selectivity) is therefore 1 / π A (R) v Range predicate (p is val 1 A val 2 ) Q R * ((val2 val1) / ((high(r.a) low(r.a)) Translation: Selected range size divided by full range size Again assumes uniform value distribution for R.A (Simply replace val i with high/low bound for a one-sided range predicate) Ex: SELECT Emp WHERE age 21; RF is ((val 2 val 1 ) / ((high(r.a) low(r.a)) Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 4

3 Boolean Selection Predicates v Conjunctive predicate (p is p1 and p2 ) RF p RF p1 * RF p2 Ex 1: SELECT Emp WHERE age = 21 AND gender = m Assumes independence of the two predicates p1 and p2 (uncorrelated) Ex 2: SELECT Student WHERE major = EE AND gender = m v Negative predicate (p is not p 1 ) RF p 1 RF p1 Translation: All tuples minus the fraction that satisfy p 1 Ex: SELECT Emp WHERE age 21 v Disjunctive predicate (p is p1 or p2 ) RF p RF p1 + RF p2 (RF p1 * RF p2 ) Translation: All (unique) tuples satisfying either predicate Ex: SELECT Student WHERE major = EE OR gender = m Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 5 Two-way Equijoin Predicates v Query Q: R join S on R.A = S.B v Assume join value set containment (FK/PK case) π A (R) is a subset of π B (S) or vice versa Translation: Smaller relation s join value set is subset of larger relation s join value set (where size is based on # unique values) Ex: SELECT Student S, Dept D WHERE S.major = D.deptname Q ( R * S ) / max( π A (R), π B (S) ) Ex: 100 D.deptname values but 85 S.major values used by 10,000 students Size of result is 1/100 of the cartesian product of Student and Dept Again making a uniformity assumption (i.e., about students majors) I.e., RF is 1 / max( π A (R), π B (S) ) v Repeat same principle to deal with N-way joins Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 6

4 Improved Estimation: Histograms v We have been making simplistic assumptions Specifically: uniform distribution of values This is definitely violated (all the time J ) in reality Violations can lead to huge estimation errors Huge estimation errors can lead to Very Bad Choices v By the way: In the absence of info, System R assumed 1/3 and 1/10 for range queries and exact match queries, respectively (Q: What might lead to the absence of info?) v How can we do better in the OTHER direction: Keep track of most and/or least frequent values Use histograms to better approximate value distributions Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 7 Equi-width Histograms v Divide the domain into B buckets of equal width E.g., partition Kid.age values into buckets v Store the bucket boundaries and the sum of frequencies of the values with each bucket Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 8

5 Histogram Construction v Initial Construction: Make one full one pass over R to construct an accurate equi-width histogram Keep a running count for each bucket If scanning is not acceptable, use sampling Construct a histogram on R sample, and scale the frequencies by R / R sample v Maintenance Options: Incremental: for each update or R, increment/decrement the corresponding bucket frequencies (Q: Cost?) Periodic recomputation: distribution changes slowly! Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 9 Using Equi-width Histograms v Q: σ Α=5 (R) 5 is in bucket [5,8] (with 19 tuples) Assume uniform distribution within the bucket Thus Q 19/4 5. Actual value is 1 v Q: σ Α>=7 & Α <= 16 (R) [7,16] covers [9,12] (27 tuples) and [13,16] (13 tuples) [7,16] partially covers [5,8] (19 tuples) Thus Q 19/ Actual value = 52. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 10

6 Equi-height Histogram v Divide the domain into B buckets with (roughly) the same number of tuples in each bucket v Store this number and the bucket boundaries v Intuition: high frequencies are more important than low frequencies Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 11 Histogram Construction v Construction: Sort all R.A values, and then take equally spaced slices Ex: Sampling also applicable here v Maintenance: Incremental maintenance Split/merge buckets (B+ tree like) Periodic recomputation Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 12

7 Using an equi-height histogram v Q: σ Α=5 (R) 5 is in bucket [1,7] (with 16 tuples) Assume uniform distribution within the bucket Thus Q 16/7 2. (actual value = 1) v Q: σ Α>=7 & Α <= 16 (R) [7,16] covers [8,9], [10,11],[12,16] (all with tuples) [7,16] partially covers [1,7] (16 tuples) Thus Q 16/ Actually Q = 52. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 13 Can Combine Approaches v If values are badly skewed Keep high/low frequency value information in addition to histograms Could even apply the idea recursively: keep this sort of information for each bucket Divide by converting values to some # ranges Conquer by keeping some statistics for each range v Some statistical glitches to be aware of Parameterized queries Runtime situations Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 14

8 Nested Queries v Nested block is optimized independently, with the outer tuple considered as providing a selection condition. v Outer block is optimized with the cost of `calling nested block computation taken into account. v Implicit ordering of these blocks means that some good strategies are not considered. The nonnested version of the query is typically optimized better! SELECT S.sname FROM Sailors S WHERE EXISTS (SELECT * FROM Reserves R WHERE R.bid=103 AND R.sid=S.sid) Nested block to optimize: SELECT * FROM Reserves R WHERE R.bid=103 AND S.sid= outer value Equivalent non-nested query: SELECT S.sname FROM Sailors S, Reserves R WHERE S.sid=R.sid AND R.bid=103 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 15 Summary v Query optimization is an important task in a relational DBMS. v Must understand optimization to understand the performance impact of a given database design (relations, indexes) on a workload (set of queries). v Two parts to optimizing a query: Consider a set of alternative plans. Must prune search space; typically, left-deep plans only. Must estimate cost of each plan that is considered. Must estimate size of result and cost for each plan node. Key issues: Statistics, indexes, operator implementations. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 16

9 Summary (Contd.) v Single-relation queries: All access paths considered, cheapest is chosen. Issues: Selections that match index, whether index key has all needed fields and/or provides tuples in a desired order. v Multiple-relation queries: All single-relation plans are first enumerated. Selections/projections considered as early as possible. Next, for each 1-relation plan, all ways of joining another relation (as inner) are considered. Next, for each 2-relation plan that is `retained, all ways of joining another relation (as inner) are considered, etc. At each level, for each subset of relations, only best plan for each interesting order of tuples is `retained. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 17