Query Evaluation Overview. Query Optimization: Chap. 15. Evaluation Example. Cost Estimation. Query Blocks. Query Blocks

Transcription

1 Query Evaluation Overview Query Optimization: Chap. 15 CS634 Leture 12 SQL query first translated to relational algebra (RA) Atually, some additional operators needed for SQL Tree of RA operators, with hoie of algorithm among available implementations for eah operator Main issues in query optimization For a given query, what plans are onsidered? Algorithm to searh plan spae for heapest (estimated) plan How is the ost of a plan estimated? Objetive Ideally: Find best plan Pratially: Avoid worst plans! Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke Evaluation Example FROM Reserves R, Sailors S WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5 Annotated Tree sname bid=100 rating > 5 sid=sid (On-the-fly) (On-the-fly) (Simple Nested Loops) Cost Estimation For eah plan onsidered, must estimate: Cost of eah operator in plan tree Depends on input ardinalities Operation and aess type: sequential san, index san, joins Size of result for eah operation in tree Use information about the input relations For seletions and joins, assume independene of prediates Reserves Sailors Query Bloks SQL query parsed into a olletion of query bloks Bloks are optimized one at a time Nested bloks an be treated as alls to a subroutine One all made one per outer tuple In some ases ross-blok optimization is possible A good query optimizer an unnest queries Outer blok WHERE S.age IN (SELECT MAX (S2.age) 2) Nested blok Query Bloks WHERE S.age IN (SELECT MAX (S2.age) 2) In fat this is an unorrelated subquery: The inner blok an be evaluated one!

2 Query Bloks WHERE S.age IN (SELECT MAX (S2.age) 2 WHERE S2.rating = S.rating) Looking for sailors who are of max age in their own rating group. Correlated subquery: eah row in S needs its own exeution of the inner blok Blok Optimization Blok = Unit of optimization For eah blok, onsider: 1. All available aess methods, for eah relation in FROM lause 2. All left-deep join trees Left-deep defined pg. 415: right hild of eah join is a base table Start with all ways to join the relations one-at-a-time Consider all relation permutations and join methods Reall: Left table = outer table of a nested loop join Left table of NLJ an be pipelined: rows used one at a time in order But need to onsider other join methods too, giving up pipelining in many ases Expressions Query is simplified to a seletion-projetion-ross produt expression Aggregation and grouping an be done afterwards Optimization with respet to suh expressions Cross-produt inludes oneptually joins Will talk about equivalenes in a bit Statistis and Catalogs To hoose an effiient plan, we need information about the relations and indexes involved Catalogs ontain information suh as: Tuple ount (NTuples) and page ount (NPages) for eah relation Distint key value ount (NKeys) for eah index, INPages Index height, low/high key values (Low/High) for eah tree index Histograms of the values in some fields (optional) Catalogs updated periodially Approximate information used, slight inonsisteny is ok Databases provide tools for updating stats on demand Size Estimation and Redution Fators SELECT attribute list FROM relation list WHERE term 1 AND... AND term k Maximum number of tuples is ardinality of ross produt Redution fator (RF) assoiated with eah term reflets its impat in reduing result size Impliit assumption that terms are independent! ol = value has RF =1/NKeys(I), given index I on ol ol1 = ol2 has RF = 1/max(NKeys(I1), NKeys(I2)) ol > value has RF = (High(I)-value)/(High(I)-Low(I)) Example: rating > 6 has RF = 4/10 = 0.4 Histograms Most often, data values are not uniformly distributed within domain Skewed distributions result in inaurate ost estimations Histograms More aurate statistis Break up the domain into bukets Store the ount of reords that fall in eah buket Tradeoff Histograms are aurate, but take some spae The more fine-grained the partition, the better auray But more spae required

3 Histogram Classifiation Equiwidth Domain split into equal-length partitions Large differene between ounts in different bukets Dense areas not suffiiently haraterized Equidepth Histograms adapts to data distribution Fewer bukets in sparse areas, more bukets in dense areas Used by Orale (pg. 485) Why are they important? They allow us to: Convert ross-produts to joins Cross produts should always be avoided (when possible) Choose different join orders Reall that hoie of outer/inner influenes ost Push-down seletions and projetions ahead of joins When doing so dereases ost Seletions: 1... nr 1... nr Casade property: R R Casade Commute Allows us to hek multiple onditions in same pass Allows us to push down only partial onditions (when not possible/advantageous to push entire ondition) Projetions: a1r a1... anr Casade If every a i set is inluded in a i+1, Example: a1 = {a,b}, a2 = {a,b,} a2 (T) has (a, b, ) olumns a1 ( a2 (T)) has (a,b) olumns, same as a1 (T) Joins: Joins: (R S) (S R) Commute (R S) (S R) Commute R (S T) (R S) T Assoiative R (S T) (R S) T Assoiative Sketh of proof: Show for ross produt Add join onditions as seletion operators Use asading seletions in assoiative ase Commutative property: Allows us to hoose whih relation is inner/outer Assoiative property: Allows us to restrit plans to left-deep only, i.e., any query tree an be turned into a left-deep tree.

4 Commuting seletions with projetions Projetion an be done before seletion if all attributes in the ondition evaluation are retained by the projetion R) ( ) ( R a a Commute seletion with join Only if all attributes in ondition appear in one relation and not in the other: inludes only attributes from R R S R S Condition an be deomposed and pushed down before joins R S R S Here, 1 inludes only attributes from R and 2 only attributes from S Commute projetion with join Only if attributes in join ondition appear in the orresponding projetion lists (so they aren t projeted out ) R S R S) a a1 a 2( System R Optimizer Developed at IBM starting in the 1970 s Most widely used urrently; works well for up to 10 joins Cost estimation Statistis maintained in system atalogs Used to estimate ost of operations and result sizes Query Plan Spae Only the spae of left-deep plans is onsidered Left-deep plans allow output of eah operator to be pipelined into the next operator without storing it in a temporary relation Cartesian produts avoided System R Optimizer Developed at IBM starting in the 1970 s Most widely used urrently; works well for up to 10 joins Cost estimation Statistis maintained in system atalogs Used to estimate ost of operations and result sizes Query Plan Spae Only the spae of left-deep plans is onsidered Left-deep plans allow output of eah operator to be pipelined into the next operator without storing it in a temporary relation Cartesian produts avoided SQL Query Semantis (pg. 136, 156) 1. ompute the ross produt of tables in FROM 2. delete rows that fail the WHERE lause 3. projet out olumns not mentioned in selet list or group by or having lauses 4. group rows by GROUP BY 5. apply HAVING to the groups, dropping some out 6. if neessary, apply DISTINCT 7. if neessary, apply ORDER BY Note this all follows the order of the SELECT lauses, exept for projetion and DISTINCT, so it s not hard to remember.

5 Single-Relation Plans Single-Relation Plans FROM lause ontains single relation Query is ombination of seletion, projetion, and aggregates (possibly GROUP BY and HAVING, but these ome late in the logial progression, so usually less ruial to planning) Main issue is to selet best from all available aess paths (either file san or index) Aess path involves the table and the WHERE lause Another fator is whether the output must be sorted E.g., GROUP BY requires sorting (or hashing) Sorting may be done as separate step, or using an index if an indexed aess path is available Plans Without Indexes Only aess path is file san Apply seletion and projetion to eah retrieved tuple Projetion may or may not use dupliate elimination, depending on whether there is a DISTINCT keyword present GROUP BY: Write out intermediate relation after seletion/projetion (or pipeline into sort) Sort intermediate relation to reate groups Apply aggregates on-the-fly per eah group HAVING also performed on-the-fly, no additional I/O needed Plans With Indexes There are four ases: 1. Single-index aess path Eah index offers an alternative aess path Choose one with lowest I/O ost Non-primary onjunts, projetion, aggregates/grouping applied next 2. Multiple-index aess path Eah index used to retrieve set of rids Rid sets interseted, result sorted by page id (Alternatively, join indexes as tables) Retrieve eah page only one Non-primary onjunts, projetion, aggregates/grouping applied next Plans With Indexes (ontd.) 3. Tree-index aess path: extra possible use If GROUP BY attributes prefix of tree index, retrieve tuples in order required by GROUP BY Apply seletion, projetion for eah retrieved tuple, then aggregate Works well for lustered indexes Example: With tree index on rating SELECT ount(*), max(age) GROUP BY rating Plans With Indexes (ontd.) 3. Index-only aess path If all attributes in query inluded in index, then there is no need to aess data reords: index-only san If index mathes seletion, even better: only part of index examined Does not matter if index is lustered or not! If GROUP BY attributes prefix of a tree index, no need to sort! Example: With tree index on rating SELECT max(rating),ount(*) Note ount(*) doesn t require aess to row, just RID.

6 Example Shema Sailors (sid: integer, sname: string, rating: integer, age: real) Reserves (sid: integer, bid: integer, day: dates, rname: string) Similar to old shema; rname added Reserves: 40 bytes long tuple, 100K reords, 100 tuples per page, 1000 pages Sailors: 50 bytes long tuple, 40K tuples, 80 tuples per page, 500 pages Assume index entry size 10% of data reord size Cost Estimates for Single-Relation Plans Sequential san of file: NPages(R) Index I on primary key mathes seletion Cost is Height(I)+1 for a B+ tree, about 1.2 for hash index Clustered index I mathing one or more onjunts: NPages(CI) * produt of RF s of mathing onjunts Quik estimate: Npages(CI) = 1.1*NPages(TableData) i.e. 10% more for needed keys Non-lustered index I mathing one or more onjunts: (NPages(I)+NTuples(R)) * produt of RF s of mathing onjunts Quik estimate: Npages(I) =.1*Npages(R) (10% of data size) Note: these formulas are not in the text, but are onsistent with the disussions and examples there. Example SELECT S.sid WHERE S.rating=8 File san: retrieve all 500 pages Clustered Index I on rating (1/NKeys(I)) * (NPages(CI)) = (1/10) * (50+500) pages Note: One rating value owns a fration (1/10) of the index at all levels, so #pages aessed = (1/10) Npages(CI) Or (1/10) Npages(I) Multiple-Relation Plans Unlustered Index I on rating (1/NKeys(I)) * (NPages(I)+NTuples(S)) = (1/10) * ( ) pages Queries Over Multiple Relations In System R only left-deep join trees are onsidered In order to restrit the searh spae Left-deep trees allow us to generate all fully pipelined plans Intermediate results not written to temporary files. Not all left-deep trees are fully pipelined (e.g., sort-merge join) A B C D A B Left-deep C D C A B D Enumeration of Left-Deep Plans Among all left-deep plans, we need to determine: the order of joining relations the aess method for eah relation the join method for eah join Enumeration done in N passes (if N relations are joined): Pass 1: Find best 1-relation plan for eah relation Pass 2: Find best way to join result of eah 1-relation plan (as outer) to another relation - result is the set of all 2-relation plans Pass N: Find best way to join result of a (N-1)-relation plan (as outer) to the N th relation - result is the set of all N-relation plans Speed-up omputation using dynami programming (remember details of good plans to avoid real)

7 Enumeration of Left-Deep Plans (ontd.) For eah subset of relations, retain only: Cheapest plan overall, plus Cheapest plan for eah interesting order of the tuples Interesting order: order that allows exeution of GROUP BY without requiring an additional step of sorting, aggregates Avoid Cartesian produts if possible An N-1 way plan is not ombined with an additional relation unless there is a join ondition between them Exeption is ase when all prediates in WHERE have been used up (i.e., query itself requires a ross-produt) Ex: selet from T1, T2, T3 where T1.x = T2.x Only one join ondition, 3 tables, so end up with ross produt Cost Estimation for Multi-Relation Plans Two omponents: 1. Size of intermediate relations Maximum tuple ount is the produt of the ardinalities of relations in the FROM lause Redution fator (RF) assoiated with eah ondition term Result ardinality estimate = Max # tuples * produt of all RF s Example query on next slide: Result ardinality estimate = (40K*100K) * ((1/100)*(5/10)*(1/40K)) = 500 This means we estimate the query returns 500 rows as its result It is not a ost alulation Here 1/40K = RF of join ondition, 1/100 assumes 100 boats. 2. Cost of eah join operator Depends on join method Example, Reserves R WHERE S.sid=R.sid AND S.rating>5 AND R.bid=100 sname bid=100 rating > 5 sname sid=sid Example Pass 1 Single-relation plans Sailors B+ tree mathes rating>5 Most seletive aess path But index unlustered! Sometimes may prefer san Reserves B+ tree on bid mathes seletion bid=100 Cheapest plan for this table Sailors: Unlustered B+ tree on rating Unlustered Hash on sid Reserves: Unlustered B+ tree on bid Reserves sid=sid Sailors bid=100 rating > 5 Reserves Sailors Note: Here we are evaluating plans as andidates for the leftmost spot in the final plan Result of Pass 1: One plan for eah table. But this is not left-deep! Example Pass 2 Consider eah plan retained from Pass 1 as the outer, and how to join it with the (only) other relation Sailors outer, Reserves inner No index mathes join ondition, this ould be done as blok nested loop Reserves outer, Sailors inner Sine we have hash index on sid for Sailors, this ould be a heap plan using an indexed nested loop This would mean S.rating>5 is done after join. Also see disussion of this on pg. 412, point 3 End up with left-deep plan. Example, ont. (pipelining not in book) Also need to hek sort-merge join But that requires materialization of input tables, an extra expense (or use pipelining into sort) Need to ost all three ompeting plans, hoose least expensive Note that left-deep plans assume nested-loop joins are in use, so may miss good hash join plans Note on pg. 500: Orale onsiders non-left-deep plans to better utilize hash joins.

8 Nested Queries Nested blok is optimized independently, with the outer tuple onsidered as providing a seletion ondition Outer blok is optimized with the ost of alling nested blok omputation taken into aount Impliit ordering of these bloks means that some good strategies are not onsidered The non-nested version of the query is typially optimized better Nested Queries WHERE EXISTS (SELECT * FROM Reserves R WHERE R.bid=103 AND R.sid=S.sid) Equivalent non-nested query:, Reserves R WHERE S.sid=R.sid AND R.bid=103