Elena Baralis, Silvia Chiusano Politecnico di Torino. Pag. 1. Query optimization. DBMS Architecture. Query optimizer. Query optimizer.

Transcription

1 DBMS Architecture SQL INSTRUCTION OPTIMIZER Dtbse Mngement Systems MANAGEMENT OF ACCESS METHODS BUFFER MANAGER CONCURRENCY CONTROL RELIABILITY MANAGEMENT Index Files Dt Files System Ctlog DATABASE 2 Query optimizer Query optimizer It selects n efficient strtegy for query execution It is fundmentl building block of reltionl DBMS It gurntees the dt independence property The form in which the SQL query is written does not ffect the wy in which it is implemented A physicl reorgniztion of dt does not require rewriting SQL queries It utomticlly genertes query execution pln It ws formerly hrd-coded by progrmmer The utomticlly generted execution pln is usully more efficient It evlutes mny different lterntives It exploits sttistics on dt, stored in the system ctlog, to mke decisions It exploits the best known strtegies It dynmiclly dpts to chnges in the dt distribution 3 4 Query optimizer Lexicl, syntctic nd semntic nlysis SQL QUERY LEXICAL, SYNTACTIC AND SEMANTIC ANALYSIS Anlysis of sttement to detect Lexicl errors e.g., misspelled keywords Syntctic errors errors in the grmmr of the SQL lnguge Semntic errors references to objects which do not ctully exist in the dtbse (e.g, ttributes or tbles) informtion in the dt dictionry is needed 5 6 Pg.

2 Lexicl, syntctic nd semntic nlysis Query optimizer Output Internl representtion in (extended) reltionl lgebr Why reltionl lgebr? It explicitly represents the order in which opertors re pplied It is procedurl (different from SQL) There is corpus of theorems nd properties exploited to modify the initil query tree SQL QUERY LEXICAL, SYNTACTIC AND SEMANTIC ANALYSIS INTERNAL REPRESENTATION BASED ON RELATIONAL ALGEBRA ALGEBRAIC DATA DICTIONARY 7 8 Algebric optimiztion Query optimizer Execution of lgebric trnsformtions considered to be lwys beneficil Exmple: nticiption of selection with respect to join Should eliminte the difference mong different formultions of the sme query This step is usully independent of the dt distribution Output Query tree in cnonicl form SQL QUERY LEXICAL, SYNTACTIC AND SEMANTIC ANALYSIS INTERNAL REPRESENTATION BASED ON RELATIONAL ALGEBRA ALGEBRAIC CANONICAL QUERY TREE COST BASED DATA DICTIONARY 9 Cost bsed optimiztion Selection of the best execution pln by evluting execution cost Selection of the best ccess method for ech tble the best lgorithm for ech reltionl opertor mong vilble lterntives Bsed on cost model for ccess methods nd lgorithms Genertion of the code implementing the best strtegy Cost bsed optimiztion Output Access progrm in executble formt It exploits the internl structures of the DBMS Set of dependencies conditions on which the vlidity of the query pln depends e.g., the existence of n index 2 Pg. 2 2

3 Query optimizer Execution modes SQL QUERY LEXICAL, SYNTACTIC AND SEMANTIC ANALYSIS INTERNAL REPRESENTATION BASED ON RELATIONAL ALGEBRA DATA DICTIONARY Compile nd go Compiltion nd immedite execution of the sttement No storge of the query pln Dependencies re not needed ALGEBRAIC CANONICAL QUERY TREE COST BASED DATA PROFILES (STATISTICS ON DATA) ACCESS PROGRAM SET OF DEPENDENCIES 3 4 Execution modes Compile nd store The ccess pln is stored in the dtbse together with its dependencies It is executed on demnd It should be recompiled when the dt structure chnges Dtbse Mngement Systems Algebric optimiztion 5 6 Algebric optimiztion Algebric optimiztion SQL QUERY LEXICAL, SYNTACTIC AND SEMANTIC ANALYSIS INTERNAL REPRESENTATION BASED ON RELATIONAL ALGEBRA ALGEBRAIC CANONICAL QUERY TREE COST BASED DATA DICTIONARY DATA PROFILES (STATISTICS ON DATA) It is bsed on equivlence trnsformtions Two reltionl expressions re equivlent if they both produce the sme query result for ny rbitrry dtbse instnce Interesting trnsformtions reduce the size of the intermedite result to be stored in memory prepre n expression for the ppliction of trnsformtion which reduces the size of the intermedite result ACCESS PROGRAM SET OF DEPENDENCIES 7 8 Pg. 3 3

4 . Atomiztion of selection s F Ʌ F2 (E) s F2 (s F (E)) s F (s F2 (E)). Atomiztion of selection s F Ʌ F2 (E) s F2 (s F (E)) s F (s F2 (E)) 2. Cscding projections p X (E) p X (p X,Y (E)) 9 2. Atomiztion of selection s F Ʌ F2 (E) s F2 (s F (E)) s F (s F2 (E)) 2. Cscding projections p X (E) p X (p X,Y (E)) 3. Anticiption of selection with respect to join (pushing selection down) s F (E E 2 ) E (s F (E 2 )) F is predicte on ttributes in E 2 only 4. Anticiption of projection with respect to join p L (E p E 2 ) p L ((p L, J (E )) p (p L2,J (E 2 ))) L = L - Schem(E 2 ) L2 = L - Schem(E ) J = set of ttributes needed to evlute join predicte p Join derivtion from Crtesin product s F (E E 2 ) E F E 2 predicte F only reltes ttributes in E nd E 2 5. Join derivtion from Crtesin product s F (E E 2 ) E F E 2 predicte F only reltes ttributes in E nd E 2 6. Distribution of selection with respect to union s F (E E 2 ) (s F (E ) ) (s F (E 2 ) ) Pg. 4 4

5 5. Join derivtion from Crtesin product s F (E E 2 ) E F E 2 predicte F only reltes ttributes in E nd E 2 6. Distribution of selection with respect to union s F (E E 2 ) (s F (E ) ) (s F (E 2 ) ) 7. Distribution of selection with respect to difference s F (E E 2 ) (s F (E ) ) (s F (E 2 ) ) 8. Distribution of projection with respect to union p X (E E 2 ) (p X (E )) (p X (E 2 )) (s F (E )) E Distribution of projection with respect to union p X (E E 2 ) (p X (E )) (p X (E 2 )) Cn projection be distributed with respect to difference? p X (E - E 2 ) (p X (E )) - (p X (E 2 )) 8. Distribution of projection with respect to union p X (E E 2 ) (p X (E )) (p X (E 2 )) Cn projection be distributed with respect to difference? p X (E - E 2 ) (p X (E )) - (p X (E 2 )) This equivlence only holds if X includes the primry key or set of ttributes with the sme properties (unique nd not null) Other properties s F V F2 (E) (s F (E)) (s F2 (E)) s F Ʌ F2 (E) (s F (E)) (s F2 (E)).Distribution of join with respect to union E (E E 2 ) (E E ) (E E 2 ) All binry opertors re commuttive nd ssocitive except for difference 29 3 Pg. 5 5

6 Exmple Exmple: Algebric trnsformtions Tbles EMP (Emp#,, Dept#, Slry) DEPT (Dept#, DNme,) p DNme (s EMP.Dept#=DEPT.Dept# Ʌ Slry > (EMP DEPT)) SQL query SELECT DISTINCT DNme FROM EMP, DEPT WHERE EMP.Dept#=DEPT.Dept# AND Slry > ; 3 32 Exmple: Algebric trnsformtions Exmple: Algebric trnsformtions p DNme (s EMP.Dept#=DEPT.Dept# Ʌ Slry > (EMP DEPT)) Prop # p DNme (s EMP.Dept#=DEPT.Dept# Ʌ Slry > (EMP DEPT)) Prop # p DNme (s Slry > (s EMP.Dept#=DEPT.Dept# (EMP DEPT)) p DNme (s Slry > (s EMP.Dept#=DEPT.Dept# (EMP DEPT)) Prop #5 p DNme (s Slry > (EMP DEPT) Exmple: Algebric trnsformtions Exmple: Algebric trnsformtions p DNme (s Slry > (EMP DEPT) p DNme (s Slry > (EMP DEPT) Prop #3 Prop #3 p DNme (s Slry > (EMP)) DEPT) p DNme (s Slry > (EMP)) DEPT) Prop #2 nd #4 p DNme ((p Dept# (s Slry > (EMP)) (p Dept#,DNme (DEPT))) Pg. 6 6

7 Exmple: Query tree Exmple: Crdinlities Finl query tree p DNme Crdinlity (EMP), Crdinlity (DEPT) Crdinlity (EMP where Slry > ) 5 p Dept# p Dept#,DNme s Slry> EMP DEPT Cost bsed optimiztion SQL QUERY LEXICAL, SYNTACTIC AND SEMANTIC ANALYSIS DATA DICTIONARY INTERNAL REPRESENTATION BASED ON RELATIONAL ALGEBRA Dtbse Mngement Systems ALGEBRAIC Cost bsed optimiztion CANONICAL QUERY TREE COST BASED DATA PROFILES (STATISTICS ON DATA) 39 ACCESS PROGRAM SET OF DEPENDENCIES 4 Cost bsed optimiztion It is bsed on Dt profiles sttisticl informtion describing dt distribution for tbles nd intermedite reltionl expressions Approximte cost formuls for ccess opertions Allow evluting the cost of different lterntives for executing reltionl opertor Dtbse Mngement Systems Dt profiles 4 42 Pg. 7 7

8 Tble profiles Tble profiles Quntittive informtion on the chrcteristics of tbles nd columns crdinlity (# of tuples) in ech tble T lso estimted for intermedite reltionl expressions size in bytes of tuples in T size in bytes of ech ttribute A j in T number of distinct vlues of ech ttribute in T crdinlity of the ctive domin of the ttribute min nd mx vlues of ech ttribute A j in T Tble profiles re stored in the dt dictionry Profiles should be periodiclly refreshed by renlyzing dt in the tbles Updte sttistics commnd Executed on demnd immedite execution during trnsction processing would overlod the system Dt profiles Tble profiles re exploited to estimte the size of intermedite reltionl expressions For the selection opertor Crd (s Ai = v (T)) Crd (T)/ Vl (A i in T) Vl (A i in T) = # of distinct vlues of A i in T (ctive domin) It holds only under the hypothesis of uniform distribution Dtbse Mngement Systems Access opertors Query tree Query tree Internl representtion of the reltionl expression s query tree p DNme Leves correspond to the physicl structures tbles, indices Intermedite nodes re opertions on dt supported by the given physicl structure e.g., scn, join, group by p Dept# p Dept#,DNme s Slry> EMP DEPT Pg. 8 8

9 Disk spce (MB) Dtbse Mngement Systems Sequentil scn Executes sequentil ccess to ll tuples in tble lso clled full tble scn Opertions performed during sequentil scn Projection discrds unnecessry columns Selection on simple predicte (A i =v) Sorting bsed on n ttribute list Insert, updte, delete Clssicl lgorithms in computer science re exploited e.g., quick sort Size of dt is relevnt memory sort sort on disk Sorting 49 5 Predicte evlution B+-tree versus bitmp If vilble, it my exploit index ccess B + -tree, hsh, or bitmp Simple equlity predicte A i =v Hsh, B + -tree, or bitmp re pproprite Rnge predicte v A i v 2 only B + -tree is pproprite For predictes with limited selectivity full tble scn is better if vilble, consider bitmp 5 B-tree Bitmp Len(Pointer) = 4 8 bit Bitmp VS B-Tree NK B-Tree NR Len(Pointer) NR NK bit Bitmp Courtesy of Golfrelli, Rizzi, Dt wrehouse, teori e prtic dell progettzione, McGrw Hill Predicte evlution Exmple: Predicte evlution Conjunction of predictes A i = v Ʌ A j = v 2 The most selective predicte is evluted first Tble is red through the index Next the other predictes re evluted on the intermedite result Optimiztion First compute the intersection of bitmps or RIDs coming from vilble indices Next tble red nd evlution of remining predictes 53 Which femle students living in Piemonte re exempt from enrollment fee? RID Gender Exempt Region M Y Piemonte 2 F Y Liguri 3 M N Pugli 4 M N Sicili 5 F Y Piemonte Gender D M BG 54 RID 5 Exempt Piemonte Pg. 9 9

10 Predicte evlution Join opertion Disjunction of predictes A i = v V A j = v 2 Index ccess cn be exploited only if ll predictes re supported by n index otherwise full tble scn A criticl opertion for reltionl DBMS connection between tbles is bsed on vlues insted of pointers size of the intermedite result is typiclly lrger thn the smller tble Different join lgorithms Nested loop Merge scn join Hsh join Bitmpped join Nested loop Nested loop Outer tble A externl scn A Inner tble A single full scn is done on the outer tble For ech tuple in the outer tble full scn of the inner tble is performed, looking for corresponding vlues Also clled brute force internl or direct scn join D M BG ttribute Nested loop Merge scn Efficient when inner tble is smll nd fits in memory optimized scn join ttribute in the inner tble is indexed index scn Execution cost The nested loop join technique is not symmetric The execution cost depends on which tble tkes the role of inner tble Left tble A b b c e left scn right scn A b d e Right tble 59 join D M BG ttribute 6 Pg.

11 Merge scn Both tbles re sorted on the join ttributes The two tbles re scnned in prllel tuple pirs re generted on corresponding vlues Execution cost The merge scn technique is symmetric requires sorting both tbles my be sorted by previous opertion my be red through clustered index on join ttributes More used in the pst efficient for lrge tbles, becuse sorted tbles D M B my be stored on disk G 6 From left tble HASH() Buckets for left tble d e c j p Buckets for right tble Hsh Join D M B Join G 62 Attribute e m w j z From right tble HASH() Hsh join Bitmpped join index Appliction of the sme hsh function to the join ttributes in both tbles Tuples to be joined end up in the sme buckets collisions re generted by tuples yielding the sme hsh function result with different ttribute vlue A locl sort nd join is performed into ech bucket Very fst join technique 63 Bit mtrix tht precomputes the join between two tbles A nd B One column for ech RID in tble A One row for ech RID in tble B Position (i, j) of the mtrix is if tuple with RID j in tble A joins with tuple with RID i in tble B RID 2 n otherwise Updtes my be slow D M BG 64 Bitmpped join Typiclly used in OLAP queries joining severl tbles with lrge centrl tble Exmple Exm tble, joined to Student nd Course tbles Exploits one or more bitmpped join indices One for ech pir of joined tbles Access to the lrge centrl tble is the lst step Bitmpped join Complex queries my exploit jointly bitmpped join indices bitmp indices for predictes on single tbles Pg.

12 Exmple: Bitmpped join Bitmpped join Averge score of mle students for exms of courses in the first yer of the mster degree STUDENT (Reg#, SNme, Gender) COURSE (Course#, CNme, CourseYer) EXAM (Reg#, Course#, Dte, Grde) SELECT AVG (Grde) FROM STUDENT S, EXAM E, COURSE C WHERE E.Reg# = S.Reg# AND E.Course# = C.Course# AND CourseYer = M AND Gender = M ; 67 RIDs nd 4 Bitmpped join index for Course-Exms join FROM EXAM E, COURSE C WHERE E.Course# = C.Course# AND CourseYer = M RID Bitmp for CourseYer ttribute RID M OR 4 = RID CY 68 Bitmpped join Group by RID CY AND bitmp for Course-Exm predictes nd join RID G = RID bitmp for Student-Exm predictes nd join RIDs of Exm tble for tuples to be red Sort bsed Sort on the group by ttributes Next compute ggregte functions on groups Hsh bsed Hsh function on the group by ttributes Next sort ech bucket nd compute ggregte functions Mterilized views my be exploited to improve the performnce of ggregtion opertions 69 7 Cost bsed optimiztion Dtbse Mngement Systems Execution pln selection 7 Inputs Dt profiles Internl representtion of the query tree Output Optiml query execution pln Set of dependencies It evlutes the cost of different lterntives for reding ech tble executing ech reltionl opertor It exploits pproximte cost formuls for ccess opertions 72 Pg. 2 2

13 Generl pproch to optimiztion Generl pproch to optimiztion The serch for the optiml pln is bsed on the following dimensions The wy dt is red from disk e.g., full scn, index The execution order mong opertors e.g., join order between two join opertions The technique by mens of which ech opertor is implemented e.g., the join method When to perform sort (if sort is needed) The optimizer builds tree of lterntives in which ech internl node mkes decision on vrible ech lef represents complete query execution pln Exmple Exmple Given 3 tbles R, S, T Compute the join R S T Execution lterntives 4 join techniques to evlute (for both joins) 3 join orders In totl, t most 4 * 4 * 3 = 48 different lterntives R S T 2 R INNER 2 NESTED LOOP NESTED LOOP R S T 2 S T R 2 S INNER 2 NESTED LOOP NESTED LOOP MERGE SCAN R T S 2 HASH JOIN T INNER T OUTER 75 LEAF NODE 76 Best execution pln selection Best execution pln selection The optimizer selects the lef with the lowest cost Generl formul C Totl = C I/O x n I/O + C cpu x n cpu n I/O is the number of I/O opertions n cpu is the number of CPU opertions The selection is bsed on opertion reserch optimiztion techniques e.g., brnch nd bound The finl execution pln is n pproximtion of the best solution The optimizer looks for solution which is of the sme order of mgnitude of the best solution For compile nd go it stops when the time spent in serching is comprble to the time required to execute the current best pln Pg. 3 3