UNIT 11. Query Optimization

Transcription

1 UNIT Query Optimiztion

2 Contents Introduction to Query Optimiztion 2 The Optimiztion Process: An Overview 3 Optimiztion in System R 4 Optimiztion in INGRES 5 Implementing the Join Opertors Wei-Png Yng, Informtion Mngement, NDHU -2

3 Introduction to Query Optimiztion -3

4 The Problem How to choose n efficient strtegy for evluting given expression ( query) Expression ( query): eg select distinct SSNAME from S, SP where SS# =SPS# nd SPP#= 'p2' Evlute: Efficient strtegy: First clss eg (A join B) where condition-on-b (A join (B where condition-on-b) ) eg SPP# = 'p2' Second clss eg from S, SP ==> S join SP P-3 How to implement join opertion efficiently? Improvement" my not be n "optiml" version Wei-Png Yng, Informtion Mngement, NDHU -4

5 Query Processing in the DBMS Query in SQL: SELECT CUSTOMER NAME FROM CUSTOMER, INVOICE WHERE REGION = 'NY' AND AMOUNT > 0000 AND CUTOMERC#=INVOICEC# Internl Form : P(σ (S SP) Opertor : SCAN C using region index, crete C SCAN I using mount index, crete I SORT C?nd I?on C# JOIN C?nd I?on C# EXTRACT nme field Clls to Access Method: OPEN SCAN on C with region index GET next tuple Clls to file system: GET0th to 25th bytes from block #6 of file #5 Lnguge Processor Optimizer? Opertor Processor Access Method File System Lnguge Processor Access Method Wei-Png Yng, Informtion Mngement, NDHU -5 dtbse

6 Suppose: S = 00, An Exmple SP = 0,000, nd there re 50 tuples in SP with p# = 'p2'? Results re plced in Min Memory Query in SQL: SELECT S* FROM S,SP WHERE SS# = SPS# AND SPP# = 'p2 Method : itertion (Join + Restrict) 2 00 S S# S2 S5 S SNAME STATUS CITY 2 0,000 SP S# P# QTY S3 S S2 Cost = 00 * 0,000 =,000,000 tuple I/O's Wei-Png Yng, Informtion Mngement, NDHU -6

7 An Exmple (cont) Method 2: Restriction itertion Join SP SP' S# P# QTY S3 S S2 P4 P2 P2 restrict p#= 'p2' 2 50 S# P# QTY S P2 S3 P2 S2 P2 S SP' 2 00 S# S2 S5 S SNAME STATUS CITY 2 50 S# P# QTY S P2 S3 P2 S2 P2 cost = 0, * 50 = 5,000 I/O Wei-Png Yng, Informtion Mngement, NDHU -7

8 An Exmple (cont) Method 3: Sort-Merge Join + Restrict Suppose S, SP re sorted on S# S 2 00 S# SNAME STATUS CITY S S2 S00 SP 2 0,000 S# P# QTY S S S00 cost = ,000 = 0,00 I/O Wei-Png Yng, Informtion Mngement, NDHU -8

9 2 The Optimiztion Process: An Overview () Query => internl form (2) Internl form => efficient form (3) Choose cndidte low-level procedures (4) Generte query plns nd choose the chepest one -9

10 Step : Cst the query into some internl representtion Query: "get nmes of suppliers who supply prt p2" SQL: select distinct SSNAME from S,SP where SS# = SPS# nd SPP# = 'p2' Query tree: Query => Algebr result project (SNAME) restrict (SPP# = 'p2') join (SS# = SPS#) S SP Algebr: ( (S join SP) where P#= 'P2') [SNAME] or ( ( S SP) ) SS# = SPS# Wei-Png Yng, Informtion Mngement, NDHU -0 SNAME 'P2'

11 Step 2: Convert to equivlent nd efficient form Def: Cnonicl Form Given set Q of queries, for q, q2 belong to Q, q re equivlent to q2 (q q2) iff they produce the sme result, Subset C of Q is sid to be set of cnonicl forms for Q iff q Q! c C q c Note: Sufficient to study the smll set C Trnsformtion Rules Step2 output of step trns equivlent nd more Algebr efficient form C Q Efficient Algebr Wei-Png Yng, Informtion Mngement, NDHU -

12 Step 2: Convert to equivlent nd efficient form (cont) eg [restriction first] (A join B) where restriction_b q C A join ( B where restriction_b) q 2 q q 2 eg2 [More generl cse] (A join B) where restriction_a nd restriction_b (A where rest_on_a) join ( B where rest_on_b) eg3 [ Combine restriction] ( A where rest_ ) where rest_2 scn 2 A where rest_ nd rest_2 Wei-Png Yng, Informtion Mngement, NDHU -2

13 Step 2: Convert to equivlent nd efficient form (cont) eg4 [projection] lst ttribute (A [ttribute_list_] ) [ttri_2] A [ttri_2] eg5 [restriction first] (A [ttri_]) where rest_ n<n n+n (A where rest _) [ttri_] Wei-Png Yng, Informtion Mngement, NDHU -3

14 Step 2: Convert to equivlent nd efficient form (cont) eg6 [Introduce extr restriction] SP JOIN (P WHERE PP#= 'P2') spp# = pp# if restriction on join ttribute (SP WHERE SPP# = 'P2') JOIN (P WHERE PP# = 'P2') eg7 [Semntic trnsformtion] (SP join P ) [S#] spp# = pp# SP[S#] if SPP# is foreign key mtching the primry term PP# Note: very significnt improvement Ref[727] P57 J J King, VLDB8 P P# P P2 P3 P4 P5 SP S# P# QTY S S2 Wei-Png Yng, Informtion Mngement, NDHU -4

15 Step 3: Choose cndidte low-level procedures Low-level procedure eg Join, restriction re low-level opertors there will be set of procedures for implementing ech opertor, eg Join (ref p-3) <> Nested Loop ( brute force) <2> Index lookup (if one reltion is indexed on join ttribute) <3> Hsh lookup (if one reltion is hshed by join ttribute) <4> Merge (if both reltions re indexed on join ttribute) Wei-Png Yng, Informtion Mngement, NDHU -5

16 Step 3: Choose cndidte low-level procedures (cont) Dt flow SQL Algebr System ctlog Cnonicl Form eg ( (C I)) Lib existence of indexes crdinlities of reltions Optimizer step3 : ccess pth selection predefined low-level procedures Ref p-3 p554 One or more cndidte procedures for ech opertor 6 choose 2 Wei-Png Yng, Informtion Mngement, NDHU -6 Step4 eg,, 2 3 2

17 Step 4: Generte query plns nd choose the chepest Query pln is built by combing together set of cndidte implementtion procedures for ny given query mny mny resonble plns Note: my not be good ide to generte ll possible plns heuristic technique "keep the set within bound" (reducing the serch spce) Wei-Png Yng, Informtion Mngement, NDHU -7

18 Step 4: Generte query plns nd choose the chepest (cont) Dt flow output of step 3 Step 4() ( (C I)) query plns 2 Step 4(b) choose the chepest chepest 2 2 Wei-Png Yng, Informtion Mngement, NDHU -8

19 Step 4: Generte query plns nd choose the chepest (cont) Choosing the chepest require method for ssigning cost to ny given pln fctor of cost formul: () # of disk I/O (2) CPU utiliztion (3) size of intermedite results difficult problem [Jrke 84, 73 p564 ACM computing surveys] [Yo 79, 78 TODS] Wei-Png Yng, Informtion Mngement, NDHU -9

20 3 Optimiztion in System R -20

21 Optimiztion in System R Only minor chnges to DB2 nd SQL/DS Query in System R (SQL) is set of "select-from-where" block System R optimizer step: choosing block order first in cse of nested => innermost block first step2: optimizing individul blocks Note: certin possible query pln will never be considered The sttisticl informtion for optimizer Where: from the system ctlog Wht: # of tuples on ech reltion 2 # of pges occupied by ech reltion 3 percentge of pges occupied by ech reltion 4 # of distinct dt vlues for ech index 5 # of pges occupied by ech index Note: not updted every time the dtbse is updted (overhed??) Wei-Png Yng, Informtion Mngement, NDHU -2

22 Optimiztion in System R (cont) Given query block cse involves just restriction nd/or projection sttisticl informtion (in ctlog) 2 formuls for size estimtes of intermedite results 3 formuls for cost of low-level opertions (next section) choose strtegy for constructing the query opertion cse 2 involves one or more join opertions eg A join B join C join D ((A join B) join C) join D Never: (A join B) join (C join D) Why? See next pge Wei-Png Yng, Informtion Mngement, NDHU -22

23 Optimiztion in System R (cont) Note: "reducing the serch spce" 2 heuristics for choosing the sequence of joins re given in [734] P573 3 (A join B) join C ((A join B) join C) join D not necessry to compute entirely before join C ie if ny tuple hs been produced Never: (A join B) join (C join D) pss to join C It my never be necessry to finish reltion "A B ", why? C hs run out?? Wei-Png Yng, Informtion Mngement, NDHU -23

24 Optimiztion in System R (cont) How to determine the order of join in System R? consider only sequentil execution of multiple join <eg> ((A B) C) D (A B) (C D) STEP: Generte ll possible sequences <eg> () ((A B) C) D (2) ((A B) D) C (3) ((A C) B) D (4) ((A C) D) B (5) ((A D) B) C (6) ((A D) C) B (7) ((B C) A) D (8) ((B C ) D) A (9) ((B D) A) C (0) ((B D) C) A () ((C D) A) B (2) ((C D) B) A Totl # of sequences = ( 4! )/ 2 = 2 Wei-Png Yng, Informtion Mngement, NDHU -24

25 Optimiztion in System R (cont) STEP 2: Eliminte those sequences tht involve Crtesin Product if A nd B hve no ttribute nmes in common, then A B = A x B STEP 3: For the reminder, estimte the cost nd choose chepest Wei-Png Yng, Informtion Mngement, NDHU -25

26 4 Optimiztion in INGRES -26

27 Query Decomposition generl ide for processing queries in INGRES bsic ide: brek query involving multiple tuple vribles down into sequence of smller queries involving one such vrible ech, using detchment nd tuple substitution void to build Crtesin Product keep the # of tuple to be scnned to minimum <eg> "Get nmes of London suppliers who supply some red prt weighing less thn 25 pounds in quntity greter thn 200" Initil query: Q0: RETRIEVE (SSNAME) WHERE SCITY= 'London' AND SS# = SPS# AND SPQTY > 200 AND SPP# = PP# AND PCOLOR = Red AND PWEIGHT < 2 5 detch P Wei-Png Yng, Informtion Mngement, NDHU -27

28 Query Decomposition (cont) D: RETRIEVE INTO P' (PP#) WHERE PCOLOR= 'Red' AND PWEIGHT < 25 Q: RETRIVE (SSNAME) WHERE SCITY = 'London' AND SS# = SPS# AND SPQTY > 200 AND SPP# = P'P# S join SP join P detch SP D2: RETRIEVE INTO SP' (SPS#, SPP#) WHERE SPQTY > 200 Q2: RETRIEVE (SSNAME) WHERE SCITY = 'London' AND SS#=SP'S# AND SP'P#=P'P# detch S Wei-Png Yng, Informtion Mngement, NDHU -28

29 Query Decomposition (cont) D3: RETRIEVE INTO S' (SS#, SSNAME) WHERE SCITY = 'LONDON' Q3: RETRIEVE (S'SNAME) WHERE S'S# =SP'S# AND SP'P# = P'P# D4: RETRIEVE INTO SP"(SP'S#) WHERE SP'P# =P'P# Q4: RETRIEVE (S'SNAME) WHERE S'S# = SP"S# D5: RETRIEVE INTO SP"(SP'S#) WHERE SP'P# = 'P' OR SP'P#= 'P3 detch P' nd SP' D4: two vr --> tuple substitution ( Suppose D evlute to {P, P3 } Q5: RETRIEVE (S'SNAME) WHERE S'S# = 'S' OR S'S# = 'S2' OR S'S# = 'S4' Q4 : two vr --> tuple substitution ( Suppose D5 evlute to { S, S2, S4}) Wei-Png Yng, Informtion Mngement, NDHU -29

30 Query Decomposition (cont) Decomposition tree for query Q0: Overll result D, D2, D3: queries involve only one vrible => evlute D4, Q4: queries involve tow vrible => tuple substitution D3 S' Q4 (Q5) P' SP'' D4 (D5) SP' - Objectives : void to build Crtesin Product keep the # of tuple to be scnned to minimum S D D2 P SP Wei-Png Yng, Informtion Mngement, NDHU -30

31 5 Implementing the Join Opertors Method : Nested Loop Method 2: Index Lookup Method 3: Hsh Lookup Method 4: Merge bck -3

32 Wei-Png Yng, Informtion Mngement, NDHU -32 Join Opertion Suppose R S is required, RA nd SA re join ttributes R A b e m S A b n

33 Method : Nested Loop Suppose R nd S re not sorted on A m R A b e n S A b - O (mn) - the worst cse - ssume tht S is neither indexed nor hshed on A - will usully be improved by constructing index or hsh on SA dynmiclly nd then proceeding with n index or hsh lookup scn Wei-Png Yng, Informtion Mngement, NDHU -33

34 Wei-Png Yng, Informtion Mngement, NDHU -34 Method 2: Index Lookup Suppose S in indexed on A S A b A b e m R b SA_index n

35 Suppose S is hshed on A Method 3: Hsh Lookup m R RA b z e h(e) h(e) = h(b) = 0 h() = 2 h(z) = 2 h() 0 2 S SA b e z -Clculte hsh function is fster thn serch in index Wei-Png Yng, Informtion Mngement, NDHU -35

36 Wei-Png Yng, Informtion Mngement, NDHU -36 Method 4: Merge Suppose R nd S re both sorted (for indexed) on A = A b z c b A b d b R S b A b b c z m R A b b d S n Only index is retrieved for ny unmtched tuple Bck to p-4