Fundamentals of DB2 Query Optimization

Transcription

1 Session: <G05> Fundamentals of DB2 Query Optimization Michelle Guo IBM, Silicon Valley Lab May 08, :20 a.m. 10:20 a.m. Platform: <DB2 for z/os> Through this 1-hr session, we will walk through the fundamental elements in DB2 z/os query optimization. SQL is declarative language, it tells database WHAT not HOW. And in general, more than one way to evaluate query Query optimization will decide what is the most efficient access path to return the correct result. It is beneficial to all of us, DBA, application developer, DB2 service, and DB2 developer to know what are the key elements in query optimization. What are the candidates, what you need to do, and what DB2 does to choose the optimal access path. 1

2 Objectives 1. Learn the architecture of DB2 query optimization. 2. Learn the fundamental techniques DB2 Query Optimization uses today. 3. Learn what influences query optimizer on access path decision. 4. Get familiar with EXPLAIN and how to communicate performance issue with DB2 support. 5. Learn the new functions in latest release and how to exploit them. 2 By end of this session, we are to achieve these 5 objectives. 1. Architecture of db2 query optimization: query transformation, access path enumeration, cost estimation, etc. 2. Fundamental techniques are: within each architecture component, what are the typical techniques used. 1. Query transformation: subquery transformation, predicate push down, etc. 2. Access path enumeration: access method, join method, index matching, screening, etc. 3. Access path decision: cost estimation & statistics 4. Explain: how to read it and what to report for a PMR. 5. New functions: are included in each techniques whenever appropriate. 2

3 Agenda Part A Session 1: Overview Session 2: Access path and explain table Session 3: Predicate application Session 4: Access methods Part B Session 5: Join methods Session 6: Query transformation Session 7: Statistics and cost estimation Session 8: Related optimization sessions 3 3

4 Overview of DB2 Optimizer Query transformation Access path enumeration Cost estimation Parallelism optimization (not covered in this presentation) Explain of optimal access path 4 Cost estimation is embedded within each access path enumeration. 1. Join sequence 2. Access method 3. Parallelism decision (new in V9) 4. Sort decision //sm optimization is not covered in this session. Explain output details will be shown later. 4

5 Agenda Part A Session 1: Overview Session 2: Access path and explain table Session 3: DB2 Runtime Architecture and predicate application Session 4: Access methods Part B Session 5: Join methods Session 6: Query transformation Session 7: Statistics and cost estimation Session 8: Related optimization sessions 5 5

6 Access Path Enumeration Final Access Path Query SELECT d.name, sum(e.id), avg(e.compensation) FROM employees e, departments d, projects p WHERE d.location in ( SVL, ARC ) and d.id = e.dept and e.id = p.employeeid and p.class in ( top secret, confidential ) GROUP BY d.name ORDER BY d.name Assumptions Index: ix1(d.location) ix2(e.id), ix3(e.dept) ix4(p.class) Cardinality: card(e) = card(d) = card(p) = Filtering factor: FF(P1) = 1%, FF(P4) = 10% Number of join sequences (3! = 6) Number of access methods (RSCAN, ISCAN, M-idx, etc) Number of join methods (NLJ, SMJ, HBJ) Total number of access paths Cost estimation (elapsed time) Access path selection (shortest elapsed time) 6 RSCAN e NLJ ISCAN d SORT NLJ ISCAN p Also called dynamic programming 6

7 Plan Clipping Query is getting bigger and more complicated Big query consumes lots of resources at bind time Time Storage Answer: Plan clipping MAX_OPT_STOR: maximum storage allowed for bind time optimization MAX_OPT_CPU: maximum CPU time allowed for bind time optimization MXQBCE: maximum number of intermediate composites allowed for each query block at bind time 7 Check when started to build N-table composite. 1. MAX_OPT_STOR: integer of MBs. Default is 20MB. Maximum is 100MB 2. MAX_OPT_CPU: integer of Secs. Default is 100s. Maximum is 1000s. Emergency Clipping: 1. Condition: The current resource consumption, either CPU or Storage, exceeds total resource limit, then enter an emergency clipping mode 2. Algorithm: trims down choices to only keep the best plan. * best: 1. weighted based on 1).cost 2).composite size 3).#tables left to join 2. best means with cheaper cost (spreaded to #tables left to join) + smaller composite size Adaptive Clipping: Condition: the current resource consumption, either CPU or Storage, exceeds the proportional limit, then enter an adaptive clipping mode Algorithm: trims down choices to make it within the proportional quota Continue dynamic programming after clipping. 7

8 Mini Plan Query SELECT d.name, sum(e.id), avg(e.compensation) FROM employees e, departments d, projects p WHERE d.location in ( SVL, ARC, Toronto, Raleigh ) and d.id = e.dept and e.id = p.employeeid and p.class in ( top secret, confidential ) GROUP BY d.name Mini plans Mini plan Plan record identification Access of new table Sort of new table Join method Sort of the composite SORT NLJ NLJ ISCAN SORT RSCAN e 8 ISCAN d p Mini Plan break down. Each step is joining a new table or a sort at the end. Old composite + new table = new composite Join: 1. Access method 2. Join method 3. Sort (for join) decision Sort: 1.Sort for groupby, orderby, distinct, etc. 8

9 Agenda Part A Session 1: Overview Session 2: Access path and explain table Session 3: Predicate application Session 4: Access methods Part B Session 5: Join methods Session 6: Query transformation Session 7: Statistics and cost estimation Session 8: Related optimization sessions 9 9

10 Predicate sargability This section: Objectives To obtain an understanding of the different predicate types and differentiate the sargability (or stage of application) of SQL predicates. Introduces Predicate Properties Predicate Types Predicate Application Stages Indexability Improving predicate performance 10 10

11 DB2 Runtime architecture join RDS Rscan (d) Iscan (e) Stage 2 Stage 1 fetch fetch find key screening DMS IDX fixpg fixpg matching select * from dept d, emp e where d.budget > // Stage 1 andd.name like '%Opt%' // Stage 2 and d.id = e.dept // Matching and e.salary > // Matching ande.bonus < 7000 // Screening Index: e.dept, e.salary, e.bonus data page index page BPS Database 11 e.dept = 30 e.salary > e.bonus < 7000 Predicate push down. Not the same as predicate push down in query transformation. Here it means pushed for earlier evaluation. IDX, then DMS, and then RDS. The earlier the evaluation, the more efficient it is. The goal is to have the best filtering at the earliest possible stage. 11

12 Predicate application Stage 1 and Stage 2 Predicates: Stage 1 Predicates Sometimes referred to as Sargable Can be applied at the 1st stage of predicate processing All indexable predicates are also stage 1 before v9 But not all stage 1 predicates are indexable V9 index on expression, non-sargable predicate can be indexable Stage 2 Predicates Sometimes referred to as Nonsargable or Residual Cannot be applied until the 2nd stage of predicate processing And are therefore not indexable The following may determine whether a predicate is stage 1 or 2 Predicate syntax (see following tables for examples) Type and length of constants or columns in the predicate DB2 V8 resolves most of these Whether the predicate is applied before or after a join Table join sequence 12 Administration Guide has detail information on what kind of predicates are stage1, indexable or stage2. In general stage1 predicates are those simple predicates, e.g. filtering on a single column. If the filtering column matches any index key column, then it becomes indexable. All complicated predicates are left as stage-2. 12

13 V8 improvement Unmatched datatype, length, CCSID Unknown join sequence: Column-expression SELECT e1.* FROM emp AS e1, emp AS e2, dept WHERE e1.deptid = dept.id AND dept.mgr = e2.name AND e1.salary > e2.salary * 1.10 AND e1.salary > e2.salary * 1.10; dec(12,2) prior to v8 v8 Stage-2 predicate If e2 is inner table Stage-2 predicate If e1 is inner table Stage-1 predicate Could use index on emp.salary 13 A Query searching for employee whose salary is 10% more than his/her manager. 1. Predicate e1.salary 13

14 V9 improvement Index on expression, non-sargable predicates on key expression can be indexable SELECT e1.* FROM emp AS e1, dept AS d1 WHERE e1.deptno = d1.deptno AND d1.headcount > 0 AND e1.salary+e1.bonus > 80000; prior to v9 v9 Stage-2 predicate With index on EMP(SALARY+BONUS) it is indexable when index is used otherwise, it is stage-2 14 A Query searching for employee whose salary is 10% more than his/her manager. 1. Predicate e1.salary 14

15 Agenda Part A Session 1: Overview Session 2: Access path and explain table Session 3: Predicate application Session 4: Access methods Part B Session 5: Join methods Session 6: Query transformation Session 7: Statistics and cost estimation Session 8: Related optimization sessions 15 15

16 Single table access method execution This section: Objectives To highlight the different access methods available for single table access. Single Table Access Methods Tablespace Scan Index Access List Prefetch 16 16

17 Tablespace Scan Segmented T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T3 T3 T3 T3 database 16 pages for each segment CREATE TABLESPACE tablespace IN database SEGSIZE ; SELECT * FROM T3 WHERE... ; Non-Segmented T1 T1 T1 T1 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T1 T1 T2 T2 T2 T2 T1 T1 T1 T1 T2 T2 T3 T3 T1 T1 T2 T2 T2 T2 T1 T1 T1 T1 T2 T2 T2 T2 T2 T2 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T3 T3 T1 T1 T1 CREATE TABLESPACE tablespace IN database... ; database 17 17

18 Page range screening Table space scan Page Range Screening Also known as Partition Elimination or Limited Partition Scan P.C1 = R-SCAN ACCESS TYPE = R SELECT * FROM P WHERE P.C1 <= 2 AND... ; P.C1 = R-SCAN ` ACCESS TYPE = R, PAGE RANGE = Y Page Range column of plan table indicates a subset of the partitions are R-scanned When there is simple, local predicates on partitioning key column, then page range screening can be derived from it. 2. Not necessary a consecutive range. Could be something like part1, part5, part99, etc. 18

19 Sequential prefetch Reads a sequential set of pages into the bufferpool with one asynchronous I/O Usually the max number of pages is 32 for base table and 8 for workfile Generally used for table space scan Sometimes used for index scan before V8, when Index clusterratiof > 0.8 For index only: number of qualified leaf pages > 8 For index + data: number of qualified clustered data pages > 8 V9 uses dynamic prefetch 19 19

20 Tablespace scan EXPLAIN PLAN SET QUERYNO = 1 FOR SELECT * FROM DSN8710.EMP; EXPLAIN PLAN SET QUERYNO = 2 FOR SELECT * FROM DSN8710.EMP OPTIMIZE FOR 1 ROWS; EXPLAIN PLAN SET QUERYNO = 3 FOR SELECT * FROM DSN8710.EMP FETCH FIRST 1 ROWS ONLY; Optimize or Fetch First QQP M T H TNAME AT MC A_NM YP OL IXO SORT NCCCC J UJOG PF QB_TYP PQ B TB_TYP EMP R 0 N S SELECT 0 T EMP R 0 N SELECT 0 T EMP R 0 N SELECT 0 T Sequential Prefetch Impact 20 20

21 Index scan matching SELECT * FROM DSN8710.EMP WHERE WORKDEPT =? AND EMPNO =? QQP M T H TNAME A T Y P M C O L Index + data access A_NM IX O 21 SORT NCCCC J UJOG P F MI X QB_TYP EMP I 2 EMPX2 N 0 SELECT 0 T P Q B TB_T YP Index access with 2 matching columns 21

22 Index scan non-matching SCAN ` SELECT WORKDEPT FROM DSN8710.EMP WHERE EMPNO >? ORDER BY WORKDEPT QQP M T H TNAME A T Y P Data retrieval satisfied by index only. No access to table required. M C O L A_NM IX O SORT NCCCC J UJOG 22 P F MI X QB_TYP EMP I 0 EMPX2 Y 0 SELECT 0 T P Q B TB_T YP Index access with zero matching columns Non-matching, especially and non screening index scan, is usually picked for it can preserve an interested order, such as OBY, GBY, DISTINCT, etc. 22

23 Multiple-index access 1 2 index: WORKDEPT, EMPNO index: LOCALE 3 Union WHERE ((WORKDEPT =? AND EMPNO =?) OR (WORKDEPT =? AND EMPNO =?)) AND LOCALE = CA Intersection ` List Prefetch 23 23

24 Multiple-index access - example SELECT EMPNO, WORKDEPT, SALARY FROM DSN8710.EMP WHERE ((WORKDEPT =? AND EMPNO =?) OR (WORKDEPT =? AND EMPNO =?)) AND LOCALE= CA ORDER BY SALARY Data Access using List Prefetch QQP M T H TNAME ATY P M C O L A_NM IX O 24 SORT NCCCC J UJOG P F MI X QB_TYP EMP M 0 N L 0 SELECT 0 T EMP MX 2 EMPX2 Y 1 SELECT 0 T EMP MX 2 EMPX2 Y 2 SELECT 0 T EMP MU 0 N 3 SELECT 0 T EMP MI 2 EMPX1 Y 4 SELECT 0 T EMP MU 0 N 5 SELECT 0 T N NNNYN 0 SELECT 0 Union output from step 1 & 2 Intersection output from step 3 & 4 P Q B TB_T YP Multi-Index Access Steps If predicate LOCALE= CA is very selective, then it might just choose EMPX1 for single-index access. It is a cost-based decision. 24

25 Page range screening - NPI Page Range Screening can be applied before data access on a NPI to limit the partitions accessed if a predicate exists that can be applied Similar to index screening Without requiring the screening column to be indexed C1 C1= 10 NPI on C1 SELECT cols FROM T1 WHERE C1 = 10 AND YEAR = Partitioned by YEAR 25

26 List Prefetch List Prefetch concepts Reads set of pages into bufferpool with one asynchronous I/O The set of pages are determined by a list of RIDs taken from an index Currently the max number of pages prefetched is 32 within 180 page swath (list prefetch can skip pages) Generally used with Index scan when clusterratiof is less than 0.8 Accessing data from the inner table during a hybrid join Multi-index access When direct access not possible (for update of...) With high clusterratiof if number of qualified pages between 1 and sequential prefetch 26 26

27 List Prefetch SELECT EMPNO, SALARY FROM DSN8710.EMP WHERE WORKDEPT =? ORDER BY SALARY RID Pool Sort ` QQP M T H TNAME ATY P M C O L A_NM IX O SORT NCCCC J UJOG P F QB_TYP EMP I 0 EMPX2 N NNNNN L SELECT 0 T N NNNYN SELECT 0 P Q B TB_T YP ORDER BY Sort unavoidable 27 Prefetch Indicator 27

28 Agenda Part A Session 1: Overview Session 2: Access path and explain table Session 3: Predicate application Session 4: Access methods Part B Session 5: Join methods Session 6: Query transformation Session 7: Statistics and cost estimation Session 8: Related optimization sessions 28 28

29 Join Method Execution This Section: Objectives To cover the three join methods used for processing SQL containing table joins. Join Methods Nested Loop Join Sort Merge Join Hybrid Join NOTE: The fourth join method, Star Join, will not be discussed in this presentation 29 29

30 Table join terminology Composite Table Outer table of the join In a two table join, this is the first table accessed New Table Inner table of the join In a two table join, this is the second table accessed T1 & T2 become Composite New the composite for T1 T2 the join to T3 T3 Composite 30 New 30

31 Plan Table columns Nested Loop Join (NLJ) Access outer (composite) table using efficient single table access T1 For each qualifying outer table row access the inner table using efficient single table access Join the results R-scan SELECT T1.C4, T2.C6 FROM T1, T2 WHERE T1.C1 = T2.C1 AND T1.C4 > 1 ; T

32 Nested Loop Join Local Predicate applied first SELECT * FROM DSN8710.EMP E JOIN DSN8710.PROJ P ON E.WORKDEPT = P.DEPTNO WHERE E.WORKDEPT IN ('A00', 'B01', 'C01') WORKDEPT EMPNO A A A A A B C C C C No Match P.DEPTNO = 'A00' One Match P.DEPTNO = 'B01' DEPTNO B01 C01 C01 PROJNO PL2100 IF1000 IF2000 Two Matches P.DEPTNO = 'C01' (for each row where E.WORKDEPT = 'C01') 1. Access outer (composite) table using efficient single table access 1. For each qualifying outer table row access the inner table using efficient single table access 2. Join the results 32

33 Nested Loop Join SELECT * FROM DSN8710.EMP E JOIN DSN8710.PROJ P ON E.WORKDEPT = P.DEPTNO WHERE E.WORKDEPT IN ('A00', 'B01', 'C01') QQP M T H TNAME A T Y P M C O L A_NM IX O SORT NCCCC J UJOG CORR _NM QB_TYP EMP N 1 EMPX2 N E SELECT 0 T PROJ I 1 PROJX2 N P SELECT 0 T P Q B TB_T YP Nested Loop Join IN list/index access, 1 matchcol for local and join predicate 33 33

34 Plan Table columns Sort Merge Join (SMJ) Also known as Merge Scan Join. Access inner/outer table using efficient single table access and apply eligible S1/S2/SubQry predicates Sort inner/outer tables (can avoid sort if index provides ordering) Inner table always written to workfile Merge filtered, sorted inputs 34 SELECT T1.C4, T2.C6 FROM T1, T2 WHERE T1.C1 = T2.C1 AND T1.C2 = 1 AND T2.C5 > 10 ; Index IX1 on T1 (C2,C1) Clusterratiof = No index on T2 T1 merge T2 R-scan sort workfile 34

35 Sort Merge Join QQP SELECT * FROM DSN8710.EMP E FULL JOIN DSN8710.DEPT D ON E.WORKDEPT = P.DEPTNO M T H TNAME A T Y P M C O L A_NM IX O SORT NCCCC J UJOG P F M J C CORR _NM EMP I 0 EMPX2 N E DEPT R N YNNNN S 1 D F Full Join used to force SMJ J T Sort Merge Join SORTN_JOIN = 'Y'. Merge Join Cols =

36 Sort Merge Join sort new QQP M T H TNAME A T Y P M C O L A_NM IX O SORT NCCCC J UJOG P F M J C CORR _NM EMP I 0 EMPX2 N E DEPT R N YNNNN S 1 D F J T EMP Scan Scan merge Data accessed in index sequence DEPT R-scan sort workfile 36 SORTN_JOIN = 'Y' Read DEPT using R-scan into a workfile Sort workfile into join col seq Access EMP using non-matching index scan (to avoid sort) Match/merge EMP with workfile (while reading EMP) 1. New table can obtain the join order by index access. 2. Composite can obtains the join order by leading table index access or a sort along the join already happened. 3. Depends on the order, query optimization decides where to sort composite, sort new, sort both or sort none. 36

37 Join Methods Hybrid Join (HBJ) Apply only to an inner join and requires an index on the join column(s) of the inner table Access the outer table using efficient single table access Optionally sort the outer table into inner table join sequence Join the outer table with RIDs from the inner table index --> workfile Optionally sort the workfile into RID sequence (outer table data + inner table RIDs) Retrieve the inner table data with list prefetch Concatenate inner table data with outer table data 37 37

38 Hybrid Join Steps 1 R-scan 2 inner rid outer data outer data step3 is optional 4a list prefetch rid list 3 sort new & rids 5 concatenate 4b outer data inner rid composite 38 Better utilization of List Prefetch than Nested Loop Join Inner table is accessed once using List Prefetch, rather than once for each outer row. Outer table local predicates applied before the join/sort All indexable, stage 1 & 2 (including subqueries) are applied on the outer table before a composite sort (if required) and before the inner table is accessed Inner table predicates applied before/after join/sort All index matching predicates are applied as the inner table index is accessed, and before the sort if required. Non-index matching predicates are applied after data access (thus after sort). 38

39 Hybrid Join sorting In addition to SORTN_JOIN as seen on previous page... QQP M T H TNAME A T Y P M C O L A_NM IX O SORT NCCCC J UJOG EMP R 0 N DEPT I 1 DEPTX1 N NNNNN L P F No sort due to high clusterratio index. Inner table accessed with List Prefetch, without RID sort. No sort QQP M T H TNAME A T Y P M C O L A_NM IX O SORT NCCCC J UJOG EMP R 0 N DEPT I 1 DEPTX3 N YNYNN L P F Composite (outer) table sorted before inner table access. Sort on inner required also. SORTN_JOIN & SORTC_JOIN = 'Y' 39 39

40 Agenda Part A Session 1: Overview Session 2: Access path and explain table Session 3: Predicate application Session 4: Access methods Part B Session 5: Join methods Session 6: Query transformation Session 7: Statistics and cost estimation Session 8: Related optimization sessions 40 40

41 Query transformation This section introduces: Purpose of transformations Unlock more possible access path choices Allow cost model to estimate and choose most efficient Transformations Predicate transformations Join transformations View / table expression transformations Distribution and pruning 41 41

42 Predicate transformation In-list / between ==> equal OR ==> In-list Predicate transitive closure Predicate pushdown 42 PTC ************************************************************************************************ PTC not supported for all predicates. IN-LIST, LIKE COL IN (NON-CORRELATED SUBQ) Compound predicates Why these predicates not transformed 1. Like predicate can eliminate previous index only access 2. IN-LIST / compound predicates 1. Prepare cost to look in compound cases expensive 2. Can cause more SQLCODE -101 errors 3. COL-IN SUBQUERY SUBQUERY would be executed twice User action: Consider manually adding these predicates Will increase optimization oppurtunities New oppurtunity may be more efficient path Example Red predicates not currently transitively closed to T2. SELECT T1.* FROM T1, T2 WHERE T1.C1 = T2.C1 <-- Join predicates AND T1.C2 = T2.C2 <-- Join predicates AND T1.C1 IN (SELECT C1 FROM T3 WHERE T3.Cx =?) NO AND T1.C1 LIKE 'XX% NO AND T1.C1 IN ('A', 'B', 'C') NO AND (T1.C1 =? OR T1.C2 =?) NO 42

43 Join transformations Join type reduction Distribution and pruning Subquery to join transformation 43 43

44 Join type reduction Full outer join transformed to left outer join Full outer join can only use sort merge join Left outer join allows nested loop join also Uni-directional join operation typically more efficient Left / right outer join transformed to inner join Inner join allows all join sequences and join methods Opens up other transformation possibilities 44 44

45 Left to inner join reduction SELECT * FROM EMP E transform LEFT OUTER JOIN DEPT D ON E.DEPTNO = D.DEPTNO WHERE D.DIVISION = 'MARKETING' ; SELECT * FROM EMP E INNER JOIN DEPT D ON E.DEPTNO = D.DEPTNO WHERE D.DIVISION = 'MARKETING' ; The where clause predicate filters all the nulls from DEPT table, so DB2 determines the join type can be reduced to inner join Benefits Inner join allows alternative join sequences so DEPT table to be outer table. Can use index access via DEPT table Also opens up Hybrid join as possible join method Join type transformation can also reduce materialization 45 45

46 Full to left join reduction SELECT * FROM EMP E transform FULL OUTER JOIN DEPT D ON E.DEPTNO = D.DEPTNO WHERE E.EMP_TYPE <> 'MANAGER' ; SELECT * FROM EMP E LEFT OUTER JOIN DEPT D ON E.DEPTNO = D.DEPTNO WHERE E.EMP_TYPE <> 'MANAGER' ; The where clause predicate filters all the nulls from EMP table, so DB2 determines the join type can be reduced to LEFT OUTER JOIN. Benefits LEFT OUTER JOIN allows NESTED LOOP JOIN and SORT MERGE JOIN, FULL OUTER JOIN --> stuck with SMJ Uni-directional sort merge join typically more efficient than bi-directional sort merge join Can cascade to allow more transformations to EMP table 46 46

47 Subquery to join transformation example SELECT * transform FROM EMP WHERE DEPTNO IN ( SELECT DEPTNO FROM DEPT WHERE DIVISION = 'MARKETING' ) ; SELECT EMP.* FROM EMP, DEPT WHERE EMP.DEPTNO = DEPT.DEPTNO AND DEPT.DIVISION = 'MARKETING' ; Contains Unique index on (DIVISION, DEPTNO) --> Unique index guarantees no redundancy No local filtering provided on EMP table Benefits Can consider different join sequences such as DEPT table first using index and filtering on division Can consider different join methods which previously were not available 47 47

48 Query transformation V7 and V8 SELECT * SELECT EMP.* FROM EMP FROM EMP, DEPT WHERE EXISTS Transform WHERE EMP.DEPTNO = DEPT.DEPTNO ( SELECT * FROM DEPT AND DEPT.LOCALE = LA ; WHERE DEPT.DEPTNO = EMP.DEPTNO AND DEPT.LOCALE = LA ); Correlated subquery to join transformation A cost-independent decision Benefits Can consider different join sequences such as DEPT table first if it has good filtering from local predicate Can consider different join methods which previously were not available 48 Correlated subquery to join transformation is introduced in V7, For UPDATE/SELECT/DELETE with IN/EQANY/EXISTS. The conditions are very restricted. They are: 1. No outer join 2. No multiple subquery predicates connected by OR 3. No nested subquery predicates 4. No multiple tables in either parent or child query block 5. No GroupBy, Set Function, or Distinct in subquery 6. No self referencing UPDATE or DELETE 7. No table UDF 8. No residual join predicates between parent query block and child query block 9. No For-Update-Of Benefits: Depending on the table size and selectivity of each table, parent or child. 1. If parent (EMP) is a big table with no or not so good filtering, then it would be a disaster to evaluate subquery as correlated. 2. If child (DEPT) is a small table with good filtering, then by doing transformation, it allows optimizer to exploit the filtering and then joining to the big(emp) table with a a good filtering index on EMP.DEPTNO. 48

49 Query Transformation V9 Q1: Q2: SELECT * FROM BIG_TABLE B WHERE EXISTS ( SELECT * FROM SMALL_TABLE S WHERE B.C1 = S.C1); SELECT * FROM SMALL TABLE WHERE S.C1 IN ( SELECT B.C1 FROM BIG_TABLE B); For those not qualified for transformation before Now have a choice to be treated as either correlated or non-corrlated A cost-based decision Benefits Maximize the benefit of more possible join sequence and access methods 49 49

50 Agenda Part A Session 1: Overview Session 2: Access path and explain table Session 3: DB2 Runtime Architecture and predicate application Session 4: Access methods Part B Session 5: Join methods Session 6: Query transformation Session 7: Statistics and cost estimation Session 8: Related optimization sessions 50 50

51 Statistics Elapsed Time CPU Cost IO Cost Base Cost Row Cost Page Cost Scan Cost Filter Factor Statistics 51 51

52 Statistics for Cost Estimation Appendix #1: Statistics for Cost Estimation All statistics used by optimizer for access path selection Cardinality, high2key/low2key, frequency, histogram Collected by RUNSTATS on table, partition, index, column, colgroup 52 52

53 Agenda Part A Session 1: Overview Session 2: Access path and explain table Session 3: DB2 Runtime Architecture and predicate application Session 4: Access methods Part B Session 5: Join methods Session 6: Query transformation Session 7: Statistics and cost estimation Session 8: Related optimization sessions 53 53

54 Related optimizer sessions Terry Purcell G06: More Ways to Challenge the DB2 z/os Optimizer 54 54

55 Session: <G05> Fundamentals of DB2 Query Optimization Michelle Guo IBM, Silicon Valley Lab 55 55

56 Appendix #1: Statistics for Cost Estimation Table Statistics Index Statistics Selectivity Statistics Single-column Cardinality, high2key/low2key, frequency, histogram Multiple-column Cardinality, frequency, histogram 56 56

57 Table statistics (SYSIBM.SYSTABLES) CARDF Number of rows in a partition/table NACTIVEF Number of active pages for tablespace Only used for simple tablespace NPAGESF Number of pages where rows appear in a partition/table PCTROWCOMP Percentage of compressed rows 57 57

58 Index statistics (SYSIBM.SYSINDEXES) NLEAF Number of active leaf pages NLEVELS Number of levels in index tree CLUSTERRATIOF Percentage of rows in clustered order New algorithm in V9 DATAREPEATFACTORF Number of data pages accessed through a non filtering full index scan 58 58

59 Single column cardinality Single column cardinality is Number of distinct values of a column Stored as SYSCOLUMNS.COLCARDF SYSINDEXES.FIRSTKEYCARDF Used when Better statistics (frequency, histogram) is not available Better statistics is not applicable Searching with unknown values Combined with frequency statistics 59 59

60 High2key/Low2key Stored as SYSCOLUMNS.HIGH2KEY SYSCOLUMNS.LOW2KEY Used when Interpolation on Range, Like, Between predicates Searching with known values Interpolation is based on how much searching ranges overlaps full value range 60 60

61 Single-column frequency (SYSIBM.SYSCOLDIST)s Stored as SYSCOLDIST.(COLVALUE, FREQUENCYF) TYPE= F and NUMCOLUMNS=1 Provides important information on data skew Used when EQ, Range, Between, Like, INLIST, ISNULL, Searching with known values Or combined with other statistics such as cardinality, histogram, etc

62 Single-column histogram Stored as SYSCOLDIST.(FREQUENCYF, CARDF, LOWVALUE, HIGHVALUE) Type= H and NUMCOLUMNS=1 Describes how many distinct values exist between range [LOWVALUE, HIGHVALUE] and what percentage of rows it represents Provides data skew information on full value range Used when EQ, Range, Between, Like, INLIST, ISNULL, Searching with known values Or combined with frequency statistics 62 62

63 Multi-column cardinality (MCARD) Stored as SYSINDEXES.FULLKEYCARDF SYSCOLDIST.CARDF TYPE= C and NUMCOLUMNS>1 Provides importation information on column correlation Used when Compound filterfactor Subquery or Groupby size estimation 63 63

64 Multi-column frequency Similar to single column frequency Column values are concatenated Stored as SYSCOLDIST.FREQUENCYF TYPE= F and NUMCOLUMNS>1 Provides important information on correlated column data skew Used when Multiple EQ predicates searching with known values 64 64

65 Multi-column histogram Stored as SYSCOLDIST.(FREQUENCYF, CARDF, LOWVALUE, HIGHVALUE) Type= H and NUMCOLUMNS>1 Describes how many distinct concatenated values exist between range [LOWVALUE, HIGHVALUE] and what percentage of rows it represents Provides data skew information on full value range Used when Multiple EQ predicates searching with know values Or combined with multi-column frequency statistics 65 65

66 What to collect and when to collect? Optimization Service Center (OSC) Stats Advisor (SA) Optimization Expert (OE) Stats Advisor (SA) 66 66

67 Appendix #2: Plan Table columns VERSION COLLID APPLNAME PROGNAME QUERYNO QBLOCKNO Version identifier of a package - in embedded SQL Collection id for a package. VARCHAR(128) in V8. Name of application plan for static SQL Name of the program or package that contains the statement. VARCHAR(128) in V8. A number intended to identify the query being explained A number indicating a query or subquery block, showing the block s order in the SQL. Subqueries can be merged into one block by the optimizer. PLANNO Processing sequence of the step within QBLOCKNO; each new table accessed has a new step in the plan. TIMESTAMP When the EXPLAIN was executed 67 Combination of (QUERYNO, QBLOCKNO, PLANNO) uniquely identifies a plan (step in whole access path). 1. Each query may contains multiple query blocks. 2. Access path selection is usually done at each query block level (exception, V9 cross query block, global optimization). 3. Each plan is a step of joining/access a new table or a sort. 67

68 Plan Table columns Access of new table(1 of 4) CREATOR TNAME TABNO ACCESSTYPE I I1 M MX MI MU N R RW V blank Creator of the table accessed in the plan step. VARCHAR(128) in V8. Name of a table, created or declared temporary table, materialized view, table expression or outer join workfile accessed in this step. VARCHAR(128) in V8. Identifies the FROM clause table showing the position of reference in the SQL How the table is accessed Index scan One-fetch index scan Multi-index access. Always followed by MX, MI or MU Multi-index scan on referenced index Intersection of multiple indexes Union of multiple indexes IN list index scan Tablespace scan (Relational Scan) Workfile scan of the result of a materialized user-defined table function Buffers for an INSERT statement within a SELECT Not applicable to this row 68 68

69 Plan Table columns Access of new table(2 of 4) CORRELATION_NAME PAGE_RANGE TABLE_TYPE B C F M Q RB T W NULL Correlation name of the view/table specified in the statement. VARCHAR(128) in V8. Whether the table qualifies for page range screening The type of new table Buffers for INSERT statement within a SELECT Common Table Expression Table function Materialized Query Table Non-materialized temporary intermediate result table Recursive Common Table Expression Table Work file Implicit sort for GROUP BY, ORDER BY, or DISTINCT 69 69

70 Plan Table columns Access of new table(3 of 4) PRIMARY_ACCESS_TYPE D T blank TSLOCKMODE PREFETCH S L D blank COLUMN_FN_EVAL R S blank Indicates whether direct row access will be attempted DB2 will try to use direct row access Sparse index access DB2 will not try to use direct row access Show the tablespace lock mode LOCK is IS, IX, S, U, X, SIX, N (NS, NIS, NISS, SS) Shows which form of Prefetch is used Sequential Prefetch List Prefetch Optimizer cost assumes Dynamic Prefetch at runtime No prefetch or unknown Shows when an SQL column function was evaluated At data retrieval time At sort time At data manipulation or unknown 70 70

71 Plan Table columns Access of new table(4 of 4) ACCESSTYPE I I1 M MX MI MU N R RW T V blank ACCESSCREATOR ACCESSNAME INDEXONLY MATCHCOLS How the table is accessed Index scan One-fetch index scan Multi-index access. Always followed by MX, MI or MU Multi-index scan on referenced index Intersection of multiple indexes Union of multiple indexes IN list index scan Tablespace scan (Relational Scan) Workfile scan of the result of a materialized user-defined table function Sparse Index or In-memory Workfile Access Buffers for an INSERT statement within a SELECT Not applicable to this row Index Creator if ACCESSTYPE is I, I1, N, or MX. VARCHAR(128) in V8. Index Name if ACCESSTYPE is I, I1, N, MX. VARCHAR(128) in V8. Y/N for index access with no data reference (write to data pages for UPDATE is ignored by this flag) Number of matched columns in the INDEX key where ACCESSTYPE is I, I1, N, or MX 71 71

72 Plan Table columns Join Method METHOD Number showing the join method used in the plan MERGE_JOIN_COLS JOIN_TYPE F L S blank First table accessed, continuation of previous table or not used Nested loop join Sort Merge (Merge scan) join Additional sorts for ORDER BY, GROUP BY, DISTINCT, UNION etc. Hybrid join Number of columns joined using a Merge Scan Join (Method=2) The type of join Full Outer Join Left Outer Join (Optimizer converts Right Joins to Left Joins) Star Join Inner Join or no join 72 Check JOIN_TYPE for possible outer join reductions. 72

73 Plan Table columns Sorting new table SORTN_UNIQ Sort on new (inner) table to remove duplicates (not used) SORTN_JOIN SORTN_ORDERBY Sort on new (inner) table for join method 2 or 4. Only valid for method 1 during outside in phase of star join. Sort on new (inner) table for an ORDER BY (not used) SORTN_GROUPBY SORTC_UNIQ SORTC_JOIN SORTC_ORDERBY SORTC_GROUPBY Sort on new (inner) table for a GROUP BY (not used) Sort on composite (outer) table to remove duplicates Sort on composite (outer) table for a join method 1, 2 or 4 Sort on composite (outer) table for an ORDER BY or a quantified predicate Sort on composite (outer) table for a GROUP BY 73 73