DB2 for z/os Optimizer: What have you done for me lately?

Transcription

1 Session: A08 DB2 for z/os Optimizer: What have you done for me lately? Terry Purcell IBM Silicon Valley Lab 14 th October :45 17:45 Platform: DB2 for z/os You can always read about the features/enhancements of a new release, such as DB2 9 for z/os, in the manuals, in redbooks or by attending highlight presentations. But what you generally miss from these sources is the answer to your question, "Why do I care about these enhancements?". And if there is one key DB2 focal area that needs extra explanation, it is the optimizer. This presentation will deep dive into some of the key SQL performance enhancements delivered recently in DB2 for z/os. 1

2 Introduction Most important to discuss Enhancements that require you to act OR Enhancements that provide a compelling reason to migrate or apply maintenance Main themes continue with each release Ongoing improvements to cost formulas Open up new access path choices Increase access path stability 2 2

3 Agenda V8 highlights Anything new for V8? What about DB2 9 enhancements in V8? DB2 9 highlights Any other stuff we don t tell you about? 3 3

4 V8 Optimizer Highlights RUNSTATS enhancements Frequency/Correlation on any column(s) Indexability for mismatched datatype/length Visual Explain With Statistics Advisor & Service SQL Page range screening enhancements Support for host vars and multi-column 4 4

5 V8 Optimizer Highlights MQTs Reduced prepare for complex statements Star Join enhancements In-memory index for materialized workfiles REOPT(ONCE) for dynamic SQL VOLATILE table support 5 5

6 V8 Optimizer Highlights Backward index scan VARCHAR index improvements Index-only access Variable length index keys Alter clustering index 6 6

7 V8 SQL Highlights 225 tables in a SQL statement 2MB statements GROUP BY expression ORDER BY within Singleton SELECT Multiple DISTINCT clauses SELECT from INSERT Common Table Expressions Recursive SQL Subquery in the SELECT list 7 7

8 Agenda V8 highlights Anything new for V8? What about DB2 9 enhancements in V8? DB2 9 highlights Any other stuff we don t tell you about? 8 8

9 Prune Always False Predicates V8 APAR PK49265 (ZPARM PREDPRUNE) Literal IN or = only (no host vars or REOPT) WHERE ('A' = 'B' OR T1.COL1 IN ('B', 'C')) Becomes. WHERE T1.COL1 IN ('B', 'C') Original OR is stage 2 Disables index access and many query transformations Documented tricks are NOT pruned (OR 0=1) 9 9

10 Index-only Preference Ever created an index to support index-only? Only to have optimizer choose index + data? SELECT C2 FROM T1 WHERE C1 =? Index 1 (C1) Index 2 (C1, C2) Index + data Index-only V8 APAR PK51734 ZPARM OPTIXOPREF Optimizer will prioritize index-only over index + data given same index prefix 10 10

11 Full index matching preference Optimizer already preferences 1 row index Full equals match on a unique index APAR PK59731 When FETCH FIRST 1 ROW ONLY is specified Will choose index with matching on all predicates WHERE C1 =? AND C2 =? AND C3 >? FETCH FIRST 1 ROW ONLY Index 1 (C1, C2, C3) Preference this over other indexes 11 11

12 Agenda V8 highlights Anything new for V8? What about DB2 9 enhancements in V8? DB2 9 highlights Any other stuff we don t tell you about? 12 12

13 Optimization Service Center DB2 for z/os Optimization Service Center (OSC) New no-charge offering Included in DB2 Accessories Suite for z/os V1.2 Available via web download DB2 9 in March 2007 DB2 V8 in April 2008 www-306.ibm.com/software/data/db2/zos/downloads/osc.html 13 13

14 VE OSC Feature comparison Functions Visual Explain Optimization Service Center Requires DB2 9? Queries from Cache, Catalog Yes Yes Query Formatter, Annotation Yes Access Plan Graph Visual Plan Hint Yes Yes Yes Query Statistics Advisor Yes Yes Workload Statistics Advisor Yes Profile based Monitoring Yes Yes 14 14

15 OSC Welcome Screen 15 15

16 What-If? Optimization Determine the optimizer impact of Creating or dropping indexes Without physically creating or dropping the index DB2 9 GA, retrofitted to V8 via APAR PK46687 When activated Will affect queries issuing EXPLAIN only Will NOT affect execution choice 16 16

17 What-If? Optimization Table DSN_VIRTUAL_INDEXES Table DSN_VIRTUAL_INDEX in V8 CM DDL is in SDSNSAMP(DSNTESC) Many columns similar to SYSINDEXES PLUS ENABLE (Y/N) to enable/disable the entry MODE (C/D) to model CREATE or DROP of the index COLNOn to list the col numbers of up to 64 index cols ORDERINGn (A/D) to list ordering of up to 64 index cols 17 17

18 What-If? Recommendation Beware of the documented recommendation: IBM s intention is that this feature will be used by tool vendors (including IBM)* 18 18

19 What-If? CREATE How would a tool test the optimizer effect? CREATE index AUSPTEST INSERT INTO DSN_VIRTUAL_INDEXES (TBCREATOR, TBNAME, IXCREATOR, IXNAME, ENABLE, MODE, UNIQUERULE, COLCOUNT, CLUSTERING, NLEAF, NLEVELS, INDEXTYPE, PGSIZE, FIRSTKEYCARDF, FULLKEYCARDF, CLUSTERRATIOF, PADDED, COLNO1, ORDERING1, COLNO2, ORDERING2, COLNO3, ORDERING3) VALUES ('SAPR3', 'AUSP', 'SAPR3', 'AUSPTEST', 'Y', 'C', 'D', 7, 'N', , 5, '2', 4, 1, , 0.95, 'N', 1, 'A', 6, 'A', 3, 'A'); Create 19 19

20 What-If? DROP How would a tool test the optimizer effect? DROP index AUSP~Z1 Populate detail from SYSIBM.SYSINDEXES INSERT INTO DSN_VIRTUAL_INDEXES (TBCREATOR, TBNAME, IXCREATOR, IXNAME, ENABLE, MODE, UNIQUERULE, COLCOUNT, CLUSTERING, NLEAF, NLEVELS, INDEXTYPE, PGSIZE, FIRSTKEYCARDF, FULLKEYCARDF, CLUSTERRATIOF, PADDED) SELECT TBCREATOR, TBNAME, CREATOR, NAME, 'Y', 'D', UNIQUERULE, COLCOUNT, CLUSTERING, NLEAF, NLEVELS, '2', 4, FIRSTKEYCARDF, FULLKEYCARDF, CLUSTERRATIOF, PADDED FROM SYSIBM.SYSINDEXES WHERE CREATOR = 'SAPR3' AND NAME = 'AUSP~Z1'; Drop 20 20

21 What If? Don t forget to.. Don t forget to DISABLE the What If? Choices Otherwise it will continue to impact all explains or UPDATE DSN_VIRTUAL_INDEXES SET ENABLE = N ; DELETE FROM DSN_VIRTUAL_INDEXES; 21 21

22 What If? Statistics Considerations Physically dropping indexes Removes statistics that can be attributed to that index Regardless if collected via the index or via COLGROUP Physically creating indexes Allows RUNSTATS to collect additional statistics What If? Optimization does not drop/create statistics 22 22

23 Agenda V8 highlights Anything new for V8? What about DB2 9 enhancements in V8? DB2 9 highlights Any other stuff we don t tell you about? 23 23

24 DB2 9 Optimizer Highlights Plan Stability Discussed in session A13 REOPT(AUTO) for dynamic SQL Discussed in session A13 Histogram Statistics Page range screening enhancements Global Optimization Generalized sparse index/in-memory WF 24 24

25 DB2 9 Optimizer Highlights Dynamic Index ANDing Index on expression Tracking index usage Optimization Service Center Sort performance and Sort avoidance Parallelism enhancements 25 25

26 DB2 9 SQL Highlights ORDER BY & FETCH FIRST in subq/table expr MERGE SELECT FROM MERGE/UPDATE/DELETE INTERSECT/EXCEPT SQL Operators OLAP Ranking functions RANK/DENSE_RANK/ROW_NUMBER 26 26

27 Dynamic Index Anding Pre-fact table filtering Filtering dimensions accessed concurrently Runtime optimization Terminate poorly filtering legs at runtime D1 More aggressive parallelism Fallback to workfile for RID pool failure Instead of r-scan D5 F D2 D4 D

28 Global Optimization V8 Limitations on subq-to-join transformations Subq costing is independent of outer table Whereas joins consider context of access DB2 9 Non-transformed subq are considered by optimizer In correlated and non-correlated forms Cross query block context considered Subqueries represented as virtual tables Thus table dependency can change 28 28

29 Generalized Sparse Index DB2 9 can create a sparse index at runtime To support join Previously only available for Star Join 29 29

30 How does Sparse Index work? Sparse index may be a subset of workfile (WF) Example, WF may have 10,000* entries Sparse index may have enough space (240K) for 1,000* entries Sparse index is binary searched to find target location of search key At most 10 WF entries are scanned NLJ Workfile sorted T 1 T 2 (WF) in t2.c order Binary Search of sparse index to look up approximate location of qualified key t1.c = t2.c Key RID Sparse Index sorted in t2.c order T 2 (WF) *Note: 1,000 & 10,000 are examples only 30 30

31 Data Caching vs Sparse Index Data Caching Also known as In-Memory WF Is a runtime enhancement to sparse index Sparse Index/In-Memory WF Extended to non-star join in DB2 9 New ZPARM MXDTCACH Maximum extent in MB, for data caching per thread If memory is insufficient Fall-back to sparse index at runtime 31 31

32 How does In-Memory WF work? Whereas sparse index may be a subset of WF IMWF contains the full result (not sparse) Example, WF may have 10,000 entries IMWF is binary searched to find target location of search key Binary Search of WF to look up exact location of qualified key T 1 NLJ t1.c = t2.c T 2 (WF) T 2 (WF) Workfile sorted in t2.c order 32 32

33 Index on Expression DB2 9 supports index on expression Can turn a stage 2 predicate into indexable SELECT * FROM CUSTOMERS WHERE YEAR(BIRTHDATE) = 1971 CREATE INDEX ADMF001.CUSTIX3 ON ADMF001.CUSTOMERS (YEAR(BIRTHDATE) ASC) Previous FF = 1/25 Now, RUNSTATS collects frequencies. Improved FF accuracy 33 33

34 Agenda V8 highlights Anything new for V8? What about DB2 9 enhancements in V8? DB2 9 highlights Any other stuff we don t tell you about? 34 34

35 Anything else we don t tell you about? Sure.. Every DB2 release has improvements to optimizer But most enhancements are too detailed for the interest of most customers, or involve the formulas which are proprietary Then, what else is interesting? 35 35

36 Predicate Localization Cross table OR predicates can be localized to each table V8 within Star Join context only DB2 9 for non-star join SELECT * FROM T1, T2 WHERE T1.C1 = T2.C1 AND ((T1.C2 = A AND T2.C3 > 5) OR (T1.C2 = A AND T2.C3 > 10)) Optimizer generates redundant BT predicates AND (T1.C2 = A ) Exact duplicates are consolidated AND (T2.C3 > 5 OR T2.C3 > 10) 36 36

37 Sort Avoidance Improvements Sort avoidance for GROUP BY Order of GROUP BY columns re-arranged to match index Data may be returned in a different order Relational theory states that order is NOT guaranteed without ORDER BY SELECT C2, C1 FROM T1 GROUP BY C2, C1 GROUP BY in C2, C1 sequence Index 1 (C1, C2) Index in C1, C2 sequence 37 37

38 Sort Improvements Reduced workfile usage for very small sorts Final sort step requiring 1 page will NOT allocate workfile More efficient sort with FETCH FIRST clause V8 and prior, Sort would continue to completion Then return only the requested n rows From V9, If the requested n rows will fit into a 32K page, As the data is scanned, Only the top n rows are kept in memory Order of the rows is tracked No requirement for final sort 38 38

39 FETCH FIRST V8 Example Sort is not avoided via index Must sort all qualified rows SELECT C1 FROM T ORDER BY C1 FETCH FIRST 3 ROWS ONLY C1 C Scan Fetch Sort row table. Who cares? But, 1 million rows?

40 Scan FETCH FIRST DB2 9 Example Sort is not avoided via index But in-memory swap avoids sort C Pointers maintain order Memory st Fetch 3 3 rd Fetch 2 2 nd Fetch SELECT C1 FROM T ORDER BY C1 FETCH FIRST 3 ROWS ONLY Suggestion: Add FETCH FIRST when subset is required

41 Parallelism Enhancements In V8 Lowest cost is BEFORE parallelism In DB2 9 Lowest cost is AFTER parallelism Only a subset of plans are considered for parallelism Optimizer Parallelism How to parallelize these plans? One Lowest cost plan survives 41 41

42 Clusterratio Enhancement New Clusterratio formula in DB2 9 Better awareness of prefetch range More accurate CR for lower cardinality indexes DB2 9 adds new statistic collected by RUNSTATS DATAREPEATFACTOR differentiates density and sequential Dense (and sequential) Sequential (not dense) Recommend RUNSTATS before mass REBIND in DB

43 Tracking Index Usage Additional indexes require overhead for Utilities REORG, RUNSTATS, LOAD etc Data maintenance INSERT, UPDATE, DELETE Disk storage Optimization time Increases optimizer s choices But identifying unused indexes is a difficult task Especially in a dynamic SQL environment 43 43

44 Tracking Index Usage RTS records the index last used date. SYSINDEXSPACESTATS.LASTUSED Updated once in a 24 hour period RTS service task updates at 1st externalization interval (set by STATSINT) after 12PM. if the index is used by DB2, update occurs. If the index was not used, no update. "Used", as defined by DB2 as: As an access path for query or fetch. For searched UPDATE / DELETE SQL statement. As a primary index for referential integrity. To support foreign key access 44 44

45 Session A08 DB2 for z/os Optimizer: What have you done for me lately? Terry Purcell IBM www-306.ibm.com/software/data/db2/zos/downloads/osc.html 45 45