DB2 10 for z/os Optimization and Query Performance Improvements

Transcription

1 DB2 10 for z/os Optimization and Query Performance Improvements James Guo DB2 for z/os Performance IBM Silicon Valley Lab August 11, PM 7 PM Session Number 9524

2 Disclaimer Copyright IBM Corporation All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE. IBM, the IBM logo, ibm.com, DB2 and z/os are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Other company, product, or service names may be trademarks or service marks of others. 2

3 Agenda Bind/Prepare Plan management Hints/Bind options Explain Dynamic Statement Caching REOPT Optimizer costing Runtime query performance Indexing Complex queries 3

4 Plan Management Overview Ability to backup your static SQL packages (DB2 9) At REBIND Save old copies of packages in Catalog/Directory Switch back to previous or original version Two flavors BASIC 2 copies: Current and Previous EXTENDED 3 copies: Current, Previous, Original Default controlled by a ZPARM Also supported as REBIND options 4

5 Plan Management - BASIC support REBIND PLANMGMT(BASIC) REBIND SWITCH(PREVIOUS) Incoming copy Current copy move move current copy move previous copy delete previous copy Chart is to be read from bottom to top 5

6 Plan Management - EXTENDED support REBIND PLANMGMT(EXTENDED) REBIND SWITCH(ORIGINAL) Incoming copy original copy clone clone current copy move current copy move original copy previous copy delete previous copy delete 6

7 DB2 10 Updates to Plan Management DB2 10 for z/os SYSIBM.SYSPACKCOPY New catalog table Hold SYSPACKAGE-style metadata for any previous or original package copies No longer need to SWITCH to see information on inactive copies Complaint from DB2 9 APRETAINDUP option of REBIND Default YES Retain duplicate for BASIC or EXTENDED Optional NO Do not retain duplicate access path as PREVIOUS or ORIGINAL PREVIOUS/ORIGINAL must be from DB2 9 or later 7

8 DB2 10 for z/os Access Path Stability with statement level hints Current limitations in hint matching QUERYNO is used to link queries to their hints a bit fragile For dynamic SQL, require a change to apps can be impractical New mechanisms: Associate query text with its corresponding hint more robust Hints enforced for the entire DB2 subsystem irrespective of static vs. dynamic, etc. Hints integrated into the access path repository PLAN_TABLE isn t going away Only the hint lookup mechanism is being improved. 8

9 Access Path Repository Hints/Statement level New SYSIBM tables SYSQUERY SYSQUERYOPTS SYSQUERYPLAN Options & Overrides REOPT, STARJOIN, etc. access path copy 1 access path copy N 9

10 DB2 10 for z/os Statement level hints (cont.) Steps to use new hints mechanism Populate a user table DSN_USERQUERY_TABLE with query text Populate PLAN_TABLE with the corresponding hints Run new command BIND QUERY To integrate the hint into the repository. FREE QUERY can be used to remove the hint. 10

11 Statement-level BIND options DB2 10 for z/os Statement-level granularity may be required rather than: Subsystem level ZPARMs (STARJOIN, SJTABLES, MAX_PAR_DEGREE) Package level BIND options (REOPT, DEF_CURR_DEGREE) For example Only one statement in the package needs REOPT(ALWAYS) New mechanism for statement-level bind options: Similar to mechanism used for hints DSN_USERQUERY_TABLE can also hold per-statement options 11

12 Literal Replacement DB2 10 for z/os Dynamic SQL with literals can now be re-used in the cache Literals replaced with & Similar to parameter markers but not the same To enable either you:- Put CONCENTRATE STATEMENTS WITH LITERALS in the PREPARE ATTRIBUTES clause Or set LITERALREPLACEMENT in the ODBC initialization file Or set the keyword enableliteralreplacement= YES in the JCC Driver Lookup Sequence Original SQL with literals is looked up in the cache If not found, literals are replaced and new SQL is looked up in the cache Additional match on literal usability Can only match with SQL stored with same attribute, not parameter marker If not found, new SQL is prepared and stored in the cache 12

13 DB2 10 for z/os Literal Replacement Example: WHERE ACCOUNT_NUMBER = This would be replaced by WHERE ACCOUNT_NUMBER = & Performance Expectation Using parameter marker still provides best performance Biggest performance gain for repeated SQL with different literals NOTE: Access path is not optimized for literals True for parameter markers/host variables today Need to use REOPT for that purpose 13

14 Agenda Bind/Prepare Optimizer costing RUNSTATS Cost model enhancements Subquery costing Runtime query performance Indexing Complex queries 14

15 DB2 10 for z/os Autonomic Statistics Solution Overview Autonomic Statistics is implemented though a set of Stored Procedures Stored procedures are provided to enable administration tools and packaged applications to automate statistics collection. ADMIN_UTL_MONITOR ADMIN_UTL_EXECUTE ADMIN_UTL_MODIFY Working together, these SP s Determine what stats to collect Determine when stats need to be collected Schedule and Perform the stats collection Records activity for later review See Chapter 11 "Designing DB2 statistics for performance" in the DB2 10 for z/os Performance Monitoring and Tuning Guide for details on how to configure autonomic monitoring directly within DB2. 15

16 RUNSTATS Simplification/Performance Overview DB2 10 for z/os RUNSTATS options to SET/UPDATE/USE a stats profile Integrate specialized statistics into generic RUNSTATS job RUNSTATS TABLE tbl COLUMN(C1) SET PROFILE Alternatively use SET PROFILE FROM EXISTING STATS RUNSTATS TABLE tbl COLUMN(C5) UPDATE PROFILE RUNSTATS TABLE tbl USE PROFILE New option for page-level sampling But what percentage of sampling to use? RUNSTATS TABLE tbl TABLESAMPLE SYSTEM AUTO 16

17 DB2 10 for z/os Optimizer Validation with Realtime Stats Index Probing & RTS lookup Estimate # of rids within a given start/stop index key range at bind/prepare Exploited when these two conditions are met. Query has matching index-access local predicate Predicate contain literals, or REOPT(ALWAYS ONCE AUTO) And 1 of the following is also true Predicate is estimated to qualify no rows Stats indicate the table contains no rows Table is defined as VOLATILE or qualifies for NPGTHRSH 17 New EXPLAIN table to externalize runtime estimates User managed DSN_COLDIST_TABLE

18 DB Minimizing Optimizer Challenges DB2 10 for z/os Potential causes of sub-optimal plans Insufficient statistics Unknown literal values used for host variables or parameter markers DB2 10 Optimizer will evaluate the risk for each predicate For example: WHERE BIRTHDATE <? Could qualify 0-100% of data depending on literal value used As part of access path selection Compare access paths with close cost and choose lowest risk plan 18

19 DB2 10 for z/os Extending VOLATILE TABLE usage VOLATILE TABLE support added in DB2 V8 Targeted to SAP Cluster Tables Use Index access whenever possible Avoids list prefetch Can be a problem for OR predicates or UPDATEs at risk of loop DB2 10 provides VOLATILE to general cases Tables matching SAP cluster tables will maintain original limitations Table with 1 unique index Tables with > 1 index will follow NPGTHRSH rules Use Index access whenever possible No limitation on list prefetch Less chance of getting r-scan when list-prefetch plan is only alternative 19

20 Agenda Bind/Prepare Optimizer costing Runtime query performance Sort/sort avoidance Sparse index Predicate application Indexing Complex queries 20

21 Sort Performance Enhancements DB2 9 & 10 for z/os FETCH FIRST n ROWS ONLY (FFnR) and Sort DB2 9 added in-memory replacement for FFnR to avoid sort Provided (n * (sort key + data)) < 32K DB2 10 extends this to 128K Avoid workfile usage for small sorts DB2 9 avoided allocating WF for final sort only If <= 255 rows and result < 32K (sort key + data) DB2 10 extends this to intermediate sorts also Except for parallelism or SET function 21

22 DB2 10 for z/os Improvements to predicate application Major enhancements to OR and IN predicates Improved performance for AND/OR combinations and long IN-lists General performance improvement to stage 1 predicate processing IN-list matching Matching on multiple IN-lists Transitive closure support for IN-list predicates List prefetch support Trim IN-lists from matching when preceding equals are highly filtering SQL pagination Single index matching for complex OR conditions Many stage 2 expressions to be executed at stage 1 Stage 2 expressions eligible for index screening Not applicable for list prefetch Externalized in DSN_FILTER_TABLE column PUSHDOWN 22

23 DB2 10 for z/os IN-list Table - Table Type 'I' and Access Type 'IN' The IN-list predicate will be represented as an in-memory table if: List prefetch is chosen, OR More than one IN-list is chosen as matching. The EXPLAIN output associated with the in-memory table will have: New Table Type: TBTYPE I New Access Type: ACTYPE IN SELECT * FROM T1 WHERE T1.C1 IN (?,?,?); QBNO PLANNO METHOD TNAME ACTYPE MC ACNAME QBTYPE TBTYPE PREFETCH DSNIN001(01) IN 0 SELECT I T1 I 1 T1_IX_C1 SELECT T L 23

24 IN-list Predicate Transitive Closure (PTC) DB2 10 for z/os SELECT * FROM T1, T2 WHERE T1.C1 = T2.C1 AND T1.C1 IN (?,?,?) AND T2.C1 IN (?,?,?) Optimizer can generate this predicate via PTC Without IN-list PTC (DB2 9) Optimizer will be unlikely to consider T2 is the first table accessed With IN-list PTC (DB2 10) Optimizer can choose to access T2 or T1 first. 24

25 DB2 10 for z/os SQL Pagination Targets 2 types of queries Cursor scrolling (pagination) SQL Retrieve next n rows Common in COBOL/CICS and any screen scrolling application Not to be confused with scrollable cursors Complex OR predicates against the same columns Common in SAP In both cases: The OR (disjunct) predicate refers to a single table only. Each OR predicate can be mapped to the same index. Each disjunct has at least one matching predicate. 25

26 Simple scrolling Index matching and ORDER BY Scroll forward to obtain the next 20 rows DB2 10 for z/os Assumes index is available on (LASTNAME, FIRSTNAME) WHERE clause may appear as: WHERE (LASTNAME='JONES' AND FIRSTNAME>'WENDY') OR (LASTNAME>'JONES') ORDER BY LASTNAME, FIRSTNAME; DB2 10 supports Single matching index access with sort avoided DB2 9 requires Multi-index access, list prefetch and sort OR, extra predicate (AND LASTNAME >= JONES ) for matching single index access and sort avoidance 26

27 DB2 10 for z/os Complex OR predicates against same index Given WHERE clause And index on one or both columns WHERE (LASTNAME= SMITH' AND FIRSTNAME= JOHN') OR (LASTNAME= JONES ); QBlockno Planno Accessname Access_Type Matchcols Mixopseq 1 1 IX1 NR IX1 NR

28 DB2 10 for z/os Minimizing impact of RID failure RID overflow can occur for Concurrent queries each consuming shared RID pool Single query requesting > 25% of table or hitting RID pool limit DB2 9 will fallback to tablespace scan* DB2 10 will continue by writing new RIDs to workfile Work-file usage may increase Mitigate by increasing RID pool size (default increased in DB2 10). MAXTEMPS_RID zparm for maximum WF usage for each RID list * Hybrid join can incrementally process. Dynamic Index ANDing will use WF for failover. 28

29 Agenda Bind/Prepare Optimizer costing Runtime query performance Indexing Index on expression Tracking index use Sparse index Include columns Complex queries 29

30 DB2 10 for z/os Index Include Columns Index INCLUDE columns Create an Index as UNIQUE, and add additional columns Ability to consolidate redundant indexes INDEX1 UNIQUE (C1) INDEX2 (C1,C2) Consolidate to INDEX1 UNIQUE (C1) INCLUDE (C2) 30

31 Agenda Bind/Prepare Optimizer costing Runtime query performance Indexing Complex queries Parallelism BI/DW 31

32 Parallelism Enhancements - Effectiveness DB2 10 for z/os Previous Releases of DB2 may use Key Range Partitioning Key Ranges Decided at Bind Time Based on Statistics (low2key, high2key, column cardinality) Assumes uniform data distribution Histograms can help But rarely collected If Statistics are outdated or data is not uniformly distributed what happens to performance? 32

33 Key range partition - Today SELECT * FROM Medium_T M, Large_T L WHERE M.C2 = L.C2 AND M.C1 BETWEEN (CURRENTDATE-90) AND CURRENTDATE DB2 10 for z/os Large_T 10,000,000 rows C2 C3 Medium_T 10,000 rows C1 C degree parallelism Workfile 25% SORT ON C ,500 rows 5,000,000 rows Partition the records according to the key ranges M.C1 is date column, assume currentdate is , after the between predicate is applied, only rows with date between and survived, but optimizer chops up the key ranges within the whole year after the records are sorted :-( 33

34 Parallelism Effectiveness Record range DB2 10 for z/os DB2 10 can use Dynamic record range partitioning Materialize the intermediate result in a sequence of join processes Results divided into ranges with equal number of records Division doesn't have to be on the key boundary Unless required for group by or distinct function Record range partitioning is dynamic no longer based on the key ranges decided at bind time Now based on number of composite records and parallel degree Data skew, out of date statistics etc. will not have any effect on performance 34

35 Dynamic record range partition SELECT * FROM Medium_T M, Large_T L WHERE M.C2 = L.C2 AND M.C1 BETWEEN (CURRENTDATE-90) AND CURRENTDATE DB2 10 for z/os Large_T 10,000,000 rows C2 C3 Medium_T 10,000 rows C1 C2 3-degrees parallelism Workfile SORT ON C2 2,500 rows Partition the records - each range has same number of records 35

36 DB2 10 for z/os Parallelism Effectiveness - Straw Model Previous releases of DB2 divide the number of keys or pages by the number representing the parallel degree One task is allocated per degree of parallelism The range is processed and the task ends Tasks may take different times to process DB2 10 can use the Straw Model workload distribution method More key or page ranges will be allocated than the number of parallel degrees The same number of tasks as before are allocated (same as degree) Once a task finishes it s smaller range it will process another range Even if data is skewed this new process should make processing faster 36