What Developers must know about DB2 for z/os indexes

Transcription

1 CRISTIAN MOLARO What Developers must know about DB2 for z/os indexes Mardi 22 novembre 2016 Tour Europlaza, Paris-La Défense

2 What Developers must know about DB2 for z/os indexes Introduction Agenda 1 Introduction 2 DB2 indexes and access paths 3 Reorg avoidance 4 Performance topics 5 Summary Cristian Molaro - cristian@molaro.be / 42

3 What Developers must know about DB2 for z/os indexes Introduction About the author Cristian Molaro, MConsulting, Belgium Independent DB2 specialist Member of the IBM Gold Consultant program Recognized by IBM as an Information Champion since 2009 Recognized by IBM as "TOP" EMEA Consultant in 2011 / 2013 / 2016 Co-author of 9 IBM Redbooks related to DB2, including: DB2 10 for z/os Performance Topics DB2 11 for z/os Technical overview DB2 11 for z/os Performance Topics Holder of the merit badge Platinum IBM Redbook Author 12 books published about DB2 for z/os and DB2 for LUW Cristian Molaro - cristian@molaro.be / 42

4 What Developers must know about DB2 for z/os indexes Introduction Disclaimer PLEASE BE AWARE THAT THE ACTUAL PROGRAMMING TECHNIQUES, ALGORITHMS AND ALL NUMERICAL PARAMETERS USED IN EXAMPLES GIVEN IN THIS PRESENTATION ARE SUBJECT TO CHANGE AT SOME FUTURE DATE EITHER BY A NEW VERSION OF DB2, A NEW RELEASE, A SMALL PROGRAMMING ENHANCEMENT (SPE) OR A PROGRAMMING TEMPORARY FIX (PTF). THE INFORMATION CONTAINED IN THIS PRESENTATION HAS NOT BEEN SUBMITTED TO ANY FORMAL REVIEW AND IS DISTRIBUTED ON AN AS IS BASIS WITHOUT ANY WARRANTY EITHER EXPRESS OR IMPLIED. THE USE OF THIS INFORMATION OR THE IMPLEMENTATION OF ANY OF THESE TECHNIQUES IS A CUSTOMER RESPONSIBILITY AND DEPENDS ON THE CUSTOMER S ABILITY TO EVALUATE AND INTEGRATE THEM INTO THE CUSTOMER S OPERATIONAL ENVIRONMENT. WHILE EACH ITEM MAY HAVE BEEN REVIEWED FOR ACCURACY IN A SPECIFIC SITUATION, THERE IS NO GUARANTEE THAT THE SAME OR SIMILAR RESULTS WILL BE OBTAINED ELSEWHERE. CUSTOMERS ATTEMPTING TO ADAPT THESE TECHNIQUES TO THEIR OWN ENVIRONMENTS DO SO AT THEIR OWN RISK. DB2 IS A TRADEMARK OF INTERNATIONAL BUSINESS MACHINE CORPORATION. THIS PRESENTATION USES MANY TERMS THAT ARE TRADEMARKS. WHEREVER WE ARE AWARE OF TRADEMARKS THE NAME HAS BEEN SPELLED IN CAPITALS. Cristian Molaro - cristian@molaro.be / 42

5 What Developers must know about DB2 for z/os indexes Introduction Agenda 1 Introduction 2 DB2 indexes and access paths 3 Reorg avoidance 4 Performance topics 5 Summary Cristian Molaro - cristian@molaro.be / 42

6 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths DB2 indexes An index is an ordered set of pointers to rows of a table B-tree data structure DB2 10 introduces HASH access as an index access alternative IMPORTANT: Indexes have both advantages and disavantages Cristian Molaro - cristian@molaro.be / 42

7 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths DB2 indexes considerations Advantages of indexes Required for uniqueness Partitioning Alternativelly: Table controlled partitioning Can improve access paths performance Read fewer pages/rows Can avoid data sort Used for physical clustering sequence of table data Some inconvenients More DASD space Utility overhead: LOAD, REORG, RECOVER OVERHEAD BIND overhead INDEX maintenance cost INSERTS, DELETES, UPDATE (possible) RID SORTS RIDPOOL FAILURES Cristian Molaro - cristian@molaro.be / 42

8 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths The perfect index 1 Optimal matching columns 2 In right sequence to allow to avoid sort Sort considered for ORDER BY DISTINCT GROUP BY JOIN UNION (BUT NOT UNION ALL) EXCEPT & INTERSECT 3 With enough columns to allow index only access Good starting point for index design Sometimes it is not possible to create the perfect index Key length Number of columns Points 1 and 2 in conflict Cristian Molaro - cristian@molaro.be / 42

9 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths DB2 Access methods at a glance DATA SCAN through data Always FULL scan, minimum scope is a partition Always uses sequential prefetch Index (RID Needed) Single row Unique KEY = RID Multiple rows Matching scan (= partial scan) Non-matching scan (= full scan) Access can be index only = all data is in index Stage 1 predicates: Fast and low CPU & Could be indexable Stage 2 predicates: Slow and high CPU & Can not be indexable Cristian Molaro - cristian@molaro.be / 42

10 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths Data Access Via Clustered Index Matching Index Scan Non-matching Index Scan IN-List Index Access Multiple Index Access One-Fetch Access Index-Only Access Equal Unique Index Access Clustered data allows efficient I/O using sequential prefetch Cristian Molaro - cristian@molaro.be / 42

11 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths Data Access Via Unclustered Index Poorly clustered data means each I/O is random, and the same page may be accessed many times Normally only the clustering index has a high cluster ratio IMPORTANT Always explicitly define a CLUSTERING index on every table Cristian Molaro - cristian@molaro.be / 42

12 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths Data Access using List Prefetch 1 RID retrieval: RIDs for data pages is found by matching index scans 2 RID sort: RID list is sorted by page # 3 Data retrieval: The needed data pages are prefetched in order List prefetch allows Sequential access of random data Cristian Molaro - cristian@molaro.be / 42

13 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths RID failures RID pool used for record identifier (RID) processing Enforcing unique keys for multi-row updates List prefetch, including single index list prefetch access paths Multiple index access paths Hybrid joins All concurrent work shares the RID pool MAXRBLK controls the maximum size of the RID pool Limit is 10,000 MB. RID overflow scenarios Concurrent queries each consuming shared RID pool Single query requesting > 25% of table or hitting RID pool limit DB2 9 will fallback to Tablespace Scan DB2 10 will continue by writing new RIDs to workfile Avoid the performance problems of falling back to TS-Scan Has no effect on access path selection Cristian Molaro - cristian@molaro.be / 42

14 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths RID failures in DB2 reports STATISTICS report ACCOUNTING REPORT LONG Cristian Molaro - cristian@molaro.be / 42

15 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths Fixing the problem Often caused by: Statistics inadequacy Use of LIKE, range predicates with HOST variables Often solved by Runstats + REBIND Cristian Molaro - cristian@molaro.be / 42

16 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths Index to Data Access Path vs. Hash Access Performance benefits : Up to 30% DB2 CPU reduction with random access Savings in index maintenance once removed clustering index Performance concerns Not for sequential FETCH nor INSERT Significant Sync I/O increase if accessed in clustering order Cristian Molaro - cristian@molaro.be / 42

17 What Developers must know about DB2 for z/os indexes DB2 indexes and access paths Hash Access Candidates Candidate tables For queries needing single row access via the unique key Queries having equal predicates on keys With known and static approximate size of data Having large N-level indexes Not for tables Whenever there is sequential processing Frequently updated Either using BETWEEN or > and < Carefully check EXPLAIN + PLAN_TABLE to check access path SYSTABLESPACESTATS.REORGHASHACCESS # of times data is read using hash access in the last interval SYSIBM.SYSINDEXSPACESTATS Check LASTUSED & REORGINDEXACCESS for overflow Cristian Molaro - cristian@molaro.be / 42

18 What Developers must know about DB2 for z/os indexes Reorg avoidance Agenda 1 Introduction 2 DB2 indexes and access paths 3 Reorg avoidance 4 Performance topics 5 Summary Cristian Molaro - cristian@molaro.be / 42

19 What Developers must know about DB2 for z/os indexes Reorg avoidance Concerns about REORG in Production environments Common reasons for REORG Better performance High CPU usage due to space searching High get page during the insert/update/query transaction High I/O accessing data is disorganized Space reclaim Materialize pending changes Common problems caused by REORG Resource utilization Timeouts (better in DB2 11) Work-space issues (specially with big partitions) Can avoid REORGS? Probably NOT 100% But REORG avoidance / delay would be nice! Cristian Molaro - cristian@molaro.be / 42

20 What Developers must know about DB2 for z/os indexes Reorg avoidance DB2 10 Improvement in using Disorganized Index Index scan using disorganized index causes high sync I/O wait Disorganized index detection at execution Use List Prefetch on index leaf pages with range scan Reduce Synchronous I/O waits for queries accessing disorganized indexes Reduce the need of REORG Index Throughput improvement in Reorg, Runstats, Check Index Performance results Observed 2 to 6 times faster with simple SQL statements with small key size using list prefetch compared to Sync I/Os Cristian Molaro - cristian@molaro.be / 42

21 What Developers must know about DB2 for z/os indexes Reorg avoidance Row Level Sequential Detection Dynamic prefetch sequential works poorly when the number of rows per page is large Row Level Sequential Detection (RLSD) Count rows, not pages to track the sequential detection Side effect: DB2 10 will trigger prefetch more quickly To compensate, it uses progressive prefetch quantity I.e. with 4K pages: The first prefetch I/O reads 8 pages Then 16 pages All subsequent I/Os will prefetch 32 pages (as previously) Also applies to indexes Row level sequential detection (RLSD) preserves good sequential performance for the clustered pages Cristian Molaro - cristian@molaro.be / 42

22 What Developers must know about DB2 for z/os indexes Reorg avoidance Pseudo deleted index keys and pages When a row is deleted: The index entry for the key to the row must be removed in each index DB2 cannot immediately delete the index entry, unless it has exclusive access control DB2 sets a bit in the index to mark the index entry as being pseudo-deleted Pseudo-empty index pages are pages that contain only pseudo-deleted index entries Pseudo-deleted index entries occupy space in the index SQL index access performance may degrade over time Performance impact of index pseudo delete entries SELECT, FETCH, UPDATE or DELETE: more getpages INSERT/UPDATE/DELETE may see concurrency issue Collisions with committed pseudo-deleted index entries RID reuse by INSERT following DELETE could cause deadlock Frequent execution of REORG INDEX utility to reduce the impact Cristian Molaro - cristian@molaro.be / 42

23 What Developers must know about DB2 for z/os indexes Reorg avoidance Cleanup of Index Pseudo Deleted Pages and Entries DB2 11 provides automatic cleanup of pseudo deleted entries Clean up both pseudo empty index pages and pseudo deleted index entries Asynchronous service tasks to cleanup committed pseudo-deleted index entries The service task overhead is not associated with any DELETE or UPDATE activity Cleanup process only occurs when the index is being updated by DB2 Performance benefits Consistent performance for index access over time Reducing the need of REORG INDEX utility Potential concerns on automatic clean up CPU overhead ziip eligible under DBM1 address space Disruption to other concurrent threads Control through zparm INDEX_CLEANUP_THREADS Control through SYSIBM.SYSINDEXCLEANUP Cristian Molaro - cristian@molaro.be / 42

24 What Developers must know about DB2 for z/os indexes Reorg avoidance Performance observations Cristian Molaro - cristian@molaro.be / 42

25 What Developers must know about DB2 for z/os indexes Reorg avoidance Controlling the cleanup processes Monitoring IFCID 377: tracks cleanup activity INDEX_CLEANUP_THREADS subsystem parameter Number of threads allowed to work on the cleanup of pseudo deleted index entries Value range from 0 to 128. Default is 10 Set to 0 to disable SYSIBM.SYSINDEXCLEANUP catalog table Specify time windows to enable or disable index cleanup For the entire subsystem For indexes in specific databases or for specific indexes The performance of index cleanup processing might be reduced if the number of rows in this catalog table becomes too large Recommendation: Use rows in the SYSIBM.SYSINDEXCLEANUP for exceptions only when the default index cleanup behavior is unacceptable Cristian Molaro - cristian@molaro.be / 42

26 What Developers must know about DB2 for z/os indexes Performance topics Agenda 1 Introduction 2 DB2 indexes and access paths 3 Reorg avoidance 4 Performance topics 5 Summary Cristian Molaro - cristian@molaro.be / 42

27 What Developers must know about DB2 for z/os indexes Performance topics Predicate indexability enhancements in DB2 11 Stage 1 predicates: Fast and low CPU & Could be indexable Stage 2 predicates: Slow and high CPU & Can not be indexable DB2 11 Query transformation improvements Promote some common STAGE 2 predicates to indexable DB2 11 enhancements Example Improved indexability for OR COL IS NULL Enhanced pruning of "always true" and "always false" predicates YEAR(COLx) =? DATE(COLy) =? SUBSTR(COLz,1,x) =? VALUE BETWEEN COLa AND COLb TIP: Please be kind, and REBIND Cristian Molaro - cristian@molaro.be / 42

28 What Developers must know about DB2 for z/os indexes Performance topics What performance feature shows after REBIND? Feature General code optimizations Customized code generation for sort Access against table with large # of partitions REL(DEALLOCATE) optimization DB2 latch reductions Buffer Pool enhancements More ziip processing DDF Sync Receive enhancements Data sharing performance improvements Customized code generation for column processing Decompression Improvement SQL Access Path improvements Trigger None None None None None None None (enough ziip capacity) None (TCP/IP APAR) None REBIND REBIND REBIND Cristian Molaro - cristian@molaro.be / 42

29 What Developers must know about DB2 for z/os indexes Performance topics What if? indexes WHAT-IF analysis DB2 has the ability to do explain with virtual indexes One additional table needs to be created for the Explain feature VIRTUAL index will show up in PLAN_TABLE if it can be used Cristian Molaro - cristian@molaro.be / 42

30 What Developers must know about DB2 for z/os indexes Performance topics DROP never-used indexes Overhead Space used SQL / Utility overhead due to maintenance Optimization time: increases optimizer s choices SYSINDEXSPACESTATS.LASTUSED records last used date Updated only once in a 24 hour period Make sure you shutdown DB2 NORMAL IMPORTANT: Be careful before dropping STATS collector indexes Used" as defined by DB2 means: As an access path for query or fetch For searched UPDATE / DELETE SQL statement As a primary index for referential integrity To support foreign key access Cristian Molaro - cristian@molaro.be / 42

31 What Developers must know about DB2 for z/os indexes Performance topics Table-controlled partitioning Table-controlled partitioning does not require an index for partitioning Can convert existing tables Allows to drop partitioning index Use PARTITION in CREATE TABLE DDL example Cristian Molaro - cristian@molaro.be / 42

32 What Developers must know about DB2 for z/os indexes Performance topics Index on Expression DB2 supports index on expression Index key values are transformed by an expression Uniqueness is enforced against the values that are stored in the index Can turn a stage 2 predicate into indexable Query example Index DDL Cristian Molaro - cristian@molaro.be / 42

33 What Developers must know about DB2 for z/os indexes Performance topics Additional Non-key Columns in a Unique Index DB2 9 definition Possible DB2 10 definition or INCLUDE columns are not allowed in non-unique indexes Indexes on Expression do not support INCLUDE columns Cristian Molaro - cristian@molaro.be / 42

34 What Developers must know about DB2 for z/os indexes Performance topics Insert Performance with INCLUDE index Potential performance improvement by reduction of number of indexes Check RTS and LASTUSED for deciding what INDEX to drop Cristian Molaro - cristian@molaro.be / 42

35 What Developers must know about DB2 for z/os indexes Performance topics Exclude Null Index DB2 will not create an index entry for key columns with NULL Effective on indexes with large numbers of NULL Support in CREATE and ALTER INDEX Potential cost reduction on index maintenance Index update Utilities maintaining indexes DDL CREATE INDEX Disk storage Potential fetch performance improvement EXCLUDE NULL KEYS index cannot be used whenever a NULL key could satisfy a predicate predicate Cristian Molaro - cristian@molaro.be / 42

36 What Developers must know about DB2 for z/os indexes Performance topics Large index page size You have the choice to use index pages > 4K Available in DB2 9 NFM Introduced to support index compression NOTE: Can be used independently of index compression Advantages Potential to reduce the number of index leaf page splits Big problem in Data Sharing environments Reduce index tree lotch contention Reduce index tree p-lock contention Potential to reduce the number of index levels Reduce the number of getpages for index traversal Reduce CPU resource consumption BUT: large index page may require increasing index bufferpool size Cristian Molaro - cristian@molaro.be / 42

37 What Developers must know about DB2 for z/os indexes Performance topics Index compression Designed for reduction of index disk space utilization Indexes can consume more DASD space than table spaces Trade off disk for CPU and memory NO hardware supported Compression of data ONLY on disk IMPORTANT: Index compression is NOT a performance feature Cristian Molaro - cristian@molaro.be / 42

38 What Developers must know about DB2 for z/os indexes Performance topics Index compression The compression ratio depends on the data and how the columns are distributed in the index The choice of the buffer pool determines the maximum disk space savings and the efficiency of memory utilization Cristian Molaro - cristian@molaro.be / 42

39 What Developers must know about DB2 for z/os indexes Performance topics Index compression: planning DSN1COMP supports INDEX COMPRESSION Cristian Molaro - cristian@molaro.be / 42

40 What Developers must know about DB2 for z/os indexes Summary This was our Agenda 1 Introduction 2 DB2 indexes and access paths 3 Reorg avoidance 4 Performance topics 5 Summary Cristian Molaro - cristian@molaro.be / 42

41 Any questions?

42