An Index's Guide to the Universe

Transcription

1 An Index's Guide to the Universe Brad Price Werner Enterprises Session Code: D12 Thu, May 02, 2013 (03:30 PM -04:30 PM) Platform: DB2 LUW An Index's Guide to the Universe Brad Price Werner Enterprises Session Code: D12 Thu, May 02, 2013 (03:30 PM -04:30 PM) Platform: DB2 LUW

2 3 Agenda Indexes 101 -the basics Important Index info in the catalog Index considerations when tuning Special Commands related to Indexes Index Potpourri for $500, Alex 4 Why Do We Need Indexes?? Performance improvement Reducing I/O to find desired rows Avoiding sorts Enforce Uniqueness Support a Primary Key definition Make the DBA look like a hero!! If you don t find it in the index, look very carefully through the entire catalogue. - Sears, Roebuck and Co., Consumer s Guide, 1897

3 5 What Does an Index Look Like? B-tree structure Leaf pages and non-leaf pages One or more columns An index with more than one column is called a composite index Unique or non-unique 6 How Does an Index Work?? Navigate the b-tree structure to the appropriate leaf pages The leaf page stores the RID (Row Identifier) that points to the physical location in the table for the row The DBMS then requests a read of the pagecontaining the requested row(s) of data (Pssst this isn t free!!!)

4 7 B+ Tree Structure (unique IX) E N Z Root page(s) B E F L N T U Z Non-leaf pages F,rid G,rid I,rid K,rid L,rid M,rid N,rid Leaf pages 8 What Makes a Good Index?? Columns that have high cardinality What is the cardinality of a flag column with Y and N? From left to right, build the index with: 1. Columns used in local equal predicates or Columns used as join predicates (if this is inner table of join) Where city = OMAHA Where a.city = b.city 2. Columns used as local range predicates Where age > Columns used as join to other tables (if this is the outer tableof join) Where a.city = b.city 4. Columns in the select clause Index-only access

5 9 What Makes a Good Index?? Select col_a, col_b, col_c From edw.tractor_history Where pk_tractor_nb = and pk_tractor_change_ts > Two possible indexes pk_tractor_nb, pk_tractor_change_ts desc pk_tractor_nb, pk_tractor_change_ts desc, col_a, col_b, col_c What about this index? col_a, col_b, col_c, pk_tractor_nb, pk_tractor_change_ts desc 10 Index Only access If all of the columns referenced in a query are in the index there is no need to access the data portion of the table. Very fast read performance Increasing penalty for insert/update/delete operations as more columns are added Indexes can become very large and less desirable for the dbms to use Index Only can break if additional columns are added to query Use this sparingly.

6 11 What Makes a Good Index?? Select col_a, col_b, col_c,col_d From edw.tractor_history Where pk_tractor_nb = and pk_tractor_change_ts > Two possible indexes pk_tractor_nb, pk_tractor_change_ts desc pk_tractor_nb, pk_tractor_change_ts desc, col_a, col_b, col_c Do we still have index only access? 12 Can an Index Have Too Many Columns?? YES!! Makes an index much bigger and less likely to be used by other queries Brad s rule-of-thumb is start with three columns or less If a query is super-critical and runs frequently, then consider additional columns to facilitate index only access

7 13 Can a Table Have Too Many Indexes?? YES!! Every index must be updated whenever there is an Insert/Delete Updates to columns in indexes require updating both table and index Indexes are included in backups Indexes are rebuilt during loads/reorgs Indexes are scanned during DB2 Runstats operations Use with sampled detailed indexes to reduce index I/O on Runstats 14 What Is The Right Number of Indexes for a Table?? In OLTP, where access patterns are somewhat predictable, most tables should do quite well with 5 indexes or less Exceptions abound however In Data Warehouse, where the access patterns are much more varied, you can see larger numbers of indexes How to identify and remove unused indexes Monitor index scans using db2pd tcbstats index or mon_get_index Lastused column in syscat.indexes How long should something be unused before you get rid of it????

8 15 What is a Clustering Index?? One, and only one, index that defines the physical order of rowson a table DBMS attemptsto insert new rows based upon the clustering index Over time, clustering degrades and requires a reorg to re-order the rows Significant performance improvements for queries that need groups of similar rows or that process rows in a certain order i.e. all rows for a given date range 16 Regular non-clustering indexes Index On Region Table Index on Year

9 Clustering Index 17 Clustering Index On Region Table Index on Year 18 Why are reads faster when a table is clustered?? The first I/O reads a page into memory which contains many rows with the same key or a range of key values Example: App needs 500 rows for a given region. If the DBMS knows that it will need to fetch several or many consecutive pages, then it can begin prefetching extents (multiple pages) into memory before application needs it 18 IOs vs 3 IOs vs 1 IO

10 19 Agenda Indexes 101 -the basics Important Index info in the catalog Index considerations when tuning Special Commands related to Indexes Index PotPourri for $500, Alex 20 Important Index Info in the Catalog select substr(indname,1,20) indname, substr(colnames,1,50) colnames, firstkeycard, fullkeycard from syscat.indexes where tabname = 'TRACTOR_HISTORY'; INDNAME COLNAMES FIRSTKEYCARD FULLKEYCARD TRC_HIST_IX1 -DSTAGE_IDENTITY_NB TRACTOR_HIST_IX1 +TRACTOR_HISTORY_TYPE_CD+PK_TRACTOR_CHANGE_TS+COST TRACTOR_HIST_IX2 +PK_TRACTOR_CHANGE_TS+PK_TRACTOR_NB+TRACTOR_HISTOR TRACTOR_HIST_IX3 +PK_TRACTOR_CHANGE_TS+DRIVER_EMPLOYEE_NB TRACTOR_HIST_IX4 +PK_TRACTOR_CHANGE_TS+TRACTOR_DRIVER2_NB TRACTOR_HIST_IX5 +PK_TRACTOR_NB-PK_TRACTOR_CHANGE_TS+TRACTOR_HISTOR TRACTOR_HIST_IX6 +TRACTOR_DRIVER2_NB+PK_TRACTOR_CHANGE_TS TRACTOR_HIST_IX7 +DRIVER_EMPLOYEE_NB+PK_TRACTOR_CHANGE_TS Some more columns: FULLKEYCARD CLUSTERRATIO CLUSTERFACTOR SEQUENTIAL_PAGES CREATE_TIME STATS_TIME REVERSE_SCANS DENSITY IID NLEAF NLEVELS FIRSTKEYCARD FIRST2KEYCARD FIRST3KEYCARD FIRST4KEYCARD

11 21 Index Cardinality Columns in the Catalog select substr(indname,1,20) indname, firstkeycard, first2keycard, first3keycard, first4keycard, fullkeycard from syscat.indexes where tabname = 'TRACTOR_HISTORY'; INDNAME FIRSTKEYCARD FIRST2KEYCARD FIRST3KEYCARD FIRST4KEYCARD FULLKEYCARD TRACTOR_HIST_IX TRACTOR_HIST_IX TRACTOR_HIST_IX TRACTOR_HIST_IX TRACTOR_HIST_IX TRACTOR_HIST_IX TRACTOR_HIST_IX TRACTOR_HIST_IX TRC_HIST_IX Density is Important If Density is low Tend to have more physical index IO Indicates a need to reorg index Procedure sysproc.reorgchk_ix_stats (and tb_stats) CALL SYSPROC.REORGCHK_IX_STATS('T','MAPW.ADDRESS')

12 23 Agenda Indexes 101 -the basics Important Index info in the catalog Index considerations when tuning Special Commands related to Indexes Index PotPourri for $500, Alex 24 Monitor Logical Data Reads and Logical Index Reads to Look for Indexing Opportunities Use db2 get snapshot for application or db2batch with -o p 3 If Logical Data Reads are high track down which tables are causing the high reads and design appropriate index If Logical Index Reads are high, look for high IO cost on Explain Could be caused by a non-matching index scan Jump Scans will help on this in v10 Can also be caused by an inappropriate NLJoin doing high numbersof index seeks Buffer pool data logical reads = Buffer pool data physical reads = Buffer pool temporary data logical reads = 5800 Buffer pool temporary data physical reads = 9 Buffer pool data writes = 0 Buffer pool index logical reads = Buffer pool index physical reads = 20693

13 25 Is This a Good Query? select * from edw.freight_bill where billing_invoice_nb = ' ' Returns 112 rows Select colnames from syscat.indexes where tabname = 'FREIGHT_BILL' COLNAMES PK_FREIGHT_BILL_NB+PK_TRIP_ID -DSTAGE_IDENTITY_NB +RECORD_UPDATE_TS +PK_TRIP_ID+RECORD_DELETE_FG+AMOUNT_PAID_ON_BILL_AM+TOTAL_CHARGE_AM+PK_FREIGHT_BILL_NB +RECORD_DELETE_FG-BILLING_INVOICE_NB 26 Poor Man s Way to Find Hot Tables Needing Indexes Works best on a low activity system Great for finding problem tables in monster queries From clp: 1. db2 reset monitor all 2. Start query 3. Repeatedly issue: db2 get snapshot for tables on db grep e Table Name e Rows Read Or: 1. Select tabname, rows_read from TABLE(MON_GET_TABLE('','',-2)) AS t where tabname in ( TAB1, TAB2, ) 2. Run query The tables in your query 3. Run step1 again and compare differences of rows_read

14 27 Poor Man s Hot Table example db2 reset monitor all Start query db2 get snapshot for tables on db grep e Table Name e Rows Read Wait a few seconds Table Name = EMPLOYEE_CHECK_DETAIL Rows Read = 1023 Table Name = EMPLOYEE_CHECK_HEADER Rows Read = 6 Table Name = TEMP (00027,00006) Rows Read = 221 Table Name = DRIVER_RETENTION_PAY Rows Read = db2 get snapshot for tables on db grep e Table Name e Rows Read Table Name = EMPLOYEE_CHECK_DETAIL Rows Read = Table Name = EMPLOYEE_CHECK_HEADER Rows Read = Table Name = VERSIONS Rows Read = 8 Table Name = LINEAGES_MODIFIED Rows Read = 4 Table Name = CALENDAR_DATE Rows Read = 1333 Table Name = DRIVER_RETENTION_PAY Rows Read = Evolution of tuning a query Indexes for our table: INDNAME COLNAMES SQL PK_EVENT_NB EVENT_IX1 -DSTAGE_IDENTITY_NB SELECT event_start_type_cd, event_end_type_cd, record_update_ts FROM brad.event WHERE event_start_type_cd = 'USPPC'

15 29 Evolution of tuning a query Let s create a new index: CREATE INDEX brad.event_ix2 ON brad.event ( event_start_type_cd asc, event_end_type_cd asc) SELECT event_start_type_cd, event_end_type_cd, record_update_ts FROM brad.event WHERE event_start_type_cd = 'USPPC' COMPRESS NO ALLOW REVERSE SCANS; Indexes for our table: INDNAME COLNAMES SQL PK_EVENT_NB EVENT_IX1 -DSTAGE_IDENTITY_NB EVENT_IX2 +EVENT_START_TYPE_CD+EVENT_END_TYPE_CD 30 Evolution of tuning a query Indexes for our table: INDNAME COLNAMES SQL PK_EVENT_NB EVENT_IX1 -DSTAGE_IDENTITY_NB EVENT_IX2 +EVENT_START_TYPE_CD+EVENT_END_TYPE_CD Old Plan SELECT event_start_type_cd, event_end_type_cd, record_update_ts FROM brad.event WHERE event_start_type_cd = 'USPPC' New Plan What do these numbers mean? Estimated number of I/Os (These are from Control Center and I changed the metric)

16 31 Evolution of tuning a query Indexes for our table: INDNAME COLNAMES SQL PK_EVENT_NB EVENT_IX1 -DSTAGE_IDENTITY_NB EVENT_IX2 +EVENT_START_TYPE_CD+EVENT_END_TYPE_CD Old Plan SELECT event_start_type_cd, event_end_type_cd, record_update_ts FROM brad.event WHERE event_end_type_cd = 'PETOF event_start_type_cd = 'USPPC' New Plan 32 Evolution of tuning a query Indexes for our table: INDNAME COLNAMES SQL PK_EVENT_NB EVENT_IX1 -DSTAGE_IDENTITY_NB EVENT_IX2 +EVENT_START_TYPE_CD+EVENT_END_TYPE_CD EVENT_IX3 +EVENT_END_TYPE_CD+EVENT_START_TYPE_CD Old Plan SELECT event_start_type_cd, event_end_type_cd, record_update_ts FROM brad.event WHERE event_end_type_cd = 'PETOF' New Plan

17 33 Evolution of tuning a query Indexes for our table: INDNAME COLNAMES SQL PK_EVENT_NB EVENT_IX1 -DSTAGE_IDENTITY_NB EVENT_IX2 +EVENT_START_TYPE_CD+EVENT_END_TYPE_CD EVENT_IX3 +EVENT_END_TYPE_CD+EVENT_START_TYPE_CD+RECORD_UPDATE_TS SELECT event_start_type_cd, event_end_type_cd, record_update_ts FROM brad.event WHERE event_end_type_cd = 'PETOF' Old Plan Should we get rid of EVENT_IX2? New Plan 34 Welcome to DBA world.

18 35 Agenda Indexes 101 -the basics Important Index info in the catalog Index considerations when tuning Special Commands related to Indexes Index PotPourri for $500, Alex Jumpscans v10 36 Performance improvement to avoid costly index scans when the index has gaps SELECT * FROM mytab T WHERE T.COL2 = 20 Index on (COL1, COL2) COL1 is a gap column (No predicate on COL1) COL2 is a non-gap column When does Jump Scan work best? When the gap column has small cardinality and the non-gap column predicate is highly selective. Ideal case for the example: COL1 has 10 distinct values COL2 has high cardinality, and the query predicate only selects a few rows

19 Jumpscan example SELECT * FROM mytab T WHERE T.COL2 = 20 Artificially add predicate for col1 Where col2 = 20 and col1 = 1 Where col2 = 20 and col1 = 2 Where col2 = 20 and col1 = 3 Where col2 = 20 and col1 = 4 Where col2 = 20 and col1 = 5 Prior method was to scan entire index comparing on col2 With jumpscan, able to skip large amounts of index Index 37 COL1 COL Runstats now supports IndexSample v10 Similar to the existing TABLESAMPLE command parameter Speed up the time it takes to generate statistics by Reducing the total number of leaf nodes processed by RUNSTATS (when INDEXSAMPLE SYSTEM is specified) Reducing the total number of index entries processed by RUNSTATS (when INDEXSAMPLE BERNOULLI is specified) Start with a 10% page-level sample, by specifying TABLESAMPLE SYSTEM(10) and INDEXSAMPLE SYSTEM(10) If results are not desirable, try a 10% row-level sample instead, by specifying INDEXSAMPLE BERNOULLI(10)

20 39 Virtual Indexes db2advis, the Index Advisor, populates the advise_index table with potential (Virtual) indexes You can produce a what if explain with db2exfmt that will consider the virtual indexes 40 Steps to Create an Explain Plan With Virtual Indexes 1. db2 set current explain mode recommend indexes 2. Execute your SQL statement db2 -stvf test.sql SQL0217W The statement was not executed as only Explain information requests are being processed. SQLSTATE= Query advise_index table to see what indexes were recommended select explain_time, name, tbname, colnames, use_index use, exists from advise_index order by explain_time; 4. Format last explain in explain tables db2exfmt -d dbedwq -1 -o explain.txt

21 41 advise_index table entries select explain_time, name, tbname, colnames, use_index use, exists from advise_index order by explain_time; EXPLAIN_TIME NAME TBNAME COLNAMES USE EXISTS SQL TOUR_ORDER +ORDER_ID+TOUR_ID Y Y IDX ORDER +CUSTOMER_CONTROL_ROLE_ID+CUST Y N IDX TOUR +SCHEDULED_ARRIVAL_TS-TOUR_ID Y N N means these are virtual indexes 42 Format the Explain / \ db2exfmt -d dbedwq -1 -o explain.txt NLJOIN IXSCAN ( 3) ( 10) / \ e+06 TBSCAN IXSCAN INDEX: SYSTEM ( 4) ( 9) IDX Q e+06 SORT INDEX: SYSIBM ( 5) SQL Q NLJOIN ( 6) / \ TBSCAN IXSCAN ( 7) ( 8) e e+06 TABFNC: SYSIBM INDEX: SYSTEM GENROW IDX Q1 Q3 Index ~55050 is the virtual index on Tour table I/Os have gone from 1331 to 2 Index ~55080 is the virtual index on Order table

22 43 To ignore a virtual index, update advise_index to set USE to N select explain_time, name, tbname, colnames, use_index use, exists from advise_index order by explain_time; EXPLAIN_TIME NAME TBNAME COLNAMES USE EXISTS SQL TOUR_ORDER +ORDER_ID+TOUR_ID Y Y IDX ORDER +CUSTOMER_CONTROL_ROLE_ID+CUST Y N IDX TOUR +SCHEDULED_ARRIVAL_TS-TOUR_ID N N Let s update from Y to N to prevent this virtual index from being used by explain Now, the real index is being used for Tour table db2exfmt -d dbedwq -1 -o explain.txt / \ e+06 TBSCAN IXSCAN INDEX: VASDW ( 4) ( 9) IDX_TOUR_ Q e+06 SORT INDEX: SYSIBM ( 5) SQL Q NLJOIN ( 6) / \ TBSCAN IXSCAN ( 7) ( 8) e e+06 TABFNC: SYSIBM INDEX: SYSTEM GENROW IDX Q1 Q3 Existing index on Tour Still using virtual index on Order. 44

23 45 Section Actuals and the db2caem Command 1. Create Explain tables 2. Create Activity Event Monitor 3. Enable Event Monitor 4. Call wlm_set_conn_env to capture on my connection 5. Execute SQL statement(s) of interest 6. Call wlm_set_conn_env to disable collection 7. Turn off Event Monitor 8. Query Event Monitor tables to find query 9. Call explain_from_activity to explain query 10. Format explain using db2exfmt 46 Or just use db2caem!!!!!!! db2caem -d dbedwp -sf po_summary.sql This actually executes the query /brad/sqltuning/brad# ls -l inedwp db2admin inedwp db2admin 256 Feb 11 14:28 DB2CAEM_ Apr po_summary.sql /brad/sqltuning/brad/db2caem_ # ls -l inedwp db2admin 4096 Feb 11 14:28 EXPORTS inedwp db2admin 7654 Feb 11 14:28 db2caem.exfmt.1 inedwp db2admin 1872 Feb 11 14:28 db2caem.log inedwp db2admin 38 Feb 11 14:28 db2caem_options.in

24 47 db2exfmt output Estimated Cardinality Actual Cardinality Rows Rows Actual RETURN ( 1) Cost I/O ^HSJOIN ( 2) NA / \ NLJOIN IXSCAN ( 3) ( 9) NA NA / \ NA TBSCAN IXSCAN INDEX: VASDW ( 4) ( 8) IDX_TOUR_ Q5 NA NA 48 Section Actuals in db2exfmt output Input Streams: ) From Operator #7 Estimated number of rows: Actual number of rows: ) From Object BRAD.ORDER Estimated number of rows: e+06 Column Names: Q3.CUSTOMER_CONTROL_NM+Q3.STATUS_ID +Q3.STATUS_DESCRIPTION_TX+Q3.ORDER_ID Output Streams: ) To Operator #5 Estimated number of rows: Actual number of rows: 2110

25 49 Index Advisor and db2advis For 9.7 and earlier, can invoke Design Advisor from Control Center In Data Studio, it appears that Index Recommendations require purchasing a licensed product Easier/faster just to run db2advis command db2advis -db dbedwd -i po_summary.sql Warning: Index Advisor was built to analyze workloads Will tend to over-recommend indexes and index columns The DBA should be selective about what to implement It would be VERY nice if Index Advisor would provide indexes in a ranked manner 50 Agenda Indexes 101 -the basics Important Index info in the catalog Index considerations when tuning Special Commands related to Indexes Index PotPourri for $500, Alex

26 51 How To Find Unused Indexes Lastused column in syscat.indexes Date when the index was last used by any DML statement to perform a scan, or used to enforce referential integrity constraints. This column is not updated when the index is used on an HADR standby database, nor is it updated when rows are inserted into the table on which the index is defined. The default value is ' '. This value is updated asynchronously not more than once within a 24 hour period and might not reflect usage within the last 15 minutes. Beware of apar IC70265: lastused column not always updated if < v9.7 fp4 db2pd db mydb tcbstats Look at scans column Resets to zero on db2stop/db2start db2pd -db dbedwp -tcbstats index tableid=14 tbspaceid=4 TableName IID Scans TRACTOR TRACTOR 4 0 TRACTOR mon_get_index The MON_GET_INDEX table function returns metrics for one or more indexes. MON_GET_INDEX--(--tabschema--,--tabname--,--member--) select * from TABLE(MON_GET_INDEX('','', -2)) V9.7 interesting columns Index_scans Index_only_scans V10 more interesting columns Object_index_l_reads Object_index_p_reads Jump_scans

27 53 Volatile Tables By default, autostats will not collect stats on volatile tables But, if stats exist on volatile tables, the optimizer will use them So, if you run stats manually on a volatile table, they will factor into the optimizer s equation Wait for a time that the data best represents itself, then perform a single runstats There is no un-runstat command to return you to -1 Can be updated manually, but I tend to avoid this MDC Block Indexes 54 Used by MDC tables to point to blocks of data within an MDC Very small, since there is only one pointer for each block as opposed to one pointer for each row Remember with MDC to primarily use lower cardinality columns foryour dimensions

28 55 Brad Price Werner Enterprises Session D12 An Index s Guide to the Universe Evaluate my session online:

29 57 Why Are My Index Scans Slow? High ratio of Physical Index Reads/Logical Index Reads Runstats are slow during index phase Likely in need of an Index Reorg 58 An Example of Index Physical I/O and Reorg Runstats before reorg index: qa-edw01:/home/ibm/inedwq# snapapp grep -i reads Buffer pool data logical reads = Buffer pool data physical reads = 489 Buffer pool temporary data logical reads = 0 Buffer pool temporary data physical reads = 0 Buffer pool index logical reads = Buffer pool index physical reads = Buffer pool temporary index logical reads = 0 Runstats after reorg index: qa-edw01:/home/ibm/inedwq# snapapp grep -i reads Buffer pool data logical reads = Buffer pool data physical reads = 351 Buffer pool temporary data logical reads = 0 Buffer pool temporary data physical reads = 0 Buffer pool index logical reads = Buffer pool index physical reads = 23 Buffer pool temporary index logical reads = 0 V10: Index Smart Prefetch will affect this.