Indexing 3 Index Types Two types of indexing technologies are supported Radix Index The default unless you explicitly say otherwise Encoded Vector Ind

Transcription

1 Boosting SQL Performance with Indexing Technology Rob Bestgen IBM - DB2 for i 215 IBM Corporation in dex Something that serves to guide, point out, or otherwise facilitate reference 2 1

2 Indexing 3 Index Types Two types of indexing technologies are supported Radix Index The default unless you explicitly say otherwise Encoded Vector Index Each type of index has specific uses and advantages Radix is the best general purpose index Respective indexing technologies complement each other 4 2

3 Index Essence Indexes: Are helper objects In SQL they exist simply to improve performance (try referencing them in a FROM clause) Are used for implementation and statistics Are maintained automatically by the database as the table rows come and go Only contain active rows (no deleted rows!) Contain data (the key) and a row reference (RRN) Are most often used in conjunction with table access They assist in determining the desired rows of the table, then the rows are retrieved from the table Can be used without having to access the table index only access 5 Index Usage - Probe vs Scan Indexes can be probed or scanned Probe using the leading (contiguous) key columns for fast lookups If more than one key is used, equal selection is required for all but the last key Scan can occur on any key column Probe and scan can be used together Probe to a subset of entries, then scan each of those A Radix index can provide ordering of data (ORDER BY) Can leverage this side effect with either probe or scan Indexes are cooperative More than one index can be used for a table in a given query Requires use of RRN list or bitmap of RRNs Mixing and matching Radix and EVI supported 6 3

4 Using Indexes - Probe v Scan Probe (key positioning) with leading, n contiguous key columns Scan (test) with any other key columns Why is the last WHERE example a scan?? Index Key Columns (CUSTOMER, ORDER, ITEM) CUSTOMER ORDER ITEM 1 B57 AB-27 1 B67 CD-2 2 B1 XY-15 2 B12 AZ-5 3 B79 HH-65 4 B43 HH-65 WHERE ORDER = B12 AND CUSTOMER = 2 WHERE ITEM = HH-65 WHERE ORDER = B12 OR CUSTOMER = 2 7 Two Radix - ANDing Example Radix State Radix Workdept SELECT * FROM EMPLOYEE WHERE STATE = MINNESOTA' AND WORKDEPT IN ( 'B1', C1, 'E1') Intermediate RRN list Intermediate RRN list Final RRN list State AND (Merge) Workdepts Represents all the local selection 4 8

5 Encoded Vector Index (EVI) Index for delivering fast data access in analytical and reporting environments Used to produce dynamic bitmaps and RRN lists Fast access to statistics to improve query optimizer decision making Can also include non-key summary data (SUM, COUNT) Not a tree structure Contains two parts: 1 Symbol table of unique key values Each unique value is encoded (assigned) to a number 2 Vector with one entry per row in the underlying table Each row contains the associated key value s assigned number Can only be created through an SQL interface or System i Navigator GUI CREATE ENCODED VECTOR INDEX SchemaName/IndexName ON SchemaName/TableName (ColumnName) INCLUDE ( SUM(ColumnName)); 9 Two EVIs - ANDing Example EVI State EVI Workdept SELECT * FROM EMPLOYEE WHERE STATE = MINNESOTA' AND WORKDEPT IN ( 'B1', C1, 'E1') Intermediate RRN list Intermediate RRN list Final RRN list State AND (Merge) Workdepts Represents all the local selection 5 1

6 Radix + EVI Usage - ORing Example Radix State EVI Workdept SELECT * FROM EMPLOYEE WHERE STATE = IOWA' OR WORKDEPT IN ( 'B1', C1, 'E1') State Intermediate Bitmap 1 1 OR (Merge) Workdepts Intermediate Bitmap Final Bitmap Represents all the local selection 11 EVI Index for Aggregate Answers CREATE ENCODED VECTOR INDEX idx2 ON employee (state) INCLUDE (SUM(x), SUM(y), COUNT(*)) SELECT COUNT(*) FROM EMPLOYEE WHERE STATE = Wisconsin ; SELECT COUNT(DISTINCT STATE) FROM EMPLOYEE; SELECT STATE, SUM(x), SUM(y) FROM EMPLOYEE GROUP BY STATE; Search / Scan symbol table for key(s) and counts Symbol Table Key Value Code Count Include Include Sum(x) Sum(y) Arizona Arkansas Wisconsin Wyoming

7 The right index for the job cardinality The number of elements in a set High cardinality = large number of distinct values Low cardinality = small number of distinct values In general A radix index is best when accessing a small set of rows and the key cardinality is high An encoded vector index is best when accessing a subset of rows and the key cardinality is low Understanding the data and query is essential 13 Creating Indexes How? CREATE INDEX SQL statement CREATE INDEX MY_IX on MY_TABLE (KEY1, KEY2) CREATE ENCODED VECTOR INDEX SQL statement CREATE ENCODED VECTOR INDEX MY_EVI on MY_TABLE (KEY1) System i Navigator Database graphical interface CRTPF and CRTLF CL commands (under the covers) Keyed access path within the physical file or logical file Join logical file Caution: Do not use this (native) approach for SQL! Illustration purposes only Primary Key, Foreign Key and Unique Key Constraints have indexes created automatically under the covers 14 7

8 Creating Indexes through inav 15 Derived Keys Creation of indexes with derived keys via SQL supported CREATE INDEX ORDERPRIORITYUPPER ON T1 (UPPER(ORDERPRIORITY) AS UORDERPRIORITY ASC); CREATE ENCODED VECTOR INDEX YEARQTR ON T1 (YEAR(ORDERDATE) AS ORDYEAR ASC, QUARTER(ORDERDATE) AS ORDQTR ASC); CREATE INDEX TOTALEXTENDEDPRICE ON T1 (QUANTITY * EXTENDEDPRICE AS TOTEXTPRICE ASC); 16 8

9 When to create a Derived Index? Derived indexes can be useful for Case insensitive searches Data extracted from a column eg SUBSTR, YEAR, MONTH Derive Common Grouping columns eg YEAR(ORDERDATE) Results of operations eg COL1+COL2, QTY * COST Might be useful to allow index only access in more cases Including the INCLUDE support for EVIs Could replace some logical files with SQL indexes for use by RLA native, high level language programs Modernize those objects Big logical page size (8K v 64K) 17 Query Optimization (using indexes) 9

10 Data Access Methods Cost based optimization dictates that the fastest access method for a given table will vary based upon selectivity of the query High Response Time Table Scan Clustered, Skip Seq Low Few Probe Number of rows searched / accessed Many 19 Strategy for Query Optimization Optimizing indexes will generally follow this (simplified) strategy: Gather list of indexes for statistics and costing implementation methods Consider the indexes based on how the index can be used Local selection Joining Grouping Ordering Index only access One index may be useful for statistics, and another useful for implementation 2 If an index is missing that could be useful for the query, the optimizer will often Advise it! 1

11 Query Optimization Feedback Advised Index Catalog SYSIXADV SQE Plan Cache Visual Explain SQE Plan Cache Snapshots SQL request Detailed DB Monitor Data Query Optimization 21 Index Advice from the Query Optimizer Index advice collected into a system file (catalog) Catalog (file) QSYS2/SYSIXADV Each iasp has its own SYSIXADV catalog View through System i Navigator Index advice also stored for each query and viewable through Visual Explain Both Radix and EVI indexes advised (Radix advised more frequently) Advice based on all parts of the query (selection, join, grouping, ordering) Multiple indexes can be advised for the same query Advice is limited to columns (fields) only No derivations/expressions Temporary Index creation also provides some insight Advise can be Condensed into a smaller, more general purpose set of indexes 22 11

12 System Wide Index Advisor 23 Index Advisor System-wide 24 12

13 Too Much IBM Advice? Training Index Advisor - Condenser Queries: WHERE YEAR = 28 AND QUARTER = 1 AND COLOR = BLUE; WHERE YEAR = 28 AND QUARTER = 1; WHERE YEAR = 28; Index advice by query: YEAR, QUARTER, COLOR YEAR, QUARTER YEAR Can be used for all three queries Condensed advice: YEAR, QUARTER, COLOR 25 Index Advisor - Condenser 26 13

14 MTIs 27 Autonomic (Temporary) Index Creation Optimizer can have the DB Engine create a maintained temporary index (MTI) As the name implies, MTIs are maintained as table rows change Temporary indexes consume temporary storage Temporary indexes are usually a good indicator that a permanent index should be created 28 14

15 Index Evaluator (Show Existing Indexes) Meta data for Indexes 29 Visual Explain 3 15

16 16

17 Run SQL Scripts A JDBC client that supports dynamic SQL 17

18 18

19 Making Choices 38 19

20 Create all advised indexes? Nothing, let the system handle it? Monitor, analyze, and tune important tables and queries? 39 DB2 for i The goals of creating indexes are: 1 Provide the optimizer implementation choices, based on the selectivity of the query Probe Scan Index only access 2 Provide the optimizer the statistics needed to understand the data, based on the query In all cases, the point is to 4 Improve Performance! 2

21 The Process of Identifying Indexes Proactive method Analyze the data model, application and SQL requests Add Database Primary Key, Unique Key and Referential Integrity constraints are helpful Reactive method Rely on optimizer feedback and actual implementation methods Rely on SQE s ability to auto tune using temporary indexes Understand the data being queried Column selectivity Column cardinality 41 Indexing Strategy Radix Indexes Common local selection columns Join columns Local selection columns + join columns Local selection columns + grouping columns Local selection columns + ordering columns Minimum Note: Columns used with equal conditions are first in key list 42 21

22 Indexing Strategy Other Radix Index usage considerations Index only access index contains all columns needed by the query(s) Avoid deleted records Replace an MTI

23 Indexing Strategy - Examples -- Query 1 SELECT ACUSTOMER_NO, AORDER_DATE, AQUANTITY FROM ORDERS A WHERE ACUSTOMER_NO = ; CREATE INDEX ORDERS_IX1 ON ORDERS (CUSTOMER_NO); -- Query 2 SELECT ACUSTOMER_NO, AORDER_DATE, AQUANTITY FROM ORDERS A WHERE ACUSTOMER_NO = AND AITEM_ID = ABC123YXZ ; CREATE INDEX ORDERS_IX2 ON ORDERS (CUSTOMER_NO, ITEM_ID); 45 Indexing Strategy - Examples -- Query 3 SELECTACUSTOMER_NO, ACUSTOMER, AORDER_DATE FROM ORDERS A WHERE ACUSTOMER_NO IN (112358, , ) AND AORDER_DATE > 25/6/3 ORDER BY AORDER_DATE; CREATE INDEX ORDERS_IX3a ON ORDERS (CUSTOMER_NO, ORDER_DATE); CREATE INDEX ORDERS_IX3b ON ORDERS (ORDER_DATE, CUSTOMER_NO); Q: Which is better? -- Query 4 SELECTACUSTOMER_NO, ACUSTOMER, AORDER_DATE FROM ORDERS A WHERE ACUSTOMER_NO = OR AORDER_DATE = 25/6/3 ; CREATE INDEX ORDERS_IX4 ON ORDERS (CUSTOMER_NO); CREATE ENCODED VECTOR INDEX ORDERS_EVI4 ON ORDERS (ORDER_DATE); 46 Q: Why not just create one index? 23

24 Indexing Strategy - Examples -- Query 5 SELECT ACUSTOMER_NO, BCUSTOMER, AORDER_DATE, AQUANTITY FROM ORDERS A, CUSTOMERS B, ITEMS C WHERE ACUSTKEY = BCUSTKEY AND AITEMKEY = CITEMKEY AND ACUSTOMER_NO = ; CREATE INDEX ORDERS_IX5a ON ORDERS (CUSTOMER_NO, CUSTKEY); CREATE INDEX ORDERS_IX5b ON ORDERS (CUSTOMER_NO, ITEMKEY); CREATE INDEX CUSTOMERS_IX5 ON CUSTOMERS (CUSTKEY); CREATE INDEX ITEMS_IX5 ON ITEMS (ITEMKEY); Q: Are there other useful index combinations? 47 Indexing Strategy - Examples -- Query 6 SELECT YEAR(AORDER_DATE),SUM(AQUANTITY), COUNT(*) FROM ORDERS A GROUP BY YEAR(AORDER_DATE); CREATE ENCODED VECTOR INDEX ORDERS_IX6A ON ORDERS (YEAR(ORDER_DATE)) INCLUDE (SUM(QUANTITY), COUNT(*)); 48 24

25 Indexing Strategy - Examples -- Query 7 SELECT YEAR(AORDER_DATE),QUARTER(AORDER_DATE), MONTH(ORDER_DATE), SUM(AQUANTITY), COUNT(*) FROM ORDERS A WHERE QUARTER(AORDER_DATE) = 4 GROUP BY YEAR(AORDER_DATE), QUARTER(AORDER_DATE), MONTH(ORDER_DATE) ORDER BY YEAR(AORDER_DATE),QUARTER(AORDER_DATE), MONTH(ORDER_DATE), CREATE ENCODED VECTOR INDEX ORDERS_IX6A ON ORDERS (YEAR(ORDER_DATE), QUARTER(ORDER_DATE), MONTH(ORDER_DATE) ) INCLUDE (SUM(QUANTITY), COUNT(*)); 49 Practice Examples select * from orders where shipdate = ; select * from orders where orderdate = and quantity = ; 5 select * from orders o, customers c where ocustkey = ccustkey; 25

26 Practice Examples select * from orders o inner join customers c on ocustkey = ccustkey; select * from orders o, customers c where ocustkey = ccustkey and cstate = MN ; select * from orders o inner join customers c on ocustkey = ccustkey and oorderkey = ; 51 Practice Examples select * from customers where country in ('CANADA', 'FRANCE', GERMANY'); select * from customers where upper(customer) = JOE COOL' and country in ('CANADA', 'FRANCE', GERMANY'); 52 select * from customers where country = 'FRANCE' or continent = 'EUROPE and rating = BEST ; 26

27 Practice Examples select from group by count(*) customers country; select from where group by count(*) customers continent = ASIA country; select * from orders where orderdate > ' and shipdate < ' '; select * from orders where orderpriority = '1-URGENT' and orderdate > ' and shipdate < ' '; 53 Practice Examples update orders set shipmode = AIR where orderkey = ; update orders o set status = BACK ORDERED where opartkey in (select ppartkey from parts p where inventory_amount = ); 27 54

28 To Index or Not to Index? For best query performance, create the appropriate indexes Eliminating table scans and temporary data structures will more than make up for index maintenance overhead Consider the number of indexes when doing high volume changes Drop indexes when inserting into any empty table Consider dropping and recreating indexes when adding, changing, or deleting more than 4% of the rows Consider parallel index maintenance for INSERTs and parallel index builds DB2 SMP feature installed and enabled Use SMP to create indexes in parallel Especially useful when (INSERT + INDEX CREATION) < (INSERT + INDEX MAINT) Summary View creating indexes as an opportunity rather than a chore! Take advantage of the integrated indexes provided by the query optimizer Manage your indexes, creating those that demonstrably improve performance and dropping those that are just taking up space Modernize your DDS logical file indexes to take advantage of larger maximum sizes and larger logical page sizes The right set of indexes will pay for themselves in terms of user satisfaction and system capacity! 28

29 Are you experiencing performance problems? Are you using SQL? Are you getting the most out of DB2 for i? Lab Services IBM DB2 for i Team Database modernization DB2 Web Query Database architecture and design DB2 SQL performance analysis and tuning Data warehousing and Business Intelligence DB2 for i education and training Need help? Contact: Mike Cain mcain@usibmcom IBM Systems and Technology Group Rochester, MN USA Thank you! 58 29

30 Special notices This document was developed for IBM offerings in the United States as of the date of publication IBM may not make these offerings available in other countries, and the information is subject to change without notice Consult your local IBM business contact for information on the IBM offerings available in your area Information in this document concerning non-ibm products was obtained from the suppliers of these products or other public sources Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products IBM may have patents or pending patent applications covering subject matter in this document The furnishing of this document does not give you any license to these patents Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY USA All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients Rates are based on a client's credit rating, financing terms, offering type, equipment type and options, and may vary by country Other restrictions may apply Rates and offerings are subject to change, extension or withdrawal without notice IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary IBM hardware products are manufactured from new parts, or new and serviceable used parts Regardless, our warranty terms apply Any performance data contained in this document was determined in a controlled environment Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration Some measurements quoted in this document may have been made on development-level systems There is no guarantee these measurements will be the same on generallyavailable systems Some measurements quoted in this document may have been estimated through extrapolation Users of this document should verify the applicable data for their specific environment 59 Special notices (cont) IBM, the IBM logo, ibmcom AIX, AIX (logo), AIX 6 (logo), AS/4, BladeCenter, Blue Gene, ClusterProven, DB2, ESCON, i5/os, i5/os (logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/4, OS/4, PartnerLink, PartnerWorld, PowerPC, pseries, Rational, RISC System/6, RS/6, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xseries, z/os, zseries, AIX 5L, Chiphopper, Chipkill, Cloudscape, DB2 Universal Database, DS4, DS6, DS8, EnergyScale, Enterprise Workload Manager, General Purpose File System,, GPFS, HACMP, HACMP/6, HASM, IBM Systems Director Active Energy Manager, iseries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER6+, System i, System p, System p5, System Storage, System z, Tivoli Enterprise, TME 1, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at wwwibmcom/legal/copytradeshtml The Power Architecture and Powerorg wordmarks and the Power and Powerorg logos and related marks are trademarks and service marks licensed by Powerorg UNIX is a registered trademark of The Open Group in the United States, other countries or both Linux is a registered trademark of Linus Torvalds in the United States, other countries or both Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both Intel, Itanium, Pentium are registered trademarks and Xeon is a trademark of Intel Corporation or its subsidiaries in the United States, other countries or both AMD Opteron is a trademark of Advanced Micro Devices, Inc Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc in the United States, other countries or both TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC) SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC) NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both AltiVec is a trademark of Freescale Semiconductor, Inc Cell Broadband Engine is a trademark of Sony Computer Entertainment Inc InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association Other company, product and service names may be trademarks or service marks of others 3 6