CA Chorus for DB2 Database Management

Transcription

1 CA Chorus for DB2 Database Management CA Performance Handbook for DB2 for z/os Version

2 This Documentation, which includes embedded help systems and electronically distributed materials (hereinafter referred to as the Documentation ), is for your informational purposes only and is subject to change or withdrawal by CA at any time. This Documentation is proprietary information of CA and may not be copied, transferred, reproduced, disclosed, modified or duplicated, in whole or in part, without the prior written consent of CA. If you are a licensed user of the software product(s) addressed in the Documentation, you may print or otherwise make available a reasonable number of copies of the Documentation for internal use by you and your employees in connection with that software, provided that all CA copyright notices and legends are affixed to each reproduced copy. The right to print or otherwise make available copies of the Documentation is limited to the period during which the applicable license for such software remains in full force and effect. Should the license terminate for any reason, it is your responsibility to certify in writing to CA that all copies and partial copies of the Documentation have been returned to CA or destroyed. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENTATION AS IS WITHOUT WARRANTY OF ANY KIND, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. IN NO EVENT WILL CA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY LOSS OR DAMAGE, DIRECT OR INDIRECT, FROM THE USE OF THIS DOCUMENTATION, INCLUDING WITHOUT LIMITATION, LOST PROFITS, LOST INVESTMENT, BUSINESS INTERRUPTION, GOODWILL, OR LOST DATA, EVEN IF CA IS EXPRESSLY ADVISED IN ADVANCE OF THE POSSIBILITY OF SUCH LOSS OR DAMAGE. The use of any software product referenced in the Documentation is governed by the applicable license agreement and such license agreement is not modified in any way by the terms of this notice. The manufacturer of this Documentation is CA. Provided with Restricted Rights. Use, duplication or disclosure by the United States Government is subject to the restrictions set forth in FAR Sections , , and (c)(1) - (2) and DFARS Section (b)(3), as applicable, or their successors. Copyright 2014 CA. All rights reserved. All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.

3 CA Technologies Product References This document references the following CA Technologies products: CA Database Analyzer for DB2 for z/os (CA Database Analyzer) CA Detector for DB2 for z/os (CA Detector) CA Index Expert for DB2 for z/os (CA Index Expert) CA Plan Analyzer for DB2 for z/os (CA Plan Analyzer) CA Rapid Reorg for DB2 for z/os (CA Rapid Reorg) CA SQL-Ease for DB2 for z/os (CA SQL-Ease) CA Subsystem Analyzer for DB2 for z/os (CA Subsystem Analyzer)This document references the following CA Technologies products: CA SYSVIEW Performance Management Option for DB2 (CA SYSVIEW for DB2) Contact CA Technologies Contact CA Support For your convenience, CA Technologies provides one site where you can access the information that you need for your Home Office, Small Business, and Enterprise CA Technologies products. At you can access the following resources: Online and telephone contact information for technical assistance and customer services Information about user communities and forums Product and documentation downloads CA Support policies and guidelines Other helpful resources appropriate for your product Providing Feedback About Product Documentation If you have comments or questions about CA Technologies product documentation, you can send a message to techpubs@ca.com. To provide feedback about CA Technologies product documentation, complete our short customer survey which is available on the CA Support website at

4

5 Contents Chapter 1: Introducing DB2 Performance 11 Using This Handbook Key Topics Audience About DB2 Performance Measuring DB2 Performance DB2 Performance Problem Root Cause Designing for Performance Identifying Performance Problems Fixing the Performance Problems Chapter 2: SQL and Access Paths 17 The DB2 Optimizer Importance of Catalog Statistics Cardinality Statistics Frequency Distribution Statistics Histogram Statistics Access Paths Table space Scans Limited Partition Scan and Partition Elimination Index Access Virtual Buffer Pools What EXPLAIN Tells You List Prefetch Nested Loop Join Hybrid Join Merge Scan Join Star Join Read Mechanisms Sequential Prefetch Dynamic Prefetch and Sequential Detection Index Lookaside Sorting Avoiding a Sort for an ORDER BY Avoiding a Sort for a GROUP BY Avoiding a Sort for a DISTINCT Contents 5

6 Avoiding a Sort for a UNION EXCEPT or INTERSECT Parallelism Chapter 3: Predicates and SQL Tuning 37 DB2 Predicates Stage 1 Indexable Other Stage Stage Stage Combining Predicates Boolean Term Predicates Predicate Transitive Closure Coding Efficient SQL Avoid Unnecessary Processing Coding SQL for Performance Promoting Predicates Functions, Expressions, and Performance Advanced SQL and Performance Recommendations for Distributed Dynamic Influencing the Optimizer Proper Statistics Run-time Reoptimization OPTIMIZE FOR Clause Encouraging Index Access and Table Join Chapter 4: Designing Tables and Indexes for Performance 59 Table Space Performance Recommendations Use Segmented or Universal Table Spaces Clustering and Partitioning Free Space Allocations Column Ordering Table Space Compression Referential Constraints Indexing Efficient Database Design Index Design Recommendations Special Tables Used for Performance Materialized Query Tables Volatile Tables Clone Tables CA Performance Handbook for DB2 for z/os

7 Table Designs for Special Situations UNION in View for Large Table Design Append Processing for High Volume Inserts Building an Index on the Fly Denormalization "Light" Chapter 5: EXPLAIN Facility and Predictive Analysis 79 EXPLAIN Facility What EXPLAIN Tells You What EXPLAIN Does Not Tell You Predictive Analysis Tools Optimization Service Center and Visual EXPLAIN DB2 Estimator Predicting Database Performance Methods to Generate Data Methods to Generate Statements Determine Your Access Patterns Simulating Access with Tests Chapter 6: Monitoring 91 DB2 Traces Statistics Trace Accounting Trace Performance Trace Accounting and Statistics Reports Statistics Report Accounting Report Online Performance Monitors Overall Application Performance Monitoring Setting the Appropriate Accounting Traces Summarizing Accounting Data Reporting to Management Finding the Statements of Interest The Tools You Need Chapter 7: Application Design and Tuning for Performance 105 Avoiding Redundant Statements Caching Data System-Generated Keys Avoiding Programmatic Joins Contents 7

8 Multi-Row Operations Placing Data-Intensive Business Logic in the Database Organizing Your Input Search Strategies Separate Predicates Boolean Term Predicate Name Searching Existence Checking Non-Correlated Subqueries Correlated Subqueries Joins ORDER BY and FETCH Lock Avoidance Strategies Optimistic Locking How Optimistic Locking Works How to Use the ROWCHANGE Option Heuristic Control Tables and Commit Strategies Chapter 8: Tuning Subsystems 125 Buffer Pools Page Types in Virtual Pools Virtual Buffer Pools Buffer Pool Queue Management I/O Requests and Externalization Checkpoints and Page Externalization Buffer Pool Sizing Sequential Versus Random Processing Write Thresholds Buffer Pool Parallelism Page Fixing Internal Thresholds Virtual Pool Design Strategies Buffer Pool Tuning RID Pool RID Pool Size RID Pool Statistics to Monitor The SORT Pool Sort Pool Size Small Sort Pool Size Environmental Descriptor Manager Pool Efficiency Dynamic SQL Caching CA Performance Handbook for DB2 for z/os

9 Logging Log Reads Log Writes Locking and Contention Thread Management The DB2 Catalog and Directory Chapter 9: Understand and Tune Your Packaged Applications 143 Database Utilization and Packaged Applications Find and Fix SQL Performance Problems Subsystem Tuning for Packaged Applications Table and Index Design Options for High Performance Monitoring and Maintaining Your Objects Real-Time Statistics Chapter 10: Tuning Tips 153 Code DISTINCT Only When Needed Prevent Java in WebSphere From Defaulting the Isolation Level Locking a Table in Exclusive Mode Will Not Always Prevent Applications from Reading It Put Data in Your Empty Control Table Use Recursive SQL to Assign Multiple Sequence Numbers in Batch Code Correlated Nested Table Expression versus Left Joins to Views Use GET DIAGNOSTICS Only When Necessary for Multi-Row Operations Database Design Tips for High Concurrency Database Design Tips for Data Sharing Tips for Avoiding Locks Index 161 Contents 9

10

11 Chapter 1: Introducing DB2 Performance Using This Handbook This handbook helps DB2 performance tuners to complete the following tasks: Design databases for high performance. Understand how DB2 processes SQL and transactions, and tune those processes for performance. For database performance management, the philosophy of performance tuning begins with database design and extends through the database and application development and production implementation. Key Topics This guide includes the following key topics: Fundamentals of DB2 performance. Descriptions of features and functions that help you manage and optimize performance. Methods for identifying DB2 performance problems. Guidelines for tuning static and dynamic SQL. Guidelines for the proper design of tables and indexes for performance. DB2 subsystem performance information, and advice for subsystem configuration and tuning. Commentary for proper application design for performance. Real-world tips for performance tuning and monitoring. Chapter 1: Introducing DB2 Performance 11

12 Audience Audience The primary audience for this handbook is physical and logical database administrators. Because performance management has many stakeholders, additional users may benefit from this information, including application developers, data modelers, and data center managers. For successful performance initiatives, DBAs, developers, and managers must cooperate and contribute to performance management. There are no right ways or wrong ways exist to tune performance. However, trade-offs are common as people decide about database design and performance. Are you designing for ease of use, minimized programming effort, flexibility, availability, or performance? As you make these choices, consider organizational costs and benefits and whether they are measured in the amount of time and effort by the personnel involved in development and maintenance, response time for end users, or CPU metrics. This handbook assumes that you have a good working knowledge of DB2 and SQL. This book helps you build good performance into the application, database, and the DB2 subsystem. This book provides techniques to help you monitor DB2 for performance and to identify and tune production performance problems. About DB2 Performance When one of the following events occurs, DB2 performance becomes a focal point: An online production system does not respond within the time specified in a service level agreement (SLA). An application uses more CPU (and thus costs more money) than is desired. Most of the efforts involved in tuning during these situations revolve around fire fighting, and when the problem is solved, the tuning effort is abandoned. Performance tuning can be categorized as follows: Designing for performance (application and database) Testing and measuring performance design efforts Understanding SQL performance and tuning SQL Tuning the DB2 subsystem Monitoring production applications An understanding of DB2 performance, even at a high level, can help ensure the performance of applications, databases, and subsystems. 12 CA Performance Handbook for DB2 for z/os

13 Measuring DB2 Performance Measuring DB2 Performance Performance design and tuning are vital because users want accessible, accurate, and secure data. Strong performance is a balance between expectations and reality. For effective performance tuning, work with each stakeholder to identify and understand their expectations. After identifying and understanding the expectations of the stakeholders, you realize the full benefits of performance tuning. How you define strong DB2 database and application performance centers on two areas. First, it means that users do not need to wait for the information that they require to do their jobs. Second, it means that the organization does not spend too much money running the application. The methods for designing, measuring, and tuning for performance vary from application to application. The optimal approach depends upon the type of application. Your site may use one or all of the following factors to measure performance: User complaints Overall response time measurements Service level agreements (SLAs) System availability I/O time and waits CPU consumption and associated charges Locking time and waits More Information: EXPLAIN Facility and Predictive Analysis (see page 79) DB2 Performance Problem Root Cause When the measurements that you use at your site warn you of a performance problem, you face the challenge of finding the root cause. The following list details common root causes of performance problems: Poor database design Poor application design Catalog statistics that do not reflect the data size, organization, or both Poorly written SQL Chapter 1: Introducing DB2 Performance 13

14 Designing for Performance Generic SQL design Inadequately allocated system resources, such as buffers and logs In other situations, you may be working with packaged software applications that cause performance problems. In these situations, it may be difficult to tune the application, but it is possible to improve performance through database and subsystem tuning. More Information: Understand and Tune Your Packaged Applications (see page 143) Designing for Performance Teams can optimize an application for high availability, ease of programming, flexibility and reusability of code, and performance. The most thorough application design must consider all of these areas. If a design team commits the following errors, major performance problems can occur: Overlooks performance Considers it late in the design cycle Fails to understand, test, and address performance before production Designing an application for performance can help avoid many performance problems after the application has been deployed. Proactive performance engineering can help eliminate the redesign, recoding, and retrofitting during implementation to satisfy performance expectations or alleviate performance problems. Proactive performance engineering also lets you analyze and stress test the designs before you implement them. Using the techniques and tools in this handbook can turn performance engineering research and testing from a lengthy process inhibiting development to one that can be performed efficiently before development. Many options exist to index and table design for performance. These options center on the type of application that you are implementing, specifically the type of data access (random versus sequential, read versus update, and so on). In addition, the application has to be properly designed for performance. The design process requires an understanding of the application data access patterns, as well as the units of work (UOWs) and inputs. Considering these points and designing the application and database accordingly are the best defense against performance problems. 14 CA Performance Handbook for DB2 for z/os

15 Identifying Performance Problems More Information: Designing Tables and Indexes for Performance (see page 59) EXPLAIN Facility and Predictive Analysis (see page 79) Identifying Performance Problems How do we find performance problems? Are SQL statements in an application causing problems? Does the area of concern involve the design of our indexes and tables? Or, perhaps our data has become disorganized, or the catalog statistics do not accurately reflect the data in the tables? It can also be an issue of dynamic SQL, and poor performing statements, or perhaps the overhead of preparing the dynamic SQL statements. When it comes to database access, finding and fixing the worst performing SQL statements is the top priority. However, how does one prioritize the worst culprits? Are problems caused by the statements that are performing table space scans or the ones with matching index access? Performance is relevant, and finding the least efficient statement is a matter of understanding how that statement is performing relative to the business need that it serves. This handbook can help you prioritize site-specific performance needs. Fixing the Performance Problems After you identify a performance problem, determine the best approach to address it. The problem could be updating catalog statistics, tuning an SQL statement, or adjusting parameters for indexing, table space, or subsystem. With the proper monitoring in place, you can identify, understand, and fix the problem. More Information: Understand and Tune Your Packaged Applications (see page 143) Designing Tables and Indexes for Performance (see page 59) Application Design and Tuning for Performance (see page 105) Tuning Subsystems (see page 125) Tuning Tips (see page 153) Chapter 1: Introducing DB2 Performance 15

16

17 Chapter 2: SQL and Access Paths DB2 offers you a level of abstraction from the data because you do not have to know the following information: Where the data physically exists Where the data sets are Which is the access method for the data What index to use All that you should know is the name of the table and the columns you are interested in. You can quickly build applications that access data using standardized language that is independent of any access method or platform. This design enables development of applications, portability across platforms, and all of the power and flexibility that comes with a relational or object-relational design. The DB2 Optimizer The DB2 optimizer performs the following functions: Accepts a statement Checks for syntax Checks the DB2 system catalog Builds access paths to the data The DB2 optimizer uses catalog statistics to attach a cost to the possible access paths and then chooses the cheapest path. Understanding the DB2 optimizer, the DB2 access paths, and how your data is accessed helps you adjust the DB2 performance. The DB2 optimizer is responsible for interpreting your queries and determining how to access your data in the most efficient manner. However, it can use only the best of the available access paths. The DB2 optimizer does not know what access paths are possible that are not available. The DB2 optimizer can deal only with the information that you provide. This information is primarily stored in the DB2 system catalog, which includes the basic information about the tables, columns, indexes, and statistics. The DB2 optimizer does not know about the indexes that could be built, or anything about the possible inputs to our queries (unless all literal values are provided in a query), input files, batch sequences, or transaction patterns. Therefore, an understanding of the DB2 optimizer, statistics, and access paths is important, but the design of applications and databases for performance is also important. Chapter 2: SQL and Access Paths 17

18 Importance of Catalog Statistics The DB2 optimizer interprets your queries and determines the most efficient method to access your data. The DB2 optimizer can use access paths that are only available when determining the most efficient way to access your data. If other access paths exist but are not available, these access paths are not considered. The DB2 optimizer can use only the information you provide when determining access paths and interpreting queries. The primary location for this information is the DB2 system catalog, which includes the basic information about tables, columns, indexes, and statistics. However, the DB2 optimizer does not know about indexes that could be built, possible inputs to queries, input files, batch sequences, or transactions patterns unless you specifically provide this information. Importance of Catalog Statistics Catalog statistics include information about the amount of data and the organization of that data in DB2 tables. The DB2 optimizer selects access paths to the data based on cost, so catalog statistics are critical to proper performance. Maintaining accurate, up-to-date statistics is critical to performance. The DB2 optimizer uses various statistics to determine the cost of accessing the data in tables. The type of query (static or dynamic), use of literal values, and run-time optimization options dictates which statistics are used. The cost that is associated with various access paths affects whether indexes are chosen, or which indexes are chosen, the access path, and table access sequence for multiple table access paths. In addition to catalog statistics and database design, the DB2 optimizer uses other information to calculate and choose an efficient access path. The following information is used to determine the access path: Model of the central processor Number of processors Size of the RID pool Various installation parameters Buffer pool sizes Ensure that you are aware of these factors as you tune your queries. 18 CA Performance Handbook for DB2 for z/os

19 Importance of Catalog Statistics Following are the three types of DB2 catalog statistics: Cardinality Frequency Distribution Histogram Each type provides different details about the data in your tables, and DB2 uses these statistics to determine the best way to access the data dependent on the objects and the query. Cardinality Statistics Cardinality statistics reflect the number of rows in a table or the number of distinct values for a column in a table. These statistics provide the main source for the filter factor. Filter factor is a percentage of the number of rows that are expected to be returned for a column or a table. DB2 uses cardinality statistics to determine which of the following methods to use to access data: Whether access is using an index scan or a table space scan Which table to access first when joining tables DB2 needs an accurate estimate of the number of rows that qualify after applying various predicates in your queries to determine the optimal access path. When multiple tables are accessed, the number of qualifying rows that are estimated for the tables can also affect the table access sequence. Column and table cardinalities provide the basic, but critical information for the DB2 optimizer to make these estimates. Example: EMP Table Access When DB2 has to choose the most efficient way to access the EMP table, this access depends on the following factors: Size of the table Cardinality of the SEX column Any index that might be defined on the table DB2 uses the catalog statistics to determine the filter factor for the SEX column. In this case, 1/COLCARDF, COLCARDF is a column in the SYSIBM.SYSCOLUMNS catalog table. The resulting fractional number represents the number of rows that are expected to be returned based on the predicate. DB2 uses this filter factor number to decide whether to use an index (if available) or table space scan. Chapter 2: SQL and Access Paths 19

20 Importance of Catalog Statistics Example: EMP Table Access Path If the cardinality of the SEX column is three (male, female, and unknown), the table has three unique occurrences of a value of SEX. In this case, DB2 determines a filter factor of 1/3, or 33 percent of the values. Consider that the cardinality of the table, as reflected in the CARDF column of the SYSIBM.SYSTABLES catalog table, is 10,000 employees. Then, the estimated number of rows that are returned from the query is 3,333. DB2 uses this information to decide which access path to choose. Consider that you are using the predicate in the following query in a join to another table. The filter factor that is calculated will be used not only to determine the access path to the EMP table, but it could also influence which table will be accessed first in the join. SELECT EMPNO FROM EMP WHERE SEX = :SEX Example: Column Groups Statistics can also be gathered in groups of columns. This method is called column correlation statistics. This method is important because the DB2 optimizer can use only the information it is given. The following command shows an example that uses this method: SELECT E.EMPNO, D.DEPTNO, D.DEPTNAME FROM EMP E INNER JOIN DEPT D ON E.WORKDEPT = D.DEPTNO WHERE LASTNAME = :LAST-NAME Example: Filter Factor Imagine that the amount of money that someone is paid is correlated to the amount of education they have received. For example, an intern with one year of college is not paid as much as an executive with an MBA. However, if the DB2 optimizer is not given this information, it has to multiply the filter factor for the SALARY predicate by the filter factor for the EDLEVEL predicate. This calculation may result in an exaggerated filter factor for the table, and it could negatively influence the choice of an index or the table access sequence of a join. For this reason, it is important to gather column correlation statistics on any columns that might be correlated. More conservatively, you may want to gather column correlation statistics on any columns that are referenced together in the WHERE clause. The following command shows an example that uses this method: SELECT E.EMPNO FROM EMP E WHERE SALARY > :SALARY AND EDLEVEL = :EDLEVEL 20 CA Performance Handbook for DB2 for z/os

21 Importance of Catalog Statistics Frequency Distribution Statistics Frequency distribution statistics reflect the percentage of frequently occurring values. You can collect the information about the percentage of values that occur most or least frequently in a column in a table. These frequency distribution statistics can dramatically impact the performance of the following types of queries: Dynamic or static SQL with embedded literals Static SQL with host variables bound with REOPT(ALWAYS) Static or dynamic SQL against nullable columns with host variables that cannot be null Dynamic SQL with parameter markers bound with REOPT(ONCE), REOPT(ALWAYS), or REOPT(AUTO) Note: DB2 can collect distribution statistics using the RUNSTATS utility. In some cases, cardinality statistics are not enough. With only cardinality statistics, the DB2 optimizer has to assume that the data is evenly distributed. For example, consider the SEX column from the previous queries. The column cardinality is 3. However, it is common that the values in the SEX column will be M for male or F for female, with each representing approximately 50 percent of the values. The value U occurs only a small fraction of the time. This example is called skewed distribution. Skewed distributions can have a negative influence on query performance if DB2 does not know about the distribution of the data in a table, the input values in a query predicate, or both. In the following query: SELECT EMPNO FROM EMP WHERE SEX = :SEX Likewise, if the following statement is issued: SELECT EMPNO FROM EMP WHERE SEX = 'U' Most of the values are M or F, and DB2 has only cardinality statistics available, then the same formula and filter factor applies. In such situations, the distribution statistics can pay off. Chapter 2: SQL and Access Paths 21

22 Importance of Catalog Statistics If frequency distribution statistics for the SEX column are gathered in the previous examples, the following frequency values in the SYSIBM.SYSCOLDIST table may occur (simplified in this example): VALUE FREQUENCY M.49 F.49 U.01 If DB2 has this information, and if the following query is issued: SELECT EMPNO FROM EMP WHERE SEX = 'U' DB2 can determine that the value U represents about one percent of the values of the SEX column. This additional information can have dramatic effects on the access path and table access sequence decisions. Histogram Statistics Histogram statistics summarize data distribution on an interval scale and span the entire distribution of values in a table. Histogram statistics divide values into quantiles. The quantile defines an interval of values. The quantity of intervals is determined using a specification of quantiles in a RUNSTATS execution. Thus, a quantile represents a range of values within the table. Each quantile contains approximately the same percentage of rows. This percentage can be more than the distribution statistics in that all values are represented. Histogram statistics improve on distribution statistics because they provide value-distribution statistics that are collected over the entire range of values in a table. Collecting statistics over the entire table goes beyond the capabilities of distribution statistics because distribution statistics are limited to only the most or least occurring values in a table. To query on the middle initial of a person's name, the initials are most likely in the range of A to Z. If you used only cardinality statistics, any predicate referencing the middle initial column would receive a filter factor of 1/COLARDF or 1/26. If you have used distribution statistics, any predicate that used a literal value could take advantage of the distribution of values to determine the best access path. That determination would depend on the value being recorded as one of the most or least occurring values. If it is not, the filter factor is determined based on the difference in frequency of the remaining values. Histogram statistics eliminate this guesswork. 22 CA Performance Handbook for DB2 for z/os

23 Access Paths If histogram statistics are calculated for the middle initial, they would appear similar to the following values: QUANTILE LOWVALUE HIGHVALUE CARDF FREQ 1 A G % 2 H L % 3 M Q % 4 R U % 5 V Z % The DB2 optimizer has information about the entire span of values in the table and any possible skew. This information is especially useful for the following range predicates: SELECT EMPNO FROM EMP WHERE MIDINIT BETWEEN 'A' and 'G' Access Paths DB2 can use the following access paths when accessing the data in your database: Tablespace Scan Limited Partition Scan Index Access List Prefetch Nested Loop Join Hybrid Join Merge Scan Join Star Join The access paths are exposed using the EXPLAIN facility. You can invoke the EXPLAIN facility using one of these techniques: EXPLAIN SQL statement EXPLAIN(YES) BIND or REBIND parameter Visual EXPLAIN Optimization Service Center Chapter 2: SQL and Access Paths 23

24 Access Paths Table space Scans You can use the DB2 table space scan to search an entire table space to satisfy a query. The type of table space determines what is scanned. Simple table spaces will be scanned in their entirety. Segmented and universal table spaces will be scanned for only the table specified in the query. Note: An R in the PLAN_TABLE ACCESSTYPE column indicates a table space. A table space scan is selected as an access method when: A matching index scan is not possible because no index is available or no predicates match the index column. DB2 calculates that a high percentage of rows is retrieved, so any index access would not be beneficial. The indexes that have matching predicates have a low cluster ratio and are not efficient for large amounts of data that the query is requesting. Limited Partition Scan and Partition Elimination DB2 uses partitioning key values to eliminate partitions from a table space scan. DB2 uses the column values that define the partitioning key to eliminate partitions from the query access path. You must code queries explicitly to eliminate partitions. This partition elimination can apply to table space scans and index access. For example, if the EMP table is partitioned by the EMPNO column, the following query would eliminate any partitions not qualified by the predicate: SELECT EMPNO, LASTNAME FROM EMP WHERE EMPNO BEWTEEN '200000' AND '299999' If an index was available on the EMPNO column, DB2 can choose an index-based access method. DB2 could also choose a table space scan, but use the partitioning key values to access only the partition in which the predicate matches. 24 CA Performance Handbook for DB2 for z/os

25 Access Paths If the partitioning key values are supplied in a predicate, DB2 can also choose a secondary index for the access path and can still eliminate partitions using those values. For example, in the following query DB2 can choose a partitioned secondary index on the WORKDEPT column and can eliminate partitions based on the range provided in the query for the EMPNO: SELECT EMPNO, LASTNAME FROM EMP WHERE EMPNO BEWTEEN '200000' AND '299999' AND WORKDEPT = 'C01' Index Access DB2 uses indexes on your tables to perform these actions: Access the data based on the predicates in the query Avoid a sort in support of the use of the DISTINCT clause ORDER BY and GROUP BY clauses INTERSECT and EXCEPT processing DB2 matches the predicates in your queries to the leading key columns of the indexes that are defined against your tables. The MATCHCOLS column of the PLAN_TABLE indicates this match. If the number of columns matched is greater than zero, the index access is considered as a matching index scan. If the number of matched columns is equal to zero, the index access is considered as a non-matching index scan. In a non-matching index scan, all of the key values and their record identifiers (RIDs) are read. Note: A non-matching index scan is used if DB2 can use the index to avoid a sort when the query contains any of the following clauses: ORDER BY DISTINCT GROUP BY NTERSECT EXCEPT Chapter 2: SQL and Access Paths 25

26 Access Paths The matching predicates on the leading key columns are equal (=) or IN predicates. This match would correspond to an ACCESSTYPE value of either I or N, respectively. The predicate that matches the last index column can be an equal, IN, NOT NULL, or a range predicate (<, <=, >, >=, LIKE, or between). For example, the following query assumes that the EMPPROJACT DB2 sample table has an index on the PROJNO, ACTNO, EMSTDATE, and EMPNO columns: SELECT EMENDATE, EMPTIME FROM EMPPROJACT WHERE PROJNO = 'AD3100' AND ACTNO IN (10, 60) AND EMSTDATE > ' ' AND EMPNO = In the previous example, DB2 chooses a matching index scan matching on the PROJNO, ACTNO, and EMSTDATE columns. DB2 does not choose a match on the EMPNO column because the previous predicate is a range predicate. However, the EMPNO column can be applied as an index screening column. Because the EMPNO column is a stage 1 predicate, DB2 can apply it to the index entries after the index is read. Although the EMPNO column does not limit the range of entries that are retrieved from the index, it can eliminate the entries as the index is read. This elimination reduces the number of data rows that have to be retrieved from the table. If all of the columns specified in a statement can be found in an index, DB2 may choose index only access. The value Y in the INDEXONLY column of the PLAN_TABLE indicates the index only access. DB2 can access indexes using multi-index access. An ACCESSTYPE of M in the PLAN_TABLE indicates the multi-index access. A multi-index access involves reading multiple indexes or the same index multiple times, gathering qualifying RID values together in a union or intersection of values and then sorting them in data page number sequence. In this way, a table can still be accessed efficiently for more complicated queries. With multi-index access, each operation is represented in the PLAN_TABLE in an individual row. These steps include the matching index access (ACCESSTYPE = MX or DX) for regular indexes or DOCID XML indexes, and then unions or intersections of the RIDs (ACCESSTYPE = MU, MI, DU, or DI). For example, the following SQL statement could use multi-index access if an index exists on the LASTNAME and FIRSTNME columns of the EMP table: SELECT EMPNO FROM EMP WHERE (LASTNAME = 'HAAS' AND FIRSTNME = 'CHRISTINE') OR (LASTNAME = 'CHRISTINE' AND FIRSTNME = 'HAAS') 26 CA Performance Handbook for DB2 for z/os

27 Access Paths With the previous query, DB2 could possibly access the index twice, union the resulting RID values together (due to the OR condition in the WHERE clause), and then access the data. Virtual Buffer Pools You can have up to 80 virtual buffer pools. This feature allows up to any one of the following buffer pool sizes: Fifty 4-KB page buffer pools BP0 - BP49 Ten 32-KB page buffer pools BP32K - BP32K9 Ten 8-KB page buffer pools Ten 16-KB page buffer pools The physical memory available on your system limits the size of the buffer pools, with a maximum size for all buffer pools of 1 TB. It does not use additional resources to search a large pool versus a small pool. If you exceed the available memory in the system, the system begins swapping pages from physical memory to disk. This swapping can severely affect performance. What EXPLAIN Tells You The PLAN_TABLE is the key table for determining the access path for a query. The PLAN_TABLE provides the following critical information: METHOD Indicates the joint method, or whether an additional sort step is required. ACCESSTYPE Indicates whether the access to is through a table space scan or through index access. If it is by index access, the specific type of index access is indicated. MATCHCOLS Indicates the number of columns that are matched against the index if an index access is used. INDEXONLY Indicates that the access required can be served by accessing the index only, and avoiding any table access when a value of Y is in this column. Chapter 2: SQL and Access Paths 27

28 Access Paths SORTN####, SORTC#### Indicates any sorts that may happen in support of a UNION, grouping, joint operation, and so on. PREFETCH Indicates whether the prefetch can play a role in the access. By manually running EXPLAIN and examining the PLAN-TABLE, you can retrieve the following information: Access path Indexes that are used Join operations Any sorting that occurs as part of your query If you have additional EXPLAIN tables (for example, those created by Visual Explain or the Optimization Service Center), those tables are populated automatically. This is done by using those tools or by manually running EXPLAIN. If you do not have the remote access that is required from a PC to use the tools, you can also query those tables manually. The tables provide detailed information about such entities as predicates stages, filter factors, eliminated partitions, parallel operations, detailed cost information, and so on. Note: For more information about tables, see the DB2 Performance Monitoring and Tuning Guide (DB2 9). List Prefetch List prefetch is used for access to a moderate number of rows when access to the table is by non-clustering or clustering index when the data is disorganized. List prefetch is less useful for queries that process large quantities of data or small amounts of data. The preferred method of table access is by using a clustered index. When an index is defined as clustered, DB2 tries to keep the data in the table in the same sequence as the key values in the clustering index. When the table is accessed using the clustering index and the data in the table is organized, the access to the data in the table typically is sequential and predictable. If accessing the data in a table using a non-clustering index or the clustering index with a low cluster ratio, the access to the data can be random and unpredictable. In addition, the same data pages could be read multiple times. 28 CA Performance Handbook for DB2 for z/os

29 Access Paths When DB2 detects that a non-clustering index or a clustering index that has a low cluster ratio is used to access a table, DB2 may choose the list prefetch index access method instead. Note: List prefetch is indicated in the PLAN_TABLE with a value of L in the PREFETCH column. List prefetch accesses the index using a matching index scan of one or more indexes and collects the RIDs into the RID pool. RID pool is a common area of memory. The RIDs are sorted in ascending order by the page number, and the pages in the table space are retrieved in a sequential or a skip sequential way. If DB2 determines that the RIDs to be processed will take more than 50 percent of the RID pool when the query is executed, list prefetch is not chosen as the access path. If DB2 determines that more than 25 percent of the rows of the table must be accessed, list prefetch can terminate during the execution. These situations are known as RDS failures. During RDS failures, the access path changes to a table space scan. Nested Loop Join The nested loop join is the access method that DB2 has selected when joining two tables together and you do not specify join columns. A nested loop join repeatedly accesses the outer table as rows are accessed in the inner table. The nested loop join is most efficient when a few rows qualify for the inner table. The nested loop join is a good access path for transactions that are processing little or no data. Note: Nested loop join is indicated using a value of 1 in the METHOD column of the PLAN_TABLE. In a nested loop join, DB2 scans the outer or first table accessed in the join operation using a table space scan or index access. For each qualifying row in that table, DB2 searches for matching rows in the inner or second table accessed. DB2 concatenates any rows that it finds in both tables. An index that supports the join on the outer table is important. For the best performance, that index should be a clustering index in the same order as the inner table. In this way, a join operation can be a sequential and efficient process. DB2 can also dynamically build a sparse index on the outer table when no index is available on that table. If DB2 determines one of the following facts, it may sort the inner table: The join columns are not in the same sequence as the outer table. The join columns are not in an index. The join columns are in a clustering index where the table data is disorganized. Chapter 2: SQL and Access Paths 29

30 Access Paths Hybrid Join When DB2 detects that the inner table access is a non-clustering index or a clustering index in which the cluster ratio is low, the hybrid join access method is used. Note: The hybrid join is indicated with a value of 4 in the METHOD column of the PLAN_TABLE. In a hybrid join, DB2 scans the outer table using a table space scan or an index. DB2 then joins the qualifying rows of the outer table with the RIDs from the matching index entries. DB2 creates a RID list for all the qualifying RIDs of the inner table. DB2 sorts the outer table and RIDs creating a sorted RID list and an intermediate table. DB2 then accesses the inner table using the list prefetch and joins the inner table to the intermediate table. Hybrid join can outperform a nested loop join when the inner table access is a non-clustering index or clustering index for a disorganized table. The hybrid join is better if the equivalent nested loop join would be accessing the inner table in a random fashion and accessing the same pages multiple times. Hybrid join takes advantage of the list prefetch processing on the inner table to read all the data pages only once in a sequential or skip sequential manner. The hybrid join is also better when the query processes more than a tiny amount of data or less than a very large amount of data. Because the list prefetch is used, the hybrid join can experience RID list failures. RID list failures result in one of the following actions: A table space scan of the inner table on initial access to the inner table. A restart of the list prefetch processing on subsequent accesses. If you are experiencing RID list failures in a hybrid join, try to encourage DB2 to use a nested loop join. Merge Scan Join Merge scan join is used as the table access method when DB2 detects one of the following facts: Many rows of the inner and outer table qualify for the join. The tables have no indexes on the matching columns of the join. Note: A merge scan join is indicated with a value of 2 in the METHOD column of the PLAN_TABLE. 30 CA Performance Handbook for DB2 for z/os

31 Read Mechanisms In a merge scan join, DB2 scans both tables in the order of the join columns. If no efficient indexes are providing the join order for the join columns, DB2 sorts the outer table, the inner table, or both tables. The inner table is placed into a work file. If the outer table has to be sorted, it is placed into a work file. DB2 then reads both tables and join the rows together using a match/merge process. Star Join DB2 employs a star join when it detects that the query is joining tables that are a part of a star schema design of a data warehouse. Note: The star join is indicated with the value of S in the JOIN_TYPE column of the PLAN_TABLE. Read Mechanisms DB2 can apply read mechanisms to enhance performance when accessing your data. These performance enhancers can have a dramatic effect on the performance of your queries and applications. These read mechanisms are as follows: Sequential Prefetch Dynamic Prefetch and Sequential Detection Index Lookaside Sequential Prefetch When your query reads data in a sequential manner, DB2 elects to use the sequential prefetch to read the data ahead of the query requesting it. DB2 launches asynchronous read engines to read data from the index and table page sets into the DB2 buffers. This prefetch loads the data pages into the buffers in expectation of the query accessing them. The size of the buffer pool that is used determines the maximum number of pages that are read by a sequential prefetch operation. When the buffer pool is smaller than 1000 pages, the prefetch quantity is limited due to the risk of filling up the buffer pool. If the buffer pool has more than 1000 pages, 32 pages can be read with a single physical I/O. Note: A value of S in the PREFETCH column of the PLAN_TABLE indicates the sequential prefetch. Chapter 2: SQL and Access Paths 31

32 Read Mechanisms Dynamic Prefetch and Sequential Detection DB2 can perform the sequential prefetch dynamically at execution time. This method is called dynamic prefetch. This method uses a technique called sequential detection to detect the forward reading of table pages and index leaf pages. DB2 tracks the pattern of pages that is accessed to detect sequential or near sequential access. The most recent eight pages are tracked, and data access is determined to be sequential if more than four of the last eight pages are page-sequential. Note: A page is considered page-sequential if it is within one half of the prefetch quantity for the buffer pool. Five of eight sequentially available pages activate dynamic prefetch. If DB2 determines that the query or application is no longer reading sequentially, it can deactivate the dynamic prefetch. If a query is likely to get dynamic prefetch at execution time, DB2 indicates in an access path. Note: A value of D in the PREFETCH column of the PLAN_TABLE indicates the dynamic prefetch. The area of memory that is used to track the last eight pages that are accessed is destroyed and rebuilt for RELEASE(COMMIT). So, dynamic prefetch is dependent on the bind parameter RELEASE(DEALLOCATE). Because RELEASE(DEALLOCATE) is not effective for remote connections, sequential detection and dynamic prefetch are less likely for remote applications. Index Lookaside Index lookaside is a performance enhancer that is active only after an initial access to an index. For repeated probes of an index, DB2 checks the most current index leaf page to see if the key is found on that page. If DB2 finds the key, DB2 does not read from the root page of the index down to the leaf page, but it uses the values from the leaf page. This method can affect performance because getpage operations and I/Os can be avoided. Index lookaside depends on the query reading index data in a predictable pattern. This happens for transactions that are accessing the index using a common cluster with other tables. Index lookaside is dependent on the RELEASE(DEALLOCATE) bind parameter. 32 CA Performance Handbook for DB2 for z/os

33 Sorting Sorting Sorting is used to support certain clauses in your SQL statement or in support of certain access paths. DB2 can sort in support of the following functions: ORDER BY clause GROUP BY clause To remove duplicates (DISTINCT, UNION, EXCEPT, or INTERSECT) In join or subquery processing Sorting in an application is always faster than sorting in DB2 for the small result sets. However, by placing the sort in the program, you remove some of the flexibility, power, and portability of the SQL statement, along with the ease of coding. Avoiding a Sort for an ORDER BY DB2 can avoid a sort for an ORDER BY by using the leading columns of an index to match the ORDER BY columns. If the columns are the leading columns of the index that are used to access a single table or the leading columns of the index to access the first table in a join operation. DB2 uses order by pruning to avoid a sort. As an example, in this query, we have an index on WORKDEPT, LASTNAME, and FIRSTNME. DB2 avoids this sort if it uses the index on those three columns to access the table. SELECT * FROM EMP WHERE WORKDEPT = 'D01' ORDER BY WORKDEPT, LASTNAME, FIRSTNME DB2 can also avoid a sort in one of the following situations: When any number of the leading columns of an index matches the ORDER BY clause. When the query contains an equals predicate on the leading columns ORDER BY pruning. Chapter 2: SQL and Access Paths 33

34 Sorting In this example, the following queries avoid a sort if the index is used: SELECT * FROM EMP ORDER BY WORKDEPT SELECT * FROM EMP WHERE WORKDEPT = 'D01' ORDER BY WORKDEPT, LASTNAME Avoiding a Sort for a GROUP BY A GROUP BY can use any of the following methods to avoid a sort: Unique index Duplicate index Full or partial index key If the index on WORKDEPT, LASTNAME, and FIRSTNME is used, you can avoid each of the following statements: SELECT WORKDEPT, LASTNAME, FIRSTNME, COUNT(*) FROM EMP GROUP BY WORKDEPT, LASTNAME, FIRSTNAME SELECT WORKDEPT, COUNT(*) FROM EMP GROUP BY WORKDEPT SELECT WORKDEPT, FIRSTNME, COUNT(*) FROM EMP WHERE LASTNAME = 'SMITH' GROUP BY WORKDEPT, FIRSTNAME 34 CA Performance Handbook for DB2 for z/os

35 Parallelism Avoiding a Sort for a DISTINCT A DISTINCT can avoid a sort by using a unique index only if that index is used to access the table. You can use a GROUP BY on all of the columns in the SELECT list to take advantage of the additional capabilities of GROUP BY to avoid a sort. As an example, the following two statements are identical: SELECT DISTINCT WORKDEPT, LASTNAME, FIRSTNME FROM EMP SELECT WORKDEPT, LASTNAME, FIRSTNME FROM EMP GROUP BY WORKDEPT, LASTNAME, FIRSTNME Avoiding a Sort for a UNION EXCEPT or INTERSECT DB2 can avoid a sort for an EXCEPT or INTERSECT. DB2 cannot avoid a sort for a union in one of the following situations: If there is a matching index with the UNIONed, intersected, and excepted columns. If the inputs to the operation are already sorted by a previous operation. DB2 cannot avoid a sort for a union, but if duplicates are possible change each to a UNION ALL, EXCEPT ALL, or INTERSECT ALL. Parallelism DB2 uses parallel operations to access tables and indexes. This configuration impacts performance for queries that process large quantities of data across multiple table and index partitions. The response time for these data-or processor-intensive queries can be reduced. The two types of query parallelism are as follows: Query I/O parallelism Query CP parallelism I/O parallelism manages concurrent I/O requests for a single query. I/O parallelism can fetch data using these multiple concurrent I/O requests into the buffers and significantly reduce the response time for large I/O bound queries. Query CP parallelism breaks a large query into smaller queries and runs the smaller queries in parallel on multiple processors. This type of parallelism can also significantly reduce the elapsed time for large queries. Chapter 2: SQL and Access Paths 35

36 Parallelism You can enable query parallelism by using the DEGREE ANY parameter of a BIND command or by setting the value of the CURRENT DEGREE special register to the value of ANY. Because there is overhead to the start of the parallel tasks, ensure that you use query parallelism for larger queries. For small queries, query parallelism can be a performance detriment. 36 CA Performance Handbook for DB2 for z/os

37 Chapter 3: Predicates and SQL Tuning To write efficient SQL statements and tune SQL statements, you need a basic understanding of how DB2 optimizes SQL statements and how it accesses your data. When writing and tuning SQL statements, filter as much as possible as early as possible. The most expensive thing you can do is to travel from your application to DB2. Filter as close to the indexes and data as possible while processing in DB2. Understand how to reduce repeat processing. The most efficient SQL statement is the statement that never executes. Do only what is necessary to complete the job. DB2 Predicates Predicates in SQL statements are classified. These classifications dictate how DB2 processes the predicates and how much data is filtered during the process. These classifications are as follows: Stage 1 and Stage 1 Indexable Stage 2 Stage 3 The DB2 stage 1 engine understands your indexes and tables and can use an index for efficient access to your data. Only a stage 1 predicate can limit the range of data accessed on a disk. The stage 2 engine processes functions and expressions. But, it is unable to access data in indexes and tables directly. Data from stage 1 is passed to stage 2 for further processing. So, stage 1 predicates are more efficient than stage 2 predicates. Stage 2 predicates cannot use an index, and thus cannot limit the range of data retrieved from a disk. A stage 3 predicate is processed in the application layer. Filtering is performed after the data is retrieved from DB2 and processed in the application. Stage 3 predicates are the least efficient. The IBM DB2 Performance Monitoring and Tuning Guide contains lists of the various predicate forms, and whether they are stage 1 indexable, stage 1, or stage 2. Chapter 3: Predicates and SQL Tuning 37

38 DB2 Predicates Stage 1 Indexable Stage 1 indexable predicates offer the best performance for filtering. However, just because a predicate is listed as stage 1 indexable does not mean that the predicate is used for an index or processed in stage 1. Many other factors must also be in place. Following are the first and second points that determine if a predicate is stage 1: The syntax of the predicate. The type and length of constants used in the predicate. If the predicate, which is classified as a stage 1 predicate, is evaluated after a join operation, it is a stage 2 predicate. All indexable predicates are stage 1, but not all stage 1 predicates are indexable. You can use stage 1 indexable predicates as predicates to match the columns of an index. Example of a Stage 1 Predicate: col op value col op Value Indicates a column of a table. Indicates an operator such as =, >, <, >=, <=, and so on. Indicates an expression that does not contain a column from the table (a non-column expression). Predicates that contain BETWEEN, IN (for a list of values), and LIKE (without a leading search character) can also be stage 1 indexable. When an index is used, the stage 1 indexable predicate provides the best filtering. Because the stage 1 indexable predicate can actually limit the range of data accessed from the index. Try to use stage 1 matching predicates whenever possible. Assuming that an index exists on the EMPNO column of the EMP table, the predicate in the following query is a stage 1 indexable predicate: SELECT LASTNAME, FIRSTNME FROM EMP WHERE EMPNO = '000010' 38 CA Performance Handbook for DB2 for z/os

39 DB2 Predicates Other Stage 1 Just because a predicate is stage 1 does not mean that it can use an index. Some stage 1 predicates are not available for index access. These predicates (again, the best reference is the chart in the IBM manuals) are generally of the form col NOT op value, where col is a column of a table, op is an operator, and value represents a non-column expression, host variable, or value. Predicates containing NOT BETWEEN, NOT IN (for a list of values), NOT LIKE (without a leading search character), or LIKE (with a leading search character) can also be stage 1 indexable. Although non-indexable stage 1 predicates cannot limit the range of data that is read from an index, they are available as index screening predicates. The following is an example of a non-indexable stage 1 predicate: SELECT LASTNAME, FIRSTNME FROM EMP WHERE EMPNO <> '000010' Stage 2 DB2 manuals provide a complete description of when a predicate can be stage 2 versus stage 1. Generally, stage 2 occurs after data accesses and performs such actions as sorting and evaluation of functions and expressions. Stage 2 predicates cannot take advantage of indexes to limit the data access and are more expensive than stage 1 predicates because they are evaluated later in the processing. Stage 2 predicates generally contain column expressions, correlated subqueries, CASE expressions, and so on. A predicate can also appear to be stage 1, but it can be processed as stage 2. For example, any predicate that is processed after a join operation is stage 2. Although DB2 promotes mismatched data types to stage 1 through casting (as of version DB2 V8), some predicates with mismatched data types are stage 2. One example is a range predicate comparing a character column to a character value that exceeds the length of the column. The following are examples of stage 2 predicates (EMPNO is a character column of fixed length 6): SELECT LASTNAME, FIRSTNME FROM EMP WHERE EMPNO > ' ' SELECT LASTNAME, FIRSTNME FROM EMP WHERE SUBSTR(LASTNAME,1,1) = 'B' Chapter 3: Predicates and SQL Tuning 39

40 DB2 Predicates Stage 3 A stage 3 predicate is a fictitious predicate that is made up to describe filtering that occurs in the application program after retrieving data from DB2. Avoid stage 3 predicates. Performance is best served when all filtering is done in the data server and not in the application code. Imagine a COBOL program has read the EMP table. After reading the EMP table, the following statement is executed: IF (BONUS + COMM) < (SALARY*.10) THEN CONTINUE ELSE PERFORM PROCESS-ROW END-IF. The previous SQL is an example of a stage 3 predicate. Data that is retrieved from the table is not used unless the condition is false. A stage 2 predicate always outperforms a stage 3 predicate. The following is an example of a stage 2 predicate that performs the same function as the previous stage 3 predicate: SELECT EMPNO FROM EMP WHERE BONUS + COMM >= SALARY * 0.1 Combining Predicates When simple predicates are connected by an OR condition, the resulting compound predicate is evaluated at the higher stage of the two simple predicates. The following example contains two simple predicates that are combined by an OR. The first predicate is stage 1 indexable, and the second is non-indexable stage 1. The result is that the entire compound predicate is stage 1 and not indexable: SELECT EMPNO FROM EMP WHERE WORKDEPT = 'C01' OR SEX <> 'M' In the next example, the first simple predicate is stage 1 indexable, but the second (connected again by an OR) is stage 2. Thus, the entire compound predicate is stage 2: SELECT EMPNO FROM EMP WHERE WORKDEPT = 'C01' OR SUBSTR(LASTNAME,1,1) = 'L' 40 CA Performance Handbook for DB2 for z/os

41 DB2 Predicates Boolean Term Predicates A Boolean term predicate is a simple or compound predicate, that when evaluated false for the row, makes the entire WHERE clause evaluate to false. This is important because only Boolean term predicates are considered for single index access. At best, non-boolean term predicates can at best be considered for multi-index access. The following example contains Boolean term predicates: WHERE LASTNAME = 'HAAS' AND MIDINIT = 'T' If the predicate on the LASTNAME column evaluates as false, the entire WHERE clause is false. The same is true for the predicate on the MIDINIT column. DB2 can take advantage of this because it can use an index (if available) on the LASTNAME column or the MIDINIT column. This is opposed to the following WHERE clause: WHERE LASTNAME = 'HAAS' OR MIDINIT = 'T' In this case, if the predicate on LASTNAME or MIDINIT evaluates as false, the rest of the WHERE clause must be evaluated. These predicates are non-boolean term. You can modify predicates in the WHERE clause to take advantage of this to improve index access. The following query contains non-boolean term predicates: WHERE LASTNAME > 'HAAS' OR (LASTNAME = 'HAAS' AND MIDINIT > 'T') A redundant predicate can be added that makes the WHERE clause functionally equivalent. This Boolean term predicate can make better use of an index on the LASTNAME column: WHERE LASTNAME >= 'HAAS' AND (LASTNAME > 'HAAS' OR (LASTNAME = 'HAAS' AND MIDINIT > 'T')) Predicate Transitive Closure DB2 can add redundant predicates if it determines that the predicate can be applied by transitive closure. Remember the rule of transitive closure: If a=b and b=c, then a=c. DB2 can take advantage of this closure to introduce redundant predicates. For example, in a join between two tables such as the following: SELECT * FROM T1 INNER JOIN T2 ON T1.A = T2.B WHERE T1.A = 1 Chapter 3: Predicates and SQL Tuning 41

42 Coding Efficient SQL DB2 generates a redundant predicate on T2.B = 1. This predicate gives DB2 more choices for filtering and table access sequence. You can add your own redundant predicates; however, DB2 does not consider them when it applies predicate transitive closure, and so they are redundant. Predicates of the form col op value, where the operation is an equals or range predicate, are available for predicate transitive closure. BETWEEN predicates are also available for this type of closure. However, predicates that contain a LIKE, an IN, or subqueries are not available for transitive closure, and so it may benefit you to code those predicates redundantly if you believe they can provide a benefit. Coding Efficient SQL The following views are popular in the database sector: The most efficient SQL statement is the SQL statement that is never executed. The most efficient SQL statement is the one that is never written. Some truth exists in both views. You need to perform two tasks when you place SQL statements into your applications. Minimize the number of times you go to the data server. Go to the data server in the most efficient way. Avoid Unnecessary Processing Ask yourself if an SQL statement is necessary. The biggest performance issue is the amount of SQL issued. The main reason for this is typically a generic development or an object-oriented design. These types of designs run contrary to performance, but speed development and create flexibility and adaptability to business requirements. The price that you pay for this is performance. Consider the following: Is the SQL statement needed? Do you need to run the statement again? Does the statement have a DISTINCT, and are duplicates possible? Does the statement have an ORDER BY, and is the order important? Are you repeatedly accessing code tables, and can the codes be cached within the program? 42 CA Performance Handbook for DB2 for z/os

43 Coding Efficient SQL Coding SQL for Performance These basic guidelines for coding SQL statements provide the best performance: Retrieve the fewest number of rows. Only rows of data that are needed for the process should be returned to the application. Do not locate data filtering statements in a program. DB2 should handle all row filtering. When a process must be performed on the data, decide where to do the process in the program or in DB2 then leave it in DB2. DB2 is evolving into the "ultimate" server, and a growing number of applications are distributed across the network and use DB2 as the database. A situation may occur when all data must be brought to the application, filtered, transformed, and processed. This situation is due to an inadequacy somewhere else in the design, normally a shortcoming in the physical design, missing indexes, or some other implementation problem. Retrieve only columns that are needed by the program. Retrieving extra column results in the column being moved from the page in the buffer pool to a user work area, passed to stage 2, and returned cross-memory to the work area of the program. This is an unnecessary expenditure of CPU. Code your SQL statements to return only the columns required. This code design may mean coding multiple SQL statements against the same table for different sets of columns. This effort is required for improved performance and reduction of CPU. Reduce the number of SQL statements. Each SQL statement is a programmatic call to the DB2 subsystem that incurs fixed overhead for each call. Careful evaluation of program processes can reveal situations in which unnecessary SQL statements are issued. This is especially true for programmatic joins. Separate application programs that retrieve data from related tables can result in extra SQL statements issued. Code SQL joins instead of programmatic joins. When comparing the programmatic join of two tables in a COBOL program and the equivalent SQL join, the SQL join consumed 30 percent less CPU. Use stage 1 predicates. Familiarize yourself with stage 1 indexable, stage 1, and stage 2 predicates. Try to use stage 1 predicates for filtering whenever possible. Determine if you can convert your stage 2 predicates to stage 1, or stage 1 non-indexable to indexable. Never use generic SQL statements. Generic SQL equals generic performance. Generic SQL generally has logic that only retrieves specific data or data relationships and leaves the entire business rule processing in the application. Common modules, SQL that is used to retrieve current or historical data, and modules that read all data into a copy area only to have the caller use a few fields are all examples of generic SQL usage. Generic SQL and generic I/O layers do not belong in high-performing systems. Chapter 3: Predicates and SQL Tuning 43

44 Coding Efficient SQL Avoid unnecessary sorts. Avoiding unnecessary sorting is a requirement in any application but more so in any high-performance environment, especially when a database is involved. Generally, if ordering is always necessary, indexes should support the ordering. Sorting can be caused by GROUP BY, ORDER BY, DISTINCT, UNION, INTERSECT, EXCEPT, and join processing. If ordering is required for a join process, the ordering generally is a clear indication that an index is missing. If ordering is necessary after a join process, hopefully the result set is small. The worst-case scenario is when a sort is performed as the result of the SQL in a cursor, and the application processes only a subset of the data. In this case, the sort overhead must be removed. If the SQL has to sort 100 rows and only 10 are processed by the application it may be better to sort in the application, or you can create an index to help avoid the sort. Only sort necessary columns. When DB2 sorts data, the columns that are used to determine the sort order actually appear twice in the sort rows. Make sure that you specify the sort only on necessary columns. For example, any column specified in an equals predicate in the query does not need to be in the ORDER BY clause. Use the ON clause for all join predicates. By using explicit join syntax instead of implicit join syntax, you make a statement easier to read, easier to convert between inner and outer joins, and harder to forget to code a join predicate. Avoid UNIONs (not necessarily UNION ALL). To avoid duplication, replace the UNION with a UNION ALL. Otherwise, see if it is possible to produce the same result with an outer join. If all the subqueries of the UNION are against the same table, try using a CASE expression and only one pass through the data. Use joins instead of subqueries. Joins can outperform subqueries for existence checking if there are good matching indexes for both tables that are involved, and if the join does not introduce any duplicate rows in the result. DB2 can take advantage of predicate transitive closure and also pick the best table access sequence. These two outcomes are not possible with subqueries. 44 CA Performance Handbook for DB2 for z/os

45 Coding Efficient SQL Code the most selective predicates first. DB2 processes the predicates in a query in the following order: Indexable predicates are applied in the order of the columns of the index. Other stage 1 predicates are applied. Stage 2 predicates are applied. In each of the stages, DB2 processes the predicates in the following order: All equals predicates (including single IN list and BETWEEN with only one value). All range predicates and predicates of the form IS NOT NULL. All other predicate types. In each grouping, DB2 processes the predicates in the order they are coded in the SQL statement. Write all SQL queries to evaluate the most restrictive predicates first to filter unnecessary rows earlier. This action reduces processing cost at a later stage. This suggestion includes subqueries (in the grouping of correlated and non-correlated). Use the proper method for existence checking. For existence checking using a subquery, an EXISTS predicate generally outperforms an IN predicate. For general existence checking, code a singleton SELECT statement that contains a FETCH FIRST 1 ROW ONLY clause. Avoid unnecessary materialization. When you run a transaction that processes little or no data, use correlated references to avoid materializing large intermediate result sets. Correlated references encourage nested loop join and index access for transactions. Promoting Predicates Code SQL predicates as efficiently as possible. When you are writing predicates, stage 1 indexable predicates whenever possible. If possible, promote stage 1 non-indexable predicates or stage 2 predicates to a more efficient stage. If you have a stage 1 non-indexable predicate, promote it to stage 1 indexable. For example: WHERE END_DATE_COL <> ' ' If you use the end of the DB2 world as an indicator of data that is still active, change the predicate to something that is indexable. For example: WHERE END_DATE_COL < ' ' Chapter 3: Predicates and SQL Tuning 45

46 Coding Efficient SQL Promote stage 2 predicates to stage 1 or even stage 1 indexable. For example: WHERE BIRTHDATE + 30 YEARS < CURRENT DATE The previous predicate applies date arithmetic to a column. This is a column expression, which makes the predicate a stage 2 predicate. By moving the arithmetic to the right side of the inequality, you make the predicate stage 1 indexable. For example: WHERE BIRTHDATE < CURRENT DATE 30 YEARS These examples can be applied as general rules for promoting predicates. Using DB2 9, it is possible to create an index on an expression. An index on an expression can be considered for improved performance of column expressions when it is not possible to eliminate the column expression in the query. Functions, Expressions, and Performance DB2 has many scalar functions and other expressions that let you manipulate data. This functionality contributes to the fact that the SQL language is indeed a programming language as much as a data access language. However, the processing of functions and expressions in DB2, if not for the filtering of data, but instead for pure data manipulation, are generally more expensive than the equivalent application program process. Even the simple functions such as concatenation are executed for every row processed: SELECT FIRSTNME CONCAT LASTNAME FROM EMP If you need the ultimate in ease of coding, time to delivery, portability, and flexibility, code expressions and functions using this practice in your SQL statements. If you need the ultimate in performance, perform the manipulation of data in your application program. CASE expressions can be expensive, but CASE expressions use "early out" logic when processing. Examine the following example of a CASE expression: CASE WHEN C1 = 'A' OR C1 = 'K' OR C1 = 'T' OR C1 = 'Z' THEN 1 ELSE NULL END 46 CA Performance Handbook for DB2 for z/os

47 Coding Efficient SQL If most of the time the value of the C1 column is a blank, the following functionally equivalent CASE expression consumes less CPU: CASE WHEN C1 <> ' ' AND (C1 = 'A' OR C1 = 'K' OR C1 = 'T' OR C1 = 'Z') THEN 1 ELSE NULL END DB2 takes the early out from the CASE expression for the first NOT TRUE of an AND, or the first TRUE of an OR. In addition to testing for the previous blank value, all the other values should be tested with the most frequently occurring values first. Advanced SQL and Performance Advanced SQL queries can be a performance advantage or performance disadvantage. Advanced SQL possibilities are endless, and it is impossible to document all the various situations and the performance impacts. Complex SQL is always a performance improvement over an application program process when the complex SQL is filtering or aggregating data. Chapter 3: Predicates and SQL Tuning 47

48 Coding Efficient SQL Correlation In general, correlation encourages nested loop join and index access. This functionality is good for transaction queries that process small out quantities of data but not for report queries that process large quantities of data. The following query works well when processing large quantities of data: SELECT TAB1.EMPNO, TAB1.SALARY, FROM TAB2.AVGSAL,TAB2.HDCOUNT (SELECT EMPNO, SALARY, WORKDEPT FROM WHERE LEFT OUTER JOIN EMP JOB='SALESREP') AS TAB1 (SELECT AVG(SALARY) AS AVGSAL, COUNT(*) AS HDCOUNT, FROM GROUP WORKDEPT EMP BY WORKDEPT) AS TAB2 ON TAB1.WORKDEPT = TAB2.WORKDEPT The aforementioned query is the most efficient way to retrieve the data if there is one sales representative per department, or if most of the employees are sales representatives. The employee table is read and materialized in the nested table expression called TAB2. It is likely that the merge scan join method will be used to join the materialized TAB2 to the first table expression called TAB1. The aforementioned query is not the most efficient if the employee table is extremely large with few sales representatives, or with sales representatives in one or a few departments. The entire employee table still must be read in TAB2, but most of the results of the nested table expression are returned in the query. In this case, the following query is more efficient: SELECT TAB1.EMPNO, TAB1.SALARY, FROM,TABLE(SELECT WHERE FROM TAB2.AVGSAL,TAB2.HDCOUNT EMP TAB1 AVG(SALARY) AS AVGSAL, EMP COUNT(*) AS HDCOUNT WHERE WORKDEPT = TAB1.WORKDEPT) AS TAB2 TAB1.JOB = 'SALESREP' This statement is functionally equivalent to the previous statement, but it operates in a different way. In this query, the employee table referenced as TAB1 is read first. The nested table expression is executed repeatedly in a nested loop join for each row that qualifies. An index on the WORKDEPT column is necessary. 48 CA Performance Handbook for DB2 for z/os

49 Coding Efficient SQL Merge Versus Materialization for Views and Nested Table Expressions When you have a reference to a nested table expression or view in your SQL statement, DB2 may merge that nested table expression or view with the referencing statement. If DB2 cannot merge the statement, then DB2 materializes the view or nested table expression into a work file. DB2 then applies the referencing statement to that intermediate result. IBM states that merge is more efficient than materialization. In general, that statement is correct. However, materialization is more efficient if your complex queries have the following combined conditions: Nested table expressions or view references, especially multiple levels of nesting. Columns generated in the nested expressions or views through application of functions, user-defined functions, or other expressions. References to the generated columns in the outer referencing statement. In general, DB2 materializes when some sort of aggregate processing is required inside the view or nested table expression. This means that the view or nested table expression contains aggregate functions, grouping (GROUP BY), or DISTINCT. If materialization is not required, the merge process occurs. The following query is an example: SELECT MAX(CNT) FROM (SELECT ACCT_RTE, COUNT(*) AS CNT FROM YLA.ACCT_TABLE) AS TAB1 DB2 materializes TAB1 in the aforementioned example. Examine the following query: SELECT SUM(CASE WHEN COL1=1 THEN 1 END) AS ACCT_CURR FROM,SUM(CASE WHEN COL1>1 THEN 1 END) AS ACCT_LATE (SELECT CASE ACCT_RTE WHEN 'AA' THEN 1 WHEN 'BB' FROM THEN 2 WHEN 'CC' THEN '2' WHEN 'DD' THEN 2 WHEN 'EE' THEN 3 END AS COL1 YLA.ACCT_TABLE) AS TAB1 In this query, DISTINCT, GROUP BY, and aggregate functions do not exist. DB2 merges inner table expression with the outer referencing statement. There are two references to COL1 in the outer referencing statement. The CASE expression in the nested table expression is calculated twice during query execution. The merged statement resembles the following example: SELECT SUM(CASE WHEN CASE ACCT_RTE WHEN 'AA' THEN 1 WHEN 'BB' THEN 2 WHEN 'CC' THEN '2' WHEN 'DD' THEN 2 WHEN 'EE' THEN 3 END =1 THEN 1 END) AS ACCT_CURR,SUM(CASE WHEN CASE ACCT_RTE WHEN 'AA' THEN 1 WHEN 'BB' THEN 2 WHEN 'CC' THEN '2' WHEN 'DD' THEN 2 WHEN 'EE' THEN 3 END >1 THEN 1 END) AS ACCT_LATE Chapter 3: Predicates and SQL Tuning 49

50 Coding Efficient SQL FROM YLA.ACCT_TABLE For this particular query, the merge is probably more efficient than materialization. However, if you have multiple levels of nesting and many references to generated columns, merge can be less efficient than materialization. In these specific cases, introduce a non-deterministic function into the view or nested table expression to force materialization. Use the RAND() function. 50 CA Performance Handbook for DB2 for z/os

51 Coding Efficient SQL UNION in a View or Nested Table Expression You can place a UNION or UNION ALL into a view or nested table expression. This practice allows for complex SQL processing, but also lets you create logically partitioned tables. This means that you can store data in multiple tables and reference them together as one table in a view. This practice is useful for quickly rolling through yearly tables or for creating optional table scenarios with little maintenance overhead. Although each SQL statement in a union in view or table expression results in an individual query block, and SQL statements written against the view are distributed to each query block, DB2 uses a technique to prune (eliminate query blocks for efficiency. DB2 can, depending upon the query, prune query blocks at statement compile time or during statement execution. Consider the following account history view: CREATE VIEW V_ACCOUNT_HISTORY (ACCOUNT_ID, AMOUNT) AS SELECT ACCOUNT_ID, AMOUNT FROM HIST1 WHERE ACCOUNT_ID BETWEEN 1 AND UNION ALL SELECT ACCOUNT_ID, AMOUNT FROM HIST1 WHERE ACCOUNT_ID BETWEEN AND Then consider the following query: SELECT * FROM V_ACCOUNT_HISTORY WHERE ACCOUNT_ID = The predicate of this query contains the literal value , and this predicate is distributed to the query blocks generated. However, DB2 compares the distributed predicate against the predicates coded in the UNION inside your view, looking for redundancies. In any situations where the distributed predicate renders a particular query block unnecessary, DB2 prunes that query block from the access path. Examine the following example:... WHERE ACCOUNT_ID BETWEEN 1 AND AND ACCOUNT_ID = WHERE ACCOUNT_ID BETWEEN AND AND ACCOUNT_ID = Chapter 3: Predicates and SQL Tuning 51

52 Coding Efficient SQL DB2 prunes the query blocks generated at statement compile time, based upon the literal value supplied in the predicate. In the previous example, two query blocks are generated, but one of them is pruned when the statement is compiled. DB2 compares the literal predicate supplied in the query against the view that contains the predicates. Any unnecessary query blocks are pruned. Because one of the resulting combined predicates is impossible, DB2 eliminates that query block. Only one underlying table is accessed. Query block pruning can happen at statement compile (bind) time or at run time if a host variable or parameter marker is supplied for a redundant predicate. Replace the literal with a host variable in the previous example. The new query resembles the following example: SELECT * FROM V_ACCOUNT_HISTORY WHERE ACCOUNT_ID = :H1 If you embed this statement in a program, and it is bound into a plan or package, two query blocks are generated. This outcome occurs because DB2 does not know the value of the host variable in advance and distributes the predicate among both generated query blocks. However, at run time, DB2 examines the supplied host variable value, and dynamically prunes the query blocks appropriately. If the value is supplied for the host variable value, one of the two query blocks is pruned at run time, and only one underlying table is accessed. This complicated process does not always succeed. Test the process by stopping one of the tables and running a query with a host variable that prunes the query block on that table. If the statement is successful, the run-time query block pruning works properly. You can prune query blocks on literals and host variables. However, you cannot prune query blocks on joined columns. In certain situations with many query blocks (UNIONs), many rows of data, and many index levels for the inner view or table expression of a join, use programmatic joins in situations in which the query can benefit from run-time query block pruning using the joining column. This is an extremely specific recommendation. There are limits to UNION in view (or table) expressions. Test to see if you get bind time or run-time query block pruning. In some cases this outcome does not occur, and APARs are available that address the problems, but they are not comprehensive; therefore testing is important. You can influence proper use of run-time query block pruning by encouraging distribution of joins and predicates into the UNION in view. Accomplish this task by reducing the number of tables in the UNION, or by repeating host variables in predicates instead of, or in addition to using correlation. Examine the following query: SELECT {history data columns} FROM V_ACCOUNT_HISTORY HIST WHERE HIST.ACCT_ID = :acctid AND HIST.HIST_EFF_DTE = ' ' AND HIST.UPD_TSP = 52 CA Performance Handbook for DB2 for z/os

53 Coding Efficient SQL WITH UR; (SELECT MAX(UPD_TSP) FROM V_ACCOUNT_HISTORY HIST2 WHERE HIST2.ACCT_ID = :acctid AND HIST2.HIST_EFF_DTE = ' ') The predicate on ACCT_ID in the subquery could be correlated to the outer query, but it is not. The same applies to the predicate on the HIST_EFF_DTE predicate in the subquery. The reason for repeating the host variable and value references is to take advantage of the runtime pruning. Correlated predicates are not pruned in this case. Recommendations for Distributed Dynamic DB2 for z/os is the ultimate data server, and there is a proliferation of dynamic distributed SQL hitting the system, through JDBC, ODBC, or DB2 CLI. Ensuring this SQL is always performing the best can sometimes be a challenge. We offer the following general recommendations to improve distributed dynamic SQL performance: Turn on the dynamic statement cache. Use a fixed application level authorization ID. This practice lets accounting data be grouped by the ID, and applications can be identified by their ID. It also makes better use of the dynamic statement cache in that statements are reused by authorization ID. Parameter markers are almost always better than literal values. You need an exact statement match to reuse the dynamic statement cache. Literal values are helpful only when a skewed distribution of data exists, and that skewed distribution is properly reflected through frequency distribution or histogram statistics in the system catalog. Use the EXPLAIN STMTCACHE ALL command to expose all statements in the dynamic statement cache. The output from this command goes into a table that contains runtime performance information. In addition, you can explain individual statements in the dynamic statement cache after they have been identified. Use connection pooling. Consolidate calls into stored procedures. This practice works only if multiple statements and program logic, representing a transaction, are consolidated in a single stored procedure. Use the DB2 CLI/ODBC/JDBC static profiling feature. This CLI feature turns dynamic statements into static statements. Static profiling lets you capture dynamic SQL statements on the client and then bind the statements on the server into a package. After the statements are bound, the initialization file can be modified, instructing the interface software to use the static statement whenever the equivalent dynamic statement is issued from the program. Chapter 3: Predicates and SQL Tuning 53

54 Influencing the Optimizer If you are moving a local batch process to a remote server, you will lose some of the efficiencies that go along with the RELEASE(DEALLOCATE) bind parameter, in particular sequential detection and index lookaside. Influencing the Optimizer The DB2 optimizer is good at picking the correct access path, when it is given the proper information. In situations where it does not pick the optimal access path, it typically does not have enough information. This is sometimes the case when you use parameter markers or host variables in a statement. Proper Statistics DB2 uses a cost-based optimizer, and that optimizer needs accurate statistical information about your data. Collecting the proper statistics is necessary for good performance. With each new version of DB2, the optimizer better utilizes catalog statistics. This point means that with each new version, DB2 is more dependent on catalog statistics. You should have statistics on every column referenced in every WHERE clause. If you are using parameter markers and host variables, you need at least cardinality statistics. If you have skewed data or you are using literal values in your SQL statements, you need frequency distribution or histogram statistics. If you suspect columns are correlated, gather column correlation statistics. To determine if columns are correlated, run these two queries: SELECT COUNT (DISTINCT CITY) AS CITYCNT * FROM CUSTOMER COUNT (DISTINCT STATE) AS STATECNT SELECT COUNT (*) FROM (SELECT DISTINCT CITY, STATE FROM CUSTOMER) AS FINLCNT If the number from the second query is lower than the number from the first query, the columns are correlated. You can also run GROUP BY queries against tables for columns used in predicates to count the occurrences of values in these columns. These counts indicate whether you need frequency distribution statistics or histogram statistics, and run-time reoptimization for skewed data distributions. 54 CA Performance Handbook for DB2 for z/os

55 Influencing the Optimizer Run-time Reoptimization If your query contains a predicate with an embedded literal value, DB2 knows about the input to the query. DB2 can take advantage of frequency distribution or histogram statistics, if available. The result is a much improved filter factor and better access path decisions by the optimizer. However, DB2 may not know about your input value: SELECT * FROM EMP WHERE MIDINIT > :H1 In this case, if the values for MIDINT are highly skewed, DB2 could make an inaccurate estimate of the filter factor for some input values. DB2 can employ the run-time reoptimization to help your queries. For static SQL, the option of REOPT(ALWAYS) is available. This bind option instructs DB2 to recalculate access paths at runtime using the host variable parameters. This practice can improve execution time for large queries. However, if many queries are in the package, they are all reoptimized. This practice can affect the statement execution time for these queries. When you use REOPT(ALWAYS), consider separating the query that can benefit in its own package. Dynamic SQL statements have three options: REOPT(ALWAYS) Reoptimizes a dynamic statement with parameter markers based upon the values that are provided on every execution. REOPT(ONCE) Reoptimizes a dynamic statement the first time it is executed based on the values that are provided for parameter markers. The access path is reused until the statement is removed from the dynamic statement cache and needs to be prepared again. Use this reoptimization option with care because the first execution should have good representative values. REOPT(AUTO) Tracks how the values for the parameter markers change on every execution and reoptimizes the query that is based upon those values if it determines that the values have changed significantly. A system parameter called REOPTEXT (DB2 9) enables REOPT(AUTO)-like behavior, subsystem wide, for any dynamic SQL queries (without NONE, ALWAYS, or ONCE already specified) that contain parameter markers when changes are detected in the values that could influence the access path. Chapter 3: Predicates and SQL Tuning 55

56 Influencing the Optimizer OPTIMIZE FOR Clause The optimizer does not know how much data you are going to fetch from the query. DB2 uses the system catalog statistics to estimate the number of rows that are returned if the entire query is processed by the application. However, if you are not going to read all the rows of the result set, use the OPTIMIZE FOR n ROWS clause, where n is the number of rows you intend to fetch. The OPTIMIZE FOR clause lets you tell DB2 how many rows you intend to process. DB2 can then make access path decisions to determine the most efficient way to access the data for that quantity. The use of this clause discourages such actions as the list prefetch, sequential prefetch, and multi-index access. It encourages index usage to avoid a sort, and a join method of the nested loop join. A value of 1 is the strongest influence on these factors. Insert the number of rows you intend to fetch in the OPTIMIZE FOR clause. Incorrectly representing the number of rows you intend to fetch can result in a poorly performing query. Encouraging Index Access and Table Join You can use certain techniques to encourage DB2 to choose a certain index or a table access sequence in a query that accesses multiple tables. When DB2 joins tables in an inner join, it tries to select the table that qualifies the fewest rows first in the join sequence. If DB2 chooses the incorrect table first (maybe due to statistics or host variables), change the table access sequence by using one or more of these techniques: Enable predicates on the table that you want to be first. By increasing potential match cols on this table, DB2 can select an index for more efficient access and can change the table access sequence. Disable predicates on the table that you do not want accessed first. Predicate disablers are documented in the IBM DB2 Performance Monitoring and Tuning Guide. We do not recommend using predicate disablers. Force materialization of the table that you want accessed first by placing the table in a nested table expression with a DISTINCT or GROUP BY. This technique changes the join type and the join sequence. This technique is especially useful when a nested loop join is randomly accessing the inner table. Convert joins to subqueries. When you code subqueries, you tell DB2 the table access sequence. Non-correlated subqueries are accessed first, the outer query is executed, and then any correlated subqueries are executed. This technique is effective only if the table moved from a join to a subquery does not have to return data. 56 CA Performance Handbook for DB2 for z/os

57 Influencing the Optimizer Convert a joined table to a correlated nested table expression. This technique forces another table to be accessed first. Data for the correlated reference is required before the table in the correlated nested table expression being accessed. Convert an inner join to a left join. By coding a left join, you dictate the table join sequence to DB2, and that the right table filters no data. Add a CAST function to the join predicate for the table you want accessed first. By placing this function on that column, you encourage DB2 to access that table first to avoid a stage 2 predicate against the second table. Code an ORDER BY clause on the columns of the index of the table that you want to be accessed first in the join sequence. This technique may influence DB2 to use that index to avoid the sort, and access that table first. Change the order of the tables in the FROM clause. Try converting the implicit join syntax to explicit join syntax and the reverse. Code a predicate on the table you want accessed first as a non-transitive closure predicate. An example of this is a non-correlated subquery against the SYSDUMMY1 table that returns a single value rather than an equals predicate on a host variable or literal value. The subquery is not eligible for transitive closure, and DB2 does not generate the predicate redundantly against the other table, and has less encouragement to choose that table first. If DB2 chooses one index over another and you disagree, try one of these techniques to influence index selection: Code an ORDER BY clause on the leading columns of the index you want DB2 to choose. This technique may encourage DB2 to choose that index to avoid a sort. Add columns to the index to make the access index-only. Modify the query or the index to increase the index match cols. Disable predicates that match other indexes. We do not recommend predicate disablers. Use the OPTIMIZE FOR clause. Note: For details about predicate disablers, see the IBM documentation. Chapter 3: Predicates and SQL Tuning 57

58

59 Chapter 4: Designing Tables and Indexes for Performance The best way to perform logical database modeling is to use strong guidelines that are developed by an expert in relational data modeling. You could also use one of the many relational database modeling tools that is supplied by vendors, but it is important to remember that using a tool to migrate your logical model into a physical model does not mean that the physical model is the most optimal for performance. It is acceptable to modify the physical design to improve performance provided the logical model is not compromised. DB2 objects must be designed for availability, ease of maintenance, and overall performance, as well as for business requirements. Table Space Performance Recommendations This section includes general recommendations for table space performance. Use Segmented or Universal Table Spaces Segmented Table Spaces DB2 is eliminating simple tablespaces support. Although you can create them in DB2 V7 and DB2 V8, they cannot be created in DB2 9 (although you can still use previously created simple tablespaces). Whatever version you are using, we recommend that you use a segmented table space instead of a simple tablespace. You gain several advantages by using segmented table spaces. Because the pages in a segment contain only rows from one table, no locking interference with other tables exists. In simple table spaces, rows are intermixed on pages, and if one table page is locked, it can inadvertently lock a row of another table just because it is on the same page. This is not an issue if you have only one table per table space; however, you can obtain following benefits from having a segmented table space for one table: If a table scan is performed, the segments belonging to the table being scanned are the only ones accessed; empty pages are not scanned. If a mass delete or a DROP table occurs, segment pages are available for immediate reuse, and it is not necessary to run a REORG utility. Mass deletes are much faster for segmented table spaces, and they produce less logging, provided the table has not been defined with DATA CAPTURE CHANGES. Chapter 4: Designing Tables and Indexes for Performance 59

60 Table Space Performance Recommendations Universal Table Spaces The COPY utility does not have to copy empty pages that are left by a mass delete. When inserting records, some read operations can be avoided by using the more comprehensive space map of the segmented table space. DB2 9 introduces a new type of table space that is called a universal table space. A universal table space is segmented and partitioned. Two types of universal table spaces are available: The partition-by-growth table space The range-partitioned table space Partition-by-Data-Growth Table Space A universal table space offers the following benefits: Better space management relative to varying-length rows. A segmented space map page provides more information about free space than a regular partitioned space map page. Improved mass delete performance. A mass delete in a segmented table space organization tends to be faster than in table spaces that are organized differently. In addition, you can immediately reuse all or most of the segments of a table. Before DB2 9, partitioned tables required key ranges to determine the target partition for row placement. Partitioned tables provide more granular locking and parallel operations by spreading the data over more data sets. In DB2 9, you can partition according to data growth. This practice enables segmented tables to be partitioned as they grow, without the need for key ranges. As a result, segmented tables benefit from increased table space limits and SQL and utility parallelism that were formerly available only to partitioned tables. Also, with a partition-by-data-growth table space, you do not need to reorganize a table space to change the limit keys. You can implement partition-by-growth table space organization in one of the following ways: Use the new MAXPARTITIONS clause on the CREATE TABLESPACE statement to specify the maximum number of partitions that the partition-by-growth table space can accommodate. The value that you specify in the MAXPARTITIONS clause is used to protect against run away applications that perform an insert in an infinite loop. Use the MAXPARTITIONS clause on the ALTER TABLESPACE statement to alter the maximum number of partitions to which an existing partition-by-growth table space can grow. This ALTER TABLESPACE operation acts as an immediate ALTER. 60 CA Performance Handbook for DB2 for z/os

61 Table Space Performance Recommendations Range-Partitioned Table Space A range-partitioned table space is a type of universal table space that is based on partitioning ranges and that contains a single table. The new range-partitioned table space does not replace the existing partitioned table space, and operations that are supported on a regular partitioned or segmented table space are supported on a range-partitioned table space. To create a range-partitioned table space, specify the SEGSIZE and NUMPARTS keywords on the CREATE TABLESPACE statement. With a range-partitioned table space, you can also control the partition size, choose from a wide array of indexing options, and take advantage of partition-level operations and parallelism capabilities. Because the range-partitioned table space is also a segmented table space, you can run table scans at the segment level. As a result, you can immediately reuse all or most of the segments of a table after the table has been dropped or a mass delete has been performed. Range-partitioned universal table spaces follow the same partitioning rules as partitioned table spaces in general. That is, you can add, rebalance, and rotate partitions. The maximum number of partitions possible for both range-partitioned and partition-by-growth universal table spaces (as for partitioned table spaces) is controlled by the DSSIZE and page size. Clustering and Partitioning The clustering index is not always the primary key. It is generally a sequential range retrieval key, and should be chosen by the most frequent range access to the table data. Range and sequential retrieval are the primary requirements, but partitioning is an important requirement and can be the most critical requirement, especially as tables get extremely large. If you do not specify an explicit clustering index, DB2 clusters by the index that is the oldest by definition (often referred to as the first index created). If the oldest index is dropped and recreated, that index will now be a new index and clustering will now be by the next oldest index. The basic rule to clustering is that if your application will have a certain sequential access pattern or a regular batch process, you should cluster the data according to that input sequence. Clustering and partitioning can be independent, and a log of options is available for organizing your data as follows: In a single dimension (clustering and partitioning are based on the same key) Dual dimensions (clustering inside each partition by a different key) Multiple dimensions (combining different tables with different partitioning unioned inside a view). Chapter 4: Designing Tables and Indexes for Performance 61

62 Table Space Performance Recommendations You should choose a partitioning strategy that is based on a concept of application-controlled parallelism, separating old and new data, grouping data by time, or grouping data by some meaningful business entity (for example, sales region or office location). Within those partitions, you can cluster the data by your most common sequential access sequence. Note: For more information about dismissing clustering for inserts, see Append Processing for High Volume Inserts (see page 74). For large tables, partitioning is the only way to store large amounts of data, but partitioning also has advantages for smaller tables. Consider the following: DB2 lets you define up to 4096 partitions of up to 64 GB each; however, total table size is limited depending on the DSSIZE specified. Non-partitioned table spaces are limited to 64 GB of data. You can take advantage of the ability to execute utilities on separate partitions in parallel. This practice also lets you access data in certain partitions while utilities are executing on others. In a data-sharing environment, you can spread partitions among several members to split workloads. You can also spread your data over multiple volumes and need not use the same storage group for each data set belonging to the table space. This practice also lets you place frequently accessed partitions on faster devices. Free Space The FREEPAGE and PCTFREE clauses can help improve the performance of updates and inserts by allowing free space to exist on table spaces. Performance improvements include improved access to the data through the better clustering of data, faster inserts, fewer row overflows, and a reduction in the number of REORGs required. Some tradeoffs include an increase in the number of pages, fewer rows per I/O, less efficient use of buffer pools, and more pages to scan. Therefore, it is important to achieve a good balance for each individual table space and index space when deciding on free space. The balance depends on the processing requirements of each table space or index space. When inserts and updates are performed, DB2 uses the available free space, and by doing so, it can keep records in clustering sequence as much as possible. When the free space is used up, the records must be located elsewhere; when this happens, performance can begin to suffer. 62 CA Performance Handbook for DB2 for z/os

63 Table Space Performance Recommendations Read-only tables do not require any free space, and tables with a pure insert-at-end strategy (append processing) generally do not require free space. Exceptions to this guideline would be tables with VARCHAR columns and tables using compression that are subject to updates. When DB2 attempts to maintain clustering during inserting and updating, it searches nearby for free space or free pages for the row. If DB2 does not find this space, it searches the table space for a free place to put the row before extending a segment or a dataset. Indications of this activity are as follows: A gradual increase in insert CPU times in your application; you can see this when you examine the accounting records. Increasing getpage counts Relocated row counts When this activity happens, perform a REORG and reevaluate your free space allocations. Allocations The PRIQTY and SECQTY clauses of the CREATE TABLESPACE and ALTER TABLESPACE SQL statements specify the space that is to be allocated for the table space if the table space is managed by DB2. These settings influence the allocation by the operating system of the underlying VSAM datasets in which table space and index space data is stored. PRIQTY SECQTY Specifies the minimum primary space allocation for a DB2-managed data set of the table space or partition. The primary space allocation is in kilobytes, and the maximum that can be specified is 64 GB. DB2 requests a data set allocation corresponding to the primary space allocation, and the operating system attempts to allocate the initial extent for the data set in one contiguous piece. Specifies the minimum secondary space allocation for a DB2-managed data set of the table space or partition. DB2 requests secondary extents in a size according to the secondary allocation. However, the actual primary and secondary data set sizes depend on various settings and installation parameters. You can specify the primary and secondary space allocations for table spaces and indexes or can let DB2 choose them. Having DB2 choose the values, especially for the secondary space quantity, increases the possibility of reaching the maximum data set size before running out of extents. In addition, the MGEXTSZ subsystem parameter influences the SECQTY allocations, and when set to YES (NO is the default), it changes the space calculation formulas to help use all of the potential space allowed in the table space before running out of extents. Chapter 4: Designing Tables and Indexes for Performance 63

64 Table Space Compression You can alter the primary and secondary space allocations for a table space. The secondary space allocation takes effect immediately. However, because the primary allocation happens when the data set is created, that allocation does not take effect until a data set is added, depending on the type of table space, or until the data set is recreated through utility execution such as a REORG or LOAD REPLACE. Column Ordering When defining a table, you can order your columns in specific ways to achieve the following results: Reduce CPU consumption when reading and writing columns with variable-length data Reduce the amount of logging performed when updating rows The version of DB2 you are running influences the way you or DB2 organizes your columns. Table Space Compression Using the COMPRESS clause of the CREATE TABLESPACE and ALTER TABLESPACE SQL statements allows for the compression of data in a table space or in a partition of a partitioned table space. In many cases, using the COMPRESS clause can significantly reduce the amount of DASD space that is required to store data. But the compression ratio that is achieved depends on the characteristics of the data. Compression lets you place more rows on a page resulting in the following performance benefits, depending on the SQL workload and the amount of compression: Higher buffer pool hit ratios Fewer I/Os Fewer getpage operations Reduced CPU time for image copies The processing cost when using compression is relatively low, but consider the following: The processor cost to decode a row using the COMPRESS clause is less than the cost to encode the same row. The data access path that DB2 uses affects the processor cost for data compression. In general, the relative overhead of compression is higher for table space scans and less costly for index access. 64 CA Performance Handbook for DB2 for z/os

65 Referential Constraints Some data does not compress well, so you should query the PAGESAVE column in SYSIBM.SYSTABLEPART to be sure that you are getting a savings (at least 50 percent is average). Data that does not compress well includes binary data, encrypted data, and repeating strings. Additionally, do not compress small tables or rows if you are concerned about concurrency issues because this configuration places more rows on a page. Note: When you compress, the row is treated as varying length and the length may change when updates occur, resulting in potential row relocation causing high numbers in NEARINDREF and FARINDREF. This outcome indicates that you are now doing more I/O to get to your data because it has been relocated, and you will have to REORG to get it back to its original position. Referential Constraints Referential integrity lets you define required relationships between and within tables. The database manager maintains these relationships, which are expressed as referential constraints. These relationships require that all values of a given attribute or table column also exist in some other table column. In general, DB2-enforced referential integrity is much more efficient than coding the equivalent logic in your application program. In addition, having the relationships enforced in a central location in the database is much more powerful than making it dependent upon application logic. You will need indexes to support the relationships that are enforced by DB2. Referential integrity checking has a cost that is associated with it. Referential integrity is meant for checking parent/child relationships, not code checking. Better options for code checking include check constraints, or even better, to put codes in memory and check them there. Table check constraints enforce data integrity at the table level. After a table-check constraint has been defined for a table, every UPDATE and INSERT statement involves checking the restriction or constraint. If the constraint is violated, the data record is not inserted or updated, and an SQL error is returned. A table-check constraint can be defined when you create a table or by using the ALTER TABLE statement. The table-check constraints can help implement specific rules for the data values contained in the table by specifying the values allowed in one or more columns in every row of a table. This practice can save time for the application developer because the validation of each data value can be performed by the database and not by each of the applications accessing the database. However, check constraints should, in general, not be used for data edits in support of data entry. For this scenario, it is best to cache code values locally within the application and perform the edits local to the application. This practice avoids numerous trips to the database to enforce the constraints. Chapter 4: Designing Tables and Indexes for Performance 65

66 Indexing Indexing Depending upon your application and the type of access, indexing can enhance or impair performance. If your application is a heavy reader or even a read-only application, numerous indexes can enhance performance. If your application is constantly inserting, updating, and deleting from your table, numerous indexes might prove detrimental. If you are adding a secondary index to a table for inserts and deletes, and even updates, you are adding another random read to these statements. Consider whether your application can afford that in support of queries that may use the index. Efficient Database Design When designing your database, for tables that will contain a significant amount of data and considerable DML activity, aim to have only one index per table. This practice provides for an efficient database design. Each index would need to support the following: Insert strategy Primary key Foreign key (if a child table) SQL access path Clustering Note the following design objectives: Avoid surrogate keys. Use meaningful business keys instead. Let the child table inherit the parent key as part of the child's primary key. Cluster all tables in a common sequence. Determine the common access paths, respect them, and try not to change them during design. Never entertain a just in case type of design mentality. Index Design Recommendations After you have determined the indexes that you require, design them properly for performance. 66 CA Performance Handbook for DB2 for z/os

67 Indexing Index Compression In DB2 9, an index can be defined with the COMPRESS YES option; COMPRESS NO is the default. Index compression can be used when you have to reduce the amount of disk space that an index consumes. We recommend index compression for applications that do sequential insert operations with few, or no, delete operations. Random inserts and deletes can adversely affect compression. We also recommend index compression for applications where the indexes are created primarily for scan operations. A buffer pool that is used to create the index must be 8 KB, 16 KB, or 32 KB. The physical page size for the index on the disk will be 4 KB. The buffer pool size is larger than the page size because index compression saves space only on the disk. The data in the index page is expanded when read into the pool. So, index compression can possibly save you read time for sequential, and random operations (but far less likely). Index compression can have a significant impact on the REORGs and index rebuilds, resulting in significant savings in this area; however, if you use the copy utility to back up an index, that image copy is uncompressed. Index Free Space Setting the PCTFREE and FREEPAGE for your indexes depends upon how much insert and delete activity is going to occur against those indexes. For indexes that have little or no inserts and deletes, you could use a small PCTFREE with no free pages. Note: Updates that change key columns are actually inserts and deletes. For indexes with heavy changes, consider larger amounts of free space. Adding free space may increase the number of index levels, and then increase the amount of I/O for random reads. If you do not have enough free space, you might experience an increased frequency of index page splits. When DB2 splits a page, it looks for a free page in which to place one of the split pages. If it does not find a page nearby, it searches the index for a free page. For large indexes, this search could lead to CPU and locking problems. We recommend that you set a predictive PCTFREE that anticipates growth over a period such that you do not split pages. You should then monitor the frequency of page splits to determine when to REORG the index or establish a regular REORG policy for that index. Chapter 4: Designing Tables and Indexes for Performance 67

68 Indexing Secondary Indexes Two types of secondary indexes exist; non-partitioning secondary indexes and data-partitioned secondary indexes as follow: Non-Partitioning Secondary Indexes (NPSIs) Are used on partitioned tables. They are not the same as the clustered partitioning key, which is used to order and partition the data, but rather they are for access to the data. NPSIs can be unique or non unique. You can have only one clustered partitioning index, but you can have several NPSIs on a table, if necessary. NPSIs can be broken into multiple pieces (data sets) by using the PIECESIZE clause on the CREATE INDEX statement. Pieces can vary in size from 254 KB to 64 GB the optimum size depends on the amount of data you have and how many pieces you want to manage. If you have several pieces, you can achieve more parallelism on processes, such as heavy INSERT batch jobs, by alleviating the bottlenecks that are caused by contention on a single data set. In DB2 V8 and later, the NPSI can be the clustering index. Note: NPSIs are good for fast read access because there is a single index b-tree structure. NPSIs can become extremely large and can cause maintenance and availability issues. Data-Partitioned Secondary Indexes (DPSIs) Provides many advantages over the traditional NPSIs for secondary indexes on a partitioned table space in terms of availability and performance. The partitioning scheme of the DPSI is the same as the table space partitions, and the index keys in 'x' index partition matches those in 'x' partition of the table space. Some of the benefits that this provides include the following: Clustering by a secondary index Ability to easily rotate partitions Efficient utility processing on secondary indexes (no BUILD-2 phase) Allow for reducing overhead in data sharing (affinity routing) Drawbacks of DPSIs While DPSIs provide gains in furthering partition independence, some queries may not perform as well. If the query has predicates that reference columns in a single partition and are therefore restricted to a single partition of the DPSI, the query benefits from this new organization. To accomplish this task, design the queries to allow for partition pruning through the predicates. This practice means that the leading column of the partitioning key has to be supplied in the query for DB2 to prune partitions from the query access path. However, if the predicate references only columns in the DPSI, it may not perform well because it may need to probe several partitions of the index. Other limitations of DPSIs include that they cannot be unique (some exceptions in DB2 9), and they may not be the best candidates for ORDER BYs. 68 CA Performance Handbook for DB2 for z/os

69 Special Tables Used for Performance Rebuild or Recover? In DB2 V8 and later, you can define an index as COPY YES. This definition means, as with a table space, you can use the COPY and RECOVER utilities to back up and recover these indexes. This practice may be especially useful for large indexes; however, large NPSIs cannot be copied in pieces. You will need large data sets to hold the backup. This requirement could mean large quantities of tapes or reaching the 59 volume limit for a dataset on DASD. REBUILD requires large quantities of temporary DASD to support sorts, as well as more CPU than a RECOVER. Carefully consider whether your strategy for an index should be backup and recover or should be rebuild. Special Tables Used for Performance Some table designs can dramatically help boost application performance. Materialized Query Tables Decision support queries are often difficult and expensive. They typically operate over a large amount of data and may have to scan or process terabytes of data and possibly perform multiple joins and complex aggregations. With these types of queries, traditional optimization and performance is not always optimal. In DB2 V8 and later, one solution is the use of Materialized Query Tables (MQTs). This practice lets you precompute whole or parts of each query and then use computed results to answer future queries. MQTs provide the means to save the results of previous queries and then reuse the common query results in subsequent queries. This practice helps avoid redundant scanning, aggregating, and joins. MQTs are also useful for data warehouse type applications. MQTs do not completely eliminate optimization problems, but rather move optimization issues to other areas. Some challenges include finding the best MQT for an expected workload, maintaining the MQTs when underlying tables are updated, ability to recognize usefulness of MQT for a query, and the ability to determine when DB2 will actually use the MQT for a query. Most of these types of problems are addressed by OLAP tools, but MQTs are the first step. The main advantage of the MQT is that DB2 can recognize a summary query against the source tables for the MQT, and can rewrite the query to use the MQT instead. It is, however, your responsibility to move data into the MQT by using a REFRESH TABLE command, or by manually moving the data yourself. Chapter 4: Designing Tables and Indexes for Performance 69

70 Special Tables Used for Performance Volatile Tables In DB2 V8 and later, volatile tables provide a way to prefer index access over table space scans or non-matching index scans for tables that have statistics that make them appear to be small. They are good for tables that shrink and grow, allowing matching index scans on tables that have grown larger without new RUNSTATS. They also improve cluster table support. Cluster tables are tables that have groups or clusters of data that logically belong together. Within each group, rows should be accessed in the same sequence to avoid lock contention during concurrent access. The sequence of access is determined by the primary key, and if DB2 changes the access path lock, contention can occur. To best support cluster tables (volatile tables), DB2 uses index-only access when possible. This practice minimizes application contention on cluster tables by preserving the access sequence by primary key. You should ensure that indexes are available for single table access and joins. You can specify the keyword VOLATILE on the CREATE TABLE or the ALTER TABLE statements. If specified, you are forcing an access path of index accessing and no list prefetch. Clone Tables In DB2 9, you can create a clone table on an existing base table at the current server by using the ALTER TABLE statement. Although ALTER TABLE syntax is used to create a clone table, the authorization that is granted as part of the clone creation process is the same as you would get during regular CREATE TABLE processing. The schema (creator) for the clone table is the same as the base table. You can create a clone table only if the base table is in a universal table space. To create a clone table, issue an ALTER TABLE statement with the ADD CLONE option. For example: ALTER TABLE base-table-name ADD CLONE clone-table-name The creation or drop of a clone table does not affect the applications that are accessing base table data. No base object quiesce is necessary and this process does not invalidate plans, packages, or the dynamic statement cache. You can exchange the base and clone data by using the EXCHANGE statement. To exchange table and index data between the base table and clone table, issue an EXCHANGE statement with the DATA BETWEEN TABLE table-name1 AND table-name2 syntax. Note: This is a method of performing an online load replace. 70 CA Performance Handbook for DB2 for z/os

71 Table Designs for Special Situations After a data exchange, the base and clone table names remain the same as they were before the data exchange. No data movement actually occurs. The instance numbers in the underlying VSAM data sets for the objects (tables and indexes) change, which changes the data that appears in the base and clone tables and their indexes. Example A base table that exists with the data set name *I0001.*. The table is cloned and the clone's data set is initially named *.I0002.*. After an exchange, the base objects are named *.I0002.* and the clones are named *I0001.*. Each time that an exchange happens, the instance numbers that represent the base and the clone objects change, which immediately changes the data contained in the base and clone tables and indexes. When the clone is dropped and an uneven number of EXCHANGE statements have been executed, the base table will have an *I0002.* data set name. This could be confusing. Table Designs for Special Situations When you are storing large amounts of data or maximizing transaction performance, you can do many creative actions. UNION in View for Large Table Design The amount of data being stored in DB2 is increasing dramatically. Availability of databases is also an increasingly important issue. As you build these giant tables in your system, you must help ensure that they are built in a way that they are available 24-hours a day, 7-days per week. Large tables must be easy to access, hold significant quantities of data, and easy to manage. Traditionally, larger database tables have been placed into partitioned table spaces. Partitioning helps with database management because it is easier to manage several small objects versus one very large object. Some limits to partitioning exist. For example, each partition is limited to a maximum size of 64 GB, a partitioning index is required (DB2 V7 only), and if efficient alternate access paths to the data are desired, then non-partitioning secondary indexes (NPSIs) are required. These NPSIs are not partitioned, and exist as single large database indexes. Thus, NPSIs can present themselves as an obstacle to availability (that is, a utility operation against a single partition may potentially make the entire NPSI unavailable), and as an impairment to database management because it is more difficult to manage large database objects. Chapter 4: Designing Tables and Indexes for Performance 71

72 Table Designs for Special Situations You can use a UNION in a view as an alternative to table partitioning in support of very large database tables. In this type of design, several database tables can be created to hold different subsets of the data that would have otherwise been held in a single table. Key values, similar to what may be used in partitioning, can be used to determine which data goes into which of the various tables. For example, the following view definition: CREATE VIEW V_ACCOUNT_HISTORY AS (ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT, PAYMENT_TYPE, INVOICE_NUMBER) SELECT ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT, FROM PAYMENT_TYPE, INVOICE_NUMBER ACCOUNT_HISTORY1 WHERE ACCOUNT_ID BETWEEN 1 AND UNION ALL SELECT ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT, FROM PAYMENT_TYPE, INVOICE_NUMBER ACCOUNT_HISTORY2 WHERE ACCOUNT_ID BETWEEN AND UNION ALL SELECT ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT, FROM PAYMENT_TYPE, INVOICE_NUMBER ACCOUNT_HISTORY3 WHERE ACCOUNT_ID BETWEEN AND UNION ALL SELECT ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT, FROM Advantages of UNION in a View PAYMENT_TYPE, INVOICE_NUMBER ACCOUNT_HISTORY4 WHERE ACCOUNT_ID BETWEEN AND ; By separating the data into different tables and creating the view over the tables, you can create a logical account history table with these distinct advantages over a single physical table: You can add or remove tables with small outages, usually just the time it takes to drop and recreate the view. You can partition each of the underlying tables, creating still smaller physical database objects. NPSIs on each of the underlying tables can be much smaller and easier to manage than they would be under a single table design. Utility operations can execute against an individual underlying table, or just a partition of that underlying table. This practice greatly decreases utility times against these individual pieces, and improves concurrency, as well as providing full partition independence. 72 CA Performance Handbook for DB2 for z/os

73 Table Designs for Special Situations Limitations of UNION in a View The view can be referenced in any SELECT statement in the same way as a physical table would be. Each underlying table could be as large as 16 TB, logically setting the size limit of the table that is represented by the view at 64 TB. Each underlying table could be clustered differently, or could be a segmented or partitioned tablespace. DB2 distributes predicates against the view to every query block within the view and then compares the predicates. Any impossible predicates will result in the query block being pruned (not executed). Consider the following limitations to UNION in a view: The view is read-only, which means you would have to use special program logic and possibly even dynamic SQL to perform inserts, updates, and deletes against the base tables. If you are using DB2 9, however, you could use INSTEAD OF triggers to provide this functionality. Predicate transitive closure happens after the join distribution. So, if you are joining from a table to a UNION in a view and predicate transitive closure is possible from the table to the view, you have to code the redundant predicate. DB2 can apply query block pruning for literal values and host variables, but not joined columns. For that reason, if you expect query block pruning on a joined column, code a programmatic join (this is the only case). Also, in some cases, the pruning does not always work for host variables, so you have to test. When using UNION in a view, keep the number of tables in a query to a reasonable number. This recommendation is especially true for joining because DB2 distributes the join to each query block. This practice multiplies the number of tables in the query, which can increase bind time and execution time. Also, you could exceed the 225 table limit in which case DB2 will materialize the view. In general, we recommend that you limit the number of tables UNIONed in the view to 15 or under. Chapter 4: Designing Tables and Indexes for Performance 73

74 Table Designs for Special Situations Append Processing for High Volume Inserts In situations in which you have very high insert rates, you may want to use append processing. When the append processing is turned on, DB2 uses an alternate insert algorithm that simplifies add data to the end of a partition or a table space. Append processing can be turned on for DB2 V7 or DB2 V8 by setting the table space options PCTFREE 0 FREEPAGE 0 MEMBER cluster. Make sure that APARS PQ86037 and PQ87381 are applied. Note: In DB2 9, turn on append processing by using the APPEND YES option of the CREATE TABLE statement. When the append processing is turned on, DB2 does not respect the cluster during inserts and simply puts all new data at the end of the table space. If you have a single index for read access, append processing may mean more random reads. This activity may require more frequent REORGs to keep the data organized for read access. Also, if you are partitioning, and the partitioning key is not the read index, then you still have random reads during the insert to the non-partitioned index. Ensure that you have adequate free space to avoid index page splits. You can also use append processing to store historical or seldom read audit information. In these cases, perform your partitioning based upon an ascending value (for example, a date) and have all new data go to the end of the last partition. In this situation, all table space maintenance, such as copies and REORGs, are against the last partition. All other data is static and does not require maintenance. You may need to create a secondary index, or each read query will have to be for a range of values within the ascending key domain. Building an Index on the Fly In situations in which you are storing data through a key that is designed for a high speed insert strategy with minimal table maintenance, avoid secondary indexes. Scanning billions of rows is not an efficient use of resources. A solution may be to build a lookup table that acts as a sparse index. This lookup table will contain nothing more than your ascending key values. One example would be dates (one date per month for every month possible in your database). If the historical data is organized and partitioned by the date, and you have only one date per month (to sub categorize the data), you can use the new sparse index to access the data you need. Using user-supplied dates as starting and ending points, the look-up table can be used to fill the gap with the dates in between. This practice provides the initial path to the history data. Read access is performed by constructing a key during SELECT processing. In the following examples, we access an account history table (ACCT_HIST) that has a key on HIST_EFF_DTE, ACCT_ID, and the date lookup table that is called ACCT_HIST_DATES, which contains one column and one row for each legitimate date value corresponding to the HIST_EFF_DTE column of the ACCT_HIST table. 74 CA Performance Handbook for DB2 for z/os

75 Table Designs for Special Situations Current Data Access Current data access is simple. You can retrieve the account history data directly from the account history table. SELECT {columns} FROM ACCT_HIST WHERE ACCT_ID = :ACCT-ID AND HIST_EFF_DTE = :HIST_EFF-DTE; Range of Data Access Accessing a range of data is a little more complicated than simply retrieving the most recent account history data. Use the sparse index history date table to build the key on demand. Apply the range of dates to the date range table, and then join the date range table to the history table. SELECT {columns} FROM ACCT_HIST HIST INNER JOIN ACCT_HIST_DATES DTE ON HIST.HIST_EFF_DTE = DTE.EFF_DTE WHERE HIST.ACCT_ID = :ACCT-ID AND HIST.HIST_EFF_DTE BETWEEN :BEGIN-DTE AND :END-DTE; Full Data Access To access all of the data for an account, you need a version of the previous query without the date range predicate. SELECT {columns} FROM ACCT_HIST HIST INNER JOIN ACCT_HIST_DATES DTE ON HIST.HIST_EFF_DTE = DTE.EFF_DTE WHERE HIST.ACCT_ID = :ACCT-ID; Denormalization "Light" In many situations, especially those in which there is a conversion from a legacy flat file-based system to a relational database, there is a performance concern (or more importantly a performance problem in that an SLA is not being met) for reading the multiple DB2 tables. These are situations in which the application is expecting to read all of the data that was once represented by a single record, but it is now in many DB2 tables. In these situations, a typical response is to begin denormalizing the tables. Chapter 4: Designing Tables and Indexes for Performance 75

76 Table Designs for Special Situations Denormalizing tables counteracts all the advantages of moving your data into DB2; that is, efficiency, portability, flexibility, and faster time to delivery for your new applications. In some situations, however, the performance issues resulting from reading multiple tables that are compared to the equivalent single record read are unacceptable. In these instances, you can consider implementing denormalization "light" instead. This type of denormalization can be applied to parent and child tables, when the child table data is in an optional relationship to the parent. Instead of denormalizing the optional child table data into the parent table, add a column to the parent table. This configuration indicates whether the child table has any data for that parent key. Note: If you apply this method, ensure that you factor in maintenance of that indicator column. However, DB2 can use a during join predicate to avoid probing the child table when there is no data for the parent key. For example, assume that you have an account table and an account history table. The account may or may not have account history, and so the following query would join the two tables together to list the basic account information (balance) along with the history information if present: SELECT A.CURR_BAL, B.DTE, B.AMOUNT FROM ACCOUNT A LEFT OUTER JOIN ACCT_HIST B ON A.ACCT_ID = B.ACCT_ID ORDER BY B.DTE DESC In this example, the query will always probe the account history table in support of the join, whether or not the account history table has data. You can employ denormalization "light" by adding an indicator column to the account table. You can then use a during join predicate. DB2 will perform the join operation only when the join condition is true. In this case, the access to the account history table is avoided when the indicator column has a value not equal to Y: SELECT A.CURR_BAL, B.DTE, B.AMOUNT FROM ACCOUNT A LEFT OUTER JOIN ACCT_HIST B ON A.ACCT_ID = B.ACCT_ID AND A.HIST_IND = 'Y' ORDER BY B.DTE DESC DB2 is going to test that indicator column first before performing the join operation, and supply nulls for the account history table when data is not present as indicated. 76 CA Performance Handbook for DB2 for z/os

77 Table Designs for Special Situations This type of design provides significant benefits when you are doing a legacy migration from a single record system to around 40 relational tables with lots of optional relationships. This form of denormalizing can improve performance in support of legacy system access, while maintaining the relation design for efficient future applications. Chapter 4: Designing Tables and Indexes for Performance 77

78

79 Chapter 5: EXPLAIN Facility and Predictive Analysis EXPLAIN Facility The DB2 EXPLAIN facility is used to expose query access path information. This result enables application developers and DBAs to see what access path DB2 is going to take for a query and decide if the query tuning is needed. DB2 can gather basic access path information in a special table called the PLAN_TABLE (DB2 V7, DB2 V8, DB2 9), as well as detailed information about predicate stages, filter factor, predicate matching, and dynamic statements that are cached (DB2 V8, DB2 9). You can invoke EXPLAIN by doing one of the following: Executing the EXPLAIN SQL statement for a single statement Specifying the BIND option EXPLAIN(YES) for a plan or package bind Executing the EXPLAIN through the Optimization Service Center or through Visual EXPLAIN EXPLAIN can populate many tables when it is executed. The target set of EXPLAIN tables depends on the authorization ID associated with the process. The creator (schema) of the EXPLAIN tables is determined by the CURRENT SQLID of the person running the EXPLAIN statement, or the owner of the plan or package at bind time. The EXPLAIN tables are optional, and DB2 only populates the tables that it finds under the SQLID or owner of the process invoking the EXPLAIN. Chapter 5: EXPLAIN Facility and Predictive Analysis 79

80 EXPLAIN Facility The following tables can be defined manually, and the DDL can be found in the DB2 sample library member DSNTESC: PLAN_TABLE Contains basic access path information for each query block of your statement. This information includes, details about index usage and the number of matching index columns, which join method is used, which access method is used, and whether a sort will be performed. The PLAN_TABLE forms the basis for access path determination. DSN_STATEMNT_TABLE Contains estimated cost information for the cost of a statement. If the statement table is present when you run EXPLAIN on a query, the table is populated with the cost information that corresponds to the access path information for the query stored in the PLAN_TABLE. For a given statement, this table contains the estimated processor cost in milliseconds and service units and it places the cost values into the following categories: Category A - DB2 has enough information to make a cost estimation without using any defaults. Category B - DB2 has to use default values to make some cost calculations. The statement table can be used to compare estimated costs when you are attempting to modify statements for performance. This is a cost estimate, and is not truly reflective of how your statement will be used in an application (given input values, transaction patterns, and so on). You should always test your statements for performance in addition to using the statement table and EXPLAIN. DSN_FUNCTION_TABLE Contains information about user-defined functions that are a part of the SQL statement. Information from this table can be compared to the cost information (if populated) in the DB2 System Catalog table, SYSIBM.SYSROUTINES, for the user-defined functions. DSN_STATEMENT_CACHE_TABLE Is not populated by a normal invocation of EXPLAIN, but instead by the EXPLAIN STMTCACHE ALL statement. Issuing this statement results in DB2 reading the contents of the dynamic statement cache, and putting runtime execution information into the table. This includes information about the frequency of execution of these statements, the statement text, the number of rows that are processed by the statement, lock and latch requests, I/O operations, number of index scans, number of sorts, and so on. This is valuable information about the dynamic queries executing in a subsystem. This table is available only for DB2 V8 and DB2 9. Note: Many EXPLAIN tables exist. Additional EXPLAIN tables are typically populated by the optimization tools that use them. You can make some tables without using the various optimization tools. For information about these tables, see the DB2 Performance Monitoring and Tuning Guide (DB2 9). 80 CA Performance Handbook for DB2 for z/os

81 EXPLAIN Facility What EXPLAIN Tells You The PLAN_TABLE is the key table for determining the access path for a query. The PLAN_TABLE provides the following critical information: METHOD Indicates the joint method, or whether an additional sort step is required. ACCESSTYPE Indicates whether the access to is through a table space scan or through index access. If it is by index access, the specific type of index access is indicated. MATCHCOLS Indicates the number of columns that are matched against the index if an index access is used. INDEXONLY Indicates that the access required can be served by accessing the index only, and avoiding any table access when a value of Y is in this column. SORTN####, SORTC#### Indicates any sorts that may happen in support of a UNION, grouping, joint operation, and so on. PREFETCH Indicates whether the prefetch can play a role in the access. By manually running EXPLAIN and examining the PLAN-TABLE, you can retrieve the following information: Access path Indexes that are used Join operations Any sorting that occurs as part of your query If you have additional EXPLAIN tables (for example, those created by Visual Explain or the Optimization Service Center), those tables are populated automatically. This is done by using those tools or by manually running EXPLAIN. If you do not have the remote access that is required from a PC to use the tools, you can also query those tables manually. The tables provide detailed information about such entities as predicates stages, filter factors, eliminated partitions, parallel operations, detailed cost information, and so on. Note: For more information about tables, see the DB2 Performance Monitoring and Tuning Guide (DB2 9). Chapter 5: EXPLAIN Facility and Predictive Analysis 81

82 EXPLAIN Facility What EXPLAIN Does Not Tell You EXPLAIN does not tell you everything about your queries. You have to be aware of this limitation to effectively performance tune and predictively analyze. EXPLAIN does not tell you about the following: INSERT indexes EXPLAIN does not tell you the index that DB2 will use for an INSERT statement. It is important that you understand your clustering indexes, and whether DB2 will be using APPEND processing for your inserts. This understanding is important for INSERT performance, and the proper organization of your data. For more information, see Designing Tables and Indexes for Performance (see page 59). Access path information for enforced referential constraints If you have INSERTS, UPDATES, and DELETES in the program to which you have applied EXPLAIN, the database enforced RI relationships and associated access paths are not exposed in the EXPLAIN tables. Ensure, therefore, that proper indexes in support of the RI constraints are established and in use. Predicate evaluation sequence The EXPLAIN tables do not show you the order in which predicates of the query are evaluated. For more information on predicate evaluation sequence, see Predicates and SQL Tuning. Statistics used The optimizer used catalog statistics to help determine the access path at the time EXPLAIN was applied to the statement. Unless you have historical statistics that correspond to the time that EXPLAIN was executed, you do not know that how the statistics appeared when the EXPLAIN was executed. Additionally, you do not know if they are different now and therefore you provide a different access path. Input values If you are using host variables in your programs, EXPLAIN does not know about the potential input values to those variables. Therefore, it is important for you to understand these values, what the most common values are, and if data is skewed relative to the input values. SQL statement The SQL statement is not captured in the EXPLAIN tables, although some of the predicates are. If you dynamically applied EXPLAIN to a statement, or you applied it through one of the tools, you know what the statement looks like. However, if you applied EXPLAIN to a package or plan, you have to see the program source code. 82 CA Performance Handbook for DB2 for z/os

83 Predictive Analysis Tools Order of input to your transactions The SQL statement may appear to be correct when you view the access path, but you cannot tell the order of the input data in relation to the statement, what ranges are supplied, or how many transactions are issued. EXPLAIN output does not let you see whether it is possible to order the input or the data in the tables more efficiently. Program source code To understand the impact of the access path that a statement used, you have to see how that statement is used in the application program. We recommend that you regularly review the program source code of the program in which the statement is embedded in. By reviewing the program source code, you can answer the following questions: How many times will the statement be executed? Is the statement in a loop? Can you avoid executing the statement? Is the program issuing 100 statements on individual key values in a range predicate when one statement would suffice? Is the programming performing programmatic joins? Although the EXPLAIN output may show an efficient access path, the statement might be unnecessary. You can also use a monitor or performance trace, which lets you see exactly which statements are executed. Predictive Analysis Tools You can use several predictive analysis tools to help you plan your implementation, including the Optimization Service Center (OCS), Visual EXPLAIN, and DB2 Estimator. Optimization Service Center and Visual EXPLAIN As the complexity of managing and tuning various workloads continues to escalate, many database administrators (DBAs) are struggling to maintain quality of service (QoS) while also managing costs. IBM Optimization Service Center for DB2 for z/os (OSC) is a Windows workstation tool that is designed to ease the workload of DBAs through a set of automation tools that help optimize query performance and workloads. You can use OSC to identify and analyze problem SQL statements and to receive expert advice about statistics that are gathered to improve the performance of problematic and poorly performing SQL statements on a DB2 subsystem. It provides the following: The ability to snap the statement cache Collect statistics information Chapter 5: EXPLAIN Facility and Predictive Analysis 83

84 Predicting Database Performance Analyze indexes Group statements into workloads Monitor workloads An easy-to-understand display of a selected access path Suggestions for changing a SQL statement An ability to invoke EXPLAIN for dynamic SQL statements An ability to provide DB2 catalog statistics for referenced objects of an access path, or for a group of statements A subsystem parameter browser with the keyword find capabilities The ability to graphically create optimization hints (a feature not found in Visual EXPLAIN V8) You can use OSC to analyze previously generated EXPLAIN data, or to gather EXPLAIN data and clarify dynamic SQL statements. OSC is available for DB2 V8 and DB2 9. DB2 Estimator DB2 Estimator is another useful predictive analysis tool. This product runs on a PC and provides a graphical user interface for entering information about DB2 tables, indexes, and queries. Table and index definitions and statistics can be entered directly or imported to view DDL files. SQL statements can be imported from text files. The table definitions and statistics can be used to accurately predict database sizes. SQL statements can be organized into transactions, and then information about DASD models, CPU size, access paths, and transaction frequencies can be set. After all of this information is put into DB2 Estimator, capacity reports can be produced. These reports contain estimates of the DASD required, as well as the amount of CPU required for an application. These reports are helpful during the initial stage of capacity planning, before any real programs or test data is available. DB2 Estimator is not available with DB2 9. Predicting Database Performance An approach for large systems design is to spend little time considering performance during the development and set aside project time for performance testing and tuning. This strategy frees the developers from having to consider performance in every aspect of their programming, and gives them the incentive to code more logic in their queries. This strategy makes for faster development time, and more logic in the queries means more opportunities for tuning. 84 CA Performance Handbook for DB2 for z/os

85 Predicting Database Performance If you choose to make performance decisions during the design phase, each decision should be backed by solid evidence, and not assumptions. You can create test tables, generate some test data, and write some small programs to simulate a performance situation and test your assumptions about how the database behaves. The feedback from these tests provides data to help you design your system for optimum performance. Reports can then be generated and reviewed by stakeholders. Tools that you can use for testing statements, design ideas, or program processes include, but are not limited to, the following: REXX DB2 programs COBOL test programs Recursive SQL to generate data Recursive SQL to generate statements Data in tables to generate more data Data in tables to generate statements Methods to Generate Data To simulate program access, you need data in tables. You can simply type some data into INSERT statements, insert them into a table, and then use data from that table to generate more data. For example, you might have to test the various program processes against a PERSON_TABLE table and a PERSON_ORDER table. No actual data has been created yet, but you have to test the access patterns of incoming files. You can key some INSERT statements for the parent table, and then use the parent table to propagate data to the child table. For example, if the parent table, PERSON_TABLE, contains this data: PERSON_ID NAME 1 JOHN SMITH 2 BOB RADY The following statement can be used to populate the child table, PERSON_ORDER, with some test data: INSERT INTO PERSON_ORDER (PERSON_ID, ORDER_NUM, PRDT_CODE, QTY, PRICE) SELECT PERSON_ID, 1, 'B100', 10, FROM YLA.PERSON_TABLE UNION ALL SELECT PERSON_ID, 2, 'B120', 3, 1.95 FROM YLA.PERSON_TABLE Chapter 5: EXPLAIN Facility and Predictive Analysis 85

86 Predicting Database Performance The resulting PERSON_ORDER data would look like this: PERSON_ID ORDER_NUM PRDT_CDE QTY PRICE 1 1 B B B B You can repeatedly use the statements to add more data, or additional statements can be executed against the PERSON_TABLE to generate more PERSON_TABLE data. Recursive SQL (DB2 V8 and DB2 9) is a useful way to generate test data. The following is a simple recursive SQL statement: WITH TEMP(N) AS (SELECT 1 FROM SYSIBM.SYSDUMMY1 UNION ALL SELECT N+1 FROM TEMP WHERE N < 10) SELECT N FROM TEMP This statement generates the numbers 1 through 10, one row each. You can use the power of recursive SQL to generate mass quantities of data that can be inserted into DB2 tables, ready for testing. 86 CA Performance Handbook for DB2 for z/os

87 Predicting Database Performance Example The following piece of an SQL statement was used to insert 300,000 rows of data into a large test lookup table. The table was quickly populated with data, and a test was conducted to determine the performance. It was determined that the performance of this large lookup table would not be adequate, but that could not have been confirmed without testing: WITH LASTPOS (KEYVAL) AS (VALUES (0) UNION ALL SELECT KEYVAL + 1 FROM LASTPOS WHERE KEYVAL < 9),STALETBL (STALE_IND) AS (VALUES 'S', 'F') SELECT STALE_IND, KEYVAL,CASE STALE_IND WHEN 'S' THEN CASE KEYVAL WHEN 0 THEN 1 WHEN 1 THEN 2 WHEN 2 THEN 3 WHEN 3 THEN 4 WHEN 4 THEN 4 WHEN 5 THEN 6 WHEN 6 THEN 7 WHEN 7 THEN 8 WHEN 8 THEN 9 WHEN 9 THEN 10 END WHEN 'F' THEN CASE KEYVAL WHEN 0 THEN 11 WHEN 1 THEN 12 WHEN 2 THEN 13 WHEN 3 THEN 14 WHEN 4 THEN 15 WHEN 5 THEN 16 WHEN 6 THEN 17 WHEN 7 THEN 18 WHEN 8 THEN 19 WHEN 9 THEN 20 END END AS PART_NUM FROM LASTPOS INNER JOIN STALETBL ON 1=1; Methods to Generate Statements Just as data can be generated, so can statements. You can write SQL statements that generate statements. For example, you might need to generate single select statements against the EMP table to test a possible application process or scenario. You can write a statement such as the following to generate those statements: SELECT 'SELECT LASTNAME, FIRSTNME ' CONCAT 'FROM EMP WHERE EMPNO = ''' CONCAT EMPNO CONCAT ''';' FROM SUSAN.EMP WHERE WORKDEPT IN ('C01', 'E11') AND RAND() <.33 Chapter 5: EXPLAIN Facility and Predictive Analysis 87

88 Predicting Database Performance The previously noted query generates SELECT statements for approximately 33 percent of the employees in departments C01 and E01. The output appears similar to the following: SELECT LASTNAME, FIRSTNME FROM EMP WHERE EMPNO = '000030'; SELECT LASTNAME, FIRSTNME FROM EMP WHERE EMPNO = '000130'; SELECT LASTNAME, FIRSTNME FROM EMP WHERE EMPNO = '200310'; You can also use recursive SQL statements to generate statements. The following statement was used during the testing of high performance INSERTs to an account history table. The following statement generated 50,000 random insert statements: WITH GENDATA (ACCT_ID, HIST_EFF_DTE, ORDERVAL) AS (VALUES (CAST(2 AS DEC(11,0)), CAST(' ' AS DATE), CAST(1 AS FLOAT)) UNION ALL SELECT ACCT_ID + 5, HIST_EFF_DTE, RAND() FROM GENDATA WHERE ACCT_ID < ) SELECT 'INSERT INTO YLA.ACCT_HIST (ACCT_ID, HIST_EFF_DTE)' CONCAT ' VALUES(' CONCAT CHAR(ACCT_ID) CONCAT ',' CONCAT '''' CONCAT CHAR(HIST_EFF_DTE,ISO) CONCAT '''' CONCAT ');' FROM GENDATA ORDER BY ORDERVAL; Determine Your Access Patterns If you have access to input data or transactions, build small sample test programs to determine the effect of these inputs on the database design. This action involves simple REXX or COBOL programs that contain little or no business logic, just simple queries that simulate the anticipated database access. Running these programs can give you an idea about the impact of random versus sequential processing, or the impact of sequential processing versus skip sequential processing. It can also give you an idea of how buffer settings can affect performance and whether or not performance enhancers, such as sequential detection or index lookaside, will be effective. Simulating Access with Tests Consider writing a simple COBOL program to access a table to understand the following: What is the proper clustering? How effective compression will be for performance? Whether or not one index performs better than another? If adding an index adversely affects INSERTs and DELETEs? 88 CA Performance Handbook for DB2 for z/os

89 Predicting Database Performance Example: A database team debated whether they would gain the greatest flexibility for their database by having an entire application interface use large joins between parent and child tables, or whether all access should be through individual table access (programmatic joins). Coding for both types of access meant extra programming effort, but they had to understand the cost of the programmatic joins for the application. They wrote two simple COBOL programs against a test database one with a two-table programmatic join and the other with the equivalent SQL join. As a result, the team determined that the SQL join consumed 30 percent less CPU than the programmatic join. Chapter 5: EXPLAIN Facility and Predictive Analysis 89

90

91 Chapter 6: Monitoring It is critical to design for performance when building applications, databases, and SQL statements. You have designed the correct SQL, avoided programmatic joins, clustered commonly accessed tables in the same sequence, and have avoided inefficient repeat processing. Your application may be in production and running without problems, but there is still more you can do to increase its performance. Which statement is the most expensive? Is it the tablespace scan that runs once per day, or the matching index scan running millions of times per day? Are all your SQL statements sub-second responders that do not need tuning? Which statement is the number one consumer of CPU resources? All of these questions can be answered by monitoring your DB2 subsystems and the applications that access them. DB2 provides facilities for monitoring the behavior of the subsystem, as well as the applications that are connected to it. This feature is achieved primarily though the DB2 trace facility. DB2 has several different types of traces. DB2 Traces DB2 provides a trace facility to help track and record events within a DB2 subsystem. There are six types of traces: Statistics Accounting Audit Performance Monitor Global These traces should play an integral part in your performance monitoring process. Statistics Trace The data collected in the statistics trace lets you conduct DB2 capacity planning and tune the entire set of DB2 programs. The statistics trace reports information about how much the DB2 system services and database services are used. It is a system-wide trace and should not be used for charge-back accounting. Statistics trace classes 1, 3, 4, 5, and 6 are the default classes for the statistics trace if statistics are specified as 'yes' in the installation panel DSNTIPN. If the statistics trace is started using the START TRACE command, class 1 is the default class. Chapter 6: Monitoring 91

92 DB2 Traces The statistics trace can collect information about the number of threads that are connected, the number of SQL statements that are executed, and the amount of storage that is consumed within the database manager. The statistics trace also collects information about address space, deadlocks, timeouts, logging, and buffer pool utilization. This information is collected at regular intervals for an entire DB2 subsystem. The interval is typically 10 minutes or 15 minutes per record. Accounting Trace The accounting trace provides data that lets you assign DB2 costs to individual authorization IDs and to tune individual programs. The DB2 accounting trace provides information that is related to application programs, such as: Start and stop times Number of commits and aborts The number of times certain SQL statements are issued Number of buffer pool requests Counts of certain locking events Processor resources that are consumed Thread wait times for various events RID pool processing Distributed processing Resource limit facility statistics Accounting times are usually the prime indicator of performance problems, and most often should be the starting point for analysis. DB2 times are classified as follows: Class 1: Specifies the time that the application spent since connecting to DB2, including the time that is spent outside DB2. Class 2: Specifies the elapsed time spent in DB2. It is divided into CPU time and waiting time. Class 3: Specifies elapsed time is divided into various waits, such as the duration of suspensions due to waits for locks and latches or waits for I/O. DB2 trace begins collecting this data at successful thread allocations to DB2 and writes a completed record when the thread terminates or when the authorization ID changes. Having the accounting trace active is critical for proper performance monitoring, analysis, and tuning. When an application connects to DB2, it is executing across address spaces, and the DB2 address spaces are shared by perhaps thousands of users across many address spaces. The accounting trace provides information about the time that is spent within DB2, as well as the overall application time. Class 2 time is a component of class 1 time, and class 3 time a component of class 2 time. 92 CA Performance Handbook for DB2 for z/os

93 DB2 Traces Accounting data for class 1 (the default) is accumulated by several DB2 components during normal execution. This data is collected at the end of the accounting period and does not involve as much overhead as the individual event tracing. Conversely, when you start class 2, 3, 7, or 8, many additional trace points are activated. Every occurrence of these events is traced internally by DB2 trace, however these traces are not written to any external destination. The accounting facility rather uses these traces to compute the additional total statistics that appear in the accounting record when class 2 or class 3 is activated. Accounting class 1 must be active to externalize the information. We recommend that you set accounting classes 1,2,3,7,8. This can add between 4 percent and 5 percent of your overall system CPU consumption. However, if you are already writing accounting classes 1, 2, 3, adding 7 and 8 typically should not add much overhead. Also, if you are using an online performance monitor, it could already have these classes started. If that is the case, adding SMF as a destination for these classes should not add any CPU overhead. Performance Trace The performance trace provides information about various DB2 events, including those related to distributed data processing. You can use this information to identify a suspected problem or to tune DB2 programs and resources for individual users or for DB2 as a whole. To start a performance trace, you must use the START TRACE(PERFM) command. Performance traces cannot be automatically started. Performance traces are expensive to run, and consume much CPU. They also collect a large volume of information. Performance traces are usually run using an online monitor tool, or the output from the performance trace can be sent to SMF and then analyzed using a monitor reporting tool, or sent to IBM for analysis. Because performance traces can consume numerous resources and generate numerous data, you have many options when starting the trace to balance the information that is desired with the resources consumed. These options include limiting the trace data that is collected by plan, package, trace class, and even IFCID. Performance traces are typically used by online monitor tools to track a specific problem for a given plan or package. Reports can then be produced by the monitor software, and can detail SQL performance, locking, and many other detailed activities. Performance trace data can also be written to SMF records, and batch reporting tools can read those records to produce detailed information about the execution of SQL statements in the application. Chapter 6: Monitoring 93

94 Accounting and Statistics Reports Accounting and Statistics Reports After DB2 trace information has been collected, you can create reports by reading the SMF records. These reports are typically produced by running reporting software that is provided by a performance reporting product. These reports are a critical part of performance analysis and tuning. You should familiarize yourself with the statistics and accounting reports, as they are the best way to gauge the health of a DB2 subsystem and the applications using it. These reports are also the best way to monitor performance trends and proactively detect potential problems before they become critical. Statistics Report RID Pool Failures Because statistics records are collected typically at 10-minute or 15-minute intervals, many records can be collected on a daily basis. Your reporting software should be able to produce summary reports, which can gather and summarize the data for a period, or detail reports that can report on every statistics interval. Start with a daily summary report, and look for specific problems within the DB2 subsystem. After you detect a problem, you can produce a detailed report to determine the specific period that the problem occurred, and also coordinate the investigation with detailed accounting reports for the same time period to attribute the problem to a specific application or process. We recommend that you use statistics reports on a regular basis, and use monitoring software documentation, along with the DB2 Administration Guide (DB2 V8) or DB2 Performance Monitoring and Tuning Guide (DB2 9). Note the following in a statistics report: The statistics report should include a reason to detail the usage of the RID pool for activities such as list prefetch, multiple index access, and hybrid joins. The report will also indicate RID failures. There can be RDS failures, DM failures, and failures due to insufficient size. If you are getting failures due to insufficient storage you can increase the RID pool size. However, if you are getting RDS or DM failures in the RID pool, the access path that is selected may be reverting to a table space scan. In these situations, determine which applications are getting these RID failures. Therefore, you have to produce a detailed statistics report that can identify the time of the failures, and also produce detailed accounting reports that show which threads are getting the failures. You to determine the packages within the plan. DB2 EXPLAIN can be used to determine which statements are using list prefetch, hybrid join, or multi-index access. You may have to test the queries to determine if they are the one's seeing the failures, and if they are, try to influence the optimizer to change the access path. 94 CA Performance Handbook for DB2 for z/os

95 Accounting and Statistics Reports Bufferpool Issues One of the most valuable piece of information in a statistics report is the section covering buffer utilization and performance. For each buffer pool in use, the report includes the size of the pool, sequential and random getpages, prefetch operations, pages that are written, and number of sequential I/Os, buffer thresholds that are reached, random I/Os, and write I/Os, and much more. Watch for the number of synchronous reads for sequential access, which may be an indication that the number of pages is too small and pages for a sequential prefetch are stolen before they are used. Additionally watch whether any critical thresholds are reached, if there are write engines not available, and whether deferred write thresholds are triggering. It is also important to monitor the number of getpages per synchronous I/O, as well as the buffer hit ratio. Logging Problems The statistics report provides important information about logging. This information includes the number of system checkpoints, number of reads from the log buffer, active log data sets, or archived log datasets, number of unavailable output buffers, and total log writes. This information could give you an indication whether you have to increase log buffer sizes or you have to investigate frequent application rollbacks or other activities that could cause excessive log reads. EDM Pool Hit Ratio Deadlocks and Timeouts The statistics report details how often database objects such as DBDs, cursor tables, and package tables are requested as well as how often those requests have to be satisfied using a disk read to one of the directory tables. You can use this information to determine if you should increase the EDM pool size. You also receive statistics about the use of dynamic statement cache and the number of times statement access paths were reused in the dynamic statement cache. This information could let you see the size of your cache and its effectiveness, but it could also highlight potential reusability of the statements in the cache. The statistics report provides subsystem-wide perspective on the number of deadlocks and timeouts your applications have experienced. You can use this information as an overall method of detecting deadlocks and timeouts across all applications. If the statistics summary report shows a positive count, you can use the detailed report to determine what time the problems are occurring. You can also use accounting reports to determine which applications are experiencing the problem. Chapter 6: Monitoring 95

96 Accounting and Statistics Reports Accounting Report Class 1 and Class 2 Timings Buffer Usage Package Execution Times An accounting report reads the SMF accounting records to produce the thread or application level information from the accounting trace. These reports typically can summarize information at the level of a plan, package, correlation ID, authorization ID, and so on. In addition, you can also produce one report per thread. This accounting detail report can provide detailed performance information for the execution of a specific application process. If you have accounting classes 1, 2, 3, 7, and 8 turned on, the information will be reported at the plan and the package level. You can use the accounting report to find specific problems within certain applications, programs, or threads. Indicates the entire application time, including the time that is spent within DB2. Class 2 is a component of class 1, and represents the amount of time the application spent in DB2. The first question to ask when an application is experiencing a performance problem is, "Where is the time being spent?" The first indication that the performance issue is with DB2 will be a high class 2 time relative to the class 1 time. Within class 2, you could be having a CPU issue (CPU time represents most of the class 2 time), or a wait issue (CPU represents little of the overall class 2 time, but class 3 wait represents most of the time), or maybe your entire system is CPU bound (Class 2 overall elapsed is not reflected in class 2 CPU and class 3 wait time combined). Displays information about buffer usage at the thread level. This information can be used to determine how a specific application or process is using the buffer pools. If you have situations in which certain buffers have high random getpage counts, look at which applications are causing the high number of random getpages. You can use this thread level information to determine which applications are accessing buffers randomly versus sequentially. Perhaps then you can see which objects the application uses, and can use this information to separate sequentially accessed objects from randomly accessed objects into different buffer pools. The buffer pool information in the accounting report also indicates how well the application is using the buffers. The report can be used during buffer pool tuning to determine the impact of buffer changes on an application. Displays information about the DB2 processing on a package level, if accounting classes 7 and 8 are turned on. This information is important for performance tuning because it lets you determine which programs in a poorly performing application should be reviewed first. 96 CA Performance Handbook for DB2 for z/os

97 Online Performance Monitors Deadlocks, Timeouts, and Lock Waits Excessive Synchronous I/Os RID Failures High Getpage Counts and High CPU Time Indicates the number of deadlocks and timeouts that occurred on a thread level. It also reports the time that the thread spent waiting on locks. This information lets you know whether you have to do additional investigation into applications that are having locking issues. Indicates if there are many excessive random synchronous I/Os being issued, and how much time the application spends waiting on I/O, if you have a slow running job or online process. The report also indicates exactly what is slow about that job or process. The information in the report can also be used to approximately determine the efficiency of your DASD by simply dividing the number of synchronous I/O's into the total synchronous I/O wait time. Displays thread level RID pool failure information. This information provides important details to help you determine if you have access path problems in a specific application. Indicates if your performance problem is related to an inefficient repeat process. If the report shows a high getpage count, or that most of the elapsed time is actually class 2 CPU time, you may have an inefficient repeat process in one of the programs for the plan. Often it is hard to determine if an application is doing repeat processing when there are not many I/Os being issued, so you can use the package level information to determine which program uses the most CPU and you can try to identify any inefficiencies in that program. Online Performance Monitors DB2 can write accounting, statistics, monitor, and performance trace data to line buffers. This ability allows for online performance reporting software to read those buffers, and report on DB2 subsystem and application performance in real time. These online monitors can be used to review subsystem activity and statistics, thread level performance information, deadlocks and timeouts, I/O activity, and dynamically execute and report on performance traces. Chapter 6: Monitoring 97

98 Overall Application Performance Monitoring Overall Application Performance Monitoring You will not find one method for improving the overall performance of large systems. We can tune the DB2 subsystem parameters, logging and storage, but often we are only making a bad situation worse. In these circumstances, the "80/20" rule applies, and you will eventually have to address application performance. You can quickly identify the easy to recognize performance issues in a report to your boss, and change the application or database to support a more efficient path to the data. Management support is required and an effective manner of communicating performance tuning opportunities and results is crucial. Setting the Appropriate Accounting Traces DB2 accounting traces play a valuable role in reporting on application performance tuning opportunities. DB2 accounting traces 1, 2, 3, 7, and 8 must be set to monitor performance at the package level. After you do that, you can further examine the most expensive programs (identified by package) to look for tuning changes. This reporting process can serve as a quick solution to identifying an application performance problem, but you can incorporate it into a long-term solution that identifies problems and tracks changes. Some have expressed concern about the performance impact of this level of DB2 accounting. The IBM DB2 Administration Guide (DB2 V8) or the DB2 Performance Monitoring and Tuning Guide (DB2 9) states that the performance impact of setting these traces is minimal and the benefits can be substantial. Tests that are performed at a customer site demonstrated an overall system impact of 4.3 percent for all DB2 activity when accounting classes 1, 2, 3, 7, and 8 are started. In addition, adding accounting classes 7 and 8 when 1, 2, and 3 are already started has the nominal impact, as does the addition of most other performance monitor equivalent traces (that is, your online monitor software). Summarizing Accounting Data To communicate application performance information to management, the accounting data must be organized and summarized up to the application level. You need a reporting tool that formats DB2 accounting traces from System Management Facilities (SMF) files to produce the type of report you are interested in. Most reporting tools can produce DB2 accounting reports at a package summary level. Some can even produce customized reports that can filter only the information you want out of the wealth of information in trace records. 98 CA Performance Handbook for DB2 for z/os

99 Overall Application Performance Monitoring You can process whatever types of reports you produce so that a concentrated amount of information about DB2 application performance can be extracted. This information is reduced to the amount of elapsed time and CPU time the application consumes daily and the number of SQL statements each package issues daily. This highly specific information is your first indication as to which packages provide the best DB2 tuning opportunity. The following example is from a package-level report with the areas of interest highlighted (Package Name, Total DB2 Elapsed, Total SQL Count, Total DB2 TCB): PACKAGE EXECUTIONS - AVERAGE TOTAL ---- HHH:MM:SS.TTT HHH:MM:SS.TTT DB2 TCB : COLL ID CPG2SU01 I/O :28: PROGRAM FHPRDA2 LOCK/LATCH OTHER RD I/O : HHH:MM:SS.TTT OTHER WR I/O AVG DB2 ELAPSED DB2 SERVICES TOTAL DB2 ELAPSED 010:55: LOG QUIESCE DRAIN LOCK AVG SQL COUNT 86.9 CLAIM RELEASE TOTAL SQL COUNT ARCH LOG READ AVG STORPROC EXECUTED 0.0 PG LATCH CONT TOT STORPROC EXECUTED 0 WT DS MSGS AVG UDFS SCHEDULED 0.0 WT GLBL CONT TOT UDFS SCHEDULED 0 GLB CHLD L-LK GLB OTHR L-LK GLB PSET P-LK GLB PAGE P-LK GLB OTHR P-LK OTHER DB Chapter 6: Monitoring 99

100 Overall Application Performance Monitoring If you do not have access to a reporting tool that can filter pieces of information, you can write a simple program in any language to read the standard accounting reports and pull out the information you need. REXX is an excellent programming language that is well-suited to this type of "report scraping"; you can write a REXX program to do such work in a few hours. If you want to avoid dependency on reporting software, you could write a slightly more sophisticated program to read the SMF data directly to produce similar summary information. After the standard reports are processed and summarized, the information for a specific interval (say one day) can appear in a simple spreadsheet. You can sort the spreadsheet by descending CPU. With high consumers at the top of the report, the obvious issues are then easy to spot. The following spreadsheet can be derived by extracting the fields of interest from a package-level summary report: Package Executions Total Elapsed Total CPU Total SQL ElapsedExecution CPUExecution ElapsedSQL CPUSQL ACCT , ACCT ACCTB RPTS ACCT RPTS HRPK HRPKB HRBKB CUSTB CUSTB CUST CUSTB RPTS CUST CUST CUSTB CUSTB Identify basic issues when choosing the first programs to address. For example, package ACCT001 consumes the most CPU each day, and issues nearly 2 million SQL statements. Although the CPU consumed per statement on average is low, the quantity of statements that are issued indicates an opportunity to save significant resources. If just a small amount of CPU can be saved, it will quickly add up. The same recommendation applies to package ACCT002 and packages RPTS001 and RPTS002. These are some of the highest consumers of CPU, and they also have a relatively high average CPU per SQL statement. This information indicates that there may be some inefficient SQL statements involved. Because the programs consume significant CPU per day, tuning these inefficient statements could yield significant savings. ACCT001, ACCT002, RPTS001, and RPTS002 represent the best opportunities for saving CPU, so examine those first. Without this type of summarized reporting, it is difficult to do any sort of truly productive tuning. 100 CA Performance Handbook for DB2 for z/os

101 Overall Application Performance Monitoring Reporting to Management To do this type of tuning, you need approval from management and application developers. These approvals can sometimes be the most difficult part because, unfortunately, most application tuning involves costly changes to programs. To demonstrate the potential Return On Investment (ROI) for programming time, report the cost of application performance problems in terms of dollars. The summarized reports can report on information on the application level. An in-house naming standard can be used to combine all the performance information from various packages into application-level summaries. This practice lets you classify applications and address the ones that use the most resources. For example, if the in-house accounting application has a program naming standard where all program names begin with "ACCT," the corresponding DB2 package accounting information can be grouped by this header. Thus, the DB2 accounting report data for programs ACCT001, ACCT002, and ACCT003 can be grouped together, and their accounting information summarized to represent the "ACCT" application. Most capacity planners have formulas for converting CPU time into dollars. If you get this formula from the capacity planner, and you categorize the package information by application, you can easily turn your daily package summary report into an annual CPU cost per application. The following shows a simple chart that is developed using an in-house naming standard and a CPU-to-dollars formula. Chapter 6: Monitoring 101