Best practices. IBMr. How to use OMEGAMON XE for DB2 Performance Expert on z/os to identify DB2 deadlock and timeout. IBM DB2 Tools for z/os

Transcription

1 IBMr IBM DB2 Tools for z/os Best practices How to use OMEGAMON XE for DB2 Performance Expert on z/os to identify DB2 deadlock and timeout Hong Zhou Advisory Software Engineer Issued: December, 2013

2 Identifying DB2 deadlock and timeout page 2 of 26

3 How to use OMEGAMON XE for DB2 Performance Expert on z/os to identify DB2 deadlock and timeout...1 Executive Summary...4 Introduction...5 Best Practices...5 Monitor DB2 Deadlock And Timeout Activities...6 Collect Deadlock And Timeout IFCIDs...8 Analyze Trace Data Using OMPE Report...10 DEADLOCK Trace Output...11 Deadlock Analysis Tips...14 Timeout Trace Output...14 Timeout Analysis Tips...15 Analyze History Data Using Near Term History Analyze Deadlock Scenario Using NTH...19 Analyze Timeout Scenario Using NTH...20 Conclusion...21 Further reading...23 Contributors...23 Notices...24 Trademarks...25 Contacting IBM...25 Identifying DB2 deadlock and timeout page 3 of 26

4 Executive Summary Deadlocks and timeouts are unresolved contentions in DB2. These unresolved contentions might cause incomplete transactions and might degrade the overall performance of your system. It is important to monitor and resolve these contentions. This best practice discusses steps to detect and analyze deadlocks and timeouts using Tivoli OMEGAMON XE for DB2 Performance Expert on z/os (OMPE). Various scenarios and setups using OMPE batch report, OMPE's real-time monitoring interface, and OMPE's Near Term History (NTH) component are provided. Identifying DB2 deadlock and timeout page 4 of 26

5 Introduction DB2 deadlock and timeout are thread termination situations due to unresolved resource contentions. Timeouts appear as lock conflicts initially. When the contention between threads is unable to resolve within a period of time set by system parameter IRLMRWT(resource timeout field), the thread that is waiting for the resource is terminated with the timeout status. A deadlock is a situation in which two or more requesters are waiting for resources that are held by each other. These contentions cannot be resolved by themselves. When DB2 detects these deadlock situations, one thread is canceled, and the other thread continues. The deadlock detection is controlled by the DEADLOCK TIME system parameter for local deadlock detection. In a data sharing environment, the DEADLOCK CYCLE system parameter controls the global deadlock detection process as well. Not all deadlock situations are detected by DB2 as some of the resources might not be controlled by DB2. In these situations, the deadlock might appear as a timeout event. DB2 provides IFCID 0172 and IFCID 0196 for thread deadlock and timeout events. These IFCIDs can be collected for deadlock and timeout analysis. OMEGAMON XE for DB2 Performance Expert on z/os (OMPE) provides several alternatives to show the real time and interval statistics from DB2 IFCIDs. You can either use OMPE batch report, OMPE's classic interface, and Near Term History (NTH) component to monitor and analyze the deadlock and timeout activities in your DB2 subsystems. Best Practices Monitor system level deadlock and timeout statistics. Collect IFCID 0172 and IFCID 0196 data when deadlock and timeout appear in DB2 subsystem by o o o Start NTH with LOCKCONT(YES) And/Or Start class 3 DB2 statistics trace to SMF Use OMPE to analyze collected data to pinpoint the cause of the deadlock or timeout. Correct the application or environmental problem to prevent deadlock and timeout from happening in the future. Identifying DB2 deadlock and timeout page 5 of 26

6 Monitor DB2 Deadlock And Timeout Activities When deadlocks and timeouts happen in your DB2 subsystem, these activities are logged in DB2 system services address space (MSTR address space) log. You can scan DSNT375I and DSNT376I messages from the MSTR address space log for DB2 deadlock and timeout activities. Illustration 1: MSTR Address Space Log You can also use OMPE's classic interface or OMPE batch report to monitor DB2 deadlock and timeout activities. From classic interface main menu, you can enter fast path R.I to view the Lock Manager Information menu. Both deadlock and timeout counts in total quantity, and interval quantity are displayed. The total quantity reflects the amount of activity that occurred since DB2 was started. The interval quantity reflects the amount of activity that occurred during the reporting interval, which is 1 second in this example. Identifying DB2 deadlock and timeout page 6 of 26

7 Illustration 2: Classic Interface Main Menu Illustration 3: Classic Interface Lock Manager Informationm You can also run 'Statistics Lock Activity' batch report to monitor DB2 deadlock and timeout activities if the statistics trace data collection is started with the deadlock and timeout IFCIDs. The Locking Activity report section shows the total timeout and deadlock counts since DB2 was started. Identifying DB2 deadlock and timeout page 7 of 26

8 Illustration 4: Statistics - Lock Activity Batch Report Ideally, the deadlock and timeout counts are zero. When deadlock and timeout activities are increasing as monitored from the classic Lock Manager Information display, DB2 MSTR address space log, or 'Statistics - Locking Activity' batch report, start collecting the deadlock and timeout IFCIDs, so that deadlock and timeout conditions can be further analyzed. Collect Deadlock And Timeout IFCIDs If your DB2 subsystem has the statistics trace to SMF turned on already, you could start deadlock and timeout analysis immediately when deadlock and timeout activities appear in your DB2 subsystem. Statistics trace class 3 includes the deadlock and timeout IFCIDs, which are IFCID 0172 and IFCID By using OMPE batch report, you can find the details about deadlock and timeout events. If the required trace is not turned on yet, you can use the DB2 start trace command to start the class 3 statistics trace. For example, -START TRACE(STAT) CLASS(1,3) DEST(SMF) OMPE classic online Active Trace Detail display (use short path R.F from classic main menu, then press PF11 while the cursor is on the trace entry) shows the list of IFCIDs that each active DB2 trace starts. You can use the trace detail display to find out whether the required IFCIDs for deadlock and timeout are started on your DB2 subsystem or not. Identifying DB2 deadlock and timeout page 8 of 26

9 Illustration 5: Classic Interface Active Trace Detail OMPE NTH provides another alternative to collect and analyze deadlock and timeout related thread data. The NTH data collection is a configurable component, and can be started when history data is needed for each DB2 subsystem. The data can be collected to VSAM data sets or sequential data sets or both. The classic NTH online interface uses the data that is collected in VSAM data sets, while the trace data in sequential data sets can be used by the OMPE batch report. With a similar user interface as the rest of the classic, the NTH online provides easy navigation to various thread related history data panels such as SQL call information, thread plan and package detail, sort and scan data that makes deadlock and timeout analysis a lot easier than the batch report approach. To collect deadlock and timeout related data in NTH, set LOCKCONT option to YES in your COPTssid member in HLQ.RKD2PAR of your OMPE installation, then start NTH subtask for that DB2 subsystem. To verify that the required IFCIDs for deadlock and timeout are started by the NTH subtask, you can check the OMPE server log after the required DB2 traces are started. The NTH trace output for DB2 subsystem DB1H is displayed here and it has both IFCID 0172 and IFCID 0196 started for deadlock and timeout analysis. Identifying DB2 deadlock and timeout page 9 of 26

10 Illustration 6: OMPE Server log You can also dynamically alter the deadlock and timeout IFCIDs collection without restarting your NTH subtask or OMPE server started task. If you start the NTH subtask without the deadlock and timeout IFCIDs, and you want to start monitoring these events in NTH, you can use the following steps: 1. Change the COPTssid in your install RKD2PAR, set LOCKCONT(YES), save the changes. 2. Issue the modify command to alter the already started NTH subtask H2ssid: /f ompetask,f H2ssid,VARY OPTION=COPTssid Analyze Trace Data Using OMPE Report When you have deadlock and timeout IFCIDs data that is captured in SMF data sets, or in the NTH sequential data sets, you can use various OMPE batch reports to analyze these events. In this best practice, deadlock and timeout trace reports are used to view the collected IFCID data. The deadlock trace report contains an entry for every occurrence of a deadlock during a specified time period. While the timeout trace report shows the timestamp when a timeout event occurred, the resource involved and information about the threads either held the resource or wait to get hold of the resource. To prepare for these batch reports, you need: 1. Input trace data sets. They can be SMF data sets, or NTH sequential data sets. 2. Deadlock and timeout messages from DB2 MSTR address space log if you have them. The messages provide the timestamps when deadlock or timeout events happened. By having the time frame of the events, you can reduce the amount of report output. 3. JCL for the batch Locking Trace report. In the following scenario, DB2 statistics trace data is captured in the SMF data set while some deadlock or timeout activities show in the DB2 MSTR address space log: Identifying DB2 deadlock and timeout page 10 of 26

11 Illustration 7: MSTR Address Space Log Deadlock Event Illustration 8: MSTR Address Space Log - Timeout Event To limit the report output, set the date and time collected from the DB2 MSTR address space log close to the deadlock and timeout events that are used in the report input (that is, keywords FROM and TO). The report outputs are directed to the default DDs in job output: the deadlock report is directed to the first default DD (LOTRCDD1), and the timeout report is directed to the second default DD (LOTRCDD2): Illustration 9: Batch Report Sample JCL Identifying DB2 deadlock and timeout page 11 of 26

12 DEADLOCK Trace Output The Illustration-10 and Illustration-11 are deadlock report output from default DD LOTRCDD1 of the locking trace report. Illustration 10: Locking Trace Report - Deadlock Page 1-1 Identifying DB2 deadlock and timeout page 12 of 26

13 Illustration 11: Locking Trace Report - Deadlock Page 1-2 To help analyzing the deadlock situation, some key attributes from the deadlock report are extracted to the following table. A deadlock is between two active threads and due to incompatible locks held among them. Both report pages present data using the victim thread's viewpoint. Illustration-10 shows the resource contention when the victim thread is a blocker of the resource #1, while Illustration-11 shows the resource contention when the victim thread is a waiter of the resource#2. In this scenario, the deadlock is between two threads with incompatible locks on one single resource (DB=TDKDB, OB=4). In other deadlock situations, you might see more threads that are involved with different resources. The victim (thread#1) is the blocker and the holder of resource#1 in the first contention, while the thread#2 is waiting for blocker (thread#1) to release the resource. In the second contention, thread#2 is the blocker that is holding the shared lock on resource#2, while thread#1 is waiting for the blocker (thread#2) to release the lock on resource#2. DB2 detects the deadlock situation, and victimizes the thread#1. Identifying DB2 deadlock and timeout page 13 of 26

14 Illustration-10 Thread#1 (Victim) Thread#2 Blocker/Holder/Waiter Block/Holder Waiter Correlation ID HONGLD01 HONGLD02 Lock Attributes Statement Type/ID Type=Table Lock State=Shared Duration=Commit Type=Static ID='7336'x Type=Table Lock State=Exclusive Duration=Commit Request=Change Lock Type=Static ID='733A' Resource #1 DB=TDKDB, OB=4 DB=TDKDB, OB=4 Illustration-11 Thread#1(Victim) Thread#2 Blocker/Holder/Waiter Waiter Blocker/Waiter Correlation ID HONGDL01 HONGDL02 Lock Attributes Statement Type/ID Type=Table Lock State=Exclusive Duration=Commit Request=Change Lock Type=Static ID='7337'x Type=Table Lock State=Exclusive Duration=Commit Type=Static ID='733A'x Resource #2 DB=TDKDB, OB=4 DB=TDKDB, OB=4 Table 1: Deadlock Attribute Table Deadlock Analysis Tips Most deadlocks situations require programming changes to correct the situation. Consider the following when deadlock Identifying DB2 deadlock and timeout page 14 of 26

15 situations are analyzed. Access resources in the same sequences in concurrent applications might avoid deadlock situation. In some cases, by using page locks or row locks to replace existing table level, partition level, or table space level locks might resolve the deadlock issues. Commit more frequently, change the bind options to release the held locks quickly might also help with deadlock situations. Timeout Trace Output The Illustration-12 is the timeout data from default DD LOTRCDD2 of the locking activity report: Illustration 12: Locking Trace Report - Timeout Page 1-1 To help analyzing the timeout situation, the key attributes are extracted from the timeout report. In this scenario, both threads were requesting the same table level lock when timeout happened. The holder (Thread#2) has the shared table lock that is released only at commit time, causing the waiter (Thread#1) requesting an exclusive table lock to timeout. Identifying DB2 deadlock and timeout page 15 of 26

16 Illustration-12 Thread#1(Victim) Thread#2 Holder/Waiter Waiter Holder Correlation ID HONGDLK2 HONGDLK1 Lock Attributes State=Exclusive Duration=Commit Unconditional Statement Type Static Static Statement ID '733D'x '733F'x State=Shared Duration=Commit Requested Resource DB=DLOCKDB,OBID=3 DB=DLOCKDB,OBID=3 Table 2: Timeout Attribute Table Timeout Analysis Tips System parameter IRLMRWT setting. A low IRLMRWT setting sometimes might cause unnecessary timeout situations. Lock granularity with LOCKSIZE Some of the timeouts can be avoided when page lock or row lock is used. Use these locks with caution as they increase the lock overhead with more storage usage and more processing time. Commit frequency Frequent commit can make sure that locks are released at unit of work boundary. Close the cursor before commit is also important as opened cursor might hold the lock on resources, thus causing other threads to timeout. Bind options Some plan or package bind options affect how locks are held by the application. Identifying DB2 deadlock and timeout page 16 of 26

17 ISOLATION options specify how quickly certain DB2 locks on pages and rows are released. Choosing the correct isolation option might reduce the number of timeouts for these applications. RELEASE options affects locks on table, partition, and table space. It applies mostly on static SQLs. But when system parameter CACHEDYN is set to YES and KEEPDYNAMIC bind option is set to YES, the RELEASE bind option applies to some dynamic SQL also. CURRENTDATA is another bind option that might affect data currency. Use it with ISOLATION(CS) and system parameter SKIPUNCI to control the data concurrency access. Possible Deadlock situations Some timeout situations are actually deadlocks. This could happen when system parameter DEADLOCK TIME is not set properly causing a thread to timeout before a deadlock can be detected. Or, the resource contention is outside the scope of DB2 deadlock detection. For example, when threads are originating from a distributed environment or when semaphores are used to control the resources. Analyze History Data Using Near Term History To analyze deadlock and timeout conditions with NTH after data is collected to the VSAM data sets, you can use the following paths: 1. From classic main menu Identifying DB2 deadlock and timeout page 17 of 26

18 Illustration 13: Classic Interface Main Menu 2. Use the DEADLK/TIMEOUT filter option to limit the thread detail display to only those threads with the non-zero deadlock or timeout count. Illustration 14: NTH Filter Options 3. On Thread History by Report Interval display Identifying DB2 deadlock and timeout page 18 of 26

19 Illustration 15: NTH Thread History By Report Interval 4. On Thread History Summary display Illustration 16: NTH Thread History Summary Identifying DB2 deadlock and timeout page 19 of 26

20 5. From Thread History Detail display, enter C from command line, Thread History Lock Waits with the deadlock or timeout information is displayed. Illustration 17: NTH Thread History Detail Analyze Deadlock Scenario Using NTH In this scenario, two programs are involved in the deadlock situation. Both programs access the same table, but update two different rows. Because of the way the locks are obtained, two programs are ended with deadlock situation and DB2 deadlock detect process cancels one thread with the deadlock termination condition. The corresponding NTH Thread History Lock Waits display shows the deadlock information. You can find the two parties that are involved in this deadlock event (HONGDLA1 and HONGDLA2). Both threads were waiting for the same resource (DBID=0270 TABL=0004) which are held by each other. By using NTH, you can also find out more information about these threads such as what SQL calls were issued by this thread, what packages were involved, or any SQL text if it is available. Identifying DB2 deadlock and timeout page 20 of 26

21 Illustration 18: NTH Thread History Lock Wait Deadlock Event Analyze Timeout Scenario Using NTH Using the similar path as deadlock analysis, you can find the timeout information from NTH's Thread History Lock Wait display. You can find the thread detail about the thread that was terminated by DB2 (victim) which was waiting for the specific resource that is not available. You can also find out the owner (holder) of the resource, the higher priority waiters of the resource if there are any, and other useful information. In data sharing environment, you might also see retained locks that are holding the resource. Identifying DB2 deadlock and timeout page 21 of 26

22 In this timeout scenario, the resource is a row in page 2 of the table space TMOUTDB1.TMOUTS1. The owner has exclusive lock on that row and only releases it at commit time. There are three other threads, including the current timed out thread (victim), that are requesting the same row lock. The other two threads have a higher priority than the current timed out thread (Showed as PWait). You can also find out that the system parameter of IRLMRWT that is set to 30 seconds (Timeout Interval) on the same display. Illustration 19: NTH Thread History Lock Waits Timeout Event Conclusion Deadlock and timeout events cause applications to terminate without successful completion. They waste resources and slow down system performance. OMPE can detect and analyze these events. You can use an OMPE batch report,the OMPE real-time monitoring interface, and the OMPE Near Term History (NTH) interface to find the information you need to correct deadlock and timeout events. You can resolve issues that are found by tuning the system parameters or correcting application logic errors. Identifying DB2 deadlock and timeout page 22 of 26

23 By following suggestions that are given in this paper, you can improve the operation of your DB2 environment. Identifying DB2 deadlock and timeout page 23 of 26

24 Further reading DB2 Best Practices : Tivoli OMEGAMON XE for DB2 Performance Expert on z/os: on.xe.pe_db2.doc_5.2.0/ko2welcome_pe.htm Contributors David Salinero Senior Software Engineer Identifying DB2 deadlock and timeout page 24 of 26

25 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON- INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. Without limiting the above disclaimers, IBM provides no representations or warranties regarding the accuracy, reliability or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS. The use of this information or the implementation of any recommendations or techniques herein is a customer responsibility and depends on the customer s ability to evaluate and integrate them into the customer s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Anyone attempting to adapt these techniques to their own environment do so at their own risk. This document and the information contained herein may be used solely in connection with the IBM products discussed in this document. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Identifying DB2 deadlock and timeout page 25 of 26

26 Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: Copyright IBM Corporation All Rights Reserved. This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Windows is a trademark of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Contacting IBM To provide feedback about this paper, write to db2zinfo@us.ibm.com To contact IBM in your country or region, check the IBM Directory of Worldwide Contacts at To learn more about IBM Information Management products, go to Identifying DB2 deadlock and timeout page 26 of 26