Time Breakdowns A novel way to analyze performance Problems

Transcription

1 Time Breakdowns A novel way to analyze performance Problems Roy Cecil IBM Session Code: 3 16 April 2014 Platform: LUW

2 2 Agenda Introduction Amdahl's law Time Breakdowns Examples Bottlenecks & Little law

3 Every performance problem is an optimization problem! #IDUG

4 What are we optimizing? Business Metric Transactions processed per second Number of orders processed per day Number of containers moved per hour at a dock Defects in a factory line In the context of DB2 Transactions per second (Throughput) Number of Reports processed per second in a Data Warehouse Response time of shopping cart transactions

5 What are the attributes of a business metric? Is it quantifiable? Under identical conditions is it repeatable? Is the problem within the accuracy with which the metric is expressed? If being compared across systems are they comparable?

6 Setting Goals Do all stakeholders sign off? #IDUG

7 7 Agenda Introduction Amdahl's law Time Breakdowns Examples Bottlenecks & Little law

8 Amdahl's law : The speedup achievable from an improvement to a computation that affects a proportion P of that computation where the improvement has a speedup of Sis given by 1 / [ ( 1 P ) + P/S ] Q P = 0.8 Q P/2 =0.4 Q = 1- P 1.67 x ie 67% overall improvement Q/2 P = x ie only 11% improvement It doesnt pay a lot to improve breakdowns that consume a negligible fraction of the overall elapsed time.

9 Law of Diminishing return : If the response time is broken down to various sub units and one picked optimally what to improve then one would see monotonically decreasing improvement as one improves. Time breakdown of a Transaction Optimal A B -> A -> C -> D B C D Sub-optimal D -> A -> C -> B

10 10 Agenda Introduction Amdahl's law Time Breakdowns Examples Bottlenecks & Little law

11 Time Breakdowns The fundamental unit of work in DB2 is the Transactions. Most often the Business metric being optimized is directly translated to Database throughput. Throughput is defined as the number of transactions that completedinagivenunitoftime. An alternate view of the Business metric that derives directly from the Throughput is the Average Response Time of the Transactions. Throughput ( T ) = 1 / R [ R is the Average Response Time of the transactions]

12 Time Breakdowns To obtain the Time breakdowns, the response time of the individual transactions or the total time spent in DB2 are broken down into the time spent in various components while processing that transaction. Coupled with Amdahl's law we should have a better understanding of what component to optimize to be able to get a greater throughput

13 Two dimensions of viewing Time breakdowns System wide Dimension Used to see system wide time breakdowns. Can also be used to see the time breakdowns per workload. Useful if there is a systemic performance problem rather than a problem with specific activities. Specific Activity Dimension Can be used to see time breakdowns of specific activities. For example time breakdowns of a Unit of Work Rolled up into System wide time breakdowns.

14 Three aspects of time monitored in DB2 Wait Times Waiting for latches or Locks Waiting for incoming client requests Reading or Writing from Bufferpool Component processing Times Committing or rolling back transactions Compiling SQL Component elapsed times Total Commit time Total compile time

15 Some facts to remember Time Breakdown monitor elements The Wait time elements and Component processing times are complementary to each other. Σ ( wait times ) + Σ ( component processing time ) total_rqst_time The small difference arises from components not monitored in DB2. This is sufficient to provide good insight into problem areas. Component elapsed time includes component processing times. Component elapsed times already includes wait times Not meaningful to use component times and wait times as that would amount to double counting.

16 Breaking down Total Elapsed Time System Dimension Total Elapsed Time ( wall clock ) is the sum of CLIENT_IDLE_WAIT_TIME and TOTAL_RQST_TIME CLIENT_IDLE_WAIT_TIME and TOTAL_RQST_TIME should be normalized by the number of connections. CLIENT_IDLE_WAIT_TIME is the amount of time DB2 spent waiting for work from Clients. TOTAL_RQST_TIME is the amount of time spent doing useful work in DB2.

17 Breaking down Total Elapsed Time - Example A snippet from MON_GET_WORKLOAD. Monitoring Interval = 200 seconds ; Number of clients = 40 Both TOTAL_RQST_TIME and CLIENT_IDLE_WAIT_TIME is expressed in milliseconds The elapsed time is the same as the Monitoring interval. TOTAL_RQST_TIME CLIENT_IDLE_WAIT_TIME TOTAL_RQST_TIME = CLIENT_IDLE_WAIT_TIME = Elapsed Time = ( ) / ( 40 * 1000 ) = 200 TOTAL_RQST_TIME = / ( 40 * 1000) % TOTAL_RQST_TIME = / 200 * 100 = 43.48%

18 Breaking down Total Request Time Time spent in DB2 Total Time spent processing Requests can be broken down into Total Wait time events in DB2 Total Compile Times Explicit ( initiated by user prep/bind/rebind etc ) Implicit ( not initiated by user ) Total user code in routines Total Section processing times Total Commit & Rollback times Total Runstat, Reorg and Load processing times Total Connection Request times Others

19 Performance Objective Reduce Wait times #IDUG

20 Significant Wait times Performance opportunities Lock Wait Times. Log Buffer and Log Disk wait times. Network Wait Times I/O Wait times TCP / IPC / FCM Send & Receive wait times Pool Read/Write times Direct Read/Write times Prefetch wait times. CF ( Caching Facility in DB2 purescale ) wait times. Reclaim Wait Times Latch Wait times Others.

21 Understanding Lock Wait times. The average time the applications spend waiting for locks is given by lock_wait_time / lock_waits Are the waits due to Lock Escalations ( lock_escals )? Increase LOCKLIST / MAXLOCKS mon_get_locks and mon_get_appl_lockwait can be used to diagnose Lock wait problems and help break down the picture to individual packages/statements. General good ideas for reducing lock wait Use the lowest isolation level that meets business requirement Commit optimally. Strive for highly selective access plans ( indexed vs scanned ) If we see a lot of row locks with reads ( S locks ) waiting on writes ( X locks ) enabling CUR_COMMIT database configuration can help when CS isolation is in force.

22 Log Buffer Wait times. A large wait by the agent for Log buffer is indicative of a higher rate at which the buffer fills up before being flushed to the disk. num_log_buffer_full need to be checked to see if applications cannot insert into the log buffer because the full buffer is being flushed to the disk. If num_log_buffer_full is zero a very probable cause for this wait is implicit commits. Tuning the LOGBUFSZ will ensure that there is sufficient space in the log buffer between successive log flushes. Ideally the value for log_buffer_wait_time should be zero. Increasing the LOGBUFSZ when log_buffer_wait_time or num_log_buffer_full have a value of zero could have adverse impact on the performance of the system.

23 Log Disk Wait times. A higher log disk wait time can limit the transaction rateand often would require a more faster dedicated I/O subsystem Log disks writes are sequential and would benefit from Faster disks available in the market with higher throughput. SSD's have higher throughput than spinning disks. Striping the log device across multiple disks will improve the write response times. Use dedicated filesystems separate from Tablespaces and other applications. Enabling write cache on the storage controller will improve log write times. Not inlining lobs that are updated would help reduce impact on the Log file system. Instead these can be cached in the Filesystem. Evaluate possibility of using 'Not logged initially' for bulk operations. Care should be taken to have a minimum amount of rollbacks. Rollbacks would require random disk access and could degrade the performance of the system.

24 Network Wait times. FCM send and receive wait times are incurred when a query is run in a DPF environment or when intra query parallelism is turned on. A small amount of FCM wait times will also be observed in purescale owing to member to member chatter Tuning FCM buffers ( fcm_num_buffers ) and FCM channels ( fcm_num_channels ) can help alleviate the problem if there are insufficient FCM buffers or channels configured. TCP/IP and IPC send and receive wait times are incurred when large amount of data needed to be send to the client. Eg. When entire tables are fetched from the database to be cached in the application layers. In the absence of such use non-selective data fetch use cases consider tuning the network stack or improving the network bandwidth between the agent and the DB2 server.

25 I/O Bottlenecks Pool Read time A higher value of pool_read_time is indicative of agents waiting for data to be fetched from the physical storage device into the bufferpool Can be broken down with MON_GET_ACTIVITY/MON_GET_PKG_CACHE_STMT into specific statements. Are you getting an unwanted tablescan? Monitor rows_read >> num_executions? Confirm plan with db2expln or db2exfmt Query is repeated, not ad hoc Are statistics out of date? Old/inaccurate statistics can trigger a tablescan Is the table sufficiently indexed? The DB2 Design Advisor may be able to help Other possibilities Is Materialized Query Table or Multi Dimensional clustering suitable?

26 I/O Bottlenecks Pool write time A higher value of pool_write_time is indicative of a high level of page cleaning activity. Consider tuning SOFTMAX to a higher value if your Recovery Time Objectives ( RTO ) is not super aggressive. Note: From version 10.5 page_age_target_mcr or page_age_target_gcr replaces SOFTMAX parameter. Analyze the iostat output to see if the disk service times are higher. And consider adding more spindles or faster disks. Tuning the NUM_IO_CLEANERS can help if the disk service times are not high.

27 I/O Bottlenecks Direct Read time and Direct write time MON_GET_ACTIVITY/mon_get_package_cache_stmt table function can be used to break this down to specific statements. LOB inlining can be used to improve the performance of data access by caching frequently accessed LOBS in bufferpools. Moving tables with LOBS to its own table-spaces and enabling filesystem caching can also improve the performance of systems bottle-necked on Direct read/write operations.

28 I/O Bottlenecks Prefetch wait times. Consider tuning the NUM_IO_SERVERS by potentially adding more prefetchers who can prefetch data into the bufferpool. Consider the possibility of tuning the PREFETCHSIZE of the tablespaces. Data is prefetched in size of PREFETCHSIZE pages. Smaller bufferpools can cause data to be replaced from the bufferpool requiring more cases when pages accessed are not in the bufferpool. This can cause prefetcher to work harder. Keep up with improvements in Prefetching in DB2! DB2 LUW v10.5 has specific improvements in the prefetching area. Smart prefetching uses either readahead or sequential prefetching depending on the clustering of data/indexes. Reduces the need to do expensive operations like reorgs.

29 Latches What are they? Latches are internal DB2 locks used to serialize access to shared data structures. Eg. Pages. MON_GET_EXTENDED_LATCH_WAIT table function can be used to get an understanding of the latch waits. A higher latch wait could be an indirect manifestation of a poorly designed application. For eg. A Page latch is obtained before making a change to the data page. If there are multiple EDU's making concurrent changes to a hot page the access is serialized and could result in latch contention. If the contented page is an index page, using RANDOM indexes and partitioned tables might help.

30 CF Wait times & Reclaim Times Check if the IB/10GE cards are saturated Consider adding more I/O channels on the CF machines if they are saturated IF we see high reclaim times, consider using Member sub-setting / partitioned workloads with EHL ( Explicit Hierarchical Locking ) CURRENT_MEMBER registry variables can be used to improve contention issues. Note : CF stands for Caching Facility in DB2 purescale

31 Activity Dimension There are two cases under which we want to look at activity dimension When there is a specific wait time in the system dimension that we need to break down into specific activities to understand the cause of the wait time. ( Eg. Breaking down the pool_read_time to understand hot tables and statements ) When we are interested in improving the performance of specific activities. ( Eg. Improve the performance of Order entry transactions in an order processing system ) The principles of breaking down the activity to specific sub activities is very similar to what we saw in the system dimension.

32 Monitoring Table functions that give Time spent breakdowns System Dimension MON_GET_SERVICE_SUBCLASS MON_GET_WORKLOAD Activity Dimension MON_GET_CONNECTION MON_GET_UNIT_OF_WORK MON_GET_PACKAGE_CACHE_STMT

33 33 Agenda Introduction Amdahl's law Time Breakdowns Examples Bottlenecks & Little law

34 Case Study We have a workload that is performing poorly and we want to use the above principles to understand and try and improve the performance. The workload we chose is TPCC benchmark and we used data from the MON_GET_WORKLOAD to understand the time breakdowns from the system dimension.

35 Example of a time-spent breakdown TOTAL_APP_RQST_TIME Throughput = Transactions per Second TOTAL_WAIT_TIME Throughput = Transactions per Second 30% 39% TOTAL_WAIT_TIME OTHERS POOL_READ_TIME POOL_WRITE_TIME OTHERS 61% 0% 70% Workload is TPCC

36 Example of a time-spent breakdown TOTAL_APP_RQST_TIME T hroughput = T ransact ions per Second TOTAL_WAIT_TIME T hroughput = T ransact ions per Second 43% TOTAL_WAIT_TIME OTHERS 32% POOL_READ_TIME POOL_WRITE_TIME OTHERS 57% 68% 0% Culprit! Number of Bufferpool Pages Old Run = New Run =

37 Percentages dont tell the whole story! TOTAL_WAIT_TIME T hroughput = T ransact ions per Second TOTAL_WAIT_TIME T hroughput = T ransact ions per Second % POOL_READ_TIME POOL_WRITE_TIME OTHERS % POOL_READ_TIME POOL_WRITE_TIME OTHERS % % % % Applying Amdahl's Law Improvement = 1/ [ ( 1-P) + P/S ] P ( %age TOTAL_WAIT_TIME) = 0.61 S ( Speedup ) = / = 2.13 Improvement = 47% which matches the TPS improvement we got.

38 38 Agenda Introduction Amdahl's law Time Breakdowns Examples Bottlenecks & Little law

39 39 Bottlenecks What are they?

40 40 Little's Law Littles's law : At steady state, the average number of outstanding requests to a system is equal to the effective arrival rate multiplied by the average service time. Note : At steady state, the arrival rate is equal to the throughput of the system.

41 41 Where is the bottleneck? Webserver L = λ * S L1 = λ1 1 * S1 L2 = λ2 2 * S2 DB2 Total Queue Length ( L ) = Queue Length at Webserver ( L1 ) +Queue Length at DB2 ( L2 ) Bottleneck is the component with the slowest service time.

42 How to compute Little's Law from DB2 Monitoring Data 42 Average Service Time ( S ) = TOTAL_RQST_TIME ( System Dimension ) / ( TOTAL_APP_COMMITS + ROLLBACK_SQL_STMTS) ( in milliseconds ) Arrival Rate ( λ )= ( TOTAL_APP_COMMITS + ROLLBACK_SQL_STMTS) / Monitoring Interval From Little's Law we have L = λ * S Queue Length = TOTAL_RQST_TIME / Monitoring interval This is the same as the number of applications in 'UOW Executing' state from db2pd -applications at any given time.

43 How to compute Little's Law from DB2 Monitoring Data 43 If the Queue Length at DB2 is far less than the Queue lengths elsewhere in the system, DB2 is not the bottleneck. DB2 itself can be decomposed into multiple queuing systems where at each category of wait_time we can imagine a queue. Extending the above formula for queue length we can derive the queue length of specific wait events ( I/O, network, prefetch etc ) by taking the ratio of specific wait times to the monitoring interval. Queue Length ( L ) = TOTAL_RQST_TIME / Monitoring interval Monitoring interval = Σ ( agent_wait_time + lock_wait_time + ) / From the above formula is easy to see that the length of the queue is proportional to the wait times in specific components.

44 44 Questions?

45 Roy Cecil IBM Session 3 Time Breakdowns A novel way to analyze performance Problems Please fill out your session evaluation before leaving!