Performance Monitoring

Performance Monitoring Goals Monitoring should check that the performanceinfluencing database parameters are correctly set and if they are not, it should point to where the problems are Monitoring (preventive method) OR Alternatively, one can sit and wait for the usersʼ complaints Correct problems when detected (reactive method)

How does one monitor a DBMS? By extracting relevant performance indicators, (counters, metrics, and details of internal DBMSʼs activities) By comparing the obtained values of these indicators against ideal values But there are many indicators!

Recurrent Patterns of Problems An overloading high-level consumer Ex: query accessing too many rows A poorly parameterized subsystem Ex: disk subsystem that stores tables, indexes, logging in a single disk An overloaded primary resource Ex: CPU busy because non-db processes are using it at the same time

A Consumer-Producer Chain of a DBMS s Resources sql commands High Level Consumers PARSER OPTIMIZER DISK SYBSYSTEM EXECUTION SUBSYSTEM LOCKING SUBSYSTEM Intermediate Resources/ Consumers probing spots for indicators CACHE MANAGER LOGGING SUBSYSTEM MEMORY CPU DISK/ CONTROLLER NETWORK Primary Resources

A Systematic Approach to Monitoring Extract indicators to answer the following questions Question 1: Are critical queries being served in the most efficient manner? Question 2: Are DBMS subsystems making optimal use of resources? Question 3: Are there enough primary resources available and are they configured adequately, given the current and expected workload?

Investigating High Level Consumers: Critical query monitoring Find critical queries Investigate intermed. level no Found any? yes Check Quest. 1 no Overconsumption? yes Tune problematic queries

Investigating Intermediate and Primary Resources: routine monitoring Check Quest. 3 Check Quest. 2 no Problems at OS level? yes Tune low-level resources Tune subsystems yes Problematic subsystems? no Investigate upper level

Tools used to gather information Event monitors Query plan explainers Performance monitors

Investigating High Level Consumers Answer question 1: Are critical queries being served in the most efficient manner? 1. Identify the critical queries 2. Analyze their access plans 3. Profile their execution

Identifying Critical Queries Critical queries are usually those that: Take a long time Consume a great amount of resources (read a lot of data, perform heavy aggregation or sorting, lock an entire table, etc) Are frequently executed Have to be answered in a short time Often, a user complaint give us a hint.

Event Monitors to Identify Critical Queries If no user complains... Record DBMSʼs performance measurements, but only when a system event happens Ex: new user connection, or beginning/end of the execution of an SQL statement Logs the event time, duration and associated performance indicator values Typical measures include CPU used, IO used, locks obtained etc.

What should be measured and evaluated Interested in the end-of-statement or end-oftransaction event Carry relevant info that indicate the resources the executed query consumed: SQL of the statement, duration of execution, CPU, IO, cache,lock consumption statistics Sort the collected data over a given resource consumption statistic and we can find the most expensive queries by that criterion

An example Event Monitor Several CPU indicators sorted by Oracle s Trace Data Viewer Similar tools: DB2 s Event Monitor and MSSQL s Server Profiler

Diagnose Expensive Queries: analyzing access plans PARSER/OPTIMIZER sql parse rewrite enumerate/ cost plans generate chosen plan plan SQL commands are translated into an internal executable format before they are processed After parsing, the optimizer enumerates and estimates the cost of a number of possible ways to execute that query The best choice, according to the existing statistics, is chosen But this choice may not be the best one!

Query Plan Explainers to Analyze Plans Explainers usually depict an access plan as a (graphical) single-rooted tree in which sources at the leaf end are tables or indexes, and internal nodes are operators These form an assembly line (actually tree) of tuples! Most explainers will let you see the estimated costs for CPU consumption, I/O, and cardinality of each operator. If you see something strange, change an index.

An example Plan Explainer Access plan according to MSSQL s Query Analyzer Similar tools: DB2 s Visual Explain and Oracle s SQL Analyze Tool

Finding strange in Access Plans What to pay attention to on a plan Access paths for each table For every table involved in the query, determine how it is accessed: scan, index? Sorts or intermediary results Check if the plan includes a sorting (because of DISTINCT, ORDER BY, GROUP BY) and see if it can be eliminated Materialization of temporary results should be avoided Order of operations May want to force an ordering if we notice the order does not favor the early reduction in the number of tuples Algorithms used in the operators

Profiling a Query s Execution A query was found critical but its plan looks okay. Whatʼs going on? Profiling it would determine the quantity of the resources used by a query and assessing how efficient this use was Profile of a query execution consists of detailed information about the duration and resource consumption Resources DBMS subsystems: cache, IO, locks, log OS raw resources: CPU

Query profiling indicators Duration information involves 3 indicators: Elapsed time for the query: time it took to process it as perceived by the user CPU time: time the CPU was actually used to process the query Wait time: time the query was not processing and waiting for a resource to become available Resource consumption information includes: (I/O) Physical and logical reads/writes (Locking) Maximum nbr locks held, nbr lock escalations, nbr deadlocks/timeouts, total time spent waiting for locks (SQL activity) Nb sorts and temporary area usage: gives an idea of the time spent in expensive overhead activity

Two common scenarios 1. Elapsed query time close to CPU time, wait time negligible. Consumption fair: reasonable nb pages being accesses, most of them are logical accesses; nb locks low or inexistent and no deadlocks or lock escalations; if there were sorts, they didnʼt augment the nb of logical/physical reads/writes Access plan is executed in the best way possible; if the rest of the chain is balanced, this is the best performance the system can deliver for this query 2. Noticeable discrepancy between elapsed and CPU time, wait time fills the gap. So there must be a problem in resource consumption: contention (probably waiting for locks) or poorly performing resource (ex. If area allocated for sorting is small, additional I/O). To distinguish between contention and physical resource consumption problem, run the query in isolation

Performance Monitors for Profiling Queries Access or compute performance indicatorsʼ values at any time Many flavors: Scope of the indicators: Generic (all indicators from OS and DBMS) or specific (correlated indicators of a given subsystem or a given query) Choose the generic one when interested in how several distinct indicators behave together Frequency of data gathering: snapshot perf indicators: allow to freeze a situation at a specific moment in time Continuous perf. Indicator: snaphotting at regular intervals Alarm modes: allows us to enter thresholds beyond which the system is acting abnormally Presentation of data: textual or graphical Textual good for seeing the precise values of indicators Graphical chosen when wants to see the trends

An example Performance Monitor (query level) Statement number: 1 select C_NAME, N_NAME from DBA.CUSTOMER join DBA.NATION on C_NATIONKEY = N_NATIONKEY where C_ACCTBAL > 0 Details of buffer and CPU consumption on a query s report according to DB2 s Benchmark tool Similar tools: MSSQL s SET STATISTICS switch and Oracle s SQL Analyze Tool Number of rows retrieved is: 136308 Number of rows sent to output is: 0 Elapsed Time is: 76.349 seconds Buffer pool data logical reads = 272618 Buffer pool data physical reads = 131425 Buffer pool data writes = 0 Buffer pool index logical reads = 273173 Buffer pool index physical reads = 552 Buffer pool index writes = 0 Total buffer pool read time (ms) = 71352 Total buffer pool write time (ms) = 0 Summary of Results ================== Elapsed Agent CPU Rows Rows Statement # Time (s) Time (s) Fetched Printed 1 76.349 6.670 136308 0

Investigating Intermediate and Primary Resources: routine monitoring Answer Q3 Answer Q2 no Problems at OS level? yes Tune low-level resources Tune subsystems yes Problematic subsystems? no Investigate upper level

Investigating Primary Resources Answer question 3: Are there enough primary resources available for a DBMS to consume? Primary resources: CPU, disk/controllers, memory, and network Analyze specific OS-level indicators to discover bottlenecks. A system-level performance monitor is the right tool here

CPU Consumption Indicators at the OS Level CPU % of utilization 100% 70% total usage system usage time Sustained utilization over 70% should trigger the alert. System utilization shouldn t be more than 40%. DBMS (in a nondedicated machine) should be getting a decent time share.

Disk Performance Indicators at the OS Level Average Queue Size Should be close to zero Disk Transfers /second Idle disk with pending requests? Check controller contention. Also, transfers should be balanced among disks/controllers New requests Wait queue Wait times should also be close to zero

Memory Consumption Indicators at the OS Level Page faults/time should be close to zero. If paging happens, at least not DB cache pages. real memory virtual memory % of pagefile in use (it s used a fixed file/partition) will tell you how much memory is lacking. pagefile

Investigating Intermediate Resources/Consumers Answer question 2: Are subsystems making optimal use of resources? Main subsystems: Cache Manager, Disk subsystem, Lock subsystem, and Log/Recovery subsystem Extract and analyze relevant perf. indicators A Performance Monitor is usually useful, but sometimes specific tools apply

Cache Manager Performance Indicators Cache Manager Pick victim strategy Page reads/ writes Table scan readpage() Data Pages Free Page slots If page is not in the cache, readpage (logical) generate an actual IO (physical). Ratio of readpages that did not generate physical IO (cache hit ration) should be 90% or more Pages are regularly saved to disk to make free space. # of free slots should always be > 0

Disk Manager Performance Indicators rows Row displacement: should kept under 5% of rows Storage Hierarchy (simplified) page extent file Free space fragmentation: pages with few space should not be in the free list Data fragmentation: ideally files that store DB objects (table, index) should be in one or few contiguous extents (<5) disk File position: should balance workload evenly among all disks

Lock Manager Perf. Indicators Lock List Lock request Object Lock Type TXN ID Locks pending list Lock wait time for a transaction should be a small fraction of the whole transaction time. Number of locks on wait should be a small fraction of the number of locks on the lock list, under 20% Deadlocks and timeouts should be virtually inexistent or extremely low (no more than 1% of the transactions)

Logging Subsystem Performance Indicators Log is used by every transaction that alters, inserts or deletes data Well tuned logging subsyst. Should write log records at a pace that doesnʼt slow active transactions Indicatores to measure: Number of log waits to confirm that the disks storing the log files are keeping pace with the transaction workload should be zero Nb of log expansions or log archives due to lack of space if bigger than zero indicates a configuration problem Log cache-hit ratio to check whether the size of the log buffer is adequate

Conclusion Monitoring a DBMSʼs performance can be done in a systematic way The consumption chain helps distinguishing problems causes from their symptoms Existing tools help extracting relevant performance indicators The three questions guide the whole monitoring process