Performance Problems

Transcription

1 Tools and Techniques to Address Performance Problems Biju

2 Biju Thomas Principal Solutions Architect with OneNeck IT Solutions Over 20 years of Oracle Database development and administration expertise Over 10 years of Oracle E-Business Suite Architecture & Tuning expertise First book published in September 2000, seventh in 2015 DBA blog since Daily Oracle Tidbits #oratidbit Oracle ACE Director 3

3 OneNeck IT Solutions at a Glance Hybrid & Multi-Cloud Solutions Managed Services, Advisory Services Coast to Coast Data Centers 4

4 Performance Troubleshooting Cycle Typical Reactive Performance Management User opens ticket performance issue Check Database Single user/program issue or database wide issue What changed Collect information How to reproduce the issue Review information Remediation Root Cause Analysis Prevention or Alert 5

5 Why SQL statements perform slow? Optimizer Inefficient plan Application Coding Literal usage Resource CPU, Memory, I/O Contention Locks 6

6 Which diagnostics tool? SQL Scripts Oracle Enterprise Manager Cloud Control 13c Database Express 12c SQL Developer v18.1 Third party products Enterprise Edition Options Diagnostic & Tuning Pack Automatic Workload Repository (AWR) Active Sessions History (ASH) Automatic Database Diagnostics Monitor (ADDM) Real-time SQL and Database Monitoring SQL Tuning Advisor 7

7 Database Quick Check OEM Performance Home 8

8 Database Quick Check: OEM Top Activity 9

9 SQL Developer: Instance Viewer 10

10 Database Wide or Single Program? Most performance issues are single program or single session Database wide issue Resource Contention? Disk CPU Memory Oracle tools OEM OS Watcher ( ) Oratop ( ) ADDM & AWR Performance Hub Single program or session Oracle tools OEM SQL Monitoring ADDM & AWR Trace Session Solution Fix Code Better Statistics SQL Profile SQL Plan Baseline Bug? Questions New program? Code change? Parameters valid? Problem start time? Statistics current? New statistics? Wait event? 11

11 Performance Hub OEM Database Express / SQL*Plus OEM Database Express or SQL*Plus The Performance Hub shows all the performance data available for a specified time period. Single view for ADDM, SQL Tuning, Real-time monitoring, ASH Real-time and historical $ORACLE_HOME/rdbms/admin/perfhubrpt.sql DBMS_PERF Package REPORT_PERFHUB ( ) function entire database REPORT_SESSION ( ) function specific database session REPORT_SQL ( ) function specific SQL Summary Activity Workload RAC Monitored SQL ADDM Current ADDM Findings Database Time Resources System Statistics Basic Typical All 12

12 Performance Hub Report 13

13 Automatic Workload Repository (AWR) STATISTICS_LEVEL Retention Default 8 days Snapshot Frequency Ways to generate AWR reports Types of AWR data Baselines Comparison reports Using *\_HIST\_* views =TYPICAL (BASIC turns AWR off. ALL is problem) Change to 30 or 60 or more days for meaningful history. Default 60. OK for normal systems OEM, SQL Developer, SQL Non-CDB, CDB, PDB, Standby, RAC Periodic baselines help to compare performance metrics Compare with baselines or other snapshots Write powerful scripts to analyze performance 14

14 STATISTICS_LEVEL Default is TYPICAL SELECT statistics_name,activation_level FROM v$statistics_level ORDER BY activation_level; ALL: only at session level, to debug specific issues. Huge WRH$_LATCH_CHILDREN 15

15 Generate AWR Reports Use SQL*Plus, OEM or SQL Developer Objective is to reduce DB Time (CPU, IO & Non-idle waits) AWR Reports Standard: awrrpt.sql / awrrtpi.sql Compare (difference): awrddrpt.sql / awrddrpi.sql Global (RAC): awrgrpt.sql / awrgrpti.sql Global difference: awrgrdpt.sql Single SQL Statement: awrsqrpt.sql / awrsqrpi.sql General info: awrinfo.sql ADDM Report addmrpt.sql ASH Report ashrpt.sql 16

16 What to look for in AWR When to check ADDM report Top-down approach unless looking for specific issues Duration of AWR report Number of sessions at the beginning and end Load profile & Instance efficiency Timed foreground events Instance CPU SQL Sections Tablespace I/O SGA & PGA RAC related waits Always check corresponding ADDM report for actionable recommendations. ADDM report also shows what is not a problem. 17

17 AWR Report Quick Review 18

18 ADDM Report Quick Review 19

19 Real-Time ADDM 20

20 Finding the Problem Session & Offending SQL OEM Top Activity SQL Details Search Sessions SQL Developer Session Monitor SQL*Plus V$ & GV$ views 21

21 Active Session History (ASH) OEM / SQL Developer / SQL*Plus / ASH ASH information for a specified Instance # or ALL for all RAC instances Filter by Session ID, SQL ID, Wait Class, Service Hash, Module, Action, Client ID 22

22 OEM ASH Analytics & ASH Report 23

23 Real-time SQL Monitoring (SQLDev) 25

24 (Real-Time) SQL Monitoring (OEM) 27

25 OEM SQL Details Statistics / Activity / Plan / Plan Control / Tuning History / SQL Monitoring 28

26 Tracing Sessions OEM / SQLDev / SQL / DBMS_MONITOR DBMS_MONITOR.SESSION_TRACE_ENABLE( session_id IN BINARY_INTEGER DEFAULT NULL, serial_num IN BINARY_INTEGER DEFAULT NULL, waits IN BOOLEAN DEFAULT TRUE, binds IN BOOLEAN DEFAULT FALSE, plan_stat IN VARCHAR2 DEFAULT NULL); If serial_num is NULL but session_id is specified, a session with a given session_id is traced irrespective of its serial number. If both session_id and serial_num are NULL, the current user session is traced. 29

27 Reading Trace File TKPROF or Use Trace Analyzer TKPROF on Steroids! (MOS ) Trace File Name Name of Control File HTML Report Control File HTML Report Event Trace Trace Analyzer Text Report Event Event Event Event Trace Event Trace Trace Trace Trace Trace Analyzer Text Report Log Files Log Files 30

28 OEM SQL Details Use Search SQL from Cursor Cache and AWR Reports 31

29 SQL Tuning Advisor Automatic Tuning Optimizer is the central brain used by SQL Tuning Advisor. 32

30 SQL Tuning Set (STS) Create a tuning task DBMS_SQLTUNE.CREATE_TUNING_TASK (task_name=>'&&task_name', sql_id=>'&sql_id ) Execute and report DBMS_SQLTUNE.EXECUTE_TUNING_TASK DBMS_SQLTUNE.REPORT_TUNING_TASK Accept Recommendations DBMS_SQLTUNE.ACCEPT_TUNING_TASK Accept SQL Profile DBMS_SQLTUNE.ACCEPT_SQL_PROFILE 33

31 Kerry s Scripts Find the SQL ID for the Use dbms_xplan.display_cursor to see the Statements that have experienced significant variances in execution Shows plan_hash_value changes and execution Execution stats 34

32 Directing the Optimizer Proper Statistics Histograms Multi-column stats SQL Profiles SQL Plan Baselines 35

33 SQL Plan Baselines Plan Capture Creation of SQL plan baselines that store accepted execution plans for all relevant SQL statements. Plan Selection Ensures only accepted execution plans are used for statements with a SQL plan baseline and records any new execution plans found for a statement as unaccepted plans. Plan Evolution Evaluate all unaccepted execution plans for a given statement, with only plans that show a performance improvement becoming accepted plans. 36

34 Plan Capture Automatic Capture OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES Manual Capture From SQL Tuning Set Use DBMS_SPM.LOAD_PLANS_FROM_SQLSET Plans loaded are automatically accepted. From Cursor Cache Use DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE Filter SQL by text, module, parsing schema, SQLID From AWR Repository (12cR2) Use DBMS_SPM.LOAD_PLANS_FROM_AWR Specify range of snapshots DBMS_SPM.PACK_STGTAB_BASELINE Export / Import Staging Table DBMS_SPM.UNPACK_STGTAB_BASELINE 37

35 Plan Selection OPTIMIZER_USE_SQL_PLAN_BASELINES (default TRUE) 38

36 SQL Plan Baselines Things to know Query DBA_SQL_PLAN_BASELINES An enabled plan is a plan that is eligible for use by the optimizer. An accepted plan is a plan that is in a SQL plan baseline for a SQL statement and thus available for use by the optimizer. A fixed plan is an accepted plan that is marked as preferred, so that the optimizer considers only the fixed plans in the baseline. Unused plans are purged after 53 weeks by default. DBMS_SPM.CONFIGURE('plan_retention_weeks',n); Explain Plan on SQL shows if a baseline is used. 39

37 Copying Optimizer Plans Using OEM SQL Profiles & SQL Plan Baselines 40

38 Copying Optimizer Plans SQL Profiles Using Oracle COE Scripts SQLT (MOS ) Download SQLT (sqlt_latest.zip) Extract files Scripts under sqlt/utl directory Run coe_xfr_sql_profile.sql with SQL_ID as parameter Shows each plan hash value and average elapsed_time/executions Creates one file for each plan hash value of SQL_ID found in cursor cache and AWR. Scripts: coe_xfr_sql_profile_<sql_id>._<phv>.sql Run the required profile create script in desired database. 41

39 Copying Optimizer Plans SQL Plan Baselines Using Oracle COE Scripts SQLT (MOS ) Scripts to copy SQL Plan Baselines under sqlt/utl/spm directory. Option 1: Create SQL Plan Baseline (SPB) in Source From a Cursor; or From AWR Package & Export SPB from Source Import & Restore SPB into Target Option 2: Create SQL Tuning Set (STS) in Source From a Cursor; or From AWR Package & Export STS from Source Import & Restore STS into Target Create SPB from STS in Target 42

40 The Proactive Part

41 Proactive Actions Saves you from the fire drill! Gather Optimizer Statistics Dictionary Stats Fixed Object Stats Database/Schema Stats Capture Periodic AWR Baselines Periodic review: Database Health Resource utilization Top SQL Optimizer plan stability Alert/terminate blocking sessions (frequency?) 44

42 Optimizer Statistics Advisor Database 12c Feature The Optimizer Statistics Advisor runs a task called AUTO_STATS_ADVISOR_TASK during the maintenance window Can be personalized on what rules to check (v$stats_advisor_rules) DBMS_STATS.CONFIGURE_ADVISOR_OBJ_FILTER DBMS_STATS.CONFIGURE_ADVISOR_RULE_FILTER Report recommendations dbms_stats.report_advisor_task Remediation script DBMS_STATS.SCRIPT_ADVISOR_TASK Manually execute the advisor DBMS_STATS.CREATE_ADVISOR_TASK DBMS_STATS.EXECUTE_ADVISOR_TASK Implement all recommendations DBMS_STATS.IMPLEMENT_ADVISOR_TASK 46

43 Optimizer Statistics Advisor Tasks 47

44 Database and System Health checks Develop a strategy If nobody complains, define scope by yourself. Baseline metrics & configuration Performance thresholds Report/review frequency Tools ORAchk DBSAT OSWatcher Health Monitor (DBMS_HM) Third party tools (SolarWinds, Foglight, APM, etc) 51

45 Familiarize with the environment Review OEM Performance Home, Top Activity and AWR reports periodically, and know the typical performance profile. When problem happens you can easily point out what stands out. Review reports from normal periods and acquaint with the norm. Understand the load on the system and busy times, especially large batch jobs. And, read Oracle Database Performance Tuning Guide 52

46 RMOUG TD18 Sessions Thursday, February 22 (tomorrow) Oracle Database 12c Features for Developers! 1:30pm - Standley 2 Essential (free) Tools for DBA 2:45pm - Cotton Creek 2 53

47 Essential (free) tools for EBS Administrator

48 Thank you! Daily #oratidbit on Facebook and Twitter. Follow me! Facebook: facebook.com/oraclenotes Google+: +bijoosoraclenotes Blog: bijoos.com/oraclenotes

49 Please Complete Session Evaluation