OpenWorld 2018 SQL Tuning Tips for Cloud Administrators GP (Prabhaker Gongloor) Senior Director of Product Management Bjorn Bolltoft Dr. Khaled Yagoub Systems and DB Manageability Development Oracle Corporation Copyright 2018, Oracle and/or its affiliates. All rights reserved.
Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle s products may change and remains at the sole discretion of Oracle Corporation. Copyright 2018, Oracle and/or its affiliates. All rights reserved.
Program Agenda 1 2 3 4 SQL Tuning for Administrators: Evolving Landscape Proactive Approach and Tips Reactive Approach and Tips Conclusion Copyright 2018, Oracle and/or its affiliates. All rights reserved. 4
Program Agenda 1 2 3 4 SQL Tuning for Administrators: Evolving Landscape Proactive Approach and Tips Reactive Approach and Tips Conclusion Copyright 2018, Oracle and/or its affiliates. All rights reserved. 5
Traditional DB Admin Responsibility Maintenance Tasks Configuration and tuning of systems, network, storage Database provisioning, patching Database backups, H/A, disaster recovery Maintain indexes, baselines, etc. Tasks Specific to Business and Innovation Application related tuning Cross-tier application diagnostics Data movement Data lifecycle management Copyright 2018, Oracle and/or its affiliates. All rights reserved. 6
Autonomous DB Admin Responsibility Maintenance Tasks Configuration and tuning of systems, network, storage Database provisioning, patching Database backups, H/A, disaster recovery Maintain indexes, baselines, etc. Autonomous Database Tasks Specific to Business and Innovation Application related tuning Cross-tier application diagnostics Data movement Data lifecycle management Copyright 2018, Oracle and/or its affiliates. All rights reserved. 7
SQL Tuning for Administrators Oracle Enterprise Manager Cloud Control (EMCC) On-premise product; installed, managed and maintained by customer Provides administration, performance management, and lifecycle management capabilities for databases running on-premises, on the cloud, or Exadata platform Will not support Autonomous Databases Oracle Management Cloud (OMC) Cloud service; fully managed and maintained by Oracle Provides monitoring, log search and analysis, performance analytics, and advanced security analytics for on-premise and cloud databases Complements EMCC capabilities Will support management of Autonomous Databases Autonomous Databases are self-tuning, however DBAs still need to diagnose application related problems in the data tier and perform cross-tier application diagnostics Poorly constructed application SQL from applications Application inefficiencies (row locks, block contention, literal SQL, etc.) Performance and SQL tuning using OMC and EMCC uses same DB Time and Time/Wait model methodology Same software running on-premise and Cloud Copyright 2018, Oracle and/or its affiliates. All rights reserved. 8
Program Agenda 1 2 3 4 SQL Tuning for Administrators: Evolving Landscape Proactive Approach and Tips Reactive Approach and Tips Conclusion 9
Proactive Approach Become the Proactive DBA using Oracle s rich tool set! A DBA has Oracle s rich tool set available to identify problems and remediate problems before they happen in production. Proactive DBA should use the following tools: ADDM to detect systemic problems and remediate them by following ADDM recommendations* Automatic SQL Tuning advisor to tune high load SQL statements over time it is available out of the box* SQL Access Advisor to make sure your application has optimal access structures (indexes and type, MVs, MV logs, etc.) and partitioning scheme, tune entire workload* SPA Quick Check in production environments to quickly assess impact of routine system changes (for e.g., SQL Profile validation, optimizer related changes, optimizer refresh statistics) without impacting production* SPA to validate system changes such as migration to Oracle Database Cloud, Exadata Cloud Machine, etc.* *OOW 2016: Eliminating Guesswork from SQL Tuning [CON6975] 10
SQL Performance Analyzer Helps users predict the impact of system changes on SQL workload Low overhead capture of SQL workload to SQL Tuning Set (STS) on production system Build different SQL trials (experiments) of SQL statements performance by test execution or explain plan Integrated with STS, SQL Plan Baselines, & SQL Tuning Advisor to form an end-to-end solution SQL Plans + Statistics Pre-change Trial Compare SQL Performance SQL Plans + Statistics Post-change Trial Analysis Report 11
DBCS Migration Use Case As a DBA, you have been tasked by management to migrate your 11.2 database to the latest Cloud DB release. At the same time, the requirement is to make sure the performance is same or better than before, how can I accomplish this? 12
Solution: How to Validate Cloud Migration with SPA? Step 1: Capture representative workload into SQL Tuning Set (STS) on Production (On-premise) Step 2: Clone Database to Cloud using Oracle supported methods For PDB use one-click migration Non-PDB use Transportable Tablespaces or Datapump features Step 3: SPA Validation in Cloud Can use EM13 Cloud Control, EM Express or API Trial 1: Build from STS (Convert from STS) Trial 2: Run against Cloud PDB (Test execute or explain plan) Generate Reports to validate plan changes and performance differences Use various metrics such as Buffer Gets, CPU time and Elapsed time to assess performance and fix any issues Note: None of the other vendors have capability test on-premise and Cloud seamlessly Copyright 2018, Oracle and/or its affiliates. All rights reserved. 13
Validate Cloud Migration with SPA On-Premise Oracle Cloud Step3a: Conduct SPA trials SPA Task Trial 1: Build (Convert) from STS Trial 2: Test Execute or Explain Plan Step 1: Capture representative workload to STS Step 2: Clone On-premise database to Cloud Step 3b: Generate SPA Report and fix regressions Analysis Report 14
15
16
Concurrent SPA Trials New in 18.1 SPA trials are currently executed serially one statement at a time For large SQL workloads this can result in Extended testing duration. For e.g., EBS suite has 1 million unique SQL statements Under-utilized test server resources if large system like Exadata SQL Plans + Statistics 17
Concurrent SPA Trials Concurrent SQL execution within a SPA trial Reduces duration to complete the trial significantly, therefore overall testing time User specifies degree of parallelism with SPA parameter TEST_EXECUTE_DOP Granted DOP is based on CPU availability Honors PDB-level instance caging limits If critical errors, SPA trial will seamlessly continue to run albeit at a reduced DOP without missing any of the SQLs SQL Plans + Statistics Pre-change Trial Compare SQL Performance Analysis Report New in 18.1 SQL Plans + Statistics Post-change Trial 18
Concurrent SPA Trials New in 18.1 Linearity based on CPU, Memory and I/O resources Consider that parallel Query will spawn additional parallel processes Tip: Use SPA concurrent SQL execution in test environment for STS with a large number (thousands or higher) of SQL statements 20299 statements executed in parallel 124665 Statements executed in parallel 45 800 40 700 35 30 25 20 15 10 Time in minutes 600 500 400 300 200 Time in minutes 5 100 0 1 8 16 0 1 8 16 19
SPA Results Validation New in 18.1 SELECT client, ROUND(sum(VALUE_X)/sum(VALUE_Y),15) Result from client_experiment where experiment_date = 18-SEP-17 group by client; Before Change AfterChange CLIENT RESULT ------- ----------------------- A 1.736594539257736 B 22.768452036538209 C 12.978975936783168!= CLIENT RESULT ------- ----------------------- A 1.736594539257736 B 22.768452036538208 C 12.978975936783168 20
SPA Results Validation Required for certain environments, for e.g., government drug research trials, pharma industries, etc. Besides performance, an additional assurance that SQL produces correct results with new environment / software SPA result set validation enabled with SPA parameter COMPARE_RESULTSET (defaults to true) Produces a result set hash value for each query SPA report can be generated for before and after change trials comparing whether identical result sets were produced Enabled only for SPA trials that are execute locally or remote Tip: Run before and after trials with exact same dataset for result set validation Tip: If result sets differ with same input dataset contact Oracle Support 21
SPA Results Validation New in 18.1 22
Program Agenda 1 2 3 4 SQL Tuning for Administrators: Evolving Landscape Proactive Approach and Tips Reactive Approach and Tips Conclusion 23
SQL Tuning Tips - Reactive Approach Proactive SQL Tuning should resolve most of your day-to-day issues, however sometimes reactive tuning is needed DBA should use the following tools for reactive SQL Tuning: ASH Analytics for transient performance analysis and drilling down on SQL dimensions (by execution plan operation, SQL_ID, PDB, Action, Module, etc.) SQL Monitor for complex run-time execution statistics on long running or Parallel executions such as which plan operation line that consumes resources such as CPU and I/O, overall PGA usage and Parallel distribution SQL Tuning Advisor getting comprehensive analysis and recommendations on problematic SQL statements such as profiles, statistics and new access structure 24
Tip: Use Real-Time SQL Monitoring For Detailed Execution Statistics On Long Running Or Parallel Executions Looking inside the SQL Enabled out-of-the-box with no performance impact Automatically monitors SQL executions that: Consume more than 5 seconds of CPU or I/O time Are running parallel: PQ, PDML, PDDL Monitors each execution independently Exposes monitoring statistics at multiple levels Global execution level Plan operation level (Plan Tuning) Parallel Execution level (PX Tuning) Guides your tuning efforts SQL level metrics CPU, I/O requests, throughput, PGA, temp space Graphical explain plan I/O statistics for each operation 26
Real-Time SQL Monitoring Use cases Case Study 1 Adaptive Plans Case Study 2 Long Execution plan Case Study 3 PL/SQL Monitoring Case Study 4 Poor indexing Case Study 5 Parallel Execution Downgrade 30
SQL Monitoring Use Cases Case Study 4: Poor Indexing Q: I have added many indexes to my database. I still have several queries with bad performance. How do I determine effectiveness of my tuning? Solution: 1. Use SQL Monitoring and looking at the plan operations statistics we can easily figure out the issues
SQL Monitoring Use Cases Case Study 4: Poor Indexing Q: I have added many indexes to my database. I still have several queries with bad performance. Can I determine if my indexes are used efficiently? Initial Query 49
SQL Monitoring Use Cases Case Study 4: Poor Indexing Q:I have added a large amount of indexes to my database. I still have several queries with bad performance. Can I identify that my indexes are used efficient? Initial Query Tuned Query 50
SQL Monitoring Use Cases Case Study 4: Poor Indexing Q: I have added many indexes to my database. I still have several queries with bad performance. Can I determine if my indexes are used efficiently? Initial Query Tuned Query Tuned Query / Cached 51
SQL Tuning Advisor Exadata Enhancement NEW IN 18.1 SQL Tuning Advisor detects if SQL is executing on Exadata SQL Tuning Advisor privately gathers system statistics and does analysis with and without these statistics If a better execution plan is found with these system statistics, an Exadata-aware SQL Profile is recommended Can result in 10x or better performance improvement for SQLs which can benefit from Exadata hardware - e.g. cell smart scans 56
SQL Tuning Advisor Exadata Enhancement Without Exadata aware SQL Profile 1.9 minutes: NEW IN 18.1 With-Exadata aware SQL Profile 13 seconds: 57
SQL Profiles Exadata aware profiles 1- SQL Profile Finding (see explain plans section below) -------------------------------------------------------- 2 potentially better execution plans were found for this statement. Choose one of the following SQL profiles to implement. Recommendation (estimated benefit: 78.69%) ------------------------------------------ Consider accepting the recommended SQL profile. It is an Exadata-aware SQL profile. execute dbms_sqltune.accept_sql_profile(task_name => 'task_sqltext_1_5_single_off2_d', task_owner => 'MSTR', replace => TRUE); New in 18.1 Validation results ------------------ The SQL profile was tested by executing both its plan and the original plan and measuring their respective execution statistics. A plan may have been only partially executed if the other could be run to completion in less time. Original Plan With SQL Profile % Improved ------------- ---------------- ---------- Completion Status: PARTIAL COMPLETE Elapsed Time (s): 60.24843 13.498967 77.59 % CPU Time (s): 5.802411 2.031372 64.99 % User I/O Time (s): 56.799954 12.364276 78.23 % Buffer Gets: 180162 500025-177.54 % Physical Read Requests: 70518 40354 42.77 % Physical Write Requests: 0 0 Physical Read Bytes: 2051678208 3768688640-83.68 % Physical Write Bytes: 0 0 Rows Processed: 0 1 Fetches: 0 1 Executions: 0 1 58
SQL Profiles Exadata aware profiles No profile Exadata aware profile New in 18.1 59
Tips for Multitenant and Developers Use local AWR AWR data for a PDB (Top N SQL per PDB vs at CDB level) AWR statistics, Time-Wait model, sysmetrics statistics per PDB AWR transportable along with PDB Tip: Enable AWR snapshots at PDB-level as necessary SQL Monitoring for Developers SQL Monitor reports were only accessible to privileged users, for e.g., DBA or users having select_catalog_role For autonomous and for DBCS/ExaCS, a PDB-level developer user may not have these privileges Supports both SQL and PLSQL monitored reports Developer users can only see reports of SQL they have executed and that have been monitored Behavior of SQL Monitor feature and report is unchanged for privileged users like DBAs New in 12.2 New in 19.1 Developer users may have access to views, but not the underlying tables. In SQL monitor report for such queries, the table names are not exposed 60
New Features New in 19.1 Automatic SQL Tuning for Autonomous DB Runs ADDM for hourly findings Self-identifies SQL that needs tuning, and can optionally auto-tune it without requiring user intervention under strict time budget Goal is to prevent regressions by automatically creating and enabling SQL plan baselines for the regressed SQL SQL Profiles and Index recommendations are not recommended/implemented 61
Tip: SQL Profiles or SQL Plan Baselines? SQL Profiles Contains auxiliary information that mitigates the defects in a sub-optimal plan Profiles provide additional information to the optimizer to help select the best plan Doesn't constrain the optimizer to any specific plan, which is why they can be shared Are more like extended statistics, better information for the optimizer to decide about how many rows will flow out of a step in the plan Useful when you want the system to adapt immediately to changes like new object statistics SQL Plan Baselines Consists of a set of accepted plans that are existing plans known to be efficient Aids the optimizer to select the best plan from among the accepted plans Constrain the optimizer to only select from a set of accepted plans A preventive mechanism that records and evaluates the execution plans of SQL statements over time Useful if you are more conservative and want to control which plans are used Used by Autonomous Databases 62
Tip: SQL Profiles or SQL Plan Baselines? SQL Profiles Use case: Reactive, SQL performance is suboptimal, invoke SQL Tuning Advisor on high load or business critical SQL Pros: A SQL Profile contains auxiliary information that corrects the cardinality estimated in a suboptimal Plan. Flexible and better approach with volatile data, not locked in by table sizes or existing access structures SQL Profiles with Indexes (recommendations) can significantly improve performance SQL Plan Baselines Use case: Proactive, you already know SQL performance is good and would like to seed the known plan to the optimizer Pros: Preserves the known plans, useful in a very predictable environment Cons: In a volatile environment, hard to know a good plan Application SQL should be executed more than once to capture the baseline If access structures change or dropped, may not be able to use the required plan 63
Program Agenda 1 2 3 4 SQL Tuning for Administrators: Evolving Landscape Proactive Approach and Tips Reactive Approach and Tips Conclusion 64
SQL Tuning for Cloud Administrators: Summary Proactive Tips SPA to validate system changes such as migration to Oracle Database Cloud, Exadata Cloud Machine, etc. SPA Quick Check in production environments to quickly assess impact of routine system changes (for e.g., optimizer refresh statistics) ADDM to detect systemic problems and remediate them by following ADDM recommendations Automatic SQL Tuning Advisor to tune high load SQL statements over time SQL Access Advisor to assure that your application has optimal access structures (indexes and type, MVs, MV logs, etc.). Tunes the entire workload Reactive Tips ASH Analytics to identify your top resource consumers SQL Monitor for detailed execution statistics on long running or parallel executions SQL Tuning Advisor to identify and remediate: Stale statistics Incorrect cardinality estimations Data correlation 65