Successful Upgrade Secrets: Preventing Performance Problems with Database Replay

Successful Upgrade Secrets: Preventing Performance Problems with Database Replay Prabhaker Gongloor (GP), Leonidas Galanis, Karl Dias Database Manageability Oracle Corporation

Oracle s Complete Enterprise Software Stack Built-in & Integrated Manageability Leader in the complete enterprise application stack Built-in manageability in every tier Integrated manageability across the entire stack

Oracle Enterprise Manager Increases Business Efficiency Manage applications topdown, from the business perspective by understanding user experiences and business impact of IT issues Manage entire application lifecycle to increase business agility with comprehensive application quality management and compliance solutions Reduce operational costs through intelligent diagnostics and automated IT processes

Agenda Why Upgrade? Upgrade Steps and Challenges Real Application Testing and Database Replay Database Replay for 9.2/10.2 DB Releases Upgrade Steps: EM Support Using 11g Snapshot Standby Performance Analysis Conclusion New! Please visit us at the OOW Demogrounds Booth Moscone West: L52/L51 : Database Replay/SPA

Why Upgrade? Benefits: Oracle Database 11g vs. 10g* Database Resident Connection Pool Java Just-In-Time Compiler Query Result Caching TimesTen Cache 2 0 0 0 % 1 5 0 0 % 1 2 1 0 8 1 4 0 1 2 0 1 0 0 1000 800 1 0 0 0 % 5 0 0 % 6 4 2 8 0 6 0 4 0 2 0 600 400 200 0 % N b r C o n n e c t i o n s Up to 20 x connections 0 S p e e d Up to 11 x Faster 0 Q u e r i e s Up to 25% Faster 0 TimesTen Up to 10 x Faster RAC Performance Enhancements Oracle Secure Backup Oracle Streams Enhancements Optimizer Stats Collection 1 8 0 % 1 4 0 2 0 0 1000 1 6 0 % 1 4 0 % 1 2 0 % 1 0 0 % 8 0 % 6 0 % 4 0 % 2 0 % 0 % Q u e r y I n t e n s i v e Up to 70% Faster 1 2 0 1 0 0 8 0 6 0 4 0 2 0 0 B a c k u p Up to 25% Faster 1 5 0 1 0 0 5 0 0 R e p l i c a t i o n Up to 2 x Faster 800 600 400 200 0 Stats Collection Up to 10 x Faster * Quantify the benefit of upgrade/new feature for YOUR workload using methodology discussed today

Why Upgrade? Lifetime Support Policy Today August 2012 August 2015 R2 July 2010 July 2013 January 2009 January 2012 Premier Support Extended Support Sustaining Support R2 July 2007 Jul 2008 July 2010 t 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 2004 2003 2002 http://www.oracle.com/support/library/data-sheet/oracle-lifetime-support-policy-datasheet.pd

Database Upgrade Steps and Challenges

Database Upgrade Process: Steps 1. Gather information about PROD environment 2. Determine the upgrade path & method 3. Prepare B/R strategy, clone and setup Test DB 4. Establish performance baseline before upgrade 5. Develop tests for database & applications 6. Test upgraded database & applications 7. Check performance before & after upgrade 8. Remediate regressions 9. Go Live! Majority of effort spent in developing tests, detecting and remediating regressions

Real Application Testing Reduces risk of new technology adoption through higher testing quality Reduces time for effective testing from weeks to days Features Database Replay SQL Performance Analyzer (SPA) Test Change Remediate Deploy Real Application Testing provides realistic testing for real-world systems

Real Application Testing Real Application Testing = SPA + Database Replay SPA + Database Replay are complementary Together, provide comprehensive & flexible testing solution SQL Performance Analyzer Predicts impact of of change on on SQL response time Assess change by by executing query part of of SQL in in isolation without concurrency Unit testing of of SQL, e.g., optimizer stats/config, drop index, etc. Can use for for upgrades from 9.2/10.1 10.2 or or 11g 10.2 10.2 or or 11g Database Replay Predicts impact of of change on workload throughput Assess change by replaying workload with concurrency Comprehensive testing of of DB stack, e.g, memory, RAC, concurrency related Can use for upgrades from 9.2 or 10.2 11g

Database Replay

Testing Today Production 1,000s of Real Online Users PRODUCTION

Testing Today Test 1-2 testers trying to be 1,000s of users PRODUCTION TEST

Database Replay Workload for 1,000s of online users captured Capture Workload PRODUCTION

Database Replay Workload for 1,000s of online users replayed Capture Replay Workload PRODUCTION TEST

Database Replay Test your system changes at production levels Capture Replay Workload PRODUCTION TEST

Database Replay Overview Replay actual production database workload in test environment Identify, analyze and fix potential instabilities before making changes to production Capture Workload in Production Capture full production workload with real load, timing & concurrency characteristics Move the captured workload to test system Replay Workload in Test Make the desired changes in test system Replay workload with full production characteristics Honor commit ordering Analyze & Report Errors Data divergence Performance divergence Analysis & Reporting

Supported Changes Client Client Client Changes Unsupported Middle Tier Changes Supported Database Upgrades, Patches Schema, Parameters RAC nodes, Interconnect OS Platforms, OS Upgrades CPU, Memory Storage Etc. Storage Recording of External Client Requests

Clients Database Replay Workflow Production ( 9.2.0.8) Test ( 11.1) Replay Driver * Mid-Tier Storage Storage Capture Process Replay Analysis & Reporting * No middle/client tier setup required

Step1 :Workload Capture Client Client Client Minimal overhead Platform and protocol independent Workload filters Backup Capture interesting workload Middle Tier OS Directory File 1 Server 1 Server 2 Server N Capture Infrastructure File 2 Background Production System File N

Workload Capture Overhead Performance overhead Workload dependent Proportional to the data sent from the client TPC-C throughput degradation about 4.5% Workload capture size TPC-C 20min 100 users 10 warehouses: 1.2G Enable capture for few minutes to assess size DSS OLTP LOW Overhead HIGH Long Running SQL Short SQL/DML Insert Intensive Large LOBs

Step 2: Process Workload Files Setup test system Application data should be same as production system as of capture start time Use RMAN, Snapshot Standby, imp/exp, Data Pump, etc. to create test system Make change: upgrade db and/or OS, change storage, migrate platforms, etc. Processing makes captured data into replay ready format Once processed, workload can be replayed many times For RAC copy all capture files to single location for processing File 1 File 2 File n Capture Files Test System File 1 File 2 File n Metadata Replay Files

Replay System Replay captured workload Replayed operations see the same data and perform the same work Preserve timing and concurrency characteristics Same number of user connections Replay Client Multithreaded OCI Client Drives multiple captured processes Scalable Architecture Interprets capture into sequence of OCI calls Functional replay Server Replay Clients Timing Preservation Re-mapping Server Commit Order Synchronization Server Sequence Replay Background File 1 File 2 File N Metadata Replay Files

Replay Options Synchronized Replay (Default) Workload is replayed in full synchronized mode Same concurrency and timing as production workload Transaction commit order is honored Ensures minimal data divergence Synchronization controls Workload can be replayed in unsynchronized mode Useful for load/stress testing High data divergence Parameters for controlling synchronization Commit order synchronization: SYNCHRONIZATION Think time synchronization: THINK_TIME_SCALE Connect (logon) time synchronization: CONNECT_TIME_SCALE Request rate preservation: THINK_TIME_AUTO_CORRECT

Replay Options Connection Remapping Capture-time connection string can be remapped for replay time Number of Replay Clients Configurable by user Calibration mode recommends number of replay clients needed for specific workload Replay clients are multithreaded clients that can drive multiple workload sessions

Comparison of LoadRunner & DB Replay Testing e-business Suite 80 80 Time Taken (Days) 60 40 20 0 4 Install & Setup 24 0 Understand Application Usage 20 Identify Key Transactions 20 2 Generate Workload Total Testing Time DB Replay: ½ month LoadRunner: 7 ½ months 0 5 Run Test 15X 5 DB Replay LoadRunner

Database Replay for Prior Releases

Real Application Testing for Prior Releases Helps Smooth Transition to Newer Releases New! Database Replay: Capture on older release; Replay on 11.1 and above SQL Performance Analyzer: Execute tests on 10.2 and above Feature Upgrade From Upgrade To Database Replay SQL Performance Analyzer 10g R2 9i R2 10g R2 10g R1 9i R2 11g 11g 10g R2 or 11g 10g R2 or 11g 10g R2 or 11g ML 560977.1: Real Application Testing for Earlier Releases

Database Upgrade Steps: EM Support

Database Upgrade Steps: EM Support 1. Gather information about PROD environment 2. Determine the upgrade path & method 3. Prepare B/R strategy, clone and setup Test DB 4. Establish performance baseline before upgrade 5. Develop tests for database & applications 6. Test upgraded database & applications 7. Check performance before & after upgrade 8. Remediate regressions 9. Go Live!

Database Upgrade Process: Steps 1. Gather information about PROD environment 2. Determine the upgrade path & method 3. Prepare B/R strategy, clone and setup Test DB 4. Establish performance baseline before upgrade 5. Develop tests for database & applications 6. Test upgraded database & applications 7. Check performance before & after upgrade 8. Remediate regressions 9. Go Live! 1,2: Session S298732: A Hitchhiker's Guide to Database Upgrades: Wednesday 09/24/2008 13:00-14:00 Moscone South

3. Backup/Recovery, Clone and Setup Test DB End-to-End Automation of Database Replay in EMGC 10.2.0.5 (when available) Capture production workload & performance data Setup Test DB : Provision, deploy software, clone database, RAC conversion Deploy Replay Clients Orchestrate Replay through GC Copy Workload and performance data to staging

Database Replay End-to-End Automation

Setup Test Database Workflow

Database Upgrade Process: Steps 1. Gather information about PROD environment 2. Determine the upgrade path & method 3. Prepare B/R strategy, clone and setup Test DB 4. Establish performance baseline before upgrade Use Snapshot Standby for Database Replay!! 5. Develop tests for database & applications 6. Test upgraded database & applications 7. Check performance before & after upgrade 8. Remediate regressions 9. Go Live!

Using Snapshot Standby DB with Database Replay Increase ROI through better utilization of standby DBs for Test/Dev Provides full scale test environment Simultaneously serves as DR environment Zero Data loss but no real-time query/fast fail-over Redo shipped but not applied Uses single copy like Storage snapshots but provides DR at the same time Session S298770: Beat Up Your Oracle Data Guard Standby with Oracle Real Application Testing: It s Payback Time! 09/24/2008 17:00-18:00 Moscone South 102

Using Snapshot Standby DB with Database Replay Physical Standby Apply Redo 1. ALTER DATABASE CONVERT TO SNAPSHOT STANDBY; Open Database Back out Changes 2. PERFORM REAL APPLICATION TESTING! Snapshot Standby Perform Testing * Continuous Redo Shipping 3. BACK OUT CHANGES 4. ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

Performance Analysis Workflow & Tools 1. CAPTURE INTERESTING WORKLOAD (PROD) Capture Profile Capture Report AWR / ASH/ ADDM Report 2. ESTABLISH REPLAY BASELINE WITHOUT CHANGE (TEST) EM interface Replay Profile 4. MAKE CHANGE, REPLAY (TEST) 5. ANALYZE REPLAY PERFORMANCE, REMEDIATE 6. DEPLOY CHANGE (PROD) Yes DONE? No Replay Report AWR / ASH/ Compare Period / ADDM Report Compare to Capture or to other Replays EM interface

Database Replay and Performance Baseline and Analysis AWR infrastructure can be used during/after capture and replay as before!!! ASH/ADDM/Compare Period Analysis Advisors Automatic snapshots at begin & end of capture and replay Transportable AWR enables comparing performance across databases Use Datapump API and requires staging schema Exposed through EM and command line interface Session: S298703: Performance Fundamentals for Oracle Database 10g and Oracle Database 11g, 09/22/2008 14:30-15:30 MS 305

Database Replay: Workflow

Database Replay: Workload Capture

Database Replay: Workload Replay

Database Replay: Analysis

Replay Report: Replay Options, Statistics, Divergence

Replay Report: Error & Divergence: Which Sessions and SQL? Sessions SQL: Errors SQL: Divergence ORA-947: Not enough values

Database Replay: Performance Analysis

AWR Compare Period Report: Load Profile 1. Capture Vs Replay 2. Replay Vs Replay With Change

AWR Compare Period Report: Top SQL by Elapsed Time, #Unique Plans Seen Capture Vs Replay Replay Vs Replay With Change

David Mitchell Senior Vice President, OVUM Oracle Real Application Testing reduces the time required to test changes by as much as 80%, lower testing costs by as much as 70%, mitigate risks by reducing the number of unexpected outages, and improve the quality of service for their IT operations. Source: Oracle Real ApplicationTesting business agility through superior testing, Jan 2008

Sean McCown Contributing Editor, Infoworld Oracle Database 11g will be welcome relief to DBAs responsible for testing changes against a production workload. The way that Data Guard and Real Application Testing work together is sheer poetry, and no doubt most shops will quickly come to rely on the combination Source: Oracle Real ApplicationTesting business agility through superior testing, Jan 2008

Resources on OTN Real Application Testing for Earlier Releases Database Replay: Technical White Paper Testing Performance Impact of an Oracle Database 9i/10g Release 1 to Oracle Database 10g Release 2 Upgrade with SQL Performance Analyzer Migration to Cost-Based Optimizer Upgrading from Oracle 9i to 10g: What to expect from the Optimizer Upgrade Companion: Metalink Note: 466181.1: One-stop shop for Upgrades Oracle By Example Series: Manageability on OTN FAQ

Conclusion Database Replay provides comprehensive and easy to use solution for assessing impact of changes on workload Higher quality testing Full coverage of workflows Database Replay can be used to test many changes Upgrade testing including 9.2, 10.2 to 11g releases Enabling new features, RAC, schema, parameter, OS/HW changes Helps adopt technology faster by cutting down testing time from months for days With Database Replay and Real Application Testing businesses can Stay competitive Improve profitability Be compliant

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

Recommended Campground Demos Complete Data Center Management Oracle Real Application Testing: Database Replay and SQL Performance Analyzer Demo Self-Managing Database: Automatic Performance Diagnostics Self-Managing Database: Automatic Application & SQL Tuning Self-Managing Database: Automatic Fault Diagnostics Change Management & Data Masking for DBAs Application Quality Management Location Moscone West Exhibit Hall Moscone West Exhibit Hall Moscone West Exhibit Hall Moscone West Exhibit Hall Moscone West Exhibit Hall Moscone West Exhibit Hall Moscone West Exhibit Hall

Recommended Sessions (Monday sessions) Session Title Date Time Location Optimizing Application Performance: Application Testing Suite to the Rescue Monday, Sept. 22 1:00 pm Moscone West 2001 Performance Fundamentals for Oracle Database 10g and Oracle Database 11g Monday, Sept. 22 2:30 pm Moscone South 302 First-Failure Fault Diagnosability and Diagnostics: Oracle Database 11g Features and Novel Approaches Monday, Sept. 22 2:30 pm Moscone South 304 Oracle Enterprise Manager: Oracle's Management Solution for Your Enterprise Monday, Sept. 22 4:00 pm Moscone West 2003

Recommended Sessions (Tuesday sessions) Session Title Date Time Location Application Upgrade Secrets: Avoid Surprises While Making Database Changes Tuesday, Sept. 23 9:00 am Moscone West 2003 Using Oracle Database 11g Real Application Testing to Simulate Production System Patching Tuesday, Sept. 23 9:00 am Moscone South 304 Advanced Performance Diagnostics: What the GUI Doesn't Tell You Tuesday, Sept. 23 11:30 am Moscone West 2003 Demystifying SQL Tuning: Tips and Techniques for SQL Experts Tuesday, Sept. 23 1:00 pm Moscone South 303

Recommended Sessions (Wednesday sessions) Session Title Date Time Location Storage Monitoring Made Easy: Diagnosing I/O Performance Problems Wednesday, Sept. 24 9:00 am Moscone South 303 Applications Data Privacy: An Expert Panel Discussion Wednesday, Sept. 24 11:30 am Moscone West 2001 SQL Tuning Roundtable with the Experts Wednesday, Sept. 24 1:00 pm Moscone West 2001 Deploying Oracle Enterprise Manager in a Secure Maximum Availability Architecture Wednesday, Sept. 24 5:00 pm Moscone West 2001

Recommended Sessions (Thursday sessions) Session Title Date Time Location The Danish Experiment: Oracle Database 11g Shock Upgrades and Massive Workload Reduction via COBS Thursday, Sept. 25 9:00 am Moscone South 301 Integrating 40 Data Centers in Three Years: How Oracle's DBAs Control the Data Center Explosion Thursday, Sept. 25 10:30 am Moscone South 303 Application Testing Best Practices: Real-World Customer Testimonials Thursday, Sept. 25 12:00 pm Moscone South 303 Proactive Performance Monitoring with Baselines and Adaptive Thresholds Thursday, Sept. 25 1:30 pm Moscone South 303 Managing Oracle Grid Computing: Oracle Real Application Clusters, Oracle Automatic Storage Management, Oracle Data Guard Thursday, Sept. 25 3:00 pm Moscone South 303