... Surya Rao Service Line Architect, IBM Senior Certified IT Specialist IBM GTS, Service Delivery, Data Management July 15, 2012

Transcription

1 Database Replay Surya Rao Service Line Architect, IBM Senior Certified IT Specialist IBM GTS, Service Delivery, Data Management July 15, 2012 In Collaboration with the IBM Oracle International Competency Center Copyright IBM Corporation, All Rights Reserved. All trademarks or registered trademarks mentioned herein are the property of their respective holders

2 Table of contents Abstract... 1 Prerequisites Executive Summary Introduction Licensing Considerations Bird s eye view of Workload Capture & Replay PRE CAPTURE CHECKLIST Limitations & Restrictions Check List Capturing a Workload Creating a Capture Directory Capture Directory and its contents Filtering the capture workload Start workload capture Completing the Capture process Workload Capture Files Workload Capture Report Exporting AWR data Useful Views PRE REPLAY CHECKLIST Pre-process the captured workload Prepare the Test System Replaying a Workload Workload Analyzer Calibrating the workload before replay Initialize the Workload Replay Connection Maps Prepare the Workload Replay and check status Starting the Workload Replay Clients (wrc) Starting the Workload Replay Changing Workload Replay execution status Replay information extraction Extracting the Replay report for a specific Workload Sample Replay Report Exporting AWR data Useful Views Data Divergence on Replay Sample Simulation A Test case for demonstrating Data Divergence Replaying the workload to cause a Data Divergence Checking for Data Divergences...41 Oracle Real Application testing

3 8.4 Extract the Diverging SQL statement Using Enterprise Manager for Real Application Testing Summary Operating system...45 Storage System Management...45 Resources About the author Appendix Common Errors and Resolutions Patch level for RAT...47 Trademarks and special notices Oracle Real Application testing

4 Abstract This white paper describes product functionality. This is a tool that captures workload including timing, concurrency and dependency information and is able to run this captured load into an alternate environment. Real application Testing is two tools. It is SQL Performance Analyzer where you perform SQL unit testing to identify SQL regression. Database replay is what is described in this document. The target audience is installations running Oracle Database on IBM hardware and IBM field specialist that support those environments. Prerequisites The reader of this document is expected to have a general and basic knowledge of the Oracle Database and IBM servers. 1.0 Executive Summary Change is an inevitable part of the IT industry and it is almost always associated with risks. To mitigate risks in production systems due to planned changes, there are various approaches prevalent in the industry the most prominent approach pertains to load testing with third party tools on nonproduction environments. Majority of these conventional tools rely on the Test Scripter s understanding of the Production workload and generate their workloads based on certain inputs and parameters specified. However, none of these Load Testing tools available today are able to generate an exact workload of production and run them on non-production databases and hence open to risks This results in issues being faced in production (Post the change activity) even after successfully completing the conventional load testing. To drastically mitigate this risk and address the shortcomings prevalent in conventional Load Testing tools, Oracle introduced the Real Application Testing offering. This is the solution in the industry which captures an actual production workload including timing, concurrency and dependency information and is able to run this captured load into an alternate environment. With this load testing, we are able to know the exact impact of a prospective change and hence avoid unknown risks. Below are some of change examples where customer will benefit with Real Application Testing tools. Database upgrades Database Patches Storage changes Network Changes 1

5 Hardware upgrades OS upgrades Database configuration changes Scale up testing and consolidation Some of Key features of Real Application Testing tools Web based interface Able to access impact of complex change on complex production environment from months to days Able to capture load from prior releases and replay them only on 11g upwards SQL Performance Analyzer will analyze change on SQL execution plan and performance The option includes the following features: Database Replay SQL Performance Analyzer (SPA) SQL Tuning Sets (STS). This is also part of the Tuning Pack SPA and STS are not in the scope of this white paper. 2.0 Introduction is a separately licensed option with Oracle Database Enterprise Edition 11g. The purpose of this product is to allow extraction of production workloads and simulate running them on preproduction or test databases to replicate a current performance issue for deeper analysis and resolution on non production environments. The Database Replay feature of the Real Application Testing suite can be used to test any substantial changes such as Database upgrades, Storage or network changes, Hardware or OS upgrades, Database configuration changes etc. It can also be used in Change Management related baselines and regression testing scenarios where the behavior of the system has to be observed 2

6 based on a fixed and unchanging workload to ensure that the results are comparable across test runs. Note that although this feature was introduced in Oracle Database 11g, it is possible to capture workloads from databases but it is possible to replay them only on Oracle Database 11g and upwards. We can also capture workloads from , & provided that the RAT capture patch is installed. This could be an excellent avenue for testing pre and post upgrade environments, an area that is not very easy to cover at the moment, in the absence of Real Application Testing 2.1 Licensing Considerations The option includes the following features: Database Replay SQL Performance Analyzer (SPA) SQL Tuning Sets (STS) In order to use the features listed in this whitepaper, it is required to purchase licenses for the option, with one exception: SQL Tuning Sets (STS) can be used if you have licensed either the Tuning Pack or. Oracle Real Application Testing functionality can be accessed from the Enterprise Manager as well as through the database server command-line APIs. The use of either interface requires an option license. Additionally, the license is required on both the capture and replay systems for Database Replay and is charged by the total number of CPUs on those systems. In production Oracle charges according to the license model that is used. On Test you can either choose between CPU or NUP (No. of Processors). Refer to this link for more information on the various licensing options 3

7 3.0 Bird s eye view of Workload Capture & Replay Oracle strongly recommends the usage of SPA to address SQL regression analysis before using the Database Replay option. The steps involved in using can be summarized as follows: 1. Workload Capture (On Source/Production database) 2. Workload Pre-processing (On the Target/Test database) 3. Prepare / Initialize the Target DB for the workload replay 4. Calibrate the workload to estimate number of workload replay clients needed 5. Initiate the workload replay clients (wrc clients) 6. Start the Workload replay 7. Generate the Replay report 8. Analyze the reports 4

8 x 5

9 4.0 PRE CAPTURE CHECKLIST 4.1 Limitations & Restrictions 1. For versions prior to 11g, i.e x, and , there are mandatory patches to be applied before workloads can be captured. These patches (applied via opatch utility) install the required Real Application Testing functionality in the Pre-11g releases. 2. For a source 10g R2 database, the minimum version for which capture is possible is for non-windows operating systems and for Windows operating systems. 3. Note that the captured workloads can only be replayed on versions 11.1 and above. 4. The entire list of patches for each source (capture) and target (replay) version is mentioned in the My Oracle Support article The following types of client requests are not captured in a workload as per the Oracle Documentation * Direct path load of data from external files using utilities such as SQL*Loader * Non-PL/SQL based Advanced Queuing (AQ) * Flashback queries * Oracle Call Interface (OCI) based object navigations * Non SQL-based object access * Distributed transactions (any distributed transactions that are captured will be replayed as local transactions). Distributed transactions using two phase commit are not supported and will not be replayed as even local transactions. 4.2 Check List 1. Ensure Oracle Home binary is at the correct patch level for Real Application Testing capture. (see appendix ) 2. Real Application Testing option must be installed 3. Activate the Real Application Testing option. 6

10 a. For Oracle Database 11g it is enabled by default but package DBMS_WORKLOAD_CAPTURE must be valid. If this package is absent, it can be created by running dbmswrr.sql. b. For Oracle Database release and execute wrrenbl.sql to set PRE_11G_ENABLE_CAPTURE. 4. Setting up a Database Replay Administrator The SYS or SYSTEM account is not required to be used for the capture and replay steps described above. Any user that is granted the below privileges can be used to administer Capture and Replay CREATE USER <USERNAME> IDENTIFIED BY oracle; GRANT CREATE SESSION, RESOURCE to <USERNAME>; GRANT EXECUTE ON DBMS_WORKLOAD_CAPTURE TO <USERNAME>; GRANT EXECUTE ON DBMS_WORKLOAD_REPLAY TO <USERNAME>; GRANT CREATE SESSION TO <USERNAME>; GRANT CREATE ANY DIRECTORY TO <USERNAME>; GRANT SELECT_CATALOG_ROLE TO <USERNAME>; GRANT BECOME USER TO <USERNAME>; GRANT UNLIMITED TABLESPACE TO <USERNAME> 5. Save the spfile / database initialization parameters and environment specifics in order to replicate in replay environment. This is particularly applicable for Oracle Real Application Clusters (RAC) environments. 6. Capture scripts should be run as SYS 7. A Database Directory Object has to be created for capture file. In a RAC environment must be on shared disk storage. Ensure enough capacity is available. 8. Identify workload period processing window based on the target technical goal of completing the replay as outlined in introduction section of this document. 9. Plan to capture the logical/physical state of the database to clone in the test environment, has to be time synchronized with the start of the capture window to avoid data divergence results on the test environment.. Other alternative methods are the use of datapump or snapshot standby. 10. May need to consider restarting the database or commencing capture in dormant periods to ensure transaction activity is at a consistent point. This will lessen the volume of divergence events on the Test environment when replayed. 11. The recommendation is to put the database into restricted mode to allow the capture process to be started. Once capture is started then database is automatically taken out of restricted mode.in a 7

11 production environment that would limit client access for a short period of time. Can be bypassed but at the expense of transaction capture integrity ( auto_unrestrict => FALSE ) 8

12 5.0 Capturing a Workload Capturing a workload is performed either using the Oracle supplied PL/SQL APIs or using Enterprise Manager. In this document, we have predominantly shown examples using the Oracle supplied PL/SQL APIs. 5.1 Creating a Capture Directory SQL> create directory CAPTURE_DIR as 'c:\app\capture'; Directory created. 5.2 Capture Directory and its contents This is the directory where the metadata related to the workload capture will be placed. Initially the directory and its sub-directories will either be empty or not even exist. SQL>!ls -l /home/oracle/capture total 0 drwxr-xr-x 2 oracle oinstall drwxr-xr-x 3 oracle oinstall 256 Oct 07 00:41 cap 256 Oct 07 00:24 capfiles If the capture processing is initiated with a previous workload's captured files still existing in the directory, the below error would be encountered. ORA-15505: cannot start workload capture because instance 1 encountered errors while accessing directory Reason: There is an existing file in the CAPTURE Directory. There should be no files including no hidden files, before the dbms_workload_capture.start_capture is executed. Delete all content in the capture directory and retry 9

13 The schema that is used to run the workload can belong to any user who has the necessary privileges to perform the required DML, DDL against the objects referred to in the workload SQL sets. 5.3 Filtering the capture workload Filtering can be done on the following attributes. The default action is to include all transactions in the replay. To include or exclude specific workloads, the following filters can be applied before the Capture or during the Replay 1. Instance_Number 2. User 3. Module 4. Program 5. Action 6. Service The steps for the Workload capture are as demonstrated below. The recommendation is to filter out monitoring activities like those performed by Enterprise Manager. Adding a Filter is basically performed to restrict and exclude unwanted workloads from being captured. By default everything is captured in the absence of the filters BEGIN DBMS_WORKLOAD_CAPTURE.ADD_FILTER ( fname => 'user_scott', fattribute => 'USER', fvalue => 'SCOTT'); DBMS_WORKLOAD_CAPTURE.ADD_FILTER ( fname => 'instance_filter', fattribute => 'INSTANCE_NUMBER', fvalue => '1' ); END; / 10

14 If a filter that was previously added needs to be removed before capturing the workload, the following command can be used to delete it. BEGIN DBMS_WORKLOAD_CAPTURE.DELETE_FILTER (fname => user_scott ); END; / To check the filters defined and their statuses i.e NEW or USED, use the below step SQL> select * from dba_workload_filters; TYPE ID STATUS NAME ATTRIBUTE VALUE CAPTURE 0 NEW USER_SCOTT USER SCOTT 5.4 Start workload capture BEGIN DBMS_WORKLOAD_CAPTURE.START_CAPTURE ( name => 'PEAK_PROCESSING_2', dir => 'CAPTURE_DIR', default_action=>'exclude', duration => null, --means capture has to be ended manually auto_unrestrict => FALSE --if the above is TRUE all instances started up in RESTRICT mode will be made -- unrestricted when the capture ends ); END; / PL/SQL procedure successfully completed. 11

15 To check for the captures that have been completed on this database or to query the ones that are currently in progress, follow the step provided below set lines 200 col status format a15 col name format a15 col dbversion format a12 SELECT ID, NAME, DBID,DBNAME,DBVERSION, PARALLEL,STATUS, to_char(start_time,'dd/mm/yyyy hh24:mi'), to_char(end_time,'dd/mm/yyyy hh24:mi'),duration_secs FROM DBA_WORKLOAD_CAPTURES; Once capture complete, status should be completed. SQL> select id, name, status,start_scn, end_scn, dbtime_total, connects, dir_path from dba_workload_captures; ID NAME STATUS START_SCN END_SCN DBTIME_TOTAL CONNECTS DIR_PATH PEAK_PROCESSING COMPLETED c:\app\capture 3 PEAK_PROCESSING_2 IN PROGRESS c:\app\capture For RAC, capture is executed on one node only For onwards. In earlier releases the capture will have to be initiated on all the nodes. 5.5 Completing the Capture process 12

16 BEGIN DBMS_WORKLOAD_CAPTURE.FINISH_CAPTURE (); END; / PL/SQL procedure successfully completed. set lines 200 col status format a15 col name format a15 col dbversion format a12 SELECT ID, NAME, DBID,DBNAME,DBVERSION, PARALLEL,STATUS, to_char(start_time,'dd/mm/yyyy hh24:mi'), to_char(end_time,'dd/mm/yyyy hh24:mi'),duration_secs FROM DBA_WORKLOAD_CAPTURES; 13

17 5.6 Workload Capture Files The following files are created in the designated capture directory. These files are release dependent and may vary for older releases SQL>!ls -l /home/oracle/capture/cap total 104 -rw-r--r-- 1 oracle oinstall Oct 07 00:41 wcr_cr.html -rw-r--r-- 1 oracle oinstall Oct 07 00:41 wcr_cr.text -rw-r--r-- 1 oracle oinstall 268 Oct 07 00:40 wcr_fcapture.wmd -rw-r--r-- 1 oracle oinstall 171 Oct 07 00:24 wcr_scapture.wmd SQL>!ls -l /home/oracle/capture/capfiles/inst1 total 16 drwxr-xr-x 2 oracle oinstall 4096 Oct 07 00:40 aa drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 ab drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 ac drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 ad drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 ae drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 af drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 ag drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 ah drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 ai drwxr-xr-x 2 oracle oinstall 256 Oct 07 00:24 aj -rw-r--r-- 1 oracle oinstall 143 Oct 07 00:24 wcr_8x37xh rec NOTE - Once a set of filters are defined and used by starting and finishing a capture, these filters are marked as USED in dba_workload_filters and can't be used again. The same filter names can be redefined and will show up as NEW in the dba_workload_filters view. TYPE ID STATUS NAME ATTRIBUTE VALUE CAPTURE 1 USED USER_SCOTT USER SCOTT CAPTURE 1 USED INSTANCE_FILTER INSTANCE_NUMBER 1 CAPTURE 0 NEW USER_SCOTT USER SCOTT CAPTURE 0 NEW INSTANCE_FILTER INSTANCE_NUMBER 1 14

18 5.7 Workload Capture Report The workload capture report is very similar to an AWR report detailing most aspects of the workload as shown below SQL> set pagesize 0 long longchunksize 2000 SQL> select dbms_workload_capture.report (capture_id =>1,format=>'TEXT') from Database Capture Report For ORCL DB Name DB Id Release RAC Capture Name Status ORCL NO PEAK_PROCESSING_2 COMPLETED Start time: 11-Aug-11 15:24:53 (SCN = ) End time: 11-Aug-11 15:48:47 (SCN = ) Duration: 23 minutes 54 seconds Capture size: KB Directory object: CAPTURE_DIR Directory path: c:\app\capture Directory shared in RAC: TRUE Filters used: 2 INCLUSION filters Statistic Name Value % Total DB time (secs) Average Active Sessions 0.00 User calls captured User calls captured with Errors 0 Session logins Transactions Top Events Captured DB: ORCL Snaps: 7-8 Avg Active Event Event Class % Event Sessions db file scattered read User I/O CPU + Wait for CPU CPU WCR: capture file IO write Other control file sequential read System I/O

19 Sid, Serial# % Activity Event % Event User Program # Samples Active XIDs , control file parallel write SYS ORACLE.EXE (CKPT) 3/1,434 [ 0%] 0 0 control file sequential read /1,434 [ 0%] 5, db file parallel write 9.09 SYS ORACLE.EXE (DBW0) 2/1,434 [ 0%] 0 21, WCR: capture file IO write 4.55 SYS ORACLE.EXE (J001) 1/1,434 [ 0%] 0 24, WCR: capture file IO write 4.55 SYS ORACLE.EXE (M001) 1/1,434 [ 0%] 0 58, control file sequential read 4.55 SYS ORACLE.EXE (ARC1) 1/1,434 [ 0%] Workload Filters DB: ORCL Snaps: 7-8 # Filter Name Type Attribute Value INSTANCE_FILTER INCLUDE INSTANCE_NUM 1 2 USER_SCOTT INCLUDE USER SCOTT Exporting AWR data AWR data can be exported during the time of the capture as well as during the replays. Exporting the workload capture AWR data is is recommended since it is used for replay performance comparison. At the end of each capture or replay, the relevant AWR snapshots for the period during which the capture or replay was executing will be automatically exported and included in the captured or replayed data. The export log would look something like the below. 16

20 .. exported "SYS"."WRH$_SYSMETRIC_HISTORY" KB 100 rows.. exported "SYS"."WRH$_SQL_PLAN" MB rows.. exported "SYS"."WRH$_SYSMETRIC_SUMMARY" KB 296 rows.. exported "SYS"."WRH$_ENQUEUE_STAT" KB 174 rows.. exported "SYS"."WRH$_WAITCLASSMETRIC_HISTORY" KB 0 rows These exported AWR snapshots can then be imported into another system into a specified and empty schema. The commands to export AWR reports during Capture and Replay are provided below EXEC DBMS_WORKLOAD_CAPTURE.EXPORT_AWR (capture_id => 12); EXEC DBMS_WORKLOAD_REPLAY.EXPORT_AWR (replay_id => 13); 5.9 Useful Views This section summarizes the views that you can display to monitor workload replay. You need DBA privileges to access these views. The DBA_WORKLOAD_CAPTURES view lists all the workload captures that have been captured in the current database. The DBA_WORKLOAD_FILTERS view lists all workload filters for workload captures defined in the current database. The DBA_WORKLOAD_REPLAYS view lists all the workload replays that have been replayed in the current database. The DBA_WORKLOAD_REPLAY_DIVERGENCE view enables you to view information about diverged calls, such as the replay identifier, stream identifier, and call counter. The DBA_WORKLOAD_REPLAY_FILTER_SET view lists all workload filters for workload replays defined in the current database. The DBA_WORKLOAD_CONNECTION_MAP view lists the connection mapping information for workload replay. The V$WORKLOAD_REPLAY_THREAD view lists information about all sessions from the replay clients. The following Oracle supplied PL/SQL API packages contain the subprograms to completely manage and administer the Capture and Replay processes DBMS_WORKLOAD_CAPTURE package DBMS_WORKLOAD_REPLAY package 17

21 6.0 PRE REPLAY CHECKLIST 6.1 Pre-process the captured workload The preprocessing should always be done on a system which has the same DB version as the source DB where the workload was captured. This step converts the captured data into replay files and also creates the necessary metadata. This step is required to be done only once (for each version of the target system) after which it can repeatedly be replayed on a target system. Preprocess of captured work load is resource intensive hence this step should be done on a test system separate from the Production system from where the workload was captured. This system can be the same as the replay system or a different one. For a RAC configuration, it is required to do the pre-processing against any one node only. Workload capture happen into 'c:\app\capture directory in previous section while for pre-processing we are doing on 'c:\app\capture2 directory. CREATE OR REPLACE DIRECTORY capture_dir AS 'c:\app\capture2'; Directory created. BEGIN DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE (capture_dir => 'CAPTURE_DIR'); END; / PL/SQL procedure successfully completed. Pre-processed files are created on c:\app\capture2 directory. These preprocessed files will be used by the replay clients in constructing the workload during the replay stage. The preprocessed files contain the metadata (i.e. attributes) describing the workload 18

22 6.2 Prepare the Test System The test system can be prepared as follows Ensure Oracle Home binary on database replay hosts and replay client hosts, is at the correct patch level for Real Application Testing replay. (see appendix ). Setting up a Database Replay Administrator The SYS or SYSTEM account is not required to be used for the capture and replay steps described above. Any user that is granted the below privileges can be used to administer Capture and Replay CREATE USER <USERNAME> IDENTIFIED BY oracle; GRANT CREATE SESSION, RESOURCE to <USERNAME>; GRANT EXECUTE ON DBMS_WORKLOAD_CAPTURE TO <USERNAME>; GRANT EXECUTE ON DBMS_WORKLOAD_REPLAY TO <USERNAME>; GRANT CREATE SESSION TO <USERNAME>; GRANT CREATE ANY DIRECTORY TO <USERNAME>; GRANT SELECT_CATALOG_ROLE TO <USERNAME>; GRANT BECOME USER TO <USERNAME>; GRANT UNLIMITED TABLESPACE TO <USERNAME> Ensure key database initialization parameters are set correctly based on target goal of replay. Check host(s) environment configuration is set up correctly e.g RAC. Restore the Production system and clone it to the test system OR Restore only selected and relevant schemas from Production to Test In addition, the System time of the replay system must be close to the Production system if the workload has time sensitive data Note from the Oracle documentation: It is recommended that the system time on the replay system host be changed to a value that approximately matches the capture start time just before replay is started. Otherwise, an invalid data set may result when replaying time-sensitive workloads. For example, a captured workload that contains SQL statements using the SYSDATE and SYSTIMESTAMP functions may cause replay divergence when replayed on a system that has a different system time. Resetting the system time will also minimize job scheduling inconsistencies between capture and replay. The Workload Analyzer will recommend if the captured workload is sensitive to time and if system clock needs to be reset. 19

23 This is not mandatory, but recommended for workloads that are sensitive to time(e.g., sysdate) If the source system makes use of DB Links, these DB links have to be created or modified on the target system. The same consideration applies to external directories and external tables. Ensure all SQLNET connectivity files do not point to the production environment target host and distributed interfaces. Ensure object code is checked and amended for all external interface communication e.g url access, mail, ftp. Execute Workload analyzer on preprocessed directory to validate capture files before replay. Analyze the result and address the recommendations. For repeated testing i.e. to repeatedly replay the captured workload, each time making changes to the code, DB configuration or other variables, it is useful to create a Savepoint as described below. In addition, if we find that the replay runs for long periods of time, we could make use of the create restore point <savepoint name> guarantee flashback database. drop restore point <Savepoint Name>; create restore point <Savepoint Name>; Once a Savepoint has been created, the DB can be reverted back to the Savepoint as follows and then the replay can be executed once again. See the steps described in the below box. 20

24 SQL> connect / as sysdba Connected. SQL> shutdown immediate Database closed. Database dismounted. ORACLE instance shut down. SQL> startup mount ORACLE instance started. Total System Global Area bytes Fixed Size bytes Variable Size bytes Database Buffers bytes Redo Buffers bytes Database mounted. SQL> flashback database to restore point <Savepoint_name>; Flashback complete. SQL> alter database open resetlogs; Database altered. After the DB has been reverted back to a clean slate as described above, the workload can be replayed once again. This approach allows us to make changes to the configuration, code or other variables and replay the workload any number of times until we achieve a satisfactory performance. 7.0 Replaying a Workload 7.1 Workload Analyzer Refer to this Metalink article for full details. A snippet from the My Oracle Support note states as follows. The workload analyzer can be downloaded from the above article. It will be available in the RDBMS and can also be directly executed from EM 12c. 21

25 The Workload Analyzer is a Java program that analyzes a workload capture directory and identifies parts of a captured workload that may not replay accurately due to factors that include insufficient data, errors that occurred during workload capture, or usage of features that are not supported by Database Replay. The results of the workload analysis are saved to an HTML report named wcr_cap_analysis.html located in the capture directory that is being analyzed. If an error can be prevented, the workload analysis report displays available preventive actions that can be implemented before replay. If an error cannot be corrected, the workload analysis report provides a description of the error so it can be accounted for during replay. Running Workload Analyzer is the default option and is strongly recommended. 7.2 Calibrating the workload before replay Calibrating the workload is done to understand the characteristics of the captured workload such as number of concurrent sessions that were in play and total number of sessions. The calibration feature also provides a recommended estimate of the number of wrc clients per CPU that could be used for the actual replay for best simulation of the workload. In addition, it also reports on the memory needed per client process wrc mode=calibrate replaydir=c:\app\capture2 Workload Replay Client: Release Production on Thu Aug 11 19:33:10.11 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. Report for Workload in: c:\app\capture Recommendation: Consider using at least 1 clients divided among 1 CPU(s) You will need at least 3 MB of memory per client process. wrc If your system machine(s) mode=replay cannot match replaydir=c:\app\capture that number, consider using more clients. Password: Workload Characteristics: Errors - max in concurrency: file : 1 sessions ORA-15554: - total number cannot of sessions: start workload 3 replay client because the database server is Assumptions: not in PREPARE mode - 1 client process per 50 concurrent sessions - 4 client process per CPU KB of memory cache per concurrent session - think time scale = connect time scale = synchronization = TRUE 22

26 7.3 Initialize the Workload Replay This step loads the metadata contained in the pre-processed files into internal tables. Basically, this step places the DB in a state of readiness for the replay. Connections strings are loaded into the DBA_WORKLOAD_CONNECTION_MAP view BEGIN DBMS_WORKLOAD_REPLAY.INITIALIZE_REPLAY (replay_name => 'PEAK_REPLAY', replay_dir => 'CAPTURE_DIR'); END; / PL/SQL procedure successfully completed. 7.4 Connection Maps When the workload is captured on the source system, all the connection strings used to establish connections to the target DB are captured as part of the workload. Needless to say, these connection strings will not be valid for the replay system as it will be a different DB and even perhaps on a different server. Therefore, connections strings need to be remapped prior to the replay on the target system. Connections can be remapped as follows. The connection_id is obtained by querying the DBA_WORKLOAD_CONNECTION_MAP view exec dbms_workload_replay.remap_connection(<connection_id>, '<replay_connection>'); e.g. exec dbms_workload_replay.remap_connection(1, '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=test_host)(PORT=1521))(CON NECT_DATA=(SERVICE_NAME=TEST_SERVICE)))'); 23

27 For Oracle Real Application Clusters databases, you can map all connection strings to a load balancing connection string. This is especially useful if the number of nodes on the replay system is different from the capture system. Alternatively, if you want to direct workload to specific instances, you can use services or explicitly specify the instance identifier in the remapped connection strings. 7.5 Prepare the Workload Replay and check status BEGIN DBMS_WORKLOAD_REPLAY.PREPARE_REPLAY (SYNCHRONIZATION=>TRUE, --means transactions are replayed in SCN order CONNECT_TIME_SCALE=>100, --means same time between connections as in captured workload THINK_TIME_SCALE=>100, --means 1.5x the time between subsequent calls THINK_TIME_AUTO_CORRECT=>false, SCALE_UP_MULTIPLIER=>1); --means how many times to multiply the workload by during replay END; / PL/SQL procedure successfully completed. Checking for replays in progress COL NAME FORMAT A15 COL STATUS FORMAT A20 SET LINES 200 SELECT ID,NAME, STATUS FROM DBA_WORKLOAD_REPLAYS; 24

28 Workload replay filters can be added if required similar to the filters defined during the capture process. If no filters are defined, the entire captured workload is replayed. The following replay options can be specified with the DBMS_WORKLOAD_REPLAY.PREPARE_REPLAY procedure during the Prepare stage synchronization connect_time_scale Think_time_scale Controls whether or not the commit order is preserved during replay. I.e. the transactions are committed in exactly the same order as they were committed in the Capture system. Alternatively the transactions can be committed in any order as long as they do not violate their transactional dependencies. Synchronization can also be done at the Object level. Scales the time elapsed between the start of the replay and the start of each session Scales the time elapsed between two successive user calls from the same session. This parameter can be used to reduce or scale up the think time to simulate different types of variations to the same workload Think_time_auto_correct Auto corrects the think time between calls when user calls takes longer during the replay than during the capture The below query displays one row for every replay client session that is connected to the database and executing the captured workload. SQL> select * from v$workload_replay_thread; no rows selected 25

29 7.6 Starting the Workload Replay Clients (wrc) wrc system mode=replay replaydir=c:\app\capture2 Workload Replay Client: Release Production on Thu Aug 11 19:58: Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. Password:*********** Wait for the replay to start (19:58:14) COL LOGON_USER FORMAT A15 COL PROGRAM FORMAT A8 COL EVENT FORMAT A25 COL SID FORMAT 99 COL FILE FORMAT A5 COL CLOCK FORMAT 99 SET LINES 200 SELECT CLOCK, SID, LOGON_USER, PROGRAM, FILE_NAME,CLIENT_OS_USER,EVENT FRO V$WORKLOAD_REPLAY_THREAD; Note: Once a workload replay is started, new replay clients will not be able to connect to the database. Only replay clients that were started before the START_REPLAY procedure is executed will be used to replay the captured workload. 26

30 7.7 Starting the Workload Replay A few pre-checks to ensure that we have a blank slate to start with. SQL> SHOW USER USER is "SYS" SQL> DESC SCOTT.PEAK_PROCESSING2 ERROR: ORA-04043: object SCOTT.PEAK_PROCESSING2 does not exist Note The error is expected since the workload is expected to create this object Now start the workload replay, using the START_REPLAY procedure as follows: BEGIN DBMS_WORKLOAD_REPLAY.START_REPLAY (); END; / Then, we will see the below appearing in the output of the terminals where the wrc.exe was started 27

31 7.8 Changing Workload Replay execution status BEGIN DBMS_WORKLOAD_REPLAY.PAUSE_REPLAY (); END; / BEGIN DBMS_WORKLOAD_REPLAY.RESUME_REPLAY (); END; / Completion Messages Replay started (20:02:43) Replay finished (20:04:50) 28

32 7.9 Replay information extraction SET SERVEROUTPUT ON DECLARE cap_id NUMBER; rep_id NUMBER; rep_rpt CLOB; BEGIN cap_id := DBMS_WORKLOAD_REPLAY.GET_REPLAY_INFO(dir => 'CAPTURE_DIR'); /* Get the latest replay for that capture */ SELECT max(id) INTO rep_id FROM dba_workload_replays WHERE capture_id = cap_id; dbms_output.put_line('max REP_ID is :' rep_id); end; / Max REP_ID is : Extracting the Replay report for a specific Workload set pagesize 0 long longchunksize 2000 select DBMS_WORKLOAD_REPLAY.REPORT(replay_id => 1, format => 'TEXT') from dual; 29

33 7.11 Sample Replay Report DB Replay Report for PEAK_REPLAY DB Name DB Id Release RAC Replay Name Replay Status ORCL NO PEAK_REPLAY COMPLETED Replay Information Information Replay Capture Name PEAK_REPLAY PEAK_PROCESSING_ Status COMPLETED COMPLETED Database Name ORCL1 ORCL Database Version Start Time :32: :24: End Time :32: :48: Duration 13 seconds 23 minutes 54 seconds Directory Object CAPTURE_DIR CAPTURE_DIR Directory Path c:\app\capture c:\app\capture Replay Options Option Name Value Synchronization SCN Connect Time 100% Think Time 150% Think Time Auto Correct FALSE 30

34 Number of WRC Clients 1 (1 Completed, 0 Running ) Replay Statistics Statistic Replay Capture DB Time seconds seconds Average Active Sessions User calls Network Time seconds Think Time seconds Exporting AWR data AWR data can be exported during the time of the capture as well as during the replays. The purpose of exporting the AWR data is primarily to compare At the end of each capture or replay, the relevant AWR snapshots for the period during which the capture or replay was executing can be exported and included in the captured or replayed data. The export log would look something like the below. To run the export, the command is as follows: BEGIN DBMS_WORKLOAD_REPLAY.EXPORT_AWR (capture_id => <CAPTURE ID>); END; /.. exported "SYS"."WRH$_SYSMETRIC_HISTORY" KB 100 rows.. exported "SYS"."WRH$_SQL_PLAN" MB rows.. exported "SYS"."WRH$_SYSMETRIC_SUMMARY" KB 296 rows.. exported "SYS"."WRH$_ENQUEUE_STAT" KB 174 rows.. exported "SYS"."WRH$_WAITCLASSMETRIC_HISTORY" KB 0 rows 31

35 These exported AWR snapshots can then be imported into another system into a specified and empty schema. The commands to export AWR reports during Capture and Replay are provided below EXEC DBMS_WORKLOAD_CAPTURE.EXPORT_AWR (capture_id => 12); EXEC DBMS_WORKLOAD_REPLAY.EXPORT_AWR (replay_id => 13); 32

36 7.13 Useful Views This section summarizes the views that you can display to monitor workload replay. You need DBA privileges to access these views. The DBA_WORKLOAD_CAPTURES view lists all the workload captures that have been captured in the current database. The DBA_WORKLOAD_FILTERS view lists all workload filters for workload captures defined in the current database. The DBA_WORKLOAD_REPLAYS view lists all the workload replays that have been replayed in the current database. The DBA_WORKLOAD_REPLAY_DIVERGENCE view enables you to view information about diverged calls, such as the replay identifier, stream identifier, and call counter. The DBA_WORKLOAD_REPLAY_FILTER_SET view lists all workload filters for workload replays defined in the current database. The DBA_WORKLOAD_CONNECTION_MAP view lists the connection mapping information for workload replay. The V$WORKLOAD_REPLAY_THREAD view lists information about all sessions from the replay clients. The following Oracle supplied PL/SQL API packages contain the subprograms to completely manage and administer the Capture and Replay processes DBMS_WORKLOAD_CAPTURE package DBMS_WORKLOAD_REPLAY package 8.0 Data Divergence on Replay Sample Simulation When the captured workload is replayed on the target system it is highly possible that some workload statements may fail due to DML, DDL or privilege related errors. These errors are called divergences and the effectiveness of the replay is measured by the % of the captured workload that could successfully be replayed on the target system. This allows the tester to make appropriate adjustments while reviewing the system performance results. Data Divergences could occur due to lack of privileges, synonyms on the objects that are referred to in the workload, missing objects and/or schemas, unique key violations, referential integrity violations, space related issues etc. 33

37 8.1 A Test case for demonstrating Data Divergence To understand a typical data divergence scenario, a sample test case has been devised based on a unique key constraint violation. In this test case, an employee table has been created with a unique key index and during replay we add a duplicate row into the table and that causes a unique index violation in the data divergence dictionary views. Same db and schema is used for capture and reply process during test case. Below are the high level steps outlined below 1) Create an employee table in the SCOTT schema. 2) Start the capture process and insert unique rows into the EMPLOYEE table 3) Once capture completes, remove inserted rows from employee table to start with a clean slate for the replay 4) Before the replay process is started, insert one duplicate row into the EMPLOYEE table. This gives rise to an unique key constraint violation which will then be populated in the divergence views 5) Check divergence in the Divergence view for the details of the SQL and bind variables that caused the divergence The following section provides a step by step approach to simulating and diagnosing a Data Divergence scenario during replay Create a sample table which will be used for demonstrating the data divergence scenario create table employee (employeeid number, employee_name varchar2(40)); Remove all previous capture metadata if desired. This is not a mandatory step. select id from DBA_WORKLOAD_CAPTURES; begin DBMS_WORKLOAD_CAPTURE.DELETE_CAPTURE_INFO(<Capture ID>); Add a end; filter on the SCOTT schema / 34

38 Add the required filters to restrict the workload that is captured as described below. BEGIN DBMS_WORKLOAD_CAPTURE.ADD_FILTER ( fname => 'user_scott', fattribute => 'USER', fvalue => 'SCOTT'); DBMS_WORKLOAD_CAPTURE.ADD_FILTER ( fname => 'instance_filter', fattribute => 'INSTANCE_NUMBER', fvalue => '1' ); END; / Check the current filters and their statuses SQL> select * from dba_workload_filters; TYPE ID STATUS NAME ATTRIBUTE VALUE CAPTURE 0 NEW USER_SCOTT USER SCOTT 35

39 Create a capture directory (if non-existent) and start the capture process SQL> create or replace directory CAPTURE_DIR as 'c:\app\capture'; Directory created. SQL> BEGIN DBMS_WORKLOAD_CAPTURE.START_CAPTURE ( name => 'PEAK_PROCESSING_2', dir => 'CAPTURE_DIR', default_action=>'exclude', duration => null, --means capture has to be ended manually auto_unrestrict => FALSE --if the above is TRUE all instances started up in RESTRICT mode will be made -- unrestricted when the capture ends ); END; / PL/SQL procedure successfully completed. Check the status of the capture process that has been started as described in the steps below set lines 200 col status format a15 col name format a15 col dbversion format a12 SELECT ID, NAME, DBID,DBNAME,DBVERSION, PARALLEL,STATUS, to_char(start_time,'dd/mm/yyyy hh24:mi'), to_char(end_time,'dd/mm/yyyy hh24:mi'),duration_secs FROM DBA_WORKLOAD_CAPTURES; ID NAME DBID DBNAME DBVERSION PAR STATUS TO_CHAR(START_TI TO_CHAR(END_TIME DURATION_SECS PEAK_PROCESSING_ ORCL NO IN PROGRESS 30/10/ :24 36

40 Insert data into employee table to generate work load in the SCOTT schema. insert into employee values (1,'Akash'); Insert into employee values (2,'Kehul'); insert into employee values (3,'Milan'); insert into employee values (4,'Ajay'); Complete the workload capture step BEGIN DBMS_WORKLOAD_CAPTURE.FINISH_CAPTURE (); END; / PL/SQL procedure successfully completed. Check the capture process status to verify it shows as COMPLETED set lines 200 col status format a15 col name format a15 col dbversion format a12 SELECT ID, NAME, DBID,DBNAME,DBVERSION, PARALLEL,STATUS, to_char(start_time,'dd/mm/yyyy hh24:mi'), to_char(end_time,'dd/mm/yyyy hh24:mi'),duration_secs FROM DBA_WORKLOAD_CAPTURES; ID NAME DBID DBNAME DBVERSION PAR STATUS TO_CHAR(START_TI TO_CHAR(END_TIME DURATION_SECS PEAK_PROCESSING_ ORCL NO COMPLETED 30/10/ :24 30/10/ : This concludes creation and capture of the workload which will be used to simulate the Data Divergence in the subsequent steps 37

41 8.2 Replaying the workload to cause a Data Divergence To simulate a data divergence, after the capture completes, we remove inserted data from employee table and add a unique key index and one dummy row which will get duplicated when the replay runs delete from employee; create unique index employee_empid on employee (employeeid); insert into employee values (1,'Raja'); Pre-process the captured workload BEGIN DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE (capture_dir => 'CAPTURE_DIR'); END; / PL/SQL procedure successfully completed. Initialize and calibrate the workload before replay BEGIN DBMS_WORKLOAD_REPLAY.INITIALIZE_REPLAY (replay_name => 'PEAK_REPLAY', replay_dir => 'CAPTURE_DIR'); END; / PL/SQL procedure successfully completed. $ wrc mode=calibrate replaydir=c:\app\capture 38

42 Prepare the Workload Replay BEGIN DBMS_WORKLOAD_REPLAY.PREPARE_REPLAY (SYNCHRONIZATION=>TRUE, --means transactions are replayed in SCN order CONNECT_TIME_SCALE=>100, --means same time between connections as in captured workload THINK_TIME_SCALE=>100, --means 1.5x the time between subsequent calls THINK_TIME_AUTO_CORRECT=>false, SCALE_UP_MULTIPLIER=>1); --means how many times to multiply the workload by during replay END; / PL/SQL procedure successfully completed. Check for the replay status as shown below. COL NAME FORMAT A15 COL STATUS FORMAT A20 SET LINES 200 SELECT ID,NAME, STATUS FROM DBA_WORKLOAD_REPLAYS; ID NAME STATUS PEAK_REPLAY COMPLETED 1 PEAK_REPLAY CANCELLED 13 PEAK_REPLAY CANCELLED 17 PEAK_REPLAY COMPLETED 34 PEAK_REPLAY COMPLETED 12 PEAK_REPLAY COMPLETED 43 PEAK_REPLAY PREPARE 21 PEAK_REPLAY COMPLETED Start the replay clients as described in the steps below. wrc system mode=replay replaydir=c:\app\capture Workload Replay Client: Release Production on Thu Aug 11 19:58: Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights 39 reserved. Password:*********** Wait for the replay to start (19:58:14)

43 Start the workload replay as shown below BEGIN DBMS_WORKLOAD_REPLAY.START_REPLAY (); END; / Then, we will see the below appearing in the output of the terminals where the wrc.exe was started wrc system mode=replay replaydir=c:\app\capture Workload Replay Client: Release Production on Sun Oct 30 11:00: Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. Password: Wait for the replay to start (11:00:21) Replay started (11:00:40) To check what SQLs are being currently replayed by the wrc clients, the following query can be executed. SQL> select sid,sql_id from v$session where program like 'wrc%'; SID SQL_ID wn3wubg7gjds 40

44 8.3 Checking for Data Divergences After the replay clients have completed, check for any Data Divergences that might have occurred as depicted below. SQL> set long 5000 SQL> select * from DBA_WORKLOAD_REPLAY_DIVERGENCE where replay_id=43; 41

45 8.4 Extract the Diverging SQL statement To extract the details of the diverging SQL statement and its associated bind variables, follow the steps described below. This can then be used to ascertain if the exclusion of the diverging statements causes an impact to the workload simulation. If it is important that the diverged statements must be executed, then the data must be corrected on the target system prior to re-running the workload replay select DBMS_WORKLOAD_REPLAY.GET_DIVERGING_STATEMENT(replay_id => 43, stream_id => stream_id, call_counter => 17) from DBA_WORKLOAD_REPLAY_DIVERGENCE where replay_id=43 and call_counter=17; Diverging Statement Details <replay_divergence_info> <sql_id>786cwqgg8m5j7</sql_id> <sql_text> insert into employee values (1,&apos;Akash&apos;) </sql_text> <full_sql_text> insert into employee values (1,&apos;Akash&apos;) </full_sql_text> <binds/> </replay_divergence_info> SQL> 42

46 9.0 Using Enterprise Manager for Real Application Testing Oracle Enterprise Manager is integrated with the APIs and can be used for performing Database Replays as an alternative to the command prompt. As shown below (for EM 11g), the Software and Support tab contains the links to the Real Application Testing suite. Clicking on the Database Replay takes us to the next screen shown further below. The Database Replay screen contains the links for performing the tasks related to Capturing Workloads, Preprocessing workloads and Replaying workloads. 43

47 In the preprocess workload screen, a list of available directory objects are displayed from which the desired directory, containing the captured data, is to be chosen The detailed steps for performing Capture and Replay using Enterprise Manager have intentionally been excluded from this document as the steps and screens are self explanatory 44