Oracle Data Guard Tips and Tricks Direct from Oracle Development

Transcription

1 1

2 2

3 <Insert Picture Here> Oracle Data Guard Tips and Tricks Direct from Oracle Development Larry M. Carpenter Oracle Server Technologies Distinguished Product Manager John Smiley Amazon.com Senior Database Engineer

4 Are You Reluctant or Unable to? Use Automatic Storage Management (ASM)? Implement a Flash Recovery Area? Use RMAN on your standby? Schedule Jobs on a Logical standby? Stop doing no-logging operations? Use Flashback Database? Execute a Failover? Allow Data Guard to Automatically Failover? Perform a Rolling Upgrade? Open your Physical standby Read Write? 4

5 These fall into 3 areas Ease of Management Protecting Data Using your Standby Databases 5

6 Data Guard Tips & Tricks Ease of Management Use ASM with your databases Incorporate a Flash Recovery Area Using RMAN to create and backup your standbys Use jobs on a Logical standby 6

7 Getting to ASM ASM provides a vertically integrated file system and volume manager directly in the Oracle kernel Significantly less work to provision database storage Higher level of availability Elimination of the expense, installation, and maintenance of specialized storage products Unique capabilities for database applications 7

8 But converting to ASM means Production downtime! True, if you are converting your Production database. Why not convert your Physical Standby first? If you already have a Physical Standby Migrate it to ASM If you do not, shame on you. Create a Physical standby with RMAN and use ASM. Run in this mixed environment until you are satisfied Execute a switchover and move Production. Migrate the Physical standby (original Production) to ASM. Switchover again to return to your original setup Still some downtime for the switchovers Tuned to less than 1 minute and is a planned event. 8

9 So how hard is it? Database Backup and Recovery User's Guide Chapter 26 Performing ASM Data Migration /rcmasmmi.htm#i Procedures well documented in MAA Oracle Database 10g Best Practices. Minimal Downtime Migration to ASM f/maa_wp_10gr2_asmmigrationwithdg.pdf Migration to Automatic Storage Management (ASM) f/maa_wp_10gasmmigration.pdf 9

10 Start using the Flash Recovery Area Centralized disk location for all recovery files. Ensures that the database is completely recoverable by using files stored in the recovery area Current control file Online redo logs Archived redo logs Flashback logs Control file autobackups Datafile and control file copies Backup pieces 10

11 Why use it in my Data Guard setup? Simplifies the management of archived redo log files You don t have to worry about cleaning up log files that are no longer necessary to your standby databases. More in the RMAN section Required for Flashback Database Which can be used for many things with Data Guard Discussed in a few minutes. 11

12 Where do I use it in my Data Guard Configuration? Primary Physical Standby Logical Standby In Oracle Database 10g you may only use the Flash Recovery Area to hold the archive logs generated by your Logical standby. The Redo coming from the Primary must be stored in a separate directory and will be maintained by SQL Apply. No longer necessary in Oracle Database 11g. Migrate slowly if you wish. Put it on your Physical standby first. Then add it to your Primary Does not require downtime. 12

13 Tuning the Flash Recovery Area Use a fast file system for your flash recovery area, Preferably in ASM. Test and configure for peak IO bandwidth. (Peak IO bandwidth/(#spindles * avg spindle throughput) ) Consider one archive copy, one redo member, flashback logs Configure sufficient spindles for 3 X peak redo rate EG 3 x 10 MB/sec = 30MB/sec. Average spindle throughput is 5 MB/sec then 30/5= a minimum of 6 spindles. If using it for your backups too then you will need to adjust for the amount of backup redo generation as well. For large databases, set LOG_BUFFER to at least 8 MB. Monitor database wait events for any of the RVWR events 13

14 So, how do I turn it on? Set DB_RECOVERY_FILE_DEST_SIZE parameter Specifies the maximum total bytes to be used. How big? Size of a copy of database + Size of an incremental backup + Size of (n+1) days of archived redo logs + Size of (y+1) days of foreign archived redo logs (for Logical) + Size of control file + Size of an online redo log member * number of log groups + Size of flashback logs (Retention Target * Redo Generated~) Set DB_RECOVERY_FILE_DEST parameter Location of the flash recovery area. Can be a directory, file system, or an ASM disk group. It cannot be a raw file system. 14

15 What about my old archiving? Change it to use the Flash Recovery Area LOG_ARCHIVE_DEST_1 = 'LOCATION = /disk2/archive VALID_FOR=(ALL_LOGFILES,ALL_ROLES) To LOG_ARCHIVE_DEST_1 = 'LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) Remember, in Oracle Database 10g, if the database is a Logical standby database you must change ALL_LOGFILES to ONLINE_LOGFILES No longer required in Oracle Database 11g 15

16 Running jobs on your Logical Standby There is basic support for DBMS_JOB. Job execution is suspended on a Logical standby database Jobs submitted on the Primary database are replicated to the Logical standby database. In the event of a switchover or failover, jobs scheduled on the original Primary database will automatically begin running on the new Primary database. Need more support for running jobs on a Logical standby database. DBMS_SCHEDULER enhanced to include a Role Jobs will on execute if the current Role of the database matches the Role of the job. 16

17 Enter the new and improved DBMS_SCHEDULER support New attribute of a scheduler job called database_role Contents match the database_role attribute of V$DATABASE. When a scheduler job is created, it defaults to the local role A job created on the standby defaults to a database_role of LOGICAL STANDBY. Job scheduler executes only jobs specific to the current role. On switchover or failover, the scheduler automatically switches to running jobs specific to the new role. Scheduler jobs are not replicated to the standby. Activate existing jobs with DBMS_SCHEDULER.Set_Attribute Clone jobs that should run in both roles DBA_SCHEDULER_JOB_ROLES view shows which jobs are specific to which role. Set the database guard to STANDBY. 17

18 Getting more out of RMAN with Data Guard Create Standbys without temporary extra storage Move the backups to the standby Use the Flash Recovery area and RMAN to manage your archive logs automatically 18

19 Standby Creation 11g New FROM ACTIVE DATABASE clause Performs the backup, transport to the standby site and restore over the network using parallel channels. No interim storage needed for a copy of the backup file. Simple steps performed on the Standby server 1. Do a software-only install on the standby database system. 2. Setup OracleNet. Create a static listener entry for the standby, start the listener. Create Oracle Net connect descriptors for the Standby at the Primary site and for the Primary at the Standby site. 3. Create an init.ora file with only the DBNAME in it. 4. Create a password file with the same SYS password. 5. Create any necessary directories. 6. Startup nomount the standby instance. 7. Execute the command on the next slide. 19

20 Standby Creation 11g Across the network! rman connect target connect auxiliary sys/oracle; run { allocate channel prmy1 type disk; allocate channel prmy2 type disk; allocate channel prmy3 type disk; allocate channel prmy4 type disk; allocate auxiliary channel stby type disk; duplicate target database for standby from active database spfile parameter_value_convert 'Chicago','Boston' set db_unique_name='boston' set db_file_name_convert='/chicago/','/boston/' set log_file_name_convert='/chicago/','/boston/' set control_files='/oracle/oradata/boston/control.ctl' } 20

21 RMAN and Physical standby RMAN now tracks filenames for all database files in a Data Guard environment using the catalog. RMAN commands now behave transparently across different physical databases in the Data Guard environment Back up a tablespace on a physical standby database and restore and recover it on the primary database and vice versa. Backups of standby control files and non-standby control files are interchangeable The recovery catalog tracks the files in a Data Guard setup Associates every database file or backup file with a DB_UNIQUE_NAME See Chapter 11 "Using RMAN to Back Up and Restore Files" of the Data Guard Concepts and Administration manual. 21

22 Move ALL backups to your Physical Standby database With the Active Data Guard option Block Change Tracking is now possible on your Physical standbys Now your incremental RMAN backups can go as fast on your standby as on your Primary. No need from a performance point of view to run any backups on your Primary database. 22

23 Automate Archive Log Management Use RMAN to setup the Archive log retention policy Primary database retention policy (Note ) CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY On the Physical Standby where the backups are being done. CONFIGURE ARCHIVELOG DELETION POLICY TO NONE On all other standby databases CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY Requires a Flash Recovery Area on all databases. After Switchover or Failover, re-execute the proper configure commands again as appropriate 23

24 Data Guard Tips & Tricks Protecting your data - always Implementing Flashback Database On your Standby On your Primary Logging all transactions on the Primary 24

25 Easing into Flashback Database Enables you to rewind an Oracle database to a previous time to correct problems caused by logical data corruptions or user errors. 25

26 Why use Flashback Database? Enables easy correction of User errors without impacting you re the standby lagging behind. Rewind the standby to fix user errors Failovers can happen without affecting RTO Enables a failed Primary to become a standby after a failover without re-copying all the files. Next topic Enables Snapshot standby Opening your Physical standby for read and write More on that later 26

27 What about performance impact? Try it out on your standby first, just like ASM. ALTER DATABASE FLASHBACK ON; Database must be in a Mounted state A Physical standby is usually just mounted. If you want real world measurements Enable Flashback Database on your physical standby Convert it to a Read Write Standby (Snapshot standby) Run your workload (Real Application Testing) Revert to a Physical standby Requires a Flash Recovery Area If necessary, tune the Flash Recovery Area, again 27

28 What about my Nologging jobs? Many People are still worried about the performance hit if they turn logging on. Most have not tried it lately. Why do something that used to hurt? No logging is not turning archive logging off. Turn archive logging off and you cannot have a standby. And it doesn t really help anyway since the redo still gets generated. Using no logging is possible with a Data Guard standby but you put your DR strategy at risk. 28

29 Transactions How Does Nologging affect Data Guard? Oracle Net Physical/Logical Standby Database LGWR Sync LNS RFS MRP - physical LSP - logical Primary Database Online Redo Logs ARCH Async Arch & Gaps Standby Redo Logs ARCH Transform Redo to SQL for Logical Standby Queries, Reports, Testing, Backups Archived Redo Logs Archived Redo Logs 29

30 To Log or Not to Log? That is the Question! Results from a very skeptical 10g Release 2 customer Generate up to 1TB of REDO a day. Difference between non-logging and logging must be < 60 minutes. Their test results? 10,000,000 row data load With no-logging 5:17 elapsed time With logging and no tuning 8:22 elapsed time With logging and tuning 5:44 elapsed time They are believers today and are implementing Data Guard 30

31 So what did they do? Created a separate ASM REDO disk group. Optimized disk I/O. Measured and calculated IOPS Changed Online Redo Log configuration Moved from 8 logs x 2 groups at 4GB to 8 logs x 2 groups at 6GB. Checked SGA memory usage Made sure memory was available. Checked read/write disk cache Adjusted to 90-95% write cache. Checked AVG_READ times Created partitions to improve performance. 31

32 If you are still not convinced Make sure you read Note The Gains and Pains of Nologging Operations If you perform no-logging operations on your Primary, You: Must ALTER DATABASE NO FORCE LOGGING; Should ALTER TABLESPACE <name> FORCE LOGGING; on all but the one tablespace where you will keep the nologging tables. Must keep a close eye on the database to make sure nobody puts important data in the nologging tablespace. Have to fix the unrecoverable objects at the standby when you switchover or failover. 32

33 Data Guard Tips & Tricks Using your standby database Getting friendly with Failover Allowing Data Guard to manage failover Perform a Rolling Upgrade Even if you don t have a Logical standby Move your testing to your Physical standby 33

34 Executing a Failover Failover is what you do when something bad happens You lost the Primary site, disks, database etc. Unlike Switchover (which is something you plan to do) it is more of a one way operation, right? Not really anymore! If you were listening to me in the past few slides you ll see why this is not true. Recreating the old Primary was always a big factor. 34

35 Flashback Database & Failover Post Failover Primary database re-instantiation Recreating the primary database after a failover no longer problematic with Flashback Database. All the datafiles no longer have to be re-copied from the new primary system back to the original primary system. Requires that Flashback Database was enabled on the original Primary database before the failover. There is no longer a reason to be wary of Failover Getting back up and running as fast as possible should be your main concern. 35

36 Reinstatement to a Physical Standby After a Failover to the Physical Standby database Get the failover SCN from the new primary database. SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE; 36

37 Reinstatement to a Physical Standby Continued Start up the old Primary in mount state. Flash back the old primary to the SCN retrieved from the new Primary SQL> FLASHBACK DATABASE TO SCN <standby_became_primary_scn>; Convert the Primary to a Physical Standby database! SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY; 37

38 Reinstatement to a Physical Standby Continued Shutdown and restart the new standby. Ensure REDO is being transferred from the new Primary to the new Standby Start the Apply on the new Standby database Ensure that REDO is being applied correctly on the new Standby database Similar procedure for Logical standby databases One command if you are using the Broker One click if you are using Grid Control. 38

39 Reinstatement to a Physical Standby Actual steps on the previous 3 slides and in the manual I ve hidden the slides for the sake of time. Similar procedure for Physical and Logical standby databases One command if you are using the Broker One click if you are using Grid Control. 39

40 40

41 Best Practices Failover Enable Flashback Database now, not later Use Data Guard real-time apply Use the Data Guard Broker Otherwise For Real Application Clusters SHUTDOWN ABORT on all secondary RAC instances on the standby database prior to performing a failover. Tune the Apply OPEN the database from the MOUNTED state No bounce of the standby If Oracle Database 10g Physical standby was opened Read Only, bounce it first No longer necessary in Oracle Database 11g 41

42 How about doing it automatically? If I have convinced you that failover is not all bad anymore then the next leap of faith is to allow Data Guard to automatically failover for you. Even the most adamant anti-automatic failover customers are coming around. Fast-Start Failover Introduced in Oracle Database 10g Release 2. Fast, Simple, Secure. Configurable to allow Zero Data Loss or User Defined Limits No longer just for the nearby SYNC user Across the WAN with ASYNC transport in 11g. But you don t have to believe me. 42

43 Data Guard Fast-Start Failover At Amazon.com November 14, 2007 John Smiley 43

44 Goals for Fast-Start Failover Reduce time to failover Simplify failover process Set standards for Fast-Start Failover deployment Reduce costs 44

45 Implementation Observer Multiple hosts Wallet for SYS password Wrapper to start/restart automatically Easily relocated Monitor Incorporated into existing monitoring infrastructure Alerts if FSFO readiness is compromised Warns if flashback database history is insufficient to reinstate DB_ROLE_CHANGE trigger Directory service update Configuration verifier Autostop Script 45

46 Failover Only the observer should initiate failover Database failover is not enough need client failover Re-establish connectivity Service relocation or Change directory service to point to new primary location Notify clients Reconnect / retry 46

47 Experience Application failover time = database failover time + directory service propagation time Reliable Always have a good primary after a failover No split brain conditions Data integrity maintained Fast, automatic standby reinstatement Flashback restore time depends upon the number of distinct blocks changed during the 30 minutes prior to failover Much faster than RMAN or other restore methods Recovery time depends upon amount of redo generated between the restore SCN and the standby_became_primary_scn 47

48 Experience Flashback database storage requirements v$flashback_database_stat estimate can be inaccurate Flashback database retention target Number of distinct blocks changed Measure peak generation rate for the required retention period Maximum Availability Mode Increased commit latency Small percentage increase in typical application transaction time Decreased throughput can be regained by increasing parallelism DML pause for LGWR NetTimeout duration DML may see short delay when mounting/dismounting the standby 48

49 Experience No application changes required Use cases Instance failure Host failure Network failure Fleet migration Test! Only reliable if the configuration is correct Verify configuration and procedures by performing failovers 49

50 Results Failover Time 60 Failover Time (min) Resolve Respond Identify 0 Pre-DG DG w/o FSFO FSFO 50

51 Other Possible Uses Use inexpensive commodity hardware Less durable Compensate with more standbys Overall TCO lowered 11g custom failover conditions Hung primary Brownouts 11g Maximum Performance Mode FSFO Asynchronous redo transfer No impact to commit/transaction latency Disaster recovery over long distances 51

52 Enhanced Fast-Start Failover Automatic failover for Maximum Performance Mode Data Guard configurations using ASYNC redo transport Maximum allowable data loss is user configurable via Data Guard Broker property: FastStartFailoverLagLimit Default setting = 30 seconds, minimum threshold = 10 seconds) High Availability for Fast-Start Failover Observer Automatically restart the Data Guard Observer on a second host if the primary Observer host fails Available with Grid Control 52

53 Configurable Fast-Start Failover Immediate automatic failover for user-configurable health conditions ENABLE FAST_START FAILOVER [CONDITION <value>]; Condition examples: Datafile Offline Corrupted Controlfile Corrupted Dictionary Inaccessible Logfile Stuck Archiver Any explicit ORA-xyz error Apps can request fast-start failover using API DBMS_DG.INITIATE_FS_FAILOVER 53

54 Oracle Rolling Upgrades for Physical People Advantages of using SQL Apply rolling upgrade Production database will incur very little downtime. The overall downtime can be as little as the time it takes to perform a switchover. Eliminate application downtime due to PL/SQL recompilation. Validate the upgraded database release without affecting the production database. Now, not just for Logical Standby customers

55 Rolling Upgrades and Physical Standby Physical Logical Upgrade Physical Transient Logical Standby Execute rolling database upgrades using a physical standby database Temporarily convert physical standby to logical to perform the upgrade Potential impact of SQL Apply data type restrictions limited to shorter upgrade window When upgrade is complete revert to physical standby No need for separate logical standby 55

56 On Original Primary Enable Flashback Database. If not already enabled. Create a guaranteed restore point create restore point pre_upgrade guarantee flashback database; Create a physical standby control file: alter database create physical standby controlfile as '/tmp/control.phys' reuse; 56

57 Convert the Original Physical standby database Convert it into a logical standby database alter database recover managed standby database cancel; alter database recover to logical standby keep identity; alter database open; execute dbms_ logstdby.apply_set( LOG_AUTO_DELETE, false ); alter database start logical standby apply immediate; Upgrade this new logical standby database. Catch up with the primary database Perform the switchover Move Production to the newly upgraded database. 57

58 On the Original Primary Flashback to the guaranteed restore point shutdown immediate; startup mount flashback database to restore point pre_upgrade; Restore the standby control file from first step shutdown immediate; RMAN> startup nomount RMAN> restore controlfile from '/tmp/control.phys'; RMAN> shutdown 58

59 On the Original Primary Continued Upgrade Oracle software Switch to upgraded home Bring up physical standby startup mount; alter database recover managed standby database using current logfile disconnect; Allow new Physical standby database to catch up with new Primary If desired, perform a switchover and move Production back to original site. 59

60 What is Snapshot Standby? Updates Updates Queries Primary Database Snapshot Physical Standby Database Preserves zero data loss continuous redo transport while open read-write Truly leverages standby database and DR hardware for multiple purposes Similar to storage snapshots, but provides DR at the same time and uses single copy of storage 60

61 But What about Oracle Database 10g? 10.2 Steps Required Standby > alter database recover managed standby database cancel; > create restore point before_lt guarantee flashback database; Primary > alter system archive log current; > alter system set log_archive_dest_state_2=defer; Standby > alter database activate standby database; > startup mount force; > alter database set standby database to maximize performance; > alter system set log_archive_dest_state_2=defer; > alter database open; PERFORM TESTING, ARCHIVE LOGS NOT SHIPPED > startup mount force; > flashback database to restore point before_lt; > alter database convert to physical standby; > startup mount force; > alter database recover managed standby database disconnect from session; Primary > Alter system set log_archive_dest_state_2=enable 11.1 Steps Required Standby > alter database convert to snapshot standby; PERFORM TESTING, ARCHIVE LOGS CONTINUE TO BE SHIPPED > alter database convert to physical standby; Can you work around this? 61

62 Maintaining Protection (RPO) in 10.2 while in Snapshot mode Create an Archive Log Repository (ALR) Note Data Guard Redo Log Repository Example This will ensure that the redo is being shipped and stored at the standby site while the standby is open read write and not able to receive the redo. Put the Archive Log Repository in place before you begin the Snapshot process. When the Physical standby has been returned Use these logs to catch it up with the Primary Or use an incremental backup of the Primary This is not required in Oracle Database 11g 62

63 Basic Steps for an ALR 1. Create a standby controlfile 2. Create a pfile from the primary 3. Copy the standby controlfile and pfile to the ALR site 4. Make any changes to the ALR pfile 5. Create the Flash Recovery Area and Dump directories 6. Copy the password file from the Primary database to the ALR 7. Setup the Oracle Network connection descriptors 8. Restore the standby controlfile to the ALR 9. Add standby redo logs to the ALR 10.Setup Primary to Transmit to the ALR 11.Verify the ALR is receiving logs 63

64 So, Are You Still Reluctant to? Use Automatic Storage Management (ASM)? Implement a Flash Recovery Area? Stop doing no-logging operations? Use Flashback Database? Execute a Failover? Allow Data Guard to Automatically Failover? Perform a Rolling Upgrade? Schedule Jobs on a Logical standby? Use RMAN on your standby? Open your Physical standby Read Write? 64

65 Come on in, the water s fine! 65

66 Database HA Sessions From Oracle Development Monday, Nov 12 S The Fastest and the Most Cost-Effective Backup for Oracle Database: What s New in Oracle Secure Backup 10.2, 11:00 am - 12:00 pm, Moscone South 304 S Oracle Database 11g: Next-Generation High Availability, 12:30-1:30 pm, Moscone South 103 S Implementing Oracle Maximum Availability Architecture (MAA) at Allstate Insurance Using Oracle 10g RAC, ASM, Oracle Data Guard and Oracle Grid Control, 3:15-4:15 pm, Moscone South 304 S Oracle Database 11g Data Repair Technologies: Comprehensive, Intelligent Recovery, 4:45-5:45 pm, Moscone South 304 Tuesday, Nov 13 S Maximum Availability Architecture Best Practices: Oracle E-Business Suite 12, 12:15-1:15 pm, Marriott Salon 10 & 11 Wednesday, Nov 14 S What s New in Oracle Data Guard 11g: Revolutionizing Data Protection and Availability, 9:45-10:45 am, Moscone South

67 Database HA Sessions From Oracle Development Wednesday, Nov 14 S Backup and Recovery Best Practices for Very Large Databases (VLDB), 11:15 am - 12:15 pm, Moscone South 304 S Oracle Active Data Guard: How to Utilize Your Standby Databases for Production Workload - What They Didn t Print in the Manuals, 3:00-4:00 pm, Moscone South 304 S Oracle Data Guard Tips and Tricks: Direct From Oracle Development, 4:30-5:30 pm, Moscone South 102 Thursday, Nov 15 S Oracle Streams Replication and Advanced Queuing (AQ): What's New in Oracle Database 11g, 8:30-9:30 am, Moscone South 304 S Best Practices for Implementing Replication with Oracle Streams in Oracle Database 10g and 11g, 10:00-11:00 am, Moscone South 304 S Maximum Availability Architecture (MAA) Best Practices: Online Patching, Rolling Upgrades and Planned Maintenance with Minimal Downtime with Oracle Database, 11:30 am - 12:30 pm, Moscone South 104 S Maximum Availability Architecture (MAA) Best Practices for Siebel 8.0, 2:30 pm - 3:30 pm, Marriott Salon 10 & 11 67

68 Database HA Demos From Oracle Development Monday, Nov 12 Thursday, Nov 15 Oracle DEMOgrounds, Moscone West Oracle Active Data Guard Oracle Streams: Replication and Advanced Queuing Oracle Secure Backup Recovery Manager (RMAN) and Flashback Technologies Maximum Availability Architecture 68

69 For More Information search.oracle.com or oracle.com 69

70 70

71 71