Mike Hughes Allstate Oracle Tech Lead, Oracle Performance DBA

Implementing Oracle Maximum Availability Architecture at Allstate Insurance, Using Oracle 10g RAC, ASM, Oracle Data Guard, Flashback Database, RMAN and Oracle Grid Control November 12, 2007 Mike Hughes Allstate Oracle Tech Lead, Oracle Performance DBA 1

Legacy ConversionOracle MAA at Allstate Insurance Co. Legacy Application Conversion (Next Gen) RTO/RPO HA Options and Considerations Oracle RAC Extended Distance Clusters Data Guard / Replication Options Chosen Architecture Outages and Upgrades ODP and HA Closing thoughts 2

Business Critical Claims Applications Consolidation and Integration 70+ claims apps scattered across several DB2 and SQL Server databases Multiple data and application integration opportunities 3

What is Next Gen? Legacy Claim system replacement Started in 2004 Phased approach across all Insurance Lines/States Over 500 people involved including 75 architects New Functionality - Operational Reporting NextGen 4

Next Gen Database Goals NextGen Near 24x7 availability Rolling upgrade capability Local and remote resiliency Protection from software corruption Remote copy of the database for DR 5

Operational Reporting What is Ops Reporting? Critical reporting component used for Next Gen Uses Crystal Reporting infrastructure Mainly used by front line management and leaders Current Activities Benchmarking and monitoring existing reports Tuning reporting queries and views Researching denormalization, partitioning, and performance options Reclassifying reports and where they should run Goals Data that is being reporting on should be within 20 minutes of current 6

RTO/RPO Oracle MAA at Allstate Insurance Co. Legacy Application Conversion (Next Gen) ** RTO/RPO HA Options and Considerations Oracle RAC Extended Distance Clusters Data Guard / Replication Options Chosen Architecture Outages and Upgrades ODP and HA Closing thoughts 7

RPO Recovery Point Objective RPO is the maximum acceptable level of data loss following an unplanned event. The RPO represents the point in time, prior to such an event, to which lost data can be recovered. Expressed backward in time Important consideration in resiliency and disaster recovery planning (DRP) Helps determine frequency of backups Helps DBAs choose optimal disaster recovery (DR) technology and procedures 8

RTO Recovery Time Objective RTO is the maximum tolerable length of time that your app can be down after a failure or disaster occurs. Potential revenue loss per unit of time during the disaster Disruption of normal operations Impacts the selection of recovery technology 9

HA Options Oracle MAA at Allstate Insurance Co. Legacy Application Conversion (Next Gen) RTO/RPO HA Options and Considerations Oracle RAC Extended Distance Clusters Data Guard / Replication Options Chosen Architecture Outages and Upgrades ODP and HA Closing thoughts 10

High Availability Options Oracle RAC Oracle RAC on extended distance clusters Oracle Data Guard 11

HA Option #1 - Oracle RAC Workload Manager (10g Services) Server 1 Server 2 Server 3 Server 4 Oracle ORA1 Oracle ORA1 Oracle ORA1 Oracle ORA1 Application A Application A Application B Application A Application B Application B Server 1 Fails Appl A Shared Storage Appl B If a server fails, the work running on that server moves to surviving servers 12

RAC Concepts Workload Management Workload Manager (10g Services) Server 1 Server 2 Server n Oracle ORA1 Oracle - ORA2 Oracle - ORAn Application A Application B Application A Application B... Application A Application B Appl A Shared Storage Appl B All servers have access to all databases (Read and Update) Any application, anywhere 13

RAC Cost / Benefits Workload Manager (10g Services) Server 1 Server 2 Server n Oracle ORA1 Application A Oracle - ORA2 Application A... Oracle - ORAn Application A Application B Application B Application B Appl A Shared Storage Appl B Benefits Scalability Horizontal Growth Redundancy / Failover Workload management Costs Overhead (CPU, elapsed time) Licensing Administration, patching Rolling upgrade capabilities 14

HA Option #2 - Extended Distance Clusters Location 1 Location 2 Server 1 Server 2 Dedicated Dark Fiber Server 3 Server 4 Oracle Oracle Max 25 km between sites Oracle Oracle Dictionary User Data Disk Mirroring Dictionary User Data 15

Extended Distance Clusters - Costs/Benefits Server 1 Oracle Dedicated Dark Fiber Max 25 km between sites Server 2 Oracle Dictionary User Data Disk Mirroring Dictionary User Data Benefits Inter-site HA Fully Active Active for complete resource utilization Costs Performance overhead due to network latency Direct cabling is very expensive Vulnerable to mirrored corruptions causing data loss and downtime 16

HA Option #3 - Oracle Data Guard Location 1 Location 2 Server 1 Server 2 LAN or WAN TCP/IP Server 3 Server 4 Oracle Oracle Data Guard Continuous Redo Shipment and Apply Oracle Oracle Dictionary User Data Dictionary User Data Redo data transmitted from primary to one or more standby locations Choice of synchronized physical or logical standby database Fast failover/switchover of production to standby site Standby databases support reporting, testing, and rolling upgrades 17

Oracle Data Guard Benefits Location 1 Server 1 Server 2 LAN or WAN TCP/IP Location 2 Server 3 Server 4 Oracle Oracle Data Guard Continuous Redo Shipment and Apply Oracle Oracle Dictionary User Data Dictionary User Data Benefits HA/DR without distance limits Optimum data protection Reduce unplanned downtime Reduce planned downtime Standby supports reporting No production overhead. Standard hardware components Costs Logical Standby database is not completely active, read-only access, no updates/inserts/deletes Physical Standby can not be used for up-to-date reporting (10g) Additional hardware cost for standby servers compared to extended distance cluster 18

Data Guard Asynchronous Redo Transport DML DML DML LOG Online Redo Log/s RLB RLB SCHEMAS Oracle Primary Site Redo data read/shipped from online logs Commit is immediately acknowledged to the application set init.ora parm log_archive_dest_n to LGWR ASYNC Transport services common to all Data Guard standbys Redo Stream NETWORK 19

Hot Swappable - Data Guard Physical Standby R U R U R U R U R Site 1... Redo Stream............... Apply Redo Apply Redo Redo Apply... Site 2 Pros Fast and Efficient Easy to setup and maintain No data type limitations Protection from hardware corruption Few if any issues Cons Not open for reads (until 11g) Limited protection from software corruption Not able to perform rolling upgrades (until 11g) 20

Active / Active(RO) - Data Guard Logical Standby R U R U R U R U R R R R R R Site 1... SQL Statements... Site 2 Redo Stream SQL Apply Pros Protection from software corruption Standby open for reads during apply Supports rolling database upgrades Can create additional indexes, mv s, and denormalized tables. Cons Workload profile impacts standby apply performance Logical copy, not physical replica Data type restrictions 21

Data Guard SQL Apply Process (logical standby) Redo From Primary Shared Pool Reader Undo Records Preparer LCR LCR LCR LCR.. Ungrouped LCRs Builder Transaction Groups Applier Coordinator Analyzer SQL DB Datafiles 22

Data Guard SQL Apply Bulk Updates TABLE: PERSON Col: STATE Col: ZIP IL 60631 IL 60514 IL 60412 IA 52001 UPDATE PERSEON SET STATE = AL WHERE STATE = IL ; COMMIT; Before: State IL, Key Mike After: State AL, Key Mike Before: State IL, Key Matt After: State AL Key Matt Before: State IL Key Mark After: State AL Key Mark Col: NM MIKE MATT MARK MARY Statements run on the logical Update PERSON SET STATE = AL WHERE NM = MIKE ; Update PERSON SET STATE = AL WHERE NM = MATT ; Update PERSON SET STATE = AL WHERE NM = MARK ; COMMIT; Redo Log 23

SQL Based Apply Misconception - Parallelism Sessions Updating Primary Database Network Sessions Updating Logical Standby Database Apply tends to serialize the work that took place on the primary If preserve commit order is true, data must be applied in the order it was processed on the primary 24

SQL Based Apply Can it Keep up? R U R U R U R U R R R R R R Site 1... SQL Statements... Site 2 Replication Know your application. What is the batch schedule? How much data do they update? How do they do their updates? (array processing, direct path load, commit scope, bulk updates) Make sure you understand the limitations of any SQL based replication tool these considerations are NOT limited to Data Guard SQL Apply 25

GRP Guaranteed Restore Point GRP Point #2 GRP Point #nn GRP Point #1 SCN # 200+ SCN # 100 SCN # 50 Flash Back Log / Archive Logs A guaranteed restore point enables you to flash the database back deterministically to the restore point. The guaranteed depends on sufficient space being available in the flash recovery area. Online DBA only function. 26

Chosen Architecture Oracle MAA at Allstate Insurance Co. Legacy Application Conversion (Next Gen) RTO/RPO HA Options and Considerations Oracle RAC Extended Distance Clusters Data Guard / Replication Options Chosen Architecture Outages and Upgrades ODP and HA Closing thoughts 27

Next Gen Selected Architecture MAA Oracle RAC, Data Guard, ASM, Flashback Database, RMAN & Enterprise Manager The 3 headed dragon RAC Primary database RAC Logical Standby RAC Physical Standby 28

Primary Database Oracle RAC Server 1 Server 2 Server 3 Server n Oracle Oracle Oracle... Oracle Dictionary ASM Shared Storage User Data Currently 2 nodes, upgrading to 4 nodes at end of 2007 12 dual core processors per node 24 total cores per node Extensive use of Oracle 10g Services to place work where we need it, when we need it 29

Oracle RAC Primary + Hot Swappable R U R U R U R U R Physical Standby Site 1...... Site 2 Primary Database Redo Stream Apply Redo Apply Redo Redo Apply Physical is the primary DR / switchover site flashback database used Standby is currently a 2 node RAC, will upgrade to 4 nodes Real-time apply on standby (but not open read-only until 11g) 12 dual processors per node, 24 total cores 30

Site 1 Hot Swappable and Active/Active (RO) R U R U R U R U R Primary...... Database Redo Stream Redo Stream SQL Statements Logical... Physical R R R R R R R R R Site 2 A logical standby resides on the same hardware as the physical standby Logical is open for read access, supports up-to-date Next Gen reporting Provides extra corruption prevention and rolling database upgrades. 31

Selected Architecture - Benefits Site 2 Site 1 Application Workload Servers Balance... Redo Transport Application Workload Servers Balance... Logical Redo Transport Application Servers... Physical Good Dragon!!!! Maximum data protection Protection from software corruption Transparent to most failures Near 24x7 availability Shortened recovery window Rolling upgrades minimizes planned downtime Role transitions are non-destructive Continuous access to data Continuous protection after failover to physical standby Offload reporting to DR system Support pre-prod test activity Able to use industry standard 32 hardware components

Outages and Upgrades Oracle MAA at Allstate Insurance Co. Legacy Application Conversion (Next-Gen) RTO/RPO HA Options and Considerations Oracle RAC Extended Distance Clusters Data Guard / Replication Options Chosen Architecture Outages and Upgrades ODP and HA Closing thoughts 33

Switchover / Failover to Recovery Site Site 1 New... Physical A switchover is a planned reversal of roles between production and standby DBs to avoid downtime during scheduled maintenance. Redo Stream SQL Statements... Logical Redo Stream New... Primary R R R R R R R R R R U R U R U R U R Site 2 34

Planned Switchover Steps 1 Blackout Monitoring Jobs (OEM) 2 Let logs / apply catch up 3 GRP all sites 4 Switchover from Primary to Physical 5 Switchover application 6 Open up Application for full processing 5 Site 1... New Physical U R U R U R 4 Redo Shipping Site 2... Logical Redo... New Primary U R U R U R OEM OEM 1 35

Switchover - Benefits Site 1... New Physical 4 Redo Shipping Site 2... Logical Redo... New Primary Practice makes perfect Allstate performs monthly switchovers for this application Planned switchover / switchback activities have occurred 15+ times to date and have become routine Not possible with legacy systems infrastructure 36

Rolling Upgrades Oracle RAC Data Guard Logical Standby 37

Oracle RAC Rolling Upgrades Workload Manager (10g Services) Server 1 Server 2 Server 3 Server 4 Oracle ORA1 Oracle ORA1 Oracle ORA1 Oracle ORA1 Application A Application A Application B Application A Application B Application B Server 1: Patch Appl A ASM Shared Storage Appl B Each server is patched one at a time. Oracle 10g Services are used to move apps around the cluster during the upgrade Rolling upgrades are available only for patches certified by Oracle to be eligible for rolling upgrades 38

RAC Rolling Upgrades - Patches that apply Server 1 Server 2 Server 3 Server 4 Oracle ORA1 Application A Application B Server 1 Fails Workload Manager (10g Services) Oracle ORA1 Application A Oracle ORA1 Application B Oracle ORA1 Application A Application B Appl A ASM Shared Storage Appl B Patches that do not touch the data dictionary Patches not related to RAC internode communication Patches related to client-side tools such as SQL*Plus, Utilities Patches that do not change shared database resources such as database headers, control files, etc. RAC Rolling upgrades applicable to one -off patching only, full patch sets do not apply 39

RAC Rolling Upgrades Actual Benefits Workload Manager (10g Services) Server 1 Server 2 Server 3 Server 4 Oracle ORA1 Application A Application B Server 1 Fails Oracle ORA1 Application A Oracle ORA1 Application B Oracle ORA1 Application A Application B Appl A ASM Shared Storage Appl B Little or no downtime. Only the node being patched needs to be down. Have been performed at Allstate on several occasions to apply critical one off patches that were causing application issues. Previous Allstate Claim system architectures did not take advantage of this feature. 40

Rolling Upgrades Using SQL Apply Protection mode must be maximum availability or maximum performance (see protection_mode in v$database) LOG_ARCHIVE_DEST_n must be set to OPTIONAL The COMPATIBLE init.ora parm must match the software release prior to the upgrade 41

Rolling Upgrades - SQL Apply 1 Upgrade the Logical Standby to version n+1 - Defer redo transport during upgrade 2 Re-start transport and apply on Logical Standby - Redo accumulated at primary will now start processing and automatically resync logical at n+1 with the primary still at n Version n Site 1... U R U R U 3 Switchover to the logical promotes it to being the new primary running at version n+1 - switchover time is the only downtime required for upgrade Primary 3 2 Redo Stream Redo Stream Version n+1 Site 2 1... Logical... Physical Version n 42

Rolling Upgrades - SQL Apply (continued) 4 Reconnect the application to the new primary - Redo will can not be transmitted to a lower Oracle version - New transactions running on primary running at n+1 are unprotected at this time Site 1 Version n... New Logical Redo Shipping Site 2 Version n+1... New Primary U R U R U Redo Stream 4... Physical Version n 43

Rolling Upgrades - SQL Apply (continued) 6 Upgrade Physical (binaries only) 7 Upgrade the old primary. Start SQL Apply on new Logical 8 -Redo will start to transmit and process -Primary is unprotected at this time. Site 1... New Logical 7 version N+1 8 Redo Stream Redo Shipping Site 2... New Primary U R U R U This is optional, but once complete, most people switch back so everything is in its original location!... Physical 6 version N+1 44

Rolling Upgrades Oracle RAC vs Data Guard Switchover RAC Rolling Switchover ** qualifying one off patches. ** OS Upgrades ** Major Oracle releases and full patch sets ** Denotes already done at Allstate. 45

Rolling Upgrades Greatly reduces down time for mission critical applications compared to previous architecture Multiple options available RAC Rolling for OS and one off patches SQL Apply major patch sets Use the appropriate tool for the appropriate job Practice and Test thoroughly 46

Unplanned Outages 47

Unplanned Failover Steps (Physical) 1 If possible.. Manually move remaining archive logs to the physical copy 2 Manually apply archive logs 3 Failover from Primary to Physical Site 1 4 If needed. Synch up logical with the new primary... 3 Site 2... Logical Redo... New Primary 4 Logical re-instate may require flashback 5 Fail application over 6 Open up Application for full processing 48

Unplanned Failover Steps (Logical) 1 If possible.. Manually move remaining archive logs to the logical copy 2 Manually force Logical to apply data from standby redo log 3 Failover from Primary to Logical 4 Fail application over 5 Open up Application for full processing Site 1 Corruption... If the original primary and physical standby can be reinstated using flashback database, Data Guard will resynchronize them with the new primary without requiring a restore from backup! 4 Site 2... New Primary... Physical 49

ODP and HA Oracle MAA at Allstate Insurance Co. Legacy Application Conversion (Next-Gen) RTO/RPO HA Options and Considerations Oracle RAC Extended Distance Clusters Data Guard / Replication Options Chosen Architecture Outages and Upgrades ODP and HA Closing thoughts 50

ODP Best Practices Workload Manager (10g Services) Server 1 Server 2 Server 3 Server 4 Oracle ORA1 Application A Application B Server 1: Outage Oracle ORA1 Application A Oracle ORA1 Application B Oracle ORA1 Application A Application B Appl A Shared Storage Appl B Connections to the failing member need to be reconnected Applications with persistent connections need to have error handling on the connection level Persistent connections are reused (pooled connections) Keep the min pool size small to avoid connection storms Leverage the connection lifetime, incr and decr pool size parameter(s) At a minimum, use 10g DTP services for two phase commit 51

Closing Thoughts Oracle MAA at Allstate Insurance Co. Legacy Application Conversion (Next-Gen) RTO/RPO HA Options and Considerations Oracle RAC Extended Distance Clusters Data Guard / Replication Options Chosen Architecture Outages and Upgrades ODP and HA Closing thoughts 52

Know your application MAA Lessons Learned Let RTO/RPO drive your HA needs Make sure your MAA solution fits your needs Be aware of limitations to SQL based replication Exercise your DR / switchover plan frequently Incorporate proper error handling on your connection objects 53

References Oracle Database High Availability http://www.oracle.com/technology/deploy/availability/index.html Oracle Data Guard Concepts and Administration, Oracle 10g Release 2 http://download.oracle.com/docs/cd/b19306_01/server.102/b14239/concepts.htm Oracle Database High Availability Best Practices, Oracle 10g Release 2 http://download.oracle.com/docs/cd/b19306_01/server.102/b25159/toc.htm Oracle Maximum Availability Architecture Best Practices http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm 54