Maximize Availability on Private Clouds

Transcription

1 Maximize Availability on Private Clouds

2 No Time for Downtime? 2

3 Twenty Two Hour Outage Popular E-Commerce Site From Barron s Online their Web site has been down for hours. Around noon Pacific time, the site said it would be back up at 1 P.M. Pacific; a few minutes ago it said the site will be up again at 4 p.m. And now there is no time reference at all. The site now says: The web site is temporarily unavailable. We apologize for any inconvenience this causes you. Please visit us again soon. You can contact Customer Service at xxx-xxxx Note: The company s stock price fell 6% on the same day 3

4 Five Day Outage US State Government Storage Area Network (SAN) failure Physical corruption was SAN-mirrored to DR site DR site also unusable Data loss, lengthy recovery time Twenty different state agencies impacted Knocked 500 servers offline 45,000 person backlog for license renewals at the Department of Motor Vehicles 4

5 Ten Hour Outage National Grocery Chain Storage admin accidentally deleted online redo log files on a primary Oracle RAC cluster Caused Oracle Database to crash SAN mirroring was used to synchronize the DR site Deletion was mirrored to the DR copy Oracle Database at DR site unable to start Data loss and downtime Impacted critical distribution system serving 2000 supermarkets 5

6 Retail Brokerage Firm Downtime Incident Reported through Oracle Support ORA Corrupted file header. This could be caused due to missed read or write or hardware problem or process external to oracle overwriting the information in file header. 6

7 Data Protection & High Availability (HA) Universal Requirements for Today s Business Site outage due to large-scale disasters Fire, floods, hurricanes, earthquakes... Local outages that occur more frequently Faulty system components Data corruptions Backup/recovery of bad data Wrong batch job Operator errors Planned maintenance Bad HW/SW installations, upgrades 7

8 Challenges with Traditional HA Limited Protection, Idle Redundancy Production Server IdleFailover Server 3 rd Party Cold Failover Cluster 3 rd Party File Systems, Volume Managers 3 rd Party Backup & Tape Management Software Proprietary Storage Array 8

9 Challenges with Traditional HA Limited Protection, Idle Redundancy Production Server IdleFailover Server Impossible to know if it all works until you try to failover 3 rd Party Backup & Tape Management Software 3 rd Party Cold Failover Cluster Proprietary Storage Array 3 rd Party File Systems, Volume Managers Storage Remote-Mirroring Idle Disaster Recovery Systems Matching Proprietary Storage Array 9

10 Challenges with Traditional HA Limited Protection, Idle Redundancy Production Server IdleFailover Server Impossible to know if it all works until you try to failover 3 rd Party Backup & Tape Management Software 3 rd Party Cold Failover Cluster Proprietary Storage Array 3 rd Party File Systems, Volume Managers Storage Remote-Mirroring Idle Disaster Recovery Systems Matching Proprietary Storage Array 10

11 Challenges with Traditional HA Limited Protection, Idle Redundancy Production Server IdleFailover Server Impossible to know if it all works until you try to failover 3 rd Party Backup & Tape Management Software 3 rd Party Cold Failover Cluster Proprietary Storage Array 3 rd Party File Systems, Volume Managers Storage Remote-Mirroring Idle Disaster Recovery Systems Matching Proprietary Storage Array 11

12 Remember the Brokerage Firm? Actual Service Request: Application Down ORA Corrupted file header. This could be caused due to missed read or write or hardware problem or process external to oracle overwriting the information in file header. 12

13 Positive Outcome With Oracle Maximum Availability Architecture They had a Data Guard standby database Corruptions isolated to the primary database Zero data loss failover over to standby database Resumed operation in minutes Investigated & resolved h/w problems with no impact 13

14 Evolving Computing Environments Moving to Cloud Computing 14

15 Maximum Availability Architecture (MAA) Architecture and Best Practices Oracle-optimized architecture to ensure high availability (HA) is maintained for the Oracle stack despite planned and unplanned outages An extensive series of technical best practices on Exadata Database Machine Oracle Database Oracle Fusion Middleware Oracle Applications Grid Control Partner solutions Ensuring the most optimal HA using these solutions Ref. 15

16 Evolution Towards MAA Start with a Single Server Database Instance Database Storage Oracle Database Automatic Storage Management (ASM) Recovery Manager (RMAN) Oracle Secure Backup (OSB) Oracle benefits: Integrated corruption detection Integrated volume management (ASM) Optimum I/O performance Provision or migrate storage online Automatic storage tuning Integrated disk mirroring Integrated Oracle backups (RMAN + OSB) Block validation Online block-level recovery Unused block compression Online, multi-streamed backup Native encryption Multiple compression levels Fast backups to disk, tape, cloud (Amazon S3) 16

17 Evolution Towards MAA Start with a Single Server, but Database Instance Database Storage Vulnerable to downtime and data loss: Server failures Database instance crashes Administrator errors Data corruptions Network outages CRASH Site failures Most planned maintenances All users impacted by any outage 17

18 Evolution Towards MAA contd. Protect Instance with RAC One Node Database Instance Database Storage Active Passive Protection from some downtime Unplanned: Server failures Instance crashes Planned Online relocation of Oracle instance Rolling upgrades for maintenance & patches Risks of other downtime remain: Administrator errors Data corruptions Network outages Site failures Patch set and database upgrades Users impacted by many outages Unused server resource, till failure occurs 18

19 Evolution Towards MAA contd. Enhance Server Scalability & Availability with RAC RAC: All Servers Active Database Instance Database Storage Real Application Clusters (RAC) best solution for Server HA: Protection from server failures Protection from database instance crashes Enables rolling patch upgrades All instances / servers are active Scale throughput by simply adding servers online Automated workload management Still need to eliminate downtime from: Administrator errors Data corruptions Network outages Site failures Patch set and database upgrades 19

20 Oracle Real Application Clusters What customers think 2010 Oracle Corporation 20

21 Evolution Towards MAA contd. Eliminate Administrator Errors with Flashback Flashback Query Flashback Table Flashback Transaction Flashback Database Fast recovery from human errors Oracle-integrated Continuous Data Protection (CDP) Operates only on changed data in a highly optimized manner Reduces correction time from hours to minutes Correction Time = Error Time + f(db_size) Simple commands instead of complex procedure 21

22 Evolution Towards MAA contd. Disaster Protection: How about Storage Mirroring? Primary Site: RAC - All Servers Active DR Site: All Servers Inactive Database Instance Database Storage Storage Mirroring Storage mirroring: Redundant storage protects from storage failures, but: No protection from physical data corruptions DR systems offline during mirroring no real-time data validation Distance limited, storage vendor lock-in, manual failover, no rolling upgrades, high network use 22

23 Example: Storage Mirroring Weakness High Usage of Network Resources Primary Volumes Log Buffer Online Logs fil Archive Logs Flashback Logs Control Files Data Files SYSTEM USER TEMP UNDO Network I/O Target Volumes Oracle Apply & Validation 7X more network volume 27X more network I/Os Zero Oracle awareness Poor isolation Idle standby systems 23

24 Evolution Towards MAA contd. Integrated HA & DR with RAC and Active Data Guard Primary Site: RAC - All Servers Active Standby Site: All Servers Read-Only Active Database Instance Data Guard Database Storage Data Guard: Oracle-integrated disaster recovery & data protection solution: Protection from storage, site, network failures and data corruptions Reporting, queries, backups offloaded to DR systems supports real-time data validation No distance limitation, storage agnostic, automatic failover, rolling upgrades, optimized network use 24

25 Example: Data Guard Strength Optimized Usage of Network Resources Primary Database Log Buffer Online Logs fil Archive Logs Flashback Logs Control Files Data Files SYSTEM USER TEMP UNDO Network I/O Oracle-aware physical replication Strong isolation End-to-end validation Automatic block repair Real-time reporting Storage agnostic Standby Database Oracle Apply & Validation 25

26 Data Guard: Rolling Maintenance Minimize Planned Downtime Data center moves Technology refresh 32 bit to 64 bit Windows to Linux AIX 64 bit to Solaris Sparc Migrate to RAC Migrate to ASM Migrate to Exadata System maintenance Database rolling upgrades Index and storage changes Implement Advanced Compression Migrate to SecureFiles Test new features See My Oracle Support Note for details 26

27 Evolution Towards MAA contd. Scorecard so far Open read-write Primary Database Data Guard Oracle Enterprise Manager Open read-only Active Standby Database Oracle Database, ASM, RMAN, Oracle RAC, Flashback, Data Guard, Enterprise Manager Protection against downtime and data loss: Server failures Database instance crashes Storage subsystem failures System induced data corruptions Administrator errors Network outages Site failures System maintenance One-off patches and CPUs Database patch-sets and upgrades 27

28 Planned Maintenance Not Yet Addressed Completely Database rolling upgrades from pre-oracle Database 10g Flexible upgrade strategies that incorporate active-active multimaster replication Cross-endian platform migrations Migration from non-oracle databases to Oracle A number of other cross platform migrations Application upgrades 28

29 Evolution Towards MAA contd. GoldenGate for Active-Active Replication Best-in-class data replication Real-time information Real-time Access Broad support for heterogeneous platforms Flexible solution for minimal/zero downtime upgrades and migrations 29

30 Oracle GoldenGate Architecture Capture Trail Trail Pump Delivery LAN/WAN Internet TCP/IP Source Oracle & Non-Oracle Database(s) Bi-directional Target Oracle & Non-Oracle Database(s) 30

31 Oracle GoldenGate Support for Heterogeneous Databases / Platforms Databases Oracle GoldenGate Capture: Oracle DB2 Microsoft SQL Server Sybase ASE Teradata Enscribe SQL/MP SQL/MX MySQL JMS message queues Oracle GoldenGate Delivery: All listed above, plus: TimesTen Netezza, Greenplum, and HP Neoview ETL products O/S and Platforms Linux Sun Solaris Windows 2000, 2003, XP HP NonStop HP-UX HP OpenVMS IBM AIX IBM z Series zlinux 31

32 RAC, Data Guard, GoldenGate Integrated HA, DR and Active-Active Replication RAC - Scalability - Server HA Active Data Guard - DR & Data Protection - Real-time Query Primary Database GoldenGate - Information Distribution - Heterogeneous Standby Database Bi-directional Replication Subsetting MySQL 32

33 Evolution Towards MAA contd. Edition-based Redefinition: Application Upgrades Enables online application upgrade Code changes installed in the privacy of a new edition Data changes are made safely by writing only to new columns or new tables not seen by the old edition An editioning view exposes a different projection of a table into each edition to allow each to see just its own columns A crossedition trigger propagates data changes made by the old edition into the new edition s columns, or (in hot-rollover) vice-versa 33

34 Oracle Maximum Availability Architecture Low-cost, Integrated, Fully Active, High ROI Production Site RAC Scalability Server HA Flashback Human error correction Active Replica Active Data Guard Data Protection, DR Query Offload GoldenGate Active-active Heterogeneous ASM Volume Management Edition-based Redefinition, Data Guard, GoldenGate Minimal downtime maintenance, upgrades, migrations RMAN & Fast Recovery Area On-disk backups Oracle Secure Backup Backup to tape / cloud 34

35 Maximum Availability Architecture What customers think 2010 Oracle Corporation 35

36 MAA with Exadata Database Machine Complete, Open, Integrated Availability Real Application Clusters Active Data Guard ASM RMAN, Fast Recovery Area Oracle Secure Backup WAN Comprehensive protection from failures: server, storage, network, site, corruptions Correction from human errors: database, table, row, transaction Active DR: Real-time remote standby open for query offload Online indexing and table redefinition Online patching and upgrades 36

37 MAA Recommendations Protection from Unplanned Outages Outage Oracle Solution (s) Downtime Server failure Oracle RAC, Clusterware Zero Storage failure Automatic Storage Management (ASM) Zero Database and site failure Oracle Data Guard < 60 seconds Data corruptions Oracle Data Guard Zero to < 60 seconds Human Errors Oracle Flashback Technologies 80x faster than restore 37

38 MAA Recommendations Minimizing Planned Downtime Planned Event Oracle Solution(s) Downtime Operating system and hardware maintenance, add/remove cluster nodes or storage Oracle one-off patches, critical patch updates, file system and clusterware upgrades Site maintenance, cluster-wide maintenance Oracle patch-set and full Oracle release upgrades Oracle RAC, Clusterware, ASM Oracle RAC, Clusterware, ASM Oracle Data Guard, GoldenGate Oracle Data Guard, GoldenGate Zero Zero Minimal or zero Minimal or zero Platform Migrations Oracle Data Guard, GoldenGate Minimal or zero Application upgrades Edition-based redefinition, GoldenGate Zero 38

39 Jon Waldron Executive Architect Commonwealth Bank of Australia High availability is absolutely essential for us we now use Oracle RAC for instance failover, Data Guard for site failover, ASM to manage our storage, and Oracle Clusterware to hang the whole thing together. Ref. 39

40 Resources Database High Availability oracle.com/ha High Availability Best Practices oracle.com/goto/maa Active Data Guard Hands-On Lab (HOL) oracle.com/technetwork/database/features/availability/data-guard-hol html Same experience as on-site HOL at Oracle OpenWorld 2010 Oracle Database 11g Resource Center oracledatabase11g.com 40

41 41