DB2 purescale Active/Active High Availability is Here!

Transcription

1 purescale Active/Active High Availability is Here! Session C04, for LUW Aamer Sachedina STSM, IBM Toronto Lab November 9, 2010, 8:30am 0 purescale is state of the art for LUW technology which offers active/active high availability on distributed systems. Five years in the making, this technology is now generally available and is expected to see wide deployment in critical ERP and OLTP systems. This presentation dives into details of the high availability architecture of purescale. Details of Cluster Services - the component responsible for recovery orchestration are presented and the overall architecture is compared and contrasted with competitive technology to display the clear advantages of purescale. The presentation also includes an in-depth walkthrough of several failure-recovery scenarios including software and hardware failures. 0

2 Copyright IBM Corporation [current year]. All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE. Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere,, Maximo, Clearcase, Lotus, etc IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at If you have mentioned trademarks that are not from IBM, please update and add the following lines: [Insert any special 3rd party trademark names/attributions here] Other company, product, or service names may be trademarks or service marks of others. 1 1

3 Agenda Cluster Services Failure scenarios Member SW failure and restart on home host Member Host failure and restart light on guest host I/O fencing Member SW failure and restart light on guest host Primary PowerHA purescale hardware failure 2 2

4 Cluster Services Overview Integrated component Integrated component Single install as part of installation Upgrades and maintenance through fixpack Cluster Services: Cluster Manager (RSCT) Cluster Automation (Tivoli SA MP) CF Cluster Services: Cluster File System (GPFS) CF 3 3

5 Cluster Services Cluster Services Reliable Scalable Cluster Technology Tivoli Systems Automation for Multi-Platforms IBM General Parallel File System tightly integrates these IBM products into purescale instance creation creates RSCT and GPFS domains across hosts Single command used to add hosts to the instance: db2iupdt add -m newhost.acme.com db2inst1 4 4

6 Up and Running with Cluster Services purescale install requires two disk devices to be provided: 1. Tie-breaker 2. Instance file system for shared instance files (sqllib_shared) Install includes, PowerHA purescale and Cluster Services PowerHA purescale Cluster Services Binaries installed, cluster domain created, sqllib_shared file system mounted on all hosts. 5 5

7 Cluster Services: Storage Management Provision shared LUNs from storage administrator Create two file systems for each database: One file system for tablespaces One file system for database logs and meta-data Instance owner can create file systems does not require sys admin. System administrator needs to change ownership of LUNs to instance owner 6 6

8 Cluster Services: Storage Management myhost:/home/db2inst1> ls -l /dev/sda* brw-r db2inst1 dbag 8, :02 /dev/sda1 brw-r db2inst1 dbag 8, :01 /dev/sda2 brw-r db2inst1 dbag 8, :02 /dev/sda3 brw-r db2inst1 dbag 8, :02 /dev/sda4 myhost:/home/db2inst1> db2cluster cfs create filesystem datafs disk /dev/sda1 /dev/sda2 /dev/sda3 mount /db2/data myhost:/home/db2inst1> db2cluster cfs create filesystem logfs disk /dev/sda4 -mount /db2/logs 7 7

9 Cluster Services: Storage Management Member Member Member Member SAN /db2/data /db2/logs /db2/data /db2/logs /db2/data /db2/logs /db2/data /db2/logs datafs sda1 sda2 sda3 datafs logfs sda4 logfs 8 8

10 Cluster Services: Storage Management myhost:/home/db2inst1> CREATE DATABASE MYDB ON /db2/datafs DBPATH ON /db2/logfs CFS does fine-grained striping over LUNs. Stripe size fixed at 1 MB No there is no hidden registry variable to change the stripe size myhost:/home/db2inst1> db2cluster cfs -add filesystem datafs disk /dev/sda5 myhost:/home/db2inst1> db2cluster cfs rebalance filesystem datafs myhost:/home/db2inst1> db2cluster cfs -remove filesystem datafs disk /dev/sda3 9 9

11 purescale HA Architecture Member Member Member Member Cluster Interconnect GPFS Secondary 2 nd -ary Primary 10 10

12 purescale Instance Status Clients > db2instance -list ID TYPE STATE HOME_HOST CURRENT_HOST ALERT host0 Single Database View host1 host2 host3 0 MEMBER STARTED host0 host0 NO 1 MEMBER STARTED host1 host1 NO 2 MEMBER STARTED host2 host2 NO 3 MEMBER STARTED host3 host3 NO 4 CF PRIMARY host4 host4 NO 5 CF PEER host5 host5 NO host5 host4 HOST_NAME STATE INSTANCE_STOPPED ALERT host0 ACTIVE NO NO host1 ACTIVE NO NO host2 ACTIVE NO NO host3 ACTIVE NO NO host4 ACTIVE NO NO host5 ACTIVE NO NO Secondary Shared Data Primary db2nodes.cfg 0 host0 0 - MEMBER 1 host1 0 - MEMBER 2 host2 0 - MEMBER 3 host3 0 - MEMBER 4 host4 0 - CF 5 host5 0 - CF 11 11

13 Member SW Failure : Member Restart on Home Host kill -9 erroneously issued to a member Clients Cluster Services automatically detects member s death Informs other members & powerha purescale servers Initiates automated member restart of light member on same ( home ) host Automatic; In the mean-time, client connections are transparently re-routed to healthy members Ultra Fast; Based on least load (by default), or, Pre-designated failover member Online Other members remain fully available throughout Online Failover Primary retains update locks held by member at the time of failure Other members can continue to read and update data not locked for write access by failed member Member restart completes Retained locks released and all data fully available Full member started and available for transaction processing Updated Pages Global Locks Secondary Single Database View Shared Data Log Records Pages Updated Pages Global Locks Primary 12 kill -9 12

14 Member SW Failure and Restart on Home Host > db2instance -list kill -9 host0 host1 host2 host3 ID TYPE STATE HOME_HOST CURRENT_HOST ALERT 0 MEMBER STARTED host0 host0 NO 1 MEMBER STARTED host1 host1 NO 2 MEMBER STARTED host2 host2 NO 3 MEMBER RESTARTING STARTED host3 host3 NO 4 CF PRIMARY host4 host4 NO 5 CF PEER host5 host5 NO host5 Secondary Shared Data db2nodes.cfg 0 host0 0 - MEMBER 1 host1 0 - MEMBER 2 host2 0 - MEMBER 3 host3 0 - MEMBER 4 host4 0 - CF 5 host5 0 - CF host4 Primary HOST_NAME STATE INSTANCE_STOPPED ALERT host0 ACTIVE NO NO host1 ACTIVE NO NO host2 ACTIVE NO NO host3 ACTIVE NO NO host4 ACTIVE NO NO host5 ACTIVE NO NO 13 13

15 Member HW Failure and Restart Light Power cord tripped over accidentally Cluster Services looses heartbeat and declares member down Informs other members & PowerHA purescale servers Fences member from logs and data Initiates automated member restart on another ( guest ) host Using reduced, and pre-allocated memory model Automatic; Member restart is like a database crash recovery in a single system database, but is much faster Redo limited to inflight transactions (due to FAC) Ultra Fast; Benefits from page cache in PowerHA purescale In the mean-time, client connections are automatically re-routed to healthy members Based on least load (by default), or, Pre-designated failover member Online Other members remain fully available throughout Online Failover Primary retains update locks held by member at the time of failure Other members can continue to read and update data not locked for write access by failed member Member restart completes Retained locks released and all data fully available Updated Pages Global Locks Secondary Clients Single Database View Shared Data Fence Log Recs Pages Updated Pages Global Locks Primary 14 14

16 Member Hardware Failure and Restart Light on Guest Host > db2instance -list ID TYPE STATE HOME_HOST CURRENT_HOST ALERT host0 host1 host2 host3 Fence 0 MEMBER STARTED host0 host0 NO 1 MEMBER STARTED host1 host1 NO 2 MEMBER STARTED host2 host2 NO 3 MEMBER WAITING_FOR_FAILBACK RESTARTING STARTED host3 host3 host2 NO 4 CF PRIMARY host4 host4 NO 5 CF PEER host5 host5 NO host5 Secondary host4 Shared Data Log Recs db2nodes.cfg 0 host0 0 - MEMBER 1 host1 0 - MEMBER 2 host2 0 - MEMBER 3 host3 0 - MEMBER 4 host4 0 - CF 5 host5 0 - CF Pages host4 Primary HOST_NAME STATE INSTANCE_STOPPED ALERT host0 ACTIVE NO NO host1 ACTIVE NO NO host2 ACTIVE NO NO host3 INACTIVE NO NO YES host4 ACTIVE NO NO host5 ACTIVE NO NO 15 15

17 Cluster Services Robust heart-beating and failure detection algorithm. SI-3 Persistent Reserve I/O Fencing for split-brain avoidance. Guarantees protection of shared data in the event that one or more errant hosts splits from the network. Substantially more robust than technology used by others (self initiated reboot based algorithms or STONITH). Allows re-integration of a split host into the cluster when the network heals without requiring a reboot. Host failure detection and I/O fencing in ~ 3 seconds 16 Cluster Services has a robust heart-beating and failure detection algorithm. It is designed to prevent the reporting of false downs when network congestion and heavy CPU utilization might suggest there is a problem. Fencing of a host will occur if AIX fails, one of the adapters supporting the high speed interconnect fail or the host itself fails. If present, Cluster Services can use SI-3 Persistent Reserve I/O Fencing to avoid split-brain situations. This guarantees that if one or more errant hosts splits from the network that the shared data will be protected. SI-3 Persistent Reserve is substantially more robust than the technology used by others (self initiated reboot based algorithms or STONITH). Note: Here is the simplest explanation of STONITH that was found on the web: stonith: Is an interface for remotely powering down a node in a cluster. The STONITH module (a.k.a. STOMITH) provides an extensible interface for remotely powering down a node in the cluster. The idea is quite simple: When the software running on one machine wants to make sure another machine in the cluster is not using a resource (e.g. in this case it does not want the other machine to continue using the disk cluster), pull the plug on the other machine. It's simple and reliable, but it does lead to a complete outage that requires a reboot of the node that was killed. 16

18 Member SW Failure and Restart Light on Guest Host > db2instance db2cluster list -list alert host0 host1 host2 host3 kill -9 ID 1. TYPE STATE HOME_HOST CURRENT_HOST ALERT Alert: The member 3 failed to start on its home 0 host MEMBER host3. STARTED Check host0 the db2diag.log host0 for messages NO 1 concerning MEMBER STARTED failures on host1 host3 host1 for member NO 3. 2 See MEMBER the STARTED Information host2 Center host2 for more details. NO 3 MEMBER WAITING_FOR_FAILBACK host3 host2 YES Action: 4 CF This PRIMARY alert must host4 be cleared host4 manually NO with the 5 CF command: PEER db2cluster host5 -cm -clear host5 -alert. NO host5 Secondary Shared Data Log Recs db2nodes.cfg 0 host0 0 - MEMBER 1 host1 0 - MEMBER 2 host2 0 - MEMBER 3 host3 0 - MEMBER 4 host4 0 - CF 5 host5 0 - CF Pages host4 Primary Impact: HOST_NAME Member STATE 3 will INSTANCE_STOPPED not be able to service ALERT requests until this alert has been cleared and the host0 member returns ACTIVE to its NO home host. NO host1 ACTIVE NO NO > host2 db2cluster ACTIVE -clear alert NO NO host3 ACTIVE NO NO The host4 alert(s) ACTIVE has been successfully NO cleared. NO host5 ACTIVE NO NO 17 17

19 Primary PowerHA purescale Failure Power cord tripped over accidentally Cluster Services looses heartbeat and declares primary down Informs members and secondary PowerHA purescale service momentarily blocked All other database activity proceeds normally Eg. accessing pages in bufferpool, existing locks, Automatic; sorting, aggregation, etc Clients Single Database View Members send missing data to secondary Eg. read locks Ultra Fast; Online Secondary becomes primary PowerHA purescale service continues where it left off No errors are returned to members Updated Pages Global Locks Shared Data Updated Pages Global Locks Secondary Primary Primary 18 18

20 Primary PowerHA purescale Failure > db2instance -list host5 host0 Secondary Primary host1 Shared Data host2 db2nodes.cfg 0 host0 0 - MEMBER 1 host1 0 - MEMBER 2 host2 0 - MEMBER 3 host3 0 - MEMBER 4 host4 0 - CF 5 host5 0 - CF host3 host4 ID TYPE STATE HOME_HOST CURRENT_HOST ALERT ALERT 0 MEMBER STARTED STARTED host0 host0 host0 host0 NO NO 1 MEMBER STARTED host1 host1 NO 2 MEMBER MEMBER STARTED STARTED host2 host1 host2 host1 NO NO 3 MEMBER STARTED STARTED host3 host2 host3 host2 NO NO 34 CF MEMBER ERROR STARTED host4 host3 host4 host3 NO NO 54 CF PRIMARY ERROR host5 host4 host5 host4 NO NO 5 HOST_NAME CF BECOMING_PRIMARY STATE INSTANCE_STOPPED host5 host5 ALERT NO HOST_NAME host0 ACTIVE STATE INSTANCE_STOPPED NO NO host1 ACTIVE NO NO ALERT host2 ACTIVE NO NO host0 host3 ACTIVE ACTIVE NO NO NO NO host1 host4 INACTIVE ACTIVE NO NO YES NO host2 host5 ACTIVE NO NO NO NO host3 ACTIVE NO NO host4 INACTIVE NO YES host5 ACTIVE NO NO 19 19

21 PowerHA purescale Re-integration > db2instance Power restored -listand system rebooted ID TYPE STATE HOME_HOST CURRENT_HOST ALERT 0 MEMBER STARTED host0 host0 NO 1 MEMBER STARTED host1 host1 NO 2 MEMBER STARTED host2 host2 NO 3 MEMBER STARTED host3 host3 NO 4 CF CATCHUP PEER host4 host4 NO 5 CF PRIMARY host5 host5 NO Cluster Services automatically detects system availability Informs members and primary HOST_NAME STATE INSTANCE_STOPPED ALERT New system assumes secondary role in catchup state Members resume duplexing Members asynchronously send lock and other state information to secondary Members asynchronously castout pages from primary to disk host0 ACTIVE NO NO host1 ACTIVE NO NO host2 ACTIVE NO NO host3 ACTIVE NO NO host4 ACTIVE NO NO host5 ACTIVE NO NO Updated Pages Global Locks Primary Clients Single Database View Shared Data Updated Pages Global Locks Secondary (Catchup (Peer state) 20 20

22 Cluster Services State of the art active-active HA and automated recovery out of the box No cluster management or failover policy configuration required Cluster automation and status is DBA and tools friendly SQL / Admin views etc

23 purescale Unlimited Capacity Start small Grow easily, with your business Application Transparency Avoid the risk and cost of tuning your applications to the database topology Continuous Availability Maintain service across planned and unplanned events 22 22

24 Thank You! Your feedback is important. Please fill out your session evaluation. Session: C04 Speaker: Aamer Sachedina is welcome: 23 23