VERITAS Storage Foundation 4.0 for Oracle RAC. Oz Melamed E&M Computing

Transcription

1 VERITAS Storage Foundation 4.0 for Oracle RAC Oz Melamed E&M Computing

2 Cluster Basics

3 Private Network LLT

4 Failover and Switchover

5 Failover Configurations Asymmetric

6 Failover Configurations Symmetric

7 What is RAC? Real Application Clusters (RAC) is a parallel database environment that takes advantage of the processing power of multiple, interconnected computers. A cluster comprises two or more computers, also known as nodes or servers. In RAC environments, all nodes concurrently run Oracle instances and execute transactions against the same database. RAC coordinates each node s access to the shared data to provide consistency and integrity. Each node adds its processing power to the cluster as a whole and can increase overall throughput or performance.

8 RAC Architecture From a high level, RAC is multiple Oracle instances, accessing a single Oracle database and carrying out simultaneous transactions. An Oracle database is the physical data stored in tablespaces on disk. An Oracle instance represents the software processes necessary to access and manipulate the database. In traditional environments, only one instance accesses a database at a specific time. With Oracle RAC, multiple instances communicate to coordinate access to a physical database, greatly enhancing scalability and availability.

9 Architecture

10 Data Stack Overview The various Oracle processes making up an instance (such as DB Writer, Log Writer, Checkpoint, Archiver, and others) read and write data to storage via the I/O stack shown in the diagram. Oracle communicates via the Oracle Disk Manager (ODM) interface to the VERITAS Cluster File System (CFS), which in turn accesses the storage via the VERITAS Cluster Volume Manager (CVM).

11 Data Stack Overview

12 Communications Stack Overview RAC instances must communicate to coordinate protection of data blocks in the database. ODM processes must communicate to coordinate data file protection and access across the cluster. CFS coordinates metadata updates for file systems, and finally CVM must coordinate the status of logical volumes and distribution of volume metadata across the cluster. VERITAS Cluster Server (VCS) controls starting and stopping of components in the Storage Foundation for Oracle RAC stack and provides monitoring and notification on failure.

13 Communications Stack Overview

14 Cluster Communications GAB s second function is to provide reliable cluster communications, used by many Storage Foundation for Oracle RAC modules. GAB provides guaranteed delivery of point to point messages and broadcast messages to all nodes. Point to point messaging uses a send and acknowledgment. Atomic Broadcast ensures all systems within the cluster receive all messages. If a failure occurs while transmitting a broadcast message, GAB s atomicity ensures that, upon recovery, all systems have the same information.

15 Cluster Communications

16 Cluster Volume Manager (CVM) Cluster Volume Manager is an extension of VERITAS Volume Manager (VxVM), the industry standard storage virtualization platform. CVM extends the concepts of VxVM across multiple nodes. Each node sees the same logical volume layout, and more importantly, the same state of all volume resources. In a Storage Foundation for Oracle RAC cluster, all storage is managed with standard VxVM commands from one node in the cluster. All other nodes immediately recognize any changes in disk group and volume configuration. CVM supports all performance enhancing capabilities, such as striping, mirroring, and mirror break-off (snapshot) for offhost backup.

17 CVM Architecture Cluster Volume Manager is designed with a master/slave architecture. One node in the cluster acts as the configuration master for logical volume management, and all others are slaves. Any node can take over as master if the existing master fails. The CVM master is established on a percluster basis.

18 Oracle Disk Manager The Oracle Disk Manager (ODM) is a standard API specified by Oracle for doing database I/O. For example, when Oracle wishes to write, it calls the odm_io function. ODM improves both performance and manageability of the file system. The VERITAS-provided implementation of ODM improves performance by providing direct access for the database to the underlying storage without passing through the actual file system interface. This means the database sees performance that is equivalent to using raw devices. The administrator sees the storage as easy-to-manage file systems, including the support of resizing data files while in use.

19 VCS Architecture VCS communicates the status of resources running on each system to all systems in the cluster. The High Availability Daemon, or HAD, is the main VCS daemon running on each system. HAD collects all information about resources running on the local system, forwarding it to all other systems in the cluster. It also receives information from all other cluster members to update its own view of the cluster.

20 VCS Architecture Each type of resource supported in a cluster is associated with an agent. An agent is an installed program designed to control a particular resource type. Each system runs necessary agents to monitor and manage resources configured to run on that node. The agents communicate with HAD on the node. HAD distributes its view of resources on the local node to other nodes in the cluster using GAB and LLT.

21 VCS Architecture

22 Storage Foundation for Oracle RAC Service Groups Storage Foundation for Oracle RAC uses parallel service groups to support RAC. There is one CVM group per server. This group has the CVM resource and the necessary resources for support of CFS. Users can modify this group to contain all common components needed by Oracle to support RAC. This includes a shared ORACLE_HOME directory and the Oracle Net Services process (LISTENER).

23 Service Groups

24 I/O Fencing I/O fencing is a feature within a kernel module of Storage Foundation for Oracle RAC designed to guarantee data integrity, even in the case of faulty cluster communications causing a split brain condition.

25 Understanding Split Brain and the Need for I/O Fencing Split brain is an issue faced by all cluster solutions. To provide high availability, the cluster must be capable of taking corrective action when a node fails. In Storage Foundation for Oracle RAC, this is carried out by the reconfiguration of CVM, CFS, and RAC to change membership. Problems arise when the mechanism used to detect the failure of a node breaks down.

26 Split Brain The symptoms look identical to a failed node. For example, if a system in a two-node cluster were to fail, it would stop sending heartbeats over the private interconnects and the remaining node would take corrective action. However, the failure of the private interconnects would present identical symptoms. In this case, both nodes would determine that their peer has departed and attempt to take corrective action. This typically results in data corruption when both nodes attempt to take control of data storage in an uncoordinated manner.

27 SCSI-3 Persistent Reservations Storage Foundation for Oracle RAC uses a relatively new enhancement to the SCSI specification, known as SCSI-3 Persistent Reservations, (SCSI-3 PR). SCSI-3 PR is designed to resolve the issues of using SCSI reservations in a modern clustered SAN environment. SCSI-3 PR supports multiple nodes accessing a device while at the same time blocking access to other nodes. SCSI-3 PR reservations are persistent across SCSI bus resets and SCSI-3 PR also supports multiple paths from a host to a disk

28 Coordinator Disks Coordinator disks are special purpose disks in a Storage Foundation for Oracle RAC environment. Coordinator disks are three (or an odd number greater than three) standard disks, or LUNs, set aside for use by I/O fencing during cluster reconfiguration.

29

30 Check Cluster Status ~]# hastatus -sum -- SYSTEM STATE -- System State Frozen A gali RUNNING 0 A mali RUNNING 0 A tali RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B cvm gali Y N ONLINE B cvm mali Y N ONLINE

31 Check Cluster Interconnect ~]# /sbin/lltstat -nvv more LLT node information: Node State Link Status Address * 0 tali OPEN eth0 UP 00:04:23:B2:0F:EE eth2 UP 00:09:3D:11:A3:84 1 mali OPEN eth0 UP 00:09:3D:11:AA:88 eth2 UP 00:09:3D:11:AA:88 2 gali OPEN eth0 UP 00:04:23:AE:E3:14 eth2 UP 00:04:23:AE:E3:14

32 Verifying GAB Port Membership galaxy# /sbin/gabconfig -a GAB Port Memberships ============================================= Port a gen 4a1c0001 membership 01 Port b gen g8ty0002 membership 01 Port d gen membership 01 Port f gen f membership 01 Port h gen d membership 01 Port o gen f membership 01 Port v gen 1fc60002 membership 01 Port w gen 15ba0002 membership 01

33 GAB Port Function a GAB b I/O fencing d ODM (Oracle Disk Manager) f CFS (Cluster File System) h VCS (VERITAS Cluster Server: high availability daemon) o VCSMM driver v CVM (Cluster Volume Manager) w vxconfigd (module for CVM)

34 To verify the I/O Fencing configuration # vxfenadm -d The output of this command should look similar to: I/O Fencing Cluster Information: ================================ Cluster Members: * 0 (galaxy) 1 (nebula) RFSM State Information: node 0 in state 8 (running) node 1 in state 8 (running)

35 To determine whether a node is the CVM master or a CVM slave On one node, determine whether it is the master or the slave node by using the command: # vxdctl -c mode Expected output: cluster active - MASTER or: cluster active - SLAVE

36 To enable write access to the volumes in the disk group You must enable write access to the shared volumes so you can create databases on them. You can use the following commands on the CVM master node: # vxdg -s import shared_disk_group # vxvol -g shared_disk_group startall # vxdg -g shared_disk_group set activation=sw

37 Mount CFS On all nodes, mount the file system: mount t vxfs o cluster /dev/vx/dsk/ora_dg/od01 /dwh/od01

38 How I/O Fencing Works in Different Event Scenarios