Oracle RAC 10g Celerra NS Series NFS

Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide Revision 1.0 EMC Solutions Practice/EMC NAS Solutions Engineering. EMC Corporation RTP Headquarters RTP, NC 27709 www.emc.com

Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide Trademark/Copyright Information Copyright 2006 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part Number H2373 ii Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide

Contents About this Document Purpose... v Audience... v Scope... v Related documents... vi Chapter 1 Chapter 2 Solution Overview The business challenge... 2 The technology solution... 2 Solution advantages... 2 Solution Architecture Overall architecture... 6 General characteristics... 8 Network architecture... 9 VLANS... 9 Switches... 10 EMC NS502 Data Mover ports... 10 Storage architecture... 10 High availability and failover... 11 RAID type and RAID group configuration... 12 Disk volume setup... 13 MVM vs. AVM... 14 File systems, exports and mount points... 14 Oracle RAC 10g NS Series NFS Reference Architecture Guide iii

Contents Database server architecture... 15 Oracle Database 10g server network architecture... 15 Oracle cluster ready services (CRS)... 16 NFS client... 16 High availability and failover... 16 Application architecture... 17 Chapter 3 Hardware and Software Resources Hardware resources... 20 Software resources... 21 iv Contents

About this Document This document provides an overview of the architecture of the EMC solution for Oracle RAC 10g on NS Series NFS developed jointly by the EMC Technology Solutions Practice and EMC NAS Solutions Engineering. Purpose Audience Scope Information in this document can be used as the basis for a solution build, white paper, best practices document, or training. Information in this document can also be used by other EMC organizations (for example, the technical services or sales organization) as the basis for producing documentation for a technical services or sales kit. This document is intended for EMC personnel, partners, and customers. This document describes the architecture of an EMC solution built and tested jointly by the EMC Technology Solutions Practice and EMC NAS Solutions Engineering, both located in Research Triangle Park, N.C. Implementation instructions and best practices are beyond the scope of this document. The Oracle RAC 10g Celerra NS Series NFS Validation Test Report provides detailed information about the tests performed to validate this solution. Note: The validation test report is available only to EMC personnel and partners. Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide v

Preface Related documents The following documents provide additional, relevant information: Oracle Database 10g/Oracle RAC 10g Celerra NS Series NFS Best Practices Planning Oracle Database 10g/Oracle RAC 10g Celerra NS Series NFS Applied Technology Celerra Network Server documentation Oracle RAC 10g presentations and white papers on EMC Powerlink vi Related documents

1 Solution Overview This chapter includes the following sections: The business challenge...2 The technology solution...2 Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide 1

Solution Overview The business challenge The technology solution Mid-size enterprises face the same challenges as their larger counterparts when it comes to managing database environments. Typical challenges include cost, control, resource utilization, and scaling. Oracle RAC 10g supports numerous functions ranging from manufacturing to online web applications. These deployments may be in smaller data centers that often support applications ranging from custom database applications to standard packages such as SAS, Siebel, PeopleSoft, and SAP, or they may be in large, centralized data centers Mid-size enterprises need scalable and RAID-protected storage but must be extremely cost-conscious. They may lack the IT resources to deploy, manage, and maintain complex environments at the departmental level. The Oracle RAC 10g - Celerra NS Series NFS solution enables a mid-size enterprise to deploy an EMC network-attached storage (NAS) architecture with NFS connectivity for its Oracle Database 10g database applications with lower cost and complexity than direct-attached storage (DAS) or a storage area network (SAN). Solution advantages The solution offers the following advantages: Table 1-1 Solution advantages Benefit Lower total cost of ownership (TCO) Greater manageability Simplified Real Application Clusters (RAC) implementation High availability Details Reduces acquisition, administration, and maintenance costs than equivalent DAS or SAN Eases implementation, provisioning, and volume management Provides NFS-mounted shared file systems Can implement a clustering architecture that provides very high levels of data availability 2 The business challenge

Solution Overview Table 1-1 Solution advantages Benefit Increased flexibility Improved protection Benefits of EMC Information Lifecycle Management (ILM) Details Easily make databases, or copies of database, available (via remounts) to other servers Integrate both availability and backup Implement tiered storage Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide 3

Solution Overview 4 The technology solution

2 Solution Architecture This chapter includes the following sections: Overall architecture... 6 Network architecture...9 Storage architecture... 10 Database server architecture... 15 Application architecture... 17 Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 5

Solution Architecture Overall architecture Table 2-1 The following figures show the overall architecture of the Oracle RAC 10g NFS solution: Oracle RAC 10g NFS Solution configurations Figure Figure 2-1 Figure 2-2 Figure 2-3 Configuration Consolidation Backup Protect Each of the configurations include the following components: Client, RAC interconnect, and storage networks consisting of dedicated network switches and VLANs Oracle RAC 10g servers connected to the client, RAC interconnect, and storage networks Clients connected to the Oracle RAC 10g servers via the client network A Celerra NS Series (NS502) network file server connected to the Oracle RAC 10g servers via the storage network 6 Overall architecture

Solution Architecture Figure 2-1 Overall architecture of the Oracle RAC 10g NFS consolidation solution Figure 2-2 Overall architecture of the Oracle RAC 10g NFS backup solution Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 7

Solution Architecture Figure 2-3 General characteristics Overall architecture of the Oracle RAC 10g - Celerra NS Series NFS protect solution Each solution configuration has the following general characteristics: Each configuration consists of either a two-node or four-node active/active Oracle RAC 10g RAC. Oracle database files, online redo log files, archived log files and the flashback recovery area each reside on their own NFS file system. Online redo log files are mirrored across two different file systems using Oracle software multiplexing. The control files are mirrored across the online redo log file systems. The voting disk and ocr files are mirrored across multiple NFS file systems. 8 Overall architecture

Solution Architecture Network architecture NFS file systems are designed to satisfy the I/O demands of individual database objects, using either RAID 5, RAID 3, or RAID 1, as appropriate. IP connections are trunked on the storage and RAC interconnect networks to balance and distribute network I/O. All database files are stored on the EMC Celerra NS Series storage system, making server replacement relatively simple. Oracle RAC 10g for x86 or x86-64 runs under Red Hat Enterprise Linux. TCP/IP and NFS version 3 provide network connectivity and file system semantics. This section describes the network architecture of the validated solution. VLANS VLANs, or virtual local area networks, logically group devices that are on different network segments, or subnetworks. The validated solution uses three VLANs to segregate network traffic of different types. This improves throughput, manageability, application separation, high availability, and security. Figure 2-1 on page 7 shows the three VLANs: A client VLAN supports connectivity between the Oracle Database 10g servers and the client workstations. The client VLAN also supports connectivity between the EMC Celerra NS Series and the client workstations to provide network file services to the clients. Control and management of these devices is also provided via the client network. A RAC interconnect VLAN network supports connectivity between the Oracle Database 10g servers for network I/O required by Oracle cluster ready services (CRS). Two network interface cards (NICs) are configured on each Oracle Database 10g server to the RAC interconnect network. Link aggregation is configured on the servers to provide load balancing and port failover between the two ports for this network. The storage VLAN uses the NFS protocol to provide connectivity between servers and storage. Each database server connected to the storage VLAN has two NICs dedicated to the storage VLAN. Link aggregation is configured on the servers to provide load balancing and port failover between the two ports. Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 9

Solution Architecture Switches EMC NS502 Data Mover ports For all intents and purposes, the network pipes shown in Figure 2-1 on page 7 represent IP switches. In addition, the RAC interconnect connections are also on a dedicated switch. For real-world solution builds, it is recommended that these switches support Gigabit Ethernet (GbE) connections, jumbo frames, and port channelling. For example, Figure 2-4 illustrates the network ports on the rear of two EMC NS502 Data Movers. Figure 2-4 EMC NS502 Data Mover ports and traffic types Port csp handles administrative and control traffic. Port cge0 (the last character is a zero) and cge1 are aggregated and connected to the storage network. They handle all I/O required by the database servers to the datafiles, online redo log files, archived log files, control files, ocr file and voting disk. Ports cge2 and cge3 are left open for future growth. Note: Celerra Manager can be used to create Ethernet channels, link aggregations, and fail-safe networks. Storage architecture Setting up storage involves: Establishing the RAID levels 10 Storage architecture

Solution Architecture Allocating hot spares Creating disk volumes Combining disk volumes (stripes and slices) into metavolumes, or disk pools, which can span disks Creating file systems, which can be exported to servers High availability and failover The EMC Celerra NS Series has built-in high-availability features. These HA features allow the Celerra NS Series to survive various failures without a loss of access to the Oracle Database 10g database. These features protect against the following: Data Mover failure Network port failure Power loss affecting a single circuit connected to the storage array Storage processor failure Disk failure Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 11

Solution Architecture RAID type and RAID group configuration Two sets of RAID and disk configurations were tested. The first is shown in Figure 2-5. Hot Spare 4 + 1 RAID 3 8 + 1 RAID 3 Shelf 2_0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 4 + 1 RAID 5 4 + 1 RAID 5 4 + 1 RAID 5 Shelf 1_0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 4 + 1 RAID 5 Hot 1 + 1 Spare RAID 1 1 + 1 RAID 1 4 + 1 RAID 5 Shelf 0_0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Data Disk Hot Spare FC Disk ATA Disk Figure 2-5 Two FC shelf RAID configuration 12 Storage architecture

Solution Architecture The second is shown in Figure 2-6. Hot Spare 4 + 1 RAID 3 8 + 1 RAID 3 Shelf 3_0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 4 + 1 RAID 5 4 + 1 RAID 5 4 + 1 RAID 5 Shelf 2_0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 4 + 1 RAID 5 4 + 1 RAID 5 4 + 1 RAID 5 Shelf 1_0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 4 + 1 RAID 5 Hot 1 + 1 Spare RAID 1 1 + 1 RAID 1 1 + 1 RAID 1 1 + 1 Hot RAID 1 Spare Shelf 0_0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Data Disk Hot Spare FC Disk ATA Disk Disk volume setup Figure 2-6 Three FC shelf RAID configuration Once the RAID groups are created, the EMC Celerra NS Series automatically creates disk volumes that are accessible to the Data Movers. Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 13

Solution Architecture MVM vs. AVM MVM, or manual volume management, allows the greatest flexibility for the user to configure storage in whatever way he sees fit, including providing the highest performance. However, this is a more complex method for configuring storage. AVM, or automatic volume management, on the other hand, is a simpler, more automatic way of configuring storage but is not as flexible as MVM in its configuration options. For MVM, once the disk volumes are created, metavolumes are created from sliced or striped volumes. Each metavolume is set up to stripe across multiple disk volumes so that multiple disks can work concurrently. The metavolumes shown in Table 2-3 were created. For disks managed using AVM, system-defined storage pools are used. Table 2-2 shows the RAID configurations used for each solution. Table 2-2 RAID configurations per solution Solution Consolidation Consolidation Consolidation Backup Protect Configuration MVM 3 FC shelves MVM 2 FC shelves AVM 2 FC shelves AVM 2 FC shelves AVM 2 FC shelves File systems, exports and mount points In the next step, the database file systems are created, which are exported to the servers. In the case of the testing covered here the file 14 Storage architecture

Solution Architecture systems shown in Table 2-3 were created, exported, and mounted on the database servers. Table 2-3 File system layout file system / export AVM MVM /datafs /log1fs /log2fs /archfs /flashfs clar_r5_performance (system-defined storage pool consisting of all RAID 5 4+1 groups) clar_r1 (system-defined storage pool consisting of all RAID 1 groups) clar_r1 (system-defined storage pool consisting of all RAID 1 groups) clarata_r3 (system-defined storage pool consisting of all RAID 3 groups) clarata_r3 (system-defined storage pool consisting of all RAID 3 groups) stripe_data (metavol consisting of all available RAID 5 groups) meta_log1 (metavol using half of the RAID 1 groups) meta_log2 (metavol using half of the RAID 1 groups) meta_arch (metavol using the 8+1 RAID 3 group) meta_flash (metavol using the 4+1 RAID 3 group) Database server architecture Oracle Database 10g server network architecture This section describes the database server architecture. Each Oracle Database 10g server has five network interfaces. Two interfaces connect to the storage network, using a link aggregation trunk. Two interfaces connect the server to the RAC interconnect network, enabling the heartbeat and other network I/O required by Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 15

Solution Architecture Oracle cluster ready services. One interface connects to the client network. Table 2-4 lists each interface and describes its use. Table 2-4 Application server network interface configuration Interface port ID eth0 eth1 eth2 eth3 eth4 Description Client network RAC interconnect (trunked) RAC interconnect (trunked) Storage network (trunked) Storage network (trunked) Oracle cluster ready services (CRS) Oracle cluster ready services (CRS) is enabled on each of the Oracle Database 10g servers. The servers operate in active/active mode to provide local protection against a server failure and to provide load balancing. CRS required files (including the voting disk and the ocr file) can reside on NFS volumes provided that the required mount point parameters are used. See Oracle Database 10g/Oracle RAC 10g Celerra NS Series NFS Best Practices Planning for more information on the mount point parameters required for the Oracle Clusterware files. NFS client Each Oracle RAC 10g server uses the NFS protocol to connect to the Celerra NS Series network file server. NFS runs on top of TCP/IP. High availability and failover TCP/IP provides the ability to establish redundant paths for sending I/O from a networked computer to another networked computer. This approach uses the link aggregation protocol, commonly referred to as trunking. Redundant paths facilitates high availability and load balancing for the networked connection. Two NICs on each Oracle Database 10g server were used in the NFS connection, referred to above as the storage network. The RAC interconnect network was trunked in a similar manner. 16 Database server architecture

Solution Architecture Application architecture The Oracle Database 10g binary files, including the Oracle Cluster Ready Services software, were installed on the database servers local disks. Datafiles, online redo log files, archive log files, tempfiles, and CRS files resided on EMC Celerra NS Series NFS file systems. These file systems were designed (in terms of the RAID level and number of disks used) to be appropriate for each type of file. Table 2-5 lists each file or activity type and indicates where it resides. Table 2-5 Application file or activity types and locations File or activity type Database binary files Datafiles Tempfiles Online redo log files Archive log files Flashback recovery area Control files CRS ocr file CRS voting disk Location database servers local disk /datafs /datafs mirrored across /log1fs and /log2fs /archfs /flashfs mirrored across /datafs, /log1fs, and /log2fs mirrored across /log1fs and /log2fs mirrored across /datafs, /log1fs, and /log2fs Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 17

Solution Architecture 18 Application architecture

3 Hardware and Software Resources This chapter includes the following sections: Hardware resources...20 Software resources... 21 Oracle RAC 10g NS Series NFS Reference Architecture Guide 19

Hardware and Software Resources Hardware resources The Oracle RAC 10g - Celerra NS Series NFS solution uses the following hardware resources. Table 3-1 Hardware specifications Equipment Quantity Configuration Dell 1850 server Dell 5324switch Celerra NS502 networked server Two for backup and protect, four for consolidation Three for each solution (one for client, one for RAC inteconnect and one for storage) One Two 3.0-GHz Intel Pentium 4 dual-core processors 8 GB of memory for backup and protect, 12 GB for consolidation Two 73 GB 10K internal SCSI disks Two onboard 10/100/1000 MB Ethernet NICs Two additional 10/100/1000 MB Ethernet NICs (one single-port and one dual-port) 24 ports per switch Two Data Movers Four GbE network connections per Data Mover Two or three FC shelves One ATA shelf 30 or 45 146 GB FC disks (depending on configuration) Fifteen 320 GB ATA disks One Control Station 20 Hardware resources

Hardware and Software Resources Software resources The Oracle RAC 10g - Celerra NS Series NFS solution uses the software resources in Table 3-2. Table 3-2 Software specifications Software title Red Hat Enterprise Linux Oracle RAC 10g Enterprise Edition Celerra Manager Advanced Edition Number of licenses One per database server One per database server One per Celerra NS Server Oracle RAC 10g NS Series NFS Reference Architecture Guide 21

Hardware and Software Resources 22 Software resources