Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM DS3400, DS4000 Series, or DS5000 Series Storage Subsystems

Transcription

1 Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM DS3400, DS4000 Series, or DS5000 Series Storage Subsystems Technical White Paper , Rev. A March 2009 Copyright 2009, IBM Corporation

2 Copyright 2009 by International Business Machines Corporation. All rights reserved.

3 Contents CHAPTER 1: DEPLOYING ORACLE 11G ON AIX 6.1 WITH IBM DS3400, DS4000 SERIES, OR DS5000 SERIES STORAGE SUBSYSTEMS Benefits of Oracle RAC 11g Release 1 Database Example Configuration Configuring the Hosts Configuring the IBM Storage Subsystems Installing Oracle Creating Directories for Oracle Setting Up the Oracle User's Profile Installing Oracle Clusterware Installing and Configuring ASM Installing an Oracle Database Configuring Listener Names Creating and Configuring a Database Conclusion Contact Information APPENDIX A: REFERENCES AIX... A-1 IBM Storage Subsystems... A-1 Oracle... A-1 Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems iii

4 iv Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

5 Chapter 1 Deploying Oracle 11g on AIX 6.1 with IBM DS3400, DS4000 Series, or DS5000 Series Storage Subsystems Oracle Real Application Clusters (Oracle RAC) supports the deployment of a single database across a cluster of servers, providing excellent fault tolerance, performance, high availability, and scalability. Oracle RACs are growing in importance in a wide range of customer implementations across all industries for transaction processing and data warehousing applications. This document provides an overview and some tips and techniques in deploying the Oracle RAC 11g Release 1 database with Oracle RAC on the IBM AIX operating system using IBM storage subsystems. The database storage is administered using the Oracle Automatic Storage Management (Oracle ASM) feature. This document summarizes the procedure of creating a simple configuration using a two-node cluster to illustrate the principles of performing this type of implementation. Actual implementations vary in complexity and require significant expertise in all of the technologies involved. This document is intended to show any deviations from the Oracle installation guide and any enhancements in implementation because of using the DS4800 storage subsystem. This document does not provide a comprehensive reference for installing all of these components. For full implementation instructions, refer to the product documentation for the hardware being used, for the AIX operating system, for the Oracle Database, and for Oracle Clusterware. For IBM storage documentation, contact your IBM business partner or service representative. For AIX documentation, go to: doc/base/aixparent.htm For Oracle documentation, go to: Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems 1-1

6 Benefits of Oracle RAC 11g Release 1 Database Before you install Oracle Database 11g, you must know how to best configure the storage subsystem for enhanced performance. For this information, refer to Best Practices for Running Oracle RAC Databases with ASM on IBM DS4800 Storage Subsystems. Go to: Benefits of Oracle RAC 11g Release 1 Database The reference configuration shows you how to configure these products to work together to achieve numerous benefits. The reference configuration uses Oracle ASM to manage the Oracle RAC 11g Release 1 database storage running under Oracle Clusterware on AIX with an IBM DS storage subsystem. The IBM DS4800 storage subsystem is a robust, high-performance storage subsystem that provides a scalable, flexible, and expandable storage foundation upon which to build a highly reliable database system using Oracle RAC. Oracle RAC offers these benefits: High availability Provides enhanced uptime for database applications. Scalability Expands capacity by adding servers to the cluster. Lower computing costs Uses lower-cost hardware and reduces expenses associated with application downtime. Excellent performance Spreads the database load across multiple cluster members, which lets performance scale with the size of the cluster. The IBM DS3400, DS4000, and DS5000 storage subsystems offer these benefits, which go hand-in-hand with those of Oracle RAC: High availability The storage subsystem s fully redundant hardware and redundant array of independent disks (RAID) functionality makes sure that the storage is as available as the RAC database. Scalability You can add additional storage capacity to the storage subsystem while the storage subsystem continues operation. Lower costs Shared storage within a storage area network (SAN) provides reduced cost compared to storage that is local to individual hosts and reduces management expenses associated with administering the storage. Excellent performance The IBM DS3400, DS4000, and DS5000 storage subsystems are among some of the best performing storage subsystems in their class. Their full front-to-back 4-Gb Fibre Channel 2 interface architecture provides bandwidth for the most demanding applications. 1.Solution also works with DS3400, DS5000, and other DS4000 models. 2.DS3400 has a Fibre Channel host interface with a SAS disk drive interface. 1-2 Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

7 Example Configuration The DS4000 Storage Manager software package configures and manages the fully redundant hardware and high performance architecture with ease. This application provides management in-band from servers to attached storage, or remotely through the ethernet. The storage subsystems offer several premium features that add functionality when used in a database environment. Storage Partitioning feature and the Enhanced Remote Mirroring feature are two of the important features available. The Storage Partitioning Premium Feature lets you distribute logical drives on the storage subsystem to different hosts in your SAN, associating each logical drive with one particular host or a group of hosts. The logical drives are presented to each host according to the host type for the different operating systems that are running on those hosts. The Storage Partitioning premium feature provides great flexibility in the use of the storage subsystem for storage consolidation. The Enhanced Remote Mirroring premium feature provides the ability to mirror logical drives between two IBM storage subsystems. This premium feature provides an additional redundant copy of the data that you can use for rapid disaster recovery. You also can use this premium feature to create a copy of live data for test purposes. Although these premium features are not used in the simple example configuration shown in this document, many opportunities exist to use these premium features in a real-world Oracle implementation. For more information, discuss these premium features with IBM. Example Configuration The example configuration contains the following equipment: 2 IBM with IBM eserver pseries 630 servers AIX 6.1, Technology Level 1, Service Pack 2 IBM DS4800 storage subsystem Oracle Clusterware (CRS) 11.1 Oracle Database The systems are configured as a two-node RAC cluster. ASM is employed to simplify storage management. The AIX MPIO multi-pathing driver was used to provide failover support. Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems 1-3

8 Example Configuration Figure 1 1 Configuration Diagram DS4800 Storage Subsystem 1-4 Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

9 Configuring the Hosts Configuring the Hosts You must configure hosts to function correctly with Oracle RAC. The following steps describe, in general terms, how to prepare the systems for the Oracle installation. IMPORTANT Refer to Metalink Bulletin , Minimum Software Versions and Patches Required to Support Oracle Products on IBM pseries, for the latest requirements. This bulletin is available to Metalink users at: Metalink Bulletin is regularly updated, and the information that the bulletin contains supersedes any information found in this paper. 1 Install the AIX operating system if AIX is not already installed. The example configuration uses AIX 6.1 Level 1. 2 Install additional software packages required for Oracle. bos.adt.base bos.adt.lib bos.adt.libm bos.perf.libperfstat bos.perf.perfstat bos.perf.proctools rsct.basic.rte rsct.compat.clients.rte xlc.aix61.rte: (See the previous important note) xlc.rte: (See the previous important note) 3 Download and install the updates to the AIX operating system from IBM's website: The example configuration uses AIX 6.1 Level 1. 4 Configure the network interfaces of the host. The example configuration requires at least two ethernet interfaces: One for the Oracle public network One for the private network used by Oracle Clusterware Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems 1-5

10 Configuring the Hosts Oracle recommends that you put the private interface on a separate network isolated from all other traffic. Use a separate gigabit ethernet switch for the private network. Consider using additional interfaces for most deployments to provide more flexibility in configuration and higher performance. In the example configuration, the VIP network shares the interface used for the public network. This network is configured later in the installation process using the Vipca utility. Entries for the internet protocol (IP) address, and the public and private networks, are inserted into the /etc/hosts file. 5 Create users and groups. a Create two groups, dba and oinstall. b Add a user, Oracle, with the oinstall primary group and with the dba group. NOTE Make sure that the user ID for the Oracle user and the group numbers for the dba group and the oinstall group are consistent on all nodes. 6 Set the shell soft limits for the Oracle and root users using the smit chuser command. Soft FILE size = -1 Soft CPU time = -1 Soft DATA segment = -1 Soft STACK size = -1 7 Use the smit chgsys command to set the maximum number of processes for each user to at least Use the /usr/sbin/no -a command to examine the minimum values for the following network tuning parameters. ipqmaxlen = 512 rfc1323 = 1 sb_max = (2*655360) tcp_recvspace = tcp_sendspace = udp_recvspace = (10 * udp_sendspace, must be less than sb_max) udp_sendspace = Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

11 Configuring the Hosts NOTE If you modify these parameters, the method depends on whether the system is in compatibility mode. For specific instructions, refer to the AIX documentation for the no command. If you change the value of the ipqmaxlen parameter, you must restart the system for the change to take effect. 9 Download and install the Open Secure Shell (SSH) package from the IBM web site. NOTE OpenSSH also requires OpenSSL, which you can download from the IBM website. This software is cryptographic software and the website requires that you log on to download this type of package. The example configuration uses OpenSSH. The SSH utility is not required for Oracle to function, however this utility is recommended because the SSH utility provides a more secure method of communication between nodes than does rsh. If SSH is available, Oracle uses SSH. Otherwise, Oracle uses rsh. Configuration of rsh user equivalence is not described in this document. 10 Configure SSH for root login permission. a Locate the sshd_config (/etc/ssh/sshd_config) file. b Change the sshd_config file to change PermitRootLogin to yes. c Use the startsrc -s sshd command to start sshd. 11 Configure SSH for user equivalents for root and oracle on all nodes. a Log on as root. b Use these commands to generate the SSH key. cd ~/.ssh ssh-keygen -t dsa cat id_dsa.pub >> authorized_keys c Copy the authorized_keys file to ~/.ssh file on the next system. d Repeat step b and step c on each system. e When all systems have a public key in the file, copy the file back to all of the previous systems so that they all have an identical file. f Test the SSH function from each system to itself and to each of the other systems by issuing the ssh host1 date command. Make sure that the function completes without using a password. Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems 1-7

12 Configuring the Hosts g Log on as the oracle user and repeat step 11so that Oracle also can perform remote commands without passwords. NOTE The permissions on both the base user directory and the.ssh directory must restrict write access. If the permissions are not correctly set, OpenSSH refuses to provide access without a password, even if the appropriate public key is in the authorized_keys file. 12 Change the.profile file in the Oracle user s home directory to set the Oracle user's parameters. Table 1 1 Oracle User s Parameters Parameter to Add Description umask 022 AIXTHREAD_SCOPE=S CRS_HOME ASM_HOME ORACLE_HOME LD_LIBRARY_PATH ORACLE_BASE ORACLE_SID PATH CLASSPATH An environment variable that directs the system to place user threads within a system-wide scope rather than a process-wide contention scope. This placement provides better performance and more efficient memory use. A unique home directory for Oracle CRS binaries. Use this parameter if you want to install separate Oracle binaries for Oracle ASM. This method is preferred if more than one database instance is supported by Oracle ASM. A unique home for the database binaries. The complete pathway is $ORACLE_HOME/lib. Identifies the Oracle base directory. Identifies the instance. Include $ORACLE_HOME/bin. Include $ORACLE_HOME/jdbc/lib/ <classes files>. 13 Change the.profile file in the root user s home directory to set the Oracle user's parameters. Add this entry to the.profile file: AIXTHREAD_SCOPE=S 14 Synchronize the system time between the hosts in the cluster using any effective method. If an Network Time Protocol (NTP) server is available, use this method to set up xntpd. a Change the /etc/ntp.conf file to add or change the name or IP address of the NTP servers. 1-8 Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

13 Configuring the IBM Storage Subsystems b Add a line server <hostname> for each host. c Start the service using the smit xntpd command. The example configuration uses NTP to automatically perform this function. 15 Because some Oracle utilities expect lsattr in the /etc directory and cannot function correctly without a link to lsattr, use the ln -s /usr/sbin/ lsattr /etc/lsattr command to add a soft link. Configuring the IBM Storage Subsystems IMPORTANT You must know how best to configure the storage for enhanced performance. For this information, refer to Best Practices for Running Oracle RAC Database with ASM on an IBM DS4800 Storage Subsystem. Go to: WP Use the DS4000 Storage Manager application and various AIX system utilities to configure the IBM storage subsystem. 1 Use the DS4000 Storage Manager utility to create logical drives on the storage subsystem that you can share for CRS communication and database storage. NOTE You also might use additional logical drives for Oracle binaries and similar purposes. However, using logical drives local to one particular system requires the Storage Partitioning premium feature. Analyze your implementation to determine the appropriate logical drive sizes, wanted RAID levels, and other parameters. The reference configuration uses the following minimal configuration of shared logical drives: One Oracle Cluster Registry (OCR) logical drive (300 MB, RAID 1) One Clusterware voting disk drives (300 MB, RAID 1) One logical drive for Oracle ASM's SPFILE (100 MB, RAID 1) Two logical drives for Oracle ASM data storage (300 GB, RAID 5) The number and the size of the logical drives created for Oracle ASM storage varies based on the individual implementation, but the sizes used for the OCR disk drives, voting disk drives, and the Oracle ASM SPFILE disk drives in the example configuration are sufficient for any configuration. In the reference configuration, the Oracle binaries are installed on internal disk drives within each individual server. This configuration uses the JFS2 file system and is mounted as /u01. Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems 1-9

14 Configuring the IBM Storage Subsystems Configure the shared disk drives in the operating system. After you have created the logical drives on the storage subsystem, and mapped to the servers, the cfgmgr command rescans the AIX system for changes to the physical devices. The lspv command shows the physical devices that AIX has configured. The mpio_get_config -Av command displays LUN numbers, disk drive name, and the associated logical drive names configured on the storage subsystem. 3 If the hdisk names for each logical drive on the storage subsystem that the system can share in the cluster are not identical on all cluster nodes, create new device nodes for each of the logical drives to give them a consistent name across all nodes. For example, if the logical drive for the first OCR disk drive was hdisk2 on node1 and hdisk3 on node2, you can create a new devnode called /dev/ora_ocr1 on each host. The new devnode uses the major numbers and the minor numbers that correspond with the same logical drive on each system. Show the major numbers and the minor numbers with the mknod /dev/ora_ocr1 c <major# of OCR> <minor# of OCR> command to create the new devnode. Even when the disk drive names correlate on all of the nodes, still create these named device nodes for easier location and administration. Named device nodes were created on the reference configuration for all shared logical drives using the convention /dev/ora_<use>. 4 Set the reserve_policy attribute for each logical drive that you want to use for shared access for OCR logical drives, voting logical drives, and Oracle ASM logical drives. For an IBM storage subsystem, reserve_policy=no_reserve is the correct value. This value lets the nodes share the logical drives instead of permitting the first system that accesses the logical drive to reserve the logical drive for exclusive use. Oracle coordinates the access to the shared logical drives. NOTE You must perform this operation on the hdisk* device name rather than on the named devnode created in step 4. a Use the lsattr -E -l hdisk? grep reserve command to check the policy. b Use the chdev -l hdisk? -a reserve_policy=no_reserve command to change the policy. c Repeat step a and step b on each cluster node Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

15 Installing Oracle 5 Set ownership and permissions on devnodes for use by Oracle. NOTE If you created new devnodes for the shared logical drives, set the ownership and permissions on the new one that you created, not the original hdisk* devnode. a Use these commands to set the ownership for each OCR device file: chown root:oinstall /dev/<devnode> chmod 640 /dev/<devnode> b Use these commands to set the ownership and the permission on the voting disk drive and database device files: chown oracle:dba /dev/<devnode> chmod 660 /dev/<devnode> c Repeat step a and step b on each cluster node. NOTE Do not assign a physical volume identifier (PVID) to hdisks for use with Oracle ASM. Installing Oracle Here are the basic steps for installing Oracle: Create directories for Oracle. Set up the Oracle user s profile. Install Oracle Clusterware. Install and configure Oracle ASM. Install the Oracle Database. Configure the listener names. Create and configure a database. Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems 1-11

16 Installing Oracle Creating Directories for Oracle Although you can create directory structures for Oracle files in many different ways, for ease of support, use a standard directory structure based on the Optimal Flexible Architecture (OFA) guidelines published by Oracle. For more information about the OFA Guidelines, see Appendix D of the Oracle Database Installation Guide 11g Release 1 (11.1) for AIX Based Systems (B ). The example configuration is based on the OFA guidelines. Follow these two important guidelines when creating directories for Oracle: You must separate the Oracle Clusterware home directory from the database home directories. If more than one database is serviced by Oracle ASM, create a separate home directory for Oracle ASM. An Oracle ASM home directory lets you modify the Oracle release for Oracle ASM while not changing the release of any of the databases on the same server. The directory tree must be owned by oracle. Setting Up the Oracle User's Profile 1 Before you install the Oracle binaries, set up the Oracle user's.profile. 2 Add or modify entries for ORACLE_HOME, ORACLE_BASE, PATH, LD_LIBRARY_PATH, and similar environment variables to reflect the directories where the Oracle binaries are placed during the Oracle Clusterware installation. Installing Oracle Clusterware As you install the Oracle Clusterware, keep the following important points in mind: Because of permission requirements for Oracle CRS, do not make the CRS_HOME path a subdirectory of the Oracle base directory. For more information, refer to Appendix E of the Oracle Clusterware Installation Guide. At the end of the installation process, run the root.sh script as the root user and run this script on only one node at a time, not concurrently. Installing and Configuring ASM Because a single ASM instance can service many database instances, Oracle recommends that you install ASM in its own home directory. Follow the naming convention of the OFA guidelines for the directory path. As you install and configure Oracle ASM, keep the following important points in mind: If the Oracle ASM instance was not correctly created by the Oracle Universal Installer Wizard during installation, use the dbca utility to add the instance and to create the disk groups Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

17 Installing Oracle If you want additional disk groups, create them using the dbca utility. NOTE For more detailed information on configuring ASM for an Oracle database, refer to Oracle Best Practices for Running Oracle RAC Database with ASM on IBM DS4800 or DS5000 Series Storage Subsystems. Installing an Oracle Database If you want a separate database home, use the Oracle Universal Installer to install another database home for a database instance. Select Install Database Software Only at the Create Database page. Configuring Listener Names If your RAC environment contains both an Oracle ASM instance and a database instance, supporting only one listener.ora configuration file for the Oracle ASM and database instances is easier. This environment variable lets the DBA maintain one set of tnsnames.ora, sqlnet.ora, and listener.ora files for the entire environment. Creating and Configuring a Database Use the Database Configuration Assistant (DBCA) to create a database. For more information, refer to the Oracle documentation about using DBCA. As you create the database, keep the following important points in mind: Using DBCA provides you with the capability to create the Oracle ASM disk groups if you choose to use Oracle ASM. The best way to configure listener names is to create an environment variable TNS_ADMIN and to assign the variable to the $ORACLE_HOME/network/ admin directory of the database instance. For example: export TNS_ADMIN=$ORACLE_HOME/network/admin Using the Flash Recovery Area (FRA) option is highly recommended. A Flash Recovery Area permits the setup of a separate area on the disk drive for flashback capability, an area for the archive logs, a second copy of the multiplexed redo log files, and an area for RMAN backups. Make sure that the FRA is correctly setup to handle the performance of the database and that the FRA contains enough space. Space to hold all of the data for the data retention periods that are set by the database Space for the type of backups to use Space for the number of backups to hold Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems 1-13

18 Conclusion For more information about using the Flash Recovery Area, refer to the Oracle documentation and the following white paper: Best Practices for Running Oracle RAC Database with ASM on an IBM DS4800 (Models 82, 84, 88) Storage Subsystem. Go to: Conclusion As you have followed the example of setting up a two-node cluster described in this document, you have seen the simplicity and effectiveness of a deployment of Oracle RAC 11g Release 1. When you are performing your own installation, make sure to use the applicable installation information as the basis for your work. This document describes only tips and techniques. The person who installs Oracle 11g should have the expertise to know how to and when to modify instructions as necessary. If your environment is complex and you require outside help, contact your IBM business partner or sales representative. Contact Information For more information and sales office locations, visit the IBM web site at: Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

19 Appendix A References AIX For AIX operating system downloads, go to Fix Central: For AIX documentation, go to AIX Information: doc/base/aixparent.htm IBM Corporation, Get the Latest Version of OpenSSH for AIX. IBM Storage Subsystems Oracle For information about the IBM DS3000, DS4000, and DS5000 storage subsystems, go to: IBM Corporation, Best Practices for Running Oracle RAC Database with ASM on an IBM DS4800 Storage Subsystems. For Oracle documentation, go to: IBM Corporation, Tuning IBM AIX 5L for an Oracle Database. VPAA-6V2C4E Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems A-1

20 Oracle A-2 Deploying Oracle RAC 11g Release 1 with ASM on AIX 6.1 with IBM Storage Subsystems

21 Trademarks and Special Notices Copyright IBM Corporation All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

22