Supplemental Implementation Guide

Transcription

1 Dell EqualLogic Best Practices Series Supplemental Implementation Guide Sizing and Best Practices for Deploying Oracle 11g Transaction Processing Databases on Dell EqualLogic Storage Chidambara Shashikiran Storage Infrastructure and Solutions Dell Product Group October 2010

2 Table of Contents 1 Introduction Audience Test System Configuration ORION Test Configuration TPC-C Test Configuration System Configuration Red Hat Enterprise Linux Installation Broadcom NIC Driver Installation Ethernet Bonding Configuration Configure Jumbo Frames iscsi Initiator Installation MPIO Configuration Blacklist Internal Disk Devices Set EqualLogic Device Specific Settings in Multipath.conf Setup Persistent Multipath Device Mapping Installation and Configuration of the Oracle 11g RAC Database Oracle GRID Infrastructure Kernel parameters IP Addresses System security settings Shell process resources Hangcheck-Timer Module Configure System Services SSH Equivalency ASM Configuration Oracle Grid Infrastructure and Database Creation Appendix A Switch Configuration A.1 PowerConnect 8024F SAN Switch Configuration A.2 PowerConnect 6248 RAC Switch Configuration ii

3 1 Introduction This paper provides a supplemental technical implementation guide for use in conjunction with the following Dell whitepaper: Sizing and Best Practices for Deploying Oracle 11g Transaction : This paper is for systems engineers and administrators that are interested in additional system configuration information that was not included in the sizing and best practices paper. The information in this document contains configuration details that are specific to a test environment created at Dell Labs. The design and configuration methods in this document provide examples that Solution Architects and Storage Consultants can adapt to their own solution design requirements. 1.1 Audience This paper is for professional services implementation engineers and end user system administrators that need to understand how to configure Red Hat Enterprise Linux based systems to support Oracle Database deployments in an Oracle RAC configuration that uses Dell EqualLogic iscsi SAN storage systems. Readers should first refer to the Sizing and Best Practices whitepaper mentioned above, and then refer to this paper for additional system configuration guidance. We assume the reader has prior experience in configuring and operating the following: Dell PowerEdge servers Dell PowerConnect switches Dell EqualLogic PS Series SANs Red Hat Enterprise Linux Oracle Real Application Clusters (RAC) This paper does not provide a comprehensive implementation guide for installing and running Oracle RAC on Red Hat. It provides details for accomplishing select portions of the system configuration and Oracle RAC installation. See XXX for a process overview and specific process details included in this document. 1

4 2 Test System Configuration We created two test configurations in this project. The first configuration supported Oracle I/O Calibration Tool (ORION 1 ) testing. The second configuration supported running Quest Benchmark Factory 2 for Databases to simulate a real TPC-C style workload using an actual Oracle 11g R2 RAC database. 2.1 ORION Test Configuration ORION test configuration components: A single Dell PowerEdge R710 server running Red Hat Enterprise Linux 5.5 o Two Broadcom dual-port 10GbE NICs installed in the R710 server Two Dell PowerConnect 8024F switches (iscsi SAN) One Dell EqualLogic PS6010XV array o 14 x 300GB 15K RPM SAS disk drives configured as RAID 10 (two hot spares). o Dual 10GbE dual-port controllers running firmware version One Dell EqualLogic PS6101S array o 14 x 100GB Solid-state disk drives configured as RAID 10 (two hot spares). o Dual 10GbE dual-port controllers running firmware version Figure 1 ORION Test System

5 2.2 TPC-C Test Configuration TPC-C test system components: Two Dell PowerEdge R710 servers running Red Hat Enterprise Linux 5.5 (database servers) o Two Broadcom dual-port 10GbE NICs installed in each R710 server Two Dell PowerEdge R610 servers running VMware vsphere ESX version 4.1 o Hosting Windows Server 2008 R2 Enterprise Edition in virtual machines (workload generators) Two Dell PowerConnect 6248 switches (Oracle RAC interconnect) Two Dell PowerConnect 8024F switches (iscsi SAN) One Dell EqualLogic PS6010XV array o 14 x 300GB 15K RPM SAS disk drives configured as RAID 10 (two hot spares). o Dual 10GbE dual-port controllers running firmware version Figure 2 TPC-C Test System 3

6 Additional TPC-C system configuration details: We created an Oracle Real Application Cluster (RAC) using the two R710 servers. Three of the 1GbE Broadcom 5709 LOM NICs on each R710 server were trunked together and used for RAC interconnection (connecting to the PowerConnect 6248 switches). We used Oracle ASM cluster file system (ACFS) to store the Oracle clusterware files (OCR and voting disk). Test configuration SAN design references: EqualLogic Configuration Guide Dell EqualLogic PS Series Network Performance Guidelines 3 System Configuration Only selected portions of the entire solution deployment process are included in this document. The solution deployment process breaks out into two main phases: 1. System Configuration (this section) 2. Oracle 11g RAC Installation and Configuration (see Section 4) Figure 3 shows a high-level outline of all system configuration steps. We include implementation details in this document for the highlighted process steps shown in Figure 3. 4

7 Figure 3 System Configuration Process Overview 3.1 Red Hat Enterprise Linux Installation Install the operating system using the Red Hat Enterprise Linux 5 Installation Guide 3. In addition to the default operating system installation, install the packages listed in Example 1 to configure the system to support Oracle Database, iscsi and Oracle ASM. Example 1 Additional RHEL Packages binutils el5.x86_64.rpm compat-libstdc i386.rpm compat-libstdc x86_64.rpm 3 5

8 elfutils-devel-static el5.i386.rpm elfutils-devel-static el5.x86_64.rpm elfutils-libelf el5.i386.rpm elfutils-libelf el5.x86_64.rpm elfutils-libelf-devel el5.i386.rpm elfutils-libelf-devel el5.x86_64.rpm elfutils-libelf-devel-static el5.i386.rpm elfutils-libelf-devel-static el5.x86_64.rpm gcc el5.x86_64.rpm gcc-c el5.x86_64.rpm glibc x86_64.rpm glibc-common x86_64.rpm glibc-devel i386.rpm glibc-devel x86_64.rpm glibc-headers x86_64.rpm kernel-headers el5.x86_64.rpm ksh el5.x86_64.rpm libaio i386.rpm libaio x86_64.rpm libaio-devel i386.rpm libaio-devel x86_64.rpm libgcc el5.i386.rpm libgcc el5.x86_64.rpm libgomp el5.i386.rpm libgomp el5.x86_64.rpm libstdc el5.i386.rpm libstdc el5.x86_64.rpm libstdc++-devel el5.i386.rpm libstdc++-devel el5.x86_64.rpm make el5.x86_64.rpm numactl-devel el5.i386.rpm numactl-devel el5.x86_64.rpm sysstat el5.x86_64.rpm unixodbc i386.rpm unixodbc x86_64.rpm unixodbc-devel i386.rpm unixodbc-devel x86_64.rpm ### Packages required for iscsi and Oracle ASM: iscsi-initiator-utils el5 oracleasm el el5.x86_64.rpm oracleasmlib el5.x86_64.rpm oracleasm-support el5.x86_64.rpm 3.2 Broadcom NIC Driver Installation We recommend using the latest supported version of the Broadcom NIC drivers. Go to the Drivers and Downloads section of to obtain the latest supported drivers. The ReadMe file provided by the driver download includes instructions for performing the update. 6

9 Both 1GbE and 10GbE NIC devices were installed on our test server. For Broadcom Netxtreme NICs, the alias for the 1GbE interfaces is bnx2, For the 10GbE interfaces it is bnx2x. The contents of the system file /etc/modprobe.conf will show which physical NIC devices correspond to each. Example 2 shows that eth[0-3] are the 10GbE devices and eth[4-7] are the 1GbE devices. Example 2 Determining NIC type from modprobe.conf [root@sieora1 ~]# cat /etc/modprobe.conf alias eth0 bnx2x alias eth1 bnx2x alias eth2 bnx2x alias eth3 bnx2x alias eth4 bnx2 alias eth5 bnx2 alias eth6 bnx2 alias eth7 bnx2 alias scsi_hostadapter megaraid_sas alias scsi_hostadapter1 ata_piix alias scsi_hostadapter2 usb-storage You can identify the current NIC driver versión using the modinfo command. Example 3 shows the version provided by our Red Hat installation. Example 3 Determining currently loaded NIC driver version using modinfo command [root@sieora1 ~]# modinfo bnx2x filename: /lib/modules/ el5/kernel/drivers/net/bnx2x.ko version: license: GPL description: Broadcom NetXtreme II BCM57710/57711/57711E Driver (command output truncated) You should update the NIC driver to the latest supported versión provided by Dell. Go to the Drivers and Downloads page at We updated the kernel driver for the Broadcom Netxtreme II 10GbE NIC to the latest version. See the README.TXT file in the driver download package for installation notes and a general upgrade procedure. Example 4 shows the installation process we used to update the driver. Example 4 Install, Build and Replace the NIC driver Install the source RPM package: [root@sieora1 ~]# rpm -ivh netxtreme src.rpm 1:netxtreme2 ########################################### [100%] Build the new package: [root@sieora1 redhat]# cd /usr/src/redhat/specs 7

10 SPECS]# rpmbuild -bb./netxtreme2.spec Install the new package: x86_64]# cd /usr/src/redhat/rpms/x86_64 x86_64]# ls netxtreme x86_64.rpm x86_64]# rpm -ivh netxtreme x86_64.rpm Preparing... ########################################### [100%] 1:netxtreme2 ########################################### [100%] Unload the current driver: x86_64]# rmmod bnx2x Replace the current driver with the new one: x86_64]# cd /lib/modules/ el5/kernel/drivers/net x86_64]# insmod bnx2x.ko Confirm the module update: ~]# modinfo bnx2x filename: /lib/modules/ el5/kernel/drivers/net/bnx2x.ko version: license: GPL description: Broadcom NetXtreme II BCM57710/57711/57711E Driver (command output truncated) Important: restart system network services to complete the driver replacement process. 3.3 Ethernet Bonding Configuration Note in Figure 2 where three 1GbE NICs were trunked or bonded on each R710 database server to create two 3Gb aggregate channels between the servers and the RAC interconnect switches. For each database server, the NIC bonding process consisted of four steps: 1. Create a new file called ifcfg-bond0 for the new virtual device bond0. 2. Modify scripts in /etc/sysconfig/network-scripts to add physical interfaces to the virtual device. 3. Modify /etc/modprobe.conf to set bonding options and an interface alias 4. Restart the network interfaces and confirm using ifconfig. Example 5 Ethernet Bonding Configuration Create the ifcfg-bond0 file in /etc/sysconfig/network-scripts for the virtual device bond0. [root@sieora1 network-scripts]# cat ifcfg-bond0 # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet DEVICE=bond0 8

11 BOOTPROTO=none IPADDR= NETMASK= ONBOOT=yes MTU=9000 Modify the ifcfg-ethx scripts for the affected interfaces. In this case eth5, eth6 and eth7 are bonded to the bond0 virtual interface. network-scripts]# cat ifcfg-eth5 # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet DEVICE=eth5 BOOTPROTO=none HWADDR=A4:BA:DB:27:A0:11 ONBOOT=yes MASTER=bond0 SLAVE=yes TYPE=Ethernet [root@sieora1 network-scripts]# cat ifcfg-eth6 # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet DEVICE=eth6 BOOTPROTO=none HWADDR=A4:BA:DB:27:A0:13 ONBOOT=yes MASTER=bond0 SLAVE=yes TYPE=Ethernet [root@sieora1 network-scripts]# cat ifcfg-eth7 # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet DEVICE=eth7 BOOTPROTO=none HWADDR=A4:BA:DB:27:A0:15 ONBOOT=yes MASTER=bond0 SLAVE=yes TYPE=Ethernet Add the following lines to /etc/modprobe.conf Note: mode=4 specifies use of IEEE 802.3ad Link Aggregation Mode (LAG). These interfaces were connected the ports in channel-group 1 on the PowerConnect 6248 switch (see Example 32 in Appendix A.2) options bonding max_bonds=1 mode=4 alias bond0 bonding Restart the network interfaces and use ifconfig to check the bonding configuration. [root@sieora1 ~]# ifconfig bond0 bond0 Link encap:ethernet HWaddr A4:BA:DB:27:A0:11 inet addr: Bcast: Mask: inet6 addr: fe80::a6ba:dbff:fe27:a011/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1 RX packets: errors:0 dropped:0 overruns:0 frame:0 9

12 TX packets: errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes: (298.8 MiB) TX bytes: (32.3 MiB) ~]# ifconfig eth5 eth5 Link encap:ethernet HWaddr A4:BA:DB:27:A0:11 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets: errors:0 dropped:0 overruns:0 frame:0 TX packets:88943 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes: (99.4 MiB) TX bytes: (10.9 MiB) Interrupt:122 Memory:d d ~]# ifconfig eth6 eth6 Link encap:ethernet HWaddr A4:BA:DB:27:A0:11 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets: errors:0 dropped:0 overruns:0 frame:0 TX packets:87861 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes: (99.7 MiB) TX bytes: (10.7 MiB) Interrupt:130 Memory:d d ~]# ifconfig eth7 eth7 Link encap:ethernet HWaddr A4:BA:DB:27:A0:11 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets: errors:0 dropped:0 overruns:0 frame:0 TX packets:87861 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes: (99.7 MiB) TX bytes: (10.7 MiB) Interrupt:138 Memory:d d Configure Jumbo Frames We used jumbo frames for the server NICs connected to iscsi SAN, and for the server NICs used for RAC interconnections. We accomplished this by setting MTU=9000 in the interface configuration file for the NIC. We repeated this change (shown in Example 6) for each NIC configuration file that required jumbo frame support. Example 6 Set NIC MTU for jumbo frame support [root@sieora1 ~]# cd /etc/sysconfig/network-scripts [root@sieora1 network-scripts] cat ifcfg-eth1 # Broadcom Corporation NetXtreme II BCM Gigabit PCIe DEVICE=eth1 BOOTPROTO=static HWADDR=00:10:18:67:D6:46 IPADDR= NETMASK= ONBOOT=yes MTU=9000 Important: restart system network services to load the new setting on the NICs 10

13 Use the ping command to verify. network-scripts]# ping -I eth1 -s PING ( ) from eth1:9000(9028) bytes of data bytes from : icmp_seq=1 ttl=255 time=0.841 ms 9008 bytes from : icmp_seq=2 ttl=255 time=0.202 ms 3.5 iscsi Initiator Installation Installation and configuration of the iscsi initiator in the host operating system is required. The Red Hat Enterprise Linux distribution includes the required iscsi-initiator-utils package. Example 7 shows the steps taken to install and configure the iscsi service. Example 7 Configure iscsi Check availability: [root@sieora1 ~]# service iscsi iscsi: unrecognized service Install the iscsi initiator utils package: [root@sieora1 ~]# rpm -ivh iscsi-initiator-utils warning: iscsi-initiator-utils el5.x86_64.rpm gnature: NOKEY, key ID Preparing... ################################ 1:iscsi-initiator-utils ################################ Start the iscsi service: [root@sieora1 ~]# service iscsi start Starting iscsi daemon [ OK ] [ OK ] Setting up iscsi targets: iscsiadm: No records found! [ OK ] [root@sieora1 ~]# service iscsi status iscsid (pid 3447) is running... Automatically start the iscsi service at system boot: [root@sieora1 ~]# chkconfig --add iscsi [root@sieora1 ~]# chkconfig iscsi on [root@sieora1 ~]# chkconfig --list iscsi iscsi 0:off 1:off 2:on 3:on 4:on 5:on 6:off 3.6 MPIO Configuration The first step in this process is to make sure the multipath daemon is enabled and running. 11

14 Example 8 Check multipath daemon [root@sieora1 ~]# chkconfig --add multipathd [root@sieora1 ~]# chkconfig multipathd on [root@sieora1 ~]# [root@sieora1 ~]# chkconfig --list multipathd multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@sieora1 ~]# service multipathd start Starting multipathd daemon: [ OK ] Before discovering iscsi targets, you will need to make target volumes available using the EqualLogic Group Administrator. Assuming you volumes with appropriate access control settings available, you use the iscsiadm command to discover all available iscsi targets. Example 9 shows target discovery on our test system. Example 9 Using iscsiadm to discover targets [root@sieora1 ~]# iscsiadm -m discovery -t st -p : :3260,1 iqn com.equallogic:0-8a dbe6406- d a44bf5e-orion :3260,1 iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion :3260,1 iqn com.equallogic:0-8a fbe d193516a74bf5e-orion :3260,1 iqn com.equallogic:0-8a0906-6fdbe6406- ec a14bf5e-orion2 Now we use iscsiadm again to login to each target and verify connection status. For each target, you will see output similar to Example 10. Example 10 only shows orion1 target information. Example 10 Login to targets and verify connection status [root@sieora1 ~]# iscsiadm -m node -l Logging in to [iface: default, target: iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1, portal: ,3260] Login to [iface: default, target: iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1, portal: ,3260]: successful... [root@sieora1 ~]# iscsiadm -m session tcp: [1] :3260,1 iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1... You must bind the NIC interface hardware addresses to the physical NIC ports for the MPIO configuration. Example 11 shows how we completed this step in our test system, where we configure interfaces eth1 and eth3 for MPIO. 12

15 Example 11 Configure network devices for MPIO First, create the interfaces for MPIO: ~]# iscsiadm -m iface -I eth1 -o new New interface eth1 added ~]# iscsiadm -m iface -I eth3 -o new New interface eth3 added Now update the interface name for each port. (This step binds the NIC interface hardware address (the MAC address) to the physical NIC ports.) ~]# iscsiadm -m iface -I eth1 --o=update -n iface.net_ifacename -v eth1 eth1 updated. ~]# iscsiadm -m iface -I eth3 --o=update -n iface.net_ifacename -v eth3 eth3 updated. Verify that the iscsi interfaces and actual hardware interfaces are mapped together (the iface.iscsi and iface.net values match): ~]# cat /var/lib/iscsi/ifaces/eth1 # BEGIN RECORD iface.iscsi_ifacename = eth1 iface.net_ifacename = eth1 iface.transport_name = tcp # END RECORD [root@sieora1 ~]# cat /var/lib/iscsi/ifaces/eth3 # BEGIN RECORD iface.iscsi_ifacename = eth3 iface.net_ifacename = eth3 iface.transport_name = tcp # END RECORD At this point you must log out of all targets, then rediscover, then log back in to all targets. [root@sieora1 ~]# iscsiadm -m session logout... When you rediscover targets below you will see two paths per volume (output for only one volume shown below): [root@sieora1 ~]# iscsiadm -m discovery -t st -p : :3260,1 iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion :3260,1 iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1 Now login to all targets. The login output below shows two MPIO paths to a single volume, orion1, using interfaces eth1 and eth3. [root@sieora1 ~]# iscsiadm -m node l... 13

16 Logging in to [iface: eth1, target: iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1, portal: ,3260] Logging in to [iface: eth3, target: iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1, portal: ,3260]... Login to [iface: eth1, target: iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1, portal: ,3260]: successful Login to [iface: eth3, target: iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1, portal: ,3260]: successful Now you must register the multipath devices. Execute the command multipath -v2 to register all paths. Then execute multipath -ll to view registered paths. Example 12 shows a portion of the output, illustrating how the path mpath9 has two physical devices assigned to it. By default, the system will register paths with mpathx naming, where X is a unique number for each path. Also, note how the sdp and sdq devices are associated to the same path. Example 12 Default Multipath Device Naming [root@sieora1 ~]# multipath -ll mpath9 (36090a06840e6db6f5ebf146a519311ec) dm-10 EQLOGIC,100E-00 [size=50g][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 29:0:0:0 sdp 8:240 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 30:0:0:0 sdq 65:0 [active][ready] Blacklist Internal Disk Devices You will need to blacklist internal disk devices. In the multipath configuration file all devices are blacklisted by default. The system internal disks in our configuration were labeled /sda1 and /sda2. We modified the blacklist section in /etc/multipath.conf as shown in Example 13 below to allow MPIO for all devices except for the system internal disks. Example 13 Blacklist section modifications in multipath.conf blacklist { # devnode "*" devnode "^sd[a]$" } Note: It is important to add the $ symbol to the pattern match shown in Example 13, otherwise devices names in the range /dev/sdaa to /dev/sdaz will also be blacklisted because they would match sda. This becomes an issue if the machine has more than 26 /dev/sd* devices. For example, if you have at least seven LUNs assigned to the machine with four paths per LUN then you would end up with more than 26 devices. 14

17 3.6.2 Set EqualLogic Device Specific Settings in Multipath.conf You will need to set EqualLogic specific settings in multipath.conf. We used the device section of multipath.conf to set attribute values that are specific for tuning EqualLogic PS Series SANs for OLTP I/O workloads. Example 14 shows the modifications we made in multipath.conf. These modifications will apply to all EqualLogic devices. Example 14 multipath.conf attributes for EqualLogic device { vendor "EQLOGIC" product "PS6010" path_grouping_policy multibus getuid_callout "/sbin/scsi_id -g -u -s /block/%n" features "1 queue_if_no_path" path_checker readsector0 failback immediate path_selector "round-robin 0" rr_min_io 20 rr_weight priorities } Setup Persistent Multipath Device Mapping Next, you need to establish persistent multipath device mapping. Example 15 shows the mpathx path names assigned to each path in our test system. There is no guarantee that this mpathx naming sequence will remain the same after a system restart. You should assume they will not. To compensate for this, you need to establish persistent multipath device mapping. Example 15 Automatically Assigned mpathx Names [root@sieora1 ~]# multipath -ll grep mpath mpath2 (36090a06840e65b835ebf046b ) dm-3 EQLOGIC,100E-00 mpath1 (36090a06840e6db745ebf446a519311d2) dm-2 EQLOGIC,100E-00 mpath0 (36090a06840e6fb6c5ebfe f) dm-1 EQLOGIC,100E-00 mpath9 (36090a06840e6db6f5ebf146a519311ec) dm-10 EQLOGIC,100E-00 mpath8 (36090a06840e6eb895ebf646b5193d1f1) dm-9 EQLOGIC,100E-00 mpath7 (36090a06840e6cb8b5ebf946b5193d12d) dm-8 EQLOGIC,100E-00 mpath6 (36090a06840e61b875ebf346b5193d193) dm-7 EQLOGIC,100E-00 mpath5 (36090a06840e6cb7a5ebfa46a519341f2) dm-6 EQLOGIC,100E-00 mpath4 (36090a06840e62b7f5ebfd46a ) dm-5 EQLOGIC,100E-00 mpath3 (36090a06840e6fb765ebf746a5193d108) dm-4 EQLOGIC,100E-00 To establish persistent device mapping, you create aliases for each UUID in the multipath.conf file. This is a four-step process: 1. Get the device IDs corresponding to each volume 2. Get the UUIDs corresponding to each device 3. For each volume, create a volume alias in multipath.conf that associates the volume to its UUID 15

18 4. Reregister the multipath devices to load the persistent device mapping Step 1: Get the device IDs corresponding to each volume Example 16 shows how to obtain the device ID for each volume. This example shows that for the volume orion1 the corresponding device ID is sds. Example 16 Finding the Device ID for a Volume [root@sieora1 ~]# ls -l /dev/disk/by-path grep orion1 lrwxrwxrwx 1 root root 9 May 22 02:50 ip :3260,1 iqn com.equallogic:0-8a0906-6cfbe6406-0f e4bf5e-orion1-lun-0 ->../../sds Step 2: Get the UUIDs corresponding to each device Example 17 shows how to find the UUID corresponding to each device. This example shows how to get the UUID for the device labeled sds. Example 17 Finding the UUID for the Device [root@sieora1 ~]# ls -l /dev/disk/by-id grep sds lrwxrwxrwx 1 root root 9 May 22 02:50 scsi-36090a06840e6fb6c5ebfe f - >../../sds Step 3: Create a volume alias in multipath.conf to associate the volume to its UUID Example 18 shows how to create an alias for each volume in multipath.conf. You will need to create a multipath alias similar to the one shown in the example for each volume that requires multipath access. Example 18 Create a Volume Alias for the UUID in multipath.conf multipaths {... multipath { wwid 36090a06840e6fb6c5ebfe f alias orion1 rr_min_io 20 } multipath { wwid 36090a06840e6db6f5ebf146a519311ec alias orion2 rr_min_io 20 }... } 16

19 Step 4: Reregister the multipath devices to load the persistent device mapping Use the following commands to reregister and reload the multipath devices: # multipath v2; service multipathd restart Example 19 shows use of the multipath ll command to verify changes. Note how the volume alias now appears (instead of the mpathx name as shown Example 12 above). Example 19 Login to targets and verify connection status [root@sieora1 ~]# multipath ll... orion1 (36090a06840e6fb6c5ebfe f) dm-1 EQLOGIC,100E-00 [size=50g][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][enabled] \_ 13:0:0:0 sdp 8:16 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 14:0:0:0 sdq 65:32 [active][ready]... Example 20 shows the set of /dev/mapper devices created. Example 20 Contents of /dev/mapper [root@sieora1 ~]# ls -l /dev/mapper total 0 crw root root 10, 63 May 22 02:49 control brw-rw root disk 253, 1 May 23 03:16 orion1 brw-rw root disk 253, 8 May 23 03:16 orion10 brw-rw root disk 253, 10 May 23 03:16 orion2 brw-rw root disk 253, 2 May 23 03:16 orion3 brw-rw root disk 253, 4 May 23 03:16 orion4 brw-rw root disk 253, 6 May 23 03:16 orion5 brw-rw root disk 253, 5 May 23 03:16 orion6 brw-rw root disk 253, 3 May 23 03:16 orion7 brw-rw root disk 253, 7 May 23 03:16 orion8 brw-rw root disk 253, 9 May 23 03:16 orion9 17

20 4 Installation and Configuration of the Oracle 11g RAC Database We followed the installation instructions provided by the following document: Oracle Database Installation Guide 11g Release 2 (11.2) for Linux We include only the pre-installation process details that were specific to our test setup in this section. Figure 4 shows a high-level outline of all installation and configuration steps. We include implementation details in this document for the highlighted process steps shown in Figure 4. Figure 4 Oracle Installation and Configuration Process 18

21 4.1 Oracle GRID Infrastructure Create the groups, inventory, users and installation path. Example 21 Create groups, inventory, users and installation path ~]# grep -q oinstall /etc/group groupadd -g 501 oinstall ~]# grep -q dba /etc/group groupadd -g 502 dba ~]# id oracle >> /dev/null 2>&1 ( useradd -u 500 -g oinstall -G dba -d /home/oracle -s /bin/bash oracle ) The above commands create a user oracle and two groups oinstall and dba [root@sieora1 ~]# id oracle uid=500(oracle) gid=501(oinstall) groups=501(oinstall),502(dba) [root@sieora1 ~]# Set the password for user oracle [root@sieora1 ~]# echo "passw0rd" passwd --stdin oracle Changing password for user oracle. passwd: all authentication tokens updated successfully Create the required directories with permission settings for Oracle grid infrastructure and database software [root@sieora1 ~]# mkdir -p /u01/app/11.2.0/grid [root@sieora1 ~]# mkdir -p /u01/app/oracle [root@sieora1 ~]# chown -R oracle:oinstall /u01/ 4.2 Kernel parameters Configure runtime settings in /etc/sysctl.conf. Example 22 Runtime Kernel Parameter Changes in /etc/sysctl.conf # Oracle Settings fs.aio-max-nr = fs.file-max = kernel.sem = net.ipv4.ip_local_port_range = net.core.rmem_default = net.core.rmem_max = net.core.wmem_default = net.core.wmem_max = net.ipv4.tcp_wmem = net.ipv4.tcp_rmem = Use sysctl to make the above changes available in the running kernel [root@sieora1 ~]# sysctl -p 19

22 4.3 IP Addresses The following IP address information must be set before you can continue with the grid infrastructure installation: Public and virtual IP addresses for each RAC node. RAC interconnect IP addresses for each node. Proper number of IP addresses for each node depending on the number of node NICs used for SAN connection. A Single Client Access Name (SCAN) IP address for the RAC configuration. This address must be unique and on the same subnet as the virtual IP address. Example 23 shows the /etc/hosts file additions used in our test configuration. Example 23 IP Address information added to /etc/hosts sieora1 #Public IP Node sieora2 #Public IP Node sieora1-vip #Virtual IP Node sieora2-vip #Virtual IP Node sieora-scan #SCAN IP - RAC sieora1-int #RAC Interconnect Node sieora2-int #RAC Interconnect Node sieora1-stg1 #Storage Interface :1- Node sieora1-stg2 #Storage Interface :2- Node sieora2-stg1 #Storage Interface :1- Node sieora2-stg2 #Storage Interface :2- Node2 4.4 System security settings Add settings to /etc/security/limits.conf and modify /etc/pam.d/login. Ensure SELinux is disabled. Example 24 Additional entries to /etc/security/limits.conf and /etc/pam.d/login Added the following entries to /etc/security/limits.conf: oracle soft nproc 2047 oracle hard nproc oracle soft nofile 1024 oracle hard nofile Added the following entries to /etc/pam.d/login: session required /lib/security/pam_limits.so 2 0

23 Note - as long as the string oracle is not already present in the target files, you can paste the following command sequences into a shell prompt to make these changes: grep -q oracle /etc/security/limits.conf if [ $? -ne 0 ]; then echo "" >> /etc/security/limits.conf echo "oracle soft nproc 2047" >> /etc/security/limits.conf echo "oracle hard nproc 16384" >> /etc/security/limits.conf echo "oracle soft nofile 1024" >> /etc/security/limits.conf echo "oracle hard nofile 65536" >> /etc/security/limits.conf fi grep -q pam_limits /etc/pam.d/login if [ $? -ne 0 ]; then echo "session required /lib/security/pam_limits.so" >> /etc/pam.d/login fi Disable SELinux: [root@sieora1 ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - SELinux is fully disabled. SELINUX=disabled # SELINUXTYPE= type of policy in use. Possible values are: # targeted - Only targeted network daemons are protected. # strict - Full SELinux protection. SELINUXTYPE=targeted # SETLOCALDEFS= Check local definition changes SETLOCALDEFS=0 4.5 Shell process resources Add ulimit settings to /etc/profile to change the per user process resources required for the oracle user. Example 25 Set ulimit parameters in /etc/profile Add the following section to /etc/profile: if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p ulimit -n else ulimit -u n fi fi 21

24 Note - as long as the string oracle is not already present in /etc/profile, you can paste the following command sequence into a shell prompt to make this change: grep -q oracle /etc/profile if [ $? -ne 0 ]; then echo "" >> /etc/profile echo "if [ \$USER = \"oracle\" ]; then" >> /etc/profile echo " if [ \$SHELL = \"/bin/ksh\" ]; then" >> /etc/profile echo " ulimit -u 16384" >> /etc/profile echo " ulimit -n 65536" >> /etc/profile echo " else" >> /etc/profile echo " ulimit -u n 65536" >> /etc/profile echo " fi" >> /etc/profile echo "fi" >> /etc/profile fi 4.6 Hangcheck-Timer Module Override the default hangcheck-timer default values with settings for Oracle 10g/11g. For details, see Oracle Metalink ID and the following Oracle publication: Oracle Clusterware Installation Guide Example 26 Override Hangcheck-Timer in /etc/modules.conf Add the following line to /etc/modules.conf: options hangcheck-timer hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1 Load the new setting: [root@sieora1 ~]# modprobe hangcheck-timer 4.7 Configure System Services Enable required system services. Disable unnecessary system services. Example 27 Configure System Services Enable nscd: chkconfig --levels 35 nscd on service nscd start Disable ntpd, sendmail, cups, cups-config, rhnsd, acpid, messagebus: chkconfig --levels 35 ntpd off 22

25 service ntpd stop chkconfig sendmail off service sendmail stop chkconfig cups off service cups stop chkconfig cups-config-daemon off service cups-config-daemon stop chkconfig rhnsd off service rhnsd stop chkconfig acpid off service acpid stop chkconfig messagebus off service messagebus stop 4.8 SSH Equivalency To establish SSH equivalency between RAC nodes run the setup script provided in the./oracle/grid/sshssetup/ directory. Example 28 Run sshusersetup.sh to Establish SSH Equivalency sshsetup]# sh sshusersetup.sh -hosts "sieora1 sieora2" -user oracle advanced 4.9 ASM Configuration Configure the Oracle ASM library driver and then verify volume discovery and login. Example 29 ASM Configuration Steps Configure the ASM library driver: ~]# /etc/init.d/oracleasm configure -i Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracle Default group to own the driver interface []: dba Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ] Verify volumes are discovered and logged in. In our test environment we need to check for 23

26 availability of the following volumes: oradb1, oradb2, oraflash1, oraflash2, The command and result for oradb1 is shown below. ~]# iscsiadm -m session grep oradb1 tcp: [26] :3260,1 iqn com.equallogic:0-8a ab beb0018d7534c2b1-oradb1 tcp: [29] :3260,1 iqn com.equallogic:0-8a ab beb0018d7534c2b1-oradb1 Create the ASM disks using the device-mapper devices. Example 30 ASM Disk Creation and Scan Create ASM disks (this step required only on the first cluster node): ~]# service oracleasm createdisk ORADB1 /dev/mapper/oradb1 Marking disk "ORADB1" as an ASM disk: [ OK ] [root@sieora1 ~]# service oracleasm createdisk ORADB2 /dev/mapper/oradb2 Marking disk "ORADB2" as an ASM disk: [ OK ] [root@sieora1 ~]# service oracleasm createdisk ORAFLASH1 /dev/mapper/oraflash1 Marking disk "ORAFLASH1" as an ASM disk: [ OK ] [root@sieora1 ~]# service oracleasm createdisk ORAFLASH2 /dev/mapper/oraflash2 Marking disk "ORAFLASH2" as an ASM disk: [ OK ] [root@sieora1 ~]# service oracleasm createdisk ORACRS /dev/mapper/oracrs Marking disk "ORACRS" as an ASM disk: [ OK ] Scan the ASM disks and verify the output. You must repeat the following command on all cluster nodes. [root@sieora1 ~]# service oracleasm scandisks Scanning the system for Oracle ASMLib disks: [ OK ] List all ASM disks on the node: [root@sieora1 ~]# service oracleasm listdisks ORACRS ORADB1 ORADB2 ORAFLASH1 ORAFLASH2 24

27 5 Oracle Grid Infrastructure and Database Creation You complete the installation and configuration tasks for Oracle Grid Infrastructure and database creation in the following phases: 1. Oracle Grid Infrastructure installation 2. Oracle database software installation 3. Oracle database creation We completed the above tasks using the setup and installation guide provided by Dell Support. Please refer to the following document: Dell PowerEdge Systems Oracle Database on Enterprise Linux x86_64 Database Setup and Installation Guide Version

28 Appendix A Switch Configuration XXX and XXX show the test system configurations. We configured the Dell PowerConnect 8024F and 6248 switches as follows: Created LAG (Link Aggregation Group) connections between the switches. Enabled Flow control on all switch ports used for iscsi SAN traffic. Enable jumbo frames. Created redundant connection paths. A.1 PowerConnect 8024F SAN Switch Configuration Example 31 shows the switch configuration used for the 10GbE SAN. Example 31 PowerConnect 8024F 10GbE SAN Switch Configuration 8024F-1#show running-config!current Configuration:!System Description "Powerconnect 8024F, , VxWorks 6.5"!System Software Version ! configure hostname "8024F-1" ip address dhcp username "root" password 5f4dcc3b5aa765d61d8327deb882cf99 level 15 encrypted! [ ports xg1 through xg20 configured identically as shown for port xg1 below ] interface ethernet 1/xg1 spanning-tree portfast mtu 9216 exit! [ ports xg21 through xg24 configured identically as shown for port xg21 below ] interface ethernet 1/xg21 channel-group 1 mode on mtu 9216 exit! [ XXX configured for XXX as shown below ] interface port-channel 1 mtu 9216 exit! interface out-of-band ip address exit exit 26

29 A.2 PowerConnect 6248 RAC Switch Configuration Example 32 shows the switch configuration used for the RAC interconnected network. Example 32 PowerConnect 6248 RAC Switch Configuration #show running-config!current Configuration:!System Description "Powerconnect 6248, , VxWorks 6.5"!System Software Version !Cut-through mode is configured as disabled! configure vlan database vlan 2 exit hostname "6248-2" stack member 1 2 exit ip address ip address vlan 2 username "root" password bed c ed3add75fb level 15 encrypted! [ ports g1 through g43 configured identically as shown for port g1 below ] interface ethernet 1/g1 spanning-tree portfast mtu 9216 exit [ ports g44 through g47 configured identically as shown for port g44 below ] interface ethernet 1/g44 channel-group 1 mode on mtu 9216 exit [ port g48 configured as shown below ] interface ethernet 1/g48 switchport access vlan 2 exit! interface port-channel 1 mtu 9216 exit exit 27

30 THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.