Enabling IBM Storwize V7000 Unified storage for Oracle x86 Linux RAC databases A reference guide for storage and database administrators

Transcription

1 Enabling IBM Storwize V7000 Unified storage for Oracle x86 Linux RAC databases A reference guide for storage and database administrators Shashank Shingornikar IBM Systems and Technology Group ISV Enablement November 2012 Copyright IBM Corporation, 2011

2 Table of contents Abstract... 1 Intended audience... 1 Scope... 1 Introduction to the IBM Storwize V7000 Unified system... 1 IBM Storwize V7000 Unified system overview... 2 IBM Storwize V7000 functionalities... 3 IBM Storwize V7000 terminology... 4 Benefits of Oracle deployment on the IBM Storwize V7000 Unified system... 5 Planning Oracle databases on an IBM V7000 Unified system... 6 Preparing the Storwize V7000 Unified system for Oracle deployment... 7 NFS-based deployment of Oracle database... 7 Creating NFS shares (using GUI)... 7 Creating NFS shares (using CLI)... 9 Creating Oracle 11gR2 RAC Oracle 11gR2 Certification & Supportability Statement Overview of NFS shares for Oracle RAC NFS shares mount options for Oracle RAC Preinstall considerations for Oracle Grid Software OS packages Kernel parameters User limits Profiles Single client access names setup and Oracle Grid Naming Service (GNS) Secure Shell (SSH) setup Installing Oracle 11gR2 grid infrastructure software Cluvfy check Installing the 11gR2 grid infrastructure software Installing the 11gR2 database software Database creation Direct NFS setup Direct NFS configuration Verification of the Direct NFS setup Summary Resources About the author Trademarks and special notices... 25

3 Abstract This paper explores the use of the IBM Storwize V7000 Unified storage system as an essential component of advanced Oracle infrastructure solution. The paper demonstrates a practical implementation of a high-availability solution for Oracle Real Application Cluster (RAC) databases; Oracle 11gR2 RAC on Red Hat Enterprise Linux (RHEL) 5.8 operating environment through network-attached storage (NAS) services using protocols such as Network File System (NFS) or Oracle Direct Network File System (Direct NFS). Intended audience This technical report is intended for: Scope Customers and prospects looking to implement effective database solution on the IBM Storwize V7000 Unified system using the Oracle software. Administrators and management seeking detailed information to implement Oracle solution on the IBM Storwize V7000 Unified system. This technical report provides: Detailed instructions for the implementation of Oracle database on the IBM Storwize V7000 Unified system by using NFS The paper does not include block based implementation of Oracle database on IBM V7000 Unified system; however there is a link provided that directs the users to another white paper that describes such implementation in detail. This technical report does not: Discuss any performance impact and analysis from a user perspective. Replace any official manuals and documents from IBM on Storwize V7000 or Oracle Corporation. Introduction to the IBM Storwize V7000 Unified system The IBM Storwize V7000Unified system solution provides a modular storage system that includes the capability to virtualize external storage area network (SAN) attached storage and its own internal storage. IBM Storwize V7000 Unified system provides a number of preset configuration options that are aimed at simplifying the implementation process. It also provides an active-active solution, and is a clustered, scalable, and midrange storage, and an external virtualization device. Included with IBM Storwize V7000 Unified system is a simple and easy-to-use graphical user interface (GUI) that allows storage to be deployed quickly and efficiently. The GUI runs on the IBM Storwize V7000 Unified system and therefore, there is no need for a separate console. The IBM Storwize V7000 Unified solution consists of a control enclosure which includes the chassis, node canisters, drives, and energy sources that include batteries, and expansion enclosures (which is a hardware unit that includes expansion canisters, drives, and energy sources) that do not include batteries. Within each enclosure are two canisters which are the hardware units that include the node hardware, fabric and service interfaces, and serial-attached SCSI (SAS) expansion ports, determining the role of the enclosure. 1

4 IBM Storwize V7000 Unified system overview IBM Storwize V7000 Unified system, when attached to a customer s Internet Protocol (IP) network, serves files using the industry-standard protocols, such as NFS, Common Internet File System (CIFS), File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), and Hypertext Transfer Protocol Secure (HTTPS), and at the same time offers block devices by means of storage logical unit number (LUN) which can be mapped to host using Fibre Channel (FC) or using Internet Small Computer System Interface (iscsi). Figure 1: IBM Storwize V7000 Unified system As shown in Figure 1, the unique architecture of the IBM Storwize V7000 Unified system offers greater flexibility and high availability when compared to traditional NAS or FC solutions. For more information about the IBM Storwize V7000 Unified system, refer to the URL: ibm.com/systems/storage/disk/storwize_v7000/specifications.html 2

5 IBM Storwize V7000 functionalities The following functionalities are available with the IBM Storwize V7000 system: Thin provisioning: With thin provisioning, applications can grow dynamically, but consume only the space that they are actually using. Without thin provisioning, preallocated space is reserved irrespective of whether the application uses it or not. Volume Mirroring: Volume Mirroring provides a single volume image to the attached host systems while maintaining pointers to two copies of data in separate storage pools. Copies can be on completely separate disk storage systems that are being virtualized. IBM Tivoli Storage FlashCopy Manager: FlashCopy creates instant application copies that can be used for backup or application testing. FlashCopy makes better use of the space with incremental copy, where only changed blocks are copied. Metro Mirror: Metro Mirror provides synchronous remote mirroring function up to approximately 300 km between sites. As the host I/O completes only after the data is cached at both the locations, performance requirements might limit the practical distance. Metro Mirror is designed to provide fully-synchronized copies at both the sites with zero data loss after the initial copy is completed. Global Mirror: Global Mirror provides long distance asynchronous remote mirroring function up to approximately 8,000 km between sites. With Global Mirror, the host I/O completes locally, and the changed data is sent to the remote site later. This is designed to maintain a consistent, recoverable copy of data at the remote site which lags behind the local site. Data migration: A data migration wizard can be used to import external storage system into the IBM Storwize V7000 system. IBM System Storage Easy Tier : Easy Tier provides a mechanism to seamlessly migrate hot spots to a higher performing storage pool within the IBM Storwize V7000 solution. Migration can be to internal drives within the IBM Storwize V7000 Unified system or to external storage systems that are virtualized by the IBM Storwize V7000 system. Figure 2: Storwize V7000 Unified system virtualizing external and internal storage 3

6 IBM Storwize V7000 terminology Table 1 lists the Storwize V7000 terminology used in the paper. For an entire list of the Storwize V7000 terminology, refer to IBM Storwize V7000 Introduction and Implementation Guide at: ibm.com/redbooks/redpieces/abstracts/sg html?open IBM Storwize V7000 term Control enclosure Expansion enclosure Node canister Host mapping Internal storage Managed disk (MDisk) Storage pool Thin provisioning or thin provisioned Volume Extent Definition A hardware unit that includes the chassis, node canisters, drives, and energy sources including batteries. A hardware unit that includes expansion canisters, drives, and energy sources that do not include batteries. A hardware unit that includes the node hardware, fabric and service interfaces, and SAS expansion ports. A controlling process in which hosts have access to only specific volumes within a cluster. Array-managed disks and drives that are held in enclosures and nodes that are part of the cluster. A component of a storage pool that is managed by a cluster. An MDisk is either a part of a Redundant Array of Independent Disks (RAID) array of internal storage or a SCSI logical unit (LU) for external storage. An MDisk is not visible to a host system on the SAN. A collection of storage capacity that provides the capacity requirements for a volume. The ability to define a storage unit (full system, storage pool, or volume) with a logical capacity size that is larger than the physical capacity assigned to that storage unit. A discrete unit of storage on disk, tape, or other data recording medium that supports some form of identifier and parameter list, such as a volume label or I/O control. Each MDisk is divided into segments of equal size called extents. When a volume is created from a storage pool, the volume is allocated based on the number of extents required to satisfy the capacity requirements for the volume. Table 1: IBM Storwize V7000 Unified system terminology 4

7 Benefits of Oracle deployment on the IBM Storwize V7000 Unified system The IBM Storwize V7000 Unified system provides greater flexibility for deploying database solution though its architecture. Customers can use the file protocol (NFS) or block storage to deploy database environments. Both, the file and block storages, can be used on the same database host. Table 2 gives a summary of deployment benefits of Oracle database on the IBM Storwize V7000 Unified system. Benefit Lower total cost of operation (TCO) Central management Greater flexibility Single global name space Intelligent policy engine High availability Description The IBM Storwize V7000 Unified system reduces the cost for administration and maintenance by scaling capacity and performance independent of each other. The IBM Storwize V7000 Unified system provides ease of use, maintenance, provisioning and implementation, and centralized management with alert and event logs through a single GUI or command line interface (CLI). The database can be created on NFS shares or on the block device. Data islands can be consolidated into a single globally-accessible network-based store alongside block databases store. Intelligent policy engine and tiered storage are built into each IBM Storwize V7000 Unified system. Automatic striping of file system allows the use of all available storage based on user-defined file system policies. The IBM Storwize V7000 Unified system with its unique robust architecture is able to provide the high-availability feature without any additional changes or setup. Table 2: Oracle database deployment benefits on the IBM Storwize V7000 Unified system 5

8 Planning Oracle databases on an IBM V7000 Unified system This guide describes the deployment of the Oracle databases on Linux using the IBM Storwize V7000 Unified system NFS data store or using FC connectivity. The topology components include IBM Stowize V7000 Unified System and IBM xseries systems. Figure 3 provides a pictorial representation of the setup created in the lab environment. Figure 3: Lab topology of RAC setup with the IBM Storwize V7000 Unified system Lab setup: A four node RAC cluster is created using the IBM xseries M3 systems. All the systems have RHEL 5.8 installed with the same kernel level and the systems had multiple network interfaces to cater various network requirements for RAC setup. A 10 Gb network interface is used to mount the NFS shares from the file module of the Storwize V7000 Unified system. 6

9 Preparing the Storwize V7000 Unified system for Oracle deployment This section describes the ways to prepare the Storwize V7000 Unified system for Oracle database deployment using file based storage by means of the NFS protocol and FC connectivity for block storage. NFS-based deployment of Oracle database This section provides the prerequisites for the creation of Oracle database on NFS storage provided by the IBM Storwize V7000 Unified system. The NFS shares needed for the Oracle database are created as described in the following sections. Creating NFS shares (using GUI) The IBM Storwize V7000 Unified system s NFS shares can be created using the GUI interface when connected through a browser or using CLI tools (such as mkexport) when connected as cliuser using PuTTY or a terminal emulator of your choice. When using the GUI, you need to provide the fully qualified domain name (FQDN). FQDN is given at the time of installation of the Storwize V7000 Unified system. You need to perform the following steps to create NFS shares using the Storwize V7000 Unified system s management console. In the lab setup, the Storwize V7000 Unified System is named ifs5.storage.tucson.ibm.com 1. Go to the URL: 2. Log in to the console using the user ID, admin and the password that has been set at the time of the installation. Figure 4: Storwize V700 Unified system login 3. On the left side, click Shares under the Files submenu to start the file share creation wizard. In the panel displayed, enter the details regarding the share name that you want to create along with the directory path. 7

10 Figure 5: Storwize V7000 Unified system export wizard - step 1 4. Select the NFS type for the share and enter the values in the Path and Share name fields. Clear the Read only access, Root squash, and Secure check boxes. Figure 6: Storwize V7000 Unified System export wizard - step 2 5. Now, the Storwize V7000 Unified system has sufficient information to create the NFS share. Click OK to create the share. 8

11 Note: When creating or configuring shares for the Oracle database, leave the Root Squash check box cleared. Clearing the Root Squash check box here allows root user on the client systems to act as root user (only) within the share path. As soon as the share is created, a confirmation indicating the success or failure information is displayed. Creating NFS shares (using CLI) File shares can also be created using the Storwize V7000 Unified system s CLI interface. In order to create shares using the CLI, log in to the Storwize V7000 Unified system s integrated management node as cliuser. Syntax: mkexport <Share-Name> <Share-Path> --nfs <Client-FQDN/IP Address/*>(NFS options) Example: mkexport oradata01 /ibm/gpfs0/oradata_nfs nfs *(rw,no_root_squash,insecure,anonuid=505,anongid=500) Table 3 provides an explanation of the options used in the command. Option Share-name Share-path FQDN Description Name to be assigned to the share. Storwize V7000 Unified system or IBM General Parallel File System (IBM GPFS ) path to be exported. This command creates the path in the GPFS directory if it does not exist. FQDN option is mandatory only if both the systems (Storwize V7000 Unified system and the client) are in different domains. If the systems are in the same domain, just the short name or host name might suffice the purpose. IP addresses and FQDN can be used irrespective of the domain of the two machines. NFS options rw,no_root_squash,insecure,anonuid=uidn,anongid=gidn (UIDn and GIDn are Oracle user s numeric user IDs on Linux clients) Table 3: Storwize V7000 Unified system command line syntax for creating NFS shares Note: After creating the shares (using GUI or the mkexport command) they need to be mounted on the client Linux systems. Special mount options must be used for mounting these shares. Also upon mounting the shares on the client systems, permissions on the local mount directory must be changed to Oracle user and dba group, incase anonuid and anongid options are not used. 9

12 Creating Oracle 11gR2 RAC Now that it is clear how to create NFS shares using the Storwize V7000 Unified system, the following shares (as shown in Table 4) needs to be created on the Storwize V7000 Unified System for Oracle RAC. Oracle 11gR2 Certification & Supportability Statement IBM supports the use of IBM SONAS and IBM Storwize V7000 Unified systems as NAS attached architectures, with Oracle Database 11gR2 on IBM Power (AIX) and on IBM x86 (RHEL Linux) server platforms. Certification and supportability details are available on My Oracle Support link ( as Oracle Metalink [ ]. Overview of NFS shares for Oracle RAC Table 4 shows an overview of the NFS shares for Oracle RAC. Share name NFS path Comment CRS1 /ibm/gpfs0/oradata_nfs/crs1 RAC cluster registry location 1 CRS2 /ibm/gpfs0/oradata_nfs /CRS2 RAC cluster registry location 2 CRS3 /ibm/gpfs0/oradata_nfs /CRS3 RAC cluster registry location 3 VOTING1 /ibm/gpfs0/oradata_nfs/voting1 Voting disk location 1 VOTING2 /ibm/gpfs0/oradata_nfs/voting2 Voting disk location 2 VOTING3 /ibm/gpfs0/oradata_nfs/voting3 Voting disk location 3 DB1 /ibm/gpfs0/oradata_nfs/db1 DB share REDO1 /ibm/gpfs0/oradata_nfs/redo1 Redo logs location FLASH1 /ibm/gpfs0/oradata_nfs/flash1 Flash recovery area ARCH1 /ibm/gpfs0/oradata_nfs/arch1 Archive log area Table 4: Overview of the NFS shares /ibm/gpfs0 here is the base file system created by the Storwize V7000 Unified storage administrator. Creating the Oracle directory provides a logical separation point from the remaining data on /ibm/gpfs0. This is not mandatory though. Note: As of the date when this paper was written, Oracle does not support reading or writing to Oracle cluster registry (OCR) or quorum disks (VOTING disks) using Oracle supplied Direct NFS client. Direct NFS client is known to support reading and writing to only database files. Refer to the metalink document ID for more information. Metalink is an Oracle support website. Refer to the Resources section for more details. 10

13 NFS shares mount options for Oracle RAC The NFS shares are listed in the /etc/fstab on cluster nodes. Oracle has strict requirements for using NFS shares with RAC. The metalink document ID has detailed information about these options for various operating systems. When using NFS mounts, care must be taken while writing the mount option and mount point in the same order as mentioned by Oracle. Misplacing mount options can cause unintended behavior or serious errors. For example, misplacing the hard parameter can cause the CREATE CONTROLFILE FAILED error (metalink note: and ). Table 5 shows an overview of NFS shares mount options needed for Oracle RAC. /etc/fstab lines for mounting Oracle cluster registry and voting disks 9.x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/VOTING1 /u03/voting1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo= x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/VOTING2 /u03/voting2 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo= x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/VOTING3 /u03/voting3 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo= x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/CRS1 /u03/crs1 nfs rw,bg,hard, nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo= x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/CRS2 /u03/crs2 nfs rw,bg,hard, nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo= x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/CRS3 /u03/crs3 nfs rw,bg,hard, nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0 0 0 /etc/fstab lines for mounting Oracle DB files 9.x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/DB1 /u04/db1 nfs rw, bg, hard, nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo= x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/FLASH1 /u05/flash1 nfs rw, bg, hard, nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo= x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/REDO1 /u04/redo1 nfs rw, bg, hard, nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo= x.x.x:/ibm/gpfs0/Oracle_1/ORACLENFS/ARCHIVE1 /u04/archive1 nfs rw, bg, hard, nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo= Table 5: Overview of the NFS shares mount options Note: Here you certainly do not want to misconfigure NFS file systems. For example, missing actimeo=0 for CRS can cause the PROT-1 failed to initialize ocrconfig error while running root.sh after installing the cluster software. (Refer to metalink document ID: and ). Misplacing the hard parameter can cause a CREATE CONTROLFILE FAILED error. 11

14 Preinstall considerations for Oracle Grid Software Following prerequisites must be met before starting the Oracle grid infrastructure software installation. OS packages You need to check for the missing operating system software packages needed by both: Oracle grid infrastructure and database software. Table 6 lists the required packages. These packages can be installed using Red Hat package manager (RPM installer). Note: Certain 32-bit packages are also needed when your system s architecture is 64 bit. Package binutils compat-libstdc compat-libstdc elfutils-libelf elfutils-libelf-devel gcc gcc-c glibc glibc glibc-common-2.5 glibc-devel-2.5 glibc-devel-2.5 glibc-headers-2.5 ksh libaio libaio libaio-devel libaio-devel libgcc Architecture 32 bit 32 bit 32 bit 32 bit 32 bit 12

15 libgcc libstdc libstdc libstdc++-devel make-3.81 sysstat unixodbc unixodbc unixodbc-devel unixodbc-devel bit 32 bit 32 bit 32 bit Table 6: OS Packages Kernel parameters Oracle RAC also has specific requirements for certain kernel parameters. The required parameters for Oracle RAC listed in Table 7 need to be added in the /etc/sysctl.conf file. Added / modified in file: /etc/sysctl.conf net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = kernel.msgmax = kernel.shmmax = kernel.shmall = kernel.shmmni = 4096 kernel.sem = kernel.shmmni = 4096 fs.file-max = fs.aio-max-nr= net.ipv4.ip_local_port_range = net.core.rmem_default = net.core.rmem_max = net.core.wmem_default = net.core.wmem_max = Reload the configuration change made using /sbin/sysctl p Table 7: Kernel parameters 13

16 User limits User limits indicate the number of process allowed per user. Table 8 lists the soft and hard process limit settings for grid and Oracle user. This needs to be modified in the /etc/security/limits.conf file. Added / modified in file: /etc/security/limit.conf grid soft nproc 2047 grid hard nproc grid soft nofile 1024 grid hard nofile oracle soft nproc 2047 oracle hard nproc oracle soft nofile 1024 oracle hard nofile Table 8: User limits information For the management of the grid infrastructure, as of Oracle 11g Release 2 (formerly known as Oracle clusterware) Oracle has introduced a dedicated user known as grid. This user is the owner of all the cluster-related binaries. Along with the grid user, a dedicated user to run database processes is also needed on the system. Table 9 shows an example to create both these users. Commands to use: useradd / groupadd / passwd groupadd -g 500 oinstall groupadd -g 501 dba useradd -g oinstall -G dba -c Oracle User -m oracle useradd -g oinstall -G dba -c Grid Infrastructure User -m grid passwd oracle < Set password for oracle user here > passwd grid < Set password for grid user here > Table 9: Required users and groups Profiles Log in as both Oracle and grid user. For both the users, modify the.bash_profile or.profile depending on your shell and add the umask 022 line. Single client access names setup and Oracle Grid Naming Service (GNS) For 11gR2, you have to register a minimum of three Single Client Access Names (SCANs) in your Domain Name System (DNS). A SCAN is used to provide service access for clients to the cluster. The SCAN is associated with the cluster as a whole and then with an individual cluster node. The resolution of these SCAN s to IP addresses needs to be done through the DNS server. Furthermore, if you need to use the Oracle provided GNS, you also need to delegate to cluster subdomain. In this setup, the test team did not use GNS. Secure Shell (SSH) setup An option to log in without a password needs to be established between all cluster nodes for grid and Oracle users. There are two ways to perform this task: 14

17 During grid infrastructure software installation Manual setup For manual setup, you need to run the following commands on all the systems as the respective user. In Table 10, you see the sample output of the ssh-keygen command during the creation of the rsa key for the grid user. All defaults have been accepted. SSH Key generation ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/grid/.ssh/id_rsa): Created directory '/home/grid/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_rsa. Your public key has been saved in /home/grid/.ssh/id_rsa.pub. The key fingerprint is: 39:e5:42:3e:24:37:74:78:a1:6g:55:6a:67:0d:7d:d0 Table 10: SSH key generation The command ssh-keygen t rsa creates RSA encryption-based public and private SSH keys. This process needs to be repeated on all cluster nodes. After generating the key, the ssh-copy-id command can be used to distribute the public key across all the cluster nodes in order to enable the passwordless login. The SSH key encryption used must be homogenous such as RSA only or Digital Signature Algorithm (DSA) only. After completing the SSH setup, log in to all hosts listed in the /etc/hosts files, except hosts ending with vip. This action puts the key entry in cache in the.ssh/known_hosts file. Login must be done from every node to the remaining cluster nodes. 15

18 Installing Oracle 11gR2 grid infrastructure software To ensure that all the prerequisites are met for the grid infrastructure, a check script called runcluvfy.sh is provided by Oracle. After successful running of the check script, the installation can be started. Cluvfy check Cluvfy helps to determine whether all the prerequisites are met for the installation of the grid infrastructure or for database software installation. The Cluvfy check can be done at various stages such as: After hardware and OS setup Before and after grid infrastructure installation Before and after database software installation The following example lists post hardware and OS check and grid infrastructure software installation phases../runcluvfy.sh stage -post hwos -n isvsonas25,isvsonas26,isvsonas27,isvsonas28 -verbose tee post_hwos.log./runcluvfy.sh stage -pre crsinst -n isvsonas25,isvsonas26,isvsonas27,isvsonas28 tee pre_crsinst.log Here, Cluvfy should pass all the tests. Installing the 11gR2 grid infrastructure software Optimal Flexible Architecture (OFA) guidelines are recommended for the installation of the grid infrastructure, and eventually the database software, thus every software installation is done in a separate home. Panel Response Comment Installation option Installation type Install and configure the grid infrastructure for cluster Advanced installation 16 Choose an appropriate option here Do not choose the Typical Installation Language selection English Language selection screen Grid plug and play information Cluster node information cluster name: orcl scan name: orcl-scan.cluster.net scan-port: 1521 Clear configure GNS Check the name selected by default Use nslookup orcl-scan before entering this name. The SCAN name is configured in DNS and the DNS will give the response This reflects the current node by default. Other nodes need to be manually added. In the same panel, the option is given to set up SSH password-less connectivity.

19 Network interfaces usage eth0: 9.x.x.x eth1: 10.x.x.x Public Private Confirm that all the interfaces are listed as expected. Also they must be the same across eth2: 9.x.x.x Do not use all the nodes. Do not mix and match Storage option information OCR storage Voting disk storage Failure isolation support Privileged operating system groups Specify installation location Shared File System /u03/crs1/ocr1 /u04/crs2/ocr2 /u05/crs3/ocr3 /u03/voting1/vdisk1 /u04/voting2/vdisk2 /u05/voting3/vdisk3 Choose Do not use IPMI Accept defaults Oracle base: /u01/app/grid Software location: /u01/app/11.2.0/grid Choice between Automatic Storage Manager (ASM) and NFS Use the Linux touch command to create empty files ocr1,ocr2 and ocr3. Change the ownership of these files to grid:oinstall Click Yes when prompted to overwrite the file. Use the Linux touch command to create empty files vdsk1, vdsk2 and vdsk3.change the ownership of these files to grid:oinstall Click Yes when prompted to overwrite the file. This response depends on whether the Intelligent Peripheral Management Interface (IPMI) device is present and has been configured on the hardware Default listing is grid:oinstall Using OFA here Inventory directory /u01/app/orainventory Using OFA here Summary Run configuration scripts Run configuration scripts Save response file Run orainstroot.sh Run root.sh Table 11: Installation screens and possible responses for the grid infrastructure software Check summary. Click Finish to run setup Needs to be done on all the nodes Needs to be done on all the nodes After running root.sh successfully on one node, you need to run it on the remaining nodes too. 17

20 Installing the 11gR2 database software After setting up the grid infrastructure successfully, you can start to set up the database software. According to the OFA standards, the database software is installed in separate home. Panel Response Comment Configure security updates Nothing is entered Click Next Installation options Grid options Product language Database edition Oracle base Software location Operating system group Perform prerequisite checks Summary Finish Install database software only Select the listed nodes English Enterprise Edition 4.29 GB /u01/app /u01/app/oracle/11.2.0/dbhome_1 dba:oinstall Should not list any issues Installation is successful You might have to check your configuration setup again, if any (serious) issues are listed here Table 12: Installation screens and the possible responses for the 11gR2 database software Database creation It is recommended to use the Oracle Database Configuration Assistant (DBCA) for creating a database. Apart from creating the databases, DBCA allows creation and management of templates used for databases, both with structure and data. Panel Response Comment Welcome Choose the Oracle Real Choose the appropriate option Application Cluster database Operation Create the database Choose the appropriate option Database templates Database identification General purpose or transaction processing Choose the Admin Managed option and enter a descriptive name, make sure you choose all the nodes using the Select All option. For any production database, you would normally choose Custom database. 18

21 Management option Configure with Enterprise Manager If you do not have a grid console, choose local database control. For a grid managed system, you might install Oracle grid agent on the system before creating the database. Database credentials Same password for all the users You definitely do not want this for your production or test system. Database file locations Use the common location for all the database files Recovery configuration Specify flash recovery area Database contents Sample schemas Initialization parameters Custom is the best option here Largely depends on how you want to configure this. Database storage Check locations for all files again Creation options Summary Choose Create database, generate database creation scripts View summary and click OK Table 13: Creating database using DBCA It is best not to put the database in archivelog mode at creation time. It can be easily done after the creation is completed. Direct NFS setup Direct NFS is a functionality offered by Oracle by means of Oracle supplied NFS client. You can set up the Direct NFS client by: Creating the required number of database shares as listed in the Creating NFS shares (using GUI) section Creating Oracle Direct NFS specific configuration files For Oracle 11gR2 RAC installation, IBM Storwize V7000 Unified is used as a NAS device, making the storage available through the NFS protocol. Oracle cluster and database software binary files are installed locally across each RAC cluster node. The cluster registry, voting disk, and database reside on Storwize V7000 Unified system. Oracle database can be configured to make use of Oracle Direct NFS client. Note: The Direct NFS functionality is used by the database software only. Cluster software does not make use of the Direct NFS functionally for reading / writing to OCR and VOTING disks. It is important that no databases are running during the configuration of Direct NFS. If any RAC databases are active, you need to stop them using the srvctl command. 19

22 Direct NFS configuration The Direct NFS configuration must be done on all cluster nodes. To configure Direct NFS, perform the following steps: 1. Install a checksum patch on all the Storwize V7000 Unified nodes. You need to contact the IBM Advanced Technical Services (ATS) team in order to obtain the patch and perform the installation. The installation of the patch needs a reboot of the interface node. Stop all the databases accessing Storwize V7000 Unified System shares before you install the patch. 2. Stop all running instances on all cluster nodes. srvctl stop database d DATABASE_NAME 3. Rename the libodm library from ORACLE_HOME and create a symbolic link mv $ORACLE_HOME/lib/libodm11.so $ORACLE_HOME/lib/libodm11.so.BAK ln s $ORACLE_HOME/lib/libnfsodm11.so libodm11.so 4. Create the oranfstab file at one of the locations listed in Table 14. The lab setup configuration options are listed in Table 14. Care must be taken when choosing the location of the oranfstab file as it can impact a single or all the databases on the host system. File location ORACLE_HOME/dbs/oranfstab /etc/oranfstab /etc/mtab Purpose Specific to single database Globally available to all databases running on the same host Oracle reads NFS mount points from this file Table 14: Direct NFS configuration setup files specification Table 15 shows a sample oranfstab file created in the /etc location. /etc/oranfstab server: ifs5 path: local: export /ibm/gpfs0/oradata_nfs/db1 export /ibm/gpfs0/oradata_nfs/flash1 export /ibm/gpfs0/oradata_nfs/redo1 export /ibm/gpfs0/oradata_nfs/archive1 Table 15: Direct NFS configuration options (sample oranfstab file) mount /u04/db1 mount /u05/flash1 mount /u06/redo1 mount /u07/archive1 The database alert log entries listed in Table 15 shows that the server name ifs5 is resolved through DNS to multiple IP addresses. It is recommended to put in all the public IP address here, during Direct NFS setup. It is recommended that you use only one Storwize V7000 Unified interface node here. 20

23 Verification of the Direct NFS setup You can check if Direct NFS is active through: Database alert log entries Existence of symbolic link libodm11.so pointing to libnfsodm11.so Querying the database instance v$ views Database alert log entries Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0 Direct NFS: channel id [0] path [ ] to filer [ifs5] via local [ ] is UP Direct NFS: channel id [1] path [ ] to filer [ifs5] via local [ ] is UP Table 16: Database alert log entries Linux symbolic link $] cd /u01/app/oracle/product/11.2.0/dbhome_1/lib $] ls -l libodm11.so lrwxrwxrwx 1 oracle oinstall 56 Nov 10 22:50 libodm11.so ->libnfsodm11.so Table 17: Linux symbolic link V v$dnfs_server col svrname for a10 col dirname for a30 select * from v$dnfs_servers; ID SVRNAME DIRNAME MNTPORT NFSPORT WTMAX RTMAX ifs5 /ibm/gpfs0/oradata_nfs/db ifs5 /ibm/gpfs0/oradata_nfs/redo Table 18: Database instance v$ view query Note: The Oracle Direct NFS setup white paper with the metalink document ID: describe the guidelines for configuring Direct NFS using the NFS server. The Storwize V7000 Unified system provides access to data files using the NFS protocol. Because of this, the configuration of Direct NFS differs when using Storwize V7000 Unified System, mainly when Microsoft Active Directory (AD) is used for authentication. 21

24 When a share is created on the Storwize V7000 Unified system, it is owned by root by default. These shares must be changed to database user ID and database user s group ID. For ease of administration, this user should be centrally authenticated. This user is assigned as anonuid and anongid (with the same UID and GID as on the local system for oracle:dba) to the Storwize V7000 Unified share. When using Lightweight Directory Access Protocol (LDAP) as a preferred authentication method, anonuid and anongid can be created with the same UID and GID as that of the Oracle user (existing on cluster nodes). When AD is used as the preferred authentication method, following considerations must be made. AD does not support UNIX UID; instead it uses names (user names or group names) which are internally translated as S-nnn (where nnn is some numeric identifier) When mapping is done for a UID in AD, mapped user IDs are registered in Samba WINBIND with numbers greater than Oracle supports user IDs only up to For AD to understand UIDs, Microsoft Services for UNIX must be installed. You need to manually change the ownership of the database directories to the UID or GID of the Oracle user when using AD. When using LDAP, you can simply use anonuid or anongid that you created in LDAP, as it understands UID and GID. 22

25 Summary Based on the tests performed, it is evident that the Storwize V7000 Unified system, with its robust architecture, is capable of supporting the demanding high-availability and scalable compute and storage requirements of an enterprise-level, Oracle RAC-based applications. The tests validated the use of both NFS and Oracle Direct NFS; however for the use of the Direct NFS method, it is advised to take a note of the special considerations described in this paper. 23

26 Resources The following websites provide useful references to supplement the information contained in this paper: IBM Storwize V7000 Unified Oracle support (formerly known as metalink) Linux Foundation IBM Techdocs ibm.com/support/techdocs/atsmastr.nsf/webindex/wp Virtual Loaner Program ibm.com/systems/vlp IBM Redbooks ibm.com/redbooks IBM Publications Center About the author Shashank Shingornikar is an IT Specialist in the IBM Systems and Technology Group ISV Enablement Organization. He has more than 10 years of experience working with Oracle database on various platforms. You can reach Shashank at 24

27 Trademarks and special notices Copyright IBM Corporation References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. 25

28 Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. 26