Shashank Shingornikar IBM Systems and Technology Group ISV Enablement. Mayur Shetty IBM Systems and Technology Group ISV Enablement for Storwize V7000

Transcription

1 Enabling IBM Storwize V7000 Unified storage for Oracle x86 LINUX single instance databases Reference guide for Oracle database and storage administrators Shashank Shingornikar IBM Systems and Technology Group ISV Enablement Mayur Shetty IBM Systems and Technology Group ISV Enablement for Storwize V7000 November 2012 Copyright IBM Corporation, 2012.

2 Table of contents Abstract... 1 Intended audience... 1 Scope... 1 Introduction to the IBM Storwize V7000 Unified system... 1 IBM Storwize V7000 Unified system overview... 2 IBM Storwize V7000 functionalities... 3 IBM Storwize V7000 terminology... 4 Benefits of Oracle deployment on the IBM Storwize V7000 Unified system... 5 Planning Oracle databases on an IBM Storwize V7000 Unified system... 6 Preparing the Storwize V7000 Unified system for Oracle deployment... 7 NFS-based deployment of Oracle database... 7 Creating NFS shares (using GUI)... 7 Creating NFS shares (using CLI)... 9 Block-based deployment of Oracle database IBM Storwize V7000 configuration SAN configuration planning Storwize V7000 overview Storage pool configuration MDisk configuration Volume configuration Mapping volumes to the host Prerequisite on the Linux host Installation requirements for Oracle Oracle 11gR2 Certification & Supportability Statement Oracle software installation Database creation Direct NFS setup Lab setup Storwize V Host nodes Summary Acknowledgements Resources About the author Trademarks and special notices... 34

3 Abstract This paper explores the use of IBM Storwize V7000 Unified storage product as an essential component of the infrastructure solution using Oracle relational database management system (RDBMS). The paper demonstrates the practical implementation of an Oracle single instance database on the Red Hat Enterprise Linux (RHEL) operating environment through networkattached storage (NAS) services using the Network File System (NFS) protocol or using block storage provided by the Storwize V7000 Unified storage system. Intended audience This technical report is intended for: Scope Customers and prospects looking to implement effective database solution on the IBM Storwize V7000 Unified system using the Oracle software. Administrators and management seeking detailed information to implement Oracle solution on the IBM Storwize V7000 Unified system. This technical report provides: Detailed instructions for the implementation of Oracle database on the IBM Storwize V7000 Unified system by using both NFS and block storage. This technical report does not: Discuss any performance impact and analysis from a user perspective. Replace any official manuals and documents from IBM on Storwize V7000 or Oracle Corporation. Introduction to the IBM Storwize V7000 Unified system The IBM Storwize V7000Unified solution provides a modular storage system that includes the capability to virtualize external storage area network (SAN) attached storage and its own internal storage. IBM Storwize V7000 Unified system provides a number of preset configuration options that are aimed at simplifying the implementation process. It also provides an active-active solution, and is a clustered, scalable, and midrange storage, and an external virtualization device. Included with IBM Storwize V7000 Unified system is a simple and easy-to-use graphical user interface (GUI) that allows storage to be deployed quickly and efficiently. The GUI runs on the IBM Storwize V7000 Unified system and therefore, there is no need for a separate console. The IBM Storwize V7000 Unified solution consists of a control enclosure which includes the chassis, node canisters, drives, and energy sources that include batteries and expansion enclosures (which is a hardware unit that includes expansion canisters, drives, and energy sources) that do not include batteries. Within each enclosure are two canisters which are the hardware units that include the node hardware, fabric and service interfaces, and serial-attached SCSI (SAS) expansion ports, determining the role of the enclosure. 1

4 IBM Storwize V7000 Unified system overview IBM Storwize V7000 Unified system, when attached to a customer s Internet Protocol (IP) network, serves files using the industry-standard protocols, such as NFS, Common Internet File System (CIFS), File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), and Hypertext Transfer Protocol Secure (HTTPS), and at the same time offers block devices by means of storage logical unit number (LUN) which can be mapped to host using Fibre Channel (FC) or using Internet Small Computer System Interface (iscsi). Figure 1: IBM Storwize V7000 Unified system As shown in Figure 1, the unique architecture of the IBM Storwize V7000 Unified system offers greater flexibility and high availability when compared to traditional NAS or FC solutions. For more information about the IBM Storwize V7000 Unified system, refer to the URL: ibm.com/systems/storage/disk/storwize_v7000/specifications.html 2

5 IBM Storwize V7000 functionalities The following functionalities are available with the IBM Storwize V7000 system: Thin provisioning: With thin provisioning, applications can grow dynamically, but consume only the space that they are actually using. Without thin provisioning, preallocated space is reserved irrespective of whether the application uses it or not. Volume Mirroring: Volume Mirroring provides a single volume image to the attached host systems while maintaining pointers to two copies of data in separate storage pools. Copies can be on completely separate disk storage systems that are being virtualized. IBM Tivoli Storage FlashCopy Manager: FlashCopy creates instant application copies that can be used for backup or application testing. FlashCopy makes better use of the space with incremental copy, where only changed blocks are copied. Metro Mirror: Metro Mirror provides synchronous remote mirroring function up to approximately 300 km between sites. As the host I/O completes only after the data is cached at both the locations, performance requirements might limit the practical distance. Metro Mirror is designed to provide fully-synchronized copies at both the sites with zero data loss after the initial copy is completed. Global Mirror: Global Mirror provides long distance asynchronous remote mirroring function up to approximately 8,000 km between sites. With Global Mirror, the host I/O completes locally, and the changed data is sent to the remote site later. This is designed to maintain a consistent, recoverable copy of data at the remote site which lags behind the local site. Data migration: A data migration wizard can be used to import external storage system into the IBM Storwize V7000 system. IBM System Storage Easy Tier : Easy Tier provides a mechanism to seamlessly migrate hot spots to a higher performing storage pool within the IBM Storwize V7000 solution. Migration can be to internal drives within the IBM Storwize V7000 Unified system or to external storage systems that are virtualized by the IBM Storwize V7000 system. Figure 2: Storwize V7000 Unified system virtualizing external and internal storage 3

6 IBM Storwize V7000 terminology Table 1 lists the Storwize V7000 terminology used in the paper. For an entire list of the Storwize V7000 terminology, refer to IBM Storwize V7000 Introduction and Implementation Guide at ibm.com/redbooks/redpieces/abstracts/sg html?open IBM Storwize V7000 term Control enclosure Expansion enclosure Node canister Host mapping Internal storage Managed disk (MDisk) Storage pool Thin provisioning or thin provisioned Volume Extent Definition A hardware unit that includes the chassis, node canisters, drives, and energy sources including batteries. A hardware unit that includes expansion canisters, drives, and energy sources that do not include batteries. A hardware unit that includes the node hardware, fabric and service interfaces, and SAS expansion ports. A controlling process in which hosts have access to only specific volumes within a cluster. Array-managed disks and drives that are held in enclosures and nodes that are part of the cluster. A component of a storage pool that is managed by a cluster. An MDisk is either a part of a Redundant Array of Independent Disks (RAID) array of internal storage or a SCSI logical unit (LU) for external storage. An MDisk is not visible to a host system on the SAN. A collection of storage capacity that provides the capacity requirements for a volume. The ability to define a storage unit (full system, storage pool, or volume) with a logical capacity size that is larger than the physical capacity assigned to that storage unit. A discrete unit of storage on disk, tape, or other data recording medium that supports some form of identifier and parameter list, such as a volume label or I/O control. Each MDisk is divided into segments of equal size called extents. When a volume is created from a storage pool, the volume is allocated based on the number of extents required to satisfy the capacity requirements for the volume. Table 1: IBM Storwize V7000 Unified system terminology 4

7 Benefits of Oracle deployment on the IBM Storwize V7000 Unified system The IBM Storwize V7000 Unified system provides greater flexibility for deploying database solution though its architecture. Customers can use the file protocol (NFS) or block storage to deploy database environments. Both, the file and block storages, can be used on the same database host. Table 2 gives a summary of deployment benefits of Oracle database on the IBM Storwize V7000 Unified system. Benefit Lower total cost of operation (TCO) Central management Greater flexibility Single global name space Intelligent policy engine High availability Description The IBM Storwize V7000 Unified system reduces the cost for administration and maintenance by scaling capacity and performance independent of each other. The IBM Storwize V7000 Unified system provides ease of use, maintenance, provisioning and implementation, and centralized management with alert and event logs through a single GUI or command line interface (CLI). The database can be created on NFS shares or on the block device. Data islands can be consolidated into a single globally-accessible network-based store alongside block databases store. Intelligent policy engine and tiered storage are built into each IBM Storwize V7000 Unified system. Automatic striping of file system allows the use of all available storage based on user-defined file system policies. The IBM Storwize V7000 Unified system with its unique robust architecture is able to provide the high-availability feature without any additional changes or setup. Table 2: Oracle database deployment benefits on the IBM Storwize V7000 Unified system 5

8 Planning Oracle databases on an IBM Storwize V7000 Unified system This guide describes the deployment of the Oracle databases on Linux using the IBM Storwize V7000 Unified system NFS data store or using FC connectivity. The topology components include IBM Storwize V7000 Unified System and IBM xseries systems. Figure 3 provides a pictorial representation of the setup created in the lab environment. Figure 3: Lab topology Lab setup: Two separate Linux hosts have been used to map storage. The Isvsonas1 host uses a dedicated 10 Gb network interface to mount NFS share from the file module of the Storwize V7000 Unified System using the NFS protocol. Alternatively, the isvx4 host uses FC connectivity to map the storage LUNs created on the IBM Storwize V7000 Unified system. For the ease of understanding, separate instructions are provided for configuring Storwize V7000 Unified system for file protocol and block access. Note: When creating the database, decide in advance on the storage media. It is not recommended to mix file and block storages in creating a single database. 6

9 Preparing the Storwize V7000 Unified system for Oracle deployment This section describes the ways to prepare the Storwize V7000 Unified system for Oracle database deployment using file-based storage by means of the NFS protocol and FC connectivity for block storage. NFS-based deployment of Oracle database This section provides the prerequisites for the creation of Oracle database on NFS storage provided by the IBM Storwize V7000 Unified system. The NFS shares needed for the Oracle database are created as described in the following sections. Creating NFS shares (using GUI) The IBM Storwize V7000 Unified system s NFS shares can be created using the GUI interface when connected through a browser or using CLI tools (such as mkexport) when connected as cliuser using PuTTY or a terminal emulator of your choice. When using the GUI, you need to provide the fully qualified domain name (FQDN). FQDN is given at the time of installation of the Storwize V7000 Unified system. You need to perform the following steps to create NFS shares using the Storwize V7000 Unified system s management console. In the lab setup, the Storwize V7000 Unified System is named ifs5.storage.tucson.ibm.com. 1. Go to the URL: 2. Log in to the console using the user ID, admin and the password that has been set at the time of the installation. Figure 4: Storwize V700 Unified system login 3. On the left side, click Shares under the Files submenu to start the file share creation wizard. In the panel displayed, enter the details regarding the share name that you want to create along with the directory path. 7

10 Figure 5: Storwize V7000 Unified system export wizard - step 1 4. Select the NFS type for the share and enter the values in the Path and Share name fields. Clear the Read only access, Root squash, and Secure check boxes. Figure 6: Storwize V7000 Unified system export wizard - step 2 5. Now, the Storwize V7000 Unified system has sufficient information to create the NFS share. Click OK to create the share. 8

11 Note: When creating or configuring shares for the Oracle database, leave the Root Squash check box cleared. Clearing the Root Squash check box here allows root user on the client systems to act as root user (only) within the share path. As soon as the share is created, a confirmation indicating the success or failure information is displayed. Creating NFS shares (using CLI) File shares can also be created using the Storwize V7000 Unified system s CLI interface. In order to create shares using the CLI, log in to the Storwize V7000 Unified system s integrated management node as cliuser. Syntax: mkexport <Share-Name> <Share-Path> --nfs <Client-FQDN/IP Address/*>(NFS options) Example: mkexport oradata01 /ibm/gpfs0/oradata_nfs nfs *(rw,no_root_squash,insecure,anonuid=505,anongid=500) Table 3 provides an explanation of the options used in the command. Option Share-name Share-path FQDN Description Name to be assigned to the share. Storwize V7000 Unified system or IBM General Parallel File System (IBM GPFS ) path to be exported. This command creates the path in the GPFS directory if it does not exist. FQDN option is mandatory only if both the systems (Storwize V7000 Unified system and the client) are in different domains. If the systems are in the same domain, just the short name or the host name might suffice the purpose. IP addresses and FQDN can be used irrespective of the domain of the two machines. NFS options rw,no_root_squash,insecure,anonuid=uidn,anongid=gidn (UIDn and GIDn are Oracle user s numeric user IDs on Linux clients) Table 3: Storwize V7000 Unified system command line syntax for creating NFS shares Note: After creating the shares (using GUI or the mkexport command) they need to be mounted on the client Linux systems. Special mount options must be used for mounting these shares. Also upon mounting the shares on the client systems, permissions on the local mount directory must be changed to Oracle user and dba group, in case anonuid and anongid options are not used. 9

12 Block-based deployment of Oracle database For simplicity in explaining the block storage configuration, refer to Figure 7. The figure shows how the RAID setup is done at the Storwize V7000 Unified system and how the mapping of Oracle volume group to Linux host is done using multiple controllers. Figure 7: RAID overview and V7000 Unified volume mappings to Linux hosts The following section explains how the Easy Tier configuration is done at the Storwize V7000 Unified systems in order to achieve the lab setup. 10

13 IBM Storwize V7000 configuration This section explains the configuration of the various components that were involved in testing. It includes: SAN configuration, MDisks configuration, storage pools, volumes, and hosts. SAN configuration planning For the best practices on SAN configuration planning, refer to the IBM Storwize V7000 Introduction and Implementation Guide from IBM Redbooks. Storwize V7000 overview In the Storwize V7000 system, the internal drives cannot be directly added to storage pools. They need to be included in a RAID to provide protection against the failure of individual drives. A RAID array is created as an MDisk, and during the array creation, wizards and presets are available to suggest configurations to users based on the hardware attached to the system. The recommendation is to use these presets for easy configuration and best performance. The following content shows how to create arrays on the system using the GUI. The Storwize V7000 CLI might not be needed though it is flexible in most cases. When you first set up the Storwize V7000 system, the initial setup wizard prompts the user to use the fullyautomated RAID setup. Now, start from the getting started page on the GUI, as shown in Figure 8 Figure 8: Getting started with storage configuration 11

14 Figure 9 shows the internal drives with an Online status before the MDisk configuration. Figure 9: Internal drives configuration Storage pool configuration This section explains how to create a storage pool on the system using the GUI. The Storwize V7000 CLI might not be needed though it is flexible in most cases. 1. On the Storwize V7000 GUI, click Pools, which is the third icon from the top. Then click MDisks by Pools. Figure 10: Storwize V7000 Unified pools configuration 12

15 2. Next, click New Poll on the GUI, and in the pop-up window that is displayed, enter StoragePool_Oracle_FC as the name of the new storage pool that will be used for the block storage testing, and then click Next. Figure 11: Creating storage pool 3. You will see the screen as shown in the Figure 12, that we have not created any MDisks at this point. Click Finish to continue with the creation of the storage pool. Figure 12: Creating a storage pool 13

16 MDisk configuration You can choose to use the recommended (fully-automatic) configuration or a different configuration for customization. (Refer to Figure 13). In this example, a different configuration has been used. In this step, first select the drive class. The GUI then provides a preset list based on the drive class. You can decide whether to configure spares automatically, and optimize the array for performance or capacity. You can also indicate the number of drives to be provisioned. The configuration summary at the bottom of the page (in Figure 14) displays information about what the system will attempt to do, or indicates that it cannot meet the RAID creation requirement. The team configured the internal SAS drives, and chose the preset of RAID-5. In this example, the team used 21 drives, optimized for a capacity to create three RAID-5 arrays. Figure 13: Configuring internal storage step 1 With the Optimize for Performance option selected, all arrays in a pool will have the same performance characteristics, and potentially leave unused drives in the system. The Optimize for Capacity option uses all that drives in the system, which means that some of the arrays might not meet the array width goals. It will do this by reducing the number of drives in one or more arrays to be created by an equal amount, and you will get an optimal performance layout (all arrays have the same width, or the width with one drive more or one drive less, but the width will not be the same as the width goal). Note: If the perfect number of drives exists, the Optimize for Capacity option creates the same layout as the Optimize for Performance option. Both the algorithms are explained in the IBM Redbooks, Implementing the IBM Storwize V7000 at ibm.com/redbooks/redpieces/abstracts/sg html?open Next, you need to decide whether to expand an exiting pool or create one or more new pools with the arrays. In this example, a new pool dedicated for internal storage (as shown in Figure 14) has been created. 14

17 Figure 14: Configuring internal storage step 2 Next, you need to decide whether to expand an exiting pool or create one or more new pools with the arrays. In this example, the test team added the new MDisks to the StoragePool_Oracle_FC storage pool that they created earlier. Figure 15: Configuring internal storage step 3 15

18 Now, you will see the new storage pool creation progress and the command line result, if the Details option is turned on (as shown in Figure 16). Figure 16: Details of the RAID array creation Figure 17 shows that the new MDisks: mdisk9, mdisk10, and mdisk11 that are now part of the StoragePool_Oracle_FC storage pool. Figure 17: MDisks overview by storage pool Next, the test team created another RAID-5 array, as shown in Figure

19 Figure 18: Creating another MDisk The team then added this to the existing StoragePool_Oracle_FC storage pool, as shown in Figure 19. Figure 19: Adding the MDisk to existing storage pool Figure 20 shows the actual command that is run to create the RAID-5 array, and add it to the StoragePool_Oracle_NFS storage pool. 17

20 Figure 20: Actual command that runs in the background On the Pools page, you can notice that a new StoragePool_Oracle_FC pool has been created, and it has three members mdisk9, mdisk10, and mdisk11, all having an Online status (as shown in Figure 21). Figure 21: Storage pools and arrays status You have now completed managing the internal HDDs. Volume configuration A volume is a logical disk that is presented to a host system by the IBM Storwize V7000 system. The IBM Storwize V7000 storage system translates this volume into a number of extents which are allocated across the MDisks present in the storage pool. 18

21 Figure 22: Volume configuration step 1 Select the StoragePool_Oracle_FC storage pool from which you want to create the new volumes. On the left pane, click Volumes by Pool and then click New Volume to launch the New Volume wizard. Then select the storage pool from which you want to create volumes (as shown in the Figure 23). Figure 23: Volume configuration step 2 19

22 Figure 24: Volume configuration step 3 Figure 25 shows detailed messages that are displayed when you click Create and Map to Host in the new volume creation wizard. Figure 25: Background command to create the volumes Mapping volumes to the host A host system is connected to the Storwize V7000 system either through a FC connection or an iscsi connection. For this paper, the team used FC to connect host to the IBM Storwize V7000 system. Hosts are defined to the IBM Storwize V7000 system by identifying their worldwide port numbers (WWPNs) for FC hosts. 20

23 After viewing the details, when you click Continue, the Modify Mappings dialog box (as shown in Figure 26) is displayed. In this example, the volumes are mapped to the isvx4 host, which is selected from the list. Figure 26: Volume mapping to host step 1 Click Next to navigate to the Modify Mappings page, as shown in Figure 27. Figure 27: Volume mapping to host step 2 On clicking Map Volumes, you can see that the volumes are mapped to the host, as shown in Figure

24 Figure 28: Volume mapping to host step 3 Figure 29 shows the one 1.0 TB volume (orcl_vol1) mapped to the isvx4 host. Figure 29: Volume overview 22

25 Prerequisite on the Linux host On the node running Linux, detect and configure the new device. It is assumed here that the zoning (soft/ hard) is already in place between the Storwize V7000 system and the Linux host. Run the multipath command to list the LUNs already configured on the system. isvx4 #> multipath d -l To configure the new devices that are created on the Storwize V7000 system and to map them to the isvx4 host, run the ls /sys/class/fc_host command. isvx4 # > ls /sys/class/fc_host The hosts returned by the ls command are then further scanned for attached SCSI targets using the echo command. echo "- - -" > /sys/classes/scsi_host/hostname_returned_by_ls_command/scan The echo command scans and maps the exposed LUN s to the isvx4 host. The sample output of the multipath command after mapping the LUNs on the isvx4 host is as follows. [root@isvx4 app]# multipath -d -l mpath28 ( f) dm-2 IBM,2145 [size=500g][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 5:0:2:2 sdj 8:144 [active][undef] \_ 5:0:3:2 sdk 8:160 [active][undef] \_ 6:0:2:2 sdl 8:176 [active][undef] \_ 6:0:3:2 sdm 8:192 [active][undef] mpath27 ( ) dm-1 IBM,2145 [size=75g][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 5:0:2:1 sdc 8:32 [active][undef] \_ 5:0:3:1 sde 8:64 [active][undef] \_ 6:0:2:1 sdg 8:96 [active][undef] \_ 6:0:3:1 sdi 8:128 [active][undef] mpath26 ( ) dm-0 IBM,2145 [size=1.0t][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 5:0:2:0 sdb 8:16 [active][undef] \_ 5:0:3:0 sdd 8:48 [active][undef] \_ 6:0:2:0 sdf 8:80 [active][undef] \_ 6:0:3:0 sdh 8:112 [active][undef] 23

26 Installation requirements for Oracle Before you install the Oracle database product on the Linux operating systems, ensure that the system you choose meets the necessary operating system, hardware, software, and communications requirements. Oracle software installation has specific requirements for setting up kernel parameters, user limits, and bash profile setting, which is part of the preinstallation checklist. A detailed documentation regarding installation is available on the Oracle website and is also part of the installation software bundle. If you plan to install Oracle Automatic Storage Manager (ASM), the Oracle Grid Infrastructure software is needed. For grid infrastructure setup, considerations such as grid software owner, multiple disk groups for database or log files, and flash recovery area must be made before installation. Multiple LUNs can be created and mapped easily to database host using the process described in this paper for the preparation of storage. Oracle 11gR2 Certification & Supportability Statement IBM supports the use of IBM SONAS and IBM Storwize V7000 Unified systems as NAS attached architectures, with Oracle Database 11gR2 on IBM Power (AIX) and on IBM x86 (RHEL Linux) server platforms. Certification and supportability details are available on My Oracle Support link ( as Oracle Metalink [ ]. Oracle software installation It is assumed at this point that the prerequisites for the Storwize V7000 Unified system and for the Linux database host are complete. The LUNs / NFS shares are already mapped to the Linux database host, and the Oracle user has write permissions on the needed block devices or the NFS shares. The test team lab setup was done using the Oracle 11gR2 install kit 2 ( ), with a specific patch Detailed instructions regarding the application for the patch during or after the installation of the Oracle software are included with the software patch. For more details about the software installation, refer to Oracle installation guide for Linux. Database creation In the lab setup, the ext3 file system was created on the block devices mapped to the Linux database host and were mounted as /oradata for storing the data and log file. For the NFS-based database creation, the NFS shares were mounted using specific mount options discussed in the Oracle support document, For the actual creation of the database, Oracle Database Creation Assistant (DBCA) was used. Refer to the Oracle database administrator s guide details for using DBCA. Note: The database creation using ASM and block devices is not discussed here. For details regarding deployment or creation of a single instance database using ASM and block storage, refer to the Deploying Oracle 11g RAC Release 2 with IBM Storwize V7000 on Red Hat Enterprise Linux document. 24

27 Direct NFS setup Direct NFS is a functionality offered by Oracle by means of Oracle supplied NFS client. You can set up the Direct NFS client by: Creating the required number of database shares as listed in the Creating NFS shares (using GUI) section Creating Oracle Direct NFS specific configuration files For Oracle 11gR2 single instance installation, IBM Storwize V7000 Unified is used as a NAS device, making the storage available through the NFS protocol. Oracle database software binary files are installed locally across the database server and the database resides on Storwize V7000 Unified system. Note: The Direct NFS functionality is used by the database software only. It is important that no database instances are running during the configuration of Direct NFS. Direct NFS configuration To configure Direct NFS, perform the following steps: Install a checksum patch on all the Storwize V7000 Unified nodes. You need to contact the IBM Advanced Technical Services (ATS) team in order to obtain the patch and perform the installation. The installation of the patch needs a reboot of the interface node. Stop all the databases instances accessing Storwize V7000 Unified System shares before you install the patch. Stop all running instances on all database hosts using SQL*Plus shutdown immediate command. You must be connected to the database instance with SYSDBA or SYSOPER privilege in order to shut it down. SQL> shutdown immediate; Rename the libodm library from ORACLE_HOME and create a symbolic link mv $ORACLE_HOME/lib/libodm11.so $ORACLE_HOME/lib/libodm11.so.BAK ln s $ORACLE_HOME/lib/libnfsodm11.so libodm11.so Create the oranfstab file at one of the locations listed in Table 14. The lab setup configuration options are listed in Table 15. Care must be taken when choosing the location of the oranfstab file as it can impact a single or all the databases on the host system. File location ORACLE_HOME/dbs/oranfstab /etc/oranfstab /etc/mtab Purpose Specific to single database Globally available to all databases running on the same host Oracle reads NFS mount points from this file 25

28 Table 4: Direct NFS configuration setup files specification 26

29 Table 15 shows a sample oranfstab file created in the /etc location. /etc/oranfstab server: ifs5 path: local: export /ibm/gpfs0/oradata_nfs/db1 export /ibm/gpfs0/oradata_nfs/flash1 export /ibm/gpfs0/oradata_nfs/redo1 export /ibm/gpfs0/oradata_nfs/archive1 Table 5: Direct NFS configuration options (sample oranfstab file) mount /u04/db1 mount /u05/flash1 mount /u06/redo1 mount /u07/archive1 The database alert log entries listed in Table 16 shows that the server name ifs5 is resolved through DNS to multiple IP addresses. It is recommended to put in all the public IP address here, during Direct NFS setup. It is recommended that you use only one Storwize V7000 Unified interface node here. Verification of the Direct NFS setup You can check if Direct NFS is active through: Database alert log entries Existence of symbolic link libodm11.so pointing to libnfsodm11.so Querying the database instance v$ views The alert log file for the database will show the entries similar to those listed in Table 16 Database alert log entries Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0 Direct NFS: channel id [0] path [ ] to filer [ifs5] via local [ ] is UP Direct NFS: channel id [1] path [ ] to filer [ifs5] via local [ ] is UP Table 6: Database alert log entries On Linux, the existence of symbolic link indicates whether Direct NFS is enabled. Linux symbolic link [oracle@sonasisvlinux1: $] cd /u01/app/oracle/product/11.2.0/dbhome_1/lib [oracle@sonasisvlinux1: $] ls -l libodm11.so lrwxrwxrwx 1 oracle oinstall 56 Nov 10 22:50 libodm11.so ->libnfsodm11.so Table 7: Linux symbolic link 27

30 The database instance V$dnfs views can also be queried to check the use of Direct NFS client. V v$dnfs_server col svrname for a10 col dirname for a30 select * from v$dnfs_servers; ID SVRNAME DIRNAME MNTPORT NFSPORT WTMAX RTMAX ifs5 /ibm/gpfs0/oradata_nfs/db ifs5 /ibm/gpfs0/oradata_nfs/redo Table 8: Database instance v$ view query Note: The Oracle Direct NFS setup white paper with the metalink document ID: describe the guidelines for configuring Direct NFS using the NFS server. The Storwize V7000 Unified system provides access to data files using the NFS protocol. Because of this, the configuration of Direct NFS differs when using Storwize V7000 Unified System, mainly when Microsoft Active Directory (AD) is used for authentication. When a share is created on the Storwize V7000 Unified system, it is owned by root by default. These shares must be changed to database user ID and database user s group ID. For ease of administration, this user should be centrally authenticated. This user is assigned as anonuid and anongid (with the same UID and GID as on the local system for oracle:dba) to the Storwize V7000 Unified share. When using Lightweight Directory Access Protocol (LDAP) as a preferred authentication method, anonuid and anongid can be created with the same UID and GID as that of the Oracle user (existing on cluster nodes). When AD is used as the preferred authentication method, following considerations must be made. AD does not support UNIX UID; instead it uses names (user names or group names) which are internally translated as S-nnn (where nnn is some numeric identifier) When mapping is done for a UID in AD, mapped user IDs are registered in Samba WINBIND with numbers greater than Oracle supports user IDs only up to For AD to understand UIDs, Microsoft Services for UNIX must be installed. You need to manually change the ownership of the database directories to the UID or GID of the Oracle user when using AD. When using LDAP, you can simply use anonuid or anongid that you created in LDAP, as it understands UID and GID. 28

31 Lab setup The lab environment under which this setup was tested had the following configuration and the tests validated the setup that the team created using the configuration mentioned in this section. However, when creating the setup for the production environment, depending on the number of database instances that run on the system, the hardware configuration of the system must be validated. Storwize V7000 The following table lists the Storwize V7000 configuration used in the exercise. Storage name Node model Micro code level Table 9: Storwize V7000 Unified system details Host nodes ifs5isv.storage Storwize V7000 The following table lists the Linux hosts that were used in the exercise. Server type Processor Memory IBM xseries 3650 M3 servers Intel Xeon processor 128 GB Operating system Red Hat Enterprise Linux 5.5 Table 10: Linux hosts details 29

32 Summary This paper described the tests performed to validate a database solution consisting of the Oracle database software, the IBM xseries 3650 M3 server hosts running Red Hat Enterprise Linux, and the IBM Storwize V7000 Unified storage for external storage. The planning section listed the various components, their compatible versions, and the recommended steps for ease of planning and deployment. It is recommended that you review the information about the Oracle software and the Storwize V7000 Unified system at the websites included in the Resources section. 30

33 Acknowledgements This paper could not have been completed without the valuable suggestions and validation efforts from the Storwize V7000 team and Oracle OCE team. Special thanks to Mayur Shetty for all his efforts in setting up the block storage configuration on the Storwize V7000 Unified system and co-authoring the block configuration setup section in this white paper. 31

34 Resources The following websites provide useful references to supplement the information contained in this paper: IBM Systems on PartnerWorld ibm.com/partnerworld/systems IBM Storwize V7000 Unified Information Center IBM Redbooks ibm.com/redbooks Deploying Oracle 11g RAC Release 2 with IBM Storwize V7000 on Red Hat Enterprise Linux at IBM Techdocs ibm.com/support/techdocs/atsmastr.nsf/webindex/wp IBM Power Systems Information Center Oracle Database administrators reference guide Oracle Database Installation guide for Linux 32

35 About the author Shashank Shingornikar is a senior software engineer in the IBM Systems and Technology Group ISV Enablement Organization. He has more than 10 years of experience working with DBMS technologies, involved in high availability areas. You can reach Shashank at shashank.shingornikar@in.ibm.com. Mayur Shetty is a senior software engineer in the IBM Systems and Technology Group ISV Enablement Organization for Storwize V7000. He has more than 10 years of experience working with Storwize V7000 technologies. You can reach Mayur at mayor.shetty@us.ibm.com 33

36 Trademarks and special notices Copyright IBM Corporation References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is 34

37 presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. 35