Oracle 12c deployment using iscsi with IBM Storwize V7000 for small and medium businesses Reference guide for database and storage administrators

Transcription

1 Oracle 12c deployment using iscsi with IBM Storwize V7000 for small and medium businesses Reference guide for database and storage administrators Shashank Shingornikar IBM Systems and Technology Group ISV Enablement March 2014 Copyright IBM Corporation, 2014

2 Table of contents Abstract...1 Intended audience... 1 Scope... 1 Introduction to the IBM Storwize V7000 system...2 Introduction to Oracle 12c...3 IBM Storwize V7000 configuration...4 IBM Storwize V7000 terminology... 4 iscsi overview...5 Understanding iscsi basics... 5 Planning for deployment...6 Ethernet network design... 6 Windows considerations... 6 Linux considerations... 7 Configuring iscsi on Storwize V Creating and mapping Storwize V7000 volumes using iscsi... 8 Configuring the Linux host...10 Configuring Windows host...17 Lab setup...21 Storwize V Linux host details Windows host details Lab topology Oracle 12c software options Container database creation...24 Pluggable database creation...25 Database backup and restore...25 Summary...26 Resources...27 About the author...27 Trademarks and special notices...28

3 Abstract This paper explores the use of the Internet Small Computer System Interface (iscsi) protocol provided by the IBM Storwize V7000 storage product as an essential component of the infrastructure solution with Oracle database. The Oracle 12c database offers an excellent choice for small- and medium-sized businesses, looking to reduce IT complexity and cost through the deployment of private database cloud. Combined with the iscsi protocol offered by Storwize V7000, customers can quickly deploy a database cloud by reusing / making minimal changes to the existing infrastructure and also benefit from Oracle 12c enhancements such as storage management, availability, and security and continue to get a high return on their database investments. Intended audience This technical paper is intended for: Customers and prospects looking to implement the iscsi protocol of the IBM Storwize V7000 system in their existing infrastructure. Database, network, and storage administrators seeking detailed information about Oracle 12c database implementation using the iscsi protocol of the IBM Storwize V7000 system for Microsoft Windows and Linux operating systems. Scope Although IBM Storwize V7000 is perfectly capable of supporting both Fibre Channel (FC) and iscsi protocols, in this paper, the discussion is limited to storage exposed over iscsi only. This technical paper provides: Guidelines for implementing the Oracle 12c database using the iscsi protocol with the IBM Storwize V7000 storage. Configuration aspects when using iscsi in Linux and Windows environment. Links to relevant documents or web pages. This technical paper does not: Discuss any performance impact and analysis from a user perspective. Replace any official manuals and documents from IBM on Storwize V7000. Replace any official manual or support document provided by Oracle. 1

4 Introduction to the IBM Storwize V7000 system The IBM Storwize V7000 system combines hardware and software to control the mapping of storage into volumes in a storage area network (SAN) environment. The Storwize V7000 system provides many benefits to storage administrators, including simplified storage administration, integrated management of IBM servers and storage, and enterprise-class performance, function, and reliability. Figure 1: IBM Storwize V7000 The Storwize V7000 system includes rack-mounted units called enclosures. Each enclosure can be a 12- or 24-drive model, and has two canisters and two power supplies located at the back. There are two types of enclosures: control and expansion. A system can support more than one control enclosure, and a single control enclosure can have several expansion units attached to it. Node canisters are always installed in pairs as part of a control enclosure. Each control enclosure represents an I/O group. Any expansion enclosure that is attached to a specific control enclosure also belongs to the same I/O group. The system also includes an easy-to-use management graphical user interface (GUI), which helps you to configure, troubleshoot, and manage the system. For more information about the IBM Storwize V7000 system, refer to the URL: pic.dhe.ibm.com/infocenter/storwize/ic/index.jsp 2

5 Introduction to Oracle 12c The following new features are introduced in the Oracle 12c database release. Application development - Oracle Application Express Business intelligence and data warehousing Compression and Archiving Database consolidation - Multitenant architecture - Pluggable databases High availability and manageability Performance Security As the database features usage varies per organization, it might be difficult to visualize all the scenarios. To keep the deployment simple, the paper highlights the prerequisites for implementation of the database on Windows and Linux operating systems. Both these environments are widely used in small and medium business (SMB) segments. Storwize V7000 is also a popular storage choice due to its simplified storage administration, integrated management of IBM servers and storage, and enterprise-class performance, function, and reliability. iscsi is an Internet Protocol (IP)-based storage networking protocol. The basic concept is to take SCSI commands and encapsulate them in Transmission Control Protocol/Internet Protocol (TCP/IP) packets to transmit data from the storage drives to the server. Because TCP packets can be lost and retransmitted, they do not have to arrive in order. iscsi also has to keep track of incoming packets to make sure that all of the SCSI commands are queued in the correct order. To understand the networking configuration best practices when using the iscsi protocol, refer to the Networking best practices with IBM Storwize V7000 and iscsi guide at: ibm.com/partnerworld/wps/servlet/redirectservlet?cmsid=stg_ast_sto_wp_best-practices-with-ibmstorwize-v7000&attachmentname=networking_best_practices_with_storwize_v7000.pdf. The networking best practices guide also describes iscsi configuration in great detail along with the creation and mapping of the Storwize V7000 volumes. 3

6 IBM Storwize V7000 configuration This section explains the configuration of the various components that were involved in testing. It includes: SAN configuration, managed disks (MDisks) configuration, storage pools, volumes, and hosts. This section explains the common terminology used in Storwize V7000. IBM Storwize V7000 terminology Table 1 lists the IBM Storwize V7000 terminology used in the paper. For an entire list of the Storwize V7000 terminology, refer to IBM Storwize V7000 Introduction and Implementation Guide at ibm.com/redbooks/redpieces/abstracts/sg html?open IBM Storwize V7000 term Control enclosure Expansion enclosure Node canister Host mapping Internal storage MDisk Storage pool Thin provisioning or thin provisioned Volume Extent Definition A hardware unit that includes the chassis, node canisters, drives, and energy sources including batteries. A hardware unit that includes expansion canisters, drives, and energy sources that do not include batteries. A hardware unit that includes the node hardware, fabric and service interfaces, and serial-attached SCSI (SAS) expansion ports. A controlling process in which hosts have access to only specific volumes within a cluster. Array-managed disks and drives that are held in enclosures and nodes that are part of the cluster. A component of a storage pool that is managed by a cluster. An MDisk is either a part of a Redundant Array of Independent Disks (RAID) array of internal storage or a SCSI logical unit (LU) for external storage. An MDisk is not visible to a host system on the SAN. A collection of storage capacity that provides the capacity requirements for a volume. The ability to define a storage unit (full system, storage pool, or volume) with a logical capacity size that is larger than the physical capacity assigned to that storage unit. A discrete unit of storage on disk, tape, or other data recording medium that supports some form of identifier and parameter list, such as a volume label or I/O control. Each MDisk is divided into segments of equal size called extents. When a volume is created from a storage pool, the volume is allocated based on the number of extents required to meet the capacity requirements for the volume. Table 1: IBM Storwize V7000system terminology 4

7 iscsi overview This section provides an overview of iscsi and explains the basics of it. Understanding iscsi basics iscsi is an IP-based storage networking protocol. The basic concept is to take SCSI commands and encapsulate them in thetcp/ip packets to transmit data from the storage drives to the server. Because TCP packets can be lost and retransmitted, they do not have to arrive in order. iscsi also has to keep track of incoming packets to make sure that all the SCSI commands are queued in the correct order. iscsi was originally started by IBM and was developed as a proof of concept in In March 2000, the first draft of the iscsi standard was presented to the Internet Engineering Task Force (IETF). IBM offered iscsi-based storage devices in July 2001 even before the iscsi specification was passed by the IP Storage Working Group in August The two fundamental concepts to explain in iscsi are initiator and target. The initiator is the server, or client, which initiates the request for storage block. The target is the storage device that accepts the request and responds by providing the storage blocks. Each initiator and target is given a unique iscsi name such as an iscsi qualified name (IQN) or an extended unique identifier (EUI). iscsi is fundamentally a SAN protocol similar to Fibre Channel. The key difference is that FC uses a specialized (FC) network and iscsi uses TCP networks. This allows you to use a single network for storage data and other communication, as shown in Figure 2. Figure 2: iscsi overview 5

8 Planning for deployment Following are some of the key points to remember when planning for deployment. Ethernet network design For the best performance, the iscsi SAN network must be isolated from the other networks by dedicating Ethernet switches for iscsi traffic, or using virtual local area networks (VLANs) to separate networks within a switch. Plan in advance the network infrastructure and the storage network infrastructure that your deployment will require. Consider following key points when designing the network. Network infrastructure bandwidth Switches should be evaluated for required proper backplane bandwidth Queue depth (applies to both iscsi initiator and Storwize V7000) Flow control Unicast storm control Jumbo frames enablement Windows considerations Following section provides Windows platform specific information. Microsoft iscsi Software Initiator Microsoft iscsi Software Initiator enables the connection between Windows host computer to an external iscsi-based storage array through an Ethernet network adapter. It enables access to block-based SANs in your existing network infrastructure, which provide iscsi target functionality without having to invest in additional hardware, and to enable the use of iscsi storage devices for small and medium businesses. For more information regarding Microsoft iscsi Software Initiator, refer to: technet.microsoft.com/en-us/library/ee338476%28v=ws.10%29.aspx Microsoft Multipath I/O To meet the growing requirement of data availability, centralized storage must be readily available and immune to outages. Multipathing is the ability of a system to use more than one read/write path to a storage device. It is a solution that provides fault tolerance against a single point-of-failure in hardware components. The Microsoft Multipath I/O (MPIO) framework helps ensure data availability at all times. MPIO supports multiple data paths to storage, improves fault tolerance of the storage connection, and in some cases, provides greater aggregate throughput by using multiple paths at the same time. This helps improve system and application performance. For more information and configuration of MPIO, refer to: technet.microsoft.com/en-us/library/ee619778%28v=ws.10%29.aspx 6

9 Linux considerations Following section lists the terminology used on Linux platform when working with iscsi. Session The group of TCP connections that link an initiator with a target form a session. TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one and the same target. iscsid iscsid is a daemon on Red Hat Enterprise Linux (RHEL) that implements the control path of iscsi protocol, along with some management facilities. This daemon can be configured to automatically restart discovery at startup, based on the contents of a persistent iscsi database available under /var/lib/iscsi/. iscsiadm On RHEL, the iscsiadm utility is a command-line tool allowing discovery and login to iscsi targets, as well as access and management of the open-iscsi database. Open-iscsi uses the term node to refer to a portal on a target. For session mode, a session ID (sid) is used. The ID sid of a session can be found by running iscsiadm m session P 1 comamnd. The sid and sysfs path are not currently persistent and are partially determined when the session is setup. Many of the node and discovery operations require the iscsi daemon (iscsid) to be running. Multipath To maintain a constant connection between a host and its storage, a technique called multipathing is used. This technique allows the use of more than one physical path that transfers data between the host and an external storage device. For example, when multiple IP addresses are configured per iscsi target, each logical unit number (LUN) is shown per IP target address. So, if four IP address are configured on Storwize V7000, then on RHEL system with two network interfaces, mapped LUN device will be shown eight times. On the RHEL system, the dm-multipath module provides the functionality of multipathing. Based on the configuration created in /etc/multipath.conf, the multipath command creates the multipathed devices. For this paper, Oracle Automatic Storage Manager (ASM) is used as the storage manager for database files. In the following sections, the complete process of ASM volume creation is described. The process is involves the following steps. Points 4 to 7 need operating system specific considerations for Windows and Linux. 1. Configuring iscsi on Storwize V Creating virtual disks (volumes) from MDisks 3. Mapping volumes to physical host 4. Using the iscsiadm utility to log on to targets 5. Configuring multipath.conf 6. Cerating udev rules for persistent volume mapping 7. Using the ASM configuration assistant (asmca) to create and mount ASM disk groups. 7

10 Configuring iscsi on Storwize V7000 This section describes the iscsi configuration setup on Storwize V7000 systems created using GUI. You can start the iscsi configuration wizard by clicking Settings Network. 10 Gbps ports are configured for iscsi with a unique IP address per port on each node, thus providing redundant channels of communication per node (refer to Figure 3). The iscsi name assigned to the system is also displayed on this page and it cannot be modified, although an iscsi alias can be given in a free-form text available. Figure 3: iscsi configuration Both 1 GB and 10 GB Ethernet ports can be used for iscsi traffic, but only the 1 GB Ethernet ports can be used for management traffic All IP addresses (service and configuration) associated with a clustered-system Ethernet port must be on the same subnet. However, IP addresses associated with a node Ethernet port used for iscsi traffic can be configured to belong to different subnets. For more information about iscsi configuration, refer to: pic.dhe.ibm.com/infocenter/storwize/ic/topic/com.ibm.storwize.v7000.doc/svc_rulesiscsi_334gow.html Creating and mapping Storwize V7000 volumes using iscsi A volume is a logical disk that the clustered system presents to a host connected over a Fibre Channel or Ethernet network. Using volumes allows administrators to more efficiently manage storage resources. A storage pool is a collection of managed disks that jointly contains all the data for a specified set of volumes. First, the system divides each managed disk in the pool into extents storage blocks that are typically of equal size. Next, the administrator creates the volumes, a process that consumes the extents, and then maps them to the host objects. As a result, the clustered system presents the volumes to the hosts. 8

11 Volumes are presented to hosts by host mapping, the process of controlling where hosts have access to specific volumes in the clustered system. Administrators can create logical hosts using Fibre Channel, Fibre Channel over Ethernet (FCoE), or iscsi technology. Volumes can then be mapped to a host. You have the ability to create different types of volumes, including mirrored, thin provisioned, and compressed volumes. Although it is possible to map the volumes using command-line interface (CLI), same can also be done through a GUI wizard, as shown in Figure 4. Figure 4: New volume mapping (step 1) Selecting a new volume opens a dialog box as shown in Figure 5, allowing users to create a volume based on presets. Figure 5: Volume creation based on presets 9

12 After completing the final step (as shown in Figure 6), the volumes are mapped to the host. Figure 6: Volume mapping Configuring the Linux host The following section describes how to perform volume mapping on Linux. 1. Discover the targets using the sendtargets command. ~]# iscsiadm --mode discovery --type sendtargets --portal :3260,1 iqn com.ibm:2145.ifs3.node :3260,1 iqn com.ibm:2145.ifs3.node1 ~]# iscsiadm --mode discovery --type sendtargets --portal :3260,1 iqn com.ibm:2145.ifs3.node :3260,1 iqn com.ibm:2145.ifs3.node1 ~]# iscsiadm --mode discovery --type sendtargets --portal :3260,1 iqn com.ibm:2145.ifs3.node :3260,1 iqn com.ibm:2145.ifs3.node2 ~]# iscsiadm --mode discovery --type sendtargets --portal :3260,1 iqn com.ibm:2145.ifs3.node :3260,1 iqn com.ibm:2145.ifs3.node2 2. Log in to the discovered targets. ~]# iscsiadm --mode node --target iqn com.ibm:2145.ifs3.node1 --portal login -L automatic Logging in to [iface: default, target: iqn com.ibm:2145.ifs3.node1, portal: ,3260] (multiple) Login to [iface: default, target: iqn com.ibm:2145.ifs3.node1, portal: ,3260] successful. ~]# iscsiadm --mode node --target iqn com.ibm:2145.ifs3.node1 --portal login Logging in to [iface: default, target: iqn com.ibm:2145.ifs3.node1, portal: ,3260] (multiple) Login to [iface: default, target: iqn com.ibm:2145.ifs3.node1, portal: ,3260] successful. ~]# iscsiadm --mode node --target iqn com.ibm:2145.ifs3.node2 --portal login Logging in to [iface: default, target: iqn com.ibm:2145.ifs3.node2, portal: ,3260] (multiple) Login to [iface: default, target: iqn com.ibm:2145.ifs3.node2, portal: ,3260] successful. ~]# iscsiadm --mode node --target iqn com.ibm:2145.ifs3.node2 --portal login Logging in to [iface: default, target: iqn com.ibm:2145.ifs3.node2, portal: ,3260] (multiple) Login to [iface: default, target: iqn com.ibm:2145.ifs3.node2, portal: ,3260] successful. 10

13 After successful login, the iscsi database is created under /var/lib/iscsi ~]# find /var -name "*node*" /var/lib/iscsi/nodes /var/lib/iscsi/nodes/iqn com.ibm:2145.isvifs2.node1 /var/lib/iscsi/nodes/iqn com.ibm:2145.ifs3.node1 /var/lib/iscsi/nodes/iqn com.ibm:2145.ifs3.node2 /var/lib/iscsi/nodes/iqn com.ibm:2145.isvifs1pune.node1 /var/lib/iscsi/nodes/iqn com.ibm:2145.tiger.node1 /var/lib/iscsi/send_targets/ ,3260/iqn com.ibm:2145.isvifs1pune.node1, ,3260,1,default /var/lib/iscsi/send_targets/ ,3260/iqn com.ibm:2145.ifs3.node1, ,3260,1,default /var/lib/iscsi/send_targets/ ,3260/iqn com.ibm:2145.ifs3.node1, ,3260,1,default /var/lib/iscsi/send_targets/ ,3260/iqn com.ibm:2145.tiger.node1, ,3260,1,default /var/lib/iscsi/send_targets/ ,3260/iqn com.ibm:2145.ifs3.node2, ,3260,1,default /var/lib/iscsi/send_targets/ ,3260/iqn com.ibm:2145.ifs3.node2, ,3260,1,default /var/lib/iscsi/send_targets/ ,3260/iqn com.ibm:2145.isvifs2.node1, ,3260,1,default 3. Check the availability of the multipath daemon. ~]# /etc/init.d/multipathd status multipathd (pid 2177) is running Run the multipath ll command. ~]# multipath -ll mpathaf ( d5c b) dm-5 IBM,2145 size=2.0t features='1 queue_if_no_path' hwhandler='0' wp=rw -+- policy='round-robin 0' prio=50 status=active - 14:0:0:1 sdh 8:112 active ready running `- 15:0:0:1 sdj 8:144 active ready running `-+- policy='round-robin 0' prio=10 status=enabled - 12:0:0:1 sdd 8:48 active ready running `- 13:0:0:1 sdf 8:80 active ready running mpathae ( d5c a) dm-3 IBM,2145 size=2.0t features='1 queue_if_no_path' hwhandler='0' wp=rw -+- policy='round-robin 0' prio=50 status=active - 12:0:0:0 sdc 8:32 active ready running `- 13:0:0:0 sde 8:64 active ready running `-+- policy='round-robin 0' prio=10 status=enabled - 14:0:0:0 sdg 8:96 active ready running `- 15:0:0:0 sdi 8:128 active ready running 5. Edit the multipath.conf file to create aliases. [root@isvhost2 ~]# vi /etc/multipath.conf multipaths{ multipath{ wwid d5c a alias VOL-2TB-3 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight priorities no_path_retry 5 } } multipath{ wwid d5c b alias VOL-2TB-4 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight priorities no_path_retry 5 } 11

14 6. Reload the multipathing configuration. This renames the devices discovered to the respective aliases configured in /etc/multipath.conf. ~]# multipath -r Dec 24 12:02: d5c a: rename mpathae to VOL-2TB-3 rename: VOL-2TB-3 ( d5c a) undef IBM,2145 size=2.0t features='1 queue_if_no_path' hwhandler='0' wp=undef `-+- policy='round-robin 0' prio=30 status=undef - 12:0:0:0 sdc 8:32 active ready running - 13:0:0:0 sde 8:64 active ready running - 14:0:0:0 sdg 8:96 active ready running `- 15:0:0:0 sdi 8:128 active ready running Dec 24 12:02: d5c b: rename mpathaf to VOL-2TB-4 rename: VOL-2TB-4 ( d5c b) undef IBM,2145 size=2.0t features='1 queue_if_no_path' hwhandler='0' wp=undef `-+- policy='round-robin 0' prio=30 status=undef - 12:0:0:1 sdd 8:48 active ready running - 13:0:0:1 sdf 8:80 active ready running - 14:0:0:1 sdh 8:112 active ready running `- 15:0:0:1 sdj 8:144 active ready running 7. Create the partitions on the discovered devices. (User responses are indicated in bold letters) [root@isvhost2 ~]# fdisk /dev/sdd WARNING: The size of this disk is 2.2 TB ( bytes). DOS partition table format can not be used on drives for volumes larger than ( bytes) for 512-byte sectors. Use parted(1) and GUID partition table format (GPT). WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): p Disk /dev/sdd: GB, bytes 255 heads, 63 sectors/track, cylinders Units = cylinders of * 512 = bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x69a2d0e4 Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder ( , default 1): Using default value 1 Last cylinder, +cylinders or +size{k,m,g} ( , default ): Using default value Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. 12

15 Same steps are run for device /dev/sdd. The partitions created can be checked using the fdisk command. ~]# fdisk -l grep [MGT]B Disk /dev/sdc: GB, bytes Disk /dev/sdd: GB, bytes Disk /dev/mapper/vol-2tb-3: GB, bytes /dev/mapper/vol-2tb-3p Linux Disk /dev/mapper/vol-2tb-3p1: GB, bytes Disk /dev/mapper/vol-2tb-4: GB, bytes /dev/mapper/vol-2tb-4p Linux Disk /dev/mapper/vol-2tb-4p1: GB, bytes Verify the created dm devices listing major and minor numbers (Entries listed in bold letters indicates new partitions) [root@isvhost2 ~]# dmsetup ls VOL-2TB-4 (253, 5) VOL-2TB-3p1 (253, 4) VOL-2TB-4p1 (253, 6) VOL-2TB-3 (253, 3) vg_isvhost2-lv_home (253, 2) vg_isvhost2-lv_swap (253, 1) vg_isvhost2-lv_root (253, 0) [root@isvhost2 ~]# ls -l /dev/dm-* brw-rw root disk 253, 0 Dec 12 13:07 /dev/dm-0 brw-rw root disk 253, 1 Dec 12 13:07 /dev/dm-1 brw-rw root disk 253, 2 Dec 12 13:07 /dev/dm-2 brw-rw root disk 253, 3 Dec 24 12:02 /dev/dm-3 brw-rw root disk 253, 4 Dec 24 12:02 /dev/dm-4 brw-rw root disk 253, 5 Dec 24 12:02 /dev/dm-5 brw-rw root disk 253, 6 Dec 24 12:02 /dev/dm-6 9. Create oinstall, asamadmin, asmdba and asmoper groups. [root@isvhost2 ~]# groupadd oinstall [root@isvhost2 ~]# groupadd dba [root@isvhost2 ~]# groupadd asmdba [root@isvhost2 ~]# groupadd asmoper [root@isvhost2 ~]# groupadd asmadmin 10. Create grid user defining oinstall as primary group and asmadmin, asmdba, asmoper as secondary groups so that you can set the permissions on iscsi devices using udev rules. [root@isvhost2 ~]# useradd -g oinstall -G asmadmin,asmdba,asmoper,dba -c "Grid user" grid [root@isvhost2 ~]# passwd grid Changing password for user grid. New password: Retype new password: passwd: all authentication tokens updated successfully. 11. Create persistant mappings with udev rules. 13

16 ~]# cd /etc/udev/rules.d rules.d]# vi 97-iscsi-dm-permissions.rules ENV{DM_NAME}=="VOL-2TB-3p1",OWNER:="grid",GROUP:="asmadmin",MODE:="660",SYMLINK+="iscsi/oraasm- $env{dm_name}" ENV{DM_NAME}=="VOL-2TB-4p1",OWNER:="grid",GROUP:="asmadmin",MODE:="660",SYMLINK+="iscsi/oraasm- $env{dm_name}" 12. Restart the multipath devices so that udev rules will be read, and iscsi devices mapping will take place accordingly. Newly mapped defices are listed in bold letters. ~]# service multipathd restart && ls -l /dev/dm-* brw-rw root brw-rw root brw-rw root brw-rw root brw-rw root brw-rw grid brw-rw grid disk 253, 0 Dec 24 15:14 /dev/dm-0 disk 253, 1 Dec 24 15:14 /dev/dm-1 disk 253, 2 Dec 24 15:14 /dev/dm-2 disk 253, 3 Dec 24 17:55 /dev/dm-3 disk 253, 4 Dec 24 17:55 /dev/dm-4 asmadmin 253, 5 Dec 24 17:55 /dev/dm-5 asmadmin 253, 6 Dec 24 17:55 /dev/dm Run the asmca command to create the ASM disk groups from the newly created multipath devices. Oracle grid infrastructure installation has the asmca command that helps in configuring the disk groups. The following four screen captures explain the user input and the process flow. 14

17 Figure 7: ASM disk group creation (step 1) Figure 8: ASM disk group creation (step 2) 15

18 Figure 9: ASM disk group creation (step 3) Figure 10: ASM disk group creation (step 4) 16

19 Configuring Windows host This section explains the configuration steps for the Windows 2008 operating system. As grid infrastructure is expected to span across multiple computers, when running a setup as a local administrator or as a user with local administrator s privilege for stand-alone grid infrastructure installation, Oracle setup issues a warning. To avoid this, create an administrative user at domain-level and add it to the local administrator s group using the Restricted Groups policy defined at domain controller. Policies created on the domain are pushed to the defined server every 3 hours, but they can also be pulled on the local system to apply them. For the Restricted Groups policy to work on the local system, System Preparation Tool (sysprep) must have been run after Windows setup. Sysprep generalizes the operating system image on the reference computer to remove machine-identifying data and enable the image to run on other bare metal machines. Oracle also suggests creating a local non-privileged user as database software owner and a domain privileged user as part of local administrator s group. Microsoft iscsi initiator is used to log on to the iscsi targets. The process is fairly straightforward. You can find more information on how to map iscsi targets at the following location: technet.microsoft.com/en-us/library/ee338476%28v=ws.10%29.aspx To use iscsi multipathing, MPIO needs to be installed on the system. MPIO is a feature and can be enabled from the Windows Server Manager, Features panel. Confirm that iscsi support is enabled for MPIO as shown in Figure 11. The disabled Add support for iscsi devices check box indicates that support is enabled for iscsi devices. Figure 11: MPIO support enabled for iscsi devices After the volumes are available, the Windows diskpart utility is used to configure them further. 17

20 Note: During initial configuration, MPIO for Windows was not enabled. Due to this, some disks are shown offline. C:\Users\Administrator.ISVPUNE>diskpart Microsoft DiskPart version Copyright (C) Microsoft Corporation. On computer: ISVHOST3 DISKPART> list disk Disk ### Status Size Free Dyn Gpt Disk 0 Online 837 GB 0 B * Disk 1 Online 4186 GB 0 B * Disk 2 Online 2048 GB 2047 GB <<< Good. MUST NOT BE GPT. Disk 3 Online 2048 GB 2047 GB <<< Good. MUST NOT BE GPT. Disk 4 Online 50 GB 49 GB <<< Good. MUST NOT BE GPT. Disk 5 Online 50 GB 49 GB <<< Good. MUST NOT BE GPT. Disk 6 Offline 2048 GB 1023 MB Disk 7 Offline 2048 GB 1023 MB Disk 8 Offline 50 GB 50 GB Disk 9 Offline 50 GB 50 GB As the next step, a specific disk is selected and two types of partitions (extended and logical) are created on it. DISKPART> select disk 2 Disk 2 is now the selected disk. DISKPART> list disk Disk ### Status Size Free Dyn Gpt Disk 0 Online 837 GB 0 B * Disk 1 Online 4186 GB 0 B * * Disk 2 Online 2048 GB 2047 GB Disk 3 Online 2048 GB 2047 GB Disk 4 Online 50 GB 49 GB Disk 5 Online 50 GB 49 GB Disk 6 Offline 2048 GB 1023 MB Disk 7 Offline 2048 GB 1023 MB Disk 8 Offline 50 GB 50 GB Disk 9 Offline 50 GB 50 GB * Displayed in front of Disk 2 indicates that it is the current disk. DISKPART> create part extended DiskPart succeeded in creating the specified partition. DISKPART> create partition logical size= DiskPart succeeded in creating the specified partition. When creating a logical partition, the size is provided in KB. In the test setup a logical partition is created with KB. 18

21 DISKPART> list part Partition ### Type Size Offset Partition 0 Extended 2047 GB 1024 KB * Partition 1 Logical 2047 GB 2048 KB For Disk 2, the setup is now complete. Same steps are repeated for Disk 3, Disk 4, and Disk 5 in order to create extended and logical partitions. After completing these steps, the mapped storage looks as shown in Figure 12. Figure 12: Completed disk configuration Before using the mapped disks to create the ASM disk groups, you need to stamp them. The stamping is done using asmtool.exe (a command-line utility) or asmtoolg.exe (a graphical utility). Both are found in GRID_HOME/bin. Note: asmtool.exe / asmtoolg.exe must be started with administrator rights. Failing to start the utility as an administrator user results in access denied error message. Figure 13 shows the disks stamped using asmtoolg. 19

22 Figure 13: Stamped ASM devices 20

23 Lab setup The lab environment under which this setup was tested had the following components. Storwize V7000 The following table lists the Storwize V7000 configuration used in the exercise. Storage name Ifs3 Node model Storwize V7000 Micro code level 7.2 Table 2: Storwize V7000 system details Linux host details The following table lists the configuration for Linux hosts used in the exercise. Server type IBM System x 3650 M4 servers Processor Intel Xeon processor Memory 238 GB Operating system RHEL 6.2 Table 3: Linux host details Windows host details The following table lists the configuration for Windows hosts used in the exercise. Server type Processor Memory Operating system IBM System x3650 M4 servers Intel Xeon processor 238 GB Windows 2008 R2 SP1 Table 4: Windows host details 21

24 Lab topology Error! Reference source not found. shows the lab setup for iscsi configuration. Figure 14: Lab setup for iscsi configuration As seen from Figure 14, the database (DB) host can be a Windows or a Linux system. The DB host connects to two types of networks. Public network is for the application or user communication. The DB host is further connected to Storwize V7000 through redundant switches over a 10Gb interface link. In this case, the DB host logs on to four targets with one session each. 22

25 Oracle 12c software options This section describes the Oracle database editions. Oracle Database is available in five editions. Oracle also offers several database options, packs, and other products that enhance the capabilities of Oracle database for specific purposes. Oracle Database Standard Edition One The Oracle Database Standard Edition One provides ease of use for workgroup, department-level, and web applications. It can be deployed from single-server environments for small business to highly distributed branch environments. Oracle Database Standard Edition Oracle Database Standard Edition provides all the features of Standard Edition One, along with support for larger systems and clustering of services with Oracle Real Application Clusters (Oracle RAC). Oracle Database Enterprise Edition Oracle Database Enterprise Edition is a high-end solution suitable for high-volume online transaction processing (OLTP) applications, query-intensive data warehouses, and demanding Internet applications. Oracle Database Enterprise Edition contains all of the components of an Oracle database, and can be further enhanced with the purchase of the options and packs. Oracle Database Express Edition Oracle Database Express Edition (Oracle Database XE) is an entry-level edition of Oracle Database that is quick to download, simple to install and manage, and is free to develop, deploy, and distribute. Oracle Database XE can be installed on a system of any size with any number of processors, stores up to 11 GB of user data, using up to 1 GB of memory, and using only one processor on the host computer. The current version of Oracle Database XE is Oracle Database 11g Release 2. Oracle Database Personal Edition Oracle Database Personal Edition supports single-user development and deployment environments that require full compatibility with Oracle Database Standard Edition One, Oracle Database Standard Edition, and Oracle Database Enterprise Edition. Personal Edition includes all of the components that are included with the Enterprise Edition, as well as all of the options that are available with the Enterprise Edition, with the exception of the Oracle RAC option, which cannot be used with the Personal Edition. Personal Edition is available on Windows and Linux platforms only. The management packs are not included in the Personal Edition. For more information on Oracle licensing, refer to: docs.oracle.com/cd/e16655_01/license.121/e17614/toc.htm 23

26 Container database creation Container database (CDB) is a new feature available with Oracle 12c. Each container database can have a single or multiple pluggable databases (PDBs). The multitenant architecture with one PDB (tenant) is available in all database versions described earlier. There is an extra cost option for up to 252 PDB per CDB. Figure 15 explains the multitenant architecture of Oracle 12c. Figure 15: Multitenant architecture The process to create a CDB or a non-cdb is exactly the same. Both can be created using database configuration assistant (DBCA) or by issuing the CREATE DATABASE statement on the SQL*Plus prompt. PDBs are created using the seed database template provided. The multitenant architecture imposes a single database sharing following components Background processes Shared / processes memory Oracle metadata Redo log files Control files 24

27 Pluggable database creation PDB can be also be created using DBCA or SQL*Plus prompt. An example of the creation of a pluggable database, PDB1, from the seed database is shown in this section. When using a file system, the FILE_NAME_CONVERT parameter must be provided to specify the location for the data files of PDB. SQL> CREATE PLUGGABLE DATABASE PDB1 ADMIN USER PDB1ADMIN IDENTIFIED BY PDB1ADMIN; Where: PDB1 is the name of the pluggable database. PDB1ADMIN is the local administrator for the pluggable database. Database backup and restore Both CDB and PDB can be backed up using Oracle Recovery Manager (RMAN). When CDB is backed up, RMAN automatically backs up root, all the PDB s, and the archived redo logs. Perform the following steps to back up root database with RMAN 1. Start rman [oracle@isvhost2 ~]$ rman Recovery Manager: Release Production on Fri Feb 21 12:42: Copyright (c) 1982, 2013, Oracle and/or its affiliates. All rights reserved. 2. Connect to root database as common user with SYSBACKUP or SYSDBA privilege RMAN> connect target / connected to target database: ORCL (DBID= ) 3. Start the backup of database RMAN> backup database root; The following example shows backing up one or more PDBs when connected to root database. 1. Start rman [oracle@isvhost2 ~]$ rman Recovery Manager: Release Production on Fri Feb 21 12:42: Copyright (c) 1982, 2013, Oracle and/or its affiliates. All rights reserved. 2. Connect to root as common user with SYSBACKUP or SYSDBA privilege 25

28 RMAN> connect target / connected to target database: ORCL (DBID= ) 3. Start the backup of database RMAN> backup pluggable database pdb1,pdb2; The following example shows backing up of PDB while connected to PDB 1. Start rman [oracle@isvhost2 ~]$ rman Recovery Manager: Release Production on Fri Feb 21 12:42: Copyright (c) 1982, 2013, Oracle and/or its affiliates. All rights reserved. 2. Connect to root as common user with SYSBACKUP or SYSDBA privilege RMAN> connect target sys/oracle@pdb1 connected to target database: ORCL (DBID= ) 3. Start the backup of database RMAN> backup database; Summary This paper describes the considerations for deploying Oracle database 12c (Linux/Windows) using the Storwize V7000 iscsi protocol. This paper provides guidelines for Ethernet network and OS-specific considerations. 26

29 Resources The following websites provide useful references to supplement the information contained in this paper: IBM Systems on PartnerWorld ibm.com/partnerworld/systems IBM Storwize V7000 Information Center iscsi networking best practices iscsi storage research Microsoft iscsi initiator information Microsoft MPIO information Oracle licensing information Oracle 12c database administrators guide Oracle 12c grid infrastructure information About the author Shashank Shingornikar is an IT specialist in the IBM Systems and Technology Group ISV Enablement Organization. He has more than 16 years of experience working with DBMS technologies, involved in high availability areas. You can reach Shashank at 27

30 Trademarks and special notices Copyright IBM Corporation References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the web at "Copyright and trademark information" at Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon 28

31 considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. 29