Oracle 11g RAC on Dell Compellent Storage Center. Dell Best Practices

Transcription

1 Oracle 11g RAC on Dell Compellent Storage Center Dell

2 Document Revision Table 1. Revision History Date Revision Description 1/16/2012 A Initial 4/19/2012 B Content and format change THIS TECHNICAL TIP IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Trademarks used in this text: Dell TM, the DELL TM logo, and Compellent TM are trademarks of Dell Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own. April 2012 Rev. B ii

3 Contents Introduction... 1 Customer Support... 1 Document Conventions... 1 Audience... 2 Scope... 2 Overview of Dell Compellent Storage Center... 3 Dynamic Block Architecture... 3 Storage Virtualization... 4 Dynamic Capacity (Thin Provisioning)... 4 Data Progression... 5 Fast Track... 5 Consistency Group... 6 Benefits of Oracle on Dell Compellent Storage Center... 7 Storage setup and configuration... 7 Disk drives & RAID recommendations... 7 RAID levels and Data Progression for Oracle databases... 8 Solid State Disks (SSD)... 8 SAS or Fibre Channel Disks... 8 Recommended disk configuration layouts ASM disk group ASM disk group devices Network setup Network configuration Database setup and configuration Direct I/O and Async I/O Conclusion Appendix A Linux / Clusterware / ASM / RAC setup... 1 Hardware:... 1 Servers:... 1 Storage:... 1 Storage Option chosen for file type... 1 iii

4 Disk Layout / Storage Centers... 2 Network:... 3 Software:... 4 OS:... 4 Linux packages:... 4 Oracle:... 4 Configuration steps:... 4 Create and present boot volumes:... 4 Installing OEL5U5:... 5 Create Oracle SCAN address and add to DNS... 6 Create and present data volumes:... 6 Prep Linux for Oracle pre-install... 7 Configure the network:... 7 Update Kernel with Oracle fixes and configuration:... 9 Download and install oracleasm: Configure Device Mapper: Create file systems: Create users and groups: Create directories for ORACLE_BASE and binaries: Add Oracle s Grid home to PATH in root s.bashrc: Generate authentication keys: Configure oracleasm: Disable OS network time protocol daemon: Upload Oracle Software and install grid package: Verify system readiness for Oracle clusterware: Install Oracle Grid: Create ASM Disk Groups: Install RDBMS Create the RAC database Appendix B: Document / Website References iv

5 Figures and Tables Table 1. Revision History... ii Table 2. Conventions... 1 Figure 1: Two Node Oracle RAC Configuration... 3 Figure 2: Dynamic Block Architecture... 3 Figure 3: Storage Virtualization... 4 Figure 4: Dynamic Capacity (Thin Provisioning)... 4 Figure 5: Data Instant Replay (DIR)... 5 Figure 6: Data Progression... 5 Table 3. Benefits of Oracle on Dell Compellent Storage Center... 7 Table 4. Disk Drives & RAID Recommendations... 7 Figure 7: Oracle with ASM Table 5. ASM Disk Group Layout Table 6. Network Setup v

6 Introduction This white paper describes general best practices for running Oracle Clusterware, ASM, and RAC Databases on Linux with Dell Compellent Storage Center. It does not cover architecture, design, implementation, performance tuning and best practices of Oracle clusterware, ASM, and RAC databases as those topics are covered in great detail in Oracle published documentation. Document and web site references are provided at the end of this document. Also note that the recommended practices in this whitepaper are not necessarily the best practices in all environments as your business requirements will dictate your configuration. Customer Support Dell Compellent provides live support EZSTORE ( ), 24 hours a day, 7 days a week, 365 days a year. For additional support, Compellent at support@compellent.com. Dell Compellent responds to s during normal business hours. Document Conventions Table 2. Conventions Item Menu items, dialog box titles, field names, keys Mouse click required User Input Value substitution required by User System response to commands Output omitted for brevity Website addresses addresses Convention Bold Click Monospace Font <value> Blue < snipped > name@dell.com 1

7 Notes are used to convey special information or instructions. Timesavers are tips specifically designed to save time or reduce the number of steps. Caution indicates the potential for risk including system or data damage. Warning indicates that failure to follow directions could result in bodily harm. Audience This paper is intended for database administrators (DBAs), system administrators and storage administrators that need to understand how to configure Oracle clustered databases on a Dell Compellent Storage Center. Readers should be familiar with Dell Compellent Storage Center and have a solid background, configuring and operating the following: Oracle architecture Oracle 11g Oracle Clusterware Oracle Real Application Clusters (RAC) Oracle s Automated Storage Management (ASM) General understanding of SAN technologies Scope This white paper explains how to merge Oracle Clusterware and a 2-node RAC cluster with Dell Compellent storage technology to meet the challenges of providing a solution that delivers protection against storage hardware failures, such as an HBA or disk, and delivers high performance I/O, while allowing the DBA to concentrate on other business critical database administration functions and services. A lab environment was used to setup an Oracle Clusterware and RDBMS configuration on Dell Compellent Storage. Instructions for configuring the lab environment are included in this document as a learning aid, and should not be used to configure a real-time production environment. 2

8 Figure 1: Two Node Oracle RAC Configuration Overview of Dell Compellent Storage Center Dynamic Block Architecture Dynamic Block Architecture records and tracks specific information, called metadata, about every block of data, and provides the system intelligence on how every block is being used. Metadata about blocks is gathered without system overhead and can be extensive, including time written, the type of disk drive used, the type of data stored, RAID level, and more. All of this metadata or data about the data enables Storage Center to take a more sophisticated and intelligent approach to storing, recovering and managing your data. For example, metadata that describes that specific blocks of data have been accessed the most can be used by Storage Center to move the blocks to a faster tier of storage. Metadata describing inactive blocks can be used by Storage Center to automatically place the blocks on lower cost, lower performance drives. Figure 2: Dynamic Block Architecture 3

9 Storage Virtualization Dell Compellent virtualizes enterprise storage at the disk level, creating a dynamic pool of storage resources shared by all servers. Because read/write operations are spread across all available drives, multiple requests are processed in parallel, boosting system performance. With Dell Compellent Storage Virtualization, users can create hundreds of volumes in seconds to support any server platform and optimize the placement of applications. Storage Center automatically restripes all data across all drives in the storage pool when adding disks, and can dynamically scale the storage pool without disruptions. Virtualization also allows Storage Center to change RAID levels on the fly. Figure 3: Storage Virtualization Dynamic Capacity (Thin Provisioning) With traditional storage systems, Administrators must purchase, allocate and manage capacity upfront, speculating where to place storage resources and creating large, underutilized volumes with long term growth built in. This practice leaves the majority of disk space allocated yet unused, and only available to specific applications. Dell Compellent s Thin Provisioning, called Dynamic Capacity, delivers the highest storage utilization possible by eliminating allocated but unused capacity. Dynamic Capacity completely separates storage allocation from Figure 4: Dynamic Capacity (Thin Provisioning) utilization, enabling users to create any size virtual volume upfront, yet only consume actual physical capacity when data is written by the application. Data Instant Replay (DIR) Data Instant Replay is most compared to "snapshot" technology and provides continuous space-efficient data protection using a feature called Replays. Replays create point-in-time copies of your volumes where further changes to a volume are journaled in a way that allows the volume to be rolled back to its original state when the replay was created. Replays can be mounted as volumes, called view volumes, for the sake of partial or full volume data restore. There are various other uses of replays, but for the sake of this paper we will focus on database backup/recovery. 4

10 Policy based schedules, with varying intervals and expiration, can be created to manage Replays and provide greater recovery capabilities to a previous know state, and there is no limit on the number or replays taken. Replays, when used in conjunction with another feature call Consistency Groups, provide data integrity of data spanning multiple volumes. Data Progression Fluid data storage or automated tiered storage is another feature, called Data Progression, offered by Dell Compellent. Data Progression automatically migrates enterprise data to the most optimal storage tier based on a set of predefined or custom policies or storage profiles. Over time, infrequently accessed data migrates to lower-cost drives (SATA), while the most active, mission-critical Figure 5: Data Instant Replay (DIR) Figure 6: Data Progression data migrates to high-performance (expensive FC/SAS) drives. You can change the number of days on a volume to alert Data Progression when to move the data. The default is 12 days. Vice versa, if the data has been accessed for a certain amount of cycles that data will then migrate back to higher tier disks for performance. This also can be changed based on your preference. Data progression eliminates the need to manually classify and migrate data to different storage tiers while reducing drive number and cost, and cooling and power costs. Fast Track Taking advantage of the outer-most tracks on a drive is referred to as short-stroking, and Fast Track provides benefits far beyond that. Short-stroking is sometimes incorrectly associated with placing data on faster tracks. So if short stroking is not related to faster tracks, what is it related to? There are two parts to the answer. Firstly, short-stroking by its name suggest that the disk arm only has to move across a smaller number of total tracks, which are the outer-most tracks, yielding lower seek times. Secondly, since there is more surface area of the outer-most tracks than inner tracks, the outer tracks can accommodate more 5

11 data. And since there s more data on the outer most tracks, latency times will be lower for retrieving data on those tracks. The lower seek and latency times are sometimes translated into faster tracks. Fast Track enhances Data Progression by dynamically placing data on the outer-most tracks of each drive, thereby reducing seek and latency times for that data within each storage tier. Fast Track also utilizes the full capacity of the drive as infrequently accessed data is placed on the inner zones of the drives. Unlike other traditional SAN technologies, Fast track also takes advantage of Thin Provisioning by only writing actual data to the outer-most tracks and not allocated, unused blocks. Combining both Thin Provisioning and Data Progression with Fast Track, wasted space is eliminated from the highest-performing tracks, increases in performance are realized by decreasing seek and latency times, and overall power consumption is decreased by reducing the number of high performance, power-hungry disks. Figure 7: Fast Track and Tiered Storage Consistency Group Dell Compellent s Consistency Group feature allows storage administrators to take a snapshot of an Oracle RAC database atomically. When creating a snapshot of a running Oracle RAC database, you must ensure that all storage volumes (LUNs) that make up your database be atomically snapped. Without a consistency group, the database must be configured with all control files in one volume and all redo log files in the same volume or another volume but cannot be spread across volumes whether file system or Oracle ASM was used. Consistency Groups gives you the ability to create a usable snapshot of a database with control files and redo log files spread across volumes which is to safeguard against single point of failure. Figure 8: Consistency Group and Oracle 6

12 Benefits of Oracle on Dell Compellent Storage Center Some of the benefits of deploying Oracle in a Dell Compellent SAN solution are listed in the below table. Table 3. Benefits of Oracle on Dell Compellent Storage Center Benefit Details Lower total cost of ownership (TCO) Greater manageability Simplified RAC Implementation High availability and scalability Compellent Information Life Cycle (ILM) benefits Reduces acquisition, administration, and maintenance costs Ease of use, implementation, provisioning, and management Provides shared storage (raw or filesystems) Clustering provides higher levels of data availability and combined processing power of multiple server for greater throughput and scalability Provides tiered storage, dynamic capacity, data progression, thin provisioning, instant replay (snapshot) and more Storage setup and configuration Disk drives & RAID recommendations Table 4. Disk Drives & RAID Recommendations Description RAID10 SSD RAID10/FC/SAS 15K rpm Data files OK (W) Recommended (W) Control files Online Redo Logs Archived Redo Logs Flashback Recovery Recommended (W) Recommended (W) Not Required Not Required RAID10/FC/SAS 10K rpm RAID5/FC/SAS 15K rpm RAID5/FC/SAS 10K rpm All RAID FC/SAS/SATA OK (W) DP DP DP OK (W) OK (W) Not Required Not Required Not Required OK (W) OK (W) Avoid Avoid Avoid Recommended (W) Recommended (W) OK (W) Not Required Not Required DP OK (W) Not Required Not Required DP 7

13 OCR files / Voting Disk Not Required Recommended (W) OK (W) Avoid Avoid Avoid Abbreviation: W Writes DP Data Progression If Fast Track is licensed, then it is enabled by default and will be utilized behind the scenes. No manual configuration is required. Drives with higher RPM provide higher overall random-access throughput and shorter response times than drives with lower RPM. Because of better performance, SAS or Fibre Channel drives with 15K rpm are always recommended for storing Oracle datafiles and online redo logs. Serial ATA and lower cost Fibre Channel drives have slower rotational speed and therefore recommended for Oracle archived redo logs and flashback recovery area with Data Progression. RAID levels and Data Progression for Oracle databases Dell Compellent highly recommends using Fast Track for datafile volumes and online redo log volumes for all production databases. For non-production systems, the use of RAID 10 is sufficient. Whether you are using 15k rpm drives or 10k rpm drives, the same rules apply to Data Progression and Data Instant Replay. Please refer to the table 4 below under the Data Progression and Data Instant Replay section for more information on how to configure the settings appropriately. Solid State Disks (SSD) Before implementing SSD on Dell Compellent Storage Center for Oracle databases, you must determine if the database in question warrants the high performance that SSD provides. Since the price per gigabyte of SSD is more expensive than SAS or Fibre Channel, you need to carefully evaluate your database performance. Dell Compellent recommends using SSD RAID10 for database online redo logs and frequently used indices if the database is highly transactional and is IO constrained. Data Warehouse databases should not use SSD for online redo logs as you will not have any performance gain unless the whole database resides on SSD which may be very costly depending on how large the database is. If using SSD for online redo logs, do not configure Data Progression on this volume as you will not gain any space back due to the nature of online redo logs. SAS or Fibre Channel Disks Disk types are determined by the performance requirement for a database. In a highly transactional database, Dell Compellent recommends using RAID10 with 15K rpm disks for 8

14 datafile volumes, online redo log volumes (if SSD is not available), and archived log volumes for all production databases. If SSD drives are available, place online redo logs and frequently used indices on SSD. For non-production systems, the use of RAID10 with 10K rpm disks is sufficient. Whether you are using 15K rpm disks or 10K rpm disks, the same rules apply to Data Progression and Data Instant Replay. Please refer to the table 4 below under the Data Progression and Data Instant Replay section for more information on how to configure the settings appropriately. 9

15 Recommended disk configuration layouts Figure 9: Oracle with Cooked File Systems 10

16 Figure 10: Oracle with SSD 11

17 Figure 7: Oracle with ASM If your Storage Center is on 4.x code, there is no consistency group available when using Data Instant Replay for snapshot. In order to take a snapshot of your database while it is running, you need to setup your Oracle database online redo log files and control files on the same volume. You cannot multiplex your online redo logs and control files across volumes. If your Storage Center is on 5.x code, then there is no restriction on multiplexing across volumes as long as the volumes are part of a consistency replay profile. Also note that you need to test to determine the number of LUNs required for your database for optimal configuration in terms of performance since every operating system is different. 12

18 ASM disk group Table 5. ASM Disk Group Layout ASM Disk Group Contents RAW Device Support ASMLib Suport +DATA Oracle Datafiles Yes Yes +FLASH Flashback Recovery Area Yes Yes ASM disk group devices Dell Compellent Storage Center supports RAW devices with or without ASMLib service. Oracle recommends that generally no more than two disk groups be maintained and managed per RAC cluster or single ASM instance. Select External Redundancy when creating ASM disk group. When creating LUNs for ASM disk group, make sure the LUNs are on the same disk characteristics (ex. FC 15K rpm or 10K rpm and 146GB or 300GB). LUNs should not contain mixed speed drives. When creating multiple LUNs for an ASM disk group, make sure to create the same size LUN to avoid imbalance. Create larger LUNs to reduce LUN management overhead. This will reduce LUN management and will allow future growth. Since the Compellent Storage Center provides thin provisioning, disk space only taken up when there is data written to it. Network setup Table 6. Network Setup VLAN ID or Separate Description CRS Setting Switches 1 or switch A Client Network Public 2 or switch B RAC Interconnect Private 13

19 Network configuration If possible, configure jumbo frames for private network. Note: when configuring jumbo frames, you need to configure on all legs of the RAC interconnect networks (i.e. servers, switches, etc.). If configuring jumbo frames is not possible, you need to configure the interconnect network with at least 1Gbps link. Database setup and configuration Direct I/O and Async I/O Oracle recommends using direct I/O and async I/O. The use of Direct I/O bypasses the filesystem buffer cache, hence reduces CPU overhead on your server. Direct I/O is very beneficial to Oracle s log writer, both in terms of throughput and latency. Async I/O is beneficial for datafiles I/O. When mounting filesystems for your Oracle database, make sure to understand how your Operating System supports Direct I/O. Some Operating Systems require certain mount options to enable Direct I/O or Async I/O. Conclusion With Dell Compellent SAN technologies combined with Oracle Real Application Clusters, has become a very attractive storage system for Oracle RAC Databases. Dell Compellent Storage Center provides the best availability, scalability, manageability, and performance for your database applications. 14

20 Appendix A Linux / Clusterware / ASM / RAC setup Oracle11gR2 Grid, ASM, RDBMS, and RAC Installation on Linux 5.5 The following steps were used to configure an 11gR2 Grid, ASM, RDBMS and RAC database on Oracle Enterprise Linux 5 update 5 x86-64, including oracle fixes and mpath, in a lab environment. Care should be used when using the information in this appendix as it does not represent the configuration of a production environment. It is recommended that Oracle s published documents be used during your Clusterware and RAC configuration to ensure proper hardware requirements, Oracle architecture, design, installation, configuration and best practices are perform appropriately for your business needs. A single 2-node RAC configuration was tested consisting of a two node cluster running Oracle Grid, ASM, and RDBMS. Hardware: Servers: Two HP ProLiant DL360 G5 servers (Bach (node1), Mozart (node2)) were used as the nodes of the RAC cluster. Memory: Node 1: MB 667 MHz DIMMs Node 2: MB 667 MHz DIMMs CPU: 2333 MHz, 4/4 cores; 4 maximum threads L1 Cache: 128 KB L2 Cache: 8192 KB Storage: Four Dell Compellent Storage Centers (sc) were used: sc6 sc10 sc11 sc14 Storage Option chosen for file type 1

21 Table 1: Storage Options chosen for lab environment Storage Option OCR & Voting Disk Files Oracle Clusterware binaries Oracle RAC binaries ASM Database Files RAC Database Files Oracle Recovery Files ASM Yes No No Yes Yes NA Local file system No Yes Yes No No NA Disk Layout / Storage Centers Storage Center Size LUN Map Purpose ASM ASM and GB ID to Data Fail Volumes Node Group Group ================================ === === ==== ======= ====== ===== sc Bac-OEL5U5-x64-boot-mp-RAC11gR Boot vol Moz-OEL5U5-x64-boot-mp-RAC11gR Boot vol sc Bac-OEL5U5-mp-orahome-RAC-sc Grid/Oracle Home Moz-OEL5U5-mp-orahome-RAC-sc Grid/Oracle Home Bac-Moz-OEL5U5-mp-ocr-vote-RAC-sc , 2 OCR & Voting disk +CRS sc Bac-Moz-OEL5U5-ocr-vote-sc , 2 OCR & Voting disk +CRS Bac-Moz-OEL5U5-data1-sc , 2 DATA Disk group +DG1 FG1 Bac-Moz-OEL5U5-data2-sc , 2 +DG1 FG1 Bac-Moz-OEL5U5-fra1-sc , 2 FRA Disk group +FRA1 FG1 Bac-Moz-OEL5U5-fra2-sc , 2 +FRA1 FG1 sc Bac-Moz-OEL5U5-ocr-vote-sc , 2 OCR & Voting disk +CRS 2

22 Storage File Volume ASM Volume Group Name ASM Component System Size Group Name Redundancy ===================== ====== ======= ====================== ========== OCR/Voting Disk ASM 5GB +CRS Normal Database Files ASM 100GB +DG1 Normal Fast Recovery Area ASM 100GB +FRA1 Normal Oracle Home/Grid Home ext3 200GB Boot ext3 50GB Network: Public interconnect: Private interconnect: NIC 1 on both servers connected to Ethernet switch. NIC 2 on both servers connected to each other. IP address requirements: Three IPs were used for Oracle s Single Client Access Name (SCAN) and resolved to the same hostname in DNS. SCAN name format: scan.<lab_domain>. Eg. # nslookup scan.techsol.beer.town techsol.beer.town Server: Address: #53 Name: scan.techsol.beer.town Address: Name: scan.techsol.beer.town Address: Name: scan.techsol.beer.town Address: Networking IPV6 disabled Each RAC node was configured with: 1 public IP address 1 private IP address 1 VIP IP address 3

23 Software: OS: OEL5U5 Linux with mpath. Linux packages: kernel-headers el5.x86_64.rpm kernel-devel el5.x86_64.rpm kernel el5.x86_64.rpm oracleasm-support el5.x86_64.rpm oracleasmlib el5.x86_64.rpm oracleasm el el5.x86_64.rpm libaio-devel i386.rpm libaio-devel x86_64.rpm sysstat el5.x86_64.rpm numactl-devel el5.i386.rpm numactl-devel el5.x86_64.rpm cvuqdisk rpm Oracle: Oracle Grid Infrastructure Oracle ASM Oracle RDBMS Configuration steps: Create and present boot volumes: In Storage Center create two boot volumes one for each node. Present each boot volume as LUN 0, using a single HBA and single path, to each a node. 4

24 Installing OEL5U5: Install Oracle Enterprise Linux 5, update 5 with the mpath option (see below: linux mpath) along with Oracle kernel fixes. o o o o o o o o o o o o o o o o o Insert OEL5U5 cdrom. Type linux mpath at the boot prompt. On the CD Found window appears, select Skip On the initial Enterprise Linux window, select Next On the language selection window, select Next On the keyboard selection window, select Next On the Upgrade or Install window, select Next On the partitioning window, select: Remove all partitions on selected drives and create default layout Next On the Warning window, select Yes On the Network Devices window, select: Edit Under section Enable IPv4 Support : o Select Manual configuration o Enter the IP Address o Enter the Prefix (Netmask) Uncheck section Enable IPv6 support Select OK Select manually and enter the hostname and its domain Enter the Gateway Enter the Primary DNS Enter the Secondary DNS Select Next On the Time Zone screen, select your region and Next Provide root password and select Next On the Software Application screen, select Customize Now and Next Select Applications, and uncheck Games and Entertainment Graphical Internet Graphics Office/Productivity Sound and Video Text-Based Internet Select Development, and check: Development Libraries Development Tools GNOME Software Development Java Development Legacy Software Development X Software Development Select Servers, and uncheck: Printing Support Select Base System Check System Tools Uncheck Dialup Networking Support Select Next 5

25 o o o On the Begin Installation window, select Next On the Congratulations window, select Reboot On the Welcome window, select Forward Select Yes, I agree to the License Agreement Select Forward Disable the firewall Select Forward Select Yes Set SELinux Setting to Disable Select Forward Select Yes On window kdump, select Forward Set the date and time appropriately in window Date and Time, and select Forward Create a user in window Create User and select Forward In window Sound Card, select Forward In window, Additional CDs, select Finish Select OK to reboot the system Create Oracle SCAN address and add to DNS Create and present data volumes: In Storage Center create and present the remaining volumes as defined above to node1 and node2. Make sure the volumes that comprise the +DATA and +FRA disk groups are defined in a consistent replay profile. 6

26 Prep Linux for Oracle pre-install IMPORTANT: All commands below need to be executed by root on both nodes unless otherwise noted. Configure the network: Set the server domainname at boot by adding the domainname to /etc/rc.local # echo "domainname techsol.beer.town >> /etc/rc.local Execute the following to set the current domainname: domainname techsol.beer.town Configure /etc/hosts on all nodes: 1) remove the qualified hostname, and hostname from the localhost.localdomain ip address, 2) add the private cluster ip addresses of all nodes, 3) add the public addresses and aliases of all RAC nodes, 4) add the VIPs of all RAC nodes # (echo "%s/`hostname`//g"; echo 'wq') ex -s /etc/hosts # (echo "%s/`hostname -s`//g"; echo 'wq') ex -s /etc/hosts # echo " # Private bach-pri.techsol.beer.town bach-pri mozart-pri.techsol.beer.town mozart-pri # Public bach.techsol.beer.town bach mozart.techsol.beer.town mozart # VIP bach-vip.techsol.beer.town bach-vip mozart-vip.techsol.beer.town mozart-vip" >> /etc/hosts Add the following to /etc/sysconfig/network on both nodes changing HOSTNAME values as appropriate, where [n] represents 1 for node 1, and 2 for node 2. Do not remove the. between the node name and fully qualified server domain. NETWORKING=yes 7

27 NETWORKING_IPV6=no HOSTNAME=<node[n]>.<fully qualified server domain> GATEWAY= Comment deprecated directives: BROADCAST and NETWORK in network script ifcfg-eth0 # cd /etc/sysconfig/network-scripts # (echo "/^BR/s/^BR/#BR/"; echo "/^NETW/s/^N/#N/"; echo 'wq') ex -s ifcfg-eth0 # cd Configure the private interconnect by making the following updates to network script ifcfg-eth1 on both nodes: Change directive BOOTPROTO from dhcp to static Change directive ONBOOT from NO to YES Comment directive DHCP_HOSTNAME Add directive NETMASK= # cd /etc/sysconfig/network-scripts # (echo "/^BOOT/s/dhcp/static/ > /^ONBOOT/s/no/yes/ > /^DHCP/s/^D/#D/ > wq ") ex -s ifcfg-eth1 # echo 'NETMASK= ' >> ifcfg-eth1 On node 1, add IPADDR= to ifcfg-eth1 # echo 'IPADDR= ' >> ifcfg-eth1 On node 2, execute the following # (echo "/^HOTPLUG/s/^H/#H/"; echo 'wq') ex -s ifcfg-eth1 # echo 'IPADDR= ' >> ifcfg-eth1 Reboot both servers to activate the new network configuration: # shutdown -r now 8

28 Update Kernel with Oracle fixes and configuration: Update the unmodified Linux kernel (kernel el5) which was installed by default above, with the bug fixes added by Oracle (kernel el5]. Copy the below RPMs from folder Server\oracle_updated located on CDROM OEL5U5 to both RAC nodes and install them. See for additional information kernel-headers el5.x86_64.rpm kernel-devel el5.x86_64.rpm kernel el5.x86_64.rpm # rpm -ihv kernel-headers el5.x86_64.rpm warning: kernel-headers el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:kernel-headers ########################################### [100%] # rpm -ihv kernel-devel el5.x86_64.rpm warning: kernel-devel el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:kernel-devel ########################################### [100%] # rpm -ihv kernel el5.x86_64.rpm warning: kernel el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:kernel ########################################### [100%] # Reboot both servers to take advantage of the Kernel enhancements provided by Oracle: # shutdown -r now Change the system configuration for Oracle. # echo 'kernel.shmall = > fs.file-max = > kernel.msgmni = 2878 > kernel.sem = > kernel.shmmni = 4096 > net.core.rmem_default = > net.core.rmem_default = > net.core.rmem_max = > net.core.wmem_default = > net.core.wmem_max = > fs.aio-max-nr = > net.ipv4.ip_local_port_range = > vm.min_free_kbytes = > SYSCTL_LOC="/sbin/sysctl" > SEMMNS="32000" 9

29 > INSTALL_USER="oracle" > SEMOPM="100" > SEMMSL="250" > SEMMNS="32000" > SEMOPM="100" > SEMMNI="128" > SHMMNI="4096" > FILE_MAX_KERNEL=" " > IP_LOCAL_PORT_RANGE=" " > RMEM_DEFAULT="262144" > RMEM_MAX=" " > WMEM_DEFAULT="262144" > WMEM_MAX=" " > AIO_MAX_NR=" "' >> /etc/sysctl.conf # Reboot both servers to take advantage of the new system configuration: # shutdown -r now Download and install oracleasm: The three following ASM RPMs were required (see Oracle documentation and OTN for more information): Note: oracleasm-support el5.x86_64.rpm is provided on CDROM OEL5U5. However the latest version oracleasm-support el5.x86_64.rpm was chosen. See release notes: and OTN for more information. RPM File Contents Source Location oracleasm-support el5.x86_64.rpm Tools Oracle OTN website * oracleasmlib el5.x86_64.rpm Library Oracle OTN website * oracleasm el el5.x86_64.rpm Driver CDROM OEL5U5 \server\oracle_updated * 10

30 Install the ASM RPMS # rpm -ihv oracleasm-support el5.x86_64.rpm warning: oracleasm-support el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:oracleasm-support ########################################### [100%] # rpm -ihv oracleasm el el5.x86_64.rpm warning: oracleasm el el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:oracleasm ########################################### [100%] # rpm -ihv oracleasmlib el5.x86_64.rpm warning: oracleasmlib el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:oracleasmlib ########################################### [100%] # Download and install the following additional Linux RPMs which are required for oracle. (see Oracle documentation for more information): libaio-devel i386.rpm libaio-devel x86_64.rpm sysstat el5.x86_64.rpm numactl-devel el5.i386.rpm numactl-devel el5.x86_64.rpm Install the additional Linux RPMs # rpm -ihv libaio-devel i386.rpm warning: libaio-devel i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:libaio-devel ########################################### [100%] # rpm -ihv libaio-devel x86_64.rpm warning: libaio-devel x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:libaio-devel ########################################### [100%] # rpm -ihv sysstat el5.x86_64.rpm warning: sysstat el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:sysstat ########################################### [100%] # rpm -ihv numactl-devel el5.i386.rpm 11

31 warning: numactl-devel el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:numactl-devel ########################################### [100%] # rpm -ihv numactl-devel el5.x86_64.rpm warning: numactl-devel el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:numactl-devel ########################################### [100%] Update security limits for grid and oracle: # echo "grid soft nofile > grid hard nofile > grid soft nproc > grid hard nproc > oracle soft nofile > oracle hard nofile > oracle soft nproc > oracle hard nproc > oracle soft core unlimited > oracle soft core unlimited > oracle hard core unlimited > oracle soft memlock > oracle hard memlock " >> /etc/security/limits.conf # Update /etc/profile for users grid and oracle. Add the following to the bottom of /etc/profile. if [ $USER = "oracle" ] [ $USER = "grid" ] ; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p ulimit -n else ulimit -u n fi fi Configure Device Mapper: Disable the blacklist of wwids in multipath by commenting the wwid entry. # (echo "/wwid/s/^/#/ > wq") ex -s /etc/multipath.conf 12

32 # Execute the next five steps to add the storage devices to /etc/multipath.conf using user-friendly names: 1) Display the WWID of all multipath devices by reloading multipath and executing multipath l. Make note of the highlighted parts of all WWIDs # /etc/init.d/multipathd reload Reloading multipathd: [ OK ] # multipath -l < snipped > mpath1 (36000d e3) dm-3 COMPELNT,Compellent Vol [size=500g][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 0:0:6:5 sdb 8:16 [active][undef] \_ 1:0:9:5 sdi 8:128 [active][undef] < snipped > # 2) Identify the Serial Number of all volumes in Storage Center presented to all RAC nodes. 3) Match corresponding entries between the highlighted multipath WWIDs and Storage Center Serial Numbers. 4) Create the multipath user-friendly name (alias) based off of the Storage Center volume name. 5) Add the multipath directive to multipath.conf and define the mpath devices. Eg: multipaths { multipath { wwid "36000d d " alias orabin } 13

33 } multipath { wwid "36000d d " alias OCRSC6 } multipath { wwid "36000d e5" alias OCRSC11 } multipath { wwid "36000d bf e" alias OCRSC14 } multipath { wwid "36000d e3" alias DATASC11 } multipath { wwid "36000d e4" alias FRASC11 } Reload multipath so aliases are used. # /etc/init.d/multipathd reload Reloading multipathd: [ OK ] # Create file systems: Create an ext3 fs for the Oracle Grid and RDBMS binaries and label the volumes. Use the dm alias when creating the volume. Labels will not be used to mount the volumes because they are dm devices. # mkfs.ext3 /dev/mapper/orabin mke2fs 1.39 (29-May-2006) < snipped > # Create mount point /u01 for volume orabin to hold the Oracle binaries and add an entry to fstab, then mount the device. Use the device mapper name of the volume in fstab. # mkdir /u01 14

34 # echo "/dev/mapper/orabin /u01 ext3 defaults 0 0" >> /etc/fstab # mount -a Create users and groups: # groupadd -g 1000 oinstall # groupadd -g 1031 dba # useradd -u g oinstall -G dba oracle # useradd -u g oinstall -G dba grid # passwd oracle < snipped > # passwd grid < snipped > # Create directories for ORACLE_BASE and binaries: # mkdir -p /u01/sw_grid # chown grid:oinstall /u01/sw_grid # mkdir -p /u01/sw_oracle # chown oracle:oinstall /u01/sw_oracle # mkdir -p /u01/app/ # mkdir -p /u01/app/grid # mkdir -p /u01/app/oracle # chown -R grid:oinstall /u01 # chown -R oracle:oinstall /u01/app/oracle # chmod -R 775 /u01/ Add Oracle s Grid home to PATH in root s.bashrc: # echo 'PATH=$PATH:/u01/app/11.2.0/grid/bin' >> ~/.bashrc Generate authentication keys: As each user (root, oracle, and grid), generate authentication keys by logging in as each user and execute the 7 steps below: 1. Generate the key 15

35 # su - <user> # ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/<user>/.ssh/id_rsa):<enter> Enter passphrase (empty for no passphrase): <Enter> Enter same passphrase again: <Enter> < snipped > 2. On node 1 execute the following to share <user> s key with node2. For root, use target directory /root/.ssh. For other users, use target directory /home/<user>/.ssh: # cd.ssh # scp id_rsa.pub <node2>:/[root home/<user>]/.ssh/<node1>.pub < snipped > id_rsa.pub 100% KB/s 00:00 # 3. On node 2 execute the following to share root s key with node1. For root, use target directory /root/.ssh. For other users, use target directory /home/<user>/.ssh: # cd.ssh # scp id_rsa.pub <node1>:/[root home/<user>]/.ssh/<node2>.pub < snipped > id_rsa.pub 100% KB/s 00:00 # 4. Create the authorized key file on node 1 by executing the following: # cat id_rsa.pub > authorized_keys # cat <node2>.pub >> authorized_keys 5. Create the authorized key file on node 2 by executing the following: # cat id_rsa.pub > authorized_keys # cat <node1>.pub >> authorized_keys 6. Verify that ssh works on both nodes by executing the following on both nodes. Respond with yes when prompted with Are you sure you want to continue connecting (yes/no)? 16

36 # ssh <node2> date Wed Jun 6 10:32:13 CDT 2012 # ssh <node2>.<qualified server domain> date < snipped > Wed Jun 6 10:32:19 CDT 2012 # ssh <node2>-pri date < snipped > Wed Jun 6 10:32:26 CDT 2012 # ssh <node2>-pri.<qualified server domain> date < snipped > Wed Jun 6 10:32:33 CDT 2012 # ssh <node1> date Wed Jun 6 10:32:42 CDT 2012 # ssh <node1>.<qualified server domain> date < snipped > Wed Jun 6 10:32:48 CDT 2012 # ssh <node1>-pri date < snipped > Wed Jun 6 10:32:54 CDT 2012 # ssh <node1>-pri. <qualified server domain> date < snipped > Wed Jun 6 10:33:00 CDT 2012 # 7. Logout of the user oracle and grid. # exit Configure oracleasm: Configure oracleasm on both nodes and scan for ASM disks. # /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: grid Default group to own the driver interface []: oinstall 17

37 Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ] # echo '# ORACLEASM_SCANORDER: Matching patterns to order disk scanning > ORACLEASM_SCANORDER="dm" > > # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan > ORACLEASM_SCANEXCLUDE="sd"' >> /etc/sysconfig/oracleasm # On node1, get a list of the devices to be used as ASM disks and create the ASM disks. # ls -1 /dev/mapper control DATASC11 FRASC11 OCRSC11 OCRSC14 OCRSC6 orabin VolGroup00-LogVol00 VolGroup00-LogVol01 # /etc/init.d/oracleasm createdisk DATASC11 /dev/mapper/datasc11 Marking disk "DATASC11" as an ASM disk: [ OK ] # /etc/init.d/oracleasm createdisk FRASC11 /dev/mapper/frasc11 Marking disk "FRASC11" as an ASM disk: [ OK ] # /etc/init.d/oracleasm createdisk OCRSC11 /dev/mapper/ocrsc11 Marking disk "OCRSC11" as an ASM disk: [ OK ] # /etc/init.d/oracleasm createdisk OCRSC14 /dev/mapper/ocrsc14 Marking disk "OCRSC14" as an ASM disk: [ OK ] # /etc/init.d/oracleasm createdisk OCRSC6 /dev/mapper/ocrsc6 Marking disk "OCRSC6" as an ASM disk: [ OK ] # On node2, scan for the newly created ASM disks. # /etc/init.d/oracleasm scandisks Scanning the system for Oracle ASMLib disks: [ OK ] # 18

38 Disable OS network time protocol daemon: # mv /etc/ntp.conf /etc/ntp.conf.orig Upload Oracle Software and install grid package: Upload Oracle s Grid Clusterware and RDBMS software to node 1 only and place them in the directories created above (/u01/sw_grid, and /u01/sw_oracle). # ls -ltr /u01/sw_oracle total rw-r--r-- 1 root root Feb 22 11:53 p _112030_linux-x86-64_2of7.zip -rw-r--r-- 1 root root Feb 22 11:57 p _112030_linux-x86-64_1of7.zip # ls -ltr /u01/sw_grid total rw-r--r-- 1 root root Feb 22 11:53 p _112030_linux-x86-64_3of7.zip # On node 1 as user grid, extract the contents of the clusterware software file. # su - grid $ cd /u01/sw_grid/ $ unzip p _112030_linux-x86-64_3of7.zip Archive: p _112030_linux-x86-64_3of7.zip creating: grid/ creating: grid/doc/ < snipped > inflating: grid/stage/properties/oracle.crs_complete.properties inflating: grid/stage/properties/userpaths.properties $ exit logout # On node 1 as user oracle, extract the contents of the RDBMS software files. # su - oracle $ cd /u01/sw_oracle/ $ unzip p _112030_linux-x86-64_1of7.zip Archive: p _112030_linux-x86-64_1of7.zip creating: database/ creating: database/install/ < snipped > inflating: database/welcome.html 19

39 inflating: database/readme.html $ unzip p _112030_linux-x86-64_2of7.zip Archive: p _112030_linux-x86-64_2of7.zip creating: database/stage/components/oracle.ctx/ creating: database/stage/components/oracle.ctx/ / < snipped > inflating: database/stage/components/oracle.sysman.console.db/ /1/datafiles/filegroup13.jar inflating: database/stage/components/oracle.sysman.console.db/ /1/datafiles/filegroup2.jar $ exit logout # As root on node1, execute the following to install the Grid package that came with the install media. # cd /u01/sw_grid/grid/rpm # rpm -ihv cvuqdisk rpm Preparing... ########################################### [100%] Using default group oinstall to install package 1:cvuqdisk ########################################### [100%] # On node2, execute the following to install the Grid package that came with the install media. # mkdir -p /u01/sw_grid/grid/rpm # chown -R grid:oinstall /u01/sw_grid # chmod 777 /u01/sw_grid/grid # chmod 777 /u01/sw_grid/grid/rpm # cd /u01/sw_grid/grid/rpm # scp <node1>:/u01/sw_grid/grid/rpm/cvuqdisk rpm. cvuqdisk rpm 100% KB/s 00:00 # chmod 755 cvuqdisk rpm # rpm -ihv cvuqdisk rpm Preparing... ########################################### [100%] Using default group oinstall to install package 1:cvuqdisk ########################################### [100%] # cd ~ As user grid, add Oracle s Grid home to PATH in.bashrc and add umask to.bash_profile # su - grid $ echo 'PATH=$PATH:/u01/app/11.2.0/grid/bin' >> ~/.bashrc $ echo 'umask 022' >> ~/.bash_profile 20

40 $ exit Verify system readiness for Oracle clusterware: As user grid on node 1, verify the server is configured for the clusterware installation. Resolve any verification check that returns a value other than passed # su - grid $ cd /u01/sw_grid/grid $./runcluvfy.sh stage -pre crsinst -fixup -n <node1>,<node2> -r 11gR2 -verbose Performing pre-checks for cluster services setup Checking node reachability... < snipped > File "/etc/resolv.conf" is consistent across nodes Check: Time zone consistency Result: Time zone consistency check passed Pre-check for cluster services setup was successful. [grid@bach grid]$ $ exit Install Oracle Grid: Start an xterm session and execute the following to install the Grid Infrastructure. # su grid # cd /u01/sw_grid/grid #./runinstaller NOTE: select all default values unless otherwise instructed. Oracle GUI changes terminal step number in installation windows as necessary Oracle Grid Infrastructure Setting up Grid Infrastructure Step 1 of 10 Download Software Updates Select 'Skip software updates' Select 'Next >' 21

41 Oracle Grid Infrastructure Setting up Grid Infrastructure Step 2 of 9 Select Installation Option Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 3 of 9 Select Installation Type Select 'Advanced Installation' Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 4 of 10 Select Product Languages Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 5 of 16 Grid Plug and Play Information Change Cluster Name and SCAN Name to your chosen names. Append the server domain to your SCAN Name value. Remove the GNS values and uncheck the Configure GNS check box. Example: Cluster Name: rac-cluster SCAN Name: rac-scan.techsol.beer.town Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 6 of 16 Cluster Node Information Select Add Add Cluster Node Information Set 'Public Hostname:' to the qualified hostname and domain of node 2 Set Virtual Hostname: to the qualified vip and domain of node 2 22

42 Example: Select 'OK' Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 7 of 16 Specify Network Interface Usage Select Next> Oracle Grid Infrastructure Setting up Grid Infrastructure Step 8 of 16 Storage Option Information Select Next> Oracle Grid Infrastructure Setting up Grid Infrastructure Step 9 of 16 Create ASM Disk Group Change 'Disk Group Name' from DATA to OCRDG Select button 'Change Discovery Path' Change Disk Discovery Path Change Disk Discovery Path from '/dev/raw/*' to '/dev/oracleasm/disks/*' Select OK 23

43 Select the appropriate OCR disks Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 10 of 16 Specify ASM Password Select radio button ' Use same passwords for these accounts' Entered 'oracle' for the password Select 'Next >' Oracle Grid Infrastructure Select Yes Oracle Grid Infrastructure Setting up Grid Infrastructure Step 11 of 17 Failure Isolation Support Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 12 of 17 Privileged Operating System Groups 24

44 Select 'Next >' Oracle Grid Infrastructure Select Yes Oracle Grid Infrastructure Setting up Grid Infrastructure Step 13 of 17 Specify Installation Location Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 14 of 17 Create Inventory Select 'Next >' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 15 of 18 Perform Prerequisite Checks Select 'Next >' After the prerequisite check is performed, select hyper-link (more details) Details If the error shown is: And if error codes PRVF-5150 and PRVF-5184 are shown, verify that Linux user grid can read/write to one of the files listed. If it can, ignore the warning message as it is a known issue. Select Close 25

45 Select check-box 'Ignore All' Select 'Next >' Oracle Grid Infrastructure Select 'Yes' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 16 of 18 Summary Select 'Install' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 17 of 18 Install Product Execute Configuration scripts Login to node 1 as root and execute the requested scripts: # /u01/app/orainventory/orainstroot.sh 26

46 Changing permissions of /u01/app/orainventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/orainventory to oinstall. The execution of the script is complete. # # /u01/app/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]:<enter> Copying dbhome to /usr/local/bin... Copying oraenv to /usr/local/bin... Copying coraenv to /usr/local/bin... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation OLR initialization - successful root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding Clusterware entries to inittab CRS-2672: Attempting to start 'ora.mdnsd' on 'bach' CRS-2676: Start of 'ora.mdnsd' on 'bach' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'bach' CRS-2676: Start of 'ora.gpnpd' on 'bach' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'bach' CRS-2672: Attempting to start 'ora.gipcd' on 'bach' 27

47 CRS-2676: Start of 'ora.cssdmonitor' on 'bach' succeeded CRS-2676: Start of 'ora.gipcd' on 'bach' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'bach' CRS-2672: Attempting to start 'ora.diskmon' on 'bach' CRS-2676: Start of 'ora.diskmon' on 'bach' succeeded CRS-2676: Start of 'ora.cssd' on 'bach' succeeded ASM created and started successfully. Disk Group OCRDG created successfully. clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-4256: Updating the profile Successful addition of voting disk a44fc6bff162fbf305c6f6. Successful addition of voting disk 2b318eed83e44fb7bfcc86cc334e979e. Successful addition of voting disk abf4fbfbf514813d2b607fd. Successfully replaced voting disk group with +OCRDG. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group ONLINE a44fc6bff162fbf305c6f6 (/dev/oracleasm/disks/ocrsc6) [OCRDG] 2. ONLINE 2b318eed83e44fb7bfcc86cc334e979e (/dev/oracleasm/disks/ocrsc11) [OCRDG] 3. ONLINE abf4fbfbf514813d2b607fd (/dev/oracleasm/disks/ocrsc14) [OCRDG] Located 3 voting disk(s). CRS-2672: Attempting to start 'ora.asm' on 'bach' CRS-2676: Start of 'ora.asm' on 'bach' succeeded CRS-2672: Attempting to start 'ora.ocrdg.dg' on 'bach' CRS-2676: Start of 'ora.ocrdg.dg' on 'bach' succeeded CRS-2672: Attempting to start 'ora.registry.acfs' on 'bach' CRS-2676: Start of 'ora.registry.acfs' on 'bach' succeeded Configure Oracle Grid Infrastructure for a Cluster... succeeded # Login to node 2 as root and execute the requested scripts. # /u01/app/orainventory/orainstroot.sh Changing permissions of /u01/app/orainventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/orainventory to oinstall. The execution of the script is complete. # /u01/app/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: <Enter> Creating /usr/local/bin/ directory... Copying dbhome to /usr/local/bin... 28

48 Copying oraenv to /usr/local/bin... Copying coraenv to /usr/local/bin... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation OLR initialization - successful Adding Clusterware entries to inittab CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node bach, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster Configure Oracle Grid Infrastructure for a Cluster... succeeded # Return to the installer window Select 'Ok' Oracle Grid Infrastructure Setting up Grid Infrastructure Step 18 of 18 Finish Select 'Close' Create ASM Disk Groups: Start an xterm session and execute the following to create the ASM Disk Groups. In the xterm window, as user grid from node 1, perform the following to create the ASM diskgroups. # su grid # cd /u01/app/11.2.0/grid/bin #./asmca ASM Configuration Assistant Configure ASM Disk Groups Select tab Disk Groups Select button Create' Crete Disk Group 29

49 Set Disk Group Name to DATADG Select radio-button External (None) Select the check-box for the disk that will participate in the DATADG diskgroup Select button 'Show Advanced Options' Set 'Database Compatibility' to Select OK Disk Group Creation Select OK ASM Configuration Assistant Configure ASM Disk Groups Select button Create' Crete Disk Group Set Disk Group Name to FRADG Select radio-button External (None) Select the check-box for the disk that will participate in the FRADG diskgroup Select Show Advanced Options Enter for Database Compatibility Select OK 30

50 Disk Group Creation Select OK ASM Configuration Assistant Configure ASM Disk Groups Select button Mount All' Confirm Mount All Select Yes Disk Group: Mount All Select button Exit ASM Configuration Assistant Select Yes Install RDBMS In the xterm window connected to node 1, logout from user grid and login to user oracle and install RDBMS. $ exit # su oracle $ cd /u01/sw_oracle/database $./runinstaller Oracle Database 11g Release 2 Installer Installing database Step 1 of 11 Configure Security Updates Uncheck check-box: I wish to receive security updates via My Oracle Support Select Next > My Oracle Support Username/ Address Not Specified Select Yes 31

51 Oracle Database 11g Release 2 Installer Installing database Step 2 of 11 Download Software Updates Select Skip software updates Select Next > Oracle Database 11g Release 2 Installer Installing database Step 3 of 11 Select Installation Option Select Install database software only Select Next > Oracle Database 11g Release 2 Installer Installing database Step 4 of 11 Grid Installation Options Select Next > Oracle Database 11g Release 2 Installer Installing database Step 5 of 12 Select Product Languages Select Next > Oracle Database 11g Release 2 Installer Installing database Step 6 of 12 Select Database Edition Select Next > Oracle Database 11g Release 2 Installer Installing database Step 7 of 12 Specify Installation Location Oracle Base: Software Location: Select Next > /u01/app/oracle /u01/app/oracle/product/11.2.0/dbhome_1 Oracle Database 11g Release 2 Installer Installing database Step 8 of 12 32

52 Privileged Operating System Groups Select Next > Oracle Database 11g Release 2 Installer Installing database Step 9 of 12 Perform Prerequisite Checks Oracle Database 11g Release 2 Installer Installing database Step 10 of 12 Summary Select Install Oracle Database 11g Release 2 Installer Installing database Step 11 of 12 Install Product Execute Configuration scripts Login to node 1 as root and execute the requested scripts: # /u01/app/oracle/product/11.2.0/dbhome_1/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1 Enter the full pathname of the local bin directory: [/usr/local/bin]:<enter> The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. 33

53 Now product-specific root actions will be performed. Finished product-specific root actions. # Login to node 2 as root and execute the requested scripts: # /u01/app/oracle/product/11.2.0/dbhome_1/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1 Enter the full pathname of the local bin directory: [/usr/local/bin]:<enter> The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Finished product-specific root actions. # Return to the installer window Select OK Oracle Database 11g Release 2 Installer Installing database Step 12 of 12 Finish Select Close Return to the xterm window and press Enter 34