Author A.Kishore Introduction Oracle RAC Software Oracle VirtualBox Software... 4

Transcription

1 CONTENTS Introduction... 2 Oracle RAC Software... 3 Oracle VirtualBox Software... 4 Oracle Enterprise Linux 6.7 software... 4 Oracle VirtualBox Installation... 8 Virtual Machine Setup... 8 Guest Operating System Installation Prepare Steps Edit the hosts file of rac Install Guest Additions Configure SCAN DNS for RAC 11G RAC using dnsmasq in OEL Disable Firewall Disable secure linux Install oracle-rdbms-server-11gr2-preinstall using yum Deconfigure Network Time Protocol Install and Configure Automatic Storage Management and Disks Verify ASMLib Packages Configuring and Loading the ASMLib Packages Backup VIRTUAL MACHINE TO add shared disks click on storage Clone the Virtual Machine Configure the Shared Disks Create ASM disks Take Snapshot node1 and nodes Grid Installation NFS sharing of the software Install the missing package on node1 and node Setup SSH Verify using Cluvfy utility KICK OFF THE INSTALLATION Post-Install Actions

2 Install Database References Problems & Solutions INTRODUCTION Here is my setup, I have 16 GB Desktop with Linux installed and with 250 SSD on which the installation was done. I have created 2 VMs using Oracle Virtual Machine Linux Desktop 16 GB RAM / 250 SSD RAC1 - Linux GB RAM RAC2 - Linux GB RAM If you want to get through all steps of the Oracle RAC installation and your laptop or desktop computer has 8 GB or more of RAM, then this is entirely feasible using Oracle VirtualBox as demonstrated in this article. You can get a running RAC system which can host a small test database. The created system is not, and should not be considered, a production-ready system. It's simply to allow you to get used to installing and using RAC and test various administration procedures. The article also explains how to save the images and restore RAC from the images in a matter of minutes. Even if you break your test system, it will be easy to restore. We have used dnsmasq for scan configuration, we can now have 3 scan listeners on our 2 node RAC. I had to refer multiple docs to make it work and appreciate all the people who have contributed. This article uses the 64-bit versions of Oracle Linux, version 6.7, and Oracle 11g Release 2, version Using VirtualBox you can run multiple Virtual Machines (VMs) on a single server, allowing you to run both RAC nodes on a single machine. In addition, it allows you to set up shared virtual disks. The finished system includes two guest operating systems, two sets of Oracle Grid Infrastructure (Clusterware + ASM) and two Database instances all on a single server.

3 This article was originally inspired by the article "Oracle Database 11g Release 2 RAC On Linux Using VirtualBox" written by Sergei Romanenko and published in his blog. Then it was almost entirely revised and reworked, now this article seems to have very little resemblance with the original work. Note. When this article was written, Oracle Database 11g Release 2 ( ) for Linux 64-bit (both clusterware and database) was available through the Oracle support to licensed customers only. As of now, can be downloaded in Oracle support site, in "Patches & Updates", then select "Latest Patchsets", then select "Oracle Database", then select "Linux x86-64", then select " ". The number of this patch set is , it is possible to jump to the download page using this number. This patch set is a full installation of the Oracle Database software. This means that you do not need to install Oracle Database 11g Release 2 ( ) before installing Oracle Database 11g Release 2 ( ). For installing RAC database you will need only 3 files: ORACLE RAC SOFTWARE I have used which comes as a patch , if you are not having support userid you can download from otn.oracle.com software Oracle Database (includes Oracle Database and Oracle RAC), part 1: p _112030_linux-x86-64_1of7.zip 1.3G Oracle Database (includes Oracle Database and Oracle RAC), part 2: p _112030_linux-x86-64_2of7.zip 1.1G Oracle Grid Infrastructure (includes Oracle ASM, Oracle Clusterware): p _112030_linux-x86-64_3of7.zip 933M

4 Software from OTN Oracle 11g client on Windows for testing TAF (transaction application failover) ORACLE VIRTUALBOX SOFTWARE On Linux host VirtualBox _102096_el5-1.x86_64.rpm ORACLE ENTERPRISE LINUX 6.7 SOFTWARE Click Sign In/Register

5 Enter Username & Password

6 Accept Terms & Restrictions, Click Continue Select Oracle Linux from Product Pack & select x86 (64 bit) from Platform, Click Go

7 Select Oracle Linux Release 6 Update 7 Media Pack for x86 (64 bit), Click Continue. V iso Note:, you need to burn them using the image option available with your cd-writer software Refer:

8 ORACLE VIRTUALBOX INSTALLATION rpm -Uvh VirtualBox _102096_el5-1.x86_64.rpm Create below folders on the HOST machine cd /d05/oracle/rac mkdir rac1 rac2 asm VIRTUAL MACHINE SETUP Go to System Tools Oracle VM VirtualBox

9 Now we must define the two virtual RAC nodes. We can save time by defining one VM, then cloning it when it is installed. Start VirtualBox and click the "New" button on the toolbar. Click the "Next" button on the first page of the Virtual Machine Wizard.

10 Enter the name "rac1", OS "Linux" and Version "Oracle (64 bit)", then click the "Next" button.

11 Set Base Memory to 3 GB or more, then click the Next button

12 Accept the default option to create a new virtual hard disk by clicking the Create button

13 Select the default VMDK type and click the Next button on the Create Virtual Hard Drive screen Select Dynamically Allocated

14

15 Provide the location where you want to save the VM files. In my case I have provided path as below. Also increase the size of your vm from default to say 30GB and Click Create

16 The "rac1" VM will appear on the left hand pane. Scroll down the "Details" tab on the right

17 Click on General->Advanced and change shared clipboard to Bidirectional and Change Drag n Drop to Bidirectional and click Ok

18 Click on Network

19 Add two Adaptors. Make sure Adapter 1 is enabled, attached to Bridged Adapter. This interface will be used for public network, for example, for connection to the RAC database from other applications.

20 Then click on the Adapter 2 tab. Make sure Adapter 2 is enabled and attach to Internal Network. Name this network privnet :

21 Click on storage to load OEL6.7 ISO

22

23 GUEST OPERATING SYSTEM INSTALLATION Please note that during installation Virtual Box will be keeping the mouse pointer inside VM area. To exit, press Right Control key on the keyboard. Click on Start as shown below

24 After starting we get below screen and select Install or Upgrade an Existing System Select skip

25 Scroll Down and click NEXT

26 Select basic Storage and click Next Select Discard any Data

27 Change the hostname and Scroll down and click on Configure Network

28 Select System eth0 and click on edit Select Connect Automatically and select IPV4 Setting

29

30 Make sure that Connect automatically is checked. Select IPv4 Settings tab; change Method to DHCP ; Press Add and fill Address: ; Netmask: ; Gateway: Press Apply then done: In the Network Connections screen select System eth1 interface, this will be used for private network, then press Edit. Then check the box Connect automatically. Select IPv4 Settings tab; change Method to Manual. Press Add and fill Address: ; Netmask: ; Gateway: When done, press Apply :

31

32

33 Finally select System eth2 interface, this will be used for Internet, then press Edit. Check the box Connect automatically. Select IPv4 Settings tab make sure the Method is set to Automatic (DHCP).Press Apply button: Click Next

34 Give password for root Select Replace Existing Linux

35 Select Write Changes to disk Select Database Server option and select Customize now and Click Next

36

37

38

39 Select Tiger VNC

40 Click all the options under Desktops and Click Next

41 Click Yes Reboot the Guest Machine

42 Click Forward

43

44

45

46 PREPARE STEPS First of all check whether we can connect to the internet or not ping PING ( ) 56(84) bytes of data. 64 bytes from sb-in-f103.1e100.net ( ): icmp_seq=1 ttl=46 time=63.4 ms 64 bytes from sb-in-f103.1e100.net ( ): icmp_seq=2 ttl=46 time=63.5 ms 64 bytes from sb-in-f103.1e100.net ( ): icmp_seq=3 ttl=46 time=62.2 ms EDIT THE HOSTS FILE OF RAC1. In this chapter we are going to prepare RAC1 and later clone to RAC2 Follow the below steps on rac1 vi /etc/hosts localhost.localdomain localhost # Public Network - (eth0) rac1.appsdba.info rac rac2.appsdba.info rac2 # Private Interconnect - (eth1) rac1-priv.appsdba.info rac1-priv rac2-priv.appsdba.info rac2-priv # Public Virtual IP (VIP) addresses for - (eth0) rac1-vip.appsdba.info rac1-vip rac2-vip.appsdba.info rac2-vip

47 INSTALL GUEST ADDITIONS Click Ok

48 Click on RUN

49 If It fails. We need to update packages Solution: To overcome this error update gcc using yum update gcc yum update - It will take 20 minutes yum install kernel-uek-devel Note: Reboot the Server once above steps are completed

50 Once again try to install VM tools Author A.Kishore Purpose we will be able to view the shared folders in Linux Guest

51 CONFIGURE SCAN DNS FOR RAC 11G RAC USING DNSMASQ IN OEL 6 Install and configure dnsmasq 1. To install dnsmasq run as root following command yum install dnsmasq Loaded plugins: refresh-packagekit, security Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package dnsmasq.x86_64 0: el6 will be updated ---> Package dnsmasq.x86_64 0: el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Updating: dnsmasq x86_ el6 public_ol6_latest 149 k Transaction Summary ================================================================================ Upgrade 1 Package(s) Total download size: 149 k Is this ok [y/n]: y Downloading Packages: dnsmasq el6.x86_64.rpm 149 kb 00:00 warning: rpmts_hdrfromfdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOK EY Retrieving key from Importing GPG key 0xEC551F03: Userid: "Oracle OSS group (Open Source Software group) <build@oss.oracle.com>" From :

52 Is this ok [y/n]: y Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : dnsmasq el6.x86_64 1/2 Cleanup : dnsmasq el6.x86_64 2/2 Verifying : dnsmasq el6.x86_64 1/2 Verifying : dnsmasq el6.x86_64 2/2 Updated: dnsmasq.x86_64 0: el6 Complete! 2. Configure dnsmasq create new file /etc/racdns with settings for SCAN [root@rac1 ~]# vi /etc/racdns #scan rac-scan rac-scan.appsdba.info rac-scan rac-scan.appsdba.info rac-scan rac-scan.appsdba.info modifydnsmasq default configuration file /etc/dnsmasq.conf. One parameter addn-hosts should be changed to point to file /etc/racdns. cat /etc/dnsmasq.conf grep addn-hosts addn-hosts=/etc/racdns 3. Start dnsmasq service dnsmasq start chkconfig dnsmasq on

53 4. Next step is to resolve problem with file /etc/resolv.conf I m using third network card as NAT with DHCP so each time you restart network card or reboot host the file is overwritten with automatically generated settings. Nameserver points for which is required to resolve internet entries but not enough to resolve SCAN settings via dnsmasq. cat /etc/resolv.conf # Generated by NetworkManager search appsdba.info nameserver As default dnsmasq is running on ipadress so it requires following settings in /etc/resolv.conf. cat /etc/resolv.conf # Generated by NetworkManager nameserver search appsdba.info nameserver You must protect the file from being automatically overwritten by host reboot, network card restart etc. [root@rac1 ~]# chattr +i /etc/resolv.conf To write we use below command chattr -i /etc/resolv.conf

54 Verification It s just running nslookup to verify all is working fine nslookup rac-scan Server: Address: #53 Name: rac-scan.appsdba.info Address: Name: rac-scan.appsdba.info Address: Name: rac-scan.appsdba.info Address: [root@rac1 ~]# nslookup rac-scan Server: Address: #53 Name: rac-scan.appsdba.info Address: Name: rac-scan.appsdba.info Address: Name: rac-scan.appsdba.info Address:

55 ~]# nslookup rac1 Server: Address: #53 Name: rac1.appsdba.info Address: ~]# nslookup rac1-priv Server: Address: #53 Name: rac1-priv.appsdba.info Address: ~]# nslookup rac1-vip Server: Address: #53 Name: rac1-vip.appsdba.info Address:

56 DISABLE FIREWALL Now we need to disable the firewall: Linux Main menu System Administration Firewall. Click on Disable icon, then on Apply. Click on "Disable" icon, then on "Apply". DISABLE SECURE LINUX Disable secure linux by editing the /etc/selinux/config file, making sure the SELINUX flag is set as follows. SELINUX=permissive

57 INSTALL ORACLE-RDBMS-SERVER-11GR2-PREINSTALL USING YUM yum install oracle-rdbms-server-11gr2-preinstall DECONFIGURE NETWORK TIME PROTOCOL Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes. If you want to deconfigure NTP do the following. Below steps will disable NTP - we have followed below steps only # service ntpd stop Shutting down ntpd: [ OK ] # chkconfig ntpd off # mv /etc/ntp.conf /etc/ntp.conf.orig # rm /var/run/ntpd.pid Below steps will enable NTP If you want to use NTP, you must add the -x option into the following line in the /etc/sysconfig/ntpd file. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" Then restart NTP. # service ntpd restart

58 Edit the sysctl.conf of rac1 with the following values. vi /etc/sysctl.conf kernel.core_uses_pid = 1 kernel.shmmax= kernel.shmmni=4096 kernel.sem= fs.file-max = kernel.msgmnb = kernel.msgmni = 2878 net.core.rmem_default = net.core.rmem_max= net.core.wmem_default = net.core.wmem_max = net.ipv4.tcp_wmem = net.ipv4.tcp_rmem = fs.aio-max-nr = net.ipv4.ip_local_port_range =

59 Edit the limits.conf file of rac1 with the following values. vi /etc/security/limits.conf * hard nofile * soft nofile * hard nproc * soft nproc * hard memlock * soft memlock * hard stack * soft stack Create "oracle/grid" User and Directories in both nodes. groupadd -g 1000 oinstall groupadd -g 1020 asmadmin groupadd -g 1021 asmdba groupadd -g 1022 asmoper groupadd -g 1031 dba useradd -u g oinstall -G asmoper,asmdba,asmadmin,dba grid userdel oracle useradd -u g oinstall -G dba,asmdba oracle cd /home chown -R oracle:dba oracle

60 Create directories and grant permissions cd /home chown -R oracle:oinstall oracle/ mkdir -p /u01/app/grid chown -R grid:oinstall /u01/app/ chmod -R 775 /u01/app/grid mkdir -p /u01/app/oracle chown oracle:oinstall /u01/app/oracle chmod -R 775 /u01/app/oracle Change the password of grid and oracle users passwd grid passwd oracle passwd grid Changing password for user grid. New password: BAD PASSWORD: it is too short BAD PASSWORD: is too simple Retype new password: passwd: all authentication tokens updated successfully. home]# passwd oracle Changing password for user oracle. New password: home]# passwd oracle Changing password for user oracle. New password: BAD PASSWORD: it is based on a dictionary word BAD PASSWORD: is too simple

61 Retype new password: passwd: all authentication tokens updated successfully. INSTALL AND CONFIGURE AUTOMATIC STORAGE MANAGEMENT AND DISKS. Install Oracleasmlib. I have downloaded file from yum install oracleasm-support oracleasmlib el6.i686.rpm rpm -Uvh oracleasmlib el6.i686.rpm Preparing... ########################################### [100%] 1:oracleasmlib ########################################### [100%]

62 VERIFY ASMLIB PACKAGES. This installation needs to be performed on all nodes as the root user account: rpm -qa grep asm oracleasmlib el6.i686 oracleasm-support el6.x86_64 CONFIGURING AND LOADING THE ASMLIB PACKAGES. Now that we downloaded and installed the ASMLib Packages for Linux, we need to configure and load the ASM kernel module. This task needs to be run on all nodes as root: $ su /etc/init.d/oracleasm configure /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: grid Default group to own the driver interface []: asmadmin Start Oracle ASM library driver on boot (y/n) [n]: y

63 Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ] Login as the grid user and add the following lines at the end of the.bash_profile file. su - grid vi.bash_profile # Oracle Settings TMP=/tmp; export TMP TMPDIR=$TMP; export TMPDIR ORACLE_HOSTNAME=rac1.appsdba.info; export ORACLE_HOSTNAME ORACLE_BASE=/u01/app/grid; export ORACLE_BASE GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME ORACLE_SID=+ASM1; export ORACLE_SID ORACLE_HOME=$GRID_HOME; export ORACLE_HOME PATH=$PATH:$ORACLE_HOME/bin; export PATH ORACLE_TERM=xterm; export ORACLE_TERM LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384

64 ulimit -n else ulimit -u n fi fi Login as the oracle user and add the following lines at the end of the.bash_profile file. su oracle vi.bash_profile # User specific environment and startup programs # database name - RACDB # instance name 1 - RACDB1 # instance name 2 - RACDB2 PATH=$PATH:$HOME/bin export PATH PATH=$PATH:$HOME/bin export PATH # Oracle Settings TMP=/tmp; export TMP TMPDIR=$TMP; export TMPDIR DB_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 ORACLE_HOSTNAME=lb.appsdba.info; export ORACLE_HOSTNAME export ORACLE_UNQNAME=RACDB ORACLE_SID=RACDB1; export ORACLE_SID ORACLE_HOME=$DB_HOME; export ORACLE_HOME PATH=$ORACLE_HOME/bin:$PATH; export PATH

65 LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH ORACLE_TERM=xterm; export ORACLE_TERM if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p ulimit -n else ulimit -u n fi fi

66 BACKUP VIRTUAL MACHINE. Now we will shut down the server as we need to add asm disks. Before we add asm disks. We will backup our vm Click File export Appliance

67 Select our VM and click next Save to some location Click next and click export Ok We done with the backup now add disk s for ASM storage.

68 TO ADD SHARED DISKS CLICK ON STORAGE

69 Pending

70

71

72

73

74

75 CLONE THE VIRTUAL MACHINE Later versions of VirtualBox allow you to clone VMs, but these also attempt to clone the shared disks, which is not what we want. Instead we must manually clone the VM. Shut down the "rac1" virtual machine using the following command. # shutdown -h now If you are using VirtualBox version or later, so the following. VBoxManage clonehd rac1.vdi /d05/oracle/rac/rac2/rac2.vdi Create rac2

76

77

78 Change the disks to shared that were created earlier for ASM

79 Do the same for all the ASMDISK s

80 Now once this is done click on RAC2 Vm and Storage (Attach the disks)

81

82

83 Modify the Network Settings

84

85 Now start the Rac2 server make sure RAC1 IS DOWN The main purpose of starting the server is that we have to configure Network If we see here it is still rac1. Hostname has to be changed

86 Change the hostname and reboot rac2 cat /etc/sysconfig/network NETWORKING=yes HOSTNAME=rac2.appsdba.info Start "Network Connections" tool (Main menu System Preferences Network Connections). Edit eth0 and eth1 interfaces and set in IPv4 addresses and correspondingly.

87

88 REBOOT NODE2

89 Check the MAC address of each of the available network connections. Don't worry that they are listed as "eth2" and "eth3". These are dynamically created connections because the MAC address of the "eth0" and "eth1" connections is incorrect. ifconfig -a grep eth eth2 eth3 Link encap:ethernet HWaddr 08:00:27:57:F1:AB Link encap:ethernet HWaddr 08:00:27:64:EC:15 Edit the "/etc/sysconfig/network-scripts/ifcfg-eth0", amending only the IPADDR and HWADDR settings as follows and deleting the UUID entry. Note, the HWADDR value comes from the "eth2" interface displayed above.

90 eth2 eth3 Link encap:ethernet HWaddr 08:00:27:57:F1:AB Link encap:ethernet HWaddr 08:00:27:64:EC:15 Edit the "/etc/sysconfig/network-scripts/ifcfg-eth1", amending only the IPADDR and HWADDR settings as follows and deleting the UUID entry. Note, the HWADDR value comes from the "eth3" interface displayed above. If the adapter names do not reset properly, check the HWADDR in the "/etc/udev/rules.d/70-persistentnet.rules" file. If it is incorrect, amend it to match the settings described above. init 0 Verify the adapter settings

91

92 Verify the ipaddress

93 CONFIGURE THE SHARED DISKS Start the "rac1" virtual machine by clicking the "Start" button on the toolbar. When the server has started, log in as the root user so you can configure the shared disks. The current disks can be seen by issuing the following commands. # cd /dev # lssd* sda sda1 sda2 sdbsdcsdd fdisk l Use the "fdisk" command to partition the disks sdb to sdf. The following output shows the expected fdisk output for the sdb disk. # fdisk /dev/sdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.

94 The number of cylinders for this disk is set to There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1305, default 1): Using default value 1 Last cylinder or +size or +sizem or +sizek (1-1305, default 1305): Using default value 1305 Command (m for help): p Disk /dev/sdb: 10.7 GB, bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of * 512 = bytes Device Boot Start End Blocks Id System

95 /dev/sdb Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. # In each case, the sequence of answers is "n", "p", "1", "Return", "Return", "p" and "w". Once all the disks are partitioned, the results can be seen by repeating the previous "ls" command. # cd /dev # lssd* sda sda1 sda2 sdb sdb1 sdc sdc1 sdd sdd1 #

96 CREATE ASM DISKS. Create the ASM disks on any one node as the root user. /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1 /usr/sbin/oracleasm createdisk DISK2 /dev/sdc1 /usr/sbin/oracleasm createdisk DISK3 /dev/sdd1 Verify that the ASM disks are visible from every node. (Right now it will not work) # /etc/init.d/oracleasm scandisks Scanning system for ASM disks: [ OK ] We can see the disk are now visible to ASM using the "listdisks" command. # /usr/sbin/oracleasm listdisks DISK1 DISK2 DISK3

97 Edit the "/home/oracle/.bash_profile" file on the "rac2" node to correct the ORACLE_SID and ORACLE_HOSTNAME values. ORACLE_SID=RACDB2; export ORACLE_SID ORACLE_HOSTNAME=rac2.appsdba.info; export ORACLE_HOSTNAME Also, amend the ORACLE_SID setting in the "/home/oracle/db_env" and "/home/oracle/grid_env" files. ORACLE_SID=+ASM2; export ORACLE_SID Start the "rac1" virtual machine and restart the "rac2" virtual machine. When both nodes have started, check they can both ping all the public and private IP addresses using the following commands. ping -c 3 rac1 ping -c 3 rac1-priv ping -c 3 rac2 ping -c 3 rac2-priv At this point the virtual IP addresses defined in the "/etc/hosts" file will not work, so don't bother testing them. The virtual machine setup is now complete. rac-scan [root@rac2 ~]# nslookup rac2-vip Server: Address: #53 Name: rac2-vip.appsdba.info Address: [root@rac2 ~]# nslookup rac2 Server: Address: #53 Name: rac2.appsdba.info Address:

98 ~]# nslookup rac-scan Server: Address: #53 Name: rac-scan.appsdba.info Address: Name: rac-scan.appsdba.info Address: Name: rac-scan.appsdba.info Address:

99 TAKE SNAPSHOT NODE1 AND NODES2 Shutdown node1 and node2 After shutdown go to Oracle VM Click Snapshots on the right hand side

100

101 LOCATION OF SNAPSHOTS

102 GRID INSTALLATION Clean the old installation (on both nodes, if required) cd /etc/init.d rm rf ohasdinit.ohasd rm -rf /etc/oracle rm -rf /etc/orainst.loc rm -rf /etc/oratab cd /u01/app rm -rf oracle rm -rf /u01/app/*.* rm -rf /u01/app/11.2.0/grid/*.* cd /u01/app/grid rm -rforainventory mkdir -p /u01/app/grid chown -R grid:oinstall /u01/app/ chmod -R 775 /u01/app/grid mkdir -p /u01/app/oracle chownoracle:oinstall /u01/app/oracle chmod -R 775 /u01/app/oracle rm -rf /u01/app/grid/* rm -rf /usr/tmp/.oracle/ rm -rf /u01/app/orainventory/*

103 -- On one of the node -- If there are 3 diskgroups then use the below commands /usr/sbin/oracleasmdeletedisk DISK1 /usr/sbin/oracleasmdeletedisk DISK2 /usr/sbin/oracleasmdeletedisk DISK3 /usr/sbin/oracleasmcreatedisk DISK1 /dev/sdb1 /usr/sbin/oracleasmcreatedisk DISK2 /dev/sdc1 /usr/sbin/oracleasmcreatedisk DISK3 /dev/sdd1 /etc/init.d/oracleasmlistdisks Always Verify Public ip s Private ip s Date between the both the nodes are same NFS SHARING OF THE SOFTWARE The software is available on host machine On the server (host machine)- vi /etc/exports /software/rac_software/ (ro) Service nfs restart On the client node1 mkdir p /software

104 mount :/d04/oracle/RAC_SOFTWARE /software Software p _112030_linux-x86-64_3of7.zip unzipp _112030_linux-x86-64_3of7.zip

105 INSTALL THE MISSING PACKAGE ON NODE1 AND NODE2 cd /software/11gr11203/grid/rpm rpm -Uvh cvuqdisk rpm Preparing... ########################################### [100%] Using default group oinstall to install package 1:cvuqdisk ########################################### [100%] SETUP SSH Connect as grid user cd /software/11gr11203/grid/sshsetup./sshusersetup.sh -user grid -hosts "rac1 rac2" -nopromptpassphrase -advanced -exverify Verifying SSH connectivity has been setup from rac1 to rac IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Tue Sep 1 05:12:35 EDT Verifying SSH connectivity has been setup from rac1 to rac IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Tue Sep 1 05:12:35 EDT Verification from rac1 complete Verifying SSH connectivity has been setup from rac2 to rac IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Tue Sep 1 05:12:35 EDT 2015

106 Verifying SSH connectivity has been setup from rac2 to rac IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Tue Sep 1 05:12:36 EDT Verification from rac2 complete- SSH verification complete. VERIFY USING CLUVFY UTILITY Prior to 11gR2 we would probably use the "runcluvfy.sh" utility in the clusterware root directory to check the prerequisites have been met. If you are intending to configure SSH connectivity using the installer this check should be omitted as it will always fail. If you want to setup SSH connectivity manually, then once it is done you can run the "runcluvfy.sh" with the following command. [grid@racnode1 ~]$ cd /software/11gr11203/grid [grid@racnode1 grid]$ sh runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose If you get any failures be sure to correct them before proceeding.

107 KICK OFF THE INSTALLATION

108

109

110

111

112

113

114

115

116

117 Every ASM disk is divided into allocation units (AU). An AU is the fundamental unit of allocation within a disk group. A file extent consists of one or more AU. An ASM file consists of one or more file extents. When you create a disk group, you can set the ASM AU size to be between 1 MB and 64 MB in powers of two, such as, 1, 2, 4, 8, 16, 32, or 64. Larger AU sizes typically provide performance advantages for data warehouse applications that use large sequential reads.

118

119

120

121

122 Only for grid installation (this is already done)

123 The results of prerequisite checks are shown on the next screen. You should see two warnings and one failure. The failure was caused by inability to lookup SCAN in DNS and that should be expected. Check "Ignore All" box and press "Next".

124

125 /u01/app/orainventory/orainstroot.sh Changing permissions of /u01/app/orainventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/orainventory to oinstall. The execution of the script is complete. ~]# /u01/app/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/app/11.2.0/grid

126 Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin... Copying oraenv to /usr/local/bin... Copying coraenv to /usr/local/bin... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation OLR initialization - successful root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert

127 Adding Clusterware entries to upstart CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1' CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1' CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1' CRS-2672: Attempting to start 'ora.gipcd' on 'rac1' CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rac1' CRS-2672: Attempting to start 'ora.diskmon' on 'rac1' CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded ASM created and started successfully. Disk Group DATA created successfully. clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-4256: Updating the profile Successful addition of voting disk 5f5866ae4c0e4fafbffab1ab690a130b. Successfully replaced voting disk group with +DATA. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group ONLINE 5f5866ae4c0e4fafbffab1ab690a130b (/dev/oracleasm/disks/disk1) [DATA] Located 1 voting disk(s). CRS-2672: Attempting to start 'ora.asm' on 'rac1' CRS-2676: Start of 'ora.asm' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.data.dg' on 'rac1' CRS-2676: Start of 'ora.data.dg' on 'rac1' succeeded Configure Oracle Grid Infrastructure for a Cluster... succeeded

128 On node2 /u01/app/orainventory/orainstroot.sh Changing permissions of /u01/app/orainventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/orainventory to oinstall. The execution of the script is complete. ~]# /u01/app/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin... Copying oraenv to /usr/local/bin... Copying coraenv to /usr/local/bin... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation OLR initialization - successful Adding Clusterware entries to upstart CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating

129 An active cluster was found during exclusive startup, restarting to join the cluster Configure Oracle Grid Infrastructure for a Cluster... succeeded

130 Hurray :) we have 3 scans On node1 [root@rac1 ~]# ps -ef grep tns root Sep01? 00:00:00 [netns] grid :25? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit grid :25? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit grid :31? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit root :33 pts/0 00:00:00 greptns On node2 ps -ef grep tns root Sep01? 00:00:00 [netns] grid :29? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit grid :31? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit

131 root :33 pts/0 00:00:00 greptns [root@orcl1 ~]# su - grid cluvfy comp scan -verbose [grid@rac1 ~]$ cluvfy comp scan -verbose Verifying scan Checking Single Client Access Name (SCAN)... SCAN Name Node Running? ListenerName Port Running? rac-scan.appsdba.info rac2 true LISTENER_SCAN true rac-scan.appsdba.info rac1 true LISTENER_SCAN true rac-scan.appsdba.info rac1 true LISTENER_SCAN true Checking TCP connectivity to SCAN Listeners... Node ListenerName TCP connectivity? rac1 LISTENER_SCAN1 yes rac1 LISTENER_SCAN2 yes rac1 LISTENER_SCAN3 yes TCP connectivity to SCAN Listeners exists on all cluster nodes Checking name resolution setup for "rac-scan.appsdba.info"... SCAN Name IP Address Status Comment rac-scan.appsdba.info passed rac-scan.appsdba.info passed rac-scan.appsdba.info passed Verification of SCAN VIP and Listener setup passed Verification of scan was successful.

132 crsctl stat res -t ~]$ crsctl stat res -t NAME TARGET STATE SERVER STATE_DETAILS Local Resources ora.data.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.listener.lsnr ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac Cluster Resources ora.listener_scan1.lsnr 1 ONLINE ONLINE rac2

133 ora.listener_scan2.lsnr 1 ONLINE ONLINE rac1 ora.listener_scan3.lsnr 1 ONLINE ONLINE rac1 ora.cvu 1 ONLINE ONLINE rac1 ora.oc4j 1 ONLINE ONLINE rac1 ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE ONLINE rac2 ora.scan1.vip 1 ONLINE ONLINE rac2 ora.scan2.vip 1 ONLINE ONLINE rac1 ora.scan3.vip 1 ONLINE ONLINE rac1 POST-INSTALL ACTIONS By default, the Global Services Daemon (GSD) is not started on the cluster. To start GSD, change directory to the <CRS_HOME> and issue the following commands: srvctl enable nodeapps n rac1 srvctl enable nodeapps n rac2 srvctl start nodeapps Successful Oracle Clusterware operation can also be verified using the following command: [grid@racnode1 oraagent_grid]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online

134 CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online olsnodes -n racnode1 1 racnode2 2 srvctl status asm -a ASM is running on rac2,rac1 ASM is enabled. ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : Used space (kbytes) : 2676 Available space (kbytes) : ID : Device/File Name : +DATA Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check bypassed due to non-privileged user

135 crsctl query css votedisk ## STATE File Universal Id File Name Disk group ONLINE 5f5866ae4c0e4fafbffab1ab690a130b (/dev/oracleasm/disks/disk1) [DATA] Located 1 voting disk(s). crsctl check cluster -all ************************************************************** ## STATE File Universal Id File Name Disk group ONLINE 5f5866ae4c0e4fafbffab1ab690a130b (/dev/oracleasm/disks/disk1) [DATA] Located 1 voting disk(s). [grid@rac1 ~]$ crsctl check cluster -all ************************************************************** rac1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** rac2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online **************************************************************

136 INSTALL DATABASE Confirm that the oracle user in this session is a member of the required groups Connect as oracle user setup SSH cd /software/11gr11203/grid/sshsetup./sshusersetup.sh -user grid -hosts "rac1 rac2" -nopromptpassphrase -advanced -exverify root@rac1 Desktop]#su - oracle [oracle@rac1 ~]$ cd /tmp/database/ [oracle@rac1 database]$ ls doc readme.html rpm sshsetup welcome.html install responseruninstaller stage [oracle@rac1 database]$./runinstaller

137

138

139

140

141

142

143

144

145

146 On node1 and node2 /u01/app/oracle/product/11.2.0/dbhome_1/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1 Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Entries will be added to the /etc/oratab file as needed by

147 Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Finished product-specific root actions.

148

149 REFERENCES nice one PROBLEMS & SOLUTIONS 11gR2 Grid: root.sh fails to start the clusterware on the second node (Doc ID ) How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID ] Issues and solutions cd /home/oracle/app/11.2.0/grid/log/linux2 cat alertlinux2.log [ctssd(27935)]crs-2409:the clock on host linux2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode :26: [ctssd(27935)]crs-2412:the Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /home/oracle/app/11.2.0/grid/log/linux2/ctssd/octssd.log :26: [ctssd(27935)]crs-2409:the clock on host linux2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

150 How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID ] -- If root.sh has failed, run the below commands to clean up -- first node cd /home/oracle/app/11.2.0/grid/crs/install./rootcrs.pl -verbose -deconfig -force -- second node cd /home/oracle/app/11.2.0/grid/crs/install./rootcrs.pl -verbose -deconfig -force lastnode To add vip manually bin]#./srvctl add vip -n linux2 -A / /eth0 -k 1 [root@linux2 bin]#./srvctlconfigvip -n linux2 TIP - to start the RAC services manually cd /etc/oracle/scls_scr/racnode1/root/ Modify files crsstart and ohasdstr change enable to disable