This RAC installation article is based on RAC Attack hands-on learning lab https://en.wikibooks.org/wiki/rac_attack_-_oracle_cluster_database_at_home/rac_attack_12c The article performs the following task:- 1. Deploying Oracle Grid Infrastructure 12cR1 (12.1.0.2.0) 2. Deploying Oracle database software 12cR1 (12.1.0.2.0) 3. Using Oracle ASM with udev rules Below the hardware details Windows 7 64 bit 8 CPUs 16Gb memory, 8GB is used for the RAC 50Gb for VDI (Dynamic Allocation) 30Gb for ASM disk (Fixed Size) Below are the software components The VirtualBox for Windows: https://www.virtualbox.org/wiki/downloads The Oracle Linux (OEL 7.2 OS -- V100082-01.iso) : https://edelivery.oracle.com/linux The Oracle Database 12c R1 for Linux x86-64: http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html 1. Complete the Installation of Oracle VM VirtualBox 2. Run VirtualBox Manager. 3. From the main screen, choose File > Preferences > Network. 4. Double click on VirtualBox Host-Only Ethernet Adapter. VirtualBox will suggest a default address of 192.168.56.1. Please update the settings as shown and click Ok twice. 12cR1 RAC OEL 7.2 Page 1
IPv4 Address: 192.168.78.1 IPv4 Network Mask: 255.255.255.0 5. In VirtualBox, from the main screen, click the New icon in the upper left hand corner. 12cR1 RAC OEL 7.2 Page 2
6. Set the memory size. I have set to 4G. 7. Create virtual hard drive. 12cR1 RAC OEL 7.2 Page 3
8. Leave VDI selected and click next. 9. Leave dynamically allocated selected and click Next. 12cR1 RAC OEL 7.2 Page 4
10. Type in the full path or choose through the browse icon the Location for the disk file to be created. Have named the file racnode1.vdi and all the vdi files created are in location D:\rac12c\. 12cR1 RAC OEL 7.2 Page 5
11. From the main screen, select the virtual machine racnode1 and click the Settings icon in the upper left hand corner. 12cR1 RAC OEL 7.2 Page 6
12. Open the Storage sub-menu. Delete the Empty disk under the IDE Controller. 13. Click on the Add CD/DVD/Optical Device. 12cR1 RAC OEL 7.2 Page 7
14. Locate and open the Oracle_Linux_7_64.iso (V100082-01.iso) file. 12cR1 RAC OEL 7.2 Page 8
15. Open the Network sub-menu. Under the Adapter 1 tab, change the Attached to: dropdown to Host-only Adapter. 16. Choose the Adapter 2 tab. Check the box for Enable Network Adapter. Change the Attached to: dropdown to Internal Network and type the name rac-priv in the Name field. 12cR1 RAC OEL 7.2 Page 9
17. Choose the Adapter 3 tab. Check the box for Enable Network Adapter. Change the Attached to: dropdown to NAT. 18. Select the Shared Folders tab. Click Add Shared Folder. Enter the path where you have downloaded the Oracle installation media and enter a name for your folder. 12cR1 RAC OEL 7.2 Page 10
19. Open the USB sub-menu. Uncheck the Enable USB Controller check box. 20. Click on Start to boot the virtual machine. 12cR1 RAC OEL 7.2 Page 11
21. The Oracle Linux install screen will automatically come up. 22. Select the language and click on Continue. 12cR1 RAC OEL 7.2 Page 12
23. Next is the Installation Summary screen. 24. Select the Installation Destination. If you are happy with automatic partitioning of the whole disk, click Done button to return to previous screen. 12cR1 RAC OEL 7.2 Page 13
25. On Software Selection screen select Server with GUI for your Base Environment. Then select extra software you want to install by selecting it in Add-Ons for Selected Environment. For Oracle software installation and general development on Oracle Linux 7 please choose the below option and click Done button. 12cR1 RAC OEL 7.2 Page 14
26. Select the Network & Host Name section from Installation Summary. 12cR1 RAC OEL 7.2 Page 15
27. Select Ethernet (enp0s3), the Public IP and click on Configure. In earlier releases of OEL we had eth0, eth1, eth2, which is now enp0s3, enp0s8, enp0s9 a. Click the Connect Automatically checkbox in General Tab b. Click on IPv4 Settings and select Manual method. c. Click Add and type 192.168.78.51 for the address and 255.255.255.0 for Netmask d. Type 192.168.78.51, 192.168.78.52 in DNS servers and racattack in search domain. e. In Hostname type racnode1.racattack. f. Click on Save 12cR1 RAC OEL 7.2 Page 16
28. Select Ethernet (enp0s8) and click on Configure a. Click on Connect Automatically checkbox in General Tab b. Move to IPv4 Settings, change to Manual method and click on Add c. Type 172.16.100.51 in Address and 255.255.255.0 in Netmask d. Save the settings 12cR1 RAC OEL 7.2 Page 17
29. Select Ethernet (enp0s9) and click on Configure a. Click on Connect Automatically checkbox in General Tab b. Move to IPv4 Settings, change to Automatic (DHCP) addresses only method 12cR1 RAC OEL 7.2 Page 18
12cR1 RAC OEL 7.2 Page 19
30. Click on Begin Installation and the screen changes to CONFIGURATION window. 31. Set the ROOT PASSWORD and USER CREATION while the installation is going on. 12cR1 RAC OEL 7.2 Page 20
32. Completion Screen: - After the Oracle Linux is successfully installed, go ahead and reboot it. After reboot you would need to complete License Information. 12cR1 RAC OEL 7.2 Page 21
12cR1 RAC OEL 7.2 Page 22
33. Turn off and disable the Firewall. systemctl and systemd have replaced chkconfig and init under OEL7. 34. Change the setting of SELinux to disabled by editing /etc/selinux/config. 12cR1 RAC OEL 7.2 Page 23
35. Check the ntpd status. Either configure NTP, or make sure it is not configured so that CTSSd can synchronize the times of RAC node. In my installation I de-configured NTP. # systemctl disable ntpd # mv /etc/ntp.conf /etc/ntp.conf.orig 36. Verify that all the network interfaces are up. 12cR1 RAC OEL 7.2 Page 24
37. Configure DNS. 12cR1 RAC OEL 7.2 Page 25
Original /etc/named.conf file before any changes - Add 192.168.78.51 to listen-on port 53 and 192.168.78.0/24 to allow-query and allowtransfer. 12cR1 RAC OEL 7.2 Page 26
At the end of file add the zone and reverse zone as shown below. Create the zone file for the racattack domain on racnode1. 12cR1 RAC OEL 7.2 Page 27
Create the reverse zone file Verify the changes made to /etc/named.conf file as shown below. In case of any errors, rectify it and re-check. 12cR1 RAC OEL 7.2 Page 28
Enabled the named service and started it Check the forward and reverse lookup using nslookup command. 12cR1 RAC OEL 7.2 Page 29
12cR1 RAC OEL 7.2 Page 30
38. Install the database preinstall package oracle-rdbms-server-12cr1-preinstall. Select the Devices > Optical Drives > oracle_linux_7_64.iso 12cR1 RAC OEL 7.2 Page 31
Run the following commands. Make a backup copy of file, delete the content and add the below lines to /etc/yum.repos.d/publicyum-ol7.repo. Install the oracle-rdbms-server-12cr1-preinstall package. It performs all prerequisite setup like installing required rpms, creating oracle user, required groups, setting parameters in /etc/sysctl.conf etc. 12cR1 RAC OEL 7.2 Page 32
One thing to note is, Oracle user limit does not reflect values in /etc/security/limit.conf if oraclerdbms-server-12cr1-preinstall RPM is used. The file to be checked is /etc/security/limits.d/oraclerdbms-server-12cr1-preinstall.conf. 12cR1 RAC OEL 7.2 Page 33
39. Unmount/Eject the device (oracle_linux_7_64.iso) and Make the VirtualBox guest additions available to the OS by clicking Devices->Install Guest Additions. 12cR1 RAC OEL 7.2 Page 34
40. After the guest addition, the Shared Folders added in step 18 should be visible. The folder is visible as /media/sf_12cr1. 41. Create the directories in which the Oracle Software will be installed and add oracle to vboxsf group. 12cR1 RAC OEL 7.2 Page 35
42. Create Virtualbox Shared Storage (ASM Disks) a. Shutdown the server. b. In the VirtualBox Manager, select the machine racnode1, click Settings -> Storage. c. Select Controller SATA and click on the Add Hard Disk button. Click Create new disk. 12cR1 RAC OEL 7.2 Page 36
12cR1 RAC OEL 7.2 Page 37
43. Repeat the same steps to create three more disks: 44. Make all the asm devices sharable: from VirtualBox Manager, click File -> Virtual Media Manager. 12cR1 RAC OEL 7.2 Page 38
Repeat the same above steps for all the asm*.vdi disk. 12cR1 RAC OEL 7.2 Page 39
45. Start the racnode1 and you must see 4 disks named /dev/sd*. 46. Once the new disks are visible in the server, add primary partition on each of them using fdisk. 12cR1 RAC OEL 7.2 Page 40
47. Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn't already exist. # echo options=-g > /etc/scsi_id.config Check if scsi_id returns data. # /usr/lib/udev/scsi_id g u d /dev/sdb Prepare the /etc/udev/rules.d/99-oracle-asmdevices.rules by running the below script. #!/bin/bash #file ~/createoracleasmudevrules.sh i=1 # ol7 / rhel7 cmd="/usr/lib/udev/scsi_id -g -u -d" for disk in sdb sdc sdd sde ; do cat <<EOF >> /etc/udev/rules.d/99-oracle-asmdevices.rules KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="$cmd /dev/\$parent", \ RESULT=="`$cmd /dev/$disk`", SYMLINK+="asm-disk$i", OWNER="oracle", GROUP="dba", \ MODE="0660" EOF i=$(($i+1)) done 12cR1 RAC OEL 7.2 Page 41
Load updated block device partition tables. Test the rules are working as expected. Restart UDEV Service. 12cR1 RAC OEL 7.2 Page 42
48. Clone VirtualBox VM a. Shutdown the VM. # shutdown h now b. Clone the disk racnod1.vdi: from VirtualBox Manager, click File -> Virtual Media Manager. 12cR1 RAC OEL 7.2 Page 43
12cR1 RAC OEL 7.2 Page 44
Once the copy is complete, click Close to exit from the Virtual Media Manager. 12cR1 RAC OEL 7.2 Page 45
49. In VirtualBox, click New icon in the upper left corner to create the new VM. 12cR1 RAC OEL 7.2 Page 46
12cR1 RAC OEL 7.2 Page 47
50. Open the Network sub-menu. Under the Adapter 1 tab, change the Attached to: dropdown to Host-only Adapter. 51. Choose the Adapter 2 tab. Check the box for Enable Network Adapter. Change the Attached to: dropdown to Internal Network and type the name rac-priv in the Name field. 52. Choose the Adapter 3 tab. Check the box for Enable Network Adapter. Change the Attached to: dropdown to NAT. 12cR1 RAC OEL 7.2 Page 48
53. Open the USB sub-menu. Uncheck the Enable USB Controller check box. 54. Select the Shared Folders tab. Click Add Shared Folder. Enter the path where you have downloaded the Oracle installation media and enter a name for your folder. 12cR1 RAC OEL 7.2 Page 49
55. Click on Controller SATA and Choose existing disk to add the asm disk vdi file. 56. Repeat the operation to add all asm disks and then click OK to save the modifications. 12cR1 RAC OEL 7.2 Page 50
57. Review the summary of new virtual machine. 58. Start the racnode2 and login as root. 12cR1 RAC OEL 7.2 Page 51
59. On the top right had side end you would see Ethernet setting. Choose one and click on Wired Settings. 60. Below is the screen-shot for enp0s3 (Public IP). Click on the right end bottom mark. 12cR1 RAC OEL 7.2 Page 52
61. Change the Address to 192.168.78.52 for enp0s3. 62. In the similar way change the Address to 172.16.100.52 (Private-IP) for enp0s8. 12cR1 RAC OEL 7.2 Page 53
63. For enp0s9 uncheck Connect Automatically 64. On terminal as root type nmtui (NetworkManager Text User Interface) 65. Change the Hostname 12cR1 RAC OEL 7.2 Page 54
66. Delete the UUID entry for ifcfg-enp0s[389]. # sed i e /UUID/d ifcfg-enp0s[389] 12cR1 RAC OEL 7.2 Page 55
67. Reboot the server and then verify all the network interfaces are up and ASM devices are visible. 68. Check the firewall status (should be inactive) and stop the DNS. 12cR1 RAC OEL 7.2 Page 56
69. Delete the racattack.zone and racattack.revzone from /var/named. 70. Edit the /etc/named.conf and change.51 to.52 and the end of the file as below. 12cR1 RAC OEL 7.2 Page 57
71. Review the /etc/resolv.conf. 72. Start DNS and check the forward and reverse lookup using nslookup command. 12cR1 RAC OEL 7.2 Page 58
73. Check that both the master on racnode1 and slave on racnode2 DNS servers are working. Run all the below statements to verify correct configuration of both DNS servers :- [root@racnode2 ~]# dig @racnode1 racnode1.racattack [root@racnode2 ~]# dig @racnode1 racnode2.racattack [root@racnode2 ~]# dig @racnode1 racnode1-vip.racattack [root@racnode2 ~]# dig @racnode1 racnode2-vip.racattack [root@racnode2 ~]# dig @racnode1 racnode1-priv.racattack [root@racnode2 ~]# dig @racnode1 racnode2-priv.racattack [root@racnode2 ~]# dig @racnode1 rac-cluster-scan.racattack [root@racnode2 ~]# dig @racnode2 racnode1.racattack [root@racnode2 ~]# dig @racnode2 racnode2.racattack [root@racnode2 ~]# dig @racnode2 racnode1-vip.racattack [root@racnode2 ~]# dig @racnode2 racnode2-vip.racattack [root@racnode2 ~]# dig @racnode2 racnode1-priv.racattack [root@racnode2 ~]# dig @racnode2 racnode2-priv.racattack [root@racnode2 ~]# dig @racnode2 rac-cluster-scan.racattack 12cR1 RAC OEL 7.2 Page 59
74. When both nodes have started, check they can both ping all the public and private IP addresses using the following commands. 12cR1 RAC OEL 7.2 Page 60
75. Install the Grid Infrastructure a. Install the following package for the Oracle Grid media as root user on both the nodes. b. Setup the passwordless ssh for as Oracle user../sshusersetup.sh user oracle hosts racnode1 racnode2 nopromptpassphrase advanced exverify logfile /home/oracle/sshsetup.log c. At this point you can run the runcluvfy.sh (optional) 12cR1 RAC OEL 7.2 Page 61
d. Run the Grid Infrastructure runinstaller as Oracle [oracle@racnode1 grid]./runinstaller e. Below are the screen-shots for the configuration. 12cR1 RAC OEL 7.2 Page 62
12cR1 RAC OEL 7.2 Page 63
f. In the SCAN Name box, enter rac-cluster-scan.racattack. In the central box, only the first node is displayed. Click on Add and enter the network names of the second node. Public Hostname: - racnode2.racttack Virtual Hostname: - racnode2-vip.racattack Since the SCAN Name chosen is more than 15 character, Oracle proposes different name as Cluster Name. Click Yes. 12cR1 RAC OEL 7.2 Page 64
g. Check the correct addresses are set as below. h. Make sure the Software Location is correct and change Cluster Registry Storage Type to Oracle Automatic Storage Management. Enter SYSASM password and confirm. Click Yes. 12cR1 RAC OEL 7.2 Page 65
i. Set the redundancy to external, click the Change Discovery Path button and set the path to /dev/asm-disk*. I have changed the disk group name to DATA01.Once completed click on Next. 12cR1 RAC OEL 7.2 Page 66
j. If you want the root script to run automatically, enter the relevant credentials. I prefer to run them manually. 12cR1 RAC OEL 7.2 Page 67
k. Wait while the prerequisite checks complete. l. The below checks can be ignored for this type of installation. 12cR1 RAC OEL 7.2 Page 68
m. Save the response file and review the summary. Click the Install button. n. Installation progress 12cR1 RAC OEL 7.2 Page 69
o. When prompted, run the configuration scripts in each node. Once both the scripts are run on all the nodes, click OK. 12cR1 RAC OEL 7.2 Page 70
p. The installation continues and may take longtime to complete. MGMTDB creation may take some time. 12cR1 RAC OEL 7.2 Page 71
q. Optional step: - You can increase the timeout of CRS before it causes fencing of the node. 12cR1 RAC OEL 7.2 Page 72
r. You can check the crs patch details. 76. As oracle user add new diskgroup for FRA The asmca opens on the Disk Groups tab, showing the DATA01 Diskgroup created by the installer. Click on create and enter FRA in Disk Group Name. Choose External (node) redundancy. Check one disk to create 10GB diskgroup. Click OK 12cR1 RAC OEL 7.2 Page 73
77. After the diskgroup is created 78. Install Database Software a. You can use cluvfy to perform prerequisite checks (Optional) cluvfy stage pre dbinst n racnode1,racnode2 d /u01/app/oracle/product/12.1.0.2/db_1 -verbose 12cR1 RAC OEL 7.2 Page 74
b. Make sure all nodes are started and then login to racnode1 as oracle user and start the Oracle Installer. 12cR1 RAC OEL 7.2 Page 75
c. Make sure both nodes are selected, then click Next button 12cR1 RAC OEL 7.2 Page 76
12cR1 RAC OEL 7.2 Page 77
d. Make sure the Oracle Base and Software Location are as needed 12cR1 RAC OEL 7.2 Page 78
e. For my installation I ignored both the warnings. 12cR1 RAC OEL 7.2 Page 79
f. Review the summary information and click the Install button. g. Run the configuration script on each node. When the script has been run on each node, click OK button. 12cR1 RAC OEL 7.2 Page 80
79. Create Database a. Run the Database Configuration Assistance (DBCA) 12cR1 RAC OEL 7.2 Page 81
b. Global Database Name:- cdbrac, Pluggable Database Name:- PDB 12cR1 RAC OEL 7.2 Page 82
c. Review the Create Database Summary and Click Finish 12cR1 RAC OEL 7.2 Page 83
12cR1 RAC OEL 7.2 Page 84
12cR1 RAC OEL 7.2 Page 85