PART 1 of 4 - GRID INFRASTRUCTURE INSTALLATION Environment details: Red Hat Enterprise Linux Server release 6.4 (Santiago) Oracle 11.2.0.3 PSU4 Preinstallation Steps ## Information provided by OS admin: [root@laguna2:/u01/orainst]# nslookup laguna-scan.turkcell.tgc Server: 10.210.154.150 Address: 10.210.154.150#53 Name: laguna-scan.turkcell.tgc Address: 10.200.160.196 Name: laguna-scan.turkcell.tgc Address: 10.200.160.197 Name: laguna-scan.turkcell.tgc Address: 10.200.160.213 [root@laguna2:~]# oracleasm listdisks CONF1 CONF2 CONF3 CONF4 CONF5 PROD1 PROD2 PROD3 Laguna1.turkcell.tgc Root pass:123456 ## OS version [grid@laguna1 raw]$ uname -a Linux laguna1 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux [grid@laguna1 raw]$ cat /etc/*-release LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics- 4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Red Hat Enterprise Linux Server release 6.4 (Santiago) root:oinstall /u01
root:oinstall /u01/app oracle ORACLE_BASE oracle:oinstall /u01/app/oracle oracle ORACLE_HOME oracle:oinstall /u01/app/oracle/product/11.2.0/dbhome_1 grid ORACLE_HOME root:oinstall /u01/app/11.2.0/grid grid $ORACLE_BASE grid:oinstall /u01/app/grid #### Creating required directories #### mkdir /u01/app chown root:oinstall /u01/app ## Create grid OB directory on both nodes. mkdir /u01/app/grid chown grid:oinstall /u01/app/grid ## Create grid OH directory on both nodes. mkdir -p /u01/app/11.2.0/grid chown -R grid:oinstall /u01/app/11.2.0/grid ## Create oracle OB directory on both nodes. mkdir /u01/app/oracle chown oracle:oinstall /u01/app/oracle ## Create oracle OH directory on both nodes. mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1 chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/dbhome_1 ## additioal rights &ownership chmod -R 775 /u01 chown grid:oinstall /u01 [root@laguna2:/u01/orainst]# mkdir /u01/app [root@laguna2:/u01/orainst]# chown root:oinstall /u01/app [root@laguna2:/u01/orainst]# [root@laguna2:/u01/orainst]# mkdir /u01/app/oracle [root@laguna2:/u01/orainst]# chown oracle:oinstall /u01/app/oracle [root@laguna2:/u01/orainst]# [root@laguna2:/u01/orainst]# mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1 [root@laguna2:/u01/orainst]# chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/dbhome_1 [root@laguna2:/u01/orainst]# [root@laguna2:/u01/orainst]# mkdir -p /u01/app/11.2.0/grid [root@laguna2:/u01/orainst]# chown -R grid:oinstall /u01/app/11.2.0/grid [root@laguna2:/u01/orainst]# [root@laguna2:/u01/orainst]# mkdir /u01/app/grid [root@laguna2:/u01/orainst]# chown grid:oinstall /u01/app/grid [root@laguna2:~]# chmod -R 775 /u01/ [root@laguna2:/u01/app]# chown grid:oinstall /u01 [root@laguna2:/u01/app]#
## Continue from below installation documentation. ##Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux E22489-03 ## Manual preinstallation checks, on both nodes. #Verify System Requirements 1.3.1 grep MemTotal /proc/meminfo grep SwapTotal /proc/meminfo [root@laguna1:/u01/app]# grep MemTotal /proc/meminfo MemTotal: 132141688 kb [root@laguna1:/u01/app]# [root@laguna1:/u01/app]# grep SwapTotal /proc/meminfo SwapTotal: 4194296 kb [root@laguna2:/u01/orainst]# grep MemTotal /proc/meminfo MemTotal: 132141688 kb [root@laguna2:/u01/orainst]# grep SwapTotal /proc/meminfo SwapTotal: 4194296 kb [root@laguna2:/u01/orainst]# # Ensure you have at least 6.5 GB of space for the Oracle Grid Infrastructure /u01 disk # Ensure that you have at least 1 GB of space in /tmp [root@laguna1:/u01/app]# df -h /tmp Filesystem Size Used Avail Use% Mounted on /dev/mapper/vgroot-lvroot 20G 4.1G 15G 22% / [root@laguna2:/u01/orainst]# df -h /tmp Filesystem Size Used Avail Use% Mounted on /dev/mapper/vgroot-lvroot 20G 4.1G 15G 22% / [root@laguna1:/u01/app]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vgroot-lvroot 20G 4.1G 15G 22% / tmpfs 64G 0 64G 0% /dev/shm /dev/sda1 485M 87M 373M 19% /boot /dev/mapper/oravg-u01lv 99G 7.0G 87G 8% /u01 [root@laguna2:/u01/orainst]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vgroot-lvroot 20G 4.1G 15G 22% / tmpfs 64G 0 64G 0% /dev/shm
/dev/sda1 485M 87M 373M 19% /boot /dev/mapper/oravg-u01lv 99G 5.9G 88G 7% /u01 ## 2.Advanced Installation Oracle Grid Infrastructure for a Cluster Preinstallation Tasks Navigate to /u01/orainst/grid, unzipped GI installation files folder. [grid@laguna1 grid]$ /u01/orainst/grid/runcluvfy.sh stage -pre crsinst -n laguna1,laguna2 -fixup -verbose /u01/orainst/cvu/bin/cluvfy stage -pre crsinst -n laguna1,laguna2 -fixup -verbose Performing pre-checks for cluster services setup Checking node reachability... Check: Node reachability from node "laguna1" Destination Node Reachable? ------------------------------------ ------------------------ laguna1 yes laguna2 yes Result: Node reachability check passed from node "laguna1" Checking user equivalence... Check: User equivalence for user "grid" Node Name Status ------------------------------------ ------------------------ laguna2 failed laguna1 failed Result: PRVF-4007 : User equivalence check failed for user "grid" ERROR: User equivalence unavailable on all the specified nodes Verification cannot proceed Pre-check for cluster services setup was unsuccessful on all the nodes. [grid@laguna1 grid]$ id grid uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1020(asmadmin),1021(asmdba),1022(asmoper) [grid@laguna2 grid]$ id grid uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1020(asmadmin),1021(asmdba),1022(asmoper) ==> SSH connectivity issue ### setup grid passwordless ssh connection for each node. example below is for grid@laguna1 to grid@laguna2. ### [grid@laguna1 grid]$ cd ~ [grid@laguna1 ~]$ cd.ssh [grid@laguna1.ssh]$ ls -ltr total 0 [grid@laguna1.ssh]$
[grid@laguna1.ssh]$ ls -las total 8 4 drwx------ 2 grid oinstall 4096 May 8 14:30. 4 drwx------ 5 grid oinstall 4096 May 8 14:40.. [grid@laguna1.ssh]$ [grid@laguna1.ssh]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/grid/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_rsa. Your public key has been saved in /home/grid/.ssh/id_rsa.pub. The key fingerprint is: 95:9f:af:20:5e:ef:28:22:bf:42:dc:b0:91:1b:c7:33 grid@laguna1 The key's randomart image is: +--[ RSA 2048]----+. o o = E.... O os o =.... o. o...o +. +oo...+ +-----------------+ [grid@laguna1.ssh]$ ls -las total 16 4 drwx------ 2 grid oinstall 4096 May 8 14:54. 4 drwx------ 5 grid oinstall 4096 May 8 14:40.. 4 -rw------- 1 grid oinstall 1675 May 8 14:54 id_rsa 4 -rw-r--r-- 1 grid oinstall 394 May 8 14:54 id_rsa.pub [grid@laguna1.ssh]$ ssh ssh ssh-add ssh-agent ssh-copy-id sshd ssh-keygen sshkeyscan [grid@laguna1.ssh]$ ssh-copy-id Usage: /usr/bin/ssh-copy-id [-i [identity_file]] [user@]machine [grid@laguna1.ssh]$ ssh-copy-id -i id_rsa.pub grid@laguna2.turkcell.tgc The authenticity of host 'laguna2.turkcell.tgc (10.200.160.194)' can't be established. RSA key fingerprint is e5:c8:18:3f:af:7b:a2:85:d0:25:21:da:f2:03:81:e2. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'laguna2.turkcell.tgc,10.200.160.194' (RSA) to the list of known hosts... grid@laguna2.turkcell.tgc's password: Now try logging into the machine, with "ssh 'grid@laguna2.turkcell.tgc'", and check in:.ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. [grid@laguna1.ssh]$ ssh grid@laguna2 The authenticity of host 'laguna2 (10.200.160.194)' can't be established. RSA key fingerprint is e5:c8:18:3f:af:7b:a2:85:d0:25:21:da:f2:03:81:e2. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'laguna2' (RSA) to the list of known hosts.
.. [grid@laguna2 ~]$ [grid@laguna1 grid]$ export DISPLAY=10.200.20.147:0.0 [grid@laguna1 grid]$ ### setup oracle passwordless ssh connection for each node. example below is for oracle@laguna2 to oracle@laguna1. ### [oracle@laguna2.ssh]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_rsa): /home/oracle/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_rsa. Your public key has been saved in /home/oracle/.ssh/id_rsa.pub. The key fingerprint is: d8:31:d3:34:07:f5:58:3e:9e:5d:c4:99:cb:46:fc:bf oracle@laguna2 The key's randomart image is: +--[ RSA 2048]----+ +oo o.+ o o = *. +.. * + o +. Bo. S + o.. E +-----------------+ [oracle@laguna2.ssh]$ [oracle@laguna2.ssh]$ ssh-copy-id -i id_rsa.pub oracle@laguna1.turkcell.tgc The authenticity of host 'laguna1.turkcell.tgc (10.200.160.211)' can't be established. RSA key fingerprint is cf:28:91:95:74:1c:4e:eb:80:5f:02:22:a9:26:55:1a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'laguna1.turkcell.tgc,10.200.160.211' (RSA) to the list of known hosts. oracle@laguna1.turkcell.tgc's password: Now try logging into the machine, with "ssh 'oracle@laguna1.turkcell.tgc'", and check in:.ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. [oracle@laguna2.ssh]$ ssh laguna1 The authenticity of host 'laguna1 (10.200.160.211)' can't be established. RSA key fingerprint is cf:28:91:95:74:1c:4e:eb:80:5f:02:22:a9:26:55:1a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'laguna1' (RSA) to the list of known hosts. [oracle@laguna1 ~]$
ssh-copy-id -i id_rsa.pub root@laguna1.turkcell.tgc ### setup root passwordless ssh connection for each node. example below is for root@laguna1 to root@laguna2. ### [root@laguna2:~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 06:f5:74:89:2b:6b:7e:3e:a9:54:4f:1f:15:e7:62:e9 root@laguna2 The key's randomart image is: +--[ RSA 2048]----+....... o... oo... + o... o o So.. E.o. o.. o...... +.+.. +-----------------+ [root@laguna2:~]# [root@laguna2:~]# cd.ssh [root@laguna2:~/.ssh]# ls -las total 20 4 drwx------ 2 root root 4096 May 21 13:50. 4 dr-xr-x---. 5 root root 4096 May 21 00:46.. 4 -rw------- 1 root root 1675 May 21 13:50 id_rsa 4 -rw-r--r-- 1 root root 394 May 21 13:50 id_rsa.pub 4 -rw-r--r-- 1 root root 629 May 8 17:19 known_hosts [root@laguna2:~/.ssh]# [root@laguna2:~/.ssh]# ssh-copy-id -i id_rsa.pub root@laguna1.turkcell.tgc The authenticity of host 'laguna1.turkcell.tgc (10.200.160.211)' can't be established. RSA key fingerprint is cf:28:91:95:74:1c:4e:eb:80:5f:02:22:a9:26:55:1a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'laguna1.turkcell.tgc' (RSA) to the list of known hosts. root@laguna1.turkcell.tgc's password: Now try logging into the machine, with "ssh 'root@laguna1.turkcell.tgc'", and check in:.ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. [root@laguna2:~/.ssh]#
### 2.8 additional manual package check ### rpm -qa grep elfutils-libelf-devel rpm -qa grep elfutils-libelf-devel rpm -qa grep gcc rpm -qa grep gcc-c++ rpm -qa grep glibc rpm -qa grep glibc-common rpm -qa grep glibc-devel rpm -qa grep glibc-headers rpm -qa grep kernel-headers rpm -qa grep ksh-20060214 rpm -qa grep libaio rpm -qa grep libaio-devel rpm -qa grep libgcc rpm -qa grep libgomp rpm -qa grep libstdc++ rpm -qa grep libstdc++-devel rpm -qa grep make rpm -qa grep sysstat rpm -qa grep unixodbc rpm -qa grep unixodbc-devel [root@laguna1:~]# rpm -qa grep glibc-common glibc-common-2.12-1.107.el6.x86_64 [root@laguna1:~]# rpm -qa grep glibc-devel glibc-devel-2.12-1.107.el6.x86_64 [root@laguna1:~]# rpm -qa grep glibc-headers glibc-headers-2.12-1.107.el6.x86_64 compat-glibc-headers-2.5-46.2.x86_64 [root@laguna1:~]# rpm -qa grep kernel-headers kernel-headers-2.6.32-358.el6.x86_64 [root@laguna1:~]# rpm -qa grep ksh-20060214 [root@laguna1:~]# rpm -qa grep ksh mksh-39-7.el6.x86_64 ksh-20100621-19.el6.x86_64 [root@laguna1:~]# rpm -qa grep libaio libaio-devel-0.3.107-10.el6.x86_64 libaio-0.3.107-10.el6.x86_64 [root@laguna1:~]# rpm -qa grep libaio-devel libaio-devel-0.3.107-10.el6.x86_64 [root@laguna1:~]# rpm -qa grep libgcc libgcc-4.4.7-3.el6.i686 libgcc-4.4.7-3.el6.x86_64 [root@laguna1:~]# rpm -qa grep libgomp libgomp-4.4.7-3.el6.x86_64 [root@laguna1:~]# rpm -qa grep libstdc++ compat-libstdc++-296-2.96-144.el6.i686 libstdc++-devel-4.4.7-3.el6.x86_64 compat-libstdc++-33-3.2.3-69.el6.x86_64 libstdc++-4.4.7-3.el6.x86_64 libstdc++-4.4.7-3.el6.i686 [root@laguna1:~]# rpm -qa grep libstdc++-devel libstdc++-devel-4.4.7-3.el6.x86_64 [root@laguna1:~]# rpm -qa grep make makebootfat-1.4-10.el6.x86_64 make-3.81-20.el6.x86_64 [root@laguna1:~]# rpm -qa grep sysstat sysstat-9.0.4-20.el6.x86_64 [root@laguna1:~]# rpm -qa grep unixodbc unixodbc-devel-2.2.14-12.el6_3.x86_64
unixodbc-2.2.14-12.el6_3.x86_64 [root@laguna1:~]# rpm -qa grep unixodbc-devel unixodbc-devel-2.2.14-12.el6_3.x86_64 [root@laguna1:~]# ## 2.10. Verifying UDP and TCP Kernel Parameters cat /proc/sys/net/ipv4/ip_local_port_range [root@laguna1:~]# cat /proc/sys/net/ipv4/ip_local_port_range 9000 65500 ## 2.15.3 Check Resource Limits for the Oracle Software Installation Users ## grid@laguna1 grid]$ ulimit -Sn 131072 [grid@laguna1 grid]$ ulimit -Hn 131072 [grid@laguna1 grid]$ ulimit -Su 8192 [grid@laguna1 grid]$ ulimit -Hu 16384 [grid@laguna1 grid]$ ulimit -Ss 10240 [grid@laguna1 grid]$ ulimit -Hs unlimited [grid@laguna1 grid]$ [root@laguna1:~]# oracleasm listdisks CONF1 CONF2 CONF3 CONF4 CONF5 PROD1 PROD2 PROD3 PROD4 [root@laguna1:~]# [root@laguna1:~]# [root@laguna1:~]# oracleasm status Checking if ASM is loaded: yes Checking if /dev/oracleasm is mounted: yes [root@laguna1:~]# [grid@laguna1 grid]$./runinstaller Starting Oracle Universal Installer... Checking Temp space: must be greater than 120 MB. Actual 15005 MB Passed Checking swap space: must be greater than 150 MB. Actual 4095 MB Passed Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed Preparing to launch Oracle Universal Installer from /tmp/orainstall2013-05-21_05-54-43pm. Please wait...[grid@laguna1 grid]$
Installation Steps
Admin123
[root@laguna1:~]# /u01/app/orainventory/orainstroot.sh Changing permissions of /u01/app/orainventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/orainventory to oinstall. The execution of the script is complete. [root@laguna1:~]# /u01/app/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin... Copying oraenv to /usr/local/bin... Copying coraenv to /usr/local/bin... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory
User ignored Prerequisites during installation OLR initialization - successful root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding Clusterware entries to upstart CRS-2672: Attempting to start 'ora.mdnsd' on 'laguna1' CRS-2676: Start of 'ora.mdnsd' on 'laguna1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'laguna1' CRS-2676: Start of 'ora.gpnpd' on 'laguna1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'laguna1' CRS-2672: Attempting to start 'ora.gipcd' on 'laguna1' CRS-2676: Start of 'ora.cssdmonitor' on 'laguna1' succeeded CRS-2676: Start of 'ora.gipcd' on 'laguna1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'laguna1' CRS-2672: Attempting to start 'ora.diskmon' on 'laguna1' CRS-2676: Start of 'ora.diskmon' on 'laguna1' succeeded CRS-2676: Start of 'ora.cssd' on 'laguna1' succeeded ASM created and started successfully. Disk Group CONFIG created successfully. clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-4256: Updating the profile Successful addition of voting disk 6dae7976ff034f6abf23a9eeb07866c5. Successful addition of voting disk 1d807080e86c4fe2bf0acf6d2f3dc3fe. Successful addition of voting disk 453f08f20d4b4f92bff4d6bcf979b655. Successful addition of voting disk 4fe8507cb22f4ff6bf2a6934eb56a330. Successful addition of voting disk 832af65698704fd0bf2e8cf4a865b9b7. Successfully replaced voting disk group with +CONFIG. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 6dae7976ff034f6abf23a9eeb07866c5 (ORCL:CONF1) [CONFIG] 2. ONLINE 1d807080e86c4fe2bf0acf6d2f3dc3fe (ORCL:CONF2) [CONFIG] 3. ONLINE 453f08f20d4b4f92bff4d6bcf979b655 (ORCL:CONF3) [CONFIG]
4. ONLINE 4fe8507cb22f4ff6bf2a6934eb56a330 (ORCL:CONF4) [CONFIG] 5. ONLINE 832af65698704fd0bf2e8cf4a865b9b7 (ORCL:CONF5) [CONFIG] Located 5 voting disk(s). CRS-2672: Attempting to start 'ora.asm' on 'laguna1' CRS-2676: Start of 'ora.asm' on 'laguna1' succeeded CRS-2672: Attempting to start 'ora.config.dg' on 'laguna1' CRS-2676: Start of 'ora.config.dg' on 'laguna1' succeeded Configure Oracle Grid Infrastructure for a Cluster... succeeded [root@laguna1:~]# [root@laguna2:~]# /u01/app/11.2.0/grid/root.sh Performing root user operation for Oracle 11g The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin... Copying oraenv to /usr/local/bin... Copying coraenv to /usr/local/bin... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation OLR initialization - successful Adding Clusterware entries to upstart CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node laguna1, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster Configure Oracle Grid Infrastructure for a Cluster... succeeded [root@laguna2:~]#
### Oracle Cluster Verification Utility NTP related error: There are three problems: 1) ntpd daemon is dead. Restarting ntpd process resolved the problem. service ntpd status service ntpd stop service ntpd start 2) ntp server entry is not valid /etc/ntp.conf Deleting invalid server names, inserting valid server, resolved. 3) time is beyond drift value: adjust date according to ntp server. ntpdate <ntp server ip> ntpdate 10.200.100.12 After getting NTP related error: INFO: CTSS is in Observer state. Switching over to clock synchronization checks using NTP INFO: Starting Clock synchronization checks using Network Time Protocol(NTP)... INFO: NTP Configuration file check started... INFO: NTP Configuration file check passed INFO: Checking daemon liveness... INFO: Liveness check passed for "ntpd" INFO: Check for NTP daemon or service alive passed on all nodes INFO: NTP daemon slewing option check failed on some nodes INFO: Check failed on nodes: INFO: laguna2 INFO: PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "-x" [root@laguna2:~/.ssh]# service ntpd status ntpd dead but pid file exists [root@laguna2:~/.ssh]# ^C [root@laguna2:~/.ssh]# service ntpd stop Shutting down ntpd: [FAILED] [root@laguna2:~/.ssh]# service ntpd start Starting ntpd: [ OK ] [root@laguna2:~/.ssh]# [root@laguna2:~/.ssh]# service ntpd status ntpd (pid 47784) is running... [root@laguna2:~/.ssh]# service ntpd stop Shutting down ntpd: [ OK ] [root@laguna2:~/.ssh]# service ntpd start Starting ntpd: [ OK ] [root@laguna2:~/.ssh]# service ntpd status ntpd (pid 48323) is running... [root@laguna2:~/.ssh]# vi /etc/ntp.conf ### NTP server ntp.conf file was containing illegal server entries. [root@laguna2:~/.ssh]# service ntpd stop Shutting down ntpd: [ OK ] ### ntpdate command for updating date accoring to supplied ntp server.
[root@laguna2:~/.ssh]# ntpdate 22 May 09:26:29 ntpdate[61843]: no servers can be used, exiting [root@laguna2:~/.ssh]# ntpdate 10.200.100.12 22 May 09:26:40 ntpdate[62471]: step time server 10.200.100.12 offset 2.175496 sec [root@laguna2:~/.ssh]# ### After pressing Retry it successfully finished the step: Oracle Cluster Verification Utility
[It is not over, continueing in other page...]
### It worths seeing the change in ifconfig output before and after GI installation: ###BEFORE [root@laguna1:~]# ifconfig bond0 Link encap:ethernet HWaddr AC:16:2D:B0:29:74 inet addr:10.200.160.211 Bcast:10.200.160.255 Mask:255.255.255.0 inet6 addr: fe80::ae16:2dff:feb0:2974/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:300879 errors:0 dropped:0 overruns:0 frame:0 TX packets:68500 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:71872763 (68.5 MiB) TX bytes:67371829 (64.2 MiB) bond1 eth0 eth1 eth2 eth3 lo Link encap:ethernet HWaddr AC:16:2D:B0:29:76 inet addr:90.90.90.91 Bcast:90.90.90.255 Mask:255.255.255.0 inet6 addr: fe80::ae16:2dff:feb0:2976/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:69690 errors:0 dropped:0 overruns:0 frame:0 TX packets:224 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:4181476 (3.9 MiB) TX bytes:10704 (10.4 KiB) Link encap:ethernet HWaddr AC:16:2D:B0:29:74 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:199869 errors:0 dropped:0 overruns:0 frame:0 TX packets:68500 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:65440529 (62.4 MiB) TX bytes:67371829 (64.2 MiB) Interrupt:69 Link encap:ethernet HWaddr AC:16:2D:B0:29:74 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:101010 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6432234 (6.1 MiB) TX bytes:0 (0.0 b) Interrupt:73 Link encap:ethernet HWaddr AC:16:2D:B0:29:76 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:34843 errors:0 dropped:0 overruns:0 frame:0 TX packets:224 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2090656 (1.9 MiB) TX bytes:10704 (10.4 KiB) Interrupt:77 Link encap:ethernet HWaddr AC:16:2D:B0:29:76 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:34847 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2090820 (1.9 MiB) TX bytes:0 (0.0 b) Interrupt:81 Link encap:local Loopback inet addr:127.0.0.1 Mask:255.0.0.0
[root@laguna1:~]# inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:96832 errors:0 dropped:0 overruns:0 frame:0 TX packets:96832 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:23219019 (22.1 MiB) TX bytes:23219019 (22.1 MiB) ### AFTER [root@laguna1:~]# ifconfig bond0 Link encap:ethernet HWaddr AC:16:2D:B0:29:74 inet addr:10.200.160.211 Bcast:10.200.160.255 Mask:255.255.255.0 inet6 addr: fe80::ae16:2dff:feb0:2974/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:2291765 errors:0 dropped:0 overruns:0 frame:0 TX packets:611590 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:251820629 (240.1 MiB) TX bytes:5014093262 (4.6 GiB) bond0:1 bond0:3 bond0:4 bond1 bond1:1 eth0 eth1 Link encap:ethernet HWaddr AC:16:2D:B0:29:74 inet addr:10.200.160.212 Bcast:10.200.160.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 Link encap:ethernet HWaddr AC:16:2D:B0:29:74 inet addr:10.200.160.196 Bcast:10.200.160.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 Link encap:ethernet HWaddr AC:16:2D:B0:29:74 inet addr:10.200.160.197 Bcast:10.200.160.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 Link encap:ethernet HWaddr AC:16:2D:B0:29:76 inet addr:90.90.90.91 Bcast:90.90.90.255 Mask:255.255.255.0 inet6 addr: fe80::ae16:2dff:feb0:2976/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:1973966 errors:0 dropped:0 overruns:0 frame:0 TX packets:1436794 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1974215073 (1.8 GiB) TX bytes:1467319327 (1.3 GiB) Link encap:ethernet HWaddr AC:16:2D:B0:29:76 inet addr:169.254.79.25 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 Link encap:ethernet HWaddr AC:16:2D:B0:29:74 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:2093220 errors:0 dropped:0 overruns:0 frame:0 TX packets:611590 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:239175211 (228.0 MiB) TX bytes:5014093262 (4.6 GiB) Interrupt:69 Link encap:ethernet HWaddr AC:16:2D:B0:29:74 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:198545 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:12645418 (12.0 MiB) TX bytes:0 (0.0 b) Interrupt:73 eth2 eth3 lo Link encap:ethernet HWaddr AC:16:2D:B0:29:76 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1905257 errors:0 dropped:0 overruns:0 frame:0 TX packets:1436794 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1970081759 (1.8 GiB) TX bytes:1467319327 (1.3 GiB) Interrupt:77 Link encap:ethernet HWaddr AC:16:2D:B0:29:76 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:68709 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4133314 (3.9 MiB) TX bytes:0 (0.0 b) Interrupt:81 Link encap:local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:433336 errors:0 dropped:0 overruns:0 frame:0 TX packets:433336 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:236033817 (225.0 MiB) TX bytes:236033817 (225.0 MiB) [root@laguna1:~]# ### 2 scan names and vip is up in first node. [root@laguna1:~]# nslookup laguna1vip.turkcell.tgc Server: 10.210.154.150 Address: 10.210.154.150#53 Name: laguna1vip.turkcell.tgc Address: 10.200.160.212 [root@laguna1:~]# nslookup laguna-scan Server: 10.210.154.150 Address: 10.210.154.150#53 Name: laguna-scan.turkcell.tgc Address: 10.200.160.213 Name: laguna-scan.turkcell.tgc Address: 10.200.160.196 Name: laguna-scan.turkcell.tgc Address: 10.200.160.197 [root@laguna1:~]#
### The orher san ip and laguna2vip is up in laguna2 node. [root@laguna2:~]# nslookup laguna2vip.turkcell.tgc Server: 10.210.154.150 Address: 10.210.154.150#53 Name: laguna2vip.turkcell.tgc Address: 10.200.160.195 [root@laguna2:~]# ifconfig bond0 Link encap:ethernet HWaddr AC:16:2D:BC:8D:50 inet addr:10.200.160.194 Bcast:10.200.160.255 Mask:255.255.255.0 inet6 addr: fe80::ae16:2dff:febc:8d50/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:2831324 errors:0 dropped:0 overruns:0 frame:0 TX packets:1018048 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3541772648 (3.2 GiB) TX bytes:71026022 (67.7 MiB) bond0:1 bond0:2 bond1 bond1:1 eth0 eth1 eth2 Link encap:ethernet HWaddr AC:16:2D:BC:8D:50 inet addr:10.200.160.213 Bcast:10.200.160.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 Link encap:ethernet HWaddr AC:16:2D:BC:8D:50 inet addr:10.200.160.195 Bcast:10.200.160.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 Link encap:ethernet HWaddr AC:16:2D:BC:8D:52 inet addr:90.90.90.92 Bcast:90.90.90.255 Mask:255.255.255.0 inet6 addr: fe80::ae16:2dff:febc:8d52/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:1639471 errors:0 dropped:0 overruns:0 frame:0 TX packets:1803434 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1488826628 (1.3 GiB) TX bytes:1977180379 (1.8 GiB) Link encap:ethernet HWaddr AC:16:2D:BC:8D:52 inet addr:169.254.187.201 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 Link encap:ethernet HWaddr AC:16:2D:BC:8D:50 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:199106 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:12680250 (12.0 MiB) TX bytes:0 (0.0 b) Interrupt:71 Link encap:ethernet HWaddr AC:16:2D:BC:8D:50 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:2632219 errors:0 dropped:0 overruns:0 frame:0 TX packets:1018050 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3529092452 (3.2 GiB) TX bytes:71027162 (67.7 MiB) Interrupt:75 Link encap:ethernet HWaddr AC:16:2D:BC:8D:52
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1570562 errors:0 dropped:0 overruns:0 frame:0 TX packets:1803434 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1484681314 (1.3 GiB) TX bytes:1977180379 (1.8 GiB) Interrupt:79 eth3 lo Link encap:ethernet HWaddr AC:16:2D:BC:8D:52 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:68909 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4145314 (3.9 MiB) TX bytes:0 (0.0 b) Interrupt:83 Link encap:local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:333267 errors:0 dropped:0 overruns:0 frame:0 TX packets:333267 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:100537601 (95.8 MiB) TX bytes:100537601 (95.8 MiB) [root@laguna2:~]# Post Installation Steps --5.2.1 Back Up the root.sh Script for both nodes. [root@laguna1:~]# cp /u01/app/11.2.0/grid/root.sh /u01/orainst/root.shbackup_gi_inst --5.2.3.2 Creating the Fast Recovery Area Disk Group [grid@laguna1 grid]$ cd /u01/app/11.2.0/grid/bin [grid@laguna1 bin]$ [grid@laguna1 bin]$ [grid@laguna1 bin]$./asmca
A.7 About Using CVU Cluster Healthchecks After Installation [grid@laguna1 bin]$ cd /u01/orainst/cvu/bin [grid@laguna1 bin]$./cluvfy comp healthcheck -collect cluster -bestpractice deviations Verifying OS Best Practice Verifying Hardware Clock synchronization at shutdown...not met Verifying Clusterware Best Practice *********************************************************************************** ******* Summary of environment *********************************************************************************** ******* Date (mm/dd/yyyy) : 05/23/2013 Time (hh:mm:ss) : 16:40:45 Cluster name : laguna-cluster Clusterware version : 11.2.0.3.0 Grid home : /u01/app/11.2.0/grid Grid User : grid Operating system : Linux2.6.32-358.el6.x86_64 *********************************************************************************** ******* System recommendations *********************************************************************************** ******* Verification Check : Hardware Clock synchronization at shutdown Verification Description : Checks whether Hardware Clock is synchronized with the system clock during system shutdown Verification Result : NOT MET Verification Summary : Check for Hardware Clock synchronization at shutdown failed Node Status Expected Value Actual Value ----------------------------------------------------------------------------------- ------- laguna2 FAILED HWCLOCKSYNC=1 HWCLOCKSYNC=0 laguna1 FAILED HWCLOCKSYNC=1 HWCLOCKSYNC=0 [grid@laguna1 bin]$
[grid@laguna1 bin]$./cluvfy comp healthcheck -collect cluster -mandatory -save - savedir /u01/orainst/cvu/bin [grid@laguna1 bin]$./cluvfy comp healthcheck -collect cluster -mandatory -save - savedir /u01/orainst/cvu/bin cvucheckreport_5232013165416.txt