Monitoring Dell 9G Servers in Oracle GRID

Transcription

1 Monitoring Dell 9G Servers in Oracle GRID By Dave Jaffe Dell Enterprise Technology Center Dell Enterprise Technology Center dell.com/techcenter January 2007

2 Contents Executive Summary...3 Introduction...4 Installation and Configuration of Oracle Enterprise Manager...6 Installation...6 Patching...8 Accessing OEM...8 Installation of Agents on Target Servers...10 Preparation for Install...10 Remote Node Agent Installation...11 Installation and Configuration of Dell OpenManage Server Administrator...12 OMSA Installation...12 SNMP Configuration...13 Checking Integration of OMSA and OEM...13 Configuration and Testing of Alert Notifications and Corrective Actions...15 Basic Monitoring of Dell Hardware in OEM...15 Configuring Alert Notification for Hardware in OEM...16 Conclusions...19 Acknowledgements...20 Tables Table 1: Oracle Software Installed... 6 Table 2: Oracle Grid Control Installation Parameters... 7 Table 3: Oracle Grid Control Configuration Parameters... 7 Figures Figure 1: Block Diagram of Systems in Demo... 5 Figure 2: Oracle Enterprise Manager Targets Screen Figure 3: Oracle Enterprise Manager Metrics with Dell OpenManage Figure 4: Oracle Enterprise Manager Group Dashboard Figure 5: Oracle Enterprise Manager Group Showing Temperature Alert January 2007 Page 2 Dell Enterprise Technology Center

3 Section 1 Executive Summary Oracle Enterprise Manager Grid Control enables Oracle system and database administrators to manage large Oracle Grids. Oracle and Dell have recently worked together to integrate Oracle Enterprise Manager with Dell s OpenManage software to provide Dell hardware-specific data within Oracle Enterprise Manager. The server hardware metrics can be monitored and managed just like Oracle database metrics. This integration of Oracle Enterprise Manager and Dell OpenManage provides a powerful tool for managing Oracle Grid on Dell PowerEdge servers within a single console. The simplification stemming from a single console, with its single log for hardware, operating system and application exceptions, is a key value proposition of Dell s Scalable Enterprise. Through concrete examples this paper documents best practices in the installation and configuration of Oracle Enterprise Manager and Dell OpenManage, including detailed instructions on how to monitor and manage alerts coming from the servers. January 2007 Page 3 Dell Enterprise Technology Center

4 Section 2 Introduction To help customers manage large Oracle Grids running on Dell PowerEdge servers, Dell and Oracle have integrated their management tools. This integration adds the capability of handling hardware-based data emanating from Dell s OpenManage server monitoring tools within the Oracle Enterprise Manager Grid Control console. By exposing this data within the main Oracle management console, Dell and Oracle provide system and database administrators a single point of operations, thereby eliminating the need to learn and monitor multiple consoles. After the installation and configuration of the Dell OpenManage Server Administrator software on each Dell server to be managed, the hardware-specific data appears in Oracle Enterprise Manager alongside all the Oracle-specific items and is managed identically. As part of Dell s initiative to simplify system management by driving industry standards, the interface between Oracle Enterprise Manager and Dell OpenManage Server Administrator is accomplished through the Simple Network Management Protocol (SNMP). This standards-based approach means that no other Dell hardware or software components are required. An Oracle Enterprise Manager console coupled with OpenManage-enabled Dell PowerEdge servers provides the following benefits: Discovery & Reporting of Dell Servers Hardware Event Notification Event-driven Policy Execution Server Asset and Inventory This white paper documents best practices for the configuration of Oracle Enterprise Manager and Dell OpenManage Server Administrator and demonstrates how the combination can simplify server management. As seen in Error! Reference source not found., for this demo we installed the Oracle Enterprise Manager console on a PowerEdge 1950 and installed Dell OpenManage Server Administrator and the Oracle Enterprise Manager agent, emagent, on the PowerEdge 1950 as well as two PowerEdge 2950 nodes of an Oracle RAC cluster. OpenManage Server Administrator communicates with emagent using SNMP. The emagent software in turn communicates with the central Oracle Enterprise Manager console via secure HTTP. The systems used for this study are running Red Hat Enterprise Linux 4 but the same integration works on other Linux distributions and Windows Server by following the same basic steps. The installation and configuration of the Oracle Enterprise Manager Grid Control console is described in the next section, followed by details on how to install the Oracle Enterprise Manager agent, emagent, on target servers in Section 4. Section 5 describes the installation of Dell OpenManage Server Administrator and Section 6 shows how to configure and use the Dell hardware alerts within Oracle Enterprise Manager. January 2007 Page 4 Dell Enterprise Technology Center

5 Figure 1: Block Diagram of Systems in Demo January 2007 Page 5 Dell Enterprise Technology Center

6 Section 3 Installation and Configuration of Oracle Enterprise Manager In this section we will cover the installation and patching of Oracle Enterprise Manager, and show the first steps in how to use it. The commands as shown were tested with Red Hat Enterprise Linux 4. They will need to be slightly varied for other Linux distributions and Windows. The steps here have been distilled from several Oracle manuals. Please see the complete Oracle documentation available at for details. Installation Oracle Enterprise Manager (OEM) is an additional component used by most enterprise Oracle users, and requires additional licenses. The current version is based on Oracle 10g R2 32b, which must be installed on a 32b operating system. For our management system we used Red Hat Enterprise Linux 4 U3 AS 32b edition (RHEL4). The operating system must be configured for a standard Oracle 10g R2 32b installation, with the proper packages installed, kernel parameters set, and users, groups and directories defined as per the Oracle Enterprise Manager 10GR2 Grid Control Quick Installation Guide for Linux x86 at To prepare RHEL4 for installation of OEM on our PowerEdge 1950 (r3rhel1950a) we followed this quick installation guide, including defining user oracle as a member of groups oinstall and dba, and creating directory /opt/oracle with owner oracle.oinstall. The Oracle Enterprise Manager Grid Control files, including agent software for 64b systems and Patch must be downloaded from Oracle. The full names of the five files used are shown in Table 1. Linux_Grid_Control_full_102010_disk1.zip Linux_Grid_Control_full_102010_disk2.zip Linux_Grid_Control_full_102010_disk3.zip Linux_x86_64_Grid_Control_agent_download_10_2_0_2_0.zip p _10202_linux.zip Table 1: Oracle Software Installed January 2007 Page 6 Dell Enterprise Technology Center

7 Run the Oracle Enterprise Manager installer in an X-Window shell as user oracle in the directory where the grid control files were unzipped. First, run the Linux command export ORACLE_BASE=/opt/oracle Then execute./runinstaller and specify the parameters shown in Table 2: Parameter Installation Type Installation Location Inventory directory and credentials Value Enterprise Manager 10g Grid Control Using a New Database Parent Directory:/opt/oracle/OracleHomes (default) Path: /opt/oracle/orainventory (default) OS group name: oinstall (default) Table 2: Oracle Grid Control Installation Parameters After all the Product-Specific Prerequisite Checks succeed enter the configuration information shown in Table 3: Parameter Repository Database Name Repository Database File Location Database Administrator (OSDBA) group Database Operator (OSOPER) group Value emrep (default) /opt/oracle/oradata (default) dba oinstall (default) Table 3: Oracle Grid Control Configuration Parameters Skip the Specify Optional Configuration step and set your passwords in Specify Security Options (for this demo we used oracle1 for all passwords). Finally, issue the Install command. When finished, run the configuration scripts as instructed, accepting the default answer to all questions. After the configuration scripts finish, return to the main installation screen. Various configuration assistants will run. We found that the Agent Configuration Assistant failed despite repeated attempts, so we ignored it at that point. It will be configured in a later step. After the configuration assistants have run, return to the Installation screen, select End and then Exit. In our case there was an information message that included the following info: Use the following URL to access the Enterprise Manager Grid Control: January 2007 Page 7 Dell Enterprise Technology Center

8 Patching Next it is necessary to patch the Oracle software you just installed. To apply patch , first make sure the repository database (emrep) is up. As oracle: export ORACLE_SID=emrep /opt/oracle/oraclehomes/db10g/bin/lsnrctl status /opt/oracle/oraclehomes/db10g/bin/lsnrctl start # if necessary /opt/oracle/oraclehomes/db10g/bin/sqlplus "/ as sysdba" # if connected to idle instance: startup Then you must stop the management server. As root: /opt/oracle/oraclehomes/oms10g/opmn/bin/opmnctl stopall Next, stop the enterprise manager agent running on the server. As oracle: /opt/oracle/oraclehomes/agent10g/bin/emctl stop agent You will need to install the patch 4 times for the Oracle Configuration Manager (OCM) software, the Oracle Management Server (OMS), DB Repository and Agent. In our installation we declined the OCM license agreement, which resulted in the software being installed but not started (it is not needed for this demo). Before installing the patch, run the Linux command unset ORACLE_HOME and then run the command./runinstaller from the directory where the patch was unzipped four times, specifying as Oracle Home in turn OCM, oms10g, db10g and agent10g. Accessing OEM Now you are ready to start up OEM. As root: /opt/oracle/oraclehomes/oms10g/opmn/bin/opmnctl startall At this point you can access Oracle Enterprise Manager Grid Control: Login: sysman/oracle1 To start OEM after reboot use the following process: - As oracle: export ORACLE_SID=emrep /opt/oracle/oraclehomes/db10g/bin/lsnrctl start /opt/oracle/oraclehomes/db10g/bin/sqlplus "/ as sysdba" SQL> startup SQL> quit January 2007 Page 8 Dell Enterprise Technology Center

9 - As root: /opt/oracle/oraclehomes/oms10g/opmn/bin/opmnctl startall - As oracle: /opt/oracle/oraclehomes/agent10g/bin/emctl start agent In the next section we will discuss how to install the Oracle Enterprise Manager agent, emagent, on remote servers. January 2007 Page 9 Dell Enterprise Technology Center

10 Section 4 Installation of Agents on Target Servers Preparation for Install To be able to manage remote Oracle systems (known as target systems to Oracle Enterprise Manager) we first need to install the OEM agent, emagent, on the target systems. In this section we will demonstrate how that is done using a 2-node RAC cluster consisting of two PowerEdge 2950s running RHEL4 64b, r3rhel2950a and r3rhel2950b. The RAC database instance is r3rac1. After installation of OEM on the OEM host, r3rhel1950a, as described in Section 3 the directory <OMS_HOME>/sysman/agent_download/ (where <OMS_HOME> = /opt/oracle/oraclehomes/oms10g) contains subdirectories for linux and win32. To avoid an installation error it necessary to fix a file that was already installed, <OMS_HOME>/sysman/agent_download/ /agent_download. rsp by changing s_omshost="r3rhel1950a.setc.com\:4889 Management_Service" to s_omshost="r3rhel1950a.setc.com" The nodes that we are adding are Linux 64b hosts and the 64b Linux agent is not included by default. To add the Linux 64b agent to the agents available use the following command: cd <OMS_HOME>/sysman/agent_download/ unzip <path>/linux_x86_64_grid_control_agent_ download_10_2_0_2_0.zip This creates a new subdirectory, linux_x64 Make the following changes in <OMS_HOME>/sysman/agent_download/ /linux_x64/agentDownload.linux_x64: OMSHost="r3rhel1950a.setc.com" httpport="4889" and copy it to a directory owned by oracle (we created /home/oracle/10g/agent for the purpose) on each target (only copy to one node of a cluster). On each target, check that user oracle has wget and jar in its path. If jar is not present, get the Java Developers Kit for Linux 64 from java.sun.com (we used jdk-1_5_0_06-linux-amd64-rpm.bin) and install to each target. January 2007 Page 10 Dell Enterprise Technology Center

11 Remote Node Agent Installation As oracle, run the agent installation script on each target to be managed. For the two node cluster, run the following command on one node only:./agentdownload.linux_x64 -b /opt/oracle/product/ c "r3rhel2950a,r3rhel2950b" -n r3rac1 Enter oracle1 password when prompted When prompted, as root on each node, run the following script, accepting all defaults: /opt/oracle/product/10.2.0/agent/agent10g/root.sh This will create a new agent home at /opt/oracle/product/10.2.0/agent10g. To verify that the agent is running, as oracle on each target run: /opt/oracle/product/10.2.0/agent10g/bin/emctl status agent Upon successful completion of the emagent installation on the targets, after a few minutes the Oracle Enterprise Manager screen should look like Figure 2. Figure 2: Oracle Enterprise Manager Targets Screen In the next section we will show how to install OpenManage Server Administrator on each system and get it to talk to Oracle Enterprise Manager via emagent. January 2007 Page 11 Dell Enterprise Technology Center

12 Section 5 Installation and Configuration of Dell OpenManage Server Administrator In this section we will show how to install Dell OpenManage Server Administrator (OMSA) and how to configure Simple Network Management Protocol (SNMP) to get OMSA to interface with the Oracle Enterprise Manager agent, emagent, running on each system. These instructions are for installation on the system that is also the OEM host, r3rhel1950a, but they apply to all systems to be managed by OEM. The OMSA documentation is collected at OMSA Installation To prepare for OMSA installation, as root check that the snmpd agent is installed. Your command and output should look something like this: rpm -qa grep -i snmp net-snmp-libs el4.6 net-snmp el4.6 Note: OMSA installation will modify the snmp configuration file, /etc/snmp/snmpd.conf, so copy it to a different name before the next step. Download the OMSA software for Linux (we used OM_5.1_ManNode_LIN_A00.tar.gz) from support.dell.com (select Drivers & Downloads for the PowerEdge 1950, then Enterprise Linux 4, then Systems Management). To install OMSA 5.1: Create a directory, say /root/srvadmin, copy OM_5.1_ManNode_LIN_A00.tar.gz to it, cd to it, then gunzip OM_5.1_ManNode_LIN_A00.tar.gz tar xvf OM_5.1_ManNode_LIN_A00.tar cd linux/supportscripts/./srvadmin-install.sh --express./srvadmin-services.sh start This will install OMSA 5.1 to /opt/dell/srvadmin. If you get dependency errors, install necessary rpms from /root/srvadmin/linux/rpms/supportrpms/rh4_x86_64. The URL to access the OpenManage Server Administrator is the hostname with port 1311, in our case Or, from a Linux shell, omreport about Product name : Server Administrator Install Core (subscription) Version : Copyright : Copyright (C) Dell Inc All January 2007 Page 12 Dell Enterprise Technology Center

13 rights reserved. Company : Dell Inc. Other omreport options include: omreport server summary omreport chassis fans omreport chassis temp (use omreport -? to drill down into commands) The web access through port 1311 is an installation option that may be omitted for security or performance reasons if not required (it is not needed to integrate OMSA with OEM). SNMP Configuration Installing OMSA adds these lines to snmpd.conf to allow OMSA to communicate via SNMP to other services: # Allow Systems Management Data Engine SNMP to connect to snmpd using SMUX smuxpeer To enable OMSA to communicate with the emagent running on that server, add the localhost IP address to snmpd.conf: rocommunity public Finally, restart the snmpd service (if this doesn t work, reboot): service snmpd restart To make the snmpd process start up upon boot, as root chkconfig --level 35 snmpd on Note that the use of the localhost IP address restricts the services that can access OMSA to services running on that server, most importantly, emagent. This is very important to prevent external access to the critical hardware-level features provided by OMSA, although the system may still be configured to pass SNMP traps to other destinations without compromising security. Also note that while OMSA uses SNMP to support remote inventory, it does not support remote configuration (SNMP SET or "write" actions). All remote configuration of OMSA is done through secure channels (HTTPS). The interface between emagent and the OEM console is through secure HTTP. Checking Integration of OMSA and OEM You should now be able to see all the Dell-specific hardware metrics in Oracle Enterprise Manager. Bring up OEM using the URL shown in the previous section ( login, select Targets, and drill down into the OEM host system and select All Metrics near the bottom of the screen. You should see a display similar to Figure 3. Note that the following metrics are Dellspecific and come through OMSA: Fans, Memory Devices, Power Supplies, Processors, Remote Access Card, System BIOS and Temperature. January 2007 Page 13 Dell Enterprise Technology Center

14 Figure 3: Oracle Enterprise Manager Metrics with Dell OpenManage January 2007 Page 14 Dell Enterprise Technology Center

15 Section 6 Configuration and Testing of Alert Notifications and Corrective Actions Basic Monitoring of Dell Hardware in OEM Now that we have the Oracle Enterprise Manager host and a few remote target systems managed in our OEM console let s see how to monitor and manage Dell hardware data from those systems. OEM manages Dell hardware data through a set of critical and warning alerts, and can be configured to take actions or provide notifications in certain situations, allowing data center employees to be freed up from watching screens to do more productive activities. First, let s read the temperature of one of our servers through OEM and compare the result using the OMSA omreport command on that server: OEM Console: Targets->r3rhel2950a->All Metrics->Temperature ->Current Temperature Last Known Value: 30 In shell on r3rhel2950a: omreport chassis temps Temperature Probes Information Main System Chassis Temperatures: Ok Index : 0 Status : Ok Probe Name : System Board Ambient Temp Reading : 30.0 C Minimum Warning Threshold : 8.0 C Maximum Warning Threshold : 42.0 C Minimum Failure Threshold : 3.0 C Maximum Failure Threshold : 47.0 C Notice that the two methods report the same value! The maximum and minimum warning and failure thresholds are used by OMSA to send out-of-bounds parameter alerts up through OEM. To see this you will first want to lower the parameter collection schedule from 15 minutes to 1 minute: Targets->r3rhel1950a->Metric and Policy Settings ->Click on "Collection Schedule" entry for Temperature Probe Status Collection Frequency: Frequency Type: By Minutes Repeat Every: 1 minutes Use of Metric Data: Alerting and Historical Trending, Upload Interval: 1 Collections January 2007 Page 15 Dell Enterprise Technology Center

16 Note that modifying collection schedules will impact more than one metric at a time since the collection schedules manage a group of metrics. Now we can force an alert by intentionally mis-setting the temperature Maximum Warning Threshold to 25 (which is lower than the current temperature). As root in a Linux shell: omconfig chassis temps index=0 maxwarnthresh=25 This should force a warning alert on the OEM console within a minute. You should see the yellow (warning) number under Alerts for the r3rhel2950a in Figure 2 change to 1. At this point you can drill down into the Alert Metric and display a graph of the parameter over time. After the problem is fixed the alert should disappear. Configuring Alert Notification for Hardware in OEM What if you are not in front of the screen when the alert occurs? To have the alert sent to one or more system administrators via or text message it is necessary to first add the /text message address, then create a new Notification Rule that uses it: Preferences->General-> addresses->add Another Row Select: check address: @cingularme.com Message Format: Short Format, Apply Preferences->Notification Rules->select Host Availability and Critical States->Create Like General Name: Temp_notification_rule Description: Notification rule for monitoring host chassis temperature. Select Make Public Target Type: Host, Apply rule to all Host targets Availability: unselect all Metrics: Select All, Remove Add Temperature Probe Status Select Critical, Warning, Continue Methods: notification, select Send Me Now try setting off the temperature alert and you should get an /text message at the address you specified. Furthermore, OEM can respond to an alert with a user-definable Corrective Action. In our example the corrective action is to merely reset the temperature warning threshold: Setup->Corrective Action Library->Create Library Corrective Action->select OS Command from dropdown->go Create "OS Command" Corrective Action General: Name: fix_temp Description: Reset temperature warning threshold Parameters: January 2007 Page 16 Dell Enterprise Technology Center

17 Command Type: Single Operation Command: /usr/bin/omconfig chassis temps index=0 warnthresh=default Credentials: Override Preferred Credentials, enter root/password Save to Library Targets->r3rhel2950a->Metric and Policy Settings->Edit (right hand column) Temperature Probe Status Monitored Objects: select, Edit Edit Advanced Settings: Edit->Corrective Action(Warning)->Add Add Corrective Action->select From Library from dropdown->continue Select fix_temp, Continue Type in name (you can reuse fix_temp), Continue, Continue, Continue, OK, OK Figure 4: Oracle Enterprise Manager Group Dashboard Now when you intentionally mis-set the temperature warning threshold, an alert will appear, which will trigger the correction, so the alert will disappear about a minute later. Finally, another useful thing to do is create a Group to display the most critical targets in a single screen: Targets->Groups->Add Name: Group1 Members: Add->add desired hosts and databases->select->ok January 2007 Page 17 Dell Enterprise Technology Center

18 Select Group1, Configure->Dashboard->Refresh Frequency: 1 minute Groups->Group1->Launch Dashboard A group containing the three hosts is shown before and after an alert is received in Figure 4 and Figure 5. The screen would turn red in case the alert was critical. Figure 5: Oracle Enterprise Manager Group Showing Temperature Alert January 2007 Page 18 Dell Enterprise Technology Center

19 Section 7 Conclusions The addition of Dell PowerEdge server metrics to the Oracle Enterprise Manager Grid Control console provides Oracle system and database administrators a single place to monitor and manage the hardware and software comprising large Oracle Grids. As shown in this paper, the integration of Dell OpenManage Server Administrator and Oracle Enterprise Manager is easy to do through the industry standard Simple Network Management Protocol. The gain in simplicity due to a single console and a single log, together with the productivity boost due to automatic notifications or actions are key advantages in running Oracle on Dell PowerEdge servers. For further reference, the use of Dell server data inside OEM is documented in the Oracle Enterprise Manager Concepts manual ( d-party_management.htm#sthref1463) and the Oracle Metric Reference Manual ( 435). THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Dell and PowerEdge are trademarks of Dell Inc. EMC, Navisphere and PowerPath are registered trademarks and Access Logix is a trademark of EMC Corp. Intel and Xeon are registered trademark of Intel Corp. Red Hat is a registered trademark of Red Hat Inc. Linux is a registered trademark of Linus Torvalds. Oracle is a registered trademark of Oracle Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others. Copyright 2006 Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Information in this document is subject to change without notice. January 2007 Page 19 Dell Enterprise Technology Center

20 Section 8 Acknowledgements The author would like to acknowledge Logan McLeod of Dell IT for several good tips on Oracle Enterprise Manager and Todd Muirhead of the Dell Scalable Enterprise Technology Center for many valuable discussions on Oracle. January 2007 Page 20 Dell Enterprise Technology Center