Using High Availability Monitors Manufacturing Part Number: B E0603

Transcription

1 Using High Availability Monitors Manufacturing Part Number: B E0603 Copyright 1997, 2003 Hewlett-Packard Company.

2 Legal Notices The information contained in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Warranty A copy of the specific warranty terms applicable to your Hewlett-Packard product and replacement parts can be obtained from your local Sales and Service Office. Restricted Rights Legend Use, duplication or disclosure by the U.S. Government is subject to restrictions as set forth in subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer Software clause at DFARS for DOD agencies, and subparagraphs (c) (1) and (c) (2) of the Commercial Computer Software Restricted Rights clause at FAR for other agencies. HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P S.H. 249 Houston, Texas Use of this document and any supporting software media supplied for this pack is restricted to this product only. Additional copies of the programs may be made for security and back-up purposes only. Resale of the programs, in their present form or with alterations, is expressly prohibited. 2

3 Copyright Notice Copyright Hewlett-Packard Development Company, L.P. All rights reserved. Reproduction, adaptation, or translation of this document without prior written permission is prohibited, except as allowed under the copyright laws. High Availability Monitors, Event Monitoring Service, HP OpenView, HP OpenView IT/Operations, ServiceGuard Extension for RAC, and MC/ServiceGuard are products of Hewlett-Packard Development Company L.P., and all are protected by copyright. Trademark Notices X Window System is a trademark of the Massachusetts Institute of Technology. MS-DOS and Microsoft are U.S. registered trademarks of Microsoft Corporation. OSF/Motif is a trademark of the Open Software Foundation, Inc. in the U.S. and other countries. 3

4 4

5 Contents 1. Understanding the Event Monitoring Service Event Monitoring Service Overview EMS Requirements EMS Resource Classes Client and Target Applications EMS with ServiceGuard EMS GUI Client Application EMS CLI Client Application EMS and Target Applications Resource Monitors Installing Resource Monitors Configuring Resource Monitors Writing Resource Monitors EMS Framework Components The EMS API The Registrar The Resource Dictionary Monitoring Disk Resources HA Disk Monitor Reference Physical Volume Summary Physical Volume and Physical Volume Link Status Logical Volume Summary Logical Volume Status Logical Volume Number of Copies Rules for Using the HA Disk Monitor with ServiceGuard Setting Failover Parameters Working with Physical Volume Groups Rules for RAID Arrays Adding PVGs to Existing Volume Groups Creating Volume Groups on Disk Arrays Using PV Links Creating Logical Volumes Rules for Mirrored Individual Disks Creating Disk Monitoring Requests Disk Monitoring Request Suggestions Resources to Monitor for RAID Arrays Resources to Monitor for Mirrored Disks Resources to Monitor for Lock Disks

6 Contents Resources to Monitor for Root Volumes Excluding Volume Groups from being Monitored Monitoring Database Resources Database Monitor Reference Database Resources Server Resources Rules for Using the HA Database Monitor with ServiceGuard Setting Failover Parameters Sample File Settings Setting Up the ServiceGuard Package ServiceGuard Sample Run/Halt Scripts Creating Database Monitoring Requests HA Database Monitor Command-Line Options HA Disk Monitor Command-Line Options General MIB Monitor Troubleshooting Database Monitor Troubleshooting for Oracle Database Monitor Troubleshooting for Informix Debug Logging of EMS HA Monitors Troubleshooting 0 Byte snmpd.conf Problem Steps to Obtain EMS Data to Reproduce an EMS Problem

7 Table 1 Printing History Printing Date Part Number Edition August 1997 B Edition 1 October 1998 B Edition 2 March 1999 B Edition 3 May 1999 B Edition 4 August 1999 B Edition 5 November 1999 B Edition 6 June 2003 B Edition 7 This edition documents configuring High Availability Monitors. The printing date changes when a new edition is printed. (Minor corrections and updates are incorporated at reprint and do not cause the date to change.) The part number is revised when technical changes are incorporated. New editions of this manual incorporate all material updated since the previous edition. HP Printing Division: Business Critical Computing Business Unit (BCC) Hewlett-Packard Co Pruneridge Ave. Cupertino, CA

8 8

9 Preface This guide describes how to install and configure the High Availability Monitors to monitor system health. The chapters are as follows: Understanding the Event Monitoring Service, which describes the Event Monitoring Service components and operations, including the role of High Availability Monitors. Monitoring Disk Resources, which provides guidelines on using the disk monitor, including using it with MC/ServiceGuard or ServiceGuard OPS Edition. Monitoring Database Resources, which provides guidelines on using the database monitor. Audience HP intends this manual for system managers or administrators responsible for configuring and maintaining the High Availability Monitors (HA Monitors) on HP-UX 11i Versions 1 and 2. This manual is based on the assumption that you have: An understanding of distributed network concepts and client-server computing Demonstrated knowledge of UNIX An understanding of EMS and HA Monitors basics Related Publications The following documents contain additional related information: Using the Event Monitoring Service (HP Part Number B ) EMS Hardware Monitors User s Guide (HP Part Number B ) Managing MC/ServiceGuard (HP Part Number B ). Configuring OPS Clusters with ServiceGuard OPS Edition (HP Part Number ). Peter Weygant, Clusters for High Availability: A Primer of HP-UX Solutions (ISBN ). HP Press: Prentice Hall, Inc.,

10 Tom Madell, Disk and File Management Tasks on HP-UX (ISBN X). HP Press; Prentice Hall, Inc., Managing Systems and Workgroups (HP Part Number B ) HP OpenView IT/Operations Admnistrator s Reference (HP Part Number B ) Managing Highly Available NFS (HP Part Number B ) Website for information about Hewlett-Packard s high-availability technologies where you can find documents such as Writing Monitors for the Event Monitoring Service (EMS) (HP Part Number B ). Select HP-UX then High Availability. Website for designing and building an EMS monitor. Select High Availability then Event Monitoring Service Developer s Kit. Problem Reporting If you have any problems with the software or documentation, please contact your local Hewlett-Packard Sales Office or Customer Service Center. 10

11 Understanding the Event Monitoring Service 1 Understanding the Event Monitoring Service This document, Using High Availability Monitors, describes how to configure high availability monitors. The chapters in this book are specific to each HA Monitor and describe the options, settings, and provide suggestions for configuring your HA Monitor. HA Monitors is part of a total Event Monitoring Service. This chapter describes the components of the Event Monitoring Service. This chapter contains the following sections: Event Monitoring Service Overview EMS Requirements EMS Resource Classes Client and Target Applications Resource Monitors EMS Framework Components Chapter 1 11

12 Understanding the Event Monitoring Service Event Monitoring Service Overview Event Monitoring Service Overview The Event Monitoring Service (EMS) monitors system resources. EMS is used by system administrators to configure monitoring requests, check resource status, and send notification when configured conditions are met. EMS can work in a high availability environment. It can report a loss of redundant resources. Identifying and reporting single points of failure helps maintain a proactive approach to preventing the loss of data and availability. EMS only observes a system, and does not modify the system. Use EMS with additional software to take or specify action. The three basic components of EMS are: Client and Target Applications System administrators use client applications to set, modify, or remove monitoring requests. System administrators use target applications to receive event notifications and possibly take actions. Client applications include ServiceGuard (MC/ServiceGuard or ServiceGuard Extension for RAC), the Event Monitoring Service (EMS) GUI (Graphical User Interface), the Event Monitoring Service CLI (Command Line Interface) or other applications that comply with the EMS API. The target application can be any application that supports the EMS protocols. The supported protocols are: TCP/IP or UDP/IP This includes any application that accepts these protocols and follows the rules defined in the EMS Developer s Kit. opcmsg method (for ITO) This option is used for IT/Operations notifications. SNMP traps This option can be used with any application that accepts SNMP traps, such as NNM or IT/O. You need to set up the application to recognize the SNMP traps generated. 12 Chapter 1

13 Understanding the Event Monitoring Service Event Monitoring Service Overview This option does not require any extra handling. Specify the address when the monitoring request is created. syslog and textlog This option does not require any extra handling. Specify the log file when the monitoring request is created. Syslog notifications go to the local system. console This option does not require any extra handling. Specify the console when the monitoring request is created. Notifications go to the local system. ServiceGuard This option requires that the client and target application both be ServiceGuard running on the same local system. Resource Monitors Resource monitors observe designated resources and report back resource values or events to the Event Monitoring Service. Hewlett-Packard provides monitors with the High Availability Monitors package and with the Event Monitoring Service. The monitors available through Hewlett-Packard include: HA Database Monitor, HA Disk Monitor, HA Cluster Monitor, HA Network Interface Monitor, and HA System Resource Monitor. Event Monitoring Service Framework The EMS framework provides the interface between the client applications, monitors, and target applications. The EMS framework contains the Applications Programmers Interface (API), registrar, and the Resource Dictionary. Developers use the API to create additional monitors for use with client and target applications, such as the EMS GUI, EMS CLI or ServiceGuard. Monitor components to be created include: resource dictionary, resource monitor binary file, manpage (recommended), and message catalog (recommended). Chapter 1 13

14 Understanding the Event Monitoring Service Event Monitoring Service Overview Figure 1-1 shows the relationships between the Event Monitoring Service components. Figure 1-1 Event Monitoring Service Components The process is as follows: 1. The system administrator enters the client application, for example, the EMS GUI, or the EMS CLI, to begin the discovery phase of creating a monitoring request. The discovery phase, includes identifying the resources to be monitored and configuring the request. It can be accomplished through many methods, including: EMS GUI EMS CLI ServiceGuard monconfig utility resls or resdata commands 14 Chapter 1

15 Understanding the Event Monitoring Service Event Monitoring Service Overview 2. The EMS API provides the interface between the client request and the registrar. There is a one to one correspondence between the client and registrar. 3. The registrar refers to the dictionary for a list of available resources and related monitors. The resources listed in the dictionary are passed back to the client. 4. When a discovery request is made that exceeds the scope of the information in the dictionary, the registrar launches the appropriate resource monitor application, if it is not already running, and passes the request on to the monitor. Multiple registrars may access the same monitor. 5. The EMS API provides the interface between the registrar and the monitor. 6. The monitor identifies the resources. The list of resources is passed back through the registrar to the client requestor. 7. The system administrator, through the client application: Continues to drill down through the list of available resources supplied by the registrar, dictionary, and monitor Identifies the resources to monitor Completes the monitoring request defines conditions of where and how to send the event notification A completed monitoring request identifies: What resources to monitor What events to watch for and how often What notifications to send when an event occurs Where to send notifications Events are defined for either of two resource state types: Periodic checking against either thresholds or state/value changes Continuous checking for asynchronous stateless events 8. The registrar passes completed monitoring requests down to the appropriate resource monitor application. Chapter 1 15

16 Understanding the Event Monitoring Service Event Monitoring Service Overview 9. The monitor checks the resource as specified in the monitor request. It passes back to the EMS API whether the request is accepted or rejected. 10. The EMS API provides the interface between the monitor and the target. 11. The monitor begins collecting data as specified in the monitoring request. 12. The EMS API interprets the information received from the monitor, determines if an event occurred, and forwards the notification to the target applications. The method of informing the target application of a critical resource value can vary for different target applications. In the case of ServiceGuard, the client application and the target application are the same and reside on the same system. 16 Chapter 1

17 Understanding the Event Monitoring Service EMS Requirements EMS Requirements The following are system requirements for the Event Monitoring Service: All hardware you intend to monitor, such as disks and LAN cards, have been configured and tested prior to configuring EMS. EMS must be installed on an HP 9000 Series 700 or Series 800 system running HP-UX version or later. When installing one or more EMS components, check that the version levels for the other components are compatible. Refer to the recent Release Notes for each component, such as EMS, HA Monitors, or ServiceGuard. NOTE For more information on system requirements, refer to the EMS and HA Monitors Release Notes. 1. Go to the Website: 2. From the Website, select HP-UX, then High Availability, then scroll to identify and select the latest Release Notes for each component. Chapter 1 17

18 Understanding the Event Monitoring Service EMS Resource Classes EMS Resource Classes EMS groups resources into classes in a hierarchy similar to that of a file system structure. Figure 1-2 is a example of a resource hierarchy. Figure 1-2 Event Monitoring Service Resource Class Hierarchy 18 Chapter 1

19 Understanding the Event Monitoring Service Client and Target Applications Client and Target Applications This section describes some of the client and target application options and processes. Target applications can be written using the EMS API. EMS with ServiceGuard ServiceGuard can be configured with EMS to monitor the health of selected resources, such as disks. Based on the status of the resources, ServiceGuard can decide to fail packages over. When working with EMS, ServiceGuard acts as both the client and target application. EMS works with both the MC/ServiceGuard and ServiceGuard Extension for RAC. Configure EMS requests for use with ServiceGuard packages by editing the package configuration ASCII file. In addition, it is recommended that you also create requests through EMS to: Enable a redundant notification system Monitor events that affect high availability Enable alerts due to a package failover ServiceGuard may already be configured to monitor the health of nodes, services, and subnets, and to make failover decisions based on the status of these resources. Using EMS with ServiceGuard adds to the set of failures or events that trigger failover and affect availability. EMS GUI Client Application The EMS GUI client application found in the Resource Management area of SAM is used to create monitoring requests for resources and targets. The EMS GUI starts from the graphical version of SAM. Click through and select from the various screens to define your monitoring request. The options include: 1. Select the resources to be monitored. The full path of a resource includes the resource class hierarchy and instance. An example of a full resource path for the physical volume status of the device /dev/dsk/c0t1d2 belonging to volume group vgdatabase, is /vg/vgdatabase/pv_pvlink/status/c0t1d2. Chapter 1 19

20 Understanding the Event Monitoring Service Client and Target Applications 2. Specify when to collect the value. Select either and/or all: When value is... If you are setting up a request for an asynchronous monitor, this is the only option available. When value changes At each interval Select this option to send an event periodically, regardless of the value. Define a polling interval that is appropriate to your system performance and reaction time needs. See Step Specify a polling interval for how often the monitor checks the resource and reports the value. This applies only to non-asynchronous monitors and goes with the At each interval option in Step Specify how often the monitor should check and send notification about the resource: The options are listed below. Initial option immediately checks and returns the resource value regardless of threshold conditions Repeat option checks and returns the resource value at each polling interval if threshold conditions have been met Return option checks and returns the resource value after a threshold condition has been resolved and the threshold condition is not longer true. 5. Specify the notification protocols: opcmsg (IT/O), by severity or map severity from values listed tcp or udp snmp trap, by severity or map severity from values listed console syslog 20 Chapter 1

21 Understanding the Event Monitoring Service Client and Target Applications textlog EMS CLI Client Application emscli is a command line utility that is used to configure and manage persistent monitoring requests for Event Monitoring Service (EMS) monitors, such as, HA Monitors, Hardware Monitors and Kernel Monitors. The emscli utility can be used to add, modify, delete, list and view monitoring requests and resources. It also allows the user to generate scripts of the configured requests. The emscli utility also has a help feature (emscli -h) to view the usage of various options supported by emscli. For more information on the command line options supported by emscli, refer to the emscli (1m) manpage. NOTE emscli should be used on HP-UX versions where the EMS GUI is not available (HP-UX 11i Version 1.6 onwards). EMS and Target Applications Target applications receive notification messages about the monitored resources. To help configure your Network Node Manager and IT/Operations or other system management software for EMS, refer to the Writing Monitors for the Event Monitoring Service (EMS) (HP Part Number B )) developer s kit web page: 1. Go to the Website: 2. From the Website, select High Availability, then select Event Monitoring Service Developers Kit. 3. Select Templates. Chapter 1 21

22 Understanding the Event Monitoring Service Resource Monitors Resource Monitors Resource monitors are applications written to gather and report information about specific resources on the system. The resource monitor: Provides a list of resources that can be monitored Provides information about the resources Monitors the resources it supports Provides values to the EMS API notification The EMS framework evaluates the data to determine if an event has occurred. If an event has occurred, the EMS API sends notification in the appropriate format to the configured target(s). Installing Resource Monitors To obtain additional information about installed monitors: 1. Navigate to the /etc/opt/resmon/dictionary directory. Each monitor registered with EMS has a dictionary file that is stored in this directory. 2. View the monitor dictionary file. Each dictionary file name is descriptive of its monitor. The file extension is typically.dict. For example, the mibmonitor dictionary filename is mibmond.dict. Configuring Resource Monitors To configure your HA Monitors use either the Event Monitoring Service (EMS) through SAM, the emscli, or the package configuration area of ServiceGuard GUI or edit the package configuration ascii file. 22 Chapter 1

23 Understanding the Event Monitoring Service Resource Monitors Writing Resource Monitors The EMS API provides a method for writing new resource monitors. To create your own monitor, read the Writing Monitors for the Event Monitoring Service (EMS) (HP Part Number B ) manual and install the developer s kit. Both are available at the following Website: 1. Go to the Website: 2. From the Website, select High Availability, then select the Event Monitoring Service Developers Kit, which includes: Writing Monitors for the Event Monitoring Service (EMS) (HP Part Number B ) ReadMe, Installation, and ReleaseNotes Developers Kit Chapter 1 23

24 Understanding the Event Monitoring Service EMS Framework Components EMS Framework Components This section describes the EMS framework components. The EMS API The EMS API is the interface between the registrar, client applications, target applications, and resource monitors as illustrated in Figure 1-1. The EMS API is provided as part of the EMS product. The EMS API manages: Client to registrar communication, which puts clients in contact with the appropriate monitor for discovery and registering monitor requests. Registrar to monitor communication, which passes client requests to the appropriate monitor. Comparison between the current resource values and pre-selected threshold values. Monitoring of target application communication: Sends events to configured targets (pre-existing targets or target you create) Sends notifications to target applications when the resource values meet event criteria For example, a target TCP application uses EMS API to translate TCP messages into EMS objects. This enables the fields to be real. The target application then reads the fields of the EMS objects. The Registrar The registrar is a link between the client applications and the resource monitors. It communicates with the resource monitors on behalf of the client applications to retrieve information requested by the clients. The registrar runs on the same system as the resource monitors. The registrar is provided as part of the EMS product. 24 Chapter 1

25 Understanding the Event Monitoring Service EMS Framework Components The registrar does not need to keep any state information and does not need to be highly available. It does not need to be running while a resource is being monitored. The registrar is needed only to start the monitors and to provide communication between clients and monitors. One registrar process is started each time a client application calls rm_client_connect(), so a registrar is always connected to one client. Depending on the requests sent by the client, the registrar may be connected to 0, 1, 2, or more resource monitors concurrently. The information in the messages contains enough information to allow the registrar to route the requests and replies correctly. Figure 1-3 illustrates how connections are made between clients and registrars. Figure 1-3 Connections Among Clients and registrars Each time the registrar starts, it reads the resource dictionary, exchanges internal version information with the client application, and prepares to receive client requests. When a request arrives, the registrar analyzes it to determine if it is one that it can reply to, or whether it needs to pass the request to a resource monitor. Chapter 1 25

26 Understanding the Event Monitoring Service EMS Framework Components When the registrar needs to pass the request to a resource monitor, it needs to determine if the resource monitor is currently running. If the appropriate resource monitor process is not found, the registrar starts the process and waits until the resource monitor can communicate with the registrar. The Resource Dictionary The resource dictionary is the mechanism by which the resource monitor identifies itself to EMS. The purpose of the resource dictionary is to give a preliminary picture of the resource structure on a given system. Its main function is to indicate to the registrar which resource monitors should be contacted when information is needed about a certain resource. The resource dictionary defines resources on the local system. 26 Chapter 1

27 Monitoring Disk Resources 2 Monitoring Disk Resources This chapter recommends ways to configure requests to the HA Disk Monitor for most high-availability configurations. You can monitor the following SE (single-ended) or F/W (fast/wide) SCSI disks: Hewlett-Packard High Availability Disk Array, for both Fast/Wide SCSI and Fibre Channel Hewlett-Packard Disk Array with AutoRAID, Models 12 and 12H Hewlett-Packard SureStore E Disk Array XP256 for both Fast/Wide SCSI and Fibre Channel Hewlett-Packard High Availability Storage System EMC Symmetrix arrays for both Fast/Wide SCSI and Fibre Channel Single-spindle SCSI disks HP-IB and HP-FL disks are not supported by the HA Disk Monitor. You should be familiar with how the physical and logical volumes are configured on all the nodes in your system. You should know whether disks are configured with redundant PV links, mirroring, or both. You should also know if they are standalone disks. See Managing Systems and Workgroups (HP Part Number B ). This chapter contains the following sections: HA Disk Monitor Reference Rules for Using the HA Disk Monitor with ServiceGuard Creating Disk Monitoring Requests Chapter 2 27

28 Monitoring Disk Resources HA Disk Monitor Reference HA Disk Monitor Reference The HA Disk Monitor reports information about the physical and logical volumes configured by LVM (Logical Volume Manager). Anything not configured through LVM cannot be monitored from the HA Disk Monitor. Monitored disk resources are: /vg/vgname/pv_summary, a summary status of all physical volumes in a volume group /vg/vgname/pv_pvlink/status/devicename, the status of a given physical volume or PV links in a volume group /vg/vgname/lv_summary, a summary status of all logical volumes in a volume group /vg/vgname/lv/status/lvname, the status of a given logical volume in a volume group /vg/vgname/lv/copies/lvname, the number of copies of data available in a volume group Monitoring both the physical and logical volumes allows you to detect failures in volume groups (both active and inactive) and in logical volumes. With these warnings, you can correct hardware problems that put node, application, or data availability at risk. 28 Chapter 2

29 Monitoring Disk Resources HA Disk Monitor Reference Figure 2-1 shows the class hierarchy for the HA Disk Monitor. Figure 2-1 Disk Monitor Resource Class Hierarchy Bold items are resource instances that can be monitored. Bold italic variables represent specific instances of volume groups, devices, and logical volumes on the system. Physical Volume Summary The pv_summary is the summary status of all physical volumes in a volume group. This status is based on the compiled results of SCSI inquiries to all physical volumes in a volume group. If you have configured physical volumes as package dependencies in ServiceGuard, this resource is used to trigger package failover. Refer the manual Using the Event Monitoring Service (HP Part Number B ) for information on configuring ServiceGuard package dependencies. If you are using the disk monitor with ServiceGuard, it is important that you configure physical volume groups (PVGs) to give you the most accurate pv_summary for ServiceGuard package failover. Chapter 2 29

30 Monitoring Disk Resources HA Disk Monitor Reference Table 2-1 Table 2-1 lists how conditions compare in logical operations. Specify the logical operation in the monitor request parameters portion of the monitor request. For example, to create a request that alerts you when the condition is SUSPECT or DOWN, specify greater than or equal to 3 (>=3). Interpreting Physical Volume Summary Resource Name: /vg/vgname/pv_summary Condition Value Interpretation UP 1 All physical volumes containing data are accessible. PVG_UP 2 At least 1 PV has failed. All data is accessible. If more than 1 PV is down and the failed PVs are from the same PVG, all data is still accessible. This condition can only occur in mirrored set or in PV links in PVGs. SUSPECT 3 Two or more physical volumes from different PVGs are not available. The disk monitor cannot conclude that all data is available. For example, on a 2-way mirrored system, if a physical volume fails on each side of the mirror, data may be available if the failed volumes are holding different data. But data may be unavailable if the failed volumes hold the same data. Because the disk monitor only knows that disks have failed, and not what data is on the disks, it marks the volume group SUSPECT. DOWN 4 Some data missing or no data accessible. The pv_summary resource may not be available for a given volume group in the following cases: Devices are on an unsupported bus (such as HP-IB or HP-FL) or an unrecognized bus, (such as a new bus technology). The /var/adm/syslog/syslog.log entry would say: diskmond[5699]: pv_summary will be unavailable for /dev/vg00 because there are physical volumes in this volume group which are on an unrecognized bus. (DRM-502). 30 Chapter 2

31 Monitoring Disk Resources HA Disk Monitor Reference PVGs (physical volume groups) exist in a volume group, but not all physical volumes are assigned to a PVG. The /var/adm/syslog/ syslog.log entry would say: diskmond[18323]: pv_summary will be unavailable for /dev/vgtest because the physical volume groups (PVGs) in this volume group do not have an equal number of PVs or there are PVs not in a PVG. (DRM-503) Unequal numbers of physical volumes exist in each PVG in the volume group. The /var/adm/syslog/syslog.log entry would say: diskmond[18323]: pv_summary will be unavailable for /dev/vgtest because the physical volume groups (PVGs in this volume group do not have an equal number of PVs or there are PVs not in a PVG. (DRM-503) Two cases where this would occur are: There is both 2-way and 3-way mirroring in the same volume group. The mirrored disks contain a different number of physical disks that equate to the same amount of disk space. For example, one 4GB drive in one PVG mirrored with 2-2G drives in the redundant PVG. Checks for the validity of pv_summary are logged with the name of the local node and the identifier diskmond to both /var/adm/syslog/syslog.log and /etc/opt/resmon/log/api.log. Physical Volume and Physical Volume Link Status Requests to monitor physical volumes and physical volume links give you status of the individual physical volumes and PV links in a volume group. In the case of most RAID arrays, this means the HA Disk Monitor can talk to a physical link LUN (logical unit number) in the array. In the case of stand-alone disks, it means the HA Disk Monitor can talk to the disk itself. The pv_pvlink status is used to calculate pv_summary. Although it is somewhat redundant to use both, you might want to have more specific status sent by pv_summary, and only have status sent on pv_pvlinks if a device is DOWN. Chapter 2 31

32 Monitoring Disk Resources HA Disk Monitor Reference Table 2-2 pv_pvlinks and pv_summary supplement lv_summary by giving status on the accessibility of volume groups (both active and inactive) and logical volumes. To pinpoint a failure of a particular disk, bus, or I/O card, you need to use the HA Disk Monitor alerts in conjunction with standard troubleshooting methods: reading log files and inspecting the actual devices. The HA Disk Monitor uses the data in /etc/lvmtab to see what is available for monitoring, and /etc/lvmtab does not distinguish between physical volumes and physical volume links, so you need to investigate to detect whether a disk, bus, or I/O card has failed. Table 2-2 lists how conditions compare in logical operations. You specify the logical operation in the monitor request parameters portion of the monitor request. For example, to create a request that alerts you when the condition is BUSY, you would specify greater than or equal to 2 (>=2). Interpreting Physical Volume and Physical Volume Link Status Resource Name: /vg/vgname/pv_pvlink/status/devicename Condition Value Interpretation UP 1 SCSI inquiry was successful. BUSY 2 SCSI inquiry returned with DEVICE BUSY. The HA Disk Monitor will try 3 times to see if it gets either an UP or DOWN result before marking a device BUSY. DOWN 3 SCSI inquiry failed. The bus and/or the disk is not accessible. While configuring requests from the SAM interface, a wildcard (*) can be used in place of devicename to monitor all physical volumes and physical volume links in a volume group. Logical Volume Summary The logical volume summary describes how accessible the data is in all logical volumes in an active volume group. Sometimes the physical connection may be working, but the application cannot read or write data on the disk. The HA Disk Monitor determines I/O activity by querying LVM, and marks a logical volume as DOWN if a portion of its data is unavailable. 32 Chapter 2

33 Monitoring Disk Resources HA Disk Monitor Reference NOTE If the logical volume is in an inactive volume group, the HA Disk Monitor cannot determine if the data can be accessible. Table 2-3 Table 2-3 lists how conditions compare in logical operations. You specify the logical operation in the monitor request parameters portion of the monitor request. For example, to create a request that alerts you when the condition is INACTIVE_DOWN, you would specify greater than or equal to 3 (>=3). Interpreting Logical Volume Summary Resource Name: /vg/vgname/lv_summary Condition Value Interpretation UP 1 All logical volumes are accessible and all data is accessible. INACTIVE 2 The volume group is inactive. This could be because: The volume group is active in exclusive mode on another node in an ServiceGuard cluster. (This is not valid for clusters running ServiceGuard Extension for RAC, because it can support a volume group being active on more than one node.) Note that ServiceGuard does allow a volume group to be active in read-only mode, if it is already active on another node. The volume group is made inactive using vgchange -a n for maintenance or other reasons. There is no quorum of active physical volumes at system boot. For example, not enough disks in the volume group were working. INACTIVE_DO3 The last time the inactive volume was activated, it WN was DOWN. At least one logical volume in the volume was inaccessible. Chapter 2 33

34 Monitoring Disk Resources HA Disk Monitor Reference Table 2-3 Interpreting Logical Volume Summary (Continued) Resource Name: /vg/vgname/lv_summary Condition Value Interpretation DOWN 4 At least one logical volume in the volume group reports a status of either INACTIVE or DOWN. Note that an inactive logical volume in an active volume group is rare, but possible. See Logical Volume Status on page 34. Logical Volume Status Table 2-4 Logical volume status gives you status on each logical volume in a volume group. While the lv_summary gives you information on whether data in a volume group is available, the lv/status/lvname gives you information on whether specific logical volumes have failed. Table 2-4 lists how conditions compare in logical operations. You specify the logical operation in the monitor request parameters portion of the monitor request. For example, to create a request that alerts you when the condition is INACTIVE, you would specify greater than or equal to 2 (>=2). Interpreting Logical Volume Status Resource Name: /vg/vgname/lv/status/lvname Condition Value Interpretation UP 1 All logical volumes are accessible and all data is accessible. INACTIVE 2 The logical volume is inactive. DOWN 3 The logical volume is DOWN. A complete copy of the data is not available for this logical volume. When configuring requests from the SAM interface, use a wildcard (*) in place of lvname to monitor all logical volumes in a volume group. If you split off mirrors from your mirrored configuration, you will see new logical volume resource instances when the split mirror is created. 34 Chapter 2

35 Monitoring Disk Resources HA Disk Monitor Reference Logical Volume Number of Copies Table 2-5 The logical volume number of copies is most useful to monitor in a mirrored disk configuration. It tells you how many copies of the data are available. The HA Disk Monitor monitors all copies of data, and therefore counts the original as part of the total number of copies. MirrorDisk/UX supports up to 3-way mirroring, so the range can be from 0 to 3 copies (see Table 2-5). In a RAID configuration that is not mirrored using LVM, the only possible number is 0 or 1; either the data is accessible or it is not. When you first configure mirroring in LVM, it lists 0 mirrors, meaning you have only the original copy of the data. Likewise, 2 mirrors mean you have one original plus 2 mirrored copies. Interpreting Logical Volume Copies Resource Name: vg/vgname/lv/copies/lvname Condition Interpretation 0 No additional copies, (only the original copy), either physical parts of the disk array have problems, the lv is inactive, or a physical extent is stale or unavailable. 1 One complete copy of data is available. If the data is not mirrored, then all physical extents are fine. If the data is mirrored, all other copies have problems. 2 Two complete copies of data are available. If the data is two-way mirrored, then all physical disks are up and data is available. If the data is 3-way mirrored, at least one logical extent has a missing or stale physical extent. 3 All copies of a 3-way mirror are available. When configuring requests from the SAM interface, use a wildcard (*) in place of lvname to request status for all logical volumes in a volume group. If you split off mirrors from your mirrored configuration, you will see the number of copies reduced by 1 when the split mirror is created. Chapter 2 35

36 Monitoring Disk Resources Rules for Using the HA Disk Monitor with ServiceGuard Rules for Using the HA Disk Monitor with ServiceGuard The HA Disk Monitor is designed for use with ServiceGuard to trigger package failover if host adapters, buses, controllers, or disks fail. Here are some examples: In a cluster where one copy of data is shared between all nodes in the cluster, you may want to fail a package if the host adapter has failed on the node running the package. Because buses, controllers, and disks are shared, package failover will not run the package successfully. ServiceGuard can then compare the resource UP values on all nodes and fail the package over to the node that has the correct resources available. In a cluster where each node has its own copy of data, you may want to fail a package over to another node for any number of reasons: Host adapter, bus, controller, or disk failure Unprotected data (the number of copies is reduced to one) Performance has degraded because one of the PV links has failed For example, in a cluster of Web servers where each node has a copy of the data and users are distributed for load balancing, you can fail a package over to another node with the correct resources available. Disk availability is based on pv_summary. See the manual Using the Event Monitoring Service (HP Part Number B ) for information on configuring package dependencies. In addition to configuring disks as ServiceGuard package dependencies, you may also want to have alerts sent to a system management tool such as HP OpenView IT/Operations or Network Node Manager. Although ServiceGuard and EMS work together to provide package failover, they do not send events or log the source of the failure. Also, failures may not cause a package to fail over, but may expose a single point of failure that you want to know about. Therefore, it is recommended that you also configure requests from the SAM interface to EMS. 36 Chapter 2

37 Monitoring Disk Resources Rules for Using the HA Disk Monitor with ServiceGuard Setting Failover Parameters Table 2-6 When using the HA Disk Monitor with ServiceGuard, the parameters listed in Table 2-6 should be set so that a package failover will occur when access to a disk resource fails. Setting Failover Parameters Parameter Setting Notes RUN_SCRIPT_ TIMEOUT HALT_SCRIPT_ TIMEOUT non-zero timeout value non-zero timeout value Do not leave these parameters set to the default, NO_TIMEOUT.If you do, the run or halt script hangs the package and will not proceed with its action. Working with Physical Volume Groups The pv_summary is calculated based on the compiled results of SCSI inquiries to all physical volumes in a volume group. To help you determine the best way to configure your disks for monitoring, here are the assumptions made by the monitor when calculating pv_summary: PVGs (physical volume groups) are set up to be bus-specific sides of a mirror or redundant links, PVGs have an equal number of physical volumes. All logical volumes within a volume group are mirrored in the same way: all 2-way or all 3-way mirroring. If you are configuring a monitoring request through ServiceGuard, a package depends on all logical volumes in the volume group. The SCSI inquiry will retry on devices that are BUSY. BUSY devices are not considered UP when calculating pv_summary. These rules apply when creating a PVG. If the rules are not followed, pv_summary will not be available for monitoring: If PVGs are used, all physical volumes in a volume group must be in a PVG. All PVGs in a volume group must have the same number of physical volumes. Table 2-7 is a summary of how pv_summary is calculated where Chapter 2 37

38 Monitoring Disk Resources Rules for Using the HA Disk Monitor with ServiceGuard Table 2-7 n is the number of paths for the volume group in /etc/lvmtab (physical volumes, paths, or LUNs). p is the number of PVGs (physical volume groups) in the volume group x is the number of paths currently available from a SCSI inquiry pv_summary Calculations Case Conclusion State x = n All physical volumes and all data are available. UP n>x>=n - (p-1) All data is available. PVG_UP n/p <= x <= n-(p-1) If there are PVGs, and one PVG has all paths, then all data is available. If there are PVGs, and none of the PVGs has all paths, then the HA Disk Monitor cannot determine if all data is available. PVG_UP SUSPECT x < n/p Some data is missing. x=0 No data or physical volumes are available. DOWN To give pv_summary the most accurate picture of data availability, you need to use PVGs to define your physical volumes as separate access points to data. Mirroring should be PVG-strict. Arrays should have PV links, with redundant links in a separate PVG. Note that if you do not configure PV links into separate PVGs, p in Table 2-7 will always be equal to 1. Therefore any SCSI inquiry that does not return a value of UP for every path will result in a calculation of DOWN for pv_summary. Rules for RAID Arrays RAID configurations must be configured with PV links. PV links are redundant links attached to separate controllers on the array. If PV links are configured, and one fails, LVM automatically switches to the alternate controller when one fails. 38 Chapter 2

39 Monitoring Disk Resources Rules for Using the HA Disk Monitor with ServiceGuard To use the HA Disk Monitor with ServiceGuard, PV links must be configured in separate PVGS (physical volume groups). This new requirement allows pv_summary to accurately calculate data availability based on physical volume availability, thus including both ACTIVE and INACTIVE volume groups. If PV links are not configured in separate PVGs, the HA Disk Monitor sees all links to the array as one physical volume. If one link fails, pv_summary will register DOWN, and your package will fail over, even if the other link is still up and data is available. The following sections describe how to make sure your PV links are in physical volume groups. Adding PVGs to Existing Volume Groups If you have already created volume groups, you can create PVGs and put PV links into them. VG /dev/vgdance PVG busa /dev/dsk/c1t0d0 /dev/dsk/c1t2d0 PVG busb /dev/dsk/c2t1d0 /dev/dsk/c2t3d0 VG /dev/vgsing PVG busa /dev/dsk/c1t0d1 /dev/dsk/c1t2d1 PVG busb /dev/dsk/c2t1d1 /dev/dsk/c2t3d1 1. Create a file called /etc/lvmpvg with permissions 600. See the lvmpvg manpage and Managing Systems and Workgroups (HP Part Number B ). 2. Create an entry for each volume group and assign a different PVG name to each PV link. The PVG names can be any arbitrary name, but must be unique on the system. For example, an array containing 2 volume groups, vgdance and vgsing, each containing a single LUN and each with 2 PV links (see Figure 2-4 on page 49) should have the following /etc/lvmpvg file: 3. Carefully copy the /etc/lvmpvg to each system connected to the disk array. Chapter 2 39

40 Monitoring Disk Resources Rules for Using the HA Disk Monitor with ServiceGuard NOTE Make sure you edit lvmpvg to contain the correct link names in /dev/dsk/device for that system. Creating Volume Groups on Disk Arrays Using PV Links If you will be monitoring volume groups that use mass storage on disk arrays, you should use redundant I/O channels from each node, and connect them to separate controllers on the array. Then you can define alternate links to the LUNs or logical disks you have defined on the array. Alternate links (known as PV links) to the same disk should be assigned to different physical volume groups. In SAM, choose the type of disk array you want to configure, and follow the menus to define alternate links. Be sure to specify a different physical volume group for each link to the same disk. The following example shows how to configure alternate links using LVM commands. In the example, the following disk configuration is assumed: 8/ /dev/dsk/c0t15d0 /* I/O Channel 0 (8/0) SCSI address 15 LUN 0 */ 8/ /dev/dsk/c0t15d1 /* I/O Channel 0 (8/0) SCSI address 15 LUN 1 */ 8/ /dev/dsk/c0t15d2 /* I/O Channel 0 (8/0) SCSI address 15 LUN 2 */ 8/ /dev/dsk/c0t15d3 /* I/O Channel 0 (8/0) SCSI address 15 LUN 3 */ 8/ /dev/dsk/c0t15d4 /* I/O Channel 0 (8/0) SCSI address 15 LUN 4 */ 8/ /dev/dsk/c0t15d5 /* I/O Channel 0 (8/0) SCSI address 15 LUN 5 */ 10/0.3.0 /dev/dsk/c1t3d0 /* I/O Channel 1 (10/0) SCSI address 3 LUN 0 */ 10/0.3.1 /dev/dsk/c1t3d1 /* I/O Channel 1 (10/0) SCSI address 3 LUN 1 */ 10/0.3.2 /dev/dsk/c1t3d2 /* I/O Channel 1 (10/0) SCSI address 3 LUN 2 */ 10/0.3.3 /dev/dsk/c1t3d3 /* I/O Channel 1 (10/0) SCSI address 3 LUN 3 */ 10/0.3.4 /dev/dsk/c1t3d4 /* I/O Channel 1 (10/0) SCSI address 3 LUN 4 */ 10/0.3.5 /dev/dsk/c1t3d5 /* I/O Channel 1 (10/0) SCSI address 3 LUN 5 */ /dev/dsk/c0t15d0 /dev/dsk/c1t3d0 Assume that the disk array has been configured, and that the following device files appear for the same LUN (logical disk) when you run the ioscan command: Use the following steps to configure a volume group for this logical disk: 1. First, set up the group directory for vgdatabase: 40 Chapter 2

41 Monitoring Disk Resources Rules for Using the HA Disk Monitor with ServiceGuard # mkdir /dev/vgdatabase 2. Next, create a control file named group in the directory /dev/vgdatabase, as follows: # mknod /dev/vgdatabase/group c 64 0xhh0000 # ls -l /dev/*/group # pvcreate /dev/dsk/c0t15d0 The major number is always 64, and the hexadecimal minor number has the form: 0xhh0000 where hh must be unique to the volume group you are creating. Use an appropriate hexadecimal number that is available on your system, after the volume groups are already configured. On a single system, this might be the next hexadecimal number. On a cluster, this number must be assigned cluster-wide, so it should be one of the hexadecimal numbers used in the cluster. Use the following command to display a list of existing volume groups: 3. Use the pvcreate command on one of the device files associated with the LUN to define the LUN to the LVM (logical volume manager) as a physical volume. It is only necessary to do this with one of the device file names for the LUN. 4. Use the following commands to create the volume group itself with the first link assigned to a physical volume group called bus1and the second link assigned to a physical volume group called bus2: # vgcreate -g bus1 /dev/vgdatabase /dev/dsk/c0t15d0 # vgextend -g bus2 /dev/vgdatabase /dev/dsk/c0t3d0 LVM will now recognize the I/O channel represented by /dev/dsk/ c0t15d0 as the primary link to the disk; if the primary link fails, LVM will automatically switch to the alternate I/O channel represented by /dev/dsk/c1t3d0. Creating Logical Volumes Use the following command to create logical volumes (the example is for /dev/vgdatabase): # lvcreate -L 120 -m 1 -s g /dev/vgdatabase If you are using disk arrays in RAID 1 or RAID 5 mode, omit the -m 1 option. Chapter 2 41

42 Monitoring Disk Resources Rules for Using the HA Disk Monitor with ServiceGuard This command creates a 120 MB mirrored volume named lvol1. The name is supplied by default, since no name is specified in the command. The -s g option means that mirroring is PVG-strict, that is, the mirror copies of data will be in different physical volume groups. Rules for Mirrored Individual Disks The following rules apply to configuring mirrored disks for use with ServiceGuard and the HA Disk Monitor: Mirroring must be PVG-strict. Mirrored volumes must reside on a different bus from the original volume to avoid a single point of failure and to obtain the best pv_summary value for that mirror. This is done automatically by LVM if you have created the PVGs while setting up mirroring. See the lvextend manpage and Managing MC/ServiceGuard (HP Part Number B ) for more information. Logical volumes that are 2-way mirrored should be in separate volume groups from those that are 3-way mirrored. Putting differently mirrored volumes in the same volume group makes it difficult to accurately interpret the pv_summary data. Take the example of a volume group containing both 2- and 3-way mirroring. If 2 host adapters fail on that volume group, it could mean no data available for the 2-way mirrored logical volume, but one copy still available for the 3-way mirrored volume. The pv_summary would be wrong for one of those mirrored disk configurations. Volume groups representing the same hardware for failover must be created with exactly the same name on all nodes. For example, a bus connecting 3 nodes to a disk array must be defined as part of vg01 on all 3 nodes. It is also recommended that you use the same names for PVGs containing the same actual disks. Mirrors that have been split off are treated the same by the HA Disk Monitor as they are by LVM. When you split off a mirror, you will see a change in the following resources: /vg/vgname/lv/copies/lvname will be reduced by one when the mirror is split off. If you have created a monitoring request for that resource that alerts you when the number of copies is changed or is reduced, you will see an event. 42 Chapter 2

43 Monitoring Disk Resources Rules for Using the HA Disk Monitor with ServiceGuard /vg/vgname/lv/status will have a new /lvname resource instance that represents the split-off mirror. /vg/vgname/lv_summary will change depending on the state of the new logical volume created by the split mirror. If you restore the split mirror normally using supported LVM commands, the HA Disk Monitor will detect the merged mirror and report it. Chapter 2 43

44 Monitoring Disk Resources Creating Disk Monitoring Requests Creating Disk Monitoring Requests There are two ways to create HA Disk Monitor requests: From EMS GUI, to send alerts to HP OpenView ITO, Network Node Manager, addresses, the console, a textlog file, or the system log. From ServiceGuard, to configure any HA Disk Monitor resource as a package dependency. These requests are not exclusive. You can configure the HA Disk Monitor from both ServiceGuard and EMS. If you are using EMS to monitor disks for ServiceGuard package dependencies, it is recommended you also configure EMS to send events to alert you when something threatens data or application availability. NOTE When ServiceGuard packages are configured, the package configuration (including package dependencies) is automatically distributed to all nodes in the cluster. If you also want to send event notification to your system monitoring software (IT/O), you must configure the same monitoring requests on all nodes in the cluster using the EMS GUI. The following sections take some common disk configurations in a high availability environment and give examples of the types of monitor requests you might want to create. 44 Chapter 2

45 Monitoring Disk Resources Creating Disk Monitoring Requests Disk Monitoring Request Suggestions Table 2-8 The examples listed in Table 2-8 are valid for both RAID and mirrored configurations. Suggestions for Creating Disk Monitor Requests To be alerted when... Resources to monitor Monitoring Parameters Notify Value Option you are at risk for pv_ data loss (most summary common for use with ServiceGuard) lv_ summary when value is when value is >= SUSPECT >= INACTIVE_D OWN any disks fail pv_pvlink/ status/* when value is not equal UP any disks fail, and you want to know when they are back up pv_pvlink/ status/* when value is not equal UP RETURN you want regular reminders to fix inoperative disks, controllers, buses, and host adapters, and you want notification when they are fixed pv_pvlink/st at each = DOWN REPEAT atus/* interval (use a long polling interval, 1 hour or more) RETURN any logical volume becomes unavailable lv/status/* when value is not equal UP you have lost a mirror in your 2-way mirroring environment lv/copies/* when value is < 2 Chapter 2 45

46 Monitoring Disk Resources Creating Disk Monitoring Requests The following series of screens provide a sample process for creating an HA Disk Monitor request. These samples use the EMS GUI, though the Package Dependency screens in ServiceGuard are similar. Refer to the Using the Event Monitoring Service (HP Part Number B ) for specific instructions. Assume you want to be alerted when any disks fail and when they are back up. Figure 2-2 shows you can select all instances of pv_pvlink, so you only have to enter the parameters once for each volume group. You still need to create multiple pv_pvlink requests, one for each volume group on your system. Click OK to set monitoring parameters. Figure 2-2 Example: Selecting All Instances of /system/filesystem/availmb 46 Chapter 2

47 Monitoring Disk Resources Creating Disk Monitoring Requests The parameters for the monitoring request in Figure 2-3 request an event notification when the resource value is not equal to UP. The polling interval for checking the resources value is 300. The notification method is an SNMP trap with a minor severity level. No initial, repeat or return values are requested. Figure 2-3 Example: Configuring /vg/vg01/pv_pvlink/status Parameters to Notify When Disks Fail All requests are created in a similar way. You need to make sure you perform these steps for all instances in all volume groups you want to monitor. Chapter 2 47

48 Monitoring Disk Resources Creating Disk Monitoring Requests Resources to Monitor for RAID Arrays Table 2-9 These considerations are relevant to all RAID supported configurations listed at the beginning of this chapter. To adequately monitor a RAID system, create requests to monitor at least the following resources for all volume groups on a node: Resources to Monitor for RAID Arrays /vg/vgname/ pv_summary vg/vgname/ pv_pvlink/ status/* vg/vgname/ lv_summary This gives you an overview of the status of the entire physical volume group and is recommended when using EMS in conjunction with ServiceGuard; see Rules for Using the HA Disk Monitor with ServiceGuard on page 36. This gives you the status of each PV link in the array and is redundant to pv_summary. It is recommended when using EMS outside of the ServiceGuard environment, or if you require specific status on each physical device. This gives you the status of data availability on the array. 48 Chapter 2

49 Monitoring Disk Resources Creating Disk Monitoring Requests Figure 2-4 represents a node with two RAID arrays and two PV links. Figure 2-4 RAID Array Example Each LUN on the RAID array is in its own volume group: vgdance and vgsing. Assume this is one node in a 2-node cluster and you want to be notified when there is a failover, when any physical device fails, and when any logical volume becomes unavailable. If you do not have ServiceGuard Manager installed with OpenView, to be notified when a package fails over, you must configure an EMS request that is the same as the package dependency you configured in ServiceGuard. See Using the Event Monitoring Service (HP Part Number B ). For this example, assume the package UP values were set as UP and PVG_UP. Chapter 2 49

50 Monitoring Disk Resources Creating Disk Monitoring Requests Table 2-10 To configure the EMS alerts, create the following requests: Sample Disk Monitoring Requests Resource Monitoring Parameters Notify Condition Option /vg/vgdance/pv_summary when value is > PVG_UP RETURN /vg/vgsing/pv_summary when value is > PVG_UP RETURN /vg/dance/lv_summary when value is >= INACTIVE RETURN /vg/vgsing/lv_summary when value is >= INACTIVE RETURN If pv_summary is SUSPECT, you know a physical device failed. If pv_summary status is SUSPECT, you may want to look at your lv_summary to see if you can still access all data. If lv_summary is DOWN or INACTIVE_DOWN, you do not have a complete copy of data. Resources to Monitor for Mirrored Disks Table 2-11 This section is valid for mirrored disks created with MirrorDisk/UX. Mirroring is required to be PVG-strict if you are using the HA Disk Monitor. Mirrored configurations that are not PVG-strict will not give you a correct pv_summary. To adequately monitor mirrored disks, create requests for the following resources for all volume groups on a node: Resources to Monitor for Mirrored Disks /vg/vgname/ pv_summary vg/vgname/pv_ pvlink/status/* vg/vgname/ lv_summary This gives you summary status of all physical volumes in a volume group. A high availability system must be configured PVG strict. If not, pv_summary cannot accurately determine disk availability. This gives you the status of each physical disk and each link. This gives you the status of data. If it is available on the logical volumes. 50 Chapter 2

51 Monitoring Disk Resources Creating Disk Monitoring Requests Table 2-11 Resources to Monitor for Mirrored Disks (Continued) vg/vgname/lv/ copies/* This gives you the total number of copies of data currently available. Copies in addition to the original copy. Figure 2-5 represents two nodes with 2-way mirrored configuration with 10 disks on 2 buses. Both copies are in a single volume group. Assume you want to be notified when: Any physical device fails You only have one copy of data There is a ServiceGuard failover. Chapter 2 51

52 Monitoring Disk Resources Creating Disk Monitoring Requests To configure this last request, you must duplicate your ServiceGuard package dependency. Figure 2-5 Mirrored Disks Example Table 2-12 To configure the EMS alerts, create the requests listed in Table 2-12 on each node: EMS Alert Requests Resource Monitoring Parameters Notify Condition Option /vg/vg01/pv_summary when value is >= PVG_UP RETURN /vg/vg01/lv_summary when value is >= INACTIVE RETURN 52 Chapter 2

53 Monitoring Disk Resources Creating Disk Monitoring Requests Table 2-12 EMS Alert Requests (Continued) Resource Monitoring Parameters Notify Condition Option /vg/vg01/lv/copies/* when value is <= 1 RETURN Table 2-13 Alerts need to be interpreted in relation to each other. In the table above, you would get an alert when PVG_UP is true. Although all data is available, the condition PVG_UP implies there are physical volumes that are not functioning and need to be fixed. See Table You may want to examine lv/copies to see how many copies of data are accessible and determine how urgently you need to repair the failures. For example, if you have 3-way mirroring and only one copy of data is available, you may want to correct the failure immediately to eliminate the single point of failure. Table 2-13 is an example of how the HA Disk Monitor determines whether data is available in a mirrored configuration with 5 disks on each bus. Example for Interpreting the pv_summary for Mirrored Disks Number of Valid Devices Meaning pv_summary Value 10 all PVs and data accessible UP 9 1 PV down, all data accessible PVG_UP 8-5 if 5 PVs are from the same PVG, then all data is available if 2 or more physical volumes from different PVGs are DOWN, the HA Disk Monitor cannot conclude that all data is available PVG_UP SUSPECT 4-1 some data missing 0 no data available DOWN Chapter 2 53

54 Monitoring Disk Resources Creating Disk Monitoring Requests Resources to Monitor for Lock Disks Table 2-14 Lock disks are used as a tie-breaker in forming or reforming a cluster. If the lock disk is unavailable during cluster formation, the cluster may fail to reform. If you are using a lock disk with your cluster, you should configure a monitoring request for that disk and send an alert to your system management software if the lock disk is unavailable. Requests to monitor the lock disk might look like those listed in Table 2-14: Lock Disk Monitoring Requests Resource Monitoring Parameters Notify Condition Option /vg/vg02/pv_pvlink/c0t0d1 when value is >= BUSY REPEAT The Repeat value in the Options will send an alert until the lock disk is available. You need to create a request on each node in the cluster. Because the bus name and SCSI path to the lock disk may be different on each node, the resource instance may have a different name. It is merely a different path to the same lock disk. Resources to Monitor for Root Volumes Table 2-15 In a high availability system, it is recommended that you mirror your root volume, and have them on separate links in separate PVGs. Note that the root volume should always be ACTIVE. Requests to monitor the root volume might look like those listed in Table 2-15: Root Volumes Monitoring Requests Resource Notify Monitoring Parameters Condition Option /vg/vg00/pv_pvlink/c0t0d0 when value is >= BUSY REPEAT /vg/vg00/pv_pvlink/c1t0d0 when value is >= BUSY REPEAT /vg/vg00/lv_summary when value is not equal UP RETURN 54 Chapter 2

55 Monitoring Disk Resources Creating Disk Monitoring Requests Table 2-15 Root Volumes Monitoring Requests (Continued) Resource Notify Monitoring Parameters Condition Option exclude_vg=/dev/vgxx /vg/vg00/lv/copies/lv01 when value is < 1 RETURN If one of the root volumes is unavailable, you are alerted and told which one has failed (pv_pvlink/status). You are alerted if you lose a root disk mirror. With the RETURN option, you are also notified when the mirror is restored. Excluding Volume Groups from being Monitored This feature implements an optional config file to the HA Disk Monitor. If the config file is not present when the HA Disk Monitor begins, the HA Disk Monitor processes all volume groups found in /etc/lvmtab. This is the default behavior. If the HA Disk Monitor config file is present, the HA Disk Monitor reads the config file. Any volume groups in the file that are found are excluded (filtered out) from monitoring by the HA Disk Monitor. The config file must be located in the directory /etc/opt/resmon/monitors and be named diskmond.config. You must create this file in order to enable the filtering-out. There is a sample config file supplied in /etc/opt/resmon/monitors/diskmond.config.sample. The file and format are also documented in the diskmond manpage. The diskmond.config file contains: where vgxx is the volume group to be excluded. Add one line for each volume group to exclude. Ownership on the config file should be bin:bin. If the config file is present but empty, HA Disk Monitor assumes the default behavior and monitors all the disks. If there are typographical errors in it, HA Disk Monitor attempts to honor the exclude_vg s that it can find, if any. Chapter 2 55

56 Monitoring Disk Resources Creating Disk Monitoring Requests 56 Chapter 2

57 Monitoring Database Resources 3 Monitoring Database Resources The HA Database Monitor monitors values and sends events regarding the status of databases and the database servers that support them. These values are defined as part of the rdbms public MIB definition (RFC1697). This chapter contains the following sections: Database Monitor Reference Rules for Using the HA Database Monitor with ServiceGuard Creating Database Monitoring Requests Chapter 3 57

58 Monitoring Database Resources Database Monitor Reference Database Monitor Reference The HA Database Monitor reports events based on the status of supported databases configured on HP-UX systems. These MIB resources can be monitored: Database resources: /rdbms/database/resource_class/database_name Information about a database on a given system, such as status and disk usage. Server resources: /rdbms/server/resource_class/server_name Information about a database server for the database, such as connections and completed transactions. To fix a problem detected by the HA Database Monitor, refer to your vendor s database documentation and your server documentation. Figure 3-1 HA Database Monitor Resource Class Hierarchy Figure 3-1 shows the HA Database Monitor class hierarchy. Bold items are resource instances that can be monitored. Bold italic variables represent specific instances of the databases and database servers on the system. 58 Chapter 3

59 Monitoring Database Resources Database Monitor Reference Database Resources The database resources available for monitoring are defined under: /rdbms/database The database resource class name is then specified, followed by the database name. The database resource class names are: status allocated usage used The database name varies depending upon your environment and the number of databases installed. The minimum polling interval for all database resources is 30 seconds. You may want a longer interval, especially if system performance is affected. Table 3-1 lists the database resources and potential values. Chapter 3 59

60 Monitoring Database Resources Database Monitor Reference Table 3-1 Interpreting Database Resource Classes Resource Name: /rdbms/database/resource_class/database_name resource_ class Condition Description status The values are: This resource class ACTIVE describes the state The database is of the database. currently being used by a database server. AVAILABLE UNAVAILABLE The database is accessible, but it is not currently being used by a database server. The database is not accessible to any database server. allocated a floating-point value in kilobytes This resource class describes the amount of disk space reserved for the database representing the maximum amount of disk space that could be used for user-created objects in the database. 60 Chapter 3

61 Monitoring Database Resources Database Monitor Reference Table 3-1 Interpreting Database Resource Classes (Continued) Resource Name: /rdbms/database/resource_class/database_name resource_ class usage used Condition a floating-point value expressed as a percentage a value of 100 indicates that all of the allocated space is currently used a value of -1 indicates that no space has been allocated for the database a floating-point value in kilobytes if the used value exceeds the allocated value, it indicates that the database is out of disk space Description This resource class describes the percentage of allocated space currently being used in the database indicating how full the database is and whether it is approaching capacity. This resource class describes the estimated amount of disk space currently being used in the database based on the space actually being used for user-created objects within the database. Chapter 3 61

62 Monitoring Database Resources Database Monitor Reference Server Resources Table 3-2 The database server resources available for monitoring are defined under /rdbms/server/, followed by the server resource class name, and then followed by the server name. The server resource class names are listed in Table 3-2. The server name varies, depending on your environment. Interpreting Server Resource Classes Resource Name: /rdbms/server/resource_class/server_name resource_ class Condition Description status 1 The values are: This resource class describes the state of the database UP the database server supporting the database server is instance. operational and available for processing requests DOWN HALTED the database server is not available for processing requests the database server is temporarily unavailable for administrative reasons 62 Chapter 3

63 Monitoring Database Resources Database Monitor Reference Table 3-2 Interpreting Server Resource Classes (Continued) Resource Name: /rdbms/server/resource_class/server_name resource_ class Condition Description status 1 The values are: (continued) This resource class describes the state of the database CON- the database server supporting the database GESTED server is not instance. accepting requests due to resource limitations RE- STARTING the database server was restarted and will be available soon allowed_ max_ connects an integer value from the database server configuration a value of zero (0) indicates that the number of connections is unlimited This resource class describes the maximum number of simultaneous connections allowed by the database server. commits an integer This resource class describes the number of transactions that have been completed by the database server. commits_ per_sec a floating-point number This resource class describes the number of transactions per second completed by the database server. connects an integer This resource class describes the current number of simultaneous connections to the database server. Chapter 3 63

64 Monitoring Database Resources Database Monitor Reference Table 3-2 Interpreting Server Resource Classes (Continued) Resource Name: /rdbms/server/resource_class/server_name resource_ class disk_ reads disk_ reads_ per_sec disk_ writes disk_ writes_ per_sec logical_ reads logical_ reads_ per_sec logical_ writes logical_ writes_ per_sec Condition an integer a floating-point number an integer a floating-point number an integer a floating-point number an integer a floating-point number Description This resource class describes the number of disk reads on the database server since it started. This resource class describes the number of disk reads per second on the database server This resource class describes the number of disk writes on the database server since it started. This resource class describes the number of disk writes per second on the database server. This resource class describes the number of logical reads on the database server since it started. This resource class describes the number of logical reads per second on the database server. This resource class describes the number of logical writes on the database server since it started. This resource class describes the number of logical disk writes per second on the database server. 64 Chapter 3

65 Monitoring Database Resources Database Monitor Reference Table 3-2 Interpreting Server Resource Classes (Continued) Resource Name: /rdbms/server/resource_class/server_name resource_ class peak_ connects an integer Condition if it keeps increasing over time, this could be an indication that a configuration parameter needs to be increased Description This resource class describes the greatest number of simultaneous connections made to the database server since the database server started. read_ cache_ hit_rate a floating-point number, expressed as a percentage a value of 100 would imply that all reads are from the cache; in other words, the closer to 100, the better the performance a value of -1 implies that no logical reads have occurred This resource class describes the ratio of logical reads less physical reads to logical reads, done by the database server: (logical_reads) - (physical_reads) (logical_reads) this is a measure of the effectiveness of the read cache. started character numbers in the following format (year, month, day, hour, minute, and second): YYYY/MM/DD HH:MM:SS.S This resource class describes date and time when the database server was last started. uptime a floating-point value in seconds a value of minus one (-1) might indicate that the database server started before the SNMP subagent This resource class describes length of time the database server has been running. Chapter 3 65

66 Monitoring Database Resources Database Monitor Reference Table 3-2 Interpreting Server Resource Classes (Continued) Resource Name: /rdbms/server/resource_class/server_name resource_ class Condition Description usage a floating-point number This resource class describes expressed as a percentage the percentage of maximum allowed connections to the a value of 100 indicates that database server currently in all available connections are use. active a value of -1 indicates that the number of connections is unlimited write_ cache_ hit_rate a floating-point number, expressed as a percentage a value of 100 would imply that all writes are from the cache; in other words, the closer to 100, the better the performance This resource class describes the ratio of logical writes less physical writes to logical writes, done by the database server: (logical_writes) - (physical_writes) (logical_writes) a value of -1 implies that no this is a measure of the logical writes have occurred effectiveness of the write cache. 1. The Oracle implementation supports only UP and DOWN. The minimum polling interval for all database server resources is 30 seconds. You may want a longer interval, especially if system performance is affected. 66 Chapter 3

67 Monitoring Database Resources Rules for Using the HA Database Monitor with ServiceGuard Rules for Using the HA Database Monitor with ServiceGuard The HA Database Monitor with ServiceGuard provides package failover if database servers fail or if the usage or number of connections exceeds specified levels. For example: In a cluster where a database is accessed by multiple database servers on the same host, you may want to fail a package over if the database servers on that host are stopped for some reason, such as shutting down the instance or if one or more of the server processes aborts. In a cluster where database access is limited by a maximum number of connections, you may want to fail a package over when this maximum number is reached, to a host where there is no limit set. In a cluster where the host cache is not being effectively utilized, you may want to fail a package over to a host that is less busy. Database availability is based on database status and database server status. See the manual Using the Event Monitoring Service (B ) for information on configuring package dependencies. In addition to configuring databases as ServiceGuard package dependencies, you may also want to send notifications about events to a system management tool such as HP Open View IT/Operations or Network Node Manager. Although ServiceGuard and EMS work together to provide package failover, they do not send events or log the source of the failure. Also, failures may not cause a package failover, but may expose a single point of failure that you wish to know about. Therefore, it is recommended that you also configure resource monitoring requests from the SAM interface through the Resource Management icon to EMS. NOTE The following sections provide an example of the parameters and settings for the vendor database software from Oracle Corporation. Currently this is the only vendor database that can be configured as a ServiceGuard package dependency. Chapter 3 67

68 Monitoring Database Resources Rules for Using the HA Database Monitor with ServiceGuard Setting Failover Parameters Table 3-3 When using the HA Database Monitor with ServiceGuard, the ServiceGuard parameters listed in Table 3-3 should be set so that a package failover will occur when access to the database resource fails. Setting Failover Parameters Parameter Recommen File Location ded Setting Notes RUN_ SCRIPT_ TIMEOUT HALT_ SCRIPT_ TIMEOUT RESOURCE_ START RESOURCE_ NAME non-zero timeout value non-zero timeout value DEFERRED database resource ServiceGuard package configuration file ServiceGuard package configuration file ServiceGuard package configuration file Do not leave these parameters set to the default, NO_TIMEOUT. If you do and if the run or halt script hangs for some reason, the package will not fail over successfully to another node. This means that the resource will not be checked for accessibility by the ServiceGuard daemon until after the cmstartres command is issued by the control script. cmstartres verifies that ServiceGuard will initialize successfully and the database resources are ready to be monitored before any dependency checking begins. The name of the database or database resource to be monitored, for example, /rdbms/server/ status/database_server_n ame RESOURCE_UP_ UP value VALUE ServiceGuard package configuration file The condition that defines the UP state, for example, RESOURCE_UP_VALUE=UP 68 Chapter 3

69 Monitoring Database Resources Rules for Using the HA Database Monitor with ServiceGuard Table 3-3 Setting Failover Parameters (Continued) Parameter Recommen File Location ded Setting Notes RESOURCE_ POLLING_ INTERVAL DEFERRED_ RESOURCE_ NAME number of seconds database resource ServiceGuard package configuration file ServiceGuard package control script Specify how often ServiceGuard will check the resource, for example once every 30 seconds The name of the database resource that must be started prior to monitoring, for example, /rdbms/server/status/ database_server_name Chapter 3 69

70 Monitoring Database Resources Rules for Using the HA Database Monitor with ServiceGuard Sample File Settings The following is an example of how you might set up an ServiceGuard cluster, mycluster, with two nodes, nestle and whitman, to monitor the availability of a database, db_1, that is defined on a volume group, VG01, that can be accessed in exclusive mode by either nestle or whitman. Figure 3-2 shows the sample cluster setup. Figure 3-2 Two Node Cluster with HA Database Monitor Failover This sample illustrates a single database failover situation. You can also configure failover for multiple databases, for example: Configure two different packages where each has a dependency on the status of two different databases Configure one package with multiple dependencies In the above scenario, ServiceGuard handles the activation and mounting of the Oracle database, as well as starting the database instance for you. Setting Up the ServiceGuard Package This section provides an example, based on Figure 3-2, of setting the parameters in the ServiceGuard package configuration file. The sample lists the parameters for the HA Database Monitor, not all the 70 Chapter 3