Designing high-availability for Xen Virtual Machines with HP Serviceguard for Linux

Transcription

1 Designing high-availability for Xen Virtual Machines with HP Serviceguard for Linux Executive Summary... 3 Introduction... 3 Scope... 3 Support requirements... 4 Xen Host (Dom0) Configuration... 4 Xen Dom0 Installation and Configuration on x86_64 platforms... 4 Xen Dom0 Installation and Configuration on HP Integrity Server platforms... 5 Serviceguard for Linux Configuration... 6 Storage Configuration... 6 Network Configuration... 6 Secure Linux Settings... 7 Domain Memory Ballooning... 7 Serviceguard for Linux on Xen host... 7 RPM package dependency for installing Serviceguard for Linux Packaging the Xen virtual machine with Serviceguard for Linux... 8 Creation of Xen VM and dependencies... 8 Consolidate all Xen Guest VM files in one Volume Group... 8 Modify the Serviceguard for Linux Package Configuration File... 9 Basic Principles of Xen Control and Management Xen VM Startup Xen VM Shutdown Xen VM Monitoring Virtualization-mode independent approach Virtualization-mode dependent approach Pros and Cons of Independent vs. Dependent Approach Monitoring Xen virtual machine network interfaces Summary Requirements:... 13

2 For more information: Appendix Section I Xen Legacy Configuration File (xenhost.cnf) Appendix Section II Xen Legacy Control Script (xenhost.sh) Appendix Section III Xen Legacy Monitor Script (cmxenvmd) Appendix IV Table of Acronyms and Abbreviations... 22

3 3 Executive Summary This white paper describes how to integrate Xen Dom0 hosts into a Serviceguard for Linux (SG/LX) cluster, and how to configure Xen Virtual Machines (VMs) as SG/LX packages. It also makes recommendations for eliminating single points of failure and provides pointers to other useful documents. Introduction Virtual machines on Xen are increasingly being deployed for server consolidation and flexibility. Virtual machine technology allows one physical server to emulate multiple servers, each capable of running its own operating system (OS) concurrently with other virtual servers. The virtualization layer, also known as the hypervisor, abstracts the physical resources so that each instance of an OS appears to have its own network and storage adapters, processors, disks, and memory, when in fact they are virtual instances. A significant single point of failure for Xen virtual machines is the physical hardware which runs the Xen dom0 hypervisor. Dom0, or domain zero to expand the abbreviation, is the first domain started by the Xen hypervisor on boot. It has special privileges, such as being able to cause new domains to start, and being able to access the hardware directly. A failure of the Xen hypervisor or the server it is running on can bring down all of the Xen virtual machines running on that server. One can protect the Xen virtual machines from these failures by using SG/LX clustering to make the Xen hypervisor highly available. This document describes how to configure an SG/LX cluster consisting of multiple Xen dom0 hosts. The Xen virtual machines (VMs) can then be configured as SG/LX packages. In the event of a failure, or to maintain application availability while performing online upgrades and maintenance, a Xen virtual machine protected within a Serviceguard for Linux package can be restarted on the same node, or on another node in the cluster. The other nodes would also be Xen dom0 hypervisors running Serviceguard for Linux. This solution protects the Xen virtual machine from the following failures: Failure of a Xen virtual machine Failure of networking Failure of storage Failure of the physical machine (running as Xen dom0 host) Scope This document describes how to provide high availability for Xen virtual machines using Serviceguard for Linux running on multiple Xen dom0 hosts. See the Support Requirements section for a list of supported versions of Xen Server and Linux distributions. As new versions of Xen server or Linux distributions are certified, this whitepaper and the Serviceguard for Linux Certification Matrix will be updated accordingly. The most recent version of the Serviceguard for Linux Certification Matrix can be found at in the Solution planning section. Note: Serviceguard for Linux is currently certified to run on Xen dom0 hosts, not on Xen virtual machine guests. Applications running on the dom0 host, including the virtual machines can be configured to run as SG/LX packages. This configuration provides high availability to host based applications and virtual machines.

4 4 However, high availability for the virtual machine based applications is not guaranteed unless a monitoring technique is also used on the applications themselves. Serviceguard for Linux does not currently provide monitoring and high availability for applications running on the Xen virtual machine. You can develop alternative approaches to monitoring the applications running on the Virtual Machine, for example using a network-based or a file-based protocol between the guest and the host. A reasonable expertise in the installation and configuration of Xen server on x86_64 platforms and familiarity with its capabilities and limitations is assumed. It is also assumed that the reader is familiar with Serviceguard for Linux. Any special issues arising out of changes due to the Xen environment are discussed in following sections. Note: Except as noted in this whitepaper, all the Serviceguard for Linux configuration options documented in the Managing HP Serviceguard for Linux manual are supported for Xen hosts, and all documented requirements apply. You can find the latest version of the manual at High Availability Serviceguard for Linux. Support requirements The requirements for configuring highly-available Xen virtual machines with Serviceguard for Linux are: Serviceguard for Linux A or later HP ProLiant or non-hp x86_64-based server certified with Serviceguard for Linux A Linux distribution certified with Serviceguard for Linux that supports Xen virtualization technologies on the selected platform specifically, one of: o Red Hat Enterprise Linux 5 Update 1 and Update 2 Note: Xen Release 3.0 supported by Red Hat and SG/LX on RHEL5-x86_64 platforms. However SG/LX configuration to provide HA for VMs on RHEL5- Xen based-hp Integrity server platform is not supported. o SUSE Linux Enterprise Server Service Pack 1 and Service Pack 2 Note: Xen Release 3.0 is supported by Novell only on x86_64 platforms Citrix XenSource is not currently supported. Note: For the most recent list of servers and storage, Linux distributions and hypervisors certified for use with Serviceguard for Linux, see the Serviceguard for Linux Certification Matrix, available at Xen Host (Dom0) Configuration For detailed documentation on Xen installation see either the Red Hat Enterprise Linux Virtualization Guide [6] or Xen Installation section of the SUSE Linux 10 Reference Guide [5]. Xen Dom0 Installation and Configuration on x86_64 platforms The installation procedure for Xen involves the setup of a domain-0 domain and the installation of Xen clients. Before you install Xen, SLES10 SP1 or SP2 or RHEL5.1, 5.2 must be installed on the machine. Refer to RHEL5 Deployment Guide [8] or SLES10 Deployment Guide [10] for OS Installation.

5 5 You can use the command yast2 (on SLES10) or rpm (on RHEL5) to install additional rpms required for Xen. These rpms are : python python-virtinst libvirt bridge-utils libvirt-python dnsmasq xen-libs xen kernel-xen Xen is added to the GRUB configuration. The installation process places an entry in /boot/grub/menu.lst. This entry should be similar to the following: title Xen2 kernel (hd0,0)/boot/xen.gz dom0_mem= module (hd0,0)/boot/vmlinuz-xen <parameters> module (hd0,0)/boot/initrd-xen Reboot the machine to boot from Xen kernel. This sets up the Xen Host (or Domain 0) for SLES10/RHEL5- Xen. You can use console client virt-install or Linux GUI client virt-manager can to install client VMs on the domain0. Refer to the Xen User Manual [4] for installation, configuration and administration details for Xen. Xen Dom0 Installation and Configuration on HP Integrity Server platforms Only RHEL5 supports Xen on HP Integrity Server (Integrity) platforms. Be aware that during Serviceguard testing, there was at least one instance of a kernel panic related to Xen. This problem is not seen on x86_64 platforms. A bugzilla is filed for the issue at In such an instance Serviceguard will fail over any protected guests. Novell SUSE10 SP1, SP2 does not support the HP Integrity Server platform. Until Novell supports HP Integrity Server it cannot be supported with SG/LX. Before installing Xen on Integrity platforms read the whitepaper Xen-Based Virtualization Technology Preview for Red Hat Enterprise Linux 5.0 on HP Integrity servers [7] and follow the recommendations given in the current paper. By default, the hypervisor on Integrity systems reserves only 512 MB of memory for the dom0 instance. Dom0 will be allocated only one virtual CPU by default. The default value for memory and CPU must be increased to prevent delays under heavy load. A recommended value of 1 GB memory and 2 virtual CPUs for Dom0 is optimum.

6 6 Serviceguard for Linux Configuration Storage Configuration When you install the OS (xen-dom0), the fibre channel drivers qla1280.ko and qla2xxx/qla2xxx.ko do not get loaded by default. A modprobe on the driver module name results in Invalid module format. The procedure for loading the SCSI QLogic drivers is as follows: 1. Edit /etc/modprobe.conf, remove any qla* and scsi_adapter* alias lines. Then add alias scsi_adapter qla1280 alias scsi_adapter1 qla2xxx Rebuild the initrd file using the mkinitrd command 2. On RHEL5.1, 5.2 use the mkinitrd command with the listed arguments, to create an initrd mkinitrd -v /boot/initrd-(uname -r).img $(uname -r) On SUSE10 SP1, SP2 use mkinitrd command with no arguments to create an initrd in the /boot directory mkinitrd Network Configuration On RHEL5.1, 5.2 Xen dom0, a pseudo-network device with the name of virbr0 (default IP address: ) will appear in the ifconfig list. This device is configured and controlled by libvirt. The device does not serve any purpose other than to provide a private VM subnet. This is not necessary if the guest virtual machines are configured to use bridge networking. If the device is left configured (in the default state), SG/LX automatic network probing may report a possible network partition on the subnet, when it detects it as a potential heartbeat network. Since is meant to be a private subnet within Xen dom0 and not meant to be visible beyond the local dom0, we suggest deconfiguring the device. To unconfigure the device; remove the entries in the file default.xml located at: /etc/libvirt/qemu/networks/autostart/default.xml Leaving it configured does not affect the operation of Serviceguard for Linux in any other way other than what is mentioned above. SG/LX does not do automatic network probing for heartbeat networks and manually configured heartbeat networks will not have this issue. The virbr0 device is only seen on RHEL5.1 and RHEL5.2 platform. SLES10 SP1 and SLES10 SP2 do not bring up the device by default.

7 7 Secure Linux Settings On RHEL5, for successful creation of virtual machines, and also for creation of volume groups, it is required that appropriate role-based selinux rules be placed on the files and directories related to Xen and Serviceguard for Linux. On SLES10, apparmor provides equivalent role-based security for files and directories. For additional details on configuring selinux or apparmor, refer to RHEL5 [8] Deployment Guide or SLES10 Apparmor Administration Guide[9]. Domain Memory Ballooning Memory allocation to dom0 and domu can be modified using the max-mem and mem-set parameters. These parameters are used in conjunction with the xm commands. When a domu is running, the max-mem and mem-set can be modified dynamically. Use the xm mem-set command to increase the memory allocation for a dom0/domu xm mem-set <dom-id> <mem> The maximum memory allocation limit for a dom0 / domu can be set using the command xm mem-max <dom-id> <mem> For domu, expanding only works up to the amount initially allocated to a domain. However, it is possible to modify these parameters before a guest VM is powered up through the DomU configuration script. maxmem = 500 memory = 500 Memory Ballooning is useful when Serviceguard for Linux is installed on machines with similar architecture and mismatched hardware configuration. A Xen virtual machine package running with a given amount of memory cannot be failed over to a machine where at least that same amount of memory is not available. Using memory ballooning makes it possible to set the amount of memory used by the guest VM at the time of package startup on the other machine. Without this, the package may fail to start up on the recipient node. Serviceguard for Linux on Xen host This section describes installation and configuration tasks specific to Serviceguard for Linux running on a Xen host. The configuration aims to provide high availability to virtual machines running on the Xen host. The Serviceguard for Linux in-host model defines virtual machines as Serviceguard for Linux packages. RPM package dependency for installing Serviceguard for Linux To install SG/LX successfully on a Xen Dom0, the kernel-xen-devel rpm must be installed first. This package is required to build the deadman driver, which is compiled as part of SG/LX rpm installation on Linux. RHEL5.1, RHEL5.2, SLES10 SP1, and SLES10 SP2 provide the kernel-xen-devel rpm with the distribution. Note: Citrix XenSource is currently not supported.

8 8 Packaging the Xen virtual machine with Serviceguard for Linux Creation of Xen VM and dependencies A Xen VM guest can be created using virt-install (Redhat) or vm-install (Novell). The VM install script prompts for the following information at the time of installation: VM types: fully-virtualized (FV) or para-virtualized (PV). For detailed information on the VM types, refer to Xen Administration Guide [4] Storage: Virtual Disk or Physical Shared Disk as the primary disk for the guest OS Network: Virtual Network Interface (An Internet protocol address is configured for the Xen VM at the time of guest OS installation) Installation Repository for guest OS: nfs, ftp, http, cdrom, iso The virt-install/vm-install script creates a Xen VM guest configuration file to store configuration information for the guest VM. The virt-install/vm-install picks up the boot image from the install repository and boots the guest VM into install mode. The installation of the OS proceeds as usual on the guest VM. After installation of the guest OS, the guest VM can be started using the command xm create <absolute path to guest configuration file> and shutdown using the commands xm shutdown <name of guest configuration file> See the section Xen Control and Management to learn more about the management commands. To provide high availability to the Xen virtual machine guests running on the Xen host, the Xen VM must be registered as an SG/LX package. In order to do this, the following requirements must be met: The Xen VM guest configuration file must reside on a volume group on the shared storage. All default paths in the Xen VM guest configuration file must be adjusted to reflect the new location of the Xen VM files (virtual disks, config files etc.) If a Xen VM uses a virtual disk, it must reside on a volume group on the shared storage. If a Xen VM uses a physical disk, the physical disk must be located on the shared storage. This ensures that the Xen VM storage is available from other nodes when a Xen VM package fails over In the case of the SG/LX legacy package used in the examples in this paper, the package control script must be created, modified and applied. (See the latest version of Managing Serviceguard for Linux for information about legacy versus modular packages.) Consolidate all Xen Guest VM files in one Volume Group When creating a Serviceguard for Linux package for the Xen VM, you need to identify all of the resources used by the Xen VM and include them in the package.

9 9 The two main resources used by a Xen VM are as follows: Xen virtual disk file located in a directory specified by the user at time of VM creation Xen VM configuration file located in a sub-directory of /etc/xen/. Follow these steps to consolidate the VM resources: 1. Create a logical volume large enough to hold the virtual disk file and the guest configuration file. If the Xen VM uses a physical shared disk, then the logical volume would hold the configuration file. 2. Create a file system on the logical volume, for instance, an ext2 or ext3 file system using mke2fs or simply mkfs 3. Mount the logical volume. 4. Create identical mount points on other nodes of the Serviceguard for Linux cluster. These mount points would be used when the package is failed over to the other node. 5. Create a Xen VM guest using virt-install (Redhat) or vm-install (Novell). Specify the install options so that the VM virtual disk and the VM configuration file resides on the shared logical volume. In the case of a pre-created guest VM, copy the VM configuration files to the logical volume and edit the VM guest configuration file to change the path entries to reflect the new location of the VM guest files. 6. Unmount the logical volume. Modify the Serviceguard for Linux Package Configuration File An application can be packaged using the legacy method or the modular method. This white paper describes the legacy method. A modular package scheme will be released in the future. Legacy Method Create a package configuration file using the following command cmmakepkg v -p pkg.conf Create a package control script using the command cmmakepkg v -s pkg.cntl Modify the package control script to enter the start, stop functions in the control script. Appendix I lists the Xen configuration file (xenhost.cnf), which sets the environment for the Xen Control Script and the Xen Monitor Script. Refer to Appendix Section II for more details on the Xen control script (xenhost.sh). The Xen control script can be called from within the SG/LX package control script; define the following functions: function customer_defined_run_cmds { ADD customer defined run commands. : do nothing instruction, because a function must contain some command. ${XENPATH}/xenhost.sh start }

10 10 This function is a place holder for customer define functions. You should define all actions you want to happen here, after the service is halted. function customer_defined_halt_cmds { ADD customer defined halt commands. : do nothing instruction, because a function must contain some command. ${XENPATH}/xenhost.sh stop } Where ${XENPATH} is an absolute path to the Xen Legacy Scripts. Refer to Appendix III for details on the monitor script (cmxenvmd). The monitor script must be registered as a SERVICE_CMD in SG Package Control Script. Modify the package configuration file with the following entires: SERVICE_NAME cmxenvmd SERVICE_FAIL_FAST_ENABLED no SERVICE_HALT_TIMEOUT 300 The package control script must contain the following entries SERVICE_NAME[0]="cmxenvmd" SERVICE_CMD[0]="$SGSBIN/cmxenvmd" SERVICE_RESTART[0]="0" Note: Once a VM is launched as a package under Serviceguard for Linux, you should never start or stop a VM manually using the xm command. A manual start or stop of a virtual machine would lead to virtual machine package failure. If a VM is already started manually, it must be stopped before starting the VM through the package control script. Distribute the package control script and apply the package configuration using the command: cmapplyconf P pkg.conf Basic Principles of Xen Control and Management The Xen control and management script used the xm command to start, stop and monitor Xen virtual machine guests. Xen VM Startup A Xen VM can be started by using the command: xm create <xen vm guest config filename>

11 11 When a Xen VM guest is started, it first goes into a running state, which is then followed by a booted up state. It is necessary to wait for the guest to get into the booted up state before using the Xen virtual machine. This is usually significant when the Xen guest is started up using SG/LX package control script. The package control script expects the VM to get into a booted up state before it can perform further user-defined operations on the virtual machine. At the time of startup, specific run-time information is captured to enable monitoring of the VM machine. Various monitoring approaches are discussed in the monitoring section. Xen VM Shutdown A Xen VM can be gracefully shut down using the command: xm shutdown <config filename> After invocation of the command, it is necessary to wait until all the VM-specific daemons stop to achieve a clean shutdown. A clean shutdown ensures that the file systems and volume groups associated with the machine are successfully unmounted and deactivated respectively. The Xen virtual machine probe function can be used to check for a complete shutdown of the VM. Once the probe function declares the VM is no longer running, post shutdown steps can be performed. If a Xen VM is packaged as a SG/LX package, the post-shutdown steps performed are as follows: 1. Unmount the file systems 2. Deactivate the volume groups 3. Re-assign IP address (optional): The virtual machine IP address should not be registered with the package control script for monitoring. However, it is possible to monitor the subnet in which the virtual machine IP resides. The subnet may be monitored by placing an entry in the package configuration script: Example : MONITORED_SUBNET (netmask= ) These steps are performed by the SG/LX package control script. The Xen control script is callable from the SG/LX Package Control Script. Xen VM Monitoring Xen supports two modes of virtualization - Para-Virtualized (PV) and Fully-Virtualized (FV). The monitoring of a Xen VM is typically specific to the mode of virtualization used by the VM, but it is also possible to implement a technique which allows monitoring that is independent of the virtualization mode used. Virtualization-mode independent approach Xen Manager 'xm' commands can be used to monitor the Xen Virtual Machine (both full and paravirtualized VMs) through the 'xm list' commands. For example: xm list grep ${Virtual Machine Name} awk '{print $5}' grep -e "r-" -e "b-"

12 12 A return of '0' from the command indicates that the machine is in the 'booting up' state (represented by the r in the grep command) or in the 'booted up' (represented by the b ) state. The command can be called periodically to check the status of the virtual machine. Virtualization-mode dependent approach Fully-Virtualized mode For every Fully-Virtualized VM, a 'qemu-dm' process is spawned. It is possible to identify which instance of 'qemu-dm' belong to a particular VM with the following command: ps -ef grep "qemu-dm" grep ${Virtual Machine Name} Capture the PID of qemu-dm and touch a file and write the PID into a file. The monitoring can be done by probing this particular PID on regular intervals. Para-Virtualized mode No specific approach has been identified for Para-Virtualized Guests. Hence, usage of the virtualization-mode independent approach described above should be used. Pros and Cons of Independent vs. Dependent Approach The virtualization-mode dependent approach provides more accurate monitoring of the Fully- Virtualized VMs than the independent approach. The independent approach relies on the xm commands to gather monitoring information. There are limitations to this type of monitoring. The virtual machine name is removed from the 'xm list' before the qemu-dm process goes down. This small time gap may lead to an incorrect interpretation of the virtual machine status at that specific time. Since the qemu-dm process belongs to Fully-Virtualized VMs, the time gap only effects the Fully-Virtualized VM monitoring. For these reasons the virtualization-mode independent approach works best for Para-Virtualized guests as there are no latencies involved, and the virtualization-mode dependent approach is recommended for Fully-Virtualized VMs. Monitoring Xen virtual machine network interfaces Xen provides network availability to virtual machines through the virtual interfaces (vif) configured on a Xen dom0 host. A 'vif' interface is created for each virtual machine, at the time of startup. (using xm create command). The naming convention for the 'vif' interface is dynamic. vif<vmid>.<interface index> Replace the vmid and interface index values to get the correct name for the vm interface. The vmid of a virtual machine change everytime the virtual machine is halted and started up. Run the command listed below to extract the vmid using the name of the virtual machine. For example:

13 13 1. To get the virtual machine ID of the virtual machine s102vm1, use the command xm list grep s102vm1 awk {print $2} 2. Now use the vmid to get the name of the virtual interface that is mapped to the virtual machine on the dom0 host. vif < vmid >.<interface index> so, the derived interface is vif Probe the interface using the command ifconfig vif2.0 grep UP. vif2.0 Link encap:ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:805 errors:0 dropped:0 overruns:0 frame:0 TX packets:3744 errors:0 dropped:18 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:26624 (26.0 Kb) TX bytes: (241.5 Kb) The virtual machine network can be monitored from the host by simply probing the 'vif' interface associated with the VM. A failure of the physical interface or the virtual machine would bring the virtual interface down, thus notifying that the network on the virtual machine has failed. Serviceguard for Linux can then fail the package and restart it on a different node where the network is available. Repeat the steps listed above to identify the new virtual interface on the other node and monitor it accordingly. Summary Protecting Xen virtual machines with clustering provided by Serviceguard for Linux delivers lower total cost of ownership by enabling Xen virtual machines to be used for business critical applications. This consolidation leads to an overall reduced data center footprint. Requirements: Install kernel-xen-devel rpm on all nodes configured as Xen dom0 nodes. The rpm does not get installed by default on RHEL5. SLES10 installs the package by default when configured through Yast2 tool. The package is required for successful installation of the SG/LX rpm. The Xen Control and Management Script is necessary to automate the start, stop and monitor operations for a VM when configured as a Serviceguard for Linux package. Refer Appendix I, II and II for developing the Xen control and management script. Serviceguard for Linux require specific configurations at install time mentioned earlier. These requirements are mandatory to ensure proper operation of SG/LX on a Xen dom0 platform. Modification to the grub bootloader sequence is required to ensure that the machines boots into Xen by default.

14 14 For more information: 1. Serviceguard for Linux 2. Serviceguard for Linux Certification Matrix under Solution Planning 3. Managing HP Serviceguard for Linux, Eighth Edition, March Xen User Administration Guide 5. Xen Installation section of the SUSE Linux 10 Reference Guide e/data/sec_xen_inst.html 6. Red Hat Enterprise Linux Virtualization Guide Hat.com/docs/manuals/enterprise/ -> Virtualization Guide 7. Xen-Based Virtualization Technology Preview for Red Hat Enterprise Linux 5.0 on HP Integrity servers 8. Red Hat Enterprise Linux Deployment Guide US/index.html 9. Novell SLES 10 AppArmor Administration Guide age=/documentation/apparmor/apparmor201_sp2_admin/data/apparmor201_sp2_admi n.html 10. Novell SLES 10 Linux Deployment Guide n/sles10/sles_admin/data/sles_admin.html

15 15 Appendix Section I Xen Legacy Configuration File (xenhost.cnf) The file xenhost.cnf defines a set of configuration parameters for the Xen Legacy Control Script and Xen Legacy Monitor Script. The file xenhost.cnf along with the control script xenhost.sh and the monitor script cmxenvmd is used to package Xen VMs as SG/LX packages in Legacy Mode. (C) Copyright 2008 Hewlett-Packard Development Company, Serviceguard for Linux Attribute Definition Product Name : HP Serviceguard for Product Version : Patch Name : *** Note: This file MUST NOT be edited. ***** Any changes made to it will be overwritten when you upgrade to the next release of HP Serviceguard for Linux. Changing this file may lead to unrecoverable package configuration problems. "XEN_BIN" is the path of the Xen Management Binary. SG_XEN_BIN=/usr/sbin/xm "XEN_VM_PATH" is the path to the Xen Configuration File. SG_XEN_VM_PATH= "XEN_VM_NAME" is the name of the Xen Configuration File. SG_XEN_VM_NAME= "XEN_VM_PID_STATUS" is the absolute path to the Xen VM PID file. This file can be probed and may be useful for monitoring the VM. SG_XEN_VM_PID_STATUS= "XEN_RETRY_INTERVAL" is the time gap, the Xen Control Script waits to check if the Xen VM has halted completely. SG_XEN_RETRY_INTERVAL=10 "XEN_PROBE_TIMEOUT" The amount of time that must elapse before the VM probe function attempts a forced shutdown on the VM SG_XEN_PROBE_TIMEOUT=20

16 16 Appendix Section II Xen Legacy Control Script (xenhost.sh) The script was designed for the purpose of Xen VM Legacy Packages. The script functions may be called from a Legacy Package Control Script through customer_defined_run_cmds and customer_defined_halt_cmds. The script depends on the xenhost.cnf file for Xen environment setup. Template for xenhost.sh script (C) Copyright 2008 Hewlett-Packard Development Company, Serviceguard for Linux Xen Legacy Package Control Product Name : HP Serviceguard for Product Version : Patch Name : *** Note: This file MUST NOT be edited. ***** Any changes made to it will be overwritten when you upgrade to the next release of HP Serviceguard for Linux. Changing this file may lead to unrecoverable package startup and/or shutdown problems. Source SG and Xen Environment Setup set -a. ${SGCONFFILE:=/etc/cmcluster.conf}. ${SGXCONFFILE:=./xenhost.cnf} set +a echo ${SG_XEN_BIN} sg_validate_services() is called while validating the package configuration (through cmcheckconf/cmapplyconf) function sgx_validate_vm {

17 17 echo "sgx_validate_vm" typeset -i retval=0 validate if the package is run on xen host if [ -d /proc/xen ]; then grep -q control_d /proc/xen/capabilities case $? in 0) echo "Xen Dom0 Host found!" retval=0 ;; 1) echo "ERROR:: Xen Package Module cannot be configured on a Xen VM Guest" retval=1 ;; *) echo "ERROR:: Xen Environment Detection Failed" retval=255 ;; esac fi if (( retval!= 0 )) then echo "ERROR:" $retval " Failed to validate Xen VM Package" to_exit=1 fi } sg_xen_start_vm() is called while starting a xen vm package (cmrunpkg) For each {service name/service command string} pair, start the service command string at the service name using cmrunserv(1m) function sgx_start_vm { typeset -i retval=0 echo "sgx_start_vm" if [ -f ${SG_XEN_VM_PATH}${SG_XEN_VM_NAME} ]; then [ -x ${SG_XEN_BIN} ] ( echo "xm not found" && exit 255 ) ${SG_XEN_BIN} create ${SG_XEN_VM_PATH}${SG_XEN_VM_NAME} case $? in 0) echo "Xen VM ${SG_XEN_VM_NAME} successfully started!!"

18 18 xm else fi 1) *) esac retval=0 ;; An error occured while running xm echo "ERROR: Xen VM startup failed" retval=1 ;; xm command failed with an undocumented error or an error occured while starting echo "Unrecoverable error occured " retval=255 ;; echo "ERROR: 255 Xen VM configuration file not found!!!" exit 255 if (( retval!= 0 )) then echo "ERROR:" $retval " Function sgx_start_vm" echo "ERROR:" $retval " Failed to start xen vm" to_exit=1 fi } sg_xen_stop_vm() is called while stopping a xen vm package (cmhaltpkg) or during rollback operation to recover from the package startup. Halt each service using xm command. function sgx_stop_vm { typeset -i retval=0 echo "sgx_stop_vm" [ -x ${SG_XEN_BIN} ] ( echo "xm not found" && exit 255 ) echo "Halting xen vm ${SG_XEN_VM_NAME}" ${SG_XEN_BIN} shutdown ${SG_XEN_VM_NAME} case $? in 0) sgx_probe_vm_status echo "Xen VM ${SG_XEN_VM_NAME} successfully halted!!" retval=0 ;; 1)

19 19 An error occured while stopping vm echo "ERROR: Xen VM shutdown failed" retval=0 ;; *) xm command failed with an undocumented error echo "Unrecoverable error occured " retval=255 ;; esac } if (( retval!= 0 )) then echo "ERROR:" $retval " Function sgx_stop_vm" echo "ERROR:" $retval " Failed to halt xen vm ${SG_XEN_VM_NAME}" to_exit=1 fi sgx_probe_vm_status() is called to probe the status of a running vm; it exits only when the vm is completely halted. sgx_probe_vm_status is called by sgx_stop_vm to ensure the package is completely halted probe done using the 'xm list' command function sgx_probe_vm_status { typeset -i retval=0 typeset -i count=${sg_xen_probe_timeout} echo "sgx_probe_vm_status" "b" if [ -x ${SG_XEN_BIN} ]; then while [ 1 ]; do ${SG_XEN_BIN} list grep ${SG_XEN_VM_NAME} awk '{print $5}' grep -e "r" -e case $? in 0) Just wait on the Vm to go down completely count=count-1 if [ $count -le 0 ]; then force power down the VM; this is guaranteed to succeed echo "WARNING:: Attempting force shutdown of the VM" ${SG_XEN_BIN} destroy ${SG_XEN_VM_NAME} fi sleep ${SG_XEN_RETRY_INTERVAL} continue ;; 1)

20 20 } fi *) esac done exit only when Vm is completely halted return 1 ;; echo "ERROR: 255 Unrecoverable error occured " return 255 ;; main routine Module script must be specified with three required entry points: start, stop, and validate. The variable to_exit indicates the success or failure of the entry point. typeset -i to_exit=0 echo "sg_xen_vm" case ${1} in start) sgx_start_vm ;; stop) sgx_stop_vm ;; validate) sgx_validate_vm ;; *) echo "ERROR: Illegal entry point specification $1." to_exit=1 ;; esac Exit to indicate success/failure to the master_control_script.sh exit $to_exit

21 21 Appendix Section III Xen Legacy Monitor Script (cmxenvmd) The script is designed to monitor Xen VMs which are started using the Xen control script. The script depends on the Xen Configuration file to source Xen specific environment variables. The script must be registered as a SERVICE_CMD in the package control script. (C) Copyright 2008 Hewlett-Packard Development Company, Serviceguard for Linux - Xen VM monitor Product Name : HP Serviceguard for Product Version : Patch Name : *** Note: This file MUST NOT be edited. ***** Any changes made to it will be overwritten when you upgrade to the next release of HP Serviceguard for Linux. Changing this file may lead to unrecoverable package configuration problems. set -a. ${SGCONFFILE:=/etc/cmcluster.conf}. ${SGXENCONFFILE:=./xenhost.cnf} set +a wait for the VM to be up first time before monitoring run=0 if [ -x ${SG_XEN_BIN} ]; then while [ 1 ]; do ${SG_XEN_BIN} list grep ${SG_XEN_VM_NAME} awk '{print $5}' grep -e "r" -e "b" case $? in 0) run=1 sleep ${SG_XEN_RETRY_INTERVAL} continue ;; 1) exit only when Vm is completely halted if [ $run -eq 1 ]; then echo "Xen VM halted" exit 1 else continue fi fi ;; *) echo "Unrecoverable error occured " exit 255 ;; esac done

22 Appendix IV Table of Acronyms and Abbreviations Product Name Abbreviation Serviceguard Serviceguard for Linux Quorum Server Business Continuity and Availability High Availability Integrity Virtual Machines SUSE Linux Enterprise SUSE Linux Enterprise Server Red Hat Enterprise Linux Xen Domain0 Xen DomainU Virtual Machine SG or SG/UX SG/LX QS BC&A HA Integrity VM SUSE SLES RH or RHEL Dom0 DomU VM 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Itanium is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries. August