Using NFS as a filesystem type with HP Serviceguard A on HP-UX 11i v3

Transcription

1 Using NFS as a filesystem type with HP Serviceguard A on HP-UX 11i v3 Technical white paper Table of contents Introduction... 2 Audience... 2 Terms and Definitions... 2 Serviceguard support for NFS on HP-UX 11i v Overview... 2 Supported configurations... 3 Recommendations... 3 Limitations... 3 Setting up the NFS server... 4 Configuring the NFS package and cluster parameters... 4 Creating a sample NFS package... 9 Sample cmviewcl line output For more information Call to action... 12

2 Introduction This white paper explains how to configure packages that need access to an NFS-imported filesystem as shared storage. This capability is available as of HP Serviceguard version A.11.20; in previous releases, Serviceguard has supported only direct-attached storage, using technologies such as SCSI, Fibre Channel, or SAS. Note: Although HP Serviceguard A supports NFS-imported storage, direct attached storage is still recommended over NFS for performance reasons. Audience This document is for users of HP Serviceguard A who want to configure packages that import and mount remote filesystems via NFS. HP also provides the HP Serviceguard NFS toolkit, which provides high availability for an NFS server. This capability is different from what is described in this white paper; see the HP Serviceguard NFS Toolkit documentation for more information. It is assumed that the reader has an understanding of HP Serviceguard. Terms and Definitions Term HA MBTD NFS NFS client Definition High Availability Maximum Bridge Transit Delay part of the IEEE 802.1D network specification Network Filesystem A system that mounts and accesses the NFS server s shared filesystems. In the context of this white paper, the NFS client is one of the Serviceguard cluster nodes NFSPkg NFS server A Serviceguard failover package that uses an NFS filesystem shared by an NFS server A system that shares its filesystems over a network Serviceguard support for NFS on HP-UX 11i v3 Overview In HP Serviceguard A and earlier, the storage groups and filesystems which are activated by the Serviceguard packages must be available locally; remote mounting is not supported. This has meant that an application could not be managed by Serviceguard if the application s storage was remote. As of A.11.20, HP Serviceguard can manage such applications via packages that import remote storage using the Network Filesystem. This feature does not in itself provide HA for the NFS client or server, but HA for the NFS server is supported by the HP Serviceguard NFS Toolkit. 2

3 Supported configurations Supported only on HP-UX 11i v3. Supported only for modular-style failover packages (see chapter 6 of the latest version of Managing Serviceguard for information about package types). Supported only on Serviceguard A or later Recommendations If the storage is directly connected to all the nodes and shared, configure the storage as a local filesystem rather than using via NFS. The same NFS share or files should not be written by multiple nodes or applications at the same time. A given NFS share should be mounted on only one mount point at any time. Only the cluster nodes may have access to filesystems they import. Do not use AutoFS on the cluster nodes which are configured to run the NFS package. Serviceguard assumes that an NFS share which is used as part of the Serviceguard package is not mounted on any nodes before the NFS package starts. If AutoFS mounts the NFS share on the Serviceguard node, the mount may fail on startup or failover. Disable CacheFS for all the filesystems used by the Serviceguard NFS packages on all the cluster nodes which are configured to run the NFS package. The NFS server itself should be highly available. The specific configuration will depend on the server platform being used. It is possible to have the package that uses the NFS filesystem as shared storage and the package that provides the NFS filesystem running in the same cluster. In such a configuration, the NFS client package must not depend on the NFS server package. If the NFS server package fails over, the client package need not be failed over. The NFS client package can continue to run and can reconnect to the NFS server once the NFS server package is up. Any NFS filesystems used by the Serviceguard packages should be configured to restrict access to the nodes in the Serviceguard cluster. Serviceguard supports exclusive activation for volume groups for example, LVM volume groups. When you use NFS client-side locks, which HP recommends, exclusive activation is not available, so you must make sure you follow these recommendations: Only the cluster nodes can have access to the filesystem. The filesystem is used by only one Serviceguard failover package. If the package fails, you never manually restart it without first ensuring that the filesystem has been properly unmounted. Limitations An NFS filesystem must be used by only one Serviceguard failover package. NFS filesystems used by Serviceguard packages must be mounted with the llock mount option to enforce local locking semantics on the NFS client. Mount NFS-imported filesystems used by Serviceguard packages only as part of starting the package. A cluster node must not mount NFS shares that are configured as part of any Serviceguard package as part of the boot up process; otherwise the package may fail when it starts or fails over. 3

4 So that Serviceguard can ensure that all I/O from a node on which a package has failed is flushed before the package starts on an adoptive node, all the switches and routers between the NFS server and client must support a worst-case timeout, after which packets and frames are dropped. This timeout is known as the Maximum Bridge Transit Delay (MBTD). Switches and routers that do not support MBTD must not be used in a Serviceguard configuration. This might lead to delayed packets that could lead to data corruption. Networking among the Serviceguard nodes must be configured in such a way that a single failure in the network does not cause a package failure. Setting up the NFS server See the NFS Services Administrator s Guide for instructions on configuring the NFS server and shares. Configuring the NFS package and cluster parameters Configuring the NFS package parameters In the modular package configuration file, the new parameter fs_server specifies the name of the NFS server. The value of this parameter can be either the hostname of the NFS server or its IP address (both IPv4 and IPv6 addresses are supported). The NFS server can be configured on a different subnet or in a different domain than the Serviceguard cluster. fs_types specifies the filesystem type. Set this to NFS to use this feature. fs_mount_opt specifies the mount option. This must include -o llock in addition to any other options you specify. -o llock specifies local locking for the NFS filesystem. fs_fsck_opt should not be used. If any option is found in fs_fsck_opt for an NFS-imported filesystem, a warning will be logged and the value will be ignored. Configuring the cluster parameter CONFIGURED_IO_TIMEOUT_EXTENSION In a Serviceguard cluster in which NFS-imported filesystems are used, an unlikely but possible scenario exists in which data corruption could occur. The scenario is as follows: 1. A Serviceguard package using an NFS filesystem ( NFSPkg ) is running on cluster node client-1 2. Node client-1 issues an NFS write request immediately before NFSPkg moves to another cluster node. 3. NFSPkg is started on the adoptive node client-2 4. Adoptive node client-2 begins sending NFS write requests to the same file and offset as the write request previously sent by client-1 just before the package was moved. 5. If the original NFS write request from "client-1" arrives on the NFS server after the new write requests from client-2, the server would overwrite the data sent from client-2, thus resulting in data corruption. 4

5 To prevent this, you must determine a maximum delay between when a write is issued from any Serviceguard node and when it can arrive at the NFS server. Use the discussion that follows for guidance. The NFS write may go through network switches before it reaches the NFS server. In each switch, the packet will be dropped after some specific time has elapsed. The IEEE Bridge specification, 802.1D, refers to this value as Maximum Bridge Transit Delay (MBTD). IMPORTANT: All switches and routers that are configured between the NFS server and Serviceguard nodes must support MBTD. You can calculate the lifetime of an NFS client s write packet by adding the MBTD value of all the switches and routers that are configured between the NFS server and the Serviceguard nodes. You must set the Serviceguard cluster parameter CONFIGURED_IO_TIMEOUT_EXTENSION for any cluster in which packages use NFS mounts. See the section on cluster configuration parameters in chapter 4 of the Managing Serviceguard manual for more information about CONFIGURED_IO_TIMEOUT_EXTENSION. To set the value for the CONFIGURED_IO_TIMEOUT_EXTENSION, first determine MBTD for each switch and router. The value should be in the vendors' documentation. Set the CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of the values for the switches and routers. If there is more than one possible path between the NFS server and the cluster nodes, sum the values for each path and use the largest number. The CONFIGURED_IO_TIMEOUT_EXTENSION value will be greater the more routers and switches there are between the NFS server and Serviceguard nodes. The cluster reformation time is increased by the CONFIGURED_IO_TIMEOUT_EXTENSION, so keep this value as low as possible by appropriate routing between the NFS server and Serviceguard nodes or by using hardware that supports smaller MBTD values. The CONFIGURED_IO_TIMEOUT_EXTENSION parameter must also be set in some cases in an extended-distance cluster (EDC); see the discussion of this parameter in Managing Serviceguard for details. If packages use NFS imports in an EDC, calculate the settings for each case separately (that is, the value required for the EDC configuration, and the value required for NFS) and use the greater of the two values. For example, if the EDC configuration requires CONFIGURED_IO_TIMEOUT_EXTENSION to be (microseconds) and the NFS configuration requires it to be , set it to In the example in Figure 1, with only one switch between the NFS server and the NFS client, set the CONFIGURED_IO_TIMEOUT_EXTENSION to the MBTD value of the switch in microseconds. In this case is the MBTD of the switch is 1 second, so set CONFIGURED_IO_TIMEOUT_EXTENSION to

6 Figure 1: The NFS client and the NFS server with one network switch configured between them If there is more than one switch between the NFS server and the Serviceguard nodes, set CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of the MBTD values for each of the switches in the path (remember to use microseconds). In the example in Figure 2, with two switches between the NFS server and the NFS client, set the CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of all switches MBTD value in microseconds, which in this case is 1+1 = 2 seconds; so set CONFIGURED_IO_TIMEOUT_EXTENSION to

7 Figure 2: The NFS client and the NFS server with two network switches configured between them In the example in Figure 3, there are multiple routes or paths between the NFS server and the NFS client, and the packet can travel by any path. Configure CONFIGURED_IO_TIMEOUT_EXTENSION to sum of the MBTD values in the longest route. 7

8 Figure 3: The NFS client and the NFS server use more than one route to communicate When the NFS client writes packets from the NFS client to the NFS server, the data might travel through switch 1 and 2, or instead go through switch 3 and reach the NFS server. For path 1 (via switch 1 and switch 2) the sum of the MBTD values is 2 seconds. For path 2 (via switch 3) the MBTD value is 1 second. In this case, set CONFIGURED_IO_TIMEOUT_EXTENSION is to

9 Creating a sample NFS package You can use Serviceguard Manager to create Serviceguard packages. If you decide instead to use Serviceguard commands to create packages, follow these steps to create a package that uses NFSimported storage. Before you start: Verify that the cluster configuration adheres to the recommendations and guidelines in this white paper. Step 1: Set the CONFIGURED_IO_TIMEOUT_EXTENSION parameter as described in section 4.3.1, and apply the new configuration (see the section Reconfiguring the Cluster in chapter 7 of Managing Serviceguard). Step 2: Create a modular failover package. In this example, we call the output file pkg.ascii, and only filesystem, service, and failover modules are included. Include additional modules as required. truck:/>cmmakepkg -m sg/failover -m sg/filesystem -m sg/service pkg.ascii Package template is created. This file must be edited before it can be used. Step 3: Edit the pkg.ascii file as follows for the NFS filesystem parameters. Do not set fs_fsck_opt for an NFS filesystem. The fs_umount_opt can be set as required. fs_name /var/opt/nfs/share1 fs_server wagon fs_directory /nfs/mnt/share1 fs_type nfs fs_mount_opt -o llock #fs_umount_opt #fs_fsck_opt Follow the instructions in chapter 6 of Managing Serviceguard to configure the other package parameters. Step 4: Verify that the NFS server is available and the NFS share to be used in the NFS package is shared by the NFS server. The showmount e <NFS SERVER> command can be used for the verification. The NFS server must be accessible and the share should be available on the NFS server before applying the package configuration. A sample execution of the command will be as follows. truck:/>showmount -e wagon export list for wagon: /var/opt/nfs/share2 (everyone) /var/opt/nfs/share1 (everyone) 9

10 Step 5: Check the package configuration file. truck:/>cmcheckconf -v -P pkg.ascii Begin package verification... Checking existing configuration... Done Attempting to add package NFSPkg. Validating package NFSPkg via /etc/cmcluster/scripts/mscripts/master_control_script.sh... Waiting for up to 1200 seconds for the validation. Validation for package NFSPkg succeeded via /etc/cmcluster/scripts/mscripts/master_control_script.sh. Maximum configured packages parameter is 300. Configuring 1 package(s). Adding the package configuration for package NFSPkg. cmcheckconf: Verification completed with no errors found. Use the cmapplyconf command to apply the configuration Step 6: Apply the package configuration file. truck:/>cmapplyconf -v -P pkg.ascii Begin package verification... Checking existing configuration... Done Attempting to add package NFSPkg. Validating package NFSPkg via /etc/cmcluster/scripts/mscripts/master_control_script.sh... Waiting for up to 1200 seconds for the validation. Validation for package NFSPkg succeeded via /etc/cmcluster/scripts/mscripts/master_control_script.sh. Maximum configured packages parameter is 300. Configuring 1 package(s). Adding the package configuration for package NFSPkg. Modify the package configuration ([y]/n)? y Completed the cluster update 10

11 Step 7: Start the cluster if it is not already running. Run the package. truck:/>cmrunpkg NFSPkg Running package NFSPkg on node blocks Successfully started package NFSPkg on node blocks cmrunpkg: All specified packages are running Step 8: Verify that the package is up and running. truck:/>cmviewcl -p NFSPkg PACKAGE STATUS STATE AUTO_RUN NODE NFSPkg up running enabled blocks Sample cmviewcl line output The cmviewcl line output is enhanced to display the NFS server. truck:/>cmviewcl v fline -p NFSPkg... package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 filesystem=[wagon ]/var/opt/nfs/share1 package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_name=/var/opt/ nfs/share1 package:nfspkg filesystem:[wagon/var/opt/nfs/share1 fs_server=wagon package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_directory=/nfs /mnt/share1 package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_type= nfs package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_mount_opt= -o llock package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_umount_opt= package: NFSPkg filesystem:[wagon]/var/opt/nfs/share1 fs_fsck_opt=... 11

12 For more information To learn more, please visit: HP Serviceguard Solutions for HP-UX 11i: For technical documentation: HP Serviceguard Solutions for HP-UX 11i: Managing Serviceguard Manual: HP Serviceguard User Guide HP Serviceguard NFS Toolkit: HP Serviceguard NFS Toolkit HP Serviceguard NFS Toolkit Administrator's Guide: Call to action HP welcomes your input. Please give us comments about this white paper, or suggestions for related documentation, through our technical documentation feedback website: Share with colleagues Copyright 2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. 4AA2-7632ENW, Created September 2010