Using NFS as a file system type with HP Serviceguard A on HP-UX and Linux

Transcription

1 Technical white paper Using NFS as a file system type with HP Serviceguard A on HP-UX and Linux Table of contents Introduction 2 Audience 2 Serviceguard support for NFS on HP-UX and Linux 2 Overview 2 Supported configurations 2 Recommendations 2 Limitations 3 Setting up the NFS server 3 Configuring the NFS package and cluster parameters 3 Creating a sample NFS package 8 Sample cmviewcl line output 10 Terms and definitions 10 Conclusion 10 For more information 11

2 Introduction This white paper explains how to configure packages that need access to an NFS-imported file system as shared storage. This capability is available as of HP Serviceguard version A In previous releases, Serviceguard has supported only direct-attached storage, using technologies such as SCSI, Fibre Channel, or SAS. Note: Although HP Serviceguard A supports NFS-imported storage, direct attached storage is still recommended over NFS for performance reasons. Audience This document is for users of HP Serviceguard A who want to configure packages that import and mount remote file systems via NFS. HP also provides the HP Serviceguard NFS Toolkit, which provides high availability for an NFS server. This capability is different from what is described in this white paper; see the hp.com/go/hpux-serviceguard-manual documentation for more information. It is assumed that the reader has an understanding of HP Serviceguard. Serviceguard support for NFS on HP-UX and Linux Overview In HP Serviceguard A and earlier, the storage groups and file systems that were activated by the Serviceguard packages had to be available locally; remote mounting was not supported. This meant that an application could not be managed by Serviceguard if the application s storage was remote. As of A.11.20, HP Serviceguard can manage such applications via packages that import remote storage using the Network File System. This feature does not in itself provide High Availability (HA) for the NFS client or server, but HA for the NFS server is supported by the HP Serviceguard NFS Toolkit. Supported configurations Supported only on HP-UX 11i v3, Red Hat Enterprise Linux 5.6 and later, 6 and later. Supported only for modular-style failover packages (see chapter 6 of the latest version of Managing Serviceguard Manual for information about package types). Supported only on Serviceguard A.11.20, or later Recommendations If the storage is directly connected to all the nodes and shared, configure the storage as a local file system rather than using the NFS option. The same NFS share or files should not be written by multiple nodes or applications at the same time. A given NFS share should be mounted on only one mount point at any time. Only the cluster nodes may have access to the file systems they import. Do not use AutoFS on the cluster nodes that are configured to run the NFS package. Serviceguard assumes that an NFS share that is used as part of the Serviceguard package is not mounted on any node before the NFS package starts. If AutoFS mounts the NFS share on the Serviceguard node, the mount may fail on startup or failover. Disable CacheFS for all the file systems used by the Serviceguard NFS packages on all the cluster nodes that are configured to run the NFS package. 2

3 The NFS server itself should be highly available. The specific configuration will depend on the server platform being used. It is possible to have the package that uses the NFS file system as shared storage and the package that provides the NFS file system running in the same cluster. In such a configuration, the NFS client package must not depend on the NFS server package. If the NFS server package fails over, the client package need not be failed over. The NFS client package can continue to run and can reconnect to the NFS server once the NFS server package is up. Any NFS file systems used by the Serviceguard packages should be configured to restrict access to the nodes in the Serviceguard cluster. Serviceguard supports exclusive activation for volume groups for example, LVM volume groups. When you use NFS client-side locks, which HP recommends, exclusive activation is not available so you must make sure you follow these recommendations: Only the cluster nodes can have access to the file system. The file system is used by only one Serviceguard failover package. If the package fails, you never manually restart it without first ensuring that the file system has been properly unmounted. Limitations An NFS file system must be used by only one Serviceguard failover package. NFS file systems used by Serviceguard packages must be mounted with the -o llock mount option on HP-UX and -O local_lock=all on Linux to enforce local locking semantics on the NFS client. Mount NFS-imported file systems used by Serviceguard packages only as part of starting the package. A cluster node must not mount NFS shares that are configured as part of any Serviceguard package, as part of the boot up process; otherwise the package may fail when it starts or fails over. So that Serviceguard can verify that all I/O from a node on which a package has failed is flushed before the package starts on an adoptive node, all the switches and routers between the NFS server and client must support a worst-case timeout; after which, packets and frames are dropped. This timeout is known as the Maximum Bridge Transit Delay (MBTD). Switches and routers that do not support MBTD must not be used in a Serviceguard configuration. This might lead to delayed packets that could lead to data corruption. Networking among the Serviceguard nodes must be configured in such a way that a single failure in the network does not cause a package failure. Setting up the NFS server See the latest Serviceguard NFS Toolkit Administrator s Guide available at hp.com/go/hpux-serviceguard-docs, under HP Serviceguard NFS Toolkit for instructions on configuring the NFS server and shares. Configuring the NFS package and cluster parameters Configuring the NFS package parameters In the modular package configuration file, the new parameter fs_server specifies the name of the NFS server. The value of this parameter can be either the hostname of the NFS server or its IP address (both IPv4 and IPv6 addresses are supported). The NFS server can be configured on a different subnet or in a different domain than the Serviceguard cluster. fs_types specifies the file system type. Set this to NFS to use this feature. fs_mount_opt specifies the mount option. On HP-UX this must include -o llock in addition to any other options you specify. -o llock specifies local locking for the NFS file system. On Linux this must include O local_lock=all. fs_fsck_opt should not be used. If any option is found in fs_fsck_opt for an NFS-imported file system, a warning will be logged and the value will be ignored. 3

4 Configuring the cluster parameter CONFIGURED_IO_TIMEOUT_EXTENSION In a Serviceguard cluster in which NFS-imported file systems are used, an unlikely but possible scenario exists in which data corruption could occur. The scenario is as follows: 1. A Serviceguard package using an NFS file system ( NFSPkg ) is running on cluster node client-1 2. Node client-1 issues an NFS write request immediately before NFSPkg moves to another cluster node. 3. NFSPkg is started on the adoptive node client-2 4. Adoptive node client-2 begins sending NFS write requests to the same file and offset as the write request previously sent by client-1 just before the package was moved. 5. If the original NFS write request from client-1 arrives on the NFS server after the new write requests from client-2, the server would overwrite the data sent from client-2, thus resulting in data corruption. To prevent this, you must determine a maximum delay between when a write is issued from any Serviceguard node and when it can arrive at the NFS server. The following typical scenarios and illustrations could give you some guidance. The NFS write may go through network switches before it reaches the NFS server. In each switch, the packet will be dropped after some specific time has elapsed. The IEEE Bridge specification 802.1D, refers to this value as MBTD. Important: All switches and routers that are configured between the NFS server and Serviceguard nodes must support MBTD. You can calculate the lifetime of an NFS client s write packet by adding the MBTD value of all the switches and routers that are configured between the NFS server and the Serviceguard nodes. You must set the Serviceguard cluster parameter CONFIGURED_IO_TIMEOUT_EXTENSION for any cluster in which packages use NFS mounts. See the section on cluster configuration parameters in the Managing Serviceguard Manual for more information about CONFIGURED_IO_TIMEOUT_EXTENSION. To set the value for the CONFIGURED_IO_TIMEOUT_EXTENSION, first determine MBTD for each switch and router. The value should be in the vendors documentation. Set the CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of the values for the switches and routers. If there is more than one possible path between the NFS server and the cluster nodes, add the values for each path and use the largest number. The CONFIGURED_IO_TIMEOUT_EXTENSION value will increase with the increase in routers and switches there are between the NFS server and Serviceguard nodes. The cluster reformation time is increased by the CONFIGURED_IO_TIMEOUT_EXTENSION, so keep this value as low as possible by appropriate routing between the NFS server and Serviceguard nodes, or by using hardware that supports smaller MBTD values. The CONFIGURED_IO_TIMEOUT_EXTENSION parameter must also be set in some cases in an extended-distance cluster (EDC). See the discussion of this parameter in the Managing Serviceguard Manual for details. If packages use NFS imports in an EDC, calculate the settings for each case separately (that is, the value required for the EDC configuration, and the value required for NFS) and use the greater of the two values. For example, if the EDC configuration requires CONFIGURED_IO_TIMEOUT_EXTENSION to be (microseconds) and the NFS configuration requires it to be , set it to

5 In the example in figure 1, with only one switch between the NFS server and the NFS client set the CONFIGURED_IO_TIMEOUT_EXTENSION to the MBTD value of the switch in microseconds. In this case, if the MBTD of the switch is one second, set the CONFIGURED_IO_TIMEOUT_EXTENSION to Figure 1. The NFS client and the NFS server with one network switch configured between them 5

6 If there is more than one switch between the NFS server and the Serviceguard nodes, set CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of the MBTD values for each of the switches in the path (remember to use microseconds). In the example in figure 2, with two switches between the NFS server and the NFS client, set the CONFIGURED_IO_TIMEOUT_EXTENSION to the sum of all switches MBTD value in microseconds, which in this case is 1+1 = 2 seconds; so set CONFIGURED_IO_TIMEOUT_EXTENSION to Figure 2. The NFS client and the NFS server with two network switches configured between them 6

7 In the example in figure 3, there are multiple routes or paths between the NFS server and the NFS client, and the packet can travel by any path. Configure CONFIGURED_IO_TIMEOUT_EXTENSION to sum of the MBTD values in the longest route. Figure 3. The NFS client and the NFS server use more than one route to communicate When the NFS client writes packets from the NFS client to the NFS server, the data might travel through switch 1 and 2, or instead go through switch 3 and reach the NFS server. For Path 1 (via switch 1 and switch 2) the sum of the MBTD values is two seconds. For Path 2 (via switch 3) the MBTD value is one second. In this case, set CONFIGURED_IO_TIMEOUT_EXTENSION to Note: While configuring the parameter CONFIGURED_IO_TIMEOUT_EXTENSION, users have to ensure that the parameter HALT_SCRIPT_TIMEOUT is greater than CONFIGURED_IO_TIMEOUT_EXTENSION. 7

8 Creating a sample NFS package You can use Serviceguard Manager to create Serviceguard packages. If you decide instead to use Serviceguard commands to create packages, follow these steps to create a package that uses NFS-imported storage. Before you start: Verify that the cluster configuration adheres to the recommendations and guidelines in this white paper. Step 1: Set the CONFIGURED_IO_TIMEOUT_EXTENSION parameter as described in the section Configuring the cluster parameter CONFIGURED_IO_TIMEOUT_EXTENSION and apply the new configuration (see the section Reconfiguring the Cluster in the Managing Serviceguard Manual). Step 2: Create a modular failover package. In this example, we call the output file pkg.ascii, and only file system, service, and failover modules are included. Include additional modules as required. truck:/>cmmakepkg -m sg/failover -m sg/filesystem -m sg/service pkg.ascii Package template is created. This file must be edited before it can be used. Step 3: Edit the pkg.ascii file as follows for the NFS file system parameters. Do not set fs_fsck_opt for an NFS file system. The fs_umount_opt can be set as required. fs_name fs_server fs_directory fs_type fs_mount_opt #fs_umount_opt #fs_fsck_opt /var/opt/nfs/share1 wagon /nfs/mnt/share1 nfs -O local_lock=all Follow the instructions in chapter 6 of Managing Serviceguard Manual to configure the other package parameters. Step 4: Verify that the NFS server is available and the NFS share to be used in the NFS package is shared by the NFS server. The showmount e <NFS SERVER> command can be used for the verification. The NFS server must be accessible and the share should be available on the NFS server before applying the package configuration. A sample execution of the command will be as follows. truck:/>showmount -e wagon export list for wagon: /var/opt/nfs/share2 (everyone) /var/opt/nfs/share1 (everyone) Step 5: Check the package configuration file. truck:/>cmcheckconf -v -P pkg.ascii Begin package verification... Checking existing configuration... Done Attempting to add package NFSPkg. Validating package NFSPkg via /etc/cmcluster/scripts/mscripts/master_control_script.sh... Waiting for up to 1200 seconds for the validation. Validation for package NFSPkg succeeded via /etc/cmcluster/scripts/mscripts/master_control_script.sh. Maximum configured packages parameter is 300. Configuring 1 package(s). Adding the package configuration for package NFSPkg. cmcheckconf: Verification completed with no errors found. Use the cmapplyconf command to apply the configuration 8

9 Step 6: Apply the package configuration file. truck:/>cmapplyconf -v -P pkg.ascii Begin package verification... Checking existing configuration... Done Attempting to add package NFSPkg. Validating package NFSPkg via /etc/cmcluster/scripts/mscripts/master_control_script.sh... Waiting for up to 1200 seconds for the validation. Validation for package NFSPkg succeeded via /etc/cmcluster/scripts/mscripts/master_control_script.sh. Maximum configured packages parameter is 300. Configuring 1 package(s). Adding the package configuration for package NFSPkg. Modify the package configuration ([y]/n)? y Completed the cluster update Step 7: Start the cluster if it is not already running. Run the package. truck:/>cmrunpkg NFSPkg Running package NFSPkg on node blocks Successfully started package NFSPkg on node blocks cmrunpkg: All specified packages are running Step 8: Verify that the package is up and running. truck:/>cmviewcl -p NFSPkg PACKAGE STATUS STATE AUTO_RUN NODE NFSPkg up running enabled blocks 9

10 Sample cmviewcl line output The cmviewcl line output is enhanced to display the NFS server. truck:/>cmviewcl v fline -p NFSPkg... package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 filesystem=[wagon] /var/opt/nfs/share1 package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_name=/var/opt/ nfs/share1 package:nfspkg filesystem:[wagon/var/opt/nfs/share1 fs_server=wagon package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_directory=/nfs /mnt/share1 package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_type= nfs package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_mount_opt= -O local_lock=all package:nfspkg filesystem:[wagon]/var/opt/nfs/share1 fs_umount_opt= package: NFSPkg filesystem:[wagon]/var/opt/nfs/share1 fs_fsck_opt=... Terms and definitions Term HA MBTD NFS NFS client NFSPkg NFS server Definition High Availability Maximum Bridge Transit Delay part of the IEEE 802.1D network specification Network File System A system that mounts and accesses the NFS server s shared file systems. In the context of this white paper, the NFS client is one of the Serviceguard cluster nodes A Serviceguard failover package that uses an NFS file system shared by an NFS server A system that shares its file systems over a network Conclusion This document helps Serviceguard users in configuring packages that need access to an NFS-imported file system as shared storage, thus enabling customers to scale their storage capabilities. 10

11 For more information To learn more, please visit: HP Serviceguard Solutions for HP-UX 11i: hp.com/go/serviceguardsolutions For technical documentation: HP Serviceguard Solutions for HP-UX 11i: hp.com/go/hpux-serviceguard-docs HP Serviceguard Solutions for Linux: hp.com/go/linux-serviceguard-docs Managing Serviceguard Manual: hp.com/go/hpux-serviceguard-docs HP Serviceguard User Guide Managing Serviceguard Manual: hp.com/go/linux-serviceguard-docs HP Serviceguard User Guide HP Serviceguard NFS Toolkit: hp.com/go/hpux-serviceguard-docs HP Serviceguard NFS Toolkit HP Serviceguard NFS Toolkit: hp.com/go/linux-serviceguard-docs HP Serviceguard NFS Toolkit Latest NFS Serviceguard NFS Toolkit Administrator s Guide is available at: hp.com/go/hpux-serviceguard-docs HP Serviceguard NFS Toolkit on configuring the NFS server and shares. HP welcomes your input. Please give us comments about this white paper, or suggestions for related documentation, through our technical documentation feedback website: hp.com/bizsupport/feedback/ww/webfeedback.html. Manage remote applications via packages that import remote storage using the Network File System, featured in the NFS server and supported by the HP Serviceguard NFS Toolkit. Visit: hp.com/go/sglx to know more. Get connected hp.com/go/getconnected Current HP driver, support, and security alerts delivered directly to your desktop Copyright Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. 4AA2-7632ENW, Created September 2010; Updated July 2012, Rev. 2 11