PRIMECLUSTER DR/PCI Hot Plug User s Guide

Transcription

1 PRIMECLUSTER DR/PCI Hot Plug User s Guide - SPARC Solaris - Version 4.2 November, 2003

2 Introduction This manual describes how to use the Dynamic Reconfiguration (DR) function and the PCI Hot Plug function on a PRIMECLUSTER product Single-node system A single-node system, which does not contain a cluster software product but redundant disk units or network, support the following: - Hot swap of the system board to which I/O is connected - Hot swap, hot expansion, and movement between partitions of the system board* on which memory and the CPU are mounted - Hot swap and hot expansion of the PCI card For further details, see 1.3 General Topology of DR and 1.4 General Topology of PCI Hot Plug. Cluster system A cluster system, which uses a cluster software product (PRIMECLUSTER) supports the following: - Hot swap, hot expansion, and movement between partitions of the system board* on which memory and the CPU are mounted - Hot swap and hot expansion of the PCI card The above operations are enabled even on a board on which an I/O device is mounted by disabling that I/O device. See Preset. Readers The document is intended for all users who use PRIMEPOWER450 (pedestal or rack-mount types) (7U))/800/900/1000/1500/2000/2500 server types. PRIMEPOWER450, as referred to in this manual, indicates a pedestal or rack-mount type (7U). PRIMEPOWER1500, as referred to in this manual, indicates a model in which SPARC64 V is mounted. Target product The document covers the following products PRIMECLUSTER GDS (hereinafter abbreviated to GDS) PRIMECLUSTER GFS (hereinafter abbreviated to GFS) PRIMECLUSTER GLS (hereinafter abbreviated to GLS) PRIMECLUSTER Related document Dynamic Reconfiguration User's Guide (Hereinafter abbreviated to DR User's Guide. ) Dynamic Reconfiguration User s Guide I/O device edition (Hereinafter abbreviated to DR User s Guide I/O device edition. ) PCI Hot Plug User's Guide PCI Hot Plug User's Guide I/O device edition Trademarks UNIX is a trademark or registered trademark of Open Group in the United States and other countries. Sun, Sun Microsystems, the Sun logo, Solaris and all trademarks and logos related to Solaris are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. All SPARC trademarks licensed by US-based SPARC International, Inc. are trademarks or registered trademarks of SPARC International, Inc. in the US and other countries. Products with SPARC trademarks attached are based on the architecture developed by US-based Sun Microsystems. Other company names and product names are trademarks or registered trademarks of their respective companies. Request No part of this document may be reproduced or copied without the prior permission of FUJITSU LIMITED. All Rights Reserved, Copyright FUJITSU 2001,2002,2003 2

3 CONTENTS 1. OVERVIEW Target Products and Versions of DR Target Products and Versions of PCI Hot Plug General Topology of DR General Topology of PCI Hot Plug PRIMECLUSTER Support of DR PRIMECLUSTER Support of PCI Hot Plug GDS Prerequisites Conditions Common to DR and PCI Hot Plug Condition Unique to DR Hot-swapping System Board using DR Separating the System Board Reconnecting the System Board Hot-Swapping the PCI Card with PCI Hot Plug Disconnecting a Disk Unit Separating, Swapping, and Incorporating a PCI Card Reconnecting a Disk Unit Hot Expansion of the PCI Card with PCI Hot Plug GLS Prerequisites for DR DR Associated Script Interface Hot Swap and Hot Expansion of NIC with PCI Hot Plug GFS LOCAL FILE SYSTEM Separating the System Board with DR Handling an I/O Error Association between DR Separation Target System and Slice Name Confirming the Multi-partition Configuration with the sfxinfo Command Hot Swapping the PCI Card with PCI Hot Plug Handling an I/O Error Association between PCI Hot Plug Swap Target PCI Card and Slice Name Confirming the Multi-partition Configuration PRIMECLUSTER DR Function Preset

4 5.1.2 Notes PCI Hot Plug Function Swapping and Expanding NIC with PCI Hot Plug Using PCI Hot Plug to Swap or Expand a PCI Card Connected to a Shared Disk

5 1. Overview 1.1 Target Products and Versions of DR The document covers the following products and versions that operate in a Solaris 8 or Solaris 9 environment of PRIMEPOWER800/900/1000/1500/2000/2500 units: PRIMECLUSTER Storage Suite (PRIMECLUSTER GDS/GFS), PRIMECLUSTER GLS 4.1A10 or later PRIMECLUSTER Enterprise Edition, HA Server 4.1A10 or later 1.2 Target Products and Versions of PCI Hot Plug This manual covers those products and versions that run, in a Solaris 8 or Solaris 9 environment, on PRIMEPOWER450 (pedestal or rack-mount type (7U))/900/1500 (with the SPARC64 V) /2500 units, as shown below: The document covers the following products and versions that operate in a Solaris 8 or Solaris 9 environment of PRIMEPOWER450 (pedestal or rack-mount type (7U))/900/1500 (with the SPARC64 V) /2500 units: PRIMECLUSTER Storage Suite (PRIMECLUSTER GDS/GFS), PRIMECLUSTER GLS 4.1A10 or later PRIMECLUSTER Enterprise Edition, HA Server 4.1A10 or later 1.3 General Topology of DR The Dynamic Reconfiguration (DR) is used to attach or detach a system board operating in a partition without disrupting ongoing operations. The DR can be used as follows: - Hot swap of failed component Hot swap* of a system board to replace the I/O device, memory, or CPU that has failed. - Hot expansion of the hardware resources (system board) Addition of a new system board to add a CPU, memory, or an I/O device to the partition. - Transfer of the hardware resources (system board) and system board pool Transferring the system board between partitions that are themselves divided into two or more partitions, as required. To this end, the system board pool function is used to pool a specific system board such that it belongs to none of these partitions. DR is enabled even on a board on which an I/O device is mounted by disabling that I/O device (See 5.1 Preset ). * If you are adding a system board that has previously been deleted, the I/O mounting state (the type of I/O controller and I/O device, and the location of the mounted slots) must be the same as that before the deletion. 1.4 General Topology of PCI Hot Plug The PCI Hot Plug is used to add or remove/separate a PCI card to/from the Solaris operating environment without rebooting a system. The system resource on the PCI card is an I/O device. The PCI Hot Plug can be used as follows: Hot swap of PCI card Swapping or removing of the PCI card, in which a failure has occurred (or may occur), during system operation. Hot expansion of the PCI card Addition of a new PCI card during system operation. 1.5 PRIMECLUSTER Support of DR PRIMECLUSTER products support the following topologies with DR. The range of support varies depending on whether the I/O is mounted on the system board on which DR is performed. 5

6 Hot swap (*1) Hot expansion Transfer/SB pool Without I/O With I/O Without I/O With I/O Without I/O With I/O GDS yes yes(*2) yes no yes no GLS GFS PRIMECLUSTER Yes(*7) yes yes (*5)(*6) yes (*3)(*7) Depend on the situation (*4) yes no Yes(*7) no yes no yes no no yes(*5) no yes (*5)(*6) (*1) If the system board, for which hot swap is to be performed, constitutes multiple partitions by the use of expansion system boards (XSB), all the XSBs must be separated from the partitions. (*2) For a system board to which a disk unit that is mirrored with GDS is connected, you must confirm the prerequisites for GDS. For information on the prerequisites and procedures, see Chapter 2. (*3) If a Network Interface Card (NIC) that is redundant with GLS is mounted on the system board, you must confirm the prerequisites for GLS. For information on the prerequisites and procedures, see Chapter 3. (*4) If a disk unit having a non-redundant I/O configuration is connected to the system board and is used for GFS, you must stop (umount) the use ofgfs. For details, see Chapter 4. (*5) There are cautions to be observed. For details, see Chapter 5. (*6) The kernel memory board cannot be removed (exchanged or transferred), regardless of the machine type. (*7)The kernel memory board cannot be removed (exchanged or transferred) on PRIMEPOWER 900/1500/2500 systems where the Redundant Line Control Function of GLS is running. For the removal (exchange or transferring) of the kernel memory board on PRIMEPOWER 900/1500/2500 systems where the Multipath Function of GLS is running, be sure to apply the patch or later of the Multipath Function. no 1.6 PRIMECLUSTER Support of PCI Hot Plug PRIMECLUSTER supports swapping or expansion with the PCI Hot Plug function of the PCI card connected to the unit, as shown below. Single node system Node type Part of a cluster system Unit to RAID device yes yes which PCI card is Public LAN yes yes(*1)(*2) connected Interconnect no (*1) The CF setup must have been completed. (*2) Using the PCI Hot Plug, you cannot change NIC of a public LAN when the Scalable Internet Services (SIS) is used (SMAWdtcp is installed). 6

7 2. GDS The Global Disk Services (GDS) allows you to execute DR without disrupting ongoing operations by mirroring the disk units that are connected to different system boards. Similarly, GDS allows you to perform hot swap and hot expansion of the PCI card by using PCI Hot Plug without disrupting ongoing operations. This part describes how to support the following operations: Hot swap of the system board by using DR (only for root or local class) Hot swap of the PCI card by using PCI Hot Plug Hot expansion of PCI card by using PCI Hot Plug 2.1 Prerequisites To enable hot swap and hot expansion for Global Disk Services (GDS) with DR and PCI Hot Plug, a disk unit must satisfy the following conditions Conditions Common to DR and PCI Hot Plug 1. Required patch To perform DR or PCI Plug using GDS you need to apply the following patch: GDS : or later 2. All parts excluding the failed portion must be reconnected in the original configuration. When the system board that has been separated using DR is to be reconnected, the disk unit that was under the control of GDS and the card that was used to connect that disk unit must be allocated exactly as they were prior to the failure. Even with PCI Hot Plug, only the PCI card is the target of swapping and the disk unit under the control of GDS is not a swap target. For information on repairing disk unit failures, see PRIMECLUSTER Global Disk Services Configuration and Administration Guide The synchronization of the mirrored volumes must be maintained. When the following conditions arise, hot swap with DR/PCI Hot Plug is disabled: 1) If a mirror disk unit group (to which a disk unit - that is connected to the hot swap target system board or PCI card belongs) includes a volume whose synchronization is not maintained, and 2) If a slice that is solely in the ACTIVE state in that volume exists on the target disk unit. You may presume the following causes if synchronization is not maintained. You need to check the state of the slice before performing hot swap using the sdxinfo(1) command. If you separate the copy source disk unit while re-synchronization copy is being performed. When TEMP slices are detached and only a single slice is ACTIVE. When you sequentially separate the system boards (or PCI cards) to which each mirrored disk is connected using the DR (or PCI Hot Plug). 4. The disk units must be mirrored between the system boards (or PCI cards). If mirrored disk units are connected to the same system board, you cannot use the DR function to separate that system board. Or, if disk units connected to a single PCI card are mirrored, hot swap cannot be performed for that PCI card by using the PCI Hot Plug. 5. The I/O paths to the disk units must not be redundant. If the redundant I/O paths to the disk units are controlled by Multipath Disk Control, you need to follow the DR/PCI Hot Plug procedure for Multipath Disk Control. Also, if you are using non-fujitsu redundant I/O software products (for example, EMC s PowerPath), check how the DR/PCI Hot Plug is supported for each software product, and follow the procedure. 6. The class attribute (scope) must be set to access the disk of a shared class on all nodes. The class attribute (scope) can be confirmed from the Class Configuration screen of GDS Management View. 7

8 2.1.2 Condition Unique to DR 1. Single partition environment Since PRIMECLUSTER supports Dynamic Reconfiguration (DR) for CPU and memory in which I/O is disabled, DR is enabled only on a single partition environment. 2.2 Hot-swapping System Board using DR This section explains how to perform hot swap using DR in an environment in which GDS is used on a single partition Separating the System Board (1) Prior check (checkremove or preremove of an associated script) Check that GDS mirrored the disk unit that is connected to the system board using the following steps. The following example shows that the disk that will be separated is controlled with the sd driver. 1-1) Use the drcstat command to find a name of the disk unit that is connected to the system board. Example: When sb01 is to be separated # drcstat -device sb01 grep sd 01-PCI#0B "/pci@81,4000/scsi@2/sd@1,0" 1 "sd" (Note) Use the dr_info command to to make the system board separation automatic with an associated script. 1-2) Find the logical device name (c#t#d#) of the target disk unit. # ls -l /dev/dsk grep /pci@81,4000/scsi@2/sd@1,0:c <-(Note) ":c" is suffixed. lrwxrwxrwx 1 root root 55 Sep ) Confirm that the target disk unit is under the control of GDS and then find an SDX disk unit name. # sdxinfo -D grep c1t1d0 disk disk01 mirror class01 grp01 c1t1d * ENABLE If the target disk unit is not under the control of GDS, check whether redundant I/O paths to the disk unit are controlled by other software, and use the appropriate DR procedure. 1-4) Confirm that an ACTIVE slice of a mirror volume is on other than the target disk unit. # sdxinfo -S -o disk01 OBJ CLASS GROUP DISK VOLUME STATUS slice class01 grp01 disk01 v01 ACTIVE slice class01 grp01 disk01 v02 ACTIVE slice class01 grp01 disk02 v01 ACTIVE slice class01 grp01 disk02 v02 ACTIVE If there is a volume that has no ACTIVE slice on other than the target disk unit, DR cannot be executed. (2) Before the system board is removed(preremove of an associated script) 2-1) Prior check and name confirmation Confirm whether the conditions for executing DR are met using the procedure of (1), and then check the class name and the SDX disk name with steps 1-3) and 1-4). 8

9 2-2) Disconnect the target disk unit by executing the sdxswap command. # sdxswap -O -c class01 -d disk01 2-3) Confirm that the target disk unit has been separated correctly (whether it has entered the SWAP state). # sdxinfo -D -o disk01 OBJ NAME TYPE CLASS GROUP DEVNUM DEVBLKS DEVCONNECT STATUS disk disk01 mirror class01 grp01 c1t1d * SWAP (3) Separate with DR. Execute separation with DR. For details, see the DR User's Guide. # drc -disconnect sb01 -keep Reconnecting the System Board (1) Reconnecting with DR Execute reconnection with DR. For details, see the DR User's Guide. # drc -connect sb01 (2) After addition (postadd) 2-1) Use the drcstat command to find the device name of the disk unit that is connected to the system board. # drcstat -device sb01 grep sd 01-PCI#0B "/pci@81,4000/scsi@2/sd@1,0" 1 "sd" 2-2) Find the logical device name (c#t#d#) of the target disk unit. # ls -l /dev/dsk grep /pci@81,4000/scsi@2/sd@1,0:c <-(Note) ":c" is suffixed. lrwxrwxrwx 1 root root 55 Sep c1t1d0s2 2-3) Confirm that the target disk unit is under the control of GDS and that it is in the SWAP state. Then, find the disk class and SDX disk name. # sdxinfo -D grep c1t1d0 disk disk01 mirror class01 grp01 c1t1d * SWAP 2-4) Incorporate the target disk unit by executing the sdxswap command. # sdxswap -I -c class01 -d disk01 2-5) Confirm that the target disk is incorporated correctly (if it is changed to the ENABLE state). # sdxinfo -D -o disk01 OBJ NAME TYPE CLASS GROUP DEVNUM DEVBLKS DEVCONNECT STATUS disk disk01 mirror class01 grp01 c1t1d * ENABLE 9

10 2-6) Wait until synchronization copy for the target disk is complete. # sdxinfo -S -o disk01 OBJ CLASS GROUP DISK VOLUME STATUS slice class01 grp01 disk01 v01 COPY <- Wait for change to ACTIVE. slice class01 grp01 disk01 v02 COPY slice class01 grp01 disk02 v01 ACTIVE slice class01 grp01 disk02 v02 ACTIVE 2.3 Hot-Swapping the PCI Card with PCI Hot Plug This section explains how to hot-swap the PCI card using the PCI Hot Plug Disconnecting a Disk Unit 1) Stopping hardware monitoring by machine management Stop the hardware monitoring performed by the machine management. For details, see the PCI Hot Plug User s Guide I/O device edition. 2) Identifying the PCI card Identify the interface name of the path for connecting the I/O device to the PCI card to be swapped, and the connected physical disk. For details, see the PCI Hot Plug User s Guide I/O device edition. 3) Searching for an SDX object that is associated with the disk unit connected to the PCI card Search for an SDX object that is associated with the disk unit that is detected upon the identification of the PCI card. # sdxinfo -D OBJ NAME TYPE CLASS GROUP DEVNAM DEVBLKS DEVCONNECT STATUS disk disk01 mirror class01 grp01 c2t10d sdxst3:sdxst4 ENABLE disk disk02 mirror class01 grp01 c2t11d sdxst3:sdxst4 ENABLE # 4) Disconnecting a disk unit from the mirrored disk unit group Disconnect the target disk unit from the mirror disk group. # sdxdisk D c class01 g grp01 d disk02 5) Removing a disk unit from the disk class Remove a target disk unit from the disk class. # sdxdisk R c class01 d disk Separating, Swapping, and Incorporating a PCI Card For details, see the PCI Hot Plug User s Guide I/O device edition. 10

11 2.3.3 Reconnecting a Disk Unit 1) For a cluster environment, edit the information on the resources as described below. If a cluster system has not been configured, you need to work on the steps after 2). 1-1) Search for a node ID. # /etc/opt/fjsvlcluster/bin/clgetnode RID 3 KEY sdxst3 RNAME sdxst3 NODEID 0 # 1-2) Search for a resource ID. # /etc/opt/fjsvcluster/bin/clgetrid -c DISK -k c2t11d0 -s 0 87 # 1-3) Delete the resource information. # /etc/opt/fjsvcluster/bin/cldelrsc -r ) Execute the automatic registration of resources in the cluster, and then execute the following command on all the nodes. # /etc/opt/fjsvcluster/bin/clautoconfig -r 2) Registering a disk unit to the disk class Register the target disk unit to the disk class again. # sdxdisk M c class01 d c1t1d0=disk02 You can confirm that the disk unit has been registered by executing the following command: # sdxinfo -D OBJ NAME TYPE CLASS GROUP DEVNAM DEVBLKS DEVCONNECT STATUS disk disk01 mirror class01 grp01 c2t10d sdxst3:sdxst4 ENABLE disk disk02 undef class01 * c2t11d sdxst3:sdxst4 ENABLE # 3) Connecting a disk unit to a mirror disk group Connect the target disk unit to the mirrored disk group again. # sdxdisk C c class01 g grp01 d disk01 You can confirm that the disk unit has been connected by executing the following command: # sdxinfo -D OBJ NAME TYPE CLASS GROUP DEVNAM DEVBLKS DEVCONNECT STATUS disk disk01 mirror class01 grp01 c2t10d sdxst3:sdxst4 ENABLE disk disk02 mirror class01 grp01 c2t11d sdxst3:sdxst4 ENABLE # 11

12 2.4 Hot Expansion of the PCI Card with PCI Hot Plug 1) Hot expansion of the PCI card For information on the PCI card expansion procedure using the PCI Hot Plug, see the PCI Hot Plug User s Guide I/O device edition. 2) Executing automatic resource registration For a cluster configuration, execute clautoconfig(1m) with the r and d options specified, or register the disk resource by using the CRM main view of ClusterAdmin (GUI). # /etc/opt/fjsvcluster/bin/clautoconfig -r d 3) Confirming LU addition Execute clgettree(1) or use the CRM main view of ClusterAdmin (GUI) to confirm whether the added LU resource has been registered. # /etc/opt/fjsvcluster/bin/clgettree You can confirm the LU addition from the GDS Management View. The added LU may not be available on the view immediately. You should display the latest information by selecting [View] -> [Update Now] from the main menu. 4) Creating a GDS volume You can create a GDS volume using the normal procedure. For details, see the PRIMECLUSTER Global Disk Services Configuration and Administration Guide. 12

13 3. GLS You can execute DR (Dynamic Reconfiguration) without disrupting ongoing operations by configuring a redundant LAN between the Network Interface Cards (NICs) that are connected to different system boards using Global Link Services (GLS). Similarly, the hot swapping and hot expansion of an NIC can be performed without disrupting ongoing operations. 3.1 Prerequisites for DR To provide hot swap by using GLS, NIC needs to meet the following conditions for DR: 1. NIC redundancy must be established for the system boards. If NIC redundancy is created for the NICs connected to a single system board, that system board cannot be separated using DR (Dynamic Reconfiguration). 2. All components, with the exception of the failed portions, must be reconnected in their original configuration. When you replace an NIC by separating the system board, the NIC must be allocated to the same slot as that before the replacement. 3.2 DR Associated Script Interface [When the Redundant Line Control function (NIC switching mode, fast switching mode, RIP mode, and GS/SURE linkage mode) is used] Since DR is automatically executed as part of Global Link Services (GLS) by using the DR associated script, you do not have to stop the NIC. For details on the DR associated script interface of GLS, see the following chapters in the PRIMECLUSTER Global Link Services Configuration and Administration Guide: Redundant Line Control Function : DR (Dynamic Reconfiguration) linkage function Setting DR (Dynamic Reconfiguration) How to recover when an error occurred in a transfer route at the execution of DR [When the Multipath function is used] Similarly, you do not have to stop the NIC since DR is automatically executed by the DR associated script. For details on the DR associated script interface of GLS, see the following chapter in the online manual, PRIMECLUSTER Global Link Services Configuration and Administration Guide: Multipath Function. Chapter 8 Dynamic Reconfiguration (DR) Function 3.3 Hot Swap and Hot Expansion of NIC with PCI Hot Plug For more information, see the PCI Hot Plug User s Guide I/O device edition. Note, however, that the CF settings must have been completed when the cluster environment is established. For details, see Replacing and Expanding NIC with PCI Hot Plug. of this guide 13

14 4. GFS Local File System The procedure for Dynamic Reconfiguration (DR) of the disk units that are being used in a GFS local file system is fundamentally the same as that for UFS. If a device unit that does not have a redundant I/O configuration on the relevant system board is being used as the GFS local file system, you have to stop (umount) the running file system. You do not have to stop the file system if the disk unit has a redundant I/O configuration and can be separated while being used by the file system. Similarly, if a device (without a redundant I/O configuration on the relevant PCI system board) is being used as the GFS local file system when PCI Hot Plug is being executed for a disk unit that is used by the GFS local file system, you have to stop (umount) the running file system. You do not have to stop the file system if the device has a redundant I/O configuration and can be separated while being used by the file system. Note, however, that the GFS local file system supports the management of multiple slices as a single file system. If you are using this multi-partition configuration, you must execute a specific GFS local file system command to determine which of the configuration slices are mounted. This manual explains how to separate a system board using DR as well as with PCI Hot Plug while the GFS local file system is being created and mounted on a multi-partition configuration within a standalone partition. 4.1 Separating the System Board with DR This section explains how to stop a failed file system: a) Execute umount for a file system for which an I/O error has been displayed by executing the df command. b) Using the sfxinfo command, identify the failed file system, and then execute umount for the file system that uses the system board to be separated Handling an I/O Error Using the df command, investigate the file system for which an I/O error is displayed, and then execute umount. 1) Execute the df command as specified by the GFS local file system. The mounting state of the GFS local file system can be confirmed by specifying -F sfxfs with the df command. # df -F sfxfs /DOC (/dev/dsk/c1t9d0s1 ): blocks files /PKG (/dev/dsk/c1t10d0s1): blocks files df: cannot statvfs /sfxfs3: I/O error 2) Remove the file system in which the error has occurred. If you identify a file system for which an I/O error is displayed as shown in the output example in 1), execute umount for that file system. If there are any active processes or unavailable data, execute umount with f specified. # umount -f /sfxfs Association between DR Separation Target System and Slice Name Confirm the association between the separation target system board and the affected physical disk slice name as described in the DR User s Guide. Note When the device used by the file system is a logical device, you must determine whether that logical device will become unusable as a result of separating the system board, based on the specifications of each logical device driver. 14

15 4.1.3 Confirming the Multi-partition Configuration with the sfxinfo Command If you execute the sfxinfo command without arguments, you can determine the slice configuration and mount points of the GFS local file system that are currently mounted. If a slice included in this output impacts the separation of the system board, unmount the file system based on the mount point name. # sfxinfo ID special size mount Type 0 /dev/dsk/c1t9d0s1(800041) /sfxfs1 META 0 /dev/dsk/c1t9d0s1(800041) /sfxfs1 DATA 0 /dev/dsk/c1t9d0s1(800041) /sfxfs1 LOG ID special size mount Type 0 /dev/dsk/c1t10d0s1(800049) /sfxfs2 META 0 /dev/dsk/c1t10d0s1(800049) /sfxfs2 DATA 0 /dev/dsk/c1t10d0s1(800049) /sfxfs2 LOG 1 /dev/dsk/c1t11d0s1(800051) /sfxfs2 DATA Note When the umount command is executed while an application is using a file system, this command fails. You need to stop the application, or unmount the file system using the umount f command. 4.2 Hot Swapping the PCI Card with PCI Hot Plug This section explains how to stop an affected file system. a) Execute umount for a file system for which an I/O error is displayed upon the execution of the df command. b) Using the sfxinfo command, identify the file system that is affected by the separation of the PCI card, and then execute umount for the file system Handling an I/O Error Using the df command, investigate the file system for which an I/O error is displayed, and then execute unmount. 1) Execute the df command that is specified by the GFS local file system. The mounting state of the GFS local file system can be confirmed by specifying -F sfxfs with the df command. # df -F sfxfs /DOC (/dev/dsk/c1t9d0s1 ): blocks files /PKG (/dev/dsk/c1t10d0s1): blocks files df: cannot statvfs /sfxfs3: I/O error 2) Remove the file system in which an error occurred. If you identify a file system for which an I/O error is displayed as shown in the output example in 1), execute umount. If there are any active processes or unreflected data, execute umount with f specified. # umount -f /sfxfs3 15

16 4.2.2 Association between PCI Hot Plug Swap Target PCI Card and Slice Name Confirm the association between the PCI card and the affected physical disk slice name, as described in the PCI Hot Plug User s Guide I/O device edition. Note When the device used by the file system is a logical device, you must determine whether that logical device will become unusable as a result of separating the PCI card, according to the specifications of each logical device driver Confirming the Multi-partition Configuration Executing the sfxinfo command without an argument allows you to determine the slice configuration and mount points of the currently mounted GFS local file system If a slice included in this output will impact the separation of the PCI card, unmount the file system based on the mount point name. # sfxinfo ID special size mount Type 0 /dev/dsk/c1t9d0s1(800041) /sfxfs1 META 0 /dev/dsk/c1t9d0s1(800041) /sfxfs1 DATA 0 /dev/dsk/c1t9d0s1(800041) /sfxfs1 LOG ID special size mount Type 0 /dev/dsk/c1t10d0s1(800049) /sfxfs2 META 0 /dev/dsk/c1t10d0s1(800049) /sfxfs2 DATA 0 /dev/dsk/c1t10d0s1(800049) /sfxfs2 LOG 1 /dev/dsk/c1t11d0s1(800051) /sfxfs2 DATA Note When the umount command is executed while an application is using a file system, this command fails. You need to stop the application, or unmount the file system using the umount f command. 16

17 5. PRIMECLUSTER 5.1 DR Function In the PRIMECLUSTER environment, it is recommended that you stop some of the nodes, and then swap, attach or detach the system board while operation continues through failover or degenerated operation. However, DR can also be used to perform hot maintenance, hot expansion, or hot degeneration in the following cases: Hot swap when an insignificant failure that does not disrupt ongoing operations occurs in a CPU or memory such as 1-bit intermittent failure. Hot expansion or degeneration of the CPU and memory due to business expansion or reduction Note You cannot execute DR for a system board (for which no OBP I/O validation option has yet been set) on which I/O is mounted. Also, before attempting hot swap or degenerated operation, you must first determine whether this is possible with reduced CPU and memory capacities. The kernel memory board cannot be detached (swapped or moved) from a cluster configuration regardless of the machine type Preset For a system board for which you intend to execute hot swap or an inter-partition move, you must set the OBP I/O disabling option in advance. Set the number of the system boards that will execute DR in the OBP variable, no-io-sbcx (X indicates a chassis number). For details, see the DR User s Guide Notes Response Messages Related to Real time Process PRIMECLUSTER allows part of the process to operate in a real time class. If you use DR to attach or detach a system board, the following message will appear: Dec 25 21:12:41 Real time processes[pid= ] is running. Do you continue DR? [YES]/[NO] Confirm that the displayed process IDs are for PRIMECLUSTER daemons, and then enter yes to continue with DR. (The same process ID will be displayed repeatedly for a multithreaded process.) The real time processes that are started with PRIMECLUSTER are shown below: PRIMECLUSTER rcsd, bm, hvdet_*(note 1) (Note 1) The detector will be started up from the process bm as a real time process according to the configuration of the applications. You should judge that the process whose parent process is bm and the process name starts with hvdet_ is a detector of PRIMECLUSTER. PRIMECLUSTER GFS shared file system sfcfsmg If you execute DR frequently or if you want to automate the processing, it may be advantageous to create a associated script and use the dr_conf command. For details, see the DR User s Guide. 17

18 Messages output during DR RMS may stop temporarily during DR. Should this occur, the following error or warning messages may be displayed: (BM, 65): NOTICE: Base monitor heartbeat between this host and host <coll0e12rms> is off. (NOD, 27): NOTICE: Detector for cluster host coll0e12rms has reported offline. (WRP, 34): ERROR: Cluster host node is no longer in time sync with local node. Sane operation of RMS can no longer be guaranteed. Further out-of-sync messages will appear in the syslog. (SYS, 88): WARNING: Cluster hostcoll0e12rms transitioned from online to offline unexpectedly. Allowing 45 more seconds to expire before reacting. (BM, 66): NOTICE: Base monitor heartbeat between this host and host <coll0e12rms> is on. (NOD, 27): NOTICE: Detector for cluster host coll0e12rms has reported online. (US, 11): NOTICE: Cluster host coll0e12rms has recovered and reported online once again.. These messages indicate that the RMS heartbeat has been temporarily suspended. If these messages are displayed during DR, you do not have to take any action in response. Ignore them Restrictions Imposed if an Error Occurs During DR If an error occurs in the cluster interconnect or another node during DR, you should pay careful attention to the following restrictions. [Restrictions] Since the PRIMEPOWER SCF driver stops temporarily during DR, you cannot use SCF/RCI with the inter-node monitoring function of PRIMECLUSTER. Therefore, you must pay particularly careful attention to the following: 1. In an environment with a single cluster interconnect, redundancy cannot be maintained if an error occurs during DR. Do not, therefore, execute DR in an environment in which the cluster interconnect does not have a redundant configuration. 2. A panic that occurs in another node during DR cannot be detected immediately by the RCI asynchronous monitoring function. Instead, it will be detected by the heartbeat function after a delay of approximately 10 seconds. 5.2 PCI Hot Plug Function Swapping and Expanding NIC with PCI Hot Plug To use PCI Hot Plug to support the hot swapping or hot expansion of a NIC, you must configure the network interface redundantly. Note In the exchange of NIC by PCI Hot Plug, the one that the network interface has been activated is an object. Therefore, Stopping the node so far and exchanging public LAN set, the interface attribute of public LAN which uses PRIMECLUSTER GLS that the network interface becomes non-active state by the standby node or the stop node (multipass function) and succession Internet Protocol address is "Base". If the CF settings have not been complete, you cannot perform NIC swap by using PCI Hot Plug. If you need to swap the NIC, therefore, replace it after terminating the node. For details on the CF settings, see the PRIMECLUSTER Cluster Foundation Configuration and Administration Guide. You cannot change NIC of a public LAN using PCI Hot Plug when the Scalable Internet Services (SIS) is used (SMAWdtcp is installed). If you use PCI Hot Plug to change NIC, remove SMAWdtcp in single user mode first. If you use the Scalable Internet Services (SIS), stop the node, and then change NIC Procedure for Swapping Public LAN Apply the swap procedure described in Redundant configuration in the PCI Hot Plug User s Guide I/O device edition Expanding a public LAN For information on adding a path for PCI card expansion, follow the expansion procedure described in PCI Hot Plug User s Guide I/O device edition. 18

19 Procedure for Swapping cluster interconnect LAN In the event of a cluster interconnect, shut down a node and swap the cluster interconnect Using PCI Hot Plug to Swap or Expand a PCI Card Connected to a Shared Disk To use PCI Hot Plug to support the hot swapping or hot expansion of a PCI card that is connected to a shared disk, you must configure a shared disk that satisfies the following conditions: A RAID unit must be redundant with MPHD/GRMPD. A SCSI device must be redundant with GDS (for swapping only) Swap or addition of a PCI card connected to a RAID unit For details on swapping a PCI card that is controlled by MPHD/GRMPD or adding a path for PCI card expansion, see the PCI Hot Plug User s Guide I/O device edition Swap a PCI card connected to a SCSI device For details on swapping a PCI card that is connected to a SCSI device that is made redundant using GDS, see Chpater 2 GDS. 19