Critical Resource Analysis (CRA) White Paper

Transcription

1 Critical Resource Analysis (CRA) White Paper SUMMARY... 3 Introduction to Critical Resource Analysis... 4 CRA return values... 4 Mass Storage CRA Scenarios... 5 Boot Path Configuration Scenarios... 5 Scenario Scenario Scenario Scenario Swap and Crash Dump Device Configuration Scenarios... 8 Scenario Scenario Scenario Service Guard Configuration Scenarios... 9 Scenario Scenario Scenario Scenario Scenario I/O in Progress Scenarios Scenario Scenario File System Configuration Scenarios Scenario Scenario Scenario Scenario Scenario Scenario DLKM Driver unload scenarios Scenario Scenario Scenario

2 Cards in PCI Error and Suspended State Scenarios Scenario Scenario Scenario Scenario SAN Storage Configuration Scenario Scenario LAN CRA Scenarios IP Address Configured NIC Scenarios Scenario Scenario Scenario Scenario Scenario VLAN and APA Configuration Scenarios Scenario Scenario Scenario Scenario Scenario Scenario Scenario Scenario Scenario Service Guard Configuration Scenarios Scenario Scenario Scenario Dynamic npartitions Scenarios I/O Chassis Deletion Scenarios Scenario For more information Call to action... 31

3 SUMMARY Critical Resource Analysis (CRA) provides usage analysis for I/O resources during destructive operations, such as online PCI card replacement and deletion, lun or lunpath removal, and during unload of an interface card driver. The complete CRA framework consists of multiple subsystem CRA modules, and one infrastructure CRA module. The infrastructure CRA module provides basic functionality and co-ordination between the subsystem CRA modules. During analysis, the infrastructure CRA module interacts with and queries each subsystem CRA module through a welldefined interface. For a given set of I/O components, these subsystem specific CRA modules perform usage analysis specific to their subsystems, and report any potential impacts. The infrastructure CRA module collates the usage analysis reports from all the subsystem CRA modules to provide a combined CRA report for the system. This white paper provides an in-depth account of the reports generated by Critical Resource Analysis (CRA) for various configurations and scenarios on a system installed with the HP-UX 11iv3 operating system. The paper also discusses various recommendations to safely perform the intended operation for each of the described scenarios. These recommendations will help a system administrator to deal with situations when an operation, that may need appropriate precaution to be taken, needs to be performed on the system. This whitepaper is primarily targeted at the HP-UX 11iv3 operating environment. Though it may have some scenarios common with HP-UX 11iv2, this document is not intended for HP-UX 11i v2 and earlier releases. The rest of this document assumes HP-UX 11iv3 operating environments. 3

4 Introduction to Critical Resource Analysis Starting with HP-UX 11i v3, given a set of I/O components, the CRA module performs usage analysis and reports any potential impacts for specific subsystems, such as Mass Storage, LAN, or Infiniband. The CRA module returns the highest critical usage reported. The CRA module also collates detailed analysis logs from all the subsystem CRA modules and returns the combined logs for the user under the /var/adm directory on the system. Example: Consider a CRA framework comprising an infrastructure CRA, a networking subsystem CRA module, and a mass storage subsystem CRA module. Assume that a 4-port LAN card is being taken offline (suspended or deleted). The I/O resources to be analyzed for usage are the 4 LAN interface ports on the card. For CRA purposes, an I/O resource is represented by its hardware path. The infrastructure CRA module invokes both the networking and mass storage subsystem CRA modules with the hardware paths for the 4 LAN ports on the IO card. The subsystem CRA modules verify if any LAN ports are in use within the networking or mass storage subsystems. Each subsystem CRA module returns its analysis report to the infrastructure CRA module. The infrastructure CRA module then returns a combined report to its caller. The CRA is invoked in the following scenarios: Card Delete (using olrad d) Card Suspend (using olrad r) Card Usage Report (using olrad C) LAN Card Usage Report (using nwmgr cra c) LUN Delete (using scsimgr command) Disable a lunpath (using scsimgr command) Driver unload (using kcmodule command) Removal of legacy device special files (using rmsf L command) Deletion of VLAN/APA interface (using nwmgr delete command) Reset of the lan interface (using nwmgr reset command ) Chassis Delete (using olrad D s) during Dynamic npartition operations CRA return values Cumulative CRA is a resource analysis method that checks the overall impact of a destructive operation on any or all components in the subsystem. The CRA returns the following values at the end of the analysis: CRA_SUCCESS - Indicates that no critical usage is detected for the set of components that were analyzed. CRA_WARNING Indicates a warning issued to the user because of a redundant resource being deleted or taken offline which does not cause any loss of service. A CRA_WARNING could be returned owing to the potential performance impact of a hardware redundancy that is being removed. For example, a warning is issued when one of the network links from a networking aggregate is going down. CRA_DATA_CRITICAL - Indicates loss of resources that are classified as not critical to the running system. However, applications might be affected. If a CRA_DATA_CRITICAL flag is returned, the 4

5 olrad command provides an option to override the flag and proceed with the removal of I/O components. For example, if an IP address is configured on a networking port, it is considered a DATA CRITICAL usage. CRA_SYSTEM_CRITICAL - Indicates critical resources are in use. If these resources are taken offline, the system could crash. Swap and boot configurations are examples. With system critical resources in use, the corresponding I/O component should not be allowed to be taken offline. CRA_FAILURE - Indicates that internal errors occurred during the analysis. The Critical Resource Analysis may take anywhere between less than a minute to several minutes to complete, depending on system configurations. In most of the typical system configurations, the CRA will complete in less than a minute and in large configurations with large number of interface cards, and storage devices with multi-paths, the CRA may take several minutes to complete. Each subsystem CRA module provides a detailed log of resource usage for its corresponding subsystem. The infrastructure CRA module combines these logs and provides a system wide usage report for I/O components. The subsystem CRA log format has a Detailed Report section. The information contained in the Detailed Report provides a system administrator with detailed information on resource usage in a subsystem. The CRA log can be found on the system at the following location /var/adm/cra.log. This whitepaper provides insight into the typical CRA scenarios. The scenarios are discussed at the following levels: Various possible system configuration scenarios. Operations that may be performed on the machine, which, if appropriate precaution is not taken, can sometimes lead to system instability or data loss issues. The CRA resource usage report generated for that configuration and operation to be performed If the operation is blocked by the CRA and if it is necessary to go ahead with the operation, a recommendation to safely carry out the intended operation. Mass Storage CRA Scenarios Mass storage CRA is a subsystem invoked by the CRA infrastructure to perform resource analysis on mass storage components on the system. Mass storage components are those that belong to class of devices, such as FC (Fiber Channel), SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), etc. Boot Path Configuration Scenarios Scenario 1 System Configuration: HP-UX OS running from a boot disk with a single lunpath under an HBA. 5

6 # scsimgr disable H <boot lunpath> Criticality reported by CRA: CRA_SYSTEM_CRITICAL The above operations with scsimgr and olrad (1M) with r or d options (if it were to be allowed) will render the boot lunpath unavailable. This operation will result in system crash, hang, or any other indeterminate system state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with a CRA_SYSTEM_CRITICAL error. If any of these operations must be performed for system administration purposes, the user must ensure that the boot disk is reachable through an alternative path. In case of the olrad operation, the alternative path can be configured using another healthy card. For the scsimgr operation, a different port of the same card (if it is a multi-port card) or another card can be used to configure the alternate path. Once the alternate paths are configured thus, the above operations can be performed. Scenario 2 System Configuration: HP-UX OS running from a boot disk with multiple lunpaths which originate from different ports of the same card (for example, AB379B dual ported 4GB FC card). Criticality reported by CRA: CRA_SYSTEM_CRITICAL Though there are multiple lunpaths to the boot disk, the olrad(1m) with r or d options (if it were to be allowed) will render all the boot lunpaths unavailable because all lunpaths are from a single card with multiple ports. This will result in system crash, hang, or any other indeterminate system state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with CRA_SYSTEM_CRITICAL error. If any of these operations must be performed for system administration purposes without resulting in system instability, the user must ensure that the boot disk is reachable through an alternative healthy card before performing the above operations. Scenario 3 System Configuration: A running HP-UX OS is configured with an alternate boot disk under an HBA. # scsimgr disable H <alternate boot path> 6

7 The above operations with scsimgr and the olrad(1m) with r or d options will fail with CRA_DATA_CRITICAL error. If any of these operations must be performed for system administration purposes, the user must ensure that the alternate boot disk is reachable through an alternative path (in case of the scsimgr operation) or an alternative healthy card (in case of the olrad operation) before performing the above operations. However, the user may override this error using the olrad f option and the scsimgr disable F option. However, overriding the error and proceeding with the above operation, without making an alternative lunpath available, will leave the system without any alternate boot path set. Scenario 4 System Configuration: HP-UX OS running from a boot disk with one or more lunpath(s) through I/O cards controlled by the same driver. Operation Performed: # kcmodule <interface driver associated with card>=unused Criticality reported by CRA: CRA_SYSTEM_CRITICAL The above operation with kcmodule will render the boot lunpath unavailable and this will result in system crash, hang, or any other indeterminate system state. Because of the above reasons, the requested operation will not be allowed by CRA to proceed and the operation will fail with CRA_SYSTEM_CRITICAL error. If this operation must be performed for system administration purposes, then, before performing the operation, the user must configure another lunpath to the boot disk through a different I/O card driver, than the one that is going to be currently unloaded. NOTE: If the boot disk had a single lunpath and a kcmodule operation is attempted, as mentioned above, though the kcmodule operation fails with CRA_SYSTEM_CRITICAL, it still marks the interface driver as unused for the next boot. That is, the corresponding interface card driver is removed from the kernel configuration that is marked to be used during the next boot of the system. Therefore, when the system is rebooted, the system will panic on the next boot since it can not find the interface card driver for boot path. In such a scenario, to avoid the panic due to the interface card driver for the boot path not being configured, before rebooting the system, the user can run one of the following commands: Discard all kernel configuration changes marked for the next boot of the system, by running: # kconfig H To verify that all the kernel configuration changes have been discarded, run: # kconfig D 7

8 However, running the above kconfig H command will discard all the kernel configuration changes, not only the kernel configuration change caused by the above kcmodule driver unload operation, but also any other changes held for next boot of the system. Load the interface card driver, that has been marked as unused, back into the kernel configuration, by running: # kcmodule <interface driver associated with card>=loaded Swap and Crash Dump Device Configuration Scenarios Scenario 5 System Configuration: HP-UX OS running with swap device configured either on a logical volume (could be either LVM or VxVM volume) or on a physical disk (PV) with a single lunpath through an HBA. Criticality reported by CRA: CRA_SYSTEM_CRITICAL The olrad(1m) with r or d options (if it were to be allowed) will render the swap device unavailable and this will result in system crash, hang, performance issues, or any other indeterminate state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with CRA_SYSTEM_CRITICAL error. If any of these operations must be performed for system administration purposes, the user must ensure that the swap device is reachable through an alternative healthy card before performing the above operations. If the swap device has been configured using a path dependent device special file, it will not be possible to provide any alternative path to it without loss of swap functionality and hence the requested olrad operation is not recommended to be performed. Scenario 6 System Configuration: HP-UX OS running with swap device configured on a File System mount point and this file system is configured either on a logical volume (could be either LVM or VxVM volume) or on a physical disk (PV) with a single lunpath through an HBA. Criticality reported by CRA: CRA_SYSTEM_CRITICAL 8

9 The olrad(1m) with r or d options (if it were to be allowed) will render the swap device unavailable and this will result in system crash, hang, performance issues, or any other indeterminate state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with CRA_SYSTEM_CRITICAL error. If any of these operations must be performed for system administration purposes, the user must ensure that the swap device is reachable through an alternative healthy card before performing the above operations. This recommendation is possible only if the swap device in question is configured using the path independent device special file (DSF). If the swap device has been configured using path dependent device special file, it will not be possible to provide any alternative path to the swap device without loss of swap functionality, which may lead to system crash, hang or any indeterminate system state. Scenario 7 System Configuration: HP-UX kernel dump device is configured on a logical volume (could be either LVM or VxVM volume) on a physical disk (PV) with a single lunpath through an HBA # scsimgr disable H <dump device lunpath> The above operations with scsimgr and the olrad(1m) with r or d options will fail with CRA_DATA_CRITICAL error. Overriding the error and proceeding with the operation will make the dump device unavailable for capturing kernel dump, such as in the event of any subsequent system crash or hang, thereby resulting in inability to debug the system problems. If any of these operations must be performed for system administration purposes, then, before performing the operation the user must configure another disk as the dump device or alternative path (in case of scsimgr operation) or alternative healthy card (in case of olrad operation). Service Guard Configuration Scenarios Scenario 8 System Configuration: HP-UX OS running in Service Guard (SG) cluster environment with common lock disk between the cluster nodes. The common lock disk has only one lunpath through an HBA. # scsimgr disable H <lunpath to quorum disk> 9

10 Criticality reported by CRA: CRA_SYSTEM_CRITICAL The above operations with scsimgr and olrad(1m) with r or d options (if it were to be allowed) will render the quorum disk unavailable if the requested operation is allowed to proceed. This process will result in the node being evicted from the cluster. Because the proper functioning of the SG cluster is considered one of the critical configurations, the above requested operations will not be allowed by CRA to proceed and the operation will fail with CRA_SYSTEM_CRITICAL error. If any of these operations must be performed for system administration purposes, the user must configure another lunpath to the SG quorum disk, which is reachable through an alternative healthy card before performing the above operations. However, this recommendation is possible only if the SG quorum disk has been configured using path-independent device special file (DSF). If the quorum disk has been configured using path dependent device special file, it will not be possible to provide any alternative path to the quorum disk without losing the SG cluster functionality and hence the requested olrad operation is not recommended. However, if the user is prepared to stop the SG cluster service, a different quorum disk can be configured using an alternative healthy card (in case of the olrad operation) or an alternative path can be configured using the same or a different card (in case of the scsimgr operation) and the required operation can be performed. Scenario 9 System Configuration: HP-UX OS running in Service Guard (SG) cluster environment with common lock disk between the cluster nodes, and the common lock disk has multiple lunpaths which originate from different ports of the same card (for example, AB379B dual ported 4GB FC card). Criticality reported by CRA: CRA_SYSTEM_CRITICAL Though there are more than one lunpaths to the quorum disk, the olrad(1m) with r or d options (if it were to be allowed) will render the quorum disk unavailable if the requested operation is allowed to proceed because all lunpaths are from a single card with multiple ports. Because this will result in the node being evicted from the cluster and the proper functioning of the SG cluster is considered as one of the critical configurations, the above requested olrad operations will not be allowed by CRA to proceed and the operation will fail with a CRA_SYSTEM_CRITICAL error. If any of these operations must be performed for system administration purposes, the user must ensure that the SG quorum disk is reachable through an alternative healthy card before performing the above operations. However, this recommendation is possible only if the quorum disk has been configured using path-independent device special file (DSF). If the quorum disk has been configured using path dependent device special file, it will not be possible to provide an alternative path to the quorum disk without losing the SG cluster functionality and hence the requested olrad operation is not recommended. However, if the user is prepared to 10

11 stop the SG cluster service, a different quorum disk can be configured using an alternative healthy card and the required olrad operation can be performed. Scenario 10 System Configuration: HP-UX OS running in Service Guard (SG) cluster environment with common lock disk between the cluster nodes, and the cluster is configured with a common lock disk using path dependent device special file. Operation Performed: # rmsf -L Criticality reported by CRA: CRA_SYSTEM_CRITICAL The above rmsf operation will render the quorum disk unavailable if the requested operation is allowed to proceed. Since this will result in the node being evicted from the cluster and because the proper functioning of the SG cluster is considered one of the critical configurations, the above requested operation will not be allowed by CRA to proceed and the operation will fail with a CRA_SYSTEM_CRITICAL error. If the quorum disk has been configured using path dependent device special file, it will not be possible to provide an alternative path to the quorum disk without losing the SG cluster functionality and hence the requested operation is not recommended. However, if the user is prepared to stop the SG cluster service, a quorum disk can be configured using path-independent DSF, which has one or more lunpaths to the quorum disk and the above operation can be performed. Scenario 11 System Configuration: HP-UX OS running in Service Guard (SG) cluster environment with common lock disk between the cluster nodes, and the cluster is configured with a common lock disk, which has one or more lunpath(s) through I/O cards controlled by the same driver. Operation Performed: # kcmodule <interface driver associated with card>=unused Criticality reported by CRA: CRA_SYSTEM_CRITICAL The above operation will render the quorum disk unavailable if the requested operation is allowed to proceed. Since this will result in the node being evicted from the cluster and the proper functioning of the SG cluster is considered as one of the critical configurations, the above requested operation will not be allowed by CRA to proceed and the operation will fail with CRA_SYSTEM_CRITICAL error. If this operation must be performed for system administration purposes, then, before performing the above operation, the user must configure another lunpath to the SG quorum disk through a card controlled by a different I/O driver. However, this is possible only if the quorum disk has been configured using path-independent device special file (DSF). If the quorum disk has been configured using a path dependent device special file, it will not be possible to provide any alternative path to the quorum disk without losing the SG cluster functionality and hence the requested operation is not recommended to be performed. However, if the user is prepared to stop the SG cluster, and an alternative path to the quorum disk can be configured through a card controlled by different I/O driver, then the required operation can be performed. 11

12 Scenario 12 System Configuration: HP-UX OS running in Service Guard (SG) cluster environment with common lock disk between the cluster nodes, but the SG cluster is not currently running. Criticality reported by CRA: CRA_FAILURE Though the SG cluster is not currently running, as long as the quorum disk is configured, the above olrad operations on the card under which the quorum disk is present will fail with CRA Internal error. In this scenario, if the above olrad operations must be performed for system administration purposes, the recommended procedure is to remove the specified quorum disk from the SG cluster configuration or remove the SG cluster configuration itself (if the user is prepared to take this recommendation and after ensuring that the cluster configuration has been backed-up) and then re-try the above operation. I/O in Progress Scenarios Scenario 13 System Configuration: I/O is in progress to a logical volume (could be either LVM or VxVM volume) configured on a physical disk (PV) with a single lunpath through an HBA. Note that this PV is neither a boot disk nor a disk that has any root file system mounted. The logical volume in use is one which is other than the logical volume for the current boot disk. # scsimgr disable H <lunpath> The above operations with scsimgr and the olrad(1m) with r or d options will fail with CRA_DATA_CRITICAL error. If any of these operations must be performed for system administration purposes, then the user must ensure that the disk under I/O is reachable through an alternative path (in case of scsimgr operation) or alternative healthy card (in case of olrad operation) before performing the above operations. However, the user may override this error using the olrad f option or the scsimgr disable F option. Overriding the error and proceeding with the operation will make the disk unavailable and this may result in loss of data or data inconsistency problems. Scenario 14 System Configuration: HP-UX OS is running and the user applications are performing disk I/O using the path dependent device special file (DSF). Operation Performed: # rmsf -L 12

13 The above operation with the rmsf option will fail with a CRA_DATA_CRITICAL error. Before performing the above operation, it is recommended to make the user applications use the path independent device special file. File System Configuration Scenarios Scenario 15 System Configuration: A File system is mounted on a disk and the disk has a single lunpath through a card. # scsimgr disable H <lunpath> The above operations with scsimgr and olrad(1m) with r or d options will fail with a CRA_DATA_CRITICAL error. Overriding the error and proceeding with the operation will make the lunpath unavailable and this will make the file system, residing on the disk, inaccessible. If any of these operations must be performed for system administration purposes, an alternative path must be configured using the same or a different card (in case of the scsimgr operation) or through an alternative healthy card (in case of the olrad operation) before performing the above operations. Scenario 16 System Configuration: A File system is mounted on a disk and the disk has one or more lunpath(s), which originate from different ports of the same card (for example, AB379B dual-ported 4GB FC card). Though there are multiple lunpaths to the disk, the above operations with olrad(1m) with r or d options will fail with a CRA_DATA_CRITICAL error. Overriding the error and proceeding with the operation will make the lunpaths unavailable as all the paths are through the same card and this will render the file system residing on the disk inaccessible. If any of these operations must be performed for system administration purposes, an alternative path must be configured using an alternative healthy card before performing the above operations. 13

14 Scenario 17 System Configuration: Disk has been configured as a mirror copy of a logical volume (could be either LVM or VxVM volume) with file system mounted on it. The mirror disk has only one lunpath, where the lunpath originates from a different I/O card as compared to the primary mirror copy. # scsimgr disable H <mirror disk lunpath> Criticality reported by CRA: CRA_WARNING or CRA_DATA_CRITICAL The olrad (1M) command with C option reports the resource usage and its criticality. The above operations with scsimgr and olrad (1M) with r or d options will display a WARNING indicating that the operation will make the lunpath to mirror disk unavailable. However, if there is any process that is keeping the mirror disk open, then the above operations will return the CRA_DATA_CRITICAL error. If the user chooses to proceed with the requested operation or override the error (in case of CRA_DATA_CRITICAL), the user must remember that the system is then vulnerable and a failure might occur while reading data from the primary mirror copy. This could potentially bring down the system. If any of these operations must be performed for system administration purposes, then an alternative path must be configured using the same or a different card (in case of the scsimgr operation) or through an alternative healthy card (in case of the olrad operation) before performing the above operations. Scenario 18 System Configuration: Disk has been configured as a mirror copy of a logical volume (could be either LVM or VxVM volume) with file system mounted on it. This mirror disk has multiple lunpaths which originate from different ports of the same card (for example, AB379B dual-ported 4GB FC card). # scsimgr disable H <mirror disk lunpath> Criticality reported by CRA: CRA_WARNING or CRA_DATA_CRITICAL The above operation with scsimgr will not have any user impact as long as there is at least one another lunpath available to the mirror disk. However, if the last lunpath is disabled, then this operation will report a WARNING. Though there are multiple paths, the above operations with olrad(1m) with r or d options will display a WARNING indicating that the operation will make all lunpaths to the mirror disk unavailable as they are all from the same card. 14

15 However, if there is any process that is keeping the mirror disk open, then the above operations will return the CRA_DATA_CRITICAL error. If the user chooses to proceed with the requested operation or overrides the error (that is, in case of CRA_DATA_CRITICAL error), it may result in system failure and/or loss of data, in the event of failure of the primary mirror copy. If any of these operations must be performed for system administration purposes, an alternative path must be configured, perhaps using an alternative healthy card (in case of the olrad operation) before performing the above operations. Scenario 19 System Configuration: A logical volume (could be either LVM or VxVM volume) has been created with two or more disks (that is, PVs), each PV has a single lunpath through same or different cards and a file system is residing on that logical volume. # scsimgr disable H <PV lunpath> The above operations with scsimgr on any of lunpaths to the PV in the lvol and olrad(1m) with r or d options on any of the cards, which provides a lunpath to the PV, will fail with CRA_DATA_CRITICAL error. Overriding the error and proceeding with the operation will make the lunpath unavailable and this will make the file system residing on the logical volume inaccessible and/or result in loss of data on the file system. If any of these operations must be performed for system administration purposes, then an alternative path must be configured using the same or a different card (in case of the scsimgr operation) or through an alternative healthy card (in case of the olrad operation) before performing the above operations. Scenario 20 System Configuration: File system is mounted on a disk and the disk has multiple lunpaths through same (that is, with multiple ports) or different cards. Operation Performed: # scsimgr disable H <lunpath> Criticality reported by CRA: No user impact The above operation will not have any user impact and no user action is required as long as there is at least one another lunpath to the disk. However, if the last lunpath is being disabled, the scsimgr will report a CRA_DATA_CRITICAL error. However, the user may override this error using the scsimgr disable F option. Overriding the error and proceeding with the operation will make the disk unavailable and this will make the file system residing on the disk inaccessible. 15

16 DLKM Driver unload scenarios Scenario 21 System Configuration: A logical volume (could be either LVM or VxVM volume) has been created with two or more disks (that is, PVs), each PV has a single lunpath through same or different cards and a file system is residing on that logical volume. Operation Performed: # kcmodule <interface driver associated with card>=unused The above kcmodule operation on any of the I/O drivers controlling a card, which provides a lunpath to the PV, will fail with CRA_DATA_CRITICAL error. If this operation must be performed for system administration purposes, then an alternative path must be configured using different I/O driver than the one that is about to be unloaded, before performing the above operation. Scenario 22 System Configuration: HP-UX OS is running and a card in one of the I/O slots is connected to some critical resources, such as boot disk, disk with a File system, SG quorum disk, etc. The driver associated with the card is a Dynamically Loadable Kernel Module (DLKM) driver. Operation Performed: User tries to remove the driver software of the dynamically loaded kernel I/O driver associated with the card, using the SD-UX swremove command. Criticality reported by CRA: The CRA result is not reported to the user. In such a scenario, where the dynamically loaded kernel I/O driver is in use, removing the driver software using the swremove command will need a system reboot to complete operation. The recommended procedure to avoid this system reboot is to perform CRA analysis using the kcmodule unload operation first before removing the driver software. If the CRA reports any critical resource usage, configure another alternative path to the resource using a different dynamically loadable I/O driver, then unload the I/O driver, for which the associated software must be removed from the system. Thereafter, proceed to remove the driver software. If the system has an earlier version of the I/O driver, which is statically linked, then regardless of criticality of the driver usage the swremove operation will result in system reboot. However, if the earlier version of the I/O driver is also a dynamically loadable kernel module, then the swremove operation will result in system reboot only if the concerned I/O driver has any critical resource usage. If there is no critical usage of the I/O driver, the swremove will not result in system reboot. 16

17 Scenario 23 System Configuration: HP-UX OS running in Service Guard (SG) cluster environment with common lock disk between the cluster nodes and the cluster is configured with a common lock disk, which has multiple lunpath(s) through different cards, controlled by different I/O drivers. Operation Performed: # kcmodule <interface driver associated with card>=unused Criticality reported by CRA: No user impact The above kcmodule operation on any one of the I/O drivers controlling a card, which provides a lunpath to the common lock disk, will not have any user impact. If the common lock disk has been configured using path dependent device special file, then the above operation will report a CRA_WARNING and the user will be allowed to proceed with the operation. No user action is required. Cards in PCI Error and Suspended State Scenarios Scenario 24 System Configuration: HP-UX OS is running and an interface card in one of the PCI I/O slots is in PCI ERROR state. That is, ioscan(1m) displays the nodes below the slot in ERROR state. Criticality reported by CRA: CRA_WARNING Under normal situations, when an I/O slot occupied with a card encounters any PCI errors, the automatic PCI error recovery feature of HP-UX 11iv3 will clear the PCI error and make the slot and card functional again. No user intervention is needed to recover from this scenario. However under certain conditions (for example, the card is faulty), the automatic PCI error recovery may fail to clear the error and the slot will be left in ERROR state to prevent any system crash leading to system downtime. In such scenario, the user can choose to perform one of the following operations: The user can suspend the card using the olrad r option. Then, a manual recovery operation using the olrad R option can be attempted to recover from the error. If the card is in a recoverable state, the card should start functioning normally after the olrad -R operation. However, if the card is faulty, the above manual recovery operation may also fail. In such a scenario, the user can remove the faulty card from the slot, replace with a new but same type of card and then run the olrad R operation. Alternatively, the user can choose to delete the card using the olrad d option. Once this operation is complete, the user can physically add a new card to the slot and run olrad A and thus recover from the PCI error scenario. 17

18 Scenario 25 System Configuration: HP-UX OS is running and an interface card in one of the PCI I/O slots is in PCI ERROR state. That is, ioscan(1m) displays the nodes below the slot in ERROR state. Operation Performed: # kcmodule <interface driver associated with card>=unused Criticality reported by CRA: Driver Unload failure Under normal situations, when an I/O slot occupied with a card encounters any PCI errors, the automatic PCI error recovery feature of HP-UX 11iv3 will clear the PCI error and make the slot and card functional again. However, under certain conditions (for example, the card is faulty), the automatic PCI error recovery may fail to clear the error and in which case, the slot will be left in ERROR state to prevent any system crash leading to system downtime. In such a scenario, the user can perform olrad(1m) SUSPEND or DELETE operation with r or d options, respectively, to clear the PCI error manually and then perform the driver unload operation again. Scenario 26 System Configuration: HP-UX OS is running and a card is in olrad (1M) suspended state. That is, ioscan(1m) displays the nodes below the slot in SUSPENDED state. Operation Performed: # olrad -d <slot id> Criticality reported by CRA: CRA_WARNING In such a scenario, the above olrad (1M) DELETE operation will complete with a WARNING and no user action is required. Scenario 27 System Configuration: HP-UX OS is running and a card is in olrad (1M) suspended state. That is, ioscan(1m) displays the nodes below the slot in SUSPENDED state. Operation Performed: # kcmodule <interface driver associated with card>=unused Criticality reported by CRA: Driver Unload failure In such a scenario, the recommended procedure is to perform the olrad(1m) RESUME or DELETE operation with r or d, respectively, and then perform the intended kcmodule driver unload operation. 18

19 SAN Storage Configuration Scenario Scenario 28 System Configuration: A set of LUNs is configured on the storage device to be visible to a particular fiber channel world wide name (WWN), which belongs to a fiber channel HBA on a system in the SAN. Possible usages of these LUNs could be: A file system is mounted on one of the LUNs. An I/O is in progress to one of the LUNs. This LUN is being used as a dump device. Operation Performed: # olrad -r -d <slot id>. The above operations with olrad(1m) with r or d options (if it were to be allowed) will render the lunpaths unavailable to reach the LUNs on the device through this HBA. This will result in data loss. If any of these operations must be performed for system administration purposes, ensure that the disks are reachable through an alternative card. This will involve reconfiguration on the device to enable the same LUNs to be visible to the WWN of the alternative card. LAN CRA Scenarios IP Address Configured NIC Scenarios Scenario 1 System Configuration: HP-UX System is running, which has a Network interface card (NIC), configured with an IP address and connected to network. User Scenario: Replace the NIC with another card of same model. Delete and add a different type of card which can be claimed by the same or different driver. The above operations with olrad(1m) with r or d options (if it were to be allowed) will render the system inaccessible to network. This may result in network applications running on the system hanging, inaccessible to the network client applications, leading to application downtime or any other indeterminate system state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with CRA_DATA_CRITICAL error. 19

20 However, if the user wants to override the CRA criticality report, he can do so at his own risk by using the f option of the olrad command. If any of these operations must be performed for system administration purposes, the system must be made accessible through alternative network interface card, which does not have IP address already configured. To do this, first verify that the alternative NIC has a physical connection to the network. This can be verified using one of these commands: nwmgr, linkloop. Perform the following steps to re-assign the IP address, from the currently configured NIC to an alternative NIC with a physical connection to the network. 1. Open the /etc/rc.config.d/netconf file and set the value of X in below lines with the instance number of the alternative NIC. INTERFACE_NAME[0]="lanX" IP_ADDRESS[0]=" " 2. Stop the current networking configuration using the ifconfig command to down and unplumb the IP address configured for the NIC to be replaced. 3. Run the following command: /sbin/init.d/net stop 4. Run the following command: /sbin/init.d/net start 5. Manually reconfigure any additional networking setup as needed. 6. Verify the network connection and IP address using the netstat and ifconfig command. Scenario 2 System Configuration: HP-UX System is running, which has a Network interface card (NIC), configured with an IP address and connected to network and the NIC card driver is in SUSPENDED state. User Scenario: Card driver is in SUSPENDED state and the user tries to delete and add a different type of card, which can be claimed by the same or a different driver. Operation Performed: # olrad C -d <slot id> Criticality reported by CRA: CRA_WARNING The olrad(1m) d option (if it were to be allowed) will delete the IP configuration and its LAN port from the system. This will result in stale or undefined state of network applications running on the system using that IP address. Because of the above reason, CRA reports WARNING to notify the usage of IP address. If the user needs to delete the card, before performing the olrad operation, ensure that the IP address is removed from the configuration of applications, which are using the IP address. To avoid the stale 20

21 configuration, re-assign the IP address to an alternative Network interface card (NIC) and then proceed with the olrad deletion. Scenario 3 System Configuration: HP-UX System is running, which has a multi-port Network interface card (NIC), with one or more NIC ports configured with IP addresses and connected to network. User Scenario: Replace the card with another card of same model. Delete and add a different type of card which can be claimed by the same or different driver. The above operations with olrad(1m) with r or d option (if it were to be allowed) will render the system inaccessible to the network. This may result in network applications running on the system hanging, inaccessible to the network client applications, leading to application downtime or any other indeterminate system state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with CRA_DATA_CRITICAL error. However, if the user wants to override the CRA criticality report, he can do so at his own risk by using the f option of the olrad command. If any of these operations must be performed for system administration purposes, then the user must reassign all the IP addresses configured on the ports of the NIC to be replaced or deleted to alternative NIC card(s). Follow the procedure in Scenario 1 to delete and re-assign the IP addresses. Scenario 4 System Configuration: HP-UX System is running, which has a single or multi-port Network interface card (NIC), with one or more NIC ports configured with IP addresses and connected to network. User Scenario: Unload the interface driver of the LAN card to which the IP address is set. Operation Performed: # kcmodule <interface driver associated with card>=unused The above operation with kcmodule(1m) will render the system inaccessible to network. This may result in network applications running on the system hanging, inaccessible to the network client applications, leading to application downtime or any other indeterminate system state. 21

22 Because of the above reasons, the requested unload operation will not be allowed by CRA to proceed and the operation will fail with CRA_DATA_CRITICAL error. If this operation must be performed for system administration purposes, the user must re-assign all the IP addresses configured on the ports of the NIC, for which the driver is about to be unloaded to alternative NIC card(s) controlled by different NIC driver than the one about to be unloaded. Follow the procedure in Scenario 1 to delete and re-assign the IP addresses. Scenario 5 System Configuration: HP-UX system is running with a multi-function combo card (that is, a card with NIC and mass-storage ports) and an IP address is assigned to the NIC port. Also, a storage device with a single lunpath and used as critical resource, such as boot/dump/swap device, Service Guard quorum disk, is connected to the mass-storage port of the combo card. User Scenario: Replace the card with another card of same model. Delete and add a different type of card which can be claimed by the same or different driver. Criticality reported by CRA: CRA_SYSTEM_CRITICAL The olrad(1m) with r or d option (if it were to be allowed) will render the boot/dump/swap device and Service Guard quorum disk unavailable, thereby resulting in system crash, hang, or any other indeterminate system state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with CRA_SYSTEM_CRITICAL error. If any of these operations must be performed for system administration purposes, before performing these olrad operations, ensure that the system critical storage device is reachable through an alternative healthy card; also, re-assign the IP address configured for the NIC port of the combo card to an alternative NIC (either with a combo card of similar type or separate NIC and mass-storage interface cards). VLAN and APA Configuration Scenarios Scenario 6 System Configuration: HP-UX System is running, which has a single or multi-port Network interface card (NIC), with one or more NIC ports configured with VLAN and IP addresses and connected to network. User Scenario: Replace the card with another card of same model. Delete and add a different type of card which can be claimed by the same or different driver. 22

23 The above operations with olrad(1m) with r or d option (if it were to be allowed) will render the system inaccessible to the network. This may result in network applications running on the system hanging, inaccessible to network client applications, leading to application downtime or any other indeterminate system state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with CRA_DATA_CRITICAL error. However if the user wants to override the CRA criticality report, he can do so at his own risk by using the f option of olrad command. If any of these operations must be performed for system administration purposes, the user must ensure that all the IP addresses configured for VLAN IDs are deleted (see ifconfig(1m)) and VLAN configurations are removed from all NIC port(s) (see lanadmin(1m), nwmgr(1m)). Select NIC port(s) from alternative card(s), reconfigure the VLAN on alternative ports (see lanadmin(1m), nwmgr(1m)) and re-assign the IP addresses set for VLAN IDs, as done earlier. Scenario 7 System Configuration: HP-UX system is running, which has a single or multi-port Network interface card (NIC), with one or more NIC ports configured with VLAN and IP addresses and connected to network. User Scenario: Deletion of VLAN interface. Operation Performed: #nwmgr --cra --delete c lan<vppa> The nwmgr(1m) command with the --cra option reports the resource usage and its criticality. The above operation with nwmgr(1m) with the delete option (if it were to be allowed) will delete the vlan and IP address making the system inaccessible to the network. This may result in network applications running on the system hanging, inaccessible to the network client applications, leading to application downtime or any other indeterminate system state. Because of the above reasons, the requested nwmgr operation will not be allowed by CRA to proceed and the operation will fail with CRA_DATA_CRITICAL error. If the user wants to perform the delete operation without impacting system or application availability, another LAN port must be configured with a similar VLAN configuration. Scenario 8 System Configuration: HP-UX system is running with IP address assigned to an APA (Auto-port Aggregation) group and the link aggregate contains LAN ports from two or more NICs. User Scenario: Replace the card with another card of same model. 23

24 To delete and add a different type of card which can be claimed by the same or different driver. Criticality reported by CRA: CRA_WARNING The above olrad(1m) operations with r or d option will make alteration or reduction of the NIC ports in the link aggregate, which may result in reduction of the available network I/O bandwidth of the APA group and also possibly may impact LAN failover capability. Because of the above reasons, the CRA will allow the requested olrad operations with a WARNING. If any of these operations must be performed for system administration purposes without degrading the link aggregate bandwidth or without compromising the LAN failover capability, before proceeding with the intended olrad operations, the user must configure additional LAN ports from alternative NICs into the affected APA group. Scenario 9 System Configuration: HP-UX system is running with IP address assigned to an APA (Auto-port Aggregation) group and the link aggregate contains LAN ports from a single multi-port NIC. User Scenario: Replace the card with another card of same model. To delete and add a different type of card which can be claimed by the same or different driver. The above operations with olrad(1m) with r or d option (if it were to be allowed) will make the system inaccessible to the network. This may result in network applications running on the system hanging, inaccessible to the network client applications, leading to application downtime or any other indeterminate system state. Because of the above reasons, the requested olrad operations will not be allowed by CRA to proceed and the operation will fail with CRA_DATA_CRITICAL error. However, if the user wants to override the CRA criticality report, he can do so at his own risk by using the f option of olrad command. If any of these operations must be performed for system administration purposes without degrading the link aggregate bandwidth or without compromising the LAN failover capability, before proceeding with the intended olrad operations, the user must configure additional LAN ports from alternative NICs into the affected APA group. Scenario 10 System Configuration: HP-UX system is running with IP address assigned to an APA (Auto-port Aggregation) group and the link aggregate contains LAN ports from two or more NICs, some of which controlled by different NIC driver. 24