High perfomance clustering using the 9125-F2C Management Guide

Transcription

1 Power Systems High perfomance clustering using the 9125-F2C Management Guide Revision 1.3 p. 1 of 98

2 p. 2 of 98

3 Power Systems High perfomance clustering using the 9125-F2C Management Guide Revision 1.3 p. 3 of 98

4 Note Before using this information and the product it supports, read the information in the Safety Notices section and in the IBM Systems Safety Notices manual, G , and the IBM Environmental Notices and User Guide, Z This edition applies to IBM Power Systems 9125-F2C servers that contain the POWER 7 processor Copyright IBM Corporation 2011.US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp p. 4 of 98

5 Table of Contents 1Safety notices...7 2Manage High performance clustering using the 9125-F2C Using the Cluster Guides Cluster Guide Revision History Clustering systems by using 9125-F2C Cluster information resources General cluster information resources Cluster hardware information resources Cluster management software information resources Cluster software and firmware information resources Cluster management Introduction to Cluster management Cluster management flow Cluster management components and their use xCAT TEAL management ISNM LoadLeveler GPFS RAID HMC Cluster management tasks Managing performance Network management overview ISNM use of the cluster database HFI network performance data collection Network problem reporting Managing and monitoring the network Performance data gathering TEAL - Event management Monitoring with TEAL Monitoring of the cluster xCAT monitoring ISNM monitoring TEAL monitoring Other monitoring Power 775 Availability Plus Management HFI network health checks Maintenance Applying System Firmware Updates Applying power code updates Updating the EMS Updating Management Software Managing Multiple Operating System Images Command references Network management (ISNM) command reference...48 p. 5 of 98

6 3.1.1chnwconfig chnwfence chnwm chnwolct chnwsvrconfig cnm.snap, cnm.snap.linux isrmon isrperf lsnwcomponents lsnwconfig lsnwdownhw lsnwexpnbrs lsnwgc lsnwlinkinfo ISNM Link Status lsnwloc ISNM Drawer State lsnwmiswire lsnwtopo nwlinkdiag ISNM location naming conventions Operating system command reference for network management chdev (AIX only) hf_dump (Linux) ifhf_dump (AIX) hfi_dump hfi_snap Notices...92 p. 6 of 98

7 1 Safety notices Safety notices may be printed throughout this guide: DANGER notices call attention to a situation that is potentially lethal or extremely hazardous to people. CAUTION notices call attention to a situation that is potentially hazardous to people because of some existing condition. Attention notices call attention to the possibility of damage to a program, device, system, or data. World Trade safety information Several countries require the safety information contained in product publications to be presented in their national languages. If this requirement applies to your country, a safety information booklet is included in the publications package shipped with the product. The booklet contains the safety information in your national language with references to the U.S. English source. Before using a U.S. English publication to install, operate, or service this product, you must first become familiar with the related safety information in the booklet. You should also refer to the booklet any time you do not clearly understand any safety information in the U.S. English publications. German safety information Das Produkt ist nicht für den Einsatz an Bildschirmarbeitsplätzen im Sinne 2 der Bildschirmarbeitsverordnung geeignet. Laser safety information IBM servers can use I/O cards or features that are fiber-optic based and that utilize lasers or LEDs. Laser compliance IBM servers may be installed inside or outside of an IT equipment rack. p. 7 of 98

8 DANGER When working on or around the system, observe the following precautions: Electrical voltage and current from power, telephone, and communication cables are hazardous. To avoid a shock hazard: Connect power to this unit only with the IBM provided power cord. Do not use the IBM provided power cord for any other product. Do not open or service any power supply assembly. Do not connect or disconnect any cables or perform installation, maintenance, or reconfiguration of this product during an electrical storm. The product might be equipped with multiple power cords. To remove all hazardous voltages, disconnect all power cords. Connect all power cords to a properly wired and grounded electrical outlet. Ensure that the outlet supplies proper voltage and phase rotation according to the system rating plate. Connect any equipment that will be attached to this product to properly wired outlets. When possible, use one hand only to connect or disconnect signal cables. Never turn on any equipment when there is evidence of fire, water, or structural damage. Disconnect the attached power cords, telecommunications systems, networks, and modems before you open the device covers, unless instructed otherwise in the installation and configuration procedures. Connect and disconnect cables as described in the following procedures when installing, moving, or opening covers on this product or attached devices. To Disconnect: 1. Turn off everything (unless instructed otherwise). 2. Remove the power cords from the outlets. 3. Remove the signal cables from the connectors. 4. Remove all cables from the devices To Connect: 1. Turn off everything (unless instructed otherwise). 2. Attach all cables to the devices. 3. Attach the signal cables to the connectors. 4. Attach the power cords to the outlets. 5. Turn on the devices. (D005) p. 8 of 98

9 DANGER Observe the following precautions when working on or around your IT rack system: Heavy equipment personal injury or equipment damage might result if mishandled. Always lower the leveling pads on the rack cabinet. Always install stabilizer brackets on the rack cabinet. To avoid hazardous conditions due to uneven mechanical loading, always install the heaviest devices in the bottom of the rack cabinet. Always install servers and optional devices starting from the bottom of the rack cabinet. Rack-mounted devices are not to be used as shelves or work spaces. Do not place objects on top of rack-mounted devices. Each rack cabinet might have more than one power cord. Be sure to disconnect all power cords in the rack cabinet when directed to disconnect power during servicing. Connect all devices installed in a rack cabinet to power devices installed in the same rack cabinet. Do not plug a power cord from a device installed in one rack cabinet into a power device installed in a different rack cabinet. An electrical outlet that is not correctly wired could place hazardous voltage on the metal parts of the system or the devices that attach to the system. It is the responsibility of the customer to ensure that the outlet is correctly wired and grounded to prevent an electrical shock. CAUTION Do not install a unit in a rack where the internal rack ambient temperatures will exceed the manufacturer's recommended ambient temperature for all your rack-mounted devices. Do not install a unit in a rack where the air flow is compromised. Ensure that air flow is not blocked or reduced on any side, front, or back of a unit used for air flow through the unit. Consideration should be given to the connection of the equipment to the supply circuit so that overloading of the circuits does not compromise the supply wiring or overcurrent protection. To provide the correct power connection to a rack, refer to the rating labels located on the equipment in the rack to determine the total power requirement of the supply circuit. (For sliding drawers.) Do not pull out or install any drawer or feature if the rack stabilizer brackets are not attached to the rack. Do not pull out more than one drawer at a time. The rack might become unstable if you pull out more than one drawer at a time. (For fixed drawers.) This drawer is a fixed drawer and must not be moved for servicing unless specified by the manufacturer. Attempting to move the drawer partially or completely out of the rack might cause the rack to become unstable or cause the drawer to fall out of the rack. (R001) p. 9 of 98

10 Removing components from the upper positions in the rack cabinet improves rack stability during relocation. Follow these general guidelines whenever you relocate a populated rack cabinet within a room or building: (R002) (L001) Reduce the weight of the rack cabinet by removing equipment starting at the top of the rack cabinet. When possible, restore the rack cabinet to the configuration of the rack cabinet as you received it. If this configuration is not known, you must observe the following precautions: Remove all devices in the 32U position and above. Ensure that the heaviest devices are installed in the bottom of the rack cabinet. Ensure that there are no empty U-levels between devices installed in the rack cabinet below the 32U level. If the rack cabinet you are relocating is part of a suite of rack cabinets, detach the rack cabinet from the suite. Inspect the route that you plan to take to eliminate potential hazards. Verify that the route that you choose can support the weight of the loaded rack cabinet. Refer to the documentation that comes with your rack cabinet for the weight of a loaded rack cabinet. Verify that all door openings are at least 760 x 230 mm (30 x 80 in.). Ensure that all devices, shelves, drawers, doors, and cables are secure. Ensure that the four leveling pads are raised to their highest position. Ensure that there is no stabilizer bracket installed on the rack cabinet during movement. Do not use a ramp inclined at more than 10 degrees. When the rack cabinet is in the new location, complete the following steps: Lower the four leveling pads. Install stabilizer brackets on the rack cabinet. If you removed any devices from the rack cabinet, repopulate the rack cabinet from the lowest position to the highest position. If a long-distance relocation is required, restore the rack cabinet to the configuration of the rack cabinet as you received it. Pack the rack cabinet in the original packaging material, or equivalent. Also lower the leveling pads to raise the casters off of the pallet and bolt the rack cabinet to the pallet. (L002) p. 10 of 98

11 (L003) or All lasers are certified in the U.S. to conform to the requirements of DHHS 21 CFR Subchapter J for class 1 laser products. Outside the U.S., they are certified to be in compliance with IEC as a class 1 laser product. Consult the label on each part for laser certification numbers and approval information. CAUTION: This product might contain one or more of the following devices: CD-ROM drive, DVD-ROM drive, DVD-RAM drive, or laser module, which are Class 1 laser products. Note the following information: (C026) Do not remove the covers. Removing the covers of the laser product could result in exposure to hazardous laser radiation. There are no serviceable parts inside the device. Use of the controls or adjustments or performance of procedures other than those specified herein might result in hazardous radiation exposure. p. 11 of 98

12 Data processing environments can contain equipment transmitting on system links with laser modules that operate at greater than Class 1 power levels. For this reason, never look into the end of an optical fiber cable or open receptacle. (C027) CAUTION: This product contains a Class 1M laser. Do not view directly with optical instruments. (C028) CAUTION: Some laser products contain an embedded Class 3A or Class 3B laser diode. Note the following information: laser radiation when open. Do not stare into the beam, do not view directly with optical instruments, and avoid direct exposure to the beam. (C030) Power and cabling information for NEBS (Network Equipment-Building System) GR-1089-CORE The following comments apply to the IBM servers that have been designated as conforming to NEBS (Network Equipment-Building System) GR-1089-CORE: The equipment is suitable for installation in the following: Network telecommunications facilities Locations where the NEC (National Electrical Code) applies The intrabuilding ports of this equipment are suitable for connection to intrabuilding or unexposed wiring or cabling only. The intrabuilding ports of this equipment must not be metallically connected to the interfaces that connect to the OSP (outside plant) or its wiring. These interfaces are designed for use as intrabuilding interfaces only (Type 2 or Type 4 ports as described in GR-1089-CORE) and require isolation from the exposed OSP cabling. The addition of primary protectors is not sufficient protection to connect these interfaces metallically to OSP wiring. Note: All Ethernet cables must be shielded and grounded at both ends. The ac-powered system does not require the use of an external surge protection device (SPD). The dc-powered system employs an isolated DC return (DC-I) design. The DC battery return terminal shall not be connected to the chassis or frame ground. p. 12 of 98

13 2 Manage High performance clustering using the 9125-F2C You can use this information to guide you through the process of managing the 9125-F2C clusters. It is part of of a series of guides to High Performance clustering using 9125-F2C High performance clustering using the 9125-F2C Planning and Installation Guide. High performance clustering using the 9125-F2C Management Guide. High performance clustering using the 9125-F2C Service Guide. This document is intended to serve as a consolidation point for important information for managing, maintaining and monitoring IBM High Performance Computing (HPC) Cluster using POWER technology and the Host Fabric Interface with the 9125-F2C server. It will serve as a consolidation point for the documents of the many component and subsystems that comprise a cluster. It will aid the reader in navigating these other documents in an efficient manner. Where necessary, it will recommend additional information to a person who has general skills within the discipline of tasks that is being documented. This document is not intended to replace existing guides for the various hardware units, firmware, operating system, software or applications publications produced by IBM or other vendors. Therefore, most detailed procedures that already exist in other documents will not be duplicated here. Instead, those other documents will be referenced by this document, and as necessary guidance will be given on how to work with generic information and procedures in the other documents. The intended audiences for this document are: HPC clients, including: System, network and site planners System administrators Network administrators System operators Other Information Technology professionals IBM personnel Planner for the cluster and the site System Service Representatives 2.1 Using the Cluster Guides The document sections are roughly in the order in which you will need them. Reference the Table of Contents for an outline of the topics in this document. The following table is a guide to finding topics in the the High performance clustering using the 9125-F2C Cluster Guides: High performance clustering using the 9125-F2C Planning and Installation Guide. High performance clustering using the 9125-F2C Management Guide. High performance clustering using the 9125-F2C Service Guide. p. 13 of 98

14 Once directed to a document for a certain topic, it is good practice to become familiar with the table of contents as a detailed outline of sub-topics within the document. Content highlights Content Description Document Clustering systems by using 9125-F2C A chapter which provides overview information. It is customized within each guide. All of them have references to information resources, and a a brief overview of cluster components, as well as how to use the cluster guides. The planning and installation guide also has a detailed overview of the cluster, its subsystems, its components, and unique characteristics. All Cluster Guides Detailed Overview Planning information Supported devices and software Planning worksheets In depth overview of the cluster, its subsystems, its components, and unique characteristics. This includes planning information and references for all major subsystems and components in the cluster. This is a list of supported devices and software at the time of publication. More up to date information is available in the HPC Central website. (see References) Worksheets to help plan the Planning and Installation Guide Planning and Installation Guide Planning and Installation Guide Planning and Installation Guide Cluster Installation This includes: the following. References to other documentation are frequent. Planning and Installation Guide Cluster management Installation Responsibilities Overview of the Installation Flow Installation steps organized by teams and objectives Detailed Installation procedures for topics not covered elsewhere This includes: the following. References to other documentation are frequent. Introduction to cluster management A cluster management flow HFI Network Management Monitoring the cluster Availability Plus monitoring Data gathering for problem resolution Cluster maintenance Command references especially for HFI network management Management Guide p. 14 of 98

15 Content Description Document Cluster service This includes: the following. References to other documentation are frequent. Introduction to cluster service Tables to narrow down to proper procedures References to detailed procedures documented elsewhere Hardware problem isolation topics Software problem isolation topics Power 775 Availability Plus actions EMS failover references Service Guide Cluster Guide Revision History The following outlines changes to the Cluster Guide. This includes changes across all of the individual guides that comprise the cluster guide. Table 1: Cluster Guide Revision History Revision Guide Changes 1 All Initial release 1.1 Planning and Installation Guide In the Overview: Diskless nodes: New section; information on updating the Cluster Service and Management Networks Updated LoadLeveler configuration information Section on Highly Available Service Nodes and LoadLeveler and TEAL GPFS Barrier-synchronization register overview Availability Plus info on improving system availability In Planning: Service and login nodes supported LoadLeveler Planning on Service Nodes Planning HIghly Available Service Nodes and LoadLeveler and GPFS TEAL monitoring and GPFS Typos and terminology cleanup Installation: Typos and grammar Installation Documentation links updated for TEAL; LoadLeveler Bringup of LoadLeveler in the installation flow Diskelss node logging configuration in the installation flow Placement of LoadLeveler and TEAL GPFS daemons in the installation flow Barrier Sync Register (BSR) Configuration in the installation flow p. 15 of 98

16 Revision Guide Changes Management Guide Changes to command references: chnwfence nwlinkdiag Service Guide Add Start Here section Update User reported problems Update System Log reported problems Update Problem reported in TEAL Update Problem reported in Service Focal Point Update Problem reported in component or subsystem log Add Node will not boot Add Checking routing mode Add xcat problems Add TEAL Service Procedures Add Acting on HFI Network Alert Add Acting on HFI Network Serviceable Event Isolation Procedure updates and additions: HFI_DDG, HFI_LDG, HFINSFP, HFINNBR, HFILONG Add Data collection section Extensive HFI Network Locations updates Add HFI Network Event Reference Terminology updates 1.2 All Add section for Cluster Guide Revision History. Planning and Installation Guide Management Guide Service Guide Power 775 Availability Plus Overview updates: Some numbered list format issues for the Availability Plus overview. Statement regarding when A+ Management begins Command reference updates: Network Management commands: cnm.snap output info OS system commmand reference for network management chghfi changed to chdev add ifhf_dump for AIX add hfi_snap Data collection updates for: ISNM network HFI network or HFI driver specific to a node TEAL 1.3 Planning and Installation Guide Information on planning highly available service nodes Management Guide Fix some typos Add isrmon and isrperf commands to Command references Service Guide Add information on TEAL, ISNM and HFI driver logs. p. 16 of 98

17 p. 17 of 98

18 2.2 Clustering systems by using 9125-F2C Clustering systems by using 9125-F2C provides references to information resources, and a brief overview of cluster components.the cluster. The cluster consists of many components and subsystems, each with an important task aimed at accomplishing user work and maintaining the ability to do so in the most efficient manner possible. The following paragraphs introduce and briefly describe various subsystems and their main components. The compute subsystem consists of: The 9125-F2C systems configured as nodes dedicated to performing computational tasks. These are diskless nodes. Operating System images customized for compute nodes Applications The storage subsystem consists of: 9125-F2C systems configured as IO nodes dedicated to serving the data for the other nodes in the cluster. These are diskless nodes. Operating System images customized for IO nodes. SAS adapters in the 9125-F2C systems which are attached to Disk Enclosures Disk enclosures Global Parallel file system (GPFS ) The communications subsystem consists of: The Host Fabric Interface technology in the 9125-F2C Busses from processor modules to the switching hub in an octant. For more information see, Octant on page 16. Local links (LL-links) between octants in a 9125-F2C. For more information see, Octant on page 16. Local remote links (LR-links) between drawers in a SuperNode. Distance links (D-links) between SuperNodes The operating system drivers The IBM User space protocols AIX and Linux IP drivers The management subsystem consists of: Executive Management Server (EMS) running key management software. Different types of servers might be used. For more details, see <<to be added>>. Operating system on the EMS Utility Nodes used as xcat service nodes. These serve operating systems to local diskless nodes and provide a hierarchical access to hardware and nodes from the EMS console. Extreme Cloud Administration Toolkit (xcat) running on the EMS and service Utility Nodes. For information on xcat, go to xcat website. ( DB2 - running on the EMS and service Utility Nodes. p. 18 of 98

19 Integrated Switch Network Manager (ISNM) running on the EMS Toolkit for Event Analysis and Logging (TEAL) Other Utility nodes are customizable for each site. These utility nodes must be 9125-F2C servers. Login Nodes are required Other site unique nodes such as tape subsystem servers. These unique nodes are optional, but must be 9125-F2C systems. A summary of node types: Compute Nodes: provide computational capability. Compute nodes generally comprise most of the cluster. IO Nodes: provide connectivity to the storage subsystem. The number of IO nodes is driven by the amount of required storage. Utility Nodes provide unique functions. Service Nodes running xcat, which serve operating systems to local diskless nodes and provide a hierarchical access to hardware and nodes from the EMS console. These service nodes are required. Login nodes which provide a log-in gateway into the cluster. These login nodes are required. Other site unique nodes, such as tape subsystem servers. These unique nodes are optional, but must be 9125-F2C systems. Note: The EMS and HMC are considered to be management consoles and not nodes. Key concepts that are introduced with this cluster are: Most of the nodes are diskless and get their operating systems and scratch space served by the service Utility nodes. Diskless boot is performed over the HFI. Power 775 Availability Plus configuration for processors, switching hubs and HFI cables provides extra resource in the cluster to let these components to fail. And not be replaced until the cluster is nearing the possibility of not being able to achieve agreed upon workload capabilities Cluster information resources Cluster information resources provides references to information resources for the cluster, its subsystems and components. The following tables indicate important documentation for the cluster, where to get it and when to use it relative to Planning, Installation, and Management and Service phases of a clusters life. The tables are arranged into categories of components: General cluster information resources, on page 1 Cluster hardware information resources, on page 21 Cluster management software information resources, on page 21 Cluster software and firmware information resources, on page 22 p. 19 of 98

20 General cluster information resources The following table lists general cluster information resources: Table 2. General cluster resources Component Document Plan Install Manage and service* IBM Cluster Information IBM Clusters with HFI and 9125-F2C website HPC Central wiki and HPC Central forum Power Systems High performance clustering using the 9125-F2C Planning and Installation Guide Power Systems High performance clustering using the 9125-F2C Management Guide Power Systems High performance clustering using the 9125-F2C Service Guide IBM HPC Clustering with Power 775 servers - Service Packs portion of the IBM High Performance Computing Clusters Service Packs website ( BM+High+Performance+Computing+Clusters+Service+Packs#IBM HighPerformanceComputingClustersServicePacks- IBMHPCClusteringwithPower775serversServicePacks) The HPC Central wiki enables collaboration between customers and IBM teams. This wiki includes questions and comments. display/hpccentral/hpc+central IBM Fix Central Note: IBM Fix Central should only be used in conjunction with the readme website for IBM Clusters with HFI and 9125-F2C, above. This is because Fix Central may contain code levels that have been verified for other environments, but not for this cluster * M = Management only; S=Service only; X=Both management and service. X X M S X X X X X X X X p. 20 of 98

21 Cluster hardware information resources The following table lists cluster hardware resources: Table 3. Cluster hardware information resources Component Document Plan Install Manage and service Site planning for all IBM systems POWER F2C system Logical partitioning for all systems System i and System p Site Preparation and Physical Planning Guides Site and Hardware Planning Guide Installation Guide for 9125-F2C Servicing the IBM system p 9125-F2C PCI Adapter Placement x x Worldwide Customized Installation Instructions (WCII) IBM service representative installation instructions for IBM machines and features Logical Partitioning Guide x x x x x x Install Instructions for IBM LPAR on System i and System P x IBM Power Systems documentation is available in the IBM Power Systems Hardware Information Center. Any exceptions to the location of information resources for cluster hardware as stated above have been noted in the table. Any future changes to the location of information that occur before a new release of this document will be noted in the HPC Central website Cluster management software information resources The following table lists cluster management software information resources: Table 4. Cluster management software resources Component Document Plan Install Manage and service Hardware Management Console (HMC) xcat Installation and Operations Guide for the HMC x x Operations Guide for the HMC and Managed Systems For xcat documentation, go to xcat x x x x Integrated Switch Network Manager (ISNM) documentation ( /index.php?title=xcat_documentation) This document x x x p. 21 of 98

22 Toolkit For Event Analysis and Logging (TEAL) On sourceforge.net: eal/index.php?title=main_page x x x When installed, on EMS: /opt/teal/doc/teal_guide.pdf IBM Power Systems documentation is available in the IBM Power Systems Hardware Information Center Cluster software and firmware information resources The following table lists cluster software and firmware information resources. Table 5. Cluster software and firmware information resources Component Document Plan Install Manage and service AIX Linux AIX Information Center x x x Obtain information from your Linux distribution source x x x DB2 For information, go to DB2 x x x IBM HPC Clusters Software GPFS: Concepts, Planning, and Installation Guide x x GPFS: Administration and Programming Reference GPFS: Problem Determination Guide GPFS: Data Management API Guide Tivoli Workload Scheduler LoadLeveler for AIX: Installation Guide (SC ) Tivoli Workload Scheduler LoadLeveler for Linux: Installation Guide (SC ) Tivoli Workload Scheduler LoadLeveler: Using and administering (SC ) Tivoli Workload Scheduler LoadLeveler: Command and API Reference (SC ) Tivoli Workload Scheduler LoadLeveler: Diagnosis and Messages Guide (SC ) Tivoli Workload Scheduler LoadLeveler: Resource Manager Guide (SC ) Parallel Environment: Installation x x Parallel Environment: Messages x x Parallel Environment: Operation and Use, Volumes 1 and 2 Parallel Environment: MPI Programming Guide Parallel Environment: MPI Subroutine Reference x x x x x x x x x x x x x x x x The IBM HPC Clusters Software Information can be found at the IBM Cluster Information Center. p. 22 of 98

23 p. 23 of 98

24 2.3 Cluster management Use this information to learn about the activities, applications, and tasks required for cluster management. This cluster management topic will bealong the lines of theory and best practice rather than detailed procedures. Documents and webites referenced in this section can be found in Cluster information resources, on page 19. Begin with the Introduction to Cluster management. You will also find useful commands that are unique to managing a cluster of p775 servers described in Command references, on page Introduction to Cluster management This section provides information about typical as well as crucial tasks in managing a cluster of 9125-F2C systems. Locations of and links to documents referred to in this section can be found in Cluster information resources, on page 19. This section is broken into the following sections. A brief description of how to use each section is included. Cluster management flow, on page 24 illustrates the flow of tasks for typical management activities in a cluster Cluster management components and their use, on page 25 briefly describes various components used to manage the cluster. Cluster management tasks, on page 27 describes the various management tasks and the components used to perform them Network management overview, on page 29 describes management of the communications network. Monitoring of the cluster, on page 40 describes monitoring the cluster Power 775 Availability Plus Management, on page 41 describes management activities required for Power 775 Availability Plus. Maintenance, on page 45 describes typical maintenance activities. For an overview of the Management subsystem see Management subsystem overview in the High perfomance clustering using the 9125-F2C Planning and Installation Guide. You will also find the Table of Contents is a good navigation resource to reference specific topics in this chapter Cluster management flow The following illustrates a typical flow of cluster management activities from the point of a successful installation onward. It can be referenced while reading the Cluster management tasks on page 54. p. 24 of 98

25 Figure 1: Management Task Flow Cluster management components and their use This topic provides information about cluster management components and their use. There is also information in the Overview section of the High perfomance clustering using the 9125-F2C Planning and Installation Guide xcat p. 25 of 98

26 xcat (Extreme Cluster Administration Tool) is a toolkit that provides support for the deployment and administration of large cluster environments. In a 9125-F2C server cluster, there is a hierarchical implementation used for xcat where one or more server nodes are given control of a group of managed nodes (such as compute or IO nodes). There is a central Management Node (EMS) that is used as the administration console. For information about xcat Architecture, see xcat 2 Architecture. ( A list of xcat Features includes: Deploying diskless and diskfull nodes. Node group support. Node discovery. Operating system image management. upport for user-provided customization scripts. xcat supports the automatic running of user-provided customization scripts on the nodes when they are deployed (installed or booted). xcat data store in plug-in relational database using DB2 Hardware control commands for discovering hardware, gathering MAC addresses, VPD, and environments, power control, initiating a network boot, and LPAR creation/deletion. Hierarchical support to allow large system to distribute the management of the cluster to service nodes. Remote console support. Parallel remote shell and remote copy commands. rsync support. Monitoring plug-in infrastructure (RMC, Ganglia) Notification infrastructure which lets users monitor xcat database table changes. Predefined conditions, responses and sensors for RMC Software and firmware inventory Allow continuous operation during cluster software updates using plug-in job scheduler ( LoadLeveler, Moab). Automatic setup for syslog, remote shell, DNS, DHCP, and ntp for both the xcat management node and the cluster nodes. For more information, see xcat features ( title=xcat_features) For information on xcat architecture, see xcat 2 architecture ( %01) TEAL management Toolkit for Event Analysis and Logging (TEAL) can be instrumented during installation to help monitor the cluster for asynchronous events affecting: Server hardware, power and cooling and the disk enclosure The HFI network LoadLeveler PNSD GPFS p. 26 of 98

27 For some more information about TEAL, see references incluster information resources, on page ISNM The Integrated Switch Network Manager (ISNM) is used to manage and monitor the HFI network. For the most part, ISNM is autonomous in managing the network once you have configured it during installation. Therefore, after installation, ISNM is largely used to monitor the HFI network. Scripts can be written and called by cronjobs to perform periodic health checks on the network using ISNM. More information is available in Network management overview, on page 29, and Monitoring of the cluster, on page LoadLeveler LoadLeveler is used for job management. Refer to LoadLeveler documentation for more information. For references, see Cluster information resources, on page GPFS RAID GPFS RAID is a function that is used to manage the disks in the disk enclosures. In particular, the disk hospital function can be queried on a regular basis to see the health of the disk subsystem. This is not completely necessary, because disk problems that require service are reported to the HMC serviceable events and to TEAL. For more information on GPFS, see the GPFS references in Cluster information resources, on page HMC The Hardware Management Console (HMC) runs on a single server and is used to help manage the 9125-F2C servers. The traditional HMC functions for configuring and controlling the servers can now be done via xcat. The HMC is largely used: During installation For reporting hardware serviceable events especially through Electronic Service Agent (ESA), which is also commonly known as call-home. By service personal to perform guided service actions Cluster management tasks This topic provides information about cluster management tasks. Typical tasks to be performed while managing the cluster are: Monitoring TEAL for alerts indicating problems Maintaining job management pools via LoadLeveler Periodic HFI network health checks Monitoring GPFS RAID for disk problems Updating code levels Managing performance p. 27 of 98

28 Managing performance This topic provides information about managing performance. Managing performance of a cluster is an ongoing task. Periodic monitoring of the health of a cluster is a major task, especially once acceptable performance levels have been achieved. However, as new applications are deployed, it may be necessary to tune their performance environment as well. Or, perhaps, how the cluster is tasked in general has changed and some higher level tuning needs to be done. Parallel Environment environment variables Parallel Environment environment variables' affect on performance vary depending on the application. There is typically some experimentation that must occur to find the best combination of settings. Therefore, the following is only a guideline and various combinations could be used as site-specific settings. This section is intended for reference rather than in-depth discussion of performance topics. See the Performance Planning Worksheet for any cluster planning that was done. If you are tuning a particular application, the following environment variables are typically considered. The following environment variable settings help improve performance of most applications: export MP_TASK_AFFINITY=CORE # or use some other form of processor affinity export LAPI_DEBUG_SLOT_ATT_THRESH= # set a high threshold to effectively disable shared mem attach mode export MP_SHM_ATTACH_THRESH= # set a high threshold to effectively disable shared mem attach mode export MP_EAGER_LIMIT=32768 # this is the first variable to experiment with tuning export MP_EUIDEVICE=sn_single # although "sn_all" might be a good choice for some apps export MP_INSTANCES=1 # although "2" might be a good choice for apps which need more per-task bandwidth export MP_USE_BULK_XFER=no # use "no" for FIFO mode; "yes" for RDMA export MEMORY_AFFINITY=MCM export MP_DEVTYPE=hfi # direct the user-space protocol to use the HFI fabric export MP_EUILIB=us export MP_SHARED_MEMORY=yes export MP_SINGLE_THREAD=yes # only if one thread per task calls protocol lib export MP_EUIDEVELOP=min export MP_RFIFO_SIZE= # receive FIFO size, default 8-16MB/adapter window, use for "sn_all" export MP_POLLING_INTERVAL= # although a lower value is appropriate if we are retransmitting export MP_REXMIT_BUF_SIZE=32800 # set to eager limit + MPI msg header (32 bytes) export MP_BUFFER_MEM=128MB # default: 64MB, max: 256MB export MP_WAIT_MODE=poll # "poll"/"nopoll" export MP_CSS_INTERRUPT=no # "no"/"yes" (must be consistent with MP_WAIT_MODE) The following environment variable settings are more application dependent: export LAPI_DEBUG_MAX_MSGS_PER_DEST=50 # increase size of protocol flow control window export LAPI_DEBUG_SLOT_DATA_SIZE=65536 # default is 32768; sometimes is optimal for shared memory export LAPI_DEBUG_SLOTS_PER_TASK=8 # a shared memory variable (default is 32) export MP_REXMIT_BUF_COUNT=256 export MP_BULK_MIN_MSG_SIZE=$MP_EAGER_LIMIT # to establish minimum message size for RDMA (when RDMA mode is enabled) export MP_USE_BULK_XFER=yes # enables RDMA mode (otherwise FIFO mode) export HFI_RDMA_BREAKUP_COUNT=32 # number of segments HFI h/w p. 28 of 98

29 will work on for current rdma before switching to next pending rdma export LAPI_DEBUG_RDMA_MAX_SEG_SIZE= # underlying message size that LAPI/PAMI will chunk RDMA messages to export MP_FIFO_ROUTE_MODE=3 # "0" (hardwaredirect routing) is the default; "3" specifies hardware-indirect export MP_RDMA_ROUTE_MODE=3 # (ditto) Network management overview This topic provides overview about the Network management. Network management software provides a set of services to manage the Integrated Switch Routers (ISR) network. It configures and initializes the network, diagnoses hardware problems, and detects and provides recovery from network hardware errors. The ISR network management software is called Integrated Switch Network Manager (ISNM) and has two main components that cooperate to provide network management services. These components are: Central Network Manager (CNM) executes on the Executive Management Server (EMS). It provides a global view of the network, with administration of all network hardware from a central point. Local Network Management Controller (LNMC) runs on the server service processor as part of the drawer firmware (GFW). LNMC works in concert with CNM to provide setup and monitoring for the network hardware within the drawer. ISNM forwards network events to the Toolkit for Event Analysis and Logging (TEAL) for further diagnosis to root cause. Events of interest result in Alerts in the TEAL Alert DB and Serviceable Events in an instance of Service Focal Point (SFP) on a prescribed HMC. ISNM functions include: Validating the network topology Configuring ISR route tables during drawer power on Configuring the ISR multicast tables to support broadcast traffic Dynamically maintaining the hardware route tables in response to network hardware events Maintaining data to support software-directed indirect routing, this is available to the operating system. Configuring and maintaining the hardware globally synchronized counter Monitoring network hardware events Forwards events to TEAL for root cause analysis Provides an interface to Resource Monitoring and Control (RMC) for certain high level network events, such as Host Fabric Interface (HFI) Down. R e p o r t s drawer and frame level events that affect the fabric. These are generally used to account for a large number of link errors caused by such events. Providing support for hardware install and hardware verification Providingand supports diagnostic test capabilities and methods for isolating FRUs if the first error capture is insufficient for isolation to a single FRU Gathering Host Fabric Interface (HFI) and ISR performance counter data Providing status for network components Enabling and disabling of optical ports. This is needed primarily for disabling unused or faulty hardware and saving power. This is also used during phased installation of a cluster. Certain diagnostic procedures also use this function. p. 29 of 98

30 The ISNM has a Command Line Interface (CLI) and runs on the Executive Management Server (EMS) ISNM use of the cluster database This topic provides information about the ISNM use of the cluster database. ISNM requires that certain data specified in the cluster topology configuration file be provided to configure the fabric components. This information must be stored in the Cluster Database before CNM is started. When CNM starts, it reads this data from the Cluster database and uses it to configure the fabric components. For example, ISNM configures the expected neighbor information, which is used to detect miswires. The following information is stored in several database tables (nodelist, ppc and site). This information is: The network topology, loosely specified as a number of D-links between supernodes. For each 9125-F2C, the following : The supernode number, needed for route generation and Torrent hub and HFI identification. The position or id (0-3) of the drawer within the supernode, needed for route generation and Torrent hub and HFI identification. CNM will store certain HFI network information in the Cluster database if configured to do so. ISR and HFI performance counters can be stored. This information can be accessed using xcat tabdump and standard SQL. The ISNM database tables in the Cluster database are specified by using the tabspec files installed in /etc/xcat/lib/perl/xcat_schema HFI network performance data collection This topic provides information about the ISR network performance data collection. CNM collects performance counters from the ISR and the HFI. ISR counters are implemented for each of the 40 optical links, the seven Llocal links, and the ISR-to-HFI links. The four physical ports between the ISR and each HFI are treated as one logical link for performance counting, so there is one set of counters for each single logical ISR-to-HFI connection, bringing the total number of links for which data is collected to 49. The ISR counters that network management will collect for these 49 links are: Cycles Blocked From Sending (64 bits) Flits Sent (64 bits) Flits Dropped (32 bits) Link Level Retries (32 bits) From the HFI, CNM collects counters that aggregate the counts for all windows on the HFIs. The HFI counters are 56 bits but CNM stores these as 64-bit quantities. Window-specific counters implemented on the HFI are not collected by CNM. The HFI counters collected are: Packet Sent Count Packet Receive Count Packet Dropped from Sending Count Packet Dropped from Receive Count Immediate Send Packet Sent Count p. 30 of 98

31 Send/Receive Packet Send Count Full-RDMA Packet Sent Count Half-RDMA Packet Sent Count Small-RDMA Packet Sent Count IP Packet Sent Count CAU Packet Sent Count GUPS Packet Sent Count CNM will collect performance counters every five minutes as a default. Along with the counters collected at each hub chip, the global counter value will be collected and stored with the data. The collection interval is configurable. Detailed data is kept for a short period; for example, one week. If desired, data from a longer period can be kept, in a summary format. Performance data is stored in the Cluster database so that the data can be accessed using xcat tabdump or standard SQL interfaces Network problem reporting This topic provides information about network problem reporting. The network manager gathers information about the network, including hardware error register information, and forwards it to TEAL for analysis. If an event or group of events is considered important, it is forwarded to the Alert table in the Cluster Database. If the event or group of events is further considered to be serviceable, it is forwarded to a designated HMC to be displayed along with other serviceable events. The flow is illustrated in the following figure: Figure 15. Network problem reporting flow Managing and monitoring the network This topic provides information about managing and monitoring the network. Table 11. Management procedures Task Procedure p. 31 of 98

32 Display ISR network link status Use lsnwlinkinfo. For information on lsnwlinkinfo, see lsnwlinkinfo, on page 78. Display miswired links Display global counter information (location of master counter, number of active backups) Set or change ISNM configuration values Display the set of P7 IH drawers reachable on the service network. This command also displays the LNMC state. Display the topology information in use by the CNM and/or the LNMCs. Used to determine whether ISNM configuration mismatches are present in the system. Set/change the ISNM network configuration information (topology setting, supernode id, drawer id) on a P7 IH FSP. Display a list of faulty ISR network hardware. Configure the Hardware Server with the BPA and FSP IP addresses on the service network Use lsnwmiswires For information on lsnwmiswire, see lsnwmiswire, on page 85. Use lsnwgc. For information on lsnwgc, see lsnwgc on page 78 Use chnwconfig. For information on chnwconfig, see chnwconfig on page 49. Use lsnwcomponents. For information on lsnwcomponents, see lsnwloc on page 83 on Use page lsnwtopo. 78. For information on lsnwtopo, see lsnwtopo on page 87. Use chnwsvrconfig. For information on chnwsvrconfig, see chnwsvrconfig on page 55. Use lsdownhw. For information on lsdownhw, see lsdownhw on page 75. Use the xcat mkhwconn command Managing the network: This topic provides information about managing the network. The following are management procedures related to the network: Table 12. Management procedures Task Procedure Enable/Disable a link Use chnwfence; See chnwfence, on page 50. Changing performance counter monitoring Use chnwconfig. See chnwconfig on page 49. Turn on/off ISNM forwarding of certain events to RMC Use chnwconfig. See chnwconfig on page 49. Procedure to resolve network miswires: In order to resolve a network miswire, one must first obtain the complete list of miswires. It is then important to determine how cable connectors may have been swapped or left off completely. This is done by beginning with the actual placement and comparing with the expected placement. Some typical miswires might occur because of one or more of the following: Cable connectors may be swapped between adjacent D-link ports. These can be horizontal or vertical neighbors. This is the simplest and most likely miswire, and will may show up randomly. Also, p. 32 of 98

33 watch for patterns that indicate that someone may have repeated the same miswire. Cable connectors may be placed on what would otherwise be empty bulkhead connectors on the back of a server. This may happen on sparsely populated servers, or during a phased install when connections are checked before all links are cabled in the final installation. Cable connectors may be swapped between ports of neighboring servers in the same frame. These sorts of miswires may be isolated, but can also show up in a pattern of many miswires. When many such miswires occur, the cause could be human error, or it could be the result of faulty labeling or cable planning. Cable connectors may be swapped between supernodes such that they are attached to the corresponding server in each supernode. For example, D-link port 0 of hub 14 in drawer 1 of supernode 1 may be swapped with D-link port 0 of hub 14 in drawer 1 of supernode 2. This sort of miswire will tend to show up in a pattern of many similar miswires, and may be caused by faulty labeling or cable planning. Cable connectors may be swapped between corresponding ports in different building blocks. For example, if supernode 1 is the first supernode in building block 1 and supernode. This sort of miswire will tend to show up in a pattern of many similar miswires, and may be caused by faulty labeling or cable planning. Besides simple human error while running and attaching many cables, miswires can be caused by faulty labeling or planning. Such labeling or planning problems typically manifest themselves in many miswires. If you experience many miswires, consider investigating these possible causes. The following is an example output from lsnwmiswire where there is a simple swap between neighboring connectors on the server bulkhead: [c111ems1] > lsnwmiswire FR004-CG14-SN015-DR0-HB0-D3 DOWN_MISWIRE ExpNbr: FR004-CG11-SN012-DR0-HB0-D0 ActualNbr: FR004-CG12-SN013- DR0-HB0-D0 FR004-CG14-SN015-DR0-HB0-D2 DOWN_MISWIRE ExpNbr: FR004-CG12-SN013-DR0-HB0-D0 ActualNbr: FR004-CG11-SN012- DR0-HB0-D0 For a description of the ISNM hardware locations, see Command references, on page 48. In the previous example, by using the following method, you can see how the cables between two ports have been swapped: The first port, FR004-CG14-SN015-DR0-HB0-D3, should be cabled to FR004-CG11-SN012-DR0-HB0- D0. Searching for FR004-CG11-SN012-DR0-HB0-D0 in the rest of the output we find it as an ActualNbr to FR004-CG14-SN015-DR0-HB0-D2. The ExpNbr for FR004-CG14-SN015-DR0-HB0-D2 is FR004-CG12-SN013-DR0-HB0-D0, which you find is the ActualNbr to FR004-CG14-SN015-DR0-HB0-D3. You will note that the FR004-CG14-SN015-DR0-HB0 is common to both cables. Using the bulkhead diagram in ISNM location naming conventions, on page 90, you will note that Hub0 D3 (0-3) is to the right of Hub0 D2 (0-2). You can now surmise that the swap occurred between FR004-CG14-SN015-DR0-HB0-D3 and FR004- CG14-SN015-DR0-HB0-D Performance data gathering This topic provides information about performance data gathering. A separate CNM thread whose actions are guided by a set of configuration parameters manages p. 33 of 98

34 performance data gathering. The configuration parameters are read from the Cluster Database at CNM initialization and stored in CNM global memory. When the performance counter thread starts up, it first reads the configuration parameters from CNM global storage, and then queries the device database for a list of drawers that are in a state (Runtime) that allows the hardware performance counters to be read. If a new drawer changes to the runtime state, the performance counter thread is notified to add the new drawer to its list. The CNM performance data thread gathers performance counter data in accordance with several configurable values: the desired collection interval the period for which detailed data is kept in the Cluster Database the number of previous periods for which summary data should be kept how often to check for expired records in the Cluster Database how often to summarize (average) the performance data in the Cluster Database A collection interval of zero means that no collection is to be done. Minimums for collection intervals are enforced. Every X' minutes, the performance counter thread requests counters from LNMC by issuing the CNMLNMC synchronous command Get LNMC Performance Registers' to each LNMC, to retrieve all local ISR and HFI performance counters. As the reads come back to the EMS, the performance counter thread stores the information into the Cluster database. At regular intervals, the performance counter thread issues SQL statements to perform calculations on each record in the database generating statistics for the PerfSummary table. Once the save period has expired for a given record, the performance counter thread issues an SQL command to have that data removed from the database. CNM will be sensitive to the amount of message traffic that it is in the process of handling, and will throttle back on collecting performance counter data if CNM's message queues are too backed up. Available performance counters are broken into two categories: ISR and HFI. ISR performance counters: This topic provides information about ISR performance counters. There are several port types in the ISR and various counters are common between them, whereas some counters have meaning only for a few port types. The following is a table of four counter types cross-referenced to the types of ports that use them. An x' is placed in a corresponding table cell when a port type has a particular counter available. In addition, below the port type, there is a number in parentheses, which represents the number of ports of that type in each ISR. Table 13. ISR performance counters Counter D-Link (16) LR-Link (24) LL-Link (7) HFI-ISR (2) Cycles Blocked from Sending a FLIT X X X FLIT Sent X X X X FLITS Dropped X X X X Link Level Retries X X p. 34 of 98

35 Two primitive techniques for using the performance counters are documented here. More techniques that are sophisticated will typically require IBM support. To look for congestion consider looking for any ISR ports that seem to be very much different than others with respect to the number of Cycles Blocked versus the amount of data being sent through them. You must also consider that if a port has a disproportionate amount of data being sent through it, it may be blocked more than other ports. FLITs Sent + Link Level Retries + FLITs Dropped = Total FLITs handled FLIT = the lowest flow control unit for physical network data To look for errors, consider the following counters. There is an acceptable bit error rate that is taken into account in the network error reporting design. Therefore, one can expect to see a relatively low proportion of FLITs dropped or link level retries without any reporting of a problem. FLITs Sent FLITs Dropped Link Level Retries HFI performance counters: This topic provides information about HFI performance counters. The following table describes the various counters available for each of the two HFIs in each hub module Table 14. HFI performance counters Counter Aggregate Packet Sent Count Aggregate Packet Receive Aggregate Packet Dropped from Sending Count Aggregate Packet Dropped from Receive Count Aggregate Immediate Send Packet Sent Count Aggregate Send/Receive Packet Send Count Aggregate Full-RDMA Packet Sent Count Aggregate Half-RDMA Packet Sent Count Aggregate Small-RDMA Packet Sent Count Aggregate IP Packet Sent Count Aggregate CAU Packet Sent Count Aggregate GUPS Packet Sent Count Description Count of packets sent regardless of window Count of network packets received, regardless of window Count of packets from a Send FIFO, but dropped (not sent), regardless of window Count of packets from the ISR, but dropped (not receive), regardless of window Count of Immediate Send packets sent, regardless of window Count of Send/Receive packets sent, regardless of window Count of Full-RDMA packets sent, regardless of window Count of Half-RDMA packets sent, regardless of window Count of Small-RDMA packets sent, regardless of window Count of IP packets sent, regardless of window Count of CAU packets sent, regardless of window Count of GUPS packet sent, regardless of window p. 35 of 98

36 TEAL - Event management This section provides information about TEAL. The Toolkit for Event Analysis and Logging (TEAL) runs on the EMS. TEAL receives events and then analyzes them to determine if they are noteworthy. Noteworthy events or groups of events are transformed into alerts, which are stored in an Alert log, which can be monitored by a user. Some alerts are forwarded to other components in the system; for example, network hardware events that require service are forwarded to Service Focal Point. This is summarized in the following figure: At this time, TEAL supports receiving events and supporting plug-ins from xcat, PNSD, Loadleveler, ISNM and GPFS and from HMC serviceable events. A connector for each component or subsystem will take the event information in the format provided by the component and transform it so that it adheres to TEAL event table schema. Analysis is done using plug-ins for components. The analyzers will produce alerts that may also need to be processed. For example, an alert may occur more than once, but only the first occurrence (within a specified time period) needs to be communicated to the alert consumers until the alert is acknowledged by the user as having been resolved at which time communication may resume. Once it is determined that an alert needs to go to an alert consumer, alert listener plug-ins are used to define how to get the alert to the consumer, such as the system administrator, and what information needs to be provided. Available alert listener plug-ins are: A network event listener to create serviceable events from alerts. These will be available on a designated HMC. An listener that a user can configure to alerts to a given address. An RMC listener that can be used to trigger conditions and responses as configured by the user. File listener that a user can configure to send alerts to a file. Flow summary: This topic summarizes the flow of data through TEAL. The flow of data through TEAL is summarized in the following figure: p. 36 of 98

37 The flow through TEAL (as shown in the figure) begins when the monitor notifies the analyzers that a new event is available to be analyzed. The analyzers evaluate the events, possibly using rules defined for GEAR, and produce alerts. Analyzers may use a collector to get information outside of TEAL. Alerts are passed to the alert filters and recorded in the alert log. The alert filters to use for a particular alert listener are configured as part of the alert listener. Each configured alert filter is given the opportunity to filter (reject) the alert. It can query the alert log for information about other alerts to make its decision. If the alert is filtered, the alert entry in the alert log is updated to indicate that the alert was filtered. Alerts that are not filtered are passed on to their associated alert listener, which send them to its associated alert consumers. Once the listeners have all processed the alert the alert log will be updated to indicate that the alert has been reported. Once the alert is resolved the TEAL user to indicate this must update the alert log. This will allow filtering to handle subsequent occurrences of the alert. Connectors: Connectors transform incoming data into a common format so that TEAL can analyze the key attributes of events from many different components and subsystems. Event log: The event log contains all of the events from all of the components. Monitor: The monitor watches the event log and notifies the analyzers when there are new events to process. The monitor tracks the last event it retrieved from the event log and uses that as a basis to generate the list of events that have occurred since that point to pass to the analyzers when it is triggered. Historic analysis works in a similar manner except the monitor is not triggered by changes to the event log; rather it will query the event log for the historic events and notify the analyzers as if the p. 37 of 98

38 events being historically analyzed were being added to the event log. Analyzer: This topic provides information about the analyzer. The analyzer is summarized in the following figure: Analyzers sends events by the monitor in the following situations: Real-time analysis when a new event is added to the event table. TEAL is run and provides real-time monitoring of components under this standard method. Historic analysis as specified by the user. This is used for in-depth problem analysis. When historic analysis is done, the monitor will be run in a separate environment from the realtime analysis. This environment will have its own configuration that specifies the plug-ins to use. TEAL provides GEAR that can be supplied a component specific set of rules to be used to process events. GEAR supports operations such as scoping of events (to allow matching at different levels of location), suppression of events, and creating alerts. Rules also allow calling out to external routines, such as when data from other sources is needed during historic analysis. Analyzers produce alerts and suppress events. Alerts are passed to the alert filter and added to the alert log. Analyzers do not create events. Instead they create alerts which represent combinations of events. In order to support analysis of events that occur over time, TEAL provides support for an event pool that can be used by the analyzers. When using GEAR this pool support is used. Pool support is provided such that the pool starts with a default duration, which can be extended by the analyzers. Pools support using the time the event was logged or the time the event occurred when controlling the pool. Pools using the p. 38 of 98

39 time the event was logged can also use a timer to cause the pool to automatically close when the pool duration has been reached. Pools that don't use timers either close when an event is added which has a time past the pool's close time (start time plus duration) or it is explicitly closed (via a rule or a control message). Explicit closure will also occur during historical analysis when all selected events have been processed. Typically, when a pool duration is reached any events that were not suppressed will be changed into alerts Monitoring with TEAL This section provides information about monitoring the TEAL Table 15. Monitoring with TEAL Task Procedure When to use Getting the current list of alerts See Getting a list of alerts:, on page 39 Typical monitoring Closing alerts See Closing alerts:, on page 39 When an alert has been resolved Getting a list of events Getting a list of events:, on page 40 For more in-depth problem analysis when using alerts and their recommendations does not resolve a problem Pruning Logs Log pruning:, on page 40 Based on periodic filesystem checks indicating that more filesystem space is required Getting a list of alerts: This section provides information about list of alerts. To review the set of alerts that TEAL has logged/reported, the administrator might: dump all of the Alerts ~> tllsalert dump alerts within a time period (11AM 7/20/2011 to 11AM 7/21/2011) and output in the text format: ~> tllsalert -f text -q creation_time> :00:00 creation_time< :00:00 dump a subset of Alerts. In this case, all alerts that have been reported and have not been closed and output in csv format ~> tllsalert --query= state=open --format=csv dump a single Alert and all of its associated events and alerts. Associated events and alerts are all of the events and alerts that went into the creation of this alert; including those suppressed by this alert. ~> tllsalert query rec_id=123 --with-assoc Closing alerts: This section provides information about closing alerts. There are times when a user will fix an alert condition outside of the scope of normal service actions. In this case, the user will want to resolve an Alert condition manually. If the user already knows the id of the Alert that was acted on, they simply need to resolve it: p. 39 of 98

40 ~> tlchalert --rec_id=55 - state=closed If the user does not know the record id of the alert to resolve, they can get the list of alerts as documented above. Getting a list of events: This section provides information about list of events. The user may also review the set of events that have been logged by all of the connectors: dump all of the events ~> tllsevent dump a subset of events. In this case, all events that have been reported by a specific connector and output in csv format ~> tllsevent --query= src_comp=gpfs format=csv dump a single event ~> tllsevent query rec_id=123 format=text Log pruning: This section provides information about log pruning. If the event and alert logs get too large, the administrator may want to prune out old data. It is suggested that the user does the following steps: 1. Ensure that all alerts that have been completed are closed using the tlchalert command. 2. Save a list of the alerts and associated events using the tllsalert command specifying the events that are closed and any additional criteria used for the remove. 3. Issue the tlrmalert command to remove the old alerts 4. Issue the tlrmevent command to remove the old events The order of execution above is because alert relationships are pruned as part of alert pruning. Removing these relationships prior to pruning events reduces the number of events that cannot be pruned due to being in an alert relationship Monitoring of the cluster If problems are found while monitoring the cluster, the High perfomance clustering using the 9125-F2C Service Guide is the place to start to determine recovery and service actions xcat monitoring This section provides information about xcat monitoring. Third party monitoring software can be integrated into your xcat cluster. The idea is to use monitoring plug-in modules that act as bridges to connect xcat and the 3rd party software. Though you can write your own monitoring plug-in modules, over time, xcat will supply a list of built-in plug-in modules for the most common monitoring software. They are: p. 40 of 98

41 xcat (xcatmon.pm) (monitoring node statue using fping. released) SNMP (snmpmon.pm) (snmp monitoring. released) RMC (rmcmon.pm) (released) Ganglia (gangliamon.pm) (released) Nagios (nagiosmon.pm) Performance Co-pilot (pcpmon.pm) Pick one or more monitoring plug-ins to monitor the xcat cluster. For information on setting up xcat Monitoring, see Monitoring an xcat Cluster. ( %01) If problems are found monitoring the cluster with xcat, xcat's Cluster Recovery ( ) is a good place to look for possible resolutions. Another possibility is to use the High perfomance clustering using the 9125-F2C Service Guide ISNM monitoring ISNM is used to monitor the HFI network. Detailed information on using ISNM for monitoring is found in Network management overview, on page 29. If problems are found while monitoring the cluster, the High perfomance clustering using the 9125-F2C Service Guide is the place to start to determine recovery and service actions TEAL monitoring TEAL is used to monitor and analyze events from key components in the cluster. Detailed information on using TEAL for monitoring is found in Monitoring with TEAL, on page 39. If problems are found while monitoring the cluster, the High perfomance clustering using the 9125-F2C Service Guide is the place to start to determine recovery and service actions Other monitoring Other components in the cluster have their own service guides, which is a good place to start when they indicate problems. References to component specific documentation can be found in Cluster information resources, on page 19. If nothing is found in the component's documentation, try he High perfomance clustering using the 9125-F2C Service Guide Power 775 Availability Plus Management Power 775 Availability Plus (A+) management is largely about monitoring failures and then performing recovery and service actions. Another key aspect is managing A+ resources according to local policies. Beyond the recovery and service actions, tracking the use of A+ resources will be important. The following table can be used: p. 41 of 98

42 Resource hostname Location Spare Policy Date deployed (Frame/Slot; SuperNode/Drawer) (hot/cold/warm) The resource hostname is recorded. The location records the physical location of the resource. The specific spare policy for this resource is recorded. The data deployed indicates when it is no longer a spare. Before that date it is considered to be a spare. The details of A+ actions are in High perfomance clustering using the 9125-F2C Planning and Installation Guide. If a serviceable event is reported with a A+ resource in the FRU list, go to the High perfomance clustering using the 9125-F2C Planning and Installation Guide section on A+ recovery actions. A+ resources are identified in the physical/service location code in a FRU list with the -R identifier followed by a number. In the HMC managing serviceable events function, you must open up an event to query its FRU list. In TEAL, the FRU list is included in the raw_data field. The following are examples of how to interpret the FRU list: Using raw_data, the FRU List is presented like this: raw_data : {'FRU List': [['74Y0601', 'ACT04216I FRU', 'U78A B001-P1-R39', 'YH10HA0CN010', '', '2E00']]... Each FRU has several fields: 'Part number', 'FRU class', 'location code', 'serial number', 'ecid', 'ccin'. The location code is the important field for determining if an A+ resource is affected by the problem. They key is to look for - R followed by a number. In the above example, there is a -R39. In the following example from an event reported by a server against the processor subsystem, -R39 is an A+ resource contained within U78A B001-P1. -> /opt/teal/tllsalert -f text pg rec_id : 7760 alert_id : B112E504 creation_time : :27: severity : E urgency : N event_loc : U9125.F2C.028B506 event_loc_type : P fru_loc : None recommendation : reason : Processor subsystem (0x12) reported an error. src_name : SFPEventAnalyzer state : 1 raw_data : {'FRU List': [['74Y0601', 'ACT04216I FRU', 'U78A B001-P1-R39', 'YH10HA0CN010', '', '2E00']], 'SFP': 'c250hmc52.pok.ibm.com', 'Problem Number': 867} In the following example from an event reported by ISNM against a D-link, -R3 on both U78A CM002- P1-R3 and 52Y3020,FRU,U78A S000-P1-R3 are A+ resources. p. 42 of 98

43 -> /opt/teal/tllsalert -f text pg rec_id : 7769 alert_id : BD creation_time : :07: severity : E urgency : S event_loc : FR052-CG04-SN001-DR0-HB1-OM11-LD11 event_loc_type : H fru_loc : None recommendation : There is a problem with a D-Link. Record the alert ID and call IBM Service. Log on to the Management Server. To isolate to the proper FRU, run Link Diags and perform the actions that it recommends. If no action is recommended, because Diags cannot isolate to the proper FRU, replace the FRUs in the order listed. reason : D-link problem between frame FR052 cage CG04 (supernode SN001 drawer DR0) hub HB1 port LD11 and frame FR052 cage CG07 (supernode SN004 drawer DR0) hub HB1 port LD14 (D Link Port Dropped Flit Threshold Exceeded) src_name : CnmEventAnalyzer state : 1 raw_data : {"fru_list":"{ HFI_DDG,Isolation Procedure,,,, },{ HFI_CAB,Symbolic Procedure,U78A CM002-P1-T17-T5,,, },{ CBLCONT,Symbolic Procedure,U78A S000-P1-T15-T6,,, },{ 52Y3020,FRU,U78A CM002-P1- R3,YA193P601115,ABC123,TRMD },{ 52Y3020,FRU,U78A S000-P1- R3,YA193N076455,ABC123,TRMD }","nbr_loc":"fr052-cg07-sn004-dr0-hb1-om14- LD14","nbr_typ":"H","pwr_enc":"78AC- 100BC50052","eed_loc":"c250mgrs52:/var/opt/isnm/cnm/log","encl_mtms":"9125- F2C/028B536"} The following example has no A+ resources. The FRU locations have no -R in them. They are U78AC.100.BC50052-P1-C1, U78AD.001.TMPJ521-E2, U78AC.100.BC50052-P1-C1-T38, and 78AD.001.TMPJ521-P1-E2-T3. -> /opt/teal/tllsalert -f text pg rec_id : 7856 alert_id : 1402B038 creation_time : :56: severity : E urgency : N event_loc : U78AC.100.BC50052 event_loc_type : P fru_loc : None recommendation : reason : Power/Cooling subsystem & control (0x60) reported an error. src_name : SFPEventAnalyzer state : 1 raw_data : {'FRU List': [['IQYRISM', 'ACT04219I Isolate procedure', '', '', '', ''], ['45D9084', 'ACT04216I FRU', 'U78AC.100.BC50052-P1-C1', 'YH10SBC50071', '', '2C82'], ['45D9841', 'ACT04216I FRU', 'U78AD.001.TMPJ521-E2', '', '', ''], ['15R8592', 'ACT04216I FRU', 'U78AC.100.BC50052-P1-C1-T38', '', '', ''], ['CBLCONT', 'ACT04223I Symbolic procedure', 'U78AD.001.TMPJ521-P1-E2-T3', '', '', '']], 'SFP': 'c250hmc52.pok.ibm.com', 'Problem Number': 696} HFI network health checks In order to check the network health periodically, write a script that can check the following and run it as a cron p. 43 of 98

44 job. The details of the commands are in Command references, on page 48. Certain checks should be done more frequently than others. The breakpoint should be that checks for failures or errors should be performed more frequently than checks for configuration changes, which should not occur very often. Frequent checking: lsnwdownhw will check for hardware that is down. Bear in mind that certain hardware may be expected to be down for service actions, or because the A+ refresh threshold has not been reached. Less frequent checking: Results of lsnwtopo, lsnwtopo -C and lsnwtopo -A should all match. Where lsnwtopo without a parameter is the topology stored in the Cluster Database, and the -C parameter return the CNM copy of the topology, and the -A returns the copy from each FSP. This is most likely to change after repowering or servicing CEC drawers. Results of lsnwconfig should typically be the same from run to run. If you periodically change the collection and summary parameters for the performance counters, that should be taken into account when interpreting results. lsnwmiswire will display any miswires. After installation, the most likely time for a miswire is after a service action on a CEC drawer or a link. lsnwlinkinfo will display all of the links. This list of links should remain the same from run to run except for when there is a configuration change. Typically, lsnwdownhw should catch problems that are found in lsnwlinkinfo. lsnwcomponents will display the list of main cluster hardware components such as FSPs and BPAs. Typically, the results should be the same from run to run after installation is complete. If the CEC drawers or frames are added to or removed from the configuration, then the results will change. Other reasons for missing components are that they are powered off, have failed, or there is a problem communicating with them. It is a good idea to take a baseline and compare results to the baseline. Whenever there is a configuration change p. 44 of 98

45 2.3.9 Maintenance For the most up-to-date information on applying code maintenance refer to the appropriate service pack on the IBM HPC Clustering with Power 775 servers - Service Packs portion of the IBM High Performance Computing Clusters Service Packs website ( Service+Packs#IBMHighPerformanceComputingClustersServicePacks- IBMHPCClusteringwithPower775serversServicePacks) If your service pack is not listed, go to the most recent service pack and access the archived service packs, which should list your service pack Applying System Firmware Updates To apply system firmware updates use the xcat rinv command to get the current level and the rflash command to update the code level, then validate that the update was succesfful: # check current level rinv cec firm # flash new level rflash cec -p <directory> --activate disruptive # check for new level rinv cec firm # Check health of the CEC rpower cec state rvitals cec lcds -p indicates the directory in which the firmware code is available on the EMS. Details can also be found in the xcat Power 775 Hardware Management document in the Update the CEC firmware, and Validate CECs Can Power Up section ( title=xcat_system_p7_775_hardware_management#update_the_cec_firmware.2c_and_validate_cecs_can_ Power_Up) Applying power code updates To apply system firmware updates use the xcat rinv command to get the current level and the rflash command to update the code level. # check current level rinv frame firm # flash new level rflash frame -p <directory> --activate disruptive # check for new level rinv frame firm # Check health of the CEC rpower frame state rvitals frame lcds p. 45 of 98

46 -p indicates the directory in which the firmware code is available on the EMS. Details can also be found in the xcat Power 775 Hardware Management document in the Update the CEC firmware, and Validate CECs Can Power Up section ( title=xcat_system_p7_775_hardware_management#update_the_cec_firmware.2c_and_validate_cecs_can_ Power_Up) Updating the EMS For an EMS running AIX, see XCAT AIX Cluster Overview and Mgmt Node ( For an EMS running Linux, see Setting Up a Linux xcat Mgmt Node ( Updating Management Software For an xcat running on AIX, see XCAT AIX Cluster Overview and Mgmt Node ( For ISNM running on AIX, see IBM Clusters with HFI and 9125-F2C website as referenced in Cluster information resources, on page 19. For TEAL running on AIX, see IBM Clusters with HFI and 9125-F2C website as referenced in Cluster information resources, on page 19. For an xcat running Linux, see Setting Up a Linux xcat Mgmt Node ( For ISNM running on Linux see IBM Clusters with HFI and 9125-F2C website as referenced in Cluster information resources, on page 19. For TEAL running on Linux, see IBM Clusters with HFI and 9125-F2C website as referenced in Cluster information resources, on page Managing Multiple Operating System Images For certain deployments and phases of deployment it may be advantageous to keep multiple operating system images available for use. There may be various production images used depending on the use of the cluster and there may be test images created before deploying them into production. Typical images are: p. 46 of 98

47 Compute images used by compute nodes Service node images (diskful) Storage node images Utility node images Impact: When applying a new image to a node, the partition must be rebooted. The partition will be unavailable during this time. If there is a backup node and the rebooted node is partition, it will take over during this time frame. Perform the following procedure to apply a different operating system on a diskless node. 1. If necessary build the image and propagate it to the service node(s) which own the node(s) that will be rebooted with the new image. 2. Assign the image to the nodes that will receive it using the appropriate xcat commands. 3. Reboot the nodes to receive the new image using the xcat rpower command. 4. This procedure ends here. p. 47 of 98

48 3 Command references This appendix provides information about the network management and Operating system commands that are used in managing, monitoring and maintaining the HFI fabric. 3.1 Network management (ISNM) command reference This section provides information about ISNM command reference. Network management provides a command interface to support the tasks of integrated switch network configuration, verification, and status queries. The commands are run on the EMS. The lists of commands are categorized as follows: Installation and Configuration: Start and stop CNM (chnwm (AIX), service cnmd (Linux)) Set Network Topology Information (chnwtopo; lsnwtopo) Enable/Disable a Link (chnwfence) Set ISNM Configuration Parameters, including performance counter gathering (chnwconfig; lsnwconfig) Start/Stop OLCT mode (chnwolct) Diagnose an optical link (nwlinkdiag) Gather the OLCT link status information from LNMC (lsnwolctstat) Network Status: Display ISNM Configuration Parameters (lsnwconfig) Display ISR Link Information (lsnwlinkinfo, lsnwdownhw) Display Frame and Supernode Information (lsnwloc) Display Faulty Links List all of the known faulty network links (lsnwdownhw) Display Faulty HFIs List all of the known faulty HFIs (lsnwdownhw) Display Faulty ISRs List all of the known faulty ISRs (lsnwdownhw) Display Expected Neighbors (lsnwexpnbrs) Display Cluster Topology (lsnwtopo) Display Global Counter Information (lsnwgc) Display Miswired Links (lsnwmiswire) Display Cluster Components Displays network hardware that is reachable on the control network. (lsnwcomponents) Network Monitor: Continuously monitor Network performance counters (isrmon) Report Network performance counters for a given period (isrperf) p. 48 of 98

49 Debug: Collect ISNM logs (cnm.snap, cnm.linux.snap) Dump Hardware Route Table Data Dump Hardware Link States Restart LNMC Route table Analysis NOTE: Unless otherwise stated, ISNM commands are on the EMS in /usr/bin chnwconfig This command is used to set or change the Network Management Configuration parameters. SYNTAX chnwconfig -p <parameter> -v <value> [ -p <parameter> -v <value>... ] [ -h --help ] DESCRIPTION The chnwconfig command allows modifications to the network management configuration data in both Cluster database and in Central Network Manager (CNM). All the configuration parameters can be changed at runtime, without restarting Central Network Manager(CNM) software. But, to change the hardware indirect routing scheme, it is required that no jobs be running that use the Integrated Switch Router network. It is possible that the drawer will require reboot in order to make this change. The user can use this command only when the 'Central Network Manager' software is enabled and running in the system. Multiple parameter and value pairs can be given to set values for more than one parameter. The command line options can be given with either '-' or '--'. OPTIONS -p Specifies the parameter for which the value needs to be set. Valid parameters and their corresponding values are: Parameter Default Value Minimum Value Maximum Value EXPIRED_PERIOD_TIMER INT_MAX MONITOR_RMC NUM_PERF_DATA_SAVE_PERIOD 1 0 INT_MAX PERF_DATA_INTERVAL INT_MAX PERF_DATA_SAVE_PERIOD INT_MAX RECOV_POOL_INTERVAL INT_MAX SUMMARY_DATA_TIMER INT_MAX -v Specifies the value to be set for the corresponding parameter. p. 49 of 98

50 -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1. To change the Performance Data Interval to 70 seconds and the Performance Data Collection Save Period to 2 hours, type: chnwconfig -p PERF_DATA_INTERVAL -v 70 p PERF_DATA_SAVE_PERIOD v 2 2. To change the RMC Monitoring Support to 1 (ON) type: chnwconfig -p MONITOR_RMC chnwfence This command is used to remove a link or set of links from the Integrated Switch Network, or restores links that have previously been removed. Single links, all links between two supernodes, links between two drawers, or links between a drawer and target supernode may be fenced or unfenced. Fenced links are identified by the link status DOWN_DISABLED_USR or DOWN_DISABLED_ERR. NAME chnwfence - Fence or unfence optical links PURPOSE Removes a link or set of links from the Integrated Switch Network, or restores links that have previously been removed. Single links, all links between two supernodes, links between two drawers, or links between a drawer and target supernode may be fenced or unfenced. Fenced links are identified by the link status DOWN_DISABLED_USR or DOWN_DIS- ABLED_ERR. SYNTAX chnwfence { -sf <source-frame> [-sc <source-cage>] -ts <target-supernode> [-td <target-drawer>] [-u] } [ -h --help ] chnwfence { -ss <source-supernode> [-td <target-drawer>] -tf <targetframe> [-tc <target-cage>] [-u] } [ -h --help ] chnwfence { -ss <source-supernode> [-sd <source-drawer>] [-u] } [ -h --help ] chnwfence { -sf <source-frame> [-sc <source-cage>] [-u] } [ -h --help ] p. 50 of 98

51 chnwfence { -sf <source-frame> [-sc <source-cage>] -tf <target-frame> [-tc <target-cage>] [-u] } [ -h --help ] chnwfence { -ss <source-supernode> [-sd <source-drawer>] -ts <targetsupernode> [-td <target-drawer>] [-u] } [ -h --help ] chnwfence { -sf <source-frame> -sc <source-cage> -sm <source-hub-module> {-lr <LR-link-id> -ld <D-link-id> } [-u] } [ -h --help ] chnwfence { -ss <source-supernode> -sd <source-drawer> -sm <sourcehub-module> { -lr <LR-link-id> -ld <D-link-id> } [-u] } [ -h --help ] DESCRIPTION The chnwfence command removes a link or set of links from service in the Integrated Switch Network. A link that is removed from service using the chnwfence command can only be restored to service using the chnwfence unfence option. If a server containing fenced links is power-cycled, the link remains fenced even after the server is powered back up. The chnwfence command fences at only the source side of the link. That is, it does not fence the link at both ends of the link. Fencing one end of a link removes the entire link from service. This command can only be executed against servers that are in the in the LNMC Runtime state. The lsnwlinkinfo command will show a status of DOWN_DISABLED_USR for a link that is fenced using the chnwfence command. The command line options can be given with either â-â or â--â. OPTIONS -ss Specifies the source supernode at which to fence or unfence the link -sd Specifies the drawer or server of the source supernode -sf Specifies the source frame at which to fence or unfence the link -sc Specifies the cage id of the source frame -sm Specifies the hub module of the source supernode -lr Specifies the optical LR link on the hub module to fence or unfence. Valid values are ld Specifies the optical D link on the hub module to fence or unfence. Valid values are ts Specifies the target neighbor supernode, when fencing or unfencing links between two drawers p. 51 of 98

52 -td Specifies the target neighbor drawer, when fencing or unfencing links between two drawers -tf Specifies the target neighbor frame, when fencing or unfencing links between two drawers -tc Specifies the target neighbor cage, when fencing or unfencing links between two drawers -u Specifies that the link(s) are to be unfenced -h, --help Displays the commandâs usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. 1 Indicates the CNM software is currently not running or disabled. 2 Indicates if there were any errors. EXAMPLES 1 To fence all optical D links between supernode 3 and supernode 55, type: chnwfence -ss 3 -ts 55 2 To fence all optical D links between supernode 3 drawer 0 and supernode 4 drawer 0, type: chnwfence -ss 3 -sd 0 -ts 4 -td 0 3 To fence D link 4 of supernode 100 drawer 3 hub 7, type: chnwfence -ss 100 -sd 3 -sm 7 -ld 4 4 To unfence LR link 22 of supernode 32 drawer 2 hub 5, type: chnwfence -ss 32 -sd 2 -sm 5 -lr 22 -u chnwm If the EMS is running AIX, use this command to activate and deactivate network management software. If the EMS is running Linux, use the service command, which is documented at the end of this command. SYNTAX chnwm { -a -d -q -h --help } DESCRIPTION The chnwm command activates or deactivates 'Central Network Manager' (CNM), the Integrated p. 52 of 98

53 Switch Network Management software. The command line options can be given with either '-' or '--'. OPTIONS -a Activates Central Network Manager (CNM) software. -d Deactivates Central Network Management (CNM) software. -q Used to query the activation status of the CNM software. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To activate the Integrated Switch Network Management software, type: chnwm -a 2 To deactivate the Integrated Switch Network Management software, type: chnwm d If the EMS is running Linux, use: service cnmd start service cnmd stop service cnmd restart chnwolct This command is used to put optical modules into or take optical modules out of Optical Link Connectivity Test (OLCT) mode PURPOSE Puts an optical module or a set of optical modules from the Integrated Switch Network into OLCT mode, or takes optical modules out of OLCT mode. SYNTAX chnwolct { -sf <source-frame> [-sc <source-cage>] ts <target-supernode> [-td <target-drawer>] [-r] } [ -h --help ] chnwolct { -ss <source-supernode> [-td <target-drawer>] tf <target-frame> [-tc <target-cage>] [-r] } [ -h --help ] chnwolct { -ss <source-supernode> [-sd <source-drawer>] [-r] } [ -h --help ] chnwolct { -sf <source-frame> [-sc <source-cage>] [-r] } [ -h --help ] p. 53 of 98

54 chnwolct { -sf <source-frame> [-sc <source-cage>] tf <target-frame> [-tc <target-cage>] [-r] } [ -h --help ] chnwolct { -ss <source-supernode> [-sd <source-drawer>] ts <target-supernode> [-td <target-drawer>] [- r] } [ -h --help ] chnwolct { -sf <source-frame> -sc <source-cage> -sm <source-hub-module> -sl <source-link_id> [-r] } [ -h -- help ] chnwolct { -ss <source-supernode> -sd <source-drawer> -sm <source-hub-module> -sl <source-link_id> [-r] } [ -h --help ] The options can be given either with '-' or '--'. DESCRIPTION The chnwolct command puts an optical module or a set of optical modules from the Integrated Switch Network into OLCT mode, or takes optical modules out of OLCT mode. Both "sides" of a link must be put into OLCT mode in order for cable validation to take place. That is, for a D-link, the optical modules at each end of the D-link cable must be put into OLCT mode in order for cabling validation to take place. For LR-link cabling validation, only full-drawer validation is supported. That is, two entire connected drawers of a supernode must be put into or taken out of OLCT mode. If a module is put into OLCT mode, it must be taken out of OLCT mode in order for the link to be used for routing. This command can only be executed against servers that are in the "LNMC Runtime" or "LNMC Low Power" states. OPTIONS -ss Specifies the source supernode at which to put optical modules into or take optical modules out of OLCT mode -sd Specifies the drawer or server of the source supernode -sf Specifies the source frame at which to put optical modules into or take optical modules out of OLCT mode -sc Specifies the cage id of the source frame -sm Specifies the hub module of the source supernode -sl Specifies the optical link id associated with the optical module to put into or take out of OLCT mode. Valid values are 32-47; these map to D0 - D15. This option is not supported for LR modules. This option cannot be used if the -td option is used -ts Specifies the target neighbor supernode, when putting optical modules into or taking them out of OLCT mode -td Specifies the target neighbor drawer, when putting optical modules into or taking them out of OLCT mode. This option cannot be used with the -sl option p. 54 of 98

55 -tf Specifies the target neighbor frame, when putting optical modules into or taking them out of OLCT mode -tc Specifies the target neighbor cage, when putting optical modules into or taking them out of OLCT mode -r Specifies that the link(s) are to be taken out of OLCT mode -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To put all optical D links between supernode 3 and supernode 55 into OLCT mode, type: chnwolct -ss 3 -ts 55 2 To put all optical D links between supernode 3 drawer 0 and supernode 4 drawer 0 into OLCT mode, type: chnwolct -ss 3 -sd 0 -ts 4 -td 0 3 To put D link 4 of supernode 100 drawer 3 hub 7 into OLCT mode, type: chnwolct -ss 100 -sd 3 -sm 7 -sl 36 4 To put all optical links of drawer 2 in supernode 12 into OLCT mode, type: chnwolct -ss 12 -sd 2 5 To put all LR links between drawers 2 and 3 of supernode 6 into OLCT mode, type: chnwolct -ss 6 -sd 2 -ts 6 -td 3 6 To remove all optical links of drawer 2 in supernode 12 from OLCT mode, type: chnwolct -ss 12 -sd 2 r chnwsvrconfig This command is used to set the network configuration data in a server. PURPOSE To set up a server with configuration data required for configuring the HFI and ISR or to erase the existing configuration data on a server. SYNTAX chnwsvrconfig [ { { { -f <frame> --frame <frame> } [ -c <cage> --cage <cage> ] } { -A --all } } [ -e --erase ] ] [ -h --help ] p. 55 of 98

56 DESCRIPTION The chnwsvrconfig command sets up one or more servers with the configuration data as defined in the Cluster Database. Certain data items - supernode id, drawer id and cluster topology - must be stored at each server for configuring the HFI and ISR hardware and to allow the server to join the ISR network. The chnwsvrconfig command extracts the specified configuration data items for a given server from the cluster database, and sends the data to the server. Using the chnwsvrconfig -e option erases existing configuration information from a server or from all servers. This command can only be executed against servers that are in the STANDBY state. The procedure for using this command is: 1. Deactivate the CNM software (See chnwm -d (AIX) or service cnmd stop (Linux)). 2. Update the topology/supernode/drawer information in the cluster database to reflect the desired changes. 3. Activate the CNM software (See chnwm -a (AIX) or service cnmd start (Linux)). 4. Issue the chnwsvrconfig command to update the server information as per the cluster database. The command line options can be given with either '-' or '--'. OPTIONS -f, --frame Specifies the frame number to access the server. -c, --cage Specifies the cage number in the frame. -A, --all Sets the topology information for all the servers currently managed by CNM. -e, --erase To erase the topology information for a specified frame/frame-cage/all. by CNM. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To set the supernode, drawer and topology for all the servers in frame 12 as defined in cluster database type : chnwsvrconfig -f 12 2 To set the supernode, drawer and topology for the server in frame 12 and cage 4 as defined in cluster database type : chnwsvrconfig -f 12 -c 4 3 To erase the supernode, drawer and topology for all the servers as defined in cluster database type : chnwsvrconfig -A -e p. 56 of 98

57 RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To set the supernode, drawer and topology for all the servers in frame 12 as defined in cluster database type : chnwsvrconfig -f 12 2 To set the supernode, drawer and topology for the server in frame 12 and cage 4 as defined in the cluster database type : chnwsvrconfig -f 12 -c 4 3 To erase the supernode, drawer and topology for the server in frame 12 and cage 4 as defined in the cluster database type : chnwsvrconfig -f 12 -c 4 -e 4 To erase the supernode, drawer and topology for all the servers as defined in cluster database type : chnwsvrconfig -A -e cnm.snap, cnm.snap.linux Use this command to collect data to send back to IBM support and engineering. The output file is /var/opt/isnm/cnm/log/[ems hostname][timestamp].snap.tar.gz. Timestamp format is [year]-[month]-[day].[hour]:[minute]:[second] isrmon NAME isrmon - Monitor the ISR Network Performance Counters PURPOSE Monitors the ISR Network Performance Counters continuously and displays the network activity based on different input options provided. SYNTAX isrmon [-e -i] [-n] [-s <supernode> [ -d <drawer> [-m <hubmodule> ]]] { { ll { send block drop } } { { lr d isr } { send block drop retry } } p. 57 of 98

58 { hfi { send drop packetsend senddrop receive receivedrop immediate sendreceive fullrdma halfrdma smallrdma ip cau gups } } } [ -h ] DESCRIPTION The isrmon command provides performance counters for individual links or HFI adapters of ISR network. It provides flits sent, cycles blocked, flits dropped and retries for links and flits sent, flits dropped, packets sent, packets received, packets dropped from sending, packets dropped from receiving, immediate send packets sent, send or receive packets, Full-RDMA packets sent, Half-RDMA packets sent, Small-RDMA packets sent, IP packets sent, CAU packets sent, and GUPS packets sent for HFI adapters. The counters are provided for the time the command running or for the latest data collection period or from the time the Hub Chips started or reset. The counters for the time the command running are provided by default. The command can be used to provide performance counters of links or HFI adapters of entire cluster, or a set of Supernodes, or particular drawers of a set of Supernodes or particular Hub Chips of a set of Supernodes. All or some relevant performance counters of links or HFI adapters are displayed dynamically in decreasing order of a given performance counter. For example, if isrmon is called with options "lr send" the LRs are displayed with all the performance in decreasing order of flits sent. The HFI adapter counters are divided into different groups and each group is displayed separately. The HFI counters are divided into the following groups: 1. HFI Flits Flits Sent, Flits Dropped 2. HFI Packets Packets sent, Packets received, Packets dropped from sending, Packets dropped from receiving and Packets send/receive 3. HFI RDMA Full-RDMA packets sent, Half-RDMA packets sent, Small-RDMA packets sent p. 58 of 98

59 4. HFI Protocols IP packets sent, Immediate send packets sent, CAU packets sent, GUPS packets sent The individual HFI group counters can be displayed by specifying an option corresponding to any counter of that group. For example; by specifying "hfi send" or "hfi drop" the counters of HFI Flits group is displayed. Single instance of isrmon command can display performance counters of links and HFI adapters. The following hotkeys can be used to switch between displays of links and HFI adapter groups. l The LL links counters are displayed. r The LR links counters are displayed. d The D Links counters are displayed. i The ISR links counters are displayed. f The HFI Flits counters ( Flits sent, Flits dropped) are displayed. p The HFI Packets counters ( Packets Sent, Packets Received, Packets dropped from sending, Packets dropped from receiving and Packets send/receive) are displayed. m The HFI RDMA counters (Full-RDMA packets sent, Half-RDMA packets sent, Small-RDMA packets sent) are displayed. t The HFI Protocols counters (IP packets sent, Immediate send packets, CAU packets sent and GUPS packets ) are displayed. While switching between different displays of Links and HFI adapter groups, the Links or HFI adapters are sorted in decreasing order of a default performance counter. The default performance counter for LL, p. 59 of 98

60 LR, D, ISR and HFI Flits group is the Flits sent. Similarly for HFI Packets group the Packets sent, for HFI RDMA group the Full-RDMA packets sent and for HFI Protocols group the IP Packets sent are default counters. The performance counters of links or HFI adapter groups can be sorted by any performance counter of the corresponding link or HFI adapter groups. The column number of the corresponding performance counter can be used as hotkey to display the performance counters of Links or HFI adapter groups in decreasing order of that performance counter. OPTIONS -e The command lists the performance counters from the time (epoch) the counters started or from the time the counters reset. Basically with this option the raw performance counters of the links or adapters are listed. -i The command lists the performance counters for latest iteration always. If either -e or -i flags are missed, then command lists the performance counters from the time the isrmon command is started. The performance counters from the start of the command is listed by default. -n The flits blocked, dropped, retried performance counters of LL, LR, and D Links and the flits dropped from sending, the packets dropped from sending, packets dropped from receiving of HFI adapters are listed with normalization. The blocked, dropped, retried performance counters are listed in percentage with respect to sum of flits sent and flits dropped. Similarly packets dropped from sending are listed in percent with respect to sum of packets sent and packets dropped from sending. The packets dropped from receiving are listed in percentage with respect to sum of packets received and packets dropped from receiving. -s <supernodes> A subset of supernodes can be specified to list performance counters of Links or HFIs of the specified supernodes. The supernodes can be specified by comma separated list or range or both, where range is integers separated by - (hyphen). -d <drawers> A set of drawers can be specified to list performance counters of links or HFIs of the drawers of the specified super nodes. The drawers can be specified by comma separated list or range or both, where range is integers separated by - (hyphen). If -d p. 60 of 98

61 option along with set of drawers is specified then -s option along with a set of super nodes must be specified. -m <hubs> A set of hub modules can be specified to list performance counters of Links or HFIs of the hubs of the specified drawers and Supernodes. The hub modules can be specified by comma separated list or range or both, where range is integers separated by - (hyphen). ). If -m option along with set of hub modules is specified then -d option along with set of drawers and -s option along with a set of super nodes must be specified. ll lr d The command displays the performance counters of LL links only. The command displays the performance counters of LR links only. The command displays the performance counters of D links only. isr The command accumulates the respective performance counters of LL, LR and D links of each ISR and the accumulated performance counters are displayed for each ISR. hfi The command displays the performance counters of HFI adapters only. One of the ll, lr, d, hfi, isr argument is mandatory. These command line arguments are not case sensitive. send The command displays the performance counters of Links, HFIs or ISRs in decreasing order of their flits sent. block The command displays the performance counters of links, HFIs or ISRs in the decreasing order of their cycles blocked before sending the flits. drop The command displays the performance counters of Links, HFIs or ISRs in the decreasing order of their number of flits dropped. retry The command displays the performance counters of inks or ISRs in the decreasing order of their number of retries attempted before sending the flits. Packetsend The command displays the performance counters of HFI adapters in the decreasing order of their number of packets sent. senddrop p. 61 of 98

62 The command displays the performance counters of HFI adapters in the decreasing order of their number packets dropped from sending. receive The command displays the performance counters of HFI adapters in the decreasing order of their number of packets received. receivedrop The command displays the performance counters of HFI adapters in the decreasing order of their number packets dropped from receiving. immediate The command displays the performance counters of HFI adapters in the decreasing order of their number of immediate packets sent. sendreceive The command displays the performance counters of HFI adapters in the decreasing order of their number of packets send and receive. fullrdma The command displays the performance counters of HFI adapters in the decreasing order of their number of fullrdma packets sent. halfrdma The command displays the performance counters of HFI adapters in the decreasing order of their number of halfrdma packets sent. smallrdma The command displays the performance counters of HFI adapters in the decreasing order of their number of smallrdma packets sent. ip The command displays the performance counters of HFI adapters in the decreasing order of their number of IP packets sent. cau The command displays the performance counters of HFI adapters in the decreasing order of their number of CAU packets sent. gups The command displays the performance counters of HFI adapters in the decreasing order of their number of GUPs packets sent. One of the send, block, drop, retry, packetsend, senddrop, receive, p. 62 of 98

63 receivedrop, immediate, sendreceive, fullrdma, halfrdma, smallrdma, ip, cau,and gups argument is mandatory. The send, block, and drop are applicable for LL Links. The send, block, drop and retry are applicable for LR, D links and ISR only. The options send, drop, packetsend, senddrop, receive, receivedrop, immediate, fullrdma, halfrdma, smallrdma, ip, cau,and gups are applicable to HFI adapters only. The command line arguments send, block, drop, retry, packetsend, senddrop, receive, receivedrop, immediate, sendreceive, fullrdma, halfrdma, smallrdma, ip, cau,and gups are not case sensitive. -h Displays the command usage statement to standard output. EXAMPLES 1 To display the raw (counters from epoch) performance counters of LL Links sorted by "Flits Sent" enter: isrmon -e LL send 2 To display the raw (counters from epoch) performance counters of LR Links sorted by "Flits Drop" enter: isrmon -e LR drop 3 To display the raw (counters from epoch) performance counters of D Links sorted by "Flits Sent" enter: isrmon -e D send 4 To display the raw (counters from epoch) performance counters of HFI Packets group sorted by "Packets Sent" enter: isrmon -e hfi packetsend 5 To display the latest performance counters of HFI Packets group sorted by "Packets Sent" enter: isrmon hfi packetsend isrperf NAME isrperf - List Performance counters of ISR Network Links and HFI Adapters. PURPOSE Displays Performance counters of ISR Links LL, LR, D and ISR HFI adapters present in the ISR network. SYNTAX isrperf -t <start-time> [-e <end-time>] [-n] [-s <supernode> [-d <drawer> [-m <hubmodule>]]] p. 63 of 98

64 { { LL [{ send block drop } <threshold>] } { { LR D ISR } [{ send block drop retry } <threshold>] } { HFI [ {send drop packetsend senddrop receive receivedrop immediate sendreceive fullrdma halfrdma smallrdma ip cau gups } <threshold> ] } } [ -h ] DESCRIPTION The isrperf command displays the historical ISR Network performance counters for a given time period for ISR Links ( LL, LR, and D) and for ISR HFI adapters OPTIONS -s <supernode> A subset of supernodes can be specified to list performance counters of Links or HFIs of the specified supernodes. The supernodes can be specified by comma separated list or range or both, where range is integers separated by - (hyphen). -d <drawer> A set of drawers can be specified to list performance counters of links or HFIs of the drawers of the specified super nodes. The drawers can be specified by comma separated list or range or both, where range is integers separated by - (hyphen). If -d option along with set of drawers is specified then -s option along with a set of super nodes must be specified. -m <hub-module> A set of hub modules can be specified to list performance counters of Links or HFIs of the hubs of the specified drawers and Supernodes. The hubmodules can be specified by comma separated list or range or both, where range is integers separated by - (hyphen). ). If -m option along with set of hub modules is specified then -d option along with set of drawers and -s option along with a set of super nodes must be specified. -t <start-time> Specifies the starting time either in the format [mm/dd[/[cc]yy]] HH:MM or in the format ccyy-mm-dd HH:MM. The hours must be specified using a 24-hour clock. The starting time p. 64 of 98

65 is mandatory. -e <end-time> Specifies the ending time either in the format [mm/dd[/[cc]yy]] HH:MM or in the format ccyy-mm-dd HH:MM. The hours must be specified using a 24-hour clock. The end time is optional. If not specified, the recent time at which the counters are collected is considered. -n The flits blocked, dropped, retried performance counters of LL, LR, and D Links and the flits dropped from sending, the packets dropped from sending, packets dropped from receiving of HFI adapters are listed with normalization. The blocked, dropped, retried performance counters are listed in percentage with respect to sum of flits sent and flits dropped. Similarly packets dropped from sending are listed in percent with respect to sum of packets sent and packets dropped from sending. The packets dropped from receiving are listed in percentage with respect to sum of packets received and packets dropped from receiving. ll lr d To display performance counters of LL links To display performance counters of LR links To display performance counters of D links isr The command accumulates the respective performance counters of LL, LR and D links of each ISR and the accumulated performance counters are displayed for each ISR. hfi To display performance counters of HFI adapters One of the ll, lr, d, hfi, isr argument is mandatory. These command line arguments are not case sensitive. send To display the performance counters of links whose flits sent exceeds the given threshold value. block To display the performance counters of links whose cycles blocked exceeds the given threshold value. drop To display the performance counters of links whose flits dropped exceeds the given threshold value. retry To display the performance counters of links whose link level retries exceeds the given threshold value. p. 65 of 98

66 packetsend To display the performance counters of HFI links whose packet sent exceeds the given threshold value. senddrop To display the performance counters of HFI links whose packets dropped exceeds the given threshold value. receive To display the performance counters of HFI links whose packet received exceeds the given threshold value. receivedrop To display the performance counters of HFI links whose packet receive dropped exceeds the given threshold value. immediate To display the performance counters of HFI links whose immediate send exceeds the given threshold value. sendreceive To display the performance counters of HFI links whose send receive packets exceeds the given threshold value. fullrdma To display the performance counters of HFI links whose full RDMA packets sent exceeds the given threshold value. halfrdma To display the performance counters of HFI links whose half RDMA packets sent exceeds the given threshold value. smallrdma To display the performance counters of HFI links whose small RDMA packets sent exceeds the given threshold value. ip To display the performance counters of HFI links whose ip packets sent exceeds the given threshold value. cau To display the performance counters of HFI links whose cau packets sent exceeds the given threshold value. gups To display the performance counters of HFI links whose gups packets sent exceeds the given threshold value. -h Displays the command usage statement to standard output. p. 66 of 98

67 RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To display performance counters of LL Links for Supernode 0, Drawer 1 and hub 1 for the time period between 2012/09/19 10:00 to 2012/09/20 14:00 type: isrperf -s 0 -d 1 -m 1 -t :00 -e :00 LL 2 To display performance counters of HFI Links for Supernode 0, Drawer 1 and hub 1 for the time period from 2012/09/19 10:00 onwards type: isrperf -s 0 -d 1 -m 1 -t :00 HFI lsnwcomponents This command is used to display the cluster hardware components that are visible to the Integrated Switch Network Manager.. Bulk Power Assembly (BPA) units and server Flexible Service Processor (FSP) units that are reachable on the service network are displayed. SYNTAX lsnwcomponents [ -B --BPA -F --FSP ] [ -p --page ] [ -h --help ] DESCRIPTION The lsnwcomponents command displays the cluster components i.e, network hardware that is reachable on the control network. It lists the Bulk Power Assembly (BPA) units and server Flexible Service Processor (FSP) units that are visible to the Integrated Switch Network Manager. The user can use this command only when the Central Network Manager (CNM) software is enabled and running in the system. The command line options can be given with either '-' or '--'. OPTIONS -B, --BPA To display only the Bulk Power Assembly (BPA) units in the cluster. -F, --FSP To display only the server Flexible Service Processor (FSP) units in the cluster. -p, --page Displays the output one screen at a time. Each screen is followed by a prompt. If you press the Enter key, another page is displayed. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To display the cluster components, type: lsnwcomponents Sample output of lsnwcomponents' command : $ lsnwcomponents p. 67 of 98

68 BPA Backup ip= MTMS=78AC-100*211S000 FR008 BPA Backup ip= MTMS=78AC-100*211S000 FR008 FSP Primary ip= MTMS=9125-F2C*0286C36 FR008-CG03-SN000-DR0 connection DOWN FSP Primary ip= MTMS=9125-F2C*0286C36 FR008-CG03-SN000-DR0 connection DOWN FSP Primary ip= MTMS=9125-F2C*0286C46 FR008-CG04-SN000-DR1 FSP Backup ip= MTMS=9125-F2C*0286C46 FSP Primary ip= MTMS=9125-F2C*0286C56 FR008-CG05-SN000-DR2 FSP Backup ip= MTMS=9125-F2C*0286C56 FSP Primary ip= MTMS=9125-F2C*0286C86 FR008-CG06-SN000-DR3 FSP Backup ip= MTMS=9125-F2C*0286C86 FSP Primary ip= MTMS=9125-F2C*0286C76 FR008-CG07-SN001-DR0 FSP Backup ip= MTMS=9125-F2C*0286C76 FSP Primary ip= MTMS=9125-F2C*0286C66 FR008-CG08-SN001-DR1 FSP Backup ip= MTMS=9125-F2C*0286C66 FSP Primary ip= MTMS=9125-F2C*0286C96 FR008-CG09-SN001-DR2 FSP Backup ip= MTMS=9125-F2C*0286C96 FSP Primary ip= MTMS=9125-F2C*0286CA6 FR008-CG10-SN001-DR3 FSP Backup ip= MTMS=9125-F2C*0286CA6 FSP Primary ip= MTMS=9125-F2C*0286CB6 FR008-CG11-SN002-DR0 FSP Backup ip= MTMS=9125-F2C*0286CB6 FSP Primary ip= MTMS=9125-F2C*0286CE6 FR008-CG12-SN002-DR1 FSP Backup ip= MTMS=9125-F2C*0286CE6 FSP Backup ip= MTMS=9125-F2C*0286CD6 FSP Primary ip= MTMS=9125-F2C*0286CD6 FR008-CG13-SN002-DR2 connection DOWN FSP Primary ip= MTMS=9125-F2C*0286CC6 FR008-CG14-SN002-DR3 connection DOWN FSP Backup ip= MTMS=9125-F2C*0286CC6 : lsnwconfig This command is used to list the Integrated Switch Network Management configuration parameters. SYNTAX lsnwconfig [ -C ] [-h --help ] DESCRIPTION The lsnwconfig command displays the active network configuration parameters defined in the system. To display the active network management configuration parameter used by the Central Network Manager (CNM) software, the 'Central Network Manager' software must be enabled and running in the system. The command line options can be given with either '-' or '--'. OPTIONS -C To get the configuration parameters specified in CNM database. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. p. 68 of 98

69 EXAMPLES 1 To display the active network configuration parameters defined in the system, type: lsnwconfig 2 To display the active network configuration parameters in use by CNM, type: lsnwconfig -C Sample output of lsnwconfig' command : $ lsnwconfig ISNM Configuration parameter values from Cluster Database Performance Data Interval: 300 seconds Performance Data collection Save Period: 48 hours No.of Previous Performance Summary Data: 1 Hardware Indirect Routing Scheme: Round-Robin (0) RMC Monitoring Support: ON (1) CNM Expired Records Timer Check: 3600 seconds CNM Summary Data Timer: seconds CNM Recovery Consolidation timer: 300 seconds lsnwdownhw This command is used to display faulty network hardware: Links, HFIs, and ISRs. SYNTAX lsnwdownhw [ -H --HFI -I --ISR { -L --LINK [ -a ] } -a [ {-f <frame> --frame <frame> } [ -c <cage> --cage <cage> ] { -s <supernode> --supernode <supernode> } [ -d <drawer> --drawer <drawer> ] ] ] [ -p --page ] [ -h --help ] DESCRIPTION The lsnwdownhw command displays faulty Links, HFIs, and ISRs. If no options are specified then it displays faulty links, HFIs and ISRs. By default, only hardware whose status indicates a faulty state is displayed. That is, links having status such as DOWN_NBR_POWEREDOFF are not displayed by default. To list the hardware that is in any state other than UP_OPERATIONAL, use the -a option. The user can use this command only when the Central Network Manager (CNM) software is enabled and running in the system. To determine the valid range of values for frame, cage, supernode and drawer for the cluster, use the ISNM command 'lsnwloc'. The command line options can be given with either '-' or '--'. OPTIONS -H, --HFI Displays all faulty HFIs. -I, --ISR Displays all faulty ISRs. -L, --LINK Displays all faulty Links. -a, --a Displays hardware that is not OPERATIONAL, that is, this option displays hardware that is down for normal reasons such as a P775 that is powered down. -f, --frame Specifies the frame number in the cluster. -c, --cage Specifies the cage number in the frame. p. 69 of 98

70 -s, --supernode Specifies the supernode number in the cluster. A valid range for supernode number is 0-511; however, for a given cluster a subset of these supernode ids may be in use. -d, --drawer Specifies the drawer number in the supernode. A valid range for drawer within a supernode is p, --page Displays the output one screen at a time. Each screen is followed by a prompt. If you press the Enter key, another page is displayed. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To display all faulty Links in supernode 1, drawer 2, type: lsnwdownhw -L -s 1 -d 2 2 To display all faulty HFIs, type: lsnwdownhw -H or lsnwdownhw --HFI 3 To display all faulty Links, HFIs and ISRs, type: lsnwdownhw :. 4 To display all faulty links, including links that are offline for normal causes, type: lsnwdownhw L -a lsnwexpnbrs This command is used to list the expected neighbors of all the links in a specified server. SYNTAX lsnwexpnbrs [ { -f <frame> --frame <frame> } [ { -c <cage> --cage <cage> } [ { -m <hubmodule> --hub_module <hubmodule> } ] ] { -s <supernode> --supernode <supernode> } [ { -d <drawer> --drawer <drawer> } [ { -m <hubmodule> --hub_module <hubmodule> } ] ] ] [ -p --page ] [ -h --help ] DESCRIPTION The lsnwexpnbrs command displays the expected neighbors of the links in a specified server in the cluster. The server could be specified as either 'frame-cage' combination or 'supernode-drawer' combination. The user can use this command only when the Central Network Manager (CNM) software is enabled and running in the system. To determine the valid range of values for frame, cage, supernode and drawer for the cluster, use the ISNM command 'lsnwloc'. The command line options can be given with either '-' or '--'. p. 70 of 98

71 OPTIONS -f, --frame Specifies the frame number to access the server. -c, --cage Specifies the cage number in the frame. -s, --supernode Specifies the supernode number in the cluster. A valid range for supernode number is 0-511; however, for a given cluster a subset of these supernode ids may be in use. -d, --drawer Specifies the drawer number in the supernode. A valid range for drawer within a supernode is m, --hub_module Specifies the hub module number in the server. Valid values are p, --page Displays the output one screen at a time. Each screen is followed by a prompt. If you press the Enter key, another page is displayed. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. 1 Indicates the CNM software is currently not running or disabled. 2 Indicates if there were any errors. EXAMPLES 1 To display the expected neighbors of the links in supernode 1 drawer 3, type : lsnwexpnbrs -s 1 -d 3 2 To display the expected neighbors of the links in frame 3 cage 2 hub module 1, type : lsnwexpnbrs -f 3 -c 2 -m 1 Sample output of lsnwexpnbr for supernode 0' and drawer 0' $./lsnwexpnbrs s = 0 -d =0 Loc: FR001-CG01-SN000-DR0-HB0-LR00 ExpNbr: FR001-CG02-SN000-DR1-HB0-LR00 ActualNbr: FR001-CG02-SN000-DR1-HB0-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR01 ExpNbr: FR001-CG02-SN000-DR1-HB1-LR00 ActualNbr: FR001-CG02-SN000-DR1-HB1-LR00 Loc : FR001-CG01-SN000-DR0-HB0-LR02 ExpNbr: FR001-CG02-SN000-DR1-HB2-LR00 ActualNbr: FR001-CG02-SN000-DR1-HB2-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR03 ExpNbr: FR001-CG02-SN000-DR1-HB3-LR00 ActualNbr: FR001-CG02-SN000-DR1-HB3-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR04 ExpNbr: FR001-CG02-SN000-DR1-HB4-LR00 ActualNbr: FR001-CG02-SN000-DR1-HB4-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR05 ExpNbr: FR001-CG02-SN000-DR1-HB5-LR00 ActualNbr: FR001-CG02-SN000-DR1-HB5-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR06 ExpNbr: FR001-CG02-SN000-DR1-HB6-LR00 ActualNbr: FR001-CG02-SN000-DR1-HB6-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR07 ExpNbr: FR001-CG02-SN000-DR1-HB7-LR00 ActualNbr: FR001-CG02-SN000-DR1-HB7-LR00 :. p. 71 of 98

72 lsnwgc This command is to display current Global Counter Information. SYNTAX lsnwgc [ -a -h --help ] DESCRIPTION The lsnwgc command displays the current Global Counter master ISR, the locations of the active Global Counter backups and Global Counter parameters if requested using the '-a' option. The command line options can be given with either '-' or '--'. OPTIONS -a Lists the Global Counter parameters along with Global Counter master and backups. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1. To display the Global Counter master and backups, type: lsnwgc 2. To display the Global Counter master, backups and Global Counter parameters, type: lsnwgc -a Sample output of lsnwgc' command : $ l s n w g c M a s t e r : F R C G S N D R 0 - H B 0 Counter ID: 2 No. of Configured Backups: 6 Backup: F R C G S N D R 0 - H B 0 Backup: F R C G S N D R 0 - H B 0 Backup: F R C G S N D R 0 - H B 0 Backup: F R C G S N D R 0 - H B 0 Backup: F R C G S N D R 0 - H B 0 Backup: F R C G S N D R 0 - H B lsnwlinkinfo This command is used to display information about the links in the Integrated Switch Router Network. SYNTAX lsnwlinkinfo [ { -f <frame> --frame <frame> } [ { -c <cage> --cage <cage> } [ { -m <hubmodule> --hub_module <hubmodule> } ] ] { -s <supernode> --supernode <supernode> } [ { -d <drawer> --drawer <drawer> } [ { -m <hubmodule> --hub_module <hubmodule> } ] ] ] [ -p --page ] [ -h --help ] p. 72 of 98

73 DESCRIPTION The lsnwlinkinfo command displays all known information about the links in the integrated switch router network. The links in a specific server can be obtained by specifying the optional parameters. The server can be specified as either 'frame-cage' combination or 'supernode-drawer' combination. The user can use this command only when the 'Central Network Manager' (CNM) software is enabled and running in the system. To determine the valid range of values for frame, cage, supernode and drawer for the cluster, use the ISNM command 'lsnwloc'. The command line options can be given with either '-' or '--'. OPTIONS -f, --frame -c, --cage -s, --supernode -d, --drawer Specifies the frame number to access the server. Specifies the cage number in the frame. Specifies the supernode number in the cluster. A valid range for supernode number is 0-511; however, for a given cluster a subset of these supernode ids may be in use. Specifies the drawer number in the supernode. A valid range for drawer within a supernode is m, --hub_module Specifies the hub module number in the server. Valid values are p, --page -h, --help Displays the output one screen at a time. Each screen is followed by a prompt. If you press the Enter key, another page is displayed. Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1. To display all known information about the links in supernode 2, drawer 1, hub module 3, type: lsnwlinkinfo -s 2 -d 1 -m 3 2. To display all known information about the links in frame 2, cage 3, hub module 1, type: lsnwlinkinfo -f 2 -c 3 -m 1 Sample output of lsnwlinkinfo' command for supernode 0', drawer 0' and hub 0' $./lsnwlinkinfo -s 0 d 0 m 0 FR001-CG01-SN000-DR0-HB0-LL0 Status: DOWN_POWEROFF ExpNbr: FR001-CG01-SN000-DR0-HB3- LL0 ActualNbr: FR001-CG01-SN000- DR0-HB3-LL0 FR001-CG01-SN000-DR0-HB0-LL1 Status: DOWN_POWEROFF ExpNbr: FR001-CG01-SN000-DR0-HB5- p. 73 of 98

74 LL0 ActualNbr: FR001-CG01-SN000- DR0-HB5-LL0 FR001-CG01-SN000-DR0-HB0-LL2 Status: DOWN_POWEROFF ExpNbr: FR001-CG01-SN000-DR0-HB3- LL2 ActualNbr: FR001-CG01-SN000- DR0-HB3-LL2 FR001-CG01-SN000-DR0-HB0-LL3 Status: DOWN_POWEROFF ExpNbr: FR001-CG01-SN000-DR0-HB4- LL0 ActualNbr: FR001-CG01-SN000- DR0-HB4-LL0 :. FR001-CG01-SN000-DR0-HB0-LR1 Status: DOWN_POWEROFF ExpNbr: : FR001-CG02-SN000-DR1- HB1-LR0 ActualNbr: FR001-CG02-SN000- DR1-HB1-LR0 FR001-CG01-SN000-DR0-HB0-LR2 Status: DOWN_POWEROFF ExpNbr: : FR001-CG03-SN000-DR2- HB1-LR0 ActualNbr: FR001-CG03-SN000- DR2-HB1-LR0 :. Sample output of lsnwlinkinfo' command for supernode 0', drawer 0' and hub 0' $./lsnwlinkinfo -s 0 d 0 t 0 FR001-CG01-SN000-DR0-HB0-LL0 Status: DOWN_POWEROFF ExpNbr: FR001-CG01-SN000-DR0-HB3-LL0 ActualNbr: FR001-CG01-SN000- DR0-HB3-LL0 FR001-CG01-SN000-DR0-HB0-LL1 Status: DOWN_POWEROFF ExpNbr: FR001-CG01-SN000-DR0-HB5-LL0 ActualNbr: FR001-CG01-SN000- DR0-HB5-LL0 FR001-CG01-SN000-DR0-HB0-LL2 Status: DOWN_POWEROFF ExpNbr: FR001-CG01-SN000-DR0-HB3-LL2 ActualNbr: FR001-CG01-SN000- DR0-HB3-LL2 FR001-CG01-SN000-DR0-HB0-LL3 Status: DOWN_POWEROFF ExpNbr: FR001-CG01-SN000-DR0-HB4-LL0 ActualNbr: FR001-CG01-SN000- DR0-HB4-LL0 :. FR001-CG01-SN000-DR0-HB0-LR1 Status: DOWN_POWEROFF ExpNbr: : FR001-CG02-SN000-DR1-HB1-LR0 ActualNbr: FR001-CG02-SN000- DR1-HB1-LR0 FR001-CG01-SN000-DR0-HB0-LR2 Status: DOWN_POWEROFF ExpNbr: : FR001-CG03-SN000-DR2-HB1-LR0 ActualNbr: FR001-CG03-SN000- DR2-HB1-LR0 : ISNM Link Status The following describes status of links as displayed by the lsnwlinkinfo command. Procedures to take for each status follow the list. DOWN_POWEROFF - the server in which this link is installed is powered off DOWN_PENDING_POWER_OFF the server in which this link is installed has received a power off command. Route table updates are underway on neighboring servers, to exclude links to the server that is pending power off. DOWN_ISRINVALID - the ISR ID on the hub module to which this link is attached is not valid. This usually means that a required piece of configuration information (frame id, cage id, supernode id, drawer id, topology indicator) has not been set up in the FSP. INVALID_NOTFUNCTIONAL the ISR to which this link is attached is garded or in an otherwise non-functional state. INVALID_NOTPRESENT the ISR to which this link is attached is not installed (physically not p. 74 of 98

75 present), or is garded; or, the optics for this port are not populated. DOWN_DISABLED_USR - the port has been disabled by issuing the chnwfence command. The neighbor port will show a status of DOWN_FAULTY. DOWN_DISABLED_ERR - the port has been disabled by firmware due to a link error or optical module error threshold event. The neighbor port will show a status of DOWN_FAULTY. DOWN_DISABLED_MISMATCH the transmit side and the receive side of the port do not have matching state. Either the transmit side is disabled and the receive side is not, or vice versa. DOWN_MISWIRED - the link is miswired. Use the lsnwmiswire or the lsnwexpnbrs command to determine the expected cabling based on the cluster topology. DOWN_NBRPOWEROFF - the link is down because the server at the other end of the link is powered off. DOWN_NBRPENDINGPOWEROFF the server in which this link is installed is connected to a server which has received a power off command. DOWN_FAULTY - the port is not operational. DOWN_NBRNOTINSTALLED - this port does not have a cable installed because the neighbor drawer to which it should be connected (according to the defined topology) is not installed. DOWN_UNREACHABLE - the Central Network Management daemon has lost contact with the FSP that manages the drawer in which this link is installed. The last known state of the port was DOWN. UP_UNREACHABLE - the Central Network Management daemon has lost contact with the FSP that manages the drawer in which this link is installed. The last known state of the port was UP. UP_OPERATIONAL - the link is in working order with 10 optical lanes for a D link or 6 optical lanes for a LR link. UP_WIDTH9 this D link is in working order with 9 optical lanes. UP_WIDTH8 this D link is in working order with 8 optical lanes. UP_WIDTH5 this LR link is in working order with 5 optical lanes. UP_WIDTH4 this LR link is in working order with 4 optical lanes. DOWN_OLCT - Indicates that this port is in OLCT mode but no OLCT status has been explicitly collected. DOWN_OLCT_EXCHANGE_PENDING - Indicates that this port has been put into OLCT mode on one side of the link, but ID exchange has not occurred because the other side of the link has not been put into OLCT mode. DOWN_OLCT_MISWIRED - Indicates that this port has been put into OLCT mode, that ID exchange has occurred and has indicated a miswired cable. DOWN_OLCT_CORRECT- Indicates that this port has been put into OLCT mode, that ID exchange has occurred and has indicated a correctly wired cable. DOWN_TRANSITIONAL - this port is transitioning between up and down and has not reached a steady state. DOWN_RECV_GOOD - CNM has received notification that this port is operational, but has not yet been notified regarding the neighbor port. The following are recommended actions to take in response to status displayed by the ISNM lsnwlinkinfo command. p. 75 of 98

76 The following are recommended actions to take in response to status displayed by the ISNM lsnwlinkinfo command. DOWN_DISABLED_USR - the port has been disabled by issuing the chnwfence command. If the port should be enabled and there is no recommended action reported against it, enable it using the chnwfence command. For information on the chnwfence command, see chnwfence in this command reference section. DOWN_DISABLED_ERR - the port has been disabled because too many serious error events have occurred. Check the TEAL alert database or the ISNM Master Event Analysis HMC s serviceable events list to determine if a problem has been reported on this link. If it has, execute the procedure for diagnosing a faulty link. If the link should be enabled and there is no recommended action reported against it, enable it using the chnwfence command. For information on the chnwfence command, see chnwfence in this command reference section. DOWN_DISABLED_MISMATCH either the transmit side of the port is disabled and the receive side is not, or vice versa. The chnwfence command acts against both the transmit and the receive sides of the port. If the port should be enabled and there is no recommended action reported against it, enable it using the chnwfence command with the -u option. If the port should not be enabled, disable it using the chnwfence command. For information on the chnwfence command, see chnwfence in this command reference section. DOWN_FAULTY - the port is not operational. Check the neighbor port for DOWN_DISABLED states and take the recommended actions for those states. Check the TEAL alert database or the ISNM Master Event Analysis HMC's serviceable events list to determine if a problem has been reported on this link. If it has, execute the procedure for diagnosing a faulty link. DOWN_ISRINVALID - the ISR ID on the hub module to which this link is attached is not valid. This usually means that a required piece of configuration information (frame id, cage id, supernode id, drawer id, topology indicator) has not been set up in the FSP. In order to recover from this, first inspect the Cluster Database to determine that the topology indicator, frame id, and supernode id have been configured. DOWN_MISWIRED - the link is miswired. Use the lsnwmiswire or the lsnwexpnbrs command to determine the expected cabling based on the cluster topology. Make corrections to cabling to match the planned topology. DOWN_NBRNOTINSTALLED - this port does not have a cable installed because the neighbor drawer to which it should be connected (according to the defined topology) is not installed. Verify that the neighbor should not be installed at this time. If it should be installed, ensure that it is connected to the service network and can be recognized by xcat. If it can be recognized by xcat, assure that it can be recognized by ISNM by issuing the lsnwcomponents command. DOWN_NBRPOWEROFF - the link is down because the drawer at the other end of the link is powered off. If the drawer should not be powered off, power it on. DOWN_OLCT - Indicates that this link is in OLCT mode but no OLCT status has been explicitly collected. If the link should not be in OLCT mode, use chnwolct to take it out of OLCT mode. DOWN_OLCT_CORRECT- Indicates that this link has been put into OLCT mode, that ID exchange has occurred and has indicated a correctly wired cable. No action is required until you wish to bring the link into an operational state. At that point, use chnwolct to bring the port out of OLCT mode. DOWN_OLCT_EXCHANGE_PENDING - Indicates that this link has been put into OLCT mode on one side of the link, but ID exchange has not occurred because the other side of the link has not been p. 76 of 98

77 put into OLCT mode. Continue to query status until DOWN_OLCT_EXCHANGE_PENDING is no longer displayed, and assure that the link has gone into the proper state (DOWN_OLCT_CORRECT if in OLCT mode and UP_OPERATIONAL if not in OLCT mode). If after 10 minutes the status has not changed, call your next level of support. DOWN_OLCT_MISWIRED - Indicates that this link has been put into OLCT mode, that ID exchange has occurred and has indicated a miswired cable. Make corrections to cabling to match the planned topology. DOWN_PENDINGNBRPOWEROFF the drawer in which this link is installed is connected to a server which has received a power off command. Continue to query drawer status using the lsnwloc command to ensure that the power off is completed. The next expected state is DOWN_NBRPOWEROFF. If the port does not change to the DOWN_ NBRPOWEROFF within 12 minutes, query the state of the drawer using the xcat command rpower [noderange] state to determine if there is a problem with the power off. DOWN_PENDINGPOWEROFF the drawer in which this link is installed has received a power off command. Continue to query the drawer status using the lsnwloc command to ensure that the power off is completed. The next expected state is DOWN_POWEROFF. If the link does not change to the DOWN_ POWEROFF within 12 minutes, query the state of the server using the xcat command rpower [noderange] state to determine if there is a problem with the power off. DOWN_POWEROFF - the drawer in which this link is installed is powered off. If the drawer should not be powered off, power it on using the xcat rpower command. DOWN_UNREACHABLE - the Central Network Management daemon has lost contact with the drawer in which this port is installed. Check the Ethernet network between the EMS and the FSP. Look to see if multiple FSPs (and BPAs) are having issues. For example, if the frame BPA and all FSPs in the frame are unreachable, look for a frame level problem. UP_OPERATIONAL - the link is in working order. No action is required. UP_WIDTH9 - the link is in working order. No action is required. UP_WIDTH8 - the link is in working order. No action is required. UP_WIDTH5 - the link is in working order. No action is required. UP_WIDTH4 - the link is in working order. No action is required. DOWN_TRANSITIONAL - Use the lsnwlinkinfo command to examine the status of the port several times in a row. If the port status does not change to a steady state, perform link diagnosis to determine whether the cable is at fault. DOWN_RECV_GOOD - Use the lsnwlinkinfo command to examine the status of the port several times in a row. If the link status does not change, check the "ActualNbr" for this link, and use the lsnwcomponents command to check that CNM can communicate with the drawer in which the other end of the link resides lsnwloc This command is used to display information about the list of servers with which the Central Network Manager is communicating. SYNTAX lsnwloc [ -f <frame> --frame <frame> -s <supernode> --supernode <supernode> ] [ -p --page ] [ -h --help ] p. 77 of 98

78 DESCRIPTION The lsnwloc command displays list of servers that are successfully contacted and which are actively managed by the Central Network Manager (CNM). The frame id and supernode number information about the servers are listed. If no options are specified, it lists all the server information. If only frame or supernode option is specified, then information about the servers in that frame or supernode is listed. The user can use this command only when the Central Network Manager (CNM) software is enabled and running in the system. The command line options can be given with either '-' or '--'. OPTIONS -f, --frame Specifies the frame number in the cluster. -s, --supernode Specifies the supernode number in the cluster. A valid range for supernode number is 0-511; however, for a given cluster a subset of these supernode ids may be in use. -p, --page Displays the output one screen at a time. Each screen is followed by a prompt. If you press the Enter key, another page is displayed. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To display frame-cage and supernode-drawer information for the servers in frame number 4, type: lsnwloc -f 4 The following describes the LNMC states as displayed by the lsnwloc command. An explanation of each state follows. STANDBY FUNCTIONAL_TORRENT RUNTIME TERMINATE PENDINGPOWEROFF LOW_POWER_IPL RUNTIME_CNM_EXCLUDED STANDBY_CNM_EXCLUDED RUNTIME_CNM_UNREACHABLE STANDBY_CNM_UNREACHABLE Sample output of lsnwloc' command : $./lsnwloc FR001-CG01-SN000-DR0 RUNTIME FR001-CG02-SN000-DR1 RUNTIME FR001-CG03-SN000-DR2 RUNTIME FR001-CG04-SN000-DR3 RUNTIME FR001-CG05-SN001-DR0 RUNTIME FR001-CG06-SN001-DR1 RUNTIME FR001-CG07-SN001-DR2 RUNTIME FR001-CG08-SN001-DR3 RUNTIME FR001-CG09-SN002-DR0 RUNTIME FR001-CG10-SN002-DR1 RUNTIME FR001-CG11-SN002-DR2 RUNTIME p. 78 of 98

79 FR001-CG12-SN002-DR3 RUNTIME FR002-CG01-SN003-DR0 RUNTIME : ISNM Drawer State The following describes the LNMC states as displayed by the lsnwloc command. STANDBY - the drawer is powered off. The FSP is at standby power. FUNCTIONAL_TORRENT the drawer is the process of powering on. IPL has completed to the point where the P775 I/O hub is in the process of initialization. RUNTIME the drawer is powered on. The operating system may or may not be loaded. PENDINGPOWEROFF the drawer has received a power off command. The P775 will remain in the PENDINGPOWEROFF state until route table updates are complete on neighbor drawers or until a time limit is reached on the drawer that is pending power off. LOW_POWER_IPL TERMINATE the drawer has experienced a checkstop or other serious error. RUNTIME_CNM_EXCLUDED the p775 is powered on, but a mismatch exists between the cluster configuration data contained in the cluster database and the configuration data stored on the FSP for this drawer. This drawer cannot participate as part of the ISR network until the configuration mismatch is corrected. See the /var/opt/isnm/cnm/log/eventsummary log to determine where configuration data needs correcting. If a configuration data update on the FSP is needed, you will need to power down the P775 server, use the chnwsvrconfig command to update the FSP data, and then power up the server. STANDBY_CNM_EXCLUDED the p775 is powered off, and a mismatch exists between the cluster configuration data contained in the cluster database and the configuration data stored on the FSP for this server. This server cannot participate as part of the ISR network until the configuration mismatch is corrected. See the /var/opt/isnm/cnm/log/eventsummary log to determine where configuration data needs correcting. RUNTIME_CNM_UNREACHABLE - the Central Network Management daemon has lost contact with the FSP. The drawer state last known to CNM was RUNTIME. Check the Ethernet network between the EMS and the FSP. Look for patterns to see if multiple servers are having issues. STANDBY_CNM_UNREACHABLE - the Central Network Management daemon has lost contact with the FSP. The drawer state last known to CNM was STANDBY. Check the Ethernet network between the EMS and the FSP. Look for patterns to see if multiple servers are having issues lsnwmiswire This command is used to display the list of miswired links. t SYNTAX lsnwmiswire [ { -f <frame> --frame <frame> } [ -c <cage> --cage <cage> ] { -s <supernode> --supernode <supernode> } [ -d <drawer> --drawer <drawer> ] ] [ -a ] [ -p --page ] [ -h --help ] DESCRIPTION The lsnwmiswire command displays the miswired links in the cluster. To get the miswired links in a specific frame or supernode, user needs to provide the optional parameter frame or supernode number. The user can use this command only when the Central Network Manager (CNM) software is enabled and running in the system. To determine the valid values for frame and supernode for the cluster, use the ISNM command 'lsnwloc'. The command line options can be given with either '-' or '--'. OPTIONS p. 79 of 98

80 -f, --frame Specifies the frame number in the cluster. -c, --cage Specifies the cage number in the frame. -s, --supernode Specifies the supernode number in the cluster. A valid range for supernode number is 0-511; however, for a given cluster a subset of these supernode ids may be in use. -d, --drawer Specifies the drawer number in the supernode. A valid range for drawer within a supernode is a Displays the Service Location Code for each location in the output -p, --page Displays the output one screen at a time. Each screen is followed by a prompt. If you press the Enter key, another page is displayed. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To display all the Miswired links in supernode 2, type: lsnwmiswire -s 2 2 To display all the Miswired links in supernode 2, drawer 1, type: lsnwmiswire -s 2 -d 1 3 To display all the Miswired links in frame 3, cage 2, type: lsnwmiswire -f 3 -c 2 4 To display all the Miswired links in frame 2, cage 4 along with the location codes, type: lsnwmiswire -f 2 -c 4 a Sample output of lsnwmiswire' command : $./lsnwmiswire s 0 Loc: FR001-CG01-SN000-DR0-HB0-LR00 ExpNbr: FR001- CG02-SN000-DR1-HB0- LR00 ActualNbr: FR001-CG02- SN001-DR1-HB0-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR01 ExpNbr: FR001- CG02-SN000-DR1-HB1- LR00 ActualNbr: FR001-CG02- SN001-DR1-HB1-LR00 Loc : FR001-CG01-SN000-DR0-HB0-LR02 ExpNbr: FR001- CG02-SN000-DR1-HB2- LR00 ActualNbr: FR001-CG02-SN001- DR1-HB2-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR03 ExpNbr: FR001- CG02-SN000-DR1-HB3- LR00 ActualNbr: FR001-CG02- SN001-DR1-HB3-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR04 ExpNbr: FR001- CG02-SN000-DR1-HB4- LR00 ActualNbr: FR001-CG02- SN001-DR1-HB4-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR05 ExpNbr: FR001- CG02-SN000-DR1-HB5- LR00 ActualNbr: FR001-CG02- SN001-DR1-HB5-LR00 Loc: FR001-CG01-SN000-DR0-HB0-LR06 ExpNbr: FR001- CG02-SN000-DR1-HB6- LR00 ActualNbr: FR001-CG02- SN001-DR1-HB6-LR00 p. 80 of 98

81 Loc: FR001-CG01-SN000-DR0-HB0-LR07 ExpNbr: FR001- CG02-SN000-DR1-HB7- LR00 ActualNbr: FR001-CG02- SN001-DR1-HB7-LR00 : lsnwtopo This command is to display the cluster topology information used by the network management software. SYNTAX lsnwtopo [ -C { { -f <frame> --frame <frame> } { -c <cage> --cage <cage> } } { { -s <supernode> -- supernode <supernode> } { -d <drawer> --drawer <drawer> } } { -A --all } ] [ -h --help ] DESCRIPTION The lsnwtopo command displays the cluster topology Information. If no options are specified the topology information stored in the PERCS Database will be displayed. To display the network topology information used by the Local Network Management Controller (LNMC) in a particular server, specify the server by either 'frame + cage' combination or 'supernode + drawer' combination. The user can use this command only when the Central Network Manager' (CNM) software is enabled and running in the system. To determine the valid values for frame, cage, supernode and drawer for the cluster, use the ISNM command 'lsnwloc'. The command line options can be given with either '-' or '--'. OPTIONS -C To get topology information used by CNM. -f, --frame Specifies the frame number to access the server. -c, --cage Specifies the cage number in the frame. -s, --supernode Specifies the supernode number in the cluster. A valid range for supernode number is 0-511; however, for a given cluster a subset of these supernode ids may be in use. -d, --drawer Secifies the drawer number in the supernode. A valid range for drawer within a supernode is A, --all Gets the topology information for all the FSPs currently managed by CNM. -h, --help Displays the command's usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. >0 Indicates if there were any errors. EXAMPLES 1 To display the cluster topology information stored in Cluster Database, type: lsnwtopo p. 81 of 98

82 2 To display the cluster topology information used by CNM type : lsnwtopo -C 3 To display the cluster topology information used by the Local Network Management Controller (LNMC) in supernode 2 drawer 0, type : lsnwtopo -s 2 -d 0 4 To display the cluster topology information used by the Local Network Management Controller (LNMC) in frame 3 cage 2, type : lsnwtopo -f 3 -c 2 Sample Output: > # Note: All copies should match > # Cluster Database > lsnwtopo ISR network topology specified by cluster configuration data is 128D > # CNM copy > lsnwtopo ISR network topology specified by cluster configuration data is 128D > # Supernode 0 Drawer 0 local copy [> lsnwtopo -s 0 -d 0 Supernode 0 Drawer 0 : Topology 128D, Frame 8, Cage nwlinkdiag NAME nwlinkdiag - Diagnose an ISR network link PURPOSE Gathers link state training information from a link in the Integrated Switch Network. This link training information, combined with the use of an optical wrap device, can isolate a link failure caused by a faulty cable. SYNTAX nwlinkdiag { { -f <frame> --frame <frame> } { -c <cage> --cage <cage> } { -m <hubmodule> --hub_module <hubmodule> } { -lr <LR-linkid> -ld <D-link-id> } } { { -s <supernode> --supernode <supernode> } { -d <drawer> --drawer <drawer> } { -m <hubmodule> --hub_module <hubmodule> } { -lr <LR-link-id> -ld <D-link-id> } } [ -h --help ] DESCRIPTION The nwlinkdiag command gathers link state training information from a link in the Integrated Switch Network. The nwlinkdiag command can only be executed against servers that are in the LNMC Runtime state. The link training information can be read any time the server is in the p. 82 of 98

83 LNMC Runtime state but in order to diagnose a link, an optical wrap device must be installed. The steps for diagnosing a link are: 1. Uninstall the optical cable from the port(s) to be diagnosed. 2. Install an optical wrap device in the port(s) to be diagnosed. 3. Wait a few seconds. 4. Issue the nwlinkdiag command for the link(s) to be diagnosed. 5. The nwlinkdiag command will display information about whether the link is able to train with the wrap device installed, and whether the fault can be isolated to a cable. The command line options can be given with either â-â or â--â. OPTIONS -f, --frame Specifies the frame ID in which the link to diagnose resides. -c, --cage Specifies the cage ID in which the link to diagnose resides.. -s, --supernode Specifies the supernode ID in which the link to diagnose resides. -d, --drawer Specifies the drawer ID in which the link to diagnose resides. -m, --hub_module Specifies the hub module within the cage/drawer in which the link to diagnose resides. -lr Specifies the link ID of the LR link to diagnose. Valid values are ld Specifies the link ID of the D link to diagnose. Valid values are h, --help Displays the commandâs usage statement to standard output. RETURN VALUES 0 Indicates successful completion of the command. 1 Indicates the CNM software is currently not running or disabled. 2 Indicates if there were any errors. EXAMPLES 1 To diagnose D link 3 in supernode 3 drawer 1 hub 4, type: nwlinkdiag -s 3 -d 1 -m 4 -ld 3 p. 83 of 98

84 2 To diagnose LR link 9 in supernode 49 drawer 0 hub 3, type: nwlinkdiag -s 49 -d 0 -m 3 -lr 9 3 To diagnose D link 2 in frame 7 cage 12 hub 5, type: nwlinkdiag -f 7 -c 12 -m 5 -ld 2 Sample output of the nwlink diag command for LR link 9: on hub module 5 in frame 7 cage 12: $ nwlinkdiag f 7 c 12 m 5 lr 8 FR007-CG12-SN049-DR0-HB5-LR9 LOCATION_CODE=U78A P1-T9 This link is not operational. If a cable is installed in the optical port, remove the cable and install an optical wrap device instead, then re-issue the nwlinkdiag command. If a wrap device is already installed, then the problem lies behind the optical port ISNM location naming conventions This topic provides information about the ISNM location naming conventions. The following naming conventions are found in ISNM: v ISNM Hardware Location v Service or Physical Location There are several location codes that are directly pertinent to ISNM. In addition to those, it is important to note those that are indirectly related and are likely to be referenced in problem analysis. The hardware location code is a logical location code that is defined in the TEAL System Design. It is constructed based on the hierarchical nature of hardware function. Such codes have been developed partly in response to a lack of ability for service location codes to fully identify function location and to also provide a location code that is more natural for code to manipulate. For ISNM, and the 9125-F2C cluster solution, the ISNM hardware location code takes the form: FRfff-CGcc-SNsss-DRddd-HBh-L[DRL]pp FR = Frame number fff CG = cage number ccc, which is the location of the server drawer within the frame numbered from the bottom of the frame up. Valid cc numbers are from 0 to 11. SN = supernode number sss. The supernode number is relative all of the supernodes in the cluster. Valid sss numbers are from 0 to 511. DR = Drawer number d within the supernode. Valid d numbers are 0-3. HB = Hub number h within the drawer. Valid h numbers are 0-7. LD = D-link number pp in the Hub. Valid pp numbers for LD are LR = LR-link number pp in the Hub. Valid pp numbers for LD are LL = LL-link number pp in the Hub. Valid pp numbers for LD are 0-6. p. 84 of 98

85 The physical, or service location code is defined the POWER Architecture. A key difference between service locations and other locations is that the service locations never have a 0 as an instance identifier; whereas, logical location codes quite often use 0. The pertinent service locations are indicated below. Typical use of the Service Location Code can be seen in: v v v v v v CEC/Planar = U78A9.001.sssssss-P1 Hub = U78A9.001.sssssss-P1-R[1-8] Optical module = U78A9.001.sssssss-P1-R[1-8]-R[1-40] D-link = U78A9.001.sssssss-P1-R[1-8] LR-link = U78A9.001.sssssss-P1-T9 LL-link = U78A9.001.sssssss-P1-R[1-8] # there is no port designator for LL link service locations. The hub module is the pertinent location that is required for service. U78A9.001.ssssss is the Unit location. This identifies the server. The sssssss string identifies the serial number of the server. A typical short notation of the unit location is U*, or simply U. The planar P1 is always the same and designates the entire server drawer. The R is an element name that stands for "resource". This is something can be defined below the typical FRU level, such as the hub module. Below the hub module R-number is an R number for each optical module. The architected element name for a port connector is "T". Because of the assemblies that are used for 9125-F2C HFI cabling there is a hierarchy to the naming convention. Each group of 8 D-link cables and the LR link connector assembly for a drawer is labeled T1-T17, where T1-T8 and T10-T17 are Dlink assemblies and T9 is the LR link assembly. The Dlink assemblies have individual cable connector numbers of T1 through T8. The LR link connector assembly has no lower "T" names in the hierarchy. So, D-links will look like U78A9.001.ssssss-P1-R[1-8]-T[1-8;10-17]-T[1-8] and LR links will look like: U78A9.001.ssssss-P1-T9. The following describes how to translate from service location to hardware location codes. Table 16. Service to hardware location code Component Service/Physical Location Code Hardware location code CEC Uffff.001.ssssss-P1 BBx-FRx-SNx-DRx Hub Uffff.001.ssssss-P1-R[1-8] BBx-FRx-SNx-DRx-HBx Dlink optical module Uffff.001.ssssss-P1-R[1-8]-R[1-15] BBx-FRx-SNx-DRx-HBx-LDx Dlink Uffff.001.ssssss-P1-R[1-8]-T[1-8;10-17]-T[1-8] BBx-FRx-SNx-DRx-HBx-LDx LRlink optical module Uffff.001.ssssss-P1-R[1-8]-R[16-39] BBx-FRx-SNx-DRx-HBx-LRx LRlink port Uffff.001.ssssss-P1-T9 BBx-FRx-SNx-DRx-HBx-LRx LLlink port Uffff.001.ssssss-P1-R[1-8]* Note, there is no port designator for LL links. BBx-FRx-SNx-DRx-HBx-LLx p. 85 of 98

86 The following describes the physical placement of the hub hardware and service locations: D Optical links PCIe 2.0 LR Optical links PCIe 2.0 D Optical links Hub 0 U*-P1-R1 Hub 1 U*-P1-R2 Hub 2 U*-P1-R3 Hub 3 U*-P1-R4 Hub 4 U*-P1-R5 Hub 5 U*-P1-R6 Hub 6 U*-P1-R7 Hub 7 U*-P1-R8 DIMMs QCMs DIMMs DCCAs p. 86 of 98

87 The following describes the physical placement of the optical module hardware and service locations: p. 87 of 98

88 The following figures describe D-link and LR-link port hardware and service locations. The rear bulkhead is laid out with the D-links to the left and right, PCIe 2.0 adapters next to the D-links and LR-links in the center. The Optical link connectors are implemented using two assemblies; one for the D-links and one for the LR-links. The D-links are implemented with 8 port assemblies that are oriented horizontally. The D- link optical assemblies loosely assemble individual D-link optical connectors. The individual optical connectors can be disassembled from the assembly for service. There are 16 of these assemblies. There is a single LR-link assembly that groups all LR-link connectors. The previous figure illustrates the relationship between Hub and hub port and service location. The top of the figure indicates the connector within the D-link assembly (T1-T8). The hub is indicated below that. Each connector is indicated with the format P1-Tx-Ty with the Hub-Port below it (eg. 7-0 = hub 7 port 0). The first T location indicates the optical link assembly. The D-link assemblies are 1-8 and The LR-link assembly is T-9. For example, Hub5 Dlink port 6 uses optical connector location P1-T4-T5. This uses D-link assembly T4. Hub2 Dlink port 13 uses optical connector location P1-T16-T4. This uses D-link assembly T p. 88 of 98

89 3.2 Operating system command reference for network management The listed network management commands are available on the running partition chdev (AIX only) This command is used to change the ODM parameters. Examples of attributes for hfi will follow. /usr/sbin/chdev -l [device] -a '[attribute=value]' For hfi, devices are hfi0 and hfi1. Valid attributes for hfi are: dd_trc_lvl ip_trc_lvl num_bounce_pts num_dynamic_win recv_limit rpoolsize send_limit spoolsize test_mode Example: chdev -l hfi0 -a dd_trc_lvl=0xf0f0f0f hf_dump (Linux) This command is to dump internal IP interface structures for debug. hf_dump -h hf_dump -l hfi_devname -i hf_devname -a -t [-r] -h: display usage information -l: hfi device name hfi{0 1} -i: hf device name hf{ } -a: dump data structure -t: dump trace -r: used with -t, dump trace and return -Z: dump everything and return ifhf_dump (AIX) This command dumps hfi interface information used for debug. /usr/hfi/bin/ifhf_dump -h ifhf_dump -l hf_devname -a ifhf_dump -l hf_devname -f dump_file -a -h: display usage information -l: hf device name hf{ } -a: dump internal data structure -o: dump content of fifos -x: dump content of rcxt -Z: dump ALL p. 89 of 98

90 -f <dump_file>: dump from dump file hfi_dump This command is to dump internal HFI Device Driver structures for debug. /usr/sbin/hfi_dump -h hfi_dump [-v] -l -r -g -d -m -s -c -i -t [-f] -w win [-o offset] -Z dev dev: logical device name, hfi#. -c index: run Hcall CAU_READ_INDEX. -d: dump hfidd_dds. -f: display formatted ROUTE_INFO output -g: dump hfidd_global. -h: display usage information. -i: run Hcall DUMP_INFO. -l: display trace information. -m: dump hfidd_dr. -o offset: window rdma rcxt offset -r: return snapshot of -s: dump hfidd_state_change. -t: Hcall ROUTE_INFO output. -v: verbose output. trace information. -w win: dump hfidd_window and registers. -Z: dump all hfi_snap In order to facilitate the data collection for HFI device and IP problems, use the snap command which collects a variety of information related to the HFI device and the corresponding IP driver. This allows service to identify problem areas within either the IP stack or the HFI driver. The hfi_snap command is invoked by: /usr/hfi/bin/hfi.snap [ -l minor_number] where minor_number is the device minor number, i.,e., 0 for hfi0 and 1 for hfi1. The output of this command goes into /var/adm/hfi/snaps, where the snap data is stored in a compressed tar file. The following information is saved by hfi_snap: System configuration information including that of the hfi device driver. System initialization logs Current process activity snapshot The device tree Dynamic reconfiguration traces netstat output Device statistics UDP statistic TCP statistic IP statistic Socket information p. 90 of 98

91 hfi_dump output ifhf_dump output p. 91 of 98

92 4 Notices This information was developed for products and services offered in the U.S.A. The manufacturer may not offer the products, services, or features discussed in this document in other countries. Consult the manufacturer's representative for information on the products and services currently available in your area. Any reference to the manufacturer's product, program, or service is not intended to state or imply that only that product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any intellectual property right of the manufacturer may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any product, program, or service. The manufacturer may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to the manufacturer. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: THIS INFORMATION IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. The manufacturer may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to Web sites not owned by the manufacturer are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this product and use of those Web sites is at your own risk. The manufacturer may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning products not produced by this manufacturer was obtained from the suppliers of those products, their published announcements or other publicly available sources. This manufacturer has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to products not produced by this manufacturer. Questions on the capabilities of products not produced by this manufacturer should be addressed to the suppliers of those products. All statements regarding the manufacturer's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. p. 92 of 98

93 The manufacturer's prices shown are the manufacturer's suggested retail prices, are current and are subject to change without notice. Dealer prices may vary. This information is for planning purposes only. The information herein is subject to change before the products described become available. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. If you are viewing this information in softcopy, the photographs and color illustrations may not appear. The drawings and specifications contained herein shall not be reproduced in whole or in part without the written permission of the manufacturer. The manufacturer has prepared this information for use with the specific machines indicated. The manufacturer makes no representations that it is suitable for any other purpose. The manufacturer's computer systems contain mechanisms designed to reduce the possibility of undetected data corruption or loss. This risk, however, cannot be eliminated. Users who experience unplanned outages, system failures, power fluctuations or outages, or component failures must verify the accuracy of operations performed and data saved or transmitted by the system at or near the time of the outage or failure. In addition, users must establish procedures to ensure that there is independent data verification before relying on such data in sensitive or critical operations. Users should periodically check the manufacturer's support websites for updated information and fixes applicable to the system and related software. Ethernet connection usage restriction This product is not intended to be connected directly or indirectly by any means whatsoever to interfaces of public telecommunications networks. Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Red Hat, the Red Hat "Shadow Man" logo, and all Red Hat-based trademarks and logos are trademarks or registered trademarks of Red Hat, Inc., in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. p. 93 of 98

94 Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Electronic emission notices Class A Notices The following Class A statements apply to the IBM servers that contain the POWER7 processor and its features unless designated as electromagnetic compatibility (EMC) Class B in the feature information. Federal Communications Commission (FCC) statement Note: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits. IBM is not responsible for any radio or television interference caused by using other than recommended cables and connectors or by unauthorized changes or modifications to this equipment. Unauthorized changes or modifications could void the user's authority to operate the equipment. This device complies with Part 15 of the FCC rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation. Industry Canada Compliance Statement This Class A digital apparatus complies with Canadian ICES-003. Avis de conformité à la réglementation d'industrie Canada Cet appareil numérique de la classe A est conforme à la norme NMB-003 du Canada. European Community Compliance Statement This product is in conformity with the protection requirements of EU Council Directive 2004/108/EC on the approximation of the laws of the Member States relating to electromagnetic compatibility. IBM cannot accept responsibility for any failure to satisfy the protection requirements resulting from a non-recommended modification of the product, including the fitting of non-ibm option cards. This product has been tested and found to comply with the limits for Class A Information Technology Equipment according to European Standard EN The limits for Class A equipment were derived for commercial and industrial environments to provide reasonable protection against interference with licensed communication equipment. European Community contact: IBM Deutschland GmbH p. 94 of 98

95 Technical Regulations, Department M456 IBM-Allee 1, Ehningen, Germany Tele: Warning: This is a Class A product. In a domestic environment, this product may cause radio interference, in which case the user may be required to take adequate measures. VCCI Statement - Japan The following is a summary of the VCCI Japanese statement in the box above: This is a Class A product based on the standard of the VCCI Council. If this equipment is used in a domestic environment, radio interference may occur, in which case, the user may be required to take corrective actions. Japanese Electronics and Information Technology Industries Association (JEITA) Confirmed Harmonics Guideline (products less than or equal to 20 A per phase) Japanese Electronics and Information Technology Industries Association (JEITA) Confirmed Harmonics Guideline with Modifications (products greater than 20 A per phase) Electromagnetic Interference (EMI) Statement - People's Republic of China p. 95 of 98

96 Declaration: This is a Class A product. In a domestic environment this product may cause radio interference in which case the user may need to perform practical action. Electromagnetic Interference (EMI) Statement - Taiwan The following is a summary of the EMI Taiwan statement above. Warning: This is a Class A product. In a domestic environment this product may cause radio interference in which case the user will be required to take adequate measures. IBM Taiwan Contact Information: Electromagnetic Interference (EMI) Statement - Korea Please note that this equipment has obtained EMC registration for commercial use. In the event that it has been mistakenly sold or purchased, please exchange it for equipment certified for home use. Germany Compliance Statement Deutschsprachiger EU Hinweis: Hinweis für Geräte der Klasse A EU-Richtlinie zur Elektromagnetischen Verträglichkeit Dieses Produkt entspricht den Schutzanforderungen der EU-Richtlinie 2004/108/EG zur Angleichung der Rechtsvorschriften über die elektromagnetische Verträglichkeit in den EU-Mitgliedsstaaten und hält die Grenzwerte der EN Klasse A ein. Um dieses sicherzustellen, sind die Geräte wie in den Handbüchern beschrieben zu installieren und zu p. 96 of 98

97 betreiben. Des Weiteren dürfen auch nur von der IBM empfohlene Kabel angeschlossen werden. IBM übernimmt keine Verantwortung für die Einhaltung der Schutzanforderungen, wenn das Produkt ohne Zustimmung von IBM verändert bzw. wenn Erweiterungskomponenten von Fremdherstellern ohne Empfehlung von IBM gesteckt/eingebaut werden. EN Klasse A Geräte müssen mit folgendem Warnhinweis versehen werden: "Warnung: Dieses ist eine Einrichtung der Klasse A. Diese Einrichtung kann im Wohnbereich Funk-Störungen verursachen; in diesem Fall kann vom Betreiber verlangt werden, angemessene Maßnahmen zu ergreifen und dafür aufzukommen." Deutschland: Einhaltung des Gesetzes über die elektromagnetische Verträglichkeit von Geräten Dieses Produkt entspricht dem Gesetz über die elektromagnetische Verträglichkeit von Geräten (EMVG). Dies ist die Umsetzung der EU-Richtlinie 2004/108/EG in der Bundesrepublik Deutschland. Zulassungsbescheinigung laut dem Deutschen Gesetz über die elektromagnetische Verträglichkeit von Geräten (EMVG) (bzw. der EMC EG Richtlinie 2004/108/EG) für Geräte der Klasse A Dieses Gerät ist berechtigt, in Übereinstimmung mit dem Deutschen EMVG das EG-Konformitätszeichen - CE - zu führen. Verantwortlich für die Einhaltung der EMV Vorschriften ist der Hersteller: International Business Machines Corp. New Orchard Road Armonk, New York Tel: Der verantwortliche Ansprechpartner des Herstellers in der EU ist: IBM Deutschland GmbH Technical Regulations, Abteilung M456 IBM-Allee 1, Ehningen, Germany Tel: tjahn@de.ibm.com Generelle Informationen: Das Gerät erfüllt die Schutzanforderungen nach EN und EN Klasse A. Electromagnetic Interference (EMI) Statement - Russia Terms and conditions Permissions for the use of these publications is granted subject to the following terms and conditions. Personal Use: You may reproduce these publications for your personal, noncommercial use provided that all p. 97 of 98