TS7650 and TS7650G Problem Determination and Service Guide

Transcription

1 IBM System Storage TS7650 ProtecTIER Deduplication Solutions ProtecTIER version TS7650 and TS7650G Problem Determination and Service Guide GA

2 Note Before using this information and the product it supports, read the information in Safety and Environmental notices on page xi and Notices on page 327. This edition applies to ProtecTIER version of the TS7650 Appliance and the TS7650G and to all subsequent releases and modifications until otherwise indicated in new editions. This edition replaces GA Copyright IBM Corporation 2011, US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents Figures vii Tables ix Safety and Environmental notices... xi Safety notices xi Power cords xii Environmental notices xiii Homologation statement xv About this document xvii Terminology xvii Who should read this document xx What's new in this edition xx Getting information, help, and service..... xx Websites xx Help and service xxi Before you call for service xxii Getting help by telephone xxii Related publications xxiii TS7650 Appliance and TS7650G publications xxiii Server publications xxiv Remote Supervisor Adapter publications... xxiv System console publication xxiv Disk storage publications xxiv Tape device driver publications xxiv How to send your comments xxiv Chapter 1. Maintenance and troubleshooting Hardware ship group CDs Software ship group DVDs Overview of ProtecTIER Manager Available configurations Disk configuration Problem resolution Problem resolution considerations Problem resolution map Troubleshooting server problems Troubleshooting disk controller problems Troubleshooting disk expansion unit problems.. 9 Resolving preferred path critical events Remote support through Call Home Chapter 2. System troubleshooting tools Running command line tools ProtecTIER Service menu Using Dynamic System Analysis Problem Manager Accessing Problem Manager from the ProtecTIER Service menu Accessing Problem Manager from the command line System health monitoring System health monitoring command line tools.. 19 Call Home Troubleshooting the server-to-tssc connection 24 Call Home command line tools User-initiated Call Home Using SNMP traps Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) Component labeling Power, controls, and indicators Front view Rear view of the 3958 DD3/DD1 server Identifying problems using status LEDs Diagnostics General checkout procedure Diagnostic tools overview Power-on self-test error logs Diagnostic programs and error messages Light path diagnostics Server power features DD3/3958 DD1 server power checkout Ethernet controller troubleshooting Server power-on password override Passwords System recovery Testing a clustered system Syncing the system times Performing the system validation test Chapter 4. ProtecTIER server (3958 DD4, DD5, and AP1) Component labeling Power, controls, and indicators Front view Operator information panel Rear view Identifying problems using status LEDs Diagnostics General checkout procedure Diagnostic tools overview Power-on self-test error log Diagnostic programs and error messages Light path diagnostics Server power features ProtecTIER 3958 DD4, DD5, and AP1 server power checkout Ethernet controller troubleshooting Gb Ethernet adapter (Feature Code 3457) Server power-on password override I/O board jumpers Passwords Copyright IBM Corp. 2011, 2012 iii

4 System recovery Testing a clustered system Syncing the system times Performing the system validation test Chapter 5. TSSC TSSC connections for DD4, DD5, and AP1 servers 113 TSSC connections for DD1 and DD3 servers Chapter 6. Disk controller Verifying and upgrading the DS4700 storage system firmware, NVSRAM, and ESM Verifying and upgrading the storage system firmware, NVSRAM, and ESM (ProtecTIER version 2.3 and earlier) Verifying and upgrading the storage system firmware, NVSRAM, and ESM (ProtecTIER version 2.4 and later) Determining firmware levels for the DS3950 storage subsystem Method One Method Two Upgrading the firmware of the DS3950 storage subsystem Disk controller components Hot-swap drive bays Disk controller front indicators Disk controller rear indicators Disk controller Fibre Channel ports Accessing disk components through Storage Manager Removing SFP modules Operating the disk controller Disk controller speed setting Connecting disk expansion modules Chapter 7. Disk expansion units Checking the LEDs Disk expansion unit front LEDs Disk expansion unit rear LEDs Operating the disk expansion unit Disk expansion unit speed setting Chapter 8. CD and DVD overview Documentation CD RAS/BIOS and Firmware update DVD Disk component firmware Recovery disk Software CDs Chapter 9. Parts catalog Field-replaceable units unique to the 3958 DD4, 3958 DD5, and 3958 AP1 servers Field-replaceable units unique to the 3958 DD3 server Field-replaceable units unique to the 3958 DD1 server Chapter 10. FRU replacement for TS7600 systems Removing, replacing, and upgrading FRUs in the ProtecTIER server Preparing the system for FRU replacement Replacing the field-replaceable unit Updating the firmware of an Emulex or Qlogic adapter Updating the server system planar board firmware and basic input/output system (BIOS) settings Initializing a used hard disk drive for reuse Replacing the RAID controller in 3958 DD4, DD5, and AP1 servers Replacing the ServeRAID battery backup unit in 3958 DD4, DD5, and AP1 servers Replacing a Dual-Port or Quad-Port Ethernet card and recovering Ethernet settings Removing and replacing the microprocessor-board assembly or the I/O board Shuttle assembly Removing and replacing the RSA Recovering a node Automatic node recovery Removing and replacing Disk Storage controller FRUs Removing and replacing disk expansion unit FRUs 215 Removing and replacing TSSC FRUs Removing and replacing Ethernet switch FRUs Removing the Ethernet switch Reinstalling Ethernet switches Removing and replacing the Western Telematic Inc. switch Recovering the basic input/output system (BIOS) code Updating the server firmware and basic input/output system settings Updating the firmware of an MR10k MegaRAID or ServeRAID M5015 card Updating the SAS hard disk drive firmware Configuring the server Starting the Configuration/Setup Utility program Verifying cable connections Chapter 11. Power off and power on procedures Power procedures for ProtecTIER version 3.1 or later Powering off a stand-alone TS7650 server version 3.1 or later Powering off a TS7650 cluster version 3.1 or later Powering on a stand-alone TS7650 server version 3.1 or later Powering on a TS7650 cluster version 3.1 or later Power procedures for ProtecTIER version 2.5 or earlier iv TS7650 and TS7650G Problem Determination and Service Guide

5 Powering off the ProtecTIER server version 2.5 or earlier Powering on the ProtecTIER server version 2.5 or earlier Performing an emergency shutdown Additional power information Chapter 12. End-of-call procedure Appendix A. Logging into the 3958 DD1/3958 DD3 server through the RSA connection Enabling remote control Configuring remote supervisor adapter keystrokes 260 Appendix B. Logging into the 3958 DD4, DD5, and AP1 servers through the IMM web interface Enabling remote control Updating IMM firmware using the web interface 267 Appendix C. Checkpoint codes Appendix D. Manually upgrading the DS4700 storage system firmware (ProtecTIER version 2.4 and later) Appendix E. ProtecTIER Network Performance Validation Utility Appendix F. ProtecTIER Replication Network Performance Validation Utility for VTL Systems Appendix G. ProtecTIER Network Performance Validation Utility for OpenStorage Systems Appendix H. Checking the Western Telematic Inc. power ports Appendix J. Overview of basic Linux commands Appendix K. Worldwide time zone codes Accessibility for publications and ProtecTIER Manager About the Windows-based accessibility features 315 About the Java-based tools Installing the Java Runtime Environment Installing the Java Access Bridge Using a screen reader to install ProtecTIER Manager Enabling the Windows High Contrast option Using the Windows high contrast scheme with ProtecTIER Manager Customizing the color palette Notices Trademarks Electronic emission notices Federal Communications Commission statement 329 Industry Canada compliance statement European Union Electromagnetic Compatibility Directive Australia and New Zealand Class A Statement 330 Germany Electromagnetic compatibility directive People's Republic of China Class A Electronic Emission statement Taiwan Class A compliance statement Taiwan contact information Japan Voluntary Control Council for Interference (VCCI) Class A Statement Japan Electronics and Information Technology Industries Association (JEITA) Statement Korean Communications Commission (KCC) Class A Statement Russia Electromagnetic Interference (EMI) Class A Statement Index Appendix I. Configuring the Western Telematic Inc. switch Contents v

6 vi TS7650 and TS7650G Problem Determination and Service Guide

7 Figures 1. ProtecTIER Service menu ProtecTIER system Health Monitoring menu Canceling a problem Displaying status of a hard drive Displaying status of a power supply and fan Listing all possible checks Listing status of all health point checks Performing a specific check immediately SNMP trap report Server front view Server operator information panel Server rear view RSA controls, connectors, and indicators Power supply rear view Server rear view RSA controls, connectors, and indicators eserver logo screen for basic input/output system (BIOS) setup Operator information panel Server light path diagnostics panel Ethernet controller LEDs Jumpers on I/O board (3958 DD3) Stopping cluster services Alerts Log Server front view Operator information panel Server rear view Server LEDs rear view Power supply rear view Server rear view eserver logo screen for basic input/output system (BIOS) setup Operator information panel Server light path diagnostics panel Ethernet controller LEDs Jumpers on I/O board Alerts Log Stand-alone TS7650G (3958 DD4/3958 DD5) or TS7650 Appliance (3958 DD4) attachment to the TSSC (x3250 M3 MT4252) Clustered TS7650G (3958 DD4/3958 DD5) or TS7650 Appliance (3958 DD4) attachment to the TSSC (x3250 M3 MT4252) Stand-alone TS7650 Appliance (3958 DD4) attachment to the TSSC (x3250 M4 MT 2538) Clustered TS7650 Appliance (3958 DD4) attachment to the TSSC (x3250 M4 MT 2538) Stand-alone TS7650G (3958 DD5) attachment to the TSSC (x3250 M4 MT 2538) Clustered TS7650G (3958 DD5) attachment to the TSSC (x3250 M4 MT 2538) Stand-alone TS7650 Appliance (3958 DD3) attachment to the TSSC Clustered TS7650 Appliance (3958 DD3) attachment to the TSSC Stand-alone TS7650G (3958 DD1/3958 DD3) attachment to the TSSC Clustered TS7650G (3958 DD1/3958 DD3) attachment to the TSSC Disk controller hot-swap drive bays Disk controller front indicators DS4700 disk controller rear indicators DS3950 disk controller rear indicators Battery unit LEDs Numeric display LEDs Disk controller seven-segment enclosure ID DS4700 disk controller Fibre Channel ports DS3950 disk controller Fibre Channel ports Storage Manager: Running controller diagnostics Storage Manager: Downloading firmware Storage Manager: Executing scripts Unlocking the SFP module latch - plastic variety Disk expansion unit front LEDs Disk expansion unit rear LEDs Power supply and fan unit LEDs Numeric display LEDs Example output of versions check WWNN main menu Display Names By Port menu Change Name By Port menu Change Name By Port menu (updated) Main Menu for Qlogic adapter command line interface HBA Information Menu HBA Instance submenu HBA information display for a port Removing the top cover Removing the top cover bracket Extracting the RAID adapter carrier Removing the RAID adapter from the carrier Removing the battery carrier card from the adapter SAS ports on the ServeRAID M5015 controller SAS ports on the ServeRAID BR10i controller Inserting the RAID adapter carrier Removing the top cover Removing the top cover bracket Extracting the RAID adapter carrier Removing the RAID adapter from the carrier Removing the battery carrier card from the adapter SAS ports on the ServeRAID M5015 controller Inserting the RAID adapter carrier Product label on server RSA II settings Login Profiles window Login Profile dialog box RSA event log Copyright IBM Corp. 2011, 2012 vii

8 92. TS3000 System Console - CD Copy to Console window Removing the Ethernet switch (3958 AP1 installations) Installing the Ethernet switch (3958 AP1 installations) Ethernet switch power cable routing into the frame Rear of rack, showing installation of the Ethernet switches Ethernet switch power cable routing through the mounting rail ProtecTIER Configuration menu IP Network configuration menu On-screen instructions for Ethernet switch configuration Clustered power connections Clustered 1 Gb Ethernet switch Ethernet connections (3958 DD1/3958 DD3) Clustered 1 Gb Ethernet switch connections for VTL configuration Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.1.8 or earlier) Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.2) Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code System x logo screen for BIOS setup Server operator information panel Server operator information panel Configured GFS mountpoints Active GFS mountpoints Service laptop to RSA connection Local Area Connection Properties Local Area Connection Properties: Internet Protocol (TCP/IP) Remote Supervisor Adapter II Refresh 1 Welcome window Remote Supervisor Adapter II Refresh 1 window: Remote Control link Maximize video speed ASM Remote Control preferences Window ASM Remote Console Preferences window Service laptop to IMM connection Local Area Connection Properties Local Area Connection Properties: Internet Protocol (TCP/IP) Integrated Management Module Welcome window Integrated Management Module window: Remote Control link Dual node cluster cabling for 3958 AP1 and 3958 DD Network power switch connections Ethernet cable connections in the VTL configuration of two clustered 3958 DD4/3958 DD5 servers Power connections to the WTI switch for clustered 3958 DD4 and 3958 DD5 servers (TS7650 Appliance shown) Plug parameters Display tab Settings for High Contrast ProtecTIER Manager window Preferences dialog box Normal contrast versus high contrast Color selection, Swatches tab Default color versus custom color viii TS7650 and TS7650G Problem Determination and Service Guide

9 Tables 1. IBM websites for help, services, and information xxi 2. Remote support capabilities ProtecTIER software Call Home events Power supply indicators Power-supply LED combinations LEDs on the 3958 DD1/3958 DD3 light path diagnostics panel Disk controller front indicators Power supply indicators Power supply LED combinations LEDs on the light path diagnostics panel Quad-port Ethernet LED indicators I/O board jumper blocks Stand-alone and clustered TSSC, KVM, and customer network Ethernet connections for the TS7650 Appliance and TS7650G Disk controller front indicators DS4700 disk controller rear indicators DS3950 disk controller rear indicators Host and drive channel LED definitions Battery unit LEDs Numeric display diagnostic codes Disk controller seven-segment enclosure ID Disk controller Fibre Channel ports Back-end storage controllers LED colors Disk expansion unit front LEDs Disk expansion unit rear LEDs Power supply and fan unit LEDs Numeric display diagnostic codes Symptom-to-FRU index DD4, DD5, and AP1 server FRUs Additional FRUs for feature code 3447, Cluster Connection Kit DD3 server FRUs Additional FRUs for feature code 3447, Cluster Connection Kit with Alpha Networks switch Additional FRUs for feature code 3437, Cluster Connection Kit with SMC switch DD1 server FRUs Additional FRUs for feature code 3447, Cluster Connection Kit with Alpha Networks switch Ethernet switch reinstallation procedures Clustered 1 Gb Ethernet switch Ethernet connections for VTL configuration Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.1.8 or earlier) Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.2) Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code Clustered 1 Gb Ethernet switch Ethernet connections for VTL configuration Clustered DD4 and DD5 servers to WTI switch power cabling Basic Linux commands ProtecTIER device-specific Linux commands and directories Copyright IBM Corp. 2011, 2012 ix

10 x TS7650 and TS7650G Problem Determination and Service Guide

11 Safety and Environmental notices This section contains information about safety notices that are used in this guide and environmental notices for this product. Safety notices Observe the safety notices when using this product. These safety notices contain danger and caution notices. These notices are sometimes accompanied by symbols that represent the severity of the safety condition. Most danger or caution notices contain a reference number (Dxxx or Cxxx). Use the reference number to check the translation in the IBM Systems Safety Notices, G manual. The sections that follow define each type of safety notice and give examples. Danger notice A danger notice calls attention to a situation that is potentially lethal or extremely hazardous to people. A lightning bolt symbol always accompanies a danger notice to represent a dangerous electrical condition. A sample danger notice follows: DANGER: An electrical outlet that is not correctly wired could place hazardous voltage on metal parts of the system or the devices that attach to the system. It is the responsibility of the customer to ensure that the outlet is correctly wired and grounded to prevent an electrical shock. (D004) Caution notice A caution notice calls attention to a situation that is potentially hazardous to people because of some existing condition, or to a potentially dangerous situation that might develop because of some unsafe practice. A caution notice can be accompanied by one of several symbols: If the symbol is... It means... A generally hazardous condition not represented by other safety symbols. This product contains a Class II laser. Do not stare into the beam. (C029) Laser symbols are always accompanied by the classification of the laser as defined by the U. S. Department of Health and Human Services (for example, Class I, Class II, and so forth). A hazardous condition due to mechanical movement in or around the product. Copyright IBM Corp. 2011, 2012 xi

12 If the symbol is... It means... This part or unit is heavy but has a weight smaller than 18 kg (39.7 lb). Use care when lifting, removing, or installing this part or unit. (C008) Sample caution notices follow: Caution The battery is a lithium ion battery. To avoid possible explosion, do not burn. Exchange only with the IBM-approved part. Recycle or discard the battery as instructed by local regulations. In the United States, IBM has a process for the collection of this battery. For information, call Have the IBM part number for the battery unit available when you call. (C007) Caution The system contains circuit cards, assemblies, or both that contain lead solder. To avoid the release of lead (Pb) into the environment, do not burn. Discard the circuit card as instructed by local regulations. (C014) Caution When removing the Modular Refrigeration Unit (MRU), immediately remove any oil residue from the MRU support shelf, floor, and any other area to prevent injuries because of slips or falls. Do not use refrigerant lines or connectors to lift, move, or remove the MRU. Use handholds as instructed by service procedures. (C016) Caution Do not connect an IBM control unit directly to a public optical network. The customer must use an additional connectivity device between an IBM control unit optical adapter (that is, fibre, ESCON, FICON ) and an external public network. Use a device such as a patch panel, a router, or a switch. You do not need an additional connectivity device for optical fibre connectivity that does not pass through a public network. Power cords For your safety, IBM provides a power cord with a grounded attachment plug to use with this IBM product. To avoid electrical shock, always use the power cord and plug with a properly grounded outlet. IBM power cords used in the United States and Canada are listed by Underwriter s Laboratories (UL) and certified by the Canadian Standards Association (CSA). For units intended to be operated at 115 volts: Use a UL-listed and CSA-certified cord set consisting of a minimum 18 AWG, Type SVT or SJT, three-conductor cord, a maximum of 15 feet in length and a parallel blade, grounding-type attachment plug rated 15 amperes, 125 volts. For units intended to be operated at 230 volts (U.S. use): Use a UL-listed and CSA-certified cord set consisting of a minimum 18 AWG, Type SVT or SJT, xii TS7650 and TS7650G Problem Determination and Service Guide

13 Environmental notices three-conductor cord, a maximum of 15 feet in length and a tandem blade, grounding-type attachment plug rated 15 amperes, 250 volts. For units intended to be operated at 230 volts (outside the U.S.): Use a cord set with a grounding-type attachment plug. The cord set should have the appropriate safety approvals for the country in which the equipment will be installed. IBM power cords for a specific country or region are usually available only in that country or region. The environmental notices that apply to this product are provided in the Environmental Notices and User Guide, Z xx manual. A copy of this manual is located on the publications CD. Safety and Environmental notices xiii

14 xiv TS7650 and TS7650G Problem Determination and Service Guide

15 Homologation statement Attention: This product is not intended to be connected directly or indirectly by any means whatsoever to interfaces of public telecommunications networks, neither to be used in a Public Services Network. Copyright IBM Corp. 2011, 2012 xv

16 xvi TS7650 and TS7650G Problem Determination and Service Guide

17 About this document This document provides problem determination information for the IBM System Storage TS7650 ProtecTIER Deduplication Appliance (TS7650 Appliance) and the IBM System Storage TS7650G ProtecTIER Deduplication Gateway (TS7650G). Note: Cache modules and cache controllers are supported separately within the TS7650G. If the problem is known to be in the IBM attached storage component, select the hardware option and enter the appropriate Machine Type and serial number for the component. If the attached storage is not IBM branded, contact the appropriate service provider for the component. For the TS7650 Appliance, all components inside the frame are supported under the 3958 AP1 MTM. Terminology IBM offers two virtualization solutions: TS7650 When used alone, this term signifies IBM's family of virtualization solutions that operate on the ProtecTIER platform. TS7650 Appliance or appliance These are terms for IBM's self-contained virtualization solution from the TS7650 family that includes a disk storage repository. The TS7650 Appliance consists of the following: Server The 3958 AP1 server is based on the IBM System x3850 X5 Type 7145-PBR at the ProtecTIER version 3.2 release. When used as a server in the TS7650 Appliance, its machine type and model are 3958 AP1. Use this machine type and model for service purposes. System console The system console is a TS3000 System Console (TSSC). This document uses the terms system console and TSSC interchangeably. Disk controller The disk controller for the TS7650 Appliance is an IBM Feature Code 3708: 4.8 TB Fibre Channel Disk Controller. Use this feature code for service purposes. Disk expansion unit The disk expansion unit for the TS7650 Appliance is an IBM Feature Code 3707: 4.8 TB Fibre Channel Disk Expansion Unit. Use this feature code for service purposes. IBM Tivoli Assist On-site (AOS) IBM Tivoli Assist On-site (AOS) is a web-based tool that enables a remote support representative in IBM to view or control the management node desktop. More information is located at the Tivoli AOS website. TS7650G or Gateway These are terms for IBM's virtualization solution from the TS7650 family that does not include a disk storage repository, allowing the customer to choose from a variety of storage options. IBM does support two TS7650 Gateway servers in a single frame (a two-node cluster). The TS7650G consists of the following: Copyright IBM Corp. 2011, 2012 xvii

18 Server There are four types of server that have been used in the Gateway: 3958 DD5 This is a newer, higher performance server available in May This server is based on the IBM System x 7145 model. When used as a server in the TS7650G, its machine type and model are 3958 DD5. Use this machine type and model for service purposes DD4 This is a higher performance server available in December This server is based on the IBM System x3850 X5 Type 7145-PBR. When used as a server in the TS7650G, its machine type and model are 3958 DD4. Use this machine type and model for service purposes DD3 This is a higher performance server available in March This server is based on the IBM System x3850 M2 Type When used as a server in the TS7650G, its machine type and model are 3958 DD3. Use this machine type and model for service purposes DD1 This is the original server available in August This server is based on the IBM System x3850 M2 Type When used as a server in the TS7650G, its machine type and model are 3958 DD1. Use this machine type and model for service purposes. System console The system console is a TS3000 System Console (TSSC). This document uses the terms system console and TSSC interchangeably. Under IBM best practices, the TS7650G also contains the following: Disk controller The customer must choose the disk controller for use with the TS7650G. A list of compatible controllers is located at the IBM Tape Systems Resource Library website in the TS7650/TS7650G ISV and interoperability matrix document. Disk expansion unit The customer must choose the disk expansion unit for use with the TS7650G. A list of compatible expansion units is located at the IBM Tape Systems Resource Library website in the TS7650/TS7650G ISV and interoperability matrix document. IBM Tivoli Assist On-site (AOS) IBM Tivoli Assist On-site (AOS) is a web-based tool that enables a remote support representative in IBM to view or control the management node desktop. More information is located at the Tivoli AOS website. OpenStorage OpenStorage allows ProtecTIER to be integrated with NetBackup to provide the means for backup-to-disk without using a virtual tape library (VTL) emulation. Using a plug-in that is installed on an OpenStorage-enabled media server, ProtecTIER can implement a communication protocol that supports data transfer and control between xviii TS7650 and TS7650G Problem Determination and Service Guide

19 the backup server and the ProtecTIER server. Therefore, to support the plug-in, ProtecTIER implements a storage server emulation. replication A process that transfers logical objects like cartridges from one ProtecTIER repository to another. The replication function allows ProtecTIER deployment to be distributed across sites. Each site has a single or clustered ProtecTIER environment. Each ProtecTIER environment has at least one ProtecTIER server. The ProtecTIER server that is a part of the replication grid has two dedicated replication ports that are used for replication. Replication ports are connected to the customer's WAN and are configured on two subnets as default. replication grid A set of repositories that share a common ID and can potentially transmit and receive logical objects through replication. A replication grid defines a set of ProtecTIER repositories and actions between them and is configured using the ProtecTIER Replication Manager. The ProtecTIER Replication Manager is a software component that is installed on a ProtecTIER server or a dedicated host. The ProtecTIER Replication Manager should be able to recognize all the members of the entire network the ProtecTIER Replication Manager handles on both replication subnets. The ProtecTIER Replication Manager manages the configuration of multiple replication grids in an organization. An agent on every node in each ProtecTIER server interacts with the server and maintains a table of its grid members. Note: Customers must license the Replication features on all ProtecTIER systems participating in the replication grid whether the system is sending or receiving data (or both). replication grid ID A number from 0 to 63 that identifies a replication grid within an organization. replication grid member A repository that is a member in a replication grid. replication pairs Two repositories within a replication grid that replicate from one to another. replication policy A policy made up of rules that define a set of objects (for example, VTL cartridges) from a source repository to be replicated to a target repository. repository unique ID (RID) A number that uniquely identifies the repository. The RID is created from the replication grid ID and the repository internal ID in the grid. replication timeframe A scheduled period of time for replication to take place for all policies. shelf A container of VTL cartridges within a ProtecTIER repository. virtual tape library (VTL) The ProtecTIER virtual tape library (VTL) service emulates traditional tape libraries. By emulating tape libraries, ProtecTIER VTL enables you to transition to disk backup without having to replace your entire backup environment. Your existing backup application can access virtual robots to move virtual cartridges between virtual slots and drives. The backup About this document xix

20 application perceives that the data is being stored on cartridges while ProtecTIER actually stores data on a deduplicated disk repository. visibility switching The automated process that transfers the visibility of a VTL cartridge from its master to its replica and vice versa. The visibility switching process is triggered by moving a cartridge to the source library Import/Export (I/E) slot. The cartridge will then disappear from the I/E slot and appear at the destination library's I/E slot. To move the cartridge back to the source library, the cartridge must be ejected to the shelf from the destination library. The cartridge will then disappear from the destination library and reappear at the source I/E slot. Who should read this document What's new in this edition This publication is for IBM service personnel who are installing, diagnosing, or repairing the TS7650 Appliance or the TS7650G. This publication is intended for use by IBM service personnel only. Technical changes occurring in this edition are identified with a vertical bar ( )in the left hand margin of the page. The following is new for the version PGA release: v TS3000 System Console cabling diagrams, see TSSC connections for DD4, DD5, and AP1 servers on page 113 Getting information, help, and service If you need help, service, technical assistance, or just want more information about IBM products, you will find a wide variety of sources available from IBM to assist you. Available services, telephone numbers, and web links are subject to change without notice. Websites IBM maintains pages on the World Wide Web where you can get the most up-to-date information about your product, including documentation and the most recent downloads. Be sure to visit the support page for the TS7650 Appliance or the TS7650G, complete with FAQs, parts information, technical hints and tips, technical publications, and downloadable files, if applicable. The support pages are available at IBM Tape systems. The translated publications for this product are included with the product. These documents and product specification sheets are also available from the IBM Support Portal. For information about the TS7650, refer to the following sites: v IBM TotalStorage Tape Storage Systems v IBM System Storage ProtecTIER TS7610 and TS7650 Combined Service Information Center v IBM System Storage ProtecTIER TS7650 Customer Information Center xx TS7650 and TS7650G Problem Determination and Service Guide

21 v IBM Tape Systems Resource Library For additional websites, see Table 1. Table 1. IBM websites for help, services, and information Description Web address (URL) IBM Support Portal IBM home page Directory of worldwide contacts IBM Publications Center DS4000 information Independent Software Vendor (ISV) support TS7650/TS7650G ISV and interoperability matrix Information about SAN switches and directors Information about IBM xseries products, services, and support DS4000 Interoperability Matrix Firmware and software downloads, as well as associated driver code Accessibility information Product recycling programs servlet/pbi.wss disk.html TS7650_Support_Matrix.pdf interop-matrix.html index.html recycling.shtml Help and service You can call 1 (800) IBM SERV for help and service if you are in the U.S. or Canada. You must choose the software or hardware option when calling for assistance. Note: This product is equipped with a Software Call Home feature. When enabled, it notifies IBM Service of software error events. Not all countries currently support this feature. Contact your next level of support for more information. The Software Call Home feature is supported in all EMEA/CEEMEA countries. Choose the software option if you are uncertain if the problem involves TS7650 software or TS7650 hardware. Choose the hardware option only if you are certain the problem solely involves the TS7650 hardware. When calling IBM for service regarding the TS7650, follow these guidelines for the software and hardware options: Software option Identify the TS7650 as your product and supply your customer number as About this document xxi

22 proof of purchase. The 7-digit customer number ( to ) is assigned by IBM when the PID is purchased. It should be located on the customer information worksheet or on the invoice from the software purchase. Hardware option Provide the serial number and appropriate 4-digit Machine Type for the hardware component that displays a problem (for example: 3958 DD1, 3958 DD3, 3958 DD4, 3958 DD5, or 3958 AP1). Note: Cache modules and cache controllers are supported separately within the TS7650G. If the problem is known to be in the IBM attached storage component, select the hardware option and enter the appropriate Machine Type and serial number for the component. If the attached storage is not IBM branded, contact the appropriate service provider for the component. For the TS7650 Appliance, all components inside the frame are supported under the 3958 AP1 MTM. Before you call for service Some problems can be solved without outside assistance, by using the online help, by looking in the online or printed documentation that comes with the unit, or by consulting the support Web page. Also, be sure to read the information in any README files and release notes. Getting help by telephone With the original purchase of the TS7650 Appliance or the TS7650G, you have access to extensive support coverage. During the product warranty period, you can call the IBM Support Center ( in the U.S.) for product assistance covered under the terms of the hardware IBM warranty or the software maintenance contract that comes with product purchase. Have the following information ready when you call: v IBM TS7650 Appliance or TS7650G software identifier, or the machine type and model. The software identifier can be either the product name or the Product Identification (PID) number. v Serial numbers of the TS7650 Appliance or TS7650G components, or your proof of purchase. v Description of the problem. v Exact wording of any error messages. v Hardware and software configuration information If possible, have access to your computer when you call. In the U.S. and Canada, these services are available 24 hours a day, 7 days a week. In the U.K., these services are available Monday through Friday, from 9:00 a.m. to 6:00 p.m. In all other countries, contact your IBM reseller or IBM marketing representative. When calling IBM for support for the TS7650 Appliance or TS7650G, follow these guidelines: v If you are certain the problem involves the TS7650 Appliance or TS7650G software, or if you are uncertain whether the problem involves the software or xxii TS7650 and TS7650G Problem Determination and Service Guide

23 v hardware, choose the software option. Then identify the TS7650 Appliance or TS7650G as your product and supply your customer number as proof of purchase Choose the hardware option only if you are certain the problem involves solely the hardware. After you select hardware, provide the serial number and appropriate 4-digit Machine Type for the hardware component that displays a problem (for example: 3958 DD1, 3958 DD3, 3958 DD4, 3958 DD5, or 3958 AP1). Cache modules and cache controllers are supported separately within the TS7650G. If the problem is known to be in the IBM attached storage component, select the hardware option and enter the appropriate Machine Type and serial number for the component. If the attached storage is not IBM branded, contact the appropriate service provider for the component. For the TS7650 Appliance, all components inside the frame are supported under the 3958 AP1 MTM. Related publications Note: US or Canadian Customers calling 1 (800) IBM SERV are asked to select a hardware or software option. Unless you are certain the problem involves the hardware, choose the software option. The following documents provide information about the TS7650 Appliance and TS7650G components and related hardware. TS7650 Appliance and TS7650G publications This topic lists TS7650 Appliance and TS7650G publications. Publications common to both the TS7650 Appliance and the TS7650G v IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA v IBM System Storage Problem Determination and Service Guide for the TS7650 ProtecTIER Deduplication Appliance and TS7650G ProtecTIER Deduplication Gateway, GA v IBM System Storage TS7650 ProtecTIER Software Upgrade Guide, SC v IBM ProtecTIER Implementation and Best Practices, IBM Redbooks publication SG v IBM System Storage Labeling Instructions for the TS7650 ProtecTIER Deduplication Appliance and TS7650G ProtecTIER Deduplication Gateway, PN 95P8942 For use only with V2.3 and earlier: v IBM System Storage TS7600 with ProtecTIER Installation Instructions for the RAS Package, BIOS, and Firmware updates following a FRU replacement for models 3958 DD1, 3958 DD3, and 3958 AP1, PN 46X2459 TS7650 Appliance-only publications v IBM System Storage TS7650 ProtecTIER Deduplication Appliance and TS7650G Deduplication Gateway Introduction and Planning Guide, GA v IBM System Storage TS7650 ProtecTIER Deduplication Appliance Installation Roadmap Guide, GA TS7650G-only publications v IBM System Storage TS7650G ProtecTIER Deduplication Gateway Installation Roadmap Guide, GA About this document xxiii

24 Server publications v v v v IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide IBM System x3850 M2 and System x3950 M2 Type 7141 and 7233 User's Guide IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Installation and User's Guide IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide Remote Supervisor Adapter publications v v Remote Supervisor Adapter II Slimline and Remote Supervisor Adapter II Installation Guide Remote Supervisor Adapter II Slimline and Remote Supervisor Adapter II User's Guide System console publication v IBM System Storage TS3000 System Console (TSSC) Maintenance Information Disk storage publications v v v IBM System Storage DS4700 Express Storage Subsystem Installation, User's and Maintenance Guide IBM TotalStorage DS4000 EXP810 Storage Expansion Unit Installation, User's and Maintenance Guide IBM TotalStorage DS4000 Storage Manager Version 10 Installation and Host Support Guide for Windows 2000/Server 2003, NetWare, ESX Server, and Linux Tape device driver publications How to send your comments v IBM Tape Device Drivers Installation and User's Guide, GC Your feedback is important in helping to provide the most accurate and highest quality information. To submit any comments about this book or any other IBM System Storage TS7600 with ProtecTIER documentation: v Send your comments by to starpubs@us.ibm.com. Be sure to include the following information: Exact publication title and version Publication form number (for example, GC ) Page, table, or illustration numbers that you are commenting on with a detailed description of any information that should be changed xxiv TS7650 and TS7650G Problem Determination and Service Guide

25 Chapter 1. Maintenance and troubleshooting Hardware ship group CDs This section is the starting point for maintenance, troubleshooting, or repair of a TS7650 Appliance or a TS7650G. The hardware ship group for the TS7650 Appliance and the TS7650G includes the following CDs. IBM System Storage TS7650 with ProtecTIER Publications CD The IBM System Storage TS7650 with ProtecTIER Publications CD contains the following service and customer documentation for the TS7650 and TS7650G: v IBM System Storage TS7650 ProtecTIER Deduplication Appliance and TS7650G Deduplication Gateway Introduction and Planning Guide, GA v IBM System Storage TS7650 ProtecTIER Deduplication Appliance Installation Roadmap Guide, GA v IBM System Storage TS7650G ProtecTIER Deduplication Gateway Installation Roadmap Guide, GA v IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA v IBM System Storage Problem Determination and Service Guide for the TS7650 ProtecTIER Deduplication Appliance and TS7650G ProtecTIER Deduplication Gateway, GA v IBM System Storage TS7650 ProtecTIER Software Upgrade Guide, SC v IBM System Storage Labeling Instructions for the TS7650 ProtecTIER Deduplication Appliance and TS7650G ProtecTIER Deduplication Gateway, PN 95P8942 v IBM System Storage TS3000 System Console (TSSC) Maintenance Information v Statements of limited warranty Recovery Disk This disk contains the Red Hat Linux operating system and a kick-start file containing scripts to automate the operating system installation process. The customer can use this disk to reinstall the operating system on the server during a recovery scenario. Copyright IBM Corp. 2011,

26 Software ship group DVDs As of version 3.2, the TS7650 software ship group includes the following DVDs. IBM System Storage ProtecTIER Enterprise Edition V3.2 DVD The IBM System Storage ProtecTIER Enterprise Edition V3.2 DVD contains the software for the TS7650G server that runs on the Red Hat Linux operating system installed on the server. The server uses the software to present the attached disk storage to host systems as virtual tape and to perform other functions such as data deduplication. IBM System Storage ProtecTIER Appliance Edition V3.2 DVD The IBM System Storage ProtecTIER Appliance Edition V3.2 DVD contains the software for the TS7650 Appliance that runs on the Red Hat Linux operating system installed on the appliance. The appliance uses the software to present the attached disk storage to host systems as virtual tape and to perform other functions such as data deduplication. IBM System Storage ProtecTIER Manager V3.2 DVD The IBM System Storage ProtecTIER Manager V3.2 DVD contains the files required to install the ProtecTIER Manager graphical user interface on workstations connected to the TS7650G or TS7650 Appliance through a customer's Ethernet network. ProtecTIER Manager allows the user to manage the virtual tape presented to host systems by the server. Overview of ProtecTIER Manager Available configurations ProtecTIER Manager is the primary interface to the TS7650 Appliance and the TS7650G. ProtecTIER Manager provides the Ethernet connection to the server Ethernet and RSA ports. It also has the client/server application software for managing the TS7650 Appliance and the TS7650G. Refer to the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA , for installation and setup information. The TSSC can run ProtecTIER Manager for the IBM Service Representative. Note: The optimal screen resolution for the ProtecTIER Manager is 1280 x Both the TS7650 Appliance and the TS7650G are available in a stand-alone configuration (with one server) and a clustered configuration (with two servers). For more information regarding the setup of these configurations, see the section titled Configurations in the IBM System Storage TS7650 ProtecTIER Deduplication Appliance Installation Roadmap Guide, GA , or the IBM System Storage TS7650G ProtecTIER Deduplication Gateway Installation Roadmap Guide, GA Disk configuration The server supports three RAID configurations. Configuration RAID 1 RAID 5 RAID 10 Application Server internal hard disk drives User data on attached disk storage Metadata on attached disk storage 2 TS7650 and TS7650G Problem Determination and Service Guide

27 Problem resolution Configure other items such as disk storage block size and channel protection, in accordance with the instructions provided by your disk storage manufacturer. This section provides problem resolution procedures. Important: For any disaster recovery situations, contact your next level of support. Review the information provided in Problem resolution considerations, and then continue with Problem resolution map on page 5 to determine the appropriate problem resolution procedure. If there is a problem with the RSA, see Appendix A, Logging into the 3958 DD1/3958 DD3 server through the RSA connection, on page 255 and Chapter 2, System troubleshooting tools, on page 15. Problem resolution considerations Because of the variety of hardware and software combinations that can be encountered, use the following information to assist you in problem determination. If possible, have this information available when requesting assistance from Service Support and Engineering functions. h h Machine type and model (see Terminology on page xvii) Microprocessor or hard disk upgrades Failure symptom h Do diagnostics fail? h What message appears in the diagnostics log? h h h h h What, when, where, single, or multiple systems? Is the failure repeatable? Has this configuration ever worked? If it has been working, what changes were made prior to it failing? Is this the original reported failure? Diagnostics version h Type and version level Hardware configuration h Print (print screen) configuration currently in use Chapter 1. Maintenance and troubleshooting 3

28 h BIOS level Operating system software h Type and version level Comparing the configuration and software setup between working and non-working systems often leads to problem resolution. Note: To eliminate confusion, systems are considered identical only if they are exactly alike in all the following criteria: v Machine type and models v BIOS level v Adapters/attachments in the same locations v Address jumpers/terminators/cabling v Software versions and levels v Diagnostics code (version) v Configuration options set in the system v Setup for the operation system control files LC wrap plugs are required to run the loopback test at the host bus adapter or at the end of cables. The part numbers for the wrap plugs are as follows: v 24P0950 (wrap connector and coupler kit) 11P3847 (wrap connector) 05N6766 (coupler) The following is a list of general symptoms that can be encountered in your external disk storage devices: v RAID controller passive v Failed or moved cluster resource v Startup long delay v Systems Management or Storage Manager performance problems In all cases, refer to the manufacturer's documentation provided with the disk storage devices. 4 TS7650 and TS7650G Problem Determination and Service Guide

29 Problem resolution map Answer the following questions to determine the correct problem resolution procedure to perform. 1. Has the customer-set power-on password been forgotten? v Yes, go to Server power-on password override on page 65. v No, go to step Has a problem been reported against the server? v Yes, go to Troubleshooting server problems on page 6. v No, go to step Has a problem been reported against the preferred path? v Yes, go to Resolving preferred path critical events on page 10. v No, go to step Has a problem been reported against a disk controller? v Yes, go to Troubleshooting disk controller problems on page 8. v No, go to step Has a problem been reported against a disk expansion unit? v Yes, go to Troubleshooting disk expansion unit problems on page 9. v No, go to step Has a problem been reported against an Ethernet switch? v Yes, go to Removing and replacing Ethernet switch FRUs on page 216. v No, go to step Has a problem been reported against a two-port Ethernet or fibre channel PCI Express adapter? v Yes, refer to Field-replaceable units unique to the 3958 DD3 server on page 163 or Field-replaceable units unique to the 3958 DD1 server on page 165. v No, go to step Has a problem been reported with a code update or load? v Yes, restart the system. If the problem persists, see Chapter 2, System troubleshooting tools, on page 15 and contact your next level of support. v No, go to step Is there a problem with an MES or a feature? v Yes, go to the MES or feature documentation provided. Review the installation procedure. v No, return to Chapter 1, Maintenance and troubleshooting, on page 1; a wrong path has been followed. Chapter 1. Maintenance and troubleshooting 5

30 Troubleshooting server problems You can troubleshoot problems with the server by answering a series of questions. Before you begin Attention: Before you proceed, become familiar with the server by reviewing the following topics. v For 3958 DD3 or 3958 DD1: Power, controls, and indicators on page 38 Front view on page 38 Rear view of the 3958 DD3/DD1 server on page 40 v For 3958 DD5 or 3958 DD4: Power, controls, and indicators on page 73 Front view on page 74 Rear view on page 77 You can also use IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide or IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide as a guide for errors and topics related to X System. Procedure To troubleshoot problems with the server, answer the following questions. 1. Is the server power-on light-emitting diode (LED) (see Figure 12 on page 40) solid and green? v Yes, go to step 2. v No, go to Power supply error LEDs on page 43 to determine the status of the power supply. Also, follow the directions in the note at the bottom of the table, if necessary. If you suspect a power supply problem, see 3958 DD3/3958 DD1 server power checkout on page 63 or ProtecTIER 3958 DD4, DD5, and AP1 server power checkout on page Is the server system-error LED (see Figure 11 on page 39) lit? v No, go to step 5 on page 7. v Yes, do the following: a. To determine which area of the server has the error, refer to the following sections. v Light path diagnostics on page 54 v For 3958 DD3 or 3958 DD1 servers: Light path diagnostic light-emitting diodes on page 56 Component labeling on page 37 v For 3958 DD5 or 3958 DD4 servers: Light path diagnostic light-emitting diodes on page 93 Component labeling on page 73 b. For remote supervisor adapter (RSA) failure, see the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. To log in to the RSA, see Appendix A, Logging into the 3958 DD1/3958 DD3 server through the RSA connection, on page TS7650 and TS7650G Problem Determination and Service Guide

31 c. For system-management information control of the 3958 DD5 or 3958 DD4 over the network, use the SYS MGMT Ethernet port on the rear of the server for web access. This Ethernet connector is used only by the Integrated Management Module (IMM). To log into the IMM, see Appendix B, Logging into the 3958 DD4, DD5, and AP1 servers through the IMM web interface, on page 261. d. After you identify the LEDs on the server, see Identifying problems using status LEDs on page 46 for removal and replacement procedures. e. Continue with step Refer to Starting the diagnostic programs on page 50 to run diagnostics. a. Ask the customer to stop all activity to the specific server. b. Refer to Server power features on page 63 to cycle the power. c. At the boot screen, press F2 to run the diagnostics. d. Follow the on-screen options. 4. Did the diagnostics run error-free and is the server system-error LED lit? v No, complete Chapter 12, End-of-call procedure, on page 249 and then return control of the server to the customer. v Yes, go to Problem resolution on page 3 to review probable causes and return to step 6 when the problem is resolved. If the errors persist, see User-initiated Call Home on page 25 and contact your next level of support. 5. If you came here from step 2 on page 6, or if the diagnostics do not detect any hardware errors but the problem remains during server operations, use the Problem Manager to check for any problems listed as open. See Problem Manager on page 17 for information about how to display an open problem, which can aid in resolution of the problem. 6. If ProtecTIER Manager is available, check for alerts and clear any open alerts. a. Select the node on which you want to check alerts. b. Click the Alerts button. The Alerts window is displayed. c. Take appropriate action to clear any open alerts (see Table 3 on page 32) or contact IBM Software Support for assistance. d. Repeat for the second node, if applicable. Chapter 1. Maintenance and troubleshooting 7

32 Troubleshooting disk controller problems You can troubleshoot problems with a disk controller by answering a series of questions. Before you begin Attention: Before you proceed, become familiar with the disk controller by reviewing Figure 46 on page 131, Figure 47 on page 132, Figure 50 on page 136, Disk controller rear indicators on page 133 and Figure 53 on page 139. v If you have the DS4700 disk controller, you can also use the IBM System Storage DS4700 Express Storage Subsystem Installation, User's and Maintenance Guide as a guide in diagnosis. v If you have the DS3950 disk controller, you can also use the IBM System Storage DS3950 Storage Subsystem Installation, User's, and Maintenance Guide as a guide in diagnosis. Procedure To troubleshoot problems with the disk controller, answer the following questions. 1. Are all disk controller LEDs solid green or flashing green? v Yes, return the TS7650 Appliance or TS7650G to the customer. v No, go to step Are any amber LEDs lit on the disk controller? v No, return the TS7650 Appliance or TS7650G to the customer. v Yes, go to Disk controller components on page 131 and continue with the diagnosis and any required removal and replacement procedures, then return here. v If the diagnostics do not detect any hardware errors but the problem remains during server operations, use the Problem Manager to check for any problems listed as open. See Problem Manager on page 17 for information about how to display an open problem, which can aid in resolution of the problem. 3. Verify that you have completed all the repairs. Are all disk controller LEDs solid green or flashing green? v Yes, return the TS7650 Appliance or TS7650G to the customer. If you found a Problem Manager open problem related to the error, see Problem Manager on page 17 for information about how to close the open problem. v No, see User-initiated Call Home on page 25 and contact the next level of support. 8 TS7650 and TS7650G Problem Determination and Service Guide

33 Troubleshooting disk expansion unit problems You can troubleshoot problems with a disk expansion unit by answering a series of questions. Before you begin Attention: If you are not already familiar with the disk expansion unit, begin by reviewing Figure 46 on page 131 (the hot-swap drive bays for the disk controller and the disk expansion unit are identical), Disk expansion unit front LEDs on page 146, and Disk expansion unit rear LEDs on page 147. You can also use the IBM System Storage DS4700 Express Storage Subsystem Installation, User's and Maintenance Guide and the IBM TotalStorage DS4000 EXP810 Storage Expansion Unit Installation, User's and Maintenance Guide as guides in diagnosis. Procedure 1. Are all disk expansion unit LEDs solid green or flashing green? v Yes, return the TS7650 Appliance or TS7650G to the customer. v No, go to step Are any amber LEDs lit on the disk expansion unit? v No, return the TS7650 Appliance or TS7650G to the customer. v Yes, go to Chapter 7, Disk expansion units, on page 145 and continue with the diagnosis and any required removal and replacement procedures, then return here. v If the diagnostics do not detect any hardware errors but the problem remains during server operations, use the Problem Manager to check for any problems listed as open. See Problem Manager on page 17 for information about how to display an open problem, which can aid in resolution of the problem. 3. Verify that you have completed all the repairs. Are all disk expansion unit LEDs solid green or flashing green? v Yes, return the TS7650 Appliance or TS7650G to the customer. If you found a Problem Manager open problem related to the error, see Problem Manager on page 17 for information about how to close the open problem. v No, see User-initiated Call Home on page 25 and contact the next level of support. Chapter 1. Maintenance and troubleshooting 9

34 Resolving preferred path critical events A preferred path critical event can be caused by several activities. You can resolve the critical event by trying a series of corrective actions until one of them works. About this task Several normal activities can generate a preferred path critical event on the disk controllers in a TS7650 Appliance or TS7650G. v Restarting a server in a clustered configuration v Failover between servers in a clustered configuration v v Note: Unwanted fencing is a known problem in a clustered environment. Fencing is the disconnection of a node from the shared storage of the cluster. Fencing cuts off I/O from shared storage to ensure data integrity. If you experience unwanted fencing and cannot stop it, contact your next level of IBM support. Issuing a scan operation from ProtecTIER Manager Generating a diagnostic summary Each scan event causes the host bus adapter (HBA) initiators to query both the A controller and the B controller of the disk controllers. This results in the Automatic Volume Transfer/Automatic Data Transfer (AVT/ADT) being signaled and the logical units (LUNs) being moved to those controllers for a brief moment. In a properly configured environment, after the devices are actually used by the server, the logical drives move back to the preferred path. In a configuration that does not access every LUN (idle node, archive data, and so on) those LUNs might not be accessed on a sufficiently regular basis. Procedure To resolve a preferred path critical event, try the following corrective actions in order until one of them is successful. 1. Within Storage Manager, select Advanced > Recovery > Redistribute logical drives. Important: Do not select Change Preferred Path to resolve the temporary messages. Only perform this when you are correcting a misconfiguration. 2. If the drives immediately or continually revert to the Logical Drive Not on Preferred Path state, type the command multipath at the CLI prompt of the affected node. This normally realigns the preferred paths. 3. If the multipath command does not resolve the problem, type multipath -v2. 4. If the multipath -v2 command does not resolve the problem, use Storage Manager to modify the AVT Default Host Type to LNXCLUSTER. For a detailed procedure, refer to the ProtecTIER and DS4000 & DS5000 AVT setting web page. 5. If the problem persists, open a service call to either have the drives reconfigured or diagnose failing hardware components. Remote support through Call Home Remote support is available for the TS7650G and the TS7650 Appliance through the Call Home capability on the TSSC. The Call Home feature reports failures detected by the ProtecTIER servers. Whenever a failure is detected, Call Home sends detailed error information to IBM (home). The IBM Service Representative 10 TS7650 and TS7650G Problem Determination and Service Guide

35 can then prepare an action plan to handle the problem before traveling to the affected installation. The appliance or gateway might also periodically send support information (such as configuration, code versions, and error logs) to IBM. Doing so speeds-up problem determination and fault resolution. When enabled on the appliance and gateway, Call Home uses a connection on your Ethernet network to transmit hardware and software problem reports to IBM. Call Home is enabled and tested by IBM Service Representatives during initial system installation. Tip: To enable Call Home, go to the TSSC General Settings page. The Call Home option allows you to select either a Modem or Ethernet interface. Set the Call Home option to use the Ethernet interface for the most reliable error notification. When the Reliability, Availability, and Serviceability (RAS) software on the ProtecTIER server detects an error condition, Call Home sends detailed error information through the TSSC to IBM (home). If the error indicates a problem with a field replaceable unit (FRU), an IBM Service Representative can then prepare an action plan to handle the problem before traveling to your site. The TS7650G and the TS7650 Appliance provide four Call Home capabilities: Problem Call Home, Heartbeat Call Home, Test Call Home, and User-Initiated Call Home; descriptions follow. RAS sends data files that may be helpful to IBM Support Center personnel for all four types of Call Home. These data files include error logs and configuration information, such as the Machine Reported Product Data (MRPD) log. Test Call Home The IBM Service Representative sends a Test Call Home signal after enabling the Call Home feature during initial installation. You can also send a Test Call Home to ensure that the setup is correct and that the appliance or gateway can successfully open a Problem Management Record (PMR) in the IBM Remote Technical Assistance Information Network (RETAIN). Problem Call Home When RAS detects a problem, RAS initiates a Call Home operation to create a PMR in RETAIN. The PMR is a single page of text data that enables the Support Center or the Service Representative to access an action plan and a list of applicable FRU components. Heartbeat Call Home To ensure proper ongoing Call Home functionality, the system sends a Heartbeat Call Home on a regularly-scheduled basis. The heartbeat interval is user-defined. User-Initiated Call Home You can manually initiate Call Home from the TSSC GUI to collect a product engineering (PE) package. Call Home through the TSSC The TSSC is a service tool that must be present in an IBM-supported TS7650G or TS7650 Appliance. You can either order a TSSC with your appliance or gateway, or use a TSSC already installed at your site. Attention: While it is possible to operate a TS7650G or TS7650 Appliance without a connected TSSC, note that IBM does not support this configuration. Chapter 1. Maintenance and troubleshooting 11

36 For both the TS7650G and the TS7650 Appliance, FC 2724 provides a new TSSC, while FC 2714 and FC 2715 allow connection of an existing TSSC. For more information on these feature codes, see the IBM System Storage TS7650 ProtecTIER Deduplication Appliance and TS7650G Deduplication Gateway Introduction and Planning Guide, GA Table 2 presents the capabilities of remote support with a TSSC. Table 2. Remote support capabilities Customer site Call Home events v Error initiated v v Heartbeat (regular interval) Test TSSC support capability v Error-initiated problem reporting for up to 43 subsystems v Staged, error-specific data gathering v Subsystem and system console heartbeat reporting v Wellness checking v Log file storage (daily) v Code image and documentation repository (from media and RETAIN Fix Distribution Library) TSSC and remote support service tools v v v Code image broadcast Call home event log review End-of-call completion report IBM support TSSC remote access v Authenticated, secure remote access v v Simultaneous call in and call home Data transmission (TCP/IP) supported IBM call home database v 24/7 access by IBM support staff v Error analysis and search capability 12 TS7650 and TS7650G Problem Determination and Service Guide

37 Troubleshooting problems with Call Home Call Home is IBM's means to automatically capture data and report errors in the TS7650 Appliance and the TS7650G. This minimizes the need for human intervention in gathering and offloading data to the IBM Remote Technical Assistance and Information Network (RETAIN ). For instructions on how to install, configure or troubleshoot Call Home, see the IBM System Storage TS3000 System Console (TSSC) Maintenance Information. For additional Call Home troubleshooting information, see Troubleshooting the server-to-tssc connection on page 24. For more instructions on using Call Home to aid in system troubleshooting, see Chapter 2, System troubleshooting tools, on page 15. Enabling Call Home: In order to use Call Home, you must ensure that it is enabled. For instructions on how to install, configure or troubleshoot Call Home, see the IBM System Storage TS3000 System Console (TSSC) Maintenance Information. To ensure that Call Home is enabled on the server, log into the server with user ID root and password admin (or an ID and password provided by the customer). Then you can either run the Call Home enablement command from the command line, or enable Call Home using the ProtecTIER Service menu. v To enable Call Home from the command line, enter the following command: /opt/ras/bin/rscerchfunction -e v To enable Call Home from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16), type the option numbers to select the appropriate options: For ProtecTIER v.2.5 or earlier, select Call Home Commands > Enable Call Home. For ProtecTIER v.3.1 or later, select Problem Alerting > Enable/Disable. If the return message is Call Home is already enabled, then the problem might be a communication failure between the server and the TSSC. Verify that the server is listed within the Attached Systems list of the TSSC, and under the Call Home Switched column, there should be a 1 (enabled) listed in the row for the specific server. If the above criteria are correct, run a test Call Home (see Running a test Call Home ). Running a test Call Home: To verify that Call Home is installed and enabled, you can run a test Call Home. v To run a test Call Home from the command line, enter the following command: /opt/ras/bin/rscerchtest v To run a test Call Home from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16), type the option numbers to initiate a test Call Home. For ProtecTIER V2.5 or earlier, select Call Home Commands > Test Call Home. Chapter 1. Maintenance and troubleshooting 13

38 For ProtecTIER V3.1 or later, select Problem Alerting > Send a Test Call Home. If the return message is Test Call Home sent successfully, then the system has completed the test. Contact your next level of support if you encounter any of the following problems: v The test Call Home fails. v The test Call Home does not show in the Call Home Queue. v The Call Home Switched column retains an NA after enabling. 14 TS7650 and TS7650G Problem Determination and Service Guide

39 Chapter 2. System troubleshooting tools Running command line tools The ProtecTIER server includes a suite of system troubleshooting tools. The troubleshooting tools included with the TS7650 Appliance and the TS7650G can perform the following tasks: v Collect system information to aid in problem determination (see Using Dynamic System Analysis on page 16) v Manage problems (see Problem Manager on page 17) v Monitor system health (see System health monitoring on page 19) v Capture data and report errors automatically using Call Home (see Call Home on page 23) v Collect system logs and deliver them to the Call Home database (see User-initiated Call Home on page 25) These are discussed individually in the following sections after instructions on how to run any of the system troubleshooting command line tools. You can use the TSSC to perform system troubleshooting tasks from the command line. Procedure To run the command line interface tools, complete the following steps: 1. From the TSSC login screen, log in with user ID service and password service. 2. Right-click the TSSC desktop and select Console Actions > Console Configuration Utility. 3. If prompted for a user ID and password, enter service and service, respectively. 4. Click Attached Systems. 5. In the Attached Systems list: a. Verify that the server is listed. b. Note or record the IP address of the server. You will need the IP address when you open the SSH session. c. Click Close this window as necessary to return to the TSSC desktop. 6. If the customer system administrator has changed the password for the ptadmin account from the default value ptadmin, obtain the new password. 7. Open an SSH session with the server: a. Right-click the TSSC desktop and then click Terminal. b. At the Terminal command prompt type the command tsys to display a list of attached systems and type the number of the server to which you want to connect or type the following: ssh ptadmin@ xxx where xxx is the last octet of the server's IP address in the TSSC's Attached Systems list, and then press the Enter key. Copyright IBM Corp. 2011,

40 ProtecTIER Service menu Note: You can also use Telnet to access an attached tape system. Right-click from anywhere on the desktop to view the main menu. Select Browser Functions > Telnet to Attached Tape System. c. At the password prompt, type the password for the ptadmin account you obtained above (or, if the password has not changed from the default value, type ptadmin) and then press the Enter key. The ProtecTIER Service menu is a text-based menu that provides access to the ProtecTIER command line interface. The ProtecTIER Service menu eliminates the need for you to remember commands or look them up in the documentation. By navigating the menus and sub-menus, you can easily understand and choose the possible actions or queries to perform. To access the ProtecTIER Service menu, log in with username root and password admin. Then type the appropriate command from any directory on the server. v For ProtecTIER V2.5 or earlier, type rasmenu. v For ProtecTIER V3.1 or later, type menu. The script resides in the /opt/ras/bin directory. It displays a menu. Type the number of the operation you want to use and press Enter. The top-level menu is similar to the example shown in Figure ProtecTIER Service Menu running on rasgen ) ProtecTIER Configuration (...) 2) Manage ProtecTIER services (...) 3) Health Monitoring (...) 4) Problem Alerting (...) 5) Version Information (...) 6) Generate a service report 7) Generate a system view 8) Update ProtecTIER code E) Exit >>> Your choice? Figure 1. ProtecTIER Service menu Note: Some of the items lead to sub-menus, which are denoted by (...). See Problem Manager on page 17 for the Health Monitoring menu structure. Using Dynamic System Analysis Dynamic System Analysis (DSA) collects and analyzes system information to aid in diagnosing server problems. Procedure 1. To access DSA from the ProtecTIER Service menu, select Generate a service report. The following options display: 16 TS7650 and TS7650G Problem Determination and Service Guide

41 Please choose a profile: 1) Default - Collects all system information reports and all log files from the either the last 4 days, or a maxiumum of 10 file entries, whichever comes first 2) Performance - Used to troubleshoot performance-related issues 3) Deduplication - Used to troubleshoot deduplication-related issues 4) Basic - Used when customer has an issue that requires a quick problem determination 5) Monitoring - Used in cases where Support provides frequent summaries on the system s health and behavior 6) Full - Collects all system information reports and all log files in their entirety 7) Systemview - Collect system information required to generate the system view html output 8) Hardware - Collects system X server hardware report (please shutdown all services before collecting this profile) 9) Coredump - Collects the vmcore dumps from the system q) Quit Choose: Problem Manager Note: To prevent an unexpected fence/reboot, the system must be offline before collecting the hardware profile. 2. To view the server hardware report, type 8 and press Enter. Problem Manager is the Reliability, Availability, and Serviceability (RAS) component that processes problems detected by system health monitoring. Problem Manager consolidates the detected problems, filters out duplicate and sympathetic problems (secondary problems that are generated by primary problems), and maintains a problem log of the results (all open primary problems) for use in service actions and information. Accessing Problem Manager from the ProtecTIER Service menu You can access Problem Manager from the ProtecTIER Service menu. Problem Manager tools are available from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16) and from the command line. The preferred way to access the tools is through the ProtecTIER Service menu. To access Problem Manager tools from the ProtecTIER Service menu, type the number of the operation you want to use and press Enter. v To get a list of open problems, type the numbers to select the appropriate options. For ProtecTIER V2.5 or earlier, select Problem management > List open problems. For ProtecTIER V3.1 or later, select Health Monitoring > List open problems. Chapter 2. System troubleshooting tools 17

42 v ProtecTIER Service Menu running on rasgen Health Monitoring (...) ) Display health summary for this node 2) Display detailed health for this node 3) Run a full check on this node 4) Reset RSA/IMM 5) List open problems 6) Service Mode B) Back E) Exit >>> Your choice? Figure 2. ProtecTIER system Health Monitoring menu To close or cancel an open problem once it has been fixed, choose the option for Problem management and then follow the on-screen instructions to either run the full refresh process or skip it, and respond Y or N to close out the problem records. Read the screen carefully to determine the correct choice of action in your responses. Problem Record 1: problemid=< > problem type=sw component type=protectier component location=node 0/PT problem severity=degraded number of occurrences=14 time of first occurrence= ,19:39:59 time of last occurrence= ,22:40:41 notify state=sent by node 0 notify time= ,20:02:56 message ids={0xaa043040} ["MESSAGE:< >"][1] => {messageid(0xaa043040) message("internal error ( ) in file source_dir/cmnt/process/service_quorum.cc line ") debuginfo(".")} Number of Problem Records is 1 >>> You can choose to cancel a problem. Please note that if problem still exists a new similar problem may reoccur. Do you want to cancel a problem? Enter y or yes to cancel a problem or any other key to exit. y >>> Please enter Problem Record Number to cancel : 1 >>> Are you sure you want to cancel Problem Record 1? (yes no) yes End Processing Procedure Successfully Press <ENTER> to continue Figure 3. Canceling a problem Note: Allow up to 20 minutes for Service Mode to be disabled. Service Mode is used to inhibit Call Home, SNMP, and alerts. Service Mode is available with ProtecTIER V3.1 or later. It should be enabled at the start of a service call and disabled at the End-of-call. A health check shows degraded when Service Mode is enabled. Service Mode is automatically disabled after 48 hours. To enable Service Mode, select Health Monitoring > Service Mode. Then type yes and press Enter. 18 TS7650 and TS7650G Problem Determination and Service Guide

43 If the ProtecTIER Service menu is not available, run the Problem Manager commands from a server command line (see Accessing Problem Manager from the command line ). Accessing Problem Manager from the command line You can access Problem Manager directly from the command line. System health monitoring If the ProtecTIER Service menu is not available, run the Problem Manager commands from a server command line. Get open problems: The following command retrieves and displays all open problems from Problem Manager. Each problem is identified by a unique 12-digit problem ID. rscerpmgetopenproblems Arguments This command takes no arguments. Cancel problem: This command closes an open problem in the problem log after it is fixed. rscerpmcancelproblem problemid Arguments Other than the problem ID, this command takes no arguments. problemid: The unique 12-digit number (displayed by rscerpmgetopenproblems) that identifies the problem. Refer to Chapter 2, System troubleshooting tools, on page 15 for instructions about how to run the command line tools. Various tools are provided for monitoring system health. The preferred method for monitoring system health is using the ProtecTIER Service menu. See Accessing Problem Manager from the ProtecTIER Service menu on page 17. If the ProtecTIER Service menu is not available you can monitor system health using the command line tools. System health monitoring command line tools This topic describes system health monitoring command line tools. The system health monitoring tool consists of the following: System health monitoring daemon The system health monitoring daemon controls system component status changes, schedules and runs required health monitoring programs and scripts, and provides an interface for checking execution based on user requests. The daemon runs upon system startup. Health monitoring scripts and programs These perform checks against specific system components or elements known as health point elements. Health point element summary data include: v Name of the health point element (for example, internal network ). Chapter 2. System troubleshooting tools 19

44 v Type of health point element. These include Network, Filesystem, Server, Devices, DS4000, DS4700, DS3950, DS5020, ProtecTIER Software, and Reliability, Availability, and Serviceability (RAS). v v Note: The ProtecTIER server only calls home for the Disk Controller and expansion units when attached to the TS7650 Appliance. Point identifier, which together with the path uniquely identifies the health point. Path format. This includes the node identifier (Node 0 or Node 1). If a disk repository related element, the path includes the disk array identifier and the disk tray identifier. v v v v v Note: Node 0 refers to the lower server normally described as Node A and Node 1 refers to the upper server which is normally described as Node B. Location description (for example, left power supply ). Timestamp indicating the latest health point update. Health point element status. Values are: OK WARNING DEGRADED FAILED OFFLINE SERVICE (currently not used) UNCONFIGURED Expiration time/status interval. The maximum amount of time for each of the components listed below before the system rechecks itself by running its own internal health checks: Network: 10 minutes Filesystem: 20 minutes Server: 25 minutes Devices: 15 minutes DS4000: 45 minutes ProtecTIER software: 30 minutes RAS: 30 minutes Health status message XML status file The current status of system components resides in this file. Interface tools The interface tools allow collection of information from the XML status file and display the information collected. They also allow the user to monitor the status of currently running checks or to run any check immediately. See Using system health monitoring command line tools. Log files The log files record the results of checks, status changes, and detailed error correction action information. Using system health monitoring command line tools You can use system health monitoring tools directly from the command line. 20 TS7650 and TS7650G Problem Determination and Service Guide

45 In a command line session to the server, you can use two commands related to system health monitoring: Command rscerhmdisplay rscerhmstatusctl Description System health monitoring display. This tool is used to view the results of the latest check that was performed on the system. This utility is also available through the ProtecTIER Service menu (see ProtecTIER Service menu on page 16). Select System Health Monitoring > Display All System Health Points. System health monitoring status control. This tool is used to immediately invoke one or more checks on the system. To view the results, type the display command. System health monitoring display The rscerhmdisplay command provides access to the system health monitoring display. rscerhmdisplay -i id -p path -v Server -x -n DiskArray Networking Software -vall --all --gall -L -Lv -? -H -i id -p path Display the status of the specified healthpoint (ID and path). -v Display the status of the specified virtual object. -vall --all --gall Display the status of all virtual objects. Display the status of all health points. Display the status of all health points in groups. -L Return the status of the system: 0 All OK or unconfigured 1 Any number of OFFLINEs 2 Any number of WARNINGs 3 Any number of DEGRADEDs 4 Any number of FAILEDs -Lv Display the status of the system in words. -x Convert any output to XML. -n Do not check for the system state before trying to run. Chapter 2. System troubleshooting tools 21

46 -? Display the syntax for the command. -h Display the syntax for the command. Some examples are shown in Figure 4 and Figure 5. rscerhmdisplay -i "harddrive1" -p "Node 0" TS7650 Checkout Version _ executed on: T04:13:07 Verify state of harddrive1 (Node 0)... DEGRADED *DEGRADED: Component Location: messi.haifa.ibm.com *DEGRADED: SRN: 0xAA *DEGRADED: Hard Drive 1 Fault Figure 4. Displaying status of a hard drive rscerhmdisplay -i "PowerFan 2" -p "DiskArray-180-1/Enclosure85" TS7650 Checkout Version _ executed on: T04:13:07 Verify state of PowerFan 2 (DiskArray-180-1/Enclosure85)... OK Figure 5. Displaying status of a power supply and fan System health monitoring status control The rscerhmstatusctl command provides access to system health monitoring status control. rscerhmstatusctl -r -s -l -b -e check_id -all -b -r List the checks currently running on the component queried. -s List the current status of all health point checks, including name, when last run, and whether the check is currently running on the component queried. -l List the check IDs of all possible checks, one per line. -b Perform one or more checks, as indicated by the additional arguments: -e check_id Perform the specified check immediately. -all Perform all checks immediately. -b Force the health monitoring daemon to run the specified checks the next time it starts. Some examples of arguments are shown in Figure 6 on page 23, Figure 7 on page 23, and Figure 8 on page TS7650 and TS7650G Problem Determination and Service Guide

47 >rscerhmstatusctl -l hwcheck fscheck ds4kcheck rascheck ptcheck networkcheck servercheck Figure 6. Listing all possible checks > rscerhmstatusctl -s There are currently 7 checks registered Check ID: hwcheck Name: Hardware check Duration: 900 Description: Checks the frontend and backend adapters Instances Running: 0 Last Started at: Dec 22 13:37:44 Last Completed at: Dec 22 13:37:53 Last Return Code: 0 Next Scheduled at: Dec 22 13:52:44 Check ID: fscheck Name: Filesystem check Duration: 1200 Description: Checks the local filesystem Instances Running: 0 Last Started at: Dec 22 13:42:44 Last Completed at: Dec 22 13:42:45 Last Return Code: 0 Next Scheduled at: Dec 22 14:02:44 Figure 7. Listing status of all health point checks > rscerhmstatusctl -e ds4kcheck ds4kcheck:0 > rscerhmstatusctl -r Check ID: ds4kcheck Name: DS4000 hardware check Started at: Dec 22 13:50:01 Timeout at: Dec 22 13:55:01 There are 1 checks currently active Figure 8. Performing a specific check immediately Call Home Call Home is the IBM means to automatically capture data and report errors in the TS7650 Appliance and the TS7650G. This minimizes the need for human intervention in gathering and offloading data to IBM support. Call Home is used in four situations: Error-initiated Call Home Error-initiated Call Home is triggered by TS7650 Appliance or TS7650G failures. Heartbeat/machine reported product data (MRPD) Call Home Heartbeat/MRPD Call Home regularly reports system status, operability, and configuration information. Chapter 2. System troubleshooting tools 23

48 Call Home test Call Home test is manually initiated from the command line to test the path to IBM RETAIN. User-initiated Call Home User-initiated Call Home is discussed in detail in User-initiated Call Home on page 25. Troubleshooting the server-to-tssc connection You can use the ProtecTIER Service menu to troubleshoot the connection between the server and the TS3000 System Console (TSSC). Procedure 1. Access the ProtecTIER Service menu (see ProtecTIER Service menu on page 16). 2. To ensure that Call Home is enabled on the server, type the option numbers to select the appropriate options. v For ProtecTIER V.2.5 or earlier, select Call Home Commands > Enable Call Home. v For ProtecTIER V.3.1 or later, select Problem Alerting > Enable/Disable. 3. If the return message is Call Home is already enabled, then the problem might be a communication failure between the server and the TSSC. Verify that the server is listed within the Attached Systems list of the TSSC, and under the Call Home Switched column, there should be a 1 (enabled) listed in the row for the specific server. 4. If the above criteria is correct, type the option numbers to initiate a test Call Home. v For ProtecTIER V2.5 or earlier, select Call Home Commands > Test Call Home. v For ProtecTIER V3.1 or later, select Problem Alerting > Send a Test Call Home. Results If the return message is Test Call Home sent successfully, then the system has completed the test. What to do next Contact your next level of support if any of the following problems occur: v Failures are encountered. v The test Call Home does not show in the Call Home Queue. v The Call Home Switched column retains an NA after enabling. Call Home command line tools On servers running ProtecTIER V2.5 or earlier, there are several commands related to Call Home that you can issue in a command line session to the server. Command rscerchtest rscerchsethbinterval rscerchfunction Description Call Home test Heartbeat interval modification Call Home enabling or disabling 24 TS7650 and TS7650G Problem Determination and Service Guide

49 All of these options are also available using the ProtecTIER Service menu (see ProtecTIER Service menu on page 16). Choose the appropriate option from the Call Home Commands submenu. Note: These commands are not supported in ProtecTIER V3.1 or later. Call Home test Note: This command is not supported in ProtecTIER V3.1 or later. This command invokes the rscerchdispatch command, which generates a test Call Home signal to the TSSC and to IBM RETAIN. rscerchtest Arguments This command takes no arguments. Heartbeat interval modification Note: This command is not supported in ProtecTIER V3.1 or later. This command sets the interval between Heartbeat/MRPD call home events. In a clustered environment, running this command for one node changes the settings for both nodes. rscerchsethbinterval <intervalindays (1 to 14)> Arguments intervalindays: how often (in days) the Heartbeat/MRPD Call Home events are sent. Valid values are 1 to 14. Other than the interval, this command takes no arguments. Call Home enabling or disabling Note: This command is not supported in ProtecTIER V3.1 or later. This command enables or disables the Call Home function. In a clustered environment, running this command for one node changes the settings for both nodes. rscerchfunction -d/-e Arguments -d This argument disables the Call Home function. -e This argument enables the Call Home function. -q This argument queries the Call Home status. Refer to Chapter 2, System troubleshooting tools, on page 15 for instructions about how to run the command line tools. User-initiated Call Home You can manually initiate Call Home and specify which information to collect. When you manually initiate Call Home, you can customize the information to be collected by specifying combinations of the following sources: Chapter 2. System troubleshooting tools 25

50 Date The server collects information from log files modified after the specified date and time. Component The server collects information from log files defined in component-specific log definition files. File name The server collects information from all log files in a specified directory. The server compiles the collected information in a package file in the /opt/ras/pepackage/out directory. You can then offload the package file from the server or forward it to the call home database for detailed analysis. Note: The /opt/ras/pepackage/out/latestpepackage.tar.gz file is overwritten each time the logs are collected. To save existing logs before collecting new logs, move the logs to another directory. Example: mv /opt/ras/pepackage/out/ latestpepackage.tar.gz /tmp You can collect logs manually at the TSSC by the following two methods: From the TS3000 System Console This is the preferred method. See Using the TS3000 System Console graphical user interface. From the TS7650 Terminal command line See Using the TS3000 System Console command line on page 27. Using the TS3000 System Console graphical user interface Complete the following task to initiate Call Home from the TS3000 System Console (TSSC) graphical user interface (GUI). Procedure 1. If not already logged into the TSSC, log in using user ID service and password service. 2. Right-click the TSSC desktop and then click Console Actions > Console Configuration Utility. 3. If prompted for a user ID and password, enter service and service, respectively. 4. Click Attached Systems. 5. Verify that the server is listed and then click Close this window. 6. Click PE Packages. 7. Click Subsystem Log Retrieval. 8. On the menu in the Select Logs area, click one of the following: v 3958DD1 v 3958DD3 v 3958DD4 v 3958DD5 v 3958AP1 9. On the Select an Attached System menu, click the server you want. 10. In the Start Time area, enter or select values for the year, month, day, and hour corresponding to some time before the occurrence of the problem. 11. (Optional) From the Component list, select one or more components. The components are as follows: 26 TS7650 and TS7650G Problem Determination and Service Guide

51 COMMON The log definition file for this component specifies items such as Linux settings and command line history. COMMON logs are always collected. SERVER The log definition file for this component specifies various logs generated by the server. ProtecTIER (PT) The log definition file for this component specifies various ProtecTIER management system logs. DISK RAS The log definition file for this component specifies various logs generated by the disk controller for the attached disk repository. The log definition file for this component specifies Reliability, Availability, and Serviceability (RAS) code logs used for RAS code analysis. All components This log definition file specifies that the logs in all log definition files listed above be collected. 12. (Optional) In the Specify a File field, type the complete path to a specific file to be collected. To collect more than one file, separate complete paths by a space. You can use * as a wildcard character. 13. For Package Size, click either Big or Small. At this time, this selection only changes the PT component. Clicking Big collects all logs defined in the PT log definition file. Clicking Small only collects a subset of the defined log files. The default is Small. 14. Click Retrieve Logs. 15. Allow the log collection operation to complete; this can take from several minutes to over an hour. When complete, the Log Retrieval Page again appears. 16. In the Action area, click Click here. 17. In the File Name list, select the check box or boxes for the log files. 18. Click an offload option. The choices are as follows: v Copy to universal serial bus (USB) Device v Copy to compact disc (CD) v Copy to Call Home 19. Follow the on-screen instructions for the choice you made in step 18. Using the TS3000 System Console command line You can initiate Call Home from the TS3000 System Console (TSSC) command line. Command syntax Issue the log collection command in a secure shell (SSH) command line or by using the ProtecTIER Service menu. v To issue the log collection command in an SSH command line, type the following command: rscerlogcollect [-d x] [-c x] [-b/-s] [-f "x"] [-o x] [-h/-?] Chapter 2. System troubleshooting tools 27

52 v where x indicates parameter definitions, discussed below. All parameters enclosed in square brackets above are optional; however, you must specify at least one parameter. To issue the log collection command from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16), type the appropriate option number. For ProtecTIER V2.5 or earlier, select Collect Logs. For ProtecTIER V3.1 or later, select Generate a Service Report. Enter the number for the option to collect the appropriate logs. Parameters The parameters for the log collection command in a secure shell (SSH) command line are defined as follows: -d date This optional parameter defines the oldest modification date of files to be collected. If no date is specified, the default is 24 hours before the command is issued. You can use any date format supported by the generic Linux date parameter -d. The following are examples of supported date formats: v yyyy-mm-dd (for example, ). This defaults to midnight at the start of the specified date. v yyyy/mm/dd (for example, 2010/10/24). This defaults to midnight at the start of the specified date. v dd-mmm (for example, 24-OCT). This defaults to midnight at the start of the specified date of the current year. v hh:mm:ss (for example, 02:34:00). This defaults to the time specified of the current date and year. v hh:mm:ss dd MMM yyyy (for example, 02:34:00 24 OCTOBER 2010). v h:mm dd MMM (for example, 2:34 24 OCT). This defaults to the current year. -c component name This parameter specifies one or more component-specific log definition file to use for log collection. The components are as follows: COMMON This log definition file specifies items such as Linux settings and command line history. SERVER This log definition file specifies various logs generated by the server. ProtecTIER (PT) This log definition file specifies various ProtecTIER management system logs. DISK RAS all This log definition file specifies various logs generated by the disk controller for the attached disk repository. This log definition file specifies Reliability, Availability, and Serviceability (RAS) code logs used for RAS code analysis. This log definition file specifies that the logs in all log definition files listed above be collected. 28 TS7650 and TS7650G Problem Determination and Service Guide

53 The definition file for each component above consists of a list of log files to collect and the type of file (text, command output, or a file that requires special handling). The log files defined in the COMMON log definition file are always collected whether or not COMMON is specified. If all is specified, the log files defined in all log definition files are collected. -b or -s You can use either of these parameters if the PT component is specified with the -c parameter. At this time, setting these parameters has no effect on components other then PT. If you specify -b, the command collects all logs defined in the PT log definition file. If you specify -s, the command collects only a subset of the logs specified in the PT log definition file. If not specified, only a small subset of logs defined in the PT log definition file are collected. -f "complete_file_path" This parameter specifies a complete path to the file to be collected. The path must be enclosed in double quotes as shown. To collect more than one file, you can specify more than one path. (For example, -f "complete_file_path_1" "complete_file_path_2".). You can use * as a wildcard character. -o complete_file_path_and_name This parameter specifies an alternative path and file name for the command's output to the default. No quotes are used. If this parameter is not used, the default is /opt/ras/pepackage/out/latestpepackage.tar.gz. -h or -? This parameter displays the detailed help screen for the command. Examples rscerlogcollect -c all -d Mar 06 12:36 This command collects logs defined in all component-specific log definition files modified after 12:36, March 6, of the current year, including a subset of the logs in the PT log definition file. rscerlogcollect -c all -d s This command collects logs defined in all component-specific log definition files modified after 00:00, December 5, Only a subset of the logs in the PT log definition file are collected. rscerlogcollect -f "/dir1/log" This command collects the following files: v All files in the /dir1/log directory v All files defined in the COMMON log definition file that were modified in the last 24 hours rscerlogcollect -o -c PT -b /usr/package.tar.gz This command collects all files defined in the COMMON log definition file that were modified in the last 24 hours, including all logs in the PT log definition file, and creates the package file /usr/package.tar.gz. rscerlogcollect -c DISK RAS This command collects logs defined in the COMMON, DISK, and RAS log definition files that were modified in the last 24 hours. rscerlogcollect -c SERVER -f "/directory/logname*" -d :30:55 This command collects the following files: Chapter 2. System troubleshooting tools 29

54 v v v Procedure All files defined in the COMMON log definition file that were modified after 07:30:55, April 10, 2010 All files defined in the SERVER log definition file that were modified after 07:30:55, April 10, 2010 All files in the /directory/logname* directory To issue the rscerlogcollect command, complete the following steps: 1. Right-click the TSSC desktop and then click Console Actions > Console Configuration Utility. 2. If prompted, log in with user ID service and password service. 3. Click Attached Systems. 4. In the Attached Systems list: a. Verify that the server is listed. b. Note or record the IP address of the server. You will need the address in step 6. c. Click Close this window as necessary to return to the TSSC desktop. 5. If the customer's system administrator has changed the password for the ptadmin account from the default value ptadmin, obtain the new password. 6. Open an SSH session with the server: a. Right-click the TSSC desktop and then click Terminal. b. At the Terminal command prompt, enter the command tsys and select the system to which you want to connect or type the following command: ssh ptadmin@ xxx where xxx is the last octet of the server's IP address in the TSSC Attached Systems list, and then press the Enter key. c. At the password prompt, type the password for the ptadmin account you obtained in step 5 (or, if the password has not changed from the default value, type ptadmin) and then press the Enter key. 7. Do one of the following: v From the ProtecTIER Service menu (see ProtecTIER Service menu on page 16), type the appropriate number for the sub-menu and action to be taken. v At the SSH command prompt, type the following command: rscerlogcollect [-d x] [-c x] [-b/-s] [-f "x"] [-o x] [-h/-?] with the parameters defined as desired, and then press the Enter key. Define the parameters using the following guidelines: Define the -c parameter to best fit the situation; for example, use SERVER with the -c parameter for server issues and use DISK with the -c parameter for disk repository system issues. If the location of the issue is uncertain, use all with the -c parameter. Specify a time with the -d parameter corresponding to the time of the occurrence of the issue. 8. Allow the process to finish collecting logs and create the package file. (This can take from several minutes to more than an hour.) When the process is complete, the message Exiting Successfully is displayed and the SSH command prompt returns. 9. At the SSH command prompt, type the following: 30 TS7650 and TS7650G Problem Determination and Service Guide

55 scp /tsscfilepath/filename where v xxx is the last octet of the server's IP address in the TSSC Attached Systems list. v /serverfilepath/filename is the full path and file name of the log package file on the server. If you do not specify an alternate with the -o parameter, the default is /opt/ras/pepackage/out/latestpepackage.tar.gz. If you did specify an alternate with the -o parameter, use that value. Note: The /opt/ras/pepackage/out/latestpepackage.tar.gz file is overwritten each time the logs are collected. To save existing logs before collecting new logs, move the logs to another directory. Example: mv /opt/ras/pepackage/out/latestpepackage.tar.gz /tmp v /tsscfilepath/filename is the full path and file name of the target location on the TSSC. Use of /var/enc/offload as the target directory is recommended to simplify locating the file for offloading later in the procedure. 10. Press the Enter key to copy the package file from the server to the TSSC. 11. If prompted for the password for the ptadmin account, type the password for the account you obtained in step 5 on page 30 (or, if the password has not changed from the default value, type ptadmin) and press Enter. 12. Wait for the file transfer status in the SSH window to indicate that the file status is complete. 13. At the SSH command prompt, type exit and press Enter to close the SSH session. 14. Right-click the TSSC desktop and then click Console Actions > Console Configuration Utility. 15. If prompted for a user ID and password, enter service and service, respectively. 16. In the TS3000 System Console icon window: a. Click PE Packages. b. Click Subsystem Log Retrieval. c. On the menu in the Select Logs area, click one of the following machine types (as appropriate for the environment in which you are working): v 3958DD1 v 3958DD3 v 3958DD4 v 3958DD5 v 3958AP1 Note: The machine type selections are limited to the machine types that are listed in the Attached Systems list. (For example, if there are only 3958DD3s in the Attached Systems list, that is the only machine type option in the list.) d. In the Action area, click Click here. e. If you specified a target directory different from the default in step 9 on page 30: 1) In the Specify a different offload directory field, type the file path of the directory you specified in step 9 on page 30. 2) Click Refresh. Chapter 2. System troubleshooting tools 31

56 f. In the File Name list, select the check box for the package file. g. Click one of the following offload options: v v v Copy to universal serial bus (USB) Device Copy to compact disc (CD) Copy to Call Home h. Follow the on-screen instructions for the choice you made in step 16g. Software call home events ProtecTIER software Call Home events are recorded in the Alerts Log. Table 3. ProtecTIER software Call Home events Error ID Description Error message 5060 The Cluster Manager is fencing the peer node The Cluster Manager detected too many recent reboots. The Cluster Manager is fencing the peer node because... The Cluster Manager detected that there have been more than value unscheduled restarts in the last value minutes. It has stopped the process of bringing up the services to prevent endless reboot cycles File access error. File access error: value Storage unit is now in error. Storage unit value is now in error value The Cluster Manager detected that the virtual tape library (VTL) has died. The Cluster Manager detected that the VTL has died. The Cluster Manager will not reboot this server because there have already been value unscheduled restarts in the last value minutes Internal error. Internal error value in file value line value Distributed file service (Dfs) Statistics failed too many times. Dfs Statistics failed too many times value Storage unit is in read only mode. Storage unit value is in read only mode value Display file (DSPF) recovery failed, entering read only mode The Cluster Manager is exiting, node will be rebooted Vtf data cyclic redundancy check (CRC) error Decompression failed in service terminal manager (STM). DSPF in read only mode name = value. The Cluster Manager is exiting because value, the node will be rebooted. Vtf data cyclic redundancy check (CRC) error: value. Decompression failed in store word with update (STU) [value] on logical block address (LBA) value Reserved for future use A fence event occurred as a result of a fence test that is part of validation, and not due to a real problem. Fence Test: fencing node node name 9521 ProtecTIER did not start cluster services because it failed to validate the proper connections to the WTI power switch. Fence Port Test: description of problem 32 TS7650 and TS7650G Problem Determination and Service Guide

57 Using SNMP traps About this task In the event of hardware or software degradation or failure, ProtecTIER systems which are configured to use Simple Network Management Protocol (SNMP) can send a problem notification to designated recipients. SNMP notifications, or traps, can be sent even if the ProtecTIER Manager interface is unavailable. To use SNMP traps you need the following items: v SNMP trap receiver software installed on an SNMP trap server. Follow the instructions from the manufacturer to install and configure the SNMP trap receiver software. v The file name and location of the management information base (MIB) file for the SNMP trap receiver. On the ProtecTIER server, the file name is: IBM-TS7600-SNMP-MIBV2.mib located in: /usr/share/snmp/mibs. The full path is: /usr/share/snmp/mibs/ibm-ts7600-snmp-mibv2.mib. v The IBM-TS7600-SNMP-MIBV2.mib file needs to be copied onto the SNMP trap receiver and the trap receiver software must point to the directory location of the MIB file for translation of the trap messaging. v SNMP trapping enabled on one or more of your ProtecTIER servers. Use the ProtecTIER Manager Configuration wizard to enable the SNMP trap option on 3959 SM1 servers. See the IBM System Storage TS7620 ProtecTIER Deduplication Appliance Express Installation and Setup Guide for VTL, and OpenStorage Systems, GA for instructions on SNMP configuration. For 3958 AP1 or 3958 DD4 servers, see the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA The ProtecTIER servers have the following improvements in SNMP support. v ProtecTIER software events that send specific notifications based on the error that occurred. v ProtecTIER hardware events that trigger specific notifications are based on the error that occurred, such as a CPU event or power event. v Send enough detailed information with the SNMP notification so that you can understand the problem. The ProtecTIER Manager Configuration menu gives you the option to filter SNMP traps based on severity. Error-level severities can be filtered by: - Error - Warning - Information - Software error categories include: v VTL v Replication v OpenStorage v Repository storage v Cluster v System - Hardware error categories include: v CPU memory module v Cooling module (fan) Chapter 2. System troubleshooting tools 33

58 v v v v v v v v v v Internal boot drives Ethernet cards Power supplies RAID card RAID battery Front end adapter, if VTL enabled General server errors General network errors v Ethernet switch, if cluster enabled with SMC switch (TS7650 or TS7650G only) v Network power switch, if cluster enabled with new network power switch (TS7650 or TS7650G only) v 3958 AP1 specific v Back end adapter (TS7650 or TS7650G only) v Disk controller (TS7650 or TS7650G only) v Disk expansion (TS7650 or TS7650G only) v 3959 SM1 specific v SAS expander v SATA hard disk drives Warning-level severity includes: - Replication warnings - VTL warnings - OpenStorage warnings - Capacity warnings - RAS warnings Information-level severity includes: - VTL configuration change events - OpenStorage configuration change events - Replication events SNMP in ProtecTIER version 3.1 or later supports threshold monitoring and allows the user to specify thresholds for the following system runtime behavior: Repository space issues - Nominal capacity - Physical capacity There are two threshold levels a user can set: - Information level: a trap is sent when the repository regains free space and rises about the information level. - Warning level: a trap is sent when the free space in the repository falls below the warning level Going below the informational threshold issues an SNMP trap only if the warning threshold has been crossed. This method is to ensure that the user is not flooded with alerts when the normal operation crosses the low water mark threshold frequently. Capacity thresholds can be set specifying % from the repository or specifying space (GBs). Using an IBM-registered management information base (MIB) file. 34 TS7650 and TS7650G Problem Determination and Service Guide

59 v v The MIB file is implemented in a tree structure and has a unique OID for each message supported. The MIB file ships on the ProtecTIER server. Provide reporting to the network management application software. Improved communication options: SNMP traps are sent through the customer network (eth0) by using the UDP protocol. By default, port 162 is used and up to five destinations are supported. Customers can optionally select a different port for SNMP traffic by using theprotectier Manager Configuration menu. On systems configured to use SNMP traps, an agent monitors the ProtecTIER server and reports fault information to a network management application. Periodically the data is sent to the designated SNMP server in the form of an SNMP trap report, a portion of which is shown in Figure 9 on page 36. SNMP trap reports allow you to receive hardware or software fault notifications whether or not you have access to the ProtecTIER Manager interface. The display format of the trap report varies between different trap receiver software applications. Your trap report might not look exactly like the following example. Chapter 2. System troubleshooting tools 35

60 ts Figure 9. SNMP trap report Procedure 1. Run the SNMP trap catcher program on the SNMP server. 2. Open the SNMP report and review the information. 3. Make note of the defective component's name (or type) and FRU ID. Note: The FRU ID (part number) for both CRU and FRU components, is expressed as a FRU ID. 4. Verify the alert received was not caused by an easily-resolved condition, such as a loose power cord or a defective cable. 36 TS7650 and TS7650G Problem Determination and Service Guide

61 Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) Component labeling This section provides detailed information about the 3958 DD1 and 3958 DD3 ProtecTIER servers, such as explanations of the controls and indicators. For detailed information about the 3958 DD4, 3958 DD5, and 3958 AP1 ProtecTIER servers, see Chapter 4, ProtecTIER server (3958 DD4, DD5, and AP1), on page 73. The colors of component labels indicate different types of components. A blue label on a component or near a component indicates touch points, where you can grip the component to remove it from or install it in the server, open or close a latch, and so on. An orange label on a component or near a component indicates that the component can be hot-swapped. This indicates that if the server and operating system support hot-swap capability, you can remove or install the component while the server is running. Note: Orange can also indicate the touch points on hot-swap components. Use the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide for removing or installing a specific hot-swap component for any additional procedures that you might have to perform before you remove or install the component. Note: Adapters in the TS7650 (including the hot-swap adapter ports) are not hot-swap capable due to the limitations of the Linux operating system. The system must be shut down when an adapter needs to be replaced. Copyright IBM Corp. 2011,

62 Power, controls, and indicators The following sections describe the controls and indicators for the 3958 DD3 or 3958 DD1 server. Front view This topic describes the front view of the 3958 DD3/3958 DD1 server. Hard disk drive activity LED Hard disk drive status LED USB connectors DVD-eject button DVD drive activity LED Operator information panel Scalability LED Electrostatic-discharge connector ts Figure 10. Server front view Note: Scalability light-emitting diode (LED) is not used. Hard disk drive activity light-emitting diode (LED): When this LED is flashing, it indicates that the drive is in use. Hard disk drive status LED: When this LED is lit continuously, that individual drive is faulty. When the drive is connected to the integrated serial attached SCSI (SAS) controller with RAID capabilities, a flashing status LED indicates that the drive is a secondary drive in a mirrored pair and the drive is being synchronized. DVD-eject button: Press this button to release a CD from the DVD-ROM drive. Note: The drive does not eject the CD until issued a umount /mnt command. DVD drive activity LED: When this LED is lit, it indicates that the DVD-ROM drive is in use. Operator information panel: This panel contains controls, LEDs, and connectors. Figure 11 on page 39 shows the controls, LEDs, and connectors on the operator information panel. 38 TS7650 and TS7650G Problem Determination and Service Guide

63 Power-control button/power-on LED Ethernet icon LED Information LED System-error LED 1 2 Power-control button cover Ethernet port activity LEDs Figure 11. Server operator information panel Locator button/locator LED The following controls and LEDs are on the operator information panel: v Power-control button cover: Slide this cover over the power-control button to prevent the server from being turned off accidentally. v Power-control button: Press this button to turn the server on and off manually. v Power-on LED: When this LED is lit and not flashing, it indicates that the server is turned on. When this LED is flashing, it indicates that the server is turned off and still connected to an ac power source. When this LED is off, it indicates that ac power is not present or the power supply or the LED itself has failed. ts v v v v v v Note: If this LED is off, it does not mean that there is no electrical power in the server. The LED might be burned out. To remove all electrical power from the server, you must disconnect the power cords from the electrical outlets. Ethernet-icon LED: This LED lights the Ethernet icon. Ethernet activity LEDs: When these LEDs flash, they indicate that there is activity between the server and the network on the indicated port. Locator LED: Use this LED to visually locate the server among other servers. You can use the System Locator LED button under System Status in a remote supervisor adapter (RSA) session to light this LED remotely; see Appendix A, Logging into the 3958 DD1/3958 DD3 server through the RSA connection, on page 255 for the procedure to open a remote supervisor adapter (RSA) session. To light the LED manually, press the locator button. This LED is also lit during startup. In clustered configurations, when this LED flashes, it indicates that the server is the primary server. When this LED is lit continuously, it indicates that the server is the secondary server. Locator button: Press this button to turn the locator LED on and off manually. Information LED: When this LED is lit, it indicates that a suboptimal condition exists in the server. Light path diagnostics lights an additional LED to help isolate the condition. This LED and LEDs on the light path diagnostics panel remain lit until you resolve the condition or you press the remind button. System-error LED: When this LED is lit, it indicates that a system error has occurred. An LED on the light path diagnostics panel is also lit to help isolate the error. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 39

64 Rear view of the 3958 DD3/DD1 server This topic describes the rear view of the 3958 DD3/DD1 server. USB SAS Power-on LED Gigabit Ethernet 2 LED Gigabit Ethernet 2 System-error LED Locator LED Gigabit Ethernet 1 LED Gigabit Ethernet 1 Remote Supervisor Adapter II Power supply 1 AC Power DC Power Powersupply error Powersupply 2 System serial SMP Expansion Port 1 link LED SMP Expansion Port 1 SMP Expansion Port 2 link LED SMP Expansion Port 2 Figure 12. Server rear view SMP Expansion Port 3 SMP Expansion Port 3 link LED ts Note: Symmetric multiprocessing (SMP) Expansion Port 1, SMP Expansion Port 2, and SMP Expansion Port 3 are not used. Power-on LED When this LED is lit and not flashing, it indicates that the server is turned on. When this LED is flashing, it indicates that the server is turned off and still connected to an ac power source. When this LED is off, it indicates that ac power is not present or the power supply or the LED itself has failed. Note: An LED that is off can also indicate a burned out LED. To remove all power from the server, you must disconnect the power cords from the electrical outlets. Locator LED Use this LED to visually locate the server among other servers. You can use IBM Director to light this LED remotely or press the locator button to light the LED manually. This LED is also lit during startup. System-error LED When this LED is lit, it indicates that a system error has occurred. An LED on the light path diagnostics panel is also lit to help isolate the error. Gigabit Ethernet 2 LED When this LED flashes, it indicates activity between the server and the network. When this LED is lit continuously, it indicates that there is an active connection on the Ethernet port. Gigabit Ethernet 2 connector Use this connector to connect the server to a network. 40 TS7650 and TS7650G Problem Determination and Service Guide

65 Gigabit Ethernet 1 LED When this LED flashes, it indicates activity between the server and the network. When this LED is lit continuously, it indicates an active connection on the Ethernet port. Gigabit Ethernet 1 connector Use this connector to connect the server to a network. This connector is shared with the baseboard management controller and is assigned two Media Access Control (MAC) addresses. Power supply 1 connector Connect the power cord to this connector. Power supply 2 connector Connect the power cord to this connector. System serial connector Connect a 9-pin serial device to this connector. Serial attached SCSI (SAS) connector Connect an internal SAS device to this connector. Universal serial bus (USB) connectors Connect USB devices to these connectors. ac power LED This green LED provides status information about the power supply. During typical operation, both the ac and dc power LEDs are lit. For any other combination of LEDs, see Power supply error LEDs on page 43. dc power LED This green LED provides status information about the power supply. During typical operation, both the ac and dc power LEDs are lit. For any other combination of LEDs, see Power supply error LEDs on page 43. Power-supply error LED When this amber LED is lit, it indicates an error condition within the power supply. It is also used, in combination with the ac and dc power LEDs, to provide more information about error conditions (see Power supply error LEDs on page 43). Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 41

66 Remote supervisor adapter (RSA) controls, connectors, and indicators The controls, connectors, and LEDs on the rear panel of the RSA are used for systems-management information and control. Adapter activity LED Power LED ASM connector Reset button (recessed) Mini-USB connector External power supply connector Ethernet connector (RJ-45) Video connector ts Figure 13. RSA controls, connectors, and indicators Adapter activity light-emitting diode (LED) This LED can be flashing, lit continuously, or off. LED state Flashing Lit continuously Off Explanation The RSA is functioning normally. Indicates a problem with the RSA. The RSA is not functioning. Power LED When this LED is lit, the RSA is receiving power from the server or from an external power supply. Reset button Trained service technician only: To manually reset the RSA, insert the open end of a paper clip (or similar object) into the recessed reset button and press. Mini-USB connector This connector is not supported. Video connector Use this connector to connect the server monitor. 42 TS7650 and TS7650G Problem Determination and Service Guide

67 Ethernet connector (RJ45) Use this connector to connect a Category 3 (10 Mbps) or Category 5 (100 Mbps) Ethernet cable to enable a local area network (LAN) connection. External power-supply connector Use this connector to connect an external power-supply to the RSA. Advanced system management (ASM) connector This connector is not supported. Power supply error LEDs Combinations of error LEDs on the rear of the server power supply indicate different error conditions. When the amber LED on the rear panel of a server power supply (Figure 14) is lit, it indicates an error condition within the power supply. Note: If a power supply fails, it shows up in System Health Monitoring as a Power Supply 1 or 2 good fault. Replacing the power supply restores the power, but does not clear the problem for System Health Monitoring. To clear the error, refer to Clearing system errors after power supply replacement on page 45. The specific error condition is indicated by combinations of four LEDs. One of these LEDs is the power indicator on the operator information panel. The other three are the indicators on the rear panel of the power supply (Table 4) ts Figure 14. Power supply rear view Table 4. Power supply indicators Number Indicator Color 1 ac power Green 2 dc power Green 3 Power supply error Amber To interpret the combinations displayed by these four LEDs, refer to Table 5 on page 44. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 43

68 Table 5. Power-supply LED combinations Operator information panel power LED Power-supply LEDs ac dc Error Description Off Off Off Off No alternating current (ac) power to the server, or a problem with the ac power source. Off Lit Off Off Direct current (dc) source power problem or system error Off Lit Lit Off Standby power problem or server is turned off Action 1. Check the ac power to the server. 2. Make sure that the power cord is connected to a functioning power source. 3. Check the Western Telematic Inc. (WTI) power ports. v For Node A, 1 and 5 should be lit. v For Node B, 2 and 6 should be lit. If the power ports are not lit, the node has been fenced. See Appendix H, Checking the Western Telematic Inc. power ports, on page 291 to restore power. Note: On newer versions of the WTI switch, the LEDs are not visible unless you remove the side panel. If this is not possible, use Telnet to connect a working node to the WTI switch (IP address ) and view the port status. 4. Remove one power supply at a time. 1. Make sure that the microprocessor tray is connected to the power backplane. 2. Reseat one power supply at a time. 3. (Trained service technician only) Replace the power backplane. 4. View the system-error log. 1. Press the power-control button on the operator information panel. 2. View the system-error log. 3. Isolate by removing one power supply at a time. 4. (Trained service technician only) Replace the power backplane. 44 TS7650 and TS7650G Problem Determination and Service Guide

69 Table 5. Power-supply LED combinations (continued) Operator information panel power LED Power-supply LEDs ac dc Error Description Flashing Lit Lit Off System power-on problem Lit or off Lit Lit or off Lit Internal power supply fault (for example, thermal fault, or over-voltage or under-voltage condition) Lit Lit Lit Off Power is available Action 1. View the system-error log. 2. Press the power-control button on the operator information panel. 3. (Trained service technician only) Use the force-power-on jumper as a debugging aid to determine whether the information panel switch and cable are faulty. 4. Remove the remote supervisor adapter (RSA) and try to turn on the server. 5. (Trained service technician only) Reseat the microprocessor tray. 6. (Trained service technician only) Replace the microprocessor tray. 1. View the system-error log. 2. Replace the power supply. No action. Note: See the Removing and replacing server components chapter in the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide to determine which components a field engineer should replace. Clearing system errors after power supply replacement If a power supply fails, it shows up in system Health Monitoring as a Power Supply 1 or 2 good fault. Replacing the power supply restores the power, but does not clear the problem for system Health Monitoring. Procedure To clear the system error, perform the following steps. 1. From the ProtecTIER Service menu, select Health Monitoring > Reset RSA/IMM. The RSA card resets. This takes about a minute. Command returns when the reset command is issued but not complete yet. 2. Display a list of open problems. v For ProtecTIER V2.5 or earlier, select Back > Problem management > Process Problems > List open problems. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 45

70 v For ProtecTIER V3.1 or later, select Health Monitoring > List open problems. 3. On a sheet of paper, record the single-digit ProblemNumber associated with the power supply problem. (Example: If the power supply problem is the second open problem listed, the ProblemNumber is 2.) 4. Cancel the problem. v For ProtecTIER V2.5 or earlier, Select Cancel Problem. Enter the ProblemNumber to cancel the problem. v For ProtecTIER V3.1 or later, select Health Monitoring > List open problems. Follow the prompts to list open problems and to cancel the power supply problem. Identifying problems using status LEDs Diagnostics You can identify server problems using the status light-emitting diodes (LEDs) on the front of the server. The server is designed so that any LEDs that are lit remain lit when the server shuts down, as long as the ac power source is operable, and the power supply can provide +5 V dc. See Power supply error LEDs on page 43 for troubleshooting with the power supply and power-on LEDs. If the system-error LED on the front of the server is on, one or more LEDs inside the server or on the power supply are on. The server has LEDs to help you identify problems with some server components. These LEDs are part of the light path diagnostic feature built into the server. By following the light path, you can quickly identify the type of system error that occurred. See Light path diagnostics on page 54. The topics in this section provide basic troubleshooting information to help you resolve some common problems that might occur with the server. General checkout procedure Perform this task to diagnose hardware problems in the subcomponents of the ProtecTIER server hardware. Before you begin Review the following information before performing the checkout procedure: v The server diagnostic programs are stored in upgradeable read-only memory (ROM) on the system board. These programs provide the primary methods of testing the major components of the server. If you are not sure whether a problem is caused by the hardware or by the software, you can run the diagnostic programs. See Diagnostic programs and error messages on page 50 to confirm whether the hardware is working correctly. v When you run the diagnostic programs, a single problem might cause several error messages. If you receive several error messages, correct the cause of the first error message. The other error messages might not occur the next time that you run the diagnostic programs. Important: 46 TS7650 and TS7650G Problem Determination and Service Guide

71 v v v v If the server is suspended and a power-on self-test (POST) error code is displayed, see Power-on self-test error logs on page 49. If the server is suspended and no error message is displayed, see the Troubleshooting tables and Solving undetermined problems sections in the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. For information about power-supply problems, see 3958 DD3/3958 DD1 server power checkout on page 63. For intermittent problems, check the error logs; see Diagnostic programs and error messages on page 50. About this task Complete the following steps to perform the checkout procedure to identify system problems. Procedure 1. Check all cables and power cords. 2. Record any power-on self test (POST) error messages that are displayed on the screen. If an error is displayed, look up the first error in the POST error codes section of the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. 3. Check the system-error light-emitting diode (LED) on the operator information panel. If it is on, see Light path diagnostic light-emitting diodes on page Check the remote supervisor adapter (RSA) event log. If any errors or warnings or any relevant informational messages are recorded, see the appropriate section of the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. Access the RSA via the TS3000 System Console (TSSC) browser using the RSA IP address xxx. For an alternate procedure to log in to the RSA, see Appendix A, Logging into the 3958 DD1/3958 DD3 server through the RSA connection, on page Check for any open problems posted by the Problem Manager See Problem Manager on page 17 and use the rscerpmgetopenproblems command to display any open problems. 6. Start the diagnostic programs. Refer to Starting the diagnostic programs on page Check for the following responses: v One beep v Readable instructions or the main menu What to do next Did you receive both correct responses? v NO. Find the failure symptom in the Symptom-to-field replaceable unit (FRU) index in the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. v YES. Run the diagnostic programs. If necessary, see Diagnostic programs and error messages on page 50. If you receive an error, see Symptom-to-field replaceable unit (FRU) index in the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 47

72 If the diagnostic programs were completed successfully and you still suspect a problem, see the Solving undetermined problems section in the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. Diagnostic tools overview Several tools are available to help you diagnose problems with the 3958 DD1/3958 DD3 server. The following tools are available to help you diagnose and solve hardware-related problems: Power-on self test (POST) beep codes, error messages, and error logs v The POST generates beep codes and messages to indicate successful test completion or the detection of a problem. v A beep code is a combination of short or long beeps or a series of short beeps separated by pauses. For example, a beep code is one beep, a pause, two beeps, a pause, and three beeps. When the POST is completed, one beep is emitted to indicate that the server is working correctly. If the POST detects a problem during startup, other beep codes might occur. Error messages also appear during startup if the POST detects a hardware configuration problem. Problem isolation tables Use these tables to help you diagnose various symptoms. Light path diagnostics Use the light path diagnostics to diagnose system errors quickly. Diagnostic programs and error messages The diagnostic programs are stored in memory on the microprocessor tray. These programs are the primary method of testing the major components of the server. For detailed information, refer to the IBM System x3850 M2 and System x3950 M2 Type 7141 and 7233 User's Guide. Note: After a remote supervisor adapter (RSA) field-replaceable unit (FRU) replacement, if you update server BIOS or firmware through the RSA using the Preboot Dynamic System Analysis (DSA) diagnostic program discussed in the IBM System x3850 M2 and System x3950 M2 Type 7141 and 7233 User's Guide, the RSA restarts. During the restart, you lose connection to the server through the RSA and must log back in after the restart completes. v For more information about the Preboot DSA diagnostic program, refer to the IBM System x3850 M2 and System x3950 M2 Type 7141 and 7233 User's Guide. v For the procedure to log into the server through the RSA, see Appendix A, Logging into the 3958 DD1/3958 DD3 server through the RSA connection, on page TS7650 and TS7650G Problem Determination and Service Guide

73 Power-on self-test error logs During power-on self-test (POST), a beep code indicates whether problems were detected. Any problems that occur during POST are recorded in the error logs. If the POST finishes without detecting any problems, a single beep sounds, and the first screen of your operating system or application program is displayed. If the POST detects a problem, more than one beep sounds, and an error message displays on your screen. See the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide for detailed failure information. Notes: v Before the POST continues, if you have a power-on password or an administrator password set, you must type the password and press Enter when prompted. (If you do not have the password, obtain it from the customer.) v A single problem can cause several error messages. If this occurs, work to correct the cause of the first error message and then run the test again. In many cases, the other error messages are also eliminated. The POST error log contains the three most recent error codes and messages that the system generated during the POST. The system-error log or baseboard management controller (BMC) log contains all messages issued during the POST and all system status messages from the service processor. You can view the contents of the system-error log and BMC log from the Configuration/Setup Utility program or from the diagnostic programs. Viewing error logs from the Configuration/Setup Utility program You can view error logs from the Configuration/Setup Utility program. Procedure 1. Start the Configuration/Setup Utility program and then select Error Logs from the main menu. 2. Select Error Logs from the main menu. What to do next See Configuring the server on page 234 for more information. Viewing error logs from diagnostic programs You can use diagnostic programs to view the diagnostics error logs. Procedure To view the diagnostics error logs, perform the following steps. 1. Start the diagnostic programs. 2. Select Hardware Info from the top of the diagnostic programs screen. 3. Select System Error Log or baseboard management controller (BMC) Log from the list that is displayed, then follow the instructions on the screen. See Starting the diagnostic programs on page 88 for more information. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 49

74 Diagnostic programs and error messages The server diagnostic programs are stored in upgradeable read only memory (ROM) on the system board. These programs are the primary method of testing the major components of your server. They display error messages to indicate that a problem exists. Diagnostic error messages are not intended to be used to identify a failing part. Troubleshooting and servicing of complex problems that are indicated by error messages should be performed by trained service personnel. Sometimes the first error to occur causes additional errors. In this case, the server displays more than one error message. Always follow the suggested action instructions for the first error message that appears. Error codes that might be displayed are listed in the Diagnostic error codes section in the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. Notes: v Depending on the server configuration, some of the error codes might not appear when you run the diagnostic programs. v If diagnostic error codes appear that are not listed in the tables, make sure that the server has the latest levels of basic input output system (BIOS), service processor, and ServeRAID code installed. Check the latest IBM System Storage RAS/BIOS and Firmware Update DVD to update to the latest levels. Diagnostic text messages Diagnostic text messages are displayed while the tests are running. A diagnostic text message contains one of the following results: Passed This test was completed without any errors. Failed This test discovered an error. User Aborted You stopped the test before it was completed. Not Applicable You attempted to test a device that is not present in the server. Aborted The test could not proceed because of the server configuration. Warning The test could not be run. There was no failure of the hardware that was being tested, but there might be a hardware failure elsewhere, or another problem prevented the test from running. (For example, there might be a configuration problem, or the hardware might be missing or not recognized.) The result is followed by an error code or other additional information about the error. Starting the diagnostic programs Complete this task to start the diagnostic programs. 50 TS7650 and TS7650G Problem Determination and Service Guide

75 Before you begin To perform this task, you need the following items: v Monitor v Universal serial bus (USB) keyboard v USB mouse v Associated cables Procedure Complete the following steps to start the diagnostic programs: 1. Connect the monitor to the video connector on the remote supervisor adapter (RSA) and connect the USB keyboard and mouse to universal serial bus (USB) ports on the back of the server. See Figure 15 and Figure 16 on page 52. USB SAS Power-on LED Gigabit Ethernet 2 LED Gigabit Ethernet 2 System-error LED Locator LED Gigabit Ethernet 1 LED Gigabit Ethernet 1 Remote Supervisor Adapter II Power supply 1 AC Power DC Power Powersupply error Powersupply 2 System serial SMP Expansion Port 1 link LED SMP Expansion Port 1 SMP Expansion Port 2 link LED SMP Expansion Port 2 Figure 15. Server rear view SMP Expansion Port 3 SMP Expansion Port 3 link LED ts Note: Symmetric multiprocessing (SMP) Expansion Port 1, SMP Expansion Port 2, and SMP Expansion Port 3 are not used. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 51

76 Adapter activity LED Power LED ASM connector Reset button (recessed) Mini-USB connector External power supply connector Ethernet connector (RJ-45) Video connector ts Figure 16. RSA controls, connectors, and indicators 2. Turn on the server and watch the screen. cvt00029 Figure 17. eserver logo screen for basic input/output system (BIOS) setup 3. When the message F2 for Diagnostics appears, press F2. 4. Type the appropriate password and then press Enter. 5. Select either Extended or Basic from the top of the screen. 6. When the diagnostic programs screen is displayed, select the test that you want to run from the list that appears, and then follow the instructions on the screen. 52 TS7650 and TS7650G Problem Determination and Service Guide

77 Notes: v You can press F1 while running the diagnostic programs to obtain help information. You also can press F1 from within a help screen to obtain online documentation from which you can select different categories. To exit from the help information and return to where you left off, press Esc. v If the server stops during testing and you cannot continue, restart the server and try running the diagnostic programs again. If the problem remains, replace the component that was being tested when the server stopped. v The keyboard and mouse (pointing device) tests assume that a keyboard and mouse are attached to the server. v If you run the diagnostic programs with either no mouse or a USB mouse attached to your server, you cannot navigate between test categories using the Next Cat and Prev Cat buttons. All other functions provided by mouse-selectable buttons are also available using the function keys. v You can test the USB keyboard by using the regular keyboard test. The regular mouse test can test a USB mouse. v You can run the USB interface test only if no USB devices are attached. v You can view server configuration information (such as system configuration, memory contents, interrupt request (IRQ) use, direct memory access (DMA) use, device drivers, and so on) by selecting Hardware Info from the top of the screen. What to do next If the diagnostic programs do not detect any hardware errors but the problem persists during normal server operations, a software error might be the cause. If you suspect a software problem, refer to the information that comes with the software package. Viewing the test log Complete this task in order to view the test log. Procedure Complete the following steps in order to view the test log after tests are completed. 1. Select Utility from the top of the screen and then select View Test Log. 2. View the test log or save the log to a file on a diskette or on the hard disk so that you can view it later by performing the following steps: a. Select Save Log on the diagnostic programs screen. b. Specify a location and name for the saved log file. Note: The test log data is maintained only while the diagnostic programs are active. When you exit from the diagnostic programs, the test log is cleared (saved test logs are not affected). What to do next For a detailed listing of diagnostic error messages, see Running DSA Preboot and DSA messages in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 53

78 Viewing the system error log or baseboard management controller (BMC) log Complete this task to view the system error log or baseboard management controller (BMC) log. You can view the system error log or BMC log from the diagnostic programs. See the instructions in the Viewing error logs from the Configuration/Setup Utility program and Viewing the BMC log from the diagnostic programs sections in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide for an explanation of the error codes. Light path diagnostics A system of light emitting diodes (LEDs) is used to convey the results from light path diagnostics. Light path diagnostics is a system of LEDs on various external and internal components of the server. When an error occurs, LEDs are lit throughout the server. By viewing the LEDs in a particular order, you can often identify the source of the error. The server is designed so that LEDs remain lit when the server is connected to an ac power source but is not turned on, provided the power supply is operating correctly. This feature helps you to isolate the problem when the operating system is shut down. Any memory card LED can be lit while the memory card is removed from the server so that you can isolate a problem. After ac power is removed from the server, power remains available to these LEDs for up to 24 hours. To view the memory card LEDs, press and hold the light path diagnostics button on the memory card to light the error LEDs. The LEDs that were lit while the server was turned on are lit again while the button is pressed. Many errors are first indicated by a lit information LED or system-error LED on the operator information panel on the front of the server. If one or both of these LEDs are lit, one or more LEDs elsewhere in the server might also be lit and can direct you to the source of the error. If an error occurs, view the LEDs in the following order: 1. Check the operator information panel on the front of the server. v If the information LED is lit, it indicates that information about a suboptimal condition in the server is available in the baseboard management controller (BMC) log or in the system-error log. Proceed to step 2 on page 55. v If the system-error LED is lit, a system error occurred. Go to step 2 on page TS7650 and TS7650G Problem Determination and Service Guide

79 Power-control button/power-on LED Ethernet icon LED Information LED System-error LED 1 2 Power-control button cover Ethernet port activity LEDs Figure 18. Operator information panel Locator button/locator LED 2. To view the light path diagnostics panel, press the release latch on the front of the operator information panel to the left, then slide it forward. This reveals the light path diagnostics panel. Lit LEDs on this panel indicate the type of error that occurred. ts OVER SPEC LOG LINK PS PCI FAN TEMP MEM NMI CNFG CPU VRM DASD RAID REMIND SP BRD NMI button (trained service technician only) Light Path Diagnostics ts Figure 19. Server light path diagnostics panel Note: (Trained IBM service technician only) The non-maskable interrupt (NMI) button is used for operating system debug purposes and causes the server to reset if pressed. Look at the system service label on the top of the server, which gives an overview of internal components that correspond to the LEDs on the light path diagnostics panel. This information and the information in Troubleshooting with light path diagnostics can often provide enough information to correct the error. Note: The checkpoint display, indicated by 88 on the light path diagnostic panel, provides checkpoint codes. Checkpoint codes are not error codes; they identify firmware checks when the server stops. There are field-programmable gate array (FPGA) hardware checkpoint codes and basic input/output system (BIOS) checkpoint codes. For more information, see the IBM System x3850 M2 (7141, 7144) Checkpoint listing web page. For a list of checkpoint codes, see Appendix C, Checkpoint codes, on page Remove the server cover and look inside the server for lit LEDs. Certain components inside the server have LEDs that are lit to indicate the location of a problem. See the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. Troubleshooting with light path diagnostics You can use light path diagnostics to diagnose problems that might occur during installation. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 55

80 About this task Light-emitting diodes (LEDs) in two locations on the server are available to help you diagnose problems: v Light path diagnostics panel v LEDs on the system board Procedure 1. Check the light path diagnostics panel first. If a system error has occurred, the system-error LED on the front of the light path diagnostics drawer is lit. Slide the latch to the left on the front of the light path diagnostics drawer to access the light path diagnostics panel. Note any LEDs that are lit, and then close the drawer. 2. Next, look at the LEDs on the system board to identify the component that is causing the error. Note the lit LED on or beside the component. What to do next Refer to the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide for additional detailed information. Light path diagnostic light-emitting diodes Light path diagnostics uses light-emitting diodes (LEDs) that indicate specific problems and associated actions. Table 6 on page 57 describes the LEDs on the light path diagnostics panel and on the boards inside the server, and lists the suggested actions to correct the detected problems. v Follow the suggested actions in the order in which they are listed in the Action column until the problem is solved. v If an Action step is preceded by Trained service technician only, that step must be performed only by a trained service technician. 56 TS7650 and TS7650G Problem Determination and Service Guide

81 Table 6. LEDs on the 3958 DD1/3958 DD3 light path diagnostics panel Lit light path diagnostics LED with the system-error LED or information LED also lit All light-emitting diodes (LEDs) are off (only the power LED is lit or flashing). All LEDs are off (the power LED is lit or flashing and the system-error LED is lit). OVERSPEC LOG Description A machine check has occurred. The server is identifying the machine check, the server was interrupted while identifying the machine check, or the server was unable to identify the machine check. There is insufficient power to power the system. The LOG LED might also be lit. Information is present in the baseboard management controller (BMC) log and system-error log. No action necessary. Action 1. Wait several minutes for the server to identify the machine check and restart. 2. (Trained service technician only) Extract the machine check data, which is used to identify the machine check. 1. Add a power supply if only one power supply is installed. 2. Use 220 volt (V) alternate current (AC) instead of 110 V AC. 3. Reseat the following components: a. Power supply b. Power backplane 4. Remove optional devices. 5. Replace the components one at a time, in the order shown, restarting the server each time. 1. Save the log if necessary and clear. 2. Check the log for possible errors. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 57

82 Table 6. LEDs on the 3958 DD1/3958 DD3 light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit LINK Description There is a fault in an symmetric multiprocessing (SMP) Expansion Port or SMP Expansion cable (requires scalability enablement). Note: 1. This LED remains lit until the problem is resolved and the server is turned off and restarted. 2. If a fault occurs, the SMP Expansion Port link LED on the failed port is off. Action 1. Check the SMP Expansion Port link LEDs to find the failing port or cable. 2. Reseat the SMP Expansion cables. 3. Replace the SMP Expansion cables. 4. (Trained service technician only) Replace the microprocessor tray. 58 TS7650 and TS7650G Problem Determination and Service Guide

83 Table 6. LEDs on the 3958 DD1/3958 DD3 light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit PS PCI SP Description A power supply has failed or has been removed. Note: In a redundant power configuration, the DC power LED on one power supply might be off. A peripheral component interface (PCI) adapter has failed. Note: The error LED next to the failing adapter on the I/O board shuttle is also lit. The remote supervisor adapter (RSA) has failed or is missing or the planar cable is not connected. Action 1. Reinstall the removed power supply. 2. Check the individual power-supply LEDs to find the failing power supply. 3. Reseat the following components: a. Failing power supply b. Power backplane 4. Make sure that the power cord is fully seated in the power-supply inlet and the AC power source. 5. Check the Western Telematic Inc. (WTI) power ports. v For Node A, 1 and 5 should be lit. v For Node B, 2 and 6 should be lit. If the power ports are not lit, the node has been fenced. See Appendix H, Checking the Western Telematic Inc. power ports, on page 291 to restore power. Note: On newer versions of the WTI switch, the LEDs are not visible unless you remove the side panel. If this is not possible, use Telnet to connect a working node to the WTI switch (IP address ) and view the port status. 6. Replace the components one at a time, in the order shown, restarting the server each time. 7. Disconnect the AC power cord for 20 seconds, then reconnect the AC power cord and restart the server. 1. See the BMC log or the system-error log. 2. Reseat the following components one at a time, in the order shown, restarting the server each time: a. Failing adapter b. I/O board shuttle 1. Reseat the RSA and planar cable. 2. Update the firmware for the RSA. 3. Replace the RSA. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 59

84 Table 6. LEDs on the 3958 DD1/3958 DD3 light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit FAN TEMP Description A fan has failed or has been removed. Note: A failing fan can also cause the TEMP LED to be lit. A system temperature or component has exceeded specifications. Note: A fan LED might also be lit. Action 1. Reinstall the removed fan. 2. If an individual fan LED is lit, replace the fan. Note: A failing fan might not cause the fan LED to be lit. 3. (Trained service technician only) Reseat the microprocessor tray. 4. (Trained service technician only) Replace the microprocessor tray. 1. See the BMC log or the system-error log for the source of the fault. 2. Make sure that the airflow of the server is not blocked. 3. If a fan LED is lit, reseat the fan. 4. Replace the fan for which the LED is lit. 5. Make sure that the room is neither too hot nor too cold. 6. If one of the voltage regulator modules (VRMs) indicates hot, remove AC power before you restore DC power. MEM Memory failure. 1. Remove the memory card that has a lit error LED, then press the light path diagnostics button on the memory card to identify the failed card or dual inline memory module (DIMM). 2. Reseat the DIMM. NMI CNFG A hardware error has been reported to the operating system. Note: The peripheral component interface (PCI) or memory (MEM) LED might also be lit. A configuration error has occurred. 1. See the BMC log and the system-error log. 2. If the peripheral component interface (PCI) LED is lit, follow the instructions for that LED. 3. If the memory (MEM) LED is lit, follow the instructions for that LED. 4. Restart the server. 1. Find the failing or missing component by checking the other light path diagnostics LEDs. 2. Make sure that the fans, power supplies, microprocessors, VRMs, and memory cards are installed in the correct sequence. 60 TS7650 and TS7650G Problem Determination and Service Guide

85 Table 6. LEDs on the 3958 DD1/3958 DD3 light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit Central processing unit (CPU) VRM Description A microprocessor has failed, is missing, or has been incorrectly installed. A DC-DC regulator has failed or is missing. Action 1. Check the BMC log or the system-error log to determine the reason for the lit LED. 2. Find the failing, missing, or mismatched microprocessor by checking the LEDs on the microprocessor board. 3. Reseat the following components: a. Failing microprocessor b. Microprocessor board 4. Replace the following components one at a time, in the order shown, restarting the server each time: a. (Trained service technician only) Failing microprocessor b. (Trained service technician only) Microprocessor board 1. Check the BMC log or the system-error log to determine the reason for the lit LED-for a voltage regulator module (VRM). 2. Find the failing or missing VRM by checking the LEDs on the microprocessor board. 3. Install any missing VRMs. 4. Reseat the following components: a. Failing VRM b. Microprocessor associated with the VRM c. Microprocessor board 5. Replace the following components one at a time, in the order shown, restarting the server each time: a. Failing VRM b. (Trained service technician only) Microprocessor associated with the VRM c. (Trained service technician only) Microprocessor board Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 61

86 Table 6. LEDs on the 3958 DD1/3958 DD3 light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit DASD RAID BRD Description A hard disk drive has failed or has been removed. Note: The error LED on the failing hard disk drive is also lit. The RAID controller has indicated a fault. The microprocessor board or I/O board has failed. Action 1. Reinstall the removed drive. 2. Reseat the following components: a. Failing hard disk drive b. Serial attached SCSI (SAS) hard disk drive backplane c. SAS signal cable d. I/O board shuttle assembly 3. Replace the components listed in the step above one at a time, in the order shown, restarting the server each time. 1. Check the BMC log or the system-error log for information. 2. Reseat the following components: a. RAID controller, if possible b. Hard disk drives c. I/O board shuttle assembly 3. Replace the components listed in the step above one at a time, in the order shown, restarting the server each time. 1. Find the failing board by checking the LEDs on the microprocessor board and I/O board. 2. Reseat the failing board 3. Replace the failing board 62 TS7650 and TS7650G Problem Determination and Service Guide

87 Server power features A master power switch is located in the front on the upper right side of the cabinet. Attention: Use the master power switch to turn cabinet power on and off completely. v When the server is turned off, the operating system does not run, and all core logic except for the service processor is shut down. v However, if the switched-off server is connected to an alternating current (ac) power source, it can still respond to requests from the service processor (also called the baseboard management controller). The power-on light-emitting diode (LED) flashes to indicate that the server is connected to ac power but is not turned on DD3/3958 DD1 server power checkout Power problems can be difficult to solve; for example, a short circuit can exist anywhere on any of the power distribution buses. A short circuit usually causes the power subsystem to shut down because of an overcurrent condition. About this task Note: Use this procedure only along with your next level of support. Procedure A general procedure for troubleshooting power problems is as follows: 1. Turn off the server, and disconnect all alternate current (ac) power cords. 2. Check for loose cables in the power subsystem. 3. Check for short circuit causes. For example, check to see whether a loose screw might have caused a short circuit on a circuit board. 4. Remove adapters and disconnect the cables and power connectors to all internal and external devices until the server is at the minimum configuration required to start. See the Solving undetermined problems section in the IBM System x3850 M2 and System x3950 M2 Type 7141 and 7233 User's Guide for the minimum configuration. 5. Reconnect all ac power cords and turn on the server. v If the server starts up successfully, replace adapters and devices one at a time until the problem is isolated. v If the server does not start up from the minimal configuration, replace field replaceable units (FRUs) of minimal configuration one at a time until the problem is isolated. What to do next v For specific power-supply problems, see Power supply error LEDs on page 43. v For light panel information, see Light path diagnostics on page 54. Ethernet controller troubleshooting Use the following guidelines to find solutions to 10/100/1000 Mbps Ethernet controller problems that have definite symptoms. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 63

88 v v v v Make sure that the correct device drivers are installed in the server and are at the latest level. Make sure that the Ethernet cable is installed correctly. The cable must be securely attached at all connections. If the cable is attached but the problem remains, try a different cable. If you set the Ethernet controller to operate at 100 Mbps or higher, you must use Category 5 cabling. For 1000 Mbps, use Category 5e or above. If you directly connect two servers (without a hub), or if you are not using a hub with X ports, use a crossover cable. To determine whether a hub has an X port, check the port label. If the label contains an X, the hub has an X port. Determine whether the hub supports auto-negotiation. If it does not, try configuring the integrated Ethernet controller manually to match the speed and duplex mode of the hub. Check the Ethernet controller light-emitting diodes (LEDs) on the rear panel of the server. These LEDs indicate whether there is a problem with the connector, cable, or hub. See Figure 20. Figure 20. Ethernet controller LEDs 64 TS7650 and TS7650G Problem Determination and Service Guide

89 Table 7. Disk controller front indicators Number Label Indication Meaning 1, 3 ACT/LNK Green on (not flashing) The adapter is connected to a valid link partner. Green on flashing Off Off Green Yellow Orange flashing Data activity No link 10 Mbps 100 Mbps 1000 Mbps Identity. Use the "Identify Adapter" button in Intel PROSet to control blinking. See Intel PROSet Help for more information. 2, 4 10=OFF 100=GRN 1000=YLW v v v The Ethernet link status LED is lit when the Ethernet controller receives a link pulse from the hub. If the LED is off, there might be a defective connector or cable or a problem with the hub. The Ethernet transmit/receive activity LED is lit when the Ethernet controller sends or receives data over the Ethernet network. If the Ethernet transmit/receive activity light is off, make sure that the hub and network are operating and that the correct device drivers are installed. Check the local area network (LAN) activity LED on the rear of the server. The LAN activity LED is lit when data is active on the Ethernet network. If the LAN activity LED is off, make sure that the hub and network are operating and that the correct device drivers are installed. Check for operating-system-specific causes of the problem. Make sure that the device drivers on the client and server are using the same protocol. If the Ethernet controller still cannot connect to the network but the hardware appears to be working, the network administrator must investigate other possible causes of the error. If the Ethernet adapter is defective, see to Replacing a Dual-Port or Quad-Port Ethernet card and recovering Ethernet settings on page 196. Server power-on password override If the power-on password for the server is lost or forgotten, regain access to the server by performing any of the following actions. v If an administrator password is set, type the administrator password at the password prompt. Start the Configuration/Setup Utility program and reset the power-on password. v Remove the battery from the server and then reinstall it. For instructions for removing the battery, see the Removing the battery section in the IBM System x3850 M2 and System x3950 M2 Type 7141 and 7233 User's Guide. v Change the position of the power-on password override jumper (J33 on the I/O board) to bypass the power-on password check. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 65

90 Attention: Before changing any switch settings or moving any jumpers, turn off the server, then disconnect all power cords and external cables. Do not change settings or move jumpers on any system-board switch or jumper blocks that are not shown in the IBM System x3850 M2 and System x3950 M2 Type 7141 and 7233 User's Guide. Figure 21 shows the location of the power-on password override, force power-on, and Wake on LAN (WOL) bypass jumpers. Power-on password (J33) Wake on Lan bypass (J38) Force power-on (J32) ts Figure 21. Jumpers on I/O board (3958 DD3) Passwords From the User Security menu choice, you can set, change, and delete a power-on password and an administrator password. The User Security choice is on the full Setup utility menu only. From the User Security menu choice, you can set, change, and delete a power-on password and an administrator password. The User Security choice is on the full Setup utility menu only. If you set only a power-on password, you must type the power-on password to complete the system startup and to have access to the full Setup utility menu. An administrator password is intended to be used by a system administrator; it limits access to the full Setup utility menu. If you set only an administrator password, you do not have to type a password to complete the system startup, but you must type the administrator password to access the Setup utility menu. If you set a power-on password for a user and an administrator password for a system administrator, you can type either password to complete the system startup. A system administrator who types the administrator password has access to the full Setup utility menu; the system administrator can give the user authority to set, change, and delete the power-on password. A user who types the power-on 66 TS7650 and TS7650G Problem Determination and Service Guide

91 System recovery password has access to only the limited Setup utility menu; the user can set, change, and delete the power-on password, if the system administrator has given the user that authority. Complete this task to recover the server to an operational state. About this task Testing a clustered system The general procedure to recover the server to an operational state is as follows. (For specific procedures for the TS7650 Appliance, refer to IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA ) Procedure 1. Reinstall Red Hat Linux. 2. Configure the system settings to match those recorded at initial installation. After a recovery and after using the ptconfig command to successfully configure both nodes in the cluster, run the system validation tests. About this task The validation tests perform checks on the Red Hat Linux cluster, including fencing the other node, and waits for the system to reboot. This test cannot be performed on a stand-alone configuration. Important: Extensive discrepancies in the system time set on each server in the cluster can result in unnecessary fencing between the nodes during system validation testing. To avoid this, it is recommended that you synchronize the servers' system times before you start the test. Refer to Syncing the system times for instructions. Note: In a replication environment, the procedures in this section should be performed on the clustered servers at both the source and the destination locations. Syncing the system times You can set the date, time, and time zone manually or sync with a timeserver. Procedure 1. If you are not already connected to a node, plug a USB keyboard and monitor into the back of the server. 2. Log in to the server with the username ptadmin and password ptadmin. 3. Access the /install directory by typing the command: cd /opt/dtc/install 4. Perform either of the following actions to begin the set clock configuration: v From the command line, type./ptconfig -setclock v From the ProtecTIER Service menu (see ProtecTIER Service menu on page 16), select ProtecTIER Configuration > Update Time, Date, Timezone and Timeserver(s) The following menu is displayed: Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 67

92 Date, Time, Timezone & Timeserver(s) configuration ================================================== 1. Set date & time 2. Set Timezone 3. Set Timeserver(s) c. Commit changes and exit q. Exit without committing changes Please Choose: 5. Synchronize the date and time by typing 1 in the Please Choose: field. The following is displayed: Please Choose:1 Please specify the date in DD/MM/YYYY format [08/11/2010]: 6. Type the date in the specified format and press Enter. If you do not enter a date, the default (displayed in brackets) is used. The following is displayed: Please specify the date in DD/MM/YYYY format [08/11/2010]: 08/11/2010 Please specify the time in HH:MM:SS format [11:56:16]: 7. Type the time in the specified format and press Enter. If you do not enter a time, the default (displayed in brackets) is used. The following is displayed: Date, Time, Timezone & Timeserver(s) configuration ================================================== 1. Set date & time 2. Set Timezone 3. Set Timeserver(s) c. Commit changes and exit * q. Exit without committing changes Please Choose: Note: If you typed in new information, an asterisk (*) appears at the end of the c. Commit changes and exit menu option to show that there are modifications to be saved. You can choose either c to save the changes and exit the setclock configuration, choose q to exit the configuration without saving the changes, or continue synchronizing the local timezone (step 5) or adding the timeservers to the system (step 12 on page 69). 8. Synchronize the local timezone by typing 2 in the Please Choose: field. The following is displayed: Please Choose:2 Enter a 2 letter country code (or type m to enter the timezone manually): If you typed m to manually enter the timezone, the following is displayed: Enter the time zone (case sensitive): 9. Select a country. v Type a two-letter country code (for example, US for the United States) 68 TS7650 and TS7650G Problem Determination and Service Guide

93 v If you chose to enter the timezone manually, type the name (for example, Italy). If you need a list of time zone codes from countries other than the United States, see Appendix K, Worldwide time zone codes, on page 303. The following is displayed: Time zones under US: ==================== 1. America/New_York 2. America/Detroit 3. America/Kentucky/Louisville 4. America/Kentucky/Monticello 5. America/Indiana/Indianapolis 6. America/Indiana/Vincennes 7. America/Indiana/Knox 8. America/Indiana/Winamac 9. America/Indiana/Marengo 10. America/Indiana/Vevay 11. America/Chicago 12. America/Indiana/Tell_City 13. America/Indiana/Petersburg 14. America/Menominee 15. America/North_Dakota/Center 16. America/North_Dakota/New_Salem 17. America/Denver 18. America/Boise 19. America/Shiprock 20. America/Phoenix 21. America/Los_Angeles 22. America/Anchorage 23. America/Juneau 24. America/Yakutat 25. America/Nome 26. America/Adak 27. Pacific/Honolulu Please choose a timezone: 10. Type a number corresponding to the timezone you want to synchronize from the list under the country code entered. The following is displayed: Date, Time, Timezone & Timeserver(s) configuration ================================================== 1. Set date & time 2. Set Timezone 3. Set Timeserver(s) c. Commit changes and exit * q. Exit without committing changes Please Choose: 11. You can choose either c to save the changes and exit the setclock configuration, choose q to exit the configuration without saving the changes, or continue synchronizing the local timezone or adding the timeservers to the system. 12. Add timeservers to the system by typing 3 in the Please Choose: field. The following is displayed: a. Please specify the timeserver's IP Address: Type the timeserver's IP address. For example: b. Would you like to set a secondary timeserver? (yes no) Type y to set a secondary timeserver. c. Please specify the secondary timeserver's IP Address: Type the secondary timeserver's IP address. For example: Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 69

94 You can search your region for the appropriate NTP timeserver you want to synchronize your system with using your internet search engine. There are several sites available, one example is the NIST Internet Time Servers web page. After specifying the timeserver s addresses, the following is displayed: Date, Time, Timezone & Timeserver(s) configuration ================================================== 1. Set date & time 2. Set Timezone 3. Set Timeserver(s) c. Commit changes and exit * q. Exit without commiting changes Please Choose: 13. Confirm and save your changes to the configuration by typing c in the Please Choose: field. A list of the configuration changes is displayed for review. Results Note: If only the date was modified, and not the timezones or timeservers, only the date is displayed in the contents. Example If you modify the date, add a primary and secondary timeserver, and specify America/Chicago as the time zone, when you choose c to commit the changes and exit, the file contents are displayed as follows: Please review the following information: ======================================== Date: Mon Nov 8 14:51: Primary time server: Secondary time server: Timezone: America/Chicago Do you wish to apply those settings? (yes no) Type y to apply the changes. Note: the cluster & VTFD services on all nodes must be stopped in order to continue. Do you wish to continue? (yes no) Type y to stop and restart the cluster and VTFD services on all the nodes. Stopping Cluster Services [ Done ] Stopping NTPD [ Done ] Setting Time Zone [ Done ] Setting Timeserver [ Done ] Setting Date & Time [ Done ] Starting NTPD [ Done ] Starting cluster [ Done ] Cluster Started Figure 22. Stopping cluster services 70 TS7650 and TS7650G Problem Determination and Service Guide

95 What to do next When both nodes finish rebooting, use ProtecTIER Manager to verify that fencing completed successfully and both nodes are online: To do so: 1. On the ProtecTIER Manager workstation or the TSSC, open ProtecTIER Manager. 2. Confirm that both nodes display as Online. v If both nodes are online, proceed to Performing the system validation test. v If Node 1 is offline, manually reboot Node 1: a. If necessary, connect the USB keyboard and monitor to Node 1 and log in to the server with the username root and password admin. b. At the command line, type: reboot c. If both nodes display as online, proceed to Performing the system validation test. d. If one (or both) nodes displays as offline, contact your next level of support. Performing the system validation test After system recovery, configuration, and syncing the system times, you must perform the system validation test. Procedure Run the test on both nodes, starting with Node If you are not already connected to Node 1, plug a USB keyboard and monitor into the back of the server. 2. Log in to Node 1 with username root and password admin. 3. At the command prompt, enter the following command: cd /opt/dtc/install 4. From the command prompt, enter the following command:./ptconfig -validate The following messages are displayed: Testing fence ports [Done ] Validation is about to execute a fence on 1 node in the cluster, the node will be forcefully shutdown and rebooted To Continue please type "fence test", or "q" to quit: 5. Type fence test. The following messages are displayed: Testing connectivity to the fence device [ Done ] Would you like to stop the VTFD service on both nodes? (yes no) 6. Type yes. The following message is displayed: Stopping VTFD locally [-] 7. After vtfd is stopped and the test finishes on Node 1, disconnect the keyboard and monitor from Node 1. Connect them to Node Log in to Node 2 with username root and password admin. 9. Repeat steps 3 through 6 on Node B. Chapter 3. ProtecTIER server (3958 DD1 and 3958 DD3) 71

96 Note: Alerts might be generated as the validation tests are running. The ProtecTIER Manager Alerts Log allows you to monitor the alerts as they occur. To do so, on the ProtecTIER Manager workstation, click Alerts in the lower right corner of the ProtecTIER Manager screen. The Alerts Log is displayed (Figure 23). ts Figure 23. Alerts Log When you are finished reviewing alerts, click Clear Alerts in the lower-right corner of the Alerts Log screen. 10. If the test succeeds on both nodes, TS7650G installation and setup is complete. Turn the system over to the customer and refer them to the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA for information on customizing, using, and maintaining ProtecTIER; and creating, configuring, and managing replication grids, if applicable. 11. If the test fails on either node, contact your next level of support to resolve the failure conditions. 72 TS7650 and TS7650G Problem Determination and Service Guide

97 Chapter 4. ProtecTIER server (3958 DD4, DD5, and AP1) Component labeling This section provides detailed information about the (3958 DD4, DD5, and AP1) ProtecTIER server, such as explanations of the controls and indicators. For detailed information about the 3958 DD1 and 3958 DD3 ProtecTIER servers, see Chapter 3, ProtecTIER server (3958 DD1 and 3958 DD3), on page 37. The colors of component labels indicate different types of components. A blue label on a component or near a component indicates touch points, where you can grip the component to remove it from or install it in the server, open or close a latch, and so on. An orange label on a component or near a component indicates that the component can be hot-swapped. This indicates that if the server and operating system support hot-swap capability, you can remove or install the component while the server is running. Note: Orange can also indicate the touch points on hot-swap components. Use the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide for removing or installing a specific hot-swap component for any additional procedures that you might have to perform before you remove or install the component. Note: Adapters in the TS7650 (including the hot-swap adapter ports) are not hot-swap capable due to the limitations of the Linux operating system. The system must be shut down when an adapter needs to be replaced. Power, controls, and indicators The topics in this section describe the controls and indicators for the 3958 DD4, DD5, and AP1 servers. Copyright IBM Corp. 2011,

98 Front view This topic describes the front view of the 3958 DD4, DD5, and AP1 servers. The following illustration shows the controls, LEDs, and connectors on the front of the server. The DVD drive is optional in some models. Electrostatic-discharge connector DVD-eject button DVD drive activity LED USB connectors Scalability LED Operator information panel Hard disk drive status LED Hard disk drive activity LED ts Figure 24. Server front view Electrostatic-discharge connector: Connect a electrostatic-discharge wrist strap to this connector when you work with static-sensitive devices. DVD-eject button: Press this button to release a CD or DVD from the DVD drive. DVD drive activity LED: When this LED is lit, it indicates that the DVD drive is in use. USB 1 and 2 connectors: Connect USB devices to these connectors. Scalability LED: This LED is lit and remains on during POST on the primary server when the UEFI and the IMM detect more than four microprocessors. This LED is lit and remains on after POST on the secondary server. Hard disk drive activity LED: When this LED is flashing, it indicates that the drive is in use. Hard disk drive status LED: On some server models, each hot-swap hard disk drive has a status LED. When this LED is lit, it indicates that the drive has failed. If an optional IBM ServeRAID controller is installed in the server, when this LED is flashing slowly (one flash per second), it indicates that the drive is being rebuilt. When the LED is flashing rapidly (three flashes per second), it indicates that the controller is identifying the drive. 74 TS7650 and TS7650G Problem Determination and Service Guide

99 Operator information panel This topic describes the operator information panel of the 3958 DD4, DD5, and AP1 servers. Figure 25 shows the controls and LEDs on the operator information panel. Power-control button/power-on LED Ethernet icon LED Information LED System-error LED 1 2 Power-control button cover Ethernet port activity LEDs Locator button/locator LED ts Figure 25. Operator information panel The following controls and LEDs are on the operator information panel: Power-control button and power-on LED Press this button to turn the server on and off manually or to wake the server from a reduced-power state. The states of the power-on LED are as follows: Off ac power is not present, or the power supply or the LED itself has failed. Flashing rapidly (4 times per second) The server is turned off and is not ready to be turned on. The power-control button is disabled. In a fully configured server, it could take up to 8 minutes after the server is connected to ac power, before the power-control button becomes active. Flashing slowly (once per second) The server is turned off and is ready to be turned on. You can press the power-control button to turn on the server. Lit The server is turned on. Fading on and off The server is in a reduced-power state. To wake the server, press the power-control button or use the IMM web interface (see Logging on to the Integrated Management Module web interface on page 76). Ethernet icon LED This LED lights the Ethernet icon. Information LED When this LED is lit, it indicates that a noncritical event has occurred. An LED on the light path diagnostics panel is also lit to help isolate the error. System-error LED When this LED is lit, it indicates that a system error has occurred. An LED on the light path diagnostics panel is also lit to help isolate the error. Locator button and locator LED Use this LED to visually locate the server among other servers. It is also used as the physical presence for Trusted Platform Module (TPM). Press Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 75

100 this button to turn on or turn off this LED locally. You can use IBM Systems Director to light this LED remotely. In a two-node configuration, this LED is lit on the primary server and flashes on the secondary server during POST. You can press this button or use an IPMI command to turn this LED on or off. Ethernet port activity LEDs When either of these LEDs is lit, it indicates that the server is transmitting to or receiving signals from the Ethernet LAN that is connected to the Ethernet port that corresponds to that LED. Logging on to the Integrated Management Module web interface You can use a web browser to log on to the Integrated Management Module (IMM) web interface. Procedure To log on to the IMMweb interface, perform the following steps. 1. Open a web browser and in the Address or URL field, type the IP address or host name of the IMM to which you want to connect. Note: If you are logging on to the IMM for the first time after installation, the IMM defaults to DHCP. If a DHCP host is not available, the IMM assigns a static IP address of On the Login page, type the user name and password. If you are using the IMM for the first time, obtain the user name and password from your system administrator. All login attempts are documented in the event log. Note: The IMM is set initially with a user name of USERID and password of PASSW0RD (with a zero, not the letter O). You have read/write access. You must change this default password the first time you log on. 3. On the Welcome page, type a timeout value (in minutes) in the field that is provided. You are logged off the web interface if your browser is inactive for the number of minutes that you entered for the timeout value. 4. Click Continue to start the session. The System Status page provides a quick view of the server status. 76 TS7650 and TS7650G Problem Determination and Service Guide

101 Rear view This topic describes the connectors located on the rear of the 3958 DD4, DD5, and AP1 servers. Figure 26 shows the connectors on the rear of the server. Thumbscrew is unpopulated in this position except when 2-node lock bracket is installed (2-node Scalability Kit Option) Power connector System management Ethernet AC AC DC DC!! QPI ports 1-2 Gigabit Ethernet 1-2 System serial Video USB QPI ports 3-4 ts Figure 26. Server rear view Power-cord connector Connect the power cord to this connector. System-management Ethernet connector Use this connector to connect the server to a network for systems-management information control. This connector is used only by the Integrated Management Module (IMM). QPI ports 1-4 In a single-node configuration, use these connectors to insert either a QPI wrap card or a filler panel. The QPI wrap cards enable increased performance in certain models. In a two-node configuration, insert the QPI cables in these ports to connect another server or a MAX5 memory expansion module to your server. See the documentation that came with your QPI cable kit for detailed cabling instructions. USB connectors Connect USB devices, such as a USB mouse or keyboard, to these connectors. Video connector Connect a monitor to this connector. In a two-node configuration, connect the monitor to the primary server to see standard system output. Serial connector Connect a 9-pin serial device to this connector. The serial port is shared with the IMM. The IMM can take control of the shared serial port to perform text console redirection and to redirect serial traffic, using Serial over LAN (SOL). Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 77

102 Gigabit Ethernet 1 and 2 connectors Use these connectors to connect the server to a network. Rear view LEDs of the 3958 DD4, DD5, and AP1 servers This topic describes the LEDs located on the rear of the 3958 DD4, DD5, and AP1 servers. Figure 27 shows the LEDs on the rear of the server. AC DC! AC power LED (green) DC power LED (green) Error LED (amber) Thumbscrew is unpopulated in this position except when 2-node lock bracket is installed (2-node Scalability Kit Option) AC AC DC DC! AC DC! QPI Link 1 LED QPI Link 2 LED QPI Link 3 LED QPI Link 4 LED Power on LED System error LED Figure 27. Server LEDs rear view Locator LED ts ac power LED Each hot-swap power supply has an ac power LED and a dc power LED. When the ac power LED is lit, it indicates that sufficient power is coming into the power supply through the power cord. During typical operation, both the ac and dc power LEDs are lit. For any other combination of LEDs, see Power supply LEDs on page 79. dc power LED Each hot-swap power supply has a dc power LED and an ac power LED. When the dc power LED is lit, it indicates that the power supply is 78 TS7650 and TS7650G Problem Determination and Service Guide

103 supplying sufficient dc power to the system. During typical operation, both the ac and dc power LEDs are lit. For any other combination of LEDs, see Power supply LEDs. Error LED When the power-supply error LED is lit, it indicates a problem with the power supply. It is also used, in combination with the ac and dc power LEDs, to provide more information about error conditions (see Power supply LEDs ). QPI Link LEDs 1-4 When the QuickPath Interconnect (QPI) link LEDs are lit, they indicate that the QPI links are fully established. Locator LED Use this LED to visually locate the server among other servers. You can use IBM Systems Director to light this LED remotely. In a two-node configuration, this LED is lit on the primary server and flashes on the secondary server during POST. You can press this button or use an IPMI command to turn this LED on or off. System-error LED When this LED is lit, it indicates that a system error has occurred. An LED on the light path diagnostics panel is also lit to help isolate the error. Power-on LED See Operator information panel on page 75 for the various states of the power-on LED Remote system management access The system management Ethernet port of the 3958 DD4 and 3958 DD5 server is used for system management information and control, in the same way that the remote supervisor adapter controls are used on the 3958 DD3 server. See Figure 26 on page 77 for the location of the system management Ethernet port. Power supply LEDs The ac power LED, the dc power LED and the power supply Error LED on the rear of the 3958 DD4, DD5, and AP1 server power supply, along with the power LED on the operator information panel, are used to indicate various error conditions. When a combination of these LEDs is lit, the combination indicates a specific error. When the amber LED on the rear panel of a server power supply is lit, it indicates an error condition within the power supply. Note: If a power supply fails, it shows up in System Health Monitoring as a Power Supply 1 or 2 good fault. Replacing the power supply restores the power, but does not clear the problem for System Health Monitoring. To clear the error, refer to Clearing system errors after power supply replacement on page 83. The specific error condition is indicated by combinations of four LEDs. One of these LEDs is the power indicator on the operator information panel. The other three are the indicators on the rear panel of the power supply ( Clearing system errors after power supply replacement on page 83). The following minimum configuration is required for the dc LED on the power supply to be lit: Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 79

104 v v v v I/O board Power supply Power cord Microprocessor board The following minimum configuration is required for the server to start: v I/O board v Power supply v Power cord v Microprocessor board v One microprocessor v Two DIMMs on one memory card v Operator information panel Figure 28 shows the locations of the power supply LEDs. AC DC! AC power LED (green) DC power LED (green) Error LED (amber) ts Figure 28. Power supply rear view Table 8. Power supply indicators Number Indicator Color 1 ac power Green 2 dc power Green 3 Power supply error Amber To interpret the combinations displayed by these four LEDs, refer to Table 9 on page TS7650 and TS7650G Problem Determination and Service Guide

105 Table 9. Power supply LED combinations Operator information panel power LED Power supply LEDs ac dc Error Description Off Off Off Off No alternating current (ac) power to the server, or a problem with the ac power source. Off Lit Off Off Direct current (dc) source power problem or system error Action 1. Check the ac power to the server. 2. Make sure that the power cord is connected to a functioning power source. 3. Make sure that the power cord is fully seated in the power supply inlet. 4. Check the Western Telematic Inc. (WTI) power ports. v For Node A, 1 and 5 should be lit. v For Node B, 2 and 6 should be lit. If the power ports are not lit, the node has been fenced. See Appendix H, Checking the Western Telematic Inc. power ports, on page 291 to restore power. Note: On newer versions of the WTI switch, the LEDs are not visible unless you remove the side panel. If this is not possible, use Telnet to connect a working node to the WTI switch (IP address ) and view the port status. 1. Reseat one power supply at a time (see Removing a hot-swap power supply and Replacing the hot-swap power supply in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 2. View the system-error log (see Event logs in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 3. (Trained service technician only) If the system-board error LED is lit, replace the system (microprocessor) board. Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 81

106 Table 9. Power supply LED combinations (continued) Operator information panel power LED Power supply LEDs ac dc Error Description Off Lit Lit Off Standby power problem or server is turned off Flashing Lit Lit Off Normal standby state Lit or off Lit Lit or off Lit Internal power supply fault (for example, thermal fault, or over-voltage or under-voltage condition) Lit Lit Lit Off Power is available Action 1. Press the power-control button on the operator information panel. 2. View the system-error log (see Event logs in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 3. Isolate by removing one power supply at a time (see Removing a hot-swap power supply and Replacing the hot-swap power supply in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 4. (Trained service technician only) If the system board error LED is lit, replace the (microprocessor) system board. No action. 1. View the system-event log (see Event logs in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 2. Replace the power supply (see Removing a hot-swap power supply and Replacing the hot-swap power supply in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). No action. Note: See the Removing and replacing server components chapter in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide to determine which components a field engineer should replace. Note: For detailed analysis of other power problems, see Power problems in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. 82 TS7650 and TS7650G Problem Determination and Service Guide

107 Clearing system errors after power supply replacement If a power supply fails, it shows up in System Health Monitoring as a Power Supply 1 or 2 good fault. Replacing the power supply restores the power, but does not clear the problem for System Health Monitoring. You must clear the error manually. Procedure To clear the error, perform the following steps. 1. From the ProtecTIER Service menu (see ProtecTIER Service menu on page 16), select System Health Monitoring > Reset RSA/IMM. IMM restarts. This takes about a minute. The command prompt returns when IMM restart is completed. 2. Select Back > Problem management > Process Problems > List open problems. 3. On a sheet of paper, record the 13-digit problem number associated with the power supply problem. (Example: problemid=< >) 4. Select Cancel Problem. Enter the problem ID to cancel the problem. Identifying problems using status LEDs Diagnostics You can identify server problems using the status light-emitting diodes (LEDs) on the front of the server. The server is designed so that any LEDs that are lit remain lit when the server shuts down, as long as the server is connected to ac power, and is able to remain in standby mode (that is, dc power is provided to the system board). See Power supply LEDs on page 79 for troubleshooting with the power supply LEDs and power-on LED. If the system-error LED on the front of the server is on, one or more LEDs inside the server or on the power supply are on. The server has LEDs to help you identify problems with some server components. These LEDs are part of the light path diagnostic feature built into the server. By following the light path, you can quickly identify the type of system error that occurred. See Light path diagnostics on page 91. The topics in this section provide basic troubleshooting information to help you resolve some common problems that might occur with the server. General checkout procedure Complete this task in order to checkout the subcomponents of the ProtecTIER server hardware. About this task Follow the checkout procedure for diagnosing hardware problems. Review the following information before performing the checkout procedure. The following diagnostic tools are available to you for diagnosis and solving of hardware-related problems: Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 83

108 Light path diagnostics Use light path diagnostics to diagnose system errors quickly. See Light path diagnostics on page 91 for more information. Dynamic System Analysis (DSA) DSA collects and analyzes system information to aid in diagnosing server problems. See Using Dynamic System Analysis on page 16 for more information. v The server diagnostic programs (DSA Preboot) are stored on the system board as part of the firmware load. These programs provide the primary methods of testing the major components of the server. If you are not sure whether a problem is caused by the hardware or by the software, you can run the diagnostic programs. See Diagnostic programs and error messages on page 87 to confirm whether the hardware is working correctly. v When you run the diagnostic programs, a single problem might cause several error messages. If you receive several error messages, correct the cause of the first error message. The other error messages might not occur the next time that you run the diagnostic programs. Important: v If the server is hung and a power-on self-test (POST) error code is displayed, see POST error codes in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide for suggested action to solve the problem. v If the server is hung, perform the following steps. 1. Do one of the following: Restart the server and press F2 to start DSA Preboot and view the event logs. Restart the server. Press F1 to start the Setup utility. View the POST event log or system-event log for the error indication. (Select POST Event Viewer to view the POST event log. Select System Event Log to view the system-event log.) 2. If you still have insufficient information to solve the problem, go to the Undetermined problems section in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide v For information about power-supply problems, see ProtecTIER 3958 DD4, DD5, and AP1 server power checkout on page 102. v For intermittent problems, check the error logs; see Diagnostic programs and error messages on page 87. Complete the following steps to perform the checkout procedure to identify system problems: 1. Check the power supply LEDs (see Power supply LEDs on page 79). 2. Turn off the server and all external devices. a. Undo any recent system changes,such as new settings or newly installed devices. b. Remove all hardware that is not listed on the ServerProven website. 3. Check all cables and power cords. 4. Turn on all external devices. 5. Restart the server. If the server does not start, see Troubleshooting tables in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. 84 TS7650 and TS7650G Problem Determination and Service Guide

109 6. Record any power-on self test (POST) error code and message that is displayed on the screen. If an error is displayed, look it up in the POST error codes section of the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. 7. Check the system-error light-emitting diode (LED) on the operator information panel; if it is on, see Light path diagnostic light-emitting diodes on page Check the IMM event log. If any errors or warnings or any relevant informational messages are recorded, see the appropriate section of the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Access the IMM via the TS3000 System Console (TSSC) browser using the IMM IP address xxx. For an alternate procedure to log in to the IMM, see Appendix A, Logging into the 3958 DD1/3958 DD3 server through the RSA connection, on page Check for any open problems posted by the Problem Manager. See Problem Manager on page 17 and use the rscerpmgetopenproblems command to display any open problems. 10. If the system stops with no readable messages in the operating system, try to find the failure symptom in the Troubleshooting tables section of the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. If necessary, see the Solving undetermined problems section. If there is a readable display on the screen, go to the next step. 11. Start the diagnostic programs. Go directly to the IBM Dynamic System Analysis (DSA) section in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide, or refer to Starting the diagnostic programs on page If DSA reports an error, then follow the instructions in the DSA messages section of the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide If the diagnostic programs were completed successfully and you still suspect a problem, see the Solving undetermined problems section in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Diagnostic tools overview This topic describes the tools that are used to diagnose the ProtecTIER server. The following tools are available to help you diagnose and solve hardware-related problems: Power-on self test (POST) error codes, error messages, and error logs The POST generates error codes and messages to indicate detection of a problem. Problem isolation tables Use these tables to help you diagnose various symptoms. Light path diagnostics Use the light path diagnostics to diagnose system errors quickly. Diagnostic programs and error messages The diagnostic programs are stored in memory on the system board. These programs are the primary method of testing the major components of the server. Event logs, error logs Error codes and messages are displayed in the following logs: Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 85

110 v v v v POST event log System-event log IMM event log DSA log Power-on self-test error log Problems that occur during the power-on self-test (POST) are recorded in the error log. If the POST finishes without detecting any problems, the first screen of your operating system or application program is displayed. If the POST detects a problem, an error message is displayed. See the POST error codes section in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide for detailed failure information. Notes: v Before the POST continues, if you have a power-on password or an administrator password set, you must type the password and press Enter when prompted. (If you do not have the password, ask the customer for it.) v A single problem might cause several error messages. When this occurs, work to correct the cause of the first error message. After you correct the cause of the first error message, the other error messages usually do not occur the next time you run the test. The POST error log contains the three most recent error codes and messages that the system generated during the POST. The System Event log contains POST and system management interrupt (SMI) events, and all events generated by the baseboard management controller (BMC) which is embedded in IMM. Also logged are system status messages from the service processor. You can view the contents of the POST error log and System Event log from the Setup Utility program or from the diagnostic programs. Viewing error logs from the Setup Utility program This topic describes how to view error logs. Procedure To view the POST event log or System event log, perform the following steps. 1. Turn on the server. 2. When the prompt <F1> Setup is displayed, press F1. 3. If a password and administrator password have been set, type the administrator password to view the event logs. 4. Select the log you want to view. v To view the POST event log, select POST Event Viewer. v To view the system event log, select System Event Log. Viewing error logs from diagnostic programs Complete this task to view the diagnostics error logs. 86 TS7650 and TS7650G Problem Determination and Service Guide

111 Procedure To view error logs from diagnostic programs, complete the following steps. 1. Start the diagnostic programs. 2. Select Hardware Info from the top of the diagnostic programs screen. A list of logs is displayed. 3. Select one of the following logs: v System event log v IMM event log v Operating-system event log v Merged DSA log 4. Follow the instructions on the screen. See Starting the diagnostic programs on page 88 for more information. Diagnostic programs and error messages This topic describes the diagnostic programs and their error messages. The server diagnostic programs are stored in upgradeable read only memory (ROM) on the system board. These programs are the primary method of testing the major components of your server. Diagnostic error messages indicate that a problem exists; they are not intended to be used to identify a failing part. Troubleshooting and servicing of complex problems that are indicated by error messages should be performed by trained service personnel. Sometimes the first error to occur causes additional errors. In this case, the server displays more than one error message. Always follow the suggested action instructions for the first error message that appears. Diagnostic error messages that might be displayed are listed in the DSA messages section in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Notes: v Depending on the server configuration, some of the error codes might not appear when you run the diagnostic programs. v If diagnostic error codes appear that are not listed in the tables, make sure that the server has the latest levels of firmware. To display the currently flashed firmware versions, do the following: For ProtecTIER software V3.1 or later, from the ProtecTIER Service menu, select Version Information > Display Firmware Levels (see ProtecTIER Service menu on page 16). For ProtecTIER software V2.4 or V2.5, from the ProtecTIER Service menu, select Display Firmware Versions. For ProtecTIER software V2.3 or earlier, from a command line type versions. If the command is not found, type the following commands to mount the IBM System Storage RAS/BIOS and Firmware Update DVD and display the firmware versions: mount /dev/hda /mnt/cdrom cd /mnt/cdrom./versions Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 87

112 v Note: Displaying the list of firmware versions can take several minutes. You can find all of the firmware modules flashed to this system listed in directory /opt/dtc/firmwares. List the files in this directory to find the latest version of the DSA utility (which must be used to create the DSA log). Diagnostic text messages Diagnostic text messages are displayed while the tests are running. A diagnostic text message contains one of the following results: Passed This test was completed without any errors. Failed This test discovered an error. User Aborted You stopped the test before it was completed. Not Applicable You attempted to test a device that is not present in the server. Aborted The test could not proceed because of the server configuration. Warning The test could not be run. There was no failure of the hardware that was being tested, but there might be a hardware failure elsewhere, or another problem prevented the test from running. (For example, there might be a configuration problem, or the hardware might be missing or not recognized.) The result is followed by an error code or other additional information about the error. Starting the diagnostic programs Complete this task to start the diagnostic programs. Before you begin To perform this task, you need the following items: v Monitor v Universal serial bus (USB) keyboard v USB mouse v Associated cables Procedure To start the diagnostic programs, perform the following steps. 1. Connect the monitor to the video connector on the rear of the server and connect the USB keyboard and mouse to the USB ports on the back of the server (Figure 29 on page 89). 88 TS7650 and TS7650G Problem Determination and Service Guide

113 Thumbscrew is unpopulated in this position except when 2-node lock bracket is installed (2-node Scalability Kit Option) Power connector System management Ethernet AC AC DC DC!! QPI ports 1-2 Gigabit Ethernet 1-2 System serial Video USB QPI ports 3-4 ts Figure 29. Server rear view Note: QPI ports 1 and 2 are wrapped by QPI Wrap card 1. QPI ports 3 and 4 are wrapped by QPI Wrap card 2. QPI ports are not used as Expansion ports. 2. Turn on the server and watch the screen. ts Figure 30. eserver logo screen for basic input/output system (BIOS) setup 3. When the message F2 for Diagnostics appears, press F2. 4. Type the appropriate password and then press Enter. 5. Select either Extended or Basic from the top of the screen. 6. When the diagnostic programs screen is displayed, select the test that you want to run from the list that appears, and then follow the instructions on the screen. Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 89

114 Notes: v You can press F1 while running the diagnostic programs to obtain help information. You also can press F1 from within a help screen to obtain online documentation from which you can select different categories. To exit from the help information and return to where you left off, press Esc. v If the server stops during testing and you cannot continue, restart the server and try running the diagnostic programs again. If the problem remains, replace the component that was being tested when the server stopped. v The keyboard and mouse (pointing device) tests assume that a keyboard and mouse are attached to the server. v If you run the diagnostic programs with either no mouse or a USB mouse attached to your server, you cannot navigate between test categories using the Next Cat and Prev Cat buttons. All other functions provided by mouse-selectable buttons are also available using the function keys. v You can test the USB keyboard by using the regular keyboard test. The regular mouse test can test a USB mouse. v Note: You can run the USB interface test only if no USB devices are attached. You can view server configuration information (such as system configuration, memory contents, interrupt request (IRQ) use, direct memory access (DMA) use, device drivers, and so on) by selecting Hardware Info from the top of the screen. What to do next If the diagnostic programs do not detect any hardware errors but the problem persists during normal server operations, a software error might be the cause. If you suspect a software problem, refer to the information that comes with the software package. Otherwise, see Undetermined problems in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Viewing the test log Complete this task in order to view the test log. Procedure Complete the following steps in order to view the test log after tests are completed. 1. Select Utility from the top of the screen and then select View Test Log. 2. View the test log or save the log to a file on a diskette or on the hard disk so that you can view it later by performing the following steps: a. Select Save Log on the diagnostic programs screen. b. Specify a location and name for the saved log file. Note: The test log data is maintained only while the diagnostic programs are active. When you exit from the diagnostic programs, the test log is cleared (saved test logs are not affected). What to do next For a detailed listing of diagnostic error messages, see Running DSA Preboot and DSA messages in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. 90 TS7650 and TS7650G Problem Determination and Service Guide

115 Viewing the system event log This topic provides information for viewing the system event log. To view the system error log, perform one of the following actions. v Refer to Viewing error logs from the Setup Utility program on page 86. v View the IMM event log online through the IMM Web interface. For a list of error messages, refer to the System-event log section of the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Light path diagnostics A system of light emitting diodes (LEDs) is used to convey the results from light path diagnostics. Light path diagnostics is a system of LEDs on various external and internal components of the server. When an error occurs, LEDs are lit throughout the server. By viewing the LEDs in a particular order, you can often identify the source of the error. The server is designed so that LEDs remain lit when the server is connected to an ac power source but is not turned on, provided the power supply is operating correctly. This feature helps you to isolate the problem when the operating system is shut down. Any memory card LED can be lit while the memory card is removed from the server so that you can isolate a problem. After ac power is removed from the server, power remains available to these LEDs for up to 24 hours. To view the memory card LEDs, press and hold the light path diagnostics button on the memory card to light the error LEDs. The LEDs that were lit while the server was turned on are lit again while the button is pressed. Many errors are first indicated by a lit information LED or system-error LED on the operator information panel on the front of the server. If one or both of these LEDs are lit, one or more LEDs elsewhere in the server might also be lit and can direct you to the source of the error. If an error occurs, view the LEDs in the following order: 1. Check the operator information panel on the front of the server. v If the information LED is lit, it indicates that information about a suboptimal condition in the server is available from the baseboard management controller in the system-event log and the IMM event log. Proceed to step 2 on page 92. v If the system-error LED is lit, a system error occurred. Go to step 2 on page 92. Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 91

116 Power-control button/power-on LED Ethernet icon LED Information LED System-error LED 1 2 Power-control button cover Ethernet port activity LEDs Locator button/locator LED ts Figure 31. Operator information panel 2. To view the light path diagnostics panel, press the release latch on the front of the operator information panel to the left, then slide it forward. This reveals the light path diagnostics panel. Lit LEDs on this panel indicate the type of error that occurred. OVER SPEC LOG LINK PS PCI FAN TEMP MEM NMI CNFG CPU VRM DASD RAID REMIND SP BRD NMI button (trained service technician only) Light Path Diagnostics ts Figure 32. Server light path diagnostics panel NMI button The NMI button is reserved for trained IBM service technicians, who use it to debug the operating system. Pressing the button sends a non-maskable interrupt to the microprocessor, causing the server to reset. Reset button The reset button is in the lower-right corner of the light path diagnostics panel. Press it to reset the server and run the power-on self-test (POST). You might have to use a pen or the end of a straightened paper clip to press the button. Remind button You can use the remind button on the light path diagnostics panel to put the system-error LED on the operator information panel into Remind mode. When you press the remind button, you acknowledge the error but indicate that you are not taking immediate action. The system-error LED flashes while it is in Remind mode and stays in Remind mode until one of the following conditions occurs: v All known errors or suboptimal conditions are corrected. v The server is powered back on. v A new error or suboptimal condition occurs, causing the system-error LED to be lit again. 92 TS7650 and TS7650G Problem Determination and Service Guide

117 You can also use the remind button to turn off the LOG LED on the light path diagnostics panel and the information LED. In multi-node configurations, you can press this button during startup to start the server as a stand-alone server. Checkpoint display The checkpoint display, indicated by 88 on the light path diagnostic panel, provides checkpoint codes. Checkpoint codes are not error codes; they identify firmware checks when the server stops. There are field-programmable gate array (FPGA) hardware checkpoint codes and basic input/output system checkpoint codes. For more information, see the IBM System x3850 M2 (7141, 7144) Checkpoint listing web page. For a list of checkpoint codes, see Appendix C, Checkpoint codes, on page 269. System service label The system service label on the top of the server provides an overview of internal components that correspond to the LEDs on the light path diagnostics panel. This information and the information in Troubleshooting with light path diagnostics can often provide enough information to correct the error. 3. Remove the server cover and look inside the server for lit LEDs. Certain components inside the server have LEDs that are lit to indicate the location of a problem. See the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Troubleshooting with light path diagnostics Complete the task in this topic to troubleshoot with light path diagnostics. About this task Light-emitting diodes (LEDs) in two locations on the server are available to help you diagnose problems that might occur during installation. Use them in the following order: Procedure 1. Light path diagnostics panel Look at this panel first. If a system error has occurred, the system-error LED on the front of the light path diagnostics drawer is lit. Slide the latch to the left on the front of the light path diagnostics drawer to access the light path diagnostics panel. Note any LEDs that are lit, and then close the drawer. 2. LEDs on the system board To identify the component that is causing the error, note the lit LED on or beside the component. What to do next See the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide for additional detailed information. Light path diagnostic light-emitting diodes This topic describes the light path diagnostic light-emitting diodes (LEDs) and their associated actions. Table 10 on page 94 describes the LEDs on the light path diagnostics panel and on the boards inside the server and suggested actions to correct the detected problems. Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 93

118 v v Follow the suggested actions in the order in which they are listed in the Action column until the problem is solved. If an Action step is preceded by Trained service technician only, that step must be performed only by a trained service technician. Table 10. LEDs on the light path diagnostics panel Lit light path diagnostics LED with the system-error LED or information LED also lit All light-emitting diodes (LEDs) are off (only the power LED is lit or flashing). All LEDs are off (the power LED is lit or flashing and the system-error LED is lit). OVERSPEC LOG LINK Description A machine check has occurred. The server is identifying the machine check, the server was interrupted while identifying the machine check, or the server was unable to identify the machine check. There is insufficient power to power the system. The LOG LED might also be lit. Information is present in the system-event log. There is a fault in a QPI port or the QPI scalability cables. Note: 1. This LED remains lit until the problem is resolved and the server is turned off and restarted. 2. If a fault occurs, the SMP Expansion Port link LED on the failed port is off. No action necessary. Action 1. Wait several minutes for the server to identify the machine check and restart. 2. (Trained service technician only) Extract the machine check data, which is used to identify the machine check. 1. Add a power supply if only one power supply is installed. 2. Use 220 V ac instead of 110 V ac. 3. Reseat the power supply (see Removing a hot-swap power supply and Replacing the hot-swap power supply in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 4. Remove optional devices. 5. Replace the power supply. 1. Check the log for possible errors. 2. Save the log if necessary and clear. 1. Check the QPI port link LEDs to find the failing port or cable. (The rear LED will be off on the failing port or cable.) 2. Reseat the QPI wrap cards or the QPI scalability cables (if installed). 3. Replace the QPI wrap cards or the QPI scalability cables (if installed). 4. (Trained service technician only) Replace the I/O board (see Removing the I/O-board shuttle and Replacing the I/O-board shuttle in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 94 TS7650 and TS7650G Problem Determination and Service Guide

119 Table 10. LEDs on the light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit PS Description A power supply has failed or has been removed. Note: In a redundant power configuration, the dc power LED on one power supply might be off. Action 1. Reinstall the removed power supply (see Replacing the hot-swap power supply in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 2. Check the individual power supply LEDs to find the failing power supply (see Rear view LEDs of the 3958 DD4, DD5, and AP1 servers on page 78). 3. Reseat the failing power supply (see Removing a hot-swap power supply and Replacing the hot-swap power supply in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 4. Check the Western Telematic Inc. (WTI) power ports. v For Node A, 1 and 5 should be lit. v For Node B, 2 and 6 should be lit. If the power ports are not lit, the node has been fenced. See Appendix H, Checking the Western Telematic Inc. power ports, on page 291 to restore power. Note: On newer versions of the WTI switch, the LEDs are not visible unless you remove the side panel. If this is not possible, use Telnet to connect a working node to the WTI switch (IP address ) and view the port status. 5. Make sure that the power cord is fully seated in the power supply inlet and the ac power source. 6. Replace the power supply. Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 95

120 Table 10. LEDs on the light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit PCI FAN Description A peripheral component interface (PCI) adapter has failed. Note: The error LED next to the failing adapter on the I/O board shuttle is also lit. A fan has failed or has been removed. Note: A failing fan can also cause the TEMP LED to be lit. Action 1. Examine the system-event log. 2. Reseat the following components one at a time, in the order shown, restarting the server each time: a. Failing adapter (see Removing an adapter and Replacing an adapter in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). b. I/O board assembly (see Replacing the front hot-swap fans and Replacing the I/O-board shuttle in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 1. Reinstall the removed fan (see Replacing the front hot-swap fans in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 2. If an individual fan LED is lit, replace the fan. Note: A failing fan might not cause the fan LED to be lit. 3. (Trained service technician only) Reseat the microprocessor board (see Removing the microprocessor-board assembly and Replacing the microprocessor-board assembly in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 4. (Trained service technician only) Replace the microprocessor board. 96 TS7650 and TS7650G Problem Determination and Service Guide

121 Table 10. LEDs on the light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit TEMP Description A system temperature or component has exceeded specifications. Note: A fan LED might also be lit. Action 1. Examine the system-event log for the source of the fault (see Event logs in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 2. Make sure that the airflow of the server is not blocked. 3. If a fan LED is lit, reseat the fan (see Removing the front hot-swap fans and Replacing the front hot-swap fans in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 4. Replace the fan for which the LED is lit. 5. Make sure that the room is neither too hot nor too cold (see Environment in the Server features and specifications section of IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 97

122 Table 10. LEDs on the light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit Description Action MEM Memory failure. 1. Remove the memory card that has a lit error LED (see Removing a memory card in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 2. Press the light path diagnostics button on the memory card to identify the failed card or DIMM (see Remind button in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 3. Reseat the DIMM with the lit LED (see Removing a DIMM and Replacing a DIMM in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 4. Swap the failed DIMM with a known good DIMM, or move the failed DIMM to another connector to see whether the error follows the DIMM or stays with the connector. Restart the server. 5. Replace the following components one at a time, in the order shown, restarting the server each time: v DIMM v Memory card v (Trained service technician only) Microprocessor board 6. If the DIMM was disabled, run the Setup utility and enable the DIMM. See Configuring the server in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. NMI A hardware error has been reported to the operating system. Note: The peripheral component interface (PCI) or memory (MEM) LED might also be lit. 1. Examine the system-event log (see Event logs in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 2. If the peripheral component interface (PCI) LED is lit, follow the instructions for that LED. 3. If the memory (MEM) LED is lit, follow the instructions for that LED. 4. Restart the server. 98 TS7650 and TS7650G Problem Determination and Service Guide

123 Table 10. LEDs on the light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit CNFG Central processing unit (CPU) VRM Description A configuration error has occurred. A microprocessor has failed, is missing, or has been incorrectly installed. Reserved Action 1. Find the failing or missing component by checking the other light path diagnostic LEDs on the operator information panel. Make sure the microprocessors match each other (speed, cache, etc.) 2. Make sure that the fans, power supplies, microprocessors, and memory cards are installed in the correct sequence. 1. If the CNFG light is lit, then make sure that the microprocessors are installed in the correct sequence (see Microprocessor in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) and make sure the microprocessors match each other (speed, cache, etc.) 2. Check the system-event log to determine the reason for the lit LED (see Event logs in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 3. Find the failing, missing, or mismatched microprocessor by checking the LEDs on the microprocessor board. 4. (Trained service technician only) Reseat the following components: a. Failing microprocessor (see Microprocessor in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) b. Microprocessor board (see Removing the microprocessor-board assembly and Replacing the microprocessor-board assembly in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) 5. (Trained service technician only) Replace the following components one at a time, in the order shown, restarting the server each time: a. Failing microprocessor b. Microprocessor board Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 99

124 Table 10. LEDs on the light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit DASD Description A hard disk drive has failed or has been removed. Note: The error LED on the failing hard disk drive is also lit. Action 1. Reinstall the removed drive. 2. Reseat the following components one at a time, in the order shown, restarting the server each time: a. Failing hard disk drive (see Removing a hot-swap hard disk drive and Replacing a hot-swap hard disk drive in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) b. SAS hard disk drive backplane (see Removing the SAS hard disk drive backplane assembly and Replacing the SAS hard disk drive backplane assembly in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) c. SAS signal cable d. I/O board assembly (see Removing the I/O-board shuttle and Replacing the I/O-board shuttle in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) 100 TS7650 and TS7650G Problem Determination and Service Guide

125 Table 10. LEDs on the light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit RAID Description The RAID controller has indicated a fault. Action 1. Check the system-event log for information (see Event logs in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 2. Reseat the following components: a. RAID controller, if possible b. Hard disk drives c. I/O board shuttle assembly 3. Reseat the following components one at a time, in the order shown, restarting the server each time: a. RAID controller (see Removing the RAID adapter carrier and the RAID adapter assembly and Replacing the RAID adapter carrier and the RAID adapter assembly in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) b. RAID power and SAS signal cables c. Hard disk drives (see Removing a hot-swap hard disk drive and Replacing a hot-swap hard disk drive in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) d. I/O board assembly (see Removing the I/O-board shuttle and Replacing the I/O-board shuttle in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide) Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 101

126 Table 10. LEDs on the light path diagnostics panel (continued) Lit light path diagnostics LED with the system-error LED or information LED also lit BOARD Description The I/O-board shuttle or microprocessor board has failed. Action 1. Find the failing board by checking the LEDs on the I/O-board shuttle and microprocessor board. 2. If the I/O board LED is lit, reseat the I/O board (see Removing the I/O-board shuttle and Replacing the I/O-board shuttle in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). If the I/O board LED remains lit, replace the I/O board. 3. If the I/O board LED is not lit, remove memory card 7 to see if the microprocessor-board LED is lit (see Removing a memory card in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). 4. (Trained service technician only) If the microprocessor-board LED is lit, reseat the microprocessor board (see Removing the microprocessor-board assembly and Replacing the microprocessor-board assembly in IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide). If the microprocessor-board LED remains lit, replace the microprocessor board. Server power features A master power switch is located in the front on the upper right side of the cabinet. Attention: Use the master power switch to turn cabinet power on and off completely. v When the server is turned off, the operating system does not run, and all core logic except for the service processor is shut down. v However, if the switched-off server is connected to an alternating current (ac) power source, it can still respond to requests from the service processor (also called the baseboard management controller). The power-on light-emitting diode (LED) flashes to indicate that the server is connected to ac power but is not turned on. ProtecTIER 3958 DD4, DD5, and AP1 server power checkout Complete this task to troubleshoot power problems. 102 TS7650 and TS7650G Problem Determination and Service Guide

127 About this task Power problems can be difficult to solve. For example, a short circuit can exist anywhere on any of the power distribution buses. A short circuit usually causes the power subsystem to shut down because of an overcurrent condition. Note: Use this procedure only along with your next level of support. Procedure To troubleshoot power problems, perform the following steps. 1. Turn off the server, and disconnect all ac power cords. 2. Check for loose cables in the power subsystem. Also check for short circuit causes; for example, check to see whether a loose screw might have caused a short circuit on a circuit board. 3. Remove adapters and disconnect the cables and power connectors to all internal and external devices until the server is at the minimum configuration required to start. See the Solving undetermined problems section in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide for the minimum configuration. 4. Reconnect all ac power cords and turn on the server. If the server starts up successfully, replace adapters and devices one at a time until the problem is isolated. If the server does not start up from the minimal configuration, replace field replaceable units (FRUs) of minimal configuration one at a time until the problem is isolated. For specific power supply problems, see Power supply LEDs on page 79. Ethernet controller troubleshooting This topic provides information to help with troubleshooting the Ethernet controller. Use the following troubleshooting guidelines to find solutions to 10/100/1000 Mbps Ethernet controller problems that have definite symptoms. v Make sure that the correct device drivers are installed in the server and are at the latest level. v Make sure that the Ethernet cable is installed correctly. The cable must be securely attached at all connections. If the cable is attached but the problem remains, try a different cable. If you set the Ethernet controller to operate at 100 Mbps or higher, you must use Category 5 cabling. For 1000 Mbps, use Category 5e or above. If you directly connect two servers (without a hub), or if you are not using a hub with X ports, use a crossover cable. To determine whether a hub has an X port, check the port label. If the label contains an X, the hub has an X port. v Determine whether the hub supports auto-negotiation. If it does not, try configuring the integrated Ethernet controller manually to match the speed and duplex mode of the hub. v Check the Ethernet controller light-emitting diodes (LEDs) on the rear panel of the server. These LEDs indicate whether there is a problem with the connector, cable, or hub. See Figure 33 on page 104. Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 103

128 ACT/LNK A 10=OFF 100=GRN 1000=ORG Intel=PRO ACT/LNK A ts Figure 33. Ethernet controller LEDs Table 11. Quad-port Ethernet LED indicators Number Label Indication Meaning 1, 3, 5, 7 2, 4, 6, 8 ACT/LNK Green on (not flashing) The adapter is connected to a valid link partner. Green on flashing Data activity Off No link 10=OFF Off 10 Mbps 100=GRN Green 100 Mbps 1000=ORG Orange 1000 Mbps Orange flashing Identity. Use the "Identify Adapter" button in Intel PROSet to control blinking. See Intel PROSet Help for more information. 104 TS7650 and TS7650G Problem Determination and Service Guide

129 v v v The Ethernet link status LED is lit when the Ethernet controller receives a link pulse from the hub. If the LED is off, there might be a defective connector or cable or a problem with the hub. The Ethernet transmit/receive activity LED is lit when the Ethernet controller sends or receives data over the Ethernet network. If the Ethernet transmit/receive activity light is off, make sure that the hub and network are operating and that the correct device drivers are installed. Check the local area network (LAN) activity LED on the rear of the server. The LAN activity LED is lit when data is active on the Ethernet network. If the LAN activity LED is off, make sure that the hub and network are operating and that the correct device drivers are installed. Check for operating-system-specific causes of the problem. Make sure that the device drivers on the client and server are using the same protocol. If the Ethernet controller still cannot connect to the network but the hardware appears to be working, the network administrator must investigate other possible causes of the error. If the Ethernet adapter is defective, see to Replacing a Dual-Port or Quad-Port Ethernet card and recovering Ethernet settings on page Gb Ethernet adapter (Feature Code 3457) This topic describes the 10 Gb Ethernet adapter. The 3958 DD4, DD5, and AP1 servers feature two 10 Gb dual-port Ethernet adapters that you can order in place of the 1 Gb quad-port Ethernet adapters in PCI slots 3 and 4 in OpenStorage and FSI configurations. The 10 Gb Ethernet adapter ports connect to Fibre Channel cables. Server power-on password override This topic provides information about how to override the server power-on password. If the power-on password for the server is lost or forgotten, regain access to the server by performing any of the following: v If an administrator password is set, type the administrator password at the password prompt. Start the Configuration/Setup Utility program and reset the power-on password. v Remove the battery from the server and then reinstall it. For instructions for removing the battery, see the Removing the battery section in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. v Change the position of the power-on password override jumper (J29 on the I/O board) to bypass the power-on password check. Attention: Before changing any switch settings or moving any jumpers, turn off the server, then disconnect all power cords and external cables. Do not change settings or move jumpers on any system-board switch or jumper blocks that are not shown in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. I/O board jumpers This topic describes the jumpers on the I/O board. Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 105

130 Figure 34 shows the jumpers on the I/O board. Wake on LAN bypass (J31) Password override (J29) Boot recovery (J22) ts Figure 34. Jumpers on I/O board Table 12 describes the function of each jumper block. Table 12. I/O board jumper blocks Jumper name Description Wake on LAN bypass (J31) The default position is pins 1 and 2. Move the jumper to pins 2 and 3 to prevent a Wake on LAN packet from waking the system when the system is in the powered-off state. Password override (J29) Boot recovery (J22) The default position is pins 1 and 2. Change the position of this jumper to pins 2 and 3 to bypass the power-on password check. Changing the position of this jumper does not affect the administrator password check if an administrator password is set. If you forget the administrator password, you must replace the microprocessor board. For more information about passwords, see Passwords. The default position is pins 1 and 2 (to use the primary page during startup). Move the jumper to pins 2 and 3 to use the secondary page during startup. Passwords This topic explains the use of passwords. 106 TS7650 and TS7650G Problem Determination and Service Guide

131 System recovery From the User Security menu choice, you can set, change, and delete a power-on password and an administrator password. The User Security choice is on the full Setup utility menu only. If you set only a power-on password, you must type the power-on password to complete the system startup and to have access to the full Setup utility menu. An administrator password is intended to be used by a system administrator; it limits access to the full Setup utility menu. If you set only an administrator password, you do not have to type a password to complete the system startup, but you must type the administrator password to access the Setup utility menu. If you set a power-on password for a user and an administrator password for a system administrator, you can type either password to complete the system startup. A system administrator who types the administrator password has access to the full Setup utility menu; the system administrator can give the user authority to set, change, and delete the power-on password. A user who types the power-on password has access to only the limited Setup utility menu; the user can set, change, and delete the power-on password, if the system administrator has given the user that authority. Complete this task to recover the server to an operational state. About this task Testing a clustered system The general procedure to recover the server to an operational state is as follows. (For specific procedures for the TS7650 Appliance, refer to IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA ) Procedure 1. Reinstall Red Hat Linux. 2. Configure the system settings to match those recorded at initial installation. After a recovery and after using the ptconfig command to successfully configure both nodes in the cluster, run the system validation tests. About this task The validation tests perform checks on the Red Hat Linux cluster, including fencing the other node, and waits for the system to reboot. This test cannot be performed on a stand-alone configuration. Important: Extensive discrepancies in the system time set on each server in the cluster can result in unnecessary fencing between the nodes during system validation testing. To avoid this, it is recommended that you synchronize the servers' system times before you start the test. Refer to Syncing the system times on page 108 for instructions. Note: In a replication environment, the procedures in this section should be performed on the clustered servers at both the source and the destination locations. Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 107

132 Syncing the system times You can set the date, time, and time zone manually or sync with a timeserver. Procedure 1. If you are not already connected to a node, plug a USB keyboard and monitor into the back of the server. 2. Log in to the server with the username root and password admin. 3. Access the /install directory by typing the command: cd /opt/dtc/install 4. Perform either of the following actions to begin the set clock configuration: v From the command line, type./ptconfig -setclock v From the ProtecTIER Service menu (see ProtecTIER Service menu on page 16), select ProtecTIER Configuration > Update Time, Date, Timezone and Timeserver(s) The following menu is displayed: Date, Time, Timezone & Timeserver(s) configuration ================================================== 1. Set date & time 2. Set Timezone 3. Set Timeserver(s) c. Commit changes and exit q. Exit without committing changes Please Choose: 5. Synchronize the date and time by typing 1 in the Please Choose: field. The following is displayed: Please Choose:1 Please specify the date in DD/MM/YYYY format [08/11/2010]: 6. Type the date in the specified format and press Enter. If you do not enter a date, the default (displayed in brackets) is used. The following is displayed: Please specify the date in DD/MM/YYYY format [08/11/2010]: 08/11/2010 Please specify the time in HH:MM:SS format [11:56:16]: 7. Type the time in the specified format and press Enter. If you do not enter a time, the default (displayed in brackets) is used. The following is displayed: Date, Time, Timezone & Timeserver(s) configuration ================================================== 1. Set date & time 2. Set Timezone 3. Set Timeserver(s) c. Commit changes and exit * q. Exit without committing changes Please Choose: Note: If you typed in new information, an asterisk (*) appears at the end of the c. Commit changes and exit menu option to show that there are modifications to be saved. 108 TS7650 and TS7650G Problem Determination and Service Guide

133 You can choose either c to save the changes and exit the setclock configuration, choose q to exit the configuration without saving the changes, or continue synchronizing the local timezone (step 5 on page 108) or adding the timeservers to the system (step 12 on page 110). 8. Synchronize the local timezone by typing 2 in the Please Choose: field. The following is displayed: Please Choose:2 Enter a 2 letter country code (or type m to enter the timezone manually): If you typed m to manually enter the timezone, the following is displayed: Enter the time zone (case sensitive): 9. Select a country. v Type a two-letter country code (for example, US for the United States) v If you chose to enter the timezone manually, type the name (for example, Italy). If you need a list of time zone codes from countries other than the United States, see Appendix K, Worldwide time zone codes, on page 303. The following is displayed: Time zones under US: ==================== 1. America/New_York 2. America/Detroit 3. America/Kentucky/Louisville 4. America/Kentucky/Monticello 5. America/Indiana/Indianapolis 6. America/Indiana/Vincennes 7. America/Indiana/Knox 8. America/Indiana/Winamac 9. America/Indiana/Marengo 10. America/Indiana/Vevay 11. America/Chicago 12. America/Indiana/Tell_City 13. America/Indiana/Petersburg 14. America/Menominee 15. America/North_Dakota/Center 16. America/North_Dakota/New_Salem 17. America/Denver 18. America/Boise 19. America/Shiprock 20. America/Phoenix 21. America/Los_Angeles 22. America/Anchorage 23. America/Juneau 24. America/Yakutat 25. America/Nome 26. America/Adak 27. Pacific/Honolulu Please choose a timezone: 10. Type a number corresponding to the timezone you want to synchronize from the list under the country code entered. The following is displayed: Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 109

134 Date, Time, Timezone & Timeserver(s) configuration ================================================== 1. Set date & time 2. Set Timezone 3. Set Timeserver(s) c. Commit changes and exit * q. Exit without committing changes Please Choose: 11. You can choose either c to save the changes and exit the setclock configuration, choose q to exit the configuration without saving the changes, or continue synchronizing the local timezone or adding the timeservers to the system. 12. Add timeservers to the system by typing 3 in the Please Choose: field. The following is displayed: a. Please specify the timeserver's IP Address: Type the timeserver's IP address. For example: b. Would you like to set a secondary timeserver? (yes no) Type y to set a secondary timeserver. c. Please specify the secondary timeserver's IP Address: Type the secondary timeserver's IP address. For example: You can search your region for the appropriate NTP timeserver you want to synchronize your system with using your internet search engine. There are several sites available, one example is the NIST Internet Time Servers web page. After specifying the timeserver s addresses, the following is displayed: Date, Time, Timezone & Timeserver(s) configuration ================================================== 1. Set date & time 2. Set Timezone 3. Set Timeserver(s) c. Commit changes and exit * q. Exit without commiting changes Please Choose: 13. Confirm and save your changes to the configuration by typing c in the Please Choose: field. A list of the configuration changes is displayed for review. Results Note: If only the date was modified, and not the timezones or timeservers, only the date is displayed in the contents. Example If you modify the date, add a primary and secondary timeserver, and specify America/Chicago as the time zone, when you choose c to commit the changes and exit, the file contents are displayed as follows: 110 TS7650 and TS7650G Problem Determination and Service Guide

135 Please review the following information: ======================================== Date: Mon Nov 8 14:51: Primary time server: Secondary time server: Timezone: America/Chicago Do you wish to apply those settings? (yes no) Type y to apply the changes. Note: the cluster & VTFD services on all nodes must be stopped in order to continue. Do you wish to continue? (yes no) Type y to stop and restart the cluster and VTFD services on all the nodes. Stopping Cluster Services [ Done ] Stopping NTPD [ Done ] Setting Time Zone [ Done ] Setting Timeserver [ Done ] Setting Date & Time [ Done ] Starting NTPD [ Done ] Starting cluster [ Done ] Cluster Started What to do next When both nodes finish rebooting, use ProtecTIER Manager to verify that fencing completed successfully and both nodes are online: To do so: 1. On the ProtecTIER Manager workstation or the TSSC, open ProtecTIER Manager. 2. Confirm that both nodes display as Online. v If both nodes are online, proceed to Performing the system validation test. v If Node 1 is offline, manually reboot Node 1: a. If necessary, connect the USB keyboard and monitor to Node 1 and log in to the server with the username root and password admin. b. At the command line, type: reboot c. If both nodes display as online, proceed to Performing the system validation test. d. If one (or both) nodes displays as offline, contact your next level of support. Performing the system validation test After system recovery, configuration, and syncing the system times, you must perform the system validation test. Procedure Run the test on both nodes, starting with Node If you are not already connected to Node 1, plug a USB keyboard and monitor into the back of the server. 2. Log in to Node 1 with the username root and password admin. 3. At the command prompt, enter the following command: Chapter 4. ProtecTIER server (3958 DD4, 3958 DD5, and AP1) 111

136 cd /opt/dtc/install 4. From the command prompt, enter the following command:./ptconfig -validate The following messages are displayed: Testing fence ports [Done ] Validation is about to execute a fence on 1 node in the cluster, the node will be forcefully shutdown and rebooted To Continue please type "fence test", or "q" to quit: 5. Type fence test. The following messages are displayed: Testing connectivity to the fence device [ Done ] Would you like to stop the VTFD service on both nodes? (yes no) 6. Type yes. The following message is displayed: Stopping VTFD locally [-] 7. After vtfd is stopped and the test finishes on Node 1, disconnect the keyboard and monitor from Node 1. Connect them to Node Log in to Node 2 with the username root and password admin. 9. Repeat steps 3 on page 111 through 6 on Node B. Note: Alerts might be generated as the validation tests are running. The ProtecTIER Manager Alerts Log allows you to monitor the alerts as they occur. To do so, on the ProtecTIER Manager workstation, click Alerts in the lower right corner of the ProtecTIER Manager screen. The Alerts Log is displayed (Figure 35). ts Figure 35. Alerts Log When you are finished reviewing alerts, click Clear Alerts in the lower-right corner of the Alerts Log screen. 10. If the test succeeds on both nodes, TS7650G installation and setup is complete. Turn the system over to the customer and refer them to the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA for information on customizing, using, and maintaining ProtecTIER; and creating, configuring, and managing replication grids, if applicable. 11. If the test fails on either node, contact your next level of support to resolve the failure conditions. 112 TS7650 and TS7650G Problem Determination and Service Guide

137 Chapter 5. TSSC The TS7650 Appliance and TS7650G Gateway servers require a connection to a TS3000 System Console (TSSC). Note: Depending upon the model of the TSSC and TSSC network switch being used, the port layout might be different than shown. If so, attach and label the cables according to the port number assignments specified, regardless of the position of the ports on the TSSC or the TSSC network switch. Refer to the IBM System Storage TS3000 System Console (TSSC) Maintenance Information for detailed port information. See TSSC connections for DD4, DD5, and AP1 servers for cabling for the DD4, DD5, and AP1 servers. See TSSC connections for DD1 and DD3 servers on page 120 for cabling for the DD1 and DD3 servers. See the IBM System Storage TS3000 System Console (TSSC) Maintenance Information document for detailed information about troubleshooting and repairing the TSSC. Ethernet connections between the TSSC and the disk controllers are not implemented, and therefore are not shown in the figures in this section. Note: To test operational connectivity between the server and TSSC for call home, see Troubleshooting the server-to-tssc connection on page 24. TSSC connections for DD4, DD5, and AP1 servers This topic provides details for the TSSC connections to the DD4, DD5, and AP1 servers. Note: Depending upon the model of the TSSC and TSSC network switch (16-port or 26-port) being used, the port layout might be different than shown. If so, attach and label the cables according to the port number assignments specified, regardless of the position of the ports on the TSSC or the TSSC network switch. Refer to the IBM System Storage TS3000 System Console (TSSC) Maintenance Information for detailed port information. Copyright IBM Corp. 2011,

138 KVM switch P SC SW Customer Network SC Server V 9 U SC KVM 22 U ProtecTIER Server A Terminator P 13 P A1 A2 B V S 8 9 E1 E2 E3 E4 23 ts Figure 36. Stand-alone TS7650G (3958 DD4/3958 DD5) or TS7650 Appliance (3958 DD4) attachment to the TSSC (x3250 M3 MT4252) 114 TS7650 and TS7650G Problem Determination and Service Guide

139 ProtecTIER Server B 25 KVM switch P P A1 A2 B V S E1 E2 E3 E4 P SC SW Customer Network SC Server V 9 U SC KVM 22 U ProtecTIER Server A Terminator P 13 P A1 A2 B V S 8 9 E1 E2 E3 E4 23 ts Figure 37. Clustered TS7650G (3958 DD4/3958 DD5) or TS7650 Appliance (3958 DD4) attachment to the TSSC (x3250 M3 MT4252) Chapter 5. TSSC 115

140 USB Modem (USBM) KVM switch 29 Customer Telephone Line CTL USB 28 P SC SW Customer Network SC Server V 9 U1 1 2 U3 24 SC KVM 22 U ProtecTIER Server A Terminator P 13 P A1 A2 B V S 8 9 E1 E2 E3 E4 23 Figure 38. Stand-alone TS7650 Appliance (3958 DD4) attachment to the TSSC (x3250 M4 MT 2538) ts TS7650 and TS7650G Problem Determination and Service Guide

141 USB Modem (USBM) 29 Customer Telephone Line CTL USB 28 ProtecTIER Server B 25 KVM switch P P A1 A2 B V S E1 E2 E3 E4 P SC SW Customer Network SC Server V U1 U3 24 SC KVM 22 U ProtecTIER Server A Terminator P 13 P A1 A2 B V S 8 9 E1 E2 E3 E4 23 Figure 39. Clustered TS7650 Appliance (3958 DD4) attachment to the TSSC (x3250 M4 MT 2538) ts Chapter 5. TSSC 117

142 USB Modem (USBM) Customer Telephone Line 29 CTL USB 28 KVM switch P U SC Server V 9 U1 1 2 U3 Terminator 10 Customer Network SC KVM SC SW P RAS ProtecTIER Server A P RAS 13 A1 A2 B V S 8 9 Figure 40. Stand-alone TS7650G (3958 DD5) attachment to the TSSC (x3250 M4 MT 2538) P E1 E2 E3 E4 ts TS7650 and TS7650G Problem Determination and Service Guide

143 USB Modem (USBM) Customer Telephone Line 29 CTL USB 28 KVM switch P U Terminator 24 SC Server V SC KVM U1 1 2 U Customer Network 21 SC SW P RAS RAS ProtecTIER Server B P P A1 A2 B V S 8 9 E1 E2 E3 E4 14 RAS ProtecTIER Server A P RAS 13 A1 A2 B V S 8 9 Figure 41. Clustered TS7650G (3958 DD5) attachment to the TSSC (x3250 M4 MT 2538) P E1 E2 E3 E4 ts Note: Ethernet port connections from the 3958 DD4 and 3958 DD5 servers to the TSSC are the same for Virtual Tape Library, OpenStorage, or File System Interface. Table 13. Stand-alone and clustered TSSC, KVM, and customer network Ethernet connections for the TS7650 Appliance and TS7650G On device or Item From On device To location Remarks 9 Port 5 SC SW (TSSC Ethernet switch) Port 1 10 Port 2 TSSC Customer local area network SC Server (TSSC) Customer specified device Chapter 5. TSSC 119

144 Table 13. Stand-alone and clustered TSSC, KVM, and customer network Ethernet connections for the TS7650 Appliance and TS7650G (continued) On device or Item From On device To location Remarks 11 Port 4 SC SW Port 4, slot 5 Server A RAS connection 12 Port 3 SC SW Port 4, slot 5 Server B RAS connection 13 Port 6 SC SW Port B Server A RAS connection (Sys Mgmt [IMM]) 14 Port 1 SC SW Port B Server B RAS connection (Sys Mgmt [IMM]) 21 Video port KVM Switch Video port SC KVM 22 Port 3 KVM Switch Video port SC Server Port U3 23 Port 1 KVM Switch Video port Server A Port E1 24 Port U1 KVM Switch Video Port SC KVM 25 Port 2 KVM Switch Video port, Server B Port E1 28 USB USB Modem Port U1 SC Server (TSSC) 29 CTL USB Modem Customer Telephone Line Customer specified device TSSC connections for DD1 and DD3 servers This topic provides details for the TSSC connections to the DD1 and DD3 servers. Note: Depending upon the model of the TSSC and TSSC network switch being used, the port layout might be different than shown. If so, attach and label the cables according to the port number assignments specified, regardless of the position of the ports on the TSSC or the TSSC network switch. Refer to the IBM System Storage TS3000 System Console (TSSC) Maintenance Information for detailed port information. The server E2 and RSA Ethernet ports connect to the TSSC by means of the TSSC Ethernet switch (see Figure 42 on page 121). Direct connection of the RSA Ethernet port to the customer LAN is not supported. In addition, with RAS/BIOS firmware versions 2.5 and higher, Ethernet connections from the storage controllers to the TSSC Ethernet switch are no longer required. 120 TS7650 and TS7650G Problem Determination and Service Guide

145 TSSC Customer connection 1 2 TSSC network Ethernet switch xx 3958 AP1 Server A RSA E1 E2 ts Figure 42. Stand-alone TS7650 Appliance (3958 DD3) attachment to the TSSC Chapter 5. TSSC 121

146 3958 AP1 Server B RSA E1 E2 TSSC Customer connection 1 2 TSSC network Ethernet switch xx 3958 AP1 Server A RSA E1 E2 ts Figure 43. Clustered TS7650 Appliance (3958 DD3) attachment to the TSSC 122 TS7650 and TS7650G Problem Determination and Service Guide

147 TSSC Customer connection 1 2 TSSC network Ethernet switch xx 3958 DD1/DD3 Server A RSA E1 E2 ts Figure 44. Stand-alone TS7650G (3958 DD1/3958 DD3) attachment to the TSSC Chapter 5. TSSC 123

148 3958 DD1/DD3 Server B RSA E1 E2 TSSC Customer connection 1 2 TSSC network Ethernet switch xx 3958 DD1/DD3 Server A RSA E1 E2 ts Figure 45. Clustered TS7650G (3958 DD1/3958 DD3) attachment to the TSSC 124 TS7650 and TS7650G Problem Determination and Service Guide

149 Chapter 6. Disk controller This section describes disk controllers. TS7650 Appliance customers who purchased systems before August 2010 should have the DS4700 disk controller, machine-type model H. TS7650 Appliance customers who purchased systems in August 2010 or later should have the DS3950 disk controller. Important: For TS7650 and TS7650G installations, this section applies only if the customer back-end storage controller is an IBM System Storage DS4700 Express or DS3950. If the customer back-end storage controller differs, use the troubleshooting procedures that were provided by the manufacturer with the controller in the original configuration. All attached storage to the TS7650G is supported separately, and problems are not called home for it. The disk controller supports sixteen 450 GB, 4 Gbps, 15 krpm, Fibre Channel disk drive modules (DDMs). The disk controller comes with two RAID controllers, two power supplies, and two cooling units. It provides dual, redundant controllers, redundant cooling, redundant power, and battery backup of the RAID controller cache. The disk controller is designed to provide maximum host and drive-side redundancy, enabling full redundant drive loop support to additional disk storage capacity in the expansion frames via the drive loop port from each RAID controller. In addition, each RAID controller supports direct attachment of two servers that contain two Fibre Channel host bus adapters each. External cables and small form-factor pluggable (SFP) modules connect the disk controller to the expansion units and the servers. For optimum performance, RAID 5 configuration is recommended. Note: Currently, the only expansion unit that can be attached to the IBM-provided disk controller is the H disk expansion unit, Feature Code 3707: 4.8 TB Fibre Channel Disk Expansion Unit. For more support information about the disk controller, go to the IBM Support Portal and select Disk systems > DS4700 Express. Verifying and upgrading the DS4700 storage system firmware, NVSRAM, and ESM This topic provides procedures for upgrading the firmware level of your DS4700 storage system. v Before upgrading to ProtecTIER code level 2.3, you must upgrade the firmware on your DS4700 storage subsystem to a minimum of or preferred level of See Verifying and upgrading the storage system firmware, NVSRAM, and ESM (ProtecTIER version 2.3 and earlier) on page 126. v If the disk controller and disk expansion unit are at the minimum level of ProtecTIER V2.4 code, the firmware is verified and upgraded automatically. See Verifying and upgrading the storage system firmware, NVSRAM, and ESM (ProtecTIER version 2.4 and later) on page 129. Copyright IBM Corp. 2011,

150 Verifying and upgrading the storage system firmware, NVSRAM, and ESM (ProtecTIER version 2.3 and earlier) This topic describes the procedure for verifying and upgrading the storage system firmware, NVSRAM, and ESM for version 2.3 and earlier. Before you begin The firmware on your DS4700 storage subsystem must be a minimum of or recommended level of before upgrading to ProtecTIER code level 2.3. Attention: Before upgrading the ProtecTIER code level, you must update the DS4700 storage subsystem firmware, NVSRAM firmware, and the ESM firmware on your EXP810 expansion drawers. You must call IBM Service to schedule a time to update the firmware to the levels listed below. Do not proceed to upgrade the ProtecTIER code level to version 2.4.x until: v The storage subsystem firmware has been upgraded to the minimum level of or the recommended level of v The DS4700 NVSRAM firmware has been updated to the minimum level of N1814D47R1050V07 or the recommended level of N1814D47R1060V04. v The expansion unit ESM firmware has been updated to the minimum level of 98C5 or the recommended level of 98D0. Attention: The storage firmware upgrade process is not a concurrent procedure. All services must be stopped on the ProtecTIER node (both nodes if clustered), before beginning the firmware upgrade. Following the firmware update, you must restart the services on both nodes, starting with node A. 1. Stop the vtfd and cluster services on the nodes, starting with node B. Type the following commands from the command line: service vtfd stop service ptcluster stop 2. Upgrade the storage firmware using the following procedure. 3. Restart the services on the nodes, starting with node A, by typing the commands: service vtfd start service ptcluster start Procedure 1. Verify that the firmware is at the correct code level. a. If you are not already logged in to the TSSC, log in with user ID service and password service. b. Right-click the blue desktop, and from the IBM TS3000 System Console menu, select Browser Functions > Storage Manager GUI Functions > Launch GUI. Note: If you receive a message that the DS Storage Manager GUI is not installed, you must install it using the supplied DS Storage Manager Graphical User Interface CD version G5.16. Use step 3 on page 127 to perform the installation and return to this step after DS Storage Manager is installed. If the DS Storage Manager GUI is already installed, verify the version by selecting Help > About. If the DS Storage Manager version is 126 TS7650 and TS7650G Problem Determination and Service Guide

151 earlier than G5.16 use step 3 to install the new version of DS Storage Manager and return to this step. The new version of DS Storage Manager replaces the previous version. c. If multiple storage systems are listed, select a storage system in the left navigation pane. When the subsystem list is displayed in the right pane, select the list, right-click it, and select Locate Storage Subsystem from the menu. The indicator lights on the storage subsystem flash until you click OK. If the storage system you want to update is not listed, select Edit > Add Storage Subsystem. When prompted in the Add New Storage Subsystem menu, enter the Out-of-Band or In-Band IP address of the storage subsystem and click Add. DS Storage Manager attempts to locate the subsystem and add it to the storage list. If this process fails, verify the IP addresses entered. d. After the correct subsystem has been identified, in the right pane the storage subsystem should display as Optimal. If it does not, resolve any failure indictions before proceeding with the firmware upgrade. If you cannot resolve a failure condition and return the system to optimal status, contact your next level of IBM support. e. If the storage displays as optimal status, double-click to open the array. At the Synchronize window, click Cancel. If prompted to set a storage system password, click No. f. Click the Logical/Physical tab. Controller A and B should be displayed. g. Right-click Controller A in the Storage Subsystem window. h. Select Properties and verify that the levels displayed for the DS5020 Firmware and NVSRAM are as listed in the example: v Firmware Version: DS5020: DS3950: v NVSRAM Version: DS4700: N1814D47R1060V04 DS5020: N1814D20R106V13 DS3950: N1814D50R1060V01.dlp i. Close the Properties window. From the title bar, select Advanced > Maintenance > Download > ESM Configuration Settings. Verify that Card A and Card B firmware are at firmware level 98D0. j. Right-click Close to close the window. k. If the code level is correct, no upgrade is required. If the code level is not at the correct level, continue with the upgrade process. Go to step 2 and continue upgrading the DS5020 firmware and NVSRAM. 2. Copy the minimum level of 98C5 or recommended level of 98D0 V or higher to the TSSC. a. Right-click the blue desktop of the TSSC and from the Systems Console menu select System Console Actions > CD copy to console. b. On the CD copy to console screen, select option 3, /home/service/offload. c. Follow the instructions to copy the code. d. After the code has been copied, the disk is ejected. 3. Upgrade the DS Storage Manager to version using the TSSC. a. Right-click the TSSC blue desktop and select Browser Functions > SM GUI Functions > Install GUI. Chapter 6. Disk controller 127

152 b. Insert the DS Storage Manager Graphical User Interface CD version G5.16 (in the TSSC ship group) into the TSSC and follow the instructions to complete the installation. 4. Locate the DS5020 firmware and NVSRAM files using the DS Storage Manager. a. Launch the DS Storage Manager from the TSSC by right-clicking the TSSC and selecting Browser Functions > SM GUI Functions > Launch GUI. b. Select the Storage Subsystem to be upgraded (identified in step 1 on page 126) by right-clicking on the storage and selecting Manage Storage Subsystem. If prompted to set a password, click No. If prompted with a Synchronize Controller Clocks window, click Cancel. c. In the Manage Storage Subsystem window, select the Advanced button in the top menu bar. d. Select Maintenance > Download > Controller Firmware. The Download Controller Firmware window is displayed. DS Storage Manager checks the health of the storage subsystem. This check might take several minutes. If there is a problem associated with the storage, it is identified, and you cannot continue with the upgrade until the problem is resolved. If you cannot isolate or resolve the error condition, contact your next level of support. e. In the Controller Firmware panel, click the Select File button. A new window is displayed. f. Enter the path of the controller firmware and press Enter. The path is similar to the following: /home/service/offload/0760/ds4700. In the same window, click to highlight the FW.DS4700_ dpl file from the Files section. g. Click the OK button at the bottom of the window. The Download Controller Firmware window is displayed. h. In the Download Controller Firmware window, select the Transfer the NVSRAM file with controller firmware button. i. Click the Select File button in the NVSRAM section. A new window is displayed. Enter the path of the NVSRAM and press Enter. The path is similar to the following: /home/service/offload/0760/ds4700. In the same window, click to highlight the N1814D47R1060V04.dlp file from the Files section. j. Click the OK button at the bottom of the window. The Download Controller Firmware window is displayed. 5. Upgrade the DS4700 firmware and NVSRAM using the DS Storage Manager. In the Download Controller Firmware window, select the Transfer button. When prompted with a version compatibility window to confirm the download and asked, "Do you want to continue?" click Yes. The transfer and activation process should take minutes to complete. Note: During this activation, both A and B controllers are upgraded. When the activation is completed, a New Firmware Detected window is displayed. Click OK. 6. Upgrade the ESM level. a. In the Manage Storage Subsystem window, click the Advanced button in the top menu bar. b. Select Maintenance > Download > ESM Firmware. The Download Environmental (ESM) Card Firmware window is displayed. c. In the Drive Enclosure section, select the enclosure to be upgraded. d. In the ESM Firmware section, click the Select File button. A new window is displayed. 128 TS7650 and TS7650G Problem Determination and Service Guide

153 e. Type the path of the ESM and press Enter. The path is similar to the following: /home/service/offload/0760/ds4700. f. In the same window, select the esm98c5.esm file from the Files section. g. Click the OK button at the bottom of the window. The Download Environmental (ESM) Card Firmware window is displayed again. h. Click the Start button in the Download Environmental Card Firmware window to start the ESM download process. A Confirm Download dialog is displayed. i. Type yes and click OK to continue. The enclosures are updated. This process takes 5-7 minutes for each enclosure. The Status column displays Transferring until the process is complete. The enclosures can only be updated one at a time. j. Repeat substeps 6c on page 128 through 6h for each enclosure. Results The upgrade process of the DS4700 firmware and NVSRAM is complete. What to do next Close any open DS Storage Manager windows, right-click the blue desktop of the TSSC and select Logout. Verifying and upgrading the storage system firmware, NVSRAM, and ESM (ProtecTIER version 2.4 and later) This topic describes how to verify and upgrade the storage system firmware, NVSRAM, and ESM for version 2.4 and later. If the disk controller and disk expansion unit are at the minimum level of ProtecTIER V2.4 code, the firmware is verified and upgraded automatically. During installation, you are asked whether you want to update the controller code. Answer yes and the disk controller and disk expansion unit firmware is automatically updated. Restriction: ProtecTIER V3.1.8 does not support automatic firmware updates. You must use the procedure for manual updates in Appendix D, Manually upgrading the DS4700 storage system firmware (ProtecTIER version 2.4 and later), on page 271. If it is necessary for you to verify and upgrade the firmware manually, see Appendix D, Manually upgrading the DS4700 storage system firmware (ProtecTIER version 2.4 and later), on page 271. Determining firmware levels for the DS3950 storage subsystem There are two different methods to determine the DS3950 storage subsystem and firmware versions. Each method uses the DS Storage Manager client that manages the DS3950 storage subsystem. Method One Procedure 1. Go to the Subsystem Management window and select Storage Subsystem > View > Profile. The Storage Subsystem Profile window is displayed. Chapter 6. Disk controller 129

154 2. Select the Controllers tab first and note the Firmware, NVSRAM, Appware and Bootware versions. 3. Select the Drives tab and note the Firmware version. 4. Select the Enclosures tab and note the ESM Firmware version. 5. Scroll through each list to locate the following information. DS3950 Storage Controllers v v v v Firmware Version NVSRAM version Appware version Bootware version Drives v v Firmware version ATA translator card firmware version (SATA E-DDMs only) Note: The ATA translator card firmware and drive firmware are typically packaged together as a single drive firmware upgrade file. ESM ESM card firmware version Method Two Complete the applicable procedure to obtain the specified firmware version. Controller firmware version Click the Controller icon in the Physical View pane of the Subsystem Management window (see Properties in the right pane). The Controller Enclosure properties window displays the properties for that controller. You must perform this step for each individual controller. Drive firmware (and ATA translator card firmware) version Click the Controller icon in the Physical View pane of the Subsystem Management window (see Properties in the right pane). The Drive Properties window displays the properties for that drive. You must perform this step for each individual drive. ESM and drive enclosure component firmware versions 1. In the Physical View pane of the Subsystem Management window, click View Enclosure Components for each Expansion enclosure. The ESM information is displayed in the right pane of the Drive Enclosure Component Information window. 2. Locate the firmware version of each ESM in the drive enclosure. Upgrading the firmware of the DS3950 storage subsystem To upgrade the DS3950 firmware, see Appendix D, Manually upgrading the DS4700 storage system firmware (ProtecTIER version 2.4 and later), on page 271 or refer to the DS3950 Storage Subsystem Installation, User's, and Maintenance Guide. 130 TS7650 and TS7650G Problem Determination and Service Guide

155 Disk controller components The following sections show the components of the disk controller. Hot-swap capable devices enable you to maintain the availability of your system while you remove, install, or replace a hot-swap device. The disk controller supports hot-swap Fibre Channel hard disk drives (HDDs) and cooling fan field-replaceable units (FRUs), so you can remove and replace these components without turning off the disk controller. In addition, the disk controller supports hot-swap RAID controllers and power supplies. Hot-swap drive bays This topic describes the disk controller hot-swap drive bays. The hot-swap drive bays that are accessible from the front of your disk controller are shown in Figure 46. ts Figure 46. Disk controller hot-swap drive bays The disk controller supports up to 16 Fibre Channel hard disk drives (HDDs). These drives are preinstalled in drive trays. The unit consisting of the drive and the drive carrier assembly is called a drive field-replaceable unit (FRU). The Fibre Channel drive FRU includes the drive tray, the Fibre Channel drive, and the HDD interposer card. You install the drive FRUs in the 16 drive bays on the front of the disk controller. There are no serviceable parts in a drive FRU. If a drive FRU fails, it must be replaced in its entirety (drive, interposer, bezel, and tray). Attention: v Never hot-swap a drive FRU when its associated green activity LED is flashing. Hot-swap a drive FRU only when its associated amber fault LED light is not flashing, or when the drive is inactive and its associated green activity LED light is not flashing. v The Fibre Channel drive and its HDD card are digitally signed together at the factory. Do not swap drives and interposers between drive FRUs. Do not replace one Fibre Channel drive with another Fibre Channel drive. These actions result in a drive FRU that cannot be recognized by the disk controller RAID controller. v After you remove a drive FRU, wait at least 70 seconds before replacing or reseating the drive FRU to allow the drive to properly spin down. Important: If the disk drive you want to remove is not in a failed or bypass state, always use Storage Manager either to place the drive in a failed state or to place the array that is associated with the drive (or drives) in an offline state before you remove the drive from the enclosure. Chapter 6. Disk controller 131

156 Disk controller front indicators This section describes the primary indicators on the front of the disk controller ts Figure 47. Disk controller front indicators Table 14. Disk controller front indicators Number Indicator Normal Status Problem Status 1 Drive activity LED On (not flashing) - No data is being processed Off 2 Drive fault LED Off Flashing - data is being processed On (not flashing) Flashing - Drive, volume, or storage array locate function 3 Global locate LED On N/A 4 Global summary fault LED Off On 5 Global power LED On Off 6 Link rate switch LED Set to the 2 Gbps or 4 Gbps position, depending on your configuration. N/A 132 TS7650 and TS7650G Problem Determination and Service Guide

157 The global power, global summary fault, and global locate LEDs are general indicators for the entire disk controller (or configuration, in the case of the global summary fault). The global summary fault LED turns on if a fault condition is detected in any component in the storage subsystem configuration, including all attached disk expansion modules. The global power, global summary fault, and global locate LEDs shine through the front bezel. Attention: If the global summary fault LED lights, refer to the client software to determine the problem with the disk controller, and check the needs attention LEDs on the other disk components to help isolate the fault to a specific component. Disk controller rear indicators This section describes the primary indicators on the rear of the DS4700 and DS3950 disk controllers ts Figure 48. DS4700 disk controller rear indicators Table 15. DS4700 disk controller rear indicators Number LED Normal Status Problem Status 1 Service action allowed (OK to remove) Off On Chapter 6. Disk controller 133

158 Table 15. DS4700 disk controller rear indicators (continued) Number LED Normal Status Problem Status 2 Service action required (fault) Off On 3 Cache active On - Data is in cache Not applicable Off - Caching is turned off. No data in cache 4 Diagnostic On - Seven-segment LEDs indicate diagnostic code Not applicable Off - Seven-segment LEDs indicate enclosure ID 5 Heartbeat Flashing Off 6 Host channel speed - L1 See Table 17 on page Host channel speed - L2 See Table 17 on page Ethernet link speed On Mbps Not applicable Off - 10 Mbps 9 Ethernet link activity On - link established Not applicable Off - no link established 10 Drive channel port bypass (One LED per port) Note: The drive channel consists of two fibre channel ports. This LED indicates the drive port bypass status of one of the two fibre channel ports that comprise a drive channel. The LED marked 13 shows the status of the other port. Flashing - activity Off (Also off if no SFP connected) 11 Drive channel speed - L1 See Table 17 on page Drive channel speed - L2 See Table 17 on page 136. On - No valid device detected and port is bypassed 134 TS7650 and TS7650G Problem Determination and Service Guide

159 Table 15. DS4700 disk controller rear indicators (continued) Number LED Normal Status Problem Status 13 Drive channel port 2 bypass (One LED per port) Note: The drive channel consists of two fibre channel ports. This LED indicates the drive port bypass status of one of the two fibre channel ports that comprise a drive channel. The LED marked 10 shows the status of the other port. Off (Also off if no SFP connected) On - No valid device detected and port is bypassed ts Figure 49. DS3950 disk controller rear indicators Table 16. DS3950 disk controller rear indicators Number Description 1 Direct current enabled LED 2 Service action required (fault) LED 3 Service action allowed (ok to remove) LED 4 Power supply and fan unit power LED The L1 and L2 LEDs for each host and drive channel combine to indicate the status and the operating speed of each host and drive channel. Chapter 6. Disk controller 135

160 Table 17. Host and drive channel LED definitions L1 L2 Definition Off Off When both LEDs for host or drive channel ports are off (unlit), this indicates one or more of the following conditions: v The host or drive channel ports are bad or the SFPs are faulty. v An SFP module is inserted with no Fibre Channel cable attached. Note: If the SFP is inserted in a drive port with no fibre channel attached, the associated drive port bypassed LED is also lit. In addition, the L1 and L2 LEDs are unlit. v No SFP module is inserted in one or both of the host or drive ports in the channel. On Off The host port is operating at 1 Gbps. Note: This pattern is not applicable because the H disk controller is not does not support a Fibre Channel speed of 1 Gbps. Off On The host or drive channel port is operating at 2 Gbps. On On The host or drive channel port is operating at 4 Gbps. Battery unit LEDs This section describes the primary LEDs on the disk controller battery units ts Figure 50. Battery unit LEDs Table 18. Battery unit LEDs Number LED Normal status Problem status 1 Service action allowed (OK to remove) Off On 2 Service action required (fault) 3 Battery charging Off On, ready Flashing, charging On Off 136 TS7650 and TS7650G Problem Determination and Service Guide

161 ds48042 Seven-segment numeric display LEDs The numeric display consists of two seven-segment LEDs that provide information about enclosure identification and diagnostics. Figure 51 shows the numeric display and the diagnostic LED. Sevensegment display Figure 51. Numeric display LEDs Diagnostic LED When you power on the disk controller, the seven-segment LEDs show various codes as the controllers are going through the boot process. When the server successfully completes the power-on boot process, the numeric display shows the enclosure identification (enclosure ID) of the disk controller. If an error has occurred and the Needs attention LED is on, the numeric display shows diagnostic information. The numeric display indicates the information is diagnostic by illuminating an LED that appears as a decimal point between the display numbers. The diagnostic LED tuns off when the numeric display shows the disk controller enclosure ID. The numeric display shows various diagnostic codes as the disk controller performs the startup process after each power cycle or reset. After diagnostics are complete, the current disk controller enclosure ID is displayed. Diagnostic codes in the form of Lx, where x is a hexadecimal digit, indicate disk controller state information. In general, these codes are displayed only when the disk controller is in a non-operational state. The disk controller might be non-operational due to a configuration problem (such as mismatched types), or it might be non-operational due to hardware faults. If the disk controller is non-operational due to system configuration, the disk controller Needs attention LED is off. If the disk controller is non-operational due to a hardware fault, the disk controller Needs attention LED is on. The definitions for Lx diagnostic codes are listed in Table 19 on page 138. Chapter 6. Disk controller 137

162 Table 19. Numeric display diagnostic codes Controller Value State Description Storage Manager View L0 Suspend Mismatched controller board ID Needs Attention condition for board ID mismatch L2 Suspend Persistent memory errors Needs Attention condition for offline controller L3 Suspend Persistent hardware errors Needs Attention condition for offline controller L5 Offline The alternate controller has incompatible firmware but automatic controller firmware synchronization (ACS) cannot be performed L7 Suspend/ Offline Controller with different controller submodel ID inserted. L8 Offline Unsupported memory is present or memory is not populated in the correct memory slots 88 Reset Controller is held in reset by the other controllers Needs Attention condition for offline controller Needs Attention for offline controller Need Attention because the controller is in offline/failed state Needs Attention condition for offline controller Disk controller enclosure ID setting: The enclosure ID, comprised of two seven-segment numbers, is located on the back of each controller next to the indicator lights. It provides a unique identifier for each enclosure in the disk storage configuration. The controller automatically sets the enclosure ID number. You can change the setting through the storage management software if necessary. Both controller enclosure ID numbers are identical under normal operating conditions. The allowable ranges for enclosure ID settings are Do not set the enclosure ID to 00 or any number less than 80. The disk controller enclosure ID is normally set to a value of 85 at the factory. Figure 52 on page 139 shows the enclosure ID. 138 TS7650 and TS7650G Problem Determination and Service Guide

163 Figure 52. Disk controller seven-segment enclosure ID Table 20. Disk controller seven-segment enclosure ID Number Description 1 Enclosure ID Disk controller Fibre Channel ports This topic describes the Fibre Channel ports of the DS4700 and DS3950 disk controllers. Figure 53 shows the locations of controllers A and B and the Fibre Channel ports on the back of the DS4700 disk controller ts Figure 53. DS4700 disk controller Fibre Channel ports Figure 54 on page 140 shows the locations of controllers A and B and the Fibre Channel ports on the back of the DS3950 disk controller. Chapter 6. Disk controller 139

164 ts Figure 54. DS3950 disk controller Fibre Channel ports Table 21 provides a list of the Fibre Channel ports associated with each of the controllers in the disk controller. Table 21. Disk controller Fibre Channel ports Number Description 1 Controller A 2 Controller B 3 Drive channel 1 - Port 2 on controller A 4 Drive channel 2 - Port 2 on controller B Drive channel 1 - Port 1 on controller A Drive channel 2 - Port 1 on controller B 5 Host port 2 6 Host port 1 Accessing disk components through Storage Manager Storage Manager provides access to the disk controller and attached disk expansion modules. Access to Storage Manager is accomplished either through the server RSA port or on a workstation connected to the TS7650 Appliance or TS7650G through the customer network. Note: Storage Manager is not included in TS7650G initial installations. This section assumes that customers with TS7650G installations are using the IBM H disk controller and H disk expansion units as back-end storage, and that Storage Manager is installed, configured, and available at the customer site to support those components. You can use Storage Manager to run diagnostics. See Figure 55 on page TS7650 and TS7650G Problem Determination and Service Guide

165 Figure 55. Storage Manager: Running controller diagnostics Whenever the controller cards are replaced with new field-replaceable units (FRUs), the firmware must be reloaded. Use the Storage Manager Download Firmware feature to update the firmware and NV-RAM. Refer to Figure 56. Figure 56. Storage Manager: Downloading firmware Use the Storage Manager Execute Script feature to execute scripts included on the disk storage configuration script CD to establish the file system on the disk components. Refer to Figure 57 on page 142. Chapter 6. Disk controller 141

166 Removing SFP modules Figure 57. Storage Manager: Executing scripts This topic describes the steps for removing SFP modules. Before you begin Attention: To prevent damage to the cable or to the SFP module, unplug the LC-LC Fibre Channel cable before you remove the SFP module. Procedure 1. Remove the LC-LC Fibre Channel cable from the SFP module. 2. Unlock the SFP module latch. For SFP modules that contain plastic tabs, unlock the SFP module latch by pulling the plastic tab outward at an angle of ten degrees, as shown in Figure 58 on page TS7650 and TS7650G Problem Determination and Service Guide

167 Plastic tab Protective cap 10 o SFP module ts With the SFP latch in the unlocked position, extract the SFP module. For SFP modules that contain plastic tabs, slide the SFP module out of the port. 4. Replace the protective cap on the SFP module. 5. Place the SFP module into a static-protective package. 6. Replace the protective cap on the port. What to do next Operating the disk controller Figure 58. Unlocking the SFP module latch - plastic variety For additional information see the IBM System Storage DS4700 Express Storage Subsystem Installation, User's and Maintenance Guide. To ensure optimal operation of your disk controller, always follow these guidelines. v To maintain power redundancy, plug the disk controller's right and left power supplies into two independent external power circuits through ac distribution units inside a rack cabinet or directly into external receptacles. Similarly, the right and left power supplies of the expansion frames attached to the disk controller should be plugged into the same two independent external power circuits as the disk controller. v Ensure that the disk controller is in an optimal state before you shut it down. Attention: Never turn the power off if any fault light is lit; be sure to resolve any error conditions before you shut down the system. v During any maintenance or attended power on procedure, carefully follow the power on sequence listed in Power procedures for ProtecTIER version 3.1 or later on page 237 or Powering on the ProtecTIER server version 2.5 or earlier on page 245. Check that each component of the TS7650 Appliance or TS7650G is turned on in the proper order during this entire power on procedure to ensure that the controller can optimally access all of your disk components. Chapter 6. Disk controller 143

168 v v The TS7650 Appliance and TS7650G support simultaneous power on to the system components; however, you should always follow the power on sequence listed in Power procedures for ProtecTIER version 3.1 or later on page 237 or Powering on the ProtecTIER server version 2.5 or earlier on page 245 during any attempted power on procedure. A disk controller in an optimal state should recover automatically from an unexpected shutdown and unattended simultaneous restoration of power to system components. After power is restored, call IBM support if any of the following conditions occur: The storage subsystem logical drives and arrays are not displayed in Storage Manager. The disk component logical drives and arrays do not come online. The disk component logical drives and arrays seem to be degraded. Disk controller speed setting The default switch setting is 4 Gbps. This is the only speed setting allowed for the disk controller. Attention: Do not remove the metal plate that covers the disk controller speed setting. Do not adjust the disk controller speed setting. Connecting disk expansion modules This topic addresses connecting disk expansion modules. Attention: In the initial installation of the disk controller, you can add only new disk expansion modules to the disk controller. This means that there must be no existing configuration information on the disk expansion module that you want to install. Improper drive migration can cause loss of configuration and other disk controller problems. 144 TS7650 and TS7650G Problem Determination and Service Guide

169 Chapter 7. Disk expansion units This topic lists the supported disk expansion units. Important: For TS7650G installations, this section does not apply. This section is for the TS7650 Appliance only. The back-end storage controller is as specified in Table 22. Table 22. Back-end storage controllers Systems purchased... Back-end storage controller Before August 2010 DS4700 August 2010 or later DS3950 Checking the LEDs The disk expansion units are IBM System Storage EXP810 Storage Expansion Units. If the customer has other back-end storage expansion units, use the troubleshooting procedures that were provided by the manufacturer with the expansion units in the original configuration. The disk expansion unit supports sixteen 450 GB, 4 Gbps, 15 krpm, Fibre Channel disk drive modules (DDMs). For optimum performance, RAID 5 configuration is recommended. The disk expansion unit supports redundant, dual-loop configurations with the disk controller and other disk expansion units. External cables and small form-factor pluggable (SFP) modules connect the controller to the disk expansion unit. The disk expansion unit provides continuous, reliable service and uses hot-swap technology. For more support information about the disk expansion unit, go to the IBM Support Portal and select Disk Systems > DS4700 EXP810 Storage Expansion Unit. The LEDs display the status of the disk expansion unit and components. Table 23. LED colors Color Meaning Green Normal operating status Amber Needs attention (possible failure) Blue v Safe to remove a component v Disk expansion unit is being identified or located It is important to check all the LEDs on the front and back of the disk expansion unit when you turn on the power. During power-on, the indicator LEDs flash intermittently as the disk expansion unit and components complete the power-on Copyright IBM Corp. 2011,

170 process. In addition to checking for faults, you can use the LEDs on the front of the disk expansion unit to determine if the drives are responding to I/O transmissions from the host. Disk expansion unit front LEDs This section describes the primary LEDs on the front of the disk expansion unit exp81021 Figure 59. Disk expansion unit front LEDs Table 24. Disk expansion unit front LEDs Number LED Normal status Problem status 1 Drive activity On (not flashing): No data is being processed. Off 2 Drive fault Off Flashing: Data is being processed. Flashing: Drive, volume, or storage array locate function. On (not flashing) Note: In certain ESM firmware versions, the Drive fault LED is lit when no drive is inserted in the drive slot. 3 Global locate On N/A See Removing and replacing disk expansion unit FRUs on page 215 for more information. 4 Global summary fault Off On 5 Global power On Off 6 Link rate switch The Link rate switch should be set to 2 Gbps or 4 Gbps, depending on your configuration. N/A 146 TS7650 and TS7650G Problem Determination and Service Guide

171 Disk expansion unit rear LEDs This section describes the primary LEDs on the rear of the disk expansion unit A ESM B ESM exp81022 Figure 60. Disk expansion unit rear LEDs Table 25. Disk expansion unit rear LEDs Number LED Normal Status Problem Status 1 ESM power On Off 2 ESM fault Off On 3 ESM service action allowed (OK to remove) Off On 4 ESM port 1 in bypass (This port is labeled 1A) 5 ESM ports1&2 2 or 4 Gbps data rate 6 ESM port 2 in bypass (This port is labeled 1B) 7 ESM port 3 in bypass (This port is labeled 2A) 8 ESM ports3&4 Off - Cable connected On - No cable connected One LED is lit if 2 Gbps Both LEDs are lit if 4 Gbps Off - Cable connected On - No cable connected Off - Cable connected On - No cable connected One LED is lit if 2 Gbps On, flashing for 30 seconds N/A On, flashing for 30 seconds On, flashing for 30 seconds N/A 2 or 4 Gbps data rate Both LEDs are lit if 4 Gbps 9 ESM port 4 in bypass (This port is labeled 2B) Off - Cable connected On - No cable connected On, flashing for 30 seconds Chapter 7. Disk expansion units 147

172 Power supply and fan unit LEDs This section describes the primary power supply and fan unit LEDs on the disk expansion unit. Figure 61. Power supply and fan unit LEDs Table 26. Power supply and fan unit LEDs Number LED Normal Status Problem Status 1 Power supply and fan unit power On Off 2 Service action allowed (OK to remove) Off On 3 Service action required (fault) Off On 4 Direct current enabled On Off Seven-segment numeric display LEDs The numeric display consists of two seven-segment LEDs that provide information about enclosure identification and diagnostics. Figure 62 on page 149 shows the numeric display and the diagnostic LED. 148 TS7650 and TS7650G Problem Determination and Service Guide

173 ds48042 Sevensegment display Diagnostic LED Figure 62. Numeric display LEDs When you power on the disk expansion modules, the seven-segment LEDs show 88 before the boot firmware starts to run. After the boot code starts, --is displayed. When the application code starts, the display shows either aa or bb, depending on whether it is ESM A or B. When the disk expansion module has completely started, the numeric display shows the enclosure identification (enclosure ID) of the disk expansion module. If an error has occurred and the Needs attention LED is on, the numeric display shows diagnostic information. The numeric display indicates the information is diagnostic by illuminating an LED that appears as a decimal point between the display numbers. The diagnostic LED tuns off when the numeric display shows the storage subsystem enclosure ID. The numeric display shows various diagnostic codes as the enclosures perform the startup process after each power cycle or reset. After diagnostics are complete, the current storage subsystem enclosure ID is displayed. Diagnostic codes in the form of Lx or Hx, where x is a hexadecimal digit, indicate disk expansion module state information. In general, these codes are displayed only when the disk expansion module is in a non-operational state. The disk expansion module might be non-operational due to a configuration problem (such as mismatched types), or it might be non-operational due to hardware faults. If the disk expansion module is non-operational due to system configuration, the disk expansion module Needs attention LED is off. If the disk expansion module is non-operational due to a hardware fault, the disk expansion module Needs attention LED is on. The definitions for Lx and Hx diagnostic codes are listed in Table 27 on page 150. Chapter 7. Disk expansion units 149

174 Table 27. Numeric display diagnostic codes Disk expansion Value module state Description SANtricity View L0 Suspend Mismatched ESM types L2 Suspend Persistent memory errors L3 Suspend Persistent hardware errors L9 Suspend Over temperature H0 Suspend SOC Fibre Channel interface failure H1 Suspend SFP speed mismatch H2 Suspend Invalid/incomplete configuration H3 Suspend Maximum reboot attempts exceeded H4 Suspend Cannot communicate with other ESM H5 Suspend Midplane harness failure Needs attention condition for board type mismatch Needs attention condition for offline ESM Needs attention condition for offline ESM Needs attention condition for offline ESM Needs attention condition for offline ESM Needs attention condition for offline ESM Needs attention condition for offline ESM H6 Suspend Firmware failure Needs attention condition for offline ESM H8 Suspend SFPs present in currently unsupported slot 88 Reset ESM is held in reset by the other ESMs 150 TS7650 and TS7650G Problem Determination and Service Guide

175 Table 28. Symptom-to-FRU index Problem indicator Component Possible cause Possible solutions Amber LED is lit Drive customerreplaceable unit (CRU) (Drive Fault LED) Drive failure Replace the failed drive. Note: The amber LED on the drive might also be lit if the drive slot has a blank drive tray inserted in certain ESM firmware versions. Amber LED is lit and green LED is off Drive CRUs Drive not certified Verify that the drive option or FRU/CRU P/N is listed in the support list for the appropriate disk controllers. ESM (Fault LED) ESM (Port Bypass LED) Front panel (Global Summary Fault LED) Power supply CRU ESM failure No incoming signal detected Incorrect enclosure speed setting ESM failure General machine fault Fibre Channel connection failure Power supply failure or power switch is turned off or there is an AC power failure. Replace the ESM. See your controller documentation for more information. Reconnect the SFP modules and the Fibre Channel cables. Verify the input and output SFP using the Fibre Channel loopback and female-female LC connector. Replace either input or output SFP modules or Fibre Channel cables as necessary. When connecting the disk expansion module to the existing operating redundant drive loop with a different speed setting, the port goes into bypass mode because mixing different speed enclosures within the same redundant drive loop/channel is not supported. If the ESM Fault and Service Action LEDs are lit, replace the ESM. A Fault LED is lit somewhere on the disk expansion module (check for Amber LEDs on CRUs). Check that the CRUs are properly installed. If none of the amber LEDs are lit on any of the CRUs, this indicates an SFP module transmission fault in the disk expansion module. Replace the failed SFP module. Replace the failed power supply, turn on all power supply switches, or check the main or the rack power unit circuit breaker. Chapter 7. Disk expansion units 151

176 Table 28. Symptom-to-FRU index (continued) Problem indicator Component Possible cause Possible solutions Amber and green LEDs are lit All green LEDs are off Power supply CRU v Fault and Power LEDs lit v Direct Current Enabled LED is not lit Power supply failure Replace the failed power supply. Drive CRUs Drive not certified Verify that the drive option or FRU/CRU P/N is listed in the support list for the appropriate disk controllers. Note: In certain cases, only the amber LED is lit, and the green activity LED is not lit. Use the Recovery Guru to better identify the cause of the drive failure. All CRUs Subsystem power is off AC power failure Power supply failure Check that all disk expansion module power cables are plugged in and the power switches are on. If applicable, check that the main circuit breakers for the rack are turned on. Check the main circuit breaker and AC outlet. Replace the power supply. 152 TS7650 and TS7650G Problem Determination and Service Guide

177 Table 28. Symptom-to-FRU index (continued) Problem indicator Component Possible cause Possible solutions Amber LED is flashing Drive CRUs (Fault LED lit) ESM port bypass ESM SFP port bypass LED Front panel (Global Summary Fault LED) Drive identity is in process A Fibre Channel loop initialization process (LIP) is being generated in the drive loop due to a faulty component. The enclosure speed was set to 4 Gbps but the SFP inserted in the ESM SFP port does not operate at 4 Gbps. SFPs are inserted into unused ESM ports labeled 2A and 2B. One or multiple CRUs has intermittent problems. No corrective action needed. Use the Read Link Status window in the Storage Manager Subsystem Management window and the storage subsystem event logs to isolate the faulty component. Use the Recovery Guru menu function in the Storage Manager Subsystem Management window to verify the problem and replace the SFP with a 4 Gbps SFP. Verify the port connections in the back of the disk expansion module and make corrections (using port 1A and 1B) if required. Use the Recovery Guru menu function in the Storage Manager Subsystem Management window and the disk controller storage subsystem Major Event Log to troubleshoot the problem. Chapter 7. Disk expansion units 153

178 Table 28. Symptom-to-FRU index (continued) Problem indicator Component Possible cause Possible solutions One or more green LEDs are off Power supply CRUs All drive CRUs Power cable is unplugged or switches are turned off The disk expansion module is connected to an unsupported disk component instead of a disk controller. The enclosure speed is not the same as other disk components in the same redundant drive channel pair. Incorrect SFPs are used such as a 2 Gbps SFP used when the disk expansion module speed is set to 4 Gbps. The disk expansion module is not cabled or incorrectly cabled to other disk expansion modules or the disk controller. For example, ESM ports labeled 2A and 2B were used instead of ports labeled 1A and 1B. Drive CRUs are not fully inserted in the drive slots. Midplane failure Make sure that the power cable is plugged in and that the switches are turned on. Verify that the disk expansion module is connected to a disk controller. Verify that all of the disk expansion module and disk controller speeds are the same. Verify and replace the SFPs with the correct speed SFPs. Verify and correct the incorrect cabling if required. Verify that the drives are seated correctly in the drive slots. Remove and reinsert the drive CRUs if required. Replace the midplane. Contact an IBM technical support representative. 154 TS7650 and TS7650G Problem Determination and Service Guide

179 Table 28. Symptom-to-FRU index (continued) Problem indicator Component Possible cause Possible solutions One or more green LEDs are off (continued) Green LED slowly flashing one per 2 seconds Intermittent or sporadic power loss to the disk expansion module Several CRUs Hardware failure Replace the affected CRUs. If this does not correct the problem, replace the ESMs, followed by the midplane. Contact an IBM technical support representative. Front panel Drive CRUs Some or all CRUs The disk controller was not turned on, or all Fibre Channel connections between the disk expansion module and the disk controller failed. Power supply problem Hardware failure The disk controller was not turned on, or all Fibre Channel connections between the disk controller and the disk expansion modules failed. Defective AC power source or improperly connected power cable Power supply failure Midplane failure Perform one of the following actions: v Power on the disk component. v Verify that the Fibre Channel connections between the disk expansion module and the disk controller are made. Make sure that the power cables are plugged in and that the power supplies are turned on. If any other LEDs are lit, replace the midplane. Contact an IBM technical support representative. Perform one of the following actions: v Power on the disk component. v Verify that the Fibre Channel connections between the disk controller and all disk expansion modules are made. v Ensure that all the disk components in the same redundant drive loop/channel pair have the same speed setting. Check the AC power source. Reseat all installed power cables and power supplies. If applicable, check the power components (power units or universal power supply). Replace defective power cables. Check the power supply Fault LED on the power supply. If the LED is lit, replace the failed CRU. Have the midplane replaced. Contact an IBM technical support representative. Chapter 7. Disk expansion units 155

180 Table 28. Symptom-to-FRU index (continued) Problem indicator Component Possible cause Possible solutions Unable to access drives Drives and Fibre Channel loop Incorrect disk expansion module ID settings ESM failure Drive or drives failed Drive is not certified Ensure that the Fibre Channel optical cables are undamaged and properly connected. Check the disk expansion module ID settings. Note: Change the switch position only when your disk expansion module is powered off. Replace one or both ESMs. Contact an IBM technical support representative. Replace failed drive or drives. Check the TS7650G product website for the correct drive option P/N or FRU P/N certified for your storage subsystem. Random errors Subsystem Midplane failure Have the midplane replaced. Contact an IBM technical support representative. Hard disk drive not visible in RAID management software Several CRUs Drive or drives failed Fibre Channel cable failed SFP failed ESM failure Midplane failure Incorrect firmware version Replace failed drive or drives. Replace Fibre Channel cable. Replace SFP. Replace ESM. Have the midplane replaced. Contact an IBM technical support representative. Verify that the storage subsystem has the correct firmware version. 156 TS7650 and TS7650G Problem Determination and Service Guide

181 Operating the disk expansion unit To ensure optimal operation of your disk expansion unit, always follow these guidelines. v To maintain power redundancy, plug the disk expansion unit right and left power supplies into two independent external power circuits through ac distribution units inside a rack cabinet or directly into external receptacles. Similarly, the right and left power supplies of the expansion frames attached to the disk expansion unit should be plugged into the same two independent external power circuits as the disk expansion unit. v Ensure that the disk expansion unit is in an optimal state before you shut it down. Never turn the power off if any fault light is lit. Resolve any error conditions before you shut down the system. v During any maintenance or attended power on procedure, check that each component of the disk expansion unit is turned on in the proper order during this entire power on procedure to ensure that the controller will optimally access all of your disk components. v The TS7650 Appliance and TS7650G supports simultaneous power on to the system components. v A disk expansion unit in an optimal state recovers automatically from an unexpected shutdown and unattended simultaneous restoration of power to system components. After power is restored, call IBM support if any of the following conditions occur: The disk component logical drives and arrays are not displayed in Storage Manager. The disk component logical drives and arrays do not come online. The disk component logical drives and arrays seem to be degraded. Disk expansion unit speed setting The default switch setting is 4 Gbps. This is the only speed setting allowed for the disk expansion unit. Attention: Do not remove the metal plate that covers the disk expansion unit speed setting. Do not adjust the disk expansion unit speed setting. Chapter 7. Disk expansion units 157

182 158 TS7650 and TS7650G Problem Determination and Service Guide

183 Chapter 8. CD and DVD overview Documentation CD The topics in this section provide information about the documentation and software CDs and DVDs of the TS7650 Appliance and the TS7650G Gateway ProtecTIER software. The Red Hat Linux and ProtecTIER code are loaded by manufacturing. The IBM System Storage TS7650 with ProtecTIER Publications CD contains documentation for the TS7650 Appliance and the TS7650G Gateway. See Hardware ship group CDs on page 1 for more information. RAS/BIOS and Firmware update DVD Disk component firmware The RAS/BIOS and Firmware DVD is no longer included in the ship group and does not apply for ProtecTIER R2.4 and above. It is available as a FRU for those customers who choose not to move to the R2.4 software. In all TS7650 Appliance installations and in TS7650G Gateway IBM best practices installations with the H disk controller and H disk expansion modules installed, the disk component firmware CD contains a script file that establishes the file system on the installed disk components AP1 installations with 3958 DD4 servers feature disk storage attachments of the new H 3950 disk controller and H disk expansion units. Disk component firmware for these installations is provided by ProtecTIER Appliance Edition software V2.5 DVD. Following a reload of the Recovery Disk, both the ProtecTIER CD and the IBM System Storage RAS/BIOS and Firmware Update DVD need to be reloaded. Note: If the reload is of R2.4 code or above, the RAS/BIOS and Firmware update is no longer available. These functions are included in the ProtecTIER software. Recovery disk There are multiple levels of Recovery Disks. Make sure the Recovery Disk matches the level of ProtecTIER software you are recovering. Software CDs This topic addresses the CDs included in the software ship group. For an overview of the CDs included in the software ship group, refer to Software ship group DVDs on page 2. Copyright IBM Corp. 2011,

184 160 TS7650 and TS7650G Problem Determination and Service Guide

185 Chapter 9. Parts catalog The topics in this section provide information about the field-replaceable units unique to each ProtecTIER server. Field-replaceable units unique to the 3958 DD4, 3958 DD5, and 3958 AP1 servers This topic lists the field-replaceable units (FRUs) that are unique to the 3958 DD4, 3958 DD5, and 3958 AP1 servers. CAUTION: The Ethernet and Fibre Channel adapter FRUs in server peripheral component interconnect (PCI) slots 1 through 7 are not hot-swap capable because the server's operating system does not permit the adapters to be varied offline prior to removal. Request that the customer shut down the server before attempting to replace any of these adapters. Use Table 29 and Table 30 on page 162 to order FRUs for the 3958 DD4, DD5, and AP1 servers. Table DD4, DD5, and AP1 server FRUs IBM part number FRU 59Y4815 Top cover (all models) 46M0000 PWR 3.0 m ROHS Power supply filler (see Filler Kit) 49Y7760 Power supply (models 1Rx 2Rx 2Sx 3Rx 4Dx 4Rx 4Sx 5Dx 5Rx 5Sx ARx) 42D0500 Emulex 8 Gb dual-port FC adapter, Model Lpe D Gb QLogic FC dual-port HBA adapter 46M0003 I/O board shuttle (all models) 59Y4814 Chassis assembly (models 1Rx, 2Rx, 2Sx, 3Rx, 4Dx, 4Rx, 4Sx, 5Dx, 5Rx, 5Sx, ARx) Hard disk drive backplane filler (see Filler Kit) Hard disk drive backplane carrier (see Miscellaneous hardware parts kit) Hard disk drive backplane power cables and carrier (see Cabling Kit) RAID card carrier (see Miscellaneous hardware parts kit) 44E8690 ServeRAID-BR10i SAS/SATA Controller (all models except 5Dx) 46M0851 ServeRAID-M5015 SAS/SATA Controller (optional) 81Y4451 ServerRAID-M5015 BBU Battery Backup Unit ( M5000 battery ) 49Y2004 Hard disk drive, 600 GB 10K, 2.5-inch SAS (optional), for DD5 42D0638 Hard disk drive, 300 GB 10K, 2.5-inch SAS (optional) 42D0673 Hard disk drive, 73 GB 15K, 2.5-inch SAS (optional) 42D0678 Hard disk drive, 146 GB 15K, 2.5-inch SAS (optional) 42D0633 Hard disk drive, 146 GB (optional) Copyright IBM Corp. 2011,

186 Table DD4, DD5, and AP1 server FRUs (continued) IBM part number FRU 46M Y W W Y Y M T T C C C Y Y Y Y Y Y Y Y Y Y Y6227 Hard disk drive filler (see Filler Kit) Operator information panel (all models) Front bezel (all models) DVD/CD-RW SATA drive (some models only) DVD/CD multi-burner SATA drive (some models only) Battery tray (see Miscellaneous hardware parts kit) Front fans (120 mm) (all models) Memory card cage (included in chassis assembly) Middle fans (dual 60 mm) (all models) Memory card filler (see Filler Kit) Memory card (models ARx (2 per) 1Rx 2Sx 2Rx 3Rx 4Dx 4Rx 4Sx 5Dx 5Rx 5Sx) DIMM, 1 GB (optional) DIMM, 2 GB (models 4Rx, ARx) DIMM, 4 GB (models 1Rx, 2Rx, 2Sx, 3Rx, 4Dx, 4Rx, 4Sx, 5Dx, 5Rx, 5Sx) DIMM, 8 GB (optional) DIMM, 16 GB (optional) Microprocessor board (for DD4 models) Microprocessor board (for DD5 server) Microprocessor (1.86 GHz, 18M 4-core), insertion tool, and heat sink (models 1Rx, ARx) Microprocessor (1.86 GHz, 12M, 6-core), insertion tool, and heat sink (models 2Rx, 2Sx) Microprocessor (1.86 GHz, 18M, 6-core), insertion tool, and heat sink (optional) Microprocessor (1.86 GHz, 24M, (8-core), insertion tool, and heat sink (optional) Microprocessor (2.00 GHz, 18M, 6-core), insertion tool, and heat sink (model 3Rx) Microprocessor (2.00 GHz, 18M, 8-core), insertion tool, and heat sink (models 4Dx, 4Rx, 4Sx) Microprocessor (2.26 GHz, 24M, 8-core), insertion tool, and heat sink (models 4Rx, 5Dx, 5Rx, 5Sx) Microprocessor (Intel Xeon E GHz, 30M, 10-core), (model B7x) for DD5 Microprocessor (2.67 GHz 6-core), insertion tool, and heat sink (optional) Table 30. Additional FRUs for feature code 3447, Cluster Connection Kit IBM part number FRU 46M2177 Ethernet Switch 1, SMC8126L2 26-Port 1Gb Switch 46M2177 Ethernet Switch 2, SMC8126L2 26-Port 1Gb Switch 162 TS7650 and TS7650G Problem Determination and Service Guide

187 Table 30. Additional FRUs for feature code 3447, Cluster Connection Kit (continued) IBM part number FRU 22R V dc jump I/O 23R m C13/C19 power cord 46X8454 Western Telematic Inc. (WTI) Network Managed Power Switch, Model VMR-8 95P m Ethernet cable 39M5377 ac Power cord 2 45E m Ethernet cable 1 45E m power cord 45E m power cord For FRUs not listed in Table 29 on page 161 and Table 30 on page 162, see the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146: Problem Determination and Service Guide. Field-replaceable units unique to the 3958 DD3 server This topic lists the field-replaceable units (FRUs) that are unique to the 3958 DD3 server. CAUTION: The Ethernet and Fibre Channel adapter FRUs in server peripheral component interconnect (PCI) slots 1 through 7 are not hot-swap capable because the server's operating system does not permit the adapters to be varied offline prior to removal. Request that the customer shut down the server before attempting to replace any of these adapters. Use Table 31 and Table 32 on page 164 to order FRUs for the 3958 DD3 server. Table DD3 server FRUs IBM part number FRU 15F8409 Battery system planar board 23R3245 PWR 3.0 m ROHS 26K8680 Hard disk drive (HDD) filler 39M5377 Alternate current (AC) power cord 39R6528 Qlogic 4 Gbps 2-port Fibre Channel host bus adapter (HBA) 39Y6128 Dual-port Ethernet card 39Y watt H/S power supply 41Y GB Memory dual inline memory module (DIMM) 42D3062 Slide kit 43W4283 Server RAID-MR10k battery 43W7512 Emulex 4 Gbps 2-port Fibre Channel HBA 43W8673 Kodiak power backplane 43W9559 Central processing unit (CPU) heat sink 43W mm cooling fan Chapter 9. Parts catalog 163

188 Table DD3 server FRUs (continued) IBM part number FRU 43X GB 10,000 RPM serial attached SCSI (SAS) HDD 44E4372 Operator panel 44E4373 Raven high speed (HS) PCI switch 44E4483 Dunnington 2.67 GHz 6C 16 M processor 44E4485 I/O planar 44E4488 Central processing unit (CPU) planar 44E4499 Label SAN volume controller (SVC) kit 44E4529 DVD housing 44E cable kit 44E4553 Voltage regulator module (VRM) processor 44E mm cooling fan 44E4564 System x3850 M2 bezel 44E4566 CMA cable management arm 44E4570 Rack latch kit 44E4572 Top cover 44E4575 Memory bulkhead 44E4576 Fan cage 44E4581 PCI retention bracket 44E4582 Rear shuttle 44E4587 Miscellaneous hardware kit 44T1413 Raptor remote supervisor adapter (RSA) 2 44W2728 Serial attached small computer system interconnect (SCSI) (SAS) backplane 44W3255 Serial advanced technology attachment (SATA) CD-RW/DVD combination drive 44W4314 Front scalable bezel 44W4321 Miscellaneous plastic parts kit 46M0827 Ocelot L1 memory expansion card 46M3510 System chassis 46M3511 Hood assembly Table 32. Additional FRUs for feature code 3447, Cluster Connection Kit with Alpha Networks switch IBM part number FRU 22R4916 Alpha Networks Ethernet switch 22R V dc jump I/O 23R m C13/C19 power cord 45E5808 Western Telematic Inc. (WTI) 1 Gbps network power switch 95P m Ethernet cable 164 TS7650 and TS7650G Problem Determination and Service Guide

189 Table 33. Additional FRUs for feature code 3437, Cluster Connection Kit with SMC switch IBM part number FRU 46M2177 Ethernet Switch 1, SMC8126L2 26-Port 1Gb Switch 46M2177 Ethernet Switch 2, SMC8126L2 26-Port 1Gb Switch 23R m C13/C19 power cord 46X8454 Western Telematic Inc. (WTI) Network Managed Power Switch, Model VMR-8 95P m Ethernet cable For FRUs not listed in Table 31 on page 163 and Table 32 on page 164, see the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. Field-replaceable units unique to the 3958 DD1 server This topic lists the field-replaceable units (FRUs) that are unique to the 3958 DD1 server. CAUTION: The Ethernet and Fibre Channel adapter FRUs in server PCI slots 1 through 7 are not hot-swap capable because the server operating system does not permit the adapters to be varied offline prior to removal. Request that the customer shut down the server before attempting to replace any of these adapters or FRUs unless listed as a concurrent replacement. Use Table 34 and Table 35 on page 166 to order FRUs for the 3958 DD1 server. Table DD1 server FRUs IBM part number FRU 95P ft Ethernet cable 23R m cable 44E4555 Tigerton 2.93 GHz/8M processor 39Y port Ethernet card 95P m Ethernet cable 41Y GB DIMM 43W7512 Emulex 4 Gbps 2-port Fibre Channel HBA 15F8409 Battery System planar board 44E4564 System x3850 M2 bezel 44E4357 Archimides 2.5 inch HDD backplane 43W8673 Kodiak power backplane 44E cable kit 44E4566 Enterprise cable management arm 40K6556 Scalable cable management arm 39M5377 AC power cord 44E4573 System chassis Chapter 9. Parts catalog 165

190 Table DD1 server FRUs (continued) IBM part number FRU 43W9559 CPU heat sink 44E4552 Mica - DVD housing with interposer card 43W4603 DVD/CD combination drive 46M0827 L1 MI RAID 1 primary array, SAS 43W4283 Server RAID-MR10k battery 22R4916 Ethernet switch 44E mm cooling fan 43W mm cooling fan 44E4576 Fan cage 26K8680 Filler 44W4314 Front scalable bezel 44E4574 Hood assembly 44E4567 Label kit 46M0827 L1 memory card 44E4575 Memory bulkhead 44E4587 Miscellaneous hardware kit 44W4321 Miscellaneous plastic parts kit 44E4372 Operator panel 44E4373 Raven HS PCI switch 44E4581 PCI retention bracket 43W8670 Panther front system board 43W8671 Cougar PCIe board 39Y watt H/S power supply 23R3245 PWR 3.0 meter ROHS 39R6528 Qlogic 4 Gbps dual-port Fibre Channel HBA 44E4570 Rack latch kit 44E4582 Rear shuttle 43X GB 10,000 RPM SAS HDD 44W2728 SAS backplane 42D3062 Slide kit 46M5968 Peregrine service processor 44E4572 Top cover 44E4553 VRM processor Table 35. Additional FRUs for feature code 3447, Cluster Connection Kit with Alpha Networks switch IBM part number FRU 22R4916 Alpha Networks Ethernet switch 22R V dc jump I/O 166 TS7650 and TS7650G Problem Determination and Service Guide

191 Table 35. Additional FRUs for feature code 3447, Cluster Connection Kit with Alpha Networks switch (continued) IBM part number FRU 23R m C13/C19 power cord 45E5808 Western Telematic Inc. (WTI) 1 Gbps network power switch 95P m Ethernet cable For FRUs not listed in Table 34 on page 165 and Table 35 on page 166, see the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. Chapter 9. Parts catalog 167

192 168 TS7650 and TS7650G Problem Determination and Service Guide

193 Chapter 10. FRU replacement for TS7600 systems This section discusses field replacement procedures for systems with ProtecTIER software V2.3 or earlier and for V2.4 or later. The replacement procedures are common, but the firmware update commands are different. v For ProtecTIER software V2.3 or earlier, use the install commands on the IBM System Storage RAS/BIOS and Firmware Update DVD. v For ProtecTIER software V2.4 or V2.5, use the following command: /opt/dtc/install/ptconfig -updatefirmwares v For ProtecTIER V3.1 or later, select ProtecTIER Configuration > Update Firmware from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16). The TS7650 Appliance and TS7650G Gateway use standard field-replaceable units (FRUs) and some unique FRUs. To replace a FRU, refer to the service guide that shipped with the component in which you are installing the FRU. Some FRUs are hot-swap capable and can be hot-swapped if the service guide instructs you to do so (for example, power supplies). For FRUs that are not hot-swap capable (for example, Fibre Channel adapters and the RSA), you might have to turn off the component. Note: Update the server base firmware and run either the /opt/dtc/install/ ptconfig -updatefirmwares command or the IBM System Storage RAS/BIOS and Firmware Update DVD V or later for each FRU (except as noted below) after any of the following maintenance actions: v Server BIOS update v RSA replacement v RSA firmware update v QLogic Fibre Channel adapter replacement v Emulex Fibre Channel adapter replacement v Ethernet adapter replacement v System planar replacement v MegaRAID (MR10k) adapter card replacement v Replacement of RAID controller with M5015 ServeRAID adapter (in 3958 DD4, DD5, and AP1 servers) Use one the following Problem Determination Guides for removing or installing a specific hot-swap component for any additional procedures that you might have to perform before you remove or install the component. v IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide v IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide Note: Adapters in the TS7650 (including the hot-swap adapter ports) are not hot-swap capable due to the limitations of the Linux operating system. The system must be shut down when an adapter needs to be replaced. Copyright IBM Corp. 2011,

194 The IBM System Storage RAS/BIOS and Firmware Update DVD contains the following: v Files for updating firmware after a FRU has been replaced v Files for enabling Call Home v A script to configure the server E2 port for connection to the system console network v FRU BIOS and firmware levels For systems running PT software 2.3 and below, see the IBM System Storage TS7600 with ProtecTIER Installation Instructions for the RAS Package, BIOS, and Firmware updates following a FRU replacement for models 3958 DD1, 3958 DD3, and 3958 AP1, PN 46X2459 and Updating the server firmware and basic input/output system settings on page 228 for more information. You must follow the directions in this document on how to update the firmware and to power down the TS7650 Appliance and TS7650G Gateway. After replacing a Fibre Channel HBA or Ethernet adapter, additional steps might be required in order to complete the replacement. The Fibre Channel HBA might require ports set to target or initiator mode. To set the IP address on the Ethernet adapter, see the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA Removing, replacing, and upgrading FRUs in the ProtecTIER server The topics in this section explain how to remove, replace, and upgrade field-replaceable units (FRUs) in a ProtecTIER server. Attention: Review this procedure before performing it. The hot-swappable FRUs are: v Power supplies and fans v Internal serial attached SCSI (SAS) HDD v SMC switch Field replaceable units include the system planar, MR10k RAID adapter, Ethernet adapters, and SAS disk drives. Often a FRU does not have the current tested level of BIOS or firmware. New FRUs need to have their BIOS or firmware levels set to the tested level for assured reliability. Perform the procedures for the FRU that needs replacement and the BIOS or firmware level set. Important: For the internal SAS drives on the server, when a disk drive failure occurs on one of the drives, and a replacement is needed, after a physical replacement is complete, the mirroring process on the remaining drive restores the configuration to the new drive. The rebuild process takes several hours to complete. Complete this drive replacement process concurrently while the system is up and operational. This replacement process relates to either disk drive in bay 0 or 1. For a failure on both drives, contact your next level of support for recovery. The replacement drive must be either new or newly formatted. Preparing the system for FRU replacement Complete the task in this topic to prepare the ProtecTIER server for replacement of a field-replaceable unit (FRU). 170 TS7650 and TS7650G Problem Determination and Service Guide

195 Before you begin Attention: The Ethernet and Fibre Channel adapter FRUs in server peripheral component interface (PCI) slots 1 through 7 are not hot-swap capable because the server's operating system does not permit the adapters to be varied offline prior to removal. You must stop all jobs and turn off the system and the server before attempting to replace any of these adapters. Procedure 1. Have the customer stop all activities from attached hosts and quiesce all jobs. 2. Enable Service Mode on the server: a. From a command line on Node A, access the ProtecTIER Service Menu by entering menu. b. From the ProtecTIER Service Menu, select Health Monitoring > Service Mode. c. Type yes and press Enter to enable Service Mode. d. Repeat steps 2a thru 2c on Node B. 3. Stop the VTFD and cluster services: a. From a command line on Node B, type the following commands: service vtfd stop service ptcluster stop b. Repeat step 3a on Node A. 4. Shut down the node on which the FRU must be replaced: a. Right-click the system console desktop and click Terminal. b. At the Terminal command prompt, enter the command tsys and select the system to which you want to connect. Note: Alternatively, you can type the following command: ssh ptadmin@xxx.xxx.xxx.xxx where xxx.xxx.xxx.xxx is the server IP address c. At the password prompt, type ptadmin. d. At the command prompt, type poweroff. Replacing the field-replaceable unit Complete the task in this topic to replace a field-replaceable unit (FRU) in the ProtecTIER server. Before you begin CAUTION: The Ethernet and Fibre Channel adapter FRUs in server peripheral component interface (PCI) slots 1 through 7 are not hot-swap capable because the server's operating system does not permit the adapters to be varied offline prior to removal. Before attempting to replace any of these adapters, request that the customer stop all jobs and turn off the system and the server (see Preparing the system for FRU replacement on page 170). See the appropriate FRU replacement section in this document for each PCI adapter. Chapter 10. FRU replacement for TS7600 systems 171

196 Procedure 1. Identify the location of the FRU to be replaced. Important: v Use the lists of FRUs in Chapter 9, Parts catalog, on page 161 to find FRU part numbers for most components in those servers. v When replacing PCI Express adapters, pay particular attention to the adapter slots. The slots identified with the orange tabs are hot-swap capable slots. The other slots require a complete system shutdown in order to replace the adapters. Refer to Chapter 9, Parts catalog, on page 161 for important information regarding the adapter slots. The adapters in the ProtecTIER server are not hot-swap capable due to the Linux operating system limitations. The system must be powered down for replacement. 2. After identifying the location, determine whether the FRU is secured by retaining screws. If the screws are in place, remove them. Working from the front of the frame, remove the retaining screws from the server front panel and set them aside. 3. For FRU replacement procedures, refer to the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide (or, for the 3958 DD3 and 3958 DD1 servers, IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide). Important: If the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide replacement procedures ask for the server to be turned off, insert the IBM System Storage RAS/BIOS and Firmware Update DVD (for R2.3 only) or the R2.4 TS7650 or TS7650G DVD into the server DVD drive and follow the steps in Powering off the ProtecTIER server version 2.5 or earlier on page 243 and Powering on the ProtecTIER server version 2.5 or earlier on page 245 to cycle power to the server. Note: No DVD is required for R2.5 or later TS7650 or TS7650G. The IBM System Storage RAS/BIOS and Firmware Update DVD (for R2.3 only) or the R2.4 TS7650 or TS7650G DVD is used to restore the server BIOS and firmware settings. No DVD is needed for V2.5 or later. The DVD is used to restore the BIOS and firmware settings after replacing any of the following FRUs: v System planar v Fibre Channel host bus adapters (HBAs) v M5015 ServeRAID adapter ( MR10k MegaRAID adapter in 3958 DD3 and 3958 DD1 servers) v Ethernet adapter v RSA Card For R2.3 or earlier: Refer to the IBM System Storage TS7600 with ProtecTIER Installation Instructions for the RAS Package, BIOS, and Firmware updates following a FRU replacement for models 3958 DD1, 3958 DD3, and 3958 AP1, PN 46X2459 (included with the ship group) for instructions for using this DVD. 4. Replace the FRU following the steps in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. For 3958 DD3 and 3958 DD1 servers, perform steps in the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. 5. After replacing the adapter, you must update the adapter firmware (see Updating the firmware of an Emulex or Qlogic adapter on page 174). 172 TS7650 and TS7650G Problem Determination and Service Guide

197 6. After any firmware level has been updated and power has been cycled, check the versions. v For ProtecTIER software V3.1 or later, from the ProtecTIER Service menu, select Version Information > Display Firmware Levels (see ProtecTIER Service menu on page 16). v For ProtecTIER software V2.4 or V2.5, from the ProtecTIER Service menu, select Display Firmware Versions. v For ProtecTIER software V2.3 or earlier, from a command line type versions. If the command is not found, type the following commands to mount the IBM System Storage RAS/BIOS and Firmware Update DVD and display the firmware versions: mount /dev/hda /mnt/cdrom cd /mnt/cdrom./versions The output for Reliability, Availability, and Serviceability (RAS) package version 3.2 or later looks similar to the example shown in Figure 63. Resource ID Version Description DiskArray Disk Array Controller 1 Controller AC1 78GABHN C35D B A7C AFB6 30C2 DAC MAIN A3EP45A 18 IBM Remote Superviosor Adapter ISMP A3BT55A 4.07 IBM BaseBoard Managment Controlle BIOS A3E170AUS 16 BIOS Version DSA DSYT89EUS IBM Preeboot Diagnostics FPGA IBM Processor Card FPGA FPGA IBM PCI I/O Planar FPGA Fibre Channel Emulex Emulex Corporation Zephyr-X Light Fibre Channel Emulex Emulex Corporation Zephyr-X Light Fibre Channel Emulex Emulex Corporation Zephyr-X Light Fibre Channel Emulex Emulex Corporation Zephyr-X Light Fibre Channel QLE QLE2462 PCI Express to 4Gb FC Dua Fibre Channel QLE QLE2462 PCI Express to 4Gb FC Dua Fibre Channel QLE QLE2462 PCI Express to 4Gb FC Dua Fibre Channel QLE QLE2462 PCI Express to 4Gb FC Dua Raid IBM ServeRAID-MR10k IBM ServeRAID-MR10k SAS/SATA Cont Ethernet e1000e Intel Corporation 82571EB Gigabit Ethernet e1000e Intel Corporation 82571EB Gigabit Ethernet e1000e Intel Corporation 82571EB Gigabit Ethernet e1000e Intel Corporation 82571EB Gigabit Ethernet bnx Broadcom Corporation NetXtreme II Ethernet bnx Broadcom Corporation NetXtreme II Disk Drive SAS B52B Slot 1 Disk Drive SAS SB07 Slot 2 Figure 63. Example output of versions check 7. Make a note of the displayed levels and compare them to the levels listed in this document. 8. If any firmware displayed is lower than the recommended level, follow the instructions to update that component (using RAS package version 3.2 or later or the /opt/dtc/install/ptconfig -updatefirmwares command). If the firmware level is up to date, you can skip the firmware update. Results What to do next If the remote supervisor adapter (RSA) field replaceable unit (FRU) was replaced, continue to Configuring the RSA card on page 204 to set the IP addresses and Chapter 10. FRU replacement for TS7600 systems 173

198 update the Reliability, Availability, and Serviceability (RAS) code, BIOS and firmware. Updating the firmware of an Emulex or Qlogic adapter This procedure is performed after the replacement of an Emulex or QLogic adapter to ensure that the firmware levels remain consistent across all the adapters. It requires a restart with the CD selected as the boot device. You can then select the firmware update needed from the menu provided. Procedure 1. Connect a USB keyboard and monitor to the 3958 server or use the TSSC to switch to the node you are working on. 2. Insert the appropriate software DVD. ProtecTIER software version DVD 2.3 IBM System Storage RAS/BIOS and Firmware Update DVD (if not already inserted) 2.4 ProtecTIER 2.4 DVD (if not already inserted) 2.5 or later No DVD needed 3. ProtecTIER software V2.3 only: If you cannot boot from the IBM System Storage RAS/BIOS and Firmware Update DVD (the system displays an Abort, Retry, Fail? error), remove the DVD and insert the TS7600 HBA Firmware CD (PN 46X5505, EC M11251). Boot from this CD. (No DVD is needed for V2.5 or later.) 4. Power on the node. ProtecTIER software V2.3 and V2.4 only The CD/DVD should be the default boot device. If it is not, when the IBM logo and the function key options are displayed, press F12 to select the CD/DVD drive as the boot device. The IBM TS7600 BIOS and Firmware tool menu is displayed. Skip to step 5. ProtecTIER software V2.5 or later Wait until the node has completed power-on. From the command line, type the following command to update the firmware: /opt/dtc/install/ptconfig -updatefirmwares After the firmware update is complete, type the command reboot. Skip to step 7 on page 175. ProtecTIER software V3.1 or later Wait until the node has completed power-on. To update the firmware, go to the ProtecTIER Service menu (see ProtecTIER Service menu on page 16) and, select ProtecTIER Configuration > Update Firmware. After the firmware update is complete, type the command reboot. Skip to step 7 on page Select the option to Update the Emulex Firmware or Update the Qlogic Firmware, depending on which Host Bus Adapter (HBA) card you replaced: 174 TS7650 and TS7650G Problem Determination and Service Guide

199 IBM TS7600 BIOS and Firmware tool Select from the following options: [1] Update the Qlogic Firmware [2] Update the Emulex Firmware [3] Run Intel Boot Agent Utility [4] Run all updates [X] Exit to command prompt [0] Reboot Note: The Reboot option restarts the server and ejects the CD-ROM. The firmware is updated. A message states that the firmware update completed successfully. 6. Select [0] to reboot. The system completes the boot cycle. 7. Verify that no errors are reported on screen or from the error indicators on the light path display panel. Note: After the reboot completes, the following message might be presented at the login prompt: The module sctp cannot be unloaded due to unsafe usage... Ignore this message and press the enter key to return to the login prompt. Results The firmware update procedure for the Emulex or Qlogic adapter is complete. What to do next v If a customer uses a fibre switch to connect the TS7650 to the cache, warn the customer that replacing a Qlogic adapter creates a different WWNN from the one that is currently used. If the customer is zoning the switch, they must rezone it with the new Qlogic WWNN (see Obtaining WWNN and WWPN values for a Qlogic adapter on page 177). v If the customer has configured LUN Mapping on the backend disk system the LUN Mapping must be reconfigured to the new WWNN. v If your system uses ProtecTIER V2.4 or earlier, you might need to manually reset the World Wide Node Name (WWNN) after updating the firmware. For instructions, see Changing the World Wide Node Name on an Emulex adapter. v If your system uses ProtecTIER V2.5 or later, the WWNN for the new Emulex adapter is automatically reset to its original value by the firmware update and reboot in step 4 on page 174. Changing the World Wide Node Name on an Emulex adapter If the system uses an Emulex adapter and is running ProtecTIER V2.4 or earlier, perform this task to change the World Wide Node Name (WWNN). About this task Note: This task cannot be performed on a system running ProtecTIER V2.5 or later. Chapter 10. FRU replacement for TS7600 systems 175

200 Procedure To change the WWNN, perform the following steps. 1. Stop vtfd services by entering the following command: service vtfd stop 2. To access the WWNN main menu, type the following command: /opt/dtc/app/utils/wwnutil The Main Menu is displayed (Figure 64) MAIN MENU [1] Show available FE ports names [2] Edit WWN by port instance [3] View Changes [4] Submit or discard changes [x] Exit Figure 64. WWNN main menu 3. Select option 1 to show the available FE port names. The Display Names By Port menu is displayed (Figure 65) DISPLAY NAMES BY PORT [port] original nodename original portname ====== ================= ================= [ 0] c97e c97e3434 [ 1] c97e c97e3435 [ 2] c97e2dc c97e2dc8 [ 3] c97e2dc c97e2dc9 [m] back to main menu Figure 65. Display Names By Port menu 4. Select option m to return to the main menu. 5. Select option 2 to edit the WWNN by port instance. 6. Enter the port number. In this example, the port entered is 0 as listed above in the Display Names By Port menu. The Change Name By Port menu is displayed (Figure 66) CHANGE NAME BY PORT [port] original nodename original portname ====== ================= ================= [ 0] c97e c97e Figure 66. Change Name By Port menu 7. Enter the WWN node name: c97e Enter the WWN port name: c97e3434. The updated values are displayed (Figure 67 on page 177). 176 TS7650 and TS7650G Problem Determination and Service Guide

201 CHANGE NAME BY PORT [port] original nodename original portname ====== ================= ================= [ 0] c97e c97e3434 [port] new nodename new portname ====== ============ ============ [ 0] c97e c97e3434 [m] back to main menu [r] repeat Figure 67. Change Name By Port menu (updated) 9. Enter option m to return to the main menu. 10. Enter option x to exit. What to do next If you are performing any other FRU or firmware updates, proceed to the appropriate section. Otherwise, go to Chapter 12, End-of-call procedure, on page 249. Obtaining WWNN and WWPN values for a Qlogic adapter After replacing a Qlogic adapter, you might have to provide the new WWNN and WWPN values to the customer. To obtain the new values, you must use the command line interface of the Qlogic adapter. Procedure 1. At the command line, type the following command: /opt/qlogic_corporation/sansurfercli/scli The Main Menu for the Qlogic adapter is displayed (Figure 68). SANsurfer FC/CNA HBA CLI v1.7.3 Build 32 Main Menu 1: General Information 2: HBA Information 3: HBA Parameters 4: Target/LUN List 5: iidma Settings 6: Boot Device 7: Utilities 8: Beacon 9: Diagnostics 10: Statistics 11: Virtual 12: Help 13: Exit Enter Selection: Figure 68. Main Menu for Qlogic adapter command line interface 2. Select option 2 (HBA Information). The HBA Information Menu is displayed, showing the WWPN values of the ports (Figure 69 on page 178). Chapter 10. FRU replacement for TS7600 systems 177

202 SANsurfer FC/CNA HBA CLI v1.7.3 Build 32 HBA Information Menu HBA Model QLE2462 1: Port 1: WWPN: B-32-1B-DE-75 Online 2: Port 2: WWPN: B-32-3B-DE-75 Online HBA Model QLE2462 3: Port 1: WWPN: B ED-7C Online 4: Port 2: WWPN: B-32-A3-ED-7C Online 5: All HBAs 6: Return to Previous Menu Note: 0 to return to Main Menu Enter Selection: Figure 69. HBA Information Menu 3. Record the WWPN values for the new ports. 4. To obtain the WWNN value of a port, type the number to select it from the HBA Information Menu. An HBA Instance submenu for that port is displayed (Figure 70). SANsurfer FC/CNA HBA CLI v1.7.3 Build 32 HBA Information Menu HBA Instance 2 (QLE2462 Port 1) : Online WWPN: B ED-7C Desc: QLE2462 PCI Express to 4Gb FC Dual Channel 1: Information 2: HBA Alias 3: HBA Port Alias 4: VPD 5: Return to Previous Menu Note: 0 to return to Main Menu Enter Selection: Figure 70. HBA Instance submenu 5. Select option 1 (Information). The information for the selected port is displayed (Figure 71 on page 179). 178 TS7650 and TS7650G Problem Determination and Service Guide

203 Host Name : raspool_node2 HBA Instance : 2 HBA Model : QLE2462 HBA Description : QLE2462 PCI Express to 4Gb FC Dual Channel HBA ID : 2-QLE2462 HBA Alias : HBA Port : 1 Port Alias : Node Name : B ED-7C Port Name : B ED-7C Port ID : EF Serial Number : RFC0836C28589 Driver Version : k BIOS Version : 2.10 Driver Firmware Version : (486) Flash BIOS Version : 2.10 Flash FCode Version : 2.04 Flash EFI Version : 2.04 Flash Firmware Version : Actual Connection Mode : Loop Actual Data Rate : 4 Gbps PortType (Topology) : NLPort Target Count : 1 PCI Bus Number : 18 PCI Device Number : 0 PCIe Max Bus Width : x4 PCIe Max Bus Speed : 2.5 Gbps PCIe Negotiated Width : x4 PCIe Negotiated Speed : 2.5 Gbps HBA Status : Online Record the WWNN value of the selected port (displayed next to the label Port Name). 7. To return to the HBA Information Menu, press Return and then select option 5 (Return to Previous Menu). 8. Repeat steps 4 on page 178 through 7 as needed to obtain the WWNN values for other ports. 9. From the HBA Information Menu, select option 6 to return to the Main Menu. 10. Select option 13 to exit from the Qlogic adapter command line interface. 11. Deliver the list of WWPN and WWNN values to the customer. Updating the server system planar board firmware and basic input/output system (BIOS) settings After the replacement of the system planar board, the BIOS level must be checked and updated to the tested level. About this task Hit <RETURN> to continue: Figure 71. HBA information display for a port For 3958 DD4 and 3958 DD5 servers, it is important to adhere closely to the replacement procedure of the microprocessor-board assembly in the IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Procedure 1. Connect a USB keyboard and monitor to the server or use the TSSC keyboard and monitor to access the command line of the node. Chapter 10. FRU replacement for TS7600 systems 179

204 Note: On 3958 DD4 and 3958 DD5 servers, in a small number of cases, it is necessary to update the firmware through the IMM Web interface. See Appendix B, Logging into the 3958 DD4, DD5, and AP1 servers through the IMM web interface, on page Insert the appropriate software DVD. ProtecTIER software version DVD 2.3 IBM System Storage RAS/BIOS and Firmware Update DVD (if not already inserted) 2.4 ProtecTIER 2.4 DVD (if not already inserted) 2.5 or later No DVD needed 3. At the login prompt, log in as ID root and password admin. 4. Display the current firmware versions. v For ProtecTIER software V3.1 or later, from the ProtecTIER Service menu, select Version Information > Display Firmware Levels (see ProtecTIER Service menu on page 16). v For ProtecTIER software V2.4 or V2.5, from the ProtecTIER Service menu, select Display Firmware Versions. v For ProtecTIER software V2.3 or earlier, from a command line type versions. If the command is not found, type the following commands to mount the IBM System Storage RAS/BIOS and Firmware Update DVD and display the firmware versions: mount /dev/hda /mnt/cdrom cd /mnt/cdrom./versions The output is similar to that shown in Figure 63 on page 173. Make a note of the displayed levels. 5. Compare the displayed levels to the levels listed in this document. v If the firmware levels are up to date, STOP. Skip the rest of this procedure. v If the firmware is not at the current level, perform the following steps to update the firmware to current supported field levels. 6. From the command line, type the appropriate commands to mount the software DVD (if any) and update the sysplanar. ProtecTIER software version Commands V2.3 and earlier mount /dev/hda /mnt/cdrom cd /mnt/cdrom./update_sysplanar V2.4 and later /opt/dtc/install/ptconfig -updatefirmwares 7. Type the appropriate commands to update the Sysplanar Broadcom Ethernet firmware. ProtecTIER software version Commands V2.3 and earlier mount /dev/hda /mnt/cdrom cd /mnt/cdrom./installbrcm V2.4 and later /opt/dtc/install/ptconfig -updatefirmwares 180 TS7650 and TS7650G Problem Determination and Service Guide

205 Note: Depending upon the model of the TSSC and TSSC network switch being used, the port layout might be different than shown. If so, attach and label the cables according to the port number assignments specified, regardless of the position of the ports on the TSSC or the TSSC network switch. Results The process of updating the server system planar board firmware and basic input/output system (BIOS) settings is complete. For both 7141 (3958 DD1) and 7233 (3958 DD3), the updated firmware levels are as follows: v brcm_fw_nic_2.1.0a_linux_32-64.bin (Sysplanar Broadcom Ethernet firmware update) v ibm_fw_bios_a3e166a_linux_i386.sh (Sysplanar BIOS) v ibm_fw_bmc_a3bt54a_linux_i386.sh (Baseboard Management Controller) v ibm_fw_fpga_a3ud23c_linux_i386.sh (Field Programmable Gate Array) v ibm_fw_dsa_3.00_dsyt55a_linux_32_64.bin (Dynamic System Analysis) What to do next v The Broadcom firmware update script updates only active ports with unique IP addresses. Sysplanar Port 1 (eth4) is used for the replication feature. Replication configuration is required before the firmware for Port 1 is updated. Sysplanar Port 2 (eth5) is used for connection to the TSSC. Installation of the RAS package is also required before firmware for Port 2 is updated DD4 and 3958 DD5 servers in VTL configuration have eth4 and eth5 onboard. Port eth5 is already active as a replication port, but eth4 must be temporarily reconfigured as an another active replication port for correct firmware update. After the firmware update, port eth4 can be restored to its previous configuration. v For 3958 DD4 and 3958 DD5 servers in OpenStorage or File System Interface configurations, ports eth12 and eth13 are onboard. Both ports must be temporarily reconfigured as active replication ports (with unique IP addresses) for correct firmware update. After the firmware update, eth12 must be restored to its previous configuration. (Restoring eth13 to its previous configuration is optional.) You must reboot the server to allow the new IMM, uefi, FPGA, DSA pre-boot and BRCM firmware to take effect. v After firmware updates have been completed on 3958 DD4 and 3958 DD5 servers, you must rebuild the RAS package. For ProtecTIER V2.5 or earlier, type the following command: /opt/dtc/install/ptconfig -configras For ProtecTIER V3.1 or later, select ProtecTIER Configuration > Configure RAS from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16). v After the rebuild is complete, continue with Chapter 12, End-of-call procedure, on page 249. If you are performing any other FRU or firmware updates, proceed to the appropriate section. Initializing a used hard disk drive for reuse Re-initialized hard disk drives (HDDs) cannot be used for hot-swapping. They can be used as a replacement field-replaceable unit (FRU) as shown in the following procedure. Chapter 10. FRU replacement for TS7600 systems 181

206 About this task Attention: This procedure initializes both of the HDDs in the server and requires a reload of the Red Hat Linux operating system and ProtecTIER code. Note: Only a fresh field-replaceable unit HDD can be used for hot-swapping. However, if a used HDD, with the same FRU part number, from another 7141 (3958 DD1) or 7233 (3958 DD3) system with the MR10K RAID controller, is available for use as a replacement FRU, it must be placed in one of the two HDD slots when the 3958 DD1/3958 DD3 is powered down. The same applies to a used HDD, previously used in a 3958 DD4 or 3958 DD5 (Type 7145) with a ServeRAID M5015 controller, and to be re-installed in a 7145 machine. Plug the used HDD as a replacement FRU into the appropriate HDD slot when the 7145 is powered down. Procedure To initialize the HDD, perform the following steps. 1. On the powered-down 7141/7233 machine or 7145 machine, plug the used HDD into the SCSI ID 0 slot or SCSI ID1 slot. 2. When the system starts up, after the System x logo is displayed, press Ctrl-H. 3. Click the Start button. 4. Click Configuration Wizard on the WebBIOS main screen. The Configuration Wizard screen is displayed. 5. Select the New Configuration option, which clears the existing configuration on the drives and lets you create a new configuration. 6. Click Next. The WebBIOS Configuration Method screen is displayed. 7. Select Automatic Configuration to create RAID 1 for the disk pair. Leave Redundancy when possible selected in the menu. 8. Click Next to continue. WebBIOS carries out the automatic configuration to create a virtual drive, based on two available drives. The Virtual Drive group and RAID 1 configuration are displayed. 9. Click Fast Initialize > Go. The HDD is initialized. 10. When the process completes, exit WebBIOS for a fresh load of Red Hat Linux and ProtecTIER code. Replacing the RAID controller in 3958 DD4, DD5, and AP1 servers This topic describes the procedure for replacing the RAID controller in 3958 DD4, DD5, and AP1 servers with the M5015 ServeRAID adapter. Procedure Perform the following steps to remove the existing RAID controller from the server and install an M5015 ServeRAID adapter: 1. Turn off the server and peripheral devices, and disconnect the power cords and all external cables as necessary to replace the device. 2. Remove the top cover. Attention: Operating the server for more than 2 minutes with the top cover removed might damage server components. For proper cooling and airflow, replace the top cover before you turn on the server. a. Slide the server out of the rack until the slide rails lock into place. 182 TS7650 and TS7650G Problem Determination and Service Guide

207 b. Press the button and rotate the cover-release latch (Figure 72). The cover slides to the rear approximately 13 mm (0.5 inch). c. Lift the cover off the server. Top cover Cover release latch ts Figure 72. Removing the top cover 3. Remove the top cover bracket: a. Slide the blue latches on the top-cover bracket toward the center of the server (Figure 73 on page 184). b. Tilt and lift the top-cover bracket out of the server. Chapter 10. FRU replacement for TS7600 systems 183

208 Latch Latch ts Figure 73. Removing the top cover bracket 4. Pull the blue handle on the RAID adapter carrier up to extract it from the server. 184 TS7650 and TS7650G Problem Determination and Service Guide

209 Tab RAID card carrier Slot ts Figure 74. Extracting the RAID adapter carrier 5. Disconnect the SAS cables from the RAID adapter. 6. Press the release latch and remove the RAID adapter from the carrier (Figure 75 on page 186). Chapter 10. FRU replacement for TS7600 systems 185

210 Post Release latch Tabs ts Figure 75. Removing the RAID adapter from the carrier 7. If a battery is installed on the RAID adapter, remove the battery carrier card and the battery from the RAID adapter. You must remove the three screws to separate them (Figure 76). ts Figure 76. Removing the battery carrier card from the adapter 186 TS7650 and TS7650G Problem Determination and Service Guide

211 8. If you are instructed to return the RAID adapter assembly, follow all packaging instructions and use any packaging materials for shipping that are supplied to you. 9. If you removed a battery carrier and battery from the former RAID adapter, use the three screws to install it on the new RAID adapter. 10. Install the replacement RAID adapter onto the RAID adapter carrier. 11. Connect the SAS cables to the RAID adapter. Note: Attach the SAS cable from port 0 to the lower backplane and the SAS cable from port 1 to the upper backplane (if one is installed). See Figure 77 and Figure 78 for port locations. ServeRAID M5015 controller Port 1 Port 0 ts Figure 77. SAS ports on the ServeRAID M5015 controller ServeRAID BR10i controller Port 1 Port 0 ts Figure 78. SAS ports on the ServeRAID BR10i controller 12. Slide the RAID adapter carrier and RAID adapter assembly into the slot on the side of the server (Figure 79 on page 188). Make sure the carrier is flat against the side wall of the server so that the adapter is installed in the connector correctly. Chapter 10. FRU replacement for TS7600 systems 187

212 Slot RAID card carrier Tab ts Figure 79. Inserting the RAID adapter carrier 13. Install the top cover bracket: a. Make sure all internal cables are correctly routed. b. Align the top-cover bracket on top of the server so that the metal tabs line up correctly on the chassis, and then rotate it into place. c. Slide the blue latches on the top cover bracket toward the outside of the server to lock it in place. 14. Install the top cover: a. Make sure all internal cables are correctly routed. b. Set the cover on top of the server so that approximately 13 mm (0.5 inch) extends from the rear. c. Make sure that the cover-release latch is up. d. Slide the top cover forward and into position, pressing the release latch closed. 15. Connect the cables and power cords. 16. Slide the server into the rack. 17. Turn on all attached devices and the server. The server restarts. The following message is displayed: Press any key to continue, or C to load the configuration utility 18. Press c to load the configuration utility. You are prompted to press Y to continue loading the configuration, or to power off the system and restart. 188 TS7650 and TS7650G Problem Determination and Service Guide

213 19. Press Y. The LSI Adapter Selection window is displayed. 20. Click the Start button. The LSI MegaRAID BIOS Config Utility Foreign Configuration window is displayed. 21. Click the Preview button. The Foreign Configuration Preview As Imported window is displayed, with the following message visible: Click IMPORT to Import and Merge this configuration. 22. Click the Import button. A Logical View of the virtual disk is displayed, showing both physical drives in online status with RAID1 enabled. 23. Click Exit. The Exit Application dialog is displayed. 24. Click Yes. The server resumes restarting with the newly merged ServeRAID controller card. Replacing the ServeRAID battery backup unit in 3958 DD4, DD5, and AP1 servers This topic describes the procedure for replacing a battery backup unit (BBU) on a ServeRAID adapter in 3958 DD4, DD5, and AP1 servers. Procedure Perform the following steps to replace a BBU on a ServeRAID adapter: 1. Turn off the server and peripheral devices, and disconnect the power cords and all external cables as necessary to replace the device. 2. Remove the top cover. Attention: Operating the server for more than 2 minutes with the top cover removed might damage server components. For proper cooling and airflow, replace the top cover before you turn on the server. a. Slide the server out of the rack until the slide rails lock into place. b. Press the button and rotate the cover-release latch (Figure 80 on page 190). The cover slides to the rear approximately 13 mm (0.5 inch). c. Lift the cover off the server. Chapter 10. FRU replacement for TS7600 systems 189

214 Top cover Cover release latch ts Figure 80. Removing the top cover 3. Remove the top cover bracket: a. Slide the blue latches on the top-cover bracket toward the center of the server (Figure 81 on page 191). b. Tilt and lift the top-cover bracket out of the server. 190 TS7650 and TS7650G Problem Determination and Service Guide

215 Latch Latch ts Figure 81. Removing the top cover bracket 4. Pull the blue handle on the RAID adapter carrier up to extract it from the server. Chapter 10. FRU replacement for TS7600 systems 191

216 Tab RAID card carrier Slot ts Figure 82. Extracting the RAID adapter carrier 5. Disconnect the SAS cables from the RAID adapter. Note: Mark the cables before removing them so they can be returned to their original positions. 6. Press the release latch and remove the RAID adapter from the carrier (Figure 83 on page 193). 192 TS7650 and TS7650G Problem Determination and Service Guide

217 Post Release latch Tabs ts Figure 83. Removing the RAID adapter from the carrier 7. Remove the battery carrier card and the battery from the RAID adapter. You must remove the three screws to separate them (Figure 84 on page 194). Note: The location of the screws will be different depending on the model of RAID adapter. The most current RAID adapter contains the three screws on the top of the BBU. Chapter 10. FRU replacement for TS7600 systems 193

218 ts Figure 84. Removing the battery carrier card from the adapter 8. Install the replacement BBU onto the adapter using the three screws. 9. Install the RAID adapter onto the RAID adapter carrier. 10. Connect the SAS cables to the RAID adapter. Note: Attach the SAS cable from port 0 to the lower backplane and the SAS cable from port 1 to the upper backplane (if one is installed). See Figure 85 for port locations. ServeRAID M5015 controller Port 1 Port 0 ts Figure 85. SAS ports on the ServeRAID M5015 controller 194 TS7650 and TS7650G Problem Determination and Service Guide

219 11. Slide the RAID adapter carrier and RAID adapter assembly into the slot on the side of the server (Figure 86). Make sure the carrier is flat against the side wall of the server so that the adapter is installed in the connector correctly. Slot RAID card carrier Tab ts Figure 86. Inserting the RAID adapter carrier 12. Install the top cover bracket: a. Make sure all internal cables are correctly routed. b. Align the top-cover bracket on top of the server so that the metal tabs line up correctly on the chassis, and then rotate it into place. c. Slide the blue latches on the top cover bracket toward the outside of the server to lock it in place. 13. Install the top cover: a. Make sure all internal cables are correctly routed. b. Set the cover on top of the server so that approximately 13 mm (0.5 inch) extends from the rear. c. Make sure that the cover-release latch is up. d. Slide the top cover forward and into position, pressing the release latch closed. 14. Connect the cables and power cords. 15. Slide the server into the rack. 16. Turn on all attached devices and the server. Chapter 10. FRU replacement for TS7600 systems 195

220 Replacing a Dual-Port or Quad-Port Ethernet card and recovering Ethernet settings None of the 3958 adapter cards are hot-swap capable because the server operating system does not permit the adapters to be varied offline prior to removal. The Ethernet replacement tool is an interactive tool that is designed to recover the Ethernet settings after replacing an Ethernet card. Before you begin Attention: (ProtecTIER V2.3 and earlier only) When using the Ethernet Replacement Tool procedure, you must load the latest RAS package (verson 3.1.x or higher) prior to running the tool from the menu selection. Refer to the RAS package update information in the IBM System Storage TS7600 with ProtecTIER Installation Instructions for the RAS Package, BIOS, and Firmware updates following a FRU replacement for models 3958 DD1, 3958 DD3, and 3958 AP1, PN 46X2459 and then return to this procedure. Procedure Perform the following steps to replace a Dual-Port or Quad-Port Ethernet card and recover Ethernet settings. 1. Prepare the system for FRU replacement (see Preparing the system for FRU replacement on page 170). 2. Replace the Ethernet adapter (see Replacing the field-replaceable unit on page 171). After replacing the FRU, continue with step Replug the disconnected Ethernet cables and power on the server. 4. Log in with user ID ptadmin and password ptadmin. 5. Run the Ethernet card replacement tool on the server by using one of the following methods. v ProtecTIER V2.5 or later: The Ethernet card replacement tool is activated automatically on bootup of the server. (This activation includes the automatic update of the quad-port Ethernet replacement card on the 3958 DD4 or 3958 DD5 server.) To check that all is repaired as expected, from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16), select the options to run a full check on the node. For ProtecTIER V2.5 or earlier, select System Health Monitoring > Run a full check on this node. For ProtecTIER V3.1 or later, select Health Monitoring > Run a full check on this node. If no degraded check points are indicated, then the Ethernet settings of the replaced adapter have been successfully recovered. After the health check has been successfully completed, STOP. Skip the remaining steps of this procedure and continue with Chapter 12, End-of-call procedure, on page 249. v ProtecTIER V2.4 or earlier, at the command line, type: rscercfgupdateethconnections. -ORv ProtecTIER 2.4 or earlier, select Manage Configuration > Update Eth Connections. 6. To verify, ping other parties on the network. 196 TS7650 and TS7650G Problem Determination and Service Guide

221 Example The following example shows activation of this tool for the dual-port Ethernet card on a system running ProtecTIER V2.4 or earlier. 1. Type the command rscercfgupdateethconnections. # /opt/ras/bin/rscercfgupdateethconnections 2. The Ethernet replace tool identifies the replaced card: The following replacement(s) were discovered for 2 port Ethernet card on slot 3 eth1 Old MAC address = 00:15:17:1e:2d:71 New MAC address= 00:15:17:7f:28:8d eth0 Old MAC address = 00:15:17:1e:2d:70 New MAC address= 00:15:17:7f:28:8c IP: Do you wish to correct the ethernet configuration? YES/NO If you wish to correct the Ethernet configuration, type yes. Do you wish to correct the ethernet configuration? YES/NO yes port to configure:0 Please disconnect ethernet cables from the replaced card, press Enter to continue or type exit 3. Disconnect all Ethernet cables from the replaced card, and press Enter. Verifying cables are disconnected... may take a few minutes Configuring eth0 Please connect ethernet cable to eth0 press Enter to continue or type exit 4. Connect a single cable to the requested port (Eth0 in our case) and press Enter. Checking for connected cable... may take a few minutes About to configure eth0 IP= Please confirm YES/NO Note: In some instances, you are requested to connect the other cable as well. In such a case you do not have to follow steps 6 through 8 on page 198. This action would also update eth1 If you wish to connect a cable to eth1 Please connect it before typing YES Please confirm YES/NO 5. If you agree with the settings, type yes. Please confirm YES/NO Yes Successfully configured eth0 6. In the following steps, the configuration for the port on the replaced card is continued. Chapter 10. FRU replacement for TS7600 systems 197

222 port to configure:1 Configuring eth1 Please connect ethernet cable to eth1 press Enter to continue or type exit 7. If you wish to configure the port, connect the Ethernet cable to the port and press Enter. Checking for connected cable... may take a few minutes About to configure eth1 Please confirm YES/NO 8. If you accept the settings, type yes. Successfully configured eth1 Please remember to connect all disconnected cables! The Ethernet settings are recovered. 9. To verify, ping other parties on the network. 10. The Intel Network DHCP-boot can be disabled by booting from one of the DVDs listed below (depending on the installed version of ProtecTIER) and disabling the Intel Network DHCP-boot. a. Insert one of the following in the CD/DVD drive: v IBM System Storage RAS/BIOS and Firmware Update DVD v TS7650 ProtecTIER 2.4.x server DVD v TS7650G ProtecTIER 2.4.x server DVD b. Restart the server. The TS7600 BIOS and Firmware tool menu is displayed. IBM TS7600 BIOS and Firmware tool Select from the following options: [1] Update the Qlogic Firmware [2] Update the Emulex Firmware [3] Run Intel Boot Agent Utility [4] Run all updates [X] Exit to command prompt [0] Reboot c. Select Run Intel Boot Agent Utility. d. When the Intel Boot Agent Utility menu is displayed, select Disable DHCP-Boot on all Intel NIC adapters. e. Select the option to Reboot. What to do next If the Ethernet adapter replacement procedure is complete, and no further firmware is being updated, go to Chapter 12, End-of-call procedure, on page TS7650 and TS7650G Problem Determination and Service Guide

223 If additional updates are being performed, continue to the relevant section of this document. Removing and replacing the microprocessor-board assembly or the I/O board Shuttle assembly These topics provide instructions for removing and replacing the microprocessor-board assembly or the I/O board Shuttle assembly. Removing and replacing the microprocessor-board assembly or the I/O board Shuttle assembly in the 3958 DD4/3958 DD5 server Use these instructions to remove and replace the microprocessor-board assembly or the I/O board Shuttle assembly. About this task To remove and replace the microprocessor-board assembly in the 3958 DD4/3958 DD5 server, refer to Removing the microprocessor-board assembly and Replacing the microprocessor-board assembly in chapter 5 of IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. To remove and replace the I/O board Shuttle in the 3958 DD4/3958 DD5 server, refer to Removing the I/O-board shuttle and Replacing the I/O-board shuttle in chapter 5 of IBM System x3850 X5 and x3950 X5 Types 7145 and 7146 Problem Determination and Service Guide. Procedure 1. Pay close attention to the steps in the procedure Replacing the microprocessor-board assembly that refer to the use of the Advanced Settings Utility (ASU) to restore the product name, machine type, and serial number of the server. The ASU is used when the server is booted into the operating system. Consult with the trained xseries representative to imprint this vital information via ASU or from a saved configuration file. 2. If necessary, set the system product fields (Product Name, Model/Type, and Serial Number) on the new microprocessor board. a. Locate the product label ( 1 in Figure 87 on page 200) on the right side of the server. b. Enter the correct values using the following commands: cd /usr/bin./asu64 set SYSTEM_PROD_DATA.SysInfoProdName "XXXXXXX"./asu64 set SYSTEM_PROD_DATA.SysInfoProdIdentifier "System x3850 X5"./asu64 set SYSTEM_PROD_DATA.SysInfoSerialNum "nnnnnnn" where XXXXXXX is the Type and Model shown on the label 1, and nnnnnnn is the seven-digit serial number. Chapter 10. FRU replacement for TS7600 systems 199

224 1 ts Figure 87. Product label on server 3. Verify that the information was entered correctly. a. Type the following command: dmidecode -s System-Product-Name Output similar to the following is displayed: System x3850 X5 -[7145PBR]- b. Type the following command: dmidecode -s System-Serial-Number The seven-digit serial number is displayed. What to do next v v v Following the replacement of the I/O board Shuttle assembly, it is necessary to update firmware to the two onboard BroadCom (BRCM) Ethernet interfaces. Go to Updating the onboard Ethernet adapter firmware on page 202. When new MAC addresses are recorded on bootup (due to replacement of an Ethernet card with the new microprocessor-board assembly or due to replacement of the I/O board shuttle), the MAC addresses are correctly configured to the appropriate Ethernet ports. Further configuration of the Ethernet ports to their previous settings occurs automatically on startup of the Ethernet ports. See Replacing a Dual-Port or Quad-Port Ethernet card and recovering Ethernet settings on page 196. If problems with automatic reconfiguration of the network ports occur during startup, you must reinstall the Red Hat Linux operating system and the ProtecTIER code. 200 TS7650 and TS7650G Problem Determination and Service Guide

225 Removing and replacing the microprocessor-board assembly or the I/O board Shuttle assembly in the 3958 DD1/3958 DD3 server Use these instructions to remove and replace the microprocessor-board assembly or the I/O board Shuttle assembly. Before you begin Following the replacement of the microprocessor-board assembly, it is necessary to restore the serial number (S/N) of the server. Note: The server S/N and system product name must be gathered before conducting a remove and replace of the microprocessor board. If the server is still active, issue the following commands and record the information. 1. dmidecode -s system-serial-number A seven-digit serial number is displayed. Write down the serial number. 2. dmidecode -s system-product-name The system product name information is displayed. Write down the text in bold. For example: IBM 3850 M2 / x3950 M2 -[7233PAF]- If the server is not accessible, verify the serial number and product name from the previous Call Home concerning this issue. About this task To remove and replace the microprocessor-board assembly in the 3958 DD1/3958 DD3 server, see Removing the microprocessor-board assembly and Replacing the microprocessor-board assembly in chapter 5 of IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. To remove and replace the I/O board Shuttle in the 3958 DD1/3958 DD3 server, see Removing the I/O-board shuttle and "Replacing the I/O-board shuttle" in chapter 5 of IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide. Procedure Perform the following steps to change the S/N of the new microprocessor board. 1. Download the x3850 M2 BIOS 170A.ISO file from the IBM Fix Central website, or contact IBM Support (Level 2 Support) for ProtecTIER to forward this.iso file to you, and save it to your laptop. Note: If the.iso file is not readily available at Fix Central, consult immediately with IBM Support for ProtecTIER to provide the.iso file to you. 2. Burn the x3850 M2 BIOS 170A.ISO file to a new CD. Use of ROXIO (on your laptop) is recommended to burn the.iso file to the new CD. The.iso file must be the only file on the CD. 3. Use the new CD on the x3850 M2 system to boot up to DOS. 4. Select option 1 Update POST/BIOS from the first menu. 5. Continue through the menus as if you are going to flash the BIOS. 6. You are prompted to enter a new S/N or keep the existing one. 7. You are prompted to enter a new system product name or keep the existing one. Chapter 10. FRU replacement for TS7600 systems 201

226 Note: This prompt is provided as part of the S/N update program. Expect to keep the existing system product name, except in extremely rare cases when wrong data might have been imprinted. 8. Activate the changes, and allow the vital product data (VPD) to be rewritten, by reflashing BIOS. Note: BIOS Firmware (FW) version 1.16 is flashed. This is the latest level of BIOS FW currently available. If BIOS FW code has been updated to a level higher than version 1.16, then select the Update Firmware menu option after boot up, or execute the command: /opt/dtc/install/ptconfig -updatefirmwares This upgrades the BIOS FW, if a new version greater than version 1.16 has been incorporated in the ProtecTIER code. 9. Upon completion of the reboot, the S/N (and the system product name if modified) is changed. What to do next v Following the replacement of the I/O board Shuttle assembly, it is necessary to update firmware to the two onboard BroadCom (BRCM) Ethernet interfaces. Go to Updating the onboard Ethernet adapter firmware. v If MAC addresses are not correctly recorded on replacement of the microprocessor-board assembly or of the I/O board shuttle on bootup, see Replacing a Dual-Port or Quad-Port Ethernet card and recovering Ethernet settings on page 196. v The procedure Replacing a Dual-Port or Quad-Port Ethernet card and recovering Ethernet settings on page 196 might not recover the new MAC addresses for the appropriate network ports. In this case, you must reinstall the Red Hat Linux operating system and the ProtecTIER 2.4.x (or lower) code and reconfigure the system. v When ProtecTIER 2.5.x or 3.1.x is loaded in the 3958 DD1/3958 DD3, all MAC addresses are correctly configured to the appropriate Ethernet ports. Further configuration of the Ethernet ports to their previous settings occurs automatically on startup of the Ethernet ports. See Replacing a Dual-Port or Quad-Port Ethernet card and recovering Ethernet settings on page 196. v If problems with automatic reconfiguration of the network ports occur during startup, you must reinstall the Red Hat Linux operating system and the ProtecTIER code. Updating the onboard Ethernet adapter firmware Use these instructions to update the firmware on the two onboard BroadCom (BRCM) Ethernet adapters. About this task Following the replacement of the I/O board Shuttle assembly, it is necessary to update firmware on the two onboard BRCM Ethernet interfaces. Procedure Perform the following steps to update the onboard Ethernet adapter firmware. 1. Contact IBM Support (Level 2 Support) for ProtecTIER to forward the brcm_fw file to you, and save it to your laptop. 2. Copy the brcm_fw file to a writable CD. 202 TS7650 and TS7650G Problem Determination and Service Guide

227 Note: Ensure that ProtecTIER code has been loaded to the server. The brcm_fw file uses the current version of BRCM firmware which is provided by ProtecTIER code in directory /opt/dtc/firmwares. 3. Insert the CD into the DVD drive of the server. 4. Copy the brcm_fw file to the /var/tmp directory on the server by typing the following commands: a. mount /dev/hda /mnt/cdrom (for DD4/DD5) -ORmount /dev/cdrom /mnt/cdrom (for DD3) b. cd /mnt/cdrom c. cp brcm_fw /var/tmp d. cd /var/tmp e. eject (remove the CD from the DVD drive) 5. Verify that the BRCM firmware is present in the /opt/dtc/firmwares directory by typing the following commands: cd /opt/dtc/firmwares ls grep brcm The filename will be similar to: brcm_fw_nic_2.x.xx_linux_32-64.bin. Note: The exact filename of the BRCM firmware file must be: brcm_fw_nic_2.1.7a_linux_32-64.bin 6. Go to the /var/tmp directory by typing the command: cd /var/tmp. 7. Update the firmware by typing the command:./brcm_fw. 8. Allow the script to finish, then reboot the server. Note: The script displays progress of the firmware update. The updated firmware levels are shown at the end of the script, as well as a reminder to reboot the system. Reboot is necessary to activate the updated firmware code level. 9. Verify that the firmware is updated: a. Type menu at the command prompt and press Enter. The ProtecTIER Service menu is displayed. b. Select Version Information > Display Firmware Versions. Example: Resource ID Version Description Ethernet bnx2 2.x.xx Broadcom Corporation NetXtreme II Ethernet bnx2 2.x.xx Broadcom Corporation NetXtreme II Note: The updated firmware version for both of the onboard Ethernet ports should be: bc NCSI Removing and replacing the RSA This topic describes the process of removing and replacing the Remote Supervisor Adapter (RSA). v See Preparing the system for FRU replacement on page 170 for the procedure to prepare the system for RSA removal and replacement. Chapter 10. FRU replacement for TS7600 systems 203

228 v See the IBM System x3850 M2 and x3950 M2 Types 7141, 7233, 7144, and 7234 Problem Determination and Service Guide for the procedures to remove and replace the RSA in the server. v After replacing the RSA card, you must set the IP address. The RSA card IP address is determined by the frame number and node ID. Set the IP address for the RAS card to the IP address for the frame number plus one for Node A, or the frame number plus 6 for Node B. Configuring the RSA card Following the replacement of the remote supervisor adapter (RSA) card, the IP address needs to be set. The RSA card IP address is determined by the frame number and node ID. Set the IP address for the RSA card to the IP address for the frame number plus one for Node A or the frame number plus 6 for Node B. Before you begin Attention: contains. Review this procedure before performing any service actions it Procedure After the RSA card has been replaced, perform the following procedure to update the IP addresses. 1. Power on the 3958 server. When the IBM logo displays and the function key options appear, press F1 (multiple times) to enter Setup Mode. 2. From the Configuration/Setup Utility menu, select Advanced Setup RSA II Settings. The RSA II settings are displayed (Figure 88). RSA II Settings RSA II MAC Address E-2A-06-4F DHCP IP Address DHCP Control [ Use Static IP ] Static IP Settings [ ] Static IP Address [ ] Subnet Mask [ ] Gateway OS USB Selection [ Linux OS ] Save Values and Reboot RAS II <<<RESTORE RSA II DEFAULTS>>> Figure 88. RSA II settings 3. Configure the settings. a. If it is not already set, change the DHCP Control to Use Static IP. Use the right and left arrow keys to select the correct option. b. Set the value of the Static IP Address of the RSA II to xx1 (lower server, Node A) or xx6 (upper server, Node B), where xx is the frame number. Examples: v For a frame of 10, set the IP address of the RSA card in the lower server (Node A) to v For a frame of 10, set the IP address of the RSA card in the upper server (Node B) to TS7650 and TS7650G Problem Determination and Service Guide

229 v For a frame of 20, set the IP address of the RSA card in the lower server (Node A) to v For a frame of 20, set the IP address of the RSA card in the upper server (Node B) to c. Set the Subnet Mask to , and set the Gateway to d. Set the OS USB Selection to Linux. Use the right and left arrow keys to select the correct option. e. Select Save the Values and Reboot RSA II with the arrow keys and press Enter. The system might stop for a few seconds. This is normal. f. When the menu returns, press the Esc key until you return to the Configuration/Setup Utility. Select Save Settings. g. After the settings have been saved, select Exit Setup, then select Yes to exit the setup utility and reboot the system. The 3958 server restarts to activate the new settings. 4. To verify the new RSA card is accessible from the TSSC with the new IP address set and the basic input/output system (BIOS) change, open a terminal window on the TSSC and issue a ping to the new RSA IP address. v If the ping of the new RSA IP address is successful, proceed to Installing the RSA II firmware. v If the ping of the new RSA IP address is not successful, repeat steps 2 on page 204 and 3 on page 204. What to do next Proceed to Installing the RSA II firmware. Installing the RSA II firmware You can update the firmware of the RSA II through its Web interface for System x3850 M2, Type 7141 and Type Before you begin The RSA II firmware can be updated from the server's command line, or from the ProtecTIER Service menu (V3.1 or later), if the server's firmware levels are at the levels listed in Figure 63 on page 173. Note: To update the RSA II firmware for ProtecTIER V2.4 or later, do not use the IBM System Storage RAS/BIOS and Firmware Update DVD. Two firmware files are required for the RSA II update, which are transferred from the IBM System Storage RAS/BIOS and Firmware Update DVD (ProtecTIER V2.3 only) in the x3850 M2 server to the TSSC service directory (/home/service) and then flashed remotely to the RSA II via the TSSC web browser. Note: This procedure requires the use of the TSSC with functioning Ethernet connections to the 3850 M2 xserver (IP address xx) and to the RSA II Web interface (IP address xx+1). The IBM System Storage RAS/BIOS and Firmware Update DVD (ProtecTIER V2.3 only) must be mounted in the server before you begin. Both the xseries 7141 and 7233 servers use the same RSA II card and firmware. Chapter 10. FRU replacement for TS7600 systems 205

230 Procedure 1. For ProtecTIER V3.1 or later, from the ProtecTIER Service menu, select ProtecTIER Configuration > Update Firmware (see ProtecTIER Service menu on page 16). Skip to step 4 2. To update the RSA II adapter card firmware from the command line: v v For ProtecTIER V2.4 and later, type the following command: /opt/dtc/install/ptconfig -updatefirmwares For ProtecTIER V2.3 and earlier, type the following commands: mount /dev/hda /mnt/cdrom./installrsafw cd / eject 3. If the installrsafw command fails, proceed to Loading the RSA firmware from the TSSC on page You must reboot the RSA II adapter for the firmware to take effect: a. In the TSSC browser, enter the IP address of the RSA II adapter ( xx) in the URL field. Example: From the TSSC attached system list, if the server TSSC IP is , then the RSA II IP is If the second node is , then the RSA II IP is b. In the RSA II, select Restart ASM. 5. Add the user ID Service to the RSA Adapter: a. On the TSSC, open a browser window. b. Enter the IP address of the RSA II adapter ( xx) in the URL field. Example: From the TSSC attached system list, if the server TSSC IP is , then the RSA II IP is If the second node is , then the RSA II IP is c. In the Login window, enter USERID as the username and PASSW0RD (with a zero) as the password, and then click OK. d. From the left pane, select Login Profiles. The Login Profiles window displays. ts Figure 89. Login Profiles window 206 TS7650 and TS7650G Problem Determination and Service Guide

231 e. In the Login ID column, click not used item 3. The Login Profile 3 dialog box displays. ts Figure 90. Login Profile dialog box f. Fill in the Login Profile: 1) In the Login ID field, type Service. 2) In the Password and Confirm password fields, type ibm2serv. 3) Select the Read-Only Authority Level. 4) Press Enter to save the profile. g. Clear the RSA event log. h. Run a full check on the corresponding node. i. Check the RSA event log and verify Service logged in successfully, as shown in the following example: ts Figure 91. RSA event log What to do next Use the procedures described within this document to upgrade all the other firmware to the latest levels. Loading the RSA firmware from the TSSC You can use the TSSC web browser to update the RSA II firmware to the latest released levels. About this task The following procedure explains how to display the current levels so you can determine whether the procedure to update the levels needs to be performed. If the installed levels are up to date, the server can be turned back over to the customer. If this is a clustered system, you must check both servers. Chapter 10. FRU replacement for TS7600 systems 207

232 Procedure 1. If you are not already logged into the TSSC, log in with the user ID service and the password service. 2. Right-click an empty area of the TSSC desktop, and from the IBM TS3000 System Console menu, select Browser Functions > Launch Browser. 3. In the address or URL field, type the IP address of the RSA II card in the 3850 M2 xserver to which you want to connect. The IP address is xx+1. Example: If the server's TSSC IP address is , the RSA IP address is Note: In a clustered configuration, update the second server using the xxx + 6. Example: If your TSSC has an IP address of , the RSA IP address for the second server is In the User Name prompt, enter the user ID USERID and the password PASSW0RD (the 0 is a zero). Click OK. 5. Select a timeout value of 10 minutes in the field provided and click Continue. Note: If the browser is inactive for the number of minutes in the timeout value, the RSA II logs off the Web interface. You must log in again. 6. Check the current levels of the firmware. At the System Status page in the left navigation frame under Monitors, click on Vital Products Data. Note: To verify the correct model of server you are upgrading, check the RSA Vital Product Data panel under the top panel heading of Machine Level VPD. Make a note of the Machine Type (7141 or 7233). 7. Scroll down to ASM VPD to verify that the build level of the two firmware files is at the same, updated level (Example: A3EP29B). If the firmware code needs to be updated, proceed to the next step. If the firmware does not need to be updated, proceed to step 33 on page 211. The current levels of firmware for both 7141 and 7233 model types are listed below: v Main Application: A3EP41B PAETMNUS.PKT v Boot ROM: A3BP41B PAETBRUS.PKT Both with date codes of If the RSA II firmware needs to be updated, right-click the desktop and from the IBM TS3000 System Console menu, select System Console Actions, then select CD Copy to Console. ACD Copy to Console window is displayed. See Figure 92 on page TS7650 and TS7650G Problem Determination and Service Guide

233 Figure 92. TS3000 System Console - CD Copy to Console window 9. Select item 3 for the /home/service/offload directory and press Enter. 10. The DVD tray opens automatically. Insert the IBM System Storage RAS/BIOS and Firmware Update DVD in the DVD drive of the TSSC and press Enter. A warning is displayed if files exist in the /home/service/offload directory. This operation will erase contents. Continue [ y/ n]. Note: This directory is only used for storing temporary data. It does not contain any files that are needed for future use. It is safe to erase the contents of this directory. 11. Press y and then Enter to continue. 12. After the copy is completed, press Enter to exit. The terminal closes and the DVD is ejected automatically. 13. Right-click on the desktop and from the IBM TS3000 System Console menu, select Terminal. 14. Change directories to the /home/service/offload directory by entering the following command: cd /home/service/offload 15. Expand the file using the following command: tar -xvf rsacode.tar -C / The following directories and files are placed on the TSSC in the /home/service directory: v TSSC_server:/home/service/ # tar -xvf rsacode.tar -C / /home/service/code/ /home/service/code/7141/ /home/service/code/7141/paetbrus.pkt /home/service/code/7141/rtalert.mib Chapter 10. FRU replacement for TS7600 systems 209

234 /home/service/code/7141/rtrsaag.mib /home/service/code/7141/paetmnus.pkt /home/service/code/7141/readme.txt /home/service/code/7233/ /home/service/code/7233/paetbrus.pkt /home/service/code/7233/rtalert.mib /home/service/code/7233/rtrsaag.mib /home/service/code/7233/paetmnus.pkt /home/service/code/7233/readme.txt v TSSC_server:/home/service/offload 16. Switch back to the browser screen and click on System Status. 17. The System Status window appears, which provides a view of the system status. In the left pane navigation frame, select Tasks > Firmware Update > Browse. 18. Select the directory path /home/service/code/server_type (where server_type is 7141 or 7233) to allow navigation to the PKT file you will use to do the update. 19. The two packet files you copied to the TSSC need to be found: PAETBRUS.PKT PAETMNUS.PKT Attention: The packets must be flashed in the order listed above. 20. Click the PAETBRUS.PKT file to highlight it, and then click Open. The file, including the full path, appears in the box beside Browse. 21. Click Update to begin the update process. A progress indicator opens as the file is transferred to temporary storage in the RSA II adapter. 22. Verify that the PKT file shown on the Confirm Firmware Update window is what you intend to update (RSA Boot ROM). If it is not correct, click the Cancel button, and click OK when asked Are you sure?. Browse again for the PAETBRUS.PKT file and repeat The first firmware file to be flashed (PAETBRUS.PKT) is confirmed by the message that the RSA Boot ROM firmware file is ready to be flashed. Click Continue. 24. A progress indicator and confirmation page opens to verify that the update was successful. The status message for a successful flash of the first firmware file displays: Update of the RSA Boot ROM firmware successful. 25. In the left navigation pane, click Firmware Update again and repeat the file retrieval and transfer process or the second (PAETMNUS.PKT) file (RSA Main Application), but verify it is the PAETMNUS.PKT file. 26. Click Continue to complete the update process. A message is displayed, notifying you that the RSA Main Application firmware file is ready to flash. A progress indicator is displayed as the firmware on the RSA II adapter is flashed. The status message for successful flash of the second firmware file displays: Update of RSA Main Application firmware was successful. 27. After you receive confirmation that the update process is complete for the second packet file, in the left navigation pane, click the option Restart ASM and click the Restart button. 28. Click OK to confirm that you want to restart the RSA II adapter. Your current session is disconnected. 210 TS7650 and TS7650G Problem Determination and Service Guide

235 29. Close your browser window, or if you see a confirmation message, click OK or logoff to close the current browser window. The RSA restarts. (This takes a few minutes.) 30. After the RSA restart, log into the RSA II again to verify the new RSA II firmware build level. 31. Open the TSSC browser and follow the login process again. 32. At the System Status window, in the left navigation frame under Monitors, click on Vital Products Data. Scroll down to ASM VPD to verify that the build level of the two firmware files is at the same, updated level (Example: A3EP41A). 33. In the left navigation pane, scroll to the bottom of the directory tree, and click Log Off, then close the current browser session by selecting File > Close. 34. Return to your previous Terminal window on the TSSC Console. 35. If this is a dual node clustered environment, repeat the firmware update procedure on the second node. Otherwise, continue to the next step. 36. Go to the /home/service/code/7233 directory for a 3958 DD3 or 3958 AP1 or the /home/service/code/7141 directory for a 3958 DD1, and inform the customer that this is the location of the MIB files for the SNMP trap catcher. The MIB files can be manually copied to portable media or sent by FTP to the SNMP server. These MIB files are used with the customer SNMP catcher to provide detailed information on the RSA SNMP traps, and are needed to decipher the text to a readable format. For more information, refer to the IBM System Storage TS3000 System Console (TSSC) Maintenance Information document provided on the IBM System Storage TS7650 with ProtecTIER Publications CD in the TS7650 ship group. On either node, the MIB file for the ProtecTIER code is in the /usr/share/snmp/mibs/diligent-mib.txt file. Provide this file to the customer also. 37. Close any open terminal sessions. At the command prompts, you can type exit to close the window, and from an empty area of the TSSC desktop, right-click and select Logout. Results Recovering a node The process of loading the RSA firmware from the TSSC is complete. Reinstalling the Red Hat Linux operating system overlays all the Reliability, Availability, and Serviceability (RAS), basic input/output system (BIOS), and firmware scripts and utility rpms. As a result, an IBM Service Representative must reload the RAS package after the customer reloads Red Hat Linux. Attention: TS7600 products require a TSSC. The TSSC is a vital part of the service strategy. For the RAS package to gather logs at the time of the failure and send a call home packet, a TSSC is required. If the RAS package is not installed, the customer must call 1(800) IBM SERV (7378) (in North America, only) or visit the IBM Directory of worldwide contacts web page to obtain hardware support. ProtecTIER must also be reinstalled following reinstallation of Red Hat Linux. This is typically a customer responsibility. Refer to the system recovery information in the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA , provided on the IBM System Storage TS7650 with ProtecTIER Publications CD. Chapter 10. FRU replacement for TS7600 systems 211

236 The following items are required when recovering a node in a clustered installation: 1. Installation of the appropriate version of Red Hat Linux on the new server. See the system recovery information in the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA for more information. 2. Extraction and reload of the ProtecTIER package. See the Unpacking the ProtecTIER package section of the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA for more information. 3. Reinstallation of the RAS package on the new server. Reinstallation of the RAS package is required. A reload of the Red Hat Linux operating system erases all components of the RAS package. To reinstall the RAS package, see Reinstalling the RAS package (ProtecTIER software V2.3 or earlier only). Automatic node recovery All of the procedures for automatic node recovery are documented in the Recovery Procedures appendix of the IBM System Storage ProtecTIER User's Guide for Enterprise Edition and Appliance Edition, GA The recovery initiates the automatic node replacement procedure. This includes missing node identification and performing the following tasks: v Configures network settings v Configures hostname v Configures bond v Configures Red Hat Linux cluster v Restarts all cluster activities on the other node in the cluster if in a clustered installation v Discovers and registers the repository file systems in the /etc/fstab directory v Configures multipath Reinstalling the RAS package (ProtecTIER software V2.3 or earlier only) This topic provides the steps for reinstalling the RAS package on a system running ProtecTIER software V2.3 or earlier. About this task For more detailed information about installing the RAS package on a system running ProtecTIER software V2.3 or earlier, see the IBM System Storage TS7600 with ProtecTIER Installation Instructions for the RAS Package, BIOS, and Firmware updates following a FRU replacement for models 3958 DD1, 3958 DD3, and 3958 AP1, PN 46X2459. v To perform the RAS package installation, you must first establish a server connection. v If you are installing the RAS package in a clustered configuration, perform the installation on Server A (the bottom server) first. v If this is the second node installation in a clustered configuration, make sure the first node is up and running before proceeding. Procedure 1. Insert the IBM System Storage RAS/BIOS and Firmware Update DVD into the DVD drive on the 3958 server on which you are installing this package. 212 TS7650 and TS7650G Problem Determination and Service Guide

237 2. If not already logged in, log into the 3958 server with the user ID root and the password admin. 3. From the command line, enter in the following commands to mount the IBM System Storage RAS/BIOS and Firmware Update DVD and launch the installation application: mkdir /mnt/cdrom Note: If the /mnt/cdrom directory already exists, the following message is displayed: mkdir: cannot create directory /mnt/cdrom : File exists If this message is displayed, ignore it and proceed to the next step. mount /dev/hda /mnt/cdrom cd /mnt/cdrom./rscercfginstall Follow the instructions on the screen to complete the installation. Important: Use the same frame number for both nodes when clustered together in the same frame. 4. The 3958 DD1 first release (August 2008) supported RSA Ethernet connections on the local customer LAN. This procedure alters that configuration to now connect the 3958 RSA ethernet connection to connect to the TSSC private network ( xxx). Disconnect the local LAN Ethernet connection to the RSA adapter port, and connect a new Ethernet cable between the RSA Ethernet port and the TSSC SMC hub. You are prompted for this action during the installation. If the connection between the RSA Ethernet port and the TSSC SMC hub is already established, proceed to the next step. Note: Depending upon the model of the TSSC and TSSC network switch being used, the port layout might be different than shown. If so, attach and label the cables according to the port number assignments specified, regardless of the position of the ports on the TSSC or the TSSC network switch. 5. After the installation is complete, unmount the IBM System Storage RAS/BIOS and Firmware Update DVD and eject it using the following commands: cd / eject /dev/hda 6. Repeat steps 1 on page 212 through 5 for the second 3958 server if applicable. You are prompted for frame number. 7. Enter the same number that you entered for the first node. For example, if you configured the frame on the first node to be 110, configure the frame on the second node to 110 also, provided the two servers are being clustered together. 8. When all procedures have been completed and you are returned to a command line, type exit to log out of the server. Results The process of reinstalling the RAS package (for ProtecTIER software V2.3 or earlier only) is complete. Removing and replacing Disk Storage controller FRUs Use the procedure in this topic to replace any field-replaceable unit (FRU) or customer-replaceable unit (CRU) in the disk controller. Chapter 10. FRU replacement for TS7600 systems 213

238 Before you begin Attention: Before upgrading the ProtecTIER code level, you must update the DS4700 storage subsystem firmware, NVSRAM firmware, and the ESM firmware on your EXP810 expansion drawers. You must call IBM Service to schedule a time to update the firmware to the levels listed below. Do not proceed to upgrade the ProtecTIER code level to version 2.4.x until: v The storage subsystem firmware has been upgraded to the minimum level of or the recommended level of v The DS4700 NVSRAM firmware has been updated to the minimum level of N1814D47R1050V07 or the recommended level of N1814D47R1060V04. v The expansion unit ESM firmware has been updated to the minimum level of 98C5 or the recommended level of 98D0. Procedure 1. It is necessary to power off the server (or servers) before starting work on the Disk Storage controller. Refer to Power procedures for ProtecTIER version 3.1 or later on page 237 or Powering off the ProtecTIER server version 2.5 or earlier on page Identify the location of the FRU or CRU to be replaced and follow the replacement procedures for the FRU or CRU. Refer to the IBM System Storage DS4700 Express Storage Subsystem Installation, User's and Maintenance Guide. 3. Once the location has been identified, use the IBM System Storage DS4700 Express Storage Subsystem Installation, User's and Maintenance Guide for replacement procedures. 4. Power on the server (or servers), refer to Power procedures for ProtecTIER version 3.1 or later on page 237 or Powering on the ProtecTIER server version 2.5 or earlier on page Referring to the IBM System Storage DS4700 Express Storage Subsystem Installation, User's and Maintenance Guide, perform any checkout procedures. Results The process of removing and replacing Disk Storage controller FRUs is complete. 214 TS7650 and TS7650G Problem Determination and Service Guide

239 Removing and replacing disk expansion unit FRUs This topic provides the steps for replacing any FRU in the disk expansion unit. Before you begin Attention: Before upgrading the ProtecTIER code level, you must update the DS4700 storage subsystem firmware, NVSRAM firmware, and the ESM firmware on your EXP810 expansion drawers. You must call IBM Service to schedule a time to update the firmware to the levels listed below. Do not proceed to upgrade the ProtecTIER code level to version 2.4.x until: v The storage subsystem firmware has been upgraded to the minimum level of or the recommended level of v The DS4700 NVSRAM firmware has been updated to the minimum level of N1814D47R1050V07 or the recommended level of N1814D47R1060V04. v The expansion unit ESM firmware has been updated to the minimum level of 98C5 or the recommended level of 98D0. Procedure 1. It is necessary to power off the server (or servers) before starting work on the DS controller. Refer to Power procedures for ProtecTIER version 3.1 or later on page 237 or Powering off the ProtecTIER server version 2.5 or earlier on page Identify the location of the FRU to be replaced and follow the replacement procedures for the FRU. Refer to the IBM TotalStorage DS4000 EXP810 Storage Expansion Unit Installation, User's and Maintenance Guide. 3. Once the location has been identified, use the IBM TotalStorage DS4000 EXP810 Storage Expansion Unit Installation, User's and Maintenance Guide for replacement procedures. 4. Power on the server (or servers), refer to Power procedures for ProtecTIER version 3.1 or later on page 237 or Powering on the ProtecTIER server version 2.5 or earlier on page Referring to the IBM TotalStorage DS4000 EXP810 Storage Expansion Unit Installation, User's and Maintenance Guide, perform any checkout procedures. Results The process of removing and replacing disk expansion module FRUs is complete. Removing and replacing TSSC FRUs Use the following steps to replace any FRU in the system console. Before you begin v The TSSC must be removed from the rack in order to service it. v Because there is no cable management arm standard with the TSSC, all cables must be unplugged before removing it from the rack in order to service it. Ensure that all cables are properly tagged with their locations prior to disconnecting. Chapter 10. FRU replacement for TS7600 systems 215

240 Procedure 1. Identify the location of the FRU to be replaced and follow the replacement procedures for the FRU. Refer to the IBM System Storage TS3000 System Console (TSSC) Maintenance Information, included on the documentation CD. 2. After the location has been identified, use the IBM System Storage TS3000 System Console (TSSC) Maintenance Information for replacement procedures. 3. Using the IBM System Storage TS3000 System Console (TSSC) Maintenance Information as a reference, perform any checkout procedures. Removing and replacing Ethernet switch FRUs The topics in this section describe how to remove and replace Ethernet switch field-replaceable units (FRUs). In all TS7650 Appliance installations, and in ProtecTIER server best practices installations, the Ethernet switches are FRUs. If an Ethernet switch fails, it must be replaced. Note: If you are required to remove either switch, refer to Removing the Ethernet switch. Removing the Ethernet switch Perform this task to remove the Ethernet switch. About this task Note: Configuration might vary for 3958 DD3 installations. Procedure In all ProtecTIER server installations and in IBM best practices installations, perform the following steps to remove the Ethernet switch. 1. Identify the switch that you need to remove. 2. Remove the power cord from the power receptacle on the Ethernet switch. See Figure 93 on page Using a Phillips head screwdriver, remove the screw 1 from the rails from both sides of the switch. 4. Slide the defective network switch out of the frame (Figure 93 on page 217). 216 TS7650 and TS7650G Problem Determination and Service Guide

241 ts Figure 93. Removing the Ethernet switch (3958 AP1 installations) 5. Remove side brackets and power cord. Save all screws and power cord for reuse. See Figure 93. Reinstalling Ethernet switches To reinstall the Ethernet switch, follow the procedure for the specific switch that you are replacing. Table 36 lists the reinstallation procedures. Table 36. Ethernet switch reinstallation procedures Switch manufacturer Procedure Alpha Networks Reinstalling the Alpha Networks Ethernet switch SMC Reinstalling the SMC Ethernet switch on page 218 Reinstalling the Alpha Networks Ethernet switch Complete the task in this topic to reinstall the Alpha Networks Ethernet switch. Procedure Perform the following steps to complete the reinstallation of the Alpha Networks Ethernet switch. Note: Use caution during this process to avoid dislodging or damaging the cabling within the frame. 1. To attach the mounting brackets and power cords to the Ethernet switches, perform the following steps: a. Attach the mounting brackets to the short-sides of the Ethernet switch (Figure 94 on page 218), making sure to align the perforated fan vents on the switch with the cut-out areas on the brackets. b. Thread the power cord through the holes in the mounting bracket, making sure that the tab extensions on the power cord are positioned on the outside of the bracket. Chapter 10. FRU replacement for TS7600 systems 217

242 c. Attach a protective grommet to the power cord, and fit the grommeted part of the cord into the retention slot on the top of mounting bracket. d. Plug the power cord into the power receptacle on the Ethernet switch. e. Insert screws through the tab extensions on the power cord and into the mounting bracket. Tighten the screws to attach the power cord to the bracket ts Figure 94. Installing the Ethernet switch (3958 AP1 installations) 2. To install the Ethernet switches into the air duct housing, perform the following steps (working from the back side of the frame): a. Carefully slide one of the Ethernet switches into the air duct housing at position 3 on the frame, with the Ethernet ports facing outward (Figure 94). b. Align the top hole in the mounting bracket on the Ethernet switch with the guide pins on the rails. c. Insert a short flanged screw into the bottom holes on each side of the mounting bracket, and tighten the screws to secure the Ethernet switch to the rails. d. Repeat the steps above for the other Ethernet switch, if applicable. e. Insert the power cord into the power receptacle on the Ethernet switch to reapply power. Reinstalling the SMC Ethernet switch Complete the task in this topic to reinstall the SMC Ethernet switch. Procedure Perform the following steps to complete the reinstallation of the SMC Ethernet switch. Note: Use caution during this process to avoid dislodging or damaging the cabling within the frame. 1. From the rear of the frame, route the female end of the two power cables for the Ethernet switch through the oblong opening on the right side (as you face it from the rear) of the frame (Figure 95 on page 219). Perform this action now 218 TS7650 and TS7650G Problem Determination and Service Guide

243 to ease connection of the cable to the switch after it is installed. ts Figure 95. Ethernet switch power cable routing into the frame 2. Install the right mounting rail 1 and the left mounting rail 2 in the frame using one screw 3 each (Figure 96) ts Figure 96. Rear of rack, showing installation of the Ethernet switches 3. From the front of the frame, route the female end of the Ethernet power cable from step 1 on page 218 through square hole 12 in the left mounting rail Chapter 10. FRU replacement for TS7600 systems 219

244 2. Consider removing the left side panel of the frame, if possible, to perform this task more easily. For example, Figure 97 shows the power cables routed through the mounting rail in an installation with the frame side panel removed and the Ethernet switches already installed. ts Figure 97. Ethernet switch power cable routing through the mounting rail 4. Attach the left and right mounting brackets 4 and 5 to the Ethernet switch 9 with four screws 10 each. 5. Attach the left side of the switch 9 to the left mounting rail 2 using one screw 11 through the mounting bracket Attach the right side of the switch 9 to the right mounting rail 1 using one screw 11 through the mounting bracket 4. Note: The switch might hang slightly below the bottoms of the mounting rails. 7. From the front of the frame, connect the Ethernet switch power cable to the Ethernet switch. 8. To configure the newly installed switch, access the ProtecTIER Service menu (see ProtecTIER Service menu on page 16). 9. Select ProtecTIER Configuration. The ProtecTIER Configuration menu is displayed (Figure 98 on page 221). 220 TS7650 and TS7650G Problem Determination and Service Guide

245 ProtecTIER Service Menu running on rasddx ProtecTIER Configuration (...) ) Configure ProtecTIER node 2) Recover Configuration for a replaced server 3) Configure machine serial number for a replaced server 4) Configure RAS 5) Update Time, Date, Timezone & Timeserver(s) 6) File Systems Management (...) 7) Configure replication (...) 8) IP Network configuration (...) 9) Update Firmware 10) Update the System s name 11) Validate configuration B) Back E) Exit >>> Your choice? Figure 98. ProtecTIER Configuration menu 10. Select IP Network configuration. The IP Network configuration menu is displayed (Figure 99) ProtecTIER Service Menu running on rasddx ProtecTIER Configuration (...) IP Network Configuration (...) ) Configure hostname 2) Configure ProtecTIER s IP interfaces 3) Configure Static Routes 4) Configure cluster Ethernet switches B) Back E) Exit >>> Your choice? Figure 99. IP Network configuration menu 11. Select Configure cluster Ethernet switches. 12. Follow the on-screen instructions to complete the replacement of the switch (Figure 100). Note: This is a non-concurrent procedure. Begin Processing Procedure Collecting system information [ Done ] note: This action will reconfigure the switches note: cluster & VTFD services on this node must be stopped in order to continue Do you wish to continue? (yes no) Figure 100. On-screen instructions for Ethernet switch configuration Removing and replacing the Western Telematic Inc. switch To remove and replace the Western Telematic Inc. (WTI) switch, perform the following steps. Chapter 10. FRU replacement for TS7600 systems 221

246 Procedure 1. Before removing the old WTI switch, issue a poweroff command to the upper ProtecTIER server Node B. 2. Wait until the upper ProtecTIER Node B is powered off, then issue a poweroff command to the lower ProtecTIER server Node A. 3. On a sheet of paper, record the following information for each power cable attached to both ProtecTIER nodes: v The label of the power cable. v The WTI port where the cable is connected (Figure 101). WTI Network Power Switch TJ8 BJ8 BJ5 BJ4 P P E Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) TSSC (1u) TSSC Network Switch & KVM (1u) Top Power Distribution Unit (PDU) TJ5 TJ DD PCS 3850 M2 (4u) Empty (1u) 3958 DD PCS 3850 M2 (4u) Bottom Power Distribution Unit (PDU) BJ1 Empty (1u) 1 GB Ethernet Switch (1u) 1 GB Ethernet Switch (1u) Empty (1u) Empty (1u) TJ1 ts Figure 101. Clustered power connections 4. Remove the old WTI switch. 5. Install the new WTI power device and connect it in exactly the same way as the old WTI. 222 TS7650 and TS7650G Problem Determination and Service Guide

247 v Connect port 1 to the power supply 1 of Node A v Connect port 5 to the power supply 2 of Node A v Connect port 2 to the power supply 1 of Node B v Connect port 6 to the power supply 2 of Node B v Connect the ETH of WTI to the internal switch 2 port 3. Ethernet switch P Ethernet switch P WTI Network Power Switch 3958 Server B P P R S U E1 E2 A V P P Server A 7 8 E R S U E1 E2 A V P P ts Figure 102. Clustered 1 Gb Ethernet switch Ethernet connections (3958 DD1/3958 DD3) From On device To On device/location Port 1 Ethernet switch 2 Port 1, slot 4 Server A Port 2 Ethernet switch 2 Port 1, slot 4 Server B Port 3 Ethernet switch 2 Ethernet port, E WTI network power switch Port 4 Ethernet switch 2 Port 4 Ethernet switch 1 Port 1 Ethernet switch 1 Port 2, slot 3 Server A Port 2 Ethernet switch 1 Port 2, slot 3 Server B Chapter 10. FRU replacement for TS7600 systems 223

248 Ethernet switch 2 P Ethernet switch 1 P WTI network power switch P P A1 A2 A3 A4 B1 B2 B3 B4 E 4 3 ProtecTIER Server B ProtecTIER Server A 8 8 Replication 1 P1 A1 A2 P2 B V S E1 E3 E2 E Replication P1 P A1 A2 B V S E1 E Replication 1 Replication 2 Customer Network Customer Network E3 E4 ts Figure 103. Clustered 1 Gb Ethernet switch connections for VTL configuration Table 37. Clustered 1 Gb Ethernet switch Ethernet connections for VTL configuration Callout From On Device To On Device/Location 1 Slot 5, port 1 Server A Customer network Customer designated device 2 Port 1 Ethernet switch 1 Slot 5, port 2 Server A 3 Port 1 Ethernet switch 2 Port A1 Server A 4 Port 3 Ethernet switch 2 Ethernet port, E WTI network power switch 5 Port 26 Ethernet switch 2 Port 26 Ethernet switch 1 6 Slot 5, port 1 Server B Customer network Customer designated device 7 Port 2 Ethernet switch 1 Slot 5, port 2 Server B 8 Port 2 Ethernet switch 2 Port A1 Server B 19 Slot 5, port 3 Server A Customer Customer specified device replication network 20 Port A2 Server A Customer Customer specified device replication network 26 Slot 5, port 3 Server B Customer Customer specified device replication network 27 Port A2 Server B Customer replication network Customer specified device 224 TS7650 and TS7650G Problem Determination and Service Guide

249 Ethernet switch 2 P Ethernet switch 1 P WTI network power switch P P A1 A2 A3 A4 B1 B2 B3 B4 E ProtecTIER Server B P1 A Replication 2 Replication 1 ProtecTIER Server A A2 P2 B V S E1 E2 E3 E4 9 6 Customer Network P1 P A1 A2 B V S E1 E Replication 2 Replication 1 1 Customer Network E3 E4 ts Figure 104. Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.1.8 or earlier) Table 38. Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.1.8 or earlier) Callout From On Device To On Device/Location 1 Port 1, slot 5 Server A Customer network Customer designated device 2 Port 1 Ethernet switch 1 Port 2, slot 5 Server A 3 Port 1 Ethernet switch 2 Port A1 Server A 4 Port 3 Ethernet switch 2 Ethernet port, E WTI network power switch 5 Port 26 Ethernet switch 2 Port 26 Ethernet switch 1 6 Port 1, slot 5 Server B Customer network Customer designated device 7 Port 2 Ethernet switch 1 Port 2, slot 5 Server B 8 Port 2 Ethernet switch 2 Port A1 Server B 15 Port 4, slot 3 Server A Customer replication Customer specified device network 16 Port 4, slot 4 Server A Customer replication Customer specified device network 17 Port 4, slot 3 Server B Customer replication network Customer specified device Chapter 10. FRU replacement for TS7600 systems 225

250 Table 38. Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.1.8 or earlier) (continued) Callout From On Device To On Device/Location 18 Port 4, slot 4 Server B Customer replication network Customer specified device Ethernet switch 2 P Ethernet switch 1 P WTI network power switch P P A1 A2 A3 A4 B1 B2 B3 B4 E ProtecTIER Server B ProtecTIER Server A P1 A1 Replication 1 P1 Replication 1 A2 18 P2 B V S E1 E2 E3 E4 9 6 Customer Network P2 A1 A2 B V S E1 E Replication 2 1 Replication 2 Customer Network E3 E4 ts Figure 105. Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.2) Table 39. Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.2) Callout From On Device To On Device/Location 1 Port 1, slot 5 Server A Customer network Customer designated device 2 Port 1 Ethernet switch 1 Port 2, slot 5 Server A 3 Port 1 Ethernet switch 2 Port A1 Server A 4 Port 3 Ethernet switch 2 Ethernet port, E WTI network power switch 5 Port 26 Ethernet switch 2 Port 26 Ethernet switch 1 6 Port 1, slot 5 Server B Customer network Customer designated device 7 Port 2 Ethernet switch 1 Port 2, slot 5 Server B 8 Port 2 Ethernet switch 2 Port A1 Server B 226 TS7650 and TS7650G Problem Determination and Service Guide

251 Table 39. Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3456 (For ProtecTIER V3.2) (continued) Callout From On Device To On Device/Location 15 Port 3, slot 5 Server A Customer replication Customer specified device network 16 Port A2 Server A Customer replication Customer specified device network 17 Port 3, slot 5 Server B Customer replication Customer specified device network 18 Port A2 Server B Customer replication network Customer specified device Ethernet switch 2 P Ethernet switch 1 P WTI network power switch P P A1 A2 A3 A4 B1 B2 B3 B4 E ProtecTIER Server B ProtecTIER Server A 8 8 Replication 1 P1 A1 A2 B V S 18 6 Replication P2 E1 E P1 P A1 A2 B V S E1 E Replication 1 Replication 2 E3 E4 Customer Network Customer Network Figure 106. Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3457 E3 E4 ts Chapter 10. FRU replacement for TS7600 systems 227

252 Table 40. Clustered 1 Gb Ethernet switch connections for OpenStorage configuration, Feature Code 3457 Callout From On Device To On Device/Location 1 Slot 5, port 1 Server A Customer network Customer designated device 2 Port 1 Ethernet switch 1 Slot 5, port 2 Server A 3 Port 1 Ethernet switch 2 Port A1 Server A 4 Port 3 Ethernet switch 2 Ethernet port, E WTI network power switch 5 Port 26 Ethernet switch 2 Port 26 Ethernet switch 1 6 Slot 5, port 1 Server B Customer network Customer designated device 7 Port 2 Ethernet switch 1 Slot 5, port 2 Server B 8 Port 2 Ethernet switch 2 Port A1 Server B 15 Slot 5, port 3 Server A Customer replication Customer specified device network 16 Port A2 Server A Customer replication Customer specified device network 17 Slot 5, port 3 Server B Customer replication Customer specified device network 18 Port A2 Server B Customer replication network Customer specified device 6. Power on the lower ProtecTIER Node A. The message Updating WTI firmware might be displayed. The update takes at least five minutes. 7. After ProtecTIER Node A is completely powered on, power on ProtecTIER Node B. Recovering the basic input/output system (BIOS) code If the BIOS code in the server has become damaged (for instance, from a power failure during an update), contact your next level of support. Updating the server firmware and basic input/output system settings Complete the task in this topic to update server firmware and basic input/output system (BIOS) settings. Before you begin The server must be completely booted up to a login prompt before proceeding with this procedure. Note: (ProtecTIER V2.3 or earlier only) Only the items in the menu are updated. To update all components in the x-system, perform the procedure in IBM System Storage TS7600 with ProtecTIER Installation Instructions for the RAS Package, BIOS, and Firmware updates following a FRU replacement for models 3958 DD1, 3958 DD3 and 3958 AP1, IBM part number 46X1339. Procedure Perform the following steps to update server firmware and BIOS settings. 1. Insert the appropriate software DVD. 228 TS7650 and TS7650G Problem Determination and Service Guide

253 ProtecTIER software version DVD 2.3 IBM System Storage RAS/BIOS and Firmware Update DVD (if not already inserted) 2.4 ProtecTIER 2.4 DVD (if not already inserted) 2.5 or later No DVD needed 2. For ProtecTIER V2.4 or earlier, type the following commands: mount /dev/hda /mnt/cdrom cd /mnt/cdrom./update_sysplanar 3. For ProtecTIER V2.5 or later, start the firmware update. v For ProtecTIER V2.5, type the following command at the command line prompt: /opt/dtc/install/ptconfig -updatefirmwares v For ProtecTIER V3.1 or later, select ProtecTIER Configuration > Update Firmware from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16). The firmware update takes several minutes to complete. All defined components are checked and updated to supported levels. Results The firmware update process is complete. Return to the procedure that sent you here or you can check the system health to verify that everything is OK. Example For both 7141 and 7233 the updated firmware levels are as follows: v ibm_fw_fpga linux_i386.sh (Field Programmable Gate Array) v brcm_fw_nic linux_32-64.bin (Sysplanar Broadcom Ethernet firmware update) v ibm_fw_dsa linux_32_64.sh (Dynamic System Analysis) Notes: 1. The Broadcom firmware update script updates only active ports. Port 1 is used for the replication feature. Replication configuration is required before the firmware for Port 1 is updated. Port 2 is used for connection to the TS3000 System Console (TSSC). 2. Installation of the Reliability, Availability, and Serviceability (RAS) package is also required before firmware for Port 2 is updated. Updating the firmware of an MR10k MegaRAID or ServeRAID M5015 card If the MR10k MegaRAID card has been replaced and the firmware is not at the current level, you must update the firmware to current supported field levels. Before you begin The mrdump script is provided to gather the MegaRAID logs needed by IBM System x support to help diagnose problems with the MR10k card and battery. Enter the following commands on the command line: Chapter 10. FRU replacement for TS7600 systems 229

254 cd /opt/megaraid/megacli./mrdump.sh The script creates two log files (mrlog.log and MegaSAS.log) in the same directory for support. The ProtecTIER software V2.4.0 or later installs the MegaCli during the autorun command. 1. Check whether the /opt/megaraid/megacli/mrdump.sh command is present. If it is not, you must install the MegaCli commands for ProtecTIER software V2.3 and below. 2. Display the current firmware versions. v For ProtecTIER software V3.1 or later, from the ProtecTIER Service menu, select Version Information > Display Firmware Levels (see ProtecTIER Service menu on page 16). v For ProtecTIER software V2.4 or V2.5, from the ProtecTIER Service menu, select Display Firmware Versions. v For ProtecTIER software V2.3 or earlier, from a command line type versions. If the command is not found, type the following commands to mount the IBM System Storage RAS/BIOS and Firmware Update DVD and display the firmware versions: mount /dev/hda /mnt/cdrom cd /mnt/cdrom./versions Make a note of the displayed levels and compare them to the levels listed in this document. 3. Update the MR10K firmware. v ProtecTIER V2.3 or earlier: If any firmware displayed is lower than the recommended level, use the Reliability, Availability, and Serviceability (RAS) package version 3.2 or later and follow the instructions in this section to update that component. If the firmware level is current, you can skip the firmware update. v ProtecTIER V2.4 or later: Type the following command: /opt/dtc/install/ptconfig -updatefirmwares The update takes several minutes to complete. All components are updated to supported levels About this task The MR10k MegaRAID card provides battery backup of the hard disk drive (HDD) cache data. This card also provides the mirroring of the two HDDs used for the operating system (OS). See the IBM System Storage Problem Determination and Service Guide for the TS7650 ProtecTIER Deduplication Appliance and TS7650G ProtecTIER Deduplication Gateway, GA , included on the IBM System Storage TS7650 with ProtecTIER Publications CD, for more information. This procedure is used to update the firmware on the MR10k after a hardware replacement of the adapter. Procedure 1. Connect a universal serial bus (USB) keyboard and monitor to the server. 2. Insert the appropriate software DVD. 230 TS7650 and TS7650G Problem Determination and Service Guide

255 ProtecTIER software version DVD 2.3 IBM System Storage RAS/BIOS and Firmware Update DVD (if not already inserted) 2.4 ProtecTIER 2.4 DVD (if not already inserted) 2.5 or later No DVD needed 3. Log with the ID root and the password admin. 4. From the command line, enter the following commands to start the firmware load process: mkdir /mnt/cdrom Note: If the /mnt/cdrom directory already exists, the following message is displayed: mkdir: cannot create directory /mnt/cdrom : File exists Ignore the message and proceed to the next command. mount /dev/hda /mnt/cdrom cd /mnt/cdrom 5. Enter the following command to update the MR10k MegaRAID card../installmrfw The updated firmware level is as follows (ServerRAID MR10k): ibm_fw_sraidmr_10k _linux_32-64.bin Note: The version listed is subject to change. Download the latest version as necessary. 6. Enter the following command to update the MegaRAID command line utilities:./installmrcli 7. After a successful update of the MRCli, eject the IBM System Storage RAS/BIOS and Firmware Update DVD by entering the following commands: cd / eject /dev/hda Results The process of replacing the MR10k MegaRAID card and updating the firmware is complete. What to do next If you are performing any other field replaceable unit (FRU) or firmware updates, proceed to the appropriate section. Otherwise, go to Chapter 12, End-of-call procedure, on page 249. Updating the SAS hard disk drive firmware Complete the task in this topic to update the firmware for a serial attached SCSI (SAS) hard disk drive. Before you begin Check the current version levels by completing the following steps: Chapter 10. FRU replacement for TS7600 systems 231

256 1. If the IBM System Storage RAS/BIOS and Firmware Update DVD is not already in the CD/DVD drive of the 3958 server, insert it into the CD/DVD drive. 2. Log in with the ID root and the password admin. 3. Display the current firmware versions. v v v For ProtecTIER software V3.1 or later, from the ProtecTIER Service menu, select Version Information > Display Firmware Levels (see ProtecTIER Service menu on page 16). For ProtecTIER software V2.4 or V2.5, from the ProtecTIER Service menu, select Display Firmware Versions. For ProtecTIER software V2.3 or earlier, from a command line type versions. If the command is not found, type the following commands to mount the IBM System Storage RAS/BIOS and Firmware Update DVD and display the firmware versions: mount /dev/hda /mnt/cdrom cd /mnt/cdrom./versions The output is similar to that shown in Figure 63 on page 173. Make a note of the displayed levels and compare them to the levels listed in this document. 4. If any firmware displayed is lower than the recommended level, use the appropriate method and follow the instructions in this section to update that component. If the firmware level is up to date, you can skip the firmware update. About this task If one or more of these parts has been replaced and the firmware is not at the current level, the following procedure must be performed to update the firmware to current supported field levels. Procedure 1. Connect a universal serial bus (USB) keyboard and monitor to the server. 2. Update the firmware. v For ProtecTIER software V2.4 or later, type the following command at a command line: /opt/dtc/install/ptconfig -updatefirmwares When this process completes, go to step 10 on page 233. v For ProtecTIER software V2.3 or earlier, type versions at a command line. If the command is not found, perform the following steps. 3. If the IBM System Storage RAS/BIOS and Firmware Update DVD is not already installed in the CD/DVD drive of the 3958 server, insert it into the CD/DVD drive. 4. Log with the ID root and the password admin. 5. From the command line, enter the following commands to start the firmware load process: mkdir /mnt/cdrom Note: If the /mnt/cdrom directory already exists, the following message is displayed: mkdir: cannot create directory /mnt/cdrom : File exists Ignore the message and proceed to the next command. 232 TS7650 and TS7650G Problem Determination and Service Guide

257 mount /dev/hda /mnt/cdrom cd /mnt/cdrom 6. Enter the following command to update the SAS hard disk drive. The level of the basic input/output system (BIOS) or firmware is in bold. This value is the result shown on the BIOS screens during the update process. Note: The SAS hard disk update can take up to one hour to complete../installhddfw The updated firmware level is as follows: ibm_fw_hdd_sas _linux_32-64.bin Note: When updating the SAS hard disk firmware, multiple reboots might be required. Press Enter when asked to continue. Important: After the firmware updates have been applied, the system must be shut down to a standby state and restarted for them to take effect. 7. Following the successful update of the components, enter the following command from the command line to shut down the system: poweroff After the poweroff command has been entered, ignore any further error messages presented. 8. Press the power button to restart the server. 9. As needed: Remove the IBM System Storage RAS/BIOS and Firmware Update DVD as soon as the CD/DVD drive allows. After the system completes the boot cycle, verify that no errors are reported on screen or from the error indicators on the light path display panel. Note: After the reboot completes, a message might be presented at the login prompt that the module sctp cannot be unloaded due to unsafe usage... Ignore this message and press the Enter key to return to the login prompt. 10. Display the current firmware versions. v For ProtecTIER software V3.1 or later, from the ProtecTIER Service menu, select Version Information > Display Firmware Levels (see ProtecTIER Service menu on page 16). v For ProtecTIER software V2.4 or V2.5, from the ProtecTIER Service menu, select Display Firmware Versions. v For ProtecTIER software V2.3 or earlier, from a command line type versions. If the command is not found, type the following commands to mount the IBM System Storage RAS/BIOS and Firmware Update DVD and display the firmware versions: mount /dev/hda /mnt/cdrom cd /mnt/cdrom./versions Results The process of replacing the SAS hard disk drive and updating the firmware is complete. Chapter 10. FRU replacement for TS7600 systems 233

258 What to do next Configuring the server If you are performing any other field replaceable unit (FRU) or firmware updates, proceed to the appropriate section. Otherwise, go to Chapter 12, End-of-call procedure, on page 249. Detailed information about configuring the server is in the IBM System x3850 M2 and System x3950 M2 Type 7141 and 7233 User's Guide. The Configuration/Setup Utility program is part of the basic input/output system (BIOS) code on the server. You can use this utility to change the remote supervisor adapter (RSA) static IP address. This might be required during TS7650 Appliance or TS7650G installation and after replacing the RSA with a new field replaceable unit (FRU). Starting the Configuration/Setup Utility program Complete the following steps to start the Configuration/Setup Utility program. Procedure 1. Log in to the RSA using the procedure in Appendix A, Logging into the 3958 DD1/3958 DD3 server through the RSA connection, on page 255. the System x logo screen shown in Figure 107 is displayed. cvt00029 Figure 107. System x logo screen for BIOS setup 2. Turn on or restart the server and press the F1 key. 3. Scroll down until the selection menu is visible. 4. Select Settings to view or change. 234 TS7650 and TS7650G Problem Determination and Service Guide

259 Verifying cable connections Fibre Channel cables from the disk controller to the server come pre-configured from the factory. For more information, see the IBM System Storage Labeling Instructions for the TS7650 ProtecTIER Deduplication Appliance and TS7650G ProtecTIER Deduplication Gateway, PN 95P8942. Chapter 10. FRU replacement for TS7600 systems 235

260 236 TS7650 and TS7650G Problem Determination and Service Guide

261 Chapter 11. Power off and power on procedures This section provides procedures for turning on or off all components of the ProtecTIER server. Power procedures for ProtecTIER version 3.1 or later This section provides procedures for turning on or off all the components of the ProtecTIER server V3.1 or later. Powering off a stand-alone TS7650 server version 3.1 or later About this task This is the recommended manual power off sequence for a single node (stand-alone TS7650 server) TS7650G or TS7650 Appliance server. Procedure 1. Ask the customer to suspend I/O to the server in the TS7650G or TS7650 Appliance. Indicate below when the customer reports that I/O is suspended. Customer reports that all I/O has been suspended to the server U 2. If the TS7650G or TS7650 Appliance does not use a TSSC, attach a keyboard and monitor to the TS7650 server. Log in with the ID ptconfig and the password ptconfig to access the ProtecTIER Service Menu. 3. From the ProtecTIER Service Menu, select the Manage ProtecTIER services option ProtecTIER Service Menu running on rasddx ) ProtecTIER Configuration (...) 2) Manage ProtecTIER services (...) 3) Health Monitoring (...) 4) Problem Alerting (...) 5) Version Information (...) 6) Generate a service report 7) Generate a system view 8) Update ProtecTIER code E) Exit >>> Your choice? 4. From the Manage ProtecTIER services menu, select Stop all services. Copyright IBM Corp. 2011,

262 ProtecTIER Service Menu running on rasddx Manage ProtecTIER Services (...) ) Display services status 2) Start all services 3) Stop all services 4) Stop ProtecTIER services only (including GFS) 5) Stop VTFD service only B) Back E) Exit >>> Your choice? When the services are stopped, the system shows the following messages: Stopping ptrasd [ Done ] Stopping vtfd [ Done ] Stopping ptcluster [ Done ] 5. Press Enter to return to the ProtecTIER Service Menu. 6. Power off the server. v ProtecTIER V3.1.8 or earlier: a. Type E to exit from the ProtecTIER Service Menu. b. Log in to the server with the user ID root, and the password admin. c. From the server command line, type poweroff and press Enter. v ProtecTIER V3.2: a. From the ProtecTIER Service Menu, select Manage ProtecTIER Services. b. In the Manage ProtecTIER Services menu, select Poweroff This Node ProtecTIER Service Menu running on rasddx Manage ProtecTIER Services (...) ) Display services status 2) Start all services 3) Stop all services 4) Stop ProtecTIER services only (including GFS) 5) Stop VTFD service only 6) Poweroff This Node 7) Reboot This Node B) Back E) Exit >>> Your choice? When the power off processes complete, the Power LED on the server's front panel blinks steadily. This indicates that server is in standby mode. 7. Power off all attached storage controllers that are located in the frame with the server, or in a separate storage frame. The control units must be powered off before the expansion drawers. If multiple controllers are present, power them all off. CAUTION: See storage controller/expansion unit documentation for detailed power off instructions to prevent damage to the controllers and expansion drawer units. 8. When the controllers have finished powering off, power off all expansion units. 9. Shut down the TSSC. To do so: 238 TS7650 and TS7650G Problem Determination and Service Guide

263 a. Log in to the TSSC with the username: service and password: service. b. Right-click on the TSSC's blue desktop. The TS3000 System Console Menu displays. c. Select Shutdown Console. d. A dialog box appears with this question: Shutdown the machine? <OK> <Cancel> e. Select OK to start the shutdown process. f. The shut down process is complete when you see this: It is now safe to turn off the TS3000 System Console Hold down the power button for 5 seconds to power off. Press the power button, it is not necessary to hold the button down. g. When the power off process is complete, the power LED on the server's front panel will blink steadily. This indicates that the server is in standby mode. 10. When all relevant components in the frame (or frames) are powered off, power off the frame (or frames) using the Unit Power On/Off (UPO) switch or the customer's circuit breaker. Powering off a TS7650 cluster version 3.1 or later About this task This is the recommended manual power off sequence for a TS7650G or TS7650 Appliance cluster. Procedure 1. Ask the customer to suspend I/O to the servers in the TS7650G or TS7650 Appliance. Indicate below when the customer reports that I/O is suspended. Customer reports that all I/O has been suspended to the servers U 2. If the TS7650G or TS7650 Appliance uses a KVM, press the PrtSc key on the KVM to select the node you are going to access. If the TS7650G or TS7650 Appliance does not use a KVM, then use an external monitor and keyboard to access the nodes. 3. On Node A (the bottom server) attach a keyboard and monitor to the server. Log in with the ID ptconfig and the password ptconfig to access the ProtecTIER Service Menu. 4. From the ProtecTIER Service Menu, select the Manage ProtecTIER services option. Chapter 11. Power off and power on procedures 239

264 ProtecTIER Service Menu running on rasddx ) ProtecTIER Configuration (...) 2) Manage ProtecTIER services (...) 3) Health Monitoring (...) 4) Problem Alerting (...) 5) Version Information (...) 6) Generate a service report 7) Generate a system view 8) Update ProtecTIER code E) Exit >>> Your choice? 5. From the Manage ProtecTIER services menu, select Stop all services ProtecTIER Service Menu running on rasddx Manage ProtecTIER Services (...) ) Display services status 2) Start all services 3) Stop all services 4) Stop ProtecTIER services only (including GFS) 5) Stop VTFD service only B) Back E) Exit >>> Your choice? When the services are stopped, the system shows the following messages: Stopping ptrasd [ Done ] Stopping vtfd [ Done ] Stopping ptcluster [ Done ] 6. Press Enter to return to the ProtecTIER Service Menu. 7. Power off the server. v ProtecTIER V3.1.8 or earlier: a. Type E to exit from the ProtecTIER Service Menu. b. Log in to the server with the user ID root, and the password admin. c. From the server command line, type poweroff and press Enter. v ProtecTIER V3.2: a. From the ProtecTIER Service Menu, select Manage ProtecTIER Services. b. In the Manage ProtecTIER Services menu, select Poweroff This Node ProtecTIER Service Menu running on rasddx Manage ProtecTIER Services (...) ) Display services status 2) Start all services 3) Stop all services 4) Stop ProtecTIER services only (including GFS) 5) Stop VTFD service only 6) Poweroff This Node 7) Reboot This Node B) Back E) Exit >>> Your choice? 240 TS7650 and TS7650G Problem Determination and Service Guide

265 When the power off processes complete, the Power LED on the server's front panel blinks steadily. This indicates that server is in standby mode. 8. On Node B (the top server), repeat steps 2 through 7 to power off the server. 9. Power off all attached storage controllers that are located in the frame with the server, or in a separate storage frame. The control units must be powered off before the expansion drawers. If multiple controllers are present, power them all off. CAUTION: See storage controller/expansion unit documentation for detailed power off instructions to prevent damage to the controllers and expansion drawer units. 10. When the controllers have finished powering off, power off all expansion units. 11. Shut down the TSSC. To do so: a. Press the PrtSc key on the KVM to select TSSC. b. Log in to the TSSC with the username service and password service. c. Right-click on the TSSC's blue desktop. The TS3000 System Console Menu displays. d. Select Shutdown Console. e. A dialog box appears with this question: Shutdown the machine? <OK> <Cancel> f. Select OK to start the shutdown process. g. The shut down process is complete when you see this: It is now safe to turn off the TS3000 System Console Hold down the power button for 5 seconds to power off. Press the power button, it is not necessary to hold the button down. h. When the power off process is complete, the power LED on the server's front panel will blink steadily. This indicates that server is in standby mode. 12. When all components in the frame (or frames) are powered off, power off the frame (or frames) using the Unit Power On/Off (UPO) switch or the customer's circuit breaker. Powering on a stand-alone TS7650 server version 3.1 or later About this task This is the recommended manual power on sequence for a single node (stand-alone TS7650 server) TS7650G or TS7650 Appliance server. Procedure 1. Using the frame's UPO switch or the customer's circuit breaker, restore power to the frame (or frames). 2. Power on all expansion units. 3. Power on all storage controllers. 4. Press the power button on the TSSC. 5. Log in to the TSSC with the username service and password service. 6. Press the power button on the server's front panel. Approximately 15 minutes after the server is powered-on, following the reboot, you can log in to the ProtecTIER Service Menu. Chapter 11. Power off and power on procedures 241

266 7. Access the ProtecTIER Service Menu with a monitor and keyboard plugged into the TS7650 server. Log in with the ID ptconfig and the password ptconfig. 8. From the ProtecTIER Service Menu, select the Manage ProtecTIER services option ProtecTIER Service Menu running on rasddx ) ProtecTIER Configuration (...) 2) Manage ProtecTIER services (...) 3) Health Monitoring (...) 4) Problem Alerting (...) 5) Version Information (...) 6) Generate a service report 7) Generate a system view 8) Update ProtecTIER code E) Exit >>> Your choice? 9. From the Manage ProtecTIER services menu, select Display services status ProtecTIER Service Menu running on rasddx Manage ProtecTIER Services (...) ) Display services status 2) Start all services 3) Stop all services 4) Stop ProtecTIER services only (including GFS) B) Back E) Exit >>> Your choice? 10. Wait until the display shows all the services have started. 11. The stand-alone TS7650 server power on process is complete. Powering on a TS7650 cluster version 3.1 or later About this task This is the recommended manual power on sequence for clustered TS7650G or TS7650 Appliance servers. Procedure 1. Using the frame's UPO switch or the customer's circuit breaker, restore power to the frame (or frames). 2. Power on all expansion units. 3. Power on all controllers. 4. Press the power button on the TSSC. 5. Log in to the TSSC with the username service and password service. 6. On Node A, press the power button on the server's front panel. Approximately 15 minutes after the server is powered-on, following the reboot, you can log in to the ProtecTIER Service Menu on Node A. 242 TS7650 and TS7650G Problem Determination and Service Guide

267 7. Access the ProtecTIER Service Menu with a monitor and keyboard plugged into the TS7650 server or from the KVM on the TSSC's console, using the PrtSc key to select Node A. Log on with the ID ptconfig and the password ptconfig. 8. From the ProtecTIER Service Menu, select the Manage ProtecTIER services option ProtecTIER Service Menu running on rasddx ) ProtecTIER Configuration (...) 2) Manage ProtecTIER services (...) 3) Health Monitoring (...) 4) Problem Alerting (...) 5) Version Information (...) 6) Generate a service report 7) Generate a system view 8) Update ProtecTIER code E) Exit >>> Your choice? 9. From the Manage ProtecTIER services menu, select Display services status ProtecTIER Service Menu running on rasddx Manage ProtecTIER Services (...) ) Display services status 2) Start all services 3) Stop all services 4) Stop ProtecTIER services only (including GFS) B) Back E) Exit >>> Your choice? 10. Wait until the display shows that all services have started on Node A before powering-on Node B. Note: If Node B is powered-on before Node A is completely online, a fence error might occur. This will cause both servers to reboot to synchronize, and they might reboot more than once. 11. Repeat steps 6 through 10 to power on Node B. Approximately 15 minutes after the server is powered on, following the reboot, log in to the ProtecTIER Service Menu on Node B. Use the ProtecTIER Service Menu to ensure that all the services are up and running on both nodes. 12. The TS7650 cluster startup is complete. Power procedures for ProtecTIER version 2.5 or earlier This section provides procedures for turning on or off all the components of the ProtecTIER server V2.5 or earlier. Powering off the ProtecTIER server version 2.5 or earlier Complete the task in this topic to turn off all the components of the ProtecTIER server V2.5 or earlier. Chapter 11. Power off and power on procedures 243

268 This section provides tasks for turning off all components of the TS7650 Appliance or the TS7650G in both stand-alone and clustered installations. This section also provides information about the following related topics. v Performing an emergency shutdown on page 247 v Additional power information on page 247 To turn off all components in the TS7650 Appliance or the TS7650G in both stand-alone and clustered installations, perform the following steps: 1. Ensure that no drive fault light-emitting diodes (LEDs) are lit on all disk components. If any drive fault LEDs are lit, correct the fault before continuing. 2. Ask the customer to suspend I/O to the server in the TS7650 Appliance or the TS7650G in both stand-alone and clustered installations. Indicate below when the customer reports that I/O is suspended. Customer reports that all I/O has been suspended to the server U 3. Turn off the server by performing the following substeps: a. Open a secure shell (SSH) session with the server using one of the following methods: Using Terminal on the system console: 1) Right-click the system console desktop and then click Terminal. 2) At the Terminal command prompt, enter the command tsys and select the system to which you want to connect or type the following command: ssh ptadmin@xxx.xxx.xxx.xxx where xxx.xxx.xxx.xxx is the server IP address, and then press the Enter key. 3) At the password prompt, type ptadmin and then press the Enter key. 4) Go to step 3b. Using a direct connection through the attached universal serial bus (USB) keyboard and monitor: 1) Connect a USB keyboard and monitor to the back of the server. 2) At the login prompt, log in with the user ID root and the password admin. 3) At the command prompt, type the following command: poweroff 4) Go to step 4. b. At the SSH command prompt, type the following command: poweroff This action dismounts the file system and shuts down the server to standby mode. 4. Ensure that the server has shut down to standby mode by verifying that the power-on LED on the operator information panel on the front of the server is flashing. See Figure 108 on page TS7650 and TS7650G Problem Determination and Service Guide

269 Power-control button/power-on LED Ethernet icon LED Information LED System-error LED 1 2 Power-control button cover 5. If in a two-node environment, repeat steps 3 on page 244 and 4 on page 244 for the other server in the node. 6. Turn off both power supply switches on the back of the disk controller or controllers. 7. Turn off both power switches on the back of all disk expansion modules connected to the disk controller or controllers. 8. Verify that no external systems are connected to frame power supplies. 9. Set the Emergency Power Off (EPO) switch on the front of the frame or rack to the OFF ( V ) position. Powering on the ProtecTIER server version 2.5 or earlier Complete this task to power on the ProtecTIER server V2.5 or earlier. About this task This section provides tasks for turning on all components of the TS7650 Appliance or the TS7650G in both stand-alone and clustered installations. Procedure Ethernet port activity LEDs Figure 108. Server operator information panel Locator button/locator LED To turn on all components in the TS7650 Appliance or the TS7650G, perform the following steps: 1. Verify the following: v All data and power cables are connected to their components v All power cables are connected to a correct power outlet v Both power switches on the back of all disk controllers and disk expansion modules are turned off v All disk drive modules are located securely in place in the drive slots of the disk controller and disk expansion module enclosures 2. If necessary, set the emergency power off (EPO) switch on the front of the rack or frame to the ON ( )position. Note: 19-inch racks (Gateway) do not have an EPO switch. Set circuit breakers to ON for all power distribution units in use in the rack. Attention: You must turn on the disk expansion modules before turning on the disk controllers. The disk controllers might not recognize the correct configuration if the disk expansion modules are turned on after the disk controllers. 3. Turn on both power switches on the back of all disk expansion modules. 4. Ensure that no drive fault light-emitting diodes (LEDs) or any other amber LEDs on any disk expansion module are lit. ts Chapter 11. Power off and power on procedures 245

270 5. Wait at least 30 seconds after starting the disk expansion modules before performing step Turn on both power switches on the back of all disk controllers. 7. Ensure that no drive fault LEDs or any other amber LEDs on any disk controller are lit. 8. Press the white recessed power-control button on the operator information panel on the front of all servers. See Figure 109. Power-control button/power-on LED Ethernet icon LED Information LED System-error LED 1 2 Power-control button cover Ethernet port activity LEDs Figure 109. Server operator information panel Locator button/locator LED 9. Open a secure shell (SSH) session with the server using one of the following methods: Using Terminal on the system console: a. Right-click the system console desktop and then click Terminal. b. At the Terminal command prompt, type the following: ssh ptadmin@xxx.xxx.xxx.xxx where xxx.xxx.xxx.xxx is the server IP address, and then press the Enter key. c. At the password prompt, type ptadmin and press the Enter key. v If this is a 3958 DD4 server or a 3958 DD1/3958 DD3 server with ProtecTIER v2.5 or later installed, all services have been started automatically. Skip to 15 on page 247. v If this is a 3958 DD1/3958 DD3 server with ProtecTIER v2.4 or earlier, go to step 10. Using a direct connection via an attached universal serial bus (USB) keyboard and monitor: a. Connect a USB keyboard and monitor to the back of the server. b. At the login prompt, log in with the user ID root and the password admin. v If this is a 3958 DD4 server or a 3958 DD1/3958 DD3 server with ProtecTIER v2.5 or later installed, all services have been started automatically. Skip to step 15 on page 247. v If this is a 3958 DD1/3958 DD3 server with ProtecTIER v2.4 or earlier, go to step At the SSH prompt, type the following command: chkconfig ptcluster on chkconfig vtfd on 11. Set the following services to ON by typing the following commands: chkconfig ptcluster on chkconfig vtfd on ts TS7650 and TS7650G Problem Determination and Service Guide

271 12. Restart the following services by typing the following commands: service ptcluster start service vtfd start 13. At the SSH prompt, type exit and then press the Enter key to close the SSH session. 14. On the ProtecTIER Manager workstation, launch ProtecTIER Manager by using one of the following methods (as applicable): v On a Windows-based PC, click Start > All Programs > IBM > ProtecTIER Manager > IBM ProtecTIER Manager. v On a Linux-based PC, double-click the ProtecTIER Manager icon on the Linux desktop. Note: If you do not have the ProtecTIER Manager icon on your Linux desktop, navigate to the directory where you installed ProtecTIER Manager (the default location is /opt/ibm/ptmanager) and double-click the ProtecTIER Manager icon. 15. Turn system control back over to the customer. Performing an emergency shutdown Complete the task in this topic to perform an emergency shut down of the ProtecTIER server. About this task This topic describes how to shut down the TS7650 Appliance or TS7650G in case of an emergency. Attention: Emergency situations might include fire, flood, extreme weather conditions, or other hazardous circumstances. If a power outage or emergency situation occurs, always turn off all power switches on all computing equipment. This helps to safeguard your equipment from potential damage due to electrical surges when power is restored. If the disk controller or disk expansion module loses power unexpectedly, it might be due to a hardware failure in the power system or in the midplane of the disk controller or disk expansion module. Additional power information Procedure 1. Stop all activity. 2. Check all of the light-emitting diodes (LEDs). Make a note of any Fault LEDs that are lit so that you can correct the problem when you turn on the power. 3. Turn off the emergency power off (EPO) switch and back-end disk repository. Important: 19-inch racks (Gateway) do not have an EPO switch. You can only power off by toggling the circuit breakers on the power distribution units. 4. Unplug the power cables from the disk controller and disk expansion module. This topic provides additional power information. Chapter 11. Power off and power on procedures 247

272 The TS7650 Appliance and the TS7650G support simultaneous power on to the system components. Always follow the power on sequence in Power procedures for ProtecTIER version 3.1 or later on page 237 or Powering on the ProtecTIER server version 2.5 or earlier on page 245 during any attended power on procedure. In an optimal state, the disk controllers and disk expansion modules should recover automatically from an unexpected shutdown and unattended simultaneous restoration of power to system components. After power is restored, call IBM support if any of the following conditions occur: v The disk component logical drives and arrays are not displayed in Storage Manager. v The disk component logical drives and arrays do not come online. v The disk component logical drives and arrays seem to be degraded. Note: Always wait at least 30 seconds between the time that you turn off a power switch and the time that you turn on the power again. 248 TS7650 and TS7650G Problem Determination and Service Guide

273 Chapter 12. End-of-call procedure Before handing the system back to the customer, finish the call by checking the following items on both nodes. Procedure 1. Check the ProtecTIER Manager alerts and clear any open alerts. To check the ProtecTIER Manager for alerts, select the node on which you want to check alerts and click on the Alerts button. The Alerts window is displayed. Take appropriate action to clear any open alerts or contact IBM Software Support for assistance. Repeat for the second node, if applicable. 2. Check the remote supervisor adapter (RSA) log (or the IMM log, for 3958 DD4 servers) for problems and clear the log. You can check the RSA log from a browser or a Telnet session. To check the RSA Adapter/IMM Log from a browser, complete the following steps: a. At the TS3000 System Console (TSSC), log in with user ID service and password service. b. Right-click on the blue desktop and from the menu, select Browser Functions > Launch Browser. c. In the URL field, enter the TSSC IP address of the RSA adapter, which is the assigned frame number ID+1. Example: If the frame number in the attached systems list is 10 ( ), then the RSA IP address is d. When prompted for a username and password, enter the username USERID and the password PASSW0RD (the 0 is a zero). e. Select the timeout value to 10 minutes and click Continue. f. Under Monitors, select Event Log. g. Review the log for error entries (a red E), or warnings (a yellow W) and verify that the notifications are valid. h. After the problems have been identified, if the fix has been implemented, scroll to the bottom of the log and choose either the option to save the log as a text file (recommended in case it is needed later), or to clear the log. If you save the log as a text file, clear the log after saving. i. After the log has been cleared, from the left navigation pane, select Log Off. Close the browser window. j. Continue to the next step. To check the RSA/IMM log from a Telnet session, complete the following steps: a. Telnet to the IP Address of the RSA Adapter from the TSSC or a server node. The IP addresses are x1 for Node A or x6 for Node B, where x is the Frame/Subsystem number. ap1tssc:~ # telnet Trying Connected to Escape character is ^]. username: h b. Log in with user ID USERID and password PASSW0RD (the 0 is a zero). Copyright IBM Corp. 2011,

274 username: USERID password: ******** SN# K116191D0S0> h c. Enter the command readlog -f. The first five entries in the list of logged errors are displayed. SN# K116191D0S0> readlog -f 1 I SERVPROC 07/22/10 00:43:28 Remote Login Suc cessful. Login ID: USERID CLI authenticated from (Telnet). 2 I SERVPROC 07/22/10 00:42:11 Remote Login Suc cessful. Login ID: USERID CLI authenticated from (Telnet). 3 I SERVPROC 07/21/10 09:53:48 Remote Login Suc cessful. Login ID: Service CLI authenticated from (Telnet). 4 I SERVPROC 07/21/10 09:53:46 Remote Login Suc cessful. Login ID: Service CLI authenticated from (Telnet). 5 I SERVPROC 07/21/10 09:53:44 Remote Login Suc cessful. Login ID: Service CLI authenticated from (Telnet). SN# K116191D0S0> h d. To display the next five entries, type the command readlog. Repeat until you reach the end of the list. e. If all displayed errors were resolved, clear the log by typing the command clearlog. SN# K116191D0S0> clearlog ok SN# K116191D0S0> readlog (There are no more entries in the event log.) SN# K116191D0S0> h 3. Use the ProtecTIER Service menu (see ProtecTIER Service menu on page 16) to check for open problems in each node. Close all open problems. To check for open problems, select Problem management from the ProtecTIER Service menu and then select List open problems from the submenu. To close (cancel) an open problem select Problem management from the ProtecTIER Service menu and then select Cancel problem from the submenu. See below for the detailed procedure: a. If this is an Appliance and a keyboard, video, mouse (KVM) is available, press the PrtSC key and select the server and press Enter. If this is a stand-alone server, connect a universal serial bus (USB) keyboard and monitor at the rear of the server. b. To check for open problems, select Problem management from the ProtecTIER Service menu and then select List open problems from the submenu. c. When asked if you want to continue, type y. d. Review any listed open problem records and verify that they have been resolved. e. To cancel a problem, record the 12-digit problem ID number. Example: ProblemRecord 2: problemid=< > From the ProtecTIER Service menu, select Problem management > Cancel problem. When prompted to continue, type y and type the 12-digit problem ID number to cancel the record. This can take several moments to complete. When prompted with a question if you are sure you want to cancel the specified problem, type y and press Enter. 250 TS7650 and TS7650G Problem Determination and Service Guide

275 Note: In ProtecTIER V2.5 or later, you are prompted for the ProblemRecord number. If so, make sure you type the one-digit ProblemRecord number, not the 12-digit problem ID number. f. After closing out the records, check for open problems again to verify that no further error records are listed. g. If you are finished, select the ProtecTIER Service menu option to Exit. See ProtecTIER Service menu on page 16 for more information about the ProtecTIER Service menu. 4. Check the TSSC Call Home Queue and clear all fixed problems. See Call Home on page Verify there are no visible failure indicators on the front or rear of the server, the light path display panel shows no failures, and any attached disk storage is free of any failure conditions: a. At the TSSC, right-click on the blue desktop and select System Console Actions > Console Configuration Utility. b. If prompted, log in with user ID service and password service. c. Select Attached Systems. From the list of attached systems, click to add a check mark in the box next to the server, then click the Update Health button. Click OK. d. When the health status is returned, verify that the state reports no problems found. e. If a Warning, Failure, orcommunications error exists, you must resolve it. To assist in identifying the fault, place a check mark next to the server again and click the button to View Health. Review the returned information assistance in the cause. If it cannot be isolated, contact your next level of support. Note: If the message "RAS Service Mode Started" displays, ignore it at this time. Service Mode is disabled in the final steps of this procedure. f. Continue to the next step. 6. Verify that both nodes are online and operational and that the filesystems are mounted: v For ProtecTIER V3.1 or later, select Manage ProtecTIER Services > Display Services Status from the ProtecTIER Service menu (see ProtecTIER Service menu on page 16). Check the status of the vtfd, cman, clvmd, and gfs services. v For ProtecTIER V2.5 or earlier, from the command line of the server, enter the following commands to verify that the services have all been started (indicated by a running state): service vtfd status service cman status service clvmd status Also, verify that the filesystems are mounted by running the following command (this is variable, but must show a list of filesystems that are mounted): service gfs status The output is similar to the examples shown in Figure 110 on page 252 and Figure 111 on page 252. Chapter 12. End-of-call procedure 251

276 /mnt/vg0-lv_vg0 /mnt/vg1-lv_vg1 /mnt/vg2-lv_vg2 /mnt/vg3-lv_vg3 /mnt/vg4-lv_vg4 /mnt/vg5-lv_vg5 /mnt/vg6-lv_vg6 /mnt/vg7-lv_vg7 /mnt/vg8-lv_vg8 /mnt/vg9-lv_vg9 /mnt/vg10-lv_vg10 /mnt/vg11-lv_vg11 Figure 110. Configured GFS mountpoints /mnt/vg0-lv_vg0 /mnt/vg1-lv_vg1 /mnt/vg2-lv_vg2 /mnt/vg3-lv_vg3 /mnt/vg4-lv_vg4 /mnt/vg5-lv_vg5 /mnt/vg6-lv_vg6 /mnt/vg7-lv_vg7 /mnt/vg8-lv_vg8 /mnt/vg9-lv_vg9 /mnt/vg10-lv_vg10 /mnt/vg11-lv_vg11 Figure 111. Active GFS mountpoints 7. After verifying all the services are operational and running, check the ProtecTIER Manager and verify that the node or nodes (if clustered) are accessible and online. This can be checked from the TSSC. Note: It is assumed that all nodes have already been established on the TSSC ProtecTIER Manager interface. If the nodes have not been established, they must be added using the customer-assigned IP address of each node, and added using the ProtecTIER Manager Add Node option: a. At the TSSC, if not already logged in, log in with user ID service and password service. b. Right-click on the blue desktop and from the menu, select Browser Functions > ProtecTIER Manager Functions > Launch GUI. c. In the left navigation pane, select the tab for Nodes. Select the server. At the login for the node, click the login button. Type the user ID ptoper and the password ptoper. Click OK. d. Once logged into the node, from the left navigation pane, select the tab for Systems. Verify that the node (or nodes if clustered) show a Status of OK, the Management Service shows as Online and the VT shows as Online. e. If the status is online and OK, close the ProtecTIER Manager. Right-click on the TSSC blue desktop and select Logout. Note: Allow up to 20 minutes for Service Mode to be disabled. 8. Disable Service Mode on the servers: a. From a command line on Node A, access the ProtecTIER Service menu by entering menu. b. From the ProtecTIER Service menu, select Health Monitoring > Service Mode. c. Type no to disable Service Mode. 252 TS7650 and TS7650G Problem Determination and Service Guide

277 d. Repeat steps 8a on page 252 thru 8c on page 252 on Node B. e. Notify the customer that the system is available. 9. If the server or servers continue to report problems or show as offline, contact your next level of support. What to do next Return control of the system to the customer. Chapter 12. End-of-call procedure 253

278 254 TS7650 and TS7650G Problem Determination and Service Guide

279 Appendix A. Logging into the 3958 DD1/3958 DD3 server through the RSA connection Complete the task in this topic to log into the 3958 DD1 or 3958 DD3 server through the remote supervisor adapter connection. Before you begin This appendix provides an alternative method for establishing a connection to the the 3958 DD1 or 3958 DD3 server during Reliability, Availability, and Serviceability (RAS) package configuration, using the remote supervisor adapter (RSA) port. This alternative method should be used only if the server connection cannot be made using a universal serial bus (USB) keyboard and graphics-capable monitor. In order to use the RSA to connect to the servers, you must use the TS3000 System Console (TSSC) (code level 5.9.x or higher required), or your IBM service laptop. If you use your laptop, you might also need an extension cord to reach a standard power outlet, or a 240 V to 110 V step-down voltage converter to allow the service laptop to run off of the power distribution unit (PDU). Important: By default, the RSA ports on both servers in a cluster have the same IP address. This causes a remote supervisor access IP address conflict when connecting to the RSA card through the TSSC network for the very first time. To avoid the conflict, you must change the IP address on one of the RSA ports so that its IP address is unique or unplug the Ethernet cable to the remote supervisor access to which you are not connecting. To do so, perform steps 2 through 11 on page 257. Change the RSA port IP address on Server B in the cluster. Do not change the default RSA port IP address on Server A. New remote supervisor access field replaceable unit (FRU) replacement's default IP address is Procedure 1. If you are using the TSSC, skip ahead to step 12 on page 257. Otherwise, continue with step 2 2. Connect an Ethernet cable from the Ethernet port on your laptop to the SYS MGMT (remote supervisor access port) on a stand-alone server, or on Server B in a cluster. See Figure 112 on page 256. Copyright IBM Corp. 2011,

280 Figure 112. Service laptop to RSA connection 3. In a cluster, disconnect the Ethernet cable from the RSA port on Server A. 4. On the laptop, click Start > Control Panel. 5. Double-click Network Connections. 6. Right-click Local Area Connection > Properties. See Figure 113. Figure 113. Local Area Connection Properties 7. From the list, select Internet Protocol (TCP/IP), then click Properties. See Figure 114 on page TS7650 and TS7650G Problem Determination and Service Guide

281 Figure 114. Local Area Connection Properties: Internet Protocol (TCP/IP) 8. Select Use the following IP address. 9. In the IP Address field, type or if a new field replaceable unit (FRU). In the Subnet mask field, type: Click OK and then click Close to exit the Local Area Connections Properties window. 11. In a cluster, reconnect the Ethernet cable to the RSA port on Server A. 12. On the TSSC or service laptop, open an Internet Explorer window. 13. In the Address Bar area of the browser window, type the default RSA card IP address or for a new FRU and then press Enter. 14. In the Login window, enter the user ID USERID and the password PASSW0RD (the 0 is a zero). Click OK. Note: The username and password are case sensitive and must be entered in all uppercase. The Remote Supervisor Adapter II Refresh 1 Welcome window is displayed. 15. In the Remote Supervisor Adapter II Refresh 1 Welcome window: a. Use the Inactive session timeout value menu to specify a reasonable amount of time that keeps your session active until you can complete your task. Do not select the No Timeout setting, which prevents anyone else from logging into the RSA until you have logged out. b. Click Continue. See Figure 115 on page 258. Appendix A. Logging into the 3958 DD1/3958 DD3 server through the RSA connection 257

282 ts Figure 115. Remote Supervisor Adapter II Refresh 1 Welcome window Results You are now logged into the server RSA. What to do next Go to Enabling remote control on page TS7650 and TS7650G Problem Determination and Service Guide

283 Enabling remote control Complete the task in this topic to enable remote control. Procedure 1. In the Remote Supervisor Adapter II Refresh 1 window, click Remote Control in the left-hand menu under the IBM logo. See Figure 116. Figure 116. Remote Supervisor Adapter II Refresh 1 window: Remote Control link 2. In the Remote Control window, click Start Remote Control in Single User Mode. See Figure 116. The advanced system management (ASM) Remote Control window is displayed. Note: If the browser on the TS3000 System Console (TSSC) or your laptop is configured to block pop-ups, temporarily disable this function. Accept any Java versions that are displayed, as well as any Windows security prompts displayed as a result of trying to open this window. What to do next After remote control is established, go on to Configuring remote supervisor adapter keystrokes on page 260. Appendix A. Logging into the 3958 DD1/3958 DD3 server through the RSA connection 259

284 Configuring remote supervisor adapter keystrokes Use the following procedure to specify unique remote supervisor adapter (RSA) keystroke combinations for use during Reliability, Availability, and Serviceability (RAS) package configuration. Procedure 1. In the asynchronous transfer mode (ASM) Remote Control window: a. Drag the Video Speed (bps) slider to the 5m setting (all the way to the right) to maximize video speed. You might have to adjust this setting later so characters show up on the screen when typed. b. Click the Windows Maximize button to maximize the size of the ASM Remote Control window. See Figure 117. ts Figure 117. Maximize video speed 2. In the ASM Remote Control window, click the Preferences link on the far right side of the Windows Terminal Services toolbar. See Figure 118. The Remote Console Preferences window opens. ts Figure 118. ASM Remote Control preferences Window 3. In the Remote Console Preferences window, do the following: a. Scroll down to locate the first row in which all input boxes are empty. b. Click inside the leftmost box in the row and press the F12 key. c. Click Save Buttons. d. Click the X to close the Remote Console Preferences window. See Figure 119 on page TS7650 and TS7650G Problem Determination and Service Guide

285 Figure 119. ASM Remote Console Preferences window Appendix B. Logging into the 3958 DD4, DD5, and AP1 servers through the IMM web interface Complete the task in this topic to log into a 3958 DD4, or 3958 DD5, or 3958 AP1 server through the Integrated Management Module (IMM) system management port. Before you begin This appendix provides an alternative method for establishing a connection to the server during Reliability, Availability, and Serviceability (RAS) package configuration, using the IMM port. This alternative method should be used only if the server connection cannot be made using a universal serial bus (USB) keyboard and graphics-capable monitor. In order to use the IMM to connect to the servers, you must use the TS3000 System Console (TSSC) (code level 5.9.x or higher required), or your IBM service laptop. If you use your laptop, you might also need an extension cord to reach a standard power outlet, or a 240 V to 110 V step-down voltage converter to allow the service laptop to run off of the power distribution unit (PDU). Copyright IBM Corp. 2011,

286 Important: By default, the IMM ports on both servers in a cluster have the same IP address. This causes a remote supervisor access IP address conflict when connecting to the IMM card through the TSSC network for the very first time. To avoid the conflict, you must change the IP address on one of the IMM ports so that its IP address is unique or unplug the Ethernet cable from the IMM to which you are not connecting. To do so, perform steps 2 through 12 on page 264. Change the IMM port IP address on Server B in the cluster. Do not change the default IMM port IP address on Server A. The default IP address of a new IMM is Procedure 1. If you are using the TSSC, skip ahead to step 13 on page 264. Otherwise, continue with step Connect an Ethernet cable from the Ethernet port on your laptop to the SYS MGMT (remote supervisor access port) on a stand-alone server, or on Server B in a cluster. See Figure 120. ProtecTIER Server ts Figure 120. Service laptop to IMM connection 3. In a cluster, disconnect the Ethernet cable from the IMM port on Server A. 4. On the laptop, click Start > Control Panel. 5. Double-click Network Connections. 6. Right-click Local Area Connection > Properties. See Figure 121 on page TS7650 and TS7650G Problem Determination and Service Guide

287 Figure 121. Local Area Connection Properties 7. From the list, select Internet Protocol (TCP/IP), then click Properties. See Figure 122. Figure 122. Local Area Connection Properties: Internet Protocol (TCP/IP) 8. Select Use the following IP address. 9. In the IP Address field, type an IP address. Appendix B. Logging into the 3958 DD4, DD5, and AP1 servers through the IMM web interface 263

288 v For the IMM of an existing system, type v For a new IMM, type In the Subnet mask field, type Click OK and then click Close to exit the Local Area Connections Properties window. 12. In a cluster, reconnect the Ethernet cable to the IMM port on Server A. 13. Open a web browser window. v v On the TSSC, use Mozilla Firefox. On a service laptop, use Internet Explorer. 14. In the address bar of the browser window, type an IP address. v To use the IMM static IP address of the existing system, type that address ( xx). v For a new IMM, type Note: If you are using the TSSC browser to address the IMM at the default IP address , you must set up a temporary alias address of in the TSSC in order to access the IMM. 15. Press Enter. 16. In the Login window, type the user ID and password. Field Value Notes User ID USERID Must be entered in all uppercase. Password PASSW0RD Must be entered in all uppercase. The 0 is a zero. 17. Click OK The Integrated Management Welcome window is displayed (see Figure 123 on page 265). 18. In the Integrated Management Module Welcome window: a. Use the Inactive session timeout value menu to specify a reasonable amount of time that keeps your session active until you can complete your task. (Example: A value of 10 minutes means that you will automatically be logged out only after 10 minutes of inactivity.) Note: Do not select the No Timeout setting. It prevents anyone else from logging into the IMM until you have logged out. b. Click Continue. See Figure 123 on page TS7650 and TS7650G Problem Determination and Service Guide

289 ts Figure 123. Integrated Management Module Welcome window You are now logged into the server IMM. 19. In the ASM section of the of the left navigation pane of the Integrated Management Welcome window, click Network Interface. 20. In the Static IP Address field, change the IP address of the IMM, if necessary. (For example, change the IP address to ) 21. In the Netmask field, type: Leave the Gateway field blank. 23. Click Save. An Alert dialog prompting you to click IMM Restart is displayed. 24. Click Restart. Click OK to continue. The ASM restarts. (This process takes a few minutes, during which the screen appears to freeze. Wait until the reset is complete.) The IMM IP address on server B is now unique. 25. On the TSSC or service laptop, open a browser window. v On the TSSC, use Mozilla Firefox. v On a service laptop, use Internet Explorer. 26. In the address bar of the browser, type the unique IMM IP address that you set in step In the Login window, type the user ID and password. Field Value Notes User ID USERID Must be entered in all uppercase. Password PASSW0RD Must be entered in all uppercase. The 0 is a zero. 28. Click OK The Integrated Management Welcome window is displayed (see Figure 123). 29. Use the Inactive session timeout value menu to specify a reasonable amount of time that keeps your session active until you can complete your task. (Example: A value of 10 minutes means that you will automatically be logged out only after 10 minutes of inactivity.) Appendix B. Logging into the 3958 DD4, DD5, and AP1 servers through the IMM web interface 265

290 Note: Do not select the No Timeout setting. It prevents anyone else from logging into the IMM until you have logged out. 30. Click Continue. What to do next Enabling remote control Go to Enabling remote control. (This might not be necessary for the IMM.) You must use a web browser running on a laptop to access remote control of the 3958 DD4 or 3958 DD5 through the IMM server. About this task The TSSC does not currently support access to remote control. To access remote control, use one of the following browsers: v Microsoft Internet Explorer v Mozilla Firefox v Google Chrome Procedure 1. In the Integrated Management Module window, click Remote Control in the left-hand menu under the IBM logo. See Figure 124. ts Figure 124. Integrated Management Module window: Remote Control link 2. In the Integrated Management Module window, click Start Remote Control in Single User Mode. See Figure 124. The advanced system management (ASM) Remote Control window is displayed. 266 TS7650 and TS7650G Problem Determination and Service Guide

291 Note: If the browser is configured to block pop-ups, temporarily disable this function. Accept any Java versions that are displayed, as well as any Windows security prompts displayed as a result of trying to open this window. What to do next After remote control is established, go to Configuring remote supervisor adapter keystrokes on page 260. Updating IMM firmware using the web interface This topic describes how to update Integrated Management Module (IMM) firmware using the web interface. About this task Important: This procedure enables you to update IMM, UEFI, FPGA and DSA firmware via the IMM web interface. It should only be used in an emergency, when firmware cannot be loaded normally for inband flashing by the operating system. Normally, firmware updates are performed during the autorun phase of ProtecTIER code load or by selecting the Update Firmware menu option. Note: The IMM, UEFI, FPGA and DSA firmware files can be downloaded from the IBM website. 1. Go to IBM Fix Central. 2. Click the IBM Sign in link in the masthead and log in using your IBM user ID. If you do not have an IBM user ID, follow the procedures to create one from any IBM web page. 3. In the Select product area, select System x from the Product Group menu. 4. Select System x3850 X5 from the Product type menu. 5. Select 7145 from the Product menu. 6. Select All (or your specific operating system) from the Operating system menu. 7. Click Continue to display the matrix of downloadable firmware files. 8. Scroll to the IMM area or other firmware area, select the link for the firmware update, and save the update file to your laptop or to the TSSC. 9. The IMM can use either the EXE type or BIN type of file to perform the update. Procedure 1. Connect an Ethernet cable from the Ethernet port on your laptop to the SYS MGMT port on the 3958 DD4 or 3958 DD5 server (see Figure 120 on page 262). 2. To facilitate opening of a web browser on your laptop, perform preparatory steps 4 on page 262 through 14 on page 264. Then return to this procedure. Note: If you are using the TSSC, perform only steps 13 on page 264 and 14 on page 264. Then return to this procedure. You have already opened the web browser with the appropriate IP address of the IMM Web interface to which you have to connect. Note: The IMM assigns an initial, default static IP address of Type your user name and password on the Login page. Appendix B. Logging into the 3958 DD4, DD5, and AP1 servers through the IMM web interface 267

292 v v If the user name and password have been changed, obtain them from the system administrator. Otherwise, type the initial values for the IMM: Field Value Notes User ID USERID Must be entered in all uppercase. Password PASSW0RD Must be entered in all uppercase. The 0 is a zero. 4. Select a timeout value, in minutes, in the field provided. If your browser is inactive for that number of minutes, the IMM logs you off the web interface. 5. Click Continue to start the session. The browser displays the System Health page, which gives you a quick view of the system status. 6. In the navigation frame, click Firmware Update under Tasks. Click Browse. 7. Navigate to the IMM firmware file (ibm_fw_imm_yuooxxx_linux_32-64.bin) v If you are performing the firmware update from the TSSC, the firmware file was saved in directory path /home/service/offload/. v If you are performing the firmware update from a laptop, the firmware file was saved in directory path C:\firmware_updt\. 8. Select the file and click Open. The filename, including the full path, is displayed in the box beside Browse. 9. To begin the update process, click Update. A progress indicator is displayed as the file is transferred to temporary storage on the IMM. After the file transfer is complete, a confirmation page is displayed. 10. Verify that the upd file shown on the Confirm Firmware Update page is the correct update file. If not, click the Cancel button. 11. To complete the update process, click Continue. A progress indicator is displayed as the firmware on the IMM is flashed. A confirmation page is displayed to verify that the update is successful. 12. Click the option Restart IMM. Click the Restart button. 13. Click OK to confirm that you want to restart the IMM. Your current session is disconnected. 14. Click OK to close the current browser window. 15. Wait a few minutes to allow the IMM to complete its restart before logging into the IMM again. 268 TS7650 and TS7650G Problem Determination and Service Guide

293 Appendix C. Checkpoint codes This topic lists some of the checkpoint codes displayed by the light path diagnostic panel. Affected configurations IBM System x3850 M2 (Type 7141, 7144) Checkpoint (cp) code table: Time (mm:ss) Checkpoint(s) Major System Components Successfully Initialized Video Display Comment - 0:0 Standby Blank. 00:00 Push power button to turn on system System power up sequence starts with code format 0:x or 1:x (digits separated by dots) 00:02 0:1.. 0:8 Turn on 12V, Turn on 5V, 3.3V volt regs Blank. 00:04 1:A..1:b..1:d Turn on chipset, processors Blank DVD drive LED on 00:15 1:E Reset chipset Blank 1A to 1E may repeat; 5-15 seconds - System processors begin executing Power On Self Test (POST) with checkpoint format yy (no dots) 00:00 72 BMC initialize Blank. 00:05 60 Memory controller initialized, memory enabled Blank. 00:06 0C..0d..0E..10 Checksumming BIOS flash ROM, copy flash ROM to RAM, execute from RAM Blank. 00:09 d4 Initialize early video RN :15 07 Check memory Checking system memory for errors, please wait :16 61 Initialize system memory Initializing system memory to good ECC may take up to 3 sec for 2 GB may take up to 2 min for 512 GB 00:19 0d..0E..10 Copy flash ROM to RAM again.. 00:21 d9..da..dc Internal PCI devices Initializing PCI devices, please wait :52 73 RSA II initialization RN50 200m/200e DDR1 BIOS. 00:56 82 Report POST errors, including previous memory error Clear screen. 00:58 C3 Sign on Logo. 01:01 b8 Multi-processor initialize (a) Logo: n GB memory: Installed. 01:07 b0 Multi-processor initialize (b) Logo: n Processor Packages Installed. 01:09 bd SMI initialize.. 01:11 Ad..bd SMI routines to initialize USB Logo (C7 in video). 01:20 3C Prompt Logo: Press F1 for Setup... Prompt to press F1, F2, or F12 01: IDE initialization Clear screen with cursor. 01:30 1E PCIe adapters ROM BIOS Adapter ROM BIOS messages Time here depends on number and type of installed PCIe adapters Copyright IBM Corp. 2011,

294 02: SMBIOS initialize Adapter ROM msgs or clear screen. 02:11 4C ACPI initialize Adapter ROM msgs or clear screen. 02:15 71 *beep* End POST. Operating System starts to load here 00:00 Push power button to turn off system. System power down sequence starts with code format 2:x (digits separated by dots) 00:05 2:0..2:7 Turn off processors, chipset, 3.3V and 5V volt regs, 12V Last video display when button was pressed.. 00:07 0:0 Standby Blank DC power off, fans spin down 270 TS7650 and TS7650G Problem Determination and Service Guide

295 Appendix D. Manually upgrading the DS4700 storage system firmware (ProtecTIER version 2.4 and later) Use this procedure if it is necessary for you to verify and upgrade the firmware manually. Before you begin Attention: The storage firmware upgrade process is not a concurrent procedure. All services must be stopped on the ProtecTIER node (both nodes if clustered), before beginning the firmware upgrade. Following the firmware update, you must restart the services on both nodes, starting with node A. 1. Stop the services on the nodes, starting with node B. From the ProtecTIER Service menu, select Manage ProtecTIER services > Stop ProtecTIER services only (including GFS). 2. Upgrade the storage firmware using the following procedure. 3. Restart the services on the nodes, starting with node A. From the ProtecTIER Service menu, select Manage ProtecTIER services > Start all services. About this task Attention: For currently released ProtecTIER products (V3.1 or later), storage firmware for the following devices is supported. Do not upgrade to higher versions of storage firmware. They are not supported. v DS4700: v DS3950: Note: As you perform the steps to upgrade the DS4700 storage system firmware, note that the IBM System Storage ProtecTIER Manager V3.2 DVD provides controller firmware, NVSRAM firmware, and ESM firmware. DS3950 firmware must be copied to the /home/service/offload directory in the TSSC from the ProtecTIER code in the 3958 AP1 (3958 DD4 server). The steps to upgrade DS3950 firmware are identical to the steps in this procedure for the DS4700, except that the DS3950 firmware files are provided to the TSSC from the ProtecTIER V3.1 software installed on the 3958 AP1. v DS3950 Controller firmware is in the /opt/dtc/ibm_ds4000/firmware/ts7650 directory. v DS3950 NVSRAM firmware is in the /opt/dtc/ibm_ds4000/nvsram/ts7650 directory. v DS3950 ESM (Enclosure) firmware is in the /opt/dtc/ibm_ds4000/esm_hdd/ EXP810 directory. Procedure 1. Verify that the firmware is at the correct code level. a. If you are not already logged in to the TSSC, log in with user ID service and password service. b. Right-click the blue desktop, and from the IBM TS3000 System Console menu, select Browser Functions > SM GUI Functions > Launch GUI. Copyright IBM Corp. 2011,

296 Note: If you receive a message that the DS Storage Manager GUI is not installed, you must install it using the supplied TS3000 System Console Graphical User Interface CD version x5.16. Use step 4 on page 273 to perform the installation, and return to this step after DS Storage Manager is installed. If the DS Storage Manager GUI is already installed, verify the version by selecting Help > About. If the DS Storage Manager version is below x5.16 use step 4 on page 273 to install the new version of DS Storage Manager and return to this step. The new version of DS Storage Manager replaces the previous version. c. If multiple storage systems are listed, you must identify the storage system you are updating. Select a storage system in the left navigation pane. When the subsystem list appears in the right pane, select the storage system, right-click it and select Locate Storage Subsystem from the menu. The indicator lights on the storage subsystem flash in blue until you click OK. d. If the storage system you want to update is not listed, or you are using DS Storage Manager for the first time and no systems are displayed, select Edit > Add Storage Subsystem. e. When prompted in the Add New Storage Subsystem menu, type the In-Band IP address of the storage subsystem, and click Add. The DS Storage Manager attempts to locate the subsystem and add it to the storage list. If the subsystem cannot be located, verify the IP addresses entered. f. After the correct subsystem has been identified, in the right pane, the storage subsystem should display as Optimal. If it does not, resolve any failure indictions before proceeding with the firmware upgrade. If you cannot resolve a failure condition and return the system to optimal status, contact your next level of IBM support. g. If the storage displays as optimal status, double-click to open the array. h. At the Synchronize window, click Cancel. i. If prompted for a cache subsystem password, enter ibm2serv. Click No when prompted to change the password. Attention: Do not change the password when prompted. This password must not change to preserve access for service. j. Click the Physical tab. Controller A and B are displayed. k. Click Controller A in the Storage Subsystem window. Properties for controller A are displayed in the right pane of the window. l. Record the levels displayed for the DS4700 firmware and NVSRAM. Verify that they are correct according to the TS7650/TS7650G ISV and interoperability matrix. (To view the matrix, go to the IBM Tape systems web page and click the link for TS7650/TS7650G ISV and interoperability matrix.) m. From the title bar, select Advanced > Maintenance > Download > ESM Configuration Settings. n. Verify that Card A and Card B firmware are at the minimum level of 98C5 or the recommended level of 98D0. o. Click Close to close the window. p. If the storage firmware level is lower than (for example: or ), it must be updated to minimum supported levels with ProtecTIER v. 2.4 microcode. Go to step 2 on page 273 to copy the latest storage firmware from the IBM System Storage ProtecTIER Manager V3.2 DVD to the TSSC. q. If the storage firmware levels are or higher (recommended level is ), proceed to the ProtecTIER v. 2.4 upgrade. 272 TS7650 and TS7650G Problem Determination and Service Guide

297 2. Insert the IBM System Storage ProtecTIER Manager V3.2 DVD into the TSSC drive. a. If you are not already logged in to the TSSC, log in with user ID service and password service. b. Right-click the blue desktop of the TSSC. From the System Console menu, select System Console Actions > CD copy to console. c. On the CD copy to console screen, select option 3, /home/service/offload. d. Follow the instructions on the screen to copy the code. After the code has been copied, the disk ejects. e. Verify the /home/service/offload/ds4k Storage manager v10.60.x5.17 for Windows 32-bit/DS_FW directory contents: v esm98d0.esm v IBM_EXP810_CFG_v0150.DL v ST FC_B989.LOD v FW_DS4700_ dlp v FW_DS4700_ dlp v FW_DS5020_ dlp v N1814D47R1060V04.dlp v N1814D47R1060V17.dlp v N1814D20R1060V19.dlp f. Right-click the TSSC blue desktop. From the System Console menu, select Terminal. 3. Type the following commands: cd /home/service/offload/ds4k Storage manager v10.60.x5.17 for Windows 32-bit/DS_FW ls Note: The storage system firmware is on the IBM System Storage ProtecTIER Manager V3.2 DVD under a x5.17 for Windows 32-bit directory. It is used on the TSSC Linux version as well, and the files are the same. 4. Upgrade the DS Storage Manager application to version x5.16 on the TSSC. a. Right-click the blue desktop of the TSSC and select Browser Functions > Storage Manager GUI Functions > Install GUI. The TSSC CD tray opens automatically. b. Insert the TS3000 System Console Storage Manager CD, version x5.16 (in the TSSC ship group) into the TSSC and follow the instructions on the screen to complete the installation. 5. Locate the firmware and NVSRAM files using the DS Storage Manager. a. Launch the DS Storage Manager from the TSSC by right-clicking the TSSC and selecting Browser Functions > Storage Manager GUI Functions > Launch GUI. b. Select the Storage Subsystem to be upgraded (identified in step 1 on page 271) by right-clicking the storage and selecting Manage Storage Subsystem. v If prompted to enter a password, enter: ibm2serv. v If prompted to change or set a password, click No. v If prompted with a Synchronize Controller Clocks window, click Cancel. c. In the Subsystem Management window, select the Advanced button in the top menu bar. Appendix D. Manually upgrading the DS4700 storage system firmware (ProtecTIER version 2.4 and later) 273

298 d. Select Maintenance > Download > Controller Firmware. The Download Controller Firmware window is displayed. The DS Storage Manager checks the health of the storage subsystem. This check might take several minutes. Any problem associated with the storage is identified. You cannot continue with the upgrade until the problem is resolved. If you cannot isolate or resolve the error condition, contact your next level of support. e. In the Controller Firmware panel, click the Select File button. A new window is displayed. f. Type the path of the controller firmware and press Enter. The path might be similar to the following: /home/service/offload/ds4k Storage manager v10.60.x5.17 for Windows 32-bit/DS_FW. g. In the same window, click to highlight the FW.DS4700_ dlp file from the Files section. Note: The storage system firmware is on the IBM System Storage ProtecTIER Manager V3.2 DVD under a x5.17 for Windows 32-bit directory, DS4k Storage Manager v10.60.x5.17 for Windows 32-bit. It is used on the TSSC Linux version as well, and the files are the same. h. Click the OK button at the bottom of the window. The Download Controller Firmware window is displayed. i. In the Download Controller Firmware window, click the Transfer the NVSRAM file with controller firmware button. j. Click the Select File button in the NVSRAM section. A new window is displayed. k. Type the path of the NVSRAM and press Enter. The path might be similar to the following: /home/service/offload/ds4k Storage manager v10.60.x5.17 for Windows 32-bit/DS_FW. l. In the same window, select the N1814D47R1060V04.dlp file from the Files section. Note: The storage system firmware is on the IBM System Storage ProtecTIER Manager V3.2 DVD under a x5.17 for Windows 32-bit directory, DS4k Storage Manager v10.60.x5.17 for Windows 32-bit. It is used on the TSSC Linux version as well, and the files are the same. m. Click the OK button at the bottom of the window. The Download Controller Firmware window is displayed. 6. Upgrade the firmware and NVSRAM using the DS Storage Manager. In the Download Controller Firmware window, click the Transfer button. 7. When you are prompted with a version compatibility window to confirm the download and you are asked Do you want to continue?, click Yes. The transfer and activation process takes minutes to complete. Note: During this activation, both A and B controllers are upgraded. When the activation is completed, a New Firmware Detected window is displayed. Click OK. 8. Upgrade the ESM level. a. In the Manage Storage Subsystem window, click Advanced button in the top menu bar. b. Select Maintenance > Download > ESM Firmware. The Download Environmental (ESM) Card Firmware window is displayed. c. In the Drive Enclosure section, select the enclosure to be upgraded. 274 TS7650 and TS7650G Problem Determination and Service Guide

299 d. In the ESM Firmware section, click the Select File button. A new window is displayed. e. Type the path of the ESM and press Enter. The path might be similar to the following: /home/service/offload/ds4k Storage manager v10.60.x5.17 for Windows 32-bit/DS_FW. f. In the same window, click to highlight the esm98d0.esm file from the Files section. Note: The storage system firmware is on the IBM System Storage ProtecTIER Manager V3.2 DVD under a x5.17 for Windows 32-bit directory, DS4k Storage Manager v10.60.x5.17 for Windows 32-bit. It is used on the TSSC Linux version as well, and the files are the same. g. Click the OK button at the bottom of the window. The Download Environmental (ESM) Card Firmware window is displayed. h. Click the Start button in the Download Environmental Card Firmware window to start the ESM download process. A Confirm Download window is displayed. i. Type yes and click OK to continue. This process takes 5-7 minutes for each enclosure. The Status column displays Transferring until the process is complete. The enclosures can only be updated one at a time. j. Repeat substeps 8c on page 274 through 8h for each enclosure. Results The upgrade process of the firmware and NVSRAM is now complete. What to do next Close any open DS Storage Manager windows. Right-click the blue desktop of the TSSC and select Logout. Appendix D. Manually upgrading the DS4700 storage system firmware (ProtecTIER version 2.4 and later) 275

300 276 TS7650 and TS7650G Problem Determination and Service Guide

301 Appendix E. ProtecTIER Network Performance Validation Utility The objective of the pt_net_perf_util is to test maximal network performance in order to discover potential performance bottlenecks. On VTL systems, the utility tests maximal replication performance between two future ProtecTIER VTL repositories by emulating the network usage patterns of the ProtecTIER Native Replication component. On OpenStorage systems, the utility tests network performance between an OpenStorage host and a ProtecTIER server to identify possible bottlenecks in a backup and restore scenario. Copyright IBM Corp. 2011,

302 278 TS7650 and TS7650G Problem Determination and Service Guide

303 Appendix F. ProtecTIER Replication Network Performance Validation Utility for VTL Systems The objective of the VTL pt_net_perf_util utility is to test maximal replication performance between two future ProtecTIER VTL repositories by emulating the network usage patterns of the ProtecTIER Native Replication component. This utility can not predict replication performance, but it can discover performance bottlenecks. Before you begin The requirements of this utility are as follows: v Red Hat Linux version 5.6 or Red Hat Linux version 5.4 v Standard external utilities expected to be in the current path: ping, netstat, getopt, echo. The pt_net_perf_util utility and the iperf tool are part of the ProtecTIER software installation. To test the replication performance, use the following tool: v iperf /usr/local/bin/iperf This utility has two modes of operation, client and server. The server has to be started before the client. Before running the utility, shut down all other programs on both the client and server ProtecTIER systems. The client is the ProtecTIER system that transmits the test data and the server is the ProtecTIER system that receives the data (also known as the target server). Based on the data sent by the client and received by the server, the script outputs key network parameter values which indicate certain attributes of the network. The goal of these tests is to benchmark the throughput of the network. The most important benchmark is the direction that replication actually takes place, that is, test the target as the server since the flow of data will be to that server from the client. However, it is also important to also test the reverse direction to measure the bandwidth performance during disaster recovery failback. Network bandwidth is not always the same in both directions. For the utility to work, the replication ports must be configured on the source and the destination servers, and there must be a static route defined from source to destination and from destination to source. About this task In the following procedure, the goal is to test network performance between two machines on a WAN, server1 and server2. Each test runs for five minutes. Since there are five tests, the process takes a total of 25 minutes. Repeat these tests for each server and for each replication port on the server (two tests total per server, one for each replication port). Copyright IBM Corp. 2011,

304 Procedure 1. At the login prompt, log in with user name root and password admin. Press Enter. On the command line, type menu and press Enter. The ProtecTIER Service Menu is displayed ProtecTIER Service Menu running on rasddx ) ProtecTIER Configuration (...) 2) Manage ProtecTIER services (...) 3) Health Monitoring (...) 4) Problem Alerting (...) 5) Version Information (...) 6) Generate a service report 7) Generate a system view 8) Update ProtecTIER code E) Exit >>> Your choice? 2. Stop the VTFD services. a. Select the option to Manage ProtecTIER services (...).. b. The Manage ProtecTIER services menu is displayed ProtecTIER Service Menu running on rasgen Manage ProtecTIER Services (...) ) Display services status 2) Start all services 3) Stop all services 4) Stop ProtecTIER services only (including GFS) 5) Stop VTFD service only B) Back E) Exit >>> Your choice? c. Select the option to Stop ProtecTIER services only (including GFS). Note: You can also stop the VTFD services from the command line. From the command line on the server, enter the following command: service vtfd stop <enter> 3. Wait for the services to stop. A message similar to the following displays: Shutting down vtfd: [ OK ] Please wait... [root@austin ~]# 4. Start the server mode of the utility on server1 by entering the following commands on the command line: cd /opt/dtc/app/sbin./pt_net_perf_util -s 5. Start the client mode of the utility on server2 by entering the following command on the command line: cd /opt/dtc/app/sbin./pt_net_perf_util -c <x.x.x.x> -t 300 Where <x.x.x.x> is the IP address of the server replication port. 280 TS7650 and TS7650G Problem Determination and Service Guide

305 6. The utility automatically performs the tests in sequence. The client output (server2 in the example shown) looks similar to the following: Note: In the sample output below the test ran for only 5 seconds instead of 300. *** Latency PING ( ) 56(84) bytes of data ping statistics packets transmitted, 5 received, 0% packet loss, time 4001ms rtt min/avg/max/mdev = 0.257/0.406/0.484/0.079 ms *** Throughput - Default TCP [ 3] sec 56.6 MBytes 94.8 Mbits/sec *** Throughput - 1 TCP stream(s), 1MB send buffer [ 3] sec 57.0 MBytes 95.0 Mbits/sec *** Throughput - 16 TCP stream(s), 1MB send buffer [SUM] sec 65.0 MBytes 94.3 Mbits/sec *** Throughput TCP stream(s), 1MB send buffer [SUM] sec 127 MBytes 94.1 Mbits/sec Number of TCP segments sent: Number of TCP retransmissions detected: 21 (0%) Done. See the next section for information about interpreting the results of the tests. Interpreting the results The utility performs five foreground tests (Tests 1-5 below), and one background test (Test 6 below). The example outputs shown are from the client side. Each of the first five tests ran for 300 seconds (-t 300), while the last test monitored TCP performance during that time. Test 1: Latency This test checks the nominal network link latency and packet loss. Example result: *** Latency PING ( ) 56(84) bytes of data ping statistics packets transmitted, 120 received, 0% packet loss, time ms rtt min/avg/max/mdev = /78.491/ /9.872 ms Interpreting the results: v The average round-trip-time (rtt) was 78.4 ms and there was 0% packet loss. v The latency in WAN topologies can vary, but should never exceed 200 ms. Contact your network administrator if latency reports more than 200 ms, as it can significantly decrease replication throughput. v Higher latency values can cause a major deterioration in replication throughput. v Packet loss should be 0%. Any other value indicates a major network problem. Test 2: Throughput - default settings Appendix F. ProtecTIER Replication Network Performance Validation Utility for VTL Systems 281

306 This test checks maximal TCP throughput of a single data stream with default TCP settings. Example result: *** Throughput - Default TCP [ 3] sec 2.41 GBytes 173 Mbits/sec Interpreting the results: v The test ran for seconds, transferred 2.41 GB (2.24 GiB), with an average throughput of 173 Mbits/sec. Note: 1 MByte = 1,048,576 bytes. 1 Mbit/sec = 1,000,000 bits/sec. Test 3: Throughput - single stream, 1 MB send buffer This test checks maximal TCP throughput of a single data stream with a1mb (0.95 MiB) send buffer. Example result: *** Throughput - 1 TCP stream(s), 1MB send buffer [ 3] sec 2.51 GBytes 180 Mbits/sec Interpreting the results: v The test ran for seconds, transferred 2.51 GB (2.34 GiB), with an average throughput of 180 Mbits/sec. v There might be an improvement here on high-latency links. Test 4: Throughput - 16 streams, 1 MB send buffer Example result: *** Throughput - 16 TCP stream(s), 1MB send buffer [SUM] sec 5.91 GBytes 418 Mbits/sec Interpreting the results: v The test ran for seconds, transferred 5.91 GB (5.5 GiB), with an average throughput of 418 Mbits/sec. v The extra streams yielded higher utilization of the connection. v The Mbits/sec reported in this test is the maximum replication performance your system will achieve if your backup environment is using up to 2-3 cartridges in parallel. Test 5: Throughput streams, 1 MB send buffer Example result: *** Throughput TCP stream(s), 1MB send buffer [SUM] sec 8.08 GBytes 550 Mbits/sec Interpreting the results: v The test ran for seconds, transferred 8.08 GB (7.53 Gb), with an average throughput of 550 Mbits/sec. v TCP takes a while to reach its maximal throughput. Longer testing times, 300 seconds or more, produces more accurate results. v The throughput value given by this test is the potential physical replication throughput for this system. It is directly affected by the available bandwidth, latency, packet loss, and retransmission rate. 282 TS7650 and TS7650G Problem Determination and Service Guide

307 v The Mbits/sec reported in this test is the maximum replication performance your system might achieve. If this number is lower than anticipated, contact your network administrator. Test 6: TCP Retransmissions versus Total TCP segments sent Example result: Number of TCP segments sent: Number of TCP retransmissions detected: (12%) Interpreting the results: v A total of TCP segments were sent during the five tests, out of which, were lost and retransmitted. v The retransmission rate imposes a direct penalty on the throughput, as the retransmission of these packets take up bandwidth. The retransmission can be caused by the underlying network (for example, packet dropping by an overflowed router) or by the TCP layer itself (for example, retransmission due to packet reordering). v Segment loss can be caused by each of the network layers. v TCP retransmission larger than 2% can cause performance degradation and unstable network connectivity. Contact your network administrator to resolve this issue and reduce it to approximately 0%. What to do next You might want to run these tests again to test the reverse throughput in the network. To run the tests in reverse, change server1 to the client and server2 to the server and repeat the procedures. Appendix F. ProtecTIER Replication Network Performance Validation Utility for VTL Systems 283

308 284 TS7650 and TS7650G Problem Determination and Service Guide

309 Appendix G. ProtecTIER Network Performance Validation Utility for OpenStorage Systems The objective of the OpenStorage pt_net_perf_util utility is to test network performance between an OpenStorage host and a ProtecTIER server. This utility can not predict backup and restore performance, but it can discover performance bottlenecks. Before you begin The requirements of this utility are as follows: v Red Hat Linux version 5.6 or 5.4 v Standard external utilities expected to be in the current path such as ping and netstat About this task The pt_net_perf_util utility and the iperf tool are part of the ProtecTIER software installation. The location of the utility varies according to the platform: v Windows plug-in: %PROGRAMFILES%\IBM\ost_plugin_tools v AIX plug-in: /opt/ibm/ost_plugin_tools v ProtecTIER server: /opp/dtc/app/sbin This utility has two modes of operation, client and server. The server has to be started before the client. Before running the utility, shut down all other programs on both the client and server systems. The client is the system that transmits the test data and the server is the system that receives the data (also known as the target server). Based on the data sent by the client and received by the server, the script outputs key network parameter values which indicate certain attributes of the network. The goal of these tests is to benchmark the throughput of the network. It is important to test both OpenStorage host-to-protectier server and ProtecTIER server-to-openstorage host directions to measure the bandwidth performance during both backup and restore activities. Network bandwidth is not always the same in both directions. In addition, if an OpenStorage host or a ProtecTIER server uses more than one IP address for OpenStorage operation, it is essential to test each IP address separately. For the utility to work, the OpenStorage ports must be configured on the source and the destination servers, and there must be a static route defined from source to destination and from destination to source. In the following procedure, the goal is to test network performance between two machines server1 and server2. Each test runs for five minutes. Since there are five tests, the process takes a total of 25 minutes. Repeat these tests for each server and for each OpenStorage port on the server (between two and six tests total per server, one for each OpenStorage port). Copyright IBM Corp. 2011,

310 Procedure 1. At the login prompt, log in with user name root and password admin. Press Enter. On the command line, type menu and press Enter. The ProtecTIER Service Menu is displayed ProtecTIER Service Menu running on rasddx ) ProtecTIER Configuration (...) 2) Manage ProtecTIER services (...) 3) Health Monitoring (...) 4) Problem Alerting (...) 5) Version Information (...) 6) Generate a service report 7) Generate a system view 8) Update ProtecTIER code E) Exit >>> Your choice? 2. Stop the VTFD services. a. Select the option to Manage ProtecTIER services (...).. b. The Manage ProtecTIER services menu is displayed. c. Select the option to Stop ProtecTIER services only (including GFS). Note: You can also stop the VTFD services from the command line. From the command line on the server, enter the following command: service vtfd stop <enter> 3. Wait for the services to stop. A message similar to the following displays: Shutting down vtfd: [ OK ] Please wait... [root@austin ~]# 4. Start the server mode of the utility on server1 by entering the following commands on the command line:./pt_net_perf_util -s 5. Start the client mode of the utility on server2 by entering the following command on the command line:./pt_net_perf_util -c server1_ip_address -t 300 where server1_ip_address is one of the IP addresses to be tested for server1. The default size of a buffer used for sending data is 1024 KB. To override this size to another value, for example 256 KB, use the -l parameter as follows:./pt_net_perf_util -c server1_ip_address -t 300 -l 256 Note: If server1 is a Windows system, use following command: pt_net_perf_util.bat -c server1_ip_address -t win-server where server1_ip_address is one of the IP addresses to be tested for server1. The same buffer override command can also be used to change the size of the buffer. 6. The utility automatically performs the tests in sequence. The client output (server2 in the example) looks similar to the following: Note: In the sample output below the test ran for only 5 seconds instead of TS7650 and TS7650G Problem Determination and Service Guide

311 *** Latency PING ( ) 56(84) bytes of data ping statistics packets transmitted, 5 received, 0% packet loss, time 4001ms rtt min/avg/max/mdev = 0.257/0.406/0.484/0.079 ms *** Throughput - Default TCP [ 3] sec 56.6 MBytes 94.8 Mbits/sec *** Throughput - 1 TCP stream(s), 1MB send buffer [ 3] sec 57.0 MBytes 95.0 Mbits/sec *** Throughput - 16 TCP stream(s), 1MB send buffer [SUM] sec 65.0 MBytes 94.3 Mbits/sec *** Throughput TCP stream(s), 1MB send buffer [SUM] sec 127 MBytes 94.1 Mbits/sec Number of TCP segments sent: Number of TCP retransmissions detected: 21 (0%) Done. 7. See the next section for information about interpreting the results of the tests. Interpreting the results The utility performs five foreground tests (Tests 1-5 below), and one background test (Test 6 below). The example outputs are from the client side. Each of the first five tests ran for 300 seconds (-t 300), while the last test monitored TCP performance during that time. Test 1: Latency This test checks the nominal network link latency and packet loss. Example result: *** Latency PING ( ) 56(84) bytes of data ping statistics packets transmitted, 120 received, 0% packet loss, time ms rtt min/avg/max/mdev = /78.491/ /9.872 ms Interpreting the results: v The average round-trip-time (rtt) was 78.4 ms and there was 0% packet loss. v The round-trip-time in network topologies can vary, but should never exceed 200 ms on WAN. Contact your network administrator if round-trip-time reports more than 200 ms on WAN or 6 ms on LAN, as it can significantly decrease throughput. v Higher latency values can cause a major deterioration in throughput. v Packet loss should be 0%. Any other value indicates a major network problem. Test 2: Throughput - default settings This test checks maximal TCP throughput using a single data stream with default TCP settings. Example result: *** Throughput - Default TCP [ 3] sec 2.41 GBytes 173 Mbits/sec Interpreting the results: v The test ran for seconds, transferred 2.41 GB (2.24 Gb), with an average throughput of 173 Mbits/sec. Note: 1 MByte = 1,048,576 bytes. 1 Mbit/sec = 1,000,000 bits/sec. Test 3: Throughput - single stream, 1 MB send buffer Appendix G. ProtecTIER Network Performance Validation Utility for OpenStorage Systems 287

312 This test checks maximal TCP throughput of a single data stream with a 1 MB (0.95 MiB) send buffer. Example result: *** Throughput - 1 TCP stream(s), 1MB send buffer [ 3] sec 2.51 GBytes 180 Mbits/sec Interpreting the results: v The test ran for seconds, transferred 2.51 GB (2.34 GiB), with an average throughput of 180 Mbits/sec. v There might be an improvement here on high-latency links. Test 4: Throughput - 16 streams, 1 MB send buffer Example result: *** Throughput - 16 TCP stream(s), 1MB send buffer [SUM] sec 5.91 GBytes 418 Mbits/sec Interpreting the results: v The test ran for seconds, transferred 5.91 GB (5.5 Gb), with an average throughput of 418 Mbits/sec. v The extra streams yielded higher utilization of the connection. Test 5: Throughput streams, 1 MB send buffer Example result: *** Throughput TCP stream(s), 1MB send buffer [SUM] sec 8.08 GBytes 550 Mbits/sec Interpreting the results: v The test ran for seconds, transferred 8.08 GB (7.53 Gb), with an average throughput of 550 Mbits/sec. v TCP takes a while to reach its maximal throughput. Longer testing times, 300 seconds or more, produce more accurate results. v The throughput value given by this test is the potential physical throughput for this system. It is directly affected by the available bandwidth, latency, packet loss, and retransmission rate. v The Mbits/sec reported in this test is the maximum throughput your system can achieve. If this number is lower than anticipated, contact your network administrator. Test 6: TCP Retransmissions versus Total TCP segments sent Example result: Number of TCP segments sent: Number of TCP retransmissions detected: (12%) Interpreting the results: v A total of TCP segments were sent during the five tests, out of which, were lost and retransmitted. v The retransmission rate imposes a direct penalty on the throughput, as the retransmission of these packets take up bandwidth. The retransmission can be caused by the underlying network (for example, packet dropping by an overflowed router) or by the TCP layer itself (for example, retransmission due to packet reordering). 288 TS7650 and TS7650G Problem Determination and Service Guide

313 v v Segment loss can be caused by each of the network layers. TCP retransmission larger than 2% can cause performance degradation and unstable network connectivity. Contact your network administrator to resolve this issue and reduce it to approximately 0%. What to do next You might want to run these tests again to test the reverse throughput in the network. To run the tests in reverse, change server1 to the client and server2 to the server and repeat the procedures. Appendix G. ProtecTIER Network Performance Validation Utility for OpenStorage Systems 289

314 290 TS7650 and TS7650G Problem Determination and Service Guide

315 Appendix H. Checking the Western Telematic Inc. power ports This topic provides a method to access the Western Telematic Inc. (WTI) switch to view power port status and to power on or power off the power ports. This method must be performed by an IBM service representative. About this task Important: The commands within this topic must be initiated from only one server. Ports 1 and 5 supply power to Node A. Port 1 is the upper power supply of Node A. Ports 2 and 6 supply power to Node B. Port 2 is the upper power supply of Node B. Figure 125 shows the dual node clustered cabling. Ethernet switch P Ethernet switch P WTI Network Power Switch 3958 Server B P P R S U E1 E2 A V P P Server A 7 8 E R S U E1 E2 A V P P ts Figure 125. Dual node cluster cabling for 3958 AP1 and 3958 DD3 Figure 126 on page 292 shows the network power switch connections. Copyright IBM Corp. 2011,

316 WTI Network Power Switch TJ8 BJ8 BJ5 BJ4 P P E Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) Empty (1u) TSSC (1u) TSSC Network Switch & KVM (1u) Top Power Distribution Unit (PDU) TJ5 TJ DD PCS 3850 M2 (4u) Empty (1u) 3958 DD PCS 3850 M2 (4u) Bottom Power Distribution Unit (PDU) BJ1 Empty (1u) 1 GB Ethernet Switch (1u) 1 GB Ethernet Switch (1u) Empty (1u) Empty (1u) TJ1 ts Figure 126. Network power switch connections 292 TS7650 and TS7650G Problem Determination and Service Guide

317 Ethernet switch 2 P Ethernet switch 1 P WTI network power switch P P A1 A2 A3 A4 B1 B2 B3 B4 E 4 3 ProtecTIER Server B ProtecTIER Server A Replication 1 8 Replication 1 P1 A1 P1 A2 20 P2 B V S E1 E3 E2 E Replication 2 1 Replication 2 9 Customer Network Customer Network P2 A1 A2 B V S E1 E2 9 E3 E4 ts Figure 127. Ethernet cable connections in the VTL configuration of two clustered 3958 DD4/3958 DD5 servers Appendix H. Checking the Western Telematic Inc. power ports 293

318 Table 41. Clustered 1 Gb Ethernet switch Ethernet connections for VTL configuration Callout From On Device To On Device/Location 1 Port 1, slot 5 Server A Customer network Customer designated device 2 Port 1 Ethernet switch 1 Port 2, slot 5 Server A 3 Port 1 Ethernet switch 2 Port A1 Server A 4 Port 3 Ethernet switch 2 Ethernet port, E WTI network power switch 5 Port 4 Ethernet switch 2 Port 4 Ethernet switch 1 6 Port 1, slot 5 Server B Customer network Customer designated device 7 Port 2 Ethernet switch 1 Port 2, slot 5 Server B 8 Port 2 Ethernet switch 2 Port A1 Server B 19 Port 3, slot 5 Server A Customer's Customer specified device replication network 20 Port A2 Server A Customer's Customer specified device replication network 26 Port 3, slot 5 Server B Customer's Customer specified device replication network 27 Port A2 Server B Customer's replication network Customer specified device 294 TS7650 and TS7650G Problem Determination and Service Guide

319 Figure 128. Power connections to the WTI switch for clustered 3958 DD4 and 3958 DD5 servers (TS7650 Appliance shown) Note: The Network Power Switch is shown vertically at the side for ease of understanding where the connections should go. Actual placement and orientation of the switch might differ. Appendix H. Checking the Western Telematic Inc. power ports 295

320 Table 42. Clustered DD4 and DD5 servers to WTI switch power cabling Callout From On Device To On Device/Location 1 Power outlet Ethernet switch 1 (bottom) (E SW-1) BJ10 Bottom PDU 2 Power outlet Ethernet switch 2 (top) (E SW-2 TJ10 Top PDU 3 Power outlet TSSC Ethernet switch (if you BJ4 Bottom PDU installed one) (SC SW) 4 Power outlet TSSC KVM switch (if you TJ4 Top PDU installed one) (SC TSSC) 5 Power outlet 1 TSSC (if you installed one) BJ5 Bottom PDU (SC) 6 Power outlet 2 TSSC (if you installed one with TJ5 Top PDU a second power outlet) (SC) 7 Power outlet KVM Switch BJ3 Bottom PDU 14 Top power outlet WTI network power switch TJ8 Top PDU 15 Bottom power outlet WTI network power switch BJ8 Bottom PDU 16 Power outlet 1 Server A Power outlet 1 WTI network power switch 17 Power outlet 2 Server A Power outlet 5 WTI network power switch 18 Power outlet 1 Server B Power outlet 2 WTI network power switch 19 Power outlet 2 Server B Power outlet 6 WTI network power switch 20 Power outlet 2 CNA-1 BJ1 Bottom PDU Power outlet 2 EXPA Power outlet 1 CNB-1 TJ1 Top PDU Power outlet 1 EXPB Power outlet 2 EXPA1-2 BJ2 Bottom PDU Power outlet 2 EXPA Power outlet 1 EXPB1-2 TJ2 Top PDU Power outlet 1 EXPB Power outlet 2 CNA-2 BJ6 BottomPDU Power outlet 2 EXPA Power outlet 1 CNB-2 TJ6 Top PDU Power outlet 1 EXPB Power outlet 2 EXPA2-2 BJ7 Bottom PDU Power outlet 2 EXPA Power outlet 1 EXPB2-2 TJ7 Top PDU Power outlet 1 EXPB TS7650 and TS7650G Problem Determination and Service Guide

321 Procedure 1. Use Telnet to connect to the WTI switch (from the running node) by entering the following command: telnet ~]# telnet Trying Connected to ( ). Escape character is ^]. 2. When you are prompted, log in by typing the password. The default WTI password is password. Enter Password: ******** The overview is displayed. Internet Power Switch v1.41h Site ID: (undefined) Plug Name Password Status Boot/Seq. Delay Default internal_node1 (undefined) ON 5 Secs ON 2 internal_node2 (undefined) OFF 5 Secs ON 3 (undefined) (undefined) ON 0.5 Secs ON 4 (undefined) (undefined) ON 0.5 Secs ON 5 internal_node1 (undefined) ON 5 Secs ON 6 internal_node2 (undefined) ON 5 Secs ON 7 (undefined) (undefined) ON 0.5 Secs ON 8 (undefined) (undefined) ON 0.5 Secs ON "/H" for help. 3. Identify which port is turned off. In the example above, port 2 is currently turned off and is the port that you need to turn on. Turn on the identified port by entering the command /On [n] where n is the port number. Type y when asked if you are sure. IPS> /On 2 Plugs to be turned on: Plug 2: internal_node2 Sure? (Y/N): y Processing - please wait... Internet Power Switch v1.41h Site ID: (undefined) Plug Name Password Status Boot/Seq. Delay Default internal_node1 (undefined) ON 5 Secs ON 2 internal_node2 (undefined) ON 5 Secs ON 3 (undefined) (undefined) ON 0.5 Secs ON 4 (undefined) (undefined) ON 0.5 Secs ON 5 internal_node1 (undefined) ON 5 Secs ON 6 internal_node2 (undefined) ON 5 Secs ON 7 (undefined) (undefined) ON 0.5 Secs ON 8 (undefined) (undefined) ON 0.5 Secs ON Note: You can use the command /H (help) to bring up further details. Also refer to the WTI Power Switch User's Guide on the IBM System Storage TS7650 with ProtecTIER Publications CD and the documentation on the WTI website. Appendix H. Checking the Western Telematic Inc. power ports 297

322 IPS> /H Internet Power Switch v1.41h Site ID: (undefined) Display Configuration /H Display Help Screen /G View/Set General Parameters /S Display Plug Status /P [n] View/Set Plug Parameters /SN Display Network Status /C View/Set Serial Parameters /N View/Set Network Parameters Control /T View/set Telnet Parameters /D Set Plugs to Default /W View/Set Web Server /Boot <n> Boot Plug n /E Save Parameters /On <n> Turn On Plug n /R Recall Parameters /Off <n> Turn Off Plug n /DL Download Parameters to File /X Exit/Disconnect Utilities /I Reset Network Interface [n] = optional plug name or number /U Upgrade Firmware <n> = required plug name or number n+nornn=plug n and plug n n:n = plug n through plug n * = all plugs,y = bypass "Sure? (y/n)" Exit the telnet session and type y when asked if you are sure. IPS> /X Exit command mode. Sure? (Y/N): y Disconnected. Connection closed by foreign host. [root@austin ~]# 5. If fencing occurs again and cannot be stopped, contact your next level of support. 298 TS7650 and TS7650G Problem Determination and Service Guide

323 Appendix I. Configuring the Western Telematic Inc. switch This topic describes the procedure for manually configuring the Western Telematic Inc. (WTI) switch. About this task The WTI switch is automatically configured on reboot after a firmware upgrade. If it becomes necessary to manually reconfigure the switch, perform the following steps. Procedure 1. Log on to the lower ProtecTIER server. 2. Create an IP alias by typing in the following command: ifconfig eth1: From the ProtecTIER server command prompt, telnet to the WTI switch. telnet Enter the Password : password 4. At the IPS prompt, type /H to list the configuration commands. 5. Type /G to set the General Parameters menu. 6. Change the system password. a. Type 1. b. Type the new password. c. Type the new password again to confirm. d. Press the ESC key to exit the General Parameters menu. 7. Change the plug parameters for Node A power plug. a. Display the plug numbers and names by typing /S at the IPS prompt. b. Type /P 1 to change the parameters for plug 1. c. Type 1 to select the name. d. Type the plug name internal_node1. e. Type 3 to select the Seq/Delay parameter. f. Type 4 to set the delay to 5 seconds. g. Press ESC to exit the Plug #1 parameters. h. Type /P 5 to change the parameters for plug 5. i. Type 1 to select the name. j. Type the plug name internal_node1. k. Type 3 to select the Seq/Delay parameter. l. Type 4 to set the delay to 5 seconds. m. Press the ESC key to exit the Plug #5 parameters. 8. Change the plug parameters for Node B power plug. a. Display the plug numbers and names by typing /S at the IPS prompt. b. Type /P 2 to change the parameters for plug 2. c. Type 1 to select the name. d. Type the plug name internal_node2. e. Type 3 to select the Seq/Delay parameter. Copyright IBM Corp. 2011,

324 f. Type 4 to set the delay to 5 seconds. g. Press ESC to exit the Plug #2 parameters. h. Type /P 6 to change the parameters for plug 6. i. Type 1 to select the name. j. Type the plug name internal_node2. k. Type 3 to select the Seq/Delay parameter. l. Type 4 to set the delay to 5 seconds. m. Press the ESC key to exit the Plug #6 parameters. 9. Verify that the plug parameters are set correctly. a. Type /S at the IPS prompt. The plug parameters are displayed (Figure 129). Plug Name Password Status Boot/Seq. Delay Default internal_node A (undefined) ON 5 Secs ON 2 internal_node B (undefined) ON 5 Secs ON 3 (undefined) (undefined) ON 0.5 Secs ON 4 (undefined) (undefined) ON 0.5 Secs ON 5 internal_node A (undefined) ON 5 Secs ON 6 internal_node B (undefined) ON 5 Secs ON 7 (undefined) (undefined) ON 0.5 Secs ON 8 (undefined) (undefined) ON 0.5 Secs ON Figure 129. Plug parameters b. Verify the plug numbers and names. 10. Change the network parameters. a. Type /N at the IPS prompt. The Network Parameters menu is displayed. b. Type 2 to change the subnet mask. c. Type d. Type 1 to change the IP address. e. Type The IP address and subnet mask settings are displayed. f. Verify that the IP address and subnet mask settings are correct. g. Press the ESC key to exit the Network Parameters menu. 11. Type /E to save the parameters. 12. When prompted, type Y to confirm the change. 13. Exit the WTI by typing /X. 14. When prompted, type Y to confirm. 300 TS7650 and TS7650G Problem Determination and Service Guide

325 Appendix J. Overview of basic Linux commands The following reference lists some basic Linux commands. Basic Linux commands Table 43. Basic Linux commands Command Description cd change directory rm <filename> delete a file ls -l list the contents of a directory Mkdir <name> create a directory touch <name> create a file find / -name <name> search for a file or directory cat <name> view a file df -B G display disk space for each mount in Giga bytes, use M for Mega bytes ifconfig mv <oldname> <newname> pwd reboot shutdown sudo <command> cat /proc/scsi/scsi sets up network interfaces (show IP config) rename a file, can also be used to move a file show working directory restart the system shut down the system perform command as super user, might require a password shows attached disk system The following wesites can also be useful references for Linux commands: v Computer Hope v UNIXguide Copyright IBM Corp. 2011,

326 ProtecTIER Linux commands Table 44. ProtecTIER device-specific Linux commands and directories Command or Directory Description cat /proc/scsi/scsi shows attached disk system service vtfd status check if vtfd service is running - can also be used for other services such as cman, gfs, clvmd, and so on service vtfd start service vtfd stop rpm -qa grep RAS rpm -qa more cat /pt_work/log/vtf_event.log cat /pt_work/log/vtf_eror.log cat /pt_work/log/vtf_internal.log cat /pt_work/log/server.log cat /pt_work/log/name.log use to start vtfd - can also be used for other services such as cman, gfs, clvmd, and so on use to stop vtfd - can also be used for other services such as cman, gfs, clvmd, and so on show RAS version on server show all program packages displays the event log, including fibre channel logins, user commands, cartridge load or unload displays the error log displays the internal log displays the server log with info about fibre channel ports other logs are also available using this command where name is the name of the log you want to display - list all the logs with the command ls -l 302 TS7650 and TS7650G Problem Determination and Service Guide

327 Appendix K. Worldwide time zone codes Use the information in the following table to help you set the system's time zone. Time zone codes The following table lists all of the worldwide time zone codes and the associated time zone descriptions. Additional information about the time zone is located in the Comments column. Code Time zone Comments AD Europe/Andorra AE Asia/Dubai AF Asia/Kabul AG America/Antigua AI America/Anguilla AL Europe/Tirane AM Asia/Yerevan AN America/Curacao AO Africa/Luanda AQ Antarctica/McMurdo McMurdo Station, Ross Island AQ Antarctica/South_Pole Amundsen-Scott Station, South Pole AQ Antarctica/Rothera Rothera Station, Adelaide Island AQ Antarctica/Palmer Palmer Station, Anvers Island AQ Antarctica/Mawson Mawson Station, Holme Bay AQ Antarctica/Davis Davis Station, Vestfold Hills AQ Antarctica/Casey Casey Station, Bailey Peninsula AQ Antarctica/Vostok Vostok Station, S Magnetic Pole AQ Antarctica/DumontDUrville Dumont-d'Urville Station, Terre Adelie AQ Antarctica/Syowa Syowa Station, E Ongul I AR America/Argentina/Buenos_Aires Buenos Aires (BA, CF) AR America/Argentina/Cordoba most locations (CB, CC, CN, ER, FM, LP, MN, NQ, RN, SA, SE, SF, SL) AR America/Argentina/Jujuy Jujuy (JY) AR America/Argentina/Tucuman Tucuman (TM) AR America/Argentina/Catamarca Catamarca (CT), Chubut (CH) AR America/Argentina/La_Rioja La Rioja (LR) AR America/Argentina/San_Juan San Juan (SJ) AR America/Argentina/Mendoza Mendoza (MZ) AR America/Argentina/Rio_Gallegos Santa Cruz (SC) AR America/Argentina/Ushuaia Tierra del Fuego (TF) AS Pacific/Pago_Pago Copyright IBM Corp. 2011,

328 Code Time zone Comments AT Europe/Vienna AU Australia/Lord_Howe Lord Howe Island AU Australia/Hobart Tasmania - most locations AU Australia/Currie Tasmania - King Island AU Australia/Melbourne Victoria AU Australia/Sydney New South Wales - most locations AU Australia/Broken_Hill New South Wales - Yancowinna AU Australia/Brisbane Queensland - most locations AU Australia/Lindeman Queensland - Holiday Islands AU Australia/Adelaide South Australia AU Australia/Darwin Northern Territory AU Australia/Perth Western Australia - most locations AU Australia/Eucla Western Australia - Eucla area AW America/Aruba AX Europe/Mariehamn AZ Asia/Baku BA Europe/Sarajevo BB America/Barbados BD Asia/Dhaka BE Europe/Brussels BF Africa/Ouagadougou BG Europe/Sofia BH Asia/Bahrain BI Africa/Bujumbura BJ Africa/Porto-Novo BL America/St_Barthelemy BM Atlantic/Bermuda BN Asia/Brunei BO America/La_Paz BR America/Noronha Atlantic islands BR America/Belem Amapa, E Para BR America/Fortaleza NE Brazil (MA, PI, CE, RN, PB) BR America/Recife Pernambuco BR America/Araguaina Tocantins BR America/Maceio Alagoas, Sergipe BR America/Bahia Bahia BR America/Sao_Paulo S & SE Brazil (GO, DF, MG, ES, RJ, SP, PR, SC, RS) BR America/Campo_Grande Mato Grosso do Sul BR America/Cuiaba Mato Grosso BR America/Porto_Velho W Para, Rondonia BR America/Boa_Vista Roraima 304 TS7650 and TS7650G Problem Determination and Service Guide

329 Code Time zone Comments BR America/Manaus E Amazonas BR America/Eirunepe W Amazonas BR America/Rio_Branco Acre BS America/Nassau BT Asia/Thimphu BW Africa/Gaborone BY Europe/Minsk BZ America/Belize CA America/St_Johns Newfoundland Time, including SE Labrador CA America/Halifax Atlantic Time - Nova Scotia (most places), PEI CA America/Glace_Bay Atlantic Time - Nova Scotia - places that did not observe DST CA America/Moncton Atlantic Time - New Brunswick CA America/Goose_Bay Atlantic Time - Labrador - most locations CA America/Blanc-Sablon Atlantic Standard Time - Quebec - Lower North Shore CA America/Montreal Eastern Time - Quebec - most locations CA America/Toronto Eastern Time - Ontario - most locations CA America/Nipigon Eastern Time - Ontario & Quebec - places that did not observe DST CA America/Thunder_Bay Eastern Time - Thunder Bay, Ontario CA America/Iqaluit Eastern Time - east Nunavut - most locations CA America/Pangnirtung Eastern Time - Pangnirtung, Nunavut CA America/Resolute Eastern Time - Resolute, Nunavut CA America/Atikokan Eastern Standard Time - Atikokan, Ontario and Southampton I, Nunavut CA America/Rankin_Inlet Central Time - central Nunavut CA America/Winnipeg Central Time - Manitoba & west Ontario CA America/Rainy_River Central Time - Rainy River & Fort Frances, Ontario CA America/Regina Central Standard Time - Saskatchewan - most locations CA America/Swift_Current Central Standard Time - Saskatchewan - midwest CA America/Edmonton Mountain Time - Alberta, east British Columbia & west Saskatchewan CA America/Cambridge_Bay Mountain Time - west Nunavut CA America/Yellowknife Mountain Time - central Northwest Territories CA America/Inuvik Mountain Time - west Northwest Territories CA America/Dawson_Creek Mountain Standard Time - Dawson Creek & Fort Saint John, British Columbia CA America/Vancouver Pacific Time - west British Columbia CA America/Whitehorse Pacific Time - south Yukon CA America/Dawson Pacific Time - north Yukon CC Indian/Cocos Appendix K. Worldwide time zone codes 305

330 Code Time zone Comments CD Africa/Kinshasa west Dem. Rep. of Congo CD Africa/Lubumbashi east Dem. Rep. of Congo CF Africa/Bangui CG Africa/Brazzaville CH Europe/Zurich CI Africa/Abidjan CK Pacific/Rarotonga CL America/Santiago most locations CL Pacific/Easter Easter Island & Sala y Gomez CM Africa/Douala CN Asia/Shanghai east China - Beijing, Guangdong, Shanghai, etc. CN Asia/Harbin Heilongjiang (except Mohe), Jilin CN Asia/Chongqing central China - Sichuan, Yunnan, Guangxi, Shaanxi, Guizhou, etc. CN Asia/Urumqi most of Tibet & Xinjiang CN Asia/Kashgar west Tibet & Xinjiang CO America/Bogota CR America/Costa_Rica CU America/Havana CV Atlantic/Cape_Verde CX Indian/Christmas CY Asia/Nicosia CZ Europe/Prague DE Europe/Berlin DJ Africa/Djibouti DK Europe/Copenhagen DM America/Dominica DO America/Santo_Domingo DZ Africa/Algiers EC America/Guayaquil mainland EC Pacific/Galapagos Galapagos Islands EE Europe/Tallinn EG Africa/Cairo EH Africa/El_Aaiun ER Africa/Asmara ES Europe/Madrid mainland ES Africa/Ceuta Ceuta & Melilla ES Atlantic/Canary Canary Islands ET Africa/Addis_Ababa FI Europe/Helsinki FJ Pacific/Fiji 306 TS7650 and TS7650G Problem Determination and Service Guide

331 Code Time zone Comments FK Atlantic/Stanley FM Pacific/Truk Truk (Chuuk) and Yap FM Pacific/Ponape Ponape (Pohnpei) FM Pacific/Kosrae Kosrae FO Atlantic/Faroe FR Europe/Paris GA Africa/Libreville GB Europe/London GD America/Grenada GE Asia/Tbilisi GF America/Cayenne GG Europe/Guernsey GH Africa/Accra GI Europe/Gibraltar GL America/Godthab most locations GL America/Danmarkshavn east coast, north of Scoresbysund GL America/Scoresbysund Scoresbysund / Ittoqqortoormiit GL America/Thule Thule / Pituffik GM Africa/Banjul GN Africa/Conakry GP America/Guadeloupe GQ Africa/Malabo GR Europe/Athens GS Atlantic/South_Georgia GT America/Guatemala GU Pacific/Guam GW Africa/Bissau GY America/Guyana HK Asia/Hong_Kong HN America/Tegucigalpa HR Europe/Zagreb HT America/Port-au-Prince HU Europe/Budapest ID Asia/Jakarta Java & Sumatra ID Asia/Pontianak west & central Borneo ID Asia/Makassar east & south Borneo, Celebes, Bali, Nusa Tengarra, west Timor ID Asia/Jayapura Irian Jaya & the Moluccas IE IL IM Europe/Dublin Asia/Jerusalem Europe/Isle_of_Man Appendix K. Worldwide time zone codes 307

332 Code Time zone Comments IN Asia/Calcutta IO Indian/Chagos IQ Asia/Baghdad IR Asia/Tehran IS Atlantic/Reykjavik IT Europe/Rome JE Europe/Jersey JM America/Jamaica JO Asia/Amman JP Asia/Tokyo KE Africa/Nairobi KG Asia/Bishkek KH Asia/Phnom_Penh KI Pacific/Tarawa Gilbert Islands KI Pacific/Enderbury Phoenix Islands KI Pacific/Kiritimati Line Islands KM Indian/Comoro KN America/St_Kitts KP Asia/Pyongyang KR Asia/Seoul KW Asia/Kuwait KY America/Cayman KZ Asia/Almaty most locations KZ Asia/Qyzylorda Qyzylorda (Kyzylorda, Kzyl-Orda) KZ Asia/Aqtobe Aqtobe (Aktobe) KZ Asia/Aqtau Atyrau (Atirau, Gur'yev), Mangghystau (Mankistau) KZ Asia/Oral West Kazakhstan LA Asia/Vientiane LB Asia/Beirut LC America/St_Lucia LI Europe/Vaduz LK Asia/Colombo LR Africa/Monrovia LS Africa/Maseru LT Europe/Vilnius LU Europe/Luxembourg LV Europe/Riga LY Africa/Tripoli MA Africa/Casablanca MC Europe/Monaco MD Europe/Chisinau 308 TS7650 and TS7650G Problem Determination and Service Guide

333 Code Time zone Comments ME Europe/Podgorica MF America/Marigot MG Indian/Antananarivo MH Pacific/Majuro most locations MH Pacific/Kwajalein Kwajalein MK Europe/Skopje ML Africa/Bamako MM Asia/Rangoon MN Asia/Ulaanbaatar most locations MN Asia/Hovd Bayan-Olgiy, Govi-Altai, Hovd, Uvs, Zavkhan MN Asia/Choibalsan Dornod, Sukhbaatar MO Asia/Macau MP Pacific/Saipan MQ America/Martinique MR Africa/Nouakchott MS America/Montserrat MT Europe/Malta MU Indian/Mauritius MV Indian/Maldives MW Africa/Blantyre MX America/Mexico_City Central Time - most locations MX America/Cancun Central Time - Quintana Roo MX America/Merida Central Time - Campeche, Yucatan MX America/Monterrey Central Time - Coahuila, Durango, Nuevo Leon, Tamaulipas MX America/Mazatlan Mountain Time - S Baja, Nayarit, Sinaloa MX America/Chihuahua Mountain Time - Chihuahua MX America/Hermosillo Mountain Standard Time - Sonora MX America/Tijuana Pacific Time MY Asia/Kuala_Lumpur peninsular Malaysia MY Asia/Kuching Sabah & Sarawak MZ NA NC NE NF NG NI NL NO NP Africa/Maputo Africa/Windhoek Pacific/Noumea Africa/Niamey Pacific/Norfolk Africa/Lagos America/Managua Europe/Amsterdam Europe/Oslo Asia/Katmandu Appendix K. Worldwide time zone codes 309

334 Code Time zone Comments NR Pacific/Nauru NU Pacific/Niue NZ Pacific/Auckland most locations NZ Pacific/Chatham Chatham Islands OM Asia/Muscat PA America/Panama PE America/Lima PF Pacific/Tahiti Society Islands PF Pacific/Marquesas Marquesas Islands PF Pacific/Gambier Gambier Islands PG Pacific/Port_Moresby PH Asia/Manila PK Asia/Karachi PL Europe/Warsaw PM America/Miquelon PN Pacific/Pitcairn PR America/Puerto_Rico PS Asia/Gaza PT Europe/Lisbon mainland PT Atlantic/Madeira Madeira Islands PT Atlantic/Azores Azores PW Pacific/Palau PY America/Asuncion QA Asia/Qatar RE Indian/Reunion RO Europe/Bucharest RS Europe/Belgrade RU Europe/Kaliningrad Moscow-01 - Kaliningrad RU Europe/Moscow Moscow+00 - west Russia RU Europe/Volgograd Moscow+00 - Caspian Sea RU Europe/Samara Moscow+01 - Samara, Udmurtia RU Asia/Yekaterinburg Moscow+02 - Urals RU Asia/Omsk Moscow+03 - west Siberia RU Asia/Novosibirsk Moscow+03 - Novosibirsk RU Asia/Krasnoyarsk Moscow+04 - Yenisei River RU Asia/Irkutsk Moscow+05 - Lake Baikal RU Asia/Yakutsk Moscow+06 - Lena River RU Asia/Vladivostok Moscow+07 - Amur River RU Asia/Sakhalin Moscow+07 - Sakhalin Island RU Asia/Magadan Moscow+08 - Magadan RU Asia/Kamchatka Moscow+09 - Kamchatka 310 TS7650 and TS7650G Problem Determination and Service Guide

335 Code Time zone Comments RU Asia/Anadyr Moscow+10 - Bering Sea RW Africa/Kigali SA Asia/Riyadh SB Pacific/Guadalcanal SC Indian/Mahe SD Africa/Khartoum SE Europe/Stockholm SG Asia/Singapore SH Atlantic/St_Helena SI Europe/Ljubljana SJ Arctic/Longyearbyen SK Europe/Bratislava SL Africa/Freetown SM Europe/San_Marino SN Africa/Dakar SO Africa/Mogadishu SR America/Paramaribo ST Africa/Sao_Tome SV America/El_Salvador SY Asia/Damascus SZ Africa/Mbabane TC America/Grand_Turk TD Africa/Ndjamena TF Indian/Kerguelen TG Africa/Lome TH Asia/Bangkok TJ Asia/Dushanbe TK Pacific/Fakaofo TL Asia/Dili TM Asia/Ashgabat TN Africa/Tunis TO Pacific/Tongatapu TR Europe/Istanbul TT America/Port_of_Spain TV Pacific/Funafuti TW Asia/Taipei TZ Africa/Dar_es_Salaam UA Europe/Kiev most locations UA Europe/Uzhgorod Ruthenia UA Europe/Zaporozhye Zaporozh'ye, E Lugansk / Zaporizhia, E Luhansk UA Europe/Simferopol central Crimea Appendix K. Worldwide time zone codes 311

336 Code Time zone Comments UG Africa/Kampala UM Pacific/Johnston Johnston Atoll UM Pacific/Midway Midway Islands UM Pacific/Wake Wake Island US America/New_York Eastern Time US America/Detroit Eastern Time - Michigan - most locations US America/Kentucky/Louisville Eastern Time - Kentucky - Louisville area US America/Kentucky/Monticello Eastern Time - Kentucky - Wayne County US America/Indiana/Indianapolis Eastern Time - Indiana - most locations US America/Indiana/Vincennes Eastern Time - Indiana - Daviess, Dubois, Knox & Martin Counties US America/Indiana/Knox Eastern Time - Indiana - Starke County US America/Indiana/Winamac Eastern Time - Indiana - Pulaski County US America/Indiana/Marengo Eastern Time - Indiana - Crawford County US America/Indiana/Vevay Eastern Time - Indiana - Switzerland County US America/Chicago Central Time US America/Indiana/Tell_City Central Time - Indiana - Perry County US America/Indiana/Petersburg Central Time - Indiana - Pike County US America/Menominee Central Time - Michigan - Dickinson, Gogebic, Iron & Menominee Counties US America/North_Dakota/Center Central Time - North Dakota - Oliver County US America/North_Dakota/New_Salem Central Time - North Dakota - Morton County (except Mandan area) US America/Denver Mountain Time US America/Boise Mountain Time - south Idaho & east Oregon US America/Shiprock Mountain Time - Navajo US America/Phoenix Mountain Standard Time - Arizona US America/Los_Angeles Pacific Time US America/Anchorage Alaska Time US America/Juneau Alaska Time - Alaska panhandle US America/Yakutat Alaska Time - Alaska panhandle neck US America/Nome Alaska Time - west Alaska US America/Adak Aleutian Islands US Pacific/Honolulu Hawaii UY America/Montevideo UZ Asia/Samarkand west Uzbekistan UZ Asia/Tashkent east Uzbekistan VA Europe/Vatican VC America/St_Vincent VE America/Caracas VG America/Tortola VI America/St_Thomas 312 TS7650 and TS7650G Problem Determination and Service Guide

337 Code Time zone Comments VN Asia/Saigon VU Pacific/Efate WF Pacific/Wallis WS Pacific/Apia YE Asia/Aden YT Indian/Mayotte ZA Africa/Johannesburg ZM Africa/Lusaka ZW Africa/Harare Appendix K. Worldwide time zone codes 313

338 314 TS7650 and TS7650G Problem Determination and Service Guide

339 Accessibility for publications and ProtecTIER Manager The publications for this product are in Adobe Portable Document Format (PDF) and should be compliant with accessibility standards. Use these procedures to enable screen-reader compatibility, change the Windows contrast setting, and customize the color palette used in ProtecTIER Manager. About this task If you experience difficulties when you use the PDF files and want to request a Web-based format for a publication, send your request to the following address: International Business Machines Corporation Information Development Department GZW 9000 South Rita Road Tucson, Arizona U.S.A In the request, be sure to include the publication number and title. When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. About the Windows-based accessibility features About this task The accessibility features in ProtecTIER Manager help persons with limited vision use the ProtecTIER Manager installation wizard and software. After preparing the ProtecTIER Manager workstation for accessibility, you can use Windows-based screen-reader software and a digital voice synthesizer to hear what is displayed on the screen. The installation, configuration, and instructional screens in the Windows versions of the ProtecTIER Manager installation wizard and the ProtecTIER Manager software have been tested with JAWS. However, the associated diagrams and graphs in ProtecTIER Manager and ProtecTIER Replication Manager, do not currently support keyboard navigation or screen-reader use. You can obtain full system statistics (typically provided in the diagrams and graphs) by going to the ProtecTIER Manager toolbar and clicking: Reports > Create long term statistics report, and downloading the results. To enable screen-reader compatibility, you must prepare the ProtecTIER Manager workstation by completing these tasks. Instructions are provided in the topics that follow: Before you install ProtecTIER Manager: v Download and install the Java Runtime Environment (JRE). v Download and install the Java Access Bridge (JAB). After you install ProtecTIER Manager: Copyright IBM Corp. 2011,

340 About the Java-based tools About this task v v v Change the ProtecTIER Manager preferences to enable support of the Windows system settings (required) Select a high-contrast color scheme in Windows (optional) Customize the color palette used in the ProtecTIER Manager display (optional) Complete the following the procedures to download and install the Java-based tools that are required to enable full screen-reader compatibility on the ProtecTIER Manager workstation. Install the Java Runtime Environment (JRE) first, and then install the Java Access Bridge (JAB). Both of these tools must be installed before you install the ProtecTIER Manager software. For simplicity, download the Java-based tools by using the ProtecTIER Manager workstation on which you are installing the JRE and JAB. If this is not possible, try to use another computer that is running Windows. Installing the Java Runtime Environment About this task The JRE includes the Java Virtual Machine (JVM). These tools are necessary for your computer to run Java-based applications. Procedure 1. Go to The Java website opens. The java.com website auto-detects the operating system and Internet browser of the computer you use when you access the site. 2. Click Free Java Download, and proceed as appropriate: v If the Download Java for Windows page opens, go on to step 3 v If the Download Java for... page title contains the name of an operating system other than Windows, do the following: a. Click the See all downloads here link. The list of available downloads, categorized by operating system, displays. b. In the Windows section, click Windows 7/XP/Vista/2000/2003/2008 Online. 3. Review the information provided, and then click Agree and Start Free Download. The download dialog box opens. 4. Follow the on-screen instructions to save the executable (.exe) installer file to the hard disk drive. 5. After the download is complete, find the installer file on the hard disk drive and write down the full path to the location of the file. For example: C:\Program Files\Java\jre6\bin\java.exe. This path is needed during ProtecTIER Manager installation. 6. Proceed as appropriate: 316 TS7650 and TS7650G Problem Determination and Service Guide

341 v If you downloaded the installer on the ProtecTIER Manager workstation on which you are installing the JRE, go on to step 7. v If you downloaded the installer on a PC other than the applicable ProtecTIER Manager workstation, do the following: a. Copy the installer file onto a CD, flash memory drive, or other form of removable media. b. Copy the installer file from the removable media to the hard disk drive of the ProtecTIER Manager workstation. c. Go on to step Double-click the installer file to start the Java installation wizard. The Java Setup Welcome window opens. 8. Click Install and follow the on-screen instructions to complete the installation process. 9. When you have successfully installed the JRE, go on to Installing the Java Access Bridge. Installing the Java Access Bridge About this task The Java Access Bridge (JAB) makes it possible for you to use Java-based screen readers with the ProtecTIER Manager installation wizard and software. Procedure 1. Go to: The Java SE Desktop Accessibility page of the Oracle website opens. 2. Read the information provided, then click Access Bridge. 3. Scroll down to the Java Access Bridge for Microsoft Windows Operating System x.x.x (where x.x.x is the most recent version listed) section. Click the Download Java Access Bridge x.x.x link. The Software License Agreement page opens. 4. Read the license agreement, and then select the I agree to the Software License Agreement check box. The Download Java Access Bridge for Windows Operating System x.x.x page opens. 5. In the Required Files list, click the link to download the Access Bridge x.x.x, accessbridge-x.x.x.exe file. The download dialog box opens. 6. Follow the on-screen instructions to save the executable (.exe) installer file to the hard disk drive. 7. When the download is complete, locate the installer file on the hard disk drive and proceed as appropriate: v If you downloaded the installer by using the ProtecTIER Manager workstation on which you are installing the JAB, go on to step 8 on page 318. v If you downloaded the installer by using a PC other than the applicable ProtecTIER Manager workstation, do the following: a. Copy the installer file onto a CD, flash memory drive, or other removable media device. b. Copy the installer file from the removable media device to the hard disk drive of the ProtecTIER Manager workstation. Accessibility for publications and ProtecTIER Manager 317

342 c. Go on to step On the ProtecTIER Manager workstation, double-click the accessbridge-x.x.x.exe installer file. A security warning dialog box displays. 9. Click Run. The Java Access Bridge InstallShield Wizard opens. 10. Read the welcome information, then click Next and follow the on-screen instructions to complete the installation. 11. When the installation is complete, restart the workstation as directed. You now have the necessary Java tools for compatibility between the ProtecTIER Manager installation wizard and screen reader software. 12. Follow the instructions in Using a screen reader to install ProtecTIER Manager to start the ProtecTIER Manager installation wizard by using a screen reader. Using a screen reader to install ProtecTIER Manager About this task Install ProtecTIER Manager according to the following command line-based instructions. When entering the commands, type them exactly as shown, including any spaces or quotation marks. Any deviation in the procedure can cause the installation to start in the non-accessible mode, or fail completely. Procedure 1. If your workstation is configured to automatically open DVDs, temporarily disable the Windows AutoPlay feature for the CD/DVD device. Use the Windows Help or other Windows documentation for instructions, and then go on to step Insert the IBM System Storage ProtecTIER Manager V3.2 DVD into the CD/DVD drive of the ProtecTIER Manager workstation. 3. Access the command prompt on the ProtecTIER Manager workstation: a. Click Start > Run... The Run dialog box opens. 4. In the Open field, type: cmd and click Ok. The command window opens. 5. Browse to the ProtecTIER Manager installation directory on the DVD. To do so: a. At the command prompt, type: D: (where D: is the letter assigned to the CD/DVD drive of the workstation) and press <enter>. b. At the command prompt, list the contents of the DVD. Type: dir and press <enter>. c. Locate the name of the ProtecTIER Manager directory on the DVD. For example: PT_Manager_V3.2. d. At the command prompt, change to the ProtecTIER Manager directory. Type: cd <directory name> and press <enter>. For example: cd PT_Manager_V3.2 <enter>. e. At the command prompt, change to the Windows directory. Type: cd windows and press <enter>. 318 TS7650 and TS7650G Problem Determination and Service Guide

343 f. At the command prompt, type: Install.exe LAX_VM "C:\Program Files\ Java60\jre\bin\java.exe" and press <enter>, where the path contained within the quotation marks is the same as the path that you noted in step 5 on page 316. The screen-reader-enabled ProtecTIER Manager installation wizard starts. g. Follow the spoken prompts to compete the installation. 6. When the installation completes, proceed as appropriate: v If you do not want to enable the Windows High Contrast option or customize the color palette, resume your regular use of ProtecTIER Manager. v To change the contrast mode for ProtecTIER Manager, go to Enabling the Windows High Contrast option. To customize the color palette, go to Customizing the color palette on page 324. Enabling the Windows High Contrast option About this task To make it possible for ProtecTIER Manager display in high contrast, you must first enable the Use High Contrast option in Windows. Procedure 1. On the ProtecTIER Manager workstation, go to Windows > Control Panel > Accessibility Options. The Accessibility Options dialog box opens. 2. Select the Display tab. 3. In the High Contrast area of the Display tab, select the Use High Contrast check box, as shown in Figure 130 on page 320: Accessibility for publications and ProtecTIER Manager 319

344 ts Figure 130. Display tab 4. Click Settings. The Settings for High Contrast dialog box displays, as shown in Figure 131 on page 321: 320 TS7650 and TS7650G Problem Determination and Service Guide

345 ts Figure 131. Settings for High Contrast By default, the High Contrast Black (large) scheme is selected. 5. Do one of the following: v v To use the default, High Contrast Black (large), scheme: a. Click Ok to close the Settings for High Contrast dialog box. b. Click Ok to close the Accessibility Options dialog box. After a few moments, the display changes to the new color scheme. c. Go on to Using the Windows high contrast scheme with ProtecTIER Manager. To use a different high contrast scheme: a. Click the arrow to show the list of available color schemes. b. Select the high contrast scheme that you want to use. c. Click Ok to close the Settings for High Contrast dialog box. d. Click Ok to close the Accessibility Options dialog box. After a few moments, the display changes to the new color scheme. e. Go on to Using the Windows high contrast scheme with ProtecTIER Manager. Using the Windows high contrast scheme with ProtecTIER Manager About this task Now that you have changed the contrast scheme in Windows, you must enable the Support system settings option in ProtecTIER Manager. Procedure 1. Launch ProtecTIER Manager: Accessibility for publications and ProtecTIER Manager 321

346 a. Click: Start > All Programs > IBM > ProtecTIER Manager > IBM ProtecTIER Manager. The ProtecTIER Manager window opens, as shown in: Figure 132. ts Figure 132. ProtecTIER Manager window 2. On the toolbar, click: Tools > Preferences. The Preferences dialog box opens with the Appearance tab selected, as shown in Figure 133 on page 323: 322 TS7650 and TS7650G Problem Determination and Service Guide

347 ts Figure 133. Preferences dialog box 3. On the Appearance tab, select the Support system settings check box. You are returned to the ProtecTIER Manager window. 4. Exit and restart ProtecTIER Manager so the contrast settings take effect: a. On the ProtecTIER Manager toolbar, click: File > Exit. The ProtecTIER Manager window closes. b. Click: Start > All Programs > IBM > ProtecTIER Manager > IBM ProtecTIER Manager. When the ProtecTIER Manager window opens, the display reflects the contrast change, as shown in: Figure 134. ts Figure 134. Normal contrast versus high contrast 5. Proceed as appropriate: v If you want to change one or more of the colors used in the ProtecTIER Manager display, continue to Customizing the color palette on page 324. v If you do not want to customize the color palette, resume your regular use of ProtecTIER Manager. Accessibility for publications and ProtecTIER Manager 323

348 Customizing the color palette About this task Use this procedure to customize the color palette for ProtecTIER Manager to improve visibility in the display, or to suit your personal preferences. Procedure 1. If necessary, start ProtecTIER Manager as described in step 1 on page Open the Preferences dialog box, as described in 2 on page Scroll down (if necessary) to see the entire Color selection list, and then select the color you want to change. The Color selection dialog box opens, with the Swatches tab selected, as shown in Figure 135: ts Figure 135. Color selection, Swatches tab The color that is currently defined for your selection is shown in the Preview pane. 4. Select a new color from the color palette. You can also specify a new color by using the Hue/Saturation/Brightness (HSB) or Red/Green/Blue (RGB) color models. To do so, click the tab for the model you want to use and enter the required values. 5. When you have finished selecting or specifying the new color, click Ok. You are returned to the Appearance tab. 6. To change another color, repeat steps 3 through TS7650 and TS7650G Problem Determination and Service Guide

349 7. When you are finished making changes in the Appearance tab, click Ok. You are returned to the ProtecTIER Manager window. 8. Exit and restart ProtecTIER Manager (as described in step 4 on page 323) so the color palette changes take effect. After you log in to ProtecTIER Manager and add a node, the display reflects your custom color selections. An example of the default color versus a custom color for Allocable resources, is shown in: Figure 136 ts Figure 136. Default color versus custom color 9. Proceed as appropriate. Return to the task from which you were sent to these instructions or resume your regular use of ProtecTIER Manager. Accessibility for publications and ProtecTIER Manager 325

350 326 TS7650 and TS7650G Problem Determination and Service Guide

351 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATIONS "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been Copyright IBM Corp. 2011,

352 estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: v AIX v DS4000 v Enterprise Storage Server v ESCON v FICON v i5/os v iseries v IBM v ProtecTIER v pseries v S/390 v ServeRAID v System x v System Storage v TotalStorage v Wake on LAN v z/os v zseries IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ((R) or (TM)), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or 328 TS7650 and TS7650G Problem Determination and Service Guide

353 Electronic emission notices common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Java and all Java-based trademarks are trademarks of Oracle, Inc. in the United States, other countries, or both. Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. This section contains the electronic emission notices or statements for the United States and other regions. Federal Communications Commission statement This explains the Federal Communications Commission's (FCC) statement. This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruction manual, might cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits. IBM is not responsible for any radio or television interference caused by using other than recommended cables and connectors, or by unauthorized changes or modifications to this equipment. Unauthorized changes or modifications could void the user's authority to operate the equipment. Notices 329

354 This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) this device might not cause harmful interference, and (2) this device must accept any interference received, including interference that might cause undesired operation. Industry Canada compliance statement This Class A digital apparatus complies with Canadian ICES-003. Cet appareil numérique de la classe A est conform à la norme NMB-003 du Canada. European Union Electromagnetic Compatibility Directive This product is in conformity with the protection requirements of European Union (EU) Council Directive 2004/108/EC on the approximation of the laws of the Member States relating to electromagnetic compatibility. IBM cannot accept responsibility for any failure to satisfy the protection requirements resulting from a non-recommended modification of the product, including the fitting of non-ibm option cards. Attention: This is an EN Class A product. In a domestic environment this product might cause radio interference in which case the user might be required to take adequate measures. Responsible Manufacturer: International Business Machines Corp. New Orchard Road Armonk, New York European community contact: IBM Deutschland GmbH Technical Regulations, Department M372 IBM-Allee 1, Ehningen, Germany Tele: lugi@de.ibm.com Australia and New Zealand Class A Statement Attention: This is a Class A product. In a domestic environment this product might cause radio interference in which case the user might be required to take adequate measures. Germany Electromagnetic compatibility directive Deutschsprachiger EU Hinweis: Hinweis für Geräte der Klasse A EU-Richtlinie zur Elektromagnetischen Verträglichkeit Dieses Produkt entspricht den Schutzanforderungen der EU-Richtlinie 2004/108/EG zur Angleichung der Rechtsvorschriften über die elektromagnetische Verträglichkeit in den EU-Mitgliedsstaaten und hält die Grenzwerte der EN Klasse A ein. Um dieses sicherzustellen, sind die Geräte wie in den Handbüchern beschrieben zu installieren und zu betreiben. Des Weiteren dürfen auch nur von der IBM 330 TS7650 and TS7650G Problem Determination and Service Guide

355 empfohlene Kabel angeschlossen werden. IBM übernimmt keine Verantwortung für die Einhaltung der Schutzanforderungen, wenn das Produkt ohne Zustimmung der IBM verändert bzw. wenn Erweiterungskomponenten von Fremdherstellern ohne Empfehlung der IBM gesteckt/eingebaut werden. EN Klasse A Geräte müssen mit folgendem Warnhinweis versehen werden: "Warnung: Dieses ist eine Einrichtung der Klasse A. Diese Einrichtung kann im Wohnbereich Funk-Störungen verursachen; in diesem Fall kann vom Betreiber verlangt werden, angemessene Mabnahmen zu ergreifen und dafür aufzukommen." Deutschland: Einhaltung des Gesetzes über die elektromagnetische Verträglichkeit von Geräten Dieses Produkt entspricht dem "Gesetz über die elektromagnetische Verträglichkeit von Geräten (EMVG)." Dies ist die Umsetzung der EU-Richtlinie 2004/108/EG in der Bundesrepublik Deutschland. Zulassungsbescheinigung laut dem Deutschen Gesetz über die elektromagnetische Verträglichkeit von Geräten (EMVG) (bzw. der EMC EG Richtlinie 2004/108/EG) für Geräte der Klasse A Dieses Gerät ist berechtigt, in übereinstimmung mit dem Deutschen EMVG das EG-Konformitätszeichen - CE - zu führen. Verantwortlich für die Einhaltung der EMV Vorschriften ist der Hersteller: International Business Machines Corp. New Orchard Road Armonk,New York Tel: Der verantwortliche Ansprechpartner des Herstellers in der EU ist: IBM Deutschland GmbH Technical Regulations, Abteilung M372 IBM-Allee 1, Ehningen, Germany Tele: lugi@de.ibm.com Generelle Informationen: Das Gerät erfüllt die Schutzanforderungen nach EN und EN Klasse A. Notices 331

356 People's Republic of China Class A Electronic Emission statement Taiwan Class A compliance statement Taiwan contact information This topic contains the product service contact information for Taiwan. IBM Taiwan Product Service Contact Information: IBM Taiwan Corporation 3F, No 7, Song Ren Rd., Taipei Taiwan Tel: f2c00790 Japan Voluntary Control Council for Interference (VCCI) Class A Statement 332 TS7650 and TS7650G Problem Determination and Service Guide

357 Japan Electronics and Information Technology Industries Association (JEITA) Statement jjieta Korean Communications Commission (KCC) Class A Statement Russia Electromagnetic Interference (EMI) Class A Statement rusemi Notices 333