Troubleshooting Guide

Transcription

1 IBM System Storge SAN Volume Controller Troubleshooting Guide GC

2 Note Before using this informtion nd the product it supports, red the informtion in Notices on pge 351. This edition pplies to IBM SAN Volume Controller, Version 7.3, nd to ll subsequent releses nd modifictions until otherwise indicted in new editions. Copyright IBM Corportion 2003, US Government Users Restricted Rights Use, dupliction or disclosure restricted by GSA ADP Schedule Contrct with IBM Corp.

3 Contents Figures vii Tbles ix About this guide xi Who should use this guide xi Emphsis xi SAN Volume Controller librry nd relted publictions xi How to order IBM publictions xiv Relted websites xv Sending your comments xv How to get informtion, help, nd technicl ssistnce xv Summry of chnges for GC SAN Volume Controller Troubleshooting Guide.... xvii Summry of chnges for GC SAN Volume Controller Troubleshooting Guide... xviii Summry of chnges for GC SAN Volume Controller Troubleshooting Guide... xviii Chpter 1. SAN Volume Controller overview Systems Configurtion node Configurtion node ddressing Mngement IP filover SAN fbric overview Chpter 2. Introducing the SAN Volume Controller hrdwre components SAN Volume Controller nodes SAN Volume Controller controls nd indictors.. 9 SAN Volume Controller opertor-informtion pnel SAN Volume Controller rer-pnel indictors nd connectors Fibre Chnnel port numbers nd worldwide port nmes Requirements for the SAN Volume Controller environment Redundnt c-power switch Redundnt c-power environment requirements 46 Cbling of redundnt c-power switch (exmple) 47 Uninterruptible power supply UPS-1U Uninterruptible power-supply environment requirements Defining the SAN Volume Controller FRUs SAN Volume Controller FRUs Redundnt c-power switch FRUs Chpter 3. SAN Volume Controller user interfces for servicing your system.. 65 Mngement GUI interfce When to use the mngement GUI Accessing the mngement GUI Deleting node from clustered system using the mngement GUI Adding nodes to clustered system Service ssistnt interfce When to use the service ssistnt Accessing the service ssistnt Commnd-line interfce When to use the CLI Accessing the system CLI Service commnd-line interfce When to use the service CLI Accessing the service CLI Front pnel interfce Technicin port for node ccess Chpter 4. Performing recovery ctions using the SAN Volume Controller CLI. 77 Vlidting nd repiring mirrored volume copies using the CLI Repiring thin-provisioned volume using the CLI 78 Recovering from offline volumes using the CLI.. 79 Chpter 5. Viewing the vitl product dt Viewing the vitl product dt using the mngement GUI Displying the vitl product dt using the CLI.. 81 Displying node properties using the CLI Displying clustered system properties using the CLI Fields for the node VPD Fields for the system VPD Chpter 6. Using the front pnel of the SAN Volume Controller Boot progress indictor Boot filed Chrging Error codes Hrdwre boot Node rescue request Power filure Powering off Recovering Restrting Shutting down Vlidte WWNN? option SAN Volume Controller menu options Cluster (system) options Copyright IBM Corp. 2003, 2014 iii

4 Node options Version options Ethernet options Fibre Chnnel port options Actions options Lnguge? option Using the power control for the SAN Volume Controller node Chpter 7. Dignosing problems Strting sttistics collection Event reporting Power-on self-test Understnding events Mnging the event log Viewing the event log Describing the fields in the event log Event notifictions Inventory informtion emil Understnding the error codes Using the error code tbles Event IDs SCSI event reporting Object types Error event IDs nd error codes Resolving problem with the SAN Volume Controller 2145-DH8 boot drives Determining hrdwre boot filure Boot code reference Node error code overview Clustered-system code overview Error code rnge Procedure: SAN problem determintion Resolving problem with new expnsion enclosures Fibre Chnnel nd 10G Ethernet link filures Ethernet iscsi host-link problems Fibre Chnnel over Ethernet host-link problems 240 Servicing storge systems Chpter 8. Disster recovery Chpter 9. Recovery procedures Recover system procedure When to run the recover system procedure Fix hrdwre errors Removing clustered system informtion for nodes with error code 550 or error code 578 using the front pnel Removing system informtion for nodes with error code 550 or error code 578 using the service ssistnt Performing recovery procedure for clustered systems using the front pnel Running system recovery using the service ssistnt Recovering from offline volumes using the CLI 252 Wht to check fter running the system recovery Bcking up nd restoring the system configurtion 253 Bcking up the system configurtion using the CLI Restoring the system configurtion Deleting bckup configurtion files using the CLI Performing the node rescue when the node boots 260 Chpter 10. Understnding the medium errors nd bd blocks Chpter 11. Using the mintennce nlysis procedures MAP 5000: Strt MAP 5040: Power SAN Volume Controller 2145-DH MAP 5050: Power 2145-CG8, 2145-CF8, nd G MAP 5060: Power A MAP 5150: 2145 UPS-1U MAP 5250: 2145 UPS-1U repir verifiction MAP 5320: Redundnt c power MAP 5340: Redundnt c power verifiction MAP 5350: Powering off SAN Volume Controller node Using the mngement GUI to power off system Using the SAN Volume Controller CLI to power off node Using the SAN Volume Controller power control button MAP 5400: Front pnel MAP 5500: Ethernet Defining n lternte configurtion node MAP 5550: 10G Ethernet nd Fibre Chnnel over Ethernet personlity enbled dpter port MAP 5600: Fibre Chnnel MAP 5700: Repir verifiction MAP 5800: Light pth Light pth for SAN Volume Controller 2145-DH Light pth for SAN Volume Controller 2145-CF8 326 Light pth for SAN Volume Controller A4 331 Light pth for SAN Volume Controller G4 334 MAP 5900: Hrdwre boot MAP 6000: Replce offline SSD MAP 6001: Replce offline SSD in RAID 0 rry MAP 6002: Replce offline SSD in RAID 1 rry or RAID 10 rry Chpter 12. iscsi performnce nlysis nd tuning Notices Trdemrks Electronic emission notices Federl Communictions Commission (FCC) sttement Industry Cnd complince sttement iv SAN Volume Controller: Troubleshooting Guide

5 Austrli nd New Zelnd Clss A Sttement 354 Europen Union Electromgnetic Comptibility Directive Germny Electromgnetic Comptibility Directive People's Republic of Chin Clss A Sttement 355 Tiwn Clss A complince sttement Tiwn Contct Informtion Jpn VCCI Council Clss A sttement Jpn Electronics nd Informtion Technology Industries Assocition Sttement Koren Communictions Commission Clss A Sttement Russi Electromgnetic Interference Clss A Sttement Index Contents v

6 vi SAN Volume Controller: Troubleshooting Guide

7 Figures 1. SAN Volume Controller system in fbric 2 2. Dt flow in SAN Volume Controller system 3 3. SAN Volume Controller nodes with internl Flsh drives Configurtion node SAN Volume Controller 2145-DH8 front pnel SAN Volume Controller 2145-CG8 front pnel SAN Volume Controller 2145-CF8 front pnel SAN Volume Controller A4 front-pnel ssembly SAN Volume Controller G4 front-pnel ssembly SAN Volume Controller 2145-DH8 opertor informtion pnel SAN Volume Controller 2145-CG8 or 2145-CF8 opertor-informtion pnel SAN Volume Controller 2145-CG8 or 2145-CF8 opertor-informtion pnel SAN Volume Controller A4 opertor-informtion pnel SAN Volume Controller G4 opertor-informtion pnel SAN Volume Controller 2145-CG8 rer-pnel indictors SAN Volume Controller 2145-CG8 rer-pnel indictors for the 10 Gbps Ethernet feture Connectors on the rer of the SAN Volume Controller 2145-CG Gbps Ethernet ports on the rer of the SAN Volume Controller 2145-CG Power connector Service ports of the SAN Volume Controller 2145-CG SAN Volume Controller 2145-CG8 port not used SAN Volume Controller 2145-CF8 rer-pnel indictors Connectors on the rer of the SAN Volume Controller 2145-CG8 or 2145-CF Power connector Service ports of the SAN Volume Controller 2145-CF SAN Volume Controller 2145-CF8 port not used SAN Volume Controller A4 rer-pnel indictors SAN Volume Controller A4 externl connectors Power connector Service ports of the SAN Volume Controller A SAN Volume Controller G4 rer-pnel indictors SAN Volume Controller G4 externl connectors Power connector Service ports of the SAN Volume Controller G SAN Volume Controller 2145-CG8 or 2145-CF8 c, dc, nd power-error LEDs SAN Volume Controller G4 c nd dc LEDs Photo of the redundnt c-power switch A four-node SAN Volume Controller system with the redundnt c-power switch feture Rck cbling exmple UPS-1U front-pnel ssembly UPS-1U connectors nd switches UPS-1U dip switches Ports not used by the 2145 UPS-1U Power connector SAN Volume Controller front-pnel ssembly Exmple of boot progress disply Exmple of n error code for clustered system Exmple of node error code Node rescue disply Vlidte WWNN? nvigtion SAN Volume Controller options on the front-pnel disply Viewing the IPv6 ddress on the front-pnel disply Upper options of the ctions menu on the front pnel Middle options of the ctions menu on the front pnel Lower options of the ctions menu on the front pnel Lnguge? nvigtion Exmple of boot error code Exmple of boot progress disply Exmple of displyed node error code Exmple of node-rescue error code Exmple of crete error code for clustered system Exmple of recovery error code Exmple of n error code for clustered system Node rescue disply SAN Volume Controller 2145-DH8 front pnel Error LED on the SAN Volume Controller models SAN Volume Controller 2145-DH8 opertor-informtion pnel Hrdwre boot disply SAN Volume Controller 2145-DH8 front pnel Power LED on the SAN Volume Controller 2145-DH Power LED indictor on the rer pnel of the SAN Volume Controller 2145-DH Copyright IBM Corp. 2003, 2014 vii

8 72. AC, dc, nd power-supply error LED indictors on the rer pnel of the SAN Volume Controller 2145-DH Power LED on the SAN Volume Controller models 2145-CG8, 2145-CF8, or G4 opertor-informtion pnel Power LED indictor on the rer pnel of the SAN Volume Controller G Power LED indictor on the rer pnel of the SAN Volume Controller 2145-CG8 or 2145-CF AC nd dc LED indictors on the rer pnel of the SAN Volume Controller G Power LED indictor nd c nd dc indictors on the rer pnel of the SAN Volume Controller 2145-CG8 or 2145-CF Power LED on the SAN Volume Controller A4 opertor-informtion pnel SAN Volume Controller A4 system bord LEDs UPS-1U front-pnel ssembly Power control button on the SAN Volume Controller models SAN Volume Controller service controller error light Front-pnel disply when push buttons re pressed Port 2 Ethernet link LED on the SAN Volume Controller rer pnel Ethernet ports on the rer of the SAN Volume Controller 2145-DH SAN Volume Controller 2145-DH8 opertor-informtion pnel Press the relese ltch SAN Volume Controller 2145-DH8 light pth dignostics pnel SAN Volume Controller 2145-DH8 system bord LEDs SAN Volume Controller 2145-CG8 or 2145-CF8 opertor-informtion pnel SAN Volume Controller 2145-CG8 or 2145-CF8 light pth dignostics pnel SAN Volume Controller 2145-CF8 system bord LEDs dignostics pnel SAN Volume Controller A4 opertor-informtion pnel SAN Volume Controller A4 system bord LEDs SAN Volume Controller G4 opertor-informtion pnel SAN Volume Controller G4 light pth dignostics pnel SAN Volume Controller G4 system bord LEDs Hrdwre boot disply Node rescue disply Keybord nd monitor ports on the SAN Volume Controller models G4 nd2145-8a Keybord nd monitor ports on the SAN Volume Controller 2145-CF Keybord nd monitor ports on the SAN Volume Controller 2145-CG Keybord nd monitor ports on the SAN Volume Controller 2145-DH8, front Keybord nd monitor ports on the SAN Volume Controller 2145-DH8, rer viii SAN Volume Controller: Troubleshooting Guide

9 Tbles 1. IBM websites for help, services, nd informtion xii 2. SAN Volume Controller librry xii 3. Other IBM publictions xiv 4. IBM documenttion nd relted websites xiv 5. IBM websites for help, services, nd informtion xv 6. SAN Volume Controller communictions types 4 7. Link stte nd ctivity for the bottom Fibre Chnnel LED Link speed for the top Fibre Chnnel LED Actul link speeds Actul link speeds Input-voltge requirements Mximum power consumption Physicl specifictions Dimensions nd weight Additionl spce requirements Mximum het output of ech 2145-DH8 node Input-voltge requirements Mximum power consumption Environment requirements without redundnt c power Environment requirements with redundnt c power Dimensions nd weight Additionl spce requirements Mximum het output of ech SAN Volume Controller 2145-CG8 node Mximum het output of ech 2145 UPS-1U Input-voltge requirements Power requirements for ech node Environment requirements without redundnt c power Environment requirements with redundnt c power Dimensions nd weight Additionl spce requirements Het output of ech SAN Volume Controller 2145-CF8 node Input-voltge requirements Power requirements for ech node Environment requirements without redundnt c power Environment requirements with redundnt c power Dimensions nd weight Additionl spce requirements Het output of ech SAN Volume Controller A4 node Input-voltge requirements Power requirements for ech node Environment requirements without redundnt c power Environment requirements with redundnt c power Dimensions nd weight Additionl spce requirements Het output of ech SAN Volume Controller G4 node FRUs for SAN Volume Controller F expnsion enclosure FRU prt for the SAN Volume Controller F SAS drive units SAN Volume Controller 2145-CG8 FRU descriptions SAN Volume Controller 2145-CF8 FRU descriptions Ethernet feture FRU descriptions Flsh drive feture FRU descriptions UPS-1U FRU descriptions SAN Volume Controller A4 FRU descriptions SAN Volume Controller G4 FRU descriptions Fields for the system bord Fields for the btteries Fields for the processors Fields for the fns Fields tht re repeted for ech instlled memory module Fields tht re repeted for ech dpter tht is instlled Fields tht re repeted for ech SCSI, IDE, SATA, nd SAS device tht is instlled Fields tht re specific to the node softwre Fields tht re provided for the front pnel ssembly Fields tht re provided for the Ethernet port Fields tht re provided for the power supplies in the node Fields tht re provided for the uninterruptible power supply ssembly tht is powering the node Fields tht re provided for the SAS host bus dpter (HBA) Fields tht re provided for the SAS flsh drive Fields tht re provided for the smll form fctor pluggble (SFP) trnsceiver Fields tht re provided for the system properties When options re vilble Sttistics collection for individul nodes Sttistic collection for volumes for individul nodes Sttistic collection for volumes tht re used in Metro Mirror nd Globl Mirror reltionships for individul nodes Sttistic collection for node ports Sttistic collection for nodes Copyright IBM Corp. 2003, 2014 ix

10 77. Cche sttistics collection for volumes nd volume copies Sttistic collection for volume cche per individul nodes XML sttistics for n IP Repliction port Description of dt fields for the event log Notifiction levels SAN Volume Controller notifiction types nd corresponding syslog level codes SAN Volume Controller vlues of user-defined messge origin identifiers nd syslog fcility codes Informtionl events SCSI sttus SCSI sense keys, codes, nd qulifiers Reson codes Object types Error event IDs nd error codes Messge clssifiction number rnge Files creted by the bckup process Bd block errors UPS-1U error indictors Fibre Chnnel ssemblies SAN Volume Controller Fibre Chnnel dpter connection hrdwre Dignostics pnel LED Dignostics pnel LED prescribed ctions SAN Volume Controller A4 dignostics pnel LED prescribed ctions Dignostics pnel LED prescribed ctions 337 x SAN Volume Controller: Troubleshooting Guide

11 About this guide This guide describes how to troubleshoot the IBM System Storge SAN Volume Controller. The chpters tht follow introduce you to the SAN Volume Controller, expnsion enclosure, the redundnt c-power switch, nd the uninterruptible power supply. They describe how you cn configure nd check the sttus of one SAN Volume Controller node or clustered system of nodes through the front pnel, with the service ssistnt GUI, or with the mngement GUI. Who should use this guide The vitl product dt (VPD) chpter provides informtion bout the VPD tht uniquely defines ech hrdwre nd microcode element tht is in the SAN Volume Controller. You cn lso lern how to dignose problems using the SAN Volume Controller. The mintennce nlysis procedures (MAPs) cn help you nlyze filures tht occur in SAN Volume Controller. With the MAPs, you cn isolte the field-replceble units (FRUs) of the SAN Volume Controller tht fil. Begin ll problem determintion nd repir procedures from MAP 5000: Strt on pge 265. This guide is intended for system dministrtors or systems services representtives who use nd dignose problems with the SAN Volume Controller, the redundnt c-power switch, nd the uninterruptible power supply. Emphsis Different typefces re used in this guide to show emphsis. The following typefces re used to show emphsis: Boldfce Bold monospce Itlics Monospce Text in boldfce represents menu items. Text in bold monospce represents commnd nmes. Text in itlics is used to emphsize word. In commnd syntx, it is used for vribles for which you supply ctul vlues, such s defult directory or the nme of system. Text in monospce identifies the dt or commnds tht you type, smples of commnd output, exmples of progrm code or messges from the system, or nmes of commnd flgs, prmeters, rguments, nd nme-vlue pirs. SAN Volume Controller librry nd relted publictions Product mnuls, other publictions, nd websites contin informtion tht reltes to SAN Volume Controller. Copyright IBM Corp. 2003, 2014 xi

12 SAN Volume Controller Informtion Center The IBM Informtion Center contins ll of the informtion tht is required to instll, configure, nd mnge the system. The informtion center is updted between product releses to provide the most current documenttion. The informtion center is vilble t the following website: publib.boulder.ibm.com/infocenter/svc/ic/index.jsp SAN Volume Controller librry Unless otherwise noted, the publictions in the SAN Volume Controller librry re vilble in Adobe portble document formt (PDF) from the following website: Click on Serch for publictions to find the online publictions you re interested in, nd then view or downlod the publiction by clicking the pproprite item. Tble 1 lists websites where you cn find help, services, nd more informtion. Tble 1. IBM websites for help, services, nd informtion Website Directory of worldwide contcts Support for SAN Volume Controller (2145) Support for IBM System Storge nd IBM TotlStorge products Address plnetwide support/ support/ Ech of the PDF publictions in the Tble 2 librry is lso vilble in the informtion center by clicking the number in the Order number column: Tble 2. SAN Volume Controller librry Title Description Order number IBM SAN Volume Controller Model 2145-DH8 Hrdwre Instlltion Guide IBM System Storge SAN Volume Controller Model 2145-CG8 Hrdwre Instlltion Guide IBM System Storge SAN Volume Controller Hrdwre Mintennce Guide This guide provides the instructions tht the IBM service representtive uses to instll the hrdwre for SAN Volume Controller model 2145-DH8. This guide provides the instructions tht the IBM service representtive uses to instll the hrdwre for SAN Volume Controller model 2145-CG8. This guide provides the instructions tht the IBM service representtive uses to service the SAN Volume Controller hrdwre, including the removl nd replcement of prts. GC GC GC xii SAN Volume Controller: Troubleshooting Guide

13 Tble 2. SAN Volume Controller librry (continued) Title Description Order number IBM System Storge SAN Volume Controller Troubleshooting Guide IBM SAN Volume Controller Sfety Notices IBM System Storge SAN Volume Controller Red First Flyer IBM System Storge SAN Volume Controller nd IBM Storwize V7000 Commnd-Line Interfce User's Guide IBM Sttement of Limited Wrrnty (2145 nd 2076) IBM License Agreement for Mchine Code This guide describes the fetures of ech SAN Volume Controller model, explins how to use the front pnel or service ssistnt GUI, nd provides mintennce nlysis procedures to help you dignose nd solve problems with the SAN Volume Controller. This guide contins trnslted cution nd dnger sttements. Ech cution nd dnger sttement in the SAN Volume Controller documenttion hs number tht you cn use to locte the corresponding sttement in your lnguge in the IBM SAN Volume Controller Sfety Notices document. This document introduces the mjor components of the SAN Volume Controller system nd describes how to get strted instlling the hrdwre nd softwre. This guide describes the commnds tht you cn use from the SAN Volume Controller commnd-line interfce (CLI). This multilingul document provides informtion bout the IBM wrrnty for mchine types 2145 nd This multilingul guide contins the License Agreement for Mchine Code for the SAN Volume Controller product. GC GA GA GC Prt number: SC (contins Z ) Other IBM publictions Tble 3 on pge xiv lists n IBM publiction tht contins informtion relted to SAN Volume Controller. About this guide xiii

14 Tble 3. Other IBM publictions Title Description Order number IBM System Storge Multipth Subsystem Device Driver User's Guide This guide describes the IBM System Storge Multipth Subsystem Device Driver for IBM System Storge products nd how to use it with the SAN Volume Controller. GC IBM documenttion nd relted websites Tble 4 lists websites tht provide publictions nd other informtion bout the SAN Volume Controller or relted products or technologies. The IBM Redbooks publictions provide positioning nd vlue guidnce, instlltion nd implementtion experiences, solution scenrios, nd step-by-step procedures for vriety of products. Tble 4. IBM documenttion nd relted websites Website Address IBM Storge Mngement Pck for Microsoft System Center Opertions Mnger (SCOM) IBM Storge Mngement Console for VMwre vcenter IBM Storge Device Driver for VMwre VAAI IBM Storge Mngement Console for VMwre vcenter Site Recovery Mnger (SRM) IBM Publictions Center IBM Redbooks publictions The IBM Storge Host Softwre Solutions Informtion Center describes how to instll, configure, nd use the IBM Storge Mngement Pck for Microsoft System Center Opertions Mnger. The IBM Storge Host Softwre Solutions Informtion Center describes how to instll, configure, nd use the IBM Storge Mngement Console for VMwre vcenter, which enbles SAN Volume Controller nd other IBM storge systems to be integrted in VMwre vcenter environments. IBM Storge Host Softwre Solutions Informtion Center describes how to instll, configure, nd use the IBM Storge Device Driver for VMwre VAAI. The IBM Storge Host Softwre Solutions Informtion Center describes how to instll, configure, nd use the IBM Storge Mngement Console for VMwre vcenter Site Recovery Mnger. servlet/pbi.wss Relted ccessibility informtion To view PDF file, you need Adobe Reder, which cn be downloded from the Adobe website: How to order IBM publictions The IBM Publictions Center is worldwide centrl repository for IBM product publictions nd mrketing mteril. xiv SAN Volume Controller: Troubleshooting Guide

15 Relted websites The IBM Publictions Center offers customized serch functions to help you find the publictions tht you need. Some publictions re vilble for you to view or downlod t no chrge. You cn lso order publictions. The publictions center displys prices in your locl currency. You cn ccess the IBM Publictions Center through the following website: The following websites provide informtion bout SAN Volume Controller or relted products or technologies: Type of informtion SAN Volume Controller support Technicl support for IBM storge products IBM Electronic Support registrtion Website Sending your comments Your feedbck is importnt in helping to provide the most ccurte nd highest qulity informtion. To submit ny comments bout this book or ny other SAN Volume Controller documenttion: v Send your comments by emil to strpubs@us.ibm.com. Include the following informtion for this publiction or use suitble replcements for the publiction title nd form number for the publiction on which you re commenting: Publiction title: IBM System Storge SAN Volume Controller Troubleshooting Guide Publiction form number: GC Pge, tble, or illustrtion numbers tht you re commenting on A detiled description of ny informtion tht should be chnged How to get informtion, help, nd technicl ssistnce If you need help, service, technicl ssistnce, or just wnt more informtion bout IBM products, you will find wide vriety of sources vilble from IBM to ssist you. Informtion IBM mintins pges on the web where you cn get informtion bout IBM products nd fee services, product implementtion nd usge ssistnce, brek nd fix service support, nd the ltest technicl informtion. For more informtion, refer to Tble 5. Tble 5. IBM websites for help, services, nd informtion Website Directory of worldwide contcts Address About this guide xv

16 Tble 5. IBM websites for help, services, nd informtion (continued) Website Address Support for SAN Volume Controller (2145) Support for IBM System Storge nd IBM TotlStorge products Note: Avilble services, telephone numbers, nd web links re subject to chnge without notice. Help nd service Before clling for support, be sure to hve your IBM Customer Number vilble. If you re in the US or Cnd, you cn cll 1 (800) IBM SERV for help nd service. From other prts of the world, see for the number tht you cn cll. When clling from the US or Cnd, choose the storge option. The gent decides where to route your cll, to either storge softwre or storge hrdwre, depending on the nture of your problem. If you cll from somewhere other thn the US or Cnd, you must choose the softwre or hrdwre option when clling for ssistnce. Choose the softwre option if you re uncertin if the problem involves the SAN Volume Controller softwre or hrdwre. Choose the hrdwre option only if you re certin the problem solely involves the SAN Volume Controller hrdwre. When clling IBM for service regrding the product, follow these guidelines for the softwre nd hrdwre options: Softwre option Identify the SAN Volume Controller product s your product nd supply your customer number s proof of purchse. The customer number is 7-digit number ( to ) ssigned by IBM when the product is purchsed. Your customer number should be locted on the customer informtion worksheet or on the invoice from your storge purchse. If sked for n operting system, use Storge. Hrdwre option Provide the seril number nd pproprite 4-digit mchine type. For SAN Volume Controller, the mchine type is In the US nd Cnd, hrdwre service nd support cn be extended to 24x7 on the sme dy. The bse wrrnty is 9x5 on the next business dy. Getting help online You cn find informtion bout products, solutions, prtners, nd support on the IBM website. To find up-to-dte informtion bout products, services, nd prtners, visit the IBM website t xvi SAN Volume Controller: Troubleshooting Guide

17 Before you cll Mke sure tht you hve tken steps to try to solve the problem yourself before you cll. Some suggestions for resolving the problem before clling IBM Support include: v Check ll cbles to mke sure tht they re connected. v Check ll power switches to mke sure tht the system nd optionl devices re turned on. v Use the troubleshooting informtion in your system documenttion. The troubleshooting section of the informtion center contins procedures to help you dignose problems. v Go to the IBM Support website t to check for technicl informtion, hints, tips, nd new device drivers or to submit request for informtion. Using the documenttion Informtion bout your IBM storge system is vilble in the documenttion tht comes with the product. Tht documenttion includes printed documents, online documents, redme files, nd help files in ddition to the informtion center. See the troubleshooting informtion for dignostic instructions. The troubleshooting procedure might require you to downlod updted device drivers or softwre. IBM mintins pges on the web where you cn get the ltest technicl informtion nd downlod device drivers nd updtes. To ccess these pges, go to support/2145 nd follow the instructions. Also, some documents re vilble through the IBM Publictions Center. Sign up for the Support Line Offering If you hve questions bout how to use nd configure the mchine, sign up for the IBM Support Line offering to get professionl nswer. The mintennce supplied with the system provides support when there is problem with hrdwre component or fult in the system mchine code. At times, you might need expert dvice bout using function provided by the system or bout how to configure the system. Purchsing the IBM Support Line offering gives you ccess to this professionl dvice while deploying your system, nd in the future. Contct your locl IBM sles representtive or the IBM Remote Technicl Support Center for vilbility nd purchse informtion. Summry of chnges for GC SAN Volume Controller Troubleshooting Guide The summry of chnges provides list of new nd updted informtion since the lst version of the guide. About this guide xvii

18 New informtion The following informtion hs been dded to this guide since the previous edition, GC v SAN Volume Controller 2145-DH8 front pnel controls nd indictors on pge 9 v SAN Volume Controller 2145-DH8 opertor informtion pnel on pge 15 v SAN Volume Controller 2145-DH8 environment requirements on pge 33 v MAP 5040: Power SAN Volume Controller 2145-DH8 on pge 274 Summry of chnges for GC SAN Volume Controller Troubleshooting Guide The summry of chnges provides list of new nd updted informtion since the lst version of the guide. New informtion The following informtion hs been dded to this guide since the previous edition, GC v Front pnel interfce on pge 74 Updted informtion This version includes updtes to: v When to run the recover system procedure on pge 246 v Fix hrdwre errors on pge 246 Summry of chnges for GC SAN Volume Controller Troubleshooting Guide The summry of chnges provides list of new nd chnged informtion since the lst version of the guide. New informtion This topic describes the chnges to this guide since the previous edition, GC The following sections summrize the chnges tht hve since been implemented from the previous version. This version includes the following new informtion: v Fibre Chnnel over Ethernet Chnged informtion This version does not include ny chnged informtion. xviii SAN Volume Controller: Troubleshooting Guide

19 Chpter 1. SAN Volume Controller overview The SAN Volume Controller combines softwre nd hrdwre into comprehensive, modulr pplince tht uses symmetric virtuliztion. Symmetric virtuliztion is chieved by creting pool of mnged disks (MDisks) from the ttched storge systems. Those storge systems re then mpped to set of volumes for use by ttched host systems. System dministrtors cn view nd ccess common pool of storge on the storge re network (SAN). This functionlity helps dministrtors to use storge resources more efficiently nd provides common bse for dvnced functions. A SAN is high-speed Fibre Chnnel network tht connects host systems nd storge devices. In SAN, host system cn be connected to storge device cross the network. The connections re mde through units such s routers nd switches. The re of the network tht contins these units is known s the fbric of the network. SAN Volume Controller softwre The SAN Volume Controller softwre provides these functions for the host systems tht ttch to SAN Volume Controller: v Cretes single pool of storge v Provides logicl unit virtuliztion v Mnges logicl volumes v Mirrors logicl volumes The SAN Volume Controller system lso provides these functions: v Lrge sclble cche v Copy Services IBM FlshCopy (point-in-time copy) function, including thin-provisioned FlshCopy to mke multiple trgets ffordble Metro Mirror (synchronous copy) Globl Mirror (synchronous copy) Dt migrtion v Spce mngement IBM Esy Tier to migrte the most frequently used dt to higher-performnce storge Metering of service qulity when combined with IBM Tivoli Storge Productivity Center Thin-provisioned logicl volumes Compressed volumes to consolidte storge Figure 1 on pge 2 shows hosts, SAN Volume Controller nodes, nd RAID storge systems connected to SAN fbric. The redundnt SAN fbric comprises fult-tolernt rrngement of two or more counterprt SANs tht provide lterntive pths for ech SAN-ttched device. Copyright IBM Corp. 2003,

20 Host Host Host Host Host zone Node Node Redundnt SAN fbric Node RAID storge system RAID storge system Storge system zone svc00600 Figure 1. SAN Volume Controller system in fbric Volumes A system of SAN Volume Controller nodes presents volumes to the hosts. Most of the dvnced functions tht SAN Volume Controller provides re defined on volumes. These volumes re creted from mnged disks (MDisks) tht re presented by the RAID storge systems. All dt trnsfer occurs through the SAN Volume Controller node, which is described s symmetric virtuliztion. The following figure shows the dt flow cross the fbric. 2 SAN Volume Controller: Troubleshooting Guide

21 Host Host Host Host Hosts send I/O to volumes. Node Redundnt SAN fbric Node I/O is sent to mnged disks. Dt trnsfer RAID storge system RAID storge system svc00601 Figure 2. Dt flow in SAN Volume Controller system The nodes in system re rrnged into pirs tht re known s I/O groups. A single pir is responsible for serving I/O on volume. Becuse volume is served by two nodes, no loss of vilbility occurs if one node fils or is tken offline. System mngement The SAN Volume Controller nodes in clustered system operte s single system nd present single point of control for system mngement nd service. System mngement nd error reporting re provided through n Ethernet interfce to one of the nodes in the system, which is clled the configurtion node. The configurtion node runs web server nd provides commnd-line interfce (CLI). The configurtion node is role tht ny node cn tke. If the current configurtion node fils, new configurtion node is selected from the remining nodes. Ech node lso provides commnd-line interfce nd web interfce for inititing hrdwre service ctions. Fbric types I/O opertions between hosts nd SAN Volume Controller nodes nd between SAN Volume Controller nodes nd RAID storge systems use the SCSI stndrd. The SAN Volume Controller nodes communicte with ech other through privte SCSI commnds. Fibre Chnnel over Ethernet connectivity is supported on SAN Volume Controller models 2145-DH8 nd 2145-CG8. Tble 6 on pge 4 shows the fbric type tht cn be used for communicting between hosts, nodes, nd RAID storge systems. These fbric types cn be used t the sme time. Chpter 1. SAN Volume Controller overview 3

22 Tble 6. SAN Volume Controller communictions types Communictions type Host to SAN Volume Controller SAN Volume Controller to storge system Fibre Chnnel SAN Yes Yes Yes iscsi (1 Gbps Ethernet or 10 Gbps Ethernet) Yes No No Fibre Chnnel Over Ethernet SAN (10 Gbps Ethernet) Yes Yes Yes SAN Volume Controller to SAN Volume Controller Flsh drives Some SAN Volume Controller nodes contin flsh drives or re ttched to expnsion enclosures tht contin flsh drives. These flsh drives cn be used to crete RAID-mnged disks (MDisks) tht in turn cn be used to crete volumes. In SAN Volume Controller 2145-DH8 nodes, the flsh drives re in n expnsion enclosure tht is connected to both sides of n I/O group. Flsh drives provide host servers with pool of high-performnce storge for criticl pplictions. Figure 3 shows this configurtion. MDisks on flsh drives cn lso be plced in storge pool with MDisks from regulr RAID storge systems, nd IBM Esy Tier performs utomtic dt plcement within tht storge pool by moving high-ctivity dt onto better performing storge. Hosts send I/O to volumes, which re mpped to internl solid-stte drives. Host Host Host Host Node with SSDs Redundnt SAN fbric svc SAN Volume Controller: Troubleshooting Guide Figure 3. SAN Volume Controller nodes with internl Flsh drives SAN Volume Controller hrdwre Ech SAN Volume Controller node is n individul server in SAN Volume Controller clustered system on which the SAN Volume Controller softwre runs. The nodes re lwys instlled in pirs, with minimum of one nd mximum of four pirs of nodes constituting system. Ech pir of nodes is known s n I/O group. I/O groups tke the storge tht is presented to the SAN by the storge systems s MDisks nd trnsltes the storge into logicl disks (volumes) tht re used by

23 pplictions on the hosts. A node is in only one I/O group nd provides ccess to the volumes in tht I/O group. SAN Volume Controller 2145-DH8 node fetures The SAN Volume Controller 2145-DH8 node hs the following fetures: v A 19-inch rck-mounted enclosure v At lest one Fibre Chnnel dpter or one 10 Gbps Ethernet dpter v Optionl second nd third Fibre Chnnel dpters v 32 GB memory per processor v One or two, eight-core processors v Dul redundnt power supplies v Up to two SAN Volume Controller F expnsion enclosures to house optionl flsh drives v iscsi host ttchment (1 Gbps Ethernet nd optionl 10 Gbps Ethernet) v Supports optionl IBM Rel-time Compression SAN Volume Controller 2145-CG8 node fetures The SAN Volume Controller 2145-CG8 node hs the following fetures: v A 19-inch rck-mounted enclosure v One 4-port 8 Gbps Fibre Chnnel dpter v One optionl 2-port 10 Gbps Fibre Chnnel over Ethernet converged network dpter v Optionl second 4-port 8 Gbps Fibre Chnnel dpter v 24 GB memory v Fibre Chnnel over Ethernet host ttchment (need to dd only one) v One qud-core processor v Dul, redundnt power supplies v Supports up to four optionl flsh drives v iscsi host ttchment (1 Gbps Ethernet nd optionl 10 Gbps Ethernet) v Supports optionl IBM Rel-time Compression Note: The optionl flsh drives nd optionl 10 Gbps Ethernet cnnot be in the sme node. Systems A system is collection of SAN Volume Controller nodes. A system cn consist of between two to eight SAN Volume Controller nodes. All configurtion settings re replicted cross ll nodes in the system. Mngement IP ddresses re ssigned to the system. Ech interfce ccesses the system remotely through the Ethernet system-mngement ddresses, lso known s the primry, nd secondry system IP ddresses. Configurtion node A configurtion node is single node tht mnges configurtion ctivity of the system. Chpter 1. SAN Volume Controller overview 5

24 If the configurtion node fils, the system chooses new configurtion node. This ction is clled configurtion node filover. The new configurtion node tkes over the mngement IP ddresses. Thus you cn ccess the system through the sme IP ddresses lthough the originl configurtion node hs filed. During the filover, there is short period when you cnnot use the commnd-line tools or mngement GUI. Figure 4 shows n exmple of clustered system tht contins four nodes. Node 1 is the configurtion node. User requests (1) re hndled by node 1. 2 Node 1 Node 2 Node 3 Node 4 1 Configurtion Node IP Interfce Figure 4. Configurtion node Configurtion node ddressing At ny given time, only one node within SAN Volume Controller clustered system is ssigned n IP ddresses. An IP ddress for the clustered system must be ssigned to Ethernet port 1. An IP ddress cn lso be ssigned to Ethernet port 2. These re the only ports tht cn be ssigned mngement IP ddresses. This node then cts s the focl point for ll configurtion nd other requests tht re mde from the mngement GUI ppliction or the CLI. This node is known s the configurtion node. If the configurtion node is stopped or fils, the remining nodes in the system determine which node will tke on the role of configurtion node. The new configurtion node binds the mngement IP ddresses to its Ethernet ports. It brodcsts this new mpping so tht connections to the system configurtion interfce cn be resumed. The new configurtion node brodcsts the new IP ddress mpping using the Address Resolution Protocol (ARP). You must configure some switches to forwrd the ARP pcket on to other devices on the subnetwork. Ensure tht ll Ethernet devices re configured to pss on unsolicited ARP pckets. Otherwise, if the ARP pcket is not forwrded, device loses its connection to the SAN Volume Controller system. If device loses its connection to the SAN Volume Controller system, it cn regenerte the ddress quickly if the device is on the sme subnetwork s the system. However, if the device is not on the sme subnetwork, it might tke hours for the ddress resolution cche of the gtewy to refresh. In this cse, you cn restore the connection by estblishing commnd line connection to the system from terminl tht is on the sme subnetwork, nd then by strting secure copy to the device tht hs lost its connection. 6 SAN Volume Controller: Troubleshooting Guide

25 Mngement IP filover If the configurtion node fils, the IP ddresses for the clustered system re trnsferred to new node. The system services re used to mnge the trnsfer of the mngement IP ddresses from the filed configurtion node to the new configurtion node. The following chnges re performed by the system service: v If softwre on the filed configurtion node is still opertionl, the softwre shuts down the mngement IP interfces. If the softwre cnnot shut down the mngement IP interfces, the hrdwre service forces the node to shut down. v When the mngement IP interfces shut down, ll remining nodes choose new node to host the configurtion interfces. v The new configurtion initilizes the configurtion demons, including sshd nd httpd, nd then binds the mngement IP interfces to its Ethernet ports. v The router is configured s the defult gtewy for the new configurtion. v The routing tbles re estblished on the new configurtion for the mngement IP ddresses. The new configurtion sends five unsolicited ddress resolution protocol (ARP) pckets for ech IP ddress to the locl subnet brodcst ddress. The ARP pckets contin the mngement IP nd the Medi Access Control (MAC) ddress for the new configurtion node. All systems tht receive ARP pckets re forced to updte their ARP tbles. After the ARP tbles re updted, these systems cn connect to the new configurtion node. Note: Some Ethernet devices might not forwrd ARP pckets. If the ARP pckets re not forwrded, connectivity to the new configurtion node cnnot be estblished utomticlly. To void this problem, configure ll Ethernet devices to pss unsolicited ARP pckets. You cn restore lost connectivity by logging in to the SAN Volume Controller nd strting secure copy to the ffected system. Strting secure copy forces n updte to the ARP cche for ll systems connected to the sme switch s the ffected system. Ethernet link filures If the Ethernet link to the SAN Volume Controller system fils becuse of n event unrelted to the SAN Volume Controller, such s cble being disconnected or n Ethernet router filure, the SAN Volume Controller does not ttempt to fil over the configurtion node to restore mngement IP ccess. SAN Volume Controller provides the option for two Ethernet ports, ech with its own mngement IP ddress, to protect ginst this type of filure. If you cnnot connect through one IP ddress, ttempt to ccess the system through the lternte IP ddress. Note: IP ddresses tht re used by hosts to ccess the system over n Ethernet connection re different from mngement IP ddresses. Routing considertions for event notifiction nd Network Time Protocol SAN Volume Controller supports the following protocols tht mke outbound connections from the system: v Emil v Simple Network Mil Protocol (SNMP) v Syslog v Network Time Protocol (NTP) Chpter 1. SAN Volume Controller overview 7

26 SAN fbric overview These protocols operte only on port configured with mngement IP ddress. When mking outbound connections, the SAN Volume Controller uses the following routing decisions: v v v If the destintion IP ddress is in the sme subnet s one of the mngement IP ddresses, the SAN Volume Controller system sends the pcket immeditely. If the destintion IP ddress is not in the sme subnet s either of the mngement IP ddresses, the system sends the pcket to the defult gtewy for Ethernet port 1. If the destintion IP ddress is not in the sme subnet s either of the mngement IP ddresses nd Ethernet port 1 is not connected to the Ethernet network, the system sends the pcket to the defult gtewy for Ethernet port 2. When configuring ny of these protocols for event notifictions, use these routing decisions to ensure tht error notifiction works correctly in the event of network filure. The SAN fbric is n re of the network tht contins routers nd switches. A SAN is configured into number of zones. A device using the SAN cn communicte only with devices tht re included in the sme zones tht it is in. A system requires severl distinct types of zones: system zone, host zones, nd disk zones. The intersystem zone is optionl. In the host zone, the host systems cn identify nd ddress the nodes. You cn hve more thn one host zone nd more thn one disk zone. Unless you re using dul-core fbric design, the system zone contins ll ports from ll nodes in the system. Crete one zone for ech host Fibre Chnnel port. In disk zone, the nodes identify the storge systems. Generlly, crete one zone for ech externl storge system. If you re using the Metro Mirror nd Globl Mirror feture, crete zone with t lest one port from ech node in ech system; up to four systems re supported. Note: Some operting systems cnnot tolerte other operting systems in the sme host zone, lthough you might hve more thn one host type in the SAN fbric. For exmple, you cn hve SAN tht contins one host tht runs on n IBM AIX operting system nd nother host tht runs on Microsoft Windows operting system. All communiction between SAN Volume Controller nodes is performed through the SAN. All SAN Volume Controller configurtion nd service commnds re sent to the system through n Ethernet network. 8 SAN Volume Controller: Troubleshooting Guide

27 Chpter 2. Introducing the SAN Volume Controller hrdwre components A SAN Volume Controller system consists of SAN Volume Controller nodes nd relted hrdwre components, such s uninterruptible power supply units nd the optionl redundnt c-power switches. Note tht nodes nd uninterruptible power supply units re instlled in pirs. SAN Volume Controller nodes SAN Volume Controller supports severl different node types. The following nodes re supported: v The SAN Volume Controller 2145-DH8 node is vilble for purchse, with the following fetures: At lest one Fibre Chnnel dpter or one 10 Gbps Ethernet dpter Optionl second nd third Fibre Chnnel dpters Up to two SAN Volume Controller F expnsion enclosures to house optionl flsh drives iscsi host ttchment (1 Gbps Ethernet nd optionl 10 Gbps Ethernet) v The SAN Volume Controller 2145-CG8 node is vilble for purchse, with the following fetures: A high-speed SAS dpter with up to four flsh drives A two-port 10 Gbps Ethernet dpter A second four-port Fibre Chnnel dpter v The following nodes re no longer vilble for purchse but remin supported: SAN Volume Controller 2145-CF8 SAN Volume Controller A4 SAN Volume Controller G4 A lbel on the front of the node indictes the SAN Volume Controller node type, hrdwre revision (if pproprite), nd seril number. SAN Volume Controller controls nd indictors The controls nd indictors re used for power nd nvigtion nd to indicte informtion such s system ctivity, service nd configurtion options, service controller filures, nd node identifiction. SAN Volume Controller 2145-DH8 front pnel controls nd indictors The controls nd indictors on the front pnel re used for power nd to indicte informtion such s system ctivity, node filures, nd node identifiction. Figure 5 on pge 10 shows the controls nd indictors on the front pnel of the SAN Volume Controller 2145-DH8. Copyright IBM Corp. 2003,

28 SAN Volume Controller svc00800 Figure 5. SAN Volume Controller 2145-DH8 front pnel 1 Hrd disk drive ctivity LED 2 Hrd disk drive sttus LED 3 USB port 4 Video connector 5 Opertor-informtion pnel 6 Rck relese ltch 7 Node sttus LED 8 Node fult LED 9 Bttery sttus LED 10 Bttery fult LED 11 Btteries 12 Hrd disk drives (boot drives) Node sttus LED The node sttus LED provides the following system ctivity indictors: Off The node is not operting s member of system. On The node is operting s member of system. Slow blinking The node is in cndidte or service stte. Fst blinking The node is dumping cche nd stte dt to the locl disk in nticiption of system reboot from pending power-off ction or other controlled restrt sequence. Node fult LED A node fult is indicted by the mber node-fult LED. Off The node does not hve ny errors tht will prevent it from doing I/O or the SVC softwre is not running on the node. On The node hs ftl node error nd is not prt of the SVC system. 10 SAN Volume Controller: Troubleshooting Guide

29 Bttery sttus LED The green bttery sttus LED indictes one of the following bttery conditions. Off The SVC softwre is not running on the node or the stte of the system cnnot be sved if power to the node is lost. Fst blinking Bttery chrge level is too low for the stte of the system to be sved if power to the node is lost. Btteries re chrging. Slow blinking Bttery chrge level is sufficient for the stte of the system to be sved once if power to the node is lost. On Bttery chrge level is sufficient for the stte of the system to be sved twice if power to the node is lost. Bttery fult LED The mber bttery fult LED indictes one of the following bttery conditions. Off The SVC softwre is not running on the node or this bttery does not hve fult. Blinking This bttery is being identified. On This bttery hs fult. It cnnot be used to sve the system stte if power to the node is lost. SAN Volume Controller 2145-CG8 controls nd indictors The controls nd indictors re used for power nd nvigtion nd to indicte informtion such s system ctivity, service nd configurtion options, service controller filures, nd node identifiction. Figure 6 shows the controls nd indictors on the front pnel of the SAN Volume Controller 2145-CG svc00717 Figure 6. SAN Volume Controller 2145-CG8 front pnel 1 Node-sttus LED 2 Front-pnel disply 3 Nvigtion buttons 4 Opertor-informtion pnel 5 Select button 6 Error LED Chpter 2. Introducing the SAN Volume Controller hrdwre components 11

30 svc00541c SAN Volume Controller 2145-CF8 controls nd indictors The controls nd indictors re used for power nd nvigtion nd to indicte informtion such s system ctivity, service nd configurtion options, service controller filures, nd node identifiction. Figure 7 shows the controls nd indictors on the front pnel of the SAN Volume Controller 2145-CF Figure 7. SAN Volume Controller 2145-CF8 front pnel 1 Node-sttus LED 2 Front-pnel disply 3 Nvigtion buttons 4 Opertor-informtion pnel 5 Select button 6 Error LED SAN Volume Controller A4 controls nd indictors The controls nd indictors re used for power nd nvigtion nd to indicte informtion such s system ctivity, service nd configurtion options, service controller filures, nd node identifiction. Figure 8 shows the controls nd indictors on the front pnel of the SAN Volume Controller A svc00438 Figure 8. SAN Volume Controller A4 front-pnel ssembly 1 Opertor-informtion pnel 2 Node sttus LED 3 Front-pnel disply 4 Nvigtion buttons 5 Seril number lbel 6 Select button 12 SAN Volume Controller: Troubleshooting Guide

31 7 Node identifiction lbel 8 Error LED SAN Volume Controller G4 controls nd indictors The controls nd indictors re used for power nd nvigtion nd to indicte informtion such s system ctivity, service nd configurtion options, service controller filures, nd node identifiction. Figure 9 shows the controls nd indictors on the front pnel of the SAN Volume Controller G svc00216 Figure 9. SAN Volume Controller G4 front-pnel ssembly 1 Node sttus LED 2 Front pnel disply 3 Nvigtion buttons 4 Seril number lbel 5 Opertor informtion pnel 6 Select button 7 Node identifiction lbel 8 Error LED Node sttus LED System ctivity is indicted through the green node-sttus LED. The node sttus LED provides the following system ctivity indictors: Off The node is not operting s member of system. On The node is operting s member of system. Slow blinking The node is in cndidte or service stte. Fst blinking The node is dumping cche nd stte dt to the locl disk in nticiption of system reboot from pending power-off ction or other controlled restrt sequence. Front-pnel disply The front-pnel disply shows service, configurtion, nd nvigtion informtion. Chpter 2. Introducing the SAN Volume Controller hrdwre components 13

32 You cn select the lnguge tht is displyed on the front pnel. The disply cn show both lphnumeric informtion nd grphicl informtion (progress brs). The front-pnel disply shows configurtion nd service informtion bout the node nd the system, including the following items: v Boot progress indictor v Boot filed v Chrging v Hrdwre boot v Node rescue request v Power filure v Powering off v Recovering v Restrting v Shutting down v Error codes v Vlidte WWNN? Nvigtion buttons You cn use the nvigtion buttons to move through menus. There re four nvigtionl buttons tht you cn use to move throughout menu: up, down, right, nd left. Ech button corresponds to the direction tht you cn move in menu. For exmple, to move right in menu, press the nvigtion button tht is locted on the right side. If you wnt to move down in menu, press the nvigtion button tht is locted on the bottom. Note: The select button is used in tndem with the nvigtion buttons. Product seril number The node contins SAN Volume Controller product seril number tht is written to the system bord hrdwre. The product seril number is lso printed on the seril number lbel which is locted on the front pnel. This number is used for wrrnty nd service entitlement checking nd is included in the dt sent with error reports. It is essentil tht this number is not chnged during the life of the product. If the system bord is replced, you must follow the system bord replcement instructions crefully nd rewrite the seril number on the system bord. Select button Use the select button to select n item from menu. The select button nd nvigtion buttons help you to nvigte nd select menu nd boot options, nd strt service pnel test. The select button is locted on the front pnel of the SAN Volume Controller, ner the nvigtion buttons. Node identifiction lbel The node identifiction lbel on the front pnel displys six-digit node identifiction number. Sometimes this number is clled the pnel nme or front pnel ID. 14 SAN Volume Controller: Troubleshooting Guide

33 The node identifiction lbel is the six-digit number tht is input to the ddnode commnd. It is redble by system softwre nd is used by configurtion nd service softwre s node identifier. The node identifiction number cn lso be displyed on the front-pnel disply when node is selected from the menu. If the service controller ssembly front pnel is replced, the configurtion nd service softwre displys the number tht is printed on the front of the replcement pnel. Future error reports contin the new number. No system reconfigurtion is necessry when the front pnel is replced. Error LED Criticl fults on the service controller re indicted through the mber error LED. The error LED hs the following two sttes: OFF The service controller is functioning correctly. ON A criticl service-controller filure ws detected nd you must replce the service controller. The error LED cn light temporrily when the node is powered on. If the error LED is on, but the front pnel disply is completely blnk, wit five minutes to llow the LED time to turn off before performing ny service ction. SAN Volume Controller opertor-informtion pnel The opertor-informtion pnel is locted on the front pnel of the SAN Volume Controller. SAN Volume Controller 2145-DH8 opertor informtion pnel The opertor-informtion pnel indictes informtion such s system bord errors, Ethernet ctivity, nd power sttus. Figure 10 shows the opertor-informtion pnel for the SAN Volume Controller 2145-DH Figure 10. SAN Volume Controller 2145-DH8 opertor informtion pnel 1 Power-control button nd power-on LED (green) 2 Ethernet icon 3 System-loctor button nd LED (blue) 4 Relese ltch for the light pth dignostics pnel 5 Ethernet ctivity LEDs 6 Check log LED 7 System-error LED (yellow) ifs00064 Chpter 2. Introducing the SAN Volume Controller hrdwre components 15

34 Note: If the node hs more thn four Ethernet ports, ctivity on ports five nd bove is not reflected on the opertor-informtion pnel Ethernet ctivity LEDs. SAN Volume Controller 2145-CG8 opertor-informtion pnel The opertor-informtion pnel contins buttons nd indictors such s the power-control button, nd LEDs tht indicte informtion such s system-bord errors, hrd-drive ctivity, nd power sttus. Figure 11 shows the opertor-informtion pnel for the SAN Volume Controller 2145-CG svc00722 Figure 11. SAN Volume Controller 2145-CG8 or 2145-CF8 opertor-informtion pnel 1 Power-button cover 2 Ethernet 1 ctivity LED. The opertor-informtion pnel LEDs refer to the Ethernet ports tht re mounted on the system bord. 3 Ethernet 2 ctivity LED. The opertor-informtion pnel LEDs refer to the Ethernet ports tht re mounted on the system bord. 4 System-informtion LED 5 System-error LED 6 Relese ltch 7 Loctor button nd LED 8 Power button nd LED Note: If you instll the 10 Gbps Ethernet feture, the port ctivity is not reflected on the ctivity LEDs. SAN Volume Controller 2145-CF8 opertor-informtion pnel The opertor-informtion pnel contins buttons nd indictors such s the power-control button, nd LEDs tht indicte informtion such s system-bord errors, hrd-drive ctivity, nd power sttus. Figure 12 on pge 17 shows the opertor-informtion pnel for the SAN Volume Controller 2145-CF8. 16 SAN Volume Controller: Troubleshooting Guide

35 svc_bb1gs Figure 12. SAN Volume Controller 2145-CG8 or 2145-CF8 opertor-informtion pnel 1 Power-button cover 2 Ethernet 2 ctivity LED 3 Ethernet 1 ctivity LED 4 System-informtion LED 5 System-error LED 6 Relese ltch 7 Loctor button nd LED 8 Not used 9 Not used 10 Power button nd LED SAN Volume Controller A4 opertor-informtion pnel The opertor-informtion pnel contins buttons nd indictors such s the power-control button, nd LEDs tht indicte informtion such s system-bord errors, hrd-drive ctivity, nd power sttus. Figure 13 shows the opertor-informtion pnel for the SAN Volume Controller A svc00452 Figure 13. SAN Volume Controller A4 opertor-informtion pnel 1 System-error LED (mber) 2 Loctor LED (blue) 3 Hrd-disk drive ctivity LED (green) 4 Reset button 5 Power-control button 6 Power LED (green) Chpter 2. Introducing the SAN Volume Controller hrdwre components 17

36 SAN Volume Controller G4 opertor informtion pnel The opertor-informtion pnel contins buttons nd indictors such s the relese ltch for the light pth dignostics pnel, the power-control button, nd LEDs tht indicte informtion such s system-bord errors, hrd-drive ctivity, nd power sttus. Figure 14 shows the opertor informtion pnel for the SAN Volume Controller G svc00215 Figure 14. SAN Volume Controller G4 opertor-informtion pnel 1 Relese ltch for light pth dignostics pnel 2 System-error LED (mber) 3 System-informtion LED (mber) 4 Loctor LED (blue) 5 Hrd disk drive ctivity LED (green) 6 Power LED (green) 7 Power-control button System-error LED When it is lit, the system-error LED indictes tht system-bord error hs occurred. This mber LED lights up if the hrdwre detects ftl error tht requires new field-replceble unit (FRU). To help you isolte the fulty FRU, see MAP 5800: Light pth to help you isolte the fulty FRU. A system-error LED is lso t the rer of the SAN Volume Controller models 2145-CG8, 2145-CF8, nd G4. Hrd-disk drive ctivity LED When it is lit, the green hrd-disk drive ctivity LED indictes tht the hrd disk drive is in use. Reset button A reset button is vilble on the SAN Volume Controller A4 node, but do not use it. Attention: If you use the reset button, the node restrts immeditely without the SAN Volume Controller control dt being written to disk. Service ctions re then required to mke the node opertionl gin. Power button The power button turns min power on or off for the SAN Volume Controller. 18 SAN Volume Controller: Troubleshooting Guide

37 To turn on the power, press nd relese the power button. You must hve pointed device, such s pen, to press the button. To turn off the power, press nd relese the power button. For more informtion bout how to turn off the SAN Volume Controller node, see MAP 5350: Powering off SAN Volume Controller node. Attention: When the node is opertionl nd you press nd immeditely relese the power button, the SAN Volume Controller writes its control dt to its internl disk nd then turns off. This cn tke up to five minutes. If you press the power button but do not relese it, the node turns off immeditely without the SAN Volume Controller control dt being written to disk. Service ctions re then required to mke the SAN Volume Controller opertionl gin. Therefore, during power-off opertion, do not press nd hold the power button for more thn two seconds. Power LED The green power LED indictes the power sttus of the system. The power LED hs the following properties: Off One or more of the following re true: v No power is present t the power supply input. v The power supply hs filed. v The LED hs filed. On The SAN Volume Controller node is turned on. Flshing The SAN Volume Controller node is turned off, but is still connected to power source. Note: A power LED is lso t the rer of the SAN Volume Controller 2145-CG8, 2145-CF8, nd G4 nodes. Relese ltch The relese ltch on the SAN Volume Controller model G4 gives you ccess to the light pth dignostics pnel, which provides method for determining the loction of problem. After pressing the relese ltch on the opertor-informtion pnel, you cn slide the light pth dignostics pnel out to view the lit LEDs. The LEDs indicte the type of error tht hs occurred. See MAP 5800: Light pth for more detil. To retrct the pnel, push it bck into the node nd snp it into plce. System-informtion LED When the system-informtion LED is lit, noncriticl event hs occurred. Check the light pth dignostics pnel nd the event log. Light pth dignostics re described in more detil in the light pth mintennce nlysis procedure (MAP). Loctor LED The SAN Volume Controller does not use the loctor LED. Chpter 2. Introducing the SAN Volume Controller hrdwre components 19

38 Ethernet-ctivity LED An Ethernet-ctivity LED beside ech Ethernet port indictes tht thesan Volume Controller node is communicting on the Ethernet network tht is connected to the Ethernet port. The opertor-informtion pnel LEDs refer to the Ethernet ports tht re mounted on the system bord. If you instll the 10 Gbps Ethernet crd on SAN Volume Controller 2145-CG8, the port ctivity is not reflected on the ctivity LEDs. SAN Volume Controller rer-pnel indictors nd connectors The rer-pnel indictors for the SAN Volume Controller re locted on the bck-pnel ssembly. The externl connectors re locted on the SAN Volume Controller node nd the power supply ssembly. SAN Volume Controller 2145-CG8 rer-pnel indictors The rer-pnel indictors consist of LEDs tht indicte the sttus of the Fibre Chnnel ports, Ethernet connection nd ctivity, power, electricl current, nd system-bord errors. Figure 15 shows the rer-pnel indictors on the SAN Volume Controller 2145-CG8 bck-pnel ssembly svc00720 Figure 15. SAN Volume Controller 2145-CG8 rer-pnel indictors 1 Fibre Chnnel LEDs 2 Ethernet-link LEDs 3 Ethernet-ctivity LEDs 4 Ac, dc, nd power-supply error LEDs 5 Power, loction, nd system-error LEDs Figure 16 shows the rer-pnel indictors on the SAN Volume Controller 2145-CG8 bck-pnel ssembly tht hs the 10 Gbps Ethernet feture. 1 2 svc00729 Figure 16. SAN Volume Controller 2145-CG8 rer-pnel indictors for the 10 Gbps Ethernet feture 1 10 Gbps Ethernet-link LEDs. The mber link LED is on when this port is connected to 10 Gbps Ethernet switch nd the link is online. 20 SAN Volume Controller: Troubleshooting Guide

39 2 10 Gbps Ethernet-ctivity LEDs. The green ctivity LED is on while dt is being sent over the link. SAN Volume Controller 2145-CG8 connectors Externl connectors tht the SAN Volume Controller 2145-CG8 uses include four Fibre Chnnel ports, seril port, two Ethernet ports, nd two power connectors. The 2145-CG8 lso hs externl connectors for the 10 Gbps Ethernet feture. These figures show the externl connectors on the SAN Volume Controller 2145-CG8 bck pnel ssembly svc00732 Figure 17. Connectors on the rer of the SAN Volume Controller 2145-CG8 1 Fibre Chnnel port 1 2 Fibre Chnnel port 2 3 Fibre Chnnel port 3 4 Fibre Chnnel port 4 5 Power-cord connector for power supply 1 6 Power-cord connector for power supply 2 7 Seril connection for UPS communiction cble 8 Ethernet port 2 9 Ethernet port svc00731 Figure Gbps Ethernet ports on the rer of the SAN Volume Controller 2145-CG Gbps Ethernet port Gbps Ethernet port 4 Fibre Chnnel port 5 (not shown) Fibre Chnnel port 6 (not shown) Fibre Chnnel port 7 (not shown) Fibre Chnnel port 8 (not shown) Figure 19 on pge 22 shows the type of connector tht is locted on ech power-supply ssembly. Use these connectors to connect the SAN Volume Controller 2145-CG8 to the two power cbles from the uninterruptible power supply. Chpter 2. Introducing the SAN Volume Controller hrdwre components 21

40 Neutrl Ground Live Figure 19. Power connector SAN Volume Controller 2145-CG8 ports used during service procedures: The SAN Volume Controller 2145-CG8 contins number of ports tht re only used during service procedures. Figure 20 shows ports tht re used only during service procedures svc00724 Figure 20. Service ports of the SAN Volume Controller 2145-CG8 1 System mngement port 2 Two monitor ports, one on the front nd one on the rer 3 Four USB ports, two on the front nd two on the rer During norml opertion, none of these ports re used. Connect device to ny of these ports only when you re directed to do so by service procedure or by n IBM service representtive. SAN Volume Controller 2145-CG8 unused ports: The SAN Volume Controller 2145-CG8 cn contin one port tht is not used. Figure 21 shows the one port tht is not used during service procedures or norml use. 1 svc00730 Figure 21. SAN Volume Controller 2145-CG8 port not used 22 SAN Volume Controller: Troubleshooting Guide

41 1 Seril-ttched SCSI (SAS) port When present, this port is disbled in softwre to mke the port inctive. The SAS port is present when the optionl high-speed SAS dpter is instlled with one or more flsh drives. SAN Volume Controller 2145-CF8 rer-pnel indictors The rer-pnel indictors consist of LEDs tht indicte the sttus of the Fibre Chnnel ports, Ethernet connection nd ctivity, power, electricl current, nd system-bord errors. Figure 22 shows the rer-pnel indictors on the SAN Volume Controller 2145-CF8 bck-pnel ssembly. 1 2 svc_00219b_cf Figure 22. SAN Volume Controller 2145-CF8 rer-pnel indictors 1 Fibre Chnnel LEDs 2 Ac, dc, nd power-supply error LEDs 3 Power, loction, nd system-error LEDs 4 Ethernet-link LEDs 5 Ethernet-ctivity LEDs SAN Volume Controller 2145-CF8 connectors Externl connectors tht the SAN Volume Controller 2145-CF8 uses include four Fibre Chnnel ports, seril port, two Ethernet ports, nd two power connectors. Figure 23 shows the externl connectors on the SAN Volume Controller 2145-CF8 bck pnel ssembly svc_00219_cf Figure 23. Connectors on the rer of the SAN Volume Controller 2145-CG8 or 2145-CF8 1 Fibre Chnnel port 1 2 Fibre Chnnel port 2 3 Fibre Chnnel port 3 4 Fibre Chnnel port 4 Chpter 2. Introducing the SAN Volume Controller hrdwre components 23

42 svc00227cf8 5 Power-cord connector for power supply 1 6 Power-cord connector for power supply 2 7 Seril connection for UPS communiction cble 8 Ethernet port 2 9 Ethernet port 1 Figure 24 shows the type of connector tht is locted on ech power-supply ssembly. Use these connectors to connect the SAN Volume Controller 2145-CF8 to the two power cbles from the uninterruptible power supply. Neutrl Ground Live Figure 24. Power connector SAN Volume Controller 2145-CF8 ports used during service procedures: The SAN Volume Controller 2145-CF8 contins number of ports tht re only used during service procedures. Figure 25 shows ports tht re used only during service procedures Figure 25. Service ports of the SAN Volume Controller 2145-CF8 1 System mngement port 2 Two monitor ports, one on the front nd one on the rer 3 Four USB ports, two on the front nd two on the rer During norml opertion, none of these ports re used. Connect device to ny of these ports only when you re directed to do so by service procedure or by n IBM service representtive. SAN Volume Controller 2145-CF8 unused ports: The SAN Volume Controller 2145-CF8 cn contin one port tht is not used. 24 SAN Volume Controller: Troubleshooting Guide

43 svc00227cf8b Figure 26 shows the one port tht is not used during service procedures or norml use. 1 Figure 26. SAN Volume Controller 2145-CF8 port not used 1 Seril-ttched SCSI (SAS) port When present, this port is disbled in softwre to mke the port inctive. The SAS port is present when the optionl high-speed SAS dpter is instlled with one or more flsh drives. SAN Volume Controller A4 rer-pnel indictors The rer-pnel indictors consist of LEDs tht indicte the sttus of the Fibre Chnnel ports, Ethernet connection nd ctivity, power, electricl current, nd system-bord errors. Figure 27 shows the rer-pnel indictors on the SAN Volume Controller A4 bck-pnel ssembly svc00539 Figure 27. SAN Volume Controller A4 rer-pnel indictors 1 Fibre Chnnel LEDs 2 Ethernet port 1 ctivity LED 3 Ethernet port 1 link LED 4 Ethernet port 2 ctivity LED 5 Ethernet port 2 link LED SAN Volume Controller A4 connectors The externl connectors consist of Fibre Chnnel, seril nd Ethernet ports, nd the power supply. Figure 28 on pge 26 shows the externl connectors on the SAN Volume Controller A4 bck-pnel ssembly. Chpter 2. Introducing the SAN Volume Controller hrdwre components 25

44 svc00538 Figure 28. SAN Volume Controller A4 externl connectors 1 Fibre Chnnel port 1 2 Fibre Chnnel port 2 3 Fibre Chnnel port 3 4 Fibre Chnnel port 4 5 Power supply 6 Seril connection 7 Ethernet port 2 8 Ethernet port 1 Figure 29 shows the type of connector tht is locted on the power supply ssembly. The connector enbles you to connect the SAN Volume Controller A4 to the power source from the uninterruptible power supply. Figure 29. Power connector SAN Volume Controller A4 ports used during service procedures The SAN Volume Controller A4 contins number of ports tht re used only during service procedures. These ports re shown in Figure svc00537 Figure 30. Service ports of the SAN Volume Controller A4 1 System mngement port 2 Four USB ports, two on the front nd two on the rer 26 SAN Volume Controller: Troubleshooting Guide

45 3 One video port on the rer During norml opertion, none of these ports re used. Connect device to ny of these ports only when you re directed to do so by service procedure or by your IBM service representtive. SAN Volume Controller A4 ports not used The SAN Volume Controller A4 hs no unused ports. SAN Volume Controller G4 rer-pnel indictors The rer-pnel indictors consist of LEDs tht indicte the sttus of the Fibre Chnnel ports, Ethernet connection nd ctivity, power, electricl current, nd system-bord errors. Figure 31 shows the rer-pnel indictors on the SAN Volume Controller G4 bck-pnel ssembly svc00536 Figure 31. SAN Volume Controller G4 rer-pnel indictors 1 Fibre Chnnel LEDs 2 Ethernet port 1 ctivity LED 3 Ethernet port 1 link LED 4 Ethernet port 2 ctivity LED 5 Ethernet port 2 link LED 6 Power, loction, nd system error LEDs 7 Ac nd dc LEDs SAN Volume Controller G4 connectors The externl connectors consist of Fibre Chnnel, seril, nd Ethernet ports, nd the power supply. Figure 32 shows the externl connectors on the SAN Volume Controller G4 bck pnel ssembly svc00535 Figure 32. SAN Volume Controller G4 externl connectors Chpter 2. Introducing the SAN Volume Controller hrdwre components 27

46 1 Fibre Chnnel port 1 2 Fibre Chnnel port 2 3 Fibre Chnnel port 3 4 Fibre Chnnel port 4 5 Power supply 6 Seril connection 7 Ethernet port 2 8 Ethernet port 1 Figure 33 shows the type of connector tht is locted on the power supply ssembly. The connector enbles you to connect the SAN Volume Controller G4 to the power source from the uninterruptible power supply. Neutrl Ground Live Figure 33. Power connector SAN Volume Controller G4 ports used during service procedures The SAN Volume Controller G4 contins number of ports tht re only used during service procedures. These ports re shown in Figure svc00534 Figure 34. Service ports of the SAN Volume Controller G4 1 System mngement port 2 Four USB ports, two on the front nd two on the rer 3 Two monitor ports, one on the front nd one on the rer During norml opertion, none of these ports re used. Connect device to ny of these ports only when you re directed to do so by service procedure or by your IBM service representtive. 28 SAN Volume Controller: Troubleshooting Guide

47 SAN Volume Controller G4 ports not used The SAN Volume Controller G4 hs no unused ports. Fibre Chnnel LEDs The Fibre Chnnel LEDs indicte the sttus of the Fibre Chnnel ports. Two LEDs re used to indicte the stte nd speed of the opertion of ech Fibre Chnnel port. The bottom LED indictes the link stte nd ctivity. Tble 7. Link stte nd ctivity for the bottom Fibre Chnnel LED LED stte Link stte nd ctivity indicted Off Link inctive On Link ctive, no I/O Flshing Link ctive, I/O ctive Ech Fibre Chnnel port cn operte t one of three speeds. The top LED indictes the reltive link speed. The link speed is defined only if the link stte is ctive. Tble 8. Link speed for the top Fibre Chnnel LED LED stte Link speed indicted Off SLOW On FAST Blinking MEDIUM Tble 9 shows the ctul link speeds for the SAN Volume Controller models A4, G4, nd. Tble 9. Actul link speeds Link speed Slow Fst Medium Actul link speeds 1 Gbps 4 Gbps 2 Gbps Tble 10 shows the ctul link speeds for the SAN Volume Controller 2145-CF8 nd for the SAN Volume Controller 2145-CG8. Tble 10. Actul link speeds Link speed Slow Fst Medium Actul link speeds 2 Gbps 8 Gbps 4 Gbps Ethernet ctivity LED The Ethernet ctivity LED indictes tht the node is communicting with the Ethernet network tht is connected to the Ethernet port. There is set of LEDs for ech Ethernet connector. The top LED is the Ethernet link LED. When it is lit, it indictes tht there is n ctive connection on the Chpter 2. Introducing the SAN Volume Controller hrdwre components 29

48 Ethernet port. The bottom LED is the Ethernet ctivity LED. When it flshes, it indictes tht dt is being trnsmitted or received between the server nd network device. Ethernet link LED The Ethernet link LED indictes tht there is n ctive connection on the Ethernet port. There is set of LEDs for ech Ethernet connector. The top LED is the Ethernet link LED. When it is lit, it indictes tht there is n ctive connection on the Ethernet port. The bottom LED is the Ethernet ctivity LED. When it flshes, it indictes tht dt is being trnsmitted or received between the server nd network device. Power, loction, nd system-error LEDs The power, loction, nd system-error LEDs re housed on the rer of the SAN Volume Controller. These three LEDs re duplictes of the sme LEDs tht re shown on the front of the node. The following terms describe the power, loction, nd system-error LEDs: Power LED This is the top of the three LEDs nd indictes the following sttes: Off One or more of the following re true: v No power is present t the power supply input v The power supply hs filed v The LED hs filed On The SAN Volume Controller is powered on. Blinking The SAN Volume Controller is turned off but is still connected to power source. Loction LED This is the middle of the three LEDs nd is not used by the SAN Volume Controller. System-error LED This is the bottom of the three LEDs tht indictes tht system bord error hs occurred. The light pth dignostics provide more informtion. Ac nd dc LEDs The c nd dc LEDs indicte whether the node is receiving electricl current. Ac LED The upper LED indictes tht c current is present on the node. Dc LED The lower LED indictes tht dc current is present on the node. 30 SAN Volume Controller: Troubleshooting Guide Ac, dc, nd power-supply error LEDs on the SAN Volume Controller 2145-CF8nd SAN Volume Controller 2145-CG8: The c, dc, nd power-supply error LEDs indicte whether the node is receiving electricl current. Figure 35 on pge 31 shows the loction of the c, dc, nd power-supply error LEDs.

49 1 2 3 svc00542 Figure 35. SAN Volume Controller 2145-CG8 or 2145-CF8 c, dc, nd power-error LEDs Ech of the two power supplies hs its own set of LEDs. Ac LED The upper LED ( 1 ) on the left side of the power supply, indictes tht c current is present on the node. Dc LED The middle LED ( 2 ) to the left side of the power supply, indictes tht dc current is present on the node. Power-supply error LED The lower LED ( 3 ) to the left side of the power supply, indictes problem with the power supply. Ac nd dc LEDs on the SAN Volume Controller G4: The c LED nd dc LED re locted on the rer of the SAN Volume Controller G4. Figure 36 on pge 32 shows the loction of the c nd dc LEDs. Chpter 2. Introducing the SAN Volume Controller hrdwre components 31

50 1 2 svc00220 Figure 36. SAN Volume Controller G4 c nd dc LEDs Ac LED The upper LED ( 1 ) indictes tht c current is present on the node. Dc LED The lower LED ( 2 ) indictes tht dc current is present on the node. Fibre Chnnel port numbers nd worldwide port nmes Fibre Chnnel ports re identified by their physicl port number nd by worldwide port nme (WWPN). The physicl port numbers identify Fibre Chnnel crds nd cble connections when you run service tsks. The physicl port numbers re 1-4, counting from left to right when you view the rer pnel of the node. The WWPNs re used for tsks such s Fibre Chnnel switch configurtion nd to uniquely identify the devices on the SAN. The WWPNs re derived from the worldwide node nme (WWNN) of the SAN Volume Controller node in which the ports re instlled. The WWNN is in the form XXXXX, where XXXXX is initilly derived from the unit nd is specific to node. The WWPNs re in the form QXXXXX, where XXXXX is s previously stted nd Q is relted to the port number s follows: Port Vlue of Q SAN Volume Controller: Troubleshooting Guide

51 Requirements for the SAN Volume Controller environment Certin specifictions for the physicl site of the SAN Volume Controller must be met before the IBM representtive cn set up your SAN Volume Controller environment. SAN Volume Controller 2145-DH8 environment requirements Before the SAN Volume Controller 2145-DH8 is instlled, the physicl environment must meet certin requirements. This includes verifying tht dequte spce is vilble nd tht requirements for power nd environmentl conditions re met. Input-voltge requirements Ensure tht your environment meets the voltge requirements tht re shown in Tble 11. Tble 11. Input-voltge requirements Voltge Frequency / Vc 50 Hz or 60 Hz Mximum power requirements for ech node Ensure tht your environment meets the power requirements s shown in Tble 12. The mximum power tht is required depends on the node type nd the optionl fetures tht re instlled. Tble 12. Mximum power consumption Components SAN Volume Controller 2145-DH8 Power requirements 750 W Environment requirements without redundnt c power Ensure tht your environment flls within the following rnges if you re not using redundnt c power. If you re not using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 13. Tble 13. Physicl specifictions Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes Turned off (with stndby power) 5 C to 40 C (41 F to 104 F) 5 C to 28 C (41 F to 82 F) 5 C to 45 C (41 F to 113 F) 0 to 950 m (0 ft to 3,117 ft) 951 m to 3,050 m (3,118 ft to 10,000 ft) 0 m to 3,050 m (0 ft to 10,000 ft) Reltive humidity Mximum dew point 8% to 85% 24 C (75 F) 8% to 85% 27 C (80.6 F) Chpter 2. Introducing the SAN Volume Controller hrdwre components 33

52 Tble 13. Physicl specifictions (continued) Environment Temperture Altitude Storing 1 C to 60 C (33.8 F to F) Shipping -40 C to 60 C (-40 F to F) 0 m to 3,050 m (0 ft to 10,000 ft) 0 m to 10,700 m (0 ft to 34,991 ft) Reltive humidity Mximum dew point 5% to 80% 29 C (84.2 F) 5% to 100% 29 C (84.2 F) Note: Decrese the mximum system temperture by 1 C for every 175-m increse in ltitude. Prepring your environment The following tbles list the physicl chrcteristics of the 2145-DH8 node. Dimensions nd weight Use the prmeters tht re shown in Tble 14 to ensure tht spce is vilble in rck cpble of supporting the node. Tble 14. Dimensions nd weight Height Width Depth Mximum weight 86 mm (3.4 in.) 445 mm (17.5 in) 746 mm (29.4 in) 25 kg (55 lb) to 30 kg (65 lb) depending on configurtion Additionl spce requirements Ensure tht spce is vilble in the rck for the dditionl spce requirements round the node, s shown in Tble 15. Tble 15. Additionl spce requirements Additionl spce Loction requirements Reson Left side nd right side Minimum: 50 mm (2 in.) Cooling ir flow Bck Minimum: 100 mm (4 in.) Cble exit Mximum het output of ech 2145-DH8 node The node dissiptes the mximum het output tht is given in Tble 16. Tble 16. Mximum het output of ech 2145-DH8 node Model Het output per node 2145-DH8 v Minimum configurtion: Btu per hour (AC 123 wtts) v Mximum configurtion: Btu per hour (AC 1020 wtts) 34 SAN Volume Controller: Troubleshooting Guide

53 SAN Volume Controller 2145-CG8 environment requirements Before the SAN Volume Controller 2145-CG8 is instlled, the physicl environment must meet certin requirements. This includes verifying tht dequte spce is vilble nd tht requirements for power nd environmentl conditions re met. Input-voltge requirements Ensure tht your environment meets the voltge requirements tht re shown in Tble 17. Tble 17. Input-voltge requirements Voltge Frequency 200 V V single phse c 50 Hz or 60 Hz Attention: v If the uninterruptible power supply is cscded from nother uninterruptible power supply, the source uninterruptible power supply must hve t lest three times the cpcity per phse nd the totl hrmonic distortion must be less thn 5%. v The uninterruptible power supply lso must hve input voltge cpture tht hs slew rte of no more thn 3 Hz per second. Mximum power requirements for ech node Ensure tht your environment meets the power requirements s shown in Tble 18. The mximum power tht is required depends on the node type nd the optionl fetures tht re instlled. Tble 18. Mximum power consumption Components SAN Volume Controller 2145-CG8 nd 2145 UPS-1U Power requirements 200 W For ech redundnt c-power switch, dd 20 W to the power requirements. For the high-speed SAS dpter with from one to four solid-stte drives, dd 50 W to the power requirements. Circuit breker requirements The 2145 UPS-1U hs n integrted circuit breker nd does not require dditionl protection. Environment requirements without redundnt c power If you re not using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 19 on pge 36. Chpter 2. Introducing the SAN Volume Controller hrdwre components 35

54 Tble 19. Environment requirements without redundnt c power Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes 10 C - 35 C (50 F - 95 F) 10 C - 32 C (50 F - 90 F) Turned off 10 C - 43 C (50 F F) Storing 1 C - 60 C (34 F F) Shipping -20 C - 60 C (-4 F F) 0 m m (0 ft ft) 914 m m (3000 ft ft) 0 m m (0 ft ft) 0 m m (0 ft ft) 0 m m (0 ft ft) Reltive humidity 8% - 80% noncondensing 8% - 80% noncondensing 8% - 80% noncondensing 5% - 80% noncondensing 5% - 100% condensing, but no precipittion Mximum wet bulb temperture 23 C (73 F) 23 C (73 F) 27 C (81 F) 29 C (84 F) 29 C (84 F) Environment requirements with redundnt c power If you re using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 20. Tble 20. Environment requirements with redundnt c power Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes 15 C - 32 C (59 F - 90 F) 15 C - 32 C (59 F - 90 F) Turned off 10 C - 43 C (50 F F) Storing 1 C - 60 C (34 F F) Shipping -20 C - 60 C (-4 F F) 0 m m (0 ft ft) 914 m m (3000 ft ft) 0 m m (0 ft ft) 0 m m (0 ft ft) 0 m m (0 ft ft) Reltive humidity 20% - 80% noncondensing 20% - 80% noncondensing 20% - 80% noncondensing 5% - 80% noncondensing 5% - 100% condensing, but no precipittion Mximum wet bulb temperture 23 C (73 F) 23 C (73 F) 27 C (81 F) 29 C (84 F) 29 C (84 F) Prepring your environment The following tbles list the physicl chrcteristics of the SAN Volume Controller 2145-CG8 node. Dimensions nd weight Use the prmeters tht re shown in Tble 21 to ensure tht spce is vilble in rck cpble of supporting the node. Tble 21. Dimensions nd weight Height Width Depth Mximum weight 4.3 cm (1.7 in.) 44 cm (17.3 in.) 73.7 cm (29 in.) 15 kg (33 lb) 36 SAN Volume Controller: Troubleshooting Guide

55 Additionl spce requirements Ensure tht spce is vilble in the rck for the dditionl spce requirements round the node, s shown in Tble 22. Tble 22. Additionl spce requirements Additionl spce Loction requirements Reson Left side nd right side Minimum: 50 mm (2 in.) Cooling ir flow Bck Minimum: 100 mm (4 in.) Cble exit Mximum het output of ech SAN Volume Controller 2145-CG8 node The node dissiptes the mximum het output tht is given in Tble 23. Tble 23. Mximum het output of ech SAN Volume Controller 2145-CG8 node Model Het output per node SAN Volume Controller 2145-CG8 160 W (546 Btu per hour) SAN Volume Controller 2145-CG8 plus flsh 210 W (717 Btu per hour) drive Mximum het output of ech 2145 UPS-1U The 2145 UPS-1U dissiptes the mximum het output tht is given in Tble 24. Tble 24. Mximum het output of ech 2145 UPS-1U Model Het output per node Mximum het output of 2145 UPS-1U 10 W (34 Btu per hour) during norml opertion Mximum het output of 2145 UPS-1U 100 W (341 Btu per hour) during bttery opertion SAN Volume Controller 2145-CF8 environment requirements Before you instll SAN Volume Controller 2145-CF8 node, your physicl environment must meet certin requirements. This includes verifying tht dequte spce is vilble nd tht requirements for power nd environmentl conditions re met. Input-voltge requirements Ensure tht your environment meets the voltge requirements tht re shown in Tble 25. Tble 25. Input-voltge requirements Voltge Frequency V single phse c 50 or 60 Hz Chpter 2. Introducing the SAN Volume Controller hrdwre components 37

56 Attention: v If the uninterruptible power supply is cscded from nother uninterruptible power supply, the source uninterruptible power supply must hve t lest three times the cpcity per phse nd the totl hrmonic distortion must be less thn 5%. v The uninterruptible power supply lso must hve input voltge cpture tht hs slew rte of no more thn 3 Hz per second. Power requirements for ech node Ensure tht your environment meets the following power requirements. Ensure tht your environment meets the power requirements s shown in Tble 26. Tble 26. Power requirements for ech node Components SAN Volume Controller 2145-CF8 node nd 2145 UPS-1U power supply Power requirements 200 W Notes: v SAN Volume Controller 2145-CF8 nodes cnnot connect to ll revisions of the 2145 UPS-1U power supply unit. The SAN Volume Controller 2145-CF8 nodes require the 2145 UPS-1U power supply unit prt number 31P1318. This unit hs two power outlets tht re ccessible. Erlier revisions of the 2145 UPS-1U power supply unit hve only one power outlet tht is ccessible nd re not suitble. v For ech redundnt c-power switch, dd 20 W to the power requirements. v For ech high-speed SAS dpter with one to four flsh drives, dd 50 W to the power requirements. Circuit breker requirements The 2145 UPS-1U hs n integrted circuit breker nd does not require dditionl protection. Environment requirements without redundnt c power If you re not using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 27. Tble 27. Environment requirements without redundnt c power Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes 10 C to 35 C (50 F to 95 F) 10 C to 32 C (50 F to 90 F) Turned off 10 C to 43 C (50 F to 110 F) Storing 1 C to 60 C (34 F to 140 F) m ( ft) m ( ft) m ( ft) m ( ft) Reltive humidity 8% to 80% noncondensing 8% to 80% noncondensing 8% to 80% noncondensing 5% to 80% noncondensing Mximum wet bulb temperture 23 C (73 F) 23 C (73 F) 27 C (81 F) 29 C (84 F) 38 SAN Volume Controller: Troubleshooting Guide

57 Tble 27. Environment requirements without redundnt c power (continued) Environment Temperture Altitude Shipping -20 C to 60 C (-4 F to 140 F) m ( ft) Reltive humidity 5% to 100% condensing, but no precipittion Mximum wet bulb temperture 29 C (84 F) Environment requirements with redundnt c power If you re using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 28. Tble 28. Environment requirements with redundnt c power Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes 15 C to 32 C (59 F to 90 F) 15 C to 32 C (59 F to 90 F) Turned off 10 C to 43 C (50 F to 110 F) Storing 1 C to 60 C (34 F to 140 F) Shipping -20 C to 60 C (-4 F to 140 F) m ( ft) m ( ft) m ( ft) m ( ft) m ( ft) Reltive humidity 20% to 80% noncondensing 20% to 80% noncondensing 20% to 80% noncondensing 5% to 80% noncondensing 5% to 100% condensing, but no precipittion Mximum wet bulb temperture 23 C (73 F) 23 C (73 F) 27 C (81 F) 29 C (84 F) 29 C (84 F) Prepring your environment The following tbles list the physicl chrcteristics of the SAN Volume Controller 2145-CF8 node. Dimensions nd weight Use the prmeters tht re shown in Tble 29 to ensure tht spce is vilble in rck cpble of supporting the node. Tble 29. Dimensions nd weight Height Width Depth Mximum weight 43 mm (1.69 in.) 440 mm (17.32 in.) 686 mm (27 in.) 12.7 kg (28 lb) Additionl spce requirements Ensure tht spce is vilble in the rck for the dditionl spce requirements round the node, s shown in Tble 30 on pge 40. Chpter 2. Introducing the SAN Volume Controller hrdwre components 39

58 Tble 30. Additionl spce requirements Additionl spce Loction requirements Reson Left nd right sides 50 mm (2 in.) Cooling ir flow Bck Minimum: 100 mm (4 in.) Cble exit Het output of ech SAN Volume Controller 2145-CF8 node The node dissiptes the mximum het output tht is given in Tble 31. Tble 31. Het output of ech SAN Volume Controller 2145-CF8 node Model SAN Volume Controller 2145-CF8 SAN Volume Controller 2145-CF8 nd up to four optionl flsh drives Mximum het output of 2145 UPS-1U during typicl opertion Mximum het output of 2145 UPS-1U during bttery opertion Het output per node 160 W (546 Btu per hour) 210 W (717 Btu per hour) 10 W (34 Btu per hour) 100 W (341 Btu per hour) SAN Volume Controller A4 environment requirements Before the SAN Volume Controller A4 is instlled, the physicl environment must meet certin requirements. This includes verifying tht dequte spce is vilble nd tht requirements for power nd environmentl conditions re met. Input-voltge requirements Ensure tht your environment meets the voltge requirements tht re shown in Tble 32. Tble 32. Input-voltge requirements Voltge Frequency V single phse c 50 or 60 Hz Attention: v If the uninterruptible power supply is cscded from nother uninterruptible power supply, the source uninterruptible power supply must hve t lest three times the cpcity per phse nd the totl hrmonic distortion must be less thn 5%. v The uninterruptible power supply lso must hve input voltge cpture tht hs slew rte of no more thn 3 Hz per second. Power requirements for ech node Ensure tht your environment meets the power requirements s shown in Tble 33 on pge 41. The power tht is required depends on the node type nd whether the redundnt c power feture is used. 40 SAN Volume Controller: Troubleshooting Guide

59 Tble 33. Power requirements for ech node Components SAN Volume Controller A4 nd 2145 UPS-1U Power requirements 180 W For ech redundnt c-power switch, dd 20 W to the power requirements. Circuit breker requirements The 2145 UPS-1U hs n integrted circuit breker nd does not require dditionl protection. Environment requirements without redundnt c power If you re not using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 34. Tble 34. Environment requirements without redundnt c power Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes 10 C to 35 C (50 F to 95 F) 10 C to 32 C (50 F to 90 F) Turned off 10 C to 43 C (50 F to 109 F) Storing 1 C to 60 C (34 F to 140 F) Shipping -20 C to 60 C (-4 F to 140 F) m ( ft) m ( ft) m ( ft) m ( ft) m ( ft) Reltive humidity 8% to 80% noncondensing 8% to 80% noncondensing 8% to 80% noncondensing 5% to 80% noncondensing 5% to 100% condensing, but no precipittion Mximum wet bulb temperture 23 C (73 F) 23 C (73 F) 27 C (81 F) 29 C (84 F) 29 C (84 F) Environment requirements with redundnt c power If you re using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 35. Tble 35. Environment requirements with redundnt c power Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes 15 C to 32 C (59 F to 90 F) 15 C to 32 C (59 F to 90 F) Turned off 10 C to 43 C (50 F to 109 F) Storing 1 C to 60 C (34 F to 140 F) m ( ft) m ( ft) m ( ft) m ( ft) Reltive humidity 20% to 80% noncondensing 20% to 80% noncondensing 20% to 80% noncondensing 5% to 80% noncondensing Mximum wet bulb temperture 23 C (73 F) 23 C (73 F) 27 C (81 F) 29 C (84 F) Chpter 2. Introducing the SAN Volume Controller hrdwre components 41

60 Tble 35. Environment requirements with redundnt c power (continued) Environment Temperture Altitude Shipping -20 C to 60 C (-4 F to 140 F) m ( ft) Reltive humidity 5% to 100% condensing, but no precipittion Mximum wet bulb temperture 29 C (84 F) Prepring your environment The following tbles list the physicl chrcteristics of the SAN Volume Controller A4 node. Dimensions nd weight Use the prmeters tht re shown in Tble 36 to ensure tht spce is vilble in rck cpble of supporting the node. Tble 36. Dimensions nd weight Height Width Depth Mximum weight 43 mm (1.75 in.) 440 mm (17.32 in.) 559 mm (22 in.) 10.1 kg (22 lb) Additionl spce requirements Ensure tht spce is vilble in the rck for the dditionl spce requirements round the node, s shown in Tble 37. Tble 37. Additionl spce requirements Additionl spce Loction requirements Reson Left nd right sides Minimum: 50 mm (2 in.) Cooling ir flow Bck Minimum: 100 mm (4 in.) Cble exit Het output of ech SAN Volume Controller A4 node The node dissiptes the mximum het output tht is given in Tble 38. Tble 38. Het output of ech SAN Volume Controller A4 node Model Het output per node SAN Volume Controller A4 140 W (478 Btu per hour) SAN Volume Controller G4 environment requirements Before the SAN Volume Controller G4 is instlled, the physicl environment must meet certin requirements. This includes verifying tht dequte spce is vilble nd tht requirements for power nd environmentl conditions re met. Input-voltge requirements Ensure tht your environment meets the voltge requirements tht re shown in Tble 39 on pge SAN Volume Controller: Troubleshooting Guide

61 Tble 39. Input-voltge requirements Voltge Frequency V single phse c 50 or 60 Hz Attention: v If the uninterruptible power supply is cscded from nother uninterruptible power supply, the source uninterruptible power supply must hve t lest three times the cpcity per phse nd the totl hrmonic distortion must be less thn 5%. v The uninterruptible power supply lso must hve input voltge cpture tht hs slew rte of no more thn 3 Hz per second. Power requirements for ech node Ensure tht your environment meets the power requirements s shown in Tble 40. The power tht is required depends on the node type nd whether the redundnt c power feture is used. Tble 40. Power requirements for ech node Components SAN Volume Controller G4 nd 2145 UPS-1U Power requirements 470 W For ech redundnt c-power switch, dd 20 W to the power requirements. Circuit breker requirements The 2145 UPS-1U hs n integrted circuit breker nd does not require dditionl protection. Environment requirements without redundnt c power If you re not using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 41. Tble 41. Environment requirements without redundnt c power Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes 10 C to 35 C (50 F to 95 F) 10 C to 32 C (50 F to 90 F) Turned off 10 C to 43 C (50 F to 110 F) Storing 1 C to 60 C (34 F to 140 F) Shipping -20 C to 60 C (-4 F to 140 F) m ( ft) m ( ft) m ( ft) m ( ft) m ( ft) Reltive humidity 8% to 80% noncondensing 8% to 80% noncondensing 8% to 80% noncondensing 5% to 80% noncondensing 5% to 100% condensing, but no precipittion Mximum wet bulb temperture 23 C (73 F) 23 C (73 F) 27 C (81 F) 29 C (84 F) 29 C (84 F) Chpter 2. Introducing the SAN Volume Controller hrdwre components 43

62 Environment requirements with redundnt c power If you re using redundnt c power, ensure tht your environment flls within the rnges tht re shown in Tble 42. Tble 42. Environment requirements with redundnt c power Environment Temperture Altitude Operting in lower ltitudes Operting in higher ltitudes 15 C to 32 C (59 F to 90 F) 15 C to 32 C (59 F to 90 F) Turned off 10 C to 43 C (50 F to 110 F) Storing 1 C to 60 C (34 F to 140 F) Shipping -20 C to 60 C (-4 F to 140 F) m ( ft) m ( ft) m ( ft) m ( ft) m ( ft) Reltive humidity 20% to 80% noncondensing 20% to 80% noncondensing 20% to 80% noncondensing 5% to 80% noncondensing 5% to 100% condensing, but no precipittion Mximum wet bulb temperture 23 C (73 F) 23 C (73 F) 27 C (81 F) 29 C (84 F) 29 C (84 F) Prepring your environment The following tbles list the physicl chrcteristics of the SAN Volume Controller G4 node. Dimensions nd weight Use the prmeters tht re shown in Tble 43 to ensure tht spce is vilble in rck cpble of supporting the node. Tble 43. Dimensions nd weight Height Width Depth Mximum weight 43 mm (1.69 in.) 440 mm (17.32 in.) 686 mm (27 in.) 12.7 kg (28 lb) Additionl spce requirements Ensure tht spce is vilble in the rck for the dditionl spce requirements round the node, s shown in Tble 44. Tble 44. Additionl spce requirements Additionl spce Loction requirements Reson Left nd right sides 50 mm (2 in.) Cooling ir flow Bck Minimum: 100 mm (4 in.) Cble exit 44 SAN Volume Controller: Troubleshooting Guide

63 Het output of ech SAN Volume Controller G4 node The node dissiptes the mximum het output tht is given in Tble 45. Tble 45. Het output of ech SAN Volume Controller G4 node Model Het output per node SAN Volume Controller G4 400 W (1350 Btu per hour) Redundnt c-power switch The redundnt c-power switch is n optionl feture tht mkes the SAN Volume Controller nodes resilient to the filure of single power circuit. The redundnt c-power switch is not replcement for n uninterruptible power supply. You must still use uninterruptible power supply for ech node. Restriction: The Redundnt c-power switch does not pply to SAN Volume Controller 2145-DH8. You must connect the redundnt c-power switch to two independent power circuits. One power circuit connects to the min power input port nd the other power circuit connects to the bckup power-input port. If the min power to the SAN Volume Controller node fils for ny reson, the redundnt c-power switch utomticlly uses the bckup power source. When power is restored, the redundnt c-power switch utomticlly chnges bck to using the min power source. Plce the redundnt c-power switch in the sme rck s the SAN Volume Controller node. The redundnt c-power switch logiclly sits between the rck power distribution unit nd the 2145 UPS-1U. You cn use single redundnt c-power switch to power one or two SAN Volume Controller nodes. If you use the redundnt c-power switch to power two nodes, the nodes must be in different I/O groups. If the redundnt c-power switch fils or requires mintennce, both nodes turn off. Becuse the nodes re in two different I/O groups, the hosts do not lose ccess to the bck-end disk dt. For mximum resilience to filure, use one redundnt c-power switch to power ech SAN Volume Controller node. Figure 37 on pge 46 shows redundnt c-power switch. Chpter 2. Introducing the SAN Volume Controller hrdwre components 45

64 svc00297 Figure 37. Photo of the redundnt c-power switch You must properly cble the redundnt c-power switch units in your environment. For more informtion, see the Plnning topic in the Informtion Center. Redundnt c-power environment requirements Ensure tht your physicl site meets the instlltion requirements for the redundnt c-power switch. The redundnt c-power switch requires two independent power sources tht re provided through two rck-mounted power distribution units (PDUs). The PDUs must hve IEC320-C13 outlets. The redundnt c-power switch comes with two IEC 320-C19 to C14 power cbles to connect to rck PDUs. There re no country-specific cbles for the redundnt c-power switch. The power cble between the redundnt c-power switch nd the 2145 UPS-1U is rted t 10 A. Redundnt c-power switch specifictions The following tbles list the physicl chrcteristics of the redundnt c-power switch. Dimensions nd weight Ensure tht spce is vilble in rck tht is cpble of supporting the redundnt c-power switch. Height Width Depth Mximum weight 43 mm (1.69 in.) 192 mm (7.56 in.) 240 mm 2.6 kg (5.72 lb) Additionl spce requirements Ensure tht spce is lso vilble in the rck for the side mounting pltes on either side of the redundnt c-power switch. Loction Width Reson Left side 124 mm (4.89 in.) Side mounting plte 46 SAN Volume Controller: Troubleshooting Guide

65 Loction Width Reson Right side 124 mm (4.89 in.) Side mounting plte Het output (mximum) The mximum het output tht is dissipted inside the redundnt c-power switch is pproximtely 20 wtts (70 Btu per hour). Cbling of redundnt c-power switch (exmple) You must properly cble the redundnt c-power switch units in your environment. Figure 38 on pge 48 shows n exmple of the min wiring connections for SAN Volume Controller clustered system with the redundnt c-power switch feture. This exmple is designed to clerly show the cble connections; the components re not positioned s they would be in rck. Figure 39 on pge 50 shows typicl rck instlltion. The four-node clustered system consists of two I/O groups: v I/O group 0 contins nodes A nd B v I/O group 1 contins nodes C nd D Chpter 2. Introducing the SAN Volume Controller hrdwre components 47

66 svc00358_cf8 Figure 38. A four-node SAN Volume Controller system with the redundnt c-power switch feture 1 I/O group 0 2 SAN Volume Controller node A UPS-1U A 4 SAN Volume Controller node B UPS-1U B 6 I/O group 1 7 SAN Volume Controller node C UPS-1U C 9 SAN Volume Controller node D UPS-1U D 11 Redundnt c-power switch 1 12 Redundnt c-power switch 2 13 Site PDU X (C13 outlets) 14 Site PDU Y (C13 outlets) 48 SAN Volume Controller: Troubleshooting Guide

67 The site PDUs X nd Y ( 13 nd 14 ) re powered from two independent power sources. In this exmple, only two redundnt c-power switch units re used, nd ech power switch powers one node in ech I/O group. However, for mximum redundncy, use one redundnt c-power switch to power ech node in the system. Some SAN Volume Controller node types hve two power supply units. Both power supplies must be connected to the sme 2145 UPS-1U, s shown by node A nd node B. The SAN Volume Controller 2145-CG8 is n exmple of node tht hs two power supplies. The SAN Volume Controller A4 is n exmple of node tht hs single power supply. Figure 39 on pge 50 shows n 8 node cluster, with one redundnt c-power switch per node instlled in rck using best loction prctices, the cbles between the components re shown. Chpter 2. Introducing the SAN Volume Controller hrdwre components 49

68 20A 20A 12A Mx. ON OFF 12A Mx. ON OFF CONNECT ONLY IBM SAN VOLUME CONTROLLERS TO THESE OUTLETS. SEE SAN VOLUME CONTROLLER INSTALLATION GUIDE. CONNECT ONLY IBM SAN VOLUME CONTROLLERS TO THESE OUTLETS. SEE SAN VOLUME CONTROLLER INSTALLATION GUIDE. AC DC CONNECT ONLY IBM SAN VOLUME CONTROLLERS TO THESE OUTLETS. SEE SAN VOLUME CONTROLLER INSTALLATION GUIDE. CONNECT ONLY IBM SAN VOLUME CONTROLLERS TO THESE OUTLETS. SEE SAN VOLUME CONTROLLER INSTALLATION GUIDE. CONNECT ONLY IBM SAN VOLUME CONTROLLERS TO THESE OUTLETS. SEE SAN VOLUME CONTROLLER INSTALLATION GUIDE. CONNECT ONLY IBM SAN VOLUME CONTROLLERS TO THESE OUTLETS. SEE SAN VOLUME CONTROLLER INSTALLATION GUIDE. CONNECT ONLY IBM SAN VOLUME CONTROLLERS TO THESE OUTLETS. SEE SAN VOLUME CONTROLLER INSTALLATION GUIDE. CONNECT ONLY IBM SAN VOLUME CONTROLLERS TO THESE OUTLETS. SEE SAN VOLUME CONTROLLER INSTALLATION GUIDE. AC DC AC DC AC DC AC DC AC DC AC DC AC DC AC DC AC DC AC DC AC DC AC DC Circuit Breker AC BRANCH A Circuit Breker DC AC DC AC DC 20A 20A 12A Mx. 12A Mx. ON OFF ON OFF 8U Reserved for dt wiring ptch pnel or ller pnels SVC #8 IOGroup 3 Node B 1U Filler Pnel SVC #7 IOGroup 3 Node A 1U Filler Pnel SVC #6 IOGroup 2 Node B 1U Filler Pnel SVC #5 IOGroup 2 Node A 1U Filler Pnel SVC #4 IOGroup 1 Node B 1U Filler Pnel SVC #3 IOGroup 1 Node A 1U Filler Pnel SVC #2 IOGroup 0 Node B 1U Filler Pnel SVC #1 IOGroup 0 Node A 1U Filler Pnel 1U ller pnel or optionl 1U monitor 1U ller pnel or optionl SSPC server ATT ENTI ON ATT ENTI ON ATT ENTI ON Circuit Breker BRANCH B ATT ENTI ON BRANCH A Circuit Breker ATT ENTI ON ATT ENTI ON ATT ENTI ON ATT ENTI ON 1U Filler Pnel BRANCH B 1U Filler Pnel Left Side epdu Min Input Figure 39. Rck cbling exmple. Bckup Input Right Side epdu svc00765 Uninterruptible power supply The uninterruptible power supply protects SAN Volume Controller node ginst blckouts, brownouts, nd power surges. The uninterruptible power supply contins power sensor to monitor the supply nd bttery to provide power until n orderly shutdown of the system cn be initited. SAN Volume Controller models use the 2145 UPS-1U. 50 SAN Volume Controller: Troubleshooting Guide

69 2145 UPS-1U A 2145 UPS-1U is used exclusively to mintin dt tht is held in the SAN Volume Controller dynmic rndom ccess memory (DRAM) in the event of n unexpected loss of externl power. This use differs from the trditionl uninterruptible power supply tht enbles continued opertion of the device tht it supplies when power is lost. With 2145 UPS-1U, dt is sved to the internl disk of the SAN Volume Controller node. The uninterruptible power supply units re required to power the SAN Volume Controller nodes even when the input power source is considered uninterruptible. Note: The uninterruptible power supply mintins continuous SAN Volume Controller-specific communictions with its ttched SAN Volume Controller nodes. A SAN Volume Controller node cnnot operte without the uninterruptible power supply. The uninterruptible power supply must be used in ccordnce with documented guidelines nd procedures nd must not power ny equipment other thn SAN Volume Controller node UPS-1U opertion Ech SAN Volume Controller node monitors the opertionl stte of the uninterruptible power supply to which it is ttched. If the 2145 UPS-1U reports loss of input power, the SAN Volume Controller node stops ll I/O opertions nd dumps the contents of its dynmic rndom ccess memory (DRAM) to the internl disk drive. When input power to the 2145 UPS-1U is restored, the SAN Volume Controller node restrts nd restores the originl contents of the DRAM from the dt sved on the disk drive. A SAN Volume Controller node is not fully opertionl until the 2145 UPS-1U bttery stte indictes tht it hs sufficient chrge to power the SAN Volume Controller node long enough to sve ll of its memory to the disk drive. In the event of power loss, the 2145 UPS-1U hs sufficient cpcity for the SAN Volume Controller to sve ll its memory to disk t lest twice. For fully chrged 2145 UPS-1U, even fter bttery chrge hs been used to power the SAN Volume Controller node while it sves dynmic rndom ccess memory (DRAM) dt, sufficient bttery chrge remins so tht the SAN Volume Controller node cn become fully opertionl s soon s input power is restored. Importnt: Do not shut down 2145 UPS-1U without first shutting down the SAN Volume Controller node tht it supports. Dt integrity cn be compromised by pushing the 2145 UPS-1U on/off button when the node is still operting. However, in the cse of n emergency, you cn mnully shut down the 2145 UPS-1U by pushing the 2145 UPS-1U on/off button when the node is still operting. Service ctions must then be performed before the node cn resume norml opertions. If multiple uninterruptible power supply units re shut down before the nodes they support, dt cn be corrupted. Connecting the 2145 UPS-1U to the SAN Volume Controller To provide redundncy nd concurrent mintennce, you must instll the SAN Volume Controller nodes in pirs. For connection to the 2145 UPS-1U, ech SAN Volume Controller of pir must be connected to only one 2145 UPS-1U. Chpter 2. Introducing the SAN Volume Controller hrdwre components 51

70 Note: A clustered system cn contin no more thn eight SAN Volume Controller nodes. The 2145 UPS-1U must be ttched to source tht is both single phse nd V. The 2145 UPS-1U hs n integrted circuit breker nd does not need externl protection. SAN Volume Controller provides cble bundle for connecting the uninterruptible power supply to node. This cble is used to connect both power supplies of node to the sme uninterruptible power supply. Single-power cble plus seril cble: v SAN Volume Controller G4 v SAN Volume Controller A4 Dul-power cble plus seril cble: v SAN Volume Controller 2145-CF8 v SAN Volume Controller 2145-CG8 The SAN Volume Controller softwre determines whether the input voltge to the uninterruptible power supply is within rnge nd sets n pproprite voltge lrm rnge on the uninterruptible power supply. The softwre continues to recheck the input voltge every few minutes. If it chnges substntilly but remins within the permitted rnge, the lrm limits re redjusted. Note: The 2145 UPS-1U is equipped with cble retention brcket tht keeps the power cble from disengging from the rer pnel. See the relted documenttion for more informtion UPS-1U controls nd indictors All controls nd indictors for the 2145 UPS-1U re locted on the front-pnel ssembly. LOAD 2 LOAD yyzvm Figure UPS-1U front-pnel ssembly 1 Lod segment 2 indictor 2 Lod segment 1 indictor 3 Alrm or service indictor 4 On-bttery indictor 5 Overlod indictor 6 Power-on indictor 7 On/off button 8 Test nd lrm reset button 52 SAN Volume Controller: Troubleshooting Guide

71 Lod segment 2 indictor: The lod segment 2 indictor on the 2145 UPS-1U is lit (green) when power is vilble to lod segment 2. When the lod segment 2 indictor is green, the 2145 UPS-1U is running normlly nd power is vilble to this segment. Lod segment 1 indictor: The lod segment 1 indictor on the 2145 UPS-1U is not currently used by the SAN Volume Controller. Note: When the 2145 UPS-1U is configured by the SAN Volume Controller, this lod segment is disbled. During norml opertion, the lod segment 1 indictor is off. A Do not use lbel covers the receptcles. Alrm indictor: If the lrm on the 2145 UPS-1U is flshing red, mintennce is required. If the lrm is on, go to the 2145 UPS-1U MAP to resolve the problem. On-bttery indictor: The mber on-bttery indictor is on when the 2145 UPS-1U is powered by the bttery. This indictes tht the min power source hs filed. If the on-bttery indictor is on, go to the 2145 UPS-1U MAP to resolve the problem. Overlod indictor: The overlod indictor lights up when the cpcity of the 2145 UPS-1U is exceeded. If the overlod indictor is on, go to MAP 5250: 2145 UPS-1U repir verifiction to resolve the problem. Power-on indictor: The power-on indictor is displyed when the 2145 UPS-1U is functioning. When the power-on indictor is stedy green, the 2145 UPS-1U is ctive. On or off button: The on or off button turns the power on or off for the 2145 UPS-1U. Turning on the 2145 UPS-1U After you connect the 2145 UPS-1U to the outlet, it remins in stndby mode until you turn it on. Press nd hold the on or off button until the power-on indictor is illuminted (pproximtely five seconds). On some versions of the 2145 UPS-1U, you might need pointed device, such s screwdriver, to press the on or off button. A self-test is initited tht tkes pproximtely 10 seconds, during which Chpter 2. Introducing the SAN Volume Controller hrdwre components 53

72 time the indictors re turned on nd off severl times. The 2145 UPS-1U then enters norml mode. Turning off the 2145 UPS-1U Press nd hold the on or off button until the power-on light is extinguished (pproximtely five seconds). On some versions of the 2145 UPS-1U, you might need pointed device, such s screwdriver, to press the on or off button. This plces the 2145 UPS-1U in stndby mode. You must then unplug the 2145 UPS-1U to turn off the unit. Attention: Do not turn off the uninterruptible power supply before you shut down the SAN Volume Controller node tht it is connected to. Alwys follow the instructions tht re provided in MAP 5350 to perform n orderly shutdown of SAN Volume Controller node. Test nd lrm reset button: Use the test nd lrm reset button to strt the self-test. To strt the self-test, press nd hold the test nd lrm reset button for three seconds. This button lso resets the lrm UPS-1U connectors nd switches The 2145 UPS-1U hs externl connectors nd dip switches. Loctions for the 2145 UPS-1U connectors nd switches Figure 41 shows the loction of the connectors nd switches on the 2145 UPS-1U svc00308 Figure UPS-1U connectors nd switches 1 Min power connector 2 Communiction port 3 Dip switches 4 Lod segment 1 receptcles 5 Lod segment 2 receptcles 2145 UPS-1U dip switches Figure 42 on pge 55 shows the dip switches, which cn be used to configure the input nd output voltge rnges. Becuse this function is performed by the SAN Volume Controller softwre, both switches must be left in the OFF position. 54 SAN Volume Controller: Troubleshooting Guide

73 ON 1 2 OFF svc00147 Figure UPS-1U dip switches 2145 UPS-1U ports not used The 2145 UPS-1U is equipped with ports tht re not used by the SAN Volume Controller nd hve not been tested. Use of these ports, in conjunction with the SAN Volume Controller or ny other ppliction tht might be used with the SAN Volume Controller, is not supported. Figure 43 shows the 2145 UPS-1U ports tht re not used. Figure 43. Ports not used by the 2145 UPS-1U 1 USB interfce port 2 Network ports 3 Lod segment receptcles 2145 UPS-1U power connector Figure 44 shows the power connector for the 2145 UPS-1U. Neutrl Ground Live Figure 44. Power connector Uninterruptible power-supply environment requirements An uninterruptible power-supply environment requires tht certin specifictions for the physicl site of the SAN Volume Controller must be met UPS-1U environment All SAN Volume Controller models re supported with the 2145 UPS-1U. Chpter 2. Introducing the SAN Volume Controller hrdwre components 55

74 2145 UPS-1U specifictions The following tbles describe the physicl chrcteristics of the 2145 UPS-1U UPS-1U dimensions nd weight Ensure tht spce is vilble in rck tht is cpble of supporting the 2145 UPS-1U. Height Width Depth Mximum weight 44 mm (1.73 in.) 439 mm (17.3 in.) 579 mm (22.8 in.) 16 kg (35.3 lb) Note: The 2145 UPS-1U pckge, which includes support rils, weighs 18.8 kg (41.4 lb). Het output The 2145 UPS-1U unit produces the following pproximte het output. Het output during norml Het output during bttery Model opertion opertion 2145 UPS-1U 10 W (34 Btu per hour) 150 W (512 Btu per hour) Defining the SAN Volume Controller FRUs The SAN Volume Controller node, redundnt c-power switch, nd uninterruptible power supply ech consist of one or more field-replceble units (FRUs). SAN Volume Controller FRUs The SAN Volume Controller nodes ech consist of severl field-replceble units (FRUs), such s the Fibre Chnnel dpter, service controller, disk drive, microprocessor, memory module, CMOS bttery, power supply ssembly, fn ssembly, nd the opertor-informtion pnel. SAN Volume Controller 2145-DH8 FRUs Refer to the SAN Volume Controller 2145-DH8 prts topic for the list of SAN Volume Controller 2145-DH8 FRUs. SAN Volume Controller F FRUs The following tbles provide informtion the SAN Volume Controller F expnsion enclosure prts nd SAS drives. Tble 46 lists the FRUs for the SAN Volume Controller F expnsion enclosure. Tble 46. FRUs for SAN Volume Controller F expnsion enclosure Description FRU prt number Expnsion enclosure midplne ssembly 64P8445 Expnsion cnister 64P8448 Expnsion enclosure power supply unit 98Y2218 (PSU), AC 56 SAN Volume Controller: Troubleshooting Guide

75 Tble 46. FRUs for SAN Volume Controller F expnsion enclosure (continued) Description FRU prt number Smll-form fctor SSD, 200 GB 31P1818 Smll-form fctor SSD, 400 GB 31P1819 Smll-form fctor SSD, 800 GB 31P1820 Smll-form fctor drive, blnk expnsion 45W8680 Rck ril kit, expnsion enclosure 64P8449 Left enclosure bezel, expnsion enclosure 00Y2450 Right enclosure bezel, SFF expnsion 00Y2512 enclosure Tble 47 lists the FRUs for the SAN Volume Controller F SAS drive units. Tble 47. FRU prt for the SAN Volume Controller F SAS drive units Description FRU prt number Mini SAS HD to mini SAS HD, 1.5 m, 12 00AR311 Gb. Mini SAS HD to mini SAS HD, 3.0 m, 12 00AR317 Gb. SAN Volume Controller 2145-CG8 FRUs Tble 48 provides brief description of echsan Volume Controller 2145-CG8 FRU. Tble 48. SAN Volume Controller 2145-CG8 FRU descriptions FRU Description System bord The system bord for the SAN Volume Controller 2145-CG8 node. Short-wve smll form-fctor pluggble (SFP) trnsceiver A compct opticl trnsceiver tht provides the opticl interfce to Fibre Chnnel cble. The trnsceiver is cpble of utonegotiting 2, 4, or 8 gigbits-per-second (Gbps) short-wve opticl connection on the four-port Fibre Chnnel dpter. Note: It is possible tht smll form-fctor pluggble (SFP) trnsceivers, other thn those trnsceiver shipped with the product, re in use on the Fibre Chnnel host bus dpter. It is customer responsibility to obtin replcement prts for such SFP trnsceivers. The FRU prt number is shown s "Non stndrd - supplied by customer" in the vitl product dt. Chpter 2. Introducing the SAN Volume Controller hrdwre components 57

76 Tble 48. SAN Volume Controller 2145-CG8 FRU descriptions (continued) FRU Description Long-wve smll form-fctor pluggble (SFP) trnsceiver Four-port Fibre Chnnel host bus dpter (HBA) Service controller Service controller cble Disk drive Disk signl cble Disk power cble Disk controller USB riser crd for the disk controller Disk bckplne Memory module Microprocessor Power supply unit CMOS bttery Opertor-informtion pnel A compct opticl trnsceiver tht provides the opticl interfce to Fibre Chnnel cble. The trnsceiver is cpble of utonegotiting 2, 4, or 8 Gbps short-wve opticl connection on the 4-port Fibre Chnnel dpter. Note: It is possible tht smll form-fctor pluggble (SFP) trnsceivers other thn those shipped with the product re in use on the Fibre Chnnel host bus dpter. It is customer responsibility to obtin replcement prts for such SFP trnsceivers. The FRU prt number is shown s "Non stndrd - supplied by customer" in the vitl product dt. The SAN Volume Controller 2145-CG8 is connected to the Fibre Chnnel fbric through the Fibre Chnnel HBA, which is in PCI slot 1. The dpter ssembly includes the Fibre Chnnel PCI Express dpter, four short-wve SFP trnsceivers, the riser crd, nd brcket. The unit tht provides the service functions nd the front pnel disply nd buttons. The USB cble tht is used to connect the service controller to the system bord. The seril-ttched SCSI (SAS) 2.5" disk drive. A 200 mm SAS disk-signl cble. The power cble for the 2.5" SAS system disk. A SAS controller crd for the SAS 2.5" disk drive. The riser crd tht connects the disk controller to the system bord nd provides the USB port to which the service controller cble connects. The hot-swp SAS 2.5" disk drive bckplne. An 8 GB DDR RX4 LP RDIMM memory module. The microprocessor on the system bord: 2.53 GHz qud-core microprocessor. An ssembly tht provides dc power to the node. A 3.0 V bttery on the system bord tht mintins power to bck up the system BIOS settings. The informtion pnel tht includes the power-control button nd LEDs tht indicte system-bord errors, hrd disk drive ctivity, nd power sttus. 58 SAN Volume Controller: Troubleshooting Guide

77 Tble 48. SAN Volume Controller 2145-CG8 FRU descriptions (continued) FRU Description Opertor-informtion pnel cble A cble tht connects the opertor-informtion pnel to the system bord. Fn ssembly A fn ssembly tht is used in ll the fn positions. Power cble ssembly The cble ssembly tht connects the SAN Volume Controller nd the 2145 UPS-1U. The ssembly consists of two power cbles nd seril cble tht is bundled together. Blnk drive by filler ssembly A blnk drive by filler ssembly. Alcohol wipe A clening wipe. Therml grese Grese tht is used to provide therml sel between processor nd het sink. SAN Volume Controller 2145-CF8 FRUs Tble 49 provides brief description of ech SAN Volume Controller 2145-CF8 FRU. Tble 49. SAN Volume Controller 2145-CF8 FRU descriptions FRU Description System bord The system bord for the SAN Volume Controller 2145-CF8 node. Fibre Chnnel smll form-fctor pluggble (SFP) trnsceiver Four-port Fibre Chnnel host bus dpter (HBA) Service controller Service controller cble Disk drive Disk signl cble A compct opticl trnsceiver tht provides the opticl interfce to Fibre Chnnel cble. The trnsceiver is cpble of utonegotiting 2, 4, or 8 Gbps short-wve opticl connection on the 4-port Fibre Chnnel dpter. Note: It is possible tht SFPs other thn those shipped with the product re in use on the Fibre Chnnel host bus dpter. It is customer responsibility to obtin replcement prts for such SFP trnsceivers. The FRU prt number is shown s "Non stndrd - supplied by customer" in the vitl product dt. The SAN Volume Controller 2145-CF8 is connected to the Fibre Chnnel fbric through the Fibre Chnnel HBA, which is in PCI slot 1. The dpter ssembly includes the Fibre Chnnel PCI Express dpter, four short-wve SFP trnsceivers, the riser crd, nd brcket. The unit tht provides the service functions nd the front pnel disply nd buttons. The USB cble tht is used to connect the service controller to the system bord. The seril-ttched SCSI (SAS) 2.5" disk drive. A 200 mm SAS disk-signl cble. Chpter 2. Introducing the SAN Volume Controller hrdwre components 59

78 Tble 49. SAN Volume Controller 2145-CF8 FRU descriptions (continued) FRU Description Disk power cble A SAS disk-power cble. Disk controller A SAS controller crd for the SAS 2.5" disk drive. Disk controller / USB riser crd The riser crd tht connects the disk controller to the system bord nd provides the USB port to which the service controller cble connects. Disk bckplne The hot-swp SAS 2.5" disk drive bckplne. Memory module A 4 GB DDR RX4 LP RDIMM memory module Microprocessor The microprocessor on the system bord GHz qud-core microprocessor. Power supply unit An ssembly tht provides dc power to the SAN Volume Controller 2145-CF8 node. CMOS bttery A 3.0 V bttery on the system bord tht mintins power to bck up the system BIOS settings. Opertor-informtion pnel The informtion pnel tht includes the power-control button nd LEDs tht indicte system-bord errors, hrd disk drive ctivity, nd power sttus. Opertor-informtion pnel cble A cble tht connects the opertor-informtion pnel to the system bord. Fn ssembly A fn ssembly tht is used in ll the fn positions. Power cble ssembly The cble ssembly tht connects the SAN Volume Controller nd the 2145 UPS-1U. The ssembly consists of two power cbles nd seril cble tht is bundled together. Alcohol wipe A clening wipe. Therml grese Grese tht is used to provide therml sel between processor nd het sink. Ethernet feture FRUs Tble 50 provides brief description of ech Ethernet feture FRU. Tble 50. Ethernet feture FRU descriptions FRU Description 10 Gbps Ethernet dpter A 10 Gbps Ethernet dpter. 10 Gbps Ethernet fiber SFP A 10 Gbps Ethernet fiber SFP. 60 SAN Volume Controller: Troubleshooting Guide

79 Flsh drive feture FRUs Tble 51 provides brief description of ech flsh drive feture FRU. Tble 51. Flsh drive feture FRU descriptions FRU Description High-speed SAS dpter An ssembly tht includes high-speed SAS dpter tht provides connectivity up to four flsh drives. The ssembly lso contins riser crd, blnking plte, nd screws. High-speed SAS cble The cble tht is used to connect the high-speed SAS dpter to the disk bckplte. Flsh drive A flsh drive, in crrier ssembly; different drive sizes re vilble UPS-1U FRUs Tble 52 provides brief description of ech 2145 UPS-1U FRU. Tble UPS-1U FRU descriptions FRU Description 2145 UPS-1U ssembly An uninterruptible power-supply ssembly for use with the SAN Volume Controller. Bttery pck ssembly The bttery tht provides bckup power to the SAN Volume Controller if power filure occurs. Power cble, PDU to 2145 UPS-1U Input power cble for connecting the 2145 UPS-1U to rck power distribution unit. Power cble, mins to UPS-1 (US) Input power cble for connecting the 2145 UPS-1U to mins power (United Sttes only). SAN Volume Controller A4 FRUs Tble 53 provides brief description of ech SAN Volume Controller A4 FRU. Tble 53. SAN Volume Controller A4 FRU descriptions FRU Description Memory module A 2 GB PC ECC memory module. Riser crd, PCI Express An interconnection crd tht provides the interfce between the system bord nd the 4-port Fibre Chnnel dpter. Four-port Fibre Chnnel host bus dpter (HBA) Fibre Chnnel smll form-fctor pluggble (SFP) trnsceiver The SAN Volume Controller A4 is connected to the Fibre Chnnel fbric through the Fibre Chnnel HBA, which is in PCI slot 1. A compct opticl trnsceiver tht provides the opticl interfce to Fibre Chnnel cble. The trnsceiver is cpble of operting t up to 4 Gbps. Chpter 2. Introducing the SAN Volume Controller hrdwre components 61

80 Tble 53. SAN Volume Controller A4 FRU descriptions (continued) FRU Description System bord The system bord for the SAN Volume Controller A4 node. Disk drive bckplne with cbles A SATA simple-swp hrd disk drive bckplne with cbles. Power supply An ssembly tht provides dc power to the SAN Volume Controller A4 node. Fn A single fn. Drive cge A cge for the SATA simple-swp hrd disk drive. Hrd disk drive A SATA (Seril Advnced Technology Attchment) disk drive for the SAN Volume Controller A4. Service controller The unit tht provides the service functions nd the front pnel disply nd buttons. Opertor-informtion pnel The informtion pnel tht includes the power-control button nd LEDs tht indicte system-bord errors, hrd disk drive ctivity, nd power sttus. Opertor-informtion pnel cble A cble tht connects the opertor-informtion pnel to the system bord. Air bffle An pprtus tht redirects or contins ir flow to keep the computer components cool. Microprocessor The microprocessor on the system bord. CMOS bttery A 3.0 V bttery on the system bord tht mintins power to bck up the system BIOS settings. Het-sink ssembly retention module The unit tht is used to instll the het-sink ssembly in the SAN Volume Controller A4 node. Het-sink ssembly An pprtus tht is used to dissipte the het tht is generted by the microprocessor. Input-power cble ssembly The cble ssembly tht provides the power nd signl connections between the SAN Volume Controller A4 nd the 2145 UPS-1U ssembly. SAN Volume Controller G4 FRUs Tble 54 provides brief description of ech SAN Volume Controller G4 FRU. Tble 54. SAN Volume Controller G4 FRU descriptions FRU Description System bord The system bord for the SAN Volume Controller G4 node. 62 SAN Volume Controller: Troubleshooting Guide

81 Tble 54. SAN Volume Controller G4 FRU descriptions (continued) FRU Description Four-port Fibre Chnnel host bus dpter (HBA) Fibre Chnnel smll form-fctor pluggble (SFP) trnsceiver Riser crd, PCI Express Service controller Disk drive Disk drive cge ssembly Disk-drive bckplne Memory module Microprocessor Power supply ssembly Power bckplne CMOS bttery Front pnel signl cble Opertor-informtion pnel Fn ssembly Input-power cble ssembly The SAN Volume Controller G4 is connected to the Fibre Chnnel fbric through the Fibre Chnnel HBA, which is in PCI slot 1. A compct opticl trnsceiver tht provides the opticl interfce to Fibre Chnnel cble. The trnsceiver is cpble of operting t up to 4 Gbps. An interconnection crd tht provides the interfce between the system bord nd the four-port Fibre Chnnel dpter. The FRU tht provides the service functions nd the front pnel disply nd buttons. A SATA (Seril Advnced Technology Attchment) disk drive for the SAN Volume Controller G4. A SATA disk drive cge ssembly for the SAN Volume Controller G4. A SATA disk drive cble ssembly with bckplne. An ECC DRR2 memory module The microprocessor on the system bord. An ssembly tht provides dc power to the SAN Volume Controller G4. An ssembly tht provides power interfce between the system bord nd the power supply ssembly. A 3.0 V bttery on the system bord tht mintins power to bck up the system BIOS settings. A ribbon cble tht connects the opertor-informtion pnel to the system bord. The informtion pnel tht includes the power control button nd the light pth dignostics LEDs. A fn ssembly tht contins two fns, which is used in ll the fn positions. The cble ssembly tht provides the power nd signl connections between the SAN Volume Controller G4 nd the 2145 UPS-1U ssembly. Redundnt c-power switch FRUs The redundnt c-power switch consists of single field replceble unit (FRU). FRU Redundnt c-power switch ssembly Description The redundnt c-power switch nd its input power cbles. Chpter 2. Introducing the SAN Volume Controller hrdwre components 63

82 64 SAN Volume Controller: Troubleshooting Guide

83 Chpter 3. SAN Volume Controller user interfces for servicing your system The SAN Volume Controller provides number of user interfces to troubleshoot, recover, or mintin your system. The interfces provide vrious sets of fcilities to help resolve situtions tht you might encounter. v v v Use the mngement GUI to monitor nd mintin the configurtion of storge tht is ssocited with your clustered systems. Complete service procedures from the service ssistnt. Use the commnd-line interfce (CLI) to mnge your system. The front pnel on the node provides n lterntive service interfce. Note: The front pnel disply is replced by technicin port on some models. Mngement GUI interfce The mngement GUI is browser-bsed GUI for configuring nd mnging ll spects of your system. It provides extensive fcilities to help troubleshoot nd correct problems. About this tsk You use the mngement GUI to mnge nd service your system. The Monitoring > Events pnel provides ccess to problems tht must be fixed nd mintennce procedures tht step you through the process of correcting the problem. The informtion on the Events pnel cn be filtered three wys: Recommended ction (defult) Shows only the lerts tht require ttention nd hve n ssocited fix procedure. Alerts re listed in priority order nd should be fixed sequentilly by using the vilble fix procedures. For ech problem tht is selected, you cn: v Run fix procedure. v View the properties. Unfixed messges nd lerts Displys only the lerts nd messges tht re not fixed. For ech entry tht is selected, you cn: v Run fix procedure on ny lert with n error code. v Mrk n event s fixed. v Filter the entries to show them by specific minutes, hours, or dtes. v Reset the dte filter. v View the properties. Show ll Displys ll event types whether they re fixed or unfixed. For ech entry tht is selected, you cn: v Run fix procedure on ny lert with n error code. v Mrk n event s fixed. v Filter the entries to show them by specific minutes, hours, or dtes. Copyright IBM Corp. 2003,

84 v v Reset the dte filter. View the properties. Some events require certin number of occurrences in 25 hours before they re displyed s unfixed. If they do not rech this threshold in 25 hours, they re flgged s expired. Monitoring events re below the colesce threshold nd re usully trnsient. You cn lso sort events by time or error code. When you sort by error code, the most serious events, those with the lowest numbers, re displyed first. You cn select ny event tht is listed nd select Actions > Properties to view detils bout the event. v Recommended Actions. For ech problem tht is selected, you cn: Run fix procedure. View the properties. v Event log. For ech entry tht is selected, you cn: Run fix procedure. Mrk n event s fixed. Filter the entries to show them by specific minutes, hours, or dtes. Reset the dte filter. View the properties. When to use the mngement GUI The mngement GUI is the primry tool tht is used to service your system. Regulrly monitor the sttus of the system using the mngement GUI. If you suspect problem, use the mngement GUI first to dignose nd resolve the problem. Use the views tht re vilble in the mngement GUI to verify the sttus of the system, the hrdwre devices, the physicl storge, nd the vilble volumes. The Monitoring > Events pnel provides ccess to ll problems tht exist on the system. Use the Recommended Actions filter to disply the most importnt events tht need to be resolved. If there is service error code for the lert, you cn run fix procedure tht ssists you in resolving the problem. These fix procedures nlyze the system nd provide more informtion bout the problem. They suggest ctions to tke nd step you through the ctions tht utomticlly mnge the system where necessry. Finlly, they check tht the problem is resolved. If there is n error tht is reported, lwys use the fix procedures within the mngement GUI to resolve the problem. Alwys use the fix procedures for both system configurtion problems nd hrdwre filures. The fix procedures nlyze the system to ensure tht the required chnges do not cuse volumes to be inccessible to the hosts. The fix procedures utomticlly perform configurtion chnges tht re required to return the system to its optimum stte. Accessing the mngement GUI To view events, you must ccess the mngement GUI. 66 SAN Volume Controller: Troubleshooting Guide

85 About this tsk You must use supported web browser. For list of supported browsers, refer to the Web browser requirements to ccess the mngement GUI topic. You cn use the mngement GUI to mnge your system s soon s you hve creted clustered system. Procedure 1. Strt supported web browser nd point the browser to the mngement IP ddress of your system. The mngement IP ddress is set when the clustered system is creted. Up to four ddresses cn be configured for your use. There re two ddresses for IPv4 ccess nd two ddresses for IPv6 ccess. When the connection is successful, you will see login pnel. 2. Log on by using your user nme nd pssword. 3. When you hve logged on, select Monitoring > Events. 4. Ensure tht the events log is filtered using Recommended ctions. 5. Select the recommended ction nd run the fix procedure. 6. Continue to work through the lerts in the order suggested, if possible. Results After ll the lerts re fixed, check the sttus of your system to ensure tht it is operting s intended. Deleting node from clustered system using the mngement GUI Remove node from system if the node hs filed nd is being replced with new node or if the repir tht hs been performed hs cused tht node to be unrecognizble by the system. Before you begin The cche on the selected node is flushed before the node is tken offline. In some circumstnces, such s when the system is lredy degrded (for exmple, when both nodes in the I/O group re online nd the volumes within the I/O group re degrded), the system ensures tht dt loss does not occur s result of deleting the only node with the cche dt. If filure occurs on the other node in the I/O group, the cche is flushed before the node is removed to prevent dt loss. Before deleting node from the system, record the node seril number, worldwide node nme (WWNN), ll worldwide port nmes (WWPNs), nd the I/O group tht the node is currently prt of. If the node is re-dded to the system t lter time, recording this node informtion cn void dt corruption. Chpter 3. SAN Volume Controller user interfces for servicing your system 67

86 Attention: v If you re removing single node nd the remining node in the I/O group is online, the dt on the remining node goes into write-through mode. This dt cn be exposed to single point of filure if the remining node fils. v If the volumes re lredy degrded before you remove node, redundncy to the volumes is degrded. Removing node might result in loss of ccess to dt nd dt loss. v Removing the lst node in the system destroys the system. Before you remove the lst node in the system, ensure tht you wnt to destroy the system. v When you remove node, you remove ll redundncy from the I/O group. As result, new or existing filures cn cuse I/O errors on the hosts. The following filures cn occur: Host configurtion errors Zoning errors Multipthing-softwre configurtion errors v If you re deleting the lst node in n I/O group nd there re volumes tht re ssigned to the I/O group, you cnnot remove the node from the system if the node is online. You must bck up or migrte ll dt tht you wnt to sve before you remove the node. If the node is offline, you cn remove the node. v When you remove the configurtion node, the configurtion function moves to different node within the system. This process cn tke short time, typiclly less thn minute. The mngement GUI rettches to the new configurtion node trnsprently. v If you turn the power on to the node tht hs been removed nd it is still connected to the sme fbric or zone, it ttempts to rejoin the system. The system tells the node to remove itself from the system nd the node becomes cndidte for ddition to this system or nother system. v If you re dding this node into the system, ensure tht you dd it to the sme I/O group tht it ws previously member of. Filure to do so cn result in dt corruption. This tsk ssumes tht you hve lredy ccessed the mngement GUI. About this tsk 68 SAN Volume Controller: Troubleshooting Guide Complete the following steps to remove node from system: Procedure 1. Select Monitoring > System. 2. Find the node tht you wnt to remove. If the node tht you wnt to remove is shown s Offline, then the node is not prticipting in the system. If the node tht you wnt to remove is shown s Online, deleting the node cn result in the dependent volumes to lso go offline. Verify whether the node hs ny dependent volumes. 3. To check for dependent volumes before ttempting to remove the node, click Mnge, nd then click Show Dependent Volumes. If ny volumes re listed, determine why nd if ccess to the volumes is required while the node is removed from the system. If the volumes re ssigned from MDisk groups tht contin flsh drives tht re locted in the node, check why the volume mirror, if it is configured, is not synchronized. There cn lso be dependent volumes becuse the prtner node in the I/O

87 group is offline. Fbric issues cn lso prevent the volume from communicting with the storge systems. Resolve these problems before continuing with the node removl. 4. Click Remove Node. 5. Click OK to remove the node. Before node is removed, the SAN Volume Controller checks to determine if there re ny volumes tht depend on tht node. If the node tht you selected contins volumes within the following situtions, the volumes go offline nd become unvilble if the node is removed: v The node contins flsh drives nd lso contins the only synchronized copy of mirrored volume v The other node in the I/O group is offline If you select node to remove tht hs these dependencies, nother pnel displys confirming the removl. Adding nodes to clustered system This topic provides instructions for dding node to clustered system. It lso contins informtion bout dding node if the node previously filed nd is being replced with new node or if repir ction hs cused the node to be unrecognizble by the system. When dding nodes ensure tht they re dded in pirs to crete full I/O group. Before you dd node to system, you must mke sure tht the switch zoning is configured such tht the node being dded is in the sme zone s ll other nodes in the system. If you re replcing node nd the switch is zoned by worldwide port nme (WWPN) rther thn by switch port, mke sure tht the switch is configured such tht the node being dded is in the sme VSAN or zone. Considertions when dding node to system If you re dding node tht hs been used previously, either within different I/O group within this system or within different system, consider the following situtions before dding the node. If you dd node to the system without chnging its worldwide node nme (WWNN), hosts might detect the node nd use it s if it were in its old loction. This ction might cuse the hosts to ccess the wrong volumes. v v v v v If the new node requires level of softwre tht is higher thn the softwre level tht is vilble on the system, the entire clustered system must be upgrded before the new node cn be dded. If you re re-dding node bck to the sme I/O group fter service ction required the node to be deleted from the system nd the physicl node hs not chnged, no specil procedures re required nd the node cn be dded bck to the system. If you re replcing node in system either becuse of node filure or n upgrde, you must chnge the WWNN of the new node to mtch tht of the originl node before you connect the node to the Fibre Chnnel network nd dd the node to the system. If you re creting n I/O group in the system nd re dding new node, there re no specil procedures becuse this node ws never dded to system nd the WWNN for the node did not exist. If you re creting n I/O group in the system nd re dding new node tht hs been dded to system before, the host system might still be configured to the node WWPNs nd the node might still be zoned in the fbric. Becuse you Chpter 3. SAN Volume Controller user interfces for servicing your system 69

88 v cnnot chnge the WWNN for the node, you must ensure tht other components in your fbric re configured correctly. Verify tht ny host tht ws previously configured to use the node hs been correctly updted. If the node tht you re dding ws previously replced, either for node repir or upgrde, you might hve used the WWNN of tht node for the replcement node. Ensure tht the WWNN of this node ws updted so tht you do not hve two nodes with the sme WWNN ttched to your fbric. Also ensure tht the WWNN of the node tht you re dding is not If it is 00000, contct your IBM representtive. Considertions when using multipthing device drivers v Applictions on the host systems direct I/O opertions to file systems or logicl volumes tht re mpped by the operting system to virtul pths (vpths), which re pseudo disk objects tht re supported by the multipthing device drivers. Multipthing device drivers mintin n ssocition between vpth nd SAN Volume Controller volume. This ssocition uses n identifier (UID) which is unique to the volume nd is never reused. The UID llows multipthing device drivers to directly ssocite vpths with volumes. v Multipthing device drivers operte within protocol stck tht contins disk nd Fibre Chnnel device drivers tht re used to communicte with the SAN Volume Controller using the SCSI protocol over Fibre Chnnel s defined by the ANSI FCS stndrd. The ddressing scheme tht is provided by these SCSI nd Fibre Chnnel device drivers uses combintion of SCSI logicl unit number (LUN) nd the worldwide node nme (WWNN) for the Fibre Chnnel node nd ports. v If n error occurs, the error recovery procedures (ERPs) operte t vrious tiers in the protocol stck. Some of these ERPs cuse I/O to be redriven using the sme WWNN nd LUN numbers tht were previously used. v Multipthing device drivers do not check the ssocition of the volume with the vpth on every I/O opertion tht it performs. 70 SAN Volume Controller: Troubleshooting Guide

89 Adding nodes to system by using the mngement GUI Attention: 1. If you re dding node to the SAN gin, ensure tht you re dding the node to the sme I/O group from which it ws removed. Filure to do this ction cn result in dt corruption. You must use the informtion tht ws recorded when the node ws originlly dded to the system. If you do not hve ccess to this informtion, cll the IBM Support Center to dd the node bck into the system without corrupting the dt. 2. For ech externl storge system, the LUNs tht re presented to the ports on the new node must be the sme s the LUNs tht re presented to the nodes tht currently exist in the system. You must ensure tht the LUNs re the sme before you dd the new node to the system. 3. For ech externl storge system, LUN msking for ech LUN must be identicl for ll nodes in system. You must ensure tht the LUN msking for ech LUN is identicl before you dd the new node to the system. 4. You must ensure tht the model type of the new node is supported by the SAN Volume Controller softwre level tht is currently instlled on the system. If the model type is not supported by the SAN Volume Controller softwre level, upgrde the system to softwre level tht supports the model type of the new node. See the following website for the ltest supported softwre levels: Ech node in n I/O group must be connected to different uninterruptible power supply. Ech node must lso hve unique nme. If you do not provide nme, the system ssigns defult nme to the object. Note: Whenever possible you must provide meningful nme for objects to mke identifying tht object esier in the future. This tsk ssumes tht you hve lredy ccessed the mngement GUI. To dd node to clustered system, follow these steps: 1. Select Monitoring > System. 2. From the rck imge, click n empty slot tht is ssocited with the I/O group tht you wnt to dd the node. 3. Select the cndidte node tht you wnt to dd. If the node tht you wnt to dd is unvilble in the cndidte list, the node is in service stte. Actions re required to relese the node from service stte before it cn be dded to the system. 4. Select Add node. You re shown wrning. 5. Click OK. 6. If you re dding node into clustered systems for the first time, record the following informtion: v Node seril number v All WWPNs v The I/O group tht the node belongs to Importnt: You need this informtion to void possible dt corruption if you must remove nd dd the node to the system gin. Chpter 3. SAN Volume Controller user interfces for servicing your system 71

90 Service ssistnt interfce If node shows node error 578 or node error 690, the node is in service stte. Perform the following steps from the front pnel to exit service stte: 1. Press nd relese the up or down button until the Actions? option displys. 2. Press the select button. 3. Press nd relese the up or down button until the Exit Service? option displys. 4. Press the select button. 5. Press nd relese the left or right button until the Confirm Exit? option displys. 6. Press the select button. For ny other node errors, follow the pproprite service procedures to fix the errors. After the errors re resolved nd the node is in cndidte stte, you cn try to dd the node to the system gin. The service ssistnt interfce is browser-bsed GUI tht is used to service your nodes. When to use the service ssistnt The primry use of the service ssistnt is when node is in service stte. The node cnnot be ctive s prt of system while it is in service stte. Attention: Complete service ctions on nodes only when directed to do so by the fix procedures. If used inppropritely, the service ctions tht re vilble through the service ssistnt cn cuse loss of ccess to dt or even dt loss. The node might be in service stte becuse it hs hrdwre issue, hs corrupted dt, or hs lost its configurtion dt. Use the service ssistnt in the following situtions: v When you cnnot ccess the system from the mngement GUI nd you cnnot ccess the SAN Volume Controller to run the recommended ctions v When the recommended ction directs you to use the service ssistnt. The mngement GUI opertes only when there is n online clustered system. Use the service ssistnt if you re unble to crete clustered system. The service ssistnt provides detiled sttus nd error summries, nd the bility to modify the World Wide Node Nme (WWNN) for ech node. You cn lso complete the following service-relted ctions: v Collect logs to crete nd downlod pckge of files to send to support personnel. v Remove the dt for the system from node. v Recover system if it fils. v Instll softwre pckge from the support site or rescue the softwre from nother node. v Upgrde softwre on nodes mnully versus completing stndrd upgrde procedure. 72 SAN Volume Controller: Troubleshooting Guide

91 v v v Chnge the service IP ddress tht is ssigned to Ethernet port 1 for the current node. Instll temporry SSH key if key is not instlled nd CLI ccess is required. Restrt the services used by the system. Accessing the service ssistnt The service ssistnt is web ppliction tht helps troubleshoot nd resolve problems on node. About this tsk You must use supported web browser. For list of supported browsers, refer to the topic Web browser requirements to ccess the mngement GUI. The service ssistnt cn be ccessed through service IP ddress. On SAN Volume Controller models 2145-DH8 nd 2145-CG8, you cn connect to service ssistnt by using the technicin port. Procedure To strt the ppliction, complete the following steps. 1. Strt supported web browser nd point your web browser to serviceddress/service for the node tht you wnt to work on. 2. Log on to the service ssistnt using the superuser pssword. If you do not know the current superuser pssword, try to find out. If you cnnot find out wht the pssword is, reset the pssword. Results Commnd-line interfce Complete the service ssistnt ctions on the correct node. Use the commnd-line interfce (CLI) to mnge system with tsk commnds nd informtion commnds. For full description of the commnds nd how to strt n SSH commnd-line session, see the Commnd-line interfce section of the SAN Volume Controller Informtion Center. When to use the CLI The system commnd-line interfce is intended for use by dvnced users who re confident t using CLI. Nerly ll of the flexibility tht is offered by the CLI is vilble through the mngement GUI. However, the CLI does not provide the fix procedures tht re vilble in the mngement GUI. Therefore, use the fix procedures in the mngement GUI to resolve the problems. Use the CLI when you require configurtion setting tht is unvilble in the mngement GUI. You might lso find it useful to crete commnd scripts tht use CLI commnds to monitor certin conditions or to utomte configurtion chnges tht you mke regulrly. Chpter 3. SAN Volume Controller user interfces for servicing your system 73

92 Accessing the system CLI Follow the steps tht re described in the Commnd-line interfce section of the SAN Volume Controller Informtion Center to initilize nd use CLI session. Service commnd-line interfce Use the service commnd-line interfce (CLI) to mnge node using the tsk commnds nd informtion commnds. Note: The service commnd-line interfce cn lso be ccessed by the technicin port. For full description of the commnds nd how to strt n SSH commnd-line session, see the Commnd-line interfce topic in the Reference section of the SAN Volume Controller Informtion Center. When to use the service CLI The service CLI is intended for use by dvnced users who re confident t using commnd-line interfce. To ccess node directly, it is normlly esier to use the service ssistnt with its grphicl interfce nd extensive help fcilities. Accessing the service CLI To initilize nd use CLI session, complete the steps in the Commnd-line interfce topic in the Reference section of this informtion center. Front pnel interfce The front pnel on ech node hs smll disply, nd five control buttons. This pnel provides ccess to system nd node sttus informtion, nd mens to run certin system configurtion nd recovery ctions. For detiled description of using the front pnel, see Chpter 6, Using the front pnel of the SAN Volume Controller, on pge 93. When to use the front pnel Note: The SAN Volume Controller 2145-DH8 does not hve front pnel disply, nvigtion, nd select buttons. On this system, use the technicin port to ccess the service ssistnt interfce. Use the front pnel when you re physiclly next to the system nd re unble to ccess one of the system interfces. Technicin port for node ccess A technicin port is vilble on the rer of the SAN Volume Controller 2145-DH8 node to use direction connection for instlling nd servicing support. The technicin port replces the front pnel disply, nd provides direct ccess to the service ssistnt GUI nd commnd-line interfce (CLI). The technicin port cn be used by directly connecting computer tht hs web browsing softwre vi stndrd 1 Gbps Ethernet cble. 74 SAN Volume Controller: Troubleshooting Guide

93 If the node hs cndidte sttus when you open the web browser, the initiliztion tool is displyed. Otherwise, the service ssistnt interfce is displyed. To use the technicin port, you must be next to the node. The technicin port does not work if it is connected to n Ethernet switch. If you hve Secure Shell (SSH) softwre on the computer, you re directly connected through the technicin port. For exmple, you cn use PuTTY session to ccess the CLI. Chpter 3. SAN Volume Controller user interfces for servicing your system 75

94 76 SAN Volume Controller: Troubleshooting Guide

95 Chpter 4. Performing recovery ctions using the SAN Volume Controller CLI The SAN Volume Controller commnd-line interfce (CLI) is collection of commnds tht you cn use to mnge SAN Volume Controller clusters. See the Commnd-line interfce documenttion for the specific detils bout the commnds provided here. Vlidting nd repiring mirrored volume copies using the CLI You cn use the repirvdiskcopy commnd from the commnd-line interfce (CLI) to vlidte nd repir mirrored volume copies. Attention: Run the repirvdiskcopy commnd only if ll volume copies re synchronized. When you issue the repirvdiskcopy commnd, you must use only one of the -vlidte, -medium, or-resync prmeters. You must lso specify the nme or ID of the volume to be vlidted nd repired s the lst entry on the commnd line. After you issue the commnd, no output is displyed. -vlidte Use this prmeter if you only wnt to verify tht the mirrored volume copies re identicl. If ny difference is found, the commnd stops nd logs n error tht includes the logicl block ddress (LBA) nd the length of the first difference. You cn use this prmeter, strting t different LBA ech time to count the number of differences on volume. -medium Use this prmeter to convert sectors on ll volume copies tht contin different contents into virtul medium errors. Upon completion, the commnd logs n event, which indictes the number of differences tht were found, the number tht were converted into medium errors, nd the number tht were not converted. Use this option if you re unsure wht the correct dt is, nd you do not wnt n incorrect version of the dt to be used. -resync Use this prmeter to overwrite contents from the specified primry volume copy to the other volume copy. The commnd corrects ny differing sectors by copying the sectors from the primry copy to the copies being compred. Upon completion, the commnd process logs n event, which indictes the number of differences tht were corrected. Use this ction if you re sure tht either the primry volume copy dt is correct or tht your host pplictions cn hndle incorrect dt. -strtlb lb Optionlly, use this prmeter to specify the strting Logicl Block Address (LBA) from which to strt the vlidtion nd repir. If you previously used the vlidte prmeter, n error ws logged with the LBA where the first difference, if ny, ws found. Reissue repirvdiskcopy with tht LBA to void reprocessing the initil sectors tht compred identiclly. Continue to reissue repirvdiskcopy using this prmeter to list ll the differences. Copyright IBM Corp. 2003,

96 Issue the following commnd to vlidte nd, if necessry, utomticlly repir mirrored copies of the specified volume: repirvdiskcopy -resync -strtlb 20 vdisk8 Notes: 1. Only one repirvdiskcopy commnd cn run on volume t time. 2. Once you strt the repirvdiskcopy commnd, you cnnot use the commnd to stop processing. 3. The primry copy of mirrored volume cnnot be chnged while the repirvdiskcopy -resync commnd is running. 4. If there is only one mirrored copy, the commnd returns immeditely with n error. 5. If copy being compred goes offline, the commnd is hlted with n error. The commnd is not utomticlly resumed when the copy is brought bck online. 6. In the cse where one copy is redble but the other copy hs medium error, the commnd process utomticlly ttempts to fix the medium error by writing the red dt from the other copy. 7. If no differing sectors re found during repirvdiskcopy processing, n informtionl error is logged t the end of the process. Checking the progress of vlidtion nd repir of volume copies using the CLI Use the lsrepirvdiskcopyprogress commnd to disply the progress of mirrored volume vlidtion nd repirs. You cn specify volume copy using the -copy id prmeter. To disply the volumes tht hve two or more copies with n ctive tsk, specify the commnd with no prmeters; it is not possible to hve only one volume copy with n ctive tsk. To check the progress of vlidtion nd repir of mirrored volumes, issue the following commnd: lsrepirvdiskcopyprogress delim : The following exmple shows how the commnd output is displyed: vdisk_id:vdisk_nme:copy id:tsk:progress:estimted_completion_time 0:vdisk0:0:medium:50: :vdisk0:1:medium:50: Repiring thin-provisioned volume using the CLI You cn use the repirsevdiskcopy commnd from the commnd-line interfce to repir the metdt on thin-provisioned volume. The repirsevdiskcopy commnd utomticlly detects nd repirs corrupted metdt. The commnd holds the volume offline during the repir, but does not prevent the disk from being moved between I/O groups. If repir opertion completes successfully nd the volume ws previously offline becuse of corrupted metdt, the commnd brings the volume bck online. The only limit on the number of concurrent repir opertions is the number of volume copies in the configurtion. 78 SAN Volume Controller: Troubleshooting Guide

97 When you issue the repirsevdiskcopy commnd, you must specify the nme or ID of the volume to be repired s the lst entry on the commnd line. Once strted, repir opertion cnnot be pused or cncelled; the repir cn only be terminted by deleting the copy. Attention: Use this commnd only to repir thin-provisioned volume tht hs reported corrupt metdt. Issue the following commnd to repir the metdt on thin-provisioned volume: repirsevdiskcopy vdisk8 After you issue the commnd, no output is displyed. Notes: 1. Becuse the volume is offline to the host, ny I/O tht is submitted to the volume while it is being repired fils. 2. When the repir opertion completes successfully, the corrupted metdt error is mrked s fixed. 3. If the repir opertion fils, the volume is held offline nd n error is logged. Checking the progress of the repir of thin-provisioned volume using the CLI Issue the lsrepirsevdiskcopyprogress commnd to list the repir progress for thin-provisioned volume copies of the specified volume. If you do not specify volume, the commnd lists the repir progress for ll thin-provisioned copies in the system. Note: Only run this commnd fter you run the repirsevdiskcopy commnd, which you must only run s required by the direct mintennce procedures or the fix procedures recommended by the IBM Support Center. Recovering from offline volumes using the CLI If node or n I/O group fils, you cn use the commnd-line interfce (CLI) to recover offline volumes. About this tsk If you hve lost both nodes in n I/O group nd hve, therefore, lost ccess to ll the volumes tht re ssocited with the I/O group, you must perform one of the following procedures to regin ccess to your volumes. Depending on the filure type, you might hve lost dt tht ws cched for these volumes nd the volumes re now offline. Dt loss scenrio 1 One node in n I/O group hs filed nd filover hs strted on the second node. During the filover process, the second node in the I/O group fils before the dt in the write cche is flushed to the bckend. The first node is successfully repired but its hrdened dt is not the most recent version tht is committed to the dt store; therefore, it cnnot be used. The second node is repired or replced nd hs lost its hrdened dt, therefore, the node hs no wy of recognizing tht it is prt of the clustered system. Chpter 4. Performing recovery ctions using the SAN Volume Controller CLI 79

98 Perform the following steps to recover from n offline volume when one node hs down-level hrdened dt nd the other node hs lost hrdened dt: Procedure 1. Recover the node nd dd it bck into the system. 2. Delete ll IBM FlshCopy mppings nd Metro Mirror or Globl Mirror reltionships tht use the offline volumes. 3. Run the recovervdisk, recovervdiskbyiogrp or recovervdiskbysystem commnd. 4. Re-crete ll FlshCopy mppings nd Metro Mirror or Globl Mirror reltionships tht use the volumes. Exmple Dt loss scenrio 2 Both nodes in the I/O group hve filed nd hve been repired. The nodes hve lost their hrdened dt, therefore, the nodes hve no wy of recognizing tht they re prt of the system. Perform the following steps to recover from n offline volume when both nodes hve lost their hrdened dt nd cnnot be recognized by the system: 1. Delete ll FlshCopy mppings nd Metro Mirror or Globl Mirror reltionships tht use the offline volumes. 2. Run the recovervdisk, recovervdiskbyiogrp or recovervdiskbysystem commnd. 3. Crete ll FlshCopy mppings nd Metro Mirror or Globl Mirror reltionships tht use the volumes. 80 SAN Volume Controller: Troubleshooting Guide

99 Chpter 5. Viewing the vitl product dt Vitl product dt (VPD) is informtion tht uniquely records ech element in the SAN Volume Controller. The dt is updted utomticlly by the system when the configurtion is chnged. The VPD lists the following types of informtion: v System-relted vlues such s the softwre version, spce in storge pools, nd spce llocted to volumes. v Node-relted vlues tht include the specific hrdwre tht is instlled in ech node. Exmples include the FRU prt number for the system bord nd the level of BIOS firmwre tht is instlled. The node VPD is held by the system which mkes it possible to get most of the VPD for the nodes tht re powered off. Using different sets of commnds, you cn view the system VPD nd the node VPD. You cn lso view the VPD through the mngement GUI. Viewing the vitl product dt using the mngement GUI You cn view the vitl product dt for node from the mngement GUI. About this tsk Perform the following steps to view the vitl product dt for node: Procedure 1. From Home, click System Sttus. 2. Select the node for which you wnt to disply the detils. 3. Click VPD to view the dt. Displying the vitl product dt using the CLI You cn use the commnd-line interfce (CLI) to disply the SAN Volume Controller system or node vitl product dt (VPD). Issue the following CLI commnds to disply the VPD: sinfo lsservicesttus lsnodehw lsnodevpd nodenme lssystem system_nme lssystemip lsdrive Note: For the SAN Volume Controller A4 nd G4 nodes, the lsnodevpd nodenme commnd displys the device seril number of the Fibre Chnnel crd s N/A. Displying node properties using the CLI You cn use the commnd-line interfce (CLI) to disply node properties. Copyright IBM Corp. 2003,

100 About this tsk To disply the node properties: Procedure 1. Use the lsnode CLI commnd to disply concise list of nodes in the clustered system (system). Issue this CLI commnd to list the nodes in the system: lsnode -delim : This is n exmple of the output tht is displyed: id:nme:ups_seril_number:wwnn:sttus:io_group_id:io_group_nme:config_node:ups_unique_id:hrdwre:iscsi_nme:iscsi_lis: pnel_nme:enclosure_id:cnister_id:enclosure_seril_number 1:node1:UPS_Fke_SN: B1:online:0:io_grp0:yes: B1:8G4:iqn com.ibm:2145.cluster0.node1:000368::: 2. Issue the lsnode CLI commnd nd specify the node ID or nme of the node tht you wnt to receive detiled output. The following is n exmple of the CLI commnd you cn issue to list detiled output for node in the system: lsnode -delim : group1node1 Where group1node1 is the nme of the node for which you wnt to view detiled output. The following is n exmple of the output tht is displyed: id:1 nme:group1node1 UPS_seril_number:10L3ASH WWNN: C sttus:online IO_group_id:0 IO_group_nme:io_grp0 prtner_node_id:2 prtner_node_nme:group1node2 config_node:yes UPS_unique_id: C0D18D8 port_id: c port_sttus:ctive port_speed:2gb port_id: c port_sttus:ctive port_speed:2gb port_id: c port_sttus:ctive port_speed:2gb port_id: c port_sttus:ctive port_speed:2gb hrdwre:8a4 iscsi_nme:iqn com.ibm:2145.ndihill.node2 iscsi_lis filover_ctive:no filover_nme:node1 filover_iscsi_nme:iqn com.ibm:2145.ndihill.node1 filover_iscsi_lis Displying clustered system properties using the CLI You cn use the commnd-line interfce (CLI) to disply the properties for clustered system (system). 82 SAN Volume Controller: Troubleshooting Guide

101 About this tsk These ctions help you disply your system property informtion. Procedure Issue the lssystem commnd to disply the properties for system. The following is n exmple of the commnd you cn issue: lssystem -delim : build1 where build1 is the nme of the system. Results id: a00a0fe nme:build1 loction:locl prtnership: bndwidth: totl_mdisk_cpcity:90.7gb spce_in_mdisk_grps:90.7gb spce_llocted_to_vdisks:14.99gb totl_free_spce:75.7gb sttistics_sttus:on sttistics_frequency:15 required_memory:0 cluster_locle:en_us time_zone:522 UTC code_level: (build ) FC_port_speed:2Gb console_ip: :443 id_lis: a00a0fe gm_link_tolernce:300 gm_inter_cluster_dely_simultion:0 gm_intr_cluster_dely_simultion:0 emil_reply: emil_contct: emil_contct_primry: emil_contct_lternte: emil_contct_loction: emil_stte:stopped inventory_mil_intervl:0 totl_vdiskcopy_cpcity:15.71gb totl_used_cpcity:13.78gb totl_overlloction:17 totl_vdisk_cpcity:11.72gb cluster_ntp_ip_ddress: cluster_isns_ip_ddress: iscsi_uth_method:none iscsi_chp_secret: uth_service_configured:no uth_service_enbled:no uth_service_url: uth_service_user_nme: uth_service_pwd_set:no uth_service_cert_set:no reltionship_bndwidth_limit:25 gm_mx_host_dely:5 tier:generic_ssd tier_cpcity:0.00mb tier_free_cpcity:0.00mb tier:generic_hdd tier_cpcity:90.67gb tier_free_cpcity:75.34gb emil_contct2: emil_contct2_primry: emil_contct2_lternte: totl_llocted_extent_cpcity:16.12gb Chpter 5. Viewing the vitl product dt 83

102 Fields for the node VPD The node vitl product dt (VPD) provides informtion for items such s the system bord, btteries, processor, fns, memory module, dpter, devices, softwre, front pnel ssembly, the uninterruptible power supply, SAS flsh drive nd SAS host bus dpter (HBA). Tble 55 shows the fields tht you see for the system bord. Tble 55. Fields for the system bord Item System bord Field nme Prt number System seril number Number of processors Number of memory slots Number of fns Number of Fibre Chnnel dpters Number of SCSI, IDE, SATA, or SAS devices Note: The service controller is device. Number of compression ccelertor dpters Number of power supplies Number of high-speed SAS dpters BIOS mnufcturer BIOS version BIOS relese dte System mnufcturer System product Plnr mnufcturer Power supply prt number CMOS bttery prt number Power cble ssembly prt number Service processor firmwre SAS controller prt number Tble 56 on pge 85 shows the fields tht you see for the btteries. 84 SAN Volume Controller: Troubleshooting Guide

103 Tble 56. Fields for the btteries Item Btteries Field nme Bttery_FRU_prt Bttery_prt_identity Bttery_fult_led Bttery_chrging_sttus Bttery_cycle_count Bttery_power_on_hours Bttery_lst_recondition Bttery_midplne_FRU_prt Bttery_midplne_prt_identity Bttery_midplne_FW_version Bttery_power_cble_FRU_prt Bttery_power_sense_cble_FRU_prt Bttery_comms_cble_FRU_prt Bttery_EPOW_cble_FRU_prt Tble 57 shows the fields you see for ech processor tht is instlled. Tble 57. Fields for the processors Item Processor Field nme Prt number Processor loction Mnufcturer Version Speed Sttus Processor seril number Tble 58 shows the fields tht you see for ech fn tht is instlled. Tble 58. Fields for the fns Item Fn Field nme Prt number Loction Tble 59 on pge 86 shows the fields tht re repeted for ech instlled memory module. Chpter 5. Viewing the vitl product dt 85

104 Tble 59. Fields tht re repeted for ech instlled memory module Item Field nme Memory module Prt number Device loction Bnk loction Size (MB) Mnufcturer (if vilble) Seril number (if vilble) Tble 60 shows the fields tht re repeted for ech instlled dpter crd. Tble 60. Fields tht re repeted for ech dpter tht is instlled Item Field nme Adpter Adpter type Prt number Port numbers Loction Device seril number Mnufcturer Device Crd revision Chip revision Tble 61 shows the fields tht re repeted for ech device tht is instlled. Tble 61. Fields tht re repeted for ech SCSI, IDE, SATA, nd SAS device tht is instlled Item Field nme Device Prt number Bus Device Model Revision Seril number Approximte cpcity Hrdwre revision Mnufcturer Tble 62 on pge 87 shows the fields tht re specific to the node softwre. 86 SAN Volume Controller: Troubleshooting Guide

105 Tble 62. Fields tht re specific to the node softwre Item Field nme Softwre Code level Node nme Worldwide node nme ID Unique string tht is used in dump file nmes for this node Tble 63 shows the fields tht re provided for the front pnel ssembly. Tble 63. Fields tht re provided for the front pnel ssembly Item Field nme Front pnel Prt number Front pnel ID Front pnel locle Tble 64 shows the fields tht re provided for the Ethernet port. Tble 64. Fields tht re provided for the Ethernet port Item Field nme Ethernet port Port number Ethernet port sttus MAC ddress Supported speeds Tble 65 shows the fields tht re provided for the power supplies in the node. Tble 65. Fields tht re provided for the power supplies in the node Item Field nme Power supplies Prt number Loction Tble 66 on pge 88 shows the fields tht re provided for the uninterruptible power supply ssembly tht is powering the node. Chpter 5. Viewing the vitl product dt 87

106 Tble 66. Fields tht re provided for the uninterruptible power supply ssembly tht is powering the node Item Field nme Uninterruptible power supply Electronics ssembly prt number Bttery prt number Frme ssembly prt number Input power cble prt number UPS seril number UPS type UPS internl prt number UPS unique ID UPS min firmwre UPS communictions firmwre Tble 67 shows the fields tht re provided for the SAS host bus dpter (HBA). Tble 67. Fields tht re provided for the SAS host bus dpter (HBA) Item Field nme SAS HBA Prt number Port numbers Device seril number Mnufcturer Device Crd revision Chip revision Tble 68 shows the fields tht re provided for the SAS flsh drive. Tble 68. Fields tht re provided for the SAS flsh drive Item Field nme SAS SSD Prt number Mnufcturer Device seril number Model Type UID Firmwre Slot FPGA firmwre Speed Cpcity Expnsion try Connection type 88 SAN Volume Controller: Troubleshooting Guide

107 Tble 69 shows the fields tht re provided for the smll form fctor pluggble (SFP) trnsceiver. Tble 69. Fields tht re provided for the smll form fctor pluggble (SFP) trnsceiver Item Field nme Smll form fctor pluggble (SFP) Prt number trnsceiver Mnufcturer Device Seril number Supported speeds Connector type Trnsmitter type Wvelength Mximum distnce by cble type Hrdwre revision Port number Worldwide port nme Fields for the system VPD The system vitl product dt (VPD) provides vrious informtion bout the system, including its ID, nme, loction, IP ddress, emil contct, code level, nd totl free spce. Tble 70 shows the fields tht re provided for the system properties s shown by the mngement GUI. Tble 70. Fields tht re provided for the system properties Item Field nme Generl ID Note: This is the unique identifier for the system. Nme Loction Time Zone Required Memory Licensed Code Version Chnnel Port Speed Chpter 5. Viewing the vitl product dt 89

108 Tble 70. Fields tht re provided for the system properties (continued) Item Field nme IP Addresses 1 Ethernet Port 1 (ttributes for both IPv4 nd IPv6) v IP Address v Service IP Address v Subnet Msk v Prefix v Defult Gtewy Ethernet Port 2 (ttributes for both IPv4 nd IPv6) v IP Address v Service IP Address v Subnet Msk v Prefix v Defult Gtewy Remote Authentiction Remote Authentiction Web Address User Nme Pssword SSL Certificte Spce Totl MDisk Cpcity Spce in Storge Pools Spce Allocted to Volumes Totl Free Spce Totl Used Cpcity Totl Alloction Totl Volume Copy Cpcity Totl Volume Cpcity Sttistics Sttistics Sttus Sttistics Frequency Metro nd Globl Mirror Link Tolernce Intersystem Dely Simultion Intrsystem Dely Simultion Prtnership Bndwidth Emil SMTP Emil Server Emil Server Port Reply Emil Address Contct Person Nme Primry Contct Phone Number Alternte Contct Phone Number Physicl Loction of the System Reporting Error Emil Sttus Inventory Emil Intervl 90 SAN Volume Controller: Troubleshooting Guide

109 Tble 70. Fields tht re provided for the system properties (continued) Item Field nme iscsi isns Server Address Supported Authentiction Methods CHAP Secret 1 You cn lso use the lssystemip CLI commnd to view this dt. Chpter 5. Viewing the vitl product dt 91

110 92 SAN Volume Controller: Troubleshooting Guide

111 Chpter 6. Using the front pnel of the SAN Volume Controller The front pnel of the SAN Volume Controller hs disply, vrious LEDs, nvigtion buttons, nd select button tht re used when you service your SAN Volume Controller node. Figure 45 shows where the front-pnel disply 1 is on the SAN Volume Controller node. 1 Restrting Restrting svc00552 Figure 45. SAN Volume Controller front-pnel ssembly The SAN Volume Controller 2145-DH8 does not hve front pnel disply, but does include node sttus, node fult, nd bttery sttus LEDs. Insted, technicin port is vilble on the rer of the 2145-DH8 node to use direction connection for instlling nd servicing support. The technicin port replces the front pnel disply, nd provides direct ccess to the service ssistnt GUI nd commnd-line interfce (CLI). Boot progress indictor Boot progress is displyed on the front pnel of the SAN Volume Controller. The Boot progress disply on the front pnel shows tht the node is strting. Booting 130 Figure 46. Exmple of boot progress disply During the boot opertion, boot progress codes re displyed nd the progress br moves to the right while the boot opertion proceeds. Boot filed If the boot opertion fils, boot code 120 is displyed. Copyright IBM Corp. 2003,

112 Filed 120 See the "Error code reference" topic where you cn find description of the filure nd the pproprite steps tht you must perform to correct the filure. Chrging The front pnel indictes tht the uninterruptible power supply bttery is chrging. Chrging A node will not strt nd join system if there is insufficient power in the uninterruptible power supply bttery to mnge with power filure. Chrging is displyed until it is sfe to strt the node. This might tke up to two hours. Error codes Error codes re displyed on the front pnel disply. Figure 47 nd Figure 48 show how error codes re displyed on the front pnel. svc00433 svc00304 Figure 47. Exmple of n error code for clustered system svc00434 Hrdwre boot Figure 48. Exmple of node error code For descriptions of the error codes tht re displyed on the front pnel disply, see the vrious error code topics for full description of the filure nd the ctions tht you must perform to correct the filure. The hrdwre boot disply shows system dt when power is first pplied to the node s the node serches for disk drive to boot. 94 SAN Volume Controller: Troubleshooting Guide

113 Node rescue request If this disply remins ctive for longer thn 3 minutes, there might be problem. The cuse might be hrdwre filure or the softwre on the hrd disk drive might be missing or dmged. If softwre is lost, you cn use the node rescue process to copy ll softwre from nother node. The node-rescue-request disply, which is shown in Figure 49, indictes tht request hs been mde to replce the softwre on this node. The SAN Volume Controller softwre is preinstlled on ll SAN Volume Controller nodes. This softwre includes the operting system, the ppliction softwre, nd the SAN Volume Controller publictions. It is normlly not necessry to replce the softwre on node, but if the softwre is lost for some reson (for exmple, the hrd disk drive in the node fils), it is possible to copy ll the softwre from nother node tht is connected to the sme Fibre Chnnel fbric. This process is known s node rescue. Figure 49. Node rescue disply Power filure The SAN Volume Controller node uses bttery power from the uninterruptible power supply to shut itself down. The Power filure disply on the lptop ppliction shows tht the SAN Volume Controller is running on bttery power becuse min power hs been lost. All I/O opertions hve stopped. The node is sving system metdt nd node cche dt to the internl disk drive. When the progress br reches zero, the node powers off. Note: When input power is restored to the uninterruptible power supply, the SAN Volume Controller turns on. Powering off The progress br on the disply shows the progress of the power-off opertion. Powering Off is displyed fter the power button hs been pressed nd while the node is powering off. Powering off might tke severl minutes. The progress br moves to the left when the power is removed. Chpter 6. Using the front pnel of the SAN Volume Controller 95

114 Recovering The front pnel indictes tht the uninterruptible power supply bttery is not fully chrged. Recovering svc00305 When node is ctive in system but the uninterruptible power supply bttery is not fully chrged, Recovering is displyed. If the power fils while this messge is displyed, the node does not restrt until the uninterruptible power supply hs chrged to level where it cn sustin second power filure. Restrting The front pnel indictes when the softwre on node is restrting. Restrting The softwre is restrting for one of the following resons: v An internl error ws detected. v The power button ws pressed gin while the node ws powering off. If you press the power button while powering off, the pnel disply chnges to indicte tht the button press ws detected; however, the power off continues until the node finishes sving its dt. After the dt is sved, the node powers off nd then utomticlly restrts. The progress br moves to the right while the node is restrting. Shutting down The front-pnel indictor trcks shutdown opertions. The Shutting Down disply is shown when you issue shutdown commnd to SAN Volume Controller clustered system or SAN Volume Controller node. The progress br continues to move to the left until the node turns off. When the shutdown opertion is complete, the node turns off. When you power off node tht is connected to 2145 UPS-1U, only the node shuts down; the 2145 UPS-1U does not shut down. Shutting Down 96 SAN Volume Controller: Troubleshooting Guide

115 Vlidte WWNN? option The front pnel prompts you to vlidte the WWNN when the worldwide node nme (WWNN) tht is stored in the service controller (the pnel WWNN) does not mtch the WWNN tht is bcked up on the SAN Volume Controller disk (the disk WWNN). Typiclly, this pnel is displyed when the service controller hs been replced. The SAN Volume Controller uses the WWNN tht is stored on the service controller. Usully, when the service controller is replced, you modify the WWNN tht is stored on it to mtch the WWNN on the service controller tht it replced. By doing this, the node mintins its WWNN ddress, nd you do not need to modify the SAN zoning or host configurtions. The WWNN tht is stored on disk is the sme tht ws stored on the old service controller. After it is in this mode, the front pnel disply will not revert to its norml displys, such s node or cluster (system) options or opertionl sttus, until the WWNN is vlidted. Nvigte the Vlidte WWNN option (shown in Figure 50) to choose which WWNN tht you wnt to use. Vlidte WWNN? Select Disk WWNN: Pnel WWNN: Use Disk WWNN? Use Pnel WWNN? Select Node WWNN: svc00409 Figure 50. Vlidte WWNN? nvigtion Choose which stored WWNN you wnt this node to use: 1. From the Vlidte WWNN? pnel, press nd relese the select button. The Disk WWNN: pnel is displyed nd shows the lst five digits of the WWNN tht is stored on the disk. 2. To view the WWNN tht is stored on the service controller, press nd relese the right button. The Pnel WWNN: pnel is displyed nd shows the lst five numbers of the WWNN tht is stored on the service controller. 3. Determine which WWNN tht you wnt to use.. To use the WWNN tht is stored on the disk: 1) From the Disk WWNN: pnel, press nd relese the down button. The Use Disk WWNN? pnel is displyed. 2) Press nd relese the select button. b. To use the WWNN tht is stored on the service controller: 1) From the Pnel WWNN: pnel, press nd relese the down button. The Use Pnel WWNN? pnel is displyed. 2) Press nd relese the select button. The node is now using the selected WWNN. The Node WWNN: pnel is displyed nd shows the lst five numbers of the WWNN tht you selected. Chpter 6. Using the front pnel of the SAN Volume Controller 97

116 If neither WWNN tht is stored on the service controller pnel nor disk is suitble, you must wit until the node restrts before you cn chnge it. After the node restrts, select Chnge WWNN to chnge the WWNN to the vlue tht you wnt. SAN Volume Controller menu options During norml opertions, menu options re vilble on the front pnel disply of the SAN Volume Controller node. Menu options enble you to review the opertionl sttus of the clustered system, node, nd externl interfces. They lso provide ccess to the tools nd opertions tht you use to service the node. Figure 51 on pge 99 shows the sequence of the menu options. Only one option t time is displyed on the front pnel disply. For some options, dditionl dt is displyed on line 2. The first option tht is displyed is the Cluster: option. 98 SAN Volume Controller: Troubleshooting Guide

117 Min Options Cluster: FC Port-1 U / D Port-1 L/ R Sttus: L/ R Address: s L/ R IPv4 Address-2: IPv4 Address-1: S S L/ R IPv4 L/ R Subnet-2: IPv4 L/ R L/ R Subnet-1: Secondry Options R / L IPv4 Gtewy-2: IPv4 Gtewy-1: Port-2 Address: s R / L L/ R L/ R R / L s IPv6 Address-2: IPv6 Address-1: Select shows IPv4 nd IPv6 ddresses if vilble L/ R L/ R IPv6 L/ R Prefix-2: IPv6 Prefix-1: L/ R IPv6 Gtewy-2: IPv6 Gtewy-1: R / L Node Node L/ R Sttus: L/ R L/ R WWNN: Service Address s U / D S R / L IPv4 Address IPv4 L/ R L/ R Subnet IPv4 Gtewy L/ R IPv6 Address L/ R IPv6 Prefix L/ R IPv6 Gtewy Version: L/ R Build: L/ R Cluster Build: U / D R / L R / L Ethernet L/ R Ethernet L/ R Speed-1: L/ R Port-1: MAC Address-1: L/ R Ethernet Port-2: L/ R Speed-2: L/ R MAC Address-2: U / D Ethernet Port-3: L/ R L/ R Speed-3: L/ R MAC Address-3: L/ R Ethernet Port-4: L/ R Speed-4: L/ R MAC Address-4: R / L FC Port-1 Sttus L/ R FC Port-1 L/ R FC Port-2 L/ R Speed Sttus FC Port-2 Speed U / D R / L FC Port-3 Sttus FC Port-3 L/ R L/ R FC Port-4 L/ R Speed Sttus FC Port-4 Speed Actions x U / D Lnguge? L/ R English? L/ R L Jpnese? L x Select tkes you to the Actions menu L Select ctivtes lnguge svc00560 Figure 51. SAN Volume Controller options on the front-pnel disply Use the left nd right buttons to nvigte through the secondry fields tht re ssocited with some of the min fields. Note: Messges might not disply fully on the screen. You might see right ngle brcket (>) on the right side of the disply screen. If you see right ngle brcket, press the right button to scroll through the disply. When there is no more text to disply, you cn move to the next item in the menu by pressing the right button. Chpter 6. Using the front pnel of the SAN Volume Controller 99

118 Similrly, you might see left ngle brcket (<) on the left side of the disply screen. If you see left ngle brcket, press the left button to scroll through the disply. When there is no more text to disply, you cn move to the previous item in the menu by pressing the left button. The following min options re vilble: v Cluster v Node v Version v Ethernet v FC Port 1 Sttus v Actions v Lnguge Cluster (system) options The min cluster (system) option from the menu cn disply the cluster nme or the field cn be blnk. The min cluster (system) option displys the system nme tht the user hs ssigned. If clustered system is in the process of being creted on the node, nd no system nme hs been ssigned, temporry nme tht is bsed on the IP ddress of the system is displyed. If this node is not ssigned to system, the field is blnk. Sttus option Sttus is indicted on the front pnel. This field is blnk if the node is not member of clustered system. If this node is member of clustered system, the field indictes the opertionl sttus of the system, s follows: Active Indictes tht this node is n ctive member of the system. Inctive Indictes tht the node is member of system, but is not now opertionl. It is not opertionl becuse the other nodes tht re in the system cnnot be ccessed or becuse this node ws excluded from the system. Degrded Indictes tht the system is opertionl, but one or more of the member nodes re missing or hve filed. IPv4 Address option A clustered system must hve either n IPv4 ddress or n IPv6 ddress, or both, ssigned to Ethernet port 1. You cn lso ssign n IPv4 ddress or n IPv6 ddress, or both, to Ethernet port 2. You cn use ny of the ddresses to ccess the system from the commnd-line tools or the mngement GUI. These fields contin the IPv4 ddresses of the system. If this node is not member of system or if the IPv4 ddress hs not been ssigned, these fields re blnk. 100 SAN Volume Controller: Troubleshooting Guide

119 IPv4 Subnet options: The IPv4 subnet msk ddresses re set when the IPv4 ddresses re ssigned to the system. The IPv4 subnet options disply the subnet msk ddresses when the system hs IPv4 ddresses. If the node is not member of system or if the IPv4 ddresses hve not been ssigned, this field is blnk. IPv4 Gtewy options: The IPv4 gtewy ddresses re set when the system is creted. The IPv4 gtewy options disply the gtewy ddresses for the system. If the node is not member of system, or if the IPv4 ddresses hve not been ssigned, this field is blnk. IPv6 Address options A clustered system must hve either n IPv4 ddress or n IPv6 ddress, or both, ssigned to Ethernet port 1. You cn lso ssign n IPv4 ddress or n IPv6 ddress, or both, to Ethernet port 2. You cn use ny of the ddresses to ccess the system from the commnd-line tools or the mngement GUI. These fields contin the IPv6 ddresses of the system. If the node is not member of system, or if the IPv6 ddress hs not been ssigned, these fields re blnk. IPv6 Prefix option: The IPv6 prefix is set when system is creted. The IPv6 prefix option displys the network prefix of the system nd the service IPv6 ddresses. The prefix hs vlue of If the node is not member of system, or if the IPv6 ddresses hve not been ssigned, blnk line displys. IPv6 Gtewy option: The IPv6 gtewy ddresses re set when the system is creted. This option displys the IPv6 gtewy ddresses for the system. If the node is not member of system, or if the IPv6 ddresses hve not been ssigned, blnk line displys. Displying n IPv6 ddress After you hve set the IPv6 ddress, you cn disply the IPv6 ddresses nd the IPv6 gtewy ddresses. The IPv6 ddresses nd the IPv6 gtewy ddresses consist of eight (4-digit) hexdeciml vlues tht re shown cross four pnels, s shown in Figure 52 on pge 102. Ech pnel displys two 4-digit vlues tht re seprted by colon, the ddress field position (such s 2/4) within the totl ddress, nd scroll indictors. Move between the ddress pnels by using the left button or right button. Chpter 6. Using the front pnel of the SAN Volume Controller 101

120 svc00417 Figure 52. Viewing the IPv6 ddress on the front-pnel disply Node options The min node option displys the identifiction number or the nme of the node if the user hs ssigned nme. Sttus option The node sttus is indicted on the front pnel. The sttus cn be one of the following sttes: Active The node is opertionl, ssigned to system, nd redy to perform I/O. Service There is n error tht is preventing the node from operting s prt of system. It is sfe to shut down the node in this stte. Cndidte The node is not ssigned to system nd is not in service. It is sfe to shut down the node in this stte. Strting The node is prt of system nd is ttempting to join the system. It cnnot perform I/O. Node WWNN option The Node WWNN (worldwide node nme) option displys the lst five hexdeciml digits of the WWNN tht is being used by the node. Only the lst five digits of WWNN vry on node. The first 11 digits re lwys Service Address option Pressing select on the Service Address pnel displys the IP ddress tht is configured for ccess to the service ssistnt nd the service CLI. Version options The version option displys the version of the SAN Volume Controller softwre tht is ctive on the node. The version consists of four fields tht re seprted by full stops. The fields re the version, relese, modifiction, nd fix level; for exmple, Build option The Build: pnel displys the level of the SAN Volume Controller softwre tht is currently ctive on this node. Cluster Build option The Cluster Build: pnel displys the level of the softwre tht is currently ctive on the system tht this node is operting in. Ethernet options The Ethernet options disply the opertionl stte of the Ethernet ports, the speed nd duplex informtion, nd their medi ccess control (MAC) ddresses. 102 SAN Volume Controller: Troubleshooting Guide

121 The Ethernet pnel shows one of the following sttes: Config - Yes This node is the configurtion node. Config - No This node is not the configurtion node. No Cluster This node is not member of system. Press the right button to view the detils of the individul Ethernet ports. Ethernet Port options The Ethernet port options Port-1 through Port-4 disply the stte of the links nd indictes whether or not there is n ctive link with n Ethernet network. Link Online An Ethernet cble is ttched to this port. Link Offline No Ethernet cble is ttched to this port or the link hs filed. Speed options The speed options Speed-1 through Speed-4 disply the speed nd duplex informtion for the Ethernet port. The speed informtion cn be one of the following vlues: 10 The speed is 10 Mbps. 100 The speed is 100 Mbps. 1 The speed is 1Gbps. 10 The speed is 10 Gbps. The duplex informtion cn be one of the following vlues: Full Dt cn be sent nd received t the sme time. Hlf Dt cn be sent nd received in one direction t time. MAC Address options The MAC ddress options MAC Address-1 through MAC Address-4 disply the medi ccess control (MAC) ddress of the Ethernet port. Fibre Chnnel port options The Fibre Chnnel port options disply the opertionl sttus of the Fibre Chnnel ports. Active The port is opertionl nd cn ccess the Fibre Chnnel fbric. Inctive The port is opertionl but cnnot ccess the Fibre Chnnel fbric. One of the following conditions cused this result: v The Fibre Chnnel cble hs filed. v The Fibre Chnnel cble is not instlled. v The device tht is t the other end of the cble hs filed. Chpter 6. Using the front pnel of the SAN Volume Controller 103

122 Filed The port is not opertionl becuse of hrdwre filure. Not instlled This port is not instlled. Actions options During norml opertions, ction menu options re vilble on the front pnel disply of the node. Only use the front pnel ctions when directed to do so by service procedure. Inpproprite use cn led to loss of ccess to dt or loss of dt. Figure 53 on pge 106, Figure 54 on pge 107, nd Figure 55 on pge 108 show the sequence of the ctions options. In the figures, bold lines indicte tht the select button ws pressed. The lighter lines indicte the nvigtionl pth (up or down nd left or right). The circled X indictes tht if the select button is pressed, n ction occurs using the dt entered. Only one ction menu option t time is displyed on the front-pnel disply. Note: Options only disply in the menu if they re vlid for the current stte of the node. See Tble 71 for list of when the options re vlid. The following options re vilble from the Actions menu: Tble 71. When options re vilble Front pnel option Cluster IPv4 Cluster IPv6 Option nme Crete clustered system with n IPv4 mngement ddress Crete clustered system with n IPv6 mngement ddress When option is vilble for the current stte of the node Cndidte stte Cndidte stte Service IPv4 Set the IPv4 service ddress All sttes of the node Service IPv6 Set the IPv6 service ddress All sttes of the node Service DHCPv4 Set DHCP IPv4 service All sttes ddress Service DHCPv6 Set DHCP IPv6 service All sttes ddress Chnge WWNN Chnge the WWNN of the Cndidte or service stte node Enter Service Enter service stte Whenever error 690 is not showing. Exit Service Leve service stte if possible Whenever error 690 is showing. Recover Cluster Recover system configurtion Cndidte or service stte Remove Cluster Remove system stte Whenever the node hs clustered system stte. Pced Upgrde Perform user-pced CCU Node in service without clustered system stte 104 SAN Volume Controller: Troubleshooting Guide

123 Tble 71. When options re vilble (continued) Front pnel option Set FC Speed Option nme Set Fibre Chnnel speed When option is vilble for the current stte of the node Reset Pssword Reset pssword Not ctive or if the resetpssword commnd is enbled Rescue Node Rescue node softwre All sttes Chpter 6. Using the front pnel of the SAN Volume Controller 105

124 Cluster IPv4? IPv4 Address: IPv4 Subnet: IPv4 Gtewy: Confirm Crete? x Cncel? Cluster IPv6? IPv6 Address: IPv6 Prefix: IPv6 Gtewy: Confirm Crete? x Cncel? Service IPv4? IPv4 Address: IPv4 Subnet: IPv4 Gtewy: Confirm Address? x Cncel? Service IPv6? IPv6 Address: IPv6 Prefix: IPv6 Gtewy: Confirm Address? x Cncel? Service DHCPv4? Confirm DHCPv4? x Cncel? Service DHCPv6? Confirm DHCPv6? x Cncel? Chnge WWNN? Edit WWNN? Confirm WWNN? x Cncel? svc00657 Figure 53. Upper options of the ctions menu on the front pnel 106 SAN Volume Controller: Troubleshooting Guide

125 Enter Service? Confirm Enter? x Cncel? Exit Service? Confirm Exit? x Cncel? Recover Cluster? Confirm Recover? x Cncel? Remove Cluster? Confirm Remove? x Cncel? Pced Upgrde? Confirm Upgrde? x Cncel? svc00658 Figure 54. Middle options of the ctions menu on the front pnel Chpter 6. Using the front pnel of the SAN Volume Controller 107

126 Set FC Speed? Edit Speed? Confirm Speed? x Cncel? Reset Pssword? Confirm Reset? x Cncel? Rescue Node? Confirm Rescue? x Cncel? Exit Actions? svc00659 Figure 55. Lower options of the ctions menu on the front pnel To perform n ction, nvigte to the Actions option nd press the select button. The ction is initited. Avilble prmeters for the ction re displyed. Use the left or right buttons to move between the prmeters. The current setting is displyed on the second disply line. To set or chnge prmeter vlue, press the select button when the prmeter is displyed. The vlue chnges to edit mode. Use the left or right buttons to move between subfields, nd use the up or down buttons to chnge the vlue of subfield. When the vlue is correct, press select to leve edit mode. Ech ction lso hs Confirm? nd Cncel? pnel. Pressing select on the Confirm? pnel initites the ction using the current prmeter vlue setting. Pressing select on the Cncel? pnel returns to the Action option pnel without chnging the node. Note: Messges might not disply fully on the screen. You might see right ngle brcket (>) on the right side of the disply screen. If you see right ngle brcket, press the right button to scroll through the disply. When there is no more text to disply, you cn move to the next item in the menu by pressing the right button. Similrly, you might see left ngle brcket (<) on the left side of the disply screen. If you see left ngle brcket, press the left button to scroll through the disply. When there is no more text to disply, you cn move to the previous item in the menu by pressing the left button. 108 SAN Volume Controller: Troubleshooting Guide

127 Cluster IPv4 or Cluster IPv6 options You cn crete clustered system from the Cluster IPv4 or Cluster IPv6 ction options. The Cluster IPv4 or Cluster IPv6 option llows you to crete clustered system. From the front pnel, when you crete clustered system, you cn set either the IPv4 or the IPv6 ddress for Ethernet port 1. If required, you cn dd more mngement IP ddresses by using the mngement GUI or the CLI. Press the up nd down buttons to nvigte through the prmeters tht re ssocited with the Cluster option. When you hve nvigted to the desired prmeter, press the select button. The prmeters tht re vilble include: v IPv4 Address v IPv4 Subnet v IPv4 Gtewy v IPv4 Confirm Crete? v IPv6 Address v IPv6 Subnet v IPv6 Gtewy v IPv6 Confirm Crete? If you re creting the clustered system with n IPv4 ddress, complete the following steps: 1. Press nd relese the up or down button until Actions? is displyed. Press nd relese the select button. 2. Press nd relese the up or down button until Cluster IPv4? is displyed. Press nd relese the select button. 3. Edit the IPv4 ddress, the IPv4 subnet, nd the IPv4 gtewy. 4. Press nd relese the left or right button until IPv4 Confirm Crete? is displyed. 5. Press nd relese the select button to confirm. If you re creting the clustered system with n IPv6 ddress, complete the following steps: 1. Press nd relese the up or down button until Actions? is displyed. Press nd relese the select button. 2. Press nd relese the left or right button until Cluster Ipv6? is displyed. Press nd relese the select button. 3. Edit the IPv6 ddress, the IPv6 prefix, nd the IPv6 gtewy. 4. Press nd relese the left or right button until IPv6 Confirm Crete? is displyed. 5. Press nd relese the select button to confirm. IPv4 Address option Using the IPv4 ddress, you cn set the IP ddress for Ethernet port 1 of the clustered system tht you re going to crete. The system cn hve either n IPv4 or n IPv6 ddress, or both t the sme time. You cn set either the IPv4 or IPv6 Chpter 6. Using the front pnel of the SAN Volume Controller 109

128 mngement ddress for Ethernet port 1 from the front pnel when you re creting the system. If required, you cn dd more mngement IP ddresses from the CLI. Attention: When you set the IPv4 ddress, ensure tht you type the correct ddress. Otherwise, you might not be ble to ccess the system using the commnd-line tools or the mngement GUI. Perform the following steps to set the IPv4 ddress: 1. Nvigte to the IPv4 Address pnel. 2. Press the select button. The first IP ddress number is highlighted. 3. Press the up button if you wnt to increse the vlue tht is highlighted; press the down button if you wnt to decrese tht vlue. If you wnt to quickly increse the highlighted vlue, hold the up button. If you wnt to quickly decrese the highlighted vlue, hold the down button. Note: If you wnt to disble the fst increse or decrese function, press nd hold the down button, press nd relese the select button, nd then relese the down button. The disbling of the fst increse or decrese function lsts until the cretion is completed or until the feture is gin enbled. If the up button or down button is pressed nd held while the function is disbled, the vlue increses or decreses once every two seconds. To gin enble the fst increse or decrese function, press nd hold the up button, press nd relese the select button, nd then relese the up button. 4. Press the right button or left button to move to the number field tht you wnt to set. 5. Repet steps 3 nd 4 for ech number field tht you wnt to set. 6. Press the select button to confirm the settings. Otherwise, press the right button to disply the next secondry option or press the left button to disply the previous options. Press the right button to disply the next secondry option or press the left button to disply the previous options. IPv4 Subnet option Using this option, you cn set the IPv4 subnet msk for Ethernet port 1. Attention: When you set the IPv4 subnet msk ddress, ensure tht you type the correct ddress. Otherwise, you might not be ble to ccess the system using the commnd-line tools or the mngement GUI. Perform the following steps to set the subnet msk: 1. Nvigte to the IPv4 Subnet pnel. 2. Press the select button. The first subnet msk number is highlighted. 3. Press the up button if you wnt to increse the vlue tht is highlighted; press the down button if you wnt to decrese tht vlue. If you wnt to quickly increse the highlighted vlue, hold the up button. If you wnt to quickly decrese the highlighted vlue, hold the down button. 110 SAN Volume Controller: Troubleshooting Guide Note: If you wnt to disble the fst increse or decrese function, press nd hold the down button, press nd relese the select button, nd then relese the down button. The disbling of the fst increse or decrese function lsts until the cretion is completed or until the feture is gin enbled. If the up button

129 or down button is pressed nd held while the function is disbled, the vlue increses or decreses once every two seconds. To gin enble the fst increse or decrese function, press nd hold the up button, press nd relese the select button, nd then relese the up button. 4. Press the right button or left button to move to the number field tht you wnt to set. 5. Repet steps 3 nd 4 for ech number field tht you wnt to set. 6. Press the select button to confirm the settings. Otherwise, press the right button to disply the next secondry option or press the left button to disply the previous options. IPv4 Gtewy option Using this option, you cn set the IPv4 gtewy ddress for Ethernet port 1. Attention: When you set the IPv4 gtewy ddress, ensure tht you type the correct ddress. Otherwise, you might not be ble to ccess the system using the commnd-line tools or the mngement GUI. Perform the following steps to set the IPv4 gtewy ddress: 1. Nvigte to the IPv4 Gtewy pnel. 2. Press the select button. The first gtewy ddress number field is highlighted. 3. Press the up button if you wnt to increse the vlue tht is highlighted; press the down button if you wnt to decrese tht vlue. If you wnt to quickly increse the highlighted vlue, hold the up button. If you wnt to quickly decrese the highlighted vlue, hold the down button. Note: If you wnt to disble the fst increse or decrese function, press nd hold the down button, press nd relese the select button, nd then relese the down button. The disbling of the fst increse or decrese function lsts until the cretion is completed or until the feture is gin enbled. If the up button or down button is pressed nd held while the function is disbled, the vlue increses or decreses once every two seconds. To gin enble the fst increse or decrese function, press nd hold the up button, press nd relese the select button, nd then relese the up button. 4. Press the right button or left button to move to the number field tht you wnt to set. 5. Repet steps 3 nd 4 for ech number field tht you wnt to set. 6. Press the select button to confirm the settings. Otherwise, press the right button to disply the next secondry option or press the left button to disply the previous options. IPv4 Confirm Crete? option Using this option, you cn strt n opertion to crete clustered system with n IPv4 ddress. 1. Press nd relese the left or right button until IPv4 Confirm Crete? is displyed. 2. Press the select button to strt the opertion. If the crete opertion is successful, Pssword is displyed on line 1. The pssword tht you cn use to ccess the system is displyed on line 2. Be sure to immeditely record the pssword; it is required on the first ttempt to mnge the system from the mngement GUI. Chpter 6. Using the front pnel of the SAN Volume Controller 111

130 Attention: The pssword displys for only 60 seconds, or until front pnel button is pressed. The clustered system is creted only fter the pssword disply is clered. If the crete opertion fils, Crete Filed: is displyed on line 1 of the front-pnel disply screen. Line 2 displys one of two possible error codes tht you cn use to isolte the cuse of the filure. IPv6 Address option Using this option, you cn set the IPv6 ddress for Ethernet port 1 of the system tht you re going to crete. The system cn hve either n IPv4 or n IPv6 ddress, or both t the sme time. You cn set either the IPv4 or IPv6 mngement ddress for Ethernet port 1 from the front pnel when you re creting the system. If required, you cn dd more mngement IP ddresses from the CLI. Attention: When you set the IPv6 ddress, ensure tht you type the correct ddress. Otherwise, you might not be ble to ccess the system using the commnd-line tools or the mngement GUI. To set the IPv6 ddress: 1. From the Crete Cluster? option, press the select button, nd then press the down button. The IPv6 Address option is displyed. 2. Press the select button gin. The first IPv6 ddress number is highlighted.. 3. Move between the ddress pnels by using the left button or right button. The IPv6 ddresses nd the IPv6 gtewy ddresses consist of eight (4-digit) hexdeciml vlues tht re shown cross four pnels 4. You cn chnge ech number in the ddress independently. Press the up button if you wnt to increse the vlue tht is highlighted; press the down button if you wnt to decrese tht vlue. 5. Press the right button or left button to move to the number field tht you wnt to set. 6. Repet steps 3 nd 4 for ech number field tht you wnt to set. 7. Press the select button to confirm the settings. Otherwise, press the right button to disply the next secondry option or press the left button to disply the previous options. IPv6 Prefix option Using this option, you cn set the IPv6 prefix for Ethernet port 1. Attention: When you set the IPv6 prefix, ensure tht you type the correct network prefix.otherwise, you might not be ble to ccess the system using the commnd-line tools or the mngement GUI. Perform the following steps to set the IPv6 prefix: Note: This option is restricted to vlue Press nd relese the left or right button until IPv6 Prefix is displyed. 2. Press the select button. The first prefix number field is highlighted. 3. Press the up button if you wnt to increse the vlue tht is highlighted; press the down button if you wnt to decrese tht vlue. If you wnt to quickly increse the highlighted vlue, hold the up button. If you wnt to quickly decrese the highlighted vlue, hold the down button. 112 SAN Volume Controller: Troubleshooting Guide

131 Note: If you wnt to disble the fst increse or decrese function, press nd hold the down button, press nd relese the select button, nd then relese the down button. The disbling of the fst increse or decrese function lsts until the cretion is completed or until the feture is gin enbled. If the up button or down button is pressed nd held while the function is disbled, the vlue increses or decreses once every two seconds. To gin enble the fst increse or decrese function, press nd hold the up button, press nd relese the select button, nd then relese the up button. 4. Press the select button to confirm the settings. Otherwise, press the right button to disply the next secondry option or press the left button to disply the previous options. IPv6 Gtewy option Using this option, you cn set the IPv6 gtewy for Ethernet port 1. Attention: When you set the IPv6 gtewy ddress, ensure tht you type the correct ddress. Otherwise, you might not be ble to ccess the system using the commnd-line tools or the mngement GUI. Perform the following steps to set the IPv6 gtewy ddress: 1. Press nd relese the left or right button until IPv6 Gtewy is displyed. 2. Press the select button. The first gtewy ddress number is highlighted. The IPv6 ddresses nd the IPv6 gtewy ddresses consist of eight (4-digit) hexdeciml vlues tht re shown cross four pnels. 3. You cn chnge ech number in the ddress independently. Press the up button if you wnt to increse the vlue tht is highlighted; press the down button if you wnt to decrese tht vlue. 4. Press the right button or left button to move to the number field tht you wnt to set. 5. Repet steps 3 nd 4 for ech number field tht you wnt to set. 6. Press the select button to confirm the settings. Otherwise, press the right button to disply the next secondry option or press the left button to disply the previous options. IPv6 Confirm Crete? option Using this option, you cn strt n opertion to crete clustered system with n IPv6 ddress. 1. Press nd relese the left or right button until IPv6 Confirm Crete? is displyed. 2. Press the select button to strt the opertion. If the crete opertion is successful, Pssword is displyed on line 1. The pssword tht you cn use to ccess the system is displyed on line 2. Be sure to immeditely record the pssword; it is required on the first ttempt to mnge the system from the mngement GUI. Attention: The pssword displys for only 60 seconds, or until front pnel button is pressed. The clustered system is creted only fter the pssword disply is clered. If the crete opertion fils, Crete Filed: is displyed on line 1 of the front-pnel disply screen. Line 2 displys one of two possible error codes tht you cn use to isolte the cuse of the filure. Chpter 6. Using the front pnel of the SAN Volume Controller 113

132 Service IPv4 or Service IPv6 options You cn use the front pnel to chnge service IPv4 ddress or service IPv6 ddress. IPv4 Address option The IPv4 Address pnels show one of the following items for the selected Ethernet port: v v v v The ctive service ddress if the system hs n IPv4 ddress. This ddress cn be either configured or fixed ddress, or it cn be n ddress obtined through DHCP. DHCP Filed if the IPv4 service ddress is configured for DHCP but the node ws unble to obtin n IP ddress. DHCP Configuring if the IPv4 service ddress is configured for DHCP while the node ttempts to obtin n IP ddress. This ddress chnges to the IPv4 ddress utomticlly if DHCP ddress is llocted nd ctivted. A blnk line if the system does not hve n IPv4 ddress. If the service IPv4 ddress ws not set correctly or DHCP ddress ws not llocted, you hve the option of correcting the IPv4 ddress from this pnel. The service IP ddress must be in the sme subnet s the mngement IP ddress. To set fixed service IPv4 ddress from the IPv4 Address: pnel, perform the following steps: 1. Press nd relese the select button to put the pnel in edit mode. 2. Press the right button or left button to move to the number field tht you wnt to set. 3. Press the up button if you wnt to increse the vlue tht is highlighted; press the down button if you wnt to decrese tht vlue. If you wnt to quickly increse the highlighted vlue, hold the up button. If you wnt to quickly decrese the highlighted vlue, hold the down button. Note: If you wnt to disble the fst increse or decrese function, press nd hold the down button, press nd relese the select button, nd then relese the down button. The disbling of the fst increse or decrese function lsts until the cretion is completed or until the feture is gin enbled. If the up button or down button is pressed nd held while the function is disbled, the vlue increses or decreses once every two seconds. To gin enble the fst increse or decrese function, press nd hold the up button, press nd relese the select button, nd then relese the up button. 4. When ll the fields re set s required, press nd relese the select button to ctivte the new IPv4 ddress. The IPv4 Address: pnel is displyed. The new service IPv4 ddress is not displyed until it hs become ctive. If the new ddress hs not been displyed fter 2 minutes, check tht the selected ddress is vlid on the subnetwork nd tht the Ethernet switch is working correctly. IPv6 Address option The IPv6 Address pnels show one of the following conditions for the selected Ethernet port: 114 SAN Volume Controller: Troubleshooting Guide

133 v v v v The ctive service ddress if the system hs n IPv6 ddress. This ddress cn be either configured or fixed ddress, or it cn be n ddress obtined through DHCP. DHCP Filed if the IPv6 service ddress is configured for DHCP but the node ws unble to obtin n IP ddress. DHCP Configuring if the IPv6 service ddress is configured for DHCP while the node ttempts to obtin n IP ddress. This chnges to the IPv6 ddress utomticlly if DHCP ddress is llocted nd ctivted. A blnk line if the system does not hve n IPv6 ddress. If the service IPv6 ddress ws not set correctly or DHCP ddress ws not llocted, you hve the option of correcting the IPv6 ddress from this pnel. The service IP ddress must be in the sme subnet s the mngement IP ddress. To set fixed service IPv6 ddress from the IPv6 Address: pnel, perform the following steps: 1. Press nd relese the select button to put the pnel in edit mode. When the pnel is in edit mode, the full ddress is still shown cross four pnels s eight (four-digit) hexdeciml vlues. You edit ech digit of the hexdeciml vlues independently. The current digit is highlighted. 2. Press the right button or left button to move to the number field tht you wnt to set. 3. Press the up button if you wnt to increse the vlue tht is highlighted; press the down button if you wnt to decrese tht vlue. 4. When ll the fields re set s required, press nd relese the select button to ctivte the new IPv6 ddress. The IPv6 Address: pnel is displyed. The new service IPv6 ddress is not displyed until it hs become ctive. If the new ddress hs not been displyed fter 2 minutes, check tht the selected ddress is vlid on the subnetwork nd tht the Ethernet switch is working correctly. Service DHCPv4 or DHCPv6 options The ctive service ddress for system cn be either configured or fixed ddress, or it cn be n ddress obtined through DHCP. If service IP ddress does not exist, you must ssign service IP ddress or use DHCP with this ction. To set the service IPv4 ddress to use DHCP, perform the following steps: 1. Press nd relese the up or down button until Service DHCPv4? is displyed. 2. Press nd relese the down button. Confirm DHCPv4? is displyed. 3. Press nd relese the select button to ctivte DHCP, or you cn press nd relese the up button to keep the existing ddress. 4. If you ctivte DHCP, DHCP Configuring is displyed while the node ttempts to obtin DHCP ddress. It chnges utomticlly to show the llocted ddress if DHCP ddress is llocted nd ctivted, or it chnges to DHCP Filed if DHCP ddress is not llocted. To set the service IPv6 ddress to use DHCP, perform the following steps: 1. Press nd relese the up or down button until Service DHCPv6? is displyed. 2. Press nd relese the down button. Confirm DHCPv6? is displyed. Chpter 6. Using the front pnel of the SAN Volume Controller 115

134 3. Press nd relese the select button to ctivte DHCP, or you cn press nd relese the up button to keep the existing ddress. 4. If you ctivte DHCP, DHCP Configuring is displyed while the node ttempts to obtin DHCP ddress. It chnges utomticlly to show the llocted ddress if DHCP ddress is llocted nd ctivted, or it chnges to DHCP Filed if DHCP ddress is not llocted. Note: If n IPv6 router is present on the locl network, SAN Volume Controller does not differentite between n utoconfigured ddress nd DHCP ddress. Therefore, SAN Volume Controller uses the first ddress tht is detected. Chnge WWNN? option The Chnge WWNN? option displys the lst five hexdeciml digits of the WWNN tht is being used by the node. Only the lst five digits of WWNN vry on node. The first 11 digits re lwys To edit the WWNN, complete the following steps: Importnt: Only chnge the WWNN when you re instructed to do so by service procedure. Nodes must lwys hve unique WWNN. If you chnge the WWNN, you might hve to reconfigure hosts nd the SAN zoning. 1. Press nd relese the up or down button until Actions is displyed. 2. Press nd relese the select button. 3. Press nd relese the up or down button until Chnge WWNN? is displyed on line 1. Line 2 of the disply shows the lst five numbers of the WWNN tht is currently set. The first number is highlighted. 4. Edit the highlighted number to mtch the number tht is required. Use the up nd down buttons to increse or decrese the numbers. The numbers wrp F to 0 or 0 to F. Use the left nd right buttons to move between the numbers. 5. When the highlighted vlue mtches the required number, press nd relese the select button to ctivte the chnge. The Node WWNN: pnel displys nd the second line shows the lst five chrcters of the chnged WWNN. Enter Service? option You cn enter service stte from the Enter Service? option. Service stte cn be used to remove node from cndidte list or to prevent it from being redded to clustered system. If the node is ctive, entering service stte cn cuse disruption to hosts if other fults exist in the system.while in service stte, the node cnnot join or run s prt of clustered system. To exit service stte, ensure tht ll errors re resolved. You cn exit service stte by using the Exit Service? option or by restrting the node. Exit Service? option You cn exit service stte from the Exit Service? option. This ction releses the node from the service stte. If there re no noncriticl errors, the node enters cndidte stte. If possible, the node then becomes ctive in clustered system. To exit service stte, ensure tht ll errors re resolved. You cn exit service stte by using this option or by restrting the node. 116 SAN Volume Controller: Troubleshooting Guide

135 Recover Cluster? option You cn recover n entire clustered system if the dt hs been lost from ll nodes by using the Recover Cluster? option. Perform service ctions on nodes only when directed by the service procedures. If used inppropritely, service ctions cn cuse loss of ccess to dt or dt loss. For informtion bout the recover system procedure, see Recover system procedure on pge 245. Remove Cluster? option The Remove Cluster? option deletes the system stte dt from the node. Use this option s the finl step in decommissioning clustered system fter the other nodes hve been removed from the system using the commnd-line interfce (CLI) or the mngement GUI. Attention: Use the front pnel to remove stte dt from single node system. To remove node from multi-node system, lwys use the CLI or the remove node options from the mngement GUI. To delete the stte dt from the node using the Remove Cluster? pnel: 1. Press nd hold the up button. 2. Press nd relese the select button. 3. Relese the up button. After the option is run, the node shows Cluster: with no system nme. If this option is run on node tht is still member of system, the system shows error 1195, Node missing, nd the node is displyed in the list of nodes in the system. Remove the node by using the mngement GUI or CLI. Pced Upgrde? option Use this option to control the time when individul nodes re upgrded within concurrent code upgrde. Note: This ction cn be used only when the following conditions exist for the node: v The node is in service stte. v The node hs no errors. v The node hs been removed from the clustered system. For dditionl informtion, see the Upgrding the softwre mnully topic in the informtion center. Set FC Speed? option You cn chnge the speed of the Fibre Chnnel ports on SAN Volume Controller by using the Set FC Speed? option Reset Pssword? option The Reset Pssword? option is useful if the system superuser pssword hs been lost or forgotten. Use the Reset pssword? option if the user hs lost the system superuser pssword or if the user is unble to ccess the system. If it is permitted by the user's pssword security policy, use this selection to reset the system superuser pssword. Chpter 6. Using the front pnel of the SAN Volume Controller 117

136 If your pssword security policy permits pssword recovery, nd if the node is currently member of clustered system, the system superuser pssword is reset nd new pssword is displyed for 60 seconds. If your pssword security policy does not permit pssword recovery or the node is not member of system, completing these steps hs no effect. If the node is in ctive stte when the pssword is reset, the reset pplies to ll nodes in the system. If the node is in cndidte or service stte when the pssword is reset, the reset pplies only to the single node. Rescue Node? option You cn strt the utomtic softwre recovery for this node by using the Rescue Node? option. Note: Another wy to rescue node is to force node rescue when the node boots. It is the preferred method. Forcing node rescue when node boots works by booting the operting system from the service controller nd running progrm tht copies ll the SAN Volume Controller softwre from ny other node tht cn be found on the Fibre Chnnel fbric. See Performing the node rescue when the node boots on pge 260. Exit Actions? option Return to the min menu by selecting the Exit Actions? option. Lnguge? option You cn chnge the lnguge tht displys on the front pnel. Before you begin The Lnguge? option llows you to chnge the lnguge tht is displyed on the menu. Figure 56 shows the Lnguge? option sequence. Lnguge? Select English? Jpnese? svc00410 Figure 56. Lnguge? nvigtion The following lnguges re vilble: v English v Jpnese About this tsk To select the lnguge tht you wnt to be used on the front pnel, perform the following steps: Procedure 1. Press nd relese the up or down button until Lnguge? is displyed. 2. Press nd relese the select button. 118 SAN Volume Controller: Troubleshooting Guide

137 3. Use the left nd right buttons to move to the lnguge tht you wnt. The trnslted lnguge nmes re displyed in their own chrcter set. If you do not understnd the lnguge tht is displyed, wit for t lest 60 seconds for the menu to reset to the defult option. 4. Press nd relese the select button to select the lnguge tht is displyed. Results If the selected lnguge uses the Ltin lphbet, the front pnel disply shows two lines. The pnel text is displyed on the first line nd dditionl dt is displyed on the second line. If the selected lnguge does not use the Ltin lphbet, the disply shows only one line t time to clerly disply the chrcter font. For those lnguges, you cn switch between the pnel text nd the dditionl dt by pressing nd relesing the select button. Additionl dt is unvilble when the front pnel displys menu option, which ends with question mrk (?). In this cse, press nd relese the select button to choose the menu option. Note: You cnnot select nother lnguge when the node is displying boot error. Using the power control for the SAN Volume Controller node Some SAN Volume Controller nodes re powered by n uninterruptible power supply tht is in the sme rck s the nodes. Other nodes hve internl btteries insted, such s the SAN Volume Controller 2145-DH8. The power stte of the SAN Volume Controller is displyed by power indictor on the front pnel. If the uninterruptible power supply bttery is not sufficiently chrged to enble the SAN Volume Controller to become fully opertionl, its chrge stte is displyed on the front pnel disply of the node. The power to SAN Volume Controller is controlled through the mngement GUI, or by the power button on the front pnel of the node.. Note: Never turn off the node by removing the power cble. You might lose dt. For more informtion bout how to power off the node, see MAP 5350: Powering off SAN Volume Controller node on pge 298 If the SAN Volume Controller softwre is running nd you request it to power off from the mngement GUI, CLI, or power button, the node strts its power off processing. During this time, the node indictes the progress of the power-off opertion on the front pnel disply. After the power-off processing is complete, the front pnel becomes blnk nd the front pnel power light flshes. It is sfe for you to remove the power cble from the rer of the node. If the power button on the front pnel is pressed during power-off processing, the front pnel disply chnges to indicte tht the node is being restrted, but the power-off process completes before the restrt. If the SAN Volume Controller softwre is not running when the front pnel power button is pressed, the node immeditely powers off. Chpter 6. Using the front pnel of the SAN Volume Controller 119

138 Note: The 2145 UPS-1U does not power off when the node is shut down from the power button. If you turn off node using the power button or by commnd, the node is put into power-off stte. The SAN Volume Controller remins in this stte until the power cble is connected to the rer of the node nd the power button is pressed. During the strtup sequence, the SAN Volume Controller tries to detect the sttus of the uninterruptible power supply through the uninterruptible power supply signl cble. If n uninterruptible power supply is not detected, the node puses nd n error is shown on the front pnel disply. If the uninterruptible power supply is detected, the softwre monitors the opertionl stte of the uninterruptible power supply. If no uninterruptible power supply errors re reported nd the uninterruptible power supply bttery is sufficiently chrged, the SAN Volume Controller becomes opertionl. If the uninterruptible power supply bttery is not sufficiently chrged, the chrge stte is indicted by progress br on the front pnel disply. When n uninterruptible power supply is first turned on, it might tke up to 2 hours before the bttery is sufficiently chrged for the SAN Volume Controller node to become opertionl. If input power to the uninterruptible power supply is lost, the node immeditely stops ll I/O opertions nd sves the contents of its dynmic rndom ccess memory (DRAM) to the internl disk drive. While dt is being sved to the disk drive, Power Filure messge is shown on the front pnel nd is ccompnied by descending progress br tht indictes the quntity of dt tht remins to be sved. After ll the dt is sved, the node is turned off nd the power light on the front pnel turns off. Note: The node is now in stndby stte. If the input power to the uninterruptible power supply unit is restored, the node restrts. If the uninterruptible power supply bttery ws fully dischrged, Chrging is displyed nd the boot process wits for the bttery to chrge. When the bttery is sufficiently chrged, Booting is displyed, the node is tested, nd the softwre is loded. When the boot process is complete, Recovering is displyed while the uninterruptible power supply finlizes its chrge. While Recovering is displyed, the system cn function normlly. However, when the power is restored fter second power filure, there is dely (with Chrging displyed) before the node cn complete its boot process. 120 SAN Volume Controller: Troubleshooting Guide

139 Chpter 7. Dignosing problems You cn dignose problems with the control nd indictors, the commnd-line interfce (CLI), the mngement GUI, or the Service Assistnt GUI. The dignostic LEDs on the SAN Volume Controller nodes nd uninterruptible power supply units lso help you dignose hrdwre problems. Event logs By understnding the event log, you cn do the following tsks: v Mnge the event log v View the event log v Describe the fields in the event log Error codes The following topics provide informtion to help you understnd nd process the error codes: v Event reporting v Understnding the events v Understnding the error codes v Determining hrdwre boot filure If the node is showing boot messge, filure messge, or node error messge, nd you determined tht the problem ws cused by softwre or firmwre filure, you cn restrt the node to see whether tht might resolve the problem. Perform the following steps to properly shut down nd restrt the node: 1. Follow the instructions in MAP 5350: Powering off SAN Volume Controller node on pge Restrt only one node t time. 3. Do not shut down the second node in n I/O group for t lest 30 minutes fter you shut down nd restrt the first node. Strting sttistics collection You cn strt the collection of cluster sttistics from the Strting the Collection of Sttistics pnel in the mngement GUI. Introduction For ech collection intervl, the mngement GUI cretes four sttistics files: one for mnged disks (MDisks), nmed Nm_stt; one for volumes nd volume copies, nmed Nv_stt; one for nodes, nmed Nn_stt; nd one for drives, nmed Nd_stt. The files re written to the /dumps/iostts directory on the node. To retrieve the sttistics files from the non-configurtion nodes onto the configurtion node, svctsk cpdumps commnd must be used. A mximum of 16 files of ech type cn be creted for the node. When the 17th file is creted, the oldest file for the node is overwritten. Copyright IBM Corp. 2003,

140 Fields The following fields re vilble for user definition: Intervl Specify the intervl in minutes between the collection of sttistics. You cn specify 1-60minutes in increments of 1 minute. Tbles The following tbles describe the informtion tht is reported for individul nodes nd volumes. Tble 72 describes the sttistics collection for MDisks, for individul nodes. Tble 72. Sttistics collection for individul nodes Sttistic Description nme id Indictes the nme of the MDisk for which the sttistics pply. idx Indictes the identifier of the MDisk for which the sttistics pply. rb Indictes the cumultive number of blocks of dt tht is red (since the node hs been running). re Indictes the cumultive red externl response time in milliseconds for ech MDisk. The cumultive response time for disk reds is clculted by strting timer when SCSI red commnd is issued nd stopped when the commnd completes successfully. The elpsed time is dded to the cumultive counter. ro Indictes the cumultive number of MDisk red opertions tht re processed (since the node hs been running). rq Indictes the cumultive red queued response time in milliseconds for ech MDisk. This response is mesured from bove the queue of commnds to be sent to n MDisk becuse the queue depth is lredy full. This clcultion includes the elpsed time tht is tken for red commnds to complete from the time they join the queue. wb Indictes the cumultive number of blocks of dt written (since the node hs been running). we Indictes the cumultive write externl response time in milliseconds for ech MDisk. The cumultive response time for disk writes is clculted by strting timer when SCSI write commnd is issued nd stopped when the commnd completes successfully. The elpsed time is dded to the cumultive counter. wo Indictes the cumultive number of MDisk write opertions processed (since the node hs been running). wq Indictes the cumultive write queued response time in milliseconds for ech MDisk. This is mesured from bove the queue of commnds to be sent to n MDisk becuse the queue depth is lredy full. This clcultion includes the elpsed time tken for write commnds to complete from the time they join the queue. Tble 73 on pge 123 describes the VDisk (volume) informtion tht is reported for individul nodes. Note: MDisk sttistics files for nodes re written to the /dumps/iostts directory on the individul node. 122 SAN Volume Controller: Troubleshooting Guide

141 Tble 73. Sttistic collection for volumes for individul nodes Sttistic Description nme id Indictes the volume nme for which the sttistics pply. idx Indictes the volume for which the sttistics pply. rb Indictes the cumultive number of blocks of dt red (since the node hs been running). rl Indictes the cumultive red response time in milliseconds for ech volume. The cumultive response time for volume reds is clculted by strting timer when SCSI red commnd is received nd stopped when the commnd completes successfully. The elpsed time is dded to the cumultive counter. rlw Indictes the worst red response time in microseconds for ech volume since the lst time sttistics were collected. This vlue is reset to zero fter ech sttistics collection smple. ro Indictes the cumultive number of volume red opertions processed (since the node hs been running). wb Indictes the cumultive number of blocks of dt written (since the node hs been running). wl Indictes the cumultive write response time in milliseconds for ech volume. The cumultive response time for volume writes is clculted by strting timer when SCSI write commnd is received nd stopped when the commnd completes successfully. The elpsed time is dded to the cumultive counter. wlw Indictes the worst write response time in microseconds for ech volume since the lst time sttistics were collected. This vlue is reset to zero fter ech sttistics collection smple. wo Indictes the cumultive number of volume write opertions processed (since the node hs been running). xl Indictes the cumultive dt trnsfer response time in milliseconds for ech volume since the lst time sttistics were collected. When this sttistic is viewed for multiple volumes nd with other sttistics, it cn indicte if the ltency is cused by the host, fbric, or the SAN Volume Controller. Tble 74 describes the VDisk informtion relted to Metro Mirror or Globl Mirror reltionships tht is reported for individul nodes. Tble 74. Sttistic collection for volumes tht re used in Metro Mirror nd Globl Mirror reltionships for individul nodes Sttistic Description nme gwl gwo Indictes cumultive secondry write ltency in milliseconds. This sttistic ccumultes the cumultive secondry write ltency for ech volume. You cn clculte the mount of time to recovery from filure bsed on this sttistic nd the gws sttistics. Indictes the totl number of overlpping volume writes. An overlpping write is when the logicl block ddress (LBA) rnge of write request collides with nother outstnding request to the sme LBA rnge nd the write request is still outstnding to the secondry site. Chpter 7. Dignosing problems 123

142 Tble 74. Sttistic collection for volumes tht re used in Metro Mirror nd Globl Mirror reltionships for individul nodes (continued) gwot gws Indictes the totl number of fixed or unfixed overlpping writes. When ll nodes in ll clusters re running SAN Volume Controller version 4.3.1, this records the totl number of write I/O requests received by the Globl Mirror feture on the primry tht hve overlpped. When ny nodes in either cluster re running SAN Volume Controller versions erlier thn 4.3.1, this vlue does not increment. Indictes the totl number of write requests tht hve been issued to the secondry site. Tble 75describes the port informtion tht is reported for individul nodes Tble 75. Sttistic collection for node ports Sttistic Description nme bbcz Indictes the totl time in microseconds for which the port hd dt to send but ws prevented from doing so by lck of buffer credit from the switch. cbr Indictes the bytes received from controllers. cbt Indictes the bytes trnsmitted to disk controllers. cer Indictes the commnds received from disk controllers. cet Indictes the commnds initited to disk controllers. cpu_busy Indictes the number of busy milliseconds since the node ws reset. This sttistic reports the mount of the time the processor hs spent polling while witing for work versus ctully doing work. This sttistic ccumultes from zero. hbr Indictes the bytes received from hosts. hbt Indictes the bytes trnsmitted to hosts. her Indictes the commnds received from hosts. het Indictes the commnds initited to hosts. icrc Indictes the number of CRC tht re not vlid. id Indictes the port identifier for the node. itw Indictes the number of trnsmission word counts tht re not vlid. lf Indictes link filure count. lnbr Indictes the bytes received to other nodes in the sme cluster. lnbt Indictes the bytes trnsmitted to other nodes in the sme cluster. lner Indictes the commnds received from other nodes in the sme cluster. lnet Indictes the commnds initited to other nodes in the sme cluster. lsi Indictes the lost-of-signl count. lsy Indictes the loss-of-synchroniztion count. pspe Indictes the primitive sequence-protocol error count. rmbr Indictes the bytes received to other nodes in the other clusters. rmbt Indictes the bytes trnsmitted to other nodes in the other clusters. rmer Indictes the commnds received from other nodes in the other clusters. rmet Indictes the commnds initited to other nodes in the other clusters. wwpn Indictes the worldwide port nme for the node. 124 SAN Volume Controller: Troubleshooting Guide

143 Tble 76 describes the node informtion tht is reported for ech nodes. Tble 76. Sttistic collection for nodes Sttistic Description nme cluster_id Indictes the nme of the cluster. cluster Indictes the nme of the cluster. id Indictes the nme of the node. node_id Indictes the unique identifier for the node. rb Indictes the number of bytes received. re Indictes the ccumulted receive ltency, excluding inbound queue time. This sttistic is the ltency tht is experienced by the node communiction lyer from the time tht n I/O is queued to cche until the time tht the cche gives completion for it. ro Indictes the number of messges or bulk dt received. rq Indictes the ccumulted receive ltency, including inbound queue time. This sttistic is the ltency from the time tht commnd rrives t the node communiction lyer to the time tht the cche completes the commnd. wb Indictes the bytes sent. we Indictes the ccumulted send ltency, excluding outbound queue time. This sttistic is the time from when the node communiction lyer issues messge out onto the Fibre Chnnel until the node communiction lyer receives notifiction tht the messge hs rrived. wo Indictes the number of messges or bulk dt sent. wq Indictes the ccumulted send ltency, including outbound queue time. This sttistic includes the entire time tht dt is sent. This time includes the time from when the node communiction lyer receives messge nd wits for resources, the time to send the messge to the remote node, nd the time tken for the remote node to respond. Tble 77 describes the sttistics collection for volumes. Tble 77. Cche sttistics collection for volumes nd volume copies Sttistic Acronym Sttistics for volume cche Sttistics for volume copy cche Sttistics for volume cche prtition Sttistics for volume copy cche prtition Sttistics for the Node Overll Cche Cche sttistics for mdisks Units nd stte red ios ri Yes Yes ios, cumultive write ios wi Yes Yes ios, cumultive reds r Yes Yes ios, cumultive red hits rh Yes Yes sectors, cumultive flush_through writes fst_write writes write_through writes ft Yes Yes sectors, cumultive fw Yes Yes sectors, cumultive wt Yes Yes sectors, cumultive write hits wh Yes Yes sectors, cumultive Chpter 7. Dignosing problems 125

144 Tble 77. Cche sttistics collection for volumes nd volume copies (continued) Sttistic Acronym Sttistics for volume cche Sttistics for volume copy cche Sttistics for volume cche prtition Sttistics for volume copy cche prtition Sttistics for the Node Overll Cche Cche sttistics for mdisks Units nd stte prefetches p Yes sectors, cumultive prefetch hits (prefetch dt tht is red) prefetch misses (prefetch pges tht re discrded without ny sectors red) ph Yes sectors, cumultive pm Yes pges, cumultive destges d Yes Yes sectors, cumultive fullness Averge fv Yes Yes %, noncumultive fullness Mx fmx Yes Yes %, noncumultive fullness Min fmn Yes Yes %, noncumultive Destge Trget Averge Destge Trget Mx Destge Trget Min Destge In Flight Averge Destge In Flight Mx Destge In Flight Min destge ltency verge destge ltency mx destge ltency min dtv Yes Yes IOs cpped 9999, noncumultive dtmx Yes IOs, noncumultive dtmn Yes IOs, noncumultive dfv Yes Yes IOs cpped 9999, noncumultive dfmx Yes IOs, noncumultive dfmn Yes IOs, noncumultive dv Yes Yes Yes Yes Yes Yes µs cpped , noncumultive dmx Yes Yes Yes µs cpped , noncumultive dmn Yes Yes Yes µs cpped , noncumultive 126 SAN Volume Controller: Troubleshooting Guide

145 Tble 77. Cche sttistics collection for volumes nd volume copies (continued) Sttistic Acronym Sttistics for volume cche Sttistics for volume copy cche Sttistics for volume cche prtition Sttistics for volume copy cche prtition Sttistics for the Node Overll Cche Cche sttistics for mdisks Units nd stte destge count dcn Yes Yes Yes Yes Yes ios, noncumultive stge ltency verge stge ltency mx stge ltency min sv Yes Yes Yes µs cpped , noncumultive smx Yes µs cpped , noncumultive smn Yes µs cpped , noncumultive stge count scn Yes Yes Yes ios, noncumultive prestge ltency verge prestge ltency mx prestge ltency min prestge count Write Cche Fullness Averge Write Cche Fullness Mx Write Cche Fullness Min Red Cche Fullness Averge Red Cche Fullness Mx Red Cche Fullness Min pv Yes Yes µs cpped , noncumultive pmx Yes µs cpped , noncumultive pmn Yes µs cpped , noncumultive pcn Yes Yes ios, noncumultive wfv Yes %, noncumultive wfmx Yes %, noncumultive wfmn Yes %, noncumultive rfv Yes %, noncumultive rfmx Yes %, noncumultive rfmn Yes %, noncumultive Chpter 7. Dignosing problems 127

146 Tble 77. Cche sttistics collection for volumes nd volume copies (continued) Sttistic Pinned Percent dt trnsfer ltency verge Trck Lock Ltency (Exclusive) Averge Trck Lock Ltency (Shred) Averge Cche I/O Control Block Queue Time Cche Trck Control Block Queue Time Owner Remote Credit Queue Time Non-Owner Remote Credit Queue Time Admin Remote Credit Queue Time Cdcb Queue Time Buffer Queue Time Hrdening Rights Queue Time Acronym Sttistics for volume cche Sttistics for volume copy cche Sttistics for volume cche prtition Sttistics for volume copy cche prtition Sttistics for the Node Overll Cche Cche sttistics for mdisks Units nd stte pp Yes Yes Yes Yes Yes % of totl cche snpshot, noncumultive tv Yes Yes µs cpped , noncumultive tev Yes Yes µs cpped , noncumultive tsv Yes Yes µs cpped , noncumultive hpt Yes Averge µs, noncumultive ppt Yes Averge µs, noncumultive opt Yes Averge µs, noncumultive npt Yes Averge µs, noncumultive pt Yes Averge µs, noncumultive cpt Yes Averge µs, noncumultive bpt Yes Averge µs, noncumultive hrpt Yes Averge µs, noncumultive Note: Any sttistic with nme v, mx, mn, nd cn is not cumultive. These sttistics reset every sttistics intervl. For exmple, if the sttistic does not hve nme with nme v, mx, mn, nd cn, nd it is n Ios or count, it will be field contining totl number. v The term pges mens in units of 4096 bytes per pge. v The term sectors mens in units of 512 bytes per sector. v The term µs mens microseconds. v Non-cumultive mens totls since the previous sttistics collection intervl. Tble 78 on pge 129 describes the sttistic collection for volume cche per individul nodes. 128 SAN Volume Controller: Troubleshooting Guide

147 Tble 78. Sttistic collection for volume cche per individul nodes. This tble describes the volume cche informtion tht is reported for individul nodes. Sttistic Description nme cm ctd ctds ctp ctps ctrh ctrhp ctrhps ctrhs ctr ctrs ctwft ctwfts ctwfw ctwfwsh ctwfwshs ctwfws ctwh ctwhs Indictes the number of sectors of modified or dirty dt tht re held in the cche. Indictes the totl number of cche destges tht were initited writes, submitted to other components s result of volume cche flush or destge opertion. Indictes the totl number of sectors tht re written for cche-initited trck writes. Indictes the number of trck stges tht re initited by the cche tht re prestge reds. Indictes the totl number of stged sectors tht re initited by the cche. Indictes the number of totl trck red-cche hits on prestge or non-prestge dt. For exmple, single red tht spns two trcks where only one of the trcks obtined totl cche hit, is counted s one trck red-cche hit. Indictes the number of trck reds received from other components, treted s cche hits on ny prestged dt. For exmple, if single red spns two trcks where only one of the trcks obtined totl cche hit on prestged dt, it is counted s one trck red for the prestged dt. A cche hit tht obtins prtil hit on prestge nd non-prestge dt still contributes to this vlue. Indictes the totl number of sectors tht re red for reds received from other components tht obtined cche hits on ny prestged dt. Indictes the totl number of sectors tht re red for reds received from other components tht obtined totl cche hits on prestge or non-prestge dt. Indictes the totl number of trck reds received. For exmple, if single red spns two trcks, it is counted s two totl trck reds. Indictes the totl number of sectors tht re red for reds received. Indictes the number of trck writes received from other components nd processed in flush through write mode. Indictes the totl number of sectors tht re written for writes tht re received from other components nd processed in flush through write mode. Indictes the number of trck writes received from other components nd processed in fst-write mode. Indictes the trck writes in fst-write mode tht were written in write-through mode becuse of the lck of memory. Indictes the trck writes in fst-write mode tht were written in write through due to the lck of memory. Indictes the totl number of sectors tht re written for writes tht re received from other components nd processed in fst-write mode. Indictes the number of trck writes received from other components where every sector in the trck obtined write hit on lredy dirty dt in the cche. For write to count s totl cche hit, the entire trck write dt must lredy be mrked in the write cche s dirty. Indictes the totl number of sectors tht re received from other components where every sector in the trck obtined write hit on lredy dirty dt in the cche. Chpter 7. Dignosing problems 129

148 Tble 78. Sttistic collection for volume cche per individul nodes (continued). This tble describes the volume cche informtion tht is reported for individul nodes. ctw ctws ctwwt ctwwts cv Indictes the totl number of trck writes received. For exmple, if single write spns two trcks, it is counted s two totl trck writes. Indictes the totl number of sectors tht re written for writes tht re received from components. Indictes the number of trck writes received from other components nd processed in write through write mode. Indictes the totl number of sectors tht re written for writes tht re received from other components nd processed in write through write mode. Indictes the number of sectors of red nd write cche dt tht is held in the cche. Tble 79 describes those XML sttistics prticulr to n IP Repliction port. Tble 79. XML sttistics for n IP Repliction port Sttistic nme Description ipbz Indictes the verge size (in bytes) of dt tht re being submitted to the IP Repliction driver since the lst sttistics collection period. ipre Indictes the bytes retrnsmitted to other nodes in other clusters by the IP Repliction driver. iprt Indictes the verge round-trip time in microseconds for the IP Repliction link since the lst sttistics collection period. iprx Indictes the bytes received from other nodes in other clusters by the IP Repliction driver. ipsz Indictes the verge size (in bytes) of dt tht re being trnsmitted by the IP Repliction driver since the lst sttistics collection period. iptx Indictes the bytes trnsmitted to other nodes in other clusters by the IP Repliction driver. Actions The following ctions re vilble to the user: OK Click this button to chnge sttistic collection. Cncel Click this button to exit the pnel without chnging sttistic collection. XML formtting informtion The XML is more complicted now, s seen in this rw XML from the volume (Nv_sttistics) sttistics. Notice how the nmes re similr but becuse they re in different section of the XML, they refer to different prt of the VDisk. <vdsk idx="0" ctrs=" " ctps="0" ctrhs=" " ctrhps="0" ctds=" " ctwfts="9635" ctwwts="0" ctwfws=" " ctwhs="9117" ctws=" " ctr=" " ctw=" " ctp="0" ctrh="123056" ctrhp="0" ctd=" " ctwft="200" ctwwt="0" ctwfw=" " ctwfwsh="0" ctwfwshs="0" ctwh="538" cm=" " cv=" " gwot="0" gwo="0" gws="0" gwl="0" 130 SAN Volume Controller: Troubleshooting Guide

149 Event reporting id="mster_iogrp0_1" ro="0" wo="0" rb="0" wb="0" rl="0" wl="0" rlw="0" wlw="0" xl="0"> Vdisk/Volume sttistics <c r="0" rh="0" d="0" ft="0" wt="0" fw="0" wh="0" ri="0" wi="0" dv="0" dcn="0" pv="0" pcn="0" tev="0" tsv="0" tv="0" pp="0"/> <cpy idx="0"> volume copy sttistics <c r="0" p="0" rh="0" ph="0" d="0" ft="0" wt="0" fw="0" wh="0" pm="0" ri="0" wi="0" dv="0" dcn="0" sv="0" scn="0" pv="0" pcn="0" tev="0" tsv="0" tv="0" pp="0"/> </cpy> <vdsk> The <cpy idx="0"> mens its in the volume copy section of the VDisk, wheres the sttistics shown under Vdisk/Volume sttistics re outside of the cpy idx section nd therefore refer to VDisk/volume. Similrly for the volume cche sttistics for node nd prtitions: <uc><c dv="18726" dcn=" " dmx="749846" dmn="89" sv="20868" scn=" " smx="980941" smn="3" pv="0" pcn="0" pmx="0" pmn="0" wfv="0" wfmx="2" wfmn="0" rfv="0" rfmx="1" rfmn="0" pp="0" hpt="0" ppt="0" opt="0" npt="0" pt="0" cpt="0" bpt="0" hrpt="0" /><prtition id="0"><c dv="18726" dcn=" " dmx="749846" dmn="89" fv="0" fmx="2" fmn="0" dfv="0" dfmx="0" dfmn="0" dtv="0" dtmx="0" dtmn="0" pp="0"/></prtition> This output describes the volume cche node sttistics where <prtition id="0"> the sttistics re described for prtition 0. Replcing <uc> with <lc> mens tht the sttistics re for volume copy cche prtition 0. Events tht re detected re sved in n event log. As soon s n entry is mde in this event log, the condition is nlyzed. If ny service ctivity is required, notifiction is sent, if you hve set up notifictions. Event reporting process The following methods re used to notify you nd the IBM Support Center of new event: v v The most serious system error code is displyed on the front pnel of ech node in the system. If you enbled Simple Network Mngement Protocol (SNMP), n SNMP trp is sent to n SNMP mnger tht is configured by the customer. Chpter 7. Dignosing problems 131

150 v v v The SNMP mnger might be IBM Systems Director, if it is instlled, or nother SNMP mnger. If enbled, log messges cn be forwrded on n IP network by using the syslog protocol. If enbled, event notifictions cn be forwrded by emil by using Simple Mil Trnsfer Protocol (SMTP). Cll Home cn be enbled so tht criticl fults generte problem mngement record (PMR) tht is then sent directly to the pproprite IBM Support Center by using emil. Power-on self-test When you turn on the SAN Volume Controller, the system bord performs self-tests. During the initil tests, the hrdwre boot symbol is displyed. All models perform series of tests to check the opertion of components nd some of the options tht hve been instlled when the units re first turned on. This series of tests is clled the power-on self-test (POST). Note: Some SAN Volume Controller nodes do not hve front pnel disply. The node sttus LED is off until booting finishes nd the SAN Volume Controller softwre is loded. Understnding events If criticl filure is detected during the POST, the softwre is not loded nd the system error LED on the opertor informtion pnel is illuminted. If this filure occurs, use MAP 5000: Strt on pge 265 to help isolte the cuse of the filure. When the softwre is loded, dditionl testing tkes plce, which ensures tht ll of the required hrdwre nd softwre components re instlled nd functioning correctly. During the dditionl testing, the word Booting is displyed on the front pnel long with boot progress code nd progress br. If test filure occurs, the word Filed is displyed on the front pnel. The service controller performs internl checks nd is vitl to the opertion of the SAN Volume Controller. If the error (check) LED is illuminted on the service controller front pnel, the front-pnel disply might not be functioning correctly nd you cn ignore ny messge displyed. The uninterruptible power supply lso performs internl tests. If the uninterruptible power supply reports the filure condition, the SAN Volume Controller displys criticl filure informtion bout the front-pnel disply or sends noncriticl filure informtion to the event log. If the SAN Volume Controller cnnot communicte with the uninterruptible power supply, it displys boot filure error messge on the front-pnel disply. Further problem determintion informtion might lso be displyed on the front pnel of the uninterruptible power supply. When significnt chnge in sttus is detected, n event is logged in the event log. Error dt Events re clssified s either lerts or messges: 132 SAN Volume Controller: Troubleshooting Guide

151 v v An lert is logged when the event requires some ction. Some lerts hve n ssocited error code tht defines the service ction tht is required. The service ctions re utomted through the fix procedures. If the lert does not hve n error code, the lert represents n unexpected chnge in stte. This sitution must be investigted to see if it is expected or represents filure. Investigte n lert nd resolve it s soon s it is reported. A messge is logged when chnge tht is expected is reported, for instnce, n IBM FlshCopy opertion completes. Mnging the event log The event log hs limited size. After it is full, newer entries replce entries tht re no longer required. To void hving repeted event tht fills the event log, some records in the event log refer to multiple occurrences of the sme event. When event log entries re colesced in this wy, the time stmp of the first occurrence nd the lst occurrence of the problem is sved in the log entry. A count of the number of times tht the error condition hs occurred is lso sved in the log entry. Other dt refers to the lst occurrence of the event. Viewing the event log You cn view the event log by using the mngement GUI or the commnd-line interfce (CLI). About this tsk You cn view the event log by using the Monitoring > Events options in the mngement GUI. The event log contins mny entries. You cn, however, select only the type of informtion tht you need. You cn lso view the event log by using the commnd-line interfce (lseventlog). See the Commnd-line interfce topic for the commnd detils. Describing the fields in the event log The event log includes fields with informtion tht you cn use to dignose problems. Tble 80 describes some of the fields tht re vilble to ssist you in dignosing problems. Tble 80. Description of dt fields for the event log Dt field Event ID Error code Sequence number Event count Object type Object ID Description This number precisely identifies why the event ws logged. This number describes the service ction tht should be followed to resolve n error condition. Not ll events hve error codes tht re ssocited with them. Mny event IDs cn hve the sme error code becuse the service ction is the sme for ll the events. A number tht identifies the event. The number of events colesced into this event log record. The object type to which the event log reltes. A number tht uniquely identifies the instnce of the object. Chpter 7. Dignosing problems 133

152 Tble 80. Description of dt fields for the event log (continued) Dt field Description Fixed When n lert is shown for n error condition, it indictes if the reson for the event ws resolved. In mny cses, the system utomticlly mrks the events fixed when pproprite. There re some events tht must be mnully mrked s fixed. If the event is messge, this field indictes tht you hve red nd performed the ction. The messge must be mrked s red. First time The time when this error event ws reported. If events of similr type re being colesced together, so tht one event log record represents more thn one event, this field is the time the first error event ws logged. Lst time The time when the lst instnce of this error event ws recorded in the log. Root sequence number Sense dt If set, this number is the sequence number of n event tht represents n error tht probbly cused this event to be reported. Resolve the root event first. Additionl dt tht gives the detils of the condition tht cused the event to be logged. Event notifictions The system cn use Simple Network Mngement Protocol (SNMP) trps, syslog messges, nd Cll Home emils to notify you nd the IBM Support Center when significnt events re detected. Any combintion of these notifiction methods cn be used simultneously. Notifictions re normlly sent immeditely fter n event is rised. However, there re some events tht might occur becuse of ctive service ctions. If recommended service ction is ctive, these events re notified only if they re still unfixed when the service ction completes. Ech event tht the system detects is ssigned notifiction type of Error, Wrning, or Informtion. When you configure notifictions, you specify where the notifictions should be sent nd which notifiction types re sent to tht recipient. Tble 81 describes the levels of event notifictions. Tble 81. Notifiction levels Notifiction level Description Error Error notifiction is sent to indicte problem tht must be corrected s soon s possible. This notifiction indictes serious problem with the system. For exmple, the event tht is being reported could indicte loss of redundncy in the system, nd it is possible tht nother filure could result in loss of ccess to dt. The most typicl reson tht this type of notifiction is sent is becuse of hrdwre filure, but some configurtion errors or fbric errors lso re included in this notifiction level. Error notifictions cn be configured to be sent s Cll Home to the IBM Remote Technicl Support. 134 SAN Volume Controller: Troubleshooting Guide

153 Tble 81. Notifiction levels (continued) Notifiction level Description Wrning A wrning notifiction is sent to indicte problem or unexpected condition with the system. Alwys immeditely investigte this type of notifiction to determine the effect tht it might hve on your opertion, nd mke ny necessry corrections. Informtion A wrning notifiction does not require ny replcement prts nd therefore should not require IBM Support Center involvement. The lloction of notifiction type Wrning does not imply tht the event is less serious thn one tht hs notifiction level Error. An informtionl notifiction is sent to indicte tht n expected event hs occurred: for exmple, FlshCopy opertion hs completed. No remedil ction is required when these notifictions re sent. Events with notifiction type Error or Wrning re shown s lerts in the event log. Events with notifiction type Informtion re shown s messges. SNMP trps Simple Network Mngement Protocol (SNMP) is stndrd protocol for mnging networks nd exchnging messges. The system cn send SNMP messges tht notify personnel bout n event. You cn use n SNMP mnger to view the SNMP messges tht the system sends. You cn use the mngement GUI or the commnd-line interfce to configure nd modify your SNMP settings. You cn specify up to mximum of six SNMP servers. You cn use the Mngement Informtion Bse (MIB) file for SNMP to configure network mngement progrm to receive SNMP messges tht re sent by the system. This file cn be used with SNMP messges from ll versions of the softwre. More informtion bout the MIB file for SNMP is vilble t this website: Serch for, then serch for MIB. Go to the downlods results to find Mngement Informtion Bse (MIB) file for SNMP. Click this link to find downlod options. Syslog messges The syslog protocol is stndrd protocol for forwrding log messges from sender to receiver on n IP network. The IP network cn be either IPv4 or IPv6. The system cn send syslog messges tht notify personnel bout n event. The system cn trnsmit syslog messges in either expnded or concise formt. You cn use syslog mnger to view the syslog messges tht the system sends. The system uses the User Dtgrm Protocol (UDP) to trnsmit the syslog messge. You cn specify up to mximum of six syslog servers.you cn use the mngement GUI or the commnd-line interfce to configure nd modify your syslog settings. Tble 82 on pge 136 shows how SAN Volume Controller notifiction codes mp to syslog security-level codes. Chpter 7. Dignosing problems 135

154 Tble 82. SAN Volume Controller notifiction types nd corresponding syslog level codes SAN Volume Controller notifiction type Syslog level code Description ERROR LOG_ALERT Fult tht might require hrdwre replcement tht needs immedite ttention. WARNING LOG_ERROR Fult tht needs immedite ttention. Hrdwre replcement is not expected. INFORMATIONAL LOG_INFO Informtion messge used, for exmple, when configurtion chnge tkes plce or n opertion completes. TEST LOG_DEBUG Test messge Tble 83 shows how SAN Volume Controller vlues of user-defined messge origin identifiers mp to syslog fcility codes. Tble 83. SAN Volume Controller vlues of user-defined messge origin identifiers nd syslog fcility codes SAN Volume Controller vlue Syslog vlue Syslog fcility code Messge formt 0 16 LOG_LOCAL0 Full 1 17 LOG_LOCAL1 Full 2 18 LOG_LOCAL2 Full 3 19 LOG_LOCAL3 Full 4 20 LOG_LOCAL4 Concise 5 21 LOG_LOCAL5 Concise 6 22 LOG_LOCAL6 Concise 7 23 LOG_LOCAL7 Concise Cll Home emil The Cll Home feture trnsmits opertionl nd event-relted dt to you nd service personnel through Simple Mil Trnsfer Protocol (SMTP) server connection in the form of n event notifiction emil. When configured, this function lerts service personnel bout hrdwre filures nd potentilly serious configurtion or environmentl issues. To send emil, you must configure t lest one SMTP server. You cn specify s mny s five dditionl SMTP servers for bckup purposes. The SMTP server must ccept the relying of emil from the mngement IP ddress. You cn then use the mngement GUI or the commnd-line interfce to configure the emil settings, including contct informtion nd emil recipients. Set the reply ddress to vlid emil ddress. Send test emil to check tht ll connections nd infrstructure re set up correctly. You cn disble the Cll Home function t ny time using the mngement GUI or the commnd-line interfce. 136 SAN Volume Controller: Troubleshooting Guide

155 Inventory informtion emil Dt tht is sent with notifictions Notifictions cn be sent using emil, SNMP, or syslog. The dt sent for ech type of notifiction is the sme. It includes: v Record type v Mchine type v Mchine seril number v Error ID v Error code v Softwre version v FRU prt number v Cluster (system) nme v Node ID v Error sequence number v Time stmp v Object type v Object ID v Problem dt Emils contin the following dditionl informtion tht llow the Support Center to contct you: v Contct nmes for first nd second contcts v Contct phone numbers for first nd second contcts v Alternte contct numbers for first nd second contcts v Offshift phone number v Contct emil ddress v Mchine loction To send dt nd notifictions to service personnel, use one of the following emil ddresses: v v For systems tht re locted in North Americ, Ltin Americ, South Americ or the Cribben Islnds, use cllhome1@de.ibm.com For systems tht re locted nywhere else in the world, use cllhome0@de.ibm.com An inventory informtion emil summrizes the hrdwre components nd configurtion of system. Service personnel cn use this informtion to contct you when relevnt softwre upgrdes re vilble or when n issue tht cn ffect your configurtion is discovered. It is good prctice to enble inventory reporting. Becuse inventory informtion is sent using the Cll Home emil function, you must meet the Cll Home function requirements nd enble the Cll Home emil function before you cn ttempt to send inventory informtion emil. You cn djust the contct informtion, djust the frequency of inventory emil, or mnully send n inventory emil using the mngement GUI or the commnd-line interfce. Chpter 7. Dignosing problems 137

156 Inventory informtion tht is sent to IBM includes the following informtion bout the clustered system on which the Cll Home function is enbled. Sensitive informtion such s IP ddresses is not included. v Licensing informtion v Detils bout the following objects nd functions: Drives Externl storge systems Hosts MDisks Volumes RAID types Esy Tier FlshCopy Metro Mirror nd Globl Mirror For detiled informtion bout wht is included in the Cll Home inventory informtion, configure the system to send n inventory emil to yourself. Understnding the error codes Error codes re generted by the event-log nlysis nd system configurtion code. Error codes help you to identify the cuse of problem, filing component, nd the service ctions tht might be needed to solve the problem. Note: If more thn one error occurs during n opertion, the highest priority error code displys on the front pnel. The lower the number for the error code, the higher the priority. For exmple, error code 1020 hs higher priority thn error code Using the error code tbles The error code tbles list the vrious error codes nd describe the ctions tht you cn tke. About this tsk Perform the following steps to use the error code tbles: Procedure 1. Locte the error code in one of the tbles. If you cnnot find prticulr code in ny tble, cll IBM Support Center for ssistnce. 2. Red bout the ction you must perform to correct the problem. Do not exchnge field replceble units (FRUs) unless you re instructed to do so. 3. Normlly, exchnge only one FRU t time, strting from the top of the FRU list for tht error code. Event IDs The SAN Volume Controller softwre genertes events, such s informtionl events nd error events. An event ID or number is ssocited with the event nd indictes the reson for the event. 138 SAN Volume Controller: Troubleshooting Guide

157 Informtionl events provide informtion bout the sttus of n opertion. Informtionl events re recorded in the event log, nd, depending on the configurtion, informtionl event notifictions cn be sent through emil, SNMP, or syslog. Error events re generted when service ction is required. An error event mps to n lert with n ssocited error code. Depending on the configurtion, error event notifictions cn be sent through emil, SNMP, or syslog. Informtionl events The informtionl events provide informtion bout the sttus of n opertion. Informtionl events re recorded in the event log nd, bsed on notifiction type, cn be notified through emil, SNMP, or syslog. Informtionl events cn be either notifiction type I (informtion) or notifiction type W (wrning). An informtionl event report of type (W) might require user ttention. Tble 84 provides list of informtionl events, the notifiction type, nd the reson for the event. Tble 84. Informtionl events Notifiction Event ID type Description I Bttery protection unvilble I Bttery protection temporrily unvilble; one bttery is expected to be vilble soon I Bttery protection temporrily unvilble; both btteries re expected to be vilble soon I Bttery cpcity is reduced becuse of cell imblnce I The error log is clered I The SSH key ws discrded for the service login user I User nme hs chnged I Degrded or offline mnged disk is now online I A degrded or offline storge pool is now online I Offline volume is now online W Volume is offline becuse of degrded or offline storge pool I All nodes cn see the port W All ports in this host re now offline I A node hs been successfully dded to the cluster (system) I The node is now functionl member of the cluster (system) I A noncriticl hrdwre error occurred I Attempt to utomticlly recover offline node strting I Both nodes in the I/O group re vilble I One node in the I/O group is unvilble W Both nodes in the I/O group re unvilble. Chpter 7. Dignosing problems 139

158 Tble 84. Informtionl events (continued) Notifiction Event ID type Description I Cluster (system) recovery completed W Filed to obtin directory listing from remote node W Filed to trnsfer file from remote node I The migrtion is complete I The secure delete is complete W The virtuliztion mount is close to the limit tht is licensed W The FlshCopy feture is close to the limit tht is licensed W The Metro Mirror or Globl Mirror feture is close to the limit tht is licensed I Fibre Chnnel discovery occurred; configurtion chnges re pending I Fibre Chnnel discovery occurred; configurtion chnges re complete I Fibre Chnnel discovery occurred; no configurtion chnges were detected W The mnged disk is not on the preferred pth W The initiliztion for the mnged disk filed W The LUN discovery hs filed. The cluster (system) hs connection to device through this node but this node cnnot discover the unmnged or mnged disk tht is ssocited with this LUN W The LUN cpcity equls or exceeds the mximum. Only prt of the disk cn be ccessed W The mnged disk error count wrning threshold hs been met I Mnged disk offline imminent, offline prevention strted I Drive firmwre downlod completed successfully I Drive FPGA downlod completed successfully I Drive firmwre downlod strted I Drive FPGA downlod strted I Drive firmwre downlod cncelled by user I SAS discovery occurred; no configurtion chnges were detected I SAS discovery occurred; configurtion chnges re pending I SAS discovery occurred; configurtion chnges re complete W The LUN cpcity equls or exceeds the mximum cpcity. Only the first 1 PB of disk will be ccessed I The drive formt hs strted I The drive recovery ws strted. 140 SAN Volume Controller: Troubleshooting Guide

159 Tble 84. Informtionl events (continued) Notifiction Event ID type Description W Insufficient virtul extents W The migrtion suspended becuse of insufficient virtul extents or too mny medi errors on the source mnged disk W Migrtion hs stopped I Migrtion is complete W Copied disk I/O medium error I The FlshCopy opertion is prepred I The FlshCopy opertion is complete W The FlshCopy opertion hs stopped W First customer dt being pinned in volume working set I All customer dt in volume working set is now unpinned W The volume working set cche mode is in the process of chnging to synchronous destge becuse the volume working set hs too much pinned dt I Volume working set cche mode updted to llow synchronous destge becuse enough customer dt hs been unpinned for the volume working set I The debug from n IERR ws extrcted to disk I An ttempt ws mde to power on the slots I All the expnders on the strnd were reset I The component firmwre updte pused to llow the bttery chrging to finish I The updte for the component firmwre pused becuse the system ws put into mintennce mode I A component firmwre updte is needed but is prevented from running I The Metro Mirror or Globl Mirror bckground copy is complete I The Metro Mirror or Globl Mirror is redy to restrt W Unble to find pth to disk in the remote cluster (system) within the timeout period W The thin-provisioned volume copy dt in node is pinned I All thin-provisioned volume copy dt in node is unpinned I The thin-provisioned volume copy import hs filed nd the new volume is offline; either upgrde the SAN Volume Controller softwre to the required version or delete the volume I The thin-provisioned volume copy import is successful. Chpter 7. Dignosing problems 141

160 Tble 84. Informtionl events (continued) Event ID Notifiction type Description W A thin-provisioned volume copy spce wrning hs occurred I A thin-provisioned volume copy repir hs strted I A thin-provisioned volume copy repir is successful I A thin-provisioned volume copy vlidtion is strted I A thin-provisioned volume copy vlidtion is successful I The import of the compressed-virtul volume copy ws successful W A compressed-virtul volume copy spce wrning hs occurred I A compressed-virtul volume copy repir hs strted I A compressed-virtul volume copy repir is successful I A compressed-virtul volume copy hs too mny bd blocks I A medium error hs been repired for the mirrored copy W A mirror copy repir, using the vlidte option cnnot complete I A mirror disk repir is complete nd no differences re found I A mirror disk repir is complete nd the differences re resolved W A mirror disk repir is complete nd the differences re mrked s medium errors I The mirror disk repir hs been strted W A mirror copy repir, using the set medium error option, cnnot complete W A mirror copy repir, using the resync option, cnnot complete W Node coldstrted W A node power-off hs been requested from the power switch I Additionl Fibre Chnnel ports were connected W The connection to configured remote cluster (system) hs been lost W The node unexpectedly lost power but hs now been restored to the cluster (system) W An overnight mintennce procedure hs filed to complete. Resolve ny hrdwre nd configurtion problems tht you re experiencing on the cluster (system). If the problem persists, contct your IBM service representtive for ssistnce W An rry MDisk is offline becuse it hs too mny missing members. 142 SAN Volume Controller: Troubleshooting Guide

161 Tble 84. Informtionl events (continued) Notifiction Event ID type Description I The rebuild for n rry MDisk ws strted I The rebuild for n rry MDisk hs finished I A RAID rry hs strted exchnging n rry member I A RAID rry hs completed exchnging n rry member I A RAID rry needs resynchroniztion W A storge pool spce wrning hs occurred. SCSI event reporting Nodes cn notify their hosts of events for SCSI commnds tht re issued. SCSI sttus Some events re prt of the SCSI rchitecture nd re hndled by the host ppliction or device drivers without reporting n event. Some events, such s red nd write I/O events nd events tht re ssocited with the loss of nodes or loss of ccess to bckend devices, cuse ppliction I/O to fil. To help troubleshoot these events, SCSI commnds re returned with the Check Condition sttus nd 32-bit event identifier is included with the sense informtion. The identifier reltes to specific event in the event log. If the host ppliction or device driver cptures nd stores this informtion, you cn relte the ppliction filure to the event log. Tble 85 describes the SCSI sttus nd codes tht re returned by the nodes. Tble 85. SCSI sttus Sttus Code Description Good 00h The commnd ws successful. Check condition 02h The commnd filed nd sense dt is vilble. Condition met 04h N/A Busy 08h An Auto-Contingent Allegince condition exists nd the commnd specified NACA=0. Intermedite 10h N/A Intermedite - condition 14h N/A met Reservtion conflict 18h Returned s specified in SPC2 nd SAM-2 where reserve or persistent reserve condition exists. Tsk set full 28h The inititor hs t lest one tsk queued for tht LUN on this port. ACA ctive 30h This code is reported s specified in SAM-2. Tsk borted 40h This code is returned if TAS is set in the control mode pge 0Ch. The node hs defult setting of TAS=0, which cnnot be chnged; therefore, the node does not report this sttus. Chpter 7. Dignosing problems 143

162 SCSI Sense Nodes notify the hosts of events on SCSI commnds. Tble 86 defines the SCSI sense keys, codes nd qulifiers tht re returned by the nodes. Tble 86. SCSI sense keys, codes, nd qulifiers Key Code Qulifier Definition Description 2h 04h 01h Not Redy. The logicl unit is in the process of becoming redy. 2h 04h 0Ch Not Redy. The trget port is in the stte of unvilble. The node lost sight of the system nd cnnot perform I/O opertions. The dditionl sense does not hve dditionl informtion. The following conditions re possible: v The node lost sight of the system nd cnnot perform I/O opertions. The dditionl sense does not hve dditionl informtion. v The node is in contct with the system but cnnot perform I/O opertions to the specified logicl unit becuse of either loss of connectivity to the bckend controller or some lgorithmic problem. This sense is returned for offline volumes. 3h 00h 00h Medium event This is only returned for red or write I/Os. The I/O suffered n event t specific LBA within its scope. The loction of the event is reported within the sense dt. The dditionl sense lso includes reson code tht reltes the event to the corresponding event log entry. For exmple, RAID controller event or migrted medium event. 4h 08h 00h Hrdwre event. A commnd to logicl unit communiction filure hs occurred. 5h 25h 00h Illegl request. The logicl unit is not supported. The I/O suffered n event tht is ssocited with n I/O event tht is returned by RAID controller. The dditionl sense includes reson code tht points to the sense dt tht is returned by the controller. This is only returned for I/O type commnds. This event is lso returned from FlshCopy trget volumes in the prepred nd prepring stte. The logicl unit does not exist or is not mpped to the sender of the commnd. 144 SAN Volume Controller: Troubleshooting Guide

163 Reson codes The reson code ppers in bytes of the sense dt. The reson code provides the node with specific log entry. The field is 32-bit unsigned number tht is presented with the most significnt byte first. Tble 87 lists the reson codes nd their definitions. If the reson code is not listed in Tble 87, the code refers to specific event in the event log tht corresponds to the sequence number of the relevnt event log entry. Tble 87. Reson codes Reson code (deciml) Description 40 The resource is prt of stopped FlshCopy mpping. 50 The resource is prt of Metro Mirror or Globl Mirror reltionship nd the secondry LUN in the offline. 51 The resource is prt of Metro Mirror or Globl Mirror nd the secondry LUN is red only. 60 The node is offline. 71 The resource is not bound to ny domin. 72 The resource is bound to domin tht hs been recreted. 73 Running on node tht hs been contrcted out for some reson tht is not ttributble to ny pth going offline. 80 Wit for the repir to complete, or delete the volume. 81 Wit for the vlidtion to complete, or delete the volume. 82 An offline thin-provisioned volume hs cused dt to be pinned in the directory cche. Adequte performnce cnnot be chieved for other thin-provisioned volumes, so they hve been tken offline. 85 The volume hs been tken offline becuse checkpointing to the quorum disk filed. 86 The repirvdiskcopy -medium commnd hs creted virtul medium error where the copies differed. 93 An offline RAID-5 or RAID-6 rry hs cused in-flight-write dt to be pinned. Good performnce cnnot be chieved for other rrys nd so they hve been tken offline. 94 An rry MDisk tht is prt of the volume hs been tken offline becuse checkpointing to the quorum disk filed. 95 This reson code is used in MDisk bd block dump files to indicte the dt loss ws cused by hving to resync prity with rebuilding strips or some other RAID lgorithm reson due to multiple filures. 96 A RAID-6 rry MDisk tht is prt of the volume hs been tken offline becuse n internl metdt tble is full. Object types You cn use the object code to determine the type of the object the event is logged ginst. Tble 88 on pge 146 lists the object codes nd corresponding object types. Chpter 7. Dignosing problems 145

164 Tble 88. Object types Object code Object type 1 mdisk 2 mdiskgrp 3 volume 4 node 5 host 7 iogroup 8 fcgrp 9 rcgrp 10 fcmp 11 rcmp 12 wwpn 13 cluster (system) 16 device 17 SCSI lun 18 quorum 34 Fibre Chnnel dpter 38 volume copy 39 Syslog server 40 SNMP server 41 Emil server 42 User group 44 Cluster (mngement) IP 46 SAS dpter Error event IDs nd error codes Error codes describe service procedure tht must be followed. Ech event ID tht requires service hs n ssocited error code. Error codes cn be either notifiction type E (error) or notifiction type W (wrning). Tble 89 lists the event IDs nd corresponding error codes, the notifiction type, nd the condition of the event. Tble 89. Error event IDs nd error codes Event ID Notifiction type Condition Error code E An utomtic system recovery hs strted. All 1001 configurtion commnds re blocked E The error event log is full W The following cuses re possible: 1196 v The node is missing. v The node is no longer functionl member of the system E A node hs been missing for 30 minutes SAN Volume Controller: Troubleshooting Guide

165 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition Error code W The softwre instll process hs filed W The softwre upgrde pckge delivery hs filed W Unble to connect to the SMTP (emil) server W Unble to send mil through the SMTP (emil) server W The Metro Mirror or Globl Mirror feture cpcity is 3030 not set W The FlshCopy feture cpcity is not set W The Virtuliztion feture hs exceeded the mount tht 3032 is licensed W The FlshCopy feture hs exceeded the mount tht is 3032 licensed W The Metro Mirror or Globl Mirror feture hs exceeded 3032 the mount tht is licensed W The usge for the thin-provisioned volume is not 3033 licensed W The vlue set for the virtuliztion feture cpcity is not 3029 vlid E A physicl disk FlshCopy feture license is required E A physicl disk Metro Mirror nd Globl Mirror feture 3036 license is required E A virtuliztion feture license is required E Automtic recovery of offline node filed W Unble to send emil to ny of the configured emil 3081 servers W The externl virtuliztion feture license limit ws 3032 exceeded W Unble to connect to LDAP server W The LDAP configurtion is not vlid E The limit for the compression feture license ws 3032 exceeded E The limit for the compression feture license ws 3032 exceeded E Unble to connect to LDAP server tht hs been 2256 utomticlly configured E Invlid LDAP configurtion for utomticlly configured 2255 server W A licensble feture's tril-timer hs reched 0. The 3082 feture hs now been dectivted W A tril of licensble feture will expire in 5 dys W A tril of licensble feture will expire in 10 dys W A tril of licensble feture will expire in 15 dys W A tril of licensble feture will expire in 45 dys W Esy Tier feture license limit exceeded Chpter 7. Dignosing problems 147

166 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition Error code W FlshCopy feture license limit exceeded W Externl virtuliztion feture license limit exceeded W Remote copy feture license limit exceeded E The node rn out of bse event sources. As result, the 2030 node hs stopped nd exited the system W The number of device logins hs reduced E A softwre error hs occurred E The block size is invlid, the cpcity or LUN identity 1660 hs chnged during the mnged disk initiliztion E The mnged disk is excluded becuse of excessive 1310 errors E The remote port is excluded for mnged disk nd 1220 node E The locl port is excluded E The login is excluded E A timeout hs occurred s result of excessive 1340 processing time E An error recovery procedure hs occurred E A mnged disk I/O error hs occurred E The mnged disk error count threshold hs exceeded W There re too mny devices presented to the clustered 1200 system W There re too mny mnged disks presented to the 1200 cluster (system) W There re too mny LUNs presented to node W There re too mny drives presented to cluster 1200 (system) W A disk I/O medium error hs occurred W A suitble MDisk or drive for use s quorum disk ws 1330 not found W The quorum disk is not vilble W A controller configurtion is not supported E A login trnsport fult hs occurred E A mnged disk error recovery procedure (ERP) hs occurred. The node or controller reported the following: 1370 v Sense v Key v Code v Qulifier E One or more MDisks on controller re degrded W The controller configurtion limits filover E The controller configurtion uses the RDAC mode; this is not supported SAN Volume Controller: Troubleshooting Guide

167 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition Error code E Persistent unsupported controller configurtion E The controller system device is only connected to the 1627 node through single inititor port E The controller system device is only connected to the 1627 node through single trget port E The controller system device is only connected to the 1627 clustered system nodes through single trget port E The controller system device is only connected to the clustered system nodes through hlf of the expected trget ports E The controller system device hs disconnected ll trget 1627 ports to the clustered system nodes W An unrecognized SAS device E SAS error counts exceeded the wrning thresholds E SAS errors exceeded criticl thresholds W Controller indictes tht it does not support descriptor 1625 sense for LUNs tht re greter thn 2 TBs W Too mny enclosures were presented to cluster 1200 (system) W Too mny controller trget ports were presented to the 1200 cluster (system) W Too mny trget ports were presented to the clustered 1200 system from single controller W There re too mny drives presented to cluster 1200 (system) W Reserved SAS port hs devices ttched E Too mny long IOs to drive E A drive is reported s continous slow with contributory 1215 fctors E Too mny long IOs to drive (Mercury drives) E A drive is reported s continous slow with contributory 1215 fctors (Mercury drives) W Drive firmwre downlod cncelled becuse of system 3090 chnges W Drive firmwre downlod cncelled becuse of drive 3090 downlod problem W Too mny drives ttched to the system E There re too mny medium errors on the mnged disk E A storge pool is offline W There re insufficient virtul extents W The mnged disk hs bd blocks. On n externl 1840 controller, this cn only be copied medium error E The system filed to crete bd block becuse MDisk lredy hs the mximum number of llowed bd blocks Chpter 7. Dignosing problems 149

168 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition E The system filed to crete bd block becuse the clustered system lredy hs the mximum number of llowed bd blocks W The trigger prepre commnd hs filed becuse of cche flush filure W The mpping is stopped becuse of the error tht is indicted in the dt W The mpping is stopped becuse of clustered system or complete I/O group filure, nd the current stte of the reltionship could not be recovered W The reltionship is stopped becuse of clustered system or complete I/O group filure, nd the current stte of the mpping could not be recovered W A Metro Mirror or Globl Mirror reltionship or consistency group exists within clustered system, but its prtnership hs been deleted. Error code W A Globl Mirror reltionship hs stopped becuse of 1920 persistent I/O error W A remote copy hs stopped becuse of persistent I/O 1915 error W Remote copy hs stopped W There re too mny clustered system prtnerships. The 1710 number of prtnerships hs been reduced W There re too mny clustered system prtnerships. The 1710 system hs been excluded W Bckground copy process for the Remote Copy ws 1960 blocked W The Globl Mirror secondry volume is offline. The reltionship hs pinned hrdened write dt for this volume W The Globl Mirror secondry volume is offline due to missing IO group prtner node. The reltionship hs pinned hrdened write dt for this volume but the node contining the required dt is currently offline W Globl Mirror performnce is likely to be impcted. A lrge mount of pinned dt for the offline volumes hs reduced the resource vilble to the globl mirror secondry disks W The thin-provisioned volume copy is offline becuse there is insufficient spce W The thin-provisioned volume copy is offline becuse the metdt is corrupt W The thin-provisioned volume copy is offline becuse the repir hs filed W The compressed volume copy is offline becuse there is insufficient spce W The compressed volume copy is offline becuse the metdt is corrupt SAN Volume Controller: Troubleshooting Guide

169 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition Error code W The compressed volume copy is offline becuse the 1860 repir hs filed W The compressed volume copy hs bd blocks W Unble to mirror medium error during volume copy 1950 synchroniztion W The mirrored volume is offline becuse the dt cnnot 1870 be synchronized W The repir process for the mirrored disk hs stopped 1600 becuse there is difference between the copies E Unrecognized node error E Detected memory size does not mtch the expected 1022 memory size E The WWNN tht is stored on the service controller nd 1192 the WWNN tht is stored on the drive do not mtch E Unble to detect ny Fibre Chnnel dpter E The system bord processor hs filed W The internl disk file system of the node is dmged E Unble to updte BIOS settings E Unble to updte the service processor firmwre for the 1020 system bord W The mbient temperture is too high while the system is 1182 strting E Cnnot form clustered system due to lck of resources E Duplicte WWNN detected on the SAN E A node is unble to communicte with other nodes E Bttery cbling fult E Bttery bckplne or cbling fult E The node hrdwre does not meet minimum 1183 requirements E Too mny softwre filures E The internl drive of the node is filing E The node softwre is dmged E The clustered system dt cnnot be red E The clustered system dt ws not sved when power 1194 ws lost E Bttery subsystem hs insufficient chrge to sve system 1107 dt E Unble to red the service controller ID E 2145 UPS-1U seril link error E 2145 UPS-1U bttery error E 2145 UPS-1U electronics error E 2145 UPS-1U overloded Chpter 7. Dignosing problems 151

170 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition Error code E 2145 UPS-1U filure E Power supply to 2145 UPS-1U does not meet 1141 requirements E Incorrect type of uninterruptible power supply detected E 2145 UPS-1U is not cbled correctly E The mbient temperture limit for the 2145 UPS-1U ws 1136 exceeded E Repeted node restrts becuse of 2145 UPS-1U errors W Insufficient uninterruptible power supply chrge to llow 1193 node to strt W Node held in service stte E High-speed SAS dpter is missing. This error pplies to 1120 only the SAN Volume Controller 2145-CG8 model E Ethernet dpter is missing. This error pplies to only 1072 the SAN Volume Controller 2145-CG8 model W Technicin port connection invlid E Bttery is missing E Bttery hs filed E Bttery is below the minimum operting temperture E Bttery is bove the mximum operting temperture E Bttery communictions error E Bttery is nering end of life E Bttery VPD checksum error E Bttery is t hrdwre revision level not supported by 1473 the current code level E Bttery protection temporrily unvilble; both btteries 1473 re expected to be vilble soon E Bttery subsystem hs insufficient chrge to sve system 1473 dt W Detected hrdwre is not vlid configurtion W Detected hrdwre needs ctivtion E A CMOS bttery filure hs occurred. This error pplies 1670 to only the SAN Volume Controller G4 model E A CMOS bttery filure hs occurred. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E The Fibre Chnnel ports re not opertionl E Clustered system pth filure W The SAN is not correctly zoned. As result, more thn 512 ports on the SAN hve logged into one SAN Volume Controller port E The 4-port Fibre Chnnel dpter in slot 1 is missing. This error pplies to only the SAN Volume Controller G4 model SAN Volume Controller: Troubleshooting Guide

171 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition E The 4-port Fibre Chnnel dpter in slot 1 hs filed. This error pplies to only the SAN Volume Controller G4 model E The 4-port Fibre Chnnel dpter in slot 1 hs detected PCI bus error. This error pplies to only the SAN Volume Controller G4 model E The 4-port Fibre Chnnel dpter in slot 1 hs detected PCI bus error. This error pplies to only the SAN Volume Controller A4 model E The 4-port Fibre Chnnel dpter in slot 1 hs detected PCI bus error. This error pplies to only the SAN Volume Controller A4 model E The 4-port Fibre Chnnel dpter in slot 1 hs detected PCI bus error. This error pplies to only the SAN Volume Controller A4 model E The 4-port Fibre Chnnel dpter in slot 1 hs detected PCI bus error. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E The 4-port Fibre Chnnel dpter in slot 1 hs detected PCI bus error. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E The 4-port Fibre Chnnel dpter in slot 1 hs detected PCI bus error. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models W One or more Fibre Chnnel ports re running t speed tht is lower thn the lst sved speed E A duplicte Fibre Chnnel frme hs been detected, which indictes tht there is n issue with the Fibre Chnnel fbric. Other Fibre Chnnel errors might lso be generted. Error code E Enclosure VPD is inconsistent E The system bord service processor hs filed E Ambient temperture is too high during system strtup W Unble to determine the vitl product dt (VPD) for n FRU. This is probbly becuse new FRU hs been instlled nd the softwre does not recognize tht FRU. The clustered system continues to operte; however, you must upgrde the softwre to fix this wrning E The node wrm strted fter softwre error W A connection to configured remote system hs been 1715 lost becuse of connectivity problem W A connection to configured remote system hs been 1716 lost becuse of too mny minor errors E The flsh boot device hs filed. This error pplies to only the SAN Volume Controller G4 model Chpter 7. Dignosing problems 153

172 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition E The flsh boot device hs recovered. This error pplies to only the SAN Volume Controller G4 model E A service controller red filure hs occurred. This error pplies to only the SAN Volume Controller G4 model E The flsh boot device hs filed. This error pplies to only the SAN Volume Controller A4 model E The flsh boot device hs recovered. This error pplies to only the SAN Volume Controller A4 model E A service controller red filure hs occurred. This error pplies to only the SAN Volume Controller A4 model E The flsh boot device hs filed. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E The flsh boot device hs recovered. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E A service controller red filure hs occurred. This error pplies to only the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models. Error code E The internl disk for node hs filed E The hrd disk is full nd cnnot cpture ny more 2030 output E One of the two power supply units in the node hs 1096 filed E One of the two power supply units in the node cnnot 1096 be detected E One of the two power supply units in the node is 1097 without power E A high-speed SAS dpter is missing. This error pplies 1120 to only the SAN Volume Controller 2145-CF8 model E Degrded PCIe lnes on high-speed SAS dpter E A PCI bus error occurred on high-speed SAS dpter E A high-speed SAS dpter requires PCI bus reset E Vitl product dt (VPD) is corrupt on high-speed SAS 1121 dpter E The service processor shows fn filure. This error pplies to only the SAN Volume Controller G4 model E The service processor shows fn filure. This error pplies to only the SAN Volume Controller A4 model E The service processor shows fn filure. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models SAN Volume Controller: Troubleshooting Guide

173 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition E The node mbient temperture threshold hs exceeded. This error pplies to only the SAN Volume Controller G4 model E The node processor wrning temperture threshold hs exceeded. This error pplies to only the SAN Volume Controller G4 model E The node processor or mbient criticl threshold hs exceeded. This error pplies to only the SAN Volume Controller G4 model E The node mbient temperture threshold hs exceeded. This error pplies to only the SAN Volume Controller A4 model E The node processor wrning temperture threshold hs exceeded. This error pplies to only the SAN Volume Controller A4 model E The node processor or mbient criticl threshold hs exceeded. This error pplies to only the SAN Volume Controller A4 model E System bord - ny voltge high. This error pplies to only the SAN Volume Controller G4 model E System bord - ny voltge high. This error pplies to only the SAN Volume Controller A4 model E System bord - ny voltge high. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E System bord - ny voltge low. This error pplies to only the SAN Volume Controller G4 model E System bord - ny voltge low. This error pplies to only the SAN Volume Controller A4 model E System bord - ny voltge low. This error pplies to only the SAN Volume Controller 2145-CF8 model E A power mngement bord voltge filure hs occurred. This error pplies to only the SAN Volume Controller G4 model E The node mbient temperture threshold hs exceeded. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E The node processor wrning temperture threshold hs exceeded. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E The node processor or mbient criticl threshold hs exceeded. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models. Error code Chpter 7. Dignosing problems 155

174 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition E A power mngement bord voltge filure hs occurred. This error pplies to the SAN Volume Controller 2145-CF8 nd the SAN Volume Controller 2145-CG8 models E A power domin error hs occurred. Both nodes in pir re powered by the sme uninterruptible power supply. Error code W The limit on the number of clustered system secure shell 2500 (SSH) sessions hs been reched W Unble to ccess the Network Time Protocol (NTP) 2700 network time server I Technicin port connection inctive I Technicin port connection ctive E An Ethernet port filure hs occurred E A server error hs occurred E An uninterruptible power supply communictions filure hs occurred. The RS232 connection between node nd its uninterruptible power supply is fulty. This error pplies to only the 2145 UPS-1U model E The uninterruptible power supply output is unexpectedly high. The uninterruptible power supply is probbly connected to non-san Volume Controller lod. This error pplies to only the 2145 UPS-1U model E The uninterruptible power supply bttery hs reched end of life. This error pplies to only the 2145 UPS-1U model E An uninterruptible power supply bttery filure hs occurred. This error pplies to only the 2145 UPS-1U model E An uninterruptible power supply electronics filure hs occurred. This error pplies to only the 2145 UPS-1U model E Uninterruptible power supply overcurrent. This error pplies to only the 2145 UPS-1U model E An uninterruptible power supply filure hs occurred. This error pplies to only the 2145 UPS-1U model E Uninterruptible power supply c input power fult. This error pplies to only the 2145 UPS-1U model E An uninterruptible power supply configurtion error hs occurred. This error pplies to only the 2145 UPS-1U model E Uninterruptible power supply mbient over temperture. This error pplies to only the 2145 UPS-1U model E Uninterruptible power supply over temperture wrning. This error pplies to only the 2145 UPS-1U model E An uninterruptible power supply softwre error hs occurred. This error pplies to only the 2145 UPS-1U model SAN Volume Controller: Troubleshooting Guide

175 Tble 89. Error event IDs nd error codes (continued) Event ID Notifiction type Condition Error code W An rry MDisk hs deconfigured members nd hs lost 1689 redundncy W An rry MDisk is corrupt becuse of lost metdt W Arry MDisk hs tken spre member tht does not 1692 mtch rry gols W An rry hs members tht re locted in different I/O 1688 group W An rry MDisk is no longer protected by n pproprite 1690 number of suitble spres W No spre protection exists for one or more rry MDisks W An rry MDisk is offline. The metdt for the inflight 1243 writes is on missing node W An rry MDisk is offline. Metdt on the missing node 1243 contins needed stte informtion W An rry hs lost redundncy due to slow write by member drive Resolving problem with the SAN Volume Controller 2145-DH8 boot drives Follow these resolution steps to resolve most problems with boot drives. Before you begin The node seril number (lso known s the product or mchine seril number) is on the MT-M SN lbel (Mchine Type - Model nd Seril Number lbel) on the front (left side) of the 2145-DH8 node. The node seril number is written to the system bord nd to ech of the two boot drives during the mnufcturing process. When the SAN Volume Controller softwre strts, it reds the node seril number from the system bord (using the node seril number for the pnel nme) nd compres it with the node seril numbers stored on the two boot drives. Specific node errors re produced under the following conditions: v Ftl node error 543: When none of the node seril numbers tht re stored in the three loctions mtch. The node seril number from the system bord must mtch with t lest one of the two boot drives for the SAN Volume Controller softwre to ssume tht node seril number is good. v Ftl node error 545: If the node seril numbers on ech boot drive mtch ech other but re not the sme s the node seril number from the system bord. In this cse, the node seril number on the system bord might be wrong or the node seril number on the boot drives might be wrong. For exmple, the system bord chnged or the boot drives come from nother 2145-DH8 node. v Node error 743: If the node seril number cnnot be red from one of the two boot drives becuse tht drive filed, is missing, or is out of sync with the other boot drive. Chpter 7. Dignosing problems 157

176 v Node error 744: If the node seril number form one of the boot drives identifies s belonging to different 2145-DH8 node. If boot drives were swpped between drive slots 1 nd 2, node error 744 is produced. v Node error 745: If boot drive is found in n unsupported slot. Drives slots 3-8 re not supported by version 7.3 SAN Volume Controller softwre. About this tsk There is n event in the Monitoring > Events pnel of the mngement GUI if the problem produces node error 743, 744, or 745. Run the fix procedure for tht event. Otherwise, connect to the technicin port to use the MT-M SN lbel on the node to see the boot drive slot informtion nd determine the problem. Attention: If drive slot hs Yes in the Active column, the operting system depends on tht drive. Do not remove tht drive without first shutting down the node. v Do not swp boot drives between slots. v Ech boot drive hs copy of the VPD on the system bord. v Softwre upgrding is to one boot drive t time to prevent filures during CCU. Procedure To resolve problem with boot drive, complete the following steps in order: 1. Remove ny drive tht is in n unsupported slot. Move the drive to the correct slot if you cn. 2. If possible, replce ny drive tht is shown s missing from slot. Otherwise, reset the drive or replce it with drive from FRU stock. 3. Move ny drive tht is in the wrong node bck to the correct node. Note: If the node seril number does not mtch the node seril number on the system bord, drive slot hs sttus of wrong_node. You cn ignore this sttus if the seril number on the MT-M SN lbel mtches the node seril number on the drive. 4. Move ny drive tht is in the wrong slot bck to the correct slot. 5. Reset the drive in ny slot tht hs sttus of filed. If the sttus remins filed, replce the drive with one from FRU stock. 6. If the drive slot hs sttus out of sync nd Yes in the cn_sync column, then: v Use the service ssistnt GUI to synchronize boot drives, or v Use the commnd-line interfce (CLI) commnd stsk chbootdrive -sync. v If No is displyed in the cn_sync column, you must resolve nother boot drive problem first. Replcing the 2145-DH8 system bord: 7. Replce the SAN Volume Controller 2145-DH8 system bord. When neither of the boot drives hve usble SAN Volume Controller softwre: For exmple, if you replce both of the boot drives from FRU stock t the sme time, neither boot drive hs usble SAN Volume Controller softwre. If the SAN Volume Controller softwre is not running, the node sttus, node fult, bttery sttus, nd bttery fult LEDs remin off. 8. If you cnnot replce t lest one of the originl boot drives with drive tht contins usble SAN Volume Controller softwre nd hs node seril number 158 SAN Volume Controller: Troubleshooting Guide

177 tht mtches the MT-M SN lbel on the front of the node, contct IBM Remote Technicl support. IBM Remote Technicl support cn help you instll the SAN Volume Controller softwre with bootble USB flsh drive. v Field-bsed USB instlltion lso repirs the node seril number nd WWNN stored on ech boot drive by finding vlues tht re stored on the system bord during mnufcturing. v If the WWNN of this node chnged in the pst, you must chnge the WWNN gin fter completing the SAN Volume Controller softwre instlltion. For exmple, if the node replced legcy SAN Volume Controller node, you would hve chnged the WWNN to tht of the legcy node. You cn repet the chnge to the WWNN fter the SAN Volume Controller softwre instlltion with the service ssistnt GUI or by commnd. When every copy of the node seril number is lost: For exmple, if you replce the system bord nd both of the boot drives with FRU stock t the sme time, every copy of the node seril number is lost. 9. If you cnnot replce one of the originl boot drives or the originl system bord so tht t lest one copy of the originl node seril number is present, you cnnot repir the node in the field. You must return the node to IBM for repir. Results The sttus of drive slot is uninitilized only if the SAN Volume Controller softwre might not utomticlly initilize the FRU drive. This sttus cn hppen if the node seril number on the other boot drive does not mtch the node seril number on the system bord. If the node seril number on the other boot drive mtches the MT-M SN lbel on the front tht is left of the node, you cn rescue the uninitilized boot drive from the other boot drive sfely. Use the service ssistnt GUI or the stsk recuenode commnd to rescue the drive. Determining hrdwre boot filure During the hrdwre boot, you see progress messges, if the model hs front pnel disply. If the model does not hve front pnel disply, light pth LEDs indicte hrdwre boot filure. Before you begin Line 1 of the front pnel displys the messge Booting tht is followed by the boot code. Line 2 of the disply shows boot progress indictor. If the boot code detects n error tht mkes it impossible to continue, Filed is displyed. You cn use the code to isolte the fult. The following figure shows n exmple of hrdwre boot disply. Filed 120 Figure 57. Exmple of boot error code Chpter 7. Dignosing problems 159

178 About this tsk If the boot detects sitution where it cnnot continue, it fils. The cuse might be tht the softwre on the hrd disk drive is missing or dmged. If possible, the boot sequence lods nd strts the SAN Volume Controller softwre. Any fults tht re detected re reported s node error. Perform the following steps to determine boot filure: Procedure 1. Attempt to restore the softwre by using the node rescue procedure. 2. If node rescue fils, perform the ctions tht re described for ny filing node-rescue code or procedure. Boot code reference Boot codes re displyed on the screen when node is booting. The codes indicte the progress of the boot opertion. Line 1 of the front pnel displys the messge Booting tht is followed by the boot code. Line 2 of the disply shows boot progress indictor. Figure 58 provides view of the boot progress disply. Booting 130 Figure 58. Exmple of boot progress disply Node error code overview Node error codes describe filures tht relte to specific node. Node rescue codes re displyed on the front pnel disply during node rescue. A 2145-DH8 node does not hve front pnel disply nd does not hve node rescue codes. Use the service ssistnt GUI by the Technicin port to view node errors on node tht does not hve front pnel disply such s 2145-DH8 node. Becuse node errors re specific to node, for exmple, memory filures, the errors might be reported only on tht node. However, if the node cn communicte with the configurtion node, then it is reported in the system event log. When the node error code indictes tht criticl error ws detected tht prevents the node from becoming member of clustered system, the Node fult LED is on for the 2145-DH8 node, or Line 1 of the front pnel disply contins the messge Node Error. Line 2 contins either the error code or the error code nd dditionl dt. In errors tht involve node with more thn one power supply, the error code is followed by two numbers. The first number indictes the power supply tht hs problem (either 1 or 2).The second number indictes the problem tht is detected. Figure 59 on pge 161 provides n exmple of node error code. This dt might exceed the mximum width of the menu screen. You cn press the Right 160 SAN Volume Controller: Troubleshooting Guide

179 nvigtion to scroll the disply. Figure 59. Exmple of displyed node error code The dditionl dt is unique for ny error code. It provides the necessry informtion to isolte the problem in n offline environment. Exmples of dditionl dt re disk seril numbers nd field replceble unit (FRU) loction codes. When these codes re displyed, you cn do dditionl fult isoltion by browsing the defult menu to determine the node nd Fibre Chnnel port sttus. There re two types of node errors: criticl node errors nd noncriticl node errors. Criticl errors A criticl error mens tht the node is not ble to prticipte in clustered system until the issue tht is preventing it from joining clustered system is resolved. This error occurs becuse prt of the hrdwre filed or the system detects tht the softwre is corrupted. If node hs criticl node error, it is in service stte, nd the fult LED on the node is on. The exception is when the node cnnot connect to enough resources to form clustered system. It shows criticl node error but is in the strting stte. Resolve the errors in priority order. The rnge of errors tht re reserved for criticl errors re Noncriticl errors A noncriticl error code is logged when hrdwre or code filure tht is relted to just one specific node. These errors do not stop the node from entering ctive stte nd joining clustered system. If the node is prt of clustered system, n lert describes the error condition. The rnge of errors tht re reserved for noncriticl errors re Node rescue codes To strt node rescue, press nd hold the left nd right buttons on the front pnel during power-on cycle. The menu screen displys the Node rescue request. See the node rescue request topic. The hrd disk is formtted nd, if the formt completes without error, the softwre imge is downloded from ny vilble node. During node recovery, Line 1 of the menu screen displys the messge Booting followed by one of the node rescue codes. Line 2 of the menu screen displys boot progress indictor. Figure 60 shows n exmple of displyed node rescue code. Booting 300 Figure 60. Exmple of node-rescue error code The three-digit code tht is shown in Figure 60 represents node rescue code. Note: The 2145 UPS-1U does not power off following node rescue filure. Chpter 7. Dignosing problems 161

180 Clustered-system code overview The error codes for creting clustered system re displyed on the menu screen when you re using the front pnel to crete new system, but the crete opertion fils. Recovery codes for clustered systems indicte tht criticl softwre error hs occurred tht might corrupt your system. Error codes for clustered systems describe errors other thn cretion nd recovery errors. Ech error-code topic includes n error code number, description, ction, nd possible field-replceble units (FRUs). Error codes for creting clustered system Figure 61 provides n exmple of crete error code. Figure 61. Exmple of crete error code for clustered system Line 1 of the menu screen contins the messge Crete Filed. Line 2 shows the error code nd, where necessry, dditionl dt. Error codes for recovering clustered system You must perform softwre problem nlysis before you cn perform further opertions to void the possibility of corrupting your configurtion. Figure 62 provides n exmple of recovery error code. Figure 62. Exmple of recovery error code Error codes for clustered systems Error codes for clustered systems describe errors other thn recovery errors. Figure 63 provides n exmple of clustered-system error code. svc00433 Figure 63. Exmple of n error code for clustered system Error code rnge This topic shows the number rnge for ech messge clssifiction. Tble 90 on pge 163 lists the number rnge for ech messge clssifiction. 162 SAN Volume Controller: Troubleshooting Guide

181 Tble 90. Messge clssifiction number rnge Messge clssifiction Rnge Booting codes Node errors Node rescue errors Log-only node errors Criticl node errors Noncriticl node errors Error codes when creting 700, 710 clustered system Error codes when recovering clustered system 920, 990 Error codes for clustered system Boot is running Explntion: The node hs strted. It is running dignostics nd loding the runtime code. User response: Go to the hrdwre boot MAP to resolve the problem. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Service controller (47%) v Service controller cble (47%) v System bord ssembly (6%) G4 or A4 v Service controller (95%) v System bord (5%) 120 Disk drive hrdwre error Explntion: The internl disk drive of the node hs reported n error. The node is unble to strt. User response: Ensure tht the boot disk drive nd ll relted cbling is properly connected, then exchnge the FRU for new FRU. (See Possible Cuse-FRUs or other. ) Possible Cuse-FRUs or other: 2145-CF8 or 2145-CG8 v Disk drive (50%) v Disk controller (30%) v Disk bckplne (10%) v Disk signl cble (8%) v Disk power cble (1%) v System bord (1%) G4 or A4 v Disk drive ssembly (95%) v Disk cble ssembly (4%) v System bord (1%) 130 Checking the internl disk file system Explntion: The file system on the internl disk drive of the node is being checked for inconsistencies. User response: If the progress br hs been stopped for t lest five minutes, power off the node nd then power on the node. If the boot process stops gin t this point, run the node rescue procedure. Possible Cuse-FRUs or other: v None. 132 Updting BIOS settings of the node Explntion: The system hs found tht chnges re required to the BIOS settings of the node. These chnges re being mde. The node will restrt once the chnges re complete. User response: If the progress br hs stopped for more thn 10 minutes, or if the disply hs shown codes 100 nd 132 three times or more, go to MAP 5900: Hrdwre boot to resolve the problem. 135 Verifying the softwre Explntion: The softwre pckges of the node re being checked for integrity. User response: complete. Allow the verifiction process to 137 Updting system bord service processor firmwre Explntion: The service processor firmwre of the Chpter 7. Dignosing problems 163

182 node is being updted to new level. This process cn tke 90 minutes. Do not restrt the node while this is in progress. User response: the commnd. Begin new SSH session nd re-issue User response: complete. Allow the updting process to 150 Loding cluster code Explntion: The SAN Volume Controller code is being loded. User response: If the progress br hs been stopped for t lest 90 seconds, power off the node nd then power on the node. If the boot process stops gin t this point, run the node rescue procedure. Possible Cuse-FRUs or other: v None. 155 Loding cluster dt Explntion: The sved cluster stte nd cche dt is being loded. User response: If the progress br hs been stopped for t lest 5 minutes, power off the node nd then power on the node. If the boot process stops gin t this point, run the node rescue procedure. Possible Cuse-FRUs or other: v None. 160 Updting the service controller Explntion: The firmwre on the service controller is being updted. This cn tke 30 minutes. User response: When node rescue is occurring, if the progress br hs been stopped for t lest 30 minutes, exchnge the FRU for new FRU. When node rescue is not occurring, if the progress br hs been stopped for t lest 15 minutes, exchnge the FRU for new FRU. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Service controller (95%) v Service controller cble (5%) All previous 2145 models v Service Controller (100%) 168 The commnd cnnot be initited becuse uthentiction credentils for the current SSH session hve expired. Explntion: Authentiction credentils for the current SSH session hve expired, nd ll uthoriztion for the current session hs been revoked. A system dministrtor my hve clered the uthentiction cche. 170 A flsh module hrdwre error hs occurred. Explntion: occurred. User response: A flsh module hrdwre error hs Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Service controller (95%) v Service controller cble (5%) All previous 2145 models v Service controller (100%) Exchnge the FRU for new FRU. 182 Checking uninterruptible power supply Explntion: The node is checking whether the uninterruptible power supply is operting correctly. User response: complete. Allow the checking process to 232 Checking uninterruptible power supply connections Explntion: The node is checking whether the power nd signl cble connections to the uninterruptible power supply re correct. User response: complete. Allow the checking process to 300 The 2145 is running node rescue. Explntion: The 2145 is running node rescue. User response: If the progress br hs been stopped for t lest two minutes, exchnge the FRU for new FRU. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Service controller (95%) v Service controller cble (5%) G4 or A4 v Service controller (100%) 310 The 2145 is running formt opertion. Explntion: The 2145 is running formt opertion. User response: If the progress br hs been stopped for two minutes, exchnge the FRU for new FRU. Possible Cuse-FRUs or other: 164 SAN Volume Controller: Troubleshooting Guide

183 CG8 or 2145-CF8 v Disk drive (50%) v Disk controller (30%) v Disk bckplne (10%) v Disk signl cble (8%) v Disk power cble (1%) v System bord (1%) G4 or A4 v Disk drive ssembly (90%) v Disk cble ssembly (10%) 320 A 2145 formt opertion hs filed. Explntion: A 2145 formt opertion hs filed. User response: Exchnge the FRU for new FRU. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Disk drive (50%) v Disk controller (30%) v Disk bckplne (10%) v Disk signl cble (8%) v Disk power cble (1%) v System bord (1%) G4 or A4 v Disk drive ssembly (90%) v Disk cble ssembly (10%) 330 The 2145 is prtitioning its disk drive. Explntion: The 2145 is prtitioning its disk drive. User response: If the progress br hs been stopped for two minutes, exchnge the FRU for new FRU. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Disk drive (50%) v Disk controller (30%) v Disk bckplne (10%) v Disk signl cble (8%) v Disk power cble (1%) v System bord (1%) G4 or A4 v Disk drive ssembly (90%) v Disk cble ssembly (10%) Other: v Configurtion problem v Softwre error 340 The 2145 is serching for donor node. Explntion: The 2145 is serching for donor node. User response: If the progress br hs been stopped for more thn two minutes, exchnge the FRU for new FRU. Possible Cuse-FRUs or other: v Fibre Chnnel dpter (100%) 345 The 2145 is serching for donor node from which to copy the softwre. Explntion: donor node. The node is serching t 1 Gb/s for User response: If the progress br hs stopped for more thn two minutes, exchnge the FRU for new FRU. Possible Cuse-FRUs or other: v Fibre Chnnel dpter (100%) 350 The 2145 cnnot find donor node. Explntion: The 2145 cnnot find donor node. User response: If the progress br hs stopped for more thn two minutes, perform the following steps: 1. Ensure tht ll of the Fibre Chnnel cbles re connected correctly nd securely to the cluster. 2. Ensure tht t lest one other node is opertionl, is connected to the sme Fibre Chnnel network, nd is donor node cndidte. A node is donor node cndidte if the version of softwre tht is instlled on tht node supports the model type of the node tht is being rescued. 3. Ensure tht the Fibre Chnnel zoning llows connection between the node tht is being rescued nd the donor node cndidte. 4. Perform the problem determintion procedures for the network. Possible Cuse-FRUs or other: v None Other: v Fibre Chnnel network problem 360 The 2145 is loding softwre from the donor. Explntion: donor. The 2145 is loding softwre from the User response: If the progress br hs been stopped for t lest two minutes, restrt the node rescue procedure. Possible Cuse-FRUs or other: Chpter 7. Dignosing problems 165

184 v None 365 Cnnot lod SW from donor Explntion: User response: None. None. 370 Instlling softwre Explntion: The 2145 is instlling softwre. User response: 1. If this code is displyed nd the progress br hs been stopped for t lest ten minutes, the softwre instll process hs filed with n unexpected softwre error. 2. Power off the 2145 nd wit for 60 seconds. 3. Power on the The softwre upgrde opertion continues. 4. Report this problem immeditely to your Softwre Support Center. Possible Cuse-FRUs or other: v None 500 Incorrect enclosure Explntion: The node cnister hs sved cluster informtion, which indictes tht the cnister is now locted in different enclosure from where it ws previously used. Using the node cnister in this stte might corrupt the dt held on the enclosure drives. User response: Follow troubleshooting procedures to move the nodes to the correct loction. 1. Follow the Procedure: Getting node cnister nd system informtion using the service ssistnt tsk to review the node cnister sved loction informtion nd the sttus of the other node cnister in the enclosure (the prtner cnister). Determine if the enclosure is prt of n ctive system with volumes tht contin required dt. 2. If you hve unintentionlly moved the cnister into this enclosure, move the cnister bck to its originl loction, nd put the originl cnister bck in this enclosure. Follow the Replcing node cnister procedure. 3. If you hve intentionlly moved the node cnister into this enclosure you should check it is sfe to continue or whether you will lose dt on the enclosure you removed it from. Do not continue if the system the node cnister ws removed from is offline, rther return the node cnister to tht system. 4. If you hve determined tht you cn continue, follow the Procedure: Removing system dt from node cnister tsk to remove cluster dt from node cnister. 5. If the prtner node in this enclosure is not online, or is not present, you will hve to perform system recovery. Do not crete new system, you will lose ll the volume dt. Possible Cuse-FRUs or other cuse: v None 501 Incorrect slot Explntion: The node cnister hs sved cluster informtion, which indictes tht the cnister is not locted in the expected enclosure, but in different slot from where it ws previously used. Using the node cnister in this stte might men tht hosts re not ble to connect correctly. User response: Follow troubleshooting procedures to relocte the node cnister to the correct loction. 1. Follow the Procedure: Getting node cnister nd system informtion using the service ssistnt tsk to review the node cnister sved loction informtion nd the sttus of the other node cnister in the enclosure (the prtner cnister). If the node cnister hs been indvertently swpped, the other node cnister will hve the sme error. 2. If the cnisters hve been swpped, use the Replcing node cnister procedure to swp the cnisters. The system should strt. 3. If the prtner cnister is in cndidte stte, use the hrdwre remove nd replce cnister procedure to swp the cnisters. The system should strt. 4. If the prtner cnister is in ctive stte, it is running the cluster on this enclosure nd hs replced the originl use of this cnister. Follow the Procedure: Removing system dt from node cnister tsk to remove cluster dt from this node cnister. The node cnister will then become ctive in the cluster in its current slot. 5. If the prtner cnister is in service stte, review its node error to determine the correct ction. Generlly, you will fix the errors reported on the prtner node in priority order, nd review the sitution gin fter ech chnge. If you hve to replce the prtner cnister with new one you should move this cnister bck to the correct loction t the sme time. Possible Cuse-FRUs or other: v None 502 No enclosure identity exists nd sttus from the prtner node could not be obtined. Explntion: The enclosure hs been replced nd communiction with the other node cnister (prtner node) in the enclosure is not possible. The prtner node could be missing, powered off, unble to boot, or n 166 SAN Volume Controller: Troubleshooting Guide

185 internode communiction filure my exist. User response: Follow troubleshooting procedures to configure the enclosure: 1. Follow the procedures to resolve problem to get the prtner node strted. An error will still exist becuse the enclosure hs no identity. If the error hs chnged, follow the service procedure for tht error. 2. If the prtner hs strted nd is showing loction error (probbly this one), then the PCI link is probbly broken. Since the enclosure midplne ws recently replced, this is likely the problem. Obtin replcement enclosure midplne, nd replce it.. 3. If this ction does not resolve the issue, contct IBM Support Center. They will work with you to ensure tht the system stte dt is not lost while resolving the problem. Also see Resolving problem. Possible Cuse FRUs or other: v Enclosure midplne (100%) 503 Incorrect enclosure type Explntion: The node cnister hs been moved to n expnsion enclosure. A node cnister will not operte in this environment. This cn lso be reported when replcement node cnister is instlled for the first time. User response: Otherwise, follow troubleshooting procedures to relocte the nodes to the correct loction. 1. Follow the procedure: Getting node cnister nd system informtion using USB flsh drive nd review the sved loction informtion of the node cnister to determine which control enclosure the node cnister should be in. 2. Follow the procedure to move the node cnister to the correct loction, then follow the procedure to move the expnsion cnister tht is probbly in tht loction, to here. If there is node cnister tht is in ctive stte where this node cnister should be, do not replce tht node cnister with this one. 504 No enclosure identity nd prtner node mtches. Explntion: The enclosure vitl product dt indictes tht the enclosure midplne hs been replced. This node cnister nd the other node cnister in the enclosure were previously operting in the sme enclosure midplne. User response: Follow troubleshooting procedures to configure the enclosure. 1. This is n expected sitution during the hrdwre remove nd replce procedure for control enclosure midplne. Continue following the remove nd replce procedure nd configure the new enclosure. Possible Cuse FRUs or other: v None 505 No enclosure identity nd prtner hs system dt tht does not mtch. Explntion: The enclosure vitl product dt indictes tht the enclosure midplne hs been replced. This node cnister nd the other node cnister in the enclosure do not come from the sme originl enclosure. User response: Follow troubleshooting procedures to relocte nodes to the correct loction. 1. Follow the Procedure: Getting node cnister nd system informtion using the service ssistnt tsk to review the node cnister sved loction informtion nd the sttus of the other node cnister in the enclosure (the prtner cnister). Determine if the enclosure is prt of n ctive system with volumes tht contin required dt. 2. Decide wht to do with the node cnister tht did not come from the enclosure tht is being replced.. If the other node cnister from the enclosure being replced is vilble, use the hrdwre remove nd replce cnister procedures to remove the incorrect cnister nd replce it with the second node cnister from the enclosure being replced. Restrt both cnisters. The two node cnister should show node error 504 nd the ctions for tht error should be followed. b. If the other node cnister from the enclosure being replced is not vilble, check the enclosure of the node cnister tht did not come from the replced enclosure. Do not use this cnister in this enclosure if you require the volume dt on the system from which the node cnister ws removed, nd tht system is not running with two online nodes. You should return the cnister to its originl enclosure nd use different cnister in this enclosure. c. When you hve checked tht it is not required elsewhere, follow the Procedure: Removing system dt from node cnister tsk to remove cluster dt from the node cnister tht did not come from the enclosure tht is being replced. d. Restrt both nodes. Expect node error 506 to be reported now, then follow the service procedures for tht error. Possible Cuse FRUs or other: v None Chpter 7. Dignosing problems 167

186 No enclosure identity nd no node stte on prtner Explntion: The enclosure vitl product dt indictes tht the enclosure midplne hs been replced. There is no cluster stte informtion on the other node cnister in the enclosure (the prtner cnister), so both node cnisters from the originl enclosure hve not been moved to this one. User response: Follow troubleshooting procedures to relocte nodes to the correct loction: 1. Follow the procedure: Getting node cnister nd system informtion nd review the sved loction informtion of the node cnister nd determine why the second node cnister from the originl enclosure ws not moved into this enclosure. 2. If you re sure tht this node cnister cme from the enclosure tht is being replced, nd the originl prtner cnister is vilble, use the Replcing node cnister procedure to instll the second node cnister in this enclosure. Restrt the node cnister. The two node cnisters should show node error 504, nd the ctions for tht error should be followed. 3. If you re sure this node cnister cme from the enclosure tht is being replced, nd tht the originl prtner cnister hs filed, continue following the remove nd replce procedure for n enclosure midplne nd configure the new enclosure. Possible Cuse FRUs or other: v None 507 No enclosure identity nd no node stte Explntion: The node cnister hs been plced in replcement enclosure midplne. The node cnister is lso replcement or hs hd ll cluster stte removed from it. User response: Follow troubleshooting procedures to relocte the nodes to the correct loction. 1. Check the sttus of the other node in the enclosure. It should show node error 506. Unless it lso shows error 507, check the errors on the other node nd follow the corresponding procedures to resolve the errors. 2. If the other node in the enclosure is lso reporting 507, the enclosure nd both node cnisters hve no stte informtion. You should contct IBM technicl support. They will ssist you in setting the enclosure vitl product dt nd running cluster recovery. Possible Cuse-FRUs or other: v None 508 Cluster identifier is different between enclosure nd node Explntion: The node cnister loction informtion shows it is in the correct enclosure, however the enclosure hs hd new clustered system creted on it since the node ws lst shut down. Therefore, the clustered system stte dt stored on the node is not vlid. User response: Follow troubleshooting procedures to correctly relocte the nodes. 1. Check whether new clustered system hs been creted on this enclosure while this cnister ws not operting or whether the node cnister ws recently instlled in the enclosure. 2. Follow the Procedure: Getting node cnister nd system informtion using the service ssistnt tsk, nd check the prtner node cnister to see if it is lso reporting node error 508. If it is, check tht the sved system informtion on this nd the prtner node mtch. If the system informtion on both nodes mtches, follow the Replcing control enclosure midplne procedure to chnge the enclosure midplne. 3. If this node cnister is the one to be used in this enclosure, follow the Procedure: Removing system dt from node cnister tsk to remove clustered system dt from the node cnister. It will then join the clustered system. 4. If this is not the node cnister tht you intended to use, follow the Replcing node cnister procedure to replce the node cnister with the one intended for use. Possible Cuse FRUs or other: v Service procedure error (90%) v Enclosure midplne (10%) 509 The enclosure identity cnnot be red. Explntion: The cnister ws unble to red vitl product dt (VPD) from the enclosure. The cnister requires this dt to be ble to initilize correctly. User response: Follow troubleshooting procedures to fix the hrdwre: 1. Check errors reported on the other node cnister in this enclosure (the prtner cnister). 2. If it is reporting the sme error follow the hrdwre remove nd replce procedure to replce the enclosure midplne. 3. If the prtner cnister is not reporting this error, follow the hrdwre remove nd replce procedure to replce this cnister. Note: If newly instlled system hs this error on both node cnisters, the dt tht needs to be written to the 168 SAN Volume Controller: Troubleshooting Guide

187 enclosure will not be vilble on the cnisters; contct IBM support for the WWNNs to use. Remember: Review the lsservicenodes output for wht the node is reporting. Possible Cuse FRUs or other: v Node cnister (50%) v Enclosure midplne (50%) 510 The detected memory size does not mtch the expected memory size. Explntion: The mount of memory detected in the node is less thn the mount required for the node to operte s n ctive member of system. The error code dt shows the detected memory, in MB, followed by the minimum required memory, in MB, there is then series of vlues indicting the mount of memory, in GB, detected in ech memory slot. Dt: v Detected memory on MB v Minimum required memory in MB v Memory in slot 1 in GB v Memory in slot 2 in GB v... etc. User response: Check the memory size of nother 2145 tht is in the sme cluster. For the G4, A4, 2145-CF8, nd 2145-CG8, if you hve just replced memory module, check tht the module tht you hve instlled is the correct size, then go to the light pth MAP to isolte ny possible filed memory modules. Possible Cuse FRUs or other: v Memory module (100%) 511 Memory bnk 1 of the 2145 is filing. Explntion: Memory bnk 1 of the 2145 is filing. User response: For the G4 nd A4, go to the light pth MAP to resolve this problem. Possible Cuse-FRUs or other: v Memory module (100%) 512 Enclosure VPD is inconsistent Explntion: The enclosure midplne VPD is not consistent. The mchine prt number is not comptible with the mchine type nd model. This indictes tht the enclosure VPD is corrupted. User response: 1. Check the support site for code updte. 2. Use the remove nd replce procedures to replce the enclosure midplne. Possible Cuse FRUs or other: v Enclosure midplne (100%) 517 The WWNNs of the service controller nd the disk do not mtch. Explntion: The node is unble to determine the WWNN tht it should use. This is becuse of the service controller or the nodes internl drive being replced. User response: Follow troubleshooting procedures to configure the WWNN of the node. 1. Continue to follow the hrdwre remove nd replce procedure for the service controller or disk these explin the service ctions. 2. If you hve not followed the hrdwre remove nd replce procedures, you should determine the correct WWNN. If you do not hve this informtion recorded, exmine your Fibre Chnnel switch configurtion to see whether it is listed there. Follow the procedures to chnge the WWNN of node. Possible Cuse-FRUs or other: v None 521 Unble to detect Fibre Chnnel dpter Explntion: The 2145 cnnot detect ny Fibre Chnnel dpter crds. Explntion: The 2145 cnnot detect ny Fibre Chnnel dpter crds. User response: Ensure tht Fibre Chnnel dpter crd hs been instlled. Ensure tht the Fibre Chnnel crd is seted correctly in the riser crd. Ensure tht the riser crd is seted correctly on the system bord. If the problem persists, exchnge FRUs for new FRUs in the order shown. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v 4-port Fibre Chnnel host bus dpter ssembly (95%) v System bord ssembly (5%) G4 or A4 v 4-port Fibre Chnnel host bus dpter (80%) v Riser crd (19%) v System bord (1%) v 4-port Fibre Chnnel host bus dpter (99%) v Frme ssembly (1%) Chpter 7. Dignosing problems 169

188 The system bord service processor hs filed. Explntion: The service processor on the system bord hs filed. User response: Possible Cuse FRUs or other: Exchnge the FRU for new FRU G4, A4, 2145-CF8, or 1245-CG8 v System bord ssembly (100%) 523 The internl disk file system is dmged. Explntion: The node strtup procedures hve found problems with the file system on the internl disk of the node. User response: Follow troubleshooting procedures to relod the softwre. 1. Follow Procedure: Rescuing node cnister mchine code from nother node (node rescue). 2. If the rescue node does not succeed, use the hrdwre remove nd replce procedures. Possible Cuse FRUs or other: v Node cnister (80%) v Other (20%) 524 Unble to updte BIOS settings. Explntion: Unble to updte BIOS settings. User response: Power off node, wit 30 seconds, nd then power on gin. If the error code is still reported, replce the system bord. Possible Cuse-FRUs or other: v System bord (100%) 525 Unble to updte system bord service processor firmwre. Explntion: The node strtup procedures hve been unble to updte the firmwre configurtion of the node cnister. The updte my tke 90 minutes. User response: 1. If the progress br hs been stopped for more thn 90 minutes, power off nd reboot the node. If the boot progress br stops gin on this code, replce the FRU shown. 2. Try removing the power cords nd then restrting to fix the problem, if the power off / restrt doesn't work. Possible Cuse FRUs or other: G4, A4, 2145-CF8, or 2145-CG8 v System bord (100%) 528 Ambient temperture is too high during system strtup. Explntion: The mbient temperture red during the node strtup procedures is too high for the node to continue. The strtup procedure will continue when the temperture is within rnge. User response: Reduce the temperture round the system. 1. Resolve the issue with the mbient temperture, by checking nd correcting:. Room temperture nd ir conditioning b. Ventiltion round the rck c. Airflow within the rck Possible Cuse FRUs or other: v Environment issue (100%) 530 A problem with one of the node's power supplies hs been detected. Explntion: The 530 error code is followed by two numbers. The first number is either 1 or 2 to indicte which power supply hs the problem. The second number is either 1, 2 or 3 to indicte the reson. 1 indictes tht the power supply is not detected. 2 indictes tht the power supply hs filed. 3 indictes tht there is no input power to the power supply. If the node is member of cluster, the cluster will report error code 1096 or 1097, depending on the error reson. The error will utomticlly cler when the problem is fixed. User response: 1. Ensure tht the power supply is seted correctly nd tht the power cble is ttched correctly to both the node nd to the 2145 UPS-1U. 2. If the error hs not been utomticlly mrked fixed fter two minutes, note the sttus of the three LEDs on the bck of the power supply. For the 2145-CG8 or 2145-CF8, the AC LED is the top green LED, the DC LED is the middle green LED nd the error LED is the bottom mber LED. 3. If the power supply error LED is off nd the AC nd DC power LEDs re both on, this is the norml condition. If the error hs not been utomticlly fixed fter two minutes, replce the system bord. 4. Follow the ction specified for the LED sttes noted in the tble below. 5. If the error hs not been utomticlly fixed fter two minutes, contct support. 170 SAN Volume Controller: Troubleshooting Guide

189 Error,AC,DC:Action ON,ON or OFF,ON or OFF:The power supply hs fult. Replce the power supply. OFF,OFF,OFF:There is no power detected. Ensure tht the power cble is connected t the node nd 2145 UPS-1U. If the AC LED does not light, check whether the 2145 UPS-1U is showing ny errors. Follow MAP UPS-1U if the UPS-1U is showing n error; otherwise, replce the power cble. If the AC LED still does not light, replce the power supply. OFF,OFF,ON:The power supply hs fult. Replce the power supply. OFF,ON,OFF:Ensure tht the power supply is instlled correctly. If the DC LED does not light, replce the power supply. Possible Cuse-FRUs or other: Reson 1: A power supply is not detected. v Power supply (19%) v System bord (1%) v Other: Power supply is not instlled correctly (80%) Reson 2: The power supply hs filed. v Power supply (90%) v Power cble ssembly (5%) v System bord (5%) Reson 3: There is no input power to the power supply. v Power cble ssembly (25%) v UPS-1U ssembly (4%) v System bord (1%) v Other: Power supply is not instlled correctly (70%) 535 Cnister internl PCIe switch filed Explntion: The PCI Express switch hs filed or cnnot be detected. In this sitution, the only connectivity to the node cnister is through the Ethernet ports. User response: Follow troubleshooting procedures to fix the hrdwre: 541 Multiple, undetermined, hrdwre errors Explntion: Multiple hrdwre filures hve been reported on the dt pths within the node cnister, nd the threshold of the number of cceptble errors within given time frme hs been reched. It hs not been possible to isolte the errors to single component. After this node error hs been rised, ll ports on the node will be dectivted. The reson for this is tht the node cnister is considered unstble, nd hs the potentil to corrupt dt. User response: 1. Follow the procedure for collecting informtion for support, nd contct your support orgniztion. 2. A softwre [code] upgrde my resolve the issue. 3. Replce the node cnister. 550 A cluster cnnot be formed becuse of lck of cluster resources. Explntion: The node cnnot become ctive in cluster becuse it is unble to connect to enough cluster resources. The cluster resources re the nodes in the system nd the ctive quorum disk. The node needs to be ble to connect to mjority of the resources before tht group will form n online cluster. This prevents the cluster splitting into two or more ctive prts, with both prts independently performing I/O. Supplementl dt displyed with this error code list the missing IDs for the 2145s nd the quorum disk controller. Ech missing node is listed by its node ID. A missing quorum disk is listed s WWWWWWWWWWWWWWWW/LL, where WWWWWWWWWWWWWWWW is worldwide port nme (WWPN) on the disk controller tht contins the missing quorum disk nd LL is the Logicl Unit Number (LUN) of the missing quorum disk on tht controller. If the system topology is stretched nd the number of opertionl nodes re less thn hlf then Node Error 550 will be displyed. In this cse, the Site Disster Recovery feture cn not be used s the number of opertionl nodes is less thn the quorum required to crete cluster using Site DR feture. User response: Follow troubleshooting procedures to correct connectivity issues between the cluster nodes nd the quorum devices. 1. Check for ny node errors tht indicte issues with Fibre Chnnel connectivity. Resolve ny issues. 2. Ensure tht the other 2145s in the cluster re powered on nd opertionl. 3. From the front pnel, disply the Fibre Chnnel port sttus. If ny port is not ctive, perform the Fibre Chnnel port problem determintion procedures. 4. Ensure tht Fibre Chnnel network zoning chnges hve not restricted communiction between nodes or between the nodes nd the quorum disk. 5. Perform the problem determintion procedures for the network. 6. The quorum disk filed or cnnot be ccessed. Perform the problem determintion procedures for the disk controller. Chpter 7. Dignosing problems 171

190 A cluster cnnot be formed becuse of lck of cluster resources. Explntion: The node does not hve sufficient connectivity to other nodes or the quorum device to form cluster. Attempt to repir the fbric or quorum device to estblish connectivity. If disster hs occurred nd the nodes t the other site cnnot be recovered, then it is possible to llow the nodes t the surviving site to form system using locl storge. User response: Follow troubleshooting procedures to correct connectivity issues between the cluster nodes nd the quorum devices. 1. Check for ny node errors tht indicte issues with Fibre Chnnel connectivity. Resolve ny issues. 2. Ensure tht the other 2145s in the cluster re powered on nd opertionl. 3. From the front pnel, disply the Fibre Chnnel port sttus. If ny port is not ctive, perform the Fibre Chnnel port problem determintion procedures. 4. Ensure tht Fibre Chnnel network zoning chnges hve not restricted communiction between nodes or between the nodes nd the quorum disk. 5. Perform the problem determintion procedures for the network. 6. The quorum disk filed or cnnot be ccessed. Perform the problem determintion procedures for the disk controller. 7. As lst resort when the nodes t the other site cnnot be recovered, then it is possible to llow the nodes t the surviving site to form system using locl site storge s described below: To void dt corruption ensure tht ll host servers tht were previously ccessing the system hve hd ll volumes un-mounted or hve been rebooted. Ensure tht the nodes t the other site re not opertionl nd re unble to form system in the future. After invoking this commnd full re-synchroniztion of ll mirrored volumes will be performed when the other site is recovered. This is likely to tke mny hours or dys to complete. Contct IBM support personnel if you re unsure. Note: Before continuing confirm tht you hve tken the following ctions - filure to perform these ctions cn led to dt corruption tht will be undetected by the system but will ffect host pplictions.. All host servers tht were previously ccessing the system hve hd ll volumes un-mounted or hve been rebooted. b. Ensure tht the nodes t the other site re not operting s system nd ctions hve been tken to prevent them from forming system in the future. After these ctions hve been tken the stsk overridequorum cn be used to llow the nodes t the surviving site to form system using locl storge. 555 Power Domin error Explntion: Both 2145s in n I/O group re being powered by the sme uninterruptible power supply. The ID of the other 2145 is displyed with the node error code on the front pnel. User response: Ensure tht the configurtion is correct nd tht ech 2145 is in n I/O group is connected from seprte uninterruptible power supply. 556 A duplicte WWNN hs been detected. Explntion: The node hs detected nother device tht hs the sme World Wide Node Nme (WWNN) on the Fibre Chnnel network. A WWNN is 16 hexdeciml digits long. For cluster, the first 11 digits re lwys The lst 5 digits of the WWNN re given in the dditionl dt of the error nd pper on the front pnel displys. The Fibre Chnnel ports of the node re disbled to prevent disruption of the Fibre Chnnel network. One or both nodes with the sme WWNN cn show the error. Becuse of the wy WWNNs re llocted, device with duplicte WWNN is normlly nother cluster node. User response: Follow troubleshooting procedures to configure the WWNN of the node: 1. Find the cluster node with the sme WWNN s the node reporting the error. The WWNN for cluster node cn be found from the node Vitl Product Dt (VPD) or from the Node menu on the front pnel. The node with the duplicte WWNN need not be prt of the sme cluster s the node reporting the error; it could be remote from the node reporting the error on prt of the fbric connected through n inter-switch link. The WWNN of the node is stored within the service controller, so the dupliction is most likely cused by the replcement of service controller. 2. If cluster node with duplicte WWNN is found, determine whether it, or the node reporting the error, hs the incorrect WWNN. Generlly, it is the node tht hs hd its service controller tht ws recently replced or hd its WWNN chnged incorrectly. Also consider how the SAN is zoned when mking your decision. 3. Determine the correct WWNN for the node with the incorrect WWNN. If the service controller hs been replced s prt of service ction, the WWNN for 172 SAN Volume Controller: Troubleshooting Guide

191 the node should hve been written down. If the correct WWNN cnnot be determined contct your support center for ssistnce. 4. Use the front pnel menus to modify the incorrect WWNN. If it is the node showing the error tht should be modified, this cn sfely be done immeditely. If it is n ctive node tht should be modified, use cution becuse the node will restrt when the WWNN is chnged. If this node is the only opertionl node in n enclosure, ccess to the volumes tht it is mnging will be lost. You should ensure tht the host systems re in the correct stte before you chnge the WWNN. 5. If the node showing the error hd the correct WWNN, it cn be restrted, using the front pnel power control button, fter the node with the duplicte WWNN is updted. 6. If you re unble to find cluster node with the sme WWNN s the node showing the error, use the SAN monitoring tools to determine whether there is nother device on the SAN with the sme WWNN. This device should not be using WWNN ssigned to cluster, so you should follow the service procedures for the device to chnge its WWNN. Once the duplicte hs been removed, restrt the node cnister. 558 The node is unble to communicte with other nodes. Explntion: The 2145 cnnot see the Fibre Chnnel fbric or the Fibre Chnnel crd port speed might be set to different speed thn the Fibre Chnnel fbric. User response: Ensure tht: 1. The Fibre Chnnel network fbric switch is powered-on. 2. At lest one Fibre Chnnel cble connects the 2145 to the Fibre Chnnel network fbric. 3. The Fibre Chnnel crd port speed is equl to the Fibre Chnnel fbric. 4. At lest one Fibre Chnnel dpter is instlled in the Go to the Fibre Chnnel MAP. Possible Cuse-FRUs or other: v None 560 Bttery cbling fult Explntion: A fult exists in one of the cbles connecting the bttery bckplne to the rest of the system. User response: Follow troubleshooting procedures to fix the hrdwre: 1. Reset the cble. 2. If reseting the cble does not fix the problem, replce the cble. 3. If replcing the cble does not fix the problem, replce the bttery bckplne. 561 Bttery bckplne or cbling fult Explntion: Either the bttery bckplne hs filed, or the power or LPC cbles connecting the bttery bckplne to the rest of the system re not connected properly. User response: Follow troubleshooting procedures to fix the hrdwre: 1. Check the cbles connecting the bttery bckplne. 2. Reset the power nd LPC cbles. 3. If reseting the cbles does not fix the problem, replce the cbles. 4. Once the cbles re well connected, but the problem persists, replce the bttery bckplne. 5. Conduct the corrective service procedure described in 1108 on pge The nodes hrdwre configurtion does not meet the minimum requirements Explntion: The node hrdwre is not t the minimum specifiction for the node to become ctive in cluster. This my be becuse of hrdwre filure, but is lso possible fter service ction hs used n incorrect replcement prt. User response: Follow troubleshooting procedures to fix the hrdwre: 1. View node VPD informtion, to see whether nything looks inconsistent. Compre the filing node VPD with the VPD of working node of the sme type. Py prticulr ttention to the number nd type of CPUs nd memory. 2. Replce ny incorrect prts. 564 Too mny mchine code crshes hve occurred. Explntion: The node hs been determined to be unstble becuse of multiple resets. The cuse of the resets cn be tht the system encountered n unexpected stte or hs executed instructions tht were not vlid. The node hs entered the service stte so tht dignostic dt cn be recovered. The node error does not persist cross restrts of the mchine code on the node. User response: Follow troubleshooting procedures to relod the mchine code: 1. Get support pckge (snp), including dumps, from the node, using the mngement GUI or the service ssistnt. 2. If more thn one node is reporting this error, contct IBM technicl support for ssistnce. The support pckge from ech node will be required. Chpter 7. Dignosing problems 173

192 Check the support site to see whether the issue is known nd whether mchine code upgrde exists to resolve the issue. Updte the cluster mchine code if resolution is vilble. Use the mnul upgrde process on the node tht reported the error first. 4. If the problem remins unresolved, contct IBM technicl support nd send them the support pckge. Possible Cuse FRUs or other: v None 565 The internl drive of the node is filing. Explntion: The internl drive within the node is reporting too mny errors. It is no longer sfe to rely on the integrity of the drive. Replcement is recommended. User response: Follow troubleshooting procedures to fix the hrdwre: 1. View hrdwre informtion. 2. Replce prts (cnister or disk). Possible Cuse FRUs or other: v 2145-CG8 or 2145-CF8 Disk drive (50%) Disk controller (30%) Disk bckplne (10%) Disk signl cble (8%) Disk power cble (1%) System bord (1%) v A4 Disk drive ssembly (80%) Disk cble ssembly (15%) System bord (5%) v G4 Disk drive ssembly (80%) Disk drive cbles (10%) System bord (10%) 570 Bttery protection unvilble Explntion: The node cnnot strt becuse bttery protection is not vilble. Both btteries require user intervention before they cn become vilble. User response: fix hrdwre. Follow troubleshooting procedures to The pproprite service ction will be indicted by n ccompnying non-ftl node error. Exmine the event log to determine the ccompnying node error. 571 Bttery protection temporrily unvilble; one bttery is expected to be vilble soon Explntion: The node cnnot strt becuse bttery protection is not vilble. One bttery is expected to become vilble shortly with no user intervention required, but the other bttery will not become vilble. User response: fix hrdwre. Follow troubleshooting procedures to The pproprite service ction will be indicted by n ccompnying non-ftl node error. Exmine the event log to determine the ccompnying node error. 572 Bttery protection temporrily unvilble; both btteries re expected to be vilble soon Explntion: The node cnnot strt becuse bttery protection is not vilble. Both btteries re expected to become vilble shortly with no user intervention required. User response: Wit for sufficient bttery chrge for enclosure to strt. 573 The node mchine code is inconsistent. Explntion: Prts of the node mchine code pckge re receiving unexpected results; there my be n inconsistent set of subpckges instlled, or one subpckge my be dmged. User response: Follow troubleshooting procedures to relod the mchine code. 1. Follow the procedure to run node rescue. 2. If the error occurs gin, contct IBM technicl support. Possible Cuse FRUs or other: v None 574 The node mchine code is dmged. Explntion: A checksum filure hs indicted tht the node mchine code is dmged nd needs to be reinstlled. User response: 1. If the other nodes re opertionl, run node rescue; otherwise, instll new mchine code using the service ssistnt. Node rescue filures, s well s the repeted return of this node error fter reinstlltion, re symptomtic of hrdwre fult with the node. Possible Cuse FRUs or other: v None 174 SAN Volume Controller: Troubleshooting Guide

193 The cluster stte nd configurtion dt cnnot be red. Explntion: The node hs been unble to red the sved cluster stte nd configurtion dt from its internl drive becuse of red or medium error. User response: In the sequence shown, exchnge the FRUs for new FRUs. Possible Cuse FRUs or other: v 2145-CG8 or 2145-CF8 Disk drive (50%) Disk controller (30%) Disk bckplne (10%) Disk signl cble (8%) Disk power cble (1%) System bord (1%) v A4 Disk drive ssembly (80%) Disk cble ssembly (15%) System bord (5%) v G4 Disk drive ssembly (80%) Disk drive cbles (10%) System bord (10%) 578 The stte dt ws not sved following power loss. Explntion: On strtup, the node ws unble to red its stte dt. When this hppens, it expects to be utomticlly dded bck into cluster. However, if it hs not joined cluster in 60 sec, it rises this node error. This is criticl node error, nd user ction is required before the node cn become cndidte to join cluster. User response: Follow troubleshooting procedures to correct connectivity issues between the cluster nodes nd the quorum devices. 1. Mnul intervention is required once the node reports this error. 2. Attempt to reestblish the cluster using other nodes. This my involve fixing hrdwre issues on other nodes or fixing connectivity issues between nodes. 3. If you re ble to reestblish the cluster, remove the cluster dt from the node showing 578 so it goes to cndidte stte, it will then be utomticlly dded bck to the cluster.. To remove the cluster dt from the node, either go to the service ssistnt, select the rdio button for the node with 578, click Mnge System, then choose Remove System Dt. b. Or use the CLI to stsk levecluster. If the node does not utomticlly dd bck to the cluster, note the nme nd I/O group of the node, then delete the node from the cluster configurtion (if this hs not lredy hppened) nd then dd the node bck to the cluster using the sme nme nd I/O group. 4. If ll nodes hve either node error 578 or 550, follow the cluster recovery procedures. 5. Attempt to determine wht cused the nodes to shut down. Possible Cuse FRUs or other: v None 579 Bttery subsystem hs insufficient chrge to sve system dt Explntion: Not enough cpcity is vilble from the bttery subsystem to sve system dt in response to series of bttery nd boot-drive fults. User response: fix hrdwre. Follow troubleshooting procedures to The pproprite service ctions re indicted by the series of bttery nd boot-drive fults. Exmine the event log to determine the ccompnying fults. Service the other fults. 580 The service controller ID could not be red. Explntion: The 2145 cnnot red the unique ID from the service controller, so the Fibre Chnnel dpters cnnot be strted. User response: In the sequence shown, exchnge the following FRUs for new FRUs. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Service controller (70%) v Service controller cble (30%) G4 or A4 Service controller (100%) Other: v None 581 A seril link error in the 2145 UPS-1U hs occurred. Explntion: There is fult in the communictions cble, the seril interfce in the uninterruptible power supply 2145 UPS-1U, or User response: Check tht the communictions cble is correctly plugged in to the 2145 nd the 2145 Chpter 7. Dignosing problems 175

194 UPS-1U. If the cble is plugged in correctly, replce the FRUs in the order shown. Possible Cuse-FRUs or other: G4, A4, 2145-CF8, or 2145-CG8 v 2145 power cble ssembly (40%) v 2145 UPS-1U ssembly (30%) v 2145 system bord (30%) 582 A bttery error in the 2145 UPS-1U hs occurred. Explntion: A problem hs occurred with the uninterruptible power supply 2145 UPS-1U bttery. User response: Exchnge the FRU for new FRU. After replcing the bttery ssembly, if the 2145 UPS-1U service indictor is on, press nd hold the 2145 UPS-1U Test button for three seconds to strt the self-test nd verify the repir. During the self-test, the rightmost four LEDs on the 2145 UPS-1U front-pnel ssembly flsh in sequence. Possible Cuse-FRUs or other: v UPS-1U bttery ssembly (50%) v UPS-1U ssembly (50%) 583 An electronics error in the 2145 UPS-1U hs occurred. Explntion: A problem hs occurred with the 2145 UPS-1U electronics. User response: Possible Cuse-FRUs or other: v 2145 UPS-1U ssembly Exchnge the FRU for new FRU. 584 The 2145 UPS-1U is overloded. Explntion: A problem with output overlod hs been reported by the uninterruptible power supply 2145 UPS-1U. The Overlod Indictor on the 2145 UPS-1U front pnel is illuminted red. User response: 1. Ensure tht only one 2145 is receiving power from the 2145 UPS-1U. Also ensure tht no other devices re connected to the 2145 UPS-1U. 2. Disconnect the 2145 from the 2145 UPS-1U. If the Overlod Indictor is still illuminted, on the disconnected 2145 replce the 2145 UPS-1U. 3. If the Overlod Indictor is now off, nd the node is G4 or A4, on the disconnected 2145, with ll outputs disconnected, in the sequence shown, exchnge the FRUs for new FRUs. 4. If the Overlod Indictor is now off, nd the node is 2145-CG8 or 2145-CF8, on the disconnected 2145, with ll outputs disconnected, determine whether it is one of the two power supplies or the power cble ssembly tht must be replced. Plug just one power cble into the left hnd power supply nd strt the node nd see whether the error is reported. Then shut down the node nd connect the other power cble into the left hnd power supply nd strt the node nd see whether the error is repeted. Then repet the two tests for the right hnd power supply. If the error is repeted for both cbles on one power supply but not the other, replce the power supply tht showed the error; otherwise, replce the power cble ssembly. Possible Cuse-FRUs or other: v Power cble ssembly (45%) v Power supply ssembly (45%) v UPS-1U ssembly (10%) 586 The power supply to the 2145 UPS-1U does not meet requirements. Explntion: None. User response: Follow troubleshooting procedures to fix the hrdwre. 587 An incorrect type of uninterruptible power supply hs been detected. Explntion: instlled. An incorrect type of 2145 UPS-1U ws User response: Exchnge the 2145 UPS-1U for one of the correct type. Possible Cuse-FRUs or other: v 2145 UPS-1U (100%) 588 The 2145 UPS-1U is not cbled correctly. Explntion: The signl cble or the 2145 power cbles re probbly not connected correctly. The power cble nd signl cble might be connected to different 2145 UPS-1U ssemblies. User response: 1. Connect the cbles correctly. 2. Restrt the node. Possible Cuse-FRUs or other: v None. Other: v Cbling error (100%) 589 The 2145 UPS-1U mbient temperture limit hs been exceeded. Explntion: The mbient temperture threshold for the 2145 UPS-1U hs been exceeded. 176 SAN Volume Controller: Troubleshooting Guide

195 User response: Reduce the temperture round the system: 1. Turn off the 2145 UPS-1U nd unplug it from the power source. 2. Cler the vents nd remove ny het sources. 3. Ensure tht the ir flow round the 2145 UPS-1U is not restricted. 4. Wit t lest five minutes, nd then restrt the 2145 UPS-1U. If the problem remins, exchnge 2145 UPS-1U ssembly. 590 Repetitive node trnsitions into stndby mode from norml mode becuse of power subsystem-relted node errors. Explntion: Multiple node restrts occurred becuse of 2145 UPS-1U errors, which cn be reported on ny node type This error mens tht the node mde the trnsition into stndby from norml mode becuse of power subsystem-relted node errors too mny times within short period. Too mny times re defined s three, nd short period is defined s 1 hour. This error lerts the user tht something might be wrong with the power subsystem s it is clerly not norml for the node to repetedly go in nd out of stndby. If the ctions of the tester or engineer re expected to cuse mny frequent trnsitions from norml to stndby nd bck, then this error does not imply tht there is ny ctul fult with the system. User response: Follow troubleshooting procedures to fix the hrdwre: 1. Verify tht the room temperture is within specified limits nd tht the input power is stble. 2. If 2145 UPS-1U is connected, verify tht the 2145 UPS-1U signl cble is fstened securely t both ends. 3. Look in the system event log for the node error tht is repeting. Note: The condition is reset by powering off the node from the node front pnel. 650 The cnister bttery is not supported Explntion: The cnister bttery shows product dt tht indictes it cnnot be used with the code version of the cnister. User response: This is resolved by either obtining bttery which is supported by the system's code level, or the cnister's code level is updted to level which supports the bttery. 1. Remove the cnister nd its lid nd check the FRU prt number of the new bttery mtches tht of the replced bttery. Obtin the correct FRU prt if it does not. 2. If the cnister hs just been replced, check the code level of the prtner node cnister nd use the service ssistnt to upgrde this cnister's code level to the sme level. Possible cuse FRUs or other cuse v cnister bttery 651 The cnister bttery is missing Explntion: The cnister bttery cnnot be detected. User response: 1. Use the remove nd replce procedures to remove the node cnister nd its lid. 2. Use the remove nd replce procedures to instll bttery. 3. If there is bttery present ensure it is fully inserted. Replce the cnister. 4. If this error persists, use the remove nd replce procedures to replce the bttery. Possible cuse FRUs or other cuse v cnister bttery 652 The cnister bttery hs filed Explntion: The cnister bttery hs filed. The bttery my be showing n error stte, it my hve reched the end of life, or it my hve filed to chrge. Dt Number indictors with filure resons v 1 bttery reports filure v 2 end of life v 3 filure to chrge User response: 1. Use the remove nd replce procedures to replce the bttery. Possible cuse FRUs or other cuse v cnister bttery 653 The cnister bttery s temperture is too low Explntion: The cnister bttery s temperture is below its minimum operting temperture. User response: v Wit for the bttery to wrm up, the error will cler when its minimum working temperture is reched. Chpter 7. Dignosing problems 177

196 v If the error persists for more thn n hour when the mbient temperture is norml, use the remove nd replce procedures to replce the bttery. Possible cuse FRUs or other cuse v cnister bttery 654 The cnister bttery s temperture is too high Explntion: The cnister bttery s temperture is bove its sfe operting temperture. User response: v If necessry, reduce the mbient temperture. v Wit for the bttery to cool down, the error will cler when norml working temperture is reched. Keep checking the reported error s the system my determine the bttery hs filed. v If the node error persists for more thn two hours fter the mbient temperture returns to the norml operting rnge, use the remove nd replce procedures to replce the bttery. Possible cuse FRUs or other cuse v cnister bttery 655 Cnister bttery communictions fult. Explntion: the bttery. The cnister cnnot communicte with User response: v Use the remove nd replce procedures to replce the bttery. v If the node error persists, use the remove nd replce procedures to replce the node cnister. Possible Cuse-FRUs or other cuse: v Cnister bttery v Node cnister 656 The cnister bttery hs insufficient chrge Explntion: The cnister bttery hs insufficient chrge to sve the cnister s stte nd cche dt to the internl drive if power were to fil. User response: v Wit for the bttery to chrge, the bttery does not need to be fully chrged for the error to utomticlly cler. Possible cuse FRUs or other cuse v none 668 The remote setting is not vilble for users for the current system. Explntion: On the current systems, users cnnot be set to remote. User response: Any user defined on the system must be locl user. To crete remote user the user must not be defined on the locl system. 670 The UPS bttery chrge is not enough to llow the node to strt. Explntion: The uninterruptible power supply connected to the node does not hve sufficient bttery chrge for the node to sfely become ctive in cluster. The node will not strt until sufficient chrge exists to store the stte nd configurtion dt held in the node memory if power were to fil. The front pnel of the node will show "chrging". User response: Wit for sufficient bttery chrge for enclosure to strt: 1. Wit for the node to utomticlly fix the error when there is sufficient chrge. 2. Ensure tht no error conditions re indicted on the uninterruptible power supply. 671 The vilble bttery chrge is not enough to llow the node cnister to strt. Two btteries re chrging. Explntion: The bttery chrge within the enclosure is not sufficient for the node to sfely become ctive in cluster. The node will not strt until sufficient chrge exists to store the stte nd configurtion dt held in the node cnister memory if power were to fil. Two btteries re within the enclosure, one in ech of the power supplies. Neither of the btteries indicte n error both re chrging. The node will strt utomticlly when sufficient chrge is vilble. The btteries do not hve to be fully chrged before the nodes cn become ctive. Both nodes within the enclosure shre the bttery chrge, so both node cnisters report this error. The service ssistnt shows the estimted strt time in the node cnister hrdwre detils. User response: Wit for the node to utomticlly fix the error when sufficient chrge becomes vilble. 672 The vilble bttery chrge is not enough to llow the node cnister to strt. One bttery is chrging. Explntion: The bttery chrge within the enclosure is not sufficient for the node to sfely become ctive in cluster. The node will not strt until sufficient chrge exists to store the stte nd configurtion dt held in the node cnister memory if power were to fil. Two btteries re within the enclosure, one in ech of the 178 SAN Volume Controller: Troubleshooting Guide

197 power supplies. Only one of the btteries is chrging, so the time to rech sufficient chrge will be extended. The node will strt utomticlly when sufficient chrge is vilble. The btteries do not hve to be fully chrged before the nodes cn become ctive. Both nodes within the enclosure shre the bttery chrge, so both node cnisters report this error. The service ssistnt shows the estimted strt time, nd the bttery sttus, in the node cnister hrdwre detils. Possible Cuse-FRUs or other: v None User response: 1. Wit for the node to utomticlly fix the error when sufficient chrge becomes vilble. 2. If possible, determine why one bttery is not chrging. Use the bttery sttus shown in the node cnister hrdwre detils nd the indictor LEDs on the PSUs in the enclosure to dignose the problem. If the issue cnnot be resolved, wit until the cluster is opertionl nd use the troubleshooting options in the mngement GUI to ssist in resolving the issue. Possible Cuse-FRUs or other: v Bttery (33%) v Control power supply (33%) v Power cord (33%) 673 The vilble bttery chrge is not enough to llow the node cnister to strt. No btteries re chrging. Explntion: A node cnnot be in ctive stte if it does not hve sufficient bttery power to store configurtion nd cche dt from memory to internl disk fter power filure. The system hs determined tht both btteries hve filed or re missing. The problem with the btteries must be resolved to llow the system to strt. User response: Follow troubleshooting procedures to fix hrdwre: 1. Resolve problems in both btteries by following the procedure to determine sttus using the LEDs. 2. If the LEDs do not show fult on the power supplies or btteries, power off both power supplies in the enclosure nd remove the power cords. Wit 20 seconds, then replce the power cords nd restore power to both power supplies. If both node cnisters continue to report this error replce the enclosure chssis. Possible Cuse-FRUs or other: v Bttery (33%) v Power supply (33%) v Power cord (33%) v Enclosure chssis (1%) 674 The cycling mode of Metro Mirror object cnnot be chnged. Explntion: The cycling mode my only be set for Globl Mirror objects. Metro Mirror objects cnnot hve cycling mode defined. User response: The object's type must be set to 'globl' before or when setting the cycling mode. 677 System code upgrde cnnot strt becuse component firmwre updte is in progress. Explntion: An ttempt ws mde to initite n system code upgrde (CCU) while the system ws updting the firmwre of vrious hrdwre components. A CCU cn not be done while firmwre downlod is in progress nd so the request to strt CCU filed. User response: The firmwre downlod needs to complete before you cn perform CCU. Due to the dynmic nture of firmwre downlods there is no wy of following the progress of one. Wit pproximtely 10 minutes nd retry the commnd. 690 The node is held in the service stte. Explntion: The node is in service stte nd hs been instructed to remin in service stte. While in service stte, the node will not run s prt of cluster. A node must not be in service stte for longer thn necessry while the cluster is online becuse loss of redundncy will result. A node cn be set to remin in service stte either becuse of service ssistnt user ction or becuse the node ws deleted from the cluster. User response: When it is no longer necessry to hold the node in the service stte, exit the service stte to llow the node to run: 1. Use the service ssistnt ction to relese the service stte. Possible Cuse FRUs or other: v none 700 The Fibre Chnnel dpter tht ws previously present hs not been detected. Explntion: A Fibre Chnnel dpter tht ws previously present hs not been detected. The dpter might not be correctly instlled, or it might hve filed. This node error does not, in itself, stop the node cnister from becoming ctive in the system; however, the Fibre Chnnel network might be being used to communicte between the node cnisters in clustered Chpter 7. Dignosing problems 179

198 system. It is possible tht this node error indictes why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node cnister. Dt: v Loction A number indicting the dpter loction. The loction indictes n dpter slot, see the node cnister description for the definition of the dpter slot loctions User response: 1. If possible, this noncriticl node error should be serviced using the mngement GUI nd running the recommended ctions for the service error code. 2. There re number of possibilities.. If you hve delibertely removed the dpter (possibly replcing it with different dpter type), you will need to follow the mngement GUI recommended ctions to mrk the hrdwre chnge s intentionl. b. If the previous steps hve not isolted the problem, use the remove nd replce procedures to replce the dpter, if this does not fix the problem replce the system bord. Possible Cuse FRUs or other cuse: v Fibre Chnnel dpter v System bord 701 A Fibre Chnnel dpter hs filed. Explntion: A Fibre Chnnel dpter hs filed. This node error does not, in itself, stop the node becoming ctive in the system. However, the Fibre Chnnel network might be being used to communicte between the nodes in clustered system. Therefore, it is possible tht this node error indictes the reson why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse-FRUs or other cuse: v Fibre Chnnel dpter v System bord 702 A Fibre Chnnel dpter hs PCI error. Explntion: A Fibre Chnnel dpter hs PCI error. This node error does not, in itself, stop the node becoming ctive in the system. However, the Fibre Chnnel network might be being used to communicte between the nodes in clustered system. Therefore, it is possible tht this node error indictes the reson why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse-FRUs or other cuse: v Fibre Chnnel dpter v System bord 703 A Fibre Chnnel dpter is degrded. Explntion: A Fibre Chnnel dpter is degrded. This node error does not, in itself, stop the node becoming ctive in the system. However, the Fibre Chnnel network might be being used to communicte between the nodes in clustered system. Therefore, it is possible tht this node error indictes the reson why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. 180 SAN Volume Controller: Troubleshooting Guide

199 Possible Cuse-FRUs or other cuse: v Fibre Chnnel dpter v System bord 704 Fewer Fibre Chnnel ports opertionl. Explntion: A Fibre Chnnel port tht ws previously opertionl is no longer opertionl. The physicl link is down. This node error does not, in itself, stop the node becoming ctive in the system. However, the Fibre Chnnel network might be being used to communicte between the nodes in clustered system. Therefore, it is possible tht this node error indictes the reson why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node. Dt: Three numeric vlues re listed: v The ID of the first unexpected inctive port. This ID is deciml number. v The ports tht re expected to be ctive, which is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is expected to be ctive. v The ports tht re ctully ctive, which is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is ctive. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Possibilities: v If the port hs been intentionlly disconnected, use the mngement GUI recommended ction for the service error code nd cknowledge the intended chnge. v Check tht the Fibre Chnnel cble is connected t both ends nd is not dmged. If necessry, replce the cble. v Check the switch port or other device tht the cble is connected to is powered nd enbled in comptible mode. Rectify ny issue. The device service interfce might indicte the issue. v Use the remove nd replce procedures to replce the SFP trnsceiver in the 2145 node nd the SFP trnsceiver in the connected switch or device. v Use the remove nd replce procedures to replce the dpter. Possible Cuse-FRUs or other cuse: v Fibre Chnnel cble v SFP trnsceiver v Fibre Chnnel dpter 705 Fewer Fibre Chnnel I/O ports opertionl. Explntion: One or more Fibre Chnnel I/O ports tht hve previously been ctive re now inctive. This sitution hs continued for one minute. A Fibre Chnnel I/O port might be estblished on either Fibre Chnnel pltform port or n Ethernet pltform port using FCoE. This error is expected if the ssocited Fibre Chnnel or Ethernet port is not opertionl. Dt: Three numeric vlues re listed: v The ID of the first unexpected inctive port. This ID is deciml number. v The ports tht re expected to be ctive, which is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is expected to be ctive. v The ports tht re ctully ctive, which is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is ctive. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Follow the procedure for mpping I/O ports to pltform ports to determine which pltform port is providing this I/O port. 3. Check for ny 704 (Fibre chnnel pltform port not opertionl) or 724 (Ethernet pltform port not opertionl) node errors reported for the pltform port. 4. Possibilities: v If the port hs been intentionlly disconnected, use the mngement GUI recommended ction for the service error code nd cknowledge the intended chnge. v Resolve the 704 or 724 error. v If this is n FCoE connection, use the informtion the view gives bout the Fibre Chnnel forwrder (FCF) to troubleshoot the connection between the port nd the FCF. Possible Cuse-FRUs or other cuse: v None 706 Fibre Chnnel clustered system pth filure. Explntion: One or more Fibre Chnnel (FC) input/output (I/O) ports tht hve previously been ble to see ll required online nodes cn no longer see them. This sitution hs continued for 5 minutes. This Chpter 7. Dignosing problems 181

200 error is not reported unless node is ctive in clustered system. A Fibre Chnnel I/O port might be estblished on either FC pltform port or n Ethernet pltform port using Fiber Chnnel over Ethernet (FCoE). Dt: Three numeric vlues re listed: v The ID of the first FC I/O port tht does not hve connectivity. This is deciml number. v The ports tht re expected to hve connections. This is hexdeciml number, nd ech bit position represents port - with the lest significnt bit representing port 1. The bit is 1 if the port is expected to hve connection to ll online nodes. v The ports tht ctully hve connections. This is hexdeciml number, ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port hs connection to ll online nodes. User response: 1. If possible, this noncriticl node error should be serviced using the mngement GUI nd running the recommended ctions for the service error code. 2. Follow the procedure: Mpping I/O ports to pltform ports to determine which pltform port does not hve connectivity. 3. There re number of possibilities. v If the port s connectivity hs been intentionlly reconfigured, use the mngement GUI recommended ction for the service error code nd cknowledge the intended chnge. You must hve t lest two I/O ports with connections to ll other nodes. v Resolve other node errors relting to this pltform port or I/O port. v Check tht the SAN zoning is correct. Possible Cuse: FRUs or other cuse: v None. 710 The SAS dpter tht ws previously present hs not been detected. Explntion: A SAS dpter tht ws previously present hs not been detected. The dpter might not be correctly instlled or it might hve filed. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Possibilities: v v v If the dpter hs been intentionlly removed, use the mngement GUI recommended ctions for the service error code, to cknowledge the chnge. Use the remove nd replce procedures to remove nd open the node nd check the dpter is fully instlled. If the previous steps hve not isolted the problem, use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse-FRUs or other cuse: v High-speed SAS dpter v System bord 711 A SAS dpter hs filed. Explntion: A SAS dpter hs filed. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse-FRUs or other cuse: v High-speed SAS dpter v System bord 712 A SAS dpter hs PCI error. Explntion: A SAS dpter hs PCI error. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Replce the dpter using the remove nd replce procedures. If this does not fix the problem, replce the system bord. 182 SAN Volume Controller: Troubleshooting Guide

201 Possible Cuse-FRUs or other cuse: v SAS dpter v System bord 713 A SAS dpter is degrded. Explntion: A SAS dpter is degrded. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse-FRUs or other cuse: v High-speed SAS dpter v System bord 715 Fewer SAS host ports opertionl Explntion: A SAS port tht ws previously opertionl is no longer opertionl. The physicl link is down. Dt: Three numeric vlues re listed: v The ID of the first unexpected inctive port. This ID is deciml number. v The ports tht re expected to be ctive, which is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is expected to be ctive. v The ports tht re ctully ctive, which is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is ctive. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Possibilities: v If the port hs been intentionlly disconnected, use the mngement GUI recommended ction for the service error code nd cknowledge the intended chnge. v Check tht the SAS cble is connected t both ends nd is not dmged. If necessry, replce the cble. v v Check the switch port or other device tht the cble is connected to is powered nd enbled in comptible mode. Rectify ny issue. The device service interfce might indicte the issue. Use the remove nd replce procedures to replce the dpter. Possible Cuse-FRUs or other cuse: v SAS cble v SAS dpter 720 Ethernet dpter tht ws previously present hs not been detected. Explntion: An Ethernet dpter tht ws previously present hs not been detected. The dpter might not be correctly instlled or it might hve filed. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. If the loction is 0, the dpter integrted into the system bord is being reported. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. If the dpter loction is 0, use the remove nd replce procedures to replce the system bord. 3. If the loction is not 0, there re number of possibilities:. Use the remove nd replce procedures to remove nd open the node nd check tht the dpter is fully instlled. b. If the previous steps hve not locted nd isolted the problem, use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse-FRUs or other cuse: v Ethernet dpter v System bord 721 An Ethernet dpter hs filed. Explntion: An Ethernet dpter hs filed. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. If the loction is 0, the dpter integrted into the system bord is being reported. User response: Chpter 7. Dignosing problems 183

202 If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. If the dpter loction is 0, use the remove nd replce procedures to replce the system bord. 3. If the dpter loction is not 0, use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse FRUs or other cuse: v Ethernet dpter v System bord 722 An Ethernet dpter hs PCI error. Explntion: An Ethernet dpter hs PCI error. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. If the loction is 0, the dpter integrted into the system bord is being reported. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. If the dpter loction is 0, use the remove nd replce procedures to replce the system bord. 3. If the dpter loction is not 0, use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse FRUs or other cuse: v Ethernet dpter v System bord 723 An Ethernet dpter is degrded. Explntion: An Ethernet dpter is degrded. Dt: v A number indicting the dpter loction. The loction indictes n dpter slot. See the node description for the definition of the dpter slot loctions. If the loction is 0, the dpter integrted into the system bord is being reported. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. If the dpter loction is 0, use the remove nd replce procedures to replce the system bord. 3. If the dpter loction is not 0, use the remove nd replce procedures to replce the dpter. If this does not fix the problem, replce the system bord. Possible Cuse FRUs or other cuse: v Ethernet dpter v System bord 724 Fewer Ethernet ports ctive. Explntion: An Ethernet port tht ws previously opertionl is no longer opertionl. The physicl link is down. Dt: Three numeric vlues re listed: v The ID of the first unexpected inctive port. This is deciml number. v The ports tht re expected to be ctive. This is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is expected to be ctive. v The ports tht re ctully ctive. This is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is ctive. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Possibilities:. If the port hs been intentionlly disconnected, use the mngement GUI recommended ction for the service error code nd cknowledge the intended chnge. b. Mke sure the Ethernet cble is connected t both ends nd is undmged. If necessry, replce the cble. c. Check tht the switch port, or other device the cble is connected to, is powered nd enbled in comptible mode. Rectify ny issue. The device service interfce might indicte the issue. d. If this is 1 Gbps port, use the remove nd replce procedures to replce the SFP trnsceiver in the SAN Volume Controller nd the SFP trnsceiver in the connected switch or device. e. Replce the dpter or the system bord (depending on the port loction) by using the remove nd replce procedures. Possible Cuse FRUs or other cuse: v Ethernet cble v Ethernet SFP trnsceiver v Ethernet dpter v System bord 184 SAN Volume Controller: Troubleshooting Guide

203 The bus dpter hs not been detected. Explntion: The bus dpter tht connects the cnister to the enclosure midplne hs not been detected. This node error does not, in itself, stop the node cnister becoming ctive in the system. However, the bus might be being used to communicte between the node cnisters in clustered system. Therefore, it is possible tht this node error indictes the reson why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node cnister. Dt: v A number indicting the dpter loction. Loction 0 indictes tht the dpter integrted into the system bord is being reported. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. As the dpter is locted on the system bord, replce the node cnister using the remove nd replce procedures. Possible Cuse-FRUs or other cuse: v Node cnister 731 The bus dpter hs filed. Explntion: The bus dpter tht connects the cnister to the enclosure midplne hs filed. This node error does not, in itself, stop the node cnister becoming ctive in the system. However, the bus might be being used to communicte between the node cnisters in clustered system. Therefore, it is possible tht this node error indictes the reson why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node cnister. Dt: v A number indicting the dpter loction. Loction 0 indictes tht the dpter integrted into the system bord is being reported. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. As the dpter is locted on the system bord, replce the node cnister using the remove nd replce procedures. Possible Cuse-FRUs or other cuse: v Node cnister 732 The bus dpter hs PCI error. Explntion: The bus dpter tht connects the cnister to the enclosure midplne hs PCI error. This node error does not, in itself, stop the node cnister becoming ctive in the system. However, the bus might be being used to communicte between the node cnisters in clustered system; therefore it is possible tht this node error indictes the reson why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node cnister. Dt: v A number indicting the dpter loction. Loction 0 indictes tht the dpter integrted into the system bord is being reported. User response: 1. If possible, this noncriticl node error should be serviced using the mngement GUI nd running the recommended ctions for the service error code. 2. As the dpter is locted on the system bord, replce the node cnister using the remove nd replce procedures. Possible Cuse-FRUs or other cuse: v Node cnister 733 The bus dpter degrded. Explntion: The bus dpter tht connects the cnister to the enclosure midplne is degrded. This node error does not, in itself, stop the node cnister becoming ctive in the system. However, the bus might be being used to communicte between the node cnisters in clustered system. Therefore, it is possible tht this node error indictes the reson why the criticl node error 550 A cluster cnnot be formed becuse of lck of cluster resources is reported on the node cnister. Dt: v A number indicting the dpter loction. Loction 0 indictes tht the dpter integrted into the system bord is being reported. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. As the dpter is locted on the system bord, replce the node cnister using the remove nd replce procedures. Possible Cuse-FRUs or other cuse: v Node cnister Chpter 7. Dignosing problems 185

204 Fewer bus ports. Explntion: One or more PCI bus ports tht hve previously been ctive re now inctive. This condition hs existed for over one minute. Tht is, the internode link hs been down t the protocol level. This could be link issue but is more likely cused by the prtner node unexpectedly filing to respond. Dt: Three numeric vlues re listed: v The ID of the first unexpected inctive port. This is deciml number. v The ports tht re expected to be ctive. This is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is expected to be ctive. v The ports tht re ctully ctive. This is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is ctive. User response: 1. If possible, this noncriticl node error should be serviced using the mngement GUI nd running the recommended ctions for the service error code. 2. Follow the procedure for getting node cnister nd clustered-system informtion nd determine the stte of the prtner node cnister in the enclosure. Fix ny errors reported on the prtner node cnister. 3. Use the remove nd replce procedures to replce the enclosure. Possible Cuse-FRUs or other cuse: v Node cnister v Enclosure midplne 740 The commnd filed becuse of wiring error described in the event log. Explntion: It is dngerous to exclude ss port while the topology is invlid, so we forbid the user from ttempting it to void ny potentil loss of dt ccess. User response: commnd. 741 CPU missing Correct the topology, then retry the Explntion: A CPU tht ws previously present hs not been detected. The CPU might not be correctly instlled or it might hve filed. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Select one of the following ctions: v v v If removing the CPU ws deliberte, follow the mngement GUI recommended ctions to mrk the hrdwre chnge s intentionl. If it is not possible to isolte the problem, use the remove nd replce procedures to replce the CPU. Replce the system bord. 746 Technicin port connection invlid. Explntion: The code hs detected more thn one MAC ddress though the connection, or the DHCP hs given out more thn one ddress. The code thus believes there is switch ttched. User response: 1. Plug cble from the technicin port to switch, nd plug 2 or more mchines into tht switch. They must hve IP ddresses in the rnge Request DHCP lese to trigger the detection. 747 The Technicin port is being used. Explntion: used The Technicin port is ctive nd being User response: No service ction is required. Use the worksttion to configure the node. 768 Ambient temperture wrning. Explntion: The mbient temperture of the node is close to the point where it stops performing I/O nd enters service stte. The node is currently continuing to operte. Dt: v A text string identifying the therml sensor reporting the wrning level nd the current temperture in degrees (Celsius). User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Check the temperture of the room nd correct ny ir conditioning or ventiltion problems. 3. Check the irflow round the system to mke sure no vents re blocked. Possible Cuse-FRUs or other cuse: v None 769 CPU temperture wrning. Explntion: The temperture of the CPU within the node is close to the point where the node stops performing I/O nd enters service stte. The node is currently continuing to operte. This is most likely n 186 SAN Volume Controller: Troubleshooting Guide

205 mbient temperture problem, but it might be hrdwre problem. Dt: v A text string identifying the therml sensor reporting the wrning level nd the current temperture in degrees (Celsius). User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Check the temperture of the room nd correct ny ir conditioning or ventiltion problems. 3. Check the irflow round the system. Ensure no vents re blocked. 4. Mke sure the node fns re opertionl. 5. If the error is still reported, replce the node s CPU. Possible Cuse FRUs or other cuse: v CPU 770 Shutdown temperture reched Explntion: The node temperture hs reched the point t which it is must shut down to protect electronics nd dt. This is most likely n mbient temperture problem, but it could be hrdwre issue. Dt: v A text string identifying the therml sensor reporting the wrning level nd the current temperture in degrees (Celsius). User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Check the temperture of the room nd correct ny ir conditioning or ventiltion problems. 3. Check the irflow round the system nd mke sure no vents re blocked. Possible Cuse-FRUs or other cuse: v CPU 779 Bttery is missing Explntion: The bttery is not instlled in the system. User response: No service ction required, use the console to mnge the node nd fix the hrdwre. You cn power up the system without the bttery instlled. 780 Bttery hs filed Explntion: 1. The bttery hs filed. 2. The bttery is pst the end of its useful life. 3. The bttery filed to provide power on previous occsion nd is therefore, regrded s unfit for its purpose. User response: No service ction required, use the console to mnge the node. 781 Bttery is below the minimum operting temperture Explntion: The bttery cnnot perform the required function becuse it is below the minimum operting temperture. This error is reported only if the bttery subsystem cnnot provide full protection. An inbility to chrge is not reported if the combined chrge vilble from ll instlled btteries cn provide full protection t the current chrge levels. User response: No service ction required, use the console to mnge the node. Wit for the bttery to wrm up. 782 Bttery is bove the mximum operting temperture Explntion: The bttery cnnot perform the required function becuse it is bove the mximum operting temperture. This error is reported only if the bttery subsystem cnnot provide full protection. An inbility to chrge is not reported if the combined chrge vilble from ll instlled btteries cn provide full protection t the current chrge levels. User response: No service ction required, use the console to mnge the node. Wit for the bttery to cool down. 783 Bttery communictions error Explntion: A bttery is instlled, but communictions vi I2C re not functioning. This might be either fult in the bttery unit or fult in the bttery bckplne. User response: No service ction required, use the console to mnge the node. Conduct the corrective service procedure described in 1108 on pge 201. Chpter 7. Dignosing problems 187

206 Bttery is nering end of life Explntion: The bttery is ner the end of its useful life. You should replce it t the erliest convenient opportunity. This might be either fult in the bttery unit or fult in the bttery bckplne. User response: No service ction required, use the console to mnge the node. Replce the bttery. 785 Bttery cpcity is reduced becuse of cell imblnce Explntion: The chrge levels of the cells within the bttery pck re out of blnce. Some cells become fully chrged before others, which cuses chrging to terminte erly, before the entire bttery pck is fully chrged. Ending rechrging premturely effectively reduces the vilble cpcity of the pck. Circuitry within the bttery pck corrects such errors normlly, but cn tke tens of hours to complete. If this error is not fixed fter 24 hours, or if the error reoccurs fter it fixes itself, the error is likely indictive of problem in the bttery cells. In such cse, replce the bttery pck. User response: No service ction required, use the console to mnge the node. Wit for the cells to blnce. 786 Bttery VPD checksum error Explntion: The checksum on the vitl product dt (VPD) stored in the bttery EEPROM is incorrect. User response: No service ction required, use the console to mnge the node. Replce the bttery. 787 Bttery is t hrdwre revision level not supported by the current code level Explntion: The bttery currently instlled is t hrdwre revision level tht is not supported by the current code level. User response: No service ction required, use the console to mnge the node. Either updte the code level to one tht supports the currently instlled bttery or replce the bttery with one tht is supported by the current code level. 803 Fibre Chnnel dpter not working Explntion: A problem hs been detected on the node s Fibre Chnnel (FC) dpter. This node error is reported only on SAN Volume Controller 2145-CG8 or older nodes. User response: Follow troubleshooting procedures to fix the hrdwre. 818 Unble to recover the service controller flsh disk. Explntion: A nonrecoverble error occurred when ccessing the service controller persistent memory. User response: 1. Restrt the node nd see if it recovers. 2. Replce the field replceble units (FRUs) in the order listed. Possible Cuse-FRUs or other cuse: v Service controller v Service controller cble 820 Cnister type is incomptible with enclosure model Explntion: The node cnister hs detected tht it hs hrdwre type tht is not comptible with the control enclosure MTM, such s node cnister type 300 in n enclosure with MTM This is n expected condition when control enclosure is being upgrded to different type of node cnister. User response: 1. Check tht ll the upgrde instructions hve been followed completely. 2. Use the mngement GUI to run the recommended ctions for the ssocited service error code. Possible Cuse-FRUs or other cuse: v None 840 Unsupported hrdwre chnge detected. Explntion: A chnge hs been detected to this node s hrdwre configurtion. The new configurtion is not supported by the node cnister softwre. User ction is required to repir the hrdwre or updte the softwre. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Follow the procedure for getting node nd clustered-system informtion. A chnge to the hrdwre configurtion is expected. 188 SAN Volume Controller: Troubleshooting Guide

207 If the hrdwre configurtion is unexpectedly reduced, mke sure the component hs not been unseted. Hrdwre replcement might be necessry. 4. If new hrdwre component is shown s unsupported, check the softwre version required to support the hrdwre component. Upgrde the softwre to version tht supports the hrdwre. If the hrdwre detected does not mtch the expected configurtion, replce the hrdwre component tht is reported incorrectly. Possible Cuse-FRUs or other cuse: v One of the optionl hrdwre components might require replcement 841 Supported hrdwre chnge detected. Explntion: A chnge hs been detected in the node hrdwre configurtion. The new configurtion is supported by the node softwre. The new configurtion does not become ctive until it is ctivted. A node configurtion is remembered only while it is ctive in system. This node error is therefore resolved using the mngement GUI. User response: 1. Use the mngement GUI to run the recommended ctions for the ssocited service error code. Use the directed mintennce to ccept or reject the new configurtion. 850 The cnister bttery is reching the end of its useful life. Explntion: The cnister bttery is reching the end of its useful life. It should be replced within week of the node error first being reported. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Replce the node cnister bttery by using the remove nd replce procedures. Possible Cuse-FRUs or other cuse: v Cnister bttery 860 Fibre Chnnel network fbric is too big. Explntion: The number of Fibre Chnnel (FC) logins mde to the node exceeds the llowed limit. The node continues to operte, but only communictes with the logins mde before the limit ws reched. The order in which other devices log into the node cnnot be determined, so the node s FC connectivity might vry fter ech restrt. The connection might be with host systems, other storge systems, or with other nodes. This error might be the reson the node is unble to prticipte in system. The number of llowed logins per node is Dt: v None User response: This error indictes problem with the Fibre Chnnel fbric configurtion. It is resolved by reconfiguring the FC switch: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Rezone the FC network so only the ports the node needs to connect to re visible to it. Possible Cuse-FRUs or other cuse: v None 870 Too mny cluster cretions mde on node Explntion: Too mny SAN Volume Controller clustered systems hve been creted on this node. The count of the number of clustered systems creted on the node is stored within the node service controller. Dt: v None User response: 1. Try to crete the clustered system on different node. 2. Replce the service controller using the remove nd replce procedures. Possible cuse-frus or other cuse: v Service controller 871 Filed to increment cluster ID Explntion: The clustered system crete option filed becuse the clustered system, which is stored in the service controller, could not be updted. Dt: v None User response: 1. Try to crete the clustered system on different node. 2. Replce the service controller using the remove nd replce procedures. Possible cuse-frus or other cuse: v Service controller Chpter 7. Dignosing problems 189

208 Request to cluster rejected. Explntion: A cndidte node could not be dded to the clustered system. The node contins hrdwre or firmwre tht is not supported in the clustered system. Dt: This node error nd extr dt is viewble through sinfo lsservicesttus on the cndidte node only. The extr dt lists full set of feture codes tht re required by the node to run in the clustered system. User response: v Choose different cndidte tht is comptible with the clustered system. v Upgrde the clustered system to code tht is supported by ll components. v Do not dd cndidte to the clustered system. v Where pplicble, remove nd replce the hrdwre tht is preventing the cndidte from joining the clustered system. Possible Cuse FRUs or other cuse. For informtion on feture codes vilble, see the SAN Volume Controller nd Storwize fmily Chrcteristic Interoperbility Mtrix on the support website: Attempting recovery fter loss of stte dt. Explntion: During strtup, the node cnnot red its stte dt. It reports this error while witing to be dded bck into clustered system. If the node is not dded bck into clustered system within set time, node error 578 is reported. User response: 1. Allow time for recovery. No further ction is required. 2. Keep monitoring in cse the error chnges to error code Too mny Fibre Chnnel logins between nodes. Explntion: The system hs determined tht the user hs zoned the fbric such tht this node hs received more thn 16 unmsked logins originting from nother node or node cnister - this cn be ny non service mode node or cnister in the locl cluster or in remote cluster with prtnership. An unmsked login is from port whose corresponding bit in the FC port msk is '1'. If the error is rised ginst node in the locl cluster, then it is the locl FC port msk tht is pplied. If the error is rised ginst node in remote cluster, then it is the prtner FC port msks from both clusters tht pply. More thn 16 logins is not supported configurtion s it increses internode communiction nd cn ffect bndwidth nd performnce. For exmple, if node A hs 8 ports nd node B hs 8 ports where the nodes re in different clusters, if node A hs prtner FC port msk of nd node B hs prtner FC port msk of there re 4 unmsked logins possible (1,7 1,8 2,7 2,8). Fbric zoning my be used to reduce this mount further, i.e. if node B port 8 is removed from the zone there re only 2 (1,7 nd 2,7). The combintion of msks nd zoning must leve 16 or fewer possible logins. Note: This count includes both FC nd Fibre Chnnel over Ethernet (FCoE) logins. The log-in count will not include msked ports. When this event is logged. the cluster id nd node id of the first node whose logins exceed this limit on the locl node will be reported, s well s the WWNN of sid node. If logins chnge, the error is utomticlly fixed nd nother error is logged if pproprite (this my or my not choose the sme node to report in the sense dt if the sme node is still over the mximum llowed). Dt Text string showing v WWNN of the other node v Cluster ID of other node v Arbitrry node ID of one other node tht is logged into this node. (node ID s it ppers in lsnode) User response: The error is resolved by either re-configuring the system to chnge which type of connection is llowed on port, or by chnging the SAN fbric configurtion so ports re not in the sme zone. A combintion of both options my be used. The system reconfigurtion is to chnge the Fibre Chnnel ports msk to reduce which ports cn be used for internode communiction. The locl Fibre Chnnel port msk should be modified if the cluster id reported mtches the cluster id of the node logging the error. The prtner Fibre Chnnel port msk should be modified if the cluster id reported does not mtch the cluster id of the node logging the error. The prtner Fibre Chnnel port msk my need to be chnged for one or both clusters. SAN fbric configurtion is set using the switch configurtion utilities. Use the lsfbric commnd to view the current number of logins between nodes. Possible Cuse-FRUs or other cuse: v None Service error code 190 SAN Volume Controller: Troubleshooting Guide

209 Filed to crete remote IP connection. Explntion: Despite request to crete remote IP repliction port connection, the ction hs filed or timed out. User response: Fix the remote IP link so tht trffic cn flow correctly. Once the connection is mde, the error will uto-correct. 920 Unble to perform cluster recovery becuse of lck of cluster resources. Explntion: The node is looking for quorum of resources which lso require cluster recovery. User response: Contct IBM technicl support. 921 Unble to perform cluster recovery becuse of lck of cluster resources. Explntion: The node does not hve sufficient connectivity to other nodes or quorum device to form cluster. If disster hs occurred nd the nodes t the other site cnnot be recovered, then it is possible to llow the nodes t the surviving site to form system using locl storge. User response: Repir the fbric or quorum device to estblish connectivity. As lst resort when the nodes t the other site cnnot be recovered, then it is possible to llow the nodes t the surviving site to form system using locl site storge s described below: To void dt corruption ensure tht ll host servers tht were previously ccessing the system hve hd ll volumes un-mounted or hve been rebooted. Ensure tht the nodes t the other site re not opertionl nd re unble to form system in the future. After invoking this commnd full re-synchroniztion of ll mirrored volumes will be performed when the other site is recovered. This is likely to tke mny hours or dys to complete. Contct IBM support personnel if you re unsure. Note: Before continuing confirm tht you hve tken the following ctions - filure to perform these ctions cn led to dt corruption tht will be undetected by the system but will ffect host pplictions. 1. All host servers tht were previously ccessing the system hve hd ll volumes un-mounted or hve been rebooted. 2. Ensure tht the nodes t the other site re not operting s system nd ctions hve been tken to prevent them from forming system in the future. After these ctions hve been tken the stsk overridequorum cn be used to llow the nodes t the surviving site to form system using locl storge. 950 Specil upgrde mode. Explntion: User response: Specil upgrde mode. None. 990 Cluster recovery hs filed. Explntion: User response: Cluster recovery hs filed. Contct IBM technicl support Automtic cluster recovery hs run. Explntion: blocked. User response: All cluster configurtion commnds re Cll your softwre support center. Cution: You cn unblock the configurtion commnds through the cluster GUI, but you must first consult with your softwre support to void corrupting your cluster configurtion. Possible Cuse-FRUs or other: v None 1002 Event log full. Explntion: User response: the strt MAP. Event log full. Possible Cuse-FRUs or other: v Unfixed errors in the log. To fix the errors in the event log, go to 1007 Cnister to cnister communiction error. Explntion: A cnister to cnister communiction error cn pper when one cnister cnnot communicte with the other. User response: Reset the pssive cnister, nd then try reseting the ctive cnister. If neither resolve the lert, try replcing the pssive cnister, nd then the other cnister. A cnister cn be sfely reseted or replced while the system is in production. Mke sure tht the other cnister is the ctive node before removing this cnister. It is preferble tht this cnister shuts down completely before removing it, but it is not required. 1. Reset the pssive cnister ( filover is not required). 2. Reset the second cnister ( filover is required). 3. If necessry, replce the pssive cnister ( filover is not required). Chpter 7. Dignosing problems 191

210 If necessry, replce the ctive cnister ( filover is required). If second new cnister is not vilble, the previously removed cnister cn be used, s it pprently is not t fult. 5. An enclosure replcement might be necessry. Contct IBM support. Possible Cuse-FRUs or other: Cnister (95%) Enclosure (5%) 1009 DIMMs re incorrectly instlled. Explntion: DIMMs re incorrectly instlled. User response: Ensure tht memory DIMMs re spred evenly cross ll memory chnnels. 1. Shut down the node. 2. Ensure tht memory DIMMs re spred evenly cross ll memory chnnels. 3. Restrt the node. 4. If the error persists, replce system bord. Possible Cuse-FRUs or other: v None 1011 Fibre Chnnel dpter (4 port) in slot 1 is missing. Explntion: Fibre Chnnel dpter (4 port) in slot 1 is missing. User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v 4-port Fibre Chnnel host bus dpter (98%) v System bord (2%) G4 or A4 v 4-port Fibre Chnnel host bus dpter (90%) v PCI Express riser crd (8%) v System bord (2%) 1013 Fibre Chnnel dpter (4-port) in slot 1 PCI fult. Explntion: Fibre Chnnel dpter (4-port) in slot 1 PCI fult. User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v 4-port Fibre Chnnel host bus dpter (98%) v System bord (2%) G4 or A4 v 4-port Fibre Chnnel host bus dpter (80%) v PCI Express riser crd (10%) v System bord (10%) N/A 1014 Fibre Chnnel dpter in slot 1 is missing. Explntion: missing. Fibre Chnnel dpter in slot 1 is User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1015 Fibre Chnnel dpter in slot 2 is missing. Explntion: missing. Fibre Chnnel dpter in slot 2 is 192 SAN Volume Controller: Troubleshooting Guide

211 User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1016 Fibre Chnnel dpter (4 port) in slot 2 is missing. Explntion: Fibre Chnnel dpter (4 port) in slot 2 is missing. User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1017 Fibre Chnnel dpter in slot 1 PCI bus error. Explntion: error. Fibre Chnnel dpter in slot 1 PCI bus User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1018 Fibre Chnnel dpter in slot 2 PCI fult. Explntion: Fibre Chnnel dpter in slot 2 PCI fult. User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1019 Fibre Chnnel dpter (4-port) in slot 2 PCI fult. Explntion: Fibre Chnnel dpter (4-port) in slot 2 PCI fult. User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1020 The system bord service processor hs filed. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 522. See the detils of node error 522 for more informtion. User response: See node error Incorrect enclosure Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 500. See the detils of node error 500 for more informtion. Chpter 7. Dignosing problems 193

212 User response: See node error The detected memory size does not mtch the expected memory size. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 510. See the detils of node error 510 for more informtion. User response: See node error The 2145 system ssembly is filing. Explntion: The 2145 system ssembly is filing. User response: 1. Go to the light pth dignostic MAP nd perform the light pth dignostic procedures. 2. If the light pth dignostic procedure isoltes the FRU, mrk this error s fixed nd go to the repir verifiction MAP. If you hve just replced FRU but it hs not corrected the problem, ensure tht the FRU is instlled correctly nd go to the next step. 3. Replce the system bord or frme ssembly s indicted in the Possible Cuse list below. 4. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of online, go to the strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to the repir verifiction MAP. Possible Cuse-FRUs or other: G4, 2145-CF8, or 2145-CG8 v The FRUs tht re indicted by the Light pth dignostics (98%) v System bord (2%) 1026 System bord device problem. Explntion: System bord device problem. User response: The ction depends on the extr dt tht is provided with the node error nd the light pth dignostics. Possible Cuse-FRUs or other: v Vrible 1027 Unble to updte BIOS settings. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 524. See the detils of node error 524 for more informtion. User response: See node error System bord service processor filed. Explntion: System bord service processor filed. User response: Complete the following steps: 1. Shut down the node. 2. Remove the min power cble. 3. Wit for the lights to stop flshing. 4. Plug in the power cble. 5. Wit for node to boot. 6. If the node still reports the error, replce system bord. Possible Cuse-FRUs or other: v System bord 1029 Enclosure VPD is unvilble or invlid. Explntion: Enclosure VPD is unvilble or invlid. User response: Overwrite the enclosure VPD or replce the power interposer bord. Possible Cuse-FRUs or other: PIB crd (10%) Other: No FRU (90%) 1030 The internl disk of node hs filed. Explntion: An error hs occurred while ttempting to red or write dt to the internl disk of one of the nodes in the cluster. The disk hs filed. User response: Determine which node's internl disk hs filed using the node informtion in the error. Replce the FRUs in the order shown. Mrk the error s fixed. Possible Cuse-FRUs or other: Node Cnister (100%) v disk drive (50%) v Disk controller (30%) v Disk bckplne (10%) v Disk signl cble (8%) v Disk power cble (1%) v System bord (1%) 1032 Fibre Chnnel dpter not working Explntion: A problem hs been detected on the node s Fibre Chnnel (FC) dpter. This node error is reported only on SAN Volume Controller 2145-CG8 or older nodes. User response: Follow troubleshooting procedures to fix the hrdwre. 194 SAN Volume Controller: Troubleshooting Guide

213 If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. Possible Cuse-FRUs or other cuse: v None 1034 Cnister fult type 2 Explntion: There is cnister internl error. User response: Reset the cnister, nd then replce the cnister if the error continues. Possible Cuse-FRUs or other: Cnister (80%) Other: No FRU (20%) 1035 Boot drive problems Explntion: Boot drive problems User response: Complete the following steps: 1. Look t boot drive view to determine the problems. 2. Run the commnds lsnodebootdrive / lsbootdrive to disply sttus for ech slot for users nd DMPs to dignose nd repir problems. 3. If you pln to move ny drives, shut down the node if booted yes is shown for tht drive in the boot drive view (lsbootdrive). After you move the drives, different node error will probbly be displyed for you to work on. 4. If you pln to set the seril number of the system bord, see stsk chvpd. 5. If there is still no usble persistent dt on the boot drives, then contct IBM Remote Technicl Support. Possible Cuse-FRUs or other: v System drive 1036 The enclosure identity cnnot be red. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 509. See the detils of node error 509 for more informtion. User response: See node error Cnister filure, cnister replcement required. Explntion: required. User response: Cnister filure, cnister replcement Replce the cnister. A cnister cn be sfely replced while the system is in production. Mke sure tht the other cnister is the ctive node before removing this cnister. It is preferble tht this cnister shuts down completely before removing it, but it is not required. Possible cuse-frus or other: Interfce crd (50%) SFP (20%) Cnister (20%) Internl interfce crd cble (10%) 1040 A flsh module error hs occurred fter successful strt of Explntion: Note: The node contining the flsh module hs not been rejected by the cluster. User response: 1. Replce the FRUs below in the order listed 2. Check node sttus. If ll nodes show sttus of Online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of Online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Service controller (50%) v Service controller cble (50%) G4 or A4 Service controller (100%) 1044 A service controller red filure occurred. Explntion: A service controller red filure occurred. User response: 1. Replce the FRUs below in the order listed. 2. Check node sttus. If ll nodes show sttus of Online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of Online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Service controller (50%) v Service controller cble (50%) Chpter 7. Dignosing problems 195

214 G4 or A4 Service controller (100%) 1052 Incorrect type of uninterruptible power supply detected Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 587. See the detils of node error 587 for more informtion. User response: See node error Fibre Chnnel dpter in slot 1 dpter present but filed. Explntion: Fibre Chnnel dpter in slot 1 dpter present but filed. User response: 1. Replce the Fibre Chnnel dpter. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1055 Fibre Chnnel dpter (4 port) in slot 1 dpter present but filed. Explntion: Fibre Chnnel dpter (4 port) in slot 1 dpter present but filed. User response: 1. Exchnge the FRU for new FRU. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4, A4, 2145-CF8, or 2145-CG8 v 4-port Fibre Chnnel host bus dpter (100%) 1056 Fibre Chnnel dpter in slot 2 dpter present but filed. Explntion: Fibre Chnnel dpter in slot 2 dpter present but filed. User response: 1. Replce the Fibre Chnnel dpter. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1057 Fibre Chnnel dpter (4 port) in slot 2 dpter present but filed. Explntion: Fibre Chnnel dpter (4 port) in slot 2 dpter present but filed. User response: 1. Exchnge the FRU for new FRU. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1060 One or more Fibre Chnnel ports on the 2072 re not opertionl. Explntion: One or more Fibre Chnnel ports on the 2072 re not opertionl. User response: 1. Go to MAP 5600: Fibre Chnnel to isolte nd repir the problem. 2. Go to the repir verifiction MAP. Possible Cuse-FRUs or other: v Fibre Chnnel cble (80%) v Smll Form-fctor Pluggble (SFP) connector (5%) v 4-port Fibre Chnnel host bus dpter (5%) 196 SAN Volume Controller: Troubleshooting Guide

215 Other: v Fibre Chnnel network fbric (10%) 1061 Fibre Chnnel ports re not opertionl. Explntion: Fibre Chnnel ports re not opertionl. User response: An offline port cn hve mny cuses nd so it is necessry to check them ll. Strt with the esiest nd lest intrusive possibility such s resetting the Fibre Chnnel or FCoE port vi CLI commnd. Possible Cuse-FRUs or other: Externl (cble, HBA/CNA, switch, nd so on) (75%) SFP (10%) Interfce (10%) Cnister (5%) 1065 One or more Fibre Chnnel ports re running t lower thn the previously sved speed. Explntion: The Fibre Chnnel ports will normlly operte t the highest speed permitted by the Fibre Chnnel switch, but this speed might be reduced if the signl qulity on the Fibre Chnnel connection is poor. The Fibre Chnnel switch could hve been set to operte t lower speed by the user, or the qulity of the Fibre Chnnel signl hs deteriorted. User response: v Go to MAP 5600: Fibre Chnnel to resolve the problem. Possible Cuse-FRUs or other: Node Cnister (100%) v Fibre Chnnel cble (50%) v Smll Form-fctor Pluggble (SFP) connector (20%) v 4-port Fibre Chnnel host bus dpter (5%) Other: v Fibre Chnnel switch, SFP connector, or GBIC (25%) 1067 Fn fult type 1 Explntion: User response: The fn hs filed. Replce the fn. Possible Cuse-FRUs or other: Fn (100%) 1068 Fn fult type 2 Explntion: The fn is missing. User response: Reset the fn, nd then replce the fn if reseting the fn does not correct the error. Note: If replcing the fn does not correct the error, then the cnister will need to be replced. Possible Cuse-FRUs or other: Fn (80%) Other: No FRU (20%) 1083 Unrecognized node error Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 562. See the detils of node error 562 for more informtion. User response: See node error System bord device exceeded temperture threshold. Explntion: System bord device exceeded temperture threshold. User response: Complete the following steps: 1. Check for externl ir flow blockges. 2. Remove the top of the mchine cse nd check for missing bffles, dmged het sinks, or internl blockges. 3. If problem persists, follow the service instructions for replcing the system bord FRU in question. Possible Cuse-FRUs or other: v Vrible 1085 PCI Riser crd exceeded temperture threshold. Explntion: threshold. PCI Riser crd exceeded temperture User response: Complete the following steps: 1. Check irflow. 2. Remove the top of the mchine cse nd check for missing bffles or internl blockges. 3. Check for fulty PCI crds nd replce s necessry. 4. If problem persists, replce PCI Riser FRU. Possible Cuse-FRUs or other: v None Chpter 7. Dignosing problems 197

216 Shutdown temperture threshold exceeded Explntion: exceeded. Shutdown temperture threshold User response: Inspect the enclosure nd the enclosure environment. 1. Check environmentl temperture. 2. Ensure tht ll of the components re instlled or tht there re fillers in ech by. 3. Check tht ll of the fns re instlled nd operting properly. 4. Check for ny obstructions to irflow, proper clernce for fresh inlet ir, nd exhust ir. 5. Hndle ny specific obstructed irflow errors tht re relted to the drive, the bttery, nd the power supply unit. 6. Bring the system bck online. If the system performed hrd shutdown, the power must be removed nd repplied. Possible Cuse-FRUs or other: Cnister (2%) Bttery (1%) Power supply unit (1%) Drive (1%) Other: Environment (95%) 1089 One or more fns re filing. Explntion: One or more fns re filing. User response: 1. Determine the filing fn(s) from the fn indictor on the system bord or from the text of the error dt in the log. The reported fn for the A4, 2145-CF8, or 2145-CG8 mtches the fn module position. For the G4, if you hve determined the filing fn number from the error dt in the log, use the following list to determine the position of the fn module to replce. Ech fn module contins two fns. 2. Exchnge the FRU for new FRU. 3. Go to repir verifiction MAP. v Fn number: Fn module position v 1or2:1 v 3or4:2 v 5or6:3 v 7or8:4 v 9 or 10:5 v 11 or 12:6 Possible Cuse-FRUs or other: G4, A4, 2145-CF8, or 2145-CG8 v Fn module (100%) 1090 One or more fns (40x40x28) re filing. Explntion: One or more fns (40x40x28) re filing. User response: 1. Determine the filing fn(s) from the fn indictor on the system bord or from the text of the error dt in the log. 2. If ll fns on the fn bckplne re filing or if no fn fult lights re illuminted, verify tht the cble between the fn bckplne nd the system bord is connected. 3. Exchnge the FRU for new FRU. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1091 One or more fns (40x40x56) re filing. Explntion: One or more fns (40x40x56) re filing. User response: 1. Determine the filing fn(s) from the fn indictor on the system bord or from the text of the error dt in the log. 2. If ll fns on the fn bckplne re filing or if no fn fult lights re illuminted, verify tht the cble between the fn bckplne nd the system bord is connected. 3. Exchnge the FRU for new FRU. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 v N/A 1092 The temperture soft or hrd shutdown threshold of the 2072 hs been exceeded. The 2072 hs utomticlly powered off. Explntion: The temperture soft or hrd shutdown threshold of the 2072 hs been exceeded. The 2072 hs utomticlly powered off. User response: 198 SAN Volume Controller: Troubleshooting Guide

217 Ensure tht the operting environment meets specifictions. 2. Ensure tht the irflow is not obstructed. 3. Ensure tht the fns re opertionl. 4. Go to the light pth dignostic MAP nd perform the light pth dignostic procedures. 5. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of online, go to the strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to the repir verifiction MAP. Possible Cuse-FRUs or other: Node Cnister (100%) v The FRU tht is indicted by the Light pth dignostics (25%) v System bord (5%) Other: System environment or irflow blockge (70%) 1093 The internl temperture sensor of the 2145 hs reported tht the temperture wrning threshold hs been exceeded. Explntion: The internl temperture sensor of the 2145 hs reported tht the temperture wrning threshold hs been exceeded. User response: 1. Ensure tht the internl irflow of the node hs not been obstructed. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to the strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4, A4, 2145-CF8, or 2145-CG8 v Fn module (25%) v System bord (5%) Other: Airflow blockge (70%) 1094 The mbient temperture threshold hs been exceeded. Explntion: The mbient temperture threshold hs been exceeded. User response: 1. Check tht the room temperture is within the limits llowed. 2. Check for obstructions in the ir flow. 3. Mrk the errors s fixed. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: None Other: System environment (100%) 1096 A Power Supply Unit is missing or hs filed. Explntion: One of the two power supply units in the node is either missing or hs filed. NOTE: This error is reported when hot-swp power supply is removed from n ctive node, so it might be reported when fulty power supply is removed for replcement. Both the missing nd fulty conditions report this error code. User response: Error code 1096 is reported when the power supply either cnnot be detected or reports n error. 1. Ensure tht the power supply is seted correctly nd tht the power cble is ttched correctly to both the node nd to the 2145 UPS-1U. 2. If the error hs not been utomticlly mrked fixed fter two minutes, note the sttus of the three LEDs on the bck of the power supply. For the 2145-CG8 or 2145-CF8, the AC LED is the top green LED, the DC LED is the middle green LED nd the error LED is the bottom mber LED. 3. If the power supply error LED is off nd the AC nd DC power LEDs re both on, this is the norml condition. If the error hs not been utomticlly fixed fter two minutes, replce the system bord. 4. Follow the ction specified for the LED sttes noted in the tble below. 5. If the error hs not been utomticlly fixed fter two minutes, contct support. 6. Go to repir verifiction MAP. Error,AC,DC:Action ON,ON or OFF,ON or OFF:The power supply hs fult. Replce the power supply. Chpter 7. Dignosing problems 199

218 OFF,OFF,OFF:There is no power detected. Ensure tht the power cble is connected t the node nd 2145 UPS-1U. If the AC LED does not light, check the sttus of the 2145 UPS-1U to which the power supply is connected. Follow MAP UPS-1U if the UPS-1U is showing no power or n error; otherwise, replce the power cble. If the AC LED still does not light, replce the power supply. OFF,OFF,ON:The power supply hs fult. Replce the power supply. OFF,ON,OFF:Ensure tht the power supply is instlled correctly. If the DC LED does not light, replce the power supply. Possible Cuse-FRUs or other: Filed PSU: v Power supply (90%) v Power cble ssembly (5%) v System bord (5%) Missing PSU: v Power supply (19%) v System bord (1%) v Other: Power supply not correctly instlled (80%) 1097 A Power Supply Unit reports no A/C power. Explntion: One of the two power supply units in the node is reporting tht no min power is detected. User response: 1. Ensure tht the power supply is ttched correctly to both the node nd to the 2145 UPS-1U. 2. If the error hs not been utomticlly mrked fixed fter two minutes, note the sttus of the three LEDs on the bck of the power supply. For the 2145-CG8 or 2145-CF8, the AC LED is the top green LED, the DC LED is the middle green LED nd the error LED is the bottom mber LED. 3. If the power supply error LED is off nd the AC nd DC power LEDs re both on, this is the norml condition. If the error hs not been utomticlly fixed fter two minutes, replce the system bord. 4. Follow the ction specified for the LED sttes noted in the tble below. 5. If the error hs not been utomticlly fixed fter two minutes, contct support. 6. Go to repir verifiction MAP. Error,AC,DC:Action ON,ON or OFF,ON or OFF:The power supply hs fult. Replce the power supply. OFF,OFF,OFF:There is no power detected. Ensure tht the power cble is connected t the node nd 2145 UPS-1U. If the AC LED does not light, check whether the 2145 UPS-1U is showing ny errors. Follow MAP UPS-1U if the UPS-1U is showing n error; otherwise, replce the power cble. If the AC LED still does not light, replce the power supply. OFF,OFF,ON:The power supply hs fult. Replce the power supply. OFF,ON,OFF:Ensure tht the power supply is instlled correctly. If the DC LED does not light, replce the power supply. Possible Cuse-FRUs or other: v Power cble ssembly (85%) v UPS-1U ssembly (10%) v System bord (5%) 1099 Temperture exceeded wrning threshold Explntion: threshold. Temperture exceeded wrning User response: Inspect the enclosure nd the enclosure environment. 1. Check environmentl temperture. 2. Ensure tht ll of the components re instlled or tht there re fillers in ech by. 3. Check tht ll of the fns re instlled nd operting properly. 4. Check for ny obstructions to irflow, proper clernce for fresh inlet ir, nd exhust ir. 5. Wit for the component to cool. Possible Cuse-FRUs or other: Hrdwre component (5%) Other: Environment (95%) 1100 One of the voltges tht is monitored on the system bord is over the set threshold. Explntion: One of the voltges tht is monitored on the system bord is over the set threshold. User response: 1. See the light pth dignostic MAP. 2. If the light pth dignostic MAP does not resolve the issue, exchnge the frme ssembly. 3. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired 200 SAN Volume Controller: Troubleshooting Guide

219 s fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: 1101 One of the voltges tht is monitored on the system bord is over the set threshold. Explntion: One of the voltges tht is monitored on the system bord is over the set threshold. User response: 1. See the light pth dignostic MAP. 2. If the light pth dignostic MAP does not resolve the issue, exchnge the system bord ssembly. 3. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4, A4, 2145-CF8, or 2145-CG8 v Light pth dignostic MAP FRUs (98%) v System bord (2%) 1105 One of the voltges tht is monitored on the system bord is under the set threshold. Explntion: One of the voltges tht is monitored on the system bord is under the set threshold. User response: 1. Check the cble connections. 2. See the light pth dignostic MAP. 3. If the light pth dignostic MAP does not resolve the issue, exchnge the frme ssembly. 4. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP One of the voltges tht is monitored on the system bord is under the set threshold. Explntion: One of the voltges tht is monitored on the system bord is under the set threshold. User response: 1. Check the cble connections. 2. See the light pth dignostic MAP. 3. If the light pth dignostic MAP does not resolve the issue, exchnge the system bord ssembly. 4. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4, A4, 2145-CF8, or 2145-CG8 v Light pth dignostic MAP FRUs (98%) v System bord (2%) 1107 The bttery subsystem hs insufficient cpcity to sve system dt due to multiple fults. Explntion: This messge is n indiction of other problems to solve before the system cn successfully rechrge the btteries. User response: No service ction is required for this error, but other errors must be fixed. Look t other indictions to see if the btteries cn rechrge without being put into use Bttery bckplne cbling fulty or possible bttery bckplne requires replcing. Explntion: Fulty cbling or fulty bckplne re preventing the system from full communiction with nd control of the btteries. User response: Check the cbling to the bttery bckplne, mking sure tht ll the connectors re properly mted. Four signl cbles (EPOW, LPC, PWR_SENSE & LED) nd one power cble (which uses 12 red nd 12 blck hevy guge wires) re involved: v The EPOW cble runs to 20-pin connector t the front of the system plnr, which is the edge nerest the drive bys, ner the left side. To check tht this connector is mted properly, it is necessry to remove the plstic irflow bffle, which lifts up. Chpter 7. Dignosing problems 201

220 v v v v A number of wires run from the sme connector to the disk bckplne locted to the left of the bttery bckplne. The LPC cble runs to smll dpter bord tht is plugged into the bck of the system plnr between two PCI Express dptor crd cges. It is helpful to remove the left dpter crd cge when checking tht these connectors re mted properly. The PWR_SENSE cble runs to 24-pin connector t the bck of the system plnr between the PSUs nd the left dpter crd cge. Check the connections of both femle connector (to the system plnr) nd mle connector (to the connector from the top PSU). Agin, it cn be helpful to remove the left dpter crd cge to check the proper mting of the connectors. The power cble runs to the system plnr between the PSUs nd the left dpter crd cge. It is locted just in front of the PWR_SENSE connector. This cble hs both femle connector tht connects to the system plnr, nd mle connector tht mtes with the connector from the top PSU. Due to the bulk of this cble, cre must be tken to not disturb PWR_SENSE connections when dressing it wy in the spce between the PSUs nd the left dpter crd cge. The LED cble runs to smll PCB on the front bezel. The only consequence of this cble not being mted correctly is tht the LEDs do not work The power mngement bord detected voltge tht is outside of the set thresholds. Explntion: The power mngement bord detected voltge tht is outside of the set thresholds. User response: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145-CG8 or 2145-CF8 v Power supply unit (50%) v System bord (50%) G4 v Power bckplne (90%) v Power supply ssembly (5%) v System bord (5%) If no problems exist, replce the bttery bckplne s described in the service ction for You do not replce either bttery t this time. To verify tht the bttery bckplne works fter replcing it, check tht the node error is fixed Bttery or possibly bttery bckplne requires replcing. Explntion: Bttery or possibly bttery bckplne requires replcing. User response: Complete the following steps: 1. Replce the drive by bttery. 2. Check to see whether the node error is fixed. If not, replce the bttery bckplne. 3. To verify tht the new bttery bckplne is working correctly, check tht the node error is fixed. Possible Cuse-FRUs or other: v Drive by bttery (95%) v Bttery bckplne (5%) 1111 Btteries hve insufficient chrge. Explntion: The insufficient chrge messge cn pper for vrious resons such s the bttery is chrging; the bttery is missing or hs filed; there is communiction error, or there hs been n over temperture event. User response: This node error cn be corrected by correcting ech of the underlying bttery problems. 1. If bttery is missing, replce the bttery. 2. If bttery is filed, replce the bttery. 3. If bttery is chrging, this error should go wy when the bttery is chrged. 4. If bttery is hving communiction error (comm errror), try to reset the bttery s described in the replcement procedure. If reseting the bttery does not correct the problem, replce the bttery. 5. If bttery is too hot, the system cn be strted fter it hs cooled. Inspect the bttery for dmge fter n over-temperture event. Possible Cuse - FRUs or other: If both btteries hve errors, bttery chrging might be underwy. (No FRU) 202 SAN Volume Controller: Troubleshooting Guide

221 If both btteries hve errors tht do not resolve fter sufficient time to chrge, bttery chrging might be impired, such s by fulty bttery bckplne FRU. Communiction errors re often correctble by reseting the bttery or by llowing the temperture of the bttery to cool without the need to replce the bttery. (No FRU) If bttery is missing or filed, the solution is to replce the bttery FRU. Bttery (50%) Other: No FRU (50%) 1112 Enclosure bttery is missing. Explntion: Enclosure bttery is missing. User response: Instll bttery in the missing slot. If the bttery is present in the slot, reset the bttery. Attention: Do not reset bttery unless the other bttery hs enough chrge, or dt loss might occur. Possible Cuse-FRUs or other: Bttery (95%) Other: No FRU (5%) 1114 Enclosure bttery fult type 1 Explntion: Enclosure bttery fult type 1. User response: Replce the bttery. Possible Cuse-FRUs or other: Bttery (100%) 1115 Enclosure Bttery fult type 4 Explntion: Enclosure Bttery fult type 4. User response: Reset the bttery. Replce the bttery if the error continues. Note: Do not reset bttery unless the other bttery hs enough chrge, or dt loss might occur. Possible Cuse-FRUs or other: Bttery (95%) Other: Bd connection (5%) 1120 A high speed SAS dpter is missing. Explntion: This node hs detected tht high speed SAS dpter tht ws previously instlled is no longer present. User response: If the high speed SAS dpter ws delibertely removed, mrk the error fixed. Otherwise, the high speed SAS dpter hs filed nd must be replced. In the sequence shown, exchnge the FRUs for new FRUs. Go to the repir verifiction MAP. Possible Cuse-FRUs or other: 1. High speed SAS dpter (90%) 2. System bord (10%) 1121 A high speed SAS dpter hs filed. Explntion: A fult hs been detected on high speed SAS dpter. User response: In the sequence shown, exchnge the FRUs for new FRUs. Go to the repir verifiction MAP. Possible Cuse-FRUs or other: 1. High speed SAS dpter (90%) 2. System bord (10%) 1122 A high speed SAS dpter error hs occurred. Explntion: The high speed SAS dpter hs detected PCI bus error nd requires service before it cn be restrted. The high speed SAS dpter filure hs cused ll of the flsh drives tht were being ccessed through this dpter to go Offline. User response: If this is the first time tht this error hs occurred on this node, complete the following steps: 1. Power off the node. 2. Reset the high speed SAS dpter crd. 3. Power on the node. 4. Submit the lsmdisk tsk nd ensure tht ll of the flsh drive mnged disks tht re locted in this node hve sttus of Online. If the sequence of ctions bove hs not resolved the problem or the error occurs gin on the sme node, complete the following steps: 1. In the sequence shown, exchnge the FRUs for new FRUs. 2. Submit the lsmdisk tsk nd ensure tht ll of the flsh drive mnged disks tht re locted in this node hve sttus of Online. 3. Go to the repir verifiction MAP. Chpter 7. Dignosing problems 203

222 Possible Cuse-FRUs or other: 1. High speed SAS dpter (90%) 2. System bord (10%) 1124 Power Supply Unit fult type 1 Explntion: A fult hs been detected on power supply unit (PSU). User response: Replce the PSU. Attention: To void losing stte nd dt from the node, use the stsk strtservice commnd to put the node into service stte so tht it no longer processes I/O. Then you cn remove nd replce the top power supply unit (PSU 2). This precution is due to limittion in the power-supply configurtion. Once the service ction is complete, run the stsk stopservice commnd to let the node rejoin the system. Possible Cuse-FRUs or other: PSU (100%) 1125 Power Supply Unit fult type 1 Explntion: supported. User response: version. The power supply unit (PSU) is not Replce the PSU with supported Attention: To void losing stte nd dt from the node, use the stsk strtservice commnd to put the node into service stte so tht it no longer processes I/O. Then you cn remove nd replce the top power supply unit (PSU 2). This precution is due to limittion in the power-supply configurtion. Once the service ction is complete, run the stsk stopservice commnd to let the node rejoin the system. Possible Cuse-FRUs or other: PSU (100%) 1126 Power Supply Unit fult type 2 Explntion: (PSU). A fult exists on the power supply unit User response: 1. Reset the PSU in the enclosure. Attention: To void losing stte nd dt from the node, use the stsk strtservice commnd to put the node into service stte so tht it no longer processes I/O. Then you cn remove nd replce the top power supply unit (PSU 2). This precution is due to limittion in the power-supply configurtion. Once the service ction is complete, run the stsk stopservice commnd to let the node rejoin the system. 2. If the fult is not resolved, replce the PSU. Possible Cuse-FRUs or other: 1. No Prt (30%) 2. PSU (70 %) 1128 Power Supply Unit missing Explntion: The power supply unit (PSU) is not seted in the enclosure, or no PSU is instlled. User response: 1. If no PSU is instlled, instll PSU. 2. If PSU is instlled, reset the PSU in the enclosure. Attention: To void losing stte nd dt from the node, use the stsk strtservice commnd to put the node into service stte so tht it no longer processes I/O. Then you cn remove nd replce the top power supply unit (PSU 2). This precution is due to limittion in the power-supply configurtion. Once the service ction is complete, run the stsk stopservice commnd to let the node rejoin the system. Possible Cuse-FRUs or other: 1. No Prt (5%) 2. PSU (95%) Reset the power supply unit in the enclosure. Possible Cuse-FRUs or other: Power supply (100%) 1129 The node bttery is missing. Explntion: Instll new btteries to enble the node to join clustered system. User response: Instll bttery in bttery slot 1 (on the left from the rer) nd in bttery slot 2 (on the right). Leve the node running s you dd the btteries. Align ech bttery so tht the guide rils in the enclosure engge the guide ril slots on the bttery. Push the bttery firmly into the bttery by until it stops. The cm on the front of the bttery remins closed during this instlltion. To verify tht the new bttery works correctly, check tht the node error is fixed. After the node joins clustered system, use the lsnodebttery commnd to view informtion bout the bttery The node bttery requires replcing. Explntion: When bttery must be replced, you get this messge. The proper response is to instll new btteries. 204 SAN Volume Controller: Troubleshooting Guide

223 User response: Bttery 1 is on the left (from the rer), nd bttery 2 is on the right. Remove the old bttery by disengging nd pulling down the cm hndle to lever out the bttery enough to pull the bttery from the enclosure. This service procedure is intended for filed or offline bttery. To prevent losing dt from bttery tht is online, run the svctsk chnodebttery -remove -bttery bttery_id node_id. Running the commnd verifies when it is sfe to remove the bttery. Instll new btteries in bttery slot 1 nd in bttery slot 2. Leve the node running s you dd the btteries. Align ech bttery so tht the guide rils in the enclosure engge the guide ril slots on the bttery. Push the bttery firmly into the bttery by until it stops. The cm on the front of the bttery remins closed during this instlltion. To verify tht the new bttery works correctly, check tht the node error is fixed. After the node joins clustered system, use the lsnodebttery commnd to view informtion bout the bttery Bttery conditioning is required but not possible. Explntion: possible. Bttery conditioning is required but not One possible cuse of this error is tht bttery reconditioning cnnot occur in clustered node if the prtner node is not online. User response: This error cn be corrected on its own. For exmple, if the prtner node comes online, the reconditioning begins. Possible Cuse-FRUs or other: Other: Wit, or ddress other errors A duplicte WWNN hs been detected. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 556. See the detils of node error 556 for more informtion. User response: See node error The 2145 UPS hs reported n mbient over temperture. Explntion: The 2145 UPS hs reported n mbient over temperture. The uninterruptible power supply switches to Bypss mode to llow the 2145 UPS to cool. User response: 1. Power off the nodes ttched to the 2145 UPS. 2. Turn off the 2145 UPS, nd then unplug the 2145 UPS from the min power source. 3. Ensure tht the ir vents of the 2145 UPS re not obstructed. 4. Ensure tht the ir flow round the 2145 UPS is not restricted. 5. Wit for t lest five minutes, nd then restrt the 2145 UPS. If the problem remins, check the mbient temperture. Correct the problem. Otherwise, exchnge the FRU for new FRU. 6. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 7. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145 UPS electronics unit (50%) Other: The system mbient temperture is outside the specifiction (50%) 1136 The 2145 UPS-1U hs reported n mbient over temperture. Explntion: The 2145 UPS-1U hs reported n mbient over temperture. User response: 1. Power off the node ttched to the 2145 UPS-1U. 2. Turn off the 2145 UPS-1U, nd then unplug the 2145 UPS-1U from the min power source. 3. Ensure tht the ir vents of the 2145 UPS-1U re not obstructed. 4. Ensure tht the ir flow round the 2145 UPS-1U is not restricted. 5. Wit for t lest five minutes, nd then restrt the 2145 UPS-1U. If the problem remins, check the mbient temperture. Correct the problem. Otherwise, exchnge the FRU for new FRU. 6. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 7. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145 UPS-1U ssembly (50%) Other: Chpter 7. Dignosing problems 205

224 The system mbient temperture is outside the specifiction (50%) 1138 Power supply unit input power filed. Explntion: Power Supply Unit input power filed. User response: Check the power cord. 1. Check tht the power cord is plugged in. 2. Check tht the wll power is good. 3. Replce the power cble. 4. Replce the power supply unit. Possible Cuse-FRUs or other: Power cord (20%) PSU (5%) Other: No FRU (75%) 1140 The 2145 UPS hs reported tht it hs problem with the input AC power. Explntion: The 2145 UPS hs reported tht it hs problem with the input AC power. User response: 1. Check the input AC power, whether it is missing or out of specifiction. Correct if necessry. Otherwise, exchnge the FRU for new FRU. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v 2145 UPS input power cble (10%) v Electronics ssembly (10%) Other: v The input AC power is missing (40%) v The input AC power is not in specifiction (40%) 1141 The 2145 UPS-1U hs reported tht it hs problem with the input AC power. Explntion: The 2145 UPS-1U hs reported tht it hs problem with the input AC power. User response: 1. Check the input AC power, whether it is missing or out of specifiction. Correct if necessry. Otherwise, exchnge the FRU for new FRU. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v 2145 UPS-1U input power cble (10%) v 2145 UPS-1U ssembly (10%) Other: v The input AC power is missing (40%) v The input AC power is not in specifiction (40%) 1145 The signl connection between 2145 nd its 2145 UPS is filing. Explntion: The signl connection between 2145 nd its 2145 UPS is filing. User response: 1. If other 2145s tht re using this uninterruptible power supply re reporting this error, exchnge the 2145 UPS electronics unit for new one. 2. If only this 2145 is reporting the problem, check the signl cble, exchnge the FRUs for new FRUs in the sequence shown. 3. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: G4 or 2145-CF8 or 2145-CG8 N/A 1146 The signl connection between 2145 nd its 2145 UPS-1U is filing. Explntion: The signl connection between 2145 nd its 2145 UPS-1U is filing. User response: 1. Exchnge the FRUs for new FRUs in the sequence shown. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired s fixed. If ny nodes do not show sttus of 206 SAN Volume Controller: Troubleshooting Guide

225 online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145-CF8 or 2145-CG8 v Power cble ssembly (40%) v 2145 UPS-1U ssembly (30%) v System bord (30%) G4 v Power cble ssembly (40%) v 2145 UPS-1U ssembly (30%) v System bord (30%) 1150 Dt tht the 2145 hs received from the 2145 UPS suggests the 2145 UPS power cble, the signl cble, or both, re not connected correctly. Explntion: Dt tht the 2145 hs received from the 2145 UPS suggests the 2145 UPS power cble, the signl cble, or both, re not connected correctly. User response: 1. Connect the cbles correctly. See your product's instlltion guide. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Configurtion error 1151 Dt tht the 2145 hs received from the 2145 UPS-1U suggests the 2145 UPS-1U power cble, the signl cble, or both, re not connected correctly. Explntion: Dt tht the 2145 hs received from the 2145 UPS-1U suggests the 2145 UPS-1U power cble, the signl cble, or both, re not connected correctly. User response: 1. Connect the cbles correctly. See your product's instlltion guide. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Configurtion error 1152 Incorrect type of uninterruptible power supply detected. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 587. See the detils of node error 587 for more informtion. User response: See node error A power domin error hs occurred. Explntion: Both 2145s of pir re powered by the sme uninterruptible power supply. User response: 1. List the 2145s of the cluster nd check tht 2145s in the sme I/O group re connected to different uninterruptible power supply. 2. Connect one of the 2145s s identified in step 1 to different uninterruptible power supply. 3. Mrk the error tht you hve just repired, fixed. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Configurtion error 1160 The output lod on the 2145 UPS exceeds the specifiction. Explntion: The 2145 UPS is reporting tht too much power is being drwn from it. The power overlod wrning LED, which is bove the lod level indictors, on the 2145 UPS will be on. User response: 1. Determine the 2145 UPS tht is reporting the error from the error event dt. Perform the following steps on just this uninterruptible power supply. 2. Check tht the 2145 UPS is still reporting the error. If the power overlod wrning LED is no longer on, go to step Ensure tht only 2145s re receiving power from the uninterruptible power supply. Ensure tht there re no switches or disk controllers tht re connected to the 2145 UPS. Chpter 7. Dignosing problems 207

226 Remove ech connected 2145 input power in turn, until the output overlod is removed. 5. Exchnge the FRUs for new FRUs in the sequence shown, on the overcurrent Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the 2145 UPS. 7. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v Power cble ssembly (50%) v Power supply ssembly (40%) v 2145 UPS electronics ssembly (10%) 1161 The output lod on the 2145 UPS-1U exceeds the specifictions (reported by 2145 UPS-1U lrm bits). Explntion: The output lod on the 2145 UPS-1U exceeds the specifictions (reported by 2145 UPS-1U lrm bits). User response: 1. Ensure tht only 2145s re receiving power from the uninterruptible power supply. Also, ensure tht no other devices re connected to the 2145 UPS-1U. 2. Exchnge, in the sequence shown, the FRUs for new FRUs. If the Overlod Indictor is still illuminted with ll outputs disconnected, replce the 2145 UPS-1U. 3. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the 2145 UPS-1U. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v Power cble ssembly (50%) v Power supply ssembly (40%) v 2145 UPS-1U ssembly (10%) 1165 The 2145 UPS output lod is unexpectedly high. The 2145 UPS output is possibly connected to n extr non-2145 lod. Explntion: The 2145 UPS output lod is unexpectedly high. The 2145 UPS output is possibly connected to n extr non-2145 lod. User response: 1. Ensure tht only 2145s re receiving power from the uninterruptible power supply. Ensure tht there re no switches or disk controllers tht re connected to the 2145 UPS. 2. Check node sttus. If ll nodes show sttus of online, the problem no longer exists. Mrk the error tht you hve just repired fixed nd go to the repir verifiction MAP. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: None Other: v Configurtion error 1166 The 2145 UPS-1U output lod is unexpectedly high. Explntion: The uninterruptible power supply output is possibly connected to n extr non-2145 lod. User response: 1. Ensure tht there re no other devices tht re connected to the 2145 UPS-1U. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the 2145 UPS-1U. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v 2145 UPS-1U ssembly (5%) Other: v Configurtion error (95%) UPS electronics fult (reported by the 2145 UPS lrm bits). Explntion: 2145 UPS electronics fult (reported by the 2145 UPS lrm bits). User response: 1. Replce the uninterruptible power supply electronics ssembly. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the UPS. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 208 SAN Volume Controller: Troubleshooting Guide

227 UPS electronics ssembly (100%) UPS-1U electronics fult (reported by the 2145 UPS-1U lrm bits). Explntion: 2145 UPS-1U electronics fult (reported by the 2145 UPS-1U lrm bits). User response: 1. Replce the uninterruptible power supply ssembly. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the 2145 UPS-1U. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145 UPS-1U ssembly (100%) 1175 A problem hs occurred with the uninterruptible power supply frme fult (reported by uninterruptible power supply lrm bits). Explntion: A problem hs occurred with the uninterruptible power supply frme fult (reported by the uninterruptible power supply lrm bits). User response: 1. Replce the uninterruptible power supply ssembly. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: Uninterruptible power supply ssembly (100%) 1179 Too mny drives ttched to the system. Explntion: The cluster only supports fixed number of drives. A drive hs been dded tht mkes the number of drives lrger thn the totl number of supported drives per cluster. User response: 1. Disconnect ny excessive unmnged enclosures from the system. 2. Un-mnge ny offline drives tht re not present in the system. 3. Identify unused drives nd remove them from the enclosures. 4. Identify rrys of drives tht re no longer required. 5. Remove the rrys nd remove the drives from the enclosures if they re present. 6. Once there re fewer thn 1056 drives in the system, consider re-engineering system cpcity by migrting dt from smll rrys onto lrge rrys, then removing the smll rrys nd the drives tht formed them. Consider the need for n dditionl Storwize system in your SAN solution UPS bttery fult (reported by 2145 UPS lrm bits). Explntion: 2145 UPS bttery fult (reported by 2145 UPS lrm bits). User response: 1. Replce the 2145 UPS bttery ssembly. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145 UPS bttery ssembly (100%) UPS-1U bttery fult (reported by 2145 UPS-1U lrm bits). Explntion: 2145 UPS-1U bttery fult (reported by 2145 UPS-1U lrm bits). User response: 1. Replce the 2145 UPS-1U bttery ssembly. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145 UPS-1U bttery ssembly (100%) 1182 Ambient temperture is too high during system strtup. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 528. See the detils of node error 528 for more informtion. User response: See node error 528. Chpter 7. Dignosing problems 209

228 The nodes hrdwre configurtion does not meet the minimum requirements. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 562. See the detils of node error 562 for more informtion. User response: See node error UPS fult, with no specific FRU identified (reported by uninterruptible power supply lrm bits). Explntion: 2145 UPS fult, with no specific FRU identified (reported by 2145 UPS lrm bits). User response: 1. In the sequence shown, exchnge the FRU for new FRU. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the 2145 UPS. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v 2145 UPS electronics ssembly (60%) v 2145 UPS bttery ssembly (20%) v 2145 UPS ssembly (20%) 1186 A problem hs occurred in the 2145 UPS-1U, with no specific FRU identified (reported by 2145 UPS-1U lrm bits). Explntion: A problem hs occurred in the 2145 UPS-1U, with no specific FRU identified (reported by 2145 UPS-1U lrm bits). User response: 1. In the sequence shown, exchnge the FRU for new FRU. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145 UPS-1U ssembly (100%) 1187 Node softwre is inconsistent or dmged Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node errors 523, 573, 574. See the detils of node errors 523, 573, 574 for more informtion. User response: See node errors 523, 573, Too mny softwre crshes hve occurred. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 564. See the detils of node error 564 for more informtion. User response: See node error The node is held in the service stte. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 690. See the detils of node error 690 for more informtion. User response: See node error The 2145 UPS bttery hs reched its end of life. Explntion: of life. The 2145 UPS bttery hs reched its end User response: 1. Replce the 2145 UPS bttery ssembly. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145 UPS bttery ssembly (100%) 1191 The 2145 UPS-1U bttery hs reched its end of life. Explntion: end of life. The 2145 UPS-1U bttery hs reched its User response: 1. Replce the 2145 UPS-1U bttery ssembly. 2. Check node sttus. If ll nodes show sttus of online, mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the uninterruptible power supply. 210 SAN Volume Controller: Troubleshooting Guide

229 Go to repir verifiction MAP. Possible Cuse-FRUs or other: 2145 UPS-1U bttery ssembly (100%) 1192 Unexpected node error Explntion: A node is missing from the cluster. The error tht it is reporting is not recognized by the system. User response: Find the node tht is in service stte nd use the service ssistnt to determine why it is not ctive The UPS bttery chrge is not enough to llow the node to strt. Explntion: The cluster is reporting tht node is not opertionl becuse of criticl node error 587. See the detils of node error 587 for more informtion. User response: 1194 Automtic recovery of offline node hs filed. Explntion: The cluster hs n offline node nd hs determined tht one of the cndidte nodes mtches the chrcteristics of the offline node. The cluster hs ttempted but filed to dd the node bck into the cluster. The cluster hs stopped ttempting to utomticlly dd the node bck into the cluster. If node hs incomplete stte dt, it remins offline fter it strts. This occurs if the node hs hd loss of power or hrdwre filure tht prevented it from completing the writing of ll of the stte dt to disk. The node reports node error 578 when it is in this stte. If three ttempts to utomticlly dd mtching cndidte node to cluster hve been mde, but the node hs not returned online for 24 hours, the cluster stops utomtic ttempts to dd the node nd logs error code 1194 Automtic recovery of offline node filed. Two possible scenrios when this error event is logged re: 1. The node hs filed without sving ll of its stte dt. The node hs restrted, possibly fter repir, nd shows node error 578 nd is cndidte node for joining the cluster. The cluster ttempts to dd the node into the cluster but does not succeed. After 15 minutes, the cluster mkes second ttempt to dd the node into the cluster nd gin does not succeed. After nother 15 minutes, the cluster mkes third ttempt to dd the node into the cluster nd gin does not succeed. After nother 15 minutes, the cluster logs error code The node never cme online during the ttempts to dd it to the cluster. 2. The node hs filed without sving ll of its stte dt. The node hs restrted, possibly fter repir, nd shows node error 578 nd is cndidte node for joining the cluster. The cluster ttempts to dd the node into the cluster nd succeeds nd the node becomes online. Within 24 hours the node fils gin without sving its stte dt. The node restrts nd shows node error 578 nd is cndidte node for joining the cluster. The cluster gin ttempts to dd the node into the cluster, succeeds, nd the node becomes online; however, the node gin fils within 24 hours. The cluster ttempts third time to dd the node into the cluster, succeeds, nd the node becomes online; however, the node gin fils within 24 hours. After nother 15 minutes, the cluster logs error code A combintion of these scenrios is lso possible. Note: If the node is mnully removed from the cluster, the count of utomtic recovery ttempts is reset to zero. User response: 1. If the node hs been continuously online in the cluster for more thn 24 hours, mrk the error s fixed nd go to the Repir Verifiction MAP. 2. Determine the history of events for this node by locting events for this node nme in the event log. Note tht the node ID will chnge, so mtch on the WWNN nd node nme. Also, check the service records. Specificlly, note entries indicting one of three events: 1) the node is missing from the cluster (cluster error 1195 event ), 2) n ttempt to utomticlly recover the offline node is strting (event ), 3) the node hs been dded to the cluster (event ). 3. If the node hs not been dded to the cluster since the recovery process strted, there is probbly hrdwre problem. The node's internl disk might be filing in mnner tht it is unble to modify its softwre level to mtch the softwre level of the cluster. If you hve not yet determined the root cuse of the problem, you cn ttempt to mnully remove the node from the cluster nd dd the node bck into the cluster. Continuously monitor the sttus of the nodes in the cluster while the cluster is ttempting to dd the node. Note: If the node type is not supported by the softwre version of the cluster, the node will not pper s cndidte node. Therefore, incomptible hrdwre is not potentil root cuse of this error. 4. If the node ws dded to the cluster but filed gin before it hs been online for 24 hours, investigte the root cuse of the filure. If no events Chpter 7. Dignosing problems 211

230 in the event log indicte the reson for the node filure, collect dumps nd contct IBM technicl support for ssistnce. 5. When you hve fixed the problem with the node, you must use either the cluster console or the commnd line interfce to mnully remove the node from the cluster nd dd the node into the cluster. 6. Mrk the error s fixed nd go to the verifiction MAP. Possible Cuse-FRUs or other: None, lthough investigtion might indicte hrdwre filure Node missing. Explntion: You cn resolve this problem by repiring the filure on the missing User response: 1. If it is not obvious which node in the cluster hs filed, check the sttus of the nodes nd find the 3700 with sttus of offline. 2. Go to the Strt MAP nd perform the repir on the filing node. 3. When the repir hs been completed, this error is utomticlly mrked s fixed. 4. Check node sttus. If ll nodes show sttus of online, but the error in the log hs not been mrked s fixed, mnully mrk the error tht you hve just repired fixed. If ny nodes do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None 1200 The configurtion is not vlid. Too mny devices, MDisks, or trgets hve been presented to the system. Explntion: The configurtion is not vlid. Too mny devices, MDisks, or trgets hve been presented to the system. User response: 1. Remove unwnted devices from the Fibre Chnnel network fbric. 2. Strt cluster discovery opertion to find devices/disks by rescnning the Fibre Chnnel network. 3. List ll connected mnged disks. Check with the customer tht the configurtion is s expected. Mrk the error tht you hve just repired fixed. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: Fibre Chnnel network fbric fult (100%) 1201 A flsh drive requires recovery. Explntion: The flsh drive tht is identified by this error needs to be recovered. User response: To recover this flsh drive, submit the following commnd: chdrive -tsk recover drive_id where drive_id is the identity of the drive tht needs to be recovered A flsh drive is missing from the configurtion. Explntion: The offline flsh drive identified by this error must be repired. User response: In the mngement GUI, click Troubleshooting > Recommended Actions to run the recommended ction for this error. Otherwise, use MAP 6000 to replce the drive A duplicte Fibre Chnnel frme hs been received. Explntion: A duplicte Fibre Chnnel frme should never be detected. Receiving duplicte Fibre Chnnel frme indictes tht there is problem with the Fibre Chnnel fbric. Other errors relted to the Fibre Chnnel fbric might be generted. User response: 1. Use the trnsmitting nd receiving WWPNs indicted in the error dt to determine the section of the Fibre Chnnel fbric tht hs generted the duplicte frme. Serch for the cuse of the problem by using fbric monitoring tools. The duplicte frme might be cused by design error in the topology of the fbric, by configurtion error, or by softwre or hrdwre fult in one of the components of the Fibre Chnnel fbric, including inter-switch links. 2. When you re stisfied tht the problem hs been corrected, mrk the error tht you hve just repired fixed. 3. Go to MAP 5700: Repir verifiction. Possible Cuse-FRUs or other: v Fibre Chnnel cble ssembly (1%) v Fibre Chnnel dpter (1%) Other: 212 SAN Volume Controller: Troubleshooting Guide

231 v Fibre Chnnel network fbric fult (98%) 1210 A locl Fibre Chnnel port hs been excluded. Explntion: excluded. A locl Fibre Chnnel port hs been User response: 1. Repir fults in the order shown. 2. Check the sttus of the disk controllers. If ll disk controllers show good sttus, mrk the error tht you just repired s fixed. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v Fibre Chnnel cble ssembly (75%) v Smll Form-fctor Pluggble (SFP) connector (10%) v Fibre Chnnel dpter (5%) Other: v Fibre Chnnel network fbric fult (10%) 1212 Power supply exceeded temperture threshold. Explntion: threshold. Power supply exceeded temperture User response: Complete the following steps: 1. Check irflow. Remove the top of the mchine cse nd check for missing bffles or internl blockges. 2. If problem persists, replce the power supply. Possible Cuse-FRUs or other: v Power supply 1213 Boot drive missing, out of sync, or filed. Explntion: Boot drive missing, out of sync, or filed. User response: Complete the following steps: 1. Look t boot drive view to determine the missing, filed or out of sync drive. 2. Insert missing drive. 3. Replce filed drive. 4. Synchronize n out of sync drive by running the commnds svctsk chnodebootdrive -sync nd/or stsk chbootdrive -sync. Possible Cuse-FRUs or other: v System drive 1214 Boot drive is in the wrong slot. Explntion: Boot drive is in the wrong slot. User response: Complete the following steps: 1. Look t boot drive view to determine which drive is in the wrong slot, which node nd slot it should be in, nd which drive should be in this slot. 2. Swp the drive for the correct one but shut down the node first if booted yes is shown for tht drive in boot drive view. 3. If you wnt to use the drive in this node, synchronize the boot drives by running the commnds svctsk chnodebootdrive -sync nd/or stsk chbootdrive -sync. 4. The node error clers, or new node error is displyed for you to work on. Possible Cuse-FRUs or other: v None 1215 A flsh drive is filing. Explntion: The flsh drive hs detected fults tht indicte tht the drive is likely to fil soon. The drive should be replced. The cluster event log will identify drive ID for the flsh drive tht cused the error. User response: In the mngement GUI, click Troubleshooting > Recommended Actions to run the recommended ction for this error. If this does not resolve the issue, contct your next level of support SAS errors hve exceeded thresholds. Explntion: The cluster hs experienced lrge number of SAS communiction errors, which indictes fulty SAS component tht must be replced. User response: In the sequence shown, exchnge the FRUs for new FRUs. Go to the repir verifiction MAP. Possible Cuse-FRUs or other: 1. SAS Cble (70%) 2. High speed SAS dpter (20%) 3. SAS drive bckplne (5%) 4. flsh drive (5%) 1217 A flsh drive hs exceeded the temperture wrning threshold. Explntion: The flsh drive identified by this error hs reported tht its temperture is higher thn the wrning threshold. User response: Tke steps to reduce the temperture of the drive. 1. Determine the temperture of the room, nd reduce the room temperture if this ction is pproprite. Chpter 7. Dignosing problems 213

232 Replce ny filed fns. 3. Ensure tht there re no obstructions to ir flow for the node. 4. Mrk the error s fixed. If the error recurs, contct hrdwre support for further investigtion. Possible Cuse-FRUs or other: v Flsh drive (10%) Other: v System environment or irflow blockge (90%) 1220 A remote Fibre Chnnel port hs been excluded. Explntion: excluded. A remote Fibre Chnnel port hs been User response: 1. View the event log. Note the MDisk ID ssocited with the error code. 2. From the MDisk, determine the filing disk controller ID. 3. Refer to the service documenttion for the disk controller nd the Fibre Chnnel network to resolve the reported problem. 4. After the disk drive is repired, strt cluster discovery opertion to recover the excluded Fibre Chnnel port by rescnning the Fibre Chnnel network. 5. To restore MDisk online sttus, include the mnged disk tht you noted in step Check the sttus of the disk controller. If ll disk controllers show good sttus, mrk the error tht you hve just repired, fixed. 7. If ll disk controllers do not show good sttus, contct your support center to resolve the problem with the disk controller. 8. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Enclosure/controller fult (50%) v Fibre Chnnel network fbric (50%) event log for other 1230 errors. Ensure tht ll higher priority errors re fixed. This error event is usully cused by fbric problem. If possible, use the fbric switch or other fbric dignostic tools to determine which link or port is reporting the errors. If there re error events for links from this node to number of different controllers or clusters, then it is probbly the node to switch link tht is cusing the errors. Unless there re other contrry indictions, first replce the cble between the switch nd the remote system. 1. From the fbric nlysis, determine the FRU tht is most likely cusing the error. If this FRU hs recently been replced while resolving 1230 error, choose the next most likely FRU tht hs not been replced recently. Exchnge the FRU for new FRU. 2. Mrk the error s fixed. If the FRU replcement hs not fixed the problem, the error will be logged gin; however, depending on the severity of the problem, the error might not be logged gin immeditely. 3. Strt cluster discovery opertion to recover the login by re-scnning the Fibre Chnnel network. 4. Check the sttus of the disk controller or remote cluster. If the sttus is not good, go to the Strt MAP. 5. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v Fibre Chnnel cble, switch to remote port, (30%) v Switch or remote device SFP connector or dpter, (30%) v Fibre Chnnel cble, locl port to switch, (30%) v Cluster SFP connector, (9%) v Cluster Fibre Chnnel dpter, (1%) Note: The first two FRUs re not cluster FRUs A node hs encountered n error upgrding. Explntion: upgrde. One or more nodes hs filed the User response: Check lssoftwreupgrdesttus for the node tht filed nd continue troubleshooting with the error code it provides A login hs been excluded. Explntion: A port to port fbric connection, or login, between the cluster node nd either controller or nother cluster hs hd excessive errors. The login hs therefore been excluded, nd will not be used for I/O opertions. User response: Determine the remote system, which might be either controller or cluster. Check the 1310 A mnged disk is reporting excessive errors. Explntion: errors. A mnged disk is reporting excessive User response: 1. Repir the enclosure/controller fult. 2. Check the mnged disk sttus. If ll mnged disks show sttus of online, mrk the error tht 214 SAN Volume Controller: Troubleshooting Guide

233 you hve just repired s fixed. If ny mnged disks show sttus of excluded, include the excluded mnged disks nd then mrk the error s fixed. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: Enclosure/controller fult (100%) 1311 A flsh drive is offline due to excessive errors. Explntion: The drive tht is reporting excessive errors hs been tken offline. User response: In the mngement GUI, click Troubleshooting > Recommended Actions to run the recommended ction for this error. If this does not resolve the issue, contct your next level of support A disk I/O medium error hs occurred. Explntion: A disk I/O medium error hs occurred. User response: 1. Check whether the volume the error is reported ginst is mirrored. If it is, check if there is 1870 Mirrored volume offline becuse hrdwre red error hs occurred error relting to this volume in the event log. Also check if one of the mirror copies is synchronizing. If ll these tests re true then you must delete the volume copy tht is not synchronized from the volume. Check tht the volume is online before continuing with the following ctions. Wit until the medium error is corrected before trying to re-crete the volume mirror. 2. If the medium error ws detected by red from host, sk the customer to rewrite the incorrect dt to the block logicl block ddress (LBA) tht is reported in the host systems SCSI sense dt. If n individul block cnnot be recovered it will be necessry to restore the volume from bckup. (If this error hs occurred during migrtion, the host system does not notice the error until the trget device is ccessed.) 3. If the medium error ws detected during mirrored volume synchroniztion, the block might not be being used for host dt. The medium error must still be corrected before the mirror cn be estblished. It my be possible to fix the block tht is in error using the disk controller or host tools. Otherwise, it will be necessry to use the host tools to copy the volume content tht is being used to new volume. Depending on the circumstnces, this new volume cn be kept nd mirrored, or the originl volume cn be repired nd the dt copied bck gin. 4. Check mnged disk sttus. If ll mnged disks show sttus of online, mrk the error tht you hve just repired s fixed. If ny mnged disks do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the disk controller. 5. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: Enclosure/controller fult (100%) 1330 A suitble mnged disk (MDisk) or drive for use s quorum disk ws not found. Explntion: A quorum disk is needed to enble tie-brek when some cluster members re missing. Three quorum disks re usully defined. By defult, the cluster utomticlly lloctes quorum disks when mnged disks re creted; however, the option exists to mnully ssign quorum disks. This error is reported when there re mnged disks or imge mode disks but no quorum disks. To become quorum disk: v The MDisk must be ccessible by ll nodes in the cluster. v The MDisk must be mnged; tht is, it must be member of storge pool. v The MDisk must hve free extents. v The MDisk must be ssocited with controller tht is enbled for quorum support. If the controller hs multiple WWNNs, ll of the controller components must be enbled for quorum support. A quorum disk might not be vilble becuse of Fibre Chnnel network filure or becuse of Fibre Chnnel switch zoning problem. User response: 1. Resolve ny known Fibre Chnnel network problems. 2. Ask the customer to confirm tht MDisks hve been dded to storge pools nd tht those MDisks hve free extents nd re on controller tht is enbled for use s provider of quorum disks. Ensure tht ny controller with multiple WWNNs hs ll of its components enbled to provide quorum disks. Either crete suitble MDisk or if possible enble quorum support on controllers with which existing MDisks re ssocited. If t lest one mnged disk Chpter 7. Dignosing problems 215

234 shows mode of mnged nd hs non-zero quorum index, mrk the error tht you hve just repired s fixed. 3. If the customer is unble to mke the pproprite chnges, sk your softwre support center for ssistnce. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: Configurtion error (100%) 1335 Quorum disk not vilble. Explntion: Quorum disk not vilble. User response: 1. View the event log entry to identify the mnged disk (MDisk) being used s quorum disk, tht is no longer vilble. 2. Perform the disk controller problem determintion nd repir procedures for the MDisk identified in step Include the MDisks into the cluster. 4. Check the mnged disk sttus. If the mnged disk identified in step 1 shows sttus of online, mrk the error tht you hve just repired s fixed. If the mnged disk does not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the disk controller. 5. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: Enclosure/controller fult (100%) 1340 A mnged disk hs timed out. Explntion: This error ws reported becuse lrge number of disk timeout conditions hve been detected. The problem is probbly cused by filure of some other component on the SAN. User response: 1. Repir problems on ll enclosures/controllers nd switches on the sme SAN s this 2145 cluster. 2. If problems re found, mrk this error s fixed. 3. If no switch or disk controller filures cn be found, tke n event log dump nd cll your hrdwre support center. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Enclosure/controller fult v Fibre Chnnel switch 1350 IB ports re not opertionl. Explntion: IB ports re not opertionl. User response: An offline port cn hve mny cuses nd so it is necessry to check them ll. Strt with the esiest nd lest intrusive possibility. 1. Reset the IB port vi CLI commnd. 2. If the IB port is connected to switch, double-check the switch configurtion for issues. 3. Reset the IB cble on both the IB side nd the HBA/switch side. 4. Run temporry second IB cble to replce the current one to check for cble fult. 5. Reset the IB interfce crd; reset the cnister; reboot the system. Note: If the system is in production, scheduling mintennce downtime before performing ny of the ctions in Step 5 is highly recommended s other ports will be ffected. Possible Cuse-FRUs or other: Externl (cble, HCA, switch, nd so on) (85%) Interfce (10%) Cnister (5%) 1360 A SAN trnsport error occurred. Explntion: This error hs been reported becuse the 2145 performed error recovery procedures in response to SAN component ssocited trnsport errors. The problem is probbly cused by filure of component of the SAN. User response: 1. View the event log entry to determine the node tht logged the problem. Determine the 2145 node or controller tht the problem ws logged ginst. 2. Perform Fibre Chnnel switch problem determintion nd repir procedures for the switches connected to the 2145 node or controller. 3. Perform Fibre Chnnel cbling problem determintion nd repir procedures for the cbles connected to the 2145 node or controller. 4. If ny problems re found nd resolved in step 2 nd 3, mrk this error s fixed. 216 SAN Volume Controller: Troubleshooting Guide

235 If no switch or cble filures were found in steps 2 nd 3, tke n event log dump. Cll your hrdwre support center. 6. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Fibre Chnnel switch v Fibre Chnnel cbling 1370 A mnged disk error recovery procedure (ERP) hs occurred. Explntion: This error ws reported becuse lrge number of disk error recovery procedures hve been performed by the disk controller. The problem is probbly cused by filure of some other component on the SAN. User response: 1. View the event log entry nd determine the mnged disk tht ws being ccessed when the problem ws detected. 2. Perform the disk controller problem determintion nd repir procedures for the MDisk determined in step Perform problem determintion nd repir procedures for the fibre chnnel switches connected to the 2145 nd ny other Fibre Chnnel network components. 4. If ny problems re found nd resolved in steps 2 nd 3, mrk this error s fixed. 5. If no switch or disk controller filures were found in steps 2 nd 3, tke n event log dump. Cll your hrdwre support center. 6. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Enclosure/controller fult v Fibre Chnnel switch 1400 The 2145 cnnot detect n Ethernet connection. Explntion: connection. The 2145 cnnot detect n Ethernet User response: 1. Go to the Ethernet MAP. 2. Go to the repir verifiction MAP. Possible Cuse-FRUs or other: G4, A4, 2145-CF8, or 2145-CG8 v Ethernet cble (25%) v System bord (25%) Other: v Ethernet cble is disconnected or dmged (25%) v Ethernet hub fult (25%) 1450 Fewer Fibre Chnnel I/O ports opertionl. Explntion: One or more Fibre Chnnel I/O ports tht hve previously been ctive re now inctive. This sitution hs continued for one minute. A Fibre Chnnel I/O port might be estblished on either Fibre Chnnel pltform port or n Ethernet pltform port using FCoE. This error is expected if the ssocited Fibre Chnnel or Ethernet port is not opertionl. Dt: Three numeric vlues re listed: v The ID of the first unexpected inctive port. This ID is deciml number. v The ports tht re expected to be ctive, which is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is expected to be ctive. v The ports tht re ctully ctive, which is hexdeciml number. Ech bit position represents port, with the lest significnt bit representing port 1. The bit is 1 if the port is ctive. User response: 1. If possible, use the mngement GUI to run the recommended ctions for the ssocited service error code. 2. Follow the procedure for mpping I/O ports to pltform ports to determine which pltform port is providing this I/O port. 3. Check for ny 704 (Fibre chnnel pltform port not opertionl) or 724 (Ethernet pltform port not opertionl) node errors reported for the pltform port. 4. Possibilities: v If the port hs been intentionlly disconnected, use the mngement GUI recommended ction for the service error code nd cknowledge the intended chnge. v Resolve the 704 or 724 error. v If this is n FCoE connection, use the informtion the view gives bout the Fibre Chnnel forwrder (FCF) to troubleshoot the connection between the port nd the FCF. Possible Cuse-FRUs or other cuse: v None Chpter 7. Dignosing problems 217

236 Interfce crd is unsupported. Explntion: Interfce crd is unsupported. User response: Replce the wrong interfce crd with the correct type. Possible Cuse-FRUs or other: Interfce crd (100%) 1473 The instlled bttery is t hrdwre revision level tht is not supported by the current code level. Explntion: To updte the code level to one tht supports the currently instlled bttery, perform service mode code upgrde. To render your bttery opertive, you must hve synchroniztion of the code level within both bttery nd system. User response: To replce the bttery with one tht is supported by the current code level, follow the service ction for 1130 on pge 204. Alwys instll the ltest level of the system softwre to void problems with upgrdes nd component comptibility Bttery is nering end of life. Explntion: When bttery ners the end of its life, you must replce it if you intend to preserve the cpcity to filover power to btteries. User response: Replce the bttery by following this procedure s soon s you cn. If the node is in clustered system, ensure tht the bttery is not being relied upon to provide dt protection before you remove it. Issue the chnodebttery -remove -bttery bttery_id node_id commnd to estblish the lck of relince on the bttery. If the commnd returns with The commnd hs filed becuse the specified bttery is offline (BATTERY_OFFLINE) error, replce the bttery immeditely. If the commnd returns with The commnd hs filed becuse the specified bttery is not redundnt (BATTERY_NOT_REDUNDANT) error, do not remove the relied-on bttery. Removing the bttery compromises dt protection. In this cse, without other bttery-relted errors, use the chnodebttery -remove -bttery bttery_id node_id commnd periodiclly to force the system to remove relince on the bttery. The system often removes relince within one hour (TBC). Alterntively, remove the node from the clustered system. Once the node is independent, you cn replce its bttery immeditely. If the node is not prt of cluster, or the bttery is offline, or the chnodebttery commnd returns without error, conduct the service ction for 1130 on pge Bttery is too hot. Explntion: You must wit for the bttery to cool down before it cn resume its norml opertion. User response: The bttery might be slow to cool if the mbient temperture is high. If node error 768 is reported, service tht s well Bttery is too cold. Explntion: You must wit for the bttery to wrm before it cn resume its norml opertion. User response: The bttery might be slow to wrm if the mbient temperture is low. If node error 768 is reported, service tht s well. Otherwise, wit for the bttery to wrm A cluster pth hs filed. Explntion: One of the V3700 Fibre Chnnel ports is unble to communicte with ll the other V3700s in the cluster. User response: 1. Check for incorrect switch zoning. 2. Repir the fult in the Fibre Chnnel network fbric. 3. Check the sttus of the node ports tht re not excluded vi the system's locl port msk. If the sttus of the node ports shows s ctive, mrk the error tht you hve repired s fixed. If ny node ports do not show sttus of ctive, go to strt MAP. If you return to this step contct your support center to resolve the problem with the V Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: Fibre Chnnel network fbric fult (100%) 1570 Quorum disk configured on controller tht hs quorum disbled Explntion: This error cn occur with storge controller tht cn be ccessed through multiple WWNNs nd hve defult setting of not llowing quorum disks. When these controllers re detected by cluster, lthough multiple component controller definitions re creted, the cluster recognizes tht ll of the component controllers belong to the sme storge system. To enble the cretion of quorum disk on this storge system, ll of the controller components must 218 SAN Volume Controller: Troubleshooting Guide

237 be configured to llow quorum. A configurtion chnge to the SAN, or to storge system with multiple WWNNs, might result in the cluster discovering new component controllers for the storge system. These components will tke the defult setting for llowing quorum. This error is reported if there is quorum disk ssocited with the controller nd the defult setting is not to llow quorum. User response: v Determine if there should be quorum disk on this storge system. Ensure tht the controller supports quorum before you llow quorum disks on ny disk controller. You cn check the support website for more informtion. v If quorum disk is required on this storge system, llow quorum on the controller component tht is reported in the error. If the quorum disk should not be on this storge system, move it elsewhere. v Mrk the error s fixed. Possible Cuse-FRUs or other: v None Other: Fibre Chnnel network fbric fult (100%) 1600 Mirrored disk repir hlted becuse of difference. Explntion: During the repir of mirrored volume two copy disks were found to contin different dt for the sme logicl block ddress (LBA). The vlidte option ws used, so the repir process hs hlted. Red opertions to the LBAs tht differ might return the dt of either volume copy. Therefore it is importnt not to use the volume unless you re sure tht the host pplictions will not red the LBAs tht differ or cn mnge the different dt tht potentilly cn be returned. User response: Perform one of the following ctions: v Continue the repir strting with the next LBA fter the difference to see how mny differences there re for the whole mirrored volume. This cn help you decide which of the following ctions to tke. v Choose primry disk nd run repir resynchronizing differences. v Run repir nd crete medium errors for differences. v Restore ll or prt of the volume from bckup. v Decide which disk hs correct dt, then delete the copy tht is different nd re-crete it llowing it to be synchronized. Then mrk the error s fixed. Possible Cuse-FRUs or other: v None 1610 There re too mny copied medi errors on mnged disk. Explntion: The cluster mintins virtul medium error tble for ech MDisk. This tble is list of logicl block ddresses on the mnged disk tht contin dt tht is not vlid nd cnnot be red. The virtul medium error tble hs fixed length. This error event indictes tht the system hs ttempted to dd n entry to the tble, but the ttempt hs filed becuse the tble is lredy full. There re two circumstnces tht will cuse n entry to be dded to the virtul medium error tble: 1. FlshCopy, dt migrtion nd mirrored volume synchroniztion opertions copy dt from one mnged disk extent to nother. If the source extent contins either virtul medium error or the RAID controller reports rel medium error, the system cretes mtching virtul medium error on the trget extent. 2. The mirrored volume vlidte nd repir process hs the option to crete virtul medium errors on sectors tht do not mtch on ll volume copies. Normlly zero, or very few, differences re expected; however, if the copies hve been mrked s synchronized inppropritely, then lrge number of virtul medium errors could be creted. User response: Ensure tht ll higher priority errors re fixed before you ttempt to resolve this error. Determine whether the excessive number of virtul medium errors occurred becuse of mirrored disk vlidte nd repir opertion tht creted errors for differences, or whether the errors were creted becuse of copy opertion. Follow the corresponding option shown below. 1. If the virtul medium errors occurred becuse of mirrored disk vlidte nd repir opertion tht creted medium errors for differences, then lso ensure tht the volume copies hd been fully synchronized prior to strting the opertion. If the copies hd been synchronized, there should be only few virtul medium errors creted by the vlidte nd repir opertion. In this cse, it might be possible to rewrite only the dt tht ws not consistent on the copies using the locl dt recovery process. If the copies hd not been synchronized, it is likely tht there re now lrge number of medium errors on ll of the volume copies. Even if the virtul medium errors re expected to be only for blocks tht hve never been written, it is importnt to cler the virtul medium errors to void inhibition of other opertions. To recover the dt for ll of these virtul medium errors it is likely tht the volume will hve to be Chpter 7. Dignosing problems 219

238 recovered from bckup using process tht rewrites ll sectors of the volume. 2. If the virtul medium errors hve been creted by copy opertion, it is best prctice to correct ny medium errors on the source volume nd to not propgte the medium errors to copies of the volume. Fixing higher priority errors in the event log would hve corrected the medium error on the source volume. Once the medium errors hve been fixed, you must run the copy opertion gin to cler the virtul medium errors from the trget volume. It might be necessry to repet sequence of copy opertions if copies hve been mde of lredy copied medium errors. An lterntive tht does not ddress the root cuse is to delete volumes on the trget mnged disk tht hve the virtul medium errors. This volume deletion reduces the number of virtul medium error entries in the MDisk tble. Migrting the volume to different mnged disk will lso delete entries in the MDisk tble, but will crete more entries on the MDisk tble of the MDisk to which the volume is migrted. Possible Cuse-FRUs or other: v None 1620 A storge pool is offline. Explntion: A storge pool is offline. User response: 1. Repir the fults in the order shown. 2. Strt cluster discovery opertion by rescnning the Fibre Chnnel network. 3. Check mnged disk (MDisk) sttus. If ll MDisks show sttus of online, mrk the error tht you hve just repired s fixed. If ny MDisks do not show sttus of online, go to strt MAP. If you return to this step, contct your support center to resolve the problem with the disk controller. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Fibre Chnnel network fbric fult (50%) v Enclosure/controller fult (50%) 1623 One or more MDisks on controller re degrded. Explntion: At lest one MDisk on controller is degrded becuse the MDisk is not vilble through one or more nodes. The MDisk is vilble through t lest one node. Access to dt might be lost if nother filure occurs. In correctly configured system, ech node ccesses ll of the MDisks on controller through ll of the controller's ports. This error is only logged once per controller. There might be more thn one MDisk on this controller tht hs been configured incorrectly, but the error is only logged for one MDisk. To prevent this error from being logged becuse of short-term fbric mintennce ctivities, this error condition must hve existed for one hour before the error is logged. User response: 1. Determine which MDisks re degrded. Look for MDisks with pth count lower thn the number of nodes. Do not use only the MDisk sttus, since other errors cn lso cuse degrded MDisks. 2. Ensure tht the controller is zoned correctly with ll of the nodes. 3. Ensure tht the logicl unit is mpped to ll of the nodes. 4. Ensure tht the logicl unit is mpped to ll of the nodes using the sme LUN. 5. Run the console or CLI commnd to discover MDisks nd ensure tht the commnd completes. 6. Mrk the error tht you hve just repired s fixed. When you mrk the error s fixed, the controller's MDisk vilbility is tested nd the error will be logged gin immeditely if the error persists for ny MDisks. It is possible tht the new error will report different MDisk. 7. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Fibre Chnnel network fbric fult (50%) v Enclosure/controller fult (50%) 1624 Controller configurtion hs unsupported RDAC mode. Explntion: The cluster hs detected tht n IBM DS series disk controller's configurtion is not supported by the cluster. The disk controller is operting in RDAC mode. The disk controller might pper to be operting with the cluster; however, the configurtion is unsupported becuse it is known to not work with the cluster. User response: 1. Using the IBM DS series console, ensure tht the host type is set to 'IBM TS SAN VCE' nd tht the AVT option is enbled. (The AVT nd RDAC options re mutully exclusive). 220 SAN Volume Controller: Troubleshooting Guide

239 Mrk the error tht you hve just repired s fixed. If the problem hs not been fixed it will be logged gin; this could tke few minutes. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Enclosure/controller fult 1625 Incorrect disk controller configurtion. Explntion: While running n MDisk discovery, the cluster hs detected tht disk controller's configurtion is not supported by the cluster. The disk controller might pper to be operting with the cluster; however, the configurtion detected cn potentilly cuse issues nd should not be used. The unsupported configurtion is shown in the event dt. User response: 1. Use the event dt to determine chnges required on the disk controller nd reconfigure the disk controller to use supported configurtion. 2. Mrk the error tht you hve just repired s fixed. If the problem hs not been fixed it will be logged gin by the mnged disk discovery tht utomticlly runs t this time; this could tke few minutes. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Enclosure/controller fult 1627 The cluster hs insufficient redundncy in its controller connectivity. Explntion: The cluster hs detected tht it does not hve sufficient redundncy in its connections to the disk controllers. This mens tht nother filure in the SAN could result in loss of ccess to the ppliction dt. The cluster SAN environment should hve redundnt connections to every disk controller. This redundncy llows for continued opertion when there is filure in one of the SAN components. To provide recommended redundncy, cluster should be configured so tht: v v ech node cn ccess ech disk controller through two or more different inititor ports on the node. ech node cn ccess ech disk controller through two or more different controller trget ports. Note: Some disk controllers only provide single trget port. v ech node cn ccess ech disk controller trget port through t lest one inititor port on the node. If there re no higher-priority errors being reported, this error usully indictes problem with the SAN design, problem with the SAN zoning or problem with the disk controller. If there re unfixed higher-priority errors tht relte to the SAN or to disk controllers, those errors should be fixed before resolving this error becuse they might indicte the reson for the lck of redundncy. Error codes tht must be fixed first re: v 1210 Locl FC port excluded v 1230 Login hs been excluded Note: This error cn be reported if the required ction, to rescn the Fibre Chnnel network for new MDisks, hs not been performed fter deliberte reconfigurtion of disk controller or fter SAN rezoning. The 1627 error code is reported for number of different error IDs. The error ID indictes the re where there is lck of redundncy. The dt reported in n event log entry indictes where the condition ws found. The mening of the error IDs is shown below. For ech error ID the most likely reson for the condition is given. If the problem is not found in the suggested res, check the configurtion nd stte of ll of the SAN components (switches, controllers, disks, cbles nd cluster) to determine where there is single point of filure A disk controller is only ccessible from single node port. v v v A node hs detected tht it only hs connection to the disk controller through exctly one inititor port, nd more thn one inititor port is opertionl. The error dt indictes the device WWNN nd the WWPN of the connected port. A zoning issue or Fibre Chnnel connection hrdwre fult might cuse this condition A disk controller is only ccessible from single port on the controller. v v v A node hs detected tht it is only connected to exctly one trget port on disk controller, nd more thn one trget port connection is expected. The error dt indictes the WWPN of the disk controller port tht is connected. A zoning issue or Fibre Chnnel connection hrdwre fult might cuse this condition Only single port on disk controller is ccessible from every node in the cluster. Chpter 7. Dignosing problems 221

240 1630 v Only single port on disk controller is ccessible to every node when there re multiple ports on the controller tht could be connected. v v The error dt indictes the WWPN of the disk controller port tht is connected. A zoning issue or Fibre Chnnel connection hrdwre fult might cuse this condition A disk controller is ccessible through only hlf, or less, of the previously configured controller ports. v v v Although there might still be multiple ports tht re ccessible on the disk controller, hrdwre component of the controller might hve filed or one of the SAN fbrics hs filed such tht the opertionl system configurtion hs been reduced to single point of filure. The error dt indictes port on the disk controller tht is still connected, nd lso lists controller ports tht re expected but tht re not connected. A disk controller issue, switch hrdwre issue, zoning issue or cble fult might cuse this condition A disk controller is not ccessible from node. v A node hs detected tht it hs no ccess to disk controller. The controller is still ccessible from the prtner node in the I/O group, so its dt is still ccessible to the host pplictions. v The error dt indictes the WWPN of the missing disk controller. v A zoning issue or cbling error might cuse this condition. User response: 1. Check the error ID nd dt for more detiled description of the error. 2. Determine if there hs been n intentionl chnge to the SAN zoning or to disk controller configurtion tht reduces the cluster's ccess to the indicted disk controller. If either ction hs occurred, continue with step Use the GUI or the CLI commnd lsfbric to ensure tht ll disk controller WWPNs re reported s expected. 4. Ensure tht ll disk controller WWPNs re zoned ppropritely for use by the cluster. 5. Check for ny unfixed errors on the disk controllers. 6. Ensure tht ll of the Fibre Chnnel cbles re connected to the correct ports t ech end. 7. Check for filures in the Fibre Chnnel cbles nd connectors. 8. When you hve resolved the issues, use the GUI or the CLI commnd detectmdisk to rescn the Fibre Chnnel network for chnges to the MDisks. Note: Do not ttempt to detect MDisks unless you re sure tht ll problems hve been fixed. Detecting MDisks premturely might msk n issue. 9. Mrk the error tht you hve just repired s fixed. The cluster will revlidte the redundncy nd will report nother error if there is still not sufficient redundncy. 10. Go to MAP 5700: Repir verifiction. Possible Cuse-FRUs or other: v None 1630 The number of device logins ws reduced. Explntion: The number of port to port fbric connections, or logins, between the node nd storge controller hs decresed. This might be cused by problem on the SAN or by deliberte reconfigurtion of the SAN. User response: 1. Check the error in the cluster event log to identify the object ID ssocited with the error. 2. Check the vilbility of the filing device using the following commnd line: lscontroller object_id. If the commnd fils with the messge CMMVC6014E The commnd filed becuse the requested object is either unvilble or does not exist, sk the customer if this device ws removed from the system. v If yes, mrk the error s fixed in the cluster event log nd continue with the repir verifiction MAP. v If no or if the commnd lists detils of the filing controller, continue with the next step. 3. Check whether the device hs regined connectivity. If it hs not, check the cble connection to the remote-device port. 4. If ll ttempts to log in to remote-device port hve filed nd you cnnot solve the problem by chnging cbles, check the condition of the remote-device port nd the condition of the remote device. 5. Strt cluster discovery opertion by rescnning the Fibre Chnnel network. 6. Check the sttus of the disk controller. If ll disk controllers show good sttus, mrk the error tht you hve just repired s fixed. If ny disk controllers do not show good sttus, go to strt MAP. If you return to this step, contct the support center to resolve the problem with the disk controller. 7. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None 222 SAN Volume Controller: Troubleshooting Guide

241 Other: v Fibre Chnnel network fbric fult (50%) v Enclosure/controller fult (50%) 1660 The initiliztion of the mnged disk hs filed. Explntion: hs filed. The initiliztion of the mnged disk User response: 1. View the event log entry to identify the mnged disk (MDisk) tht ws being ccessed when the problem ws detected. 2. Perform the disk controller problem determintion nd repir procedures for the MDisk identified in step Include the MDisk into the cluster. 4. Check the mnged disk sttus. If ll mnged disks show sttus of online, mrk the error tht you hve just repired s fixed. If ny mnged disks do not show sttus of online, go to the strt MAP. If you return to this step, contct your support center to resolve the problem with the disk controller. 5. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: Enclosure/controller fult (100%) 1670 The CMOS bttery on the V3700 system bord filed. Explntion: bord filed. User response: vilble. The CMOS bttery on the V3700 system Possible Cuse-FRUs or other: CMOS bttery (100%) 1680 Drive fult type 1 Explntion: Drive fult type 1 User response: Replce the node until the FRU is Replce the drive. Possible Cuse-FRUs or other: Drive (95%) Cnister (3%) Midplne (2%) 1684 Drive is missing. Explntion: Drive is missing. User response: Instll the missing drive. The drive is typiclly dt drive tht ws previously prt of the rry. Possible Cuse-FRUs or other: Drive (100%) 1686 There re too mny drives ttched to the system. Explntion: of drives. The cluster only supports fixed number User response: 1. Disconnect ny excessive unmnged enclosures from the system. 2. Unmnge ny offline drives tht re not present in the system. 3. Identify unused drives nd remove them from the enclosures. 4. Identify rrys of drives tht re no longer required. Remove the rrys nd remove the drives from the enclosures if they re present. 5. Once there re fewer thn 1056 drives in the system, consider re-engineering system cpcity by migrting dt from smll rrys onto lrge rrys, then removing the smll rrys nd the drives tht formed them. 6. Consider the need for n dditionl Storwize system in your SAN solution Arry MDisk hs lost redundncy. Explntion: Arry MDisk hs lost redundncy. The RAID 5 system is missing dt drive. User response: Possible Cuse-FRUs or other: Drives removed or filed (100%) Replce the missing or filed drive No spre protection exists for one or more rry MDisks. Explntion: The system spre pool cnnot immeditely provide spre of ny suitbility to one or more rrys. User response: 1. Configure n rry but no spres. 2. Configure mny rrys nd single spre. Cuse tht spre to be consumed or chnge its use. Chpter 7. Dignosing problems 223

242 A bckground scrub process hs found n inconsistency between dt nd prity on the rry. Explntion: The rry hs t lest one stride where the dt nd prity do not mtch. RAID hs found n inconsistency between the dt stored on the drives nd the prity informtion. This could either men tht the dt hs been corrupted, or tht the prity informtion hs been corrupted. User response: Follow the directed mintennce procedure for inconsistent rrys Arry MDisk hs tken spre member tht does not mtch rry gols. Explntion: 1. A member of the rry MDisk either hs technology or cpbility tht does not mtch exctly with the estblished gols of the rry. 2. The rry is configured to wnt loction mtches, nd the drive loction does not mtch ll the loction gols. User response: The error will fix itself utomticlly s soon s the rebuild or exchnge is queued up. It does not wit until the rry is showing blnced = exct (which indictes tht ll populted members hve exct cpbility mtch nd exct loction mtch) Persistent unsupported disk controller configurtion. Explntion: A disk controller configurtion tht might prevent filover for the cluster hs persisted for more thn four hours. The problem ws originlly logged through event, service error code User response: 1. Fix ny higher priority error. In prticulr, follow the service ctions to fix the 1625 error indicted by this error's root event. This error will be mrked s fixed when the root event is mrked s fixed. 2. If the root event cnnot be found, or is mrked s fixed, perform n MDisk discovery nd mrk this error s fixed. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Enclosure/controller fult 1700 Unrecovered Metro Mirror or Globl Mirror reltionship Explntion: This error might be reported fter the recovery ction for cluster filure or complete I/O group filure. The error is reported becuse some Metro Mirror or Globl Mirror reltionships, whose control dt is stored by the I/O group, were ctive t the time of the filure nd the current stte of the reltionship could not be recovered. User response: To fix this error it is necessry to delete ll of the reltionships tht could not be recovered nd then re-crete the reltionships. 1. Note the I/O group index ginst which the error is logged. 2. List ll of the Metro Mirror nd Globl Mirror reltionships tht hve either mster or n uxiliry volume in this I/O group. Use the volume view to determine which volumes in the I/O group you noted hve reltionship defined. 3. Note the detils of the Metro Mirror nd Globl Mirror reltionships tht re listed so tht they cn be re-creted. 4. Delete ll of the Metro Mirror nd Globl Mirror reltionships tht re listed. Note: The error will utomticlly be mrked s fixed once the lst reltionship on the I/O group is deleted. New reltionships should not be creted until the error is fixed. 5. Using the detils noted in step 3, re-crete ll of the Metro Mirror nd Globl Mirror reltionships tht you just deleted. Note: You re ble to delete Metro Mirror or Globl Mirror reltionship from either the mster or uxiliry cluster; however, you must re-crete the reltionship on the mster cluster. Therefore, it might be necessry to go to nother cluster to complete this service ction. Possible Cuse-FRUs or other: v None 1710 There re too mny cluster prtnerships. The number of cluster prtnerships hs been reduced. Explntion: A cluster cn hve Metro Mirror nd Globl Mirror cluster prtnership with one or more other clusters. Prtnership sets consist of clusters tht re either in direct prtnership with ech other or re in indirect prtnership by hving prtnership with the sme intermedite cluster. The topology of the prtnership set is not fixed; the topology might be str, loop, chin or mesh. The mximum supported number of clusters in prtnership set is four. A cluster is member of prtnership set if it hs prtnership with nother cluster in the set, regrdless of whether tht prtnership hs ny defined consistency groups or reltionships. 224 SAN Volume Controller: Troubleshooting Guide

243 These re exmples of vlid prtnership sets for five unique clusters lbelled A, B, C, D, nd E where prtnership is indicted by dsh between two cluster nmes: v A-B, A-C, A-D. E hs no prtnerships defined nd therefore is not member of the set. v v v v A-B, A-D, B-C, C-D. E hs no prtnerships defined nd therefore is not member of the set. A-B, B-C, C-D. E hs no prtnerships defined nd therefore is not member of the set. A-B, A-C, A-D, B-C, B-D, C-D. E hs no prtnerships defined nd therefore is not member of the set. A-B, A-C, B-C. D-E. There re two prtnership sets. One contins clusters A, B, nd C. The other contins clusters D nd E. These re exmples of unsupported configurtions becuse the number of clusters in the set is five, which exceeds the supported mximum of four clusters: v A-B, A-C, A-D, A-E. v A-B, A-D, B-C, C-D, C-E. v A-B, B-C, C-D, D-E. The cluster prevents you from creting new Metro Mirror nd Globl Mirror cluster prtnership if resulting prtnership set would exceed the mximum of four clusters. However, if you restore broken link between two clusters tht hve prtnership, the number of clusters in the set might exceed four. If this occurs, Metro Mirror nd Globl Mirror cluster prtnerships re excluded from the set until only four clusters remin in the set. A cluster prtnership tht is excluded from set hs ll of its Metro Mirror nd Globl Mirror cluster prtnerships excluded. Event ID 0x is reported if the cluster is retined in the prtnership set. Event ID 0x is reported if the cluster is excluded from the prtnership set. All clusters tht were in the prtnership set report error All inter-cluster Metro Mirror or Globl Mirror reltionships tht involve n excluded cluster will lose connectivity. If ny of these reltionships re in the consistent_synchronized stte nd they receive write I/O, they will stop with error code User response: To fix this error it is necessry to delete ll of the reltionships tht could not be recovered nd then re-crete the reltionships. 1. Determine which clusters re still connected nd members of the prtnership set, nd which clusters hve been excluded. 2. Determine the Metro Mirror nd Globl Mirror reltionships tht exist on those clusters. 3. Determine which of the Metro Mirror nd Globl Mirror reltionships you wnt to mintin, which determines which cluster prtnerships you wnt to mintin. Ensure tht the prtnership set or sets tht would result from configuring the cluster prtnerships tht you wnt contin no more thn four clusters in ech set. NOTE: The reduced prtnership set creted by the cluster might not contin the clusters tht you wnt in the set. 4. Remove ll of the Metro Mirror nd Globl Mirror reltionships tht you do not wnt to retin. 5. Remove ll of the Metro Mirror nd Globl Mirror cluster prtnerships tht you do not wnt to retin. 6. Restrt ll reltionships nd consistency groups tht were stopped. 7. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None 1720 Metro Mirror (Remote copy) - Reltionship hs stopped nd lost synchroniztion, for reson other thn persistent I/O error (LSYNC) Explntion: A remote copy reltionship or consistency group needs to be restrted. In Metro Mirror (remote copy) or Globl Mirror opertion, the reltionship hs stopped nd lost synchroniztion, for reson other thn persistent I/O error. User response: The dministrtor must exmine the stte of the system to vlidte tht everything is online to llow restrt to work. Exmining the stte of the system lso requires checking the prtner Fibre Chnnel (FC) port msks on both clusters. 1. If the prtner FC port msk ws chnged recently, check tht the correct msk ws selected. 2. Perform whtever steps re needed to mintin consistent secondry, if desired. 3. The dministrtor must issue strt commnd. Possible Cuse-FRUs or other: v None 1740 Recovery encryption key not vilble. Explntion: Recovery encryption key is not vilble. User response: Mke the recovery encryption key vilble. 1. If the key is not vilble: v Instll USB drive with the encryption key. v Ensure correct file is on the USB drive. 2. If the key is not vlid: v Get USB drive with vlid key for this MTMS. The key does not hve vlid CRC. Possible Cuse-FRUs or other: No FRU Chpter 7. Dignosing problems 225

244 Drive uninitilized Explntion: Drive uninitilized User response: If encryption is enbled, ensure tht there is ccess to encryption key on the USB drive on the ctive cnister. Formt the drive if necessry. Possible Cuse-FRUs or other: No FRU 1750 Arry response time too high. Explntion: A number of cuses cn led to higher-thn-usul rry response time. User response: 1. Fix higher priority errors first. 2. Fix ny other known errors. 3. Chnge the rry into redundncy mode by using the chrry interfce. Possible Cuse-FRUs or other: Environment or configurtion issues: Volume config 30% Slow drive 30% Enclosure 20% SAS port 20 Dt: v None User response: Chnge the zoning nd/or Fibre Chnnel port msking so tht no more thn 16 logins re possible between pir of nodes. See Non-criticl node error 888 on pge 190 for detils. Use the lsfbric commnd to view the current number of logins between nodes. Possible Cuse-FRUs or other cuse: v None 1802 Fibre Chnnel network settings Explntion: Fibre Chnnel network settings User response: Follow these troubleshooting steps to reduce the number of hosts tht re logged in to the port: 1. Increse the grnulrity of the switch zoning to reduce unnecessry host port logins. 2. Chnge switch zoning to spred out host ports cross other vilble ports. 3. Use interfces with more ports, if not lredy t the mximum. 4. Scle out by using nother FlshSystem enclosure. Possible Cuse-FRUs or other: No FRU 1800 The SAN hs been zoned incorrectly. Explntion: This hs resulted in more thn 512 other ports on the SAN logging into one port of 2145 node. User response: 1. Ask the user to reconfigure the SAN. 2. Mrk the error s fixed. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: v Fibre Chnnel switch configurtion error v Fibre Chnnel switch 1801 A node hs received too mny Fibre Chnnel logins from nother node. Explntion: This event ws logged becuse the node hs received more thn sixteen Fibre Chnnel logins originting from nother node. This indictes tht the Fibre Chnnel storge re network tht connects the two nodes is not correctly configured IB network settings Explntion: IB network settings User response: Follow these troubleshooting steps to reduce the number of hosts tht re logged in to the port: 1. Increse the grnulrity of the switch zoning to reduce unnecessry host port logins. 2. Chnge switch zoning to spred out host ports cross other vilble ports. 3. Use interfces with more ports, if not lredy t the mximum. 4. Scle out by using nother FlshSystem enclosure. Possible Cuse-FRUs or other: No FRU 1840 The mnged disk hs bd blocks. Explntion: These re "virtul" medium errors which re creted when copying volume where the source hs medium errors. During dt moves or dupliction, such s during flsh copy, n ttempt is mde to 226 SAN Volume Controller: Troubleshooting Guide

245 move medium errors; to chieve this, virtul medium errors clled bd blocks re creted. Once bd block hs been creted, no ttempt will be mde to red the underlying dt, s there is no gurntee tht the old dt still exists once the bd block is creted. Therefore, it is possible to hve bd blocks, nd thus medium errors, reported on trget volume, without medium errors ctully existing on the underlying storge. The bd block records re removed when the dt is overwritten by host. Note: On n externl controller, this error cn only result from copied medium error. User response: 1. The support center will direct the user to restore the dt on the ffected volumes. 2. When the volume dt hs been restored, or the user hs chosen not to restore the dt, mrk the error s fixed. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None 1850 Compressed volume copy hs bd blocks Explntion: A system recovery opertion ws performed, but dt on one or more volumes ws not recovered; this is normlly cused by combintion of hrdwre fults. If dt contining medium error is copied or migrted to nother volume, bd blocks will be recorded. If host ttempts to red the dt in ny of the bd block regions, the red will fil with medium error. User response: 1. The support center will direct the user to restore the dt on the ffected volumes. 2. When the volume dt hs been restored, or the user hs chosen not to restore the dt, mrk the error s fixed. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None 1860 Thin-provisioned volume copy offline becuse of filed repir. Explntion: The ttempt to repir the metdt of thin-provisioned volume tht describes the disk contents hs filed becuse of problems with the utomticlly mintined bckup copy of this dt. The error event dt describes the problem. User response: Delete the thin-provisioned volume nd reconstruct new one from bckup or mirror copy. Mrk the error s fixed. Also mrk the originl 1862 error s fixed. Possible Cuse-FRUs or other: v None 1862 Thin-provisioned volume copy offline becuse of corrupt metdt. Explntion: A thin-provisioned volume hs been tken offline becuse there is n inconsistency in the cluster metdt tht describes the disk contents. This might occur becuse of corruption of dt on the physicl disk (e.g., medium error or dt miscompre), the loss of cched metdt (becuse of cluster recovery) or becuse of softwre error. The event dt gives informtion on the reson. The cluster mintins bckup copies of the metdt nd it might be possible to repir the thin-provisioned volume using this dt. User response: The cluster is ble to repir the inconsistency in some circumstnces. Run the repir volume option to strt the repir process. This repir process, however, cn tke some time. In some situtions it might be more pproprite to delete the thin-provisioned volume nd reconstruct new one from bckup or mirror copy. If you run the repir procedure nd it completes, this error is utomticlly mrked s fixed ; otherwise, nother error event (error code 1860) is logged to indicte tht the repir ction hs filed. Possible Cuse-FRUs or other: v None 1865 Thin-provisioned volume copy offline becuse of insufficient spce. Explntion: A thin-provisioned volume hs been tken offline becuse there is insufficient llocted rel cpcity vilble on the volume for the used spce to increse further. If the thin-provisioned volume is uto-expnd enbled, then the storge pool it is in lso hs no free spce. User response: The service ction differs depending on whether the thin-provisioned volume copy is uto-expnd enbled or not. Whether the disk is uto-expnd enbled or not is indicted in the error event dt. If the volume copy is uto-expnd enbled, perform one or more of the following ctions. When you hve performed ll of the ctions tht you intend to perform, mrk the error s fixed ; the volume copy will then return online. v Determine why the storge pool free spce hs been depleted. Any of the thin-provisioned volume copies, with uto-expnd enbled, in this storge pool might hve expnded t n unexpected rte; this could Chpter 7. Dignosing problems 227

246 v v v v indicte n ppliction error. New volume copies might hve been creted in, or migrted to, the storge pool. Increse the cpcity of the storge pool tht is ssocited with the thin-provisioned volume copy by dding more MDisks to the storge pool. Provide some free cpcity in the storge pool by reducing the used spce. Volume copies tht re no longer required cn be deleted, the size of volume copies cn be reduced or volume copies cn be migrted to different storge pool. Migrte the thin-provisioned volume copy to storge pool tht hs sufficient unused cpcity. Consider reducing the vlue of the storge pool wrning threshold to give more time to llocte extr spce. If the volume copy is not uto-expnd enbled, perform one or more of the following ctions. In this cse the error will utomticlly be mrked s fixed, nd the volume copy will return online when spce is vilble. v v v v Determine why the thin-provisioned volume copy used spce hs grown t the rte tht it hs. There might be n ppliction error. Increse the rel cpcity of the volume copy. Enble uto-expnd for the thin-provisioned volume copy. Consider reducing the vlue of the thin-provisioned volume copy wrning threshold to give more time to llocte more rel spce. Possible Cuse-FRUs or other: v None 1870 Mirrored volume offline becuse hrdwre red error hs occurred. Explntion: While ttempting to mintin the volume mirror, hrdwre red error occurred on ll of the synchronized volume copies. The volume copies might be inconsistent, so the volume is now offline. User response: v Fix ll higher priority errors. In prticulr, fix ny red errors tht re listed in the sense dt. This error event will utomticlly be fixed when the root event is mrked s fixed. v If you cnnot fix the root error, but the red errors on some of the volume copies hve been fixed, mrk this error s fixed to run without the mirror. You cn then delete the volume copy tht cnnot red dt nd re-crete it on different MDisks Unrecovered FlshCopy mppings Explntion: This error might be reported fter the recovery ction for cluster filure or complete I/O group filure. The error is reported becuse some FlshCopies, whose control dt is stored by the I/O group, were ctive t the time of the filure nd the current stte of the mpping could not be recovered. User response: To fix this error it is necessry to delete ll of the FlshCopy mppings on the I/O group tht filed. 1. Note the I/O group index ginst which the error is logged. 2. List ll of the FlshCopy mppings tht re using this I/O group for their bitmps. You should get the detiled view of every possible FlshCopy ID. Note the IDs of the mppings whose IO_group_id mtches the ID of the I/O group ginst which this error is logged. 3. Note the detils of the FlshCopy mppings tht re listed so tht they cn be re-creted. 4. Delete ll of the FlshCopy mppings tht re listed. Note: The error will utomticlly be mrked s fixed once the lst mpping on the I/O group is deleted. New mppings cnnot be creted until the error is fixed. 5. Using the detils noted in step 3, re-crete ll of the FlshCopy mppings tht you just deleted. Possible Cuse-FRUs or other: v None 1900 A FlshCopy, Trigger Prepre commnd hs filed becuse cche flush hs filed. Explntion: A FlshCopy, Trigger Prepre commnd hs filed becuse cche flush hs filed. User response: 1. Correct higher priority errors, nd then try the Trigger Prepre commnd gin. 2. Mrk the error tht you hve just repired s fixed. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: Cche flush error (100%) Possible Cuse-FRUs or other: v None 228 SAN Volume Controller: Troubleshooting Guide

247 A FlshCopy mpping tsk ws stopped becuse of the error tht is indicted in the sense dt. Explntion: A stopped FlshCopy might ffect the sttus of other volumes in the sme I/O group. Prepring the stopped FlshCopy opertions s soon s possible is dvised. User response: 1. Correct higher priority errors, nd then prepre nd strt the FlshCopy tsk gin. 2. Mrk the error tht you hve just repired s fixed. 3. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None 1920 Globl nd Metro Mirror persistent error. Explntion: This error might be cused by problem on the primry cluster, problem on the secondry cluster, or problem on the inter-cluster link. The problem might be filure of component, component becoming unvilble or hving reduced performnce becuse of service ction or it might be tht the performnce of component hs dropped to level where the Metro Mirror or Globl Mirror reltionship cnnot be mintined. Alterntively the error might be cused by chnge in the performnce requirements of the pplictions using Metro Mirror or Globl Mirror. This error is reported on the primry cluster when the copy reltionship hs not progressed sufficiently over period of time. Therefore, if the reltionship is restrted before ll of the problems re fixed, the error might be reported gin when the time period next expires (the defult period is five minutes). This error might lso be reported becuse the primry cluster hs encountered red errors. You might need to refer to the Copy Services fetures informtion in the softwre instlltion nd configurtion documenttion while dignosing this error. User response: 1. If the 1920 error hs occurred previously on Metro Mirror or Globl Mirror between the sme clusters nd ll the following ctions hve been ttempted, contct your product support center to resolve the problem. 2. On both clusters, check the prtner fc port msk to ensure tht there is sufficient connectivity. If the prtner fc port msk ws chnged recently, ensure the msk is correct. 3. On the primry cluster reporting the error, correct ny higher priority errors. 4. On the secondry cluster, review the mintennce logs to determine if the cluster ws operting with reduced cpbility t the time the error ws reported. The reduced cpbility might be becuse of softwre upgrde, hrdwre mintennce to 2145 node, mintennce to bckend disk system or mintennce to the SAN. 5. On the secondry 2145 cluster, correct ny errors tht re not fixed. 6. On the intercluster link, review the logs of ech link component for ny incidents tht would cuse reduced cpbility t the time of the error. Ensure the problems re fixed. 7. If reson for the error hs been found nd corrected, go to Action On the primry cluster reporting the error, exmine the 2145 sttistics using SAN productivity monitoring tool nd confirm tht ll the Metro Mirror nd Globl Mirror requirements described in the plnning documenttion re met. Ensure tht ny chnges to the pplictions using Metro Mirror or Globl Mirror hve been tken into ccount. Resolve ny issues. 9. On the secondry cluster, exmine the 2145 sttistics using SAN productivity monitoring tool nd confirm tht ll the Metro Mirror nd Globl Mirror requirements described in the softwre instlltion nd configurtion documenttion re met. Resolve ny issues. 10. On the intercluster link, exmine the performnce of ech component using n pproprite SAN productivity monitoring tool to ensure tht they re operting s expected. Resolve ny issues. 11. Mrk the error s fixed nd restrt the Metro Mirror or Globl Mirror reltionship. When you restrt the Metro Mirror or Globl Mirror reltionship there will be n initil period during which Metro Mirror or Globl Mirror performs bckground copy to resynchronize the volume dt on the primry nd secondry clusters. During this period the dt on the Metro Mirror or Globl Mirror uxiliry volumes on the secondry cluster is inconsistent nd the volumes could not be used s bckup disks by your pplictions. Note: To ensure the system hs the cpcity to hndle the bckground copy lod you my wnt to dely restrting the Metro Mirror or Globl Mirror reltionship until there is quiet period when the secondry cluster nd the SAN fbric (including the intercluster link) hve the required cpcity. If the required cpcity is not vilble you might experience nother 1920 error nd the Metro Mirror or Globl Mirror reltionship will stop in n inconsistent stte. Chpter 7. Dignosing problems 229

248 Note: If the Metro Mirror or Globl Mirror reltionship hs stopped in consistent stte ( consistent-stopped ) it is possible to use the dt on the Metro Mirror or Globl Mirror uxiliry volumes on the secondry cluster s bckup disks by your pplictions. You might therefore wnt to strt Flsh Copy of your Metro Mirror or Globl Mirror uxiliry disks on the secondry system before restrting the Metro Mirror or Globl Mirror reltionship. This mens you mintin the current, consistent, imge until the time when the Metro Mirror or Globl Mirror reltionship is gin synchronized nd in consistent stte. Possible Cuse-FRUs or other: v None Other: v Primry 2145 cluster or SAN fbric problem (10%) v Primry 2145 cluster or SAN fbric configurtion (10%) v Secondry 2145 cluster or SAN fbric problem (15%) v Secondry 2145 cluster or SAN fbric configurtion (25%) v Intercluster link problem (15%) v Intercluster link configurtion (25%) 1925 Cched dt cnnot be destged. Explntion: Problem dignosis is required. User response: 1. Run the directed mintennce procedure to fix ll errors of higher priority. This will llow the cched dt to be destged nd the originting event to be mrked fixed. Possible Cuse-FRUs or other: v None 1930 Migrtion suspended. Explntion: Migrtion suspended. User response: 1. Ensure tht ll error codes of higher priority hve lredy been fixed. 2. Ask the customer to ensure tht ll storge pools tht re the destintion of suspended migrte opertions hve vilble free extents. 3. Mrk this error s fixed. This cuses the migrte opertion to be restrted. If the restrt fils, new error is logged. 4. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None 1950 Unble to mirror medium error. Explntion: During the synchroniztion of mirrored volume copy it ws necessry to duplicte the record of medium error onto the volume copy, creting virtul medium error. Ech mnged disk hs tble of virtul medium errors. The virtul medium error could not be creted becuse the tble is full. The volume copy is in n inconsistent stte nd hs been tken offline. User response: Three different pproches cn be tken to resolving this problem: 1) the source volume copy cn be fixed so tht it does not contin medium errors, 2) the number of virtul medium errors on the trget mnged disk cn be reduced or 3) the trget volume copy cn be moved to mnged disk with more free virtul medium error entries. The mnged disk with full medium error tble cn be determined from the dt of the root event. Approch 1) - This is the preferred procedure becuse it restores the source volume copy to stte where ll of the dt cn be red. Use the norml service procedures for fixing medium error (rewrite block or volume from bckup or regenerte the dt using locl procedures). Approch 2) - This method cn be used if the mjority of the virtul medium errors on the trget mnged disk do not relte to the volume copy. Determine where the virtul medium errors re using the event log events nd re-write the block or volume from bckup. Approch 3) - Delete the offline volume copy nd crete new one either forcing the use of different MDisks in the storge pool or using completely different storge pool. Follow your selection option(s) nd then mrk the error s fixed. Possible Cuse-FRUs or other: v None 2008 A softwre downgrde hs filed. Explntion: Cluster configurtion chnges re restricted until the downgrde is completed. The cluster downgrde process wits for user intervention when this error is logged. User response: The ction required to recover from stlled downgrde depends on the current stte of the cluster being downgrded. Cll IBM Support for n ction pln to resolve this problem. Possible Cuse-FRUs or other: v None Other: 2145 softwre (100%) 230 SAN Volume Controller: Troubleshooting Guide

249 A softwre upgrde hs filed. Explntion: Cluster configurtion chnges re restricted until the upgrde is completed or rolled bck. The cluster upgrde process wits for user intervention when this error is logged. User response: The ction required to recover from stlled upgrde depends on the current stte of the cluster being upgrded. Cll IBM technicl support for n ction pln to resolve this problem. Possible Cuse-FRUs or other: v None Other: 2145 softwre (100%) 2020 IP Remote Copy link unvilble. Explntion: IP Remote Copy link is unvilble. User response: Fix the remote IP link so tht trffic cn flow correctly. Once the connection is mde, the error will uto-correct Prtner cluster IP ddress unrechble. Explntion: Prtner cluster IP ddress unrechble. User response: 1. Verify the system IP ddress of the remote system forming the prtnership. 2. Check if remote cluster IP ddress is rechble from locl cluster. The following cn be done to verify ccessibility:. Use svctsk to ping the remote cluster IP ddress. If the ping works, there my be block on the specific port trffic tht needs to be opened in the network. If the ping does not work, there my be no route between the system. Check the IP gtewy configurtion on the SAN Volume Controller nodes nd the IP network configurtion. b. Check the configurtion of the routers nd firewll to ensure tht TCP/IP port 3620 used for IP repliction is not blocked. c. Use the ssh commnd from nother system to ttempt to estblish session with the problemtic remote cluster IP ddress to confirm tht the remote cluster is opertionl Cnnot uthenticte with prtner cluster. Explntion: Cnnot uthenticte with prtner cluster. User response: Verify the CHAP secret set of prtnership using mkipprtnership or chprtnership CLIs mtch remote system CHAP secret set using chsystem CLI. If they don't mtch, use pproprite commnds to set the right CHAP secrets Unexpected cluster ID for prtner cluster. Explntion: Unexpected cluster ID for prtner cluster. User response: After deleting ll reltionships nd consistency group, remove the prtnership. This is n unrecoverble error when one of the sites hs undergone T3 recovery nd lost ll prtnership informtion. Contct IBM support Softwre error. Explntion: The V3700 softwre hs restrted becuse of problem in the cluster, on disk system or on the Fibre Chnnel fbric. User response: 1. Collect the softwre dump file(s) generted t the time the error ws logged on the cluster. 2. Contct your product support center to investigte nd resolve the problem. 3. Ensure tht the softwre is t the ltest level on the cluster nd on the disk systems. 4. Use the vilble SAN monitoring tools to check for ny problems on the fbric. 5. Mrk the error tht you hve just repired s fixed. 6. Go to repir verifiction Mp. Possible Cuse-FRUs or other: v Your support center might indicte FRU bsed on their problem nlysis (2%) Other: v V3700 softwre (48%) v Enclosure/controller softwre (25%) v Fibre Chnnel switch or switch configurtion (25%) 2040 A softwre upgrde is required. Explntion: The softwre cnnot determine the VPD for FRU. Probbly, new FRU hs been instlled nd the softwre does not recognize tht FRU. User response: 1. If FRU hs been replced, ensure tht the correct replcement prt ws used. The node VPD indictes which prt is not recognized. 2. Ensure tht the cluster softwre is t the ltest level. 3. Sve dump dt with configurtion dump nd logged dt dump. 4. Contct your product support center to resolve the problem. Chpter 7. Dignosing problems 231

250 Mrk the error tht you hve just repired s fixed. 6. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: 2145 softwre (100%) 2070 A drive hs been detected in n enclosure tht does not support tht drive. Explntion: A drive hs been detected in n enclosure tht does not support tht drive. User response: Remove the drive. If the result is n invlid number of drives, replce the drive with vlid drive. Possible Cuse-FRUs or other: Drive (100%) 2100 A softwre error hs occurred. Explntion: One of the V3700 server softwre components (sshd, crond, or httpd) hs filed nd reported n error. User response: 1. Ensure tht the softwre is t the ltest level on the cluster. 2. Sve dump dt with configurtion dump nd logged dt dump. 3. Contct your product support center to resolve the problem. 4. Mrk the error tht you hve just repired s fixed. 5. Go to repir verifiction MAP. Possible Cuse-FRUs or other: v None Other: V3700 softwre (100%) 2500 A secure shell (SSH) session limit for the cluster hs been reched. Explntion: Secure Shell (SSH) sessions re used by pplictions tht mnge the cluster. An exmple of such n ppliction is the commnd-line interfce (CLI). An ppliction must initilly log in to the cluster to crete n SSH session. The cluster imposes limit on the number of SSH sessions tht cn be open t one time. This error indictes tht the limit on the number of SSH sessions hs been reched nd tht no more logins cn be ccepted until current session logs out. The limit on the number of SSH sessions is usully reched becuse multiple users hve opened n SSH session but hve forgotten to close the SSH session when they re no longer using the ppliction. User response: v Becuse this error indictes problem with the number of sessions tht re ttempting externl ccess to the cluster, determine the reson tht so mny SSH sessions hve been opened. v Run the Fix Procedure for this error on the pnel t Mngement GUI Troubleshooting > Recommended Actions to view nd mnge the open SSH sessions The cluster ws unble to send n emil. Explntion: The cluster hs ttempted to send n emil in response to n event, but there ws no cknowledgement tht it ws successfully received by the SMTP mil server. It might hve filed becuse the cluster ws unble to connect to the configured SMTP server, the emil might hve been rejected by the server, or timeout might hve occurred. The SMTP server might not be running or might not be correctly configured, or the cluster might not be correctly configured. This error is not logged by the test emil function becuse it responds immeditely with result code. User response: v Ensure tht the SMTP emil server is ctive. v Ensure tht the SMTP server TCP/IP ddress nd port re correctly configured in the cluster emil configurtion. v Send test emil nd vlidte tht the chnge hs corrected the issue. v Mrk the error tht you hve just repired s fixed. v Go to MAP 5700: Repir verifiction. Possible Cuse-FRUs or other: v None 2601 Error detected while sending n emil. Explntion: An error hs occured while the cluster ws ttempting to send n emil in response to n event. The cluster is unble to determine if the emil hs been sent nd will ttempt to resend it. The problem might be with the SMTP server or with the cluster emil configurtion. The problem might lso be cused by filover of the configurtion node. This error is not logged by the test emil function becuse it responds immeditely with result code. User response: 232 SAN Volume Controller: Troubleshooting Guide

251 v If there re higher-priority unfixed errors in the log, fix those errors first. v Ensure tht the SMTP emil server is ctive. v v v v Ensure tht the SMTP server TCP/IP ddress nd port re correctly configured in the cluster emil configurtion. Send test emil nd vlidte tht the chnge hs corrected the issue. Mrk the error tht you hve just repired s fixed. Go to MAP 5700: Repir verifiction. Possible Cuse-FRUs or other: v None 2700 Unble to ccess NTP network time server Explntion: Cluster time cnnot be synchronized with the NTP network time server tht is configured. User response: There re three min cuses to exmine: v The cluster NTP network time server configurtion is incorrect. Ensure tht the configured IP ddress mtches tht of the NTP network time server. v The NTP network time server is not opertionl. Check the sttus of the NTP network time server. v The TCP/IP network is not configured correctly. Check the configurtion of the routers, gtewys nd firewlls. Ensure tht the cluster cn ccess the NTP network time server nd tht the NTP protocol is permitted. The error will utomticlly fix when the cluster is ble to synchronize its time with the NTP network time server. Possible Cuse-FRUs or other: v None 2702 Check configurtion settings of the NTP server on the CMM Explntion: The node is configured to utomticlly set the time using n NTP server within the CMM. It is not possible to connect to the NTP server during uthentiction. The NTP server configurtion cnnot be chnged within S-ITE. Within the CMM, there re chngeble NTP settings. However, these settings configure how the CMM gets the time nd dte - the internl CMM NTP server tht is used by the S-ITE cnnot be chnged or configured. This event is only rised when n ttempt is mde to use the server - once every hlf hour. Note: The NTP configurtion settings re re-red from the CMM before ech connection. The reson for connection error cn be due to the following: v ll suitble Ethernet ports re offline v the CMM hrdwre is not opertionl v the CMM is ctive but the CMM NTP server is offline. The reson for n uthentiction issue cn be due to the following: v the uthentiction vlues provided were invlid v the NTP server rejected the uthentiction key provided to the node by the CMM. If the NTP port is n unsupported vlue, port error cn disply. Currently, only port 123 is supported. Only the current configurtion node ttempts to resync with the server. User response: 1. Mke sure tht CMM is opertionl by logging in nd confirming its time. 2. Check tht the IP ddress in the event log cn be pinged from the node. 3. If there is n error, try rebooting the CMM The 2145 UPS temperture is close to its upper limit. If the temperture continues to rise the 2145 UPS will power off. Explntion: The temperture sensor in the 2145 UPS is reporting temperture tht is close to the opertionl limit of the unit. If the temperture continues to rise the 2145 UPS will power off for sfety resons. The sensor is probbly reporting n excessively high temperture becuse the environment in which the 2145 UPS is operting is too hot. User response: 1. Ensure tht the room mbient temperture is within the permitted limits. 2. Ensure tht the ir vents t the front nd bck of the 2145 UPS re not obstructed. 3. Ensure tht other devices in the sme rck re not overheting. 4. When you re stisfied tht the cuse of the overheting hs been resolved, mrk the error fixed The 2145 UPS-1U temperture is close to its upper limit. If the temperture continues to rise the 2145 UPS-1U will power off. Explntion: The temperture sensor in the 2145 UPS-1U is reporting temperture tht is close to the opertionl limit of the unit. If the temperture continues to rise the 2145 UPS-1U will power off for Chpter 7. Dignosing problems 233

252 sfety resons. The sensor is probbly reporting n excessively high temperture becuse the environment in which the 2145 UPS-1U is operting is too hot. User response: 1. Ensure tht the room mbient temperture is within the permitted limits. 2. Ensure tht the ir vents t the front nd bck of the 2145 UPS-1U re not obstructed. 3. Ensure tht other devices in the sme rck re not overheting. 4. When you re stisfied tht the cuse of the overheting hs been resolved, mrk the error fixed Internl uninterruptible power supply softwre error detected. Explntion: Some of the tests tht re performed during node strtup did not complete becuse some of the dt reported by the uninterruptible power supply during node strtup is inconsistent becuse of softwre error in the uninterruptible power supply. The node hs determined tht the uninterruptible power supply is functioning sufficiently for the node to continue opertions. The opertion of the cluster is not ffected by this error. This error is usully resolved by power cycling the uninterruptible power supply. User response: 1. Power cycle the uninterruptible power supply t convenient time. The one or two nodes ttched to the uninterruptible power supply should be powered off before powering off the uninterruptible power supply. Once the nodes hve powered down, wit 5 minutes for the uninterruptible power supply to go into stndby mode (flshing green AC LED). If this does not hppen utomticlly then check the cbling to confirm tht ll nodes powered by this uninterruptible power supply hve been powered off. Remove the power input cble from the uninterruptible power supply nd wit t lest 2 minutes for the uninterruptible power supply to cler its internl stte. Reconnect the uninterruptible power supply power input cble. Press the uninterruptible power supply ON button. Power on the nodes connected to this uninterruptible power supply. 2. If the error is reported gin fter the nodes re restrted replce the 2145 UPS electronics ssembly. Possible Cuse-FRUs or other: v 2145 UPS electronics ssembly (5%) Other: v Trnsient 2145 UPS error (95%) 3024 Technicin port connection invlid Explntion: The code hs detected more thn one MAC ddress through the connection, or the DHCP hs given out more thn one ddress. The code thus believes there is switch ttched. User response: 1. Remove the cble from the technicin port. 2. (Optionl) Disble dditionl network dpters on the lptop to which it is to connected. 3. Ensure DHCP is enbled on the network dpter. 4. If this ws not possible, mnully set the IP to Connect stndrd Ethernet cble between the network dpter nd the technicin port. 6. If this still does not work, reboot the node nd repet the bove steps. 7. This event will uto-fix once either no connection or vlid connection hs been detected A virtuliztion feture license is required. Explntion: The cluster hs no virtuliztion feture license registered. You should hve either n Entry Edition Physicl Disk virtuliztion feture license or Cpcity virtuliztion feture license tht covers the cluster. The cluster will continue to operte, but it might be violting the license conditions. User response: v If you do not hve virtuliztion feture license tht is vlid nd sufficient for this cluster, contct your IBM sles representtive, rrnge license nd chnge the license settings for the cluster to register the license. v The error will utomticlly fix when the sitution is resolved. Possible Cuse-FRUs or other: v None 3029 Virtuliztion feture cpcity is not vlid. Explntion: The setting for the mount of spce tht cn be virtulized is not vlid. The vlue must be n integer number of terbytes. This error event is creted when cluster is upgrded from version prior to to version or lter. Prior to version the virtuliztion feture cpcity vlue ws in gigbytes nd therefore could be set to frction of terbyte. With version nd lter the licensed cpcity for the virtuliztion feture must be n integer number of terbytes. 234 SAN Volume Controller: Troubleshooting Guide

253 User response: v Review the license conditions for the virtuliztion feture. If you hve one cluster, chnge the license settings for the cluster to mtch the cpcity tht is licensed. If your license covers more thn one cluster, pportion n integer number of terbytes to ech cluster. You might hve to chnge the virtuliztion cpcity tht is set on the other clusters to ensure tht the sum of the cpcities for ll of the clusters does not exceed the licensed cpcity. v You cn view the event dt or the feture log to ensure tht the licensed cpcity is sufficient for the spce tht is ctully being used. Contct your IBM sles representtive if you wnt to chnge the cpcity of the license. v This error will utomticlly be fixed when vlid configurtion is entered. Possible Cuse-FRUs or other: v None 3030 Globl nd Metro Mirror feture cpcity not set. Explntion: The Globl nd Metro Mirror feture is set to On for the cluster, but the cpcity hs not been set. This error event is creted when cluster is upgrded from version prior to to version or lter. Prior to version the feture cn only be set to On or Off; with version nd lter the licensed cpcity for the feture must lso be set. User response: Perform one of the following ctions: v Chnge the Globl nd Metro Mirror license settings for the cluster either to the licensed Globl nd Metro Mirror cpcity, or if the license pplies to more thn one cluster, to the portion of the license llocted to this cluster. Set the licensed Globl nd Metro Mirror cpcity to zero if it is no longer being used. v View the event dt or the feture log to ensure tht the licensed Globl nd Metro Mirror cpcity is sufficient for the spce ctully being used. Contct your IBM sles representtive if you wnt to chnge the licensed Globl nd Metro Mirror cpcity. v The error will utomticlly be fixed when vlid configurtion is entered. Possible Cuse-FRUs or other: v None 3031 FlshCopy feture cpcity not set. Explntion: The FlshCopy feture is set to On for the cluster, but the cpcity hs not been set. This error event is creted when cluster is upgrded from version prior to to version or lter. Prior to version the feture cn only be set to On or Off; with version nd lter the licensed cpcity for the feture must lso be set. User response: Perform one of the following ctions: v Chnge the FlshCopy license settings for the cluster either to the licensed FlshCopy cpcity, or if the license pplies to more thn one cluster, to the portion of the license llocted to this cluster. Set the licensed FlshCopy cpcity to zero if it is no longer being used. v View the event dt or the feture log to ensure tht the licensed FlshCopy cpcity is sufficient for the spce ctully being used. Contct your IBM sles representtive if you wnt to chnge the licensed FlshCopy cpcity. v The error will utomticlly be fixed when vlid configurtion is entered. Possible Cuse-FRUs or other: v None 3032 Feture license limit exceeded. Explntion: The mount of spce tht is licensed for cluster feture is being exceeded. The feture tht is being exceeded might be: v Virtuliztion feture - event identifier v FlshCopy feture - event identifier v Globl nd Metro Mirror feture - event identifier The cluster will continue to operte, but it might be violting the license conditions. User response: v Determine which feture license limit hs been exceeded. This might be: v Virtuliztion feture - event identifier v FlshCopy feture - event identifier v Globl nd Metro Mirror feture - event identifier v Ensure tht the feture cpcity tht is reported by the cluster hs been set to mtch either the licensed size, or if the license pplies to more thn one cluster, to the portion of the license tht is llocted to this cluster. v Decide whether to increse the feture cpcity or to reduce the spce tht is being used by this feture. v To increse the feture cpcity, contct your IBM sles representtive nd rrnge n incresed license cpcity. Chnge the license settings for the cluster to set the new licensed cpcity. Alterntively, if the license pplies to more thn one cluster modify how the licensed cpcity is pportioned between the clusters. Updte every cluster so tht the sum of the Chpter 7. Dignosing problems 235

254 v v v v license cpcity for ll of the clusters does not exceed the licensed cpcity for the loction. To reduce the mount of disk spce tht is virtulized, delete some of the mnged disks or imge mode volumes. The used virtuliztion size is the sum of the cpcities of ll of the mnged disks nd imge mode disks. To reduce the FlshCopy cpcity delete some FlshCopy mppings. The used FlshCopy size is the sum of ll of the volumes tht re the source volume of FlshCopy mpping. To reduce Globl nd Metro Mirror cpcity delete some Globl Mirror or Metro Mirror reltionships. The used Globl nd Metro Mirror size is the sum of the cpcities of ll of the volumes tht re in Metro Mirror or Globl Mirror reltionship; both mster nd uxiliry volumes re counted. The error will utomticlly be fixed when the licensed cpcity is greter thn the cpcity tht is being used. Mirror license registered on the cluster. The cluster will continue to operte, but it might be violting the license conditions. User response: v Check if you hve n Entry Edition Physicl Disk Globl nd Metro Mirror license for this cluster tht you hve not registered on the cluster. Updte the cluster license configurtion if you hve license. v Decide whether you wnt to continue to use the Globl Mirror or Metro Mirror fetures or not. v If you wnt to use either the Globl Mirror or Metro Mirror feture contct your IBM sles representtive, rrnge license nd chnge the license settings for the cluster to register the license. v If you do not wnt to use both the Globl Mirror nd Metro Mirror fetures, you must delete ll of the Globl Mirror nd Metro Mirror reltionships. v The error will utomticlly fix when the sitution is resolved. Possible Cuse-FRUs or other: v None 3035 Physicl Disk FlshCopy feture license required Explntion: The Entry Edition cluster hs some FlshCopy mppings defined. There is, however, no Physicl Disk FlshCopy license registered on the cluster. The cluster will continue to operte, but it might be violting the license conditions. User response: v Check whether you hve n Entry Edition Physicl Disk FlshCopy license for this cluster tht you hve not registered on the cluster. Updte the cluster license configurtion if you hve license. v Decide whether you wnt to continue to use the FlshCopy feture or not. v If you wnt to use the FlshCopy feture contct your IBM sles representtive, rrnge license nd chnge the license settings for the cluster to register the license. v If you do not wnt to use the FlshCopy feture, you must delete ll of the FlshCopy mppings. v The error will utomticlly fix when the sitution is resolved. Possible Cuse-FRUs or other: v None 3036 Physicl Disk Globl nd Metro Mirror feture license required Explntion: The Entry Edition cluster hs some Globl Mirror or Metro Mirror reltionships defined. There is, however, no Physicl Disk Globl nd Metro Possible Cuse-FRUs or other: v None 3080 Globl or Metro Mirror reltionship or consistency group with deleted prtnership Explntion: A Globl Mirror or Metro Mirror reltionship or consistency group exists with cluster whose prtnership is deleted. Beginning with SAN Volume Controller softwre version this configurtion is not supported nd should be resolved. This condition cn occur s result of n upgrde to SAN Volume Controller softwre version or lter. User response: The issue cn be resolved either by deleting ll of the Globl Mirror or Metro Mirror reltionships or consistency groups tht exist with cluster whose prtnership is deleted, or by recreting ll of the prtnerships tht they were using. The error will utomticlly fix when the sitution is resolved. v v v List ll of the Globl Mirror nd Metro Mirror reltionships nd note those where the mster cluster nme or the uxiliry cluster nme is blnk. For ech of these reltionships, lso note the cluster ID of the remote cluster. List ll of the Globl Mirror nd Metro Mirror consistency groups nd note those where the mster cluster nme or the uxiliry cluster nme is blnk. For ech of these consistency groups, lso note the cluster ID of the remote cluster. Determine how mny unique remote cluster IDs there re mong ll of the Globl Mirror nd Metro Mirror reltionships nd consistency groups tht you hve identified in the first two steps. For ech of 236 SAN Volume Controller: Troubleshooting Guide

255 v v v these remote clusters, decide if you wnt to re-estblish the prtnership with tht cluster. Ensure tht the totl number of prtnerships tht you wnt to hve with remote clusters does not exceed the cluster limit. In version this limit is 1. If you re-estblish prtnership, you will not hve to delete the Globl Mirror nd Metro Mirror reltionships nd consistency groups tht use the prtnership. Re-estblish ny selected prtnerships. Delete ll of the Globl Mirror nd Metro Mirror reltionships nd consistency groups tht you listed in either of the first two steps whose remote cluster prtnership hs not been re-estblished. Check tht the error hs been mrked s fixed by the system. If it hs not, return to the first step nd determine which Globl Mirror or Metro Mirror reltionships or consistency groups re still cusing the issue. Possible Cuse-FRUs or other: v None 3081 Unble to send emil to ny of the configured emil servers. Explntion: Either the system ws not ble to connect to ny of the SMTP emil servers, or the emil trnsmission hs filed. A mximum of six emil servers cn be configured. Error event 2600 or 2601 is rised when n individul emil server is found to be not working. This error indictes tht ll of the emil servers were found to be not working. User response: v Check the event log for ll unresolved 2600 nd 2601 errors nd fix those problems. v If this error hs not lredy been utomticlly mrked fixed, mrk this error s fixed. v Perform the check emil function to test tht n emil server is operting properly. Possible Cuse-FRUs or other: v None 3090 Drive firmwre downlod is cncelled by user or system, problem dignosis required. Explntion: Drive firmwre downlod is cncelled by user or system, problem dignosis required. User response: 1. If system stte cuses this error, check the system stte. 2. Usesvctsk chdrive -use to chnge the drive use. v Drive stte becomes degrded, offline or filed. v Node stte becomes pending. v Redundncy stte becomes loss or check. Itis necessry to crete new volume on RAID0. v Dependency stte becomes hppen or check. 3. If file ccess error occurs, check the pckge file. 4. If device error occurs, check the drive itself. 5. Finlly, downlod gin. Procedure: SAN problem determintion You cn solve problems on the SAN Volume Controller system nd its connection to the storge re network (SAN). About this tsk SAN filures might cuse SAN Volume Controller volumes to be inccessible to host systems. Filures cn be cused by SAN configurtion chnges or by hrdwre filures in SAN components. The following list identifies some of the hrdwre tht might cuse filures: v Power, fn, or cooling switch v Appliction-specific integrted circuits v Instlled smll form-fctor pluggble (SFP) trnsceiver v Fiber-optic cbles If either the mintennce nlysis procedures or the error codes sent you here, complete the following steps: Procedure 1. If the customer chnged the SAN configurtion by chnging the Fibre Chnnel cble connections or switch zoning, verify tht the chnges were correct nd, if necessry, reverse those chnges. Chpter 7. Dignosing problems 237

256 2. Verify tht the power is turned on to ll switches nd storge controllers tht the SAN Volume Controller system uses, nd tht they re not reporting ny hrdwre filures. If problems re found, resolve those problems before you proceed further. 3. Verify tht the Fibre Chnnel cbles tht connect the systems to the switches re securely connected. 4. If the customer is running SAN mngement tool, you cn use tht tool to view the SAN topology nd isolte the filing component. Resolving problem with new expnsion enclosures Determine why newly instlled expnsion enclosure ws not detected by the system. When you instll new expnsion enclosure, follow the mngement GUI Add Enclosure wizrd, which is vilble from the Mnge Devices Actions menu. If the expnsion enclosure is not detected, complete the following verifictions: v Verify the sttus of the LEDs t the bck of the expnsion enclosure. At lest one power supply unit must be on with no fults shown. At lest one cnister must be ctive, with no fult LED on. The SAN Volume Controller F hs two LEDs per Seril-ttched SCSI (SAS) port: one green link-sttus LED nd one mber fult LED. The link sttus LED of the ports tht re in use is on while the fult LED is off. For detils bout LED sttus, see SAN Volume Controller F expnsion cnister SAS ports nd indictors. v Verify tht the SAS cbling to the expnsion enclosure is correctly instlled. To review the requirements, see Connecting the optionl SAN Volume Controller F expnsion enclosures. Fibre Chnnel nd 10G Ethernet link filures You might need to replce the smll form-fctor pluggble (SFP) trnsceiver when filure occurs on single Fibre Chnnel or 10G Ethernet link (pplicble to Fibre Chnnel over Ethernet personlity enbled 10G Ethernet link). Before you begin The following items cn indicte tht single Fibre Chnnel or 10 G Ethernet link filed: v The Fibre Chnnel port sttus on the front pnel of the node v The Fibre Chnnel sttus light-emitting diodes (LEDs) t the rer of the node v An error tht indictes tht single port filed (703, 723). About this tsk Use only IBM supported 10 Gb SFP trnsceivers with the SAN Volume Controller 2145-DH8. Using ny other SFP trnsceivers cn led to unexpected system behvior. Copper DAC is not supported by these 10 Gb ports. The SFP trnsceiver replcement in 10 Gbps Ethernet dpter port is governed by the following rules: v v An existing 10 Gb SFP trnsceiver replced with new 10 Gb SFP trnsceiver: The 10 Gbps Ethernet dpter port detects new SFP trnsceiver nd becomes opertionl immeditely. If the 10 Gbps Ethernet dpter port detects new SFP trnsceiver nd becomes opertionl immeditely, the port hs n incorrect SFP trnsceiver since the lst 238 SAN Volume Controller: Troubleshooting Guide

257 v reboot nd is replced with the correct 10 Gb SFP trnsceiver. This sitution cn occur with n incomptible SFP trnsceiver (8 Gb SFP or 4 Gb SFP) tht is inserted in the 10 Gbps Ethernet dpter port. The node will require reboot for detecting the new SFP trnsceiver. The new SFP trnsceiver will be opertionl only fter reboot (no DMP is produced). The 10 Gbps Ethernet dpter port contins no SFP trnsceiver since the lst reboot nd the correct 10 Gb SFP trnsceiver is instlled: System reboot is required for detecting the new SFP trnsceiver. Procedure Attempt ech of these ctions, in the following order, until the filure is fixed. 1. Replce the SFP trnsceiver for the filing port on the node. Note: SAN Volume Controller nodes re supported by both longwve SFP trnsceivers nd shortwve SFP trnsceivers. You must replce n SFP trnsceiver with the sme type of SFP trnsceiver. If the SFP trnsceiver to replce is longwve SFP trnsceiver, for exmple, you must provide suitble replcement. Removing the wrong SFP trnsceiver might result in loss of dt ccess. 2. Replce the Fibre Chnnel dpter on the node.replce the Fibre Chnnel dpter or Fibre Chnnel over Ethernet dpter on the node. Ethernet iscsi host-link problems If you re hving problems ttching to the Ethernet hosts, your problem might be relted to the network, the SAN Volume Controller system, or the host. Before you begin For network problems, you cn ttempt ny of the following ctions: v Test your connectivity between the host nd SAN Volume Controller ports. v Try to ping the SAN Volume Controller system from the host. v Ask the Ethernet network dministrtor to check the firewll nd router settings. v Check tht the subnet msk nd gtewy re correct for the SAN Volume Controller host configurtion. Using the mngement GUI for SAN Volume Controller problems, you cn ttempt ny of the following ctions: v View the configured node port IP ddresses. v v View the list of volumes tht re mpped to host to ensure tht the volume host mppings re correct. Verify tht the volume is online. For host problems, you cn ttempt ny of the following ctions: v Verify tht the host iscsi qulified nme (IQN) is correctly configured. v Use operting system utilities (such s Windows device mnger) to verify tht the device driver is instlled, loded, nd operting correctly. Chpter 7. Dignosing problems 239

258 Fibre Chnnel over Ethernet host-link problems Problems ttching to the Fibre Chnnel over Ethernet hosts might be relted to the network, the SAN Volume Controller system, or the host. Before you begin If error code 705 on node is displyed, this mens the FC I/O port is inctive. Fibre Chnnel over Ethernet uses Fibre Chnnel s protocol nd Ethernet s n inter-connect. Note: Concerning Fibre Chnnel over Ethernet enbled port: either the fibre chnnel forwrder (FCF) is not seen, or the Fibre Chnnel over Ethernet feture is not configured on switch. v Verify tht the Fibre Chnnel over Ethernet feture is enbled on the FCF. v Verify the remote port (switch port) properties on the FCF. If connecting the host through Converged Enhnced Ethernet (CEE) Switch: v Test your connectivity between the host nd CEE Switch. v Ask the Ethernet network dministrtor to check the firewll nd router settings. Run lsfbric, nd verify the host is seen s remote port in the output. If the host is not seen, in order: v v v Verify tht SAN Volume Controller nd host get n Fibre Chnnel ID (FCID) on the FCF. If unble to verify, check the VLAN configurtion. Verify tht SAN Volume Controller nd host port re prt of zone nd tht zone is currently in force. Verify the volumes re mpped to host nd re online. For more informtion, see lshostvdiskmp nd lsvdisk in the description in the SAN Volume Controller Informtion Center. Wht to do next Servicing storge systems If the problem is not resolved, verify the stte of the host dpter. v Unlod nd lod the device driver v Use the operting system utilities (for exmple, Windows Device Mnger) to verify the device driver is instlled, loded, nd operting correctly. Storge systems tht re supported for ttchment to the SAN Volume Controller system re designed with redundnt components nd ccess pths to enble concurrent mintennce. Hosts hve continuous ccess to their dt during component filure nd replcement. The following ctegories represent the types of service ctions for storge systems: v Controller code upgrde v Field replceble unit (FRU) replcement Controller code upgrde Ensure tht you re fmilir with the following guidelines for upgrding controller code: 240 SAN Volume Controller: Troubleshooting Guide

259 v v v Check to see if the SAN Volume Controller supports concurrent mintennce for your storge system. Allow the storge system to coordinte the entire upgrde process. If it is not possible to llow the storge system to coordinte the entire upgrde process, perform the following steps: 1. Reduce the storge system worklod by 50%. 2. Use the configurtion tools for the storge system to mnully filover ll logicl units (LUs) from the controller tht you wnt to upgrde. 3. Upgrde the controller code. 4. Restrt the controller. 5. Mnully filbck the LUs to their originl controller. 6. Repet for ll controllers. FRU replcement Ensure tht you re fmilir with the following guidelines for replcing FRUs: v If the component tht you wnt to replce is directly in the host-side dt pth (for exmple, cble, Fibre Chnnel port, or controller), disble the externl dt pths to prepre for upgrde. To disble externl dt pths, disconnect or disble the pproprite ports on the fbric switch. The SAN Volume Controller ERPs reroute ccess over the lternte pth. v If the component tht you wnt to replce is in the internl dt pth (for exmple, cche, or drive) nd did not completely fil, ensure tht the dt is bcked up before you ttempt to replce the component. v If the component tht you wnt to replce is not in the dt pth, for exmple, uninterruptible power supply units, fns, or btteries, the component is generlly dul-redundnt nd cn be replced without dditionl steps. Chpter 7. Dignosing problems 241

260 242 SAN Volume Controller: Troubleshooting Guide

261 Chpter 8. Disster recovery Use these disster recovery solutions for Metro Mirror, Globl Mirror, nd Stretched Cluster, where ccess to storge is still possible fter the filure of site. Metro Mirror nd Globl Mirror Note: Inpproprite use of these procedures cn llow host systems to mke independent modifictions to both the primry nd secondry copies of dt. The user is responsible for ensuring tht no host systems re continuing to use the primry copy of the dt before you enble ccess to the secondry copy. In Metro Mirror or Globl Mirror configurtion system is configured t ech site. Reltionships re configured between the systems to mirror dt from storge t the primry site to storge t the secondry site. If n outge occurs t the secondry site the primry site continues opertion without ny intervention. If n outge occurs t the primry site, then it is necessry to enble ccess to storge t the secondry site. Use the stoprcreltionship-ccess or stoprcconsistgrp-ccess commnd to enble ccess to the storge t the secondry site. Stretched Cluster In stretched cluster (formerly split-site) configurtion, system is configured with hlf the nodes t ech site nd quorum device t third loction. If n outge occurs t either site, then the other nodes t the other site ccesses the quorum device nd continue opertion without ny intervention. If connectivity between the two sites is lost, then whichever nodes ccess the quorum device first continues opertion. For disster recovery purposes user might wnt to enble ccess to the storge t the site tht lost the rce to ccess the quorum device. Use the stsk overridequorum commnd to enble ccess to the storge t the secondry site. This feture is only vilble if the system ws configured by ssigning sites to nodes nd storge controllers, nd chnging the system topology to stretched. Importnt: If the user rn disster recovery on one site nd then powered up the remining, filed site (which contined the configurtion node t the time of the disster), then the cluster would ssert itself s designed. This procedure would strt second, identicl cluster in prllel, which cn cuse dt corruption. The user must follow these steps: Exmple 1. Remove the connectivity of the nodes from the site tht is experiencing the outge 2. Power up or recover those nodes 3. Run stsk levecluster-force or svctsk rmnode commnd for ll the nodes in the cluster 4. Bring the nodes into cndidte stte, nd then 5. Connect them to the site on which the site disster recovery feture ws run. Copyright IBM Corp. 2003,

262 Other configurtions To recover ccess to the storge in other configurtions, use Recover system procedure on pge SAN Volume Controller: Troubleshooting Guide

263 Chpter 9. Recovery procedures Recover system procedure This topic describes these recovery procedures: recover system nd bck up nd restore system configurtion. This topic lso contins informtion bout performing the node rescue. The recover system procedure recovers the entire system if the block cluster stte hs been lost from ll nodes. The procedure recretes the storge system by using sved configurtion dt. The recovery might not be ble to restore ll volume dt. This procedure is lso known s Tier 3 (T3) recovery. T3 filure stte CAUTION: If the cluster encounters T3 filure stte, where: v No nodes re ctive, nd v One or more nodes hve node errors requiring node rescue STOP nd contct IBM support. Inititing this T3 procedure while in this specific stte cn result in loss of the XML bckup. Attention: Run service ctions only when directed by the fix procedures. If used inppropritely, service ctions cn cuse loss of ccess to dt or even dt loss. Before ttempting to recover storge system, investigte the cuse of the filure nd ttempt to resolve those issues by using other fix procedures. Red nd understnd ll of the instructions before completing ny ction. Do not ttempt the recovery procedure unless the following conditions re met: v All hrdwre errors re fixed. v All nodes hve cndidte sttus. The system recovery procedure is one of severl tsks tht must be completed. The following list is n overview of the tsks nd the order in which they must be completed: 1. Prepring for system recovery. Review the informtion regrding when to run the recover system procedure b. Fix your hrdwre errors c. Remove the system informtion for node cnisters with error code 550 or error code 578 by using the service ssistnt. 2. Running the system recovery. After you prepred the system for recovery nd met ll the pre-conditions, run the system recovery. Note: Run the procedure on one system in fbric t time. Do not run the procedure on different nodes in the sme system. This restriction lso pplies to remote systems. 3. Completing ctions to get your environment opertionl v Recovering from offline volumes by using the CLI Copyright IBM Corp. 2003,

264 v Checking your system, for exmple, to ensure tht ll mpped volumes cn ccess the host. You cn run the recovery procedure by using the front pnel of nodes other thn the 2145-DH8 or the service ssistnt. When to run the recover system procedure Attempt recover procedure only fter complete nd thorough investigtion of the cuse of the system filure. Attempt to resolve those issues by using other service procedures. Attention: If you experience filures t ny time while running the recover system procedure, cll the IBM Support Center. Do not ttempt to do further recovery ctions, becuse these ctions might prevent IBM Support from restoring the system to n opertionl sttus. Certin conditions must be met before you run the recovery procedure. Use the following items to help you determine when to run the recovery procedure: 1. Check tht no node in the cluster is ctive nd tht the mngement IP is not ccessible from ny other node. If this is the cse, there is no need to recover the cluster. 2. Resolve ll hrdwre errors in nodes so tht only nodes 578 or 550 re present. If this is not the cse, go to Fix hrdwre errors. 3. Ensure ll bckend-storge tht is dministered by the cluster is present before you run the recover system procedure. 4. If ny nodes hve been replced, ensure tht the WWNN of the replcement node mtches tht of the replced node, nd tht no prior system dt remins on this node. Fix hrdwre errors Before running system recovery procedure, it is importnt to identify nd fix the root cuse of the hrdwre issues. Identifying nd fixing the root cuse cn help recover system, if these re the fults tht re cusing the system to fil. The following re common issues which cn be esily resolved: v The node hs been powered off or the power cords were unplugged. v v A 2145 UPS-1U might hve filed nd shut down one or more nodes becuse of the filure. In generl, this cuse might not hppen becuse of the redundncy provided by the second 2145 UPS-1U. Check the node sttus of every node tht is member of the system. Resolve ll errors. All nodes must be reporting either node error 578, or no cluster nme is shown on the Cluster: disply. These error codes indicte tht the system hs lost its configurtion dt. If ny nodes report nything other thn these error codes, do not perform recovery. You cn encounter situtions where non-configurtion nodes report other node errors, such s node error 550. The 550 error cn lso indicte tht node is not ble to join system. Note: If ny of the buttons on the front pnel hve been pressed fter these two error codes re reported, the report for the node returns to the 578 node error. The chnge in the report hppens fter pproximtely 60 seconds. Also, 246 SAN Volume Controller: Troubleshooting Guide

265 if the node ws rebooted or if hrdwre service ctions were tken, the node might show no cluster nme on the Cluster: disply. If ny nodes show Node Error: 550, record the dt from the second line of the disply. If the lst chrcter on the second line of the disply is >, use the right button to scroll the disply to the right. - In ddition to the Node Error: 550, the second line of the disply cn show list of node front pnel IDs (seven digits) tht re seprted by spces. The list cn lso show the WWPN/LUN ID (16 hexdeciml digits followed by forwrd slsh nd deciml number). - If the error dt contins ny front pnel IDs, ensure tht the node referred to by tht front pnel ID is showing Node Error 578:. If it is not reporting node error 578, ensure tht the two nodes cn communicte with ech other. Verify the SAN connectivity nd restrt one of the two nodes by pressing the front pnel power button twice. - If the error dt contins WWPN/LUN ID, verify the SAN connectivity between this node nd tht WWPN. Check the storge system to ensure tht the LUN referred to is online. After verifying, restrt the node by pressing the front pnel power button twice. Note: If fter resolving ll these scenrios, hlf or greter thn hlf of the nodes re reporting Node Error: 578, it is pproprite to run the recovery procedure. For ny nodes tht re reporting node error 550, ensure tht ll the missing hrdwre tht is identified by these errors is powered on nd connected without fults. If you hve not been ble to restrt the system, nd if ny node other thn the current node is reporting node error 550 or 578, you must remove system dt from those nodes. This ction cknowledges the dt loss nd puts the nodes into the required cndidte stte. Removing clustered system informtion for nodes with error code 550 or error code 578 using the front pnel The recovery procedure for clustered systems works only when ll nodes re in cndidte sttus. If ny nodes disply error code 550 or error code 578, you must remove their system dt. About this tsk To remove clustered system informtion from node with n error 550 or 578, follow this front pnel procedure : Procedure 1. Press nd relese the up or down button until the Actions menu option is displyed. 2. Press nd relese the select. 3. Press nd relese the up or down button until Remove Cluster? option is displyed. 4. Press nd relese the select. 5. The node displys Confirm Remove?. 6. Press nd relese the select. 7. The node displys Cluster:. Chpter 9. Recovery procedures 247

266 Results When ll nodes show Cluster: on the top line nd blnk on the second line, the nodes re in cndidte sttus. The 550 or 578 error is removed. You cn now run the recovery procedure. Removing system informtion for nodes with error code 550 or error code 578 using the service ssistnt The system recovery procedure works only when ll nodes re in cndidte sttus. If there re ny nodes tht disply error code 550 or error code 578, you must remove their dt. About this tsk Before performing this tsk, ensure tht you hve red the introductory informtion in the overll recover system procedure. To remove system informtion from node with n error 550 or 578, follow this procedure using the service ssistnt: Procedure 1. Point your browser to the service IP ddress of one of the nodes, for exmple, If you do not know the IP ddress or if it hs not been configured, configure the service ddress in one of the following wys: v On SAN Volume Controller models 2145-CG8, 2145-CF8, A4 nd G4 nodes, use the front pnel menu to configure service ddress on the node. v On SAN Volume Controller 2145-DH8 nodes, use the technicin port to connect to the service ssistnt nd configure service ddress on the node. 2. Log on to the service ssistnt. 3. Select Mnge System. 4. Click Remove System Dt. 5. Confirm tht you wnt to remove the system dt when prompted. 6. Remove the system dt for the other nodes tht disply 550 or 578 error. All nodes previously in this system must hve node sttus of Cndidte nd hve no errors listed ginst them. 7. Resolve ny hrdwre errors until the error condition for ll nodes in the system is None. 8. Ensure tht ll nodes in the system disply sttus of cndidte. Results When ll nodes disply sttus of cndidte nd ll error conditions re None, you cn run the recovery procedure. Performing recovery procedure for clustered systems using the front pnel Strt recovery when ll nodes tht were members of the system re online nd re in cndidte sttus. If there re ny nodes tht disply error code 550 or error code 248 SAN Volume Controller: Troubleshooting Guide

267 578, remove their system dt to plce them into cndidte sttus. Do not run the recovery procedure on different nodes in the sme system; this restriction includes remote clustered systems. About this tsk Attention: This service ction hs serious implictions if not performed properly. If t ny time n error is encountered not covered by this procedure, stop nd cll IBM Support. Any one of the following ctegories of messges my be displyed: v T3 successful v v The volumes re online. Use the finl checks to mke the environment opertionl; see Wht to check fter running the system recovery on pge 252. T3 incomplete One or more of the volumes is offline becuse there ws fst write dt in the cche. Further ctions re required to bring the volumes online; see Recovering from offline volumes using the CLI on pge 252 for detils (specificlly, see the tsk concerning recovery from offline VDisks using the commnd-line interfce (CLI)). T3 filed Cll IBM Support. Do not ttempt ny further ction. Strt the recovery procedure from ny node in the system; the node must not hve prticipted in ny other system. To receive optiml results in mintining the I/O group ordering, run the recovery from node tht ws in I/O group 0. Note: Ech individul stge of the recovery procedure might tke significnt time to complete, dependnt upon the specific configurtion. Procedure 1. Click the up or down button until the Actions menu option is displyed; then click Select. 2. Click the up or down button until the Recover Cluster? option is displyed, nd then click Select; the node displys Confirm Recover?. 3. Click Select; the node displys Retrieving. After short dely, the second line displys sequence of progress messges indicting the ctions re tking plce; for exmple, Finding qdisks. The bckup files re scnned to find the most recent configurtion bckup dt. After the file nd quorum dt retrievl is complete, the node displys T3 dt: on the top line. 4. Verify the dte nd time on the second line of the disply. The time stmp shown is the dte nd time of the lst quorum updte nd must be less thn 30 minutes before the filure. The time stmp formt is YYYYMMDD hh:mm, where YYYY is the yer, MM is the month, DD is the dy, hh is the hour, nd mm is the minute. Attention: If the time stmp is not less thn 30 minutes before the filure, cll IBM support. 5. After verifying the time stmp is correct, press nd hold the UP ARROW nd click Select. The node displys Bckup file on the top line. Chpter 9. Recovery procedures 249

268 6. Verify the dte nd time on the second line of the disply. The time stmp shown is the dte nd time of the lst configurtion bckup nd must be less thn 24 hours before the filure. The time stmp formt is YYYYMMDD hh:mm, where YYYY is the yer, MM is the month, DD is the dy, hh is the hour, nd mm is the minute. Attention: If the time stmp is not less thn 24 hours before the filure, cll IBM support. Note: Chnges mde fter the time of this configurtion bckup might not be restored. 7. After verifying the time stmp is correct, press nd hold the UP ARROW nd click Select. The node displys Restoring. After short dely, the second line displys sequence of progress messges indicting the ctions tking plce; then the softwre on the node restrts. The node displys Cluster on the top line nd mngement IP ddress on the second line. After few moments, the node displys T3 Completing. Note: Any system errors logged t this time might temporrily overwrite the disply; ignore the messge: Cluster Error: After short dely, the second line displys sequence of progress messges indicting the ctions tking plce. When ech node is dded to the system, the disply shows Cluster: on the top line, nd the cluster (system) nme on the second line. Attention: After the lst node is dded to the system, there is short dely to llow the system to stbilize. Do not ttempt to use the system. The recovery is still in progress. Once recovery is complete, the node displys T3 Succeeded on the top line. 8. Click Select to return the node to norml disply. Results Recovery is complete when the node displys T3 Succeeded. Verify the environment is opertionl by performing the checks provided in Wht to check fter running the system recovery on pge 252. Running system recovery using the service ssistnt Strt recovery when ll nodes tht were members of the system re online nd re in cndidte sttus. If ny nodes disply error code 550 or 578, remove system informtion to plce them into cndidte sttus. Do not run the recovery procedure on different nodes in the sme system; this restriction includes remote systems. Before you begin Note: Ensure tht the web browser is not blocking pop-up windows. If it does, progress windows cnnot open. Before beginning this procedure, red the recover system procedure introductory informtion; see Recover system procedure on pge SAN Volume Controller: Troubleshooting Guide

269 About this tsk Attention: This service ction hs serious implictions if not completed properly. If t ny time n error is encountered not covered by this procedure, stop nd cll IBM Support. Run the recovery from ny nodes in the system; the nodes must not hve prticipted in ny other system. Note: Ech individul stge of the recovery procedure cn tke significnt time to complete, depending on the specific configurtion. Procedure 1. Point your browser to the service IP ddress of one of the nodes. If you do not know the IP ddress or if it hs not been configured, configure the service ddress in one of the following wys: v On SAN Volume Controller models 2145-CG8, 2145-CF8, A4 nd G4 nodes, use the front pnel menu to configure service ddress on the node. v On SAN Volume Controller 2145-DH8 nodes, use the technicin port to connect to the service ssistnt nd configure service ddress on the node. 2. Log on to the service ssistnt. 3. Select Recover System from the nvigtion. 4. Follow the online instructions to complete the recovery procedure.. Verify the dte nd time of the lst quorum time. The time stmp must be less thn 30 minutes before the filure. The time stmp formt is YYYYMMDD hh:mm, where YYYY is the yer, MM is the month, DD is the dy, hh is the hour, nd mm is the minute. Attention: If the time stmp is not less thn 30 minutes before the filure, cll IBM Support. b. Verify the dte nd time of the lst bckup dte. The time stmp must be less thn 24 hours before the filure. The time stmp formt is YYYYMMDD hh:mm, where YYYY is the yer, MM is the month, DD is the dy, hh is the hour, nd mm is the minute. Attention: If the time stmp is not less thn 24 hours before the filure, cll IBM Support. Chnges mde fter the time of this bckup dte might not be restored. Results Any one of the following ctegories of messges my be displyed: v T3 successful v v The volumes re bck online. Use the finl checks to get your environment opertionl gin. T3 recovery completed with errors T3 recovery completed with errors: One or more of the volumes re offline becuse there ws fst write dt in the cche. To bring the volumes online, see Recovering from offline volumes using the CLI on pge 252 for detils. T3 filed Cll IBM Support. Do not ttempt ny further ction. Chpter 9. Recovery procedures 251

270 Verify the environment is opertionl by completing the checks provided in Wht to check fter running the system recovery. If ny errors re logged in the error log fter the system recovery procedure completes, use the fix procedures to resolve these errors, especilly the errors relted to offline rrys. If the recovery completes with offline volumes, go to Recovering from offline volumes using the CLI. Recovering from offline volumes using the CLI If Tier 3 recovery procedure completes with offline volumes, then it is likely tht the dt which ws in the write-cche of the node cnisters ws lost during the filure tht cused ll of the node cnisters to lose the block storge system cluster stte. You cn use the commnd-line interfce (CLI) to cknowledge tht there ws lost dt lost from the write-cche, nd bring the volume bck online to ttempt to del with the dt loss. About this tsk If you hve run the recovery procedure but there re offline volumes, you cn complete the following steps to bring the volumes bck online. Any volumes tht re offline nd re not thin-provisioned (or compressed) volumes re offline becuse of the loss of write-cche dt during the event tht led ll node cnisters to lose their cluster stte. Any dt lost from the write-cche cnnot be recovered. These volumes might need dditionl recovery steps fter the volume is brought bck online. Note: If you encounter errors in the error log fter running the recovery procedure tht re relted to offline rrys, use the fix procedures to resolve the offline rry errors before fixing the offline volume errors. Exmple Complete the following steps to recover n offline volume fter the recovery procedure hs completed: 1. Delete ll IBM FlshCopy function mppings nd Metro Mirror or Globl Mirror reltionships tht use the offline volumes. 2. Run the recovervdisk, recovervdiskbyiogrp or recovervdiskbysystem commnd. (This will only bring the volume bck online so tht you cn ttempt to del with the dt loss.) 3. Refer to Wht to check fter running the system recovery for wht to do with volumes tht hve been corrupted by the loss of dt from the write-cche. 4. Recrete ll FlshCopy mppings nd Metro Mirror or Globl Mirror reltionships tht use the volumes. Wht to check fter running the system recovery Severl tsks must be completed before you use the system. The recovery procedure recretes the old system from the quorum dt. However, some things cnnot be restored, such s cched dt or system dt mnging in-flight I/O. This ltter loss of stte ffects RAID rrys mnging internl storge. The detiled mp bout where dt is out of synchroniztion hs been 252 SAN Volume Controller: Troubleshooting Guide

271 lost, mening tht ll prity informtion must be restored, nd mirrored pirs must be brought bck into synchroniztion. Normlly this results in either old or stle dt being used, so only writes in flight re ffected. However, if the rry hd lost redundncy (such s syncing, or degrded or criticl RAID sttus) prior to the error requiring system recovery, then the sitution is more severe. Under this sitution you need to check the internl storge: v v v v Prity rrys will likely be syncing to restore prity; they do not hve redundncy when this opertion proceeds. Becuse there is no redundncy in this process, bd blocks might hve been creted where dt is not ccessible. Prity rrys could be mrked s corrupt. This indictes tht the extent of lost dt is wider thn in-flight IO, nd in order to bring the rry online, the dt loss must be cknowledged. Rid-6 rrys tht were ctully degrded prior the system recovery might require full restore from bckup. For this reson, it is importnt to hve t lest cpcity mtch spre vilble. Be wre of these differences regrding the recovered configurtion: v FlshCopy mppings re restored s idle_or_copied with 0% progress. Both volumes must hve been restored to their originl I/O groups. v The mngement ID is different. Any scripts or ssocited progrms tht refer to the system-mngement ID of the clustered system (system) must be chnged. v Any FlshCopy mppings tht were not in the idle_or_copied stte with 100% progress t the point of disster hve inconsistent dt on their trget disks. These mppings must be restrted. v Intersystem remote copy prtnerships nd reltionships re not restored nd must be re-creted mnully. v Consistency groups re not restored nd must be re-creted mnully. v Intrsystem remote copy reltionships re restored if ll dependencies were successfully restored to their originl I/O groups. v The system time zone might not hve been restored. v Any Globl Mirror secondry volumes on the recovered system might hve inconsistent dt if there ws repliction I/O from the primry volume cched on the secondry system t the point of the disster. A full synchroniztion is required when recreting nd restrting these remote copy reltionships. Before using the volumes, complete the following tsks: v Strt the host systems. v Mnul ctions might be necessry on the hosts to trigger them to rescn for devices. You cn complete this tsk by disconnecting nd reconnecting the Fibre Chnnel cbles to ech host bus dpter (HBA) port. v Verify tht ll mpped volumes cn be ccessed by the hosts. v Run file system consistency checks. v Note: Any dt tht ws in the SAN Volume Controller write cche t the time of the filure is lost. Run the ppliction consistency checks. Bcking up nd restoring the system configurtion You cn bck up nd restore the configurtion dt for the system fter preliminry tsks re completed. Chpter 9. Recovery procedures 253

272 Configurtion dt for the system provides informtion bout your system nd the objects tht re defined in it. The bckup nd restore functions of the svcconfig commnd cn bck up nd restore only your configurtion dt for the SAN Volume Controller system. You must regulrly bck up your ppliction dt by using the pproprite bckup methods. You cn mintin your configurtion dt for the system by completing the following tsks: v Bcking up the configurtion dt v Restoring the configurtion dt v Deleting unwnted bckup configurtion dt files Before you bck up your configurtion dt, the following prerequisites must be met: v v No independent opertions tht chnge the configurtion for the system cn be running while the bckup commnd is running. No object nme cn begin with n underscore chrcter (_). Note: v The defult object nmes for controllers, I/O groups, nd mnged disks (MDisks) do not restore correctly if the ID of the object is different from wht is recorded in the current configurtion dt file. v All other objects with defult nmes re renmed during the restore process. The new nmes pper in the formt nme_r where nme is the nme of the object in your system. Before you restore your configurtion dt, the following prerequisites must be met: v v v v v You hve the Security Administrtor role ssocited with your user nme nd pssword. You hve copy of your bckup configurtion files on server tht is ccessible to the system. You hve bckup copy of your ppliction dt tht is redy to lod on your system fter the restore configurtion opertion is complete. You know the current license settings for your system. You did not remove ny hrdwre since the lst bckup of your system configurtion. If you hd to replce fulty node, the new node must use the sme worldwide node nme (WWNN) s the fulty node tht it replced. v Note: You cn dd new hrdwre, but you must not remove ny hrdwre becuse the removl cn cuse the restore process to fil. No zoning chnges were mde on the Fibre Chnnel fbric which would prevent communiction between the SAN Volume Controller nd ny storge controllers which re present in the configurtion. Use the following steps to determine how to chieve n idel T4 recovery: v Open the pproprite svc.config.bckup.xml (or svc.config.cron.xml) file with suitble text editor or browser nd nvigte to the node section of the file. v For ech node entry, mke note of the vlue of following properties; IO_group_id, cnister_id, enclosure_seril_number. v Use the CLI sinfo lsservicenodes commnd nd the dt to determine which nodes previously belonged in ech I/O group. 254 SAN Volume Controller: Troubleshooting Guide

273 Restoring the system configurtion should be performed vi one of the nodes previously in IO group zero. For exmple, property nme="io_group_id" vlue="0". The remining enclosures should be dded, s required, in the pproprite order bsed on the previous IO_group_id of its nodes. Note: It is not currently possible to determine which cnister within the identified enclosure ws previously used for cluster cretion. Typiclly the restortion should be performed vi cnister 1. The SAN Volume Controller nlyzes the bckup configurtion dt file nd the system to verify tht the required disk controller system nodes re vilble. Before you begin, hrdwre recovery must be complete. The following hrdwre must be opertionl: hosts, SAN Volume Controller, drives, the Ethernet network, nd the SAN fbric. Bcking up the system configurtion using the CLI You cn bck up your configurtion dt using the commnd-line interfce (CLI). Before you begin Before you bck up your configurtion dt, the following prerequisites must be met: v v No independent opertions tht chnge the configurtion cn be running while the bckup commnd is running. No object nme cn begin with n underscore chrcter (_). About this tsk The bckup feture of the svcconfig CLI commnd is designed to bck up informtion bout your system configurtion, such s volumes, locl Metro Mirror informtion, locl Globl Mirror informtion, storge pools, nd nodes. All other dt tht you wrote to the volumes is not bcked up. Any ppliction tht uses the volumes on the system s storge, must use the pproprite bckup methods to bck up its ppliction dt. You must regulrly bck up your configurtion dt nd your ppliction dt to void dt loss, such s fter ny significnt chnges to the system configurtion. Note: The system utomticlly cretes bckup of the configurtion dt ech dy t 1 AM. This bckup is known s cron bckup nd is written to /dumps/svc.config.cron.xml_seril# on the configurtion node. Use the these instructions to generte mnul bckup t ny time. If severe filure occurs, both the configurtion of the system nd ppliction dt might be lost. The bckup of the configurtion dt cn be used to restore the system configurtion to the exct stte it ws in before the filure. In some cses, it might be possible to utomticlly recover the ppliction dt. This bckup cn be ttempted with the Recover System Procedure, lso known s Tier 3 (T3) procedure. To restore the system configurtion without ttempting to recover the ppliction dt, use the Restoring the System Configurtion procedure, lso known s Tier 4 (T4) recovery. Both of these procedures require recent bckup of the configurtion dt. Complete the following steps to bck up your configurtion dt: Chpter 9. Recovery procedures 255

274 Procedure 1. Use your preferred bckup method to bck up ll of the ppliction dt tht you stored on your volumes. 2. Issue the following CLI commnd to bck up your configurtion: svcconfig bckup The following output is n exmple of the messges tht might be displyed during the bckup process: CMMVC6112W io_grp io_grp1 hs defult nme CMMVC6112W io_grp io_grp2 hs defult nme CMMVC6112W mdisk mdisk14... CMMVC6112W node node1... CMMVC6112W node node The svcconfig bckup CLI commnd cretes three files tht provide informtion bout the bckup process nd the configurtion. These files re creted in the /dumps directory of the configurtion node cnister. Tble 91 describes the three files tht re creted by the bckup process: Tble 91. Files creted by the bckup process File nme Description svc.config.bckup.xml_<seril#> Contins your configurtion dt. svc.config.bckup.sh_<seril#> Contins the nmes of the commnds tht were issued to crete the bckup of the system. svc.config.bckup.log_<seril#> Contins detils bout the bckup, including ny reported errors or wrnings. 3. Check tht the svcconfig bckup commnd completes successfully, nd exmine the commnd output for ny wrnings or errors. The following output is n exmple of the messge tht is displyed when the bckup process is successful: CMMVC6155I SVCCONFIG processing completed successfully. If the process fils, resolve the errors, nd run the commnd gin. 4. Keep bckup copies of the files outside the system to protect them ginst system hrdwre filure. Copy the bckup files off the system to secure loction; use either the mngement GUI or SmrtCloud Provisioning commnd line. For exmple: pscp -unsfe superuser@cluster_ip:/dumps/svc.config.bckup.* /offclusterstorge/ The cluster_ip is the IP ddress or DNS nme of the system nd offclusterstorge is the loction where you wnt to store the bckup files. Tip: To mintin controlled ccess to your configurtion dt, copy the bckup files to loction tht is pssword-protected. Restoring the system configurtion For directions on the recover procedure, see Recover system procedure on pge SAN Volume Controller: Troubleshooting Guide

275 Before you begin This configurtion restore procedure is designed to restore informtion bout your configurtion, such s volumes, locl Metro Mirror informtion, locl Globl Mirror informtion, storge pools, nd nodes. All the dt tht you wrote to the volumes is not restored. To restore the dt on the volumes, you must restore ppliction dt from ny ppliction tht uses the volumes on the clustered system s storge seprtely. Therefore, you must hve bckup of this dt before you follow the configurtion recovery process. About this tsk You must regulrly bck up your configurtion dt nd your ppliction dt to void dt loss. If system is lost fter severe filure occurs, both configurtion for the system nd ppliction dt is lost. You must reinstte the system to the exct stte it ws in before the filure, nd then recover the ppliction dt. Importnt: v There re two phses during the restore process: prepre nd execute. You must not chnge the fbric or system between these two phses. v For SAN Volume Controller with 10 GE Interfce crd or second Fibre Chnnel Interfce crd tht is instlled, the system loclfcportmsk nd prtnerfcportmsk settings re mnully repplied before you restore your dt. See step 9 on pge 258. v For SAN Volume Controller with internl flsh drives (including nodes tht re connected to expnsion enclosures), ll nodes must be dded into the system before you restore your dt. See step 10 on pge 258. If you do not understnd the instructions to run the CLI commnds, see the commnd-line interfce reference informtion. To restore your configurtion dt, follow these steps: Procedure 1. Verify tht ll nodes re vilble s cndidte nodes before you run this recovery procedure. You must remove errors 550 or 578 to put the node in cndidte stte. 2. Crete system. If possible, use the node tht ws originlly in I/O group 0. v For SAN Volume Controller 2145-DH8 systems, use the technicin port. v For ll other erlier models, use the front pnel. 3. In supported browser, enter the IP ddress tht you used to initilize the system nd the defult superuser pssword (pssw0rd). 4. Issue the following CLI commnd to ensure tht only the configurtion node is online: lsnode The following output is n exmple of wht is displyed: id nme sttus IO_group_id IO_group_nme config_node 1 nodel online 0 io_grp0 yes 5. Using the commnd-line interfce, issue the following commnd to log on to the system: plink -i ssh_privte_key_file superuser@cluster_ip Chpter 9. Recovery procedures 257

276 Where ssh_privte_key_file is the nme of the SSH privte key file for the superuser nd cluster_ip is the IP ddress or DNS nme of the system for which you wnt to restore the configurtion. Note: Becuse the RSA host key chnged, wrning messge might disply when you connect to the system using SSH. 6. If the clustered system ws previously configured s repliction lyer, then use the chsystem commnd to chnge the lyer setting. 7. Identify the configurtion bckup file from which you wnt to restore. The file cn be either locl copy of the configurtion bckup XML file tht you sved when you bcked-up the configurtion or n up-to-dte file on one of the nodes. Configurtion dt is utomticlly bcked up dily t 01:00 system time on the configurtion node. Downlod nd check the configurtion bckup files on ll nodes tht were previously in the system to identify the one contining the most recent complete bckup. From the mngement GUI, click Settings > Support. b. Click Show full log listing. c. For ech node (cnister) in the system, complete the following steps: 1) Select the node to operte on from the selection box t the top of the tble. 2) Find ll the files with nmes tht mtch the pttern svc.config.*.xml*. 3) Double-click the files to downlod them to your computer. d. If recent configurtion file is not present on this node, configure service IP ddresses for other nodes nd connect to the service ssistnt to look for configurtion files on other nodes. For more informtion, see the service IPv4 or service IPv6 t Service IPv4 or Service IPv6 options on pge 114. The XML files contin dte nd time tht cn be used to identify the most recent bckup. After you identify the bckup XML file tht is to be used when you restore the system, renme the file to svc.config.bckup.xml. 8. Copy onto the system the XML bckup file from which you wnt to restore. pscp full_pth_to_identified_svc.config.file superuser@cluster_ip:/tmp/svc.config.bckup.xml 9. If the system contins ny nodes with 10 GB interfce crd or second Fibre Chnnel Interfce crd tht is instlled nd non-defult loclfcportmsk nd prtnerfcportmsk settings were previously configured, then mnully reconfigure these settings before restoring your dt. 10. If the system contins ny nodes with internl flsh drives (including nodes tht re connected to expnsion enclosures), these nodes must be dded to the system now. To dd these nodes, determine the pnel nme, node nme, nd I/O groups of ny such nodes from the configurtion bckup file. To dd the nodes to the system, issue this commnd: svctsk ddnode -pnelnme pnel_nme -iogrp iogrp_nme_or_id -nme node_nme Where pnel_nme is the nme tht is displyed on the pnel, iogrp_nme_or_id is the nme or ID of the I/O group to which you wnt to dd this node, nd node_nme is the nme of the node. 258 SAN Volume Controller: Troubleshooting Guide

277 11. Issue the following CLI commnd to compre the current configurtion with the bckup configurtion dt file: svcconfig restore -prepre This CLI commnd cretes log file in the /tmp directory of the configurtion node. The nme of the log file is svc.config.restore.prepre.log. Note: It cn tke up to minute for ech 256-MDisk btch to be discovered. If you receive error messge CMMVC6200W for n MDisk fter you enter this commnd, ll the mnged disks (MDisks) might not be discovered yet. Allow suitble time to elpse nd try the svcconfig restore -prepre commnd gin. 12. Issue the following commnd to copy the log file to nother server tht is ccessible to the system: pscp superuser@cluster_ip:/tmp/svc.config.restore.prepre.log full_pth_for_where_to_copy_log_files 13. Open the log file from the server where the copy is now stored. 14. Check the log file for errors. v If you find errors, correct the condition tht cused the errors nd reissue the commnd. You must correct ll errors before you cn proceed to step 15. v If you need ssistnce, contct the IBM Support Center. 15. Issue the following CLI commnd to restore the configurtion: svcconfig restore -execute Note: Issuing this CLI commnd on single node system dds the other nodes to the system. This CLI commnd cretes log file in the /tmp directory of the configurtion node. The nme of the log file is svc.config.restore.execute.log. 16. Issue the following commnd to copy the log file to nother server tht is ccessible to the system: pscp superuser@cluster_ip:/tmp/svc.config.restore.execute.log full_pth_for_where_to_copy_log_files 17. Open the log file from the server where the copy is now stored. 18. Check the log file to ensure tht no errors or wrnings occurred. Note: You might receive wrning tht sttes tht licensed feture is not enbled. This messge mens tht fter the recovery process, the current license settings do not mtch the previous license settings. The recovery process continues normlly nd you cn enter the correct license settings in the mngement GUI lter. When you log in to the CLI gin over SSH, you see this output: IBM_2145:your_cluster_nme:superuser> Wht to do next You cn remove ny unwnted configurtion bckup nd restore files from the /tmp directory on your configurtion by issuing the following CLI commnd: svcconfig cler -ll Chpter 9. Recovery procedures 259

278 Deleting bckup configurtion files using the CLI You cn use the commnd-line interfce (CLI) to delete bckup configurtion files. About this tsk Complete the following steps to delete bckup configurtion files: Procedure 1. Issue the following commnd to log on to the system: plink -i ssh_privte_key_file where ssh_privte_key_file is the nme of the SSH privte key file for the superuser nd cluster_ip is the IP ddress or DNS nme of the clustered system from which you wnt to delete the configurtion. 2. Issue the following CLI commnd to erse ll of the files tht re stored in the /tmp directory: svcconfig cler -ll Performing the node rescue when the node boots If it is necessry to replce the hrd disk drive or if the softwre on the hrd disk drive is corrupted, you cn use the node rescue procedure to reinstll the SAN Volume Controller softwre. Before you begin Similrly, if you hve replced the service controller, use the node rescue procedure to ensure tht the service controller hs the correct softwre. About this tsk Attention: If you recently replced both the service controller nd the disk drive s prt of the sme repir opertion, node rescue fils. Node rescue works by booting the operting system from the service controller nd running progrm tht copies ll the SAN Volume Controller softwre from ny other node tht cn be found on the Fibre Chnnel fbric. Attention: When running node rescue opertions, run only one node rescue opertion on the sme SAN, t ny one time. Wit for one node rescue opertion to complete before strting nother. Perform the following steps to complete the node rescue: Procedure 1. Ensure tht the Fibre Chnnel cbles re connected. 2. Ensure tht t lest one other node is connected to the Fibre Chnnel fbric. 3. Ensure tht the SAN zoning llows connection between t lest one port of this node nd one port of nother node. It is better if multiple ports cn connect. This is prticulrly importnt if the zoning is by worldwide port nme (WWPN) nd you re using new service controller. In this cse, you might need to use SAN monitoring tools to determine the WWPNs of the node. If you need to chnge the zoning, remember to set it bck when the service procedure is complete. 260 SAN Volume Controller: Troubleshooting Guide

279 4. Turn off the node. 5. Press nd hold the left nd right buttons on the front pnel. 6. Press the power button. 7. Continue to hold the left nd right buttons until the node-rescue-request symbol is displyed on the front pnel (Figure 64). Results Figure 64. Node rescue disply The node rescue request symbol displys on the front pnel disply until the node strts to boot from the service controller. If the node rescue request symbol displys for more thn two minutes, go to the hrdwre boot MAP to resolve the problem. When the node rescue strts, the service disply shows the progress or filure of the node rescue opertion. Note: If the recovered node ws prt of clustered system, the node is now offline. Delete the offline node from the system nd then dd the node bck into the system. If node recovery ws used to recover node tht filed during softwre upgrde process, it is not possible to dd the node bck into the system until the upgrde or downgrde process hs completed. This cn tke up to four hours for n eight-node clustered system. Chpter 9. Recovery procedures 261

280 262 SAN Volume Controller: Troubleshooting Guide

281 Chpter 10. Understnding the medium errors nd bd blocks A storge system returns medium error response to host when it is unble to successfully red block. The SAN Volume Controller response to host red follows this behvior. The volume virtuliztion tht is provided extends the time when medium error is returned to host. Becuse of this difference to non-virtulized systems, the SAN Volume Controller uses the term bd blocks rther thn medium errors. The SAN Volume Controller lloctes volumes from the extents tht re on the mnged disks (MDisks). The MDisk cn be volume on n externl storge controller or RAID rry tht is creted from internl drives. In either cse, depending on the RAID level used, there is normlly protection ginst red error on single drive. However, it is still possible to get medium error on red request if multiple drives hve errors or if the drives re rebuilding or re offline due to other issues. The SAN Volume Controller provides migrtion fcilities to move volume from one underlying set of physicl storge to nother or to replicte volume tht uses FlshCopy or Metro Mirror or Globl Mirror. In ll these cses, the migrted volume or the replicted volume returns medium error to the host when the logicl block ddress on the originl volume is red. The system mintins tbles of bd blocks to record where the logicl block ddresses tht cnnot be red re. These tbles re ssocited with the MDisks tht re providing storge for the volumes. The dumpmdiskbdblocks commnd nd the dumpllmdiskbdblocks commnd re vilble to query the loction of bd blocks. Importnt: The dumpmdiskbdblocks only outputs the virtul medium errors tht hve been creted, nd not list of the ctul medium errors on MDisks or drives. It is possible tht the tbles tht re used to record bd block loctions cn fill up. The tble cn fill either on n MDisk or on the system s whole. If tble does fill up, the migrtion or repliction tht ws creting the bd block fils becuse it ws not possible to crete n exct imge of the source volume. The system cretes lerts in the event log for the following situtions: v When it detects medium errors nd cretes bd block v When the bd block tbles fill up Tble 92 lists the bd block error codes. Tble 92. Bd block errors Error code Description 1840 The mnged disk hs bd blocks. On n externl controller, this cn only be copied medium error The system hs filed to crete bd block becuse the MDisk lredy hs the mximum number of llowed bd blocks. Copyright IBM Corp. 2003,

282 Tble 92. Bd block errors (continued) Error code Description 1225 The system hs filed to crete bd block becuse the system lredy hs the mximum number of llowed bd blocks. The recommended ctions for these lerts guide you in correcting the sitution. Cler bd blocks by dellocting the volume disk extent, by deleting the volume or by issuing write I/O to the block. It is good prctice to correct bd blocks s soon s they re detected. This ction prevents the bd block from being propgted when the volume is replicted or migrted. It is possible, however, for the bd block to be on prt of the volume tht is not used by the ppliction. For exmple, it cn be in prt of dtbse tht hs not been initilized. These bd blocks re corrected when the ppliction writes dt to these res. Before the correction hppens, the bd block records continue to use up the vilble bd block spce. 264 SAN Volume Controller: Troubleshooting Guide

283 Chpter 11. Using the mintennce nlysis procedures MAP 5000: Strt The mintennce nlysis procedures (MAPs) inform you how to nlyze filure tht occurs with SAN Volume Controller node. About this tsk SAN Volume Controller nodes must be configured in pirs so you cn perform concurrent mintennce. When you service one node, the other node keeps the storge re network (SAN) opertionl. With concurrent mintennce, you cn remove, replce, nd test ll field replceble units (FRUs) on one node while the SAN nd host systems re powered on nd doing productive work. Note: Unless you hve prticulr reson, do not remove the power from both nodes unless instructed to do so. When you need to remove power, see MAP 5350: Powering off SAN Volume Controller node on pge 298. Procedure v To isolte the FRUs in the filing node, complete the ctions nd nswer the questions given in these mintennce nlysis procedures (MAPs). v When instructed to exchnge two or more FRUs in sequence: 1. Exchnge the first FRU in the list for new one. 2. Verify tht the problem is solved. 3. If the problem remins:. Reinstll the originl FRU. b. Exchnge the next FRU in the list for new one. 4. Repet steps 2 nd 3 until either the problem is solved, or ll the relted FRUs hve been exchnged. 5. Complete the next ction indicted by the MAP. 6. If you re using one or more MAPs becuse of system error code, mrk the error s fixed in the event log fter the repir, but before you verify the repir. Note: Strt ll problem determintion procedures nd repir procedures with MAP 5000: Strt. MAP 5000: Strt is n entry point to the mintennce nlysis procedures (MAPs) for the SAN Volume Controller. Before you begin Note: The service ssistnt interfce should be used if there is no front pnel disply, for exmple on the SAN Volume Controller 2145-DH8. If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures. Copyright IBM Corp. 2003,

284 This MAP pplies to ll SAN Volume Controller models with front pnel disply. However the front pnel will only disply the sttus of the first four Fibre Chnnel ports; the service ssistnt GUI should be used if the node hs more thn four Fibre Chnnel ports. Be sure tht you know which model you re using before you strt this procedure. To determine which model you re working with, look for the lbel tht identifies the model type on the front of the node. You might be sent here for one of the following resons: v The fix procedures sent you here v A problem occurred during the instlltion of SAN Volume Controller v Another MAP sent you here v A user observed problem tht ws not detected by the system SAN Volume Controller nodes re configured in pirs. While you service one node, you cn ccess ll the storge mnged by the pir from the other node. With concurrent mintennce, you cn remove, replce, nd test ll FRUs on one SAN Volume Controller while the SAN nd host systems re powered on nd doing productive work. Notes: v Unless you hve prticulr reson, do not remove the power from both nodes unless instructed to do so. v If n ction in these procedures involves removing or replcing prt, use the pplicble procedure. v If the problem persists fter you complete the ctions in this procedure, return to step 1 of the MAP to try gin to fix the problem. Procedure 1. Were you sent here from fix procedure? NO Go to step 2 YES Go to step 6 on pge (from step 1) Log in to the mngement GUI. Use the user ID nd pssword tht is provided by the user nd strt the mngement GUI for the system tht you re repiring. See Accessing the mngement GUI on pge (from step 2) Does the mngement GUI strt? NO Go to step 6 on pge 267. YES Go to step (from step 3) When the SAN Volume Controller system tht you wnt to service is selected, is the Welcome window displyed? NO Go to step 6 on pge 267. YES Go to step (from step 4) Strt the fix procedures. Did the fix procedures find n error tht is to be fixed? NO Go to step 6 on pge SAN Volume Controller: Troubleshooting Guide

285 YES Follow the fix procedures. 6. (from steps 1 on pge 266, 3 on pge 266, 4 on pge 266, nd 5 on pge 266) Is the power indictor on the front pnel off? Check to see whether the power LED on the opertor-informtion pnel is off. NO Go to step 7. YES Try to turn on the nodes. See Using the power control for the SAN Volume Controller node on pge 119. Note: The uninterruptible power supply unit tht supplies power to the node might lso be turned off. The uninterruptible power supply must be turned on before the node is turned on. SAN Volume Controller 2145-DH8 does not hve n externl uninterruptible power supply unit. This system hs bttery modules in its front pnel insted. If the nodes re turned on, go to step 7; otherwise, go to the pproprite Power MAP: MAP 5060: Power A4 on pge 285 or MAP 5050: Power 2145-CG8, 2145-CF8, nd G4 on pge 279 or MAP 5040: Power SAN Volume Controller 2145-DH8 on pge (from step 6) Does the front pnel of the node show hrdwre error? There is hrdwre error if ny of the following conditions re true for the node: v None of the LEDs on the front pnel re on, nd the front-pnel disply, if there is one, is blnk. v The error LED is on, 8 on Figure 65 on pge 268. Chpter 11. Using the mintennce nlysis procedures 267

286 SAN Volume Controller svc Hrd disk drive ctivity LED 2 Hrd disk drive sttus LED 3 USB port 4 Video connector 5 Opertor-informtion pnel 6 Rck relese ltch 7 Node sttus LED 8 Node fult LED 9 Bttery sttus LED 10 Bttery fult LED 11 Btteries 12 Hrd disk drives (boot drives) Figure 65. SAN Volume Controller 2145-DH8 front pnel NO Go to step 8. YES The service controller for the SAN Volume Controller filed. (The SAN Volume Controller 2145-DH8 does not hve service controller.). Check tht the service controller tht is indicting n error is correctly instlled. If it is, replce the service controller. b. Go to MAP 5700: Repir verifiction on pge (from step 7 on pge 267) Is the opertor-informtion pnel error LED ( 1 in Figure 66 on pge 269) ( 7 in Figure 67 on pge 269) illuminted or flshing? 268 SAN Volume Controller: Troubleshooting Guide

287 CF A G CG8 svc00714 Figure 66. Error LED on the SAN Volume Controller models Figure 67 shows the opertor-informtion pnel for the SAN Volume Controller 2145-DH svc Power-control button nd power-on LED 2 Ethernet icon 3 System-loctor button nd LED 4 Relese ltch for the light pth dignostics pnel 5 Ethernet ctivity LEDs 6 Check log LED 7 System-error LED Note: If the node hs more thn 4 Ethernet ports, ctivity for ports 5 nd bove is not indicted by the Ethernet ctivity LEDs on the opertor-informtion pnel. Figure 67. SAN Volume Controller 2145-DH8 opertor-informtion pnel NO Go to step 9. YES Go to MAP 5800: Light pth on pge 319. Is the opertor-informtion pnel error LED illuminted or flshing? NO Go to step 9. YES Go to MAP 5800: Light pth on pge (from step 8 on pge 268) Is the hrdwre boot disply tht you see in Figure 68 on pge 270 displyed on the node? (The SAN Volume Controller 2145-DH8 does not hve front pnel disply.) Chpter 11. Using the mintennce nlysis procedures 269

288 Figure 68. Hrdwre boot disply NO Go to step 11. YES Go to step (from step 9 on pge 269) Hs the hrdwre boot disply tht you see in Figure 68 displyed for more thn 3 minutes? NO Go to step 11. YES. Go to MAP 5900: Hrdwre boot on pge 338. b. Go to MAP 5700: Repir verifiction on pge (from step 9 on pge 269 ) Is Filed displyed on the top line of the front-pnel disply of the node? Or, is the Node Fult LED (from step 8 on pge 268), which is the middle of the 3 sttus LEDs on the front pnel of SAN Volume Controller 2145-DH8, on? Figure 69 shows the node fult LED SAN Volume Controller svc00800 Figure 69. SAN Volume Controller 2145-DH8 front pnel NO Go to step 12. YES Complete these steps:. If the node hs front pnel disply, note the filure code nd go to Boot code reference on pge 160 to follow the repir ctions. b. Access the service ssistnt interfce vi the Technicin port for node ccess on pge 74 nd follow the service recommendtion presented. c. Go to MAP 5700: Repir verifiction on pge (from step 11) Is Booting displyed on the top line of the front-pnel disply of the node? 270 SAN Volume Controller: Troubleshooting Guide

289 NO Go to step 14. YES Go to step (from step 12 on pge 270) A progress br nd boot code re displyed. If the progress br does not dvnce for more thn 3 minutes, the progress is stlled. Hs the progress br stlled? NO Go to step 14. YES. Note the filure code nd go to Boot code reference on pge 160 to complete the repir ctions. b. Go to MAP 5700: Repir verifiction on pge (from step 12 on pge 270 nd step 13) If you pressed ny of the nvigtion buttons on the front pnel, wit for 60 seconds to ensure tht the disply switched to its defult disply. Is Node Error displyed on the top line of the front-pnel disply of the node? Or, is the node fult LED (from step 8 on pge 268), which is the middle of the 3 sttus LEDs on the front pnel of SAN Volume Controller 2145-DH8, on? Figure 69 on pge 270 shows the node fult LED. NO Go to step 15. YES Complete these steps:. Note the filure code nd go to Node error code overview on pge 160 to complete the repir ctions. b. If the node does not hve front pnel disply, ccess the service ssistnt interfce vi the Technicin port for node ccess on pge 74 nd follow the service recommendtion presented. c. Go to MAP 5700: Repir verifiction on pge (from step 14) Is Cluster Error displyed on the top line of the front-pnel disply of the node? NO Go to step 16. YES A cluster error ws detected. This error code is displyed on ll the opertionl nodes in the system. The fix procedures normlly repir this type of error. Follow these steps:. Go to Clustered-system code overview on pge 162 to complete the repir ctions. b. Go to MAP 5700: Repir verifiction on pge (from step 15) Is Powering Off, Restrting, Shutting Down, orpower Filure displyed in the top line of the front-pnel disply? NO Go to step 18 on pge 272. YES The progress br moves every few seconds. Wit for the opertion to complete nd then return to step 1 on pge 266 in this MAP. If the progress br does not move for 3 minutes, press power nd go to step (from step 16) Chpter 11. Using the mintennce nlysis procedures 271

290 Did the node power off? NO Complete the following steps:. Remove the power cord from the rer of the box. b. Wit 60 seconds. c. Replce the power cord. d. If the node does not power on, press power to power on the node nd then return to step 1 on pge 266 in this MAP. YES Complete the following steps:. Wit 60 seconds. b. Click power to turn on the node nd then return to step 1 on pge 266 in this MAP. Note: The 2145 UPS-1U turns off only when its power button is pressed, input power hs been lost for more thn five minutes, or the SAN Volume Controller node hs shut it down following reported loss of input power. 18. (from step 17 on pge 271) Is Chrging or Recovering displyed in the top line of the front-pnel disply of the node? NO Go to step 19. YES v If Chrging is displyed, the uninterruptible power supply bttery is not yet chrged sufficiently to support the node. If Chrging is displyed for more thn 2 hours, go to MAP 5150: 2145 UPS-1U on pge 289. v If Recovering is displyed, the uninterruptible power supply bttery is not yet chrged sufficiently to be ble to support the node immeditely following power supply filure. However, if Recovering is displyed, the node cn be used normlly. v If Recovering is displyed for more thn 2 hours, go to MAP 5150: 2145 UPS-1U on pge (from step 18) Is Vlidte WWNN? displyed on the front-pnel disply of the node? NO Go to step 20 on pge 273. YES The node is indicting tht its WWNN might need chnging. The node enters this mode when the node service controller or disk is chnged but the service procedures were not followed. Note: Do not vlidte the WWNN until you red the following informtion to ensure tht you choose the correct vlue. If you choose n incorrect vlue, you might find tht the SAN zoning for the node is lso not correct nd more thn one node is using the sme WWNN. Therefore, it is importnt to estblish the correct WWNN before you continue.. Determine which WWNN tht you wnt to use. v If the service controller ws replced, the correct vlue is probbly the WWNN tht is stored on disk (the disk WWNN). 272 SAN Volume Controller: Troubleshooting Guide

291 v If the disk ws replced, perhps s prt of frme replcement procedure, but ws not reinitilized, the correct vlue is probbly the WWNN tht is stored on the service controller (the pnel WWNN). b. Select the stored WWNN tht you wnt this node to use: v To use the WWNN tht is stored on the disk: 1) From the Vlidte WWNN? pnel, press nd relese the select button. The Disk WWNN: pnel is displyed nd shows the lst five digits of the WWNN tht is stored on the disk. 2) From the Disk WWNN: pnel, press nd relese the down button. The Use Disk WWNN? pnel is displyed. 3) Press nd relese the select button. v To use the WWNN tht is stored on the service controller: 1) From the Vlidte WWNN? pnel, press nd relese the select button. The Disk WWNN: pnel is displyed. 2) From the Disk WWNN: pnel, press, nd relese the right button. The Pnel WWNN: pnel is displyed nd shows the lst five numbers of the WWNN tht is stored on the service controller. 3) From the Pnel WWNN: pnel, press nd relese the down button. The Use Pnel WWNN? pnel is displyed. 4) Press nd relese the select button. c. After you set the WWNN, check the front-pnel disply: v If the Node WWNN: pnel is displyed on the front pnel, the node is now using the selected WWNN. The Node WWNN: pnel shows the lst five numbers of the WWNN tht you selected. v If the front pnel shows Cluster: but does not show system nme, you must use the recover procedure for clustered system to delete the node from the system nd dd the node bck into the system. 20. (from step 19 on pge 272) Is there node tht is not member of clustered system? You cn tell if node is not member of system becuse the node sttus LED is off or blinking for SAN Volume Controller 2145-DH8 or by checking the front-pnel menu. If Cluster: is displyed but no system nme is shown, the node is not member of system. If the current lnguge font llows two-line disply, the nme is on the second line of the front-pnel disply. Otherwise, you cn press the select button to disply the nme.) NO Go to step 21. YES The node is not member of system. The node might hve been deleted during mintennce procedure nd ws not dded bck into the system. Mke sure tht ech I/O group in the system contins two nodes. If n I/O group hs only one node, dd the node bck into tht system. Then, ensure tht the node is restored to the sme I/O group from which it ws deleted. 21. (from step 20) Is the front-pnel disply unredble? NO Go to step 22 on pge 274. Chpter 11. Using the mintennce nlysis procedures 273

292 YES Complete the following steps:. Check the lnguge. The disply might be set to nother lnguge. b. If the lnguge is set correctly, go to MAP 5400: Front pnel on pge (from step 21 on pge 273) No errors were detected by the SAN Volume Controller. If you suspect tht the problem tht is reported by the customer is hrdwre problem, follow these tsks:. Complete Problem Determintion procedures on your host systems, disk controllers, nd Fibre Chnnel switches. b. Ask your hrdwre support center for ssistnce. Results If you suspect tht the problem is softwre problem, see Upgrding the system documenttion for detils bout how to upgrde your entire SAN Volume Controller environment. If the problem is still not fixed, collect dignostic informtion nd contct IBM Remote Technicl Support. MAP 5040: Power SAN Volume Controller 2145-DH8 It might become necessry to solve problems tht re ssocited with power on the SAN Volume Controller 2145-DH8. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. Power problems might be ssocited with ny of the following resons: v A problem occurred during the instlltion of SAN Volume Controller node v The power switch filed to turn on the node v The power switch filed to turn off the node v Another MAP sent you here Procedure 1. Are you here becuse the node is not powered on? NO Go to step 10 on pge 278. YES Go to step (from step 1) Is the power LED on the opertor-informtion pnel continuously illuminted? Figure 70 on pge 275 shows the loction of the power LED 1 on the opertor-informtion pnel. 274 SAN Volume Controller: Troubleshooting Guide

293 Power button nd power LED (green) Figure 70. Power LED on the SAN Volume Controller 2145-DH8 NO Go to step 3. YES The node is powered on correctly. Ressess the symptoms nd return to MAP 5000: Strt or go to MAP 5700: Repir verifiction to verify the correct opertion. 3. (from step 2 on pge 274) Is the power LED on the opertor-informtion pnel flshing pproximtely four times per second? NO Go to step 4. YES The node is turned off nd is not redy to be turned on. Wit until the power LED flshes t rte of once per second, then go to step 5. If this behvior persists for more thn 3 minutes, complete the following procedure:. Remove ll input power from the SAN Volume Controller node by removing the power supply from the bck of the node. See Removing SAN Volume Controller 2145-DH8 power supply when you re removing the power cords from the node. b. Wit 1 minute nd then verify tht ll power LEDs on the node re extinguished. c. Reinsert the power supply. d. Wit for the flshing rte of the power LED to slow down to one flsh per second. Go to step 5. e. If the power LED keeps flshing t rte of four flshes per second for second time, replce the prts in the following sequence: v System bord Verify the repir by continuing with MAP 5700: Repir verifiction. 4. (from step 3) Is the Power LED on the opertor-informtion pnel flshing once per second? YES The node is in stndby mode. Input power is present. Go to step 5. NO Go to step 6 on pge (from step 3 nd step 4) Press Power on the opertor-informtion pnel of the node. Is the Power LED on the opertor-informtion pnel illuminted solid green? ifs00064 Chpter 11. Using the mintennce nlysis procedures 275

294 NO Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If the node still fils to power on, replce prts in the following sequence:. Opertor-informtion pnel ssembly b. System bord Verify the repir by continuing with MAP 5700: Repir verifiction. YES The power LED on the opertor-informtion pnel shows tht the node successfully powered on. Verify the correct opertion by continuing with MAP 5700: Repir verifiction. 6. (from step 4 on pge 275) Is the rer pnel power LED on or flshing?figure 71 shows the loction of the power LED 1 on the SAN Volume Controller 2145-DH Figure 71. Power LED indictor on the rer pnel of the SAN Volume Controller 2145-DH8 svc00574 NO Go to step 7. YES The opertor-informtion pnel is filing. Verify tht the opertor-informtion pnel cble is seted on the system bord. If the node still fils to power on, replce prts in the following sequence:. Opertor-informtion pnel ssembly b. System bord 7. (from step 6) Are the c LED indictors on the rer of the power supply ssemblies illuminted? Figure 72 on pge 277 shows the loction of the c LED 1, the dc LED 2, nd the power-supply error LED 3 on the rer of the power supply ssembly tht is on the rer pnel of the SAN Volume Controller 2145-DH SAN Volume Controller: Troubleshooting Guide

295 1 AC LED (green) AC 2 DC LED (green) DC 3 Power-supply error LED (yellow) AC DC Figure 72. AC, dc, nd power-supply error LED indictors on the rer pnel of the SAN Volume Controller 2145-DH8 NO Verify tht the input power cble or cbles re securely connected t both ends nd show no sign of dmge; replce dmged cbles. If the node still fils to power on, replce the specified prts tht re bsed on the SAN Volume Controller model type. Replce the SAN Volume Controller 2145-DH8 prts in the following sequence:. Power supply 750 W YES Go to step (from step 7 on pge 276) Is the power-supply error LED on the rer of the SAN Volume Controller 2145-DH8 power supply illuminted? Figure 72 shows the loction of the power-supply error LED 3. YES Replce the power supply unit. NO Go to step 9 9. (from step 8) Are the dc LED indictors on the rer of the power supply ssemblies illuminted? svc00794 Chpter 11. Using the mintennce nlysis procedures 277

296 NO Replce the SAN Volume Controller 2145-DH8 prts in the following sequence:. Power supply 750 W b. System bord YES Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If the node still fils to power on, replce prts in the following sequence:. Opertor-informtion pnel b. Cble, signl c. System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from step 1 on pge 274) The node does not power off immeditely when the power button is pressed. When the node fully boots, the node powers-off under the control of the SAN Volume Controller softwre. The power-off opertion cn tke up to 5 minutes to complete. Is the power LED on the opertor-informtion pnel flshing pproximtely four times per second? NO Go to step 11. YES Wit for the node to power off. If the node fils to power off fter 5 minutes, go to step (from step 10) Attention: Turning off the node by ny mens other thn using the mngement GUI might cuse loss of dt in the node cche. If you re performing concurrent mintennce, this node must be deleted from the system before you proceed. Ask the customer to delete the node from the system now. If they re unble to delete the node, cll your support center for ssistnce before you proceed. The node cnnot be turned off either becuse of softwre fult or hrdwre filure. Press nd hold the power button. The node cn turn off within 5 seconds. Did the node turn off? NO Determine whether you re using n Advnced Configurtion nd Power Interfce (ACPI) or non-acpi operting system. If you re using non-acpi operting system, complete the following steps: Press Ctrl+Alt+Delete. Turn off the server by pressing nd holding Power for 5 seconds. Restrt the server. If the server fils POST nd pressing Power does not work, disconnect the power cord for 20 seconds Reconnect the power cord nd restrt the server. If the problem remins or if you re using n ACPI-wre operting system, suspect the system bord. Go to step 12 on pge SAN Volume Controller: Troubleshooting Guide

297 YES Go to step (from step 11 on pge 278) Press the power button to turn on the node. Did the node turn on nd boot correctly? NO Go to MAP 5000: Strt on pge 265 to resolve the problem. YES Go to step (from step 12) The node probbly suffered softwre filure. Memory dump dt might be cptured tht helps resolve the problem. Cll your support center for ssistnce. MAP 5050: Power 2145-CG8, 2145-CF8, nd G4 This topic helps you to solve power problems tht hve occurred on SAN Volume Controller models 2145-CG8, 2145-CF8, nd G4. If you re using SAN Volume Controller A4, see the Power MAP for tht SAN Volume Controller model. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. You might hve been sent here for one of the following resons: v A problem occurred during the instlltion of SAN Volume Controller node v The power switch filed to turn the node on v The power switch filed to turn the node off v Another MAP sent you here About this tsk Perform the following steps: Procedure 1. Are you here becuse the node is not powered on? NO Go to step 11 on pge 284. YES Go to step (from step 1) Is the power LED on the opertor-informtion pnel continuously illuminted? Figure 73 on pge 280 shows the loction of the power LED 1 on the opertor-informtion pnel. Chpter 11. Using the mintennce nlysis procedures 279

298 CF G CG8 svc00715 Figure 73. Power LED on the SAN Volume Controller models 2145-CG8, 2145-CF8, or G4 opertor-informtion pnel NO Go to step 3. YES The node is powered on correctly. Ressess the symptoms nd return to MAP 5000: Strt on pge 265 or go to MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. 3. (from step 2 on pge 279) Is the power LED on the opertor-informtion pnel flshing pproximtely four times per second? NO Go to step 4. YES The node is turned off nd is not redy to be turned on. Wit until the power LED flshes t rte of pproximtely once per second, then go to step 5. If this behvior persists for more thn three minutes, perform the following procedure:. Remove ll input power from the SAN Volume Controller node by removing the power retention brckets nd the power cords from the bck of the node. See Removing the cble-retention brckets to see how to remove the cble-rentention brckets when removing the power cords from the node. b. Wit one minute nd then verify tht ll power LEDs on the node re extinguished. c. Reinsert the power cords nd power retention brckets. d. Wit for the flshing rte of the power LED to slow down to one flsh per second. Go to step 5. e. If the power LED keeps flshing t rte of four flshes per second for second time, replce the prts in the following sequence: v System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from step 3) Is the Power LED on the opertor-informtion pnel flshing pproximtely once per second? YES The node is in stndby mode. Input power is present. Go to step 5. NO Go to step 6 on pge (from step 3 nd step 4) Press the power-on button on the opertor-informtion pnel of the node. 280 SAN Volume Controller: Troubleshooting Guide

299 Is the Power LED on the opertor-informtion pnel illuminted solid green? NO Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If you re working on SAN Volume Controller 2145-CG8, or 2145-CF8, nd the node still fils to power on, replce prts in the following sequence:. Opertor-informtion pnel ssembly b. System bord If you re working on SAN Volume Controller G4, verify tht the opertor-informtion pnel cble is correctly seted on the system bord. If the node still fils to power on, replce prts in the following sequence:. Opertor-informtion pnel ssembly b. System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES The power-on indictor on the opertor-informtion pnel shows tht the node hs successfully powered on. Continue with MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. 6. (from step 4 on pge 280) Is the rer pnel power LED on or flshing?figure 74 shows the loction of the power LED 1 on the rer pnel of the G4 node. Figure 75 on pge 282 shows the loction of the power LED 1 on the 2145-CF8 or the 2145-CG8. Figure 74. Power LED indictor on the rer pnel of the SAN Volume Controller G4 Chpter 11. Using the mintennce nlysis procedures 281

300 svc00574 Figure 75. Power LED indictor on the rer pnel of the SAN Volume Controller 2145-CG8 or 2145-CF8 NO Go to step 7. YES The opertor-informtion pnel is filing. Verify tht the opertor-informtion pnel cble is seted on the system bord. If you re working on SAN Volume Controller 2145-CG8, or 2145-CF8, nd the node still fils to power on, replce prts in the following sequence:. Opertor-informtion pnel ssembly b. System bord If you re working on SAN Volume Controller G4, verify tht the opertor-informtion pnel cble is correctly seted on the system bord. If the SAN Volume Controller G4 still fils to power on, replce prts in the following sequence:. Opertor-informtion pnel ssembly b. System bord 7. (from step 6 on pge 281) Locte the 2145 UPS-1U (2145 UPS-1U) tht is connected to this node. Does the 2145 UPS-1U tht is powering this node hve its power on nd is its lod segment 2 indictor solid green? NO Go to MAP 5150: 2145 UPS-1U on pge 289. YES Go to step (from step 7) Are the c LED indictors on the rer of the power supply ssemblies illuminted? Figure 76 on pge 283 shows the loction of the c LED 1 nd the dc LED 2 on the rer of the power supply ssembly locted on the rer pnel of the G4. Figure 77 on pge 283 shows the loction of the c LED 1 nd the dc LED 2 on the rer of the power supply ssembly tht is on the rer pnel of the 2145-CF8 or the 2145-CG SAN Volume Controller: Troubleshooting Guide

301 svc00307 Figure 76. AC nd dc LED indictors on the rer pnel of the SAN Volume Controller G svc00571 Figure 77. Power LED indictor nd c nd dc indictors on the rer pnel of the SAN Volume Controller 2145-CG8 or 2145-CF8 NO Verify tht the input power cble or cbles re securely connected t both ends nd show no sign of dmge; otherwise, if the cble or cbles re fulty or dmged, replce them. If the node still fils to power on, replce the specified prts bsed on the SAN Volume Controller model type. Replce the SAN Volume Controller 2145-CG8, or 2145-CF8 prts in the following sequence:. Power supply 675W Replce the SAN Volume Controller G4 prts in the following sequence:. Power supply 670W b. Power bckplne Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Go to step 9 for SAN Volume Controller 2145-CG8, or 2145-CF8 models. Go to step 10 for ll other models. 9. (from step 8 on pge 282) Is the power supply error LED on the rer of the SAN Volume Controller 2145-CG8, or 2145-CF8 power supply ssemblies illuminted? Figure 75 on pge 282 shows the loction of the power LED 1 on the 2145-CF8 or the 2145-CG8. YES Replce the power supply unit. NO Go to step (from step 8 on pge 282 or step 9) Chpter 11. Using the mintennce nlysis procedures 283

302 Are the dc LED indictors on the rer of the power supply ssemblies illuminted? NO Replce the SAN Volume Controller 2145-CG8, or 2145-CF8 prts in the following sequence:. Power supply 675W b. System bord Replce the SAN Volume Controller G4 prts in the following sequence:. Power bckplne b. Power supply 670W c. System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If the node still fils to power on, replce prts in the following sequence:. Opertor-informtion pnel b. Cble, signl, front pnel c. System bord (if the node is SAN Volume Controller 2145-CG8, SAN Volume Controller 2145-CF8, or SAN Volume Controller G4) Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from step 1 on pge 279) The node will not power off immeditely when the power button is pressed. When the node is fully booted, power-off is performed under the control of the SAN Volume Controller softwre. The power-off opertion cn tke up to five minutes to complete. Is Powering Off displyed on the front pnel? NO Go to step 12. YES Wit for the node to power off. If the node fils to power off fter 5 minutes, go to step (from step 11) Attention: Turning off the node by ny mens other thn using the mngement GUI might cuse loss of dt in the node cche. If you re performing concurrent mintennce, this node must be deleted from the system before you proceed. Ask the customer to delete the node from the system now. If they re unble to delete the node, cll your support center for ssistnce before you proceed. The node cnnot be turned off either becuse of softwre fult or hrdwre filure. Press nd hold the power button. The node should turn off within five seconds. Did the node turn off? NO Turn off the 2145 UPS-1U tht is connected to this node. Attention: Be sure tht you re turning off the correct 2145 UPS-1U. If necessry, trce the cbles bck to the 2145 UPS-1U ssembly. Turning off the wrong 2145 UPS-1U might cuse customer dt loss. 284 SAN Volume Controller: Troubleshooting Guide

303 Go to step 13. YES Go to step (from step 12 on pge 284) If necessry, turn on the 2145 UPS-1U tht is connected to this node nd then press the power button to turn the node on. Did the node turn on nd boot correctly? NO Go to MAP 5000: Strt on pge 265 to resolve the problem. MAP 5060: Power A4 YES Go to step (from step 13) The node hs probbly suffered softwre filure. Dump dt might hve been cptured tht will help resolve the problem. Cll your support center for ssistnce. MAP 5060: Power A4 helps you to solve power problems tht hve occurred on the SAN Volume Controller A4 node. If you re using ny other SAN Volume Controller model, see the Power MAP for tht SAN Volume Controller model. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. You might hve been sent here for one of the following resons: v A problem occurred during the instlltion of A4 node. v The power switch filed to turn the node on. v The power switch filed to turn the node off. v Another MAP sent you here. About this tsk Perform the following steps: Procedure 1. Are you here becuse the node is not turned on? NO Go to step 9 on pge 288. YES Go to step (from step 1) Is the power LED on the opertor-informtion pnel continuously illuminted? Figure 78 on pge 286 shows the loction of the power LED 1 on the opertor-informtion pnel. Chpter 11. Using the mintennce nlysis procedures 285

304 Figure 78. Power LED on the SAN Volume Controller A4 opertor-informtion pnel NO Go to step 3. YES The node turned on correctly. Ressess the symptoms nd return to MAP 5000: Strt on pge 265 or go to MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. 3. (from step 2 on pge 285) Is the power LED on the opertor-informtion pnel flshing? NO Go to step 5. YES The node is in stndby mode. Input power is present. Go to step (from step 3) Press the power-on button on the opertor-informtion pnel of the node. Is the Power LED on the opertor-informtion pnel illuminted solid green? NO Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If the node still fils to turn on, replce prts in the following sequence:. Opertor-informtion pnel b. Opertor-informtion pnel cble c. System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES The power-on indictor on the opertor-informtion pnel shows tht the node hs successfully turned on. Continue with MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. 5. (from step 3) Locte the 2145 UPS-1U tht is connected to this node. Does the 2145 UPS-1U tht is powering this node hve its power on nd is its lod segment 2 indictor solid green? NO Go to MAP 5150: 2145 UPS-1U on pge 289. YES Verify tht the input-power cble is securely connected t both ends nd shows no sign of dmge; otherwise, if the cble is fulty or dmged, replce it. If the node still fils to turn on, go to step 6. If the node turns on, continue with MAP 5700: Repir verifiction on pge (from step 5) Remove the node from the rck nd remove the top cover. Reconnect the power cble, which is still connected to the 2145 UPS-1U, to the node. Is the stndby power LED tht is on the system bord illuminted? Figure 79 on pge 287 shows where the dignostics LEDs re locted on the system bord. 286 SAN Volume Controller: Troubleshooting Guide

305 NO Go to step 7. YES Replce the SAN Volume Controller A4 prts in the following sequence:. Opertor-informtion pnel b. Opertor-informtion pnel cble Figure 79. SAN Volume Controller A4 system bord LEDs 1 Fn 1 error LED 2 Fn 2 error LED 3 Fn 3 error LED 4 DIMM 1 error LED 5 DIMM 2 error LED 6 DIMM 3 error LED 7 DIMM 4 error LED 8 PCI Express slot 2 error LED 9 PCI Express slot 1 error LED 10 Fn 4 error LED 11 Fn 5 error LED 12 Voltge regultor error LED 13 Stndby power LED 14 Power good LED 15 Bsebord mngement controller hertbet LED 16 SAS/SATA controller error LED 7. (from step 6 on pge 286) Chpter 11. Using the mintennce nlysis procedures 287

306 Is the voltge regultor LED tht is on the system bord illuminted? NO Go to step 8. YES Replce the system bord. 8. (from step 7 on pge 287) Replce the SAN Volume Controller A4 prts in the following sequence:. Input-power cble (or the 2145 UPS-1U to SAN Volume Controller node power cble) b. Power supply Are you now ble to turn on the node? NO Contct your IBM service representtive for ssistnce. YES The power-on indictor on the front pnel shows tht the node hs successfully turned on. Continue with MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. 9. (from step 1 on pge 285) The node does not turn off when the power button is pressed. When the node is fully booted, power-off is performed under the control of the SAN Volume Controller softwre. The power-off opertion cn tke up to five minutes to complete. Is Powering Off displyed on the front pnel? NO Go to step 10. YES Wit for the node to turn off. If the node fils to turn off fter 5 minutes, go to step (from step 9) Attention: Turning off the node by ny mens other thn using the mngement GUI might cuse loss of dt in the node cche. If you re performing concurrent mintennce, this node must be deleted from the system before you proceed. Ask the customer to delete the node from the system now. If they re unble to delete the node, contct your IBM service representtive for ssistnce before you proceed. The node cnnot be turned off either becuse of softwre fult or hrdwre filure. Press nd hold the power button. The node should turn off within five seconds. Did the node turn off? NO Turn off the 2145 UPS-1U tht is connected to this node. Attention: Be sure tht you re turning off the correct 2145 UPS-1U. If necessry, trce the cbles bck to the 2145 UPS-1U ssembly. Turning off the wrong 2145 UPS-1U might cuse customer dt loss. Go to step 11. YES Go to step (from step 8) If necessry, turn on the 2145 UPS-1U tht is connected to this node nd then press the power button to turn on the node. Did the node turn on nd boot correctly? NO Go to MAP 5000: Strt on pge 265 to resolve the problem. YES Go to step (from step 11) 288 SAN Volume Controller: Troubleshooting Guide

307 MAP 5150: 2145 UPS-1U The node hs probbly suffered softwre filure. Dump dt might hve been cptured tht will help resolve the problem. Contct your IBM service representtive for ssistnce. MAP 5150: 2145 UPS-1U helps you solve problems tht hve occurred in the 2145 UPS-1U systems tht re used on SAN Volume Controller. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. You my hve been sent here for one of the following resons: v The system problem determintion procedures sent you here v A problem occurred during the instlltion of SAN Volume Controller v Another MAP sent you here v A customer observed problem tht ws not detected by the system problem determintion procedures About this tsk Figure 80 shows n illustrtion of the front of the pnel for the 2145 UPS-1U. LOAD 2 LOAD yyzvm Figure UPS-1U front-pnel ssembly 1 Lod segment 2 indictor 2 Lod segment 1 indictor 3 Alrm 4 On-bttery indictor 5 Overlod indictor 6 Power-on indictor 7 On or off button 8 Test nd lrm reset button Tble 93 on pge 290 identifies which sttus nd error LEDs tht disply on the 2145 UPS-1U front-pnel ssembly relte to the specified error conditions. It lso lists the uninterruptible power supply lert-buzzer behvior. Chpter 11. Using the mintennce nlysis procedures 289

308 Tble UPS-1U error indictors [1] Lod2 [2] Lod1 [3] Alrm [4] Bttery Green (see Note 1) Green Green Green Green Green Green Either on or off Either on or off Amber (see Note 2) Either on or off Either on or off Either on or off Either on or off Either on or off Either on or off Either on or off Flshing red Flshing red Flshing red Flshing red Flshing red Flshing red Flshing red [5] Overlod [6] Power-on Buzzer Error condition Green (see Note 3 ) Amber Green Beeps for two seconds nd then stops Flshing mber Flshing mber Flshing mber Amber Flshing mber Flshing red No errors; the 2145 UPS-1U ws configured by the SAN Volume Controller Green No errors; the 2145 UPS-1U is not yet configured by the SAN Volume Controller Flshing green Flshing green Flshing green Three beeps every ten seconds Solid on Solid on Solid on Beeps for four seconds nd then stops Beeps for two seconds nd then stops Red Green Beeps for one second nd then stops Amber Red Beeps for one second nd then stops The c power is over or under limit. The uninterruptible power supply hs switched to bttery mode. Bttery undervoltge Bttery overvoltge Output wve is bnorml when the chrger is open, on bttery mode The c-power output wve is under low limit or bove high limit on bttery mode On bttery (no c power) Low bttery (no c power) Overlod while on line Overlod while on bttery Green Solid on Fn filure Amber Solid on Bttery test fil Red Solid on Overlod timeout 290 SAN Volume Controller: Troubleshooting Guide

309 Tble UPS-1U error indictors (continued) [1] Lod2 [2] Lod1 [3] Alrm [4] Bttery [5] Overlod [6] Power-on Buzzer Error condition Flshing red Flshing red Amber Green Solid on Over temperture Amber Red Green Output short circuit Notes: 1. The green Lod2 LED ([1]) indictes tht power is being supplied to the right pir of c-power outlets s seen from the rer of the 2145 UPS-1U. 2. The mber Lod1 LED ([2]) indictes tht power is being supplied to the left pir of c-power outlets s seen from the rer of the 2145 UPS-1U. These outlets re not used by the SAN Volume Controller. This LED might be illuminted during power-on sequences, but it is typiclly extinguished by the SAN Volume Controller node tht is ttched to the 2145 UPS-1U. 3. A blnk cell indictes tht the light or buzzer is off. Procedure 1. Is the power-on indictor for the 2145 UPS-1U tht is connected to the filing SAN Volume Controller off? NO Go to step 3. YES Go to step (from step 1) Are other 2145 UPS-1U units showing the power-on indictor s off? NO The 2145 UPS-1U might be in stndby mode. This cn be becuse the on or off button on this 2145 UPS-1U ws pressed, input power hs been missing for more thn five minutes, or becuse the SAN Volume Controller shut it down following reported loss of input power. Press nd hold the on or off button until the 2145 UPS-1U power-on indictor is illuminted (pproximtely five seconds). On some versions of the 2145 UPS-1U, you need pointed device, such s screwdriver, to press the on or off button. Go to step 3. YES Either min power is missing from the instlltion or redundnt c-power switch hs filed. If the 2145 UPS-1U units re connected to redundnt c-power switch, go to MAP 5320: Redundnt c power on pge 295. Otherwise, complete these steps:. Restore min power to instlltion. b. Verify the repir by continuing with MAP 5250: 2145 UPS-1U repir verifiction on pge (from step 1 nd step 2) Are the power-on nd lod segment 2 indictors for the 2145 UPS-1U illuminted solid green, with service, on-bttery, nd overlod indictors off? NO Go to step 4. YES The 2145 UPS-1U is no longer showing fult. Verify the repir by continuing with MAP 5250: 2145 UPS-1U repir verifiction on pge (from step 3) Chpter 11. Using the mintennce nlysis procedures 291

310 Is the 2145 UPS-1U on-bttery indictor illuminted yellow (solid or flshing), with service nd overlod indictors off? NO Go to step 5. YES The input power supply to this 2145 UPS-1U is not working or is not correctly connected, or the 2145 UPS-1U is receiving input power tht might be unstble or outside the specified voltge or frequency rnge. (The voltge should be between 200V nd 240V nd the frequency should be either 50 Hz or 60 Hz.) The SAN Volume Controller utomticlly djusts the 2145 UPS-1U voltge rnge. If the input voltge hs recently chnged, the lrm condition might be present until the SAN Volume Controller hs djusted the lrm setting. Power on the SAN Volume Controller tht is connected to the 2145 UPS-1U. If the SAN Volume Controller strts the on-bttery indictor should go off within five minutes. If the SAN Volume Controller powers off gin or if the condition persists for t lest five minutes, do the following:. Check the input circuit protector on the 2145 UPS-1U rer pnel, nd press it, if it is open. b. If redundnt c power is used for the 2145 UPS-1U, check the voltge nd frequency t the redundnt c-power switch output receptble connected to this 2145 UPS-1U. If there is no power, go to MAP 5340: Redundnt c power verifiction on pge 296. If the power is not within specifiction, sk the customer to resolve the issue. If redundnt c power is not used for this uninterruptible power supply, check the site power outlet for the 2145 UPS-1U providing power to this SAN Volume Controller. Check the connection, voltge, nd frequency. If the power is not within specifiction, sk the customer to resolve the issue. c. If the input power is within specifiction nd the input circuit protector is stble, replce the field-replceble units (FRUs) in the following sequence: 1) 2145 UPS-1U power cord 2) 2145 UPS-1U d. Verify the repir by continuing with MAP 5250: 2145 UPS-1U repir verifiction on pge (from step 4 on pge 291) Is the 2145 UPS-1U overlod indictor illuminted solid red? NO Go to step 6 on pge 293. YES The 2145 UPS-1U output power requirement hs exceeded the 2145 UPS-1U cpcity.. Check tht only one SAN Volume Controller node is connected to the 2145 UPS-1U. b. Check tht no other lods re connected to the 2145 UPS-1U. c. After ensuring tht the output loding is correct, turn off the 2145 UPS-1U by pressing the on or off button until the power-on indictor goes off. Then unplug the input power from the 2145 UPS-1U. Wit t lest five seconds until ll LEDs re off nd restrt the 2145 UPS-1U by reconnecting it to input power nd pressing the on or off button until the 2145 UPS-1U power-on indictor is illuminted (pproximtely five seconds). On some versions of the 2145 UPS-1U, you need pointed device, such s screwdriver, to press the on or off button. 292 SAN Volume Controller: Troubleshooting Guide

311 d. If the condition persists, replce the 2145 UPS-1U. Note: If the condition recurs, replce the power supply or power supplies in the node. e. Verify the repir by continuing with MAP 5250: 2145 UPS-1U repir verifiction on pge (from step 5 on pge 292) Is the 2145 UPS-1U service indictor illuminted flshing red nd the on-bttery indictor illuminted solid yellow, with the power-on nd overlod indictors off? NO Go to step 7. YES The 2145 UPS-1U bttery might be fully dischrged or fulty.. Check tht the 2145 UPS-1U hs been connected to power outlet for t lest two hours to chrge the bttery. After chrging the bttery, press nd hold the test or lrm reset button for three seconds; nd then check the service indictor. b. If the service indictor is still flshing, replce the 2145 UPS-1U. c. Verify the repir by continuing with MAP 5250: 2145 UPS-1U repir verifiction on pge (from step 6) Is the 2145 UPS-1U service indictor illuminted flshing red, the on-bttery indictor illuminted solid yellow, nd the power-on illuminted solid green, with the overlod indictor off? NO Go to step 8. YES The 2145 UPS-1U internl temperture is too high.. Turn off the 2145 UPS-1U by pressing the on or off button until the power-on indictor goes off. Then unplug the 2145 UPS-1U. Cler vents t the front nd rer of the 2145 UPS-1U. Remove ny het sources. Ensure the irflow round the 2145 UPS-1U is not restricted. b. Wit t lest five minutes nd restrt the 2145 UPS-1U by reconnecting to input power nd pressing the on or off button until the 2145 UPS-1U power-on indictor is illuminted (pproximtely five seconds). c. If the condition persists, replce the 2145 UPS-1U. d. Verify the repir by continuing with MAP 5250: 2145 UPS-1U repir verifiction on pge (from step 7) Is the 2145 UPS-1U, service, on-bttery, overlod, nd power-on indictors illuminted nd flshing? NO The 2145 UPS-1U hs n internl fult.. Replce the 2145 UPS-1U. b. Verify the repir by continuing with MAP 5250: 2145 UPS-1U repir verifiction on pge 294. YES The 2145 UPS-1U bttery might be fully dischrged or fulty.. Check tht the 2145 UPS-1U hs been connected to power outlet for t lest two hours to chrge the bttery. After chrging the bttery, press nd hold the test or lrm reset button for three seconds nd then check the service indictor. Chpter 11. Using the mintennce nlysis procedures 293

312 MAP 5250: 2145 UPS-1U repir verifiction b. If the service indictor is still flshing, replce the 2145 UPS-1U. c. Verify the repir by continuing with MAP 5250: 2145 UPS-1U repir verifiction. MAP 5250: 2145 UPS-1U repir verifiction helps you to verify tht field replceble units (FRUs) tht you hve exchnged for new FRUs, or repir ctions tht were done, hve solved ll the problems on the SAN Volume Controller 2145 UPS-1U. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. You my hve been sent here becuse you hve performed repir nd wnt to confirm tht no other problems exist on the mchine. About this tsk Perform the following steps: Procedure 1. Are the power-on nd lod segment 2 indictors for the repired 2145 UPS-1U illuminted solid green, with service, on-bttery, nd overlod indictors off? NO Continue with MAP 5000: Strt on pge 265. YES Go to step (from step 1) Is the SAN Volume Controller node powered by this 2145 UPS-1U powered on? NO Press power-on on the SAN Volume Controller node tht is connected to this 2145 UPS-1U nd is powered off. Go to step 3. YES Go to step (from step 2) Is the node tht is connected to this 2145 UPS-1U still not powered on or showing error codes in the front pnel disply? NO Go to step 4. YES Continue with MAP 5000: Strt on pge (from step 3) Does the SAN Volume Controller node tht is connected to this 2145 UPS-1U show Chrging on the front pnel disply? NO Go to step 5. YES Wit for the Chrging disply to finish (this might tke up to two hours). Go to step (from step 4) 294 SAN Volume Controller: Troubleshooting Guide

313 Press nd hold the test/lrm reset button on the repired 2145 UPS-1U for three seconds to initite self-test. During the test, individul indictors illuminte s vrious prts of the 2145 UPS-1U re checked. Does the 2145 UPS-1U service, on-bttery, or overlod indictor sty on? NO 2145 UPS-1U repir verifiction hs completed successfully. Continue with MAP 5700: Repir verifiction on pge 318. MAP 5320: Redundnt c power YES Continue with MAP 5000: Strt on pge 265. MAP 5320: Redundnt c power helps you solve problems tht hve occurred in the redundnt c-power switches used on SAN Volume Controller. Use this MAP when 2145 UPS-1U tht is connected to redundnt c-power switch does not pper to hve input power. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. You might hve been sent here for one of the following resons: v A problem occurred during the instlltion of SAN Volume Controller. v MAP 5150: 2145 UPS-1U on pge 289 sent you here. About this tsk Perform the following steps to solve problems tht hve occurred in the redundnt c-power switches: Procedure 1. One or two 2145 UPS-1Us might be connected to the redundnt c-power switch. Is the power-on indictor on ny of the connected 2145 UPS-1Us on? NO Go to step 3 on pge 296. YES The redundnt c-power switch is powered. Go to step (from step 1) Mesure the voltge t the redundnt c-power switch output socket connected to the 2145 UPS-1U tht is not showing power-on. CAUTION: Ensure tht you do not remove the power cble of ny powered uninterruptible power supply units Is there power t the output socket? NO One redundnt c-power switch output is working while the other is not. Replce the redundnt c-power switch. Chpter 11. Using the mintennce nlysis procedures 295

314 CAUTION: You might need to power-off n opertionl node to replce the redundnt c-power switch ssembly. If this is the cse, consult with the customer to determine suitble time to perform the replcement. See MAP 5350: Powering off SAN Volume Controller node on pge 298. After you replce the redundnt c-power switch, continue with MAP 5340: Redundnt c power verifiction. YES The redundnt c-power switch is working. There is problem with the 2145 UPS-1U power cord or the 2145 UPS-1U. Return to the procedure tht clled this MAP nd continue from where you were within tht procedure. It will help you nlyze the problem with the 2145 UPS-1U power cord or the 2145 UPS-1U. 3. (from step 1 on pge 295) None of the used redundnt c-power switch outputs ppers to hve power. Are the two input power cbles for the redundnt c-power switches correctly connected to the redundnt c-power switch nd to different mins circuits? NO Correctly connect the cbles. Go to MAP 5340: Redundnt c power verifiction. YES Verify tht there is min power t both the site's power distribution units tht re providing power to this redundnt c-power switch. Go to step (from step 3) Is power vilble t one or more of the site's power distribution units tht re providing power to this redundnt c-power switch? NO YES MAP 5340: Redundnt c power verifiction Hve the customer fix the mins circuits. Return to the procedure tht clled this MAP nd continue from where you were within tht procedure. The redundnt c-power switch should operte in this sitution. Replce the redundnt c-power switch ssembly. After you replce the redundnt c-power switch, continue with MAP 5340: Redundnt c power verifiction. MAP 5340: Redundnt c power verifiction helps you verify tht redundnt c-power switch is functioning correctly. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. You might hve been sent here becuse you hve replced redundnt c-power switch or corrected the cbling of redundnt c-power switch. You cn lso use this MAP if you think redundnt c-power switch might not be working correctly, becuse it is connected to nodes tht hve lost power when only one c power circuit lost power. 296 SAN Volume Controller: Troubleshooting Guide

315 In this MAP, you will be sked to confirm tht power is vilble t the redundnt c-power switch output sockets 1 nd 2. If the redundnt c-power switch is connected to nodes tht re not powered on, use voltge meter to confirm tht power is vilble. If the redundnt c-power switch is powering nodes tht re powered on (so the nodes re opertionl), tke some precutions before continuing with these tests. Although you do not hve to power off the nodes to conduct the test, the nodes will power off if the redundnt c-power switch is not functioning correctly. About this tsk For ech of the powered-on nodes connected to this redundnt c-power switch, perform the following steps: 1. Use the mngement GUI or the commnd-line interfce (CLI) to confirm tht the other node in the sme I/O group s this node is online. 2. Use the mngement GUI or the CLI to confirm tht ll virtul disks connected to this I/O group re online. 3. Check the redundnt c-power switch output cbles to confirm tht the redundnt c-power switch is not connected to two nodes in the sme I/O group. If ny of these tests fil, correct ny filures before continuing with this MAP. If you re performing the verifiction using powered-on nodes, understnd tht power is no longer vilble if the following is true: v v The on-bttery indictor on the 2145 UPS-1U tht connects the redundnt c-power switch to the node lights for more thn five seconds. The SAN Volume Controller node disply shows Power Filure. When the instructions sy remove power, you cn switch the power off if the sitepower distribution unit hs outputs tht re individully switched; otherwise, remove the specified redundnt c-power switch power cble from the site power distribution unit's outlet. Perform the following steps: Procedure 1. Are the two site power distribution units providing power to this redundnt c-power switch connected to different power circuits? NO Correct the problem nd then return to this MAP. YES Go to step (from step 1) Are both of the site power distribution units providing power to this redundnt c-power switch powered? NO Correct the problem nd then return to the strt of this MAP. YES Go to step (from step 2) Are the two cbles tht re connecting the site power distribution units to the redundnt c-power switch connected? NO Correct the problem nd then return to the strt of this MAP. Chpter 11. Using the mintennce nlysis procedures 297

316 YES Go to step (from step 3 on pge 297) Is there power t the redundnt c-power switch output socket 2? NO Go to step 8. YES Go to step (from step 4) Is there power t the redundnt c-power switch output socket 1? NO Go to step 8. YES Go to step (from step 5) Remove power from the Min power cble to the redundnt c-power switch. Is there power t the redundnt c-power switch output socket 1? NO Go to step 8. YES Go to step (from step 6) Reconnect the Min power cble. Remove power from the Bckup power cble to the redundnt c-power switch. Is there power t the redundnt c-power switch output socket 1? NO Go to step 8. YES Reconnect the Bckup power cble. The redundnt c power verifiction hs been successfully completed. Continue with MAP 5700: Repir verifiction on pge (from steps 4, 5, 6, nd 7) The redundnt c-power switch hs not functioned s expected. Replce the redundnt c-power switch ssembly. Return to the strt of this MAP. Results MAP 5350: Powering off SAN Volume Controller node MAP 5350: Powering off SAN Volume Controller node helps you power off single node to perform service ction without disrupting host ccess to disks. Before you begin Powering off single node does not disrupt normlly the opertion of clustered system. Norml opertion within SAN Volume Controller system hs nodes in pirs clled I/O groups. An I/O group continues to hndle I/O to the disks it mnges with only single node powered on. However, performnce degrdes nd resilience to error is reduced. Be creful when powering off node to impct the system no more thn necessry. If you do not follow the procedures outlined here, your ppliction hosts might lose ccess to their dt or they might lose dt in the worst cse. You cn use the following preferred methods to power off node tht is member of system nd not offline: 1. Use the Shut Down Node option on the mngement GUI. 2. Use the CLI commnd stopsystem node nme. 298 SAN Volume Controller: Troubleshooting Guide

317 It is preferble to use either the mngement GUI or the commnd-line interfce (CLI) to power off node, s these methods provide controlled hndover to the prtner node nd provide better resilience to other fults in the system. Only if node is offline or not member of system must you power it off using the power button. About this tsk To provide the lest disruption when powering off node, ll of the following conditions should pply: v The other node in the I/O group is powered on nd ctive in the system. v v v The other node in the I/O group hs SAN Fibre Chnnel connections to ll hosts nd disk controllers mnged by the I/O group. All volumes hndled by this I/O group re online. Host multipthing is online to the other node in the I/O group. In some circumstnces, the reson you power off the node might mke these conditions impossible. For instnce, if you replce broken Fibre Chnnel crd, volumes do not show n online sttus. Use your good judgment to decide tht it is sfe to proceed when condition is not met. Alwys check with the system dministrtor before proceeding with power off tht you know disrupts I/O ccess, s the system dministrtor might prefer to wit for more suitble time or to suspend host pplictions. To ensure smooth restrt, node must sve dt structures tht it cnnot recrete to its locl, internl disk drive. The mount of dt the node sves to locl disk cn be high, so this opertion might tke severl minutes. Do not ttempt to interrupt the controlled power off. Attention: The following ctions do not llow the node to sve dt to its locl disk. Therefore, do not power off node using the following methods: v Removing the power cble between the node nd the uninterruptible power supply. Normlly the uninterruptible power supply provides sufficient power to llow the write to locl disk in the event of power filure, but obviously it is unble to provide power in this cse. v Holding down the power button on the node. When you press nd relese the power button, the node indictes this to the softwre so the node cn write its dt to locl disk before the node powers off. When you hold down the power button, the hrdwre interprets this s n emergency power off indiction nd shuts down immeditely. The hrdwre does not sve the dt to locl disk before powering down. The emergency power off occurs pproximtely four seconds fter you press nd hold down the power button. v Pressing the reset button on the light pth dignostics pnel. Using the mngement GUI to power off system Use the mngement GUI to power off system. Chpter 11. Using the mintennce nlysis procedures 299

318 Procedure To use the mngement GUI to power off system, complete the following steps: 1. Lunch the mngement GUI for the system tht you re servicing. Optionlly, you cn sign on to the IBM System Storge Productivity Center s n dministrtor to lunch the mngement GUI for the system tht you re servicing. 2. Find the system to shut down. If the nodes to power off re shown s Offline, the nodes re not prticipting in the system. In such circumstnces, use the power button on the offline nodes to power off the nodes. If the nodes to power off re shown s Online, powering off the nodes cn result in their dependent volumes lso going offline:. Select the node nd click Show Dependent Volumes. b. Mke sure the sttus of ech volume in the I/O group is Online. You might need to view more thn one pge. If ny volumes re Degrded, only one node in the I/O is processing I/O requests for tht volume. If tht node is powered off, it impcts ll the hosts tht re submitting I/O requests to the degrded volume. If ny volumes re degrded nd you believe tht this might be becuse the prtner node in the I/O group hs been powered off recently, wit until refresh of the screen shows ll volumes online. All the volumes should be online within 30 minutes of the prtner node being powered off. Note: After witing 30 minutes, if you hve degrded volume nd ll of the ssocited nodes nd MDisks re online, contct the IBM Support Center for ssistnce. Ensure tht ll volumes tht re used by hosts re online before continuing. c. If possible, check tht ll hosts tht ccess volumes mnged by this I/O group re ble to fil over to use pths tht re provided by the other node in the group. Perform this check using the multipthing device driver softwre of the host system. Commnds to use differ, depending on the multipthing device driver being used. If you use the System Storge Multipth Subsystem Device Driver (SDD), the commnd to query pths is dtpth query device. It cn tke some time for the multipthing device drivers to rediscover pths fter node is powered on. If you re unble to check on the host tht ll pths to both nodes in the I/O group re vilble, do not power off node within 30 minutes of the prtner node being powered on or you might lose ccess to the volume. d. If you decide tht it is oky to continue with powering off the nodes, select the node to power off nd click Shut Down System. e. Click OK. If the node tht you select is the lst remining node tht provides ccess to volume, for exmple node tht contins flsh drives with unmirrored volumes, the Shutting Down Node-Force pnel is displyed with list of volumes tht will go offline if the node is shut down. f. Check tht no host pplictions ccess the volumes tht re going offline. Continue with the shut down only if the loss of ccess to these volumes is cceptble. To continue with shutting down the node, click Force Shutdown. 300 SAN Volume Controller: Troubleshooting Guide

319 Wht to do next During the shut down, the node sves its dt structures to its locl disk nd destges ll write dt held in cche to the SAN disks. Such processing cn tke severl minutes. At the end of this processing, the system powers off. Using the SAN Volume Controller CLI to power off node Use the commnd-line interfce (CLI) to power off node. Procedure 1. Issue the lsnode CLI commnd to disply list of nodes in the system nd their properties. Find the node to shut down nd write down the nme of its I/O group. Confirm tht the other node in the I/O group is online. lsnode -delim : id:nme:ups_seril_number:wwnn:sttus:io_group_id: IO_group_nme:config_node: UPS_unique_id 1:group1node1:10L3ASH: C:online:0:io_grp0:yes: C0D18D8 2:group1node2:10L3ANF: :online:0:io_grp0:no: C0D1796 3:group2node1:10L3ASH: :online:1:io_grp1:no: C0D18D8 4:group2node2:10L3ANF: F4:online:1:io_grp1:no: C0D1796 If the node to power off is shown s Offline, the node is not prticipting in the system nd is not processing I/O requests. In such circumstnces, use the power button on the node to power off the node. If the node to power off is shown s Online, but the other node in the I/O group is not online, powering off the node impcts ll hosts tht re submitting I/O requests to the volumes tht re mnged by the I/O group. Ensure tht the other node in the I/O group is online before you continue. 2. Issue the lsdependentvdisks CLI commnd to list the volumes tht re dependent on the sttus of specified node. lsdependentvdisks group1node1 vdisk_id vdisk_nme 0 vdisk0 1 vdisk1 If the node goes offline or is removed from the system, the dependent volumes lso go offline. Before tking node offline or removing it from the system, you cn use the commnd to ensure tht you do not lose ccess to ny volumes. 3. If you decide tht it is oky to continue powering off the node, issue the stopsystem node <nme> CLI commnd to power off the node. Use the node prmeter to void powering off the whole system: stopsystem node group1node1 Are you sure tht you wnt to continue with the shut down? yes Note: To shut down the node even though there re dependent volumes, dd the -force prmeter to the stopsystem commnd. The force prmeter forces continution of the commnd even though ny node-dependent volumes will be tken offline. Use the force prmeter with cution; ccess to dt on node-dependent volumes will be lost. During the shut down, the node sves its dt structures to its locl disk nd destges ll write dt held in the cche to the SAN disks, which cn tke severl minutes. At the end of this process, the node powers off. Chpter 11. Using the mintennce nlysis procedures 301

320 Using the SAN Volume Controller power control button Do not use the power control button to power off node unless n emergency exists or nother procedure directs you to do so. Before you begin With this method, you cnnot check the system sttus from the front pnel, so you cnnot tell if the power off is lible to cuse excessive disruption to the system. Insted, use the mngement GUI or the CLI commnds, described in the previous topics, to power off n ctive node. About this tsk If you must use this method, notice in Figure 81 tht ech model type hs power control button 1 on the front DH CG CF A G4 Figure 81. Power control button on the SAN Volume Controller models When you determine it is sfe to do so, press nd immeditely relese the power button. On models other thn the 2145-DH8, the front pnel disply chnges to disply Powering Off nd displys progress br. Note: The 2145-DH8 is new design tht does not use front pnel disply. svc00716 The 2145-CG8 or the 2145-CF8 requires tht you remove power button cover before you cn press the power button. The A4 nd the G4 might require you to use pointed device to press the power button. If you press the power button for too long, the node immeditely powers down nd cnnot write ll dt to its locl disk. An extended service procedure is required to restrt the node, which involves deleting the node from the system before dding it bck. The following grphic shows how Powering Off is displyed on the front pnel of ll nodes but the 2145-DH8: 302 SAN Volume Controller: Troubleshooting Guide

321 Results MAP 5400: Front pnel The node sves its dt structures to disk while powering off. The power off process cn tke up to five minutes. When node is powered off by using the power button (or becuse of power filure), the prtner node in its I/O group immeditely stops using its cche for new write dt nd destges ny write dt lredy in its cche to the SAN ttched disks. The time tken by this destge depends on the speed nd utiliztion of the disk controllers. The time to complete is usully less thn 15 minutes, but it might be longer. The destging cnnot complete if there is dt witing to be written to disk tht is offline. A node tht powers off nd restrts while its prtner node continues to process I/O might not be ble to become n ctive member of the I/O group immeditely. The node must wit until the prtner node completes its destge of the cche. If the prtner node powers off during this period, ccess to the SAN storge tht is mnged by this I/O group is lost. If one of the nodes in the I/O group is unble to service ny I/O, for exmple becuse the prtner node in the I/O group is still flushing its write cche, volumes tht re mnged by tht I/O group hve sttus of Degrded. MAP 5400: Front pnel helps you to solve problems tht hve occurred on the front pnel. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. This MAP pplies to ll SAN Volume Controller models with front pnel disply. However the front pnel will only disply the sttus of the first four Fibre Chnnel ports; the service ssistnt GUI should be used if the node hs more thn four Fibre Chnnel ports. Be sure tht you know which model you re using before you strt this procedure. To determine which model you re working with, look for the lbel tht identifies the model type on the front of the node. You might hve been sent here becuse: v A problem occurred during the instlltion of SAN Volume Controller system, the front-pnel disply test filed, or the correct node number filed to be displyed v Another MAP sent you here About this tsk Perform the following steps: Procedure 1. Is the power LED on the opertor-informtion pnel illuminted nd showing solid green? Chpter 11. Using the mintennce nlysis procedures 303

322 NO Continue with the power MAP. See MAP 5050: Power 2145-CG8, 2145-CF8, nd G4 on pge 279 or MAP 5060: Power A4 on pge 285. YES Go to step (from step 1 on pge 303) Is the service controller error light 1 tht you see in Figure 82 illuminted nd showing solid mber? 1 svc00561 Figure 82. SAN Volume Controller service controller error light NO Strt the front pnel tests by pressing nd holding the select button for five seconds. Go to step 3. Attention: Do not strt this test until the node is powered on for t lest two minutes. You might receive unexpected results. YES The SAN Volume Controller service controller hs filed. v Replce the service controller. v Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from step 2) The front-pnel check light illumintes nd the disply test of ll disply bits turns on for 3 seconds nd then turns off for 3 seconds, then verticl line trvels from left to right, followed by horizontl line trvelling from top to bottom. The test completes with the switch test disply of single rectngle in the center of the disply. Did the front-pnel lights nd disply operte s described? NO SAN Volume Controller front pnel hs filed its disply test. v Replce the service controller. v Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Go to step (from step 3) Figure 83 on pge 305 provides four exmples of wht the front-pnel disply shows before you press ny button nd then when you press the up button, the left nd right buttons, nd the select button. To perform the front pnel switch test, press ny button in ny sequence or ny combintion. The disply indictes which buttons you pressed. 304 SAN Volume Controller: Troubleshooting Guide

323 Check ech switch in turn. Did the service pnel switches nd disply operte s described in Figure 83? NO The SAN Volume Controller front pnel hs filed its switch test. v Replce the service controller. v Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Press nd hold the select button for five seconds to exit the test. Go to step Is the front-pnel disply now showing Cluster:? NO Continue with MAP 5000: Strt on pge 265. YES Keep pressing nd relesing the down button until Node is displyed in line 1 of the menu screen. Go to step (from step 5) Is this MAP being used s prt of the instlltion of new node? NO Front-pnel tests hve completed with no fult found. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Go to step (from step 6) Is the node number tht is displyed in line 2 of the menu screen the sme s the node number tht is printed on the front pnel of the node? NO YES MAP 5500: Ethernet Figure 83. Front-pnel disply when push buttons re pressed Node number stored in front-pnel electronics is not the sme s tht printed on the front pnel. v Replce the service controller. v Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. Front-pnel tests hve completed with no fult found. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. MAP 5500: Ethernet helps you solve problems tht hve occurred on the SAN Volume Controller Ethernet. Chpter 11. Using the mintennce nlysis procedures 305

324 Before you begin Note: The service ssistnt GUI should be used if there is no front pnel disply, for exmple on the SAN Volume Controller 2145-DH8. If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. If you encounter problems with the 10 Gbps Ethernet feture on the SAN Volume Controller 2145-CG8 or SAN Volume Controller 2145-DH8, see MAP 5550: 10G Ethernet nd Fibre Chnnel over Ethernet personlity enbled dpter port on pge 309. You might hve been sent here for one of the following resons: v A problem occurred during the instlltion of SAN Volume Controller system nd the Ethernet checks filed v Another MAP sent you here v The customer needs immedite ccess to the system by using n lternte configurtion node. See Defining n lternte configurtion node on pge 309 About this tsk Perform the following steps: Procedure 1. Is the front pnel of ny node in the system displying Node Error with error code 805? YES Go to step 6 on pge 307. NO Go to step Is the system reporting error 1400 either on the front pnel or in the event log? YES Go to step 4. NO Go to step Are you experiencing Ethernet performnce issues? YES Go to step 9 on pge 308. NO Go to step 10 on pge (from step 2) On ll nodes perform the following ctions:. Press the down button until the top line of the disply shows Ethernet. b. Press right until the top line displys Ethernet port 1. c. If the second line of the disply shows link offline, record this port s one tht requires fixing. d. If the system is configured with two Ethernet cbles per node, press the right button until the top line of the disply shows Ethernet port 2 nd repet the previous step. e. Go to step (from step 4) Are ny Ethernet ports tht hve cbles ttched to them reporting link offline? YES Go to step 6 on pge 307. NO Go to step 10 on pge SAN Volume Controller: Troubleshooting Guide

325 6. (from step 5 on pge 306) Do the SAN Volume Controller nodes hve one or two cbles connected? One Go to step 7. Two Go to step (from step 6) Perform the following ctions:. Plug the Ethernet cble from tht node into the Ethernet port 2 from different node, s shown in Figure 84 nd Figure 85. b. If the Ethernet link light is illuminted when the cble is plugged into Ethernet port 2 of the other node, replce the system bord of the originl node svc00718 Figure 84. Port 2 Ethernet link LED on the SAN Volume Controller rer pnel 1 SAN Volume Controller 2145-CG8 port 2 (upper right) Ethernet link LED 2 SAN Volume Controller 2145-CF8 port 2 (upper right) Ethernet link LED 3 SAN Volume Controller G4 port 2 (center) Ethernet link LED 4 SAN Volume Controller A4 port 2 (upper right) Ethernet link LED Figure 85. Ethernet ports on the rer of the SAN Volume Controller 2145-DH8 2 1 Gbps Ethernet port 2 c. If the Ethernet link light does not illuminte, check the Ethernet switch or hub port nd cble to resolve the problem. d. Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from step 5 on pge 306 or step 6) Perform the following ctions:. Plug the Ethernet cble from tht node into nother device, for exmple, the SSPC. b. If the Ethernet link light is illuminted when the cble is plugged into the other Ethernet device, replce the system bord of the originl node. svc00861 Chpter 11. Using the mintennce nlysis procedures 307

326 c. If the Ethernet link light does not illuminte, check the Ethernet switch/hub port nd cble to resolve the problem. d. Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from step 3 on pge 306) Perform the following ctions:. Check ll Speed port 1 nd Speed port 2 pnels for the speed nd duplex settings. The formt is: <Speed>/<Duplex>. 1) Press the down button until the top line of the disply shows Ethernet. 2) Press the right button until the top line displys Speed 1. 3) If the second line of the disply shows link offline, record this port s one tht requires fixing. 4) If the system is configured with two Ethernet cbles per node, press the right button until the top line of the disply shows Speed 2 nd repet the previous step. b. Check tht the SAN Volume Controller port hs negotited t the highest speed vilble on the switch. All nodes hve gigbit Ethernet network ports. c. If the Duplex setting is hlf, perform the following steps: 1) There is known problem with gigbit Ethernet when one side of the link is set to fixed speed nd duplex nd the other side is set to utonegotite. The problem cn cuse the fixed side of the link to run t full duplex nd the negotited side of the link to run t hlf duplex. The duplex mismtch cn cuse significnt Ethernet performnce degrdtion. 2) If the switch is set to full duplex, set the switch to utonegotite to prevent the problem described previously. 3) If the switch is set to hlf duplex, set it to utonegotite to llow the link to run t the higher bndwidth vilble on the full duplex link. d. If none of the bove re true, cll your support center for ssistnce. 10. (from step 2 on pge 306) A previously reported fult with the Ethernet interfce is no longer present. A problem with the Ethernet might hve been fixed, or there might be n intermittent problem. Check with the customer to determine tht the Ethernet interfce hs not been intentionlly disconnected. Also check tht there is no recent history of fixed Ethernet problems with other components of the Ethernet network. Is the Ethernet filure explined by the previous checks? NO YES There might be n intermittent Ethernet error. Perform these steps in the following sequence until the problem is resolved:. Use the Ethernet hub problem determintion procedure to check for nd resolve n Ethernet network connection problem. If you resolve problem, continue with MAP 5700: Repir verifiction on pge 318. b. Determine if similr Ethernet connection problems hve occurred recently on this node. If they hve, replce the system bord. c. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. Verify the repir by continuing with MAP 5700: Repir verifiction on pge SAN Volume Controller: Troubleshooting Guide

327 Defining n lternte configurtion node A sitution cn rise where the customer needs immedite ccess to the system by using n lternte configurtion node. About this tsk If ll Ethernet connections to the configurtion node hve filed, the system is unble to report filure conditions, nd the mngement GUI is unble to ccess the system to perform dministrtive or service tsks. If this is the cse nd the customer needs immedite ccess to the system, you cn mke the system use n lternte configurtion node. Note: If the system hs no front pnel disply such s on SAN Volume Controller 2145-DH8, use the service ssistnt GUI. Service ssistnt is ccessed vi the technicin port. If only one node is displying Node Error 805 on the front pnel, perform the following steps: Procedure 1. Press nd relese the power button on the node tht is displying Node Error When Powering off is displyed on the front pnel disply, press the power button gin. 3. Restrting is displyed. Results The system will select new configurtion node. The mngement GUI is ble to ccess the system gin. MAP 5550: 10G Ethernet nd Fibre Chnnel over Ethernet personlity enbled dpter port MAP 5550: 10G Ethernet helps you solve problems tht occur on SAN Volume Controller 2145-CG8 or SAN Volume Controller 2145-DH8 with 10G Ethernet cpbility, nd Fibre Chnnel over Ethernet personlity enbled. Before you begin Note: The service ssistnt GUI might be used if there is no front pnel disply, for exmple on the SAN Volume Controller 2145-DH8. If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. This MAP pplies to the SAN Volume Controller 2145-CG8 nd SAN Volume Controller 2145-DH8 models with the 10G Ethernet feture instlled. Be sure tht you know which model you re using before you strt this procedure. To determine which model you re working with, look for the lbel tht identifies the model type on the front of the node. Check tht the 10G Ethernet dpter is instlled nd tht n opticl cble is ttched to ech port. Figure 18 on pge 21 shows the rer pnel of the 2145-CG8 with the 10G Ethernet ports. Chpter 11. Using the mintennce nlysis procedures 309

328 If you experience problem with error code 805, go to MAP 5500: Ethernet on pge 305. If you experience problem with error code 703 or 723, go to Fibre Chnnel nd 10G Ethernet link filures on pge 238. You might be sent here for one of the following resons: v A problem occurred during the instlltion of SAN Volume Controller system nd the Ethernet checks filed v Another MAP sent you here About this tsk Perform the following steps: Procedure 1. Is node error 720 or 721 displyed on the front pnel of the ffected node or is service error code 1072 shown in the event log? YES Go to step 11 on pge 311. NO Go to step (from step 1) Perform the following ctions from the front pnel of the ffected node:. Press nd relese the up or down button until Ethernet is shown. b. Press nd relese the left or right button until Ethernet port 3 is shown. Ws Ethernet port 3 found? No Go to step 11 on pge 311 Yes Go to step 3 3. (from step 2) Perform the following ctions from the front pnel of the ffected node:. Press nd relese the up or down button until Ethernet is shown. b. Press nd relese the up or down button until Ethernet port 3 is shown. c. Record if the second line of the disply shows Link offline, Link online, or Not configured. d. Press nd relese the up or down button until Ethernet port 4 is shown. e. Record if the second line of the disply shows Link offline, Link online, or Not configured. f. Go to step (from step 3) Wht ws the stte of the 10G Ethernet ports tht were seen in step 3? Both ports show Link online The 10G link is working now. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. One or more ports show Link offline Go to step 5 on pge 311. One or more ports show Not configured For informtion bout the port configurtion, see the CLI commnd cfgportip description in the SAN Volume Controller Informtion Center for iscsi. 310 SAN Volume Controller: Troubleshooting Guide

329 For Fibre Chnnel over Ethernet informtion, see the CLI commnd lsportfc description in the SAN Volume Controller Informtion Center. This commnd provides connection properties nd sttus to help determine whether the Fibre Chnnel over Ethernet is prt of correctly configured VLAN. 5. (from step 4 on pge 310) Is the mber 10G Ethernet link LED off for the offline port? YES Go to step 6 NO The physicl link is opertionl. The problem might be with the system configurtion. See the configurtion topic iscsi configurtion detils nd Fibre Chnnel over Ethernet configurtion detils in the SAN Volume Controller Informtion Center. 6. (from step 5) Perform the following ctions:. Check tht the 10G Ethernet ports re connected to 10G Ethernet fbric. b. Check tht the 10G Ethernet fbric is configured. c. Pull out the smll form-fctor pluggble (SFP) trnsceiver nd plug it bck in. d. Pull out the opticl cble nd plug it bck in e. Clen contcts with smll blst of ir, if vilble. f. Go to step (from step 6) Did the mber link LED light? YES The physicl link is opertionl. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. NO Go to step (from step 7) Swp the 10G SFPs in port 3 nd port 4, but keep the opticl cbles connected to the sme port. Is the mber link LED on the other port off now? YES Go to step 10. NO Go to step (from step 8) Swp the 10G Ethernet opticl cbles in port 3 nd port 4. Observe how the mber link LED chnges. Swp the cbles bck. Did the mber link LED on the other port go off? YES Check the 10G Ethernet opticl link nd fbric tht is connected to the port tht now hs the mber LED off. The problem is ssocited with the cble. The problem is either in the opticl cble or the Ethernet switch. Check tht the Ethernet switch shows tht the port is opertionl. If it does not show tht the port is opertionl, replce the opticl cble. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. NO Go to step (from step 8) Perform the following ctions:. Replce the SFP tht now hs the mber link LED off. b. Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from steps 1 on pge 310, 2 on pge 310, nd 9) Hve you lredy removed nd replced the 10G Ethernet dpter? YES Go to step 12 on pge 312. Chpter 11. Using the mintennce nlysis procedures 311

330 MAP 5600: Fibre Chnnel NO Perform the following ctions:. Remove nd replce the 10G Ethernet dpter. b. Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from steps 11 on pge 311) Replce the 10G Ethernet dpter with new one.. Replce the 10G Ethernet dpter. b. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. MAP 5600: Fibre Chnnel helps you to solve problems tht occur on the SAN Volume Controller Fibre Chnnel ports. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. This MAP pplies to ll SAN Volume Controller models with front pnel disply. However the front pnel will only disply the sttus of the first four Fibre Chnnel ports; the service ssistnt GUI should be used if the node hs more thn four Fibre Chnnel ports. Be sure tht you know which model you re using before you strt this procedure. To determine which model you re working with, look for the lbel tht identifies the model type on the front of the node. Note: Use the service ssistnt GUI if there is no front pnel disply, for exmple on the SAN Volume Controller 2145-DH8 where the service ssistnt GUI cn be ccessed vi the Technicin port. You might be sent here for one of the following resons: v A problem occurred during the instlltion of SAN Volume Controller system nd the Fibre Chnnel checks filed v Another MAP sent you here About this tsk Complete the following steps to solve problems tht re cused by the Fibre Chnnel ports: Procedure 1. Are you trying to resolve Fibre Chnnel port speed problem? NO Go to step 2. YES Go to step 11 on pge (from step 1) Disply Fibre Chnnel port 1 sttus on the SAN Volume Controller front-pnel disply or the service ssistnt GUI. For more informtion, see Chpter 6, Using the front pnel of the SAN Volume Controller, on pge 93. Is the front-pnel disply or the service ssistnt GUI on the SAN Volume Controller showing Fibre Chnnel port-1 ctive? 312 SAN Volume Controller: Troubleshooting Guide

331 NO A Fibre Chnnel port is not working correctly. Check the port sttus on the second line of the disply or the service ssistnt GUI. v v v Inctive: The port is opertionl but cnnot ccess the Fibre Chnnel fbric. The Fibre Chnnel dpter is not configured correctly; the Fibre Chnnel smll form-fctor pluggble (SFP) trnsceiver filed; the Fibre Chnnel cble tht is either filed or is not instlled; or the device t the other end of the cble filed. Mke note of port-1. Go to step 7 on pge 315. Filed: The port is not opertionl becuse of hrdwre filure. Mke note of port-1. Go to step 9 on pge 316. Not instlled: This port is not instlled. Mke note of port-1. Go to step 10 on pge 316. YES Press nd relese the right button to disply Fibre Chnnel port-2. Go to step (from step 2 on pge 312) Is the front pnel disply or the service ssistnt GUI on the SAN Volume Controller showing Fibre Chnnel port-2 ctive? NO A Fibre Chnnel port is not working correctly. Check the port sttus on the second line of the disply. v v v Inctive: The port is opertionl but cnnot ccess the Fibre Chnnel fbric. The Fibre Chnnel dpter is not configured correctly; the Fibre Chnnel smll form-fctor pluggble (SFP) trnsceiver filed; the Fibre Chnnel cble tht is either filed or is not instlled; or the device t the other end of the cble filed. Mke note of port-2. Go to step 7 on pge 315. Filed: The port is not opertionl becuse of hrdwre filure. Mke note of port-2. Go to step 9 on pge 316. Not instlled: This port is not instlled. Mke note of port-2. Go to step 10 on pge 316. YES Press nd relese the right button to disply Fibre Chnnel port-3. Go to step (from step 3) Is the front pnel disply or the service ssistnt GUI on the SAN Volume Controller showing Fibre Chnnel port-3 ctive? NO A Fibre Chnnel port is not working correctly. Check the port sttus on the second line of the disply. v v v Inctive: The port is opertionl but cnnot ccess the Fibre Chnnel fbric. The Fibre Chnnel dpter is not configured correctly; the Fibre Chnnel smll form-fctor pluggble (SFP) trnsceiver filed; the Fibre Chnnel cble tht is either filed or is not instlled; or the device t the other end of the cble filed. Mke note of port-3. Go to step 7 on pge 315. Filed: The port is not opertionl becuse of hrdwre filure. Mke note of port-3. Go to step 9 on pge 316. Not instlled: This port is not instlled. Mke note of port-3. Go to step 10 on pge 316. YES Press nd relese the right button to disply Fibre Chnnel port-4. Go to step (from step 4) Chpter 11. Using the mintennce nlysis procedures 313

332 Is the front pnel disply or the service ssistnt GUI on the SAN Volume Controller showing Fibre Chnnel port-4 ctive? NO A Fibre Chnnel port is not working correctly. Check the port sttus on the second line of the disply. v v v Inctive: The port is opertionl but cnnot ccess the Fibre Chnnel fbric. The Fibre Chnnel dpter is not configured correctly; the Fibre Chnnel smll form-fctor pluggble (SFP) trnsceiver filed; the Fibre Chnnel cble tht is either filed or is not instlled; or the device t the other end of the cble filed. Mke note of port-4. Go to step 7 on pge 315. Filed: The port is not opertionl becuse of hrdwre filure. Mke note of port-4. Go to step 8 on pge 315. Not instlled: This port is not instlled. Mke note of port-4. Go to step 10 on pge 316. YES If there re more thn four Fibre Chnnel ports on the node, repet step 5 on pge 313 for ech dditionl Fibre Chnnel port tht uses the service ssistnt. Go to step (from step 5 on pge 313) A previously reported fult with Fibre Chnnel port is no longer being shown. A problem with the SAN Fibre Chnnel fbric might be fixed or there might be n intermittent problem. Check with the customer to see whether ny Fibre Chnnel ports re disconnected or if ny component of the SAN Fibre Chnnel fbric filed nd ws recently fixed. Is the Fibre Chnnel port filure explined by the previous checks? NO There might be n intermittent Fibre Chnnel error.. Use the SAN problem determintion procedure to check for nd resolve ny Fibre Chnnel fbric connection problems. If you resolve problem, continue with MAP 5700: Repir verifiction on pge 318. b. Check whether similr Fibre Chnnel errors occurred recently on the sme port on this SAN Volume Controller node. If they hve, replce the Fibre Chnnel cble, unless it ws replced. c. Replce the Fibre Chnnel SFP trnsceiver, unless it ws replced. Note: SAN Volume Controller nodes re supported by both longwve SFP trnsceivers nd shortwve SFP trnsceivers. You must replce n SFP trnsceiver with the sme type of SFP trnsceiver. If the SFP trnsceiver to replce is longwve SFP trnsceiver, for exmple, you must provide suitble replcement. Removing the wrong SFP trnsceiver might result in loss of dt ccess. See the Removing nd replcing the Fibre Chnnel SFP trnsceiver on SAN Volume Controller node documenttion to find out how to replce n SFP trnsceiver. d. Replce the Fibre Chnnel dpter ssembly tht is shown in Tble 94 on pge SAN Volume Controller: Troubleshooting Guide

333 Tble 94. Fibre Chnnel ssemblies Node SAN Volume Controller 2145-CG8 port 1, 2, 3, or 4 (slot 1) SAN Volume Controller 2145-CG8 port 5, 6, 7, or 8 (slot 2) SAN Volume Controller 2145-CF8 port 1, 2, 3, or 4 SAN Volume Controller A4 port 1, 2, 3, or 4 SAN Volume Controller G4 port 1, 2, 3, or 4 SAN Volume Controller 2145-DH8 port 1, 2, 3, or 4 (slot 1 mndtory; the first FC dpter) SAN Volume Controller 2145-DH8 port 5, 6, 7, or 8 (slot 2 optionl; the second FC dpter) SAN Volume Controller 2145-DH8 port 9, 10, 11, or 12 (slot 5 optionl; the third FC dpter) Adpter ssembly Four-port Fibre Chnnel HBA Four-port Fibre Chnnel HBA Four-port Fibre Chnnel HBA Four-port Fibre Chnnel HBA Four-port Fibre Chnnel HBA Four-port Fibre Chnnel dpter Four-port Fibre Chnnel dpter Four-port Fibre Chnnel dpter e. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from steps 2 on pge 312, 3 on pge 313, 4 on pge 313, nd 5 on pge 313) The port noted on the SAN Volume Controller is showing sttus of inctive. For certin models, this inctive sttus might occur when the Fibre Chnnel speed is not set correctly. 8. (from step 7) The noted port on the SAN Volume Controller displys sttus of inctive. If the noted port still displys sttus of inctive, replce the prts tht re ssocited with the noted port until the problem is fixed in the following order:. Fibre Chnnel cbles from the SAN Volume Controller to Fibre Chnnel network. b. Fulty Fibre Chnnel fbric connections, prticulrly the SFP trnsceiver t the Fibre Chnnel switch. Use the SAN problem determintion procedure to resolve ny Fibre Chnnel fbric connection problem. c. SAN Volume Controller Fibre Chnnel SFP trnsceiver. Note: SAN Volume Controller nodes re supported by both longwve SFPs nd shortwve SFPs. You must replce n SFP with the sme type of SFP trnsceiver tht you re replcing. If the SFP trnsceiver to replce is longwve SFP trnsceiver, for exmple, you must provide suitble replcement. Removing the wrong SFP trnsceiver might result in loss of dt ccess. See the Removing nd replcing the Fibre Chnnel SFP trnsceiver on SAN Volume Controller node documenttion to find out how to replce n SFP trnsceiver. d. Replce the Fibre Chnnel dpter ssembly s shown in Tble 94. Chpter 11. Using the mintennce nlysis procedures 315

334 e. Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from steps 2 on pge 312, 3 on pge 313, 4 on pge 313, nd 5 on pge 313) The noted port on the SAN Volume Controller displys sttus of filed. Verify tht the Fibre Chnnel cbles tht connect the SAN Volume Controller nodes to the switches re securely connected. Replce the prts tht re ssocited with the noted port until the problem is fixed in the following order:. Fibre Chnnel SFP trnsceiver. Note: SAN Volume Controller nodes re supported by both longwve SFP trnsceivers nd shortwve SFP trnsceivers. You must replce n SFP trnsceiver with the sme type of SFP trnsceiver. If the SFP trnsceiver to replce is longwve SFP trnsceiver, for exmple, you must provide suitble replcement. Removing the wrong SFP trnsceiver might result in loss of dt ccess. See the Removing nd replcing the Fibre Chnnel SFP trnsceiver on SAN Volume Controller node documenttion to find out how to replce n SFP trnsceiver. b. Replce the Fibre Chnnel dpter ssembly s shown in Tble 94 on pge 315. c. Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from steps 2 on pge 312, 3 on pge 313, 4 on pge 313, nd 5 on pge 313) The noted port on the SAN Volume Controller displys sttus of not instlled. If you replced the Fibre Chnnel dpter, mke sure tht it is instlled correctly. If you replced ny other system bord components, mke sure tht the Fibre Chnnel dpter ws not disturbed. Is the Fibre Chnnel dpter filure explined by the previous checks? NO. Replce the Fibre Chnnel dpter ssembly s shown in Tble 94 on pge 315. b. If the problem is not fixed, replce the Fibre Chnnel connection hrdwre in the order tht is shown in Tble 95. Tble 95. SAN Volume Controller Fibre Chnnel dpter connection hrdwre Node SAN Volume Controller 2145-CG8 port 1, 2, 3, or 4 (slot 1) SAN Volume Controller 2145-CG8 port 5, 6, 7, or 8 (slot 2) SAN Volume Controller A4 port 1, 2, 3, or 4 SAN Volume Controller G4 port 1, 2, 3, or 4 Adpter connection hrdwre 1. PCI Express FC with Riser crd ssembly 1 2. System bord 1. PCI Express FC with Riser crd ssembly 2 2. System bord 1. Riser crd, PCI Express 2. System bord 1. Riser crd, PCI Express 2. System bord SAN Volume Controller 2145-DH8 port PCI Express Riser crd ssembly 1 2. System bord 316 SAN Volume Controller: Troubleshooting Guide

335 Tble 95. SAN Volume Controller Fibre Chnnel dpter connection hrdwre (continued) Node Adpter connection hrdwre SAN Volume Controller 2145-DH8 port PCI Express Riser crd ssembly 2 2. System bord c. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Verify the repir by continuing with MAP 5700: Repir verifiction on pge (from step 1 on pge 312) For the SAN Volume Controller models A4 nd2145-8g4 ech Fibre Chnnel port utonegotites its operting speed with the switch to which it is connected. If the operting speed is lower thn the operting speed tht is supported by the switch, high number of link errors re being detected. To disply the current speed of the link, complete the following steps, or use the service ssistnt GUI:. Press the up or down button on the front pnel until FC Port-1 Sttus: is displyed. b. Press nd relese the select button. c. Press the left or right button until FC Port-1 Speed: is displyed. d. Press nd relese the select button. e. Press the down button. The second line of the front-pnel disply shows the current Fibre Chnnel speed of the port. Is the port operting t lower thn the expected speed? NO YES Repet the check with the other Fibre Chnnel ports until the filing port is locted. If no filing port is locted, the problem no longer exists. Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. Perform the following steps:. Check the routing of the Fibre Chnnel cble to ensure tht no dmge exists nd tht the cble route contins no tight bends (no less thn 3-inch rdius). Either reroute or replce the Fibre Chnnel cble. b. Remove the Fibre Chnnel cble for 2 seconds nd then reinsert it to force the Fibre Chnnel dpter to renegotite its operting speed. c. Recheck the speed of the Fibre Chnnel port. If it is now correct, the problem is resolved. Otherwise, the problem might be cused by one of the following conditions: v Four-port Fibre Chnnel HBA v SAN Volume Controller SFP trnsceiver v Fibre Chnnel switch gigbit interfce converter (GBIC) or SFP trnsceiver v Fibre Chnnel switch Recheck the speed fter you chnge ny component until the problem is resolved nd then verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. Chpter 11. Using the mintennce nlysis procedures 317

336 MAP 5700: Repir verifiction MAP 5700: Repir verifiction helps you to verify tht field-replceble units (FRUs) tht you hve exchnged for new FRUs, or repir ctions tht hve been done hve solved ll the problems on the SAN Volume Controller. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. You might hve been sent here becuse you performed repir nd wnt to confirm tht no other problems exists on the mchine. Procedure 1. Are the Power LEDs on ll the nodes on? For more informtion bout this LED, see Power LED on pge 19. NO Go to MAP 5000: Strt on pge 265. YES Go to step (from step 1) Are ll the nodes displying Cluster: or is the node sttus LED on? NO Go to MAP 5000: Strt on pge 265. YES Go to step (from step 2) Using the SAN Volume Controller ppliction for the system you hve just repired, check the sttus of ll configured mnged disks (MDisks). Do ll MDisks hve sttus of online? NO If ny MDisks hve sttus of offline, repir the MDisks. Use the problem determintion procedure for the disk controller to repir the MDisk fults before returning to this MAP. If ny MDisks hve sttus of degrded pths or degrded ports, repir ny storge re network (SAN) nd MDisk fults before returning to this MAP. If ny MDisks show sttus of excluded, include MDisks before returning to this MAP. Go to MAP 5000: Strt on pge 265. YES Go to step (from step 3) Using the SAN Volume Controller ppliction for the system you hve just repired, check the sttus of ll configured volumes. Do ll volumes hve sttus of online? NO Go to step 5. YES Go to step 6 on pge (from step 4) Following repir of the SAN Volume Controller, number of volumes re showing sttus of offline. Volumes will be held offline if SAN Volume Controller cnnot confirm the integrity of the dt. The volumes might be the trget of copy tht did not complete, or cche write dt tht ws not written 318 SAN Volume Controller: Troubleshooting Guide

337 MAP 5800: Light pth bck to disk might hve been lost. Determine why the volume is offline. If the volume ws the trget of copy tht did not complete, you cn strt the copy gin. Otherwise, write dt might not hve been written to the disk, so its stte cnnot be verified. Your site procedures will determine how dt is restored to known stte. To bring the volume online, you must move ll the offline disks to the recovery I/O group nd then move them bck to n ctive I/O group. Go to MAP 5000: Strt on pge (from step 4 on pge 318) You hve successfully repired the SAN Volume Controller. MAP 5800: Light pth helps you to solve hrdwre problems on ll the SAN Volume Controller models tht re preventing the node from booting. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. You might be sent here becuse of the following situtions: v The Error LED on the opertor-informtion pnel is on or flshing v Another MAP sent you here: Light pth for SAN Volume Controller 2145-DH8 MAP 5800: Light pth Light pth for SAN Volume Controller 2145-CF8 on pge 326 Light pth for SAN Volume Controller A4 on pge 331 Light pth for SAN Volume Controller G4 on pge 334 Light pth for SAN Volume Controller 2145-DH8 Light pth dignostics is system of LEDs on top of the opertor-informtion pnel of the SAN Volume Controller 2145-DH8 node, which leds you to the filed component. About this tsk When n error occurs, LEDs re lit long the front of the opertor-informtion pnel, the light pth dignostics pnel, then on the filed component. By viewing the LEDs in prticulr order, you cn often identify the source of the error. LEDs tht re lit to indicte n error remin lit when the server is turned off, if the node is connected to n operting power supply. Ensure tht the node is turned on, nd then resolve ny hrdwre errors tht re indicted by the Error LED nd light pth LEDs: Procedure 1. Is the System error LED 7, shown in Figure 86 on pge 320, on the SAN Volume Controller 2145-DH8 opertor-informtion pnel on or flshing? Chpter 11. Using the mintennce nlysis procedures 319

338 Figure 86. SAN Volume Controller 2145-DH8 opertor-informtion pnel 1 Power control button nd LED 2 Ethernet LED 3 Loctor button nd LED 4 Relese ltch 5 Ethernet ctivity LEDs 6 Check log LED 7 System error LED NO Ressess your symptoms nd return to MAP 5000: Strt on pge 265. YES Go to step (from step 1 on pge 319) Press the relese ltch, s shown in Figure 87, nd open the light pth dignostics pnel, which is shown in Figure 88 on pge 321. ifs00064 Opertor informtion pnel Light pth dignostics LEDs Relese ltch Figure 87. Press the relese ltch Are one or more LEDs on the light pth dignostics pnel on or flshing? 320 SAN Volume Controller: Troubleshooting Guide

339 Checkpoint Code Remind Reset Light Pth Dignostics Figure 88. SAN Volume Controller 2145-DH8 light pth dignostics pnel NO YES Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If the error LED is still illuminted but no LEDs re illuminted on the light pth dignostics pnel, replce prts in the following sequence:. Opertor-informtion pnel b. System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. See Tble 96 on pge 322 nd perform the ction tht is specified for the specific light pth dignostics LEDs. Then, go to step 3 on pge 326. Some ctions require tht you observe the stte of LEDs on the system bord. Figure 89 on pge 322 shows the loction of the system bord LEDs. The fn LEDs re locted next to ech FAN. To view the LEDs, complete the following ctions:. Turn off the node while ensuring tht its dt is mirrored nd synchronized. b. Identify nd lbel ll the cbles tht re ttched to the node so tht they cn be replced in the sme port. Remove the node from the rck nd plce it on flt, sttic-protective surfce. For more informtion, see Removing the node from rck. c. Remove the top cover. d. See Tble 96 on pge 322 nd perform the ction tht is specified for the specific light pth dignostics LEDs. Then, go to step 3 on pge 326. Chpter 11. Using the mintennce nlysis procedures 321

340 System Error LED Loctor LED Power LED Enclosure mngement hertbet LED Imm2 hertbet LED Stndby power LED 10G Ethernet crd error LED Bttery error LED DIMM error LED (under the ltches) Microprocessor 2 error LED DIMM 1-6 error LED (under the ltches) Microprocessor 1 error LED Tble 96. Dignostics pnel LED LED Description Action The Error log or Check log LED opertorinformtion pnel Fn 4 error LED An error occurred nd cnnot be isolted without performing certin procedures. opertorinformtion pnel Systemerror An error occurred. LED Fn3 error LED DIMM 7-18 Fn2 error LED error LED (under the ltches) Figure 89. SAN Volume Controller 2145-DH8 system bord LEDs System bord error LED Fn1 error LED 1. Plug in the VGA screen nd the USB keybord. 2. Check the IMM2 system event log nd the system-error log for informtion bout the error. 3. Sve the log if necessry nd cler the log fterwrd. 1. Check the light pth dignostics LEDs nd follow the instructions. 2. Check the IMM2 system event log nd the system-error log for informtion bout the error. 3. Sve the log if necessry nd cler the log fterwrd. 322 SAN Volume Controller: Troubleshooting Guide

341 Tble 96. Dignostics pnel LED (continued) LED Description Action PS When only the PS LED is lit, power supply filed. The system might detect power supply error. Complete the following steps to correct the problem: 1. Check the power-supply with lit yellow LED. 2. Mke sure tht the power supplies re seted correctly nd plugged in good AC outlet. 3. Remove one of the power supplies to isolte the filed power supply. 4. Mke sure tht both power supplies instlled in the server re of the sme AC input voltge. 5. Replce the filed power supply. OVER SPEC PCI NMI PS + CONFIG When both the PS nd CONFIG LEDs re lit, the power supply configurtion is invlid. The system consumption reches the power supply over-current protection point or the power supplies re dmged. An error occurred on PCI bus or on the system bord. An dditionl LED is lit next to filing PCI slot. A nonmskble interrupt occurred, or the NMI button ws pressed. If the PS LED nd the CONFIG LED re lit, the system issues n invlid power configurtion error. Mke sure tht both power supplies instlled in the node re of the sme rting or wttge. 1. If the Pwr Ril (A, B, C, D, E, F, G, nd H) error ws not detected, complete the following steps:. Use the IBM Power Configurtor utility to determine current system power consumption. For more informtion nd to downlod the utility, go to bldecenter/ resources/powerconfig.html. b. Replce the filed power supply. 2. If the Pwr Ril (A, B, C, D, E, F, G, nd H) error ws lso detected, follow ctions tht re listed in MAP 5040: Power. 1. Check the riser-crd LEDs, the ServeRAID error LED, nd the dul-port network dpter error LED to identify the component tht cused the error. 2. Check the system-error log for informtion bout the error. 3. If you cnnot isolte the filing component by using the LEDs nd the informtion in the system-error log, remove one component t time. Then, restrt the server fter ech component is removed. 4. Replce the following components, in the order tht is shown, restrting the server ech time: v PCI riser crds v ServeRAID dpter v Network dpter v (Trined technicin only) System bord. 5. If the filure remins, contct your IBM service representtive. 1. Check the system-error log for informtion bout the error. 2. Restrt the server. Chpter 11. Using the mintennce nlysis procedures 323

342 Tble 96. Dignostics pnel LED (continued) LED Description Action CONFIG CONFIG + PS An invlid power configurtion error occurred. If the CONFIG LED nd the PS LED re lit, the system issues n invlid power configurtion error. Mke sure tht both power supplies instlled in the server re of the sme rting or wttge. LINK CPU CONFIG + CPU A hrdwre configurtion error occurred. CONFIG + MEM A hrdwre configurtion error occurred. CONFIG + PCI A hrdwre configurtion error occurred. CONFIG + HDD A hrd drive error occurred. Reserved. When only the CPU LED is lit, microprocessor hs filed. When both the CPU nd CONFIG LEDs re lit, the microprocessor configurtion is invlid. If the CONFIG LED nd the CPU LED re lit, complete the following steps to correct the problem: 1. Check the microprocessors tht were instlled to mke sure tht they re comptible with ech other. 2. (Trined technicin only) Replce the incomptible microprocessor. 3. Check the system-error logs for informtion bout the error. Replce ny component tht is identified in the error log. If the CONFIG LED nd the MEM LED re lit, check the system-event log in the Setup utility or IMM2 error messges. If the CONFIG LED nd the PCI LED re lit, check the system-error logs for informtion bout the error. Replce ny component tht is identified in the error log. If the CONFIG LED nd the HDD LED re lit, check the system-error logs for informtion bout the error. Replce ny component tht is identified in the error log. 1. If the CONFIG LED is not lit, microprocessor filure occurs, complete the following steps:. (Trined technicin only) Mke sure tht the filing microprocessor nd its het sink, which re indicted by lit LED on the system bord, re instlled correctly. b. (Trined technicin only) Replce the filing microprocessor. c. For more informtion, contct your IBM service representtive. 2. If the CONFIG LED nd the CPU LED re lit, the system issues n invlid microprocessor configurtion error. Complete the following steps to correct the problem:. Check the microprocessors tht were just instlled to mke sure tht they re comptible with ech other. b. (Trined technicin only) Replce the incomptible microprocessor. c. Check the system-error logs for informtion bout the error. Replce ny component tht is identified in the error log. 324 SAN Volume Controller: Troubleshooting Guide

343 Tble 96. Dignostics pnel LED (continued) LED Description Action MEM When only the MEM LED is lit, memory error occurred Note: Note: Ech time tht you instll or remove DIMM, you must disconnect the node from the power source; then, wit 10 seconds before you restrt the server. If the CONFIG LED is not lit, the system might detect memory error. Complete the following steps to correct the problem: 1. Updte the node firmwre. 2. Reset or swp the DIMMs with lit LED. 3. Check the system-event log in the Setup utility or IMM error messges. 4. Replce the filing DIMM. TEMP FAN BOARD MEM + CONFIG When both the MEM nd CONFIG LEDs re lit, the memory configurtion is invlid. The system or the system component temperture exceeded threshold level. A filing fn cn cuse the TEMP LED to be lit. A fn filed, is operting too slowly, or is removed. The TEMP LED might lso be lit. An error occurred on the system bord or the system bttery. If the MEM LED nd the CONFIG LED re lit, check the system-event log in the Setup utility or IMM2 error messges. 1. Mke sure tht the het sink is seted correctly. 2. Determine whether fn filed nd replce the fn if necessry. 3. Mke sure tht the room temperture is not too high. See the environment requirements for the server temperture informtion. 4. Mke sure tht the ir vents re not blocked. 5. Mke sure tht the het sink or the fn on the dpter, or the optionl network dpter is seted correctly. If the fn filed, replce it. 6. For more informtion, contct your IBM service representtive. 1. Check whether your node is instlled with the dul-port network dpter. If yes, mke sure tht your node compiles with the configurtion with four fns instlled. 2. Reset the filing fn, which is indicted by lit LED ner the fn connector on the system bord. 3. Replce the filing fn. 1. Check the LEDs on the system bord to identify the component tht cused the error. The BOARD LED cn be lit due to ny of the following resons: v Bttery v (Trined technicin only) System bord 2. Check the system-error log for informtion bout the error. 3. Replce the filing component. Chpter 11. Using the mintennce nlysis procedures 325

344 Tble 96. Dignostics pnel LED (continued) LED Description Action HDD A hrd disk drive tht is filed or is missing. 1. Check the LEDs on the hrd disk drives for the drive with lit sttus LED nd reset the hrd disk drive. 2. Reset the hrd disk drive bckplne. 3. If the error remins, replce the following components one t time, in the order tht is listed, restrting the server fter ech:. Replce the hrd disk drive. b. Replce the hrd disk drive bckplne. 4. If the problem remins, contct your IBM service representtive. 3. Continue with MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. Light pth for SAN Volume Controller 2145-CF8 Use the dignostics LEDs tht re on the system bord to solve hrdwre problems with thesan Volume Controller 2145-CF8 node. About this tsk Ensure tht the node is turned on, nd then perform the following steps to resolve ny hrdwre errors tht re indicted by the Error LED nd light pth LEDs: Procedure 1. Is the Error LED, shown in Figure 90, on the SAN Volume Controller 2145-CF8 opertor-informtion pnel on or flshing? svc_bb1gs Figure 90. SAN Volume Controller 2145-CG8 or 2145-CF8 opertor-informtion pnel 5 System error LED 6 Relese ltch NO Ressess your symptoms nd return to MAP 5000: Strt on pge 265. YES Go to step (from step 1) Press the relese ltch nd open the light pth dignostics pnel, which is shown in Figure 91 on pge 327. Are one or more LEDs on the light pth dignostics pnel on or flshing? 326 SAN Volume Controller: Troubleshooting Guide

345 OVERSPEC LOG LINK PS PCI SP REMIND FAN TEMP MEM NMI CNFG CPU VRM DASD RAID BRD Light Pth Dignostics RESET Figure 91. SAN Volume Controller 2145-CG8 or 2145-CF8 light pth dignostics pnel NO YES Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If the error LED is still illuminted but no LEDs re illuminted on the light pth dignostics pnel, replce prts in the following sequence:. Opertor-informtion pnel b. System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. See Tble 97 on pge 329 nd perform the ction tht is specified for the specific light pth dignostics LEDs. Then, go to step 3 on pge 331. Some ctions require tht you observe the stte of LEDs on the system bord. Figure 92 on pge 328 shows the loction of the system bord LEDs. The fn LEDs re locted next to ech FAN. To view the LEDs, perform the following ctions:. Turn off the node while ensuring tht its dt is mirrored nd synchronized. For more informtion, see MAP 5350: Powering off SAN Volume Controller node on pge 298. b. Identify nd lbel ll the cbles tht re ttched to the node so tht they cn be replced in the sme port. Remove the node from the rck nd plce it on flt, sttic-protective surfce. For more informtion, see Removing the node from rck. c. Remove the top cover. d. See Tble 97 on pge 329 nd perform the ction tht is specified for the specific light pth dignostics LEDs. Then, go to step 3 on pge 331. Chpter 11. Using the mintennce nlysis procedures 327

346 Figure 92. SAN Volume Controller 2145-CF8 system bord LEDs dignostics pnel 1 Slot 2 missing PCI riser crd LED 2 Enclosure mnger hertbet LED 3 Bttery LED 4 IMM hertbet LED 5 Slot 1 missing PCI riser crd LED 6 System error LED 7 Microprocessor 1 error LED 8 DIMM 1-8 error LEDs 9 Fn 1 error LED 10 Fn 2 error LED 11 Fn 3 error LED 12 Fn 4 error LED 13 Fn 5 error LED 14 Fn 6 error LED VA error LED 16 Power chnnel A error LED 328 SAN Volume Controller: Troubleshooting Guide

347 17 Power chnnel B error LED 18 Power chnnel C error LED 19 Power chnnel D error LED 20 Power chnnel E error LED 21 AUX power chnnel error LED 22 SAS/SATA RAID error LED 23 Microprocessor 2 error LED 24 DIMM 9-16 error LEDs Tble 97. Dignostics pnel LED prescribed ctions Dignostics pnel LED OVER SPEC LOG LINK PS Action The power supplies re using more power thn their mximum rting. If the OVER SPEC LED is lit, one or more of the six 12 V chnnel error LEDs (A, B, C, D, E, or AUX) is lso lit on the system bord. Perform the following ctions to resolve the problem: 1. Turn off the node, pull the node forwrd in the rck, nd remove the cover. Do not disconnect power from the node. 2. Check which 12 V chnnel error LED is lit on the system bord, nd remove the components tht re listed for tht LED: v LED A: fns, disk drive, ny flsh drives, or disk bckplne v LED B: Fibre Chnnel dpter nd riser, ll memory v LED C: disk controller, ll memory v LED D: microprocessor v LED E: High-speed SAS dpter nd riser, if instlled v LED AUX: Fibre Chnnel dpter nd high-speed SAS dpter, if instlled 3. Restrt the node to see whether the problem remins. 4. Reinstll ech device one t time tht you removed for the LED problems. Strt the node ech time to isolte the filing device. 5. Replce ny filing device. 6. If no device ws isolted, nd if LED C or LED D is lit, turn off the node nd remove the microprocessor. You need lcohol wipes nd therml grese to replce the microprocessor. Power on the server by toggling switch block 3 (SW3) bit 6. Restrt the server. Restrt the server. If the problem is resolved, replce the microprocessor; otherwise, reinstll the microprocessor. In either cse, toggle switch block 3 (SW3) bit 6 bck to its originl position. 7. If no device ws isolted, nd if LED AUX is lit, turn off the node nd remove the opertor-informtion pnel. Power on the server by toggling switch block 3 (SW3) bit 6. Restrt the server. Restrt the server. If the problem ws resolved, replce the opertor-informtion pnel; otherwise, reinstll the opertor-informtion pnel. In either cse, toggle switch block 3 (SW3) bit 6 bck to its originl position. 8. If no filing device is isolted, replce the system bord. An error occurred. Connect keybord nd monitor. Check the IMM system event log nd the system event log for informtion bout the error. Replce ny components tht re identified in the event logs. This LED is not used on the SAN Volume Controller 2145-CF8. Replce the system bord. Power supply 1 or power supply 2 filed. Perform the following ctions to resolve the problem: 1. Check the power supply tht hs lit mber LED. 2. Mke sure tht the power supplies re seted correctly. 3. Remove one of the power supplies to isolte the filed power supply. 4. Replce the filed power supply. Chpter 11. Using the mintennce nlysis procedures 329

348 Tble 97. Dignostics pnel LED prescribed ctions (continued) Dignostics pnel LED Action PCI An error occurred on PCI bus or on the system bord. An dditionl LED is lit next to filing PCI slot. Perform the following ctions to resolve the problem: 1. Identify the filing dpter by checking the LEDs on the PCI slots. 2. If PCI slot 1 shows n error, replce the four-port Fibre Chnnel dpter ssembly. 3. If PCI slot 2 shows n error, replce the high-speed SAS dpter ssembly. 4. If the error is not resolved, replce the system bord. SP A service processor error ws detected. Perform the following ctions to resolve the problem: 1. Remove power from the node. Reconnect the server to the power, nd restrt the node. 2. If the problem remins, replce the system bord. FAN A fn filed, is operting too slowly, or ws removed. A filing fn cn lso cuse the TEMP LED to be lit. Perform the following ctions to resolve the problem: 1. Reset the filing fn, which is indicted by lit LED ner the fn connector on the system bord. 2. If the problem remins, replce the filing fn. TEMP The system temperture exceeded threshold level. A filing fn cn cuse the TEMP LED to be lit. Perform the following ctions to resolve the problem: 1. Mke sure tht the het sink is seted correctly. 2. Determine whether fn filed. If it hs, replce it. 3. Verify tht the mbient temperture is within norml operting specifictions. 4. Mke sure tht irflow in nd round the SAN Volume Controller 2145-CF8 is not obstructed. MEM A memory configurtion or memory error tht is not vlid occurred. Both the MEM LED nd CNFG LED might be lit. Perform the following ctions to resolve the problem: 1. Check tht ll the memory DIMMs re correctly instlled. 2. If ny memory error LEDs re lit, replce the indicted memory module. 3. If the MEM LED nd the CNFG LED re lit, djust the memory so tht DIMM slots 2, 3, 5, 6, 7, nd 8 re the only ones used. NMI A non-mskble interrupt occurred or the NMI button ws pressed. This sitution should not occur. If the NMI button on the light pth dignostic pnel ws pressed by mistke, restrt the node; otherwise, cll your support center. CNFG A hrdwre configurtion error occurred. If the MEM LED is lso lit, follow the ctions tht re shown for MEM LED. If the CPU LED is lit, check to see whether microprocessor is instlled in CPU 2. If one is instlled, remove it becuse the configurtion is not supported. If no other light pth LEDs re lit, replce the FRUs in the order tht is shown until the problem is resolved: 1. Opertor-informtion pnel 2. Opertor-informtion pnel cble 3. System bord CPU A microprocessor tht is filed, or microprocessor configurtion is not vlid. Both the CPU LED nd the CNFG LED might be lit. Perform the following ctions: 1. Check the system bord error LEDs. 2. If CPU 1 error LED is lit, check tht the microprocessor is correctly instlled. 3. If the error persists, replce the microprocessor. 4. If the error persists, replce the system bord. VRM This LED is not used on the SAN Volume Controller 2145-CF SAN Volume Controller: Troubleshooting Guide

349 Tble 97. Dignostics pnel LED prescribed ctions (continued) Dignostics pnel LED DASD Action A disk drive tht is filed or is missing. A SAN Volume Controller 2145-CF8 must hve its system hrd disk drive tht is instlled in drive slot 4. Up to four flsh drives cn be instlled in drive slots 0-3. If n Flsh drive is delibertely removed from slot, the system error LED nd the DASD dignostics pnel LED lights. The error is mintined even if the Flsh drive is replced in different slot. If n Flsh drive is removed or moved, the error is clered by powering off the node using MAP 5350, removing both the power cbles, replcing the power cbles, nd then restrting the node. Resolve ny node or system errors tht relte to Flsh drives or the system disk drive. If n error is still shown, power off the node nd reset ll the drives. If the error remins, replce the following components in the order listed: 1. The system disk drive 2. The disk bckplne RAID BRD This LED is not used on the SAN Volume Controller 2145-CF8. An error occurred on the system bord. Perform the following ctions to resolve the problem: 1. Check the LEDs on the system bord to identify the component tht cused the error. The BRD LED cn be lit becuse of ny of the following resons: v Bttery v Missing PCI riser-crd ssembly. There must be riser crd in PCI slot 2 even if the optionl dpter is not present. v Filed voltge regultor 2. Replce ny filed or missing replcement components, such s the bttery or PCI riser-crd ssembly. 3. If voltge regultor fils, replce the system bord. 3. Continue with MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. Light pth for SAN Volume Controller A4 Use the dignostics LEDs tht re on the system bord to solve hrdwre problems with the SAN Volume Controller A4 node. About this tsk Ensure tht the node is turned on nd then perform the following steps to resolve ny hrdwre errors tht re indicted by the Error LED nd light pth LEDs: Procedure 1. Is the Error LED, shown in Figure 93 on pge 332, on the SAN Volume Controller A4 opertor-informtion pnel on or flshing? Chpter 11. Using the mintennce nlysis procedures 331

350 Figure 93. SAN Volume Controller A4 opertor-informtion pnel 1 Error LED NO Ressess your symptoms nd return to MAP 5000: Strt on pge 265. YES Go to step (from step 1 on pge 331) Observe the stte of the dignostic LEDs on the system bord. To view the LEDs, follow these steps:. Turn off the node while ensuring tht its dt is mirrored nd synchronized. For more informtion, see MAP 5350: Powering off SAN Volume Controller node on pge 298. b. Identify nd lbel ll the cbles tht re ttched to the node so tht they cn be replced in the sme port. Remove the node from the rck nd plce it on flt, sttic-protective surfce. c. Remove the top cover. d. Turn on the node. 3. (from step 2) Other thn the Stndby Power, Power good, nd the Bsebord mngement controller hertbet LEDs, re one or more LEDs on the system bord on or flshing? NO Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If the error LED is still on but no error LEDs re illuminted on the system bord, replce prts in the following sequence:. Opertor-informtion pnel b. Opertor-informtion pnel cble c. System bord Go to step 5 on pge 334. YES Identify ny dignostic LEDs on the system bord tht re on. Figure 94 on pge 333 shows the loction of the system bord LEDs. The fn LEDs re locted next to ech fn. You cn ignore the three LEDs tht do not indicte n error: 13, 14, nd SAN Volume Controller: Troubleshooting Guide

351 Figure 94. SAN Volume Controller A4 system bord LEDs 1 Fn 1 error LED 2 Fn 2 error LED 3 Fn 3 error LED 4 DIMM 1 error LED 5 DIMM 2 error LED 6 DIMM 3 error LED 7 DIMM 4 error LED 8 PCI Express slot 2 error LED 9 PCI Express slot 1 error LED 10 Fn 4 error LED 11 Fn 5 error LED 12 Voltge regultor error LED 13 Stndby power LED 14 Power good LED 15 Bsebord mngement controller hertbet LED 16 SAS/SATA controller error LED 4. (from step 3 on pge 332) Are ny dignostic LEDs other thn 13, 14, nd 15 on the system bord illuminted? NO Go to step 5 on pge 334. YES See Tble 98 on pge 334 nd replce the prts tht re specified for the specific LEDs one-t--time in the following order until the error is repired. Then, go to step 5 on pge 334. Chpter 11. Using the mintennce nlysis procedures 333

352 Tble 98. SAN Volume Controller A4 dignostics pnel LED prescribed ctions Dignostics pnel LED DIMM error LEDs (1-4) Fn error LEDs (1-5) PCI Express slot 1 error LED PCI Express slot 2 error LED Voltge regultor error LED SAS/SATA controller error LED Action Replce prts in the following sequence: 1. Indicted DIMM 2. System bord Replce prts in the following sequence: 1. Indicted fn 2. System bord Replce prts in the following sequence: 1. PCI riser crd 2. System bord 3. Fibre Chnnel dpter This LED is not used on the SAN Volume Controller A4. Replce the system bord. Replce the system bord. This LED is not used on the SAN Volume Controller A4. Replce the system bord. 5. (from step 4 on pge 333) Replce the top cover nd plce the node in the rck. For more informtion, see Removing the node from rck. Then, continue with MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. Light pth for SAN Volume Controller G4 Use light pth dignostics to solve hrdwre problems with the SAN Volume Controller G4 node. About this tsk Ensure tht the node is turned on nd then perform the following steps to resolve ny hrdwre errors tht re indicted by the Error LED nd light pth LEDs: Procedure 1. Is the Error LED, shown in Figure 95, on the SAN Volume Controller G4 opertor-informtion pnel illuminted or flshing? 2 1 svc00230 Figure 95. SAN Volume Controller G4 opertor-informtion pnel 1 Relese ltch 2 Error LED NO Ressess your symptoms nd return to MAP 5000: Strt on pge SAN Volume Controller: Troubleshooting Guide

353 YES Go to step (from step 1 on pge 334) Press the relese ltch nd open the light pth dignostics pnel, which is shown in Figure 96. Light Pth Dignostics OVER SPEC PS1 PS2 REMIND CPU VRM CNFG MEM NMI S ERR SP DASD RAID FAN TEMP BRD PCI svc00224 Figure 96. SAN Volume Controller G4 light pth dignostics pnel Are one or more LEDs on the light pth dignostics pnel on or flshing? NO YES Verify tht the opertor-informtion pnel cble is correctly seted t both ends. If the error LED is still illuminted but no LEDs re illuminted on the light pth dignostics pnel, replce prts in the following sequence:. Opertor-informtion pnel b. System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. See Tble 99 on pge 337 nd perform the ction tht is specified for the specific light pth dignostics LEDs. Then, go to step 3 on pge 338. Some ctions require tht you observe the stte of LEDs on the system bord. Figure 97 on pge 336 shows the loction of the system bord LEDs. The fn LEDs re locted next to ech FAN. To view the LEDs, you need follow these steps:. Turn off the node while ensuring tht its dt is mirrored nd synchronized. For more informtion, see MAP 5350: Powering off SAN Volume Controller node on pge 298. b. Identify nd lbel ll the cbles tht re ttched to the node so tht they cn be replced in the sme port. Remove the node from the rck nd plce it on flt, sttic-protective surfce. For more informtion, see Removing the node from rck. c. Remove the top cover nd open the fn doors. d. Press the light pth dignostics button ( 7 in Figure 97 on pge 336). Note: The light pth dignostics button is used to illuminte the light pth dignostics LEDs when power is disconnected from the SAN Volume Controller G4 node. Chpter 11. Using the mintennce nlysis procedures 335

354 svc Figure 97. SAN Volume Controller G4 system bord LEDs 1 System-bord bttery error LED 2 DIMM 5 error LED 3 DIMM 6 error LED 4 DIMM 7 error LED 5 DIMM 8 error LED 6 Light pth dignostics ctive LED 7 Light pth dignostics button 8 Microprocessor 2 error LED 9 Microprocessor 1 error LED 10 DIMM 1 error LED 11 DIMM 2 error LED 12 DIMM 3 error LED 13 DIMM 4 error LED 14 System-bord fult LED 15 Power B error LED 16 Power A error LED 17 Power C error LED 18 Power D error LED 336 SAN Volume Controller: Troubleshooting Guide

355 Tble 99. Dignostics pnel LED prescribed ctions Dignostics pnel LED Action OVER SPEC Replce prts in the following sequence: 1. Power supply 2. Power bckplne 3. System bord PS1 If you replced the power supply, check tht it is correctly instlled. If it is correctly instlled, replce prts in the following sequence: 1. Power supply 2. Power bckplne PS2 This LED is not used on the SAN Volume Controller G4, nd is flse indiction. A sensor filed or the system bord service processor firmwre is not functioning correctly. Contct your support center to see whether firmwre updte is vilble. If not, follow this sequence to replce prts: 1. Power bckplne 2. Opertor-informtion pnel 3. System bord CPU A microprocessor filed. Mke sure tht the filing microprocessor, which is indicted by lit LED on the system bord, is instlled correctly. If it is instlled correctly, replce the microprocessor. VRM This LED is not used on the SAN Volume Controller G4. CNFG Microprocessor configurtion error. Check the instlled microprocessors for comptibility. MEM Observe the DIMM LEDs on the system bord. If ny DIMM LED is flshing, mke sure tht the correct type of DIMM is instlled in every slot. Replce prts in the following sequence: 1. Filing DIMM 2. System bord NMI S ERR SP DASD BRD FAN Note: If more thn one DIMM is indicted by the light pth dignostics, replce the DIMMs one-t--time, strting t the lowest-numbered DIMM slot tht the dignostics indicted. A non-mskble interrupt occurred. Cll your support center nd check if ny softwre updtes must be pplied to this SAN Volume Controller G4. If this node does not join the system, run node recovery. If node recovery does not resolve the problem, replce the system bord ssembly. A soft error occurred. Cll your support center nd check if ny softwre updtes must be pplied to this SAN Volume Controller G4. If this node does not join the system, run node recovery. If node recovery does not resolve the problem, replce the system bord ssembly. The Service processor filed. Replce the system bord ssembly. This LED is not used on the SAN Volume Controller G4. A sensor filed or the system bord service processor firmwre is not functioning correctly. Contct your support center to see whether firmwre updte is vilble. If not, follow this sequence to replce prts: 1. Opertor-informtion pnel 2. System bord Observe the bttery LED nd the system bord LED. If the bttery LED is illuminted, replce the bttery. If the system bord LED is illuminted, replce the system bord. A fn filed, is operting too slowly, or ws removed. A filing fn cn lso cuse the TEMP LED to be lit. Replce the filing fn, which is indicted by lit LED ner the fn connector on the system bord. Chpter 11. Using the mintennce nlysis procedures 337

356 Tble 99. Dignostics pnel LED prescribed ctions (continued) Dignostics pnel LED TEMP RAID PCI Action If ny fn filures exist, repir those fns before you ttempt this procedure. Verify tht the mbient temperture is within norml operting specifictions. Mke sure tht irflow in nd round the SAN Volume Controller G4 is not obstructed. If the error persists, replce the system bord. This LED is not used on the SAN Volume Controller G4. The Fibre Chnnel crd might be filing. Ensure tht the Fibre Chnnel crd nd the riser crd re correctly instlled. If the error persists, replce the Fibre Chnnel crd. MAP 5900: Hrdwre boot 3. Continue with MAP 5700: Repir verifiction on pge 318 to verify the correct opertion. MAP 5900: Hrdwre boot helps you solve problems tht re preventing the node from strting its boot sequence. Before you begin Note: Use the service ssistnt GUI if there is no front pnel disply, for exmple on the SAN Volume Controller 2145-DH8. If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. This MAP pplies to ll SAN Volume Controller models. However some models do not hve front pnel disply; the service ssistnt GUI should be used if the node does not hve front pnel disply. Be sure tht you know which model you re using before you strt this procedure. To determine which model you re working with, look for the lbel tht identifies the model type on the front of the node. You might hve been sent here for one of the following resons: v The hrdwre boot disply, shown in Figure 98, is displyed continuously. Figure 98. Hrdwre boot disply v The node rescue disply, shown in Figure 99, is displyed continuously. Figure 99. Node rescue disply v v v The boot progress is hung nd n error is displyed on the front pnel Another MAP sent you here The node sttus LED, node fult LED nd bttery sttus LED hve remined off 338 SAN Volume Controller: Troubleshooting Guide

357 About this tsk Perform the following steps to llow the node to strt its boot sequence: Procedure 1. Is the Error LED on the opertor-informtion pnel illuminted or flshing? NO Go to step 2. YES Go to MAP 5800: Light pth on pge 319 to resolve the problem. 2. (From step 1) If you hve just instlled the SAN Volume Controller node or hve just replced field replceble unit (FRU) inside the node, perform the following steps:. Identify nd lbel ll the cbles tht re ttched to the node so tht they cn be replced in the sme port. Remove the node from the rck nd plce it on flt, sttic-protective surfce. See the Removing the node from rck informtion to find out how to perform the procedure. b. Remove the top cover. See the Removing the top cover informtion to find out how to perform the procedure. c. If you hve just replced FRU, ensure tht the FRU is correctly plced nd tht ll connections to the FRU re secure. d. Ensure tht ll memory modules re correctly instlled nd tht the ltches re fully closed. See the Replcing the memory modules (DIMM) informtion to find out how to perform the procedure. e. Ensure tht the Fibre Chnnel dpters re correctly instlled. See the Replcing the Fibre Chnnel dpter ssembly informtion to find out how to perform the procedure. f. Ensure tht the disk drive nd its connectors re correctly instlled. See the Replcing the disk drive informtion to find out how to perform the procedure. g. Ensure tht the service controller is correctly instlled. See the Replcing the service controller informtion to find out how to perform the procedure. h. Replce the top cover. See the Replcing the top cover informtion to find out how to perform the procedure. i. Plce the node in the rck. See the Replcing the node in rck informtion to find out how to perform the procedure. j. Turn on the node. Does the boot opertion still hng? NO Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Go to step (from step 2) Check if the system BIOS is reporting ny errors. You need to ttch VGA disply nd USB keybord to see the BIOS output. The customer should be ble to supply suitble disply nd keybord.. Turn off the node while ensuring tht its dt is mirrored nd synchronized. See MAP 5350: Powering off SAN Volume Controller node on pge 298. b. Connect the keybord 1 nd the disply 2. Figure 100 on pge 340 shows the loction of the keybord nd monitor ports. Figure 101 on pge 340 Chpter 11. Using the mintennce nlysis procedures 339

358 340 shows the loction of the keybord nd monitor ports on the 2145-CF8. Figure 102 shows the loction of the keybord nd monitor ports on the 2145-CG8. Connect to the VGA port nd USB port on the front or the bck of 2145-DH8. Model type: G4 Model type: A svc00675 Figure 100. Keybord nd monitor ports on the SAN Volume Controller models G4 nd2145-8a4 2 1 svc00572 Figure 101. Keybord nd monitor ports on the SAN Volume Controller 2145-CF8 1 2 svc00723 Figure 102. Keybord nd monitor ports on the SAN Volume Controller 2145-CG8 340 SAN Volume Controller: Troubleshooting Guide

359 SAN Volume Controller svc USB port 1 nd 2 4 Video port Figure 103. Keybord nd monitor ports on the SAN Volume Controller 2145-DH8, front USB port 6 8 USB port 5 9 USB port 4 10 USB port 3 11 Seril port 12 Video port Figure 104. Keybord nd monitor ports on the SAN Volume Controller 2145-DH8, rer c. Turn on the node. d. Wtch the VGA disply. v If the POST sequence indictes n error, or if the BIOS Configurtion/Setup Utility progrm indictes n error during strtup, you need to resolve the error. v If it indictes n error with specific hrdwre item, power off the node nd remove it from the rck. Ensure the item specified is correctly svc00859 Chpter 11. Using the mintennce nlysis procedures 341

360 v instlled, replce the node in the rck, nd then restrt the node. If the error is still reported, replce the specified item. If configurtion error is reported, run the Configurtion/Setup Utility progrm option to reset the BIOS to its defult (fctory) settings. v If no boot imge cn be found on 2145-DH8 node then contct IBM Remote technicl support for help, which could led to instlling the softwre from USB flsh drive. e. Turn off the node nd remove the keybord nd disply. f. Turn on the node. Does the boot opertion still hng? NO Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Go to step (from step 3 on pge 339). Turn off the node while ensuring tht its dt is mirrored nd synchronized. See MAP 5350: Powering off SAN Volume Controller node on pge 298. b. Identify nd lbel ll the cbles tht re ttched to the node so tht they cn be replced in the sme port. Remove the node from the rck nd plce it on flt, sttic-protective surfce. See the Removing the node from rck informtion to find out how to perform the procedure. c. Remove the top cover. See the Removing the top cover informtion to find out how to perform the procedure. d. Remove some of the memory modules: v v In the SAN Volume Controller 2145-DH8, remove the memory modules in slots 1, 4, 9, nd 12, in ddition to slots 13, 16, 21 nd 24 if the second microprocessor is fitted. If you re using the SAN Volume Controller A4, remove the memory modules in slots 2 through 4. v If you re using the SAN Volume Controller G4, remove the memory modules in slots 2 nd 4 through 8. e. Remove ll instlled dpters. f. Remove the disk drive, unless this is 2145-DH8, which cn only boot from its boot drives. g. Replce the top cover. See the Replcing the top cover informtion to find out how to perform the procedure. h. Plce the node in the rck. See the Replcing the node in rck informtion to find out how to perform the procedure. i. Turn on the node. 5. Does the boot opertion still hng with the booting disply (perform the NO ction) or hs the boot opertion progressed (perform the YES ction)? Note: With the FRUs removed, the boot will hng with different boot filure code. NO Go to step 6 to replce the FRUs, one-t--time, until the filing FRU is isolted. YES Go to step 7 on pge (From step 5) 342 SAN Volume Controller: Troubleshooting Guide

361 Remove ll hrdwre except the hrdwre tht is necessry to power up. Continue to dd in the FRUs one t time nd power on ech time until the originl filure is introduced. Does the boot opertion still hng? NO Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Go to step (from steps 4 on pge 342 nd 6 on pge 342). Turn off the node. See MAP 5350: Powering off SAN Volume Controller node on pge 298 for more informtion. b. Identify nd lbel ll the cbles tht re ttched to the node so tht they cn be replced in the sme port. Remove the node from the rck nd plce it on flt, sttic-protective surfce. See the Removing the node in rck informtion to find out how to perform the procedure. c. Remove the top cover. See the Removing the top cover informtion to find out how to perform the procedure. d. Replce the dpter nd the disk drive. e. Replce the memory modules: v In the 2145-DH8, replce the memory modules in (the white slots) slots 1, 4, 9, nd 12, in ddition to slots 13, 16, 21 nd 24, if the second microprocessor is fitted. v If you re using the SAN Volume Controller A4, replce the memory module in slot 1 with ny of the removed memory modules from slots 2 through 4. v If you re using the SAN Volume Controller G4, replce the memory modules in slots 1 nd 3 with ny two of the removed memory modules from slots 2 nd 4 through 8. f. Replce the top cover. See the Replcing the top cover informtion to find out how to perform the procedure. g. Plce the node in the rck. See the Replcing the node in rck informtion to find out how to perform the procedure. h. Turn on the node. Does the boot opertion still hng with the booting disply (perform the NO ction) or does the disply progress beyond the initil booting pnel (perform the YES ction)? NO Exchnge the filing memory modules for new FRUs nd verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. YES Replce the prts in the following sequence: v For the SAN Volume Controller 2145-CG8, or 2145-CF8:. Service controller b. System bord v For the SAN Volume Controller A4 nd SAN Volume Controller G4:. Service controller b. System bord Verify the repir by continuing with MAP 5700: Repir verifiction on pge 318. Chpter 11. Using the mintennce nlysis procedures 343

362 MAP 6000: Replce offline SSD MAP 6000: This procedure replces flsh drive tht hs filed while it is still member of storge pool. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. This mp pplies to models with internl flsh drives. Be sure tht you know which model you re using before you strt this procedure. To determine which model you re working on, look for the lbel tht identifies the model type on the front of the node. About this tsk Use this MAP to determine which detiled MAP to use for replcing n offline SSD. Attention: If the drive use property is member nd the drive must be replced, contct IBM support before tking ny ctions. Procedure Are you using n SSD in RAID 0 rry nd using volume mirroring to provide redundncy? Yes Go to MAP 6001: Replce offline SSD in RAID 0 rry. No Go to MAP 6002: Replce offline SSD in RAID 1 rry or RAID 10 rry on pge 346. MAP 6001: Replce offline SSD in RAID 0 rry MAP 6001: This procedure replces flsh drive tht hs filed while it is still member of storge pool. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. This mp pplies to models with internl flsh drives. Be sure tht you know which model you re using before you strt this procedure. To determine which model you re working on, look for the lbel tht identifies the model type on the front of the node. Attention: 1. Bck up your SAN Volume Controller configurtion before you begin these steps. 2. If the drive use property is member nd the drive must be replced, contct IBM support before tking ny ctions. About this tsk Perform the following steps only if drive in RAID 0 (striped) rry hs filed: 344 SAN Volume Controller: Troubleshooting Guide

363 Procedure 1. Record the properties of ll volume copies, MDisks, nd storge pools tht re dependent on the filed drive.. Identify the drive ID nd the error sequence number with sttus equls offline nd use equls filed using the lsdrive CLI commnd. b. Review the offline reson using the lsevent <seq_no> CLI commnd. c. Obtin detiled informtion bout the offline drive or drives using the lsdrive <drive_id> CLI commnd. d. Record the mdisk_id, mdisk_nme, node_id, node_nme, nd slot_id for ech offline drive. e. Obtin the storge pools of the filed drives using the lsmdisk <mdisk_id> CLI commnd for ech MDisk tht ws identified in the substep 1c. Continue with the following steps by replcing ll the filed drives in one of the storge pools. Mke note of the node, slot, nd ID of the selected drives. f. Determine ll the MDisks in the storge pool using the lsmdisk -filtervlue mdisk_grp_id=<grp id> CLI commnd. g. Identify which MDisks re internl (ctrl_type equls 4) nd which MDisks contin SSDs (ctrl_type equls 6). h. Find the volumes with extents in the storge pool using the lsmdiskmember <mdisk_id> CLI commnd for ech MDisk found in substep 1f. It is likely tht the sme volumes will be returned for ech MDisk. i. Record ll the properties on ech volume listed in step 1h by using the lsvdisk <vdisk_id> CLI commnd. For ech volume check if it hs online volume copies which indicte it is mirrored. Use this informtion in step 9 on pge 346. j. Obtin list of ll the drives in ech internl MDisk in the storge pool using the lsdrive -filtervlue mdisk_id=<mdisk_id> CLI commnd. Use this informtion in step 8 on pge 346. k. Record ll the properties of ll the MDisks in the storge pool using the lsmdisk <mdisk_id> CLI commnd. Use this informtion in step 8 on pge 346. l. Record ll the properties of the storge pool using the lsmdisk <mdisk_id> CLI commnd. Use this informtion in step 7 on pge 346. Note: If listed volume hs mirrored, online, nd in-sync copy, you cn recover the copied volume dt from the copy. All the dt on the unmirrored volumes will be lost nd will need to be restored from bckup. 2. Delete the storge pool using the rmmdiskgrp -force <mdiskgrp id> CLI commnd. All MDisks nd volume copies in the storge pool re lso deleted. If ny of the volume copies were the lst in-sync copy of volume, ll the copies tht re not in sync re lso deleted, even if they re not in the storge pool. 3. Using the drive ID tht you recorded in substep 1e, set the use property of the drive to unused using the chdrive commnd. chdrive -use unused <id of offline drive> The drive is removed from the drive listing. 4. Follow the physicl instructions to replce or remove drive. See the Replcing SAN Volume Controller 2145-CG8 flsh drive documenttion or the Removing SAN Volume Controller 2145-CG8 flsh drive documenttion to find out how to perform the procedures. Chpter 11. Using the mintennce nlysis procedures 345

364 5. A new drive object is creted with the use ttribute set to unused. This ction might tke severl minutes. Obtin the ID of the new drive using the lsdrive CLI commnd. 6. Chnge the use property for the new drive to cndidte. chdrive -use cndidte <drive id of new drive> 7. Crete new storge pool with the sme properties s the deleted storge pool. Use the properties tht you recorded in substep 1l. mkmdiskgrp -nme <mdiskgrp nme s before> -ext <extent size s before> 8. Crete gin ll MDisks tht were previously in the storge pool using the informtion from steps 1j nd 1k. v For internl RAID 0 MDisks, use this commnd: mkrry -level rid0 -drive <list of drive IDs> -nme <mdisk_nme> <mdiskgrp id or nme> where -nme <mdisk_nme> is optionl, but you cn use the prmeter to mke the new rry hve the sme MDisk nme s the old rry. v For externl MDisks, use the ddmdisk CLI commnd. v For non-raid 0 MDisks, use the mkrry CLI commnd. 9. For ll the volumes tht hd online, in sync, mirrored volume copies before the MDisk group ws deleted, dd new volume copy in the new storge pool to restore redundncy using the following commnd: ddvdiskcopy -mdiskgrp <mdiskgrp id> -vtype striped -esytier <on or off s before> <vdisk_id> 10. For ny volumes tht did not hve n online, in sync, mirrored copy, crete the volume gin nd restore the dt from bckup or use other methods. 11. Mrk the drive error s fixed using the error sequence number from step 1b. cherrstte -sequencenumber <error_sequence_number> MAP 6002: Replce offline SSD in RAID 1 rry or RAID 10 rry MAP 6002: This procedure replces flsh drive tht hs filed while it is still member of storge pool. Before you begin If you re not fmilir with these mintennce nlysis procedures (MAPs), first red Chpter 11, Using the mintennce nlysis procedures, on pge 265. This mp pplies to models with internl flsh drives. Be sure tht you know which model you re using before you strt this procedure. To determine which model you re working on, look for the lbel tht identifies the model type on the front of the node. Attention: 1. Bck up your SAN Volume Controller configurtion before you begin these steps. 2. If the drive use property is member nd the drive must be replced, contct IBM support before tking ny ctions. About this tsk Perform the following steps if drive fils in RAID 1 or RAID 10 rry: 346 SAN Volume Controller: Troubleshooting Guide

365 Procedure 1. Mke sure the drive property use is not member. Use the lsdrive CLI commnd to determine the use. 2. Record the drive property vlues of the node ID nd the slot ID for use in step 4. These vlues identify which physicl drive to remove. 3. Record the error sequence number for use in step Use the drive ID tht you recorded in step 2 to set the use ttribute property of the drive to unused with the chdrive commnd. chdrive -use filed <id of offline drive> chdrive -use unused <id of offline drive> The drive is removed from the drive listing. 5. Follow the physicl instructions to replce or remove drive. See the Replcing SAN Volume Controller 2145-CG8 flsh drive documenttion or the Removing SAN Volume Controller 2145-CG8 flsh drive documenttion to find out how to perform the procedures. 6. A new drive object is creted with the use property set to unused. 7. Chnge the use property for the drive to cndidte. chdrive -use cndidte <id of new drive> 8. Chnge the use property for the drive to spre. chdrive -use spre <id of new drive> v If you re using spre drives, perform member exchnge. Move dt from the spre to the newly inserted device. v If you do not hve spre, when you mrk the drive object s spre, the rry strts to build on the newly inserted device. 9. If the spre is not perfect mtch for the replced drive, then the rry is considered unblnced, nd error code 1692 is recorded in the error log. 10. Follow the fix procedure to complete the procedure. 11. Mrk the drive error s fixed using the error sequence number from step 3. cherrstte -sequencenumber <error_sequence_number> Chpter 11. Using the mintennce nlysis procedures 347

366 348 SAN Volume Controller: Troubleshooting Guide

367 Chpter 12. iscsi performnce nlysis nd tuning This procedure provides solution for iscsi host performnce problems while connected to SAN Volume Controller system, nd its connectivity to the network switch. About this tsk Some of the ttributes nd host prmeters tht might ffect iscsi performnce: v Trnsmission Control Protocol (TCP) Delyed ACK v Ethernet jumbo frme v Network bottleneck or oversubscription v iscsi session login blnce Procedure 1. Disble the TCP delyed cknowledgment feture. To disble this feture, refer to OS/pltform documenttion. v VMWre: v Windows: The primry signture of this issue: red performnce is significntly lower thn write performnce. Trnsmission Control Protocol (TCP) delyed cknowledgment is technique used by some implementtions of the TCP in n effort to improve network performnce. However, in this scenrio where the number of outstnding I/O is 1, the technique cn significntly reduce I/O performnce. In essence, severl ACK responses my be combined together into single response, reducing protocol overhed. As described in RFC 1122, host my dely sending n ACK response by up to 500 ms. Additionlly, with strem of full-sized incoming segments, ACK responses must be sent for every second segment. Importnt: The host must be rebooted for these settings to tke effect. A few pltforms (for exmple, stndrd Linux distributions) do not provide wy to disble this feture. However, the issue hs been resolved with the version 7.1 relese, nd no host configurtion chnges re required to mnge TcpDelyedAck behvior. 2. Enble jumbo frme for iscsi. Jumbo frmes re Ethernet frmes with size exceeding 1500 bytes. The mximum trnsmission unit (MTU) prmeter is used to mesure the size of jumbo frmes. The SAN Volume Controller supports 9000 bytes MTU. Refer to the CLI commnd CFGPORTIP to enble jumbo frme. This commnd is disruptive s the link flips nd the I/O opertion through tht port will puse. The network must support jumbo frmes end-to-end for this to be effective; verify this by sending ping pcket to be delivered without frgmenttion. For exmple: v Windows: ping -t <iscsi trget ip> -S <iscsi inititor ip> -f -l <new mtu size - pcket overhed (usully 36, might differ)> Copyright IBM Corp. 2003,

368 The following is n exmple of commnd used to check if 9000 bytes MTU is set correctly on Windows 7 mchine: ping -t -S f -l 8964 v v The following is n exmple of successful reply: : bytes=8964 time=1ms TTL=52 Linux: ping -l <source iscsi inittior ip> -s <new mtu size> -M do <iscsi trget ip> ESXi: ping <iscsi trget ip> -I <source iscsi inititor ip> -s <new mtu size - 28> -d 3. Verify the switch's port sttistic where inititor/trget ports re connected to mke sure pcket drops re not high. Review network rchitecture to void ny bottlenecks nd oversubscription. The network needs to be blnced to void ny pcket drop; pcket drop significntly reduces storge performnce. Involve networking support to fix ny such issues. 4. Optimize nd utilize ll iscsi ports. To optimize SAN Volume Controller resource utiliztion ll iscsi ports must be utilized. v Ech port is ssigned to one CPU, nd by blncing the login, one cn mximize CPU utiliztion nd chieve better performnce. Idelly configure subnets equl to the number of ISCSI Ports on the SVC node. Configure ech port of node with n IP on different subnet nd keep it the sme for other nodes.exmple of n idel configurtion: Node 1 Port 1: Port 2: Port 3: v v Node 2: Port 1: Port 2: Port 3: Avoid situtions where 50 hosts re logged in to port 1 nd only 5 hosts re logged in to port 2. Use proper subnetting to chieve blnce between the number of sessions nd redundncy. 350 SAN Volume Controller: Troubleshooting Guide

369 Notices This informtion ws developed for products nd services offered in the U.S.A. IBM my not offer the products, services, or fetures discussed in this document in other countries. Consult your locl IBM representtive for informtion on the products nd services currently vilble in your re. Any reference to n IBM product, progrm, or service is not intended to stte or imply tht only tht IBM product, progrm, or service my be used. Any functionlly equivlent product, progrm, or service tht does not infringe ny IBM intellectul property right my be used insted. However, it is the user's responsibility to evlute nd verify the opertion of ny non-ibm product, progrm, or service. IBM my hve ptents or pending ptent pplictions covering subject mtter described in this document. The furnishing of this document does not grnt you ny license to these ptents. You cn send license inquiries, in writing, to: IBM Director of Licensing IBM Corportion North Cstle Drive Armonk, NY U.S.A. For license inquiries regrding double-byte chrcter set (DBCS) informtion, contct the IBM Intellectul Property Deprtment in your country or send inquiries, in writing, to: Intellectul Property Licensing Legl nd Intellectul Property Lw IBM Jpn, Ltd , Nihonbshi-Hkozkicho, Chuo-ku Tokyo , Jpn The following prgrph does not pply to the United Kingdom or ny other country where such provisions re inconsistent with locl lw: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some sttes do not llow disclimer of express or implied wrrnties in certin trnsctions, therefore, this sttement my not pply to you. This informtion could include technicl inccurcies or typogrphicl errors. Chnges re periodiclly mde to the informtion herein; these chnges will be incorported in new editions of the publiction. IBM my mke improvements nd/or chnges in the product(s) nd/or the progrm(s) described in this publiction t ny time without notice. Any references in this informtion to non-ibm Web sites re provided for convenience only nd do not in ny mnner serve s n endorsement of those Web sites. The mterils t those Web sites re not prt of the mterils for this IBM product nd use of those Web sites is t your own risk. Copyright IBM Corp. 2003,

370 IBM my use or distribute ny of the informtion you supply in ny wy it believes pproprite without incurring ny obligtion to you. Licensees of this progrm who wish to hve informtion bout it for the purpose of enbling: (i) the exchnge of informtion between independently creted progrms nd other progrms (including this one) nd (ii) the mutul use of the informtion which hs been exchnged, should contct: IBM Corportion Almden Reserch 650 Hrry Rod Bldg 80, D3-304, Deprtment 277 Sn Jose, CA U.S.A. Such informtion my be vilble, subject to pproprite terms nd conditions, including in some cses, pyment of fee. The licensed progrm described in this document nd ll licensed mteril vilble for it re provided by IBM under terms of the IBM Customer Agreement, IBM Interntionl Progrm License Agreement or ny equivlent greement between us. Any performnce dt contined herein ws determined in controlled environment. Therefore, the results obtined in other operting environments my vry significntly. Some mesurements my hve been mde on development-level systems nd there is no gurntee tht these mesurements will be the sme on generlly vilble systems. Furthermore, some mesurements my hve been estimted through extrpoltion. Actul results my vry. Users of this document should verify the pplicble dt for their specific environment. Informtion concerning non-ibm products ws obtined from the suppliers of those products, their published nnouncements or other publicly vilble sources. IBM hs not tested those products nd cnnot confirm the ccurcy of performnce, comptibility or ny other clims relted to non-ibm products. Questions on the cpbilities of non-ibm products should be ddressed to the suppliers of those products. All sttements regrding IBM's future direction or intent re subject to chnge or withdrwl without notice, nd represent gols nd objectives only. All IBM prices shown re IBM's suggested retil prices, re current nd re subject to chnge without notice. Deler prices my vry. This informtion is for plnning purposes only. The informtion herein is subject to chnge before the products described become vilble. This informtion contins exmples of dt nd reports used in dily business opertions. To illustrte them s completely s possible, the exmples include the nmes of individuls, compnies, brnds, nd products. All of these nmes re fictitious nd ny similrity to the nmes nd ddresses used by n ctul business enterprise is entirely coincidentl. COPYRIGHT LICENSE: 352 SAN Volume Controller: Troubleshooting Guide

371 This informtion contins smple ppliction progrms in source lnguge, which illustrte progrmming techniques on vrious operting pltforms. You my copy, modify, nd distribute these smple progrms in ny form without pyment to IBM, for the purposes of developing, using, mrketing or distributing ppliction progrms conforming to the ppliction progrmming interfce for the operting pltform for which the smple progrms re written. These exmples hve not been thoroughly tested under ll conditions. IBM, therefore, cnnot gurntee or imply relibility, servicebility, or function of these progrms. The smple progrms re provided "AS IS", without wrrnty of ny kind. IBM shll not be lible for ny dmges rising out of your use of the smple progrms. If you re viewing this informtion softcopy, the photogrphs nd color illustrtions my not pper. Trdemrks IBM, the IBM logo, nd ibm.com re trdemrks or registered trdemrks of Interntionl Business Mchines Corp., registered in mny jurisdictions worldwide. Other product nd service nmes might be trdemrks of IBM or other compnies. A current list of IBM trdemrks is vilble on the web t Copyright nd trdemrk informtion t Electronic emission notices Adobe, the Adobe logo, PostScript, nd the PostScript logo re either registered trdemrks or trdemrks of Adobe Systems Incorported in the United Sttes, nd/or other countries. Linux nd the Linux logo is registered trdemrk of Linus Torvlds in the United Sttes, other countries, or both. Microsoft, Windows, nd the Windows logo re trdemrks of Microsoft Corportion in the United Sttes, other countries, or both. Other product nd service nmes might be trdemrks of IBM or other compnies. This section contins the electronic emission notices or sttements for the United Sttes nd other countries. Federl Communictions Commission (FCC) sttement This explins the Federl Communictions Commission's (FCC s) sttement. This equipment hs been tested nd found to comply with the limits for Clss A digitl device, pursunt to Prt 15 of the FCC Rules. These limits re designed to provide resonble protection ginst hrmful interference when the equipment is operted in commercil environment. This equipment genertes, uses, nd cn rdite rdio frequency energy nd, if not instlled nd used in ccordnce with the instruction mnul, might cuse hrmful interference to rdio communictions. Opertion of this equipment in residentil re is likely to cuse hrmful interference, in which cse the user will be required to correct the interference t his own expense. Properly shielded nd grounded cbles nd connectors must be used in order to meet FCC emission limits. IBM is not responsible for ny rdio or television interference cused by using other thn recommended cbles nd connectors, or by Notices 353

372 unuthorized chnges or modifictions to this equipment. Unuthorized chnges or modifictions could void the user's uthority to operte the equipment. This device complies with Prt 15 of the FCC Rules. Opertion is subject to the following two conditions: (1) this device might not cuse hrmful interference, nd (2) this device must ccept ny interference received, including interference tht might cuse undesired opertion. Industry Cnd complince sttement This Clss A digitl pprtus complies with Cndin ICES-003. Cet ppreil numérique de l clsse A est conform à l norme NMB-003 du Cnd. Austrli nd New Zelnd Clss A Sttement Attention: This is Clss A product. In domestic environment this product might cuse rdio interference in which cse the user might be required to tke dequte mesures. Europen Union Electromgnetic Comptibility Directive This product is in conformity with the protection requirements of Europen Union (EU) Council Directive 2004/108/EC on the pproximtion of the lws of the Member Sttes relting to electromgnetic comptibility. IBM cnnot ccept responsibility for ny filure to stisfy the protection requirements resulting from non-recommended modifiction of the product, including the fitting of non-ibm option crds. Attention: This is n EN Clss A product. In domestic environment this product might cuse rdio interference in which cse the user might be required to tke dequte mesures. Responsible Mnufcturer: Interntionl Business Mchines Corp. New Orchrd Rod Armonk, New York Europen community contct: IBM Deutschlnd GmbH Technicl Regultions, Deprtment M372 IBM-Allee 1, Ehningen, Germny Tele: Emil: lugi@de.ibm.com Germny Electromgnetic Comptibility Directive Deutschsprchiger EU Hinweis: Hinweis für Geräte der Klsse A EU-Richtlinie zur Elektromgnetischen Verträglichkeit 354 SAN Volume Controller: Troubleshooting Guide

373 Dieses Produkt entspricht den Schutznforderungen der EU-Richtlinie 2004/108/EG zur Angleichung der Rechtsvorschriften über die elektromgnetische Verträglichkeit in den EU-Mitgliedsstten und hält die Grenzwerte der EN Klsse A ein. Um dieses sicherzustellen, sind die Geräte wie in den Hndbüchern beschrieben zu instllieren und zu betreiben. Des Weiteren dürfen uch nur von der IBM empfohlene Kbel ngeschlossen werden. IBM übernimmt keine Verntwortung für die Einhltung der Schutznforderungen, wenn ds Produkt ohne Zustimmung der IBM verändert bzw. wenn Erweiterungskomponenten von Fremdherstellern ohne Empfehlung der IBM gesteckt/eingebut werden. EN Klsse A Geräte müssen mit folgendem Wrnhinweis versehen werden: Wrnung: Dieses ist eine Einrichtung der Klsse A. Diese Einrichtung knn im Wohnbereich Funk-Störungen verurschen; in diesem Fll knn vom Betreiber verlngt werden, ngemessene Mbnhmen zu ergreifen und dfür ufzukommen. Deutschlnd: Einhltung des Gesetzes über die elektromgnetische Verträglichkeit von Geräten Dieses Produkt entspricht dem Gesetz über die elektromgnetische Verträglichkeit von Geräten (EMVG). Dies ist die Umsetzung der EU-Richtlinie 2004/108/EG in der Bundesrepublik Deutschlnd. Zulssungsbescheinigung lut dem Deutschen Gesetz über die elektromgnetische Verträglichkeit von Geräten (EMVG) (bzw. der EMC EG Richtlinie 2004/108/EG) für Geräte der Klsse A Dieses Gerät ist berechtigt, in übereinstimmung mit dem Deutschen EMVG ds EG-Konformitätszeichen - CE - zu führen. Verntwortlich für die Einhltung der EMV Vorschriften ist der Hersteller: Interntionl Business Mchines Corp. New Orchrd Rod Armonk,New York Tel: Der verntwortliche Ansprechprtner des Herstellers in der EU ist: IBM Deutschlnd GmbH Technicl Regultions, Abteilung M372 IBM-Allee 1, Ehningen, Germny Tele: Emil: lugi@de.ibm.com Generelle Informtionen: Ds Gerät erfüllt die Schutznforderungen nch EN und EN Klsse A. People's Republic of Chin Clss A Sttement Notices 355

374 Tiwn Clss A complince sttement Tiwn Contct Informtion This topic contins the product service contct informtion for Tiwn. IBM Tiwn Product Service Contct Informtion: IBM Tiwn Corportion 3F, No 7, Song Ren Rd., Tipei Tiwn Tel: f2c00790 Jpn VCCI Council Clss A sttement This explins the Jpn Voluntry Control Council for Interference (VCCI) sttement. Jpn Electronics nd Informtion Technology Industries Assocition Sttement This explins the Jpn Electronics nd Informtion Technology Industries Assocition (JEITA) sttement for less thn or equl to 20 A per phse. 356 SAN Volume Controller: Troubleshooting Guide

375 This explins the JEITA sttement for greter thn 20 A per phse. jjiet2 Koren Communictions Commission Clss A Sttement This explins the Koren Communictions Commission (KCC) sttement. Russi Electromgnetic Interference Clss A Sttement This sttement explins the Russi Electromgnetic Interference (EMI) sttement. rusemi Notices 357