Troubleshooting Cisco Storage Networks

Transcription

1 1 Troubleshooting Cisco Storage Networks Session 2

2 Session Agenda Fibre Channel Ports Physical Level and Type Focus on Device Discovery and Fibre Channel Services E-Port ISL Troubleshooting, Including FCIP SN5428 Specific Troubleshooting PA-FC-1G Specific Troubleshooting MDS9000 Specific Troubleshooting iscsi Driver Troubleshooting 3 Fibre Channel Port Fundamentals 4

3 Fibre Channel Port Identification N_Port Any Port on a Node Device F_Port Port on the Fabric N E_Port Ports for Interconnect of Switches F G_Port Port with Both E and F Port Capability FC Switch E NL_Port FL_Port GL_Port N_Port with Loop Capability F_Port with Loop Capability G_Port With Loop Capability FC Switch FL E U_Port Can Be a G, E, F, FL NL B_Port and TE_Ports Will Be Covered in Later Section N N NL FC-AL Hub 5 Fibre Channel Transmission Words Ordered sets that are transmitted continuously to indicate that specific conditions within the link are encountered Transmitted while the condition exist Four primitive sequences Not Operational Sequence (NOS) Offline Sequence (OLS) Link Reset Sequence (LR) Link Reset Response Sequence (LRR) 6

4 Link Initialization Flow Port A AC Port B AC Link Failure Condition LF LR AC LR Idle NOS OLS LRR Idle LF OL LR AC These Are All Special Ordered Sets of 8B/10B Coding: NOS = Not Operational Sequence OLS = Offline Sequence LR = Link Reset LRR =Link Reset Response AC = Active State LR = Link Recovery State LF = Link Failure State OL = Offline State 7 Port Type Connection Protocol What port types use what sequences N_Port Uses NOS/OLS to connect F_Port Uses NOS/OLS to attach to single N_Port FL_Port Uses LIP to attach 1 to 126 NL_Ports E_Port Uses NOS/OLS to interconnect switches (Inter-Link Switch ISL) B_Port Same as E_Port but only bridges (FCIP) G_Port Uses NOS/OLS can be a F or E port U_Port G, E, F,FL; it basically is a transition state MDS9000 Also Has TE Port and SD Port, Covered in MDS Section 8

5 Where You Find Link Level Issues MDS9000 show int fc PA-FC-1G show fc fc-stats 9 Where You Find Link Level Issues SN5428 show interface fc# stats 10

6 Other Common Link Level Issues GBIC or SFP issues MDS9000# show interface fc x/x transceiver (SFP) SN5428-2# show interface brief MM optical cable issues Check for light on TX of FC port and end of cable Differences in optics Install qualified GBIC or SPF; most SFP s will work for both Fibre Channel and GiGE, MM vs. SM, LW vs. SW 11 Agenda Next Subject Fibre Channel Ports Physical Level and Type Focus on Device Discovery and Fibre Channel Services 12

7 Device Discovery and Fibre Channel Services 13 World Wide Names Each switch element is assigned a 64 bit WWN at time of manufacture Each switch port is assigned a 64 bit WWPN at the time manufacture During Fabric Logon (FLOGI) the switch identifies the WWN in the service parameters of the accept frame These Address Assignments Can then Correlate Each Fabric Port with Switch Routing and the Fiber Channel ID (FCID) 14

8 FC World Wide Name Addressing Single Hex Digit Selected by Vendor 1= ieee Standard 2= ieee Extended 2 Port HBA Node WWN da :00 00:01:64 40:da:20 IEEE Assigned Vendor Assigned 3 HEX Digits Used by Vendor to Identify Port Port 1 WWPN da 20 Port 2 WWPN da 20 Port 1 WWPN a7 0d d8 Port 2 WWPN a7 0d d8 Disk Drive Disk WWN a7 0d d8 15 Fabric Addressing FCID The 24 bit address is partitioned into 3 fields Device Area (depends on vendor) Domain This partitioning is used in FSPF routing as source and destination ID s The Fabric Login Server at address FF FF FE assigns the address to the connecting N_Port 8 Bits 8 Bits 8 Bits Switch Domain Area Device 16

9 Name Server Repository of information regarding the components that make up the Fibre Channel network Located at address FF FF FC Components can register their characteristics with the Directory Server An N_Port Host can query the Directory Server for specific information 17 Name Server Command Requests These Are Some of the More Commonly Used Commands Used to Talk to the Name Server: Get objects from directory GA_NXT Get all next GID_FT Get FCID Register objects in directory RFT_ID Register FC-4 types Deregister objects DA_ID Deregister all 18

10 Fabric Controller Each switch has a fabric controller Assigned address FF FF FD Every fabric controller in the fabric has the same address Responsible for managing fabric, initialization, FSPF routing, responsible for RSCN s and listens for SCN s Responsible to receive request and generate responses for the switch fabric The MDS9000 also has the Fabric Domain Controller at address FFFCxx where xx=vsan domain number; this is used for LUN discovery 19 Fabric Login Host System N_Node 64-bit Node WWN 64 Bit Port WWN 24-bit port ID Registration Information Registration for State Changes FLOGI Accept PLOGI Accept SCR Accept Fabric F_Port X FF FF FE Name Server X FF FF FC Fabric Controller X FF FF FD Distributed Name Server Database Address Discovery for SCSI Storage Devices Name Server Query Accept Name Server X FF FF FC By Registering for State Change Notifications the Port or Domain Controller Can Then Notify the Connected Ports with RSCN s when a Port Changes 20

11 24-Bit Address Assignment Host Host Connected to Port 3 on Switch Switch Domain ID = 5 FLOGI Accept 24 Bit Port ID Assigned Is x FLOGI Accept 24 Bit Port ID Assigned Is x Host Connected to Port 4 on Switch Switch Domain ID = 1 Storage Array 21 Port to Port Host on Port 3 of Switch ID 5 Name Server Query Get FC-4 Devices PLOGI D_ID x S_ID x PRLI D_ID x S_ID x X Name Server X FF FF FC FSPF Routing DB Accept Accept D_ID x S_ID x SCSI Inquiry Report LUNS ID 1 Presents LUN s to Host Storage on Port 4 of Switch 22

12 Agenda Next Subject Fibre Channel Ports Physical Level and Type Focus on Device Discovery and Fibre Channel Services E-Port ISL Troubleshooting, Including FCIP 23 E-Port/B-Port ISL Troubleshooting 24

13 E-Port Troubleshooting Differences between E-Port and B-Port Understanding the E-Port inter-switch Link Connection (ISL) Troubleshooting domain E-Port and B-Port fabric connection issues 25 FCIP E-Port and B-Port Relationships FCIP E-Port VE-Port FCIP E-Port FC SAN FC Switch IP Network FC Switch FC SAN Exchange Fabric Parameters Exchange Fabric Parameters Exchange Fabric Parameters E Port Operation Exchange Link Parameters ESC Exchange FCIP- Link Parameters ESC Exchange Link Parameters ESC FC SAN B Port Operation FCIP E-Port FC Switch FCIP B-Port Exchange Link Parameters 7200 w/ PA-FC-1G or SN VB-Port IP Network Exchange FCIP-Link Parameters 7200 w/ PA-FC-1G or SN Exchange Fabric Parameters ESC (Exchange Switch Capabilities) FCIP B-Port FCIP E-Port Exchange Link Parameters FC Switch FC SAN 26

14 Understanding the ISL Connection E_Port and Cisco MDS TE_Port Link initialization Exchange link parameters Link Reset Exchange switch capabilities Exchange port parameters MDS9000 only to determine if connected to another MDS Exchange fabric Parameters Assign domain IDs Establish routes Merge zones if required B_Port Link initialization Exchange link parameters Link Reset 27 E/TE Port Link Exchange Explained Exchange Switch Capabilities Primitive Sequences Exchange Link Parameters Exchange Port Parameters LR to Initialize Flow Control Buffer Credit Parameters At This Point a B_Port Connection Is Up and E_Port to E_Port Exchange Continues Exchange Port Parameters 28

15 E/TE Port Link Exchange Explained Domain Identifier Assigned Exchange Fabric Parameters Exchange Fabric Parameters Request Domain ID 29 E/TE Port Link Exchange Explained Exchange Fabric Parameters Merge Request Merge Request Zoning FSPF Hello Merge Request Merge Request At This Point FSPF Continues to Update Link State Information Between the Switches 30

16 E_Port, TE_Port Connection Problems 1. Trunk E-Port configuration mismatch Fabric parameters and timers mismatch switch 2. Zoning parameters mismatch per VSAN or switch 3. VSAN configuration mismatch (MDS9000) 4. Domain IDs must not conflict on either VSAN or switch 31 E-Port, TE_Port Troubleshooting Output of Good TE Port ISL Error Indication Will be Displayed Here 1. Isolation due to ELP (Exchange Link Parameters) failure ED_TOV Timer RA_TOV Timer Frame size FS_TOV timer 2. Isolation due to zone merge failure Same name used to describe 2 different types or content mismatch in the merging switches If the same zones on 2 different switches contain different members 3. Isolation due to port VSAN mismatch E_ports not in same VSAN MDS9000# sh vsan membership 32

17 E-Port Troubleshooting Domain Assignment Failure Each VSAN or FC Switch has its own domain ID, what happens when they are the same? E_Port becomes isolated because the domain ID s overlap on the 2 switches New switch being added to the network will be assigned an ID if not managed with one If both switches are in production network, there are two ways to resolve the issue but both are disruptive to the FC network 1. A Cisco MDS9000 can be configured to force a RCF (Reconfigure Fabric); this will force a reselection of Domain ID s and resolve the issue 2. Take the switch offline and manually configure Domain ID and re-enable switch 33 Specifications that Define Operations ANSI: FC-SW-2 describes the operation and interaction of Fibre Channel switches, including E_Port, B_Port and Fabric operation FC-BB-2 is a mapping that pertains to the extension of Fibre Channel switched networks across a TCP/IP network backbone and defines reference models that support E_Port and B_Port IETF IPS Working Group: Fibre Channel over TCP/IP covers the TCP/IP requirements for transporting Fibre Channel frames over an IP network FC frame encapsulation defines the common Fibre Channel encapsulation format 34

18 Agenda Next Subject Fibre Channel Ports Physical Level and Type Focus on Device Discovery and Fibre Channel Services E-Port ISL Troubleshooting, Including FCIP SN5428 Specific Troubleshooting 35 SN5428 Specific Troubleshooting 36

19 SN5428 Troubleshooting Debug uses iscsi troubleshooting SN5428 Fibre channel switch troubleshooting E-Port FCIP B-Port troubleshooting 37 SN 5428 Functional Blocks SCSI Routing Engine fci1 fc0 fci2 fc15 Control Plane and Management FC Switch Console Mgmt 10/100 HA 10/100 GE1 GE2 FC1 FC2 FC3 FC4 FC5 FC6 FC7 FC8 Two internal FC ports allow access between the routing engine and FC switch ports Interfaces fci1 and fci2 will look like Initiators logged into the nameserver 38

20 SN5428 Debugs Depends on which support mode the SN5428 is in iscsi router mode debug scsirouter FCIP mode debug FCIP Both modes have debug for GigE interface Debugging different from typical Cisco IOS Debugs are enabled and captured to a circular file by default Captured debug information is then displayed via show commands 39 SN5428 Data Flow Debugs [SN5428A]# debug scsirouter scsiroutername iscsitrace scsitrace target iscsi trace facility (debugs iscsi PDU on IP side) Enable SCSI trace (debugs scsi CDB, FC side) Set variables in a SCSI Router (singles out target) [SN5428A]# debug scsirouter scsiroutername iscsitrace Enable fromto pducnt pdusize Enable iscsi trace facility (always typed at end of debug string) Trace PDUs from/to specified host/network Set max number PDUs to trace Set PDU trace size 40

21 SN 5428 Functional Blocks Show debug mailboxtrace Show devices Show devices rediscover (clears this table and forces a a GA_NXT) Fibre Channel Device Discovery Table Port Database Show debug rawlundatabase Control Plane and Management SN5428 O/S fci1 fci2 fc0 fc15 FC Switch O/S Console Mgmt 10/100 HA 10/100 GE1 GE2 FC1 FC2 FC3 FC4 FC5 FC6 FC7 FC8 Show interface GEx Show interface FCx 300 Line Circular Buffer; Latest Entry at Bottom Show debug scsirouter all scsitrace iscsi PDU and CDB Global Trace T1 T2 Targets with LUN s 41 FC Device Debugging This is a complete output of all LUN discovery that is done by the Fibre Channel firmware in the SN5428, the report LUNs is done to all WWPN s returned to the SN542X after a query to the directory server show debug rawlundatabase all Entry Address = 0x7595b6c fabricloginfailurecode=0x0,fabricloginextendedcode=0x0,fabriclogintimeoutcode=0x0 ReportLunsLLDStatus=0x0,ReportLunsLLDStatusModifier=0x0,ReportLunsSCSIStatus=0x0,ReportLunsASCASCQ=0x0,ReportLunsLunCount=1 InquiryLLDStatus=0x0,InquiryLLDStatusModifier=0x0,InquiryLastLunWithLLDError=0x0, WWPN that InquirySCSIStatus=0x0,InquiryASCASCQ=0x0,InquiryLastLunWithSCSIStatusError=0x0 Inquiry Is Being boollunsnotsupported=0x0,inquirylastlunnotsupported=0x0 Done To loopid=0x1,masterstate=0x6,slavestate=0x7,loggedin=0,roles=1,valid=1,portid=0xe8,scanluns=0x0 numberluns=0x1,reportasyncevent=0x0,node_wwn=0x x37a70dd8, port_wwn=0x x37a70dd8 lun=0, wwnn=0x x37a70dd8, reportasyncevent=0x0 stdinquiry data for lun=0x0 bytes0-7=0x x8b00500a vendorid=seagate, product=st318451fc, revision=0001 device Type=0x0 DeviceIdPage:bytes0-3= 0x c,bytes4-7= 0x ,bytes8-11= 0x :bytes12-15= 0x37a70dd8,bytes16-19=0x ,bytes20-23=0x S/N Page:bytes0-3= 0x s/n=3cc02avp epts LUN Count Returned WWN on the LUN Port LUN Is LUN 0, and This Is the Data Return from the Report LUN Command 42

22 GigE Packet Tracing Debugging A Little Different from IOS Type Outputs Debug Is Turned On, But Output Is Directed to a File Debug Monitoring Is Done with Showing of Stats Debug Is Halted With No Show Command Is Used to Display the Captured Data The Output From the Captured File Is Saved and Can Be Shown Until Another Debug Over Writes the File Ascii Output Can Be Selected for Readable Form 43 iscsi PDU Debugs Details of iscsi PDU s ascii readable Use pdusize 256 to read ascii Use stats to get packet capture status Trick is to use keyboard repeat keys to turn debugs off Readable Exchange Data 44

23 Troubleshooting SN5428 FC Switch [SN5428-2]# show fcswitch Eport fabric global-nameserver linkstate nameserver Display of eport FSPF information Display of domains in the SAN Fabric Complete SAN wide nameserver entries including host devices Display linkstate transitions Displays this FC switch target devices 45 SN5428 Outputs Show interface fci2 stats sh debug fcswitch all Will Give Log Output Frame Counts and Errors on the Internal FC Interface List of Logged Messages Are Output Here 46

24 SN5428 Fibre Channel Switch E_Port to E_port Troubleshooting Internal SN5428 WWPN s SN5428 Domain 1 Port FC 3 Port fc 1/1 E-Port ISL MDS9216 Domain 101 (0x65) 47 SN5428 E_Port to E_port Loop Targets on JBOD Port FC2 SN5428 Domain 1 Port FC3 E-Port ISL HBA Attached Host MDS9216 Domain 101 (0x65) 48

25 Interrogating E_Port Connection Via FCIP B_Port FCIP Link SN5428 internal WWPN s Will Not be Shown in FCIP Mode SN5428 Domain 1 FCIP Port 0 FCIP B_Port on SN5428 is Sourced from Port 255 Internally; the Remote E_Port Connection is Sourced Via Port 0 Internally MDS9216 Domain 99 (0x63) 49 Interrogating E_Port Connection Via FCIP B_Port SN5428 Domain 1 FCIP Link FC Port 2 Devices e4 and e5 FCIP Link Info MDS9216 Domain 99 (0x63) 50

26 Agenda Next Subject Fibre Channel Ports Physical Level and Type Focus on Device Discovery and Fibre Channel Services E-Port ISL Troubleshooting, Including FCIP SN5428 Specific Troubleshooting PA-FC-1G Specific Troubleshooting 51 C7200 and PA-FC-1G Specific Troubleshooting 52

27 PA-FC-1G FCIP Troubleshooting Port adaptor configuration troubleshooting Debugging of PA-FC-1G Debugging the E-Port connection to the Fibre Channel Switch 53 PA-FC-1G Example of FCIP connectivity between McData and Brocade using the PA-FC-1G Port Adapter Local C7200 Remote C

28 FCPA Configuration Local Side Tunnel Remote Side Tunnel interface Fcpa1/0 ip address fc-tunnel Local src-ip dest-ip src-port 3000 dest-port 2000 tcp sack tcp mws 256 tcp kad 7200 ip tos 0 inservice interface Fcpa2/0 ip address fc-tunnel Remote src-ip dest-ip src-port 2000 dest-port 3000 tcp sack tcp mws 256 tcp kad 7200 ip tos 0 inservice 55 PA-FC-1G B_Port Initiation B_Port initialization can be divided into roughly three stages: 1. Link initialization 2. Exchange Link Parameters (ELP frames) 3. Link reset Most typical problems are issues with ELP exchange PA-FC-1G only runs at 1 GIG, speed, connected Fibre Channel Switches need to be fixed at 1 GiG (no auto speed) Connecting Fibre Channel switches may need to run in SW-2 compatibility mode if different vendor types Best practice is to connect Fibre Channel Switch E_Ports together first prior to connection to PA-FC-1G FCIP connection 56

29 Show fc-tunnel Detail First Page This Is Basically the Fibre Channel Interface and B_Port Connection Configuration is Correct Between Local FCPA Interface and Local FC Tunnel Set TCP Values Per Config Highest Support MTU Session Manager State MSS Value Negotiated After Tunnel Is Inservice PWWN of B_Port Interface on PA-FC-1G BB Credits the PA-FC-1G Has Available to Send Frames to Connected FC Switch E_Port Timeout Values Set on B_Port Interface Up = FC B_Port Had Successful ELP with E_Port PWWN of E_Port Interface on Connected FC Switch; This Will be Blank Until Successful ELP Acknowledgment WWN of on Connected FC Switc; This Will be Blank Until Successful ELP Acknowledgment 57 Sh fc-tunnel Detail Second Page Forced at 1 GiG Today Successful Exchange Link Parameters Completed, 1=Yes, 0=No, 1 Needed for FC Interface to Come Up Buffers Credits for Class-F Frames; Ack ed from Remote End BB Credits Set Based on Switch Vendor ID TOV Parameters That Are Hard Configured within the B-Port Interface Echo Frames Enabled, This Is an Extended Link Service Echo Command Sent by the Port to the Remote FCIP Link End Point (B-Port); This is Monitored by TCP for a Connection Keepalive Credits and Timers Settings on E-Port of FC Switch (Learned at ELP) 58

30 show fc tcpconn Used to monitor performance issue related to WAN transport 7200#sh fc tcpconn Interface: Fcpa6/0 TCP variables for Connection Fcpa6/0: ======================================= snd_una: , snd_nxt: , snd_up: , snd_wnd: snd_max: , snd_cwnd: 92224, snd_retx_max: , snd_ssthresh: 2176 t_unack_datasz: 564, t_flags: 612, t_state: TCPS_ESTABLISHED, rcv_wnd: , rcv_nxt: , rcv_adv: TCP Timers: REXMT=0 PERSIST=0 KEEP=7189 2MSL=0 Window Sizes TCP State 59 Debugs for FCPA Many different debugs to run All are pretty safe except cordova-driver has a chance to cause the 7200 to choke Most issues are at install with connection to Fibre Channel Switch 7400#debug fc? all cli cordova-driver fcap fd northstar-driver sm tcp All Modules cli Module Cordova Driver Module FCAP Module FD Module Northstar Driver Module Session Manager Module TCP Module 60

31 A Look at Debug FCPA FD all Inservice (same as no shut) Fcpa1/0: TCP Connection Established! *Sep 5 02:08:09.658: GG_FCPA_FD_EV: Fcpa1/0 fd_process_tcm_message(): SM2FD_CONN_OPENED_EV - TCP connection success *Sep 5 02:08:09.658: GG_FCPA_FD_EV: Fcpa1/0 fd_process_tcm_message(): Initializing CPU credits to 32 Link Reset Rx state +++ LR2 +++ Link Active state AC +++ Link Initilization Completed *Sep 5 02:08:10.494: GG_FCPA_FD_EV: 9.782: Fcpa1/0 fd_process_bprt_fc_frame (): ELP frame! *Sep 5 02:08:10.526: GG_FCPA_FD_EV: Fcpa1/0 fd_process_ns_pkt_msg (): ID_FCINIT - 0xBC49BF49 LR received Link Reset Rx state +++ LR2 +++ Link Active state AC +++ Link Reset completed Link offline Rx wait state ++ OL3 +++ Link failure Rx state +++ LF1 +++ Fcpa1/0: B-port is UP! SW_ACC frame from GiGE SW_ACC frame from FC CPU_CREDITS: 24 adding 8 IP Tunnel Is Good Configuration Good F-Class Frames Established Across the FCIP Tunnel Primitive Sequences for Physical Link Connection FC Link Initialization for Buffers Indication ELP Exchange Was Good This Link Reset to Establish FC Credit Buffers for Data 61 Issues with Fibre Channel Connected Switch B-Port connections require Fibre Channel switches to be configured in SW-2 Interop mode if not the same vendor type Fibre Channel Switch options specific to that vendor may not be used i.e. proprietary vendor interworking settings All E-Port issues discussed in the E-Port troubleshooting section apply when working with B-Port connections 62

32 Agenda Next Subject Fibre Channel Ports Physical Level and Type Focus on Device Discovery and Fibre Channel Services E-Port ISL Troubleshooting, Including FCIP SN5428 Specific Troubleshooting PA-FC-1G Specific Troubleshooting MDS9000 Specific Troubleshooting 63 MDS9000 Specific Troubleshooting 64

33 Section Agenda MDS 9000 Troubleshooting Using Debug Serviceability Tools Switch to Switch Fabric Connectivity Storage and Host Discovery, Distribution across the SAN Troubleshooting MDS Gathering Traces for Analysis Using built-in FCAnalyzer Using Ethereal Using the MDS Port Analyzer Adapter Using Span Using an external FC Analyzer 66

34 MDS FCAnalyzer Output is displayed to the console in readable format Is only used to monitor Fibre Channel Traffic to and from supervisor on the MDS9000 Span is used for FC port to FC port monitoring 67 FCAnalyzer Detail Trace This is the command entered at the MDS CLI (while in config mode) to start the FCAnalyzer trace; here we show how to filter for VSAN 1 traffic only MDS-9509(config)# FCanalyzer local display-filter (mdshdr.vsan==0x01) Other MDS built in filters are found in Chapter 23 on MDS9000 configuration guide FCAnalyzer is turned off after output has been displayed to your screen and user exits configuration mode 68

35 FCAnalyzer Brief MDS Fabric Processes Login from the Device and then Will Also Log into Attached Device and Query for Device Type and LUN s eth2 Is an Internal Port Associated with the Analyzer Fiber Channel Traffic Is Seen Going to and from Fabric Login Controller at Address FE (FLOGI) Connecting Device 7e0000 Registers for State Changes and Gets Response PLOGI s and Name Query s to the MDS Nameserver Address FC Device Logs into MDS Platform Management Server Process Login s to Device on Switch 7e Port 00 Address 00 Are Captured and MDS Logs Out 69 Fibre Channel Capture Breakdown Destination is Switch Fabric Login Server From Local Port Fabric Login WWN s of MDS 00:10:73 = Vendor ID of Attached Device a JNI HBA Class 3 Device Parameters 70

36 Ethereal Protocol Analyzer Rich functionality: GUI front-end runs on Linux, Windows, Solaris and many other OSs Filter-based capture and display Minimal protocol statistics Colorized display Find frame, mark frame, read captured traces Customizable columns Supported in Ethereal version and higher 71 What Ethereal Can Do Decodes: NIC Adapter s encapsulation header All FC protocols SCSI (SPC-2, SBC-2, SSC-2) VSAN-aware Matches response with request and so can decode responses completely 72

37 What Ethereal Cannot Do Industry-strength wire-speed capture Ethereal s speed is limited by machine/os limitations No decoding of FC-1 including: No 10bit/8bit encoding No ordered sets Works with MDS s SPAN functionality Cannot act as a bump-in-the-wire like other FC protocol analyzers 73 MDS FCAnalyzer Remote with Ethereal For capture of internal MDS9000 switch traffic, FC traffic to and from supervisor not FC port to FC port traffic Make sure MDS can ping workstation IPS-TEST(config)# fcanalyzer remote active IPS-TEST# sh fcanalyzer ActiveClient = , DEFAULT IPS-TEST#clear fcanalyzer turns off analyzer SCSI HBA At DOS Prompt: ethereal -i rpcap:// /eth2 Or at Ethereal Interface Drop-Down: rpcap:// /eth2 Management 10/100 Port on MDS Address FC Workstation Running Ethereal IP Address

38 Use of Span Feature Used for FC Port to FC Port analyzing Same type of tool as used on Cisco Catalyst products Ingress and egress ports are sent to a FC port set up as a Span Destination (SD port type) No limits to where the ports are located on the MDS switch Used to output to third party test equipment or to Cisco Port Analyzer Adapter 75 Source Types Supported by SPAN Physical ports F, FL, TE, E Port channels All ports in a port channel are configured for SPAN Same filters are applied to all the ports VSAN traffic is spanned only in ingress direction All FC ports in a VSAN are included as a source Two sessions can share a source port but in different directions (ingress/egress) 76

39 MDS 9000 Port Analyzer Adapter MDS Port Analyzer Adapter is a protocol converter Converts FC frames into Ethernet frames FC-specific information is carried in encapsulation header Enables use of ethernet protocol analyzers to decode FC protocols Cisco has released decodes for all FC protocols, the encapsulation header and SCSI decoder to public domain Part of the popular Ethereal protocol analyzer 77 Span Configuration with Single Port Analyzer Ingress to the Supervisor Egress from the Supervisor FC1/1 FC 2/16 FC 6/8 Span Destination Port (SD) Dropped TX RX TX RX FC Analyzer 78

40 Port Analyzer Connection Using Port Analyzer with Ethereal For capture of device port to device port Fibre Channel traffic to a IP attached workstation running Ethereal 1. Configure Port Analyzer Adapter (dip switch settings, mostly plug and play) 2. Configure SPAN port on MDS Set Ethereal to capture on local Ethernet interface SCSI HBA Cisco Port Analyzer Adaptor FC Workstation Anywhere on IP Network 79 Fibre Channel Capture without Port Analyzer with Third Party Analyzer For capture of device port to device port Fibre Channel traffic, to a Fibre Channel attached tester MDS9000#config t span session 10 destination interface fc1/4 source interface fc 2/1 tx MDS9000#config t span session 11 destination interface fc2/13 source interface fc5/1 rx FC 2/1 FC 1/4 SCSI HBA TX RX FC 5/1 RX TX FC 2/13 Dropped Dropped TX RX TX RX 2 Port FC Analyzer FC 80

41 Fibre Channel Port Issues In order for a F_Port to come up on a MDS switch The switch port must first acquire bit and word synchronization with the N_Port N_Port must issue a FLOGI to the MDS FCOT = Fibre Channel Optical Transport 81 Using FCPing and FCtrace Need FCID or pwwn Ping uses PRLI Extended Link Service Checks connectivity and latency on devices attached to MDS Switches directly or across TE ports FCtrace command is two path descriptors that identify the path taken on a hop-by-hop basis including a timestamp at each hop in both directions IPS-TEST# show flogi database vsan INTERFACE VSAN FCID PORT NAME NODE NAME fc1/8 7 0x :00:00:00:c9:30:ba:06 20:00:00:00:c9:30:ba:06 Total number of flogi = 1. IPS-TEST# fcping fcid 0x vsan 7 28 bytes from 0x time = 1896 usec 28 bytes from 0x time = 1116 usec 28 bytes from 0x time = 305 usec 28 bytes from 0x time = 575 usec 28 bytes from 0x time = 565 usec IPS-TEST# fctrace fcid 0x4c0007 vsan 1 Route present for : 0x4c :00:00:05:30:00:37:de(0xfffc4a) Timestamp Invalid. 5 frames sent, 5 frames received, 0 timeouts Round-trip min/avg/max = 305/891/1896 usec 20:00:00:05:30:00:2f:de(0xfffc4b) Timestamp Invalid. 20:00:00:05:30:00:44:5e(0xfffc4c) Timestamp Invalid. 20:00:00:05:30:00:2f:de(0xfffc4b) 82

42 Debugging Host Login Issues Int fc1/8 SCSI HBA FC 1. Show interface fc 1/8 2. Show port internal info int fc1/8 3. debug port event int fc1/8 4. Show flogi database vsan 7 83 Interrogating Fibre Channel Name Server Database Show fcns database Show fcs database 84

43 Debugging Name Server Issues If a host is not receiving a proper response, or if not able to discover a target, enable the following debug 1. debug fcns events (this will give the request and response to a host s request) 2. Check if the host and the disk are in the same zone 3. Check if the host and the disk are in the same vsan 85 Agenda Next Subject Fibre Channel Ports Physical Level and Type Focus on Device Discovery and Fibre Channel Services E-Port ISL Troubleshooting, Including FCIP SN5428 Specific Troubleshooting PA-FC-1G Specific Troubleshooting MDS9000 Specific Troubleshooting iscsi Driver Troubleshooting 86

44 iscsi Driver Troubleshooting 87 Understanding the iscsi Connection iscsi Protocol Host Login and flows A look at W2K tools for troubleshooting iscsi A look at Linux driver utilities and status logs 88

45 iscsi Login Sequence Initiator TCP iscsi Driver iscsi Initiator Login (Discovery Session) Establish TCP Session Command Login Session Type: Discovery-Session Optional Authentication Phase Command SendTargets Target TCP Port 3260 Response Target list: com.cisco.target-disk1 Com.cisco.Target-Disk2 89 iscsi Login Sequence Initiator TCP iscsi Driver iscsi Initiator Login to Target Establish Normal TCP Session Command Login Session-Type: Normal-Session, Target: com.cisco.target-disk1 Optional Authentication Phase Operational Parameter Negotiation After This Completes We Are in Full Feature Mode Target TCP Port 3260 Command SCSI Commands Normal SCSI Commands 90

46 Windows iscsi Driver Status Relogin Will Drop TCP Session and Cause Relogin and Discovery with All Targets CHAP Password If Passwords Are Set then You Are Using CHAP, Make Sure Your Target Device Is Set to Work with CHAP 91 W2K TCP Registry Tweaks 2 ways to tweak directory 1. Use regedit and adjust values manually 2. Use a freeware tool like DR TCP Interface MTU Adjusted on Per NIC Interface All Other Options Done on TCP Stack in Windows 92

47 Windows iscsi Debug 93 Linux Support Commands Useful Commands: /var/log/iscsi.log This log contains iscsi diagnostic information if there were problems during the iscsi kernel module load /usr/bin/iscsi-ls l Utility to view current available iscsi devices on the system (new in 3.2 drivers) /proc/scsi/iscsi Identifies the controller number assigned to the iscsi HBA /etc/rc.d/init.d/iscsi reload Rediscover command for new iscsi devices that have been added to iscsi.conf (does not effect connected targets ) Read and follow release notes that come with the Cisco Linux Driver To check version of Linux driver loaded: [Linux]# cd/usr/src [Linux]# ls -l iscsi lrwxrwxrwx 1 root root 17 Feb 28 14:31 iscsi -> linux-iscsi

48 Linux Log Detail 95 Linux iscsi Display Information Detail Information About Each iscsi Device on Each Controller /proc/scsi/iscsi log]# cd /proc/scsi/iscsi iscsi]# ls 1 [root@nt5-ge iscsi]# cat 1 # iscsi driver version: (28-Feb-2003) # # SCSI: iscsi: # Bus Tgt LUN IP address Port TargetName iqn com.cisco.00.ec5bed7e2793a c585bbc09.llb5a iqn com.cisco.00.b beee6ea95f130f98c1bcd5.llb5a iqn com.cisco.00.ebd3b01484d3db4ddfaede782ab8a4f6.llb5a iqn com.cisco.00.e9829d85e3637c d6fd7967f.llb5a iqn com.cisco.00.18e8e7f6afc9bc871265c a7.llb5a iqn com.cisco.00.18e8e7f6afc9bc871265c a7.llb5a iqn com.cisco.00.18e8e7f6afc9bc871265c a7.llb5a iqn com.cisco.00.18e8e7f6afc9bc871265c a7.llb5a58 96

49 Please Complete Your Evaluation Form Session , Cisco Systems, Inc. All rights reserved. 98

50 EXTRAS on Troubleshooting MDS IPS-8 99 MDS IPS-8 iscsi and FCIP configuration troubleshooting A look at important output displays Best practices for debugging 100

51 Configured and Auto Import Targets in the iscsi Domain Manually configured targets Show iscsi local-node shows if target is manually configured or is auto-created 101 Displays: iscsi Session Auto iscsi Targets iscsi Discovery Session Manual iscsi Targets 102

52 Displays: Auto Configured iscsi Initiators iscsi configured-initiators are iscsi hosts that are automatically configured by the MDS The IQN node name will be unique for all iscsi hosts but the assigned pwwn can change TIP: This Unique iqn Number Can Be Used to Statically Configure the Initiator 103 Displays: Zoning Options Target Zone Members Tip: Use symbolic iqn Name Because This Will Not Change but wwn Could iscsi Initiator Zone Member 104

53 Displays: Details on iscsi Initiator in the iscsi Domain nwwn Auto Assigned and Can Be Used to Configure Static Entry; This Would be Required if WWN Access Is Used on Storage Array pwwn Auto Assigned and Can be Used for Static Config Assigned Targets 105 Displays: Show iscsi Interface iscsi Sessions with Targets on This Interface; TCP Sessions Also Includes iscsi Discovery Sessions MTU Path Discovery, if Using Large Frames Like Jumbo on the Host This Needs to be Set Tx Window Size as Determined by Host Setting Selective Acknowledgement Feature Disabled by Default iscsi PDU Counts 106

54 Interface Displays Transfer Rates Active TCP Connections; Port 3260 is iscsi Discovery Session and 1 iscsi Target Session 107 Displays: Specific iscsi Show Commands Configured-initiator Global information Local-node Port Remote-node Session Stats Show iscsi configured initiators Show iscsi global config Show iscsi local nodes Show iscsi FV port information Show iscsi node information Show iscsi session information Show iscsi statistics 108

55 show iscsi remote-node detail iscsi Host Created in MDS Configuration and Was Placed in VSAN 4 iscsi Host Assigned pwwn; Info on What GiGE Port Connection Is Coming in on 109 iscsi Debugs Requires attachment to module debug ips iscsi MDS9200A# Attach Module 2 Attaching to Module 2... To Exit Type Exit', to Abort Type '$.' Module-2# all api buf error event fcpevent fcpframe fcpfsm Turn on all ISCSI debug Configure ISCSI API debugging Configure ISCSI buffer debugging Configure ISCSI error debugging Configure ISCSI event debugging Configure ISCSI fcp event debugging Configure ISCSI fcp frame debugging Configure ISCSI FCP FSM debugging fsm Configure ISCSI FSM debugging login Configure ISCSI login debugging parse Configure ISCSI parsing debugging pkt Configure ISCSI packet debugging task Configure ISCSI transaction debugging transaction Configure ISCSI transaction debugging warn Configure ISCSI warning debugging Most debugs require a port or session for information you are wanting to debug 110

56 FCIP Interface MDS-MDS TE Port Connected to Another MDS Special Frame Verification and Will Display Peer Switch wwn MDS to MDS Uses 2 TCP Connections One Connection Is F-Class Flow Second Connection Is the Data Timestamp Information and Acceptable Difference Would be Displayed Here TCP MTU, Can Be Adjusted Up to 8K 111 Show TCP stats MDS9509# show ips stats tcp interface gigabitethernet 2/8 TCP Statistics for port GigabitEthernet2/8 Connection Stats 6 active openings, 8 accepts 6 failed attempts, 0 reset received, 8 established Segment stats received, sent, 109 retransmitted 0 bad segments received, 0 reset sent Excessive Retransmits Indicates Possible Core Drops and/or tcp Window Size Should Be Adjusted TCP Active Connections Local Address Remote Address State Send-Q Recv-Q : :65519 ESTABLISH : :65521 ESTABLISH : :0 LISTEN 0 0 Used for Class F Control Traffic Only; FCIP Uses Port 3225 Data Traffic Connection Only Connection You Would See for Both Class F and Data for a B-Port Connection TCP Listen Port Ready for New tcp Connections 112

57 Show counters MDS9509# show interface fcip 1 counters bri Interface Input (rate is 5 min avg) Output (rate is 5 min avg) Rate Total Rate Total Mbits/s Frames Mbits/s Frames fcip Frames averaged over 5 minutes and total count of frames since last clear counters or since tunnel up 113 debug ips ipstack IP debugging for iscsi and FCIP all Configure ALL IP Stack debugging in Configure IP Stack IN debugging arp Configure IP Stack ARP debugging ip Configure IP Stack IP debugging arp-detail Configure verbose IP Stack ARP debugging ip-detail Configure verbose IP Stack IP debugging error Configure IP Stack Error debugging route Configure IP Stack route debugging eth Configure IP Stack Ethernet debugging sdip Configure IP Stack SDIP debugging eth-cfg eth-chan eth-detail icmp if Configure IP Stack Ethernet Configuration debugging Configure IP Stack Ether Channel debugging Configure verbose debugging of Ethernet layer Configure IP Stack ICMP debugging Configure IP Stack interface debugging tcp udp udp-detail vrrp vrrp-detail Configure IP Stack TCP debugging Configure IP Stack UDP debugging Configure verbose IP Stack UDP debugging Configure IP Stack VRRP debugging Configure verbose IP Stack VRRP debugging 114