DC Compute SAN. Carlos Lopez CCIE SAN, DC #21063 David Kester CCIE SAN #19555 Ed Mazurek CCIE SNA/IP, SAN #6448

Transcription

1 DC Compute SAN Carlos Lopez CCIE SAN, DC #21063 David Kester CCIE SAN #19555 Ed Mazurek CCIE SNA/IP, SAN #6448

2 Introductions Carlos Lopez CCIE SAN, DC #21063 Technical Leader TAC David Kester CCIE SAN #19555 Team Leader Storage Networking Ed Mazurek CCIE SNA/IP, SAN #6448 Technical Leader TAC 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 2

3 Agenda Components Topology Nexus FC NPV vs FCoE-NPV Bugs MDS Slow Drain Troubleshooting Enhancements

4 Components

5 Cisco Multi-Protocol Product Portfolio: SAN, LAN, and Compute LAN/SAN SAN 16G COMPUTE Cisco Nexus 7000 Cisco Nexus 9000 Cisco MDS x16G Line-Rate FC 48x10GE Line-Rate FCoE Cisco UCS 6300 Series FI Cisco UCS 6200 Series FI Cisco Nexus 5500 Nexus 5672UP-16G Cisco Nexus 5600 Cisco MDS 9396S Cisco MDS 9148S Cisco MDS 9718 Cisco Nexus G FC: Nexus 2348UPQ Cisco Nexus x40GE Line-Rate FCoE Cisco MDS 9250i 12+ Years of Proven NX-OS Operating System Cisco Prime Data Center Network Manager (DCNM) Cisco UCS B-Series Blade Servers Cisco UCS C-Series Rack Servers Consistent, Simplified Features, Management, and Programmability 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

6 Cisco MDS 9000 Series 16G FC Director Switches Cisco 48x10G Line-rate FCoE Module Cisco 48x16G Line-rate FC Module Future Proof Reliable Multi-Protocol Flexibility Investment Protection Ease of Management Cisco MDS 9718 Director Cisco MDS 9710 Director Cisco MDS 9706 Director Cisco 24x40G FCoE Line-rate FCoE Module Driving Innovations for the Next Decade with a complete 16G Portfolio Deploy Small, Medium, Large SANs with Cisco MDS 9000 Family 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

7 Cisco MDS 9000 Series 16G FC Fabric Switches Cisco MDS 9250i 16G Multi-Service Fabric Switch Cisco MDS 9396S 16G FC Fabric Switch Cisco MDS 9148S 16G FC Fabric Switch Pay-as-You-Grow Enterprise Class Features Reliability Multi-Protocol Flexibility Ease of Management Driving Innovations for the Next Decade with a complete 16G Portfolio Deploy Small, Medium, Large SANs with Cisco MDS 9000 Family 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

8 Cisco MDS 9700 Directors Comparison 26 RU Winning Points 32G FC line-rate ready Interchangeable line cards Redundant hardware Common PSUs, Linecards Single OS, Management Better UCS Interoperability 9 RU 14 RU Hardware Feature MDS 9706 MDS 9710 MDS 9718 Line Card slots Line rate 16Gbps FC or 10 Gbps FCoE Line rate 40 Gbps FCoE Fabric Module slots (available / default) 6 / 3 6 / 3 6 / 6 Sup Slots Fabric Module location Rear Rear Rear Airflow Front to Back Front to Back Front to Back Power Supply slots Power Consumption (Typical/Max) 2425W / 2620W 4615W / 5020W 4742/8462W 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

9 Converged FEX Architecture Enabling Director-class resiliency at Converged Access Multi-Protocol Storage and Host Connectivity: FC, FCoE and IP Converged Architecture includes Cisco Data center portfolio: SAN(MDS 9700), LAN(Nexus 2k-7k) and Compute (UCS Chassis) accessibility 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

10 MDS 9250i Multiservice Fabric Switch Multi-Protocol Support Features 16G FC, 10GE FCoE, 1GE/10GE FCIP, iscsi Intelligent Storage Services for FC and FCoE SANs Fiber Channel over IP (FCIP) IO Accelerator (IOA) Data Mobility Migration (DMM) Integrated Management via Data Center Network Manager (DCNM) FICON Certified Benefits Single Platform for deploying Storage Services across FC, FCoE and IP based Storage Area Networks (SANs) High-Bandwidth SAN Extension across MAN/WAN Vendor independent array migration tools Interoperate data between FC and FCoE arrays 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

11 MDS 9250i Overview Next Generation Multiservice Intelligent Services-oriented Fabric Switch Provides FCIP, IOA and DMM Integrated 40x16G FC, 8x10GE FCoE, and 2x1GE/10GE FCIP/iSCSI ports Enclosure: 2 RU; Redundant and hot-swappable power supplies and fan trays Console USB Mgmt Cisco and/or its affiliates. All rights reserved. Cisco Public

12 Cisco MDS 9148S Fabric Switch Front Back 1 RU 48 x 16G FC Line Rate Performance 12- to 48-ports in 12-port increments Dual Power Supplies and Fans for Enterprise-Class Availability VERSATILE EASY TO USE ENTERPRISE-CLASS Line-rate 16/8/4/2G FC Ports Industry-leading port range Start with 12-port base Scale up with 12-port license Or, full 48-port option available Automated Provisioning Quick Configuration Wizard Same OS and Management across Industry s broadest SAN Portfolio Non-disruptive software upgrades Up to 32 Virtual SANs (VSANs) Inter-VSAN Routing (IVR), QOS, PortChannels, N-Port ID Virtualization (NPIV), N-Port Virtualization (NPV), Comprehensive Security Hardware-based slow-drain detection and recovery High-Performance, Easy to Deploy, Enterprise-class Fabric Switch 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

13 Introducing MDS 9396S 96-Port 16G Fabric Switch 2 RU Cisco MDS 9396S Versatile Easy to Use Enterprise-Class Start with 48-port base; Scale up with 12-port license Or full 96-port option available Automated provisioning Quick Configuration Wizard Same OS and management across industry s broadest SAN portfolio Dual-power supplies and fans, non-disruptive software upgrades Up to 4095 B2B credits per port (MDS 9396S); up to 253 B2B credits per port (MDS 9148S) Up to 32 Virtual SAN (VSANs) Hardware-based slow-drain detection and recovery, Inter-VSAN routing, QoS, PortChannels, N-Port ID Virtualization (NPIV), N-Port Virtualization (NPV) Forward Error Correction, Link Encryption (FC TrustSec) Industry s Most Affordable 16G Fabric Switch Family 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

14 MDS 9396S B2B credits All ports in a port group can have maximum of 500 B2B credits Enterprise license enables extended credits that means up to 4095 B2B credit per port in a port group Port group can have maximum of 4150 B2B credits. Best Practice: Avoid grouping all E ports in same port group/ioslice Generic Formula: For every 1 KM distance with 1GB speed, we need.5 BB credit for standard FC frame (2112 bytes). CLI command: show port-resources module Cisco and/or its affiliates. All rights reserved. Cisco Public

15 Topologies UCS-FI N5K MDS

16 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

17 Separation makes sure that the design is highly available even when one of the fabrics goes down Cisco and/or its affiliates. All rights reserved. Cisco Public

18 Redundant redundancy is not required 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

19 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

20 Working with TAC Topology Commands Use the show topology command to display the interswitch links `show topology` FC Topology for VSAN 1 : Interface Peer Domain Peer Interface Peer IP Address(Switch Name) port-channel 6 0x62(98) port-channel (sw201a) Cisco and/or its affiliates. All rights reserved. Cisco Public

21 Working with TAC Topology Commands Use the show fcs ie command to display all the switches in the VSAN Core# show fcs ie IE List for VSAN: IE-WWN IE Mgmt-Id Mgmt-Addr (Switch-name) :01:00:0d:ec:39:19:c1 S(Rem) 0xfffc0e (sw204a) 20:01:00:0d:ec:39:1a:01 S(Rem) 0xfffc (sw202a) 20:01:00:0d:ec:fb:88:41 S(Loc) 0xfffc (sw200a) 20:01:00:2a:6a:8c:0b:01 S(Adj) 0xfffc (sw201a) Loc = Local = this switch Adj = Adjacent = connected switch 21 Rem = Remote = more than one hop away 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

22 Working with TAC Topology Commands Use the show fcns database npv command to find Cisco NPV switches. Core# show fcns database npv VSAN NPV NODE-NAME :20:01:00:0d:ec:51:06:01 NPV IP_ADDR : NPV INTERFACE :port-channel 30 CORE SWITCH WWN :20:00:00:0d:ec:24:ef:c0 CORE INTERFACE :Po Cisco and/or its affiliates. All rights reserved. Cisco Public

23 Working with TAC Topology Commands Use the show npv flogi-table command to shows where the device is attached and it s uplink. NPV# show npv flogi-table SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE fc1/ x0d :00:00:00:c9:71:04:4e 20:00:00:00:c9:71:04:4e Po Cisco and/or its affiliates. All rights reserved. Cisco Public

24 Working with TAC Topology Commands Note: Ensure both links are connected to the same upstream IP address. This tells you which upstream switch the UCS FI is connected to in case you need to connect to upstream switch to verify the connectivity. NPV# show npv internal info external-interface all grep addr: fabric mgmt addr: fabric mgmt addr: Cisco and/or its affiliates. All rights reserved. Cisco Public

25 Nexus FC NPV vs FCoE-NPV

26 Nexus FC NPV vs FCoE-NPV FC and NPV FCoE-NPV FC or FCoE FC or FCoE Nexus or MDS NPIV Nexus or MDS NPIV FC or FCoE FCoE FCoE Only FCoE FC N5K NPV Both FC and FCoE FCoE N5K FCoE-NPV 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 Nexus FC NPV vs FCoE-NPV Enabling FC and NPV FC or FCoE First Enable FCoE and then NPV N5K(config)# feature fcoe FC license checked out successfully fc_plugin extracted successfully FC plugin loaded successfully FCoE manager enabled successfully FC enabled on all modules successfully Nexus or MDS NPIV FC or FCoE N5K(config)# feature npv Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y FCoE FC N5K NPV Also enable fcoe qos 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Nexus FC NPV vs FCoE-NPV Enabling FC and NPV Actually, after enabling fcoe, then NPV, What really happens N5K(config)# feature npv Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y When the switch is reloaded in the NPV mode, Some configuration is saved: switchname management ip configuration and vrf boot variable username / password details ntp configuration callhome configuration snmp-server details feature fcoe FCoE FC or FCoE FC Nexus or MDS NPIV FC or FCoE N5K NPV 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Nexus FC NPV vs FCoE-NPV Enabling FCoE-NPV Enable fcoe-npv FC or FCoE n5k(config)# feature fcoe-npv FCoE NPV license checked out successfully fc_plugin extracted successfully FC plugin loaded successfully FCoE manager enabled successfully FCoE NPV enabled on all modules successfully No reload No reconfiguration fcoe qos still required FCoE Nexus or MDS NPIV FCoE N5K FCoE-NPV 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 Nexus FC NPV vs FCoE-NPV Comparison FC NPV FCoE-NPV Protocols FC and/or FCoE FCoE License FC_FEATURES_PKG FCOE_NPV_PKG Command feature fcoe feature npv feature fcoe-npv Write Erase Reload feature npv no Nexus Models N5K, N6K N5K, N6K, N9K FCoE QoS Required Required Disable FKA on Core for ISSU No if uplink is FC yes 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 Nested NPV Can I connect two NPV Switches? After enabling NPV, NPIV can also be enabled Connecting two Cisco NPV switches is not supported Cisco NPIV Cisco NPIV FC or FCoE Cisco NPV+NPIV FC or FCoE Cisco NPV+NPIV Cisco NPV 3 rd Party Vendor Unsupported Supported 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

32 Bugs

33 Frequent Bugs CSCun Weak CBC mode and weak ciphers should be disabled in SSH server Symptom: SSH servers on Cisco Nexus devices may be flagged by security scanners due to the inclusion of SSH ciphers and HMAC algorithms that are considered to be weak. These may be identified as 'SSH Server CBC Mode Ciphers Enabled' and 'SSH Server weak MAC Algorithms Enabled' or similar. Conditions: This issue applies to Cisco Nexus 7000, Cisco Nexus 5000 and MDS 9000 series switches. SSH functionality is enabled by default in Cisco NX-OS. The current SSH server status is displayed using the show ssh server command Cisco and/or its affiliates. All rights reserved. Cisco Public 34

34 Frequent Bugs CSCun Weak CBC mode and weak ciphers should be disabled in SSH server With the Fix: If an SSH client configured to use weak ciphers is used to log in to a Cisco device with this fix, the login may fail. The following messages are logged in the switch syslog: %DAEMON-2-SYSTEM_MSG: fatal: no matching cipher found: client 3descbc,blowfish-cbc server aes128-ctr,aes192-ctr,aes256-ctr - sshd Reconfigure any SSH clients not to use weak ciphers like 3des-cbc or blowfishcbc. DCNM uses SSH to manage Cisco devices and must be upgraded to at least 7.2(1) to work with devices with this fix. Known-fixed-releases: 7.3(0)N1(1) 7.2(1)N1(1) 7.2(0)N1(1) 6.2(11c) 5.2(8g) 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35

35 Frequent Bugs CSCue SNMP crashes on SNMP bulk get query Symptom: The SNMP process may crash with the following messages displayed in the output of show logging log %KERN-2-SYSTEM_MSG: mts_is_q_space_available_new():1416:total mtsbuf size for sap 28, exceeds limit 15 perc of kernel %KERN-2-SYSTEM_MSG: mts_acquire_q_space() failing - no space in sap 28, uuid 26 send_opc 3176, pid 3616, proc_name sctpt_rx_thr - kernel %KERN-2-SYSTEM_MSG: [sap 28][pid 4406][comm:snmpd] sap recovering failed and so Killed - kernel %SYSMGR-2-SERVICE_CRASHED: Service "snmpd" (PID 4406) hasn't caught signal 6 (core will be saved) 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 36

36 Frequent Bugs CSCue SNMP crashes on SNMP bulk get query Conditions: This bug affects both Nexus and MDS switches. It has been observed when a monitoring device is using snmp-bulk-get requests on the entity- MIB for multiple FEX modules at one time, or if there is continuous polling from multiple polling stations on slow mibs. Some examples of mibs that may be affected by continuous snmp bulk walk are: qos mib entity mib entity-fru mib bridge mib 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

37 Frequent Bugs CSCue SNMP crashes on SNMP bulk get query Workaround: A possible workaround is configuring the no snmp-server counter cache enable command. This command prevents SNMP bulk gets from getting cached via the use of MTS buffers. This will prevent the MTS buffers from getting consumed and resulting in a process crash. The result of the command is that the interface table might be slower to update the statistics (since caching is disabled). Note: This workaround is only available on Nexus 7000 switches. This defect is fixed in NX-OS releases 5.2.9, (Nexus 7000), 5.2.8g (MDS), and (Nexus 7000 and MDS), 6.0(2)N1(2a) (Nexus 5K and 6k) Further Problem Description: A possible way of verifying if you are affected by this bug is to issue the command show system internal mts buffers summary and check if notifications for sap 28 are increasing Cisco and/or its affiliates. All rights reserved. Cisco Public 38

38 Frequent Bugs CSCus MDS 9700 show tech detail missing some commands Symptom: MDS 9710 and MDS 9706 show tech detail missing some commands, like 'show running-config' and 'show startup-config'. Conditions: MDS 9710 and MDS 9706 at NX-OS 6.2(11). Workaround: Collect show tech-support all along with show tech-support detail Fixed in NX-OS 6.2(11c) and above Cisco and/or its affiliates. All rights reserved. Cisco Public 39

39 Frequent Nexus 5K/6K Bugs CSCuj Unified fc interfaces come up as Ethernet after disruptive upgrades Symptom: In Cisco Nexus 5000 series switches, a disruptive upgrade with reason incompatible image causes the Unified Ports configured as FC ports to come up as Ethernet ports after upgrade. However, the FC port configuration still exists in the running configuration. Conditions: Upgrade between any two incompatible images and the fc interfaces are unified interfaces requiring the slot and port commands, slot z port x - y mode fc Backup the configuration before upgrading 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

40 Frequent Nexus 5K/6K Bugs CSCuj Unified fc interfaces come up as Ethernet after disruptive upgrades Proactive Workaround: Do ISSU only between compatible images. Please check the result of install command for image compatibility. Reactive Workaround: After the disruptive ISSU between incompatible images, do the following: a. copy startup-config bootflash: b. copy running-config startup-config c. reload After reload: d. copy bootflash: running-config e. copy running-config startup-config Now the device should have the same configurations as before upgrade. Consult Release Notes for non-disruptive upgrade path 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

41 Frequent Nexus 5K/6K Bugs CSCup FC interfaces are not listed in IF-MIB snmp Symptom: FC interfaces are not listed in IF-MIB snmp walk. Device Manager is not working correctly with the Nexus 5548UP or 5596UP (GEM modules installed) when the expansion module ports are set to fibre channel mode. Hovering over the ports with the mouse in Device Manager will display for example, "Ethernet 1/17 Status: failed". Looking at the same ports via CLI will show that the ports are really in FC mode and not configured as Ethernet ports Cisco and/or its affiliates. All rights reserved. Cisco Public 42

42 Frequent Nexus 5K/6K Bugs CSCup FC interfaces are not listed in IF-MIB snmp Conditions: Nexus 5548UP or Nexus 5596UP running NX-OS 7.0(2)N1(1) with GEM Expansion module ports configured to operate in Fibre Channel mode Some ports are in Fibre Channel mode on the base chassis. More Info: NX-OS 7.0(1)N1(1) and all previous software versions are not affected by this defect. This is an NX-OS bug, not a Device Manager bug. Fixed in 7.0(6)N1 and above Cisco and/or its affiliates. All rights reserved. Cisco Public 43

43 Nexus 5K/6K Bug CSCur10558 Trunk Protocol Enable does not show in running config when disabled Resolution Summary: 1 Made the command [no] trunk protocol enable hidden 2. Added appropriate warning message when the command is run on CLI Fixed in 7.3(0)N1(1) 7.2(1)N1(1) 7.1(3)N1(1) 7.0(7)N1(1) 6.2(9) (config-if)# no trunk protocol Warning: This will globally disable the switch's ability to form any trunks and impacts existing trunk ports Do you wish to continue(y/n)? [n] 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

44 Nexus 5K/6K Bug CSCur10558 Trunk Protocol Enable does not show in running config when disabled [no] trunk protocol is not in the running config.. or the show tech detail. Tip: show tech will contain show port internal info all You should always have... Epp state: Enabled Port Trunking Protocol (PTP) and Port Channel Protocol (PCP) use the EPP frame. No practical reason to disable fibre channel trunking Cisco and/or its affiliates. All rights reserved. Cisco Public 45

45 Cisco MDS NX-OS 7.3(0)D1(1) OUI Enhancement Example: Adding OUIs Switch(config)# wwn oui 0x10001c OUI - A 24 bit globally unique number assigned by IEEE. Port-channel functionality includes Cisco OUI check of peer switch Cisco and/or its affiliates. All rights reserved. Cisco Public 46

46 MDS Slow Drain Troubleshooting Enhancements

47 MDS Slow Drain Troubleshooting Enhancements NX-OS 6.2(9) and 6.2(13) added several enhancements system timeout no-credit-drop triggered at exact time by HW TxWait slowport-monitor New port-monitor counters txwait tx-slowport-oper-delay tx-slowport-count show tech-support slowdrain DCNM Slow Drain Analysis TAC tool MDS_show_tech_slowdrain_analysis For a more comprehensive information see: - SAN Congestion! Understanding, Troubleshooting, Mitigating in a Cisco Fabric (2016 Las Vegas) Cisco and/or its affiliates. All rights reserved. Cisco Public 48

48 system timeout no-credit-drop Triggered by HW at exact time no-credit-drop causes frames to be dropped immediately if the destination port is at 0 Tx credits for the time specified system timeout no-credit-drop 200 mode f Previously no-credit-drop was triggered by SW process at 100ms intervals NX-OS 6.2(9) and later triggered by the HW at exact time the threshold is reached Should be used in conjunction with lowering congestion-drop threshold Recommended for F ports Can drastically improve ISL performance under slow drain conditions xxx_force_timeout_on/off counter By default no-credit-drop is not enabled 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

49 TxWait enhancements txwait txwait is a counter that increments every 2.5us when port is at 0 Tx credits and there are frames queued for transmit txwait * 2.5 / = seconds of time the port was unable to transmit Only applies to the following: MDS 9500 with generation 4 linecards: MDS 9000 Family 32-Port 8-Gbps Advanced Fibre Channel Switching Module (DS-X K9) MDS 9000 Family 48-Port 8-Gbps Advanced Fibre Channel Switching Module (DS-X K9) MDS Port 16-Gbps Fibre Channel Switching Module (DS-X K9) MDS 9148S 16G Multilayer Fabric Switch MDS 9250i Multiservice Fabric Switch MDS 9396S 16G Multilayer Fabric Switch Others will return zero 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

50 TxWait enhancements - continued txwait can be seen in the following: show interface counters Raw value in 2.5us units show interface counters Percentage Tx credits are available for last 1s/1m/1h/72h show process creditmon txwait-history 60sec, 60min, 72hour graphs show logging onboard txwait SNMP fciftxwaitcount variable 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 51

51 TxWait enhancements - continued txwait - show interface counters mds9710-1# show interface fc1/13 counters i fc wait fc1/ us Txwaits due to lack of transmit credits * 2.5 / = seconds Cumulative since the interface counters were last cleared The above indicates the MDS was not able to transmit for over 15 seconds since the counters were last cleared 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 52

52 TxWait enhancements - continued txwait - Percentage Tx credits are available for last 1s/1m/1h/72h Utilizes the underlying txwait counter MDS9710-1# show interface fc1/13 counters fc1/13 5 Transmit B2B credit transitions to zero 2 Receive B2B credit transitions to zero us TxWait due to lack of transmit credits Percentage Tx credits not available for last 1s/1m/1h/72h: 1%/5%/3%/2% 32 receive B2B credit remaining 128 transmit B2B credit remaining 128 low priority transmit B2B credit remaining 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 53

53 Level 1: Latency - Troubleshooting txwait - show logging onboard txwait MDS9513# show logging onboard txwait module Module: 4 txwait count Notes: - Sampling period is 20 seconds - Only txwait delta >= 100 ms are logged Recorded every 20 seconds only when >= 100ms Interface Delta TxWait Time Congestion Timestamp 2.5us ticks seconds fc4/ % Wed May 27 13:20: fc4/ % Wed May 27 13:19: fc4/ % Wed May 27 13:19: fc4/ % Wed May 27 13:19: Delta values recorded when they are more than 100ms in the 20 second interval 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 54

54 Level 1: Latency - Troubleshooting txwait-history Graphical display of time where Tx credits are not available Similar in format to cpu history 3 graphs per port Last 60 seconds Last 60 minutes Last 72 hours Utilizes the underlying txwait counter mds9710-1# show process creditmon txwait-history module 1 port 13 TxWait history for port fc1/13: ============================== # ## 900 # ## 800 ## ## 700 ## ## 600 ### ### 500 ### ## ### 400 ### ## #### 300 ### ## #### 200 ### ## #### 100 ### ## #### Credit Not Available per second (last 60 seconds) # = TxWait (ms) 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

55 Level 1: Latency - Troubleshooting slowport-monitor system timeout slowport-monitor <1-500> mode e f Must be configured! Events are captured every 100ms Last 10 events per port captured in slowport-monitor-events Logging onboard slowport-monitor-events captures more events Currently implemented for: Gen 3 LCs - DS-X K9 and DS-X92xx-96K9 modules Gen 4 LCs - DS-X K9 and DS-X K9 modules 9700 & 9396S (Gen 5) 9250i & 9148S Differences exist between Gen3, Gen4 and 9700/9250i/9148S/9396S 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 56

56 Level 1: Latency - Troubleshooting slowport-monitor 9700/9250i/9148S/9396S (Gen 5 LCs) Gen5/9250i/9148S/9396S have enhanced HW capabilities Each 100ms interval the number of times Tx credits remained at 0 for the configured(admin) delay is counted. The average operational delay is determined This is how long the port was at 0 Tx credits Recorded when at least one complete event occurred More events available via logging onboard slowport-monitor-events MDS9710-1# show process creditmon slowport-monitor-events Module: 01 Slowport Detected: YES ========================================================================= Interface = fc1/ admin slowport oper Actual Timestamp average delay delay detection delay (ms) count (ms) /01/15 23:03: /01/15 23:03: /01/15 23:03: /01/15 23:03: Configured delay(5ms) 4 events in last 100ms Note: Oper delay limited by nocredit-drop threshold 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 57

57 Slow Drain Alerting and Mitigation Port-monitor alerting Port-monitor allows monitoring of several counters relating to slow drain credit-loss-reco Credit loss recovery counter lr-rx The number of link resets received by the fc-port lr-tx Link resets transmitted by the fc-port timeout-discards Timeout discards counter New in 6.2(13) tx-credit-not-available Credit not available counter(in 100ms increments) tx-discards Tx discards counter tx-slowport-count Number of slowport events tx-slowport-oper-delay Slowport operational delay txwait Amount of time at 0 Tx credits and packets queued rx-datarate Rx data rate as a percentage of link speed tx-datarate Tx data rate as a percentage of link speed Note: There are other counters that are valuable and should also be considered for inclusion in monitoring but are not part of slow drain 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 58

58 Troubleshooting Documentation for TAC show tech-support slowdrain Contains all the commands available that pertain to slow drain Contains context commands to understand the FC topology Contains name server commands to identify devices Contains active zonesets to understand device relationships Most useful when run from DCNM and gathered for the entire fabric SAN Client -> Tools -> Run CLI Commands When opening up a case with the TAC please have this available! Used for MDS_show_tech_slowdrain_analysis 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

59 DCNM Slow Drain Analysis DCNM 7.1(1) added Slow Drain Analysis DCNM 7.2(2) added improvements DCNM 10.0(1) added improvements Used for pulling fabric wide slow drain counters for a defined period of time Useful for ongoing slow drain problems Accessed from the Web Client Health -> Diagnostics -> Slow drain Analysis 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

60 MDS_show_tech_slowdrain_analysis Red indicates level 3 (creditloss) Slow draining end device! Yellow indicates level 1 (latency) Arrows indicate direction of congestion Orange indicates level 2 (timeout drops) Green indicates no congestion 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

61 Thank you