FCoE Cookbook for HP Virtual Connect

Transcription

1 Technical whitepaper FCoE Cookbook for HP Virtual Connect Version 4.45 Firmware Enhancements August 2015 Table of contents Change History 6 Purpose 7 Overview 7 Requirements and support 7 Supported Designs 7 Virtual Connect support 9 Upstream Switch support 11 Virtual Connect Restrictions 13 Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets 18 Benefits 18 Considerations 19 Requirements 20 Guidelines 20 Virtual Connect configuration 20 Nexus configuration 20 Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel) 26 Benefits 26 Considerations 27 Requirements 28 Guidelines 28 Virtual Connect configuration 28 Nexus configuration 29 Click here to verify the latest version of this document

2 Scenario 3: Dual-Hop FCoE using Multi- FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel) 34 Benefits 35 Considerations 35 Requirements 36 Guidelines 37 Virtual Connect configuration 37 Nexus configuration 37 Scenario 4: Multi-Hop FCoE using Multi- FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel) 44 Benefits 45 Considerations 45 Requirements 46 Guidelines 47 Virtual Connect configuration 47 Nexus and MDS configuration 47 Scenario 5: Dual-Hop FCoE using Multi- FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches. 57 Benefits 58 Considerations 58 Requirements 59 Guidelines 59 Virtual Connect configuration 60 Nexus configuration 60 Scenario 6: Dual-Hop FCoE using Multi- FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel) 66 Benefits 67 Considerations 67 Requirements 68 Guidelines 69 Virtual Connect configuration 69 Nexus configuration 69

3 Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target 75 Benefits 75 Considerations 76 Requirements 77 Guidelines 77 Virtual Connect configuration 77 HP 5900 configuration 77 Scenario 8: Dual-Hop FCoE using Multi- FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches 91 Benefits 91 Considerations 92 Requirements 93 Guidelines 93 Virtual Connect configuration 94 HP 5900CP Switch configuration 94 Scenario 9: Multi-Hop FCoE using Multi- FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel) 106 Benefits 107 Considerations 107 Requirements 108 Guidelines 108 Virtual Connect configuration 109 HP 5900 and Nexus Switch configuration 109 FCoE configuration in Virtual Connect 124 Standard design 124 Alternative design 125 Converged Shared Uplink Sets (SUS) Details and Restrictions 125 Defining two Converged Shared Uplink Sets 126 Configuring a maximum FCoE bandwidth limit on servers 129 Quality of Service Configuration 131 QoS on Nexus 5010 and

4 QoS on Nexus 5500 and 6000 series 132 Quality of Service in the Virtual Connect Domain 133 Adding a FCoE volume in a multi-hop FCoE environment 138 With HP 5900 switches 138 With Cisco Nexus switches 144 Adding a FCoE volume under VMware ESX 145 Adding a FCoE volume under Windows Server Virtual Connect Troubleshooting 152 Troubleshooting DCBX issues 152 Troubleshooting FCoE issues with FIP Snooping information 159 Nexus Troubleshooting commands 163 Useful Show commands 163 Specific NPV Mode CLI commands 172 Per-Priority Flow Control commands 175 Interface Status commands 176 HP Networking Troubleshooting commands 180 Useful FC/FCoE commands 180 Useful interface commands 182 Useful QoS commands 185 Useful LLDP commands 187 Useful log commands 189 Specific NPV Mode CLI commands 190 Issues with FCoE performance 191 Possible Cause 191 When slow FCoE performance is not due to Pause Frames 191 When slow FCoE performance is due to Pause frames transmitted by the Nexus switch. 192 When slow FCoE performance is due to Pause frames transmitted by VC/Servers. 193 Troubleshooting Common Issues 195 VMware vsphere not detecting FCoE adapters 195

5 Troubleshooting using server adapter management suite 196 Emulex OneCommand Manager 196 FCoE Frames Analysis 198 Appendix A: FCoE Port Channel limitations on Nexus switches 201 Identifying ports that belong to the same Nexus Unified Port Controller 201 Acronyms and abbreviations 204 FC/FCoE Port Types 205 FCoE Port Types 205 FC Port Types 205 Support and Other Resources 206 Contacting HP 206 Documentation feedback 207 Related documentation

6 Change History The following Change History log contains a record of changes made to this document: Publish / Revised Version # Section / Nature of change August 2015 VC 4.45 Changed recommended vfc binding method for all Cisco scenarios Added the minimum firmware version required to support FCoE on 40Gb uplink ports Added the VC FlexFabric-20/40 F8 Module s 40Gb uplink ports to the support list Added support for Nexus 6000 Series Added information on Port Chanel limitation with Nexus 6000 Series Added information on HP OneView Added information on FCoE Multi-Hop connectivity support with HP 5900 Added note on Uplink Ports in SUS showing Linked-Standby if no server profile is using the network Updated some URLs with hpe.com Other minor changes May 2014 VC 4.20 Added new Dual/Multi-Hop scenarios with 5900 switches June 2013 VC 4.01 Added new Dual/Multi-Hop scenarios with Nexus switches Added information on HP Virtual Connect FlexFabric-20/40 F8 Modules Added information on new CNAs Added troubleshooting DCBX/FIP snooping/fcoe issues Added HPN Troubleshooting commands Added the section Adding an FCoE volume in a Multi-Hop FCoE environment Other minor changes 6 - Change History

7 Purpose This guide provides concept, deployment guidelines and use case scenarios for HP Virtual Connect when using Fibre Channel over Ethernet through FIP Snooping under the T11 FC-BB-5 specification. Overview Fibre Channel over Ethernet (FCoE) through FIP Snooping under the T11 FC-BB-5 specification is supported on the HP Virtual connect modules since VC 4.01, this feature allows the FCoE traffic to be propagated out of the enclosure to an external bridge which will handle the conversion of FCoE to FC traffic or back to FCoE traffic if the bridge is either directly connected to an FCoE storage target or to an FCoE Forwarder in a Multi-Hop FCoE implementation. The traditional model of completely parallel, autonomous data and storage networks with dedicated interface cards, switches, and cabling plants can be costly and inefficient. Enterprises are looking to consolidate server and storage connectivity to reduce equipment and operations expenses, decrease clutter and complexity, and make more efficient use of shared networking resources while ensuring continuity of service. In addition to having the potential to reduce complexity at the network edge with Dual-Hop scenarios, FCoE also promises to reduce equipment costs beyond the access layer by enabling, with Multi-Hop scenarios, convergence in the extended network. Requirements and support Supported Designs Dual Hop FCoE designs are supported. LAN SAN FCoE FCoE Dual Hop FCoE means there are two FCoE hops between the server and the conversion point from FCoE to FC. The first hop is between the server CNA adapter and the VC Module. The second hop is between the VC Module and the external FCoE to FC bridge device. 7 - Purpose

8 Figure 1: Dual Hop FCoE Design Server FC Storage Array FCoE FCoE Native FC Virtual Connect (FIP Snooping) FCoE Forwarder Dual-Hop FCoE Multi Hop FCoE technology can also be used to extend convergence beyond the external FCoE device. Multi Hop FCoE means there are more than two FCoE hops between the server and the conversion point from FCoE to FC. This extended convergence design offers more flexibility and capital/operational cost savings. LAN SAN FCoE FCoE FCoE FCoE However given the critical nature of storage networking and its central importance to the integrity of the end-to-end data center architecture, customers should carefully evaluate the implications of Multi Hop solutions when considering more extensive FCoE deployments. 8 - Requirements and support

9 Figure 2: Multi Hop FCoE Design Server FCoE FCoE FCoE Native FC FC Storage Array Virtual Connect (FIP Snooping) DCB Device FCoE Forwarder Multi-Hop FCoE Virtual Connect support VC Module support Extending FCoE through FIP Snooping with HP Virtual Connect is only supported with the following modules: HP Virtual Connect FlexFabric-20/40 F8 Module HP Virtual Connect FlexFabric 10Gb/24-port Module HP Virtual Connect Flex-10/10D Ethernet Module Firmware requirement FCoE through FIP Snooping using FC-BB-5 is supported on the HP Virtual connect modules since VC Multi-FCoE Network per Uplink Set support requires VC 4.20 or later. VC 4.30 is the minimum firmware version required to support FCoE on 40Gb uplink ports. 9 - Requirements and support

10 FCoE-capable uplink ports HP Virtual Connect FlexFabric 10Gb/24-port Module supports FCoE only on uplink ports X1 to X4 HP Virtual Connect FlexFabric-20/40 F8 Module supports FCoE on all uplink ports (X1 to X8 and Q1 to Q4) HP Virtual Connect Flex-10/10D Ethernet Module supports FCoE on all uplink ports (X1 to X10) VC FlexFabric 10Gb/24-port VC Flex-10/10D X1 X4 Uplink ports available for FCoE connection X1 X10 Uplink ports available for FCoE connection VC FlexFabric-20/40 F8 Q1-Q4 and X1-X8 10Gb and 40Gb Uplink ports available for FCoE connection FcoE-enabled uplink port limitation Long Range Ethernet Optical transceivers (LR SFP+) are not supported on Virtual Connect FCoE-enabled uplink port. 40Gb FCoE connection is supported with 1x40G only. Server and enclosure limitations HP ProLiant BL2x220c and HP BladeSystem c3000 enclosure are not supported. HP OneView Virtual Connect Dual-Hop and Multi-Hop FCoE is not currently supported by HP OneView ( So if your VC Domain is configured with Shared Uplink Set using FCoE network(s), do not transition to HP OneView management as HP OneView does not support Dual-Hop and Multi-Hop FCoE. If you want to transition to HP OneView to take advantages of all innovative new features, you can configure FC connectivity under HP OneView using native FC connections instead of FCoE connections Requirements and support

11 Upstream Switch support Supported upstream switch models you can connect to the VC modules are listed in the Virtual Connect Connectivity stream documents available on the following HP s SPOCK web page: then select the Virtual Connect FCoE Modules section. (Requires HP Passport; if you do not have an HP Passport account, follow the instructions on the webpage). In short, FCoE connection on Virtual Connect uplink ports is currently supported by HP with: Cisco Nexus 50xx (i.e. Nexus 5010 and 5020) Cisco Nexus 55xx (i.e. Nexus 5548UP and 5596UP) Cisco Nexus 600x (i.e. Nexus 6001P and Nexus 6004EF) HP 5900 switch series (i.e. HP FlexFabric 5900CP, HP 5900AF) Note: HP does not provide services and installation support for Brocade VDX upstream switches. However, we do recognize Brocade VDX as a viable third party upstream FCoE multi-hop option with Virtual Connect. Brocade has released a deployment guide providing use cases for Brocade VDX/VCS Fabrics and HP Virtual Connect. See Brocade VDX Switches and HP Virtual Connect, configuring FCoE Multi-Hop, Contact Brocade for additional details and support. HP C-series Cisco Nexus support The following describes the different scenarios currently supported by HP with Nexus 50xx and Nexus 55xx: VC<------>Nexus FCF<------>FC or FCoE target VC<------>Nexus FCF/NPV<------>MDS FCoE<------>FC or FCoE target VC<------>Nexus FCF/NPV<------>MDS FC<------>FC target VC<------>Nexus NPV<------>Brocade FC<------>FC target Variants of the above with more Cisco FC or FCoE hops are supported up to the Cisco hop limit. To learn what HP C-series switches and what versions of NX-OS support Virtual Connect, visit the C-Series FCoE Switch Connectivity stream from the SPOCK site For more information about supported upstream switches you can connect to HP C-series switches, visit the C-Series FCoE Switch Connectivity stream and C-Series FC Switch Connectivity stream from the SPOCK site These HP s SPOCK documents provide detailed information on recommended NX-OS versions, supported upstream FC/FCoE switches, supported Virtual Connect modules, transceivers, Storage System, etc. HP FlexFabric 5900CP support The following describes the different scenarios currently supported by HP with HP FlexFabric 5900CP: VC<------>5900CP FCF<------>FC or FCoE target VC<------>5900CP NPV<------>FCoE target VC<------>5900CP NPV<------>Nexus 5k FCoE<------>FC or FCoE target VC<------>5900CP NPV<------>B/C/H-Series FC<------>FC target 11 - Requirements and support

12 Variants of the above with more FC or FCoE hops are supported up to the Cisco/Brocade hop limit. VC<------>5900CP<------>5900<--> <-->5900<------>FCoE target HP supports a maximum of seven FCoE hops (up to 8 HP switches including VC). To get details about the HP FlexFabric 5900CP supported with your Virtual Connect module, visit the Virtual Connect Connectivity stream documents from the SPOCK site in the Virtual Connect FCoE Modules section. For more information about supported upstream switches you can connect to HP FlexFabric 5900CP switches and to learn what versions of the firmware support Virtual Connect, visit the HP Networking Switches Connectivity stream at This HP S SPOCK document provides detailed information on recommended Firmware versions, supported upstream FC/FCoE switches, supported Virtual Connect modules, transceivers, Storage System, etc. HP 5900AF support The following describes the different scenarios currently supported by HP with HP 5900AF: VC<------>5900AF NPV/FCF<------>FCoE target VC<------>5900AF NPV<------>Nexus 5k FCoE<------>FC or FCoE target Variants of the above with more FCoE hops are supported up to the Cisco hop limit. VC<------>5900AF<------>5900<--> <-->5900<------>FCoE target HP supports a maximum of seven FCoE hops (up to 8 HP switches including VC). To get details about the HP FlexFabric 5900AF supported with your Virtual Connect module, visit the Virtual Connect Connectivity stream documents from the SPOCK site in the Virtual Connect FCoE Modules section. For more information about supported upstream switches you can connect to HP FlexFabric 5900AF switches and to learn what versions of the firmware support Virtual Connect, visit the HP Networking Switches Connectivity stream at This HP S SPOCK document provides detailed information on recommended Firmware versions, supported upstream FC/FCoE switches, supported Virtual Connect modules, transceivers, Storage System, etc Requirements and support

13 Virtual Connect Restrictions The following restrictions apply to VC modules and switches from different vendors: Up to 32 FCoE networks can be associated with any single set of uplink ports using the Multi-FCoE Network per Uplink Set support in VC 4.20 or later. With multi-fcoe Network per Uplink Set support, customers can benefit from Virtual SANs (VSANs) support but can also have the ability to provide connectivity to multiple types of SAN from the same Virtual Connect set of ports. Figure 3: Virtual Connect SUS configuration when using FCoE networks Converged-SUS defined to pass FCoE traffic and traditional Ethernet traffic VC Ethernet networks SUS-1 FCoE-1 FCoE-2 VLAN-30 VLAN-20 VLAN-10 FCoE-3 LACP Lossless FCoE Networks FCoE-capable Shared Uplink Set (SUS) is used to allow concurrent Ethernet and lossless FCoE traffic. Figure 4: Sample of a Virtual Connect SUS with 2xFCoE networks and 3xEthernet networks FCoE Networks Lossless Ethernet Networks o An FCoE network cannot be created independently outside of a Shared Uplink Set. o The number of FCoE networks is additionally limited by the 1000 networks Virtual Connect domain limit Virtual Connect Restrictions

14 For each FCoE VLAN, a unique FC VSAN must be created on the switch and association between the two must be strictly configured as 1-to-1 (i. e. VLAN x maps with VSAN y and to increase the FCoE traffic identification, we recommend to use x=y, e.g. VLAN 200 maps with VSAN 200). Figure 5: VSAN Support when using multi-fcoe networks Server 1 Server 2 FCoE vnet 200 (VLAN 200) FCoE vnet 300 (VLAN 300) Shared UplinkSet X1 X2 Eth1 Eth2 Map VLAN 200 > VSAN 200 Map VLAN 300 > VSAN 300 Map VLAN 400 > VSAN 400 VSAN Trunk 200,300,400 VSAN Trunk 200,300,400 VSAN Membership control FC Port for VSAN 200 FC Port for VSAN 300 Storage Target 1 Storage Target 2 FCoE FC Server 3 FCoE vnet 400 (VLAN 400) Virtual Connect (FIP Snooping mode) FCF Switch SAN Switch FC Port for VSAN 400 Storage Target 3 Note: Mapping of a single VLAN into multiple VSANs is not supported. Note: Like all other vendors, Virtual Connect supports a single lossless Priority Flow Control (PFC) queue for all FCoE networks that are defined on any specific shared uplink set. Sharing PFC queue between multiple VSANs will make it possible that traffic from one VSAN may have impact on traffic from other VSANs. For that reason, it is highly recommended to create enough SAN subscription to avoid any over-subscription issues that would prevent SAN to pause and therefore to impact other VSANs Virtual Connect Restrictions

15 Virtual Connect requires the creation of an Active/Active Shared Uplink Set configuration for the FCoE traffic. Figure 6: Active-Active FCoE Virtual Connect configuration FCoE ToR Switch 1 FCoE ToR Switch 2 FCoE Active Lossless Ethernet Network FCoE Active SUS-1 SUS-2 VC Domain FCoE ToR Switch 1 FCoE ToR Switch 2 Lossless Ethernet Network FCoE FCoE SUS VC Domain FCoE-capable Shared Uplink Sets must always contain ports from a single VC module in order to maintain the SAN-A / SAN-B isolation. Figure 7: FCoE-capable Shared Uplink Set configuration support FCoE ToR Switch 1 FCoE ToR Switch 2 Lossless Ethernet Network FCoE FCoE SUS-1 SUS-2 VC Domain FCoE ToR Switch 1 FCoE ToR Switch 2 Lossless Ethernet Network FCoE FCoE SUS-1 SUS-2 VC Domain 15 - Virtual Connect Restrictions

16 FCoE-capable Shared Uplink Sets must always be connected to a single Top of Rack switch in order to maintain the SAN-A / SAN-B isolation. Figure 8: FCoE-capable Shared Uplink Set LACP configuration support FCoE ToR Switch 1 FCoE ToR Switch 2 Active FCoE LACP Lossless Ethernet Network LACP Active FCoE SUS-1 SUS-2 VC Domain FCoE ToR Switch 1 FCoE ToR Switch 2 Active FCoE LACP Lossless Ethernet Network LACP Active FCoE SUS-1 SUS-2 VC Domain o Ethernet forwarding through link aggregation can break SAN A/B separation. o This limitation remains true even when technology like HP irf, Cisco vpc are used Virtual Connect Restrictions

17 FCoE traffic is not allowed to cross stacking links. In a Multiple Enclosure VC Domain, VCM automatically reserves all same uplink ports on each VC module when creating a SUS that contains associated FCoE networks. Figure 9: FCoE-capable Shared Uplink Set always reserve the same ports on each module in a ME Domain LAN FCoE uplink connections 10Gb Stacking connections enc0:bay2:x4 enc1:bay2:x4 HP BladeSystem c enclosures stacking VC Domain 17 - Virtual Connect Restrictions

18 Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets In this scenario, two Converged Shared Uplink Sets using a single uplink port and carrying a single FCoE network and multiple standard Ethernet networks are created between the Virtual Connect modules and the Nexus switches. Each Nexus switch is directly connected to a FC or FCoE-based storage target. Figure 10: Physical view LAN Storage Arrays FC FCoE Nexus 5xxx Series FCF Mode FC F-Ports FCoE VF-Ports FC F-Ports FCoE VF-Ports Converged Ethernet/FCoE Converged Ethernet/FCoE HP Virtual Connect FlexFabric or Flex-10/10D modules VC Domain Converged Ethernet/FCoE FCoE FC LAN FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 200 FCoE VLAN 201 Blade Servers Benefits This configuration offers the simplicity of managing only one redundant DCB network with a single FCoE uplink. This simple scenario maximizes the use of the VC uplink ports, reduces the total number of FCoE switch ports needed in the datacenter and saves money. FC/FCoE-based Storage devices are directly connected to the Nexus 5xxx series, this reduces complexity and cost by eliminating the need for multitier storage area networks Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets

19 Considerations In this scenario, the Nexus switches must operate as Fiber Channel Forwarders (FCF). This is the default Cisco Nexus 5xxx Series switches mode, it s also called the fabric mode. In this mode, the switch provides standard Fibre Channel switching capability and features. Figure 11: Cisco Nexus 5xxx switches operating as FC Forwarder (FCF) Server 1 (ENode) FC Storage Array VN_Port N_Port Native FC Lossless Ethernet Server 2 (ENode) VN_Port network VF_Port FCF F_Port VF_Ports FCoE VN_Port FCoE Storage Array Virtual Connect (FIP Snooping mode) Dual-Hop FCoE Nexus 5xxx Series (FCF Mode) Converged Ethernet/FCoE FC FCoE This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of Nexus FCoE configuration guidelines, refer to the Fibre Channel over Ethernet Configuration Guides available at For information on Nexus FCoE supported configuration and design with Virtual Port Channel (vpc), refer to: Data Center Design with Cisco Nexus Switches and Virtual PortChannel Configuring Storage Array for FC/FCoE is beyond the scope of this document, consult your Storage Array documentations Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets

20 Requirements Refer to the HP s SPOCK documents for information on: Minimum NX-OS version Minimum Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Note: FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG). This license is an option which is activated when the Nexus switch is shipped from Cisco. Guidelines Nexus switches must either directly connect to FC-based or FCoE-based Storage devices. Refer to the Cisco Nexus or Storage vendor interoperability Matrix. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. The FCoE VLAN should be dedicated to FCoE traffic (i.e. it should not carry IP traffic). The FCoE VLAN must not be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by default VLAN 1). Interfaces connecting to VC must be configured as trunk ports and STP edge ports. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section. Nexus configuration Details about the configuration: FCF mode is used. Interfaces eth1/5 are connected to the VC modules. Interfaces eth1/1 are connected to an FCoE-based storage device (EVA). Interfaces eth1/17 are connected to the datacenter LAN. Interfaces fc2/1 are directly connected to a FC-based storage device (3PAR). VLAN IDs 200 and 201 are used for the FCoE networks. VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 2000 bind to eth1/5 for the Virtual Connect FCoE connection. The virtual Fibre Channel interfaces vfc 2007 bind to eth1/1 for the FCoE-based storage target FCoE connection Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets

21 Figure 12: Physical diagram Switch A Nexus-5010 FCF Mode vfc 2007 VLAN 200 eth1/1 WWN: 50:01:43:80:02:5D:19:72 eth1/5 vfc 2000 VSAN 200 VC FlexFabric Bay 1 EVA Storage Array FCoE VLAN 1,10,20 eth1/17 WWN: 50:01:43:80:02:5D:19:74 FCoE VLAN 200 VLAN 1,10,20 VSAN 200 X3 VSAN 201 fc2/1 fc2/1-2 FCoE VLAN 201 VLAN 1,10,20 VSAN 201 X3 LAN vfc 2007 VLAN 201 eth1/1 WWN: 21:53:00:02:ac:00:15:9d VSAN 200 eth1/5 vfc 2000 VC FlexFabric Bay 2 3PAR Storage Array FC VLAN 1,10,20 eth1/17 WWN: 20:53:00:02:ac:00:15:9d VSAN 201 fc2/1 fc2/1-2 Switch B Nexus-5010 FCF Mode Native Ethernet + Native Fibre Channel FCoE / DCB FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:20 50:06:0b:00:00:c3:1a:22 Blade Servers FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:2450:06:0b:00:00:c3:1a:26 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 200 FCoE VLAN 201 Converged Ethernet/FCoE FCoE FC LAN HP BladeSystem c Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets

22 Nexus switch-a configuration Enable FCoE on the switch (disabled by default): o configure terminal o feature fcoe Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate a VLAN with a VSAN for FCoE traffic: o vlan 200 o fcoe vsan 200 Note: Enabling FCoE on VLAN 1 is not supported. Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to eth1/5: o interface vfc 2000 o description vfc for VC in Bay 1 o bind interface eth1/5 o no shutdown # Assign it to VSAN200 as a trunk port o switchport trunk allowed vsan 200 Create a virtual Fibre Channel interface for the FCoE-based storage target and bind it to eth1/1: o interface vfc2007 o description vfc for EVA o bind interface eth1/1 o no shutdown # Assign it to VSAN200 as a trunk port o switchport trunk allowed vsan 200 Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk between the Nexus and the VC module Bay 1 with eth1/5 to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20): o interface Ethernet1/5 o description FCoE uplink to VC-Bay1-X3 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200 o spanning-tree port type edge trunk Create a trunk on the interface connected to the FCoE-based storage device to pass the FCoE traffic (VLAN 200): o interface Ethernet1/1 o description FCoE uplink to EVA o switchport mode trunk o switchport trunk allowed vlan 200 o spanning-tree port type edge trunk 22 - Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets

23 Configure the Fiber Channel interface connected to the 3PAR array: o interface fc 2/1 o switchport description Connected to the 3PAR array o no shutdown # Assign it to VSAN200 as a trunk port o switchport trunk allowed vsan 200 Create the VSAN 200 o vsan database o vsan 200 Assign the Virtual Fibre Channel interfaces (vfc) and the Fiber Channel interface to the FCoE VSAN: o vsan 200 interface vfc 2000 o vsan 200 interface vfc 2007 o vsan 200 interface fc2/1 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the first Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset-3par vsan 200 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 200 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 200 o member pwwn 50:01:43:80:02:5D:19:72 {This is the WWN of the first FCoE EVA port} o member pwwn 50:06:0b:00:00:c3:1a:24 {This is the WWN of the second Blade FlexFabric Adapter port 1} e. Create zoneset: o zoneset name zoneset-eva vsan 200 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets

24 Nexus switch-b configuration Enable FCoE on the switch (disabled by default): o configure terminal o feature fcoe Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate a VLAN with a VSAN for FCoE traffic: o vlan 201 o fcoe vsan 201 Note: Enabling FCoE on VLAN 1 is not supported. Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to eth1/5: o interface vfc 2000 o description vfc for VC in Bay 2 o bind interface eth1/5 o no shutdown # Assign it to VSAN201 as a trunk port o switchport trunk allowed vsan 201 Create a virtual Fibre Channel interface for the FCoE-based storage target and bind it to eth1/1: o interface vfc2007 o description vfc for EVA o bind interface eth1/1 o no shutdown # Assign it to VSAN201 as a trunk port o switchport trunk allowed vsan 201 Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk between the Nexus and the VC module Bay 2 with eth1/5 to pass the FCoE (VLAN 201) and Ethernet traffic (VLAN 1,10,20): o interface Ethernet1/5 o description FCoE uplink to VC-Bay2-X3 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201 o spanning-tree port type edge trunk Create a trunk on the interface connected to the FCoE-based storage device to pass the FCoE traffic (VLAN 201): o interface Ethernet1/1 o description FCoE uplink to EVA o switchport mode trunk o switchport trunk allowed vlan 201 o spanning-tree port type edge trunk 24 - Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets

25 Configure the Fiber Channel interface connected to the 3PAR array: o interface fc 2/1 o switchport description Connected to the 3PAR array o no shutdown # Assign it to VSAN201 as a trunk port o switchport trunk allowed vsan 201 Create the VSAN 201 o vsan database o vsan 201 Assign the Virtual Fibre Channel interfaces (vfc) and the Fiber Channel interface to the FCoE VSAN: o vsan 201 interface vfc 2000 o vsan 201 interface vfc 2007 o vsan 201 interface fc2/1 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the first Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset-3par vsan 201 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 201 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 201 o member pwwn 50:01:43:80:02:5D:19:74 {This is the WWN of the second FCoE EVA port} o member pwwn 50:06:0b:00:00:c3:1a:26 {This is the WWN of the second Blade FlexFabric Adapter port 2} e. Create zoneset: o zoneset name zoneset-eva vsan 201 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan Scenario 1: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode directly connected to FC/FCoE-Based storage targets

26 Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel) In this scenario, two Converged Shared Uplink Sets using a single uplink port and carrying a single FCoE network and multiple standard Ethernet networks are created between the Virtual Connect modules and the Nexus switches. Each Nexus switch is connected to a Cisco MDS SAN Fabric through a SAN Port Channel. Figure 13: Physical view FC Storage Array VSAN 200 VSAN 201 Cisco MDS Series SAN Switch FC SAN A FC SAN B SAN Port Channel FC LAN FC-FCoE SAN Port Channel FC Nexus 5xxx Series FCF Mode E-Ports E-Ports Converged Ethernet/FCoE Converged Ethernet/FCoE HP Virtual Connect FlexFabric or Flex-10/10D modules VC Domain Converged Ethernet/FCoE FC LAN FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 200 FCoE VLAN 201 Blade Servers Benefits This configuration offers the simplicity of managing only one redundant DCB network with a single FCoE uplink. This simple scenario maximizes the use of the VC uplink ports, reduces the total number of FCoE switch ports needed in the datacenter and saves money Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel)

27 Considerations The Nexus switches operate as Fiber Channel Forwarders (FCF). This is the default Cisco Nexus 5xxx Series switches mode, it s also called the fabric mode. In this mode, the switch provides standard Fibre Channel switching capability and features. In a fully populated c7000 enclosure, the server-to-uplink ratio is 16:1. This configuration can result in poor response time and sometimes requires particular performance monitoring attention. The use of additional FCoE uplink ports, both for better performance and better redundancy might be considered. Multiple VSAN support Cisco MDS and Nexus switches offer a capability called Virtual SANs (VSANs). VSANs allow an enterprise customer to maintain single physical SAN infrastructure, while providing separation and traffic isolation to the servers based on the business unit, types of the operating systems or need for logical isolation. But with the use of a single FCoE network in this scenario, servers can only access to a single VSAN, and therefore all benefits of Virtual SANs capability are here withdrawn. SAN Port Channel and Trunking The use of a SAN Port Channel between the Nexus and MDS core switch is highly recommended to create enough SAN subscription. F-Port Port Channeling increases the uplink resiliency, optimizes the bandwidth utilization and increases the aggregated bandwidth with load balancing. SAN Port Channel between Nexus and upstream switches is only supported with Cisco switches (MDS or Nexus series). E ports that are members of the same SAN Port Channel cannot be connected to different core switches. Figure 14: Cisco Nexus 5xxx switches operating as FC Forwarder (FCF) Server 1 (ENode) VN_Port FC Storage Array Lossless Ethernet N_Port Native FC Server 2 (ENode) VN_Port network Virtual Connect (FIP Snooping mode) VF_Port FCF F_Port Nexus 5xxx Series (FCF Mode) E_Ports SAN Port Channel ISL E_Ports Native FC Cisco MDS F_Port N_Port FC Storage Array Dual-Hop FCoE Converged Ethernet/FCoE FC 27 - Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel)

28 This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of Nexus FCoE configuration guidelines, refer to the Fibre Channel over Ethernet Configuration Guides available at For information on Nexus FCoE supported configuration and design with Virtual Port Channel (vpc), refer to: Data Center Design with Cisco Nexus Switches and Virtual PortChannel Configuring Storage Array for FC/FCoE is beyond the scope of this document, consult your Storage Array documentations. Requirements Refer to the HP s SPOCK documents for information on: Minimum NX-OS version Minimum Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Note: FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG). This license is an option which is activated when the Nexus switch is shipped from Cisco. Guidelines Nexus switches must either bridge to native FC infrastructure or directly connect to FC/FCoE-based Storage devices. Refer to the Cisco Nexus or Storage vendor interoperability Matrix. When bridged to a native FC infrastructure, it is mandatory to use Cisco MDS directors or fabric switches in order to provide interoperability between fabrics. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. The FCoE VLAN should be dedicated to FCoE traffic (i.e. it should not carry IP traffic). The FCoE VLAN must not be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by default VLAN 1). Interfaces connecting to VC must be configured as trunk ports and STP edge ports. If you configure a port channel between the Nexus and MDS switch, ensure that a SAN port channel is created on the MDS switch. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel)

29 Nexus configuration Details about the configuration: FCF mode is used. Interfaces eth1/5 are connected to the VC modules. Interfaces eth1/17 are connected to the datacenter LAN. Interfaces fc2/1 and fc2/2 are directly connected to Cisco MDS 9148 switches through a SAN Port Channel. SAN port channels spo100 are configured to pass FCoE VSANs 200 and 201. VLAN IDs 200 and 201 are used for the FCoE networks. VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 2000 bind to eth1/5 for the Virtual Connect FCoE connection. Figure 15: Physical diagram 3PAR Storage Array WWN: 21:53:00:02:ac:00:15:9d WWN: 20:53:00:02:ac:00:15:9d LAN Cisco MDS-A 9148 spo100 SAN-A SAN-B fc1/9 fc1/10 fc1/9 fc1/10 VSAN 200 VSAN 201 spo100 Cisco MDS-B 9148 Native Ethernet + Native Fibre Channel Switch A Nexus-5010 FCF Mode eth1/5 vfc 2000 VLAN 1,10,20 eth1/17 FCoE VLAN 200 VLAN 1,10,20 VSAN 200 X3 fc2/1-2 FCoE VLAN 201 VLAN 1,10,20 VSAN 201 X3 eth1/5 vfc 2000 VLAN 1,10,20 eth1/17 fc2/1-2 Switch B Nexus-5010 FCF Mode FCoE / DCB VC FlexFabric Bay 1 VC FlexFabric Bay 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:20 50:06:0b:00:00:c3:1a:22 Blade Servers FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:2450:06:0b:00:00:c3:1a:26 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 200 FCoE VLAN 201 Converged Ethernet/FCoE FC LAN HP BladeSystem c Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel)

30 Nexus switch-a configuration Enable FCoE on the switch (disabled by default): o configure terminal o feature fcoe Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate a VLAN with a VSAN for FCoE traffic: o vlan 200 o fcoe vsan 200 Note: Enabling FCoE on VLAN 1 is not supported. Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to eth1/5: o interface vfc 2000 o description vfc for VC in Bay 1 o bind interface eth1/5 o no shutdown # Assign it to VSAN200 as a trunk port o switchport trunk allowed vsan 200 Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk between the Nexus and the VC module Bay 1 with eth1/5 to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20): o interface Ethernet1/5 o description FCoE uplink to VC-Bay1-X3 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200 o spanning-tree port type edge trunk Create a SAN port channel between the Nexus and the MDS-A using the two fiber channel interfaces: o interface san-port-channel 100 o switchport description SAN-Port-Channel to MDS-A o channel mode active o switchport trunk allowed vsan 200 o interface fc 2/1-2 o switchport description Connected to MDS-A-fc1/9-10 o channel-group 100 o no shutdown Note: It is also necessary to configure port channeling on the MDS-A switch. To configure of an E port-channel on the MDS switch, use the following commands: interface port-channel 100 channel mode active switchport trunk allowed vsan Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel)

31 interface fc1/9-10 switchport description Connected to Nexus-A-fc2/1-2 channel-group 100 no shutdown Create the VSAN with assigned name o vsan database o vsan 200 name "3PAR-VSAN" Assign the Virtual Fibre Channel interface (vfc) and the SAN Port Channel to the FCoE VSAN: o vsan 200 interface san-port-channel 100 o vsan 200 interface vfc 2000 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20 Configuration of the zone: a. Create zones: o zone name fcoe-zone-3par vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the first Blade FlexFabric Adapter port 1} o member pwwn 50:06:0b:00:00:c3:1a:24 {This is the WWN of the second Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset-3par vsan 200 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 200 Nexus switch-b configuration Enable FCoE on the switch (disabled by default): o configure terminal o feature fcoe Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate a VLAN with a VSAN for FCoE traffic: o vlan 201 o fcoe vsan 201 Note: Enabling FCoE on VLAN 1 is not supported. Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to eth1/5: o interface vfc 2000 o description vfc for VC in Bay Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel)

32 o bind interface eth1/5 o no shutdown # Assign it to VSAN201 as a trunk port o switchport trunk allowed vsan 201 Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk between the Nexus and the VC module Bay 2 with eth1/5 to pass the FCoE (VLAN 201) and Ethernet traffic (VLAN 1,10,20): o interface Ethernet1/5 o description FCoE uplink to VC-Bay2-X3 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201 o spanning-tree port type edge trunk Create a SAN port channel between the Nexus and the MDS-B using the two fiber channel interfaces: o interface san-port-channel 100 o switchport description SAN-Port-Channel to MDS-B o channel mode active o switchport trunk allowed vsan 201 o interface fc 2/1-2 o switchport description Connected to MDS-B-fc1/9-10 o channel-group 100 o no shutdown Note: It is also necessary to configure port channeling on the MDS-B switch. To configure of an E port-channel on the MDS switch, use the following commands: interface port-channel 100 channel mode active switchport trunk allowed vsan 201 interface fc1/9-10 switchport description Connected to Nexus-B-fc2/1-2 channel-group 100 no shutdown Create the VSAN with assigned name o vsan database o vsan 201 name "3PAR-VSAN" Assign the Virtual Fibre Channel interface (vfc) and the SAN Port Channel to the FCoE VSAN: o vsan 201 interface san-port-channel 100 o vsan 201 interface vfc 2000 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10, Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel)

33 Configuration of the zone: a. Create zones: o zone name fcoe-zone-3par vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the first Blade FlexFabric Adapter port 2} o member pwwn 50:06:0b:00:00:c3:1a:26 {This is the WWN of the second Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset-3par vsan 201 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan Scenario 2: Dual-Hop FCoE using a single FCoE network and port with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS SAN switches (using a SAN Port Channel)

34 Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel) In this scenario, two Converged Shared Uplink Sets using multiple uplinks and carrying multiple FCoE networks and multiple standard Ethernet networks are created between the Virtual Connect modules and the Nexus switches. Each Nexus switch is connected to a Cisco MDS SAN Fabric through a trunk SAN Port Channel. Figure 16: Physical view 3PAR FC Storage Arrays EVA VSAN 200 VSAN 201 VSAN 300 VSAN 301 Cisco MDS Series SAN Switch FC SAN A FC SAN B Nexus 5xxx Series FCF Mode SAN Port Channel FC TE-Ports LAN FC-FCoE SAN Port Channel FC TE-Ports LACP Converged Ethernet/FCoE LACP Converged Ethernet/FCoE HP Virtual Connect FlexFabric or Flex-10/10D modules VC Domain Converged Ethernet/FCoE FC LAN FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Blade Servers 34 - Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

35 Benefits This configuration offers increased performance and better availability. The server-to-uplink ratio is adjustable, with the current NX software limitations, up to 2:1 with the Nexus 5500 series and VC Flex-10/10D (as few as two servers share one physical Fabric uplink) and up to 4:1 with the Nexus 5000 series. Using multiple FCoE uplink ports on each VC module connected to Nexus allows dynamically distribution of the FCoE traffic across the ports using a round robin format. By providing ability to define multiple FCoE VLANs on a single set of uplinks, VC allows multi-vsan capability to be enabled on Cisco s MDS or Nexus switches. VSANs allow an enterprise customer to maintain single physical SAN infrastructure, while providing separation and traffic isolation to the servers based on the business unit, types of the operating systems or need for logical isolation. Servers can therefore access to a different VSANs and connect to storage targets located in different logical SAN environment. Port Channeling with VSAN trunking is also used between the Nexus and MDS core switches to increase the high availability and to create enough SAN subscription. Considerations The Nexus switches operate as Fiber Channel Forwarders (FCF). This is the default Cisco Nexus 5xxx Series switches mode, it s also called the fabric mode. In this mode, the switch provides standard Fibre Channel switching capability and features. Multiple VSAN support The number of FCoE networks per Shared Uplink Set is limited to 32. VSAN tagging is not supported for native FC fabrics. The number of FCoE networks is additionally limited by the 1000 network domain limit. For this feature to successfully integrate with Cisco Nexus switches it is important to note that for each FCoE VLAN a unique FC VSAN must be created and association between the two must be strictly configured as 1-to-1. Mapping of a single VLAN into multiple VSANs is not supported. This requirement is documented in Cisco s best practice recommendations. All of the FCoE networks will share the same lossless Priority Flow Control (PFC) queue on any specific shared uplink set. Sharing PFC queue between multiple VSANs will make it possible that traffic from one VSAN may have impact on traffic from other VSANs. For that reason, it is highly recommended to create enough SAN subscription (i.e. by using a suitable number of uplink between VC and Nexus but also between Nexus and Cisco MDS) to avoid any over-subscription issues that would prevent SAN to pause and therefore to impact other VSANs. Therefore the use of a SAN Port Channel with VSAN trunking between the Nexus and MDS switches is highly recommended. SAN Port Channel and Trunking The use of a SAN Port Channel between the Nexus and MDS core switch is highly recommended to create enough SAN subscription. F-Port Port Channeling increases the uplink resiliency, optimizes the bandwidth utilization and increases the aggregated bandwidth with load balancing. SAN Port Channeling between Nexus and upstream switches is only supported with Cisco switches (MDS or Nexus series). TE ports that are members of the same SAN Port Channel cannot be connected to different core switches Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

36 Figure 17: Cisco Nexus 5xxx switches operating as FC Forwarder (FCF) Server 1 (ENode) FC Storage Array A FCoE VLAN A FCoE VLAN B Server 2 (ENode) VN_Port VN_Port Lossless Ethernet network Port Channel N_Port VSAN A FCF VF_Ports F_Port Native FC SAN Port Channel and Trunking TE_Ports TE_Ports Native FC F_Port VSAN B N_Port FC Storage Array B Virtual Connect (FIP Snooping mode) Nexus 5xxx Series (FCF Mode) EISL Cisco MDS Dual-Hop FCoE Converged Ethernet/FCoE FC This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of Nexus FCoE configuration guidelines, refer to the Fibre Channel over Ethernet Configuration Guides available at For information on Nexus FCoE supported configuration and design with Virtual Port Channel (vpc), refer to: Data Center Design with Cisco Nexus Switches and Virtual PortChannel Configuring Storage Array for FC/FCoE is beyond the scope of this document, consult your Storage Array documentations. Requirements Refer to the HP s SPOCK documents for information on: Minimum NX-OS version Minimum Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Note: FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG). This license is an option which is activated when the Nexus switch is shipped from Cisco Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

37 Guidelines Nexus switches must either bridge to native FC infrastructure or directly connect to FC/FCoE-based Storage devices. Refer to the Cisco Nexus or Storage vendor interoperability Matrix. When bridged to a native FC infrastructure, it is mandatory to use Cisco MDS directors or fabric switches in order to provide interoperability between fabrics. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. FCoE VLANs should be dedicated to FCoE traffic (i.e. they should not carry IP traffic). FCoE VLAN cannot be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by default VLAN 1). Interfaces connecting to VC must be configured as trunk ports and STP edge ports. If you configure a port channel between the Nexus and MDS switch, ensure that SAN port channeling and Trunking are enabled on the MDS switch. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section. Nexus configuration Current NX-OS limitations There is currently an NX-OS limitation with FCoE Port Channels on Cisco Nexus 5000 and 5500 series. When a Port Channel is carrying FCoE traffic, all interfaces of this port channel must belong to the same Nexus Unified Port Controller (UPC) ASIC. At this time, it is not supported to configure an FCoE port channel across different Unified Port Controllers. For more information, see Appendix A. Note: These limitations will be removed in a future NX software release. Details about the configuration: FCF mode is used. Interfaces eth1/5 and eth1/6 are connected to the VC modules thought FCoE using a port channel po200. Interfaces eth1/17 are connected to the datacenter LAN. Interfaces fc2/1 and fc2/2 are directly connected to Cisco MDS 9148 switches through a SAN trunk Port Channel (TE_ports). SAN port channels spo100 are using VSAN Trunking to pass FCoE VSANs 20x and 30x. VLAN IDs 200, 300, 201 and 301 are used for the FCoE networks. VLAN 200 maps to VSAN 200, VLAN 300 to VSAN 300, VLAN 201 to VSAN 201 and VLAN 301 to VSAN 301. VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 2000 bind to port channel po200 for the Virtual Connect FCoE connection. The EVA is connected to VSAN 30x and the 3PAR is connected to VSANs 20x Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

38 Figure 18: Physical diagram LAN Cisco MDS-A 9148 WWN: 21:53:00:02:ac:00:15:9d VSAN 200 VSAN Trunk spo100 3PAR WWN: 20:53:00:02:ac:00:15:9d VSAN 201 VSAN 300 EVA WWN: WWN: 50:01:43:80:02:5D:19:7C 50:01:43:80:02:5D:19:79 SAN-A SAN-B fc1/9 fc1/10 fc1/9 fc1/10 VSAN 200 VSAN 300 VSAN 201 VSAN 301 VSAN 301 VSAN Trunk spo100 Cisco MDS-B 9148 Native Ethernet + Native Fibre Channel Switch A Nexus-5010 FCF Mode eth1/5-6 vfc 2000 VLAN 1,10,20 VLAN 1,10,20 eth1/17 eth1/17 fc2/1-2 eth1/5-6 vfc 2000 fc2/1-2 Switch B Nexus-5010 FCF Mode FCoE VLAN 200, 300 VLAN 1,10,20 VSAN 200, 300 po200 po200 FCoE VLAN 201, 301 VLAN 1,10,20 VSAN 201, 301 FCoE / DCB X3 - X4 X3 X4 VC FlexFabric Bay 1 VC FlexFabric Bay 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:20 50:06:0b:00:00:c3:1a:22 Blade Servers HP BladeSystem c7000 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:2450:06:0b:00:00:c3:1a:26 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Converged Ethernet/FCoE FC LAN 38 - Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

39 Nexus switch-a configuration Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate each VLAN with a VSAN for each FCoE networks: o vlan 200 o fcoe vsan 200 o vlan 300 o fcoe vsan 300 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the first Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the VC module Bay 1 and the Nexus switch A with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/5-6 o channel-group 200 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk on the port channel interface to pass the FCoE VLANs 200 and 300 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o description FCoE Port Channel to VC-Bay1-X3/X4 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200, 300 o spanning-tree port type edge trunk Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to port-channel 200: o interface vfc 2000 o description vfc for VC in Bay 1 o bind interface port-channel 200 o no shutdown # Assign it to VSAN200 and 300 as a trunk port o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 Create a SAN port channel with VSAN Trunking allowing the two FCoE VSANs (200 and 300) between the Nexus and MDS-A switch using the two fiber channel interfaces: o interface san-port-channel 100 o switchport description SAN-Trunk-Port-Channel to MDS-A 39 - Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

40 o channel mode active o switch trunk mode on o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 o interface fc 2/1-2 o switchport trunk mode on o switchport description Connected to MDS-A-fc1/9-10 o channel-group 100 o no shutdown Note: It is also necessary to configure port trunking and port channeling on the MDS-A switch. To configure of a TE port-channel on the MDS switch, use the following commands: interface port-channel 100 channel mode active switchport trunk allowed vsan 200 switchport trunk allowed vsan add 300 interface fc1/9-10 switchport description Connected to Nexus-A-fc2/1-2 channel-group 100 no shutdown vsan database vsan 200 vsan 200 interface port-channel 100 Create the two VSANs with assigned names o vsan database o vsan 200 name "3PAR-VSAN" o vsan 300 name "EVA-VSAN" Assign the Virtual Fibre Channel interface (vfc) and the SAN Port Channel to first FCoE VSAN: o vsan 200 interface san-port-channel 100 o vsan 200 interface vfc 2000 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the first Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset-3par vsan 200 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

41 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 300 o member pwwn 50:01:43:80:02:5D:19:7C {This is the WWN of the first EVA controller port} o member pwwn 50:06:0b:00:00:c3:1a:24 {This is the WWN of the second Blade FlexFabric Adapter port 1} e. Create zoneset: o zoneset name zoneset-eva vsan 300 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan 300 Nexus switch-b configuration Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate each VLAN with a VSAN for each FCoE networks: o vlan 201 o fcoe vsan 201 o vlan 301 o fcoe vsan 301 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the second Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the VC module Bay 2 and the Nexus switch B with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/5-6 o channel-group 200 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk on the port channel interface to pass the FCoE VLANs 201 and 301 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o description FCoE Port Channel to VC-Bay2-X3/X4 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201,301 o spanning-tree port type edge trunk 41 - Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

42 Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to port-channel 200: o interface vfc 2000 o description vfc for VC in Bay 2 o bind interface port-channel 200 o no shutdown # Assign it to VSAN201 and 301 as a trunk port o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 Create a SAN port channel with VSAN Trunking allowing the two FCoE VSANs (201 and 301) between the Nexus and MDS-B switch using the two fiber channel interfaces: o interface san-port-channel 100 o switchport description SAN-Trunk-Port-Channel to MDS-B o channel mode active o switchport trunk mode on o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 o interface fc 2/1-2 o switchport trunk mode on o switchport description Connected to MDS-B-fc1/9-10 o channel-group 100 o no shutdown Note: It is also necessary to configure port trunking and port channeling on the MDS-B switch. To configure of a TE port-channel on the MDS switch, use the following commands: interface port-channel 100 channel mode active switchport trunk allowed vsan 201 switchport trunk allowed vsan add 301 interface fc1/9-10 switchport description Connected to Nexus-B-fc2/1-2 channel-group 100 no shutdown Create the two VSANs with assigned names o vsan database o vsan 201 name "3PAR-VSAN" o vsan 301 name "EVA-VSAN" Assign the Virtual Fibre Channel interface (vfc) and the SAN Port Channel to first FCoE VSAN: o vsan 201 interface san-port-channel 100 o vsan 201 interface vfc 2000 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10, Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

43 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the first Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset-3par vsan 201 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 201 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 301 o member pwwn 50:01:43:80:02:5D:19:79 {This is the WWN of the second EVA controller port} o member pwwn 50:06:0b:00:00:c3:1a:26 {This is the WWN of the second Blade FlexFabric Adapter port 2} e. Create zoneset: o zoneset name zoneset-eva vsan 301 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan Scenario 3: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to Cisco MDS switches (using a SAN Port Channel)

44 Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel) In this Multi-hop FCoE scenario, two Converged Shared Uplink Sets using multiple uplinks and carrying multiple FCoE networks and multiple standard Ethernet networks are created between the Virtual Connect modules and the Nexus switches and each Nexus switch is connected to a Cisco MDS SAN Fabric through a trunk FCoE Port Channel. Figure 19: Physical view 3PAR FC Storage Arrays EVA Cisco MDS Gbps 8-Port FCoE Module VSAN 200 VSAN 201 VSAN 300 VSAN 301 Cisco MDS 9506 FC SAN A FC SAN B FC-FCoE LACP FCoE LAN LACP FCoE Nexus 5xxx Series FCF Mode VE-Ports VE-Ports LACP Converged Ethernet/FCoE LACP Converged Ethernet/FCoE HP Virtual Connect FlexFabric or Flex-10/10D modules VC Domain Converged Ethernet/FCoE FCoE FC LAN FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Blade Servers 44 - Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

45 Benefits This Multi-Hop FCoE scenario extends convergence beyond the Top of Rack switches but at the same time it preserves all your Fibre Channel SAN switch investments. Extended convergence offers more flexibility and capital/operational cost savings. Nexus 50xx switches have a limited number of 8Gb FC port (5010=6 max, 5020=12 max) so this scenario can help customers running out of Fibre Channel ports on Nexus 50xx switches by using FCoE links instead of FC to reach the MDS SAN switches. Using multiple FCoE uplink ports on each VC module connected to Nexus and on each Nexus connected to the MDS allow dynamically distribution of the FCoE traffic across the ports using a round robin format. FCoE Port Channeling increases the high availability and create enough FCoE SAN subscription. Considerations Enabling convergence in the extended network requires more careful bandwidth management than a Dual-Hop FCoE topology. Suitable subscription is crucial to reduce the risk of congestion and slow performance, therefore the use of Port Channeling is recommended. The server-to-vc uplink ratio is adjustable. With the current NX software limitations, the server-to-uplink can go up to: 2:1 with the Nexus 5500 series and VC Flex-10/10D (as few as two servers share one physical Fabric uplink) 4:1 with the Nexus 5000 series. The Nexus to the MDS ratio can be as good as 1:1 with up to 4 ports with the Nexus 5000 series and up to 8 ports with the Nexus 5500 series. Multiple VSAN support The number of FCoE networks per Shared Uplink Set is limited to 32. VSAN tagging is not supported for native FC fabrics. The number of FCoE networks is additionally limited by the 1000 network domain limit. For this feature to successfully integrate with Cisco Nexus switches, it is important to note that for each FCoE VLAN a unique FC VSAN must be created and association between the two must be strictly configured as 1-to-1. Mapping of a single VLAN into multiple VSANs is not supported. This requirement is documented in Cisco s best practice recommendations. All of the FCoE networks will share the same lossless Priority Flow Control (PFC) queue on any specific shared uplink set. Sharing PFC queue between multiple VSANs will make it possible that traffic from one VSAN may have impact on traffic from other VSANs. For that reason, it is highly recommended to create enough SAN subscription (i.e. by using a suitable number of uplink between VC and Nexus but also between Nexus and Cisco MDS) to avoid any over-subscription issues that would prevent SAN to pause and therefore to impact other VSANs. Therefore the use of an FCoE Port Channel between the Nexus and MDS switches is highly recommended Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

46 Figure 20: Cisco Nexus 5xxx switches operating as FC Forwarder (FCF) Server 1 (ENode) FC Storage Array A FCoE VLAN A FCoE VLAN B Server 2 (ENode) VN_Port VN_Port Lossless Ethernet network Virtual Connect (FIP Snooping mode) Port Channel VF_Ports FCoE storage FC storage Nexus 5xxx Series (FCF Mode) VE_Ports Port Channel N_Port VSAN A VE_Ports Cisco MDS Native FC F_Port F_Port VSAN B N_Port Native FC FC Storage Array B Multi-Hop FCoE Converged Ethernet/FCoE FC FCoE This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of Nexus FCoE configuration guidelines, refer to the Fibre Channel over Ethernet Configuration Guides available at For information on Nexus FCoE supported configuration and design with Virtual Port Channel (vpc), refer to: Data Center Design with Cisco Nexus Switches and Virtual PortChannel Configuring Storage Array for FC/FCoE is beyond the scope of this document, consult your Storage Array documentations. Requirements Refer to the HP s SPOCK documents for information on: Minimum NX-OS version Minimum Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Cisco MDS Gbps 8-Port FCoE Module support information Note: FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG). This license is an option which is activated when the Nexus switch is shipped from Cisco Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

47 Guidelines Nexus switches can directly connect to FC/FCoE-based Storage devices as they are running in FCF mode. Refer to the Cisco Nexus or Storage vendor interoperability Matrix. The NPV mode can also be used in this scenario. When NPV mode is enabled, FC/FCoE-based Storage devices cannot be directly connected to the Nexus. To reduce the risk of congestion, it is recommended to use FCoE Port Channeling. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. FCoE VLANs should be dedicated to FCoE traffic (i.e. they should not carry IP traffic). FCoE VLAN cannot be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by default VLAN 1). Interfaces connecting to VC must be configured as trunk ports and STP edge ports. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section. Nexus and MDS configuration Current NX-OS limitations There is currently an NX-OS limitation with FCoE Port Channels on Cisco Nexus 5000 and 5500 series. When a Port Channel is carrying FCoE traffic, all interfaces of this port channel must belong to the same Nexus Unified Port Controller (UPC) ASIC. At this time, it is not supported to configure an FCoE port channel across different Unified Port Controllers. For more information, see Appendix A. Note: These limitations will be removed in a future NX software release. Details about the Nexus configuration: FCF mode is used. Interfaces eth1/5 and eth1/6 are connected to the VC modules through FCoE using a port channel po200. Interfaces eth1/19 and eth1/20 are connected to Cisco MDS 9506 switches through FCoE using a port channel po100 (VE_Ports). Interfaces eth1/17 are connected to the datacenter LAN. VLAN IDs 200, 300, 201 and 301 are used for the FCoE networks. VLAN 200 maps to VSAN 200, VLAN 300 to VSAN 300, VLAN 201 to VSAN 201 and VLAN 301 to VSAN 301. VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 2000 bind to port channel po200 for the Virtual Connect FCoE connection. The virtual Fibre Channel interfaces vfc 1000 bind to the port channel po100 connected to the MDS Details about the MDS 9506 configuration: FCF mode is used. A Cisco MDS Gbps 8-Port FCoE Module is required to enable FCoE on MDS switches. Interfaces Eth2/1-2 are connected to the Nexus switches thought FCoE using an Ethernet port channel epo100 (VE_Ports). VLAN IDs 200, 300, 201 and 301 are used for the FCoE networks. VLAN 200 maps to VSAN 200, VLAN 300 to VSAN 300, VLAN 201 to VSAN 201 and VLAN 301 to VSAN Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

48 The virtual Fibre Channel interfaces vfc 1000 bind to the port channel epo100 connected to the Nexus switches. Interfaces fc1/1 are connected to the EVA and fc1/2 are connected to the 3PAR. The EVA is connected to VSAN 30x and the 3PAR is connected to VSANs 20x. Figure 21: Physical diagram WWN: 21:53:00:02:ac:00:15:9d 3PAR Storage Array FC VSAN 200 WWN: WWN: 20:53:00:02:ac:00:15:9d 50:01:43:80:02:5D:19:7C VSAN 201 VSAN 300 EVA Storage Array FC WWN: 50:01:43:80:02:5D:19:79 VSAN 301 Cisco MDS Gbps 8-Port FCoE Module MDS-A Cisco MDS 9506 FCF Mode eth2/1-2 vfc 1000 epo100 fc1/1 fc1/2 FC fc1/1 fc1/2 MDS-B Cisco MDS 9506 FCF Mode eth2/1-2 vfc 1000 epo100 Switch A Nexus-5010 FCF Mode eth1/5-6 vfc 2000 FCoE VLAN 200, 300 VSAN 200, 300 VLAN 1,10,20 eth1/17 po 100 vfc 1000 eth1/19-20 LAN eth1/5-6 vfc 2000 VLAN 1,10,20 eth1/17 FCoE VLAN 201, 301 VSAN 201, 301 po100 vfc 1000 eth1/19-20 Switch B Nexus-5010 FCF Mode FCoE / DCB FCoE VLAN 200, 300 VLAN 1,10,20 VSAN 200, 300 po200 po200 FCoE VLAN 201, 301 VLAN 1,10,20 VSAN 201, 301 X3 - X4 X3 X4 VC FlexFabric Bay 1 VC FlexFabric Bay 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:20 50:06:0b:00:00:c3:1a:22 Blade Servers HP BladeSystem c7000 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:2450:06:0b:00:00:c3:1a:26 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Converged Ethernet/FCoE FCoE FC LAN 48 - Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

49 Nexus switch-a configuration Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate each VLAN with a VSAN for each FCoE networks: o vlan 200 o fcoe vsan 200 o vlan 300 o fcoe vsan 300 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the first Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the VC module Bay 1 and the Nexus switch A with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/5-6 o channel-group 200 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk on the port channel interface to pass the FCoE VLANs 200 and 300 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o description FCoE Port Channel to VC-Bay1-X3/X4 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200, 300 o spanning-tree port type edge trunk Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to port-channel 200: o interface vfc 2000 o description vfc for VC in Bay 1 o bind interface port-channel 200 o no shutdown # Assign it to VSAN200 and 300 as a trunk port o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

50 Create a port channel between the Nexus switch A and the MDS-A switch with eth1/19 and eth1/20. eth1/19 and eth1/20 can be used because both are mapped to the same UPC (i.e. UPC#2) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/19-20 o channel-group 100 mode active Create a trunk on the port channel interface to pass the FCoE VLANs 200 and 300: o interface port-channel 100 o description FCoE Port Channel to MDS-A-eth2/1-2 o switchport mode trunk o switchport trunk allowed vlan 200, 300 Create a virtual Fibre Channel interface for the FCoE Port Channel connected to the MDS-A Switch: o interface vfc1000 o description vfc for MDS-A o bind interface port-channel 100 o switchport mode E o no shutdown # Assign it to VSAN200 and 300 as a trunk port o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 Create the two VSANs with assigned names o vsan database o vsan 200 name "3PAR-VSAN" o vsan 300 name "EVA-VSAN" Assign the two Virtual Fibre Channel interfaces (vfc) to the first FCoE VSAN: o vsan 200 interface vfc 1000 o vsan 200 interface vfc 2000 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the first Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset-3par vsan 200 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 200 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

51 o member pwwn 50:01:43:80:02:5D:19:7C {This is the WWN of the first EVA controller port} o member pwwn 50:06:0b:00:00:c3:1a:24 {This is the WWN of the second Blade FlexFabric Adapter port 1} e. Create zoneset: o zoneset name zoneset-eva vsan 300 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan 300 MDS 9506 switch-a configuration Enable FCoE and LACP on the switch (disabled by default): o configure terminal o install feature-set fcoe o feature-set fcoe o feature lacp Associate each VLAN with a VSAN for each FCoE networks: o vlan 200 o fcoe vsan 200 o vlan 300 o fcoe vsan 300 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the first Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the MDS-A and the Nexus switch A with eth2/1 and eth2/2 from the Cisco MDS Gbps 8-Port FCoE Module: o interface ethernet 2/1-2 o channel-group 100 mode active Create a trunk on the port channel interface to pass the FCoE VLANs 200 and 300: o interface ethernet-port-channel 100 o switchport description FCoE Port Channel to Nexus-A-eth1/19-20 o switchport mode trunk o switchport trunk allowed vlan 200, 300 Create a virtual Fibre Channel interface for the FCoE Port Channel connected to Nexus-A: o interface vfc1000 o description vfc for Nexus-A o bind interface ethernet-port-channel 100 o switchport mode E o no shutdown # Assign it to VSAN200 and 300 as a trunk port o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 o switchport trunk mode on 51 - Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

52 Configuration of the Fibre Channel ports connected to the storage arrays: o interface fc1/1 o switchport description Connected to EVA o no shutdown o interface fc1/2 o switchport description Connected to 3PAR o no shutdown Create the two VSANs with assigned names o vsan database o vsan 200 name "3PAR-VSAN" o vsan 300 name "EVA-VSAN" Assign the Virtual Fibre Channel interfaces (vfc) and the Fibre Channel interfaces (fc) to the FCoE VSANs: o vsan 200 interface vfc 1000 o vsan 200 interface fc1/1 o vsan 301 interface fc1/2 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the first Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset-3par vsan 200 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 200 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 300 o member pwwn 50:01:43:80:02:5D:19:7C {This is the WWN of the first EVA controller port} o member pwwn 50:06:0b:00:00:c3:1a:24 {This is the WWN of the second Blade FlexFabric Adapter port 1} e. Create zoneset: o zoneset name zoneset-eva vsan 300 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

53 Nexus switch-b configuration Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate each VLAN with a VSAN for each FCoE networks: o vlan 201 o fcoe vsan 201 o vlan 301 o fcoe vsan 301 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the second Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the VC module Bay 2 and the Nexus switch B with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/5-6 o channel-group 200 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk on the port channel interface to pass the FCoE VLANs 201 and 301 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o description FCoE Port Channel to VC-Bay2-X3/X4 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201,301 o spanning-tree port type edge trunk Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to port-channel 200: o interface vfc 2000 o description vfc for VC in Bay 2 o bind interface port-channel 200 o no shutdown # Assign it to VSAN201 and 301 as a trunk port o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

54 Create a port channel between the Nexus switch B and the MDS-B switch with eth1/19 and eth1/20. eth1/19 and eth1/20 can be used because both are mapped to the same UPC (i.e. UPC#2) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/19-20 o channel-group 100 mode active Create a trunk on the port channel interface to pass the FCoE VLANs 201 and 301: o interface port-channel 100 o description FCoE Port Channel to MDS-B-eth2/1-2 o switchport mode trunk o switchport trunk allowed vlan 201, 301 Create a virtual Fibre Channel interface for the FCoE Port Channel connected to the MDS-B Switch: o interface vfc1000 o description vfc for MDS-B o bind interface port-channel 100 o switchport mode E o no shutdown # Assign it to VSAN201 and 301 as a trunk port o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 Create the two VSANs with assigned names o vsan database o vsan 201 name "3PAR-VSAN" o vsan 301 name "EVA-VSAN" Assign the two Virtual Fibre Channel interfaces (vfc) to the first FCoE VSAN: o vsan 201 interface vfc 1000 o vsan 201 interface vfc 2000 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the first Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset-3par vsan 201 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 201 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

55 o member pwwn 50:01:43:80:02:5D:19:79 {This is the WWN of the second EVA controller port} o member pwwn 50:06:0b:00:00:c3:1a:26 {This is the WWN of the second Blade FlexFabric Adapter port 2} e. Create zoneset: o zoneset name zoneset-eva vsan 301 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan 301 MDS 9506 switch-b configuration Enable FCoE and LACP on the switch (disabled by default): o configure terminal o install feature-set fcoe o feature-set fcoe o feature lacp Associate each VLAN with a VSAN for each FCoE networks: o vlan 201 o fcoe vsan 201 o vlan 301 o fcoe vsan 301 Note: Enabling FCoE on VLAN 1 is not supported. Create a port channel between the MDS-B and the Nexus switch B with eth2/1 and eth2/2 from the Cisco MDS Gbps 8-Port FCoE Module: o interface ethernet 2/1-2 o channel-group 100 mode active Create a trunk on the port channel interface to pass the FCoE VLANs 201 and 301: o interface ethernet-port-channel 100 o switchport description FCoE Port Channel to Nexus-B-eth1/19-20 o switchport mode trunk o switchport trunk allowed vlan 201, 301 Create a virtual Fibre Channel interface for the FCoE Port Channel connected to Nexus-B: o interface vfc1000 o description vfc for Nexus-B o bind interface ethernet-port-channel 100 o switchport mode E o no shutdown # Assign it to VSAN201 and 301 as a trunk port o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 o switchport trunk mode on 55 - Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

56 Configuration of the Fibre Channel ports connected to the storage arrays: o interface fc1/1 o switchport description Connected to EVA o no shutdown o interface fc1/2 o switchport description Connected to 3PAR o no shutdown Create the two VSANs with assigned names o vsan database o vsan 201 name "3PAR-VSAN" o vsan 301 name "EVA-VSAN" Assign the Virtual Fibre Channel interfaces (vfc) and the Fibre Channel interfaces (fc) to the FCoE VSANs: o vsan 201 interface vfc 1000 o vsan 201 interface fc1/1 o vsan 301 interface fc1/2 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the first Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset-3par vsan 201 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 201 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 301 o member pwwn 50:01:43:80:02:5D:19:79 {This is the WWN of the second EVA controller port} o member pwwn 50:06:0b:00:00:c3:1a:26 {This is the WWN of the second Blade FlexFabric Adapter port 2} e. Create zoneset: o zoneset name zoneset-eva vsan 301 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan Scenario 4: Multi-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in FCF mode connected to a Cisco MDS switches (using a FCoE Port Channel)

57 Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches. In this scenario, the Nexus switches operate as FCoE Gateways. This particular mode of operation is enabled by using the Nexus NPV mode. In this scenario, two Converged Shared Uplink Sets using multiple uplink ports and carrying multiple FCoE networks and multiple standard Ethernet networks are created between the Virtual Connect modules and the Nexus switches running in NPV mode. Each NPV Nexus switch is connected to a Brocade SAN Fabric through multiple NP ports with each one carrying a single VSAN. Servers can access to different VSANs and connect to storage targets through different logical SAN environment. Figure 22: Physical view with a Brocade Fabric using multiple NP uplinks 3PAR FC Storage Arrays EVA VSAN 200 VSAN 201 VSAN 300 VSAN 301 Brocade SAN Switches FC SAN A FC SAN B NPIV LAN NPIV Nexus 5xxx Series NPV Mode VSAN 200 VSAN 300 VSAN 201 VSAN 301 NP-Ports NP-Ports LACP Converged Ethernet/FCoE LACP Converged Ethernet/FCoE HP Virtual Connect FlexFabric or Flex-10/10D modules VC Domain Converged Ethernet/FCoE FC LAN FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Blade Servers 57 - Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

58 Benefits This scenario can be used to simplify third-party interoperability concerns in environment where different types of Fibre Channel switches are used (i.e. when connecting Nexus 5xxx to a non-cisco Fibre Channel switch like a Brocade SAN switch) because in NPV mode, the switch does not participate in domain operations or perform local switching. This enables multivendor topologies to be implemented without the restrictions the FCF mode requires. Additionally the NPV mode increases the SAN scalability as you can add switches to the faric without adding more domain IDs but the number of ports to the fabric are increased. In NPV mode, fabric logins are passed upstream from the edge switch to the core Brocade switch. The Nexus acts as a pass through switch and converts the Fabric Login (FLOGI) command to a Fabric Discovery (FDISC) command and passes it to the upstream SAN switch. The use of multiple FCoE uplink ports on each VC module connected to Nexus allows dynamically distribution of the FCoE traffic across the ports using a round robin format. By providing ability to define multiple FCoE VLANs on a single set of uplinks, VC allows multi-vsan capability to be enabled on Nexus switches. The use of VSANs is fully transparent to the Brocade Core switches since Brocade has no concept of VSANs, however VSANs can help to maintain separation and traffic isolation to the servers on the NPV switches since zoning is not supported. On the Brocade side, zoning can be set to extend those VSANs to defined Brocade zones that are connected to different storage targets. Considerations In NPV mode, the switch does not operate as a typical Fibre Channel switch, fabric services (i.e. Name Service, Fabric Login, zoning, etc.) are not available and Fibre Channel switching is not performed: all traffic is switched upstream in the core switch. The NPV mode uses N Port ID Virtualization (NPIV) technology, F_Ports that normally connect to the fabric are configured in this mode as NP_Ports and use NPIV to register multiple WWPNs to the F_Port on the other end. Note: To convert a switch into NPV mode, you set the NPV feature to enable. This configuration command automatically triggers a switch reboot and the system configuration is erased. You cannot configure NPV mode on a per-interface basis. NPV mode applies to the entire switch. For more information about the NPV Mode and Cisco support, see Multiple VSAN support The number of FCoE networks per Shared Uplink Set is limited to 32. VSAN tagging is not supported for native FC fabrics. The number of FCoE networks is additionally limited by the 1000 network domain limit. For this feature to successfully integrate with Cisco Nexus switches it is important to note that for each FCoE VLAN a unique FC VSAN must be created and association between the two must be strictly configured as 1-to-1. Mapping of a single VLAN into multiple VSANs is not supported. This requirement is documented in Cisco s best practice recommendations. All of the FCoE networks will share the same lossless Priority Flow Control (PFC) queue on any specific shared uplink set. Sharing PFC queue between multiple VSANs will make it possible that traffic from one VSAN may have impact on traffic from other VSANs. For that reason, it is highly recommended to create enough SAN subscription (i.e. by using a suitable number of uplink between VC and Nexus but also between Nexus and Cisco MDS) to avoid any over-subscription issues that would prevent SAN to pause and therefore to impact other VSANs. Therefore the use of multiple NP Ports between the Nexus and Brocade Core switches is highly recommended Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

59 Figure 23: Cisco Nexus 5xxx switches operating as an NPV Gateway Server (ENode) Lossless Ethernet network FCoE Gateway Native FC VN_Port Port Channel Virtual Connect (FIP Snooping mode) VF_Ports Nexus 5xxx Series (NPV Mode) NP_Ports NPIV F_Ports Brocade SAN Switch (NPIV Enabled) F_Port N_Port FC Storage Array Dual-Hop FCoE Converged Ethernet/FCoE FC This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of Nexus FCoE configuration guidelines, refer to the Fibre Channel over Ethernet Configuration Guides available at For information on Nexus FCoE supported configuration and design with Virtual Port Channel (vpc), refer to: Data Center Design with Cisco Nexus Switches and Virtual PortChannel Configuring Storage Array for FC is beyond the scope of this document, consult your Storage Array documentations. Requirements Refer to the HP s SPOCK documents for information on: Minimum NX-OS version Minimum Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Note: FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG). This license is an option which is activated when the Nexus switch is shipped from Cisco. Guidelines Since the Nexus is running in NPV mode, it is mandatory to connect the Nexus switches to a native FC SAN infrastructure. This SAN infrastructure can be non-cisco FC devices (e.g. Brocade, Qlogic switches). Refer to SPOCK for supported upstream switches. FC/FCoE-based Storage targets cannot be connected to the Nexus switch when running in NPV mode. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. FCoE VLANs should be dedicated to FCoE traffic (i.e. it should not carry IP traffic). FCoE VLAN cannot be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by default VLAN 1). Interfaces connecting to VC must be configured as trunk ports and STP edge ports Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

60 NPIV support is required in FC switches that connect to the Nexus NP Ports. When multiple NP ports belonging to the same Cisco VSAN are used to connect to the Brocade fabric, the Nexus uses dynamic login distribution to load balance the server connections across all available NP uplinks. The NP uplink interface with the minimum load is selected from the available NP uplinks as the external interface: Different NP ports can be connected to different core switches as long as they belong to the same SAN Fabric. When a NP uplink fails, the disrupted server connections re-login on other NP uplink. Cisco Nexus SAN Port Channel and VSAN trunking features are not supported with Brocade/Qlogic SAN fabrics. Since Brocade has no concept of VSANs, any VSANs can be connected to a Brocade fabric. When Multiple VSANs are used, there are 2 options: o If you want to assign servers to a specific FC interface: map one NP uplink to one VSAN. o If you don t need to assign servers to any specific FC interfaces: map all NP uplinks to all VSANs. Since the Nexus is running in NPV mode, all zoning needs to be performed from the Brocade fabric. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section. Nexus configuration Current NX-OS limitations There is currently an NX-OS limitation with FCoE Port Channels on Cisco Nexus 5000 and 5500 series. When a Port Channel is carrying FCoE traffic, all interfaces of this port channel must belong to the same Nexus Unified Port Controller (UPC) ASIC. At this time, it is not supported to configure an FCoE port channel across different Unified Port Controllers. For more information, see Appendix A. Note: These limitations will be removed in a future NX software release. Details about the configuration: NPV mode is used. Interfaces eth1/5 and eth1/6 are connected to the VC modules thought FCoE using a port channel po200. Interfaces eth1/17 are connected to the datacenter LAN. Interfaces fc2/1 and fc2/2 are connected to Brocade Core switches using NPIV (NP_Ports). Interface fc2/1 is mapped to VSAN 200/201 and fc2/2 to VSAN 300/301 (Alternativelly, you can assign both interfaces to all VSANs). VLAN IDs 200, 300, 201 and 301 are used for the FCoE networks. VLAN 200 maps to VSAN 200, VLAN 300 to VSAN 300, VLAN 201 to VSAN 201 and VLAN 301 to VSAN 301 VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 2000 bind to port channel po200 for the Virtual Connect FCoE connection Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

61 Figure 24: Physical diagram WWN: 21:53:00:02:ac:00:15:9d 3PAR WWN: 20:53:00:02:ac:00:15:9d EVA WWN: WWN: 50:01:43:80:02:5D:19:7C 50:01:43:80:02:5D:19:79 LAN Brocade-A 8000 SAN-A SAN-B Brocade-B 8000 Native Ethernet + Native Fibre Channel Switch A Nexus-5010 NPV Mode eth1/5-6 vfc 2000 VLAN 1,10,20 VLAN 1,10,20 eth1/17 VSAN 200 VSAN 300 eth1/17 VSAN 201 VSAN 301 fc2/1-2 eth1/5-6 vfc 2000 fc2/1-2 Switch B Nexus-5010 NPV Mode FCoE VLAN 200, 300 VLAN 1,10,20 VSAN 200, 300 po200 X3 - X4 X3 X4 po200 FCoE VLAN 201, 301 VLAN 1,10,20 VSAN 201, 301 FCoE / DCB VC FlexFabric Bay 1 VC FlexFabric Bay 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:20 50:06:0b:00:00:c3:1a:22 Blade Servers HP BladeSystem c7000 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:2450:06:0b:00:00:c3:1a:26 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Converged Ethernet/FCoE FC LAN 61 - Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

62 Nexus switch-a configuration Enable NPV: o configure terminal o feature npv Note: Switching to NPV mode automatically performs a configuration erase and reloads the switch. Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Configure the Fibre Channel interfaces connected to the Brocade-A SAN switch: o interface fc 2/1-2 o switchport description NP Port connected to Brocade-A o switchport mode np o no shutdown Note: The upstream Brocade SAN switch needs to be NPIV-enabled. To enable NPIV on port 13 on the Brocade SAN switch, enter the following command: portcfgnpivport 13 1 {Where 1 indicates that NPIV is enabled} # if port 13 is a 8G port, it is necessary to add: portcfgfillword 13 3 {Only applicable to 8G-capable ports} For more information about NPIV support for your Fibre Channel switch, refer to the switch vendor documentation. Associate each VLAN with a VSAN for each FCoE networks: o vlan 200 o fcoe vsan 200 o vlan 300 o fcoe vsan 300 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the first Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the VC module Bay 1 and the Nexus switch A with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/5-6 o channel-group 200 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10, Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

63 Create a trunk on the port channel interface to pass the FCoE VLANs 200 and 300 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o description FCoE Port Channel to VC-Bay1-X3/X4 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200,300 o spanning-tree port type edge trunk Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to port-channel 200: o interface vfc 2000 o description vfc for VC in Bay 1 o bind interface port-channel 200 o no shutdown # Assign it to VSAN200 and 300 as a trunk port o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 Create the two VSANs with assigned names o vsan database o vsan 200 name "3PAR-VSAN" o vsan 300 name "EVA-VSAN" Assign the Virtual Fibre Channel interface (vfc) to the first FCoE VSAN: o vsan 200 interface vfc 2000 Assign the Fibre Channel interfaces to the appropriate FCoE VSANs: o vsan 200 interface fc2/1 o vsan 300 interface fc2/2 Note: Alternatively, instead of dedicating one FC interface to one VSAN, you can use both FC interfaces to carry the two VSANs: o vsan 200 interface fc2/2 o vsan 300 interface fc2/1 Note: If there are multiple FC interfaces in the same VSAN, then it is not possible to assign an end device to a specific interface. Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10, Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

64 Nexus switch-b configuration Enable NPV: o configure terminal o feature npv Note: Switching to NPV mode automatically performs a configuration erase and reloads the switch. Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Configure the Fibre Channel interfaces connected to the Brocade-B SAN switch: o interface fc 2/1 o switchport description NP Port connected to Brocade-B o switchport mode np o no shutdown Note: The upstream Brocade SAN switch needs to be NPIV-enabled. To enable NPIV on port 13 on the Brocade SAN switch, enter the following command: portcfgnpivport 13 1 {Where 1 indicates that NPIV is enabled} # if port 13 is a 8G port, it is necessary to add: portcfgfillword 13 3 {Only applicable to 8G-capable ports} For more information about NPIV support for your Fibre Channel switch, refer to the switch vendor documentation. Associate each VLAN with a VSAN for each FCoE networks: o vlan 201 o fcoe vsan 201 o vlan 301 o fcoe vsan 301 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the second Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the VC module Bay 2 and the Nexus switch B with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/5-6 o channel-group 200 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10, Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

65 Create a trunk on the port channel interface to pass the FCoE VLANs 201 and 301 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o description FCoE Port Channel to VC-Bay2-X3/X4 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201,301 o spanning-tree port type edge trunk Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to port-channel 200: o interface vfc 2000 o description vfc for VC in Bay 2 o bind interface port-channel 200 o no shutdown # Assign it to VSAN201 and 301 as a trunk port o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 Create the two VSANs with assigned names o vsan database o vsan 201 name "3PAR-VSAN" o vsan 301 name "EVA-VSAN" Assign the Virtual Fibre Channel interface (vfc) to the first FCoE VSAN: o vsan 201 interface vfc 2000 Assign the Fibre Channel interfaces to the appropriate FCoE VSANs: o vsan 201 interface fc2/1 o vsan 301 interface fc2/2 Note: Alternatively, instead of dedicating one FC interface to one VSAN, you can use both FC interfaces to carry the two VSANs: o vsan 201 interface fc2/2 o vsan 301 interface fc2/1 Note: If there are multiple FC interfaces in the same VSAN, then it is not possible to assign an end device to a specific interface. Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10, Scenario 5: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Brocade SAN switches.

66 Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel) In this scenario, the Nexus switches operate as FCoE Gateways. This particular mode of operation is enabled by using the Nexus NPV mode. In this scenario, two Converged Shared Uplink Sets using multiple uplink ports and carrying multiple FCoE networks and multiple standard Ethernet networks are created between the Virtual Connect modules and the Nexus switches running in NPV mode. Each NPV Nexus switch is connected to a Cisco MDS SAN Fabric through a trunk SAN Port Channel. Figure 25: Physical view with a Cisco MDS Fabric with SAN Port Channel 3PAR FC Storage Arrays EVA VSAN 200 VSAN 201 VSAN 300 VSAN 301 Cisco MDS Series SAN Switch FC SAN A FC SAN B SAN Port Channel NPIV LAN SAN Port Channel NPIV Nexus 5xxx Series NPV Mode TNP-Ports TNP-Ports LACP Converged Ethernet/FCoE LACP Converged Ethernet/FCoE HP Virtual Connect FlexFabric or Flex-10/10D modules VC Domain Converged Ethernet/FCoE FC LAN FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Blade Servers 66 - Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

67 Benefits Although the NPV mode can be intended to help address third-party interoperability issues in multivendor environment, in large Cisco fabrics, the NPV mode can be used to simplify the switch configuration, the points of management and also to better control the number of domain IDs to increase the SAN scalability as you can add switches to the faric without adding more domain IDs but the number of ports to the fabric are increased. In NPV mode, fabric logins are passed upstream from the edge switch to the core MDS switch, this eliminates the need to configure unique domain ID and Zoning. The Nexus acts as a pass through switch and converts the Fabric Login (FLOGI) command to a Fabric Discovery (FDISC) command and passes it to the upstream SAN switch. The use of multiple FCoE uplink ports on each VC module connected to Nexus allows dynamically distribution of the FCoE traffic across the ports using a round robin format. By providing ability to define multiple FCoE VLANs on a single set of uplinks, VC allows multi-vsan capability to be enabled on the Nexus NPV switches and Cisco s MDS. VSANs allow an enterprise customer to maintain single physical SAN infrastructure, while providing separation and traffic isolation to the servers based on the business unit, types of the operating systems or need for logical isolation. Servers can therefore access to a different VSANs and connect to storage targets located in different logical SAN environment. F-Port Port Channeling with VSAN trunking, fully supported under the NPV mode, is also used between the Nexus and MDS core switches to increase the high availability and to create enough SAN subscription. Considerations In NPV mode, the switch does not operate as a typical Fibre Channel switch, fabric services (i.e. Name Service, Fabric Login, zoning, etc.) are not available and Fibre Channel switching is not performed: all traffic is switched upstream in the core switch. The NPV mode uses N Port ID Virtualization (NPIV) technology, F_Ports that normally connect to the fabric are configured in this mode as NP_Ports and use NPIV to register multiple WWPNs to the F_Port on the other end. Note: To convert a switch into NPV mode, you set the NPV feature to enable. This configuration command automatically triggers a switch reboot and the system configuration is erased. You cannot configure NPV mode on a per-interface basis. NPV mode applies to the entire switch. For more information about the NPV Mode and Cisco support, see Multiple VSAN support The number of FCoE networks per Shared Uplink Set is limited to 32. The number of FCoE networks is additionally limited by the 1000 network domain limit. For this feature to successfully integrate with Cisco Nexus switches it is important to note that for each FCoE VLAN a unique FC VSAN must be created and association between the two must be strictly configured as 1-to-1. Mapping of a single VLAN into multiple VSANs is not supported. This requirement is documented in Cisco s best practice recommendations. All of the FCoE networks will share the same lossless Priority Flow Control (PFC) queue on any specific shared uplink set. Sharing PFC queue between multiple VSANs will make it possible that traffic from one VSAN may have impact on traffic from other VSANs. For that reason, it is highly recommended to create enough SAN subscription (i.e. by using a suitable number of uplink between VC and Nexus but also between Nexus and Cisco MDS) to avoid any over-subscription issues that would prevent SAN to pause and therefore to impact other VSANs. Therefore the use of a SAN Port Channel with VSAN trunking between the Nexus and MDS switches is highly recommended Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

68 SAN Port Channel and Trunking The use of SAN Port Channel and VSAN trunking between the Nexus and MDS core switches are fully supported under the NPV mode. F-Port Port Channeling increases the uplink resiliency, optimizes the bandwidth utilization and increases the aggregated bandwidth with load balancing. SAN Port Channel between Nexus and upstream switches is only supported with Cisco switches (MDS or Nexus series). NP ports that are members of the same SAN Port Channel cannot be connected to different core switches. Figure 26: Cisco Nexus 5xxx switches operating as an NPV Gateway Server 1 (ENode) FC Storage Array A FCoE VLAN A FCoE VLAN B Server 2 (ENode) VN_Port VN_Port Lossless Ethernet network Port Channel VF_Ports NPIV SAN Port Channel with Trunking TNP_Ports N_Port VSAN A TF_Ports Native FC F_Port VSAN B F_Port N_Port FC Storage Array B Virtual Connect (FIP Snooping mode) Nexus 5xxx Series (NPV Mode) Cisco MDS (NPIV Enabled) Dual-Hop FCoE FCoE Gateway Converged Ethernet/FCoE FC This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of Nexus FCoE configuration guidelines, refer to the Fibre Channel over Ethernet Configuration Guides available at For information on Nexus FCoE supported configuration and design with Virtual Port Channel (vpc), refer to: Data Center Design with Cisco Nexus Switches and Virtual PortChannel Configuring Storage Array for FC/FCoE is beyond the scope of this document, consult your Storage Array documentations. Requirements Refer to the HP s SPOCK documents for information on: Minimum NX-OS version Minimum Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Note: FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG). This license is an option which is activated when the Nexus switch is shipped from Cisco Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

69 Guidelines Since the Nexus is running in NPV mode, it is mandatory to connect the Nexus switches to a native FC SAN infrastructure. FC/FCoE-based Storage targets cannot be connected to the Nexus switch when running in NPV mode. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. FCoE VLANs should be dedicated to FCoE traffic (i.e. it should not carry IP traffic). FCoE VLAN cannot be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by default VLAN 1). Interfaces connecting to VC must be configured as trunk ports and STP edge ports. NPIV support is required in FC switches that connect to the Nexus NP Ports. SAN Port Channel and VSAN trunking features requires a correct configuration of the upstream MDS switch. Ensure F- port trunking, F-port port channeling, and NPIV are enabled. Since the Nexus is running in NPV mode, all zoning needs to be performed from the Cisco MDS fabric. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section. Nexus configuration Current NX-OS limitations There is currently an NX-OS limitation with FCoE Port Channels on Cisco Nexus 5000 and 5500 series. When a Port Channel is carrying FCoE traffic, all interfaces of this port channel must belong to the same Nexus Unified Port Controller (UPC) ASIC. At this time, it is not supported to configure an FCoE port channel across different Unified Port Controllers. For more information, see Appendix A. Note: These limitations will be removed in a future NX software release. Details about the configuration: NPV mode is used. Interfaces eth1/5 and eth1/6 are connected to the VC modules thought FCoE using a port channel po200. Interfaces eth1/17 are connected to the datacenter LAN. Interfaces fc2/1 and fc2/2 are directly connected to Cisco MDS switches through a SAN Port Channel using NPIV (TNP_Ports). SAN port channels spo100 are using VSAN Trunking to pass FCoE VSANs 20x and 30x. VLAN IDs 200, 300, 201 and 301 are used for the FCoE networks. VLAN 200 maps to VSAN 200, VLAN 300 to VSAN 300, VLAN 201 to VSAN 201 and VLAN 301 to VSAN 301 VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 2000 bind to port channel po200 for the Virtual Connect FCoE connection. The EVA is connected to VSAN 30x and the 3PAR is connected to VSANs 20x Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

70 Figure 27: Physical diagram LAN WWN: 21:53:00:02:ac:00:15:9d VSAN 200 EVA WWN: WWN: 50:01:43:80:02:5D:19:7C 50:01:43:80:02:5D:19:79 Cisco MDS-A 9148 SAN-A SAN-B Cisco MDS-B 9148 Native fc1/9 fc1/10 fc1/9 fc1/10 VSAN Trunk spo100 3PAR WWN: 20:53:00:02:ac:00:15:9d VSAN 201 VSAN 300 VSAN 200 VSAN 300 VSAN 201 VSAN 301 VSAN 301 VSAN Trunk spo100 Ethernet + Native Fibre Channel Switch A Nexus-5010 NPV Mode eth1/5-6 vfc 2000 VLAN 1,10,20 VLAN 1,10,20 eth1/17 eth1/17 fc2/1-2 eth1/5-6 vfc 2000 fc2/1-2 Switch B Nexus-5010 NPV Mode FCoE VLAN 200, 300 VLAN 1,10,20 VSAN 200, 300 po200 X3 - X4 X3 X4 po200 FCoE VLAN 201, 301 VLAN 1,10,20 VSAN 201, 301 FCoE / DCB VC FlexFabric Bay 1 VC FlexFabric Bay 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:20 50:06:0b:00:00:c3:1a:22 Blade Servers HP BladeSystem c7000 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 WWN: WWN: 50:06:0b:00:00:c3:1a:2450:06:0b:00:00:c3:1a:26 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Converged Ethernet/FCoE FC LAN 70 - Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

71 Nexus switch-a configuration Enable NPV: o configure terminal o feature npv Note: Switching to NPV mode automatically performs a configuration erase and reloads the switch. Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate each VLAN with a VSAN for each FCoE networks: o vlan 200 o fcoe vsan 200 o vlan 300 o fcoe vsan 300 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the first Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the VC module Bay 1 and the Nexus switch A with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/5-6 o channel-group 200 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk on the port channel interface to pass the FCoE VLANs 200 and 300 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o description FCoE Port Channel to VC-Bay1-X3/X4 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200,300 o spanning-tree port type edge trunk 71 - Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

72 Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to port-channel 200: o interface vfc 2000 o description vfc for VC in Bay 1 o bind interface port-channel 200 o no shutdown # Assign it to VSAN200 and 300 as a trunk port o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 Create a SAN port channel with VSAN Trunking allowing the two FCoE VSANs (200 and 300) between the Nexus and MDS-A switch using the two fiber channel interfaces: o interface san-port-channel 100 o switchport description SAN-Trunk-Port-Channel to MDS-A o channel mode active o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 o interface fc 2/1-2 o switchport trunk mode on o switchport description Connected to MDS-A-fc1/9-10 o channel-group 100 o no shutdown Note: The MDS-A core switch needs to be NPIV-enabled and F port trunking and channeling enabled. For more information about NPIV support for your Fibre Channel switch, refer to the Cisco MDS documentation. To enable the F port trunking and channeling protocol on the MDS switch, use feature fport-channel-trunk. To enable NPIV use feature npiv. To configure of a TF port-channel on the MDS switch, use the following commands: interface port-channel 100 channel mode active switchport trunk allowed vsan 200 switchport trunk allowed vsan add 300 interface fc1/9-10 switchport description Connected to Nexus-A-fc2/1-2 channel-group 100 no shutdown Create the two VSANs with assigned names o vsan database o vsan 200 name "3PAR-VSAN" o vsan 300 name "EVA-VSAN" Assign the Virtual Fibre Channel interface (vfc) and the SAN Port Channel interface to the first FCoE VSAN: o vsan 200 interface vfc 2000 o vsan 200 interface san-port-channel 100 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10, Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

73 Nexus switch-b configuration Enable NPV: o configure terminal o feature npv Note: Switching to NPV mode automatically performs a configuration erase and reloads the switch. Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp Note: The Nexus can require a reload. Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate each VLAN with a VSAN for each FCoE networks: o vlan 201 o fcoe vsan 201 o vlan 301 o fcoe vsan 301 Note: Enabling FCoE on VLAN 1 is not supported. Note: Make sure that these VLAN IDs are the same as the second Shared Uplink Set FCoE s VLAN IDs in VCM. Create a port channel between the VC module Bay 2 and the Nexus switch B with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/5-6 o channel-group 200 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk on the port channel interface to pass the FCoE VLANs 201 and 301 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o description FCoE Port Channel to VC-Bay2-X3/X4 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201,301 o spanning-tree port type edge trunk 73 - Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

74 Create a virtual Fibre Channel interface for the connection to Virtual Connect and bind it to port-channel 200: o interface vfc 2000 o description vfc for VC in Bay 2 o bind interface port-channel 200 o no shutdown # Assign it to VSAN201 and 301 as a trunk port o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 Create a SAN port channel with VSAN Trunking allowing the two FCoE VSANs (201 and 301) between the Nexus and MDS-B switch using the two fiber channel interfaces: o interface san-port-channel 100 o switchport description SAN-Trunk-Port-Channel to MDS-B o channel mode active o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 o interface fc 2/1-2 o switchport trunk mode on o switchport description Connected to MDS-B-fc1/9-10 o channel-group 100 o no shutdown Note: The MDS-B core switch needs to be NPIV-enabled and F port trunking and channeling enabled. For more information about NPIV support for your Fibre Channel switch, refer to the Cisco MDS documentation. To enable the F port trunking and channeling protocol on the MDS switch, use feature fport-channel-trunk. To enable NPIV use feature npiv. To configure of a TF port-channel on the MDS switch, use the following commands: interface port-channel 100 channel mode active switchport trunk allowed vsan 201 switchport trunk allowed vsan add 301 interface fc1/9-10 switchport description Connected to Nexus-B-fc2/1-2 channel-group 100 no shutdown Create the two VSANs with assigned names o vsan database o vsan 201 name "3PAR-VSAN" o vsan 301 name "EVA-VSAN" Assign the Virtual Fibre Channel interface (vfc) and the SAN Port Channel interface to the first FCoE VSAN: o vsan 201 interface vfc 2000 o vsan 201 interface san-port-channel 100 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10, Scenario 6: Dual-Hop FCoE using Multi-FCoE networks and ports with Nexus 5xxx Series switch in NPV mode connected to Cisco MDS switches (using a SAN Port Channel)

75 Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target In this scenario, two Converged Shared Uplink Sets using a single or multiple uplink ports and carrying a single FCoE network and multiple standard Ethernet networks are created between the Virtual Connect modules and the HP 5900AF or HP FlexFabric 5900CP switches. Each HP 5900 switch is directly connected to an FCoE-based storage target. Figure 28: Physical view FCoE Storage Array LAN HP 5900AF/CP FCF Mode FCoE VF-Ports FCoE VF-Ports HP Virtual Connect FlexFabric or Flex-10/10D modules LACP Converged Ethernet/FCoE LACP Converged Ethernet/FCoE VC Domain FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 Blade Servers FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 200 FCoE VLAN 201 Converged Ethernet/FCoE FCoE LAN Benefits This configuration offers the simplicity of managing only one redundant DCB network with a single FCoE uplink or with multiple FCoE uplinks to offer increased performance and better availability. The use of a single VC uplink port from each VC module connected to the HP 5900 reduces the total number of FCoE switch ports needed in the datacenter and saves money. With multiple FCoE uplink ports, the server-to-uplink ratio is adjustable and the distribution of the FCoE traffic across the ports is dynamic using a round robin format. FCoE-based Storage devices are directly connected to the HP 5900 Series switches, this reduces complexity and cost by eliminating the need for multitier storage area networks Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

76 Considerations In this scenario, the HP 5900 switches must operate as Fiber Channel Forwarders (FCF). In this mode, the switch provides standard Fibre Channel switching capability and features including expansion, fabric, trunk VF and N ports, and aggregation of E-port and N-port virtualization; fabric services such as name server, registered state change notification, and login services; per-vsan fabric services, FSPF, soft and hard zoning, Fibre Channel traceroute, ping, debugging, and FIP snooping. It is not possible to directly connect a FC-based target on HP 5900AF switches because they do not provide support for Fibre Channel interfaces. Note: With HP FlexFabric 5900CP, it is also possible to connect a FC-based storage array. Figure 29: HP 5900 series switch operating as FC Forwarder (FCF) Server 1 (ENode) VN_Port Lossless Ethernet Server 2 (ENode) VN_Port network Port Channel VF_Port FCF VF_Ports FCoE VN_Port FCoE Storage Array Converged Ethernet/FCoE FCoE Virtual Connect (FIP Snooping mode) HP 5900AF/5900CP (FCF Mode) This cookbook provides only basic HP guidelines and configuration steps for the HP 5900 Switch Series, for a more complete and detailed configuration guidelines, refer to: HP 5920 & 5900 Switch Series FCoE Configuration Guide HP FlexFabric 5900CP Solutions Configuration Guide Configuring HP 3PAR StoreServ Storage for FCoE is beyond the scope of this document, consult the HP 3PAR documentations Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

77 Requirements Refer to the HP s SPOCK documents for information on: Minimum HP Comware Software version Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Note: License is not required for FCoE on HP 5900 switches. Guidelines It is mandatory to use the FCF Mode in order to directly attach FCoE-based Storage devices. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. The FCoE VLAN should be dedicated to FCoE traffic (i.e. it should not carry IP traffic). You must enable FCoE for the same VLAN and map this VLAN to the same VSAN on the two ends. Interfaces connecting to VC must be configured as trunk ports and STP edge ports. It is currently required by Comware Software to define one vfc interface for each Blade server using MAC binding. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section. HP 5900 configuration Details about the configuration: FCF mode is used. Option 1: Interfaces XGE1/0/12 are connected to the VC modules thought FCoE using a single uplink. Option 2: Interfaces XGE1/0/12-14 are connected to the VC modules thought FCoE using a bridge aggregation BAGG 20. Interfaces XGE1/0/7 are connected to an FCoE-based storage device (3PAR). Interfaces XGE1/0/23 are connected to the datacenter LAN. VLAN IDs 200 and 201 are used for the FCoE networks. VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). VSAN 200 and 201 will be used for the FCoE traffic. The virtual Fibre Channel interfaces vfc 1 bind to the MAC addresses of the first Blade server. The virtual Fibre Channel interfaces vfc 2 bind to the MAC addresses of the second Blade server. The virtual Fibre Channel interfaces vfc 7 bind to XGE1/0/7 for the FCoE-based storage target FCoE connection Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

78 Figure 30: Physical diagram 3PAR Storage Array WWN: 20:81:00:02:ac:00:15:9d WWN: 21:81:00:02:ac:00:15:9d LAN Switch A HP 5900 FCF Mode vfc 7 XGE1/0/7 FCoE VLAN 200 XGE1/0/12-14 FCoE VLAN 1,10,20 XGE1/0/23 FCoE vfc 7 XGE1/0/7 FCoE VLAN 201 XGE1/0/12-14 VLAN 1,10,20 XGE1/0/23 Switch B HP 5900 FCF Mode Native Ethernet + FCoE / DCB BAGG 20 vfc 1 vfc 2 FCoE VLAN 200 VLAN 1,10,20 VSAN 200 X3 X4 FCoE VLAN 201 VLAN 1,10,20 VSAN 201 X3 X4 BAGG 20 vfc 1 vfc 2 FCoE / DCB VC FlexFabric Bay 1 VC FlexFabric Bay 2 FCoE NIC CNA Port 1 WWN: 50:06:0b:00:00:c3:1e:08 FCoE NIC CNA Port 2 WWN: 50:06:0b:00:00:c3:1e:0a Blade Servers HP BladeSystem c7000 FCoE NIC CNA Port 1 WWN: 50:06:0b:00:00:c3:1e:0c FCoE NIC CNA Port 2 WWN: 50:06:0b:00:00:c3:1e:0e FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 200 FCoE VLAN 201 Converged Ethernet/FCoE FCoE LAN 78 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

79 HP 5900 switch-a configuration Enable the advanced operating mode (by default, a 5900 switch operates in non-fcoe mode): o system-view o system-working-mode advance Note: The switch requires a reboot (enter: save, quit then reboot). Configure Switch A to operate in FCF mode: o system-view o fcoe-mode fcf Configure DCBX: # Enable LLDP globally in case LLDP has been disabled (enabled by default): o lldp global enable. # Create Ethernet frame header ACL numbered 4000, and configure the ACL to permit FCoE packets (whose protocol number is 0x8906) and FIP packets (whose protocol number is 0x8914) to pass through. o acl number 4000 name DCBX o rule 0 permit type 8906 ffff o rule 1 permit type 8914 ffff # Create a class named DCBX, specify the operator of the class as OR, and use ACL 4000 as the match criterion of the class. o traffic classifier DCBX operator or o if-match acl 4000 # Create a traffic behavior named DCBX, and configure the action of marking packets with 802.1p priority 3. o traffic behavior DCBX o remark dot1p 3 # Create a QoS policy named DCBX, associate the class DCBX with traffic behavior DCBX in the QoS policy, and apply the association to DCBX. o qos policy DCBX o classifier DCBX behavior DCBX mode dcbx Configure ETS: # Configure the 802.1p-lp priority map to: - Map 802.1p priority 3 to local precedence 1 (corresponding to queue 1). - Map all other 802.1p priorities to local precedence 0 (corresponding to queue 0). o qos map-table dot1p-lp o import 0 export 0 o import 1 export 0 o import 2 export 0 o import 3 export 1 o import 4 export 0 o import 5 export 0 o import 6 export 0 o import 7 export 0 Create VLANs: (by default VLAN 1 is already configured) o vlan 10 o vlan Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

80 Create a VSAN 200 for the FCoE traffic: o vsan 200 # Disable the fabric configuration function in VSAN 200 o undo domain configure enable # Configure the domain ID as 200 in VSAN 200 o domain-id 200 static # Configuration of a zone alias for the FCoE Nodes. o zone-alias name FCoE-Nodes o member pwwn 50:06:0b:00:00:c3:1e:08 {This is the WWN of the first Blade FlexFabric Adapter port 1} o member pwwn 50:06:0b:00:00:c3:1e:0c {This is the WWN of the second Blade FlexFabric Adapter port 1} # Configuration of a zone alias for the 3PAR FCoE ports. o zone-alias name FCoE-ports-3PAR o member pwwn 20:81:00:02:ac:00:15:9d {This is the WWN of the first 3PAR FCoE controller port} # Configuration of the zone for the 3PAR Array o zone name fcoe-zone-3par o member zone-alias FCoE-Nodes o member zone-alias FCoE-ports-3PAR # Configuration of the zoneset for the 5900 VSAN 200 o zoneset name vsan o member fcoe-zone-3par o zoneset distribute full # Activating the zoneset o zoneset activate name vsan Enable FCoE for VLAN 200 and map it to VSAN 200 o vlan 200 o description FCoE-ToSAN-A o fcoe enable vsan 200 Note: Make sure that this VLAN ID is the same as the first Shared Uplink Set FCoE s VLAN ID in VCM. Configure the interface connected to 3PAR o interface ten-gigabitethernet1/0/7 o description Connected to 3PAR 0:8:1 # Assign the interface to VLAN 200 as a trunk port o port link-type trunk o port trunk permit vlan 200 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets 80 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

81 o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Create a virtual Fibre Channel interface for the 3PAR Array o interface vfc 7 o description vfc for 3PAR # Configure it to operate in F mode o fc mode f # Bind it to interface Ten-GigabitEthernet 1/0/7 (this is the Ethernet port that 3PAR StoreServ is connected to) o bind interface Te1/0/7 # Assign it to VSAN200 as a trunk port o port trunk vsan 200 Configure the interface connected to the datacenter LAN: o interface ten-gigabitethernet1/0/23 o description Connected to datacenter LAN # Assign the interface to VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Option 1: Configuration with one uplink between VC and Configure the interface connected to Virtual Connect Bay 1 o interface ten-gigabitethernet1/0/12 o description Connected to VC-Bay1-X3 # Assign the interface to VLAN 200 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan # Enable Spanning Port Fast o stp edged-port # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound 81 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

82 # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Create a virtual Fibre Channel interface for the first Blade server o interface vfc 1 o description vfc for Server-1 # Bind it to the MAC address of the first Blade server FCoE adapter 1 o bind interface Ten-GigabitEthernet1/0/12 mac 0017-a477-bc38 # Assign it to the FCoE VSAN o port trunk vsan 200 Note: The MAC Address of the FCoE adapter is found under the Virtual Connect Server Profile: 82 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

83 Create a virtual Fibre Channel interface for the second Blade server o interface vfc 2 o description vfc for Server-2 # Bind it to the MAC address of the second Blade server FCoE adapter 1 o bind interface Ten-GigabitEthernet1/0/12 mac 0017-a477-bc46 # Assign it to the FCoE VSAN o port trunk vsan 200 Option 2: Configuration with a bridge aggregation between VC and Create a link-aggregation interface to connect to Virtual Connect Bay 1 o interface bridge-aggregation 20 o description Aggregation to VC-Bay1-X3/X4 o link-aggregation mode dynamic # Enable Spanning Port Fast o stp edged-port # Assign the interface to VLAN 200 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Associate each interface to the link-aggregation o interface range te 1/0/12 te 1/0/14 o port link-aggregation group 20 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp 83 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

84 Make final settings on the link-aggregation interface connected to Virtual Connect Bay 1 o interface bridge-aggregation 20 # Assign the interface to VLAN 200 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Create a virtual Fibre Channel interface for the first Blade server o interface vfc 1 o description vfc for Server-1 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 1 o bind interface Bridge-Aggregation20 mac 0017-a477-bc38 # Assign it to the FCoE VSAN o port trunk vsan 200 Note: The MAC Address of the FCoE adapter is found under the Virtual Connect Server Profile: Create a virtual Fibre Channel interface for the second Blade server o interface vfc 2 o description vfc for Server-2 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 1 o bind interface Bridge-Aggregation20 mac 0017-a477-bc46 # Assign it to the FCoE VSAN o port trunk vsan Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

85 HP 5900 switch-b configuration Enable the advanced operating mode (by default, a 5900 switch operates in non-fcoe mode): o system-view o system-working-mode advance Note: The switch requires a reboot (enter: save, quit then reboot). Configure Switch B to operate in FCF mode: o system-view o fcoe-mode fcf Configure DCBX: # Enable LLDP globally in case LLDP has been disabled (enabled by default): o lldp global enable. # Create Ethernet frame header ACL numbered 4000, and configure the ACL to permit FCoE packets (whose protocol number is 0x8906) and FIP packets (whose protocol number is 0x8914) to pass through. o acl number 4000 name DCBX o rule 0 permit type 8906 ffff o rule 1 permit type 8914 ffff # Create a class named DCBX, specify the operator of the class as OR, and use ACL 4000 as the match criterion of the class. o traffic classifier DCBX operator or o if-match acl 4000 # Create a traffic behavior named DCBX, and configure the action of marking packets with 802.1p priority 3. o traffic behavior DCBX o remark dot1p 3 # Create a QoS policy named DCBX, associate the class DCBX with traffic behavior DCBX in the QoS policy, and apply the association to DCBX. o qos policy DCBX o classifier DCBX behavior DCBX mode dcbx Configure ETS: # Configure the 802.1p-lp priority map to: - Map 802.1p priority 3 to local precedence 1 (corresponding to queue 1). - Map all other 802.1p priorities to local precedence 0 (corresponding to queue 0). o qos map-table dot1p-lp o import 0 export 0 o import 1 export 0 o import 2 export 0 o import 3 export 1 o import 4 export 0 o import 5 export 0 o import 6 export 0 o import 7 export 0 Create VLANs: (by default VLAN 1 is already configured) o vlan 10 o vlan Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

86 Create a VSAN 201 for the FCoE traffic: o vsan 201 # Disable the fabric configuration function in VSAN 201 o undo domain configure enable # Configure the domain ID as 201 in VSAN 201 o domain-id 201 static # Configuration of a zone alias for the FCoE Nodes. o zone-alias name FCoE-Nodes o member pwwn 50:06:0b:00:00:c3:1e:0a {This is the WWN of the first Blade FlexFabric Adapter port 2} o member pwwn 50:06:0b:00:00:c3:1e:0e {This is the WWN of the second Blade FlexFabric Adapter port 2} # Configuration of a zone alias for the 3PAR FCoE ports. o zone-alias name FCoE-ports-3PAR o member pwwn 21:81:00:02:ac:00:15:9d {This is the WWN of the second 3PAR FCoE controller port} # Configuration of the zone for the 3PAR Array o zone name fcoe-zone-3par o member zone-alias FCoE-Nodes o member zone-alias FCoE-ports-3PAR # Configuration of the zoneset for the 5900 VSAN 201 o zoneset name vsan o member fcoe-zone-3par o zoneset distribute full Enable FCoE for VLAN 201 and map it to VSAN 201 o vlan 201 o description FCoE-ToSAN-B o fcoe enable vsan 201 Note: Make sure that this VLAN ID is the same as the second Shared Uplink Set FCoE s VLAN ID in VCM. Configure the interface connected to 3PAR o interface ten-gigabitethernet1/0/7 o description Connected to 3PAR 1:8:1 # Assign the interface to VLAN 201 as a trunk port o port link-type trunk o port trunk permit vlan 201 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p 86 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

87 # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Create a virtual Fibre Channel interface for the 3PAR Array o interface vfc 7 o description vfc for 3PAR # Configure it to operate in F mode o fc mode f # Bind it to interface Ten-GigabitEthernet 1/0/7 (this is the Ethernet port that 3PAR StoreServ is connected to) o bind interface Te1/0/7 # Assign it to VSAN200 as a trunk port o port trunk vsan 201 Configure the interface connected to the datacenter LAN: o interface ten-gigabitethernet1/0/23 o description Connected to datacenter LAN # Assign the interface to VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Option 1: Configuration with one uplink between VC and Configure the interface connected to Virtual Connect Bay 2 o interface ten-gigabitethernet1/0/12 o description Connected to VC-Bay2-X3 # Assign the interface to VLAN 201 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan # Enable Spanning Port Fast o stp edged-port # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: 87 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

88 Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Create a virtual Fibre Channel interface for the first Blade server o interface vfc 1 o description vfc for Server-1 # Bind it to the MAC address of the first Blade server FCoE adapter 2 o bind interface Ten-GigabitEthernet1/0/12 mac 0017-a477-bc39 # Assign it to the FCoE VSAN o port trunk vsan 201 Note: The MAC Address of the FCoE adapter is found under the Virtual Connect Server Profile: 88 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

89 Create a virtual Fibre Channel interface for the second Blade server o interface vfc 2 o description vfc for Server-2 # Bind it to the MAC address of the second Blade server FCoE adapter 2 o bind interface Ten-GigabitEthernet1/0/12 mac 0017-a477-bc47 # Assign it to the FCoE VSAN o port trunk vsan 201 Option 2: Configuration with several uplinks between VC and Create a link-aggregation interface to connect to Virtual Connect Bay 2 o interface bridge-aggregation 20 o description Aggregation to VC-Bay2-X3/X4 o link-aggregation mode dynamic # Enable Spanning Port Fast o stp edged-port # Assign the interface to VLAN 201 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Associate each interface to the link-aggregation o interface range te 1/0/12 te 1/0/14 o port link-aggregation group 20 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp 89 - Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

90 Make final settings on the link-aggregation interface connected to Virtual Connect Bay 2 o interface bridge-aggregation 20 # Assign the interface to VLAN 201 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Create a virtual Fibre Channel interface for the first Blade server o interface vfc 1 o description vfc for Server-1 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 2 o bind interface Bridge-Aggregation20 mac 0017-a477-bc39 # Assign it to the FCoE VSAN o port trunk vsan 201 Note: The MAC Address of the FCoE adapter is found under the Virtual Connect Server Profile: Create a virtual Fibre Channel interface for the second Blade server o interface vfc 2 o description vfc for Server-2 # Bind the Bridge Aggregation to the MAC address of the second Blade server FCoE adapter 2 o bind interface Bridge-Aggregation20 mac 0017-a477-bc47 # Assign it to the FCoE VSAN o port trunk vsan Scenario 7: Dual-Hop FCoE using a single FCoE network and multiple ports with HP 5900 Series switch in FCF mode directly connected to an FCoE-Based storage target

91 Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches In this scenario, the HP FlexFabric 5900CP switches operate as FCoE Gateways. This particular mode of operation is enabled by using NPV mode. In this scenario, two Converged Shared Uplink Sets using multiple uplink ports and carrying multiple FCoE networks and multiple standard Ethernet networks are created between the Virtual Connect modules and 5900CP switches. Each 5900 switch is connected to a Cisco MDS SAN Fabric through multiple NP ports carrying a single or multiple VSANs. Servers can access to different VSANs and connect to storage targets through different logical SAN environment. Figure 31: Physical view EVA Storage Array FC 3PAR Storage Array FC VSAN 200 VSAN 201 VSAN 300 VSAN 301 Cisco MDS Series SAN Switch FC SAN A FC SAN B NPIV LAN NPIV HP 5900CP NP V Mode VSAN 200 VSAN 300 VSAN 201 VSAN 301 NP-Ports NP-Ports LACP Converged Ethernet/FCoE LACP Converged Ethernet/FCoE HP Virtual Connect FlexFabric or Flex-10/10D modules VC Domain Converged Ethernet/FCoE FC LAN FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3P AR ) (Connected to E VA) Blade Servers Benefits This scenario can be used to simplify third-party interoperability concerns in environment where different types of Fibre Channel switches are used (i.e. when connecting HP5900 switches to a Cisco Fibre Channel switch like a MDS SAN switch) 91 - Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

92 because in NPV mode, the switch does not participate in domain operations or perform local switching. This enables multivendor topologies to be implemented without the restrictions the FCF mode requires. In NPV mode, fabric logins are passed upstream from the edge switch to the core MDS switch. The 5900 acts as a pass through switch and converts the Fabric Login (FLOGI) command to a Fabric Discovery (FDISC) command and passes it to the upstream SAN switch. The use of multiple FCoE uplink ports on each VC module connected to the 5900 allows dynamically distribution of the FCoE traffic across the ports using a round robin format. By providing ability to define multiple FCoE VLANs on a single set of uplinks, VC allows multi-vsan capability to be enabled on the 5900 switches. The use of VSANs can help to maintain separation and traffic isolation to the servers on the NPV switches since zoning is not supported. On the MDS side, zoning can be set to extend those VSANs to defined MDS zones that are connected to different storage targets. Considerations In NPV mode, the switch does not operate as a typical Fibre Channel switch, fabric services (i.e. Name Service, Fabric Login, zoning, etc.) are not available and Fibre Channel switching is not performed: all traffic is switched upstream in the core switch. The NPV mode uses N Port ID Virtualization (NPIV) technology, F_Ports that normally connect to the fabric are configured in this mode as NP_Ports and use NPIV to register multiple WWPNs to the F_Port on the other end. Multiple VSAN support The number of FCoE networks per Shared Uplink Set is limited to 32. VSAN tagging is not supported for native FC fabrics. The number of FCoE networks is additionally limited by the 1000 network domain limit. For this feature to successfully integrate with HP 5900 switches, it is important to note that for each FCoE VLAN a unique FC VSAN must be created and association between the two must be strictly configured as 1-to-1. Mapping of a single VLAN into multiple VSANs is not supported. All of the FCoE networks will share the same lossless Priority Flow Control (PFC) queue on any specific shared uplink set. Sharing PFC queue between multiple VSANs will make it possible that traffic from one VSAN may have impact on traffic from other VSANs. For that reason, it is highly recommended to create enough SAN subscription (i.e. by using a suitable number of uplink between VC and 5900 but also between 5900 and Cisco MDS) to avoid any over-subscription issues that would prevent SAN to pause and therefore to impact other VSANs. Therefore the use of multiple ports between the 5900 and MDS switches is highly recommended Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

93 Figure 32: Dual-Hop FCoE with HP FlexFabric 5900CP-48XG-4QSFP+ Switch operating as an FCoE Gateway (NPV) Server (ENode) Lossless Ethernet network FCoE Gateway Native FC VN_Port Port Channel Virtual Connect (FIP Snooping mode) VF_Ports HP 5900CP (NPV Mode) NP_Ports NPIV F_Ports Cisco MDS SAN Switch (NPIV Enabled) F_Port N_Port FC Storage Array Dual-Hop FCoE Converged Ethernet/FCoE FC This cookbook provides only basic HP guidelines and configuration steps for the HP 5900 Switch Series, for a more complete and detailed configuration guidelines, refer to: HP 5920 & 5900 Switch Series FCoE Configuration Guide HP FlexFabric 5900CP Solutions Configuration Guide Configuring Storage Array for FC is beyond the scope of this document, consult your Storage Array documentations. Requirements Refer to the HP s SPOCK documents for information on: Minimum HP Comware Software version Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Note: License is not required for FCoE on HP 5900 switches. Guidelines Since the 5900CP is running in NPV mode, it is mandatory to connect the HPN switches to a native FC SAN infrastructure. This SAN infrastructure can also be non-cisco FC devices (e.g. Brocade, Qlogic SAN switches). Refer to SPOCK for supported upstream switches. 5900CP cannot directly connect to FC/FCoE-based Storage devices since the switch is running in NPV mode. NPIV support is required in the FC switches that connect to the 5900CP NP Ports. Since the 5900CP is running in NPV mode, all zoning needs to be performed from the MDS fabric. When multiple NP ports belonging to the same VSAN are used to connect to the MDS fabric, the 5900CP uses dynamic login distribution to load balance the server connections across all available NP uplinks. The NP uplink interface with the minimum load is selected from the available NP uplinks as the external interface. Different NP ports can be connected to different core switches as long as they belong to the same SAN Fabric. When a NP uplink fails, the disrupted server connections re-login on other NP uplink. Cisco SAN Port Channel (TNP Ports) are not supported with non Cisco SAN fabrics (i.e with HP 5900CP) Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

94 When Multiple VSANs are used, there are 2 options: o If you want to assign servers to a specific FC interface: map one NP uplink to one VSAN. o If you don t need to assign servers to any specific FC interfaces: map all NP uplinks to all VSANs. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. FCoE VLANs should be dedicated to FCoE traffic (i.e. it should not carry IP traffic). Interfaces connecting to VC must be configured as trunk ports and STP edge ports. It is currently required by Comware Software to define one vfc interface for each Blade server using MAC binding. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section. HP 5900CP Switch configuration Details about the HP 5900CP configuration: NPV mode is used. Interfaces XGE1/0/12-14 are connected to the VC modules thought FCoE using a bridge aggregation BAGG 20. Interfaces fc1/0/35 and fc1/0/36 are connected to MDS Core switches using NPIV (NP_Ports). Interface fc1/0/35 is mapped to VSAN 200/201 and fc1/0/36 to VSAN 300/301 (Alternativelly, you can assign both interfaces to all VSANs). VLAN IDs 200/300 and 201/301 are used for the FCoE networks. VSAN 200/300 and 201/301 are used for the FCoE traffic. VLAN 200 maps to VSAN 200, VLAN 300 to VSAN 300, VLAN 201 to VSAN 201 and VLAN 301 to VSAN 301 Interfaces XGE1/0/15 are connected to the datacenter LAN. VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 1 bind to the MAC addresses of the first Blade server. The virtual Fibre Channel interfaces vfc 2 bind to the MAC addresses of the second Blade server. The EVA is connected to VSAN 30x and the 3PAR is connected to VSANs 20x. Details about the MDS 9148 configuration: FCF mode is used. Interfaces fc1/9 and fc1/10 are connected to the 5900CP switches thought FC (F_Ports). VSANs 200/300 are defined on the first fabric and VSANs 201/301 on the second one. Interfaces fc1/1 are connected to the EVA and fc1/2 are connected to the 3PAR. The EVA is connected to VSAN 30x and the 3PAR is connected to VSANs 20x Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

95 Figure 33: Physical diagram EVA Storage Array FC 3PAR Storage Array FC WWN: 50:01:43:80:02:5D:19:72 WWN: 50:01:43:80:02:5D:19:74 VSAN 300 VSAN 301 WWN: 21:53:00:02:ac:00:15:9d WWN: 20:53:00:02:ac:00:15:9d VSAN 200 VSAN 201 Cisco MDS-A 9148 fc1/1 fc1/2 VSAN 200 VSAN 300 fc1/1 fc1/2 fc1/9 fc1/10 fc1/9 fc1/10 VSAN 201 VSAN 301 Cisco MDS-B 9148 Native Ethernet + Native Fibre Channel LAN Switch A HP 5900CP NPV Mode VLAN 1,10,20 XGE1/0/15 FC1/0/35-36 VLAN 1,10,20 XGE1/0/15 XGE1/0/12-14 FCoE VLAN 200,300 FCoE VLAN 201,301 XGE1/0/12-14 VLAN 1,10,20 VLAN 1,10,20 BAGG 20 VSAN 200,300 VSAN 201,301 BAGG 20 vfc 1 vfc 2 vfc 1 vfc 2 X3 X4 X3 X4 FC1/0/35-36 Switch B HP 5900CP NPV Mode FCoE / DCB VC FlexFabric Bay 1 VC FlexFabric Bay 2 FCoE NIC CNA Port 1 WWN: 50:06:0b:00:00:c3:1e:08 FCoE NIC CNA Port 2 WWN: 50:06:0b:00:00:c3:1e:0a Blade Servers HP BladeSystem c7000 FCoE NIC CNA Port 1 WWN: 50:06:0b:00:00:c3:1e:0c FCoE NIC CNA Port 2 WWN: 50:06:0b:00:00:c3:1e:0e FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Converged Ethernet/FCoE FC LAN 95 - Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

96 HP 5900 switch-a configuration Enable the advanced operating mode (by default, a 5900 switch operates in non-fcoe mode): o system-view o system-working-mode advance Note: The switch requires a reboot (enter: save, quit then reboot). Configure Switch A to operate in NPV mode: o system-view o fcoe-mode npv Configure DCBX: # Enable LLDP globally in case LLDP has been disabled (enabled by default): o lldp global enable. # Create Ethernet frame header ACL numbered 4000, and configure the ACL to permit FCoE packets (whose protocol number is 0x8906) and FIP packets (whose protocol number is 0x8914) to pass through. o acl number 4000 name DCBX o rule 0 permit type 8906 ffff o rule 1 permit type 8914 ffff # Create a class named DCBX, specify the operator of the class as OR, and use ACL 4000 as the match criterion of the class. o traffic classifier DCBX operator or o if-match acl 4000 # Create a traffic behavior named DCBX, and configure the action of marking packets with 802.1p priority 3. o traffic behavior DCBX o remark dot1p 3 # Create a QoS policy named DCBX, associate the class DCBX with traffic behavior DCBX in the QoS policy, and apply the association to DCBX. o qos policy DCBX o classifier DCBX behavior DCBX mode dcbx Configure ETS: # Configure the 802.1p-lp priority map to: - Map 802.1p priority 3 to local precedence 1 (corresponding to queue 1). - Map all other 802.1p priorities to local precedence 0 (corresponding to queue 0). o qos map-table dot1p-lp o import 0 export 0 o import 1 export 0 o import 2 export 0 o import 3 export 1 o import 4 export 0 o import 5 export 0 o import 6 export 0 o import 7 export 0 Create VLANs: (by default VLAN 1 is already configured) o vlan 10 o vlan Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

97 Create VSAN 200 and 300 for the FCoE traffic: o vsan 200 o vsan 300 Associate each VLAN with a VSAN for each FCoE network: o vlan 200 o description FCoE-To-3PAR o fcoe enable vsan 200 o vlan 300 o description FCoE-To-EVA o fcoe enable vsan 300 Note: Make sure that this VLAN ID is the same as the first Shared Uplink Set FCoE s VLAN ID in VCM. Configure the interface connected to the datacenter LAN: o interface ten-gigabitethernet1/0/15 o description Connected to datacenter LAN # Assign the interface to VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Create a link-aggregation interface to connect to Virtual Connect Bay 1 o interface bridge-aggregation 20 o description Aggregation to VC-Bay1-X3/X4 o link-aggregation mode dynamic # Enable Spanning Port Fast o stp edged-port # Assign the interface to VLANs 200 and 300 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Associate each interface to the link-aggregation o interface range te 1/0/12 te 1/0/14 o port link-aggregation group 20 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count 97 - Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

98 Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Make final settings on the link-aggregation interface connected to Virtual Connect Bay 1 o interface bridge-aggregation 20 # Assign the interface to VLANs 200 and 300 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Change the Ethernet ports to FC ports: o interface range te 1/0/35 te 1/0/36 o port-type fc Configure the Fibre Channel interfaces connected to the Cisco MDS-A SAN switch: o interface range fc1/0/35 fc1/0/36 o description NP Port connected to MDS-A # Enable the NP mode on the vfc interface o fc mode np # Assign it to the FCoE VSANs as a trunk port o port trunk vsan o qos trust dot1p Create a virtual Fibre Channel interface for the first Blade server o interface vfc 1 o description vfc for Server-1 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 1 o bind interface Bridge-Aggregation20 mac 0017-a477-bc38 # Assign it to the FCoE 3PAR VSAN o port trunk vsan Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

99 Note: The MAC Address of the FCoE adapter is found under the Virtual Connect Server Profile: Create a virtual Fibre Channel interface for the second Blade server o interface vfc 2 o description vfc for Server-2 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 1 o bind interface Bridge-Aggregation20 mac 0017-a477-bc46 # Assign it to the FCoE EVA VSAN o port trunk vsan 200 MDS 9148 switch-a configuration Enable NPIV on the switch (disabled by default): o configure terminal o feature npiv Configuration of the Fibre Channel ports connected to the 5900CP: o interface fc1/9-10 o switchport description Connected to 5900CP-A-fc1/0/35-36 o switchport mode F o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 o no shutdown Configuration of the Fibre Channel ports connected to the storage arrays: o interface fc1/1 o switchport description Connected to EVA o no shutdown o interface fc1/2 o switchport description Connected to 3PAR o no shutdown 99 - Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

100 Assign the Fibre Channel interfaces (fc) to the FCoE VSANs: o vsan database o vsan 200 o vsan 200 name 3PAR-VSAN o vsan 200 interface fc1/2 o vsan 200 interface fc 1/9 o vsan 200 interface fc1/10 o vsan 300 o vsan 300 name EVA-VSAN o vsan 300 interface fc1/1 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1e:08 {This is the WWN of the first Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset-3par vsan 200 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 200 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 300 o member pwwn 50:01:43:80:02:5D:19:72 {This is the WWN of the first EVA controller port} o member pwwn 50:06:0b:00:00:c3:1e:0c {This is the WWN of the second Blade FlexFabric Adapter port 1} e. Create zoneset: o zoneset name zoneset-eva vsan 300 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan 300 HP 5900 switch-b configuration Enable the advanced operating mode (by default, a 5900 switch operates in non-fcoe mode): o system-view o system-working-mode advance Note: The switch requires a reboot (enter: save, quit then reboot). Configure Switch B to operate in NPV mode: o system-view o fcoe-mode npv Configure DCBX: # Enable LLDP globally in case LLDP has been disabled (enabled by default): o lldp global enable. # Create Ethernet frame header ACL numbered 4000, and configure the ACL to permit FCoE packets (whose protocol number is 0x8906) and FIP packets (whose protocol number is 0x8914) to pass through Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

101 o acl number 4000 name DCBX o rule 0 permit type 8906 ffff o rule 1 permit type 8914 ffff # Create a class named DCBX, specify the operator of the class as OR, and use ACL 4000 as the match criterion of the class. o traffic classifier DCBX operator or o if-match acl 4000 # Create a traffic behavior named DCBX, and configure the action of marking packets with 802.1p priority 3. o traffic behavior DCBX o remark dot1p 3 # Create a QoS policy named DCBX, associate the class DCBX with traffic behavior DCBX in the QoS policy, and apply the association to DCBX. o qos policy DCBX o classifier DCBX behavior DCBX mode dcbx Configure ETS: # Configure the 802.1p-lp priority map to: - Map 802.1p priority 3 to local precedence 1 (corresponding to queue 1). - Map all other 802.1p priorities to local precedence 0 (corresponding to queue 0). o qos map-table dot1p-lp o import 0 export 0 o import 1 export 0 o import 2 export 0 o import 3 export 1 o import 4 export 0 o import 5 export 0 o import 6 export 0 o import 7 export 0 Create VLANs: (by default VLAN 1 is already configured) o vlan 10 o vlan 20 Create VSAN 201 and 301 for the FCoE traffic: o vsan 201 o vsan 301 Associate each VLAN with a VSAN for each FCoE network: o vlan 201 o description FCoE-To-3PAR o fcoe enable vsan 201 o vlan 301 o description FCoE-To-EVA o fcoe enable vsan 301 Note: Make sure that this VLAN ID is the same as the second Shared Uplink Set FCoE s VLAN ID in VCM. Configure the interface connected to the datacenter LAN: o interface ten-gigabitethernet1/0/15 o description Connected to datacenter LAN Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

102 # Assign the interface to VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Create a link-aggregation interface to connect to Virtual Connect Bay 2 o interface bridge-aggregation 20 o description Aggregation to VC-Bay2-X3/X4 o link-aggregation mode dynamic # Enable Spanning Port Fast o stp edged-port # Assign the interface to VLANs 201 and 301 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Associate each interface to the link-aggregation o interface range te 1/0/12 te 1/0/14 o port link-aggregation group 20 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Make final settings on the link-aggregation interface connected to Virtual Connect Bay 2 o interface bridge-aggregation 20 # Assign the interface to VLANs 201 and 301 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

103 Change the Ethernet ports to FC ports: o interface range te 1/0/35 te 1/0/36 o port-type fc Configure the Fibre Channel interfaces connected to the Cisco MDS-A SAN switch: o interface range fc1/0/35 fc1/0/36 o description NP Port connected to MDS-A # Enable the NP mode on the vfc interface o fc mode np # Assign it to the FCoE VSANs as a trunk port o port trunk vsan o qos trust dot1p Create a virtual Fibre Channel interface for the first Blade server o interface vfc 1 o description vfc for Server-1 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 2 o bind interface Bridge-Aggregation20 mac 0017-a477-bc39 # Assign it to the FCoE 3PAR VSAN o port trunk vsan 201 Note: The MAC Address of the FCoE adapter is found under the Virtual Connect Server Profile: Create a virtual Fibre Channel interface for the second Blade server o interface vfc 2 o description vfc for Server-2 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 2 o bind interface Bridge-Aggregation20 mac 0017-a477-bc47 # Assign it to the FCoE EVA VSAN o port trunk vsan Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

104 MDS 9148 switch-b configuration Enable NPIV on the switch (disabled by default): o configure terminal o feature npiv Configuration of the Fibre Channel ports connected to the 5900CP: o interface fc1/9-10 o switchport description Connected to 5900CP-B-fc1/0/35-36 o switchport mode F o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 o no shutdown Configuration of the Fibre Channel ports connected to the storage arrays: o interface fc1/1 o switchport description Connected to EVA o no shutdown o interface fc1/2 o switchport description Connected to 3PAR o no shutdown Assign the Fibre Channel interfaces (fc) to the FCoE VSANs: o vsan database o vsan 201 o vsan 201 name 3PAR-VSAN o vsan 201 interface fc1/2 o vsan 201 interface fc 1/9 o vsan 201 interface fc1/10 o vsan 301 o vsan 301 name EVA-VSAN o vsan 301 interface fc1/1 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1e:0a {This is the WWN of the first Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset-3par vsan 201 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 201 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 301 o member pwwn 50:01:43:80:02:5D:19:74 {This is the WWN of the second EVA controller port} o member pwwn 50:06:0b:00:00:c3:1e:0e {This is the WWN of the second Blade FlexFabric Adapter port 2} e. Create zoneset: o zoneset name zoneset-eva vsan Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

105 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan Scenario 8: Dual-Hop FCoE using Multi-FCoE networks and ports with HP FlexFabric 5900CP switch in NPV mode connected to Cisco MDS switches

106 Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel) In this Multi-hop FCoE scenario, two Converged Shared Uplink Sets using a single or multiple uplink ports and carrying a single FCoE network and multiple standard Ethernet networks are created between Virtual Connect modules and HP 5900 switches. The FCoE traffic is then extended farther to Nexus 5xxx Series switches that are connected to a FC/FCoEbased storage target. Figure 34: Physical view LAN EVA Storage Array FCoE VSAN 300 VSAN 301 3PAR Storage Array FC VSAN 200 VSAN 201 Nexus 5xxx Series F CF Mode FCoE FC SAN A FC FCoE FC SAN B FC HP 5900AF NP V Mode LACP Converged Ethernet/FCoE Converged Ethernet/FCoE VSAN 200 VSAN 300 VSAN 201 VSAN 301 VNP-Ports LACP VNP-Ports LACP Converged Ethernet/FCoE HP Virtual Connect Modules LACP Converged Ethernet/FCoE VC Domain Converged Ethernet/FCoE FCoE FC LAN FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE NIC CNA Port 1 FCoE NIC CNA Port 2 FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3P AR ) (Connected to E VA) Blade Servers Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

107 Benefits This Multi-Hop FCoE scenario extends convergence beyond the Top of Rack switches up to a Nexus infrastructure without requiring any Fibre Channel SAN switch investments. With this scenario having HP 5900AF switches with no Fibre Channel port is not anymore a limitation as it uses Ethernet to transport the Fibre Channel data. Using multiple FCoE uplink ports on each VC module connected to the HP 5900 and on each HP 5900 connected to the Nexus allow dynamically distribution of the FCoE traffic across the ports using a round robin format. FCoE Port Channeling increases the high availability and create enough FCoE SAN subscription. Considerations Enabling convergence in the extended network requires more careful bandwidth management than a Dual-Hop FCoE topology. Suitable subscription is crucial to reduce the risk of congestion and slow performance, therefore the use of Port Channeling is recommended. The 5900AF-to-Nexus subscription is limited by the current Cisco NX software limitations, the ratio can go up to: 1.25:1 with the Nexus 5500 series (can support a maximum of 8 ports per port-channel when carrying FCoE traffic) 2.5:1with Nexus 5000 series (can support a maximum of 4 ports per port-channel when carrying FCoE traffic). The server-to-vc uplink ratio is adjustable and can go up to: 1.6:1 with the VC Flex-10/10D Module (10 FCoE ports max) 4:1 with the VC FlexFabric Module (4 FCoE ports max) 2:1 with the VC FlexFabric-20/40 F8 Module (8 FCoE ports max) In addition to a more careful bandwidth management, Multi-Hop FCoE shared infrastructure requires also much more attention, maintenance and upgrade practices, as these activities now have the potential to affect all traffic (FC+Ethernet) in the data center. In NPV mode, the switch does not operate as a typical Fibre Channel switch, fabric services (i.e. Name Service, Fabric Login, zoning, etc.) are not available and Fibre Channel switching is not performed: all traffic is switched upstream in the core switch. Multiple VSAN support The number of FCoE networks per Shared Uplink Set is limited to 32. VSAN tagging is not supported for native FC fabrics. The number of FCoE networks is additionally limited by the 1000 network domain limit. For this feature to successfully integrate with HP 5900 switches, it is important to note that for each FCoE VLAN a unique FC VSAN must be created and association between the two must be strictly configured as 1-to-1. Mapping of a single VLAN into multiple VSANs is not supported. All of the FCoE networks will share the same lossless Priority Flow Control (PFC) queue on any specific shared uplink set. Sharing PFC queue between multiple VSANs will make it possible that traffic from one VSAN may have impact on traffic from other VSANs. For that reason, it is highly recommended to create enough SAN subscription (i.e. by using a suitable number of uplink between VC and 5900 but also between 5900 and Cisco Nexus) to avoid any over-subscription issues that would prevent SAN to pause and therefore to impact other VSANs. Therefore the use of multiple ports between the 5900 and the Nexus is highly recommended Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

108 Figure 35: Multi-Hop FCoE with HP 5900AF switch operating as an FCoE Gateway (NPV) Server 1 (ENode) FCoE Storage Array A FCoE VLAN A VN_Port VN_Port FCoE VLAN B Server 2 (ENode) VN_Port Lossless Ethernet network Virtual Connect (FIP Snooping mode) Port Channel VF_Ports VNP_Ports HP 5900AF/5900CP (NPV Mode) Port Channel VSAN A VF_Ports VF_Port Nexus 5xxx Series (FCF Mode) F_Port VSAN B N_Port Native FC FC Storage Array B Multi-Hop FCoE Converged Ethernet/FCoE FCoE FC This cookbook provides only basic HP guidelines and configuration steps for the HP 5900 Switch Series, for a more complete and detailed configuration guidelines, refer to: HP 5920 & 5900 Switch Series FCoE Configuration Guide HP FlexFabric 5900CP Solutions Configuration Guide Configuring HP 3PAR StoreServ Storage for FCoE is beyond the scope of this document, consult the HP 3PAR documentations. Requirements Refer to the HP s SPOCK documents for information on: Minimum HP Comware Software version Virtual Connect firmware version HP FlexFabric Adapter Firmware and driver Note: License is not required for FCoE on HP 5900AF switches. Guidelines 5900AF or 5900CP cannot directly connect to FC/FCoE-based Storage devices since the switch is running in NPV mode. NPIV support is required in the Nexus switches that connect to the 5900 VNP Ports. Since the 5900 is running in NPV mode, all zoning needs to be performed from the Nexus fabric. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. FCoE VLANs should be dedicated to FCoE traffic (i.e. it should not carry IP traffic). You must enable FCoE for the same VLAN and map this VLAN to the same VSAN end-to-end Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

109 Interfaces connecting to VC must be configured as trunk ports and STP edge ports. It is currently required by Comware Software to define one vfc interface for each Blade server using MAC binding. FC and FCoE-based Storage devices connected to the Nexus switch are supported. Virtual Connect configuration Before you configure the upstream switches, Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS). Refer to the FCoE configuration in Virtual Connect section. If you need to improve the Ethernet traffic service level control, refer to the Quality of Service in the Virtual Connect Domain section. HP 5900 and Nexus Switch configuration Details about the HP 5900 configuration: NPV mode is used. Interfaces XGE1/0/12-14 are connected to the VC modules thought FCoE using a bridge aggregation BAGG 20. Interfaces XGE1/0/16-17 are connected to the Nexus switches thought FCoE using a bridge aggregation BAGG 30 (VNP_Ports). VLAN IDs 200/300 and 201/301 are used for the FCoE networks. VSAN 200/300 and 201/301 are used for the FCoE traffic. VLAN 200 maps to VSAN 200, VLAN 300 to VSAN 300, VLAN 201 to VSAN 201 and VLAN 301 to VSAN 301 VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 1 bind to the MAC addresses of the first Blade server. The virtual Fibre Channel interfaces vfc 2 bind to the MAC addresses of the second Blade server. The virtual Fibre Channel interfaces vfc 1000 bind to the bridge aggregation BAGG 30 connected to the Nexus. Details about the Nexus 5xxx configuration: FCF mode is used. Interfaces Eth1/11-12 are connected to the HP 5900 switches thought FCoE using a port channel po300 (VNP_Ports). Interfaces eth1/1 are connected to an FCoE-based storage device (EVA). Interfaces eth1/17 are connected to the datacenter LAN. Interfaces fc2/1 are directly connected to a FC-based storage device (3PAR). VLAN IDs 200/300 and 201/301 are used for the FCoE networks. VSAN 200/300 and 201/301 are used for the FCoE traffic. VLAN 200 maps to VSAN 200, VLAN 300 to VSAN 300, VLAN 201 to VSAN 201 and VLAN 301 to VSAN 301 VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-fcoe networks). The virtual Fibre Channel interfaces vfc 1000 bind to the port channel po300 connected to the 5900AF. The virtual Fibre Channel interfaces vfc 2007 bind to eth1/1 for the EVA FCoE-based storage array. The EVA is connected to VSAN 30x and the 3PAR is connected to VSANs 20x Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

110 Figure 36: Physical diagram EVA Storage Array FCoE 3PAR Storage Array FC Switch A Nexus 5xxx Series FCF Mode Switch A HP 5900 NPV Mode WWN: 50:01:43:80:02:5D:19:72 VSAN 300 vfc 2007 vfc 2007 VLAN 300 VLAN 1,10,20 VLAN 301 eth1/1 eth1/17 fc2/1 eth1/1 eth1/11-12 vfc 1000 po300 BAGG 30 vfc 1000 XGE1/0/16-17 WWN: 50:01:43:80:02:5D:19:74 VSAN 301 FCoE VLAN 200,300 VLAN 1,10,20 VSAN 200,300 LAN WWN: 21:53:00:02:ac:00:15:9d VSAN 200 VLAN 1,10,20 eth1/17 FCoE VLAN 201,301 VLAN 1,10,20 VSAN 201,301 XGE1/0/12-14 XGE1/0/12-14 FCoE VLAN 200,300 FCoE VLAN 201,301 VLAN 1,10,20 VLAN 1,10,20 BAGG 20 VSAN 200,300 VSAN 201,301 BAGG 20 vfc 1 vfc 1 vfc 2 vfc 2 X3 X4 X3 X4 WWN: 20:53:00:02:ac:00:15:9d VSAN 201 eth1/11-12 vfc 1000 po300 fc2/1 BAGG 30 vfc 1000 XGE1/0/16-17 Switch B Nexus 5xxx Series FCF Mode Switch B HP 5900 NPV Mode Native Ethernet + FCoE + FC FCoE / DCB VC FlexFabric Bay 1 VC FlexFabric Bay 2 FCoE NIC CNA Port 1 WWN: 50:06:0b:00:00:c3:1e:08 FCoE NIC CNA Port 2 WWN: 50:06:0b:00:00:c3:1e:0a Blade Servers HP BladeSystem c7000 FCoE NIC CNA Port 1 WWN: 50:06:0b:00:00:c3:1e:0c FCoE NIC CNA Port 2 WWN: 50:06:0b:00:00:c3:1e:0e FCoE VLAN 200 FCoE VLAN 201 FCoE VLAN 300 FCoE VLAN 301 (Connected to 3PAR) (Connected to EVA) Converged Ethernet/FCoE FCoE FC LAN Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

111 HP 5900 switch-a configuration Enable the advanced operating mode (by default, a 5900 switch operates in non-fcoe mode): o system-view o system-working-mode advance Note: The switch requires a reboot (enter: save, quit then reboot). Configure Switch A to operate in NPV mode: o system-view o fcoe-mode npv Configure DCBX: # Enable LLDP globally in case LLDP has been disabled (enabled by default): o lldp global enable. # Create Ethernet frame header ACL numbered 4000, and configure the ACL to permit FCoE packets (whose protocol number is 0x8906) and FIP packets (whose protocol number is 0x8914) to pass through. o acl number 4000 name DCBX o rule 0 permit type 8906 ffff o rule 1 permit type 8914 ffff # Create a class named DCBX, specify the operator of the class as OR, and use ACL 4000 as the match criterion of the class. o traffic classifier DCBX operator or o if-match acl 4000 # Create a traffic behavior named DCBX, and configure the action of marking packets with 802.1p priority 3. o traffic behavior DCBX o remark dot1p 3 # Create a QoS policy named DCBX, associate the class DCBX with traffic behavior DCBX in the QoS policy, and apply the association to DCBX. o qos policy DCBX o classifier DCBX behavior DCBX mode dcbx Configure ETS: # Configure the 802.1p-lp priority map to: - Map 802.1p priority 3 to local precedence 1 (corresponding to queue 1). - Map all other 802.1p priorities to local precedence 0 (corresponding to queue 0). o qos map-table dot1p-lp o import 0 export 0 o import 1 export 0 o import 2 export 0 o import 3 export 1 o import 4 export 0 o import 5 export 0 o import 6 export 0 o import 7 export 0 Create VLANs: (by default VLAN 1 is already configured) o vlan 10 o vlan Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

112 Create VSAN 200 and 300 for the FCoE traffic: o vsan 200 o vsan 300 Associate each VLAN with a VSAN for each FCoE network: o vlan 200 o description FCoE-To-3PAR o fcoe enable vsan 200 o vlan 300 o description FCoE-To-EVA o fcoe enable vsan 300 Note: Make sure that these VLAN IDs are the same as the first Shared Uplink Set FCoE s VLAN IDs in VCM. Create a link-aggregation interface to connect to Virtual Connect Bay 1 o interface bridge-aggregation 20 o description Aggregation to VC-Bay1-X3/X4 o link-aggregation mode dynamic # Enable Spanning Port Fast o stp edged-port # Assign the interface to VLANs 200 and 300 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Associate each interface to the link-aggregation o interface range te 1/0/12 te 1/0/14 o port link-aggregation group 20 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

113 o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Make final settings on the link-aggregation interface connected to Virtual Connect Bay 1 o interface bridge-aggregation 20 # Assign the interface to VLANs 200 and 300 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Create a link-aggregation interface to connect to Nexus Switch-A o interface bridge-aggregation 30 o description Aggregation to Nexus-Sw-A-eth1/11-12 o link-aggregation mode dynamic # Assign the interface to VLANs 200 and 300 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Associate each interface to the link-aggregation o interface range te 1/0/16 te 1/0/17 o port link-aggregation group 30 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

114 o qos wrr cs7 group sp Make final settings on the link-aggregation interface connected to Nexus Switch-A o interface bridge-aggregation 30 # Assign the interface to VLANs 200 and 300 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Create a virtual Fibre Channel interface for the FCoE Bridge Aggregation connected to the Nexus Switch-A o interface vfc 1000 o description vfc for BAGG30 to Nexus-A # Enable the NP mode on the vfc interface o fc mode np # Bind the vfc interface to the Bridge Aggregation o bind interface Bridge-Aggregation30 # Assign it to the FCoE VSANs as a trunk port o port trunk vsan Create a virtual Fibre Channel interface for the first Blade server o interface vfc 1 o description vfc for Server-1 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 1 o bind interface Bridge-Aggregation20 mac 0017-a477-bc38 # Assign it to the FCoE 3PAR VSAN o port trunk vsan 200 Note: The MAC Address of the FCoE adapter is found under the Virtual Connect Server Profile: Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

115 Create a virtual Fibre Channel interface for the second Blade server o interface vfc 2 o description vfc for Server-2 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 1 o bind interface Bridge-Aggregation20 mac 0017-a477-bc46 # Assign it to the FCoE EVA VSAN o port trunk vsan 300 Nexus switch-a configuration Enable FCoE on the switch (disabled by default): o configure terminal o feature fcoe Note: The Nexus can require a reload. Enable NPIV on the switch (disabled by default): o feature npiv Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate each VLAN with a VSAN for each FCoE networks: o vlan 200 o fcoe vsan 200 o vlan 300 o fcoe vsan 300 Note: Enabling FCoE on VLAN 1 is not supported. Create a port channel with the 5900 switch A with eth1/11 and eth1/12 eth1/11 and eth1/12 can be used because both are mapped to the same UPC (i.e. UPC#2) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/11-12 o channel-group 300 mode active Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk on the port channel interface to pass FCoE VLANs 200 and 300 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 300 o description FCoE Port Channel to 5900-A-GE1/0/16-17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200,300 Create a virtual Fibre Channel interface for the FCoE Port Channel connected to the 5900 Switch-A to pass the FCoE VSANs 200 and 300: o interface vfc1000 o description vfc for Port Channel to 5900-A-GE1/0/ Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

116 o bind interface port-channel 300 o switchport trunk allowed vsan 200 o switchport trunk allowed vsan add 300 o no shutdown Create a virtual Fibre Channel interface for the FCoE-based storage target and bind it to eth1/1: o interface vfc2007 o description vfc for EVA o bind interface eth1/1 o no shutdown # Assign it to VSAN300 as a trunk port o switchport trunk allowed vsan 300 Create a trunk on the interface connected to the EVA FCoE-based storage device to pass the FCoE traffic VLAN 300: o interface Ethernet1/1 o description FCoE uplink to EVA o switchport mode trunk o switchport trunk allowed vlan 300 o spanning-tree port type edge trunk Configure the Fiber Channel interface connected to the 3PAR array using VSAN 200: o interface fc 2/1 o switchport description Connected to the 3PAR array o no shutdown # Assign it to VSAN200 as a trunk port o switchport trunk allowed vsan 200 Assign the Virtual Fibre Channel interfaces (vfc) and the Fiber Channel interface to the FCoE VSANs: o vsan database o vsan 200 o vsan 200 interface fc2/1 o vsan 200 interface vfc 1000 o vsan 300 o vsan 300 interface vfc 2007 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1e:08 {This is the WWN of the first Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset-3par vsan 200 o member fcoe-zone-3par c. Activate zoneset: Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

117 o zoneset activate name zoneset-3par vsan 200 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 300 o member pwwn 50:01:43:80:02:5D:19:72 {This is the WWN of the first FCoE EVA port} o member pwwn 50:06:0b:00:00:c3:1e:0c {This is the WWN of the second Blade FlexFabric Adapter port 1} e. Create zoneset: o zoneset name zoneset-eva vsan 300 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan 300 HP 5900 switch-b configuration Enable the advanced operating mode (by default, a 5900 switch operates in non-fcoe mode): o system-view o system-working-mode advance Note: The switch requires a reboot (enter: save, quit then reboot). Configure Switch B to operate in NPV mode: o system-view o fcoe-mode npv Configure DCBX: # Enable LLDP globally in case LLDP has been disabled (enabled by default): o lldp global enable. # Create Ethernet frame header ACL numbered 4000, and configure the ACL to permit FCoE packets (whose protocol number is 0x8906) and FIP packets (whose protocol number is 0x8914) to pass through. o acl number 4000 name DCBX o rule 0 permit type 8906 ffff o rule 1 permit type 8914 ffff # Create a class named DCBX, specify the operator of the class as OR, and use ACL 4000 as the match criterion of the class. o traffic classifier DCBX operator or o if-match acl 4000 # Create a traffic behavior named DCBX, and configure the action of marking packets with 802.1p priority 3. o traffic behavior DCBX o remark dot1p 3 # Create a QoS policy named DCBX, associate the class DCBX with traffic behavior DCBX in the QoS policy, and apply the association to DCBX. o qos policy DCBX o classifier DCBX behavior DCBX mode dcbx Configure ETS: # Configure the 802.1p-lp priority map to: - Map 802.1p priority 3 to local precedence 1 (corresponding to queue 1). - Map all other 802.1p priorities to local precedence 0 (corresponding to queue 0). o qos map-table dot1p-lp o import 0 export Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

118 o import 1 export 0 o import 2 export 0 o import 3 export 1 o import 4 export 0 o import 5 export 0 o import 6 export 0 o import 7 export 0 Create VLANs: (by default VLAN 1 is already configured) o vlan 10 o vlan 20 Create VSAN 201 and 301 for the FCoE traffic: o vsan 201 o vsan 301 Associate each VLAN with a VSAN for each FCoE network: o vlan 201 o description FCoE-To-3PAR o fcoe enable vsan 201 o vlan 301 o description FCoE-To-EVA o fcoe enable vsan 301 Note: Make sure that this VLAN ID is the same as the second Shared Uplink Set FCoE s VLAN ID in VCM. Create a link-aggregation interface to connect to Virtual Connect Bay 2 o interface bridge-aggregation 20 o description Aggregation to VC-Bay2-X3/X4 o link-aggregation mode dynamic # Enable Spanning Port Fast o stp edged-port # Assign the interface to VLANs 201 and 301 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Associate each interface to the link-aggregation o interface range te 1/0/12 te 1/0/14 o port link-aggregation group 20 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

119 o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Make final settings on the link-aggregation interface connected to Virtual Connect Bay 2 o interface bridge-aggregation 20 # Assign the interface to VLANs 201 and 301 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Create a link-aggregation interface to connect to Nexus Switch-B o interface bridge-aggregation 30 o description Aggregation to Nexus-Sw-B-eth1/11-12 o link-aggregation mode dynamic # Assign the interface to VLANs 201 and 301 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Associate each interface to the link-aggregation o interface range te 1/0/16 te 1/0/17 o port link-aggregation group 30 # For DCBX: Enable LLDP on the interface, and enable the interface to advertise DCBX TLVs o lldp enable o lldp tlv-enable dot1-tlv dcbx Apply the QoS policy named DCBX to the outgoing traffic o qos apply policy DCBX outbound # For PFC: Configure the interface to automatically negotiate with its peer to enable PFC o priority-flow-control auto Enable PFC for 802.1p priority 3 on the interface o priority-flow-control no-drop dot1p 3 Configure the interface to trust the 802.1p priority carried in packets o qos trust dot1p # For ETS: Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

120 Enable byte-count Weighted Round Robin o qos wrr byte-count Assign 50% of the interface bandwidth to queue 1 (af1) for FCoE traffic o qos wrr af1 group 1 byte-count 1 Assign the other 50% to queue 0 (be) for standard LAN traffic. o qos wrr be group 1 byte-count 1 Assign all other queues to the SP group. o qos wrr af2 group sp o qos wrr af3 group sp o qos wrr af4 group sp o qos wrr ef group sp o qos wrr cs6 group sp o qos wrr cs7 group sp Make final settings on the link-aggregation interface connected to Nexus Switch-B o interface bridge-aggregation 30 # Assign the interface to VLANs 201 and 301 (FCoE) and VLAN 1,10,20 (Ethernet traffic) as a trunk port o port link-type trunk o port trunk permit vlan Create a virtual Fibre Channel interface for the FCoE link to the Nexus Switch-B o interface vfc 1000 o description vfc for BAGG30 to Nexus-B # Configure it to operate in NP mode o fc mode np # Bind the vfc interface to the Bridge Aggregation o bind interface Bridge-Aggregation30 # Assign it to the FCoE VSANs as a trunk port o port trunk vsan Create a virtual Fibre Channel interface for the first Blade server o interface vfc 1 o description vfc for Server-1 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 2 o bind interface Bridge-Aggregation20 mac 0017-a477-bc39 # Assign it to the FCoE 3PAR VSAN o port trunk vsan Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

121 Note: The MAC Address of the FCoE adapter is found under the Virtual Connect Server Profile: Create a virtual Fibre Channel interface for the second Blade server o interface vfc 2 o description vfc for Server-2 # Bind the Bridge Aggregation to the MAC address of the first Blade server FCoE adapter 2 o bind interface Bridge-Aggregation20 mac 0017-a477-bc47 # Assign it to the FCoE EVA VSAN o port trunk vsan 301 Nexus switch-b configuration Enable FCoE on the switch (disabled by default): o configure terminal o feature fcoe Note: The Nexus can require a reload. Enable NPIV on the switch (disabled by default): o feature npiv Configuring Quality of Service to provide lossless service for FCoE traffic and best-effort service for Ethernet traffic: o Refer to the Quality of Service configuration section. Associate each VLAN with a VSAN for each FCoE networks: o vlan 201 o fcoe vsan 201 o vlan 301 o fcoe vsan Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

122 Note: Enabling FCoE on VLAN 1 is not supported. Create a port channel with the 5900 switch B with eth1/11 and eth1/12 eth1/11 and eth1/12 can be used because both are mapped to the same UPC (i.e. UPC#2) on Nexus 5010 (for more information see Appendix A): o interface ethernet 1/11-12 o channel-group 300 mode active o lacp rate fast Create the VLANs for the IP traffic: o vlan 1,10,20 Create a trunk on the port channel interface to pass FCoE VLAN 201 and 301 and Ethernet traffic (VLAN 1,10,20): o interface port-channel 300 o description FCoE Port Channel to 5900-B-GE1/0/16-17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201,301 Create a virtual Fibre Channel interface for the FCoE Port Channel connected to the 5900 Switch-B to pass the FCoE VSANs 201 and 301: o interface vfc1000 o description vfc for Port Channel to 5900-B-GE1/0/16-17 o bind interface port-channel 300 o switchport trunk allowed vsan 201 o switchport trunk allowed vsan add 301 o no shutdown Create a virtual Fibre Channel interface for the EVA FCoE-based storage target and bind it to eth1/1: o interface vfc2007 o description vfc for EVA o bind interface eth1/1 o no shutdown # Assign it to VSAN301 as a trunk port o switchport trunk allowed vsan 301 Create a trunk on the interface connected to the EVA FCoE-based storage device to pass the FCoE traffic VLAN 301: o interface Ethernet1/1 o description FCoE uplink to EVA o switchport mode trunk o switchport trunk allowed vlan 301 o spanning-tree port type edge trunk Configure the Fiber Channel interface connected to the 3PAR array using VSAN 201: o interface fc 2/1 o switchport description Connected to the 3PAR array o no shutdown # Assign it to VSAN201 as a trunk port o switchport trunk allowed vsan Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

123 Assign the Virtual Fibre Channel interfaces (vfc) and the Fiber Channel interface to the FCoE VSAN: o vsan database o vsan 201 o vsan 201 interface fc2/1 o vsan 201 interface vfc 1000 o vsan 301 o vsan 301 interface vfc 2007 Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20 Configuration of the zone: a. Create the first VSAN zone for the 3PAR Storage target: o zone name fcoe-zone-3par vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1e:0a {This is the WWN of the first Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset-3par vsan 201 o member fcoe-zone-3par c. Activate zoneset: o zoneset activate name zoneset-3par vsan 201 d. Create the second VSAN zone for the EVA Storage target: o zone name fcoe-zone-eva vsan 301 o member pwwn 50:01:43:80:02:5D:19:74 {This is the WWN of the second FCoE EVA port} o member pwwn 50:06:0b:00:00:c3:1e:0e {This is the WWN of the second Blade FlexFabric Adapter port 2} e. Create zoneset: o zoneset name zoneset-eva vsan 301 o member fcoe-zone-eva f. Activate zoneset: o zoneset activate name zoneset-eva vsan Scenario 9: Multi-Hop FCoE using Multi-FCoE networks and ports with HP 5900 Series switch in NPV mode connected to Nexus 5xxx Series switches (using a FCoE Port Channel)

124 FCoE configuration in Virtual Connect Extending FCoE through FIP Snooping with HP Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS) which contain at least one FCoE network and optionally associated non-fcoe network(s). Standard design With the Virtual Connect standard design, the upstream network switches connect to each Virtual Connect module through two separate Converged Shared Uplink Sets, providing an Active/Active configuration. The use of multiple FCoE uplink ports on each VC module is highly recommended to aggregate the FCoE links to increase the high availability and to create enough subscription. Figure 37: Standard Virtual Connect Active-Active Converged Shared Uplink Sets scenario Nexus 5xxx Series ToR Switch using Fabric Mode TOR Switch 1 TOR Switch 2 Active FcoE X1 X q 802.3ad Active FcoE X1 X q 802.3ad Lossless Ethernet Network VC Module bay 1 VC Module bay 2 Converged Shared Uplink Sets DCB-1 VC Domain DCB-2 FlexNIC FlexHBA FlexNIC FlexNIC FlexFabric Adapter Port 1 FlexNIC FlexHBA FlexNIC FlexNIC FlexFabric Adapter Port 2 vswitch or NIC Teaming Multi Pathing Blade Server FCoE configuration in Virtual Connect

125 Alternative design An alternative scenario can be created using dedicated Share Uplink Sets for the FCoE traffic combined with Shared Uplink Sets for the standard Ethernet traffic since it is not mandatory to associate non-fcoe networks to a Converged Shared Uplink Set network. A typical scenario for this is when the Virtual Connect Domain cannot be modified and is already configured with Shared Uplink Sets that are using either Non-FCoE capable VC uplink ports or Dual Homed LACP configurations (it is not supported to connect a FCoE-capable Shared Uplink Set to two Top of Rack switches). This scenario is using additional uplink ports however these extra links used by the FCoE traffic can provide a better service level because the link is dedicated to the FCoE traffic and is not shared with any other types of network. Figure 38: Alternative Virtual Connect Active-Active Converged Shared Uplink Sets scenario with Dedicated SUS for the FCoE traffic. Nexus 5xxx Series ToR Switch using Fabric Mode FCF Mode vpc peer link FCF Mode FCoE Active Non-FCoE FCoE Active Non-FCoE Dual Homed LACP SUS configuration Shared Uplink Set (Non-FCoE traffic) Converged Shared Uplink Set (FCoE traffic) Lossless Ethernet Network VC Module bay 1 VC Module bay 2 SUS-1 SUS-FCoE-1 SUS-FCoE-2 VC Domain SUS-2 FlexNIC FlexHBA FlexNIC FlexNIC FlexFabric Adapter Port 1 FlexNIC FlexHBA FlexNIC FlexNIC FlexFabric Adapter Port 2 vswitch or NIC Teaming Multi Pathing Blade Server Converged Shared Uplink Sets (SUS) Details and Restrictions FCoE-capable Shared Uplink Sets (SUS) can contain both the FCoE network and non-fcoe networks. FCoE-capable SUS can support port aggregation (802.3ad). FCoE-capable SUS can support up to 32 FCoE networks. FCoE-capable SUS must always contain ports from a single VC module. Note: For Multi Enclosure Stacking (MES) environments, all corresponding ports in remote enclosures will be included in the same SUS. (e.g. selecting enc0:bay1:x1 means bay1:x1 in all remote enclosures is also included) FCoE configuration in Virtual Connect

126 Defining two Converged Shared Uplink Sets Creating the first SUS named DCB-1 On the Virtual Connect Manager screen, click Define, Shared Uplink Set to create a SUS. Enter the Network Name of DCB-1. Select Add Port, then add one or more ports from Bay 1. Note: In order to reduce the risk of congestion and increase the performance, make sure to use a suitable number of uplink to reach a good oversubscription ratio. For Connection Mode, use Auto. For LACP Timer, leave the Domain Default, Short (1 sec). This setting controls the requested frequency of LACP control messages on a LACP capable interface. Using the "long" setting can help prevent loss of LAGs while performing inservice upgrades on upstream switch firmware. For Associated FCoE Network, click on Add then enter all FCoE networks that are required: o Network Names: DCB-1-FCoE-1, DCB-1-FCoE-2, etc. o VLAN ID: enter the VLAN ID used by the FCoE traffic, it should match with the FCoE VLAN ID used by the upstream switch. (e.g. 200, 300, etc.) For Associated Networks, click on Add then enter all Ethernet networks that are required. o Select Smart Link for all defined networks. Note: For the removal of ports from a Converged Shared Uplink Set, it is recommended to first shutdown or remove a port from the vfc (interface vfc xx - no bind interface ethx/x) or channel group (interface ethx/x no channel-group xx mode active) on the FCF prior to removing the corresponding port from the Converged Shared Uplink Set in order to allow the FCF to properly halt traffic on the link. Failure to do so may result in loss of storage access FCoE configuration in Virtual Connect

127 Creating the second SUS named DCB-2 Enter the Network Name of DCB-2. Select Add Port, then add one or more ports from Bay 2. Note: In order to reduce the risk of congestion and increase the performance, make sure to use a suitable number of uplink to reach a good oversubscription ratio. For Connection Mode, use Auto. For LACP Timer, leave the Domain Default, Short (1 sec). This setting controls the requested frequency of LACP control messages on a LACP capable interface. Using the "long" setting can help prevent loss of LAGs while performing inservice upgrades on upstream switch firmware. For Associated FCoE Network, click on Add then enter all FCoE networks that are required: o Network Names: DCB-2-FCoE-1, DCB-2-FCoE-2, etc. o VLAN ID: enter the VLAN ID used by the FCoE traffic, it should match with the FCoE VLAN ID used by the upstream switch. (e.g. 201, 301, etc.) For Associated Networks, click on Add then enter all Ethernet networks that are required. o Select Smart Link for all defined networks. Note: For the removal of ports from a Converged Shared Uplink Set, it is recommended to first shutdown or remove a port from the vfc (interface vfc xx no/undo bind interface x/x) or from an HP bridge aggregation (undo bind interface Bridge-Aggregation xx) or with a Cisco channel group (interface ethx/x no channel-group xx mode active) on the FCF/FCoE Gateway prior to removing the corresponding port from the Converged Shared Uplink Set in order to allow the FCF to properly halt traffic on the link. Failure to do so may result in loss of storage access FCoE configuration in Virtual Connect

128 Figure 39: Example of a Converged Shared Uplink Set using two 40Gb QSFP+ uplink ports Warning: Since VC Firmware 4.30, the Port Status information has changed! All uplink ports in a Shared Uplink Set will show Linked-Standby if the networks contained in the SUS are not assigned to any server profiles FCoE configuration in Virtual Connect

129 Configuring a maximum FCoE bandwidth limit on servers Unifying storage traffic and network traffic on the same link can present some challenges and usually requires more careful bandwidth management than standard Ethernet environments. One easy solution is to maintain the server traffic to reasonable connection speed by using the VC Maximum connection speed option. This option limits the server traffic by providing a maximum connection speed option for defined networks. This option is recommended for high oversubscribed environment where the number of FCoE uplinks are limited which can increase the risk of congestion and could badly impact the network performance. Note: Maximum bandwidth optimization requires Flex-10 and FlexFabric adapter firmware and drivers to be updated to SPP or higher. This feature excludes support for the following adapters: HP NC551i Dual Port FlexFabric 10Gb Converged Network Adapter HP NC551m Dual Port FlexFabric 10Gb Converged Network Adapter HP NC550m 10Gb 2-port PCIe x8 Flex-10 Ethernet Adapter To configure a maximum bandwidth limit on servers for the FCoE traffic: Go to the Ethernet Networks and select the first FCoE network and click Edit. Check the Advanced Network Settings option. Check Set Maximum Connection Speed FCoE configuration in Virtual Connect

130 Select the best suitable speed for the maximum connection speed in order to meet your network design. Click Apply and Apply again to save the new FCoE network configuration. This value will limit the maximum port speed of the server s FlexHBA associated with this FCoE network. Note: With 2 x 10Gb uplinks per VC module connected to a Top of Rack switch using 50/50 bandwidth share and 16 servers using the FCoE network, it might be necessary to limit the maximum connection speed to get a reasonable FCoE over-subscription: o For a 2:1 oversubscription, use 1.3Gb o For a 4:1 oversubscription, use 2.5Gb o For a 8:1 oversubscription, use 5Gb o For a 16:1 oversubscription, use 10Gb Repeat the same steps for other FCoE networks FCoE configuration in Virtual Connect

131 Quality of Service Configuration The default Virtual Connect configuration provides lossless service of the FCoE traffic. PFC (Priority Flow Control) mechanisms are used to provide the FCoE lossless service and prevent congestion. It is therefore not necessary to enable any particular QoS settings in the VC Domain. Since Virtual Connect provides only best effort service for Ethernet traffic (i.e. flow control does not take place, in the event of congestion, packets get dropped) Administrator can decide to configure QoS in the Virtual Connect Domain and on the upstream switches to improve the Ethernet traffic service level control. The bandwidth allocation for FCoE-capable SUS uplinks always allocates only half the link bandwidth to FCoE traffic, with the other half allocated to Ethernet traffic. Since VC 4.01, this allocation is fixed and cannot be modified. Therefore it is important to configure the upstream switches with the same Virtual Connect Domain QoS settings as follow: FCoE lossless traffic class 3. 50/50% of bandwidth sharing between regular and lossless traffic. Note: Conflicting QoS parameters between the Virtual Connect Domain and the upstream environment can badly impact the traffic and affect the performance. QoS on Nexus 5010 and 5020 On the Nexus 5010 and 5020 it is not necessary to modify the QoS settings because by default once the FCoE feature is configured (by entering feature fcoe ) the default QoS template for shared links allocates the same shares and priorities than Virtual Connect. The following two classes of service are automatically configured: class-fcoe is configured to be no-drop with a COS priority 3 and a MTU of 2158 with an ingress/egress bandwidth allocation of 50%. class-default is configured to be drop with an MTU of 1500 with an ingress/egress bandwidth allocation of 50%. To verify that the FCoE policy maps can be found in the running configuration, enter: show policy-map system Quality of Service Configuration

132 QoS on Nexus 5500 and 6000 series On the Nexus 5500 and 6000 series, enabling the FCoE feature does not create automatically no-drop policies like on the Nexus 5000 series, so before enabling FCoE, it is necessary to attach the pre-defined FCoE policy maps to the type qos, type network-qos, and type queuing policy maps. The following is an example of a service policy that needs to be configured: o class-map type qos class-fcoe o class-map type queuing class-fcoe o match qos-group 1 o class-map type queuing class-all-flood o match qos-group 2 o class-map type queuing class-ip-multicast o match qos-group 2 o class-map type network-qos class-fcoe o match qos-group 1 o class-map type network-qos class-all-flood o match qos-group 2 o class-map type network-qos class-ip-multicast o match qos-group 2 o system qos o o o o service-policy type qos input fcoe-default-in-policy service-policy type queuing input fcoe-default-in-policy service-policy type queuing output fcoe-default-out-policy service-policy type network-qos fcoe-default-nq-policy To verify the FCoE system class is active on the interface connected to VC, enter: show queuing interface Ethernet 1/5 For more detailed description of the QoS configuration on Nexus 5000 series switches, see the Cisco Quality of Service Configuration Guides available at For Nexus 6000 series, see Quality of Service Configuration

133 Quality of Service in the Virtual Connect Domain It is not necessary to enable any particular QoS settings in the VC Domain to provide Quality of Service for the FCoE traffic. The moment FCoE is configured, Virtual Connect provides lossless service for the FCoE traffic. By default, the VC QoS configuration mode is Passthrough. In this mode, two classes of service are provided, one for FCoE with lossless service and one for Ethernet without any form of flow control. To improve the Ethernet traffic service level control and make VC to take a specific action on a specified classified traffic, the VC QoS settings can be changed from Passthrough to Custom (with FCoE Lossless). In this mode, VC monitors the Ethernet traffic for Layer p Priority bits, or Layer 3 DSCP or ToS markings and places packets on the pre-defined egress queues. The Ethernet traffic is then prioritized based on the traffic classification. Optional VCM QoS Configuration On the Virtual Connect Manager screen, click Configure then Quality of Service (QoS). Change the default Pass-through mode to Custom (with FCoE Lossless) Quality of Service Configuration

134 In this mode, VC supports up to 8 configurable traffic classes: o 1 predefined system class for Best Effort. o 1 predefined system class for FCoE Lossless. o 6 user defined classes. Note: The default 802.1p priority for the Lossless FCoE traffic class is 3. Note: The FCoE Lossless traffic share is based on the VC server profile configuration. The MAX Share is based on the FCoE Fabric configuration. For each user defined classes you want to use, check the Enabled box and enter the appropriate Share/Max share and 802.1p priority. Then click Apply Quality of Service Configuration

135 Connecting Servers to the FCoE networks To connect Blade servers to FCoE networks that have been configured, go to the Server Profiles interface. Select a profile and click Edit. Select the first FCoE HBA Connection and select one FCoE network. Note: The server administrator can select FCoE networks like any other storage networks when configuring profiles. Select the second FCoE HBA Connection and select a second FCoE network. The two Active/Active FCoE networks are now assigned to the server FCoE HBA Quality of Service Configuration

136 To configure a Minimum Allocated Port Speed for the FCoE traffic: Select Custom from the Port Speed Type drop down menu of the first FCoE port. Then select the port speed, this speed is the minimum guaranteed bandwidth. Maximum Connection Speed that had been defined for FCoE-1 Note: Make sure to select the best suitable Minimum Port Speed connection in order to meet your network design. Click Ok. Note: The Minimum Allocated Port Speed feature is an additional solution to overcome the challenges of network convergence discussed before. This Virtual Connect option guarantees that the FCoE traffic from server will get its minimum allocated bandwidth even if other FlexNICs on the same port generate too much traffic. Furthermore, if a congestion event occurs, the bandwidth is allocated based on the ratio of the Minimum setting. This option can really improve the Virtual connect bandwidth utilization and network performance. Note: Minimum and maximum bandwidth optimization requires Flex-10 and FlexFabric adapter firmware and drivers to be updated to SPP or higher. This feature excludes support for the following adapters: o HP NC551i Dual Port FlexFabric 10Gb Converged Network Adapter o HP NC551m Dual Port FlexFabric 10Gb Converged Network Adapter o HP NC550m 10Gb 2-port PCIe x8 Flex-10 Ethernet Adapter Repeat the same steps for the second FCoE port. Click Apply to activate the profile configuration Quality of Service Configuration

137 The Allocated Port Speed should now be displayed with the Min and Max that have been defined. The next step is to configure under the Server Operating System the NIC teaming and the FC multi-pathing for failover Quality of Service Configuration

138 Adding a FCoE volume in a multi-hop FCoE environment We are showing in this section how we can add a new volume when Virtual Connect is using FCoE to connect to an upstream infrastructure. We will describe the different steps that are required with both HP 5900 and Cisco Nexus switches in a VMware vsphere and Microsoft Windows Server environment. With HP 5900 switches With HP 5900 switch series, it is necessary to define for each Blade server, one vfc interface with MAC Binding due to some current ComWare Software limitation. So the very first step when adding a volume to a server in a VC/FCoE environment with HP 5900 switches is to identify the MAC and WWPN addresses of the FCoE ports. Identifying MAC and WWPN addresses of the FCoE adapter from VCM GUI or CLI The first place where you can find these addresses is the VCM interface (GUI or CLI). With VCM GUI Open the Virtual Connect profile assigned to the server. Record the WWPN and MAC addresses located in the FCoE HBA connections section: With VCM CLI Open a VCM CLI session, enter the following command and record the WWPN and MAC addresses located in the FCoE HBA connections section: ->show profile ESX-FCoE-Emulex Name : ESX-FCoE-Emulex Device Bay : enc0:8 Server : ProLiant BL460c Gen8 Status : OK Serial Number : CZJ23507RC UUID : A43-4A NAG : Default Hide Unused FlexNICs : true Adding a FCoE volume in a multi-hop FCoE environment

139 Ethernet Network Connections ========================================================================== Port Network Name Status PXE MAC Address Allocated Speed (min-max) ========================================================================== 1 DCB-1-Management OK UseBIOS A4-77-BC Gb-10Gb DCB-2-Management OK UseBIOS A4-77-BC-4A 3.6Gb-10Gb DCB-1-Vmotion OK UseBIOS A4-77-BC-4C 3.5Gb-10Gb DCB-1-Vmotion OK UseBIOS A4-77-BC-4E 3.5Gb-10Gb <Unassigned> OK UseBIOS A4-77-BC <Unassigned> OK UseBIOS A4-77-BC FCoE Connections ============================================================================== Port Connected Name Type Status Allocated WWPN MAC Address To Bay Speed (min-max) ============================================================================== 1 1 DCB-1- FCoE OK 2.9Gb-8Gb 50:06:0B:00: A4-77- FCoE-3 00:C3:1E:0C BC-46 PAR DCB-2- FCoE OK 2.9Gb-8Gb 50:06:0B:00: A4-77- FCoE-3 00:C3:1E:0E BC-47 PAR > Identifying MAC and WWPN addresses of the FCoE adapter from VMware ESX If you don t have access to both VCM GUI and CLI, you can use different methods under VMware according to the FlexFabric adapter type you are using: With Broadcom FlexFabric adapters o To get the MAC addresses of the FCoE ports, enter from the ESXi CLI client: ~ # esxcli fcoe nic list vmnic2 User Priority: 3 Source MAC: 00:17:a4:77:bc:38 Active: false Priority Settable: false Source MAC Settable: false VLAN Range Settable: false vmnic3 User Priority: 3 Source MAC: 00:17:a4:77:bc:39 Active: false Priority Settable: false Source MAC Settable: false VLAN Range Settable: false o To get the WWPN addresses of the FCoE ports, enter from the ESXi CLI client: ~ # esxcli storage san fcoe list Adapter: vmhba33 Port ID: Node Name: 50:06:0b:00:00:c3:1e:09 Port Name: 50:06:0b:00:00:c3:1e:08 Speed: 0 Gbps Port Type: Unknown Port State: OFFLINE Adding a FCoE volume in a multi-hop FCoE environment

140 Controller MAC: 00:17:a4:77:bc:38 FCF MAC: 00:00:00:00:00:00 VN Port MAC: 00:00:00:00:00:00 VLAN ID: 0 Vmnic Name: vmnic2 Adapter: vmhba34 Port ID: E50600 Node Name: 50:06:0b:00:00:c3:1e:0b Port Name: 50:06:0b:00:00:c3:1e:0a Speed: 0 Gbps Port Type: Unknown Port State: ONLINE Controller MAC: 00:17:a4:77:bc:39 FCF MAC: 00:0d:ec:cf:b4:c0 VN Port MAC: 0e:fc:00:e5:06:00 VLAN ID: 1001 Vmnic Name: vmnic3 With Emulex FlexFabric adapters You can use the HBACMD utility but you must install and use the appropriate Emulex CIM Provider for ESXi to remotely interrogate the adapter. For more information about OneCommand Manager CLI, see OneCommand Manager Command Line Interface User Manual, o To get the WWPN of the FCoE ports of a remote ESXi server, enter from a Windows Command Prompt: C:\Program Files\Emulex\Util\OCManager> hbacmd h=<ip address of the ESXi server> m=cim u=root p=password listhbas Manageable HBA List Port WWN : 50:06:0b:00:00:c3:1e:0c Node WWN : 50:06:0b:00:00:c3:1e:0d Fabric Name : 10:00:b8:af:67:0c:9d:27 Flags : Host Name : esx-iscsi-emulex.bladesystem.ne Mfg : Emulex Corporation Serial No. : H D Port Number : 1 Mode : Initiator PCI Bus Number : 4 PCI Function : 2 Port Type : FCoE Model : 554FLB Discovery : CIM Port WWN : 50:06:0b:00:00:c3:1e:0e Node WWN : 50:06:0b:00:00:c3:1e:0f Fabric Name : 20:c9:00:0d:ec:cf:b4:c1 Flags : Host Name : esx-iscsi-emulex.bladesystem.ne Mfg : Emulex Corporation Serial No. : H D Port Number : 2 Mode : Initiator PCI Bus Number : 4 PCI Function : 3 Port Type : FCoE Model : 554FLB Discovery : CIM Permanent MAC : b4-b5-2f-59-f3-10 Flags : Host Name : esx-iscsi-emulex.bladesystem.ne <...> Adding a FCoE volume in a multi-hop FCoE environment

141 o Then using the WWPN just discovered, you can find out the MAC address of the FCoE port (look at the IEEEE Address field), enter: C:\Program Files\Emulex\Util\OCManager> hbacmd h= m=cim u=root p=password hbaattrib 50:06:0b:00:00:c3:1e:0c HBA Attributes for 50:06:0b:00:00:c3:1e:0c Host Name : esx-iscsi-emulex.bladesystem.net Manufacturer : Emulex Corporation Serial Number : H D Model : 554FLB Model Desc : HP FlexFabric 10Gb 2-port 554FLB Adapter Node WWN : b c3 1e 0d Node Symname : ELX UCNA ASIC:554FLB:H D HW Version : E3 A1 FW Version : Vendor Spec ID : 19A2 Number of Ports : 2 Driver Name : lpfc820 Device ID : 0714 HBA Type : 554FLB Operational FW : IEEE Address : a4 77 BC 46 Boot Version : ` Driver Version : ; HBAAPI(I) v2.3.b, Board Temperature : Normal Function Type : FCoE Sub Device ID : 337B Port Number : 1 PCI Bus Number : 4 PCI Func Number : 2 Sub Vendor ID : 103C Firmware Status : Working Service Processor FW Name : Not Available ULP FW Name : Not Available IPL Filename : L32IHPB3 NCSI Version : N/A Start-up Boot Code Version : FCoE Universal Version : 7.06a11 FCoE x86 BIOS Version : 4.03a6 FCoE EFI Version : 5.21a2 FCoE FCODE Version : 4.01a2 UEFI NIC Version : UEFI FCODE Version : UEFI iscsi Version : PCI Express Link Speed : 5.0GT/s PCI Express Bus Width : x8 Flash Firmware Version : o Do the same for the second FCoE port Adding a FCoE volume in a multi-hop FCoE environment

142 Identifying MAC and WWPN addresses of the FCoE adapter from Microsoft Windows Server If you don t have access to both VCM GUI and CLI, you can use under Windows the utilities that are provided by the manufacturer of the FlexFabric adapter. With Broadcom FlexFabric adapters: With Broadcom, you can use BACSCLI. This is a CLI utility that enables you to view information and alternatively configure Broadcom network adapters. BACSCLI is installed when BACS is installed with the installer. For more information about BACS, BACS CLI, see o To get the WWPN and MAC addresses of the FCoE ports, from a Windows Command Prompt, enter from C:\Program Files\Broadcom\BACS: > BACScli "list fcport" Broadcom BACS Command Line Interface Utility for Windows v C FPMA MAC FCoE MAC DevType Name Host WIN-DG7UDMVBFUS 0 0EFC00C A477BC38 FCPort -Port 50:06:0B:00:00:C3:1E:08 1 0EFC00E A477BC39 FCPort -Port 50:06:0B:00:00:C3:1E:0A With Emulex FlexFabric adapters: With Emulex, you can use the HBACMD utility. This is a CLI utility that enables you to view information and alternatively configure Emulex network adapters. HBACMD is installed when OneCommand Manager is installed with the installer. For more information about OneCommand Manager CLI, see o To get the WWPN of the FCoE port, from a Windows Command Prompt, enter from C:\Program Files\Emulex\Util\OCManager: > hbacmd listhba Manageable HBA List Port WWN : 50:06:0b:00:00:c3:1e:00 Node WWN : 50:06:0b:00:00:c3:1e:01 Fabric Name : 00:00:00:00:00:00:00:00 Flags : Host Name : W Mfg : Emulex Corporation Serial No. : H T Port Number : 0 Mode : Initiator PCI Bus Number : 4 PCI Function : 2 Port Type : FCoE Model : 554FLB Port WWN : 50:06:0b:00:00:c3:1e:02 Node WWN : 50:06:0b:00:00:c3:1e:03 Fabric Name : 20:c9:00:0d:ec:cf:b4:c1 Flags : Host Name : W Mfg : Emulex Corporation Serial No. : H T Port Number : 0 Mode : Initiator PCI Bus Number : 4 PCI Function : 3 Port Type : FCoE Model : 554FLB <...> Adding a FCoE volume in a multi-hop FCoE environment

143 o Then using the WWPN just discovered, you can find out the MAC address of the FCoE port, enter: >hbacmd hbaattrib 50:06:0b:00:00:c3:1e:00 HBA Attributes for 50:06:0b:00:00:c3:1e:00 Host Name : W Manufacturer : Emulex Corporation Serial Number : H T Model : 554FLB Model Desc : HP FlexFabric 10Gb 2-port 554FLB Adapter Node WWN : b c3 1e 01 Node Symname : Emulex 554FLB FV V W HW Version : E3 A1 FW Version : Vendor Spec ID : 19A2 Number of Ports : 1 Driver Name : elxcna Device ID : 0714 HBA Type : 554FLB Operational FW : IEEE Address : a4 77 bc 00 Boot Code : Enabled Boot Version : Driver Version : Board Temperature : Normal Function Type : FCoE Sub Device ID : 337B Port Number : 0 PCI Bus Number : 4 <...> o Do the same for the second FCoE port. Configuring VFC interfaces Once you have the MAC addresses of your FCoE ports, you can define the virtual Fibre Channel interfaces for the two HP 5900 upstream switches. Create two Virtual Fibre Channel interfaces, one on each 5900 switch corresponding to each FCoE adapter and bind each vfc with the MAC address discovered in the previous section: o On the first 5900: the vfc interface is bound to the first FCoE MAC address: # interface vfc # bind interface Bridge-Aggregation 20 mac 0017-a477-bc46 Note: with 20 the link aggregation number used between VC-Bay1 and Note: with no link aggregation between VC/5900, use bind interface ten-gig instead # port trunk vsan 200 Note: 200 is the VSAN number used by FCoE. o On the second 5900: the vfc interface is bound to the second FCoE MAC address: # interface vfc # bind interface Bridge-Aggregation 20 mac 0017-a477-bc47 Note: with 20 the link aggregation number used between VC-Bay2 and Note: with no link aggregation between VC/5900, use bind interface ten-gig instead # port trunk vsan 201 Note: 201 is the VSAN number used by FCoE. Configuring the Storage Presentation From the Storage Array management interface, present the new LUN to the WWPN of the two FCoE adapter ports recorded earlier Adding a FCoE volume in a multi-hop FCoE environment

144 Configuring the zoning For better security and stability, it is recommended to use zoning. Add the FCoE adapters to the Zone (not required if using NPV mode) o On the first 5900: # vsan # zone-alias name FCoE-Nodes # member pwwn 50:06:0b:00:00:c3:1e: # member pwwn 50:06:0b:00:00:c3:1e:0c o On the second 5900: # vsan # zone-alias name FCoE-Nodes # member pwwn 50:06:0b:00:00:c3:1e:0a # member pwwn 50:06:0b:00:00:c3:1e:0e With Cisco Nexus switches With Cisco Nexus switches, it is not required to define one vfc interface for each blade with MAC binding, each Blade can log into the fabric using a single vfc interface that is bound to the interface port or interface port channel connected to VC. Configuring VFC interfaces Make sure the two Virtual Fibre Channel interfaces, one on each Nexus switch are correctly defined and mapped to the correct interface. Configuring the Storage Presentation From the Storage Array management interface, present the new LUN to the WWPN of the two FCoE adapter ports that can be found in the Virtual connect profile, see the previous section. Configuring the zoning For better security and stability, it is recommended to use zoning. Add the FCoE adapters to the Zone (not required if using NPV mode) o On the first Nexus: Nexus-1 # zone name fcoe-zone-3par vsan 200 Nexus-1 # member pwwn 50:06:0b:00:00:c3:1e:08 Nexus-1 # member pwwn 50:06:0b:00:00:c3:1e:0c o On the second Nexus: Nexus-2 # zone name fcoe-zone-3par vsan 201 Nexus-2 # member pwwn 50:06:0b:00:00:c3:1e:0a Nexus-2 # member pwwn 50:06:0b:00:00:c3:1e:0e Adding a FCoE volume in a multi-hop FCoE environment

145 Adding a FCoE volume under VMware ESX Under VMware vsphere, volumes are not automatically detected and installed after they are presented to the server. So it is necessary to rescan the vmhbas. Discovering new volumes Log in to the vsphere web Client, and select your host Select the Manage menu and click on the Storage Adapters tab Click on the Rescan All Storage button Then you can select the first FCoE adapter and click on Devices to verify that a target has been discovered Click on the newly discovered FCoE volume Note: if you don t detect any volumes, consult the Troubleshooting sections Adding a FCoE volume in a multi-hop FCoE environment

146 Select Paths and verify that two paths are detected: Make sure two paths are detected Note: if you don t detect two paths, consult the Troubleshooting sections. Adding a FCoE volume under Windows Server 2012 Under Microsoft Windows Server 2012, disk drives are automatically detected and installed as soon as they are presented to the server. For that reason, there is no need to rescan the network to find a new volume but one important step that is required is the installation of a Multipath solution. Installation of Microsoft MPIO MPIO solutions are needed to logically manage the FCoE redundant connections and ensure that the FCoE connection is available at all times. MPIO provides fault tolerance against single point of failure in hardware components but can also provide load balancing of I/O traffic, thereby improving system and application performance. MPIO is an optional component in all versions of Windows 2012 Server so it must be installed: Go to Server Manager Then select Manage Then Add Roles and Features Adding a FCoE volume in a multi-hop FCoE environment

147 In the Features section, select Multipath I/O Click Next and Install When the installation has completed, you need to restart the computer, click Yes After restarting the computer, the computer finalizes the MPIO installation. Then open the MPIO control panel by clicking on Server Manager Select the Tools menu then click on MPIO The MPIO Devices tab is displaying the hardware IDs of the devices that are managed by MPIO: Adding a FCoE volume in a multi-hop FCoE environment

148 Installation of the Device Specific Module (DSM) MPIO requires the installation of device specific modules (DSM) in order to provide support for using multiple data paths to a storage device. These modules are a server-side plug-in to the Microsoft MPIO framework. A native Microsoft DSM provided by Microsoft comes by default with Microsoft MPIO software but storage providers can develop their own DSM that contain the hardware specific information needed to optimize the connectivity with their storage arrays. Vendor specific DSM usually provides optimal performance, lower latency, more advanced path decisions and quicker path failover times while the Microsoft DSM offers more basic functions. Installation of the Microsoft DSM With Windows 2003 Server, the Microsoft DSM comes with the Microsoft iscsi Initiator. With Windows 2008 and 2012 Server, the Microsoft DSM is an optional component that comes with MPIO. When MPIO is installed, the Microsoft device-specific module (DSM) is also installed. For more information, see: - Microsoft Multipath I/O (MPIO) User s Guide for Windows Server 2012 at - Installing and Configuring MPIO for Windows 2008 Server at Configuring Microsoft MPIO for HP 3PAR Storage: To install and configure the native Microsoft MPIO for HP 3PAR StoreServ Storage, follow these steps: If you have not already done so, install any required support drivers for your FlexFabric adapter. In the MPIO Devices tab, click the Add button; the Add MPIO Support popup appears. In the Device Hardware ID: text box, enter 3PARdataVV, and click OK Adding a FCoE volume in a multi-hop FCoE environment

149 Reboot as directed. Note: You can also use MPIO-cli to add 3PARdataVV. The command is: "mpclaim -r -I -d "3PARdataVV" For more information, see HP 3PAR Windows Server 2012 and Windows Server 2008 Implementation Guide Adding a FCoE volume in a multi-hop FCoE environment

150 Installation of the Storage vendor DSM (optional) To determine which DSM to use with your storage, refer to information from your hardware storage array manufacturer. Many storage arrays that are SPC-3 compliant will work by using the Microsoft MPIO DSM but some manufacturers provide their own DSMs to achieve optimal performance. Note: Refer to your storage array manufacturer for information about which DSM to use with a given storage array, as well as the optimal configuration of it. If available, download and install the Vendor s DSM for your Storage Array Note: As a recommendation, if you are not sure about your storage array DSM support with Accelerated FCoE adapters or if you are facing issues after the vendor s DSM installation, it is recommended to use the Microsoft DSM that comes with the Microsoft MPIO software. Note: Once the Vendor s DSM for MPIO is installed, there are sometimes additional parameters specific for HP FlexFabric adapters (CNAs) so refer to your storage array manufacturer s documentation. All volume connections made to a Storage System will attempt to connect with the Storage vendor s DSM for MPIO. After the reboot, re-open the MPIO control panel, the MPIO Devices tab should list additional hardware IDs that have been installed Multipath Verification From a command prompt, type the following command to open the Device Manager: > mmc devmgmt.msc In the Disk Drives section, select and right-click the FCoE volume and click Properties Note: if you don t see any volumes, consult the Troubleshooting sections Adding a FCoE volume in a multi-hop FCoE environment

151 Select the MPIO tab. You can verify: o The MPIO policy (by default Round Robin is selected) o The DSM installed (native Microsoft or vendor specific DSM) o The number of path (2 paths should be detected) Default MPIO Policy Native Microsoft DSM is active 2 paths are detected by MPIO Note: if you don t detect two paths, consult the Troubleshooting sections Adding a FCoE volume in a multi-hop FCoE environment

152 Virtual Connect Troubleshooting Troubleshooting DCBX issues The Data Center Bridging exchange protocol (DCBX) is a discovery and capability exchange protocol to discover and exchange DCB configuration information between link partners, it provides the following functions: Discovers the peer devices' capabilities and determines whether devices at both ends support these capabilities. Detects configuration errors on peer devices. Remotely configures the peer device if the peer device accepts the configuration. Note: Virtual Connect supports only the remote configuration function. These DCB capabilities include the number of priority and traffic classes supported, the priority to traffic class maps, the scheduling algorithms for traffic classes and their parameters like the ETS bandwidth for each ETS enabled traffic class, the indication of which priorities have PFC enabled, the willingness to accept the recommendations (CNA). DCBX is designed as an extension of the Logical Link Discovery Protocol (LLDP), and provides new Type, Length, Value (TLV) definitions. DCBX information is essential when troubleshooting FCoE issues. It is necessary to verify that DCBX negotiation has successfully gone through between Virtual Connect and the DCB peer. DCBX information available in VCM DCBX information is provided under the Virtual Connect GUI and CLI. The UI provides the overall status of DCBX protocol exchange with peer entity, the status of the Application protocol, the Priority Flow Control and the Priority Group feature exchanges. Note: The DCBX information is only available for physical uplink and downlink ports. It is not available for Flex-10 subports. 1. Click the Interconnect Bays link on the VCM left-menu Virtual Connect Troubleshooting

153 2. Select one of the VC modules 3. Click on Uplink Ports tab 4. The Uplink Port Information window shows the status of the FCoE connection Virtual Connect Troubleshooting

154 Figure 40: Virtual Connect Uplink Port showing a successful FCoE connection Figure 41: Virtual Connect Uplink Port showing an unsuccessful FCoE connection 5. Then for the FCoE Uplink port you want to check, select Detailed Stats / Info. The Overall status value Ok indicates that no error is detected in DCBX operation, for example the protocol exchange is successfully completed or the port is not enabled for DCBX. Figure 42: Virtual Connect Uplink Port showing a successful DCBX negotiation Virtual Connect Troubleshooting

155 Pending status indicates the status of applying local DCBX configuration changes: The value False indicates that there is no pending DCBX exchange. The value True indicates that the DCBX exchange is not completed. Figure 43: When the DCBX negotiation is successful, the FCoE network status is green The status value Failed indicates an error in one of the DCBX feature information exchanges. The specific error reason is displayed in the individual feature state field below. Peer is not running DCBX Figure 44: Virtual Connect Uplink Port showing a DCBX negotiation failure and State fields are showing: Peer is not running DCBX Figure 45: VC Domain critical alert showing an FCoE network failure due to a DCBX issue Virtual Connect Troubleshooting

156 Figure 46: FCoE network status is red Possible reason: The peer DCB port is not enabled for DCBX or it is using an unsupported version of DCB. DCBX has three versions: DCBX Rev 1.00, DCBX Rev 1.01, and IEEE Std 802.1Qaz (standard version). Note: Virtual Connect Uplink Port will show a DCBX negotiation failure when DCB peer is sending DCBX Rev 1.00 TLVs. Only Pre-CEE DCBX versions are not supported by Virtual Connect (i.e. DCBX Rev 1.00). Peer does not indicate that it supports the feature Figure 47: Virtual Connect Uplink Port showing a DCBX negotiation failure and PFC State fields is showing: Peer does not indicate that it supports the feature Possible reason: The peer DCB port is not supporting PFC or PFC is disabled Virtual Connect Troubleshooting

157 Peer has an incompatible configuration and is unwilling to accept changes Figure 48: Virtual Connect Uplink Port showing a DCBX negotiation failure and PFC State fields is showing: Peer has an incompatible configuration and is unwilling to accept changes Possible reason: PFC is enabled on the peer DCB port but Priority 3 is not no-drop. Note: VC does not take PG state into account for determining the overall DCBX status Virtual Connect Troubleshooting

158 Peer has an incompatible configuration and is unwilling to accept changes but Overall status is OK Figure 49: Virtual Connect Uplink Port showing a successful DCBX negotiation but PG State fields is showing: Peer has an incompatible configuration and is unwilling to accept changes Possible reason: This error is usually displayed when VC is connected to a HP 5900 switch. VC is showing PG State as incompatible because the 5900 Series support 8 priority groups and VC supports 2 priority groups. Therefore, this Priority Group mismatch is not an error and will not make overall DCBX status to fail because VC does not take PG state into account for determining the overall DCBX status. The number of PG support is fixed and can t be modified in VC Virtual Connect Troubleshooting

159 Troubleshooting FCoE issues with FIP Snooping information FCoE Initialization protocol (FIP) is the control plane protocol that is used in the discovery and initialization of the Fibre Channel Virtual links between servers and FCFs. During the FIP discovery stage, the FCoE VLAN is exchanged with other parameters and the maximum FCoE frame size is negotiated. During the initialization phase, the FC Login/Logout and virtual links are established. FIP Snooping information under the VC GUI FIP Snooping information is provided under the Virtual Connect GUI by clicking on an individual VC module. The UI provides a table with FCoE uplink and downlink ports information that can help to verify which servers have correctly logged into the FCF. 1. Click the Interconnect Bays link on the VCM left-menu. 2. Click on one of the VC modules Virtual Connect Troubleshooting

160 3. Click on the Details tab. 4. Then select the FIP Snooping tab. 5. This page displays the FIP Snooping information for the specified interconnect module. o Information about FCoE Uplink and downlink ports, FCF and Fabric names are provided. o The information is provided by the FCF in snooped FCoE Initialization Protocol (FIP) messages. o This table displays also all server logins which have logged into the Fabric using FCoE. Each FCoE Adapter port logs into the Fabric using a different FC ID. Downlink port sessions are listed along with the VC assigned MAC addresses of the FCoE Adapters, the FCoE network names, the FCoE VLANs, the FCoE MAC address and the FCIDs assigned by the Fabric to every single server. Note: When the TOR switch is running in NPV mode, the Fabric name snooped/displayed by VC may not be consistent with the Fabric name displayed by the principal switch and the server Virtual Connect Troubleshooting

161 Virtual Connect FIP Snooping Command Lines For more detailed information about Virtual Connect command lines, see HP Virtual Connect Manager Command Line Interface for c-class BladeSystem Version User Guide show uplinkset <UplinkSetName> o This command displays a detailed view of uplink port sets including associated FCoE networks Virtual Connect Troubleshooting

162 show interconnect encx:y -fip_snooping o This command displays the fip-snooping information for the specified interconnect module. o Information about FCoE Uplink and downlink ports, FCF and Fabric names are provided. o This table displays also all server logins which have logged into the Fabric using FCoE. Each FCoE Adapter port logs into the Fabric using a different FC ID. Downlink port sessions are listed along with the VC assigned MAC addresses of the FCoE Adapters, the FCoE network names, the FCoE VLANs, the FCoE MAC address and the FCIDs assigned by the Fabric to every single server Virtual Connect Troubleshooting

163 Nexus Troubleshooting commands For more detailed information about Nexus FCoE troubleshooting guidance, see Disclaimer: we used the Cisco Nexus 5000 FCoE Troubleshooting Guide to create this section. Useful Show commands show flogi database o This command displays all logins (server nodes and storage devices) which have logged into the Fabric using FC and FCoE. Each FCoE Adapter port logs into the Fabric using a dedicated virtual FC interface. The vfc are listed along with the WWN addresses (VC assigned or factory default), the VSAN numbers and the FCIDs assigned by the Fabric to every single server. show fcoe database Server #1 Server #2 Server #3 Server #4 Server #5 o This command is the same as the previous one but only displays the FCoE logins. o It provides also the VC assigned MAC address of the FCoE adapter. o Interfaces are listed with their FCIDs assigned by the Fabric and their PWWNs and MAC addresses (VC assigned or factory default) Nexus Troubleshooting commands

164 show interface vfc 2009 o This command should display the vfc interface as up on VSAN where successful FLOGI happened. show fcoe o This command shows if FCoE is enabled. o It displays the FCF-MAC and the FC-MAP of the Nexus device. This command is useful because every FCoE frame sent by servers always use this FCF-MAC address as their Destination MAC Address Nexus Troubleshooting commands

165 show fcdomain o This command displays the domain information of all VSANs. This command includes the Domain IDs, the Fabric names, the priorities, the state of each FC domain (must be Stable), etc. o The output shows that the domain configuration is complete (state = stable) Nexus Troubleshooting commands

166 show platform software fcoe_mgr info interface vfc2008 Interface used by the vfc interface Solicitation happened on this VSAN VN Port information o This command allows to check if FCF discovery has gone through. show vsan membership o This command displays the member interfaces of all VSANs Nexus Troubleshooting commands

167 show policy-map o 2 traffic classes are defined, one for FCoE traffic and the other for Ethernet. show class-map class-fcoe o The FCoE traffic class is defined with priority 3. Note: COS 3 is the default Industry standard value for the FCoE Lossless traffic class (IEEE 802.1p). The same priority is used by the Virtual Connect Domain. show queuing interface ethernet 1/5 50% of the link bandwidth is reserved to the FCoE traffic qos-group 1 defines the lossless type of service (FCoE). o This commands shows the ETS (Enhanced Transmission Selection - IEEE 802.1Qaz) configuration of an interface. o It shows two traffic classes, one for the FCoE traffic (no-drop, MTU 2240) and one for the Ethernet traffic (drop, MTU 1500). o The bandwidth allocation for each traffic class is 50% of the link bandwidth Nexus Troubleshooting commands

168 show lldp dcbx interface Ethernet 1/13 o This command allows you to check the LLDP/DCBX negotiation has gone through with VC Nexus Troubleshooting commands

169 show system internal dcbx info interface ethernet 1/5 LLDP TLV advertised by Virtual Connect should be displayed by the Nexus interface Nexus Troubleshooting commands

170 The peer PFC and FCoE subtypes must show 0 error The DCBX counters located at the very bottom should show incremental numbers and no error o This command allows you to check the LLDP/DCBX negotiation error. o If PFC is enabled on different CoS values between VC and the Nexus switch, then a negotiation error will occur Nexus Troubleshooting commands

171 o Absence of operating configuration indicates that the peer does not support the DCBX TLV or that there is a negotiation error. show logging last 20 o This command displays the last 20 system log messages of the device. o Always very helpful, the Nexus logs can provide useful messages to help the troubleshooting. show logging grep vfc o This command is useful to check the status of the virtual interface used for the FCoE traffic. o The vfc can be up, down, administratively down, etc. o A down but waiting for flogi can mean that the setup is ok but no blade server has been turned on yet. show tech-support fcoe o This technical support command gathers information for FCoE troubleshooting Nexus Troubleshooting commands

172 Specific NPV Mode CLI commands In NPV mode, there is no familiar fcns or flogi database CLI commands for these services are not running but some other show commands are available: show npv flogi-table o This command displays all server logins which have logged into the Fabric and their assigned NP uplinks. o The vfc interfaces are listed along with the VC assigned WWN addresses, the VSAN numbers and the FCIDs assigned by the Fabric to every single server. o With a SAN Port Channel, the external interface column displays the port channel number. show npv status o This command displays the status of the NP uplink interfaces (i.e. external interfaces) and the server interfaces Nexus Troubleshooting commands

173 o With a SAN Port Channel, the command displays the state of the port channel (i.e. trunking mode) and lists the VSAN that are enabled. o Server interfaces with the initializing state are servers which have not logged into the Fabric. show npv external-interface-usage o This command is useful to display all server logins which have logged into the Fabric and their assigned NP uplinks. o With SAN Port Channel, the command displays the assigned port channel number Nexus Troubleshooting commands

174 show npv internal info global o This command displays the status of the NPV mode and the switch WWN. show npv internal info external-interface fc2/2 o This command displays information of the upstream switch where the fc interface is connected to, the commands provides Fabric management IP address (only available with MDS switches) the VSAN number, the FCF MAC address and the Fabric WWN. show npv internal info external-interface san-port-channel 100 o With a MDS SAN Fabric, this command displays the upstream MDS switch information: Fabric management IP address. FCF MAC address. Fabric Name. o The command also provides the VSAN numbers that are part of the port channel. Note: When the Nexus is running in NPV mode, the Fabric name may not be consistent from the Fabric name display in Virtual Connect, which is provided by the FCF in snooped FCoE Initialization Protocol (FIP) messages Nexus Troubleshooting commands

175 Per-Priority Flow Control commands Make sure PFC (Per-priority Flow Control Qbb) is enabled on the interface connected to VC. Flow Control must be disabled. PFC ensures the lossless behavior of the FCoE traffic. Check the status of Flow Control; it must be turned OFF on interfaces connected to Virtual Connect. o show interface ethernet 1/5 flowcontrol Check the status of the PFC. It must be turned ON on interfaces connected to Virtual Connect. o show interface ethernet 1/5 priority-flow-control If the command shows Mode as Off or On, it should be changed to Auto. To force PFC mode Auto on an interface, enter: interface ethernet 1/5 priority-flow-control mode auto If the command shows Mode as Auto and Oper as Off or if the command returns nothing, you should check the setup of the Shared Uplink Set in VCM by making sure that an associated FCoE Network has been properly defined Nexus Troubleshooting commands

176 Interface Status commands To check the status of the port connected to VC, you can enter: show interface status err-disabled VFC is usually down due to FIP solicitation failure and possible reasons are: o Fabric is not available for flogi. o Zoning or Storage presentation are misconfigured. show interface port-channel 200 o Use the show interface command to verify that the Nexus interface connected to VC is not showing errors, drops, discards, etc Nexus Troubleshooting commands

177 show interface vfc2005 o Use the show interface command to verify that the vfc interface is not showing errors or discards. o When a vfc interface is properly connected, the VSAN number used by the interface is shown in parenthesis in front of Trunk vsans (up). o When a vfc interface is not properly connected, Trunk vsans (up) is not showing any VSAN number in the parenthesis Nexus Troubleshooting commands

178 show interface vfc2008 brief o The show interface brief command allows to verify the bind information and the VSAN number used by the vfc interface. show interface san-port-channel 100 o This command displays the SAN Port channel information o When the Nexus is running in NPV mode, the port mode should be NP or TNP with VSAN trunking Nexus Troubleshooting commands

179 o When the Nexus is running in FCF mode, the port mode should be E or TE with VSAN trunking Nexus Troubleshooting commands

180 HP Networking Troubleshooting commands For more detailed information about HP 5900 series, see Useful FC/FCoE commands display fcoe-mode display fcoe o This command shows if FCoE is enabled. o It displays the FCoE MAC and the FC-MAP of the HP Networking device. This command is useful because every FCoE frame sent by servers always use this FCoE MAC address as their Destination MAC Address. o The FKA period shows the FIP Keep Alive advertisements that are sent by a vfc interface. display fc login o This command displays all FC logins (server nodes and storage devices) which have logged into the Fabric using FC and FCoE. Each FCoE Adapter port logs into the Fabric using a dedicated virtual FC interface. The vfc are listed along with the WWPN addresses (VC assigned or factory default), the VSAN numbers and the FCIDs assigned by the Fabric to every single server. display fc name-service database o This command displays the name service database information HP Networking Troubleshooting commands

181 display fc domain o This command displays the domain information of all VSANs. This command includes the Domain IDs, the Fabric names, the priorities, the state of each FC domain (must be Stable), the state of the fabric configuration function, etc. o VSAN 200 is the FCoE VSAN. The fabric configuration function is disabled for the FCoE VSAN Domain. o The output shows that the domain configuration is complete (state = stable). display vsan port-member o This command displays the member interfaces of all VSANs HP Networking Troubleshooting commands

182 Useful interface commands display interface brief o The output displays interfaces information in brief mode, including vfc interfaces. o This command is very useful to get a quick overview of all interfaces status (linked/unlinked, speed and type) HP Networking Troubleshooting commands

183 display interface te1/0/14 o This output shows a typical trunk interface connected to a VC uplink port, passing 4 VLANS (1, 10, 20 and 200 for the FCoE VLAN). o Useful command to verify that the 5900 interface connected to VC is not showing errors, drops, etc. o The pauses counters are useful to detect potential congestion somewhere in the network HP Networking Troubleshooting commands

184 display interface vfc1 o This command displays the vfc interface information, including the state (up or down), the FCoE MAC address and interface or BAGG bound to the vfc interface, the VSAN number and the mode of the vfc interface (F/E/NP). o The output shows that an FCoE communication is taking place as interface vfc1 is UP. o Administratively DOWN shows an interface that is administratively shut down. To put the interface back to administratively up, use the command undo shutdown. o DOWN shows an interface that is shut down. This can be due to an incorrect configuration of Virtual connect/5900, a server turned off, etc HP Networking Troubleshooting commands

185 display transceiver interface te1/0/8 o This command displays some useful information about the transceiver in use. The output will show a message if the transceiver is not supported by HP. o display transceiver alarm interface is another useful command to display transceiver alarm information. Useful QoS commands display qos policy user-defined display qos policy interface te1/0/12 o This output shows a typical QOS policy configuration of an interface connected to a VC module HP Networking Troubleshooting commands

186 display qos queue interface te1/0/12 o This output shows the typical ETS QOS queue configuration with 50% of the interface bandwidth assigned to queue 1 (af1) for FCoE traffic and the other 50% to queue 0 (be) for standard LAN traffic. display qos map-table dot1p-lp o This commands shows the 802.1p-priority to local-precedence mapping table for the ETS configuration. display qos trust interface te1/0/12 o This command allows to verify if 802.1p class of service required for Per-Priority Flow Control is enabled on the interface HP Networking Troubleshooting commands

187 Useful LLDP commands display lldp local-information interface te1/0/12 Indicates the DCBX version number used by the 5900 CoS-to-local priority mapping and bandwidth allocation. 50% is assigned to FCoE (CoS 3) P3-1: Indicates PFC is enabled for priority 3 assigned to FCoE traffic. 1-Enabled. 0-Disabled. Indicates that Link Aggregation is supported and enabled on this interface. o This command displays the LLDP information the 5900 interface is sending in the LLDP TLVs to its neighboring Virtual Connect uplink port. o All DCBX parameters must match with the neighbor information sent by the VC uplink port, see the following command HP Networking Troubleshooting commands

188 display lldp neighbor-information interface te1/0/12 verbose LLDP information sent by Virtual Connect to the 5900 interface. Indicates the DCBX version number used by VC CoS-to-local priority mapping and bandwidth allocation. 50% is assigned to FCoE (CoS 3) P3-1: Indicates PFC is enabled for priority 3 assigned to FCoE traffic. 1-Enabled. 0-Disabled. o This command displays the LLDP information received by the 5900 interface from its neighboring Virtual Connect uplink port HP Networking Troubleshooting commands

189 display lldp tlv-config interface Ten-GigabitEthernet 1/0/12 o This command allows you to check the LLDP/DCBX negotiation has gone through with VC. Useful log commands Always very helpful, the logs can provide useful messages to help the troubleshooting. display logbuffer reverse level 1 o This command displays the level 1 log messages with the most recent entry at the top. o Level 1 shows alert when an action must be taken immediately. For example, traffic on an interface exceeds the upper limit. display logbuffer reverse level 3 o This command displays the level 3 log messages with the most recent entry at the top. o Level 3 shows all error condition. For example, the link state changes HP Networking Troubleshooting commands

190 Specific NPV Mode CLI commands In NPV mode, there is no familiar fcns or flogi database CLI commands for these services are not running but some other show commands are available: display npv login o This command displays all server logins which have logged into the Fabric and their assigned NP uplinks or Vfc interfaces. o The vfc interfaces are listed along with the VC assigned WWN addresses, the VSAN numbers and the FCIDs assigned by the Fabric to every single server. display npv status o This command displays the status of the NP uplink interfaces (i.e. external interfaces) and the server interfaces. o Server interfaces with the down state are servers which have not logged into the Fabric. display npv traffic-map o This command displays the traffic mapping information (downlink-to-uplink interface mappings) of the NPV switch HP Networking Troubleshooting commands

191 Issues with FCoE performance If FCoE throughput on servers is very low, this can be due to pause frames. It is necessary to check if the switch is sending pause frames or if it is getting paused. Possible Cause If the egress FC port is congested, then the switch will send PFC frames to the Virtual Connect uplink port. The PFC frames are sent to reduce its FCoE rate and avoid a drop. If the server is slow or congested, then the server will send PFC frames to the Virtual Connect downlink port. As a result Virtual Connect will eventually send PFC Frames to the Nexus interface. To verify this situation, it is necessary to check the Nexus interface that is connected to the Virtual Connect modules. When slow FCoE performance is not due to Pause Frames For a few iterations using the "show interface ethx/x priority-flow-control" command, the PFC frame count (RX/TX) is not incrementing: For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for both RX and TX as Inactive: Solution: The slow FCoE performance does not seem to be due to network congestion. Check the Nexus interface to find information about possible link issues: Use the show interface command to verify that the Nexus interface connected to VC is not showing errors, drops, discards, etc. Use the show interface command to verify that the vfc interface is not showing errors or discards. Use the show queuing interface to verify the number of discards Issues with FCoE performance

192 When slow FCoE performance is due to Pause frames transmitted by the Nexus switch. Figure 50: Nexus sending PFC Pause frames due to FC congestion. FC Target LAN Congestion Point PPP Nexus interface connected to Virtual Connect that needs to be monitored When the Nexus switch is sending Pause frames to VC, the Nexus interface that is connected to the Virtual Connect module is showing the following: For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for TX as Active: Issues with FCoE performance

193 For a few iterations using the "show interface ethx/x priority-flow-control" command, the pause TxPPP counter is incrementing: Solution: Identify the source of the congestion and try to resolve it by increasing the FC bandwidth (or by adding additional Nexus FC ports to connect to the SAN environment). When slow FCoE performance is due to Pause frames transmitted by VC/Servers. Figure 51: Virtual Connect sending PFC Pause frames due to congestion. FC Target LAN Congestion Point PPP Nexus interface connected to Virtual Connect that needs to be monitored Congestion Point PPP When the Virtual Connect uplink is sending Pause frames to the Nexus, the Nexus interface that is connected to the Virtual Connect module is showing the following:. For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for Rx as Active as shown below: Issues with FCoE performance

194 For a few iterations using the "show interface ethx/x priority-flow-control" command, the pause RxPPP counter is incrementing: Solution: Identify the source of the congestion and try to resolve it with the following action: 1. Increase the FCoE connection speed (Min and Max) of every servers. 2. Increase the number of VC FCoE Uplink ports connected to the Nexus. Note: If congestion is expected, then Pause is expected for FCoE traffic. Note: PFC Pause frames are currently not available in the Virtual Connect statistics. Only the standard Ethernet Pause frames are displayed Issues with FCoE performance

195 Troubleshooting Common Issues VMware vsphere not detecting FCoE adapters Problem: Broadcom FCoE offload adapters are not listed or displayed as storage adapters by VMware ESX in the vsphere Client s Manage tab -> Storage -> Storage Adapters. Solution: See ~ # esxcli fcoe nic list vmnic2 User Priority: 3 Source MAC: 00:17:a4:77:bc:38 Active: false Priority Settable: false Source MAC Settable: false VLAN Range Settable: false vmnic3 User Priority: 3 Source MAC: 00:17:a4:77:bc:39 Active: false Priority Settable: false Source MAC Settable: false VLAN Range Settable: false ~ # esxcli fcoe nic discover -n vmnic2 Discovery enabled on device 'vmnic2' ~ # esxcli fcoe nic discover -n vmnic3 Discovery enabled on device 'vmnic3' Troubleshooting Common Issues

196 Troubleshooting using server adapter management suite Emulex OneCommand Manager The Emulex OneCommand Manager Application is a comprehensive management utility for Emulex and converged network adapters (CNAs) and host bus adapters (HBAs) that provides a powerful, centralized adapter management suite, including discovery, reporting. It s an excellent tool for troubleshooting. OneCommand Manager contains a graphical User Interface (GUI) and a Command Line Interface (CLI) OneCommand Manager is available under: Windows Server 2003/2008/2008 R2/2012 Linux VMware ESX Server See OneCommand Manager provides interesting features available for FCoE: Viewing FCoE Port information Viewing FCoE Port statistics Viewing FC/FCoE Fabric information Viewing FCoE target/lun information Target LUNs from Port 1 The port WWN Fabric Name FC ID provided by the FCF Target LUNs from Port 2 Note: When the Nexus is running in NPV mode, the Fabric name may not be consistent from the Fabric name display in Virtual Connect, which is provided by the FCF in snooped FCoE Initialization Protocol (FIP) messages Troubleshooting using server adapter management suite

197 LUN information Statistics Troubleshooting using server adapter management suite

198 FCoE Frames Analysis For network troubleshooting, capturing frames with a protocol analyzer can be very instructive but it is necessary to understand the way FCoE works. FCoE traffic use a different MAC address than the MAC address of the FCoE Adapter, the physical MAC address of the FCoE adapter (i.e. the MAC displayed in the Virtual Connect interface) is only used during FIP negotiation (FCoE Initialization Protocol). Figure 52: MAC address of the FCoE adapter found in the Server profile interface. During the FIP negotiation, the FCF assigned a Fabric Provided MAC address (FPMA) to the server. This address is used for FCoE forwarding and is the one Wireshark will detect during FCoE network captures. The FPMA is built by appending the FCID (assigned by the Fabric to the server) to the Nexus FC-MAP. Figure 53: The way the FPMA is built. 24 Bits 24 Bits FC-MAPs FC-ID 0E-FC-00 to 0E-FC-FF E FC-MAP 0E-FC-00 FC_ID E E-FC-00-E FPMA 48 bits FCoE Frames Analysis

199 How to find Fabric Provided MAC Addresses FPMA addresses assigned to servers can be found from either the Virtual Connect GUI or CLI: Under the GUI, the FPMA can be found by clicking on an individual module, then clicking on tabs Details then FIP Snooping : Under the CLI, the FPMA can be found by entering: show interconnect encx:y -fip_snooping FCoE Frames Analysis

200 FCoE Frames sent by servers: Destination MAC: the destination of FCoE frames sent by servers is always the MAC address of the Fibre Chanel Forwarder (i.e. the FCF-MAC address of the Nexus switch). Source MAC: the source of an FCoE frame is the Fabric Provided MAC Address (FPMA) assigned to the server by the FCF during the FIP negotiation. FCoE Frames received by servers: Destination MAC: the destination of an FCoE frame is the Fabric Provided MAC Address (FPMA) assigned to the server by the FCF during the FIP negotiation. Source MAC: the source of FCoE frames sent by the FCF to a server is always the MAC address of the Fibre Chanel Forwarder (i.e. the FCF-MAC address of the Nexus switch). Figure 54: Sample of an FCoE frame sent by a server FPMA (FC-MAP+FCID) FCID of the Storage Target FCID of the FCoE Server FCoE Frames Analysis

201 Appendix A: FCoE Port Channel limitations on Nexus switches There is currently an NX-OS limitation with FCoE Port Channels on Cisco Nexus switches. When a Port Channel is carrying FCoE traffic, all interfaces of this port channel must belong to the same Nexus Unified Port Controller (UPC) ASIC. At this time, it is not supported to configure a FCoE port channel across different Unified Port Controllers. This limitation will be removed in a future NX software release. As a result, we have the following limitations: Nexus 5010/5020 switches can support a maximum of four ports per port-channel when carrying FCoE traffic. Nexus 5548/5596 switches can support a maximum of eight ports per port-channel when carrying FCoE traffic. Nexus 6001/6004 switches can support a maximum of twelve 10 GE ports per port-channel or three 40GE ports per port-channel (two for the 6001) when carrying FCoE traffic. Identifying ports that belong to the same Nexus Unified Port Controller It is necessary while configuring the FCoE port channel group between Virtual Connect modules and Nexus switches to identify ports that belong to same Unified Port Controller (e.g. Eth1/5 and Eth1/6 because both interfaces are mapped to the same ASIC, UPC#1) by using the show hardware internal command. On Nexus 5010/5020 switches: The Nexus 5010 and 5020 switches use Gatos Unified Port Controller ASIC (UPC Gen 1). Each UPC ASIC have 4 ports mapped to front panel ports. This can be seen with the following command: Nexus-5010# show hardware internal gatos all-ports Gatos Port Info: name log gat mac flag adm opr c:m:s:l ipt fab xgat xpt if_index diag xgb1/ b7 dis dn 0:0:0:f xgb1/ b7 dis dn 0:1:1:f xgb1/ b7 dis dn 1:2:2:f xgb1/ b7 en up 1:3:3:f xgb1/ b7 en up 0:0:0:f xgb1/ b7 dis dn 0:1:1:f xgb1/ b7 en up 1:2:2:f xgb1/ b7 en up 1:3:3:f xgb1/ b7 en up 0:0:0:f xgb1/ b7 dis dn 0:1:1:f xgb1/ b7 en up 1:2:2:f xgb1/ b7 dis dn 1:3:3:f xgb1/ b7 dis dn 0:0:0:f xgb1/ b7 en dn 0:1:1:f xgb1/ b7 en up 1:2:2:f xgb1/ b7 dis dn 1:3:3:f a pass 0 1a pass 4 1a pass 6 1a pass 2 1a pass 0 1a pass 4 1a pass 6 1a pass 0 1a pass 0 1a00a000 pass 2 1a pass 2 1a00b000 pass 6 1a pass 6 1a pass 4 1a pass 4 1a pass Note: Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/3, Eth1/9, Eth1/17, etc. Column 3- Indicates the UPC ASIC number. Column 4- Indicates the UPC ASIC interface number. 0-3 are the 4 port available on the Gen-1 ASIC. Note: Only ports identified as mapped to the same UPC ASIC can be member of the same port channel. For example, Eth1/1 to Eth1/4 can be used in the same port channel because they are all mapped to UPC# Appendix A: FCoE Port Channel limitations on Nexus switches

202 Figure 55: Interface Eth1/5 cannot be part of a same Port Channel as Eth1/1 Eth1/4 Port Channel Xgb1/1 = Eth1/1 10G Ethernet Eth1/1 Eth1/2 Eth1/3 Eth1/4 Eth1/5 Eth1/6... Interfaces UPC # UPC #1 Only these interfaces can be part of the same FCoE Port Channel On Nexus 5548/5596 switches: The Nexus 5548 and 5596 switches use Carmel Unified Port Controller ASIC (UPC Gen 2). Each UPC ASIC have 8 ports mapped to front panel ports. This can be seen with the following command: Nexus-5548# show hardware internal carmel all-ports Carmel Port Info: name log car mac flag adm opr m:s:l ipt fab xcar xpt if_index diag ucver xgb1/ b7 dis dn 0:0:f a pass 0.00 xgb1/ b7 dis dn 1:1:f a pass 0.00 xgb1/ b7 dis dn 2:2:f a pass 0.00 xgb1/ b7 dis dn 3:3:f a pass 0.00 xgb1/ b7 dis dn 4:4:f a pass 0.00 xgb1/ b7 dis dn 5:5:f a pass 0.00 xgb1/ b7 dis dn 6:6:f a pass 0.00 xgb1/ b7 dis dn 7:7:f a pass 0.00 xgb1/ b7 dis dn 0:0:f a pass 0.00 xgb1/ p b7 en up 1:1:f a pass 0.00 Note: Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/3, Eth1/9, Eth1/17, etc. Column 3- Indicates the UPC ASIC number. Column 4- Indicates the UPC ASIC interface number. 0-7 are the 8 port available on the Gen-2 ASIC. Note: Only ports identified as mapped to the same UPC ASIC can be member of the same port channel. For example, Eth1/1 to Eth1/8 can be used in the same port channel because they are all mapped to UPC# Appendix A: FCoE Port Channel limitations on Nexus switches