FCoE for Small and Mid-size Enterprise Hui Chen Technical Marketing Engineer BRKSAN-2101

Transcription

1

2 FCoE for Small and Mid-size Enterprise Hui Chen Technical Marketing Engineer BRKSAN-2101

3 The Session Objectives: Provide a refresh of FCoE and DCBX Understand the basic FCoE implementation on Nexus 5K FCoE design options for small and mid-size enterprise FCoE deployment with Cisco Unified architecture Step-by-step configuration examples

4 The Session Non-Objectives: Nexus hardware architecture deep dive UCS storage architecture SAN distance extension using FCoE FCOE or iscsi, who is better for small and mid-size customers

5 Related Sessions Storage Contents BRKSAN Operational Models for FCoE Deployments - Best Practices and Examples BRKSAN Advanced Storage Area Network Design BRKSAN Storage Area Network Extension Design and Operation BRKVIR Fiber Channel Networking for the IP Network Engineer and SAN Core Edge Design Best Practices BRKCOM UCS Storage Integration, Technologies, and Topologies BRKDCT End-to-End QoS Implementation and Operation with Cisco Nexus Switches

6 Traditional Data Center Design Ethernet LAN and Fibre Channel SAN Ethernet FC FC L3 L2 Fabric A Fabric B LAN NIC HBA SAN

7 Agenda Introduction to FCoE Technology FCoE SAN Design for Small and Mid-size Enterprise Basic FCoE Configuration and Troubleshooting Conclusion

8 Agenda Introduction to FCoE Technology FCoE SAN Design for Small and Mid-size Enterprise Basic FCoE Configuration and Troubleshooting Conclusion

9 Block Storage Protocols (FC/FCoE/iSCSI)

10 What is SAN (Storage Area Network) LAN (TCP/IP) SAN (Fibre Channel, iscsi, FCoE) A dedicated network that provides access to consolidated, block level data storage.

11 Small Computer Systems Interface (SCSI) SCSI enables connection and data transfer between various independent peripheral devices and computers SCSI is a common protocol to connect Disks and Tape drives to the servers Protocols enabling SCSI communication between networked devices: Fiber Channel (SCSI over FCP) iscsi ( SCSI over TCP/IP) FCoE (Fiber Channel over Ethernet)

12 The SCSI I/O Transaction The SCSI protocol defines a bus based system used to carry block based storage commands The channel provides connectivity between server and storage The following shows two sample SCSI exchanges: Host (Initiator) DATA STATUS DATA DATA SCSI READ OPERATION SCSI I/O Channel Disk (Target) READ STATUS Host (Initiator) SCSI WRITE OPERATION SCSI I/O Channel Disk (Target) WRITE DATA DATA DATA

13 SAN Protocols Network Stack Comparison SCSI iscsi FCoE FibreChannel SCSI SCSI SCSI SCSI iscsi FCP FC FCP FC TCP IP Ethernet FCoE Lossless Ethernet PHYSICAL WIRE

14 Block Storage Networking Protocols FC FCoE iscsi SCSI transport protocol that operates over Fiber Channel FC frames with the SCSI CDB payload are transported over Fiber channel Protocol Works on Fiber Channel switches OXID/RXID generated for every I_T pair conversation Needs Zoning Runs on dedicated lossless Fiber Channel networks Limited by distance Well suited for latency sensitive and high I/O applications Mapping of Fibre Channel frames over Ethernet Fibre Channel is enabled to run on a lossless Ethernet network Works on Ethernet switches with FCF capability OXID/RXID generated for every I_T pair conversation Needs Zoning Uses Ethernet and needs lossless network Limited by distance Reduces the TCO of the fabrics by preserving the advantages of FC networks SCSI transport protocol that operates over TCP Encapsulation of SCSI command descriptor blocks and data in TCP/IP byte streams Works on any Ethernet switch ISID/TSID generated for every I_T pair conversation Zoning not required Works on TCP and subject to losses in network No Distance limitations Well suited for applications with less I/O requirements while reducing the TCO

15 Host/ Server Storage Transport Storage Media Unified Fabric and FCoE All Data accessed over a common fabric FC SAN Computer System Application File System Volume Manager SCSI Device Driver FC Driver HBA SAN FC FCoE SAN Computer System Application File System Volume Manager SCSI Device Driver FCoE Driver CNA FCoE Block I/O iscsi Appliance Computer System Application File System Volume Manager SCSI Device Driver iscsi Driver TCP/IP Stack NIC NIC TCP/IP Stack iscsi Layer Bus Adapter iscsi Gateway Computer System Application File System Volume Manager SCSI Device Driver iscsi Driver TCP/IP Stack NIC Unified Fabric NIC TCP/IP Stack iscsi Layer FC HBA FC NAS Appliance Computer System IP Application File System I/O Redirector NFS/CIFS TCP/IP Stack NIC NIC TCP/IP Stack File System Device Driver File I/O NAS Gateway Computer System Application File System I/O Redirector NFS/CIFS TCP/IP Stack NIC NIC TCP/IP Stack File System FC HBA FC

16 Basics of FCoE

17 What is Fibre Channel over Ethernet (FCoE) FCoE Mapping of FC Frames over Ethernet Enables FC on a Lossless Ethernet Network Ethernet Fibre Channel Traffic Fewer Cables Both block I/O & Ethernet traffic co-exist on same cable Fewer adapters needed Overall less power Interoperates with existing SAN s Management of SAN s remains constant No Gateway Benefits

18 FCoE Protocol Fundamentals Protocol Organization Data and Control Plane FC-BB-5 defines two protocols required for an FCoE enabled Fabric Data Plane FCoE It is used to carry most of the FC frames and all the SCSI traffic Uses Fabric Assigned MAC address (dynamic) : FPMA IEEE-assigned Ethertype for FCoE traffic is 0x8906 FIP (FCoE Initialization Protocol) It is the control plane protocol It is used to discover the FC entities connected to an Ethernet cloud It is also used to login to and logout from the FC fabric Uses unique BIA on CNA for MAC IEEE-assigned Ethertype for FCoE traffic is 0x8914

19 Ethernet Header FCoE Header FC Header CRC EOF FCS FCoE Protocol Fundamentals Fibre Channel over Ethernet (FCoE) Fibre Channel over Ethernet provides a high capacity and lower cost transport option for block based storage Two protocols defined in the standard FCoE Data Plane Protocol FIP Control Plane Protocol FCoE is a standard - June 3rd 2009, the FC-BB-5 working group of T11 completed its work and unanimously approved a final standard for FCoE FCoE is Fibre Channel Bit 0 FCoE Frame Format Bit 31 Byte 0 EOF Destination MAC Address Source MAC Address (IEEE 802.1Q Tag) ET = FCoE Ver Reserved Reserved Reserved Reserved SOF Encapsulated FC Frame (with CRC) Reserved FCS Byte 2197 FC Payload

20 FCoE Protocol Fundamentals FCoE Initialization Protocol (FIP) Neighbor Discovery and Configuration (VN VF and VE to VE) Step 1: FCoE VLAN Discovery FIP sends out a multicast to ALL_FCF_MAC address looking for the FCoE VLAN FIP frames use the native VLAN Step 2: FCF Discovery FIP sends out a multicast to the ALL_FCF_MAC address on the FCoE VLAN to find the FCFs answering for that FCoE VLAN FCF s responds back with their MAC address Step 3: Fabric Login FIP sends a FLOGI request to the FCF_MAC found in Step 2 Establishes a virtual link between host and FCF FCF assigns the host a Enode MAC address to be used for FCoE forwarding ** FIP does not carry any Fibre Channel data frames Enode Initiator VLAN Discovery FCF Discovery FLOGI/FDIS C FC Command FCoE Switch FCF VLAN Discovery FCF Discovery FLOGI/FDIS C Accept FC Command Responses FCoE Initializatio n Protocol (FIP) FCoE Protocol

21 What happens after FLOGI FCoE Protocol carries FC model Name server registration/query PLOGI from initiator to target SCSI Commands and Data transmission Initiator FC PLOGI ACCEPT PRLI ACCEPT Target Report LUNs SCSI Command SCSI Command (1) (Read) SCSI Data (1) SCSI Status (1) SCSI_FCP_DATA STATUS (=good) SCSI Initiator SCSI Command (1) (Write) SCSI Data (1) SCSI Status (1) SCSI Target SCSI Inquiry Command (LUN n) SCSI_FCP_DATA STATUS (=good)

22 Recap of Fibre Channel Concepts Initiator SCSI Host System WWNs FCIDs Fibre Channel HBA Name server Fabric controller FSPF Zone Server VSANs Fibre Channel Fabric target Disk Array

23 Fibre Channel Addressing 64-bit WWNs are used as burnt-in unique addresses assigned to fabric switches, ports, and nodes by the manufacturer These addresses are registered in the fabric and mapped to an 24-bit FC_ID 8 bits 8 bits 8 bits Switch Domain Area Device FCoE Addressing Scheme: A fabric-provided Mac address (FPMA) is assigned to each Enode Enode MAC composed of a FC-MAP and FCID FCoE forwarding decisions will still be made based on FSPF and the FCID within the Enode MAC FC Switches assign FC_ID addresses to N_Ports; FSPF Forwarding decisions are made on domain ID FC-MAC Address FC-MAP (0E-FC-xx) FC-MAP (0E-FC-xx) FC-ID FC-ID

24 Basic Fibre Channel Port Types FC Switch E_Port E_Port F_Port NP_Port FC NPV Switch F_Port N_Port Node FC Switch F_Port N_Port Node FCF Switch VE_Port VE_Port VF_Port VNP_Port FCoE_NPV Switch VF_Port VN_Port End Node FCoE Switch : FCF VF_Port VN_Port End Node

25 Fibre Channel VSAN A Virtual SAN (VSAN) Provides a Method to Allocate Ports Within a Physical Fabric to Create Virtual Fabrics Analogous to VLANs in Ethernet Virtual fabrics created from larger cost-effective redundant physical fabric Reduces wasted ports of island approach Fabric events are isolated per VSAN maintains isolation for HA (i.e., RSCNs) Cisco MDS 9000 Family with VSAN Service Physical SAN Islands Are Virtualized onto Common SAN Infrastructure

26 Fibre Channel Zoning pwwn 50:06:01:61:3c:e0:1a:f6 Target Zones are the basic form of data path security zone members can only see and talk to other members of the zone devices can be members of more than one zone Default zoning is deny Zones belong to a zoneset Zoneset must be active to enforce zoning Only one active zoneset per fabric or per VSAN FC/FCoE Fabric FCF with Domain ID 10 SAN ZoneA ZoneB Disk2 Host1 Disk4 Disk3 Disk1 Host2 ZoneC Initiator pwwn 10:00:00:00:c9:76:fd:31 zoneset name ZONESET_V1 vsan 1 zone name Z_FC1_b1_FC1_e1_V1 vsan 1 fcid 0x [pwwn 10:00:00:00:c9:76:fd:31] [initiator] fcid 0x [pwwn 50:06:01:61:3c:e0:1a:f6] [target]

27 Fibre Channel Flow Control B2B Credits used to ensure that FC transport is lossless Number of credits negotiated between ports when link is brought up Each side informs the other side of the number of buffer credits it has F ports - In the Fabric Login(FLOGI) E ports In the Exchange Link Parameters(ELP) # Credits decremented with each packet placed on the wire Independent of packet size If # credits == 0, no more packet transmission # of credits incremented with each transfer ready received B2B Credits need to be taken into consideration as distance and/or bandwidth increases 15 Fibre Channel Switch R_RDY Packet Host

28

29 DCB and QoS

30 Standards for FCoE FCoE is fully defined in FC-BB-5 standard FCoE works alongside additional technologies to make I/O Consolidation a reality T11 FCoE IEEE DCB FC on FC on Other other Network network Media media PFC ETS DCBX Lossless Ethernet Priority Grouping Configuration Verification FC-BB Qbb 802.1Qaz 802.1Qaz Standard Status Technically stable October, 2008 Completed in June 2009 Published in May, 2010 Sponsor Ballot July 2010 Published Fall 2011 Sponsor Ballot October 2010 Published Fall 2011 Sponsor Ballot October 2010 Published Fall 2011

31 Data Center Bridging (DCB) Can Ethernet be lossless? DCB features extend the Ethernet capabilities to the Data Center by ensuring delivery over lossless fabrics and I/O convergence. The three main features of the DCB architecture are Priority Flow Control (PFC) Enhanced Transmission Selection (ETS) Data Center Bridging Exchange(DCBX) Priority Pause mechanism that can be controlled independently for each class of network service on a shared multiprotocol link Defines a common management framework to assign bandwidth for each class of network service on shared links Discovery and exchange of Capabilities to ensure consistent configuration between network neighbors

32 PFC: Priority Flow Control IEEE 802.1Qbb VLAN Tag enables 8 priorities for Ethernet traffic PFC enables Flow Control on a Per-Priority basis using PAUSE frames (802.1p). Ethernet Wire FCoE Therefore, we have the ability to have lossless and lossy priorities at the same time on the same wire Allows FCoE to operate over a lossless priority independent of other priorities

33 ETS: Enhanced Transmission Selection IEEE 802.1Qaz Allows you to create priority groups Can guarantee bandwidth Can assign bandwidth percentages to groups Not all priorities need to be used or in groups 80% 20% 80% FCoE 20% Ethernet Wire

34 DCBX: Data Center Bridging exchange IEEE 802.1Qaz Hello? Hello. Look s Like We All Speak the Same Language. Ethernet Wire Hello? Hello. Hello? Hello. Allows network devices to advertise their identities and capabilities over the network Enables hosts to pick up proper configuration from the network Enables switches to verify proper configuration Provides support for: PFC ETS Applications (e.g., FCoE)

35 DCBX Basics DCBX operates for priority-based flow control (PFC), Layer 2 and Layer 4 applications such as FCoE and iscsi, and RoCE. Via DCBX exchange, the switch can: o o o Discover the DCB capabilities of peers. Detect DCB feature misconfiguration or mismatches between peers. Configure DCB features on peers if the peer is configured as willing to learn the configuration from other peers. To enable DCBX negotiation for applications, the applications must be configured and be mapped to IEEE 802.1p code points in an application map, and apply the application map to interfaces.

36 DCBX Versions The CIN version (Pre-CEE/Pre-FIP version 1.0): The CEE version (DCBX version 1.01): capability-exchangediscovery-protocol-1108-v1.01.pdf The IEEE version: * Cisco supports CEE DCBX

37 Nexus 5K DCB Features DCB Features Nexus 5K Support PFC/802.1Qbb ETS/802.1Qaz Quantized Congestion Notification/802.1Qau DCBX (CEE Version) * * Nexus 5K does NOT support IEEE DCBX

38 CEE DCBX vs IEEE DCBX CEE DCBX * IEEE DCBX Organizationally Unique Identifier (OUI) 0x001b21 (Organizationally Specific TLV) 0x0080c2 (IEEE COMMITTEE) Exchnage Protocol LLDP LLDP TLV Format One TLV that includes all DCBX attributes; The Willing bit determines whether an interface can change its configuration to match connected peer. Use a unique TLV for each DCB attribute; Configuration and recommendation is sent in different TLVs TLV Subtypes One TLV with subtype of 0x02 with Feature types: 1: Control Info 2: Priority Group / ETS 3: PFC 4: Application Subtypes: 0x09 ETS Configuration; 0x0A ETS Recommendation; 0x0B PFC Configuration; 0x0C Application Priority; Peer Synchronization Symmetric Asymmetric or Symmetric * Pre-CEE DCBX versions has a subtype of 1 and an OUI of 0x001b21.

39 Nexus 5K CEE DCBX Packet Captured in LLDP Packet DCBX TLV: All attributes in one TLV.

40 CEE DCBX TLV Structure DCBX TLV Values: fe37001b a a b fe37001b2102 TLV Control Information 020a PFC Sub-TLV =0x03 080a b DCBX Protocol Sub-TLV=0x01 FCoE APP Sub-TLV=0x04 ETS/PriGroup Sub-TLV=0x02

41 CEE DCBX TLV Control Information OUI: 0x001b21 TLV Type: 127 Value: 0xfe37001b2102 DCBX Version: 0x02 (CEE 1.01)

42 CEE DCBX Protocol Sub-TLV=0x01 Value: 0x020a = Sub-type 1 (control protocol sub-tlv) 0a = Length = Operational version 0 00 = Maximum version = Sequence number = Acknowledgement number 0

43 CEE DCBX Feature PFC Sub-TLV=0x03 Value: 0x = Sub-type 3 (PFC sub-tlv) 06 = Length 6 00 = Operational version 0 00 = Maximum version 0 80 = Enabled, Not Willing, No Error * 00 = Sub-type 0 08 = PFC is enabled for CoS 3 08 = PFC is supported for 8 traffic classes * The Error bit will only be set in case of negotiation failures.

44 CEE DCBX Feature Application Sub-TLV=0x04 Value: 0x080a b = Sub-type 4 (Application protocol sub-tlv) 0a = Length = Operational version 0 00 = Maximum version 0 80 = Enabled, Not Willing, No Error * 00 = Sub-type = Application Proto ID FCoE 00 1b 21 = OUI, and indicator of APP defined by EtherType (Selector Field SF) 08 = Upper Protocol Map maps to CoS 3 * The Error bit will only be set in case of negotiation failures.

45 CEE DCBX Feature PriorityGroup/ETS Sub-TLV=0x02 Value: 0x = Sub-type 2 (Priority group sub-tlv) 11 = Length = Operational version 0 00 = Maximum version 0 80 = Enabled, Not Willing, No Error * 00 = Sub-type = CoS 0-2 is mapped to priority group 0 1 = CoS 3 is mapped to priority group = CoS 4-7 is mapped to priority group 0 32 = Percentage for priority group 0 is 50% 32 = Percentage for priority group 1 is 50% = Percentage for priority group 2-7 is 0% 02 = Number of priority groups supported is 2 (GRP 0 & 1) * The Error bit will only be set in case of negotiation failures.

46 Cisco Nexus 5K QoS Processing Flow Trust CoS/DSCP L2/L3/L4 info with ACL If Buffer Usage Crosses Threshold: Tail drop for drop class Assert pause signal to MAC Ingress UPC VoQs for Unicast (8 per ingress port) Central Scheduler MAC Traffic Classification Ingress Cos/DSCP Marking Ingress Policing MTU Checking Per-class Buffer Usage Monitoring Multicast Queues PAUSE ON/OFF Signal Truncate or Drop Packets if MTU is Violated Egress Queues Unicast Crossbar Fabric MAC ECN Marking Egress Policing Egress Scheduling Egress UPC Strict Priority + DWRR Scheduling Multicast

47 Nexus QoS Priority Flow Control and No-Drop Queues Default queuing buffer for PFC generation: Configs for nodrop class Buffer size Pause Threshold (XOFF) Resume Threshold (XON) N bytes bytes bytes N bytes bytes bytes Tuning of the lossless queues to support a variety of use cases Extended switch to switch no drop traffic lanes Support for FCoE long distance Increased number of no drop services lanes (4) for RDMA and other multi-queue HPC and compute applications Gen 2 UPC Unified Crossbar Fabric Gen 2 UPC Support tuning no drop distance for switch to switch ISLs between FCoE Forwarders 5548-FCoE(config)# policy-map type network-qos 3km-FCoE 5548-FCoE(config-pmap-nq)# class type network-qos 3km-FCoE 5548-FCoE(config-pmap-nq-c)# pause no-drop buffer-size pause-threshold resume-threshold 83520

48 Nexus QoS - Enhanced Transmission Selection Bandwidth Management When configuring FCoE by default, each class is given 50% of the available bandwidth Can be changed through QoS settings when higher demands for certain traffic exist (i.e. HPC traffic, more Ethernet NICs) CNAs 5Gig FC vhba 5Gig Ethernet vnic N5k-1# show queuing interface Ethernet 1/18 Ethernet1/18 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 50 1 WRR 50

49 Nexus QoS QoS Policy Types There are three QoS policy types used to define system behavior (qos, queuing, network-qos) There are three policy attachment points to apply these policies to Ingress interface System as a whole (defines global behavior) Egress interface Policy Type Function Attach Point qos queuing network-qos Define traffic classification rules Strict Priority queue Deficit Weight Round Robin System class characteristics (drop or nodrop, MTU), Buffer size, Marking system qos ingress Interface system qos egress Interface ingress Interface system qos

50 System Default QoS Setting for FCoE tme-5548up-1# show running-config ipqos!command: show running-config ipqos!time: Thu May 28 18:11: version 7.0(1)N1(1) system qos service-policy type queuing input fcoe-default-in-policy service-policy type queuing output fcoe-default-out-policy service-policy type qos input fcoe-default-in-policy service-policy type network-qos fcoe-default-nq-policy

51 System Default QoS Setting for FCoE tme-5548up-1# show policy-map system type network-qos tme-5548up-1# show policy-map system type queuing input Type network-qos policy-maps =============================== policy-map type network-qos fcoe-default-nq-policy class type network-qos class-fcoe match qos-group 1 pause no-drop mtu 2158 class type network-qos class-default match qos-group 0 mtu 1500 multicast-optimize Service-policy (queuing) input: fcoe-default-in-policy policy statistics status: disabled Class-map (queuing): classfcoe (match-any) Match: qos-group 1 bandwidth percent 50 Class-map (queuing): classdefault (match-any) Match: qos-group 0 bandwidth percent 50

52 System Default QoS Setting for FCoE tme-5548up-1# show policy-map system type qos tme-5548up-1# show policy-map system type queuing output fcoe-default-in- Service-policy (qos) input: policy policy statistics status: disabled Service-policy (queuing) output: fcoe-default-out-policy policy statistics status: disabled Class-map (qos): Match: cos 3 set qos-group 1 Class-map (qos): Match: any set qos-group 0 class-fcoe (match-any) class-default (match-any) class- Class-map (queuing): fcoe (match-any) Match: qos-group 1 bandwidth percent 50 class- Class-map (queuing): default (match-any) Match: qos-group 0 bandwidth percent 50

53 Does PFC really work? Customer is testing Server INTB3 BNT EN4093 Server INTB4 Server Server Server INTB5 INTB6 INTB7 Shared Buffer ext9 e1/10 10gbps Nexus 5500 Ingress VoQ Queuing Buffer fc4/13 4gbps MDS BB Credits 10gbps IBM Blade H Chassis with 14 servers, 4x10G uplinks to N5548UP, N5548UP 2x8G FC uplinks to SAN A and SAN B

54 Agenda Introduction to FCoE Technology FCoE SAN Design for Small and Mid-size Enterprise Basic FCoE Configuration and Troubleshooting Conclusion

55 Requirements of Small and Mid-size SAN s IO Consolidation - Need for converged networks with use of same links for SAN and LAN requirements? Performance and flexibility - Use the existing Ethernet network infrastructure for high performance SAN? Cost - Price might be a factor, cost efficient for dedicated SAN Growth with scale - Scalability to be taken care of without compromise in performance and much additional cost Data security High availability and resiliency

56 Think About FCoE SAN Convergence Multi-protocol support to provide the flexibility for IO consolidation Scalability Meet the increasing demands of network and data without compromising performance Bandwidth Resource sharing amongst multiple application workloads Fabric Management Unified Management tool to maintain, troubleshoot and manage the fabric Latency Low latency high throughput networks for sensitive workloads High Availability Hardware and software redundancy needs at the Server, Fabric and Storage levels

57 NX-OS Common Operating System DCNM Common Management Tool FCoE on Cisco Nexus Switches High Convergence Multi-protocol support Flexible for IO consolidation Low Latency Scalable Networks Predictable performance Ease of segregation of Storage traffic QoS, DCB, FCoE TLV Traffic Prioritization Highly secure Networks RBAC,AAA,ACL TrustSec High Speed Interconnects 10/40/100Gb Ethernet,FC,FCoE High Availability and Data Resiliency VDC,Hardware Redundancy vpc, FabricPath

58 FCoE Design Considerations

59 Traditional Data Center Design Ethernet LAN and Fibre Channel SAN Ethernet FC Physical and Logical separation of LAN and SAN traffic FC Additional Physical and Logical separation of SAN fabrics L3 L2 Fabric A Fabric B NIC HBA Isolation Convergence

60 Data Center Design with E-SAN Ethernet LAN and Ethernet SAN Ethernet FC Same topologies as existing networks, but using Nexus Unified Fabric Ethernet switches for SANs FCoE Physical and Logical separation of LAN and SAN traffic Additional Physical and Logical separation of SAN fabrics L3 L2 Fabric A Fabric B Ethernet SAN Fabric carries FC/FCoE & IP based storage (iscsi, NAS, ) Common components: Ethernet Capacity and Cost Isolation NIC or CNA CNA Convergence

61 Converged Network Unified Access Layer Network Convergence occurs at the access layer Consolidate I/O on 10G links Drastically reduced CapEx and OpEx Multiprotocol connectivity eased purchasing decisions for server refreshes Prepared Data Centers for VM mobility requirements Any VM could connect to FC storage if necessary, not just the ones with HBAs pre-installed Isolation L3 L2 LAN Fabric A CNA FC Ethernet FC Converged FCoE link Dedicated FCoE link Convergence SAN Fabric B

62 Converged Network with FCoE NPV expanding existing SAN Infrastructure FCoE CNAs can be in active-standby or active-active to have redundancy along with bandwidth utilization Fabric A Fabric B Ethernet FC Converged FCoE link Dedicated FCoE link Nexus 5000 switches in Access layer makes use of both FCoE NPV and FC NPV to leverage the advantages of NPV (prevents domain ID sprawl, better manageability) FCoE NPV and FC NPV connectivity helps easy migration of servers from legacy FC network to FCoE network. Maintains the SAN-A, SAN-B isolation for SAN while providing the vpc connectivity with existing Ethernet network. SAN can utilize higher performance, higher density, lower cost Ethernet switches for the aggregation/core FCoE L3 L2 FCoE NPV VF VF VNP V CNAN FCoE NPV Core VF NP VF FC VF F FC NPV Core VNP NP FCoE NPV LAN/SAN FC F

63 Converged Network Dual Fabrics with Dedicated Links Maintaining Dual SAN fabrics with Overlay Ethernet FC Converged FCoE link Dedicated FCoE link LAN and SAN traffic share physical switches Fabric A Fabric B LAN/SAN LAN and SAN traffic use dedicated links between switches All Access and Aggregation switches are FCoE FCF switches L3 L2 FCF Multi-hop convergence - Dedicated links between switches are VE_Ports Storage VDC for additional management / operation separation VE FCF FCF CNA FCoE FC Isolation Convergence

64 Converged Network Dual Fabrics with Dedicated Links Overlay of SAN A / SAN B with Dedicated Links Ethernet FC Converged FCoE link Dedicated FCoE link LAN Fabric A Fabric B SAN L3 L2 FCF VE FCF FCF CNA CNA FCoE FC

65 Converged Network Unified Fabric LAN and SAN traffic share physical switches Fabric A Fabric B LAN and SAN traffic share consolidated links between switches L3 L2 Supports Spine/Leaf topology 10,20 FCF 20, FCF 10,20 20, CNA1 CNA2 Array1 FCoE Array2 Isolation Convergence

66 Deploying FCoE SAN using Cisco UCS, Nexus and MDS

67 Industry s Broadest FCoE Portfolio 40G FCoE has 50% greater Data Rate than 32G FC Cisco NEXUS Cisco MDS Cisco UCS Nexus 5696Q Nexus 5648Q Nexus 5624Q Nexus 7718 Nexus 7710 Nexus 7706 Nexus 7018 Nexus 7010 Nexus 7006 Nexus 56128P Nexus 5672UP Nexus 2348UPQ Nexus 2348TQ Nexus 2332TQ MDS 9710 MDS 9706 MDS 9250i Nexus 6296UP Nexus 6248UP Density Nexus 7700 Nexus 7000 Nexus 5600 Nexus 2300 MDS 9000 UCS FI 10G FCoE (breakout) G FCoE 384 (June 15) 192 (June 15) 96 6 Future Future Converge at Host Edge with 10G FCoE, Use 40G FCoE for ISLs, Deploy 16G FC at Storage Core

68 FCoE Deployment Nexus 5K as Unified TOR Ethernet FC FCoE Converged LAN Unified Access Nexus 5K/7K L3 L2 Nexus 5K NIC Fabric A HBA FC MDS SAN Fabric B Ethernet and FC I/O is carried over the same physical link from the server Servers use Converged Network Adapters to combine traffic (Eth & FC) Significantly reduces the amount of cabling required Reduces the number of port needed, as Eth and FC traffic uses the same port and cable FCoE Host The access layer switch acts a protocol splitter

69 FCoE Deployment Choices: Nexus 5K breakout to SAN Ethernet FC FCoE Converged FC FC FC FC SAN A SAN B SAN A SAN B SAN A SAN B SAN A SAN B MD S E E MD S F F MD S VE VE MD S VF VF Nexus 5K E E Nexus 5K N P N P Nexus 5K VE VE Nexus 5K VNP VN P FCoE Host FCoE Host FCoE Host FCoE Host FC Switch Mode FC NPV Mode FCoE Switch Mode FCoE NPV Mode

70 FCoE Deployment FEX as Unified TOR Nexus 5K LAN Ethernet FC FCoE Converged FC/FCoE SAN The fabric extenders (FEX) can be used at the ToR FEX cannot be FCF ToR FEX connect to the end of row switches which manages the FEX FEX work like external line cards to the end or row switches Nexus 2K The EoR switches splits the Ethernet and FC traffic FCoE Host B22 HP FEX 1/10G FEX for HP Blade servers B22F FEX 1/10G FEX for FTS Blade servers B22 IBM FEX 1/10G FEX for IBM Blade servers B22 Dell FEX 1/10G FEX for Dell Blade servers

71 FCoE Deployment Nexus 5000/ Nexus 2000 Connectivity Models FCoE over FEX: Single-home or Dual-home Ethernet FCoE Link Fibre Channel Converged Link LAN Fabric SAN A SAN B Native FC OR Dedicated FCoE LAN Fabric SAN A SAN B Native FC OR Dedicated FCoE N5K FCF FCF N5K N5K FCF FCF N5K Nexus 2000 FEX Nexus 2000 FEX Consider FCoE Traffic Path Nexus 10GE FEX FCoE with Single-Homed FEX FCoE with Dual-Homed FEX - Supported on Nexus 5K from 5.1(3)N1(1)

72 FCoE Deployment UCS FI Connecting to Nexus 5K TOR Nexus 5K Nexus 2232 LAN FC/FCoE SAN 61xx/62xx UCS FI Ethernet FC FCoE Converged FI has the default End Host Mode; It works as NPV Switch in SAN network; NPIV feature needs to be enabled on N5K; FI can have separate FC uplinks to N5K; Or, have FCoE uplinks to N5K. UCS Servers

73 FCoE Deployment UCS FI as Unified TOR Ethernet FC FCoE Converged Fabric A FC Fabric B L3 L2 Nexus 5K/7K MDS SAN 61xx/62xx UCS FI Nexus 2232 FI connects to MDS SAN via FC links; Or, via FCoE links. UCS Servers

74 FCoE Deployment UCS FI with Direct Attach Storage Ethernet FC FCoE Converged FC Nexus 2232 L3 L2 SAN A Nexus 5K/7K 61xx/62xx UCS FI SAN B FCoE LAN/SAN IO by default converges inside UCS servers by using converged adapters; UCS FI is in FC Switch Mode; UCS FI supports Unified ports which can connect direct attached FC or FCoE storage devices; Use different FCoE VLANs/VSANs for SAN A and SAN B separation UCS Servers

75 The Power of Converged Infrastructure FLEXPOD VBLOCK VSPEX VersaStack HDS UCP- Select SmartStack UCS Director Cisco UCS Cisco Nexus Cisco MDS

76 FCoE Deployment Nexus 5K as Unified Access with Direct Attach Storage Ethernet FC FCoE Converged FC L3 L2 SAN A FCoE Host Nexus 5K/7K Nexus 5K SAN B FCoE LAN/SAN Convergence happens at the access layer; Nexus 5K as access switch is also FCoE Forwarder; Nexus 5K supports Unified ports which can connect direct attached FC or FCoE storage devices; Use different FCoE VLANs/VSANs for SAN A and SAN B separation Shared Wires connecting to HOSTS must be configured as trunk ports

77 FCoE Deployment Multi-hop FCoE extending FCoE through Aggregation FC Ethernet FC FCoE Converged LAN Unified Access Nexus 5K/7K L3 L2 Nexus 5K Fabric A MDS SAN Fabric B VE links between access and aggregation switches; Dedicated links to Storage VDC in N7K; Can be converged wires between N5K and N5K. FCoE Host

78 FCoE Deployment End-to-end Multi-hop FCoE with Nexus Platforms FCoE Ethernet FC FCoE Converged LAN Unified Access Nexus 5K/7K L3 L2 Nexus 5K FCoE Host Fabric A Nexus SAN Fabric B End-to-end FCoE ISLs between Nexus switches; N7K as storage core and N7K or N5K as storage edge; FCoE SAN Edge/Core or Edge/Core/Edge design.

79 FCoE Deployment Ethernet Fabric connecting to SAN Fabric A Fabric B FC/FCoE SAN L3 L2 Nexus 5K/7K Spine FC/FCoE SAN FabricPath Network or Standalone Fabric FCoE Uplink Nexus 5K Leaf Native FC CNA1 CNA2

80 FCoE Deployment Dynamic FCoE enables Full Convergence in the Unified Fabric All Leaf switches are FCoE FCF switches, spine is FCoE transparent Multi-hop convergence one hop away from any leaf to any leaf Nexus 5/k/7K VE instantiation is created dynamically The adjacency between FCF leafs are established dynamically Complete, load-balance ISLs dynamically created between FCF leafs No Storage VDC, all Ethernet VDCs Nexus 5K SAN A SAN B Dynamic FCoE enables full convergence in Fabricpath Ethernet Fabric Dynamic FCoE using FabricPath is simplifying the network infrastructure and achieves multiprotocol convergence in the cloud Data Center. iscsi/ NFS Converged FC

81 Logical Separation of SAN A/B in the Dynamic FCoE Unified Fabric Server/Storage nodes reside on the leafs Storage sees an edge-core topology equivalence SAN A/B separation occurs at the most vulnerable part of the network the server to the access layer switch (i.e., leaf) What Fibre Channel sees SAN A SAN B iscsi/ NFS Converged FC

82 Agenda Introduction to FCoE Technology FCoE SAN Design for Small and Mid-size Enterprise Basic FCoE Configuration and Troubleshooting Conclusion

83 FC/FCoE Configuration on Nexus 5K What basic steps are needed 1) Install License Storage Service License, or FCoE-NPV License 2) Enable features LLDP, LACP, FEX (optional), FCoE or FCoE-NPV, NPIV (optional), NPV (optional) 3) Enable system QoS and policy for FCoE * This step is only required on Nexus 55xx platforms prior to release 5.1(3)N1(1). 4) Bring up FEX connections to parent switch (optional) 5) Create and configure native FC interfaces if needed 6) Create VSAN and assign native FC interfaces to VSAN 7) Create FCoE VLAN and map to VSAN * Best Practice: Reserve a Range with same number of VSANs expected to be used 8) Configure host Ethernet interfaces for FCoE traffic 9) Create VFC interfaces and bind them to the host ports * Best Practice: Formulate a method to assign vfc numbers for the bound Ethernet interface 10) Configure VFC interfaces and assign the interfaces into VSAN 11) Zoning configuration

84 Nexus 5K Configuration Example 1 - FC/FCoE Ethernet FC FCoE Converged FC FCoE E1/25 FC1/32 Nexus 5548UP-1 E1/3 E1/7 E1/7 E1/8 E1/8 E1/4 E1/4 FC1/32 E1/25 Nexus 5548UP-2 E1/3 SAN A Nexus 2232 FEX 100 1/25 1/25 SAN B Nexus 2232 FEX 101 Nexus 5548 code version: 7.0(1)N1(1) FCoE Host

85 Configuration Example 1 on Nexus 5K 1. Install FCoE License tme-5548up-1# install license bootflash:n lic Tme-5548up-1# show license usage Feature Ins Lic Status Expiry Date Comments Count FC_FEATURES_PKG Yes - In use Never Enable FEX feature 3. Enable VPC feature 4. Enable LACP feature 5. Enable FCoE feature tme-5548up-1(config)# feature fex tme-5548up-1(config)# feature vpc tme-5548up-1(config)# feature lacp tme-5548up-1(config)# feature fcoe tme-5548up-1(config)# show feature incl ena fcoe 1 enabled fex 1 enabled lacp 1 enabled lldp 1 enabled by default lldp feature is enabled vpc 1 enabled 6. After FCoE feature is enabled, the system enables the default FCoE QoS Policy tme-5548up-1(config)# show running-config ipqos system qos service-policy type queuing input fcoe-default-in-policy service-policy type queuing output fcoe-default-out-policy service-policy type qos input fcoe-default-in-policy service-policy type network-qos fcoe-default-nq-policy

86 Configuration Example 1 on Nexus 5K 7. Configure VPC 8. Bring up FEX tme-5548up-1(config)# vpc domain 1 tme-5548up-1(config-vpc-domain)# peer-keepalive destination tme-5548up-1(config)# interface Ethernet1/7 tme-5548up-1(config-if)# channel-group 1 mode active tme-5548up-1(config-if)# no shutdown tme-5548up-1(config)# interface Ethernet1/8 tme-5548up-1(config-if)# channel-group 1 mode active tme-5548up-1(config-if)# no shutdown tme-5548up-1(config)# interface port-channel1 tme-5548up-1(config-if)# switchport mode trunk tme-5548up-1(config-if)# switchport trunk allowed vlan 1,1001 tme-5548up-1(config-if)# vpc peer-link tme-5548up-1(config)# fex 100 tme-5548up-1(config-fex)# pinning max-links 1 tme-5548up-1(config-fex)# description "FEX0100" tme-5548up-1(config-fex)# fcoe tme-5548up-1(config)# fex 101 tme-5548up-1(config-fex)# pinning max-links 1 tme-5548up-1(config-fex)# description "FEX0101 tme-5548up-1(config)# interface Ethernet1/3 tme-5548up-1(config-if)# channel-group 100 tme-5548up-1(config-if)# no shutdown tme-5548up-1(config)# interface Ethernet1/4 tme-5548up-1(config-if)# channel-group 101 tme-5548up-1(config-if)# no shutdown tme-5548up-1(config)# interface port-channel100 tme-5548up-1(config-if)# switchport mode fex-fabric tme-5548up-1(config-if)# vpc 100 tme-5548up-1(config-if)# fex associate 100 tme-5548up-1(config)# interface port-channel101 tme-5548up-1(config-if)# switchport mode fex-fabric tme-5548up-1(config-if)# vpc 101 tme-5548up-1(config-if)# fex associate 101

87 Configuration Example 1 on Nexus 5K 9. Create native FC interfaces 12. Create FCoE VLAN and map to VSAN tme-5548up-1(config)# slot 1 tme-5548up-1(config-slot)# port type fc Port type is changed. Please reload the switch 10. Configure native FC interface tme-5548up-1(config)# interface fc1/32 tme-5548up-1(config-if)# switchport mode F tme-5548up-1(config-if)# no shutdown 11. Create VSAN and assign native FC interface to VSAN tme-5548up-1(config)# vsan database tme-5548up-1(config-vsan-db)# vsan 100 tme-5548up-1(config-vsan-db)# vsan 100 interface fc1/32 tme-5548up-1(config)# vlan 100 tme-5548up-1(config-vlan)# fcoe vsan Configure FCoE host interface tme-5548up-1(config)# interface Ethernet100/1/25 tme-5548up-1(config-if)# switchport mode trunk tme-5548up-1(config-if)# switchport trunk allowed vlan 100 tme-5548up-1(config-if)# spanning-tree port type edge trunk tme-5548up-1(config-if)# channel-group 102 mode active tme-5548up-1(config)# interface vfc125 tme-5548up-1(config-if)# bind interface Ethernet100/1/25 tme-5548up-1(config-if)# switchport trunk allowed vsan 100 tme-5548up-1(config-if)# no shutdown tme-5548up-1(config)# vsan database tme-5548up-1(config-vsan-db)# vsan 100 interface vfc125

88 Configuration Example 1 on Nexus 5K 14. Configure FCoE target interface tme-5548up-1(config)# interface Ethernet1/25 tme-5548up-1(config-if)# switchport mode trunk tme-5548up-1(config-if)# switchport trunk allowed vlan 100 tme-5548up-1(config-if)# spanning-tree port type edge trunk tme-5548up-1(config)# interface vfc25 tme-5548up-1(config-if)# bind interface Ethernet1/25 tme-5548up-1(config-if)# switchport trunk allowed vsan 100 tme-5548up-1(config-if)# no shutdown tme-5548up-1(config)# vsan database tme-5548up-1(config-vsan-db)# vsan 100 interface vfc Configure Zoning tme-5548up-1(config)# zone name demo-fcoe-host-1 vsan 100 tme-5548up-1(config-zone)# member pwwn 20:00:6c:20:56:a4:75:9c tme-5548up-1(config-zone)# member pwwn 50:06:01:64:3e:a0:33:27 tme-5548up-1(config)# zone name demo-fcoe-host-2 vsan 100 tme-5548up-1(config-zone)# member pwwn 20:00:6c:20:56:a4:75:9c tme-5548up-1(config-zone)# member pwwn 50:06:01:64:3e:a4:33:27 tme-5548up-1(config)# zoneset name demo-fcoe vsan 100 tme-5548up-1(config-zoneset )# member demo-fcoe-host-1 tme-5548up-1(config-zoneset )# member demo-fcoe-host Check PWWNs of the end nodes tme-5548up-1(config)# show flogi database vsan INTERFACE VSAN FCID PORT NAME NODE NAME fc1/ x3500ef 50:06:01:64:3e:a0:33:27 50:06:01:60:be:a0:33:27 vfc x :06:01:68:3e:a4:33:27 50:06:01:60:be:a0:33:27 vfc x :00:6c:20:56:a4:75:9c 10:00:6c:20:56:a4:75:9c Total number of flogi = Activate Zoneset tme-5548up-1(config)# zoneset activate demo-fcoe vsan 100 tme-5548up-1# show zoneset active vsan 100 zoneset name demo-fcoe vsan 100 zone name demo-fcoe-host-1 vsan 100 * fcid 0x3500ef [pwwn 50:06:01:64:3e:a0:33:27] * fcid 0x [pwwn 20:00:6c:20:56:a4:75:9c ] zone name demo-fcoe-host-2 vsan 100 * fcid 0x [pwwn 20:00:6c:20:56:a4:75:9c] * fcid 0x [pwwn 50:06:01:64:3e:a4:33:27]

89 Nexus 5K Configuration Example 2 - FCoE-NPV Ethernet FC FCoE Converged PO1001 VF VN P MD VF S97 10 FCoE-NPV VNP Nexus 5696Q SAN A SAN B Nexus 2K FCoE Host

90 Configuration Example 2 on Nexus 5696Q 1. FCOE_NPV License is required to enable FCoE-NPV feature. 2. Enable FCoE-NPV feature. FCoE-NPV feature does not require to reboot the switch or module. Enabling FCoE-NPV feature enables QoS default settings for FCoE. 3. Enable LACP feature. 4. Create VSAN/VLAN Mapping. 5. Create Ethernet Port Channel. TME-N5696Q(config)# interface e1/1, e1/11, e2/1, e2/11 TME-N5696Q(config-if-range)# switchport mode trunk TME-N5696Q(config-if-range)# switchport trunk allowed vlan 1001 TME-N5696Q(config-if-range)# channel-group 1001 mode active TME-N5696Q(config-if-range)# exit TME-N5696Q(config-)# interface port-channel 1001 TME-N5696Q(config-if)# no shutdown TME-N5696Q# show license usage Feature Ins Lic Status Expiry Date Comments Count FCOE_NPV_PKG Yes - Unused Never - FM_SERVER_PKG No - Unused -. TME-N5696Q(config)# feature fcoe-npv TME-N5696Q(config)# feature lacp TME-N5696Q(config)# vsan database TME-N5696Q(config-vsan-db)# vsan 1001 TME-N5696Q(config-vsan-db)# exit TME-N5696Q(config)# vlan 1001 TME-N5696Q(config-vlan)# fcoe vsan Create VFC for Port Channel Configure VFC to NP mode for NPV uplink. TME-N5696Q(config-if)# interface vfc1001 TME-N5696Q(config-if)# bind interface po1001 TME-N5696Q(config-if)# switchport mode np TME-N5696Q(config-if)# switchport trunk allowed vsan 1001 TME-N5696Q(config-if)# no shutdown

91 Configuration Example 2 on MDS On MDS 9710, the FCoE feature-set is enabled by default without requiring a FCoE capable module to be inserted in the system.. 2. Enable NPIV feature as MDS will behave as NPIV core. 3. Enable LACP feature. 4. Enable F_Port_Channel_Trunk feature. 5. Enable system default FCoE QoS setting. MDS9710-A(config)# system qos MDS9710-A(config-sys-qos)# service-policy type network-qos default-nq-7e-1q1q-policy MDS9710-A(config)# show feature-set Feature Set Name ID State fcoe 1 enabled MDS9710-A(config)# feature npiv MDS9710-A(config)# feature lacp MDS9710-A(config)# feature fport-channel-trunk 6. Create VSAN/VLAN Mapping. 7. Create Ethernet Port Channel. MDS9710-A(config)# interface e7/25, e7/37, e8/25, e8/37 MDS9710-A(config-if-range)# switchport mode trunk MDS9710-A(config-if-range)# switchport trunk allowed vlan 1001 MDS9710-A(config-if-range)# channel-group 1001 mode active MDS9710-A(config-if-range)# exit MDS9710-A(config-if-range)# interface ethernet-port-channel 1001 MDS9710-A(config-if)# no shutdown MDS9710-A(config)# vsan database MDS9710-A(config-vsan-db)# vsan 1001 MDS9710-A(config-vsan-db)# exit MDS9710-A(config)# vlan 1001 MDS9710-A(config-vlan)# fcoe vsan 1001

92 Configuration Example 2 on MDS Create VFC for Port Channel Configure VFC to F mode. MDS9710-A(config)# interface vfc-port-channel 1001 MDS9710-A(config-if)# switchport mode F MDS9710-A(config-if)# switchport trunk allowed vsan 1001 MDS9710-A(config-if)# no shutdown 9. Verify the configuration on NPV Switch: TME-N5696Q(config)# show fcoe database INTERFACE FCID PORT NAME MAC ADDRESS vfc1001 0x :fa:54:7f:ee:ea:55:00 54:7f:ee:ea:55:00 vfc1011 0x :00:10:05:ca:71:78:cc 10:05:ca:71:78:cc vfc1012 0x :00:10:05:ca:71:78:cb 10:05:ca:71:78:cb Total number of flogi count from FCoE devices = 3. TME-N5696Q(config)# show npv flogi-table SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE vfc x :00:10:05:ca:71:78:cc 10:00:10:05:ca:71:78:cc vfc1001 vfc x :00:10:05:ca:71:78:cb 10:00:10:05:ca:71:78:cb vfc1001 TME-N5696Q(config)# show npv status npiv is disabled disruptive load balancing is disabled External Interfaces: ==================== Interface: vfc1001, State: Trunking VSAN: 1001, State: Up, FCID: 0x Number of External Interfaces: 1 Server Interfaces: ================== Interface: vfc1011, VSAN: 1001, State: Up Interface: vfc1012, VSAN: 1001, State: Up Number of Server Interfaces: 2 Total number of flogi = 2.

93 Basic of FCoE Troubleshooting on Nexus 5K Which Symptom best describes your problem? Ethernet Interface down Verify cable and SFP. Verify the VLAN configuration. Check VLAN-VSAN Mapping VFC Interface not Trunking FIP Installation Failure Check Ethernet Interface status and if VFC is bound to it. Ensure VSAN mapping is correct Check fcoe_mgr Events for FIP Transitions VFC VSAN goes down due to missing FKA VFC VSAN Initializing VSAN not up Verify the VFC interface VSAN is correct Verify the VSAN allow list is correct Check for FIP instantiation failure Check QoS/PFC Check DCBX (LLDP) Need to verify Ethernet and VFC status Network is good! Performance problems, timeouts, drops Additional FCoE Troubleshooting Tips show interface e1/1 fcoe Monitor PFC Check Ethernet Interface Check Queuing Check License and feature installation, check FEX FCoE Configuration Check CNA for proper setup Check Ethernet interface for discards, errors FCoE and Native vlan configuration

94 Check DCBX Status: CNA Connection tme-5672up-1(config)# show lldp dcbx interface e1/25 Local DCBXP Control information: Operation version: 00 Max version: 00 Seq no: 1 Ack no: 1 Type/ Subtype Version En/Will/Adv Config 003/ Y/N/Y / Y/N/Y b / Y/N/Y Peer's DCBXP Control information: Operation version: 00 Max version: 00 Seq no: 1 Ack no: 1 Type/ Max/Oper Subtype Version En/Will/Err Config 004/ /000 Y/Y/N b / /000 Y/Y/N ff08 002/ /000 Y/Y/N ffffffff o o o o The configured TLV values on the switch and the CNA can be different Sub-type of 2-4 TLVs are enabled on both sides The switch is not Willing to comprise its TLV value but will advertise all the configured values; The End-node is usually Willing to comprise the TLV values; because of that the Error Bit will not be set.

95 Check DCBX Status: CNA Connection (cont.) tme-5672up-1(config)# show system internal dcbx info interface e1/25 Interface info for if_index: 0x1a018000(Eth1/25) tx_enabled: TRUE rx_enabled: TRUE dcbx_enabled: TRUE DCX Protocol: CEE 6 Features on this intf for Protocol DCX CIN(0) [DCX CIN = 0, not CIN version, ignore the following info] 3 Features on this intf for Protocol DCX CEE(1) [DCX CEE = 1, CEE version, the following features are negotiated] Feature type PFC (3) feature type 3(DCX CEE-PFC)sub_type 0 Feature State Variables: oper_version 0 error 0 local error 0 oper_mode 1 feature_seq_no 0 remote_feature_tlv_present 1 remote_tlv_aged_out 0 remote_tlv_not_present_notification_sent 0 Feature Register Params: max_version 0, enable 1, willing 0 advertise 1 disruptive_error 0 mts_addr_node 0x101 mts_addr_sap 0x179 Desired config cfg length: 2 data bytes:08 08 Operating config cfg length: 2 data bytes:08 08 Peer config cfg length: 0 data bytes: [oper_mode= 1 for PFC/APP/ETS, operations are in active mode; Operating cfg is negotiated to follow switch s cfg. ] Feature type App (4)sub_type FCoE iscsi (0) [iscsi TLV is not enabled for negotiation (value=0)] feature type 4(DCX CEE-App)sub_type 0 Feature State Variables: oper_version 0 error 0 local error 0 oper_mode 1 feature_seq_no 0 remote_feature_tlv_present 1 remote_tlv_aged_out 0 remote_tlv_not_present_notification_sent 0 Feature Register Params: max_version 0, enable 1, willing 0 advertise 1 disruptive_error 0 mts_addr_node 0x101 mts_addr_sap 0x179 Desired config cfg length: 6 data bytes: b Operating config cfg length: 6 data bytes: b Peer config cfg length: 0 data bytes: Feature type PriGrp (2) feature type 2(DCX CEE-PriGrp)sub_type 0 Feature State Variables: oper_version 0 error 0 local error 0 oper_mode 1 feature_seq_no 0 remote_feature_tlv_present 1 remote_tlv_aged_out 0 remote_tlv_not_present_notification_sent 0 Feature Register Params: max_version 0, enable 1, willing 0 advertise 1 disruptive_error 0 mts_addr_node 0x101 mts_addr_sap 0x179 Desired config cfg length: 13 data bytes: Operating config cfg length: 13 data bytes: Peer config cfg length: 0 data bytes:

96 Check FCoE Interface Status: CNA Connection tme-5548up-1# show interface e1/13 fcoe [Check FCoE Interface status] Ethernet1/13 is FCoE UP vfc10 is Up FCID is 0x5a0000 PWWN is 20:00:b0:fa:eb:ac:0f:1a PFC Pause Statistics MAC addr is b0:fa:eb:ac:0f:1a tme-5548up-1# show interface e1/13 priority-flow-control [Monitor PFC] ================================================= Port Mode Oper(VL bmap) RxPPP TxPPP ================================================= Ethernet1/13 Auto On (8) 0 0 tme-5548up-1# show interface vfc10 [Check VFC interface status] vfc10 is trunking Bound interface is Ethernet1/13 Hardware is Ethernet Port WWN is 20:09:00:05:73:de:cd:7f Admin port mode is F, trunk mode is on snmp link state traps are enabled Port mode is TF Port vsan is 10 Trunk vsans (admin allowed and active) (10) Trunk vsans (up) (10) Trunk vsans (isolated) () Trunk vsans (initializing) () 1 minute input rate 128 bits/sec, 16 bytes/sec, 0 frames/sec 1 minute output rate 464 bits/sec, 58 bytes/sec, 0 frames/sec frames input, bytes 0 discards, 0 errors frames output, bytes 0 discards, 0 errors VSAN up Always verify no discards or errors tme-5548up-1# show queuing interface e1/13 [verify queuing on interface] Ethernet1/13 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 50 [ETS bandwidth settings for drop group 1 WRR 50 and non-drop (FCoE) group] RX Queuing qos-group 0 q-size: , HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: Statistics: Pkts received over the port : Ucast pkts sent to the cross-bar : 4 Mcast pkts sent to the cross-bar : Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 0 Pkts discarded on ingress : 0 PFC Pause Threshold Per-priority-pause status : Rx (Inactive), Tx (Inactive) qos-group 1 q-size: 79360, HW MTU: 2158 (2158 configured) drop-type: no-drop, xon: 20480, xoff: Statistics: Pkts received over the port : Ucast pkts sent to the cross-bar : Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : Pkts sent to the port : Pkts discarded on ingress : 0 [Drops upon buffer overflow] Per-priority-pause status : Rx (Inactive), Tx (Inactive) Total Multicast crossbar statistics: Mcast pkts received from the cross-bar : 0

97 Agenda Introduction to FCoE Technology FCoE SAN Design for Small and Mid-size Enterprise Basic FCoE Configuration and Troubleshooting Conclusion

98 Takeaways FCoE helps to achieve the IO consolidation in the DC networking for Small and Mid-size Enterprise customers Different FCoE design can satisfy different convergence needs for your environments at: - Access layer only - End-to-end FCoE with dedicated VE links - Fully converged Unified Fabric Cisco is the leader in SAN switching (FC + FCoE) market You can easily deploy FCoE SAN into the production with Cisco UCS, Nexus and MDS products

99 Q&A

100 Call to Action Visit the Cisco Campus at the World of Solutions to experience the following demos/solutions in action: Cisco Unified Fabric/Multiprotocol - End-to-End LAN and SAN Meet the Engineer Open for Discussions Schedule a meeting with local storage engineers: Mark Allen, Hui Chen, Craig Ashapa Discuss your project s challenges at the Technical Solutions Clinics Check out other storage networking sessions

101 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online