Unified Storage Access Design

Transcription

1

2 Unified Storage Access Design

3 Unified Storage Access Design Agenda Why Unified Fabric? 101 The Storage Basics SAN iscsi & NAS Storage Design Essentials Understanding FCoE Quality of Service At The Edge Additional FCoE and IP Edge Design Considerations 1K Cisco Nexus x86

4 Evolving Data Centre Architecture Compute Unit I/O - Aggregation and Serialized Re-Use Fewer CNAs (Converged Network adapters) instead of NICs, HBAs and HCAs Limited number of interfaces for Blade Servers FC HBA FC Traffic FC HBA NIC NIC FC Traffic Enet Traffic Enet Traffic CNA CNA All traffic goes over 10GE NIC Enet Traffic HCA IPC Traffic HCA IPC Traffic

5 Unified Fabric Why? Ethernet Economic Model Embedded on Motherboard Integrated into O/S Many Suppliers Mainstream Technology Widely Understood Interoperability by Design FC Economic Model Always a stand alone Card Specialized Drivers Few Suppliers Specialized Technology Special Expertise Interoperability by Test

6 Unified Fabric Why? FCoE SAN iscsi Appliance iscsi Gateway NAS Appliance NAS Gateway Ability to re-provision any compute unit to leverage any access method to the data stored on the spindle Serialized Re-Use (e.g. Boot from SAN and Run from NAS) Virtualization requires that the Storage Fabric needs to exist everywhere the IP fabric does Computer System Computer System Computer System Computer System Computer System Application File System Volume Manager SCSI Device Driver FCoE Driver NIC Ethernet FCoE Block I/O Application File System Volume Manager SCSI Device Driver iscsi Driver TCP/IP Stack NIC Block I/O Ethernet NIC TCP/IP Stack iscsi Layer Bus Adapter Application File System Volume Manager SCSI Device Driver iscsi Driver TCP/IP Stack NIC Ethernet NIC TCP/IP Stack iscsi Layer FC HBA Application File System I/O Redirector NFS/CIFS TCP/IP Stack NIC Ethernet NIC TCP/IP Stack File System Device Driver File I/O Application File System I/O Redirector NFS/CIFS TCP/IP Stack NIC Ethernet NIC TCP/IP Stack File System FC HBA FC Block I/O FC

7 Unified Storage Access Design Agenda Why Unified Fabric? 101 The Storage Basics SAN iscsi & NAS Storage Design Essentials Understanding FCoE Quality of Service At The Edge Additional FCoE and IP Edge Design Considerations 1K Cisco Nexus x86

8 What is a SAN? The Components - Starting Point Application Server Connection to Clients (IP/Ethernet) Just-a-Bunch-of-Disks (JBOD) Connection to Servers (SCSI, FC, iscsi) Disk Subsystem Internal Storage Connection to Additional Storage (SCSI, FC, iscsi) Commonly have internal storage for host OS Direct-attached or network-attached to additional storage Front-end IP connection for client communication JBOD are simple groups of disks with no data protection (no RAID) JBODs are commonly Parallel SCSI or Fibre Channel Loop attached Disk Subsystems are complex arrays of disks with many services (RAID)

9 What is a SAN? The SCSI I/O Channel - Starting Point SCSI is a standard that defines an interface between an initiator (usually a computer) and a target (usually a storage device such as a hard disk) Logical Unit Number (LUN): A 64-bit field within SCSI that identifies the logically addressable unit within a target SCSI device SCSI I/O channel provides half-duplex pipe for SCSI command, data, and status SCSI I/O channel can be internal or external to host Multiple SCSI I/O channels can exist within host TCP/IP Stack NIC Driver NIC Adapter Applications File System Block Device SCSI Generic Adapter Driver SCSI Adapter SCSI Initiator Half-Duplex I/O Channel SCSI SCSI Target

10 What is a SAN? Direct Attached Storage (DAS) As Hosts need more disk externally connected storage gets added Hosts directly access storage as block-level SCSI devices Storage is captive behind the server, limited mobility Limited scalability due to limited devices No efficient storage sharing possible Costly to scale; complex to manage LAN Clients Win2k Linux Unix Win2k Linux Unix SCSI Direct Attached Storage FC FC Application Servers Tape

11 And then, there was SAN!

12 Storage Area Network (SAN) Same SCSI protocol carried over a network transport via serial implementation Transport must not jeopardize SCSI payload (security, integrity, latency) Two primary transports to choose from today, namely IP and Fibre Channel Fibre Channel provides high-speed transport for SCSI payload via Host Bus Adapter (HBA) Fibre Channel overcomes many shortcomings of parallel I/O and combines best attributes of a channel and a network together Fibre Channel SAN LAN Clients Servers Block Storage Devices Characteristics and requirements of the SCSI protocol and emulating raw block disk to the OS define the necessary fabric capabilities and design

13 Fibre Channel Communications Model Fibre Channel has many similarities to IP (TCP) Point to point oriented Facilitated through device login Similar to TCP session establishment N_Port to N_Port connection Logical node connection point Similar to TCP/UDP sockets Flow Controlled Hop-by-hop and End-to-End basis Similar to TCP flow control Different mechanism (no drops) Acknowledged For certain classes of traffic, none for others Similar to TCP / UDP acknowledgement models Multiple connections allowed per device Similar to multiple TCP / UDP sockets Node Transmitter N_Port Receiver Node Transmitter N_Port Receiver Link

14 Fibre Channel Port Types N port: Node ports used to connect devices to switched fabric or point to point configurations. F port: Fabric ports residing on switches connecting N port devices E port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches L port: Loop ports are used in arbitrated loop configurations to build networks without FC switches. These ports often also have N port capabilities and are called NL ports GL port: A generic port capable of operating as either an E or F port. Its also capable of acting in an FL port capacity. Auto Discovery. N N E L NL N F E L FL

15 Fibre Channel Addressing World Wide Names (WWN), Node-WWN, Port-WWN WWNs are used as burnt-in unique addresses assigned to fabric switches, ports, and nodes by the manufacturer Each switch is assigned a WWN at time of manufacture Each switch port is assigned a WWN at the time of manufacture Each HBA is assigned a WWN at the time of manufacture WWNs are created using a MAC address and a prefix to ensure a globally unique address These addresses are registered in the fabric and mapped to an FC_ID Eg. IEEE Extended Name Format 4 bits 12 bits 24 bits 24 bits 0002 N_port or F_port Identifier IEEE Organizational Unique ID (OUI) Locally Assigned Identifier Format Identifier Port Identifier Assigned to each Vendor Cisco 00:25:B5 Vendor-Unique Assignment

16 Fibre Channel Addressing FC_ID Address Model FC_ID address models help speed up routing Switches assign FC_ID addresses to N_Ports Some addresses are reserved for fabric services Private loop devices only understand 8-bit address (0x0000xx) TL_Port can provide proxy service for private-to-public address translation Maximum switch domains = 239 (based on standard) 8 bits 8 bits 8 bits Switch Topology Model Switch Domain Area Device Private Loop Device Address Model Arbitrated Loop Physical Address (AL_PA) Public Loop Device Address Model Switch Domain Area Arbitrated Loop Physical Address (AL_PA)

17 Fibre Channel Fabric Topology Trunking and Channeling Port Channels Higher aggregate bandwidth Hardware-based load balancing Only supported on switch to switch connections (E_Port to E_Port and NP_Port to F_Port) Trunking Trunking E_Port (TE_Port) Carries tagged frames from multiple VSANs Enhanced ISL (EISL) link Trunking E_Port (TE_Port) Standardization of enhanced capabilities is less mature in the Fibre Channel Fabric than you may be used to in the Ethernet and IP world

18 Fibre Channel Topology Virtual SAN (VSAN) - Virtual Fabric Separation A Virtual SAN (VSAN) Provides a Method to Allocate Ports within a Physical Fabric and Create Virtual Fabrics Analogous to VRF s in Ethernet Fabric events are isolated per VSAN FC Features can be configured on a per VSAN basis. Overlay isolated virtual fabrics on same physical infrastructure

19 50 Fibre Channel Topology Fabric Shortest Path First (FSPF) FSPF Provides routing services within a fabric FSPF routes traffic based on destination domain ID (domain ID = single switch) This limits the max number of switches that can support in the Fabric to 239 Using summarized metric to weight multiple paths to a domain Provides load balancing between equal-cost paths Recovery time from failed paths based on set of FSPF timers Outlined in ANSI T11 FC-SW-2 standard Similar to IP OSPF routing protocol with one area Domain 100 Domain 110 Domain 156 Domain 104 Domain 112 Domain 113 Red Path Metric 500 (chosen) load balanced Blue Path Metric 525 Purple Path Metric 600 Yellow Path Metric 500 (chosen) load balanced 75 PortChannels have metric based on aggregated bandwidth

20 iscsi SCSI Transport over TCP/IP A SCSI transport protocol that operates on top of TCP Encapsulates SCSI CDBs and Data into TCP/IP bytestreams (defined by SAM-2 SCSI Architecture Model 2) Host Applications File System iscsi Host Driver Block Device SCSI/TCP Server SCSI Driver TCP/IP Driver FC HBA GigE NIC TCP/IP Stack iscsi SCSI Generic NIC Driver Adapter Driver Fibre Channel TCP/IP Network NIC Adapter SCSI Adapter Fibre Channel Storage Array

21 iscsi iscsi Architectural Model SCSI Applications (File Systems, Databases) SCSI Device-Type Commands SCSI Block Commands SCSI Stream Commands Other SCSI Commands SCSI Generic Commands SCSI Commands, Data, and Status SCSI Transport Protocols Parallel SCSI Transport FCP SCSI over FC iscsi SCSI over TCP/IP Layer 3 Network Transport TCP IP Layer 2 Network Parallel SCSI Interfaces Fibre Channel Ethernet, PPP, HDLC,

22 NAS File System Based Storage Access Network Attached Storage - file-based data storage and access to other devices in a network CIFS - Common Internet File System, successor of SMB protocol with enhanced functionality, Microsoft NFS Network File System, Sun (MAC, Unix, Linux, Novell) TCP or UDP based NAS is different from SAN (Storage Area Network) which uses block-based storage (a block may only contain a part of a single file) Client usually sends 128KB (Solaris) - 512KB (Linux) buffers in a single round-trip NFSv3 is designed for well balanced networks It supports async write and read-ahead, but uses buffer sizes that are useful only on low latency or low bandwidth networks Almost all NFS clients perform local caching of both file and directory data

23 Unified Storage Access Design Agenda Why Unified Fabric? 101 The Storage Basics SAN iscsi & NAS Storage Design Essentials Understanding FCoE Quality of Service At The Edge Additional FCoE and IP Edge Design Considerations 1K Cisco Nexus x86

24 Network vs. Fabric Classical Ethernet Ethernet/IP Goal : provide any-to-any connectivity Unaware of packet loss ( lossy ) relies on ULPs for retransmission and windowing Provides the transport without worrying about the services - services provided by upper layers East-west vs north-south traffic ratios are undefined Network design has been optimized for: High Availability from a transport perspective by connecting nodes in mesh architectures Service HA is implemented separately Takes in to account control protocol interaction (STP, OSPF, EIGRP, L2/L3 boundary, etc )????? Switch?? Switch?? Switch Fabric topology and traffic flows are highly flexible??????? Client/Server Relationships are not predefined

25 Network vs. Fabric Classical Fibre Channel Fibre Channel SAN Transport and Services are on the same layer in the same devices Well defined end device relationships (initiators and targets) Does not tolerate packet drop requires lossless transport Only north-south traffic, east-west traffic mostly irrelevant Network designs optimized for Scale and Availability High availability of network services provided through dual fabric architecture Edge/Core vs Edge/Core/Edge Service deployment I0 I1 DNS Fabric topology, services and traffic flows are structured I(c) I(c) T0 FSPF Switch Zone RSCN I2 T1 FSPF Switch Zone DNS RSCN T(s) T2 FSPF Switch DNS I3 Zone RSCN I4 I5 Client/Server Relationships are pre-defined

26 Storage Controller Fabric vs. Network or Fabric & Network What can fail in a storage fabric Failure causes: Hardware, software, or cable failures Mis-configuration or upgrades Intentional attack Application Host Failure NIC or HBA Failure Link Failure Network Failure (Software, Hardware, Links, etc) Controller or Interface Failure Disk Failure

27 Fabric vs. Network or Fabric & Network Storage Multi-pathing Failover Multipathing: multiple Hardware paths to a single drive (LUN) Active/Active: balanced i/o over both paths (implementation specific) Active/Passive: i/o over primary path switches to standby path upon failure Application Multipathing iscsi Driver Application Multipathing iscsi Driver Active Active Multipathing Software Balances i/o over Available iscsi Interfaces Multipathing Software Monitors Active iscsi Path active passive Primary Path Standby (Failover) Path WWPN a WWPN b LUN Mapped over Multiple Paths Using Different Controller WWPNs WWPN a WWPN b

28 Fabric vs. Network or Fabric & Network SAN Dual Fabric Design SAN and LAN have different design assumptions SAN leverages dual fabric with multi-pathing failover initiated by the client LAN leverages single fully meshed fabric with higher levels of component redundancy FC FC Core Core? FC

29 Fabric vs. Network or Fabric & Network Oversubscription and QoS All Network Designs Have Some Degree of Oversubscription Without oversubscription, storage networks can be costly Device capabilities (peak and sustained) must be considered along with network oversubscription Must consider oversubscription during a network failure event Remember, all traffic flows towards targets main bottlenecks Capacity and over-subscription is not a function of the protocol (NAS, FC, iscsi) but of the application I/O requirements Disk Oversubscription Disk Do Not Sustain Wire-Rate I/O with Realistic I/O Mixtures Most Major Vendors Promote 12:1 host:disk Fan-Out 70 MBps Max/Port (Common) Typical Oversubscription in Two-Tier Design Can Approach 8:1, Sometimes Even Higher 7:1 O.S. (Common) Tape Oversubscription Low Sustained I/O Rates LTO-2 Native Transfer Rate ~ 60 MBps 60 MBps Max/Port (Common) Port Channels Help Reduce Oversubscription While Maintaining HA Requirements Example: 10:1 O/S ratio 60 Servers with 4 Gb HBAs 240 G 24 G 24 G FC 40 MBps Max/HBA (Common) Host Oversubscription Most Hosts Suffer from PCI Bus Limitations, OS, and Application Limitations Thereby Limiting Maximum I/O and Bandwidth Rate

30 Unified Storage Access Design Agenda Why Unified Fabric? 101 The Storage Basics SAN iscsi & NAS Storage Design Essentials Understanding FCoE Quality of Service At The Edge Additional FCoE and IP Edge Design Considerations 1K Cisco Nexus x86

31 Unified Fabric & FCoE FCoE Benefits Encapsulation of FC Frames over Ethernet Fewer Cables Both block I/O & Ethernet traffic co-exist on same cable Enables FC to Run on a Lossless Ethernet Network Fewer adapters needed Overall less power Ethernet Fibre Channel Traffic Interoperates with existing SAN s Management of SAN s remains constant No Gateway

32 Unified Fabric & FCoE When? FCoE Projected Growth Source: Infonetics

33 What is FCoE? Standards for I/O Consolidation FCoE Fibre Channel on network media FC-BB-5 Completed June 2009 Published by ANSI in May 2010 PFC IEEE 802.1Qbb Priority-based Flow Control IEEE DCB ETS IEEE 802.1Qaz Priority Grouping Enhanced Transmission Selection DCBx IEEE 802.1Qaz Configuration Verification Completed March 2011 Forwarded to RevCom for publication

34 What is FCoE? It s Fibre Channel From a Fibre Channel standpoint it s FC connectivity over a new type of cable called Ethernet From an Ethernet standpoints it s Yet another ULP (Upper Layer Protocol) to be transported FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FC-1 Encoding FC-0 Physical Interface FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FCoE Logical End Point Ethernet Media Access Control Ethernet Physical Layer

35 What is FCoE? FCoE Initialization Protocol (FIP) Neighbour Discovery and Configuration (VN VF and VE to VE) Enode Initiator FCoE Switch FCF Step 1: FCoE VLAN Discovery FIP sends out a multicast to ALL_FCF_MAC address looking for the FCoE VLAN VLAN Discovery VLAN Discovery FIP frames use the native VLAN Step 2: FCF Discovery FIP sends out a multicast to the ALL_FCF_MAC address on the FCoE VLAN to find the FCFs answering for that FCoE VLAN FCF s responds back with their MAC address FCF Discovery FLOGI/FDISC FCF Discovery FLOGI/FDISC Accept FCoE Initialization Protocol (FIP) Step 3: Fabric Login FIP sends a FLOGI request to the FCF_MAC found in Step 2 Establishes a virtual link between host and FCF ** FIP does not carry any Fibre Channel frames FC Command FC Command Responses FCoE Protocol

36 FCoE, Same Model as FC Connecting to the Fabric Same host to target communication Host has 2 CNA s (one per fabric) Target has multiple ports to connect to fabric Connect to a capable switch Port Type Negotiation (FC port type will be handled by FIP) Speed Negotiation DCBX Negotiation Access switch is a Fibre Channel Forwarder (FCF) Dual fabrics are still deployed for redundancy Ethernet Fabric Unified Wire ENode FC Fabric CNA Target DCB capable Switch acting as an FCF

37 My port is up can I talk now? FIP and FCoE Login Process Step 1: FIP Discovery Process FCoE VLAN Discovery FCF Discovery Verifies Lossless Ethernet is capable of FCoE transmission Step 2: FIP Login Process Similar to existing Fibre Channel Login process - sends FLOGI to upstream FCF FCF assigns the host a Enode MAC address to be used for FCoE forwarding (Fabric Provided MAC Address - FPMA) FC-MAP (0E-FC-xx) FC-ID FC or FCoE Fabric VF_Port VN_Port CNA Target E_ports or VE_Port FC-MAC Address FC-MAP (0E-FC-xx) FC-ID ENode FIP Discovery

38 Login complete almost there Fabric Zoning Zoning is a feature of the fabric and is independent of Ethernet transport Zoning can be configured on the Nexus 5000/7000 using the CLI or Fabric Manager If Nexus 5000 is in NPV mode, zoning will be configured on the upstream core switch and pushed to the Nexus 5000 Devices acting as Fibre Channel Forwarders participate in the Fibre Channel security (Zoning) control DCB only bridges do not participate in zoning and require additional security mechanisms (ACL applied along the forwarding path on a per FLOGI level of granularity) name EXAMPLE_Zone fcid 0x [pwwn 10:00:00:00:c9:76:fd:31] [initiator] fcid 0x [pwwn 50:06:01:61:3c:e0:1a:f6] [target] pwwn 50:06:01:61:3c:e0:1a:f6 FC/FCoE Fabric FCF with Domain ID 10 Initiator Target pwwn 10:00:00:00:c9:76:fd:31

39 Fibre Channel Switch FCoE Explicit Roles still defined in the Fabric FCoE does not change the explicit port level relationships between devices (add a V to the port type when it is an Ethernet wire) Servers (VN_Ports) connect to Switches (VF_Ports) Switches connect to Switches via Expansion Ports (VE_Ports) FCF Switch VE_Port VE_Port VF_Port VNP_Port FCoE_ NPV Switch VF_Port VN_Port End Node FCoE Switch : FCF VF_Port VN_Port End Node

40 A FCoE Edge CNA: Converged Network Adapter Nexus Edge participates in both distinct FC and IP Core topologies Converged Network Adapter (CNA) presents two PCI address to the Operating System (OS) OS loads two unique sets of drivers and manages two unique application topologies FCF FCF Server participates in both topologies since it has two stacks and thus two views of the same unified wire Data VLAN(s) are passed to the Ethernet driver 10GbE 10GbE Link FCoE VLAN terminates on the CNA SAN Multi-Pathing provides failover between two fabrics (SAN A and SAN B ) NIC Teaming provides failover within the same fabric (VLAN) Ethernet Driver bound to Ethernet NIC PCI address Ethernet Drivers Ethernet Fibre Channel PCIe Fibre Channel Drivers FC Driver bound to FC HBA PCI address Operating System

41 A FCoE Edge N-Port Virtualizer (NPV) N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a switch to act like a server performing multiple logins through a single physical link Physical servers connected to the NPV switch login to the upstream NPIV core switch Physical uplink from NPV switch to FC NPIV core switch does actual FLOGI Subsequent logins are converted (proxy) to FDISC to login to upstream FC switch No local switching is done on an FC switch in NPV mode FC edge switch in NPV mode does not take up a domain ID F-Port Eth1/1 Server1 N_Port_ID 1 NP-Port F-Port Eth1/2 Server2 N_Port_ID 2 F_Port Eth1/3 Server3 N_Port_ID 3 N-Port FC NPIV Core Switch

42 Unified Storage Access Design Agenda Why Unified Fabric? 101 The Storage Basics SAN iscsi & NAS Storage Design Essentials Understanding FCoE Quality of Service At The Edge Additional FCoE and IP Edge Design Considerations 1K Cisco Nexus x86

43 Priority Flow Control FCoE Flow Control Mechanism 802.1Qbb Enables lossless Ethernet using PAUSE based on a COS as defined in 802.1p When link is congested, CoS assigned to no-drop will be PAUSED Other traffic assigned to other CoS values will continue to transmit and rely on upper layer protocols for retransmission Not only for FCoE traffic Fibre Channel Transmit Queues One Ethernet Link Receive Buffers One Two Two Three STOP PAUSE Three Packet R_RDY Four Five Four Five Eight Virtual Lanes Six Six Seven Seven B2B Credits Eight Eight

44 Enhanced Transmission Selection (ETS) Bandwidth Management 802.1Qaz Prevents a single traffic class of hogging all the bandwidth and starving other classes When a given load doesn t fully utilize its allocated bandwidth, it is available to other classes Helps accommodate for classes of a bursty nature Offered Traffic 10 GE Link Realized Traffic Utilization 3G/ s 3G/ s 2G/ s 3G/s HPC Traffic 3G/s 2G/s 3G/ s 3G/ s 3G/ s 3G/s Storage Traffic 3G/s 3G/s 3G/ s 4G/ s 6G/ s 3G/s LAN Traffic 4G/s 5G/s t1 t2 t3 t1 t2 t3

45 Data Center Bridging Control Protocol DCBX Overview Qaz Negotiates Ethernet capability s : PFC, ETS, CoS values between DCB capable peer devices Simplifies Management : allows for configuration and distribution of parameters from one node to another Responsible for Logical Link Up/Down signaling of Ethernet and Fibre Channel DCBX is LLDP with new TLV fields The original pre-standard CIN (Cisco, Intel, Nuova) DCBX utilized additional TLV s DCBX negotiation failures result in: per-priority-pause not enabled on CoS values vfc not coming up when DCBX is being used in FCoE environment dc # sh lldp dcbx interface eth 1/40 Local DCBXP Control information: Operation version: 00 Max version: 00 Seq no: 7 Ack no: 0 Type/ Subtype Version En/Will/Adv Config 006/ Y/N/Y 00 <snip> DCBX Switch DCBX CNA Adapter

46 Nexus QoS QoS Policy Types There are three QoS policy types used to define system behavior (qos, queuing, network-qos) There are three policy attachment points to apply these policies to Ingress interface System as a whole (defines global behavior) Egress interface Policy Type Function Attach Point qos queuing network-qos Define traffic classification rules Strict Priority queue Deficit Weight Round Robin System class characteristics (drop or nodrop, MTU), Buffer size, Marking system qos ingress Interface system qos egress Interface ingress Interface system qos

47 Some common QoS Edge Best Practice

48 Nexus 5500 QoS MTU per Class of Service (CoS Queue) MTU can be configured for each class of service (no interface level MTU) When forwarded using cut-through, frames are truncated if they are larger than MTU When forwarded using store-and-forward, frames are dropped if they are larger than MTU iscsi and NFS will benefit from enabling Jumbo frames N5k(config)# policy-map type network-qos policy-jumbo N5k(config-pmap-nq)# class type network-qos class-default N5k(config-pmap-nq-c)# mtu 9216 N5k(config-pmap-nq-c)# system qos N5k(config-sys-qos)# service-policy type network-qos policyjumbo The command to set the jumbo MTU on Netapp filer is : ifconfig e0b netmask mtusize 9000 Each CoS queue on the Nexus 5500 and F1/F2 line card supports a unique MTU

49 Enhanced Transmission Selection - N5K Bandwidth Management When configuring FCoE by default, each class is given 50% of the available bandwidth Can be changed through QoS settings when higher demands for certain traffic exist (i.e. HPC traffic, more Ethernet NICs) 1Gig FC HBAs 1Gig Ethernet NICs N5k-1# show queuing interface ethernet 1/18 Ethernet1/18 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 50 1 WRR 50 Traditional Server Best Practice: Tune FCoE queue to provide equivalent capacity to the HBA that would have been used (1G, 2G, )

50 Unified Storage Access Design Agenda Why Unified Fabric? 101 The Storage Basics SAN iscsi & NAS Storage Design Essentials Understanding FCoE Quality of Service At The Edge Additional FCoE and IP Edge Design Considerations 1K Cisco Nexus x86

51 Unified Fabric Design The FCoE VLAN Each FCoE VLAN and VSAN count as a VLAN HW resource therefore a VLAN/VSAN mapping accounts for TWO VLAN resources FCoE VLANs are treated differently than native Ethernet VLANs: no flooding, broadcast, MAC learning, etc. BEST PRACTICE: use different FCoE VLANs/VSANs for SAN A and SAN B The FCoE VLAN must not be configured as a native VLAN Shared Wires connecting to HOSTS must be configured as trunk ports and STP edge ports Note: STP does not run on FCoE vlans between FCFs (VE_Ports) but does run on FCoE VLANs towards the host (VF_Ports) Nexus 5000 FCF VLAN 10,20 LAN Fabric VSAN 2 Fabric A VLAN 10,30 Fabric B VSAN 3 Nexus 5000 FCF STP Edge Trunk! VLAN 20 is dedicated for VSAN 2 FCoE traffic (config)# vlan 20 (config-vlan)# fcoe vsan 2

52 Unified Fabric Design FCoE and vpc together vpc with FCoE are ONLY supported between hosts and N5k or N5k/2232 pairs AND they must follow specific rules A vfc interface can only be associated with a single-port port-channel While the port-channel configurations are the same on N5K-1 and N5K-2, the FCoE VLANs are different FCoE VLANs are not carried on the vpc peer-link (automatically pruned) FCoE and FIP ethertypes are not forwarded over the vpc peer link either vpc carrying FCoE between two FCF s is NOT supported Best Practice: Use static port channel configuration rather than LACP with vpc and Boot from SAN (LACP supported in NX-OS 5.1(3)N1 and above) VLAN 10,20 LAN Fabric Nexus 5000 FCF-A Fabric A VLAN 10 ONLY HERE! VLAN 10,30 Direct Attach vpc Topology Fabric B vpc contains only 2 X 10GE links one to each Nexus 5X00 Nexus 5000 FCF-B STP Edge Trunk

53 Fabric Extender - FEX Unified Fabric and FCoE FC Nexus 5000 access switches operating in NPV mode Fabric A Fabric B With NX-OS release 4.2(1) Nexus 5000 supports F- Port Trunking and Channeling on the links between an NPV device and upstream FC switch (NP port -> F port) Nexus 5000/5500 FCoE FC F_Port Trunking: Better multiplexing of traffic using shared links (multiple VSANs on a common link) Nexus 5000 as FCF or as NPV device F_Port Channeling: Better resiliency between NPV edge and Director Core No host re-login needed per link failure No FSPF recalculation due to link failure Nexus GE FEX Simplifies FC topology (single uplink from NPV device to FC director) Generation 2 CNAs

54 FCoE Enhancement: Enhanced vpc FCoE vpc from the host and from FEX at the same time Prior to this enhancement, FCoE was NOT supported in a FEX active/active topology As of Cisco NXOS Release 5.1(3)N1(1) for N5K, EVPC will be supported Why wasn t this supported? Ethernet traffic is hashed from a FEX to two switches, which violates FC fabric A/fabric B separation FCoE Storage FCoE Nexus 5000 (San A) SAN A Nexus 5000 (San B) SAN B Nexus 2000 Fabric Extender (FEX) CNA

55 FCoE Enhancement: Enhanced vpc Allowing for common FEX deployment model across the infrastrcuture How is this achieved? We must continue to maintain fabric separation SAN A SAN B Configuration reveals how this is achieved switcha(config)# fex 101 switcha(config-fex)# fcoe switchb(config)# fex 101 FCoE Storage FCoE Nexus 5000 (San A) Nexus 5000 (San B) Nexus 2000 Fabric Extender (FEX) CNA

56 Nexus 5500 Series Unified Port Flexibility and Reuse Unified Port supports multiple transceiver types 1G Ethernet Copper/Fibre 10G Ethernet Copper/Fibre 10G DCB/FCoE Copper/Fibre 1/2/4/8G Fibre Channel Change the transceiver and connect evolving end devices, Server 1G to 10G NIC migration FC to FCoE migration FC to NAS migration Unified Port supports a Unified Access Layer across the entire Fabric Any Unified Port can be configured as Ethernet Fibre Channel Traffic or Fibre Channel Traffic Fibre Channel Unified Port Any device in any rack connected to the same edge infrastructure Servers, FCoE attached Storage Servers FC Attached Storage

57 Need Director-class at the edge? Storage VDC on the Nexus 7000 Separate VDC running ONLY storage related protocols Storage VDC: a virtual MDS FC switch Running only FC related processes Only one such VDC can be created Provides control plane separation Dedicated for VE_Ports LAN VDC Storage VDC Shared Converged Port Model for host/target interfaces, not VE_Port Ingress Ethernet traffic is split based on frame ether type FCoE traffic is processed in the context of the Storage VDC LAN VDC Storage VDC Ethernet FCoE & FIP Ethernet Converged I/O FCoE & FIP

58 Creating the Storage VDC Port and VLAN allocations Create VDC of type storage and allocate non-shared interfaces: N7K-50(config)# vdc fcoe id 2 type storage N7K-50(config-vdc)# allocate interface Ethernet4/1-16, Ethernet4/19-22 Allocate FCoE vlan range from the Owner VDC to the Storage VDC. This is a necessary step for sharing interfaces to avoid vlan overlap between the Owner VDC and the Storage VDC N7K-50(config) vdc fcoe id 2 N7K-50(config-vdc)# allocate fcoe-vlan-range from vdcs n7k-50 Allocated the shared interfaces: N7K-50(config-vdc)# allocate shared interface Ethernet4/17-18 Install the license for the FCoE Module. n7k-50(config)# license fcoe module 4

59 It s happening today, moving beyond the edge

60 Traditional Data Center Design Ethernet LAN and Fibre Channel SAN Physical and Logical separation of LAN and SAN traffic Additional Physical and Logical separation of SAN fabrics L3 L2 Nexus 7000 Nexus 5000 Fabric A MDS 9000 FC Fabric B NIC HBA

61 Converged Access Sharing Access Layer for LAN and SAN Shared Physical, Separate Logical LAN and SAN traffic at Access Layer Physical and Logical separation of LAN and SAN traffic at Aggregation Layer Additional Physical and Logical separation of SAN fabrics Storage VDC for additional management / operation separation Higher I/O, HA, fast re-convergence for host LAN traffic This topology is where most customers are today. L3 L2 Nexus 7000 Nexus 5000 Fabric A FCoE MDS 9000 FC Fabric B CNA

62 Converged Network Fabrics with Dedicated Links Maintaining Dual SAN fabrics with Overlay LAN and SAN traffic share physical switches LAN and SAN traffic use dedicated links between switches Fabric A Fabric B LAN/SAN All Access and Aggregation switches are FCoE FCF switches L3 L2 Dedicated links between switches are VE_Ports VE FCF Storage VDC for additional operation separation at high function agg/core Improved HA, load sharing and scale for LAN vs. traditional STP topologies Nexus 7000 Nexus 5000 FCF FCF MDS 9500 SAN can utilize higher performance, higher density, lower cost Ethernet switches for the aggregation/core CNA FCoE FC

63 Looking forward: Converged Network Single Fabric SAN Separation at the Access Switch LAN and SAN traffic share physical switches and links FabricPath enabled All Access switches are FCoE FCF switches VE_Ports to each neighbor Access switch Single process and database (FabricPath) for forwarding Improved (N + 1) redundancy for LAN & SAN Sharing links increases fabric flexibility and scalability Distinct SAN A & B for zoning isolation and multipathing redundancy Is your IT organization ready for this move??? L3 L2 Nexus 7000 Nexus 5000 Fabric A Fabric B 10,20 CNA1 FCF CNA2 20, FCF 10,20 20, Array1 FCoE Array2

64 Transparent Bridges? FIP Snooping What does a FIP Snooping device do? FIP solicitations (VLAN Disc, FCF Disc and FLOGI) sent out from the CNA and FIP responses from the FCF are snooped How does a FIP Snooping device work? The FIP Snooping device will be able to know which FCFs hosts are logged into Will dynamically create an ACL to make sure that the host to FCF path is kept secure A FIP Snooping device has NO intelligence or impact on FCoE traffic/path selection/load balancing/login selection/etc Mentioned in the Annex of the FC-BB-5 (FCoE) standard as a way to provide security in FCoE environments Spoofed MAC 0E.FC.00.DD.EE.FF VN VF FIP Snooping ENode MAC 0E.FC ENode FCF FCF MAC 0E.FC.00.DD.EE.FF

65 FLOGI Fibre Channel Aware Device FCoE NPV FABRIC A Target FC FCoE Pass through device All FCoE Switching is performed at the upstream FCF Addressing is pass out by the upstream FCF FCF More FCoE connectivity to hosts without: Running into the domain ID issue Less-expensive Consistent management Domain ID and FC-MAP come from the FCF VF N7K, MDS or N5K Proxy s FIP functions between a CNA and an FCF FCoE VLAN configuration and assignment FCF Assignment FCoE-NPV load balance logins from the CNAs evenly across the available FCF uplink ports FCoE-NPV will take VSAN into account when mapping or pinning logins from a CNA to an FCF uplink VF VN VN P N5K in FCoE_NPV Mode FCoE_NPV does not consume a domain ID Operations and management process are in line with today s SAN-Admin practices Similar to NPV in a native Fibre Channel network E_Node MAC Address Dedicated FCoE Link Converged Link

66 FCoE-NPV configuration Details N7K w/ release 5.2.x N7K Storage VDC n7k-fcoe(config)# feature npiv MDS Global command MDS # feature npiv Becomes VNP to VF MDS w/ release 5.2.x Edge N5000/5500 n5k(config)# feature fcoe-npv Proper no drop QOS needs to be applied to all NPIV VDC s and NPV switches as shown in earlier slides Edge N5000/5500 LACP Port-channels an be configured between switches for High availability.

67 Other Multi-Hop Design Options VE Ports FC FCoE Fibre Channel Forwarders (FCF) Allows switching of FCoE frames across multiple hops E E FCF VE VE FCF Creates Standards based FCoE ISL Necessary for Multi-Hop FCoE Nothing Further Required (No TRILL, vpc or Spanning Tree) FCF supports all FC Functionalities: Support up to 7 hops Support up to 10,000 logins per fabric Supports up to 8,000 Zones per switch Supports up to 500 Zonesets per switch E-Ports w/ FC VE-Ports w/ FCoE It s Fibre Channel Same MDS FC CLI Available on the Ethernet Switch

68 UCS Multihop FCoE: Separate FCoE and Ethernet Uplinks *) *) available since UCS release 2.1.a

69 UCS Multihop FCoE: Converged FCoE and Ethernet Uplinks *) *) available since UCS release 2.1.a

70 UCS Multihop FCoE: Converged FCoE and Ethernet uplinks with separate Ethernet VPC *) *) available since UCS release 2.1.a

71 Call to Action Visit the Cisco Campus at the World of Solutions to experience Cisco innovations in action Get hands-on experience attending one of the Walk-in Labs Schedule face to face meeting with one of Cisco s engineers at the Meet the Engineer center Discuss your project s challenges at the Technical Solutions Clinics 83

72 84

73