The Confusion Arising from Converged Multi Hop FCoE Topologies

Transcription

1 Author: Simon Gordon, Juniper Networks The Confusion Arising from Converged Multi Hop FCoE Topologies Simon Gordon, Juniper Networks

2 SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. 2

3 Abstract The Confusion Arising from Converged Multi Hop FCoE Topologies There is much information on the various approaches for convergence, the different types of products that can be used, and the implications for a practical deployments. Some of this information is out of date, some product specific, and some just informal internet discussion. In addition there is an overloading of terminology that can further lead to confusion During this session, Simon will explain clearly and precisely each of the different types of product, the possible capabilities of such products, where they sit within the most typical deployments. He will also examine the implications of this on SAN and LAN administration and operation. Finally, Simon will discuss some of the key changes that arise as the industry moves from the current FC-BB-5 standard to the FC-BB-6 standard which is under development. 3

4 Hands On Lab Hear no evil, see no evil, speak no evil Go have a look for yourselves Unified Storage Infrastructure HOL Unified Storage IP Solutions HOL 4

5 Agenda Part 1 (we will go fast) Two nations divided by a common language Who said that? Ethernet & IP History Lesson Tiers & Hops L2 Switches, L3 Routers, and L2/L3 Switches TRILL Fabrics Fibre Channel History Lesson Once upon a time Now with NPIV, WDM & FCIP Not to mention Virtual Fabrics & Inter Fabric Routing 5

6 Agenda Part 2 (we will go slow) FCoE with FC-BB-5 Roles of different devices Transit Switch, Gateway, FCoE/FC Switch Capabilities of different devices Basic, Advanced Future with FC-BB-6 New types of device 6

7 Agenda Part 1 Two nations divided by a common language Who said that? Ethernet & IP History Lesson Tiers & Hops L2 Switches, L3 Routers, and L2/L3 Switches Fibre Channel History Lesson Once upon a time Now with NPIV, WDM, and FCIP Not to mention Virtual Fabrics & Inter Fabric Routing 7

8 Refer to the SNIA Dictionary Languages evolve Slowly within isolated communities Battle between simplicity and expressiveness Rapidly when communities interact or combine As is happening with network convergence Not to mention the influence of fashion and even individual idiosyncrasies Individuals, companies, organizations Electrical, mechanical, software terms This can be confusing at times Check out the SNIA Dictionary 8

9 Several Industry Groups T11 9

10 Coming back to the subject There are multiple possible valid definitions for many of the terms we use in networking Tiers and Hops, let alone bridge, switch, router, gateway. We have overloaded many of the terms and use them in different ways often in the same conversation Fabric, virtual As such we need to take care Understand common usage Understand official protocol definitions I say this now to avoid an argument later ;-) 10

11 Agenda Part 1 Two nations divided by a common language Who said that? Ethernet & IP History Lesson Tiers & Hops L2 Switches, L3 Routers, and L2/L3 Switches Fibre Channel History Lesson Once upon a time Now with FCIP, ifcp, and NPIV Not to mention Virtual Fabrics & Inter Fabric Routing 11

12 Tiers & Hops in a Network Multi-tier network With a pure L2 network every hop is an L2 hop so we know the exact L2 hop count for any path from source to destination L3 Boundary Core Router Tier 3 Tier 2 - Aggregation Tier 1 - Access 3 Tiers of L2 Switching The hop is the process of traversing the switch or as IEEE say the bridge Servers From server to server may be 1, 3, or 5 L2 hops depending on relative location L3 Router L2/L3 Switch L2 Switch 12

13 Tiers & Hops in a Network MAN & WAN Across the MAN/WAN L3 hops With a pure L3 network every hop is an L3 hop so we know the exact L3 hop count for any path from source to destination L3 Boundary The hop is the process of traversing the router Inside the Datacenter L2 hops L3 Router L2/L3 Switch L2 Switch 13

14 Tiers & Hops in a Network Multi-tier network Typically customers deploy with a defined L2/L3 boundary at a fixed layer/tier in the network. Now there is a mixture of L2 and L3 hops for some paths Core Router Tier 3 L3 Boundary Tier 2 - Aggregation Tier 1 - Access 1 Tier of L2 Switching 3 Tiers of Switching Servers NB: Many L2 switches actually have an understanding of L3 and higher protocols From server to server may be 1, 3, or 5 hops but may be a mixture of L2 and L3 hops In reality L3 could be anywhere though arbitrary placement makes a network hard to manage, firewalls further complicate the picture L3 Router L2/L3 Switch L2 Switch 14

15 Tiers & Hops in a Network Multi-tier network Core Router Tier 3 Tier 2 Tier 1 - Aggregation 5 Tiers Blade Switches may add another tier From virtual server to virtual server may even be 7 or 9 hops in total!!! Virtual Switches may add another tier Blade & Virtual switches again add tiers to the network as well as adding hops L3 Router L2/L3 Switch L2 Switch 15

16 Tiers & Hops in a Network Multi-tier network Whilst all forms of multipathing (IEEE 802.1aq, TRILL, MC LAG) can remove the effect of loops, they do not solve the problem of hops and tiers L2 multi pathing 802.1aq TRILL Fabric Tier 3 Tier 2 Tier 1 the number of tiers is the same the number of hops is the same the decision on location of L3 default router still exists Path Selection has changed and packets are encapsulated as they traverse the fabric From server to server may be 1, 3, or 5 hops TRILL Commonly using Ethernet L2 for the hop to hop protocol support (just as IP often does) L3 Router L2/L3 Switch L2 Switch 16

17 Ethernet & IP Summary Tiers Hops The definition is clear Layers of switching / routing boxes in a hierarchical structure Generic term independent of protocol based on topology Traditionally network theory considers traversing the network device from ingress to egress as the definition of the hop Specific meaning for each and every protocol IEEE: L2 Hop IETF: L3-IP Hop Fabric Hop (routing of encapsulated packets) Path Selection depends on various factors 17

18 Agenda Part 1 Two nations divided by a common language Who said that? Ethernet & IP History Lesson Tiers & Hops L2 Switches, L3 Routers, and L2/L3 Switches Fibre Channel History Lesson Once upon a time Now with NPIV, WDM, and FCIP Not to mention Virtual Fabrics & Inter Fabric Routing 18

19 Once upon a time Many topologies discussed Even tested and deployed A hop was a hop was a hop Process of traversing an fcf Gradually settled on core/edge Looks like a 2 tier network with some form of multipathing Dual Rail is assumed but not shown to simplify the diagrams E_ N_ F_ E_ F_ N_ 19

20 What did he say was the hop? The Fibre Channel standards were developed following normal network theory, leveraging from existing protocols. As such the hop was the process of traversing the Though the standards never said a hop is xxxxx Actually before that we had loops and hubs! Early on there was little or no cascading of switches Different vendors used the terms differently Books recognized that different definitions were possible However after some time common practice in the real world of suppliers and customers We all found ourselves referring to the ISL as the hop We ignored the wire hop from end device to switch Aka we ignored the wire from server to switch And we ignored the wire form switch to storage 20

21 Once upon a time core/edge 2 hops from server to storage 2 tier topology edge/core/edge 3 hops from server to storage 3 tier topology Path Selection Forwarded based on FC address Session based (early days) Exchange based (later on) Every switch had all switch services Dual Rail is assumed but not shown to simplify the diagrams North south traffic 21

22 Once upon a time edge/core/edge Is it really a 3 tier network Or is it just a 2 tier network in disguise Its just a matter of perspective Dual Rail is assumed but not shown to simplify the diagrams East West traffic 22

23 Traditional FC Summary Tiers Hops Most FC SANs were 2 tier (or 3 depending on definition) Every box in the network was an and so an FC Hop Path Selection based on source/destination FCIDs Increasingly sophisticated multipathing box to box Increasingly sophisticated multipathing end to end Most FC SANs had 2 or 3 hops from server to storage Every switch in the network was an switch in the classic sense of the term from a Fibre Channel perspective But please remember in FC World people say hop==isl! 23

24 FC-FC NPIV Gateway N_ Virtualizer FC-GS-6 brief mention Using NPIV to proxy F 5 NPIV Proxy (FLOGI/FDISC- >FDISC) Session based load balancing F F 6 N 1 Fabric 1 F 7 N 2 F F 7 F 5 N 3 F 5 F 6 F 7 NPIV Proxy (FLOGI/FDISC- >FDISC) Session based load balancing F N 4 F Fabric 2 This example shows a physical device acting as gateway for two independent FC SAN Fabrics Aka one physical gateway is in a sense acting as two virtual gateways However various possible configurations of device are possible 24

25 N_ Virtualization Gateway FC-FC NPIV Proxy Gateway Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection (GW-) Load balancing based on FLOGI Load balancing based on FDISC No exchange based load balancing Forwarding based on FC addresses FC Services Present An FC Stack but not acting as a full switch Fabric Facing Presents as a server (N ports) Server Facing Present as an (F s) Should we consider this a hop? Did you know NPIV Gateway FLOGI s the fabric N_-F_ NPIV GW maps Server FLOGI to an FDISC VF_s spawned from the F_ Server N_ maps to an VF_ F_ N_ E_ F_ N_ NPIV E_ NPIV NPIV NPIV 25

26 Now with WDM WDM No forwarding decision WDM Is not an FC hop Is not a hop of any type FC Services Present None No hops of any interest? In the optical network May be multiple boxes Do not count at the FC level FC rules must be considered E_D_TOV R_A_TOV R_T_TOV FC Services Present None E_ WDM WDM E_ The optical network is not a source of congestion but FC bandwidth is typically more limited 26

27 Now with FCIP FCIP Gateway Is making forwarding decisions FCIP Gateway Is not an FC hop but it is a hop Indeed if it is also doing routing then the FC hop count is reset or rather not counted across the boundary FC Services Present An FC Stack More hops and of interest In between the FCIP Gateway Many L2 Ethernet hops Many L3 IP hops Do not count at the FC level FC rules must still be considered E_D_TOV R_A_TOV R_T_TOV FC Services Present None FCIP FCIP The IP network is a source of congestion 27

28 Not to mention VFT and IFR Not a full description of VFT / IFR Introduced multiple switching domains Introduced routing between these domains Both VFT and IFT headers each add a hop count to Fibre Channel where previously FC had none 3 FC Hops in total 2 FC Hops in VF1 2 FC Hops in VF1 FCID VF1 FCID VF1 Converged Device VF1 Converged Device VF1/VF2 Converged Device VF2 FCID VF2 FCID VF2 L3 Forwarding L3 Forwarding Plus IFR L3 Forwarding 28

29 Pre FCoE Summary Tiers Hops The definition is clear Layers of boxes in a hierarchical structure Generic term independent of protocol based on topology The box takes ingress packets and makes forwarding decision Specific meaning for different types of box FC Hop NPIV Proxy Hop FCIP Gateway Hop Ethernet L2 and IP L3 Hops between FCIP Gateways And if you want to use the fc tradition then say isl-hop 29

30 So what is it we care about? When considering network design Hops & tiers Different types of boxes Fibre Channel Switches & NPV Switches Not to mention all that MAN/WAN Stuff We are thinking about the following Are there a forwarding decisions being made Are their points of queuing / buffering 30

31 Agenda Part 2 FCoE with FC-BB-5 Roles of different devices Transit Switch, Gateway, FCoE/FC Switch Capabilities of different devices Basic, Advanced Future with FC-BB-6 New types of device 31

32 And Now FCoE not a lesson FIP/VLAN Discovery happens using untagged packets Uses the FIP Ethertype Native VLAN can be used to control visibility during the discovery phase (separate discussion) Discovers which VLANs contain FCoE s (or gateways) Establish & Maintains our VE-VE FCoE-ISL or VN-VF connection Maintains there s protocol stuff to keep the connection working Did you notice have we overloaded the term VF_? Once we have actual FCoE FC sessions running Uses a FCoE Ethertype (different to FIP Ethertype) Run within the actual VLANS that correspond to actual Virtual Fabrics using VLAN tagged packets FIP keep alives running, connection recovery mechanisms 32

33 Key Device Types (or modes) Fibre Channel Forwarder or Switch (FC/FCoE) Fixed or configurable ports, any mix possible May also be an L2/L3 switch on Ethernet ports NPIV Proxy Gateway (N_ Virtulizer) FCoE-FC FCoE-FCoE FC-FCoE May also be an L2/L3 switch on Ethernet ports Ethernet Switch or FCoE Transit Switch L2 forwarding of FCoE frames plus some bits 33

34 FCoE Based FC SAN all If every box is an FCoE Every box is an FC Hop Same number of hops for any case Please Please Please Please The correct term is still FC Hop Hop is to do with crossing the Hop is not to do with the link layer Forwarding Mechanism Session based Exchange based Forwarded based on FC address NOTE: Ignoring virtual fabrics for now Ignoring combined /L2 Switch New FC port setup mechanism used to establish virtual ports in theory across an arbitrary L2 network (VE- VE, VF-VN, and later with FC-BB-6 VN-VN) though in this case looks no different to before Ethernet LAG below the FCoE layer Ethernet LAG below the FCoE layer VF_ VN_ Path Selection Mechanism Exactly the same as native FC At least in terms of selecting which virtual port (VN/VF/VE) to use Ethernet LAGs may exist beneath the virtual port layer VE_ Ethernet LAG below the FCoE layer VE_ VF_ VN_ May be more than just an FCoE disk array 34

35 Fibre Channel Switch or Forwarder Forwards packets/frames based on L3 address When a frame or packet is received on an ingress port the Domain ID is used to decide what egress port to transmit Has all the usual FC Services and has its own domain id With Convergence May have FC, FCoE, and/or multi personality ports May also implement L2-Ethertnet switching on FCoE ports Implication regarding FCoE Forwarding Minimum Features Same as normal FC Switch Must used enhanced Ethernet that meets the minimum set of FCoE requirements 35

36 FCoE-FC-Switch (optional) VF DCB DCB DCB DCB VF FC Services F Local Area Network MAC(s) VF E VE VE FC Services E VF VE/VF s Instantiated as required F VF Support LAG on the FCoE side VF/VE s could be n*10gbe s Trusted / untrusted modes ACL Protection only needed at server edge Ability to disable L2 End of Row / Against FC SAN Design Level of DCB & QoS Features To leverage DCB and Ethernet QoS for traffic separation Multiple 802.1p priorities & VLAN per Virtual Fabric To leverage DCB and Ethernet QoS for traffic separation Vx Scaling To support L2 layer Virtual Fabric Support Virtual fabric mapping to VLAN 36

37 Now with NPIV FCoE-FC NPIV Proxy Gateway Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection Mechanism Load balancing based on FLOGI Load balancing based on FDISC No exchange based load balancing Forwarding based on FC addresses FCoE-FCoE NPIV Proxy Gateway Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection Mechanism Load balancing based on FLOGI Load balancing based on FDISC No exchange based load balancing Forwarding based on FC addresses Effectively the same NPIV NPIV NPIV NPIV NPIV NPIV NPIV NPIV 37

38 FCoE-FC-Proxy-Gateway VF DCB DCB DCB DCB VF NPIV Proxy (FLOGI/FDISC->FDISC) Session based load balancing N Fabric 1 Local Area Network MAC(s) VF N VF VF N VF VF s Instantiated as required N VF NPIV Proxy (FLOGI/FDISC->FDISC) Session based load balancing F F F F Fabric 2 Optional Features Support LAG on the FCoE side VF could be n*10gbe s Trusted / untrusted modes ACL Protection only needed at server edge Ability to disable L2 switching to be a pure gateway End of Row / Against FC SAN Design Multiple 802.1p priorities & VLAN per Virtual Fabric To leverage DCB and Ethernet QoS for traffic separation Configurable FCload balancing by Enode/FLOGI or by FDISC 38

39 Future with FC-BB-6 / FDF All boxes do FC based forwarding Just as if they are s However a group share a domain id Hosted from the s above them The details are still be worked on Exact topologies HA models Scale models Intention is to minimize explosion of Domain ID Same intention as NPIV-Proxy Gateway Same results as using Layer 2 Clearly these are hops even though no E_s Pros All layer 3 FC Forwarding Full Knowledge of all hops Cons New set of protocols still in development Heterogeneous support issues? FDF Shared Domain ID FDF FDF FDF Shared Domain ID FDF FDF FDF FDF 39

40 Now with FCoE Transit Switch FCoE Transit Switch Core Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection Mechanism VE-VE ports created based on L2 visibility during FIP discovery Selection of ISL usual FC rules VE-VE could be a LAG with LAG rules FCoE Transit Switch Edge Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection Mechanism VN-VF ports created based on L2 visibility during FIP discovery VN-VF could be a LAG with LAG rules These hops matter L2 L2 Could could do both these things at the same time These hops matter L2 L2 L2 L2 40

41 Now with FCoE Transit Fabric FCoE Transit Core Fabric Adds tiers (or replaces them) But is not an FC hop But it IS a hop Path Selection Mechanism VE-VE ports created based on L2 visibility during FIP discovery Selection of ISL usual FC rules VE-VE could be a LAG with LAG rules FCoE Transit Edge Fabric Adds tiers (or replaces them) But is not an FC hop But it IS a hop Path Selection Mechanism VN-VF ports created based on L2 visibility during FIP discovery VN-VF could be a LAG with LAG rules These hops matter L2 L2 L2 Fabric L2 L2 L2 L2 Could could do both these things at the same time These hops matter L2 L2 L2 L2 Fabric L2 L2 L2 L2 L2 41

42 FCoE Transit Switch We all know what an L2 Ethernet Switch is (I hope) Forwards packets based on MAC Address That s all you need for a VN-VF or VE-VE connection The problem is that most standards are optional Lets ignore the fact that officially FCoE does not mandate what standards are supported just the net result From a PRACTICAL standpoint FCoE requires SOME of the standards that went in to the IEEE DCB committee PFC with at least one lossless lane DCBx & FCoE TLV (for link setup) FIP Snooping for security (optional section of FC-BB-5) 42

43 FCoE Transit Switch FCoE Server VN_ DCB FIP ACL DCB DCB FCoE Server VN_ DCB FIP ACL FCoE Transit Switch FIP Snooping Trusted connections FC/FCoE Switch DCB DCB FCoE Server VN_ DCB FIP ACL VF_ VF_ VF_ Ethernet & DCB 802.1p Level CoS & Monitoring VLAN based CoS 0xID in LAG, MC-LAG Age Discard may be considered Default / Native VLAN to control FC Dual Rail deployment FIP Snooping Enabled for switch and on a per VLAN basis (Vlan maps to virtual FC fabric) Programming of ACL to protect FCoE environment usually just at server edge Learn FCoE routes for monitoring FIP Heartbeat monitoring continuously verifies stability of the FCoE virtual connection 43

44 Future with FC-BB-6 / VN2VN All boxes do L2 based forwarding Just as if they are Transit Switches The L2 domain share a domain id Domain ID 0 (currently unused) Address discovery (think private loop) End to End FCoE only FIP now works from initiator to target Could be an L2 Fabric TRILL or other These hops matter L2 L2 L2 L2 L2 L2 Check out SNIA Tutorial: FCoE Direct End-Node to End-Node (aka VN2VN) L2 L2 L2 L2 44

45 By the way things are more complex We ignored a critical point What a given box does depends on the VLAN on which the packet arrived... 45

46 Converged or Unified Switch A given device could be doing multiple things Acting as an on 1 st VLAN Acting as an FCoE VN2VF Transit Switch on a 2 nd VLAN Acting as an FCoE-FC Gateway on a 3 rd VLAN Acting as an FCoE VN2VN Transit Switch on a 4 th VLAN And as for port types Could be pure FCoE Could be mixture of fixed FCoE and FC ports Could have multi-personality ports 46

47 One Ring to Rule them All Colors = VLANS (& Black Dashed = physical fibre channel) Fabric forwarding Fc-fc The same ports on the same device can be acting in completely different modes (forwarding based on different paradigms) on different VLANS, both on ingress and on egress this makes the rest of this presentation complicated! FCID FCID IP MAC/FCoE MAC/IP Converged Device FCoE-FC Gateway FCID over FC FCID IP MAC/FCoE MAC/IP L2 Switching L3/IP Routing L3 Forwarding 47

48 Summary All hops matter no matter they be forwarding based on primarily layer 2 or layer 3 It really doesn t matter if you think in terms of the device or the link just so long as you count all connections (aka the ISL is a poor counting method) In a fully converged world it may be easier to think in terms of the device being the hop as its technically more correct You don t have to be an FC Switch / to have an FC Stack and have good monitoring and manageability of FC or FCoE traffic 48

49 Summary Language will continue to evolve forever And for now so will viable solutions LAN DC LAN MAC-VPN FC SAN Pre-standards convergence FC-BB-5 convergence FC-BB-6 convergence 49

50 Q&A / Feedback Many thanks to the following individuals for their contributions to this tutorial. - SNIA Education Committee Simon Gordon Joseph White Samir Kumar Sharma Send any questions or comments on this presentation to SNIA: tracktutorials@snia.org 50