The confusion arising from Converged Multihop Topologies as to the roles and capabilities of FCFs, FCoE-FC Gateways, and FCoE Transit Switches

Transcription

1 The confusion arising from Converged Multihop Topologies as to the roles and capabilities of s, FCoE-FC Gateways, and FCoE Transit Switches Simon J Gordon, Juniper Networks Author: Simon J Gordon, Juniper Networks

2 SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. 2

3 Abstract There is much information on the various approaches for convergence, the different types of products that can be used, and the implications for a practical deployments. Some of this information is out of date, some product specific, and some just informal internet discussion. In addition there is an overloading of terminology that can further lead to confusion. During this session, Simon will explain clearly and precisely each of the different types of product, the possible capabilities of such products, where they sit within the most typical deployments. He will also examine the implications of this on SAN and LAN administration and operation. Finally, Simon will discuss some of the key changes that arise as the industry moves from the current FC-BB-5 standard to the FC-BB-6 standard which is under development. 3

4 See Convergence in Action Unified Storage Infrastructure The best way to understand what it really means to you is go see it and touch it 4

5 Agenda Part 1 (we will go fast) Two nations divided by a common language Who said that? Ethernet & IP History Lesson Tiers & Hops L2 Switches, L3 Routers, and L2/L3 Switches TRILL Fabrics Fibre Channel History Lesson Once upon a time Now with NPIV, WDM & FCIP Not to mention Virtual Fabrics & Inter Fabric Routing 5

6 Agenda Part 2 (we will go slow) FCoE with FC-BB-5 Roles of different devices Capabilities of different devices Future with FC-BB-6 6

7 Agenda Part 1 Two nations divided by a common language Who said that? Ethernet & IP History Lesson Tiers & Hops L2 Switches, L3 Routers, and L2/L3 Switches Fibre Channel History Lesson Once upon a time Now with NPIV, WDM, and FCIP Not to mention Virtual Fabrics & Inter Fabric Routing 7

8 Refer to the SNIA Dictionary Languages evolve Slowly within isolated communities Battle between simplicity and expressiveness Rapidly when communities interact or combine As is happening with network convergence Not to mention the influence of fashion and even individual idiosyncrasies Individuals, companies, organizations Electrical, mechanical, software terms This can be confusing at times Check out the SNIA Dictionary 8

9 Several non vendor communities T11 9

10 Agenda Part 1 Two nations divided by a common language Who said that? Ethernet & IP History Lesson Tiers & Hops L2 Switches, L3 Routers, and L2/L3 Switches Fibre Channel History Lesson Once upon a time Now with FCIP, ifcp, and NPIV Not to mention Virtual Fabrics & Inter Fabric Routing 10

11 Tiers & Hops in a Network Multi-tier network With a pure L2 network every hop is an L2 hop so we know the exact L2 hop count for any path from source to destination L3 Boundary Core Router Tier 3 Tier 2 - Aggregation 3 Tiers of L2 Switching Tier 1 - Access Servers From server to server may be 1, 3, or 5 L2 hops depending on relative location L3 Router L2/L3 Switch L2 Switch 11

12 Tiers & Hops in a Network Multi-tier network Typically customers deploy with a defined L2/L3 boundary at a fixed layer/tier in the network. Now there is a mixture of L2 and L3 hops for some paths Core Router L3 Boundary Tier 3 L3 boundary Tier 2 - Aggregation Tier 1 - Access 2 Tiers of L2 Switching 3 Tiers of Switching Servers From server to server may be 1, 3, or 5 hops but may be a mixture of L2 and L3 hops In addition firewalls may be deployed complicating the picture (not shown here) L3 Router L2/L3 Switch L2 Switch 12

13 Tiers & Hops in a Network Multi-tier network As the L2/L3 boundary moves lower the exact mix of hops changes with more paths having a mixture of L2 and L3 hops Core Router Tier 3 L3 Boundary Tier 2 - Aggregation Tier 1 - Access 1 Tier of L2 Switching 3 Tiers of Switching Servers From server to server may be 1, 3, or 5 hops but may be a mixture of L2 and L3 hops L3 Router L2/L3 Switch L2 Switch 13

14 Tiers & Hops in a Network Multi-tier network As the L3 boundary moves to the access layer all paths having a mixture of L2 and L3 hops Core Router Tier 3 Tier 2 - Aggregation 3 Tiers of L2/L3 Switching Tier 1 - Access L3 Boundary Servers From server to server may be 1, 3, or 5 hops but may be a mixture of L2 and L3 hops In reality L3 could be anywhere though arbitrary placement makes a network hard to manage L3 Router L2/L3 Switch L2 Switch 14

15 Tiers & Hops in a Network Multi-tier network Core Router Tier 3 Tier 2 Tier 1 - Aggregation 4 Tiers Blade Switches may add another tier From virtual server to virtual server may even be 7 or 9 hops in total!!! Blade switches not only add another tier but of course could be L2 or L3 in nature L3 Router L2/L3 Switch L2 Switch 15

16 Tiers & Hops in a Network Multi-tier network Core Router Tier 3 Tier 2 Tier 1 - Aggregation 5 Tiers Blade Switches may add another tier From virtual server to virtual server may even be 7 or 9 hops in total!!! Virtual Switches may add another tier Virtual switches with or without NIC offload again add tiers to the network L3 Router L2/L3 Switch L2 Switch 16

17 Tiers & Hops in a Network Multi-tier network Whilst all forms of multipathing (IEEE 802.1aq, TRILL, MC LAG) can remove the effect of loops, they do not solve the problem of hops and tiers Tier 3 the number of tiers is the same L2 multi pathing Tier aq the decision on location of L3 TRILL Fabric Tier 1 - Aggregation the number of hops is the same default router still exists Path Selection has changed and packets are encapsulated as they traverse the fabric From server to server may be 1, 3, or 5 hops L3 Router L2/L3 Switch L2 Switch 17

18 Ethernet & IP Summary Tiers The definition is clear Hops Layers of switching / routing boxes in a hierarchical structure Generic term independent of protocol based on topology The box takes ingress packets and makes a path selection Specific meaning for each and every protocol L2 Hop (reset across L3 boundary) L3-IP Hop Fabric Hop (routing of encapsulated packets) Path Selection depends 18

19 Agenda Part 1 Two nations divided by a common language Who said that? Ethernet & IP History Lesson Tiers & Hops L2 Switches, L3 Routers, and L2/L3 Switches Fibre Channel History Lesson Once upon a time Now with NPIV, WDM, and FCIP Not to mention Virtual Fabrics & Inter Fabric Routing 19

20 Once upon a time Many topologies discussed Even tested and deployed A hop was a hop was a hop Process of traversing an fcf Gradually settled on core/edge Looks like a 2 tier network with some form of multipathing Dual Rail is assumed but not shown to simplify the diagrams E_Port E_Port N_Port F_Port F_Port N_Port 20

21 Once upon a time core/edge 2 hops from server to storage 2 tier topology edge/core/edge 3 hops from server to storage 3 tier topology Path Selection Forwarded based on FC address Session based (early days) Exchange based (later on) Every switch had all switch services Dual Rail is assumed but not shown to simplify the diagrams 21

22 Once upon a time edge/core/edge Is it really a 3 tier network Or is it just a 2 tier network in disguise Its just a matter of perspective Dual Rail is assumed but not shown to simplify the diagrams 22

23 Traditional FC Summary Tiers Most FC SANs were 2 tier (or 3 depending on definition) Hops Every box in the network was an and so an FC Hop Path Selection based on source/destination FCIDs Increasingly sophisticated multipathing box to box Increasingly sophisticated multipathing end to end Most FC SANs had 2 or 3 hops from server to storage Every switch in the network was an switch in the classic sense of the term from a Fibre Channel perspective 23

24 Now with NPIV FC-FC NPIV Proxy Gateway Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection (GW-) Load balancing based on FLOGI Load balancing based on FDISC No exchange based load balancing Forwarding based on FC addresses FC Services Present An FC Stack but not acting as a full switch Fabric Facing Presents as a server (N ports) Server Facing Present as an (F Ports) E_Port E_Port Did you know NPIV Gateway FLOGI s the fabric N_Port-F_Port NPIV GW maps Server FLOGI to an FDISC VF_Ports spawned from the F_Port Server N_Port maps to an VF_Port F_Port N_Port F_Port N_Port NPIV NPIV NPIV NPIV 24

25 Now with WDM WDM No forwarding decision WDM Is not an FC hop Is not a hop of any type FC Services Present None In the optical network May be multiple boxes Do not count at the FC level FC rules must be considered E_D_TOV R_A_TOV R_T_TOV FC Services Present None E_Port WDM WDM E_Port 25

26 Now with FCIP FCIP Gateway Is making forwarding decisions FCIP Gateway Is not an FC hop but it is a hop Indeed if it is also doing routing then the FC hop count is reset or rather not counted across the boundary FC Services Present An FC Stack In between the FCIP Gateway Many L2 Ethernet hops Many L3 IP hops Do not count at the FC level FC rules must still be considered E_D_TOV R_A_TOV R_T_TOV FC Services Present None FCIP FCIP 26

27 Not to mention VFT and IFR 3 FC Hops in total 2 FC Hops in VF1 2 FC Hops in VF1 FCID VF1 Converged Device Converged Device Converged Device FCID VF2 FCID VF1 VF1 VF1/VF2 VF2 FCID VF2 L3 Forwarding L3 Forwarding Plus IFR L3 Forwarding 27

28 Pre FCoE Summary Tiers The definition is clear Hops Layers of boxes in a hierarchical structure Generic term independent of protocol based on topology The box takes ingress packets and makes forwarding decision Specific meaning for different types of box FC Hop NPIV Proxy Hop FCIP Gateway Hop Ethernet L2 and IP L3 Hops between gateways 28

29 Agenda Part 2 FCoE with FC-BB-5 What can I do now What are the implications Future with FC-BB-6 29

30 And Now FCoE not a lesson FIP/VLAN Discovery happens using untagged packets Uses the FIP Ethertype Native VLAN can be used to control visibility during the discovery phase (separate discussion) Discovers which VLANs contain FCoE s (or gateways) Establish & Maintains our VE-VE FCoE ISL or VN-VF connection Maintains there s protocol stuff to keep the connection working Did you notice have we overloaded the term VF_Port? Once we have actual FCoE FC sessions running Uses a different Ethertype Run within the actual VLANS that correspond to actual Virtual Fabrics using VLAN tagged packets FIP keep alives running, connection recovery mechanisms 30

31 Key Device Types Fibre Channel Forwarder or Switch (FC/FCoE) Fixed or configurable ports, any mix possible May also be an L2/L3 switch on Ethernet ports NPIV Proxy Gateway FCoE-FC FCoE-FCoE May also be an L2/L2 switch on Ethernet ports Ethernet Switch or FCoE Transit Switch L2 forwarding of FCoE frames plus some bits 31

32 FCoE Based FC SAN all If every box is an FCoE Every box is an FC Hop Same number of hops for any case Please Please Please Please The correct term is still FC Hop Hop is to do with crossing the Hop is not to do with the link layer Forwarding Mechanism Session based Exchange based Forwarded based on FC address Path Selection Mechanism Exactly the same as native FC At least in terms of selecting which virtual port (VN/VF/VE) to use Ethernet LAGs may exist beneath the virtual port layer VN_Port NOTE: Ignoring virtual fabrics for now Ignoring combined /L2 Switch New FC port setup mechanism used to establish virtual ports in theory across an arbitrary L2 network (VE- VE, VF-VN, and later with FC-BB-6 VN-VN) though in this case looks no different to before VF_Port VN_Port VE_Port VE_Port VF_Port 32

33 Fibre Channel Switch or Forwarder Forwards packets/frames based on L3 address When a frame or packet is received on an ingress port the DID is used to decide what egress port to transmit Has all the usual FC Services and has its own domain id With Convergence May have FC, FCoE, and/or multi personality ports May also implement L2-Ethertnet switching on FCoE ports Implication regarding FCoE Forwarding Minimum Features Same as normal FC Switch Must used enhanced Ethernet that meets the minimum set of FCoE requirements 33

34 Fibre Channel Switch or Forwarder Now lets define other possible capabilities Trusted / untrusted modes depending on topology ACL Protection only needed at server edge Support LAG on the FCoE side VF Port could be n*10gbe Ports Virtual Port Scaling Number of Vx ports per physical port Ability to disable L2 switching to be a pure gateway Eor / Against FC SAN Design Variation in level of DCB implementation PFC, ETS, DCBx have mandatory and optional capabilities To leverage DCB and Ethernet QoS for traffic separation 34

35 Now with NPIV FCoE-FC NPIV Proxy Gateway Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection Mechanism Load balancing based on FLOGI Load balancing based on FDISC No exchange based load balancing Forwarding based on FC addresses FCoE-FCoE NPIV Proxy Gateway Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection Mechanism Load balancing based on FLOGI Load balancing based on FDISC No exchange based load balancing Forwarding based on FC addresses NPIV NPIV NPIV NPIV NPIV NPIV NPIV NPIV 35

36 Future with FC-BB-6 / FDF All boxes do FC based forwarding Just as if they are s However a group share a domain id Hosted from the s above them The details are still be worked on Exact topologies HA models Scale models Intention is to minimize explosion of Domain id Same intention as NPIV-Proxy Gateway Same results as using Layer 2 FDF Shared DID FDF FDF FDF FDF FDF Shared DID FDF FDF 36

37 FCoE-FC Gateway (and the rest) Now lets define other possible capabilities Configurable load balance on the SAN Side by Enode/FLOGI or by session/fdisc Trusted / untrusted modes depending on topology ACL Protection only needed at server edge Support LAG on the FCoE side VF Port could be n*10gbe Ports Ability to disable L2 switching to be a pure gateway End of Row / Against FC SAN Design Multiple 802.1p priorities & VLAN per Virtual Fabric To leverage DCB and Ethernet QoS for traffic separation 37

38 Now with FCoE Transit Switch FCoE Transit Switch Core Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection Mechanism VE-VE ports created based on L2 visibility during FIP discovery Selection of ISL usual FC rules VE-VE could be a LAG with LAG rules FCoE Transit Switch Edge Adds another tier (or replaces one) But is not an FC hop But it IS a hop Path Selection Mechanism VN-VF ports created based on L2 visibility during FIP discovery VN-VF could be a LAG with LAG rules L2 L2 NPIV NPIV NPIV NPIV L2 L2 L2 L2 38

39 Future with FC-BB-6 / VN2VN All boxes do L2 based forwarding Just as if they are Transit Switches The L2 domain share a domain id Domain ID 0 (currently unused) Address discovery (think private loop) End to End FCoE only FIP now works from initiator to target L2 L2 L2 L2 L2 L2 Check out SNIA Tutorial: FCoE Direct End-Node to End-Node (aka VN2VN) L2 L2 L2 L2 39

40 FCoE Transit Switch We all know what an L2 Ethernet Switch is (I hope) Forwards packets based on MAC Address Each FCoE Virtual Port gets its own MAC Addresses VN, VF, VE The problem is that most standards are optional Lets ignore the fact that officially FCoE does not mandate what standards are supported just the net result From a PRACTICAL standpoint FCoE requires SOME of the standards that went in to the IEEE DCB committee 40

41 FCoE Transit Switch First lets define the minimum practical set Some sensible set of basic L2 capabilities Not going to define and may depend on topology / use PFC with at least one lossless lane Ensures FCoE is lossless from a congestions standpoint without cross channel interference with non FCoE traffic on other lanes DCBx mandated by PFC FIP Snooping Technically optional but generally accepted as a sensible precaution to provide appropriate protection to the SAN 41

42 FCoE Transit Switch Now lets define other possible capabilities Layer 2 Multipathing Various mechanisms available LAG Properties Inclusion of FC level source and destination in LAG Hash Use of Exchange ID in the LAG for Exchange based load balancing 42

43 FCoE Transit Switch Now lets define other possible capabilities Good DCB Monitoring would be useful Ability to see traffic by 802.1p user priority maybe even by VLAN CoS/QoS Would be Good ETS to control bandwidth for groups of user priorities Traditional CoS/QoS to control bandwidth by 802.1p user priority or even fine grained queuing by VLAN FIP Snooping By definition you learn the routes of the Vx-Vx connections through the L2 layer and can make this information visible You could also monitor the FIP keep alives on these sessions Age Discard s perform age discard, should an L2 switch? 43

44 Summary We ignored something 44

45 Converged or Unified Switch A given device could be doing multiple things Acting as an on one VLAN Acting as an FCoE Transit Switch on a second VLAN Acting as an FCoE-FC Gateway on a third VLAN And as for port types Could be pure FCoE Could be mixture of fixed FCoE and FC ports Could have multi-personality ports 45

46 One Ring to Rule them All Colors = VLANS (& Black Dashed = physical fibre channel) Fabric forwarding Fc-fc The same ports on the same device can be acting in completely different modes (forwarding based on different paradigms) on different VLANS, both on ingress and on egress this makes the rest of this presentation complicated! FCID FCID IP MAC/FCoE MAC/IP Converged Device FCoE-FC Gateway FCID over FC FCID IP MAC/FCoE MAC/IP L2 Switching L3/IP Routing L3 Forwarding 46

47 Summary Language will continue to evolve forever And for now so will viable solutions LAN DC LAN MAC-VPN FC SAN Prestandards convergence FC-BB-5 convergence FC-BB-6 convergence 47

48 Q&A / Feedback Please send any questions or comments on this presentation to SNIA: tracktutorials@snia.org Many thanks to the following individuals for their contributions to this tutorial. - SNIA Education Committee Simon Jonathan Gordon Joseph L White Samir Kumar Sharma 48