Cisco FabricPath Technology Introduction

Transcription

1 Cisco FabricPath Technology Introduction Marian Klas 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 1 Cisco Public 2 State of Existing Layer 2 Networks FabricPath Revolutionary Solution for Scalable, Highly- vailable Layer 2 FabricPath Forwarding Details FabricPath Design Consideration dvanced Features to integrate exiting L2/L3 networks Examples of FabricPath Use Case Conclusion 2011 Cisco and/or its affiliates. ll rights reserved. 1

2 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 3 Layer 2 or Layer 3? Both Layer 2 and Layer 3 are required for any network design Core ccess Layer 3 Network Layer 2? Layer 3? L3 L2 Subnet provide fault isolation Scalable control planes with inherent provision of multi-pathing and multi-topology H with fast convergence dditional loop-mitigation mechanism in the data plane (e.g. TTL, RPF check, etc.) Cisco has solutions for both Layer 2 and Layer 3 to satisfy Customers requirements Simplicity (no planning/configuration required for either addressing or control plane) Single control plane protocol for unicast, broadcast, and multicast Easy application development VLN VLN VLN VLN VLN VLN affiliates. VLN ll rights VLN 2011 Cisco and/or its reserved. Cisco Public 4 2

3 Maximize Bi-Sectional Bandwidth Scalable Layer 2 domain High vailability Resilient control-plane Fast convergence upon failure Fault-domain isolation Facilitate pplication Deployment Workload mobility, Clustering, etc. Multi-Pathing/Multi-Topology 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 5 Port Density on Switches Over-subscription Ratio Complex STP Configuration Data Center Core ggregation Primary vpc HSRP ctive Primary Root N VPC domain N R R R R R R R Secondary vpc HSRP Standby Secondary Root - R N Network port E Edge port - Normal port type B BPDUguard R Rootguard L Loopguard Layer 3 Layer 2 X-Chassis Port-Channel ccess L E B E B E B E B E B MC Table Size 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 6 3

4 More I/O slots More ports per I/O podule Higher Port-Density Wasted Bandwidth Higher Port Cost Limited Scale STP only allows single active link between 2 devices Multiple Inter-Switch Links Use interface with speed equals to the combination of multiple lower-speed links Higher Interface Speed More ports in a bundle (up to 16-port today) Port-Channel/Link-ggregation 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 7 11 Physical Links 5 Logical Links Branches of trees never interconnect (no loop!!!) Spanning Tree Protocol (STP) uses the same approach to build loop-free L2 logical topology Over-subscription ratio exacerbated by STP algorithm 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 8 4

5 Sub-optimal path selection Single path between any 2 bridges in the same L2 network Shortest path only from Root Bridge s perspective Under-utilized bandwidth Ensure loop-free L2 logical topologies by blocking redundant links Increased waste of available bandwidth as link-speed getting faster and faster No control plane security Root election purely based on switch-id, which is prone to problems caused by operator errors Slow and unreliable reconvergence upon link failure Up to seconds of service disruption even with RSTP 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 9 Transparent act like shared media to end devices Plug-N-Play No user configuration is required to build forwarding database Data plane learning Forwarding database built based on frame contents Flooding Default forwarding behavior for frames with unknown unicast destination is to flood the whole broadcast domain Every MC, Everywhere!!! ll unicast MCs need be learn by all bridges in the same bridge domain to minimize flooding MC Table MC Table Layer 2 Domain MC Table MC Table MC Table MC Table 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 10 5

6 MC v.s. IP Network ddress / /24 Non-hierarchical ddress Host ddress / / / / L2 Forwarding (Bridging) Data-plane learning Flat address space and forwarding table (MC everywhere!!!) Flooding required for unknown unicast destination Destination MCs need to be known for all switches in the same network to avoid flooding L3 Forwarding (Routing) Control-plane learning Hierarchical address space and forwarding Only forwarding to destination addresses with matching routes in the table Flooding is isolated within subnets No dependence on data-plane for maintaining forwarding table 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 11 What Can Be Improved? Network ddress Scheme: Flat Hierarchical dditional header is required to allow L2 Routing instead of Bridging Provide additional loop-prevention mechanism like TTL ddress Learning: Data Plane Control Plane Eliminate the needs to program all MCs on every switches to avoid flooding Control Plane: Distance-Vector Link-State Improve scalability, minimize convergence time, and allow multipathing inherently The ultimate solution needs to take both control and data plane into consideration this time!!! 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 12 6

7 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 13 Cisco FabricPath Data Plane Innovation Control Plane Innovation FabricPath encapsulation No MC learning via flooding Routing, not bridging Built-in loop-mitigation Time-to-Live (TTL) RPF Check Plug-n-Play Layer 2 IS-IS Support unicast and multicast Fast, efficient, and scalable Equal Cost Multipathing (ECMP) VLN and Multicast Pruning Cisco NX-OS Cisco Nexus Platform 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 14 7

8 Q: What s the latest status of the TRILL standard? : TRILL now officially moved from Draft to Proposed Standard in IETF RBridges: Base Protocol Specification (draft-ietf-trill-rbridge-protocol-16) Data plane, frame formats, learning, etc. RBridges: djacency (draft-ietf-trill-adj-07) IS-IS over shared media TRILL Use of IS-IS (draft-ietf-isis-trill-05) TRILL IS-IS TLV encodings Extensions to IS-IS for Layer-2 Systems (RFC 6165) rchitecture of IS-IS for L2 networks Proposed Standard status means vendors can confidently begin developing TRILL compliant software implementations Cisco and/or its affiliates. ll rights reserved. Cisco Public 15 Current Cisco implementation based on existing standards and modeled on proposed standards Easily migrated to industry standards in future Hardware already capable, software load will provide standards-based control-plane FabricPath 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 16 8

9 Nexus 7000 F-Series Module First FabricPath-capable hardware platform from Cisco Scalable to 512 ports per system High-performance 320/230 Gbps (switching/ backplane), 5µs latency Investment Protection Seamless Upgrade and Interoperability Standards Based TRILL and DCB support Flexible 1/10G ports auto-sensing Energy Efficient ~10W per 10GbE port The F-Series modules on the Cisco Nexus 7000 series are currently deployed in LLNL s high performance computing infrastructure, offering us a high density 10GE and low latency networking solution. This technology has enabled LLNL to build large storage network fabrics to support the world class supercomputing systems vital to the laboratory's national security research and development missions Matt Leininger, Deputy for dvanced Technology Projects at Lawrence Livermore National Laboratory 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 17 Encapsulation to creates hierarchical address scheme FabricPath header is imposed by ingress switch Ingress and egress switch addresses are used to make Routing decision No MC learning required inside the FabricPath Header S42 S11 C S11 S42 FabricPath Routing DT S11 STP FabricPath Domain S42 Ingress Switch C Egress Switch C DT C L2 Bridging STP Domain 1 STP Domain 2 C C 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 18 9

10 Plug-N-Play L2 IS-IS is used to manage forwarding topology ssigned switch addresses to all FabricPath enabled switches automatically (no user configuration required) Compute shortest, pair-wise paths Support equal-cost paths between any FabricPath switch pairs S1 S2 S3 S4 FabricPath Routing Table Switch S1 S2 S3 S4 S12 S42 L1 L2 L3 L4 L1, L2, L3, L4 L1, L2, L3, L4 L1 L2 L3 S11 L4 S12 S Cisco and/or its affiliates. ll rights reserved. Cisco Public 19 Forwarding decision based on FabricPath Routing Table Support more than 2 active paths (up to 16) across the Fabric Increase bi-sectional bandwidth beyond port-channel High availability with N+1 path redundancy S1 S2 S3 S4 Switch S42 L1, L2, L3, L4 MC 1/1 C S42 è C L1 L2 L3 S11 L4 S12 S42 1/1 è C 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 20 C 10

11 Forwarding through distinct Trees Several Trees are rooted in key location inside the fabric ll Switches in share the same view for each Tree Multicast traffic load-balanced across these Trees Root for Tree #1 Root for Tree #2 Root for Tree #3 Root for Tree #4 Ingress switch for FabricPath decides which tree to be used and add tree number in the header è M è M 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 21 C Minimize impact of transient loop with TTL and RPF Check Root STP Domain TTL=2 Root S1 S2 TTL=1 S10 M ç S2 TTL=3 TTL=0 Block redundant paths to ensure loop-free topology Frames loop indefinitely if STP failed Could results in complete network melt-down as the result of flooding TTL is part of FabricPath header Decrement by 1 at each hop Frames are discarded when TTL=0 RPF check for multicast based on tree info 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 22 11

12 VL10 VL20 VL30 VL10 VL30 Shared Broadcast Tree Switches indicate locally interested VLNs to the rest of the Broadcast traffic for any VLN only sent to switches that have requested for it VL20 VLN 10 VLN 20 VLN Cisco and/or its affiliates. ll rights reserved. Cisco Public Cisco and/or its affiliates. ll rights reserved. Cisco Public 24 12

13 Configuration used to determine if FabricPath should be run on an interface FabricPath Port Interfaces connected to another FabricPath Port Send/Receive traffic with FabricPath header No Spanning-Tree!!! No MC Learning Exchange topology info through L2 ISIS djacency Forwarding based on Switch Table Ethernet FP Header Classic Ethernet (CE) Port Interfaces connected to all existing NICs and Network Devices Send/Receive traffic in Ethernet frames format Participated in STP domain Forwarding based on MC Table STP Domain 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 25 Ethernet FabricPath Port CE Port FabricPath versus CE VLNs In FabricPath system, each VLN identified as either a CE VLN (default) or a FabricPath VLN CE VLN VLN Mode FabricPath VLN Only traffic in FabricPath VLNs can traverse FabricPath domain Bridging between M1 and ports possible only on CE VLNs M1 Ports Ports Ports n7k(config)# vlan 10 n7k(config-vlan)# mode? ce Classical Ethernet VLN mode fabricpath FabricPath VLN mode n7k(config-vlan)# mode 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 26 13

14 16-bytes header provide fields to help create hierarchical L2 address space and facilitate feature enhancements (Classical) Ethernet Frame DMC SMC 802.1Q Header Ether Type Payload CRC Cisco FabricPath Frame Outer D (48)* Outer S (48)* FP TG (32*) DMC SMC 802.1Q Header Ether Type Payload CRC (New) * Lengths for all fields are shown in bits EndNode_ID (5:0) I/G U/L EndNode _ID (7:6) RSVD (1)* OOO/ DL (1) Switch ID (12) Sub-Switch ID (8) Port ID (16) Ether Type Tree ID (10) TTL (6) Switch ID: Unique number assigned to help identify each device that is part of L2 Fabric Port ID: Used to provide information about MC-to-Interface association at L2 Fabric boundary Tree ID: Unique number assigned to help identify each distribution Tree TTL: Decrement at each hop to prevent frames looping infinitely in the fabric in case of unexpected failure 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 27 Control-Plane Protocol for FabricPath to Replace STP IS-IS Update IS-IS Update FabricPath Port CE Port Minimal knowledge required with no user configuration by default maintain PnP nature of Layer 2 Based on ISO/IEC Extensible protocol design allows Layer 2 info to be exchanged through IS-IS Single-level IS-IS with support for P2P links Calculate forwarding information for traffic forwarding Link-state protocol with support for ECMP improves failure detection, network reconvergence, and highavailability 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 28 14

15 Optimize Resource Utilization Learning only the MC addresses required 250 MCs 500 MCs 250 MCs 500 MCs MC MC B 2/1 STP Domain S11 MC B 500 MCs 250 MCs 500 MCs 250 MCs C C 3/1 S11 LL MCs needs to be learn on EVERY Switch Local MC: Source-MC Learning only happen to traffic received on CE Ports Large L2 domain and virtualization present challenges to MC Table scalability Remote MC: Source-MC for traffic received on FabricPath Ports are only learned if Destination-MC is already known as Local 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 29 Contains Information for Forwarding L2 Unicast Traffic inside the Switch S2 S16 S100 S200 S100 L1,, L101 L1,, L101 L1 L101 S1 S2 S16 L1 L2 L101 L16 S100 L102 FabricPath Port CE Port S200 L116 S200 C Known Unicast Destination One Ingress Switch è One Egress Switch Forwarding path selection based on destination Switch-ID inside FabricPath encapsulation imposed by Ingress Switch Each switch that is part of a calculate it s local Switch Table based on the information received from L2 IS- IS Switch Table basically contains information about {Switch-ID, Output Interfaces} Up to 16 Next-hop Interfaces (i.e. L2 ECMP) can be programmed for a given Switch-ID 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 30 15

16 Used for forwarding L2 multi-destination traffic (Unknown Unicast, Broadcast, and Multicast) inside the Tree # S100 S100 FabricPath Port CE Port S101 S105 S200 Root for Tree #1 S1 S2 S16 L1 1 L1, L101 L2 L101 L16 L102 L116 S200 C Tree topology is required to forward multi-destination traffic properly One Ingress Switch è Many Egress Switches Same method is also used by L3 (e.g. PIM Source Tree/Shared Tree) One or more Root devices are first elected for the Tree spanning from each Root is then formed and a network-wide unique ID is assigned to it Support for multiple Trees allows Cisco FabricPath to support multipathing even for multi-destination traffic Ingress Switch determines the Tree for each traffic flow 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 31 Step-by-Step Details 1. Host communicate to Host C for the first time. Send RP Request to C 2. S11 add into MC Table as the result of new source learning on CE Ports 3. Since destination MC is ll F, S11 flood this frame out all CE Ports 4. Meanwhile, S11 select Tree 1, marks this in FabricPath header and floods this frame out all FabricPath ports (L1 ~ L4) that are part of Tree 1 5. S1 flood this frame further (L5, L9) based on local info about Tree 1 6. S12 and S42 remove FabricPath header and flood the frame out all local CE Ports Tree # Tree # 1 L1, L2, L3, L4 MC 1/1 1 L1, L5, L9 S11 è FF (1) S1 S2 S3 S4 L1 L2 L3 L4 L6 L7 L8 L5 L12 S11 S12 L9 S42 è FF Encap è FF B C 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 32 L10 L11 MC 1/1 3/1 Decap Decap è FF MC No Learning on Remote MC since DMC is unknown FabricPath Port CE Port 16

17 Step-by-Step Details 1. Host C sends RP Reply back to Host 2. S42 adds C into MC Table from source learning on CE Port 3. Since is unknown, S42 floods this frame out all CE Ports 4. Meanwhile, S42 selects Tree 1, marks this in FabricPath header and floods this frame out all FabricPath ports (L9) that are part of Tree 1 5. S1 floods this frame further (L1, L5) along Tree 1 6. S11 floods this frame further (L2~L4) along Tree 1. lso, upon removing FabricPath header, S11 finds was learned locally. Therefore adds C as remote, associated with S42. Tree # 1 L1, L5, L9 S1 S2 S3 S4 L1 L2 L3 L4 L6 L7 L8 L5 L12 1 L9 Tree # S11 S12 L9 S42 1 L1, L2, L3, L4 MC HIT! MC 1/1 C S42 Decap FF ç S42 (1) ç C ç C B C 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 33 L10 FF ç S42 (1) ç C L11 MC 1/1 3/1 Encap Decap C è Tree # C 3/1 MISS FabricPath Port CE Port Step-by-Step Details 1. Host starts sending traffic to Host C after RP resolution 2. S11 finds C was learned as remote, associated with S42. Encap all subsequent frames to C with S42 as destination in FabricPath header 3. S11 s Routing Table indicates multiple paths to S42. Runs ECMP hash and selects L4 as next-hop 4. Routing Table lookup at S4 indicates L12 as next hop for S42 5. S42 finds itself as destination in FabricPath header and C is also known locally. Therefore, adds as remote, associated with S11. Switch S42 HIT! MC 1/1 C L1, L2, L3, L4 S42 Encap S1 S2 S3 S4 L1 L2 L3 L4 L6 L7 L8 L5 L12 S11 S12 L9 S42 è C è C S11 è S42 B C 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 34 L10 L11 1/1 3/1 Decap è C S11 è S42 Switch S42 L12 MC HIT! C 3/1 S11 FabricPath Port CE Port 17

18 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 35 Value-dd Enhancements 16-Way Equal Cost Multipathing (ECMP) at Layer 2 FabricPath Header Hierarchical ddressing with built in loop mitigation (RPF,TTL) Conversational MC Learning Efficient use of hardware resource by learning only MCs for interested hosts Interoperability with existing classic Ethernet networks VPC + allows VPC into a STP Boundary Termination Multi-Topology providing traffic engineering capabilities Cisco FabricPath Cisco FabricPath Up to 16-Way L2 ECMP Up to 16Way L2 ECMP 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 36 18

19 MC Table MC Table è??? B è S4 B S3 S3 S3 S1 B Payload S3 S2 B Payload S3 S4 B Payload S3 S4 B Payload vpc S1 S2 MC Table vpc+ S1 S2 MC Table B Payload B è S3 S4 B è S3 For Switches at Edge vpc is still required to provide active/active L2 paths for dualhomed CE devices or clouds However, MC Table only allows 1-to-1 mapping between MC and Switch ID B Payload Each vpc domain is represented by an unique Virtual Switch to the rest of Switch ID for such Virtual Switch is then used as Source in FabricPath encapsulation 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 37 VPC+ and ctive/ctive HSRP With VPC+ and SVIs in mixed-chassis, HSRP Hellos sent with VPC+ virtual switch ID FabricPath edge switches learn HSRP MC as reached through virtual switch Traffic destined to HSRP MC can leverage ECMP if available Either VPC+ peer can route traffic destined to HSRP MC HSRP ctive HSRP Standby DSID MC SVI SVI SSID 1000 S10 S20 S30 S40 DMC 0002 SMC HSRP Payload S1000 S100 FabricPath po1 po2 S200 1/30 MC MC B MC C 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 38 19

20 L3 FabricPath ctive Services Standby Services CE ctive po1 po2 Standby SVI GWY router MC HSRP M1 VPC+ M1 SVI GWY router MC S1 S2 GWY proxy L3 Services po1 GWY proxy L3 Services po1 L1 L2 po3 GWY L1,L2 Services L1,L2 GWY po3 Services po Cisco and/or its affiliates. ll rights reserved. Cisco Public 39 Integrating FEX with FabricPath With, requires VDCs with external crosslinks (M1 ports cannot belong to FabricPath VLNs) FabricPath CE L3 Same VLNs in FP mode in FP VDC Use ports for VDC interconnect M1 M1 M1 M1 M1 M1 VLNs in CE mode in FEX VDC 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 40 20

21 FEX with FabricPath Using F2 Modules With F2 modules, VDC requirement removed FabricPath CE L3 VLNs in FP mode F2 F2 F2 F2 F2 F2 F2 F2 FEX connected directly to F2 ports 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 41 FabricPath (L2 IS-IS) Classical Ethernet (STP) BPDU STP Domain 1 BPDU STP Domain 2 FabricPath Port CE Port is presented as a single bridge to all connected CE devices STP BPDUs are processed and terminated by CE Ports CE devices not interconnected will form separate STP domains Loops outside will be blocked within each STP domain should be the root for all connected STP domain. CE ports will be put into blocking state when superior BPDU is received 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public 42 21

22 No L2 IS-IS configuration required New feature-set keyword allows multiple conditional services required by FabricPath (e.g. L2 IS-IS, LLDP, etc.) to be enabled in one shot Simplified operational model only 3 CLIs to get FabricPath up and running N7K(config)# feature-set fabricpath N7K(config)# vlan N7K(config-vlan)# mode fabricpath N7K(config)# interface port-channel 1 N7K(config-if)# switchport mode fabricpath FabricPath Port CE Port 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public Cisco and/or its affiliates. ll rights reserved. Cisco Public 44 22

23 Building Large Scalable Compute Clusters Spine Switch 16 Chassis 8,192 10GE ports GE FabricPath ports per system 16-way ECMP 16-port Etherchannel Edge Switch FabricPath 32 Chassis 160 Tbps System Bandwidth (FDX) GE FabricPath Ports Open I/O Slots for connectivity HPC Requirements HPC Clusters require highdensity of compute nodes Minimal over-subscription Low server to server latency FabricPath Benefits for HPC FabricPath enables building a highdensity fat-tree network Fully non-blocking with FabricPath ECMP & port-channels Minimize switch hops to reduce server to server latencies 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public Cisco and/or its affiliates. ll rights reserved. Cisco Public 46 23

24 Standards-based No proprietary lock-in Plug-and-play Minimal configuration/complexity Optimal any-to-any connectivity Connect anywhere using an arbitrary topology, fabric uses the best path High-bandwidth High-performance modules/platforms with ample parallel bandwidth Resilient Routing-like convergence Scalable Easily grow the network based on business requirements Easy migration Doesn t follow the rip-and-replace model Simple administration Not black-box to network team FabricPath 2011 Cisco and/or its affiliates. ll rights reserved. Cisco Public Cisco and/or its affiliates. ll rights reserved. Cisco Public 48 24

25 25