Data Centre Interconnect with OTV and Other Solutions

Transcription

1

2 Data Centre Interconnect with and Other Solutions David Jansen CCIE#5952 Distinguished Systems Engineer

3 Session Abstract: This session features a detailed analysis of the architectural aspects, implementation details and deployment benefits behind the Overlay Transport Virtualisation () technology recently introduced by Cisco. The attendees will learn how, an industry first solution, significantly simplifies Data Centre Interconnect (DCI) deployments by extending Ethernet LANs between multiple sites over any network, making multiple data centres look like one logical data centre. The attendees will learn how is aimed at providing Layer 2 connectivity beyond the Layer 3 boundary while maintaining the scalability, failure containment and operational simplicity that the Layer 3 boundary provides. involves foundational changes to the learning and forwarding principles of traditional VPN technologies. is a "MAC in IP" technology where the MAC address reachability information is conveyed in a control protocol. The architecture is discussed in detail, giving the attendee a clear understanding of the technical aspects of the technology. The multiple benefits achieved by are discussed in detail during the session. Some of the improvement areas examined in the discussion include core transparency, multi-homing, loop prevention, failure isolation, high availability, bandwidth optimisation, etc. This session does NOT cover the technical aspects of the traditional Layer 2 VPN technologies such as EoMPLS, VPLS, etc. The session will compare and VXLAN technologies for DCI as well as design guidance for both. Target Audience: Those responsible for the Design, Deployment, Operations, and Management of SP, Enterprise and Data Centre Networks will find this session informative and useful.

4 Cisco Spark Ask Questions, Get Answers, Continue the Experience Use Cisco Spark to communicate with the Speaker and fellow participants after the session Download the Cisco Spark app from itunes or Google Play 1. Go to the Cisco Live Melbourne 2017 Mobile app 2. Find this session 3. Click the Spark button under Speakers in the session description 4. Enter the room, room name = Session ID (speaker to change) 5. Join the conversation! The Spark Room will be open for 2 weeks after Cisco Live

5 Introduction Control Plane and Data Plane Failure Isolation Multi-homing Mobility QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

6 Overlay Transport Virtualization () is a MAC-in-IP method that extends Layer 2 Single touch site configuration Transport independence High Resiliency Flexible encapsulation (IP-GRE or IP-UDP) Failure domain isolation Control-plane learning Seamless Multi-homing Dynamic neighbor discovery Maximises available bandwidth Native multi-homing Ethernet in IP MAC routing Spanning Tree Protocol isolation Multi-Data Centre scalability Unknown unicast flooding isolation Simplified Configuration & Operation Broadcast traffic separation Seamless overlay - no network re-design Multicast transport optimisation Address Resolution Protocol (ARP) optimisation Nexus 7000 Verified Scale Limits Note: Also supported on the ASR 1000

7 What is NOT transported over? Spanning-tree Unknown Unicast HSRP (when filtered) CDP LACP IGMP VTP?

8 Terminology Devices and Interfaces Edge Device Performs all functionality Usually located at the Aggregation Layer or at the Core Layer Support for multiple Edge Devices (multi-homing) in the same site Internal Interface Site facing Interfaces of the Edge Devices Carry VLANs extended through Regular Layer 2 interfaces No configuration required Supports IPv4 & IPv6 Edge Device Internal Interfaces Edge Device Core Device Aggregation Device Internal Interface Join Interface Overlay Interface

9 Terminology Devices and Interfaces Join Interface Uplink of the Edge Device Point-to-point routed interface (physical interface, subinterface or port-channel supported; now loopback as well) Used to physically join the Overlay network No specific configuration required IPv4 only Overlay Interface Virtual interface with most of the configuration Logical multi-access multicast-capable interface Encapsulates Layer 2 frames in IP unicast or multicast Overlay Interface Edge Device Join Interface Core Device Aggregation Device Internal Interface Join Interface Overlay Interface

10 Interfaces Nexus 7000 Hardware Support F3 Support for in 6.2(6) Enable on Nexus 7700 Series Utilize port-level VLAN Translation on F3 F1 and F2e support for internal Interface F1 and F2e Linecards have the ability to be internal interfaces when M series Linecard is used for M3 Support for in7.3(0)dx(1) Edge Device Core Device Aggregation Device Internal Interface Join Interface Overlay Interface

11 NX-OS Best Practices For Your Reference Best Practices Configure the site-vlan on the same switchports as the extended vlans. This will ensure that if site-vlan goes down, so will extended-vlans. edge devices in same site must be configured with same site-id Extended VLANs should be symmetric across all edge devices. The overlay interface number should be same for the same VPN Overlay network for Adjacency Server. SVI interfaces should not be created on the extended vlans. Use a dedicated VDC if SVI is present Only Extend the VLANs needed; not everything Use the same site-vlan per site, do NOT extend site-vlan on the Overlay Deploy Edge device in pairs at each Data Centre For Unicast-mode (Adj Server); it is recommend to not exceed three Data Centres Notes With NX-OS and higher Do not extend the sitevlan Recommended best practice for FHRP isolation

12 NX-OS Best Practices For Your Reference For fast site peer failure detection: Enable BFD between SVIs for site VLAN Enable Routing Protocols (no static routes) Best Practices Remember: The Nexus 7000 does not support fragmentation NXOS Notes need full MTU (42bytes) along the path

13 Control Plane Building the MAC Tables Prevents unknown unicast flooding selective unicast flooding starting NX-OS 6.2 Control Plane Learning with proactive MAC advertisement Background process with no specific configuration IS-IS used between Edge Devices MAC Addresses Advertisements West IP A IP B East IP C South

14 Control Plane Neighbor Discovery and Adjacency Formation Before any MAC address can be advertised the Edge Devices must: Discover each other Build a neighbor relationship with each other Neighbor Relationship built over a transport infrastructure: Multicast-enabled (all shipping releases) Unicast-only (from NX-OS release 5.2 & IOS-XE 3.9)

15 Control Plane Neighbor Discovery (over Multicast Transport) Control Plane Multicast-enable Transport Control Plane West IP A IP B East Mechanism Edge Devices (EDs) join an multicast group in the transport, as they were hosts (no PIM on EDs) hellos and updates are encapsulated in the multicast group End Result Adjacencies are maintained over the multicast group A single update reaches all neighbors

16 Multicast Transport Control and Data Plane over Multicast Transport Use a High-Available Multicast Rendez-Vous Point (RP) configuration PIM Anycast (RFC4610) or MSDP (Multicast Source Discovery Protocol) Requirements to Control Plane PIM Any-Source-Multicast (ASM) Sparse-Mode Requirements to Data Plane PIM Source-Specific-Multicast (SSM)

17 Control Plane (Multicast Transport) 3 Neighbor IP Addr Neighbor West IP Addr IP A Hello Control Plane 7 Hello Control Plane 4 Hello IP A è G West IP A Multicast-enabled Transport IGMP Join G IP B Hello IP A è G Hello East 6 Encap IGMP Join G Decap 1 All edge devices join control-group G 2 Multicast state for group G established throughout transport Hello IP A è G Hello IP A è G IGMP Join G Decap IP C 6 5 Transport natively replicates multicast to all OIFs Hello IP A è G Control Plane 7 Hello Neighbor IP Addr West IP A South

18 Control Plane (Multicast Transport) Hello 5 Control Plane Neighbor South IP Addr IP C Bidirectional adjacency formed Neighbor IP Addr West IP A South IP C 5 Hello Control Plane Hello West IP C è G IP A Multicast-enabled Transport IP B Hello 4 Decap Decap 4 East IP C è G Hello Hello IP C è G IP C è G 3 The South Site creates its hello with West s address in the TLV Encap IP C 2 Hello IP C è G Control Plane 1 Hello Neighbor IP Addr West IP A South

19 Control Plane MAC Advertisements (over Multicast Transport) Craft 2update with new MACs Update A VLAN MAC IF 100 MAC A IP A 100 MAC B IP A 100 MAC C IP A Update A 6 Update A IP A è G West 3 MAC Table Encap VLAN MAC IF 100 MAC A e1/ MAC B e1/ MAC C e1/1 Multicast-enabled Transport Update A Update A IP A è G IP A è G 4 5 Update A Decap East IP A è G MAC Table VLAN MAC IF 100 MAC A IP A 101 MAC B IP A 102 MAC C IP A 1 New MACs learned in VLANs that are extended Decap 5 Add MACs learned through 7 Update A IP A è G 6 VLAN MAC IF 100 MAC A IP A 100 MAC B IP A 100 MAC C IP A Update A South MAC Table VLAN MAC IF 100 MAC A IP A MAC B IP A Cisco and/or its affiliates. through All rights 102 MAC C IP A reserved. Cisco Public 7 Add MACs learned

20 Multicast NXOS Best Practices For Your Reference Multicast Best Practices Multicast routing is turned on by default in NX-OS, so there is no need for the ip multicast-routing Notes RP redundancy use IP PIM Anycast RP or MSDP If using auto-rp, need ip pim auto-rp listen forward needs to be applied in order to accept and forward the Auto-RP messages Configure Static anycast-rp addresses 1) For Multicast routing with vpc/vpc+ use PIM-SM 2) For Multicast routing with vpc+ PIM-SSM is supported with and higher 3) PIM-BiDir is not supported with vpc / vpc+ 4) PIM SSM over vpc (Nexus 9000) Note: F2 does not support BiDir PIM (F2e does)

21 Multicast Example NXOS Best Practices Anycast-RP 1: feature pim feature eigrp feature bfd interface loopback0 ip address /32 ip router eigrp 10 ip pim sparse-mode interface loopback1 ip address /32 ip router eigrp 10 ip pim sparse-mode router eigrp 10 bfd ip pim rp-address group-list /4 ip pim ssm range /8 ip pim bfd ip pim anycast-rp ip pim anycast-rp Anycast-RP 2: feature pim feature eigrp feature bfd interface loopback0 ip address /32 ip router eigrp 10 ip pim sparse-mode interface loopback1 ip address /32 ip router eigrp 10 ip pim sparse-mode router eigrp 10 bfd ip pim rp-address group-list /4 ip pim ssm range /8 ip pim bfd ip pim anycast-rp ip pim anycast-rp

22 Control Plane Neighbor Discovery (Unicast-only Transport) Ideal for connecting a small number of sites With a higher number of sites a multicast transport is the best choice Control Plane Unicast-only Transport Control Plane West IP A IP B East Mechanism Edge Devices (EDs) register with an Adjacency Server ED EDs receive a full list of Neighbors (onl) from the AS hellos and updates are encapsulated in IP and unicast to each neighbor End Result Neighbor Discovery is automated by the Adjacency Server All signaling must be replicated for each neighbor Data traffic must also be replicated at the head-end

23 Control Plane CLI Verification Establishment of control plane adjacencies between Edge Devices (multicast or unicast transport): dc1-agg-7k1# show otv adjacency Overlay Adjacency database Overlay-Interface Overlay100 : Hostname System-ID Dest Addr Up Time Adj-State dc2-agg-7k1 001b.54c2.efc :08:53 UP dc1-agg-7k2 001b.54c2.e1c :43:27 UP dc2-agg-7k2 001b.54c2.e :49:11 UP Unicast MAC reachability information: dc1-agg-7k1# show otv route Unicast MAC Routing Table For Overlay100 VLAN MAC-Address Metric Uptime Owner Next-hop(s) c07.ac01 1 3d15h site Ethernet1/ d70e 1 3d15h site Ethernet1/ f3.88ff 42 2d22h overlay dc2-agg-7k f d22h overlay dc2-agg-7k2 Local Site MAC Remote Site MAC

24 Data Plane Inter-Site Packet Flow 2 Layer 2 Lookup MAC TABLE VLAN MAC IF 100 MAC 1 Eth 2 4 Transport Infrastructure 100 MAC 2 Eth 1 Encap 100 MAC 2 IP A MAC 1 è MAC 3 IP A è IP B 100 MAC 3 IP B MAC 1 è MAC 3 IP A èip B 100 MAC 3 Eth MAC 4 IP B IP A 3 Decap IP B 5 MAC TABLE VLAN MAC IF 100 MAC 1 IP A 100 MAC 4 Eth 4 6 Layer 2 Lookup MAC 1 è MAC 3 1 West Site East Site Server 1 Server 3 7 MAC 1 è MAC 3

25 Data Plane 1.0 Encapsulation 42 Bytes overhead to the packet IP MTU size (IPv4 packet) Outer IP + Shim - Original L2 Header (w/out the.1q header) 802.1Q header is removed and VLAN field copied over to the shim header Outer shim header contains VLAN, overlay number, etc. Consider Jumbo MTU Sizing 802.1Q header removed 802.1Q 802.1Q DMAC SMAC Etype Payload CRC Original Layer 2 Frame Classic Ethernet Frame 1.0 Frame Outer MAC Outer IP Shim DMAC SMAC Etype Payload CRC (new) 14B 20B 8B 14B 4B 20B + 8B + 14B* = 42 Bytes of total overhead * The 4 Bytes of.1q header have already been removed

26 Tunnel Depolarization 1.0 Encapsulation only Secondary IP command introduced Configured within join-interface, not interface Introduction of multiple IPs results in tunnel depolarization 3 secondary IPs supported -a(config-if)# ip address /24 secondary Disabling IP Redirects on port-channel11 :secondary address configured. -a(config-if)# show run interface Ethernet1/19!Command: show running-config interface Ethernet1/19!Time: Wed Mar 27 23:05: version 6.2(2) interface Ethernet1/19 no ip redirects ip address /24 ip address /24 secondary ip ospf network point-to-point ip router ospf 1 area ip igmp version 3 -a (config-if)# sh otv Overlay Information Site Identifier Overlay interface Overlay1 VPN name : Overlay1 VPN state : UP Extended vlans : (Total:182) Control group : Data group range(s) : /24 Broadcast group : Join interface(s) : Ethernet1/19 ( ) Secondary IP Addresses: Site vlan : 1 (up) AED-Capable : Yes Capability : Multicast-Reachable 26

27 Data Plane Encapsulation Choice Nexus 7x00 NX-OS 7.2 Default Encapsulation is IP GRE Encapsulation format can be configured per VDC / Site UDP encapsulation format requires F3/M3-Linecards Encapsulation should be consistent across all DCI sites -a(config)# otv encapsulation-format ip {gre udp} # Allows to use either IP GRE or IP UDP encapsulation.

28 Data Plane 2.5 Encapsulation Nexus 7x00 NX-OS Bytes overhead to the packet IP MTU size (IPv4 packet) Outer IP + UDP + - Original L2 Header (w/out the.1q header) 802.1Q header is removed and VLAN field copied over to the shim header Outer shim header contains VLAN, overlay number, etc. Consider Jumbo MTU Sizing 802.1Q header removed 802.1Q 802.1Q DMAC SMAC Etype Payload CRC Original Layer 2 Frame Classic Ethernet Frame 2.5 Frame Outer MAC Outer IP UDP DMAC SMAC Etype Payload CRC (new) 14B 20B 8B 8B 14B 4B 20B + 8B +8B + 14B* = 50 Bytes of total overhead * The 4 Bytes of.1q header have already been removed

29 Data Plane 2.5 Encapsulation Nexus 7x00 NX-OS Bytes of Overhead Outer MAC Header Outer IP Header UDP Header Header Original Layer-2 Frame Underlay Overlay Original IETF draft of IETF draft-hasmit-otv-04 Header format is bit by bit VXLAN (RFC 7348) Uses Tunnel Depolarization UDP encapsulation utilize varying UDP source port ( UDP port 8472 and VXLAN UDP port 4789) Requires Nexus 7000/7700 M3/F3-Series Linecard Header VXLAN Flags RRRRIRRR 8 Reserved Overlay ID / Instance ID Bytes UDP Header Source Port Port (UDP 8472) UDP Length Bytes Reserved 8 Checksum 0x

30 and UDP Nexus 7x00 NX-OS Encapsulation (UDP) allows for load-balancing across multiple L3 uplinks UDP Encapsulation requires F3/M3 Modules Do not need tunnel-depolarization with UDP Can selectively enable / disable UDP option; if neighbor is M3/F3 and 7.2 code, we do UDP Encapsulation. Entropy support Release notes: os//config_guide/b_cisco_nexus_7000_series_nx- OS Configuration_Guide-RI/adv-otv.html

31 Loopback Join Interface: Ability to use a loopback address as the join interface for the overlay Ø Enables the use of multiple uplinks/ecmp paths in the core for resiliency and better traffic de-polarization of the links. Ø Loopback Join Interface adds a PIM-based control plane Ø Loopback Join interface to have multiple Uplinks into the provider Multicast core Ø The loopback join-interface is used to facilitate the following: Ø source the traffic from the edge device into the core device. Ø to source multicast traffic into the core device, in order to receive traffic from the core device.

32 Multiple uplinks/ecmp paths in Core Core L3 links L3 links L3 links Server/VM cabinet Server/VM cabinet DC1 DC2

33 Loopback Join Interface: Configuration Uplink Interfaces: join-interface: interface Ethernet1/2 ip address a.b.c.y ip router ospf 100 area ip pim sparse-mode no shutdown interface Ethernet1/3 ip address a.b.c.z ip router ospf 100 area ip pim sparse-mode no shutdown interface loopback1 ip address /32 ip router ospf 100 area ip pim sparse-mode Overlay interface: interface Overlay1 otv join-interface loopback1 otv control-group otv data-group /24 otv extend-vlan no shutdown

34 BFD for Peer Edge Device Failure Detection Nexus 7x00 NX-OS 6.2 No more waiting for IS-IS timeout! Overlay Relies on hardened multi-homing BFD between site VLAN SVIs for detection within the site Route reachability to neighbor IP address for detection on overlay Both adjacencies must be down to declare ED failure Site 150 msec detection Overlay < 5 sec (depends on routing performance and tuning in overlay) VDC Access Aggregation VPC VDC BFD otv site-vlan 1001 otv isis bfd interface Vlan1001 ip address /30 bfd interval 50 min_rx 50 multiplier 3

35 Introduction Control Plane and Data Plane Failure Isolation Multi-homing Mobility QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

36 Spanning-Tree and Site Independence Site transparency: no changes to the STP topology Total isolation of the STP domain Default behavior: no configuration is required BPDUs sent and received ONLY on Internal Interfaces The BPDUs stop here L3 L2 The BPDUs stop here

37 Unknown Unicast and No Longer Unknown Unicast Storms Across the DCI No requirements to forward unknown unicast frames Assumption: end-host are not silent or uni-directional MAC TABLE VLAN MAC IF 100 MAC 1 Eth1 Default behavior: no configuration is required L3 L2 100 MAC 2 IP B No MAC 3 in the MAC Table MAC 1 è MAC 3

38 Unknown Unicast and Selective Unicast Flooding Nexus 7x00 NX-OS Some applications are required to forward unknown unicast frames Selective Unicast Flooding can be enabled per MAC address Default behavior: no unknown unicast forwarding Use Cases: Silent-hosts/unidirectional devices L3 Microsoft Network Load-balancing (NLB).0000 Blk Overlay1 L2 Enable Flooding for MAC.1111 Unknown Unicast MAC State IF.0101 Blk Overlay Fwd Overlay1 -a(config)# otv flood mac vlan 172 MAC 1 è MAC 3 VLAN 100 MAC 6 è MAC 7 VLAN 172

39 Controlling ARP Traffic ARP Neighbor-Discovery (ND) Cache ARP cache maintained in Edge Device by snooping ARP replies First ARP request is broadcasted to all sites. Subsequent ARP requests are replied by local Edge Device ARP suppression can be disabled Timeout can be adjusted Drastic reduction of ARP traffic on DCI IPv4 only feature Default behavior: -a(config)# interface overlay 1 -a(config-if-overlay)# no otv surpress-arp-nd # Allows ARP requests over an overlay network and disables ARP caching on edge devices. This command does not support IPv6. no configuration is required -a(config)# interface overlay 1 -a(config-if-overlay)# otv arp-nd timeout 70 # Configures the time, in seconds, that an entry remains in the ARP-ND cache. The time is in seconds varying from 60 to The default timeout value is 480 seconds.

40 Introduction Distributed Data Centres: Goals and Challenges Control Plane and Data Plane Failure Isolation Multi-homing Mobility QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

41 Multi-homing Fully Automated Multi-homing No additional protocols required (i.e. BGP) site-vlan used to discover neighbor in the same site Authoritative Edge Device (AED) Election takes place Extended VLANs are split across the AEDs The AED is responsible for: MAC address advertisement for its VLANs Forwarding its VLANs traffic inside and outside the site AED Site Adjacency L3 L2 AED Site Adjacency used for AED election

42 Hardened Multi-homing Introducing Site-identifier Same site devices must use common site-identifier Site-id information is included in the control plane Makes multi-homing more robust and resilient Site Adjacency and Overlay Adjacency are now both leveraged for AED election Overlay Adjacency An overlay will not come up until a site-id is configured Site and Overlay Adjacency are both leveraged for AED election AED Site Adjacency L3 L2 AED feature otv otv site-identifier 0x1 otv site-vlan 99

43 STP BPDU Handling When STP is configured at a site, an Edge Device will send and receive BPDUs on the internal interfaces. An Edge Device will not originate or forward BPDUs on the overlay network. An Edge Device can become (but it is not required to) a root of one or more spanning trees within the site. An Edge Device will take the typical action when receiving Topology Change Notification (TCNs) messages. The BPDUs stop here Core

44 Data-plane Loop Prevention Handling Brodcast/M-cast packets reach all Edge Devices within a site. The AED for the VLAN is the only Edge Device that forwards b-cast/ m-cast packets onto the overlay network The b-cast/m-cast packet is replicated to all the Edge Devices on the overlay. Broadcast, Multicast, Unknown Unicast Only the AED at each remote site will forward the packet from the overlay onto the site. Once sent into the site, the b-cast/m-cast packet is replicated per regular switching Core AED AED

45 Multi-homing AED and Broadcast/Multicast Handling Broadcast/M-cast packets reach all Edge Devices within a site. The AED for the VLAN is the only Edge Device that forwards b-cast/ m-cast packets onto the overlay network The b-cast/m-cast packet is replicated to all the Edge Devices on the overlay. Only the AED at each remote site will forward the packet from the overlay onto the site. Once sent into the site, the b-cast/m-cast packet is replicated per regular switching Broadcast stops here Broadcast stops here Bcast pkt Core AED AED

46 Multi-homing AED and Unicast Forwarding One AED is elected for each VLAN on each site Different AEDs can be elected for each VLAN to balance traffic load Only the AED forwards unicast traffic to and from the overlay Only the AED advertises MAC addresses for any given site/vlan Unicast routes will point to the AED on the corresponding remote site/vlan MAC TABLE VLAN MAC IF 100 MAC 1 IP A 201 MAC 2 IP B AED IP A AED Core IP B AED AED

47 Multi-homing VLANs Split across AEDs Automated and deterministic algorithm In a dual-homed site: Lower IS-IS System-ID (Ordinal 0) = EVEN VLANs Higher IS-IS System-ID (Ordinal 1) = ODD VLANs Remote Device MAC Table VLAN MAC IF 100 MAC 1 IP A 101 MAC 2 IP B -a# show otv vlan Extended VLANs and Edge Device State Information (* - AED) VLAN Auth. Edge Device Vlan State Overlay East-b inactive(non AED) Overlay * East-a active Overlay East-b inactive(non AED) Overlay100 -b# show otv vlan Extended VLANs and Edge Device State Information (* - AED) VLAN Auth. Edge Device Vlan State Overlay * East-b active Overlay East-a inactive(non AED) Overlay * East-b active Overlay100 AED ODD VLANs IP A Overlay Adjacency Site Adjacency -a -b IP B AED EVEN VLANs

48 Introduction Control Plane and Data Plane Failure Isolation Multi-homing Mobility QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

49 and MAC Mobility MAC Moving and Updates (1) 1. Workload moved between Data Centre sites VM Moves MAC X MAC X MAC X Hypervisor MAC X Core Hypervisor MAC X MAC X AED AED

50 and MAC Mobility MAC Moving and Updates (2) 1. Workload moved between Data Centre sites 2. Workload is detected in East DC and control plane is triggered 2.3) AED advertises MAC X with a metric of zero MAC X MAC X MAC X MAC X MAC X Hypervisor MAC X Core MAC X Hypervisor MAC X MAC X AED 2.4) EDs in site West see MAC X advertisement with a better metric from site East and change them to remote MAC address. MAC X AED MAC X 2.2) AED detects MAC X is now local 2.1) vswitch originates an ARP (RARP) frame

51 and MAC Mobility MAC Moving and Updates (3) 1. Workload moved between Data Centre sites 2. Workload is detected in East DC and control plane is triggered 3. East to West data plane traffic allows to update the MAC tables of the L2 devices in West Site 3.2) AED in site West forwards the RARP into the site and the L2 switches update their CAM tables MAC X MAC X MAC X MAC X ESX MAC X Core MAC X ESX MAC X MAC X West AED 3.1) AED in site East forwards the RARP broadcast frame across the overlay AED East

52 Introduction Control Plane and Data Plane Failure Isolation Multi-homing Mobility L2 Multicast Forwarding QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

53 QoS and Marking on Encapsulation On Encapsulation CoS bits (802.1p) copied to the outer DSCP header If IP traffic: The original (inner) DSCP value is also copied to outer DSCP DMAC SMAC 802.1Q ETHERTYPE IP (optional) Q West CoS 802.1p IP A Inner DSCP IP (optional) Original Frame Outer DSCP shim IP B East 2 Encap BRKDCT

54 QoS and Marking on De-capsulation On De-capsulation CoS value is recovered from the outer DSCP and added to the 802.1Q header Original CoS and DSCP are both preserved Control Traffic is statically marked at CoS = 6/DSCP = 48 Decap 1 West IP A IP (optional) Original Frame Outer DSCP shim IP B 2 East 802.1Q DMAC SMAC 802.1Q ETHERTYPE IP (optional) CoS 802.1p Inner DSCP BRKDCT

55 How are we doing the DSCP to CoS mapping? RFC2474 DSCP Class DSCP (bin) DSCP (dec) CoS (bin) CoS (dec) BE CS1 / AF11 / AF12 / AF / 10 / 12 / CS2 / AF21 / AF22 / AF CS3 / AF31 / AF32 / AF / 26 / 28/ CS4 / AF41 / AF42 / AF / 34 / 36 / CS5 / EF / CS CS Default Mapping: 3 most significant bits of DSCP gets mapped to CoS bits

56 L2 Multicast with Multicast Transport Multicast Groups in the Core can leverage the benefits of a multicast-enabled transport for both control and data planes. The following summarizes the requirements for a multicast transport: Control Group Single PIM-SM or Bidir-PIM group used to form adjacencies and exchange reachability information Data Groups Range of SSM groups used to carry multicast data traffic generated by the sites Broadcast Group PIM-SM or Bidir-PIM group used for Broadcast traffic -a# sh otv interface Overlay100 otv join-interface e1/1 otv control-group otv broadcast-group otv data-group /27 otv extend-vlan Useful for QoS purposes: eg. ip multicast rate-limit Overlay Information Site Identifier Overlay interface Overlay100 VPN name : Overlay100 VPN state : UP Extended vlans : (Total:50) Control group : Data group range(s) : /27 Broadcast group : Join interface(s) : e1/11 ( ) Site vlan : 99 (up) AED-Capable : Yes Capability : Multicast-Reachable

57 Introduction Control Plane and Data Plane Failure Isolation Multi-homing Mobility L2 Multicast Forwarding QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

58 Path Optimization (Layer-2) VLAN Translation: Translation through transit VLAN When a different VLAN is used at multiple sites Usually for 3 or more sites VLAN 400 VLAN 400 VLAN 100 VLAN 200 DC DC West East

59 Path Optimization (Layer-2) VLAN Translation: Translation through transit VLAN -a(config)# int overlay1 -a(config-if-overlay)# otv vlan mapping 100 to 400 -a(config-if-overlay)# sh run int overlay1!command: show running-config interface Overlay1!Time: Fri Mar 29 19:01: version 6.2(2) interface Overlay1 otv isis hello-multiplier 9 otv join-interface port-channel11 otv control-group otv data-group /24 otv extend-vlan 25-50, otv vlan mapping 100 to 400 no shutdown -a(config-if-overlay)# sh otv vlan-mapping Original VLAN -> Translated VLAN > 400 -B(config)# int overlay1 -B(config-if-overlay)# otv vlan mapping 200 to 400 -B(config-if-overlay)# sh run int overlay1!command: show running-config interface Overlay1!Time: Fri Mar 29 19:02: version 6.2(2) interface Overlay1 otv isis hello-multiplier 9 otv join-interface port-channel21 otv control-group otv data-group /24 otv extend-vlan 25-50, otv vlan mapping 200 to 400 no shutdown -B(config-if-overlay)# sh otv vlan-mapping Original VLAN -> Translated VLAN > 400

60 Path Optimization (Layer-3) Egress Routing with LAN Extension Extended VLANs typically have associated HSRP groups By default, only one HSRP router elected active, with all servers pointing to HSRP VIP as default gateway Result: sub-optimal routing Routing HSRP Hellos ARP reply ARP for HSRP VIP Packet from Vlan 10 to Vlan 20 DMAC = DGW HSRP Active HSRP Standby HSRP Listen HSRP Listen Packet from Vlan 10 to Vlan 20 DMAC = Host Vlan 20 VLAN 20 VLAN 10

61 Egress Routing Localization FHRP Filtering Solution Filter FHRP with combination of VACL and MAC route filter Result: Still have one HSRP group with one VIP, but now have active router at each site for optimal first-hop routing HSRP Active HSRP Hellos HSRP Standby HSRP Filter HSRP Active Listen HSRP Hellos HSRP Standby Listen ARP for HSRP VIP ARP reply VLAN 20 VLAN 10

62 Egress Routing Localization Distributed IP Anycast Gateway Layer-2/Layer-3 demarcation moves to the Access-Switch (Leaf) Result: No HSRP and Distributed Anycast Gateway (DAG*), all active router at each Access-Switch (Leaf) for optimal first-hop routing HSRP DAG Active HSRP DAG Standby Active HSRP DAG Active Listen HSRP DAG Active Listen ARP reply ARP for HSRP VIP VLAN 20 VLAN 10 * Distributed Anycast Gateway requires VXLAN EVPN

63 Path Optimization (Layer-3) Optimal Routing Challenges Layer 2 extensions represent a challenge for optimal routing Challenging placement of gateway and advertisement of routing prefix/subnet Ingress: North-South / Client-Server HSRP Active HSRP Standby WAN HSRP Filter HSRP Active HSRP Standby Ingress: North-South / Client-Server Egress: South-North / Server-Client East-West / Server-Server Egress: South-North / Server-Client

64 Sample Cluster - Primary Service in Left DC FHRP localisation Path Optimisation /24 advertised into L /25 & /25 advertised into L3 ip prefix-list otv-local-prefix seq 10 permit /25 ip prefix-list otv-local-prefix seq 15 permit /25 route-map redist-otv-subnets Layer 3 Core permit 10 Data Centre Data Centre match ip address prefix-list otv-local-prefixes B A ip route /25 Null0 250 ip route /25 Null0 250 HSRP HSRP HSRP Group HSRP Group 1 HSRP HSRP Active Standby Priority 140 and 130 HSRP Filtering Priority 120 and 110 router eigrp 1 Active Standby router-id Public Network redistribute static route-map redist-otv-subnets Agg VLAN A Node A ü Asymmetrical flows No Stateful device Low ingress traffic Access Node B HA cluster Node A Cluster VIP = Preempt Default GW = HA cluster Node B

65 Sample Cluster Active / Active DC FHRP localisation Path Optimisation /25 advertised into L3 Data Centre A ip prefix-list otv-local-prefix seq 10 permit /25 route-map redist-otv-subnets permit 10 match ip address Layer prefix-list 3 Core otv-local-prefixes ip route /25 Null /25 advertised into L3 Data Centre B HSRP Active HSRP Standby HSRP router Group eigrp HSRP Group 1 Priority 140 router-id and HSRP Filtering Priority 120 and 110 redistribute static route-map redist-otv-subnets HSRP Active HSRP Standby Node A HA cluster Node A Cluster VIP = Preempt Default GW = Public Network ip prefix-list otv-local-prefix VLAN seq A 15 permit /25 route-map redist-otv-subnets permit 10 match ü ipasymmetrical address prefix-list flows otv-local-prefixes No Stateful device Low ingress traffic ip route /25 Null0 250 router eigrp 1 router-id redistribute static route-map redist-otv-subnets Agg Access HA cluster Node B Cluster VIP = Preempt Default GW = Node B

66 Ingress Routing Localization Possible Solutions Challenge Subnets are spread across locations Subnet information in the routing tables is not specific enough Routing doesn t know if a server has moved between locations Traffic may be sent to the location where the application is not available Options DNS Based Route Injection LISP Locator/ID Separation Protocol LISP and Deployment

67 Introduction Control Plane and Data Plane Failure Isolation Multi-homing Mobility L2 Multicast Forwarding QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

68 Apples vs. Oranges Data-Plane Control-Plane Multi-Homing Loop Prevention Fault Containment Multicast Optimization 1.0 (EoMPLSoGRE) 2.5 (UDP, VXLAN ) IS-IS IS-IS native native Block BPDU STP integration (ie TCN) Block BPDU STP integration (ie TCN) Stop Unknown Unicast Selective Unicast Flooding ARP Suppression Stop Unknown Unicast Selective Unicast Flooding ARP Suppression IGMP Snooping IGMP Snooping VXLAN VXLAN Flood&Learn VPC Block BPDU none Flood* VXLAN BGP EVPN VPC Block BPDU Minimized Unknown Unicast ARP Suppression Flood* * IGMP Snooping

69 Apples vs. Oranges Control-Plane Multi-Homing Loop Prevention Fault Containment Transport Agnostic Multicast Optimization Path Diversity Multi-Site Good Better FabricPath 1 VXLAN (Flood&Learn) 1 2 VXLAN BGP EVPN 1 2 VPLS 1 Best 1) Only with Multi-Chassis Link Aggregation (MC-LAG / VPC) 2) Limited Overlay Loop Prevention

70 Introduction Control Plane and Data Plane Failure Isolation Multi-homing Mobility L2 Multicast Forwarding QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

71 Interconnecting Multiple Data Centres LAN Extensions VXLAN L3 Domain VXLAN VTEP Domain Boundary: Failure and Event Containment Clear Administrative Delineation VXLAN VXLAN N7K/ASR N7K/ASR VXLAN L2/L3 Gateway VXLAN L2/L3 Gateway VXLAN L2/L3 Gateway VXLAN L2/L3 Gateway Data Centre 1 Data Centre 2 L3 Fabric L3 Fabric VXLAN L3 Gateway VXLAN L3 Gateway VXLAN L3 Gateway VXLAN L3 Gateway VXLAN L3 Gateway VXLAN L3 Gateway VXLAN L3 Gateway VXLAN L3 Gateway VNI 5000 VLAN 30 VLAN 20 VNI 5000 VLAN 20 VLAN 30

72 Principles of Interconnecting Networks Control-Plane Learn and Distribute MAC information (no Flood&Learn) Multi-Homing Automated Multi-Homing for Resiliency Core (Layer-3) /VPLS Loop Prevention Using redundant Path Providing Loop protection Fault Containment V V V V Separate Control-Plane information Limit Flood (ARP caching) Principles for Interconnecting Networks Do Apply for Ethernet, FabricPath and VXLAN

73 Principles of Interconnecting Networks (Con t) Transport Agnostic Can leverage literally any Transport Technology Simplified Transport Requirement Multicast dependent and independent Forwarding of BUM* Traffic (no hairpin) Core (Layer-3) /VPLS Multicast Optimization Offers optimised Multicast Forwarding Path Diversity V V V V Flow based Entropy Multi-Site Provides Site to Multi-Site connectivity Principles for Interconnecting Networks Do Apply for Ethernet, FabricPath and VXLAN

74 Stretched Fabric: Option #1 (if you must) Single BGP-EVPN Control-Plane Domain (AS100) Single Underlay Routing Domain for both Data Centres Single Multicast domain for both Data Centres (BUM to Underlay) Implement Anycast-RP redundancy Single end-to-end Data-Plane encapsulation (VXLAN) End-to-End reachability for VTEP Enable MSDP Between Multicast Domains Partial-mesh between Data Centres via border-leafs Since a single fabric by design, need to look at Underlay (OSPF) Cost for traffic egressing the proper Data Centre. L2 stretched fabric, need to look at LISP for Ingress (N-S) traffic draw Note: Ingress Replication can be used instead of Multicast

75 Multi-site VXLAN EVPN: Stretched Fabric #1a RR /RP RR /RP RR /RP RR /RP VXLAN EVPN BGP AS#100 Border-leaf Border-leaf ebgp ebgp Edge router BGP AS#200 Inter-DC Core (Layer-3 IP/MPLS) Edge router

76 Multi-site VXLAN EVPN: Stretched Fabric #1b RR /RP RR /RP RR /RP RR /RP VXLAN EVPN BGP AS#100 Border-leaf Border-leaf ebgp ebgp Edge router Physical Layer3 Interfaces MP-BGP (ibgp) EVPN BGP AS#200 Inter-DC Core (Layer-3 IP/MPLS) Edge router - The ibgp RR need to peer together, maintain all reachability information updated OSPF Area different. Egress cost is natural leaf to leaf, add more ospf cost.

77 Stretched Fabric Option #2 (if you must) Independent Underlay Routing Domain per Data Centre Independent Multicast domain per Data Centre (BUM to Underlay) Different BGP-EVPN Control-Plane Domain (AS100 and AS200) End-to-End Data-Plane encapsulation (VXLAN) End-to-End reachability for VTEP Enable MSDP Between Multicast Domains The /32 VTEP host routes need to advertised in both fabrics (no summaries) Summaries in core network only if needed (each fabric needs all /32 host routes) Multi-destination traffic needs to be the same between the Fabric (or ingress-replication) Avoiding Partitioning of OSPF and Virtual-links J Note: Ingress Replication can be used instead of Multicast above

78 Multi-site VXLAN EVPN RR RR RR RR VXLAN EVPN BGP AS#100 MP-BGP (ebgp) Multi-hop ebgp EVPN and IPv4 Unicast (Underlay) VXLAN EVPN BGP AS#200 Border-leaf Border-leaf 1) Two DCs are directly connected at the border-leaf (could be border-spine). ebgp ebgp 2) evpn Routes exchanged via ebgp 3) ebgp Sessions to/from WAN for External Connectivity (VRFlite/VNI) Edge router BGP AS#65500 Inter-DC Core (Layer-3 IP/MPLS) Edge router

79 Big-Fabrics or Stretched-Fabrics Interconnecting VXLAN/EVPN Pods with VXLAN/EVPN is possible Control-Plane Domains (EVPN) can be separated (ibgp/ebgp) Core (Layer-3) Data-Plane Encapsulation is End-to- End! Leaf/TOR knows about all VTEP across the two Data Centres ebgp BUM Traffic is across Pods Decision on Ingress Replication (Unicast) or Multicast is across all Pods being interconnected V VXLAN/EVPN VNI V ibgp VXLAN/EVPN VNI V V V Not All Principles Satisfied Good Enough Solution V

80 Big-Fabrics or Stretched-Fabrics Core (Layer-3) Switch# show nve peers Interface Peer-IP VNI Up Time nve :18:06 nve :44:24 nve :17:03 nve :08:44 nve :58:21 ebgp Switch# show nve peers Interface Peer-IP VNI Up Time nve :18:06 nve :06:22 nve :44:24 nve :17:03 nve :58:21 V V ibgp V V V V

81 VXLAN Multi-Site Recommendation Independent BGP-EVPN Control-Plane Domains Leverage a dedicated VXLAN/EVPN Transit Fabric Layer 2 traffic VLAN 802.1Q Trunk hand-off Layer 3 traffic vrf-lite sub-interfaces Independent VXLAN Data-Plane encapsulation per Data Centre (Domain) Independent Administrative Domains

82 Multi-site VXLAN EVPN: Separate CP and DP RR /RP RR /RP RR /RP RR /RP VXLAN EVPN EVPN ibgp AS#100 AS#100 VXLAN EVPN ibgp AS#200 Border-leaf Border-leaf Layer-3 sub-interfaces Layer q VLAN(s) Edge router BGP AS# 300 Edge router Inter-DC Core (Layer-3 IP/MPLS)

83 Introduction Control Plane and Data Plane Failure Isolation Multi-homing Mobility L2 Multicast Forwarding QoS and Scalability Path Optimization Architecture Principles Principles of Interconnecting Networks at Layer-2 Deployment Scenario Summary

84 VXLAN EVPN with Deployment Details Physical Topology Configuration CLI or DCNM CLI or DCNM POAP POAP templates Private (AS) for each DC fabric

85 MP-BGP EVPN Route Type 2 MP-BGP EVPN Route Type 2 - MAC/IP Advertisement Route Route Type 2 provides End-Host reachability information The following fields are part of the EVPN prefix in the NLRI Ethernet Tag ID (zeroed out) MAC Address Length (/48), MAC Address IP Address Length (/32, /128), IP Address [Optional] Additional Route Attributes Ethernet Segment Identifier (ESI) (zeroed out) MPLS Label1 (L2VNI) MPLS Label2 (L3VNI) RD (1 octet) ESI (10 octets) Ethernet Tag ID (4 octets) MAC Address Length (1 octet) MAC Address (6 octets) IP Address Length (1 octet) IP Address (0, 4, or 16 octets) MPLS Label1 (3 octets) MPLS Label2 (0 or 3 octets) TECDCT

86 MP-BGP EVPN Route Type 5 MP-BGP EVPN Route Type 5 - IP Prefix Route Route Type 5 provides IP Prefix advertisement in EVPN RT-5 decouples IP prefix from MAC (RT-2) and provides flexible advertisement of IPv4 and IPv6 Prefixes with variable length The following fields are part of the EVPN prefix in the NLRI IP Prefix Length (0-32 bits for IPv4 or bits for IPv6) IP Prefix (IPv4 or IPv6) GW IP Address MPLS Label (L3VNI) RD (8 octet) ESI (10 octets) Ethernet Tag ID (4 octets) IP Prefix Length (1 octet) IP Prefix (4 or 16 octets) GW IP Address (4 or 16 octets) MPLS Label (3 octets) TECDCT

87 Route Type: 2 - MAC/IP Ethernet Segment Identifier Ethernet Tag Identifier MAC Address Length MAC Address IP Address Length IP Address V2# show bgp l2vpn evpn BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: :32868 BGP routing table entry for [2]:[0]:[0]:[48]:[ a3.c2bb]:[32]:[ ]/272, version 4 Paths: (1 available, best #1) Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked L3VNI L2VNI Advertised path-id 1 Path type: internal, path is valid, is best path, no labeled nexthop AS-Path: NONE, path sourced internal to AS (metric 3) from ( ) Origin IGP, MED not set, localpref 100, weight 0 Received label Extcommunity: RT:65501:30001 RT:65501:50001 ENCAP:8 Router MAC: d Originator: Cluster list: Remote VTEP IP Address Route Target: L2VNI (VLAN) Route Target: L3VNI (VRF) Overlay Encapsulation: 8 - VXLAN Router MAC of Remote VTEP TECDCT

88 VXLAN EVPN with Deployment Details Physical Topology - The border nodes are configured with all the local VRF instances associated with Layer 3 VNI (L3VNI) for external connectivity. - Layer 2 segments and associated Layer 2 VNI (L2VNI) relevant for extension via are also terminated at the border nodes. - To ensure resiliency, the border nodes provide Layer 2 redundancy with virtual port channels (VPCs) for the connectivity to the edge devices. - In order to accommodate the Layer 2 termination on the Cisco Nexus 7000 Series Switch acting as the VXLAN tunnel endpoint (VTEP), respective bridge domain, Layer 2 VNI (L2VNI) with associated multicast group, and EVPN instances (EVI) have to be configured.

89 VXLAN EVPN with Deployment Details Layer-2 and Layer-3 Services - Layer 2 segments extended via, the fabric border node splits the IP and MAC address information advertised in EVPN route type 2 advertisements. 1) MAC-only extension is achieved via Layer 2 2) IP-only via the VRF-aware Layer 3 extension - The separation of the MAC and IP routes in EVPN (route type 2) results in an individual MAC-only (route type 2) and IP-only (route type 5) route in the remote fabric (fabric-2) - In addition to the all of the VLANs extended via, the site VLAN must be present on the inside interface and on the VPC peer link. - The extension of the site VLAN beyond the edge device facilitates native multi-homing, backdoor-link detection, and fast convergence. In the latter case, the site VLAN is enhanced with a switched virtual interface (SVI) and bidirectional forwarding detection (BFD).

90 Layer 2: Fabric to Fabric via transport

91 VXLAN EVPN with Deployment Details Forwarding from the Fabric Toward - Once a host is learned at a leaf in a VXLAN EVPN fabric, at the border node, the received IP and MAC address information is learned as part of EVPN route type 2 advertisement. - The subsequent forwarding across for Layer 2 and VRF-aware transport for Layer 3, the routing and bridging portions are separate. - Layer 2 traffic is terminated on the border node learned via the BGP EVPN route type 2 update. - From the border node toward the edge device, the MAC address is learned in a traditional way through data-plane communication. - Then is advertised via s control-plane exchange to the remote edge device. - Fabric 2 receives the individual MAC address over that is advertised as a MAC-only route type 2 advertisement into EVPN.

92 Layer 2 Communication via Toward Remote Fabric

93 Host Mobility via Toward Remote Fabric Pre-host move - The ability to move hosts between fabrics has been one of the major use cases for data centre networks. The ability to dynamically and manually move hosts to new locations provides the opportunity for better load distribution or failure handling in the sense of high availability.

94 Host Mobility via Toward Remote Fabric Post-host move When we consider a host moving from a leaf in fabric 1 to a leaf in fabric 2, the following steps are involved: 1. Host moves from the leaf in fabric 1 to a leaf in fabric 2 2. Once the host move is completed, the virtual switch at the destination server typically issues a gratuitous ARP (GARP) or reverse ARP (RARP) to signal completion 3. The GARP and RARP notification is used to withdraw and update the ARP table state to reflect the new location of the host (fixup). This message will update the Layer 2 tables along the path. 4. During fixup, the state tables (MAC, ARP, routing) are modified to reflect the correct situation after the move. a. In fabric 2, the previous individual EVPN routes are withdrawn, the new learning of the host in fabric 2, a single IP and MAC route (EVPN route type 2) with adjusted MAC mobility sequence number is now present. b. In fabric 1, the previous single EVPN route type 2 (IP and MAC) is withdrawn via Layer 2 DCI we learn an EVPN route type 2 (MAC only) and EVPN route type 5 (IP only) via Layer 3 DCI. The MAC mobility sequence number is also updated 5. The host move from the leaf in fabric 1 to the leaf in fabric 2 is now complete

95 Host Mobility via Toward Remote Fabric Post-host move

96 Fault Containment UUU-Flooding - With unknown unicast flooding, traffic is unnecessarily transported across DCI links. - Layer 2 loops - Frequent topology changes (frequent MAC/ARP table flush) - ARP/MAC timers not aligned - Disabling unknown unicast flooding across a DCI, prevents the impact of such traffic patterns from one fabric to other fabrics, thereby avoiding failure propagation and providing isolation

97 Fault Containment Broadcast Storms - In cases where Layer 2 loops in a network, the existence of broadcast traffic can create significant impact.. - As broadcasts are an integral part of Layer 2 networking, it is not possible to disable broadcast forwarding completely in a network. (it would break ARP, DHCP as an example) - The goal is to limit / minimize the impact of a broadcast storm from one fabric to adjacent ones. - From the border node of fabric 1 toward the edge devices, a classic Ethernet link exists, and thus storm control can be applied. - In addition, allows the broadcast traffic to be placed into a separate multicast group. With this separation, the broadcast, unknown unicast, and multicast can be treated differently, and excessive broadcasts can be identified and rate-limited

98 Fault Containment Backdoor Path - Layer 2 extensions bring the threat of a backdoor path during network changes and migration. - Traditional Layer 2 extensions do not provide an integrated approach to detect such a looped topology, other than the use of Spanning Tree. - The site-vlan together with the site ID understand the concept of a site, which is also part of the integrated multihoming approach. - In cases where the site VLAN is common in all fabrics but the site ID is different, a backdoor path will be detected resulting in the tunnel will be shut down, thereby ensuring that a loop is prevented.

99 Layer 3 High level discussion - For an IP subnet that is extended through Layer 2 between multiple fabrics, instantiation of the distributed IP anycast gateway on the border nodes is not supported. - By default, host routes are advertised across sites also for end-points belonging to non-stretched IP subnets. - The exchange of Layer 3 information for IP subnets is required. Technologies like VRF-lite, MPLS L3VPN, or LISP can accommodate this requirement. - The Anycast Gateway MAC (AGM) has to be the same for all fabrics. Configure the Cisco Fabric Border Provider Edge Feature for VXLAN EVPN Fabric: itches/nexus-7000-series-switches/white-paper-c pdf Optimizing Ingress Routing with LISP across Multiple VXLAN/EVPN Sites White Paper: /nexus-7000-series-switches/white-paper-c html

100 Optimizing Layer 2 DCI with between Multiple VXLAN EVPN Fabrics (Multi-fabric) White Paper CCO URL: witches/nexus-9000-series-switches/white-paperc html

101 Standalone Fabric VXLAN + (Layer2 Applications) BGP EVPN + VXLAN Fabric #1 VXLAN VXLAN RR/ RP RR/ RP BGP EVPN + VXLAN Fabric #2 BGP AS #100 BGP AS #200 Distributed Anycast Gateway Distributed Anycast Gateway Access H (VLAN 111) H (VLAN 111) VLAN(s) MPLS/IP + WAN + INET VLAN(s) Domain Boundary: Failure and Event Containment Clear Administrative Delineation

102 Nexus 7000 VXLAN to Bridging VXLAN Layer 2 tunnel (VNI) to Layer 2 tunnel () bridging The L2 VNI to L2 tunnel mapping happens within the same VDC 1 Box Solution Fabric A L3 Core Fabric B Nexus 7k Join Interface Nexus 7k L2 VNI to Stitching

103 Use Case Two Sites Connected

104 and 802.1ae Encryption (AES128bit & AES256 bit) Core Core AGG AGG AED AED AED AED

105 Advantages of : Ultimate operational simplicity Provisioning/CLI DCNM automation Seamless insertion Scalability Failure Scenarios / Optimizations Maturity Shipping since 2010, thousands of mission critical deployments Integrated DCI feature set Multi-homing with loop detection, site and flood suppression Standards based data-plane VXLAN encapsulation in

106 Minimum, Maximum Transmission Unit (MTU) Guidance: EoMPLS Port Mode: 1522 Bytes EoMPLS VLAN Mode: 1526 Bytes VPLS: 1526 Bytes (1530 Bytes with control-word) A-VPLS: 1530 with flow-label (3rd Label), (1534 with control-word) : 1542 Bytes w/udp: 1550 Bytes LISP IPv Bytes IPv bytes For Your Reference

107 Minimum, Maximum Transmission Unit (MTU) Guidance: (Con t) FabricPath: 1516 Bytes VXLAN: 1550 Bytes GRE: 1524 Bytes 802.1ae: 1540 Bytes IPSEC: 1574 Bytes For Your Reference

108 Q & A

109 Complete Your Online Session Evaluation Give us your feedback and receive a Cisco 2017 T-Shirt by completing the Overall Event Survey and 5 Session Evaluations. Directly from your mobile device on the Cisco Live Mobile App By visiting the Cisco Live Mobile Site Visit any Cisco Live Internet Station located throughout the venue T-Shirts can be collected Friday 10 March at Registration Learn online with Cisco Live! Visit us online after the conference for full access to session videos and presentations.

110 Thank you

111