Building Data Center Networks with VXLAN EVPN Overlays Part I

BRKDCT-2949 Building Data Center Networks with VXLAN EVPN Overlays Part I Lukas Krattiger, Principal Engineer

Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#brkdct-2949 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Session Objective A short Overview on Data Center Fabric A close look at Single Fabric Overlay and Underlay Details on Single Fabric Control- & Data-Plane Multi-Tenancy in VXLAN BGP EVPN environments First-Hop Gateway with Distributed Anycast Gateway Multi-Homing with Virtual Port-Channel (VPC) for VXLAN BRKDCT-2949 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

Introduction to Data Center Fabrics

Data Center Fabric Journey (Standalone) Layer-3 Layer-2 HSRP HSRP Spanning-Tree Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Baremet al Baremet al Baremet al Baremet al 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

The / Topology (Clos* Network) Wide ECMP: Unicast or Multicast Uniform Reachability Deterministic Latency High Redundancy On Node or Link Failure *Clos, Charles (1953) "A study of non-blocking switching networks" 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

A Scale Out Architecture Smallest Operational Entity s Wide vs. Big Uplinks Symmetric to all s or Pods SAYG: Scale as You Grow More More Bandwidth More Resiliency More More Ports More Capacity 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

The Super- Super Scale Out Not Limited to Port Density Simpler Capacity Planning Super Super Beyond a single Server Room Allows Interconnecting Pods Retains Intra-Pod Topology with Flexible Inter-Pod Connectivity POD 1 POD 2 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13

Data Center Fabric Properties Any Subnet, Anywhere, Rapidly Any Network on Any Reduced Failure Domain Any Default Gateway on Any - Distributed Extensible Scale and Resiliency 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

Overlay Based Data Center: Fabrics Mobility Overlay Segmentation Scale Automated and Programmable Abstracted Consumption Model Layer-2 and Layer-3 Service Physical and Virtual Workloads 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15

Overlay Based Data Center: Edge Devices Network Overlays Host Overlays Overlay Overlay - - - - Hybrid Overlays Router/Switch End-Points Protocols for Resiliency/Loops Traditional VPNs VXLAN, OTV, VPLS, LISP, FP - - Overlay Virtual End-Points only Single Admin Domain VXLAN, NVGRE, STT Physical and Virtual Resiliency and Scale Cross-Organizations/Federation Open Standards 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

Overlay Taxonomy - Overlay Tunnel Encapsulation (VNI Namespace) Overlay LAN Segment Virtual Server Physical Server : VXLAN Tunnel End-Point VNI/VNID: VXLAN Network Identifier 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

Understanding Overlay Technologies Overlay Services Layer-2 Layer-3 Layer-2 and Layer-3 Tunnel Encapsulation Underlay Transport Network Control-Plane Peer-Discovery Route Learning and Distribution Local Learning Remote Learning Data-Plane Overlay Layer-2/Layer-3 Unicast Traffic Overlay Broadcast, Unknown Unicast, Multicast traffic (BUM traffic) forwarding Ingress Replication (Unicast) Multicast 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

VXLAN Evolves as the Control Plane Evolves! Back Then Yet Another Encapsulation Flood & Learn (Multicast-based) Data-Plane only 4 Years ago VXLAN for the Data Center Intra-DC Control-Plane Active Discovery Multicast and Unicast 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21

A single Fabric with Overlay Overlay A Single Overlay Domain End-to-End Encapsulation Closest to the Source Closest to the Destination External Connectivity @ = Border @ = Border POD 1 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22

The Super- and the Overlay (Multi-POD) Super Super Super Still, a Single Overlay Domain End-to-End Encapsulation Closest to the Source Closest to the Destination Overlay External Connectivity Overlay @ = Border @ = Border @ Super- or Connected to Super- POD 1 POD 2 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25

Attributes of Multi-POD in VXLAN BGP EVPN Underlay Super Nicely Structured and Tiered Topologies Super Super Allows Efficient Scale-Out More End-Points = More More Bandwidth, Resilience or Capacity = More or Tiers Different Control-Plane Instances (BGP AS) Overlay Overlay End-to-End Encapsulation, Flat, No Hierarchy Single Control-Plane reach all in one kitchen sink Overlay POD 1 POD 2 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26

The Super- and the Overlay (Multi-POD) Super Super Super Overlay Scale-Out Model to Build a Large Intra-DC Network? Data Center Interconnect (DCI)? Overlay Domain Normalization (Coexistence and/or Migration)? POD 1 POD 2 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27

VXLAN Evolves as the Control Plane Evolves! Back Then Yet Another Encapsulation Flood & Learn (Multicast-based) Data-Plane only 4 Years ago VXLAN for the Data Center Intra-DC Control-Plane Active Discovery Multicast and Unicast Today VXLAN for DCI Inter-DC Multi-Site Control- & Data-Plane Separation Failure Domain Isolation 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

Use-Cases for DCI Connectivity Scale-Out Model to Build a Large Intra- DC Network Data Center Interconnect (DCI) Domain Normalization (Coexistence and/or Migration) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

VXLAN for Interconnecting Networks VXLAN Multi-Pod VXLAN Multi-Fabric EVPN Fabric Control- #1 Plane Domain 1 BGP EVPN EVPN Fabric Control- #2 Plane Domain 2 EVPN Fabric Control-Plane #1 Domain 1 EVPN Fabric Control-Plane #2 Domain 2 Overlay Overlay Overlay Overlay VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P Bar em eta l Bar em eta l Single Data-Plane End-to-End Bar em eta l Bar em eta l Bar em etal Bar em etal DCI Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Bar em etal Bar em etal Single Fabric with End-to-End Encapsulation Build Hierarchy in the Underlay Flatten it in the Overlay Multiple Fabrics Normalized through Ethernet Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

VXLAN for Interconnecting Networks VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site EVPN Fabric Control- #1 Plane Domain 1 BGP EVPN EVPN Fabric Control- #2 Plane Domain 2 EVPN Fabric Control-Plane #1 Domain 1 EVPN Fabric Control-Plane #2 Domain 2 EVPN Fabric Control-Plane #1 Domain 1 BGP EVPN EVPN Fabric Control-Plane #2 Domain 2 Overlay Overlay Overlay Overlay Overlay Overlay VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P Bar em eta l Bar em eta l Single Data-Plane End-to-End Bar em eta l Bar em eta l Bar em etal Bar em etal DCI Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Bar em etal Bar em etal Bar em etal Bar em etal DCI Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Bar em etal Bar em etal Single Fabric with End-to-End Encapsulation Build Hierarchy in the Underlay Flatten it in the Overlay Multiple Fabrics Normalized through Ethernet Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3) Multiple Fabrics with Integrated DCI Integrated DCI Scaling within and between Fabrics 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

VXLAN Multi-Site Introducing Overlay Hierarchies Border Gateways (BGW) (Key Functional Components of VXLAN Multi-Site Architecture) Super Super Super Multi-Site Overlay Overlay Overlay Site 1 Site 2 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33

VXLAN Multi-Site Introducing Overlay Hierarchies Super Multiple Overlay Domains Overlay Super Super Multi-Site Overlay Per-Site Encapsulation Closest to the Source Closest to the Destination Exit/Transit via Border Gateway (BGW) Overlay Multi-Site and/or External Connectivity @ = Border @ = Border Super- becomes Transit Site 1 Site 2 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

MTU and Overlays Data Center often require Jumbo MTU Most Server NIC support up to 9000 Bytes Network Switches support MTU up to 9216* Bytes Accommodates Jumbo MTU plus Overlay overhead Avoid Fragmentation Adjust the Transport Network with appropriate MTU *Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 Traffic 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

Interface Principles Routed Ports and Interfaces Layer-3 Interfaces between and (no switchport) For each Point-2-Point (P2P) connection, minimum /31 required Alternative, use IP Unnumbered (/32) Underlay Use Loopback as Source- Interface for (NVE*) *NVE: Network Virtualization Edge 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

IP Addressing Principles Prepare a IP Addressing Plan Separate Interface functions through IP Addressing (Aggregates) Unicast Routing Routing Protocol Peering (p2p*) Unicast Routing Routing Identifier (RID) and VPC Multicast Rendezvous-Point (RP) IPv4 only (today) p2p* Links / IP Unnumbered Underlay Loopback Routing Identifier Routing Identifier Rendezvous Point p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 *p2p: Point-to-Point 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

IP Addressing Principles interface ethernet4/4 description p2p-to- ip address 10.1.1.2/30 interface ethernet1/49 description p2p-to- ip address 10.1.1.1/30 interface loopback254 description RP ip address 10.254.254.1/32 interface loopback0 description RID ip address 10.10.10.201/32 Underlay interface loopback0 description RID ip address 10.10.10.101/32 interface loopback1 description ip address 10.200.200.101/32 p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

Some Math IP Addressing Principles (P2P) Example from depicted Topology 4 * 7 (28 Links) 11 Router ID (RID Loopback) 7 (Loopback) Underlay 28 Link * 2 (/31) = 56 IP Addresses 11 Router ID (RID) = 11 IP Addresses 7 = 7 IP Addresses Total: 74 IP Addresses Required 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

Simplifying the Math IP Unnumbered Example from depicted Topology 4 + 7 (11 Loopback) 11 Router ID (RID Loopback) 7 (Loopback) Underlay 11 Unnumbered IF = 11 IP Addresses 11 Router ID (RID) = 11 IP Addresses 7 = 7 IP Addresses Total: 29 IP Addresses Required 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42

Unicast Routing OSPF and IS-IS OSPF watch your Network Type Network Type Point-2-Point Preferred (only LSA type-1) No DR/BDR election Suits well for routed interfaces/ports (optimal from a LSA DB perspective) Full SPF calculation on Link Change IS-IS what was this CLNS? Independent of IP (CLNS) Well suited for routed interfaces/ports No SPF calculation on Link change; only if Topology changes Fast Re-convergence Not everyone is familiar with it 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

Unicast Routing BGP ebgp Underlay Routing Service Provider style Two Different Models Two-AS Multi-AS BGP is a Distance Vector Protocol actually Path Vector Protocol AS* are used to calculate the Path (AS_Path) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

Unicast Routing ebgp Two-AS Model All- AS#65500 ebgp Two-AS, yes it works! Underlay All- AS#65501 ebgp peering for Underlay is not a Route-Reflector (ebgp) Retain Route-Targets Disable BGP AS-Path check Next-Hop needs to be Unchanged Underlay is Reachability! Advertise your Loopbacks 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

Unicast Routing ebgp Multi-AS Model All- AS#65500 Underlay ebgp Two-AS, yes it works! ebgp peering for Underlay is not a Route-Reflector (ebgp) Retain Route-Targets Disable BGP AS-Path check Next-Hop needs to be Unchanged Underlay is Reachability! Advertise your Loopbacks Changes Overlay Routing Policy Manually define Route-Targets 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46

Unicast Routing ebgp Model Two different BGP Peering Underlay ebgp peering for Underlay Global IPv4/v6 Address-Family Use Physical Interface IP ebgp peering for Overlay Global EVPN Address-Family Use Loopback Interface IP BFD not so ok 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

Unicast Routing Why two different BGP Peering? 1) Interface Down BGP Down BGP Peering (IPv4/IPv6) BGP Peering (IPv4/IPv6) AS#65500 BGP Peering (IPv4/IPv6) Point-2-Point Link Fails BGP Peering is teared down Lights-Out Event or BFD Fast reaction to Routing Table Underlay Network Converges ECMP kicks in if available/configured IGPs do this Automatically BGP Peering (IPv4/IPv6) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48

Unicast Routing Why two different BGP Peering? 1) Interface Down BGP Down 2) Interface Down BGP Not Down AS#65500 Point-2-Point Link Fails BGP Peering (EVPN) Loopback to Loopback Peering remains Up If Alternate Path available Timers should allow Time for Network Re-Convergence No BFD Unchanged Overlay Reachability No Mass Delete/Re-Learn Underlay Path change only 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

Unicast Routing Why two different BGP Peering? 1) Interface Down BGP Down 2) Interface Down BGP Not Down 3) Down Prefix are Withdrawn (RNH*) 5 192.168.10.0/24 2 0000.3001.1101 Next-Hop: 10.200.200.102 2 0000.3001.1101, 192.168.10.101 AS#65500 and p2p Interfaces Fail Either IGP or BGP converges Loopback to Loopback Peering remains Up BGP Dead-Timer (180s) Recursive Next-Hop will trigger Convergence Event Next-Hop () disappeared in Underlay Overlay withdraws Prefixes *RNH: Recursive Next-Hop 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

Underlay - Unicast Routing and Overlay Specific to BGP as a Overlay Control-Protocol BGP Knobs can Help here Generic Concept for Underlay / Overlay Separation Use Different Routing Protocol Use Same Routing Protocol RNH* for Overlay works with ALL Underlay Routing Protocols Ensure /32-Reachability for s Other Routes can impact (Aggregates, Default-Route) IGP + BGP for true Protocol Separation BGP for single Routing Protocol approach 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51

Underlay - Multicast Routing and Rendezvous-Point PIM Any-Source-Multicast (ASM) Platform Support Nexus 9000 / Nexus 7000 (F3/M3) ASR 1000 / ASR 9000 RP Redundancy PIM Anycast-RP or MSDP Source-Trees (Unidirectional) 1 Source Tree per per Multicast Group Bidirectional PIM (Bidir) Platform Support Nexus 5600 / Nexus 7000 (F3/M3) ASR 1000 / ASR 9000 RP Redundancy Phantom-RP Shared-Trees (Bidirectional) 1 Shared Tree per Multicast Group Follows Unicast Routing Path 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52

Underlay PIM ASM with PIM Anycast-RP interface loopback254 description RP ip address 10.254.254.1/32 ip pim sparse-mode interface loopback0 description RID ip address 10.10.10.201/32 ip pim sparse-mode ip pim anycast-rp 10.254.254.1 10.10.10.201 ip pim anycast-rp 10.254.254.1 10.10.10.202 ip pim rp-address 10.254.254.1 RP RP (&) Underlay interface loopback254 description RP ip address 10.254.254.1/32 ip pim sparse-mode interface loopback0 description RID ipaddress 10.10.10.202/32 ip pim sparse-mode 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

Underlay - Multicast Routing and Rendezvous-Point The makes a good Rendezvous-Point (RP) Use multiple RP for Redundancy Watch your Multicast-Group and OIF* scale VXLAN uses Multicast for BUM Broadcast, Unknown Unicast, Multicast 1:1 Multicast-to-VNI mapping 1:N Multicast-to-VNI mapping Ingress-Replication can be valid as well *OIF: Outgoing Interface 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

Underlay Ingress Replication A Packet Multiplication EVPN assists, VNI Topology Various Platform Support Ie Nexus 9000 Underlay Ingress Replication Host sends 1 Packet to Edge-Device Edge-Device Encapsulates 1 Packet and multiplies it Ingress sends 1 Packet per Neighbor 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

VXLAN with BGP EVPN

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

What is? VXLAN Standards based Encapsulation RFC 7348 Uses UDP-Encapsulation Transport Independent Layer-3 Transport (Underlay) Flexible Namespace 24-bit field (VNID) provides ~16M unique identifier Allows Segmentations EVPN Standards based Control-Plane RFC 7432 Uses Multiprotocol BGP Uses Various Data-Planes VXLAN (EVPN-Overlay), MPLS, Provider Backbone (PBB) Many Use-Cases Covered Bridging, MAC Mobility, First-Hop & Prefix Routing, Multi-Tenancy (VPN) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

Introducing Ethernet VPN (EVPN) EVPN MP-BGP RFC 7432 MPLS (draft-ietf-l2vpn-evpn) Provider Backbone Bridges (draft-ietf-l2vpn-pbb-evpn) Overlay (NVO3) (draft-ietf-bess-evpn-overlay) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64

VXLAN and EVPN related RFCs & Drafts (IETF) ID Title Category RFC 7348 Virtual Extensible Local Area Network Data Plane RFC 7432 BGP MPLS based Ethernet VPNs Control Plane draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65

Multiprotocol BGP (MP-BGP) Primer AS#65500 Multiprotocol BGP (MP-BGP) Extension to Border Gateway Protocol (BGP) RFC 4760 VPN Address-Family Allows different types of Address- Families (i.e. VPNv4/v6, MVPN, L2VPN, EVPN) Various Information transported over single BGP Peering 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

Multiprotocol BGP (MP-BGP) Primer vrf context A rd 10.10.10.101:3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 vrf context A rd 10.10.10.102:8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 VPN Segmentation for Tenant Routing Route Distinguisher (RD) 8-byte field A Value to make a VPN Prefix unique RD + VPN Prefix [10.10.10.101:5000 + 192.168.10.0/24] 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

Multiprotocol BGP (MP-BGP) Primer vrf context A rd auto address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 vrf context A rd auto address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 Cisco provides automated Route Distinguisher derivation Macros uses Type 1 format 4-byte Router ID (RID) 4-byte VRF ID (internal number) Example of auto derived RD: 10.10.10.101:3 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68

Multiprotocol BGP (MP-BGP) Primer vrf context A rd auto address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 vrf context A rd auto address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 VPN Segmentation for Tenant Routing Route Target (RT) 8-byte field A Value to import/export a VPN Prefix Each RD + VPN Prefix have an RT [10.10.10.101:5000 + 192.168.10.0/24] [65500:5000, 65500:5000] 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69

Multiprotocol BGP (MP-BGP) Primer vrf context A rd auto address-family ipv4 unicast route-target import auto route-target export auto AS#65500 vrf context A rd auto address-family ipv4 unicast route-target import auto route-target export auto Cisco provides automated Route Target derivation Macros uses following values 4-byte Autonomous System 4-byte VNI Example of auto derived RD: Import, Export or Both 65500:5000 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70

Multiprotocol BGP (MP-BGP) Primer vrf context A rd 10.10.10.101:3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 vrf context A rd 10.10.10.102:8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 RD Prefix Next-Hop Route Target 10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000 AS#65500 10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

Multiprotocol BGP (MP-BGP) Primer vrf context A rd 10.10.10.101:3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 vrf context A rd 10.10.10.102:8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 RD Prefix Next-Hop Route Target 10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000 10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72

Multiprotocol BGP (MP-BGP) Primer vrf context A rd 10.10.10.101:3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 vrf context A rd 10.10.10.102:8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 RD Prefix Next-Hop Route Target 10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000 10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 73

Multiprotocol BGP (MP-BGP) Primer vrf context A rd 10.10.10.101:3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 vrf context A rd 10.10.10.102:8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 5 192.168.10.0/24 Next-Hop: 10.200.200.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74

EVPN - Host and Subnet Route Distribution RR RR Host Route Distribution decoupled from the Underlay protocol Overlay Use MultiProtocol-BGP (MP- BGP) on the nodes to distribute internal Host/Subnet Routes and external reachability information Route-Reflectors (RR) deployed for scaling purposes 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75

EVPN Control Plane - Host and Subnet Routes BGP EVPN NLRI* Overlay Host MAC (Route Type 2) MAC only, Single VNI, Single Route Target Host MAC+IP (Route Type 2) MAC and IP, Two VNI, Two Route Target, Router MAC Internal and External Subnet Prefixes (Route Type 5) IP Subnet Prefix, Single VNI, Single Route Target *NLRI: Network Layer Reachability Information (BGP Update Format) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76

Host Advertisements Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101 2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104 Overlay Host A MAC: 0000.3001.1101 Host B MAC: 0000.3001.1102 Host C MAC: 0000.3002.2101 *L2VNI: VNI for all Bridging operation ( VLAN-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77

Host Advertisements Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101 2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104 Overlay 2 0000.3002.2101 / 48 3002, 65500:3002 10.200.200.107 Host MAC (Route Type 2) MAC MPLS Label1 (L2VNI*) Route Target for MAC-VRF MAC attributes are Mandatory Host A MAC: 0000.3001.1101 Host B MAC: 0000.3001.1102 Host C MAC: 0000.3002.2101 *L2VNI: VNI for all Bridging operation ( VLAN-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78

Ethernet Tag Ethernet Identifier V2# show bgp l2vpn evpn 0000.3001.1101 Segment (Ethtag) Identifier (ESI) MAC Address MAC Route Type: Length BGP routing table information for VRF default, address family Address MAC/IP L2VPN EVPN Route Distinguisher: 10.10.10.101:32777 BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[0]:[0.0.0.0]/216, version 4 Paths: (1 available, best #1) Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked Next-Hop Advertised path-id 1 IP Address Path type: internal, path L2VNIis valid, is best path, no labeled nexthop AS-Path: NONE, path (MPLS sourced Label1) internal to AS 10.200.200.101 (metric 3) from L2VNI 10.10.10.201 Encap:8 (10.10.10.201) Route Target Origin IGP, MED not set, localpref 100, VXLAN weight 0 Received label 3001 Extcommunity: RT:65500:3001 ENCAP:8 Originator: 10.10.10.101 Cluster list: 10.10.10.201 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79

Host Advertisements Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101 2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104 Overlay Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 *L3VNI: VNI for all Routing operation ( VRF-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80

Host Advertisements Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101 Overlay 2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104 2 0000.3002.2101 / 48 3002, 65500:3002 192.168.20.101 /32 5000, 65500:5000 10.200.200.107 Host MAC+IP (Route Type 2) MAC and IP MPLS Label1 (L2VNI) Route Target for MAC-VRF MPLS Label2 (L3VNI*) Route Target for IP-VRF Router MAC IP Attributes are Optional Populated through ARP/ND Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 *L3VNI: VNI for all Routing operation ( VRF-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 81

Ethernet Tag Ethernet Identifier V2# show bgp l2vpn evpn 0000.3001.1101 Segment (Ethtag) Identifier (ESI) MAC Address MAC Route Type: Length BGP routing table information for VRF default, address family Address MAC/IP L2VPN EVPN Route Distinguisher: 10.10.10.101:32777 BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[32]:[192.168.10.101]/272, version 4 Paths: (1 available, best #1) IP Address Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked Length IP Address Next-Hop Advertised path-id 1 IP Address L3VNI Path type: internal, L2VNI path is (MPLS valid, Label2) is best path, no labeled nexthop AS-Path: NONE, path (MPLS sourced Label1) internal to AS 10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201) Encap:8 Origin IGP, MED not set, localpref 100, weight 0 VXLAN Received label 3001 5000 Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01 Originator: 10.10.10.101 Cluster list: 10.10.10.201 L2VNI L3VNI Route Target Router MAC Route Target 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

Subnet Route Advertisements Type IP / Length L3VNI / RT Next-Hop Seq. 101010110101 01010101010 5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101 Overlay Subnet A 192.168.10.0/24 Internal and External Subnet Prefixes (Route Type 5) IP Prefix MPLS Label (L3VNI) Route Target for IP-VRF Router MAC Populated through External Routing Protocol 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83

Subnet Route Advertisements Type IP / Length L3VNI / RT Next-Hop Seq. 5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101 5 192.168.10.0 /24 5000, 65500:5000 10.200.200.104 Overlay 5 192.168.20.0 /24 5000, 65500:5000 10.200.200.107 Subnet A 192.168.10.0/24 101010110101 01010101010 Subnet A 192.168.10.0/24 101010110101 01010101010 Subnet B 192.168.20.0/24 IP Prefix Learning via BGP with VRF-Lite via LISP on Nexus 7000/7700 via other routing protocol (static or dynamic) Default: Export of IP Host and IP Prefix Routes advertisements Filter and Summarize where appropriate 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84

Ethernet Tag Ethernet Identifier V2# show bgp l2vpn evpn 192.168.10.0 Segment (Ethtag) Identifier (ESI) IP Address Route Type: BGP routing table information for VRF default, address Length IP Address IP Prefix family L2VPN EVPN Route Distinguisher: 10.10.10.101:3 BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.10.101]/224, version 4 Paths: (1 available, best #1) Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked Next-Hop Advertised path-id 1 IP Address Path type: internal, path L3VNIis valid, is best path, no labeled nexthop AS-Path: NONE, path (MPLS sourced Label) internal to AS 10.200.200.101 (metric 3) from L3VNI 10.10.10.201 Encap:8 (10.10.10.201) Route Target Origin IGP, MED not set, localpref 100, VXLAN weight 0 Router MAC Received label 5000 Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01 Originator: 10.10.10.101 Cluster list: 10.10.10.201 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 85

Introducing VXLAN Src, Dst and Hop-by- Hop MAC Src and Dst IP Address UDP Dst Port 4789 VXLAN VNI MAC 802.1q IP Payload CRC Original Layer-2 Frame Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC Data-Plane (VXLAN) UDP Src Port Hash of L2/L3/L4 headers of original Frame 20-byte + 8-byte +8-byte + 14-byte* = 50 Bytes of total overhead *plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86

20 Bytes 8 Bytes 14 Bytes (4 Bytes Optional) 8 Bytes VXLAN Frame Format MAC in IP Encapsulation Field Value Bites Total Field Value Bites Total Dest. MAC Address Next-Hop MAC Address 48 Source Port L2/L3/L4 Hash 16 Src. MAC Address Next-Hop MAC Address 48 VLAN Type 0x8100 16 VLAN ID Tag 16 Ether Type 0x0800 16 Destination Port 4789 (UDP) 16 UDP Length 16 Checksum 0x0000 16 Outer MAC Outer IP UDP VXLAN Inner MAC Payload CRC Field Value Bites Total IP Header Misc. Data 72 Protocol 0x11 (UDP) 8 Header Checksum Various 16 Source IP Src, IP 32 Field Value Bites Total VXLAN Flags RRRRIRRR 8 Reserved 24 VNI 16M Possible Segments 24 Reserved 8 Destination IP Dest. IP 32 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 87

No Path Diversity Equal Cost Multi-Pathing (ECMP) uses Header information to form Path Diversity 101010110101010 10101010 AS#65500 Some Tunnel Protocol provide no diversity in IP or Protocol Header As a Result, all Packets travel the same Path No Path Diversity or Entropy 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88

Introducing VXLAN Entropy VXLAN provides variable UDP Source Port in Outer Header 101010110101010 10101010 AS#65500 Hash of the inner Layer-2/Layer- 3/Layer-4 Headers of the original Ethernet Frame. Enables entropy for ECMP Load balancing in the Network 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 89

Introducing VXLAN Entropy AS#65500 Entropy happens here Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC Data-Plane (VXLAN) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 90

Difference between VXLAN (F&L) and VXLAN (EVPN)? F&L Flood & Learn Data-Plane Encapsulation Layer-2 MAC-in-IP Encapsulation Follows Ethernet Semantics Learning through Flooding No Layer-3 First-Hop Gateway, Multi-Tenancy Uses Multicast for BUM BUM Broadcast, Unknown Unicast, Multicast Some static Ingress Replication (IR) possible EVPN BGP EVPN Control-Plane + Data-Plane Solution Layer-2 MAC-in-IP Encapsulation with Reachability Protocol Follows local/remote learning through Control-Plane Learn Local, advertise to Remote Integrated Layer-2 and Layer-3 First-Hop Gateway, Multi-Tenancy Uses Multicast or Ingress Replication for BUM Ingress Replication (IR) / Head-End Replication (HER) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91

VXLAN and BGP EVPN Putting it Together Control-Plane (BGP EVPN) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101/48 3001 65500:3001 192.168.10.101/32 5000 65500:5000 10.200.200.101 Dst IP 10.200.200.101 L2VNI 3001 Dst MAC 0000.3001.1101 Dst IP 192.168.10.101 Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC Data-Plane (VXLAN) Bridging 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 92

VXLAN and BGP EVPN Putting it Together Control-Plane (BGP EVPN) Extended Community Router MAC 0200.0ade.de01 Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101/48 3001 65500:3001 192.168.10.101/32 5000 65500:5000 10.200.200.101 Dst IP 10.200.200.101 L3VNI 5000 Router MAC 0200.0ade.de01 Dst IP 192.168.10.101 Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC Data-Plane (VXLAN) Routing 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 93

Routing and the Router MAC Ethernet Router MAC SMAC DMAC SIP DIP 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101 Payload SMAC DMAC SIP DIP 0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 Payload SMAC DMAC SIP DIP 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101 Payload SVI10 192.168.10.1 Switch Switch SVI20 192.168.20.1 interface: Eth2/1 MAC: 0200.0ade.de01 IP: 10.200.200.1 interface: Eth2/1 MAC: 0200.0ade.de07 IP: 10.200.200.7 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 94

Routing and the Router MAC VXLAN Router MAC SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101 Payload SMAC DMAC SIP DIP 0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 Payload SMAC DMAC SIP DIP 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101 Payload SVI10 192.168.10.1 VXLAN SVI20 192.168.20.1 interface: NVE1 MAC: 0200.0ade.de01 IP: 10.200.200.1 interface: NVE1 MAC: 0200.0ade.de07 IP: 10.200.200.7 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 95

Packet Walk ARP Request Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101 Overlay SIP DIP VXLAN SMAC DMAC 10.200.200.101 239.0.0.1 3001 0000.3001.1101 FFFF.FFFF.FFFF ARP Request for 192.168.10.102 ARP Request for 192.168.10.102 ARP Request for 192.168.10.102 SMAC: 0000.3001.1101 DMAC: FFFF.FFFF.FFFF SMAC: 0000.3001.1101 DMAC: FFFF.FFFF.FFFF Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 96

Packet Walk ARP Response Type MAC / Length / L2VNI / RT / RT IP IP / Length / L3VNI / RT / RT Next-Hop Seq. 2 2 0000.3001.1101 / 48 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101 Overlay SIP DIP VXLAN SMAC DMAC 10.200.200.104 10.200.200.101 3001 0000.3001.1102 0000.3001.1101 ARP Response for 192.168.10.102 ARP Response for 192.168.10.102 ARP Response for 192.168.10.102 SMAC: 0000.3001.1102 DMAC: 0000.3001.1101 SMAC: 0000.3001.1102 DMAC: 0000.3001.1101 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 97

Packet Walk Bridging Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101 2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102/32 5000, 65500:5000 10.200.200.104 Overlay SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.101 10.200.200.104 3001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102 Payload SMAC DMAC SIP DIP SMAC DMAC SIP DIP 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 98

Packet Walk Routing Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101 2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107 Overlay SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101 Payload Router MAC SMAC DMAC SIP DIP 2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101 SMAC DMAC SIP DIP 0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 99

Packet Walk Routing (Silent Host) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101 5 192.168.20.0/24 5000, 65500:5000 10.200.200.107 Overlay SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101 Payload SMAC DMAC SIP DIP 0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 100

Packet Walk Routing (Silent Host) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101 5 192.168.20.0/24 5000, 65500:5000 10.200.200.107 2 0000.3002.21o1 / 48 3002, 65500:3002 192.168.20.101 5000, 65500:5000 10.200.200.107 Overlay SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101 Payload ARP Response for 192.168.20.101 SMAC: 0000.3002.2101 DMAC: 2020.0000.AAAA ARP Request for 192.168.20.101 SMAC DMAC SIP DIP 0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101 SMAC: 2020.0000.AAAA DMAC: FFFF.FFFF.FFFF Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 101

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

What is Multi-Tenancy? Segregation at Layer-2 VLAN Layer-2 VNI (L2VNI) VLAN Significance Per-Fabric Per-Switch Per-Port Segregation at Layer-3 VRF Layer-3 VNI (L3VNI) VRF Significance Per-Fabric Per-Switch 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 103

Layer-2 Multi-Tenancy Bridge Domains Bridge Domain Layer-2 Segment from End-Point to End-Point Overlay Host A VLAN 10 VNI 3001 (L2VNI) VLAN 10 Host B VLAN 100 VLAN 100 Host C VLAN 20 Bridge Domains in VXLAN consists of The Ethernet Segment (VLAN) between Host and Edge Device The Hardware Resources within the Edge Device The VXLAN Segment (VNI) between Edge Device and Edge Device 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 104

Layer-3 Multi-Tenancy Routing Domains Overlay Host A 192.168.10.101 VNI 5000 (L3VNI) Host B 192.168.10.102 Host C 192.168.20.101 Routing Domain Multiple Subnets sharing the same Layer-3 forwarding policy Routing Domain in VXLAN consists of The Routing Domain local to the Edge Device (VRF) The Routing Domain (VPN) across the Edge Devices Multi-Protocol BGP with EVPN Address-Family 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 105

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

Distributed IP Anycast Gateway 192.168.10.1 2020.0000.AAAA 192.168.20.1 2020.0000.AAAA Overlay Distributed First-Hop Routing on Edge Device All Edge Device share same Gateway IP and MAC address Pervasive Gateway approach Gateway is always active No redundancy protocol for hello or state exchange Distributed and smaller state Only local End-Points ARP entries 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 107

Anycast One-to-Nearest Association Overlay Network Addressing and Routing Methodology Datagrams sent from a single Sender to the Topologically Nearest Node Group of potential Receivers, all identified by the same Destination Address Host A Host B Host C *L3VNI: VNI for all Routing operation ( VRF-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 109

Packet Walk Symmetric IRB (A to C) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101 2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107 Overlay SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101 Payload SMAC DMAC SIP DIP 2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101 SMAC DMAC SIP DIP 0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 110

Packet Walk Symmetric IRB (C to A) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101 2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107 Overlay SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.107 10.200.200.101 5000 0200.0ade.de07 0200.0ade.de01 192.168.20.101 192.168.10.101 Payload SMAC DMAC SIP DIP 2020.0000.AAAA 0000.3001.1101 192.168.20.101 192.168.10.101 SMAC DMAC SIP DIP 0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 111

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

VPC Gateway Redundancy VPC Virtual Port-Channel Multi-Chassis Link Aggregation Layer-2 Multihoming Extended for VXLAN Host-side Dual-Connect Hosts Using Port-Channels Overlay VPC VPC Fabric-side Individual s Using a common Anycast Seen as one from remote Nodes 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 113

VPC Gateway Redundancy A VXLAN perspective Overlay Both sharing an Anycast VPC Individual Node with unique Identity Individual Node with unique Identity 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 114

VPC Gateway Redundancy A VXLAN perspective Overlay VPC interface loopback0 description RID ip address 10.10.10.102/32 interface loopback0 description RID ip address 10.10.10.103/32 interface loopback1 description ip address 10.200.200.102/32 ip address 10.200.200.123/32 secondary interface loopback1 description ip address 10.200.200.103/32 ip address 10.200.200.123/32 secondary Anycast IP Address Anycast IP Address 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 115

Host Advertisements with VPC Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123 Overlay VPC VPC Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 116

Host Advertisements with VPC Independent Devices in the EVPN Control-Plane Individual Router and Peering Unique Route Distinguisher (RD) Independent Underlay Routing Devices Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123 VPC 2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123 Overlay VPC Common VXLAN Device Next-Hop is Anycast Underlay ECMP Load Share to Anycast Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 117

ECMP to the Anycast Underlay VPC AS#65500 101010110101010 10101010 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 118

Bridging to a VPC Domain VXLAN SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.104 10.200.200.123 3001 0000.3001.1102 0000.3001.1101 192.168.10.102 192.168.10.101 Payload VPC AS#65500 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 119

Routing to a VPC Domain VXLAN SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.10.101 Payload Local Station or Virtual MAC VPC AS#65500 Host A MAC: 0000.3001.1101 IP: 192.168.10.101 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 120

VPC Gateway Redundancy A VXLAN perspective VPC provides Layer-2 Gateway Redundancy From the VXLAN perspective, the next-hop is always the Anycast (VIP) Optimal for direct attached Hosts 1:1 Multicast-to-VNI mapping VPC operates at Layer-2 MAC is Synchronized Local IP (ARP) is Synchronized Routing Tables are not Synchronized 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 121

Subnet Route Advertisement with VPC Type IP / Length L3VNI / RT Next-Hop Seq. Subnet Route Advertisement Route Type 5 Next-Hop is Anycast Ensure Sync of Subnet Dual-Connect Networks (Point-2- Point not Layer-3 over VPC) Synchronize Routing Table Advertise Route Type 5 with individual IP (PIP) 5 192.168.11.0 /24 5000, 65500:5000 10.200.200.123 5 192.168.22.0 /24 5000, 65500:5000 10.200.200.123 Overlay VPC VPC Subnet X 192.168.11.0/24 Subnet Y 192.168.22.0/24 Host B MAC: 0000.3001.1102 IP: 192.168.10.102 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 122

Subnet Route Advertisement with VPC SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101 Payload Subnet X 101010110101010 192.168.11.0/24 10101010 VPC AS#65500 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 123

VPC Dual-Attach Networks SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101 Payload Subnet X 101010110101010 192.168.11.0/24 10101010 Layer-3 Point-2-Point (not Layer-3 over VPC!) VPC AS#65500 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 124

VPC Synchronizing the Routing SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101 Payload Subnet X 101010110101010 192.168.11.0/24 10101010 Dedicated Routing Session (per-vrf) VPC AS#65500 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 125

VPC Advertise Subnet Individually (Advertise-PIP) SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.107 10.200.200.102 5000 0200.0ade.de07 0200.0ade.de02 192.168.20.101 192.168.11.101 Payload Subnet X 101010110101010 192.168.11.0/24 10101010 VPC AS#65500 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 Type IP / Length L3VNI / RT Next-Hop Seq. 5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 126

VPC Advertise Subnet Individually (Advertise-PIP) SIP DIP VXLAN SMAC DMAC SIP DIP 10.200.200.107 10.200.200.102 5000 0200.0ade.de07 0200.0ade.de02 192.168.20.101 192.168.11.101 10.200.200.107 10.200.200.103 5000 0200.0ade.de07 0200.0ade.de03 192.168.20.101 192.168.11.101 Payload Subnet X 101010110101010 192.168.11.0/24 10101010 VPC AS#65500 Host C MAC: 0000.3002.2101 IP: 192.168.20.101 Type IP / Length L3VNI / RT Next-Hop Seq. 5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102 5 192.168.11.0/24 5000, 65500:5000 10.200.200.103 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 127

Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

A Deployment Story

Scalable Data Center Fabric VXLAN based Data Center Fabric BGP EVPN Control-Protocol (Overlay) OSPF for Underlay Routing (Unicast) PIM ASM with Anycast-RP for BUM Replication (Underlay) Distributed IP Anycast Gateway 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 130

A Deployment Story Underlay Routing interface loopback0 ip address 10.10.10.101/32 ip router ospf UNDERLAY area 0.0.0.0 p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 interface loopback0 ip address 10.10.10.201/32 ip router ospf UNDERLAY area 0.0.0.0 router ospf UNDERLAY router-id 10.10.10.101 router ospf UNDERLAY router-id 10.10.10.201 interface Ethernet1/1 mtu 9192 ip address 10.1.1.1/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode Underlay interface Ethernet1/1 mtu 9192 ip address 10.1.1.2/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface Ethernet1/2 mtu 9192 ip address 10.1.1.6/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface Ethernet1/3 mtu 9192 ip address 10.1.1.10/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 132

A Deployment Story Underlay Routing p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 interface loopback0 ip address 10.10.10.101/32 ip router ospf UNDERLAY area 0.0.0.0 interface loopback0 router ip address ospf UNDERLAY 10.10.10.102/32 router-id ip router 10.10.10.101 ospf UNDERLAY area 0.0.0.0 interface loopback0 ip address 10.10.10.201/32 ip router ospf UNDERLAY area 0.0.0.0 router ospf UNDERLAY router-id 10.10.10.201 interface router ospf Ethernet1/1 UNDERLAY mturouter-id 9192 10.10.10.102 ip address 10.1.1.1/30 ipinterface ospf network Ethernet1/1 point-to-point ip mtu router 9192ospf UNDERLAY area 0.0.0.0 ip ipimaddress sparse-mode 10.1.1.5/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode Underlay interface Ethernet1/1 mtu 9192 ip address 10.1.1.2/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface Ethernet1/2 mtu 9192 ip address 10.1.1.6/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface Ethernet1/3 mtu 9192 ip address 10.1.1.10/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 133

A Deployment Story Underlay Routing p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 interface loopback0 ip address 10.10.10.101/32 ip router ospf UNDERLAY area 0.0.0.0 interface loopback0 router ip address ospf UNDERLAY 10.10.10.102/32 router-id ip router 10.10.10.101 ospf UNDERLAY area 0.0.0.0 interface loopback0 interface router ip address ospf Ethernet1/1 UNDERLAY 10.10.10.103/32 mturouter-id 9192 ip router 10.10.10.102 ospf UNDERLAY area 0.0.0.0 ip address 10.1.1.1/30 ipinterface ospf router network ospf Ethernet1/1 UNDERLAY point-to-point ip mtu router router-id 9192ospf 10.10.10.103 UNDERLAY area 0.0.0.0 ip ipimaddress sparse-mode 10.1.1.5/30 ipinterface ospf network Ethernet1/1 point-to-point ip mtu router 9192ospf UNDERLAY area 0.0.0.0 ip ipimaddress sparse-mode 10.1.1.9/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode Underlay interface loopback0 ip address 10.10.10.201/32 ip router ospf UNDERLAY area 0.0.0.0 router ospf UNDERLAY router-id 10.10.10.201 interface Ethernet1/1 mtu 9192 ip address 10.1.1.2/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface Ethernet1/2 mtu 9192 ip address 10.1.1.6/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface Ethernet1/3 mtu 9192 ip address 10.1.1.10/30 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 134

A Deployment Story Underlay Routing interface loopback0 ip address 10.10.10.202/32 ip router ospf UNDERLAY area 0.0.0.0 p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 interface loopback0 ip address 10.10.10.203/32 ip router ospf UNDERLAY area 0.0.0.0 interface loopback254 ip address 10.254.254.1/32 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode ip pim anycast-rp 10.254.254.1 10.254.254.202 ip pim anycast-rp 10.254.254.1 10.254.254.203 ip pim rp-address 10.254.254.1 Underlay interface loopback254 ip address 10.254.254.1/32 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode ip pim anycast-rp 10.254.254.1 10.254.254.202 ip pim anycast-rp 10.254.254.1 10.254.254.203 ip pim rp-address 10.254.254.1 interface loopback0 ip address 10.10.10.101/32 ip router ospf UNDERLAY area 0.0.0.0 ip pim rp-address 10.254.254.1 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 135

A Deployment Story Underlay Routing interface loopback0 ip address 10.10.10.202/32 ip router ospf UNDERLAY area 0.0.0.0 p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 interface loopback0 ip address 10.10.10.203/32 ip router ospf UNDERLAY area 0.0.0.0 interface loopback254 ip address 10.254.254.1/32 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode ip pim anycast-rp 10.254.254.1 10.254.254.202 ip pim anycast-rp 10.254.254.1 10.254.254.203 ip pim rp-address 10.254.254.1 Underlay interface loopback254 ip address 10.254.254.1/32 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode ip pim anycast-rp 10.254.254.1 10.254.254.202 ip pim anycast-rp 10.254.254.1 10.254.254.203 ip pim rp-address 10.254.254.1 interface loopback0 interface loopback0 ip address 10.10.10.102/32 ip router ospf UNDERLAY area 0.0.0.0 ip address 10.10.10.101/32 ip pim rp-address 10.254.254.1 ip router ospf UNDERLAY area 0.0.0.0 ip pim rp-address 10.254.254.1 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 136

A Deployment Story Underlay Routing interface loopback0 ip address 10.10.10.202/32 ip router ospf UNDERLAY area 0.0.0.0 p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 interface loopback0 ip address 10.10.10.203/32 ip router ospf UNDERLAY area 0.0.0.0 interface loopback254 ip address 10.254.254.1/32 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode ip pim anycast-rp 10.254.254.1 10.254.254.202 ip pim anycast-rp 10.254.254.1 10.254.254.203 ip pim rp-address 10.254.254.1 Underlay interface loopback254 ip address 10.254.254.1/32 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode ip pim anycast-rp 10.254.254.1 10.254.254.202 ip pim anycast-rp 10.254.254.1 10.254.254.203 ip pim rp-address 10.254.254.1 interface loopback0 ip address 10.10.10.103/32 ip router ospf UNDERLAY area 0.0.0.0 interface loopback0 ip address 10.10.10.102/32 ip pim rp-address 10.254.254.1 ip router ospf UNDERLAY area 0.0.0.0 interface loopback0 ip address 10.10.10.101/32 ip pim rp-address 10.254.254.1 ip router ospf UNDERLAY area 0.0.0.0 ip pim rp-address 10.254.254.1 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 137

A Deployment Story the p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 Underlay interface loopback1 ip address 10.200.200.101/32 ip router ospf UNDERLAY area 0.0.0.0 interface nve1 source-interface loopback1 host-reachability protocol bgp 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 138

A Deployment Story the p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 Underlay interface loopback1 ip address 10.200.200.102/32 interface loopback1 ip router ospf UNDERLAY area 0.0.0.0 ip address 10.200.200.101/32 ip router interface ospf UNDERLAY nve1 area 0.0.0.0 source-interface loopback1 interface nve1 host-reachability protocol bgp source-interface loopback1 host-reachability protocol bgp 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 139

A Deployment Story the p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 Underlay interface loopback1 ip address 10.200.200.103/32 interface loopback1 ip router ospf UNDERLAY area 0.0.0.0 ip address 10.200.200.102/32 interface loopback1 ip router interface ospf UNDERLAY nve1 area 0.0.0.0 ip address 10.200.200.101/32 source-interface loopback1 ip router interface ospf UNDERLAY nve1 area 0.0.0.0 host-reachability protocol bgp source-interface loopback1 interface nve1 host-reachability protocol bgp source-interface loopback1 host-reachability protocol bgp 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 140

A Deployment Story Overlay Control-Plane router bgp 65500 router-id 10.10.10.202 neighbor 10.10.10.0/24 remote-as 65500 update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client Underlay p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 router bgp 65500 router-id 10.10.10.203 neighbor 10.10.10.0/24 remote-as 65500 update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client router bgp 65500 router bgp router-id 65500 10.10.10.102 router-id neighbor 10.10.10.101 10.10.10.202 remote-as 65500 neighbor 10.10.10.202 update-source remote-as loopback0 65500 update-source address-family loopback0 l2vpn evpn address-family send-community l2vpn evpn both send-community neighbor both 10.10.10.203 remote-as 65500 neighbor 10.10.10.203 update-source remote-as loopback0 65500 update-source address-family loopback0 l2vpn evpn address-family send-community l2vpn evpn both send-community both 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 142

A Deployment Story Overlay Control-Plane router bgp 65500 router-id 10.10.10.202 neighbor 10.10.10.0/24 remote-as 65500 update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client Underlay p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 router bgp 65500 router-id 10.10.10.203 neighbor 10.10.10.0/24 remote-as 65500 update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client router bgp 65500 router bgp router-id 65500 10.10.10.103 router bgp router-id neighbor 65500 10.10.10.102 10.10.10.202 remote-as 65500 router-id neighbor 10.10.10.101 10.10.10.202 update-source remote-as loopback0 65500 neighbor 10.10.10.202 update-source address-family remote-as loopback0 l2vpn evpn 65500 update-source address-family send-community loopback0 l2vpn evpn both address-family send-community neighbor l2vpn evpn both 10.10.10.203 remote-as 65500 send-community neighbor both 10.10.10.203 update-source remote-as loopback0 65500 neighbor 10.10.10.203 update-source address-family remote-as loopback0 l2vpn evpn 65500 update-source address-family send-community loopback0 l2vpn evpn both address-family send-community l2vpn evpn both send-community both 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 143

A Deployment Story Layer-2 Service Overlay vlan 100 vn-segment 30001 name Blue vlan 200 vn-segment 30002 name Green p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 evpn vni 30001 rd auto route-target both auto vni 30002 rd auto route-target both auto interface nve1 source-interface loopback1 host-reachability protocol bgp member vni 30001 mcast-group 239.239.239.1 member vni 30002 mcast-group 239.239.239.2 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 144

A Deployment Story Layer-3 Service vlan 2001 vn-segment 50001 interface Vlan2001 mtu 9192 vrf member VRF-A ip forward no ip redirects Overlay p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 vrf context VRF-A vni 50001 rd auto address-family ipv4 unicast route-target both auto route-target both auto evpn address-family ipv6 unicast route-target both auto route-target both auto evpn interface nve1 source-interface loopback1 host-reachability protocol bgp member vni 50001 associate-vrf router bgp 65500 vrf VRF-A address-family ipv4 unicast advertise l2vpn evpn redistribute direct route-map TAG 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 145

A Deployment Story First-Hop Gateway interface Vlan100 mtu 9192 vrf member VRF-A ip address 192.168.1.1/24 tag 21921 fabric forwarding mode anycast-gateway p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 router bgp 65500 vrf VRF-A address-family ipv4 unicast advertise l2vpn evpn redistribute direct route-map TAG interface Vlan200 mtu 9192 vrf member VRF-A ip address 10.10.10.1/24 tag 21921 fabric forwarding mode anycast-gateway Overlay route-map TAG permit 10 match tag 21921 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 146

A Deployment Story interface Ethernet 2/1.10 vrf member VRF-A ip address 172.16.0.1/30 encapsulation dot1q 5 interface Ethernet 2/1.20 vrf member VRF-B ip address 172.16.0.1/30 encapsulation dot1q 6 router bgp 65500 vrf VRF-A address-family ipv4 unicast advertise l2vpn evpn Overlay aggregate-address 10.10.10.0/24 summary-only aggregate-address 192.168.1.0/24 summary-only neighbor 172.16.0.1 remote-as 65599 update-source Ethernet2/1.10 address-family ipv4 unicast interface Ethernet 1/15.21 vrf member VRF-A ip address 172.16.0.2/30 encapsulation dot1q 5 interface Ethernet 1/15.22 vrf member VRF-B ip address 172.16.0.2/30 encapsulation dot1q 6 router bgp 65599 vrf VRF-A address-family ipv4 unicast neighbor 172.16.0.1 remote-as 65500 update-source Ethernet1/15.21 address-family ipv4 unicast p2p Agg: 10.1.1.0/24 RID Agg: 10.10.10.0/24 Agg: 10.200.200.0/24 RP Agg: 10.254.254.0/24 Subnet B 192.168.20.0/24 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 147

Summary

Summary Multi-Tier Topologies based on and s (aka Clos) New paradigm with Hierarchical Overlays Overlays (VXLAN) for Network Virtualization Different flavors of Overlay Solution (Flood&Learn and BGP EVPN) Layer-3 in the Underlay Defines the Topology Layer-2 and Layer-3 in the Overlay Defines the Services End-Points State exists in the Overlay BGP EVPN for integrated Layer-2 and Layer-3 Services Control-Plane driven Optimal Routing and Bridging Avoid hair pinging and reduced failure domains 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Links & Resources VXLAN Multi-Site Intro https://blogs.cisco.com/datacenter/vxlan-innovations-vxlan-evpn-multi-site-part-2-of-2 VXLAN Multi-Site @ Cisco Live online https://www.ciscolive.com/global/on-demand-library/?search=brkdcn-2035#/ ebgp for EVPN https://learningnetwork.cisco.com/blogs/community_cafe/2017/11/02/vxlan-ebgp-evpnthe-incarnation-of-a-hybrid-guest-post Configuration Example https://communities.cisco.com/community/technology/datacenter/data-centernetworking/blog/2015/05/19/vxlanevpn-configuration-example 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 151

Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/. 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public