Building Data Center Networks with VXLAN EVPN Overlays Part I

Transcription

1

2 BRKDCT-2949 Building Data Center Networks with VXLAN EVPN Overlays Part I Lukas Krattiger, Principal Engineer

3 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#brkdct Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Session Objective A short Overview on Data Center Fabric A close look at Single Fabric Overlay and Underlay Details on Single Fabric Control- & Data-Plane Multi-Tenancy in VXLAN BGP EVPN environments First-Hop Gateway with Distributed Anycast Gateway Multi-Homing with Virtual Port-Channel (VPC) for VXLAN BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 5

5 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

6 Introduction to Data Center Fabrics

7 Data Center Fabric Journey (Standalone) Layer-3 Layer-2 HSRP HSRP Spanning-Tree Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Baremet al Baremet al Baremet al Baremet al 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8

8 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

9 The / Topology (Clos* Network) Wide ECMP: Unicast or Multicast Uniform Reachability Deterministic Latency High Redundancy On Node or Link Failure *Clos, Charles (1953) "A study of non-blocking switching networks" 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

10 A Scale Out Architecture Smallest Operational Entity s Wide vs. Big Uplinks Symmetric to all s or Pods SAYG: Scale as You Grow More More Bandwidth More Resiliency More More Ports More Capacity 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

11 The Super- Super Super Super POD 1 POD Cisco and/or its affiliates. All rights reserved. Cisco Public 12

12 The Super- Super Scale Out Not Limited to Port Density Simpler Capacity Planning Super Super Beyond a single Server Room Allows Interconnecting Pods Retains Intra-Pod Topology with Flexible Inter-Pod Connectivity POD 1 POD Cisco and/or its affiliates. All rights reserved. Cisco Public 13

13 Data Center Fabric Properties Any Subnet, Anywhere, Rapidly Any Network on Any Reduced Failure Domain Any Default Gateway on Any - Distributed Extensible Scale and Resiliency 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

14 Overlay Based Data Center: Fabrics Mobility Overlay Segmentation Scale Automated and Programmable Abstracted Consumption Model Layer-2 and Layer-3 Service Physical and Virtual Workloads 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15

15 Overlay Based Data Center: Edge Devices Network Overlays Host Overlays Overlay Overlay Hybrid Overlays Router/Switch End-Points Protocols for Resiliency/Loops Traditional VPNs VXLAN, OTV, VPLS, LISP, FP - - Overlay Virtual End-Points only Single Admin Domain VXLAN, NVGRE, STT Physical and Virtual Resiliency and Scale Cross-Organizations/Federation Open Standards 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

16 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

17 Overlay Taxonomy - Underlay Layer-3 Interface Peering Underlay Edge Device LAN Segment Virtual Server Physical Server 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

18 Overlay Taxonomy - Overlay Tunnel Encapsulation (VNI Namespace) Overlay LAN Segment Virtual Server Physical Server : VXLAN Tunnel End-Point VNI/VNID: VXLAN Network Identifier 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

19 Understanding Overlay Technologies Overlay Services Layer-2 Layer-3 Layer-2 and Layer-3 Tunnel Encapsulation Underlay Transport Network Control-Plane Peer-Discovery Route Learning and Distribution Local Learning Remote Learning Data-Plane Overlay Layer-2/Layer-3 Unicast Traffic Overlay Broadcast, Unknown Unicast, Multicast traffic (BUM traffic) forwarding Ingress Replication (Unicast) Multicast 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

20 VXLAN Evolves as the Control Plane Evolves! Back Then Yet Another Encapsulation Flood & Learn (Multicast-based) Data-Plane only 4 Years ago VXLAN for the Data Center Intra-DC Control-Plane Active Discovery Multicast and Unicast 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21

21 A single Fabric with Overlay Overlay A Single Overlay Domain End-to-End Encapsulation Closest to the Source Closest to the Destination External = = Border POD Cisco and/or its affiliates. All rights reserved. Cisco Public 22

22 What is the Elephant in the Room? 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

23 The Super- and the Overlay (Multi-POD) Super Super Super Overlay Overlay POD 1 POD Cisco and/or its affiliates. All rights reserved. Cisco Public 24

24 The Super- and the Overlay (Multi-POD) Super Super Super Still, a Single Overlay Domain End-to-End Encapsulation Closest to the Source Closest to the Destination Overlay External Connectivity = = Super- or Connected to Super- POD 1 POD Cisco and/or its affiliates. All rights reserved. Cisco Public 25

25 Attributes of Multi-POD in VXLAN BGP EVPN Underlay Super Nicely Structured and Tiered Topologies Super Super Allows Efficient Scale-Out More End-Points = More More Bandwidth, Resilience or Capacity = More or Tiers Different Control-Plane Instances (BGP AS) Overlay Overlay End-to-End Encapsulation, Flat, No Hierarchy Single Control-Plane reach all in one kitchen sink Overlay POD 1 POD Cisco and/or its affiliates. All rights reserved. Cisco Public 26

26 The Super- and the Overlay (Multi-POD) Super Super Super Overlay Scale-Out Model to Build a Large Intra-DC Network? Data Center Interconnect (DCI)? Overlay Domain Normalization (Coexistence and/or Migration)? POD 1 POD Cisco and/or its affiliates. All rights reserved. Cisco Public 27

27 VXLAN Evolves as the Control Plane Evolves! Back Then Yet Another Encapsulation Flood & Learn (Multicast-based) Data-Plane only 4 Years ago VXLAN for the Data Center Intra-DC Control-Plane Active Discovery Multicast and Unicast Today VXLAN for DCI Inter-DC Multi-Site Control- & Data-Plane Separation Failure Domain Isolation 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

28 Use-Cases for DCI Connectivity Scale-Out Model to Build a Large Intra- DC Network Data Center Interconnect (DCI) Domain Normalization (Coexistence and/or Migration) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

29 VXLAN for Interconnecting Networks VXLAN Multi-Pod VXLAN Multi-Fabric EVPN Fabric Control- #1 Plane Domain 1 BGP EVPN EVPN Fabric Control- #2 Plane Domain 2 EVPN Fabric Control-Plane #1 Domain 1 EVPN Fabric Control-Plane #2 Domain 2 Overlay Overlay Overlay Overlay VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P Bar em eta l Bar em eta l Single Data-Plane End-to-End Bar em eta l Bar em eta l Bar em etal Bar em etal DCI Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Bar em etal Bar em etal Single Fabric with End-to-End Encapsulation Build Hierarchy in the Underlay Flatten it in the Overlay Multiple Fabrics Normalized through Ethernet Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

30 VXLAN for Interconnecting Networks 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31

31 VXLAN for Interconnecting Networks VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site EVPN Fabric Control- #1 Plane Domain 1 BGP EVPN EVPN Fabric Control- #2 Plane Domain 2 EVPN Fabric Control-Plane #1 Domain 1 EVPN Fabric Control-Plane #2 Domain 2 EVPN Fabric Control-Plane #1 Domain 1 BGP EVPN EVPN Fabric Control-Plane #2 Domain 2 Overlay Overlay Overlay Overlay Overlay Overlay VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P VTE P Bar em eta l Bar em eta l Single Data-Plane End-to-End Bar em eta l Bar em eta l Bar em etal Bar em etal DCI Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Bar em etal Bar em etal Bar em etal Bar em etal DCI Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Bar em etal Bar em etal Single Fabric with End-to-End Encapsulation Build Hierarchy in the Underlay Flatten it in the Overlay Multiple Fabrics Normalized through Ethernet Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3) Multiple Fabrics with Integrated DCI Integrated DCI Scaling within and between Fabrics 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

32 VXLAN Multi-Site Introducing Overlay Hierarchies Border Gateways (BGW) (Key Functional Components of VXLAN Multi-Site Architecture) Super Super Super Multi-Site Overlay Overlay Overlay Site 1 Site Cisco and/or its affiliates. All rights reserved. Cisco Public 33

33 VXLAN Multi-Site Introducing Overlay Hierarchies Super Multiple Overlay Domains Overlay Super Super Multi-Site Overlay Per-Site Encapsulation Closest to the Source Closest to the Destination Exit/Transit via Border Gateway (BGW) Overlay Multi-Site and/or External = = Border Super- becomes Transit Site 1 Site Cisco and/or its affiliates. All rights reserved. Cisco Public 34

34 VXLAN Multi-Site for Interconnecting Networks Your Happy Place! 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35

35 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

36 MTU and Overlays Data Center often require Jumbo MTU Most Server NIC support up to 9000 Bytes Network Switches support MTU up to 9216* Bytes Accommodates Jumbo MTU plus Overlay overhead Avoid Fragmentation Adjust the Transport Network with appropriate MTU *Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 Traffic 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

37 Interface Principles Routed Ports and Interfaces Layer-3 Interfaces between and (no switchport) For each Point-2-Point (P2P) connection, minimum /31 required Alternative, use IP Unnumbered (/32) Underlay Use Loopback as Source- Interface for (NVE*) *NVE: Network Virtualization Edge 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

38 IP Addressing Principles Prepare a IP Addressing Plan Separate Interface functions through IP Addressing (Aggregates) Unicast Routing Routing Protocol Peering (p2p*) Unicast Routing Routing Identifier (RID) and VPC Multicast Rendezvous-Point (RP) IPv4 only (today) p2p* Links / IP Unnumbered Underlay Loopback Routing Identifier Routing Identifier Rendezvous Point p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 *p2p: Point-to-Point 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

39 IP Addressing Principles interface ethernet4/4 description p2p-to- ip address /30 interface ethernet1/49 description p2p-to- ip address /30 interface loopback254 description RP ip address /32 interface loopback0 description RID ip address /32 Underlay interface loopback0 description RID ip address /32 interface loopback1 description ip address /32 p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: / Cisco and/or its affiliates. All rights reserved. Cisco Public 40

40 Some Math IP Addressing Principles (P2P) Example from depicted Topology 4 * 7 (28 Links) 11 Router ID (RID Loopback) 7 (Loopback) Underlay 28 Link * 2 (/31) = 56 IP Addresses 11 Router ID (RID) = 11 IP Addresses 7 = 7 IP Addresses Total: 74 IP Addresses Required 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

41 Simplifying the Math IP Unnumbered Example from depicted Topology (11 Loopback) 11 Router ID (RID Loopback) 7 (Loopback) Underlay 11 Unnumbered IF = 11 IP Addresses 11 Router ID (RID) = 11 IP Addresses 7 = 7 IP Addresses Total: 29 IP Addresses Required 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42

42 Unicast Routing OSPF and IS-IS OSPF watch your Network Type Network Type Point-2-Point Preferred (only LSA type-1) No DR/BDR election Suits well for routed interfaces/ports (optimal from a LSA DB perspective) Full SPF calculation on Link Change IS-IS what was this CLNS? Independent of IP (CLNS) Well suited for routed interfaces/ports No SPF calculation on Link change; only if Topology changes Fast Re-convergence Not everyone is familiar with it 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

43 Unicast Routing BGP ebgp Underlay Routing Service Provider style Two Different Models Two-AS Multi-AS BGP is a Distance Vector Protocol actually Path Vector Protocol AS* are used to calculate the Path (AS_Path) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

44 Unicast Routing ebgp Two-AS Model All- AS#65500 ebgp Two-AS, yes it works! Underlay All- AS#65501 ebgp peering for Underlay is not a Route-Reflector (ebgp) Retain Route-Targets Disable BGP AS-Path check Next-Hop needs to be Unchanged Underlay is Reachability! Advertise your Loopbacks 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

45 Unicast Routing ebgp Multi-AS Model All- AS#65500 Underlay ebgp Two-AS, yes it works! ebgp peering for Underlay is not a Route-Reflector (ebgp) Retain Route-Targets Disable BGP AS-Path check Next-Hop needs to be Unchanged Underlay is Reachability! Advertise your Loopbacks Changes Overlay Routing Policy Manually define Route-Targets 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46

46 Unicast Routing ebgp Model Two different BGP Peering Underlay ebgp peering for Underlay Global IPv4/v6 Address-Family Use Physical Interface IP ebgp peering for Overlay Global EVPN Address-Family Use Loopback Interface IP BFD not so ok 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

47 Unicast Routing Why two different BGP Peering? 1) Interface Down BGP Down BGP Peering (IPv4/IPv6) BGP Peering (IPv4/IPv6) AS#65500 BGP Peering (IPv4/IPv6) Point-2-Point Link Fails BGP Peering is teared down Lights-Out Event or BFD Fast reaction to Routing Table Underlay Network Converges ECMP kicks in if available/configured IGPs do this Automatically BGP Peering (IPv4/IPv6) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48

48 Unicast Routing Why two different BGP Peering? 1) Interface Down BGP Down 2) Interface Down BGP Not Down AS#65500 Point-2-Point Link Fails BGP Peering (EVPN) Loopback to Loopback Peering remains Up If Alternate Path available Timers should allow Time for Network Re-Convergence No BFD Unchanged Overlay Reachability No Mass Delete/Re-Learn Underlay Path change only 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

49 Unicast Routing Why two different BGP Peering? 1) Interface Down BGP Down 2) Interface Down BGP Not Down 3) Down Prefix are Withdrawn (RNH*) / Next-Hop: , AS#65500 and p2p Interfaces Fail Either IGP or BGP converges Loopback to Loopback Peering remains Up BGP Dead-Timer (180s) Recursive Next-Hop will trigger Convergence Event Next-Hop () disappeared in Underlay Overlay withdraws Prefixes *RNH: Recursive Next-Hop 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

50 Underlay - Unicast Routing and Overlay Specific to BGP as a Overlay Control-Protocol BGP Knobs can Help here Generic Concept for Underlay / Overlay Separation Use Different Routing Protocol Use Same Routing Protocol RNH* for Overlay works with ALL Underlay Routing Protocols Ensure /32-Reachability for s Other Routes can impact (Aggregates, Default-Route) IGP + BGP for true Protocol Separation BGP for single Routing Protocol approach 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51

51 Underlay - Multicast Routing and Rendezvous-Point PIM Any-Source-Multicast (ASM) Platform Support Nexus 9000 / Nexus 7000 (F3/M3) ASR 1000 / ASR 9000 RP Redundancy PIM Anycast-RP or MSDP Source-Trees (Unidirectional) 1 Source Tree per per Multicast Group Bidirectional PIM (Bidir) Platform Support Nexus 5600 / Nexus 7000 (F3/M3) ASR 1000 / ASR 9000 RP Redundancy Phantom-RP Shared-Trees (Bidirectional) 1 Shared Tree per Multicast Group Follows Unicast Routing Path 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52

52 Underlay PIM ASM with PIM Anycast-RP RP RP S,G S,G S,G S,G Underlay S,G 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53

53 PIM ASM S,G for 5 S,G S=1 S,G S=2 S,G S=3 S,G S=4 S,G S= Cisco and/or its affiliates. All rights reserved. Cisco Public 54

54 Underlay PIM ASM with PIM Anycast-RP interface loopback254 description RP ip address /32 ip pim sparse-mode interface loopback0 description RID ip address /32 ip pim sparse-mode ip pim anycast-rp ip pim anycast-rp ip pim rp-address RP RP (&) Underlay interface loopback254 description RP ip address /32 ip pim sparse-mode interface loopback0 description RID ipaddress /32 ip pim sparse-mode 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

55 Underlay - Multicast Routing and Rendezvous-Point The makes a good Rendezvous-Point (RP) Use multiple RP for Redundancy Watch your Multicast-Group and OIF* scale VXLAN uses Multicast for BUM Broadcast, Unknown Unicast, Multicast 1:1 Multicast-to-VNI mapping 1:N Multicast-to-VNI mapping Ingress-Replication can be valid as well *OIF: Outgoing Interface 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

56 Underlay Ingress Replication A Packet Multiplication EVPN assists, VNI Topology Various Platform Support Ie Nexus 9000 Underlay Ingress Replication Host sends 1 Packet to Edge-Device Edge-Device Encapsulates 1 Packet and multiplies it Ingress sends 1 Packet per Neighbor 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

57 VXLAN with BGP EVPN

58 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

59 What is? VXLAN Standards based Encapsulation RFC 7348 Uses UDP-Encapsulation Transport Independent Layer-3 Transport (Underlay) Flexible Namespace 24-bit field (VNID) provides ~16M unique identifier Allows Segmentations EVPN Standards based Control-Plane RFC 7432 Uses Multiprotocol BGP Uses Various Data-Planes VXLAN (EVPN-Overlay), MPLS, Provider Backbone (PBB) Many Use-Cases Covered Bridging, MAC Mobility, First-Hop & Prefix Routing, Multi-Tenancy (VPN) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

60 Introducing Ethernet VPN (EVPN) EVPN MP-BGP RFC 7432 MPLS (draft-ietf-l2vpn-evpn) Provider Backbone Bridges (draft-ietf-l2vpn-pbb-evpn) Overlay (NVO3) (draft-ietf-bess-evpn-overlay) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64

61 VXLAN and EVPN related RFCs & Drafts (IETF) ID Title Category RFC 7348 Virtual Extensible Local Area Network Data Plane RFC 7432 BGP MPLS based Ethernet VPNs Control Plane draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65

62 Multiprotocol BGP (MP-BGP) Primer AS#65500 Multiprotocol BGP (MP-BGP) Extension to Border Gateway Protocol (BGP) RFC 4760 VPN Address-Family Allows different types of Address- Families (i.e. VPNv4/v6, MVPN, L2VPN, EVPN) Various Information transported over single BGP Peering 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

63 Multiprotocol BGP (MP-BGP) Primer vrf context A rd :3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 vrf context A rd :8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 VPN Segmentation for Tenant Routing Route Distinguisher (RD) 8-byte field A Value to make a VPN Prefix unique RD + VPN Prefix [ : /24] 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

64 Multiprotocol BGP (MP-BGP) Primer vrf context A rd auto address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 vrf context A rd auto address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 Cisco provides automated Route Distinguisher derivation Macros uses Type 1 format 4-byte Router ID (RID) 4-byte VRF ID (internal number) Example of auto derived RD: : Cisco and/or its affiliates. All rights reserved. Cisco Public 68

65 Multiprotocol BGP (MP-BGP) Primer vrf context A rd auto address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 vrf context A rd auto address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 VPN Segmentation for Tenant Routing Route Target (RT) 8-byte field A Value to import/export a VPN Prefix Each RD + VPN Prefix have an RT [ : /24] [65500:5000, 65500:5000] 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69

66 Multiprotocol BGP (MP-BGP) Primer vrf context A rd auto address-family ipv4 unicast route-target import auto route-target export auto AS#65500 vrf context A rd auto address-family ipv4 unicast route-target import auto route-target export auto Cisco provides automated Route Target derivation Macros uses following values 4-byte Autonomous System 4-byte VNI Example of auto derived RD: Import, Export or Both 65500: Cisco and/or its affiliates. All rights reserved. Cisco Public 70

67 Multiprotocol BGP (MP-BGP) Primer vrf context A rd :3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 vrf context A rd :8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 RD Prefix Next-Hop Route Target : / :5000, 65500:5000 AS# : / :5001, 65500: Cisco and/or its affiliates. All rights reserved. Cisco Public 71

68 Multiprotocol BGP (MP-BGP) Primer vrf context A rd :3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 vrf context A rd :8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 RD Prefix Next-Hop Route Target : / :5000, 65500: : / :5001, 65500: Cisco and/or its affiliates. All rights reserved. Cisco Public 72

69 Multiprotocol BGP (MP-BGP) Primer vrf context A rd :3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 vrf context A rd :8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS#65500 RD Prefix Next-Hop Route Target : / :5000, 65500: : / :5001, 65500: Cisco and/or its affiliates. All rights reserved. Cisco Public 73

70 Multiprotocol BGP (MP-BGP) Primer vrf context A rd :3 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 vrf context A rd :8 address-family ipv4 unicast route-target import 65500:5000 route-target export 65500:5000 AS# /24 Next-Hop: Cisco and/or its affiliates. All rights reserved. Cisco Public 74

71 EVPN - Host and Subnet Route Distribution RR RR Host Route Distribution decoupled from the Underlay protocol Overlay Use MultiProtocol-BGP (MP- BGP) on the nodes to distribute internal Host/Subnet Routes and external reachability information Route-Reflectors (RR) deployed for scaling purposes 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75

72 EVPN Control Plane - Host and Subnet Routes BGP EVPN NLRI* Overlay Host MAC (Route Type 2) MAC only, Single VNI, Single Route Target Host MAC+IP (Route Type 2) MAC and IP, Two VNI, Two Route Target, Router MAC Internal and External Subnet Prefixes (Route Type 5) IP Subnet Prefix, Single VNI, Single Route Target *NLRI: Network Layer Reachability Information (BGP Update Format) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76

73 Host Advertisements Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: Overlay Host A MAC: Host B MAC: Host C MAC: *L2VNI: VNI for all Bridging operation ( VLAN-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77

74 Host Advertisements Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: Overlay / , 65500: Host MAC (Route Type 2) MAC MPLS Label1 (L2VNI*) Route Target for MAC-VRF MAC attributes are Mandatory Host A MAC: Host B MAC: Host C MAC: *L2VNI: VNI for all Bridging operation ( VLAN-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78

75 Ethernet Tag Ethernet Identifier V2# show bgp l2vpn evpn Segment (Ethtag) Identifier (ESI) MAC Address MAC Route Type: Length BGP routing table information for VRF default, address family Address MAC/IP L2VPN EVPN Route Distinguisher: :32777 BGP routing table entry for [2]:[0]:[0]:[48]:[ ]:[0]:[ ]/216, version 4 Paths: (1 available, best #1) Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked Next-Hop Advertised path-id 1 IP Address Path type: internal, path L2VNIis valid, is best path, no labeled nexthop AS-Path: NONE, path (MPLS sourced Label1) internal to AS (metric 3) from L2VNI Encap:8 ( ) Route Target Origin IGP, MED not set, localpref 100, VXLAN weight 0 Received label 3001 Extcommunity: RT:65500:3001 ENCAP:8 Originator: Cluster list: Cisco and/or its affiliates. All rights reserved. Cisco Public 79

76 Host Advertisements Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: / , 65500: / , 65500: Overlay Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: *L3VNI: VNI for all Routing operation ( VRF-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80

77 Host Advertisements Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: Overlay / , 65500: / , 65500: / , 65500: / , 65500: Host MAC+IP (Route Type 2) MAC and IP MPLS Label1 (L2VNI) Route Target for MAC-VRF MPLS Label2 (L3VNI*) Route Target for IP-VRF Router MAC IP Attributes are Optional Populated through ARP/ND Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: *L3VNI: VNI for all Routing operation ( VRF-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 81

78 Ethernet Tag Ethernet Identifier V2# show bgp l2vpn evpn Segment (Ethtag) Identifier (ESI) MAC Address MAC Route Type: Length BGP routing table information for VRF default, address family Address MAC/IP L2VPN EVPN Route Distinguisher: :32777 BGP routing table entry for [2]:[0]:[0]:[48]:[ ]:[32]:[ ]/272, version 4 Paths: (1 available, best #1) IP Address Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked Length IP Address Next-Hop Advertised path-id 1 IP Address L3VNI Path type: internal, L2VNI path is (MPLS valid, Label2) is best path, no labeled nexthop AS-Path: NONE, path (MPLS sourced Label1) internal to AS (metric 3) from ( ) Encap:8 Origin IGP, MED not set, localpref 100, weight 0 VXLAN Received label Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01 Originator: Cluster list: L2VNI L3VNI Route Target Router MAC Route Target 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

79 Subnet Route Advertisements Type IP / Length L3VNI / RT Next-Hop Seq / , 65500: Overlay Subnet A /24 Internal and External Subnet Prefixes (Route Type 5) IP Prefix MPLS Label (L3VNI) Route Target for IP-VRF Router MAC Populated through External Routing Protocol 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83

80 Subnet Route Advertisements Type IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: Overlay / , 65500: Subnet A / Subnet A / Subnet B /24 IP Prefix Learning via BGP with VRF-Lite via LISP on Nexus 7000/7700 via other routing protocol (static or dynamic) Default: Export of IP Host and IP Prefix Routes advertisements Filter and Summarize where appropriate 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84

81 Ethernet Tag Ethernet Identifier V2# show bgp l2vpn evpn Segment (Ethtag) Identifier (ESI) IP Address Route Type: BGP routing table information for VRF default, address Length IP Address IP Prefix family L2VPN EVPN Route Distinguisher: :3 BGP routing table entry for [5]:[0]:[0]:[24]:[ ]/224, version 4 Paths: (1 available, best #1) Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked Next-Hop Advertised path-id 1 IP Address Path type: internal, path L3VNIis valid, is best path, no labeled nexthop AS-Path: NONE, path (MPLS sourced Label) internal to AS (metric 3) from L3VNI Encap:8 ( ) Route Target Origin IGP, MED not set, localpref 100, VXLAN weight 0 Router MAC Received label 5000 Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01 Originator: Cluster list: Cisco and/or its affiliates. All rights reserved. Cisco Public 85

82 Introducing VXLAN Src, Dst and Hop-by- Hop MAC Src and Dst IP Address UDP Dst Port 4789 VXLAN VNI MAC 802.1q IP Payload CRC Original Layer-2 Frame Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC Data-Plane (VXLAN) UDP Src Port Hash of L2/L3/L4 headers of original Frame 20-byte + 8-byte +8-byte + 14-byte* = 50 Bytes of total overhead *plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86

83 20 Bytes 8 Bytes 14 Bytes (4 Bytes Optional) 8 Bytes VXLAN Frame Format MAC in IP Encapsulation Field Value Bites Total Field Value Bites Total Dest. MAC Address Next-Hop MAC Address 48 Source Port L2/L3/L4 Hash 16 Src. MAC Address Next-Hop MAC Address 48 VLAN Type 0x VLAN ID Tag 16 Ether Type 0x Destination Port 4789 (UDP) 16 UDP Length 16 Checksum 0x Outer MAC Outer IP UDP VXLAN Inner MAC Payload CRC Field Value Bites Total IP Header Misc. Data 72 Protocol 0x11 (UDP) 8 Header Checksum Various 16 Source IP Src, IP 32 Field Value Bites Total VXLAN Flags RRRRIRRR 8 Reserved 24 VNI 16M Possible Segments 24 Reserved 8 Destination IP Dest. IP Cisco and/or its affiliates. All rights reserved. Cisco Public 87

84 No Path Diversity Equal Cost Multi-Pathing (ECMP) uses Header information to form Path Diversity AS#65500 Some Tunnel Protocol provide no diversity in IP or Protocol Header As a Result, all Packets travel the same Path No Path Diversity or Entropy 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88

85 Introducing VXLAN Entropy VXLAN provides variable UDP Source Port in Outer Header AS#65500 Hash of the inner Layer-2/Layer- 3/Layer-4 Headers of the original Ethernet Frame. Enables entropy for ECMP Load balancing in the Network 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 89

86 Introducing VXLAN Entropy AS#65500 Entropy happens here Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC Data-Plane (VXLAN) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 90

87 Difference between VXLAN (F&L) and VXLAN (EVPN)? F&L Flood & Learn Data-Plane Encapsulation Layer-2 MAC-in-IP Encapsulation Follows Ethernet Semantics Learning through Flooding No Layer-3 First-Hop Gateway, Multi-Tenancy Uses Multicast for BUM BUM Broadcast, Unknown Unicast, Multicast Some static Ingress Replication (IR) possible EVPN BGP EVPN Control-Plane + Data-Plane Solution Layer-2 MAC-in-IP Encapsulation with Reachability Protocol Follows local/remote learning through Control-Plane Learn Local, advertise to Remote Integrated Layer-2 and Layer-3 First-Hop Gateway, Multi-Tenancy Uses Multicast or Ingress Replication for BUM Ingress Replication (IR) / Head-End Replication (HER) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91

88 VXLAN and BGP EVPN Putting it Together Control-Plane (BGP EVPN) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / : / : Dst IP L2VNI 3001 Dst MAC Dst IP Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC Data-Plane (VXLAN) Bridging 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 92

89 VXLAN and BGP EVPN Putting it Together Control-Plane (BGP EVPN) Extended Community Router MAC ade.de01 Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / : / : Dst IP L3VNI 5000 Router MAC ade.de01 Dst IP Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC Data-Plane (VXLAN) Routing 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 93

90 Routing and the Router MAC Ethernet Router MAC SMAC DMAC SIP DIP ade.de ade.de Payload SMAC DMAC SIP DIP :0000:AAAA Payload SMAC DMAC SIP DIP AAAA Payload SVI Switch Switch SVI interface: Eth2/1 MAC: ade.de01 IP: interface: Eth2/1 MAC: ade.de07 IP: Host A MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 94

91 Routing and the Router MAC VXLAN Router MAC SIP DIP VXLAN SMAC DMAC SIP DIP ade.de ade.de Payload SMAC DMAC SIP DIP :0000:AAAA Payload SMAC DMAC SIP DIP AAAA Payload SVI VXLAN SVI interface: NVE1 MAC: ade.de01 IP: interface: NVE1 MAC: ade.de07 IP: Host A MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 95

92 Packet Walk ARP Request Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: Overlay SIP DIP VXLAN SMAC DMAC FFFF.FFFF.FFFF ARP Request for ARP Request for ARP Request for SMAC: DMAC: FFFF.FFFF.FFFF SMAC: DMAC: FFFF.FFFF.FFFF Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 96

93 Packet Walk ARP Response Type MAC / Length / L2VNI / RT / RT IP IP / Length / L3VNI / RT / RT Next-Hop Seq / 48 / , 65500: / , 65500: Overlay SIP DIP VXLAN SMAC DMAC ARP Response for ARP Response for ARP Response for SMAC: DMAC: SMAC: DMAC: Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 97

94 Packet Walk Bridging Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: / , 65500: / , 65500: Overlay SIP DIP VXLAN SMAC DMAC SIP DIP Payload SMAC DMAC SIP DIP SMAC DMAC SIP DIP Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 98

95 Packet Walk Routing Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: / , 65500: / , 65500: Overlay SIP DIP VXLAN SMAC DMAC SIP DIP ade.de ade.de Payload Router MAC SMAC DMAC SIP DIP AAAA SMAC DMAC SIP DIP AAAA Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 99

96 Packet Walk Routing (Silent Host) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: , 65500: / , 65500: Overlay SIP DIP VXLAN SMAC DMAC SIP DIP ade.de ade.de Payload SMAC DMAC SIP DIP AAAA Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 100

97 Packet Walk Routing (Silent Host) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: , 65500: / , 65500: o1 / , 65500: , 65500: Overlay SIP DIP VXLAN SMAC DMAC SIP DIP ade.de ade.de Payload ARP Response for SMAC: DMAC: AAAA ARP Request for SMAC DMAC SIP DIP AAAA SMAC: AAAA DMAC: FFFF.FFFF.FFFF Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 101

98 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

99 What is Multi-Tenancy? Segregation at Layer-2 VLAN Layer-2 VNI (L2VNI) VLAN Significance Per-Fabric Per-Switch Per-Port Segregation at Layer-3 VRF Layer-3 VNI (L3VNI) VRF Significance Per-Fabric Per-Switch 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 103

100 Layer-2 Multi-Tenancy Bridge Domains Bridge Domain Layer-2 Segment from End-Point to End-Point Overlay Host A VLAN 10 VNI 3001 (L2VNI) VLAN 10 Host B VLAN 100 VLAN 100 Host C VLAN 20 Bridge Domains in VXLAN consists of The Ethernet Segment (VLAN) between Host and Edge Device The Hardware Resources within the Edge Device The VXLAN Segment (VNI) between Edge Device and Edge Device 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 104

101 Layer-3 Multi-Tenancy Routing Domains Overlay Host A VNI 5000 (L3VNI) Host B Host C Routing Domain Multiple Subnets sharing the same Layer-3 forwarding policy Routing Domain in VXLAN consists of The Routing Domain local to the Edge Device (VRF) The Routing Domain (VPN) across the Edge Devices Multi-Protocol BGP with EVPN Address-Family 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 105

102 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

103 Distributed IP Anycast Gateway AAAA AAAA Overlay Distributed First-Hop Routing on Edge Device All Edge Device share same Gateway IP and MAC address Pervasive Gateway approach Gateway is always active No redundancy protocol for hello or state exchange Distributed and smaller state Only local End-Points ARP entries 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 107

104 Distributed IP Anycast Gateway AAAA AAAA Overlay Distributed First-Hop Routing on Edge Device All Edge Device share same Gateway IP and MAC address Pervasive Gateway approach Gateway is always active No redundancy protocol for hello or state exchange Distributed and smaller state Only local End-Points ARP entries 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 108

105 Anycast One-to-Nearest Association Overlay Network Addressing and Routing Methodology Datagrams sent from a single Sender to the Topologically Nearest Node Group of potential Receivers, all identified by the same Destination Address Host A Host B Host C *L3VNI: VNI for all Routing operation ( VRF-VNI ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 109

106 Packet Walk Symmetric IRB (A to C) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: / , 65500: / , 65500: Overlay SIP DIP VXLAN SMAC DMAC SIP DIP ade.de ade.de Payload SMAC DMAC SIP DIP AAAA SMAC DMAC SIP DIP AAAA Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 110

107 Packet Walk Symmetric IRB (C to A) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: / , 65500: / , 65500: Overlay SIP DIP VXLAN SMAC DMAC SIP DIP ade.de ade.de Payload SMAC DMAC SIP DIP AAAA SMAC DMAC SIP DIP AAAA Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 111

108 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

109 VPC Gateway Redundancy VPC Virtual Port-Channel Multi-Chassis Link Aggregation Layer-2 Multihoming Extended for VXLAN Host-side Dual-Connect Hosts Using Port-Channels Overlay VPC VPC Fabric-side Individual s Using a common Anycast Seen as one from remote Nodes 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 113

110 VPC Gateway Redundancy A VXLAN perspective Overlay Both sharing an Anycast VPC Individual Node with unique Identity Individual Node with unique Identity 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 114

111 VPC Gateway Redundancy A VXLAN perspective Overlay VPC interface loopback0 description RID ip address /32 interface loopback0 description RID ip address /32 interface loopback1 description ip address /32 ip address /32 secondary interface loopback1 description ip address /32 ip address /32 secondary Anycast IP Address Anycast IP Address 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 115

112 Host Advertisements with VPC Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: / , 65500: / , 65500: Overlay VPC VPC Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 116

113 Host Advertisements with VPC Independent Devices in the EVPN Control-Plane Individual Router and Peering Unique Route Distinguisher (RD) Independent Underlay Routing Devices Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: VPC / , 65500: / , 65500: Overlay VPC Common VXLAN Device Next-Hop is Anycast Underlay ECMP Load Share to Anycast Host A MAC: IP: Host B MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 117

114 ECMP to the Anycast Underlay VPC AS# Host A MAC: IP: Host B MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 118

115 Bridging to a VPC Domain VXLAN SIP DIP VXLAN SMAC DMAC SIP DIP Payload VPC AS#65500 Host A MAC: IP: Host B MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 119

116 Routing to a VPC Domain VXLAN SIP DIP VXLAN SMAC DMAC SIP DIP ade.de Payload Local Station or Virtual MAC VPC AS#65500 Host A MAC: IP: Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 120

117 VPC Gateway Redundancy A VXLAN perspective VPC provides Layer-2 Gateway Redundancy From the VXLAN perspective, the next-hop is always the Anycast (VIP) Optimal for direct attached Hosts 1:1 Multicast-to-VNI mapping VPC operates at Layer-2 MAC is Synchronized Local IP (ARP) is Synchronized Routing Tables are not Synchronized 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 121

118 Subnet Route Advertisement with VPC Type IP / Length L3VNI / RT Next-Hop Seq. Subnet Route Advertisement Route Type 5 Next-Hop is Anycast Ensure Sync of Subnet Dual-Connect Networks (Point-2- Point not Layer-3 over VPC) Synchronize Routing Table Advertise Route Type 5 with individual IP (PIP) / , 65500: / , 65500: Overlay VPC VPC Subnet X /24 Subnet Y /24 Host B MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 122

119 Subnet Route Advertisement with VPC SIP DIP VXLAN SMAC DMAC SIP DIP ade.de Payload Subnet X / VPC AS#65500 Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 123

120 VPC Dual-Attach Networks SIP DIP VXLAN SMAC DMAC SIP DIP ade.de Payload Subnet X / Layer-3 Point-2-Point (not Layer-3 over VPC!) VPC AS#65500 Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 124

121 VPC Synchronizing the Routing SIP DIP VXLAN SMAC DMAC SIP DIP ade.de Payload Subnet X / Dedicated Routing Session (per-vrf) VPC AS#65500 Host C MAC: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 125

122 VPC Advertise Subnet Individually (Advertise-PIP) SIP DIP VXLAN SMAC DMAC SIP DIP ade.de ade.de Payload Subnet X / VPC AS#65500 Host C MAC: IP: Type IP / Length L3VNI / RT Next-Hop Seq / , 65500: Cisco and/or its affiliates. All rights reserved. Cisco Public 126

123 VPC Advertise Subnet Individually (Advertise-PIP) SIP DIP VXLAN SMAC DMAC SIP DIP ade.de ade.de ade.de ade.de Payload Subnet X / VPC AS#65500 Host C MAC: IP: Type IP / Length L3VNI / RT Next-Hop Seq / , 65500: / , 65500: Cisco and/or its affiliates. All rights reserved. Cisco Public 127

124 Agenda Introduction to Data Center Fabric,, Super- (CLOS) Overlay Underlay VXLAN with BGP EVPN Control & Data Plane Multi-Tenancy Distributed Anycast Gateway VPC A Deployment Story

125 A Deployment Story

126 Scalable Data Center Fabric VXLAN based Data Center Fabric BGP EVPN Control-Protocol (Overlay) OSPF for Underlay Routing (Unicast) PIM ASM with Anycast-RP for BUM Replication (Underlay) Distributed IP Anycast Gateway 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 130

127 A Deployment Story p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 Underlay 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 131

128 A Deployment Story Underlay Routing interface loopback0 ip address /32 ip router ospf UNDERLAY area p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 interface loopback0 ip address /32 ip router ospf UNDERLAY area router ospf UNDERLAY router-id router ospf UNDERLAY router-id interface Ethernet1/1 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode Underlay interface Ethernet1/1 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode interface Ethernet1/2 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode interface Ethernet1/3 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 132

129 A Deployment Story Underlay Routing p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 interface loopback0 ip address /32 ip router ospf UNDERLAY area interface loopback0 router ip address ospf UNDERLAY /32 router-id ip router ospf UNDERLAY area interface loopback0 ip address /32 ip router ospf UNDERLAY area router ospf UNDERLAY router-id interface router ospf Ethernet1/1 UNDERLAY mturouter-id ip address /30 ipinterface ospf network Ethernet1/1 point-to-point ip mtu router 9192ospf UNDERLAY area ip ipimaddress sparse-mode /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode Underlay interface Ethernet1/1 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode interface Ethernet1/2 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode interface Ethernet1/3 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 133

130 A Deployment Story Underlay Routing p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 interface loopback0 ip address /32 ip router ospf UNDERLAY area interface loopback0 router ip address ospf UNDERLAY /32 router-id ip router ospf UNDERLAY area interface loopback0 interface router ip address ospf Ethernet1/1 UNDERLAY /32 mturouter-id 9192 ip router ospf UNDERLAY area ip address /30 ipinterface ospf router network ospf Ethernet1/1 UNDERLAY point-to-point ip mtu router router-id 9192ospf UNDERLAY area ip ipimaddress sparse-mode /30 ipinterface ospf network Ethernet1/1 point-to-point ip mtu router 9192ospf UNDERLAY area ip ipimaddress sparse-mode /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode Underlay interface loopback0 ip address /32 ip router ospf UNDERLAY area router ospf UNDERLAY router-id interface Ethernet1/1 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode interface Ethernet1/2 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode interface Ethernet1/3 mtu 9192 ip address /30 ip ospf network point-to-point ip router ospf UNDERLAY area ip pim sparse-mode 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 134

131 A Deployment Story Underlay Routing interface loopback0 ip address /32 ip router ospf UNDERLAY area p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 interface loopback0 ip address /32 ip router ospf UNDERLAY area interface loopback254 ip address /32 ip router ospf UNDERLAY area ip pim sparse-mode ip pim anycast-rp ip pim anycast-rp ip pim rp-address Underlay interface loopback254 ip address /32 ip router ospf UNDERLAY area ip pim sparse-mode ip pim anycast-rp ip pim anycast-rp ip pim rp-address interface loopback0 ip address /32 ip router ospf UNDERLAY area ip pim rp-address Cisco and/or its affiliates. All rights reserved. Cisco Public 135

132 A Deployment Story Underlay Routing interface loopback0 ip address /32 ip router ospf UNDERLAY area p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 interface loopback0 ip address /32 ip router ospf UNDERLAY area interface loopback254 ip address /32 ip router ospf UNDERLAY area ip pim sparse-mode ip pim anycast-rp ip pim anycast-rp ip pim rp-address Underlay interface loopback254 ip address /32 ip router ospf UNDERLAY area ip pim sparse-mode ip pim anycast-rp ip pim anycast-rp ip pim rp-address interface loopback0 interface loopback0 ip address /32 ip router ospf UNDERLAY area ip address /32 ip pim rp-address ip router ospf UNDERLAY area ip pim rp-address Cisco and/or its affiliates. All rights reserved. Cisco Public 136

133 A Deployment Story Underlay Routing interface loopback0 ip address /32 ip router ospf UNDERLAY area p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 interface loopback0 ip address /32 ip router ospf UNDERLAY area interface loopback254 ip address /32 ip router ospf UNDERLAY area ip pim sparse-mode ip pim anycast-rp ip pim anycast-rp ip pim rp-address Underlay interface loopback254 ip address /32 ip router ospf UNDERLAY area ip pim sparse-mode ip pim anycast-rp ip pim anycast-rp ip pim rp-address interface loopback0 ip address /32 ip router ospf UNDERLAY area interface loopback0 ip address /32 ip pim rp-address ip router ospf UNDERLAY area interface loopback0 ip address /32 ip pim rp-address ip router ospf UNDERLAY area ip pim rp-address Cisco and/or its affiliates. All rights reserved. Cisco Public 137

134 A Deployment Story the p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 Underlay interface loopback1 ip address /32 ip router ospf UNDERLAY area interface nve1 source-interface loopback1 host-reachability protocol bgp 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 138

135 A Deployment Story the p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 Underlay interface loopback1 ip address /32 interface loopback1 ip router ospf UNDERLAY area ip address /32 ip router interface ospf UNDERLAY nve1 area source-interface loopback1 interface nve1 host-reachability protocol bgp source-interface loopback1 host-reachability protocol bgp 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 139

136 A Deployment Story the p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 Underlay interface loopback1 ip address /32 interface loopback1 ip router ospf UNDERLAY area ip address /32 interface loopback1 ip router interface ospf UNDERLAY nve1 area ip address /32 source-interface loopback1 ip router interface ospf UNDERLAY nve1 area host-reachability protocol bgp source-interface loopback1 interface nve1 host-reachability protocol bgp source-interface loopback1 host-reachability protocol bgp 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 140

137 A Deployment Story Overlay Control-Plane router bgp router-id neighbor /24 remote-as update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client Underlay p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 router bgp router-id neighbor /24 remote-as update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client router bgp router-id neighbor remote-as update-source loopback0 address-family l2vpn evpn send-community both neighbor remote-as update-source loopback0 address-family l2vpn evpn send-community both 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 141

138 A Deployment Story Overlay Control-Plane router bgp router-id neighbor /24 remote-as update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client Underlay p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 router bgp router-id neighbor /24 remote-as update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client router bgp router bgp router-id router-id neighbor remote-as neighbor update-source remote-as loopback update-source address-family loopback0 l2vpn evpn address-family send-community l2vpn evpn both send-community neighbor both remote-as neighbor update-source remote-as loopback update-source address-family loopback0 l2vpn evpn address-family send-community l2vpn evpn both send-community both 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 142

139 A Deployment Story Overlay Control-Plane router bgp router-id neighbor /24 remote-as update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client Underlay p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 router bgp router-id neighbor /24 remote-as update-source loopback0 address-family l2vpn evpn send-community both route-reflector-client router bgp router bgp router-id router bgp router-id neighbor remote-as router-id neighbor update-source remote-as loopback neighbor update-source address-family remote-as loopback0 l2vpn evpn update-source address-family send-community loopback0 l2vpn evpn both address-family send-community neighbor l2vpn evpn both remote-as send-community neighbor both update-source remote-as loopback neighbor update-source address-family remote-as loopback0 l2vpn evpn update-source address-family send-community loopback0 l2vpn evpn both address-family send-community l2vpn evpn both send-community both 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 143

140 A Deployment Story Layer-2 Service Overlay vlan 100 vn-segment name Blue vlan 200 vn-segment name Green p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 evpn vni rd auto route-target both auto vni rd auto route-target both auto interface nve1 source-interface loopback1 host-reachability protocol bgp member vni mcast-group member vni mcast-group Cisco and/or its affiliates. All rights reserved. Cisco Public 144

141 A Deployment Story Layer-3 Service vlan 2001 vn-segment interface Vlan2001 mtu 9192 vrf member VRF-A ip forward no ip redirects Overlay p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 vrf context VRF-A vni rd auto address-family ipv4 unicast route-target both auto route-target both auto evpn address-family ipv6 unicast route-target both auto route-target both auto evpn interface nve1 source-interface loopback1 host-reachability protocol bgp member vni associate-vrf router bgp vrf VRF-A address-family ipv4 unicast advertise l2vpn evpn redistribute direct route-map TAG 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 145

142 A Deployment Story First-Hop Gateway interface Vlan100 mtu 9192 vrf member VRF-A ip address /24 tag fabric forwarding mode anycast-gateway p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 router bgp vrf VRF-A address-family ipv4 unicast advertise l2vpn evpn redistribute direct route-map TAG interface Vlan200 mtu 9192 vrf member VRF-A ip address /24 tag fabric forwarding mode anycast-gateway Overlay route-map TAG permit 10 match tag Cisco and/or its affiliates. All rights reserved. Cisco Public 146

143 A Deployment Story interface Ethernet 2/1.10 vrf member VRF-A ip address /30 encapsulation dot1q 5 interface Ethernet 2/1.20 vrf member VRF-B ip address /30 encapsulation dot1q 6 router bgp vrf VRF-A address-family ipv4 unicast advertise l2vpn evpn Overlay aggregate-address /24 summary-only aggregate-address /24 summary-only neighbor remote-as update-source Ethernet2/1.10 address-family ipv4 unicast interface Ethernet 1/15.21 vrf member VRF-A ip address /30 encapsulation dot1q 5 interface Ethernet 1/15.22 vrf member VRF-B ip address /30 encapsulation dot1q 6 router bgp vrf VRF-A address-family ipv4 unicast neighbor remote-as update-source Ethernet1/15.21 address-family ipv4 unicast p2p Agg: /24 RID Agg: /24 Agg: /24 RP Agg: /24 Subnet B / Cisco and/or its affiliates. All rights reserved. Cisco Public 147

144 Summary

145 Summary Multi-Tier Topologies based on and s (aka Clos) New paradigm with Hierarchical Overlays Overlays (VXLAN) for Network Virtualization Different flavors of Overlay Solution (Flood&Learn and BGP EVPN) Layer-3 in the Underlay Defines the Topology Layer-2 and Layer-3 in the Overlay Defines the Services End-Points State exists in the Overlay BGP EVPN for integrated Layer-2 and Layer-3 Services Control-Plane driven Optimal Routing and Bridging Avoid hair pinging and reduced failure domains 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

146 If you haven t had enough VXLAN BGP EVPN 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 150

147 Links & Resources VXLAN Multi-Site Intro VXLAN Cisco Live online ebgp for EVPN Configuration Example Cisco and/or its affiliates. All rights reserved. Cisco Public 151

148 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

149 Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public