Taking the Risk Away from Layer 2 Interconnects

Transcription

1 Taking the Risk Away from Layer 2 Interconnects BRKDCT

2 Reference Sessions BRKDCT Active-Active Data Centre Strategies, Carlos Pereira. BRKDCT Deploying Nexus 7000 in Data Centre Networks, Nivedita Autar. BRKDCT Mobility and Virtualisation in the Data Centre with LISP and OTV, Victor Moreno. 2

3 Session BRKDCT-2840 Abstract Data Centre Networking: Taking Risk Away from Layer 2 Interconnects This intermediate session details a solution for providing a means of Layer 2 communications adjacency to support operating system clustering, file system clustering, virtual machine mobility, symmetric traffic flows, and more in a highly resilient multisite Data Centre infrastructure. Starting from the building blocks of spanning-tree implementations and considerations, the session continues with details on how to control the Layer 2 control and data planes to limit negative effects present today in geographically diverse Layer 2 domains. The emphasis is on multisite Data Centre interconnect and specifics of service advertisement and site failover. Considerations are given for tying users to either site in an active/standby, active/active per application, and active/active within an application relationship. Transport mechanisms such as tag switching, Ethernet over MPLS, Virtual Private LAN Service, MPLSoGRE, OTV, Virtual Ethernet, ServerFarm to User First Hop Redundancy, User to ServerFarm redundancy with Route Health Injection, 802.1s and w, load sharing multisite traffic on intra-data Centre VLANs, global site load balancing, and others. This session compares alternatives with direct Layer 2 links on dedicated services or DWDM lambdas, point-to-point and multipoint scenarios, configurations using existing RPVST or MST deployments within a Data Centre site, sharing Layer 2 and Layer 3 services, and operations and administration considerations. 3

4 Goals of This Session Present alternatives for interconnecting multiple Data Centre locations Present tested methods in production for minimising the risks associated with meeting these connectivity requirements. 4

5 Session Agenda Data Centre Interconnection Common Scenarios and Terms Dark Fibre / DWDM Solutions Label Based Solutions IP Based Solutions Encryption Recommended Designs for Optimising Traffic Flows Q & A 5

6 Data Centre Interconnection Common Scenarios and Terms

7 Intra-DC Domain with STP Isolation L3 L2 DC Interconnect (DCI) End-to-End Requirements Data Centre WAN Core Aggr/ Distr Access SAN WAN Same Extended VLAN SAN SAN Replication - Synchronous implies distance limitation L3 L2 Data Centre Intra-DC Domain with STP Isolation WAN Core Aggr/ Distr Access Solution requirements E2E Loop Prevention STP Isolation Redundant LAN extn. WAN load balancing Core Transparency DC site Transparency Optimal Traffic Handling VLAN Scalability Multipoint Optional Encryption HQOS STP isolation desired, storm-control (unknown unicast & broadcast) Path Optimisation Encryption requirement mainly driven by Federal Government standards Core & DC Site Transparency needed to Minimise Operational Impact 7

8 Layer 2 Use Cases Extending Operating System / File System clusters Extending Database clusters Virtual machine mobility Physical machine mobility Physical to Virtual (PtoV) Migrations Legacy devices/apps with embedded IP addressing Time to deployment and operational reasons Extend DC to solve power/heat/space limitations Data Centre co-location 8

9 Layer 2 Risks Flooding of packets between Data Centres Spanning Tree (STP) is not easily scalable and risk grows as diameter grows STP has no domain isolation issue in single DC can propagate First hop resolution and inbound service selection can cause verbose inter-data Centre traffic In general Cisco recommends L3 routing for geographically diverse locations This session focuses on making limited L2 connectivity as stable as possible 9

10 Layer 2 Solution Types Light customer owned fibre to build an extended L2 network No STP isolation between sites Virtual Switching System (VSS) / Virtual Port Channel (vpc) FabricPath (no STP) Purchase multiple wavelengths from SP Cost rises, still nothing to offer STP isolation Redesign Data Centre STP domain using Multiple Spanning Tree (MST) regions STP domain concept Fundamental change requiring large time investment Operational differences and MST database management 10

11 Layer 2 Solution Types (Con t) Implement a L2 solution to virtualise transport over L3 EoMPLS for point to point (possible STP isolation issues) Multipoint bridging using Virtual Private LAN Services (VPLS) MPLSoGRE Overlay Transport Virtualisation (OTV) Advanced VPLS (A-VPLS) 11

12 Session Agenda Data Centre Interconnection Common Scenarios and Terms Dark Fibre / DWDM Solutions Label Based Solutions IP Based Solutions Encryption Recommended Designs for Optimising Traffic Flows Q & A 12

13 Dark Fibre / DWDM Solutions

14 Layer 2 Prerequisites for All Options This session assumes a fairly detailed knowledge of Spanning Tree Protocol Items we leverage in this solution: 802.1w 802.1s Port Fast BPDU Filter BPDU Guard Root Guard Loop Guard Bridge Assurance (Catalyst 6500, Nexus 5000/5500 and 7000) 14

15 Layer 2 Extension Without Tunnels/Tags (vpc/vss) 6500 with Virtual Switching System cluster (Supported distances at 80km (ZR) Dark Fibre) Nexus 7000 with Virtual Port-Channels (Supported distances at 80km (ZR- X2) Dark Fibre) All traffic flows to a vpc/vss member node Hub-and-spoke topology from a layer 2 perspective Dedicated links to vpc/vss members from each Data Centre aggregation switch Can consume lambda or fibre strands quickly Data plane rate limiting in L2 still needs protection STP domains are not isolated unless we BPDU-filter at all vpc/vss aggregation switches 15

16 vpc / VSS Design L2 LH Fibre/DWDM L3 LH Fibre/DWDM L2 Local Fibre L3 Local Fibre Data Centre #1 Data Centre #2 vpc / VSS vpc / VSS 16

17 vpc / VSS L2 View L2 LH Fibre/DWDM L2 Local Fibre Data Centre #1 Data Centre #2 BPDU-Filtering BPDU-Filtering vpc/vss vpc/vss - vpc/vss Domain ID for facing vpc/vss layers should be different - BPDU Filter on the edge devices to avoid BPDU propagation - STP Edge Mode to provide fast failover times - No Loop must exist outside the vpc/vss domain - No L3 peering between Nexus 7000 devices (i.e. pure layer 2) 17

18 vpc / VSS Design Data Centre #3 12 Lambda/24 Strand Example 4 Additional Lambda/8 Strands per new DC L2 Service Only from Provider VSS L2 LH Fibre/DWDM L3 LH Fibre/DWDM L2 Local Fibre L3 Local Fibre Data Centre #1 Data Centre #2 VSS/vPC vpc / VSS vpc / VSS 18

19 vpc / VSS L2 View Data Centre #3 VSS L2 LH Fibre/DWDM L2 Local Fibre All links are port channels to Central VSS BPDU Filtering Data Centre #1 Data Centre #2 BPDU Filtering BPDU Filtering VSS VSS vpc/vss 19

20 vpc and Layer 3 P L2 LH Fibre/DWDM L3 LH Fibre/DWDM L2 Local Fibre L3 Local Fibre L3 Peer Data Centre #1 Data Centre #2 P P vpc vpc Nexus 7000 configured for L2 Transport only SVI passive-interface (no IGP peering) 20

21 vpc and Layer 3 P L2 LH Fibre/DWDM L3 LH Fibre/DWDM L2 Local Fibre L3 Local Fibre L3 Peer Data Centre #1 Data Centre #2 P P vpc P vpc P Peering over a vpc inter-connection on parallel routed interfaces SVI passive-interface (no IGP peering) 21

22 FabricPath Design (Partial/Full/Ring Topology) Leverage vpc+ Data Centre #3 FabricPath STP (CE) Brownfield / Greenfield DC STP Integration Conversational MAC Learning Classic Ethernet Native VLAN Pruning TTL / RPF ECMP for L2 FabricPath Data Centre #1 FabricPath Core Agg w/vpc+ Data Centre #2 22

23 Session Agenda Data Centre Interconnection Common Scenarios and Terms Dark Fibre / DWDM Solutions Label Based Solutions IP Based Solutions Encryption Recommended Designs for Optimising Traffic Flows Q & A 23

24 MPLS Solutions

25 EoMPLS (Ethernet Over MPLS) Encapsulates Ethernet frames inside MPLS packets to pass layer 3 network EoMPLS has routing separation from metro core devices providing connectivity CE flapping routes won t propagate inside MPLS Point to point links between locations Data plane rate limiting in L2 still needs protection EoMPLS Is a Pseudo-Wire CE PE PE CE MPLS 25

26 Virtual Private LAN Service (VPLS) VPLS defines an architecture that allows MPLS networks to offer Layer 2 multipoint Ethernet Services Metro Core emulates an IEEE Ethernet bridge (virtual) Virtual Bridges linked with EoMPLS Pseudo Wires Data plane rate limiting in L2 still needs protection VPLS Multipoint Services CE PE VFI PE VFI CE MPLS VFI CE 26

27 Virtual Forwarding Instance (VFI) IOS Representation of Virtual Switch Interface Flooding / Forwarding MAC table instances per customer (port/vlan) for each PE VFI will participate in learning and forwarding process Associate ports to MAC, flood unknowns to all other ports Address Learning / Aging LDP enhanced with additional MAC List TLV (label withdrawal) MAC timers refreshed with incoming frames Loop Prevention Create full-mesh of Pseudo Wire VCs (EoMPLS) Unidirectional LSP carries VCs between pair of N-PE Per VPLS Uses split horizon concepts to prevent loops 27

28 Calculating Core MTU Requirements Core MTU Edge MTU + Transport Header + (MPLS Label Stack * MPLS Header Size) Edge MTU is the MTU configured in the CE-facing PE interface Examples (all in Bytes): Edge EoMPLS Port Mode 1500 Transport 14 MPLS Stack 2 MPLS Header 4 Total 1522 EoMPLS VLAN Mode

29 End to End VPLS and EoMPLS Design Layer 3 Core Intranet WCore1 WCore2 ECore1 ECore2 DC Core Po1 WMC1 EMC1 DC Core WAgg1 VPLS / EoMPLS Domain EAgg2 Agg WAgg2 Po1 EAgg1 Agg WMC2 EMC2 Access Access Server Farm L2 Links (GE or 10GE) L3 Links (GE or 10GE) Loss of Link/Node Server Farm 29

30 Access to Aggregation Connections Rapid-PVST is existing protocol, and no desire to force a change Aggregation switches are root for all intra-dc VLANs Aggregation ARP and CAM Timers The peer aggregation switch is secondary root HSRP tested for first hop redundancy from server (more later) Server Farm 30 Agg Access

31 Layer 3 Aggregation and Core Connections Layer 3 connections from DC Core to Enterprise Core Aggregation switch L3 connected to DC Core If dual supervisor modules, need non-stop forwarding (NSF) under routing process Layer 3 Enterprise Core Hanging L3 links in diagram, are to Metro Core switches which are Ethernet over MPLS links Hanging L3 links are for peering the DC Cores in each location in a point-to-point scenario DC Core Agg Bidirectional forwarding detection (BFD) interval 100 min_rx 100 multiplier 3 31

32 EoMPLS / VPLS Infrastructure Loopbacks chosen as peering points for EoMPLS and VPLS xconnects Horizontal links represent 10GE on DWDM service between Data Centres (alternate paths) Vertical links represent intra-dc 10GE connections MPLS LDP enabled globally (not a full P / PE MPLS implementation) LDP NSF/SSO mpls ldp graceful-restart VPLS / EoMPLS Domain Links to/from aggregation switches for Layer 2, are storm-control limited for broadcasts and multicasts to 1% (protect data plane) MTU increased to 1522 bytes on the L3 MPLS links for the MPLS tagging Metro Core Metro Core 32

33 Metro Switch Interconnectivity - Link debounce timers - Aggressive-UDLD - Carrier-delay timers IGP Routing Process connecting MPLS PEs Metro Core Metro Core - Link debounce timers - Aggressive-UDLD - Carrier-delay timers L3 Links (10GE) 33

34 EoMPLS for Layer3 Layer 3 Core Intranet DC Core METRO CORE DC Core Agg PW Pseudo Wires EoMPLS Agg Metro Core Metro Core Access Access L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm Server Farm 34

35 VPLS for Layer2 Layer 3 Core Intranet DC Core METRO CORE DC Core Agg VFI Agg Metro Core PW Pseudo Wires Metro Core Access Access L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm Server Farm 35

36 VPLS for Layer2 l2 vfi vlan3700 manual vpn id 3700 neighbor encapsulation mpls neighbor encapsulation mpls neighbor encapsulation mpls Layer 3 Core Intranet l2 vfi vlan3700 manual vpn id 3700 neighbor encapsulation mpls neighbor encapsulation mpls neighbor encapsulation mpls DC Core METRO CORE DC Core Agg Server Farm Metro Core l2 vfi vlan3700 manual Access vpn id 3700 neighbor encapsulation mpls neighbor encapsulation mpls neighbor encapsulation mpls PW Pseudo Wires L2 Links (GE or 10GE) L3 Links (GE or 10GE) Metro Core l2 vfi vlan3700 manual vpn id 3700 neighbor encapsulation mpls neighbor encapsulation mpls neighbor encapsulation mpls Server Farm Agg Access 36

37 VPLS for Layer2 DC Core interface Vlan3700 no ip address load-interval 30 xconnect vfi vlan3700 Layer 3 Core Intranet METRO CORE interface Vlan3700 no ip address load-interval 30 xconnect vfi vlan3700 DC Core Agg Access interface Vlan3700 no ip address load-interval 30 xconnect vfi vlan3700 Metro Core VLAN 3700 PW Pseudo Wires L2 Links (GE or 10GE) L3 Links (GE or 10GE) Metro Core interface Vlan3700 no ip address load-interval 30 xconnect vfi vlan3700 Agg Access Server Farm Server Farm 37

38 Spanning Tree Spanning-Tree BPDUs will NOT traverse between the Data Centres It isn t needed (and blocked) with VPLS We still need to control data plane layer 2 events (i.e., limit the traffic) Since enterprises want dual N-PE devices, and VPLS blocks BPDUs, we require method to block within a local DC 38

39 End-to-End L2 View Layer 3 Core Intranet Broadcast, Multicast, Unknown Unicast DC Core DC Core Agg RSTP Metro Core VPLS / EoMPLS Domain Metro Core RSTP X X X X Agg Access Without layer 2 link between Metro Switches there is a loop. Each side has a U shape with Metro and Agg switches, broadcast storms. Access Server Farm L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm 39

40 Spanning Tree Option: MSToNPE Root Bridge in West DC for all VLANs that Go Between Data Centres Layer 3 Core Intranet Root Bridge in East DC for all VLANs that Go Between Data Centres DC Core DC Core Agg RSTP X Single L2 MST Bridge MST VPLS / EoMPLS Domain Single L2 MST Bridge MST X RSTP Agg Metro Core Metro Core Access Access L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm Server Farm 40

41 Spanning-Tree MST (802.1s) represents Metro Cores as single bridge Blue Layer 2 link is access port channel with a VLAN that represents the MST0 instance to make the MST group MST bridge priority set to 0 (Metro Core will be root of Inter-DC VLANs) Spanning tree root-guard enabled on Metro Cores toward aggregation switches (protects in case the blue MST link fails) Only inter-dc VLANs allowed on trunks to/from aggregation switches Single L2 MST Bridge Set spanning-tree VLAN cost to set the priorities on the agg switches links to metro core will allow us to put some VLANs on upper Metro Core, some on lower by default 41

42 Spanning Tree Option: MSToNPE interface Port-channel4 description Port Channel to WestMetroCore1 spanning-tree vlan 3702,3706,3710,3714,3718 cost 8 Layer 3 Core Intranet DC Core DC Core Agg X RSTP X X X X X Single L2 MST Bridge MST Metro Core VPLS / EoMPLS Domain Metro Core Single L2 MST Bridge MST X RSTP X X X X X Agg Access interface Port-channel4 description Port Channel to WestMetroCore2 spanning-tree vlan 3700,3704,3712,3716 cost 8 Access Server Farm Server Farm 42

43 STP Option: Multi-Chassis Link Aggregation Group (MC-LAG) Root Bridge in West DC for all VLANs that Go Between Data Centres Layer 3 Core Intranet Root Bridge in East DC for all VLANs that Go Between Data Centres DC Core ICCP VPLS / EoMPLS Domain ICCP DC Core vpc vpc RSTP Metro Core Metro Core RSTP Access Access L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm Server Farm 43

44 Advanced VPLS (A-VPLS) Leverages VSS MEC for DCI L2/L3/L4 Flow Based Balancing Simplified Edge Redundancy Optimal Bandwidth Utilisation PFC on SUP720 treats as a normal Ethernet port Flexibility to trunk VLANs over either an MPLS or IP transport easily A new interface type: interface virtual-ethernet x Takes switchport commands just like a normal physical Ethernet port 44

45 Advanced VPLS (A-VPLS) Integration with existing VPLS solutions MPLS Fast Re-Route (FRR) for very fast failover MPLS Traffic Engineering (TE) Requires SIP-400 / ES40+ ( SXJ1) 10GE IOS Version SXI4 Sub-1 second fail-over 4,000 VLANs 32 Sites Unified Control-Plane (Single npe Per Location) 45

46 Advanced VPLS (A-VPLS) VSS is recommended but not required. If VSS is used then the modules need to be compatible with VSS. Ie. 67xx modules. Scalability is 32k VCs; the number of VCs equals the number of neighbors * number of VLANs The solution supports MPLS L3 VPNs at the same time; MPLS L3 VPNs can exist side by side on the same PEs to provide a complete solution. 46

47 Leveraging VSS for Dual-Homing Agg Agg npe npe Agg VSL IP/MPLS Cloud VSL Agg Agg Agg VSS system VSS system Leveraging VSS at the DCI edge provides npe redundancy Use of VSS is transparent to the VPLS cloud Equivalent to having the sites single attached (single virtual PE) 47

48 The Label Setup Example Agg One Tunnel Label Per ECMP Exit Agg npe npe Agg OSPF Agg VSL VSL Agg Agg Loop0: Loop0:

49 The Label Setup Example Agg VLAN10 PW Lbl1 PW Lbl1 VLAN10 VLAN20 PW Lbl2 npe Targeted LDP Single tldp per neighbor PW Lbl2 npe VLAN20 Agg Agg Agg VSL VSL Agg Agg Loop0: Loop0:

50 Multi-Pathing with A-VPLS Agg A-VPLS Pseudowire Single Virtual Ethernet Interface across Multiple Interfaces LSP/GRE Tunnel npe npe Agg Agg VSL IP/MPLS Cloud VSL Agg Agg Agg VSS system VSS system Up to 8 equal cost paths between any two sites A label is assigned to each equal cost path based on routing reachability of neighbor Simplified CLI: Virtual Ethernet interface Loadbalancing at L2/L3/L4 50

51 Agg A-VPLS Solution Agg npe npe Agg VSL L2/L3/L4 LB between all sites VSL Agg Agg Agg VSS system Split horizon between all neighbors for loop avoidance, multipoint support. VSL Want to add a 3 rd site? VSS system 51

52 Configuration A-VPLS pseudowire-class cl1 encap mpls PE1 ( )! enable ML PW (ECMP LB) load-balance flow! enable FAT PW flow-label enable interface virtual-ethernet 1 IP/MPLS transport vpls mesh neighbor pw-class cl1 neighbor pw-class cl1 switchport switchport mode trunk switchport trunk allowed vlan 10, 20 Egress physical interface: interface TenGigabitEthernet1/1/3/0 ip address mpls ip PE2 ( ) PE3 ( ) 52

53 End to End VPLS and EoMPLS Design A-VPLS Layer 3 Core Intranet WCore1 WCore2 ECore1 ECore2 DC Core Po1 WMC1 EMC1 DC Core WAgg1 Agg WAgg2 Po1 VPLS / EoMPLS Domain EAgg1 EAgg2 Agg WMC2 EMC2 Access Access Server Farm L2 Links (GE or 10GE) L3 Links (GE or 10GE) Loss of Link/Node Server Farm 53

54 A-VPLS Routed/IRB PW MPLS Cloud WCore1 WCore2 ECore1 ECore2 DC Core VSS Agg VSL WAgg2 Po1 SIP-400 or ES40+ Core Interfaces Ten3/0/0 Ten4/0/0 Ten4/0/0 A-VPLS Virtual Ethernet Configuration Ten4/0/0 EAgg1 VSL DC Core VSS Agg Access A-VPLS with Integrated Routing and Bridging L2 Boundary does not extend beyond Aggregation layer Access Server Farm L2 Links (GE or 10GE) L3 Links (GE or 10GE) Loss of Link/Node Server Farm 54

55 Storm Control Traffic storms when packets flood the LAN Traffic storm control feature prevents LAN ports from being disrupted by broadcast or multicast flooding Rate limiting for unknown unicast (UU) must be handled at Data Centre aggregation; unknown unicast flood rate-limiting (UUFRL): mls rate-limit layer2 unknown rate-in-pps [burst-size] Storm Control is configured as a percentage of the link that storm traffic is allowed to use. storm-control broadcast level 1.00 (% of b/w may vary need to baseline) storm-control multicast level 1.00 (% of b/w may vary need to baseline) 55

56 3 or More Data Centre Locations EoMPLS will allow multiple point to point links between any 2 sites Can build a full mesh of links to interconnect layer 3 devices VPLS scales by adding peer xconnects under the VFI in the IOS configuration Split horizon with MST local to Data Centre will make for simple growth Limits dependant on amounts of L2 traffic especially multicast, as these are replicated on each PW 56

57 3 Site Drawing With EoMPLS PWs for L3 Server Farm Server Farm L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm 57

58 3 Site Drawing With VPLS PWs for L2 Server Farm Server Farm L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm 58

59 Summary of Tagging Section EoMPLS well suited for Router-Router links VPLS well suited for Switch-Switch links Straightforward to scale to multiple Data Centre locations MST and MC-LAG both work well One tradeoff is QinQ support against number of VLANs to pass Another is the root of the spanning tree for inter-dc VLANs A-VPLS Backwards Compatible Load Balancing Enhancements Simplified Configuration Single virtual npe 59

60 Session Agenda Data Centre Interconnection Common Scenarios and Terms Dark Fibre / DWDM Solutions Label Based Solutions IP Based Solutions Encryption Recommended Designs for Optimising Traffic Flows Q & A 60

61 IP Based Solutions

62 EoMPLS/VPLSoGRE Reason for ogre IP Only Core Need a solution to stand up VC with a LDP label GRE provides routing separation from metro core devices providing connectivity Customer Edge (CE) flapping routes won t propagate inside IP network Point to point links between locations Wide range of hardware support including 6500, 7600, ASR IPSec securing of tunnel straightforward Data plane rate limiting in L2 still needs protection * Please note the 7600 does not support VPLSoGRE 62

63 What Is EoMPLS and VPLS Over GRE? EoMPLS connectivity over IP-only network. EoMPLS VCs are established over MPLSoGRE Tunnels Requires SIP-400 on the 6500 with SUP720 EoMPLS instance PE MPLSoGRE Tunnels EoMPLS instance PE VPLS connectivity over IP-only network. VPLS VCs are established over MPLSoGRE Tunnels. Requires SIP-400 on the 6500 with SUP720 VPLS instance PE MPLSoGRE Tunnels VPLS instance PE IP GRE Tunnels that provide MPLS connectivity over IP-only network. MPLS LDP session is established through the GRE tunnel PE VPLS instance 63

64 Layer 2 Extension EoMPLSoGRE Catalyst 6500 Per VLAN Per VC/GRE VLAN alternate path Backup EoMPLS Pseudo-wire into Core npe Si L3 L2 Si npe MCEC with Nexus 7000 vpc Aggregation L2 Etherchannel as VSS is viewed as one device Aggregation npe Si Si L3 L2 VSL Si Si npe MEC Access Access Si VSL Si L2 Links (GE or 10GE) L3 Links (GE or 10GE) 64

65 Layer 2 Extension EoMPLSoGRE - Catalyst 6500 interface Loopback0 description tunnel source ip address interface Loopback1 description LDP Router ID ip address interface Loopback0 description tunnel source ip address interface Loopback1 description LDP Router ID ip address Interface Tunnel 10 ip address tunnel-source tunnel-destination mpls ip ip route Tunnel 10 Interface gig 1/0 Switchport Switchportmode access Switchportaccess vlan10 mtu 9216 interface GigabitEthernet3/0/1 description SIP-400 Interface mtu 9216 ip address bfd interval 100 min_rx 100 multiplier 3 Interface Tunnel 10 ip address tunnel-source tunnel-destination mpls ip ip route Tunnel 10 Interface gig 1/0 Switchport Switchportmode access Switchportaccess vlan10 mtu 9216 interface GigabitEthernet3/0/1 description SIP-400 Interface mtu 9216 ip address bfd interval 100 min_rx 100 multiplier 3! Int vlan 10 Int vlan 10 Xconnect encapsulation mpls mtu 9216 Xconnect encapsulation mpls mtu

66 Layer 2 Extension VPLSoGRE Catalyst 6500 Per VLAN Per VFI/GRE VLAN alternate path L2 Links (GE or 10GE) L3 Links (GE or 10GE) npe Si Si L3 L2 VSL Si Si npe L2 Etherchannel as VSS is viewed as one Device Aggregation npe Si L3 L2 Si npe npe L2 Etherchannel as VSS is viewed as one Device Aggregation Si Si L3 L2 VSL Si Si npe L2 Etherchannel as VSS is viewed as one Device Aggregation MEC Access Access Si VSL Si Access 66

67 Layer 2 Extension VPLSoGRE Catalyst 6500 interface Loopback0 description tunnel source ip address interface Loopback1 description LDP Router ID ip address Interface Tunnel 10 ip address tunnel-source tunnel-destination mpls ip interface Loopback0 description tunnel source ip address interface Loopback1 description LDP Router ID ip address Interface Tunnel 10 ip address tunnel-source tunnel-destination mpls ip ip route Tunnel 10 Interface gig 1/0 Switchport Switchport mode access Switchport access vlan10 mtu 9216 interface GigabitEthernet3/0/1 description SIP-400 Interface mtu 9216 ip address bfd interval 100 min_rx 100 multiplier 3 ip route Tunnel 10 Interface gig 1/0 Switchport Switchport mode access Switchport access vlan10 mtu 9216 interface GigabitEthernet3/0/1 description SIP-400 Interface mtu 9216 ip address bfd interval 100 min_rx 100 multiplier 3 l2 vfi vfi-vlan10 vpn id 10 neighbor encapsulation mpls interface Vlan 10 xconnectvfi vfi-vlan10 mtu 9216 l2 vfi vfi-vlan10 vpn id 10 neighbor encapsulation mpls interface Vlan 10 xconnectvfi vfi-vlan10 mtu

68 Overlay Transport Virtualisation (OTV) Ethernet LAN Extension over any Network Ethernet in IP MAC routing Multi-Data Centre scalability Simplified Configuration & Operation Seamless overlay - no network re-design Single touch site configuration High Resiliency Failure domain isolation Seamless Multi-homing Maximises available bandwidth Automated multi-pathing Optimal multicast replication 68

69 OTV Interface Types Edge Device Internal Interfaces External Interface OTV Overlay Interface Join Interface Overlay Interface Internal Interfaces L2 L3 Join Interface Core 69

70 OTV Control Plane Neighbor Discovery and Adjacency Formation Before any MAC address can be advertised the OTV Edge Devices must: Discover each other Build a neighbor relationship with each other The neighbor relationship can be built over a transport infrastructure, that can be: multicast-enabled unicast-only Technology Benefit: OTV can leverage any networking capability provided by the transport infrastructure (multicast, fast-reroute, ECMP) 70

71 OTV Control Plane Neighbor Discovery (over Multicast Transport) OTV Control Plane OTV Multicast-enable Transport OTV OTV Control Plane West IP A IP B East The mechanism Edge Devices (EDs) join an multicast group in the transport, as they were hosts (no PIM on EDs) OTV hellos and updates are encapsulated in the multicast group The end result Adjacencies are maintained over the multicast group A single update reaches all neighbors 71

72 OTV Control Plane OTV Control Plane Neighbor Discovery (Unicast-Only Transport) Ideal for connecting two or three sites With a higher number of sites a multicast transport is the best choice OTV Unicast-only Transport OTV OTV Control Plane West IP A Adjacency Server Mode IP B East The mechanism Edge Devices (EDs) register with an Adjacency Server ED EDs receive a full list of Neighbors (onl) from the Adjacency Server OTV hellos and updates are encapsulated in IP and unicast to each neighbor The end result Neighbor Discovery is automated by the Adjacency Server All signalling must be replicated for each neighbor Data traffic must also be replicated at the head-end 72

73 OTV Data Plane Encapsulation OTV encapsulation adds 42 Bytes to the packet IP MTU size Outer IP Header and OTV Shim Header in addition to original L2 Header stripped off of the.1q header The outer OTV shim header contains information about the overlay (VLAN, overlay number) The 802.1Q header is removed from the original frame and the VLAN 802.1Q header removed field copied over into the OTV shim header 802.1Q DMAC SMAC 802.1Q Ether Type DMAC SMAC Ether Type IP Header OTV Shim 6B 6B 2B 20B 8B L2 Header 14B* Payload CRC 4B 20B + 8B + 14B* = 42Byte of total overhead Original L2 Frame * The 4Bytes of.1q header have already been removed 73

74 OTV Data Plane: Unicast OTV Inter-Site Traffic MAC 2 1 Layer 2 Lookup OTV MAC TABLE VLAN MAC IF 100 MAC 1 Eth MAC 2 Eth MAC 3 IP B 100 MAC 4 IP B MAC Table contains MAC addresses reachable through IP addresses 5 Layer 2 Lookup OTV MAC TABLE VLAN MAC IF 100 MAC 1 IP A 100 MAC 2 IP A 100 MAC 3 Eth MAC 4 Eth 4 MAC 4 MAC 1 MAC 1 MAC 3 West Eth 1 Eth 2 External External Eth 4 IP A IP B Eth 3 MAC 1 MAC 3 IP A IP B MAC 1 MAC MAC 1 3 MAC 3 IP A IP B Core L2 L3 L3 L2 2 4 MAC 1 MAC 3 3 Encap Decap MAC 3 East 6 74

75 STP BPDU Handling When STP is configured at a site, an Edge Device will send and receive BPDUs on the internal interfaces. An OTV Edge Device will not originate or forward BPDUs on the overlay network. An OTV Edge Device can become (but it is not required to) a root of one or more spanning trees within the site. An OTV Edge Device will take the typical action when receiving Topology Change Notification (TCNs) The BPDUs messages. stop here OTV Core 75

76 Data-plane Loop Prevention AED and Broadcast/Multicast Handling Broadcast/M-cast packets reach all Edge Devices within a site. The AED for the VLAN is the only Edge Device that forwards b-cast/ m-cast packets onto the overlay network The b-cast/m-cast packet is replicated to all the Edge Devices on the overlay. Only the AED at each remote site will forward the packet from the overlay onto the site. Once sent into the site, the b-cast/m-cast packet is replicated per regular switching Broadcast, Multicast, Unknown Unicast OTV OTV OTV Core OTV AED AED 76

77 Multi-Homing Per VLAN Authoritative Edge Device OTV provides loop-free multi-homing by electing a designated forwarding device per site for each VLAN This forwarder is known as the Authoritative Edge Device (AED) The Edge Devices at the site peer with each other on the internal interfaces to elect the AED A hash based on the VLAN-ID and the number of edge devices on the site is used to elect the AED As sites merge and/or partition, internal peering is updated and AED re-election happens Internal peering for AED election 77 OTV AED OTV

78 Dual Adjacency Multi-homing With Mechanism to proactively advertise AED capability I m not AED Capable Provides additional resiliency Avoid single point of failure of site-vlan going down Proactively inform neighbors about local failures AED for VLAN A,B,C AED Adjacency election begins. Up. Exclude AED election Overlay non-aed process Adjacency capable begins EDs OTV OTV AED for VLAN AED for VLAN X, Y, Z X, Y, A, B, C Join interface down Internal Vlans down AED down or initialising Site Adjacency Vlans are split across EDs as long as At least one adjacency is up & EDs are AED capable 78

79 Multi-Homing AED and Broadcast/Multicast Handling Broadcast/M-cast packets reach all Edge Devices within a site. The AED for the VLAN is the only Edge Device that forwards b-cast/ m-cast packets onto the overlay network The b-cast/m-cast packet is replicated to all the Edge Devices on the overlay. Only the AED at each remote site will forward the packet from the overlay onto the site. Once sent into the site, the b-cast/m-cast packet is replicated per regular switching Broadcast stops here OTV Broadcast stops here OTV Bcast pkt OTV Core OTV AED AED 79

80 Multi-Homing AED and Unicast Forwarding One AED is elected for each VLAN on each site Different AEDs can be elected for each VLAN to balance traffic load Only the AED forwards unicast traffic to and from the overlay Only the AED advertises MAC addresses for any given site/vlan Unicast routes will point to the AED on the corresponding remote site/vlan MAC TABLE VLAN MAC IF 100 MAC 1 IP A 201 MAC 2 IP B OTV AED IP A OTV AED OTV Core IP B OTV AED AED 80

81 OTV Use Case Two Sites Connected With Dark-Fibre 81

82 Configuration OTV over a Multicast Transport Minimal configuration required to get OTV up and running feature otv otv site-vlan 600 interface Overlay1 description WEST-DC otv join-interface e1/1 otv control-group otv data-group /24 otv extend-vlan OTV West IP A feature otv otv site-vlan 601 interface Overlay1 description SOUTH-DC otv join-interface Po16 otv control-group otv data-group /24 IP C OTV otv extend-vlan feature otv otv site-vlan 602 interface Overlay1 description EAST-DC otv join-interface e1/1.10 otv control-group otv data-group /24 otv extend-vlan OTV IP B East South 82

83 Configuration OTV over an unicast-only transport Establishing a DCI has never been this simple feature otv otv site-vlan 600 interface Overlay1 description WEST-DC otv join-interface e1/1 otv adjacency-server local otv extend-vlan West OTV IP A feature otv otv site-vlan 601 interface Overlay1 description SOUTH-DC otv join-interface Po16 otv adjacency-server otv extend-vlan IP C OTV feature otv otv site-vlan 602 interface Overlay1 description EAST-DC otv join-interface e1/1.10 otv adjacency-server otv extend-vlan IP B OTV East South 83

84 Localised HSRP ip access-list ALL_IPs 10 permit ip any any mac access-list ALL_MACs 10 permit any any ip access-list HSRP_IP 10 permit udp any /32 eq permit udp any /32 eq 1985 mac access-list HSRP_VMAC 10 permit c07.ac ff any 20 permit c9f.f fff any vlan access-map HSRP_Localization 10 match mac address HSRP_VMAC match ip address HSRP_IP action drop vlan access-map HSRP_Localization 20 match mac address ALL_MACs match ip address ALL_IPs action forward vlan filter HSRP_Localization vlan-list ,1100,1200,1300 mac-list OTV_HSRP_VMAC_deny seq 10 deny c07.ac00 ffff.ffff.ff00 mac-list OTV_HSRP_VMAC_deny seq 11 deny c9f.f000 ffff.ffff.f000 mac-list OTV_HSRP_VMAC_deny seq 20 permit route-map OTV_HSRP_filter permit 10 match mac-list OTV_HSRP_VMAC_deny otv-isis default vpn Overlay0 redistribute filter route-map OTV_HSRP_filter otv site-vlan

85 OTV Summary STP Isolation: BPDUs are not forwarded over the overlay Multi-homing support Optimal Multicast Replication Control-plane MAC based learning and forwarding Simplified Configuration IP Based / Transport Agnostic 85

86 Calculating Core MTU Requirements Edge MTU is the MTU configured in the CE-facing PE interface Examples (all in Bytes): MPLSoGRE PE to PE MPLSoGRE PE to P Edge MPLS Label 4 ( label) 8 ( labels) GRE Header Total PWoGRE PE to PE* (vlan) PWoGRE PE to PE* (port) OTV 1500 n/a * 6 -srcmacaddr 6 -dstmacaddr 4 -VLAN information 2 -Type field 4 -Control word 4 -VC label 4 -Tunnel label

87 Session Agenda Data Centre Interconnection Common Scenarios and Terms Dark Fibre / DWDM Solutions Label Based Solutions IP Based Solutions Encryption Recommended Designs for Optimising Traffic Flows Q & A 87

88 Encryption

89 Point-to-Point Encryption Solution 802.1AE Link DC-1 DC-2 N N e1/25 e1/ Nexus 7000 Nexus 7000 Nexus 7000 Trustsec can be used to secure data across remote Data Centre if Layer 2 and BPDU transparency is ensured (e.g. dark fibre or DWDM transport). 89

90 Encryption Solution 802.1AE Link N N e1/25 DC-1 DC-2 gi 0/0/0 gi 0/0/3 Self-Managed MPLS Core gi 0/0/3 gi 0/0/0 e1/ Nexus 7000 Nexus 7000 EoMPLS PW * Remote port shutdown (ASR Only) 90

91 Nexus 7000 vpc Encryption Solution DC1-Nexus DC2-Nexus vpc Self-Managed MPLS Core vpc DC1-Nexus DC2-Nexus * Remote port shutdown (ASR) 91

92 Conclusions TrustSec SAP (Security Association Protocol) control plane is preserved through the EoMPLS pseudowire AE connectivity can be achieved between the two nexus 7000 through the ASR(s)/6500(s) devices with confidentiality and integrity. Such solution can be deployed to preserve data confidentiality and integrity through Nexus 7000 when interconnecting remote Data Centres over an EoMPLS network. 92

93 VSPA/ASR1000/ASA Solution Overview Data Centre Interconnect with MPLSoGREoIPSec DC 1 DC 2 MPLSoGREoIPSec Leverage ECMP to load balance flows over multiple GRE/IPSec Duplicate tunnels per VSPA allow redundant 10GE links to be provisioned Inherent crypto engine HA: Traffic will rebalance in the event of a VSPA outage Solution Objective Provide a high speed Layer 2 connection between two or more DCs.. Two or more redundant links are used between the DCs. VSPA Performance Three VSPAs can drive a 10 GE link with IMIX traffic. Single chassis can encrypt three 10 GE links at IMIX rates. ASR-1000 Performance ASR1000-ESP5-1.8Gbps IPSec ASR1000-ESP10-4Gbps IPSec ASR1000-ESP20-8Gbps IPSec ASR1006-2/ESP20-16Gbps IPSec ASR1006-2/ESP Gbps IPSec ASA-5585-X Performance 93

94 Session Agenda Data Centre Interconnection Common Scenarios and Terms Dark Fibre / DWDM Solutions Label Based Solutions IP Based Solutions Encryption Recommended Designs for Optimising Traffic Flows Q & A 94

95 Flow Optimisation and Symmetry Site Selection and Inbound Flows First Hop Outbound

96 Optimising Traffic Patterns and HA Design Many tradeoffs in understanding flows in multi-dc design Slides that follow are a specific recommendation that meets the following requirements: Minimise inter-dc traffic to maintenance/failure scenario s Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.) Desire to keep flows symmetric in/out of a location for DC services (FW, LB, IPS, WAAS, etc.) Site failure will allow failover, with IP mobility to resolve caching issues Single points of failure in gear won t cause site failover Indicate a location preference for a service to the Layer 3 network If broadcast storm in DC, limit impacts to other DCs If DCI Layer 2 adjacency fails Ability to connect to services in both DC locations (active/active per application) DNS to round-robin clients to DC Allow backup server farms with same service VIP (for backup connections on site fail) Localised HSRP (egress) Inbound traffic draw via LISP (ingress) This is a solution in production at some customers 96

97 Sample Cluster Service Normally in Left DC Default Gateway Shared Between Sites /25 & /25 advertised into L3 -EEM or RHI can be used to get very granular Layer3 Core Layer3 Core /24 advertised into L3 Backup should main site go down Active/Standby Pairs: FW IPS NLB SSL WAN Accel Data Centre 1 Data Centre 2 Active/Standby Pairs: FW IPS NLB SSL WAN Accel VLAN A HSRP Group 1 Priority 140 and HSRP Group 1 Priority 120 and 110 VLAN A Cluster Node A Cluster Node B Cluster VLAN C (L2 Only) Cluster VLAN D (L2 Only) -Cluster VIP = Preempt -Default GW = L2 Links (GE or 10GE) L3 Links (GE or 10GE) -Cluster VIP = Default GW =

98 Sample Cluster Broadcast Storm in Left DC Broadcast, Multicast, Unknown Unicast /25 & /25 advertised into L3 -EEM or RHI can be used to get very granular Layer3 Core /24 advertised into L3 Backup should main site go down Data Centre 1 Data Centre 2 VLAN A HSRP Group 1 Priority 140 and HSRP Group 1 Priority 120 and 110 VLAN A Cluster Node A Cluster Node B Cluster VLAN C (L2 Only) Cluster VLAN D (L2 Only) -Cluster VIP = Preempt -Default GW = Cluster VIP = Default GW =

99 Sample Cluster L2 Interconnect Failure Broadcast, Multicast, Unknown Unicast /25 & /25 advertised into L3 -EEM or RHI can be used to get very granular Layer3 Core Layer3 Core /24 advertised into L3 Backup should main site go down Data Centre 1 Data Centre 2 VLAN A HSRP Group 1 Priority 140 and HSRP Group 1 Priority 120 and 110 VLAN A Cluster Node A Cluster Node B Cluster VLAN C (L2 Only) Cluster VLAN D (L2 Only) -Cluster VIP = Preempt -Default GW = Cluster VIP = Default GW =

100 Active/Active per Application (VIP at Either) /25 & /25 advertised into L3 -EEM or RHI can be used to get very granular /24 advertised into L3 Backup should main site go down Layer3 Layer3 Core Core /24 advertised into L3 Backup should main site go down /25 & /25 advertised into L3 -EEM or RHI can be used to get very granular VLAN A HSRP Group 1 Priority 140 and HSRP Group 2 Priority 120 and 110 Data Centre 1 Data Centre 2 DNS: www-hr.acme.com -> www-news.acme.com -> HSRP Group 2 Priority 140 and 130 VLAN A HSRP Group 1 Priority 120 and 110 Cluster Node A Cluster VLAN C (L2 Only) Cluster Node B -Cluster VIP = Preempt -Default GW = Cluster VLAN D (L2 Only) -Cluster VIP = Default GW = Cluster VIP = Preempt -Default GW =

101 Active/Active per Application (VIP at Both) /25 & /25 advertised into L3 -EEM or RHI can be used to get very granular /24 advertised into L3 Backup should main site go down Layer3 Layer3 Core Core /24 advertised into L3 Backup should main site go down /25 & /25 advertised into L3 -EEM or RHI can be used to get very granular VLAN A HSRP Group 1 Priority 140 and HSRP Group 2 Priority 120 and 110 Data Centre 1 Data Centre 2 DNS: www-hr.acme.com -> HSRP Group 2 Priority 140 and 130 VLAN A HSRP Group 1 Priority 120 and 110 Cluster Node A Cluster Node B -Cluster VIP = Default GW = Cluster VIP = Preempt -Default GW = Cluster VLAN C (L2 Only) Cluster VLAN D (L2 Only) -Cluster VIP = Default GW = Cluster VIP = Preempt -Default GW =

102 Primary Service in Left DC DR/SRM Movement of VM announced via VCenter /24 is advertised into L3 Layer3 Core MAC moved Change the Agg SNAT Public Network VLAN A SNAT Agg Access Access VM= Default GW =

103 Stateful Firewall Services Layer3 Core Data Centre 1 Data Centre 2 VLAN B - Outside VLAN B - Outside VLAN C - Inside VLAN C - Inside ESX Node A VLAN A x VLAN A x ESX Node B 103

104 Localised First Hop Layer3 Core Data Centre 1 Data Centre 2 VLAN A x 1) Filter HSRP Message 2) Filter vmac VLAN A x HSRP Group 30 Priority 140 and 130 ESX Node A HSRP Group 30 Priority 140 and 130 ESX Node B -VM IP Address = VM Default GW =

105 Locator/ID Separation Protocol (LISP) and L2 Extension Workload Mobility Client in LISP Site Client in non-lisp Site C1 C2 A A OTV D MR Layer3 Layer3 Core Core MS PxTR Server-to-Server L2 traffic E B B VLAN A VLAN A FHRP: ESX Server A ESX Server B FHRP: Virtual-Machine-A -IP Address = Mask: Default GW = LISP: L3 Client-to-Server Optimise L3 Routing providing granular location information Optimised mobility within or across subnets Scale the network so host routes are in mapping database -Virtual-Machine-A -IP Address = L2 Server-to-Server -Mask: Optimise LAN Extensions -Default GW = Enable dispersion of app clusters App discovery based on MAC level broadcast and link-local multicast General application communication may require L2 connectivity 105 L3 Router LISP Router or infrastructure device

106 Routing Based Ingress Optimisation LISP Data Centre 1 ISP A 1 IP_DA = IP_DA = IP_DA = B Prefix (EID) VM IP Address Route Locator (RLOC) Moved A, B to C, D C, D C, D 2 Ingress Tunnel Router (ITR) 6 Encap IP_DA = A ETR B A, B C D 7 ETR Decap 5 IP_DA = Decap ISP B IP_DA = IP_DA = C Data Centre 2 Agg LAN Extension Agg Access Access VM= Default GW = VM= Default GW =

107 Session Agenda Data Centre Interconnection Common Scenarios and Terms Dark Fibre / DWDM Solutions Label Based Solutions IP Based Solutions Encryption Recommended Designs for Optimising Traffic Flows Q & A 107

108 Summary Discussed different deployment options and transport options Tightly coupled Data Centre with FabricPath Spanning-tree isolation Traffic Optimisation Egress and Ingress Symmetry Encryption Solutions 108

109 Q & A

110 Recommendations Recommended Reading NX-OS and Cisco Nexus Switching (ISBN: ), by David Jansen, Ron Fuller, Kevin Corbin. Cisco Press Interconnecting Data Centres Using VPLS (ISBN-10: ; ISBN-13: ), by Nash Darukhanawalla, Patrice Bellagamba. Cisco Press MPLS Fundamentals (ISBN: ), by Luc De Ghein, Cisco Press Layer 2 VPN Architectures (ISBN: ), by Wei Luo, Carlos Pignataro, Anthony Chan, Dmitry Bokotey. Cisco Press Cisco LAN Switching Configuration Handbook (2nd Edition) (ISBN ; ISBN-13: ), by Steve McQuerry, David Jansen, David Hucaby, Cisco Press

111 Recommendations Check the Recommended Reading flyer for suggested books Additional Information on LISP: Available Onsite at the Cisco Company Store 111

112 Complete Your Online Session Evaluation Complete your session evaluation: Directly from your mobile device by visiting and login by entering your username and password Visit one of the Cisco Live internet stations located throughout the venue Open a browser on your own computer to access the Cisco Live onsite portal Don t forget to activate your Cisco Live Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit 112

113 Visit the Cisco Store for Related Titles

114 114