Extending ACI to Multiple Sites: Dual Site Deployment Deep Dive

Transcription

1

2 Extending ACI to Multiple Sites: Dual Site Deployment Deep Dive Patrice Bellagamba Distinguished Systems Engineer BRKACI-3503

3 Agenda Multi-Data Center Design Options Stretched Fabric Deep Dive ACI Multi-POD Overview ACI Multi-Fabric Deep Dive Conclusion Thanks to Santiago Freitas ), Distinguished Systems Engineer Max Ardica ), Principal Engineer

4 Objectives This presentation and associated white-papers provides a guide to designing and deploying Cisco Application Centric Infrastructure (Cisco ACI TM ) in two data centers with an active-active architecture that delivers Increased uptime Disaster avoidance Easier maintenance Flexible workload placement Extremely low recovery time objective (RTO)

5 ACI Multi-DC Design Options Single APIC Cluster/Single Domain Stretched Fabric Multiple APIC Clusters/Multiple Domains Dual-Fabric Connected (L2 and L3 Extension) ACI Fabric Site 1 Site 2 ACI Fabric 1 ACI Fabric 2 DB Web L2/L3 App Multi-POD (Q3 2016) Multi-Site (CY 17) POD A POD B Site A IP Network Site B MP-BGP - EVPN MP-BGP - EVPN DB Web/App APIC Cluster Web/App DB Web App BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Stretched Fabric Supported Distances and Interconnection Technologies b-aci-stretched-fabric.html

7 Stretched ACI Fabric ACI Fabric DC Site 1 DC Site 2 APIC APIC APIC vcenter Server Transit leaf Transit leaf Single fabric stretched to two sites. Works the same way as Single fabric deployed within a single DC One APIC cluster. One management and configuration point. Anycast GW on all leaf switches. Support for up to 3 sites Work with one or more transit leaf per site. Any leaf can be transit leaf. Number of transit leaf and links is redundancy and bandwidth capacity decision BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Supported Distances and Interconnection Technologies Dark Fiber ACI Fabric DC Site 1 DC Site 2 APIC APIC APIC vcenter Server Transceivers QSFP-40G-LR4 QSFP-40GE-LR4 QSFP-40GLR4L QSFP-40G-ER4 Transit leaf Cable Distance 10 km 10 km 2 km 30 km in 1.0(4h) or earlier 40 km in 1.1 and later Transit leaf For all these transceivers, the wavelength is 1310, the cable type is SMF, and the power consumption is 3.5W. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Supported Distances and Interconnection Technologies Private DWDM Node ID 1 Node ID 2 DC Site 1 Node ID 3 DC Site 2 APIC APIC APIC QSFP-40G-SR4 40G DWDM 40G or 4x 10G 40G 40G QSFP to SFP+ breakout cable DWDM 40G or 4x 10G 40G ACI leaf or spine connects to DWDM using 40G short reach or long reach transceivers. If using 40G-CSR4 or 40G-SR4 a QSFP to SFP+ breakout cable between ACI node and DWDM system can be used and then 4x 10G lambdas can be used between DWDM systems. 4x10G lambdas = 1x 40G link. DWDM failure scenarios If DWDM lambda goes down, DWDM must shutdown the ports facing the ACI Fabric, otherwise 30 seconds outage due to Fabric IS-IS Hold Time. If one attachment circuit goes down, remote port must be shutdown, otherwise 30 seconds outage. DWDM links to be similar, same latency and etc. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 Supported Distances and Interconnection Technologies Ethernet over MPLS (EoMPLS) Node ID 1 Node ID 2 DC Site 1 Node ID 3 DC Site 2 APIC APIC QSFP-40G-SR4 10 ms RTT 800 KM / 500 miles APIC 40G 10G/40G/100G EoMPLS Pseudowire 40G 40G 10G/40G/100G 40G WAN Port mode EoMPLS used to stretch the ACI fabric over long distance. DC Interconnect links could be 10G (minimum) or higher with 40G facing the Leafs / Spines DWDM or Dark Fiber provides connectivity between two sites. Max 10ms RTT between sites. Under normal conditions 10 ms allows us to support two DCs up to 800 KMs apart. Other ports on the Router used for connecting to the WAN via L3Out Validated platform is ASR 9K with XR or later. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Fabric Topology from APIC EoMPLS, DWDN, Dark Fiber pseudowire is transparent for ACI BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 VMM Integration Focus on VMware VDS With DVS 5.x and 6.0, one DVS can be stretched across two sites Live migration supported. Same vcenter manages vsphere servers for both sites With DVS 6.0, one DVS can be used per site and the same EPG can span two (or more) VMM domains Live migration supported with Cross vcenter (Cross DVS) vmotion, APIC release 1.2(1i) and later. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Transit Leaf and WAN Traffic Same ISIS metric for inter-site links and local links When WAN router is connected to transit leaf from both sites, non-border leaf switches will see 2-way ECMP for external subnets Recommended design: WAN Router is NOT connected to transit leaf, so Local WAN router is 2 hops away and WAN router at another site is 4 hops away. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Reference Topology 10 ms RTT DC Site 1 DC Site 2 40G 10G EoMPLS Pseudowire 40G 40G 10G 40G DC1-CE1 DC1-CE2 DC2-CE1 DC2-CE2 DC1-PE DC2-PE 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 RealWeb EPG /24 S-N Traffic Flow N-S is symmetric Odd Tenants = DC 1 primary Even Tenant = DC 2 Primary WAN EPG Layer 2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Logical Topology Deep Dive WAN-CE to ASA, BGP peering through the Fabric WAN EPG with L2 BD with static binding towards ASA and WAN CE ASA/T4/act(config)#route-map set-localpref-200-inprefixes permit 10 ASA/T4/act(config-route-map)# set local-preference 200 ASA/T4/act(config-if)# interface TenGigabitEthernet0/ ASA/T4/act(config-if)# nameif outside ASA/T4/act(config-if)# ip address standby Even numbered tenants use the primary path into/out of the fabric via DC2 and odd tenants use the primary path into/out of the fabric via the left side DC1 ASA/T4/act(config)# router bgp ASA/T4/act(config-router)# address-family ipv4 unicast ASA/T4/act(config-router-af)# neighbor remote-as ASA/T4/act(config-router-af)# neighbor remote-as ASA/T4/act(config-router-af)# neighbor remote-as ASA/T4/act(config-router-af)# neighbor remote-as ASA/T4/act(config-router-af)# redistribute static ASA/T4/act(config-router-af)# neighbor route-map set-localpref-200-inprefixes in ASA/T4/act(config-router-af)# neighbor route-map set-localpref-200-inprefixes in ASA/T4/act(config)# route inside BGP towards CEs Static Towards WEB subnet, NH Fabric 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

17 Logical Topology Deep Dive External L3 out towards ASA External L3 Out Configuration Steps on ACI Create Logical Node Profile with border leafs Leaf-3 and Leaf-5, where ASA is connected Static Default route from each Border Leaf node with Next Hop pointing to ASA Inside Interface IP ASA Failover: ASA failover link and state link through the Fabric via a BD setup in Layer 2. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Logical Topology Deep Dive External L3 out towards ASA External L3 Out Configuration Steps on ACI On the Logical Interface Profile create Secondary IP Address (Floating IP) under each logical transit interface created between Border Leaf and External Physical ASA. This secondary address is a floating IP owned by the border leafs. This helps for seamless convergence during border leaf failures. Remark: DC1-ASA/T4/act(config)# route inside BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Logical Topology Deep Dive MP-BGP Route Reflector Placement Spine 1 == DC 1 Spine 3 == DC 2 The fabric uses MP-BGP to distribute external routes within ACI fabric. Tested SW Release supports a max of two MP-BGP route reflectors. In a stretched fabric implementation, place one route reflector at each site to provide redundancy. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Data Center Failure Site failure on the site with two APICs ACI Fabric DC Site 1 DC Site 2 APIC APIC APIC vcenter Server The remaining APIC controller becomes minority when Site 1 goes down. Configuration changes are not allowed with minority When site 1 goes down, user can access and monitor the ACI fabric via the controller in site 2 but user can t make configuration changes. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Data Center Failure Restoring ability to make configuration changes Node ID 1 Node ID 2 Node ID 3 DC Site 1 DC Site 2 APIC APIC APIC APIC vcenter Server De-commission APIC node 1 and 2 Standby Node ID 2 Commission new APIC node 2. Now has 2 working controllers in site 2. Connect a standby APIC appliance (4 th APIC) in Site 2 after the APIC cluster is formed and operational Standby appliance remains shutdown until needed. When site 1 is down, user de-commission APIC node 1 and 2 and commission new APIC node 2. The "standby" APIC appliance joins APIC cluster Site 2 now has majority of APIC (2 out of 3). User can start to make changes. Stretched Fabric APIC Cluster Recovery Procedures r/aci/apic/sw/kb/b_kb-aci-stretched-fabric.html BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Summary Stretched ACI Fabric For Your Reference - One APIC cluster. One management and configuration point. Anycast GW on all leaf switches. Works the same way as Single fabric deployed within a single DC. - Cisco Validated Design. Extensively tested and passed validation criteria. - 10ms RTT between the sites Under normal conditions 10 ms allows two DCs up to 800 KMs/500 Miles apart. Previous BRKACI-3503 recordings from Cisco Live USA 2015 has a DEEP dive and test results for Stretched Fabric - Interconnection could be dark fiber, DWDM or EoMPLS pseudowire If EoMPLS then DC Interconnect links could be 10G (minimum) or higher with 40G facing the Leaf/Spine QoS required, you need to protect critical control-plane traffic. - APIC Release 1.0(3f) or later. DEMO available Stretched Fabric Link failures vmotion over Stretched Fabric with EoMPLS - BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 ACI Multi-POD Overview Go to ACI MultiPod/MultiSite Deployment Options [BRKACI- 2003] for deep-dive

24 ACI Multi-POD Solution Overview Availability now: Q Inter-POD Network POD A POD n MP-BGP - EVPN IS-IS, COOP, MP-BGP Single APIC Cluster IS-IS, COOP, MP-BGP Multiple ACI PODs connected by an IP Inter-POD L3 network, each POD consists of leaf and spine nodes Managed by a single APIC Cluster Single Management and Policy Domain End-to-end policy enforcement Forwarding control plane (IS-IS, COOP) fault isolation BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 ACI Multi-POD Solution Topologies POD 1 40G/100G Intra-DC 40G/100G POD n For Your Reference 40G/100G ACI MultiPod/MultiSite Deployment Options [BRKACI-2003] Two DC sites connected back2back 10G/40G/100G 40G/100G POD 1 Dark fiber/dwdm (up POD 2 to 10 msec RTT) DB Web/App APIC Cluster Web/App DB Web/App APIC Cluster Web/App 3 DC Sites POD 1 POD 2 10G/40G/100G 40G/100G 40G/100G Dark fiber/dwdm (up to 10 msec RTT) 40G/100G Multiple sites interconnected by a generic L3 network 40G/100G 40G/100G L3 40G/100G 40G/100G POD 3 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 ACI Multi-Pod Solution Scalability Considerations Maximum number of supported ACI leaf nodes (across all Pods) 300 with a 5 nodes APIC Cluster Maximum 200 leaf nodes per Pod Up to 80 leaf nodes supported with a 3 nodes APIC cluster Up to 6 spines per Pod Maximum number of supported Pods 4 in Congo/Congo-MR releases (Q3CY16) 6 in Crystal release (Q4CY16) No current plans to increase those values before end of CY16 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 ACI Multi-POD Solution Inter-POD Network (IPN) Requirements Not managed by APIC, must be initially preconfigured Main requirements: POD A POD B 40G/100G interfaces to connect to the spine nodes MP-BGP - EVPN Multicast, specifically BiDir PIM needed to handle BUM traffic DB Web/App APIC Cluster Web/App DHCP Relay for spine/leaf discovery across PODs OSPF (only option at FCS) for advertising VTEP reachability Increased MTU support to handle VXLAN encapsulated traffic QoS (to prioritize intra APIC cluster communication) BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Inter-Pod Connectivity Frequently Asked Questions What platforms can or should I deploy in the IPN? Nexus 9200s, 9300-EX, but also any other switch or router supporting all the IPN requirements NorthStar and Donner/Donner+ based platforms not initially supported as IPN nodes SW fix is being scoped for 2HCY16 timeframe Can I use a 10G connection between the spines and the IPN network? Yes, once QSA adapters will be supported on the ACI spine devices Planned for Crystal release (Q4CY16) on EX based HW Scoped for Q1CY17 for Alpine based spines BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Inter-Pod Connectivity Frequently Asked Questions I have two sites connected with dark fiber/dwdm circuits, can I connect the spines back-toback? X POD 1 POD 2 APIC Cluster No, because of multicast requirement for L2 multidestination inter-pod traffic IPN Devices 40G/100G connections Do I need a dedicated pair of IPN devices in each Pod? POD 1 POD 2 APIC Cluster Yes, but initially mandates the use of 40G/100G inter- Pod links BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 ACI Multi-POD Solution Overlay Data Plane Policy information carried across Pods Group VTEP IP VNID Tenant Packet Policy Leaf Proxy B 13 Spine encapsulates traffic to remote Proxy B Spine VTEP Spine encapsulates traffic to local leaf Leaf Proxy A Proxy A Proxy B * Proxy A VM2 unknown, traffic is encapsulated to the local Proxy A Spine VTEP (adding S_Class information) VM1 sends traffic destined to remote VM2 Single APIC Cluster Pod1 L4 * Proxy B Leaf learns remote VM1 location and enforces policy If policy allows it, VM2 receives the packet BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 ACI Golf

32 ACI Integration with WAN at Scale Project GOLF Overview WAN GOLF (WAN Edge) Routers Connect an ACI Fabric to the external L3 domain GOLF devices functionally behave as ACI border leafs Control plane and data plane scale MP-BGP EVPN L3Out with GOLF Complementary with ACI Multi-Fabric solutions MP-BGP EVPN control plane between ACI spine and GOLF routers VXLAN data plane between ACI spine and GOLF routers VRF-1 DB Web/App L3Out with VRF-lite VRF-2 OpFlex for exchanging config parameters (VRF names, BGP Route-Targets, etc.) Consistent policy for north-south traffic applied at ACI leaf (both ingress and egress directions) BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 ACI Integration with WAN at Scale Project GOLF : Supported Platforms MP-BGP EVPN IP Network WAN Router initial choices Nexus 7000/7700: F3 line card in DX(1)ES (end of May 2016). M3 support in Q4CY16 (Atherton release) ASR 9000: IOS-XR (June/July 2016) for platforms with minimum RSP3 and Typhoon/ Tomahawk line card support ASR 1000: Polaris release 16.4 (Q4CY16), including also CSR1Kv support High level whitepaper available on CCO: /data-center-virtualization/application-centricinfrastructure/white-paper-c html BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Multi-Pod and GOLF Combined Models Centralized Scenario (Intra-DC) GOLF Devices Connected to IPN GOLF Devices Connected to Pod Spines WAN WAN MP-BGP EVPN MP-BGP EVPN GOLF devices perform a dual function: Pure L3 routing for Inter-Pod VXLAN traffic VXLAN Encap/Decap for WAN to DC traffic flows BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 ACI Multi-Fabric Deep Dive Multiple APIC Clusters / Multiple Domains

36 Dual-Fabric Design Scenarios Two independent ACI fabrics. Two management and configuration domains. Design Goals: Active/Active workload. Extend L2 and subnet across sites. Anycast GW on both fabrics Interconnect Technologies: Dark Fiber or DWDM (back to back vpc) VXLAN/OTV/VPLS/PBB for L2 extension over IP BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 Latency considerations Cisco ACI fabrics are totally independent from each other, and the only control plane between them uses BGP and learning bridges (both supported over long distances) Latency considerations are relative to the other components of the solution VMware supports RTT of up to 100 ms starting with vsphere Release 6.0 ASA clusters are supported over two sites deployed with 20 ms of RTT latency Storage replication. with asynchronous replication, there is no real limit With synchronous replication, there is a strict limitation that depends on the deployed technology. EMC VPLEX and NetApp MetroCluster solutions support a maximum RTT latency limit of 10 ms. Recommendation to deploy all application tiers at the same site with local storage When planning DCI deployments, you also need to consider path optimization BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 ACI Multi-Fabric Layer 2

39 L2 Reachability across Sites Static Binding between EPGs and VLANs ACI Fabric 1 ACI Fabric 2 APIC Static 1:1 mapping VLANs/EPGs DCI Static 1:1 mapping VLANs/EPGs App1 EP1 EPG1 BD1 VLAN = BD = EPG EPG1 BD1 App1 EP2 Internal EPGs are extended to the remote site by leveraging a static 1:1 mapping with VLANs carried on the double-sided vpc. o o o Simpler and recommended over the use of L2Out With vpc, the VLAN EPG mapping must be consistent between the APIC cluster of each Fabric With VXLAN /OTV, DCI can perform VLAN translation ACI Fabric: 1,750 BD per Border Leaf node DCI Scalability OTV 1500 VLAN, PBB-EVPN 4000 VLAN, VXLAN/EVPN 1000 VLAN BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 DCI Dual-Fabric with Active/Active GW One EPG per BD with Static binding ACI Fabric 1 VRF-1 ACI Fabric 2 VRF-1 WEB1 BD /24 VLAN300 VLAN300 BD /24 WEB2 APP1 BD /24 VLAN301 VLAN301 BD /24 APP2 DB1 BD /24 VLAN302 VLAN302 BD /24 DB2 Use static binding to extend EPG between the sites. VLAN ID to EPG mapping matches between fabrics. Fabric treats the remote end points as if they are locally attached, they are learned on the Border Leaf Flood and learn between two fabrics Recommended to turn on Unknown Unicast and ARP flooding in the BD for extended L2 segments BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 Dual-Fabric with Active/Active GW Multiple EPGs Under one BD with Static binding ACI Fabric 1 VRF-1 ACI Fabric 2 VRF-2 WEB1 BD /24 VLAN300 BD /24 WEB2 APP1 VLAN301 APP1 DB1 VLAN302 DB1 Create loop between two fabrics WEB, APP and DB reside in same flooding domain. Can t support multiple EPGs under same BD with L2 extension between ACI fabrics BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 Dual-Fabric with L2 Extension Pervasive Default Gateway (ACI version 11.2 or later) Two IP/MAC addresses should be defined for each stretched IP subnet 1. Unique IP and MAC per site for ARP resolution (Glean) 2. Common virtual IP and virtual MAC for server default GW BD1 Glean IP: Virtual IP MAC: MAC-A vmac: MAC-common ACI Fabric 1 ACI Fabric 2 BD1 Glean IP: Virtual IP MAC: MAC-B vmac: MAC-common DCI Hypervisor Hypervisor One L2 Domain One IP Subnet BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Active/Active Default Gateway Common GW MAC/IP Configuration APIC DC 1 APIC DC 2 Common Virtual MAC, used by VIP Common IP address marked as VIP BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 Routing to an Endpoint Connected to an Internal IP Subnet Inter-subnet routing locally performed by ACI Fabric 1 ACI Fabric 1 ACI Fabric 2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 Inter subnet routing to silent host using gleaning 13 EP1 Leaf 1 12 EP1 e1/3 * Proxy A Proxy A 16 Proxy B EP1 17 BL1/BL2 * Proxy A Leaf1 e1/3 ESX 1 BL1 BL2 Po1 EP1 Leaf 1 DCI BL1 BL2 Po1 EP1 vpc1 ESX * Proxy B EP EPG WEB1 * 14 Proxy A DCI * 15 Proxy B Flood and learn between two fabrics: Unknown are gleaned by the spine using ARP with physical MAC/IP Recommended to turn on Unknown Unicast and ARP flooding in the BD for extended L2 segments Leaf6 EP EPG WEB2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Inter subnet routing to silent host using gleaning 13 EP1 Leaf 1 EP2 BL1/BL2 12 EP1 e1/3 * Proxy A Proxy A 16 Proxy B 17 EP1 BL1/BL2 EP2 Leaf 6 * Proxy A Leaf1 e1/3 ESX 1 BL1 BL2 Po1 EP1 Leaf 1 EP2 Po1 DCI BL1 BL2 Po1 EP1 vpc1 EP2 Leaf 6 ESX * Proxy B EP EPG WEB1 * 14 Proxy A DCI * 15 Proxy B Leaf6 EP EPG WEB2 On ARP Reply to the Fabric 1, Silent Host is discovered BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 Policy Consistency across Sites (L2 Extension) Contract Relationship with Static Binding ACI Fabric 1 Logical EPG Extension ACI Fabric 2 BD-1 VLAN300 BD-1 WEB1 C1 WEB1 EP subject to C1 policy when communicating with APP1 and APP2 APP2 EP subject to C2 policy when communicating back to WEB-1 WEB2 C2 WEB2 EP subject to C2 policy when communicating with APP1 and APP2 APP1 BD-2 VLAN301 APP2 BD-2 Each Fabric treats the remote end points as if they were locally connected (they are added to the local COOP database) Remote endpoint classification performed with static EPG-VLAN binding Contract on two fabrics must be independently created and kept consistent WEB1 EP should be subject to the same policy when accessing an EP in APP1 or APP2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 Inter-sites storm-control Can be applied to ACI or DCI or both APIC APIC Storm-Control ingress traffic on ACI Storm-Control ingress traffic on DCI interface port-channel2 storm-control broadcast level 1.00 storm-control multicast level 1.00 storm-control unicast level 1.00 EPG Ingress static ACI Storm-Control binding DCI EPG Ingress static Storm-Control binding s/switches/datacenter/aci/apic/sw/k b/b_kb_configuring_traffic_storm _Control_in_APIC.html BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 ACI Fabric Loop Protection Applied per site independently Multiple Protection Mechanisms against external loops LLDP detects direct loopback cables between any two switches in the same fabric Mis-Cabling Protocol (MCP) is a new link level loopback packet that detects an external L2 forwarding loop MCP frame sent on all VLAN s on all Ports If any switch detects MCP packet arriving on a port that originated from the same fabric the port is errdisabled External devices can leverage STP/BPDU MAC/IP move detection and learning throttling and err-disable LLDP Loop Detection MCP Loop Detection (supported from 11.1 release) STP Loop Detection BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 Interconnecting multiple ACI Fabrics using OTV vsphere / vcenter 6.0 DVS-DC1 vsphere / vcenter 6.0 DVS-DC2 APIC APIC EPG static binding Server ESX-DC1 DVS-DC1 Server Nexus 7700 OTV EPG VLAN to static OTV binding Overlay Nexus 7700 OTV ESX-DC2 DVS-DC2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 Interconnecting multiple ACI Fabrics using OTV vsphere / vcenter 6.0 DVS-DC1 vsphere / vcenter 6.0 DVS-DC2 OTV Advantages: Server APIC ESX-DC1 DVS-DC1 Server Nexus 7700 OTV APIC Spanning-tree isolation Unknown unicast traffic suppression ARP optimization Layer 2 broadcast policy control EPG static binding OTV also offers a simple command-line interface (CLI), or it can easily be set up using a programming EPG VLAN to static language OTV such as Python. binding Overlay Nexus 7700 OTV ESX-DC2 DVS-DC2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 ACI Dual Fabric with vsphere 6.0 for Cross vcenter vmotion vsphere / vcenter 6.0 vsphere / vcenter 6.0 DVS-DC1 DVS-DC2 WEST_OTVA feature otv APIC otv site-vlan 210 otv site-identifier EAST_OTVA feature otv APIC otv site-vlan 210 otv site-identifier Server interface Overlay1 otv join-interface port-channel100 otv control-group otv data-group /24 otv extend-vlan no shutdown ESX-DC1 interface port-channel100 Nexus 7700 mtu 9216 OTV Server 2 ip address /30 ip igmp version 3 DVS-DC1 EPG static binding EPG VLAN to static OTV binding Overlay interface Overlay1 otv join-interface port-channel100 otv control-group otv data-group /24 otv extend-vlan no shutdown interface port-channel100 Nexus 7700 mtu 9216 OTV ip address /30 ip igmp version 3 ESX-DC2 DVS-DC2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Interconnecting multiple ACI Fabrics using VXLAN/EVPN vsphere / vcenter 6.0 DVS-DC1 vsphere / vcenter 6.0 DVS-DC2 APIC APIC EPG static binding Server ESX-DC1 DVS-DC1 Server Nexus 9300 NX-OS Mode VLAN to EPG static VXLAN binding mapping ESX-DC2 DVS-DC2 VXLAN EPG static overlay with binding BGP-EVPN BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 VXLAN as a DCI for Dual ACI Fabric DCI Nexus 9300 running in NX-OS mode Transport EPG VLAN over L2 VNI to remote site(s) vpc attachment-circuit Anycast VTEP for VXLAN encapsulation L3 peering between Fabrics over vpc attachment-circuit DCI Core Can be Fiber or DWDM with IGP peering. Can be a Metro or WAN network with BGP peering. Nexus 9300 NX-OS Mode VLAN to EPG static VXLAN binding mapping VXLAN EPG static overlay with binding BGP-EVPN BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 VXLAN as a DCI set-up Storm-Control ingress traffic interface port-channel2 storm-control broadcast level 1.00 storm-control multicast level 1.00 storm-control unicast level 1.00 Map VLAN_ID to VNI L2 vlan 1001 vn-segment Overlay Data-Plane: Interface NVE BUM multicast or Ingress Replication interface nve1 no shutdown source-interface loopback1 host-reachability protocol bgp member vni ingress-replication protocol bgp router bgp 100 neighbor remote-as 200 update-source loopback0 ebgp-multihop 10 address-family l2vpn evpn send-community both route-map NEXT-HOP-UNCHANGED out route-map NEXT-HOP-UNCHANGED permit 10 set ip next-hop unchanged evpn vni l2 rd auto route-target import 100:31001 route-target export 100:31001 Overlay Control-Plane: BGP peering with remote Fabric EVPN to populate MAC and MAC IP per VNI Anycast VTEP interface loopback1 ip address /32 ip address /32 secondary Create an Underlay IGP if back to back links ebgp if Metro/WAN Core Multicast or Unicast only BFD enabled BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 ACI Multi-Fabric Layer 3 connectivity between fabrics

57 Cross Fabric L3 Extension BD /24 ACI Fabric 1 ACI Fabric 2 L3Out-DCI EBGP DCI L3Out-DCI EBGP Fabric to Fabric, Per Tenant ebgp peering over Layer 2 DCI (L2VNI, OTV and back to back vpc) Not all EPGs have to be Layer 2 extended Some subnets are local to a single DC/Fabric. L3 Peering between the Fabrics is required for route exchange. ACI support multiple protocols including ibgp, ebgp and OSPF The reference design uses ebgp as it provides demarcation of the administrative domain and the option to manipulate routing policy ACI supports Layer 3 dynamic routing protocol peering over vpc. DCI BD /24 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 Policy Consistency across Sites (L3 Connectivity) One EPG per BD with Unique IP Subnet Distributed application with L3 connectivity between fabrics (EPG=BD=Subnet) Classification: group policy ID derived based on subnet (IP prefix to external EPG mapping) Policy: WEB1 EP should be subject to the C2 policy when accessing APP2 EP Local BD /24 WEB1 C1 L3Out Ext-APP2 WAN L3Out Ext- WEB1 Local BD /24 WEB2 C1 Local BD /24 APP1 Ext-WEB2 External EPG Mapping Table /24 Ext-WEB /24 Ext-APP2 Ext-APP1 External EPG Mapping Table /24 Ext-WEB /24 Ext-APP1 C2 C2 Local BD /24 APP2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 Multitenancy support WAN routers to WAN Edge multitenancy via MPLS L3 VPN or VRF-lite ACI provides L2 BD between WAN Edge and Firewall OSPF per VRF on Router FW Context per Tenant ASA and ACI OSPF within the ASA context L3out per Tenant in ACI Fabric 1 Fabric 2 Per Tenant L3Out ebgp peering over Layer 2 DCI BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 ACI Multi-Fabric Layer 3 connectivity outside the fabrics

61 Perimeter Firewall Design using Active/Active ASA Cluster Video of a demo available at ACI Fabric 1 ACI Fabric 2 Cluster Control Link (CCL) DATA CCL CCL DCI CCL CCL DATA DATA DATA Dual-DC Cisco ASA Cluster ASA Cluster inserted using IP routing, without Service Graph. North-South communication through the local ASA units for IP subnets that are not stretched. Ingress traffic from the WAN routed to the DC where the non-stretched subnet resides based on IP routing. Intra-cluster forwarding to keep symmetry for stretched IP subnets. ACI fabric provides a Layer 2 BD on a dedicated vpc for CCL VLAN which is then extended via DCI to the other site. ASA cluster in routed mode with multiple contexts using individual interfaces. OSPF used as the routing protocol between ASA units and ACI Fabric. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 Routing Peering AS100 ACI AS200 ACI OSPF ebgp Traffic AS300 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 Local Subnet to ACI 1 ACI ACI InGress / Egress OSPF ebgp Traffic BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 Local Subnet to ACI 2 ACI ACI Egress/Ingress OSPF ebgp Traffic BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 Stretched Subnet to both ACI ACI ACI Egress Egress OSPF ebgp Traffic BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 Stretched Subnet to both ACI ACI ACI If FW state Ingress OSPF ebgp Traffic BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 Stretched Subnet to both ACI ACI ACI Ingress OSPF ebgp Traffic BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Perimeter Firewall Design using Active/Active ASA Cluster Video of a demo available at Subnet A: Available only in DC1 ( /24) Subnet B: Available only in DC2 ( /24) Subnet C: Stretched across and available in both data centers ( /24) Subnet D: Represents an external Layer 3 destination in the WAN All 4 ASAsgrouped into a single logical unit. Every member of the cluster has the same configuration, is capable of forwarding every traffic flow, and can be active for all flows. Each firewall peers on its inside interface with the local ACI fabric using OSPF. On the outside interface, each firewall peers with the local WAN edge routers through the ACI fabric (the fabric performs only Layer 2 transport to enable the peering). BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 Perimeter Firewall Design using Active/Active ASA Cluster Video of a demo available at Traffic from subnet A (DC1 only) leaving the fabric and traveling to subnet D in the WAN uses one of the local ASA devices in DC1. This traffic uses the optimal forwarding path. Traffic from subnet B (DC2 only) leaving the fabric and traveling to the WAN uses one of the local ASA devices in the data center 2. This traffic uses the optimal forwarding path. Traffic from stretched subnet C: Traffic originating from DC1 uses one of the local ASA devices in DC1. This traffic uses the optimal forwarding path. Traffic for some devices currently located in DC2 use one of the local ASA devices in DC2. This traffic uses the optimal forwarding path. Ingress Traffic (WAN to DCs) Ingress is optimized to the non-stretched subnets (A and B), because only the local units in DC1 (subnet A) or DC2 (subnet B) announce them to the WAN. Stretched subnet is not optimized in ingress. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 ACI Multi-Fabric Application Integration

71 vcenter Integration Models VMM Integration with Live Migration between sites with vsphere 6 ACI Fabric 1 ACI Fabric 2 vcenter Server 1 vcenter Server 2 VLAN 100 ESX VLAN 100 ESX DVS1 DCI Live migration with vsphere 6 DVS2 VLAN 200 ESX ESX VLAN 200 VMM Domain: DC1 EPG WEB /24 One vcenter/dvs for each fabric with VMM integration VMM Domain: DC2 EPG WEB /24 vsphere 6 Cross vcenter Server vmotion supported from APIC release 1.2(1i) and later Allows live migration between two fabrics with optimized default gateway BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 Cisco UCS Director Policy (Configuration) synchronization between APIC Clusters UCS Director integrates with ACI by communicating with the APIC cluster. User provide IP of one controller, UCS-D discover the other APICs. Single UCS-D instance communicate with two or more APIC clusters UCS Director becomes the platform for the provisioning of Application Network Profiles, EPGs, Bridge Domains, etc. Changes performed directly in the APIC are discovered by UCS Director and reflected on UCS-D object model however configurations are not synchronized to other APICs. To deploy applications, UCS-D creates the ACI objects in the multiple APIC clusters simultaneously. Approval (optional) requested before executing the change on the APIC cluster(s). Support for Multi-Fabric is based on custom workflows. UCS-Director also automates the DCI devices. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 Implementation of UCS-Director for Policy Synchronization with Approval (Optional) DC 1 DC 2 BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 Implementation of UCS-Director for Policy Synchronization with Approval (Optional) BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 Implementation of UCS-Director for Policy Synchronization with Approval 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

76 Implementation of UCS-Director for Policy Synchronization with Approval BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Implementation of UCS-Director for Policy Synchronization with Approval 3 Tier Application Profile Across Both Sites BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 Implementation of UCS-Director for Policy Synchronization with Approval 3 Tier Application Profile Across Both Sites BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 Implementation of UCS-Director for Policy Synchronization with Approval 3 Tier Application Profile Across Both Sites BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 Implementation of UCS-Director for Policy Synchronization with Approval 3 Tier Application Profile Across Both Sites BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 Validated Topology and components with SW version DC1 I Spine-201 I Spine-202 DC2 APIC1-SVR-01 APIC1-SVR-02 APIC1-SVR-03 APIC2-SVR-01 APIC2-SVR-02 APIC2-SVR-03 E 2/1 E 2/2 E 2/1 E 2/2 E 2/1 E 2/2 E 1/34 E 1/35 E 1/36 E 1/34 E 1/35 E 1/36 E 2/1 E 2/2 E 2/1 E 2/2 E 2/1 E 2/2 E 1/34 E 1/35 E 1/36 E 1/34 E 1/35 E 1/36 I Leaf-101 I Leaf-102 I Leaf G A STC2 1/5 10G B I Leaf-101 I Leaf-102 I Leaf G C STC2 1/7 10G D STC2 1/8 Component Software Version E 1/2 E 1/1 E 1/3 E 2/11 E 2/12 E 1/2 E 1/1 E 1/26 E 1/46 E 1/26 E 1/45 E 1/29 E 1/31 E 1/25 E 1/27 E 1/43 E 1/29 E 1/31 E 1/3 E 2/11 E 1/25 E 1/27 E 2/12 E 1/43 E 1/46 E 1/45 E 2/11 E 2/11 E 1/3 E 1/4 STC2 1/6 E 1/2 E 1/1 E 1/3 E 2/11 E 2/12 E 1/2 E 1/1 E 1/26 E 1/46 E 1/26 E 1/45 E 1/29 E 1/31 E 1/25 E 1/27 E 1/43 E 1/29 E 1/31 E 1/3 E 2/11 E 1/25 E 1/27 E 2/12 E 1/43 E 1/46 E 1/45 E 2/11 E 2/11 E 1/3 E 1/4 APIC 1.2(1i) vpc2 Po2 Nexus 9000 ACI Leaf/Spines n (1i) e Po10 Gi0/1 Gi0/2 Gi 0/0/0 ISR Po10 Po11 Gi0/0 Gi0/1 Gi0/2 Gi0/3 Data CCL i05-asa Po10 e Po10 Po11 Gi0/0 Gi0/1 Gi0/2 Gi0/3 e1/3 e1/4 e1/3 e1/4 e1/1 Gi0/0 Gi0/1 Data e1/47-48 e1/1 Gi0/2 Gi0/3 CCL Gi0/1 Gi0/2 Data CCL Po10 Po11 Po1 e1/46 e1/46 E1/50 E1/50 e1/45 e1/45 E1/49 E1/49 i i Gi 0/0/0 i i ISR / /24 i05-asa e05-asa e05-asa Po10 Po11 Gi0/0 Gi0/1 Gi0/2 Gi0/3 e1/3 e1/4 e1/3 e1/4 e1/1 e1/1 e1/47-48 Data CCL Po ASA (1) DCI Nexus 9300 NX-OS 7.0(3)I2(2a) FW DCI FW DCI WAN Router Gi 0/1 Gi 0/2 e Gi 0/1/0 Gi 0/0/0 Gi0/1 10G E / /24 e1/47 e1/49 10G /24 40G /24 e1/48 e1/50 Branch STC2 10/ VM I07-c220m3-01 Client Versions later than the ones above also support the design presented BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 Test Results Summary ACI Dual Fabric Design has been validated Test Case On Failure On Recovery Link from ACI Leaf 1 in DC1 to the local Nexus 9300 VXLAN DCI device 320 ms 122 ms Nexus 9300 VXLAN DCI device node failure 390 ms 1529 ms Peer link failure between the Nexus 9300 DCI devices 735 ms 1593 ms ASA cluster member failure (slave node in DC1) 3255 ms 214 ms ASA cluster member failure (master node) 3947 ms 0 ms ASA cluster member failure (slave node in DC2) 3038 ms 0 ms Customer edge router: link with ACI fabric failure 3094 ms 20 ms Customer edge router WAN link failure 2745 ms 0 ms Cisco ACI border leaf node failure 2494 ms 135 ms Cisco ACI spine node failure 280 ms 0 ms Numbers shown are the worst case scenario, refer to the Whitepaper for DETAILED test results BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 Multi-Fabric Design Summary Stretched Fabric Dual-Fabric with L2Out and L3Out Multi-POD Management Domain Single Multiple Single Distance Up to 10 msec RTT No limit Up to 10 msec RTT HA and Fault Isolation One HA Domain Total independency Control protocol isolation L2 extension End to End built in (Single Fabric) Yes. Flood-N-Learn End to End Overlay Data-Plane End-to-End Policy End to End built in (Single Fabric) Yes with one EPG per BD Single APIC Cluster Scalability Same as one fabric Border Leaf scale 300 nodes across 6 PODs BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 83

84 Summary Solutions for ACI Dual Site Deployment Provides Active/Active Data Centers Business Continuity. Workload mobility and better asset utilization. Single APIC Cluster / Single Management Domain Stretched Fabric with Dark Fiber and Private DWDM. Stretched Fabric with EoMPLS for long distance or SP-managed. Multi-POD (Now in Q3CY16) Multiple APIC Clusters / Independent Fabrics Multi-Fabric with DCI (vpc, VXLAN and OTV) with L2 and L3 Extension Whitepaper published in CCO. BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 85

86 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 86

87 Please join us for the Service Provider Innovation Talk featuring: Yvette Kanouff Senior Vice President and General Manager, SP Business Joe Cozzolino Senior Vice President, Cisco Services Thursday, July 14 th, :30 am - 12:30pm, In the Oceanside A room What to expect from this innovation talk Insights on market trends and forecasts Preview of key technologies and capabilities Innovative demonstrations of the latest and greatest products Better understanding of how Cisco can help you succeed Register to attend the session live now or watch the broadcast on cisco.com

88 Thank you

89