Cisco ACI Multi-Pod Design and Deployment

Transcription

1

2 Cisco ACI Multi-Pod Design and Deployment John Weston Technical Marketing Engineer

3 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Session Objectives At the end of the session, the participants should be able to: Articulate the different deployment options to interconnect Cisco ACI networks (Multi-Pod vs. Multi-Site) Understand the functionalities and specific design considerations associated to the ACI Multi-Pod Fabric option Initial assumption: The audience already has a good knowledge of ACI main concepts (Tenant, BD, EPG, L2Out, L3Out, etc.) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 Agenda ACI Network and Policy Domain Evolution ACI Multi-Pod Deep Dive Overview, Use Cases and Supported Topologies APIC Cluster Deployment Considerations Inter-Pod Connectivity Deployment Considerations Control and Data Planes Connecting to the External Layer 3 Domain Network Services Integration Migration Scenarios

6 ACI Network and Policy Domain Evolution

7 Cisco ACI Fabric and Policy Domain Evolution ACI Single Pod Fabric ACI Stretched Fabric ACI Multi-Pod Fabric Pod A IPN Pod n DC1 APIC Cluster DC2 MP-BGP - EVPN APIC Cluster ACI Leaf/Spine Single Pod Fabric ACI Geographically Stretch a single Pod ISE 2.1 & ACI 1.2 Federation of Identity and Interconnect TrustSec and ACI using IP based EPG/SGT ACI Multiple Networks (Pods) in a single Availability Zone (Fabric) Fabric A ACI 3.0 Multiple Availability Zones (Fabrics) in a Single Region and Multi-Region Policy Management IP ACI 3.1/3.2 - Remote Leaf and vpod extends an Availability Zone (Fabric) to remote locations Fabric n ISE MP-BGP - EVPN ACI Multi-Site 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Fabric and Policy Domain Evolution Deployment Options Single APIC Cluster/Single Fabric Stretched Fabric Multiple APIC Clusters/Multiple Fabrics Multi-Fabric (with L2 and L3 DCI) DC1 ACI Fabric APIC Cluster DC2 Fabric A Inter-Site App Fabric n L2/L3 DCI Multi-Pod (from 2.0 Release) Multi-Site (3.0 Release, Q3CY17) Pod A IPN Pod n Fabric A IP Fabric n MP-BGP - EVPN MP-BGP - EVPN APIC Cluster ACI Multi-Site 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

9 Terminology Pod A Leaf/Spine network sharing a common control plane (ISIS, BGP, COOP, ) Pod == Network Fault Domain Fabric Scope of an APIC Cluster, it can be one or more Pods Fabric == Availability Zone (AZ) or Tenant Change Domain Multi-Pod Single APIC Cluster with multiple leaf spine networks Multi-Pod == Multiple Networks within a Single Availability Zone (Fabric) Multi-Fabric Multiple APIC Clusters + associated Pods (you can have Multi-Pod with Multi-Fabric)* Multi-Fabric == Multi-Site == a DC infrastructure Region with multiple AZs * Available from ACI release Cisco and/or its affiliates. All rights reserved. Cisco Public 10

10 ACI Multi-Site Overview IP Network VXLAN For More Information on ACI Multi-Site: BRKACI-2125 MP-BGP - EVPN Multi-Site Orchestrator Site 1 Site 2 Availability Zone A REST API GUI Availability Zone B Separate ACI Fabrics with independent APIC clusters ACI Multi-Site Orchestrator pushes cross-fabric configuration to multiple APIC clusters providing scoping of all configuration changes MP-BGP EVPN control plane between sites Data Plane VXLAN encapsulation across sites End-to-end policy definition and enforcement 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

11 Typical Requirement Creation of Two Independent Fabrics/AZs Fabric A (AZ 1) Fabric B (AZ 2) Application workloads deployed across availability zones BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 12

12 Typical Requirement Creation of Two Independent Fabrics/AZs Multi-Pod Fabric A (AZ 1) Classic Active/Active Pod 1.A Pod 2.A ACI Multi-Site Multi-Pod Fabric B (AZ 2) Application workloads deployed across availability zones Pod 1.B Classic Active/Active Pod 2.B BRKACI Cisco and/or its affiliates. All rights reserved. Cisco Public 13

13 ACI Multi-Pod Deep Dive

14 Overview, Use Cases and Supported Topologies

15 ACI Multi-Pod Overview Pod A Inter-Pod Network VXLAN Pod n MP-BGP - EVPN IS-IS, COOP, MP-BGP APIC Cluster IS-IS, COOP, MP-BGP Availability Zone Multiple ACI Pods connected by an IP Inter-Pod L3 network, each Pod consists of leaf and spine nodes Managed by a single APIC Cluster Single Management and Policy Domain Forwarding control plane (IS-IS, COOP) fault isolation Data Plane VXLAN encapsulation between Pods End-to-end policy enforcement 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

16 Single Availability Zone with Maintenance & Configuration Zones Scoping Network Device Changes Maintenance Zones Groups of switches managed as an upgrade group Inter-Pod Network ACI Multi-Pod Fabric APIC Cluster Configuration Zone A Configuration Zone B Configuration Zones can span any required set of switches, simplest approach may be to map a configuration zone to an availability zone, applies to infrastructure configuration and policy only 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17

17 Reducing the Impact of Configuration Errors Introducing Configuration Zones Three different zone deployment modes: Enabled (default): updates are immediately sent to all nodes part of the zone Note: a node not part of any zone is equivalent to a node part of a zone set to enabled. Disabled: updates are postponed until the zone deployment mode is changed (or a node is removed from the zone) Triggered: send postponed updates to the nodes part of the zone The deployment mode can be configured for an entire Pod or for a specified set of leaf switches Select entire Pod Change the deployment mode Select specific Leaf Switches Show the changes not applied yet to a Disabled zone 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

18 Single Availability Zone with Tenant Isolation Isolation for Virtual Network Zone and Application Changes Inter-Pod Network ACI Multi-Pod Fabric APIC Cluster Tenant Prod Configuration/Change Domain Tenant Dev Configuration/Change Domain The ACI Tenant construct provide a domain of application and associated virtual network policy change Domain of operational change for an application (e.g. production vs. test) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

19 ACI Multi-Pod Supported Topologies Intra-DC Two DC sites directly connected POD 1 10G*/40G/100G 10G*/40G/100G POD n 10G*/40G/100G 10G/40G/100G 10G*/40G/100G POD 1 Dark fiber/dwdm POD 2 (up to 50 msec RTT**) APIC Cluster APIC Cluster 3 (or more) DC Sites directly connected 10G/40G/100G 10G*/40G/100G 10G*/40G/100G POD 1 POD 2 Dark fiber/dwdm (up to 50 msec RTT**) 10G*/40G/100G Multiple sites interconnected by a generic L3 network 10G*/40G/100G 10G*/40G/100G L3 (up to 50msec RTT**) 10G*/40G/100G 10G*/40G/100G POD 3 * 10G only with QSA adapters on EX/FX spines ** msec Cisco support and/or its affiliates. added All in rights SW reserved. release Cisco 2.3(1) Public

20 ACI Multi-Pod SW/HW Support and Scalability Values All existing Nexus 9000 HW supported as leaf and spine nodes Maximum number of supported ACI leaf nodes (across all Pods) Up to 80 leaf nodes supported with a 3 nodes APIC cluster 300 leaf nodes (across Pods) with a 5 nodes APIC Cluster 400 leaf nodes (across Pods) with a 7 nodes APIC Cluster (from ACI release 2.2(2e)) Maximum 200 leaf nodes per Pod Up to 6 spines per Pod Maximum number of supported Pods 4 in 2.0(1)/2.0(2) releases 6 in 2.1(1) release 10 in 2.2(2e) release 12 in 3.0(1) release 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21

21 APIC Cluster Deployment Considerations

22 APIC Distributed Multi-Active Data Base The Data Base is replicated across APIC nodes One copy is active for every specific portion of the Data Base Shard 1 Shard 1 Shard 1 APIC APIC APIC Shard 2 Shard 3 Shard 2 Shard 3 Shard 2 Shard 3 Processes are active on all nodes (not active/standby) The Data Base is distributed as active + 2 backup instances (shards) for every attribute 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

23 APIC Cluster Deployment Considerations Single Pod Scenario X X Shards X X APIC APIC APIC in APIC APIC APIC APIC APIC read-only mode APIC will allow read-only access to the DB when only one node remains active (standard DB quorum) Hard failure of two nodes cause all shards to be in read-only mode (of course reboot etc. heals the cluster after APIC nodes are up) Shards in read-only mode Shards in read-write mode Additional APIC will increase the system scale (up to 7* nodes supported) but does not add more redundancy Hard failure of two nodes would cause inconsistent behaviour across shards (some will be in read-only mode, some in read-write mode) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24

24 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario Pod 1 Pod 2 Up to 50 msec APIC APIC APIC 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25

25 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario X Pod 1 Pod 2 Up to 50 msec APIC APIC APIC Read/Write Read Only Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod Cisco and/or its affiliates. All rights reserved. Cisco Public 26

26 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario Pod 1 Pod 2 Up to 50 msec APIC APIC APIC Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod Cisco and/or its affiliates. All rights reserved. Cisco Public 27

27 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario X Pod 1 Pod 2 Up to 50 msec X X APIC APIC APIC Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod2 P od hard failure scenario: recommendation is to activate a standby node to make the cluster fully functional again 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

28 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario X Pod 1 Pod 2 Up to 50 msec X X APIC APIC APIC APIC Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod2 P od hard failure scenario: recommendation is to activate a standby node to make the cluster fully functional again 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

29 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario X Pod 1 Pod 2 Up to 50 msec X X APIC APIC APIC APIC Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod2 P od hard failure scenario: recommendation is to activate a standby node to make the cluster fully functional again 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

30 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario X Pod 1 Pod 2 Up to 50 msec X X APIC APIC APIC APIC Pod 1 Pod 2 Up to 50 msec APIC APIC APIC APIC APIC Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod2 P od hard failure scenario: recommendation is to activate a standby node to make the cluster fully functional again 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31

31 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario X Pod 1 Pod 2 Up to 50 msec X X APIC APIC APIC APIC X Pod 1 Pod 2 Up to 50 msec APIC APIC APIC APIC APIC Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod2 P od hard failure scenario: recommendation is to activate a standby node to make the cluster fully functional again Pod isolation scenario: same considerations as with single Pod (different behaviour across shards) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

32 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario X Pod 1 Pod 2 Up to 50 msec X X APIC APIC APIC APIC Pod 1 Pod 2 Up to 50 msec APIC APIC APIC APIC APIC Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod2 P od hard failure scenario: recommendation is to activate a standby node to make the cluster fully functional again Pod isolation scenario: same considerations as with single Pod (different behaviour across shards) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33

33 APIC Cluster Deployment Considerations Multi-Pod 2 Pods Scenario X Pod 1 Pod 2 Up to 50 msec X X APIC APIC APIC APIC Pod 1 Pod 2 Up to 50 msec X XXX APIC APIC APIC APIC APIC Pod isolation scenario: changes still possible on APIC nodes in Pod1 but not in Pod2 P od hard failure scenario: recommendation is to activate a standby node to make the cluster fully functional again Pod isolation scenario: same considerations as with single Pod (different behaviour across shards) P Possible to restore the whole fabric state to the latest taken configuration snapshot ( ID Recovery procedure needs BU and TAC involvement) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

34 APIC Cluster Deployment Considerations What about a 4 Nodes APIC Cluster? Q2CY18 Pod 1 Pod 2 Up to 50 msec Pending internal validation, scoped for Q2CY18 APIC APIC APIC APIC Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to 170 leaf nodes supported) Pod isolation scenario: same considerations as with 5 nodes (different behaviour across shards) Pod hard failure scenario No chance of total loss of information for any shard Can bring up a standby node in the second site to regain full majority for all the shards 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35

35 APIC Cluster Deployment Considerations What about a 4 Nodes APIC Cluster? Q2CY18 X Pod 1 Pod 2 Up to 50 msec Pending internal validation, scoped for Q2CY18 APIC APIC APIC APIC Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to 170 leaf nodes supported) Pod isolation scenario: same considerations as with 5 nodes (different behaviour across shards) Pod hard failure scenario No chance of total loss of information for any shard Can bring up a standby node in the second site to regain full majority for all the shards 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 36

36 APIC Cluster Deployment Considerations What about a 4 Nodes APIC Cluster? Q2CY18 Pod 1 Pod 2 Up to 50 msec Pending internal validation, scoped for Q2CY18 APIC APIC APIC APIC Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to 170 leaf nodes supported) Pod isolation scenario: same considerations as with 5 nodes (different behaviour across shards) Pod hard failure scenario No chance of total loss of information for any shard Can bring up a standby node in the second site to regain full majority for all the shards 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

37 APIC Cluster Deployment Considerations What about a 4 Nodes APIC Cluster? XX Pod 1 Pod 2 Up to 50 msec XAPIC APIC APIC APIC Q2CY18 Pending internal validation, scoped for Q2CY18 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to 170 leaf nodes supported) Pod isolation scenario: same considerations as with 5 nodes (different behaviour across shards) Pod hard failure scenario No chance of total loss of information for any shard Can bring up a standby node in the second site to regain full majority for all the shards 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

38 APIC Cluster Deployment Considerations What about a 4 Nodes APIC Cluster? Q2CY18 XX Pod 1 Pod 2 Up to 50 msec XAPIC APIC APIC APIC APIC Pending internal validation, scoped for Q2CY18 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to 170 leaf nodes supported) Pod isolation scenario: same considerations as with 5 nodes (different behaviour across shards) Pod hard failure scenario No chance of total loss of information for any shard Can bring up a standby node in the second site to regain full majority for all the shards 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

39 APIC Cluster Deployment Considerations What about a 4 Nodes APIC Cluster? Q2CY18 XX Pod 1 Pod 2 Up to 50 msec XAPIC APIC APIC APIC APIC Pending internal validation, scoped for Q2CY18 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to 170 leaf nodes supported) Pod isolation scenario: same considerations as with 5 nodes (different behaviour across shards) Pod hard failure scenario No chance of total loss of information for any shard Can bring up a standby node in the second site to regain full majority for all the shards 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

40 APIC Cluster Deployment Considerations Deployment Recommendations Main recommendation: deploy a 3 nodes APIC cluster when less than 80 leaf nodes are deployed across Pods From Q2CY18 can deploy 4 nodes if the scalability requirements are met When 5 (or 7) nodes are really needed for scalability reasons, follow the rule of thumb of never placing more than two APIC nodes in the same Pod (when possible): Pod1 Pod2 Pod3 Pod4 Pod5 Pod6 2 Pods* APIC APIC APIC APIC APIC 3 Pods APIC APIC APIC APIC APIC 4 Pods APIC APIC APIC APIC APIC 5 Pods APIC APIC APIC APIC APIC 6+ Pods APIC APIC APIC APIC APIC * ID Recovery procedure possible for recovering of lost information 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

41 Inter-Pod Connectivity Deployment Considerations

42 ACI Multi-Pod Inter-Pod Network (IPN) Requirements Pod A Pod B MP-BGP - EVPN DB Web/App APIC Cluster Web/App Not managed by APIC, must be separately configured (day-0 configuration) IPN topology can be arbitrary, not mandatory to connect to all spine nodes Main requirements: Multicast BiDir PIM needed to handle Layer 2 BUM* traffic OSPF to peer with the spine nodes and learn VTEP reachability Increase MTU support to handle VXLAN encapsulated traffic DHCP-Relay * Broadcast, Unknown unicast, Multicast 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

43 Inter-Pod Connectivity Frequently Asked Questions What platforms can or should I deploy in the IPN? Nexus 9200s, 9300-EX, but also any other switch or router supporting all the IPN requirements First generation Nexus 9300s/9500s not supported as IPN nodes Can I use a 10G connection between the spines and the IPN network? Yes, with QSA adapters supported on the ACI spine devices Available from 2.1(1h) release on EX/FX based HW No plans to introduce support for first generation spines (including 9336-PQ baby spine ) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

44 Inter-Pod Connectivity Frequently Asked Questions (2) I have two sites connected with dark fiber/dwdm circuits, can I connect the spines back-toback? X POD 1 POD 2 APIC Cluster No, because of multicast requirement for L2 multidestination inter-pod traffic IPN Devices 10G*/40G/100G connections Do I need a dedicated pair of IPN devices in each Pod? POD 1 POD 2 APIC Cluster Can use a single pair of IPN devices, but before 2.1(1h) release mandates the use of 40G/100G inter-pod links 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

45 Control and Data Planes

46 ACI Multi-Pod Auto-Provisioning of Pods Provisioning interfaces on the spines facing the IPN and EVPN control plane configuration 2 3 DHCP requests are relayed by the IPN devices back to the APIC in Pod DHCP response reaches Spine 1 allowing its full provisioning For more information on how to setup an ACI Fabric from scratch: BRKACI-2004, BRKACI-2820 Spine 1 in Pod 2 connects to the IPN and generates DHCP requests 14 7 Discovery and provisioning of all the devices in the local Pod Seed Pod 1 1 APIC Node 1 connected to a Single APIC Cluster 9 APIC Node 2 joins the Cluster 8 APIC Node 2 connected to a Leaf node in Pod 2 Discovery and provisioning of all the devices in the local Pod Leaf node in Seed Pod 1 Pod 2 10 Discover other Pods following the same procedure 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

47 ACI Multi-Pod IPN Control Plane Separate IP address pools for VTEPs assigned by APIC to each Pod Summary routes advertised toward the IPN via OSPF routing IS-IS convergence events local to a Pod not propagated to remote Pods Spine nodes redistribute other Pods summary routes into the local IS-IS process Needed for local VTEPs to communicate with remote VTEPs OSPF OSPF /16 mutual redistribution /16 IP Prefix IPN Network Routing Table IPN IS-IS to OSPF APIC Cluster / /16 Leaf Node Underlay VRF Next-Hop /16 Pod1-S1, Pod1-S2, Pod1-S3, Pod1-S Cisco and/or its affiliates. All rights reserved. Cisco Public 48

48 ACI Multi-Pod Inter-Pod MP-BGP EVPN Control Plane MP-BGP EVPN to sync Endpoint (EP) and Multicast Group information All remote Pod entries associated to a Proxy VTEP next-hop address (not part of local TEP Pool) Same BGP AS across all the Pods ibgp EVPN sessions between spines in separate Pods COOP EP1 Leaf 1 EP2 Leaf 3 EP3 Proxy B EP4 Proxy B Proxy A MP-BGP - EVPN IPN EP1 EP2 EP3 EP4 Proxy B Proxy A Proxy A Leaf 4 Leaf 6 Full mesh MP-iBGP EVPN sessions between local and remote spines (default behavior) EP1 EP2 APIC Cluster EP3 EP4 Optional RR deployment (recommended one RR in each Pod for resiliency) Single BGP ASN 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

49 ACI Multi-Pod Inter-Pod Data Plane VTEP IP VNID Class-ID Tenant Packet Policy and network information carried across Pods = VXLAN Encap/Decap EP1 Leaf 4 EP2 Proxy B 3 Spine encapsulates traffic to remote Proxy B Spine VTEP IPN Spine encapsulates traffic to local leaf 4 EP2 Leaf 4 EP1 Proxy A Proxy A Proxy B EP1 e1/3 * Proxy A EP2 unknown, traffic is EP1 encapsulated to the local Proxy A Spine VTEP (adding S_Class 1 information) VM1 sends traffic destined to remote EP2 2 EP1 EPG APIC Cluster C Configured on APIC EP2 EPG EP2 6 5 If policy allows it, EP2 receives the packet EP2 e1/1 EP1 Pod1 L4 * Proxy B Leaf learns remote EP1 location and enforces policy 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

50 ACI Multi-Pod Inter-Pod Data Plane (2) = VXLAN Encap/Decap IPN Proxy A Proxy B EP1 e1/3 EP2 Pod2 L4 EP1 Pod1 L4 * Proxy A Leaf learns remote VM2 location (no need to enforce policy) 9 EP1 10 VM1 receives the packet EP1 EPG APIC Cluster C Configured on APIC EP2 EPG EP2 7 8 VM2 sends traffic back to remote VM1 * Proxy B Leaf enforces policy in ingress and, if allowed, encapsulates traffic to remote Leaf node L Cisco and/or its affiliates. All rights reserved. Cisco Public 51

51 ACI Multi-Pod Inter-Pod Data Plane (3) = VXLAN Encap/Decap IPN Proxy A Proxy B EP1 e1/3 EP2 Pod2 L4 EP1 Pod1 L4 * Proxy A * Proxy B EP1 APIC Cluster EP2 11 From this point EP1 to EP2 communication is encapsulated Leaf to Leaf (VTEP to VTEP) and policy always applied at the ingress leaf (applies to both L2 and L3 communication) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52

52 ACI Multi-Pod Use of Multicast for Inter-Pod Layer 2 BUM Traffic IGMP Join for (*, GIPo1) Spine 1 elected authoritative for BD1 BUM traffic originated in the local Pod BD1 GIPo1: IPN1 BUM traffic originated from a remote Pod IPN2 Ingress replication for BUM* traffic not supported with Multi-Pod PIM Bidir is the only validated and supported option Scalable: only a single (*,G) entry is created in the IPN for each BD Fast-convergent: no requirement for datadriven multicast state creation A spine is elected authoritative for each Bridge Domain: Generates an IGMP Join on a specific link toward the IPN Always sends/receives BUM traffic on that link BUM: Broadcast, Unknown Unicast, Multicast 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53

53 ACI Multi-Pod Use of Multicast for Inter-Pod BUM Traffic 4 IPN replicates traffic to all the PODs that joined MG1 (optimized delivery to Pods) Spine 2 is designated to send MG1 traffic toward the IPN 3 * 2 5 BUM frame is flooded along one of the trees associated to MG1 BD1 has associated MG1, traffic is flooded intra-pod via one multi-destination tree EP1 1 VM1 in BD1 generates a BUM* frame APIC Cluster EP2 6 VM2 receives the BUM frame BUM: Layer 2 Broadcast, Unknown Unicast, Multicast 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54

54 ACI Multi-Pod PIM Bidir for BUM Supported Topologies Full Mesh between remote IPN devices IPN1 IPN3 Create full-mesh connections between IPN devices IPN2 IPN4 More costly for geo-dispersed Pods, as it requires more links between sites * APIC Cluster Directly connect local IPN devices IPN1 IPN3 EP2 * Alternatively, connect local IPN devices with a portchannel interface (for resiliency) In both cases, it is critical to ensure that the preferred path toward the RP from any IPN devices is not via a spine Recommendation is to increase the OSPF cost of the interfaces between IPN and spines * IPN2 APIC Cluster IPN4 EP2 * interface Ethernet1/49.4 description L3 Link to Pod1-Spine1 mtu 9150 encapsulation dot1q 4 ip address /31 ip ospf cost 100 ip ospf network point-to-point ip router ospf IPN area ip pim sparse-mode ip dhcp relay address ip dhcp relay address e1/49 IPN1 IPN Cisco and/or its affiliates. All rights reserved. Cisco Public 55

55 Connecting to the External Layer 3 Domain

56 Connecting ACI to Layer 3 Domain Traditional L3Out on the BL Nodes PE Client L3Out PE PE WAN PE Border Leafs Connecting to WAN Edge devices at Border Leaf nodes Definition of a L3Out logical construct VRF-lite hand-off for extending L3 multitenancy outside the ACI fabric Each tenant defines one (or more) L3Out with a set of Logical Nodes, Logical Interfaces, peering protocol 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58

57 Connecting Multi-Pod to Layer 3 Domain Traditional L3Out on the BL Nodes A Pod does not need to have a dedicated WAN connection (i.e. can offer transit services to other Pods) Multiple WAN connections can be deployed across Pods Outbound traffic: by default VTEPs always select WAN connection in the local Pod based on preferred metric MP-BGP - EVPN Pod 1 Pod 2 WAN WAN Pod 3 By default traffic flows are hashed across L3Outs of remote Pods 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

58 Connecting Multi-Pod to Layer 3 Domain Traditional L3Out on the BL Nodes (2) Asymmetric traffic paths creates issues when independent active perimeter FWs are deployed across Pods Tuning routing is possible to ensure ingress/egress traffic leverages always the same Pod s L3Out MP-BGP - EVPN Reverting to an Active/Standby mode of operation for the deployed FWs Active FW Pod 1 Active FW Pod 2 Host routes advertisement is a best option to ensure all the deployed FWs are actively utilized Support for host route advertisement on BL nodes planned for a future ACI release WAN Pod 3 WAN Requires an L3Out connection in each Pod 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

59 Connecting ACI to Layer 3 Domain GOLF Design For More Information on GOLF Deployment: LABACI-2101 = VXLAN Encap/Decap VXLAN Data Plane PE PE DCI WAN GOLF Routers (ASR 9000, ASR 1000, Nexus 7000) Client OTV/VPLS Direct or indirect connection from spines to WAN Edge routers PE Better scalability, one protocol session for all VRFs, no longer constraint by border leaf HW table VXLAN handoff with MP-BGP EVPN Simplified tenant L3Out configuration Support for host routes advertisement out of the ACI Fabric VRF configuration automation on GOLF router through OpFlex exchange PE 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

60 GOLF and Multi-Pod Integration Centralized and Distributed Models Centralized WAN Edge Devices Distributed WAN Edge Devices WAN IPN WAN Edge Routers MP-BGP EVPN WAN Edge Routers IPN WAN IPN WAN Edge Routers MP-BGP EVPN Common when Pods represent rooms/halls in the same physical DC MP-BGP EVPN peering required from spines in each Pod and the centralized WAN Edge devices Pods usually represent separate physical DCs Full mesh of EVPN peerings between Pods and WAN Edge routers For more info on GOLF and Multi-Pod integration: Cisco and/or its affiliates. All rights reserved. Cisco Public

61 GOLF and Multi-Pod Integration Inter-DC Scenario Pod A Host routes for endpoint belonging to public BD subnets in Pod A MP-BGP EVPN Control Plane WAN Edge devices inject host routes into the WAN or register them in the LISP database IPN Host routes for endpoint belonging to public BD subnets in Pod B MP-BGP EVPN Control Plane Pod B APIC Cluster 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

62 GOLF and Multi-Pod Integration Inter-DC Scenario (2) Remote Router Table /32 G1,G /32 G3,G4 Granular inbound path optimization( host route advertisement into the WAN or integration with LISP) G1,G2 Routing Table /24 A /32 A WAN G3,G4 Routing Table /24 B /32 B IPN Proxy A Proxy B APIC Cluster 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64

63 Network Services Integration

64 ACI Multi-Pod Network Services Integration Models Active Active/Standby Standby Active/Standby Active and Standby pair deployed across Pods No issues with asymmetric flows but may cause traffic hair-pinning across the IPN Works in all scenarios from release 2.3 Independent Active/Standby pair deployed in each Pod Use PBR (managed or unmanaged mode) Only for perimeter FW use case assuming proper solution is adopted to keep symmetric ingress/egress flows FW cluster deployed across Pods Not currently supported (scoped for 1HCY18) Cluster 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

65 Active/Standby Pair across Pods Option 1: FW in L2 Mode IPN APIC Cluster Active MAC G L3Out-1 WAN L3Out-2 WAN Standby L2 Mode WAN L2 Mode = East-West = North-South 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

66 Active/Standby Pair across Pods Option 2: FW in L3 Mode and PBR IPN PBR Policy Applied Here APIC Cluster L3Out- 1 WAN L3Out- 2 WAN L3 Mode Active WAN L3 Mode Standby = East-West = North-South 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68

67 Active/Standby Pair across Pods Option 2: FW in L3 Mode and PBR IPN PBR Policy Applied Here APIC Cluster L3Out- 1 WAN L3Out- 2 WAN L3 Mode Active WAN L3 Mode Standby = East-West = North-South 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69

68 FW in L3 Mode and L3Outs Single L3Out Defined across Pods IPN APIC Cluster Web VM1 L3Out ASA In/Out Web VM2 L3 Mode Active BDs associated to L3Outs extended via Multi-Pod L3 Mode Standby 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70

69 FW in L3 Mode and L3Outs Single L3Out Defined across Pods (Dynamic Routing) IPN Routing table (VRF1) External IP prefix via Pod1-sLeaf1/2 APIC Cluster Routing table (VRF1) External IP prefix via Pod2-sLeaf1/2 Secondary IP: L3Out ASA In/Out Web VM1 Traffic Bounced Web VM2 Peering across Pods L3 Mode L3 Mode Active Standby Active ASA: Standby ASA: Note: supported from ACI SW releases 2.1(3), 2.2(3), 2.3(1) and 3.0(1) and deploying EX/FX HW for ACI service leaf nodes 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

70 FW in L3 Mode and L3Outs Single L3Out Defined across Pods (Static Routing) IPN Routing table (VRF1) External IP prefix via Pod1-sLeaf1/2 APIC Cluster Routing table (VRF1) External IP prefix via Pod2-sLeaf1/2 Secondary IP: Secondary IP: Secondary IP: L3Out ASA In/Out Web VM1 Static route Traffic Bounced Web VM2 injected into MP- across Pods BGP VPNv4 Fabric L3 Mode control plane L3 Mode Active Standby Active ASA: Standby ASA: Note: supported from ACI SW releases 2.1(3), 2.2(3), 2.3(1) and 3.0(1) and deploying EX/FX HW for ACI service leaf nodes 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72

71 Migration Scenarios

72 1 Migration Scenarios Adding Pods to an Existing ACI Pod1 Add connections to the IPN network Connect and auto-provision the other Pod(s) Pod2 MP-BGP - EVPN Distribute the APIC nodes across Pods 2 Add connections to the IPN network Connect and auto-provision the other Pod(s) MP-BGP EVPN Pod1 Distribute the APIC nodes across Pods Pod Cisco and/or its affiliates. All rights reserved. Cisco Public 74

73 3 Migration Scenarios Converting Stretched Fabric to Multi-Pod Pod1 Pod2 MP-BGP EVPN Re-cabling of the physical interconnection (especially when using DWDM circuits that must be reused) Re-addressing the VTEP address space for the second Pod disruptive procedure as it requires a clean install on the second Pod Not internally QA-validated or recommended 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75

74 Conclusions and Q&A

75 ACI Multi-Pod & Multi-Site A Reason for Both Multi-Pod Fabric A (AZ 1) Classic Active/Active Pod 1.A Pod 2.A ACI Multi-Site Multi-Pod Fabric B (AZ 2) Application workloads deployed across availability zones Pod 1.B Classic Active/Active Pod 2.B 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77

76 Conclusions Cisco ACI offers different multi-fabric options that can be deployed today There is a solid roadmap to evolve those options in the short and mid term Multi-Pod represents the natural evolution of the existing Stretched Fabric design MP-BGP EVPN MP-BGP EVPN Multi-Site will replace the Dual-Fabric approach Cisco will offer migration options to drive the adoption of those new solutions 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78

77 Where to Go for More Information ACI Stretched Fabric White Paper ACI Multi-Pod White Paper ACI Dual Fabric Design Guide ACI and GOLF High Level Integration Paper Cisco and/or its affiliates. All rights reserved. Cisco Public 79

78 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

79 Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public

80 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Tech Circle Meet the Engineer 1:1 meetings Related sessions 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

81 Thank you

82