Data Center InterConnect (DCI) Technologies. Session ID 20PT

Transcription

1 Data Center InterConnect (DCI) Technologies Session ID 20PT

2 Session Objectives The main goals of this session are: Highlighting the main business requirements driving Data Center Interconnect (DCI) deployments Understand the functional components of the holistic Cisco DCI solution Get a full knowledge of Cisco LAN extension technologies and associated deployment considerations 2

3 Session Non-objectives This session does not include: In depth discussion of Path Optimization technologies (ACE/GSS, LISP, etc.) Storage extension considerations associated to DCI deployments Workload mobility application specific deployment considerations 3

4 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions MPLS Based Solutions Summary and Conclusions Q&A 4

5 Data Center Interconnect Business Drivers Data Centers are extending beyond traditional boundaries Virtualization applications are driving DCI across PODs (aggregation blocks) and Data Centers DCI Drivers Business Solution Constraints IT Technology Business Continuity Disaster Recovery HA Framework Stateless Network Service Sync Process Sync GSLB Geo-clusters HA Cluster Operation Cost Containment Data Center Maintenance / Migration / Consolidation VLAN Extension Distributed Virtual Data Center Business Resource Optimization Disaster Avoidance Workload Mobility Stateful Bandwidth Latency VM Mobility Cloud Services Inter-Cloud Networking XaaS Flexibility VM Mobility Automation 5

6 Data Center Interconnect DCI Model Allowing VLAN Extension between Sites Path Optimization Any type of links STP Domain isolation + Storm-control Dual-Homing GW STP domain STP domain STP domain ALT GW ALT GW ALT Si Si Si Si DC1 DC2 DC3 Storage extension 6

7 VLAN Extension with DCI Requirements VLAN Types Type T0 Limited to a single access layer device Type T1 Extended inside an aggregation block (POD) Type T2 Extended between PODs part of the same DC site Type T3 Extended between PODs part of twin DC sites (usually connected via dedicated dark fiber links) Type T4 Extended between PODs part of remote DC sites T0 T1 T3 T2 7

8 VLAN Extension Technology Selection Criteria Tech. Nature Ethernet Selection Criteria VSS & vpc or Fabric-Path Applies easily for dual site interconnection Over dark fiber or protected D-WDM Easy crypto using end-to-end 802.1AE IP MPLS L2oL3 for link protection (Fast detection & convergence / Dampening) CE style Enterprise / DC focus Easy integration over Core, works also over MPLS transport Innovative MAC routing EoMPLS & A-VPLS & H-VPLS L2oL3 for link protection (Fast detection & convergence / Dampening) PE style Large scale & Multi-tenants Works over GRE Most deployed today 8

9 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions MPLS Based Solutions Summary and Conclusions Q&A 9

10 Dual Sites Interconnection Leveraging Etherchannel between Sites On DCI Etherchannel: STP Isolation (BPDU Filtering) Broadcast Storm Control FHRP Isolation Primary Root Primary Root interface port-channel10 desc DCI point to point connection switchport switchport mode trunk vpc 10 switchport trunk allowed vlan spanning-tree port type edge trunk spanning-tree bpdufilter enable storm-control broadcast level 1 storm-control multicast level x Si Si L 2 L 3 WAN L 3 L 2 Link utilization with Multi- Chassis EtherChannel DCI port-channel - 2 or 4 links Requires protected DWDM or Direct fibers Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N When vpc: Use dedicated L3 Links for Inter-DC routing! Alternative solutions: Shared L2/L3 link possible with static routing Use dedicated VDC for routing 10

11 VSS / vpc Data-Center Interconnect Scalability Validation Testing Public design guide ( nterconnect_design_guide.pdf VSL or vpc Peer Link extended over 100km fiber Layer 2: 200 Layer 2 VLANs VLAN SVIs 10,000 client-to-server flows at 20 Gbps Layer BGP routes also redistributed to OSPF OSPF routes Results: L2/L3 Unicast & Multicast traffic protected on any failure in less than 3 secs for any combination More recent validation testing with NX7K V4.2.6 & Cat6K SXI 1200 VLAN SVI (static routing) 6500 customer flows at 20Gbps Unicast Convergence around 4 to 5s worst cases Storm control contained on failing site Validated design 11

12 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions Overlay Transport Virtualization (): Technology Overview Overlay Transport Virtualization (): Deployment Considerations MPLS Based Solutions Summary and Conclusions Q&A 12

13 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions Overlay Transport Virtualization (): Technology Overview Overlay Transport Virtualization (): Deployment Considerations MPLS Based Solutions Summary and Conclusions Q&A 13

14 Terminology Edge Device (ED): connects the site to the (WAN/MAN) core; responsible for performing all the functions Internal Interfaces: interfaces of the ED that face the site. Join Interface: interface of the ED that faces the core. Overlay Interface: logical multi-access multicast-capable interface. It encapsulates Layer 2 frames in IP unicast or multicast headers. Overlay Interface Internal Interfaces L2 L3 Join Interface Core 14

15 Join and Internal Interfaces Deployment Guidelines Both currently supported only on M1 line cards The Internal Interfaces should carry the VLANs to be extended plus the site-vlan Only one join interface (physical or logical) can currently be specified per Overlay Multiple physical interfaces can be deployed as L3 uplinks For a higher resiliency the use of a port-channel is encouraged, but it s not mandated There are NOT requirements neither in terms of 1GE vs 10GE nor in terms of Dedicated vs Shared mode. Supported Join Interface types: Join Interface Type* Layer 3 Routed Physical Interface and Sub-interface Layer 3 Port-Channel Interface and Sub-interface Supported * Loopback interfaces and SVI support planned for future releases 15

16 Data Plane Inter-Sites Packet Flow Layer 2 Lookup MAC TABLE VLAN MAC IF 100 MAC 1 Eth Encap Transport Infrastructure 100 MAC 2 Eth MAC 2 IP A MAC 1 MAC 3 IP A IP B 100 MAC 3 IP B MAC 1 MAC 3 IP A IP B 100 MAC 3 Eth MAC 4 IP B IP A 4 Decap 5 IP B MAC TABLE VLAN MAC IF 100 MAC 1 IP A 100 MAC 4 Eth 4 6 Layer 2 Lookup MAC 1 MAC 3 1 MAC 1 MAC 3 West East Server 1 Site Site Server

17 Data Plane Encapsulation encapsulation adds 42 Bytes to the packet IP MTU size Outer IP Header and Shim Header in addition to original L2 Header Control and Data Plane packets have DF bit set The outer shim header contains information about the overlay (VLAN, overlay number) The 802.1Q header is removed from the original frame and the VLAN field copied over into the shim header 802.1Q header removed 802.1Q DMAC SMAC 802.1Q Ether Type DMAC SMAC Ether Type IP Header Shim L2 Header CRC 6B 6B 2B 20B 8B VLAN 14B* Payload 4B Original L2 Frame 42 Byte total overhead *4B of original 802.1Q header removed 17

18 Overlay Transport Virtualization The Control Plane Neighbor discovery and adjacency over Multicast Unicast (Adjacency Server Mode available with 5.2 release) proactively advertises/withdraws MAC reachability (controlplane learning) IS-IS is the Control Protocol - No specific configuration required MAC Addresses Advertisements West IP A IP B East IP C South 18

19 Control Plane Neighbor Discovery (Unicast-Only Transport) Release 5.2 and above Adjacency server is just an edge device (i.e. Nexus 7000) not a separate server or other device Advertises IP of each Edge Device (ED) to all other EDs ( Neighbor List onl) All subsequent communications happen directly between EDs without going through the Adjacency Server Site 2 Site 3 onl Site 1, IP A Site 2, IP B Site 3, IP C Site 4, IP D Site 5, IP E Site 1 IP A Adjacency Server Mode IP B IP D IP C Unicast-Only Transport IP E Site 4 Site 5 19

20 Failure Domain Isolation Spanning-Tree Site Independence Site transparency: no changes to the STP topology Total isolation of the STP domain Default behavior: no configuration is required BPDUs sent and received ONLY on Internal Interfaces The BPDUs stop here The BPDUs L3 stop here L2 20

21 Failure Domain Isolation Preventing Unknown Unicast Storms No requirements to forward unknown unicast frames Assumption: end-host are not silent or unidirectional Default behavior: no configuration is required MAC TABLE VLAN MAC IF 100 MAC 1 Eth1 100 MAC 2 L3 IP B - - L2 - No MAC 3 in the MAC Table MAC 1 MAC 3 21

22 and Multi-homing VLAN Splitting between Edge Devices Authoritative Edge Device (AED) role negotiated between the two VDCs (on a per VLAN basis) Internal IS-IS peering on the site VLAN VLANs are split between the Edge Devices belonging to the same site Achieved via a very deterministic algorithm (not configurable) AED AED Future functionality will allow to tune the behavior System-ID determines which AED will handle ODD / EVEN VLANs Internal peering on Site VLAN for AED election Highest System-ID = AED for ODD VLANs -ED# show otv site Site Adjacency Information (Site-VLAN: 1999) (* - this device) Overlay100 Site-Local Adjacencies (Count: 2) Hostname System-ID Ordinal dc2a-agg-7k2-otv 001b.54c2.e142 0 * dc2a-agg-7k1-otv f

23 Scalability Current and Future Supported Values The following values have been tested and verified: 3 Sites 128 extended VLANs* 12K MAC Addresses across all the extended VLANs 1500 Sites Multicast Data Groups The values below are supported in 5.2 release: 6 Sites 256 extended VLANs 16K MAC Addresses across all the extended VLANs 3000 Sites Multicast Data Groups 23

24 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions Overlay Transport Virtualization (): Technology Overview Overlay Transport Virtualization (): Deployment Considerations MPLS Based Solutions Summary and Conclusions Q&A 24

25 Placement of the Edge Device Option 1 in the DC Core Easy deployment for Brownfield L2-L3 boundary remains at aggregation DC Core devices performs L3 and functionalities May use a pair of dedicated Nexus 7000 VLANs extended from aggregation layer L2 Octopus design Recommended to use separate physical links for L2 & L3 traffic STP and L2 broadcast domains not isolated between PODs vpc vpc VSS SVIs SVIs SVIs SVIs vpc vpc vpc VSS SVIs SVIs SVIs SVIs vpc 25

26 and SVI Coexistence Introducing the VDC VDC VDC Currently, on the Nexus 7000 traffic belonging to a given VLAN can either be routed (associated with an SVI) or extended using L3 L2 L3 L2 This would theoretically require a dual-system solution The VDC feature allows to deploy a dual-vdc solution on the same physical device Different VDC deployment options Single Homed VDC Model Dual Homed VDC Model 26

27 Placement of the Edge Device Option 2 in the DC Aggregation L2-L3 boundary at aggregation DC Core performs only L3 role STP and L2 broadcast Domains isolated between PODs Intra-DC and Inter-DCs LAN extension provided by Requires the deployment of dedicated VDCs Ideal for single aggregation block topologies Recommended for Green Field deployments Nexus 7000 required in aggregation SVIs SVIs SVIs SVIs vpc vpc 27

28 Placement of the Edge Device Option 3 over Dark Fiber Deployments Data Centers directly connected at the Aggregation Currently mandates the deployment of dedicated VDCs Control Plane messages must always be received on the Join Interface Requires IGP/PIM peering between aggregation devices (via peer-link) Advantages over VSS-vPC solution: Provision of Layer 2 and Layer 3 connectivity leveraging the same dark fiber connections Native STP isolation: no need to explicitly configure BPDU filtering ARP Optimization with the ARP Cache Simplified provisioning of FHRP isolation Easy Addition of Sites Layer 2 Link Layer 3 Link Virtual Link Site 1 VDC VDC VDC SVIs SVIs SVIs SVIs vpc vpc Site 2 VDC 28

29 Single Homed VDC Simple Model N7K-A N7K-B VDC VDC Link-1 Link-3 Link-1 Routing VDC N7K-A Po1 Po1 Routing VDC Logical View N7K-B Link-2 VDC VDC Link-2 Link-4 May use a single physical link for Join and Internal interfaces Minimizes the number of ports required to interconnect the VDCs Single link or physical node (or VDC) failures lead to AED re-election 50% of the extended VLANs affected Failure of the routed link to the core is not related Recovery is based on IP convergence Physical View Layer 3 Layer 2 29

30 Dual Homed VDC Improving the Design Resiliency VDC VDC Links 1-2 N7K-A Routing VDC N7K-A Po1 Logical View N7K-B VDC Links 1-2 Po1 Links 3-4 Link 5 Link 7 Link 6 Link 8 Routing VDC Physical View N7K-B Links 3-4 VDC Logical Port-channels used for the Join and the Internal interfaces Increases the number of physical interfaces required to interconnect the VDCs Traffic recovery after single link failure event based on port-channel re-hashing No need for AED re-election Physical node (or VDC) failure still requires AED re-election In the current implementation may cause few seconds of outage (for 50% of the extended VLANs) Layer 3 Layer 2 30

31 Site Load Balancing Per VLAN Load Balancing Simple Appliance Model Aggregation AED role negotiated between the two VDCs (on a per VLAN basis) VDC VDC Internal IS-IS peering on the site VLAN Recommended to carry the site VLAN on vpc links and vpc peer-link AED For a given VLAN all traffic must be carried to the AED Device Part of the flows carried across the vpc peer-link Optimized traffic flows is achieved in the most resilient model leveraging Port-Channels as Internal Interfaces VDC AED Most Resilient Model Aggregation VDC The AED encapsulates the original L2 frame into an IP packet and send it back to the aggregation layer device The aggregation layer device routes the IP packet toward the DC Core/WAN edge L3 routed traffic bypasses the VDC 31

32 Device Load Balancing Single Overlay Behavior Unicast traffic directed to (received from) the same remote site (AED) will always use the same physical link encapsulated packets characterized by the same <Src-IP, Dst-IP> information In multipoint deployments unicast traffic may leverage multiple equal cost paths <Dts-IP> value changes with the remote Edge Device (AED1, AED2) Next generation HW (CY12) would allow to achieve flow based loadbalancing traffic encapsulated into UDP with variable source port # dc1-otv1# show routing hash <SRC> <DST> Load-share parameters used for software forwarding: load-share mode: address source-destination port sourcedestination Universal-id seed: 0x854d029d Hash for VRF "default" Hashing to path * For route: /24, ubest/mbest: 2/0 *via , Eth2/15, [110/84], 3d14h, ospf-10, intra *via , Eth2/16, [110/84], 3d13h, ospf-10, intra To/From Remote AED1 (IP B) IP A AED ECMP Links dc1-otv1# show port-channel load-balance forwarding-path interface port-channel 1 src-ip <SRC> dst-ip <DST> module <MOD> Missing params will be substituted by 0's. Module 1: Load-balance Algorithm: src-dst ip-l4port RBH: 0x6 Outgoing port id: Ethernet1/17 To/From Remote AED2 (IP C) 32

33 Convergence Analysis Failure Scenarios IP/MPLS Core 1 Typical Failure Extreme Failures 1. Core failure (link/node) 1x. Core Partition 1x 1 2. Join interface failure 2x. Core Partition 3. Internal interface failure 4. Device component failure 3x. Site Partition 4x. Total device failure East-B Aggregation 3 2x 2 4 4x East-A Access VPC Internal Interfaces Join Interfaces Other Uplink Interfaces 33

34 Convergence Analysis Convergence Values Robust HA is the guiding principle Common Failures: 1. Core failures Multipath routing (or TE FRR) sub-sec 2. Join interface failures Link Aggregates across line-cards sub-sec 3. Internal Interfaces failures Multipath topology (vpc) & LAGs sub-sec 4. ED component failures HW/SW resiliency sub-sec Extreme failures (unlikely): 1x. Core partition convergence 3x. Site partition (AED re-election) 4x. Device down 5 seconds (low scale) to 10+ seconds (large scale) sub-sec in future releases Internal Interfaces Join Interfaces Other Uplink Interfaces 34 VDC East-B Access IP/MPLS Core Aggregation 1x 3 VPC x 1x 2 VDC 4 East-A

35 Control Plane CLI Verification Establishment of control plane adjacencies between Edge Devices: dc1-otv1# show otv adjacency Overlay Adjacency database Overlay-Interface Overlay100 : Hostname System-ID Dest Addr Up Time Adj-State dc1-otv2 001b.54c2.efc :08:53 UP dc2-otv1 001b.54c2.e1c :43:27 UP dc2-otv2 001b.54c2.e :49:11 UP MAC reachability information: dc1-otv1# show otv route Unicast MAC Routing Table For Overlay100 VLAN MAC-Address Metric Uptime Owner Next-hop(s) c07.ac01 1 3d15h site Ethernet1/ d70e 1 3d15h site Ethernet1/ f3.88ff 42 2d22h overlay dc2-otv f d22h overlay dc2-otv1 Local Site MAC Remote Site MAC 35

36 in the DC Aggregation Storm Control Unknown Unicast are stopped and ARP are reduced by default with! Recommendation is still to install a strict rate-limiter for generic broadcast to a few ten s Mbits VDC Default VDC Default VDC VDC Multicast should also be constrained e1/1 Storm Control applied in inbound direction only N7K-Agg1 L3 Link L2 Link Storm-Control N7K-Agg2 interface ethernet1/1 storm-control broadcast level 1 storm-control multicast level x CoPP enabled by default on the default VDC Default policies apply to all the defined VDCs 36

37 Summary Extensions over any transport (IP, MPLS) Fault Domain North Data Center Fault Domain Failure boundary preservation Site independence Optimal BW utilization (no head-end replication) Automated Built-in Multihoming End-to-End loop prevention Scalability Sites, VLANs, MACs Only few CLI commands Operations simplicity Fault Domain South Data Center Fault Domain 37

38 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions MPLS based solutions EoMPLS A-VPLS H-VPLS Summary and Conclusions Q&A 38

39 EoMPLS port mode xconnect interface PE1 LDP/RSVP PE2 interface interface g1/1 description EoMPLS port mode connection no switchport no ip address xconnect vcid 1 encapsulation mpls 39

40 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions MPLS based solutions EoMPLS A-VPLS H-VPLS Summary and Conclusions Q&A 40

41 Multi-Point Topologies What is VPLS? VLAN SVI VFI PW MPLS Core PW VFI SVI VLAN PW One extended bridge-domain built using: VFI = Virtual Forwarding Instance ( VSI = Virtual Switch Instance) PW = Pseudo-Wire SVI = Switch Virtual Interface xconnect VFI SVI VLAN Mac address table population is pure Learning-Bridge 41

42 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions MPLS based solutions EoMPLS A-VPLS H-VPLS Summary and Conclusions Q&A 42

43 Service-Providers or SP-Like Enterprises High-End DCI Service Enterprise DC Interconnection models: Provider DC to Provider DC Enterprise DC to Provider DC Enterprise DCI MPLS Cloud DCI POP / WAN Edge service initiation Multi-tenants High Scale > 300 VLAN >1000 VLAN >2000 VLAN >4000 VLAN Multi-tenant SP DC (Hosting Service) 43

44 MPLS DCI Conclusion A Mature Solution EoMPLS is an easy point to point solution VPLS DCI is having two flavors: 1. A-VPLS based on node clustering Simplicity Very fast convergence Only available today with Catalyst H-VPLS based on mlacp attachment High-end devices (7600 / ASR9K, ) Multi-tenant features High scale High SLA features Standard based 44

45 Agenda DCI Business Drivers and Solutions Overview LAN Extension Deployment Scenarios Ethernet Based Solutions IP Based Solutions MPLS Based Solutions Summary and Conclusions Q&A 45

46 Data Center Interconnect - DCI Model Connecting Virtualized Data Centers L2 Domain Elasticity - Fabric Path - LAN Extensions IP Localization - Optimal Routing - Route Portability Service Localization - Any service anywhere Fabric Consolidation - Unified Fabric & I/O - Device Virtualization - Segmentation VN-link notifications Storage Elasticity - SAN Extensions VM-awareness - VN-link intelligence 46

47 Data Center Interconnect - DCI Model Connecting Virtualized Data Centers L2 Domain Elasticity - Fabric Path - LAN Extensions STP Isolation is the key element Multipoint Loop avoidance + Storm-Control Unknown Unicast & Broadcast control Link sturdiness Scale & Convergence Fabric Consolidation - Unified Fabric & I/O - Device Virtualization - Segmentation IP Localization - Considerations Optimal Routing - Route Portability Network and Security services deployment Server-Client Flows Server-Server Flows Path Optimization Options Egress Addressed by FHRP Filtering Ingress: 1. DNS redirection with ACE/GSS 2. Route Injection 3. LISP Service Localization - Any service anywhere VN-link notifications Storage Elasticity - SAN Extensions Sync or Async replication modes are driven by the applications, hence the distance/latency is a key component to select the choice Localization of Active Storage is key Distance can be improved using IO accelerator or caching Virtual LUN is allowing Active/Active VM-awareness - VN-link intelligence 47

48 Data Center Interconnect LAN Extension Technology Selection Criteria Ethernet Over dark fiber or protected D-WDM VSS & vpc Dual site interconnection FabricPath (TRILL) MPLS MPLS Transport EoMPLS A-VPLS Enterprise style MPLS H-VPLS Large scale & Multi-tenants IP IP Transport Enterprise style MAC Routing 48

49 Data Center Interconnect Where to Go for More Information 49

50 Complete Your Session Evaluation Please give us your feedback!! Complete the evaluation form you were given when you entered the room This is session 3.3 (Data Center & Virtualization) Don t forget to complete the overall event evaluation form included in your registration kit YOUR FEEDBACK IS VERY IMPORTANT FOR US!!! THANKS 50

51 51