Optimizing Layer 2 DCI with OTV between Multiple VXLAN EVPN Fabrics (Multifabric)

White Paper Optimizing Layer 2 DCI with OTV between Multiple VXLAN EVPN Fabrics (Multifabric) What You Will Learn This document describes how to achieve a VXLAN EVPN multifabric design by integrating Virtual Extensible LAN (VXLAN) Ethernet Virtual Private Network (EVPN) fabrics in conjunction with Overlay Transport Virtualization (OTV) for Layer 2 Data Center Interconnect (DCI). In addition, we provide a sample configuration to illustrate this integration. The VXLAN EVPN fabric can be extended at Layer 2 with various technologies, such as OTV, Virtual Private LAN Services (VPLS), classic Ethernet, or multipod VXLAN EVPN. The sole focus of this document is on the interconnection of VXLAN EVPN fabrics with OTV to facilitate the multifabric approach. For information about connecting VXLAN EVPN fabric in a multipod environment, please refer to the white paper VXLAN Multipod Design for Intra Data Center and Geographically Dispersed Data Center Sites at http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-737201.html. Prerequisites This document assumes that the reader is familiar with the configuration of VXLAN EVPN data center fabric. The VXLAN EVPN fabric can be configured either manually or using Cisco Data Center Network Manager (DCNM) for the underlay and the overlay (Figure 1). This document focuses entirely on providing Layer 2 DCI between fabrics, with the assumption that the individual data center fabrics are already configured and up and running. Figure 1. DCNM VXLAN EVPN Fabrics with OTV Edge Device 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 17

For more information about VXLAN EVPN, including examples showing configuration details, please refer to the following white paper: http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-seriesswitches/guide-c07-734107.html. For more information about OTV, including examples showing configuration details, please refer to the following white paper: http://www.cisco.com/en/us/docs/solutions/enterprise/data_center/dci/whitepaper/dci3_otv_intro_wp.pdf. Introduction Data center deployments require the capability to interconnect networks at Layer 2 in a resilient way, where domain isolation and failure containment can be achieved. In this document, when we refer to the interconnection, there could be a single fabric or multiple fabrics per data center. With this in mind, the interconnection of multiple fabrics with OTV can apply within as well as between data centers. Before the introduction of OTV with the Cisco Nexus 7000 Series Switches in 2010, Layer 2 extensions were achieved by extending the classic Ethernet semantics between data centers. This holds true with the use of VPLS or VXLAN in the area of interconnecting networks, as the flood and learn semantics haven t been revised. With the flood and learn approach, propagated failures are not mitigated and the blast radius extends across all interconnected networks. To eliminate such failure propagation scenarios, new features have been developed. Nevertheless, all these features have to be known and used appropriately. Even with all the measures in place, unknown unicast flooding and potential looping scenarios can still occur. OTV is a purpose-built DCI technology that encompasses transparent transport without the need for adjacent features for multihoming or domain isolation. In addition to the capability of Layer 2 extension, OTV provides: Transport independence Flexible encapsulation (IP-GRE or IP-UDP) Control-plane learning Dynamic neighbor discovery Native multihoming Spanning Tree Protocol isolation Unknown unicast flooding isolation Broadcast traffic separation Multicast transport optimization Address Resolution Protocol (ARP) optimization As of NX-OS 7.3, VXLAN EVPN and OTV operate in individual virtual device contexts (VDCs) on Cisco Nexus 7000 Series Switches. Future developments will allow VXLAN EVPN to integrate seamlessly with purpose built DCI solutions. This will enable the placement of both technologies in a single VDC or switch, thereby allowing integrated extension of VXLAN network identifiers (VNIs) to remote sites that are either VXLAN EVPN-based fabrics or classic Ethernet-based networks. Such function will become available starting with the Nexus 9000-EX Series Switches or with Cisco Nexus 7000 Series Switches with M3 line cards. Note: In addition to Layer 2 connectivity between fabrics, the exchange of Layer 3 information for IP subnets is required. Technologies like VRF-lite, MPLS L3VPN, or LISP can accommodate this requirement. 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 17

For more information about VXLAN EVPN with the extension to MPLS L3VPN (BorderPE), including examples showing configuration details, please refer to the white paper Configure the Cisco Fabric Border Provider Edge Feature for VXLAN EVPN Fabric at https://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-7000- series-switches/white-paper-c11-737109.pdf. For information about connecting VXLAN EVPN fabric with LISP for Layer 3 connectivity, please refer to the following white paper: http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/whitepaper-c11-734843.html. Hardware and Software Requirements This section lists the requirements for a VXLAN EVPN plus OTV deployment. VXLAN EVPN Plus OTV Requirements The requirements for VXLAN EVPN plus OTV are summarized in the tables that follow. Table 1 provides the hardware and software requirements for a Cisco Nexus 7000 Series Switch providing VXLAN EVPN border node and OTV edge device (two VDCs). Table 2 provides the requirements for the Cisco Nexus 9000 Series Switch to act as a VXLAN EVPN border node. Table 3 indicates the minimum requirements for the Cisco Aggregation Services Router (ASR), Integrated Services Router (ISR), or Cloud Services Router (CSR) to serve as the OTV edge device. Table 1. Minimum Software and Hardware Requirements for Cisco Nexus 7000 Series (Border Node and OTV Edge Device) Item Cisco Nexus hardware Cisco NX-OS software Requirement Cisco Nexus 7000 Series Switches with F3 or M3 * line cards Cisco NX-OS Release 7.3(0)D1(1) or later * M3 line cards will be supported in a later NX-OS code release. Table 2. Minimum Software and Hardware Requirements for Cisco Nexus 9000 Series (Border Node) Item Cisco Nexus hardware Cisco NX-OS software Requirement Cisco Nexus 9200, 9300, or 9300-EX Series Switches Cisco Nexus 9500 Series Switches with 9564, 9536, or 9732 line cards * Cisco NX-OS Release 7.0(3)I1(1) * Cisco Nexus 9500 line cards are supported starting with the NX-OS 7.0(3)I2(1) release. Table 3. Minimum Software and Hardware Requirements for ASR, ISR, or CSR (OTV Edge Device) Item Cisco ASR hardware Cisco IOS XE software Cisco ISR hardware Cisco IOS XE software Cisco CSR Virtual Router Cisco IOS XE Software Requirement Cisco ASR 1000 Series Router Cisco IOS XE 3.5S or later Cisco 4451-X ISR Cisco IOS XE 3.8S or later Cisco CSR1000V Cisco IOS XE 3.10S or later The hardware and software requirements for the leaf and spine nodes in the VXLAN EVPN fabric (programmable fabric) and the requirements for the OTV network remain the same as without OTV integration. 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 17

Fabric and OTV Requirements This document does not cover the hardware and software requirements for the VXLAN EVPN fabric. The following link provides access to the Cisco website, where you can find more information about fabric deployment: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/pf/configuration/guide/b-pf-configuration.html. Please consult the publicly available documents on the Cisco website for additional information about OTV deployment. VXLAN EVPN with OTV Deployment Details This section provides physical and logical overviews of a VXLAN EVPN deployment and the interconnection of the fabrics with OTV. Figure 2 shows how the fabrics interconnect with OTV. Figure 2. VXLAN EVPN Fabrics Interconnected with OTV Physical Overview Figure 3 shows the network topology used in this document to describe the VXLAN EVPN plus OTV integration and the DCI between fabrics. Figure 3. Physical Deployment Overview 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 17

The programmable data center fabrics in Figure 2 are assigned to private autonomous system (AS) numbers and use a Clos architecture consisting of spine and leaf nodes. The configuration of these nodes is performed using either the command-line interface (CLI) or Data Center Network Manager power-on auto provisioning (POAP) templates. The Layer 3 transport network between the VXLAN EVPN fabrics in Figure 2 can be constructed with external Border Gateway Protocol (BGP) or any other dynamic routing protocol, such as Open Shortest Path First (OSPF). The borders can be connected either back to back, best in a full-mesh, or through an intermediate network. The transport network between the border nodes needs to accommodate the requirements for the respective OTV deployment (unicast or multicast mode). Details about the OTV deployment modes can be found in the Cisco Nexus 7000 Series NX-OS OTV Configuration Guide at http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nxos/otv/config_guide/b_cisco_nexus_7000_series_nx-os_otv_configuration_guide/adv-otv.html. This document assumes that the transport network is already built and ready for use. The connectivity from the border nodes in data center fabric 1 to the border nodes in data center fabric 2 is already established, providing underlying transport for OTV as well as VRF-aware routing for the present IP subnets (tenant networks). Each spine node in the fabric is deployed as a multiprotocol BGP (MP-BGP) route reflector for the EVPN address family. The leaf nodes and border nodes peer with the fabric MP-BGP route reflectors in their respective fabrics. Within the data center fabric, the MP-BGP EVPN control plane provides host as well as internal and external prefix reachability information. This BGP EVPN control plane in the VXLAN EVPN fabric helps ensure distribution of routes between VXLAN tunnel endpoints (VTEPs) hosted on each leaf node and the border nodes within the fabric. The leaf node will forward the attached host IP and MAC addresses using the EVPN route type 2 option (for this route type, BGP has both the IP and MAC address information), including Layer 2 and Layer 3 VXLAN network identifiers (VNIs). Recall that a VXLAN VNI is a 24-bit identifier that provides a total of 16 million addressable entities. In VXLAN EVPN deployments, this addressable space is used for uniquely identifying a Layer 2 network (L2VNI) as well as a Layer 3 VRF (L3VNI). The border nodes on the two fabrics are assigned to private autonomous systems and provide redundant Layer 3 connectivity between the fabric and the VRF-aware Layer 3 transport network. Note: The Anycast Gateway MAC (AGM) has to be the same for all fabrics. The border nodes are configured with all the local VRF instances associated with Layer 3 VNI (L3VNI) for external connectivity. In addition, the Layer 2 segments and associated Layer 2 VNI (L2VNI) relevant for extension via OTV are also terminated at the border nodes. To ensure resiliency, the border nodes provide Layer 2 redundancy with virtual port channels (VPCs) for the connectivity to the OTV edge devices. In order to accommodate the Layer 2 termination on the Cisco Nexus 7000 Series Switch acting as the VXLAN tunnel endpoint (VTEP), respective bridge domain, Layer 2 VNI (L2VNI) with associated multicast group, and EVPN instances (EVI) have to be configured. Example 1 shows such a Layer 2 configuration. Example 1. Layer 2 VNI Configuration for OTV Extended VLAN (Nexus 7000 CLI) system bridge-domain 3000 bridge-domain 3000 member vni 30000 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 17

evpn vni 30000 l2 rd auto route-target import auto route-target export auto interface nve1 member vni 30000 mcast-group 239.1.1.30 encapsulation profile vni TO-OTV dot1q 100 vni 30000 interface port-channel72 service instance 1 vni no shutdown encapsulation profile TO-OTV default The port channel toward the OTV edge devices (inside interface) requires the necessary encapsulation profile and virtual service instance (VSI). Next to the VLANs required for OTV extension, the OTV site VLAN also must be present on the inside interface and on the VPC peer link. The extension of the site VLAN beyond the OTV edge device facilitates native multihoming, backdoor-link detection, and fast convergence. In the latter case, the OTV site VLAN is enhanced with a switched virtual interface (SVI) and bidirectional forwarding detection (BFD). Toward the transport network between the fabrics, the OTV join interface is recommended to be a Layer 3 port channel with dynamic unicast routing configured, as documented in the configuration guide for OTV fast convergence (http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nxos/otv/config_guide/b_cisco_nexus_7000_series_nx-os_otv_configuration_guide/advotv.html#concept_2fc307827b054586b3cf21e299d0b7e8). With dynamic route exchange and presence of the join interface s IP address (host route) in the OTV edge devices routing table, OTV fast convergence becomes capable of tracking the next-hop adjacency. OTV fast convergence avoids the need for protocol timer adjustment by adding failure detection through BFD (inside interface) and next-hop tracking (join interface). VXLAN EVPN with OTV Details The extension of Layer 2 segments with OTV requires that the fabric border node be able to split the IP and MAC address information advertised in EVPN route type 2 advertisements so that MAC-only extension is achieved via OTV Layer 2 and IP-only via the VRF-aware Layer 3 extension. This section briefly discusses the control-plane and data-plane exchange in the respective forwarding directions. In addition, EVPN route types and respective attributes, such as sequence number used for host mobility, are discussed. Further, we touch upon how the separation of the MAC and IP routes in EVPN (route type 2) results in an individual MAC-only (route type 2) and IP-only (route type 5) route in fabric 2. The control-plane and data-plane flow is explained, including details of what encapsulation and control plane is responsible at what stage and in which direction. 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 17

Forwarding from the Fabric Toward OTV Once a host is learned at a leaf in a VXLAN EVPN fabric, at the border node, the received IP and MAC address information is learned as part of EVPN route type 2 advertisement. For the subsequent forwarding across OTV for Layer 2 and VRF-aware transport for Layer 3, the routing and bridging portions are separated. Figure 4. Layer 3: Fabric to Fabric via VRF-Aware Transport For the IP subnets that are stretched via OTV, the IP address portion (host routes) of the EVPN route type 2 messages must be advertised via a dynamic routing protocol with VRF-aware transport to fabric 2, as shown in Figure 4. Such a VRF-aware routed transport can be achieved by using technologies such as MPLS L3VPN, LISP, or VRF-lite. As an alternative option to a separate VRF-aware transport, OTV can provide this service by using a dedicated OTV extended VLAN per VRF. In the case of OTV being used also for VRF-aware routed transport, a dedicated inside interface would work best, neighbored by a Layer 3 interface or subinterface on the border node. Nevertheless, in any of these cases the EVPN route type 2 MAC portion and mobility sequence attribute is lost, as routing protocols don t carry this information. For the host IP address advertisement received via the VRF-aware routed transport, the border node in fabric 2 readvertises it into EVPN. At this stage, the previous EVPN route type 2 advertisement in fabric 1 becomes a route type 5 (IP-only) advertisement in fabric 2 as learned via an external routing exchange. The reason for the exchange of host route information for stretched IP subnets resides in the need for supporting inter-subnet communication via routing. With multiple IP subnets within a given VRF, stretched or local, without the exchange of routing information, Layer 3 forwarding would be impaired. Note: By default, host routes are advertised across sites also for end-points belonging to non-stretched IP subnets. 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 17

Figure 5. Layer 2: Fabric to Fabric via OTV transport As previously mentioned, the MAC portion is not exchanged via traditional routing protocols. Along with Layer 3 traffic, Layer 2 traffic is terminated on the border node. Note that the MAC address information will be present in the local MAC address table on the border node, learned via the BGP EVPN route type 2 update. From the border node toward the OTV edge device, the MAC address is learned in a traditional way through data-plane communication and is subsequently advertised via OTV s control-plane exchange to the remote OTV edge device (Figure 5). From here, fabric 2 receives an individual MAC address over OTV (through data-plane learning) that is advertised as a MAC-only route type 2 advertisement into EVPN. Recall that the associated host IP address was advertised using an EVPN route type 5 in fabric 2. Consequently, with this separation of Layer 2 and Layer 3 communication exchange, the previous host object in fabric 1 has distinct and separate routing and bridging entries within EVPN in fabric 2. Note: For an IP subnet that is extended through Layer 2 between multiple fabrics, instantiation of the distributed IP anycast gateway on the border nodes is not supported. This is because, with the instantiation of the distributed IP anycast gateway on the border nodes that also extend the Layer 2 network, the same MAC and IP address becomes visible on the Layer 2 extension on both fabrics, and unpredictable learning and forwarding can occur. Even if the Layer 2 network is not extended between fabrics, based on potential later extension of the same, we don t recommend that the distributed IP anycast gateway be present on the border node. Layer 2 Communication via OTV Toward Remote Fabric When we consider a host connected to a leaf in fabric 1 that wants to communicate at Layer 2 to a host connected to a leaf in fabric 2 (Figure 6), the following steps are involved: 1. An initial ARP resolution between the host connected in fabric 1 and the host connected in fabric 2 is required. The ARP request must be forwarded from fabric 1 toward fabric 2 via the Layer 2 extension. This ARP request/response will generate the host s IP and MAC routes inside its own fabric within EVPN (route type 2). For bridging between hosts in the same IP subnets, the Layer 2 VNI is used for VXLAN encapsulation. 2. The EVPN route type 2 routes (host IP and MAC routes) are imported on the border node into the local VRF instances (namely the IP VRF and the MAC VRF) according to the BGP route-target filtering. 3. Imported MAC information installed into the local MAC address table is subsequently seen in the OTV edge device via data-plane learning. The OTV edge device will advertise the learned MAC address toward fabric 2. 4. In fabric 2 the MAC address is advertised as an EVPN type 2 route (MAC only). 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 17

Figure 6. Layer 2: EVPN Route Advertisements Layer 3 Communication Toward Remote Fabric When we consider a host connected to a leaf in fabric 1 that wants to communicate at Layer 3 to a host connected to a leaf in fabric 2 (Figure 7), the following steps are processed: 1. An initial ARP resolution for the default gateway is required in order to allow inter-subnet routing between hosts. The ARP request is forwarded only between the host and its connected leaf node that hosts the anycast gateway (depending on enabled ARP suppression). This ARP request/response will generate the host s IP and MAC routes within EVPN (route type 2). For the routing between hosts in different IP subnets, the Layer 3 VNI is used for VXLAN encapsulation. 2. The EVPN type 2 routes (host IP and MAC routes) are imported on the border node into the local VRF instances according to the BGP route-target filtering. 3. Imported IP host routes are advertised to the VRF-aware Layer 3 transport toward fabric 2. 4. In fabric 2 the IP address is advertised as an EVPN type 5 route (IP only). Figure 7. Layer 3: EVPN Route Advertisements 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 17

Host Mobility via OTV Toward Remote Fabric The ability to move hosts between fabrics has been one of the major use cases for data center networks. The ability to dynamically and manually move hosts to new locations provides the opportunity for better load distribution or failure handling in the sense of high availability. Figure 8. Pre-Host Move When we consider a host moving from a leaf in fabric 1 to a leaf in fabric 2, the following steps are involved: 1. Host moves from the leaf in fabric 1 to a leaf in fabric 2, as shown in Figure 8. 2. Once the host move is completed, the virtual switch at the destination server typically issues a gratuitous ARP (GARP) or reverse ARP (RARP) to signal completion. 3. The GARP and RARP notification is used to withdraw and update the ARP table state to reflect the new location of the host (fixup). This message will update the Layer 2 tables along the path. 4. During fixup, the state tables (MAC, ARP, routing) are modified to reflect the correct situation after the move. a. In fabric 2, the previous individual EVPN routes are withdrawn. As per the new learning of the host in fabric 2, a single IP and MAC route (EVPN route type 2) with adjusted MAC mobility sequence number is now present. b. In fabric 1, the previous single EVPN route type 2 (IP and MAC) is withdrawn. Via Layer 2 DCI we learn an EVPN route type 2 (MAC only) and EVPN route type 5 (IP only) via Layer 3 DCI. The MAC mobility sequence number is also updated. 5. The host move from the leaf in fabric 1 to the leaf in fabric 2 is now complete (see Figure 9). 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 17

Figure 9. Post-Host Move Fault Containment Data center redundancy helps ensure high availability. In that sense, if a failure affects one data center or a fabric, we want to ensure that respective failure containment measures are present. Layer 2 extensions are prone to replication of such failures. Various technologies and features try to introduce failure containment measures, but they require appropriate placement and configuration. Cisco OTV provides some built-in failure containment functionality. The use of OTV reduces the risk of failure propagation, as unknown unicast flooding, broadcast group separation, and backdoor paths are automatically detected and mitigated. Unknown Unicast Flooding With unknown unicast flooding, traffic is mostly unnecessarily transported across DCI links. Excessive flooding often occurs in the presence of Layer 2 loops or in cases of frequent topology changes (frequent MAC/ARP table flush). By disabling unknown unicast flooding across a DCI, OTV prevents the impact of such traffic patterns from one fabric to other fabrics, thereby avoiding failure propagation and providing isolation (see Figure 10). Figure 10. Unknown Unicast Flooding 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 17

Broadcast Storms In cases where Layer 2 loops exist in a network, the existence of broadcast traffic can create significant harm. As broadcasts are an integral part of Layer 2 networking, it is not possible to disable broadcast forwarding completely in a network. With the absence of broadcasts, some crucial learning mechanisms triggered by protocols such as ARP, DHCP, etc. would not function anymore. Regardless of the importance of certain broadcasts, in the case of a failure, we want to minimize the blast radius and minimize or limit the impact of a broadcast storm from one fabric to adjacent ones. From the border node of fabric 1 toward the OTV edge devices, a classic Ethernet link exists, and thus storm control can be applied. In addition, OTV allows the broadcast traffic to be placed into a separate multicast group. With this separation, the broadcast, unknown unicast, and multicast can be treated differently, and excessive broadcasts can be identified and rate-limited (see Figure 11). Figure 11. Broadcast Storm Backdoor Path Layer 2 extensions bring the threat of a backdoor path during network changes and migration (see Figure 12). Traditional Layer 2 extensions do not provide an integrated approach to detect such a looped topology, other than the use of Spanning Tree. In networks that provide filtering of Spanning Tree bridge protocol data units (BPDUs), detection of a backdoor path would be prevented or simply isn t possible. With Cisco OTV, the site VLAN mechanism, together with the site ID, understands the concept of a site, which is also part of the integrated multihoming approach. In cases where the OTV site VLAN is common in all fabrics but the site ID is different, a backdoor path will be detected and the potential resultant loop prevented. In the specific case of backdoor path detection, the OTV tunnel will be shut down, thereby ensuring that a loop is prevented (see Figure 12). 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 17

Figure 12. Backdoor Path Configuration Example The following configuration example refers to the diagram in Figure 13. The two border nodes can be Cisco Nexus 9000 or 7000 Series Switches, or any other VXLAN BGP EVPN-capable switch, providing VXLAN BGP EVPN functionality with virtual port channels. While the VXLAN BGP EVPN configuration might encompass additional configuration for underlay and overlay toward the fabric or external connectivity, the configuration example itself focuses on the specifics for facilitating the OTV extension. Figure 13. Connectivity Configuration Example Example 2. Cisco Nexus 9000 Border Node: VXLAN Layer 2 VNI Configuration (including Site VLAN) vlan 99 vn-segment 30099 vlan 100 vn-segment 30000 VLAN to VNI for site VLAN (99) VLAN to VNI for extended VLAN (100) 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 17

evpn vni 30000 l2 rd auto route-target import auto route-target export auto vni 30099 l2 rd auto route-target import auto route-target export auto interface nve1 member vni 30000 mcast-group 239.1.1.30 member vni 30099 mcast-group 239.1.1.2 interface port-channel72 mtu 9000 switchport switchport mode trunk switchport trunk allowed vlan 99,100 interface port-channel1 mtu 9192 ip address 10.72.72.2/30 ip ospf network point-to-point ip router ospf CORE area 0.0.0.0 ip igmp version 3 EVPN virtual instance (EVI) for VXLAN EVPN Layer 2 services VXLAN tunnel endpoint (VTEP) interface with VNI and based on multicast for broadcast, unknown unicast, and multicast replication Port channel toward OTV edge device in mode Trunk with allowed VLAN list Layer 3 Port channel toward OTV edge device, providing OTV join-interface connectivity Example 3. Cisco Nexus 7000 Border Node: VXLAN Layer 2 VNI Configuration (including Site VLAN) system bridge-domain 9,3000 vni 30000,30099 bridge-domain 9 bridge-domain 3000 bridge-domain 9,3000 member vni 30099,30000 evpn vni 30000 l2 rd auto route-target import auto route-target export auto vni 30099 l2 rd auto route-target import auto route-target export auto Bridge domain for further VNI mapping VNI to be used for VXLAN Bridge domain for site VLAN (99) Bridge domain for extended VLAN (100) Mapping of the carved bridge domain to the prepared VNI EVPN virtual instance (EVI) for VXLAN EVPN Layer 2 services 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 17

interface nve1 member vni 30000 mcast-group 239.1.1.30 member vni 30099 mcast-group 239.1.1.2 encapsulation profile vni TO-OTV dot1q 99,100 vni 30099,30000 interface port-channel72 mtu 9000 fabric forwarding port-l2dci service instance 1 vni no shutdown encapsulation profile TO-OTV default interface port-channel1 mtu 9192 ip address 10.72.72.2/30 ip ospf network point-to-point ip router ospf CORE area 0.0.0.0 ip igmp version 3 VXLAN tunnel endpoint (VTEP) interface with VNI and based on multicast for broadcast, unknown unicast, and multicast replication Encapsulation profile to map VLAN IDs from the Ethernet wire (802.1Q) to a VNI Port channel toward OTV edge device with virtual service instance (VSI) to attach encapsulation profile fabric forwarding port-l2dci provides additional topology information to the border node. End host port vs. DCI port Layer 3 Port channel toward OTV edge device, providing OTV join-interface connectivity Example 4. OTV Edge-Device: Inside Interface interface port-channel72 mtu 9000 switchport switchport mode trunk switchport trunk allowed vlan 99,100 interface Ethernet1/22 switchport switchport mode trunk channel-group 72 mode active no shutdown interface Ethernet2/22 switchport switchport mode trunk channel-group 72 mode active no shutdown OTV inside interface port channel toward VXLAN EVPN border node with increased MTU to accommodate end host Physical interfaces as part of the OTV inside interface port channel 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 17

Example 5. OTV Edge Device: Join Interface interface port-channel1 mtu 9192 ip address 10.72.72.1/30 ip ospf network point-to-point ip router ospf CORE area 0.0.0.0 ip igmp version 3 interface Ethernet1/24 mtu 9192 channel-group 1 mode active no shutdown interface Ethernet2/24 mtu 9192 channel-group 1 mode active no shutdown OTV join interface port channel toward Layer 3 network with increased MTU to accommodate end host plus encapsulation overhead Physical interfaces as part of the OTV join interface port channel Example 6. OTV Edge Device: OTV Configuration feature otv otv site-vlan 99 otv isis bfd interface Overlay0 otv join-interface port-channel1 otv extend-vlan 100 otv control-group 239.239.239.1 otv data-group 232.0.0.0/25 otv broadcast-group 239.239.239.9 no shutdown otv-isis default track-adjacency-nexthop otv site-identifier 0x7 otv encapsulation-format ip udp Enabling the OTV feature Definition of the OTV site VLAN (99) with enabled fast convergence "OTV overlay interface with definition of the join interface, the extended VLAN (100) OTV control group for control plane (PIM ASM), the data group for multicast traffic (PIM SSM) and broadcast group for broadcast traffic separation (PIM ASM) OTV next-hop tracking for fast convergence OTV site identifier (site 7) for multihoming OTV encapsulation set to UDP (VXLAN) Conclusion VXLAN EVPN multifabric is a hierarchical network design comprising individual fabrics interconnected together. The design focuses on the individuality of the data center domains, allowing independent scale and, more important, independent failure domains. The connectivity between the individual fabric domains is independent of the choice that is being used within the data center, and thus a natural separation is achieved. Overlay Transport Virtualization (OTV) provides Layer 2 extension while maintaining failure containment. With this ability and the additional attributes OTV offers for Data Center Interconnect (DCI), modern data center fabrics can be extended in an optimized fashion. With OTV as the interconnectivity technology, integrated functions are used to optimize Layer 2 DCI between multiple VXLAN EVPN fabrics. 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 17

Printed in USA C11-738358-00 01/17 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 17