Configuring Cisco Nexus 9000 Series Switches for VMware NSX OVSDB Integration

Transcription

1 White Paper Configuring Cisco Nexus 9000 Series Switches for VMware NSX OVSDB Integration First published: January 28, Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 62

2 Contents Executive summary... 5 Use cases... 6 How Cisco Nexus 9000 Series OVSDB integration with VMware NSX works... 6 OVSDB control plane... 8 OVSDB protocol... 8 OVSDB communication between VMware NSX controllers and Cisco 9000 Series Switches... 8 Packet flow for VMware NSX connected virtual machines and physical workloads BUM traffic replication using RSNs RSN availability: Bidirectional Forwarding Detection over VXLAN HW-VTEP redundancy: vpc and anycast loopback Routing for VNIs that have an HW-VTEP binding Configuring the Cisco Nexus 9000 Series for HW-VTEP OVSDB integration with VMware NSX Configuration checklist Supported Cisco Nexus 9300 platform switches Verifying that your Cisco Nexus 9300 platform switch is running the correct Cisco NX-OS Software release 19 Installing the TP Services Package license for NXDB Verifying that the correct version of the plug-in and JRE are installed Carving the TCAM to enable BFD over VXLAN on first generation Cisco Nexus 9300 platform switches Verifying the vpc configuration Configuring the anycast loopback address for vpc pairs Configuring a username and password for use by the NXDB plug-in if desired Verifying that the VMware NSX version and configuration are supported Verifying the required network reachability Reserving VLANs and other switch resources Configuring OVSDB integration with VMware NSX Configuring the required features on the switch Configuring VXLAN on the switch Configuring BFD over VXLAN Assigning VLANs and interfaces to the controllers Configuring the guest shell for the OVSDB plug-in Installing the OVSDB plug-in Configuring the OVSDB plug-in for a standalone switch Configuring the OVSDB plug-in for a pair of vpc switches Enabling the OVSDB plug-in Registering the HW-VTEP with VMware NSX Binding a logical switch in VMware NSX to a physical switch, physical port, and VLAN Verification and troubleshooting Limitations of VMware NSX OVSDB integration with HW-VTEPs Configuring the Cisco Nexus 9000 Series Switch as the default gateway for the VNI and VLAN Configuring a redundant default gateway on two vpc switches acting as HW-VTEPs using HSRP Conclusion Appendix A: Upgrading the Cisco NX-OS image and the OVSDB plug-in for vpc Check prerequisites Upgrade the OVSDB plug-in on the vpc secondary switch Upgrade the OVSDB plug-in on the vpc primary switch Upgrade the Cisco NX-OS image What to do next Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 62

3 Appendix B: Best-practice configurations for vpcs Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 62

4 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental. This product includes cryptographic software written by Eric Young (eay@cryptsoft.com). This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. ( This product includes software written by Tim Hudson (tjh@cryptsoft.com). Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R) 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 62

5 Executive summary This document is designed for network and virtualization architects interested in deploying Open vswitch Database (OVSDB) for hardware Virtual Extensible LAN (VXLAN) tunnel endpoint (HW-VTEP) integration between Cisco Nexus 9000 Series Switches and VMware NSX. The Cisco Nexus 9000 Series delivers proven high performance and density, low latency, and exceptional power efficiency in a broad range of compact form factors. The Cisco Nexus 9000 Series offers an open-object APIprogrammable model for provisioning Layer 2 and 3 features. It provides extensibility through a route processor module application package, Linux containers, and Broadcom and Linux shell access. It uses the Cisco NX-OS Software API (NX-API) for easy-to-use programmatic access. VMware NSX is a network virtualization platform software suite. It embeds networking and security functions that typically are handled in hardware directly into the hypervisor. With NSX, users can reproduce data center networking environments in software. NSX provides a set of logical networking elements and services, including logical switching, routing, firewalling, and load balancing, for VMware vsphere virtualized environments. NSX virtual networks can be programmatically provisioned and can run independent of the underlying hardware to support virtual machines running on a VMware hypervisor. In some NSX deployments, NSX logical switches must be extended into the physical environment. The goal of this integration is to connect virtualized workloads using NSX as their networking layer with physical workloads that are not virtualized and that need to be on the same network subnet. This integration requires the bridging of VXLAN network identifiers (VNIs) with virtual LANs (VLANs). The OVSDB integration discussed in this document allows the Cisco Nexus 9000 Series Switch to be an HW-VTEP, which performs the translation from VXLAN encapsulation to VLAN encapsulation, and back, in hardware to connect workloads in the virtual and physical environments. The OVSDB integration allows the NSX controllers to configure VXLAN on the Cisco Nexus 9000 Series Switch. The NSX controller maps the VXLAN segments to the VLAN segments on specific ports of the Cisco Nexus 9000 Series Switch and pushes this configuration to the switch. The NSX controller uses the OVSDB management protocol to push the configuration changes to the switch Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 62

6 Use cases From a logical networking perspective, the OVSDB integration allows the virtual workloads running within the NSX network and connected to VNIs to be on the same subnet and broadcast domain as physical workloads connected to VLANs through the Cisco Nexus 9000 Series Switches. Figure 1 shows a logical view of the connectivity achieved by the integration. Figure 1. Logical view of VNI-to-VLAN connectivity How Cisco Nexus 9000 Series OVSDB integration with VMware NSX works The best way to understand the integration of OVSDB with Cisco Nexus 9000 Series Switches and NXS is to look at the connections one step at a time. Figure 2 shows the physical locations of the workloads. Figure 2. Physical connectivity 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 62

7 Figure 2 shows three virtual workloads: VM1, VM2, and VM3. Their IP addresses are (VM1), (VM2), and (VM3). The three virtual machines are all connected to VNI They are on the same subnet as the physical workload at the bottom of the figure using IP address The physical workload is connected to VLAN 500. VM1 and VM2 are both on a VMware ESXi host (H1) running NSX. H1 has a VMkernel interface used by NSX as its VTEP with the IP address VM3 is located on H3 with a VTEP address of The physical workload is connected to a Cisco Nexus 9000 Series Switch (N9K) with a loopback address of When VM1 sends data to the physical workload, a VXLAN tunnel is formed between H1and N9K, as indicated by the purple line in Figure 3. Figure 3. VXLAN tunnel between H1 and N9K The tunnel shown in Figure 3 carries the traffic between VM1 and the physical workload. The packet from VM1 is encapsulated in a VXLAN header with a source IP address of (VTEP for H1) and a destination IP address of (loopback IP address for N9K) and using VNI It is up to the data center network to deliver the packet. So long as there is Layer 3 IP reachability between the H1 VTEP and the N9K loopback IP address, VM1 and the physical workload can communicate with each other as if they are on the same VLAN (or VXLAN) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 62

8 Figure 4 shows a simplified diagram of a packet. Figure 4. Simplified diagram of an IP packet encapsulated by a VXLAN header The HW-VTEP, in this case N9K, will take the VXLAN encapsulated packet received from H1, remove the outer headers (IP, UDP, and VXLAN headers), insert a VLAN 500 header, and deliver the packet on its physical port. It will also take all VLAN 500 encapsulated packets received from the physical workload destined for VM1, remove the VLAN header, and insert appropriate VXLAN headers (IP, UDP, and VXLAN 5000 headers), with an IP destination of H1 and a source IP address consisting of its loopback address. This process all is performed in hardware on the Cisco Nexus 9000 Series Switch. OVSDB control plane This section describes the OVSDB control plane. OVSDB protocol Open vswitch Database, or OVSDB, is a management protocol, detailed in RFC 7047, designed to be used in Software-Defined Networking (SDN) environments. OVSDB allows the NSX controllers and the Cisco Nexus 9000 Series Switches to communicate with one another. The NSX controllers use the OVSDB protocol to communicate with HW-VTEPs. OVSDB-based communication uses the VTEP 5 schema, which defines the data format for the communication between an external controller and a switch. NX-OS implements the OVSDB protocol by means of an intermediate agent in the form of a plug-in, maintained by Cisco. This plug-in accepts the OVSDB messages sent from the NSX controllers and makes the appropriate JavaScript Object Notation Remote Procedure Call (JSON-RPC) NX-API calls on the Cisco Nexus 9000 Series Switch. The plug-in runs in the Cisco Nexus 9000 Series Switch s guest shell container to provide security and isolation between different control planes that may be running on the switch. The OVSDB plug-in has OVSDB as its northbound interface (facing toward the NSX controllers) and the JSON- RPC NX-API as its southbound interface (facing toward the switch). OVSDB communication between VMware NSX controllers and Cisco 9000 Series Switches In this integration, the NSX controllers are responsible for handling the interaction with the hardware gateways (the Cisco Nexus 9000 Series Switches). For this purpose, a connection is established between the NSX controllers and a dedicated piece of software called the hardware switch controller (HSC). For the Cisco implementation, the HSC is the plug-in running in the Cisco Nexus 9000 Series Switch s guest shell. Figure 5 shows the communication path from the NSX controllers to the Cisco Nexus 9000 Series Switch as described in the preceding sections Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 62

9 Figure 5. Communication path from the VMware NSX controllers to the Cisco Nexus 9000 Series Switch After the connectivity between the NSX controllers and the Cisco Nexus 9000 Series Switches is configured and established, the NSX controllers will push the administrator-configured association between a logical switch (VNI) and the physical switch, port, and VLAN to the Cisco Nexus 9000 Series hardware gateway through the HSC. The NSX controller will also advertise a list of replication service nodes (RSNs), which the Cisco Nexus 9000 Series Switches will use to forward broadcast, unknown unicast, and multicast (BUM) traffic. The RSN function is discussed Packet flow for VMware NSX connected virtual machines and physical workloads section of this document. A typical NSX deployment generally has three NSX controllers, providing redundancy and scale-out capabilities. The HSC will open a connection to all three controllers, as shown in Figure Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 62

10 Figure 6. Connection between the HSC running in the Cisco Nexus 9000 Series Switch and the VMware NSX controllers The NSX controllers will advertise the list of hypervisor VTEP IP addresses relevant to the logical switches configured on the hardware gateway to the HSC. The NSX controllers also will advertise the association between the MAC addresses of the virtual machines in the virtual network and the VTEP through which they can be reached. Now return to example introduced in Figure 3 and depicted in Figure 7. Figure 7. VXLAN tunnel between H1 and N9K 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 62

11 After the OVSDB relationship is established, the NSX controllers will advertise to the HW-VTEP that the MAC addresses for VM1 and VM2 (MAC A and MAC B) are reachable at IP address (the VTEP IP address of H1), and that the MAC address for VM3 (MAC C) is reachable at The Cisco Nexus 9000 Series Switch s HSC will advertise to the NSX controllers that the MAC address for the physical workload (MAC D) is reachable at (the loopback address of N9K). The NSX controllers in turn let H1 and H3, the hosts that have workloads on the VNI that is mapped to VLAN 500, know that MAC D is reachable through N9K as required. Note that the NSX controllers use OVSDB only toward the HW-VTEP, and the hypervisor VTEPs are programmed using VMware s proprietary protocols. The Cisco Nexus 9000 Series Switch running as an HW-VTEP includes entries in the MAC address table for MAC A and MAC B pointed at and MAC C pointed at Listing 1 shows an example. The top MAC address belongs to VM1, and it is reachable at H1. The second MAC address is the physical workload connected to interface Eth1/31 on VLAN 500. Listing 1 Example of a MAC address table on a Cisco Nexus 9000 Series Switch N9K# sh mac address-table vlan 500 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vpc Peer-Link, (T) - True, (F) - False, C - ControlPlane MAC VLAN MAC Address Type age Secure NTFY Ports C b.7e06 dynamic 0 F F nve1( ) * f.ee84.327c dynamic 0 F F Eth1/31 N9K# The same information can be seen in the NSX controller. The physical workload is reachable through the N9K loopback address ( ), as shown in Listing 2. Listing 2 Example of the MAC address table on the NSX controller nsx-controller # show control-cluster logical-switches mac-table 5000 VNI MAC VTEP-IP Connection-ID :50:56:9b:7e: :7f:ee:84:32:7c nsx-controller # OVSDB is used by the NSX controllers to learn MAC address reachability for physical workloads connected to the Cisco Nexus 9000 Series Switches configured as HW-VTEPs, and to push relevant MAC address reachability information to the HW-VTEPs. Note: Other regular interface configurations required for the daily operation of the switch, such as storm control, Quality of Service (QoS), Access Control Lists (ACLs), and redundancy, will not be configured or monitored by NSX. This type of everyday configuration and monitoring remains the responsibility of the network administrator Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 62

12 Packet flow for VMware NSX connected virtual machines and physical workloads This section discusses the flow of packets for virtual machines and physical workloads connected to NSX. BUM traffic replication using RSNs The NSX for vsphere implementation does not use the HW-VTEP capabilities to perform replication for BUM traffic. BUM traffic replication is performed in software on the RSNs. Therefore, as discussed earlier in this document, when you configure OVSDB integration between NSX and Cisco Nexus 9000 Series Switches, you must select NSX-enabled ESXi hosts as the RSNs that the hardware gateway can use to forward BUM traffic. To better understand how BUM traffic is handled in this integration, look at Figure 8. Figure 8. Logical view of a broadcast domain in OVSDB integration In the figure, the physical workload ( ) is sending traffic to VM1. To begin communicating, the physical workload will send an Address Resolution Protocol (ARP) request for IP (VM1 s IP address). The ARP request is sent as a broadcast message and is used to map IP addresses to MAC addresses. In this example, the virtual network consists of three hypervisors. Two of those hypervisors are devices to which the traffic must be flooded, because they have active virtual machines on the VNI 5000 logical switch. In the current model, the hardware gateway will not use hardware replication for the frames it needs to flood to each of those hypervisors. Instead, the NSX controller has provided a list of RSNs that the hardware gateway will use to perform the replication in software. The RSNs are statically defined by the NSX administrator. For each frame that needs to be flooded, the hardware gateway picks one RSN per VNI to act as a replication server for the virtual world. The RSN then takes care of replicating the BUM traffic in software and forwards it to the appropriate hypervisors that need to receive it. Figure 9 illustrates the process Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 62

13 Figure 9. Sending BUM traffic to the RSN The example in Figure 9 shows that the HW-VTEP (N9K) will take the ARP broadcast packet and forward one copy of it to the chosen RSN for VNI 5000, which in this case is H3. The host should already know the MAC address of the physical workload (MAC D) and its location, which in this case is N9K ( ), from the OVSDB learning described earlier. The second part of the RSN s job replication is shown in Figure Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 62

14 Figure 10. RSN replication Figure 10 shows that H3 will then take the packet and send a broadcast packet to VM3, which is the only virtual machine that is in VNI 5000 on that host. H3 will also look at its VTEP table and replicate the broadcast ARP packet for every host in the NSX domain that has virtual machines active on VNI The replication method used depends on the setup in the NSX manager for the particular logical switch (unicast or hybrid), but it occurs in software. In the example here, the only other host with virtual machines active in VNI 500 is H1. When the replicated packet arrives at H1, H1 will send a broadcast packet to VM1 and VM2 because they are both in VNI When VM1 sees that the ARP request is for itself, it will learn the ARP entry for the physical workload and return an ARP reply with its MAC address (MAC A) directly to the physical workload at MAC D. When that reply reaches the hypervisor of H1, H1 will perform a lookup for MAC D and see that it is sitting behind N9K. H1 will encapsulate the ARP reply in a VXLAN packet with its IP address as the source and the loopback of N9K as the destination, and it will forward the packet to the network. N9K will receive the ARP reply, remove the VXLAN header, and forward the packet to the physical workload on the correct VLAN. The physical workload will learn the MAC address associated with VM1 and will now be able to send unicast packets directly to the virtual machine as described in the preceding sections. RSN availability: Bidirectional Forwarding Detection over VXLAN To avoid black-holing BUM traffic, the HW-VTEP needs to be able to remove a failed RSN from its replication servers list if an RSN host fails. To use enable function, VMware requires the use of Bidirectional Forwarding Detection (BFD) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 62

15 BFD is a detection protocol designed to provide fast forwarding-path-failure detection times for all media types, encapsulations, topologies, and routing protocols. In addition to fast forwarding-path-failure detection, BFD provides a consistent failure detection method for the network administrator. You can enable or disable BFD in the NSX GUI on a global basis for all defined service nodes in the controller. In this context, it provides fast forwarding-path-failure detection times between the NSX RSNs and the HW-VTEP (the Cisco Nexus 9300 platform switch). In this implementation, this feature is essentially BFD over VXLAN. The BFD session is hosted on the Cisco switch and on all the RSNs in the NSX controller. The BFD control packets are encapsulated and decapsulated by the Cisco switch. In virtual Port Channel (vpc) mode, the BFD session is hosted only on the vpc primary device, and the vpc secondary device can receive the encapsulated BFD control frames. In such cases, the vpc secondary device decapsulates the VXLAN packet and sends it over the vpc peer link to the vpc primary device (the BFD hosting device). Explicit Ternary Content-Addressable Memory (TCAM) carving is needed on the first generation Cisco switch 9300 switches to enable this feature. This is implemented by enabling the redirect-tunnel region as discussed in the Carving the TCAM to enable BFD over VXLAN on first generation Cisco Nexus 9300 platform switches section of this document. In a BFD-enabled setup, the VNIs are hashed to only those replication servers that have BFD enabled and are in a session up state. When a BFD session goes down in a replication server, the VNIs are rehashed dynamically to the available replication servers that have BFD enabled and are in the BFD session up state. This function provides dynamic rebalancing of BUM traffic flows upon fast forwarding-path-failure detection. It is important to note that forwarding of traffic requires that the BFD session be active between the hardware VTEP and the RSNs. HW-VTEP redundancy: vpc and anycast loopback The Cisco Nexus 9000 HW-VTEP integration with NSX can take advantage of Cisco vpc technology to provide high availability for connectivity of physical devices. vpcs allow links that are physically connected to two different Cisco switches to appear to a third downstream device to be coming from a single device and as part of a single port channel. The third device can be a switch, a server, or any other networking device that supports IEEE 802.3ad port channels. Using vpcs and an anycast loopback address, two Cisco Nexus 9000 Series Switches will appear as a single HW-VTEP device to the NSX manager, controllers, and hypervisors, as shown in Figure Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 62

16 Figure 11. Using vpc and anycast loopback addresses for high availability Routing for VNIs that have an HW-VTEP binding When an NSX logical switch is connected to an HW-VTEP using OVSDB, it cannot be attached to a Distributed Logical Router (DLR) at the same time. This limitation exists for all NSX implementations of this feature, regardless of the hardware vendor providing the HW-VTEP function. Traditionally, this limitation meant that the default gateway for the virtual machines and bare-metal devices attached to the VNI and VLAN combination had be an external device. This device could be an Edge Services Gateway (ESG) virtual machine attached to the VNI, or it could be a traditional router connected to the VLAN (or a physical firewall or another service device). These traditional options are shown in Figure 12. Figure 12. Traditional methods of providing a default gateway for VNIs and VLANs 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 62

17 With the new cloud-scale application-specific integrated circuits (ASICs) available in newer Cisco Nexus 9000 EX platform switches, the Cisco Nexus 9000 Series Switches performing the OVSDB integration can also be the default gateways for the extension of the subnet. This capability allows savings in Capital Expenditures (CapEx) because external physical routers are no longer necessary. The Cisco Nexus 9000 Series Switches can perform routing in hardware, while also saving Operating Expenses (OpEx) by providing the default gateway and routing capabilities using the well-known switch virtual interface (SVI) feature. Redundancy can be achieved by using a First-Hop Redundancy Protocol (FHRP) such as Hot-Standby Router Protocol (HSRP). This new capability is shown in Figure 13. Figure 13. New method of providing a default gateway for VNIs and VLANs Configuring the Cisco Nexus 9000 Series for HW-VTEP OVSDB integration with VMware NSX This section describes how to configure the Cisco Nexus 9000 Series for HW-VTEP OVSDB integration with VMware NSX. Configuration checklist Review the following checklist before installing the OVSDB plug-in and configuring the system: The Cisco Nexus 9300 switch must be one of the models listed in Table 1. The Cisco Nexus 9300 switches should be running the correct version of NX-OS Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 62

18 The Cisco Nexus 9000 Series Switches should have the Cisco Nexus Database (NXDB) license (N93- TP1K9) installed and the feature enabled. The Cisco Nexus 9300 switches should have the correct version of the plug-in and Java Runtime Environment (JRE) downloaded to their bootflash memory. If using first generation Cisco Nexus 9300 switches they must have been configured with the explicit TCAM carving that is needed to enable the redirect-tunnel region. This feature is used for BFD over VXLAN. If high availability is needed, vpcs must be configured correctly on the Cisco Nexus 9300 switches. If high availability with vpc is being used, an anycast loopback address must be configured on the vpc pair. A separate username and password must be configured on the Cisco Nexus 9300 switches to be used by the NXDB process to push API configuration changes, if this capability is desired. The correct supported version of NSX must be installed. The NSX configuration for controllers, NSX VIBs, and VXLAN VTEPs must be completed. The routing for the environment must be set up. The NSX hypervisors (using the NSX VTEP interface), NSX manager, and NSX controllers all should be able to ping the switch s loopback address (the one that is going to be used for HW-VTEP configuration). This address can be the anycast loopback address used if vpc configuration is desired. VLANs must be reserved for use for VXLAN-to-VLAN association. Each NSX logical switch extension will use one VLAN. Also, a VLAN will be used for BFD over VXLAN. The following sections discuss how to configure and verify the items on this checklist. Supported Cisco Nexus 9300 platform switches As of the writing of this document, the only Cisco Nexus 9300 switches supported and certified to run the HW- VTEP OVSDB Integration with NSX are the ones listed Table 1. Table 1. Supported Cisco Nexus devices for OVSDB HW-VTEP integration with VMware NSX Cisco Nexus 9300 platform switch Cisco Nexus 9372PX or 9372PX-E Switch Cisco Nexus 9372TX or 9372TX-E Switch Cisco Nexus 93120TX Switch Cisco Nexus 9332PQ Switch Cisco Nexus 9396PX Switch Cisco Nexus 9396TX Switch Cisco Nexus 93128TX Switch Cisco Nexus 93180YC-EX Cisco Nexus 93108TC-EX Cisco Nexus 93180LC-EX Switch Part number N9K-C9372PX or N9K-C9372PX-E N9K-C9372TX or N9K-C9372TX-E N9K-C93120TX N9K-C9332PQ N9K-C9396PX N9K-C9396TX N9K-C93128TX N9K-C93180YC-EX N9K-C93108TC-EX N9K-C93180LC-EX To verify the type of hardware you have, you can log in to the Cisco Nexus 9300 switch and type the command shown in Listing Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 62

19 Listing 3 Sample display for a show version inc chassis command N9K# show version inc chassis cisco Nexus9000 C9332PQ chassis N9K# Verifying that your Cisco Nexus 9300 platform switch is running the correct Cisco NX-OS Software release As of the writing of this document, the NX-OS release recommended for running the HW-VTEP OVSDB integration with NSX is the one shown in Table 2. Table 2. Recommended Cisco NX-OS Software release for OVSDB HW-VTEP integration with VMware NSX Recommended Cisco NX-OS release Release 7.0(3)I6(1) or later in same main release The recommended version of NX-OS is indicated by the main release number. For example, if the recommended release is Release 7.0(3)I6(1), then any later minor release in the same main release will work: for example, in this case Release 7.0(3)I6(6) would also be recommended. To verify the release of NX-OS in your switch, enter the command shown in Listing 4. Listing 4 Sample display from a show version inc System command N9K# show version inc System Cisco Nexus Operating System (NX-OS) Software System version: 7.0(3)I6(1) N9K# N9k# Installing the TP Services Package license for NXDB The TP Services Package (TP_SERVICES_PKG) license must be installed on the Cisco Nexus 9300 switch. The license part number is N93-TP1K9. Note that this is not an honor-based license. Without a license, you cannot enable the OVSDB features on the switch. The command to verify the licensing is shown in Listing 5. Listing 5 Sample display for a show license usage command Feature Ins Lic Status Expiry Date Comments Count FCOE_NPV_PKG No - Unused - TP_SERVICES_PKG Yes - Unused Never - NETWORK_SERVICES_PKG No - Unused - LAN_ENTERPRISE_SERVICES_PKG No - Unused N9K# 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 62

20 If the TP_SERVICES_PKG license is not listed as installed, fix the issue before proceeding. The Cisco NX-OS Licensing Guide can be found at the following URL: OS_Licensing_Guide.html. Verifying that the correct version of the plug-in and JRE are installed As discussed earlier. the plug-in that runs in the switch s guest shell provides communication between the NSX controllers running OVSDB and the switch running NX-API. The plug-in uses a Java Runtime Environment, or JRE, in the guest shell. The JRE is a set of software tools for the development of Java applications. It combines the Java Virtual Machine (JVM), platform core classes, and supporting libraries. JRE is part of the Java Development Kit (JDK), but it can be downloaded separately. Both the plug-in file and the JRE must be downloaded to the bootflash memory of the switch. To verify that the plug-in and the JRE are in the bootflash memory, enter dir bootflash: as shown in Listing 6 and look for the correct version of the plug-in and JRE. Listing 6 Sample display for a dir bootflash: command N9K# dir bootflash: 4096 Jul 05 21:21: rpmstore/ 4462 Jul 05 21:23: _212325_poap_15804_init.log 2 Jul 05 19:48: diag_bootup 53 Jul 05 20:40: disk_log.txt Jul 06 20:46: jre-8u112-linux-x64.rpm Sep 09 23:57: nxos i4.3.bin Jul 05 21:27: nxos ivm bin Sep 09 22:10: ovsdb-plugin rpm 4096 Jul 05 21:22: scripts/ 4096 Oct 03 20:22: virt_strg_pool_bf_vdc_1/ 4096 Oct 03 20:21: virtual-instance/ 59 Oct 03 20:21: virtual-instance.conf 448 Oct 08 18:18: vlan.dat Usage for bootflash://sup-local bytes used bytes free bytes total N9K# As of the writing of this document, the recommended versions of the plug-in and JRE are those shown in Table 3. Table 3. Recommended versions of the JRE and plug-in Type Recommended version Plug-in Version JRE Version jre-8u112-linux-x64.rpm 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 62

21 The plug-in can be downloaded from a special file access area on Cisco.com that is available to authorized customers. The recommended JRE can be downloaded from the Java website: The latest JREs are available at the following URL as of the writing of this document: Carving the TCAM to enable BFD over VXLAN on first generation Cisco Nexus 9300 platform switches NOTE: This step is NOT required for 9300 EX or newer switches. To enable the BFD over VXLAN feature discussed earlier in this document, explicit TCAM carving is required on the Cisco Nexus 9300 switch. Note that a reboot is mandatory after the TCAM has been recarved. The hardware access-list tcam region value for the redirect-tunnel feature must be set to 256. To accomplish this, you need to borrow some space from a feature that will not miss it. To display the current TCAM carving in the Cisco Nexus 9300 switch, use the method shown in Listing 7. Listing 7 Displaying the TCAM carving in a Cisco Nexus 9300 switch N9K# show hardware access-list tcam region IPV4 PACL [ifacl] size = 0 IPV6 PACL [ipv6-ifacl] size = 0 MAC PACL [mac-ifacl] size = 0 IPV4 Port QoS [qos] size = 0 ---CLIP-- IPV4 RACL [racl] size = 1024 Egress IPV6 VACL [ipv6-vacl] size = 0 Egress MAC VACL [mac-vacl] size = 0 Egress IPV4 RACL [e-racl] size = 256 Egress IPV6 RACL [e-ipv6-racl] size = 0 Egress IPV4 QoS Lite [e-qos-lite] size = 0 ranger+ IPV4 QoS Lite [rp-qos-lite] size = 0 ranger+ IPV4 QoS [rp-qos] size = 256 ranger+ IPV6 QoS [rp-ipv6-qos] size = 256 ranger+ MAC QoS [rp-mac-qos] size = 256 NAT ACL[nat] size = 0 Mpls ACL size = 0 MOD RSVD size = 0 sflow ACL [sflow] size = 0 mcast bidir ACL [mcast_bidir] size = 0 Openflow size = 0 Openflow Lite [openflow-lite] size = 0 Ingress FCoE Counters [fcoe-ingress] size = 0 Egress FCoE Counters [fcoe-egress] size = 0 Redirect-Tunnel [redirect-tunnel] size = 256 SPAN+sFlow ACL [span-sflow] size = 0 N9K# 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 62

22 In the example in Listing 8, the IPV4 RACL [racl] size = 1024 setting is configured to borrow some TCAM space for the redirect-tunnel feature. Listing 8 An example of changing the TCAM carving in a Cisco Nexus 9300 switch N9K(config)# hardware access-list tcam region racl 1024!<<Used as example Warning: Please save config and reload the system for the configuration to take effect N9K(config)# hardware access-list tcam region e-racl 256 Warning: Please save config and reload the system for the configuration to take effect N9K(config)# N9K(config)# copy running-config startup-config [########################################] 100% Copy complete. N9K# reload Be sure that any TCAM region that is you make smaller is one that is not being fully utilized. Note that you must run copy running-config startup-config and reload the switch to make the new TCAM template take effect. If you are not sure whether the new TCAM template has taken effect, rerun the show hardware access-list tcam region command. Look for Redirect-Tunnel [redirect-tunnel] size = 256 and verify that the smaller TCAM region for the feature chosen to be changed is also correct. Verifying the vpc configuration vpc configuration is an optional step. As discussed earlier in this document, you can use the vpc feature to improve high availability for the HW-VTEP. Verify that the vpc feature is configured correctly. All VLANs that will be used for the HW-VTEP feature (as discussed later in this document) must be allowed on the vpc peer link. The Cisco NX-OS Software Release 7.0 configuration guide contains information about configuring vpcs. It is located at the following URL: x/interfaces/configuration/guide/b_cisco_nexus_9000_series_nx- OS_Interfaces_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX- OS_Interfaces_Configuration_Guide_7x_chapter_01000.html The vpc role for the switches must be noted. Listing 9 shows an example. Listing 9 Example of the output for a show vpc role command N9K# show vpc role vpc Role status vpc role : primary Dual Active Detection Status : Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 62

23 vpc system-mac : 00:23:04:ee:be:01 vpc system-priority : vpc local system-mac : 00:f6:63:80:65:bf vpc local role-priority : 100 N9K# The vpc primary switch will contain the master plug-in, and the secondary switch will contain the slave plug-in. You must know which switch is primary so that you can configure the switches and their plug-ins in the correct order for their roles. Configuring the anycast loopback address for vpc pairs If you are using high availability with vpc, you must configure an anycast loopback address on the vpc switch pair. In this implementation, with this configuration, both switches in the vpc pair will advertise the same secondary IP address for the loopback interface they are using for sourcing and receiving the VXLAN traffic. This configuration will allow them to be seen as a single device from the perspective of the NSX control plane. This loopback address must be reachable by all NSX components. Listings 10 and 11 show examples of the loopback configuration for two switches that make up a vpc pair. Listing 10 An example of anycast loopback configuration on the vpc primary switch N9K-VPC-1# show running-config interface loopback 0!Command: show running-config interface loopback0!time: Tue Dec 20 19:18: version 7.0(3)I4(3) interface loopback0 ip address /32 ip address /32 secondary ip router ospf lab area N9K-VPC-1# Listing 11 An example of anycast loopback configuration on the vpc secondary switch N9K-VPC-2# show running-config interface loopback 0!Command: show running-config interface loopback0!time: Tue Dec 20 19:18: version 7.0(3)I4(3) interface loopback0 ip address /32 ip address /32 secondary 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 62

24 ip router ospf lab area N9K-VPC-2# Note that the primary IP addresses are different, but the secondary addresses are the same. Both of these switches will advertise their primary addresses and anycast secondary addresses in the Open Shortest Path First (OSPF) protocol. If a separate Virtual Routing and Forwarding (VRF) instance will be used for this connectivity, you also need to configure the loopback and associated routing accordingly. Configuring a username and password for use by the NXDB plug-in if desired Network administrator users can assign roles that limit access to NXDB operations on the switch. NX-OS supports two NXDB roles for users who are configured for remote use through TACACS+: nxdb-admin: Allowed to run get and set JSON-RPC NX-API calls from the external controller nxdb-operator: Allowed to run only get JSON-RPC NX-API calls from the external controller When NXDB is enabled, the nxdb-admin role is automatically assigned to the permanent user (admin). Network administrator users can assign the nxdb-admin or nxdb-operator role to other users as necessary. Note that Representational State Transfer (REST) requests using credentials received from TACACS+ will work as expected. If you want, you can configure a separate local username and password on the Cisco Nexus 9300 switches to be used by the NXDB plug-in. Use the following command: username user-id [password [0 5] password role {nxdb-admin nxdb-operator} This command configures a user account with the specified NXDB role. The user-id argument is a case-sensitive, alphanumeric character string with a maximum length of 28 characters. Valid characters are uppercase letters A through Z, lowercase letters a through z, numbers 0 through 9, hyphen (-), period (.), underscore (_), plus sign (+), and equal sign (=). The at symbol (@) is supported in remote usernames but not in local usernames. The default password is undefined. The 0 option indicates that the password is clear text, and the 5 option indicates that the password is encrypted. The default is 0 (clear text). Listing 12 shows an example of how to configure and verify a user account to be used by the NXDB process. Listing 12 Defining a user to be used by the NXDB process N9K# conf t Enter configuration commands, one per line. End with CNTL/Z. N9K(config)#username nxdb-user password 0 NX-DBpassword! [[OK?]] role nxdb-admin N9K(config)# N9K(config)# copy running-config startup-config [########################################] 100% Copy complete. N9K(config)# exit N9K# show user-account user:admin 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 24 of 62

25 this user account has no expiry date roles:network-admin user:nxdb-user this user account has no expiry date roles:nxdb-admin N9K# Note that this step is optional. The administrator user can also be used for the NXDB configuration. Verifying that the VMware NSX version and configuration are supported Verify that the correct version of VMware NSX is installed. As of the writing of this document, the version of code certified by VMware and Cisco to run this integration is the one listed in Table 4. Table 4. Supported VMware NSX release Supported VMware NSX release Release and later on the same main release The NSX configuration should follow the VMware configuration guidelines. Check the correct guidelines at the VMware website. Verifying the required network reachability You need to set up and verify the routing for the environment before performing the rest of the configuration. The NSX hypervisors, NSX manager, and NSX controller should be reachable from the switch s loopback address (the one that is going to be used for HW-VTEP configuration). Note that this address can be the anycast loopback address if you configure vpc. To verify the address, you can ping the IP addresses of the NSX manager, the NSX controllers, and the ESXi hosts VTEP VMkernel while sourcing them from the loopback address that will be used for VXLAN configuration. If you are using a separate VRF instance for this connectivity, the loopback and associated routing must be configured accordingly, and the ping should use that VRF instance. Listing 13 shows an example. Listing 13 Example of using ping to verify connectivity N9K# ping source-interface loop0 PING ( ): 56 data bytes 64 bytes from : icmp_seq=0 ttl=62 time=0.728 ms 64 bytes from : icmp_seq=1 ttl=62 time=0.479 ms 64 bytes from : icmp_seq=2 ttl=62 time=0.422 ms 64 bytes from : icmp_seq=3 ttl=62 time=0.428 ms 64 bytes from : icmp_seq=4 ttl=62 time=0.418 ms ping statistics packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.418/0.495/0.728 ms N9K# Ideally, if possible, the ping should be performed from the NSX components themselves to the loopback addresses of the Cisco Nexus 9300 switches Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 25 of 62

26 Reserving VLANs and other switch resources As discussed earlier, VLANs must be reserved for VXLAN-to-VLAN associations. Each NSX logical switch extension uses one VLAN on an HW-VTEP (or vpc pair). If 20 VXLAN-backed port groups are to be extended using the OVSDB HW-VTEP integration between NSX and the Cisco Nexus 9300 switch, then 20 VLANs must be reserved on that switch (or vpc pair of switches) for that association. A single VLAN per switch (or vpc pair of switches) must also be reserved for BDF over VXLAN. Note: In a vpc implementation, all the reserved VLANs (including the one reserved for BFD over VXLAN) must be allowed on the vpc peer link. Note the following guidelines and limitations for VLANs assigned to the external controller: The assigned VLANs must not already exist on the system. The assigned VLANs can be configured only as dedicated resources. Therefore, only the external controller can push down VLAN-related configurations for them. The VLANs are either completely owned by the external controller or completely owned by the switch. If the VLAN is owned by the external controller, the switch cannot configure the port membership for that VLAN. If the VLAN is owned by the switch, any configuration that the controller sends down will be blocked. Note the following guidelines and limitations for interfaces assigned to the external controller: The Ethernet and port-channel interfaces that are exposed to the external controller must be valid interfaces. vpcs are supported. vpc domains should be configured with the delay peer-link timer (using the delay peer-link seconds command). The recommended value is 600 seconds but must be adjusted based on the scale. The Ethernet and port-channel interfaces can be configured only as shared resources. Therefore, the configuration for these resources can be performed from both the switch Command-Line Interface (CLI) and the external controller. Trunk-mode interfaces are the only interfaces that can be shared. Access ports are not supported. If an interface is already assigned, it cannot be changed to an access-mode interface. You cannot configure the native VLAN (the untagged VLAN for trunk-mode interfaces) on the assigned interface. Therefore, an assigned interface can only have VLAN 1 as its default VLAN. Configuring OVSDB integration with VMware NSX After the tasks in the checklists discussed in the previous sections are complete, you are ready to configure the Cisco Nexus 9000 Series Switch HW-VTEP integration with VMware NSX using OVSDB. This configuration will make the Cisco Nexus switches work as HW-VTEPs with NSX. Configuring the required features on the switch You must configure the following features for the integration to work: Note: The commands shown here are the same for both standalone and vpc configurations. They must match exactly for a vpc pair Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 26 of 62

27 feature nv overlay: Enables the VXLAN feature feature vn-segment-vlan-based: Configures the global mode for all VXLAN bridge domains feature bfd: Enables the BFD feature feature nxapi: Enables the plug-in to use NX-API to communicate with the switch feature nxdb: Enables NXDB on the switch, which allows the switch to be configured through JSON-RPC NX-API calls In addition to the preceding features, other features, such as vpc, Lightweight Access Control Protocol (LACP), and routing protocols, may be required, but those should have been configured when you completed the checklist steps. The NX-API feature requires you to specify a VRF instance and ports. Choose the management option if the connection to the controller is through the management VRF instance. The default option specifies the default VRF instance. The commands for specifying the VRF instance and ports to be used for the NXAPI feature are listed here: nxapi use-vrf {default management} nxapi http port 80 nxapi https port 443 Listing 14 shows an example of how to configure the features and the NX-API to use VRF management. Any VRF instance can be specified for this procedure and will be used by the plug-in. Listing 14 Configuring the features for OVSDB integration N9k# conf t Enter configuration commands, one per line. End with CNTL/Z. N9k(config)# feature nv overlay N9k(config)# feature vn-segment-vlan-based N9k(config)# feature bfd Please disable the ICMP / ICMPv6 redirects on all IPv4 and IPv6 interfaces running BFD sessions using the command below 'no ip redirects ' 'no ipv6 redirects ' N9k(config)# feature nxapi N9k(config)# feature nxdb N9k(config)# nxapi http port 80 N9k(config)# nxapi https port 443 N9k(config)# nxapi use-vrf management Warning: Management ACLs configured will not be effective for HTTP services. Please use iptables to restrict access. N9k(config)# Configuring VXLAN on the switch Note: The commands discussed here are the same for both standalone and vpc configurations. They must match exactly for a vpc pair Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 27 of 62

28 For the switch to accept Layer 2 VXLAN configurations from an external controller, you must configure some VXLAN settings. Here, to do this, configure a Network Virtualization Edge (NVE) interface using the following command: switch(config)#interface nve 1 This command creates a VXLAN overlay interface that terminates VXLAN tunnels. The NVE interface serves as a single logical interface for the VXLAN network ports. After the interface is created, you must enable it by using the following command: switch(config-if-nve)# no shut Note that the switch supports only one NVE interface. The NVE interface then must be tied to the loopback interface that will be its source. Use the following command: switch(config-if-nve)# source-interface loopback 0 (or the correct loopback interface) This /32 IP address must be known by the transient devices in the transport network and the remote VTEPs. This requirement is met by advertising the address through a dynamic routing protocol in the transport network. The following command automatically remaps traffic to a different replication service node when a service node is added or goes down: switch(config-if-nve)# auto-remap-replication-servers You then specify that the external controller will distribute the host reachability information (such as the MAC addresses and IP addresses of the host) in the network. Use the following command: switch(config-if-nve)# host-reachability protocol controller 1 The next step is to enable the switch to receive configurations from the controller. Use the following command: switch(config-if-nve)# config-source controller Next, add a hold-down timer for the interface to keep it down until the switch is up and the routing protocols have had a chance to converge: switch(config-if-nve)# source-interface hold-down-time 30 A sample configuration for the NVE interface is shown in Listing 15. Listing 15 Configuring the NVE interface N9K# conf t Enter configuration commands, one per line. End with CNTL/Z. N9K (config)# interface nve1 N9K (config-if-nve)# N9K (config-if-nve)# N9K (config-if-nve)# no shutdown source-interface loopback0 auto-remap-replication-servers N9K (config-if-nve)# host-reachability protocol controller 1 N9K (config-if-nve)# config-source controller N9K (config-if-nve)# source-interface hold-down-time 30 N9K(config-if-nve)#end N9K# 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 28 of 62

29 Configuring BFD over VXLAN Note: The commands discussed here are the same for both standalone and vpc configurations. They must match exactly for a vpc pair. To configure the BFD-over-VXLAN feature, you define a BFD control VLAN and map it to control VNI 0. The BFD control frames are encapsulated with VNI 0. The following commands are used: switch(config)# vlan 3000!<<Or any VLAN that was reserved for BFD over VXLAN) switch(config-vlan)# vn-segment 0 Next, you must define a BFD control SVI with IP forwarding. This command is required to forward the BFD packet to the supervisor after it is received. The commands are as follows: switch(config)# interface Vlan3000!<<Or any VLAN that is reserved for BFD over VXLAN) switch(config-if)# no shutdown switch(config-if)# ip forward A sample configuration for BFD over VXLAN is shown in Listing 16. The actual BFD configuration will be pushed down by the NSX controller after the integration is configured. Listing 16 Configuring the VLAN for BFD over VXLAN N9K# conf t Enter configuration commands, one per line. End with CNTL/Z. N9K(config)# vlan 3000!<<Use the correct VLAN N9K(config-vlan)# vn-segment 0 N9K(config-vlan)# exit Warning: Enable double-wide arp-ether tcam carving if igmp snooping is enabled. Ignore if tcam carving is already configured. N9K(config)# interface vlan 3000!<<Use the correct VLAN N9K(config-if)# no shut N9K(config-if)# ip forward N9K(config-if)# end N9K# Assigning VLANs and interfaces to the controllers Note: The commands discussed here are the same for both standalone and vpc configurations. They must match exactly for the vpc pair for VLANs and any vpcs that are shared. They do not have to match if orphan ports are being shared on vpc member switches. For the switch to accept configurations from an external controller, you must identify the VLANs and interfaces whose configuration can be performed from the external controller. You will assign the reserved VLANs and interfaces to be configured by the NSX controller using OVSDB. First you must identify the type of controller that will be connecting to the Cisco Nexus 9300 switch by entering the following command: switch(config)# controller type l2-vxlan identifier 1 switch(config-ctrlr-type)# 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 29 of 62

30 After the controller type is configured and created, you can start assigning interfaces to it. These are the interfaces that will be controlled and configured by the external controller. If this is a standalone implementation, then you assign the trunked interfaces that will be connected to the VXLAN backed port groups. If this is a vpc implementation and vpc will be assigned, then the vpc peer link must be assigned to the controller first, and then the actual trunked vpcs should be assigned. The following example shows how to assign physical (non-vpc) trunked interfaces to the controller: switch(config-ctrlr-type)# assign interface ethernet 1/4-7, ethernet 1/17 shared Note that interface ranges can be used, and that the shared keyword at the end of the line is required. You can also use several lines to assign interfaces: switch(config-ctrlr-type)# assign interface ethernet 1/4 shared switch(config-ctrlr-type)# assign interface ethernet 1/5 shared switch(config-ctrlr-type)# assign interface ethernet 1/6 shared switch(config-ctrlr-type)# assign interface ethernet 1/7 shared switch(config-ctrlr-type)# assign interface ethernet 1/17 shared You can also use the no version of the command to remove an interfaces assignment. Here is an example: switch(config-ctrlr-type)# no assign interface Ethernet 1/4-5 If you are assigning vpcs, you must assign the vpc peer link first. The vpc configuration must match for both vpc member switches. As mentioned earlier, orphan ports can be assigned as well. Here is an example: switch(config-ctrlr-type)# assign interface port-channel 10 shared!<<vpc peer link switch(config-ctrlr-type)# assign interface port-channel 100 shared switch(config-ctrlr-type)# assign interface port-channel shared switch(config-ctrlr-type)# assign interface ethernet 1/20 shared!<orphan port After the interfaces have been assigned to the controllers, the next step is to assign the dedicated VLANs that have been reserved for this configuration. For a vpc implementation, these VLANs must match in both vpc member switches. The command to assign VLANs to the controller is as follows: switch(config-ctrlr-type)# assign vlan , 1000 dedicated Note that the dedicated keyword at the end of the line is required. The no version of this command can be used to remove VLANs from the controllers: switch(config-ctrlr-type)# no assign vlan 510 Note that after running the no assign command and then assigning the VLANs in the controller context, the OVSDB plug-in will need to be restarted using the command guestshell run sudo ovsdb-plugin service restart. The processes for installing and working with the plug-in are discussed in the next section of this document. If you want, you can add a description to the controller by using the following command: switch(config-ctrlr-type)# controller description <text> A sample controller configuration for a vpc member switch is shown in Listing 17. Listing 17 Sample controller configuration for a vpc member switch 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 30 of 62

31 N9K-vPC-1# N9K-vPC-1# conf t Enter configuration commands, one per line. End with CNTL/Z. N9K-vPC-1(config)# controller type l2-vxlan identifier 1 N9K-vPC-1(config-ctrlr-type)# controller description Controller-for-OVSDB N9K-vPC-1(config-ctrlr-type)# assign interface port-channel10, port-channel100 shared N9K-vPC-1(config-ctrlr-type)# assign interface Ethernet1/10-11 shared N9K-vPC-1(config-ctrlr-type)# assign vlan dedicated N9K-vPC-1(config-ctrlr-type)# end N9K-vPC-1# Configuring the guest shell for the OVSDB plug-in Note: The commands discussed here are the same for both standalone and vpc configurations. They must match exactly for a vpc pair. In addition to the NX-OS CLI and Bash access in the underlying Linux environment, the Cisco Nexus 9000 Series Switches support access to a decoupled run space within a Linux Container (LXC) called the guest shell. The OVSDB plug-in runs inside the guest shell of the Cisco Nexus 9000 Series Switch. More information about the Cisco Nexus 9000 Guest Shell can be found at the following URL: x/programmability/guide/b_cisco_nexus_9000_series_nx-os_programmability_guide_7x/guest_shell.html To run the plug-in, you must first resize the guest shell to the correct values. Use the following commands: switch# guestshell destroy switch# guestshell resize cpu 18 switch# guestshell resize mem 2500 switch# guestshell resize rootfs 1200 switch# guestshell enable Warning: The guestshell destroy command removes all existing configurations in the guest shell. After you enter the guestshell enable command, it may take a few minutes for the new guest shell to come up. An example of this configuration is shown in Listing 18. Listing 18 Configuring the guest shell N9K# guestshell destroy You are about to destroy the guest shell and all of its contents. Be sure to save your work. Are you sure you want to continue? (y/n) [n] y N9K# guestshell resize cpu 18 Note: Guest shell is currently installing, uninstalling or upgrading; please retry request!<<please wait N9K# guestshell resize cpu 18 Note: System CPU share will be resized on Guest shell enable N9K# guestshell resize mem Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 31 of 62

32 Note: System memory will be resized on Guest shell enable N9K# guestshell resize rootfs 1200 Note: Root filesystem will be resized on Guest shell enable N9K# guestshell enable N9K# N9K# show guestshell Virtual service guestshell+ detail State : Installing!<<Still not fully enabled Package information Name : guestshell.ova Path : /isanboot/bin/guestshell.ova Resource reservation Disk Memory CPU : 0 MB : 0 MB : 0% system CPU N9K# show guestshell Virtual service guestshell+ detail State : Activated!<<Guest shell is ready Package information Name : guestshell.ova Path : /isanboot/bin/guestshell.ova Application Name : GuestShell Installed version : 2.2(0.0) Description : Cisco Systems Guest Shell Resource reservation Disk Memory CPU N9K# : 1200 MB : 2500 MB : 18% system CPU Installing the OVSDB plug-in Note: The commands discussed here are the same for both standalone and vpc configurations. They must match exactly for a vpc pair Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 32 of 62

33 After the guest shell is active, it is time to install the plug-in and JRE that were copied to the bootflash memory earlier. To do that, you must first access the guest shell prompt by using the following command: switch# run guestshell ~]$ From the guest shell, install the correct JRE file that should be copied to the bootflash memory. Use the following command: ~]$ sudo rpm -i /bootflash/jre-8u112-linux-x64.rpm!<<use correct JRE After the JRE is installed, it is time to install the correct OVSDB plug-in that has been copied to the bootflash memory. Use the following command: ~]$ sudo rpm -i /bootflash/ovsdb-plugin rpm!<<note plug-in A sample configuration is shown in Listing 19. Listing 19 Installing the JRE and plug-in in the guest shell N9K# run guestshell [guestshell@guestshell ~]$ sudo rpm -i /bootflash/jre-8u112-linux-x64.rpm Unpacking JAR files... rt.jar... jsse.jar... charsets.jar... localedata.jar... jfxrt.jar... [guestshell@guestshell ~]$ sudo rpm -i /bootflash/ovsdb-plugin rpm + APPDIR=/usr/local/ovsdb + LOGDIR=/usr/local/ovsdb/log + START_SCRIPT_NAME=ovsdb.service + TIMER_SCRIPT_NAME=ovsdb.timer + START_SCRIPT_SRC=/usr/local/ovsdb/systemd/ovsdb.service + START_SCRIPT_DST=/etc/systemd/system/ovsdb.service + TIMER_SCRIPT_SRC=/usr/local/ovsdb/systemd/ovsdb.timer + TIMER_SCRIPT_DST=/etc/systemd/system/ovsdb.timer + SERVICE_NAME=ovsdb.timer + /bin/id -u guestshell USER=guestshell + mkdir -p /usr/local/ovsdb /usr/local/ovsdb/log + sed -i 's~^logging_dir.*~logging_dir = abspath('\''/usr/local/ovsdb/log/'\'')~' /usr/local/ovsdb/lib/python/ovsdb_plugin/common.py + sed -i 's/defaults\s\+requiretty/defaults \!requiretty/' /etc/sudoers + cp /usr/local/ovsdb/systemd/ovsdb.service /etc/systemd/system/ovsdb.service + cp /usr/local/ovsdb/systemd/ovsdb.timer /etc/systemd/system/ovsdb.timer + sed -i s/guestshell/guestshell/g /etc/systemd/system/ovsdb.service 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 33 of 62

34 + chmod 664 /etc/systemd/system/ovsdb.service + chmod 664 /etc/systemd/system/ovsdb.timer + chown -R guestshell:guestshell /usr/local/ovsdb /usr/local/ovsdb/log + chmod -R 777 /usr/local/ovsdb + chmod -R 777 /usr/local/ovsdb/bin/data-recv /usr/local/ovsdb/bin/data-send /usr/local/ovsdb/bin/nc /usr/local/ovsdb/bin/ovsdb-plugin + chmod -R 777 /usr/local/ovsdb/log + chmod -R u+rwx,g-rwx,o-rwx /usr/local/ovsdb/config/ssl + test -h /usr/bin/ovsdb-plugin + ln -s /usr/local/ovsdb/bin/ovsdb-plugin /usr/bin/ovsdb-plugin + systemctl enable ovsdb.timer ln -s '/etc/systemd/system/ovsdb.timer' '/etc/systemd/system/multiuser.target.wants/ovsdb.timer' [guestshell@guestshell ~]$ exit logout N9K# Configuring the OVSDB plug-in for a standalone switch Note: The following steps are for configuring the OVSDB plug-in for a standalone (non-vpc) switch. The steps for configuring the OVSDB plug-in for a vpc pair of switches are in the next section. After the plug-in is installed in the guest shell, you need to configure it. Make sure that the NSX controller IP address is reachable before you configure the OVSDB plug-in. Then configure the plug-in using the following command: switch# guestshell run sudo ovsdb-plugin config set --run-in-switch --vrf vrf-name --log-level level --log-type type controller-ip-address switch-mgmt-ip-address switch-username switch-password switch-name -- switch-description switch-description You must complete all the highlighted fields (in yellow) [[PLS ADJUST AS NECESSARY IF THE TEXT USES SOMETHING OTHER THAN YELLOW HIGHLIGHTING]] with the information for your environment. Take a closer look at this long command: guestshell run ovsdb-plugin: This entry tells the switch that you are sending the command to the plug-in running in the guest shell. config set: This entry tells the plug-in that you are setting its configuration. --run-in-switch: This entry configures the plug-in to run in the switch. --vrf vrf-name: This entry is used only when --run-in-switch is set. It configures the plug-in to use the given VRF name when communicating with the controller. The default VRF name is management. --log-level level: This entry sets the logging level. It defaults to info. The recommended setting is debug. --log-type type: This entry specifies the type of logging to use. When set to file, the path is always PLUGIN_ROOT/log/ovsdb-plugin.log. The default type is file, and that is the recommended setting. You can also set this command to UDP and define a server Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 34 of 62

35 controller-ip-address: This entry specifies is the IP address of one of the NSX controllers (not the NSX manager). The controller address is in IP:PORT format. The port defaults to 6632 if this command is not included. The port used by NSX is 6640 and must be specified: for example, :6640. The order for this specification is important. The controller IP address must be entered before the switch IP address. switch-mgmt-ip-address: This entry specifies is the switch s address in IP:PORT format. The port defaults to 443 if this command is not included. You should use the loopback address here. switch-username switch-password: This entry specifies the username and password that will be used by the NXDB process to push NX-API changes to the switch. switch-name: This entry specifies a switch name to be used. This attribute is mandatory. This attribute must come after the username and password. --switch-description switch-description: This entry specifies a description for this switch. This attribute is optional. You can use any description, including the switch name. Listing 20 shows an example of the entire command. Listing 20 Configuring the plug-in N9K# guestshell run sudo ovsdb-plugin config set --run-in-switch --vrf default -- log-level debug --log-type file : nxdb-user NX- DBpassword! N9K --switch-description N9K Configuration saved N9K# guestshell run sudo ovsdb-plugin config show Controllers : #1 addr: :6640 VPC : No In switch : Yes VRF : default Switches : #1 addr: :443 type: STANDALONE user: nxdb-user name: N9K description: N9K Log type : file Log level : debug Log server : - TTY log path : - Max JSON peers: 6 Min heap size : 2048 MB Max heap size : 2048 MB Schema : shol-nxos-3# As shown in the example, you can use the guestshell run sudo ovsdb-plugin config show command to check the OVSDB plug-in s saved configuration Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 35 of 62

36 After the OVSDB plug-in is configured, the next step is to create a certificate that will be used for secure communication between the OVSDB plug-in and the NSX controllers. The certificate is created by entering the following command: switch# guestshell run sudo ovsdb-plugin cert bootstrap Use the command guestshell run sudo ovsdb-plugin cert show. An example of this command is shown in Listing 21. Listing 21 Creating a certificate for the plug-in N9K# guestshell run sudo ovsdb-plugin cert bootstrap OVSDB plugin keypair saved to '/usr/local/ovsdb/config/ssl/ovsdb_plugin.p12' Controller CA cert saved to '/usr/local/ovsdb/config/ssl/controller_ca_cert1.der' Combined controller CA cert saved to '/usr/local/ovsdb/config/ssl/controller_ca_cert.der' Switch #1 CA cert saved to '/usr/local/ovsdb/config/ssl/switch_ca_cert_1.der' N9K# guestshell run sudo ovsdb-plugin cert show -----BEGIN CERTIFICATE----- MIICkjCCAfugAwIBAgIJAM31b/Oey9eKMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV BAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9u MRIwEAYDVQQKDAlBbmRyb21lZGExFTATBgNVBAMMDG92c2RiX3BsdWdpbjAeFw0x NjEyMjYyMDU1MjdaFw0xOTEyMjYyMDU1MjdaMGIxCzAJBgNVBAYTAlVTMRMwEQYD VQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9uMRIwEAYDVQQKDAlB bmryb21lzgexftatbgnvbammdg92c2rix3bsdwdpbjcbnzanbgkqhkig9w0baqef AAOBjQAwgYkCgYEArMr611TJIxrIwxwoG+XpoeuhW9r1TNmsog1UNIDQkPXybvLa 8GTBgSqqI7kl4K326fxstPqnJAJtA4R4xCi4OpQDoUmDp680KejjW63/EEvYokx0 XuaY48z+VO0L4zBAIOpI4o0mbGNv5aIaq/qeobbZvo3KSRKkEcGL+IeDEgUCAwEA AaNQME4wHQYDVR0OBBYEFBqi2xO54kvGVATk3IjvVnoNNAU7MB8GA1UdIwQYMBaA FBqi2xO54kvGVATk3IjvVnoNNAU7MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL BQADgYEAGkzO/7ubt8LmgKbOUu3nOK5c6fdm+YaktyZ+KemnOPw043FHu/RQ/PF3 aiphh4pajxouln1w4k9hdkgkwxvgxrqdvyxsotrdsrhhvvfxrhp+jsgwgi2x3az1 +hotol8wnoc6zytc6dftwel0jeze+53nb5/zmvgmz77fi4iewo8= -----END CERTIFICATE----- N9K# Configuring the OVSDB plug-in for a pair of vpc switches You configure the OVSDB plug-in for use by a pair of vpc switches similar to the way that you configure it for a standalone switch. After the plug-in is installed in the guest shells of both switches, it is time to configure it. Make sure that the NSX controller IP address is reachable from both switches before you configure the OVSDB plug-in. You also need to know which switch has the vpc primary role. The plug-in will be in master mode on the vpc primary switch and in slave mode on the vpc secondary switch Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 36 of 62

37 Now you are ready to configure the plug-in. Enter the following command on the vpc primary switch: switch# guestshell run sudo ovsdb-plugin config set --run-in-switch --vrf vrf-name --log-level level --log-type type controller-ip-address vpc-primary-switch-mgmt-ip-address, vpc-secondary-switch-mgmt-ip-address vpc-primary-switch-username vpc-secondary-switch-username vpc-primary-switch-password vpc- Secondary-switch-password switch-name --switch-description switch-description Enter the following command on the vpc secondary switch: switch# guestshell run sudo ovsdb-plugin config set --run-in-switch --vrf vrf-name --log-level level --log-type type controller-ip-address vpc-secondary-switch-mgmt-ip-address, vpc-primary-switch-mgmt-ip-address vpc-secondary-switch-username vpc-primary-switch-username vpc-secondary-switch-password vpc- Primary-switch-password switch-name --switch-description switch-description These commands show that, unlike with the standalone configuration, for the vpc configuration you enter two switch management IP addresses, two switch usernames, and two switch passwords. You must enter these in a specific order. On the primary vpc switch, you enter the primary management IP address, username, and password first. On the secondary vpc switch, you enter the secondary management IP address, username, and password first. Take a closer look at this long command: guestshell run ovsdb-plugin: This entry tells the switch that you are sending the command to the plug-in running in the guest shell. config set: This entry tells the plug-in that you are setting its configuration. --run-in-switch: This entry configures the plug-in to run in the switch. --vrf vrf-name: This entry is used only when --run-in-switch is set. It configures the plug-in to use the given VRF name when communicating with the controller. The default VRF name is management. --log-level level: This entry sets the logging level. It defaults to info. The recommended setting is debug. --log-type type: This entry specifies the type of logging to use. When the type is set to file, the path is always PLUGIN_ROOT/log/ovsdb-plugin.log. The default setting is file, and that is the recommended setting. You can also set the type to UDP and define a server. controller-ip-address: This entry is the IP address of one of the NSX controllers (not the NSX manager). The controller address is in IP:PORT format. The port defaults to 6632 if this command is not included. The port used by NSX is 6640 and must be specified: for example, :6640. The order for this specification is important. The controller IP address must be entered before the switch IP address. vpc-primary-switch-mgmt-ip-address,vpc-secondary-switch-mgmt-ip-address: These entries are the two switch addresses in IP:PORT format. The port defaults to 443 if these entries are not included. The real loopback addresses (not the anycast loopback address) must be used here. For the vpc primary switch, the order of IP addresses is vpc-primary-loopback, vpc-secondary-loopback. For the vpc secondary switch, the order of the IP addresses is reversed. For example, you would use , vpc-primary-switch-username,vpc-secondary-switch-username vpc-primary-switchpassword,vpc-secondary-switch-password: This entry specifies the username and password that will be used by the NXDB process to push NX-API changes to the switch. Usually these are the same and are just repeated. An example of this configuration is nxdb-user, nxdb-user NX-DBpassword!,NX- DBpassword! 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 37 of 62

38 switch-name: This entry specifies the switch name for use by the plug-in, and this attribute is mandatory. It must come after the username and password. For a vpc setup, the switch name must be identical on both the vpc primary and secondary switches. --switch-description switch-description: This entry specifies a description for this switch, and it is an optional attribute. You can use any description, including the switch name. Here is an example of the entire command on the vpc primary switch: guestshell run sudo ovsdb-plugin config set --run-in-switch --vrf default --log-level debug --log-type file : , nxdb-user,nxdb-user NXDBpassword!,NXDBpassword! N9K-VPC-1 -- switch-description N9K-VPC1-N9K-VPC2 Here is an example of the entire command on the vpc secondary switch: guestshell run sudo ovsdb-plugin config set --run-in-switch --vrf default --log-level debug --log-type file : , nxdb-user,nxdb-user NXDBpassword!,NXDBpassword! [[SOMETHING SEEMS AMISS HERE. PLS VERIFY.]] N9K-VPC-2 --switch-description N9K-VPC1-N9K-VPC2 Note: The anycast loopback IP address is not used for this part of the configuration. Listings 22 and 23 show the full configurations. Listing 22 Configuring the plug-in on the vpc primary switch N9K-VPC-1# sh vpc role vpc Role status vpc role Dual Active Detection Status : 0 vpc system-mac : primary vpc system-priority : vpc local system-mac vpc local role-priority : 100 N9K-VPC-1# : 00:23:04:ee:be:01 : 00:f6:63:80:65:bf N9K-VPC-1# guestshell run sudo ovsdb-plugin config set --run-in-switch --vrf default --log-level debug --log-type file : , nxdb-user,nxdb-user NXDBpassword!,NXDBpassword! N9K-VPC-1 --switch-description N9K-VPC1-N9K-VPC2 Configuration saved N9K-VPC-1# N9K-VPC-1# guestshell run sudo ovsdb-plugin config show Controllers : #1 addr: :6640 VPC : Yes In switch : Yes VRF : default Switches : #1 addr: :443 type: LOCAL 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 38 of 62

39 user: nxdb-user name: N9K-VPC-1 description: N9K-VPC1-N9K-VPC2 #2 addr: :443 type: REMOTE user: nxdb-user name: N9K-VPC-1 description: N9K-VPC1-N9K-VPC2 Log type : file Log level : debug Log server : - TTY log path : - Max JSON peers: 6 Min heap size : 2048 MB Max heap size : 2048 MB Schema : N9K-VPC-1# Listing 23 Configuring the plug-in on the vpc secondary switch N9K-VPC-2# show vpc role vpc Role status vpc role Dual Active Detection Status : 0 vpc system-mac : secondary vpc system-priority : vpc local system-mac vpc local role-priority : N9K-VPC-2# : 00:23:04:ee:be:01 : 00:62:ec:b3:96:93 N9K-VPC-2# guestshell run sudo ovsdb-plugin config set --run-in-switch --vrf default --log-level debug --log-type file : , nxdb-user,nxdb-user NXDBpassword!,NXDBpassword! N9K-VPC-2 --switch-description N9K-VPC1-N9K-VPC2 Configuration saved N9K-VPC-2# N9K-VPC-2# guestshell run sudo ovsdb-plugin config show Controllers : #1 addr: :6640 VPC : Yes In switch : Yes VRF : default Switches : #1 addr: : Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 39 of 62

40 type: LOCAL user: nxdb-user name: N9K-VPC-2 description: N9K-VPC1-N9K-VPC2 #2 addr: :443 type: REMOTE user: nxdb-user name: N9K-VPC-2 description: N9K-VPC1-N9K-VPC2 Log type : file Log level : debug Log server : - TTY log path : - Max JSON peers: 6 Min heap size : 2048 MB Max heap size : 2048 MB Schema : N9K-VPC-2# After the OVSDB plug-ins are configured, the next step is to create a certificate that will be used for secure communication between the OVSDB plug-ins and the NSX controllers. Because the two vpc switches will appear as a single control plane to the NSX controllers (running in a master-slave relationship locally), you must synchronize the certificate between the two devices. Then, if a failover event occurs, the secondary vpc switch s plug-in can take over seamlessly. You create the certificate by entering the following command on the vpc secondary switch: N9K-VPC-2#guestshell run sudo ovsdb-plugin cert bootstrap --receive This command will create a temporary certificate that will be displayed. Copy this certificate. On the vpc primary switch, enter this command: N9K-VPC-1#guestshell run sudo ovsdb-plugin cert bootstrap --send IP-of-vPC-Secondary --vrf vrf-name Enter the IP-of-vPC-Secondary address as well as the correct VRF instance to reach it. When prompted by the vpc primary switch, paste the temporary certificate that was provided earlier (after you entered guestshell run sudo ovsdb-plugin cert bootstrap receive) on the vpc secondary switch. After you complete the preceding steps, you can display the certificates by entering the guestshell run sudo ovsdb-plugin cert show command. The certificates in both vpc switches should match. Sample configuration steps are shown in Listings 24, 25, and Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 40 of 62

41 Listing 24 Creating a certificate for the plug-in on the vpc secondary switch N9K-VPC-2# guestshell run sudo ovsdb-plugin cert bootstrap --receive OVSDB plugin keypair saved to '/usr/local/ovsdb/config/ssl/ovsdb_plugin.p12' Waiting to receive keypair on port 6640 via SSL/TLS... Provide sender with my CA cert: -----BEGIN CERTIFICATE-----!<<This is the temporary cert you should copy MIICkjCCAfugAwIBAgIJAOV/d9zN9tvLMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV BAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9u CLIP mrzr8jyjmhml1vgfej34bjtecawzkdksqxptb6dvdssaen9lksl0mzyzxeom+4xb nt7wl1lzumolmftzfzqeexzbmctiu/hsnlgqtv5ns775sbyisda= -----END CERTIFICATE----- Listing 25 Creating a certificate for the plug-in on the vpc primary switch N9K-VPC-1# guestshell run sudo ovsdb-plugin cert bootstrap --send vrf default OVSDB plugin keypair saved to '/usr/local/ovsdb/config/ssl/ovsdb_plugin.p12' Please paste the receiver's CA cert and press ENTER: -----BEGIN CERTIFICATE-----!<<This is where the temporary cert should be pasted MIICkjCCAfugAwIBAgIJAOV/d9zN9tvLMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV BAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9u MRIwEAYDVQQKDAlBbmRyb21lZGExFTATBgNVBAMMDG92c2RiX3BsdWdpbjAeFw0x NjEyMjYyMjI3NTZaFw0xOTEyMjYyMjI3NTZaMGIxCzAJBgNVBAYTAlVTMRMwEQYD VQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9uMRIwEAYDVQQKDAlB bmryb21lzgexftatbgnvbammdg92c2rix3bsdwdpbjcbnzanbgkqhkig9w0baqef AAOBjQAwgYkCgYEAw5f/VtkvtFySQrjBLAii/aZhoVFmoVBknRMGo6BZlw4fJK// NcgWKXSaxn6FlrgqjLmWIOQtgjDfKEfvU2CwVE+3Z0FLj8TGFGJjIYYToEwGcyWl ZbnpfQy1Jy+CnnNEcfp5tFkcxIVhf/66gtjispBVlAbu/F+HyeNMQB3JK+UCAwEA AaNQME4wHQYDVR0OBBYEFJl67js6GlgV/6p8I1nov88K/YL5MB8GA1UdIwQYMBaA FJl67js6GlgV/6p8I1nov88K/YL5MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL BQADgYEAC+qLxAt3BUB5J516aPkU4OWZE4Zan1fpmyWuanca+Tkb+zKSh7bQwGbr mrzr8jyjmhml1vgfej34bjtecawzkdksqxptb6dvdssaen9lksl0mzyzxeom+4xb nt7wl1lzumolmftzfzqeexzbmctiu/hsnlgqtv5ns775sbyisda= -----END CERTIFICATE----- Connecting to : Keypair sent to ssl: Controller CA cert saved to '/usr/local/ovsdb/config/ssl/controller_ca_cert1.der' Combined controller CA cert saved to '/usr/local/ovsdb/config/ssl/controller_ca_cert.der' Switch #1 CA cert saved to '/usr/local/ovsdb/config/ssl/switch_ca_cert_1.der' Switch #2 CA cert saved to '/usr/local/ovsdb/config/ssl/switch_ca_cert_2.der' 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 41 of 62

42 N9K-VPC-1# Listing 26 Receiving a certificate for the plug-in on the vpc secondary switch Keypair received from ssl: :26021 OVSDB plugin keypair saved to '/usr/local/ovsdb/config/ssl/ovsdb_plugin.p12' Controller CA cert saved to '/usr/local/ovsdb/config/ssl/controller_ca_cert1.der' Combined controller CA cert saved to '/usr/local/ovsdb/config/ssl/controller_ca_cert.der' Switch #1 CA cert saved to '/usr/local/ovsdb/config/ssl/switch_ca_cert_1.der' Switch #2 CA cert saved to '/usr/local/ovsdb/config/ssl/switch_ca_cert_2.der' N9K-VPC-2# After the two switches have their certificates synchronized, display them to verify that they are identical. The commands to display the certificates are shown in Listings 27 and 28. Listing 27 Verifying the certificate on the vpc primary switch N9K-VPC-1# guestshell run sudo ovsdb-plugin cert show -----BEGIN CERTIFICATE----- MIICkjCCAfugAwIBAgIJAOV/d9zN9tvLMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV BAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9u MRIwEAYDVQQKDAlBbmRyb21lZGExFTATBgNVBAMMDG92c2RiX3BsdWdpbjAeFw0x NjEyMjYyMjI3NTZaFw0xOTEyMjYyMjI3NTZaMGIxCzAJBgNVBAYTAlVTMRMwEQYD VQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9uMRIwEAYDVQQKDAlB bmryb21lzgexftatbgnvbammdg92c2rix3bsdwdpbjcbnzanbgkqhkig9w0baqef AAOBjQAwgYkCgYEAw5f/VtkvtFySQrjBLAii/aZhoVFmoVBknRMGo6BZlw4fJK// NcgWKXSaxn6FlrgqjLmWIOQtgjDfKEfvU2CwVE+3Z0FLj8TGFGJjIYYToEwGcyWl ZbnpfQy1Jy+CnnNEcfp5tFkcxIVhf/66gtjispBVlAbu/F+HyeNMQB3JK+UCAwEA AaNQME4wHQYDVR0OBBYEFJl67js6GlgV/6p8I1nov88K/YL5MB8GA1UdIwQYMBaA FJl67js6GlgV/6p8I1nov88K/YL5MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL BQADgYEAC+qLxAt3BUB5J516aPkU4OWZE4Zan1fpmyWuanca+Tkb+zKSh7bQwGbr mrzr8jyjmhml1vgfej34bjtecawzkdksqxptb6dvdssaen9lksl0mzyzxeom+4xb nt7wl1lzumolmftzfzqeexzbmctiu/hsnlgqtv5ns775sbyisda= -----END CERTIFICATE----- N9K-VPC-1# Listing 28 Verifying the certificate on the vpc primary switch N9K-VPC-2# guestshell run sudo ovsdb-plugin cert show -----BEGIN CERTIFICATE----- MIICkjCCAfugAwIBAgIJAOV/d9zN9tvLMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV BAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9u MRIwEAYDVQQKDAlBbmRyb21lZGExFTATBgNVBAMMDG92c2RiX3BsdWdpbjAeFw0x NjEyMjYyMjI3NTZaFw0xOTEyMjYyMjI3NTZaMGIxCzAJBgNVBAYTAlVTMRMwEQYD VQQIDApDYWxpZm9ybmlhMRMwEQYDVQQHDApQbGVhc2FudG9uMRIwEAYDVQQKDAlB 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 42 of 62

43 bmryb21lzgexftatbgnvbammdg92c2rix3bsdwdpbjcbnzanbgkqhkig9w0baqef AAOBjQAwgYkCgYEAw5f/VtkvtFySQrjBLAii/aZhoVFmoVBknRMGo6BZlw4fJK// NcgWKXSaxn6FlrgqjLmWIOQtgjDfKEfvU2CwVE+3Z0FLj8TGFGJjIYYToEwGcyWl ZbnpfQy1Jy+CnnNEcfp5tFkcxIVhf/66gtjispBVlAbu/F+HyeNMQB3JK+UCAwEA AaNQME4wHQYDVR0OBBYEFJl67js6GlgV/6p8I1nov88K/YL5MB8GA1UdIwQYMBaA FJl67js6GlgV/6p8I1nov88K/YL5MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL BQADgYEAC+qLxAt3BUB5J516aPkU4OWZE4Zan1fpmyWuanca+Tkb+zKSh7bQwGbr mrzr8jyjmhml1vgfej34bjtecawzkdksqxptb6dvdssaen9lksl0mzyzxeom+4xb nt7wl1lzumolmftzfzqeexzbmctiu/hsnlgqtv5ns775sbyisda= -----END CERTIFICATE----- N9K-VPC-2# Enabling the OVSDB plug-in Note: The commands discussed here are the same for both standalone and vpc configurations. The plug-in must be enabled first on the primary vpc switch. After all the previous configuration steps are complete, it is time to enable the OVSDB plug-in. This step will configure the plug-in to wait for a connection and configurations from the NSX controllers. Use this command to enable the OVSDB plug-in: switch#guestshell run sudo ovsdb-plugin service start Use this command to verify the plug-in status: switch#guestshell run sudo ovsdb-plugin service status Remember that in a vpc switch pair, the plug-in on the vpc primary switch should be enabled first, followed by the plug-in in the secondary switch. Listings 29 through 33 shows sample configuration and status commands. Listing 29 shows the configuration for a standalone switch. Listing 29 Enabling the plug-in on a standalone switch N9K# guestshell run sudo ovsdb-plugin service start Starting OVSDB plugin... nohup: appending output to 'nohup.out' OVSDB plugin started N9K# guestshell run sudo ovsdb-plugin service status Status: Running Connections Switches: #1 addr : Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 43 of 62

44 type vpc state : Local : Disabled : Up Controllers: #1 addr : state : Starting!<<This is in starting mode since we have not configured NSX yet N9K# Listings 30 through 33 show the configurations for a vpc pair of switches. Listing 30 shows the configuration on the vpc primary switch. Listing 30 Enabling the plug-in on the vpc primary switch N9K-VPC-1# guestshell run sudo ovsdb-plugin service start Starting OVSDB plugin... nohup: appending output to 'nohup.out' OVSDB plugin started N9K-VPC-1# Listing 31 shows the configuration on the vpc secondary switch. Listing 31 Enabling the plug-in on the vpc secondary switch N9K-VPC-2# guestshell run sudo ovsdb-plugin service start Starting OVSDB plugin... nohup: appending output to 'nohup.out' OVSDB plugin started N9K-VPC-2# Listing 32 shows verification on the vpc primary switch. Listing 32 Verifying the plug-in on the vpc primary switch N9K-VPC-1# guestshell run sudo ovsdb-plugin service status Status: Running Connections Switches: #1 addr : type : Local vpc : Enabled/Master state : Up 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 44 of 62

45 #2 addr : type : Remote vpc : Enabled/Slave state : Up Controllers: #1 addr : state : Starting!<<This is in starting mode since we have not configured NSX yet N9K-VPC-1# Listing 33 shows verification on the vpc secondary switch. Listing 33 Verifying the plug-in on the vpc secondary switch N9K-VPC-2# guestshell run sudo ovsdb-plugin service status Status: Running Connections Switches: #1 addr : type : Local vpc : Enabled/Slave state : Up #2 addr : type : Remote vpc : Enabled/Master state : Up Controllers: #1 addr : state : Down!<<This is in down mode since the other switch is master for now N9K-VPC-2# The vpc secondary switch plug-in is in backup, or slave, mode. Therefore, it will show its state as down. The state will change if the primary plug-in or switch goes down and failover occurs Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 45 of 62

46 Registering the HW-VTEP with VMware NSX After the OVSDB plug-in is running in the Cisco Nexus 9000 Series Switch, it is time to register the HW-VTEP with the NSX controllers. The configuration on the NSX manager will be the same for both standalone and vpc setups, because from the perspective of NSX, the vpc pair appears as a single control plane. First, a hardware gateway must be registered to NSX. The steps detailed in this section need to be performed only once for this purpose. The user must configure the HSC (in the case of the Cisco Nexus 9000 Series integration, this is the plug-in) of their hardware gateway with the NSX controller IP address. Note that there are typically three redundant NSX controllers. You need to specify only one of them, and the others will be automatically discovered. This process was completed when the plug-in was configured in the earlier sections. After the plug-in is configured to point to an NSX controller, you need to collect the certificate that will be used by the OVSDB client on the NSX controller to connect to the server on the HSC (the plug-in). To collect the certificate, as mentioned in the earlier sections, you enter the following command: switch# guestshell run sudo ovsdb-plugin cert show Note that if a vpc pair is being used, the certificates have already been synchronized, so you can enter the command on either switch. From here, the registration of a new hardware gateway in the NSX GUI is relatively straightforward. In vcenter, navigate to the Networking and Security section and select the Service Definition tab. Then open the Hardware Devices menu and click the + button, as shown in Figure 14. For this example, the replication cluster is already configured with the replication service nodes for BUM traffic as discussed earlier in this document. For more information about configuring the RSN, see the VMware documentation. Figure 14. Adding a hardware device in the VMware NSX manager GUI 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 46 of 62

47 After you click the + button, a new window appears, where you enter a name and certificate. The name can be anything that makes sense to the administrator. The certificate is the one retrieved with the guestshell run sudo ovsdb-plugin cert show command on the Cisco Nexus 9000 Series Switch. There is also an optional Description field. Note that BFD is enabled by default, so the Cisco Nexus 9000 Series Switch will establish BFD sessions with the RSNs. This configuration is critical for protecting against the silent failure of an RSN, and VMware supports only configurations that run BFD. The process for enabling BFD over VXLAN on the Cisco Nexus 9000 Series was discussed in an earlier section of this document. As mentioned earlier, the configuration for this is the same for both standalone and vpc switches. The example in Figure 15 uses a vpc switch. Figure 15. Adding a certificate for a Cisco Nexus 9000 Series Switch in the VMware NSX manager GUI After a few seconds (a refresh may be required on the browser), the new HW-VTEP should appear in the Hardware Devices window with connectivity shown as Up (Figure 16) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 47 of 62

48 Figure 16. Checking the connectivity to the Cisco Nexus 9000 Series Switch in the VMware NSX manager GUI You can also verify the connectivity in the Cisco Nexus 9000 Series Switch by entering the following command: switch# guestshell run sudo ovsdb-plugin service status An example of the output for this command for a vpc primary switch is shown in Listing 34. Listing 34 Verifying the plugin connectivity to the NSX controllers N9K-VPC-1# guestshell run sudo ovsdb-plugin service status Status: Running Connections Switches: #1 addr : type : Local vpc : Enabled/Master state : Up #2 addr : type : Remote vpc : Enabled/Slave state : Up Controllers: #1 addr : Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 48 of 62

49 state : Up #2 addr : state : Up #3 addr : state : Up N9K-VPC-1# Note that there are three controllers in the Up state. The two additional controllers were pushed down by the first controller (the one you configured). The same command can be used for a standalone switch. Binding a logical switch in VMware NSX to a physical switch, physical port, and VLAN After a hardware gateway has been added to NSX, any existing logical switch can programmatically be mapped to any physical port or VLAN advertised by the switch. The ports advertised by the Cisco Nexus 9000 Series using OVSDB are the ones shared in the Assigning VLANs and interfaces to the controllers section of this document. This section illustrates the mapping of a logical switch to a particular port, using the vcenter GUI. First, select a logical switch on the Network and Security > Logical Switches tab. Then open the Actions menu and choose Manage Hardware Bindings. Note that any logical switch that will be connected to a hardware VTEP cannot be connected to a DLR at the same time. This is an NSX requirement, and it applies to every vendor s HW-VTEP integration. An example is shown in Figure 17. Figure 17. Connecting a logical switch to an HW-VETP using the VMware NSX manager GUI A new window will appear with all the hardware gateways that are configured for this set of NSX controllers Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 49 of 62

50 Click the triangle icon to the left of the switch name to open the dialog box for that hardware gateway. You can then click the + icon. The next step is to click the Select button in the Port column, as shown in Figure 18. Figure 18. Selecting the port for a binding using the VMware NSX manager GUI After you click Select, all the ports that have been shared will appear. In the case of orphan ports in a vpc pair, the serial number of the physical switch is appended to the interface name. This naming allows the plug-in to understand the difference between Ethernet1/10 on the primary vpc switch and Ethernet1/10 on secondary vpc switch. vpc ports will appear as single interfaces as they are shared. This process does not apply to standalone mode. An example is shown in Figure 19. Figure 19. Displaying the shared ports in the VMware NSX manager GUI 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 50 of 62

51 Select the interface on this switch that you want to map to this NSX logical switch. After you have selected an interface (multiple interfaces can be selected, one at a time), enter the VLAN that will be mapped to this logical switch on this interface. Again, this is one of the VLANs was shared in the Assigning VLANs and interfaces to the controllers section of this document. After you have entered the VLAN and clicked OK, the binding is complete (Figure 20). Figure 20. Completing the HW-VTEP binding using the VMware NSX manager GUI You can verify the binding by clicking the Hardware Port Binding number that now appears on the screen, as shown in Figure 21. Figure 21. Verifying the HW-VTEP binding using the VMware NSX manager GUI You can also verify that the configuration was correctly pushed to the switch by entering the following command: switch# show run controller 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 51 of 62