Deploying and Troubleshooting the Nexus 1000V Virtual Switch

Transcription

1

2 Deploying and Troubleshooting the Nexus 1000V Virtual Switch 2

3 Agenda Session Prerequisites Current Nexus 1000V Releases Virtual Supervisor Module (VSM) Virtual Ethernet Module (VEM) Port-Profiles Port Channels VXLAN VSM High Availability Cisco Nexus 1010 Appendix 3

4 Session Prerequisites

5 Related Sessions BRKVIR-2008 BRKVIR-2011 BRKVIR-2014 BRKVIR-2017 Experiences From Delivering End to End Cloud IaaS Deploying Services in a Virtualized Environment Architecting Scalable Clouds using VXLAN and Nexus 1000V The Nexus 1000V on Microsoft Hyper-V: Expanding the Virtual Edge Deploying and Troubleshooting the Nexus 1000v virtual switch 5

6 Prerequisites Understanding of VMware ESX and vcenter Server Cisco NXOS Understand the CLI General switching concepts Cisco Nexus 1000V concepts Understand VSM and VEM Port-profile concepts 6

7 Current Releases and Futures

8 Current Nexus 1000V Releases Two release trains Latest release 4.2(1)SV1(5.1) (aka 1.5.1) 1.5.1, 1.5.1a Dropped support for ESX/ESXi (1)SV1(4b) (aka 1.4) 1.4, 1.4a, and 1.4b 1.4a first version to support ESXi 5 8

9 1.5 Features Virtual Extensible LAN (VXLAN) Support networks with 16M LAN segments LAN segments across Layer-3 Integration with VMware vcloud Director vpath 1.5 L3 support for Virtual Services Microsoft Load Balancing Support Unicast mode VRRP support for Virtual Machines https support to VSM for extension download (config)# http-server no http Updated Installer Installs the VSMs and the VEMs! 9

10 Futures Hyper-V support Works with Windows Server 8 and Hyper-V 3.0 Increase configuration limits Veth ports increasing to 10,000 Increase in the number of VEMs supported Nexus 1000V and ASA-1000V integration with vcloud Director 10

11 Virtual Supervisor Module Deployment and Troubleshooting

12 Virtual Supervisor Module (VSM) VSM is a Virtual Machine On ESX/ESXi On Nexus 1010 Control plane for the Nexus 1000V solution Responsible for VMware vcenter communication Programming and managing Virtual Ethernet Modules (VEM) 1 VSM HA pair can manage 64 VEMs Nexus 1000V can coexist with VMware vswitch and DVS 12

13 VSM Deployment Scenarios ESX/ESXi VSM as a Virtual Machine Support the VSM on a VEM Support the VSM on a vswitch or VMware DVS Customers have separate management clusters Keep VSMs on different ESX/ESXi hosts Use VMware Affinity Rules Storage wise we don t care. VSM can be hosted on network storage Changes in 1.5 to make the VSM behave better on network storage 13

14 Deployment Scenarios VSMs in Mgmt Cluster VM VM VM VM VM VM VEM-1 VEM-2 VSMs on VEM VSMs on vswitch VMware ESX VMware ESX VM VM VM VSM VM VM VM VSM VM VM VM VSM VM VM VM VSM Data Cluster VEM-1 VMware ESX VEM-2 VMware ESX VEM-1 vswitch VMware ESX VEM-2 vswitch VMware ESX Management Cluster VMware ESX vswitch VM VM VM VSM 14

15 VSM Deployment Scenarios Nexus 1010 VSM on a Nexus 1010 or 1010-X It s still a Virtual Machine 6 per 1010 pair 10 per 1010-X pair Always deploy 1010s in pairs! No support to have VSM HA on a single allows for Network team to own the virtualization platform 1010s should go in the Aggregation Layer VSMs need access to different ESX/ESXi clusters 15

16 1010/1010-X Deployment Scenario Nexus 1010 Primary VSM VSM VSM VSM Nexus 1010 Secondary Aggregation L3 L2 VMware ESX VEM-1 VM VM VM VMware ESX VEM-2 VM VM VM VMware ESX VEM-1 VM VM VM VMware ESX VEM-2 VM VM VM VMware ESX VEM-1 VM VM VM VMware ESX VEM-2 VM VM VM 16

17 Stretched Nexus 1000V Model VM VM VM VM VM VM VSMs and VEMs spread across Datacenters VSMs need to be in the same DC location Prevents an active/active scenario if DC link goes down L3 control is preferred Lowers supported configuration limits Less VEMs and VLANs Check release notes for complete list VMware ESX VEM-1 Local DC VMware ESX VEM-2 VEM-3 VEM-4 VMware ESX VMware ESX Remote DC DCI VM VM VM VSM VM VM VM VSM 17

18 VSM Control Modes L2 mode Default mode Requires L2 connectivity through Control interface to all VEM modules L3 Mode Requires an IP address be assigned to the VEM L3 uses UDP port 4785 for both source and destination Uses Mgmt or Control interface of the VSM VSM mgmt 0 is default interface for L3 Can also use control 0 Ties to control adapter of the VM (Adapter 1) 18

19 Which Control Mode? L2 or L3? L3 control You will see us start to recommend L3 control over L2 More flexible Support for add on applications Easier to troubleshoot L2 is still supported and will continue to be supported 19

20 VSM L3 Configuration and Planning Two options for the L3 control interface Mgmt 0 (default) Control 0 Use control 0 If you want to separate control and management traffic If you are going to have a lot of VEMs and want to isolate the traffic Mgmt and Control use different VRF Mgmt 0 uses VRF management Control 0 uses VRF default Primary and secondary VSM still need to be L2 adjacent 20

21 VSM Installation Initial configuration from Installer or via ISO image Installer Improvements Standalone Java application now Installs VSM in L2 or L3 Installs primary and secondary VSMs Add ESX/ESXi hosts to Nexus 1000V 21

22 VSM Virtual Machine Requirements 3 network interfaces Adapter 1 is the Control interface Heartbeat between VSMs and VEM Heartbeat between VSMs Control 0 Adapter 2 is the Management interface VSM terminal connectivity Connectivity to VMware vcenter Backup Heartbeat for VSM HA Mgmt 0 Adapter 3 is the Packet interface Passes CDP and IGMP information 2GB of memory RAM reserved 22

23 VSM to VMware vcenter Communication Cisco VSMs Virtual Center VSM connects to vcenter using SSL connection. Management Interface Self-Signed certificate used for this connection VSM configures vcenter using its API VSM creates N1KV Port-Groups in vcenter VSM also stores opaque data in vcenter VSM pulls information from vcenter (DC, DVS, VM, ) 23

24 VSM to VMware vcenter Connectivity VSM registers to vcenter using extension key plug-in Contains a public SSL Certificate Extension key of the VSM Only need to register a specific VSM extension key once Can register multiple keys from different VSMs on same vcenter A VSM is tied to a VMware Datacenter Extension key only changes Brand new VSM install Change the certificate on the VSM. "install certificate" command in svs connection mode Change the extension key using "vmware vc extension-key 24

25 VSM to VMware vcenter Datacenter VSM gets assigned to a VMware Datacenter Multiple VSMs per VMware Datacenter is supported An ESX/ESXi host can only be a member of one VSM Vmotion between VSMs is not supported 25

26 Verify VSM to vcenter Connectivity Verify SVS connection settings n1000v# show svs connections connection VC-test: ip address: protocol: vmware-vim https certificate: default datacenter name: Harrison DVS uuid: 72 f b2 01 7b 8b cf df 10 5a db 55 config status: Enabled operational status: Connected If Datacenter is underneath a folder and spaces n1000v(config-svs-conn)# vmware dvs datacenter-name? LINE Datacenter name in VC with path (e.g. DCName, DCFolder/DC Name) 26

27 Connectivity Error Extension Key Below error means wrong key or key is not registered n1000v(config-svs-conn)# connect ERROR: [VMware vcenter Server build ] Extension key was not registered before its use Register the Key in VMware vcenter 27

28 Connectivity Error Connection Refused Below error could indicate port mismatch n1000v(config-svs-conn)# connect ERROR: [VMWARE-VIM] Operation could not be completed due to connection failure.connection refused. connect failed in tcp_connect() Default port for communication is port 80 Admins change default port for various reasons 28

29 What Port is VMware configured to Use? Verify Port number in vcenter Administration->vCenter Server Settings 29

30 Change the Port on the VSM Make sure VSM SVS port matches vcenter http port n1000v# show svs connections connection vcenter: ip address: remote port: 80 protocol: vmware-vim https To change the port n1000v(config)# svs connection vcenter n1000v(config-svs-conn)# remote port

31 VSM and vmotion Manual vmotion of VSM is supported Not recommended to allow DRS to vmotion Primary and Secondary VSM Aggressive DRS vmotion setting can cause VSM to drop packets and loose connectivity to VEM Best practice to keep Primary and Secondary VSM outside of DRS 31

32 Backing up the VSM A running-config is not always enough to restore VSM on ESX/ESXi We now support clone to a template You can restore from a template and saved-config VSM on Nexus 1010 You can now export a VSM Import the saved VSM to restore VSM on ESX/ESXi Snapshots Not officially supported Useful for upgrades 32

33 VSM Best Practices L3 control will become preferred method Use control 0 over mgmt 0 Primary and Standby VSM in same L2 domain!!! VSM on VEM is supported VSM primary to secondary latency <= 5ms VSM to VEM latency 5ms Backup your config!!! 33

34 Virtual Ethernet Module Deployment and Troubleshooting

35 VEM Deployment Recommending L3 Control L3 control requires a VMKernel NIC on ESX We need an L3 interface to forward control traffic We recommend creating VMK interface explicitly for the VEM An existing VMK interface can be used in test or small environments VMK interface must be moved to the VEM 35

36 L3 VEM Deployment Requirements VSM setup for L3 control Create VMK Interface on ESXi host Test ping Create uplink port-profile Create veth port-profile with capability l3control Migrate host and VMK interface VEM gets installed via VUM, manual, or installer app Host will not show up until VMK interface is moved to port-profile 36

37 VEM Installation - VUM VMware Update Manager(VUM) does all the work Requires HTTP server on the VSM Turn off the following VMware cluster settings HA, DRS, and DPM Logs on vcenter Server in C:/Documents and Settings/Application Data/All Users/VMware/VUM/logs VEM modules get stored in C:\ProgramData\VMware\VMware Update Manager\Data\hostupdate CISCO directory comes from VSM CSCO directory comes from VUM portal Uninstalling VUM does not clean out the above directories 37

38 VEM Installation - Manual With ESXi 5 use esxcli Can be run remote or locally on the host Local esxi5.0# esxcli software vib install v patch01/cross_cisco-vem-v vib Remote linux1# esxcli --server <server> software vib install -v Address>/cisco/vibs/VEM/4.1.0/VEM patch01/cross_cisco-vem-v vib ESX/ESXi 4.1 Esxupdate (local) esx4.1# esxupdate b cross_cisco-vem-v vib update Vihostupdate (remote) linux1# vihostupdate install bundle cisco-vem-v zip --server <server> 38

39 VEM Installation Nexus 1000V Installer App Installation App is now standalone Java application Requires administrator privileges to the ESXi host Allows the network admin to directly install the VEM Need to start it with VEM option java.exe jar Nexus1000V-install.jar VEM 39

40 VEM Installation ESXi Stateless VMware introduced Stateless ESXi with version 5 ESXi PXE boots No information is stored on local disks No place to install the VEM and store opaque data VEM module has to be built into the boot image Possible using VMware Powershell Instructions are in Install and Upgrade Guide 1_5_1/install_upgrade/vsm_vem/guide/n1000v_installupgrade.html 40

41 VEM Installation DVS Error DVS operation failed error VUM is not installed or configured VUM could not find the right VEM version Check the VUM logs Cluster HA, DRS, DPM was not disabled Manual installation of VEM was not performed 41

42 VEM Manual Installation Issues Dependency error ~]# esxupdate -b./cross_cisco-vem-v release.vib u pdate cross_cisco-vem-v ############################################## [100%] Unpacking cross_cisco-vem-v100-es.. ############################################## [100%] The following problems were encountered trying to resolve dependencies: No VIB provides 'vmknexus1kvapi-0-4' (required by cross_ciscovem-v100-esx_ ) Requested VIB cross_cisco-vem-v100-esx_ conflicts with the host Verify VEM VIB version to ESX/ESXi version Compatibility matrix will identify right VEM version 42

43 VEM Seeding How does VEM know VSM information? Opaque data copied to VEM to seed during install. Opaque data consists of: Domain-cfg (Domain ID, Control VLAN, Packet VLAN) Switchname VSM image version System profiles [System VLANS, profile names] IP address MAC address 43

44 Checking Opaque Data VSM stores opaque-data in vcenter as persistent data for its DVS. vcenter downloads this information to ESX for VEM to use, whenever a host is added to N1KV-DVS Checking opaque-data in VSM switch-cp# show svs domain SVS domain config: Domain id: 100 Control vlan: 150 Packet vlan: 150 Status: Config push to VC successful. Checking opaque-data in VEM [root@sfish sbin]# /usr/lib/ext/cisco/nexus/vem/sbin/vemcmd show card Switch name: switch-cp Card domain: 100 Card slot: 2 Card control VLAN: 150 Card packet VLAN: 150 Checking opaque-data in vcenter Content rootfolder (group-dx) childentity (datacenter-n) networkfolder (group-n6) childentity (group-n) -> childentity (dvs-n) config VendorSpecificConfig 44

45 VEM VSM Connectivity Troubleshooting VEM adds in vcenter but does not show up on VSM show module With L2 most of the time its a Control VLAN issue Verify Control VLAN connectivity Appendix for more L2 troubleshooting tips With L3 its usually an IP routing problem If you can ping VMK interface the VEM should connect to VSM Troubleshoot as you would all VMware L3 issues Is the VMK port-profile set with system VLAN and capability L3control 45

46 VEM L3 Troubleshooting - Steps VMK interface created on ESX/ESXi host Can ESX/ESXi host ping VSM control/mgmt interface Uplink profile created correctly L3 veth port-profile created correctly Check the opaque data VSM VMK VLAN 10 Mgmt 0: VLAN 151 Control 0: VLAN 150 VM VM VM VEM-1 VMware ESX Sample Topology VM VM VM VEM-2 VMware ESX VMK VLAN 11 46

47 VMK Interface ESX/ESXi VMK interfaces are special interfaces Need to be created by VMware Admin Verify interface exists ~ # esxcfg-vmknic -l Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type vmk0 VMkernel IPv :18:fe:72:f2:aa true STATIC vmk2 34 IPv :50:56:78:6f: true STATIC Can you ping the control interface of the VSM? ~ # vmkping PING ( ): 56 data bytes 64 bytes from : icmp_seq=0 ttl=255 time=0.570 ms 64 bytes from : icmp_seq=1 ttl=255 time=0.420 ms 47

48 Verify the VSM Check SVS domain parameters n1kv-l3# show svs domain SVS domain config: Domain id: 43 Control vlan: 1 Packet vlan: 1 L2/L3 Control mode: L3 L3 control interface: control0 Verify control 0 n1kv-l3# show run int control 0 interface control0 Verify VRF default n1kv-l3# show ip route IP Route Table for VRF "default" /0, ubest/mbest: 1/0, pending *via , control0, [1/0], 4d23h, static Can the VSM ping the VMK interface n1kv-l3# ping vrf default PING ( ): 56 data bytes 64 bytes from : icmp_seq=0 ttl=63 time=1.082 ms 64 bytes from : icmp_seq=1 ttl=63 time=0.841 ms ip address /24 48

49 Check the Uplink Port-Profile Uplink needs to allow VLANs 10 and 11 VLANs 10 and 11 need to be system vlans n1kv-l3# show run port-profile uplink port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 10-11, channel-group auto mode on mac-pinning no shutdown system vlan state enabled 49

50 Check the veth Port-Profile VMK interface needs to be migrated to this port-profile Allow VLAN 11 and have system VLAN set It also must have capability l3control n1kv-l3# show run port-profile L3-control-vlan11 port-profile type vethernet L3-control-vlan11 capability l3control vmware port-group switchport mode access switchport access vlan 11 no shutdown system vlan 11 state enabled Each VMK VLAN needs a new port-profile 50

51 Check Opaque Data Is the right Opaque data getting pushed to the ESX/ESXi host? # vemcmd show card Card UUID type 2: 414a e375a37 Card name: cae-esx-186 Switch name: n1kv-l3 Switch alias: DvsPortset-0 Should match VLAN defined veth port-profile Switch uuid: e2 ba af 6c bc 25 cf 3f 86 Card domain: 43 Card slot: 4 VEM Tunnel Mode: L3 Mode L3 Ctrl Index: 49 L3 Ctrl VLAN: 11 VEM Control (AIPC) MAC: 00:02:3d:10:2b:03 Should match MAC of control 0 or mgmt 0 VEM Packet (Inband) MAC: 00:02:3d:20:2b:03 VEM Control Agent (DPA) MAC: 00:02:3d:40:2b:03 VEM SPAN MAC: 00:02:3d:30:2b:03 Primary VSM MAC : 00:50:56:af:49:30 Primary VSM PKT MAC : 00:50:56:af:59:a1 Primary VSM MGMT MAC : 00:50:56:af:47:51 Standby VSM CTRL MAC : 00:50:56:af:61:e7 51

52 Check Counters for Heartbeat on VSM n1kv-l3# show module vem counters Mod InNR OutMI InMI OutHBeats InHBeats InsCnt RemCnt Crit Tx Errs InNR - NodeID requests received count OutMI - Module Insert Start requests sent to VEM InMI - Module Insert Start responses received from VEM OutHBeats - Number of HBs which have been broadcast by VSM InHBeats - Number of HBs received from this VEM 52

53 View Heartbeat Messages on VEM Use vempkt on the ESX/ESXi host vempkt capture ingress/egress vlan 11 Let it run vempkt cancel capture all vempkt display detail all vempkt can now export to a pcap file vempkt pcap export <filename> Look for heartbeat messages from VSM 53

54 Verifying Modules with vem status vem status v Reboot the ESX host if not all the modules have loaded. ~ # vem status -v Package vssnet-esxmn-ga-release Version Build 1 Date Mon Jan 30 18:38:49 PST 2012 Number of PassThru NICs are 0 VEM modules are loaded Switch Name Num Ports Used Ports Configured Ports MTU Uplinks vswitch vmnic0 DVS Name Num Ports Used Ports Configured Ports MTU Uplinks n1kv-l vmnic1 Number of PassThru NICs are 0 VEM Agent (vemdpa) is running 54

55 VEM Best Practices Control network should have low latency (5ms) and available bandwidth VEM and VSM running on the same versions Use one eth uplink with all vlans Segregate VM traffic with port-profiles and MAC pinning Upstream switch ports configured identically On UCS make Service Profile does not contain Dynamic VNICs VEM and VM-FEX are mutually exclusive n1000v-mv# Hard code config VEM t to module number with. n1000v-mv(config)# vem 12 n1000v-mv(config-vem-slot)# host vmware id E

56 Port-Profiles Deployment and Troubleshooting

57 Uplink(eth) Port-Profile Troubleshooting Port-Profiles with multiple NICs need a port-channel Causes duplicate packets Kicks in déjà vu driver Requires extra CPU processing Fills the logs Example: Eth 6/1 and Eth 6/3 added to below Port-Profile WRONG port-profile type ethernet uplink-nopc vmware port-group switchport mode trunk switchport trunk allowed vlan , no shutdown system vlan 11 state enabled RIGHT port-profile type ethernet uplink-nopc vmware port-group switchport mode trunk switchport trunk allowed vlan , channel-group auto mode on mac-pinning no shutdown system vlan 11 state enabled 57

58 Uplinks with Overlapping VLANs VLANs cannot overlap on uplink port-profiles Example: Two port-profiles Overlap of VLAN 12 Assign 6/1 to uplink1 and 6/3 to uplink2 Which uplink will a VM on VLAN 12 use? No way to pin a VM to an uplink port-profile Use VPC MAC Pinning port-profile type ethernet uplink1 vmware port-group switchport mode trunk switchport trunk allowed vlan 1,10,11,12 no shutdown state enabled port-profile type ethernet uplink2 vmware port-group switchport mode trunk switchport trunk allowed vlan 12,13,14 no shutdown state enabled 58

59 VM(veth) Port-Profile Troubleshooting Be careful modifying veths directly Modify the port-profile and not the veth VSM remembers VM veths until they are deleted Changes to a veth will stick around until the VM nic is deleted VM to veth mapping does not change until NIC is removed from the VM NIC is reassigned to another port-profile Use VMware VMXNET3 NIC type over E

60 Cisco Nexus 1000V System VLANs System VLANs enable interface connectivity before an interface is programmed Address chicken and egg issue VEM needs to be programmed, but it needs a working network for this to happen Port profiles that contain system VLANs are system port profiles Allowed 32 port-profiles with system VLAN System port-profiles become part of the opaque data VEM will load system port-profiles and pass traffic even if VSM is not up System vlans must be set on egress and ingress port-profiles 60

61 System VLAN Guidelines The system VLAN must be a subset of the allowed VLAN list on trunk ports Only one system VLAN on an access port The no system vlan command only when no interface is using the profile. Once a system profile is in use by at least one interface Can add to the list of system VLANs Cannot delete any VLANs from the list. Required System VLANs Control, Packet, IP Storage, Service Console, VMKernel, any Management Networks 61

62 System VLAN Example Migrate VMware Service Console to VEM SC interface uses VLAN 2 Uplink port-profile must define VLAN 2 as system vlan uplink-pinning port-profile type ethernet uplink-pinning vmware port-group switchport mode trunk switchport trunk allowed vlan all channel-group auto mode on mac-pinning no shutdown system vlan 2,10, Service Console Port-profile must also define system vlan n1000v# show run port-profile SC port-profile type vethernet SC vmware port-group switchport mode access switchport access vlan 2 no shutdown system vlan 2 62

63 How do I Recover? Vemcmd allows you to set system vlan on LTLs on ESX host From ESX/ESXi console [root@cae-esx-180 ~]# vemcmd show port LTL IfIndex Vlan Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name VIRT UP UP 4 Access l VIRT UP UP 4 Access l VIRT UP UP 4 Access l VIRT UP UP 4 Access l VIRT UP UP 4 Access l VIRT UP UP 0 Access l VIRT UP UP 4 Access l VIRT UP UP 4 Access l a T PHYS UP UP 4 Trunk vmnic0 48 1b VIRT UP UP 4 Access vmk0 49 1b VIRT UP UP 4 Access vswif T VIRT UP UP 4 Trunk [root@cae-esx-180 ~]# vemcmd set system-vlan 2 ltl 49 63

64 Jumbo Frames Support MTU setting for eth type port-profile Simply use mtu size in port-profile and nothing else Add system vlan directive to Port-profile if needed System jumbo mtu global setting all versions Sets the system wide jumbo mtu size Generally do not need to change 64

65 Jumbo Frames Support - Example Eth port-profile with jumbo MTU n1000v-av(config)# port-profile type eth uplink-jumbo n1000v-av(config-port-prof)# switchport mode trunk n1000v-av(config-port-prof)# switchport trunk allowed vlan 1,3-149, n1000v-av(config-port-prof)# vmware port-group n1000v-av(config-port-prof)# mtu 9000 n1000v-av(config-port-prof)# no shut n1000v-av(config-port-prof)# system vlan 3,10 n1000v-av(config-port-prof)# state enabled Use esxcfg-nics l on the ESX host to confirm [root@cae-esx-133 ~]# esxcfg-nics -l Name PCI Driver Link Speed Duplex MAC Address MTU Description vmnic0 08:00.00 enic Up 10000Mbps Full 00:25:b5:00:00:1e 1500 Cisco Systems Inc 10G Ethernet NIC vmnic1 09:00.00 enic Up 10000Mbps Full 00:25:b5:00:00:0e 1500 Cisco Systems Inc 10G Ethernet NIC vmnic2 0a:00.00 enic Up 10000Mbps Full 00:25:b5:00:00:0f 9000 Cisco Systems Inc 10G Ethernet NIC 65

66 VMware DVS Max-port Issues Default to 32 max-ports per port-profile Some customers set the number higher Counts toward the maximum number of VMware DVS ports 8192 by default pre-provisioned Some ports are consumed when you add an ESX host to the DVS N1KV provides Two commands to remedy this Max-ports under svs connection <name> Allows you to increase the ports of the VMware DVS Port-binding auto in veth port-profiles N1KV dynamically adds ports as VMs are added Will become default in 1.5b 66

67 Port-Profile Using Weighted QOS vmotion in 4.1 can now take advantage of 10Gb links Increases the number and bandwidth of vmotions ESX 4.1 introduced NetIOC (Network I/O Control) Allows for QOS to classify traffic based on type Nexus 1000V can do this with QOS Fair Weighted Queuing Works on egress uplink ports only Easy to setup and configure Use vemcmd show qos queue-rate ltl to verify VEM is matching packets 67

68 Port-Profile Using Weighted QOS Configuration Steps to limit vmotion traffic n1kv-l3(config)# class-map type queuing match-all vmotion-class n1kv-l3(config-cmap-que)# match protocol? n1k_control N1K control traffic n1k_mgmt N1K management traffic n1k_packet N1K inband traffic vmw_ft VMware fault tolerance traffic vmw_iscsi VMware iscsi traffic vmw_mgmt VMware management traffic vmw_nfs VMware NFS traffic vmw_vmotion VMware vmotion traffic n1kv-l3(config-cmap-que)# match protocol vmw_vmotion n1kv-l3(config-cmap-que)# policy-map type queuing vmotion-policy n1kv-l3(config-pmap-que)# class type queuing vmotion-class n1kv-l3(config-pmap-c-que)# bandwidth percent 50 n1kv-l3(config)# port-profile type eth uplink-vpc n1kv-l3(config-port-prof)# service-policy type queuing output vmotion-policy 68

69 Port Channels

70 Port Channels 3 load balancing modes LACP Port-channels Upstream switch support and configuration VPC MAC Pinning Works with any upstream switch Allows for pinning of veths (VM) to specific links. VPC Host Mode CDP/Manual NIC association is either Manual or CDP Multiple connections per physical switch require a port-channel 70

71 LACP Port Channels Use when single upstream or clustered (vpc,vss) switch Use channel-group auto mode active on N1KV Use channel-group # mode active on upstream switch Switchports must be configured with spanning-tree portfast trunk spanning-tree bpdufilter enable As of 1.4a LACP Offload is on by default LACP offload lets the VEM negotiate LACP Greatly improves LACP stability 71

72 LACP Troubleshooting Do not use Network State Tracking(NST) with LACP LACP Port-Channel configured on the upstream switches Port-profile created with channel-group auto mode active On the VEM vemcmd show lacp On the VSM and Upstream Switch show port-channel summary show lacp counters/neighbor Are you seeing LACP PDUs? 72

73 VPC - MAC Pinning Each Eth interface added is a unique Service Group SGID # get assigned based off vmnic# Use pinning id command under vethernet port-profile Pins the VM to a particular uplink Ordered list for backup n1kv-l3(config-port-prof)# pinning id 0 backup 1 2 Default assignment is Round Robin to an SGID New command to make SGID # relative n1kv-l3(config-port-prof)# channel-group auto mode on mac-pinning relative 73

74 Port Channels How to Tell Pinning Can run from the VSM now No need to run command on the VEM n1kv-l3# show int virtual pinning module Veth Pinned Associated PO List of Sub Group id interface Eth interface(s) Veth2 0 Po5 Eth5/1 Veth4 2 Po5 Eth5/3 Veth5 0 Po5 Eth5/1 Veth6 2 Po5 Eth5/3 Veth7 0 Po5 Eth5/1 74

75 Port Channels Best Practice If the upstream switch can be clustered (VPC, VBS Stack, VSS) use LACP If you are using LACP also use LACP Offload If the upstream switch can NOT be clustered use MAC-PINNING Create channel-groups in port-profile Let VSM build the port-channel All physical switch ports in port-channel configured identical 75

76 Spanning-tree and BPDU Best Practice Mandatory Spanning-tree settings per port IOS set STP portfast cat65k-1(config-if)# spanning-tree portfast trunk NXOS set port type edge n5k-1(config-if)# spanning-tree port type edge trunk Highly Recommended Global BPDU Filter/Guard IOS cat65k(config)# spanning-tree portfast bpdufilter cat65k(config)# spanning-tree portfast bpduguard NXOS n5k-1(config)# spanning-tree port type edge bpduguard default n5k-1(config)# spanning-tree port type edge bpdufilter default BPDU Filter is mandatory for LACP port-channels Set per port BPDU Filter/Guard when Global is not possible 76

77 Virtual Extensible LAN

78 What is VXLAN? What is it? Why do I need it What is required Multicast Larger MTU size How to set it up Troubleshooting tips 78

79 Virtual Extensible Local Area Network (VXLAN) Ethernet in IP overlay network Entire L2 frame encapsulated in UDP (port 8472) 50 bytes of overhead Include 24 bit VXLAN Identifier 16 M logical networks Mapped into local bridge domains Unique multicast group per segment VXLAN can cross Layer 3 Tunnel between VEMs VMs do NOT see VXLAN ID IP multicast used for L2 broadcast/multicast, unknown unicast Technology submitted to IETF for standardization Outer MAC DA Outer MAC SA Outer 802.1Q Outer IP DA Outer IP SA Outer UDP VXLAN ID (24 bits) Inner MAC DA InnerM AC SA Optional Inner 802.1Q Original Ethernet Payload CRC VXLAN Encapsulation Original Ethernet Frame 79

80 Virtual Extensible Local Area Network (VXLAN) Each overlay network is known as a VXLAN segment Each VXLAN segment identified by a 24-bit segment ID (VNI) VXLAN traffic carried between VXLAN Tunnel Endpoints (VTEP) VEM module acts as the VTEP VM traffic is carried over point to point tunnels between VTEPs VM to VM traffic is encapsulated in a VXLAN header VTEP uses multicast to discover unknown destination VTEP for a VM VM MAC to VTEP mapping is gleaned from VTEP multicast traffic 80

81 Why do I Need VXLAN? Fast on-demand provisioning of large number of isolated L2 networks On demand networks without physical network re-configuration Enables network snapshot of applications Move from dev to test to production with no re-ip Massive scale for multi-tenant environments Tenants can use the same IP addresses Allows virtual L2 network to stretch across physical L2 boundaries 81

82 VXLAN Requirements With VMware vcloud Director (vcd) and Nexus 1000V Integration Full integration with vcloud Director Automatic provisioning of VXLAN segments from vcd and vshield Manager Uses Network Segmentation Manager process on VSM VMware vshield Edge acts a the IP gateway Future will allow ASA 1000V as gateway Standalone on the Nexus 1000V Create and manage segments on the VSM VMkernel interface to act as VTEP Control Mode should be L3 82

83 VXLAN Requirements IP Multicast forwarding is required Multiple segments can be mapped to a single multicast group If VXLAN transport is contained to a single VLAN, IGMP Querier must be enabled on that VLAN If VXLAN transport is traversing routers Multicast routing must be enabled. Proxy ARP must also be enabled Increased MTU for VXLAN encapsulation overhead 1550 minimum Leverage 5-tuple hash distribution for uplink and interswitch LACP Encapsulation will generate a source UDP port based on a hash of inner packet 5-tuple 83

84 VXLAN Configuration Multicast Addresses Upstream Switch Configuration Enable IGMP Querier Set physical switch port MTU to 1550 ESX/ESXi Host Create VMK interface for VXLAN Nexus 1000V Enable feature segmentation Create a Bridge Domain Create a port-profile for VTEP VMK interface Create a veth port-profile for the VMs 84

85 VXLAN Configuration Increase the MTU on your eth port-profile n1kv-l3(config)# port-profile type eth uplink n1kv-l3(config-port-prof)# mtu 1550 Create veth port-profile for VXLAN VMK interface n1kv-l3(config)# port-profile type vethernet VXLAN-VMK n1kv-l3(config-port-prof)# switchport mode access n1kv-l3(config-port-prof)# switchport access vlan 11 n1kv-l3(config-port-prof)# no shutdown n1kv-l3(config-port-prof)# system vlan 11 n1kv-l3(config-port-prof)# vmware port-group n1kv-l3(config-port-prof)# capability vxlan n1kv-l3(config-port-prof)# state enabled 85

86 VXLAN Configuration Configure the Bridge Domain Maps a segment ID to a multicast address Segment ID >4096 n1kv-l3(config)# bridge-domain vxlan-1 n1kv-l3(config-bd)# segment id 5000 n1kv-l3(config-bd)# group Create VM port-profile n1kv-l3(config)# port-profile type veth vm-vxlan-1 n1kv-l3(config-port-prof)# vmware port-group n1kv-l3(config-port-prof)# switchport mode access n1kv-l3(config-port-prof)# switchport access bridge-domain vxlan-1 n1kv-l3(config-port-prof)# no shut n1kv-l3(config-port-prof)# state enabled 86

87 VXLAN Troubleshooting Tips Verify your Bridge Domains, VM port-profiles, and VXLAN VMK portprofiles Verify multicast on your upstream switches show ip igmp snooping Do you see the VTEPs Use vmkping on the ESXi host to verify network and MTU Use 1542 to cover the addition of the ICMP header ~ # vmkping -s d Verify the VEM has the right VXLAN capability ~ # vemcmd show vxlan interfaces LTL IP

88 VXLAN Troubleshooting Tips Verify your VM is on the right segment id ~ # vemcmd show port vlans Native VLAN Allowed LTL VSM Port Mode VLAN/ State Vlans/SegID SegID 17 Eth4/1 T 1 FWD 25, Eth4/2 T 1 FWD 25, Veth2 A 25 FWD Veth14 A 25 FWD Veth19 A 6000 FWD 6000 Verify the VEM was programmed correctly ~ # vemcmd show segment 6000 BD 23, vdc 1, segment id 6000, segment group IP , swbd 4096, 2 ports, "dvs.vcdvsvcdni vl634-backed-b69c1d1d-02bf b7e-fa06c64e8c18" Portlist: 53 vse-vcdni-6-26-vl634-backed (b6 68 vcdni-2 (5ac7d73c-d1d ef 88

89 VXLAN Other Useful Commands vemcmd show port vemcmd show igmp <vlan> vemcmd show l2 segment <segment-id> vemcmd show vxlan-encap [ltl/mac] <ltl/mac address> vemcmd show vlxan-stats all Detailed slides in the Appendix 89

90 Nexus 1000V High Availability

91 VSM High Availability (HA) Major Changes in HA in version 1.4a VSM has 3 modes Standalone, Primary, or Secondary HA is limited to 2 VSM Virtual Machines (primary and secondary) Same Management, Control and Packet networks Heartbeats VSM-VSM every second drop after 6 missed VSM-VEM every second drop after 6 missed 91

92 VSM High Availability Behavior was changed in 1.4a No longer continuous rebooting of standby VSM Single reboot and it stays in warm standby state Management interface is truly used for backup heartbeat State changes to degraded when control 0 fails Use show system internal redundancy trace for state changes n1kv-l3# show system internal redundancy trace 1 0s START_THREAD ST_NP ST_NP ST_INVALID 2 0s CP_STATUS_CHG ST_INIT ST_NP ST_INIT 3 5s DEGRADED_MODE ST_INIT ST_NP ST_INIT 4 5s STATE_TRANS ST_INIT ST_NP ST_INIT EV_OS_NP ST_AC_NP 5 0s CP_STATUS_CHG ST_AC ST_NP ST_AC_NP 6 4s SET_VER_RCVD ST_AC ST_NP ST_AC_NP 7 0s STATE_TRANS ST_AC ST_INIT ST_AC_NP EV_OS_INIT ST_AC_INIT 8 0s STATE_TRANS ST_AC ST_SB ST_AC_INIT EV_OS_SB ST_AC_SB 92

93 VSM VSM Heartbeat n1000v-mv# show system internal redundancy info My CP: slot: 0 domain: 184 role: primary status: RDN_ST_AC state: RDN_DRV_ST_AC_SB intr: enabled power_off_reqs: 0 reset_reqs: 1 Other CP: slot: 1 status: RDN_ST_SB active: true ver_rcvd: true degraded_mode: false Redun Device 0: name: ha0 Active VSM Standby VSM Statistics & Errors pdev: bc1bb000 alarm: false mac: 00:50:56:8e:5e:f5 tx_set_ver_req_pkts: 13 tx_set_ver_rsp_pkts: 2 tx_heartbeat_req_pkts: tx_heartbeat_rsp_pkts: 318 rx_set_ver_req_pkts: 2 rx_set_ver_rsp_pkts: 1 rx_heartbeat_req_pkts: 318 rx_heartbeat_rsp_pkts: rx_drops_wrong_domain: 0 rx_drops_wrong_slot: 0 rx_drops_short_pkt: 0 rx_drops_queue_full: 0 rx_drops_inactive_cp: 0 rx_drops_bad_src: 0 rx_drops_not_ready: 0 rx_unknown_pkts: 0 93

94 Reference Topology switch switch Mgmt 4 Mgmt VSM1 Active VSM2 Standby 2 Control/Packet 1 Control/Packet switch 3 switch VEM1 VEM2 VEM3 VEM4 94

95 Fail Scenario 1 Failed Interface Mgmt VSM1 4 VSM2 Mgmt #1- VSM2 Control Effect: (when communication lost) VSMs use mgmt heartbeat VSM1 remains primary 2 Control 3 1 Control VSM2 remains standby VEMs stay connected Exit: (when communication restored) VSMs use control heartbeat VSM1 remains primary VEM1 VEM2 VEM3 VEM4 VSM2 remains standby 95

96 Fail Scenario 2 2 Mgmt VSM1 Control 4 VSM2 1 Mgmt Control Failed Interface #2- VSM1 Control Effect: VEMs 1-4 become headless VSMs use mgmt for heartbeat VSM1 remains primary VSM2 remains secondary VEM1 VEM2 3 VEM3 VEM4 Exit: VEMs Reconnect VSMs use control for heartbeat VSM1 remains primary VSM2 remains secondary 96

97 Fail Scenario 3 2 Mgmt VSM1 Control 4 VSM2 1 Mgmt Control Failed Interface #3 - Split DVS Effect: VEM3,4 become headless VSMs use mgmt for heartbeat VSM1 remains primary VSM2 remains secondary VEM1 VEM2 3 VEM3 VEM4 Exit: VEM3,4 reconnect VSM use control for heartbeat VSM1 remains primary VSM2 remains secondary 97

98 Fail Scenario 4 2 VEM1 Mgmt VSM1 Control/ Packet VEM2 4 3 VSM2 1 VEM3 Mgmt Control/P acket VEM4 Failed Interface #2 & #4 - Split Brain Effect: VSM2 becomes Active, taking over VEM1-4 VSM1 stays active but drops all VEMs VSM1 and VSM2 use same IP address Exit: VSM1 reboots VSM2 stays primary VEM1-4 stay connected to VSM2 VSM1 joins as standby 98

99 Fail Scenario 5 2 VEM1 Mgmt VSM1 Control/Pa cket VEM2 4 3 VSM2 1 VEM3 Mgmt Control/Pa cket VEM4 Failed Interface #1 & #4 - Split Brain Effect: VSM2 becomes Active, but does not see any VEMs VSM1 stays active, handling all VEMs VSM1 and VSM2 use same IP address Exit: VSM2 reboots VEM1-4 remain connected to VSM1 VSM2 joins as standby 99

100 Nexus 1010 and 1010-X

101 Cisco Nexus 1010 and 1010-X Based off UCS C200 M2 server Provide 6 x 1GB network connections 4 distinct topologies 1 flexible topology Virtual Service Blade (VSB) Support 1010 supports X supports 10 Current supported VSBs Nexus 1000V VSM Virtual Security Gateway (VSG) Network Analysis Module (NAM) Data Center Network Manager (DCNM) 101

102 Cisco Nexus 1010 Must be deployed in pairs No option for a standalone 1010 Deploy in the Aggregation Layer Must be in the same L2 domain for management and control Uses same HA mechanism as VSM with domain id and control vlan Do not overlap the domain id between a 1010 and a VSM Not supported Primary and Secondary VSM on same 1010 Primary VSM on ESX and Secondary VSM on 1010 or vice versa 1010s split between data centers 102

103 VSB Import/Export Previously no way to save off a VSB Works with VSM, NAM, and VSG Can Import/Export both primary and secondary Export requires that VSB be shutdown Images are stored in export-import/ dir on bootflash Can be manually copied off to remote storage 103

104 Using Export to backup a VSM Shutdown primary VSM Secondary VSM will take over and run Nexus 1000v control plane f n1010-1(config)# virtual-service-blade training f n1010-1(config-vsb-config)# shutdown primary Export VSB on 1010 f n1010-1(config-vsb-config)# export primary Note: export started.. Note: please be patient.. Note: export completed... Verify f n1010-1(config-vsb-config)# dir bootflash:///export-import/ Oct 18 02:47: Vdisk4.img.tar

105 Using Import to Restore a VSB Copy the backup to bootflash f n1010-1# copy scp://root@ /root/vdisk4.img.tar.00 bootflash:export-import vrf management Import the image f n1010-1(config)# virtual-service-blade training f n1010-1(config-vsb-config)# import primary Vdisk4.img.tar.00 Note: import started.. Note: please be patient.. Note: Import cli returns check VSB status for completion Verify f n1010-1(config-vsb-config)# show virtual-service-blade name training 105

106 Nexus 1010 Network Classes and Topologies Network traffic is classed into 3 categories Management Carries the mgmt 0 interface of the 1010 Carries the mgmt 0 traffic for all VSMs installed No longer required that VSM mgmt and 1010 mgmt be on the same subnet Control Carries all the control and packet traffic for the VSMs installed on the 1010 Carries control traffic for HA between primary and secondary 1010 Data Used by Virtual Service Blades (VSB) other than VSM Currently carries traffic for NAM blade 5 Network Topologies you can choose 106

107 Topology 1 Single Uplink Traffic will only flow over one link (ex eth1) Should eth1 fail traffic will fail to eth2 Should eth2 fail services will fail to secondary 1010 Port-channel is not supported 1GB of throughput Best practice eth1 and eth2 connect to different switches for redundancy 107

108 Topology 2 Two uplinks 1 Management and Control are still combined 1GB throughput Data traffic for NAM for best redundancy and throughput Without VPC/VSS Eth3 and Eth5 are paired LACP portchannel Eth4 and Eth6 are paired LACP portchannel 2 GB of throughput With VPC/VSS One big LACP port-chanel 4 GB of throughput 108

109 Topology 3 Two uplinks 2 Management dedicated interfaces 1GB throughput Control and Data traffic combined Without VPC/VSS Eth3 and Eth5 are paired LACP portchannel Eth4 and Eth6 are paired LACP portchannel 2 GB of throughput With VPC/VSS One big LACP port-channel 4 GB of throughput 109

110 Topology 4 Three Uplinks Management dedicated interfaces Eth1 and Eth2 dedicated for Management 1GB of throughput Control dedicated interfaces Eth3 and Eth4 dedicated for Control 1GB of throughput Data gets dedicated interfaces Eth5 and Eth6 dedicated for Data 1GB of throughput Best for solutions where Control and Data need dedicated links 110

111 Topology 5 Flexible Assign a link explicitly to a VSB links per VSB interface Share links across VSBs Create port-channels Specific to a VSB Shared to multiple VSBs 111

112 Recommendations If you are not planning on using vnam Topology 3 gives best bandwidth and redundancy for control VLAN Negative is that is harder to configure Topology 4 would be be next best scenario Dedicated NICs but no LACP Flexible allows any configuration Recommend port-channels Remember VSM latency is key over bandwidth Use VPC or VSS upstream if you have it 112

113 Wrap Up

114 N1K Public Links N1K Download and 60-day Eval: N1K Product Page: N1K Community: N1K Twitter N1K Webinars: N1K Case Studies: N1K Whitepapers N1K Deployment Guide: N1K on UCS Best Practices: VXLAN Web Conference: 114

115 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Don t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit 115

116 Final Thoughts Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042 Come see demos of many key solutions and products in the main Cisco booth 2924 Visit after the event for updated PDFs, on-demand session videos, networking, and more! Follow Cisco Live! using social media: Facebook: Twitter: LinkedIn Group: 116

117

118 Appendix

119 L2 Control VEM VSM Troubleshooting Steps 1. VSM MAC address 2. VSM is connected to vcenter 3. VSM has Control VLAN on right interface 4. Uplink port-profile has Control vlan 5. VEM sees control VLAN 6. VEM and VSM see each others MAC 7. Physical network sees VEM and VSM MAC 8. VSM sees heartbeat messages from VEM 119

120 Step 1: VSM MAC Need for L2 troubleshooting On VSM run show svs neighbors Its the AIPC Interface MAC n1kv-l2# show svs neighbors Active Domain ID: 422 AIPC Interface MAC: a Inband Interface MAC: a

121 Step 2: VSM vcenter Connectivity Verify VSM is connected to vcenter n1kv-l2# show svs connections connection VC: ip address: remote port: 80 protocol: vmware-vim https certificate: default datacenter name: Harrington admin: max-ports: 8192 DVS uuid: 3e ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e config status: Enabled operational status: Connected 121

122 Step 3: Verify VSM VM Control interface 1 st interface listed is Control Interface Interface connected? 122

123 Step 4: Verify Uplink Port-Profile The first ESX interface added to the N1KV must have Control VLAN Verify uplink port-profile has Control VLAN defined and system VLAN n1kv-l2# show run port-profile uplink version 4.2(1)SV1(5.1) port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan , no shutdown system vlan 2 state enabled 123

124 Step 5: Verify VEM Sees Control VLAN Verify VEM sees control VLAN with commands vemcmd show card vemcmd show port vemcmd show trunk 124

125 Vemcmd show card Control, packet vlans and domain-id match with VSM [~ # vemcmd show card Card UUID type 2: e Card name: cae-esx-154 Switch name: n1kv-l2 Switch alias: DvsPortset-0 Switch uuid: 3e ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e Card domain: 422 Card slot: 5 VEM Tunnel Mode: L2 Mode VEM Control (AIPC) MAC: 00:02:3d:11:a6:04 VEM Packet (Inband) MAC: 00:02:3d:21:a6:04 VEM Control Agent (DPA) MAC: 00:02:3d:41:a6: Card control VLAN: 2 Card packet VLAN: 2 MAC the VSM should learn for VEM 125

126 Vemcmd show port-old Ports with LTLs 8, 9,10 are UP and CBL states are 1. ESX Physical ports are UP and CBL states 1. ~ # vemcmd show port-old LTL IfIndex Vlan/ Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name SegId T VIRT UP UP 1 Trunk vns VIRT UP UP 1 Access VIRT UP UP 1 Access VIRT UP UP 1 Access VIRT UP UP 1 Access VIRT UP UP 1 Access VIRT UP UP 0 Access VIRT UP UP 1 Access VIRT UP UP 1 Access T VIRT UP UP 1 Trunk ar T PHYS UP UP 1 Trunk vmnic0 Local Target Logic (LTL) is an index to address a port, or group of ports. Data path lookup engine takes LTL as input, and gives LTL as output. LTL scheme: [0-14: internal ports] [15-271: pnics,vms, etc ] 126

127 Vemcmd show trunk Control and packet are CBL states 1 on the physical ports. ~ # vemcmd show trunk Trunk port 6 native_vlan 1 CBL 1 vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1, vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1, vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1, Trunk port 16 native_vlan 1 CBL 1 vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1, vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1, vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1, Trunk port 17 native_vlan 1 CBL 1 vlan(1) cbl 1, vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1, vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1, vemcmd show port vlans ~ # vemcmd show port vlans Native VLAN Allowed LTL VSM Port Mode VLAN State Vlans 17 Eth5/1 T 1 FWD 2,10-11, ~ # 127

128 Step 6: VEM and VSM See Each Other s MAC Is the VEM learning the MAC of the VSM? On VEM vemcmd show l2 <control-vlan> do you see the mac of the VSM? ~ # vemcmd show l2 2 Bridge domain 9 brtmax 4096, brtcnt 32, timeout 300 VLAN 2, swbd 2, "" Flags: P - PVLAN S - Secure D - Drop Type MAC Address LTL timeout Flags PVLAN Static 00:02:3d:21:a6: Dynamic 00:50:56:a9:25:

129 VEM and VSM See Each Other s MAC Is the VSM learning the MAC of the VEM? n1kv-l2# show mac address-table vlan 2 VLAN MAC Address Type Age Port Mod d21.a604 static 0 N1KV Internal Port d41.a604 static 0 N1KV Internal Port 5 129

130 Step 7: Physical Switch Mac Table Check the physical switch MAC address table Are the MACs of the VEM and VSM getting learned by the physical switches in the right VLANs? cae-cat6k-1#show mac-address-table vlan 2 Legend: * - primary entry age - seconds since last seen n/a - not available vlan mac address type learn age ports * dynamic Yes 360 Gi3/48 * a dynamic Yes 0 Gi4/9 * static Yes - Switch,Stby-Switch * d41.a604 dynamic Yes 0 Gi1/4 130

131 Step 8: VEM VSM Heartbeat One Heartbeat per second per VEM from VSM Timeout for VEM from VSM is 6 seconds of missed heartbeats After 6 seconds VSM will drop VEM Use vempkt capture to view heartbeats SPAN physical switch ports for heartbeats 131

132 Assigning a Veth to a particular VM By default veths are assigned in order To specify do the following Make sure the veth you want to use is free Make sure old veth is down with reason nonparticipating Can be done by disconnecting vnic or powering down VM Get veth dvs port # with show int veth # Remove vmware dvport config line on the old veth switch(config)# interface veth1 switch(config-if)# no vmware dvport 32 Create the veth and add dvport switch(config)# interface veth100 switch(config-if)# vmware dvport 32 Power on VM or connect vnic 132

133 VEM Removal VEM can only be removed manually Removing a host from the Nexus 1000V using vsphere does not remove the VEM ESX/ESXi 4/4.1 vem-remove s r Will unload and remove the driver ESXi 5 Host has to be in Maintenance Mode esxcli software vib remove -n cisco-vem-v140-esx 133

134 Port Channels vpc HM CDP/Manual vpc-hm uses Service Group (SG) Service Group is a collection of Ethernet interfaces from ESX host One Service Group per physical path CDP is used to determine SG membership Can be a 60 second delay while VSM determines NIC membership because of CDP Can configure SG membership manually for switches without CDP support Multiple links per physical path must be configured as a port-channel upstream 134

135 iscsi Support iscsi is supported Provide automatic multipathing capability for veth type port-profiles capability iscsi-multipath When turned on it attempts to balance connections across uplinks Works on a per vlan basis So access to iscsi storage paths has to be via the same vlan One path cannot be vlan 10 and the other vlan 11 Verify with vemcmd show iscsi pinning Vemcmd set command to change pinning vemcmd set iscsi pinning <vmk-ltl> <vmnic-ltl> 135

136 Upgrades As of Support for ISSU Order is now VSMs First VEMs Second Futures Allow 2 version jumps Better integration with VUM to allow ESX/ESXi and N1KV upgrades 136

137 VSM and VEM Upgrades VSM Use the install all command on the VSM Upgrades one VSM, does a switchover, upgrades second VSM Requires a temporary IP address One needed for mgmt 0 and control 0 (if used) VEM Use the vmware vem upgrade notify/proceed commands Using VUM the VSM can initiate VEM upgrades Hosts will go into maintenance mode, upgrade, exit maintenance mode Manual method you individually update each ESX/ESXi host 137

138 VXLAN Other Useful Commands Verify Multicast Upstream Nexus 7K/5K Verify querier is enabled for vlan VMK interfaces are on switch# show run vlan configuration 634 ip igmp snooping querier

139 VXLAN Other Useful Commands Verify IGMP snooping is configured CWD # show ip igmp snooping vlan 634 IGMP Snooping information for vlan 634 IGMP snooping enabled Optimised Multicast Flood (OMF) enabled IGMP querier present, address: , version: 3, i/f Po1 Verify multicast IP address for the VXLAN is being learned CWD # show ip igmp snooping groups vlan 634 Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port Querier interval: 125 secs Querier last member query interval: 1 secs Vlan Group Address Ver Type Port list Querier robustness: 2 Switch-querier enabled, address , currently running */* - R Po v2 D Po100 Eth2/30 IGMPv3 Report suppression disabled Link Local Groups suppression disabled Router port detection using PIM Hellos, IGMP Queries Number of router-ports: 1 Number of groups: 1 VLAN vpc function enabled Active ports: Po1 Po9 Po17 Po25 Po31 Po52 Po100 Eth2/30 139

140 VXLAN Other Useful Commands vemcmd show port Will show ports that are on a vxlan ~ # vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 17 Eth4/1 UP UP F/B* vmnic0 18 Eth4/2 UP UP F/B* vmnic1 49 Veth2 UP UP FWD 0 0 vmk Veth19 UP UP FWD 0 vse-vcdni-6-26-vl634-backed (b6 54 Veth16 UP UP FWD 0 0 vse-vcdni-6-26-vl634-backed (b Veth21 UP UP FWD 0 vcdni-2 (5ac7d73c-d1d ef 69 Veth22 UP UP FWD 0 0 vmk1 vxlan 305 Po2 UP UP F/B* 0 140

141 VXLAN Other Useful Commands vemcmd show igmp <vlan> Verify that multicast is enabled ~ # vemcmd show igmp 634 IGMP is ENABLED on VLAN 634 Multicast Group Table: Group */*, Multicast LTL: 4410 vemcmd show l2 segment <segment-id> Verify the VEM is learning MAC addresses in the VXLAN ~ # vemcmd show l2 segment 6000 Bridge domain 23 brtmax 4096, brtcnt 3, timeout 300 Segment ID 6000, swbd 4096, "dvs.vcdvsvcdni-6-26-vl634-backed-b69c1d1d-02bf b7e-fa06c64e8c18" Flags: P - PVLAN S - Secure D - Drop Type MAC Address LTL timeout Flags PVLAN Remote IP Static 00:50:56:01:02:0a Dynamic 00:50:56:01:02: Static 00:50:56:01:02:

142 VXLAN Other Useful Commands vemcmd show vxlan-encap [ltl/mac] <ltl/mac address> Identify the traffic path a MAC or LTL will utilize ~ # vemcmd show vxlan-encap ltl 68 Encapsulation details for LTL 68 in BD "dvs.vcdvsvcdni-6-26-vl634-backed-b69c1d1d-02bf b7e-fa06c64e8c18": Source MAC: 00:50:56:01:02:0a Segment ID: 6000 Multicast Group IP: Encapsulating L2 LISP Interface LTL: 69 Encapsulating Source IP: Encapsulating Source MAC: 00:50:56:7e:0e:b6 Pinning of L2 LISP Interface to the Uplink: LTL IfIndex PC_LTL VSM_SGID Eff_SGID iscsi_ltl* Name 69 1c vmk1 142

143 VXLAN Other Useful Commands vemcmd show vlxan-stats all Show VXLAN traffic stats ~ # vemcmd show vxlan-stats all LTL Ucast Mcast Ucast Mcast Total Encaps Encaps Decaps Decaps Drops

144 VXLAN Load Balancing With LACP port-channel 5-tuple hash is used Use singe VMK VXLAN interface VEM does the hashing across all the links Remember to change load balancing to 5-tuple hashing On the upstream switch and on the VSM With VPC MAC Pinning Create a VMK VXLAN interface for each available uplink VEM will pin an interface to each available link The VEM will distribute the VM's flows between the vmknics based on a hash of the source MAC. 144

145 Nexus 1010/1010-X Network Ports It is important to know which ports are which on the appliance Picture of the back of the appliance Eth1 and Eth2 are the onboard LOM ports Eth3 Eth6 are on the expansion card The expansion card should always be in the left slot of the appliance Do not move the card to the right slot 145