Deploying and Troubleshooting the Nexus 1000v Virtual Switch on vsphere

Transcription

1

2 Deploying and Troubleshooting the Nexus 1000v Virtual Switch on vsphere Matthew Wronkowski Technical Leader Virtualization Services

3 Agenda Current N1K Releases and New Features Licensing Virtual Supervisor Module (VSM) & VEM VSM High Availability Upgrades Port-Profiles & Port Channels VXLAN Cisco Cloud Services Platform / Nexus1x10 3

4 Cisco Nexus 1000V Virtual Switch Build & Price 4

5 Cisco Virtual Networking and Cloud Network Services Virtualized/Cloud Data Center WAN Switches Router Servers Imperva Cloud SecureSphere Services Router 1000V WAF Citrix NetScaler VPX Cloud Network Services vwaas Network Analysis Module (vnam) ASA 1000V Cloud Firewall Cisco Virtual Security Gateway Tenant A Zone A Physical Infrastructure vpath VXLAN Nexus 1000V Zone B Multi-Hypervisor (VMware, Microsoft, Ubuntu, RedHat*) Nexus 1000V VSG ASA 1000V vwaas CSR 1k vnam Ecosystem Distributed Switch NX-OS consistency VM-level controls Zonebased FW Edge firewall, VPN Protocol Inspection WAN optimization Application traffic WAN GW Routing & VPN App Visibility (L2-L7) Citrix NetScaler Imperva Web FW 5

6 Name a feature we will not implement on Nexus 1000V. Saravan Rajendran, Cisco CNSG VP 6

7 Current Releases and New Features

8 Current Nexus 1000V Releases ESX 5.2(1)SV3(1.1)* 256 VEMs, 12K veth count VXLAN 2.0 (BGP) N1K Management Center ESX 4.2(1)SV2(2.2) Dynamic Fabric Automation Leaf VDP VSI Discovery Protocol Universal Licensing ESX - 4.2(1)SV2(2.1a) Scalability Release 128 VEMs VXLAN 1.5, VXLAN GW Geographically Separated VSMs Removed ESX 4.1 support Hyper-V 5.2(1)SM1(5.2a) SCVMM 2012 SP1 & R2 Windows Server 2012 & R2 VSG VM and Custom Attributes Universal Licensing InterCloud 5.2(1)IC1(1.2) Simplified Platform Image Local License Server or Cisco PNSC Ubuntu KVM / OpenStack Initial Release *Next Release 8

9 Evolution of VXLAN to version 1.5 Unicast mode Simplifies VXLAN deployment Reduces network dependency (no multicast) Easier troubleshooting Flood directly to VXLAN Tunnel End Points (VTEP) Unicast Mac-address Distribution Mode Flooding is eliminated VSM learns all MACs and programs mappings to VEMs Faster response time Will not support VXLAN veth trunking(multi-mac) Requires static MACs (won t work with MS NLB) 9

10 vtracker Feature Provides intuitive virtualization perspective to the network-admin Pulls data from vcenter and VEM Gives cloud view of connected objects Enabled with feature vtracker There are 5 view options module-view upstream-view vlan-view vm-view vmotion-view SV2# show vtracker vm-view info vm win3 Module 5: VM Name: win3 Guest Os: Microsoft Windows Server 2003 Standard (32-bit) Power State: Powered On VM Uuid: 423ca4df-26d0-50c1-d531-1a49b3a83aed Virtual CPU Allocated: 1 CPU Usage: 0 % Memory Allocated: 1024 MB Memory Usage: 7 % VM FT State: Unknown Tools Running status: Running Tools Version status: current Data Store: datastore1 (2) VM Uptime: 25 days 3 hours 56 minutes 15s 10

11 Nexus 1000V Manager Installation Screenshot Install / Migrate / Upgrade / Monitor Zero CLI full GUI interface Auto Host Selection Deploy Redundant VSMs Best Practices Auto-Implemented Automated prompts with suggestion for alternatives Customize Installation for Advanced Users *Available Summer

12 Licensing Info

13 Licensing Essential Edition (No Expiration) Default mode for New Installs All features except Cisco TrustSec (CTS) DHCP Snooping IP Source Guard / Dynamic ARP Inspection Virtual Security Gateway (VSG) VXLAN Gateway 128 modules with 4096 virtual ports Support Options Pay Nothing support is through the communities site off cisco.com Pay for service contract 13

14 Licensing Advanced Edition For customers that want more security features Customers with existing licenses will be considered Advanced Upgrade process will migrate VSM to Advanced Edition Required for VXLAN Gateway and VSG Licensed customers can get Virtual Security Gateway(VSG) for free Cisco Account Team can submit request VSG will no longer be sold separately 256 modules with 12k virtual ports (SV3)* 60-day Trial after which Advanced FeatureSet is disabled 14

15 Universal Licensing A common license is shared for both N1k & VSG. Cross Hypervisor portability. The license name is NEXUS1000V_LAN_SERVICES_PKG. Following upgrade, request a new Permanent license within 60 days. 15

16 Licensing New Commands Display Current Edition switch# show switch edition To switch between Essential or Advanced switch(config)# svs switch edition [essential advanced] VEM Licenses are Sticky Removed & Offline VEMs hold a license switch# show module vem license-info Licenses are Sticky Mod Socket Count License Usage License Version License Status licensed VEM license transfer to pool: switch(config)# svs license transfer src-vem <module> license_pool 16

17 Licensing Overdraft Licenses Extra licenses to use in temporary situations 16 extra sockets Sometimes more depending on number of licenses you ve purchased Can only be used after a valid license is installed No penalty Full TAC Support for Overdraft Modules SV2# show license usage NEXUS1000V_LAN_SERVICES_PKG Feature Usage Info Installed Licenses : 16 Default Eval Licenses : 0 Max Overdraft Licenses : 16 <---- Installed Licenses in Use : 12 Overdraft Licenses in Use : 0 <---- Default Eval Lic in Use : 0 Default Eval days left : 0 Licenses Available : 20 < Shortest Expiry : 04 Feb

18 Virtual Supervisor Module Deployment and Troubleshooting

19 Back Plane Cisco Nexus 1000V Architecture Network Admin Virtual Appliance VSM-1 (active) VSM-2 (standby) NX-OS Control Plane Supervisor-1 (Active) Supervisor-2 (StandBy) Linecard-1 Linecard-2 Linecard-N NX-OS Data Plane Modular Switch VEM-1 VEM-2 VEM-N VSM: Virtual Supervisor Module VEM: Virtual Ethernet Module Server Admin Hypervisor Hypervisor Hypervisor 19

20 Virtual Supervisor Module (VSM) VSM is a Virtual Machine On ESXi, Hyper-V, Ubuntu KVM / OpenStack On Nexus 1x10 / Cloud Services Platform Control plane for the Nexus 1000V solution VEM packet forwarding not impacted by reloads Responsible for Programming and Managing Virtual Ethernet Modules (VEM) Communicating with Management Applications VMware vcenter, SCVMM, Horizon Dashboard 1 VSM HA pair can manage 128 VEMs Coexist with VMware vswitch, vds, Microsoft Logical, Native Switches 20

21 Nexus 1000V VSM Interfaces Control L2/L3 VEM (AIPC) VSM-VEM Heartbeats (L2/L3) VSM-VSM Synchronization (L2) VSM-VSM HA Heartbeats (L2/L3) Packet CDP, IGMP, NetFlow, SNMP L3 Mode Collapsed Ctrl, Pkt into mgmt0 VSM-VEM flow from mgmt0 Dedicated Control: svs mode L3 interface [control mgmt0] Management SSH console access SNMP, HTTP vcenter Communication HA Heartbeat Backup Interface Order is always the same! VSM-P eth0: control eth1: mgmt0 eth2: packet 21

22 VSM Deployment Scenarios Supports the VSM on a VEM Supports the VSM on any hypervisor native, logical, or distributed switch Supports the VSM on any supported hypervisor (ESXi/Hyper-V/N1110) Keep VSMs on different physical hosts Use anti-affinity rules Storage wise we don t care. VSM can be hosted on network storage 22

23 Stretched Nexus 1000V Model VSMs and VEMs spread across Datacenters VSMs can be split across DCs Requires L2 connectivity across DCI 10ms latency across DCI Not supported with Hyper-V Supported in a future release VM VM VM VM VM VM VEM-3 VEM-4 Local DC hypervisor hypervisor Remote DC DCI VSM hypervisor VEM-1 hypervisor VEM-2 VM VM VM VM VM VM VSM 23

24 VSM Control Modes L3 Mode L3 is the recommended & default Easier to troubleshoot Flexible Requires an IP address be assigned to the VEM Uses UDP4785 for both source and destination Sourced from mgmt0 by default L2 mode Requires L2 connectivity through control0 interface to all VEM modules L2 still supported on ESX Not supported with Hyper-V or KVM 24

25 VSM L3 Configuration and Planning Two options for the L3 control interface mgmt0 (default) control0 Use Control0 to separate control and management traffic Mgmt and Control use different VRF mgmt0 uses VRF management control0 uses VRF default Primary and Secondary VSM still need to be L2 adjacent! Test with mping broadcast command. 0x201 is control between VSMs # mping broadcast 64 bytes from node 0x0201 (msg id = 0x030b1e 1) (time=0 sec, 1510 usec) 25

26 VSM Connectivity to VMware vcenter VSM connects to vcenter using SSL connection VC Extension contains the SSL cert Unique extension ID for the VSM Ability to generate own certificates VSM talks to vcenter using its API We push and pull data to/from vcenter VSMs get tied to a VMware Datacenter Multiple VSMs tied to same DC is allowed VSM can manage across clusters but not datacenters Can get confusing 26

27 VSM Connectivity Errors - ESXi If you get Extension key was not registered before it s use Re-register the Extension Key with VMware vcenter If you get Connection refused. connect failed in tcp_connect() Ping vcenter IP from VSM CLI VMware admin could have changed the http port API communication is through port 80 with VMware vcenter Find new port and change it on VSM 27

28 VSM and vmotion/live Migration Manual vmotion/live Migration is supported VMware DRS is NOT recommended for Primary & Secondary VSMs Aggressive settings could lead to excessive VSM-VEM heartbeat packet drops Best practice to keep Primary and Secondary VSM outside DRS control Use anti-affinity rules where possible 28

29 Backing up the VSM A running-config is not enough to restore VSM on ESXi Clone to a template You can restore from a template and saved-config Must be powered down VSM on Nexus 1x10 Export a VSM to a file Import the saved VSM to restore VSM on ESXi Snapshots Not officially supported I/O latency cost associated with expanding the differential file 29

30 VSM Best Practices - Summary L3 control is the preferred method Use mgmt0 for control traffic Primary and Standby VSM in same L2 domain!!! Required even if VSMs are split between datacenters VSM on VEM is supported 10ms Latency between components: VSM-VSM, VSM-VEM 10ms even for VSMs split between datacenters For VEMs at branch locations 100ms Backup your config!!! 30

31 Nexus 1000V High Availability

32 VSM Redundancy Manager HA had to evolve to support split datacenter VSMs New Redundancy Manager process polls: VEM Manager polls for number of active VEMs attached to VSM VMS process retrieves which VSM has active VC connectivity SNMP Library gets the last configuration time Runs on both primary and secondary VSM Heartbeats VSM-VSM every second. Drop after 6 missed VSM-VEM every second. Drop after 15 missed SV2# show system internal redundancy trace 1 0s START_THREAD ST_NP ST_NP ST_INVALID 2 0s CP_STATUS_CHG ST_INIT ST_NP ST_INIT 3 0s SET_VER_RCVD ST_INIT ST_NP ST_INIT 4 0s STATE_TRANS ST_INIT ST_INIT ST_INIT EV_OS_INIT ST_AC_INIT 5 0s CP_STATUS_CHG ST_AC ST_INIT ST_AC_INIT 6 0s STATE_TRANS ST_AC ST_SB ST_AC_INIT EV_OS_SB ST_AC_SB 32

33 VSM Split Brain Recovery for ESXi Redundancy Manager in SV2(2.2) Module Count vcenter Status Last Configuration Time Last Standby-Active Switch(VSM with longer primary active time) Out-of-Sync / Split-Brain causes VSM to reload 33

34 When does a VEM switch VSMs? What if we have two active VSMs? What causes a VEM to switch? Standby VSM becomes active and broadcasts to all VEMs VEM will attach depending on Connectivity between VEM and VSM VEM receives the request to switch VEM goes into headless mode after 15 seconds If a VEM is headless traffic forwarding continues! vmotion/live Migration is blocked 34

35 Upgrades

36 Upgrades First always read and follow the upgrade guides Go in order Take a backup of the VSMs On ESXi use the clone to template option On Nexus 1x10s use the export function Backup the running-config Generate a Tech-Support before the upgrade If something goes wrong STOP and call TAC Use a maintenance window VEM upgrades require ESXi hosts to be in Maintenance Mode 36

37 Supported Upgrades Starting Version Combined VMware Upgrade Notes 1.3 Yes 1.4 first* 1.4 first* 1.4 first* No 1.4 Yes Yes Yes No 1.5 Yes Yes Yes 2.1 Yes Yes 1.4 last version supporting ESX for combined 2.1 last version supporting ESX 4.1 Upgrade matrix: * Must upgrade to 1.4b first 37

38 Upgrades to 2.2 Scalability limits may require changes to the VM settings For full scalability support: CPU reservation to 2GHz Memory to 3GB VSMs do NOT support multiple vcpus Steps Shutdown Secondary VSM Make VM changes Power Secondary on System Switchover Repeat steps on Primary VSM API can be upgraded individually now show plugin status 38

39 Upgrading the VSM Changes from 2.1 VSMs can run newer software than VEMs. New features disabled until VEMs upgraded. ISSU upgrade is similar to other Nexus switches Copy new kickstart and system images to bootflash Run install all command Verifies software compatibility Copies images to secondary s bootflash. Upgrade/Reboot the Secondary VSM Switchover to Secondary VSM It s now the active VSM with VEMs attached Upgrade/Reboot the old-primary VSM Requires no outage of the VSM Change CPU/Memory after the SV2(2.2) upgrade is complete 39

40 Troubleshooting VSM Upgrades If something is wrong after the VSM upgrade STOP Call TAC Rollback using backup method Shutdown the VSM VMs Power-on the Clones (ESXi), Import the backup (Nexus 1x10) Changing boot variables to older image is not supported but often works Sometimes the VEM won t connect to the Standby VSM Try a system switchover once the old primary is upgraded Might want to verify Standby VSM before upgrade Make sure VEMs can connect to standby Use system switchover command 40

41 Upgrading the VEMs VEM module upgrade kicked off on VSM If VUM is installed everything is automatic VSM communicates with vcenter to manage the upgrade Host is placed in maintenance mode(if DRS is installed VMs are migrated off) VEM is upgraded and host exits maintenance mode Moves on to the next host If VUM is not installed Still initiate the process on the VSM User manually places ESXi hosts in maintenance mode Upgrade the VEM with esxcli command Exit maintenance mode and move to the next host Always complete the upgrade Issue the vmware vem upgrade complete command Signals vcenter to use the new VEM VIB when hosts are added 41

42 Troubleshooting VEM Upgrades Remember the VMware admin has to acknowledge upgrade in vcenter Don t upgrade the VEMs by pushing a baseline Make sure you have DRS capacity Need to be able to handle one ESXi host in maintenance mode If a particular ESXi host fails It s usually because the host cannot go into maintenance mode From vcenter attempt to put the host in maintenance mode Troubleshoot any issues that prevent it If an ESXi host is running a vcenter VM this can cause problems You can restart the VEM upgrade after it fails It will only upgrade hosts that did not succeed 42

43 Virtual Ethernet Module Deployment and Troubleshooting

44 VEM Deployment Best Practices Again we recommend L3 Control L3 control requires a VMKernel NIC on N1K DVS We need an L3 interface to forward control traffic 10/100ms latency for local vs. branch office Recommend using the ESXi management VMKernel NIC Requires management interface to the VEM Doesn t require static routes on ESXi hosts Don t create an L3 vmk on same subnet as mgmt vmk Don t use UCS Dynamic vnics in Service-Profiles VEM and VM-FEX are mutually exclusive 44

45 VEM Deployment veth Port-Profile vmk0 interface needs to be migrated to this port-profile It must have capability l3control and system VLAN Each VMKernel VLAN needs a different port-profile VSM only permits VMKs to connect to this port-profile port-profile type vethernet vmk-l3 capability l3control vmware port-group switchport mode access switchport access vlan 119 capability vxlan no shutdown system vlan 119 state enabled 45

46 VEM Deployment Uplink Port-Profile Typically a trunk Verify upstream switch allowed VLAN list matches Must have system vlans & a port-channel defined MTU must match. Especially important when using OTV. port-profile type ethernet system-uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 119,199,219,319 mtu 9000 channel-group auto mode on mac-pinning no shutdown system vlan 119,319 state enabled 46

47 VEM L3 Troubleshooting 1. VMK migrated behind VEM? 2. VSM-ESXi connectivity? Static route needed? 3. L3 veth Port-Profile correct? 4. Uplink Port-Profile correct? 5. Check the Opaque Data 6. Check Heartbeats 47

48 VEM Troubleshooting VSM Connectivity VEM adds in vcenter but does not show up on VSM show module With L3 its usually an IP routing problem If you can ping from VSM to VMK interface then VEM should connect. Troubleshoot as you would all VMware L3 issues With L2 most of the time its a Control VLAN issue Verify Control VLAN connectivity in upstream network Check upstream switches for VEM AIPC MAC address Additional Information in Appendix 2 48

49 VEM Deployment VMKs on same subnet Don t use multiple VMKs on the same subnet on different virtual switches VMware uses a single TCP/IP stack for all VMK interfaces No way to pin traffic to an uplink interface. One interface gets picked for all traffic on that subnet Check out VMware KB article Only one gateway per host VMK VEM-1 VMware ESX VMK vswitch 50

50 VSM Setting Verification Verify the VRF SV2# show ip route vrf management /0, ubest/mbest: 1/0 *via , mgmt0, [1/0], 6d20h, static Can the VSM ping the VEM Check SVS domain SV2# sh svs domain SVS domain config: Domain id: 1919 Control vlan: NA Packet vlan: L2/L3 Control mode: L3 L3 control interface: mgmt0 Status: Config push to SV2# ping VC successful. PING ( ): 56 data bytes 64 bytes from : icmp_seq=0 ttl=62 time=1.254 ms 64 bytes from : icmp_seq=1 ttl=62 time=1.057 ms 64 bytes from : icmp_seq=2 ttl=62 time=1.055 ms NA 52

51 Check Opaque Data Opaque data is bootstrap information for the VEM Pushed via SCVMM or vcenter during Host Add to DVS Is the right Opaque data getting pushed to the ESXi host? ~ # vemcmd show card Card UUID type 2: 9aed7c30-84f8-11e ff f Card name: Switch name: SV2 Switch alias: DvsPortset-0 Switch uuid: b2 40 3c e 15 f5-6a 3c 7f d1 c cd Card domain: 1919 Card slot: 3 VEM Tunnel Mode: L3 Mode L3 Ctrl Index: 49 L3 Ctrl VLAN: 119 VEM Control (AIPC) MAC: 00:02:3d:17:7f:02 VEM Packet (Inband) MAC: 00:02:3d:27:7f:02 VEM Control Agent (DPA) MAC: 00:02:3d:47:7f:02 VEM SPAN MAC: 00:02:3d:37:7f:02 Primary VSM MAC : 00:02:3d:70:1f:07 Primary VSM PKT MAC : 00:02:3d:70:1f:08 Primary VSM MGMT MAC : 00:02:3d:70:1f:06 53 Should match VLAN defined in veth Port-Profile Should match MAC of control 0 or mgmt 0

52 View Heartbeat Messages on VEM Use vempkt on the ESXi host vempkt capture [egress ingress] vlan 119 ltl 50 Run for 10s to capture several heartbeat cycles vempkt cancel capture all vempkt display detail all vempkt can now export to a pcap file vempkt pcap export <filename> Look for heartbeat messages on VSM SV2# show module vem counters Mod InNR OutMI InMI OutHBeats InHBeats InsCnt RemCnt Crit Tx Errs

53 VEM Troubleshooting - vemlog Used for detailed debugging of programming and packet flows Executed on the Hypervisor Host Enable different debug options to help troubleshoot LACP QOS VXLAN IGMP VSM<-->VEM Data ~ # vemlog show debug grep lacp Module Available Printing sflacp ENWID PL (223) ( 0) sf_lacp_pdu_utils ENWID PL (223) ( 0) sflacp_hostdata ENWID PL (223) ( 0) ~ # vemlog debug sflacp all ~ # vemlog show debug grep lacp sflacp ENWID PL (223) ENWIDTPL (255) sf_lacp_pdu_utils ENWID PL (223) ( 0) sflacp_hostdata ENWID PL (223) ( 0) ed119.shtml 55

54 Port-Profiles Deploying and Troubleshooting

55 Port-Profiles Port-Profile Port-profiles <type> vethernet Ethernet Usage VM vmk l3control / vservice UPLINK vethernet PP (default) -Virtual Interfaces (veth x/) (VMs, VMK) -Typically Access Ports -Configuration: VLAN, ACL, Pinning, QoS Ethernet PP -Physical Interfaces (Eth x/y) -Typically Trunk (could also be access) -Configuration: Port-Channel, ACLs, QoS 57

56 Switch Interface Types Ethernet Port (eth) Correspond to the physical NIC interfaces leaving the server Specific to each module or VEM VMware s vmnicx == Cisco ethx/y Up to 32 physical ports supported per host Port Channel (port-channel) Aggregation of physical Ethernet ports Up to eight Port Channels per host Virtual Ethernet Port (veth) One per virtual NIC interface (vnic) including service console / vmknic Notation is VethX No module number is assigned to keep naming persistent as VMs move between modules (hosts/vems) Eth3/1 Veth1 VM1 Po1 Eth3/2 VM2 Veth2 58

57 Loop Prevention without STP Cisco VEM Eth4/1 Cisco VEM Eth4/2 X Cisco VEM X VM1 VM2 VM3 VM4 BPDUs are Dropped VM5 VM6 VM7 VM7 No Switching from Physical NIC to NIC VM9 VM10 VM11 VM12 déjà vu check Frames with local MAC Dropped on Ingress 59

58 Spanning-tree and BPDU Best Practice Mandatory Spanning-Tree settings per port IOS set STP portfast cat65k-1(config-if)# spanning-tree portfast trunk NXOS set port type edge n5k-1(config-if)# spanning-tree port type edge trunk Highly Recommended Global BPDUFilter/BPDUGuard IOS cat65k(config)# spanning-tree portfast bpdufilter cat65k(config)# spanning-tree portfast bpduguard NXOS n5k-1(config)# spanning-tree port type edge bpduguard default n5k-1(config)# spanning-tree port type edge bpdufilter default BPDU Filter is mandatory for LACP port-channels Set per-port BPDU Guard when Global is not possible 60

59 Ethernet (uplink) Port-Profile Troubleshooting Port-Profiles with multiple NICs need a port-channel Causes duplicate packets Kicks in déjà vu driver Requires extra CPU processing Fills the logs When in doubt, use mac-pinning Also same issue if you overlap VLANs in different Port-Profiles on same host WRONG port-profile type ethernet uplink-nopc vmware port-group switchport mode trunk switchport trunk allowed vlan , no shutdown system vlan 11 state enabled RIGHT port-profile type ethernet uplink-nopc vmware port-group switchport mode trunk switchport trunk allowed vlan , channel-group auto mode on mac-pinning no shutdown system vlan 11 state enabled 61

60 Cisco Nexus 1000V System VLANs System VLANs enable interface connectivity before an interface is programmed System port-profiles become part of the opaque data VEM will load system port-profiles and pass traffic even if VSM is not up Unprotected (No ACLs, VSG) before module registers for first time Addresses chicken and egg issue VEM needs to be programmed, but it needs a working network for this to happen Port profiles that contain system VLANs are system port profiles Allowed 32 port-profiles with system VLAN 62

61 System VLAN Guidelines The system VLAN must be a subset of the allowed VLAN list on trunk ports Only one system VLAN on an access port The no system vlan command only when no interface is using the profile Once a system profile is in use by at least one interface Can add to the list of system VLANs Cannot delete VLANs from the list reason to limit usage System vlans must be set on egress and ingress port-profiles Required System VLANs Control, Packet, IP Storage, VMKernel, vcenter, any Management Networks 63

62 VMware DVS Max-Port Issues Default to 32 max-ports per port-profile Counts toward the maximum number of VMware DVS ports 8192 by default Pre-Provisioned Some ports are consumed when you add an ESX host to the DVS Two methods to remedy: Max-ports under svs connection <name> Allows you to increase the ports of the VMware DVS Port-binding auto expand in veth port-profiles N1KV dynamically adds ports as VMs are added Set port-binding as default with port-profile default port-binding static auto expand 64

63 Microsoft Network Load Balancing Support Unicast mode is officially supported method no mac auto-static-learn in veth port-profile Multicast Mode NLB virtual cluster address requires a static ARP entry on the edge router Works through flooding Multicast Mode IGMP Disable IGMP snooping on the N1KV Upstream switches enable IGMP snooping Enable IGMP Querier in the environment NLB virtual cluster address requires a static ARP entry on the edge router CSCue Add support for Microsoft NLB - Multicast+IGMP 65

64 Jumbo Frames Support System jumbo mtu 9000 Enabled globally by default in SV1(4)+ Sets the systemwide jumbo MTU size Generally do not need to change vethernet ports are 9000 by default MTU setting for ethernet type port-profile Simply use mtu size in port-profile and nothing else Still need to configure upstream network devices UCS System QoS Class UCS vnic QoS Policy Nexus 5k / 7k / etc 66

65 Port-Profile Using Weighted QOS Configuration Steps to limit vmotion traffic n1kv-l3(config)# class-map type queuing match-all vmotion-class n1kv-l3(config-cmap-que)# match protocol? n1k_control N1K control traffic n1k_mgmt N1K management traffic n1k_packet N1K inband traffic vmw_ft VMware fault tolerance traffic vmw_iscsi VMware iscsi traffic vmw_mgmt VMware management traffic vmw_nfs VMware NFS traffic vmw_vmotion VMware vmotion traffic n1kv-l3(config-cmap-que)# match protocol vmw_vmotion n1kv-l3(config-cmap-que)# policy-map type queuing vmotion-policy n1kv-l3(config-pmap-que)# class type queuing vmotion-class n1kv-l3(config-pmap-c-que)# bandwidth percent 50 n1kv-l3(config)# port-profile type eth uplink-vpc n1kv-l3(config-port-prof)# service-policy type queuing output vmotion-policy 68

66 Port Channels

67 Port Channels LACP Port-Channels Requires upstream switch support and configuration VPC MAC Pinning Works with any upstream switch Allows for pinning of veths (VM) to specific links VPC Host Mode CDP/Manual (deprecated) NIC association is either Manual or CDP 70

68 Port Channels Best Practice Configuration Guide 080c1ee1e.shtml All Ethernet Port-Profiles must be configured in a Port-Channel LACP & MAC-Pinning are recommended modes Use Manual/Static Pin Group for granular traffic steering Use Manual/Static Pin Groups with multiple vmotion VMKs in ESX 5.x Same link-speed for all members. No mixing 1G+10GE+40GE interfaces. 71

69 Port Channels Best Practice If the upstream switch can be clustered (VPC, VBS Stack, VSS) use LACP If you are using LACP also use LACP Offload UCS-B must use MAC-Pinning If the upstream switch can NOT be clustered use MAC-PINNING Create channel-groups in port-profile Let VSM build the interface port-channel & add physical NICs All physical switch ports in port-channel configured identical 72

70 Port Channels MAC Pinning MAC Pinning provides the dynamism of vpc Host-Mode without requiring CDP to be configured on the upstream switch VM VM VM VM The VM MAC address is used to select link. sys-uplink vsphere port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 1-10 channel-group auto mode on mac-pinning no shut state enable system vlan 10 73

71 Port Channels MAC Pinning (Link Failure) If a failover occurs, all the traffic pinned to an interface will be migrated to the other interfaces. VEM sends GARP to flush upstream CAM tables. VM VM VM VM The VM MAC address is used to select link. sys-uplink vsphere port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 1-10 channel-group auto mode on mac-pinning no shut state enable system vlan 10 74

72 Port Channels MAC Pinning Use Network State Tracking (NST) to detect non-link failures Each Eth interface added is a unique Service Group SGID # assigned based off vmnic# Use pinning id command under vethernet Port-Profile Pins the VM to a particular uplink Ordered list for backup n1kv(config-port-prof)# pinning id 0 backup 1 2 Default assignment is Round Robin to an SGID New command to make SGID # relative n1kv(config-port-prof)# channel-group auto mode on mac-pinning relative 75

73 MAC Pinning (Host Pinning Tables) n1kv# sh port-channel summary 1 Po1(SU) Eth NONE Eth5/1(P) Eth5/2(P) 2 Po2(SU) Eth LACP Eth6/1(P) Eth6/2(P) 3 Po3(SD) Eth NONE Eth3/3(r) [root@mw-esx15 ~]# vemcmd show channel type LTL Channel_Type MAC Pinning 18 MAC Pinning 76

74 MAC Pinning (Host Pinning Tables) ~]# vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 17 Eth3/1 UP UP F/B* vmnic0 18 Eth3/2 UP UP F/B* vmnic1 49 Veth1 UP UP FWD 0 1 vmk0 [root@mw-esx15 ~]# vemcmd show pc pce_ind chan pc_ltl pce_in_pc LACP SG_ID NumVethsPinned mbrs N , 1* 3 18, [root@mw-esx15 ~]# vemcmd show pinning LTL IfIndex PC_LTL VSM_SGID Eff_SGID iscsi_ltl* Name c0000a vmk0 50 1c0000d vmk1 77

75 Port Channels How to Tell Pinning Can run from the VSM now No need to run command on the VEM n1kv-l3# show int virtual pinning module Veth Pinned Associated PO List of Sub Group id interface Eth interface(s) Veth2 0 Po5 Eth5/1 Veth4 2 Po5 Eth5/3 Veth5 0 Po5 Eth5/1 Veth6 2 Po5 Eth5/3 Veth7 0 Po5 Eth5/1 78

76 Static Pinning to Sub-Group Static Pinning is similar to VMware s vswitch active/standby design. port-profile type ethernet uplink channel-group auto mode on macpinning relative port-profile vmkernel pinning sub-group id 0 backup 2 1 port-profile vmkernel pinning sub-group id 0 backup 2 1 port-profile type ethernet vmotion pinning sub-group id 2 vmk0 VMotion vmk0 VMotion P C After failover P C Port-channel Sub-group 1 Port-channel Sub-group 1 Sub-group 0 Sub-group 2 Sub-group 2 79

77 LACP Port Channels Use when single upstream or clustered (vpc,vss, Catalyst Stack) switch Use channel-group auto mode active on N1KV Use channel-group # mode active/passive on upstream switch Switchports must be configured with spanning-tree portfast trunk spanning-tree bpdufilter enable Not compatible with Network State Tracking(NST) with LACP 80

78 Port-Channels - LACP VM VM VM VM vsphere LACP Port-channel Upstream switch clustered (vpc,vss,vbs,stack ) LACP allows traffic from each VM to fully utilize multiple links simultaneously. Allows faster VMotion and faster VM connectivity by using flow based hashing port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 1-10 channel-group auto mode active no shut state enable 81

79 LACP Troubleshooting Do not use Network State Tracking(NST) with LACP LACP Port-Channel configured on the upstream switches Port-profile created with channel-group auto mode active On the VEM vemcmd show lacp On the VSM and Upstream Switch show port-channel summary show lacp counters/neighbor Are you seeing LACP PDUs? 82

80 LACP Debugging ~ # vemcmd show lacp LACP Offload is Enabled LACP Offload Config for LTL Channel No : 8 Channel Mode : Active Port Priority : 0x8000 LACP Bit Set : Yes SV2# show lacp counters LACPDUs Marker Marker Response LACPDUs Port Sent Recv Sent Recv Sent Recv Pkts Err port-channel8 Ethernet10/ Ethernet10/

81 LACP Debugging ~ # vemlog show debug grep lacp sflacp ENWID P ( 95) ENW ( 7) sf_lacp_pdu_utils ENWID P ( 95) ENW ( 7) sflacp_hostdata ENWID P ( 95) ENW ( 7) Debug (LTL 16, DIR TX) : Actorstate=7 agg=1 insync=0 coll=0 dis=0 active=1 short_timeout=1 Port ID (0x8000.0x602), Key (7) Debug (LTL 16, DIR TX) :Partnerstate=2 agg=0 insync=0 coll=0 dis=0 active=0 short_timeout=1 Port ID (0x0.0x0), Key (0) Debug sf_lacp_tx_pdu_to_upstream: LTL = 18 Debug sf_lacp_tx_pdu_to_upstream, NEW LACP PKT : Src(1), Dst(18), VLAN(1), FLAGS(1) [ ] Debug (LTL 18, DIR RX) :Partnerstate=3d agg=1 insync=1 coll=1 dis=1 active=1 short_timeout=0 Port ID (0x8000.0x602), Key (7) Debug (LTL 16, DIR TX) : Actorstate=3d agg=1 insync=1 coll=1 dis=1 active=1 short_timeout=0 Port ID (0x8000.0x602), Key (7) 84

82 Virtual Extensible LAN (VXLAN)

83 Virtual Extensible Local Area Network (VXLAN) Ethernet in IP overlay network Entire L2 frame encapsulated in UDP (port 4789) 50 bytes of overhead Include 24-bit VXLAN Identifier 16 M logical networks Mapped into local bridge domains Unique multicast group per segment VXLAN can cross Layer 3 Tunnel between VEMs VMs do NOT see VXLAN ID Egress to Non-VXLAN network Outer MAC DA Outer MAC SA Outer 802.1Q Outer IP DA Outer IP SA Outer UDP VXLAN ID (24 bits) Inner MAC DA InnerM AC SA Optional Inner 802.1Q Original Ethernet Payload CRC VXLAN Encapsulation Original Ethernet Frame 87

84 Virtual Extensible Local Area Network (VXLAN) Each overlay network is known as a VXLAN segment Each VXLAN segment identified by a 24-bit segment ID (VNI) VXLAN traffic carried between VXLAN Tunnel Endpoints (VTEP) VEM module acts as the VTEP VM traffic is carried over point to point tunnels between VTEPs VM to VM traffic is encapsulated in a VXLAN header 1550 MTU for encapsulation overhead Encapsulated multicast is always flooded No IGMP in VXLAN 88

85 Deployment Modes: Multicast or Unicast? Multicast used to be required for unknown broadcast/unicast on VXLAN N1KV 2.2 introduced Unicast Mode and Unicast Mac Distribution Mode Multicast (VXLAN 1.0) Needs Multicast configured throughout complete network IGMP Querier in VLAN Multicast routing and proxy ARP across subnets VTEPs all join multicast group Interoperates with N9K, CSR1K, other Nexus products Unicast Mode (VXLAN 1.5) VEMs flood each other directly for unknown broadcast/unicast Keep a list of other VEMs in each VXLAN 89

86 Deployment Modes: When to use MAC Distribution? MAC distribution will provide best performance No Flooding & Learning Full MAC table distributed to each VEM VEMs report local MACs to VSM VSM distributes {MAC,VTEP} mapping to each VEM VXLAN traffic cannot span multiple Nexus 1000V switches* Two caveats No veth VXLAN trunk mode support with MAC distribution Won t work with Microsoft NLB 90

87 VXLAN Forwarding Basics Forwarding mechanisms similar to Layer 2 bridge: Flood & Learn VEM learns VM s Source (MAC, Host VXLAN IP) tuple Broadcast, Multicast, and Unknown Unicast Traffic VM broadcast & unknown unicast traffic are sent as multicast Unicast Traffic Unicast packets are encapsulated and sent directly (not via multicast) to destination host VXLAN IP (Destination VEM) VM VM VM VM VEM 1 VEM 2 92

88 Enhanced VXLAN VXLAN (multicast mode) Enhanced VXLAN (unicast mode) Enhanced VXLAN MAC Distribution Enhanced VXLAN ARP Termination Broadcast / Multicast Multicast Encapsulation Replication plus Unicast Encap Replication plus Unicast Encap Replication plus Unicast Encap Unknown Unicast Multicast Encapsulation Replication plus Unicast Encap Drop Drop Known Unicast Unicast Encapsulation Unicast Encap Unicast Encap Unicast Encap ARP Unicast Encapsulation Replication plus Unicast Encap Replication plus Unicast Encap VEM ARP Reply 96

89 VXLAN Configuration: Unicast VMkernel interface acts as VTEP VSM Control Mode should be L3 Bridge domain is configured as Unicast or Unicast Mac Distribution feature segmentation feature vxlan-gateway port-profile type vethernet vmk-l3-vxlan-vtep capability l3control vmware port-group switchport mode access switchport access vlan 119 capability vxlan no shutdown system vlan 119 state enabled 97

90 Bridge Domain Configuration: Unicast Create a bridge-domain in unicast mode Scenario 1: Scenario 2: switch(config)# segment mode unicast-only (Global) switch(config)# bridge-domain segment-cisco switch(config-bd)# segment id 5000 switch(config-bd)# segment distribution mac switch(config)# bridge-domain segment-cisco switch(config-bd)# segment id 5000 switch(config-bd)# segment mode unicast-only (Per BD override) switch(config-bd)# segment distribution mac 98

91 Port-Profile Configuration Create an Access Port-Profile with the VXLAN Bridge Domain Assign to VM s in vcenter port-profile type vethernet bd-5000 vmware port-group switchport mode access switchport access bridge-domain bd-5000 no shutdown state enabled 99

92 VXLAN Debugging SV2# show bridge-domain bd-5000 Bridge-domain bd-5000 (2 ports in all) Segment ID: 5000 (Manual/Active) Mode: Unicast-only (override) MAC Distribution: Disable (override) Group IP: NULL State: UP Mac learning: Enabled Veth9, Veth45 SV2# show bridge-domain bd-5000 vteps Bridge-domain: bd-5000 VTEP Table Version: 21 Port Module VTEP-IP Address VTEP-Flags Veth (D) <---Designated VTEP (vmk) Veth (D) Veth (DI) <---VXGW Veth (DI*)<---VXGW (Standby) 100

93 VXLAN Debugging ~ # vemcmd show vxlan-vteps Bridge-Domain: bd-5000 Segment ID: 5000 Designated Remote VTEP IPs (*=forwarding publish incapable): (DSN: 1), (DSN: 1)* ~ # vemcmd show bd bd-name bd-5000 BD 31, vdc 1, segment id 5000, segment group IP , encap VXLAN, vff_mode Anycast,swbd 4096, VLAN 0, 1 ports, "bd-5000" Segment Mode: Unicast VTEP DSN: 1, MAC DSN: 0 Portlist: 52 win2k.eth0 Virtual Machine in VXLAN

94 VXLAN Debugging ~ # vemcmd show l2 segment 5000 Bridge domain 31 brtmax 4096, brtcnt 3, timeout 300 Segment ID 5000, swbd 4096, "bd-5000" Flags: P - PVLAN S - Secure D - Drop Type MAC Address LTL timeout PVLAN Remote IP DSN Dynamic 00:50:56:bc:73:1a Static 00:50:56:a9:00:2e Dynamic 54:7f:ee:2f:33: ESXi Host #2 VXLAN Gateway 102

95 Nexus 1010 and 1110

96 VSM Deployment Scenarios Nexus 1110 Cisco Cloud Services Platform VSM on a Nexus 1010/X or 1110-S/X It s still a Virtual Machine Up to 14 VSM pairs on one 1110-X cluster Always deploy in the appliance pairs! N110 allows for Network team to own the virtualization platform N110s should go in the Aggregation Layer Stretched Model requires L2 Connectivity 10ms latency 104 *Next Release

97 1110-S/X Deployment Scenario 105

98 Cisco Cloud Services Platform (CSP) Nexus 1010/1010-X/1110-S/1110-X Based off UCS C2x0 M3 server Same CIMC/BIOS/firmware Provide 6 x 1G network connections 1110-X 2 x 10G - SP1(7) 10G available only on purchase. No upgrade available. Encryption Accelerator Card for Citrix VPX SP1(7) Virtual Service Blade (VSB) Support 1010/1110-S supports up to /1110-X supports up to

99 Cisco Cloud Services Platform (CSP) Current supported VSBs Nexus 1000V VSM (ESX/HyperV/KVM) Virtual Security Gateway (VSG) Network Analysis Module (NAM) Data Center Network Manager (DCNM) Citrix NetScaler VPX VSB HyperV VXLAN GW Citrix Netscaler Minimum Version SP(6.1) SP1(6.1) SP1(6.2) 107

100 Cisco Cloud Services Platform (CSP) Must be deployed in pairs No option for standalone Deploy in the Aggregation Layer Must be in the same L2 domain for management and control Can be geographically diverse Uses same HA mechanism as VSM with domain-id and control vlan Do not overlap the domain-id between a 1x10 and a VSM What s not supported? Primary and Secondary VSM on same 1x10 Primary VSM on ESX and Secondary VSM on 1x10 or vice versa 108

101 VSB Backups using Import/Export Works with VSM, NAM, and VSG Can Import/Export both primary and secondary Export requires that VSB be shutdown Images are stored in export-import/ dir on bootflash Can be manually copied off to remote storage n1010-1# copy bootflash:export-import vrf management n1010-1(config)# virtual-service-blade training n1010-1(config-vsb-config)# import primary Vdisk4.img.tar.00 Note: import started.. Note: please be patient.. Note: Import cli returns check VSB status for completion 109

102 Network Classes and Topologies Management Carries the mgmt0 interface of the 1x10 Carries the mgmt0 traffic for all VSMs installed Control Carries all the control and packet traffic for the VSMs installed on the 1x10 Carries control traffic for HA between primary and secondary 1x10 Data Used by Virtual Service Blades other than VSM Passthrough Binds physical NIC to VSB 5 Network Topologies choices 110

103 Network Topologies Uplink Type Management VLAN Control VLAN Data VLAN 1 Ports 1 and 2 Ports 1 and 2 Ports 1 and 2 2 Ports 1 and 2 Ports 1 and 2 (HA) Ports 3-6 (LACP) 3 Ports 1 and 2 Ports 3-6 (LACP) Ports 3-6 (LACP) 4 Ports 1 and 2 Ports 3 and 4 Ports 5 and 6 Flexible There is no traffic segregation based on traffic class. *Must use for VXGW deployements. 111

104 Recommendations If you are not planning on using other VSBs Topology 3 gives best bandwidth and redundancy for control VLAN Negative is that is harder to configure If using VXGW, Netscaler, or shared between production / lab network Topology 5 is Flexible Flexible allows any configuration Recommend port-channels Remember VSM latency is key over bandwidth Use VPC or VSS upstream if you have it 112

105 Participate in the My Favorite Speaker Contest Promote Your Favorite Speaker and You Could be a Winner Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress) Send a tweet and include Your favorite speaker s Twitter Two hashtags: #CLUS #MyFavoriteSpeaker You can submit an entry for more than one of your favorite speakers Don t forget to View the official rules at 113

106 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online 114

107 Continue Your Education Demos in the Cisco Campus Walk-in Self-Paced Labs Table Topics Moscone Center West 3 rd Floor Lobby Discuss Experiences with Cisco Services with Distinguished Service Engineers Meet the Engineer 1:1 meetings 115

108

109

110 Appendix A L2 Troubleshooting

111 L2 Control VEM VSM Troubleshooting Steps 1. VSM MAC address 2. VSM is connected to vcenter 3. VSM has Control VLAN on right interface 4. Uplink port-profile has Control vlan 5. VEM sees control VLAN 6. VEM and VSM see each others MAC 7. Physical network sees VEM and VSM MAC 8. VSM sees heartbeat messages from VEM 119

112 Step 1: VSM MAC Need for L2 troubleshooting On VSM run show svs neighbors Its the AIPC Interface MAC n1kv-l2# show svs neighbors Active Domain ID: 422 AIPC Interface MAC: a Inband Interface MAC: a

113 Step 2: VSM vcenter Connectivity Verify VSM is connected to vcenter n1kv-l2# show svs connections connection VC: ip address: remote port: 80 protocol: vmware-vim https certificate: default datacenter name: Harrington admin: max-ports: 8192 DVS uuid: 3e ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e config status: Enabled operational status: Connected 121

114 Step 3: Verify VSM VM Control interface 1 st interface listed is Control Interface Interface connected? 122

115 Step 4: Verify Uplink Port-Profile The first ESX interface added to the N1KV must have Control VLAN Verify uplink port-profile has Control VLAN defined and system VLAN n1kv-l2# show run port-profile uplink version 4.2(1)SV1(5.1) port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan , no shutdown system vlan 2 state enabled 123

116 Step 5: Verify VEM Sees Control VLAN Verify VEM sees control VLAN with commands vemcmd show card vemcmd show port vemcmd show trunk 124

117 Vemcmd show card Control, packet vlans and domain-id match with VSM [~ # vemcmd show card Card UUID type 2: e Card name: cae-esx-154 Switch name: n1kv-l2 Switch alias: DvsPortset-0 Switch uuid: 3e ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e Card domain: 422 Card slot: 5 VEM Tunnel Mode: L2 Mode VEM Control (AIPC) MAC: 00:02:3d:11:a6:04 VEM Packet (Inband) MAC: 00:02:3d:21:a6:04 VEM Control Agent (DPA) MAC: 00:02:3d:41:a6:04.. MAC the VSM should learn for VEM.. Card control VLAN: 2 Card packet VLAN: 2 125

118 Vemcmd show port-old Ports with LTLs 8, 9,10 are UP and CBL states are 1. ESX Physical ports are UP and CBL states 1. ~ # vemcmd show port-old LTL IfIndex Vlan/ Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name SegId T VIRT UP UP 1 Trunk vns VIRT UP UP 1 Access VIRT UP UP 1 Access VIRT UP UP 1 Access VIRT UP UP 1 Access VIRT UP UP 1 Access VIRT UP UP 0 Access VIRT UP UP 1 Access VIRT UP UP 1 Access T VIRT UP UP 1 Trunk ar T PHYS UP UP 1 Trunk vmnic0 Local Target Logic (LTL) is an index to address a port, or group of ports. Data path lookup engine takes LTL as input, and gives LTL as output. LTL scheme: [0-14: internal ports] [15-271: pnics,vms, etc ] 126

119 Vemcmd show trunk Control and packet are CBL states 1 on the physical ports. ~ # vemcmd show trunk Trunk port 6 native_vlan 1 CBL 1 vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1, vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1, vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1, Trunk port 16 native_vlan 1 CBL 1 vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1, vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1, vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1, Trunk port 17 native_vlan 1 CBL 1 vlan(1) cbl 1, vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1, vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1, vemcmd show port vlans ~ # vemcmd show port vlans Native VLAN Allowed LTL VSM Port Mode VLAN State Vlans 17 Eth5/1 T 1 FWD 2,10-11, ~ # 127

120 Step 6: VEM and VSM See Each Other s MAC Is the VEM learning the MAC of the VSM? On VEM vemcmd show l2 <control-vlan> do you see the mac of the VSM? ~ # vemcmd show l2 2 Bridge domain 9 brtmax 4096, brtcnt 32, timeout 300 VLAN 2, swbd 2, "" Flags: P - PVLAN S - Secure D - Drop Type MAC Address LTL timeout Flags PVLAN Static 00:02:3d:21:a6: Dynamic 00:50:56:a9:25:

121 VEM and VSM See Each Other s MAC Is the VSM learning the MAC of the VEM? n1kv-l2# show mac address-table vlan 2 VLAN MAC Address Type Age Port Mod d21.a604 static 0 N1KV Internal Port d41.a604 static 0 N1KV Internal Port 5 129

122 Step 7: Physical Switch Mac Table Check the physical switch MAC address table Are the MACs of the VEM and VSM getting learned by the physical switches in the right VLANs? cae-cat6k-1#show mac-address-table vlan 2 Legend: * - primary entry age - seconds since last seen n/a - not available vlan mac address type learn age ports * dynamic Yes 360 Gi3/48 * a dynamic Yes 0 Gi4/9 * static Yes - Switch,Stby-Switch * d41.a604 dynamic Yes 0 Gi1/4 130

123 Step 8: VEM VSM Heartbeat One Heartbeat per second per VEM from VSM Timeout for VEM from VSM is 6 seconds of missed heartbeats After 6 seconds VSM will drop VEM Use vempkt capture to view heartbeats SPAN physical switch ports for heartbeats 131

124 Appendix B Miscellaneous Commands

125 Appendix C VXLAN Multicast

126 VXLAN Configuration: Multicast VMkernel interface to act as VTEP VSM Control Mode should be L3 Multicast for Broadcast traffic IP Multicast forwarding is required Multicast addresses Multiple segments can be mapped to a single multicast group If VXLAN transport is contained to a single VLAN, IGMP Querier must be enabled on that VLAN If VXLAN transport is traversing routers Multicast routing must be enabled. Proxy ARP must also be enabled 1550 MTU for VXLAN encapsulation overhead 134

127 VXLAN Configuration: Multicast Upstream Switch Configuration Enable IGMP Querier Set physical switch port MTU to 1550 Enable proxy-arp on upstream SVI ESXi Host Create VMK interface for VXLAN Nexus 1000V Enable feature segmentation Create a Bridge Domain Create a port-profile for VTEP VMK interface Create a veth port-profile for the VMs 135

128 VXLAN Configuration: Multicast Increase the MTU on your eth port-profile n1kv-l3(config)# port-profile type eth uplink n1kv-l3(config-port-prof)# mtu 1550 Create veth port-profile for VXLAN VMK interface n1kv-l3(config)# port-profile type vethernet VXLAN-VMK n1kv-l3(config-port-prof)# switchport mode access n1kv-l3(config-port-prof)# switchport access vlan 11 n1kv-l3(config-port-prof)# no shutdown n1kv-l3(config-port-prof)# system vlan 11 n1kv-l3(config-port-prof)# vmware port-group n1kv-l3(config-port-prof)# capability vxlan n1kv-l3(config-port-prof)# state enabled 136

129 VXLAN Configuration: Multicast Configure the Bridge Domain Maps a segment ID to a multicast address Segment ID >4096 n1kv-l3(config)# bridge-domain vxlan-1 n1kv-l3(config-bd)# segment id 5000 n1kv-l3(config-bd)# group Create VM port-profile n1kv-l3(config)# port-profile type veth vm-vxlan-1 n1kv-l3(config-port-prof)# vmware port-group n1kv-l3(config-port-prof)# switchport mode access n1kv-l3(config-port-prof)# switchport access bridge-domain vxlan-1 n1kv-l3(config-port-prof)# no shut n1kv-l3(config-port-prof)# state enabled 137

130 VXLAN Troubleshooting Tips Verify your Bridge Domains, VM port-profiles, and VXLAN VMK port-profiles Verify multicast on your upstream switches show ip igmp snooping Do you see the VTEPs Use vmkping on the ESXi host to verify network and MTU Use 1542 to cover the addition of the ICMP header ~ # vmkping -s d Verify the VEM has the right VXLAN capability ~ # vemcmd show vxlan interfaces LTL IP

131 VXLAN Troubleshooting Tips ~ # vemcmd show port vlans LTL VSM Port Mode VLAN/ State Vlans/SegID 17 Eth4/1 T 1 FWD 25, Eth4/2 T 1 FWD 25, Veth19 A 6000 FWD 6000 Verify the VEM was programmed correctly ~ # vemcmd show segment 6000 BD 23, vdc 1, segment id 6000, segment group IP , swbd 4096, 2 ports, "dvs.vcdvsvcdni-6-26-vl634-backed-b69c1d1d-02bf b7e-fa06c64e8c18" Portlist: 53 vse-vcdni-6-26-vl634-backed (b6 68 vcdni-2 (5ac7d73c-d1d ef 139

132 VXLAN Other Useful Commands vemcmd show port vemcmd show igmp <vlan> vemcmd show l2 segment <segment-id> vemcmd show vxlan-encap [ltl/mac] <ltl/mac address> vemcmd show vlxan-stats all Detailed slides in the Appendix 140

133 Appendix D - Additional VXLAN TShoot