Inside the Nexus 1000V Virtual Switch

Transcription

1 Inside the Nexus 1000V Virtual Switch BRKVIR

2 Agenda Nexus 1000V Why Nexus 1000V? VSMs, VEMs, and Port Profiles VMotion Physical Host Connectivity Port Channels and Upstream Switches Deploying and Migrating to VSM and VEM Inside the Nexus 1000V Software Upgrade and High Availability vpath and other Features 2

3 What happened to the Edge?

4 Losing the Edge VMs on Wrong VLANs! Server Admin must handle network configuration Host Host Host Host vswitch vswitch vswitch vswitch Server Admin No Network Visibility or Control! Unchaperoned VMto-VM communication! The rest of the network No Policy and VLAN control! Network Admin 4

5 And Finding it Back! Server Admin freed from networking configuration Host Host Host Host Server Admin Clear Configuration Boundaries Transparent Monitoring Boundaries Distributed Switch managed by Network Admin Network Admin 5

6 The Nexus 1000V

7 Virtual Supervisor Module (VSM) Take Two; They re Small! Familiar CLI interface into the Nexus 1000V Leverages NX-OS Controls multiple VEMs as a single network device Offered as both virtual and physical appliance Nexus 1010 or 1010-X 1u based on UCS C-Series Runs up to 6 or 10 (1010-X) VSMs or VSBs Each 1010 includes a 32 (or 48)-CPU VEM license! VSM 7

8 Virtual Ethernet Module (VEM) A Switch in Every Host o o o VEM Advanced switching capability on the hypervisor Provides each VM with dedicated switch ports VEM VSM 8

9 The Nexus 1000V Nexus 1000V is a Distributed Edge Switch VEM o o o VEM Up to 64 VEMs switching packets vcenter Server VSM Only one switch to manage! 9

10 The Switch Nexus 1000V on vcenter Virtual Side Physical Side 10

11 Flying Objects Virtual Machines and Virtual Ports...A flying saucer? You mean the kind from up there? Plan 9 From Outer Space, 1959

12 Network Customer Requirements Owned by the Server / Application Administrator Network Security Performance Monitoring DB / File Server Back-end Network Very High ERSPAN for debug DMZ Separate Access ERSPAN for recording Web / Mail Server Precise ACLs Unpredictable Call Manager / UC ACLs Predictable VDI Clients PVLANs Port Security Management Separate from Others Moderate Different Requirements for Different Networks 12

13 Network Implementation Owned by the Network Admininistrator Network admin accepts requirements Chooses VLANs Chooses security policies ACL, PVLAN, etc Chooses quality of service policies Chooses monitoring policies NetFlow, ERSPAN Chooses how physical NICs are used 13

14 Too many ports, and they move too fast Network admin needs sanity Server admin needs freedom To deploy and move virtual machines To deploy and move physical hosts switch # int gi1/0/35 switch # int gi1/0/47 switchport # mode int gi1/0/21 access switchport mode access access # int gi1/0/17 switchport mode vlan access 23 access vlan 23 etc switchport mode access switchport access vlan 23 etc switchport access vlan 23 etc etc Source: 14

15 Port Profiles Doing Your Homework Instead of individual Ports, create a Port Profile Set up ahead of time: VLANs ACLs NetFlow QoS Private VLANs and all other port config! # port-profile database switchport mode access switchport access vlan 10 ip port access-group myacl in no shut state enabled Re-use it multiple times! 15

16 Port Profiles Network View Setting Port Policies Ahead of Time # port-profile database switchport mode access switchport access vlan 10 no shut # port-profile webserver switchport mode access # port-profile webserver switchport access vlan 243 switchport mode access access list, etc. commands switchport access vlan 752 no shut access list, etc. commands no shut Port Profiles are Live : Network Admin can change them any time! 16

17 Nexus 1000V on vcenter Nexus 1000V is a Distributed Virtual Switch Physical Side Virtual Side 17

18 Port Profiles Server View Port Group Port Profile # port-profile iscsi26 switchport mode access switchport access vlan 26 Access lists, etc here state enable vcenter Server Port-group iscsi26 18

19 Server Admin s Virtual Port Virtual machine name Adapter MAC address Port profile (port group) 19

20 Network Admin s Virtual Port The Other End of the Wire vsm-main# show int veth2 Vethernet2 is up Port description is win8d, Network Adapter 4 Hardware is Virtual, address is b7.38e7 Owner is VM "win8d", adapter is Network Adapter 4 Active on module 3 VMware DVS port 280 Port-Profile is iscsi26 Port mode is access Virtual machine name, adapter, MAC address, current host all visible! 20

21 Deploying Virtual Machines with Nexus 1000V Network admin sets up port profiles in advance All features are specified that will be needed Goes to get coffee or on vacation Server admin creates VM templates Template virtual NICs use port profiles Server admin clones templates Clones bring port profiles along for the ride Possibly Thousands of VMs! Server admin starts up VMs Nexus 1000V sets up ports from port profiles Communicated by Vmware on VM startup No action from the Network Admin! 21

22 Moving Targets

23 Before the vmotion vsm-main# show int veth2 Vethernet2 is up Port description is win8d, Active on module 3 VMware DVS port 280 Port-Profile is iscsi26 o o o vsm-main# show module 3 Mod Server-IP Server-Name mbakke-ucs 23

24 After the vmotion vsm-main# show int veth2 Vethernet2 is up Port description is win8d, Active on module 17 VMware DVS port 280 Port-Profile is iscsi26 o o o vsm-main# show module 17 Mod Server-IP Server-Name mbakke-svr2 24

25 Notifying the L2 Network Learning the New Location of the VM o o o mac_move packet: Reverse ARP SA = MAC of virtual machine DA = all ones Causes switches to relearn! 25

26 Notifying the L2 Network Learning the New Location of the VM o o o mac_move packet: Reverse ARP SA = MAC of virtual machine DA = all ones Causes switches to relearn! 26

27 VMware Clusters A Group of Physical Hosts and Virtual Machines Small clusters for failover (HA) Larger clusters for load sharing (DRS) o o o vmotion vmotion within a single cluster Nexus 1000V can span multiple clusters 27

28 High Availability and Fault Tolerance High Availability X X Restart VM Fault Tolerance Both are supported by Nexus 1000V Virtual ethernet moves with the virtual machine 28

29 Physical Hosts and the Nexus 1000V

30 Host Connectivity Requirements Each Physical Host Is Typically on Several Networks Management to talk to vcenter Storage iscsi and NFS VMotion for moving VMs VSM to VEM communication the backplane Virtual machine networks (why we are all here) Virtual Side Port channels for physical NICs Many configurations possible From dual 10G to many 1G Physical Side 30

31 Port Profiles for Physical Ports Uplink or Ethernet # port-profile type ethernet trunkall switchport mode trunk switchport trunk allowed vlan channel-group auto mode on no shutdown state enabled vcenter Server Port-group trunkall 31

32 Port Profiles for Physical Ports Adding a Host to the Nexus 1000V vsm-main# 2011 May 18 19:04:50 vsm-main : Module 3 powered up 32

33 Port Profiles for Physical Ports Automatic Port Channels Inside the Nexus 1000V eth3/3 vmnic2 po1 eth3/4 vmnic3 # port-profile type ethernet trunkall switchport mode trunk switchport trunk allowed vlan channel-group auto mode on trunkall Port channel is connected to VLANs Ports on same profile cause a port channel to be created Physical ports assigned a port profile on N1k Port profile created with channel-group auto 33

34 Server Admin View of Physical Ports 34

35 Network Admin View of Physical Ports Automatic Port Channels vsm-main# show interface brief Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # Eth3/3 1 eth trunk up none 1000(D) 1 Eth3/4 1 eth trunk up none 1000(D) Port-channel VLAN Type Mode Status Reason Speed Protocol Interface Po1 1 eth trunk up none a-1000(d) none vsm-main# show cdp neighbors interface eth3/3 mbakke-r Eth3/3 173 R S I WS-C3750G-48T Gig1/0/24 35

36 Hooking It Up Virtual Ports on the Host Management and Control Networks vmotion Mgmt VEM DPA v m d ESXi Host A Host B Mgmt VLAN vmotion vcenter Server External Switch(es) Data VLANs N1k Control 36

37 Hooking It Up Network Storage NFS and iscsi Through the Nexus 1000V Mgmt v d NFS iscsi v m d n i Host A Host B NFS server NFS VLAN iscsi VLAN iscsi 37

38 Vmknics for the Host 38

39 Port Profiles for the Host # port-profile control-vmk switchport mode access switchport access vlan 30 system vlan 30 capability l3control no shutdown # port-profile management switchport mode access switchport access vlan 16 system vlan 16 # port-profile no shutdown vmotion switchport mode access switchport access vlan 27 no shutdown # port-profile nfs switchport mode access switchport access vlan 28 no shutdown m # port-profile iscsi switchport mode access switchport access vlan 26 no shutdown v d n i Host 39

40 System VLANs for Bootstrapping System VLANs must be used for: ESX Management (console) VLAN Both uplink and vethernet Nexus 1000V Control VLAN These VLANs are brought up on their ports before talking with the VSM System VLANs may be used for: Storage VLANs (NFS/iSCSI) # port-profile type ethernet trunkall switchport mode trunk switchport trunk allowed-vlan 16,26-28,30 no shutdown system vlan 16,30 m v d n i Host vcenter Server 40

41 System VLANs for VSM on its own VEM System VLANs must also be used in vethernet port profiles for: VSM Management (console) VLAN Nexus 1000V Control VLAN # port-profile management switchport mode access switchport # port-profile access vlan 16 vsm-control no shutdown switchport mode access system vlan switchport 16 access vlan 30 no shutdown system vlan 30 # port-profile type ethernet trunkall switchport mode trunk switchport trunk allowed-vlan 16,26-28,30 no shutdown system vlan 16,30 m d Host vcenter Server 41

42 VEM Automatically Found the VSM! Wait a Minute; How Did It Know? Nexus 1000V DVswitch has custom opaque data Contains VSM s contact information Initial physical port configuration, including System VLANs Switch-domain 40 VSM IP Uplink System VLANs 42

43 Port Channels and Upstream Switches

44 Common Port Channel Scenarios Port Profiles allow vmotion across different host types vmknic mgmt 1G 1G mgmt2 1G 1G data 1G 1G allmgt 1G 1G data 10G 10G all-vlans 10G 10G VLAN 10 VLANs VLANs LOM + 4x1G VLANs VLANs LOM + 2x10G VLANs Blade Server 44

45 Port Channels with Palo (on UCS) Nexus 1000V Features + Hardware Segmentation vmknic mgmt 1G 1G vmknic storage 3G 3G mgmt2 1G 1G data 5G 5G Port channels are used as normal ESX and Nexus 1000V see Palo s virtual adapters as physical NICs Each Palo can be provisioned to provide different virtual adapters, with bandwidth allocated 2x10G Palo Adapters 45

46 Common Port Channel Problems Upstream switches have many configurations Network can send N1k its own broadcasts via other switches!! N1k can also receive duplicate broadcasts or floods from outside hosts If a VM s traffic is sent on multiple links, MAC addresses could flap 46

47 Common Port Channel Solutions Essential to choose the Right One for your network LACP for clustered switches (Cisco vpc, VBS Stack, VSS) Use LACP (turn on LACP offload requires VSM reboot) vpc-hm MAC Pinning Un-clustered switches without port channels (e.g. UCS) Or when using one port per upstream switch vpc-hm Subgroup CDP or Subgroup Manual Un-clustered switches with port-channels enabled (e.g. N5k) Other vendors switches that support port channels vpc-hm + Network State Tracking Use with HP Virtual Connect when SmartLink not used Handles loops and split networks Use port profile for channel groups (let VSM create port channel automatically) 47

48 LACP for Clustered switches Cisco vpc, VSS, VBS Stack 802.3ad LACP used by both sides to agree on how to load balance Flow-based balancing allows more than one physical NIC to be used by a virtual NIC channel-group auto mode active Many hashing methods available Post 4.2(1)SV1(4) use of active mode is preferred 48

49 LACP Recovery When NIC fails, LACP recovers Pnic or upstream link fails X Tip: Use the new LACP Offload feature for FCoE or Boot from SAN LACP protocol agrees on new channel group Traffic resumes, hashing on remaining ports 49

50 MAC Pinning Keeping it Simple Simplest configuration; no upstream features required All Pnics with the same port profile create a port channel SubGroup PortChannel Each pnic is formed into its own subgroup Each vnic is pinned to a particular pnic vnics balanced across pnics static pinning also available 50

51 MAC Pinning When NIC fails, Vnics are re-pinned Pnic or upstream link fails X All vnics are re-pinned New mac_move is sent for each re-pinned vnic Tip: Do not configure upstream port channel when using MAC pinning 51

52 vpc-hm Subgroup CDP / Subgroup Manual Use when upstream switches are independent channels Use CDP for Cisco switches; manual for non-cisco gear All Pnics with the same port profile create a port channel Subgroups are formed for each physical switch using CDP info (or done manually if CDP unavailable) SubGroup PortChannel Each vnic is pinned to a particular subgroup 52

53 vpc-hm Avoiding Duplicates Use when upstream switches are independent channels Each vnic only sends packets on its subgroup One subgroup is designated receiver for broadcast packets Self-broadcasts filtered by source MAC X X Vnic drops any broadcast, multicast or flood traffic from other subgroups If a whole subgroup fails, vnics are re-pinned 53

54 Network State Tracking Use with vpc-hm when link detection doesn t work N1k sends tracking packets on each subgroup N1k expects tracking packets on its other subgroups If these are received, the network is not split Default: send every 5 seconds N1k uses all subgroups and designated receiver 54

55 Network State Tracking Detecting a network split? X X X?? Hardware failure happens but we don t know where! N1k misses tracking packets (default is 5 packets) Network is declared split If no other traffic received on a NIC for X more seconds, it s declared inactive Tip: NST can be set to just syslog instead Vnics are then re-pinned to another subgroup 55

56 Deploying the Nexus 1000V

57 Nexus 1000V How Did It Get There? VSM is created as a virtual machine From ISO image, using VSM setup wizard Enter the password for admin : Confirm the password for admin : Enter HA role[standalone/primary/secondary]: primary Enter the domain id<1-4095>: 47 Would you like to enter the basic configuration dialog? Yes Enter the switch name: mbakke-47 Mgmt0 IPv4 address: Mgmt0 IPv4 netmask: Enter SVS Control mode (L2 / L3) : L3 57

58 Nexus 1000V How Did It Get There? or from ovf package with new installer application Domain ID 46 VSM IP

59 VSM Best Practices L3 control is preferred for new installations No need to change a working L2 control setup Management, Control, and Packet can use same VLAN Networking For Your Reference Do not use VLAN 1 for Control and Packet Primary and Standby VSM must be in the same L2 domain 59

60 VSM Best Practices For Your Reference VSM primary to secondary latency up to 5ms VSM to VEM latency up to 5ms VSM on VEM is supported (but note system vlans and caveats) Caveats for putting vcenter behind VEM as well Placement Backup your configuration! VMware snapshots and cloning of the VSM are not supported Storage If deploying VSM on remote storage, know the caveats 60

61 VSM and vmotion Manual vmotion of VSM is supported Not recommended to allow DRS to vmotion Primary and Secondary VSM Aggressive DRS vmotion setting can cause VSM to drop packets and lose connectivity to VEM Best practice is to keep Primary and Secondary VSM outside of DRS Using the Nexus 1010 is a popular option that will avoid these concerns! 61

62 Nexus 1000V in vcenter Distributed Virtual Switch Nexus 1000 Virtual Switch -Locations of VEM images -Switch data for VEMs -VSM certificate vsm-main# show svs connections connection vc: ip address: certificate: default datacenter name: mbakke-main config status: Enabled vcenter Server 62

63 VEM Software Manual Installation VEM Can Be Installed Manually on Each Host PS C:\Program Files (x86)\vmware\vmware vsphere CLI\bin>./vihostupdate.pl --server mbakke-ucs --username root --install --bundle \VEM\cisco-vem-v zip Using the vsphere CLI ESX 5.0 changed to: esxcli software vib update or on the host with ssh # esxupdate -b cross_cisco-vem-v vib update Unpacking cross_cisco-vem-v13.. ######################################## [100%] Installing packages :cross_ci.. ######################################## [100%] Running [ipkg -f /tmp/ipkg.conf-n1kv remove cisco-vem-v124-esx]... ok. Running [/usr/sbin/vmkmod-install.sh]... ok. 63

64 Matching Versions (example with ESX 5.0) Pre-5.0: esxupdate vib-view query /]$ esxcli software vib list grep vem cisco-vem-v131-esx

65 Nexus 1000V on the Host Nexus 1000V VEM is an ESX Host Software Package esx-host # esxupdate --vib-view query VIB ID Package State cross_cisco-vem-v130-esx_ installed VEM Software Package esx-host # ps grep vemdpa vemdpa Data Path Agent Communicates with VSM ESX 5.0 changed to: esxcli software vib list Hypervisor Drivers Packet Switching esx-host # vmkload_mod -b Name Size Used vmkernel vem-v120-l2device vem-v120-n1kv vem-v120-vssnet vem-v120-stun

66 VMware Update Manager Or Automatically When a Physical Port Is Selected Ports and Uplink Port Profiles are Selected then VEM software is installed in the background and automatically started 66

67 Deploying Large Numbers of Hosts No Network Admin Actions Required! VUM for VEM installation Set up a host Complete with port profiles! Create a host profile Add hosts using host profile Nexus 1000V is Added! Images from VMware vsphere 67

68 Migrating to Nexus 1000V VMware vswitch and Nexus 1000V can coexist On the same servers Migrate some networks and VMs at first Then migrate the rest later No need to shut down VMs to migrate Simply use vcenter to change the port group But will momentarily disrupt traffic Very much like unplugging a cable and moving it to a port on another switch 68

69 Uplink (Ethernet) Port-Profile Notes For Your Reference Do not add multiple pnics from same ESX host to same uplink port-profile if no port-channeling is configured While it may work you will end up with the ESX host receiving duplicate packets and require extra processing from CPU to deal with this improper configuration Do not configure multiple uplink port-profiles to an ESX host carrying the same vlan Example: Uplink1 and uplink2 to same ESX host both carrying vlan 100 Use vpc-mac pinning instead If you want NIC teaming use one of the approved port-channel mechanisms 69

70 Spanning-tree and BPDU Best Practice For Your Reference Mandatory Spanning-tree settings per port IOS set STP portfast cat65k-1(config-if)# spanning-tree portfast trunk NXOS set port type edge n5k-1(config-if)# spanning-tree port type edge trunk Highly Recommended Global BPDU Filter/Guard IOS NXOS cat65k(config)# spanning-tree portfast bpdufilter cat65k(config)# spanning-tree portfast bpduguard n5k-1(config)# spanning-tree port type edge bpduguard default n5k-1(config)# spanning-tree port type edge bpdufilter default BPDU Filter is mandatory for LACP port-channels Set per port BPDU Filter/Guard when Global is not possible 70

71 VEM VSM Troubleshooting For Your Reference If VEM adds in vcenter but not on VSM show module With L2 it s usually a Control VLAN issue Verify Control VLAN connectivity Make sure that the control VLAN is a system VLAN If VSM on its own VEM, check port profiles for VSM s ports These need to be system VLANs, too! 71

72 VEM VSM Troubleshooting For Your Reference With L3 it s usually an IP routing problem If you can ping VMK interface the VEM should connect to VSM Troubleshoot as you would all VMware L3 issues Is there a vmk NIC using the port profile with l3control? Is the VMK port-profile set with system VLAN? When VSM/VEM connectivity is restored Hitless to VEM unless VSM configuration has changed Hitless to VEM even if VSM rebooted (unless config changed) 72

73 Inside the Nexus 1000V

74 Virtual NIC Types Inside the Host Standard Off the Shelf Guest OS Guest OS with VMware Tools Console, vmotion, NFS, iscsi Virtual Machines flexible e1000 vmxnet2 vmxnet3 vmknic Virtual NICs Virtual Switch Physical NICs 74

75 VMware Port IDs and Port Groups Inside the Virtual Switch 107 data data data18 Nexus 1000V Virtual Ethernet Module 480 mytrunk 481 mytrunk vmknic Virtual NICs 288 mgmt Port ID portgroup Port ID portgroup Physical NICs 75

76 VEM LTLs (port numbers) and veths Inside the VEM: global veths and local LTLs Nexus 1000V Virtual Ethernet Module vmknic veth veth veth veth port-channel3 19 ethernet7/2 20 ethernet7/ Each port has an LTL number, local to the host Veth, port-channel and ethernet assigned by VSM 76

77 VLANs tie Virtual and Physical Together Inside the VEM: VLANs connect virtual and physical ports Nexus 1000V Virtual Ethernet Module veth veth veth42 VLAN 17 VLAN po3 19 eth7/2 20 eth7/ vmknic veth56 VLAN 10 Most virtual ports are Access Ports Most physical ports are Trunk Ports 77

78 MAC Tables for Dynamic Forwarding Each VEM has its own MAC Table VLAN Type MAC LTL 48 veth veth veth42 17 Static 00:50:12:34:ab:cd Static 00:50:12:68:a5:a Dynamic 00:0c:12:34:ab:cd Dynamic 00:02:20:fe:ed: Static 00:50:12:34:ab:cd po3 19 7/2 20 7/3 vmknic 104 veth56 10 Static 00:50:12:34:ab:cd 104 VMware supplies veth MAC addresses (static) N1k learns dynamic MAC addresses 78

79 Distributed L2 Switching Each VEM is an Independent L2 Switch Host VLAN Type MAC LTL 17 Static 00:50:12:34:ab:cd Static 00:50:12:68:a5:a Dynamic 00:50:05:55:12: Dynamic 00:50:08:67:53: po3 19 7/2 20 7/3 Host 2 VLAN Type MAC LTL 17 Static 00:50:05:55:12: / Static 00:50:08:67:53: Dynamic 00:50:12:34:ab:cd Dynamic 00:50:12:68:a5:a po2 20 8/3 79

80 Physical Switch MAC Table (VSM) vsm-main# show int control0 inc Hardware Hardware: Ethernet, address: 000c df (bia 000c df) vsm-main# show mac-address-table module 6 vlan 30 VLAN MAC Address Type Age Port Mod d static 0 N1KV Internal Port d60f static 0 Veth b7.002e static 0 Veth b7.002f static 0 Veth b7.22ad static 0 Veth b3.f588 dynamic 257 Po c df dynamic 0 Po3 6 80

81 Physical Switch MAC Table (VEM) /]$ vemcmd show card grep "Primary VSM MAC" Primary VSM MAC : 00:0c:29:62:25:df [root@ucs-red /]$ vemcmd show l2 30 Type MAC Address LTL timeout Flags PVLAN Static 00:02:3d:80:28: Static 00:50:56:b7:00:2f 51 0 Static 00:50:56:b7:00:2e 52 0 Static 00:50:56:71:d6:0f 50 0 Static 00:50:56:b7:22:ad 55 0 Dynamic 00:0c:29:62:25:df

82 Upgrading the Nexus 1000V

83 Upgrading the Nexus 1000V Software Keeping the Boundaries 3. Server admin upgrades VEMs Server admin still owns the hardware o o o 2. VSM makes new VEM version available 1. Network admin upgrades VSMs vcenter Server 83

84 Upgrading the Nexus 1000V Software Special Procedure for 1.3 to Server admin upgrades VEMs Server admin still owns the hardware o o o VSM works with new VEM version 3. Network admin upgrades VSMs vcenter Server VSM enables new features 84

85 Upgrade: Example Problem For Your Reference Adding hosts failed after upgrading VSM vsm-main# show vmware vem upgrade status # Check that bundle IDs match How this can happen Network admin skips all or part of the notify/accept/proceed/complete Server admin uses VUM or vihostupdate to install new VEM version Bundle ID in DVS doesn t match When trying to add host VUM sees wrong bundle ID The fix: Disable VUM Do the notify/accept/proceed/complete procedure Verify bundle IDs match are are correct Enable VUM 85

86 High Availability

87 VSM High Availability (HA) VSM has 3 modes HA is limited to 2 VSM Virtual Machines (primary and secondary) Standby VSM is powered up Standby cannot be powered down Standalone No HA, can later be converted Primary Secondary They must be on the same L2 Management network They must be on the same Control and Packet network Keep Primary and Secondary on different ESX hosts 87

88 Mgmt Network Reference Topology switch switch Heartbeats Alternate HB Sync Data VSM1 Active VSM2 Standby Control/ Packet Network switch switch VEM1 VEM2 VEM3 VEM4 88

89 Watching VSM Heartbeats: show system internal redundancy info My CP: slot: 0 domain: 100 role: primary status: RDN_ST_AC state: RDN_DRV_ST_AC_SB intr: enabled power_off_reqs: 0 reset_reqs: 3 Other CP: slot: 1 status: RDN_ST_SB active: true ver_rcvd: true degraded_mode: false Device 0 == control (normal) Device 1 == mgmt (if control fails) Active VSM Standby VSM Redun Device 0: name: ha0 pdev: f alarm: false mac: 00:50:56:a0:00:23 tx_set_ver_req_pkts: 84 tx_set_ver_rsp_pkts: 1 tx_heartbeat_req_pkts: tx_heartbeat_rsp_pkts: 345 rx_set_ver_req_pkts: 1 rx_set_ver_rsp_pkts: 1 rx_heartbeat_req_pkts: 345 rx_heartbeat_rsp_pkts: other error counters here Heartbeats increase 1 per second 89

90 Fail Scenario 1: VSM2 Control fails VSM1 Active VSM2 Standby Effect: (communication lost) VSM2 stays as Standby Heartbeats also on management network; VSMs aware of each other VSM1-VSM2 communication goes into degraded mode TCP sync cxn broken at VSM1 VSM2 resets & waits for VSM1 VEMs stay connected to active For Your Reference VEM1 VEM2 VEM3 VEM4 Exit: (communication restored) No VEM flap VSM2 syncs with VSM1 Normal HA operation 90

91 Fail Scenario 2: VSM1 Control fails For Your Reference VSM1 Active VSM2 Standby Effect: (communication lost) VSM2 Drops VEMs VEMs run headless VSM2 stays as standby Heartbeats also on management network after 3 seconds; VSMs aware of each other Sync broken & VSM2 reset as before VEM1 VEM2 VEM3 VEM4 Exit: (communication restored) No VEM flap VSM2 comes back up in HA mode VEMs reconnect to VSM1 91

92 Fail Scenario 3: Control Network Split For Your Reference VSM1 Active VSM2 Standby Effect: (communication lost) VEMs 1 and 2 run normally VEMs 3 and 4 run headless Otherwise, same as scenario 2 Exit: (communication restored) VEMs 3 & 4 reconnect to VSM1 VSM2 comes back up in HA mode VEM1 VEM2 VEM3 VEM4 92

93 Fail Scenario 4: Split Brain Active Side For Your Reference VSM1 Active VSM2 Standby Effect: (communication lost) VSM2 becomes Active and takes over VEMs 1-4 VSM1 stays active but drops all VEMs VSM1 and VSM2 use same IP address VEM1 VEM2 VEM3 VEM4 Exit: (communication restored) VEMs will flap after split brain condition detected (hitless if configuration not changed) VSM1 is reset when connection is restored VEMs stay connected to VSM2 93

94 Fail Scenario 5: Split Brain Standby Side For Your Reference VSM1 Active VSM2 Standby Effect: (communication lost) VSM2 becomes Active buts sees no VEMs VSM1 stays active handling all VEMs VSM1 and VSM2 use same IP address Exit: (communication restored) VSM2 resets VSM1 when connection is restored VEMs connect to VSM2 VEM1 VEM2 VEM3 VEM4 94

95 HA Differences from 1.3 to 1.4/1.5 Secondary VSM no longer resets continuously Reset is now triggered by broken configuration sync (TCP pipe full) Heartbeats use management network if control network fails To help avoid split brain 95

96 vpath Integrating Appliances (such as firewalls) into Nexus 1000V

97 What is vpath? Nexus 1000V feature, enabling virtual appliance integration VSG Virtual Security Gateway ASA 1000V Virtual ASA Firewall Other services covered by BRKVIR-2011 Allows appliances to intercept data path traffic And scale virtual appliances across hosts 97

98 vpath Decision Caching for Smart Appliances 3. Flow decision (allow/drop) is sent to vpath on Nexus 1000V 2. Packet is sent to vpath capable device (e.g. VSG) vpath 1. vpath detects packet in a new, unknown flow 98

99 vpath Decision Caching for Smart Appliances vpath 4. vpath allows or drops subsequent packets in flow without intervention 99

100 Related Presentations BRKVIR-2010 BRKVIR-2011 BRKVIR-2931 BRKDCT-2010 BRKDCT-2023 Cisco VXI and End to End Architecture Deploying Services in a Virtualized Environment End-to-End Data Center Virtualization Next Generation Data Centre Architecture Evolution of the Data Centre Access Architecture 100

101 Keeping Your Sanity Virtualization brings new networking challenges As well as opportunities, with more of both to come! Nexus 1000V brings sanity to the virtual network Clear, appropriate administrative boundaries Port profiles for deployment, scaling and motion Nexus 1000V preserves freedom for servers Safely deploy hosts and VMs independently How to keep your own sanity Follow the configuration guides for Nexus 1000V Don t panic use the troubleshooting presentation 101

102 Recommended Reading BRKVIR- 2012

103 Q & A

104 Complete Your Online Session Evaluation Complete your session evaluation: Directly from your mobile device by visiting and login by entering your username and password Visit one of the Cisco Live internet stations located throughout the venue Open a browser on your own computer to access the Cisco Live onsite portal Don t forget to activate your Cisco Live Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit 104

105 105

106 References

107 Cisco Live Europe, BRKVIR-3013 Similar to this talk, but more focus on troubleshooting Register for a free user ID at Look for BRKVIR-3013 Slides and video are available For Your Reference 107

108 VXLAN deep dive & more! Resources CCO Links 1000V: : VSG: VNMC: vasa: My Cisco Community mmunity/technology/datacenter/ nexus1000v Deployment Guides Nexus 1000V Deployment Guide Nexus 1000V on UCS Best Practices Nexus 1010 Deployment Guide VSG Deployment Guide White paper Nexus 1000V and vcloud Director For Your Reference 108

109 Validated Designs For Your Reference vblock with Nexus 1000V FlexPOD with Nexus 1000V and Nexus 1010 Virtual Multi-tenant Data Center with Nexus 1000V Virtual Desktop 1000V and VMware View 1000V and VSG in VXI Reference Architecture 109

110 Extras Help and Troubleshooting

111 Verify VSM to vcenter Connectivity Verify SVS connection settings For Your Reference vsm-main# show svs connections connection vc: ip address: remote port: 80 protocol: vmware-vim https certificate: default datacenter name: mbakke-main admin: max-ports: DVS uuid: a4 ab d a-7a b1 9a e f3 8c config status: Enabled operational status: Connected sync status: Complete version: VMware vcenter Server build

112 Connectivity Error Extension Key For Your Reference The error below means the correct key is not registered n1000v(config-svs-conn)# connect ERROR: [VMware vcenter Server build ] Extension key was not registered before its use Check the key on VSM vsm-main# show vmware vc extension-key Extension ID: Cisco_Nexus_1000V_

113 Check Extension Key on VMware MOB For Your Reference Click on content Click on ExtensionManager vsm-main# show vmware vc extension-key Extension ID: Cisco_Nexus_1000V_

114 Connectivity Error Connection Refused For Your Reference The error below could indicate a port mismatch n1000v(config-svs-conn)# connect ERROR: [VMWARE-VIM] Operation could not be completed due to connection failure.connection refused. connect failed in tcp_connect() Default port for communication is port 80 All communication is https VMware accepts on port 80 and tunnels internally to port

115 VSM to VMware vcenter Connectivity For Your Reference Make sure VSM SVS port matches vcenter http port n1000v# show svs connections connection vcenter: ip address: remote port: 80 protocol: vmware-vim https To change the port number n1000v(config)# svs connection vcenter n1000v(config-svs-conn)# remote port

116 VSM to VMware vcenter Connectivity Verify Port number in vcenter Administration->vCenter Server Settings For Your Reference 116

117 Port Channels How to Tell Pinning For Your Reference Use show interface virtual pinning on the VSM vsm-main# show interface virtual pinning Veth Pinned Associated PO List of Sub Group id interface Eth interface(s) Veth1 0 Po1 Eth7/1 Veth2 0 Po1 Eth7/1 Veth3 1 Po1 Eth7/2 Veth4 1 Po1 Eth7/2 117

118 Port Channels How to Tell Pinning For Your Reference Or use vemcmd show port on ESX host From VSM preface vemcmd with module vem <module-number> execute If ssh into host, just run vemcmd without the above vsm-main# module vem 6 execute vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port 17 Eth7/1 UP UP FWD vmnic0 18 Eth7/2 UP UP FWD vmnic1 49 Veth1 UP UP FWD 0 0 VSM1.eth0 50 Veth2 UP UP FWD 0 0 VSM1.eth1 51 Veth3 UP UP FWD 0 1 VSM1.eth2 53 Veth4 UP UP FWD 0 1 vmk0 305 Po1 UP UP FWD 0 118

119 Cheat Sheet: Tracing Where Ports Go For Your Reference Get the number of veths (subtract 4) show int virtual wc lines Get the list of veths that are down show int brief grep down For a veth, find its VM and network adapter show int virtual description grep Veth34 For a veth, find its DVPort show int virtual port-mapping grep Veth34 Show Veth/DVport for all interfaces that are not up show int virtual port-mapping exclude up Show interfaces on a particular module show int virtual module 5 Show only vmknics on a module show int virtual vmk module 6 Show only VM interfaces on a module show int virtual vm module 6 119

120 Cheat Sheet: Tracing a downed Ethernet port For Your Reference Get the DVPortID for a veth show int virtual port-mapping grep Veth34 Get the VM port number for a veth show int virtual description grep Veth34 Get the host module # a veth is running on show int veth34 grep "on module" After the above: Check to see if module is up See if DVPort appears in vcenter: 1. Go to Networks, click on the DVswitch 2. Click on the Ports tab 3. Sort appropriately, and see if the DVPort ID appears 120

121 Cheat Sheet: VSM Sanity Check For Your Reference show vmware vem upgrade status Look for bundle ids that match and no errors show run Look for proper system vlans Look for mac pinning mode matching the configuration show int brief grep down Look for interfaces that are down and make sure they are expected 121

122 122