NET1350BUR Deploying NSX on a Cisco Infrastructure Jacob Rapp jrapp@vmware.com Paul A. Mancuso pmancuso@vmware.com #VMworld #NET1350BUR
Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #NET1350BUR CONFIDENTIAL 2
Agenda 1 NSX Anywhere 2 NSX Design 3 Nexus Switching Fabric 4 UCS Connectivity 5 NSX on ACI 6 Summary & Question #NET1350BUR CONFIDENTIAL 3
Agenda 1 NSX Anywhere 2 NSX Design 3 Nexus Switching Fabric 4 UCS Connectivity 5 NSX on ACI 6 Summary & Question #NET1350BUR CONFIDENTIAL 4
Apps Driving Infrastructure Requirements Infrastructure independent, with standards-based interoperability where necessary Security wrapped around the VM, container & microservice Works across hypervisors, application frameworks, clouds People, Process and Tooling Model #NET1350BUR CONFIDENTIAL 5
Infrastructure Independent Any L2/L3 Fabric Controlled Communication Controlled Communication Micro- Segment Controlled Communication Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful Stateful DFW DFW 3 rd Party Services 3 rd Party Services 3 rd Party Services 3 rd Party Services Any Transport Consistent policy Any IP network Up to 150ms Universal Distributed Logical Router Controlled Communication Controlled Communication Micro- Segment Controlled Communication Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful Stateful DFW DFW 3 rd Party Services 3 rd Party Services 3 rd Party Services 3 rd Party Services Public clouds ACI Fabric Single/Multi Pod Or Any L2/L3 Fabric #NET1350BUR CONFIDENTIAL 6
Security Everywhere Control Points Data Center Cloud Campus/Branch Mobile Endpoint Context Telemetry: Application Network Intelligence Policy Complete framework spanning use cases and service stack #NET1350BUR CONFIDENTIAL 7
Multi-Hypervisor, Cloud and Frameworks Branch offices On-Premise Data Center vcloud Air Network Public clouds VMworld 2017 Content: Not for Virtual Desktop (VDI) New app frameworks publication Mobile Devices (Airwatch) Security Inherently Secure Infrastructure Automation IT at the Speed of Business Application Continuity Data Center Anywhere #NET1350BUR CONFIDENTIAL 8
People, Process and Tooling Model Ending State Blended, cross-functional People Speed, agility, standardization Processes VMworld 2017 Content: Not for Converged, correlated publication Tooling Beginning State Siloed, specialized Slow, error prone, inconsistent Domain-specific, hardware focused Reseach paper detailing operational changes: https://tinyurl.com/y8dme6gx #NET1350BUR CONFIDENTIAL 9
Agenda 1 NSX Anywhere 2 NSX Design 3 Nexus Switching Fabric 4 UCS Connectivity 5 NSX on ACI 6 Summary & Question #NET1350BUR CONFIDENTIAL 10
Software Defined Data Center Delivers Freedom Network hardware centric application and service deployment Difficult operational model Complex protocol mix Hardware ASIC dependent Finite service offering Micro- Stateful DFW Micro- Stateful Micro- Micro- Segment Stateful DFW Micro- 3 rd Party Segment Stateful DFW DFW Micro- Micro- 3 rd Party Segment Stateful DFW Segment Stateful 3DFW 3 rd Party Services Segment rd Party Segment Stateful DFW 3 rd Party 3 rd Party Services Segment Stateful DFW 3 rd Party Services Services 3 rd Party Services Services Services Services Micro- Segment Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful DFW Stateful DFW Decouple applications from hardware infrastructure: Simple, Scalable Service model #NET1350BUR CONFIDENTIAL 11
NSX vsphere Cluster Design Tiered Logical Switches Web Application Database Transit VXLAN Management (VLAN) VM1 VM5 Transport Zone (VLAN) VM2 VM3 VM5 VM6 VMworld 2017 Content: Not for Compute Clusters Edge Cluster(s) Management Cluster Physical Network NSX Edge Control VM NSX Controller Cluster NSX Manager publication Transport Subnet A 192.168.150.0/24 Transport Subnet B 192.168.250.0/24 #NET1350BUR CONFIDENTIAL 12
VMkernel Networking Span of VLANs IP Stacks Default vmotion VXLAN SVI 66: 10.66.1.1/26 SVI 77: 10.77.1.1/26 SVI 88: 10.88.1.1/26 SVI 99: 10.99.1.1/26 VLAN 66 Mgmt 10.66.1.2/26 DGW: 10.66.1.1 VLAN 77 vmotion 10.77.1.2/26 DGW: 10.77.1.1 VLAN 88 VLAN Trunk (802.1Q) VXLAN 10.88.1.2/26 DGW: 10.88.1.1 Layer 2 or Layer 3 Uplinks VLAN 99 or distribution vsphere Host (ESXi) Storage 10.99.1.2/26 DGW: 10.99.1.1 Span of VLANs #NET1350BUR CONFIDENTIAL 13
NSX VXLAN Data and Control Plane NSX VXLAN Control Plane Unicast mode or Hybrid Mode Unicast preferred with ACI infrastructure NSX Data Plane VMKernel VTEP encaps/decaps Transport Zone 1 1 2 VXLAN VTEP 3 VXLAN VTEP 2 3 4 5 3 4 5 L2 Frame L2 IP UDP VXLAN L2 Payload IP Frame UDP VXLAN L2 Payload L2 Frame VM Sends a standard L2 Frame Source Hypervisor (VTEP) encapsulates VXLAN, UDP & IP Headers Physical Network forwards frame as standard IP frame Destination Hypervisor (VTEP) decapsulate headers Original L2 Frame delivered to VM #NET1350BUR CONFIDENTIAL 14
NSX VXLAN Infrastructure Transport Zone VXLAN prepared ESXi clusters VTEP(s) 1 prepared VDS per cluster enabled for VXLAN Logical Switch as dvportgroup Support for VMware vds managed by vsphere VTEP1 10.20.10.10 VM MAC1 vsphere Host VM MAC2 Compute VDS VTEP2 10.20.10.11 VMworld 2017 Content: Not for VXLAN 5002 VXLAN prepared Hosts VM MAC3 Edge VDS VM MAC4 VTEP3 VTEP4 vsphere 10.20.10.12 10.20.10.13 Host publication VXLAN Transport Network #NET1350BUR CONFIDENTIAL 15
VDS Uplink Design NSX host preparation Creates VXLAN dvuplink Consistent for all hosts using that VDS Consistent teaming policy Recommended teaming mode Route Based on Originating Port LACP is not possible from UCS blade VMworld 2017 Teaming Mode Route based on Originating Port Route based on Source MAC hash NSX Support Multi- VTEP LACP Route based on IP Hash (Static EtherChannnel) Explicit Failover Order Uplink 2 x 10G/40G Nexus/ACI Port Configuration All Active Standard All Active Standard Flow based All active Flow based All active Single link active vpc Port-Channel - LACP Content: Not for publication vpc Port-Channel LACP mode OFF Standard Route based on Physical NIC Load (LBT) Standard #NET1350BUR CONFIDENTIAL 16
Connect Your Workloads to a Physical Network Route and switch where you can. VLAN subnets Transit subnet VXLAN subnets VM1 VM5 VM2 VM5 VM3 VMworld 2017 Web Application Database Compute/Edge Clusters Physical Network VXLAN VLAN Bridge if you must. VM1 VM5 Content: Not for publication VM2 VM5 VM3 Each bridged segment is a separate subnet #NET1350BUR CONFIDENTIAL 17
Agenda 1 NSX Anywhere 2 NSX Design 3 Nexus Switching Fabric 4 UCS Connectivity 5 NSX on ACI 6 Summary & Question #NET1350BUR CONFIDENTIAL 18
NSX Infrastructure Requirements Only Two Requirements 1) MTU of 1600 2) IP Connectivity NSX is AGNOSTIC to underlay network topology L2 or L3 switched infrastructure #NET1350BUR CONFIDENTIAL 19
Jumbo MTU Nexus 5000 and 6000 VDS Max MTU is 9000 Byte Nexus 5xxx, 56xx and 6xxx requires MTU to be changed with Policy-Map L3 requires per interface MTU change All links belonging to fabric must be enabled with Jumbo MTU VMworld 2017 All L2 interfaces Only global configurations Create policy-map: policy-map type network-qos jumbo class type network-qos class-default mtu 9216 Apply policy-map: system qos service-policy type network-qos jumbo Layer 3 Interface interface Vlan151 SVI Interface no ip redirects ip address 10.114.221.34/27 hsrp 1 ip 10.114.221.33 description VXLAN Transport Zone no shutdown mtu 9216 Content: Not for publication interface Ethernet2/12 Layer 3 Interface description L3 Link to Spine no switchport speed 40000 duplex full mtu 9216 ip address 10.114.211.117/31 no shutdown #NET1350BUR CONFIDENTIAL 20
Jumbo MTU Nexus 7000 and 9000 VDS Max MTU is 9000 Byte Nexus 7xxx, 9xxx Series L2 only requires global configuration L3 requires per interface MTU change All links belonging to fabric must be enabled with Jumbo MTU VMworld 2017 Layer 2 Interface system jumbomtu 9216 Global configurations interface Ethernet1/9 description to esx-vmnic3-vmk switchport mode trunk switchport trunk allowed vlan 22-25 spanning-tree port type edge trunk mtu 9216 Layer 2 MTU channel-group 9 mode active Layer 3 Interface interface Vlan151 SVI Interface no ip redirects ip address 10.114.221.34/27 hsrp 1 ip 10.114.221.33 description VXLAN Transport Zone no shutdown mtu 9216 Content: Not for publication interface Ethernet2/12 Layer 3 Interface description L3 Link to Spine no switchport speed 40000 duplex full mtu 9216 ip address 10.114.211.117/31 no shutdown #NET1350BUR CONFIDENTIAL 21
Cisco DC Topology L2 Pod NSX is Agnostic Pod components can be any mix of 9k / 7k/ 6k / 5k / 2k Compute Cluster A Compute Cluster B L3 L2 POD A 95xx L3 Core L3 95xx 95xx 95xx L2 95xx 95xx 93xx 93xx 93xx 93xx 93xx 93xx 93xx 93xx VMworld 2017 Content: Not for POD B VLANs & IP Subnet Defined at 95xx for POD A SVI Interface VLAN ID IP Subnet Management 100 10.100.A.x/24 publication vmotion 101 10.101.A.x/24 Storage 102 10.102.A.x/24 VXLAN 103 10.103.A.x/24 VLANs & IP Subnet Defined at 95xx for POD B SVI Interface VLAN ID IP Subnet Management 200 10.200.B.x/24 vmotion 201 10.201.B.x/24 UCS B-Series VLAN ID 100, 101 & 102 Scope UCS B-Series VLAN ID 200, 201 and 203 Scope Storage 202 10.202.B.x/24 VXLAN 103 10.103.B.x/24 VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters) #NET1350BUR CONFIDENTIAL 22
Cisco DC Topologies L3 Design NSX is Agnostic Compute Cluster A Compute Cluster B L3 L2 POD A 95xx L3 Core 95xx 95xx 95xx 95xx 95xx L3 93xx 93xx 93xx 93xx 93xx 93xx 93xx 93xx L2 POD B VLANs & IP Subnet Defined at each ToR SVI Interface VLAN ID IP Subnet Management 100 10.100.R_ID.x/24 vmotion 101 10.101.R_ID.x/24 Storage 102 10.102.R_ID.x/24 VXLAN 103 10.103.R_ID.x/24 UCS B-Series VLAN ID 100, 101 & 102 Scope UCS B-Series VLAN ID 100, 101 & 102 Scope VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters) #NET1350BUR CONFIDENTIAL 23
Compute Cluster A Compute Cluster B Cisco DC Topologies VXLAN Leaf Spine - Leaf can be: 9xxx, 7xxx, 6xxx, 56xx UCS B-Series Spine UCS B-Series L3 Spine 95xx 95xx 95xx 95xx 56xx 56xx 93xx 93xx 93xx 93xx or UCS B-Series VLAN ID 100, 101 and 102 Scope VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters) VLANs & IP Subnet Defined at each ToR SVI Interface VLAN ID IP Subnet Management 100 10.100.100.x/24 vmotion 101 10.101.101.x/24 Storage 102 10.102.102.x/24 VXLAN 103 10.103.103.x/24 60xx UCS B-Series 60xx DC Core Internet/DMZ distribution Border Leaf Mgt / Edge Cluster Cisco s Prime or NFM may also provide underlay and VXLAN management. (NFM supports only 9ks) #NET1350BUR CONFIDENTIAL 24
Agenda 1 NSX Anywhere 2 NSX Design 3 Nexus Switching Fabric 4 UCS Connectivity 5 NSX on ACI 6 Summary & Question #NET1350BUR CONFIDENTIAL 25
NSX Connectivity Ideals vsphere Host and UCS Interconnectivity UCS Fabric Interconnects End-Host mode vpcs to Nexus switching vsphere Compute connectivity UCS vnics shared or dedicated vsphere dvuplinks equals number of vmnics VTEPs with Src ID teaming vsphere Edge connectivity Preferably UCS C-Series Separate connection, bypass FIs 95xx L3 L2 DC Fabric 93xx UCS 95xx 93xx Edge Leaf or Edge Cluster UCS C-Series DC Core Internet/DMZ distribution #NET1350BUR CONFIDENTIAL 26 VPN VPN VPN VPN
NSX Edge Routing for UCS C Series & Intel NICs Recommended design for Edge routing: Two Uplinks per ESG Per ESG, establish adjacency to each ToR (redundancy) VLAN backed networks routing neighbors on the ToRs A/B Map each VLAN of the dvportgoup to each dvuplink VLANs used for dynamic routing are local to each router Use Source ID ebgp is used between NSX ESG and routers A/B Equally applicable to OSPF Default route must follow the uplink status VMworld 2017 Loss of Uplinks will withdraw routes on that fabric and upstream link Default route advertised downstream ebgp Peering VLAN 10 SVI vnic1 Uplink A VLAN 10 95xx 9K-A vds-edge 9K-B 95xx VLAN 20 SVI Content: Not for publication vnic2 Uplink B VLAN 20 Pair of NSX Edges per ESXi host #NET1350BUR CONFIDENTIAL 27
NSX Edge Routing for UCS B Series Recommended design for Edge routing Dedicated UCS vnics pinned for routing Port-Channel connection between FI and ToRs Redundancy and scaling bandwidth Per ESG, establish an adjacency per ToR (Redundancy) Each ESG, bound to one active dvuplink ebgp is used between NSX ESG and N9Ks Equally applicable to OSPF Default route must follow the uplink status Loss of Uplinks will withdraw routes on that fabric and upstream link VMworld 2017 Content: Not for Default route advertised downstream VLAN 10 SVI Dedicated non-vpc pinned UCS FI-A 1 vnic maps to Fabric A VLAN 10 95xx 9K-A vds-edge 9K-B publication 95xx VLAN 20 SVI UCS FI-B 1 vnic maps to Fabric B VLAN 20 ebgp Peering Pair of NSX Edges per ESXi host #NET1350BUR CONFIDENTIAL 28
VDS Design, Uplink & Traffic Mapping Recommended UCS B-Series setup UCS B-Series Blade vnic 1 93xx 6248 (A) 6248 (B) 2204 FEX vnic 2 2204 FEX vnic 3 Nexus 93XX NX-OS Mode vnic 4 VMNIC 0 VMNIC 1 VMNIC 2 VMNIC 3 VDS - 1 MGMT, vmotion, NFS, VXLAN & Bridging 93xx VDS 2 Routing Traffic Type VMkernel IP Storage VMkernel vmotion VMkernel VXLAN VTEP 1 VMkernel VXLAN VTEP - 2 VMkernel Mgmt Bridging PG Routing VLAN PG 10 Routing VLAN PG 20 Teaming Mode LBT LBT SRC_ID SRC_ID Explicit Failover SRC_ID SRC_ID SRC_ID #NET1350BUR CONFIDENTIAL 29
Agenda 1 NSX Anywhere 2 NSX Design 3 Nexus Switching Fabric 4 UCS Connectivity 5 NSX on ACI 6 Summary & Question #NET1350BUR CONFIDENTIAL 30
NSX Network Virtualization with ACI underlay Cloud Management Platform vrealize, OpenStack, Custom NSX vcenter NSX Network Virtualization provides Cloud automation Integrated NSX service deployment Embedded security deployment Provides P 2 V integrated services and security Inherent Services: LB, DHCP, NAT, VPN, and 3 rd party service insertion Customer Benefits No dependency on infrastructure for service mobility Choice of underlay network Opportunistic leveraging of fabric E.g -> ACI and network infrastructure isolation #NET1350BUR CONFIDENTIAL 31
NSX Over ACI Recommendations ACI Fabric Ideals Layer 2 Fabric: Single Tenant Fewer Contract Needs Map Static vsphere EPs Map NSX Edge to ACI Border Fabric Infrastructure Minimum Requirements: 1 Physical Domain 1 External Routing Domain 2 VLAN Pools (Int & Ext) 1 AEP (Leaf & Switch Policies, Int & Int Sel Policies, etc..) Tenant: NSX over ACI Tenant Separate tenant (not common) 1 Application (Network) Profile 4 EPGs (base epgs) 4 Bridge Domains, 1 VRF 2 L3Outs; North and South #NET1350BUR CONFIDENTIAL 32
NSX Over ACI: ACI Infrastructure NSX Overlay Compute VMKernel ACI Contracts ACI EPGs Layer 2 ACI Fabric VM1 VM3 VM5 VM2 VM4 Web LS APP LS DB LS Compute VDS Edge VDS VMworld 2017 Content: Not for VM6 MGMT vmotion Storage Transport Peer VLANs ACI Infrastructure Supports attachment of hosts Define phy domain of host attachment VLANs, switch interfaces, and policies in use publication Domains, Physical and External Create Application Profile Defines EPGs Networks Private Networks Bridge Domains External L2 and L3 connectivity #NET1350BUR CONFIDENTIAL 33
ACI Infrastructure Screen Shot 4 EPGs Specify Physical domain and define Static Ports along with a VLAN encapsulation Contracts unnecessary for Intra EPG communication #NET1350BUR CONFIDENTIAL 34
NSX Edge Mapping to ACI Border Leaf Compute Clusters Compute workloads Services (Tenant LB, NAT, etc) Distributed routing, switching, DFW Edge Clusters ECMP routing with ACI border leafs ESGs single active uplink per dvuplink Vmkernel VXLAN Overlay Compute ACI Spine EPG/VLAN 10 L3Out DC Core Edge North bound L3Outs L3 L2 Border Leafs EPG/VLAN 20 L3Out ECMP Edges Routing Adjacency VM1 VM3 VM5 Control VM #NET1350BUR CONFIDENTIAL 35
NSX Edge BGP AS Configuration (Prod) BGP Connectivity DC Core to ACI Border Leafs ebgp connection DC Core: AS 1 ACI Leafs: AS 65100 ACI Border Leafs to ESGs ebgp connection ACI Leafs: AS 65100 ESGs: 65014 ESGs to DLR CVM ebgp connection ESGs: 65014 DLR Control VM: 65013 ACI Spines MPBGP AS 65001 This AS must NOT be used by ACI for L3Out Neighboring ACI Spine EPG/VLAN 1209 Vmkernel VXLAN Overlay BGP 65001 L3Out E1 E2 Control VM BGP 65013 L3 L2 E3 L3Outs North BGP 1 Border Leafs EPG/VLAN 1210 L3Out E4 DC Core Edge Cluster BGP 65100 BGP 65014 #NET1350BUR CONFIDENTIAL 36
ACI Tenant Networking Networking Bridge domains VRFs or Private Networks vzany contract for EPG collection for VRF Allow all L3Out and IP Mgmt traffic to communicate externally External Routed Networks NSX to ACI border leafs ACI border leafs to Nexus DC cloud (not pictured) Default Route Leak Policy Allow Default route #NET1350BUR CONFIDENTIAL 37
ACI L3Outs NSX Edges L3 External Routing Per ACI Leaf config BG Peer defined Utilize defined SVI per leaf, per VLAN SVI s IP address per leaf node Define switch node, ports and transit vlan #NET1350BUR CONFIDENTIAL 38
ACI BGP Peer Connectivity BGP Peer Connectivity Profile Define BGP peer, BGP Prefix policy for prefix distribution, and relative BGP neighbor attributes #NET1350BUR CONFIDENTIAL 39
NSX Edge Mapping to ACI Border Leaf L3Out Edge Connectivity Transit VLANs 1209: 10.114.219.208/29 Edges: E1:210, E3:211 ACI SVI:.209 1210: 10.114.219.216/29 Edges: E2:.218, E3:.220 ACI SVI:.217 Vmkernel VXLAN Overlay Compute 10.114.219.208/29 VLAN 1209 ACI Spine E1 Edge Cluster Transit VXLAN E2 Control VM L3 L2 E3 DC Core L3Outs North Border Leafs 10.114.219.216/29 VLAN 1210 E4 ECMP Transit VXLAN 10.114.219.184/29 DLR Forwarding IP:.185 DLR Protocol IP:.186 E1 E4: (.187 190) #NET1350BUR CONFIDENTIAL 40
Multi Site Heterogenous DR Universal Logical Switches VMworld 2017 L2/L3 DCI Any L2/L3 DCI Policy Everywhere - Stateful Firewall - Network Introspection - Identity Firewall - Decoupled - Less replacement cycles Content: Not for publication #NET1350BUR CONFIDENTIAL 41
Agenda 1 NSX Anywhere 2 NSX Design 3 Nexus Switching Fabric 4 UCS Connectivity 5 NSX on ACI 6 Summary & Question #NET1350BUR CONFIDENTIAL 42
NSX Design Guides Reference Design: Deploying NSX with Cisco UCS and Nexus 9000 Infrastructure https://communities.vmware.com/docs/doc-29373 Design Guide for Vmware NSX running with a Cisco ACI Underlay Fabric https://communities.vmware.com/docs/doc-30849 NSX-V Multi-Site Options and Cross-VC NSX Design Guide https://communities.vmware.com/docs/doc-32552 VMworld 2017 VMware NSX for vsphere Network Virtualization Design Guide version 3.0 https://communities.vmware.com/docs/doc-27683 Content: Not for publication #NET1350BUR CONFIDENTIAL 43