NET1863BE NSX-T Advanced Architecture Concepts Dimitri Desmidt / Yasen Simeonov September 2017
Agenda Introduce NSX-T: Architecture Switching Routing Firewall Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. This presentation is shared under NDA Introduction to NSX-T Architecture [NET1510BU] - Andrew Voltmer, Group Product Line Manager, VMware - Dimitri Desmidt, Senior Technical Product Manager, VMware 2
NSX-T Architecture & Switching
NSX Architecture and Components Cloud Consumption Management Plane Control Plane Data Plane vswitch ESXi HV CCP Cluster Central Control Plane (CCP) Nodes- VM form factor vswitch KVM HV Management Plane (MP) Node VM form factor VPN Transport Nodes NSX Edge (VM or Bare Metal) Layer 2 Bridge Self Service Portal OpenStack, Custom Concurrent configuration portal REST API entry-point UI Control-Plane Protocol Dynamic state Separation of Control and Data Plane High Performance Data Plane Scale-out Distributed Forwarding Model Physical Infrastructure 4
Switching Demo: Logical Switch Creation vcenter1 Web1 VIF1 vcenter2 172.16.10.11 172.16.10.12 vswitch ESXi HV1 Web2 VIF2 vswitch ESXi HV2 Virtual Interface (VIF): Compute manager object representing the VM vnic 5
Switching Demo: Logical Switch Creation On vcenter1 On vcenter2 vcenter1 Web1 VIF1 vcenter2 172.16.10.11 172.16.10.12 vswitch ESXi HV1 Web2 VIF2 vswitch ESXi HV2 Virtual Interface (VIF): Compute manager object representing the VM vnic 6
Switching Demo: Ping vcenter1 172.16.10.11 172.16.10.12 Web1 VIF1 vswitch ESXi HV1 vcenter2 Web2 VIF2 vswitch ESXi HV2 Virtual Interface (VIF): Compute manager object representing the VM vnic 7
NSX Architecture in Action Compute Manager (vcenter1) 1. Create Web-LS LS1 2. Configure Web-LS Compute Manager (vcenter2) Web-LS LIF1 VIF1 4. Attach Web1 to Web-LS Web1 vswitch ESXi HV1 TEP1 VIF1 Management Plane Node Central Control Plane Cluster 5. vswitch attach VIF1 to Web-LS ESXi HV2 TEP2 Web2 3. Advertise Web-LS to ESXi HVs VIF2 6. Configure LIF1 on Web- LS attached to VIF1 Tunnel End Point (TEP) Virtual Interface (VIF): Compute manager object representing the VM vnic Logical Interface (LIF): port on the logical switch 8
NSX Architecture in Action Compute Manager (vcenter1) Compute Manager (vcenter2) MAC@ Web-LS LS1 LIF1 VIF1 TEP IP Mac1 TEP1 Web1 vswitch ESXi HV1 TEP1 VIF1 LIF1 Management Plane Node Central Control Plane Cluster 9. vswitch LIF1 created on Web-LS ESXi HV2 TEP2 Web2 VIF2 7. Advertise Web-LS and LIF1 to CCP 8. Web-LS created, I m master 10. Mac1 associated to TEP1 Tunnel End Point (TEP) Virtual Interface (VIF): Compute manager object representing the VM vnic Logical Interface (LIF): port on the logical switch 9
Unicast Packet Walk (ESXi) LS Web1 mac1 HV1 TEP1 Central Control Plane Cluster Overlay encapsulated frame Web2 TEP1 mac1 HV2 TEP2 MAC@ TEP IP Mac2 local Mac1 TEP1 MAC@ TEP IP Mac1 TEP1 Mac2 TEP2 Mac1?Mac1 TEP1 Web2 sends a unicast to Web1 ESXi asks Controller Controller gives information 10
Identify the VIF of a KVM Virtual Machine Web1 VIF1 LIF1 Web2 VIF3 UUID:? 172.16.10.11 172.16.10.12 172.16.10.13 vswitch ESXi HV1 or distribution VIF2 LIF2 vswitch ESXi HV2 Web3 VIF3 vswitch KVM HV3 11
Identify the VIF of a KVM Virtual Machine VIF3 UUID: 57601300-2e82-48c4-8c27-1e961ac70e79 172.16.10.11 172.16.10.12 172.16.10.13 Web1 VIF1 LIF1 vswitch ESXi HV1 or distribution Web2 VIF2 LIF2 vswitch ESXi HV2 Web3 VIF3 vswitch KVM HV3 12
Attach KVM Virtual Machine to a Logical Switch with Logical Port VIF3 UUID: 57601300-2e82-48c4-8c27-1e961ac70e79 172.16.10.11 172.16.10.12 172.16.10.13 Web1 VIF1 LIF1 vswitch ESXi HV1 or distribution Web2 VIF2 LIF2 vswitch ESXi HV2 Web3 VIF3 LIF3 vswitch KVM HV3 13
Ping KVM/ESXi 172.16.10.11 172.16.10.12 172.16.10.13 Web1 VIF1 LIF1 vswitch ESXi HV1 or distribution Web2 VIF2 LIF2 vswitch ESXi HV2 Web3 VIF3 LIF3 vswitch KVM HV3 14
Adding KVM Port Compute Manager (vcenter1) MAC@ 1. Attach VIF3 to Web-LS Compute Manager Web1 (vcenter2) 6. Mac TEP associations VIF1 advertised to HV3 Web-LS LS1 LIF1 VIF1 LIF2 VIF2 LIF3 VIF3 TEP IP Mac1 TEP1 Mac2 TEP2 Mac3 TEP3 LIF1 2. Configure LIF3 attached to VIF3 on Web-LS Management Plane Node Central Control Plane Cluster Web2 VIF2 LIF2 3. Advertise LIF3 Web3 5. Mac3 associated to TEP3 VIF3 LIF3 4. LIF3 created vswitch vswitch vswitch ESXi HV1 TEP1 ESXi HV2 TEP2 KVM HV3 TEP3 MAC@ TEP IP Mac1 TEP1 Mac2 TEP2 15
Unicast Packet Walk (KVM) LS Web1 mac1 HV1 TEP1 Web2 Central Control Plane Cluster Overlay encapsulated frame Mac1? Mac1? Web3 TEP1 mac1 HV3 TEP3 MAC@ TEP IP Mac1 TEP1 Mac2 TEP2 Mac3 local MAC@ TEP IP Mac1 TEP1 Mac2 TEP2 Mac3 TEP3 Web3 sends a unicast to Web1 A lookup is made for Mac1 If it s a hit { Frame is encapsulated Frame is sent unicast to remote TEP } else { Frame is flooded } 16
BUM Traffic Handling : Unicast (MTEP) Traffic flooded from Web1 on HV1 on a Logical Switch Frame replication is achieved at two tiers, based on the TEP subnets HV2 Web1 HV1 TEP2 HV3 TEP1 TEP3 TEP1, TEP2, TEP3 have IP addresses in subnet A 1. HV1 replicates the frame to all TEPs in its subnet A 2. HV1 forwards the frame to one TEP in each remote subnet B & C 3. Remote TEPs in subnet B & C replicate the frame to other interested TEPs in their respective subnet. TEP4 TEP6 TEP7 TEP9 HV4 TEP5 HV6 HV7 TEP8 HV9 HV5 TEP4, TEP5, TEP6 have IP addresses in subnet B HV8 TEP7, TEP8, TEP9 have IP addresses in subnet C HV6 has no logical port in the logical switch 17
Flood and Learn The controller distribute Mac TEP association, but NSX can also do data plane learning Example of data plane learning of Mac1 of VM Web 1 from a flooded frame: Web1 Mac1 HV1 Web1 Mac1 MAC@ TEP IP HV1 TEP1 TEP1 L2 Payload Src Mac1: Dest Mac FF Inner Mac @s TEP4 HV4 Src IP:TEP1 Dest IP:TEP2 Tunnel Header Now, a more complex example (MTEP replication, as seen previous slide) Mac1 VMworld 2017 Mac1 TEP2 HV2 TEP5 HV5 MAC@ TEP IP Mac1 TEP1 Content: Not for publication Mac1 TEP4 wrong 18
Flood and Learn The controller distribute Mac TEP association, but NSX can also do data plane learning Example of data plane learning of Mac1 on HV5 from a frame flooded by VM Web1 Web1 Mac1 HV1 TEP1 L2 Payload Src Mac1: Dest Mac FF Inner Mac @s Src IP:TEP1 Dest IP:TEP4 Tunnel Header Now, a more complex example (MTEP replication, as seen previous slide) Solution: Carry some metadata identifying the source TEP in the encapsulation Web1 Mac1 TEP1 HV1 Mac1 S:Tep1 TEP4 HV4 Mac1 S:Tep1 TEP2 HV2 TEP5 HV5 MAC@ TEP IP Mac1 TEP1 MAC@ TEP IP Mac1 TEP1 19
Choice for NSX Overlay Encapsulation Metadata is critical to any distributed system, Encapsulations designed around hardware-based forwarding typically have fixed fields New features might require new metadata NSX is currently leveraging GENEVE as a tunneling mechanism (https://datatracker.ietf.org/doc/draft-ietf-nvo3-geneve/) It maintains the traditional offload capabilities offered by NICs for best performance Provides complete flexibility for inserting Metadata as Type Length Value (TLV) fields Note: Third party devices don t need to understand NSX tunnels Tools for looking inside GENEVE tunnels are available (Wireshark dissector for ex.) NSX can handle different types of tunnels simultaneously. 20
NSX-T Routing
Logical Routing Demo: Create Logical Router 172.16.20.11 172.16.10.11 App1 Web1 22
Logical Routing Demo: Create Logical Router 172.16.20.11 172.16.10.11 App1 app-ls to App-LS 172.16.20.1 web-ls Web1 to Web-Ls 172.16.10.1 Tenant1 Router 23
Logical Routing Demo: Traceroute 172.16.20.11 172.16.10.11 App1 app-ls to App-LS 172.16.20.1 web-ls Web1 to Web-LS 172.16.10.1 Tenant1 Router 24
Traceflow Demo App1 app-ls to App-LS 172.16.20.1 web-ls Web1 to Web-LS 172.16.10.1 Tenant1 Router 25
Edge Nodes Edge Nodes are appliances with pool of capacity for handling services that cannot be distributed. Example of services: Peering with the physical infrastructure NAT DHCP Server, MetaData Proxy Edge Firewall Edges are available in 2 form factors Bare Metal & VM VMworld 2017 Both leverage Intel s DPDK (DataPlane Development ToolKit) High forwarding performance Linear performance increase by addition of cores. Edge Node1 DHCP Edge Node2 Edge Cluster Edge Node3 Those are services, not VMs Content: Not for publication 26
Two-Tier Routing Tenants/CMP vma vmb vmc vmd Admin VMworld 2017 Provider Logical Router Tier0 LR Role Attach to the physical routing infrastructure Manual management Tenant Logical Router Tier1 LR Role Per tenant first hop router Content: Not for publication Cloud Management Platform (CMP) driven management No dynamic routing between tiers: NSX distributes the appropriate routes 27
Logical Router - Components Distributed Router () Distributed Component to provide E-W Routing in Logical Space Logical Router (LR) SR Services Router (SR) Service Component to provide on/off ramp gateway functionality Spans all the Transport Nodes (HVs and Edges) Offers Stateful & Centralized Services: NAT, BGP Runs as a Kernel Module in ESXi & OVS in KVM Runs only in Edge Nodes
ECMP Detailed Internal View RouterLink LS VM Tier0 Tenant1 HV1 Web LS 169.0.0.0/28 Network Intra-Tier Transit LS Tier0 Tenant1 HV2 Uplink-LS Tier0 SR Tier0 Tenant1 EN1 BGP to External Tier0 SR Tier0 Tenant1 EN2 Confidential 29
Packet Flow Logical Topology Tenant-A T1 LR-1 Router-Link LS 6000 T0 LR-1 VLAN LS 100 Physical Networks Router-Link LS 6001 T1 LR-2 Tenant-B LS-5000 LS- 5001 LS-5002 LS-5003 VM1 VM2 VM3 VM4 172.16.10.11 172.16.20.11 172.16.30.11 172.16.40.11
Communication within Tenants Same Host 5000 172.16.10.11 172.16.20.11 VM1 VM2 1 3 2 T1 LR-1 5001 T0 LR-1 4 VTEP T1 LR-2 NSX vswitch LS-5000 LS-5001 Transport Node A Transport Node B
Communication within Tenants Different Host 5000 2 172.16.10.11 172.16.20.11 VM1 VM2 1 T1 LR-1 T0 LR-1 5001 3 NSX vswitch 5 VTEP T1 LR-2 VTEP T0 LR-1 LS-5000 LS-5001 T1 LR-1 T1 LR-2 Transport Node A 4 GENEVE Encapsulated Traffic Transport Node B
Communication between Tenants Same Host 5000 2 172.16.10.11 172.16.40.11 VM1 VM4 1 6 T1 LR-1 NSX vswitch T0 LR-1 3 4 VTEP T1 LR-2 5 5003 LS-5000 LS-5003 Transport Node A Transport Node B
Communication between Tenants Different Hosts 5000 2 172.16.10.11 172.16.40.11 VM1 VM4 1 T1 LR-1 NSX vswitch T0 LR-1 4 3 5003 VTEP T1 LR-2 VTEP LS-5000 LS-5003 6 NSX vswitch T0 LR-1 5003 T1 LR-1 T1 LR-2 Transport Node A 5 GENEVE Encapsulated Traffic Transport Node B
Ingress from Physical Network 172.16.10.11 VM1 LS-5000 LS-5003 6 5000 NSX vswitch T0 LR-1 T1 LR-1 T1 LR-2 4 5000 T1 LR-1 T0 LR-1 3 VTEP VTEP 2 SR T1 LR-2 1 192.168.100.10 Transport Node A 5 GENEVE Encapsulated Traffic EDGE Transport
Egress to Physical Network 172.16.10.11 VM1 1 LS-5000 LS-5003 NSX vswitch 5000 2 T1 LR-1 T0 LR-1 3 4 T1 LR-1 T0 LR-1 T1 LR-2 VTEP VTEP SR T1 LR-2 7 6 192.168.100.10 Transport Node A 5 GENEVE Encapsulated Traffic EDGE Transport Node
Flowcache datapath VM Flowcache Lookup Slow-path Swsec Firewall Overlay vswitch Routing vswitch Encap VM Firewall Swsec Overlay vswitch Routing vswitch Encap Fast-path Single lookup Add 2 new Iochains (lookup and fastpath) Initial packet will take slowpath Fastpath Iochain will install flow Subsequent packets will take fast path VMworld 2017 Flowcache Fastpath Software offloads Software offloads Content: Not for publication Pnic Pnic 37
Optimized datapath with flowcache TX fastpath App 1 App 2 FC-lookup VM to VM FC-fastpath Host Virtual Switch RX fastpath Optimized datapath VM to UPLINK UPLINK to VM VM to VM FC-fastpath FC-lookup 38
NSX Firewall
Micro-Segmentation with Distributed Firewall (DFW) Web1 Web2 Web3 DB1 App1 App2 NAT01 Each VM is its own perimeter Policies align with logical groups Prevents threats from spreading DFW available on ESXi and KVM 40
Micro-Segmentation Demo: Traceflow Web1 Web3 41
Micro-Segmentation Demo: NSGroup VMworld 2017 Content: Not for Tags can be dynamically applied to: - Logical Switch - Logical Ports publication - VMs NSGroups can be created by combining tags and VM names. 42
Micro-Segmentation Demo: Preventing Web to Web Traffic 43
Micro-Segmentation Demo: New Traceflow Web1 Web3 44
NSX-T DFW: Mgmt. Plane, Control Plane & Data-Plane roles Management Plane Distributed Data Plane Control Plane VMworld 2017 Content: Not for 2 Rules expanded and converted to IP addresses 3 Rules programmed in the data-path publication 1 Policy input validations storing rules/section/groups 45
Transport Node - ESXi User Space Kernel Space MP MPA nsxa netcpa vdl2 vdrb bfd swsec kcp vsip traceflo w LCP ipfix CCP nsxvswitch CONFIDENTIAL ESXi 46
DFW Implementation on KVM VMworld 2017 Content: Not for NSX-Agent Primary LCP component on KVM Receives DFW config from VSFWD Sends wiring implementation (OpenFlow) to OVS-VswitchD Ovs-VswitchD / Ovs-ko publication Connection tracking and Log-and-drop/allow action to implement DFW features Packet forawarding Part of Open vswitch distributions Ovs-fwd Responsible for logging and reject action Closed source component Linux Contrack and conntrack utilities Part of standard Open vswitch distribution Handle packet forwarding
Where to Get Started Engage and Learn Join VMUG for exclusive access to NSX vmug.com/vmug-join/vmug-advantage Connect with your peers communities.vmware.com Find NSX Resources vmware.com/products/nsx Network Virtualization Blog blogs.vmware.com/networkvirtualization Try VMworld 2017 Experience Dozens of Unique NSX Sessions Spotlights, breakouts, quick talks & group discussions Visit the VMware Booth Product overview, use-case demos Visit Technical Partner Booths Integration demos Infrastructure, security, operations, visibility, and more Content: Not for publication Meet the Experts Join our Experts in an intimate roundtable discussion Take Free Hands-on Labs Test drive NSX yourself with expert-led or self-paces hands-on labs labs.hol.vmware.com Training and Certification Several paths to professional certifications. Learn more at the Education & Certification Lounge. vmware.com/go/nsxtraining 48