NET1863BU NSX-T Advanced Architecture, Switching and Routing François Tallet, NSBU #VMworld #NET1863BU
Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. CONFIDENTIAL 2
Agenda 1 Architecture and Switching 2 Routing 3 Distributed and Edge Firewall Introduction to NSX-T Architecture [NET1510BU] - Andrew Voltmer, Group Product Line Manager, VMware - Dimitri Desmidt, Senior Technical Product Manager, VMware #NET1863BU CONFIDENTIAL 3
NSX-T Architecture & Switching
NSX Architecture and Components Cloud Consumption Management Plane Control Plane Data Plane ESXi HV KVM HV Management Plane (MP) Node VM form factor Transport Nodes NSX Edge (VM or Bare Metal) CCP Cluster Central Control Plane (CCP) Nodes- VM form factor VPN Layer 2 Bridge Self Service Portal OpenStack, Custom Concurrent configuration portal REST API entry-point UI Control-Plane Protocol Dynamic state Separation of Control and Data Plane High Performance Data Plane Scale-out Distributed Forwarding Model Physical Infrastructure 8
Switching Demo: Logical Switch Creation vcenter1 Web1 VIF1 vcenter2 172.16.10.11 172.16.10.12 ESXi HV1 Web2 VIF2 ESXi HV2 Virtual Interface (VIF): Compute manager object representing the VM vnic 9
Switching Demo: Logical Switch Creation On vcenter1 On vcenter2 vcenter1 Web1 VIF1 vcenter2 172.16.10.11 172.16.10.12 ESXi HV1 Web2 VIF2 ESXi HV2 Virtual Interface (VIF): Compute manager object representing the VM vnic 10
Switching Demo: Ping vcenter1 172.16.10.11 172.16.10.12 Web1 VIF1 ESXi HV1 vcenter2 Web2 VIF2 ESXi HV2 Virtual Interface (VIF): Compute manager object representing the VM vnic 11
NSX Architecture in Action Compute Manager (vcenter1) 1. Create Web-LS LS1 2. Configure Web-LS Compute Manager (vcenter2) Web-LS LIF1 VIF1 4. Attach Web1 to Web-LS Web1 ESXi HV1 TEP1 VIF1 Management Plane Node Central Control Plane Cluster 5. attach VIF1 to Web-LS ESXi HV2 TEP2 Web2 3. Advertise Web-LS to ESXi HVs VIF2 6. Configure LIF1 on Web- LS attached to VIF1 Tunnel End Point (TEP) Virtual Interface (VIF): Compute manager object representing the VM vnic Logical Interface (LIF): port on the logical switch 12
NSX Architecture in Action Compute Manager (vcenter1) Compute Manager (vcenter2) MAC@ Web-LS LS1 LIF1 VIF1 TEP IP Mac1 TEP1 Web1 ESXi HV1 TEP1 VIF1 LIF1 Management Plane Node Central Control Plane Cluster 9. LIF1 created on Web-LS ESXi HV2 TEP2 Web2 VIF2 7. Advertise Web-LS and LIF1 to CCP 8. Web-LS created, I m master 10. Mac1 associated to TEP1 Tunnel End Point (TEP) Virtual Interface (VIF): Compute manager object representing the VM vnic Logical Interface (LIF): port on the logical switch 13
Identify the VIF of a KVM Virtual Machine Web1 VIF1 LIF1 Web2 VIF3 UUID:? 172.16.10.11 172.16.10.12 172.16.10.13 ESXi HV1 or distribution VIF2 LIF2 ESXi HV2 Web3 VIF3 KVM HV3 15
Identify the VIF of a KVM Virtual Machine VIF3 UUID: 57601300-2e82-48c4-8c27-1e961ac70e79 172.16.10.11 172.16.10.12 172.16.10.13 Web1 VIF1 LIF1 ESXi HV1 or distribution Web2 VIF2 LIF2 ESXi HV2 Web3 VIF3 KVM HV3 16
Attach KVM Virtual Machine to a Logical Switch with Logical Port VIF3 UUID: 57601300-2e82-48c4-8c27-1e961ac70e79 172.16.10.11 172.16.10.12 172.16.10.13 Web1 VIF1 LIF1 ESXi HV1 or distribution Web2 VIF2 LIF2 ESXi HV2 Web3 VIF3 LIF3 KVM HV3 17
Ping KVM/ESXi 172.16.10.11 172.16.10.12 172.16.10.13 Web1 VIF1 LIF1 ESXi HV1 or distribution Web2 VIF2 LIF2 ESXi HV2 Web3 VIF3 LIF3 KVM HV3 18
Adding KVM Port Compute Manager (vcenter1) MAC@ 1. Attach VIF3 to Web-LS Compute Manager Web1 (vcenter2) 6. Mac TEP associations VIF1 advertised to HV3 Web-LS LS1 LIF1 VIF1 LIF2 VIF2 LIF3 VIF3 TEP IP Mac1 TEP1 Mac2 TEP2 Mac3 TEP3 LIF1 2. Configure LIF3 attached to VIF3 on Web-LS Management Plane Node Central Control Plane Cluster Web2 VIF2 LIF2 3. Advertise LIF3 Web3 5. Mac3 associated to TEP3 VIF3 LIF3 4. LIF3 created ESXi HV1 TEP1 ESXi HV2 TEP2 KVM HV3 TEP3 MAC@ TEP IP Mac1 TEP1 Mac2 TEP2 19
Unicast Packet Walk LS Web1 mac1 HV1 TEP1 Web2 Central Control Plane Cluster Overlay encapsulated frame Mac1? Mac1? Web3 TEP1 mac1 HV3 TEP3 MAC@ TEP IP Mac1 TEP1 Mac2 TEP2 Mac3 local MAC@ TEP IP Mac1 TEP1 Mac2 TEP2 Mac3 TEP3 Web3 sends a unicast to Web1 A lookup is made for Mac1 If it s a hit { Frame is encapsulated Frame is sent unicast to remote TEP } else { Frame is flooded } 20
BUM Traffic Handling : Unicast (MTEP) Traffic flooded from Web1 on HV1 on a Logical Switch Frame replication is achieved at two tiers, based on the TEP subnets HV2 Web1 HV1 TEP2 HV3 TEP1 TEP3 TEP1, TEP2, TEP3 have IP addresses in subnet A 1. HV1 replicates the frame to all TEPs in its subnet A 2. HV1 forwards the frame to one TEP in each remote subnet B & C 3. Remote TEPs in subnet B & C replicate the frame to other interested TEPs in their respective subnet. TEP4 TEP6 TEP7 TEP9 HV4 TEP5 HV6 HV7 TEP8 HV9 HV5 TEP4, TEP5, TEP6 have IP addresses in subnet B HV8 TEP7, TEP8, TEP9 have IP addresses in subnet C HV6 has no logical port in the logical switch 21
Flood and Learn The controller distribute Mac TEP association, but NSX can also do data plane learning Example of data plane learning of Mac1 of VM Web 1 from a flooded frame: Web1 Mac1 HV1 Web1 Mac1 MAC@ TEP IP HV1 TEP1 TEP1 L2 Payload Src Mac1: Dest Mac FF Inner Mac @s TEP4 HV4 Src IP:TEP1 Dest IP:TEP2 Tunnel Header Now, a more complex example (MTEP replication, as seen previous slide) Mac1 VMworld 2017 Mac1 TEP2 HV2 TEP5 HV5 MAC@ TEP IP Mac1 TEP1 Content: Not for publication Mac1 TEP4 wrong 22
Flood and Learn The controller distribute Mac TEP association, but NSX can also do data plane learning Example of data plane learning of Mac1 on HV5 from a frame flooded by VM Web1 Web1 Mac1 HV1 TEP1 L2 Payload Src Mac1: Dest Mac FF Inner Mac @s Src IP:TEP1 Dest IP:TEP4 Tunnel Header Now, a more complex example (MTEP replication, as seen previous slide) Solution: Carry some metadata identifying the source TEP in the encapsulation Web1 Mac1 TEP1 HV1 Mac1 S:Tep1 TEP4 HV4 Mac1 S:Tep1 TEP2 HV2 TEP5 HV5 MAC@ TEP IP Mac1 TEP1 MAC@ TEP IP Mac1 TEP1 23
Choice for NSX Overlay Encapsulation Metadata is critical to any distributed system, Encapsulations designed around hardware-based forwarding typically have fixed fields New features might require new metadata NSX is currently leveraging GENEVE as a tunneling mechanism (https://datatracker.ietf.org/doc/draft-ietf-nvo3-geneve/) It maintains the traditional offload capabilities offered by NICs for best performance Provides complete flexibility for inserting Metadata as Type Length Value (TLV) fields Note: Third party devices don t need to understand NSX tunnels Tools for looking inside GENEVE tunnels are available (Wireshark dissector for ex.) NSX can handle different types of tunnels simultaneously. 24
NSX-T Routing
Logical Routing Demo: Create Logical Router 172.16.20.11 172.16.10.11 App1 Web1 29
Logical Routing Demo: Create Logical Router 172.16.20.11 172.16.10.11 App1 app-ls to App-LS 172.16.20.1 web-ls Web1 to Web-Ls 172.16.10.1 Tenant1 Router 30
Logical Routing Demo: Traceroute 172.16.20.11 172.16.10.11 App1 app-ls to App-LS 172.16.20.1 web-ls Web1 to Web-LS 172.16.10.1 Tenant1 Router 31
Distributed Routing HV1 HV2 172.16.20.1 172.16.10.1 172.16.20.1 172.16.10.1 App1 Web2 Web1 32
Distributed Routing HV1 HV2 172.16.20.1 172.16.10.1 172.16.20.1 172.16.10.1 In-kernel routing: this is not a VM or distribution App1 Web2 Web1 33
Traceflow Demo App1 app-ls to App-LS 172.16.20.1 web-ls Web1 to Web-LS 172.16.10.1 Tenant1 Router 34
Distributed Routing with a Centralized Component Introducing the Edge Node HV1 Edge Node 172.16.20.1 172.16.10.1 Physical Port HV2 172.16.20.1 172.16.10.1 172.16.20.1 172.16.10.1 VMworld 2017 Content: Not for Physical Router publication App1 Web2 Web1 35
Distributed Routing with a Centralized Component Introducing the Edge Node Intra-Router tunnel HV1 Edge Node 172.16.20.1 172.16.10.1 HV2 172.16.20.1 172.16.10.1 172.16.20.1 172.16.10.1 VMworld 2017 Content: Not for Physical Port Physical Router publication App1 Web2 Web1 36
Edge Nodes Edge Nodes are appliances with pool of capacity for handling services that cannot be distributed. Example of services: Peering with the physical infrastructure NAT DHCP Server, MetaData Proxy Edge Firewall Edge Node1 Edges are available in 2 form factors Bare Metal & VM VMworld 2017 Both leverage Intel s DPDK (DataPlane Development ToolKit) DHCP Edge Node2 High forwarding performance Linear performance More increase on NSX-T by addition Performance: of cores. NSX Performance Deep Dive [NET1343BU] - Samuel Kommu, Sr. Technical Product Manager, Vmware Edge Cluster Edge Node3 Those are services, not VMs Content: Not for publication 37
Two-Tier Routing Tenants/CMP vma vmb vmc vmd Admin VMworld 2017 Provider Logical Router Tier0 LR Role Attach to the physical routing infrastructure Manual management Tenant Logical Router Tier1 LR Role Per tenant first hop router Content: Not for publication Cloud Management Platform (CMP) driven management No dynamic routing between tiers: NSX distributes the appropriate routes 39
2-Tier Routing is Distributed Tier0 and Tier1 routers are also instantiated on the hypervisors in order to prevent hair-pinning Fully distributed architecture : as much routing as possible is performed upfront at the source vma HV1 vmd Again: the forwarding tables of the distributed components are populated by NSX There is no routing protocol involved for communication within NSX HV2 vmd 40
Gateway Service to the Physical Infrastructure Peering to a single Tier 0 router from the perspective of the physical infrastructure Tier0 logical router supports: Static routes towards physical ebgp towards physical VMworld 2017 Content: Not for ECMP supported using static routes and ebgp BFD towards physical to protect against link failures ebgp Edge Node ebgp Edge Node ebgp Edge Node ebgp publication Edge Node Edge Cluster 41
ECMP with Physical Infrastructure VMworld 2017 Content: Not for Edge Node Edge Cluster publication Edge Node Logical view Physical view HV1 vma vma 42
BGP Demo: Setting Up BGP Neighbor or Physical Router Edge TN1 192.168.240.1 Uplink-LS1 distribution Uplink1:192.168.240.3 edgecluster1 T0 Router Tenant1 Router 172.16.10.1/24 172.16.20.1/24 Web2 Web2 Web2 App1 44
BGP Demo: Setting Up BGP Neighbor or EBGP Physical Router Edge TN1 192.168.240.1 Uplink-LS1 Uplink1:192.168.240.3 edgecluster1 AS200 distribution AS100 T0 Router Tenant1 Router 172.16.10.1/24 172.16.20.1/24 Web2 Web2 Web2 App1 45
BGP Demo: Redistribute NSX Static Routes or EBGP Physical Router Edge TN1 192.168.240.1 Uplink-LS1 BGP Advertise NSX Static edgecluster1 AS200 distribution Uplink1:192.168.240.3 AS100 T0 Router Tenant1 Router 172.16.10.1/24 172.16.20.1/24 Web2 Web2 Web2 App1 46
BGP Demo: Attach Tier1 Router to Tier0 Router or EBGP Physical Router Edge TN1 192.168.240.1 Uplink-LS1 BGP Advertise NSX Static edgecluster1 AS200 distribution Uplink1:192.168.240.3 100.64.0.0/31 100.64.0.1/31 AS100 T0 Router Tenant1 Router 172.16.10.1/24 172.16.20.1/24 Web2 Web2 Web2 App1 47
BGP Demo: Show BPG Routes on Physical Infrastructure or EBGP Physical Router Edge TN1 192.168.240.1 Uplink-LS1 BGP Advertise NSX Static edgecluster1 AS200 distribution Uplink1:192.168.240.3 100.64.0.0/31 100.64.0.1/31 AS100 T0 Router Tenant1 Router 172.16.10.1/24 172.16.20.1/24 Web2 Web2 Web2 App1 48
BGP Demo: Advertise Connected Routes from Tier1 More on NSX-T Routing: NSX Logical Routing [NET1416BU] - Pooja Patel, Senior Manager, Technical Product Management, VMware - Jerome Catrouillet, Senior Product Line Manager, VMware VMworld 2017 EBGP Edge TN1 172.16.10.1/24 Physical Router 192.168.240.1 Uplink-LS1 BGP Advertise NSX Static edgecluster1 AS200 Uplink1:192.168.240.3 Advertise connected 100.64.0.0/31 100.64.0.1/31 AS100 T0 Router Content: Not for publication Tenant1 Router 172.16.20.1/24 Web2 Web2 Web2 App1 49
NSX Firewall
Micro-Segmentation with Distributed Firewall (DFW) Web1 Web2 Web3 DB1 App1 App2 NAT01 Each VM is its own perimeter Policies align with logical groups Prevents threats from spreading DFW available on ESXi and KVM 51
Micro-Segmentation Demo: Traceflow Web1 Web3 52
Micro-Segmentation Demo: NSGroup VMworld 2017 Content: Not for Tags can be dynamically applied to: - Logical Switch - Logical Ports publication - VMs NSGroups can be created by combining tags and VM names. 53
Micro-Segmentation Demo: Preventing Web to Web Traffic 54
Micro-Segmentation Demo: New Traceflow Web1 Web3 55
Adding Firewall Rule 1. Drop Web to Web communication Drop 172.16.10.11 172.16.10.12 Drop 172.16.10.12 172.16.10.13 Drop 172.16.10.11 172.16.10.13 LS 1 vma ESXi HV1 TEP1 VIF A LIF A 2. Rule saved in database Management Plane Node Central Control Plane Cluster ESXi HV2 TEP2 vmb VIF B LIF B KVM HV3 TEP3 3. Push rule to CCP vmc VIF C LIF C 5. Rule programmed in datapath of affected hosts 56
Edge Firewall Stateful Firewall we need to see both directions of the traffic Practically, the firewall has to be centralized For the DFW, the firewall is naturally centralized on the LIF where the VM vnics attach For a Firewall on the uplink of a router, we ll use an Edge node (same as peering to physical) VMworld 2017 Edge Firewall vma HV1 vmc Context Content: Not for publication Edge Node HV2 vmd 57
Packet Walk with Edge Firewall Service Edge Firewall HV1 Edge Node The FW sees traffic both ways HV2 vma vmc vmd More on Security: The Future of Networking and Security with NSX-T [NET1821BU] - Bruce Davie - CTO VMware 58
Wrapping Up
NSX-T This presentation was on understanding how basic NSX-T networking capabilities worked In particular NSX-T Decouple networking from the hardware and from vcenter Is Multi-Hypervisor Uses high performance Edge Nodes for its centralized services Interconnects its components with minimal user intervention SPL182601U VMware NSX-T Getting Started SPL182602U VMware NSX-T - NSX-T with Kubernetes 60