NET1522BU Kubernetes Networking with NSX-T Deep Dive #VMworld #NET1522BU
Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. CONFIDENTIAL 2
NET1522BU Kubernetes Networking with NSX-T Deep Dive Yves Fauser / Yasen Simeonov #VMworld #NET1522BU
Agenda 1 NSX-T Overview 2 Kubernetes Overview 3 NSX-T & Kubernetes Integration 4 Demo #NET1522BU CONFIDENTIAL 4
NSX-T Overview
NSX Vision: Driving NSX Everywhere Managing security and connectivity for many heterogeneous end points Branch offices/edge computing/iot vcloud Air Network Cloud VMworld 2017 Content: Not for publication New app frameworks On-premises data center End users Automation IT at the speed of business Security Inherently secure infrastructure Application Continuity Data center anywhere #NET1522BU CONFIDENTIAL 6
NSX-T Architecture NSX Architecture and Components Cloud Consumption Management Plane Control Plane Data Plane NSX Manager Management Plane (MP) Node VM form factor Central Control Plane (CCP) Nodes- VM form factor ESXi (+ kernel modules) NSX Controllers Transport Nodes VMworld 2017 Content: Not for KVM (+ kernel modules) Self Service Portal OpenStack, K8s, Custom Concurrent configuration portal REST API entry-point UI publication Talks to Dataplane over a Control-Plane Protocol Separation of Control and Data Plane High Performance Data Plane Scale-out Distributed Forwarding Model Hypervisors NSX Edge (L3 + Adv Services) L2 Bridge (L2 Overlay- VLAN) Physical Infrastructure #NET1522BU CONFIDENTIAL 7
NSX-T Architecture Operations Workflow Configuration is persisted Configuration is pushed to CCP MP Node CCP Node CCP Node CCP Node VMworld 2017 Content: Not for X User makes a configuration publication Configuration is realized MPA LCP Transport Node MPA LCP Transport Node MPA LCP Transport Node #NET1522BU CONFIDENTIAL 8
Data Plane Improved performance and resiliency Designed for multi-tenancy and scale Tenants/CMP GENEVE Tunnel TEP: Overlay Tunnel End Point (with its own IP address) Admin p1 TEP vswitch 1 Overlay Transport Zone p2 HV TN1 New distributed edge architecture with increased performance with DPDK p1 HV TN1 Edge Node p2 TEP vswitch 2 Edge Node Edge Node Edge Node Edge Cluster Next gen overlay maintaining performance with increased flexibility #NET1522BU CONFIDENTIAL 9
NSX-T VMworld Session & Lab NSX-T Breakout Session Introduction to NSX-T Architecture NET1510BU (US) NET1510BE (Europe) VMworld 2017 NSX-T Hands On Lab VMware NSX-T - Getting Started SPL182601U (US) SPL182601E (Europe) Content: Not for publication #NET1522BU CONFIDENTIAL 10
Kubernetes Overview
What Is Kubernetes? Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure. #NET1522BU CONFIDENTIAL 12
Kubernetes Components > _ Kubectl CLI K8s master K8s master dashboard Controller Manager Scheduler K8s Master(s) K8s Nodes K8s API Server K8s Master Key-Value Store K8s node K8s node K8s node K8s node kubelet Kube-proxy c runtime K8s Cluster Consists of Master(s) and Nodes K8s Master Components API Server Scheduler Controller Manager Dashbord K8s Node Components Kubelet Kube-Proxy Containers Runtime (Docker or Rocket) #NET1522BU CONFIDENTIAL 13
Kubernetes Pod Pod 10.24.0.0/16 nginx tcp/80 10.24.0.2 mgmt tcp/22 pause container ( owns the IP stack) logging udp/514 IPC External IP Traffic A Pod is a group of one or more containers that shares an IP address and a Data Volume #NET1522BU CONFIDENTIAL 14
Kubernetes Namespace Namespace: foo Base URI: /api/v1/namespaces/foo redis-master Pod: /api/v1/namespaces/foo/pods/redis-master redis service: /api/v1/namespaces/foo/services/redis-master Namespace: bar Base URI: /api/v1/namespaces/bar redis-master Pod: /api/v1/namespaces/bar/pods/redis-master VMworld 2017 redis service: /api/v1/namespaces/bar/services/redis-master Namespaces are a way to divide cluster resources between multiple uses They can be considered as Tenants They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Names Overlapping Content: Not for publication #NET1522BU CONFIDENTIAL 15
K8s Load Balancing 10.24.0.5/16 East-West Load Balancing Web Front-End Pods redis-slave svc Redis Slave Pods 172.30.0.24 Nginx HAProxy etc. LB Pods Web Front-End (e.g. Apache) Pods North-South Load Balancing http://*.bikeshop.com Web Front-End Ingress East-West Load Balancing is provided through K8s Service using ClusterIP & IPTables Can be achieved through K8s Ingress or External third Party Load Balancer using NodePort #NET1522BU CONFIDENTIAL 16
Kubernetes Networking Topologies Flat routed topology ip route 10.24.1.0/24 10.240.0.3 ip route 10.24.2.0/24 10.240.0.4 int eth0 int cbr0 10.240.0.3 net.ipv4.ip_forward=1 10.24.1.1/24 10.24.1.2 10.24.1.3 10.24.1.4 Node int eth0 int cbr0 10.240.0.4 net.ipv4.ip_forward=1 10.24.2.1/24 10.24.2.2 10.24.2.3 10.24.2.4 Every Node is an IP Router and responsible for its Pod Subnet Subnets are associated with Nodes, not Tenants Physical Network Configuration is required Node #NET1522BU CONFIDENTIAL 17
Kubernetes Networking Topologies Node-to-Node overlay topology int eth0 int cbr0 10.240.0.3 net.ipv4.ip_forward=1 10.24.1.1/24 10.24.1.2 10.24.1.3 10.24.1.4 Node Key-Value Store int eth0 int cbr0 10.240.0.4 net.ipv4.ip_forward=1 10.24.2.1/24 10.24.2.2 10.24.2.3 10.24.2.4 Node Overlays are typically used to avoid Physical Network Configuration Overlay #NET1522BU CONFIDENTIAL 18
NSX-T and Kubernetes Integration
NSX-T K8s Integration Namespaces & Pods admin@k8s-master:~$ kubectl create namespace foo namespace foo" created admin@k8s-master:~$ kubectl create namespace bar namespace bar" created admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo deployment "nginx-foo" created admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bar deployment "nginx-bar" created VMworld 2017 NSX / K8s topology Namespace: foo NAT boundary NAT boundary 10.24.0.0/24 10.24.1.0/24 10.24.2.0/24 Namespace: bar Content: Not for publication K8s Masters K8s nodes #NET1522BU CONFIDENTIAL 20
NSX-T K8s Integration Routed Namespaces admin@k8s-master:~$ vim no-nat-namespace.yaml apiversion: v1 kind: Namespace metadata: name: no-nat-namespace annotations: ncp/no_snat: "true admin@k8s-master:~$ kubectl create f no-nat-namespace.yaml namespace no-nat-namespace" created VMworld 2017 admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx n no-nat-namespace deployment "nginx-k8s" created NSX / K8s topology Namespace: no-nat-namespace 114.4.10.0/26 Direct Routing Content: Not for publication 114.4.10.64/26 K8s Masters K8s nodes #NET1522BU CONFIDENTIAL 21
NSX-T K8s Integration Pods Micro-Segmentations Option1: Predefined Label Based Rules admin@k8s-master:~$ kubectl label pods nginx-foo-3492604561-nltrf secgroup=web -n foo Pod "nginx-nsx-3492604561-nltrf" labeled admin@k8s-master:~$ kubectl label pods nginx-bar-2789337611-z09x2 secgroup=db -n bar pod "nginx-k8s-2789337611-z09x2" labeled admin@k8s-master:~$ kubectl get pods --all-namespaces -Lsecgroup NAMESPACE NAME READY STATUS RESTARTS AGE SECGROUP k8s nginx-foo-2789337611-z09x2 1/1 Running 0 58m web nsx nginx-bar-3492604561-nltrf 1/1 Running 0 1h db Security Groups are defined in NSX with ingress and egress policy Each Security Group could be microsegmented to protect Pods from each other VMworld 2017 NSX / K8s topology Content: Not for publication Namespace: foo NAT boundary NAT boundary Namespace: bar 10.24.0.0/24 10.24.1.0/24 114.4.10.0/26 DB Web #NET1522BU CONFIDENTIAL 22
NSX-T K8s Integration Pods Micro-Segmentations Option 2: K8s Network Policy admin@k8s-master:~$ vim nsx-demo-policy.yaml apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: nsx-demo-policy spec: podselector: matchlabels: app: web ingress: - from: - namespaceselector: matchlabels: ncp/project: db ports: - port: 80 protocol: TCP admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml VMworld 2017 State: released on K8s 1.7 (Beta on 1.6) Capability: Using Network Policy, users can define firewall rules to allow traffic into and out of a Namespace, and between Pods. The network policy is a Namespace property. The default is drop. NSX / K8s topology Namespace: foo NAT boundary Routed 10.24.0.0/24 10.24.1.0/24 114.4.10.0/26 Web Label: app=web Namespace: bar Content: Not for publication DB Label: app=db #NET1522BU CONFIDENTIAL 23
NSX-T K8s Integration Pods Micro-Segmentations Option 2: K8s Network Policy $ kubectl create -f nsx-demo-policy.yaml Once the Network Policy is applied, NSX will dynamically create source & destination Security Groups and apply the right policy Dynamic Creation of Security Groups Dynamic Creation of Security Policy based on k8s Network Policy #NET1522BU CONFIDENTIAL 24
NSX-T K8s Integration Pods Micro-Segmentations Firewalling in Kubernetes Micro-Segmentation in K8s: The data model to describe segmentation policies between Namespaces, and within namespaces is called Network Policies and is released on Kubernetes 1.7 (Beta on 1.6) K8s Network Policy NSX could utilize K8s Network Policies to define Dynamic Security Groups & Policies Capabilities are limited to K8s Network Policy capabilities VMworld 2017 Firewalling in NSX / K8s Pre-Defined Label based rules Security Groups & Policies could be predefined on NSX. Labels are used to specify Pods Membership Content: Not for publication Mapping of IP based groups, egress rules, VM based matching could be available to be used in the policy definition The NSX / K8s integration intends to support both the pre-defined label based rules and K8s network policy #NET1522BU CONFIDENTIAL 25
East-West Load Balancing K8s master K8s master dashboard Controller Manager Scheduler NSX CNI Plugin K8s API Server K8s Master OVS Node VM NSX Kube Proxy K8s Services are delivered through NSX Kube-Proxy Delivered as a container image, so that it can be run as a Kubernetes Daemon-Set on the Nodes NSX Kube-Proxy would replace the native distributed east-west load balancer in Kubernetes called Kube-Proxy OpenVSwitch (OVS) load-balancing is used Pods #NET1522BU CONFIDENTIAL 26
North-South Load Balancing Once an Ingress Controller is added, NSX will define SNAT & DNAT rules Nginx Ingress LB Pod http://*.demo.corp.local Web Front-End Ingress VMworld 2017 Content: Not for publication 10.4.0.0/24 10.4.1.0/24 10.4.0.67 #NET1522BU CONFIDENTIAL 27
K8s / NSX Components NSX Container Plugin (NCP) VMworld 2017 Content: Not for NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod publication NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems #NET1522BU CONFIDENTIAL 28
K8s / NSX Workflows Namespace / Topology creation K8s master etcd API- Server Scheduler NSX/ K8s topology NS: foo 2) 1) 3) NSX Container Plugin K8s Adapter NSX Manager NS: bar NCP Infra NSX Manager API Client 4) Namespace creation workflow 1. NCP creates a watch on K8s API for any Namespace events 2. A user creates a new K8s Namespace 3. The K8s API Server notifies NCP of the change (addition) of Namespaces 4. NCP creates the network topology for the Namespace: a) Requests a new subnet from the preconfigured IP block in NSX b) Creates a logical switch c) Creates a T1 router and attaches it to the pre-configured global T0 router d) Creates a router port on the T1 router, attaches it to the LS, and assigns an IP from the new subnet #NET1522BU CONFIDENTIAL 29
NSX-T Container Interface (CIF) mgmt network eth0 Minion Mgmt. IP Stack NSX CNI Plugin cif eth2 Vlan 10 vlan 11 cif OVS DFW Node VM DFW cif eth2 vlan 10 vlan 11 cif OVS eth0 mgmt network Minion Mgmt. IP Stack Hypervisor (ESXi & KVM) Node VM Management Interface is Separated from the interface used for Pods traffic CIF is used per K8s Pod CIFs are differentiated through locally significant VLAN tags NSX CNI Plugin is responsible for tagging the traffic with the right VLAN NCP will map the VLAN tags to a specific CIF Pods Pods NSX CNI Plugin #NET1522BU CONFIDENTIAL 30
NSX-T Operational Tools for K8s VMworld 2017 Content: Not for NSX-T Operational Tools Traceflow Port Mirroring Port Connection Tool Spoofguard Syslog Port Counters IPFIX publication NSX-T Traceflow #NET1522BU CONFIDENTIAL 31
Demo
NSX-T Values for K8s Enterprise-class Networking Unified VM-to- Pod Networking N S X - T Va l u e s f o r K 8 s Advanced Security Pods Micro- Segmentation Full Network Visibility Enhanced Operations Enterprise Support F e a t u r e s #NET1522BU CONFIDENTIAL 33
Hands On Lab Self-Paced Lab VMware NSX-T with Kubernetes SPL182602U(US) SPL182602E(Europe) Kubernetes and VMware NSX Blog https://blogs.vmware.com/networkvirtualization/2017/03/ kubecon-2017.html/ #NET1522BU CONFIDENTIAL 34
Where to Get Started Engage and Learn Join VMUG for exclusive access to NSX vmug.com/vmug-join/vmug-advantage Connect with your peers communities.vmware.com Find NSX Resources vmware.com/products/nsx Network Virtualization Blog blogs.vmware.com/networkvirtualization Try VMworld 2017 Experience Dozens of Unique NSX Sessions Spotlights, breakouts, quick talks & group discussions Visit the VMware Booth Product overview, use-case demos Visit Technical Partner Booths Integration demos Infrastructure, security, operations, visibility, and more Meet the Experts Join our Experts in an intimate roundtable discussion Content: Not for publication Take Free Hands-on Labs Test drive NSX yourself with expert-led or self-paces hands-on labs labs.hol.vmware.com Training and Certification Several paths to professional certifications. Learn more at the Education & Certification Lounge. vmware.com/go/nsxtraining #NET1522BU CONFIDENTIAL 35