NET1522BE Kubernetes Networking with NSX-T Deep Dive Ali Al Idrees Yves Fauser #VMworld #NET1522BE
Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. 2
Agenda 1 NSX-T Overview 2 Kubernetes Overview 3 NSX-T & Kubernetes Integration 4 Demo 3
NSX-T Overview
NSX Vision: Driving NSX everywhere Managing security and connectivity for many heterogeneous end points Branch offices/edge computing/iot vcloud Air Network Cloud VMworld 2017 Content: Not for publication New app frameworks On-premises data center End users Automation Security Application Continuity IT at the speed of business Inherently secure infrastructure Data center anywhere 5
NSX-T Architecture NSX Architecture and Components Cloud Consumption Management Plane Control Plane Data Plane NSX Manager Management Plane (MP) Node VM form factor Central Control Plane (CCP) Nodes- VM form factor VMworld 2017 ESXi (+ kernel modules) KVM (+ kernel modules) NSX Controllers Transport Nodes Self Service Portal OpenStack, K8s, Custom Concurrent configuration portal REST API entry-point UI Talks to Dataplane over a Control-Plane Protocol Separation of Control and Data Plane Content: Not for publication High Performance Data Plane Scale-out Distributed Forwarding Model Hypervisors NSX Edge (L3 + Adv Services) L2 Bridge (L2 Overlay- VLAN) Physical Infrastructure 6
NSX-T Architecture Operations Workflow Configuration is persisted Configuration is pushed to CCP MP Node CCP Node CCP Node CCP Node VMworld 2017 Content: Not for X User makes a configuration publication Configuration is realized MPA LCP Transport Node MPA LCP Transport Node MPA LCP Transport Node 7
Data Plane Improved performance and resiliency Designed for multi-tenancy and scale Tenants/CMP GENEVE Tunnel TEP: Overlay Tunnel End Point (with its own IP address) Admin p1 TEP vswitc h1 Overlay Transport Zone p2 HV TN1 New distributed edge architecture with increased performance with DPDK p1 HV TN1 Edge Node p2 TEP vswitc h2 Edge Node Edge Node Edge Node Edge Cluster Next gen overlay maintaining performance with increased flexibility 8
NSX-T VMworld Session & Lab NSX-T Breakout Session Introduction to NSX-T Architecture NET1510BU (US) NET1510BE (Europe) VMworld 2017 NSX-T Hands On Lab VMware NSX-T - Getting Started SPL182601U (US) SPL182601E (Europe) Content: Not for publication 9
Kubernetes Overview
What is Kubernetes? Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure. 11
Kubernetes Components > _ Kubectl CLI K8s master K8s master dashboard Controller Manager Scheduler K8s Master(s) K8s Nodes K8s API Server K8s Master K8s node K8s node K8s node K8s node Key-Value Store kubelet Kube-proxy c runtime K8s Cluster Consists of Master(s) and Nodes K8s Master Components API Server Scheduler Controller Manager Dashbord K8s Node Components Kubelet Kube-Proxy Containers Runtime (Docker or Rocket) 12
Kubernetes Pod Pod 10.24.0.0/16 nginx tcp/80 VMworld 2017 10.24.0.2 mgmt tcp/22 pause container ( owns the IP stack) logging udp/514 IPC External IP Traffic Content: Not for publication A Pod is a group of one or more containers that shares an IP address and a Data Volume 13
Kubernetes Namespace Namespace: foo Base URI: /api/v1/namespaces/foo redis-master Pod: /api/v1/namespaces/foo/pods/redis-master redis service: /api/v1/namespaces/foo/services/redis-master Namespace: bar Base URI: /api/v1/namespaces/bar redis-master Pod: /api/v1/namespaces/bar/pods/redis-master VMworld 2017 redis service: /api/v1/namespaces/bar/services/redis-master Namespaces are a way to divide cluster resources between multiple uses They can be considered as Tenants They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Names Overlapping Content: Not for publication 14
K8s Load Balancing 10.24.0.5/16 East-West Load Balancing Web Front-End Pods redis-slave svc Redis Slave Pods 172.30.0.24 Nginx HAProxy etc. LB Pods Web Front-End (e.g. Apache) Pods North-South Load Balancing http://*.bikeshop.com Web Front-End Ingress East-West Load Balancing is provided through K8s Service using ClusterIP & IPTables Can be achieved through K8s Ingress or External third Party Load Balancer using NodePort 15
Kubernetes Networking Topologies Flat routed topology ip route 10.24.1.0/24 10.240.0.3 ip route 10.24.2.0/24 10.240.0.4 int eth0 int cbr0 10.240.0.3 10.24.1.1/24 Node net.ipv4.ip_forward=1 10.24.1.2 10.24.1.3 10.24.1.4 int eth0 int cbr0 10.240.0.4 10.24.2.1/24 Node net.ipv4.ip_forward=1 10.24.2.2 10.24.2.3 10.24.2.4 Every Node is an IP Router and responsible for its Pod Subnet Subnets are associated with Nodes, not Tenants Physical Network Configuration is required 16
Kubernetes Networking Topologies Node-to-Node overlay topology int eth0 int cbr0 10.240.0.3 10.24.1.1/24 Node net.ipv4.ip_forward=1 10.24.1.2 10.24.1.3 10.24.1.4 Key-Value Store int eth0 int cbr0 10.240.0.4 10.24.2.1/24 Node net.ipv4.ip_forward=1 10.24.2.2 10.24.2.3 10.24.2.4 Overlays are typically used to avoid Physical Network Configuration Overlay 17
NSX-T and Kubernetes Integration
NSX-T K8s Integration Namespaces & Pods admin@k8s-master:~$ kubectl create namespace foo namespace foo" created admin@k8s-master:~$ kubectl create namespace bar namespace bar" created admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo deployment "nginx-foo" created admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bar deployment "nginx-bar" created VMworld 2017 NSX / K8s topology Namespace: foo NAT boundary NAT boundary 10.24.0.0/24 10.24.1.0/24 10.24.2.0/24 Namespace: bar Content: Not for publication K8s Masters K8s nodes 19
NSX-T K8s Integration Routed Namespaces admin@k8s-master:~$ vim no-nat-namespace.yaml apiversion: v1 kind: Namespace metadata: name: no-nat-namespace annotations: ncp/no_snat: "true admin@k8s-master:~$ kubectl create f no-nat-namespace.yaml namespace no-nat-namespace" created VMworld 2017 admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx n no-nat-namespace deployment "nginx-k8s" created NSX / K8s topology Namespace: no-nat-namespace 114.4.10.0/26 Direct Routing Content: Not for publication 114.4.10.64/26 K8s Masters K8s nodes 20
NSX-T K8s Integration Pods Micro-Segmentations Option1: Predefined Label Based Rules admin@k8s-master:~$ kubectl label pods nginx-foo-3492604561-nltrf secgroup=web -n foo Pod "nginx-nsx-3492604561-nltrf" labeled admin@k8s-master:~$ kubectl label pods nginx-bar-2789337611-z09x2 secgroup=db -n bar pod "nginx-k8s-2789337611-z09x2" labeled admin@k8s-master:~$ kubectl get pods --all-namespaces -Lsecgroup NAMESPACE NAME READY STATUS RESTARTS AGE SECGROUP k8s nginx-foo-2789337611-z09x2 1/1 Running 0 58m web nsx nginx-bar-3492604561-nltrf 1/1 Running 0 1h db Security Groups are defined in NSX with ingress and egress policy Each Security Group could be micro-segmented to protect Pods from each other VMworld 2017 NSX / K8s topology Content: Not for publication Namespace: foo NAT boundary NAT boundary Namespace: bar 10.24.0.0/24 10.24.1.0/24 114.4.10.0/26 DB Web 21
NSX-T K8s Integration Pods Micro-Segmentations Option 2: K8s Network Policy admin@k8s-master:~$ vim nsx-demo-policy.yaml apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: nsx-demo-policy spec: podselector: matchlabels: app: web ingress: - from: - namespaceselector: matchlabels: ncp/project: db ports: - port: 80 protocol: TCP admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml State: released on K8s 1.7 (Beta on 1.6) Capability: Using Network Policy, users can define firewall rules to allow traffic into and out of a Namespace, and between Pods. The network policy is a Namespace property. The default is drop NSX / K8s topology Namespace: foo VMworld 2017 Content: Not for publication NAT boundary Routed 10.24.0.0/24 10.24.1.0/24 114.4.10.0/26 Web Label: app=web Namespace: bar DB Label: app=db 22
NSX-T K8s Integration Pods Micro-Segmentations Option 2: K8s Network Policy $ kubectl create -f nsx-demo-policy.yaml Once the Network Policy is applied, NSX will dynamically create source & destination Security Groups and apply the right policy Dynamic Creation of Security Groups Dynamic Creation of Security Policy based on k8s Network Policy 23
NSX-T K8s Integration Pods Micro-Segmentations Firewalling in Kubernetes Micro-Segmentation in K8s: The data model to describe segmentation policies between Namespaces, and within namespaces is called Network Policies and is released on Kubernetes 1.7 (Beta on 1.6) K8s Network Policy NSX could utilize K8s Network Policies to define Dynamic Security Groups & Policies. Capabilities are limited to K8s Network Policy capabilities. VMworld 2017 Firewalling in NSX / K8s Pre-Defined Label based rules Security Groups & Policies could be predefined on NSX. Labels are used to specify Pods Membership Content: Not for publication Mapping of IP based groups, egress rules, VM based matching could be available to be used in the policy definition The NSX / K8s integration intends to support both the pre-defined label based rules and K8s network policy. 24
East-West Load Balancing K8s master K8s master dashboard Controller Manager Scheduler NSX CNI Plugin K8s API Server K8s Master OVS Node VM NSX Kube Proxy K8s Services are delivered through NSX Kube-Proxy. Delivered as a container image, so that it can be run as a Kubernetes Daemon-Set on the Nodes. NSX Kube-Proxy would replace the native distributed east-west load balancer in Kubernetes called Kube-Proxy. OpenVSwitch (OVS) load-balancing is used. Pods 25
North-South Load Balancing Once an Ingress Controller is added, NSX will define SNAT & DNAT rules Nginx Ingress LB Pod http://*.demo.corp.local Web Front-End Ingress VMworld 2017 Content: Not for publication 10.4.0.0/24 10.4.1.0/24 10.4.0.67 26
K8s / NSX Components NSX Container Plugin (NCP) VMworld 2017 Content: Not for NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod. publication NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems 27
K8s / NSX Workflows Namespace / Topology creation K8s master etcd API- Server Scheduler NSX/ K8s topology NS: foo 2) 1) 3) NSX Container Plugin K8s Adapter NSX Manager NS: bar NCP Infra NSX Manager API Client 4) Namespace creation workflow 1. NCP creates a watch on K8s API for any Namespace events 2. A user creates a new K8s Namespace 3. The K8s API Server notifies NCP of the change (addition) of Namespaces 4. NCP creates the network topology for the Namespace : a) Requests a new subnet from the preconfigured IP block in NSX b) Creates a logical switch c) Creates a T1 router and attaches it to the pre-configured global T0 router d) Creates a router port on the T1 router, attaches it to the LS, and assigns an IP from the new subnet 28
NSX-T Container Interface (CIF) mgmt network eth0 Minion Mgmt. IP Stack NSX CNI Plugin cif eth2 Vlan 10 Pods vlan 11 cif OVS DFW Node VM DFW cif eth2 vlan 10 Pods cif vlan 11 OVS eth0 Hypervisor (ESXi & KVM) mgmt network Node VM Minion Mgmt. IP Stack NSX CNI Plugin Management Interface is Separated from the interface used for Pods traffic CIF is used per K8s Pod CIFs are differentiated through locally significant VLAN tags NSX CNI Plugin is responsible for tagging the traffic with the right VLAN NCP will map the VLAN tags to a specific CIF. 29
NSX-T Operational Tools for K8s VMworld 2017 Content: Not for NSX-T Operational Tools Traceflow Port Mirroring Port Connection Tool Spoofguard Syslog Port Counters IPFIX publication NSX-T Traceflow 30
Demo
NSX-T Values for K8s Enterprise-class Networking Unified VM-to- Pod Networking N S X - T Va l u e s f o r K 8 s Advanced Security Pods Micro- Segmentation Full Network Visibility Enhanced Operations Enterprise Support F e a t u r e s 32
Hands On Lab Self-Paced Lab VMware NSX-T with Kubernetes SPL182602U(US) SPL182602E(Europe) Kubernetes and VMware NSX Blog https://blogs.vmware.com/networkvirtualization/2017/03/ kubecon-2017.html/ 33
Where to get started Engage and Learn Join VMUG for exclusive access to NSX vmug.com/vmug-join/vmug-advantage Connect with your peers communities.vmware.com Find NSX Resources vmware.com/products/nsx Network Virtualization Blog blogs.vmware.com/networkvirtualization Try VMworld 2017 Experience Dozens of Unique NSX Sessions Spotlights, breakouts, quick talks & group discussions Visit the VMware Booth Product overview, use-case demos Visit Technical Partner Booths Integration demos Infrastructure, security, operations, visibility, and more Content: Not for publication Meet the Experts Join our Experts in an intimate roundtable discussion Take Free Hands-on Labs Test drive NSX yourself with expert-led or self-paces hands-on labs labs.hol.vmware.com Training and Certification Several paths to professional certifications. Learn more at the Education & Certification Lounge. vmware.com/go/nsxtraining