Kubernetes Container Networking with NSX-T Data Center Deep Dive Yasen Simeonov, VMware, Inc. #vmworld NET1677BU #NET1677BU
Disclaimer This presentation may contain product features or functionality that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new features/functionality/ technology discussed or presented, have not been determined. 2
Agenda NSX-T Intro Quick level set on NSX-T Kubernetes Overview Technical overview of Kubernetes, nomenclature & networking details NSX-T & Kubernetes Details of the NSX-T integration with Kubernetes Demo Seeing is believing 3
NSX-T Data Center Intro Quick level set on NSX-T Data Center 4
The Virtual Cloud Network Connect and protect your business TELCO/NFV EDGE/IOT BRANCH BRANCH BRANCH BRANCH BRANCH BRANCH EDGE/IOT TELCO/NFV BRANCH BRANCH BRANCH TELCO/NFV BRANCH EDGE/IOT 5
Public Clouds VMs, Containers, Microservices Virtual Cloud Networking Connect & Protect any workload across any environment 2 d orl w M V C 8 01 Telco Networks n o i ut b i r t s i d r o n o i t a Built-in c i l b u p r o Automated f t o Private Data Centers n e t n o t: N Identity Secure Connectivity Availability Policy Scalability Analytics and Insights Things Apps and Data Programmable Application Centric Users Branch Offices 6
VMware NSX Portfolio The foundation of the Virtual Cloud Network NETWORKING AND SECURITY MANAGEMENT AND AUTOMATION Cloud-Based Management Workflow Automation Blueprints / Templates Insights / Discovery Visibility NSX Data Center Networking and security for data center workloads Network Insight Network discovery and insights NETWORK AND SECURITY VIRTUALIZATION AppDefense Modern application security vrealize Automation End-to-end workload automation Security Integration Extensibility Automation Elasticity NSX Cloud Networking and security for Public Cloud workloads NSX SD-WAN by VeloCloud WAN connectivity services NSX Hybrid Connect Data center and cloud workload migration 7
NSX-T Data Center Architecture and Components Data Plane Cloud Consumption Management Plane Control Plane ESXi (+ kernel modules) Central Control Cluster CCP KVM (+ kernel modules) Local Control Plane LCP NSX Edge VM or Bare Metal Layer 2 Bridge OpenStack, k8s or Custom Highly available and scalable Built for consumption by developers Support for endpoint heterogeneity Improved performance and resiliency 8
Data Plane Improved performance and resiliency Designed for multi-tenancy and scale Tenants/CMP GENEVE Tunnel TEP: Overlay Tunnel End Point (with its own IP address) Admin Overlay Transport Zone p1 TEP vswitc h1 p2 HV TN1 New distributed edge architecture with increased performance with DPDK p1 HV TN1 Edge Node p2 TEP vswitc h2 Edge Node Edge Cluster Next gen overlay maintaining performance with increased flexibility Edge Node Edge Node 9
Kubernetes Overview Technical overview of Kubernetes, nomenclature & networking details 10
What is Kubernetes? Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure. 11
Kubernetes Components > _ Kubectl CLI K8s master K8s master dashboard Controller Manager Scheduler K8s Master(s) K8s API Server K8s Master K8s Nodes K8s node K8s node K8s node K8s node Key-Value Store kubelet Kube-proxy c runtime K8s Cluster Consists of Master(s) and Nodes K8s Master Components API Server Scheduler Controller Manager Dashboard K8s Node Components Kubelet Kube-Proxy Containers Runtime (Docker or Rocket) 12
Kubernetes Pod Pod 10.24.0.0/16 nginx tcp/80 10.24.0.2 mgmt tcp/22 pause container ( owns the IP stack) logging udp/514 External IP Traffic A Pod is a group of one or more containers that shares an IP address and a Data Volume IPC 13
Kubernetes Namespace Namespace: foo Base URI: /api/v1/namespaces/foo redis-master Pod: /api/v1/namespaces/foo/pods/redis-master redis service: /api/v1/namespaces/foo/services/redis-master Namespace: bar Base URI: /api/v1/namespaces/bar redis-master Pod: /api/v1/namespaces/bar/pods/redis-master redis service: /api/v1/namespaces/bar/services/redis-master Namespaces are a way to divide cluster resources amongst users and groups They can be thought of as Tenants They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Name uniqueness 14
DNS: Kubernetes Service redis-slave.external.com 134.247.200.20 ExternalIP 134.247.200.20 10.24.0.5 Web Front-End Pods redis-slave svc Redis Slave Pods ClusterIP 172.30.0.24 kubectl describe svc redis-slave Name: redis-slave Namespace: default Labels: name=redis-slave Selector: name=redis-slave Type: LoadBalancer IP: 172.30.0.24 LoadBalancer Ingress: 134.247.200.20 Port: <unnamed> 6379/TCP Endpoints: 10.24.0.5:6379, 10.24.2.7:6379 10.24.2.7 DNS: redis-slave.<ns>.cluster.local 172.30.0.24 A Kubernetes Service defines a logical set of Pods, selected with matching labels Serves multiple functions: Service Discovery / DNS East/West load balancing in the Cluster (Type: ClusterIP) External load balancing for L4 TCP/UDP (Type: LoadBalancer) External access to the service through the nodes IPs (Type: NodePort) 15
Kubernetes Ingress DNS: *.bikeshop.com 134.247.200.1 http://www.bikeshop.com/shop http://www.bikeshop.com/special-offers Web Front-End Pods (shop svc) External IP: 134.247.200.1 kubectl describe ingress bikeshop-ingress-shop Name: bikeshop-shop Namespace: bikeshop Address: 100.64.240.9,134.247.200.1 Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- www.bikeshop.com /shop web-svc-1:80 (<none>) LoadBalancer Datapath (External or K8s Pods) Web Front-End Pods (special-offers svc) A Kubernetes Ingress Object is a L7 LoadBalancing rule that binds a hostname and url to a Service The LoadBalancer Datapath can be implemented as an external Load Balancer or as a K8s Pod 16
Kubernetes Networking Topologies Non-multitenant routed topology int eth0 int cbr0 10.240.0.3 Node net.ipv4.ip_forward=1 10.24.1.1/24 10.24.1.2 10.24.1.3 10.24.1.4 ip route 10.24.1.0/24 10.240.0.3 ip route 10.24.2.0/24 10.240.0.4 int eth0 int cbr0 10.240.0.4 10.24.2.1/24 10.24.2.2 10.24.2.3 10.24.2.4 Node net.ipv4.ip_forward=1 Every Node is an IP Router and responsible for its Pod Subnet Subnets are associated with Nodes, not Tenants Physical Network Configuration is required 17
Kubernetes Networking Topologies Node-to-Node overlay topology Overlay int eth0 int cbr0 10.240.0.3 Node net.ipv4.ip_forward=1 10.24.1.1/24 10.24.1.2 10.24.1.3 10.24.1.4 int eth0 int cbr0 Key-Value Store 10.240.0.4 10.24.2.1/24 10.24.2.2 10.24.2.3 10.24.2.4 Node net.ipv4.ip_forward=1 Overlays are typically used to avoid Physical Network Configuration Subnets are still associated with Nodes, not Tenants External outbound connectivity needs SNAT using the Nodes IP External inbound connectivity needs Node Port or Ingress in Host Network Mode 18
NSX-T & Kubernetes Details of the NSX-T integration with Kubernetes 19
Key Design Goals of the NSX-T Data Center Kubernetes Integration Don't stand in the way of the developer! Provide solutions to map the Kubernetes constructs to enterprise networking constructs Secure Containers, VMs and any other endpoints with overarching Firewall Policies Provide visibility & troubleshooting tools to ease the container adoption in the enterprise 20
Kubernetes NSX Topology Dynamic per Namespace Topology NSX/ K8s topology Namespace: foo Namespace: bar 10.4.0.0/26 10.4.0.64/26 34.1.2.33/26 Dynamically network topology per K8s namespace K8s Nodes are not doing IP routing Every Pod has its own logical port on a NSX logical switch, and is supporting all features a VM interface supports Every Pod has Dynamic Firewall rules applied on its logical Interface 21
K8s / NSX Components NSX Container Plugin (NCP) K8s master etcd NS: foo API- Server Scheduler NSX/ K8s topology NSX Container Plugin K8s / OS Adapter CloudFoundry Adapter More NSX Manager NS: bar NCM Infra NSX Manager API Client NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod. NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems at some point 22
Tenancy / Topology Mapping The open source way With most networking technologies in K8s like Flannel, OpenShift OVS Networking, Calico, etc. the source IP of the traffic can't be mapped to the tenancy. This is the biggest hurdle today to get K8s integrated in enterprise IT environments SNAT to Node IP Node VM mgmt IP 172.16.1.11/24 Pods 10.255.0.9/24 Tenant: foo Physical or virtual Router 172.16.1.1/24 vnic IPTables (NAT) 10.255.0.10/24 Tenant: bar Physical DC Firewall Database (VM based or Physical) Node VM mgmt IP 172.16.1.12/24 Pods 10.255.1.5/24 vnic IPTables (NAT) 10.255.1.3/24 Tenant: foo Did the traffic come from 'foo or 'bar'? SNAT to Node IP 23
Tenancy / Topology Mapping Persistent IPs for K8s Namespaces With NSX-T each Tenant (Kubernetes Namespace) either gets its own SNAT IP (NAT Mode), or is directly identifiable by its source subnet (No NAT Mode) Database (VM based or Physical) Physical DC Firewall In No-NAT NAT Mode, Mode, the the external external DC Firewall DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source IP SNAT IP Subnet that is allocated that is allocated to a specific to a specific Tenant. Tenant. Tenant: bar PAS VMs T1 router NSX-T Logical Switch Node VM mgmt IP 172.16.1.11/24 VLAN Trunk 10.12.1.8/24 vnic 10.12.5.5/24 OpenvSwitch Namesp. Foo T1 router 172.16.1.1/24 10.12.1.1/24 10.12.5.1/24 Pods A new SNAT IP is allocated on the T0 router for each Tenant for NAT Mode Tenant: foo Namesp. Bar T1 router 24
Persistent SNAT IP per K8s Service Specifying the source IP Kubernetes Workloads using the K8s service allow from: 134.247.100.10 (App) to: 134.247.200.9 (DB) For all other Pods use namespace SNAT IP Kubernetes Namespace: Foo K8s Svc for Web Web-Frontend Pods Corporate network Namespace LS(s) Tier0 LR Tier1 LR DB SNAT App Svc Pods to: 134.247.100.10 K8s Svc for App App Logic Pods Feature With this feature a set of Kubernetes Workloads (Pods) can be assigned to use a specific IP or group of SNAT IPs to source their traffic from Before this feature we only assigned a SNAT IP to a Kubernetes Namespace Benefits Infrastructure Teams can pre-create Firewall rules in existing DC physical Firewalls to allow traffic from specific workloads in K8s The K8s user / DevOps can deploy applications that are easily identifiable in the physical network 25
Central Visibility With most other networking technologies in K8s and PCF like Flannel, OpenShift OVS Networking, PCF Silk, Calico, etc. there is no centralized control plane. So, there s no counters, troubleshooting tools, 'span ports', Firewall Rules Overview, etc. 26
Central Visibility With NSX-T you are gain deep visibility into the container networks, and you can use the same troubleshooting tools we created for VM based workloads 27
Kubernetes Metadata / NSX Logical Port Mapping kubectl get pod nsx-demo-rc-c7x65 -o yaml apiversion: v1 kind: Pod metadata: creationtimestamp: 2018-07-25T12:05:56Z generatename: nsx-demo-rclabels: app: nsx-demo name: nsx-demo-rc-c7x65 namespace: nsx-ujo Metadata within Kubernetes like Namespace, Pod names, Labels all get copied to the NSX Logical Port as Port Tags 28
Pre-Created Security Groups / Firewall rules (admin rules) NSX can be configured to collect ports and switches in dynamic security groups based on Tags (Kubernetes Metadata) and apply Firewall rules on them Match on Port Tags Matching Pods are part of the Group Groups are used in Firewall sections as src and dst 29
Unified Policy for K8s, PCF & VMs Both K8s and PCF have 'built-in' micro segmentation policy languages (network policy), and there's a broad set of products and open source projects implementing micro segmentation inside of K8s or PCF. However there is no technology other than NSX-T today that allows you to define policies across K8s, PCF and VM based workloads using Metadata from each system vsphere VMs DB VMs T1 router NSX-T Logical Switch NSX-T LS PCF Org Foo T1 router PCF AIs allow: tcp/443 Kubenetes Namespace: Bar T1 router NSX-T Logical Switch K8s Pods allow: tcp/3306 (mysql) 30
Support of Kubernetes Network Policy apiversion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: nsx-demo-policy spec: podselector: matchlabels: app: nsx-demo policytypes: - Ingress ingress: - from: - ipblock: cidr: 100.64.160.11/32 ports: - port: 80 protocol: TCP apiversion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny spec: podselector: {} policytypes: - Ingress Besides supporting admin pre-defined rules, NCP is also translating Kubernetes NetworkPolicy Objects to NSX security groups and Firewall rules Admin pre-defined rules can be used concurrently in NSX, admin rules are put in sections before or after K8s network policy rules 31
Built-in Load Balancing We have built-in support for Ingress (L7 HTTP/HTTPS) and Svc Type LB (L4 TCP/UDP) in the NSX-K8s integration. Most other K8s networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX from Ingress (L7). K8s master etcd Scheduler LB Service HTTP and/or HTTPS traffic Virtual Server 10.114.209.209 NSX Container Plugin K8s / OS Adapter CloudFoundry Adapter Libnetwork Adapter More NSX Manager Rule 1 /foo/ Rule 2 /bar/ NCM Infra Server Pool 1 Server Pool 2 NSX Manager API Client K8s master etcd API- Server API- Server Scheduler LB Service TCP and/or UDP traffic Virtual Server 10.114.209.212 NSX Container Plugin K8s / OS Adapter CloudFoundry Adapter Libnetwork Adapter More NSX Manager NCM Infra Server Pool NSX Manager API Client 32
K8s / NSX Workflows Svc Type LB K8s master etcd 2) API- Server Scheduler LB Service TCP and/or UDP traffic 1) 3) Virtual Server 10.114.209.212 NSX Container Plugin K8s / OS Adapter CloudFoundry Adapter More NSX Manager Server Pool NCM Infra NSX Manager API Client 4) 1. NCP watches for Svc events in Kubernetes 2. User creates a new Svc of Type LoadBalancer 3. The Kubernetes API server notifies NCP of the new Svc 4. NCP creates a new Virtual Server with a unique IP and a Server Pool with the Pods as targets 33
K8s / NSX Workflows Ingress HTTP and/or HTTPS traffic K8s master etcd 2) API- Server Scheduler LB Service Virtual Server 10.114.209.209 1) 3) Rule 1 /foo/ Rule 2 /bar/ NSX Container Plugin K8s / OS Adapter CloudFoundry Adapter More NSX Manager Server Pool 1 Server Pool 2 NCM Infra NSX Manager API Client 4) 1. NCP watches for Ingress events in Kubernetes 2. User creates a new Ingress rule 3. The Kubernetes API server notifies NCP of the new Ingress rule 4. NCP creates a new forwarding rule sending a specific HTTP/S hostname and path to a specific Server Pool 34
NSX-T Data Center Timeline Kubernetes, OpenShift and PKS 2017 2018 September October November December January February March NSX-T 2.0: Support for 'Do It Yourself' K8s & OpenShift Core value add: Mapping of K8s Namespaces to Network Topology & source IP Addresses NAT & No-NAT modes per Namespace Network Policy (Firewall) across K8s and VM workloads Support for K8s Network Policy Logical Network Port per K8s workload (Pod) for visibility and troubleshooting NSX-T 2.1 Support for PKS 0.8 and PKS 1.0 Support for K8s Ingress and Svc Type LB with Platform LB Core value add: One of the only SDN solution in the market that includes LB with Ingress and Svc Type LB for K8s PKS / OPS MGR Integration Gives PKS support for Network Policy 35
NSX-T Timeline PCF 2.0 2018 January February March April May June July NSX-T 2.1 Support for PCF 2.0 -> PAS Core value add: Allows mapping of CF tenancy (Orgs) to Network Topology & source IP Addresses Network Policy (Firewall) support across PKS, PCF and VM workloads Only solution that allows for direct, no_nat communication from CF Apps to backend services Logical Network Port per CF workload (AI) for visibility and troubleshooting NSX-T 2.2 Operational Enhancement & Additional LB features Core value add: Persistent SNAT IP for Kubernetes Services and CF Apps TLS/SSL Offload support for Kubernetes Ingress OpenShift 'router' support for HTPP and HTTPS (feature parity with K8s Ingress) URL rewrite support for K8s Ingress Various install & operational improvements 36
NSX-T & Kubernetes Demo 37
NSX-T Data Center Values for Containers Enterprise-class Networking NSX- T Values for Containers Advanced Security Unified VMto-Container Networking Micro- Segmentation Full Network Visibility Enhanced Operations Enterprise Support Features 38
Where to Get Started Engage and Learn Join the NSX VMUG Community vmug.com/nsx Connect with your Peers communities.vmware.com Embrace the NSX Mindset nsxmindset.com Find NSX Resources vmware.com/go/networking Read the Network Virtualization Blog blogs.vmware.com/networkvirtualization Try Free Hands-on Labs labs.hol.vmware.com Virtual Cloud Network Guided Demo vcndemo.com Experience Attend the Networking and Security Sessions Showcases, breakouts, quick talks & group discussions Visit the VMware Booth Product overviews, use-case demos Visit Technical Partner Booths Integration demos Infrastructure, security, operations, visibility, and more Meet the Experts Join our experts in an intimate roundtable discussion Take VMware Education Training and Certification vmware.com/go/nsxtraining Free NSX Training on Coursera vmware.com/go/coursera 39
n o i ut b i r t s i d r o n o i t a c i l b u p r o f t o 2 d orl w M V C 8 01 N : t n e t n o 40
PLEASE FILL OUT YOUR SURVEY. Take a survey and enter a drawing for a VMware company store gift card. #vmworld #NET1677BU
THANK YOU! #vmworld #NET1677BU