Secure Kubernetes Container Workloads

Transcription

1 Secure Kubernetes Container Workloads with Production-Grade Networking Cynthia Thomas Irena Berezovsky Tim Hockin

2 CIA IT operations have top secret apps for their agents, most of which require isolation - Antoni Antoni is in Ops and wants to help CIA embrace DevOps Berta is a Dev eager to design efficiently and deliver excellent apps Berta

3 The world before Neutron: can I plug in your cable? 1. New project defined: Developer needs an environment 2. Dev asks SysAdmin for some resources 3. SysAdmin installs Server OS and asks Network people for a VLAN (ewww!) 4. Network people ask Security team to open a port in a firewall rule. Someone plugs into wrong port or wrong requirements: start over!

4 Antoni & Berta were doing it the hard way The CIA IT takes weeks, even months to deliver isolated resources for the various projects at CIA. Servers and VLANs and firewalls, oh my!

5 Neutron for higher layer network services OpenStack core project since Folsom Tenant and Admin solution agnostic API Pluggable Framework Provides extensible API to build rich topologies (vendor extensions) Advanced Services Support, i.e. LBaaS, VPNaaS, FWaaS

6 OpenStack Networking on the fly at CIA Operator (networking, security, etc) versus App Developer OpenStack has reduced the time to deliver compute resources Security policies allows the CIA to keep business units separate Each department admin can manage its own resources

7 Can we do better? Spawning a VM is slow expensive There is a lot of management overhead VMs are not portable: run on a specific Hypervisor If only there was a way to virtualize an OS to enable multiple workloads to run on a single OS... App App App Libs Libs Libs Guest OS Guest OS Guest OS Hypervisor OS Hardware

8 Along came Docker Containers are an alternative to VMs Bundle your app & deps, but not the OS Faster and lower overhead than VMs O(milliseconds) to spawn Developer-focused Enables fast iteration, less non-app concerns Ridiculously simple UX It s the technology of the decade! container container container App App App Libs Libs Libs OS Hardware

9 CIA developers demand containers Launch in milliseconds! Dev-Prod parity, on my laptop! MUST HAVE! But it is very chaotic -- they need need help managing it all...

10 Kubernetes changed everything The Kubernetes API is app-centric De-emphasizes infrastructure and operations concerns Those can exist, but are not the primary focus Integrate with existing infrastructure and ops, but don t replace it Networking is infrastructure, security is ops We still need to address the concerns of ops!

11 Kubernetes network model Assumes a single, shared network space No noun for Network (yet?) Network plugins decide what technology veth, VXLAN, OVS, etc. All connectivity is enabled by default Implicitly single tenant Also reflected in services like DNS Compare to Docker model: Noun for Network App-centric networks

12 Tension between dev and ops No, no, no! I can t have my apps be reachable by Carlos s team!

13 Namespaces: Kubernetes Scopes for named objects within a cluster Pods, Services, etc. are all namespaced Logical grouping of related things Could be 1 user...or 1 app...or 1 tier of an app No relationship to nodes or networks All Namespaces exist on all nodes Network is not segmented Seems like an obvious hook for networking

14 Kubernetes NetworkPolicy API to lock down the network Describe the graph of your app Specify which connectivity to allow Applies per Namespace Exists alongside the apps it decribes Default-deny plus explicit allow rules Network infrastructure can enforce it Many vendors have implementations Does not cover egress (yet?)

15 Kubernetes NetworkPolicy apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: front-to-mid namespace: cia-spy-app spec: podselector: role: middleware ingress: - ports: - protocol: TCP port: 6379 from: - podselector: role: frontend apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: mid-to-db namespace: cia-spy-app spec: podselector: role: db ingress: - ports: - protocol: TCP port: 3306 from: - podselector: role: middleware

16 Kubernetes NetworkPolicy apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: front-to-mid namespace: cia-spy-app spec: podselector: role: middleware ingress: - ports: - protocol: TCP port: 6379 from: - podselector: role: frontend apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: mid-to-db namespace: cia-spy-app spec: podselector: role: db ingress: - ports: - protocol: TCP port: 3306 from: - podselector: role: middleware

17 Kubernetes NetworkPolicy apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: front-to-mid namespace: cia-spy-app spec: podselector: role: middleware ingress: - ports: - protocol: TCP port: 6379 from: - podselector: role: frontend apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: mid-to-db namespace: cia-spy-app spec: podselector: role: db ingress: - ports: - protocol: TCP port: 3306 from: - podselector: role: middleware

18 Kubernetes NetworkPolicy apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: front-to-mid namespace: cia-spy-app spec: podselector: role: middleware ingress: - ports: - protocol: TCP port: 6379 from: - podselector: role: frontend apiversion: extensions/v1beta1 kind: NetworkPolicy metadata: name: mid-to-db namespace: cia-spy-app spec: podselector: role: db ingress: - ports: - protocol: TCP port: 3306 from: - podselector: role: middleware

19 Where Neutron is ahead of k8s Neutron Kubernetes Multi Tenant environment Single Tenant Rich network topologies with overlapping IPs Flat, shared network with IP per pod Security Groups, Port Security (ARP Spoofing) Network Policy (ingress only) Port Quality of Service - Admin and Tenant facing API Primarily application-centric API

20 Containers Challenges A lot of new products/companies have emerged in the container orchestration and integration ecosystem With multi-host, multi orchestration environment, networking becomes critical Run containers in VMs for better isolation and security Multi-tenancy, host or cluster per tenant VMs and Containers share the same network

21 What is Kuryr? Neutron as a production-ready networking abstraction that containers need Kuryr translates container orchestration events into Neutron entities, performs API calls and manages the response to the orchestrator

22 Kuryr as a translator between k8s and Neutron Map container networking abstractions to the Neutron API Kubernetes Neutron Allow Container, BM and VM networking under the same API Namespace Network, Subnet Pod Port Implements all the common code for Neutron vendors, allowing them to provide advanced container networking by just having a binding script Service Load Balancer External IP Floating IP Network Policy Security Groups

23 Example: CIA Security Antoni can satisfy CIA security requirements with Kubernetes & Kuryr: Kubernetes made it easy for the app devs to express the application in terms of required deployment With NetworkPolicy devs can specify the intended application connectivity With Kuryr mapping of Kubernetes requests to Neutron constructs, and NetworkPolicy realization by Neutron security groups, true isolation and security is achieved via the Kubernetes API

24 Kubernetes + Kuryr + MidoNet: Scalable Neutron Neutron plugin scaling with ease, and flexible API for fine-grain security policies Event-based design (receives events from k8s-api) Compatible with Kubernetes >=1.2 API watcher + CNI driver Asynchronous event-loop based on asyncio python 3.4 library No kube-proxy

25 How is MidoNet a Scalable Neutron Solution?

26 Example: CIA MidoNet deployment Antoni uses MidoNet for his OpenStack Neutron plugin for production-grade networking He knows he can scale compute for the CIA with confidence with an HA solution MidoNet Manager provides Antoni with every single network flow and each security policy applied

27 Example: CIA MidoNet deployment with kuryr-k8s Antoni is trying the kuryr-k8s tech preview solution with MidoNet: Today, he can launch a script to automatically deploy the k8s-master and k8s-worker to try MidoNet with k8s Antoni wants native API calls for pods and VMs while using the same operator tools

28 MidoNet-enhanced Security Neutron Security Groups white-list of allowed traffic port-level firewall MidoNet implements SG+: low-level constructs called chains and rules richer feature set for matching/filtering and actions

29 Future of networking in Kubernetes Multi-tenancy is probably unavoidable Ripples are deep and wide Possible evolution: Network objects in API More fine-grained policy Egress policy L7 policy QoS/shaping Multi-tenant DNS (and other services)

30 Kuryr-Kubernetes status Current Early stage CNI Driver and k8s API watcher in progress Future work K8s Network Policy support High Availability Kuryr-openshift Bridging OpenStack VMs and kuryr-k8s

31 Get Involved! Kuryr Kubernetes MidoNet Community Site Community Site Community Site Project Git Repo Project Git Repo Project Git Repo IRC weekly meeting Join Slack slack.k8s.io/ Join Slack slack.midonet.org Try MidoNet with one command: $> curl -sl quickstart.midonet.org sudo bash