Bringing Security and Multitenancy. Lei (Harry) Zhang

Transcription

1 Bringing Security and Multitenancy to Kubernetes Lei (Harry) Zhang

2 About Me Lei (Harry) Zhang #Microsoft MVP in cloud and datacenter management though I m a Linux guy :/ Previous: VMware, Baidu Feature maintainer of Kubernetes HyperCrew: Publications: Docker & Kubernetes Under the Hood PhD Large-scale cluster management and scheduling

3 A survey about boundary Are you comfortable with Linux containers as an effective boundary? Yes, I use containers in my private/safe environment No, I use containers to serve the public cloud

4 As long as we care security We have to wrap containers inside full-blown virtual machines But we lose cloud-native deployment reality Slow startup time Huge resources wasting dream Memory tax for every container

5 Revisit namespace cgroups container Container Runtime The dynamic view and boundary of /bin /dev /etc /home /lib / lib64 /media /mnt /opt /proc / root /run /sbin /sys /tmp / usr /var /data /temp.txt echo hello Read-Write Layer & /data read-write layer init layer your running process Container Image The static view of your program, data, dependencies, files and directories /etc/hosts /etc/hostname /etc/resolv.conf CMD [ echo hello"] VOLUME /data ADD temp.txt / json json /temp.txt read-only layer FROM busybox Docker Container FROM busybox ADD temp.txt / VOLUME /data CMD [ echo hello"]

6 HyperContainer Secure Kubernetes from runtime level

7 HyperContainer Container Runtime RunV The OCI compatible hypervisor based runtime implementation Widely adopted by companies like Huawei etc Control daemon Container Image Docker Image Spec

8 Combine the best parts Portable and behaves like a Linux container $ hyperctl run -t busybox echo helloworld sub-second startup time*, ~12MB memory cost Fully isolated sandbox with an independent guest kernel $ hyperctl exec -t busybox uname -r hyper (or your provided kernel) security, backward compatibility, maturity See:

9 HyperContainer is a Pod That s how HyperContainer fits into the Kubernetes philosophy Wait, why Pod is so important?

10 Pod: lesson learned from Borg Should sample.war be packaged with Tomcat?

11 Pod: lesson learned from Borg InitContainers: one or more containers started in sequence before the pod's normal containers are started. Share volumes, perform network operations, and perform computation prior to the app containers.

12 So, Pod is The group of super-affinity containers The atomic scheduling unit Pod The process group in container cloud log app Do right things without modifying your container image infra container init container Kubernetes = Spring Framework volume Pod = IoC

13 Pod is not easy to simulate log super affinity app Requirement: app: 1G, log: 0.5G Available: Node_A: 1.25G, Node_B: 2G What happens if app scheduled to Node_A?

14 HyperContainer is a Pod Linux container based runtimes wraps and encapsulates several app containers into a logical group Hypervisor container based runtime hypervisor serves as a natural boundary of Pod

15 HyperContainer is a Pod kubelet Container Runtime Interface create sandbox Foo --> create container C --> start container C stop container C --> remove container C --> delete sandbox Foo Sandbox Normally: the infra container HyperContainer: hypervisor with HyperKernel a HyperStart process as PID 1 setup mnt namespace, launch apps from the images etc

16 Hypernetes Kubernetes with HyperContainer Runtime

17 Hypernetes Also: h8s 1. Kubernetes + HyperContainer runtime officially supported by using kubernetes/frakti 2. Multi-tenant network and persistent volumes battle tested Neutron + Cinder plugin

18 Multi-tenant Network

19 Multi-tenant Network Goal: leveraging tenant-aware neutron network for Kubernetes following the network plugin workflow Non-goal: break k8s network model or hack k8s code

20 Define the Network Network a top class api object each tenant (created by Keystone) has its own Network Network mapping to Neutron net a Network Controller is responsible to manage Network lifecycle

21 Example proxy Call Neutron to create/delete network Desired World Real World controller-manager ControlLoop network pod replica namespace service job deployment volume petset kubelet SyncLoop api-server etcd proxy scheduler kubelet SyncLoop

22 Kubernetes Network Model Container reach container all containers can communicate with all other containers without NAT Node reach container all nodes can communicate with all containers (and vice-versa) without NAT IP addressing Pod in cluster can be addressed by its IP

23 How h8s fits that? Network can be assigned to one or more Namespaces Pods belonging to the same Network can reach each other directly through IP a Pod s network mapping to Neutron port kubelet is responsible for Pod network setup let s see how kubelet works

24 Example proxy kubelet SyncLoop 1 Pod created scheduler api-server etcd proxy kubelet SyncLoop

25 Example proxy kubelet SyncLoop scheduler 2 Pod object added api-server etcd proxy kubelet SyncLoop

26 Example proxy 3.1 New pod object detected 3.2 Bind pod with node kubelet SyncLoop scheduler api-server etcd proxy kubelet SyncLoop

27 Example proxy kubelet SyncLoop scheduler api-server etcd proxy 4.1 Detected pod bind with me 4.2 Start containers in pod kubelet SyncLoop

28 Design of kubelet Choose Runtime docker, rkt, hyper/remote NodeStatus Network Status status Manager PLEG InitNetworkPlugin SyncLoop volume Manager Pod Update Worker (e.g.add) generale Pod status check volume status (talk later) call runtime to start containers set up Pod network (see next slide) image Manager PodUpdate HandlePods {Add, Update, Remove, Delete, }

29 Set Up Pod Network

30 kubestack A standalone grpc daemon 1. to translate the SetUpPod request to the Neutron network API 2. handling multi-tenant Service proxy

31 Service OnServiceUpdate $ iptables-save grep my-service -A KUBE-SERVICES -d /32 -p tcp -m comment --comment "default/my-service: cluster IP" -m tcp --dport j KUBE-SVC-KEAUNL7HVWWSEZA6 -A KUBE-SVC-KEAUNL7HVWWSEZA6 -m comment --comment "default/my-service:" --mode random -j KUBE-SEP-6XXFWO3KTRMPKCHZ -A KUBE-SVC-KEAUNL7HVWWSEZA6 -m comment --comment "default/my-service:" --mode random -j KUBE-SEP-57KPRZ3JQVENLNBRZ -A KUBE-SEP-6XXFWO3KTRMPKCHZ -p tcp -m comment --comment "default/my-service:" -m tcp -j DNAT --to-destination :80 -A KUBE-SEP-57KPRZ3JQVENLNBRZ -p tcp -m comment --comment "default/my-service:" -m tcp -j DNAT --to-destination :80 OnEndpointsUpdate portal :8001 backend rule_ :80 random mode rules backend rule_ :80

32 Multi-tenant Service Default iptables-based kube-proxy is not tenant aware Endpoint Pods and Nodes with iptables rules are isolated into different networks Hypernetes uses a built-in HAproxy as the Service portal to handle all Service instances within same namespace the same OnServiceUpdate and OnEndpointsUpdate workflow ExternalProvider a OpenStack LB will be created as Service e.g. curl :8078

33 Persistent Volume

34 Kubernetes Persistent Volume Get mountedvolume from actualstateofworld mount Host Unmount volumes in mountedvolume but not in desiredstateofworld AttachVolume() if vol in desiredstateofworld and not attached Pod mountpath Pod mountpath MountVolume() if vol in desiredstateofworld and not in mountedvolume Verify devices that should be detached/unmounted are detached/unmounted attach path Cinder volume plugin Tips: 1. -v host:path Volume Manager desired World 2. attach VS mount 3. Totally independent from container management reconcile

35 Persistent Volume with HyperContainer Enhanced Cinder volume plugin Host Linux container: Pod Pod 1. full OpenStack cluster mountpath mountpath 2. query Nova to find node 3. attach Cinder volume to host path attach vol vol 4. bind mount host path to Pod containers Enhanced Cinder volume plugin HyperContainer: directly attach block devices to Pod thanks to the hypervisor based Pod boundary Volume Manager desired World eliminates extra time to query Nova reconcile

36 PV Example Create a Cinder volume Claim volume by reference its volumeid

37 Container Runtime Interface

38 Future of CRI Keep Docker as the only one default container runtime oci-runtime, rktlet, hyperd Frakti: the Remote Container Runtime Kit welcome to tryout, star and fork

39 if image becomes non-standard e.g. Docker image becomes somehow Docker specific Don t worry, kubelet.imagemanager is moving to runtime specific but then k8s will probably choose NO DEFAULT runtime

40 Full Topology Node Node Node KeyStone Pod Pod Pod Pod Master Neutron Object: Network kubestack kube-proxy Object: Pod Cinder Ceph Neutron L2 Agent kubelet Object: Cinder Plugin

41 Summary A new way to build secure and multi-tenant Kubernetes Kubernetes + HyperContainer + Neutron Plugin + Cinder Plugin + Keystone Project URL: Roadmap Graduate HyperContainer runtime on k8s upstream see HyperContainer in official k8s release Neutron CNI plugin Tip: is totally built on Hypernetes, try it out :)

42 END Lei (Harry)