Evolution of Kubernetes in One Year From Technical View

Size: px

Start display at page:

Download "Evolution of Kubernetes in One Year From Technical View"

Phillip Garrison
6 years ago
Views:

1 Evolution of Kubernetes in One Year From Technical View Harry Zhang

2 Background

3 Docker = Fan economy Github search stack overflow DockerCon de facto Docker Kubernetes

4 diversity Docker Image Image ACI RunC RunV Runtime rkt (AppC) Mesos Containerizer containerd ocid hyperd Containerizer Container Runtime Interface Docker Daemon Kubernetes Mesos Legacy* Control Panel

5 Architecture

6 Example proxy kubelet SyncLoop 1 Pod created scheduler api-server etcd proxy kubelet SyncLoop

7 Example proxy kubelet SyncLoop scheduler 2 Pod object added api-server etcd proxy kubelet SyncLoop

8 Example proxy 3.1 New pod object detected 3.2 Bind pod with node kubelet SyncLoop scheduler api-server etcd proxy kubelet SyncLoop

9 Example proxy kubelet SyncLoop scheduler api-server etcd proxy 4.1 Detected pod bind with me 4.2 Start containers in pod kubelet SyncLoop

10 Desired World Real World controller-manager ControlLoop Architecture pod replica namespace service endpoint job deployment volume petset kubelet SyncLoop proxy api-server etcd proxy scheduler kubelet SyncLoop

11 Kubernetes Components De-coupling Extensible Event loop driven Distributed data oriented Robust Not only HA, but also avoid some races e.g. should undesired pod be considered by scheduler? No, it should be handled by controller

12 Bootstrapping

13 old 1. $ systemctl start etcd Self-healing new 2. $ systemctl start kubelet 3. $ systemctl start kube-proxy 4. $ systemctl start kube-apiserver 1. $ hyperkube kubelet 2. $ hyperkube kube-proxy Done! 5. $ systemctl start kube-controller-manager 6. $ systemctl start kube-scheduler

14 Self-healing $ hyperkube kubelet config=/etc/kubernetes/manifests Kubelet will load all of those pods And bind them with this node $ ls /etc/kubernetes/manifests etcd.yaml kube-apiserver.yaml kube-scheduler.yaml kube-controller-manager.yaml

15 Topology etcd api-server scheduler controller-manager kubelet kubelet kubelet kubelet kube-proxy kube-proxy kube-proxy

16 With CNI Network calico-policycontroller etcd api-server scheduler controller-manager kubelet kubelet kubelet kubelet kube-proxy kube-proxy kube-proxy calico/node calico/node calico/node

17 Production Topology LB etcd api-server scheduler controller-manager calico-policycontroller etcd api-server scheduler controller-manager kubelet kubelet kubelet kubelet kube-proxy kube-proxy kube-proxy calico/node calico/node calico/node

18 Scheduling

19 Pod The atomic scheduling unit* The container (micro-service) design pattern The container in Kubernetes *EuroSys 15, Large-scale cluster management at Google with Borg, Section 8.1

20 atomic scheduling unit Should sample.war be packaged with Tomcat?

21 How? new InitContainers: one or more containers started in sequence before the pod's normal containers are started. Share volumes, perform network operations, and perform computation prior to the app containers.

22 Container Design Pattern* Spring Framework Container Boundary Self-contained Atomic deployment signal ( succeeded / failed ) + tomcat serverless Do right things without modifying your container image * HotCloud 16, Design Patterns for Container-based Distributed Systems

23 sidecar* *Google Internal: web+logstash, web+git sync etc

24 Pod Super affinity containers Share network, storage init container + Tomcat log + app Process VS process group But why?

25 gang scheduling log super affinity app Requirement: swarm peer node Omega 3. app: 1G, log: 0.5G Available: Node_A: 1.25G, Node_B: 2G What happens if app scheduled to Node_A?

26 Pod Summary Pod = container affinity ( link, volumes-from, net=xxx, ) Kubernetes is for freshman to build high quality systems think about Spring Framework DO NOT modify your image! Anyone still assumes Worldpress + MySQL belongs to one Pod?

27 Schedule Strategy Predicates NoDiskConflict NoVolumeZoneConflict PodFitsResources PodFitsHostPorts MatchNodeSelector MaxEBSVolumeCount MaxGCEPDVolumeCount CheckNodeMemoryPressure eviction, QoS tiers CheckNodeDiskPressure Priorities LeastRequestedPriority BalancedResourceAllocation SelectorSpreadPriority CalculateAntiAffinityPriority ImageLocalityPriority NodeAffinityPriority

28 Resource Model requests for scheduling limits for enforcement cpu-shares=requests.cpu cpu-quota=limits.cpu cpu-period=100ms memory=limits.memory Compressible Resource CPU, throttle Incompressible Resource Memory, killed by kernel

29 QoS Tiers Guaranteed limits is set for all resources, all containers limits == requests (if set) Be killed until they exceed their limits Burstable or if the system is under memory pressure and there are no lower priority containers that can be killed. requests is set for one or more resources, one or more containers limits (if set)!= requests killed once they exceed their requests and no Best-Effort pods exist when system under memory pressure Best-Effort requests and limits are not set for all of the resources, all containers First to get killed if the system runs out of memory

30 Multi-Scheduler The 2nd scheduler annotation: system usage labels Do NOT abuse labels

31 Control Panel

32 DaemonSet Spread daemon pod to every node DaemonSet Controller don t need scheduler even on unschedulable nodes e.g. bootstrap

33 Deployment Replicas with control Bring up a Replica Set and Pods. Check the status of a Deployment. Update that Deployment (e.g. new image, labels). Rollback to an earlier Deployment revision. Pause and resume a Deployment.

34 Create ReplicaSet Next generation of ReplicaController record: record command in the annotation of nginx-deployment

35 Check DESIRED:.spec.replicas CURRENT:.status.replicas UP-TO-DATE: contains the latest pod template AVAILABLE: pod status is ready (running)

36 Update kubectl set image will change container image kubectl edit open an editor and modify your deployment yaml trigger RollingUpdateStrategy 1 max unavailable 1 max surge can also be percentage Does not kill old Pods until a sufficient number of new Pods have come up Does not create new Pods until a sufficient number of old Pods have been killed.

Update Process The update process is coordinated by Deployment Controller Create: Replica Set (nginx-deployment-2035384211) and scaled it up to 3 replicas directly.

37 Update Process The update process is coordinated by Deployment Controller Create: Replica Set (nginx-deployment ) and scaled it up to 3 replicas directly. Update: created a new Replica Set (nginx-deployment ) and scaled it up to 1 scaled down the old Replica Set to 2 continued scaling up and down the new and the old Replica Set, with the same rolling update strategy. Finally, 3 available replicas in the new Replica Set, and the old Replica Set is scaled down to 0.

38 Rolling Back Check reversions Roll back to reversion

39 Pausing & Resuming (Canary) Tips blue-green deployment: duplicated infrastructure canary release: share same infrastructure rollback resumed deployment is WIP old way: kubectl rolling-update rc-1 rc-2

40 Horizontal Pod Autoscaling Tips Scale out/in TriggeredScaleUp (GCE, AWS, will add more) Support for custom metrics

41 Custom Metrics Endpoint (Location to collect metrics from) Name of metric Type (Counter, Gauge,...) Prometheus Data Type (int, float) Units (kbps, seconds, count) Polling Frequency Regexps (Regular expressions to specify which metrics to collect and how to parse them) The metric will be added to pod as ConfigMap volume Nginx

42 ConfigMap Decouple configuration from image configuration is a runtime attribute Can be consumed by pods thru: env volumes

43 ConfigMap Volume No need to use Persistent Volume Think about Etcd

44 Tip: credentials for accessing the k8s API is automatically added to your pods as secret Secret

45 Downward Api Get these inside your pod as ENV or volume The pod s name The pod s namespace The pod s IP A container s cpu limit A container s cpu request A container s memory limit A container s memory request

46 Service The unified portal of replica containers Portal IP:Port External load balancer GCE AWS HAproxy Nginx OpenStack

47 Headless Service *.nginx.default.svc.cluster.local app=nginx app=nginx app=nginx

48 User-space Service Etcd User Request Pod_IP:Port 1. Service change 2. Endpoint change Node :80 Node-1 // from container:prerouting target prot opt source destination REDIRECT tcp / tcp dpt:80 redir ports // from HOST:OUTPUT target prot opt source destination DNAT tcp / tcp dpt:80 to: :43318 maintain the rules kube-proxy Proxier Load Balancer Round-Robin Session Affinity old

iptables Service $ iptables-save grep my-service -A KUBE-SERVICES -d 10.

"default/my-service:" --mode random -j KUBE-SEP-6XXFWO3KTRMPKCHZ -A KUBE-SVC-KEAUNL7HVWWSEZA6 -m comment --comment "default/my-service:" --mode random -j KUBE-SEP-57KPRZ3JQVENLNBRZ -A

49 iptables Service $ iptables-save grep my-service -A KUBE-SERVICES -d /32 -p tcp -m comment --comment "default/my-service: cluster IP" -m tcp --dport j KUBE-SVC-KEAUNL7HVWWSEZA6 -A KUBE-SVC-KEAUNL7HVWWSEZA6 -m comment --comment "default/my-service:" --mode random -j KUBE-SEP-6XXFWO3KTRMPKCHZ -A KUBE-SVC-KEAUNL7HVWWSEZA6 -m comment --comment "default/my-service:" --mode random -j KUBE-SEP-57KPRZ3JQVENLNBRZ -A KUBE-SEP-6XXFWO3KTRMPKCHZ -p tcp -m comment --comment "default/my-service:" -m tcp -j DNAT --to-destination :80 -A KUBE-SEP-57KPRZ3JQVENLNBRZ -p tcp -m comment --comment "default/my-service:" -m tcp -j DNAT --to-destination :80 new Tip: ipvs solution works in nat mode which is the same with this iptables way

50 Tips User-space proxy service is slow please use: proxy-mode=iptables requires newer iptables version >= (released 2011-May-26).

51 Publishing Services Use Service.Type=NodePort <node_ip>:<node_port> External IP IPs route to one or more cluster nodes (e.g. floating IP) Use external LoadBalancer Require support from IaaS (GCE, AWS, OpenStack) Deploy a service-loadbalancer (e.g. HAproxy) Official guide:

52 Ingress The next generation external Service load balancer Deployed as a Pod on dedicated Node (with external network) Implementation Nginx, HAproxy, GCE L7 External access for service SSL support for service <IP_of_Ingress_node> s1

53 TLS Your Ingress TLS Only supports a single TLS port 443

54 Write Your Ingress Poll until apiserver reports a new Ingress Write the nginx config file based on a go text/template Reload nginx

55 PetSet Stateless application management Replicas Stateful application management Pets

56 clustered applications Stable hostname Peer discovery Ordinal index startup/teardown ordering Stable storage linked to the ordinal & hostname Databases like MySQL or PostgreSQL single instance attached to a persistent volume at any time Clustered software like Zookeeper, Etcd, or Elasticsearch, Cassandra stable membership.

57 PetSet Example $ kubectl patch petset cassandra -p '{"spec":{"replicas":10}}' cassandra-1.cassandra.default.svc.cluster.local cassandra-0.cassandra.default.svc.cluster.local cassandra-0 cassandra-1 volume 0 volume 1

58 ScheduledJobs Distributed Cron format job with container alpha kubectl run pi --image=perl --restart=onfailure --runat=" *" -- perl -Mbignum=bpi -wle 'print bpi(2000) July 21th, 2pm Fault-Tolerance Multiple controller processes Ensuring jobs are run at most once Unlike normal pods, there should never be more than one job running at the same time Deterministic name

59 Network

One Pod One IP Pod /proc/{pid}/ns/net -> net:[4026532483] Network sharing is important for affiliate containers Not all containers need

60 One Pod One IP Pod /proc/{pid}/ns/net -> net:[ ] Network sharing is important for affiliate containers Not all containers need independent network Container A --net=container:pause Infra container Container B Network implementation for pod is totally the same as for single container

61 Network Model Container reach container all containers can communicate with all other containers without NAT Node reach container all nodes can communicate with all containers (and viceversa) without NAT IP addressing Pod in cluster can be addressed by its IP

62 Network Implementation Cloud provider CNI plugin e.g. Calico, Flannel etc The kubelet cni flags: --network-plugin=cni --network-plugin-dir=/etc/cni/net.d

63 Calico Step 1: Run calico-node image as DaemonSet

64 Calico Step 2: Download and enable calico cni plugin

65 Calico Step 3: Add calico network controller Done!

66 Tips host < calico(bgp) < calico(ipip) = flannel(vxlan) = docker(vxlan) < flannel(udp) < weave(udp) Test graph comes from:

67 Persistent Volumes There is only one kind of volume in container world -v host_path:container_path networked storage mounted to host_path container volume bind mount container_path with host_path

68 Advanced Volume Model PersistentVolumeClaims Pod Host Pod mountpath Pod mountpath Persistent Volumes path networked storage Best practice: Persistent volumes should be handled by professionals

69 Why PV & PVC? Volume path on the host is always shared by containers from different users PVC: Sharing things needs fine-grained control: pod > PVC > PV System Admin: $ kubectl create -f nfs-pv.yaml create a volume with access mode, capacity, recycling mode Dev: $ kubectl create -f pv-claim.yaml request a volume with access mode, resource, selector $ kubectl create -f pod.yaml

70 Officially Supported PVs GCEPersistentDisk AWSElasticBlockStore AzureFile FC (Fibre Channel) NFS iscsi RBD (Ceph Block Device) CephFS Cinder (OpenStack block storage) Glusterfs VsphereVolume HostPath (single node testing only local storage is not supported in any way and WILL NOT WORK in a multi-node cluster)

71 SLO Kubernetes SLO API responsiveness: 99% of all API calls return in less than 1s Pod startup time: 99% of pods and their containers (with pre-pulled images) start within 5s.

72 Summary Kubernetes is not a container management, scheduling, orchestration tool that s compose + swarm Kubernetes: the Spring Framework in cloud area Pod is like IOC in Spring Framework decoupling, refactor & rescue, best practice A framework for people to build right system with container.

73 THE END Lei (Harry) Kubernetes Project Member CNCF Member Docker

Bringing Security and Multitenancy. Lei (Harry) Zhang

Bringing Security and Multitenancy to Kubernetes Lei (Harry) Zhang About Me Lei (Harry) Zhang #Microsoft MVP in cloud and datacenter management though I m a Linux guy :/ Previous: VMware, Baidu Feature