Scheduling in Kubernetes October, 2017

Size: px

Start display at page:

Download "Scheduling in Kubernetes October, 2017"

Prudence James
6 years ago
Views:

1 Scheduling in Kubernetes October, 2017

2 What to look for Kubernetes overview Scheduling algorithm Scheduling controls Advanced scheduling techniques Examples and use cases

3 Kubernetes Technology stack Docker Distribution Configuration Isolation Kubernetes Orchestration Network Configuration Service discovery Ingress Persistence

4 Docker Architecture Instance Docker CLI Docker image repository Images Docker daemon Application Application containers containers Overlay network App data

5 Kubernetes Architecture Master Node kubelet K8s node components: overlay network, discovery, connectivity K8s master components: etcd, scheduler, api, controller Docker Infrastructure Infrastructure and and application application containers containers Overlay network K8s metadata App data

6 Kubernetes Nodes and Pods Pod A-1 Pod B-1 Pod A-2 Node 2 Cnt1 Cnt3 Cnt1 Cnt Cnt

7 Kubernetes Container Orchestration K8S Master API Kubelet User Pod A Pod B Node 2 Pod C Docker Pod A K8S Controller(s) K8S Scheduler(s) Pod B

8 Kubernetes Container Orchestration K8S Master API Kubelet User Docker K8S Controller(s) K8S Scheduler(s) It all starts empty

9 Kubernetes Container Orchestration User K8S Master API Kubelet Docker K8S Controller(s) Kubelet registers Node object in Master K8S Scheduler(s)

10 Kubernetes Container Orchestration User K8S Master API Node 2 Kubelet Docker K8S Controller(s) K8S Scheduler(s)

11 Kubernetes Container Orchestration K8S Master API User Pod C Node 2 Kubelet Pod A Pod B Docker K8S Controller(s) K8S Scheduler(s) User creates (unscheduled) Pod object(s) in Master

12 Kubernetes Container Orchestration User Scheduler notices unscheduled Pods... Pod B K8S Controller(s) K8S Master API Pod A Node 2 Pod C K8S Scheduler(s) Kubelet Docker

13 Kubernetes Container Orchestration User... finds the best Node to run them... Pod B K8S Controller(s) K8S Master API Pod A Node 2 Pod C K8S Scheduler(s) Kubelet Docker

14 Kubernetes Container Orchestration User K8S Master API Pod A Pod B Node 2 Pod C Kubelet... and marks the Pods as scheduled on corresponding Nodes K8S Controller(s) K8S Scheduler(s) Docker

15 Kubernetes Container Orchestration User K8S Master API Pod A Pod B Node 2 Pod C Kubelet Docker K8S Controller(s) Kubelet notices Pods scheduled to its node... K8S Scheduler(s)

16 Kubernetes Container Orchestration User K8S Master API Pod A Pod B Node 2 Pod C Kubelet Docker Pod A K8S Controller(s)... and starts Pods containers K8S Scheduler(s) Pod B

17 Kubernetes Scheduling Algorithm User Scheduler finds the best Node to run Pods How? Pod B K8S Controller(s) K8S Master API Pod A Node 2 Pod C K8S Scheduler(s) Kubelet Docker

18 Kubernetes Scheduling Algorithm For each pod that needs scheduling: 1. Filter nodes 2. Calculate nodes priorities 3. Pick node with the highest priority

19 Scheduling Algorithm Filters Volume filters Do pod requested volumes zones fit the node s zone? Can the node attach the volumes? Are there mounted volumes conflicts? Are there additional volume topology constraints? Volume filters Resource filters Topology filters Prioritization

20 Scheduling Algorithm Filters Resource filters Does pod requested resources (CPU, RAM GPU, etc) fit the node s available resources? Can pod requested ports be opened on the node? Is there no memory or disk pressure on the node? Volume filters Resource filters Topology filters Prioritization

21 Scheduling Algorithm Filters Topology filters Is Pod requested to run on this node? Are there inter-pod affinity constraints? Does the node match Pod s node selector? Can Pod tolerate node s taints? Volume filters Resource filters Topology filters Prioritization

22 Scheduling Algorithm Priorities Prioritize with weights for: Pod replicas distribution Least (or most) node utilization Balanced resource usage Inter-pod affinity priority Node affinity priority Taint toleration priority Volume filters Resource filters Topology filters Prioritization

23 Scheduling Controlling Pods Destinyation Specify resource requirements Be aware of volumes Use node constraints Use affinity and anti-affinity Scheduler configuration Custom / multiple schedulers

24 Scheduling Controlled Resources CPU, RAM, other (GPU) Requests and limits Reserved resources kind: Node kind: Pod spec: containers: - name: main resources: requests: cpu: 100m memory: 1Gi status: allocatable: cpu: "4" memory: Ki pods: "110" capacity: cpu: "4" memory: 8Gi pods: "110"

25 Scheduling Controlled Volumes Request volumes in the right zones Make sure node can attach enough volumes Avoid volume location conflicts Use volume topology constraints (alpha in 1.7) Node 2 Pod A Unschedulable Pod C Pod B Requested Volume Zone A Zone B Volume 2

26 Scheduling Controlled Volumes Request volumes in the right zones Make sure node can attach enough volumes Avoid volume location conflicts Use volume topology constraints (alpha in 1.7) Pod A Pod B Pod C Volume 1 Volume 2 Requested Volume

27 Scheduling Controlled Volumes Request volumes in the right zones Make sure node can attach enough volumes Avoid volume location conflicts Use volume topology constraints (alpha in 1.7) Node 2 Pod A Pod C Pod B Volume 1 Volume 2

28 Scheduling Controlled Volumes Request volumes in the right zones Make sure node can attach enough volumes Avoid volume location conflicts Use volume topology constraints (alpha in 1.7) annotations: "volume.alpha.kubernetes.io/node-affinity": '{ "requiredduringschedulingignoredduringexecution": { "nodeselectorterms": [{ "matchexpressions": [{ "key": "kubernetes.io/hostname", "operator": "In", "values": ["docker03"] }] }] }}'

29 Scheduling Controlled Node Constraints Host constraints Labels and node selectors Taints and tolerations Pod A kind: Pod spec: nodename: node1 kind: Node metadata: name: node1

30 Scheduling Controlled Node Constraints Host constraints Labels and node selectors Taints and tolerations Pod A Node 2 label: tier: backend Node 3 kind: Pod spec: nodeselector: tier: backend kind: Node metadata: labels: tier: backend

31 Scheduling Controlled Node Constraints Host constraints Labels and node selectors Taints and tolerations Pod A tolerate tainted Pod B kind: Pod spec: tolerations: - key: error value: disk operator: Equal effect: NoExecute tolerationseconds: 60 kind: Node spec: taints: - effect: NoSchedule key: error value: disk timeadded: null

32 Scheduling Controlled Taints Taints communicate node conditions Key condition category Value specific condition Operator value wildcard Equal Exists Effect kind: Pod spec: tolerations: - key: <taint key> value: <taint value> operator: <match operator> effect: <taint effect> tolerationseconds: 60 NoSchedule filter at scheduling time PreferNoSchedule prioritize at scheduling time NoExecute filter at scheduling time, evict if executing TolerationSeconds time to tolerate NoExecute taint

33 Scheduling Controlled Affinity Node affinity Inter-pod affinity Inter-pod anti-affinity kind: Pod spec: affinity: nodeaffinity: {... } podaffinity: {... } podantiaffinity: {... }

34 Scheduling Controlled Node Affinity Scope Preferred during scheduling, ignored during execution Required during scheduling, ignored during execution kind: Pod spec: affinity: nodeaffinity: preferredduringschedulingignoredduringexecution: - weight: 10 preference: { <node selector term> } -... requiredduringschedulingignoredduringexecution: nodeselectorterms: - { <node selector term> } -...

35 Interlude Node Selector vs Node Selector Term... nodeselector: <label 1 key>: <label 1 value> <node selector term>: matchexpressions: - key: <label key> operator: In NotIn Exists DoesNotExist Gt Lt values: - <label value 1>......

36 Scheduling Controlled Inter-pod Affinity Scope Preferred during scheduling, ignored during execution Required during scheduling, ignored during execution kind: Pod spec: affinity: podaffinity: preferredduringschedulingignoredduringexecution: - weight: 10 podaffinityterm: { <pod affinity term> } -... requiredduringschedulingignoredduringexecution: - { <pod affinity term> } -...

37 Scheduling Controlled Inter-pod Anti-affinity Scope Preferred during scheduling, ignored during execution Required during scheduling, ignored during execution kind: Pod spec: affinity: podantiaffinity: preferredduringschedulingignoredduringexecution: - weight: 10 podaffinityterm: { <pod affinity term> } -... requiredduringschedulingignoredduringexecution: - { <pod affinity term> } -...

38 Scheduling Controlled Pod Affinity Term topologykey nodes label key defining co-location labelselector and namespaces select group of pods <pod affinity term>: topologykey: <topology label key> namespaces: [ <namespace>,... ] labelselector: matchlabels: <label key>: <label value>... matchexpressions: - key: <label key> operator: In NotIn Exists DoesNotExist values: [ <value 1>,... ]...

39 Scheduling Controlled Affinity Example Pod B group: a affinity: topologykey: tier labelselector: matchlabels: group: a tier: a tier: a Pod B group: a tier: a Node 3 tier: b Node 4 tier: b tier: b

40 Scheduling Controlled Scheduler Configuration kube-scheduler --scheduler-name=default-scheduler --algorithm-provider=defaultprovider --algorithm-provider=clusterautoscalerprovider

41 Scheduling Controlled Multiple Schedulers kind: Pod Metadata: name: pod2 spec: schedulername: my-scheduler kind: Pod Metadata: name: pod1 spec:...

42 Scheduling Controlled Custom Scheduler Naive implementation In an infinite loop: Get list of Nodes: /api/v1/nodes Get list of Pods: /api/v1/pods Select Pods with status.phase == Pending and spec.schedulername == our-name For each pod: apiversion: v1 kind: Binding Metadata: namespace: default name: pod1 target: apiversion: v1 kind: Node name: node1 Calculate target Node Create a new Binding object: POST /api/v1/bindings

43 Scheduling Controlled Custom Scheduler Better implementation Watch Pods: /api/v1/pods On each Pod event: Process if the Pod with status.phase == Pending and spec.schedulername == our-name apiversion: v1 kind: Binding Metadata: namespace: default name: pod1 target: apiversion: v1 kind: Node name: node1 Get list of Nodes: /api/v1/nodes Calculate target Node Create a new Binding object: POST /api/v1/bindings

44 Scheduling Controlled Custom Scheduler Even better implementation Watch Nodes: /api/v1/nodes On each Node event: Update Node cache Watch Pods: /api/v1/pods On each Pod event: Process if the Pod with status.phase == Pending and spec.schedulername == our-name Calculate target Node apiversion: v1 kind: Binding Metadata: namespace: default name: pod1 target: apiversion: v1 kind: Node name: node1 Create a new Binding object: POST /api/v1/bindings

45 Custom Scheduler Standard Filters Minimal set of filters kube-scheduler Extend Re-implement GitHub kubernetes/kubernetes plugin/pkg/scheduler/scheduler.go plugin/pkg/scheduler/algorithm/predicates/predicates.go

46 Use Case Distributed Pods apiversion: v1 kind: Pod metadata: name: db-replica-3 labels: db-replica-3 comp: db spec: affinity: podantiaffinity: requiredduringschedulingignoredduringexecution: - topologykey: kubernetes.io/hostname labelselector: matchexpressions: - key: comp operator: In values: [ "db" ] db-replica-1 Node 2 db-replica-2 Node 3

47 Use Case Co-located Pods apiversion: v1 kind: Pod metadata: name: app-replica-1 labels: app-replica-1 comp: app spec: affinity: podaffinity: requiredduringschedulingignoredduringexecution: - topologykey: kubernetes.io/hostname labelselector: matchexpressions: - key: comp operator: In values: [ "db" ] db-replica-1 Node 2 db-replica-2 Node 3

48 Use Case Reliable service on spot nodes fixed node group Expensive, more reliable, fixed number Tagged with label nodegroup: fixed spot node group Inexpensive, unreliable, auto-scaled Tagged with label nodegroup: spot Scheduling rules: At least two pods on fixed nodes All other pods favor spot nodes Custom scheduler

49 Thank you Oleg Chunikhin Chief Technology Officer kublr.com

Kubernetes - Load Balancing For Virtual Machines (Pods)

Kubernetes - Load Balancing For Virtual Machines (Pods) 4 th of Feb 2018 Yanir Quinn Senior Software Engineer Red Hat This presentation is licensed under a Creative Commons Attribution 4.0 International