Scheduling Applications at Scale

Transcription

1 Scheduling Applications at Scale Meeting Tomorrow's Application Needs, Today

2 SETH

3

4

5 Globally Distributed Optimistically Concurrent Scheduler Nomad

6 Globally Distributed Optimistically Concurrent Scheduler Nomad

7 sched ul er COMPUTING (n.) a program that arranges jobs or a computer's operations into an appropriate sequence.

8 sched ul er NORMAL (n.) a person or machine that organizes or maintains schedules.

9 Schedulers map a set of work to a set of resources

10

11 Datacenter Operator

12 Skywalker Vader Leia Solo Datacenter Operator

13 PYTHON GOLANG GOLANG PYTHON GOLANG Skywalker Vader Leia Solo Datacenter Operator

14 RUBY PYTHON PYTHON PYTHON GOLANG GOLANG GOLANG GOLANG NODE Skywalker Vader Leia Solo Datacenter Operator

15 RUBY :45:13:B6:87:C4 VADER CA:9A:3D:7F:8B:CB Randomly kills applications LEIA :CE:4F:C8:54:C3 SOLO :30:9C:0D:1E:74 RUBY PYTHON PYTHON PYTHON GOLANG GOLANG GOLANG GOLANG NODE Skywalker Vader Leia Solo Datacenter Operator

16 RUBY :45:13:B6:87:C4 VADER CA:9A:3D:7F:8B:CB Randomly kills applications LEIA :CE:4F:C8:54:C3 SOLO :30:9C:0D:1E:74 RUBY PYTHON PYTHON PYTHON GOLANG GOLANG GOLANG NODE Datacenter GOLANG SkywalkerFVader Leia Solo Operator

17 RUBY :45:13:B6:87:C4 VADER CA:9A:3D:7F:8B:CB Randomly kills applications LEIA :CE:4F:C8:54:C3 SOLO :30:9C:0D:1E:74 RUBY PYTHON GOLANG NODE PYTHON PYTHON PYTHON GOLANG PYTHON PYTHON GOLANG GOLANG SkywalkerFVader Leia Solo Datacenter Operator

18 RUBY :45:13:B6:87:C4 VADER CA:9A:3D:7F:8B:CB Randomly kills applications LEIA :CE:4F:C8:54:C3 SOLO :30:9C:0D:1E:74 RUBY PYTHON PYTHON GOLANG GOLANG GOLANG GOLANG NODE PYTHON Skywalker Vader Leia Solo Datacenter Operator

19 RUBY :45:13:B6:87:C4 VADER CA:9A:3D:7F:8B:CB Rebuilt on 04/20/2016 LEIA :CE:4F:C8:54:C3 SOLO :30:9C:0D:1E:74 RUBY PYTHON PYTHON GOLANG GOLANG GOLANG GOLANG NODE PYTHON Skywalker Vader Leia Solo Datacenter Operator

20 RUBY :45:13:B6:87:C4 VADER CA:9A:3D:7F:8B:CB Rebuilt on 04/20/2016 LEIA :CE:4F:C8:54:C3 SOLO :30:9C:0D:1E:74 RUBY PYTHON PYTHON GOLANG GOLANG GOLANG GOLANG NODE PYTHON Skywalker Vader Leia Solo Datacenter Operator

21 This does not scale

22 CPU Scheduler KERNEL CORE APACHE REDIS CPU SCHEDULER CORE CORE BASH CORE

23 CPU Scheduler KERNEL CORE APACHE REDIS CPU SCHEDULER CORE BASH

24 Schedulers in the Wild Type Work Resources CPU Scheduler Threads Physical Cores EC2 / Nova Virtual Machines Hypervisors Hadoop YARN MapReduce Jobs Client Nodes Cluster Scheduler Applications Machines

25 Scheduler Advantages Higher Resource Utilization Decouple Work from Resources Better Quality of Service

26 Scheduler Advantages Higher Resource Utilization Bin Packing Decouple Work from Resources Over-Subscription Better Quality of Service Job Queueing

27 Scheduler Advantages Higher Resource Utilization Abstraction Decouple Work from Resources API Contracts Better Quality of Service Standardization

28 Scheduler Advantages Higher Resource Utilization Priorities Decouple Work from Resources Resource Isolation Better Quality of Service Pre-emption

29 Not a New Concept

30 Not Alone

31 Nomad

32 Cluster Scheduler Deployments Job Specification Nomad

33 example.nomad job "redis" { datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB } } } network { mbits = 10 dynamic_ports = ["redis"] }

34 Job specification declares what to run

35 Nomad determines how and where to run

36 Nomad abstracts work from resources

37 Higher Resource Utilization Decouple Work from Resources Better Quality of Service Nomad

38 Designing Nomad

39 Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin Packing Nomad Large Scale Operationally Simple

40 Scaling Requirements Thousands of regions Tens of thousands of clients per region Thousands of jobs per region

41 Built on Experience GOSSIP CONSENSUS

42 Cluster Management Gossip Based (P2P) Membership Failure Detection Serf Event System

43 Gossip Protocol Large Scale Production Hardened Serf Operationally Simple

44 Service Discovery Configuration Coordination (Locking) Central Servers + Distributed Clients Consul

45 Multi-Datacenter Raft Consensus Large Scale Production Hardened Consul

46 Built on Experience Mature Libraries Proven Design Patterns GOSSIP CONSENSUS

47 Built on Experience Mature Libraries Proven Design Patterns GOSSIP CONSENSUS Lacking Scheduling Logic

48 Built on Research GOSSIP CONSENSUS

49

50 Optimistic vs Pessimistic Internal vs External State Single vs Multi Level Fixed vs Pluggable Service vs Batch Oriented

51 Inspired by Google Omega Optimistic Concurrency State Coordination Service & Batch workloads Nomad Pluggable Architecture

52 Consul Architecture CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT RPC LAN GOSSIP RPC SERVER SERVER SERVER REPLICATION REPLICATION WAN GOSSIP SERVER REPLICATION SERVER REPLICATION SERVER

53 Multi-Datacenter Servers per DC Failure Isolation Domain is the Datacenter Consul

54 Single-Region Architecture CLIENT CLIENT CLIENT DC1 DC2 DC3 RPC RPC RPC SERVER REPLICATION SERVER REPLICATION SERVER FORWARDING FORWARDING FOLLOWER LEADER FOLLOWER

55 Multi-Region Architecture REGION A SERVER SERVER SERVER REPLICATION REPLICATION FOLLOWER LEADER FORWARDING FOLLOWER REGION B GOSSIP REGION FORWARDING SERVER FOLLOWER REPLICATION SERVER LEADER REPLICATION FORWARDING SERVER FOLLOWER

56 Region is Isolation Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary Nomad

57 Data Model NODE EVALUATION ALLOCATION JOB

58 Evaluation ~= State Change

59 Evaluations Create / Update / Delete Job Node Up / Node Down Allocation Failed

60 Evaluations SCHEDULER func(evaluation) => []AllocationUpdates

61 Evaluations SCHEDULER func(evaluation) => []AllocationUpdates Service, Batch, System

62 Server Architecture Omega Class Scheduler Pluggable Logic Internal Coordination and State Multi-Region / Multi-Datacenter

63 Client Architecture Broad OS Support Host Fingerprinting Pluggable Drivers

64 Fingerprinting Type Examples Operating System Kernel, OS, Version Hardware CPU, Memory, Disk Apps (Capabilities) Docker, Java, Consul Environment AWS, GCE

65 Constrain Placement and Bin Pack

66 Task Requires Linux, Docker, and PCI- Compliant Hardware expressed as constraints in job file

67 Task needs 512MB RAM and 1 Core expressed as resource in job file

68 Drivers Execute Tasks Provide Resource Isolation

69 Docker Containerized rkt Qemu / KVM Virtualized Java Jar Standalone Static Binaries

70 Docker Containerized rkt Windows Server Containers Qemu / KVM Virtualized Xen Hyper-V Java Jar Standalone Static Binaries C#

71 Schedulers Fingerprints Drivers Job Specification Nomad

72 Single Binary No Dependencies Highly Available Nomad

73 Nomad Million Container Challenge 1,000 Jobs 1,000 Tasks per Job 5,000 Hosts on GCE 1,000,000 Containers

74

75 No one would ever need to schedule a million containers. Negative Nancy

76

77

78

79 Globally Distributed Optimistically Concurrent Scheduler Nomad

80 Higher Resource Utilization Decouple Work from Resources Better Quality of Service Nomad

81 QUESTIONS? SETH