Armon HASHICORP

Transcription

1 Nomad

2 Armon

3

4

5 Distributed Optimistically Concurrent Scheduler Nomad

6 Distributed Optimistically Concurrent Scheduler Nomad

7 Schedulers map a set of work to a set of resources

8 Work (Input) Resources Web Server -Thread 1 Web Server -Thread 2 Redis -Thread 1 CPU Scheduler CPU - Core 1 CPU - Core 2 Kernel -Thread 1 CPU Scheduler

9 Work (Input) Resources Web Server -Thread 1 Web Server -Thread 2 Redis -Thread 1 CPU Scheduler CPU - Core 1 CPU - Core 2 Kernel -Thread 1 CPU Scheduler

10 Type Work Resources CPU Scheduler Threads Physical Cores AWS EC2 / OpenStack Nova Virtual Machines Hypervisors Hadoop YARN MapReduce Jobs Client Nodes Cluster Scheduler Applications Servers Schedulers In the Wild

11 Higher Resource Utilization Decouple Work from Resources Better Quality of Service Advantages

12 Higher Resource Utilization Bin Packing Decouple Work from Resources Over-Subscription Better Quality of Service Job Queueing Advantages

13 Higher Resource Utilization Abstraction Decouple Work from Resources API Contracts Better Quality of Service Standardization Advantages

14 Higher Resource Utilization Priorities Decouple Work from Resources Resource Isolation Better Quality of Service Pre-emption Advantages

15

16 Nomad

17 Cluster Scheduler Easily Deploy Applications Job Specification Nomad

18 # Define our simple redis job job "redis" { # Run only in us-east-1 datacenters = ["us-east-1"] example.nomad # Define the single redis task using Docker task "redis" { driver = "docker" config { image = "redis:latest" } } } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } }

19 Declares what to run Job Specification

20 Nomad determines where and manages how to run Job Specification

21 Abstract work from resources Job Specification

22 Higher Resource Utilization Decouple Work from Resources Better Quality of Service Nomad

23 Designing Nomad

24 Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin Packing Large Scale Nomad Operationally Simple

25 Thousands of regions Tens of thousands of clients per region Thousands of jobs per region

26 gossip consensus Built on Experience

27 Cluster Management Gossip Based (P2P) Membership Failure Detection Event System

28 Gossip Protocol Large Scale Production Hardened Operationally Simple

29 Service Discovery Configuration Coordination (Locking) Central Servers + Distributed Clients

30 Multi-Datacenter Raft Consensus Large Scale Production Hardened

31 Mature Libraries Design Patterns No Scheduling Logic gossip consensus

32 gossip consensus Built on Research

33

34 Optimistic vs Pessimistic Internal vs External State Single vs Multi Level Fixed vs Pluggable Service vs Batch Oriented

35 Inspired by Google Omega Optimistic Concurrency Internal State and Coordination Service and Batch workloads Nomad Pluggable Architecture

36

37 Multi-Datacenter Servers Per DC Failure Isolation Domain is the Datacenter

38 CLIENT CLIENT CLIENT DC1 DC2 DC3 RPC RPC RPC SERVER REPLICATION SERVER REPLICATION SERVER FORWARDING FORWARDING FOLLOWER LEADER FOLLOWER Single Region Architecture

39 REGION A SERVER SERVER SERVER REPLICATION REPLICATION FOLLOWER LEADER FORWARDING FOLLOWER REGION B GOSSIP REGION FORWARDING SERVER FOLLOWER REPLICATION SERVER LEADER REPLICATION FORWARDING SERVER FOLLOWER Multi Region Architecture

40 Region is Isolation Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary Nomad

41 Data Model

42 Evaluations ~= State Change Event

43 Create / Update / Delete Job Node Up / Node Down Allocation Failed

44 Scheduler = func(eval) => []AllocUpdates

45 Scheduler func s can specialize (Service, Batch, System, etc)

46 Evaluation Enqueue

47 Evaluation Dequeue

48 Plan Generation

49 Plan Execution

50 External Event Evalua?on Crea?on Evalua?on Queuing Evalua?on Processing Op?mis?c Coordina?on State Updates

51 Omega Class Scheduler Pluggable Logic Internal Coordination and State Multi-Region / Multi-Datacenter Server Architecture

52 Broad OS Support Host Fingerprinting Pluggable Drivers Client Architecture

53 Type Examples Operating System Kernel, OS, Versions Hardware CPU, Memory, Disk Applications Java, Docker, Consul Environment AWS, GCE Fingerprinting

54 Constrain Placement and Bin Pack Fingerprinting

55 Task Requires Linux, Docker, and PCI-Compliant Hardware expressed as Constraints Fingerprinting

56 Task needs 512MB RAM and 1 Core expressed as Resource Ask Fingerprinting

57 Execute Tasks Provide Resource Isolation Drivers

58 Docker Containerized Rocket Qemu / KVM Virtualized Java Jar Standalone Static Binaries

59 Docker Containerized Rocket Windows Server Containers Qemu / KVM Virtualized Xen Hyper-V Java Jar Standalone Static Binaries C#

60 Workload Flexibility: Schedulers Fingerprints Drivers Job Specification Nomad

61 Operational Simplicity: Single Binary No Dependencies Highly Available Nomad

62

63 Released in October Service and Batch Scheduler Docker, Qemu, Exec, Java Drivers Nomad 0.1

64 Case Study

65 3 servers in NYC3 100 clients in NYC3, SFO1, AMS2/ Containers Case Study

66 <1s to schedule 1s to first start 6s to 95% 8s to 99% Case Study

67 Service Discovery System Scheduler Restart Policies Enhanced Constraints Nomad Service Workloads

68 Cron Job Queuing Latency-Aware Scheduling Nomad Batch Workloads

69 Nomad 0.2 in Prod Stress Testing Atlas Integration Production Hardening

70 Cluster Scheduler Easily Deploy Applications Job Specification Nomad

71 Higher Resource Utilization Decouple Work from Resources Better Quality of Service Nomad

72 Thanks! Q/A