Armon HASHICORP

Size: px

Start display at page:

Download "Armon HASHICORP"

Aubrey Casey
5 years ago
Views:

1 Nomad

2 Armon

5 Cluster Manager Scheduler Nomad

6 Cluster Manager Scheduler Nomad

7 Schedulers map a set of work to a set of resources

8 Work (Input) Resources Web Server -Thread 1 Web Server -Thread 2 Redis -Thread 1 CPU Scheduler CPU - Core 1 CPU - Core 2 Kernel -Thread 1 CPU Scheduler

9 Work (Input) Resources Web Server -Thread 1 Web Server -Thread 2 Redis -Thread 1 CPU Scheduler CPU - Core 1 CPU - Core 2 Kernel -Thread 1 CPU Scheduler

10 Type Work Resources CPU Scheduler Threads Physical Cores AWS EC2 / OpenStack Nova Virtual Machines Hypervisors Hadoop YARN MapReduce Jobs Client Nodes Cluster Scheduler Applications Servers Schedulers In the Wild

11 Higher Resource Utilization Decouple Work from Resources Better Quality of Service Advantages

12 Higher Resource Utilization Bin Packing Decouple Work from Resources Over-Subscription Better Quality of Service Job Queueing Advantages

13 Higher Resource Utilization Abstraction Decouple Work from Resources API Contracts Better Quality of Service Standardization Advantages

14 Higher Resource Utilization Priorities Decouple Work from Resources Resource Isolation Better Quality of Service Pre-emption Advantages

16 Nomad

17 Cluster Scheduler Easily Deploy Applications Operationally Simple Built for Scale Nomad

18 example.nomad job "redis" { datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB } } } network { mbits = 10 dynamic_ports = ["redis"] }

19 Declares what to run Job Specification

20 Nomad determines where and manages how to run Job Specification

21 Nomad abstracts work from resources Job Specification

22 Docker Containerized Rkt Windows Server Containers Qemu / KVM Virtualized Xen Hyper-V Java Jar Standalone Static Binaries C#

23 Declarative Job Specification Infrastructure-As-Code Removes Imperative Logic External Dependencies? Nomad

24 Service Discovery? Health Monitoring? Application Secrets? Stateful Applications? Nomad

25 job my-app" { task my-app" { service { port = http check { type = http path = /health interval = 5s } } } } example.nomad

26 Client Nomad Server Schedule App Nomad App 1 Consul Register Service Monitor Health Consul Server App N

27 Secret Distribution: API Keys DB Credentials SSL/TLS Certificates Nomad

28 job my-app" { task my-app" { env { DB_USERPASS = foo:bar } } } example.nomad

29 Secure secret storage Dynamic secrets Leasing, renewal, and revocation Auditing Rich ACLs Vault Multiple client authentication methods

30 Login Vault Token Vault Token + Operation Op Response

31 example.nomad job my-app" { task my-app" { env { VAULT_TOKEN = b6a10b e6-9c6f-67a52bc6b8d3 } } }

32 job my-app" { task my-app" { vault { policies = [ my-app-role ] } } } example.nomad

33 Submit Job + Vault Token Nomad Server Verify Vault Token Schedule App Client Nomad Generate + Renew Vault Token App 1 App N

34 Native Vault Integration No Secrets in Jobs No Secrets on Client Disk Minimize Trust Nomad

35 Stateless Stateful Stateful Applications

36 Stateless Stateful API Web Cache Stateful Applications

37 Stateless Stateful API Web Cache HDFS Cassandra MongoDB Stateful Applications

38 Stateless Stateful API Web Cache HDFS Cassandra MongoDB *SQL Stateful Applications

39 EASY MEDIUM HARD Stateless Stateful API Web Cache HDFS Cassandra MongoDB *SQL Stateful Applications

40 job my-app" { task my-app" { ephemeral_disk { sticky = true } } } example.nomad

41 Moves data between tasks on the same machine

42 Copies data between tasks on different machines

43 Easily Deploy Apps: Declarative Jobs Flexible Workloads Consul Integration Vault Integration Nomad Sticky Volumes

44 Operationally Simple

45 Client Server

46 Built on Experience GOSSIP CONSENSUS

47 Cluster Management Gossip Based (P2P) Membership Failure Detection Serf Event System

48 Large Scale Production Hardened Simple Clustering and Federation Serf

49 Service Discovery Configuration Coordination (Locking) Central Servers + Distributed Clients Consul

50 Multi-Datacenter Raft Consensus Large Scale Production Hardened Consul

51 Operational Simplicity: Single Binary No Dependencies Highly Available Nomad

52 Built for Scale

53 Built on Experience Mature Libraries Proven Design Patterns GOSSIP CONSENSUS Lacking Scheduling Logic

54 Built on Research GOSSIP CONSENSUS

56 CLIENT CLIENT CLIENT DC1 DC2 DC3 RPC RPC RPC SERVER REPLICATION SERVER REPLICATION SERVER FORWARDING FORWARDING FOLLOWER LEADER FOLLOWER Single Region Architecture

57 REGION A SERVER SERVER SERVER REPLICATION REPLICATION FOLLOWER LEADER FORWARDING FOLLOWER REGION B GOSSIP REGION FORWARDING SERVER FOLLOWER REPLICATION SERVER LEADER REPLICATION FORWARDING SERVER FOLLOWER Multi Region Architecture

58 Region is Isolation Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary Nomad

59 Hundreds of regions Tens of thousands of clients per region Thousands of jobs per region

60 Inspired by Google Omega Optimistic Concurrency State Coordination Service & Batch workloads Nomad Pluggable Architecture

61 Data Model NODE EVALUATION ALLOCATION JOB

62 Evaluation ~= State Change

63 Evaluations Create / Update / Delete Job Node Up / Node Down Allocation Failed / Finished

64 Evaluations SCHEDULER func(evaluation) => []AllocationUpdates

65 Evaluations SCHEDULER func(evaluation) => []AllocationUpdates Service, Batch, System

66 External Event EvaluaBon CreaBon EvaluaBon Queuing EvaluaBon Processing OpBmisBc CoordinaBon State Updates

67 Omega Architecture Optimistically Schedule 100 s of Jobs in Parallel Controls for Correctness Nomad

68 Nomad Million Container Challenge 1,000 Jobs 1,000 Tasks per Job 5,000 Hosts on GCE 1,000,000 Containers

70 640 KB ought to be enough for anybody. Bill Gates

71 2nd Largest Hedge Fund 18K Cores 5 Hours 2,200 Containers/second

72 Cluster Scheduler Easily Deploy Applications Operationally Simple Built for Scale Nomad

73 Thanks! Q/A

Armon HASHICORP

Armon HASHICORP Nomad Armon Dadgar @armon Distributed Optimistically Concurrent Scheduler Nomad Distributed Optimistically Concurrent Scheduler Nomad Schedulers map a set of work to a set of resources Work (Input) Resources