Cluster management at Google john wilkes / Principal Software Engineer

Transcription

1 Cluster management at Google john wilkes / johnwilkes@google.com Principal Software Engineer

2 For the past 15 years, Google has been building out the world s fastest, most powerful, highest quality cloud infrastructure on the planet. Images by Connie Zhou

3 Hello World job hello_world = { runtime = { cell = 'ic' } // What cluster should we run in? binary = '.../hello_world_webserver' // What program are we to run? args = { port = '%port%' } // Command line parameters requirements = { // Resource requirements ram = 100M disk = 100M cpu = 0.1 } replicas = // Number of tasks }

4 Hello World > borgcfg.../hello_world_webserver.borg up... About to affect tasks and 1 packages on cell IC. Do you wish to continue (yes/no) [no]? yes ==== Staging package hello_world_webserver.63ce1b965155c75e/johnwilkes on ic... SUCCESS ==== Making package hello_world_webserver.63ce1b965155c75e/johnwilkes on ic... SUCCESS ==== Starting job hello_world on ic... SUCCESS

5 Hello World

6 Binary Hello World Config file webbrowsers browsers web borgcfg What just happened? Cell BorgMaster BorgMaster UIshard shard BorgMaster UI BorgMaster UI shard read/ui BorgMaster UI shard shard persistent store (Paxos) Scheduler scheduler linkshard shard link link shard linkshard shard link Borglet Borglet Borglet Borglet

7 Hello World Images by Connie Zhou

8 Hello World

9 Failures task-eviction rates and causes 9

10 A 2000-machine service will have >10 machine crashes per day DRAM errors (1% AFR) Disk failures (2-10% AFR) crashes (~2/year) OS upgrades (2-6/year) Images by Connie Zhou

11 A 2000-machine service will have >10 machine crashes per day This is normal; not a problem DRAM errors (1% AFR) Disk failures (2-10% AFR) crashes (~2/year) OS upgrades (2-6/year) Images by Connie Zhou

12 Efficiency Advanced binpacking algorithms Experimental placement of production VM workload, July 2014

13 Efficiency Advanced binpacking algorithms nice round numbers There are no obvious bucket sizes (cf. cloud VMs) gaming the system 13

14 Efficiency CDF Advanced binpacking algorithms Batch jobs Service jobs Heterogeneous workloads, May 2011 Omega paper, EuroSys 2013 Job runtime [log]

15 Efficiency Utilization: sharing clusters between prod/batch helps 15

16 Efficiency Utilization: sharing clusters between prod/batch helps 16

17 Efficiency Advanced binpacking algorithms Data from a cluster with 12k machines, May 2011 Trace is publicly available Heterogeneity and dynamicity of clouds at scale: Google trace analysis. SoCC 12

18 Efficiency Resource reclamation could be more aggressive Nov/Dec

19 Efficiency Multiple applications per machine tasks/machine CPI^2 paper, EuroSys 2013 threads/machine

20 Efficiency Multiple applications per machine CPI^2 paper, EuroSys Gather CPI for all the tasks in a job Find outliers Take action μ μ+σ μ + 2σ μ + 3σ task CPI outliers => victims

21 Achieving desired behavior Exposing mechanisms is fragile Better: declarative intents

22 Achieving desired behavior Service level objective (SLO) Examples: availability obtainability reliability velocity freshness? accuracy? security? an SLO

23 A few other moving parts Config file webbrowsers browsers web borgcfg Cell UI BorgMaster UI BorgMaster UI BorgMaster UI shard BorgMaster read/ui shard BorgMaster shard shard shard persistent store (Paxos) Scheduler scheduler linkshard shard link link shard linkshard shard link Borglet Borglet Borglet Borglet

24 A few other moving parts job config master agent app

25 A few other moving parts storage job config master agent app

26 A few other moving parts storage job config master agent app

27 A few other moving parts system config storage job config master agent app

28 A few other moving parts system config storage job config master agent app monitoring

29 A few other moving parts system config storage job config master agent app monitoring binaries + data distribution

30 A few other moving parts system config security storage job config master agent app monitoring binaries + data distribution

31 A few other moving parts system config security accounting/planning storage job config master agent app monitoring binaries + data distribution Diagram from an original by Cody Smith.

32 A few other moving parts system config security accounting/billing storage job config master agent app monitoring binaries + data distribution Diagram from an original by Cody Smith.

33 s Everything at Google runs in a container -- including our VMs s give us: resource isolation execution isolation CPU QoS We start over 2 billion containers per week. Image: "" glynlowe CC-BY-2.0

34 Kubernetes κυβερνήτης: Greek for pilot or helmsman of a ship The open source cluster manager from Google.

35 Kubernetes Web server Log roller

36 Pods Web server Kubernetes master/scheduler Log roller

37 Labels FE FE FE FE FE Kubernetes master/scheduler

38 Label selectors FE FE labels: role: frontend FE FE FE Kubernetes master/scheduler

39 Label selectors FE FE labels: role: frontend stage: production FE FE FE Kubernetes master/scheduler

40 Replica controller FE FE replicas: 3 template:... labels: role: frontend FE Kubernetes - Master/Scheduler

41 Replica controller FE FE FE replicas: 4 template:... labels: role: frontend FE Kubernetes - Master/Scheduler

42 Service id: frontend-service port: 9000 labels: role: frontend frontend-service FE FE FE FE Kubernetes - Master/Scheduler

43 Kubernetes The open source cluster manager from Google. Pods: groups of containers Labels Replica controller Services

44 resources Pulling it all together Do it yourself? Sure. offered load

45

46

47 Pulling it all together We choose to go to the roof not because it is glamorous, but because it is right there!... the bulk of our success is the result of the methodical, relentless, persistent pursuit of 1.32x opportunities -- what I have come to call "roofshots". -- Luiz Barroso

48 Pulling it all together Porsche doesn't make cars: it designs and assembles them 1H2014: 1.7% (89k) of VW group's vehicles 23% ( 1.4b) of its profits Data: Volkswagen, Image: john wilkes

49 Pulling it all together Cloud system providers are getting better at everything... capacity management monitoring storage + networking reliability software development tooling... Wouldn't you like to stand on others' shoulders?

50 Three rules of thumb: 1. Resiliency is more important than performance. 2. Relax. Let go. Build on what others have done. 3. Do more monitoring. Images by Connie Zhou

51

52 Achieving desired behavior Service level agreement (SLA) SLA = SLOs + consequences of achieving or missing them Example: if availability > 99.95% (SLO) user pays xx/cpu-week else gets a 30% refund an SLO

53 For the past 15 years, Google has been building out the world s fastest, most powerful, highest quality cloud infrastructure on the planet. Images by Connie Zhou