More Containers, More Problems

Transcription

1 More Containers, More Problems Ed coreos.com

2 Agenda Define problems Define vision of the solution How CoreOS is building solutions How you can get started

3 It all started with... a server

4 Then we got... many servers

5 Then we got... VMs on our servers

6 Then we got... APIs around hosted VMs (cloud)

7 Which led to... even more servers

8 Too Many Servers! The cloud made booting servers really easy. Also Moore s law is still a thing.

9 More Servers, More Problems Patching..is hard Dependency management...is hard Managing access...is hard Managing workloads...is hard App Lifecycle management....is hard Identifying security issues...is hard

10 More Servers == More Sysadmins 1000 Servers Sysadmins 500 0

11 More Servers, More Problems 1000 Servers Sysadmins 500 0

12 Google needed more servers before the rest of us did. They solved many of these problems internally, and published some great papers.

13 We started building it CoreOS, Google, and the community... are building the open-source version.

14 #GIFEE

15 What is #GIFEE? Google s Infrastructure For Everyone Else

16 Google s Infrastructure "Fundamentally, it's what happens when you ask a software engineer to design an operations function." --Ben Treynor Sloss Vice President, Google Engineering founder of Google SRE

17

18 What is #GIFEE? Servers are not your pets Servers are the new CPU Cores Clusters are the new servers

19 Evolution of Servers

20 Clusters Server Cluster

21 Clusters Process App

22 Operating System Custom Linux Distributed Consensus Chubby Cluster Manager Borg Monitoring BorgMon RPC framework Stubby Auth private

23 Open Source Operating System Custom Linux CoreOS Linux Distributed Consensus Chubby etcd Cluster Manager Borg Kubernetes Monitoring BorgMon Prometheus RPC framework Stubby grpc Auth private Dex

24 cluster operating system

25 OS for Clusters Orchestration State Scheduler: Gets work to the servers

26 What is #GIFEE? Software manages servers Software manages workloads Declare what you want, it will become so

27 API + scheduler worker kubelet worker kubelet worker kubelet worker kubelet worker kubelet worker kubelet worker kubelet

28 API + scheduler worker kubelet

29 API + Scheduler + worker works on 1 node too

30 Primary component of the Cluster OS Fits our vision Started by Google with over 10 yrs experience running Borg

31 What is #GIFEE? Centralized administration & orchestration No more SSH Yes, that even means your favorite config mgmt tool

32 What is #GIFEE? Don t say HOW $ scp myapp host:/opt $ ssh host systemd-run /opt/myapp

33 What is #GIFEE? say WHAT $ kubectl run myapp --image=quay.io/sym3tri/hello --replicas=1 $ kubectl get pods POD IP myapp-97wt

34 What is #GIFEE? say WHAT again $ kubectl scale rc myapp --replicas=4 $ kubectl get POD myapp-97wt8 myapp-f839d myapp-98b35 myapp-e40ee pods IP

35 What is #GIFEE? say WHAT one more time $ kubectl run myapp --image=quay.io/sym3tri/hello --replicas=1 $ kubectl get pods POD IP myapp-97wt

36

37 RC web-prod Pod select(env=prod,app=web) count=1 env=prod app=web

38 RC web-prod select(env=prod,app=web) count=4 Pod Pod Pod Pod env=prod env=prod app=web env=prod app=web env=prod app=web app=web

39 automated!= automatic

40 What is #GIFEE? Dependencies are isolated per app Apps automatically migrate throughout the cluster

41 What is #GIFEE? All apps are 12-factor Configuration/Secret management prod config staging config

42 What is #GIFEE? Consistent Deployment API Deploy canary builds and experiments Rolling Updates

43 app v1 app v1 Load Balanced Service app v1 app v1

44 app v1 app v1 Load Balanced Service app v1 app v1 app v2

45 app v1 app v1 Load Balanced Service app v1 app v1 app v2

46 app v1 app v1 Load Balanced Service app v1 app v1 app v2

47 app v1 app v1 Load Balanced Service app v1 app v2 app v2

48 app v1 app v1 Load Balanced Service app v2 app v2 app v2

49 app v2 Load Balanced Service app v2 app v2 app v2

50 What is #GIFEE? Mixed workloads (staging + prod) A Team Logically partitioned resources B Team C Team

51 What is #GIFEE? Trusted & Secure from the bottom up* Cluster OS Container Runtime Only trusted code is executed OS Firmware & TPM

52 What is #GIFEE? Every {human,machine,process} is authenticated & authorized API + scheduler All communication is encrypted worker kubelet

53 What is #GIFEE? Failure is expected and handled for - Services / Apps Machines Storage Clusters Regions

54 What is #GIFEE? Logging Monitoring / Alerting

55 #GIFEE vs Google Infra? Compatibility with existing tools Work with other projects (Docker, Calico, Prometheus) Incorporates lessons learned

56 Why? Build for scale Manage your apps, not servers High Availability New paradigm of infra/development

57 #GIFEE and Security We believe: As #GIFEE becomes ubiquitous, the Internet becomes more secure overall

58 CoreOS Mission Secure the Internet

59 Journey to #GIFEE

60 Getting Started Leverage prior work + standards - Raft - Omaha Protocol - OIDC

61 Securing The Internet Start from the bottom The Operating System

62 Securing The Internet Minimal Server OS + Automatic Updates Requires: - Distributed consensus - Containers - Cluster computing

63 Containerize In this new world we containerize all the things

64 Containerize but

65 More Containers, More Problems Every solution breeds new problems -Arthur Bloch １つの問題解決 別の問題発生

66 More Containers, More Problems Problem #1 - Secure & controlled container distribution

67 More Containers, More Problems Problem #1 - Secure & controlled container distribution Solution

68 More Containers, More Problems Problem #2 - Docker security model - Docker coupling of components

69 More Containers, More Problems Problem #2 - Docker security model - Docker coupling of components Solution

70 More Containers, More Problems systemd systemd docker run redis docker engine daemon app app

71 Side Note: Spec vs Implementation Implementation:

72 Side Note: Spec vs Implementation Specification:

73 More Containers, More Problems Problem #3 - User Authentication

74 More Containers, More Problems Problem #3 - User Authentication Solution - Dex

75 More Containers, More Problems Problem #4 - Really big containers

76 More Containers, More Problems Problem #4 - Really big containers Solution - Go - Buildroot - acbuild for ACIs

77 Your container is 500MB!? NOOOOOOOOO!!! github.com/brianredbeard/minimal_containers

78 More Containers, More Problems Problems # Co-locating Containers Intelligent Scheduling Port Management Segmenting workloads Configuration Management Secrets Management Inconsistent Deployments

79 More Containers, More Problems Problems # Co-locating Containers Intelligent Scheduling Port Management Segmenting workloads Configuration Management Secrets Management Inconsistent Deployments Solution

80 More Containers, More Problems Problem #12 Networking - Too many types of SDNs - IP per POD

81 More Containers, More Problems Problem #12 Networking - Too many types of SDNs - IP per POD Solution - CNI

82 More Containers, More Problems Problem #13 - Metrics - Monitoring - Alerting

83 More Containers, More Problems Problem #13 - Metrics - Monitoring - Alerting Solution - Prometheus

84 More Containers, More Problems Problem #14 - Vulnerabilities inside containers

85 More Containers, More Problems Problem #14 - Vulnerabilities inside containers Solution

86

87 More Containers, More Problems Problem #15 - Visualize & configure clusters

88 More Containers, More Problems Problem #15 - Visualize & configure clusters Solution - Tectonic Console

89

90 More Containers, More Problems Problem #16 - Running on Bare Metal

91 More Containers, More Problems Problem #16 - Running on Bare Metal Solution - Ignition - coreos-baremetal - Tectonic baremetal installer

92 More Containers, More Problems Problem #17 - Inability to verify node trust

93 More Containers, More Problems Problem #17 - Inability to verify node trust Solution - Distributed Trusted Computing (DTC)

94 More Containers, More Problems Problem #18 - Persistent storage

95 More Containers, More Problems Problem #18 - Persistent storage Solution - Torus

96 Kubernetes is the kernel, Tectonic is the distro.

97 off-the-shelf #GIFEE

98 Kubernetes Contributions OIDC Authentication 2x Scheduler Performance RBAC Authorization etcd 3 support TLS Bootstrapping coreos-kubernetes rktnetes Bootstrap/Upgrade Simplification

99 Future More Management Tools Prometheus Enhancements Expand platform support Federated Clusters

100 Summary Open-Source is key Containers Security is key Orchestration Updates are key Automatic systems

101 More Containers, More Problems Ed coreos.com

102 CoreOS is Running the World s Containers OPEN SOURCE ENTERPRISE 90+ Projects on GitHub, 1,000+ Contributors Secure solutions, support plans, training + more CoreOS.com - github/coreos sales@coreos.com - tectonic.com - quay.io We re hiring in all departments! careers@coreos.com Positions: coreos.com/ careers