Developing Kubernetes Services

Size: px

Start display at page:

Download "Developing Kubernetes Services"

Iris Meghan Allison
5 years ago
Views:

1 / MARCH 2019 / CON LONDON Developing Kubernetes Services at Airbnb Scale

2 What is kubernetes?

3 @MELAN IECEBULA Who am I?

4 A BRIEF HISTORY

5 Why Microservices? MONOLITH LOC

6 Why Microservices? ENGINEERING TEAM

7 Why Microservices? SCALING CONTINUOUS DELIVERY

8 Why Microservices? Deploys per week (all apps, all environments)

9 Why Microservices? Monolith production deploys per week

10 Why Microservices? 125,000 production deploys per year

11 Why kubernetes? EVOLUTION OF CONFIGURATION MANAGEMENT Manually configuring boxes Automate configuration of applications with Chef Automate configuration and orchestration of containerized applications with Kubernetes

12 Why kubernetes? declarative portable human-friendly data efficient scheduling immutable standard format extensible API reproducible

13 TODAY

14 migration progress 50% of services in kubernetes

15 migration progress 250+ critical services in kubernetes

16 Challenges with kubernetes? complex configuration complex tooling integrating with your current infrastructure open issues scaling and more!

17 Challenges with kubernetes? complex configuration complex tooling integrating with your current infrastructure open issues solvable problems! scaling and more!

18 you are not alone.

19 Solutions? 1. abstract away k8s configuration 2. generate service boilerplate 3. version + refactor configuration 4. opinionated kubectl 5. custom ci/cd + validation

20 ABSTRACT AWAY CONFIGURATION

21 kubernetes config files Production Deployment Canary Deployment Dev Deployment kubernetes kubectl apply Production ConfigMap Canary ConfigMap Dev ConfigMap kubernetes cluster Production Service Canary Service Dev Service P

22 lots of boilerplate kubernetes config files repetitive by environment Production Deployment Canary Deployment Dev Deployment kubectl apply kubernetes resources Production ConfigMap Canary ConfigMap Dev ConfigMap kubernetes cluster Production Service Canary Service Dev Service P environments

23 Prefer templating over file inheritance Reducing k8s boilerplate OUR REQUIREMENTS Input should be templated YAML files Make it easier to migrate 100s of legacy services Make it easier to retrain 1000 engineers

24 generating k8s configs kubernetes config files Project Apps Production Deployment Canary Deployment Dev Deployment Containers kube-gen generate kubectl apply Files Production ConfigMap Canary ConfigMap Dev ConfigMap kubernetes cluster Volumes Production Service Canary Service Dev Service Dockerfile

25 generating k8s configs kube-gen! kubernetes config files Project Apps Production Deployment Canary Deployment Dev Deployment Containers kube-gen generate kubectl apply Files Production ConfigMap Canary ConfigMap Dev ConfigMap kubernetes cluster Volumes Production Service Canary Service Dev Service Dockerfile

26 sets params per environment Project Apps Reducing k8s boilerplate WHAT WE WENT WITH Containers Files other files access params Volumes Dockerfile

27 generating k8s configs Project Apps Containers kube-gen generate Files Volumes Dockerfile

28 generating k8s configs Project Apps Containers kube-gen generate standardized namespaces based on environments! Files Volumes Dockerfile

29 Project Apps Which shared components to use? kube-gen COMPONENTS Containers Files example components Volumes nginx logging statsd Dockerfile

30 nginx component Main Container Containers common pattern abstracted into component kube-gen COMPONENTS Main App Volumes component yams is merged into project on generate components can require or set default params Files Dockerfile

31 Reducing k8s boilerplate OPEN SOURCE OPTIONS 1. Combine with package management (ex: helm) 2. Override configuration via file inheritance (ex: kustomize) 3. Override configuration via templating (ex: kapitan)

32 Takeaways Reduce kubernetes boilerplate Standardize on environments and namespaces

33 GENERATE SERVICE BOILERPLATE

34 Everything about a service is in one place in git, and managed with one process.

35 Everything about a service is in one place in git Configuration LIVES IN ONE PLACE All configuration lives in _infra alongside project code Edit code and configuration with one pull request Easy to add new configuration Statically validated in CI/CD

36 What we support: kube-gen files framework boilerplate API boilerplate Configuration LIVES IN ONE PLACE CI/CD docs AWS IAM roles project ownership storage.. and more!

37 this hello world service was created in one command Configuration LIVES IN ONE PLACE

38 collection of config generators (ex: docs, ci) Configuration LIVES IN ONE PLACE

39 Configuration LIVES IN ONE PLACE collection of framework-specific generators (ex: Rails, Dropwizard)

40 make best practices the default (ex: deploy pipeline, autoscaling, docs) Configuration CAN BE GENERATED run generators individually or as a group support for review, update, commit

41 Takeaways Everything about a service should be in one place in git Make best practices the default by generating configuration

42 VERSION CONFIGURATION

43 Why do we version our kube configuration? add support for something new (ex: k8s version) kube.yml v1 kubev1 deploy kubev2 deploy want to change something (ex: deployment strategy) want to drop support for kubernetes cluster something (breaking kube.yml v2 change) know which versions are bad when we make a regression support release cycle and cadence

44 How do we version our kube configuration? 1. version field 2. publish binaries for each version 3. channels point to binaries (ex: stable) 4. generate and apply using the appropriate binary bonk kube-gen.yml

45 Why do we version our generated configuration? kube.yml generated by sha1 kubev1 deploy what our project generators generate changes over kubernetes cluster time kube.yml generated by sha2 best practices change and bugs in the generators are found! generator at sha2 has a bug

46 How do we version our generated configuration? generator tags generated files with version, sha, and timestamp

47 REFACTOR CONFIGURATION

48 Why do we refactor configuration? FOR HUNDREDS OF SERVICES services should be up-to-date with latest best practices update configuration to the latest supported versions apply security patches to images configuration migrations should be automated

49 (we don t want to manually refactor) 250+ critical services in kubernetes

50 How do we refactor configuration? get-repos.py close.py collection of general purpose scripts list-pr-urls.py refactor.py scripts are modular scripts cover the lifecycle of a refactor updateprs.py status.py refactorator

51 The lifecycle of a refactor Run Refactor Update Merge Checks out repo, finds project, runs refactor job, tags owners, creates PR Comments on the PR, reminding owners to verify, edit, and merge the PR Merges the PR with different levels of force

52 How do we refactor configuration? refactorator upgradekube.py refactor job refactorator will run a refactor for all services given a refactor job refactor job updates _infra file(s) ex: upgrade kube version to stable

53 Bumping stable version bump stable version runs daily on weekdays refactorator upgradekube.py cron job uses refactorator and with the upgradekube.py refactor job to create PRs k8s cron job refactor job another cron job handling updating and merging PRs

54 Takeaways Configuration should be versioned and refactored automatically.

55 OPINIONATED KUBECTL

56 lots of boilerplate kubernetes config files repetitive by environment Production Deployment Canary Deployment Dev Deployment kubectl apply kubernetes resources Production ConfigMap Canary ConfigMap Dev ConfigMap kubernetes cluster Production Service Canary Service Dev Service P environments

57 kubernetes config files Production Deployment Canary Deployment Dev Deployment kubectl apply kubernetes Production ConfigMap Canary ConfigMap Dev ConfigMap kubernetes cluster verbose Production Service Canary Service Dev Service repetitive by namespace P

58 @MELAN IECEBULA k tool KUBECTL WRAPPER Production Deployment Canary Deployment Dev Deployment kubectl apply Production ConfigMap Production Service Canary ConfigMap Canary Service Dev ConfigMap Dev Service kubernetes cluster calls kubectl commands (incl. plugins)

59 @MELAN IECEBULA k tool OPINIO NATED KUBECTL

60 Runs in the project home directory: $ cd /path/to/bonk $ k status k tool USES ENV VARS Environment variables for arguments: $ k status ENV=staging Prints the command that it will execute: $ k status ENV=staging standardized namespaces! kubectl get pods --namespace=bonk-staging

61 k generate generates kubernetes files k build performs project build, docker build and k tool SIMPLIFIES BUILDS AND DEPLOYS docker push with tags k deploy creates namespace, applies/replaces kubernetes files, sleeps and checks deployment status can chain commands; ex: k all

62 defaults to random pod, main container: k tool A DEBUGGING TOOL $ k ssh ENV=staging specify particular pod, specific container: $ k logs ENV=staging POD= CONTAINER=bonk automates debugging with k diagnose ENV=staging

63 defaults to random pod, main container: k tool A DEBUGGING TOOL $ k ssh ENV=staging specify particular pod, specific container: call kubectl diagnose $ k logs ENV=staging POD= CONTAINER=bonk automates debugging with k diagnose ENV=staging

64 What are kubectl plugins?

65 What are kubectl plugins?

66 k diagnose SETUP deploy bonk service with failing command new pod in CrashLoopBackoff

67 k diagnose MANUALLY 1. use get pods - o=yaml and look for problems 2. grab logs for unready container

68 k diagnose MANUALLY 3. get k8s events related to this pod

69 kubectl podevents KUBECTL PLUGIN kubectl podevents plugin

70 kubectl diagnose USES COBRA GO CLI // defines CLI command and flags var Namespace string var rootcmd = &cobra.command{ Use: kubectl diagnose namespace<namespace>" Short: diagnoses a namespace with pods in CrashLoopBackOff Run: func(cmd *cobra.command, arg[]string) { // Fill in with program logic } } func Execute() { rootcmd.flags().stringvarp(&namespace, "namespace", "n", ) rootcmd.markflagrequired("namespace") if err := rootcmd.execute(); err!= nil { fmt.println(err) os.exit(1) }

71 kubectl diagnose USES K8S CLIENT-GO // get pods (assume Namespace is defined) kubeconfig := filepath.join(os.getenv("home"), ".kube","config") config, err := clientcmd.buildconfigfromflags("", kubeconfig) if err!= nil { } clientset, err := kubernetes.newforconfig(config) if err!= nil { } pods, err := clientset.corev1().pods(namespace).list(metav1.listoptions{}) fmt.printf("there are %d pods in the namespace %s\n", len(pods.items), Namespace) for _, pod := range pod.items { podname := pod.name for _, c := range pod.status.containerstatuses { if c.ready!= true { // print c.lastterminatedstate and c.state } } uses k8s client-go and Namespace param to get pods

72 kubectl diagnose USES K8S CLIENT-GO // get pods (assume Namespace is defined) kubeconfig := filepath.join(os.getenv("home"), ".kube","config") config, err := clientcmd.buildconfigfromflags("", kubeconfig) if err!= nil { } clientset, err := kubernetes.newforconfig(config) if err!= nil { } pods, err := clientset.corev1().pods(namespace).list(metav1.listoptions{}) fmt.printf("there are %d pods in the namespace %s\n", len(pods.items), Namespace) for _, pod := range pod.items { podname := pod.name for _, c := range pod.status.containerstatuses { if c.ready!= true { // print c.lastterminatedstate and c.state prints info for all unready containers } }

73 // get pod events for namespace and pod kubectl diagnose USES OS/EXEC (WHEN LAZY) cmd := exec.command("kubectl", "podevents", Namespace, podname) var out bytes.buffer var stderr bytes.buffer cmd.stdout = &out podevents kubectl plugin cmd.stderr = &stderr err := cmd.run() if err!= nil { } fmt.println(fmt.sprint(err) + ": " + stderr.string()) log.fatal(err) } else { } fmt.println("events: \n" + out.string()) // also grab logs cmd = exec.command("kubectl", "logs", podname, fmt.sprintf("-- namespace=%s", Namespace), "-c", "bonk")

74 kubectl diagnose GO KUBECTL PLUGIN

75 kubectl diagnose GO KUBECTL PLUGIN 1. unready container info

76 kubectl diagnose GO KUBECTL PLUGIN 1. unready container info 2. kubectl podevents

77 kubectl diagnose GO KUBECTL PLUGIN 1. unready container info 2. kubectl podevents 3. pod logs for unready containers

78 Takeaways Create an opinionated kubectl wrapper Automate common k8s workflows with kubectl plugins

79 CI/CD

81 Each step in our CI /CD jobs are RUN steps in a build Dockerfile

82 runs k commands

83 DEPLOY PROCESS

84 A single deploy process for every change Develop Merge Deploy Write code and config under your project Open a PR and merge your code to master Deploy all code and config changes

85 A single deploy process for every change Deployment AWS Project ownership Storage kubectl apply ConfigMap Alerts Docs Service Discovery kubernetes cluster Service Dashboards Secrets API Gateway Routes

86 How do we apply k8s configuration? Deployment kubectl apply all files kubectl apply in some cases where apply fails, replace files without ConfigMap kubernetes cluster force always restart pods on deploy to pick up changes Service return atomic success or failure state by sleeping and checking status

87 How do you always restart pods on deploy? We add a date label to the pod spec, Deployment which convinces k8s to relaunch all pods kubectl apply ConfigMap kubernetes cluster Service

88 How do we apply custom configuration?

89 How do we apply custom configuration? kubectl apply aws.yml kubernetes cluster AWS AWS AWS CRD Controller webhook

90 How do we apply custom configuration? kubectl apply 1. Create a custom resource definition aws.yml kubernetes cluster for aws.yml AWS AWS AWS CRD Controller webhook

91 How do we apply custom configuration? aws.yml kubectl apply kubernetes cluster 2. Create a controller that calls a web hook when aws.yml is applied AWS AWS AWS CRD Controller webhook

92 How do we apply custom configuration? kubectl apply 3. Create a web hook that updates a aws.yml kubernetes cluster custom resource AWS AWS AWS CRD Controller webhook

93 How do we apply custom configuration? 4. AWS lambda exposes web hook to be called AWS CRD AWS Controller AWS webhook AWS lambda

94 Takeaways Code and configuration should be deployed with the same process Use custom resources and custom controllers to integrate k8s with your infra

95 VALIDATION

96 enforce best practices at build time with validation scripts Configuration at deploy time with admission controller SHOULD BE VALIDATED

97 How do we validate configuration at build time?

98 How do we validate configuration at build time? project.yml validation script kube validation script job dispatcher project build global validation docs build jobs aws.yml validation script bonk CI jobs global jobs repo

99 How do we validate configuration at build time? project.yml validation script kube validation script job dispatcher project build global validation docs build jobs aws.yml validation script 1. Define global job in global jobs repo bonk CI jobs global jobs repo

100 How do we validate configuration at build time? project.yml validation script kube validation script job dispatcher project build global validation docs build jobs 2. job dispatcher aws.yml validation script always dispatches global jobs to projects bonk CI jobs global jobs repo

101 How do we validate configuration at build time? project.yml validation script kube validation script job dispatcher project build global validation docs build jobs aws.yml validation script 3. global job runs alongside project bonk CI jobs global jobs repo jobs

102 What do we validate at build time? project.yml validation script invalid yaml invalid k8s configuration kube validation script job dispatcher bad configuration versions max namespace length (63 chars) valid project name aws.yml validation script valid team owner in project.yml global jobs repo

103 How do we validate configuration at deploy time? kubectl apply admission controller project.yml kubernetes cluster admission controller intercepts requests to the k8s api server prior to persistence of the object

104 How do we validate configuration at deploy time? metadata is encoded as admission controller annotations at generate time project.yml admission controller checks for required annotations reject any update to resources that are missing required annotations

105 What do we validate with admission controller? admission controller project ownership annotations project.yml configuration stored in git configuration uses minimally supported version

106 What do we validate with admission controller? production images must be admission controller uploaded to production ECR project.yml prevent deployment of unsafe workloads prevent deployment of development namespaces to production clusters

107 What do we validate with admission controller? production images must be admission controller uploaded to production ECR project.yml prevent deployment of unsafe workloads prevent deployment of development namespaces to production clusters standardized namespaces!

108 Takeaways CI/CD should run the same commands that engineers run locally CI/CD should run in a container Validate configuration as part of CI/ CD

109 10 Takeaways 1. Abstract away complex kubernetes configuration 2. Standardize on environments and namespaces 3. Everything about a service should be in one place in git 4. Make best practices the default by generating configuration 5. Configuration should be versioned and refactored automatically. 6. Create an opinionated kubectl wrapper that automates common workflows 7. CI/CD should run the same commands that engineers run locally, in a container 8. Code and configuration should be deployed with the same process 9. Use custom resources and custom controllers to integrate with your infrastructure 10. Validate configuration as part of CI/CD

110 2019 Challenges thousands of services running in k8s moving all configuration to gitops workflow w/ custom controllers scaling the cluster / scaling etcd / multi cluster support stateful services / high memory requirements tighter integration with kubectl plugins and more! keno

111

OpenShift Roadmap Enterprise Kubernetes for Developers. Clayton Coleman, Architect, OpenShift

OpenShift Roadmap Enterprise Kubernetes for Developers Clayton Coleman, Architect, OpenShift What Is OpenShift? Application-centric Platform INFRASTRUCTURE APPLICATIONS Use containers for efficiency Hide