Two years of on Kubernetes

Size: px

Start display at page:

Download "Two years of on Kubernetes"

Tiffany Clarke
5 years ago
Views:

1 Two years of on Kubernetes

2 Platform rebuy Once a Fullstack- and Game-Developer Got interested in container technologies in 2014 and jumped on K8s in 2015 Finished my master thesis with a case study on migrating infrastructure to K8s on AWS in 2017 %>

3 recommerce market leader in Europe Germany, Austria, France, UK and the Netherlands ~500 employees Growing and expanding We re hiring! (tech.rebuy.com) %> whoarewe rebuy.de

4 agenda Our two year journey Stack Architecture Lifecycle Struggles What we learned

5 Our two year journey

6 Why migrate? Static number of machines at Serverhousing company TV-Spot/Ads = Downtime or permanently adding new machines Manual Provisioning Scalability Reliability Automation Infrastructure as Code Observability Cattle and Pets Immutability...

7 Timeline November 2016 May 2016 Researching Fiddling around Prototyping Decided for a stack First production workloads on Kubernetes Ongoing migration efforts May 2017 Got off legacy infrastructure Final migration step took one night

8 Timeline Today ~600 containers running in production 27 m5.2xlarge worker instances deployments per working day Ongoing projects: Open Tracing Cluster Auto Scaling Service Mesh Staging Platform for Application Development

9 Stack

10 Terraform Infrastructure as Code SCM-able Descriptive (define desired state, not changes) future-proof

11 Amazon Web Services Infrastructure as a Service API Maturity Features

12 CoreOS KISS etcd Systemd Docker cloud-config/ignition Immutable filesystem

13 Kubernetes Application Lifecycle Bin-Packing Auto-Scaling...

14 Docker Container runtime

15 Architecture

16 3x master worker stateful worker stateless ignition ignition ignition ignition ASG ASG ASG

17 github.com/rebuy-de/ terraform-aws-coreos-k8s-single-node (

18 OLD DATACENTER DB Master K8s on AWS DB Replica replicate VPN Messaging Caches federation Messaging Caches

19 Lifecycles

20 Accounts Production-Cluster DNS-Root dev3 ECR Backup dev2 staging dev1 jenkins-pr integration-test jenkins-master

21 TESTING LIFECYCLE SPIN UP dostuff(); TEAR DOWN

22 SPIN UP TRIGGERS Pull request Master commit Periodic check-jobs Periodic integration tests Manually

23 SPIN UP TOOLING Scripts to provision AWS and Kubernetes Change configurations for non-production clusters through Terraform.tfvars files kubernetes-deployment to use go templating on.yaml files github.com/rebuy-de/

24 github.com/rebuy-de/ TEAR DOWN aws-nuke TOOLING (and Jenkins)

25 PRODUCTION LIFECYCLE START NEW ASG MIGRATE WORKLOADS REMOVE OLD ASG

26 MIGRATIONS... OLD ASG... NEW ASG

27 TOIL ALARM! Manually cordon and drain old nodes Remove that exact node in ASG Human Error Boring Takes time * nodecount

28 THE SOLUTION! In-cluster operator Talks to AWS Talks to Kubernetes Less manual labour Can work together with Cluster-Autoscaler Profit! github.com/rebuy-de/ node-drainer

29 HOW DOES IT WORK? ASG Scale-in lifecycle hook Amazon Simple Queue Service (SQS) scale-in message not so fast, mate! go ahead and terminate remove message Node-drainer Pod start draining done draining Kubernetes API t

30 terminated MIGRATIONS terminated... OLD ASG... NEW ASG

31 Struggles

32 ELB + Masters Every once in a while about ⅓ of our workers in different AZs went NotReady because heartbeat didn t reach the Masters Things exploded Happened again a few weeks later out of nowhere and again and again...

33 ELB + Masters

34 ELB + Masters apiserver Master 2 apiserver Master 1 apiserver Elastic Load Balancer Master 2 Worker 1 kubelet Worker 2 kubelet...

35 ELB + Masters

36 ELB + Masters

37 ELB + Masters

38 ELB + Masters

39 ELB + Masters apiserver controller-manager scheduler Network Load Balancer Master 1 Master 2 apiserver controller-manager scheduler

40 ELB + Masters apiserver controller-manager scheduler Network Load Balancer Master 1 Master 2 apiserver controller-manager scheduler

41 Missing routes in flannel Some Pods reported not being able to resolve hostnames (via kube-dns) After some time we noticed missing routes, and flannel complaining Tweaking sysctl didn t help

42 Missing routes in flannel Initial taint DaemonSet (instance-health-checker) checks if routes are properly set up Removes taint if so Otherwise trigger alert via Prometheus endpoint Run flannel as a DaemonSet instead of Systemd

43 What we learned

44 Costs Track your costs: github.com/rebuy-de/cost-exporter Use Reservations and Spot Instances where you can

45 Terraform + K8s Turn on versioning of S3 Terraform.tfstate files and use DynamoDB for locking Apply Terraform changes in small batches Use the same CI/CD strategies on your infrastructure code as on all other code Use testing clusters to experiment with risky changes Separate Pets and Cattle into separate ASGs Running self managed Kubernetes is possible and gives you More insight into how things work More flexibility Surprises :)

46 Workflow Don t be afraid to break stuff (in testing clusters) Know what context you are working in Automate everything Hang your dashboards up on a wall

47 tech.rebuy.com

The Long Road from Capistrano to Kubernetes

The Long Road from Capistrano to Kubernetes Tobias Schwab, Co-Founder of PhraseApp Slides: http://bit.ly/cap-to-kube How to deploy Ruby on Rails? Deploying Ruby on Rails required on all servers: OS + system