Pursuit of stability. Growing AWS ECS in production. Alexander Köhler Frankfurt, September 2018

Transcription

1 Pursuit of stability Growing AWS ECS in production Alexander Köhler Frankfurt, September 2018

2 Alexander Köhler DevOps Engineer Systems Engineer Big Data Engineer Application Developer

3 inovex is an IT project house with a focus on digital transformation Product Discovery Product Ownership Web UI/UX Replatforming Microservices Mobile Apps Smart Devices Robotics Big Data & Business Intelligence Platforms Data Science Data Products Search Deep Learning Data Center Automation DevOps Cloud Hosting Agile Training Technology Training Coaching Karlsruhe Pforzheim Stuttgart München Köln Hamburg Using technology to inspire our clients. And ourselves.

4 The start What the customer told us to do. host web applications that our SCRUM team develops 4

5 The start What the customer didn t told us right on the spot. different languages (e.g. java, go) Different maintainers Cloud based host web applications that our SCRUM team develops SSO solution Multistage setup Need for an android build server Distribute APKs 5

6 The start What we told the customer we could do. Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. 6

7 The start What we told the customer we could do. different languages (e.g. java, go) Different maintainers Cloud based Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. SSO solution Multistage setup Android Build Server Distribute APKs 7

8 The start What we told the customer we could do. Wrap the apps in containers Different maintainers Cloud based Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. SSO solution Multistage setup Android Build Server Distribute APKs 8

9 The start What we told the customer we could do. Wrap the apps in containers Add CI/CD pipelines Cloud based Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. SSO solution Multistage setup Android Build Server Distribute APKs 9

10 The start What we told the customer we could do. Wrap the apps in containers Add CI/CD pipelines AWS Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. SSO solution Multistage setup Android Build Server Distribute APKs 10

11 The start What we told the customer we could do. Wrap the apps in containers Add CI/CD pipelines AWS Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. Keycloak Multistage setup Android Build Server Distribute APKs 11

12 The start What we told the customer we could do. Wrap the apps in containers Add CI/CD pipelines AWS Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. 12 Keycloak Infrastructure-as- Code to deploy in multiple AWS accounts Android Build Server Distribute APKs

13 The start What we told the customer we could do. Wrap the apps in containers Add CI/CD pipelines AWS Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. Keycloak 13 Infrastructure-as- Code to deploy in multiple AWS accounts Ec2 instance with planned builds Distribute APKs

14 The start What we told the customer we could do. Wrap the apps in containers Add CI/CD pipelines AWS Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. Keycloak 14 Infrastructure-as- Code to deploy in multiple AWS accounts Ec2 instance with planned builds Provide S3 buckets for upload/download

15 15 Round 1

16 Round 1 Containers in production. Back in 2017: No EKS. But Kops with AWS support. 16

17 Round 1 Containers in production. Back in 2017: No AWS Fargate. (as long as i remember) But still Kops with AWS support. 17

18 Round 1 Let s bring containers in production, they said. AWS ECS to the rescue. Easy setup, I ve heard. 18

19 Round 1 Let s bring containers in production, they said. 19

20 Round 1 Let s bring containers in production, they said. ECS itself is pretty easy, but there are some more components involved. 20

21 Round 1 How exactly bring container now into production? ECS Service ECS Service... ECS Cluster 21

22 Round 1 How exactly bring container now into production? task. (id 42) task. (id 41) ECS Service ECS Service... ECS Cluster 22

23 Round 1 How exactly bring container now into production? ECS Service ECS Service... ECS Cluster VPC EC2 node EC2 node ECS Ami

24 Round 1 How exactly bring container now into production? ECS Service ECS Service... Docker Registry ECS Cluster VPC EC2 node EC2 node ECS Ami

25 Round 1 How exactly bring container now into production? DNS Load balancer Certs ECS Service ECS Service... Docker Registry ECS Cluster VPC EC2 node EC2 node ECS Ami

26 26 Round 2

27 Round 2 So, replace ECS Ami What we knew good: Ubuntu LTS. Should be easy. 27

28 Round 2 Move away from ECS AMI DNS Load balancer Certs ECS Service ECS Service... Docker Registry ECS Cluster VPC EC2 node EC2 node ECS Ami Ubuntu AMI

29 Round 2 Move away from ECS AMI ECS AMI Ubuntu Docker ECS agent 29

30 Round 2 Move away from ECS AMI ECS AMI Ubuntu Docker Docker ECS agent ECS agent 30

31 Round 2 Move away from ECS AMI ECS AMI Ubuntu EBS Volume Docker Docker ECS agent ECS agent Docker-Root at EBS 31

32 32 Round 3

33 Round 3 Bring up a monitoring solution It s AWS. There is cloudwatch? Or do prometheus like everyone else? 33

34 Round 3 Bring up a monitoring solution We did prometheus. And ended up with something like this. 34

35 Round 3 Let prometheus collect all the metrics from all the sources. Monitoring stack Container ECS cluster EC2 instances AWS Api 35

36 Round 3 Let prometheus collect all the metrics from all the sources. Monitoring stack Container Grafana Alertmanager ECS cluster EC2 instances AWS Api Blackbox Exporter ECS Discovery Prometheus EC2 discovery 36

37 Round 3 Let prometheus collect all the metrics from all the sources. Monitoring stack Container Grafana Alertmanager ECS cluster CAdvisor EC2 instances Node Exporter Docker Daemon Metrics AWS Api Blackbox Exporter ECS Discovery Prometheus EC2 discovery 37 ContainerSt arts

38 Round 4 Read all the shells. Ever tried to debug Keycloak in a container not starting properly on a distributed infrastructure? We did. We ended up somehow like this. 38

39 Round 4 Read all the shells. Nay. Lets collect the logs. 39

40 Round 4 Setup a log pipeline. Container ECS cluster Elasticsearch Service on AWS Docker EC2 instances 40

41 Round 4 Setup a log pipeline. Container ECS cluster filebeat processor add_docker_metadata Elasticsearch Service on AWS EC2 instances Docker json files 41

42 Round 5 Add this magic thing called CI/CD. 42

43 Round 5 The pipeline build artifact (e.g. dist/) Actually, pretty easy. AWS ECR Some different kind (docker of registry) easy. gitlab registry (docker registry) ECS cluster (running service) gitlab-runner gitlab-runner gitlab-runner gitlab-runner make docker build docker push terraform apply 43 gitlab repository build artifact gitlab registry gitlab repository

44 Round 5 Toolstack ECS Service Gitlab ECS Cluster EC2 44

45 Round 5 Toolstack ECS Service Gitlab Terraform Gitlab Runner ECS Cluster EC2 45

46 Round 5 Toolstack ECS Service ECS Service Terraform Gitlab Gitlab Runner ECS Cluster Gitlab Runner ECS Cluster EC2 EC2 Terraform (awsinfrastructure) 46

47 Round 5 The pipeline itself docker login ecr registry docker push Build App Wrap in Docker Push to ECR- Dev 1.Pull from gitlab 2.Push to ECR Add to Gitlab Docker Registry Push to next- Stage Deploy ECS services Gitlab-CI 47

48 48 Final round.

49 Conclusion What I ve talked about Wrap the apps in containers Add CI/CD pipelines AWS Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. Keycloak 49 Infrastructure-as- Code to deploy in multiple AWS accounts Ec2 instance with planned builds Provide S3 buckets for upload/download

50 Conclusion What we build but wasn t mentioned. Wrap the apps in containers Add CI/CD pipelines AWS Build a modern (cloud native) infrastructure, wrap everything in ci/cd, build management components like monitoring, add some magic sugar on top and host your web applications in AWS. Keycloak 50 Infrastructure-as- Code to deploy in multiple AWS accounts Ec2 instance with planned builds Provide S3 buckets for upload/download

51 Final words Bring container in production. ECS itself: easy and straightforward. The complete stack with supporting tools: can be tricky. 51

52 Final words Proven tool chain Provision with terraform. Gitlab-CI as CI/CD base. Gitlab-Runner as workhorse. 52

53 Final words What we might do in another way the second time. Take a clearer decision about the usage of additional AWS service. 53

54 Final words What we might do in another way the second time. Re-check if fargate isn t the better solution for most of the services 54

55 Vielen Dank Alexander Köhler DevOps Engineer inovex GmbH Ludwig-Erhard-Allee Karlsruhe alexander.koehler@inovex.de