Wrapp. Powered by AWS EC2 Container Service. Jude D Souza Solutions Wrapp Phone:

Transcription

1 Wrapp Powered by AWS EC2 Container Service Jude D Souza Solutions Wrapp Phone: jude@wrapp.com

2 About Me Jude D Souza Stockholm, Sweden ß Karachi, Pakistan jude@wrapp.com MS Distributed Systems KTH Royal Institute of Technology, Stockholm, Sweden Solutions Wrapp Passion for building large scale distributed systems

3 About Wrapp Wrapp delivers personal offers and rewards based on where you shop, dine and the brands you follow. Microservice Architecture Infrastructure 13 ECS Clusters 50+ Instances 90+ Services Technology Stack Operating System - Ubuntu Consul, Registrator, AWS(EC2, ECS, Lambda), Docker, HAProxy Monitoring Sensu, Riemann Logs - Logentries and Sumo Languages - Go, Python, Java

4 Agenda Architecture before ECS Service Orchestration Service Discovery Service Deployments Service Availability & Scalability Container Registry Transition towards ECS Microservice Wrapp Why ECS? Next Challenges QA

5 How we did it Before ECS (1/3) ü Service Orchestration Service Deployments Service Availability & Scalability Container Registry Service Orchestration Via runlist configs On instance boot: Determine autoscaling group Pull runlist for autoscaling group Run containers specified in config Add these to supervisor PULL RUNLIST CONFIGS Sample Runlist configuration file misc Autoscaling group ops Autoscaling group api Autoscaling group misc ops api

6 How we did it Before ECS (2/3) ü Service Orchestration ü Service Deployments Service Availability & Scalability Container Registry Service Deployments Find instances where service is running (via serf) SSH into these instances Do rolling deploy (Terminate container, Launch container with new version) misc Autoscaling group ops Autoscaling group api Autoscaling group misc ops api Sample serf data for services - misc-i-abc rewards/http/ misc-i-def rewards/http/ misc-i-ghi rewards/http/ SERF CLUSTER

7 How we did it Before ECS (3/3) ü Service Orchestration ü Service Deployments ü Service Availability & Scalability ü Container Registry Service Availability Supervisord misc Autoscaling group ops Autoscaling group api Autoscaling group Service Scalability Service itself can't scale Scale only by scaling autoscaling group misc ops api misc SERF CLUSTER Container Registry Hosting our own container registry Backed by s3

8 Transition to ECS Service Orchestration Runlists Supervisor Service Deployments Supervisor Service Availability Supervisor Service Scalability EC2 Autoscaling Container Registry Self-hosting Service Discovery Serf HAProxy Service Orchestration Service Deployments Service Availability Service Scalability (with lambda) AWS Container Registry + Service stats (cpu + memory) + Service monitoring + Service Provisioning Service Discovery Consul Registrator HAProxy

9 Wrapp Microservice Architecture api.wrapp.com DNS Elastic Load Balancer PUBLIC SUBNET api api api PRIVATE SUBNET Microservices ZONE A ZONE B ZONE C

10 Opsweb Tooling around controlling our Infrastructure Cluster Overview

11 Opsweb Tooling around controlling our Infrastructure Services Overview

12 Opsweb Tooling around controlling our Infrastructure Instances Overview

13 Service Scalability (Concept from Amazon) Trigger Lambda function from SNS Topic CloudWatch Alarm SNS Topic AWS Lambda +1 DesiredCount in Service Trigger CloudWatch alarm from ECS Metrics (CPU/ Memory) misc Autoscaling group EC2 Container Service misc ECS Instances Scaling Amazon ECS Services Automatically Using Amazon CloudWatch and AWS Lambda:

14 Service (CPU/ Memory) Provisioning ü ü Memory allocation by Service Allocate CPU Units by Service ü Efficient Resource utilization (Container placement) eventbus notifications api rewards s auth Typical ECS Instance

15 Container Logging Logspout Containers launched via ECS Rsyslog LogEntries logspout Log router for Docker rsyslog Forwards logs to logentries Typical ECS Instance Logentries: Logspout:

16 Why ECS? Our Experience ü Supports built-in service discovery, loadbalancing and routing Generally good service (ECS is ok) Great support ü Fleet - Container Orchestration We don t need to host these services ourselves on AWS Always improving to make their services better and better ü Docker Swarm: Supports orchestrating containers ü Weave Flux: Supports built-in service discovery, loadbalancing and routing

17 Service Wrapp (1/2) Consul server-a HA Consul Cluster Service catalog Key/value store Consul server-b Consul server-c Consul Agent Consul Agent Consul Agent Registrator Registrator Registrator api auth users users

18 Service Wrapp (2/2) HA Consul Cluster Consul server-b Consul server-a Consul server-c >> curl users/v1/users/1 { "name": "jude" "gender": "male" } Consul Template HAProxy frontend users-in bind users:80 DNS Hack mode http ( x.x) default_backend users Consul Agent backend users server Server-a :8080 server Server-c :8080 #### Registrator auth users frontend auth-in bind auth:80 mode http default_backend auth backend auth server Server-a :8081 server Server-c :8081

19 Next Challenges (1/2) Short comings perceived Built-in (like kube-proxy, weave-proxy)? Service Discovery Service Routing Service Load balancing SINGLE ELB FOR MULTIPLE SERVICES APPROACH Elastic Load Balancer Approach 1: ELB Per service? Costly services, 70+ ELBs Approach 2: Single ELB for all services? ü ü Service Routing works Service Load balancing works No health check per service No way to return HTTP 503 (but possible via haproxy http check option) ECS Boto3: create_service() Route53: ELB.dns = elb-services Benefits: Removes tools like Consul and Registrator frontend users-in bind users:80 mode http default_backend users backend users server users elb-services:8080 #### >> curl users/ping

20 Next Challenges (2/2) Short comings perceived Ø Cron scheduling Setup ecs tasks to run on cronschedule Ø ECS Event stream Actionables on certain events (when deploy completes) Ø ECS Container Logging Ø Optionally Integrate ECS Cluster with EC2 Autoscaling Group

21 Summary The Wrapp Architecture before ECS Transition to ECS Service Wrapp with Consul, Registrator and HAProxy Our perceived shortcomings from ECS Built-in Service Discovery Cron scheduling on Tasks Event stream

22 Questions? Contact: Jude D Souza jude@wrapp.com Phone: Twitter: Wrapp

23