Elastic Efficient Execution of Varied Containers. Sharma Podila Nov 7th 2016, QCon San Francisco

Size: px

Start display at page:

Download "Elastic Efficient Execution of Varied Containers. Sharma Podila Nov 7th 2016, QCon San Francisco"

Morgan Gregory
5 years ago
Views:

1 Elastic Efficient Execution of Varied Containers Sharma Podila Nov 7th 2016, QCon San Francisco

2 In other words... How do we efficiently run heterogeneous workloads on an elastic pool of heterogeneous resources, with capacity guarantees?

3 Topics Containers, Mesos, Fenzo - where are we today? Modeling an elastic Mesos cluster Capacity guarantees for varied applications Network resource and security groups Ongoing and future work

4 About Me Software engineer Resource scheduling, stream processing, distributed systems Netflix Edge Engineering Sun Microsystems + Oracle Corp. Author of Fenzo scheduling library

5 81 Million subscribers worldwide and growing! Source:

6 Microservices architecture on AWS EC2

7 Containers, Apache Mesos, Fenzo where are we today?

8 Reactive stream processing: Mantis Zuul Cluster Mantis API Cluster Stream processing Cloud native service Configurable message delivery guarantees Heterogeneous workloads Real-time dashboarding, alerting Anomaly detection, metric generation Interactive exploration of streaming data Anomaly Detection

9 Current Mantis usage Peak of 1,800 EC2 instances M3.2xlarge instances Peak of 3,700 concurrent containers Trough of 2,700 containers Mix of perpetual and interactive exploratory jobs Peak of 11 Million events / sec

10 Titus Job Control Container deployment: Titus App Cloud Platform (metrics, IPC, health) Batch Containers Containers VM VM VM VM VPC EC2 Atlas & Insight Eureka Edda

11 Current Titus usage Peak of ~1,800 instances Mix of m4.4xl, r3.8xl, g2.8xl ~800 instances at trough Mix of batch, stream processing, and some microservices #Containers (tasks) for the week of 10/24 in one of the regions

12 Core architectural components Titus/Mantis Framework Fenzo Apache Mesos AWS EC2 Fenzo at Apache Mesos at

13 Jobs, tasks, instances, containers Jobs can be one of batch, service, or stream processing type of jobs A jobs has one or more tasks to run An instance is equivalent to a task A task runs one container

14 A few common themes Heterogeneous mix of jobs and resources Resource Task request Agent sizes CPU 1-32 CPUs 8-32 CPUs Memory GB GB Network bandwidth Mbps Resource affinity based on task type Task locality

15 A few common themes Large variation in peak to trough resource requirements 11M Mantis events/sec 2M Titus concurrent containers 1000s 10s

16 Modeling an elastic Mesos cluster Can we resize agent cluster based on demand?

17 Task assignments in a cluster Consider a cluster with 4-slot hosts

18 Random assignments in a cluster An EC2 instance with 4 slots Used slot Idle slot Cluster starts random assignments of resources to tasks

19 Random assignments in a cluster Cluster starts to fill up...

20 Random assignments in a cluster About 50% utilized Cluster somewhat full. But, only 1 agent can be terminated for scale down without losing jobs

21 Random assignments in a cluster 100% utilized Cluster is now full

22 Random assignments in a cluster About 65% utilized Cluster partially used as jobs finish...

23 Random assignments in a cluster About 25% utilized Cluster partially used, but, can t terminate any instance without losing jobs

24 Ideal assignments in a cluster Similarly, 25% utilized Cluster utilized to the same level as previous, but, can now terminate 9 of the 12 instances!

25 Ideal assignments in a cluster Cluster scaled down easily due to bin packing

26 EC2 ASG attributes for setting number of servers in cluster EC2 AutoScalingGroups have three attributes to set Min - minimum number of instances to have Max - maximum number of instances Desired - current number of instances to have Fenzo sets the Desired count based on demand

27 EC2 AutoScalingGroup for Mesos agents Min Desired Max

28 EC2 AutoScalingGroup for Mesos agents Min Desired Max

29 EC2 AutoScalingGroup for Mesos agents Min Desired Max

30 Using multiple instance types

31 Using multiple instance types Amazon EC2 provides a variety of servers a.k.a instance types Algorithm model training jobs run well on memory optimized instances of R3 type Typical services run well on balanced compute instances of M4 type

32 Using multiple instance types How do we use multiple EC2 instance types in the same Mesos agent cluster?

33 Using multiple EC2 instance types Grouping agents by instance type let s us autoscale them independently Titus m4.4xlarge agent ASG r3.8xlarge agent ASG

34 Using multiple EC2 instance types User job: 2 CPUs, 5GB memory User job: 8 CPUs, 8GB memory User job: 1 CPUs, 1GB memory Titus m4.4xlarge agent ASG r3.8xlarge agent ASG

35 Continuous deployment of agents

36 Continuous deployment of agents A new version of agent introduces a new ASG m4.4xlarge agent ASG v1

37 Continuous deployment of agents A new version of agent introduces a new ASG m4.4xlarge agent ASG v1 m4.4xlarge agent ASG v2

38 Continuous deployment of agents A new version of agent introduces a new ASG m4.4xlarge agent ASG v1 Disable m4.4xlarge agent ASG v2

39 Continuous deployment of agents A new version of agent introduces a new ASG m4.4xlarge agent ASG v1 m4.4xlarge agent ASG v2 Migrate tasks Disable

40 Continuous deployment of agents A new version of agent introduces a new ASG m4.4xlarge agent ASG v1 Disable m4.4xlarge agent ASG v2

41 Continuous deployment of agents A new version of agent introduces a new ASG m4.4xlarge agent ASG v2 Old agent ASG removed

42 Bringing it all together... Titus m4.4xlarge agent ASG r3.8xlarge agent ASG v2 v1 v2 v1

43 Capacity guarantees for varied applications

44 The capacity guarantee challenge Demand for resources > Supply

45 An execution sample from a cluster Running #tasks New batch of tasks Tasks launched

46 An execution sample from a cluster Running #tasks New batch of tasks Tasks launched Waiting for agents to free up Or, for new agents from scale up

47 An execution sample from a cluster Running #tasks New batch of tasks Tasks launched Scale up and freed agents satisfy all new pending tasks

48 An execution sample from a cluster Running #tasks New batch of tasks Tasks launched What if a service was launched at this time? Waiting for agents to free up Or, new agents from scale up

49 Capacity guarantees on up d e Agre Guarantee^capacity for timely job starts Mesos support for quotas, etc. evolving

50 Capacity guarantees on up d e Agre Guarantee^capacity for timely job starts Mesos support for quotas, etc. evolving Generally, optimize throughput for batch jobs and start latency for service jobs

51 Capacity guarantees Some service style jobs may be less important Categorize by expected behavior instead

52 Capacity guarantees Some service style jobs may be less important Categorize by expected behavior instead Critical versus Flex (flexible) scheduling requirements

53 Capacity guarantees Flex Critical Quotas

54 Capacity guarantees Flex Critical Critical Flex Quotas vs. Priorities Resource Allocation Order

55 Capacity guarantees: hybrid view AppFN AppF3 AppF2 AppF1 Flex AppCN AppC3 AppC2 AppC1 Critical Resource Allocation Order

56 Capacity guarantees via Fenzo Fenzo supports multi-tiered task queues Tier 0 Multiple buckets per tier with fair sharing by dominant resource usage Tier 1

57 Translating application capacity to EC2 instances Define per application capacity guarantees Define per tier capacity guarantees Translate to number of EC2 instances

58 Defining application capacity App1-cap = num_app_instances * app_instance_dimensions app_instance_dimensions: { #cpus, memory, disk, network} Agnostic to EC2 instance types

59 Defining application capacity Applications specify resource needs, not EC2 instance types Can manage capacity guarantees using a variety of instance types Eases migration to new instance types, thereby helps capacity procurement teams

60 Defining Tier capacity Tier Capacity = SUM (App1-cap + App2-cap + + AppN-cap) + BUFFER BUFFER: Accommodate some new or ad hoc jobs with no guarantees Red-black pushes of services temporarily double capacity

61 Translate to number of instances #EC2_instances = Tier_capacity / EC2_instance_dimensions A tier may use multiple instance types Critical = { m4.4xlarge, m3.2xlarge } Flex = { r3.8xlarge, g2.8xlarge }

62 Network resource and security groups

63 Container executor T E I-T T UL M N NA + Augment missing pieces: IP per container Security - Security Groups, IAM roles Isolation for networking b/w, disk I/O <

64 Elastic Network Interfaces (ENI) Each EC2 instance in VPC has 2 or more ENIs Each ENI can have 2 or more IPs Security Groups are set on the ENI AWS EC2 Instance ENI0 ENI1 ENI2 ENI0 IP0 IP4 IP8 IP0 IP1 IP5 IP9 IP1 IP2 IP6 IP10 IP2 IP3 IP7 IP11 IP3

65 ENI+IP resource allocation model A two level resource modeled in Fenzo Each agent reports #ENIs and #IPs per ENI via custom attribute Fenzo does allocation and usage tracking ENI 1 Assigned Security Group: SG1 Used IPs Count: 2 of 7 ENI 2 Assigned Security Group: SG1,SG2 Used IPs Count: 1 of 7 ENI 3 Assigned Security Group: SG3 Used IPs Count: 7 of 7

66 Plumbing VPC Networking into Docker Task 0 Task 1 Task 2 Task 3 No IP, SecGrp A SecGrp X SecGrp X SecGrp Y,Z app app app app veth<id> pod root pod root pod root veth<id> veth<id> veth<id> * * * Titus EC2 Metadata Proxy IPTables NAT (*) * Linux Policy Based Routing + Traffic Control eth1 eth2 eth3 ENI1 ENI2 ENI3 SecGrp=A Non-routable IP SecGrp=X SecGrp=Y,Z IP 1 IP 2 IP 3 Titus EC2 Host VM

67 Network bandwidth isolation Each container gets an IP on one of the ENIs Linux tc policies used on virtual Ethernet For both incoming and outgoing traffic Bandwidth limited to the requested value No borrowing of unused bandwidth Easy to reason about

68 Ongoing and future work

69 Current and future work Fine grain capacity guarantees Hierarchical sharing policies Preemptions to satisfy priority tiers and sharing policies Execution environment security hardening Onboarding new applications Looking forward to working with the community

70 In Summary...

71 In summary... Mesos and Fenzo help us run lots of containers In an elastic fashion With guaranteed capacity for varied applications Custom AWS integration gives us network resource isolation and security groups

72 Questions? Elastic Efficient Execution of Varied Containers Sharma netflix. com linkedin. com / in / spodila

Cloud Monitoring as a Service. Built On Machine Learning

Cloud Monitoring as a Service. Built On Machine Learning Cloud Monitoring as a Service Built On Machine Learning Table of Contents 1 2 3 4 5 6 7 8 9 10 Why Machine Learning Who Cares Four Dimensions to Cloud Monitoring Data Aggregation Anomaly Detection Algorithms