SOFT CONTAINER TOWARDS 100% RESOURCE UTILIZATION ACCELA ZHAO, LAYNE PENG

Transcription

1 SOFT CONTAINER TOWARDS 100% RESOURCE UTILIZATION ACCELA ZHAO, LAYNE PENG 1

2 WHO ARE THOSE GUYS Accela Zhao, Technologist at EMC OCTO, active Openstack community contributor, experienced in cloud scheduling and container technologies. Mail: Layne Peng, Principal Technologist at EMC OCTO, experienced cloud architect, one of the earliest contributors to Cloud Foundry in China, 9 patents owner and a book author. Mail: layne.peng@emc.com 2

3 WHAT IS RESOURCE UTILIZATION? This is what we buy A gap of $$$ wasted This is what we use 3

4 ENERGY AND RESOURCE UTILIZATION Real world resource utilization is usually low: around 20% or less An idle server consumes even 70% as much energy as running in fullspeed Energy-related costs 42% of total (including buy new machines) Low resource utilization is energy inefficient Waste energy, waste money 4

5 A CLOSER LOOK TO CLOUD The key advantage - cloud consolidation Improved resource utilization Less machines, more apps. Energyefficient and saves money. 5

6 RESOURCE UTILIZATION ON CLOUD Scheduling - choose the best resource placement when app starts Examples: Green Cloud, Paragon. And the schedulers in Openstack, Kubernetes, Mesos, Migration - continuously optimize the resource placement when app is running Examples: Openstack Watcher, VMware DRS Soft Container - elastic, and dynamically adjust resource constraints in response to co-located apps Related: Google Heracles 6

7 RESOURCE UTILIZATION ON CLOUD Apps Scheduler Manages resource utilization at app kick-off Soft Container Manages resource utilization at fine granularity inside host Migration Manages resource utilization cross hosts while app running 7

8 RESOURCE UTILIZATION ON CLOUD A battle of putting more apps in each host vs. guaranteed app SLA The key problem: resource interference 8

9 THE KEY PROBLEM: RESOURCE INTERFERENCE What is resource interference? Apps co-located in one host share resources like CPU, cache, memory, They interfere with each other, result in poor performance compared to running standalone Resource interference make SLA unenforceable Related readings Google Heracles: an analysis of resource interference Paragon: resource interference-aware scheduling Bubble-up: to measure resource interference 9

10 RESOURCE INTERFERENCE: HOW IT LOOKS? MySQL standalone running vs co-located with a CPU & disk hungry task 10

11 RESOURCE INTERFERENCE: HOW TO MEASURE? Bubble-up The setup Run app co-located with resource benchmarks, each benchmark stresses one type of resource App tolerated resource interference Slowly increase resource benchmark stress until app fails its SLA. The critical point shows how much resource interference the app can tolerate. App caused resource interference Run app at what its SLA requires. The stress it causes on each type of resource is the app s caused resource interference. Where to use it? Better resource utilization management Scheduling, Migration, Soft Container, 11

12 RESOURCE INTERFERENCE: HOW TO MEASURE? MySQL standalone running, vs co-located with CPU stress, vs disk stress. In my case, MySQL is much more sensitive to CPU interference. 12

13 INTRODUCING TO SOFT CONTAINER Motivations Increase resource utilization by co-locating more apps E.g. Business services is critical but may not use all resources on the host. Add the low priority hadoop batching tasks to fill what is left. Respond to the dynamic nature of time-varying workload E.g. Business service may become more idle at lunch time, hadoop tasks can then expand its resource bubble and utilize the leftover. Guarantee the SLA of critical apps E.g. When the business service suddenly requires more resource for processing, hadoop tasks will shrink instantly to give out resources. Challenges Resource control and isolation of interference Respond to dynamic workload change 13

14 INTRODUCING TO SOFT CONTAINER What does Soft mean? Varying container resources needs based upon neighbors and SLAs. (The container becomes elastic) Expanding (bubble up) resources when idle resources exist Shrinking resources on a specific container, when another critical app demands more resources Resource Container resource bubble Time 14

15 THE FEEDBACK CONTROL LOOP Controller Soft Container Watcher Limiter Containers 15

16 RESOURCES TO LIMIT CPU Core Time Quota Disk I/O IOPS Throughput Memory Size Bandwidth 16

17 RESOURCES TO LIMIT - MISSING CPU Core Time Quota Disk I/O IOPS Throughput Memory Size Bandwidth* Cache LLC Network Ulimit Bandwidth GPU Device* Kernel 3.6, most supports can be found in the community 17

18 ISOLATION THE RESOURCES - NAMESPACE clone(): create a new process and attached to a new namespace unshare(): create a new namespace and attaches to a existed process setns(): Set a a process to a existing namespace /proc/<pid>/ns: lrwxrwxrwx 1 root root 0 Jun 21 18:38 ipc -> ipc:[ ] lrwxrwxrwx 1 root root 0 Jun 21 18:38 mnt -> mnt:[ ] lrwxrwxrwx 1 root root 0 Jun 16 18:24 net -> net:[ ] lrwxrwxrwx 1 root root 0 Jun 21 18:38 pid -> pid:[ ] lrwxrwxrwx 1 root root 0 Jun 21 18:38 user -> user:[ ] lrwxrwxrwx 1 root root 0 Jun 21 18:38 uts -> uts:[ ] We are still waiting security namespace security keys namespace device namespace time namespace 18

19 LIMIT THE RESOURCE - CGROUP Task, Control Group & Hierarchy Subsystem Control options blkio cpu cpuacct cpuset devices freezer memory net_cls net_prio ns Usage Create a cgroup subsystem Change the limitation # echo > /sys/fs/cgroup/memory/foo/memory.limit_in_b ytes 19

20 MISSING - NETWORK Isolation, does not means resource controlling Suppose two containers in a machine, totally 100Gbps b/w 100Gbps

21 MISSING - NETWORK Isolation, does not means resource controlling Suppose two containers in a machine, totally 100Gbps b/w 100Gbps If the GREEN container consumes the majority of b/w, which may have a negative impact on the BLUE one How we can avoid this case from happening? 100Gbps 95 21

22 MISSING - NETWORK Nightmare of the PaaS providers Community attempts: Base on Traffic Control (tc) 22

23 MISSING - NETWORK Nightmare of the PaaS providers Community attempts: Base on Traffic Control (tc) 23

24 MISSING - GPU Nvidia s efforts: a. GPU exposed as separated normal devices in /dev b. devices cgroup: Allow/Deny/List Access i. R ii. W iii. M Ref: 24

25 MISSING - GPU Nvidia s efforts: a. GPU exposed as separated normal devices in /dev b. devices cgroup: Allow/Deny/List Access i. R ii. W iii. M Usable, but insufficient 1. Launch multiple jobs in parallel, each one us a subset of avaiable GPUs; 2. How about share GPU between Jobs with proper isolation? Can we share a GPU like we can a CPU? Ref: 25

26 MISSING - CACHE Intel s efforts: Cache Monitor Technology (CMT) For an OS or VMM to indicate a softwaredefined ID for each of applications or VMs that are scheduled to run on a core. This ID is called the Resource Monitoring ID (RMID). To Monitor cache occupancy on a per RMID basis For an OS or VMM to read LLC occupancy for a given RMID at any time. Cache Allocation Technology (CAT) The ability to enumerate the CAT capability and the associated LLC allocation support via CPUID. Interfaces for the OS/hypervisor to group applications into classes of service (CLOS) and indicate the amount of last-level cache available to each CLOS. These interfaces are based on MSRs (Model-Specific Registers). Code and Data Prioritization (CDP) Extension to CAT a new CPUID feature flag is added within the CAT sub-leaves at CPUID.0x10.[ResID=1]:ECx[bit 2] to indicate support 26

27 MISSING MEMORY BANDWIDTH Memory Bandwidth Monitoring (MBM) Mechanisms in hardware to monitor cache occupancy and bandwidth statistics as applicable to a given product generation on a per software-id basis. Mechanisms for the OS or hypervisor to read back the collected metrics such as L3 occupancy or Memory Bandwidth for a given software ID at any point during runtime. Monitor Control Ref Memory Bandwidth Management for Efficient Performance Isolation in Multi-core Platform: Code: 27

28 MISSING MEMORY BANDWIDTH Memory Bandwidth Monitoring (MBM) Mechanisms in hardware to monitor cache occupancy and bandwidth statistics as applicable to a given product generation on a per software-id basis. Mechanisms for the OS or hypervisor to read back the collected metrics such as L3 occupancy or Memory Bandwidth for a given software ID at any point during runtime. Monitor Control Ref Memory Bandwidth Management for Efficient Performance Isolation in Multi-core Platform: Code: 28

29 WATCH THE WORKLOAD CHANGE Latencies App request latency Disk IO await Network response time Queue length CPU load average Disk request queue size Network queue length Utilization CPU util rate Disk util rate Network util rate Bandwidth DRAM bandwidth CPU bandwidth Disk bandwidth Request count App request count Disk IOPS / req/s Network IOPS / req/s Granularity Global level Per container level 29

30 THE FEEDBACK CONTROL LOOP Controller Soft Container Watcher Limiter Containers 30

31 THE FEEDBACK CONTROL LOOP Controller Soft Container Watcher Immediate response Limiter Containers 31

32 THE FEEDBACK CONTROL LOOP Controller Soft Container Watcher Immediate response Limiter Containers How to immediately resize the containers? 32

33 HOW WE LOOK AT RESIZE? a. Create a new container; b. Live migrate the contents to new container: 1. Transfer existed data to new container; 2. Transfer the instant data to new container. c. Stop the old container d. Start the new container e. Route the traffic to new container 33

34 IN CONTAINER S WORLD 9527 /usr/sbin/httpd a. Mount to new cgroup or change the value of the cgroup b. Done! Control Groups (cgroup): CPU time: 20 System memory: 1G Disk bandwidth: 2000 Network bandwidth: 100Mbs Control Groups (cgroup): CPU time: 70 System memory: 5G Disk bandwidth: 8000 Network bandwidth: 1Gbs 34

35 IN CONTAINER S WORLD a. Mount to new cgroup or change the value of the cgroup b. Done! 9527 /usr/sbin/httpd Control Groups (cgroup): CPU time: 20 We need to take a fresh look at System memory: 1G Disk bandwidth: 2000 Network bandwidth: 100Mbs the resources management from Container s perspective. Control Groups (cgroup): CPU time: 70 System memory: 5G Disk bandwidth: 8000 Network bandwidth: 1Gbs 35

36 SOFT CONTAINER: IMPLEMENTATION Controller Algorithm expand Algorithm pin_idle Container Repo RunC plugin Docker plugin Algorithm plugin N Container type N Watcher CPU plugin Disk plugin Watcher plugin N CPU statistics Disk More Metrics Store Auto discovery Limiter RunC plugin Docker plugin Limiter plugin N Containers 36

37 SOFT CONTAINER: CURRENT STATUS Early version Support RunC and Docker containers A few controller algorithms which are effective Able to expand with more plugins Completely runnable! 37

38 Demo Time :-) 38

39 BENCHMARK RESULTS: BEFORE If uncontrolled, MySQL workload is severely interfered by co-located low priority task 39

40 BENCHMARK RESULTS: BEFORE The CPU utilization is far from saturation while workload varies by time (Although in my case, disk IO is highly utilized) 40

41 BENCHMARK RESULTS: SOFT CONTAINER With Soft Container (green line), latency impact is controlled. (We can improve the algorithm to cope better with peak workload) 41

42 BENCHMARK RESULTS: SOFT CONTAINER Soft Container helps improve CPU utilization by co-locating new tasks with MySQL 42

43 BENCHMARK RESULTS: SOFT CONTAINER CPU utilization looks close to saturation, after adding in iowait time 43

44 HOW DOES SOFT CONTAINER DID THIS? Soft Container monitors app resource needs and overall resource utilization in realtime Soft Container issues resource controls in realtime, to guard app SLA and balance resource utilization 44

45 BENCHMARK RESULTS: SOFT CONTAINER How the resource bubble floats under the control of Soft Container. (The vibration threshold are made very sensitive to workload change) 45

46 Q&A