My VM is Lighter (and Safer) than your Container

Size: px

Start display at page:

Download "My VM is Lighter (and Safer) than your Container"

Timothy Norman
5 years ago
Views:

Yasukata, Sumit Sati, Costin Lupu*, Costin Raiciu*,

1 My VM is Lighter (and Safer) than your Container Filipe Manco, Florian Schmidt, Simon Kuenzer, Kenichi Yasukata, Sumit Sati, Costin Lupu*, Costin Raiciu*, Felipe Huici NEC Europe Ltd, *University Politehnica of Bucharest

Containers container container container apps apps apps libs / services Dispel 250 the myth that VMs are

2 No. of syscalls Isolating workloads: Pick your Poison Virtualization 400 VM 350 apps 300 libs GuestOS (Linux) VM apps libs GuestOS FreeBSD Hypervisor Hardware VM apps libs GuestOS Windows vs. Containers container container container apps apps apps libs / services Dispel 250 the myth that VMs are heavyweight Host OS (Linux) Linux Release Year Hardware Strong isolation Heavy weight? Lightweight Iffy isolation 2

3 LightVM VMs as fast and efficient as containers Fast instantiation, destruction and migration 10s of milliseconds or less Low per-instance memory footprint 10s of MBs or less High density 1-10K guests on a single host 3

4 Standard VM: Application on Top of Distro VM User Application 3 rd Party Applications Libraries Services Kernel

5 Most of the VM not Used Nginx User Application memcached bash 3 rd Party Applications libc libssl Libraries init ssh Services ext4 netfront blkfront Kernel

Unikernel : single app+minimalistic OS Nginx init memca ched libs sl libc ssh bas h ext4 Unikernels are lightweight: netfr ont blkfr ont Daytime - 480KB disk, 3.

6 Unikernel : single app+minimalistic OS Nginx init memca ched libs sl libc ssh bas h ext4 Unikernels are lightweight: netfr ont blkfr ont Daytime - 480KB disk, 3.4MB RAM Minipython MB disk, 8MB RAM TLS termination proxy 3.58MB disk, 8MB RAM Nginx netfront blfront MINIMALISTIC OPERATING SYSTEM (e.g., MiniOS, OSv) SINGLE ADDRESS SPACE 6

7 Tinyx: Small Linux Distro for Target App Nginx memcached bash Find dependencies objdump dpkg libssl libc ssh Install app & deps OverlayFS Cleanup init ext4 netfront Small kernel: Remove drivers User config opts blkfront

8 Tinyx: Small Linux Distro for Target App Nginx bash memcached libssl libc ssh init Tinyx VMs are also lightweight: Kernel: 1.5MB (compared to 8MB) Image size 10-30MB (compared to 1GB). Boot: 200ms instead of 2s. ext4 netfront blkfront

9 Are lightweights VMs enough for good performance? 9

10 VM, Container and Process Creation Times 10 5 Process Create 10 4 Time [ms] Number of running guests 10

11 VM, Container and Process Creation Times 10 5 Process Create Docker Boot 10 4 Time [ms] Number of running guests 11

12 VM, Container and Process Creation Times 10 5 Docker Boot Debian Boot Debian Create Time [ms] Debian: 1.1GB, secs Number of running guests 12

13 VM, Container and Process Creation Times 10 5 MiniOS Boot MiniOS Create 10 4 Time [ms] Number of running guests 13

14 VM, Container and Process Creation Times 10 5 Tinyx Boot Tinyx Create 10 4 Time [ms] Number of running guests 14

drivers NIC block netback xenbus virt drivers netfront

15 A Quick Xen Primer Dom0 (Linux/NetBSD) DomU 1 libxl libxc xl toolstac k libxs apps OS (Linux) SW switch Xen store drivers NIC block netback xenbus virt drivers netfront xenbus Xen Hypervisor Hardware (CPU, Memory, MMU, NICs, ) 15

16 Creation Times Breakdown (unikernel) Time [ms] config hypervisor xenstore devices load toolstack Biggest culprits: XenStore and device creation Number of running guests * Note: Spikes in graph due to XenStore s garbage collector (it s written in OCaml) 16

Create device x3 status Notify More than 30 Xenstore entries

17 What s wrong with the XenStore? xl toolstack Backend (dom0) Frontend (VM) Write x6 Notify Create device x3 status Notify More than 30 Xenstore entries are used per device, resulting in hundreds of XenStore accesses. Xen Hypervisor Xen store 17

18 LightVM A Lightweight Virtualization System 18

Chaos toolstack optimized for paravirtualized guests libxc SW

19 LightVM Architecture Dom0 (Linux/NetBSD) chaos daemon libchaos (prepare) chaos libchaos (execute) 1. Chaos toolstack optimized for paravirtualized guests libxc SW switch xendevd 2. Split functionality 3. Noxs - no XenStore drivers NIC block netback NOXS virt drivers 19

20 VM creation steps VM create command VM CREATE PROCESS VM creation calls (standard toolstack) 1. HYPERVISOR RESERVATION fn1 fnn 2. COMPUTE ALLOCATION fn1 fnn 3. MEMORY RESERVATION fn1 fnn 4. MEMORY PREPARATION fn1 fnn 5. DEVICE PRE-CREATION fn1 fnn 6. CONFIGURATION PARSING fn1 fnn 7. DEVICE INITIALIZATION fn1 fnn 8. IMAGE BUILD fn1 fnn 9. VIRTUAL MACHINE BOOT fn1 fnn COMMON TO ALL GUESTS VM creation calls (split toolstack) 1. HYPERVISOR RESERVATION fn1 fnn 2. COMPUTE ALLOCATION fn1 fnn 3. MEMORY RESERVATION fn1 fnn 4. MEMORY PREPARATION fn1 fnn 5. DEVICE PRE-CREATION fn1 fnn 6. CONFIGURATION PARSING fn1 fnn 7. DEVICE INITIALIZATION fn1 fnn 8. IMAGE BUILD fn1 fnn 9. VIRTUAL MACHINE BOOT fn1 fnn PRE-CREATE PHASE (DAEMON) VM create command RUN-TIME PHASE 20

21 NoXS: Xen without the Xenstore Xenstore is used to: Store data Communicate between guests Synchronization NoXS Shared memory Event channels 21

22 NoXS: Xen without the Xenstore chaos toolstack Update dev info page Device info Dev ctrl: Device info Notify Frontend (VM) Device control Device info Backend (dom0) 22

23 Evaluation 23

24 Instantiation Unikernels vs. Tinyx Time [ms] Tinyx over LightVM Unikernel over LightVM Docker Tinyx (minimalistic Linux) close to Docker times! Number of Running VMs/Containers Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions

25 Instantiation High Density (noop unikernel) Docker: 150 ms-666ms Out of memory Unikernel: 5.2ms-8.6ms Server: 4 x AMD Opteron 6376 CPU@2.3GHz (64 cores total), 128GB DDR3 RAM, Xen/Linux versions

26 Understanding LightVM s components xl Creation Time [ms] Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions

27 Understanding LightVM s components xl Creation Time [ms] chaos [XS] Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions

28 Understanding LightVM s components xl Creation Time [ms] chaos [XS] chaos [XS+split] Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions

29 Understanding LightVM s components 4096 Creation Time [ms] xl chaos [XS+split] chaos [XS] chaos [NoXS] Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions

30 Understanding LightVM s components xl Creation Time [ms] chaos [XS] chaos [XS+split] Note: 2.3 ms with all optimizations and no devices chaos [NoXS] LightVM Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions

31 Use Cases 31

32 Mobile edge computing Internet Core Network Mobile Edge Cloud

One firewall for each mobile in the cell (ClickOS unikernel) Total Throughput(Gbps) 4 3 2 1 0 Maximal theoretical LTE-advanced throughput per cell

33 One firewall for each mobile in the cell (ClickOS unikernel) Total Throughput(Gbps) Maximal theoretical LTE-advanced throughput per cell sector: 3.3 Gb/s Throughput # Running VMs Server: Intel Xeon E v4 2.6 GHz (14 cores), 64GB DDR4 RAM, Xen/Linux versions

34 One VM per mobile: low latency services CDF ping RTT Server: Intel Xeon E v4 2.6 GHz (14 cores), 64GB DDR4 RAM, Xen/Linux versions

35 Related work Intel Clear Containers Runs containers within VMs => added overheads. 70MB Linux guest, 500ms boot. Unikernel-based approaches: ukvm KVM optimizations to boot Mirage unikernels ~10ms. Jitsu [Madhvapeddy, NSDI 2015] Xen optimizations for unikernels, networking workloads. 35

36 LightVM: lightweight virtual machines Virtual machines can provide strong isolation and be lightweight: ms instantiation times, memory footprint of 10s of MBs. Achieved through lightweight guests re-architected toolstack Try LightVM: 36

Unleashing the Power of Unikernels with Unikraft

EU H2020 Superfluidity Unleashing the Power of Unikernels with Unikraft Felipe Huici felipe.huici@neclab.eu Systems and Machine Learning Group NEC Laboratories GmbH, Heidelberg Who am I? Chief Researcher