Unleashing the Power of Unikernels with Unikraft

Size: px

Start display at page:

Download "Unleashing the Power of Unikernels with Unikraft"

Piers Simpson
6 years ago
Views:

1 EU H2020 Superfluidity Unleashing the Power of Unikernels with Unikraft Felipe Huici Systems and Machine Learning Group NEC Laboratories GmbH, Heidelberg

2 Who am I? Chief l NEC Laboratories in Heidelberg, Germany l Systems and Machine Learning Group 2 NEC Corporation 2018

3 The Container Craze Google runs all services in containers Container-as-a-Service services: l Amazon Lambda, EC2 Container Service l Google Container Engine l Azure Container service l and so on 3 NEC Corporation 2018

4 The Container Advantage Very fast instantiation times l 100s ms Small per-instance memory footprints l MBs High density l instances on same host VMs 4 NEC Corporation 2018

5 The Unsafe Container Kernel API difficult to secure, lots of exploits An unsafe container affects all other containers on host l This includes DoS/exhaustion attacks (e.g., forkbombs, etc.) No. of syscalls Linux Release Year 5 NEC Corporation 2018

6 Pick Your Isolating Poison Workloads Virtualization VM VM VM vs. Containers container container container apps apps apps apps apps apps libs libs libs libs / services GuestOS (Linux) GuestOS FreeBSD GuestOS Windows Host OS (Linux) Hypervisor Hardware Hardware Strong isolation Heavy weight? Lightweight Iffy isolation 6 NEC Corporation 2018

GuestOS FreeBSD GuestOS Windows apps apps apps libs / services Dispel the myth that VMs

7 Isolating Workloads Virtualization VM VM VM vs. Containers container container container apps apps apps libs libs libs GuestOS (Linux) GuestOS FreeBSD GuestOS Windows apps apps apps libs / services Dispel the myth that VMs are heavyweight Host OS (Linux) Hypervisor Hardware Hardware Strong isolation Heavy weight? Lightweight Iffy isolation 7 NEC Corporation 2018

8 What s a Unikernel Then?

9 Standard VM: Application on Top of Distro User Application(s) 3 rd Party Applications Libraries Services Kernel 9 NEC Corporation 2018

10 Most of the VM not Used Nginx User Application memcached bash 3 rd Party Applications libc libssl Libraries init ssh Services ext4 netfront blkfront Kernel 10 NEC Corporation 2018

11 Specialization in Practice Standard OS/VM/container: lots of unnecessary code = lots of overhead (and potentially bugs/security issues)! Unikernel: only what s needed is there Nginx User Application Nginx memcached bash 3 rd Party Applications memcached bash libssl libc ssh unused! Libraries libssl libc ssh init unused! Services init unused! ext4 netfront blkfront Kernel netfront blkfront ext4 general-purpose OS unikernel 11 NEC Corporation 2018

12 Unikernels - A Few More Details USER SPACE app 1 app 2 KERNEL SPACE driver1 driver2 app N drivern Vdriver1 app vdriver2 SINGLE ADDRESS SPACE GENERAL-PURPOSE OPERATING SYSTEM (e.g., Linux, FreeBSD) MINIMALISTIC OPERATING SYSTEM (e.g., MiniOS, OSv) 12 NEC Corporation 2018

13 When do I want a Unikernel? Unikernels are applicable to many domains! Minimal SW stack Reactive VNFs, Lambda Functions, Fast boot, migration destroy Resource efficient Minimal SW stack Per-customer VNFs, IoT, MEC, Specialization NFV, MEC, High performance Mission critical Small code base à cheap verification Strong isolation Automotive, Industrial-grade, 13 NEC Corporation 2018

14 Some Unikernel Numbers My VM is Lighter (and Safer) than your Container SOSP 2017

In Numbers: Instantiation Times Server: Intel Xeon E5-1630 v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions 4.

15 In Numbers: Instantiation Times Server: Intel Xeon E v3 (4 cores), 128GB DDR4 RAM, Xen/Linux versions Process Create 10 4 Time [ms] Process: 0.7ms-10ms Number of running guests 15 NEC Corporation 2018

16 In Numbers: Instantiation Times Server: Intel Xeon E v3 (4 cores), 128GB DDR4 RAM, Xen/Linux versions Process Create Docker Boot 10 4 Time [ms] Docker: 150ms-550ms Number of running guests 16 NEC Corporation 2018

17 In Numbers: Instantiation Times Server: Intel Xeon E v3 (4 cores), 128GB DDR4 RAM, Xen/Linux versions Docker Boot Debian Boot Debian Create Debian: secs Time [ms] Number of running guests 17 NEC Corporation 2018

18 In Numbers: Instantiation Times Server: Intel Xeon E v3 (4 cores), 128GB DDR4 RAM, Xen/Linux versions Docker Boot Debian Boot Debian Create Debian: secs Time [ms] Traditional VMs are in fact heavy-weight Number of running guests 18 NEC Corporation 2018

19 In Numbers: Instantiation Times Server: Intel Xeon E v3 (4 cores), 128GB DDR4 RAM, Xen/Linux versions MiniOS Boot MiniOS Create Time [ms] unikernel:, 163ms-1.4secs Number of running guests 19 NEC Corporation 2018

20 In Numbers: Instantiation Times Server: Intel Xeon E v3 (4 cores), 128GB DDR4 RAM, Xen/Linux versions MiniOS Boot MiniOS Create Time [ms] unikernel:, 163ms-1.4secs Unikernels can be almost as fast as 10 2 containers Number of running guests 20 NEC Corporation 2018

21 In Numbers: Instantiation Times Server: Intel Xeon E v3 (4 cores), 128GB DDR4 RAM, Xen/Linux versions MiniOS Boot MiniOS Create Time [ms] unikernel:, 163ms-1.4secs Can we make them faster? Number of running guests 21 NEC Corporation 2018

22 LightVM A Lightweight Virtualization System 22 NEC Corporation 2018

23 A Quick Xen Primer Dom0 (Linux/NetBSD) DomU 1 libxl libxc xl toolstack libxs apps OS (Linux) SW switch Xen store drivers NIC block netback xenbus virt drivers netfront xenbus Xen Hypervisor Hardware (CPU, Memory, MMU, NICs, ) 23 NEC Corporation 2018

24 Creation Times Breakdown (unikernel) Time [ms] NEC Corporation 2018 config hypervisor xenstore devices load toolstack Biggest culprits: XenStore and device creation * Note: Spikes in graph due to XenStore s log rotation Number of running guests

are used per device, resulting in hundreds of

25 What s Wrong with the XenStore? xl toolstack Backend (dom0) Frontend (VM) Write x6 status Notify Notify Create device x3 Notify Notify More than 30 Xenstore entries are used per device, resulting in hundreds of XenStore accesses. Xen Hypervisor Xen store 25 NEC Corporation 2018

Noxs - no XenStore 3. Split toolstack 4.

26 LightVM Architecture Dom0 (Linux/NetBSD) chaos daemon libchaos (prepare) libxc chaos libchaos (execute) 1. Chaos specialized toolstack (xl replacement) 2. Noxs - no XenStore 3. Split toolstack 4. Xendevd optimized hotplug script SW switch xendevd drivers NIC block netback NOXS virt drivers 26 NEC Corporation 2018

27 1. Chaos - Specialized Toolstack Complete re-write of toolstack for paravirtualized guests l Replaces xl à chaos l Replaces libxl à libchaos Includes framework for easily plugging in different elements of a toolstack (e.g., use the XenStore or not) 27 NEC Corporation 2018

28 2. NoXS Xen without a XenStore 2 hypercall dom0 toolstack ioctl NOXS backend-id event channel id grant reference 1 device create NW backend netback memory page 4 NW frontend backend-id event channel id grant reference netfront 3 hypercall hypervisor 28 NEC Corporation 2018

29 3. Split Toolstack Insight: a significant portion of the code does not need to be executed at VM creation time l Functionality common to all VMs (e.g., mem alloc) can be preexecuted Split toolstack into two: l Pre-create phase: run by a daemon, periodically l Run-time phase: run when VM creation commands (and others) are issued 29 NEC Corporation 2018

30 4. Xendevd Hotplug Script Xen uses either a bash script or udevd to add virtual interfaces to software switch l Slow: 10s of milliseconds, slows creation/boot times Replace with xendevd, binary daemon l Listens to udev events from netback, execute pre-defined setup (e.g., add vif to OVS) 30 NEC Corporation 2018

31 Evaluation Unikernels We Used Noop unikernel: self explanatory J Daytime unikernel: TCP server that responds with current time Minipython: Port of Micropython to MiniOS TLS unikernel: TLS termination proxy Ping unikernel: also self-explanatory Sample sizes: Daytime - 480KB disk, 3.4MB RAM Minipython MB disk, 8MB RAM TLS 3.58MB disk, 8MB RAM 31 NEC Corporation 2018

32 In Numbers: Instantiation Times with LightVM Docker: 150 ms-666ms Out of memory Unikernel: 5.2ms-8.6ms Server: 4 x AMD Opteron 6376 CPU@2.3GHz (64 cores total), 128GB DDR3 RAM, Xen 4.8/Linux 4.8 LightVM: Guests: Noop/Unikernel and Noop Docker image / No devices 32 NEC Corporation 2018

In Numbers: Instantiation Times with LightVM Docker: 150 ms-666ms Out of memory Unikernels are even faster & lighter when using an optimized Unikernel: 5.2ms-8.

33 In Numbers: Instantiation Times with LightVM Docker: 150 ms-666ms Out of memory Unikernels are even faster & lighter when using an optimized Unikernel: 5.2ms-8.6ms hypervisor toolstack Server: 4 x AMD Opteron 6376 CPU@2.3GHz (64 cores total), 128GB DDR3 RAM, Xen 4.8/Linux 4.8 LightVM: Guests: Noop/Unikernel and Noop Docker image / No devices 33 NEC Corporation 2018

34 Checkpointing Daytime Unikernel Time [ms] Time [ms] xl 16 chaos+xenstore 4 LightVM 1 xl chaos [XS] chaos [NoXS] LightVM Number of Running VMs Number of Running VMs (a) Save (b) Restore Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

35 Migration Daytime Unikernel Time [ms] xl chaos [XS] chaos [NoXS] LightVM Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

36 CPU Usage CPU Utilization (%) Debian Tinyx Unikernel Docker Number of Running VMs/Containers Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

37 Memory Usage Memory Usage [MB] Debian Tinyx Docker Micropython Minipython Micropython Process VM/Container/Process # Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

38 Understanding LightVM s components xl Creation Time [ms] Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

39 Understanding LightVM s components xl Creation Time [ms] chaos [XS] Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

40 Understanding LightVM s components xl Creation Time [ms] chaos [XS] chaos [XS+split] Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

41 Understanding LightVM s components 4096 Creation Time [ms] xl chaos [XS+split] chaos [XS] chaos [NoXS] Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

42 Understanding LightVM s components 4096 Creation Time [ms] xl chaos [XS+split] chaos [XS] Note: 2.3 ms with all optimizations and no devices chaos [NoXS] LightVM Number of Running VMs Server: Intel Xeon E v3 CPU@3.7GHz (4 cores), 128GB DDR4 RAM, Xen/Linux versions NEC Corporation 2018

43 The Potential of Unikernels A Summary Fast instantiation, destruction and migration time l 10s of milliseconds or less (and as little as 2.3ms) (LigthVM [Manco SOSP 2017], Jitsu [Madhvapeddy, NSDI 2015]) Low memory footprint l Few MBs of RAM or less (ClickOS [Martins NSDI 2014]) High density l 8k guests on a singlex86 server (LigthVM [Manco SOSP 2017]) High Performance l 10-40Gbit/s throughput with a single guest CPU (ClickOS [Martins NSDI 2014], Elastic CDNs [Kuenzer VEE 2017]) Reduced attack surface l Small trusted compute base l Strong isolation by hypervisor 43 NEC Corporation 2018

44 So unikernels are perfect right? 44 NEC Corporation 2018

45 The (Big) Downside with Unikernels Each optimized Unikernel is manually built l Building takes several months or longer l Potentially wash, rinse, repeat for each target application 45 NEC Corporation 2018

46 Unikraft A Unikernel and Specialized OS Build Framework

47 Unikraft Goals Severely reduce development time Simultaneously target multiple platforms l Xen l KVM l Bare metal (X86_64, ARM32, etc.) Allow simple customization of the unikernel without having to tweak the actual code 47 NEC Corporation 2018

48 The Unikraft Way Decompose OSes and libraries into primitives Recompose them to meet the needs of particular applications Application(s) network stack profiling filesystem memory allocator timers scheduler drivers 48 NEC Corporation 2018

particular applications Application(s) network stack

choose which parts/libraries we actually memory need

49 The Unikraft Way Decompose OSes and libraries into primitives Recompose them to meet the needs of particular applications Application(s) network stack Once decomposed, filesystem profiling we can pick and choose which parts/libraries we actually memory need for our timers application allocator scheduler drivers 49 NEC Corporation 2018

50 Unikraft Overview Everything as a Library myapp main libs SELECT&CONFIG LIBS network stack drivers liblwip.o libtcpip.o libhttp.o libconsole.o libixgbe.o libnetfront.o filesystems libvfs.o libfat.o libext3.o memory allocators libbuddy.o libheap.o libmempool.o schedulers libcoop.o libpreempt.o librt.o runtimes libocaml.o libpython.o liberlang.o standard libs libc.o libnewlibc.o libopenssl.o debug&profiling libgdb.o libucdebug.o libperf.o platform libs Libdockerplat.o libbaremetalplat.o Libkvmplat.o libxenplat.o BUILD architecture libs libx86_64arch.o libarm32arch.o libmipsarch.o Unikernels unicore_bare_x86_64 unicore_bare_arm32 unicore_bare_mips unicore_xen_x86_64 unicore_xen_arm32 unicore_xen_mips unicore_kvm_x86_64 unicore_kvm_arm32 unicore_kvm_mips unicore_docker_x86_ RUN 1 SELECT APP 50 NEC Corporation 2018

51 Two Library Types Built-in: functionality specific to Unikraft (e.g., ukboot, ukschedpreempt, etc.) External: software projects external to Unikraft (e.g., lwip, micropython, etc.) 51 NEC Corporation 2018

for your application Choose your target platform(s) l Xen, KVM,

52 Putting Things Together The Unikraft Build Tool Kconfig/Makefile based Type make menuconfig Choose options in the menu that you want for your application Choose your target platform(s) l Xen, KVM, bare-metal, Linux Save your config and type make.config 52 NEC Corporation 2018

53 Example System Python Unikernel for KVM on x86_64 apppython.o liblwip.o liballocbuddy.o libconsole.o libschedcoop.o libkvmplat.o libfilesystem.o libx86_64arch.o Unikernel 53 NEC Corporation 2018

54 Example System VNF Unikernel for KVM on x86_64 libdpdk.o liballocbuddy.o libconsole.o libschedcoop.o libkvmlat.o libfilesystem.o libx86_64arch.o unikernel 54 NEC Corporation 2018

55 Building a Unikraft Hello World App

56 Repo Structure Clone the main Unikraft repo Clone any external library repos Create repo for the actual application!"" unikraft!"" unikraft-apps helloworld!"" unikraft-libs!"" axtls!"" lwip!"" micropython!"" newlib!"" python!"" toybox Unikraft repo (+ built-in libs) application repo(s) external libraries repos 56 NEC Corporation 2018

57 Hello World Four Required Files (I) Makefile: specify where the main Unikraft repo is, as well as repos for external libraries UK_ROOT?= $(PWD)/../../unikraft UK_LIBS?= $(PWD)/../../unikraft-libs LIBS := $(UK_LIBS)/newlib path to unikraft repo path to external libs external libs needed -C $(UK_ROOT) A=$(PWD) L=$(LIBS) -C $(UK_ROOT) A=$(PWD) L=$(LIBS) $(MAKECMDGOALS) 57 NEC Corporation 2018

58 Hello World Four Required Files (II) Makefile.uk: specifies the sources to build for the application $(eval $(call addlib,apphelloworld)) register app with uk Build system APPHELLOWORLD_SRCS-y += $(APPHELLOWORLD_BASE)/main.c Add main.c to build 58 NEC Corporation 2018

59 Hello World Four Required Files (III) Config.uk: to populate Unikraft s menu with application-specific option ### Invisible option for dependencies config APPHELLOWORLD_DEPENDENCIES bool default y select LIBNOLIBC if!have_libc ### App configuration config APPHELLOWORLD_PRINTARGS bool "Print arguments" default y help Prints argument list (argv) to stdout 59 NEC Corporation 2018

60 Hello World Four Required Files (IV) main.c: source file to provide (at least) a main() function #include <stdio.h> /* Import user configuration: */ #include <uk/config.h> int main(int argc, char *argv[]) { printf("hello world!\n"); Unikernel entry point after boot #if APPHELLOWORLD_PRINTARGS int i; defined by Config.uk #endif } printf("arguments: "); for (i=0; i<argc; ++i) printf(" \"%s\"", argv[i]); printf("\n"); 60 NEC Corporation 2018

61 Porting an External Library

62 How To Port an External Library Most of the work revolves around ensuring Unikraft supports your lib s dependencies à we re working on this! J Write Makefile.uk and add the external library source files to it Sometimes a bit of glue code is needed l E.g., in newlib to link POSIX thread creation to Unikraft s thread library 62 NEC Corporation 2018

63 How To Port an External Library Makefile.uk ################################################################################ # Library registration ################################################################################ $(eval $(call addlib_s,libaxtls,$(libaxtls))) ################################################################################ # Sources ################################################################################ # Nothing here: sources are small and included directly in the uk library repo # Note: sources are from axtls tar.gz. ################################################################################ # Library includes ################################################################################ CINCLUDES-y += -I$(LIBAXTLS_BASE)/include \ -I$(LIBAXTLS_BASE)/crypto \ -I$(LIBAXTLS_BASE)/ssl ################################################################################ # sources ################################################################################ LIBAXTLS_SRCS-y += $(LIBAXTLS_BASE)/crypto/aes.c LIBAXTLS_SRCS-y += $(LIBAXTLS_BASE)/crypto/bigint.c LIBAXTLS_SRCS-y += $(LIBAXTLS_BASE)/crypto/sha512.c 63 NEC Corporation 2018

64 Unikraft 0.2 Titan Current Status

65 Available Libraries Main Libraries l libfdt Flat device tree parser l libnolibc A tiny libc replacement l libnewlib (external) Porting ongoing l libukalloc Memory allocator abstraction l libukallocbbuddy Binary buddy allocator for libukalloc l libukargparse Argument parser library l libukboot Unikraft bootstrapping l libukdebug Debug and kernel printing Assertions, hexdump l libuksched Scheduler abstraction l libukschedcoop Cooperative scheduler for libsched Architecture Libraries l libarmmath 64bit arithmetic on ARMv7 Platform Libraries l libxenplat Xen (PV) X86_64, ARMv7 l libkvmplat X86_64 l liblinuxu Linux userspace X86_64, ARMv7 65 NEC Corporation 2018

66 Coming Soon l libukschedpreempt Pre-emptive scheduler l libukswrand Software random number generator l libukvirtio Virtio driver l liblwip (external) Network stack l libclick (external) The Click modular router (e.g., for NFV) l libmicropython (external) Micropython (Python for embedded devices) l libcpython (external) Standard Python l libaxtls (external) TLS support for embedded devices More on the way! 66 NEC Corporation 2018

o Final linking unikraft_xen-x86_64 (32,7kB) Boots and prints

67 A Baseline Example Xen PV x86_64 binary unikraft_xen-x86_64.o (50,2kB) libnolibc.o libxenplat.o libukboot.o libukdebug.o Final linking unikraft_xen-x86_64 (32,7kB) Boots and prints messages to debug console (with min. 208kB RAM) 67 NEC Corporation 2018

68 Short Video/Demo It is real!

It s Open Source Join Us! Recently: Unikraft got released as Xen incubator project l OpenSource development under Xen and LinuxFoundation umbrella Have look on the Xen Pages and Wiki l https://www.

69 It s Open Source Join Us! Recently: Unikraft got released as Xen incubator project l OpenSource development under Xen and LinuxFoundation umbrella Have look on the Xen Pages and Wiki l l Drop us an mail l minios-devel@lists.xen.org Join our IRC channel l #unikraft on Freenode We have plenty of open topics, get in touch with us! l 69 NEC Corporation 2018

70 This work has received funding from the European Union s Horizon 2020 research and innovation program under grant agreement no ( Superfluidity ). This work reflects only the author s views and the European Commission is not responsible for any use that may be made of the information it contains.

My VM is Lighter (and Safer) than your Container

My VM is Lighter (and Safer) than your Container Filipe Manco, Florian Schmidt, Simon Kuenzer, Kenichi Yasukata, Sumit Sati, Costin Lupu*, Costin Raiciu*, Felipe Huici NEC Europe Ltd, *University Politehnica