Combining Dataplane Operating Systems and Containers for Fast and Isolated Datacenter Applications

Transcription

1 EDIC RESEARCH PROPOSAL 1 Combining Dataplane Operating Systems and Containers for Fast and Isolated Datacenter Applications Mia Primorac DCSL, I&C, EPFL Abstract Commodity operating systems largely used in today s data centers were designed under different hardware assumptions and for any general type of applications. The specialization of operating system design for certain domains has showed to be successful especially in the case of exokernel architecture. In this proposal we examine and compare two recent research projects, Arrakis and IX, that follow the exokernel design while specializing for datacenter applications. We argue that the specialization of the whole operating system is not sufficient and it would be useful to provide a specializable version of more and more popular container technology with respect to demanding networking requirements of datacenter workloads. We propose the containers technique in conjunction with dataplane OS design in order to (1) improve the performance of datacenter applications running in containers, (2) reuse the container management tools for dataplane configuration and resource management and (3) improve the container isolation. Index Terms data center, dataplane, containers I. INTRODUCTION Today s data center operators face a challenge of demanding workloads with extremely high rates of small requests. While Proposal submitted to committee: June 3rd, 2015; Candidacy exam date: June 10th, 2015; Candidacy exam committee: Prof. James Larus, Prof. Edouard Bugnion, Prof. Willy Zwaenepoel. This research plan has been approved: Date: Doctoral candidate: (name and signature) Thesis director: (name and signature) Thesis co-director: (if applicable) (name and signature) Doct. prog. director: (B. Falsafi) (signature) sustaining high packets rates, datacenter services have to ensure low overall latency for each client request that involves hundreds of servers. Therefore, the low tail latency becomes an inevitable demand together with high throughput and high connection count. In addition to tackling this problem on the end points, it can also be tackled from the bottom by redesigning the software stack, especially the operating system. There are multiple reasons why one would want to redesign the commodity operating systems used in today s data centers. The first argument is almost as old as those operating systems [1] and it argues that generality of the OS design is penalizing applications with specific requirements. In the meanwhile, both, hardware and the applications have changed, but the kernel design has not followed those changes. This gap between hardware and software is especially visible in the networking stack which is primarily designed for fine-grained resource partitioning. Orthogonally to the OS development, Linux containers recently found many important use-cases in data centers and among software developers. New container technologies like LXC and Docker enable easier deployment of applications and their dependencies and server consolidation with bigger resource efficiency than with virtual machines. The purpose of this proposal is not to argue if and how one should use containers in production. We find this technology interesting and believe that it can supplement the specialized dataplane operating system like IX [7], presented in the 3rd chapter. We believe that such composed design brings benefits to both sides. The dataplane operating system would profit in easier configuration and deployment, whereas the applications on top of containers would profit from better isolation and specialized networking stack, without giving up on the comfort of container frameworks like Docker [10] for their management. In the second chapter we will present the original Exokernel design and its later application to networking and specialized applications. In the third and fourth chapter we will present and compare two modern and at the first sight similar operating systems. Both of them are specialized for datacenter applications and both are inspired by the exokernel design to some extent. Finally, we will introduce the research proposal which considers how to combine the exokernel-like design with container technology and what would be the possible benefits and shortcomings of such design. EDIC-ru/

2 EDIC RESEARCH PROPOSAL 2 A. Operating Systems Design Traditionally we can divide kernels into two main types of design: the monolithic kernel and the microkernel. They differ from each other in modularity of the design and in privilege mode under which the operating system services are executed. On x86 processor the most important prilivege modes are ring 0 (kernel or supervisor mode) and ring 3 (user mode). Kernel mode is the unrestricted mode, while user mode is the restricted mode. Rings 1 and 2 are also available, but rarely used. Monolithic kernels are implemented entirely as a single process running in a single address space. All kernel services exist and execute in kernel mode in the large kernel address space. Therefore, the communication within the kernel is as trivial as within a user-space application. It boils down to direct function invocation. Microkernels, on the other hand, are broken down into separate processes, usually called servers. All the servers run in separate address spaces and they communicate via message passing interprocess communication (IPC) mechanism. Only the servers absolutely requiring such capabilities run in kernel mode, while the rest of the servers run in user mode. The message passing between components running in different privilege modes introduces more overhead than a trivial function call in the monolithic kernels. On the other hand, the monolithic kernels suffer from the lack of flexibility and fault tolerance. Most of the data centers still run Linux commodity operating systems. Linux is a monolithic kernel, therefore running in a single address space and completely in the kernel mode. However, many good features of microkernels are incorporated in its design, for instance modular design and capability to dynamically load separate binaries (kernel modules) [14]. One thing in common to all the aforementioned systems, including Linux-based systems, is imposing the fixed set of abstractions to every user application, without providing any way of bypassing or easily extending them. This problem was addressed by the third type of kernel - the exokernel [2] [3] [4]. The principal goal of an exokernel - giving applications control - is orthogonal to the question of monolithic versus microkernel organization. If applications are restricted to inadequate interfaces, it is indifferent whether the implementations reside in the kernel or privileged user-level servers; in both cases applications lack control [3]. B. Networking Performance Gap Between Hardware and Software One of the most important pro-exokernel arguments is the stale design of operating system abstractions designed under different hardware assumptions. For instance, kernel schedulers, networking API and network stacks have been designed under the assumptions of multiple applications sharing a single processing core and packet interarrival times being several orders of magnitude higher than the latency of interrupts [7]. Nowadays that is no longer true and we are paying an overhead in throughput and latency for fine-grained scheduling of plentiful resources. Today we have multiple processors, multiple queues on the networking cards, and the packet inter-arrival times are no longer negligible with respect to the cost of an interrupt (or I/O processing in general). Linux completely decouples networking stack from application execution, hence suffers from bad cache locality. Fine-grained application scheduling prevents applications to process their packets before they are scheduled. Even if the application is scheduled, on systems with multiple processing cores and multi-queue NICs, different packets can be delivered to different queues to distribute processing among CPU cores, which may not be the same as the cores on which the application is scheduled. Further on, socket interfaces involve unnecessary data copies because the application buffers are not explicitly exchanged with the kernel. Transport-level acknowledgements and retransmissions may be redundant, but protocol implementations are hidden from the applications preventing changes. Even with NIC optimizations for increasing throughput, like receive side coalescing, or for reducing latency, polling the incoming ring buffer (RX) with NAPI (New API) [15] (instead of interrupt-based processing), there is still a substantial performance gap between hardware potentials and software achievements. This problem has been addressed by several research projects like Exokernel [2] [3] [4], Arrakis [5], and IX [7]. C. Linux Containers and the Docker Platform Linux container is a set of processes that are isolated from the rest of the machine, but, unlike the virtual machines, they share the operating system. Therefore, containers are light-weighted with respect to resource consumption, and they are much faster in bootstrapping and shutting down than virtual machines. In this chapter we will briefly introduce the important terms related to container technologies and kernel mechanisms they use, as well as often heard buzzwords like LXC and Docker. LXC [12] is a user space interface for the Linux kernel containment features such as control groups and network namespaces. The LXC project provides base OS container templates and a set of tools for container lifecycle management. However, the most often heard name in the context of containers is certainly Docker. Docker is a project originally based on the LXC project to build single application containers. Docker has now developed their own implementation, libcontainer, that uses kernel container capabilities directly. One can think of Docker container as a specific use case of Linux containers to build loosely applications as services. Docker consists of Docker Engine, a portable, lightweight runtime and packaging tool, and Docker Hub, a cloud service for sharing applications and automating workflows. Comfortable tools and image sharing environment makes it feasible for broader community of users, but as well for a use in data centers. There are multiple Linux kernel technologies in charge of the container isolation and encapsulation, for instance control groups and kernel namespaces. Control groups (cgroups) is a Linux kenrel feature that limits, accounts for and isolates the

3 EDIC RESEARCH PROPOSAL 3 Fig. 1: Exokernel architecture resource usage (CPU, memory, disk I/O, etc.) of a collection of processes and their children. Network namespaces are one of the kernel namespaces. They enable separate instances of network interfaces, routing tables and iptables, that operate independent of each other. Linux containers logically run separate networking stacks because they belong to different networking namespaces. However, they all share the same code base - the general one of the Linux kernel. The emerging container technologies motivate us to come up with an interesting system design that takes advantage of some of them, but as well to contribute to its design, in particular, to the networking stack and isolation of containers. II. EXOKERNEL Followed by the principles of good computer systems design, the authors of the Exokernel paper suggested an innovative operating system architecture in which traditional operating systems abstractions, such as virtual memory, IPC, files etc., are not imposed to all untrusted applications with rather different requirements. In traditional operating system the applications are faced with fixed interfaces to access the abstractions that may or may not fit their needs. The main idea behind Exokernel design is letting applications decide how to use their (and only their) physical resources and letting them to choose the abstractions that are a good match for them, and customize the rest of them. That is achieved through securely exporting hardware interface to user space where different customized operating system libraries and applications can use them (almost) directly. The main function of the kernel itself is securely multiplexing hardware resources without understanding them or the abstractions on top of them. In other words, the main goal is separating resource protection from resource management. The first exokernel paper [2] provides the idealistic overview of the exokernel architecture and gives the promising microbenchmarking results for a research prototype exokernel called Aegis and the library operating system ExOS. Later on, in the two followon papers [3] and [4] the authors give more detailed examples of real systems, services and applications that take advantage of the exokernel implemented on x86 called Xok and more complex ExOS. They more precisely describe the disk [3] and network [3] multiplexing. In this paper we will focus on the networking aspect of Xok given in [4]. A. Design Principles In order to achieve that, the exokernel architecture: (1) securely exposes hardware resources through system calls that check the ownership of every resource being involved, (2) exposes allocation requests for specific physical resources, (3) exposes revocation which cooperative applications can use to efficiently manage their resources and (4) exposes physical names, which is important to efficiently use (2) and (3). Driven by these design principles, the kernel implements three mechanisms: secure bindings, visible revocation and abort protocol. Secure bindings bind a specific resource to a specific application (or library OS) during the bind time and enable fast ownership checks for each access time (see Fig. 1). Secure bindings can be seen as a new abstraction introduced by the exokernel architecture whose implementation varies depending on the resource that is being multiplexed. The secure bindings for multiplexing physical memory is implemented as a combination of hardware mechanisms and caching in software TLB. Another example are the secure bindings for multiplexing the network. They are implemented as software packet filters downloaded into the kernel and dynamically compiled to machine code that can efficiently be invoked for every incoming packet. Visible revocation includes the applications in the resource revocation process and gives them an useful insight for efficient resource management. Although invisible revocation has lower latency, the visible revocation improves the overall performance as long as revocations do not occur very frequently. Although the exokernel provides power to the applications, it implements the abort protocol to tame uncooperative applications. If the application does not respond to visible revocation protocol, the kernel breaks the secure bindings to the resource and informs the library operating system. In the Fig. 1 we can see how applications with fairly different requirements manage to satisfy them simoultaneously. The Web server is designed for custom networing stack and file system, while the C shell does not have any exotic requirements and it can freely use the backwards compatible default ExOS library operating system. B. Application-Level Networking on Exokernel Systems Application-level networking allows applications with no special privileges to interact with a network interface. This idea was introduced before, but [4] provides concrete examples of how application-level networking can substantially improve end-to-end performance, for instance the Cheetah HTTP server. The ExOS library provides a user-level and extensible implementation of an UNIX operating system. They implemented full network services (UDP/IP, TCP/IP, POSIX sockets, ARP, DNS, and tcpdump) as independent application-level libraries and defined a set of base kernel mechanisms and a base network interface on which such applications and services can successfully be implemented. They focus on solving the problem of efficiently sharing a single networking interface among multiple applications using efficient polling, rather than giving ownership to a single

4 EDIC RESEARCH PROPOSAL 4 entity (like IX [7] does). This paper presents many interesting techniques to reduce the OS overhead in networking even when sharing is not the main concern. The paper argues against violating the end-to-end argument [8] in OS design. The argument argues against placing redundant functionality at low levels of system instead of the ends, which are the applications in the case of the Internet protocol. The efficient polling of applications is implemented using a mechanism called a WK (WaKeup) predicate. WK predicate is a set of conditions that describe an interest of an application. Each condition consists of a memory location and a value that is to be compared with the value on the given memory location. The Xok process scheduler evaluates the WK predicate of a sleeping process marks it runnable only if the WK predicate is true. This mechanism is an efficient optimization for several services. Application-level networking has to solve a problem of multiplexing the network interface in two cases: transmission and reception of packets. The transmission of packets from the application-level is straight-forward from the main memory - a system call takes the network interface on which to send the packet, a list of memory locations and sizes of packets, and a pointer to an integer in the memory which counts how many packets are left to send. When the driver learns that the card has sent a packet, it decrements the counter. Comparing if the counter is equal to zero is a perfect example of the aforementioned WK predicate because the application is not occupying the CPU while waiting for its packets to be sent. The reception does not dictate the behavior of applications and does not include unnecessary copies. The reception of packets consists of two phases: identifying the owner (demultiplexing) and packet delivery (buffering). To demultiplex packets among applications, Xok uses DPFs (Dynamic Packet Filters) which are downloaded and installed into the kernel. Each filter is associated with a packet ring identifier. A packet ring is a shared ring of buffers shared between an application and Xok. Each buffer has an ownership flag which points either to the application or Xok. When the application processes the packets from the buffer, it gives the ownership back to Xok. Each packet is copied once, but if we have an advanced NIC with demulitplexing support, then not even a single copy is needed (like in IX or native Arrakis API). C. Cheetah HTTP Server In [4] there are several examples of successfully exploiting the application-level networking in Xok. We will focus on one of them - the Cheetah HTTP server. Cheetah is a perfect example of how the exokernel architecture and applicationlevel networking can significantly boost the performance. Cheetah has a specialized TCP/IP and file system library. The main control loop simply waits for relevant events, like packet arrival, calls into its specialized library and checks perrequest state which determines if the request can be fully processed. The performance of the Cheetah server is partly credited to the non-blocking event loop without mediation through socket interfaces. The major improvement, however, is credited to three other extensions enabled by applicationlevel networking. First, merged file cache and TCP retransmission pool enable zero-copy disk to network transfer. Second, Cheetah avoids to send distinct TCP control packets that can be merged with forthcoming data packets. Third, TCP checksums of packets are precomputed and stored on the disk along with the data. Cheetah on Xok outperforms the Baseline server (socket-based HTTP/1.0 server as well presented in [4]) on both Xok and OpenBSD, but also outperforms two other systems running on OpenBSD; the Harvest cache [16] and the NCSA server [17]. Cheetah achieves 8x bigger throughput for small HTTP documents (less than 1KB) and roughly 4x bigger throughput for large documents (more than 10KB) than Harvest and NCSA server. The throughput was measured with six clients repeatedly requesting the same static Web document (100% cache hit rate) without any timeout between the requests. D. Discussion The Exokernel papers provide an interesting research idea and present two systems, the research prototype Aegis and Xok, more mature but still limited to research context. With Aegis they focused on tuning microbenchmarks, whereas with Xok they proved that it is possible to achieve a great macrobenchmark performance even without tuning the microbenchmarks. Some ideas of the paper remained undefined, for instance, how to achieve fairness between two competing library operating systems. The exokernel design in its extremity did not quite find its place in production. Many of their baselines are today outof-use systems and their hardware is not comparable with today s hardware. However, many exokernel design principles can still be found in later research projects, as we will see in the following chapters. It is a promising design for specialized environments, for instance data centers. III. ARRAKIS The Arrakis operating system addresses the problem mentioned in the first chapter, the performance gap between hardware and software. Arrakis tackles the problem not only for network services, but also for disk I/O operations. It is no longer true that disks are terribly slow after introducting flash-backed DRAM onto the devices. Therefore, the same issue pops out - the OS I/O overhead is not negligible any more. Obviously influenced by the exokernel design, Arrakis follows the idea of moving the operating system into the user-space libraries. Each application has its own I/O library OS through which the applications can almost directly communicate with I/O devices. The biggest change in the Arrakis design, compared with the Exokenrel, is pushing a big part of the OS functionality into the hardware. The functionality of the kernel is divided into a control plane and a dataplane. The control plane is in charge of access control, resource limits and global naming (virtual file system). The control plane is infrequently invoked to configure the fast dataplane. The dataplane is in charge of

5 EDIC RESEARCH PROPOSAL 5 Fig. 2: Arrakis architecture and hardware support protection and multiplexing of resources, I/O scheduling, I/O processing. The part of the dataplane which is not in hardware is the application and the API. For both networking and storage Arrakis provides backwards compatible POSIX interfaces. The programmers that are willing to change their applications can benefit from extra performance boost when using native zero-copy API for networking (Arrrakis/N) or a library that provides an asynchronous API persistent data structures API on low-latency storage devices. The Arrakis paper describes the changes needed in both hardware and software that would enable such redesign. A. Hardware Support The big hammer of the Arrakis operating system is the hardware I/O virtualization. Arrakis enables direct access to virtualized I/O devices in order to avoid kernel crossings as much as possible. Today (almost) all the aforementioned dataplane functions this can be provided in today s NICs and some of them in RAID controllers. In the Fig. 2 we can see the Arrakis architecture divided into the dataplane and the control plane and the hardware support on which it relies - a NIC that provides multiple virtualized NICs (VNICs), and a storage controller that can emulate regions of the disk into virtual storage areas (VSAs) and that can expose multiple virtual storage interface controllers (VSICs). The SR-IOV (Single-Root I/O Virtualization) technology multiplexes one PCI device into multiple virtual PCI devices with their own registers, descriptor queues and interrupts. Each application is mapped to specific virtual PCI and this is how the multiplexing is done. To stop the applications from the arbitrary I/O networking applications, the packet filters in the NIC can be arbitrarily programmed to filter incoming and outgoing traffic. Some NICs can already provide filtering based on rich semantics, although most of commodity NICs (including their Intel 82599) still do not have it. Using the IOMMU technology the devices can be restricted to access only certain memory locations. The I/O scheduling can as well be done in the hardware using NIC rate limiter and packet schedulers. The storage part is somehow less hardware supported. The storage controllers have some parts of the technology needed for the Arrakis model, but the required protection Fig. 3: Average memcached transaction throughput and scalability on Arrakis mechanism for restricting to access only their VSA is missing. B. Results The authors evaluated the Arrakis operating system on four cloud application workloads, all using UDP or only IP protocol. Here we will briefly present memcached benchmark. In the Fig. 3 we can see the networking stack scalability across multiple cores running memcached in-memory keyvalue store. The benchmark has 90% GET and 10% SET requests. The original memcached on Arrakis POSIX interface outperforms Linux by 1.7x on one core and 3x on 4 cores. A single multithreaded memcached instance has approximately the same throughput as the multiple memcached processes. The evaluation does not show any memcached latency numbers, it is using UDP instead of TCP and it is not clear what was the experiment setup. Therefore it is difficult to compare it with results from other papers. Among other benchmarks, it is interesting to mention Redis, single threaded NoSQL store that was extended with Arrakis persistent structures library. It reduced the latency of GET requests by 65%, SET requests by 81%, while the throughput increased by 1.75x for GET and 9x for SET requests. The experiment included 1600 connections from 16 clients. They ran a separate benchmarks for GET and for SET operations. It is not clear how Arrakis behaves in the case of high connection count. C. Discussion We will analyze how Arrakis demonstrates the important properties of the Exokernel architecture [2]: 1) Exokernels can be made efficent due to the limited number of simple primitives they must provide. The job of the kernel in Arrakis is not to be extremely efficient in the terms of microbenchmarks. Its job is to stay away and infrequently configure the hardware. 2) Low-level secure multiplexing of hardware resources can be provided with low overhead. Yes, Arrakis efficiently multiplexes hardware resources, and it does that using the hardware itself. It is difficult to be more efficient than that.

6 EDIC RESEARCH PROPOSAL 6 ring 3 ring 0 non-root ring 0 vmx-root sshd... IXCP Dune Linux C C httpd libix IX C Fig. 4: IX architecture memcached libix IX C C 3) Traditional OS abstractions can be implemented efficiently at the application level. Yes, Arrakis provides traditional abstractions at the application level. For instance, the Arrakis POSIX interface. 4) Applications can create special-purpose implementations of abstractions. Yes, Arrakis provides the persistent structures library which is an enhancement of the traditional file abstraction. To sum up, Arrakis leveraged the old design and the new hardware to remove the kernel out of the data path. This design will be more even significant as the hardware keeps improving. Designing the interface for both networking and storage library operating system is a major contribution of the paper. IV. IX IX operating system [7] removes overheads that the Webscale applications usually face on Linux-based commodity operating systems and does not trade-off latency (tail or average), throughput nor protection. The IX system architecture consists of a Linux kernel, Dune kernel module, IX dataplanes, user level library libix and applications on top of IX (see Fig. 4). Similarly to Arrakis, IX separates the control plane (IXCP in Fig. 4), in charge of multiplexing and scheduling resources among dataplanes, from the fast dataplane, which runs the networking stack and the application logic. A. Dune Dune [6] provides the virtualization of processes rather than machines and gives the ability of a direct and safe access to privileged hardware features. The provided features are privilege modes, virtual memory registers, page tables, and interrupt, exception and system call vectors. Dune extends the Linux kernel but does not require its changes. The Dune system consists of a loadable kernel module and a user-level library, libdune. Dune uses Intel VT-x technology [13], a virtualization extension to x86 ISA, to provide user programs with full access to x86 protection hardware with less overhead than trapping and emulating privileged instructions [9]. VT-x adopts a design where the CPU is split into two operating modes: VMX root (host) mode and VMX non-root (guest) mode. Privilege modes (or privilege rings) exposed to user-level applications enable the code of Dune processes to run in VMX non-root ring 0 (guest kernel mode) or in VMX non-root ring 3 (guest user mode) for untrusted code. Those features are leveraged by the Dune sandbox application. The sandbox restricts the memory that the applications inside it can access and it restricts the interfaces or system calls they can use. Linux kernel and Dune kernel module run in host VMX root ring 0 (host kernel mode) that is typically used for VMMs. Dune applications can either run in VMX non-root ring 0 or VMX non-root ring 3. VMX non-root ring 0 enables fast and safe access to hardware like traditionally guest kernels do. The untrusted applications can be restricted like VM applications through VMX non-root ring 3. B. IX Design IX separates the control plane from the dataplane and enables implementing resource allocation policies in control plane while doing efficient network processing in dataplane. IX eliminates the trade-off between the high packet rates and low latency while retaining the same protection model as commodity operating systems. It runs the dataplane kernel and the application at distinct protection levels, and it isolates the control plane from the dataplane. This three way isolation is achieved through Dune and hardware virtualization. IX leverages the privilege modes exposed through Dune. Each IX dataplane runs as a Dune processes in the sandbox application in VMX non-root ring 0, while the untrusted applications run in VMX non-root ring 3 (see Fig. 4). The execution model of the IX networking stack is optimized for both throughput and latency, but it is applicable only for event-driven applications. In the Fig. 5 we can see the event loop of packet processing. Each packet runs to completion through both the application and the networking stack, which improves the instruction cache locality. Packets are not always processed one by one. In the presence of congestion, the packets are taken from the networking queue in batches up to predetermined maximum batch size. This optimizes for data cache locality and low latency. There are no blocking operations in this event loop which enables high throughput. The batched system calls generated by the application are supposed to be non-blocking system calls. If the application wants to execute a blocking system call, it has to do it in the background using dedicated background threads. C. Results The IX operating system was compared against Linux running the in-memory key-value store memcached and against Linux and mtcp, the state-of-the-art user-level networking stack. Here we present a part of the results. The Fig. 6 shows

7 EDIC RESEARCH PROPOSAL 7 ring 0 non-root Latency (µs) ring Event Conditions tcp/ip app libix 5 4 tcp/ip timer adaptive batch Fig. 5: IX execution model Batched Syscalls ETC: Throughput (RPS x 10 3 ) Fig. 6: Average (full line) and 99th percentile (dashed line) latency for Linux (red) and IX (black) the memcached result running a Facebook s TCP benchmark. We can see that the maximal throughput before the latency shoots up is 3x bigger than for Linux. The 99th percentile stays very close to the average latency. Another interesting result is the high connection scalability. The Fig. 7 shows the results of the experiment with 18 multithreaded clients with each thread repeatedly performing a 64B RPC to the server. It shows the throughput as a function of the variable number of active connections. At the peak IX performs 10x better than Linux and it retained half of the peak Messages/sec (x 10 6 ) IX-40Gbps IX-10Gbps Linux-40Gbps Linux-10Gbps Connection Count (log scale) Fig. 7: IX connection scalability 6 SLA throughput even with 250,000 connections on the 4x10GbE. D. Discussion Now let us compare how IX demonstrates the important properties of the Exokernel architecture: 1) Exokernels can be made efficient due to the limited number of simple primitives they must provide. IX dataplane is in fact a specialized networking library OS, whereas the Dune module plays the role of the exokernel. The Dune module nicely follows this property - it efficiently exposes hardware primitives through the libdune interface. 2) Low-level secure multiplexing of hardware resources can be provided with low overhead. Hardware resources are allocated in coarse-grained manner. Whole networking queues and processing cores are allocated to a single dataplane running a single application. Therefore, multiplexing of resources is avoided as much as possible. 3) Traditional OS abstractions can be implemented efficiently at application level. Not very relevant for IX. The IX API departs from the POSIX API, but the libix userlevel library includes an event-based API similar to the popular libevent library [18]. For the missing traditional OS abstractions (the file system for instance) IX can make system calls into the Linux kernel. 4) Applications can create special-purpose implementations of abstractions. This is the main feature of IX, a datacenter-specific networking stack. V. RESEARCH PROPOSAL The main characteristics of dataplane operating systems are a clean separation between the dataplane and the control plane. Dataplane is in charge for actual packet processing, whereas control plane is allocating and managing resources for each dataplane. Docker platform does not explicitly assert that their design follows the dataplane/control plane separation, but it exists implicitly. Each container has its own networking stack with allocated resources and as such it is analog to dataplane in IX. Docker daemon manages the containers and implements resource allocation policies as the control plane does in dataplane operating systems. The intuition and the core of our proposal is to reuse the Docker framework and leverage its utilities to manage dataplanes. In other words, we would like to use Docker as a control plane and run dataplanes in Linux containers. This analogy is depicted in Fig. 8. In this write-up we presented the Exokernel paper and two its successors, Arrakis and IX, in order to analyze and compare different approaches to the same problem. We believe that IX is more feasible for the scenario above. The motivation lays in the fact that IX is an operating system and a Linux process at the same time. It originates from the duality of the Dune kernel module which enables virtualization of a process rather than virtualization of a machine. If we consider IX to be just an application, it should be possible to pack it and run it using Docker. Dune would run in the regular Linux kernel, but it can be exposed as a device to the containers.

8 EDIC RESEARCH PROPOSAL 8 Fig. 8: Combining Docker containers and IX dataplanes Fig. 9: Combining containers and virtual machines Why would one want such design? The reasons are several: REFERENCES [1] D.R. Engler, M.F. Kaaashoek, Exterminate all operating system abstractions, HOTOS, [2] D.R. Engler, M.F. Kaaashoek, and J. O Toole Jr., Exokernel: An Operating System Architecture for Application-Level Resource Management, SOSP, [3] M.F. Kaaashoek, D.R. Engler, G.R. Ganger, H.M. Briceno, R. Hunt, D. Mazieres, T. Pinckney, R. Grimm, J. Jannotti, and K. Mackenzie, Application Performance and Flexibility on Exokernel Systems, SOSP, [4] G.R. Ganger, D.R. Engler, M.F. Kaaashoek, H.M. Briceno, R. Hunt, and T. Pinckney, Fast and Flexible Application-Level Networking on Exokernel Systems, TOCS, Volume 20 Issue 1, February [5] S. Peter, J. Li, I. Zhang, D.R.K. Ports, D. Woos, A. Krishnamurthy, T. Anderson, and T. Roscoe, Arrakis: The Operating System is the Control Plane, OSDI, [6] A. Belay, A. Bittau, A. Mashtizadeh, D. Terei, D. Mazieres, C. Kozryrakis, Dune: Safe User-Level Access to Privileged CPU Features, OSDI, [7] A. Belay, G. Prekas, A. Klimovic, S. Grossman, C. Kozyrakis, and E. Bugnion, IX: A Protected Dataplane Operating System for High Throughput and Low Latency, OSDI, [8] J.H. Saltzer, D.P. Reed, and D.D. Clark, End-To-End Arguments in System Design, TOCS, Volume 2 Issue 4, November [9] G.J. Popek, R.P. Goldberg, Formal Requirements for Virtualizable Third Generation Architectures, CACM, Volume 17 Issue 7, July [10] Docker: [11] Understanding Docker Security (May 2005), blog.docker.com/2015/05/understanding-docker-security-and-bestpractices/ [12] LXC: linuxcontainers.org [13] R. Uhlig, G. Neiger, D. Rodgers, A. Santoni, F. Martins, A. Anderson, S. Bennett, A. Kagi, F. Leung, and L. Smith, Intel Virtualization Technology, Computer, 38(5):48 56, May [14] R. Love, Linux Kernel Development (3rd Edition), ISBN-13: [15] NAPI: [16] A. Chankhunthod, P.B. Danzig, C. Neerdaels, M.F. Schwartz, K.J. Worrell, A Hierarchical Internet Object Cache, ATEC, [17] NCSA: en.wikipedia.org/wiki/ncsa HTTPd [18] libevent: libevent.org Easier deployment and configuration of dataplanes using sophisticated Docker tools. We want to run and configure the IX operating system using a single Dockerfile and use Docker s resource management features to extend the Python control plane. IX execution model could bring benefit of high throughput and low latency to the applications running in the containers. However, this design considers only event-driven applications. Enhanced isolation with hardware virtualization using Dune. The isolation of containers is often discussed problem. Considering the extensive research on container security and a very recent paper from Docker [11], the only way to guarantee the same level of isolation between containers as between virtual machine is - to run them inside virtual machines (see Fig. 9 from the same paper). However, using the Dune module and hardware virtualization to isolate networking stacks, we could guarantee the same level of isolation as with virtual machines, but with less overhead.