Enabling Fast, Dynamic Network Processing with ClickOS

Transcription

1 Enabling Fast, Dynamic Network Processing with ClickOS Joao Martins*, Mohamed Ahmed*, Costin Raiciu, Roberto Bifulco*, Vladimir Olteanu, Michio Honda*, Felipe Huici* * NEC Labs Europe, Heidelberg, Germany University Politehnica of Bucharest firstname.lastname@neclab.eu, firstname.lastname@cs.pub.ro

2 The Idealized Network Application Application Transport Transport Network Page 2 Network Network Datalink Datalink Datalink Datalink Physical Physical Physical Physical

3 A Middlebox World ad insertion WAN accelerator BRAS carrier-grade NAT transcoder IDS session border controller load balancer DDoS protection firewall QoE monitor Page 3 DPI

4 Hardware Middleboxes - Drawbacks Middleboxes are useful, but Expensive Difficult to add new features, lock-in Difficult to manage Cannot be scaled with demand Cannot share a device among different tenants Hard for new players to enter market Clearly shifting middlebox processing to a software-based, multi-tenant platform would address these issues But can it be built using commodity hardware while still achieving high performance? ClickOS: tiny Xen-based virtual machine that runs Click Page 4

5 Click Runtime Modular architecture for network processing Based around the concept of elements Elements are connected in a configuration file A configuration is installed via a command line executable (e.g., click-install router.click) An element Can be configured with parameters (e.g., Queue::length) Can expose read and write variables available via sockets or the /proc system under Linux (e.g., Counter::reset, Counter::count) Compiled 262/300 elements Programmers can write new ones to extend Click runtime Page 5

6 A simple (click-based) firewall example in :: FromNetFront(DEVMAC 00:11:22:33:44:55, BURST 1024); out :: ToNetFront(DEVMAC 00:11:22:33:44:55, BURST 1); filter :: IPFilter( allow src host && dst host && udp, drop all); in -> CheckIPHeader(14) -> filter filter[0] -> Print( allow ) -> out; filter[1] -> Print( drop ) -> Discard(); Page 6

7 What's ClickOS? domu ClickOS apps Click guest OS mini OS paravirt paravirt Work consisted of: Build system to create ClickOS images (5 MB in size) Emulating a Click control plane over MiniOS/Xen Reducing boot times (roughly 30 miliseconds) Optimizations to the data plane (10 Gb/s for almost all pkt sizes) Page 7

8 Performance analysis pkt size (bytes) 10Gb rate Mp/s Mp/s Mp/s Mp/s Mp/s Driver Domain (or Dom 0) netback NW driver Linux/OVS bridge 1500 vif Xen bus/store 810 Kp/s ClickOS Domain netfront Event channel Click FromNetfront ToNetfront Xen ring API (data) 300* Kp/s 350 Kp/s 225 Kp/s * - maximum-sized packets Page 8

9 Main issues Backend switch ( bridge / openvswitch ) are slow Copying pages between domains (grant copy) greatly affects packet I/O These are done in batches, but still expensive Packet metadata (skb or mbufs) allocations MiniOS netfront not as good as Linux 225 Kpps VS 430 Kpps Tx only 8 Kpps Rx Page 9 NEC Corporation 2009

10 Optimizing Network I/O Backend Switch ClickOS Domain Driver Domain (or Dom 0) NW driver (netmap mode) netback netfront Xen bus/store VALE port Event channel Click FromNetfront ToNetfront Xen ring API (data) Introduce VALE as the backend switch NIC switches to netmap-mode Slight modifications to the netback driver only Batch more I/O requests through multi-page rings Removed packet metadata manipulation 625 Kpps (1500 size, 2.7x improvement) and 1.2 Mpps (64 size, 4.2x improvement) Page 10

11 Background - Netmap Fast packet I/O framework Mpps on 1 core at 900 Mhz Available in FreeBSD 9+ Also runs on Linux Minimal device driver modifications Critical resources (NIC registers, physical buffer addresses, and descriptors) not exposed to the user NIC works in special mode, bypassing the host stack Amortize syscalls cost by using large batches Preallocated packet buffers, and memory mapped to userspace Netmap a novel framework for fast packet I/O Luigi Rizzo Universita di Pisa Page 11

12 Background - VALE Software Switch High performance switch based on netmap API (18 Mpps between virtual ports, one CPU core) Packet processing is modular Default as learning bridge Modules are independent kernel modules Applications use the netmap API Page 12 VALE, a Virtual Local Ethernet Luigi Rizzo, Giuseppe Lettieri Universita di Pisa

13 Optimizing Network I/O ClickOS Domain Driver Domain (or Dom 0) netfront netback NW driver VALE Click Xen bus/store TX/RX Event channels FromNetfront ToNetfront Netmap API (data) No longer need the extra copy between domains Netmap rings (in the VALE switch) are mapped all the way to the guest An I/O request doesn't require a response to be consumed by the guest Event channels are used to proxy netmap operations from/to guest and VALE Breaks other (non-minios) guests :( Page 13 But we have implemented a netmap-based Linux netfront driver

14 Optimizing Network I/O Initialization and Memory usage Netmap buffers are contiguous pages in guest memory KB # grants slots (per ring) (per ring) Driver Domain VALE Buffers are 2k in size, each page fits 2 buffers Ring fits 1 page for 64 and 128 slots; (2+ for 256+ slots) buf slot [0] buf slot [1] buf slot [2] netmap buffers pool Vale Mini-OS Netback (Xen) 3. ring/bufs pages granted netfront app. netmap API netback 1. opens netmap device 2. registers a VALE port Initialization 4. ring grant refs read from the xenstore buffer refs read from the mapped ring slot

15 Optimizing Network I/O Synchronization In netmap application, operation is done in sender context Backend/Frontend private copy not included in the shared ring page(s) Event channels used for synchronization Domain-0 VALE Vale buf slot 0 buf slot 1 buf slot 2 (mapped) Guest (Mini-OS) Netback (Xen) buf slot 0 buf slot 1 buf slot 2 Packets to transmit netfront netback Backend finished TX event channel app

16 EVALUATION

17 ClickOS Base Performance RX Intel Xeon E core 3.2GHz, 16GB RAM, dual-port Intel x520 10Gb/s NIC. One CPU core assigned to VM, the rest to dom0 TX

18 Scaling out Multiple NICs/VMs Intel Xeon E core 3.2GHz, 16GB RAM, dual-port Intel x520 10Gb/s NIC. 3 cores assigned to VMs, 3 cores for dom0

19 Linux Guest Performance

20 ClickOS (virtualized) Middlebox Performance

21 ClickOS Delay vs. Other Systems

22 Conclusions Presented ClickOS: Tiny (5MB) Xen VM tailored at network processing Can be booted (on demand) in 30 milliseconds Can achieve 10Gb/s throughput using only a single core. Can run a varied range of middleboxes with high throughput Future work: Improving performance on NUMA systems High consolidation of ClickOS VMs (thousands) Service chaining Page 22

23