VALE: a switched ethernet for virtual machines

Transcription

1 L < > T H local VALE VALE -- Page 1/23 VALE: a switched ethernet for virtual machines Luigi Rizzo, Giuseppe Lettieri Università di Pisa Motivation Make sw packet processing cope with ever increasing hw speeds netmap -- fast packet I/O QFQ -- high speed fair queueing scheduler DXR -- fast IPv4 lookup Virtualization is the next step consolidation and better use of hw resources VM performance not too far from bare metal We focus on network I/O

2 The VALE switch functionally equivalent to linux bridge (or simplified OVS) implements the netmap API over 18 Mpps for 64-byte frames, and over 70 Gbit/s with 1500-byte frames available for Linux and FreeBSD as add-on kernel module dynamic creation of switch instances and ports qemu/kvm backend The rest of the paper is structured as follows:... Background VALE architecture and performance VM interaction Conclusion and future work

3 Background OSes normally run on "bare metal" With virtualizazion Guest OSes run in an imaginary world hypervisor (VMM) emulates a virtual machine frontend emulates guest devices backend talks to the actual physical hardware virtual switch interconnects the virtual machines Switch technologies and performance native bridge (linux, BSD, OVS) UDP multicast encapsulation Virtual Machine Device queue (VMDq), physical switch support Proprietary solutions 0.5-1Mpps, and up to Gbit/s for HW supported solutions

4 Netmap key ideas Efficient I/O API: simplified packet format reduced system call overhead (batching) Standard system calls: open(), ioctl() for configuration ioctl(), poll() for send and receive libpcap emulation Line rate on 10 Gbit/s (14.88 Mpps) at 0.9 GHz Creating a Virtual Local Ethernet VALE extends the netmap API to virtual interfaces dynamically create switch instances and ports ports named valex:ywhere X is the switch instance, Y is the local port name each switch runs the Learning Bridge algorithm

5 Operation VALE operation is sender-driven: each incoming packet is dispatched to the correct destination (zero, one or more) writes are non-blocking, packets dropped when input queue full reads are blocking the cost of forwarding/copying is charged to the sender. Basic performance A straightforward implementation (one packet at a time) reaches 5 Mpps much better than native bridging but can still be improved by a large factor locking overhead and cache misses are the performance bottlenecks

6 Multi-stage forwarding Build batches of packets of bounded maximum size prefetch payload, reducing cache misses control maximum latency Compute destinations for each packet in the batch allow further batching during the next stage Forward traffic iterating on interfaces one lock/unlock per interface Mpps with minimum-size frames Raw switch performance Performance analysis of the VALE switch in various conditions: packet size number of ports batch size (on sender, receiver and kernel side) Comparison is between VALE, our switch NIC, hardware-assisted switching (using PCI passthrough and NIC virtual queues, and/or external switch support) TAP and linux bridging (OpenvSwitch shows similar performance)

7 Throughput versus number of replicas Throughput decreases as the number of replicas of each packet grows. VALE and native bridging are CPU/memory bound; NIC is mostly limited by bus speed VALE is faster than hardware-supported solutions and much faster than linux/ovs bridging Processing time versus packet size Packet processing time (lower is better) grows with packet size. VALE operates at memory speed ( Gbit/s) TAP operates almost at memory speed, but has higher per-packet overhead NIC operates at link/bus speed, hence limited to Gbit/s

8 Effect of batching Batching occurs in three places transmit side: controlled by the sender, can amortize syscall and locking costs during forwarding receive side: controlled by the receiver, can amortize syscall costs kernel side: controlled by the sysadmin, guarantees bounded forwarding latency, can amortize locking costs during forwarding. The tradeoff is between efficiency and latency the kernel batching makes sure we can control the forwarding delay ( ns/pkt depending on the size) Throughput versus receive batch TX and kernel batch = 1024 packets, single output port even short receive batches give high speed.

9 Throughput versus transmit batch kernel batch = 1024, different results depending on rx batch rx-batch = 1, throughput limited by the receive side rx-batch = 1024, growth is slower than with variable rx batch because the receiver is faster than the sender Throughput versus kernel batch TX and RX batch = 1024 growth is slower than with variable rx batch because even in this case the receiver is faster than the kernel

10 Round trip latency Round trip delay for request-response over VALE these times are dominated by OS overheads VALE is slightly better than native bridging VALE and VM performance The switch is no more a bottleneck in any condition helps identifying other bottlenecks Huge dependency on other components: client (netsend - socket based, or pkt-gen - netmap based) frontend (virtio or e1000) hypervisor (QEMU or KVM) backend (TAP or VALE) Not all configurations tested (due to incompatibilities or bugs) ongoing work to support more configurations

11 VM performance summary Speed, Kpps Configuration TAP VALE TX RX TX RX Raw switch speed e1000, QEMU e1000, KVM fast e1000, KVM virtio, KVM pkt-gen e1000 QEMU pkt-gen e1000 KVM pkt-gen e1000 KVM (1514 bytes) vmxnet3 ESX (VMware docs) Other applications VALE is not limited to VM interconnection. Other applications include: performance testing (interconnecting traffic sources with analyzers, monitors, firewalls...) support for building packet processing chains (with coarse processing modules) Example: netmap enabled ipfw+dummynet has been developed and tested using VALE measured 6.5 Mpps, over 5 times faster than in-kernel version

12 Conclusions and future work VALE is extremely fast and can be used to interconnect VMs as well as native packet processing nodes readily available for Linux and FreeBSD ongoing work to improve KVM/QEMU performance ongoing work to replace the Learning Bridge algorithm with OVS forwarding code (joint work with EC project OpenLab) See for code, papers and more details