Evolution of the netmap architecture

Size: px

Start display at page:

Download "Evolution of the netmap architecture"

Emily Barnett
5 years ago
Views:

L < > T H local Evolution of the netmap architecture Evolution of the netmap architecture -- Page 1/21 Evolution of the netmap architecture Luigi Rizzo, Università di Pisa http://info.iet.unipi.

1 L < > T H local Evolution of the netmap architecture Evolution of the netmap architecture -- Page 1/21 Evolution of the netmap architecture Luigi Rizzo, Università di Pisa Starting point these slides at Packet I/O in commodity OS is not fast enough typically ~1 Mpps/core, vs Mpps on 10 Gbit/s links insufficient for software packet processing nodes several ad-hoc or proprietary solution (Click-based, PacketShader, DPDK) We tried to come up with a more generic solution. Ideally: as fast as (or better than) others device-independent, OS independent developed friendly Netmap (2011) times faster than raw sockets, PF_PACKET, bpf Mpps with 1 core at 900 MHz pcap library on top of netmap as of 2012, also available on Linux

2 Next, get rid of hardware Use netmap key ideas for virtual switches simplify experiments with high speed network applications useful to interconnect VMs VALE (2012) Virtual Local Ethernet implenting ethernet learning bridge up to 20 Mpps (64 byte frames) or 70 Gbit/s (1500 byte frames) And then, how fast networking is in VMs Focus is mostly on bulk TCP emulation of network devices very poor paravirtualized devices (vendor specific) reasonably fast for TCP Gbit/s using tricks (good ones) We want to deal with high packet rates, too SDN must be implemented, not just Defined some applications have high pps requirements Fast QEMU (2013) accelerated network I/O path in qemu paravirtualized e1000 guest-guest rates on top of e1000: >1 Mpps with sockets, 5 Mpps with netmap More at

Availability netmap and VALE are in standard FreeBSD distributions (HEAD and stable/9) same code also runs on Linux as an add-on module support for multiple NICs (Intel, Realtek, Mellanox, Nvidia)

3 Availability netmap and VALE are in standard FreeBSD distributions (HEAD and stable/9) same code also runs on Linux as an add-on module support for multiple NICs (Intel, Realtek, Mellanox, Nvidia) QEMU enhancements submitted to QEMU-dev Netmap design principles Design principles: no requirement/reliance on special hardware features amortize costs over large batches (syscalls) remove unnecessary work (copies, mbuf alloc/free) reduce runtime decisions (a single frame format) modifying device drivers is permitted, as long as the code can be maintained

ring poll()and select()are used for synchronization POLLIN and POLLOUT select the rings to monitor Netmap performance 65 cycles/packet between the wire and userspace (14.

4 Netmap data structures and API Access open("/dev/netmap") ioctl(fd, NIOCREG, arg)--> disconnect datapath from the OS; mmap(..., fd, 0) Transmit fill buffers, update netmap ring ioctl(fd,nioctxsync)queues the packets Receive ioctl(fd,niocrxsync)reports newly received packets process buffers, update netmap ring poll()and select()are used for synchronization POLLIN and POLLOUT select the rings to monitor Netmap performance 65 cycles/packet between the wire and userspace (14.88 Mpps at 900 MHz) good scalability with CPU frequency and number of cores Ported several apps (OVS, Click, ipfw,...) with 5-10x speedup netmap exposes bottlenecks in applications some kernel functions may now run more efficiently in userspace than in the kernel

VALE, a high performance Virtual Local Ethernet NIOCREG valex:ydynamically creates switch instances (X) and ports (Y) same API as physical netmap ports, but separate memory

are non-blocking, packets dropped when destination queue full reads are blocking the cost of forwarding/copying is charged to the sender.

5 VALE, a high performance Virtual Local Ethernet NIOCREG valex:ydynamically creates switch instances (X) and ports (Y) same API as physical netmap ports, but separate memory regions each switch runs the Learning Bridge algorithm (and now, also OVS) Operation is sender-driven: each incoming packet is dispatched to the correct destination(s) writes are non-blocking, packets dropped when destination queue full reads are blocking the cost of forwarding/copying is charged to the sender. VALE performance Multi-stage forwarding to amortize locking and cache miss costs 1. Fetch a batch of packets, prefetch payload 2. Compute destinations for each packet in the batch 3. Forward traffic iterating on interfaces

following: mmio 100 times more expensive than on bare metal, due to VM exits interrupts can be expensive VMM/host/switch

6 VALE Performance VALE: our software switch NIC: switch/nic-supported forwarding TAP: linux/bsd bridge, in-kernel OVS VALE achieves Mpps / 70 Gbit/s Virtual Machine network performance Emulation of network peripherals historically slow, due to the following: mmio 100 times more expensive than on bare metal, due to VM exits interrupts can be expensive VMM/host/switch performance Paravirtualized peripherals (virtio, vmxnet, xenfront) address only one part of the problem: VMM, host and switch can still be limiting

Our work A set of host, guest, VMM modifications that can be used depending on operational constraints 1. proper interrupt moderation (VMM only) 2. send combining (guest only) 3.

7 Our work A set of host, guest, VMM modifications that can be used depending on operational constraints 1. proper interrupt moderation (VMM only) 2. send combining (guest only) 3. e1000 paravirtualization (host-guest) btw, FreeBSD's DEVICE_POLLING (Rizzo 2001) is almost as good as paravirtualization! 4. use VALE instead of linux bridge (host, VMM) 5. clean up the backend-frontend datapath (VMM) Send combining Send combining reduces VM exits on TX: when there are pending interrupt, postpone transmissions until the next interrupt arrives especially useful when combined with moderation IMPLEMENTATION: ~100 lines, only in the guest device driver PERFORMANCE: TX rate, Kpps (guest-guest, TAP backend) Function 1 VCPU 2 VCPU moderation: 24 -> > 87 send comb.: 24 -> > 301 both: 24 -> > 334

e1000 paravirtualization Reduce VM exits using a shared memory mailbox (CSB) between the VCPU and the IO thread the first I/O or interrupt acts as a "kick" to wake up a process on the other side;

8 e1000 paravirtualization Reduce VM exits using a shared memory mailbox (CSB) between the VCPU and the IO thread the first I/O or interrupt acts as a "kick" to wake up a process on the other side; later, guest driver and I/O thread chase each other exchanging read/write position through the CSB; Data buffers and descriptors are in shared memory (guest physical, host virtual) IMPLEMENTATION no need to create a brand new device, easy addon for any modern NIC ~100 lines in the guest driver, ~100 lines in the frontend e1000 paravirtualization performance Almost completely remove kicks and interrupts as packet rate increases 24 -> 492 Kpps (1 VCPU), 65 -> 507 Kpps (2 VCPU) performance equivalent to that of virtio does not incur the latency of interrupt moderation reliability/portability/stability advantages

$limited to ~3 Mpps even before going through the switch removed unnecessary address translation optimized copy routine Current code can drive the switch at 10 Mpps Guest-guest, TAP, sockets \ normal$

9 Cleaning up the hypervisor datapath Data moves from frontend -> backend -> switch access guest descriptors and buffers copy to hypervisor memory (backend) pass to the switch Initial throughput was limited to ~3 Mpps even before going through the switch removed unnecessary address translation optimized copy routine Current code can drive the switch at 10 Mpps Guest-guest, TAP, sockets \ normal send_combining paravirt. itr \ 1 cpu \ 2 cpu 1 cpu 2 cpu 1 cpu 2 cpu

10 Guest-guest, VALE, sockets \ normal itr \ 1 cpu 2 cpu send_combining 1 cpu 2 cpu paravirt. 1 cpu 2 cpu \ (Black: TAP, Red: VALE; ~500 Kpps with 1500-byte frames) ~5 Mpps or 25-Gbit/s with netmap within the guest. Evolution of the netmap architecture After this experience, we decided to add features that proved useful, but trying to avoid feature bloat or additions that impact performance. "transparent mode" for netmap: packets not marked by user application are sent to the other side direct connection between VALE and NIC (as a real host bridge) programmable destination lookup function in VALE (now we can hook to OVS) indirect buffers and scatter-gather I/O in netmap (optimize bulk I/O)

11 Acknowledgements Many contributions come from colleagues and former students: Matteo Landi, Gaetano Catalli, Marta Carbone, Giuseppe Lettieri, Vincenzo Maffione, Michio Honda Funding from EU Projects (CHANGE, OPENLAB), and companies (Netapp, Google).

VALE: a switched ethernet for virtual machines

L < > T H local VALE VALE -- Page 1/23 VALE: a switched ethernet for virtual machines Luigi Rizzo, Giuseppe Lettieri Università di Pisa http://info.iet.unipi.it/~luigi/vale/ Motivation Make sw packet processing