Got Loss? Get zovn! Daniel Crisan, Robert Birke, Gilles Cressier, Cyriel Minkenberg, and Mitch Gusat. ACM SIGCOMM 2013, August, Hong Kong, China

Transcription

1 Got Loss? Get zovn! Daniel Crisan, Robert Birke, Gilles Cressier, Cyriel Minkenberg, and Mitch Gusat ACM SIGCOMM 2013, August, Hong Kong, China

2 Virtualized Server 1 Application Performance in Virtualized Datacenter Networks Virtualized Server 2 Virtualized Server 3 Virtualized Server N VM 1 VM K 1 VM 1 VM K 2 VM 1 VM K 3 VM 1 VM K N vnic vnic vnic vnic vnic vnic vnic vnic Virtual Switch Virtual Switch Virtual Switch Virtual Switch NIC NIC NIC NIC Switch Physical Datacenter Network short-and-fat links Switch Switch Switch Router Router Global Internet long-and-fat links End-users accessing datacenter services 2

3 Physical Network: Lossless Links IBM builds flow-controlled links since the 80 s High Performance Computing community - large scale lossless distributed systems Flow control improves performance HPC and Datacenter communities disconnected Why do we disregard the Ethernet flow-control? PAUSE widely available, largely ignored Converged Enhanced Ethernet applies HPC and Storage lessons Priority Flow Control (standardized 2011) Constantly improved for 1T 3

4 Virtual Networks are Different Physical Networks Virtual Networks Packet forwarding Deterministic bandwidth and delay Link level flow control Bandwidth allocation Latency µs ms Virtual Networks in embryonic stage 4

5 Contributions Loss identification and characterization in virtual networks Dirty-slate approach for latency sensitive applications Exploit a L2 technique to the benefit of TCP and application Introduce zero-loss Overlay Virtual Network Flow-controlled virtual switch Evaluation with Partition/Aggregate Prototype implementation Cross-layer simulation Flow control improves application performance 5

6 Outline Introduction Losses in Virtual Networks zovn Architecture Evaluation Conclusions 6

7 VM 1 Losses in Virtual Networks vswitch Physical Machine Source vnic Tx Port A Tx VM 3 VM 2 Port C Rx vnic Rx Sink Source vnic Tx Port B Tx Packets traverse a series of queues Producer/Consumer problem on each queue Not implemented correctly on each queue 7

8 VM 1 Losses in Virtual Networks (2) 1 2 vswitch Physical Machine Source vnic Tx Port A Tx 3 4 VM VM Port C Rx vnic Rx Sink Source vnic Tx Port B Tx 3 Numbers: measurement points Inject UDP packets at (1) Count how many still arrive at (6) Loss locations vswitch between (3) and (4) Receive stack between (5) and (6) 8

9 Injected traffic [MBps] 200 Losses in Virtual Networks (3) Stack Loss vswitch Loss Received 0 C1 C2 C3 C4 C5 C6 C7 Configuration Hypervisor vnic vswitch C1 Qemu/KVM Virtio Linux Bridge C2 Qemu/KVM Virtio Open vswitch C3 Qemu/KVM Virtio VALE C4 H2 N2 S4 C5 H2 E1000 S4 C6 Qemu/KVM E1000 Linux Bridge C7 Qemu/KVM E1000 Open vswitch 9

10 Outline Introduction Losses in Virtual Networks zovn Architecture Evaluation Conclusions 10

11 TX Path VM Application write return value Guest kernel socket Tx enqueue Qdisc free skb start_xmit Hypervisor vnic Tx receive Port B Tx vswitch start/stop queue return value NIC send frame zovn bridge NIC Tx overlay encapsulation Port A Rx Physical link receive PAUSE wake-up 11

12 VM RX Path: Fix Stack Loss Guest kernel Hypervisor vswitch Application read return value socket Rx netif_receive skb pause/resume queue NET RX Softirq vnic Rx send return value Port B Rx setsockopt Select lossy or lossless. NIC receive frame zovn bridge NIC Rx overlay decapsulation Port A Tx Physical link send PAUSE wake-up 12

13 Lossless Virtual Switch Senders: Produce packets Start forwarder Sleep vswitch Receivers: Consume packets Start forwarder Sleep Port 1 Tx Port 2 Tx Port N Tx Forwarder: Move packets from Tx to Rx Pause Tx ports if Rx port full Wake-up Tx ports when something is consumed Port 1 Rx Port 2 Rx Port N Rx 13

14 VM 1 1 Fully Lossless Path 2 vswitch Physical Machine Source vnic Tx Port A Tx 3 4 VM VM Port C Rx vnic Rx Sink Source vnic Tx Port B Tx 3 Fixed vswitch between (3) and (4) Receive stack between (5) and (6) 14

15 Outline Introduction Losses in Virtual Networks zovn Architecture Evaluation Conclusions 15

16 Partition/Aggregate Workload 1 4 Master Worker Worker Worker Worker Problem: TCP incast During Aggregate, buffers might overflow. For short flows: TCP ineffective, ACK clock stalled. Must rely on timeouts. Partition and Aggregate datacenter internal Open to optimizations 16

17 IBM x3550 M4 Server VM 1 VM 16 Testbed Setup IBM x3550 M4 Server VM 1 VM 16 IBM x3550 M4 Server VM 1 VM 16 IBM x3550 M4 Server VM 1 VM 16 vswitch vswitch vswitch vswitch 10G 1G 10G 1G 10G 1G 10G 1G Data network IBM G G Switch Control network HP G 1G Switch 4x Rack Servers 16 physical cores + HyperThreading Intel 10G adapters (ixgbe drivers) 16 VMs / server 8 VMs for PA traffic* 8 VMs produce background flow * as in DCTCP: Efficient Packet Transport for the Commoditized Data Center SIGCOMM

18 Mean completion time [ms] 1000 Testbed Results (CUBIC) LL LZ ZL Virtual Network Flow Control No No Yes Physical Network Flow Control No Yes No ZZ Yes Yes Response size [Packets] Virtual only better than physical only: vswitch primary congestion point. Physical switch congestion negligible No improvement for short/long flows: Long transfers can remain on lossy priorities 18

19 Simulation Setup Larger topology: 256 servers 4 VMs / server 3 VMs produce PA traffic 1 VM background flows Assumption: infinite CPU 19

20 Mean completion time [ms] Simulation Results (64 packets) NewReno Vegas Cubic LL LZ ZL ZZ Virtual Network Flow Control No No Yes Yes Physical Network Flow Control No Yes No Yes Confirm findings from prototype experiments (LZ) Physical only flow control: shift the drop point into the virtual network (ZZ) Both flow controls required for better performance 20

21 Faster CPUs or faster networks? Loss ratio influenced by CPU/network speed ratio TX Slow CPU coupled with a fast network is desirable e.g. Xeon + 1G network drops more than Core2 + 1G network RX Fast CPU coupled with a slow network is desirable e.g. Xeon + 10G network drops more than Xeon + 1G network Conflicting requirements: cannot solve problem by changing hardware The only solution: add flow control! 21

22 Conclusions Loss identification and characterization in OVN First flow-controlled vswitch for future Overlay Virtual Networks Dirty-slate approach for latency sensitive applications Un-tuned TCP Commodity 1-10G Ethernet fabric Result replication trivial Orthogonal to other proposals Lossless links: Order of magnitude completion time reduction in Partition/Aggregate 22

23 Backup 23

24 Encapsulation in Overlay Virtual Networks Destination Server VM VM VM (5) vswitch Cache (4) (2) Physical Network Fabric Controller (3) Source Server VM VM VM vswitch Cache (1) Workflow Payload TCP IP Eth Encap UDP IP Eth 1. Source VM sends packet to its attached vswitch. 2. vswitch queries the Controller to find the address of the destination. 3. Controller answers. The information is cached by the switch. 4. Packet sent over physical network encapsulated with new headers. 5. Packet decapsulated at destination virtual Switch. 24