Handles all kinds of traffic on a single network with one class

Transcription

1

2 Handles all kinds of traffic on a single network with one class No priorities, no reservations required Quality? Throw bandwidth at the problem, hope for the best 1000x increase in bandwidth over 2 decades

3 Biggest benefit: Slack For scheduling at switches, for forwarding engines Allows distributed protocols like TCP time to figure out path bandwidth Of course, to handle temporary oversubscriptions Unfortunate consequence Packet switching requires buffers How much buffering? How much buffering? As much as can be put on chip/linecard

4 Latency: We love circuit switching s low latency Very predictable delays: just switching + propagation No random, traffic dependent queuing delay Queuing delays: some numbers Memory per port 10G switches ~ 2MB and above Queuing delay for a TCP packet = 1.6 milliseconds Switching time ~ nanosecs!

5 Bandwidth delay product rule of thumb: A single flow needs C RTT buffers for 100% Throughput. B = C RTT uffer Size Bu B B > C RTT Buff fer Size B B < C RTT Buf ffer Size B Throughput loss! More latency!

6 Widespread conception: increase link speed to reduce latency Eg. upgrade from 1Gbps to 10Gbps network. However, increasing link speed doesn t lower queuing delay, because: Switch buffers also need to be 10 times larger and 10 times faster. Buffer Size 10G x RTT 1G x RTT Time 1G 10G

7 Build a high bandwidth, ultra low latency Interconnect fabric High port density: 10, ,000 end hosts connected Very large bisection bandwidth: 10G fat tree network of commodity switches Reduce latency down to just processing, switching, forwarding No queuing delays Motivated by RAMCloud needs, but we don t assume network has only RAMCloud traffic Need to go after latency everywhere: stack, NIC, network Focus here: network delay

8 Use ECN Make buffer occupancy independent of buffer size Use multi bit congestion signals QCN and DCTCP h l bf Use phantom queues : signal congestion before congestion

9 Signal congestion with 6 bit message F b instead of 1 bit TCP QCN ndow Size (R Rate) Wi Additive Increase: Time W W+1 per round trip time Multiplicative Decrease: W W/2 per ECN mark ndow Size (R Rate) Wi Time Additive Increase: W W+1 per round trip time Multiplicative Decrease: W W (1 F b ) per congestion msg 9

10 10

11 11

12 12

13 How can we get multi bit signaling in TCP? Use ECN

14 Sender 1 Additive Increase: W W+1 per round trip time Multiplicative Decrease: W W/2 per ECN mark Window Size (Rate) Time ECN Mark (1 bit) Receiver Sender 2 14

15 React in proportion to the extent of congestion, not its presence. Reduces variance in sending rates, lowering queuing requirements. ECN Marks TCP DCTCP Cut window by 50% Cut window by 40% Cut window by 50% Cut window by 5%

16 ytes) (Kby Setup: Win 7, Broadcom 1Gbps Switch Scenario: 2 long lived lived flows, K = 30KB 16

17 Shallow buffer switches: overflow on bursts Deep buffer switches: introduce delay to all packets Bursty Traffic Delay sensitive ii Traffic 17

18 TCP: ~10ms DCTCP: ~100us ~ Zero latency How do we get this?

19 TCP Incoming Traffic C Outgoing Link Switch Queue DCTCP K Incoming C Traffic Outgoing Link Switch Queue Avoiding queue buildup requires congestion signaling before link is 100% utilized. 19

20 Incoming Traffic Switch Queue C Outgoing Link K γc (γ < 1) Phantom Queue 20

21 qlen drop qlen drop Queue Length [Bytes] Num mber of dropped packets Queue Length [Bytes] Num mber of dropped packets time [sec] time [sec] TCP Ave Q: KBytes gth [Bytes] Queue Leng pped packets Number of dro qlen drop DCTCP Ave Q: KBytes time [sec] DCTCP + PQ (95%) Ave Q: 4.44 KBytes 21

22 Host H Core Switch Pacer Module PM PQ PQ Phantom Queue PQ PQ PQ Agg DCTCP Congestion Control PQ PQ PQ PQ PM PM PM PM H H H H

23 1000BaseT NICs/switch 2 9 senders and one receiver Static and dynamics flows Schemes compared: DCTCP or TCP reno Pacer (on/off) No PQ or PQ 800Mbps Different thresholds for marking (3 60KB)

24 Load: 20% Switch 1KB RCT 10MB RCT Latency (us) (us) (ms) Avg 90 th Avg 90 th Avg 90 th DCTCP DCTCP 6KB pacer HULL TCP QoS

25

26 The image part with relationship ID rid3 was not found in the file. The image part with relationship ID rid2 was not found in the file. Throughput: Mbps Avg Q: KBytes 26

27 Total Rate Average Qlen TCP Mbps KB DCTCP Mbps KB DCTCP+P Q (95%) DCTCP+P Q (90%) Mbps 4.44 KB Mbps 1.96 KB 27

28 Large Flows Small Flows Link (with speed C) Host Applicatio on DCTCP CC Large Burst LSO NIC Pacer Switch Empty Queue PQ γ xc ECN Thresh. h

29 Consider the source switch congestion loop Q eq Source Congestion Point (Switch) Dynamics: Sample packets, compute feedback (Fb), quantize Fb to 6 bits, and reflect only negative Fb values back to Reaction Point with a probability proportional to Fb. Fb = -(Q-Q Q eq +w. dq/dt) = -(queue offset + w.rate offset) on ility Reflecti Probabi P max 29 Fb Pmin

30 Source (reaction point): Transmit regular Ethernet frames. When congestion message arrives: perform multiplicative li decrease, fast recovery and active probing. Fast recovery very similar to BIC TCP We show it gives very good performance in high BWxDelay networks TR Fast Recovery CR Target Rate Rate Rd/8 erd/4 Rd Rd/2 Active Probing 30 Current Rate Congestion message recd Time

31 QCN: signal congestion with 6 bit message F b instead of 1 bit like TCP Window Size (Rate) TCP Additive Increase: Time W W+1 per round trip time Multiplicative Decrease: W W/2 per ECN mark Windo ow Size (Rat te) QCN Time Multiplicative Decrease: W W (1 F b ) per congestion msg 31

32 Very useful to cut sending window by factors less than 2 More congestion, more decrease

33 Define multi bit TCP, say n bit TCP Sampling probability Mark value 100% 2 n -1 1% 1 25% 50% 100% Queue occupancy 25% 50% 100% Queue occupancy Update cwnd < cwnd(1 mark_value/2 n+1 ) E.g. with 6 bit TCP, smallest cut is by 127/128 All other features (e.g. additive increase) similar to TCP

34

35 Numb ers ( KB) Sequ uence data seq num 100 ack seq num ideal slope Time (msec)

36 Interrupt Coalescing ACK ratio (KB) CPU (%) adaptive rx frames= rx frames= rx frames= rx frames= rx frames=

37 Adaptive Token Bucket Array Outgoing g Packets From Server NIC Flow Association/ Disassociation Table Hash Function Un paced Traffic R 1 R 2 R m Egress Scheduler TX TCP ACK Parser RX

38 CDF No Pacing With Pacing Inter-packet gap (μs) (a) 4 flows CDF No Pacing With Pacing Inter-packet gap (μs) (b) 8 flows

39 C VQ K γc C (γ < 1)

40 Switch Phantom Queue PQ Counter Packet Flow ECN Modifier Link Output Port

41 Throug gput (Mb bps) Drain Rate (Mbps) (a) Throughput Latency (μs) Switch Drain Rate (Mbps) (b) Switch latency, 90 th percentile

42 Host H Core Switch Pacer Module PM PQ PQ Phantom Queue PQ PQ PQ Agg DCTCP Congestion Control PQ PQ PQ PQ PM PM PM PM H H H H

43 1000BaseT NICs/switch 2 9 senders and one receiver Static and dynamics flows Schemes compared: DCTCP or TCP reno Pacer (on/off) No PQ or PQ 800Mbps Different thresholds for marking (3 60KB)

44 atency (μ μs) Switch L NoPQ Marking Threshold (KB) Switch La atency (μ μs) PQ Marking Threshold (KB)

45 30 NoPQ 30 PQ 800 Switch Latency (μs) Switch Latency (μs) Marking Threshold (KB) Marking Threshold (KB)

46 s) Throughp put (Mbp NoPQ Marking Threshold (KB) s) Throughp put (Mbp PQ 800Mbps Marking Threshold (KB)

47 Poisson requests from receiver 9 senders 16 permanent TCP connections per server Average load: 20% Schemes compared: DCTCP DCTCP 6KB pacer HULL TCP QoS: 2 queues (latency + bulk), strict priority for latency queue

48 Load: 20% Switch 1KB RCT 10MB RCT Latency (us) (us) (ms) Avg 90 th Avg 90 th Avg 90 th DCTCP DCTCP 6KB pacer HULL TCP QoS