Leveraging Multi-path Diversity for Transport Loss Recovery in Data Centers

Transcription

1 Fast and Cautious: Leveraging Multi-path Diversity for Transport Loss Recovery in Data Centers Guo Chen Yuanwei Lu, Yuan Meng, Bojie Li, Kun Tan, Dan Pei, Peng Cheng, Layong (Larry) Luo, Yongqiang Xiong, Xiaoliang Wang, and Youjian Zhao

2 Motivation n Services care about the tail flow completion time (tail FCT) Large number of flows generated in each operation Overall performance governed by the last completed flows App App App App App App App App App App Logic Logic Logic Logic Logic Logic Logic Logic Logic Logic Large-scale web application hosted in Data Center Network (DCN) Responding to a user request 16/6/25 2

3 Motivation n Services care about the tail flow completion time (tail FCT) Large number of flows generated in each operation Overall performance governed by the last completed flows n But packet loss hurts tail FCT Real case in a Microsoft Azure s DCN Spine switch 2% random drop rate --> increase of 99th percentile latency of all users DCN tail latency visualization [Pingmesh (SIGCOMM 15)] 16/6/25 (a) Normal (b) Spine failure 3

4 Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 4

5 Packet Loss in DCN n Loss characteristics Measured in a Microsoft production DCN during Dec. 1 st -5 th, % above ToR Mean loss rate 4% Similar in 5 days Loss rate and location distribution of lossy links (loss rate > 1%) 1) Loss frequently happens (the overall loss rate is low) 2) Most losses happen in the network instead of the edge 16/6/25 5

6 Packet Loss in DCN n Reasons causing loss Congestion loss Ø Ø Uneven load-balance Incast Bursty; Transient Greatly mitigated (e.g., 1%->0.01%) [Jupiter Rising SIGCOMM 15] Failure loss Ø Ø Silent random drop Packet black-hole Complex; Hard to detect Common & Huge impact on performance [Pingmesh SIGCOMM 15] 16/6/25 6

7 Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss Why loss hurts the tail? How hard loss hurts? n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 7

8 How TCP Handles Loss? n Fast recovery Wait for certain number of DACKs to detect the loss and retransmit RTT 1-2 Ack 1-2 Receiver 3-6 RTT DupAck 3 Retran 3 8

9 How TCP Handles Loss? n Fast recovery Wait for certain number of DACKs to detect the loss and retransmit n Timeout (RTO) If not enough DACKs return, retransmit after a timeout RTT 1-2 Ack Receiver RTO >> RTT e.g. RTO=5ms, RTT<100us [Pingmesh (SIGCOMM 15), DCTCP (SIGCOMM 10)] Timeout Retran 3 9

10 How TCP Handles Loss? n Fast recovery Wait for certain number of DACKs to detect the loss and retransmit n Timeout (RTO) If not enough DACKs return, retransmit after a timeout RTT 1-2 Ack Receiver RTO >> RTT e.g. RTO=5ms, RTT<100us [Pingmesh (SIGCOMM 15), DCTCP (SIGCOMM 10)] Timeout Encountering one RTO à Retran 3 dramatically increase the FCT 10

11 Loss Incurs Timeout n A little loss causes enough timeout to hurt the tail FCT 10KB(analysis) 10KB(testbed) 3% loss à ~10% timeout 99 th FCT > RTO 100KB(testbed) 100KB(analysis) Timeout probability of flows with different sizes passing a path with different packet loss rate 1. 1% loss à more than 1% flows timeout 2. Larger flows (e.g. 100KB) a. timeout ratio sharply grows when loss rate > 1% 16/6/25 11

12 Loss Incurs Timeout n A little loss causes enough timeout to hurt the tail FCT 10KB(analysis) 10KB(testbed) 3% loss à ~10% timeout 99 th FCT > RTO 100KB(testbed) 100KB(analysis) Timeout probability of flows with different sizes passing a path with different packet loss rate 1. 1% loss à more than 1% flows timeout 2. Larger flows (e.g. 100KB) To avoid RTO a. timeout ratio sharply grows when loss rate > 1% 16/6/25 12

13 Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 13

14 Challenge for TCP Loss Recovery n Prior works add aggressiveness to congestion control to do loss recovery before timeout (RTO) Tail Loss Probe (TLP) Ø transmit one prober after 2RTT Instant Recovery (TCP-IR) [SIGCOMM 13, RFC 5827] [SIGCOMM 13, RFC 5827] Ø generate an FEC packet for every group of packets (up to 16) Ø FEC packets also act as probers, delayed 1/4RTT before sent Proactive/RepFlow Ø Duplicate every packet/flow [SIGCOMM 13, INFOCOM 14] 16/6/25 14

15 Challenge for TCP Loss Recovery n How long to wait before sending recovery packets? For congestion loss Ø Should delay enough in case of worsening congestion Bursty: Lead to multiple consecutive losses [Incast (WREN 09), DCTCP (SIGCOMM 10)] 16/6/25 15

16 Challenge for TCP Loss Recovery n How long to wait before sending recovery packets? For congestion loss Ø Should delay enough in case of worsening congestion For failure loss such as random drop Ø Should recover as fast as possible, otherwise already increase the FCT Wait 2RTT is too costly [TLP SIGCOMM 13, RFC 5827] Accurate & high-precision RTT measurementis challenging 16/6/25 16

17 Brief Summary n Loss easily incurs timeout to hurt the tail n To prevent timeout, prior works add fixed aggressiveness to recover loss before timeout n Hard to adapt to various loss conditions Should be fast for failure loss Should be cautious for congestion loss How to accelerate loss recovery as soon as possible, under various loss conditions without causing congestion? 16/6/25 17

18 Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 18

19 FUSO: Fast Multi-path Loss Recovery n Utilize the good paths to proactively conduct loss recovery for bad paths Leveraging path diversity (multiple paths; a few encounter loss) n Fast and Cautious Fast Ø Proactive (immediate) recovery for potential packet loss utilizing spare transmission opportunity Cautious Ø Strictly follow congestion control without adding aggressiveness 16/6/25 19

20 Multi-path Transport Background Sub-flows: Implicitly/Explicitly mapping to physical paths Receiver Data Distribution CWND 1 CWND 2 CWND total CWND 3 Multi-path Congestion Control 16/6/25 20

21 FUSO Receiver CWND 1 P5 P4 P3 P1 CWND 2 CWND total CWND 3 16/6/25 21

22 FUSO Receiver P1 CWND 1 P5 P4 P3 CWND 2 CWND total CWND 3 16/6/25 22

23 FUSO Receiver P1 CWND 1 P5 P4 P3 CWND 2 CWND total CWND 3 16/6/25 23

24 FUSO Receiver CWND 1 P5 P4 P3 P1 CWND 2 CWND total CWND 3 16/6/25 24

25 FUSO Receiver CWND 1 P5 P4 P3 P1 CWND 2 CWND total CWND 3 16/6/25 25

26 FUSO Receiver CWND 1 Lost P5 P4 P3 P1 CWND 2 CWND total CWND 3 16/6/25 26

27 FUSO Receiver CWND 1 Lost P5 P4 P1 CWND 2 CWND total P3 CWND 3 16/6/25 27

28 FUSO Receiver CWND 1 Lost P5 P4 P1 CWND 2 CWND total P3 CWND 3 16/6/25 28

29 FUSO Receiver CWND 1 Lost P5 P4 P3 P1 CWND 2 CWND total CWND 3 16/6/25 29

30 FUSO Receiver CWND 1 Lost P5 P4 ACK P3 CWND total CWND 2 P3 P1 CWND 3 16/6/25 30

31 FUSO Receiver CWND 1 Lost P5 P4 P3 P1 CWND 2 CWND total CWND 3 16/6/25 31

32 FUSO Receiver CWND 1 Lost P3 P1 CWND total CWND 2 P5 P4 CWND 3 16/6/25 32

33 FUSO Receiver CWND 1 Lost P3 P1 CWND total CWND 2 P5 P4 CWND 3 16/6/25 33

34 FUSO Receiver CWND 1 Lost P5 P4 P3 P1 CWND total CWND 2 CWND 3 16/6/25 34

35 FUSO Receiver CWND 1 Lost ACK P1 ACK P4&P5 CWND total CWND 2 P5 P4 P3 P1 CWND 3 16/6/25 35

36 FUSO Receiver CWND 1 Lost P5 P4 P3 P1 CWND total CWND 2 CWND 3 16/6/25 36

37 FUSO Receiver Spare CWND CWND 1 Lost P5 P4 P3 P1 CWND 2 CWND total No new data CWND 3 16/6/25 37

38 FUSO Proactive loss recovery Receiver CWND 1 Lost P5 P4 P3 P1 CWND total CWND 2 CWND 3 16/6/25 38

39 FUSO Proactive loss recovery Receiver CWND 1 Lost P5 P4 P3 P1 CWND total CWND 2 Worst sub-flow CWND 3 Best sub-flow 16/6/25 39

40 FUSO Proactive loss recovery Receiver CWND 1 Lost P5 P4 P3 P1 CWND total CWND 2 Worst sub-flow CWND 3 Best sub-flow 16/6/25 40

41 FUSO Proactive loss recovery Receiver CWND 1 Lost P5 P4 P3 P1 CWND total CWND 2 Worst sub-flow CWND 3 Best sub-flow 16/6/25 41

42 FUSO Proactive loss recovery Receiver CWND 1 Lost P5 P4 P3 P1 CWND total CWND 2 Worst sub-flow CWND 3 Best sub-flow 16/6/25 42

43 FUSO Proactive loss recovery Receiver CWND 1 Lost P5 P4 P3 P1 CWND total CWND 2 Worst sub-flow Done! CWND 3 Best sub-flow 16/6/25 43

44 Standard MPTCP Receiver CWND 1 Retransmit after an RTO Lost P5 P4 P3 P1 CWND total CWND 2 CWND 3 16/6/25 44

45 FUSO: Path Selection Possibility ofencounteringloss CWND 1 Lost CWND total CWND 2 CWND 3 16/6/25 45

46 FUSO: Path Selection Possibility ofencounteringloss CWND total CWND 1 CWND 2 Worst sub-flow Lost Un-ACKed data n Worst Sub-flow With un-acked data Most likely having loss CWND 3 16/6/25 46

47 FUSO: Path Selection Possibility ofencounteringloss CWND total CWND 1 CWND 2 Spare CWND Worst sub-flow Lost n Worst Sub-flow With un-acked data Most likely having loss n Best Sub-flow CWND 3 Best sub-flow With spare CWND Least likely having loss 16/6/25 47

48 FUSO in 1 Slide Multipath Congestion Control Spare window Sub-Flow 1 Sub-Flow 1 Receiver P1 n n If (spare CWND) && (no new data) Utilize the transmission opportunity to proactively recover Use good paths to help bad paths Multi-path diversity offers many transmission opportunities App Data Recovery packets... R Send to best Sub-Flow Good paths have spare window P4 P3 Un-ACKed data Spare window Sub-Flow 2. Sub-Flow N Sub-Flow 2 Sub-Flow N 16/6/25 48 R P4 Recover P7 P3 P6 P5 App Data

49 FUSO Implementation n Implemented in Linux kernel; ~900 lines of code 16/6/25 49

50 Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 50

51 Testbed Settings n Network n TCP 1Gbps fabric & 1Gbps hosts; ECMP routing; ECN enabled Init_cwnd=16; min_rto=5ms 16/6/25 51

52 Testbed Results Loss rate: 0.125%-4% n Failure loss Random-drop Latency-sensitive flows Fast 99 th FCT % of flows encountering timeout better Reducing 99 th FCT up to Reducing the timeout flows ~82.3% up to 100% 16/6/25 52

53 Testbed Results n Congestion loss Incast Concurrent responses Performs the best Cautious better 16/6/25 53

54 Testbed Results Loss rate: 2% n Failure loss & Congestion loss From failure-loss-dominated to congestion-loss-dominated Latency-sensitive flows Background long flows better 16/6/25 Adapt to various loss condition 54

55 Larger-scale Simulations n Simulation settings NS2 simulator; 3- layer, 4-port FatTree 40Gbps fabric, 10Gbps host; 64 hosts, 20 switches Empirical failure generation Random failure Latency-sensitive flows Background long flows 16/6/25 55

56 Larger-scale Simulations n Simulation settings NS2 simulator; 3-layer, 4-port FatTree fabric 40Gbps fabric, 10Gbps host; 64 hosts, 20 switches Empirical failure generation better Reducing the average FCT up to ~60.3% Reducing the 99 th FCT up to ~87.4% 16/6/25 56

57 Outline n Motivation n Packet Loss in DCN n Impact of Packet Loss n Challenge for Loss Recovery n FUSO Design n Evaluation n Summary 16/6/25 57

58 Summary n n n Loss hurts tail latency Loss is not uncommon A little loss leads to enough timeout, hurting the tail Challenges for loss recovery How to accelerate loss recovery under various loss conditions without causing congestion? Philosophy for FUSO To be fast & cautious are equally important Fast: Proactive loss recovery utilizing spare transmission opportunity, leveraging multipath diversity Cautious: Strictly follows congestion control without adding aggressiveness 16/6/25 58

59 Thanks Q&A?