The dark side of TCP. understanding TCP on very high-speed networks. ACOMP 2008 HCMC, Bach Khoa University March 11th, 2008 LIUPPA. C.

Transcription

1 The dark side of TCP understanding TCP on very high-speed networks ACOMP 2008 HCMC, Bach Khoa University March 11th, 2008 C. Pham University of Pau, France laboratory

2 The The big-bang big-bang of of the the Internet Internet

3 # Internet host INTRODUCTION

4 Towards all IP & TCP! IPTV VOD VoIP IP Telephony Multimedia High-perf networking INTERNET, HTTP Sensor networks Grid computing Interactive gaming Pervasive networking 90% of the Internet traffic is based on TCP IP E1/T1 X.25 FR ATM PSTN INTRODUCTION

5 What TCP brings stream-based in-order delivery segments are ordered according to sequence numbers only consecutive bytes are delivered reliability missing segments are detected (ACK is missing) and retransmitted flow control receiver is protected against overload (window based) congestion control network is protected against overload (window based) protocol tries to fill available capacity connection handling explicit establishment + teardown full-duplex communication an ACK can be a data segment at the same time (piggybacking) INTRODUCTION

6 A brief history of TCP 1975 Three-way handshake Raymond Tomlinson In SIGCOMM TCP described by Vint Cerf and Bob Kahn In IEEE Trans Comm 1982 TCP & IP RFC 793 & BSD Unix 4.2 supports TCP/IP 1984 Nagel s algorithm to reduce overhead of small packets; predicts congestion collapse 1986 Congestion collapse observed 1987 Karn s algorithm to better estimate round-trip time 1988 Van Jacobson s algorithms congestion avoidance and congestion control (most implemented in 4.3BSD Tahoe) BSD Reno fast retransmit delayed ACK s INTRODUCTION

7 in the nineties 1994 T/TCP (Braden) Transaction TCP 1996 SACK TCP (Floyd et al) Selective Acknowledgement 1993 TCP Vegas (Brakmo et al) real congestion avoidance 1994 ECN (Floyd) Explicit Congestion Notification 1996 Hoe Improving TCP startup 1996 FACK TCP (Mathis et al) extension to SACK? INTRODUCTION

8 huge variety of communicating devices! Internet Wireless sensor nodes INTRODUCTION

9 1st revolution: Wireless Networks WiFi, WiMax BlueTooth, ZigBee, IrDA GSM, GPRS, EDGE, UMTS, 4G, Access Point INTRODUCTION

10 2nd revolution: going optical 2x / 18 months 2x / 7 months INTRODUCTION Source «Optical fibers for Ultra-Large Capacity Transmission» by J. Grochocinski

11 DWDM, bandwidth for free? DWDM: Dense Wavelength Division Multiplexing < 0,1 nm 10Gbps 2Gbps 10, 40, 160 Gbps are available! INTRODUCTION

12 campus Fibers everywhere? NEWS of Dec 15th, 2004 NEWS from Japan and residentials South Korea Verizon offices and SBC are NEWS of May 31st, 2005 deploying large optical fiber Japan FTTH is one of the first FTTC infrastructures in the US US Fiber-to-the-home country in FTTH technology FTTP using FTTC or FTTP (FTTH) installations with 3.2 millions have users at the scenario 10Gbps grown 83% end since of 2005 October (16 % of Internet highspeed access 2004, now reaching Data 398 market). Center In metro ring NEWS of July, communities 2006 South in 43 Korea, states FTTH users Network Provider represents Gbps %. France Telecom Verizon will deploy is on track to pass 2.5Gbps an FTTH 10Gbps test-bed three million homes with infrastructure in fiber Paris. by 2.5 the end of 2005 Network Gbps Provider in download and 1.2Gbps in upload! 1Gbps GigaEth INTRODUCTION Core 40, 160 Gbps

13 SONET/SDH in the core 95% % of exploited OF use SONET/SDH Digital switch Digital switch n*30*64 Kb/s MUX PDH/SDH n*2048 Kb/s Optical Fiber or Microwave Link SDH : INTRODUCTION STM-1 : Mb/s STM-4 : Mb/s STM-16 : Mb/s MUX PDH/SDH

14 SONET/SDH transport network infrastructure Add Drop Multiplexer rings rings INTRODUCTION SONET/SDH now offers Native Ethernet interface Generic Framing Procedure Virtual Concatenation

15 The new networks vbns Abilene SUPERNET DREN CA*NET GEANT DATATAG much more to come! INTRODUCTION

16 GEANT INTRODUCTION

17 Computational grids user application Virtually unlimited resources 1PFlops INTRODUCTION from Dorian Arnold: Netsolve Happenings

18 Real-time interactive large- scale scientific collaborations Multimodality brain mapping require the ability to process, share, and interactively visualize multiple 100Gbytes datasets! Today, to visualize and explore eight 3D images require 64Gb/s! Large data transfers require very high bandwidth

19 TCP FTP HTTP SSH SMTP

20 Very High-Speed Networks Optical fiber 40 Gbps km/s, delay of 5ms every 1000km Today s backbone links are optical, DWDMbased, and offer gigabit rates Transmission time <<< propagation time Duplicating a 10GB database should not be a problem anymore

21 The reality check: TCP on a 200Mbps link Huge capacity in network links does not mean end-to-end performances! TCP is not adapted to exploit Long Fat Networks! Packet losses

22 The things about TCP your mother never told you! vanilla TCP TCP 40 Gbps 0.3Gbps If you want to transfer a 1Go file with a standard TCP stack, you will need minutes even with a 40Gbps (how much in $?) link!

23 Let s s go back to the origin! Flow control is for receivers Congestion control is for the network From Computer Networks, A. Tanenbaum Congestion collapse was first observed in 1986 by V. Jacobson. Congestion control was added to TCP (TCP Reno) in 1988.

24 The congestion phenomenon 10 Mbps 1.5 Mbps 100 Mbps Too many packets sent to the same interface. Difference bandwidth from one network to another Main consequence: packet losses in routers

25 Flow control prevents receiver s s buffer overfow Packet Sent Source Port Dest. Port Sequence Number Acknowledgment HL/Flags Window D. Checksum Urgent Pointer Options.. Packet Received Source Port Dest. Port Sequence Number Acknowledgment HL/Flags Window D. Checksum Urgent Pointer Options.. App write acknowledged sent to be sent outside window

26 TCP congestion control: the big picture Congestion window doubles every round-trip time Sequence No From Computer Networks, A. Tanenbaum packet ack Time cwnd grows exponentially (slow start), then linearly (congestion avoidance) with 1 more segment per RTT If loss, divides threshold by 2 (multiplicative decrease) and restart with cwnd=1 packet

27 From the control theory point of view ƒ feedback Closed-loop control Feedback should be frequent, but not too much otherwise there will be oscillations Can not control the behavior with a time granularity less than the feedback period

28 Congestion: A Close-up View knee point after which throughput increases very slowly delay increases fast cliff point after which throughput starts to decrease very fast to zero (congestion collapse) delay approaches infinity Note (in an M/M/1 queue) delay = 1/(1 utilization) Throughput Delay knee cliff Load Load packet loss congestion collapse

29 Congestion Control vs. Congestion Avoidance Congestion control goal stay left of cliff Congestion avoidance goal stay left of knee Right of cliff: Congestion collapse Throughput knee cliff congestion collapse Load

30 The TCP saw-tooth curve TCP behavior in steady state N Isolated packet losses trigger the fast recovery procedure instead of the slow-start. The TCP steadystate behavior is referred to as the Additive Increase- Multiplicative Decrease process N/2 3N/4.N/2 Packets/cycle no loss: cwnd = cwnd + 1 loss: cwnd = cwnd*0.5

31 AIMD Phase plot User 2 s Allocation x 2 t 0 Convergence point Fairness Line x 1 =x 2 Efficiency Line x 1 +x 2 =C Fairness is preserved under Multiplicative Decrease since the user s allocation ratio remains the same Ex: x 2 = x.b 2 x 1 x 1.b User 1 s Allocation x 1 Assumption: decrease policy must (at minimum) reverse the load increase over-and-above efficiency line Implication: decrease factor should be conservatively set to account for any congestion detection lags etc

32 Tuning stand for TCP the dark side of speed! TCP team TCP performances depend on TCP & network parameters Congestion window size, ssthresh (threshold) RTO timeout settings SACKs NEED A Packet size System parameters SPECIALIST! TCP and OS buffer size (in comm. subsys., drivers )

33 First problem: window size The default maximum window size is 64Kbytes. Then the sender has to wait for acks. TIME RTT Time to transmit 3 packets Sender Packet #1 Packet #2 Packet #3 Receiver Packet #1 Ack. Packet #2 Ack. Packet #3 Ack. Packet #4 Packet #5 Packet #6 Packet #4 Ack. Packet #5 Ack. Packet #6 Ack.

34 First problem: window size The default maximum window size is 64Kbytes. Then the sender has to wait for acks. RTT=200ms Link is 0C-48 = 2.5 Gbps Waiting time

35 Rule of thumb on Long Fat Networks capacity High-speed network Transmission time is small Propagation time is large RTT The optimal window size should be set to the bandwidthxrtt product to avoid blocking at the sender side Need lots of memory for buffers!

36 Side effect of large windows TCP becomes very sensitive to packet losses on LFN Congestion window size Large congestion window create burst/congestion Packet losses

37 Pushing the limits of TCP Standard configuration (vanilla TCP) is not adequate on many OS, everything is undersized Receiver buffer System buffer Default block size Will manage to get near 1Gbps if well-tuned

38 Pushing the limits of TCP Standard configuration (vanilla TCP) is not adequate on many OS, everything is undersized Receiver buffer System buffer Default block size Large congestion window Socket buffer=64mo Will manage to get near 1Gbps if well-tuned Source: M. Goutelle, GEANT test campaign

39 Some TCP tuning guides cptune/ k-tcpip.htm /tcp.html

40 Problem on high capacity link? Additive increase is still too slow! Take ages to get to full speed With 100ms of round trip time, a connection needs 203 minutes (3h23) to send at 10Gbps starting from 1Mbps! Once you get high throughput, maintaining it is difficult too! From S. Floyd

41 TCP rules: slow increase, big decrease A TCP connection with 1250-Byte packet size and 100ms RTT is running over a 10Gbps link (assuming no other connections, and no buffers at routers) From Injong Rhee, Lisong Xu cwnd 1.4 hours 1.4 hours 1.4 hours slow Packet lossincrease Packet loss Packet loss Packet loss TCP 100,000 10Gbps big decrease 50,000 5Gbps Slow start Congestion avoidance Time (RTT)

42 Going faster (cheating?) n flows is better than 1 The CC limits the throughput of a TCP connection: so why not use more than 1 connection for the same file? Very big file Seg 1 Seg 2 Seg 3 Seg n-1 Seg n TCP connection TCP connection 10 Gbps link

43 Some results from IEPM/SLAC More streams is better than larger congestion windows

44 Multiple streams No/few modifications to transport protocols (i.e. TCP) Parallel socket libraries GridFTP ( bbftp (

45 New transport protocols New transport protocols are those that are not only optimizations of TCP New behaviors, new rules, new requirements! Everything is possible! New protocols are then not necessarily TCP compatible! BEYOND TCP

46 The new transport protocol strip XCP H-TCP BIC TCP FAST TCP HS-TCP S-TCP TSUNAMI BEYOND TCP

47 Response function Throughput = f(p, RTT) TCP s response function, from (N+N/2)/2 (N/2) 2 +1/2(N/2) 2 Throughput = W RTT = 3 2 MTU RTT p BEYOND TCP

48 TCP s s response function in image Throughput = W RTT = 3 2 MTU RTT p MTU: Packet Size RTT: Round-Trip Time P : Packet Loss Probability From Injong Rhee, Lisong Xu 10Gbps requires a packet loss rate of 10-10, which is an unrealistic (or at least hard) requirement for current networks. BEYOND TCP

49 AIMD, general case cwnd = cwnd + 1 cwnd = cwnd + 32 cwnd = cwnd * (1-1/2) cwnd = cwnd * (1-1/8) The throughput of AIMD is always about 13 times larger than that of TCP NOT TCP Friendly!!! What s wrong? TCP: R = MSS RTT 1.2 p 0.5 AIMD: R = MSS RTT 15.5 p 0.5 Inspired from Injong Rhee, Lisong Xu BEYOND TCP

50 High Speed TCP [Floyd[ Floyd] Modifies the response function to allow for more link utilization in current high-speed networks where the loss rate is smaller than that of the networks TCP was designed for (at most 10-2 ) TCP Throughput (Mbps) RTTs Between Losses W P Table 1: RTTs Between Congestion Events for Standard TCP, for 1500-Byte Packets and a Round-Trip Time of 0.1 Seconds. From draft-ietf-tsvwg-highspeed-01.txt BEYOND TCP

51 Modifying the response Packet Drop Rate P Congestion Window W RTTs Between Losses ^ ^ ^ ^ ^ ^ ^ ^ ^ Table 2: TCP Response Function for Standard TCP. The average congestion window W in MSS-sized segments is given as a function of the packet drop rate P. From draft-ietf-tsvwg-highspeed-01.txt To specify a modified response function for HighSpeed TCP, we use three parameters, Low_Window, High_Window, and High_P. To Ensure TCP compatibility, the HighSpeed response function uses the same response function as Standard TCP when the current congestion window is at most Low_Window, and uses the HighSpeed response function when the current congestion window is greater than Low_Window. In this document we set Low_Window to 38 MSS-sized segments, corresponding to a packet drop rate of 10^-3 for TCP. Packet Drop Rate P Congestion Window W RTTs Between Losses ^ ^ ^ ^ ^ ^ ^ ^ ^ Table 3: TCP Response Function for HighSpeed TCP. The average congestion window W in MSS-sized segments is given as a function of the packet drop rate P. BEYOND TCP

52 See it in image TCP Friendly region BEYOND TCP

53 Relation with AIMD TCP-AIMD Additive increase: a=1 Multiplicative decrease: b=1/2 no loss: cwnd = cwnd + 1 loss: cwnd = cwnd*0.5 HSTCP-AIMD Link a & b to congestion window size a = a(cwnd), b=b(cwnd) General rules the larger cwnd, the larger the increment The larger cwnd, the smaller the decrement BEYOND TCP

54 Quick to grab bandwidth, slow to give some back! No loss: cwnd=cwnd+a Loss: cwnd=cwnd*b BEYOND TCP

55 Talking about dark side Starvation of TCP flow (>10x) 1 HSTCP and 1 TCP flow 2 TCP flows SETUP RTT=100ms Bottleneck BW=50Mbps Qsize=BW*RTT Qtype=DropTail BEYOND TCP

56 XCP [Katabi02] XCP is a router-assisted solution, generalized the ECN concepts (FR, TCP-ECN) XCP routers can compute the available bandwidth by monitoring the input rate and the output rate Feedback is sent back to the source in special fields of the packet header source H_feedback Q EC FC Input rate: I r Output rate: O r XCP packet header H_cwnd (set to the sender s current cwnd) H_rtt (set to sender s RTT estimate) H_feedback (initialized to sender s demands) feedback=α.rtt.(o r -I r )-βq α=0.4, β=0.226 Q: persistent queue size BEYOND TCP

57 XCP in action Feedback value represents a window increment/decrement H_cwnd=200 H_rtt=100ms H_feedback=0 Q source I r =250Mbps O r =100Mbps cwnd=200 cwnd=194 H_cwnd=200 H_rtt=100ms H_feedback=-6 BEYOND TCP feedback=α.rtt.(o r -I r )-βq α=0.4, β=0.226 Q: persistent queue size Case without βq contribution O r -I r = =-150 feedback=-6

58 XCP Variable bandwidth environments Good fairness and stability even in variable bandwidth environments BEYOND TCP

59 XCP-r [Pacheco&Pham05] A more robust version of XCP 10 flows sharing a 1Gbps link Fast recovery after the timeouts and better fairness level BEYOND TCP

60 XCP-r performance Amount of data transfered in 50s, 10 flows, 1Gbps link, 200ms RTT BEYOND TCP

61 XCP-r fairness TCP and HSTCP are not really fair... BEYOND TCP

62 Nothing is perfect :-( Multiple or parallel streams How many streams? Tradeoff between window size and number of streams New protocol Fairness issues? Deployment issues? Still too early to know the side effects BEYOND TCP

63 Where to find the new protocols? HSTCP STCP on Linux ctk21/scalable/ FAST XCP BEYOND TCP

64 Web100 project «The Web100 project will provide the software and tools necessary for endhosts to automatically and transparently achieve high bandwidth data rates (100 Mbps) over the high performance research networks» Actually it s not limited to 100Mbps! Recommended solution for end-users to deploy and test high-speed transport solutions BEYOND TCP

65 Hostile environments Asymetric networks Satellite links & terrestrial links Wireless (WiFi, WiMax) High loss probability Losses congestions Ad-Hoc (PDA) Small capacity Wireless Sensor Networks All of the above mentioned problems! BEYOND TCP

66 New sensor applications disaster relief - security Real-time organization and optimization of rescue in large scale disasters BEYOND TCP Rapid deployment of fire detection systems in highrisk places

67 Conclusions Understanding the dark side allows to move forwards! However vanilla TCP 10GB file 40 Gbps MAY THE FORCE BE WITH YOU! BEYOND TCP