TCP. The TCP Protocol. TCP Header Format. TCP Flow Control. TCP Congestion Control Datacenter TCP

Transcription

1 The TCP Protocol TCP TCP Congestion Control Datacenter TCP Frame format Connection management Flow control TCP reliable data transfer Congestion control TCP Header Format TCP Flow Control HL 32 bits Source Port Destination Port Sequence Number Acknowledgement number U A P R S F R C S S Y I G K H T N N Window Size Checksum Urgent Pointer Options (0 or more 32-bit words) TCP uses a modified version of the sliding window In acknowledgements, TCP uses the Window size field to tell the sender how many bytes it may transmit TCP uses bytes, not packets, as sequence numbers Data 1

2 TCP Flow Control TCP Retransmission Application does a 2K write Sender Receiver Receiver s buffer 0 4K Empty When a packet remains unacknowledged for a period of time, TCP assumes it is lost and retransmits it Application does a 3K write Sender is blocked 2K Full Application reads 2K TCP tries to calculate the round trip time (RTT) for a packet and its acknowledgement From the RTT, TCP can guess how long it should wait before timing out Sender may send up to 2K 2K 1K 2K RTT Calculation Smoothing the RTT measurement 0.9 sec Sender Receiver First, we must smooth the round trip time due to variations in delay within the network: RTT SRTT = a SRTT + (1-a) RTT arriving ACK 2.2 sec RTT = 2.2 sec sec. = 1.3 sec The smoothed round trip time (SRTT) weights previously received RTTs by the a parameter a is typically equal to (1/8) a power of 2 2

3 Calculating the Retransmission Timeout Interval Problem with RTT Calculation The timeout value is then calculated by multiplying the smoothed RTT by some factor (greater than 1) called b Sender Receiver Timeout = b SRTT This coefficient of b is included to allow for some variation in the round trip times. Back-off; Whenever a timeout occurs Timeout=2*timeout Sender Timeout RTT? RTT? Karn s algorithm Use back-off as part of RTT computation On a packet loss, RTO (Retransmission Time Out) is increased by a factor Use this increased RTO as RTT estimate for the next segment (not from srtt) Only after an acknowledgment received for a successful transmission is the timer set to new RTT obtained from srtt Congestion Control 3

4 Congestion Control Too many packets in part of the subnet Performance Degrades: Congestion Congestion control is a mechanism to prevent or respond to overloaded conditions thereby ensuring timely delivery of packets from source to destination TCP Slow Start TCP defines the maximum segment size as the maximum size a TCP packet can be (including header) TCP Slow Start: Congestion window starts small, at 1 segment size Each time a transmitted segment is acknowledged, the congestion window is increased by one maximum segment size TCP Slow Start (cont d) TCP Slow Start (cont d) Congestion Window Size 1K 2K 4K 8K 16K Event A sends 1 segment to B B ACKs the segment A sends 2 segments to B B ACKs both segments A sends 4 segments to B B ACKs all four segments A sends 8 segments to B B ACKs all eight segments and so on Congestion window size grows exponentially (i.e. it keeps on doubling) Packet losses indicate congestion Packet losses are determined by using timers at the sender When a timeout occurs, the congestion window is reduced to one maximum segment size and everything starts over 4

5 TCP Slow Start (cont d) TCP Slow Start (cont d) Congestion window Timed out Transmissions TCP Slow Start by itself is inefficient Although the congestion window builds exponentially, it drops to 1 segment size every time a packet times out This leads to low throughput 1 Maximum Segment Size Transmission Number TCP Linear Increase Threshold TCP Linear Increase Threshold Establish a threshold at which the rate increase is linear instead of exponential to improve efficiency Algorithm: Start the threshold at 64K (ssthresh) Start the congestion window size at 1 segment size Increase the congestion window size exponentially using slow start until the threshold is reached Once the threshold is passed, only increase the congestion window size by 1 segment size for each congestion window of data transmitted For each ack received, increment cwnd as follows cwnd = cwnd + (mss*mss)/cwnd Note that cwnd is a multiple of mss for some k (say k*mss) During the linear phase, if we get k acks then cwnd increases by 1 mss If a timeout occurs, set thresh to max (cwnd/2,2) reset the congestion window size to 1 segment Example: Maximum segment size = 1K Congestion window 40K 32K 20K 1K Timeout occurs when MIN(sliding window, congestion window) = 40K Thresholds Transmission Number 5

6 Fast retransmission One way to infer loss is by RTT measurements (timer-driven) Another is to infer loss by counting number of duplicate acks (data-driven) TCP infers loss if it sees three duplicate acks This is simply a heuristic, works well in today s internet What other circumstances can lead to dup acks? TCP Fast Retransmit (RENO) Another enhancement to TCP congestion control Idea: When sender sees 3 duplicate ACKs, it assumes something went wrong The packet is immediately retransmitted instead of waiting for it to timeout TCP Fast Retransmit Example MSS = 1K Sender Receiver ACK of new data Datacenter TCP Duplicate ACK #1 1. Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage Systems. Amar Phanishayee, Elie Krevat, Vijay Vasudevan, David G. Andersen, Gregory R. Ganger, Garth A. Gibson, Srinivasan Seshan. 6th USENIX Conference on File and Storage Technologies (FAST '08). Feb , Fast Retransmit occurs (2nd packet is now retransmitted w/o waiting for it to timeout) Duplicate ACK #2 Duplicate ACK #3 2. Safe and Effective Fine-grained TCP Retransmissions for Datacenter Communication. Vijay Vasudevan, Amar Phanishayee, Hiral Shah, Elie Krevat, David G. Andersen, Gregory R. Ganger, Garth A. Gibson, Brian Mueller. SIGCOMM 09, August 17 21, 2009, Barcelona, Spain. 3. DaTa center TCP (DTCP), Mohammad Alizadeh et.al, SIGCOMM 2010, August, 2010, New Delhi, India 6

7 Density of servers!! Traditional Data Center Topology 100,000 servers (32 VMs/Host).. 3 M addressable hosts 100Gbps Core Switch Aggregation Switches 10Gbps 1Gbps Top of Rack Switches Racks of servers 2800 servers in flat bed truck Incast Problem Data Block Incast problem Synchronized Read R R R R Client Switch Client now sends next batch of requests Storage Servers Server Request Unit (SRU) Client sends a request to a set of servers Servers respond (almost Synchronously) Overwhelming resources at Aggregator (switch) Switch buffer is bottleneck TCP is data transport Slide from presentation in Fast 08 7

8 Link idle time due to timeouts TCP Throughput Collapse: Incast Synchronized Read 1 R R R R 2 Client 4 Switch 3 Collapse! Server 4 Request Unit (SRU) Link is idle until server experiences a timeout [Nagle04] called this Incast Cause of throughput collapse: TCP timeouts Solution Increase buffer size at switch Limit losses Losses result in timeouts Link idle during timeout interval Increasing switch buffer size: results per-port output buffer More servers supported before collapse Fast (SRAM) buffers are expensive 8

9 Increasing SRU size No throughput collapse using netperf Used to measure network throughput and latency netperf does not perform synchronized reads Hypothesis: Larger SRU size less idle time Servers have more data to send per data block One server waits (timeout), others continue to send Increasing SRU size: results SRU = 1MB SRU = 10KB SRU = 8MB Significant reduction in throughput collapse More pre-fetching, kernel memory Reducing the penalty of timeouts Reduce penalty by reducing Retransmission TimeOut period (RTO) NewReno with RTOmin = 200ms RTOmin = 200us Reduced RTOmin helps But still shows 30% decrease for 64 servers Issues with Reduced RTOmin Implementation Hurdle - Requires fine grained OS timers (us) - Very high interrupt rate - Current OS timers ms granularity - Soft timers not available for all platforms Unsafe - Servers talk to other clients over wide area - Overhead: Unnecessary timeouts, retransmissions 9

10 TCP in the Data Center Data Center TCP (DCTCP) Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, Murari Sridharan Microsoft Research Stanford University TCP RTTs are much smaller than 300msec In addition to Incast, TCP does not meet demands of apps. Builds up large queues: Adds significant latency. Wastes precious buffers, esp. bad with shallow-buffered switches. Operators work around TCP problems. Ad-hoc, inefficient, often expensive solutions No solid understanding of consequences, tradeoffs Slides from SIGCOMM presentation Case Study: Microsoft Bing Consider three Workloads Measurements from 6000 server production cluster Instrumentation passively collects logs Application-level Socket-level Selected packet-level More than 150TB of compressed data over a month Partition/Aggregate (Query) Short messages [50KB-1MB] (Coordination, Control state) Large flows [1MB-50MB] (Data update) Delay-sensitive Delay-sensitive Throughput-sensitive

11 Considers three Impairments Incast Queue Buildup Queue Buildup Sender 1 Big flows buildup queues. Increased latency for short flows. Receiver Buffer Pressure Large flows take up buffer space at the expense of small flows Sender 2 Measurements in Bing cluster For 90% packets: RTT < 1ms For 10% packets: 1ms < RTT < 15ms Data Center Transport Requirements 1. High Burst Tolerance Incast due to Partition/Aggregate is common. 2. Low Latency Short flows, queries The DCTCP Algorithm 3. High Throughput Continuous data updates, large file transfers The challenge is to achieve these three together

12 Review: The TCP/ECN Control Loop Sender 1 ECN = Explicit Congestion Notification Small Queues & TCP Throughput: The Buffer Sizing Story Bandwidth-delay product rule of thumb: A single flow needs buffers for 100% Throughput. ECN Mark (1 bit) Receiver Cwnd Sender 2 Buffer Size B Throughput 100% Two Key Ideas 1. React in proportion to the extent of congestion, not its presence. Reduces variance in sending rates, lowering queuing requirements. ECN Marks TCP DCTCP Cut window by 50% Cut window by 50% Cut window by 40% Cut window by 5% Data Center TCP Algorithm Switch side: Mark packets when Queue Length > K. Sender side: Maintain running average of fraction of packets marked (α). In each RTT: B Mark K Don t Mark 2. Mark based on instantaneous queue length. Fast feedback to better deal with bursts. 18 Adaptive window decreases: Note: decrease factor between 1 and

13 (Kbytes) DCTCP in Action Baseline Background Flows Query Flows Setup: Win 7, Broadcom 1Gbps Switch Scenario: 2 long-lived flows, K = 30KB Low latency for short flows Baseline Baseline Background Flows Query Flows Background Flows Query Flows Low latency for short flows. High throughput for long flows. 25 Low latency for short flows. High throughput for long flows. High burst tolerance for query flows

14 Scalability Conclusions DTCP satisfies all requirements for Data Center packet transport. Handles bursts well Keeps queuing delays low Achieves high throughput Availability of Multiple paths Multipath TCP References: 1. Improving Datacenter performance using MTCP- SIGCOM Design and Implementation of contestion control for MTCP NSDI IETF presentations on MTCP 4. Multipath TCP resources UCL Website (nrg.cs.ucl.ac.uk/mptcp) 5. Slides from SIGCOMM, NSDI, and NETSYS tutorial presentations Devices with more than one interface WIFI, 3G, Bluetooth Datacenter networking topologies Hierarchical FAT trees (redundant paths) ECMP Availability of Equal Cost Multi Path 14

15 What technology provides Datacenters 3G celltower IP IP How to exploit multiple paths Send different flows over different paths One TCP connection over each path Single e2e connection cannot use more than 1 path Current TCP works Stripe across ECMP Stripe across paths: not balanced Collision 15

16 How about mapping each flow to a different path? How about mapping each flow to a different path? How about mapping each flow to a different path? How about mapping each flow to a different path? Not fair Not fair Mapping each flow to a path is the wrong approach! 16

17 Instead, pool capacity from links No matter how you do it, mapping each flow to a path is the wrong goal 17

18 Instead, we should pool capacity from different Wide range of traffic sharing Two single paths Max (0< bw<100) Shared is only 50 1 multi path -Max 1 (0<bw<200) 1 flow can use 100 MB on both paths Both multi path -Max 2 (0 <bw <100) 50 in each path; for both flows Shared is 100 Improving Datacenter Performance and Robustness with Multipath TCP Costin Raiciu Department of Computer Science University Politehnica of Bucharest Sebastien Barre (UCL-BE), Christopher Pluntke (UCL), Adam Greenhalgh (UCL), Damon Wischik (UCL) and Mark Handley (UCL) Multipath Transport SIGCOMM 2011 Presentation 18

19 Design decision Multipath TCP and the architecture A Multipath TCP connection is composed of one of more regular TCP subflows that are combined Each host maintains state that glues the TCP subflows that compose a Multipath TCP connection together Each TCP subflow is sent over a single path and appears like a regular TCP connection along this path Application Transport Network Datalink Physical Application socket Multipath TCP TCP1 TCP2... TCPn A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, Architectural guidelines for multipath TCP development", RFC A regular TCP connection What is a regular TCP connection? It starts with a three-way handshake SYN segments may contain special options A naïve Multipath TCP SYN+Option SYN+ACK+Option ACK seq=123, "abc" All data segments are sent in sequence There is no gap in the sequence numbers It is terminated by using FIN or RST seq=126, "def" 19

20 A naïve Multipath TCP In today's Internet? SYN+ACK+Option seq=123, "abc" SYN+Option ACK Multipath TCP SYN+Option SYN+ACK+Option ACK seq=126, "def" There is no corresponding TCP connection SYN+OtherOption SYN+ACK+OtherOption ACK How to combine two TCP subflows? SYN+Option SYN+ACK+Option ACK How to JOIN TCP subflows? SYN+ACK[...] SYN, Port src =1234,Port dst =80+Option ACK A NAT could change addresses and port numbers SYN+OtherOption SYN+ACK+OtherOption ACK How to link with blue subflow? SYN, Port src =1235,Port dst =80 +Option[JOIN Port src =1234,Port dst =80] 20

21 How to link TCP subflows? MyToken=5678 YourToken=6543 SYN, Port src =1234,Port dst =80 +Option[Token=5678] SYN+ACK+Option[Token=6543] ACK Subflow agility Multipath TCP supports addition of subflows removal of subflows MyToken=6543 YourToken=5678 SYN, Port src =1235,Port dst =80 +Option[Token=6543] How to transfer data? seq=123,"a" How to transfer data in today's Internet? seq=123,"a" ack=124 ack=126 seq=125,"c" ack=124 ack=126 seq=125,"c" Gap in sequence numbering space Some DPI will not allow this! seq=124,"b" ack=125 seq=126,"d" ack=127 seq=124,"b" ack=125 21

22 Multipath TCP Data transfer How to assign sequence numbers Two levels of sequence numbers ABCDEF socket Multipath TCP TCP1 TCP2 Data sequence # TCP1 sequence # TCP2 sequence # socket Multipath TCP TCP1 TCP2 Sequence numbers TCP header 22

23 Two Sequence numbers Data Sequence Number: Global position of each byte in the stream TCP sequence number Multipath TCP Data transfer Dseq=0,seq=123,"a" DAck=1,ack=124 DSeq=2, seq=124,"c" DAck=3, ack=125 DSeq=1, seq=456,"b" DAck=2,ack=457 Multipath TCP How to deal with losses? Data losses over one TCP subflow Fast retransmit and timeout as in regular TCP Multipath TCP What happens when a TCP subflow fails? Dseq=0,seq=123,"a" DAck=1,ack=124 Dseq=0,seq=123,"a" DAck=1,ack=1 24 Dseq=0,seq=123,"a" DSeq=1, seq=456,"b" DSeq=0,ack=457 DAck=2,ack=458 Dseq=0,seq=457,"a" 23

24 Retransmission heuristics TCP PACKET HEADER Heuristics used by current Linux implementation Fast retransmit is performed on the same subflow as the original transmission Upon timeout expiration, reevaluate whether the segment could be retransmitted over another subflow Upon loss of a subflow, all the unacknowledged data are retransmitted on other subflows RECEIVE WINDOW Flow control How should the window-based flow control be performed? Independent windows on each TCP subflow A single window that is shared among all TCP subflows 24

25 Independent windows Independent windows possible problem Dseq=0,seq=123,"a" Dseq=0,seq=123,"a" DAck=1,ack=124,win= 0 DSeq=1, seq=456,"b" DAck=2,ack=457,win=100 Dseq=2,seq=457,"c" DAck=3,ack=458,win=100 DSeq=1, seq=456,"b" DAck=2,ack=457,win=0 Impossible to retransmit, window is already full on green subflow A single window shared by all subflows Dseq=0,seq=123,"a" DAck=1,ack=124,win= 10 DSeq=1, seq=456,"b" DAck=2,ack=457,win=10 Dseq=2,seq=457,"c" DAck=3,ack=458,win=10 Multipath TCP Windows Multipath TCP maintains one window per Multipath TCP connection Window is relative to the last acked data (Data Ack) Window is shared among all subflows It's up to the implementation to decide how the window is shared Window is transmitted inside the window field of the regular TCP header 25

26 TCP congestion control A linear rate adaption algorithm To be fair and efficient, a linear algorithm must use additive increase and multiplicative decrease (AIMD) # Additive Increase Multiplicative Decrease if congestion : rate=rate*betac # multiplicative decrease, betac<1 else rate=rate+alphan # additive increase, v0>0 AIMD in TCP Congestion control mechanism Each host maintains a congestion window (cwnd) No congestion Congestion avoidance (additive increase) increase cwnd by one segment every round-trip-time Congestion TCP detects congestion by detecting losses Mild congestion (fast retransmit multiplicative decrease) cwnd=cwnd/2 and restart congestion avoidance Severe congestion (timeout) cwnd=1, set slow-start-threshold and restart slow-start Evolution of the congestion window Congestion control for Multipath TCP Simple approach Cwnd Fast retransmit independant congestion windows Fast retransmit Threshold Threshold Threshold Threshold Slow-start exponential increase of cwnd Congestion avoidance linear increase of cwnd Time Threshold 26

27 Independent congestion windows Problem (Unfair to Regular TCP) 12Mbps Coupled congestion control Congestion windows are coupled Congestion window growth cannot be faster than TCP with single flow Coupled congestion control aims at moving traffic away from congested path Coupling the congestion windows Principle The TCP subflows are not independant and their congestion windows must be coupled EWTCP For each ACK on path r, cwin r =cwin r +a/cwin r (in segments) For each loss on path r, cwin r =cwin r /2 How does MPTCP congestion control work? Maintain a congestion window w r, one window for each path, where r R ranges over the set of available paths. -Increase w r for each ACK on path r, by Each subflow gets window size proportional to a 2 Same throughput as TCP if a = 1 n D. Wischik, C. Raiciu, A. Greenhalgh, and M. Handley, Design, implementation and evaluation of congestion control for multipath TCP, NSDI'11: Proceedings of the 8th USENIX conference on Networked systems design and implementation, M. Honda, Y. Nishida, L. Eggert, P. Sarolahti, and H. Tokuda. Multipath Congestion Control for Shared Bottleneck. In Proc. PFLDNeT workshop, May Decrease w r for each drop on path r, by w r /2 27

28 Multipath TCP: Congestion Control [NSDI, 2011] MPTCP better utilizes the FatTree network MPTCP on EC2 Amazon EC2: infrastructure as a service We can borrow virtual machines by the hour These run in Amazon data centers worldwide We can boot our own kernel A few availability zones have multipath topologies 2-8 paths available between hosts not on the same machine or in the same rack Available via ECMP Amazon EC2 Experiment 40 medium CPU instances running MPTCP For 12 hours, we sequentially ran all-to-all iperf cycling through: TCP MPTCP (2 and 4 subflows) 28

29 MPTCP improves performance on EC2 MPTCP improves fairness in VL2 topologies VL2 Same Rack Fairness is important: Jobs finish when the slowest worker finishes MPTCP improves throughput and fairness in BCube BCube Summary One flow, one path thinking has constrained datacenter design Collisions, unfairness, limited utilization Multipath transport enables resource pooling in datacenter networks: Improves throughput Improves fairness Improves robustness One flow, many paths frees designers to consider topologies that offer improved performance for similar cost 29