COSC4377. Network milestones

Transcription

1 Lecture Network milestones ARPANet TCP/IP TCP Backbone speed: Cutover to TCP/IP Tahoe HTTP 50 56kbps, ARPANet T1 NSFNet T3, NSFNet Network is exploding TCP Cambridge.pptx 622 Mbps OC12 MCI 2.5 Gbps OC48 vbns 2 10 Gbps OC192 Abilene 1

2 Congestion collapse October 1986, the first congestion collapse on the Internet was detected Link between UC Berkeley and LBL 400 yards, 3 hops, 32 Kbps throughput dropped to 40 bps factor of ~1000 drop! 1988, Van Jacobson proposed TCP congestion control throughput load TCP Cambridge.pptx 3 TCP congestion control ARPANet Network Mail TCP 50 56kbps, ARPANet Telnet TCP/IP Flow control: File Prevent Transfer overwhelming receiver Low TCP Cambridge.pptx Cutover Tahoe TCP Tahoe to TCP/IP congestion collapse detected at LBL Internet Talk Radio HTTP Internet Phone Whitehouse T1 online NSFNet T3, NSFNet Napster music OC12 MCI OC48 + Congestion control: vbns Prevent overwhelming network AT&T VoIP itunes video YouTube OC192 Abilene 4 2

3 Recall: TCP Slow Start when connection begins, increase rate exponentially until first loss event: initially cwnd = 1 MSS double cwnd every RTT done by incrementing cwnd for every ACK received summary: initial rate is slow but ramps up exponentially fast Host A RTT Host B time Slide from Kurose & Ross, 6 th Ed 5 Recall: TCP switching from slow start to CA Q: when should the exponential increase switch to linear? A: when cwnd gets to 1/2 of its value before timeout. Implementation: variable ssthresh on loss event, ssthresh is set to 1/2 of cwnd just before loss event Slide from Kurose & Ross, 6 th Ed 6 3

4 Recall: TCP detecting, reacting to loss loss indicated by timeout: cwnd set to 1 MSS; window then grows exponentially (as in slow start) to threshold, then grows linearly loss indicated by 3 duplicate ACKs: TCP RENO dup ACKs indicate network capable of delivering some segments cwnd is cut in half window then grows linearly TCP Tahoe always sets cwnd to 1 (timeout or 3 duplicate acks) Slide from Kurose & Ross, 6 th Ed 7 Recall: TCP Futures TCP over long, fat pipes example: 1500 byte segments, 100ms RTT, want 10 Gbps throughput requires W = 83,333 in flight segments throughput in terms of segment loss probability, L [Mathis 1997]: TCP throughput = MSS RTT L to achieve 10 Gbps throughput, need a loss rate of L = a very small loss rate! new versions of TCP for high speed Slide from Kurose & Ross, 6 th Ed 8 4

5 TCP congestion control history (incomplete) ARPANet TCP Cutover to TCP/IP Flow control: Prevent overwhelming receiver Derived from Low TCP Cambridge.pptx TCP Tahoe TCP Reno TCP SACK TCP Vegas TCP New Reno + Congestion control: Prevent overwhelming network BICTCP Fast TCP HSTCP STCP DCTCP 9 TCP Performance Congestion Control Congestion is inevitable Internet does not reserve resources in advance TCP actively tries to push the envelope Congestion can be handled Additive increase, multiplicative decrease Slow start, and slow start restart Active Queue Management can help (discussed in some detail in Network Layer lectures) Random Early Detection (RED) Explicit Congestion Notification (ECN) Fundamental tensions Feedback from the network? Enforcement of TCP friendly behavior? Source congestion.ppt 10 5

6 TCP Recovery Time (Simplistic Model) Recovery Rate of Recovery Time Bytes to Recover Recovery Time Bytes to Recover Rate of Recovery BW * RTT MTU RTT BW * RTT MTU RTT (BWDP) 2 RTT * BW MTU Globus Allcock/ParallelTCP.pdf 11 Recovery Time for a Single Congestion Event (Simplistic Model) T1 (1.544 Mbs) with 50ms RTT 10 KB Recovery Time (1500 MTU): 0.16 Sec GigE with 50ms RTT 6250 KB Recovery Time (1500 MTU): 104 Seconds GigE to Amsterdam (100ms) 1250 KB Recovery Time (1500 MTU): 416 Seconds GigE to CERN (160ms) 2000 KB Recovery Time (1500 MTU): 1066 Sec (17.8 min) Globus Allcock/ParallelTCP.pdf 12 6

7 TCP over High Speed Networks A TCP connection with 1250 Byte packet size and 100ms RTT is running over a 10Gbps link (assuming no other connections, and no buffers at routers) cwnd 1.4 hours 1.4 hours 1.4 hours slow Packet loss increase Packet loss Packet loss Packet loss 100,000 10Gbps big decrease TCP 50,000 5Gbps Slow start Congestion avoidance Time (RTT) 13 Link Utilization TCP Performance Utilization of a link with 5 TCP connections Cannot fully utilize the huge capacity of highspeed networks! Link Capacity (Mbps) NS 2 Simulation (100 sec) Link Capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps, Drop Tail Routers, 0.1BDP Buffer 5 TCP Connections, 100ms RTT, 1000 Byte Packet Size

8 Linux 2.4 New Reno (1994) Low performance on fast long distance paths AIMD (add a=1 pkt to cwnd / RTT, decrease cwnd by factor b=0.5 in congestion) 700 Throughput Mbps Abilene/Internet2 10 Gbps Reno RTT ms RTT (~70ms) s SLAC to Univ of Florida Network Services Interoperability Labs Information courtesy of Les Cottrell from the SLAC group at Stanford Source: TCP alternatives hicks.ppt 15 Parallel TCP Reno TCP Reno with 16 streams Parallel streams heavily used in HENP (High Energy and Nuclear Physics) & elsewhere to achieve needed performance, so it is today s de facto baseline However, hard to optimize both the window size AND number of streams since optimal values can vary due to network capacity, routes or utilization changes Information courtesy of Les Cottrell from the SLAC group at Stanford Source: TCP alternatives hicks.ppt 16 8

9 Parallel TCP: Does it Help? Reduces the severity of a congestion event Buffers are divided across streams so faster recovery Probably get more than your fair share in the router Globus Allcock/ParallelTCP.pdf 17 Reduced Severity from Congestion Events Normal TCP your BW Reduction is 50% 1000 Mbps * 50% = 500 Mbps Reduction In Parallel TCP BW Reduction for single Stream packet loss is: Total BW / N Streams * 50% Example 1 of four streams congested 1000 / 4 * 50% = 125 Mbps Reduction Globus Allcock/ParallelTCP.pdf 18 9

10 Faster Recovery from Congestion Events Optimum TCP Buffer Size is now BWDP/N where N is number of Streams Since Buffers are reduced in size by a factor of 1/N so is the recovery time. This can also help work around host limitations. If the maximum buffer size is too small for max bandwidth, you can get multiple smaller buffers. Globus Allcock/ParallelTCP.pdf 19 More than your Fair Share Routers apply fair sharing algorithms to the streams being processed. Since your logical transfer now has N streams, it is getting N times the service it otherwise normally would, unless a router discover that a logical stream has been parallelized. Globus Allcock/ParallelTCP.pdf 20 10

11 What about Striping? Typically used in a cluster with a shared file system, but it can be a multi homed host All the advantages of Parallel TCP Also get parallelism of CPUs, Disk subsystems, buses, NICs, etc.. You can, in certain circumstances, also get parallelism of network paths This is a much more complicated implementation and beyond the scope of what we are primarily discussing here. Globus Allcock/ParallelTCP.pdf 21 Affect of Parallel Streams ANL to ISI (n=5) Bandwidth (Mbs) 70 Notes: 60 1) Error bars represent +/- 1 standard deviation, n=5 2) Variance in the bandwidth decreases with number of streams 50 - More streams means lower impact per stream 40 - More streams to spread the lost packets over 3) Nearly linear climb followed by sharp knee is characteristic 30 - Where the knee is tends to depend on RTT and loss rate 20 - Higher RTT or Higher loss means more streams to reach the knee - generally between 2 and 8, with 4 being a good rule of thumb 10 - As you can see, more streams is not always good Number of Streams Globus Allcock/ParallelTCP.pdf 22 11

12 Affect of TCP Buffer Size (iperf) RTT BW (Mbs) Calculated RTT (ms) Buffer Size (MB) 0 Globus Allcock/ParallelTCP.pdf 23 When should you use Parallel TCP? Engineered, private, semi private, or very over provisioned networks are good places to use parallel TCP. Bulk data transport. It makes no sense at all to use parallel TCP for most interactive apps. QOS: If you are guaranteed the bandwidth, use it Community Agreement: You are given permission to hog the network. Lambda Switched Networks: You have your own circuit, go nuts. Globus Allcock/ParallelTCP.pdf 24 12

13 Scalable TCP (2002) Uses exponential increase everywhere (in slow-start and congestion avoidance) Multiplicative decrease factor b = Introduced by Tom Kelly of Cambridge Information courtesy of Les Cottrell from the SLAC group at Stanford Source: TCP alternatives hicks.ppt 25 High-Speed TCP (2002) Behaves like Reno for small values of cwnd Above a chosen value of cwnd (default 38) a more aggressive function is used Uses a table to indicate by how much to increase cwnd when an ACK is received Available with web100 Introduced by Sally Floyd Information courtesy of Les Cottrell from the SLAC group at Stanford Source: TCP alternatives hicks.ppt 26 13

14 Binary Increase Control TCP (BIC TCP)(2004) Combine: Additive increase for large cwnd Binary increase for small cwnd Developed by Injong Rhee at NC State University Information courtesy of Les Cottrell from the SLAC group at Stanford Source: TCP alternatives hicks.ppt 27 Fast TCP FAST Fast AQM Scalable TCP FAST attempts to keep the number of packets queued in the network constant by adjusting the congestion window based on estimates of the current RTT and the minimum RTT. FAST differs from Vegas that also use current and minimum RRTs to adjust the congestion window in how the adjustment is made. FAST unlike most TCP variants has been patented as opposed to being standardized through IETF and commercialized through FastSoft that was acquired by Akamai in

15 FAST vs Linux TCP Test results (2002) Linux TCP txqueulen=100 Linux TCP txqueulen=10000 FAST Linux TCP txqueulen=100 Linux TCP txqueulen=10000 FAST flows Bmps Peta Thruput Mbps Distance km Delay ms MTU B Duration s Transfer GB , , CERN - Sunnyvale , , CERN - Sunnyvale , , CERN - Sunnyvale , , CERN - Sunnyvale , , CERN - Sunnyvale ,797 10, , CERN - Sunnyvale Path Mbps = 10 6 b/s; GB = 2 30 bytes; Delay = propagation delay Linux TCP expts: Jan 28 29, 2003 netlab.caltech.edu 29 Network Details for FAST vs Linux Test 7, 9, 10 FAST flows 3,948 km (Sylvain Ravot, caltech/cern) 1, 2 Linux/FAST flows 10,037 km ppt 30 15

16 FAST vs Linux Aggregate Throughput FAST Standard MTU Utilization averaged over 1hr 2G 92% Average utilization 1G 95% 48% 19% 27% 16% txq=100 txq=10,000 txq=100 txq=100 txq=10,000 txq=100 Linux TCP Linux TCP FAST Linux TCP Linux TCP FAST netlab.caltech.edu 31 FAST Aggregate Throughput FAST Standard MTU Utilization averaged over > 1hr 90% 88% 90% Average utilization 95% 92% 1hr 1hr 6hr 1.1hr 6hr 1 flow 2 flows 7 flows 9 flows 10 flows netlab.caltech.edu

17 Packet level Reno AIMD(1, 0.5) ACK: W W + 1/W Loss: W W 0.5W HSTCP (High Speed TCP) AIMD(a(w), b(w)) ACK: W Loss: W W + a(w)/w W b(w)w STCP (Stratified TCP) MIMD(a, b) FAST ACK: W W Loss: W W 0.125W basertt RTT : W W RTT basertt = minimum RTT observed determines fairness and convergence rate 33 Dynamic sharing: 3 flows 10 x FAST 10 x Linux throughput (Kbps) throughput (Kbps) sec Dynamic sharing on Dummynet capacity = 800Mbps delay=120ms iperf throughput Linux 2.4.x (HSTCP: UCL) sec 34 17

18 10 x FAST Steady throughput 10 x Linux 7 7 throughput (Kbps) throughput (Kbps) sec sec 10 x HSTCP 10 x STCP throughput (Kbps) throughput (Kbps) sec netlab.caltech.edu/fast/publications/i nfocom2004.ppt sec 35 Is large queue necessary for high throughput? research.microsoft.com/en us/.../tcpsummit/.../fast 30min microsoft.ppt 36 18

19 Ultrascale protocol development: FAST TCP FAST TCP Based on TCP Vegas Uses end to end delay and loss to dynamically adjust the congestion window Defines an explicit equilibrium Capacity = OC Gbps; 264 ms round trip latency; 1 flow BW use 30% BW use 40% BW use 50% BW use 79% Linux TCP Westwood+ BIC TCP FAST research.microsoft.com/en us/.../tcpsummit/.../fast 30min microsoft.ppt (Yang Xia, Caltech) 37 RTT Parallel and FAST TCP Parallel TCP appears to dramatically affect the RTT E.g. increases by RTT by 200ms (factor 20 for short distances) Implication: P TCP would impact apps. like Voice/IP Throughput (Mbps) 0 SLAC-Caltech P-TCP 16 stream RTT Time (secs) RTT (ms) Throughput (Mbps) SLAC-Caltech FAST TCP 1 stream RTT Time (secs) 1200 RTT (ms) 38 19

20 Lots of Small Files (LOSF) Optimization Megabit/sec Send 1 GB partitioned into equi sized files over 60 ms RTT, 1 Gbit/s WAN John Bresnahan et al., Argonne File size (Kbyte) (16MB TCP buffer) 39 Data Center TCP (DCTCP) Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, Murari Sridharan Microsoft Research Stanford University 40 20

21 Data Center Packet Transport Large purpose built DCs Huge investment: R&D, business Transport inside the DC TCP rules (99.9% of traffic) How s TCP doing? 41 TCP in the Data Center We ll see TCP does not meet demands of apps. Suffers from bursty packet drops, Incast [SIGCOMM 09],... Builds up large queues: Adds significant latency. Wastes precious buffers, esp. bad with shallow buffered switches. Operators work around TCP problems. Ad hoc, inefficient, often expensive solutions No solid understanding of consequences, tradeoffs 42 21

22 Roadmap What s really going on? Interviews with developers and operators Analysis of applications Switches: shallow buffered vs deep buffered Measurements A systematic study of transport in Microsoft s DCs Identify impairments Identify requirements Our solution: Data Center TCP 43 Case Study: Microsoft Bing Measurements from 6000 server production cluster Instrumentation passively collects logs Application level Socket level Selected packet level More than 150TB of compressed data over a month 44 22

23 TLA Partition/Aggregate Application Structure Picasso Art is 1. Deadline = 250ms 2. Art is a lie 3... Picasso Time is money 1. Strict deadlines (SLAs) 2. MLA MLA Deadline = 50ms 1. Art is a lie 2. The chief Missed deadline Lower quality result TLA = Top Level Aggregator MLA = Mid Level Aggegator simula.stanford.edu/~alizade/site/ DCTCP_files/DCTCP talk.pptx Everything The It I'd is Art Computers Inspiration your chief like Bad is to you a work enemy lie live artists can that in as are does of life imagine a makes copy. useless. creativity poor that exist, man is the real. is Deadline = 10ms They but can it ultimate with Good must realize only good lots artists find give seduction. the of sense. money. you truth. steal. working. answers. Worker Nodes 45 Generality of Partition/Aggregate The foundation for many large scale web applications. Web search, Social network composition, Ad selection, etc. Example: Facebook Partition/Aggregate ~ Multiget Aggregators: Web Servers Workers: Memcached Servers Internet Memcached Protocol Web Servers Memcached Servers 46 23

24 Partition/Aggregate (Query) Workloads Delay sensitive Short messages [50KB 1MB] (Coordination, Control state) Delay sensitive Large flows [1MB 50MB] (Data update) Throughput sensitive 47 Traffic Statistics for Bing Cluster Time between arrival of new work for the Aggregator (queries) and between background flows between servers (updates and short messages). research.microsoft.com/pubs/121386/dctcp public.pdf 24

25 Traffic Statistics for Bing Cluster PDF (Probability Distribution Function) of flow size distribution for background traffic. PDF of Total Bytes shows probability a randomly selected byte would come from a flow of given size. research.microsoft.com/pubs/121386/dctcp public.pdf Traffic Statistics for Bing Cluster The number of concurrent flows in which a Mid Level Aggregator (MLA) or worker node participates during 50 msec. The median for all flows is 36 (recall the MLA partitions queries among up to 43 worker nodes) and the percentile is over 1,600. One server was observed to have a median over 1,200 flows. For flows >1MB the median is 1 and the 75 percentile is 2. research.microsoft.com/pubs/121386/dctcp public.pdf 25

26 Impairments Incast (many to one communications) Incast increases the queuing delay of flows, and decreases application level throughput to far below the link bandwidth. The problem especially affects computing paradigms in which distributed processing cannot progress until all parallel threads in a stage complete. Examples of such paradigms include distributed file systems, web search, advertisement selection, and other applications with partition or aggregation semantics pdf Queue Buildup Buffer Pressure Derived from 51 Incast Worker 1 Worker 2 Synchronized mice collide. Caused by Partition/Aggregate. Aggregator Worker 3 Worker 4 TCP timeout RTO min = 300 ms RTO = Retransmission Time Out 52 26

27 Flows and switch buffer scenarios 6(a). Many small flows may overflow buffers and cause packet drops. 6(b). Large flows can cause large delays for small flows even if there is no packet loss. 6(c). Buffer allocation to large flows from the shared switch multi ported (expensive) memory can cause small allocation to short flows with consequent packet loss for short flows. research.microsoft.com/pubs/121386/dctcp public.pdf 53 Traffic Statistics for Bing Cluster A real incast event measured in a production environment. Timeline shows queries forwarded over 0.8ms, with all but one response returning over 12.4ms. That response is lost, and is retransmitted after RTO min (300ms). RTT+Queue estimates queue length on the port to the aggregator. Application limited responses to 2kB/query and added 10 ms jitter in worker responses to limit risk of buffer overflow. research.microsoft.com/pubs/121386/dctcp public.pdf 54 27

28 Incast Really Happens MLA Query Completion Time (ms) Requests are jittered over 10ms window. Jittering switched off around 8:30 am. 55 Incast Really Happens MLA Query Completion Time (ms) Jittering trades off median against high percentiles

29 Sender 1 Queue Buildup Big flows buildup queues. Increased latency for short flows. Receiver Sender 2 Measurements in Bing cluster For 90% packets: RTT < 1ms For 10% packets: 1ms < RTT < 15ms 57 Latency Queuing Delay Bing Cluster 3.pdf 58 29

30 Data Center Transport Requirements 1. High Burst Tolerance Incast due to Partition/Aggregate is common. 2. Low Latency Short flows, queries 3. High Throughput Continuous data updates, large file transfers The challenge is to achieve these three together. 59 Tension Between Requirements High Throughput High Burst Tolerance Low Latency Deep Buffers: Queuing Delays Increase Latency Reduced RTO min (SIGCOMM 09) Doesn t Help Latency Shallow Buffers: Bad for Bursts & Throughput DCTCP AQM RED: Avg Queue Not Fast Enough for Incast 60 30

31 Tension Between Requirements High Throughput High Burst Tolerance Low Latency Deep Buffers: Shallow Buffers: Queuing Delays Bad for Bursts & Objective: Increase Latency Throughput Low Queue Occupancy & High DCTCP Throughput Reduced RTO min (SIGCOMM 09) Doesn t Help Latency AQM RED: Avg Queue Not Fast Enough for Incast 61 The DCTCP Algorithm 62 31

32 Review: The TCP/ECN Control Loop Sender 1 ECN = Explicit Congestion Notification ECN Mark (1 bit) Receiver Sender 2 63 Small Queues & TCP Throughput: The Buffer Sizing Story Bandwidth delay product rule of thumb: A single flow needs buffers for 100% Throughput. Cwnd Buffer Size B Throughput 100% 64 32

33 Small Queues & TCP Throughput: The Buffer Sizing Story Bandwidth delay product rule of thumb: A single flow needs buffers for 100% Throughput. Appenzeller rule of thumb (SIGCOMM 04): Large # of flows: is enough. Cwnd Buffer Size B Throughput 100% 65 Small Queues & TCP Throughput: The Buffer Sizing Story Bandwidth delay product rule of thumb: A single flow needs buffers for 100% Throughput. Appenzeller rule of thumb (SIGCOMM 04): Large # of flows: is enough. Can t rely on stat mux benefit in the DC. Measurements show typically 1 2 big flows at each server, at most

34 Small Queues & TCP Throughput: The Buffer Sizing Story Bandwidth delay product rule of thumb: A single flow needs buffers for 100% Throughput. Appenzeller rule of thumb (SIGCOMM 04): Large # of flows: is enough. Can t rely on stat mux benefit in the DC. Measurements show typically 1 2 big flows at each server, at most 4. B Real Rule of Thumb: Low Variance in Sending Rate Small Buffers Suffice 67 Two Key Ideas 1. React in proportion to the extent of congestion, not its presence. Reduces variance in sending rates, lowering queuing requirements. ECN Marks TCP DCTCP Cut window by 50% Cut window by 40% Cut window by 50% Cut window by 5% 2. Mark based on instantaneous queue length. Fast feedback to better deal with bursts

35 This image cannot currently be displayed. 10/11/2013 Switch side: Data Center TCP Algorithm Mark packets when Queue Length > K. B Mark K Don t Mark Sender side: Maintain running average of fraction of packets marked (α). In each RTT: Adaptive window decreases: Note: decrease factor between 1 and pdf 70 35

36 DCTCP in Action (Kbytes) Setup: Win 7, Broadcom 1Gbps Switch Scenario: 2 long lived flows, K = 30KB 71 Why it Works 1.High Burst Tolerance Large buffer headroom bursts fit. Aggressive marking sources react before packets are dropped. 2. Low Latency Small buffer occupancies low queuing delay. 3. High Throughput ECN averaging smooth rate adjustments, low variance

37 Analysis How low can DCTCP maintain queues without loss of throughput? How do we set the DCTCP parameters? Need to quantify queue size oscillations (Stability). Window Size W*+1 W* (W*+1)(1 α/2) Time 73 Analysis How low can DCTCP maintain queues without loss of throughput? How do we set the DCTCP parameters? Need to quantify queue size oscillations (Stability). Window Size Packets sent in this RTT are marked. W*+1 W* (W*+1)(1 α/2) Time 74 37

38 Analysis How low can DCTCP maintain queues without loss of throughput? How do we set the DCTCP parameters? Need to quantify queue size oscillations (Stability). 85% Less Buffer than TCP 75 Evaluation Implemented in Windows stack. Real hardware, 1Gbps and 10Gbps experiments 90 server testbed Broadcom Triumph 48 1G ports 4MB shared memory Cisco Cat G ports 16MB shared memory Broadcom Scorpion 24 10G ports 4MB shared memory Numerous micro benchmarks Throughput and Queue Length Multi hop Queue Buildup Buffer Pressure Fairness and Convergence Incast Static vs Dynamic Buffer Mgmt Cluster traffic benchmark 76 38

39 Experiment implement 45 1G servers connected to a Triumph, a 10G server extern connection 1Gbps links K=20 10Gbps link K=65 Generate query, and background traffic 10 minutes, 200,000 background, 188,000 queries Metric: Flow completion time for queries and background flows. We use RTO min = 10ms for both TCP & DCTCP. 77 DCTCP Performance Queue length on 1 Gbps switch ports Throughput for 10 Gbps switch ports research.microsoft.com/pubs/121386/dctcp public.pdf 78 39

40 DCTCP Fairness and Convergence One receiver and 5 senders connected to a 1 Gbps 48 port 4 MB shared memory switch. Flows added and removed with 30 sec intervals. research.microsoft.com/pubs/121386/dctcp public.pdf DCTCP Incast Assessment Fixed buffers Static switch buffers of 100 packets and RTO either 10 ms (min possible in test setup) and 300 ms (standard in production cluster). One receiver aggregating 1MB from n servers with 1MB/n per server. 8ms is the minium delay caused by the 1 Gbps receiver connection to the switch (8 1000Mbps) research.microsoft.com/pubs/121386/dctcp public.pdf 40

41 DCTCP Incast Assessment Dynamic buffers One receiver aggregating 1MB from n servers with 1MB/n per server. 8ms is the minium delay caused by the 1 Gbps receiver connection to the switch (8 1000Mbps). Switch has 48 1 Gbps ports and 4 MB of shared multi ported memory. The dynamic shared switch memory allocation allocates 700kB (of 4MB) to the receiver port queue.) research.microsoft.com/pubs/121386/dctcp public.pdf DCTCP Incast Assessment: All to All Incast with Dynamic buffers Each node requests 25kB from each of 40 other nodes (1MB total). research.microsoft.com/pubs/121386/dctcp public.pdf 41

42 Baseline Background Flows Query Flows 83 Baseline Background Flows Query Flows Low latency for short flows

43 Background Flows Baseline Query Flows Low latency for short flows. High throughput for long flows. 85 Baseline Background Flows Query Flows Low latency for short flows. High throughput for long flows. High burst tolerance for query flows

44 Conclusions DCTCP satisfies all our requirements for Data Center packet transport. Handles bursts well Keeps queuing delays low Achieves high throughput Features: Very simple change to TCP and a single switch parameter. Based on mechanisms already available in Silicon. 87 Chapter 3: summary principles behind transport layer services: multiplexing, demultiplexing reliable data transfer flow control congestion control instantiation, implementation in the Internet UDP TCP next: leaving the network edge (application, transport layers) into the network core Slide from Kurose & Ross, 6 th Ed

45 References Congestion Avoidance and Control, V. Jacobson, M. J. Karels, Proc. of SIGCOMM 88, ACM, vol 18, no. 4, pp , August 1988, ftp://ftp.ee.lbl.gov/papers/congavoid.ps.z (TCP Tahoe) Modified TCP Congestion Avoidance Algorithm, V Jacobson, end2end interest mailing list, mail sent by V Jacoson Apr 30, 1990, ftp://ftp.isi.edu/end2end/end2end interest 1990.mail (TCP Reno) Startup Dynamics of TCP s Congestion Control and Avoidance Schemes, J. Hoe, MS Thesis, MIT, papers/hoe thesis.ps TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms, RFC 2001, January 1997, The NewReno Modification to TCP's Fast Recovery Algorithm, RFC 2582, April, 1999, TCP Vegas: end to end congestion avoidance on a global Internet, L.S. Barkmo, L. L. Peterson, vegas.ps Understanding Vegas: a duality model, S. H. Low, L. Peterson, L. Wang, J. of the ACM, Vol 49, no. 2, pp , March 2002, A Comparative Analysis of TCP Tahoe, Reno, New Reno, SACK and Vegas, 89 References cont d FAST TCP A new TCP/AQM for Stable Operation in Fast Networks, F. Paganini, Z. Wang, S. H. Low, J. C. Doyle, infocom03.pdf FAST Kernel: Background Theory and Experimental Results, C. Jin, D. Wei, S. H. Low, G. Buhrmaster, J. Bunn, D. H. Choe, R. L. A. Cottrell, J. C. Doyle, H. Newman, F. Paganini, S. Ravot, S. Singh, Scalable TCP: Improving Performance in Highspeed Wide Area Networks, T. Kelly, pdf (2002) HighSpeed TCP for Large Congestion Windows, S. Floyd, RFC 3649, December 2003, Binary Increase Congestion Control for Fast, Long Distance Networks, (BICTCP), L. Xu, K. Harfoush, I. Rhee, Infocomm 2004, DCTCP, DCTCP: Efficient Packet Transport for the Commoditized Data Center, M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, M. Sridharan, research.microsoft.com/pubs/121386/dctcp public.pdf Analysis of DCTCP: Stability, Convergence, and Fairness, M. Alizadeh, A. Javanmard, B. Prabhakar,