Data Center TCP (DCTCP)

Transcription

1 Data Center TCP (DCTCP) Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, Murari Sridharan Stanford University MicrosoD Research

2 Case Study: Microso7 Bing Measurements from 6000 server produchon cluster InstrumentaHon passively collects logs ApplicaHon- level Socket- level Selected packet- level More than 150TB of compressed data over a month 2

3 ParCCon/Aggregate ApplicaCon Structure Picasso Art is 1. TLA Deadline = 250ms 2. Art is a lie Time is money Strict deadlines (SLAs) Picasso 3... Missed deadline 1. Lower quality result 2. MLA MLA Deadline = 50ms 1. Art is a lie 2. The chief Everything The It I'd is Art Computers InspiraCon your chief like Bad is to you a work enemy lie live arcsts can that in as are does of imagine life a makes copy. useless. creacvity poor that exist, man is the real. is Deadline = 10ms They but can it ulcmate with Good must realize only good lots arcsts find give seduccon. the sense. of money. you truth. steal. working. answers. Worker Nodes 3

4 Workloads ParHHon/Aggregate (Query) Delay- sensicve Short messages [50KB- 1MB] (CoordinaCon, Control state) Delay- sensicve Large flows [1MB- 50MB] (Data update) Throughput- sensicve 4

5 Impairments Incast Queue Buildup Buffer Pressure 5

6 Incast Worker 1 Synchronized fan- in congeshon: Caused by ParCCon/Aggregate. Worker 2 Aggregator Worker 3 RTO min = 300 ms Worker 4 TCP Cmeout 6

7 Incast in Bing MLA Query CompleCon Time (ms) Requests are ji\ered over 10ms window. Jibering trades off median against high percencles. Ji\ering switched off around 8:30 am. 7

8 Queue Buildup Sender 1 Big flows buildup queues. Increased latency for short flows. Receiver Sender 2 Measurements in Bing cluster For 90% packets: RTT < 1ms For 10% packets: 1ms < RTT < 15ms 8

9 Data Center Transport Requirements 1. High Burst Tolerance Incast due to ParHHon/Aggregate is common. 2. Low Latency Short flows, queries 3. High Throughput ConHnuous data updates, large file transfers The challenge is to achieve these three together. 9

10 The DCTCP Algorithm

11 TCP Buffer Sizing Bandwidth- delay product rule of thumb: A single flow needs C RTT buffers for 100% Throughput. B C RTT B < C RTT Buffer Size B B Throughput 100% 100% 11

12 Buffer Sizing Impacts Latency Widespread concephon: increase link speed to reduce latency Eg. upgrade from 1Gbps to 10Gbps network. However, increasing link speed doesn t lower queuing delay, because: Switch buffers also need to be 10 Cmes larger and 10 Cmes faster. Buffer Size 10G x RTT To reduce latency, 1G we x RTT MUST reduce the buffering requirements of the transport protocol. Time 1G 10G 12

13 Reducing Buffer Requirements Appenzeller rule of thumb (SIGCOMM 04): Large # of flows: is enough. Cwnd Buffer Size Throughput 100% 13

14 Reducing Buffer Requirements Appenzeller rule of thumb (SIGCOMM 04): Large # of flows: is enough. Can t rely on stat- mux benefit in the DC. Measurements show typically 1-2 big flows at each server, at most 4. Real Rule of Thumb: Low Variance in Sending Rates Small Buffers Suffice. Both QCN & DCTCP reduce variance in sending rates. QCN: Explicit mulh- bit feedback. DCTCP: Implicit mulh- bit feedback from ECN marks. 14

15 DCTCP: Two Main Ideas 1. React in proporhon to the extent of congeshon, not its presence. Reduce window size based on fraccon of marked packets. ECN Marks TCP DCTCP Cut window by 50% Cut window by 40% Cut window by 50% Cut window by 5% 2. Mark based on instantaneous queue length. Fast feedback to be\er deal with bursts. Simplifies hardware. 15

16 DCTCP: Algorithm Switch side: Mark packets when Queue Length > K. B Mark K Don t Mark Sender side: Maintain running average of frac%on of packets marked (α). each RTT : F = # of marked ACKs Total # of ACKs α (1 g)α + gf AdapCve window decreases: W (1 α 2 )W Note: decrease factor between 1 and 2. 16

17 DCTCP vs TCP (Kbytes) Setup: Win 7, Broadcom 1Gbps Switch Scenario: 2 long- lived flows, K = 30KB 17

18 Why it Works 1. High Burst Tolerance Large buffer headroom bursts fit. Aggressive marking sources react before packets are dropped. 2. Low Latency Small buffer occupancies low queuing delay. 3. High Throughput ECN averaging smooth rate adjustments, low variance. 18

19 EvaluaCon Implemented in Windows stack. Real hardware, 1Gbps and 10Gbps experiments 90 server testbed Broadcom Triumph 48 1G ports 4MB shared memory Cisco Cat G ports 16MB shared memory Broadcom Scorpion 24 10G ports 4MB shared memory Numerous micro- benchmarks Throughput and Queue Length MulC- hop Queue Buildup Buffer Pressure Fairness and Convergence Incast StaCc vs Dynamic Buffer Mgmt Cluster traffic benchmark 19

20 Cluster Traffic Benchmark Emulate traffic within 1 Rack of Bing cluster 45 1G servers, 10G server for external traffic Generate query, and background traffic Flow sizes and arrival Hmes follow distribuhons seen in Bing Metric: Flow complehon Hme for queries and background flows. We use RTO min = 10ms for both TCP & DCTCP. 20

21 Baseline Background Flows Query Flows 21

22 Baseline Background Flows Query Flows Low latency for short flows. 22

23 Baseline Background Flows Query Flows Low latency for short flows. High throughput for long flows. 23

24 Baseline Background Flows Query Flows Low latency for short flows. High throughput for long flows. High burst tolerance for query flows. 24

25 Scaled Background & Query 10x Background, 10x Query 25