Data Center Transport Mechanisms. EE384M Spring 2011

Transcription

1 Data Center Transport Mechanisms EE384M Spring 2011

2 What are Data Centers? Large enterprise networks; convergence of High speed LANs: 10, 40, 100 Gbps Ethernet Storage networks: Fibre Channel, Infiniband Related idea: Cloud CompuMng Outgrowth of high- performance compumng networks with integrated storage and server virtualizamon support Driven by Economics: One network, not many Low capex and opex Economics: Server umlizamon Resource pooling, virtualizamon, server migramon, high- speed interconnect fabrics Savings in power consumpmon Unified management of network of servers allows server and job scheduling Security 2 Storage and processing of data within a single autonomous domain

3 Why do Conges7on Control? Switches and routers send congesmon signals to end- systems to regulate the amount of network traffic; we dismnguish two types of congesmon Transient: Due to random fluctuamons in packet arrival rate EffecMvely handled with buffers, packet drops/marks or link- level PAUSE: IEEE 802.1Qbb Sustained: When link bandwidth suddenly drops or when new flows arrive Switches need to signal the sources to slow their sending rate: IEEE 802.1Qau Or, at Layer 3, use TCP, BIC TCP, DCTCP, 3

4 Stability CongesMon control algorithms aim to deliver high throughput, maintain low latencies/backlogs, be fair to all flows, be simple to implement and easy to deploy Performance is related to stability of control loop Stability refers to the non- oscillatory or non- exploding behavior of congesmon control loops. In real terms, stability refers to the non- oscillatory behavior of the queues at the switch. If the switch buffers are short, oscillamng queues can overflow (hence drop packets/pause the link) or underflow (hence lose umlizamon) In either case, links cannot be fully umlized, throughput is lost, flow transfers take longer So stability is an important property, especially for networks with high bandwidth- delay products operamng with shallow buffers 4

5 Conges7on control in the Internet Queue management schemes (e.g. RED) at the links signal congesmon by either dropping or marking packets using ECN TCP at end- systems uses these signals to vary the sending rate There exists a rich history of algorithm development, control- theoremc analysis and detailed simulamon of queue management schemes and congesmon control algorithms for the Internet Jacobson, Floyd et al, Kelly et al, Low et al, Srikant et al, Misra et al, Katabi et al

6 TCP- - RED: A basic control loop TCP TCP TCP TCP TCP: Slow start + CongesMon avoidance p CongesMon avoidance: AIMD No loss: increase window by 1; Pkt loss: cut window by half min th max th q avg RED: Drop probability, p, increases as the congesmon level goes up 6

7 TCP Dynamics CongesMon Window ~ Rate Cwnd Cwnd/2 CongesMon message recd Time

8 TCP- - RED: Analy7cal model 1/R C - q - Time Delay p TCP Control RED Control 8

9 TCP- - RED: Analy7cal model Users: dw i (t) dt = 1 RTT i (t) W i(t) 1.5 * W i(t) p(t) RTT i (t) Network: W: window size; RTT: round trip time; C: link capacity" q: queue length; q a : ave queue length p: drop probability" 9 *By V. Misra, W. Dong and D. Towsley at SIGCOMM 2000 *Fluid model concept originated by F. Kelly, A. Maullo and D. Tan at Jour. Oper. Res. Society, 1998

10 10 Accuracy of analy7cal model

11 TCP- - RED: Stability analysis Given the differenmal equamons, in principle one can figure out whether the TCP- - RED control loop is stable However, the differenmal equamons are very complicated 3rd or 4th order, nonlinear, with delays There is no general theory, specific case treatments exist Linearize and analyze Linearize equamons around the (unique) operamng point Analyze resultant linear, delay- differenmal equamons using Nyquist or Bode theory End result: Design stable control loops Determine stability condimons (RTT limits, number of users, etc) Obtain control loop parameters: gains, drop funcmons, 11

12 Instability of TCP- - RED As the bandwidth- delay- product increases, the TCP- - RED control loop becomes unstable Parameters: 50 sources, link capacity = 9000 pkts/sec, TCP- - RED Source: S. Low et. al. Infocom

13 Feedback Stabiliza7on in High BDP Networks Many new congesmon control algorithms have been proposed which use feedback stabilizamon to restore stability in these high bandwidth- delay product environments; ; e.g. High- Speed TCP, FAST, XCP, RCP, BIC- TCP, etc. The two main flavors of feedback stabilizamon used are: 1. Determine lags (round trip Mmes), apply the correct gains for the loop to be stable (e.g. FAST, XCP, RCP, HS- TCP). 2. Include higher order queue derivamves in the congesmon informamon fed back to the source (e.g. REM/PI, XCP, RCP). We shall see that BIC- TCP and QCN use a different method which we call the Averaging Principle. First, we describe the QCN algorithm.

14 QuanMzed CongesMon NoMficaMon (QCN): CongesMon control for Ethernet

15 Ethernet vs. the Internet Some significant differences 1. No per- packet acks in Ethernet, unlike in the Internet Not possible to know round trip Mme! So congesmon must be signaled to the source by switches Algorithm not automamcally self- clocked (like TCP) 2. Links can be paused; i.e. packets may not be dropped 3. No sequence numbering of L2 packets 4. Sources do not start transmission gently (like TCP slow- start); they can potenmally come on at the full line rate of 10Gbps 5. Ethernet switch buffers are much smaller than router buffers (100s of KBs vs 100s of MBs) 6. Most importantly, algorithm should be simple enough to be implemented completely in hardware Note: QCN has Internet relamves- - - BIC- TCP at the source and the REM/PI controllers

16 The Control Loop Feedback S 1 D 1 S N CongesMon Point D N ReacMon Points

17 QCN Conges7on Point Dynamics Consider the single- source, single- switch loop below Q eq Source Conges7on Point (Switch) Dynamics: Sample packets, compute feedback (Fb), quanmze Fb to 6 bits, and reflect only nega%ve Fb values back to ReacMon Point with a probability propormonal to Fb. Fb = - (Q- Q eq + w. dq/dt ) = - (queue offset + w.rate offset) ReflecMon Probability Pmin P max Fb

18 QCN Reac7on Point Dynamics TR CR Fast Recovery Target Rate Rate Rd Rd/2 Rd/4 Rd/8 AcMve Probing QCN ~ BIC- TCP Current Rate CongesMon message recd Time QCN- AIMD ~ TCP We choose this method. Rate CongesMon message recd Time

19 Stability: QCN- AIMD vs QCN

20 Fluid Model for QCN Assume N flows pass through a single queue at a switch. State variables are TR i (t), CR i (t), q(t). dtr i dt dcr i dt dq dt = = (TR i (t) CR i (t)) CR i (t τ) p(t τ) + (1 p(t τ)) 500 α CR i (t τ) 100 = (G d F b (t τ)cr i (t)) CR i (t τ) p(t τ) + TR i (t) C i (t) 2 N i=1 CR i(t) C F b (t) = q(t) Q eq + w ( CR Cp i (t) C) s p(t) = p s 1{F b (t) > 0} N i=1 CR i (t τ)p(t τ) (1 p(t τ)) 20

21 Accuracy: Equa7ons vs ns2 simula7on Throughput (Gbps) Queue Size (packets) (a) N = 10 sources, RTT = 100µs 2 ns Simulation Fluid Model Time (s) ns Simulation Fluid Model Time (s) Throughput (Gbps) Queue Size (packets) (b) N = 10 sources, RTT = 500µs 2 ns Simulation Fluid Model Time (s) ns Simulation Fluid Model Time (s)

22 Summary The algorithm has been extensively tested in deployment scenarios of interest Esp. interoperability with link- level PAUSE and TCP The theoremcal development is interesmng, but most notably because QCN (and BIC- TCP) display strong stability in the face of increasing lags, or, equivalently in high bandwidth- delay product networks While atempmng to understand why these schemes perform so well, we have uncovered a method for improving the stability of any congesmon control scheme; we present this next

23 The Averaging Principle

24 ( AP ) The Averaging Principle A source in a congesmon control loop is instructed by the network to decrease or increase its sending rate (randomly) periodically AP: a source obeys the network whenever instructed to change rate, and then voluntarily performs averaging as below TR = Target Rate CR = Current Rate

25 Recall: QCN does 5 steps of Averaging The Fast Recovery pormon of QCN, there are 5 steps of averaging In fact, QCN and BIC- TCP are the Ave Prin applied to TCP! TR CR Target Rate Rate Rd Rd/2 Rd/4 Rd/8 AcMve Probing Current Rate CongesMon message recd Time

26 A Generic Control Example As an example, we consider the plant transfer funcmon: P(s) = (s+1)/(s s s+0.6)

27 Step Response Basic AP, No Delay

28 Step Response Basic AP, Delay = 8 seconds

29 Step Response Two- step AP, Delay = 14 seconds

30 Two- step AP is even more stable than Basic AP Step Response Two- step AP, Delay = 25 seconds

31 Understanding the AP As menmoned earlier, the two major flavors of feedback compensamon are: 1. Determine lags, chose appropriate gains 2. Feedback higher derivamves of state We prove that the AP is sense equivalent to both of the above! This is great because we don t need to change network routers and switches And the AP is really very easy to apply; no lag- dependent opmmizamons of gain parameters needed

32 AP Equivalence: Single Source Case Source does AP Fb Regular source 0.5 Fb T dfb/dt Systems 1 and 2 are discrete- Mme models for an AP enabled source, and a regular source respecmvely. Main Result: Systems 1 and 2 are algebraically equivalent. That is, given idenmcal input sequences, they produce idenmcal output sequences. Therefore the AP is equivalent to adding a derivamve to the feedback and reducing the gain! Thus, the AP does both known forms of feedback compensamon without knowing RTTs or changing switch implementamons

33 AP vs PD No Delay

34 AP vs PD Delay = 8 seconds

35 Summary of AP The AP is a simple method for making many control loops (not just congesmon control loops) more robust to increasing lags Gives a clear understanding as to the reason why the BIC- TCP and QCN algorithms have such good delay tolerance: they do averaging repeatedly There is a theorem which deals explicitly with the QCN- type loop VariaMons of the basic principle are possible; i.e. average more than once, average by more than half- way, etc The theory is fairly complete in these cases