Advanced Networking Technologies

Transcription

1 Advanced Networking Technologies Chapter 3 Routers and Switches Size and Organization of Router Buffers This chapter is heavily based on material by Holger Karl (Universität Paderborn), AQM slides are based on material by Manolis Sifalakis (IBM) 1 Routers as ueue servers? Just suppose Traffic were Poissonian, routers were M/M/1 from [1] At a buffer size of B, link load ½, packet loss probability drops exponentially in buffer size B i.e., small buffers should avoid packet loss 2

2 Question for this chapter How big should the buffers in a router be? Goal: Maximum efficiency = keep every outgoing link busy all the time Argument for large buffers: Can compensate for senders not sending all the time, smooth out variations in ingoing traffic Argument for small buffers: TCP will fill every bottleneck buffer, will lead to large delays 3 Overview Brief recap: TCP congestion control scheme The single flow case Many flows: synchronized or not? Tiny buffers Optimizing Drop Behaviour 4

3 TCP ACK/self-clocking Suppose TCP has somehow determined a correct size of its congestion window Suppose also that the TCP source has injected this entire amount of data into the network but still has more data to send When to send more data? Only acceptable when there is space in the network again Space is available when packets leave the network Sender can learn about packets leaving the network by receiving an acknowledgement! Thus: ACK not only serves as a confirmation, but also as a permit to inject a corresponding amount of data into the network! ACK-clocking (self-clocking) behavior of TCP 5 AIMD Sawtooth pattern of TCP s offered load In summary: TCP uses an additive increase multiplicative decrease (AIMD) scheme Conseuence A TCP connection perpetually probes the network to check for additional bandwidth Will repeatedly exceed it and fall back, owing to multiplicative decrease Sawtooth pattern of TCP s congestion window/offered load This is still simplified; we have to introduce one more mechanism! Congestion window KB Time (seconds)

4 Quickly initialize a connection: Slow start Additive increase nice and well when operating close to network capacity Source Destination But takes a long time to converge to it for a new connection Starting at a congestion window of, say, 1 or 2 Idea: Quickly ramp up the congestion window in such an initialization phase Goal: double congestion window each RTT Implemented by: increase congestion window by one packet per arriving ACK Instead of just adding a single packet per RTT Note: Name slow start is historic it was slow compared to some earlier, even more aggressive scheme 7 7 TCP Tahoe congestion window dynamics (w/o anim) Initial congest ion threshhold Reset CW to 1, new threshold = CW/2 20=40/2 Initial congest ion window size Additive increase, congestion avoidance Slow start New slow start to new threshold, then linear increase SS 16, v

5 TCP Reno congestion control dynamics 9 Summary: TCP Reno sender congestion control Event State TCP Sender Action Comments ACK receipt for previously unacked data Slow Start (SS) CongWin = CongWin + MSS, If (CongWin > Threshold) set state to Congestion Avoidance Resulting in a doubling of CongWin every RTT ACK receipt for previously unacked data Congestion Avoidance (CA) CongWin = CongWin+MSS * (MSS/CongWin) or: CongWin += 1/CongWin Additive increase, resulting in increase of CongWin by 1 MSS every RTT Loss event detected by triple duplicate ACK SS or CA Threshold = CongWin/2, CongWin = Threshold, Set state to Congestion Avoidance Fast recovery, implementing multiplicative decrease. CongWin will not drop below 1 MSS. Timeout SS or CA Threshold = CongWin/2, CongWin = 1 MSS, Set state to Slow Start Enter slow start Duplicate ACK SS or CA Increment duplicate ACK count for segment being acked CongWin and Threshold not changed 10

6 TCP Reno, relevant scenario Assumptions: Sender is backlogged (always has packets to send) No ueuing delays for ACKs Only one bottleneck, maximum output rate C T p propagation delay, T ueuing delay Behaviour: congestion window W is always fully utilized by sender, it has W outstanding packets, it can send new packet after RTT Hence: Sending rate R = W/RTT = W/ (2T p + T ) We are interested in congestion avoidance mode, no timeouts, losses always detected via dup-acks ACK arrives: W += 1/W Loss detected: W /= 2 and has to wait for enough ACKs to arrive before it may continue sending! from [1] 11 Homework to deepen understanding of Reno Derive the period of a single, TCP Reno flow s sawtooth pattern (under the above assumptions) Hints Find a differential euation linking the change of the congestion window per RTT the size of the congestion window both expressed as a function of the ueue length at time t Solve the differential euation (think about suare roots) Assume router buffer is empty when the TCP flow starts Assume you have a router buffer size B Determine the sawtooth pattern period as a function of B and C 12

7 Overview Brief recap: TCP congestion control scheme The single flow case Many flows: synchronized or not? Tiny buffers Optimizing Drop Behaviour 14 Router buffer size for the single TCP-Reno flow Assume there is only a single TCP flow in the network and the above assumptions as well Question: How large should the router buffer be at the bottleneck link so that this link is always sending packets? from [1] 15

8 Congestion avoidance dynamics TCP increases congestion window, filling up the buffer eventually Sender will increase window a bit more, causing the first packet loss Sender detects loss (dup acks), cuts congestion window in half, and pauses Sender had W* packets in flight when loss detected (congestion window always fully used!) Note: W* measured in packets here, not bytes Now allowed to only have W*/2 packets in flight, must wait for W*/2 ACKs to arrive During this pause, bottleneck link sends packets from buffer Buffer needs to be just big enough to hold enough packets to cover this pause 16 ACK arrivals & buffer size So: how long does the sender pause? Router sends packets from buffer at rate C packets/s ACKs, hence, arrive at sender at rate C ACKs/s Sender has to wait for W*/2 ACKs, this takes W*/2/C seconds Hence: Router must be able to send packets for (at least) W*/2/C seconds from its buffer At sending rate C, that takes C (W*/2/C) = W*/2 packets! Example: I need to send 5 packets per second, for 10 seconds. That reuires 50 packets. Hence: Buffer B >= W*/2! 17

9 At what congestion window does a loss happen? Or: What is W*? Look at situation when the sender just resumes transmission Its current congestion window is W=W*/2, the router buffer is empty We are in congestion avoidance! For the bottleneck link to be fully utilized, the sender should ideally send at packet rate C But a TCP sender s rate R is determined by its congestion window and RTT; R = W/RTT Hence we reuire R = C = W/RTT RTT here is only 2T p, since buffer is empty, no ueuing delay Hence: C = W*/2 / 2T p! 18 What can we control? Buffer size! So two euations now: B >= W*/2 C = W*/2 / 2T p where C, T p are given, B can be chosen Summary: Just before packet drop: W = 2T p C + B Just after packet drop: W/2 = 2T p C Solve for buffer size B: B >= 2T p C Note: Why W*/2? Reno cuts in half! Actually: B = W max W min! Famous rule of thumb : buffer size should be at least the bandwidthdelay product of any flow 19

10 Behaviour of a single TCP flow from [1] 20 Overview Brief recap: TCP congestion control scheme The single flow case Many flows: synchronized or not? Tiny buffers Optimizing Drop Behaviour 21

11 Many TCP flows So far: only one flow in the network What happens if many flows multiplex a bottleneck link? Step 1: let all these flows be long-lived and synchronized in time Step 2: all long-lived, but non-synchronous Step 3: mix long-lived and short-lived flows 22 Many long-lived, synchronized TCP flows Synchronized: Flows are allowed to start at arbitrary times Under some circumstances: they will uickly synchronize, i.e., their congestion window evolution will become parallel, observing packets losses at nearly the same points in time Homework: Find out when and why synchronization happens! Due to synchronization, same dynamics, same buffer reuirements as a single flow 23

12 Many-long lived, desynchronized TCP flows What if flows do not synchronize? Effectively: adding up sawtooth patterns, shifted with respect to each other Smooth each other out Recall: Buffer needed for difference from (aggregated) congestion window minimum to (aggregated) congestion window minimum Smoothing: Difference max-min becomes smaller Less buffer?? 24 Long-lived, desynchronized Ansatz Assume n flows pass through router, congestion windows not synchronized (independent offsets) Each with window W i, aggregate window W = å i W i All senders backlogged again, i.e., all senders fully exploit their windows Where are the packets? Either in flight, in buffer, or lost: W(t) = Q(t) + 2T p C + ² Or looking at the ueue: Q(t) = W(t) 2T p C - ² W is a random variable, T p, C, ² constants We need random distribution of W Then we know distribution of Q Then we can bound probability of Q being empty 25

13 Long-lived, desynchronized Distribution of W Distribution of W i : Uniform U(2/3 E[W i ], 4/3 E[W i ]) Hint: Think about the sawtooth pattern, its triangular shape, express E[W i ] as function of W* Hence: standard deviation ¾ Wi = 1/( ) E[W i ] Property of uniform distribution Distribution of W: normal, by law of large numbers We want standard deviation of ueue length 26 Buffer empty How likely is it that link is utilized? I.e., how likely is it that buffer is empty? Or: buffer smaller than given threshold? Queue length is normal distributed, we know its standard deviation! we can bound deviations from mean Examples, with n= flows: Utilization: probability of not seeing a ueue shorter than some constant Typical core router: >= flows! 27

14 What about short flows? Slow flow: Never leaves slow start Assume: Access links have infinite speed Access slower than backhaul link: bursts are smoothed out anyway New flows arrive according to Poisson process Each flow generates a seuence of packet bursts With some distribution on how many bursts each flow generates Suppose flow wants to send l packets, it will send bursts 2, 4, 8, 16, 2 n-1, R where n= number of bursts, R = l mod (2 n-1-1) Burst arrivals are also Poisson process! Justification: Each flow small compared to overall load (!?) Hence: Router turns into an M/G/1 server 28 Queuing theory on M/G/1 Poisson arrival, general service time for jobs, one server Here job : a packet burst Let X be random describing job length = burst size; we need first and second moment Small homework: assume l is uniformly distributed up to some max, empirically derive these moments Known: Average number of jobs in ueue for M/G/1 for given E[X 2 ] and link load ½ E{T v } = 1 2(1 ) E{X 2 } E{X} Note: independent of number of flows! 29

15 Average ueue length for short flows Why peak at 14, not 16? 30 A first summary This analysis so far: A simplistic model seems to reuire router buffers on the order of the datarate-delay product to ensure high link utilization Understanding the behaviour of many flows sharing a link (irrespective of short- or long-lived), similar behaviour (link utilization!) can be achieved with substantially smaller routers Datarate-delay product, divided by suare root of number of concurrent flows In mixed traffic, long flows determine behaviour Open issues: Does this hold up in practice? a marketing problem! Number of flows in a real router? see later Sacrifice 100% utilization goal? 31

16 Overview Brief recap: TCP congestion control scheme The single flow case Many flows: synchronized or not? Tiny buffers Optimizing Drop Behaviour 32 Very small buffers? What happens if we make the buffers VERY small? In the model considered so far, huge packet losses, low utilization Losses come from TCP s packet bursts So let s reconsider our model what if TCP would not burst packets into the network whenever it has the chance? Pacing TCP in protocol, or implicitly by slow access links 33

17 Model assumptions for very small buffers Model: TCP sends packets via a Poisson process, with rate W/RTT TCP uses some maximum window size W max Hence, maximum rate is W max /RTT We have N flows, all have same RTT Many flows share a link of capacity C Shared link is overprovisioned by factor 1/½, i.e., C (1/½) N W max /RTT Hence, highest possible throughput is ½ C Goal: We are willing to tolerate a utilization < 100% = Achieved throughput / ½ C 34 Buffer for utilization To obtain utilization of under Poissonian packet arrivals, we need a buffer size of: Why not surprising: It depends on and ½, also on W max Why surprising: It does NOT depend on link capacity! Nor on number of flows! Example: W max = 64 (Linux), ½ =0.5, =0.9: B = 15! 35

18 Tiny buffers, realistic model Poissonian packet arrival is simply wrong But gives interesting insight; although proof complicated In practice: use Paced TCP TCP sends packets not in a burst when window opens, but at fixed rate of W/RTT. Still works surprisingly well! From [2] Buffer Size 36 From:McKeown, SIGCOMM Keynote, % ~ 90% Throughput Integrated all-optical buffer [UCSB 2008] On-chip buffers Smaller design Lower power 20 pkts log(w) RT T C p N RT T C Number of packets 10Gb/s WAN ~50 25,000 2,500,000 37

19 Overview Brief recap: TCP congestion control scheme The single flow case Many flows: synchronized or not? Tiny buffers Optimizing Drop Behaviour 38 Active Queue Management As Queues build up, RTT increases, congestion becomes hard to remediate (notice this has the opposite effect from the original intend) Takes longer for endpoints to react (TxWin adjustment) ACK spacing can get compressed increasing burstiness of transmissions The larger the capacity of the ueue The more detached are the endpoints from the truth about congestion The more likely that the ueues become persistent: bufferbloat (long standing delay is treated as the norm) Active Queue Management Advanced Networking (SS 17): 02 - Input-buffered switches 39

20 RED Model Max thresh Pkt Arrival Min thresh Maintain Exp Moving Avg (EWMA) of ueue length Byte mode vs. Packet mode depending if Tx delay is a function of packet length or not For each packet arrival Drop probability 1.0 P max Actual Avg Queue Queue Length Length min th max th Avg ueue length if (avg < min th ) do nothing if (max th avg) mark(packet) if (min th avg < max th ) calculate probability P m mark(p m, packet) Marked packets are either dropped or ECN flagged 40 RED Parameters EWMA of Queue size computed at every packet arrival (not periodically!) avg now = w * len now + (1-w ) * avg prev Special condition if ueue was idle, i.e. len now = 0 Same as if it had been 100% link utilization with 0 ueue. Approx. that m small packets could have been processed m = (t now t last_arrival )/ pkt_size nominal avg now = (1-w ) m * avg prev Packet marking probability is a function of % of the between thresholds utilitisation P m = P max * (avg now min th ) / (max th min th ) 41

21 Issues with RED: configurability avg is an EWMA w adjusts the lag and trend and window of averaging Short window: fast sensing but vulnerable to transients Long window: slow adaptation min th adjusts the power of the network Too close to 0: the ueue is likely to have idle periods (bandwidth not used) Too far from 0: increases path latency, delays feedback signal to endpoints max th min th adjusts the freuency of marking Too small : the AQM becomes spasmodic in its reaction, forces flows to sync Must be larger that the typical avg increase in an RTT A visual analogy: which vessel size for which sea condition? Think traffic bursts like the swell of the sea (height, length) Think of the AQM as a speed boat or a a tanker depending on swell 42 Issues with RED: configurability Average Queue size oscillation Difficult to control congestion when many flows esp. unresponsive ones Qlen max 8 flows Actual Avg Qlen max 32 flows Actual Avg 0 time 0 time Many optimizations 43

22 RED & variants FRED Fair RED or Flow RED: fairness among flow types Flow type differentiation based on ueue use non-adaptive (UDP), fragile (sensitive to loss), robust per-active-flow (present in the ueue buffer) accounting and lossregulation All flows entitled to admit min packets without loss Adjust min based on avg. per flow occupancy (avgc) of ueue Set upper capacity per flow type to max and count violations (strikes). Then for freuent violators lower max = avgc (to punish them more often) 44 RED & variants CHOKE CHOose and Kill unresponsive flows... or.. CHOose and Keep responsive flows Compare incoming packet with random selected packet in ueue Aggressive flows become more likely to select Max thresh Pkt Arrival Min thresh Select Randomly Flow ID Match No Apply RED (variant) Yes 45

23 RED & variants ARED Adaptive RED: reduces delay variance + introduces parameter auto-tune Adapt P max periodically, slowly, and with an AIMD policy Fix max th = 3 * min th Fix w = 1 - exp(1/link_capacity) Goal: To keep avg constant at about (max th + min th )/2 46 RED & variants SRED Stabilised RED eliminate need of avg (and w ) P m = f(inst. ueue length, # of active flows, rank of flow) Zombie list: history of K seen flows with Hit counters. On packet arrival pick Zombie flow randomly if flows match hits++ else replace Zombie with prob. Pr Statistical counting of flows based on hit freuency Rank of flow = Hit counter on match 47

24 RED & variants BLUE Putting past insights in new light Avoid parameter tuning nightmare Avoid effects of avg ueue fluctuation on AQM Adaptive marking probability P m = f (packet loss, link idle events) [Pkt loss] if (t now t last_arrival > freeze_period) P m = P m + d1 [Idle link] if (t now t last_arrival > freeze_period) P m = P m - d2 d1 >> d2 : faster reaction to congestion up-rise than decrease freeze_period works as a sort-of A/D discretizer (step-hold) Filters out high-fre transient oscillations Adjusts parameter at packet arrivals times mod a fixed uantum SFBlue uses ideas of SFQ and FRED to discriminate flows 48 CoDel Controlled delay Time-based model of ueue dynamics instead of a spatial one Objective: Monitor how long the Min ueue length remains above a threshold (desired Min) Less descriptive than Avg. Min ueue length, Yet sufficient, and simpler computationally Metric: Sojourn time as a measure of instantaneous ueue length How long packets stay in the ueue: Time delta between a packet s departure and arrival time Works with a single ueue or multiple ueues Works for variable link rates (e.g. wireless links) well statistically speaking! Simple to measure, easy to implement 49

25 CoDel Controlled delay On packet arrival: timestamp(packet) 1. dropping rate increases linearly with the measured RTT 2. if a last epoch of drops is not too long in the past, the last good drop rate is also remembered! On packet departure: sojourn = now packet.tstamp if (sojourn < Target) if (drp_mode == 1) drp_mode = 0 exit_drp = now if (now exit_drp >= Interval) drp_count = 0 else // sojourn > Target if (drp_mode==0 && now - exit_drp < Interval) drp_mode = 1 if (now >= next_drp) drp(packet) drp_count++ next_drp = now + Interval/srt(drp_count) else if (drp_mode = 0) // start drop drp_mode = 1 drp(packet) drp_count = 1 next_drp = now + Interval/srt(drp_count) else // already in drp_mode if (now >= next_drp) drp(packet) drp_count++ next_drp = now + Interval/srt(drp_count) 50 CoDel Controlled delay On packet arrival: timestamp(packet) Sojourn falls below Target Only reset drop rate memory if Sojourn is below Target for Interval Sojourn above Target again after temporary improvement, resume last drop rate Sojourn above Target first time, after Interval, start dropping Sojourn continues to remain above Target, continue dropping On packet departure: sojourn = now packet.tstamp if (sojourn < Target) if (drp_mode == 1) drp_mode = 0 exit_drp = now if (now exit_drp >= Interval) drp_count = 0 else // sojourn > Target if (drp_mode==0 && now - exit_drp < Interval) drp_mode = 1 if (now >= next_drp) drp(packet) drp_count++ next_drp = now + Interval/srt(drp_count) else if (drp_mode = 0) // start drop drp_mode = 1 drp(packet) drp_count = 1 next_drp = now + Interval/srt(drp_count) else // already in drp_mode if (now >= next_drp) drp(packet) drp_count++ next_drp = now + Interval/srt(drp_count) 51

26 CoDel Controlled delay Significantly less configuration magic involved Interval: const ( 1RTT) Target (delay): const max{ euiv of 1-2 packets worth of ueue, 5% of worst case RTT} Drop/Mark rate: const acceleration in Interval inverse-suare-root progression à linear increase of drops per RTT dropping speed up is independent of ueue accumulation speed!?!? f_codel combines CoDel with SFQ treats different traffic classes fair gives starting flows a good head start Sojourn measurement does reuire blocking of the ueue! by contrast to ueue length averaging Drop/Mark at the head of the ueue not the tail 52 Conclusions Buffer size, sending rate and RTT interact closely for link utilization maximization Rule of thumb for single flows or synchronized flows Non-synchronized flows: reuired buffer only suare-root of number of flows of rule-of-thumb size Sacrifice utilization, and even tiny, fixed-size buffers suffice! Main relevance: makes it feasible to use optical buffers Large buffers should/can be controlled, but not simple! 53

27 References 1. G. Appenzeller, I. Keslassy, and N. McKeown, Sizing router buffers, in ACM SIGCOMM Computer Communication Review, 2004, vol. 34, no. 4, p M. Enachescu, Y. Ganjali, A. Goel, N. McKeown, and T. Roughgarden, Routers with very small buffers, Proceedings IEEE INFOCOM TH IEEE International Conference on Computer Communications, Nichols, Kathleen; Jacobson, Van, "Controlling Queue Delay". ACM Queue. August