048866: Packet Switch Architectures

Transcription

1 048866: Packet Switch Architectures Output-Queued Switches Deterministic Queueing Analysis Fairness and Delay Guarantees Dr. Isaac Keslassy Electrical Engineering, Technion

2 Outline. Output Queued Switches 2. Terminology: Queues and Arrival Processes. 3. Deterministic Queueing Analysis 4. Output Link Scheduling Spring Packet Switch Architectures 2

3 Generic outer Architecture Data Hdr Header Processing Lookup IP Address Update Header Queue Packet Address Address Table Table Buffer Buffer Memory Memory Data Hdr Header Processing Lookup IP Address Update Header 2 2 N times line rate Queue Packet Address Address Table Table Buffer Buffer Memory Memory N times line rate Data Hdr Header Processing Lookup IP Address Update Header N N Queue Packet Address Address Table Table Buffer Buffer Memory Memory Spring Packet Switch Architectures 3

4 Simple Output-Queued (OQ) Switch Model Link, ingress Link, egress Link 2 Link rate, Link rate, Link Link 3 Link 2, ingress Link 2, egress Link 4 Link 3, ingress Link 3, egress Link 4, ingress Link 4, egress Spring Packet Switch Architectures 4

5 How an OQ Switch Works Output Queued (OQ) Switch Spring Packet Switch Architectures 5

6 OQ Switch Characteristics Arriving packets are immediately written into the output queue, without intermediate buffering. The flow of packets to one output does not affect the flow to another output. Spring Packet Switch Architectures 6

7 OQ Switch Characteristics An OQ switch is work conserving: an output line is always busy when there is a packet in the switch for it. OQ switches have the highest throughput, and lowest average delay. We will also see that the rate of individual flows, and the delay of packets can be controlled. Spring Packet Switch Architectures 7

8 The Shared-Memory Switch A single, physical memory device Link, ingress Link, egress Link 2, ingress Link 3, ingress Link N, ingress Link 2, egress Link 3, egress Link N, egress Spring Packet Switch Architectures 8

9 OQ vs. Shared-Memory Memory Bandwidth Buffer Size Spring Packet Switch Architectures 9

10 Memory Bandwidth (OQ) In?? Out? In??? Total: (N+) Out In??? Out Spring Packet Switch Architectures 0

11 Memory Bandwidth Basic OQ switch: Consider an OQ switch with N different physical memories, and all links operating at rate bits/s. In the worst case, packets may arrive continuously from all inputs, destined to just one output. Maximum memory bandwidth requirement for each memory is (N+) bits/s. Shared Memory Switch: Maximum memory bandwidth requirement for the memory is 2N bits/s. Spring Packet Switch Architectures

12 OQ vs. Shared-Memory Memory Bandwidth Buffer Size Spring Packet Switch Architectures 2

13 Buffer Size In an OQ switch, let Q i (t) be the length of the queue for output i at time t. Let M be the total buffer size in the shared memory switch. Is a shared-memory switch more bufferefficient than an OQ switch? Spring Packet Switch Architectures 3

14 Buffer Size Answer: Depends on the buffer management policy Static queues: Same as OQ switch For no loss, needs Q i (t) M/N for all i Dynamic queues: Better than OQ switch (multiplexing effects) Needs Spring Packet Switch Architectures 4

15 How fast can we make a centralized shared memory switch? 2 Shared Memory 5ns SAM 5ns per memory operation Two memory operations per packet Therefore, upper-bound of: N 200 byte bus Spring Packet Switch Architectures 5

16 Outline. Output Queued Switches 2. Terminology: Queues and Arrival Processes. 3. Deterministic Queueing Analysis 4. Output Link Scheduling Spring Packet Switch Architectures 6

17 Queue Terminology A(t), λ S,µ D(t) Arrival process, A(t): Q(t) In continuous time, usually the cumulative number of arrivals in [0,t], In discrete time, usually an indicator function as to whether or not an arrival occurred at time t=nt. λ is the arrival rate: the expected number of arriving packets (or bits) per second. Queue occupancy, Q(t): Number of packets (or bits) in queue at time t. Spring Packet Switch Architectures 7

18 Queue Terminology A(t), λ S,µ D(t) Q(t) Service discipline, S: Indicates the sequence of departures: e.g. FIFO/FCFS, LIFO, Service distribution: Indicates the time taken to process each packet: e.g. deterministic, exponentially distributed service time. µ is the service rate: the expected number of served packets (or bits) per second. Departure process, D(t): In continuous time, usually the cumulative number of departures in [0,t], In discrete time, usually an indicator function as to whether or not a departure occurred at time t=nt. Spring Packet Switch Architectures 8

19 More terminology Customer: Queueing theory usually refers to queued entities as customers. In class, customers will usually be packets or bits. Work: Each customer is assumed to bring some work which affects its service time. For example, packets may have different lengths, and their service time might be a function of their length. Waiting time: Time that a customer waits in the queue before beginning service. Delay: Time from when a customer arrives until it has departed. Spring Packet Switch Architectures 9

20 Arrival Processes Deterministic arrival processes: E.g. arrival every second; or a burst of 4 packets every other second. A deterministic sequence may be designed to be adversarial to expose some weakness of the system. andom arrival processes: (Discrete time) Bernoulli i.i.d. arrival process: Let A(t) = if an arrival occurs at time t, where t = nt, n=0,, A(t) = w.p. p and 0 w.p. -p. Series of independent coin tosses with p-coin. (Continuous time) Poisson arrival process: Exponentially distributed interarrival times. Spring Packet Switch Architectures 20

21 Adversarial Arrival Process Example for Knockout Switch Memory write bandwidth = k. < N. 2 3 N If our design goal was to not drop packets, then a simple discrete time adversarial arrival process is one in which:. A (t) = A 2 (t) = = A k+ (t) =, and 2. All packets are destined to output t mod N. Spring Packet Switch Architectures 2

22 Bernoulli arrival process Memory write bandwidth = N. A (t) A 2 (t) 2 A 3 (t) 3 A N (t) N Assume A i (t) = w.p. p, else 0. Assume each arrival picks an output independently, uniformly and at random. Some simple results follow:. Probability that at time t a packet arrives to input i destined to output j is p/n. 2. Probability that two consecutive packets arrive to input i = probability that packets arrive to inputs i and j simultaneously = p 2. Spring Packet Switch Architectures 22

23 Outline. Output Queued Switches 2. Terminology: Queues and Arrival Processes. 3. Deterministic Queueing Analysis 4. Output Link Scheduling Spring Packet Switch Architectures 23

24 Simple Deterministic Model Q(t) Cumulative number of bits that arrived up until time t. A(t) Cumulative number of bits A(t) D(t) Service process D(t) Cumulative number of departed bits up until time t. time Properties of A(t), D(t): A(t), D(t) are non-decreasing A(t) D(t) Spring Packet Switch Architectures 24

25 Simple Deterministic Model Cumulative number of bits A(t) Q(t) d(t) D(t) time Queue occupancy: Q(t) = A(t) - D(t). Queueing delay d(t): time spent in the queue by a bit that arrived at time t (assuming that the queue is served FCFS/FIFO). Spring Packet Switch Architectures 25

26 Discrete-Time Queueing Model Discrete-time: at each time-slot n, first a(n) arrivals, then d(n) departures. Cumulative arrivals: Cumulative departures: Queue size at end of time-slot n: Q(n)=A(n)-D(n) Spring Packet Switch Architectures 26

27 Work-Conserving Queue In?? Out? In??? Out In??? Out Spring Packet Switch Architectures 27

28 Work-Conserving Queue In? In? Out In? Spring Packet Switch Architectures 28

29 Work-Conserving Queue We saw that an output queue in an OQ switch is work-conserving: it is always busy when there is a packet for it. Let A(n), D(n) and Q(n) denote the arrivals, departures and queue size of some output queue. Let be the queue departure rate (amount of traffic that can depart at each time-slot). After arrivals at start of time-slot n, this output link contains Q(n-)+a(n) amount of traffic. Spring Packet Switch Architectures 29

30 Work-Conserving Output Link Case : Q(n-)+a(n) everything is serviced, nothing is left in the queue. Case 2: Q(n-)+a(n) > exactly amount of traffic is serviced, Q(n)=Q(n-)+a(n) -. Lindley s Equation: Q(n) = max(q(n-)+a(n)-,0) = (Q(n-)+a(n)-) + Note: to find cumulative departures, use: D(n)=A(n)-Q(n) Spring Packet Switch Architectures 30

31 Outline. Output Queued Switches 2. Terminology: Queues and Arrival Processes. 3. Deterministic Queueing Analysis 4. Output Link Scheduling Spring Packet Switch Architectures 3

32 The problems caused by FIFO output-link scheduling Fairness. A FIFO queue does not take fairness into account it is unfair. (A source has an incentive to maximize the rate at which it transmits.) Delay Guarantees 2. It is hard to control the delay of packets through a network of FIFO queues. Spring Packet Switch Architectures 32

33 Fairness A 00 Mb/s 0 Mb/s 0.55 Mb/s. Mb/s C B 0.55 Mb/s e.g. an http flow with a given (IP SA, IP DA, TCP SP, TCP DP) What is the fair allocation: (0.55Mb/s, 0.55Mb/s) or (0.Mb/s, Mb/s)? Spring Packet Switch Architectures 33

34 Fairness A 00 Mb/s 0 Mb/s. Mb/s D B C 0.2 Mb/s What is the fair allocation? Spring Packet Switch Architectures 34

35 Max-Min Fairness A common way to allocate flows N flows share a link of rate C. Flow f wishes to send at rate W(f), and is allocated rate (f).. Pick the flow, f, with the smallest requested rate. 2. If W(f) < C/N, then set (f) = W(f). 3. If W(f) > C/N, then set (f) = C/N. 4. Set N = N. C = C (f). 5. If N>0 goto. Spring Packet Switch Architectures 35

36 Max-Min Fairness An example W(f ) = 0. W(f 2 ) = 0.5 W(f 3 ) = 0 W(f 4 ) = 5 C ound : Set (f ) = 0. ound 2: Set (f 2 ) = 0.9/3 = 0.3 ound 3: Set (f 4 ) = 0.6/2 = 0.3 ound 4: Set (f 3 ) = 0.3/ = 0.3 Spring Packet Switch Architectures 36

37 Water-Filling Analogy esource equested/ Allocated Customers (sorted by requested amount) Spring Packet Switch Architectures 37

38 Max-Min Fairness How can an Internet router allocate different rates to different flows? First, let s see how a router can allocate the same rate to different flows Spring Packet Switch Architectures 38

39 Fair Queueing. Packets belonging to a flow are placed in a FIFO. This is called per-flow queueing. 2. FIFOs are scheduled one bit at a time, in a round-robin fashion. 3. This is called Bit-by-Bit Fair Queueing. Flow Bit-by-bit round robin Classification Flow N Scheduling Spring Packet Switch Architectures 39

40 Bit-by-Bit Weighted Fair Queueing (WFQ) Likewise, flows can be allocated different rates by servicing a different number of bits for each flow during each round. (f ) = 0. (f 2 ) = 0.3 (f 3 ) = 0.3 (f 4 ) = 0.3 Also called Generalized Processor Sharing (GPS) (with infinitesimal amount of flow instead of bits ) Spring Packet Switch Architectures 40 Order of service for the four queues: f, f 2, f 2, f 2, f 3, f 3, f 3, f 4, f 4, f 4, f, C

41 GPS Guarantees An output link implements GPS with k sessions, allocated rates (f ),, (f k ). Assume session i is continually backlogged. For all j, let S j (t,t 2 ) be the amount of service received by session j between times t and t 2. Then: S i (t,t 2 ) (f i ) (t 2 -t ) For all j i, Spring Packet Switch Architectures 4

42 Packetized Weighted Fair Queueing (WFQ) Problem: We need to serve a whole packet at a time. Solution:. Determine at what time a packet p would complete if we served flows bit-by-bit. Call this the packet s finishing time, F p. 2. Serve packets in the order of increasing finishing time. Also called Packetized Generalized Processor Sharing (PGPS) Spring Packet Switch Architectures 42

43 Understanding Bit-by-Bit WFQ 4 queues, sharing 4 bits/sec of bandwidth, Equal Weights Time A = 4 B = 3 C2 = C = D2 = 2 D = Weights : ::: A2 = 2 A = 4 Time A2, C3 arrive D, C Depart at = B = 3 D C B A C3 = 2 C2 = C = D2 = 2 D = ound Weights : ::: A2 = 2 A = 4 Time C2 Departs at =2 B = 3 D2 C2 B A D C B A C3 = 2 C2 = C = D2 = 2 D = ound 2 ound Weights : ::: Spring Packet Switch Architectures 43

44 Understanding Bit-by-Bit WFQ 4 queues, sharing 4 bits/sec of bandwidth, Equal Weights Time D2, B Depart at =3 A2 = 2 A = 4 B = 3 D2 C3 B A D2 C2 B A D C B A C3 = 2 C2 = C = D2 = 2 D = ound 3 ound 2 ound Weights : ::: Time A2 C3, A Depart at =4 Departs at =6 A2 = 2 A = 4 B = 3 A2 A2 C3 A D2 C3 B A D2 C2 B A D C B A C3 = 2 C2 = C = D2 = 2 D = 6 5 ound 4 ound 3 ound 2 ound Weights : ::: Sort packets Time Departure order for packet by packet WFQ: Sort by finish round of packets A2 = 2 A = 4 B = 3 A2 A2 C3 C3 A A A A D2 D2 B B B C2 D C C3 = 2 C2 = C = D2 = 2 D = Weights : ::: Spring Packet Switch Architectures 44

45 Understanding Bit-by-Bit WFQ 4 queues, sharing 4 bits/sec of bandwidth, Weights 3:2:2: Time A = 4 B = 3 C2 = C = D2 = 2 D = Weights : 3:2:2: Time A2 = 2 A = 4 3 B = 3 2 B A A A C3 = 2 C2 = C = D2 = 2 D = 2 ound Weights : 3:2:2: Time D, C2, C Depart at = A2 = 2 A = 4 3 B = 3 2 D C2 C B B A A A C3 = 2 C2 = C = D2 = 2 D = 2 ound Weights : 3:2:2: Spring Packet Switch Architectures 45

46 Understanding Bit-by-Bit WFQ 4 queues, sharing 4 bits/sec of bandwidth, Weights 3:2:2: A2 = 2 A = 4 Time 3 B, A2 A Depart at =2 B = 3 2 B A2 A2 A D C2 C B B A A A C3 = 2 C2 = C = D2 = 2 D = 2 ound 2 ound Weights : 3:2:2: Time D2, C3 Depart at =2 A2 = 2 A = 4 3 B = 3 2 D2 D2 C3 C3 B A2 A2 A D C2 C B B A A A C3 = 2 C2 = C = D2 = 2 D = 2 3 ound 2 ound Weights : 3:2:2: Sort packets Time Departure order for packet by packet WFQ: Sort by finish time of packets A2 = 2 A = 4 3 B = 3 2 D2 D2 C3 C3 B B B A2 A2 A A A A D C2 C C3 = 2 C2 = C = 2 D2 = 2 D = Weights : ::: Weights : 3:2:2: Spring Packet Switch Architectures 46

47 WFQ is complex There may be hundreds to millions of flows; the linecard needs to manage a FIFO per flow. The finishing time must be calculated for each arriving packet, Packets must be sorted by their departure time. Naively, with m packets, the sorting time is O(logm). In practice, this can be made to be O(logN), for N active flows: Egress linecard 2 Packets arriving to egress linecard Calculate F p 3 Find Smallest F p Departing packet N Spring Packet Switch Architectures 47

48 Deficit ound obin (D) [Shreedhar & Varghese, 95] An O() approximation to WFQ Step : Step 2,3,4: emaining credits Active packet queues Active packet queues Quantum Size = 200 It is easy to implement Weighted D using a different quantum size for each queue. Often-adopted solution in practice Spring Packet Switch Architectures 48

49 The problems caused by FIFO output-link scheduling Fairness. A FIFO queue does not take fairness into account it is unfair. (A source has an incentive to maximize the rate at which it transmits.) Delay Guarantees 2. It is hard to control the delay of packets through a network of FIFO queues. Spring Packet Switch Architectures 49

50 Deterministic analysis of a router queue Model of router queue Cumulative bytes FIFO delay, d(t) A(t) D(t) A(t) µ D(t) Q(t) Q(t) µ time Spring Packet Switch Architectures 50

51 So how can we control the delay of packets? Assume continuous time, bit-by-bit flows for a moment. Let s say we know the arrival process, A f (t), of flow f to a router. 2. Let s say we know the rate, (f) that is allocated to flow f. 3. Then, in the usual way, we can determine the delay of packets in f, and the buffer occupancy. Spring Packet Switch Architectures 5

52 WFQ Scheduler A (t) Flow (f ), D (t) Classification Flow N WFQ Scheduler A N (t) (f N ), D N (t) Assume a WFQ scheduler Spring Packet Switch Architectures 52

53 WFQ Scheduler Cumulative bytes A f (t) D f (t) (f) time We know the allocated rate (f) If we knew the arrival process, we would know the packet delay Key idea: constrain the arrival process Spring Packet Switch Architectures 53

54 Let s say we can bound the arrival process Cumulative bytes A (t) ρ Number of bytes that can arrive in any period of length t is bounded by: σ + ρt σ time This is called (σ,ρ) regulation Spring Packet Switch Architectures 54

55 (σ,ρ) Constrained Arrivals and Minimum Service ate d max Cumulative bytes A (t) D (t) B max σ ρ (f ) For no packet loss, B σ. If ( f ) ρ, then d( t) σ / ( f ). time Theorem [Parekh,Gallager 93]: If flows are leaky-bucket constrained, and routers use WFQ, then end-to-end delay guarantees are possible. Spring Packet Switch Architectures 55

56 The leaky bucket (σ,ρ) regulator Tokens at rate, ρ Token bucket size, σ Packets Packets Packet buffer One byte (or packet) per token Spring Packet Switch Architectures 56

57 Making the flow conform to (σ,ρ) regulation Leaky bucket as a shaper Variable bit-rate compression Tokens at rate, ρ C Token bucket size σ To network bytes ρ bytes bytes time time time Spring Packet Switch Architectures 57

58 Checking up on the flow Leaky bucket as a policer outer Tokens at rate, ρ Token bucket size σ From network C bytes ρ bytes time time Spring Packet Switch Architectures 58

59 QoS outer Classifier Policer Policer Per-flow Queue Per-flow Queue Scheduler Classifier Policer Policer Per-flow Queue Per-flow Queue Scheduler emember: These results assume that it is an OQ switch! Spring Packet Switch Architectures 59

60 eferences. [GPS] A. K. Parekh and. Gallager A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Single Node Case, IEEE Transactions on Networking, June [D] M. Shreedhar and G. Varghese Efficient Fair Queueing using Deficit ound obin, ACM Sigcomm, 995. Spring Packet Switch Architectures 60