Scaling routers: Where do we go from here?

Transcription

1 Scaling routers: Where do we go from here? HPSR, Kobe, Japan May 28 th, 2002 Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford University May 28th, 2002 Nick McKeown

2 000 Relative performance increase 00 Router capacity x2.2/8 months 0 Moore s law x2/8 m May 28th, 2002 Nick McKeown 2

3 000 Relative performance increase 00 Router capacity x2.2/8 months 0 Moore s law x2/8 m DRAM access rate x./8 m May 28th, 2002 Nick McKeown 3

4 Router vital statistics Cisco GSR Juniper M60 9 Capacity: 60Gb/s Power: 4.2kW Capacity: 80Gb/s Power: 2.6kW 6ft 3ft 2ft 2.5ft May 28th, 2002 Nick McKeown 4

5 Relative performance increase Router capacity x2.2/8 months Internet traffic x2/yr 5x May 28th, 2002 Nick McKeown 5

6 Fast (large) routers Big POPs need big routers POP with large routers POP with smaller routers Interfaces: Price >$200k, Power > 400W About 50-60% of interfaces are used for interconnection within the POP. Industry trend is towards large, single router per POP. May 28th, 2002 Nick McKeown 6

7 Job of router architect For a given set of features: Maximize capacity, C s.. t Power, P< 5kW Volume, V < 2m 3 May 28th, 2002 Nick McKeown 7

8 Mind the gap Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption. Our options:. Make routers simple 2. Use more parallelism 3. Use more optics May 28th, 2002 Nick McKeown 8

9 Mind the gap Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption. Our options:. Make routers simple 2. Use more parallelism 3. Use more optics May 28th, 2002 Nick McKeown 9

10 Make routers simple We tell our students that Internet routers are simple. All routers do is make a forwarding decision, update a header, then forward packets to the correct outgoing interface. But I don t understand them anymore. List of required features is huge and still growing, Software is complex and unreliable, Hardware is complex and power-hungry. May 28th, 2002 Nick McKeown 0

11 OC92c linecard Router linecard Optics Physical Layer Framing & Maintenance 30M gates 2.5Gbits of memory m 2 $25k cost, $200k price. Lookup Tables Packet Processing Buffer & State Memory Buffer Mgmt & Scheduling Buffer Mgmt & Scheduling Buffer & State Memory Scheduler May 28th, 2002 Nick McKeown

12 Things that slow routers down 250ms of buffering Requires off-chip memory, more board space, pins and power. Multicast Affects everything! Complicates design, slows deployment. Latency bounds Limits pipelining. Packet sequence Limits parallelism. Small internal cell size Complicates arbitration. DiffServ, IntServ, priorities, WFQ etc. Others: IPv6, Drop policies, VPNs, ACLs, DOS traceback, measurement, statistics, May 28th, 2002 Nick McKeown 2

13 An example: Packet processing CPU Instructions per minimum length packet since May 28th, 2002 Nick McKeown 3

14 Reducing complexity Conclusion Need aggressive reduction in complexity of routers. Get rid of irrelevant requirements and irrational tests. It is not clear who has the right incentive to make this happen. Else, be prepared for core routers to be replaced by optical circuit switches. May 28th, 2002 Nick McKeown 4

15 Mind the gap Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption. Our options:. Make routers simpler 2. Use more parallelism 3. Use more optics May 28th, 2002 Nick McKeown 5

16 Use more parallelism Parallel packet buffers Parallel lookups Parallel packet switches Things that make parallelism hard: Maintaining packet order, Making throughput guarantees, Making delay guarantees, Latency requirements, Multicast. May 28th, 2002 Nick McKeown 6

17 Parallel Packet Switches Router rate, R rate, R 2 rate, R N N rate, R k Bufferless May 28th, 2002 Nick McKeown 7

18 Characteristics Advantages k memory bandwidth k lookup/classification rate k routing/classification table size With appropriate algorithms Packets remain in order, 00% throughput, Delay guarantees (at least in theory). May 28th, 2002 Nick McKeown 8

19 Mind the gap Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption. Our options:. Make routers simpler 2. Use more parallelism 3. Use more optics May 28th, 2002 Nick McKeown 9

20 All-optical routers don t make sense A router is a packet-switch, and so requires A switch fabric, Per-packet address lookup, Large buffers for times of congestion. Packet processing/buffering infeasible with optics A typical 0 Gb/s router linecard has 30 Mgates and 2.5 Gbits of memory. Research Problem How to optimize the architecture of a router that uses an optical switch fabric? May 28th, 2002 Nick McKeown 20

21 00Tb/s optical router Stanford University Research Project Collaboration 4 Professors at Stanford (Mark Horowitz, Nick McKeown, David Miller and Olav Solgaard), and our groups. Objective To determine the best way to incorporate optics into routers. Push technology hard to expose new issues. Photonics, Electronics, System design Motivating example: The design of a 00 Tb/s Internet router Challenging but not impossible (~00x current commercial systems) It identifies some interesting research problems May 28th, 2002 Nick McKeown 2

22 00Tb/s optical router Electronic Linecard # Electronic Linecard #625 60Gb/s Gb/s Optical Switch Arbitration Gb/s 40Gb/s Line termination IP packet processing Packet buffering 40Gb/s 40Gb/s Line termination IP packet processing Packet buffering Request Grant 40Gb/s (00Tb/s = 625 * 60Gb/s) May 28th, 2002 Nick McKeown 22

23 Research Problems Linecard Memory bottleneck: Address lookup and packet buffering. Architecture Arbitration: Computation complexity. Switch Fabric Optics: Fabric scalability and speed, Electronics: Switch control and link electronics, Packaging: Three surface problem. May 28th, 2002 Nick McKeown 23

24 60Gb/s Linecard: Packet Buffering DRAM DRAM DRAM 60 Gb/s 60 Gb/s SRAM Queue Manager Problem Packet buffer needs density of DRAM (40 Gbits) and speed of SRAM (2ns per packet) Solution Hybrid solution uses on-chip SRAM and off-chip DRAM. Identified optimal algorithms that minimize size of SRAM (2 Mbits). Precisely emulates behavior of 40 Gbit, 2ns SRAM. klamath.stanford.edu/~nickm/papers/ieeehpsr200.pdf May 28th, 2002 Nick McKeown 24

25 The Arbitration Problem A packet switch fabric is reconfigured for every packet transfer. At 60Gb/s, a new IP packet can arrive every 2ns. The configuration is picked to maximize throughput and not waste capacity. Known algorithms are too slow. May 28th, 2002 Nick McKeown 25

26 Approach We know that a crossbar with VOQs, and uniform Bernoulli i.i.d. arrivals, gives 00% throughput for the following scheduling algorithms: Pick a permutation uar from all permutations. Pick a permutation uar from the set of size N in which each input-output pair (i,j) are connected exactly once in the set. From the same set as above, repeatedly cycle through a fixed sequence of N different permutations. Can we make non-uniform, bursty traffic uniform enough for the above to hold? May 28th, 2002 Nick McKeown 26

27 2-Stage Switch External Inputs Internal Inputs External Outputs N N N Spanning Set of Permutations Spanning Set of Permutations Recently shown to have 00% throughput Mild conditions: weakly mixing arrival processes C.S.Chang et al.: May 28th, 2002 Nick McKeown 27

28 2-Stage Switch External Inputs Spanning Set of Permutations Internal Inputs at () bt () Spanning Set of Permutations External Outputs N N π () t π () t 2 qt () N E[ bt ()] = E[ π() tat ()] = E[ π() t] E[ a() t ] = eλ. N Long-term, service opportunities exceed arrivals: τ lim bt () π 2( t) = eλ e 0. τ τ N N t= May 28th, 2002 Nick McKeown 28

29 Problem: Unbounded Mis-sequencing External Inputs 2 Internal Inputs External Outputs 2 N N N Spanning Set of Permutations Spanning Set of Permutations Side-note: Mis-sequencing is maximized when arrivals are uniform. May 28th, 2002 Nick McKeown 29

30 Preventing Mis-sequencing Small Coordination Buffers & FFF Algorithm Large Congestion Buffers N N N Spanning Set of Permutations Spanning Set of Permutations The Full Frames First algorithm: Keep packets ordered and Guarantees a delay bound within the optimum Infocom 02: klamath.stanford.edu/~nickm/papers/infocom02_two_stage.pdf May 28th, 2002 Nick McKeown 30

31 Example Optical 2-stage Switch Linecards Phase Lookup Buffer Phase 2 Lookup Buffer 2 Lookup Buffer 3 Idea: Use a single-stage twice May 28th, 2002 Nick McKeown 3

32 Example Passive Optical 2-Stage Switch Ingress Linecard R/N R/N Midstage Linecard R/N R/N Egress Linecard Ingress Linecard 2 Midstage Linecard 2 Egress Linecard 2 Ingress Linecard n R/N Midstage Linecard n R/N Egress Linecard n It is helpful to think of it as spreading rather than switching. May 28th, 2002 Nick McKeown 32

33 2-Stage spreading Buffer stage N N N May 28th, 2002 Nick McKeown 33

34 Passive Optical Switching Integrated AWGR or diffraction grating based wavelength router Ingress Linecard λ, K, λn λ,, K λn Midstage Linecard λ, K, λn λ,, K λn Egress Linecard Ingress Linecard 2 λ, K, λn 2 2 λ,, K λn 2 Midstage Linecard 2 λ, K, λn 2 2 λ,, K λn 2 Egress Linecard 2 Ingress Linecard n λ, K, λn n n λ, K, λn n Midstage Linecard n λ, K, λn n n λ, K, λn n Egress Linecard n May 28th, 2002 Nick McKeown 34

35 00Tb/s Router Optical links Optical Switch Fabric Racks of 60Gb/s Linecards May 28th, 2002 Nick McKeown 35

36 Racks with 60Gb/s linecards DRAM DRAM DRAM SRAM Queue Manager Lookup DRAM DRAM DRAM SRAM Queue Manager Lookup May 28th, 2002 Nick McKeown 36

37 Additional Technologies Demonstrated or in development ¾ ¾ ¾ May 28th, 2002 Nick McKeown TX TX TX data gen TX TX-DLL ¾ 40 µm TX-PLL ¾ Chip to chip optical interconnects with total power dissipations of several mw. Demonstration of wavelength division multiplexed chip interconnect. Integrated laser modulators. 8Gsample/s serial links. Low-power variable power supply serial links. Integrated array waveguide routers. data gen Digital Sliding Controller ¾ Buck Converter Power Transistors Testing Interface TX/RX Feedback Biasing RXPLL PRBS RX RX-DLL RX PRBS 37

38 Mind the gap Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption. Our options:. Make routers simpler 2. Use more parallelism 3. Use more optics May 28th, 2002 Nick McKeown 38

39 Some predictions about core Internet routers The need for more capacity for a given power and volume budget will mean: Fewer functions in routers: Little or no optimization for multicast, Continued overprovisioning will lead to little or no support for QoS, DiffServ,, Fewer unnecessary requirements: Mis-sequencing will be tolerated, Latency requirements will be relaxed. Less programmability in routers, and hence no network processors. Greater use of optics to reduce power in switch. May 28th, 2002 Nick McKeown 39

40 What I believe is most likely The need for capacity and reliability will mean: Widespread replacement of core routers with transport switching based on circuits: Circuit switches have proved simpler, more reliable, lower power, higher capacity and lower cost per Gb/s. Eventually, this is going to matter. Internet will evolve to become edge routers interconnected by rich mesh of WDM circuit switches. May 28th, 2002 Nick McKeown 40