Challenges for Future Interconnection Networks Hot Interconnects Panel August 24, Dennis Abts Sr. Principal Engineer

Transcription

1 Challenges for Future Interconnection Networks Hot Interconnects Panel August 24, 2006 Sr. Principal Engineer

2 Panel Questions How do we build scalable networks that balance power, reliability and performance Are there new architectures on the horizon that will swoop in and save the day? What critical reliability assumptions that we now take for granted are likely not to hold for future interconnection networks? 2

3 Complexity-Effective Routers How do design and build future routers? Conventional routers do not scale to high radix O(N 2 ) complexity of crossbar Hierarchical router organization 8x8 tiled architecture Distributed routing algorithm (input tile, crosspoint tile, and output tile) Internal speedup in the router Implemented in a 90nm CMOS standard-cell ASIC 800MHz 17 months from concept to tapeout with working first-silicon 3

4 Prior Cray Machines XT3 (2004 thru present) 3-D torus Scales to 32K procs 4 VCs 4.8 GB/s per direction T3D (early 90s) Dual-bristled 3-D torus Scales to 2048 procs 4 VCs 300 MB/s per direction T3E (mid, late 90s) 3-D torus Scales to 2048 procs 5 VCs (adaptive routing) 600 MB/s per direction X1 (2003 thru present) 16-sliced, dual-bristled 2-D torus Scales to 4096 procs 4 VCs 1.6 GB/s per direction August 24, 2006 Hot Interconnects

5 Motivation for High-Radix Routers During the past 15 years, the total bandwidth per router has increased by nearly three orders of magnitude Packet sizes have remained roughly constant By increasing the radix of the router, both the latency and cost of the network can be reduced. Utilize bandwidth by building networks with many narrow links rather than fewer fat links 5

6 YARC (Yet Another Router Chip) 64 full-duplex links Each link is 3 lanes wide Fast data rate 6.25 Gb/s 2.4Tb/s total bandwidth Scaling to high-radix requires a different microarchitecture Hierarchical organization makes high-radix router feasible YARC High-Radix Router 64 ports 3-lanes per port 6.25 Gbps/port 2.4 Tb/s total 6

7 YARC Microarchitecture Regular 8x8 array of tiles Easy to lay out chip No global arbitration All decisions local Simple routing Small 8-entry routing table per tile High routing throughput for small packets Hierarchical organization Input buffers Row buffers Column buffers 7

8 YARC Pipeline 25-stage pipeline distributed among the tiles 800MHz memories required latches on the input and outputs Each sub-chip floorplanning unit required latches on inputs and outputs 8

9 YARC Chip Implemented in a 90nm CMOS standard-cell ASIC technology 192 SerDes on the chip! (64 ports x 3-bits per port) 6.25Gbaud data rate 17mm 17mm 9

10 Reliability trends CMOS devices are becoming increasingly susceptible to external effects Electrical noise, process variation, natural radiation Robust system design techniques at various levels of abstraction in the system From circuit-level to system-level error handling protocols 10

11 Reliability trends CMOS devices are becoming increasingly susceptible to external effects Electrical noise, process variation, natural radiation Robust system design techniques at various levels of abstraction in the system From circuit-level to system-level error handling protocols 11

12 Network Fault Tolerance The router must provide graceful degradation in the presence of faults: A failed network cable or connector A faulty router chip (a YARC that stops responding) A noisy high-speed serial lane that causes excessive retransmissions Auto-degrade network links Dynamically map out a faulty lane and keep operating, albeit at a lower bandwidth Tolerates a failed lane 12

13 Network Fault Tolerance CRC protection through the router data path to detect soft errors in the buffering as the packet traverses the router switch Flexible routing table to specify the set of allowable output ports prevents the use of a faulty link. 13

14 Summary Challenges: Building complexity-effective routers Regular array of tiles simplifies design and verification Internal speedup in the router (8x speedup) YARC: ~17 months from concept to tapeout First silicon works Reliability Fully protected data paths with 16-bit CRC Error detection, but not correction Reliable link-level packet retry End-to-end packet retry too expensive for large-scale O(N 2 ) 14

15 Thank You.