IntraChip Optical Networks for a Future Supercomputer-on-a-Chip

Transcription

1 for a Future Supercomputer-on-a-Chip Jeffrey Kash, IBM Research

2 Acknowledgements IBM Research: Yurii Vlasov, Clint Schow, Will Green, Fengnian Xia, Jose Moreira, Eugen Schenfeld, Jose Tierno, Alexander Rylyakov, Columbia University: Keren Bergman, Luca Carloni, Rick Osgood Cornell University: David Albonesi, Alyssa Apsel, Michal Lipson, Jose Martinez UC Santa Barbara: Daniel Blumenthal, John Bowers 2

3 Outline Optics in today s HPCs Trends in microprocessor design Multi-core designs for power efficiency Vision for future (ICON) 3D stack of logic, memory, and global optical interconnects Required devices and processes Low power and small footprint 3

4 Today s High Performance Server Clusters: Racks are mainly electrically connected, but going optical All Electrical NEC Earth Simulator (during installation) -All copper -Next gen: Optics All Optical Snap 12 module 12 Tx or Rx at 2.5Gb/s placement at back of rack Real systems are s of server racks and several racks of switches Rack-to-rack interconnects ( 100m) now moving to optics Interconnects within racks ( 5m) now primarily copper Over time, optics will increasingly replace copper at shorter and shorter distances Backplane and card interconnects ( 1m) after rack-to-rack Trend will accelerate as bitrates (in the media) increase and costs come down 2.5Gb/s 5 Gb/s 10Gb/s 20Gb/s(?) Target ~ $1/Gb/s 4 IBM Federation Switch for ASCI Purple (LLNL) (backside of a switch rack) - Copper (bulk, bend, weight, air cooling) - Optical (very organized but more expensive)

5 Beyond bitrate, density is a major driver of optics. Cables Connectors HM-Zd 10Gbps connector 40 differential pairs (25mm wide) High-Speed Copper Cabling Fiber-Ribbon MT fiber ferrule 48 fibers extendable to 72 or 96 (7mm wide) 5 Electrical Transmission Lines and Optical Waveguides mils 400μm 6 35 x 35μm 62.5μm pitch But optics must be packaged deep within the system to achieve density improvements

6 Packaging of Optical Interconnects is Critical Better to put optics close to logic rather than at the card edge Avoids distortion, power, & cost of electrical link on each end of optical link Breaks through pin-count limitation of multi-chip modules (MCMs) Operation at 10 Gb/s: equalization required Operation to >15 Gb/s: no equalization required Bandwidth limited by # of pins Opto module ~2cm 1.7cm traces Laser+driver IC NIC Ceramic 1cm Flex Organic card fiber 1.7cm ~2cm traces NIC Ceramic Organic card Organic card Good: Optics Optics on-card on-card Opto module Laser+driver IC 1cm Flex >12.5cm traces (with or w/o via stubs) fiber Optics on-mcm Optical bulkhead connector 6 Colgan, et. al., Direct integration of dense parallel optical interconnects on a first level package for high-end servers, ECTC 2005, 55 th, pp , Vol. 1., 31 May-3 June 2005.

7 Current architecture: Electronic Packet Switching Current architecture (electronic switch chips, interconnected by electrical or optical links, in multi-stage networks) works well now--- Scalable BW & applicationoptimized cost Multiple switches in parallel Modular building blocks many identical switch chips & links) -- but challenging in the future Switch chip throughput stresses the hardest aspects of chip design I/O & packaging Multi-stage networks will require multiple E-O-E conversions N-stage Exabyte/s network = N*Exabytes/s of cost N*Exabytes/s of power Central switch racks Mare Nostrum, Barcelona Supercomputing Center 7

8 Possible new architecture: Optical Circuit Switching (Optics is not electronics, maybe a different architecture can use it better) All-Optical Packet Switches are hard e.g., IBM/Corning OSMOSIS project Expensive, and required complex electrical control network No optical memory or optical logic Probably not cost-competitive against electronic packet switches, even in But Optical Circuit Switches (~10millisecond switching time) are available today Several technologies (MEMS, piezo-, thermo-,..) Low power OCS power essentially zero, compared to electronic switch no extra O-E-O conversion But require single-mode optics In ~2015, with silicon photonics, ~1nsec switching time Does 6 orders of magnitude make approach more suitable to general-purpose computing? Scalable Optical Circuit Switch (OCS) OCS Concept Input fiber (one channel shown) Output fibers OCS 2-axis MEMS Mirror (one channel shown) MEMS-based OCS HW is commercially available (Calient, Glimmerglass,..) 20 ms switching time <100 Watts 8

9 Outline Optics in today s HPCs Trends in microprocessor design Multi-core designs for power efficiency Vision for future (ICON) 3D stack of logic, memory, and global optical interconnects Required devices and processes Low power and small footprint 9

10 Chip MultiProcessors (CMPs) IBM Cell, Sun Niagara, Intel Montecito, (note that the processors on the chip are not identical) IBM Cell: 10 Parameter Value Technology process 90nm SOI with low-κ dielectrics and 8 metal layers of copper interconnect Chip area 235mm^2 Number of transistors ~234M Operating clock frequency 4Ghz Power dissipation ~100W Percentage of power dissipation due to 30-50% global interconnect Intra-chip, inter-core communication Tbps, 2Gb/sec/lane (four shared bandwidth buses, 128 bits data + 64 bits address each) I/O communication bandwidth Tbps (includes external memory)

11 but perhaps a hierarchical design of several cores grouped into a supercore will emerge ~2017 Multiple supercores on a chip Electrical communication within supercore Optical communications between supercores After Moray McLaren, HP Labs 11

12 Theme: How to continue to get exponential performance increase over time (Moore s Law extension) from silicon ICs even though CMOS scaling by itself is no longer enough Performance (log) (Moore s Law extension) Exa-scale (~2017) Peta-scale (~2012) Tera-scale (today) Communications and Architecture Increased # of Processors Uniprocessor performance Transistors Can Si photonics provide this performance increase? (original Moore s Law applies here) Time (linear) IBM Cell Processor 9 processors, ~200GFLOPs On- and Off-chip BW~100GB/sec (0.5B/FLOP) BW requirements must scale with System Performance, ~1Byte/FLOP 12

13 Outline Optics in today s HPCs Trends in microprocessor design Multi-core designs for power efficiency Vision for future (ICON) 3D stack of logic, memory, and global optical interconnects Required devices and processes Low power and small footprint 13

14 Inter-core communication trends network on chip INTEL Polaris 2007 Research Chip: 100 Million Transistors 80 cores (tiles) 275mm 2 i.e., 3D Integration (why not go to optical plane, too?) Higher BW and lower Power with Optics? 14

15 Photonics in Multi-Core Processors Intra-Chip Communications Network Photonics changes the rules OPTICS: Modulate/receive ultra-high bandwidth data stream once per communication event Broadband switch fabric is uses very little power highly scalable Off-chip and on-chip can use essentially the same technology Much more off-chip BW available ELECTRONICS: Buffer, receive and re-transmit at every switch Off chip is pin-limited and really power hungry TX RX TX RX TX RX TX RX TX RX TX RX 15

16 Integration Concept 3D layer stacking will be prevalent in the 22nm timeframe Intra-chip optics can take advantage of this technology Photonics layer (with supporting electrical circuits) more easily integrated with high performance logic and memory layers Layers can be separately optimized for performance and yield Optical Off-chip Interconnects Memory Plane Memory Plane Memory Plane Processor System Stack BEOL vertical electrical interconnects Processor Plane w/ local memory cache Photonic Network Interconnect Plane (includes optical devices, electronic drivers & amplifiers and electronic control network) 16

17 Vision for Silicon Photonics: Intra-Chip Optical Networks Pack ~36 IBM Cell processor supercores on a single ~600mm 2 die in 22nm CMOS In each Cell supercore, there are 9 cores (PPE + 8SPEs) 324 processors in one chip Power and area dramatically lower than today at comparable clock speeds Each supercore is electrically interconnected Communication between supercores and off-chip are optical BW between supercores is similar to today s off-cell BW (i.e., 1-2Tbps per Cell) Intra-Chip Optical Network: Fundamentally alters the roadmap to scaling highperformance multi-core processors Communications sub-system and architecture that leap-frogs equivalent electronic systems Use photonics for communications, not logic May require new network architecture; not just a point to point replacement of electrical network Silicon Nanophotonics: Enormous capacity and fundamentally low power consumption Estimate optical network requires 25 Watts vs. 640 Watts for equivalent electrical network Off-chip power advantage is even more compelling, by more than an order of magnitude 17

18 Possible On-Chip Optical Network Architecture Bufferless, Deflection-switch based (OCS on a chip) P G P G P G Cell Core (on processor plane) Gateway to ICON (on processor and photonic plane) P G P G P G Thin Electrical Control Network (~1% BW, also sends small messages) Photonic Network P P P Deflection Switch G G G 18

19 On-chip Network Implementation Architecture with Improved Application Performance Bufferless optical switch network Over provisioned to optimize throughput and latency Simple electric control plane with block transfer Integration with μproc via 3D layer stacking Subsystems Supercore Gateway to optical network ~2Tbps including over provisioning and coding Combine WDM, TDM, SDM, for example: 480Gbps Optical channels 6 λ s at 80Gbps or 12 λ s at 40Gbps 4 parallel optical channels 19 Possible Devices Optical devices Metrics determined by system needs Ultra-dense: 30x area improvement compared to EPIC program CMOS compatible (22nm node) Low power Functions include: Transmitter, Receiver, Switch, Transport (waveguides, gain) e.g., ring resonator array

20 Outline Optics in today s HPCs Trends in microprocessor design Multi-core designs for power efficiency Vision for future (ICON) 3D stack of logic, memory, and global optical interconnects Required devices and processes Low power and small footprint 20

21 Devices for Implementation Ultradense Si waveguides and Optical Components: 90nm and beyond CMOS generations enables: ~20x smaller bend radius than today s EPIC designs ~30x smaller area On chip modulators: Ring resonator or MZI based 2x2 optical deflection switches: Broad band (all λ s switched simultaneously) Temperature variation tolerant (~20C) MZI, TIR or MMI devices Integrated InP layers: Provides optical gain to overcome network losses Optical or Electronic TDM : Mux up to high bitrate from logic with fast modulators/detectors or by on-chip OTDM Optical WDM Mux-DeMux Rings, MZI, MMI, AWG are choices Detectors: e.g., Integrated WG Ge photodetectors Supporting electronics: High performance, low power CMOS based drivers, amplifiers, control network/logic. Optical source: off chip lasers 1μm bend radius Device designs and estimated future performance based on work at IBM, Cornell, UC Santa Barbara and Columbia Aggressive (but not implausible) performance extrapolation Total Internal Reflection Switch Ultradense waveguides Ring Resonator Optical Gain Block WDM Lattice filter (L), -- Ring Resonator (R) 21

22 Off-chip optical coupling 22

23 Coupling from fiber to silicon photonic wire S.McNab et al Optics Express 2003 (IBM) Fiber Polymer waveguide ~500nm (~3μm by 2μm) ~500nm by 220nm Si photonic wire Cornell, IBM, NTT Coupling loss <1dB (with a lensed fiber) 23

24 WDM passives 24

25 (i) Multimode Interferometer based WDM devices An imaging device: an input field is reproduced in single or multiple images at periodic intervals along propagation direction 6μm In F.Xia et al OFC 2007 (IBM) Concept: Soldano et al., J. Lightwave Technol., 13, ,

26 (i, continued) MMI-MZI based λ demultiplexer F.Xia et al OFC 2007 (IBM) λ 1, λ 2, λ 3, λ 4 λ 1 26 R 2μm 20μm λ 2 λ 3 λ 4 As-grown No active tuning Loss: 3dB Pass band: 0.3nm Limited by crosstalk Crosstalk: -12dB Channel spacing: 3.2±0.1nm Designed channel spacing: 3.2nm Reponse (db) Footprint: ~40μm 130μm (~0.005mm 2) 10 times smaller than AWG on same SOI platform 100 times smaller than III-V AWG) Wavelength (nm) 4 1

27 (ii) WDM based on detuned ring resonators F.Xia et al OFC 2007 (IBM) λ 1, λ 2, λ 3, λ 4 20μm λ 1 λ 2 λ 3 Relative response (db) Channel #1 Channel #2 Channel #3 Channel #4 λ 4 Difference in resonance wavelength is due to different perimeters of the resonator Δλ=3.2nm ΔL=180nm Wavelength (nm) Cross talk < -20dB Limited transmission bandwidth Designed channel spacing: 3.2nm Experimental value: 2.2nm to 3.1nm 27

28 Fast modulators 28

29 Silicon Ring Modulator at 12.5 Gb/s Xu, Schmidt, and Lipson, Nature, 2005(Cornell) 0.2µm Ring, Waveguide >9dB modulation depth PRBS Appears extendable to 40Gb/s 29

30 Broadband Optical Deflection Switches (all wavelengths simultaneously deflected) Broadband ring-resonator switch ON state: carrier injection coupling into ring signal switched OFF state passive waveguide crossover negligible power CMOS Driver CMOS Driver OFF ON CMOS Driver CMOS Driver 30

31 Broadband, thermally stable deflection switch from multiple resonators (Multiple rings broaden the passband for thermal stability) Xia, et al, CLEO 2007, Green, et al., OFC 2008 (IBM) IN Switch Performance (multiple λs) DROP THRU Apodization (flattens passband) λ nm Transmission (dbm) ring (w apodization) 4-ring (w/o apodization) λ 3 λ W avelength detuning (nm)

32 Photodetectors 32

33 Ge-on-SOI Detector Design Dehlinger, et al. PTL, 2004 and Schow et al., PTL, 2006 (IBM) Lateral PIN design, direct Ge growth on thin SOI. W m Ti/Al SiO 2 W i = 300 nm W m = 200 nm S = μm = 350 nm t Ge n+ p+ n+ p+ Si W i Ge S t Ge SiO 2 Si Design Features: Epitaxy using UHV-CVD Buried oxide isolates carriers generated in substrate Eliminates low-frequency tail 20GHz bandwidth Lateral p-i-n design for low capacitance/ dark current 33

34 On-chip optics requires waveguide geometry (e.g., Luxtera, announced) Ge adiabatic taper Si strip waveguide L taper L diode Oxide layer Light in Si waveguide Adiabatically coupled, high bandwidth waveguide Ge photodiodes on SOI substrate 34

35 On-chip gain: Evanescent Optical Amplifiers Park, Bowers, et al, PTL 19, p. 210 (2007) (UCSB) Intrachip networks have many nodes. Network size is limited by loss in waveguides, switches, and waveguide crossings Solution: Silicon evanescent optical amplifiers (III-V gain medium) Initial results: Device dimensions: H= 0.7 um, W = 2 um, L = 1.36 mm Amplifier Gain: 13 db Evanescent design allows higher saturation output powers than convention III-V amp Heating effects can be minimized with package design 35 University of California Santa Barbara

36 3D Integration 36

37 Development of 3DI Process for Si Photonics Currently developing 3DI processes based upon both Cu-Cu compression and oxide fusion bonding. Oxide bonding SOI Face-to-back Cu-Cu bonding Bulk Face-to-back Additional process technology development will be necessary to adapt each approach for photonics integration. Challenges include: Photonic devices and off-chip optical coupling compatible with 3D integration Thermal management 37

38 Major Challenges Achieving required device performance, in particular for: Optical bandwidth of modulators and switches High per-channel bitrates from direct modulation Device density WDM stability against temperature variations Low power operation Integration and path to manufacturability InP with Si Electronic support circuits with Si nanophotonics devices Compatibility with 3D layer stacking technologies and CMOS processing Network performance: demonstrate system/application advantages Low latency High throughput (avoid congestion) 38

39 Summary Optical I/O Multi-core uprocessor architectures are emerging as a key concept to provide power efficient high performance computing capability On-chip optical network can overcome the intra-chip and off-chip communications power bottleneck to scaling these architectures An on-chip optical network is not just a point to point replacement of electrical network No optical logic or buffers not packet switched Silicon Nanophotonics can provide the enormous capacity and fundamentally low power consumption required for future multi-core microprocessors Work on required devices and demonstration of the utility of the architecture just beginning 39