Optical PCB Overview. Frank Libsch IBM T.J. Watson Research Center Yorktown Heights, NY. IBM Research

Transcription

1 IBM Research Optical PCB Overview Frank Libsch IBM T.J. Watson Research Center Yorktown Heights, NY IBM Internal November 16, 2011

2 Outline System view of why optics is needed Potential OPCB Technologies for Next Generation HPCs OSA Designs OSA Assemblies Packaging Optical Component Characterization 2 November 2011

3 Optimized solutions will require detailed analysis of trade-offs Power Margin, Packaging Integration, Data-rate Density Packaging Integration, Channel Integration, Margin, Cooling, Data-rate Cost Base Manufacturing Cost, Yield, Channel Integration, Data-rate, Reliability 3 November 2011

4 Why High Performance Computing? Larger scale, more complex, higher resolution, multiscale Physics Shorter time to solution real time response Materials Science Geophysical Data Processing Environment and Climate Modeling Life Sciences / Drug Discovery Fluid Dynamics / Energy Industrial Modeling Financial Modeling Transportation More than 50% of Top500 systems are for industry* Growing number of flops are industry (~15% in 1990s to ~30% today)* Courtesy L. Treinish, IBM * Image courtesy of the National Center for Computational Sciences, Oak Ridge National Laboratory 4 November 2011

5 Maintaining the HPC Performance Trend Performance (Gigaflops) 1.E+11 1.E+10 1.E+09 1.E+08 1.E+07 1.E+06 1.E+05 1.E+04 1.E+03 1.E+02 1.E+01 1.E+00 1.E-01 Nov-92 10x /3.5-4yrs = 85-90% CAGR Total 1st Nov-94 Nov-96 Nov th Nov-00 Nov-02 Nov-04 5 November 2011 Nov-06 Nov-08 Time Nov-10 Nov-12 Nov-14 Nov-16 Nov-18 Nov-20 1EF 1PF Increasing Parallelism: System BW at all levels of assembly must scale exponentially ~(0.1-1 Byte/Flop) AND/OR new architectures, topologies and algorithms required Systems: 85-90% CAGR,continuing** Increasing Parallelism Accelerators Chip ~50% 30% CAGR* Semiconductors & Pkg: ~15-20% CAGR, slowing * **chart data from

6 Cost and power of a supercomputer Year Peak Performance Machine Cost Total Power Consumption PF $150M 2.5MW PF $225M 5MW PF $340M 10MW PF (1EF) $500M 20MW Assumptions: Based on typical industry trends (See, e.g., top500.org and green500.org) 10X performance / 4yrs (from top500 chart) 10X performance costs 1.5X more 10X performance consumes 2X more power 6 November 2011 J. Kash Photonics Society Annual Meeting Nov 2010 and OWQ1 OFC 2011

7 Evolution of Optical interconnects Time of Commercial Deployment (Copper Displacement): 1980 s 1990 s 2000 s > 2012 WAN, MAN metro,long-haul LAN campus, enterprise System intra/inter-rack Board module-module Module chip-chip Chip on-chip buses Telecom Datacom Computer -com Distance Multi-km 100 s m 10 s m < 1 m < 10 cm < 20 mm Integration cards Card edge Card edge /on card Module Si C or chip On chip BW * Distance: Optics >> Copper Increasing integration of Optics with decreasing cost, decreasing power, increasing density 7 November 2011

8 Why Optics? Electrical Buses become increasingly difficult at high data rates (physics): Increasing losses & cross-talk Frequency resonant affects Optical data transmission is easier: Much lower loss, esp. at higher data rates Additional advantages include: Cable bulk, connector size, EMI Potential power savings KEY ADVANTAGE: BW * Distance > electrical Optics trends for large servers and data centers are following same trends as telecom network: Longer links first Short link optics requires tighter package integration: Higher BW closer to signal source Changes the supplier/manufacturing paradigm CPU LGA Socket Card Card Cross talk Resonance effects Optics Freq dependent losses Backplane Copper 8 November 2011

9 Bandwidth: the Bane of the Multicore Paradigm: Logic flops continue to scale faster than interconnect BW Constant Byte/Flop ratio with N cores (constant?) means: Bandwidth(N-core) = N x Bandwidth(single core) Signal + + Reference Pins Pins per chip Assumptions: 3 GHz clock ~ 3 IPC 10 Gb/s I/O 1 B/Flop mem 0.1 B/Flop data 0.05 B/Flop I/O Number of Cores 3Di (3D integration) will only exacerbate bottlenecks M. Ritter Topical Workshop on Electronics for Particle Physics, Sept November 2011

10 Implications of BW Scaling: Signal + Reference Pins Signal + Reference Pins Pins per chip Number of Cores Chip escape limit, 200?m pitch Module escape, 1mm pitch Card escape, 8 pair/mm (QCM w/8 Cores ) Module Escape Bottleneck Card Escape Bottleneck M. Ritter Topical Workshop on Electronics for Particle Physics, Sept November 2011

11 Total bandwidth, cost and power for optics in a machine Year Peak Performance PF PF PF PF (1EF) (Bidi) Optical Bandwidth 0.012PB/s (1.2x10 5 Gb/s) 1PB/s (10 7 Gb/s) 20PB/sec (2x10 8 Gb/s) 400PB/sec (4x10 9 Gb/s) Optics Power Consumption 0.012MW 11 November 2011 Optics Cost $2.4M 0.5MW $22M 2MW $68M 8MW $200M Require >0.2Byte/FLOP I/O bandwidth, >0.2Byte/FLOP memory bandwidth 2008 optics replaces electrical cables (0.012Byte/FLOP, 40mW/Gb/s) 2012 optics replaces electrical backplane (0.1Byte/FLOP, 10% of power/cost) 2016 optics replaces electrical PCB (0.2Byte/FLOP, 20% of power/cost) 2020 optics on-chip (or to memory) (0.4Byte/FLOP, 40% of power/cost) J. Kash Photonics Society Annual Meeting Nov 2010 and OWQ1 OFC 2011

12 HPC driving volume optics Computercom market Single machine volumes similar to today s WW parallel optics Number of Optical Channels WW volume in Gbps ASCI Purple MareNostrum Blue Waters* 5Gbps Roadrunner 10Gbps * Expected Summer 2011 T. Dunning, NCSA, of High Performance Computing Year? 12 November 2011

13 Electronic Packet Switching Typical architecture (electronic switch chips, interconnected by electrical or optical links, in multi-stage networks) works well now--- Scalable BW & applicationoptimized cost Multiple switches in parallel Modular building blocks many identical switch chips & links) -- but challenging in the future Switch chip throughput stresses the hardest aspects of chip design I/O & packaging Multi-stage networks will require multiple E-O-E conversions N-stage Exabyte/s network = N*Exabytes/s of cost N*Exabytes/s of power J. Kash OFC tutorial 2008 Central switch racks By courtesy of Barcelona Supercomputing Center November 2011

14 Potential Optics Technologies for Next Gen HPC transition to OPCB? Next Gen Fiber Optics Optical PCB Polymer waveguides Switch hub, optics on MCM 2x12 VCSEL array TSV Si carrier Optochips assembled 2x12 PD array 2x12 LDD IC 6.4mm x 10.4mm 2x12 RX IC TX LDD VCSEL RX PD RX Si Carrier To optical connector Organic Carrier Lens Arrays PCB or Flex Polymer Wavegu ~Known (vendor) Technologies Higher cost Moderate Density Needs commercial ecosystem Lower cost Higher Density HPC Peak Performance Optics Bitrate PF 10Gbps Gbps 300 PF EF 25-40Gbps Volume commercial use lags HPC by ~4-5 years IBM has technology expertise both in WGs and high bitrates 14 November 2011

15 Advantages of Polymer Waveguide Technology compared to Parallel Fiber Optics Integrated mass manufacturing Board, sheet, film level processing of optical interconnects Lower assembly, waveguide jumper costs Costs for wide busses should scale better Simple assembly Avoid fiber handling (integrated approach) Similar assembly procedures as for electrical components and boards (pick and place, etc.) Establish electrical and optical connections simultaneously (pick, place, reflow, etc.) Avoid separate optical layer (if integrated with board) New or compact functionality supporting new architectures Shuffles, Crossings, splitters, Enabler for multi-drop splitting, & complex re-routing that is expensive in fiber Higher density, waveguide pitch < 125 um (best future fiber pitch) Cost Higher bandwidth density, less signal layers demonstrated 62.5um Reduced Optics module cost AND jumper/connector costs both important Possible Lower Maintenance costs 15 November 2011

16 Integrated packaging is more complex, requires close relationship with suppliers (IBM PERCS) Ideal candidate for PWG Packaging. Hub ASIC (Under Heat Spreader) Strain Relief for Optical Ribbons Total of 672 Fiber I/Os per Hub, 10 Gb/s each Optical Transmitter/Receiver Devices 12 channel x 10 Gb/s 28 pairs per Hub - (2,800+2,800) Gb/s of optical I/O BW Cooling / Load Saddle for Optical Devices Heat Spreader over HUB ASIC Heat Spreader for Optical Devices A.Benner, Future Directions in Packaging (FDIP) Workshop, EPEP Oct November 2011

17 Density of optical links Optics Module and electrical connector Card edge Active cable, electrical at card edge Courtesy of Avago Technologies ~18x41mm 1.27mm pitch Courtesy of Avago Technologies ~8X8mm, ~0.75mm pitch ~60K fibers/ RACK ~10x25mm ~40K fibers / SYSTEM ROADRUNNER ~1PF) 4x4 VCSEL Array 4x4 PD Array ~5X15mm 48@10Gbps/cable ~5x3mm 0.2mm pitch PERCS >10PF Optical at card edge > 40x denser 17 November 2011

18 Optical waveguide interconnects: The Terabus project and related work Optochip SLC CMOS IC VCSEL CMOS ICCMOS CMOS IC VCSEL OE s PD Waveguide Lens Array Optomodule SLC Optomodule 2 CMOS ICCMOS CMOS IC VCSEL OE s Waveguide Lens Array Optocard Polymer waveguides Dense Hybrid Integration: demonstrate a low-cost packaging approach compatible with conventional PCB manufacturing and surface-mount board assembly Future Vision: optically-enabled MCM s Circa Transceiver Optochip Other Chips organic chip carrier Circuit Board w/ both electrical traces & optical waveguides Low-density, conventional electrical interface for power & control High-density, wide and fast optical interfaces for data I/O Much higher off-module bandwidth at low cost in $$ and power 18 November 2011 CPU OE XCVR Power Cost Datarate Density Reliability <10mw/Gbps (EOE) <$0.25/Gbps (TRx + on-card optics) 25Gbps/channel 2 Tbps/cm 2 (on module) < 10 FIT per channel

19 Outline System view of why optics is needed Potential OPCB Technologies for Next Generation HPCs OSA Designs OSA Assemblies Packaging Optical Component Characterization 19 November 2011

20 Optical Printed Circuit Boards IBM Research has invested heavily in the past 7+ years in Optical printed circuit board technology based on multi-mode polymer waveguides Partially funded by the US Government (Terabus program) We believe this technology will be needed to provide the needed BW for future server generations, allow highly integrated electrical-optical links and provide a path to much lower cost optical links. We are highly interested in establishing a market eco-system that will provide a set of suppliers, standards and specifications and users for this technology. 20 November 2011

21 Optocard Technology: Waveguides, Module Attachment and Tolerances Waveguide Lens Array 35µm x 35µm 62.5µm pitch BGA site for Optomodule TRX IC SLC Carrier x Offset (?m) OE Lens Array y Offset (?m) Module Attachment FR4 Optical coupling efficiency Coupling Efficiency (db) x Offset (? m) BGA attachment process Alignment of OE lens array to waveguide lens array Coupling Efficiency (db) Tx: ±35 µm Rx > ±65 µm Offset (?m) 21 November 2011

22 Optocard waveguides: Turning mirrors and Lens Array SLC Carrier TRX IC OE Lens Array Waveguide Lens Array FR4 TIR mirrors Integrated turning mirrors TIR laser ablation Dense waveguide pitch Integrated 48-channel collimating lens array Provides 40mm alignment tolerance for Optomodule BGA site for Optomodule 48-channel Waveguide mirror array 22 November 2011 Board Loss (db) Waveguide/Mirror channel 35, 40 not shown, in-coupling scattering Uniformity 1.6 db average loss 0.9dB for 7.5cm ~0.7dB for mirror/lens assembly Channel Number waveguide cores on 62.5um pitch Median : 1.6 Stdev: 0.1

23 Waveguide Connectors: Passive alignment with optical precision Top FR4 stack (with electrical lines) Polymer waveguide layer Connector interface 12 waveguides Alignment marker Bottom FR4 stack MT pins Alignment studs PCB Copper markers Alignment slots waveguides MT ferrule aligned by copper markers Positioned MT ferrule to polymer waveguides 23 November 2011

24 Terabus Transceivers 985nm and 850nm built Chip-to-chip optical interconnect on a PCB using 985nm transceivers (non-standard wavelength) Chip-to-chip optical interconnect on a PCB using 850nm wavelength transceivers (industry-standard wavelength) Optomodule SLC Transceiver Optochip CMOS ICCMOS CMOS IC VCSEL OE s OE s PD Optochip SC SiCarrier LDD VCSEL PD TIA LDD VCSEL PD SLC CoreEZ SLC Substrate Waveguide Lens Array Lens Array Optocard FR4 Substrate Optochip: Single-chip CMOS transceiver IC with flip-chip attached optoelectronic arrays 985nm, substrate emitting OEs Optochip: Si carrier platform for heterogeneous integration of OEs and ICs Electrical and Optical vias in Si carrier Optomodule: High-speed, high-density organic carrier Extendible to optically enabled MCM (OE-MCM) Optocard: PCB with integrated polymer waveguides Replace complex electrical PCB with simpler electrical PCB (power and control), plus waveguides 24 November 2011

25 Full Terabus Link (985-nm): 2 Transceiver Optomodules on Optocard TRX1: 16TX + 16RX 4x4 VCSEL Array TRX2: 16TX + 16RX 4x4 PD Array 16 Channels TRX1? TRX2 at 10Gb/s + 16 Channels TRX1? TRX2 at 10Gb/s 25 November 2011

26 Terabus 850 nm Transceiver Optochip Assembly VCSEL High-speed array probe PD pads array TSV Si carrier LDD IC LDD IC VCSEL array PD array RX IC RX IC Silicon Carrier Silicon Carrier provides dense high-speed wiring, mechanical support, and optical I/O through holes etched in the carrier Heterogeneous integration of ICs and OEs through dense arrays of solder microbumps IC pads:?= 35?m, pitch = 50?m OE pads:?= 35?m, pitch = 100?m ~1um Au-plating on pads Optochip assembly: four sequential flip-chip soldering processes Suss FC-150 flip-chip bonder AuSn solder pre-deposited on ICs, VCSEL & PD arrays Reflow solder at about 320 C 26 November 2011

27 Assembled Terabus 850-nm Optomodule (Optochip on CoreEZ) TX:15Gb/s 850-nm Optochip demonstrated 300Gb/s bidirectional aggregate data rate RX:12.5Gb/s First row of solder joins visible beneath the Optochip 27 November 2011

28 IBM technology expertise in high bitrate transceivers VCSEL Transmitters and Receivers can achieve 2015 metrics for density, bitrate and power 90-nm IBM CMOS-Driven VCSEL Transmitters and Compatible Receivers Power and Speed Records VCSEL Transmitters 20Gb/s 20Gb/s 29Gb/s 32Gb/s 0.7 pj/bit 2.8 pj/bit 1.9 pj/bit 1.75 pj/bit Compatible Receivers 15Gb/s 17.5Gb/s 20Gb/s 2.9 pj/bit 3.5 pj/bit 6.7 pj/bit November 2011

29 Development of VCSELs for >25 Gb/s links collaborations with OE vendors Joint work with Finisar AOC 8mA 700mV pp 26Gb/s 30Gb/s *R. Johnson and D. M. Kuchta, 30Gb/s directly modulated VCSELs, CLEO 2008 Joint work with Emcore Corp 20Gb/s 25Gb/s 6mA 375mV pp *N. Y. Li et al., High-Performance 850 nm VCSELs and Photodetector Arrays for 25 Gb/s Parallel Optical Interconnects, OFC *N. Y. Li et al., Development of High-Speed VCSELs Beyond 10 Gb/s at Emcore, Photonics West November 2011

30 IBM technology expertise in high bitrate optical PWG Link Prototype builds Embedded Waveguide Technology: 120 Gb/s Board-to-Board Optical Link Demonstrator Optical TX/RX board as building block 12 embedded waveguides Complete 12x10 Gb/s link demonstrator MT interface EO modul Embedded polymer waveguides (12 channels) Passive alignment of MT standard based connectors MT interface as standard interface for WG, fiber bundles/optical flexes and transceivers Pluggable TX/RX module (butt coupling) Eye diagrams for 2 channels at 10 Gb/s 10 Gb/s 10 Gb/s 30 November 2011

31 IBM technology expertise in field replaceable optical TRx packaging Field Replacable Optics OE Carrier/LGA, Dual Layer PWG Carrier LGA VCSEL arraypd array TIA LDD Heat Spreader Backplane Connector (TRL) Organic Carrier LGA PCB Lens arrays WG arrays with mirrors OE Carrier: Wafer based micro-optic assemblies, organic substrate for electrical interconnect (low cost). Removable carrier, passive alignment. PCB / Waveguide Board: PCB with dual layer buried polymer waveguides, integrated waveguide lens arrays. LGA Assembly: Low force LGA, course & fine alignment means. Optical Components: 1x12 arrays : 850nm VCSEL and photodiode Arrays (10Gb/s) Libsch, F.R. et. al., MCM LGA packaging for Optical I/O Passively Aligned to Dual Layer Polymer Waveguides, IEEE Electronic Components and Technology Conference (ECTC), November 2011

32 Waveguide-on-flex cables: 192 Channel flexible waveguide optical backplane 8 waveguide flex sheets, 192 waveguides, 8 connectors Optical L-Links Optical D-Links 4 connectors, 48 waveguides each Multi-layer waveguide optical connector Based on passive alignment Basic building block for optical pcb backplane technology 32 November 2011

33 48-Channel-to-4x12-Channel Waveguide Flex Fan-Out 48 WGs with 62.5?m pitch Waveguide cross-section Light incoupling 12 WGs with 250?m pitch 33 November 2011

34 Summary and Questions for Discussion As shown, optical polymer waveguide interconnect technology are a candidate for high performance computer optical interconnects. IBM is working with other system houses to define commonality, drive volumes up, and provide for lower cost to both system and component vendors. IBM is not likely to build this technology for itself, just as we currently do not manufacture PCBs and organic build-up technologies or optical transceivers IBM has developed research-level technologies for all aspects of the technology Commercialization of waveguide-on-card manufacturing would require working with a commercial PCB vendors and a waveguide materials supplier, as well as a manufacturer of the optical transceivers What role do vendors (yourself) envision for yourself in that ecosystem? Comments welcomed. 34 November 2011