Simulation Strategies for Massively Parallel Supercomputer Design

Transcription

1 Simulation Strategies for Massively Parallel Supercomputer Design Authored by: Ansoft Corporation Special Thanks to: Cray Presentation #2 Ansoft 2003 / Global Seminars: Delivering Performance

2 Introduction Cray: Red Storm Supercomputer Sandia National Laboratories awarded Cray Inc. a multiyear contract to develop and deliver a new massively parallel processing (MPP) supercomputer called Red Storm. The computer will use 10,000 Advanced Micro Devices Inc. Opteron processors connected via a high-bandwidth, three-dimensional mesh interconnect network.

3 Introduction About Cray Approximately 850 employees worldwide Corporate headquarters: Seattle, WA 3 major engineering centers: Chippewa Falls, WI, Mendota Heights, MN, Seattle, WA NASDAQ: CRAY

4 Red Storm: System Overview Introduction Theoretical peak performance: 40 trillion calculations per second 10,368 Compute Nodes: AMD 64 bit Opteron processors Connected via a low-latency, high-bandwidth, three-dimensional mesh interconnect network based on HyperTransport technology Approximately 3000 ft² including disk systems

5 Introduction Red Storm: High Speed Network (HSN) 3D Mesh that interconnects all of the compute nodes 27 x 16 x 24 (x, y, z) mesh High-Speed Serial Link Nominal Data Rate: 3.2Gbps High Speed Network (HSN) +Y -Z +X -X -Y +Z PCI - X Compute Node

6 Introduction NEC Earth Simulator Performance: 40Tflops Processor: NEC.15um vector CPU Date: Cray Red Storm Performance: 40Tflops Processor: AMD Opteron Date: Cost: $450M Development Schedule: >54 months $$ Cost: $90M Development Schedule: 26 months $$ Custom Hardware System Integration

7 100 Introduction Relative Cost of Finding Hardware Design Problems Cost = Pain = $$$, Time to Market, Your Job, etc Software Test and Measurement Preliminary Design Detailed Design Integration Validation Operation

8 Designing for High-Speed Difficult Aspects As Speed increases, luck decreases Introduction Large number of codependent terms» They are not always controllable/understood random variation New effects Large Systems composed of many sub-systems New techniques» Variables that could be ignored in the past must be known to a very high precision» Signal Channel Management How do we account for and manage information? At high-speeds: Signal Integrity Engineering = Microwave Engineering» New Design Flows Cost Increases» New Techniques and Terms: Frequency Domain vs. Time-Domain» New Tools: Harmonic Balance, Quasi-Static, Full-Wave, etc.» New Models: 2D and 3D Physical Device Models» Model Abstraction LUCK SPEED A. Fraser, S. Argyrakis, Does Signal Integrity Engineering have a Future, DesignCon 2003,.

9 Designing for High-Speed Reverse the trend Introduction Decrease Cost : Move more Integration and Validation into early design stages. Virtual Prototypes! Stop relying on Luck: Better models, techniques, and tools increase the probability of first past success. Microwave Engineers have been using these techniques for over a decade

10 Virtual Prototypes Full System & Sub-Systems Introduction Sub-System Board/Stackup Sub-System - Connectors Full System Sub-System Daughter Card Sub-System - Transitions Sub-System - Packaging Sub-System - Routing

11 Channel Management Introduction Challenge: Move Integration and Validation into Virtual Prototype System 3D Models SPICE Models Connectors Bandwidth Impedance Cross-Talk Vias Frequency Dependence Channel Model Management Boards Layout Modes Eye Diagram Isolation Packaging Skin Effect 2D Models Power Delivery Transitions ISI TDR Loss BER Load Delay Source

12 Channel Management Introduction Common Design Environment/Integrated Database Solver on Demand Circuit: Transient/Linear/Non-Linear Harmonic Balance System: Mixed Mode Analysis - Baseband-through-RF Planar EM: 2.5D Full-Wave Method of Moments 3D Full-Wave: HFSS v9 Finite Elements (Solver on Demand, Now in Ansoft Designer 1.1) 3D Quasi-Static: Spicelink Boundary Elements (Solver on Demand, Version 6.0 coming soon) Solver on Demand - Information Hiding Prevents higher levels of design from becoming dependent on low-level details such as 3D Physical Device Modeling. Mechanical CAD 3D AnsoftLinks Layout Channel Manager Ansoft Design Environment Planar EM System Matlab DXF/GDSII C code Circuit SPICE

13 Introduction Why are better models, techniques, and tools needed? Speed = Problems Evolution of a short circuit Once interconnects stop behaving as transmission lines, SPICE models and SPICE like tools can not predict performance SPEED A. Fraser, S. Argyrakis, Does Signal Integrity Engineering have a Future, DesignCon 2003,.

14 Introduction Why are better models, techniques, and tools needed? Co-dependent terms Example: As speed increases, the connector performance begins to depend on the board integration. Adopting new models, techniques, and tools that can identify these co-dependent performance factors reduces the probability of discovering hardware problems late in the product development cycle» Remember: The possibility of uncontrollable or unforeseen variables can still appear?

15 Introduction What are these uncontrollable or unforeseen variables? Virtual Prototypes are abstractions They only contain the essential details of a complex system Essential Details = Those that are critical to the electrical performance Model Abstraction efficiently uses limited computer resources and product development time Example: Cavity filter designers routinely use screws to tune the filter and account for manufacturing variations. When they simulate their filter designs they would not include the threads on the screw. The threads are essential mechanical details, not electrical details Manufacturing Process Variations Example: If the virtual prototype does not account for the substrate thickness shrinking because of thermal effects in the manufacturing process, you will not predict the performance correctly.

16 Introduction Ansoft and Cray Ansoft: Provide End-to-End Simulations of HSN Channel Five different classes of simulations / analysis 1. PCB/Interconnects Mezzanine, Module, Backplane, and Red/Black Switch 2. Connectors NexLev, GbX, and VHDM 3. Cabling Self-Equalizing Twin-Ax (1.1m - 8m) 4. Packaging HyperBGA High Performance Organic Flip-Chip BGA 5. System Frequency and Time Based Performance Extraction Cray: Provide Electrical Specifications Electrical Models Mechanical Models Board Layouts

17 Introduction Red Storm: HSN Physical Configuration VHDM Connector SerDes ASIC Backplane Compute Board AMD 64 bit Opteron GbX Connector

18 Introduction Red Storm: HSN Electrical Configuration HyperBGA + Mezzanine Board Module Board Backplane Red/Black Switch Connector Connector Connector Cable

19 Introduction Red Storm: HSN Electrical Configuration SerDes HyperBGA Mezzanine Board Teradyne Nexlev Connector Still in Model Development Module Board Teradyne GbX Connector Teradyne GbX Connector Backplane Board Molex VHDM Connector Molex VHDM Connector Molex Twin-ax Cable Molex Twin-ax Cable Red/Black Switch

20 Module Board PCB/Interconnects NexLev GBX YP3_FMBP{18,19} 32mm YM0_TOBP{18,19} 28mm NexLev GBX

21 PCB/Interconnects Module Board System (Frequency or Time Based Analysis) Port1 Planar EM Coupled Bend _EM_SLCBENDS29 W21 W=5mil S=9.11mil P=645mil W=5milP=716mil S=9.11mil P1 W=5milP=293mil S= Port1 Port3 Port2 Port1 Port3 Port4 W18 W20 _EM_SLCBENDS28 _EM_SLCBENDS27 U3 Port2 Port4 W=5mil S=9.11mil P=3095mil W=5mil S=9.11mil P=401mil W=5milP=376mil S=9.11mil PlanarEM7 Circuit Solver On Demand Planar EM Speed Choose the level of speed and accuracy Accuracy

22 PCB/Interconnects Backplane P1BYP3_FMBP{18,19} 217mm P1BYM0_TOBP{18,19} 291mm

23 Backplane Spicelink 2D PCB/Interconnects Layer Height (B): mm (10.7 mil) Trace Width (W): mm Trace Separation (S): 0.25 mm Trace Thickness: 0.5 Oz Copper (0.7 mil) ε r = 3.4, tanδ = S W B Layer B W S Zse Zd Zcom S1/S All Dimensions are in mm

24 PCB/Interconnects Backplane Routing Via Stub In the link the GbX and VHDM will contain a best and worst case via stub VHDM Connector VHDM Connector Route Layer: s1 (Via Stub: mil) Backplane VHDM Best Case Worst Case GbX Route Layer: s10 (Via Stub: 10.75mil)

25 Backplane Routing Via Stub PCB/Interconnects These results do not include loss

26 Backplane Routing Anti-pad PCB/Interconnects Antipad Radius: 0.5mm (Layout) Antipad Radius: 0.7mm

27 PCB/Interconnects Backplane Routing Anti-pad (Layer: S1) These results do not include loss

28 PCB/Interconnects Backplane Routing Anti-pad (Layer: S10) These results do not include loss

29 HFSS side view GbX Connector bottom view Teradyne s GbX advanced performance interconnect provides the highest density optimized differential connector available today. Delivering data rates greater than 5 Gb/s. High Density: GbX provides up to 55 pairs per linear inch (4-pair configuration). Reliability: Two points of contact at a separable interface. Flexibility: Choice of density configurations (3, 4 and 5-pair) for higher application flexibility. Vertical and Horizontal Routing make GbX the ideal solution for star or mesh backplane design.

30 GbX Connectors All links contain 2 backplane sections One channel outbound from SerDes ASIC. One channel inbound to SerDes ASIC. GbX models encapsulate connectors and escape vias/routing Connector performance is very dependent on board interface. Interface is critically dependent on board metrics: route layer via stub length antipad dimensions board materials Escape routing is different on the outbound and inbound channels.

31 Backplane VHDM To ASIC from backplane Module GbX Module GbX VHDM From ASIC to backplane

32 Connectors Models are generated separately for the GbX components. Each channel includes models for: 1. Backplane board escape routing, with adjacent pins. 2. GbX connector with single wafer. 3. Module board escape routing, with adjacent pins. Different levels of complexity were retained initially for the escape routing. From Backplane routing will be used to determine what level of complexity is necessary. From Backplane + Complexity - + Complexity - To Backplane

33 GbX connector module escape routing backplane escape routing

34 Connectors VHDM Very High Density Matrix

35 Connectors

36 VHDM - Backplane Connectors VHDM connector twin-ax cable feed backplane escape routing

37 Connectors Red/Black switch allows supercomputer to be physically divided for secure (classified) processing Red/Black switch is two VHDM-HSD connectors in a back-to-back configuration A center-plane circuit board provides support for the back-to-back configuration HFSS model

38 Cable Gore Twin-Ax 100 differential Self Equalization

39 Self Equalization Cable Attenuation increases with sqrt(f) due to conductor skin effects Higher frequency components attenuations >> fundamental frequency Increased jitter and inter-symbol interference Limits length of cable Dielectric loss vary directly with frequency Low loss dielectric Cable Equalization Produces a near linear attenuation response vs. frequency Use different skin depth properties of conducting materials Base material has low conductivity and/or high permeability» Coat with a good conductor

40 Cable Equalized Standard Cable

41 HyperBGA High Performance Organic Package Flip Chip Package

42 Packaging

43 Packaging

44 Conclusions Cray and Ansoft Corporation are in collaboration to verify for the 3.2Gb/s serial data channel of the Cray Red Storm Supercomputer high-bandwidth, three-dimensional mesh interconnect network. Cray recognized the value of electromagnetic-based simulation to ensure reliable supercomputer performance. This presentation showed how a combination of electromagnetic field simulation coupled with circuit and system simulation was used to predict the interconnect performance. The successful/accurate characterization of the system was made possible by utilizing: Electromagnetics based analyses software Circuit/System Level» Ansoft Designer Passive Physical Device Modeling» Ansoft HFSS» Ansoft Designer» Ansoft SpiceLink» Ansoft Optimetrics Modern high-speed designs are requiring engineers to achieve new levels of technological advances. The methodologies introduced here show how to systematically reduce a complex system to a solvable problem. This structured procedure breaks the design-build-redesign loop commonly found in the old methodology of addressing problems after signal integrity errors are encountered.