The way toward peta-flops

Transcription

1 The way toward peta-flops ISC-2011 Dr. Pierre Lagier Chief Technology Officer Fujitsu Systems Europe

2 Where things started from DESIGN CONCEPTS 2

3 New challenges and requirements! Optimal sustained flops per watt!! Low operating cost through efficient cooling, floor space and weight Mission critical ready with high reliability and availability! Scalability towards s processors 3

4 Petascale supercomputer design concepts HPC centric Environmentally efficient Mission Critical Design first a true HPC processor rather than adapt the technologies around a commodity processor Most efficient technologies fitting together toward HPC efficiency Optimal sustained flops per watt Flexible mixed water/air cooling capabilities High density integration to reduce floor space Longest mean time between interrupt (MTBI) and mean time between failure (MTBF) Shortest mean time before restart production High throughput capabilities 4

5 Building a peta-flops class computer DESIGN OUTLINE 5

6 Processor Processor HPC engine Low Power Reliable SPARC64 TM VIIIfx architecture Multi-cores technology 256 Floating point registers per core 32KB(I)+32KB(D) 2 Way L1 cache per core 6MB Shared L2 cache Inter-core hardware synchronisation Application access to cache management High performance per watt 128 GFlops 58 Watts peak Water cooling Low current leakage of the CPU Low power consumption and low failure rate of CPUs 6

7 Memory hierarchy Memory High throughput Single CPU L2 6 MB Main memory 16 / 32 / 64GB 64 GB/s ICC L1 32KB(I)+32KB(D) SPARCfx 7

8 On board interconnect controller (ICC) ICC Rich functions High BW & Low latency SPARCfx ICC Rich functions User level RDMA Hardware barrier and reduction offload engine Low latency and high-throughput Full crossbar router with redundant direct paths Dedicated data path with CPU Direct command issuing to ICC enables low latency user space interconnect DMA hardware registers memory mapped through Linux kernel driver extension 8

9 System Board Motherboard Compact Maintenance Compact design 4 x compute nodes per system board Hybrid cooling Direct 6D Mesh/Torus (4 ICC s) 32 x DIMMs for memory dimensions 526mm x 481mm x 46mm Low processor temperature Easy replacement Remote power on/off of each system board No need to stop a rack to replace a system board 9

10 Tofu Interconnect (1) Interconnect Fast Reliable Efficient Technology Very fast node to node communication, 5GB/s x 2 (bi-directional) Low latency, less than 2 µs point to point hardware latency Global hardware barrier, less than 10 µs to synchronise all compute nodes of petaflops class machine Integrated MPI support for collective operations Topology 6D Torus / Mesh physical node addressing (x, y, z, a, b, c) Logical 3D Torus partitioning (x, y, z) with 3 additional communication paths (a, b, c) z y a c x b 10

11 Rack integration Rack Density Cooling Maintenance High density integration 24 multi-node system boards 96 compute nodes 12 Tflops of compute power 6 I/O nodes per rack 48 GB/s I/O throughput Tofu Integrated switchless 6D Torus/Mesh interconnect Flexible mixed water and air cooling Easy access to all components with very fast hardware replacement Few minutes to replace a system board 11

12 Deployment Interconnect [X-Y] between racks [Z][a-b-c] inside a rack Inside a rack Z axis of 3D Torus 8 mesh, 12 nodes each with a,b,c axis Between racks X,Y axis of 3D Torus X 8 mesh Z Y a c b 12

13 Breaking 8 peta-flops sustained RIKEN NATIONAL PROJECT All images courtesy of RIKEN 13

14 Facilities in Kobe (1) Research unit building ~65m Main computer building ~65m Power supply and cooling unit building Electric power supply Kobe site ground plan Courtesy of RIKEN 14

15 Facilities at Kobe (2) Cooling towers Centrifugal chiller Power Supply and Cooling Unit Building 15 Courtesy of RIKEN

16 Inside the computer room The Next-Generation Supercomputer installed in the facility 16

17 17 17