Oakforest-PACS (OFP) Taisuke Boku

Transcription

1 Oakforest-PACS (OFP) Taisuke Boku Deputy Director, Ceter for Computatioal Scieces Uiversity of Tsukuba (with courtesy of JCAHPC members) 1 Japa-Korea HPC Witer School 2018 Ceter for Computatioal Scieces, Uiv. of Tsukuba

2 JCAHPC Joit Ceter for Advaced High Performace Computig ( Very tight collaboratio for post-t2k with two uiversities For mai supercomputer resources, uiform specificatio to sigle shared system Each uiversity is fiacially resposible to itroduce the machie ad its operatio -> uified procuremet toward sigle system with largest scale i Japa To maage everythig smoothly, a joit orgaizatio was established -> JCAHPC 2 Japa-Korea HPC Witer School 2018 Ceter for Computatioal Scieces, Uiv. of Tsukuba

3 Oakforest-PACS (OFP) U. Tokyo covetio U. Tsukuba covetio Do t call it just Oakforest! OFP is much better 25 PFLOPS peak 8208 KNL CPUs FBB Fat-Tree by OmiPath HPL PFLOPS #1 i Japa #6 #7 HPCG #3 #5 Gree500 #6 #21 Full operatio started Dec Official Program started o April Japa-Korea HPC Witer School 2018

4 Computatio ode & chassis Water coolig wheel & pipe Chassis with 8 odes, 2U size Computatio ode (Fujitsu ext geeratio PRIMERGY) with sigle chip Itel Xeo Phi (Kights Ladig, 3+TFLOPS) ad Itel Omi-Path Architecture card (100Gbps) 4 Japa-Korea HPC Witer School 2018

5 Water coolig pipes ad rear pael radiator 5 Japa-Korea HPC Witer School 2018

6 Specificatio of Oakforest-PACS Total peak performace 25 PFLOPS Total umber of compute odes 8,208 Compute ode Itercoect Logi ode Product Processor Fujitsu Next-geeratio PRIMERGY server for HPC (uder developmet) Itel Xeo Phi (Kights Ladig) Xeo Phi 7250 (1.4GHz TDP) with 68 cores Memory High BW 16 GB, > 400 GB/sec (MCDRAM, effective rate) Product Lik speed Topology Product Low BW # of servers 20 Processor Memory 96 GB, GB/sec (DDR x 6ch, peak rate) Itel Omi-Path Architecture 100 Gbps Fat-tree with full-bisectio badwidth Fujitsu PRIMERGY RX2530 M2 server Itel Xeo E5-2690v4 (2.6 GHz 14 core x 2 socket) 256 GB, 153 GB/sec (DDR x 4ch x 2 socket) 6 Japa-Korea HPC Witer School Ceter for Computatioal Scieces, Uiv. of Tsukuba

7 Specificatio of Oakforest-PACS (I/O) Parallel File System Fast File Cache System Type Total Capacity Meta data Object storage Type Product Lustre File System 26.2 PB # of MDS 4 servers x 3 set MDT Total capacity Product Product # of OSS (Nodes) Aggregate BW DataDirect Networks MDS server + SFA7700X 7.7 TB (SAS SSD) x 3 set DataDirect Networks SFA14KE 10 (20) ~500 GB/sec # of servers (Nodes) 25 (50) Aggregate BW Burst Buffer, Ifiite Memory Egie (by DDN) 940 TB (NVMe SSD, icludig parity data by erasure codig) DataDirect Networks IME14K ~1,560 GB/sec 7 Japa-Korea HPC Witer School 2018 Ceter for Computatioal Scieces, Uiv. of Tsukuba

8 Full bisectio badwidth Fat-tree by Itel Omi-Path Architecture 12 of 768 port Director Switch (Source by Itel) Uplik: 24 2 Dowlik: Firstly, to reduce switches&cables, we cosidered : All the odes ito subgroups are coected with FBB Fat-tree Subgroups are coected with each other with >20% of FBB But, HW quatity is ot so differet from globally FBB, ad globally FBB is preferredfor flexible job maagemet. 362 of 48 port Edge Switch 2 Compute Nodes 8208 Logi Nodes 20 Parallel FS 64 IME 300 Mgmt, etc. 8 Total Japa-Korea HPC Witer School 2018

9 Facility of Oakforest-PACS system Power cosumptio # of racks 102 Coolig system Compute Node Type Facility Others Type Air coolig 4.2 MW (icludig coolig) actually aroud 3.0 MW Warm-water coolig Direct coolig (CPU) Rear door coolig (except CPU) Coolig tower & Chiller Facility PAC 9 Japa-Korea HPC Witer School 2018

10 Software of Oakforest-PACS Compute ode Logi ode OS CetOS 7, McKerel Red Hat Eterprise Liux 7 Compiler gcc, Itel compiler (C, C++, Fortra) MPI Itel MPI, MVAPICH2 Library Itel MKL Applicatio Distributed FS Job Scheduler Debugger Profiler LAPACK, FFTW, SuperLU, PETSc, METIS, Scotch, ScaLAPACK, GNU Scietific Library, NetCDF, Parallel etcdf, Xabclib, ppope-hpc, ppope-at, MassiveThreads mpijava, XcalableMP, OpeFOAM, ABINIT-MP, PHASE system, FrotFlow/blue, FrotISTR, REVOCAP, OpeMX, xtapp, AkaiKKR, MODYLAS, ALPS, feram, GROMACS, BLAST, R packages, Biocoductor, BioPerl, BioRuby Globus Toolkit, Gfarm Fujitsu Techical Computig Suite Alliea DDT Itel VTue Amplifier, Trace Aalyzer & Collector 10 Japa-Korea HPC Witer School 2018

11 Post-K Computer ad OFP OFP fills gap betwee K Computer ad Post-K Computer Post-K Computer is plaed to istall time frame K Computer will be shutdow aroud ?? Two system software developed i AICS RIKEN for Post-K Computer McKerel OS for May-core era, for a umber of thi-cores without OS jitter ad core bidig Primary OS (based o Liux) o Post-K, ad applicatio developmet goes ahead XcalableMP (XMP) (i collaboratio with U. Tsukuba) Parallel programmig laguage for directive-base easy codig o distributed memory system Not like explicit message passig with MPI 11 Japa-Korea HPC Witer School 2018 Ceter for Computatioal Scieces, Uiv. of Tsukuba

12 Machie locatio: Kashiwa Campus of U. Tokyo U. Tsukuba Kashiwa Campus of U. Tokyo Hogo Campus of U. Tokyo 12 Japa-Korea HPC Witer School 2018 Ceter for Computatioal Scieces, Uiv. of Tsukuba

13 Large Scal Applicatios o Oakforest-PACS ARTED (SALMON) Electro Dyamics La/ce QCD Quatum Chroo Dyamics NICAM & COCO Atmosphere & Ocea Couplig GHYDRA Earthquake Simula<os Seism3D Seismic Wave Propaga<o Joural of Advaced Simulatio i Sciece ad Egieerig Macroscopic grids Vacuum Microscopic grids y Solids Atom Z Electric field x Figure 1: A schematic picture of the multi-scale coordiates system. Left-had side shows the macroscopic coordiate to describe propagatio of the macroscopic electromagetic fields. Righ-had side shows the microscopic coordiates to describe quatum dyamics of electros iduced by the fields. 2. Theoretical formulatio ad scietific aspects Japa-Korea HPC Witer School Theoretical framework To explai our multi-scale simulatio, we cosider a simple case: a liearly polarized laser pulse irradiatig ormally o a surface of bulk Si. We take a coordiate system show i Fig. 1 where the surface of bulk Si is take to be a Z = 0 plae, the directio of the laser electric field is parallel to x-axis, ad the directio of the laser propagatio is parallel to 13