Introduction of Oakforest-PACS Hiroshi Nakamura Director of Information Technology Center The Univ. of Tokyo (Director of JCAHPC)
Outline Supercomputer deployment plan in Japan What is JCAHPC? Oakforest-PACS system Application Summary Impacts of extreme scale computing (2017/11/2) 2
Computational Resource Providers of HPCI Tier1: K Computer at RIKEN Tier2: Supercomputers of 9 universities and 2 research institutes : Oakforest-PACS Oakforest-PACS Impacts of extreme scale computing (2017/11/2) 3
Deployment plan of Tier-2 Supercomputers (as of May. 2017) available at HPCI Consortium (www.hpci-c.jp) http://www.hpci-c.jp/news/hpci-infra-summary.pdf Power consumption indicates maximum of power supply (includes cooling facility) Impacts of extreme scale computing (2017/11/2) 4
PF 1000 100 Towards Exascale Computing Tier-1 and Tier-2 supercomputers move forward to Exascale computing like two wheels Future Exascale Post K Computer RIKEN AICS 10 Oakforest-PACS JCAHPC (The Univ. of Tokyo and Univ. of Tsukuba) 1 Tokyo Tech. TSUBAME2.0 T2K U. of Tsukuba U. of Tokyo Kyoto U. 2008 2010 2012 2014 2016 2018 2020 Impacts of extreme scale computing (2017/11/2) 5
JCAHPC Joint Center for Advanced High Performance Computing (http://jcahpc.jp) director: Hiroshi Nakamura @ ITC, U-Tokyo established in 2013 under agreement between Information Technology Center (ITC) at The University of Tokyo Center for Computational Sciences (CCS) at University of Tsukuba Design, operate and manage next-generation supercomputer system for researchers Community of advanced HPC research Impacts of extreme scale computing (2017/11/2) 6
Procurement Policy of JCAHPC joint procurement by two universities uniform specification, single shared system Each university is financially responsible to introduce the machine and its operation first attempt in Japan the largest class of budget as national universities supercomputer in Japan Oakforest-PACS : largest scale in Japan investment ratio: U. Tokyo : U. Tsukuba = 2:1 Impacts of extreme scale computing (2017/11/2) 7
Oakforest-PACS Impacts of extreme scale computing (2017/11/2) 8 Full operation started Dec. 2016 Official Program started on April 2017 25 PFLOPS peak 8208 KNL CPUs Fat-Tree (full bisection BW) by OmniPath HPL 13.55 PFLOPS: #1 in Japan (2017/6) WW #6(2016/11) #7(2017/6) HPCG WW #3(2016/11) #5(2017/6)
HPCG on Nov. 2016 Impacts of extreme scale computing (2017/11/2) 9
Location of Oakforest-PACS : Kashiwa Campus of U. Tokyo Univ. of Tsukuba Kashiwa Campus Univ. of Tokyo Hongo Campus of U. Tokyo Impacts of extreme scale computing (2017/11/2) 10
Oakforest-PACS in the Room 2 nd floor of Kashiwa Research Complex http://news.mynavi.jp/news/2016/12/02/035/ Impacts of extreme scale computing (2017/11/2) 11
Specification of Oakforest-PACS Total peak performance Total number of compute nodes Compute node Interconnect Product 25 PFLOPS 8,208 Fujitsu PRIMERGY CX600 M1 (2U) + CX1640 M1 x 8node Processor Intel Xeon Phi 7250 (Code name: Knights Landing), 68 cores, 1.4 GHz Memory High BW 16 GB, 490 GB/sec (MCDRAM, effective rate) Product Link speed Topology Low BW 96 GB, 115.2 GB/sec (peak rate) Intel Omni-Path Architecture 100 Gbps Fat-tree with (completely) full-bisection bandwidth Impacts of extreme scale computing (2017/11/2) 12
Computation node & chassis Water cooling wheel & pipe Chassis with 8 nodes, 2U size Computation node (Fujitsu next generation PRIMERGY) with single chip Intel Xeon Phi (Knights Landing, 3+TFLOPS) and Intel Omni-Path Architecture card (100Gbps) Impacts of extreme scale computing (2017/11/2) 13
Rack 15 Chassis with 120 nodes per Rack rear panel radiator Impacts of extreme scale computing (2017/11/2) 14 water cooling pipe
Full bisection bandwidth Fat-tree by Intel Omni-Path Architecture 12 of 768 port Director Switch (Source by Intel) Uplink: 24 Downlink: 24 2......... 1 24 25 48 49 72 All the nodes are connected with FBB Fat-tree globally full bisection bandwidth is preferable for flexible job management. 2/3 of system : University of Tokyo 1/3 of system : University of Tsukuba but job assignment is flexible (no boudary) 362 of 48 port Edge Switch Impacts of extreme scale computing (2017/11/2) 15 2 Compute Nodes 8208 Login Nodes 20 Parallel FS 64 IME 300 Mgmt, etc. 8 Total 8600
Specification of Oakforest-PACS (I/O) Parallel File System File Cache System Type Total Capacity Product Aggregate BW Type Total capacity Product Aggregate BW Power consumption Lustre File System 26.2 PB # of racks 102 DataDirect Networks SFA14KE 500 GB/sec Burst Buffer, Infinite Memory Engine (by DDN) 940 TB (NVMe SSD, including parity data by erasure coding) DataDirect Networks IME14K 1,560 GB/sec 4.2 MW (including cooling) actually ~3.0MW Impacts of extreme scale computing (2017/11/2) 16
Software of Oakforest-PACS Compute node Login node OS CentOS 7, McKernel Red Hat Enterprise Linux 7 Compiler gcc, Intel compiler (C, C++, Fortran), XcalbleMP MPI Intel MPI, MVAPICH2 Library Intel MKL Application Distributed FS Job Scheduler Debugger Profiler LAPACK, FFTW, SuperLU, PETSc, METIS, Scotch, ScaLAPACK, GNU Scientific Library, NetCDF, Parallel netcdf, Xabclib, ppopen-hpc, ppopen-at, MassiveThreads mpijava, XcalableMP, OpenFOAM, ABINIT-MP, PHASE system, FrontFlow/blue, FrontISTR, REVOCAP, OpenMX, xtapp, AkaiKKR, MODYLAS, ALPS, feram, GROMACS, BLAST, R packages, Bioconductor, BioPerl, BioRuby Globus Toolkit, Gfarm Fujitsu Technical Computing Suite Allinea DDT Intel VTune Amplifier, Trace Analyzer & Collector Impacts of extreme scale computing (2017/11/2) 17
Post-K Computer and Oakforest-PACS as the two wheels of HPCI in Japan Oakforest-PACS fills blank period between K Computer and Post-K Computer Installation of Post-K Computer is planned in 2020-2021 Shutdown of K Computer is planned in 2018-2019?? System software in Oakforest-PACS developed for Post-K McKernel OS for Many-core era, for a number of thin-cores without OS jitter and core binding Primary OS (based on Linux) on Post-K, and application development goes ahead XcalableMP (XMP) Parallel programming language for directive-base easy coding on distributed memory system Not like explicit message passing with MPI Impacts of extreme scale computing (2017/11/2) 18
Oakforest-PACS resource sharing program (nation-wide) As JCAHPC (20%) HPCI HPC Infrastructure program in Japan to share all supercomputers (free!) Big challenge special use (full system size) As U. Tokyo (56.7%) Interdisciplinary Joint Research Program General use Industrial trial use Educational use Young & Female special use As U. Tsukuba (23.3%) Interdisciplinary Academic Program Large scale general use Impacts of extreme scale computing (2017/11/2) 19
Applications on Oakforest-PACS 20 ARTED (SALMON) Electron Dynamics Lattice QCD Quantum Chrono Dynamics NICAM & COCO Atmosphere & Ocean Coupling GHYDRA Earthquake Simulations Seism3D Seismic Wave Propagation Impacts of extreme scale computing (2017/11/2)
Summary JCAHPC : joint resource center for advanced HPC by Univ. of Tokyo and Univ. of Tsukuba for community for advanced HPC research Oakforest-PACS is currently #1 supercomputer in Japan and available for nation-wide resource sharing programs Oakforest-PACS and Post-K : two wheels of HPCI Oakforest-PACS is also a testbed for McKernel and XcalableMP system software to support Post-K development Full system scale applications are under development with extreme scale and getting new results fundamental physics, global science, disaster simulation, material science, etc. Impacts of extreme scale computing (2017/11/2) 21