Professor Boris M. Glinsky Nikolay V. Kuchin

Transcription

1 Professor Boris M. Glinsky Nikolay V. Kuchin HP-CAST

2 The Siberian Supercomputer Center The supercomputer center of the collective use The Siberian Supercomputer Center (SSCC) is created as a laboratory of the Institute of Computational Mathematics and Mathematical Geophysics (ICM&MG) according to the Directive of Presidium of the Siberian Branch of the Russian Academy of Sciences (SB RAS) of No Now the center of the collective use The Siberian Supercomputer Center is the joint project of ICM&MG and the Institute of Cytology and Genetics (IC&G) SB RAS. Scientific Supervisor of the Siberian Supercomputer Center - Academician, Professor Boris G. Mikhailenko is the director of the ICM&MG SB RAS and the Chairman of Scientific Council on Supercomputing of the SB RAS. HPC Competence Center SB RAS Intel has been organized in

3 The Siberian Supercomputer Center The main objectives of the Siberian Supercomputer Center are as follows: providing the modern computing resources hardware and software to the researchers from the Siberian Branch of the Russian Academy of Sciences (SB RAS) and Universities of Siberia; training the researchers of the SB RAS and students of the universities the modern methods of parallel computing and solving large scale problems on supercomputers; managing the development of all Supercomputer Centers of the SB RAS (according to directives of the Scientific Council on Supercomputing of the SB RAS). 3

4 Training, schools and seminars 1) The International Conference Parallel and Computing Technologies 2012 was held by the ICMMG with 242 participants from Russia, Kazakhstan, Ukraine, Germany, France, the USA, see at 2) NVIDIA CUDA Technology 3 days trainings was held in The School was supported by the specialists of NVIDIA on computer resources of the cluster. 118 listeners from the research institutes of SB RAS, Higher Education Institutions and Firms were trained. Program and teaching materials are displayed at: 3) The Workshop on Parallel Programming of Hybrid Clusters was held in the December of 2012 : see at: 4) Trainings in high-performance computing and mathematical modeling at 5 Chairs at NSU and NSTU by the specialists of ICMMG SB RAS: Mathematical Methods in Geophysics Chair (NSU); Numerical Analysis Chair (NSU); Parallel Computations Chair (NSU); Computer Systems Chair (NSU); Parallel Computer Technologies Chair (NSTU). 5) The Architecture, System and Application Software of Cluster Supercomputers seminars are regularly provided on the basis of the SSCC, the Chair of Computer Systems of NSU and the Competence Center on High-Performance Computing of SB RAS/Intel. The Seminars are presented at: 4

5 COMPUTING RESOURCES GROWTH МVS-1000/128М NКS-30Т 4,8 TFlops 0,246 TFlops ,247 TFlops 2006 NКS-30Т 16,5 TFlops > 1 TFlops 2007 NКS-160 5,8 Т TFlops 2008 NКS-30Т 30 TFlops 7,1 Т TFlops 2009 NКS TFlops 17,5 Tflops 2010 NКS-160 > 1 TFlops Hybrid Cluster 115 TFlops 31 TFlops TFlops 2012 МVS-1000/32 NКS-160 5

6 Hardware and software Hybrid cluster HKC-30T+GPUs with Cool Aisle Containment System (CACS) 576 CPU (2688 cores) Intel Xeon Е5450/E5540/X5670; 80 CPU CPU (X5670) (480 cores); Software RedHat 5.4 operating system PBSPro 11.1 batch system HP Cluster Management Utility Toolkits: Storage subsystem 120 GPUs Tesla M2090 (61440 cores). Peak performance 115 TFlops Shared memory server (hp DL980 G7) Intel Cluster Studio XE 2013, Including Intel compilers and MPI 4.1. NVIDIA CUDA 5, Portland Group PGI Accelerator. Application programs: ANSYS CFD (Fluent), Gaussian 09, Bioscope. Software developed in ICMMG: PARMONC, AGNES. Cluster file system IBRIX 4 servers, 32 Tbytes 4 CPU (40 cores) Intel Е7-4870; RAM Gbytes; 384 GFlops. 6

7 Hybrid cluster (NKS-30Т + GPUs) NKS 30T cluster Cluster extension with GPUs 64 BL2x220c G5 128 BL2x220c G BL2x220c G SL390s G GPUs M Shared Memory server DL980 G7 RAM 512 GB 40 cores 384 GFlops I N F I N I B A N D..... IBRIX Cluster File system 4 servers 32 TBytes IBRIX Cluster File system Future extension / plans xx servers 7

8 Hardware: more details 7 HP BladeSystem c7000 Enclosure 2 c7000 Enclosure, 32 HP BL2х220c G5, 64 Compute nodes, 128 CPU Intel Xeon E5450, 512 Cores, RAM 16 GB, 6.1 TFlops. 4 c7000 Enclosure, 64 HP BL2х220c G6, 128 Compute nodes, 256 CPU Intel Xeon Е5540, 1024 Cores, RAM 16 GB, Tflops. BL2х220c G6 system boards were redesigned in the end We are going to replace all old system boards with the new redesigned system boards. 3 c7000 Enclosure, 48 HP BL2х220c G7, 96 Compute nodes, 192 CPU Intel Xeon X5670, 1152 Cores, RAM 24 GB, 13.5 Tflops. 40 SL390s G7, each server has 2 CPU Intel Xeon X5670, RAM 96 GB, 3 GPU NVIDIA Tesla M2090. Peak performance of 40 servers is 85 TFlops. DL980 G7, 4 CPU Intel Е7-4870, 40 Cores, RAM 512 GB, local RAID 1.5 Tbytes, 384 GFlops. IBRIX cluster file system 4 HP DL380 G6, 2 CPU Intel Xeon E5520, 8 Cores, RAM 48 GB. 4 HP 2000sa Modular Smart Array (MSA2000sa), 32 TB (48 TB row space). IBRIX is very good system but its configuration is too small for our cluster. IBRIX throughput is bottleneck for some user applications. 8

9 Software: more details System software RHEL 5.4 Altair PBS Pro 11.1 batch system HP Cluster Management Utility Toolkits Intel Cluster Studio XE 2013 for Linux Compilers Intel С/C++ и Intel Fortran Composer XE Intel MPI 4.1, Trace Analyzer & Collector 8.1 NVIDIA CUDA 5 Portland Group PGI Accelerator 13.4 Application programs ANSYS CFD (Fluent) Gaussian 09 (in the next 2 or 3 months) Bioscope, Gromacs, Quantum Espresso Software developed in ICMMG PARMONC, AGNES 9

10 The program library PARMONC for the parallel Monte-Carlo simulation M.A. Marchenko, ICM&MG SB RAS, The program library PARMONC (PARallel MONte Carlo) is designed for parallelization of the time-consuming Monte-Carlo simulations. The core of the library is the parallel long-period random numbers generator developed in ICM&MG. Using the PARMONC, the Monte Carlo codes written in FORTRAN or C may be easily parallelized without explicit use of MPI. Scope: time-consuming applications in the natural sciences (physics, chemistry, biology, etc.) References: 1) Marchenko, M.: PARMONC - A software library for massively parallel stochastic simulation. LNCS, vol. 6873, pp Springer, Heidelberg (2011) 2) 10

11 Usage of computing resources in ,98 % ICG Institute of Cytology and Genetics 19,84 % IC Institute of Catalysis 15,67 % ICMMG Institute of Computational Mathematics and Mathematical Geophysics 10,47 % NSU Novosibirsk State University 6,16 % ICKC Institute of Chemical Kinetics and Combustion 4,69 % ICCT Institute of Chemistry and Chemical Technologies, Krasnoyarsk 2,90 % ITP Institute of Thermophysics 1,80 % INP Institute of Nuclear Physics 1,31 % ICT Institute of Computational Technologies 1,20 % IPGG Institute of Petroleum Geology and Geophysics 0,79 % ITAM Institute of Theoretical and Applied Mechanics 0,35 % NSTU Novosibirsk State Technical University 0,27 % ICBFM Institute of Chemical Biology and Fundamental Medicine 0,26 % IEC Institute of the Earth Cryosphere, Tyumen 0,05 % NIIC Institute of Inorganic Chemistry 0,05 % ILP Institute of Laser Physics 0,02 % ISP Institute of Semiconductor Physics 11

12 Scientific Researches Areas (from annual reports of users) 1) Industry of nano-systems - ICMMG, IC, ITAM, ICKC, IFS, INP, ICCT (Krasnoyarsk), IEC (Tyumen) 2) Information-Telecommunication systems - ICT, ICMMG, NIIC, NSU, NSTU, ICG, INP 3) Energy Efficiency, Economy of Power, Nuclear Power - ICT, ICMMG, IC, ITP, ICKC, INP, NSU, NSTU, IPGG 4) Life Sciences - ICBFM, ICG, ICMMG, NSU, IC, IEC (Tyumen) 5) Rational Use of Natural Resources - ICMMG, IPGG, ITP, NSU, ICCT (Krasnoyarsk), IEC (Tyumen) 6) Transport and Cosmic Systems -ITAM, NSTU Annual reports of our users in 2012: Our users are carried out 152 grants, programs, projects, topics, including 2 international grants. The Russian Foundation for Basic Research (RFBR) grants 54, Russian Academy of Sciences Programs 16, SB RAS Projects 37, the Ministry of Education Programs 23, others

13 Usage of computing resources in Accounting statistics (NKS NKS-30T) peak performance (TFlops) 7,1 17,5 31,0 116 CPU (hours) , , , ,11 jobs number

14 Processing the results of experiments in high energy physics Сommon virtualized computing environment has been designed in the Institute of Nuclear Physics of the SB RAS (INP). It includes computing resources of HPC clusters of the Siberian Supercomputer Center and Novosibirsk State University. Additional software, including KVM, has been installed on cluster compute nodes. Special network (10 Gbits) has been used as a transport level. All data are located in INP and are accessible via NFS. PBS prologue and epilogue are used to start and stop kvm virtual machine on compute node and start / stop NFS over IB plus 10 Gbits network. KEDR KEDR is a large scale particle detector experiment being carried out at VEPP-4M electronpositron collider at INP. The offline software of the experiment was being developed since late 90 s. After several migrations the standard computing environment was frozen on Scientific Linux CERN 3 i386 and no further migrations are expected in the future. The SND detector experiment which is being carried out at INP at VEPP-2000 collider have successfully adopted the virtualization solution previously built for KEDR detector in order to satisfy their own needs for HPC resources. The INP user group doing data analysis for ATLAS experiment at the LHC machine (CERN, Switzerland) within the framework of ATLAS Exotics Working Group have joined the activity as well. 14

15 Future Work Which parallel file system the new cluster needs (Lustre, Panasas)? Some of Bioinformatics application use big volumes of data. Which accelerator is the best for user applications (Xeon Phi, Nvidia Kepler)? We can evaluate Intel Xeon Phi now. We have access to Intel Xeon Phi cluster located in Joint Supercomputer Center RAS, Moscow. HPC cluster workload optimization: combining CPU intensive applications with I/O intensive applications. But it s impossible to predict cluster workload. 15

16 Thank you 16