CSC resources intro: most powerful computational resources in Finland. Tomasz Malkiewicz CSC IT Center for Science Ltd.

Transcription

1 CSC resources intro: most powerful computational resources in Finland Tomasz Malkiewicz CSC IT Center for Science Ltd.

2 Outline Intro: Why supercomputers? CSC at glance Kajaani Datacenter Finland s new supercomputers Sisu (Cray XC30) Taito (HP cluster) Live demo/hands-on (Taito) CSC resources available for researchers

3 Supercomputing: serial and parallel processing Serial computing single processing unit (core) is used for solving a problem single task performed at once Parallel computing P r o b l e m c o r e r e s u l t multiple cores are used for solving a problem c1 problem is split into smaller subtasks multiple subtasks are performed simultaneously P r o b l e m c2 c3... cn r e s u l t

4 Types of parallel computers Shared memory all the cores can access the whole memory Distributed memory all the cores have their own memory communication is needed in order to access the memory of other cores Current supercomputers combine the distributed memory and shared memory approaches

5 Data parallelism Data is distributed to processor cores Each core performs (nearly) identical tasks with different data Example: summing the elements of a 2D array core 1: = core 2: = core 3: = core 4: = Each core sums it's part of the array The individual sums have to be combined in the end

6 Task parallelism Different cores perform different tasks with the same or different data Example: signal processing, four filters as separate tasks... data Data is processed as segments Core 2 obtains a segment after core 1 has processed it; core 1 starts to process a new segment When the first segment gets to core 4, all cores are busy f i l t e r core 1 f i l t e r core 2 f i l t e r core 3 f i l t e r core 4

7 ns/day Why supercomputers? Lipid MD, 120katoms, PME, Gromacs louhi vuori taito sisu cores

8 CSC and High Performance Computing

9 Supercomputers Supercomputer is a computer at the frontline of contemporary processing capacity particularly speed of calculation. Fastest supercomputers: China s Tianhe-2 (no.1, PFlop/s on the LINPACK benchmark), Finland s Sisu (no. 189, 245 TFlop/s) Finland s Taito (no. 267, 191 TFlop/s).

10 CSC Computing Capacity

11 CSC at glance Founded in 1971 technical support unit for Univac 1108 Connected Finland to Internet in 1988 Reorganized as a company, CSC Scientific Computing Ltd. in 1993 All shares to the Ministry of Education and Culture of Finland in 1997 Operates on a non-profit principle Facilities in Espoo and Kajaani Staff ~250 people

12 CSC s Services FUNET Services Computing Services Application Services Data Services for Science and Culture Information Management Services Universities Polytechnics Ministries Public sector Research centers Companies

13 FUNET and Data services FUNET Connections to all higher education institutions in Finland Haka-identity Management Campus Support The NORDUnet network Data services Digital Preservation and Data for Research Data for Research (TTA), National Digital Library (KDK) Database and information services nic.funet.fi freely distributable files with FTP since 1990 Memory organizations (Finnish university and polytechnics libraries, Finnish National Audiovisual Archive, Finnish National Archives, Finnish National Gallery)

14 Users About 700 active computing projects 3000 researchers use CSC s computing capacity 4250 registered customers Haka-identity federation covers all universities and higher education institutes ( users) Funet - Finnish research and education network Total of end users

15 Users of computing resources by discipline Biosciences Physics Chemistry Total active users Language research Nanoscience Computational fluid dynamics Engineering Computational drug design Earth sciences Other disciplines 153 CSC presentation 15

16 Computing usage by discipline % 5 % 1 % 1 % 5 % 38 % Physics Nanoscience Chemistry 8 % Total 201,6 million billing units Biosciences Astrophysics Computational fluid dynamics 14 % Materials sciences Computational drug design Other disciplines 23 % CSC presentation 16

17 Users of computing resources by organization University of Helsinki Aalto University 65 University of Turku University of Oulu Total active users CSC (Projects) Tampere University of Technology University of Jyväskylä CSC (PRACE) 96 University of Eastern Finland University of Tampere Other CSC presentation 17

18 THE KAJAANI DATACENTER

19

20

21 Power distribution (FinGrid)

22 Kajaani site

23 Sisu now

24 Sisu rear view

25 Taito (HP) hosted in SGI Ice Cube R80

26 SGI Ice Cube R80

27 Taito now

28 Data center specification 2.4 MW combined hybrid capacity 1.4 MW modular free air cooled datacenter Hosting e.g. Taito Upgradable in 700 kw factory built modules Order to acceptance in 5 months 35 kw per extra tall racks 12 kw common in industry PUE forecast < 1.08 (ppue L2,YC ) 1MW HPC datacenter Optimised for Cray super & T-Platforms prototype 90% Water cooling

29 CSC SUPERCOMPUTERS

30 Overview of New Systems Phase 1 Phase 2 Cray HP Cray HP Deployment Done Done Probably 2014 CPU Intel Sandy Bridge GHz Next generation processors Interconnect Aries FDR InfiniBand Aries FDR InfiniBand Cores ~40000 ~17000 Tflops 244 (2x Louhi) 180 (5x Vuori) ~1700 (16x Louhi) ~515 (15x Vuori) Tflops total 424 (3.6x Louhi) ~2215 (20.7x Louhi)

31 IT summary Sisu: Cray XC30 supercomputer Fastest computer in Finland Phase 1: 385 kw, 244 TFlop/s, 16 x 2 GB cores per computing node, 4 x 256 GB login nodes Very high density, large racks

32 IT summary cont. Taito: HP supercluster 1152 Intel CPUs 16 x 4 GB cores per node 16 fat nodes with 16 x16 GB cores per node 6 x 64 GB login nodes 180 TFlop/s 30 kw 47 U racks HPC storage PB of fast parallel storage Supports Cray and HP systems

33 Sisu Phase2: Cray Supercomputer - Future Intel Xeon processor E v3 product family - Cray Aries Interconnect - ~ cores - 64 GB memory per node CSC presentation 33

34 Cray Dragonfly Topology All-to-all network between groups 2 dimensional all-to-all network in a group Source: Robert Alverson, Cray Hot Interconnects 2012 keynote Optical uplinks to inter-group net

35 Cray environment (Sisu) Typical Cray environment Compilers: Cray, Intel and GNU Cray mpi, Cray tuned versions of all usual libraries SLURM Default shell: bash (previously tcsh) Character encoding: UTF-8 Latin-15 alias iso currently kept as is on Vuori and Hippu

36 Sisu (Phase 2) AVX-2 May need to optimize for wider vectors size DDR4 Higher bandwidth, lower power consumption Max job size likely to increase Native SLURM on the way, unlikely to be available after Sisu July hardware update CSC presentation 36

37 Taito Phase2: HP Supercluster - Intel Xeon processor E v2 product family & Future Intel Xeon processor E v3 family - FDR InfiniBand interconnect - ~ cores - Different memory per node sizes: 64, 128, 256 GB and 1.5 TB CSC presentation 37

38 HP Environment (Taito) Compilers: Intel, GNU MPI libraries: Intel, mvapich2, OpenMPI Batch queue: SLURM New more robust module system Only compatible modules shown with module avail Use module spider to see all Disk system changes Default shell: bash (used to be tcsh) Character encoding: UTF-8

39 Core development tools Intel XE Development Tools Compilers C/C++ (icc), Fortran (ifort), Cilk+ Profilers and trace utilities Vtune, Thread checker, MPI checker MKL numerical library Intel MPI library (only on HP) Cray Application Development Environment GNU Compiler Collection Tokens shared between HP and Cray TotalView debugger

40 GFlop/s Performance of numerical libraries DGEMM 1000x1000 Single-Core Performance Turbo Peak (when only 1 core is 3.5GHz * 8 Flop/Hz GHz * 8 Flop/Hz Sandy Bridge 2.7GHz Opteron Barcelona 2.3GHz (Louhi) GHz * 4 Flop/Hz ATLAS 3.8 ATLAS 3.10 ACML 5.2 Ifort 12.1 RedHat 6.2 RPM matmul MKL 12.1 LibSci ACML MKL 11 MKL the best choice on Sandy Bridge, for now. (On Cray, LibSci a good alternative)

41 Modules Some software installations are conflicting with each other For example different versions of programs and libraries Modules facilitate the installation of conflicting packages to a single system User can select the desired environment and tools using module commands Can also be done "on-the-fly"

42 Taito module system module avail shows only those modules that can be loaded to current setup (no conflicts or extra dependencies) Use module spider to list all installed modules and solve the conflicts/dependencies No PrgEnv- modules (on Taito) Changing the compiler module switches also MPI and other compiler specific modules

43 Live demo/hands-on (Taito) ssh trng01 - mkdir own_username cd own_username module avail

44 Live demo/hands-on cont. nano test_hostname.sh CTRL+O; CTRL+X to exit #!/bin/bash -l #SBATCH -J print_hostname #SBATCH -o output.txt #SBATCH -e errors.t #SBATCH -t 00:01:00 #SBATCH -p test # echo "This job runs on the host:"; hostname sbatch test_hostname.sh

45 Live demo/hands-on cont. Check out the output: less output.txt (type q to quit) less errors.t (type q to quit) More examples:

46 Disks space taito.csc.fi login nodes sisu.csc.fi login nodes Your workstation irods client SUI compute nodes compute nodes $TMPDIR $TMPDIR $TMPDIR $TMPDIR $WRKDIR $HOME $TMPDIR New tape $ARCHIVE in Espoo irods interface disk cache icp, iput, ils, irm $USERAPPL $HOME/xyz icp

47 Disks space cont. 4 PB on DDN New $HOME directory (on Lustre) $WRKDIR (not backed up), soft quota: 5 TB HPC_ARCHIVE 2 TB / user, common between Cray and HP Disk space through IDA 1 PB for Univerisities 1 PB for Finnish Academy (SA) 1 PB to be shared between SA and ESFRI more could be requested

48 Moving files, best practices tar & bzip first rsync, not scp rsync -P Blowfish may be faster than AES (CPU bottleneck) Funet FileSender (max 50 GB) Files can be downloaded also with wget Consider: SUI, IDA, irods, batch-like process, staging CSC can help to tune e.g. TCP/IP parameters /pert FUNET backbone 10 SCiP Gbit/s

49 ARCHIVE, dos and don ts Don t put small files in HPC ARCHIVE Small files waste capacity Less than 10 MB is small Keep the number of files small Tar and bzip files Don t use ARCHIVE for incremental backup (store, delete/overwrite, store, ) Space on tape is not freed up until months or years! Maximum file size 300 GB Default quota 2 TB per user

50 CSC RESOURCES AVAILABLE FOR RESEARCHERS

51 Currently available computing resources Sisu > cores, >23TB memory Taito Small and medium-sized tasks Application server Hippu Interactive usage, without job scheduler Postprocessing, e.g. vizualization FGI Cloud Bull system

52 Three service models of cloud computing Software Operating systems Computers and networks SaaS PaaS IaaS

53 Example: Virtualization in Taito Taito cluster: two types of nodes, HPC and cloud HPC node HPC node Cloud node Cloud node Host OS: RHEL Virtual machine Guest OS: Ubuntu Virtual machine Guest OS: Windows

54 Bull In pilot/project until end of August 2014 No guarantee on availability 38 NVIDIA K40 nodes (76 gpus) 12 GB memory per card 45 Intel Xeon Phi nodes (90 Xeon Phis) 16 GB memory per card Energy efficient CPU s CSC presentation 54

55 How to access Bull (plan) Accessing the resources Intel Xeon Phi: ssh taito-mic.csc.fi (TBC) NVIDIA K40: ssh taito-gpu.csc.fi CSC presentation 55

56 Grand Challenges Normal GC (in half a year / year) new CSC resources available for a year no bottom limit for number of cores Special GC call (mainly for Cray) (when needed) possibility for short (day or less) runs with the whole Cray Remember also PRACE/DECI

57 NX screenshot

58 Courses Sisu Phase 2 workshop Late 2014 Taito Phase 2 workshop Spring 2015 CSC courses: CSC HPC Summer School Spring, Autumn, Winter Schools Introduction to Linux and Using CSC Environment Efficiently Parallel Programming CSC presentation 58

59 How to get access to CSC supercomputers? sui.csc.fi (HAKA authentication) Sing up

60 ns/day Summary Sisu supercomputer Installation in July-August 2014 General availability in Q Taito supercluster Installation planned in Q cores Bull system General availability planned for Q nodes with 2 Intel Xeon Phi coprocessors each 38 nodes with 2 NVIDIA Tesla K40 accelerators each DDN HPC storage system Adding 1.9 PB, in Q totaling 4 PB of fast parallel storage Supports Cray and HP systems, aggregate bandwidth > 80 GB/s Gromacs performance Taito Sisu FGI Vuori Louhi CSC presentation 60