Advanced Topics in High Performance Scientific Computing [MA5327] Exercise 1

Size: px

Start display at page:

Download "Advanced Topics in High Performance Scientific Computing [MA5327] Exercise 1"

Beverly Tucker
5 years ago
Views:

1 Advanced Topics in High Performance Scientific Computing [MA5327] Exercise 1 Manfred Liebmann Technische Universität München Chair of Optimal Control Center for Mathematical Sciences, M17 manfred.liebmann@tum.de October 14, 2015

2 Reading List Introduction to C++ programming for the lecture with a scientific computing angle: Pitt-Francis, Guide to Scientific Computing in C++, Springer

3 Exercises Exercise sheets C/C++ Programming and theoretical exercises Gain experience with parallel programming models Support the development of the software projects 2

4 Computer Access Computer Access Options Your laptop or PC Room: Praktikumsraum (Special access required) Compute server via ssh: Mephisto cluster access via ssh: mephisto.uni-graz.at (internal network) 3

5 Mephisto GPU Cluster User Guide Mephisto GPU Cluster Hardware Configuration Rocks Cluster Software and File System Configuration How to Use the GPU Cluster? 4

6 (1) Mephisto GPU Cluster Hardware Configuration Figure 1: GPU Cluster: mephisto.uni-graz.at 5

7 Network Configuration 1x Mellanox 18 Port QDR 40Gb/s InfiniBand Switch MIS5023Q-1BFR 1x TP-Link 24 Port Gigabit Ethernet Switch TL-SG1024 1x Master Node: mephisto 1x Supermicro SuperServer 7046GT-TRF-FC407 1x Supermicro Motherboard X8DTG-QF 2x Intel Xeon Six-Core CPU 2.67GHz with 12 Cores 1x 40GB DDR3 PC1333 ECC RAM 1x Samsung SSD GB Solid State Boot Disk 1x Mellanox 40Gb/s QDR InfiniBand Board MHQH19B-XTR 4x Nvidia GeForce GTX 480 GPU with 6GB GDDR5 RAM and 1920 Cores 1x Adaptec RAID Controller 6805 with 8x 6Gb/s SATA/SAS Ports 4x Western Digital RE4 2TB Hard Disk Drive: RAID5 Array with 6TB 4x OCZ Vertex3 240GB Solid State Disk: RAID5 Array with 720GB 6

8 5x Compute Node Tesla : compute compute-0-4 5x Supermicro SuperServer 7046GT-TRF-FC407 5x Supermicro Motherboard X8DTG-QF 10x Intel Xeon Six-Core CPU 2.67GHz with 60 Cores 5x 96GB DDR3 PC1333 ECC RAM with a total of 480GB DDR3 RAM 5x OCZ Vertex3 120GB Solid State Boot Disk 5x Mellanox 40Gb/s QDR InfiniBand Board MHQH19B-XTR 20x Nvidia Tesla C2070 GPU with 120GB GDDR5 ECC RAM and 8960 CUDA Cores 1x Compute Node Kepler : compute-0-5 1x Supermicro SuperServer 7047GR-TPRF 1x Supermicro Motherboard X9DRG-QF 2x Intel Xeon Eight-Core CPU E GHz with 16 Cores 1x 256GB DDR3 PC1600 ECC RAM 1x OCZ Vertex4 128GB Solid State Boot Disk 1x Mellanox 40Gb/s QDR InfiniBand Board MCX353A-QCBT 4x Nvidia Telsa K20 GPU with 20GB GDDR5 ECC RAM and 9984 CUDA Cores 7

9 1x Compute Node Phi : compute-0-6 1x Supermicro SuperServer 7047GR-TPRF 1x Supermicro Motherboard X9DRG-QF 2x Intel Xeon Eight-Core CPU E GHz with 16 Cores 1x 256GB DDR3 PC1600 ECC RAM 1x OCZ Vertex4 256GB Solid State Boot Disk 1x Mellanox 40Gb/s QDR InfiniBand Board MCX353A-QCBT 1x Intel Xeon Phi 5110P with 8GB GDDR5 RAM and 60 CPU Cores 8

10 5x Compute Node: compute compute-0-4 5x 96GB DDR3 PC1333 ECC RAM with a total of 480GB DDR3 RAM 20x Nvidia Tesla C2070 GPU with 120GB GDDR5 RAM and 8960 CUDA Cores Figure 2: Nvidia Tesla C2070: 448 CUDA 1.15GHz and 6GB ECC 1.5GHz 9

11 10

12 (2) Rocks Cluster Software and File System Configuration Rocks 6.1 (Emerald Boa) Cluster Software: Very flexible cluster management software Based on CentOS 6.3 operating system Simple compute node installation and management File System Configuration: NFS automount, EXT3 file system 6TB RAID5 HDD array for home directories 720GB RAID5 SSD array for high performance cluster directories 11

13 (3) How to Use the GPU Cluster? Get your GPU computing project approved! Contact: Manfred Liebmann Access the GPU cluster within the University of Graz network: Login: ssh Software Compiler: icc, pgcc, nvcc, g++, mpicxx, gfortran,... Oracle Grid Engine 6.2u5: Batch processing system: qsub myjob.sh, qstat,... Scalasca: Parallel performance analysis tool OpenMPI and 1.82, MVAPICH2 2.0, Mellanox OFED 2.2.0, Nvidia CUDA

14 User Account Configuration Update the paths in the.bashrc file! # CUDA & SCALASCA export PATH=$PATH:/share/apps/cuda-7.0/bin:/share/apps/scalasca/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/cuda-7.0/lib64:/share/apps/scalasca/lib # PGI Compiler export PATH=/share/apps/pgi/linux86-64/15.7/bin:$PATH export MANPATH=$MANPATH:/share/apps/pgi/linux86-64/15.7/man export LM_LICENSE_FILE=$LM_LICENSE_FILE:/share/apps/pgi/license.dat # PGI MPICH #export PATH=/share/apps/pgi/linux86-64/15.7/mpi/mpich/bin:$PATH #export MANPATH=$MANPATH:/share/apps/pgi/linux86-64/15.7/mpi/mpich/man # PGI MVAPICH #export PATH=/share/apps/pgi/linux86-64/15.7/mpi/mvapich/bin:$PATH #export MANPATH=$MANPATH:/share/apps/pgi/linux86-64/15.7/mpi/mvapich/man # Intel Compiler source /share/apps/intel/bin/compilervars.sh intel64 13

15 Oracle Grid Engine Batch Processing System Design goals for the queueing system Maximum performance for all users! Minimum interference between parallel jobs using CPU or GPU resources Not trivial to achieve out of the box Solution for mephisto GPU cluster Queue with one slot per compute server! Only 5 slots are available on the cluster One slot provides 2 Six-Core CPUs and 4 GPUs! 24 hour time limit for all jobs! Only parallel jobs are supposed to be run on the cluster! Modified submission scripts are required for good performance Parallel jobs can run with full hardware utilization with fine grained control High bandwidth 1.44Tb/s Infiniband switch is the only shared resource. 14

16 Simple Submission Script for CPU Jobs The simple CPU submission script configures 12 CPU slots per compute node. Script Directory:/share/apps/ Output:[scriptname].o[jobID] qsub simple.sh #!/bin/sh -f #$ -V #$ -cwd #$ -j y #$ -pe mpi 4 while read line; do echo $line "slots=12" done < $TMPDIR/machines > $TMPDIR/hostfile cat $TMPDIR/hostfile mpirun --np 48 --hostfile $TMPDIR/hostfile./armocpu armo16.inp 15

17 Advanced Submission Script for CPU Jobs The advanced CPU submission script configures a MPI rank to CPU core mapping for 12 cores per compute node. qsub cpu.sh #!/bin/sh -f #$ -V #$ -cwd #$ -j y #$ -pe mpi 4 rank=0; while read line; do for i in ; do echo "rank" $rank"="$line "slot="$i let "rank += 1" done done < $TMPDIR/machines > $TMPDIR/rankfile cat $TMPDIR/rankfile mpirun --np 48 --hostfile $TMPDIR/machines --rankfile $TMPDIR/rankfile./armocpu armo16.inp 16

18 Submission Script for GPU Jobs The GPU submission script configures a MPI rank to CPU core mapping for 4 CPU cores and 4 GPU boards per compute node. qsub gpu.sh #!/bin/sh -f #$ -V #$ -cwd #$ -j y #$ -pe mpi 4 rank=0; while read line; do for i in ; do echo "rank" $rank"="$line "slot="$i let "rank += 1" done done < $TMPDIR/machines > $TMPDIR/rankfile cat $TMPDIR/rankfile mpirun --np 16 --hostfile $TMPDIR/machines --rankfile $TMPDIR/rankfile./armogpu armo16.inp 17

19 Submission Script for PHI Jobs Simple PHI submission script for offload mode. qsub phi.sh #!/bin/sh -f #$ -S /bin/bash #$ -V #$ -cwd #$ -j y #$ -pe phi 1./exe-offload.x 18

20 GNU Compiler Collection Version and The Red Hat Developer Toolset 1.1 and 2.0 can be activated with the scl utility. scl enable devtoolset-1.1 bash scl enable devtoolset-2 bash 19

21 Intel Compiler Version 15.1 Intel Parallel Studio XE 2015 (Professional Edition) source /share/apps/intel/bin/compilervars.sh intel64 20

Introduc)on to Hyades

Introduc)on to Hyades Shawfeng Dong Department of Astronomy & Astrophysics, UCSSC Hyades 1 Hardware Architecture 2 Accessing Hyades 3 Compu)ng Environment 4 Compiling Codes 5 Running Jobs 6 Visualiza)on