KISTI TACHYON2 SYSTEM Quick User Guide

Transcription

1 KISTI TACHYON2 SYSTEM Quick User Guide Ver Feb. SupercomputingCenter

2 1. TACHYON 2 System Overview Section Specs Model SUN Blade 6275 CPU Intel Xeon X GHz(Nehalem) Nodes 3,200 total Cores 25,408 cores 8 cores/node Rpeak 300 TFlops (3,200 nodes) Memory DDR3/1333MHz 76.8TB 24GB/node, 3GB/core Storage interconnect Network 234TB (Disk) 2.3PB (Disk) 2,112TB (Tape) Infiniband 40G 4X QDR 2. USER Environment a. Access Node Type (# of nodes) Hostname(IP) Available Access methods Time Limit for Interactive session login(4) tachyon2.ksc.re.kr (tachyon2a ~ tachyon2d).ksc.re.kr IP : ~ 104 ssh, sftp, X11 20 Min. datamover(3) dm.ksc.re.kr (dm02 ~ dm04).ksc.re.kr IP : ~ 184 ssh, sftp, ftp - compute(3176) tachyon0001 ~ tachyon debug(24) tachyon3177 ~ tachyon3200 ssh 120 Min. To access Tachyon2 system, users have to open a ssh connection to the login nodes, for example, "ssh -l [user_id] tachyon2.ksc.re.kr" To change user's shell, use the "ldapchsh" command, To change password, use the "passwd" command, To check the process's resource Limit on compute node, use the "ulimit -a" command, To check the amount of used SRU Time per User Account, use the "isam" command. b. Storage Section Directory Quota Purge Policy Home Directory /home01 64 GB per account - Scratch Dirctory /scratch2 100 TB per account The file not accessed for last 15 days would be erased. Applications Directory /applic - - To check the directory quota and usage, use a command like this: # lfs quota /home01 (or lfs quota /scratch2) /scratch file system was terminated the service after May

3 c. Programming Environment Name / Path Compilers (/applic/compilers/<compiler>) Debuggers (/applic/debuggers/toolworks) MPI Libraries (/applic/compilers/<compiler>/<compiler_versio N/mpi) Specification l PGI CDK / 2014 l Intel Compiler 11.1 / 2013 / 2015 l gcc (/usr) /4.4.6 / l TotalView l MVAPICH2 1.4/1.5/2.0/2.1 l OpenMPI 1.3.3/1.4.2/1.4.3/1.6.3/1.8.2/1.8.5 Math Libraries (<COMPILER DIR>/applib1, <MPI DIR>/applib2) l FFTW l FFTW /3.3.4 l LAPACK l Scalapack Others Libraries (<COMPILER DIR>/applib1) l HDF / l HDF / l NCARG 4.4.2/5.2.1/6.0.0 l NetCDF Common Libraries (/applic/common) Applications (/applic/applications) Commercial Software (/applic/applications) l CAIRO l CURL l EXPAT l FONTCONFIG l FREETYPE l JASPER l JPEG 9a l LIBPNG l PIXMAN l SZIP 2.1 l ZLIB l Amber 10 l ncview l NAMD 2.9 l PYTHON / l gromacs / /5.1.4 l lammps 5Sep14 / 10Aug15 l siesta 3.2-p1-5 / 4.0 l Gaussian 09-b01, 09-d S/W Information reference - 3 -

4 Libraries Path Pattern Pattern /applic/compilers/pgi/<version>/{applib1,mpi/<mpi_version>/applib2} /applic/compilers/intel/<version>{applib1,mpi/<mpi_version>/applib2} /applic/compilers/gcc/4.1.2/{applib1,mpi/<mpi_version>/applib2} Contents Libraries for compiled with pgi Libraries for compiled with intel Libraries for compiled with gcc Compiler Example Compiler Example GNU PGI Intel gcc o test.exe O3 test.c pgcc/pgcpp/pgf95 o test.exe fast test.c/cc/f90 icc/ifort o test.exe O3 xsse4.2 -m64 test.c/cc/f90 d. Modules Environment Section How to use contents module purge module avail module load compiler/<compiler-version> mpi/<mpi-version> module load applic/<prog_name> module list module unload <list> module add <list> module rm <list> module help <list> module whatis <list> Example module purge module avail module load compiler/intel-11.1 mpi/openmpi module rm mpi module help mpi/openmpi module whatis mpi/openmpi module add mpi/openmpi MPI module avail reference Compiler module avail reference - 4 -

5 e. Running Jobs using SUN Grid Engine(SGE) Section Command Comment Job submission Queue Information Job monitoring qsub job_script showq qstat only own jobs qstat -u '*' all user's jobs ( * or username ) Host Monitoring Job deletion f. Queue Information showhost qdel <jobid> qdel -u <username> The job having the <jobid> would be deleted All jobs submitted by <username> would be deleted. [ As of February 2017 ] Queue Name Wall Clock Limit (hours) Host Range Max CPU Max Running Job Comment normal 48 tachyon[ ] (2970 nodes) 23, public queue exclusive unlimited tachyon[ ] (134 nodes) 1, special queue To check the latest queue configuration, use a command like this: #showq (Queue configuration can be changed flexibly) Serial program job script example (serial.sh) #$ -cwd # use current directory as working directory of job #$ -N serial_job # specify Job Name #$ -q normal # Queue name ##$ -wd /scratch2/<user01>/serialtest # working directory of job, in general it's not necessary. # multi-thread case, please do change OMP_NUM_THREADS value. # For example in gaussian case, # OMP_NUM_THREADS value must be specify to same %Nproc or %Nprocshared value #$ -l OMP_NUM_THREADS=1 export OMP_NUM_THREADS=1 serial.exe - 5 -

6 mpi program job script example(mpi_mvapich2.sh) #$ -cwd # use current directory as working directory of job #$ -N mvapich_job # specify Job Name #$ -pe mpi_fu 32 # specify pe cpu number. #$ -q normal # queue name ##$ -wd /scratch2/<user01>/mvapich # working directory of job, in general it's not necessary. #$ -M my address # registration #$ -m e # send when the Job is completed mpirun_rsh -hostfile $TMPDIR/machines -np $NSLOTS./mpi.exe mpi program job script example(mpi_openmpi.sh) #$ -cwd # use current directory as working directory of job #$ -N mvapich_job # specify Job Name #$ -pe mpi_fu 32 # specify pe cpu number. #$ -q normal # queue name ##$ -wd /scratch2/<user01>/mvapich # working directory of job, in general it's not necessary. #$ -M my address # registration #$ -m e # send when the Job is completed MCAArgs="-mca btl self,openib -mca plm_rsh_num_concurrent 400" MCAArgs="$MCAArgs -mca oob_tcp_listen_mode listen_thread" mpirun $MCAArgs -np $NSLOTS./mpi.exe - 6 -

7 MPI program that uses lots of memory job script example (mpi_mem.sh) #$ -V #$ -cwd #$ -N mvapich_job #$ -pe mpi_4cpu 32 #$ -q normal #$ -R yes #$ -l h_rt=01:00:00 ##$ -M my address ##$ -m e #unset existing MPI affinities export MV2_USE_AFFINITY=0 export MV2_ENABLE_AFFINITY=0 export VIADEV_USE_AFFINITY=0 export VIADEV_ENABLE_AFFINITY=0 mpirun_rsh -hostfile $TMPDIR/machines -np $NSLOTS./numa.sh numa.sh #socket numbers in a compute node SPN=2 #get my MPI rank [ "x$pmi_rank"!= "x" ] && RANK=$PMI_RANK [ "x$mpi_rank"!= "x" ] && RANK=$MPI_RANK [ "x$mpirun_rank"!= "x" ] && RANK=$MPIRUN_RANK [ "x$ompi_mca_ns_nds_vpid"!= "x" ] && RANK=$OMPI_MCA_ns_nds_vpid [ "x$pmi_id"!= "x" ] && RANK=$PMI_ID [ "x$ompi_comm_world_rank"!= "x" ] && RANK=$OMPI_COMM_WORLD_RANK #allocate a cpu core per MPI rank socket=$(( ($RANK + 1) % $SPN )) echo "myrank: $RANK, mysocket: $socket, hostname: $(hostname)" /usr/bin/numactl --cpunodebind=$socket --membind=$socket./mpi.exe # mpi.exe is user execution file - 7 -

8 MPI program that uses redirection job script example #$ -V #$ -cwd #$ -N mvapich_job #$ -pe mpi_4cpu 32 #$ -q normal #$ -R yes #$ -l h_rt=01:00:00 ##$ -M my address ##$ -m e mpirun_rsh -hostfile $TMPDIR/machines -np $NSLOTS./run.sh run.sh./mpi.exe < input.in > std.out - 8 -

9 OpenMP program job script example (mpi_openmp.sh) #$ -cwd #use current directory as job's working directory #$ -N openmp_job #Specify Job Name #$ -pe openmp 4 # Specify number of OpenMP thread #$ -q normal # Queue name ##$ -wd /scratch2/<user01>/openmp # jobs's working directory, in general it's not necessary. #$ -l OMP_NUM_THREADS=4 export OMP_NUM_THREADS=4./omp.exe Hybrid(MPI+OpenMP) job script example (hybrid.sh) #$ -cwd # use current directory as job's working directory #$ -N hybrid_job # specify Job Name #$ -pe mpi_4cpu 8 # Specify number of Task and Threads. #$ -q normal # Queue name ##$ -wd /scratch2/<user01>/hybrid # jobs's working directory, in general it's not necessary. #$ -l OMP_NUM_THREADS=2 # Specify number of OpenMP Threads per MPI Task. # Specify Threads vaule to same below OMP_NUM_THREADS value # or not, Job terminated forcely. export OMP_NUM_THREADS=2 mpirun_rsh -hostfile $TMPDIR/machines -np $NSLOTS./hybrid.exe User Support technical support NISN Helpdesk : accounting related parallelization / optimization education staff: JinWoo Sung account@ksc.re.kr Homepage: parallel@ksc.re.kr NISN Education: