Genius Quick Start Guide

Transcription

1 Genius Quick Start Guide Overview of the system Genius consists of a total of 116 nodes with 2 Skylake Xeon Gold 6140 processors. Each with 18 cores, at least 192GB of memory and 800 GB of local SSD disk. There are 3 sections, a thin node cluster with 86 nodes, 10 large memory nodes and a GPU sections with 20 nodes. Table 1 shows the hardware details of Genius cluster compared to ThinKing: ThinKing Ivy bridge Haswell Thin nodes Genius Large memory GPU Total nodes 176/32 48/ Processor type Intel Xeon E5-2680v2 Intel Xeon E5-2680v3 Skylake Intel Xeon 6140 Base Clock Speed (GHz) Cores per node Total cores Memory per node (GB) 1886 Mhz 64/128@ 2133Mhz 192@ 2666Mhz 768@ 2666Mhz 192@ 2666Mhz Memory per core (GB) 3.2/ Total peak performance CPU(TFlop/s) Network IB QDR 2:1 IB FDR Infiniband EDR Cache (L1 KB/L2 KB/L3 MB) 10x (32i+32d) 10x x(32i+32d) 12x x(32i+32d) 18x Local disk 200 GB HDD 100 GB SSD 800 GB SSD Table 1 Hardware overview The nodes have 2 IB connection. The networkt for the storage is separated from the network of the MPI communication).

2 ThinKing K20Xm ThinKing K40c Genius P100 Total nodes GPU per node Total CUDA cores Memory 6GB 12GB 16 GB Base Clock Speed cores 732MHz 745MHz 1328 MHz Max clock speed cores 784MHz 874MHz 1480 MHz Memory Bandwidth 249,6GB/s 288GB/s 732 GB/s Peak double precision floating point performance 1,31Tflops 1,43Tflops 5,3 Tflops Peak single precision floating point performance 3,95Tflops 4,29Tflops 10,6 Tflops Features SMX, Dynamic Parallelism, Hyper-Q, GPUBoost SMX, Dynamic Parallelism, Hyper-Q, GPUboost NVLink, GPUBoost

3 Connecting to Genius during pilot phase Genius does have a 4 dedicated login nodes. In the closed pilot phase only invited users have access to the login nodes. All users having an active VSC account can connect to the login node with the same credentials using the command: $ ssh vscxxxxx@nodename Where nodename can be one of the following: Normal login nodes: login1-tier2.hpc.kuleuven.be login2-tier2.hpc.kuleuven.be With a visualization capabilities (nvidia Quadro P6000 GPU): login3-tier2.hpc.kuleuven.be login4-tier2.hpc.kuleuven.be Accessing your Data All global storage areas available on Thinking are also available on Genius, so no data migration is required. Table 2 summarizes the available storage areas and their characteristics: Name Variable Type Access Backup Default Quota /user/leuven/30x/vsc30xxx $VSC_HOME NFS Global YES 3 GB /data/leuven/30x/vsc30xxx $VSC_DATA NFS Global YES 75 GB /scratch/leuven/30x/vsc30xxx $VSC_SCRATCH $VSC_SCRATCH_SITE GPFS Global NO 100 GB /node_scratch $VSC_SCRATCH_NODE ext4 Local NO 100GB /mnt/beeond/ $VSC_SCRATCH_JOB BeeGFS Nodes in the job NO 300 GB Table 2 Storage areas overview

4 $VSC_HOME: A regular home directory which contains all files that a user might need to log on to the system, and small 'utility' scripts/programs/source code/... The capacity that can be used is restricted by quota and this directory should not be used for I/O intensive programs. Regular backups are performed. $VSC_DATA: A data directory which can be used to store programs and their results.. Regular backups are performed. This are shpuld not be used for I/O intensive programs. There is a default quota of 75 GB, but it can be enlarged. You can find more information about the price and conditions here: $VSC_SCRATCH/$VSC_SCRATCH_SITE: On each cluster you have access to a scratch directory that is shared by all nodes on the cluster. This directory is also accessible from the login nodes, so it is accessible while your jobs run, and after they finish. No backups are made for that area and files can be removed automatically if they have not been accessed for 21 days. $VSC_SCRATCH_NODE: is a scratch space local to each compute node. Thus, on each node, this directory points to a different physical location and the content is only accessible from that particular worknode, and only during the runtime of your job. Software The software stack on for Genius is still under construction, but you will already find available a all the basic software packages and a number of application software. Everything is built with the 2018a toolchain. We recommend to compile your software on the debugging nodes or request an interactive node and not use login nodes as the OS system and node configuration of the compute nodes is slightly different that the ones of the login nodes. The modules software manager tool is available on Genius as it was on ThinKing. There is a small difference since it is now Lmod. Lmod is a Lua based module system, but it is fully compatible with the TCL modulefiles we ve used in the past. All the module commands that you are used to will work. But Lmod is somewhat faster and adds a few additional features on top of the old implementation. The switch to Lmod should be mostly transparent, i.e. you should not have to change your existing job scripts, but of course you need to take into account the new toolchain. The default MODULEPATH is 2018a. Existing module commands should keep working as they were. The naming scheme for modules remains the same : PackageName/version-ToolchainName-ToolchainVersion Where PackageName is the official name of the software, keeping capital and lower letters. On Genius: $ module av Python /apps/leuven/skylake/2018a/modules/all

5 Boost/ foss-2018a-Python GDAL/2.2.3-intel-2018a-Python GEOS/3.6.2-intel-2018a-Python Mako/1.0.7-intel-2018a-Python Mesa/ foss-2018a-Python Python/ foss-2018a Python/ GCCcore bare Python/ intel-2018a Python/3.6.4-foss-2018a Python/3.6.4-intel-2018a SWIG/ intel-2018a-Python Tkinter/3.6.4-foss-2018a-Python VTK/8.0.1-foss-2018a-Python matplotlib/2.1.2-foss-2018a-python wheel/ foss-2018a-python (D) (D) TIP: Revise your job scripts to ensure the appropriate software package name is used. Use always the complete name of the package (name and version) and do not rely on defaults. On Genius you will need to use the 2018 toolchain version. Compiling and Running your Code Several compilers and libraries are available on Genius, as well as two toolchains flavors intel (based on Intel software components) and foss (based on free and open source software). A toolchain is a collection of tools to build (HPC) software consistently. It consists of: compilers for C/C++ and Fortran a communications library (MPI) mathematical libraries (linear algebra, FFT). Toolchains are versioned and refreshed twice a year. All software available on the cluster is rebuilt when a new version of a toolchain is defined to ensure consistency. Version numbers consist of the year of their definition, followed by either a or b, e.g., 2018a. Note that the software components are not necessarily the most recent releases; rather they are selected for stability and reliability. The 2018a toolchain gives the best support for the new generation of intel CPU s. Older toolchains will not be ported to Genius. Table 4 summarizes the toolchains available on Genius and their components: Intel compilers Open Source compilers Name intel foss version 2018a 2018a Compilers Intel compilers (v ) icc, icpc, ifort MPI Library Intel MPI OpenMPI Math libraries MKL GNU compilers (v ) gcc, g++, gfortran OpenBLAS, ScaLAPACK Table 3. Toolchains on Genius

6 TIP: When recompiling your codes for using them on Genius, check the results of the recompiled codes before starting production runs, and use the available Toolchains for compiling whenever possible. In order compile programs we recommend to start an interactive job in the machine. Running Jobs Torque/Moab is used for scheduling jobs on Genius, so the same commands and scripts used on Thinking will work. Credits For the pilot phase everybody is added to the project lpt2_pilot_2018. There are credits available on this shared project. So to submit you should specify this project (-A lpt2_pilot_2018) always. Later on A option will be obligatory (even for introductory credits). A CPU node in Genius A node unit in Genius is a physical server with 2 CPU, which thus contains 36 cores. The scheduling policy is SINGLE_JOB, which means that only one user per node is allowed. Single core jobs can end up on the same node, but are accounted on a job basis. You should pack single core jobs, eg. With the worker framework, to fill the node in order to be accounted only once per node A GPU node in Genius A GPU node unit in Genius is a physical server with 2 CPU and 4 P100 GPU s. The scheduling policy is SHARED. So this means the node is shared with different users. However the users are separated by cgroups. A cgroup is created based on what was requested by the user. So if a user requests 18 cores and 2 GPUs he/she will only have access to 18 cores and 2 GPUs. If you want the complete node for yourself you will need to request also the complete node: so request 36 cores and 4 GPUs. Queues The current available queues on Genius are: q1h, q24h, q72h and q7d. There will be no 21 day queue during the pilot phase. As before, we strongly recommend that instead of specifying queue names on the batch scripts you use the PBS l option to define your needs. Some useful are l options are: Resources usage -l walltime=4:30:00 (job will last 4h 30 min) -l nodes=2:ppn=36 (job needs 2 nodes and 36 cores per node)

7 -l mem=40gb (jobs request 40 GB of memory, sum for all processes) -l pmem=5gb (job request 5 GB of memory per core, which is the default for the thin node) TIP: don t forget, the CPU nodes have 36 cores Revise your batch scripts to specify correct ppn. GPU Partition As explained before, Genius is split into two partitions with different number of cores and memory configuration. By default, jobs will be scheduled by the system in one of the two partitions according to the resources requested, and the availability. However, it is also possible to manually select one partition and have full control over where the jobs are executed. To specify partitions use the following PBS option: -l partition=partition_name Where partition_name can be either gpu. An example of a job submitted using resource request could be: $ qsub l nodes=10:ppn=36:gpus=4 -l walltime=1:00:00 \ -l pmem=4gb -l partition=gpu -A lpt2_pilot_2018 \ This would request 10 nodes with each 4 GPU s. In case you only need one GPU you can request: $ qsub l nodes=1:ppn=9:gpus=1 -l walltime=1:00:00 \ -l pmem=4gb -l partition=gpu -A lpt2_pilot_2018 \ Should your program launch more than 1 process on the GPU you need to add the :default option. $ qsub l nodes=1:ppn=9:gpus=1:default -l walltime=1:00:00 \ -l pmem=4gb -l partition=gpu -A lpt2_pilot_2018 \ Note that you really need to explicitly request the number of GPUs you want to use, otherwise they will be invisible. Large memory nodes To submit to the large memory nodes you will need to also specify explicitly lpartition=bigmem together with the amount of memory you will need for your job. For example: $ qsub l nodes=2:ppn=36 -l walltime=1:00:00 \ -l pmem=20gb -l partition=bigmem -A lpt2_pilot_2018 \

8 Debugging queue At this moment Genius has 1 GPU node for compiling and debugging purposes. You can request the node for a maximum of XX minutes by specifying the QOS: $ qsub l nodes=1:ppn=36 -l walltime=30:00 \ -l qos=debugging -l partition=gpu -A lpt2_pilot_2018 \ So mind specifying qos and also partition. The gpu debug node is a shared node. So if you want the node exclusive for you, you will have to reserve all cores using lnodes=1:ppn=36. the maximum walltime is 30 minutes, which is shorter than the default walltime. So this should be specified. Running tensorflow on GPU s Install miniconda in your VSC_DATA (and never in your VSC_HOME): o o If you already had one, please update your Conda to the latest release: o $> conda update conda Start an interactive session on one of the GPU nodes: o $> qsub I l partition=gpu,nodes=1:ppn=1:gpus=1 A <myproject-name> where the 1st argument is capital I to request an interactive node, the 2nd is the lowcase l to specify the resources, and the last one is your project name for debiting the credits (e.g. for the pilot phse please use lpt2_pilot_2018) Create a new conda environment with the latest TF-gpu: o $> conda create n py36-tf19 python=3.6 tensorflow-gpu=1.9.0 Currently, tensorflow-gpu version is the latest compatible one. The newer ones require higher versions of the Nvidia driver than already installed on the cluster. Ensure that TF-gpu can be imported without error, and can identify the two devices attached to the node: o $> conda activate py36-tf19 o $> python o $> import tensorflow as tf o $>sess=tf.session(config=tf.configproto(log_device_placement=tr ue)) Running PyTorch on GPU s It is very straightforward to manage a conda environment that includes PyTorch: Install miniconda in your VSC_DATA (and never in your VSC_HOME): Create a new conda environment with the latest PyTorch: o $> conda create n py36-torch python=3.6 Install the latest PyTorch from a channel: o $> conda activate py36-torch o $> conda install pytorch torchvision cuda91 c pytorch o $> python c import pytorch; print(pytorch. path )

9 New Features Using local disks as temporary scratch with BeeOnd As an alternative to the GPFS scratch space we also now provide on Genius the possibility for the user to request BeeGFS ( ) when launching your job. This will spawn during your job a parallel shared filesystem on the compute nodes using the local disks of the nodes used within your job. During your job this filesystem will be mounted on /mnt/beeond/. After your job is completed the filesystem will be destroyed so don t forget to include a file transfer to a safe place in your jobscript. To request this filesystem you have to submit for example like this: $ qsub l nodes=2:ppn=36:beeond -l walltime=1:00:00 \ -l pmem=5gb -A lpt2_pilot_2018 \