Shifter and Singularity on Blue Waters Maxim Belkin June 7, 2018
A simplistic view of a scientific application DATA RESULTS My Application
Received an allocation on Blue Waters! DATA RESULTS My Application DATA i RESULTS My Application Your Computer Blue Waters
A more realistic view of a scientific application LIBRARY X EXECUTABLE Y HARDWARE Z DATA My Application RESULTS Your Computer
A more realistic view of a scientific application LIBRARY A EXECUTABLE B HARDWARE C DATA RESULTS My Application Blue Waters
Software Containers LIBRARY X EXECUTABLE Y HARDWARE C DATA My Application RESULTS
Shifter Docker Singularity https://github.com/nersc/shifter Designed for HPC https://github.com/docker Not suitable for HPC https://github.com/singularityware/singularity Can be used in HPC environment
Docker FROM centos:7 Dockerfile RUN yum update &&... wget COPY file.ext inside/container/ RUN./configure && make && WORKDIR /home/user 1) Build: docker build -t repo/name:tag. 2) Upload: docker push repo/name:tag 3) Download: docker pull repo/name:tag 4) Run: docker run repo/name:tag 4a) Execute: docker exec <container> command
https://github.com/nersc/shifter https://github.com/singularityware/singularity
-l gres=shifter16 Docker ssh qsub aprun User Login nodes MOM nodes Compute nodes
-l gres=shifter16 -l gres=shifter16 -v UDI=repo/name:tag export CRAY_ROOTFS=SHIFTER Docker qsub aprun -b... -- app-in-container --options Login nodes MOM nodes -v UDI=repo/name:tag UDI: User-Defined Image application application application application application application application application Compute nodes
-l gres=shifter16 Docker -l gres=shifter16 qsub module load shifter aprun -b... -- shifter --image=repo/name:tag -- app-in-container --options Login nodes MOM nodes application application application application application application application application Compute nodes
Shifter module shifter shifterimg images lookup pull login launches an app from UDI on compute nodes returns info about available UDIs returns an ID of an image (if available) downloads Docker image and converts it to UDI authenticates to DockerHub (for private images)
shifterimg images shifterimg lookup shifterimg pull shifterimg login Examples ubuntu:zesty centos:latest image on Blue Waters image on DockerHub shifterimg pull --user bw_user prvimg1:latest shifterimg pull --group bw_grp prvimg2:latest
LIBRARY X EXECUTABLE Y HARDWARE Z DATA My Application RESULTS
MPI in Shifter: requirements In a Container: 1. glibc 2.17+ (CentOS 7, Ubuntu 14) 2. Use one of the following MPIs (see MPICH-ABI compatibility) MPICH v3.1+ Intel MPI Library v5.0+ Cray MPT v7.0.0+ MVAPICH2 2.0+ Parastation MPI 5.1.7-1+ IBM MPI v2.1+ 3. Do not use package manager to install MPI *
MPI in Shifter: requirements On Blue Waters: 1. Use Intel or GNU Programming Environment 2. Load MPICH ABI-compatible MPI (on BW: cray-mpich-abi) 3. Set LD_LIBRARY_PATH to the location of MPICH-ABI libraries
MPI in Shifter: in Dockerfile FROM centos:7 RUN yum -y install file gcc make gcc-gfortran gcc-c++ wget curl RUN cd /usr/local/src/ && \ wget http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz && \ tar xf mpich-3.2.tar.gz && \ rm mpich-3.2.tar.gz && \ cd mpich-3.2 && \./configure && \ make && make install && \ cd /usr/local/src && \ rm -rf mpich-3.2*
MPI in Shifter: job batch script snippet 1 2 3 4 5 6 7 8 9 10 11 12 module unload PrgEnv-cray module unload cce module load PrgEnv-gnu module unload cray-mpich module load cray-mpich-abi LIBS=$CRAY_LD_LIBRARY_PATH:/opt/cray/wlm_detect/default/lib64 CACHE=$PWD/cache.$PBS_JOBID mkdir -p $CACHE for dir in $( echo $LIBS tr ":" " " ); do cp -L -r $dir $CACHE; done export LD_LIBRARY_PATH=$CACHE/lib:$CACHE/lib64 19
MPI in Shifter 1 qsub -l gres=shifter16 -l nodes=2:ppn=32:xe module load shifter aprun -b -n 64 -N 32 -- shifter --image=repo/image:tag -- app aprun -b -n 64 -N 32 -- shifter --image=repo/image2:tag -- app2 2 qsub -l gres=shifter16 -v UDI=repo/image:tag -l nodes=2:ppn=32:xe export CRAY_ROOTFS=SHIFTER aprun -b -n 64 -N 32 -- app 20
MPI in Shifter T(N, ppn=1) a Job start-up time, s 256 128 64 32 16 8 4 ppn = 1 36 MB 1.7 GB CLI prologue T(N=80, ppn) b Job start-up time, s 2 64 32 16 8 2 4 8 16 32 64 128 256 512 1024 2048 4096 Nodes 80 nodes 36 MB 1.7 GB CLI prologue 1 2 4 1 2 4 8 16 processes per node (ppn) Hon Wai Leong DOI: 10.1145/3219104.3219145 21
Performance of MPI applications in Shifter a Time, µs b Time, ms 10 7 10 6 10 5 Shifter CLE MPI_Alltoall MPI_Alltoallv 10 4 10 3 10 2 1 10 10 2 10 3 10 4 10 5 10 6 Message size, bytes 4 Shifter CLE MPI_Bcast 3 MPI_Reduce 2 1 0 0 200 400 600 800 1000 1200 Message size, kilobytes IOR Read, GB/s IOR Write, GB/s 8.5 8.0 7.5 3.5 3.0 2.5 Shifter Cray Linux Environment Shifter Cray Linux Environment 0 10 20 30 Benchmark number Galen Arnold DOI: 10.1145/3219104.3219145 22
Using GPUs in Shifter 1. Use Shifter CLI: shifter --image... 2. Set CUDA_VISIBLE_DEVICES: export CUDA_VISIBLE_DEVICES=0 aprun -b... -- shifter --image=centos:latest -- nvidia-smi 23
Using MPI + GPUs in Shifter Caveat: shifter command removes LD_LIBRARY_PATH Solution: use a script to pass LD_LIBRARY_PATH aprun -b -n... -N 1 -e VAR="$LD_LIBRARY_PATH... \ shifter --image=repo/name:tag... -- script.sh script.sh: export LD_LIBRARY_PATH= $VAR application 24
Directory mapping between BW and Shifter UDI shifter... --volume=/path/on/bw:/path/in/udi Important: /path/in/udi must exist in UDI qsub -v UDI="centos:latest -v /u/sciteam/$user:/home"! /projects, /scratch, and Home folders are mapped between Blue Waters and UDI
https://github.com/nersc/shifter https://github.com/singularityware/singularity
-l gres=singularity Docker ssh qsub aprun User Login nodes MOM nodes Compute nodes
-l gres=singularity module load singularity Docker -l gres=singularity qsub singularity build IMAGE.simg docker://centos:latest singularity exec -H /u:/home IMAGE.simg -- app Login nodes MOM nodes application application application application application application application application Compute nodes
Directory mapping between BW and Singularity singularity exec -B /path/on/bw:/path/in/singularity And... /path/in/singularity does not have to exist. Note: /projects on BW is /mnt/abc/projects in Singularity /scratch on BW is /mnt/abc/scratch in Singularity
GPUs in Singularity singularity exec --nv... If you re using BW directories, make sure they are mapped into Singularity image Example: aprun -n 16 -N 8 -b -- singularity exec -B /dsl/opt/cray/nvidia/default/bin:/home/staff/mbelkin /home/staff/mbelkin/nvidia-smi
Thank you! 38