MAKING CONTAINERS EASIER WITH HPC CONTAINER MAKER Scott McMillan September 2018
NVIDIA GPU CLOUD (NGC) Simple Access to Ready to-run, GPU-Accelerated Software Discover 35 GPU-Accelerated Containers Deep learning, HPC applications, NVIDIA HPC visualization tools, and partner applications Innovate in Minutes, Not Weeks Get up and running quickly and reduce complexity Access from Anywhere Containers run on supported cloud providers, NVIDIA DGX Systems, and PCs with NVIDIA Volta or Pascal architecture GPUs 2
HPC APP CONTAINERS ON NGC CANDLE CHROMA LATTICE MILC MICROBES PIConGPU BigDFT GAMESS GROMACS LAMMPS NAMD RELION PGI RAPID CONTAINER ADDITION RAPID USER ADOPTION 3
GET STARTED TODAY WITH NGC Sign Up and Access Containers for Free To learn more about all of the GPUaccelerated software from NGC, visit: nvidia.com/cloud To sign up or explore NGC, visit: ngc.nvidia.com 4
WHAT IF A CONTAINER IMAGE IS NOT AVAILABLE FROM NGC? 5
TYPICAL BARE METAL WORKFLOW Login to system (e.g., CentOS 7 with Mellanox OFED 3.4) $ module load PrgEnv/GCC+OpenMPI $ module load cuda/9.0 $ module load gcc $ module load openmpi/1.10.7 What is the equivalent container workflow? Steps to build application Result: application binary suitable for that particular bare metal system 6
CONTAINER WORKFLOW Login to system (e.g., CentOS 7 with Mellanox OFED 3.4) FROM nvidia/cuda:9.0-devel-centos7 $ module load PrgEnv/GCC+OpenMPI $ module load cuda/9.0 $ module load gcc $ module load openmpi/1.10.7 Steps to build application Result: application binary suitable for that particular bare metal system 7
OPENMPI DOCKERFILE VARIANTS Real examples which one should you use? RUN apt-get update \ && apt-get install -y --no-install-recommends \ libopenmpi-dev \ openmpi-bin \ openmpi-common \ && rm -rf /var/lib/apt/lists/* ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/openmpi/lib RUN OPENMPI_VERSION=3.0.0 && \ wget -q -O - https://www.open- mpi.org/software/ompi/v3.0/downloads/openmpi- ${OPENMPI_VERSION}.tar.gz tar -xzf - && \ cd openmpi-${openmpi_version} && \./configure --enable-orterun-prefix-by-default --with-cuda -- with-verbs \ --prefix=/usr/local/mpi --disable-getpwuid && \ make -j"$(nproc)" install && \ cd.. && rm -rf openmpi-${openmpi_version} && \ echo "/usr/local/mpi/lib" >> /etc/ld.so.conf.d/openmpi.conf && ldconfig ENV PATH /usr/local/mpi/bin:$path COPY openmpi /usr/local/openmpi WORKDIR /usr/local/openmpi RUN /bin/bash -c "source /opt/pgi/license.txt && CC=pgcc CXX=pgc++ F77=pgf77 FC=pgf90./configure --with-cuda -- prefix=/usr/local/openmpi RUN /bin/bash -c "source /opt/pgi/license.txt && make all install" A B C RUN mkdir /logs RUN wget -nv https://www.openmpi.org/software/ompi/v1.10/downloads/openmpi-1.10.7.tar.gz && \ D tar -xzf openmpi-1.10.7.tar.gz && \ cd openmpi-*&&./configure --with-cuda=/usr/local/cuda \ --enable-mpi-cxx --prefix=/usr 2>&1 tee /logs/openmpi_config && \ make -j 32 2>&1 tee /logs/openmpi_make && make install 2>&1 tee /logs/openmpi_install && cd /tmp \ && rm -rf openmpi-* WORKDIR /tmp ADD http://www.openmpi.org//software/ompi/v1.10/downloads/openmpi-1.10.7.tar.gz /tmp RUN tar -xzf openmpi-1.10.7.tar.gz && \ cd openmpi-*&&./configure --with-cuda=/usr/local/cuda \ --enable-mpi-cxx --prefix=/usr && \ make -j 32 && make install && cd /tmp \ && rm -rf openmpi-* RUN wget -q -O - https://www.openmpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.bz2 tar - xjf - && \ cd openmpi-3.0.0 && \ CXX=pgc++ CC=pgcc FC=pgfortran F77=pgfortran./configure -- prefix=/usr/local/openmpi --with-cuda=/usr/local/cuda --with-verbs --disable-getpwuid && \ make -j4 install && \ rm -rf /openmpi-3.0.0 8 E F
HPC CONTAINER MAKER Tool for creating HPC application Dockerfiles and Singularity recipe files You will still need to build the container images Makes it easier to create HPC application containers by encapsulating best practices into building blocks Open source (Apache 2.0) https://github.com/nvidia/hpc-container-maker pip install hpccm https://devblogs.nvidia.com/making-containers-easier-with-hpc-container-maker/ 9
BUILDING BLOCKS TO CONTAINER RECIPES Stage0 += openmpi() hpccm Generate corresponding Dockerfile instructions for the HPCCM building block # OpenMPI version 3.0.0 RUN yum install -y \ bzip2 file hwloc make openssh-clients perl tar wget && \ rm -rf /var/cache/yum/* RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp https://www.openmpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.bz2 && \ mkdir -p /var/tmp && tar -x -f /var/tmp/openmpi-3.0.0.tar.bz2 -C /var/tmp -j && \ cd /var/tmp/openmpi-3.0.0 && CC=gcc CXX=g++ F77=gfortran F90=gfortran FC=gfortran./configure -- prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-by-default --with-cuda=/usr/local/cuda --with-verbs && \ make -j4 && \ make -j4 install && \ rm -rf /var/tmp/openmpi-3.0.0.tar.bz2 /var/tmp/openmpi-3.0.0 ENV LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH \ PATH=/usr/local/openmpi/bin:$PATH 11
EQUIVALENT HPC CONTAINER MAKER WORKFLOW Login to system (e.g., CentOS 7 with Mellanox OFED 3.4) Stage0 += baseimage(image='nvidia/cuda:9.0-devel-centos7') Stage0 += mlnx_ofed(version= 3.4-1.0.0.0 ) $ module load PrgEnv/GCC+OpenMPI $ module load cuda/9.0 $ module load gcc $ module load openmpi/1.10.7 Steps to build application Result: application binary suitable for that particular bare metal system Stage0 += gnu() Stage0 += openmpi(version='1.10.7') Steps to build application Result: portable application container capable of running on any system 12
HIGHER LEVEL ABSTRACTION Building blocks to encapsulate best practices, avoid duplication, separation of concerns openmpi(check=false, # run make check? configure_opts=['--disable-getpwuid', ], # configure command line options cuda=true, # enable CUDA? directory='', # path to source in build context infiniband=true, # enable InfiniBand? ospackages=['bzip2', 'file', 'hwloc', ], # Linux distribution prerequisites prefix='/usr/local/openmpi', # install location toolchain=toolchain(), # compiler to use version='3.0.0') # version to download Examples: openmpi(prefix='/opt/openmpi', version='1.10.7 ) openmpi(infiniband=false, toolchain=pgi.toolchain) Full building block documentation can be found in RECIPES.md 13
INCLUDED BUILDING BLOCKS As of version 18.8 CUDA is included via the base image, see https://hub.docker.com/r/nvidia/cuda/ Compilers GNU PGI Intel (BYOL) HPC libraries Charm++ FFTW, MKL, OpenBLAS CGNS, HDF5, NetCDF, PnetCDF Miscellaneous Boost CMake InfiniBand Mellanox OFED OFED (upstream) MPI Intel MPI MVAPICH2, MVAPICH2-GDR OpenMPI Package management packages (Linux distro aware), or apt_get yum 14
RECIPES INCLUDED WITH CONTAINER MAKER HPC Base Recipes: Ubuntu 16.04 CentOS 7 GNU compilers PGI compilers OpenMPI 3.0.0 MVAPICH2 2.3rc2 CUDA 9.0 FFTW 3.3.7 HDF5 1.10.1 Mellanox OFED 3.4-1.0.0.0 Python 2 and 3 15
BUILDING APPLICATION CONTAINER IMAGES WITH HPCCM Container template generator 1. Generate the desired container template starting from one of the HPC base recipes included with HPCCM $ hpccm --recipe recipes/hpcbase-gnu-openmpi.py --single-stage --format docker > Dockerfile $ hpccm --recipe recipes/hpcbase-gnu-openmpi.py --single-stage --format singularity > Singularity 2. Edit Dockerfile to add application build instructions 3. Build the container image $ docker build -t myapp -f Dockerfile. 2. Edit Singularity recipe file to add application build instructions 3. Build the container image $ sudo singularity build myapp.simg Singularity 16
BUILDING APPLICATION CONTAINER IMAGES WITH HPCCM HPC Base Images 1. Generate the desired container base image starting from one of the HPC base recipes included with HPCCM $ hpccm --recipe recipes/hpcbase-gnu-openmpi.py --single-stage --format docker > Dockerfile $ docker build -t hpcbase-gnu-openmpi -f Dockerfile. 2. Create Dockerfile referencing the HPC base image and containing the application build instructions FROM hpcbase-gnu-openmpi 3. Build the container image $ docker build -t myapp -f Dockerfile. 2. Create Singularity recipe file referencing the HPC base image and containing the application build instructions BootStrap: Docker From: hpcbase-gnu-openmpi 3. Build the container image $ sudo singularity build myapp.simg Singularity 17
BUILDING APPLICATION CONTAINER IMAGES WITH HPCCM Application recipe $ cat mpi-bandwidth.py # Setup GNU compilers, Mellanox OFED, and OpenMPI Stage0 += baseimage(image='nvidia/cuda:9.0-devel-centos7') Stage0 += gnu() Stage0 += mlnx_ofed(version='3.4-1.0.0.0') Stage0 += openmpi(version='3.0.0') # Application build steps below # Using MPI Bandwidth from Lawrence Livermore National Laboratory (LLNL) as an example # 1. Copy source code into the container Stage0 += copy(src='mpi_bandwidth.c', dest='/tmp/mpi_bandwidth.c') # 2. Build the application Stage0 += shell(commands=['mkdir -p /workspace', 'mpicc -o /workspace/mpi_bandwidth /tmp/mpi_bandwidth.c ]) $ hpccm --recipe mpi-bandwidth.py --format 18
BUILDING APPLICATION CONTAINERS IMAGES WITH HPCCM Application recipes Dockerfile Singularity recipe file CUDA CentOS base BootStrap: docker image FROM nvidia/cuda:9.0-devel-centos7 From: nvidia/cuda:9.0-devel-centos7 %post. /.singularity.d/env/10-docker.sh # GNU compiler RUN yum install -y \ GNU compiler gcc \ # GNU gcc-c++ \ %post gcc-gfortran && \ yum install -y \ rm -rf /var/cache/yum/* gcc \ gcc-c++ \ gcc-gfortran # Mellanox OFED version 3.4-1.0.0.0 rm -rf /var/cache/yum/* RUN yum install -y \ libnl \ libnl3 \ # Mellanox OFED version 3.4-1.0.0.0 numactl-libs \ %post wget && \ yum install -y \ rm -rf /var/cache/yum/* libnl \ RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp http://content.mellanox.com/ofed/mlnx_ofed-3.4-1.0.0.0/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2- Mellanox OFED libnl3 \ x86_64.tgz && \ numactl-libs \ mkdir -p /var/tmp && tar -x -f /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64.tgz -C /var/tmp -z && \ wget rpm --install /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libibverbs-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2- rm -rf /var/cache/yum/* x86_64/rpms/libibverbs-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libibverbs-utils-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0- %post rhel7.2-x86_64/rpms/libibmad-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libibmad-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0- mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp http://content.mellanox.com/ofed/mlnx_ofed-3.4-1.0.0.0/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2- rhel7.2-x86_64/rpms/libibumad-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libibumad-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0- x86_64.tgz rhel7.2-x86_64/rpms/libmlx4-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libmlx5-*.x86_64.rpm && \ mkdir -p /var/tmp && tar -x -f /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64.tgz -C /var/tmp -z rm -rf /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64.tgz /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64 rpm --install /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libibverbs-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2- x86_64/rpms/libibverbs-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libibverbs-utils-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0- rhel7.2-x86_64/rpms/libibmad-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libibmad-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0- # OpenMPI version 3.0.0 rhel7.2-x86_64/rpms/libibumad-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libibumad-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0- RUN yum install -y \ rhel7.2-x86_64/rpms/libmlx4-*.x86_64.rpm /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64/rpms/libmlx5-*.x86_64.rpm bzip2 \ rm -rf /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64.tgz /var/tmp/mlnx_ofed_linux-3.4-1.0.0.0-rhel7.2-x86_64 file \ hwloc \ make \ # OpenMPI version 3.0.0 openssh-clients \ %post perl \ yum install -y \ tar \ bzip2 \ wget && \ OpenMPI file \ rm -rf /var/cache/yum/* hwloc \ RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.bz2 && \ make \ mkdir -p /var/tmp && tar -x -f /var/tmp/openmpi-3.0.0.tar.bz2 -C /var/tmp -j && \ openssh-clients \ cd /var/tmp/openmpi-3.0.0 &&./configure --prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-by-default --with-cuda --with-verbs && \ perl \ make -j4 && \ tar \ make -j4 install && \ wget rm -rf /var/tmp/openmpi-3.0.0.tar.bz2 /var/tmp/openmpi-3.0.0 rm -rf /var/cache/yum/* ENV LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH \ %post PATH=/usr/local/openmpi/bin:$PATH mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.bz2 mkdir -p /var/tmp && tar -x -f /var/tmp/openmpi-3.0.0.tar.bz2 -C /var/tmp -j cd /var/tmp/openmpi-3.0.0 &&./configure --prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-by-default --with-cuda --with-verbs COPY mpi_bandwidth.c /tmp/mpi_bandwidth.c make -j4 make -j4 install rm -rf /var/tmp/openmpi-3.0.0.tar.bz2 /var/tmp/openmpi-3.0.0 RUN mkdir -p /workspace && \ %environment mpicc -o /workspace/mpi_bandwidth /tmp/mpi_bandwidth.c export LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH export PATH=/usr/local/openmpi/bin:$PATH %post export LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH export PATH=/usr/local/openmpi/bin:$PATH MPI Bandwidth %files mpi_bandwidth.c /tmp/mpi_bandwidth.c %post mkdir -p /workspace mpicc -o /workspace/mpi_bandwidth /tmp/mpi_bandwidth.c 19
RECIPES INCLUDED WITH CONTAINER MAKER HPC Base Recipes: Ubuntu 16.04 GNU compilers CentOS 7 PGI compilers Reference Recipes: OpenMPI 3.0.0 MVAPICH2 2.3rc2 CUDA 9.0 FFTW 3.3.7 HDF5 1.10.1 Mellanox OFED 3.4-1.0.0.0 Python 2 and 3 MPI Bandwidth GROMACS MILC 20
CONTAINER SPECIFICATION FORMAT ABSTRACTION Single source to either Docker or Singularity HPC Container Maker Dockerfile Singularity recipe file baseimage(image='ubuntu:16.04') FROM ubuntu:16.04 Bootstrap: docker From: ubuntu:16.04 shell(commands=['a', 'b', 'c']) RUN a && \ b && \ c %post a b c copy(src='a', dest='b') COPY a b %files a b Recommendation: avoid using primitives when a building block is available 21
CONTAINER SPECIFICATION FORMAT ABSTRACTION Single source supporting either Ubuntu or CentOS/RHEL base images HPC Container Maker packages(ospackages=['gcc']) apt_get(ospackages=['g++']) Ubuntu base image baseimage(image='ubuntu:16.04') RUN apt-get update y && apt-get install y --no-install-recommends \ gcc && \ rm rf /var/lib/apt/lists/* RUN apt-get update y && apt-get install y --no-install-recommends \ g++ && \ rm rf /var/lib/apt/lists/* yum(ospackages=['gcc-c++'])!!! packages(apt=['g++'], yum=['gcc-c++']) RUN apt-get update y && apt-get install y --no-install-recommends \ g++ && \ rm rf /var/lib/apt/lists/* CentOS base image baseimage(image='centos:7') RUN yum install y \ gcc && \ rm -rf /var/cache/yum/*!!! RUN yum install y \ gcc-c++ && \ rm -rf /var/cache/yum/* RUN yum install y \ gcc-c++ && \ rm -rf /var/cache/yum/* Recommendation: always use the packages building block to install distro packages 22
USER ARGUMENTS Easy way to enable configurable recipes $ cat recipes/examples/userargs.py # Set the image tag based on the specified version (default to 9.1) cuda_version = USERARG.get('cuda', '9.1') image = 'nvidia/cuda:{}-devel-ubuntu16.04'.format(cuda_version) Stage0.baseimage(image) # Set the OpenMPI version based on the specified version (default to 3.0.0) ompi_version = USERARG.get('ompi', '3.0.0') ompi = openmpi(infiniband=false, version=ompi_version) Stage0 += ompi $ hpccm --recipe recipes/examples/userargs.py --userarg cuda=9.0 ompi=2.1.2 $ hpccm --recipe recipes/examples/userargs.py --userarg cuda=9.2 23
MULTISTAGE RECIPES Only supported by Docker $ cat recipes/examples/multistage.py # Devel stage base image Stage0.name = 'devel' Stage0.baseimage('nvidia/cuda:9.0-devel-ubuntu16.04') # Install compilers (upstream) g = gnu() Stage0 += g # Build FFTW using all default options f = fftw() Stage0 += f # Runtime stage base image Stage1.baseimage('nvidia/cuda:9.0-runtime-ubuntu16.04') # Compiler runtime (upstream) Stage1 += g.runtime() # Copy the FFTW build from the devel stage and setup the runtime environment. # The _from option is not necessary, but increases clarity. Stage1 += f.runtime(_from=stage0.name) 24
SUMMARY HPC Container Maker simplifies creating a container specification file Best practices used by default Building blocks included for many popular HPC components Flexibility and power of Python Supports Docker (and other frameworks that use Dockerfiles) and Singularity Open source: https://github.com/nvidia/hpc-container-maker pip install hpccm 25
BACKUP 26
BIG PICTURE 27
BUILDING AN HPC APPLICATION IMAGE Analogous workflows for Singularity 1. Use the HPC base image as your starting point Base recipe Dockerfile Base image App Dockerfile 2. Generate a Dockerfile from the HPC base recipe Dockerfile and manually edit it to add the steps to build your application Base recipe Dockerfile App Dockerfile 3. Copy the HPC base recipe file and add your application build steps to the recipe Base recipe App recipe See https://devblogs.nvidia.com/making-containers-easier-with-hpc-container-maker/ 28
FULL PROGRAMMING LANGUAGE AVAILABLE Conditional branching, validation, etc. # get and validate precision VALID_PRECISION = [ single, double, mixed ] precision = USERARG.get( LAMMPS_PRECISION, single ) if precision not in VALID_PRECISION: raise ValueError( Invalid precision ) Stage0 += shell(commands=[ make f Makefile.linux.{}.format(precision), ]) $ hpccm --userarg LAMMPS_PRECISION=single Courtesy of Logan Herche 29