MAKING CONTAINERS EASIER WITH HPC CONTAINER MAKER. Scott McMillan September 2018

Size: px

Start display at page:

Download "MAKING CONTAINERS EASIER WITH HPC CONTAINER MAKER. Scott McMillan September 2018"

Peregrine Nicholson
5 years ago
Views:

1 MAKING CONTAINERS EASIER WITH HPC CONTAINER MAKER Scott McMillan September 2018

Innovate in Minutes, Not Weeks Get up and running quickly and reduce complexity Access from Anywhere

2 NVIDIA GPU CLOUD (NGC) Simple Access to Ready to-run, GPU-Accelerated Software Discover 35 GPU-Accelerated Containers Deep learning, HPC applications, NVIDIA HPC visualization tools, and partner applications Innovate in Minutes, Not Weeks Get up and running quickly and reduce complexity Access from Anywhere Containers run on supported cloud providers, NVIDIA DGX Systems, and PCs with NVIDIA Volta or Pascal architecture GPUs 2

3 HPC APP CONTAINERS ON NGC CANDLE CHROMA LATTICE MILC MICROBES PIConGPU BigDFT GAMESS GROMACS LAMMPS NAMD RELION PGI RAPID CONTAINER ADDITION RAPID USER ADOPTION 3

4 GET STARTED TODAY WITH NGC Sign Up and Access Containers for Free To learn more about all of the GPUaccelerated software from NGC, visit: nvidia.com/cloud To sign up or explore NGC, visit: ngc.nvidia.com 4

5 WHAT IF A CONTAINER IMAGE IS NOT AVAILABLE FROM NGC? 5

6 TYPICAL BARE METAL WORKFLOW Login to system (e.g., CentOS 7 with Mellanox OFED 3.4) $ module load PrgEnv/GCC+OpenMPI $ module load cuda/9.0 $ module load gcc $ module load openmpi/ What is the equivalent container workflow? Steps to build application Result: application binary suitable for that particular bare metal system 6

0-devel-centos7 $ module load PrgEnv/GCC+OpenMPI $ module load cuda/9.

7 CONTAINER WORKFLOW Login to system (e.g., CentOS 7 with Mellanox OFED 3.4) FROM nvidia/cuda:9.0-devel-centos7 $ module load PrgEnv/GCC+OpenMPI $ module load cuda/9.0 $ module load gcc $ module load openmpi/ Steps to build application Result: application binary suitable for that particular bare metal system 7

8 OPENMPI DOCKERFILE VARIANTS Real examples which one should you use? RUN apt-get update \ && apt-get install -y --no-install-recommends \ libopenmpi-dev \ openmpi-bin \ openmpi-common \ && rm -rf /var/lib/apt/lists/* ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/openmpi/lib RUN OPENMPI_VERSION=3.0.0 && \ wget -q -O - mpi.org/software/ompi/v3.0/downloads/openmpi- ${OPENMPI_VERSION}.tar.gz tar -xzf - && \ cd openmpi-${openmpi_version} && \./configure --enable-orterun-prefix-by-default --with-cuda -- with-verbs \ --prefix=/usr/local/mpi --disable-getpwuid && \ make -j"$(nproc)" install && \ cd.. && rm -rf openmpi-${openmpi_version} && \ echo "/usr/local/mpi/lib" >> /etc/ld.so.conf.d/openmpi.conf && ldconfig ENV PATH /usr/local/mpi/bin:$path COPY openmpi /usr/local/openmpi WORKDIR /usr/local/openmpi RUN /bin/bash -c "source /opt/pgi/license.txt && CC=pgcc CXX=pgc++ F77=pgf77 FC=pgf90./configure --with-cuda -- prefix=/usr/local/openmpi RUN /bin/bash -c "source /opt/pgi/license.txt && make all install" A B C RUN mkdir /logs RUN wget -nv && \ D tar -xzf openmpi tar.gz && \ cd openmpi-*&&./configure --with-cuda=/usr/local/cuda \ --enable-mpi-cxx --prefix=/usr 2>&1 tee /logs/openmpi_config && \ make -j 32 2>&1 tee /logs/openmpi_make && make install 2>&1 tee /logs/openmpi_install && cd /tmp \ && rm -rf openmpi-* WORKDIR /tmp ADD /tmp RUN tar -xzf openmpi tar.gz && \ cd openmpi-*&&./configure --with-cuda=/usr/local/cuda \ --enable-mpi-cxx --prefix=/usr && \ make -j 32 && make install && cd /tmp \ && rm -rf openmpi-* RUN wget -q -O - tar - xjf - && \ cd openmpi && \ CXX=pgc++ CC=pgcc FC=pgfortran F77=pgfortran./configure -- prefix=/usr/local/openmpi --with-cuda=/usr/local/cuda --with-verbs --disable-getpwuid && \ make -j4 install && \ rm -rf /openmpi E F

9 HPC CONTAINER MAKER Tool for creating HPC application Dockerfiles and Singularity recipe files You will still need to build the container images Makes it easier to create HPC application containers by encapsulating best practices into building blocks Open source (Apache 2.0) pip install hpccm 9

10 BUILDING BLOCKS TO CONTAINER RECIPES Stage0 += openmpi() hpccm Generate corresponding Dockerfile instructions for the HPCCM building block # OpenMPI version RUN yum install -y \ bzip2 file hwloc make openssh-clients perl tar wget && \ rm -rf /var/cache/yum/* RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp && \ mkdir -p /var/tmp && tar -x -f /var/tmp/openmpi tar.bz2 -C /var/tmp -j && \ cd /var/tmp/openmpi && CC=gcc CXX=g++ F77=gfortran F90=gfortran FC=gfortran./configure -- prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-by-default --with-cuda=/usr/local/cuda --with-verbs && \ make -j4 && \ make -j4 install && \ rm -rf /var/tmp/openmpi tar.bz2 /var/tmp/openmpi ENV LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH \ PATH=/usr/local/openmpi/bin:$PATH 11

EQUIVALENT HPC CONTAINER MAKER WORKFLOW Login to system (e.g., CentOS 7 with Mellanox OFED 3.4) Stage0 += baseimage(image='nvidia/cuda:9.0-devel-centos7') Stage0 += mlnx_ofed(version= 3.4-1.0.0.0 ) $ module load PrgEnv/GCC+OpenMPI $ module load cuda/9.

11 EQUIVALENT HPC CONTAINER MAKER WORKFLOW Login to system (e.g., CentOS 7 with Mellanox OFED 3.4) Stage0 += baseimage(image='nvidia/cuda:9.0-devel-centos7') Stage0 += mlnx_ofed(version= ) $ module load PrgEnv/GCC+OpenMPI $ module load cuda/9.0 $ module load gcc $ module load openmpi/ Steps to build application Result: application binary suitable for that particular bare metal system Stage0 += gnu() Stage0 += openmpi(version='1.10.7') Steps to build application Result: portable application container capable of running on any system 12

12 HIGHER LEVEL ABSTRACTION Building blocks to encapsulate best practices, avoid duplication, separation of concerns openmpi(check=false, # run make check? configure_opts=['--disable-getpwuid', ], # configure command line options cuda=true, # enable CUDA? directory='', # path to source in build context infiniband=true, # enable InfiniBand? ospackages=['bzip2', 'file', 'hwloc', ], # Linux distribution prerequisites prefix='/usr/local/openmpi', # install location toolchain=toolchain(), # compiler to use version='3.0.0') # version to download Examples: openmpi(prefix='/opt/openmpi', version=' ) openmpi(infiniband=false, toolchain=pgi.toolchain) Full building block documentation can be found in RECIPES.md 13

13 INCLUDED BUILDING BLOCKS As of version 18.8 CUDA is included via the base image, see Compilers GNU PGI Intel (BYOL) HPC libraries Charm++ FFTW, MKL, OpenBLAS CGNS, HDF5, NetCDF, PnetCDF Miscellaneous Boost CMake InfiniBand Mellanox OFED OFED (upstream) MPI Intel MPI MVAPICH2, MVAPICH2-GDR OpenMPI Package management packages (Linux distro aware), or apt_get yum 14

14 RECIPES INCLUDED WITH CONTAINER MAKER HPC Base Recipes: Ubuntu CentOS 7 GNU compilers PGI compilers OpenMPI MVAPICH2 2.3rc2 CUDA 9.0 FFTW HDF Mellanox OFED Python 2 and 3 15

15 BUILDING APPLICATION CONTAINER IMAGES WITH HPCCM Container template generator 1. Generate the desired container template starting from one of the HPC base recipes included with HPCCM $ hpccm --recipe recipes/hpcbase-gnu-openmpi.py --single-stage --format docker > Dockerfile $ hpccm --recipe recipes/hpcbase-gnu-openmpi.py --single-stage --format singularity > Singularity 2. Edit Dockerfile to add application build instructions 3. Build the container image $ docker build -t myapp -f Dockerfile. 2. Edit Singularity recipe file to add application build instructions 3. Build the container image $ sudo singularity build myapp.simg Singularity 16

16 BUILDING APPLICATION CONTAINER IMAGES WITH HPCCM HPC Base Images 1. Generate the desired container base image starting from one of the HPC base recipes included with HPCCM $ hpccm --recipe recipes/hpcbase-gnu-openmpi.py --single-stage --format docker > Dockerfile $ docker build -t hpcbase-gnu-openmpi -f Dockerfile. 2. Create Dockerfile referencing the HPC base image and containing the application build instructions FROM hpcbase-gnu-openmpi 3. Build the container image $ docker build -t myapp -f Dockerfile. 2. Create Singularity recipe file referencing the HPC base image and containing the application build instructions BootStrap: Docker From: hpcbase-gnu-openmpi 3. Build the container image $ sudo singularity build myapp.simg Singularity 17

17 BUILDING APPLICATION CONTAINER IMAGES WITH HPCCM Application recipe $ cat mpi-bandwidth.py # Setup GNU compilers, Mellanox OFED, and OpenMPI Stage0 += baseimage(image='nvidia/cuda:9.0-devel-centos7') Stage0 += gnu() Stage0 += mlnx_ofed(version=' ') Stage0 += openmpi(version='3.0.0') # Application build steps below # Using MPI Bandwidth from Lawrence Livermore National Laboratory (LLNL) as an example # 1. Copy source code into the container Stage0 += copy(src='mpi_bandwidth.c', dest='/tmp/mpi_bandwidth.c') # 2. Build the application Stage0 += shell(commands=['mkdir -p /workspace', 'mpicc -o /workspace/mpi_bandwidth /tmp/mpi_bandwidth.c ]) $ hpccm --recipe mpi-bandwidth.py --format 18

18 BUILDING APPLICATION CONTAINERS IMAGES WITH HPCCM Application recipes Dockerfile Singularity recipe file CUDA CentOS base BootStrap: docker image FROM nvidia/cuda:9.0-devel-centos7 From: nvidia/cuda:9.0-devel-centos7 %post. /.singularity.d/env/10-docker.sh # GNU compiler RUN yum install -y \ GNU compiler gcc \ # GNU gcc-c++ \ %post gcc-gfortran && \ yum install -y \ rm -rf /var/cache/yum/* gcc \ gcc-c++ \ gcc-gfortran # Mellanox OFED version rm -rf /var/cache/yum/* RUN yum install -y \ libnl \ libnl3 \ # Mellanox OFED version numactl-libs \ %post wget && \ yum install -y \ rm -rf /var/cache/yum/* libnl \ RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp Mellanox OFED libnl3 \ x86_64.tgz && \ numactl-libs \ mkdir -p /var/tmp && tar -x -f /var/tmp/mlnx_ofed_linux rhel7.2-x86_64.tgz -C /var/tmp -z && \ wget rpm --install /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibverbs-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2- rm -rf /var/cache/yum/* x86_64/rpms/libibverbs-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibverbs-utils-*.x86_64.rpm /var/tmp/mlnx_ofed_linux %post rhel7.2-x86_64/rpms/libibmad-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibmad-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp rhel7.2-x86_64/rpms/libibumad-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibumad-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux x86_64.tgz rhel7.2-x86_64/rpms/libmlx4-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libmlx5-*.x86_64.rpm && \ mkdir -p /var/tmp && tar -x -f /var/tmp/mlnx_ofed_linux rhel7.2-x86_64.tgz -C /var/tmp -z rm -rf /var/tmp/mlnx_ofed_linux rhel7.2-x86_64.tgz /var/tmp/mlnx_ofed_linux rhel7.2-x86_64 rpm --install /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibverbs-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2- x86_64/rpms/libibverbs-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibverbs-utils-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibmad-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibmad-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux # OpenMPI version rhel7.2-x86_64/rpms/libibumad-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libibumad-devel-*.x86_64.rpm /var/tmp/mlnx_ofed_linux RUN yum install -y \ rhel7.2-x86_64/rpms/libmlx4-*.x86_64.rpm /var/tmp/mlnx_ofed_linux rhel7.2-x86_64/rpms/libmlx5-*.x86_64.rpm bzip2 \ rm -rf /var/tmp/mlnx_ofed_linux rhel7.2-x86_64.tgz /var/tmp/mlnx_ofed_linux rhel7.2-x86_64 file \ hwloc \ make \ # OpenMPI version openssh-clients \ %post perl \ yum install -y \ tar \ bzip2 \ wget && \ OpenMPI file \ rm -rf /var/cache/yum/* hwloc \ RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp && \ make \ mkdir -p /var/tmp && tar -x -f /var/tmp/openmpi tar.bz2 -C /var/tmp -j && \ openssh-clients \ cd /var/tmp/openmpi &&./configure --prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-by-default --with-cuda --with-verbs && \ perl \ make -j4 && \ tar \ make -j4 install && \ wget rm -rf /var/tmp/openmpi tar.bz2 /var/tmp/openmpi rm -rf /var/cache/yum/* ENV LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH \ %post PATH=/usr/local/openmpi/bin:$PATH mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp mkdir -p /var/tmp && tar -x -f /var/tmp/openmpi tar.bz2 -C /var/tmp -j cd /var/tmp/openmpi &&./configure --prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-by-default --with-cuda --with-verbs COPY mpi_bandwidth.c /tmp/mpi_bandwidth.c make -j4 make -j4 install rm -rf /var/tmp/openmpi tar.bz2 /var/tmp/openmpi RUN mkdir -p /workspace && \ %environment mpicc -o /workspace/mpi_bandwidth /tmp/mpi_bandwidth.c export LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH export PATH=/usr/local/openmpi/bin:$PATH %post export LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH export PATH=/usr/local/openmpi/bin:$PATH MPI Bandwidth %files mpi_bandwidth.c /tmp/mpi_bandwidth.c %post mkdir -p /workspace mpicc -o /workspace/mpi_bandwidth /tmp/mpi_bandwidth.c 19

RECIPES INCLUDED WITH CONTAINER MAKER HPC Base Recipes: Ubuntu 16.

19 RECIPES INCLUDED WITH CONTAINER MAKER HPC Base Recipes: Ubuntu GNU compilers CentOS 7 PGI compilers Reference Recipes: OpenMPI MVAPICH2 2.3rc2 CUDA 9.0 FFTW HDF Mellanox OFED Python 2 and 3 MPI Bandwidth GROMACS MILC 20

20 CONTAINER SPECIFICATION FORMAT ABSTRACTION Single source to either Docker or Singularity HPC Container Maker Dockerfile Singularity recipe file baseimage(image='ubuntu:16.04') FROM ubuntu:16.04 Bootstrap: docker From: ubuntu:16.04 shell(commands=['a', 'b', 'c']) RUN a && \ b && \ c %post a b c copy(src='a', dest='b') COPY a b %files a b Recommendation: avoid using primitives when a building block is available 21

21 CONTAINER SPECIFICATION FORMAT ABSTRACTION Single source supporting either Ubuntu or CentOS/RHEL base images HPC Container Maker packages(ospackages=['gcc']) apt_get(ospackages=['g++']) Ubuntu base image baseimage(image='ubuntu:16.04') RUN apt-get update y && apt-get install y --no-install-recommends \ gcc && \ rm rf /var/lib/apt/lists/* RUN apt-get update y && apt-get install y --no-install-recommends \ g++ && \ rm rf /var/lib/apt/lists/* yum(ospackages=['gcc-c++'])!!! packages(apt=['g++'], yum=['gcc-c++']) RUN apt-get update y && apt-get install y --no-install-recommends \ g++ && \ rm rf /var/lib/apt/lists/* CentOS base image baseimage(image='centos:7') RUN yum install y \ gcc && \ rm -rf /var/cache/yum/*!!! RUN yum install y \ gcc-c++ && \ rm -rf /var/cache/yum/* RUN yum install y \ gcc-c++ && \ rm -rf /var/cache/yum/* Recommendation: always use the packages building block to install distro packages 22

22 USER ARGUMENTS Easy way to enable configurable recipes $ cat recipes/examples/userargs.py # Set the image tag based on the specified version (default to 9.1) cuda_version = USERARG.get('cuda', '9.1') image = 'nvidia/cuda:{}-devel-ubuntu16.04'.format(cuda_version) Stage0.baseimage(image) # Set the OpenMPI version based on the specified version (default to 3.0.0) ompi_version = USERARG.get('ompi', '3.0.0') ompi = openmpi(infiniband=false, version=ompi_version) Stage0 += ompi $ hpccm --recipe recipes/examples/userargs.py --userarg cuda=9.0 ompi=2.1.2 $ hpccm --recipe recipes/examples/userargs.py --userarg cuda=9.2 23

23 MULTISTAGE RECIPES Only supported by Docker $ cat recipes/examples/multistage.py # Devel stage base image Stage0.name = 'devel' Stage0.baseimage('nvidia/cuda:9.0-devel-ubuntu16.04') # Install compilers (upstream) g = gnu() Stage0 += g # Build FFTW using all default options f = fftw() Stage0 += f # Runtime stage base image Stage1.baseimage('nvidia/cuda:9.0-runtime-ubuntu16.04') # Compiler runtime (upstream) Stage1 += g.runtime() # Copy the FFTW build from the devel stage and setup the runtime environment. # The _from option is not necessary, but increases clarity. Stage1 += f.runtime(_from=stage0.name) 24

24 SUMMARY HPC Container Maker simplifies creating a container specification file Best practices used by default Building blocks included for many popular HPC components Flexibility and power of Python Supports Docker (and other frameworks that use Dockerfiles) and Singularity Open source: pip install hpccm 25

25 BACKUP 26

26 BIG PICTURE 27

27 BUILDING AN HPC APPLICATION IMAGE Analogous workflows for Singularity 1. Use the HPC base image as your starting point Base recipe Dockerfile Base image App Dockerfile 2. Generate a Dockerfile from the HPC base recipe Dockerfile and manually edit it to add the steps to build your application Base recipe Dockerfile App Dockerfile 3. Copy the HPC base recipe file and add your application build steps to the recipe Base recipe App recipe See 28

28 FULL PROGRAMMING LANGUAGE AVAILABLE Conditional branching, validation, etc. # get and validate precision VALID_PRECISION = [ single, double, mixed ] precision = USERARG.get( LAMMPS_PRECISION, single ) if precision not in VALID_PRECISION: raise ValueError( Invalid precision ) Stage0 += shell(commands=[ make f Makefile.linux.{}.format(precision), ]) $ hpccm --userarg LAMMPS_PRECISION=single Courtesy of Logan Herche 29

HPC IN CONTAINERS: WHY CONTAINERS, WHY HPC, WHY NVIDIA GTC 18, S8642, Monday March 26, 11am

HPC IN CONTAINERS: WHY CONTAINERS, WHY HPC, WHY NVIDIA GTC 18, S8642, Monday March 26, 11am CJ Newburn, HPC Architect, NVIDIA Compute SW Principal Engineer OUTLINE Motivation What NVIDIA is doing Collaborations