XC Series Shifter User Guide (CLE 6.0.UP02) S-2571
Contents Contents 1 About the XC Series Shifter User Guide...3 2 Shifter System Introduction...6 3 Download and Convert the Docker Image...7 4 Submit Jobs to Use a UDI...9 5 Run Shifter in Interactive Mode Without a Batch Reservation...11 6 Test the Shifter Configuration for sitefs Mount...12 2
About the XC Series Shifter User Guide 1 About the XC Series Shifter User Guide The Cray XC Series Shifter User Guide includes information for using the Shifter system. XC Series Shifter User Guide S-2571 CLE 6.0 UP02 This version includes updated procedures to support Cray software release CLE 6.0 UP02, released November 3, 2016 and supports the following new features: The shifterimg command replaces the getdockerimage command used in CLE 5.2 UP04 The Docker daemon is replaced by the Shifter Image Gateway. Scope and Audience This publication is written for Cray customers using the Shifter system. Feedback Visit the Cray Publications Portal at http://pubs.cray.com and make comments online using the Contact Us button in the upper-right corner or Email pubs@cray.com. Your comments are important to us and we will respond within 24 hours. Command Prompt Conventions hostname in command prompts Hostnames in command prompts indicate where the command must be run. hostname# Run the command on this hostname. smw# cmc# boot# sdb# login# smw1# smw2# Run the command on the SMW. Run the command on the CMC. Run the command on the boot node. Run the command on the SDB node. Run the command on any login node. For a system configured with the SMW failover feature there are two SMWs one in an active role and the other in a passive role. The 3
About the XC Series Shifter User Guide SMW that is active at the start of a procedure is smw1. The SMW that is passive is smw2. smwactive# smwpassive# In some scenarios, the active SMW is smw1 at the start of a procedure then the procedure requires a failover to the other SMW. In this case, the documentation will continue to refer to the formerly active SMW as smw1, even though smw2 is now the active SMW. If further clarification is needed in a procedure, the active SMW will be called smwactive and the passive SMW will be called smwpassive. account name in command prompts The account that must run the command is also indicated in the prompt. smw# cmc# boot# sdb# login# hostname# The root or super-user account always has the # character at the end of the prompt. crayadm@smw> crayadm@login> user@hostname> Any non-root account is indicated with account@hostname. A user account that is neither root nor crayadm is referred to as user. command prompt inside chroot directory path in command prompt If the chroot command is used, the prompt changes to indicate that it is inside a chroot'd environment on the hostname. smw# chroot /path/to/chroot chroot-smw# Sometimes the current path can be so long that including it in the prompt does not add clarity to the command example. Most of the time, the command can be executed from any directory. When it matters which directory the command is invoked within, the cd command is used to change into the directory, and the directory is referenced with a period (.) to indicate the current directory. For example, here are actual prompts as they appear on the system: smw:~ # cd /etc smw:/etc# cd /var/tmp smw:/var/tmp# ls./file smw:/var/tmp# su - crayadm crayadm@smw:~> cd /usr/bin crayadm@smw:/usr/bin>./command And here are the same prompts as they would appear in examples in this publication: smw# cd /etc smw# cd /var/tmp smw# ls./file 4
About the XC Series Shifter User Guide smw# su - crayadm crayadm@smw> cd /usr/bin crayadm@smw>./command Typographic Conventions Monospace Monospaced Bold Oblique or Italics Indicates program code, reserved words, library functions, command-line prompts, screen output, file/path names, and other software constructs. Indicates commands that must be entered on a command line or in response to an interactive prompt. Indicates user-supplied values in commands or syntax definitions. Proportional Bold Indicates a GUI Window, GUI element, cascading menu (Ctrl Alt Delete), or key strokes (press Enter). \ (backslash) At the end of a command line, indicates the Linux shell line continuation character (lines joined by a backslash are parsed as a single line). Trademarks The following are trademarks of Cray Inc. and are registered in the United States and other countries: CRAY and design, SONEXION, URIKA, and YARCDATA. The following are trademarks of Cray Inc.: APPRENTICE2, CHAPEL, CLUSTER CONNECT, CRAYDOC, CRAYPAT, CRAYPORT, DATAWARP, ECOPHLEX, LIBSCI, NODEKARE. The following system family marks, and associated model number marks, are trademarks of Cray Inc.: CS, CX, XC, XE, XK, XMT, and XT. The registered trademark LINUX is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Other trademarks used in this document are the property of their respective owners. 5
Shifter System Introduction 2 Shifter System Introduction The Shifter system allows users to run commands in a user defined image (UDI). A UDI has many of the features of a Linux container. Some features (e.g., separate pid namespace) are missing because the UDI is intended to be used on a compute node where only one user's apps run at a time. A UDI is created from a Docker image. A user may download an image that has been exported by another site from the Docker Hub. This image can then be unpacked and then packaged into a UDI for the local user to use. Downloading and packaging is done with the shifterimg command. Once a user has a UDI to use, the user can submit a batch job to use the image. The batch job can request use of the image in one of two ways: 1. When the job is submitted by specifying an environment variable that indicates what UDI should be used, or 2. By submitting a job using the shifter command inside the batch job. Both of these methods are described in more detail in this User Guide. 6
Download and Convert the Docker Image 3 Download and Convert the Docker Image An image must be downloaded and converted before it can be used. This is done via the shifterimg command. Docker Image Creation Limitations and Considerations The importation of Docker images that are not on DockerHub or other reachable private registries requires either root access or inclusion in the Docker group. Only Docker images downloaded from Docker registries can be run via Shifter on Cray platforms. Cray currently does not support creation of Docker images on Cray platforms. Local Docker registries are supported by this release of Shifter, but they must be specifically included in the configuration of the Shifter image gateway. Contact a system administrator if you need access to another registry to be added. Docker registries that require authentication to access images are not supported by this release of Shifter. To delete an image, contact a system administrator. shifterimg The shifterimg command allows a user to list available images on the local system, look up a particular image, or pull (download and convert) a particular image. The command syntaxes are as follows: user@login> module load shifter user@login> shifterimg images This will display the contents of the Shifter image gateway. user@login> shifterimg lookup <image>:<tag> This will look up the specified image in the local image cache. The lookup command will not search the remote Docker hub. user@login> shifterimg -v lookup docker:<image>:<tag> This will show Shifter image meta data. user@login> shifterimg pull <image>:<tag> This will download and convert the specified image. The local image cache is searched first and then the Docker Hub (hub.docker.com). This form is used internally by the Shifter prologue and the shifter command. If the image hasn t already been downloaded, it is downloaded. If the image hasn t already been converted, it is converted for use. When the image has been downloaded and converted, its ID is printed. This ID is used internally to locate a converted image and to prepare it for use as a UDI. 7
Download and Convert the Docker Image shifterimg expire Command By default, only the root user is able to use the shifterimg expire command. shifterimg expire will mark an image for removal. Once no Shifter user is using that image, the Shifter Image Gateway will remove the UDI. user@login> shifterimg lookup <image>:<tag> 463ff6be4238c14f5b88898f17b47a9cf494f9a9be7b6170c3e852568d2b0432 user@login> shifterimg expire <image>:<tag> user@login> shifterimg lookup <image>:<tag> user@login> echo $? 1 8
Submit Jobs to Use a UDI 4 Submit Jobs to Use a UDI To initialize the UDI during the batch reservation, users must specify the requested UDI to qsub, salloc, or sbatch. Users may also specify the requested image to the shifter command in a batch job. If the shifter_slurm.so plugin has been configured, salloc and sbatch will accept the --image and --volume options directly. The UDI is specified on the qsub command line, as shown in the following examples (please note users must have the shifter module loaded before using any shifter command): Example 1: user@torque-mom> module load shifter user@torque-mom> qsub -I -v SHIFTER=docker:ubuntu:latest user@torque-mom> aprun -b shifter cat /etc/os-release Example 2: user@torque-mom> qsub -I -v SHIFTER_IMAGE=ubuntu:latest,SHIFTER_IMAGETYPE=docker user@torque-mom> aprun -b shifter cat /etc/os-release Example 3: user@torque-mom> qsub -I user@torque-mom> aprun -b shifter --image=docker:ubuntu:latest cat /etc/os-release The SHIFTER_VOLUME environment variable can also be used in place of the --volume option. The -b option to aprun is important. The -b option tells aprun not to transfer the Shifter binary from the mom node to the compute node. If no -b option is used, the transfer will remove the setuid bit required of Shifter and the command will not function. No -b option fails with the following: user@torque-mom> aprun -L 170 shifter --image ubuntu:latest cat /etc/os-release FAILED to find requested image. Application 582219 exit codes: 1 Application 582219 resources: utime ~0s, stime ~0s, Rss ~4480, inblocks ~0, outblocks ~0 The aprun command may be used multiple times and each run will get the same UDI. The aprun commands may be run concurrently or serially, one after another. A different UDI may also be run even if it was not specified at batch reservation time. Shifter will check if the UDI specified with --image is already present on the node. If it is not, Shifter will mount the UDI in a private namespace, as shown in the following example: user@torque-mom> aprun -b shifter --image docker:ubuntu:14.04 cat /etc/os-release NAME="Ubuntu" 9
Submit Jobs to Use a UDI VERSION="14.04.4 LTS, Trusty Tahr" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 14.04.4 LTS" VERSION_ID="14.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" To see that Shifter is mounting the UDI on the node at batch reservation time, first specify a UDI at batch reservation time: user@torque-mom> qsub -I -v SHIFTER=docker:ubuntu:16.04 qsub: waiting for job 90.kachina to start qsub: job 90.kachina ready Initializing udiroot, please wait. Retrieving Docker Image... Next, run Shifter with a different UDI: user@torque-mom> aprun -b shifter --image docker:ubuntu:14.04 sleep 30 Application 628 resources: utime ~0s, stime ~1s, Rss ~4536, inblocks ~2178, outblocks ~0 Finally, check the loop devices mounted on the compute node using losetup: user@torque-mom> ssh root@nid00015 nid00015:~ # shifterimg lookup docker:ubuntu:16.04 594b6e305389ce5151e289d96b1e0e1df3c86d1bd8d208c997ecbcdfd6e320a0 nid00015:~ # shifterimg lookup docker:ubuntu:14.04 9bc9537638433df5e03e4327bb57c9aa9f1372f9985928d7562a857e242b377d nid00015:~ # losetup -a /dev/loop0: [0027]:118396 (/var/opt/cray/imps-distribution/squash/store/p0/.cache/ squashfs) /dev/loop1: [0027]:152582 (/var/opt/cray/imps-distribution/squash/store/ global/.cache/squashfs) /dev/loop2: [0055]:31514062 (/home/users/alanm/shifter/kachina/udi/ 594b6e305389ce5151e289d96b1e0e1df3c86d1bd8d208c997ecbcdfd6e320a0.squashfs) /dev/loop3: [0055]:31514066 (/home/users/alanm/shifter/kachina/udi/ 9bc9537638433df5e03e4327bb57c9aa9f1372f9985928d7562a857e242b377d.squashfs) 10
Run Shifter in Interactive Mode Without a Batch Reservation 5 Run Shifter in Interactive Mode Without a Batch Reservation Shifter can be run in interactive mode without a batch reservation. To do this, Shifter will execute the command from within the container environment. Run cat /etc/os-release inside the Ubuntu 16.04 container, as shown in the following example: user@torque-mom> aprun -b shifter --image docker:ubuntu:16.04 cat /etc/os-release NAME="Ubuntu" VERSION="16.04 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" UBUNTU_CODENAME=xenial Application 102536 resources: utime ~0s, stime ~0s, Rss ~4372, inblocks ~2248, outblocks ~0 11
Test the Shifter Configuration for sitefs Mount 6 Test the Shifter Configuration for sitefs Mount If /etc/shifter/udiroot.conf is configured with an appropriate sitefs option for users' home directories, users are able to access their home directory, as shown in the following example: user@torque-mom> cat /etc/shifter/udiroot.conf grep sitefs grep -v "#" sitefs=/home:/home user@torque-mom> echo "test sitefs mount" > shifter_home_file user@torque-mom> aprun -L 130 -b shifter --image docker:ubuntu:16.04 cat ~/ shifter_home_file test sitefs mount Application 102539 resources: utime ~0s, stime ~0s, Rss ~4372, inblocks ~2152, outblocks ~0 User bind mounts inside the container environment. The --volume option will let users bind mount files and directories outside the container, into the container: user@torque-mom> mkdir ~/user_volume user@torque-mom> touch ~/user_volume/user_file user@torque-mom> aprun -L 130 -b shifter --image docker:ubuntu:16.04 --volume ~/ user_volume:/mnt ls /mnt user_file Application 102543 resources: utime ~0s, stime ~0s, Rss ~4372, inblocks ~2782, outblocks ~0 12