PBS Pro with Docker Integration

Transcription

1 PBS Pro with Docker Integration Arun Grover Technical Manager 31 August 2015

2 Table of Contents 1. Why Integrate Docker and PBS? How PBS Runs Jobs Using Docker Steps to Integrate PBS and Docker Allowing Multi- vnode Jobs to Spawn Tasks on Sister Vnodes Making Data Available to Jobs Requirements Sample Hook Scripts Job_launch.py (execjob_launch hook) Docker_shutdown.py (execjob_end hook) Usage References For more information, contact Altair at: pbssales@altair.com Copyright Altair Engineering, Inc. All rights reserved. PBS, PBS Works, PBS Professional, PBS Analytics, and e-compute are trademarks of Altair Engineering, Inc. and are protected under U.S. and international laws and treaties. All other marks are the property of their respective owners. PBS Pro with Docker Integration 2

3 1. Why Integrate Docker and PBS? PBS Professional 1 PBS Professional is a fast and powerful workload manager designed to improve productivity, optimize utilization & efficiency, and simplify administration for HPC clusters, clouds and super- computers. PBS Professional automates job scheduling, management, monitoring and reporting. Docker 2 Docker containers wrap up a piece of software in a com- plete file system that con- tains everything it needs to run: code, runtime environ- ment, system tools, system libraries anything you can install on a server. This guarantees that the software will always run the same, regardless of the envi- ronment it is running in. Docker containers protect jobs from the effects of other jobs by controlling the environment; PBS Professional works to use resources optimally, and in order to do so, it may run more than one job on a single host. Each job often has spe- cific environmental needs; for example, some jobs may re- quire that specific versions of third- party libraries are in- stalled in a particular location. However, when applications run on a host directly, they can alter the environment in which they are executing. This can render the environment unsuitable for other jobs or applications. Maintaining a pris- tine environment on the host can therefore generate a lot of administrative overhead. You can avoid this by running jobs or applications inside vir- tual machines, where each job or application gets a dedicat- ed environment to run in, but running jobs in a completely virtualized environment can pose a scaling issue for the host and cause a loss in performance for the application, which may not be acceptable in your HPC environment. A Docker container running on the host OS is much lighter- weight than a virtual machine. It also has the inherent ad- vantage of sharing the same OS libraries and binaries across containers. The workload manager s task is to ensure that a job does not use more than the amount of resources it has requested, and Docker containers make this easier by con- straining or containing the resources that a job can use. Docker containers are built on top of cgroups and LxC and use them to constrain processes. There is one downside to integrating PBS and Docker. Cur- rently, Docker containers present a security problem. For in- stance, if a user were to have root privileges inside a Docker container, there are chances that this user could tamper with the host- mounted filesystem, even though the user has no privileges to do so on the host machine. To sum up, using PBS Professional integrated with Docker, you can run jobs in containers at scale and blend them with other workloads to support heterogeneous applications and environments, maximizing performance and meeting job re- quirements. PBS Pro with Docker Integration 3

4 2. How PBS Runs Jobs Using Docker PBS starts a separate Docker container to run each job; the scope of this container ends with the job. PBS provides the exact same environment to a job in a container that it provides to any other job when staging files in and out, managing a job s output and error files, and exporting envi- ronment variables. When PBS starts a Docker container, it configures the container so that the job cannot use more resources than it explicitly requested. For multi- vnode jobs, PBS configures any Docker containers created on sister MoMs to be net- work- linked to the container running on the mother superior, allowing communication between job tasks on separate execution hosts. When a job ends, PBS manages shutdown of the Docker containers it started for that particular job. 3. Steps to Integrate PBS and Docker 1. Set the environment variable DOCKER_IMAGE to the name of the Docker image in the local repository that PBS will start for each job. 2. Write a queuejob hook that includes either - v DOCKER_IMAGE or - V so that this environ- ment variable is included with each job. 3. Create an execjob_launch hook that launches a Docker container and sets up each job s envi- ronment. We give an example hook in 7.1, Job_launch.py (execjob_launch hook) but we explain what it s doing here. The hook does the following: a. Search the job s Variable_List attribute for the environment variable named DOCK- ER_IMAGE. b. Search the job s Resource_List attribute for ncpus and mem, and pass these resource values to the Docker container as limits on the number of CPUs and amount of memory. c. Collect all of the job s environment variables, both from the job submitter and MoM, in- to a list, and pass the list to the Docker container. d. Mount the following directories in the Docker container: PBS_HOME, PBS_EXEC, the job submitter s home directory, and directories where files are to be staged in and out. This command has the following form: "docker run - v <src dir>:<target dir>" PBS Pro with Docker Integration 4

5 e. Start the Docker container using the image name, resource restrictions, environment list and all mounted directories. Use the job ID as the name of the container. This gives each container a unique name, and we can use that name when we stop the container. f. In order to allow a multi- vnode job script running inside a Docker container to distribute tasks to other vnodes, the container needs to be able to communicate with the MoM running on its host. You can do this using the host network stack inside the container via the "- - net=host" option. We explain further in section 4. g. Before starting the container, the hook sets "docker exec" as the program name to be executed, and the actual job script/executable becomes one of the program arguments. 4. Create an execjob_end hook to strip down the Docker container launched by execjob_launch hook. HOST B HOST A execjob_launch create container Docker container Launch job 1.HOSTA PBS Server PBS MoM delete container execjob_end 4. Allowing Multi- vnode Jobs to Spawn Tasks on Sister Vnodes When you use PBS with Docker containers, a multi- vnode job runs each task on each vnode in- side a container. Each task may need to communicate with the others, so you make containers on sister MoMs network- linked to those on the mother superior. PBS Pro with Docker Integration 5

6 In order to distribute tasks to other vnodes, the TM API connects to the MoM running on its lo- cal host and asks the MoM to distribute tasks to sister MoMs. By default, for a job running in- side a Docker container, TM APIs are not able to find a MoM running on the local host because the Docker container runs its own network stack and the MoM runs on the host machine. To solve this issue, modify the network stack of the container so that the TM API can connect to the MoM running on the host machine. Do this by making container use the host machine's network stack via the "- - net=host" docker run option. (This is included in the sample hook in 7.1.) 1 docker run - it - d - - name <name of container> - - net=host <image name> /bin/bash 5. Making Data Available to Jobs When PBS is integrated with Docker, you can make data files available to jobs running in con- tainers. Whether your job submitter has the data directory mounted or is going to use staging, you can use an execjob_launch hook to mount a pre- defined data directory (in the user s home directo- ry on the execution host) at a mount point in the container, and have job submitters stage files into and out of this pre- defined directory. (We show a diagram for staging, but if your job sub- mitter mounts the data directory, they simply mount it where it would have been staged.) We use this approach below in line 65 of the sample hook shown in section 7.1. In the sample hook, we mount the user s home directory as a data volume in the container: 1 docker run - it - d - - name <name of container> - v <path on host>:<path in container> <image name> /bin/bash Data Host Execution Host User s home directory Docker container Data directory Staging A: Staged- in data A mount at B B: Mount point for A There are some other ways to make data available to jobs. Here are two other options: PBS stages from data storage into a directory in the user s home on the execution host, then Docker copies the files from there into the container using docker cp. You can mount the data storage directories at a mount point in the container. PBS Pro with Docker Integration 6

7 6. Requirements The job submitter must meet the following requirements: Be part of the Docker group. This provides the rights to execute inside the container. Have an account in the OS instance that is launched inside the Docker container. The Docker software installed on the execution host should have the necessary OS instance present in its local repository so that when we create a container it is able to find that OS instance. The re- pository must be locally present on the execution host. 7. Sample Hook Scripts 7.1. Job_launch.py (execjob_launch hook) 1 import os 2 import pbs 3 import copy 4 e = pbs.event() 5 j = e.job 6 args = copy.deepcopy(e.argv) 7 pbs.logmsg(pbs.log_debug, "args are %s " % e.progname) 8 prog = e.progname 9 hdir = "" 10 edir = "" 11 def add_resource_restriction (call): 12 n = j.resource_list["ncpus"] 13 m = j.resource_list["mem"] 14 if n: 15 call += " - c " + str(n) 16 if m: 17 call += " - m " + str(m) 18 return call 19 def add_env (call): 20 for i in e.env: 21 call += " - e " + str(i)+"="+e.env[i] 22 return call 23 def get_pbs_dirs (): 24 global hdir 25 global edir 26 #get PBS_HOME & PBS_EXEC directories 27 hdir = pbs.pbs_conf['pbs_home'] 28 edir = pbs.pbs_conf['pbs_exec'] 29 return 30 def create_container (): 31 global hdir 32 global edir 33 is_ms = j.in_ms_mom() 34 docker = "" 35 call = "" 36 search_container = "docker ps grep "+str(j.id)+" grep - v grep" PBS Pro with Docker Integration 7

8 37 docker = os.popen(search_container).read() 38 jid = j.id 39 pbs.logmsg(pbs.log_debug, "Docker container: %s" % docker) 40 if docker: 41 pbs.logmsg(pbs.log_debug, "Docker container found!") 42 else: 43 v = str(j.variable_list) 44 vl = v.split(",") 45 image = "DOCKER_IMAGE=" 46 os_image = "None" 47 #get image name 48 for i in vl: 49 pbs.logmsg(pbs.log_debug, "Variable is: %s" % i) 50 if i.startswith(image): 51 name = i.split("=",1) 52 os_image = name[1] 53 if "None" in os_image: 54 # No need to run any Docker instance... just bail out! 55 e.accept() 56 pbs.logmsg(pbs.log_debug, "Image is: %s" % os_image) 57 call = "docker run - d - it - - name " + str(jid) 58 call = add_resource_restriction(call); 59 pbs.logmsg(pbs.log_debug, "Resource is : %s" % call) 60 call = add_env(call); 61 pbs.logmsg(pbs.log_debug, "Env is : %s" % call) 62 get_pbs_dirs(); 63 pbs.logmsg(pbs.log_debug, "Directory is : %s" % hdir) 64 job_file=hdir+"/mom_priv/jobs/"+str(jid)+".sc" 65 call += " - v "+str(hdir)+":"+str(hdir)+" "+" - v "+str(edir)+":"+str(edir)+" - v /home:/home - - net=host" 66 call += " " + str(os_image) + " /bin/bash" 67 pbs.logmsg(pbs.log_debug, "Call is : %s" % call) 68 os.popen(call) 69 return 70 def launch_job (): 71 global prog 72 global hdir 73 is_script = prog.find("bash") 74 prog = "" 75 get_pbs_dirs() 76 pbs.logmsg(pbs.log_debug," is_script %d" % is_script) 77 if is_script!= - 1: 78 job_file=hdir+"/mom_priv/jobs/"+str(j.id)+".sc" 79 else: 80 for arg in args: 81 prog=prog+" "+arg e.progname="/usr/bin/docker" 84 e.argv = [] 85 e.argv.append("docker") 86 e.argv.append("exec") 87 e.argv.append(str(j.id)) 88 e.argv.append("/bin/bash") 89 e.argv.append("- c") 90 executable="" PBS Pro with Docker Integration 8

9 91 if is_script!= - 1: 92 pbs.logmsg(pbs.log_debug, "It's a script") 93 e.argv.append(job_file) 94 else: 95 pbs.logmsg(pbs.log_debug, "It's an executable") 96 executable="" 97 for arg in args: 98 executable+=arg+" " 99 pbs.logmsg(pbs.log_debug, "arg[1] : %s" % arg) 100 e.argv.append(executable) 101 return 102 create_container() 103 launch_job() 104 e.accept() 7.2. Docker_shutdown.py (execjob_end hook) 1 import pbs 2 import os 3 4 e = pbs.event() 5 6 j = e.job 7 8 jid = j.id 9 v = str(j.variable_list) 10 vl = v.split(",") 11 image="docker_image=" 12 os_image="none" 13 for i in vl: 14 pbs.logmsg(pbs.log_debug, "Variable is: %s" % i) 15 if i.startswith(image): 16 name = i.split("=",1) 17 os_image = name[1] 18 if "None" in os_image: 19 e.accept() call = "docker stop " + str(jid) 22 pbs.logmsg(pbs.log_debug, "Call is : %s" %call) 23 os.system(call) 24 call = "docker rm " + str(jid) 25 os.system(call) 26 e.accept() 8. Usage To create an execjob_launch hook using the hook script in section 7.1 do the following: 1 $ qmgr - c "c h launch_docker" 2 $ qmgr - c "s h launch_docker event = execjob_launch" 3 $ qmgr - c "i h launch_docker application/x- python default tmp/docker_launcher.py" PBS Pro with Docker Integration 9

10 To create an execjob_end hook using the hook script in section 7.2 do the following: 1 $ qmgr - c "c h stop_docker" 2 $ qmgr - c "s h stop_docker event = execjob_end" 3 $ qmgr - c "i h stop_docker application/x- python default tmp/docker_shutdown.py" To submit a job requesting a specific Docker image to run on (for example, centos:7 ), do the following: 1 $ qsub - lselect=1:ncpus=<ncpus>:mem=<mem> - v DOCKER_IMAGE="centos:7" <job script> 9. References 1 PBS Professional: HPC Workload Manager and Job Scheduler ( Professional&c=Overview- and- Capabilities) 2 What is Docker? ( PBS Pro with Docker Integration 10