HPC Course Session 3 Running Applications

Transcription

1 HPC Course Session 3 Running Applications Checkpointing long jobs on Iceberg 1.1 Checkpointing long jobs to safeguard intermediate results For long running jobs we recommend using checkpointing this allows you to run a job for a shorter period, and then save the entire state of the application at the end of the job. The job can then be re submitted, and will continue from where it was saved. This protects you against loosing days/weeks of calculations if a single, long running job were to fail through a power cut or by timing out before the analysis completed. 1.2 Submitting your checkpointing jobs to ICEBERG Jobs submission scripts must include extra options for using checkpointing that indicate in the script that the intermediate results should be saved: #$ ckpt blcr #$ c sx Additionally when submitting a job you need to prefix your executable command with 'cr_run'. E.g.: cr_run [normal command to execute]. In the given example, to submit a job this would be: cr_run(space)qsub(space)colony_molecol_job.sh The scheduler will then run your job as normal, but after the time requested (i.e. h_rt) it will create a snapshot of the running process before terminating it. The checkpoint file (which is a complete copy of the application's in memory state) will be saved in the current working directory under the file name checkpoint.[jobid].[pid]. E.g. checkpoint (see below) The amount of memory in use by the application determines the size of the checkpoint file, so if you are using a large amount of memory make sure you have sufficient free disk space in your working directory to save the file. 1.3 Restarting a job that has timed out using the intermediate checkpoint file To restart a checkpointed job you need to create a new job submission script with the same options but add the cr_restart command to the script and the name of the intermediate checkpoint file as below: #!/bin/bash #$ l h_rt=168:00:00 [... any other normal options...] #$ ckpt blcr #$ c sx cr_restart checkpoint.[jobid].[pid]

2 Replace [jobid] and [Pid] with the values for the generated checkpoint file, E.g. checkpoint To restart the job, resubmit the job normally using the qsub command. i.e. qsub(space)colony_molecol_job.sh Each time the job ends a new checkpoint file will be generated, which you can then use to resubmit the job (until such time that it completes). How to run COLONY in the ICEBERG platform using checkpointing 2.1 Colony description and analysis on a Desktop PC versus Iceberg Colony is a computer program implementing a maximum likelihood method to assign/infer parentage and sibship among individuals using their multi locus genotypes. Colony can be used, among others, in estimating full and half sib relationships, assigning parentage, inferring mating systems (polygamous / monogamous, selfing rate) and reproductive skew in both diploid and haplo diploid species. It can also be used to simulate genotype data with a particular sibship and parentage structure. The method is formally described in the following papers Wang (2004, 2012, 2013) and Wang and Santure (2009). Several factors determine how fast Colony runs and thus how much time it takes to analyze your data. These are discussed in detail in the software manual. In general, the analysis method used dictates computational time. The full likelihood method (FL) and the pair likelihood score method (PLS), incur very different computational intensities; full likelihood takes longest. Depending on the analysis method used and dataset, a Colony analysis can be quite computationally intensive and to run a single job using FL may require most of the CPU on a desktop and may take several days or weeks. If you wish to run a quick and short preliminary analysis using PLS then a desktop is sufficient. For your analysis, you may wish to run several replicate jobs using different numbers of markers and different genotyping error rates. This would require you to run several jobs. The advantage of using High Performance Computing and the Iceberg computer cluster is that you can run multiple jobs simultaneously. This protocol describes how to submit a job to the serial version of Colony on the Iceberg computer cluster. Comparative tests of run times in a Linux environment indicated that serial versions were faster than the parallel. The Colony software is downloadable from the website

3 2.2 molecol ICEBERG access molecol ICEBERG is a private computer cluster belonging to NERC Biomolecular Analysis Facility (NBAF), with exclusive access for NBAF and Molecular Ecology group members. To run short and test jobs you can use the Iceberg computer cluster, the maximum run time of Iceberg is 168 hours (7 days). If you have long jobs to run, you may wish to use molecol ICEBERG which has a maximum run time of 672 hours (28 days). 2.3 Input files for Colony analysis software Example 2 Example 2 : An ant (Leptothorax acervorum) dataset The original dataset comprises of 377 ant workers (diploid) sampled from 10 known colonies, each headed by a single monogamous queen (diploid). Males are also monogamous. Therefore, the sampled workers are either fullsibs from the same colony or nonsibs from different colonies. Candidate males and females are not available. Ant workers are genotyped at up to 6 microsatellite loci, which have a number of observed alleles varying between 3 to 22. In the HPC exercise we will use a subset of the original dataset from Example 2 comprising of the first 25 ant workers (diploid) in the dataset. The ants have been genotyped at up to 6 microsatellite loci. Using Colony software, we will reconstruct the sibships amongst individuals to identify how many colonies the individuals belong to. The rates of genotyping errors are assumed in the analysis to be 5% for both allelic dropout and other kind of errors at each locus. This modified data input file requires 5 7 minutes to complete analysis. In the HPC exercise we will simulate a job timing out after three minutes prior to analysis completion. Then we will re start the job using the intermediate checkpoint file and run until completion. 2.4 Prepare your input files for Colony analysis using the ICEBERG platform To run COLONY in the molecol ICEBERG platform you need the following files: 1 Your COLONY project data input file in.dat data format (colony2.dat) as well as the other accessory files. Using the desktop version of COLONY, create a project, choosing the settings according with your species and data etc. Find the created files Colony2.dat and accessory files in the programme folder e.g. at C:\Program Files\ZSL\Colony\...(projectname). 2 The job submission file (colony_molecol_job.sh) to submit jobs to the molecol Iceberg cluster using checkpointing (see description below). The job submission file contains information regarding job run length (e.g. 672 hours, note this is the run length of the molecol cluster), ICEBERG cluster for job submission (e.g. molecol), job to use checkpointing ( ckpt blcr; c sx), e mail notification about job progress (e.g. at bea; b (beginning) e (end) or (a) abort) and e mail address for notification (e.g. c.mcinerney@sheffield.ac.uk), path directory to software executable file, path directory to input file; for more information see

4 This job submission script is specific to submitting a job to the molecol Iceberg queue. This is denoted by the lines : P molecol, q molecol.q. You must have permission to use this cluster (contact the administrator). Alternatively, for test runs and short jobs, these should be submitted to the Iceberg (short or long queues). Your job submission script to submit a job to the Iceberg queue: a) will not have the lines : P molecol, q molecol.q b) the line denoting the run time will have either 8 or 168 hours depending on which Iceberg queue you wish to submit your job to short or long queues, respectively (e.g: l h_rt=8:00:00 or l h_rt=168:00:00). For more information see: The job submission file (colony_molecol_job.sh) to submit jobs to the Iceberg using checkpointing: Prior to beginning a Colony analysis both your colony project input data file.dat and your job submission script files need to be modified. This must be done within ICEBERG using the nano function. This is explained next in the HPC exercise.

5 3. 1 HPC exercise to run Colony ICEBERG using checkpointing 1. Login to ICEBERG via the internet at my APPS Portal using the following links: or Log in with your User ID and Password for your ICEBERG account (provided by CICS). 2. Select Interactive job that opens a command window (e.g. bo1cem@testnode02:~): 3. In the ICEBERG window, move to the your data folder identified by your account username (bo1cem in this example) by typing

6 cd(space)/data/bo1cem/hpc_course/colony 4. Check the folder contents by typing the unix command to list files ls. ls Notice data files are present but no job submission script. We will now copy the job submission script from the example job submission scripts available in the Genomics folder. 5. You should now copy the colony job submission script colony_molecol_job.sh into your folder. You do this by typing the cp command followed by the directory of the job submission script and then followed by the directory of your colony folder in your file directory location cp (space) /usr/local/extras/genomics/submit_scripts/colony_molecol_job.sh (space) /data/bo1cem/hpc_course/colony 6. Check you have successfully copied the file by typing ls (list files unix command). ls The jobs submission script colony_molecol_job.sh should now be listed as a file in your folder. 3.2 Modify your Job submission script file 1. Open the job submission script using the nano text editor function by typing: nano(space)colony_molecol_job.sh This opens this job submission script which must now be edited.

7 2. Modify the time: In a real analysis the line denoting the run time will have either 8 or 168 hours depending on which Iceberg queue you wish to submit your job to short or long queues, respectively (e.g: l h_rt=8:00:00 or l h_rt=168:00:00). For the molecol Iceberg cluster the maximum time is 672 hours (28 days). In this exercise we will simulate a job timing out prior to completion; please modify the run time to 3 minutes. l h_rt=00:03:00 3. Modify the job queue: This script is specific for sending jobs to the molecol Iceberg cluster job queue. As you do not have permissions to submit jobs to the molecol Iceberg cluster queue you must now delete the lines: P molecol, q molecol.q 4. Modify the e mail address for notifications about the job progress: Change the e mail address to your own (e.g. c.mcinerney@sheffield.ac.uk) to receive job progress notifications at the job beginning (b), end (e) or when a job failed /aborted (a) m bea M c.mcinerney@sheffield.ac.uk 5. Modify the path directory to your colony data input file: Specify your folder directory containing your colony files and your data input file: /data/bo1cem/hpc_course/colony/colony2.dat 6. To save the changes and exit the nano text editor: Press Control O to write the file i.e. save the changes. Save the file using the same name by pressing enter. Press Control X to exit. 3.3 Modify your Colony data input file Newer versions of Colony save the data input file as Colony2.Dat. The source code in the Linux environment is an older version, however, and the file name required is colony2.dat. We need to modify the file name and some other parameters in the data input file. 1. Open the Colony2.Dat using the nano function by typing: nano(space)colony2.dat 2. Change the directory location in the two first lines of the file (highlighted) to match the directory location of your colony folder where you wish to carry out your analysis: /data/bo1cem/hpc_course/colony/

8 3. To save the changes and exit the nano text editor: Press Control O to write the file i.e. save the changes. Save the file using a new name, save the file as colony2.dat. Press Control X to exit. 4. Remove the old Colony2.Dat file by typing: rm(space)colony2.dat 5. Convert your colony2.dat file to a compatible unix file by typing in the ICEBERG window: dos2unix (space) colony2.dat or alternatively by typing /usr/bin/dos2unix (space) colony2.dat Now you re ready to start submitting your COLONY job(s) to ICEBERG 3.4 Submitting your COLONY jobs using checkpointing in the ICEBERG platform 1. To submit your COLONY job(s) to ICEBERG type: cr_run(space) qsub(space)colony_molecol_job.sh. 2. Check the status of your jobs submitted to Iceberg by typing:

9 Qstat 3. Alternatively, to check the status of jobs submitted to molecol ICEBERG cluster type: Qstat(space) q (space)molecol.q Running jobs are indicated by an r while queued jobs are indicated by qw. 3.5 Restarting a Colony job that has timed out using the intermediate checkpoint file In this exercise we are simulating a job that times out. After the job has timed out you should have received e mail notification and a checkpoint file (which is a complete copy of the application's in memory state) was saved in the current working directory under the file name checkpoint.[jobid].[pid]. This intermediate checkpoint file may be used to restart a Colony job. 1. Identify the intermediate checkpoint file for your Colony job by listing the folder files by typing: ls 2. Make a note of the checkpoint file, for example: checkpoint To restart a checkpointed job you need to create a new job submission script. Open the job submission script colony_molecol_job.sh using the nano text editor function by typing: nano(space)colony_molecol_job.sh 4. Modify the job submission script to change the run time to10 minutes. l h_rt=00:03:00 5. Modify the job submission script to include a line with the cr_restart command and the name of the intermediate checkpoint file as below: #!/bin/bash #$ l h_rt=00:10:00 [... any other normal options...] #$ ckpt blcr #$ c sx cr_restart checkpoint.[jobid].[pid] 6. To save the changes and exit the nano text editor: Press Control O to write the file i.e. save the changes. Save the file using the same name. Press Control X to exit. 7. Restart your job using the normal job submission command: qsub(space)colony_molecol_job.sh Your job should complete and you should receive e mail notification!

10 Examining and copying your results files using WinSCP WinSCP is a programme for file transfer from / to your desktop into ICEBERG. 1. Download and install WinsSCP using the following link: If possible, choose this option: Download the stand alone version(4.37) from a local copy. 2. Open Winscp, to login insert the Host name for ICEBERG and your User name and password for your ICEBERG account, as shown in the figure below, and select Login. On the left side of the window you can search for your input files while on the right side you will create folders and import your input files to ICEBERG from the left side. 3. Within Winscp, use the Go to option to navigate your windows to your file directories.

11 After the Colony job has finished you will have the same 17 output files as those created by COLONY in Windows. Some of these are not visible in the Linux environment but can be viewed using Winscp. To ensure that your job has completed check that you have the Bestconfig file (see below). Result files can be transferred and saved to a desktop using Winscp. References: (1) Wang J Sibship reconstruction from genetic data with typing errors. Genetics 166: (2) Wang J & Santure AW Parentage and sibship inference from multi locus genotype data under polygamy. Genetics 181: (3) Jones OR & Wang J COLONY: a program for parentage and sibship inference from multilocus genotype data. Molecular Ecology Resources 10: (4) Wang J Computationally efficient sibship and parentage assignment from multilocus marker data. Genetics 191: (5) Wang J A simulation module in the computer program COLONY for sibship and parentage analysis. Molecular Ecology Resources 13: The Colony software is downloadable from the website