Working with Shell Scripting. Daniel Balagué

Transcription

1 Working with Shell Scripting Daniel Balagué

2 Editing Text Files We offer many text editors in the HPC cluster. Command-Line Interface (CLI) editors: vi / vim nano (very intuitive and easy to use if you are a new user - it has no undo) emacs etc. Graphical User Interface (GUI) editors: gedit kwrite etc.

3 What is a Shell Script? & Why Shell Scripts? In a shell script, the code is not compiled; it is interpreted. The interpreter reads the lines of the code and it executes it sequentially. They are often less efficient than a compiled code. Then, why would you use a shell script? Because they are faster to code than C/C++ code (and do not require compilation) with an acceptable performance. It makes the tradeoff worthwhile. Shell Scripts, why? 1. Simplicity: the shell is a high-level language; you can express complex operations clearly and simply using it. 2. Portability: use just the POSIX-specific features and your script would likely run unchanged in other systems. 3. Ease of development: you can often write a powerful script in little time. Examples of scripting languages: awk, BASH, Perl, Python, Ruby, Lua, etc.

4 Creating a BASH Script: The first line BASH scripts start with the following line: #!/path/to/bash Usually it is: #!/bin/bash (it may change in different Linux or Unix flavors).

5 Executing the Script Use the following command:./script.sh (for a relative call) Here, the. (dot) means the current working directory. or /path/to/script.sh (for an absolute call)

6 Hello World (C v.s. BASH) C BASH #include <stdio.h> int main(int argc, char* argv[]) { printf("hello World\n"); return 0; } #!/bin/bash echo "Hello World"

7 You are Ready! It is time to log into the HPC cluster!!! ssh

8 The SLURM Script The SLURM script, is the script that will be read by the job scheduler, and it will be used to allocate your resources. You can think of the SLURM script as a personalized BASH script.

9 Structure of the SLURM Script These are some (but not all) of the options for the configuration: #!/bin/bash #SBATCH --account=<group_account> # Group account (-A) #SBATCH --nodes=1 # number of nodes required (-N) #SBATCH --tasks-per-node=1 # number of tasks per node (-n) #SBATCH --cpus-per-task=2 # number of cpus per task (-c) #SBATCH --partition=<partion_name> # Partition type (-p) #SBATCH -J jobname # A Job Name for the array #SBATCH --mem=1gb # memory reserved #SBATCH --time=00:02:00 # minutes #SBATCH --mail-user=abc123@case.edu # user #SBATCH --mail-type=all # notifications #SBATCH -o readgroup_job.o%j # Screen output

10 Allocate resources correctly Common errors HPCC users encounter: Long waiting time in the queue. Job cancelled because there was not enough memory allocated. Job cancelled because it exceeded the time limit. OpenMP jobs running in serial. We recommend you reading our Batch & Interactive Job documentation, as well as the SLURM sbatch documentation

11 Example of a Simple SLURM Script #!/bin/bash #SBATCH -o test.o%j #SBATCH --time=00:30:00 #SBATCH -N 1 #SBATCH -n 1 #SBATCH -c 1 module load <software-module>./<executable> [options] [inputs]

12 Using Scratch Space (explained) basic_script.slurm Scheduler Options First line #!/bin/bash #SBATCH -o test.o%j #SBATCH --time=00:30:00 #SBATCH -N 1 #SBATCH -n 1 Load necessary modules Copy files to scratch Move to the scratch folder Execute your code / tool in scratch Copy just new generated files back module load <software-module> cp -r my_files $PFSDIR cd $PFSDIR./<executable> [options] [inputs] cp -ru * $SLURM_SUBMIT_DIR

13 Useful SLURM Variables Variable Description $PFSDIR $SLURM_SUBMIT_DIR The scratch folder created for the job Directory from which the Slurm script was launched $SLURM_ARRAY_TASK_ID Index in the job array variable $CUDA_VISIBLE_DEVICES Shows the GPU identifier

14 Repetitive Jobs - Use a Job Array You can launch the same job over and over with different parameters using job arrays. What are job arrays? They are a way to launch your jobs as soon as resources are available. Jobs (sort of) run in parallel. You only need one script!

15 More on Job Arrays Run your job array with sbatch --array=1-n you_script.slurm Control the parameters inside your script with: $SLURM_ARRAY_TASK_ID

16 Thank you for your attention! Questions? It is time for some exercises!

17 ADVANCED MATERIAL

18 Using Variables in BASH hello2.sh #!/bin/bash FIRST_NAME="Daniel" LAST_NAME="Balague" echo "Hello $FIRST_NAME $LAST_NAME" WARNING: Do not put spaces between variable names, the equal sign, and the value for the variable.

19 BASH Script that Takes Parameters hello3.sh #!/bin/bash echo "Hello $1 $2" Here $1 and $2 are the first and second arguments passed when we call the script

20 Fancier Output: printf We can modify the way numbers or strings are displayed on the screen: print_numbers.sh #!/bin/bash NUMBER=12 printf "The number is %04d\n" "$NUMBER"

21 Assign the Output of a Command into a Variable print_numbers2.sh #!/bin/bash NUMBER=12 STRING=$(printf "The number is %04d\n" "$NUMBER") echo $STRING Produces the same output as before: