Introduction to shell scripting

Transcription

1 Introduction to shell scripting Thomas Röblitz Research Infrastructure Services Group Department for Research Computing (DRC) DRC Course Week, March 25-28, 2014

2 Outline What is shell scripting? Invocation Variables Control flow (tests, if-then-else-fi, loops)

3 Poll Who did already use a shell?

4 Poll Who did already use a shell? write a shell script?

5 Poll Who did already use a shell? write a shell script? write a script (R, MATLAB, Bioportal, )?

6 What is a shell script? The shell is a computer program that presents you with the command line interface used on linux/unix systems A script is a list of commands that you ask the shell to perform, like a theater director giving a script to hist cast of actors An actor reads, interprets and executes the script according to the director s instructions The shell does the same with your scripts

7 What is shell scripting? A way to store, edit & execute shell commands store avoid re-typing again and again edit easily change commands execute execute commands (interactive or batch mode) Simplifies the repetition of long & complex command sequences

8 What is shell scripting used for? 1 A wrapper for a program aka SLURM file prepare job input data, run job, check results, copy results May help in avoiding (unnoticed) job failures To manage research environment create data/jobs layout (eg, for parameter studies) perform post-processing of results May help in reproducing results if necessary 1 on Abel

9 Simple script: Edit simplesh use your favourite (ASCII) text editor: vim, emacs, #!/bin/bash WHO=All if [ $# -gt 0 ] then WHO=$1 fi DATE=$(date) echo "Welcome $WHO to DRC Course Week\n$DATE"

10 Simple script: Running > bash simplesh Welcome All to DRC Course Week Tue Mar 25 12:48:10 CET 2014 > bash simplesh Abel Welcome Abel to DRC Course Week Tue Mar 25 12:58:34 CET 2014

11 Step-by-step hashbang invocation & arguments variables control flow

12 Hashbang 2 1st line in each shell script hashbang #! command interpreter, eg /bin/bash #!/bin/bash How about perl, /bin/perl? > which perl /usr/bin/perl which shows the full path of (shell) commands (see man which) 2 aka shebang, pound-bang, hash-pling,

13 Invocation Explicit use shell interpreter explicitly > sh simplesh Why sh and not bash? > which sh /bin/sh > file /bin/sh /bin/sh: symbolic link to bash sh is a symbolic link to /bin/bash (the default shell in a Linux system)

14 Invocation Implicit Run script without sh? > /simplesh -bash: simplesh: Permission denied Check > ls -l simplesh -rw-r--r-- 1 thomarob users 117 Oct 22 18:04 simplesh Fix > chmod u+x simplesh > ls -l simplesh -rwxr--r-- 1 thomarob users 117 Oct 22 18:04 simplesh

15 Invocation Scientists have no time How about saving two more keystrokes? > simplesh -bash: simplesh: command not found Check > which simplesh /usr/bin/which: no simplesh in (/hpc/bin: which searches the directories in the environment variable PATH > PATH=$PATH:$PWD > simplesh Welcome All to DRC Course Week Tue Mar 25 12:48:10 CET 2014 Alternative: use absolute path, ie, /path/simplesh

16 Arguments Welcome everybody Passing arguments > argumentssh everybody Welcome everybody to DRC Course Week Simplified script #!/bin/bash echo "Welcome $1 to DRC Course Week" $1 is a positional parameter positional parameters are set when the shell is invoked set with everybody in the invocation above

17 Arguments Welcome everybody Passing multiple arguments > argumentsxsh bash csh Welcome bash (1) to DRC Course Week Welcome csh (2) to DRC Course Week Welcome (3) to DRC Course Week Enhanced script #!/bin/bash echo "Welcome $1 (1) to DRC Course Week" echo "Welcome $2 (2) to DRC Course Week" echo "Welcome $3 (3) to DRC Course Week" $n is a positional parameter unused positional parameters are empty

18 Variables Assignment VARIABLENAME=any (un)quoted string VARIABLENAME is of characters a-z, A-Z, 0-9, _, no space between VARIABLENAME and = and value quoting (" or ) needed if value contains spaces Use $VARIABLENAME no need to declare before use (default: empty string) Before the shell executes a line, it will expand $VARIABLENAME to the actual value contained within Note the differences (with/out $) untyped (think: everything is a string)

19 Variables Examples Script #!/bin/bash ABEL=Abel ABLE="Abel is able" echo "Welcome $ABEL (ABEL) to DRC Course Week" echo "Welcome $ABLE (ABLE) to DRC Course Week" echo "Welcome $UNSET (UNSET) to DRC Course Week" Output Welcome Abel (ABEL) to DRC Course Week Welcome Abel is able (ABLE) to DRC Course Week Welcome (UNSET) to DRC Course Week Experiment with quoting remove double quotes in assignment of ABLE remove single and/or double quotes in echo for UNSET

20 Variables Examples (2) Variables #!/bin/bash mypath=/tmp/scripting_course myfilename=magic_scriptsh myfile=$mypath/$myfilename mytext="welcome!" mynumber=0815 myarray=(apple banana strawberry) Use mkdir -p $mypath echo "$mytext" > $myfile echo $mynumber >> $myfile mv $myfile $myfile${myarray[2]} for ((i=0; i<=$mynumber; i++)); do echo $i; done

21 Variables Environment (1) Many useful variables defined in the environment General (run env after logging in), eg PWD current path/directory HOME home directory PATH searched for executables/scripts SHELL the current shell Can you (now) 3 explain the following command? > PATH=$PATH:$PWD 3 See example on slide 15

22 Variables Environment (2) Job (run env after qlogin ), eg SLURM_JOBID SCRATCH SLURM_JOB_NODELIST SLURM_TASKS_PER_NODE SLURM_MEM_PER_CPU job id directory for in/out data on /work list of nodes allocated to the job number of tasks per node memory per core

23 Status You should now understand the highlighted parts #!/bin/bash WHO=All if [ $# -gt 0 ] then WHO=$1 fi DATE=$(date) echo "Welcome $WHO to DRC Course Week\n$DATE"

24 Variables Command substitution Suppose we want to use the same date multiple times #!/bin/bash date +"Script starts at %c" sleep 3 # do sth date +"Script started at %c" date +"Script ends at %c" Script starts at Tue 25 Mar :02:46 CET Script started at Tue 25 Mar :02:49 CET Script ends at Tue 25 Mar :02:49 CET need to save the start time in a variable

25 Variables Command substitution(2) Syntax VARIABLENAME=$(command with arguments) Variable is substituted with the command s output #!/bin/bash START=$(date +%c) echo "Script starts at $START" sleep 3 # do sth echo "Script started at $START" date +"Script ends at %c" Script starts at Tue 25 Mar :10:38 CET Script started at Tue 25 Mar :10:38 CEST Script ends at Tue 25 Mar :10:41 CEST Command substitution is a powerful tool Try it!

26 Control flow Branch depending on input, result of a command Repeat some statements

27 If-then-else Example Example #!/bin/bash DATE=$(date +%s) START=$(date -d "Thu 27 Mar :20:00 CET" +%s) if [ $DATE -gt $START ] then echo "DRC course already started" else echo "DRC course not yet started" fi DRC course not yet started

28 If-then-else Syntax if [ conditional_expression ] then statements else statements fi else branch is optional [ conditional_expression ] same as test conditional_expression Note! Must not remove spaces around [ and ]

29 Tests Conditional expressions For details see man test Examples expression evaluates to true if! EXP EXP is false EXP1 -a EXP2 both EXP1/2 are true EXP1 -o EXP2 either EXP1/2 is true STRING1 = STRING2 strings are equal STRING1!= STRING2 strings are not equal INT1 -eq INT2 integers are equal INT1 -gt INT2 INT1 INT2 -e FILE FILE exists -x FILE FILE exists incl execute permission

30 Putting it all together #!/bin/bash WHO=All # $# is the number of arguments if [ $# -gt 0 ] then WHO=$1 fi DATE=$(date) echo "Welcome $WHO to DRC Course Week\n$DATE"

31 Loops Example Example #!/bin/bash for i in $(find -type f) do file $i done

32 Loops (for) Syntax for VAR in LIST do statements done VAR is left side of variable assignment (no $) LIST can be given like apple cherry kiwi or as output of a program

33 Become a shell scripting user/expert You just learned the very basics: hashbang, invocation, arguments, variables, if-then-else, loops & tests read man pages: man bash, man test search internet for tutorials study examples and scripts of co-workers Most important! Practice, practice, practice

34 Happy scripting