Unix Power Use. Martti Louhivuori. Spring School in Computational Chemistry 2013 PRACE Advanced Training CSC

Transcription

1 Unix Power Use Martti Louhivuori Spring School in Computational Chemistry 2013 PRACE Advanced Training CSC

2 Shell Scripting Martti Louhivuori CSC Tieteen tietotekniikan keskus Oy CSC IT Center for Science Ltd.

3 Introduction shell scripting is the art of writing scripts that can automate repetitive tasks here we present BASH and discuss how it can be used as a general programming languages

4 Different shells sh Bourne shell csh/tcsh C shell ksh Korn shell zsh Z shell bash Bourne again shell sh successor that includes csh and ksh features

5 When NOT to use a shell script? resource or compute intensive large-scale applications complex datastructures lot s of I/O, need libraries, floating point arithmetic, use Perl, Python or even C or Fortran instead

6 When to use a shell script? glue together Unix commands to automate repetitive tasks awk, bc, more, less, paste, simple applications that do not need lots of resources fancy scripts for batch jobs

7 How to write a script? put commands that you would execute on the command-line in a file $ cat hello.sh echo "Hello world!" echo "... this is my first script." $ bash hello.sh Hello world!... this is my first script.

8 Pipe magic commands can be chained together using a pipe $ cat hello.sh grep my echo "... this is my first script." $./configure --help less the output of the 1st command (left) is used as the input of the 2nd command (right) arbitrary number of pipes can be used

9 Pipe magic with files output can be redirected to a file with > overwrite >> append input can be read from a file with < read $ echo "Hello world!" > hello $ echo "... what's up?" >> hello $ cat < hello Hello world!... what's up?

10 Pipes & streams three special streams always opened: stdin 0< keyboard stdout 1> stderr 2> screen error messages (on screen) stdout and stderr can be joined with &> or &>> M>&N redirects file descriptor M to N i.e. 2>&1 joins stderr and stdout

11 Pipes & streams # separate stdout & stderr $./myprogram > output 2> errors # joined stdout & stderr $./myprogram &> output+errors # one-time redirect $./myprogram > filename 2>&1 # permanent redirect $ exec 2>&1 $./myprogram > filename # swap stdout and stderr $./myprogram 3>&2 2>&1 1>&3

12 Executable script, Sha-Bang! add a shebang to the script to define the shell that executes it #!shell-binary hello.sh: #!/bin/bash echo "Hello world!" $ chmod u+x hello.sh $./hello.sh Hello world!

13 Comments anything after a # is treated as a comment can be added as separate line, or at the end of a line hello.sh: #!/bin/bash # let's greet the world... echo "Hello world!" # cliche $ bash hello.sh Hello world!

14 Variables like other programming languages bash has variables variables can be set with name=value to use the variable precede the name with $, e.g. $name you may need to enclose the variable name in curly brackets: ${name}

15 Variable expansion $ name=max $ echo "Hi $name!" Hi Max! $ name="sir Max"; d=sun $ echo "Dear $name," $ echo "today is ${d}day." Dear Sir Max, today is Sunday.

16 Variable types there are two types of variables strings integers (no native floating point type!) additionally variables can be read-only an array by default variables can hold any data we assign to it this can be limited with "declare"

17 declare declare an integer variable declare I name=value declare a read-only variable declare r name=value print the variable attributes declare p name

18 Arrays bash supports 1D arrays defining an array element-by-element: myarray[123]=456 as a list: myarray=( apple 1 ) elements do not need to be contiguous value accessed as ${myarray[123]} does NOT work without the curly brackets!

19 Arrays special array syntax ${myarray[*]} all items ${!myarray[*]} all indexes ${#myarray[*]} number of items ${#myarray[0]} length of item 0 ${#myarray[1]} length of item 1

20 Command line arguments command line arguments are stored as special variables $# number of arguments all arguments $0 name of script $1 first argument $2 second argument

21 Command line arguments cmd-args.sh: #!/bin/bash echo "args: echo "name: $0" echo "1st argument: $1" echo "3rd argument: $3" $ bash cmd-args.sh one two three args: one two three name: cmd-args.sh 1st argument: one 3rd argument: three

22 Quotations single quotes (') preserve literal values $ a='hello!' $ echo '$a' $a double quotes (") preserve literal values, except for: $ ` \ $ echo "$a" hello! $ echo "$a \" " hello! "

23 Command substitution the output of a command can be used in a script with $(command) (or `command`) $(command) is substituted with the output of command just like a variable would be $ echo "Today is $(date)." Today is Fri Aug 26 10:58:03 EEST $ b=20.1 $ a=$(echo "$b*10.0" bc l) $ echo $a

24 Arithmetics simple arithmetics performed using: (( expression )) let "expression" variable and command expansion done on expression $ not needed for variables in expression $ a=10 $ (( c = a - 4 )) $ echo $c 6 $ let "a += 3" $ echo $a 13 $ echo $a $ (( a++ )) 14

25 Arithmetic expansion $(( expression )) replaced by value of expression $ a=10; b=20 $ echo $(( a + b )) 30 $ echo "$a x $b = $(( a * b ))" 10 x 20 = 200 (( expression )) used for if and for constructs $ if (( $DEBUG $MODE == "d" )) then echo "debug mode ON" fi

26 Arithmetic operators Operator Meaning + - addition, subtraction * / % multiplication, division, remainder ** exponentiation VAR++ VAR-- variable post-increment and post-decrement ++VAR --VAR variable pre-increment and pre-decrement <= >= < > comparison operators ==!= equality and inequality && logical AND logical OR! ~ logical and bitwise negation & bitwise AND ^ bitwise exclusive OR bitwise OR << >> left and right bitwise shifts expr? expr : expr conditional evaluation = *= /= %= assignment operations += -= <<= >>= &= ^= =

27 Conditional if statements basic form: if test-commands then do-stuff elif test-commands then do-stuff else $ if [[ 1 eq 1 ]] then echo A else echo B fi A fi do-stuff $ [[ 1 eq 1 ]] && echo A echo B A

28 Test commands four ways to do this (yes really ) test command or [ ] single brackets [[ ]] double brackets (( )) arithmetic expansion double brackets are safer than test or single brackets operators for strings, integers, files and Boolean algebra

29 [[ ]] file test operators [[ -e name ]] file exist [[ -f name ]] is a regular file [[ -d name ]] is a directory [[ -s name ]] is size of file not zero [[ -r name ]] file has read permission [[ -w name ]] file has write permission [[ -x name ]] file has execute permission

30 [[ ]] string operators [[ "s1" == "s2" ]] string equality [[ "s1"!= "s2" ]] string inequality [[ "s1" < "s2" ]] string lexiographic before [[ "s1" > "s2" ]] string lexiographic after [[ "s1" =~ "s2" ]] regular expression match [[ -z "s1" ]] string has zero length [[ -n "s1" ]] string has non-zero length

31 [[ ]] integer operators [[ 1 eq 2 ]] equality [[ 1 ne 2 ]] non-equality [[ 1 lt 2 ]] less than [[ 1 gt 2 ]] more than [[ 1 -le 2 ]] less than or equal [[ 1 -ge 2 ]] greater than or equal

32 [[ ]] Boolean algebra [[ A B ]] (logical) A or B [[ A && B ]] (logical) A and B [[! A ]] not A

33 Conditional example xor.sh: #!/bin/bash if [[ ($1!= $2) && \ ( $1 $2 ) ]] then echo True else echo False fi $ bash xor.sh 1 0 True $ bash xor.sh 1 1 False backslash can be used to continue the command on the next line

34 Case branch code by matching the value of a variable against expression(s) case "$variable" in expr1) commands... ;; expr2) commands... ;; *) default commands... ;; esac each block terminated with ;; always true!

35 Case example case.sh: $ bash case.sh Z #!/bin/bash uppercase letter DEBUG=0 $ bash case.sh 77 case "$1" in number -d --debug) $ bash case.sh f! DEBUG=1 something else ;; $ bash case.sh --debug [0-9]*) debug mode ON echo "number" ;; [[:upper:]]) echo "uppercase letter" ;; *) echo "something else" ;; esac (( $DEBUG )) && echo "debug mode ON"

36 Loops for loops over words in a string for name in Alice Bob Charlie or over numerical indeces for ((i=0; i<10; i++)) HUOM: {0..10} continue jumps to next iteration break steps out of the loop while, until

37 Loops over words loop.sh: #!/bin/bash for name in Alice Bob "Charlie D" do echo "Who? $name" done $ bash loop.sh Who? Alice Who? Bob Who? Charlie D

38 Loops over files loop-files.sh: #!/bin/bash for f in *.sh do l=$(wc $f awk '{ print $1 }') echo "$f has $l lines" done $ bash loop-files.sh loop-args.sh has 7 lines loop-files.sh has 8 lines

39 Loops over arguments loop-args.sh: #!/bin/bash for arg in do echo $arg done quotes needed to handle word splitting correctly! $ bash loop-args.sh one two "3 4" one two 3 4

40 Loops over integers loop-int.sh: #!/bin/bash for ((i=0; i<4; i++)) do echo part-$i done $ bash loop-int.sh part-0 part-1 part-2 part-3

41 Loops while loop-while.sh: #!/bin/bash i=0 while [[ $i lt 4 ]] do echo part-$i let "i += 1" done $ bash loop-while.sh part-0 part-1 part-2 part-3

42 Functions (limited) support for functions funcs-hello.sh: #!/bin/bash hello () { echo "Funky hello!" } echo "calling function hello()..." hello call $ bash funcs-hello.sh calling function hello()... Funky hello! declaration must be before call! declaration

43 Variables scope bash has three scopes for variables: local, global and environment variables local variables are local to current codeblock set with: local var=value use local variables in functions! global variables are global in current shell set with: var=value

44 Variables scope environment variables are inherited by all launched sub-shells set with: export var=value changes in value are not passed back! by convention, ALL_CAPS used for names

45 Variables scope variables.sh: #!/bin/bash scopetest() { local lvar=20 echo "---in function---" echo "lvar: $lvar" echo " var: $var" echo "EVAR: $EVAR" } var=30 export EVAR=40 scopetest echo "---in main scope---" echo "lvar: $lvar" echo " var: $var" echo "EVAR: $EVAR" bash print-variables.sh print-variables.sh: #!/bin/bash echo "---in next shell---" echo "lvar: $lvar" echo " var: $var" echo "EVAR: $EVAR"

46 Variables scope $ bash variables.sh ---in function--- lvar: 20 var: 30 EVAR: in main scope--- lvar: var: 30 EVAR: in next shell--- lvar: var: EVAR: 40 undefined!

47 Functions input funcs-input.sh: #!/bin/bash addup() { local total=0 while [[ $# -gt 0 ]] do case "$1" in [0-9]*) let "total += $1" ;; *) echo "invalid argument: $1" >&2 ;; esac shift done echo "Grand total: $total" } addup addup 3 8 foo input parameters work like command-line arguments $ bash funcs-input.sh Grand total: 22 invalid argument: foo Grand total: 11 $ bash funcs-input.sh 2> /dev/null Grand total: 22 Grand total: 11

48 Functions output no direct way to get output! ;( return value of function is 0 success nonzero failure options are global variable (not recommended but fast) command substitution, i.e. echo output (slow?) output variable name passed as input

49 Functions - output # set global out_a(){ val_a=$1 } # command subst out_b(){ echo $1 } # pass output var name out_c(){ eval $1='$2' } $ out_a 123 $ echo $val_a 123 $ val_b=$(out_b 123) $ echo $val_b 123 $ out_c val_c 123 $ echo $val_c 123 eval will evaluate the line after shell expansions

50 Here documents pipe an arbitrary length text block spanning over multiple lines to a command $ command << END... multiple lines of text... END same as command < file where file includes the input above END is an arbitrary tag that marks the end of input

51 Here documents heredoc.sh: cat << EndBlock A here document can be used to print out instructions using cat. It can also be used to steer an interactive program as shown below... EndBlock gnuplot << EOF f(x)=1/x plot f(x) pause 2 EOF

52 Here documents $ bash heredoc.sh A here document can be used to print out instructions using cat. It can also be used to steer an interactive program as shown below...

53 Further information on-line, free (as in beer) resources Bash guide for Beginners Advanced Bash-Scripting Guide BASH FAQ GNU Bash Reference Manual

54 Further information books Learning the Bash Shell (O Reilly) Bash Cookbook (O Reilly) Pro Bash Programming: Scripting the GNU/Linux Shell

55 Make Martti Louhivuori (Jura Tarus)

56 Why make? Module B Module A Module C Program program separated into several files multiple interdependant modules compilation and linking becomes easily a nightmare especially when developing the program!

57 Why make? when code has been modified, there are two approaches to compile the program: re-compile everything too slow keep records and re-compile only what is needed too much work make makes life easier by taking care of all the book keeping

58 Makefile defines: work-flow(s) for producing target(s) dependencies of each target library paths, compiler flags etc. directives for conditional definitions etc. # starts a comment usually called Makefile other choices: makefile, GNUmakefile

59 RULE Basic syntax name (usually filename) list of files / rules target: dependencies recipe... commands to execute Note: use tabs instead of spaces to example: indent recipes! foo.o: foo.c bar.h # module foo cc -c foo.c clean: rm *.o # remove all

60 Basic syntax target usually the file that is produced by the recipe name of an action also commonly used for example: clean, distclean dependencies a list of (source) files needed by the recipe may also be other targets recipe a list of commands to execute to make target

61 Logic of make read general macro definitions etc. call the rule for target check when dependencies were changed if any of the dependencies have changed, the target is re-built according to the recipe dependencies may also be targets for other rules in that case, make calls those rules

62 Simple example hello: main.o sub1.o sub2.o sub3.o f90 -o hello main.o sub1.o sub2.o sub3.o main.o: main.f90 f90 -c main.f90 sub1.o: sub1.f90 f90 -c sub1.f90 sub2.o: sub2.f90 f90 -c sub2.f90 sub3.o: sub3.f90 f90 -c sub3.f90 clean: rm hello main.o sub1.o sub2.o sub3.o

63 Which target? by default, the first target is called hello in the previous example target can be also specified when running make make target make clean make main.o

64 Variables contain a string of text variable = value substituted in-place when referenced $(variable) value sometimes also called macros shell variables are also available in the makefile $(HOME), $(USER),

65 Two flavors of variables in GNU make recursive variables defined as: foo = bar expanded when referenced simple / constant variables defined as: foo := bar expanded when defined foo = $(bar) bar = $(ugh) ugh = Huh? $(foo) Huh? x := foo y := $(x) bar x = later $(x) later $(y) foo bar

66 Variables by convention variables are name in ALL- CAPS in the previous example we could have used a variable to store the names of all objects OBJ = main.o sub1.o sub2.o sub3.o

67 Simple example revisited OBJ = main.o sub1.o sub2.o sub3.o hello: $(OBJ) f90 -o hello $(OBJ) main.o: main.f90 f90 -c main.f90 sub1.o: sub1.f90 f90 -c sub1.f90 sub2.o: sub2.f90 f90 -c sub2.f90 sub3.o: sub3.f90 f90 -c sub3.f90 clean: rm hello $(OBJ)

68 Common variables some common variables CC CFLAGS FC FCFLAGS LDFLAGS OBJ SRC

69 Special variables $< name of the target client: client.c $(CC) client.c -o name of the first dependency client: client.c $(CC) $< -o

70 Special variables $+ $^ $? list of all dependencies list of all dependencies (duplicates removed) list of dependencies more recent than target client: client.c $(CC) $+ -o

71 Special variables $* common prefix shared by the target and the dependencies client: client.c $(CC) -c -o $*.o $*.c

72 Special characters / continues a line # starts a executes a command quietly by default, make echos all commands executed this can be prevented by at the beginning of the quiet echo quiet echo echo normal echo echo normal echo normal echo

73 Special characters if there is an error executing a command, make stops this can be prevented by using a sign at the beginning of a command clean: -rm hello -rm $(OBJ)

74 Implicit rules one can use special characters to define an implicit rule e.g. quite often target and dependencies share the name (different extensions) define an implicit rule compiling an object file from a Fortran 90 source code file %.o: %.f90 $(F90) $(FFLAGS) -c -o $@ $<

75 Example revisited again OBJ = main.o sub1.o sub2.o sub3.o # implicit rule for compiling f90 files %.o: %.f90 f90 -c -o $@ $< hello: $(OBJ) f90 -o hello $(OBJ) clean: rm hello $(OBJ)

76 Built-in functions GNU make has also built-in functions for a complete list see: strip, patsubst, sort, dir, suffix, basename, wildcard, general syntax $(function arguments)

77 Command line options -j parallel execution -n dry-run shows the command, but does not execute them -p print defaults shows default rules and values for variables before execution -s silent-run do not print commands as they are executed

78 Command line options variables can also be defined from the command line make CC=gcc CFLAGS=-O3 g foobar

79 Complete example SRC = main.f90 sub1.f90 sub2.f90 sub3.f90 OBJ = $(patsubst %.f90, %.o, $(SRC)) F90 = gfortran FFLAGS = DEST = bin # implicit rule for compiling f90 files %.o: %.f90 $(F90) $(FFLAGS) -c -o $@ $< hello: $(DEST)/hello $(DEST)/hello: $(OBJ) $(F90) $(FFLAGS) -o $@ $(OBJ) clean: -rm $(OBJ) -rm $(DEST)/hello # extra dependencies sub2.o: modules.o

80 GNU make The material presented here assumes the use of GNU make (some parts may not work with other implementations of make) For more information see

81 sed + regular expressions Daniel Landau Martti Louhivuori

82 Regular expressions xkcd.com/208/

83 What are regular expressions? way of defining a common text pattern foo-[0-9] : foo-0, foo-1, foo-2,, foo-9 [Dd]og : Dog, dog X[a-z]*Z : XyZ, XabcZ, XZ, XfoZ, extremely useful and powerful tool

84 Regex syntax most characters are taken literally A matches A etc. some characters have special meaning. any character [] delimit a character class ^ beginning of a line $ end of a line OR (i.e. match cat or dog with cat dog) special characters can be made literal by escaping, e.g. \. matches a dot.

85 Regex syntax special shorthands available, too: \w [[:alnum:]] alphanumeric character \s [[:space:]] whitespace \d [[:digit:]] number [[:lower:]] lower case letter [[:upper:]] upper case letter wildcards: * 0 or more times + 1 or more times? 0 or 1 times {n} n times {n,m} at least n but no more than m times

86 Regex syntax character class defines all characters that will match [ab] a or b [A-Z] from A to Z, i.e. upper case letters [ab_-] a or B or _ or [0-9] any number [a-za-z] any letter [^x] anything else than x last word of warning: escaping varies between tools!

87 Regex exercises 1) two functions, a_function and b_function, are defined and used in file.c find the function declaration of a_function using grep the line should look something like 'int a_function( ) {' the function might be called many times, make sure you only find the declaration, not the places where it's called

88 Regex exercises the same problem, except this time you have to find the declaration of b_function 2) in the directory data/ s/ you will find some mega bytes of from the Linux kernel mailing list search for anything formatted like a US style address, i.e. '123 Fake street, New York City NY'

89 Regex resources ressions.html

90 sed sed has lots of commands, only one is discussed here: substitute sed -e 's/cat/dog/' animals.txt substitutes cat for dog once for each line in animals.txt and prints the result to standard output sed -e 's/cat/dog/g' animals.txt substitutes (globally) multiple occurences on the same line

91 sed refer to matching patterns: sed -e 's/\(\w\+\).c/\1.o/' Makefile \(PATTERN\) defines a PATTERN to match \1 refers to the first match sometimes syntax simplified by switching to extended regular expressions: sed -r -e 's/(\w+).c/\1.o/' Makefile

92 sed modify a file in-place: sed -i '.bak' -e 's/cat/dog/' animals.txt modifies animals.txt and copies the old contents to animals.txt.bak

93 sed resources everything: advanced things in blog form:

94 awk Daniel Landau Thomas Zwinger

95 awk Line based processing of column data with c-style programming constructs available Basic pattern: awk '/REGEX/ { commands }' file awk breaks the line into fields delimited by whitespace. Fields are available in variables from $1 onwards awk fields start with a $, but other variables don't

96 awk Special variables: NF contains the number of fields on the current line. NR contains the number of lines so far You can omit the regex, or even use any expression to decide which lines are processed, e.g. awk '{commands}' file for all lines awk 'NF == 5 {commands}' file for just the lines that have five fields awk '$1 == "yes" {commands}' file for lines where the first field is the string yes

97 awk Special words BEGIN and END awk 'BEGIN { begin_commands } { commands } END { end_commands }' The begin_commands are run before all lines and the end_commands after everything For example set sum = 0 at the beginning and print sum in the end

98 awk You can change the field delimiter with -F: awk -F ',' '{ commands }' file awk tutorial: