CS 25200: Systems Programming Lecture 10: Shell Scripting in Bash Dr. Jef Turkstra 2018 Dr. Jeffrey A. Turkstra 1
Lecture 10 Getting started with Bash Data types Reading and writing Control loops Decision making Indexed arrays Basic regular expressions 2018 Dr. Jeffrey A. Turkstra 2
Shells There are many diferent shells available for *NIX-based computers. Some of them even run under that other OS. A shell is basically a command interpreter. It provides an interface between the user and the computer (operating system). Shells may be graphical (explorer.exe, for instance) or text-based - often times called a CLI or command line interface 2018 Dr. Jeffrey A. Turkstra 3
Shells cont When we write a shell script, the first line of the file tells the operating system which shell to use. Some common *NIX shells include: #! /bin/sh Bourne shell #! /bin/csh C-Shell #! /bin/ksh KornShell (more powerful) #! /bin/bash Bourne-Again SHell 2018 Dr. Jeffrey A. Turkstra 4
Bash "Bash is the GNU Project's shell. Bash is the Bourne Again SHell. Bash is an sh-compatible shell that incorporates useful features from the Korn shell (ksh) and C shell (csh). It is intended to conform to the IEEE POSIX P1003.2/ISO 9945.2 Shell and Tools standard. It ofers functional improvements over sh for both programming and interactive use. In addition, most sh scripts can be run by Bash without modification." - https://ww.gnu.org/software/bash/ 2018 Dr. Jeffrey A. Turkstra 5
Diving in - variables Declaration and definition: NAME="whatever" with no spaces between or after the "=" Accessed as $NAME,, or better, ${NAME} Examples: MagicVariable1=7 MagicVariable2="Hello!" MagicVariable3=3.1415 MagicVariable4=${MagicVariable1} 2018 Dr. Jeffrey A. Turkstra 6
Special Bash "variables" Bash has some built-in variables that allow us to get at some important information $# Number of command line arguments $* Command line arguments $0 The name of the shell script $$ Current process ID number $? Return value from last executed command $1 to $N Command line parameters (as separated by whitespace) 2018 Dr. Jeffrey A. Turkstra 7
echo command echo <options> Arguments Data is not formatted Writes its arguments, separated by blanks (spaces) and ending with a newline, to standard output. Whitespace can be protected with double or single quotes as needed. Example: $ echo Hello, World! Hello, World! $ 2018 Dr. Jeffrey A. Turkstra 8
Common echo options See man page for more detail. -n do not add a newline at the end -e turn on the special meaning of the backslash \ character -E turn of backslash (default) 2018 Dr. Jeffrey A. Turkstra 9
Examples echo -n "Hello" echo ", World!" echo "-n hmm" echo ":-)\n:-(" echo -e ":-)\n:-(" Results in the following output: Hello, World! -n hmm :-)\n:-( :-) :-( 2018 Dr. Jeffrey A. Turkstra 10
printf command - formatted output Almost like in C: printf "format" list of variables Format string is like that found in C The list of variables should be separated by spaces. Example: A=3.45 S="Big Deal" printf "A: %5.2f, S: %s\n" $A "$S" 2018 Dr. Jeffrey A. Turkstra 11
More Bash variables Untyped variables can be used to hold string and integer values Output: #! /bin/bash Output: A="Big Deal" Big Deal echo $A 3 A=3 2.3 echo $A A=2.3 echo $A exit 0 2018 Dr. Jeffrey A. Turkstra 12
Typed variables Integers typeset -i variable_name Integers Cannot be used to store strings 2018 Dr. Jeffrey A. Turkstra 13
Example #! /bin/bash typeset -i A A=3 echo "A: $A" A="Oh Well" echo "A: $A" A=2.4 echo "A: $A" exit 0 Results in the following output: A: 3 X1: line 5: Oh Well: syntax error in expression (error token is Well ) A: 3 X1: line 7: 2.4: syntax error: invalid arithmetic operator (error token is.4 ) A: 3 2018 Dr. Jeffrey A. Turkstra 14
Floating point variables Bash does not support foating point values or math. :-( 2018 Dr. Jeffrey A. Turkstra 15
Constants We can create read only variables too typeset -r Name="Ffej" typeset -r K=8 Or how about a read only integer typeset -ri K=8 2018 Dr. Jeffrey A. Turkstra 16
Example #! /bin/bash typeset -r Name="Ffej Artskrut" echo $Name Name="Big Deal" echo $Name exit 0 Results in the following: $ C Ffej Artskrut./C: line 4: Name: readonly variable Ffej Artskrut 2018 Dr. Jeffrey A. Turkstra 17
for loops Two diferent syntaxes in Bash for variable in list do commands done where list is a set of strings separated by whitespace for (( initial_expression; loop_condition; loop_expression )) do commands done do must be on the 2 nd line in both cases! 2018 Dr. Jeffrey A. Turkstra 18
Examples for I in 1 2 3 4 5 do echo -n ${I} done for (( I = 1; I < 6; I++ )) do echo -n ${I} done Both result in the same output: 12345 2018 Dr. Jeffrey A. Turkstra 19
Examples cont #! /bin/bash ls *.c > temp:$$ touch temp:$$ for File in $(cat temp:$$) do lp -dsomeprinter ${File} done rm -f temp:$$ exit 0 2018 Dr. Jeffrey A. Turkstra 20
while loop Note that a command is considered "true"" as long as its exit status is zero.. This is at times backwards from what you might be used to. while <command exit status is true (0)> do <whatever commands you need to do the job> done 2018 Dr. Jeffrey A. Turkstra 21
Example #! /bin/bash typeset -i I=0 while (( I < 10 )) do echo -n " ${I}" (( I = I + 1 )) done echo exit 0 Outputs: 0 1 2 3 4 5 6 7 8 9 2018 Dr. Jeffrey A. Turkstra 22
read command read can be used to obtain user input. Eg, echo -n "Feed me data: " read DataVar echo "You fed me ${DataVar}!" It may also be used to read from other streams such as a fle. Everyone always forgets that the name of the file goes at the end of the while loop 2018 Dr. Jeffrey A. Turkstra 23
reading from a fle #! /bin/bash cat InFile echo while read A B C do echo "A: ${A} " \ "B: ${B} " \ "C: ${C} " done < InFile exit 0 2018 Dr. Jeffrey A. Turkstra 24
Output This is the output generated from the cat command, which simply displays a file's contents to stdout: This is Neat! CS 252 is Great! Ffej is the best. 1 2 3 4 5 6 Big Deal 1234 This is the output generated by the echo statement inside of the while loop: A: This B: is C: Neat! A: CS B: 252 C: is Great! A: Ffej B: is C: the best. A: 1 B: 2 C: 3 4 5 6 A: Big B: Deal C: A: 1234 B: C: 2018 Dr. Jeffrey A. Turkstra 25
Bash math Bash only has integer math. Use let or (( )) to isolate mathematical statements Basic math operators include: addition (+),( subtraction (-),( multiplication (*),( division (/), and modulus (%)( 2018 Dr. Jeffrey A. Turkstra 26
Integer math example #! /bin/bash typeset -i a=11 typeset -i b=3 typeset -i x (( x = a + b )) echo "(( x = $a + $b )) x = $x" (( x = a - b )) echo "(( x = $a - $b )) x = $x" (( x = a * b )) echo "(( x = $a * $b )) x = $x" (( x = a / b )) echo "(( x = $a / $b )) x = $x" (( x = a % b )) echo "(( x = $a % $b )) x = $x" exit 0 2018 Dr. Jeffrey A. Turkstra 27
Output (( x = 11 + 3 )) x = 14 (( x = 11-3 )) x = 8 (( x = 11 * 3 )) x = 33 (( x = 11 / 3 )) x = 3 (( x = 11 % 3 )) x = 2 2018 Dr. Jeffrey A. Turkstra 28
Branching if command true then commands else commands fi Note: the command is "true" if it returns 0 - neat! 2018 Dr. Jeffrey A. Turkstra 29
Nested if statements As one might expect, it is possible to nest branches in ksh if (( A < B )) then echo "A < B" else if (( A == B )) then echo "A = B" else echo "A > B" fi fi 2018 Dr. Jeffrey A. Turkstra 30
elif statement Equivalent to "else if" in C if (( A < B )) then echo "A < B" elif (( A == B )) then echo "A = B" else echo "A > B" fi 2018 Dr. Jeffrey A. Turkstra 31
Testing Tests of any sort should be surrounded by double brackets with spaces: [[ space whatever space ]] Example if [[ -r MyFile ]] then echo MyFile is readable! fi Tests include logical comparisons, string comparisons, and file permission tests 2018 Dr. Jeffrey A. Turkstra 32
File testing -a file exists -d is a directory -f is an ordinary file -r is readable -s has non-zero length -w is writable -x is executable and lots more! reverse the test 2018 Dr. Jeffrey A. Turkstra 33
Arithmetic comparison Comparisons between numbers should always be surrounded by double parentheses: (( whatever )) For example: if (( 7 <= 5 )) then echo "The world is at an end!" fi Arithmetic comparisons include: == equal >= greater than or equal > greater than <= less than or equal < less than!= not equal 2018 Dr. Jeffrey A. Turkstra 34
String tests Equality, if [[ string1 = string2orpattern ]] if [[ string1 == string2orpattern ]] if [[ string1!= string2orpattern ]] Lexicographical ordering, if [[ string1 < string2 ]] if [[ string1 > string2 ]] Emptiness, if [[ -n string1 ]] # string is not NULL if [[ -z string1 ]] # string is NULL 2018 Dr. Jeffrey A. Turkstra 35
Example #! /bin/bash # if (( $#!= 1 )); then echo "Usage: $0 <filename>" exit 1 fi File="$1" if [[! -f "${File}" ]]; then echo "File: ${File} is not an ordinary file" else echo "File: ${File} is an ordinary file" fi exit 0 2018 Dr. Jeffrey A. Turkstra 36
Output $ File_Check Usage: File_Check <filename> $ File_Check x File x is not an ordinary file $ File_Check File_Check File File_Check is an ordinary file 2018 Dr. Jeffrey A. Turkstra 37
Arrays in Bash Unlike C, bash supports sparse arrays ArrVar[5]=8 ArrVar[15]=12 ArrVar[19]=7 If the indices aren't consecutive, how do we know the array's size? How do we know the indices for all values? 2018 Dr. Jeffrey A. Turkstra 38
Special operators # and! You can obtain a list (a string of whitespace-separated values) of every element in an array: echo ${ArrVar[*]} echo ${ArrVar[@]} The size of an array can be found by using the # operator: ${#ArrVar[*]} or ${#ArrVar[@]} The array subscripts can be found by using the! operator: ${!ArrVar[*]} or ${!ArrVar[@]} 2018 Dr. Jeffrey A. Turkstra 39
Indexed array example #! /bin/bash A[5]=34 A[1]=3 A[2]=56 A[100]=89 echo "Size of array: ${#A[*]}" echo "Array indices: ${!A[*]}" for I in ${!A[*]} do echo "A[${I}]=${A[I]}" done exit 0 2018 Dr. Jeffrey A. Turkstra 40
Indexed array output Size of array: 4 Array indices: 1 2 5 100 A[1]=3 A[2]=56 A[5]=34 A[100]=89 2018 Dr. Jeffrey A. Turkstra 41
Read array example #! /bin/bash echo "Data_File:" cat Data_File # Remember, cat just dumps the file's echo # contents to stdout echo "Formatted output:" while read -a Data # Split on whitespace do # (spaces and tabs) for (( I = 0; I < ${#Data[*]}; I++ )) do printf "%6.2f" ${Data[I]} done echo done < Data_File exit 0 2018 Dr. Jeffrey A. Turkstra 42
Output Data_File: 1 2 3 77 12 12.6 6.8 7 2 1.0-3 -5.5 Formatted output 1.00 2.00 3.00 77.00 12.00 12.60 6.80 7.00 2.00 1.00-3.00-5.50 2018 Dr. Jeffrey A. Turkstra 43
More on loops Similar to C, bash has continue n Used to stop the execution of the innermost n loops and then continue with the next loop. The default is n = 1. break n Used to end the execution of the innermost n loops. Default is n = 1. 2018 Dr. Jeffrey A. Turkstra 44
Examples #! /bin/bash for (( I = 0; I <= 4; I++ )); do if (( I == 1 )); then continue fi echo -n " ${I}" done echo exit 0 Results in the following output: 0 2 3 4 2018 Dr. Jeffrey A. Turkstra 45
Examples cont #! /bin/bash I=0 while (( I <= 4 )); do if (( I == 1 )); then break fi echo -n " ${I}" (( I++ )) done echo exit 0 Results in the following: 0 2018 Dr. Jeffrey A. Turkstra 46
What about arguments? What if we want to loop through the command line arguments? Even if we know how many there are, we still can't use a loop construct #! /bin/bash for (( I = 0; I < $#; I++ )); do echo $what??? done 2018 Dr. Jeffrey A. Turkstra 47
shift n Used to left shift the parameters on the command line n places Default is n = 1 $0 is never changed Often used when an unknown number of parameters are passed, or for looping through a large number of parameters. 2018 Dr. Jeffrey A. Turkstra 48
Example #! /bin/bash echo '$0 -- ' $0 echo '$# -- ' $# X=0 while (( $#!= 0 )); do (( X = X + 1 )) echo "\"\$${X}\" was $1" shift done exit 0 Sample run $ parameters q "1 2 3" xyz $0 -- parameters $# -- 3 "$1" was q "$2" was 1 2 3 "$3" was xyz 2018 Dr. Jeffrey A. Turkstra 49
Trouble with quotes There are various kinds of quotes, and each one can mean something diferent in ksh. ' The single forward quote character " The double quote character ` The back quote character \ The backslash character (sometimes used to escape quotes) 2018 Dr. Jeffrey A. Turkstra 50
The single forward quote ' Must appear in pairs Protects all characters between the pair of quotes Ignores all special characters Protects whitespace 2018 Dr. Jeffrey A. Turkstra 51
Single quote examples Path='/b/cs252' echo The path for cs252 is $Path Displays: The path for cs252 is /b/cs252 echo 'The path for cs252 is $Path' Displays: The path for cs252 is $Path echo 'The book costs $2.00' Displays: The book cost $2.00 2018 Dr. Jeffrey A. Turkstra 52
Wildcard * exception #! /bin/bash ls echo * echo '*' yuk='*' echo $yuk echo '$yuk' exit 0 Produces the following output: Chap-1 TESTS TV memo x y Chap-1 TESTS TV memo x y * Chap-1 TESTS TV memo x y $yuk 2018 Dr. Jeffrey A. Turkstra 53
The double quote " Must come in pairs Protects whitespace Does not ignore Dollar signs - $ Back quotes - ` Backslashes - \ 2018 Dr. Jeffrey A. Turkstra 54
Double quote example Path="/b/bee264" echo "The path for ee264 is $Path" Yields: The path for ee264 is /b/ee264 echo "The book cost \$2.00" Yields: The book cost $2.00 2018 Dr. Jeffrey A. Turkstra 55
Wildcard * exception #! /bin/bash ls echo * echo "*" yuk="*" echo $yuk echo "$yuk" exit 0 Produces the following output: Chap-1 TESTS TV memo x y Chap-1 TESTS TV memo x y * Chap-1 TESTS TV memo x y * 2018 Dr. Jeffrey A. Turkstra 56
The back quote ` Used to issue a UNIX command and obtain its output... #! /bin/bash echo "Current directory is `pwd`" DIR=`pwd` echo "Directory is ${DIR}" exit 0 Outputs: Current directory is /a/turkstra/252 Directory is /a/turkstra/252 2018 Dr. Jeffrey A. Turkstra 57
$(command) "Better" way to issue a UNIX command and obtain its output Makes your code easier to read, easier to debug #! /bin/bash echo "Current working directory is $(pwd)" DIR=$(pwd) echo "Directory is ${DIR}" exit 0 Results in Current directory is /a/turkstra/252 Directory is /a/turkstra/252 2018 Dr. Jeffrey A. Turkstra 58
The backslash \ Used to remove any special meaning that a symbol may have. \$1.00 "escapes" the $ character, treating it as a literal $ instead of $1 (the first argument passed to the script) Used to add special meaning to symbols. \n for newline, for example. If it is the last symbol on a line, it will act as a continuation indicator. 2018 Dr. Jeffrey A. Turkstra 59
Backslash example #! /bin/bash echo "This item costs \$2.00" echo -n "This is line 1, " echo "this is the rest of the line" echo "This is" \ "\"$(whoami)\"" # \" used to print " exit 0 Outputs This item costs $2.00 This is line 1, this is the rest of the line This is "turkstra" 2018 Dr. Jeffrey A. Turkstra 60
Whitespace protection #! /bin/bash DATA=$(cat $0) echo ${DATA} echo echo "${DATA}" exit 0 Displays: #! /bin/ksh DATA=$(cat $0) print ${DATA} print print "${DATA}" exit 0 #! /bin/ksh DATA=$(cat $0) print ${DATA} print print "${DATA}" exit 0 2018 Dr. Jeffrey A. Turkstra 61
grep command Used to search files for lines of information. Many, many fags - see the man page. grep -flags regular_expression filename Useful fags -x Exact match of line -i Ignore upper/lower case -c Only count the number of lines which match -n Add relative line numbers -b Add block numbers -v Output all lines which do not match 2018 Dr. Jeffrey A. Turkstra 62
Simple regular expressions Regular expressions express patterns. They are used to find and/or extract pieces of information from a string.. Matches any character ^ Start of line $ End of line \ Escape character [list] Matches any character in the list [^list] Matches any character not in the list * Match zero or more occurrences of the previous regular expression \{min,max\} Matches at least min and at most max occurrences of the previous regular expression 2018 Dr. Jeffrey A. Turkstra 63
Examples grep "^string$" file_name collects all lines which contain only string grep " " file_name collects all lines which have any three characters surrounded by spaces grep " [0-9]\{1,3\} " file_name collects all lines containing a sequence of one to three digits surrounded by spaces grep "^x*[abc]" file_name collects all lines which start with zero or more x's followed by a single a, b, or c 2018 Dr. Jeffrey A. Turkstra 64
More examples Let's pretend we have a file named data 12 12 345 567 3 abd asdf And this script #! /bin/bash # begins with 1 or 2 grep "^[0-9]\{1,2\} " data # digits followed by exit 0 # a space We should get this output 12 345 3 abd 2018 Dr. Jeffrey A. Turkstra 65
-v option, inverting the match Let's pretend we have a file named data 12 12 345 567 3 abd asdf And this script #! /bin/bash grep -v "^[0-9]\{1,2\} " data exit 0 We should get this output 12 567 asdf 2018 Dr. Jeffrey A. Turkstra 66
-c option, counting the matches Let's pretend we have a file named data 12 12 345 567 3 abd asdf And this script #! /bin/bash grep -c "^[0-9]\{1,2\} " data exit 0 We should get this output 2 2018 Dr. Jeffrey A. Turkstra 67
-n option, adding line numbers Let's pretend we have a file named data 12 12 345 567 3 abd asdf And this script #! /bin/bash grep -n "^[0-9]\{1,2\} " data exit 0 We should get this output 2:12 345 4:3 abd 2018 Dr. Jeffrey A. Turkstra 68
Using grep inside a script #! /bin/bash if (( $#!= 1 )); then echo "Usage: $0 <user_id>" exit 1 fi USER="$1" if grep "${USER}" Id_File > /dev/null then echo "Bad way: ${USER} in file" else echo "Bad way: ${USER} not in file" fi if grep "^${USER}$" Id_File > /dev/null then echo "Good way: ${USER} in file" else echo "Good way: ${USER} not in file" fi exit 0 2018 Dr. Jeffrey A. Turkstra 69
Output $ cat Id_File $ Check sam Usage: Check <user_id> maryann $ Check jeff john Bad way: jeff in file jeff Good way: jeff in file jeffrey $ Check son bill Bad way: son in file william Good way: son not in file peterson 2018 Dr. Jeffrey A. Turkstra 70
Questions? 2018 Dr. Jeffrey A. Turkstra 71