Shell Script Programming 2 David Morgan Useful capabilities parameter processing validation usage checking user input custom functions filenames decomposition, tempnames, random names action hinged on compound condition evaluation (ifless if) sourcing execute commands from a file in current shell #include functions in such a file, by reference debugging 1
Supplemental supporting utilities find generate qualified file name lists xargs build command lines from stdin sed dynamic, programatic, in-place editing awk text manipulation bc arbitrary precision math capabilities lacking or weak in shell often utilized in scripts Special shell variables $0 script name from command line $1, $2, etc command line parameters $* - command line parameters collectively $$ - process ID (PID) of current process $? exit status of last command 2
shift positional parameter promoter getopts getopts vs getopt a shell built-in a stand-alone binary not identical processes command-line options 3
Command-line format command [ - options ] [ arguments ] ls -l m* tokens Some options take a value, some commands take arguments command [ - options ] [ arguments ] myprog [ -x value ] [-y] [ aaa ] [ bbb ] [ ccc ] an option (-d ) taking a value ( : ) a command ( cut ) taking an argument ( /etc/passwd ) 4
getopts command resides in a script knows the script s allowable options parses the command line that called the script executed successively, finds/returns the options in the line one at a time used in loops to read/detect them one after the other detect phase normally followed by programmed response phase getopts documentation (from man bash ) getopts optstring name [args] getopts is used by shell procedures to parse positional parameters. optstring contains the option characters to be recognized; if a character is followed by a colon, the option is expected to have an argument, which should be separated from it by white space. The colon and question mark characters may not be used as option characters. Each time it is invoked, getopts places the next option in the shell variable name, ini- tializing name if it does not exist, and the index of the next argument to be processed into the variable OPTIND. OPTIND is initialized to 1 each time the shell or a shell script is invoked. When an option requires an argument, getopts places that argument into the variable OPTARG. The shell does not reset OPTIND automatically; it must be manually reset between multiple calls to getopts within the same shell invocation if a new set of parameters is to be used. When the end of options is encountered, getopts exits with a return value greater than zero. OPTIND is set to the index of the first non-option argument, and name is set to?. getopts normally parses the positional parameters, but if more arguments are given in args, getopts parses those instead. getopts can report errors in two ways. If the first character of optstring is a colon, silent error reporting is used. In normal operation diagnostic messages are printed when invalid options or missing option arguments are encountered. If the variable OPTERR is set to 0, no error messages will be displayed, even if the first character of optstring is not a colon. If an invalid option is seen, getopts places? into name and, if not silent, prints an error message and unsets OPTARG. If getopts is silent, the option character found is placed in OPTARG and no diagnostic message is printed. If a required argument is not found, and getopts is not silent, a question mark (?) is placed in name, OPTARG is unset, and a diagnostic message is printed. If getopts is silent, then a colon (:) is placed in name and OPTARG is set to the option character found. getopts returns true if an option, specified or unspecified, is found. It returns false if the end of options is encountered or an error occurs. 5
functions install runnable code unit in memory under a callable name A shell function stores a series of commands for later execution. When the name of a shell function is used as a simple command name, the list of commands associated with that function name is executed. Functions are executed in the context of the current shell; no new process is created to interpret them (contrast this with the execution of a shell script). bash man page functions 6
filenames decomposition basename dirname temporary names random names Filename decomposition 7
Generating working/temp filenames date based Generating working/temp filenames RANDOM variable, mktemp command in /tmp in some other directory 8
if-less if if [... ]; then xxx fi [... ] && xxx seen often understood seldom a (too?) slick way to do conditionals if-less if Some examples found in rc.sysinit AND-based righthand action only if lefthand condition true: [ -x /usr/bin/plymouth ] && PLYMOUTH=yes [! -f "$file" ] && continue [ "$READONLY" = "yes" ] && return 1 [ -f /var/log/dmesg ] && mv -f /var/log/dmesg /var/log/dmesg.old OR-based righthand action only if lefthand condition not true: [ -f /.autorelabel ] touch /.autorelabel 9
source execute code from a file in current shell often used on a file containing functions effectively similar to c language #include source filename [arguments] Read and execute commands from filename in the current shell environment bash man page find searches for files in a directory tree described by an expression expression consists of elements options tests actions each element returns boolean result find evaluates as many elements of its expression as needed to know expression s outcome 10
Most common use for <all files in a set of files> next if <something about the file> do <something with the file> but the operation details are more complex than that find example expression find. -maxdepth 1 -size +1000000c -print an option a test an action find files 1) in the current directory (no subdirectory search) 2) bigger than a million bytes 3) and print their names 11
Some example elements find. options* tests actions maxdepth name print mount atime +n ls etc size +n exec executable ok type etc empty false etc * find s options, not shell command options Some example elements find. options tests actions what it returns: true always true or false true or false what it does: influence nothing their particular find behavior action 12
Operational logic "[evaluates] the given expression from left to right... until the outcome is known (the left hand side is false for and operations, true for or), at which point find moves on to the next file name. - find man page Operational logic because -name A* is false for B* files because printing happens before -name A* evaluation from the 2 nd -print (2 nd print doesn t happen for B* files) 13
exec action arbitrary response for qualifying files needs to be terminated with ; uses {} as placeholder for current file need to escape these from shell a finder script command: find. -type f exec grep l $1 {} \; print names of all files in current directory containing a given string String manipulation 14
sed stream editor used for search and replace filters standard input to standard output sed stream editor prints 2 lines replace 1 st o with *, each line all o s replace any h or e replace any vowel replace any letter or numeral replace anything neither h nor e replace anything neither letter nor numeral 15
awk (or gawk) a pattern scanning and processing language better text processing facilities than shell s often used in scripts to break text into fields gawk gawk <program> <input files> { <pattern> <action> } gawk processes all lines in the input, comparing each to the pattern and, for those that match, performing the action 16
gawk the program the pattern the action pattern only: default action is to print whole line action only: default pattern selects all lines bc arbitrary precision calculator strong computation features operates internally in decimal interactive offers command prompt programmable weak programming features complements shell (weak comp, strong prog) use with shell pipe commands to bc from shell result returns to shell from bc 17
bc expressions high integer precision high integer precision bc how shell does it (no decimals) high decimal precision how bc does it high decimal precision decimal periodicity = 1 decimal periodicity = 2 decimal periodicity = 6 18
bc 7, 97, 983 and other primes have special properties (see cyclic numbers ) periodicity = 7 1 = 6 high integer precision periodicity = 97 1 = 96 periodicity = 983 1 = 982 bc - functions enter function code at bc prompt utilize it thereafter 19
bc function library files mod( ) no such function but here s a file that defines such a function (plus others) file s contained functions are now available bc code from standard input things you do in bc you can have done by bc in the shell 20
quick RSA tutorial with suitably chosen values for e, n, and d * an integer m encrypts to: and can be recovered by: c= m e mod n c d mod n a set of suitable values (ones that work): e 15941 n 48863 d 17741 * pair {e,n} is called public key and pair {d,n} private key encrypt and recover 1001 in bc 21
encrypt and recover 1001 in shell script use command substitution where the command is bc use here doc to drop in entire bc prog any bc code that returns a desired calculated value 22
Looping helpers debugging scripts sectional debugging turn on with set x, off with set +x within script whole-script debugging set x at command line before running script shabang line in script: #!/bin/bash -x 23
Resources Advanced Bash-Scripting Guide, An in-depth exploration of the art of shell scripting; Mendel Cooper GNU manuals: find: http://www.gnu.org/software/findutils/manual/find.html grep: http://www.gnu.org/software/grep/manual/ sed: http://www.gnu.org/software/sed/manual/sed.html bc: http://www.gnu.org/software/bc/manual/bc.html Classic Shell Scripting Arnold Robbins and Nelson A. F. Beebe, O'Reilly & Associates, 2005 UNIX Shells by Example Ellie Quigley; Prentice Hall, 4th edition, 2005 Learning the bash Shell Cameron Newham and Bill Rosenblatt, O'Reily & Associates, 3rd edition, 2005 Introducing Regular Expressions Michael Fitzgerald, O'Reilly & Associates, 2012 Mastering Regular Expressions Jeffrey E. F. Friedl, O'Reilly & Associates, 3rd edition, 2006 Sed and Awk Dale Dougherty, O'Reilly & Associates, 1997 The AWK Programming Language, Alfred Aho, Brian Kernighan, Peter Weinberger, Addison-Wesly Publishing Company, 1988 24