Shell Programming. Erwin Earley Sr. @Zend

Size: px

Start display at page:

Download "Shell Programming. Erwin Earley Sr. @Zend"

Asher Phillips
6 years ago
Views:

About Me 39 Years in the Industry Worked with multiple of *NIX variants including AT&T System V Release 3, DGUX, SUN/OS, Linux Worked with many different hardware platforms including PDP 11/70, AT&T

AS/400 in 2001 Linux instructor as Adjunct Professor for University of MN in early 2000s RHCE certified LPIC level 1 & 2 certified Mirantis Openstack Professional certification Headed up OSS center

4 About Me 39 Years in the Industry Worked with multiple of *NIX variants including AT&T System V Release 3, DGUX, SUN/OS, Linux Worked with many different hardware platforms including PDP 11/70, AT&T 3B2/400, MainFrame, AS/400, PC/XT, PC/AT, Started out in development including Model 204, PL/1, Pascal, C Open Source Software (OSS) Background Test team lead for original enablement of Linux on AS/400 in 2001 Linux instructor as Adjunct Professor for University of MN in early 2000s RHCE certified LPIC level 1 & 2 certified Mirantis Openstack Professional certification Headed up OSS center of competency for the iseries Technology Center (precursor to IBM Lab Services) Lead consultant on OSS team in IBM Lab Services Also worked with Database, Quality Assurance, Technical Sales Support, Customer Enablement IBM technical advocate for initial roll-out of PHP enablement on IBM i Still working on wife 1.0 Have 3 Children 2 Married Sons 1 Daughter never getting married! Newest Edition to the team!! 2016 Rogue Wave Software, Inc. All Rights Reserved. 4

6 But before we begin Some Useless Trivia The Story of the Linux Penguin Linus Torvalds has claimed in various posts to have been attacked by a "ferocious penguin that bit me and infected me with a little known disease called penguinitis. "Penguinitis makes you stay awake at nights just thinking about penguins and feeling great love towards them" Torvalds later said that while he did indeed have a close and personal encounter with a penguin, the bird wasn't ferocious, and just timidly nibbled on Torvald's finger. Many thought a strong, smart and fierce beast would be the best representation for Linux. Sharks, foxes, eagles, and hawks were discussed on the Linux forums. Torvalds spoke up and said he thought that a penguin would be the perfect Linu logo but not just any penguin, Torvalds specifically wanted to steer away from the "macho penguin" image, and instead urged the developers to focus on creating an image of a cute, cuddly and contended penguin. Contented meaning, in Torvald's words that "the penguin had just stuffed itself on herring" So when you think penguin, Torvalds told the developers, "you should be imagining a slightly overweight penguin, sitting down after having gorged itself, and having just burped. It's sitting there with a beatific smile the world is a good place to be when you have just eaten a few gallons of raw fish and you can feel another burp coming." Rogue Wave Software, Inc. All Rights Reserved. 6

that get called There are some built in commands Unlike CL there are a number of varieties of shell sh bourne shell csh c shell ksh korn shell

7 What is a Shell? The command line used on Unix systems Like CL it can be used interactively, or run as a program Like CL most commands are actually programs that get called There are some built in commands Unlike CL there are a number of varieties of shell sh bourne shell csh c shell ksh korn shell bash bourne again shell qsh Q shell There are some similarities and some differences Most of the discussion here is not operating system specific Will work on AIX, Linux, QSH in OS/400, other nasty Unix variants, etc 2016 Rogue Wave Software, Inc. All Rights Reserved. 7

8 Why do we care about the shell? All system configuration operations can be done through the shell often more quickly then through a GUI It may be the only environment available in the case of a system crash Shell scripts can automate most routine tasks such as backups, scheduled s, etc. GUI can be used for a great amount of admin activities However, the shell tends to be a comfort zone providing ability to fix things in case something goes wrong 2016 Rogue Wave Software, Inc. All Rights Reserved. 8

Different Types of Shells A number of shells are available each providing

include: BASH (Borune Again Shell) PDKsh (Public Domain Korn Shell) csh (C

scripting capabilities and user interface Items such as command recall and

9 Different Types of Shells A number of shells are available each providing function/usability customized to a particular type of user: Popular shells include: BASH (Borune Again Shell) PDKsh (Public Domain Korn Shell) csh (C shell) mc (Midnight Commander) QSHELL (PASE shell) Difference tends to be in scripting capabilities and user interface Items such as command recall and file name completion are typically different 2016 Rogue Wave Software, Inc. All Rights Reserved. 9

10 Login Shell vs. Non-Login Shell Login Shell: Shell executed when user logs in to the system. Typically performs general setup initializes the terminal, sets some variables, etc. Non-login shell: Either subshells (started from the login shell), shells started by the GUI desktop, or disconnected shells started by a command Remember that shell is simply another command on the system Login shells unlike Non-login shells read a series of configuration files: Login Shell: On login if /etc/profile, source it if ~/.bash_profile exists, source it else if ~/.bash_login exists, source it if ~/.profile exists, source it Non Login Shell: On startup if ~/.bashrc exists, source it On exit if ~/.bash_logout exists, source it 2016 Rogue Wave Software, Inc. All Rights Reserved. 10

11 11 Starting with bash bash stands for Bourne Again Shell Developed by Brian Fox in 1987 One of the most popular shells available in Linux Bash incorporates features of the Korn and C shell (ksh and csh) Bash configuration files: /bin/bash Bash executable /etc/profile System wide initialization file for login shells ~/.bash_profile Personal initialization file for login shells ~/.bashrc Personal per-interactive-shell startup file ~/.bash_logout Login shell clean file that executes when shell exits 2016 Rogue Wave Software, Inc. All Rights Reserved. 11

12 A little more on BASH Default Linux shell This can be changed in a variety of ways /etc/profile login shell Magic string within the script (more on this in a moment) Very powerful as a command line shell Recall previous commands Command and file completion with the <TAB> key May programming features Loops and conditionals 2016 Rogue Wave Software, Inc. All Rights Reserved. 12

13 13 Shell Environment The shell is an environment where commands can be entered and the Operations system can respond to them A key concept to the environment is environment variables There are a large number of environment variables HISTFILE: points to file containing the shell history, defaults to ~/.bash_history HISTFILESIZE: how man last commands you wish to have in history HOME: points to your home directory PATH: set of directories to search when trying to execute a command PS1: Prompt variable USER: username 2016 Rogue Wave Software, Inc. All Rights Reserved. 13

14 Shell metacharacters Symbol Meaning > Output redirection >> Output redirection (append) < Input redirection * File substitution wildcard; zero or more characters? File substitution wildcard; one character [ ] File substitution wildcard; any character between brackets `cmd` Command substitution $(cmd) Command substitution The pipe : Command Sequence OR conditional execution && AND conditional execution Symbol Meaning ( ) Group commands & Run command in the background # Comment $ Expand the value of a variable \ Prevent or exape interpretation of the next character << Input redirection " $val " Literal with variable substitution ' $val ' Literal without variable substituion 2016 Rogue Wave Software, Inc. All Rights Reserved. 14

15 15 Useful Shell Constructs Arrow Up & Down: Scroll through recent commands used &&: command is only executed if preceding command was successful: command1 && command2 alias: sets a command alias or prints defined aliases alias wrklnk=ls bg[jobid]: Resumes the suspended job in the background cd: changes current directory to directory indicated cd /home 2016 Rogue Wave Software, Inc. All Rights Reserved. 15

16 16 Useful Shell Constructs (continued) echo: Outputs the arguments echo hello world find [path][expression]: searches the directory indicating looking for files that match expression: find / -name passwd print pwd Prints the absolute pathname of the current working directory unalias Removes an alias history displays command history with line numbers umask User file creation mask logout: exits the shell environment exit [n]: exits shell environment with exit status n 2016 Rogue Wave Software, Inc. All Rights Reserved. 16

18 qsh A version of a shell for System i Has many of the same features as bash RTFM This is

19 19 The First Shell Program #!/bin/bash echo Hello World This is called a shell script The first line #!/bin/bash is a magic string When a file is loaded if the first two characters are #! then the remainder of the line is used as the program to run the file #!/home/erwin/my_nice_program is a valid shell script echo is a built-in command that just displays the rest of the line 2016 Rogue Wave Software, Inc. All Rights Reserved. 19

20 20 Creating a shell program Use a text editor Traditional Unix editors include vi and emacs More user friendly editors include nano On System i use edtf 2016 Rogue Wave Software, Inc. All Rights Reserved. 20

21 Writing Your Shell Script Using nano 2016

22 22 Running Your Shell Script ls l test.sh -rw-r r 1 earley earley :54 test.sh earley@testbed:~$ test.sh -bash: test.sh: command not found earley@testbed:~$ We have a test.sh program in our directory Lets try and run it Doesn t work! Why not!?!? 2016 Rogue Wave Software, Inc. All Rights Reserved. 22

23 23 Running Your Shell Script echo $PATH /home/earley/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/ usr/bin:/sbin:/bin:/usr/bin/x11 test.sh Hello World The program is not in the path The current directory is not in the path Can run a shell script by typing. Means load the file into the current shell 2016 Rogue Wave Software, Inc. All Rights Reserved. 23

24 24 Running Your Shell Script mv test.sh ~/bin/ echo $PATH /home/earley/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/ usr/bin:/sbin:/bin:/usr/bin/x11 test.sh -bash: /home/earley/bin/test.sh: Permission denied Move the shell script into a directory in the path Try and execute it again Now we get Permission denied! Why!?!? 2016 Rogue Wave Software, Inc. All Rights Reserved. 24

25 25 Running Your Shell Script ls l ~/bin/test.sh -rw-r--r-- 1 earley earley :54 /home/earley/bin/test.sh earley@testbed:~$ chmod u+x ~/bin/test.sh earley@testbed:~$ ls l ~/bin/test.sh -rwx--r-- 1 earley earley :54 /home/earley/bin/test.sh earley@testbed:~$ test.sh Hello World earley@testbed:~$ The file is not marked with executable permissions chmod u+x says make it executable by the user who owns it NOW it works 2016 Rogue Wave Software, Inc. All Rights Reserved. 25

28 28 Running a shell script in QSH Can also get into qsh via 'STRQSH' Keep in mind that you could also simply ssh into your IBM I system (assuming the ssh daemon is running) and run your scripts 2016 Rogue Wave Software, Inc. All Rights Reserved. 28

29 Running a shell script in QSH 2016 Rogue

30 30 When good scripts go bad If you get Command not found the path to the shell is probably wrong Another error occurs if you move a file from a Windows machine to Linux and the line endings get messed up If this happens, you can use the following command: tr d \r < erwin.sh > erwin2.sh we will understand that line more later 2016 Rogue Wave Software, Inc. All Rights Reserved. 30

31 31 Input and Output Unix programs start with three open files Input (called stdin) (#0) Output (called stdout) (#1) Error output (called stderr) (#2) Our shell program uses the echo command to write to stdout We can redirect the output to go to a file by using > ls l > output.txt This will take the output of the ls l command and write it into a file called output.txt 2016 Rogue Wave Software, Inc. All Rights Reserved. 31

32 32 Input and Output You can also specify which output goes to a file ls l 1> output.txt this is the same as before. Redirecting only error output grep fred * 2> grep.err this will redirect only the error output to the file grep.err 2016 Rogue Wave Software, Inc. All Rights Reserved. 32

33 33 Input and Output Use the < operator to redirect input Equivalent to typing at the keyboard For example sed s/erwin/erwin/g < my.txt This runs the command sed (an editor) changing erwin to Erwin and taking its input from a file called my.txt Input and output sed s/erwin/erwin/g < my.txt > my2.txt 2016 Rogue Wave Software, Inc. All Rights Reserved. 33

34 34 Common *nix commands Unix is built around the idea of lots of little programs that all do one thing well ls lists files sed an editor grep a searcher cat a file outputer find a file locator sort sorts files tr translates characters ps list processes seq print a sequence of numbers a thousand others Help for all these is in the man (for manual ) command E.g. man sed Shell programs generally involve stringing lots of these together Which leads us to pipes All of these except 'seq' are available in the IFS and can be run from a IBM I shell (i.e., qsh, ssh session) 2016 Rogue Wave Software, Inc. All Rights Reserved. 34

36 36 Variables Bash (both interactively and in programs) has variables Variables do not need to be declared, and don t have types #!/bin/bash STR="Hello World!" echo $STR 2016 Rogue Wave Software, Inc. All Rights Reserved. 36

37 37 Special Variables $1 is the first argument, $2 is the second argument, etc $* is all the arguments $# is the number of arguments $? Is the status of the last command 2016 Rogue Wave Software, Inc. All Rights Reserved. 37

38 38 Command variables $( command) can be used like a variable, but executes the command FILES=$(ls) Sets the variable $FILES to the output of the ls command Use the let command to evaluate expressions COUNTER=0 COUNTER=$COUNTER+1 echo $COUNTER # This prints 0+1 COUNTER=0 let COUNTER=COUNTER+1 echo $COUNTER # This prints Rogue Wave Software, Inc. All Rights Reserved. 38

39 Interacting with a User Enter Your Name Shell script gets a name from user input and echoes it to the screen #!/bin/bash echo "Enter Your Name: " read NAME echo "Your name is $NAME" exit Rogue Wave Software, Inc. All Rights Reserved. 39

41 41 If statements Standard if/then/else Note the weird syntax involving [ ] and ; #!/bin/bash if [ "foo" = "foo" ] ; then echo expression evaluated as true else echo expression evaluated as false fi 2016 Rogue Wave Software, Inc. All Rights Reserved. 41

42 42 Conditions [ ] [ ] is a synonym for the test command man test gives the options [ -z $x ] the length of the string is zero [ STRING1 = STRING2 ] the strings are equal [ STRING1!= STRING2 ] the strings are not equal [ INTEGER1 -eq INTEGER2 ] INTEGER1 is equal to INTEGER2 [ INTEGER1 -ge INTEGER2 ] INTEGER1 is greater than or equal to INTEGER2 [ INTEGER1 -gt INTEGER2 ] INTEGER1 is greater than INTEGER2 [ INTEGER1 -le INTEGER2 ] INTEGER1 is less than or equal to INTEGER2 [ INTEGER1 -lt INTEGER2 ] INTEGER1 is less than INTEGER2 [ INTEGER1 -ne INTEGER2 ] INTEGER1 is not equal to INTEGER2 [ -d FILE ] FILE exists and is a directory [ -e FILE ] FILE exists [ -f FILE ] FILE exists and is a regular file 2016 Rogue Wave Software, Inc. All Rights Reserved. 42

43 43 If statements and variables The example makes more sense with a variable #!/bin/bash if [ "$myvar" = "foo" ]; then echo expression evaluated as true else echo expression evaluated as false fi 2016 Rogue Wave Software, Inc. All Rights Reserved. 43

44 Example Script Simple file copy script with a if/then/else loop #!/bin/bash #Name: Copy File echo "Enter a Filename to copy:" read SRC echo "Enter the filename to copy to:" read DEST if cp $SRC $DEST 2>/dev/null then echo "Task Complete" else echo "Copy Failed!" > &2 exit 1 fi exit Rogue Wave Software, Inc. All Rights Reserved. 44

45 45 Loops for: very different from other programming languages. Loops over a series of 'words' within a string. while: loops while a condition is true until: almost identical to the while loop, except that the code is executed while the control expression evaluates to false. #!/bin/bash for i in $( ls ); do echo item: $i done 2016 Rogue Wave Software, Inc. All Rights Reserved. 45

46 46 Loop Examples #!/bin/bash for i in `seq 1 10`; do echo $i done #!/bin/bash COUNTER=0 while [ $COUNTER -lt 10 ]; do echo The counter is $COUNTER let COUNTER=$COUNTER+1 done 2016 Rogue Wave Software, Inc. All Rights Reserved. 46

47 Case Statement The case statement can be used to replace nested if/then/else statements case EXRESSIOON in pattern1) statements ;; pattern2) statements ;;... esac NOTES: EXPRESSION is expanded and checks are made against each pattern (in order) When a match is found all associated statements are executed (execution terminates with ;;) After first match, case terminates exit status is status of last command executed If no match, exit status of case is zero 2016 Rogue Wave Software, Inc. All Rights Reserved. 47

48 case statement simple example #!/bin/bash echo -n "Do you agree with this? [yes or no]: " read yno case $yno in [yy] [yy][ee][ss] ) echo "Agreed" ;; [nn] [n N][O o] ) echo "Not agreed, you can't proceed the installation"; exit 1 ;; *) echo "Invalid input" ;; esac 2016 Rogue Wave Software, Inc. All Rights Reserved. 48

50 50 Functions with arguments Arguments use the same $1, $2 syntax as the main program #!/bin/bash function quit { exit } function mysay { echo $1 } mysay Hello mysay World quit echo foo 2016 Rogue Wave Software, Inc. All Rights Reserved. 50

51 51 Putting Text Inline; or Here documents Can redirect input from inside a shell script using the << operator Treats any text until the TAG as if it is input #!/bin/bash mail s "Test Mail" $1 << END_OF_MAIL Hello $1, This is some spam. Thanks for reading END_OF_MAIL 2016 Rogue Wave Software, Inc. All Rights Reserved. 51

52 52 Setting Exit Status Shell scripts return a status value exit 0 exit -1 You can also test the status of a command you call using the special $? Variable or by using the command in an if statement directly #!/bin/bash cp $1 /tmp/backup/ if [ $? == 0 ] then echo "Copy of $1 succeeded" else echo "Copy of $1 failed" fi 2016 Rogue Wave Software, Inc. All Rights Reserved. 52

54 54 A Sample (and useful) backup script #!/bin/bash if [ -z "$1" ]; then echo usage: $0 directory exit fi SRCD=$1 TGTD="/var/backups/" OF=home-$(date +%Y%m%d).tgz tar -czf $TGTD$OF $SRCD 2016 Rogue Wave Software, Inc. All Rights Reserved. 54

55 55 File Level Backup to the IFS An Example Scripts can be implemented to perform file-level backup to the IFS This facilitates the backup/restore of files form Linux without having to take the Linux server down (which would be required for a full SAV of the storage space). One method would be to have the script find all files that have been modified over a defined period, build a backup set and then write the backup set out to the IFS Additional scripts could then be developed to list the contents of the backup set and retrieve (restore) files from the backup set The following slides present some simple script examples 2016 Rogue Wave Software, Inc. All Rights Reserved. 55

56 56 File Level Backup to the IFS an Example Backup Script #!/bin/sh ################################### # Script Name: backup.sh ################################### # Environment Variables ################################### PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/s bin:/usr/bin:/usr/x11r6/bin:/root/bin SOURCE= /home TARGET= linux.backup.`date +%d^b^y` DAYS=1 HOST= USER= qsecofr PASSWD= packers FTPDST= /home/linux.backup/$target # Following find statement builds the cpio set of modifies files find $SOURCE mtime -$DAYS print cpio ocbv F /tmp/$target 2> /dev/null 2016 Rogue Wave Software, Inc. All Rights Reserved. 56

57 57 File Level Backup to the IFS an Example Backup Script # Following statements use ftp to copy the cpio set to the i5/os partition ftp n $HOST << END_SCRIPT quote USER $USER quote PASS $PASSWD bin put /tmp/$target $FTPDST quit END_SCRIPT exit Rogue Wave Software, Inc. All Rights Reserved. 57

58 58 File Level Backup to the IFS an Example List Script #!/bin/sh ###################################### # Script Name: list.sh ###################################### # Environment Variables ###################################### HOST= USER= qsecofr PASSWD= packers ################################################ # First lets get a list of the save sets that # # are available. # ################################################ list.files.sh 2> /dev/null grep linux.backup tr s cut f6 d > /tmp/linux.backup.save.sets 2016 Rogue Wave Software, Inc. All Rights Reserved. 58

59 59 File Level Backup to the IFS an Example List Script ###################################################################################### # The following statements check the usage form of the command. If the command did # # not include the file name then a usage statement is output along with a list of # # backup sets available on the server. # ###################################################################################### if [ $# -eq 0 ] then echo Usage: $0 backup-file-name echo echo Backup files are located in the /home/linux.backup echo directory of the IFS of the OS/400 parttion echo echo Files follow the name convention of linux.backup.ddmmmyy echo Current files available echo cat /tmp/linux.backup.save.sets exit 0 fi 2016 Rogue Wave Software, Inc. All Rights Reserved. 59

60 60 File Level Backup to the IFS an Example List Script if test `grep c $1 /tmp/linux.backup.save.sets` -eq 0 then echo Save set $1 does not exist echo echo Current files available echo cat /tmp/linux.backup.save.sets exit 0 fi ###################################################### # The following statements get the requested backup # # set from the server. # ###################################################### TARGET=$1 FTPDST= /home/linux.backup/$target ftp n $HOST << END_SCRIPT quote USER $USER quote PASS $PASSWD bin get $FTPDST /tmp/$target quit END_SCRIPT 2016 Rogue Wave Software, Inc. All Rights Reserved. 60

61 61 File Level Backup to the IFS an Example List Script ##################################################3############ # The following command lists the contents of the backup set. # ############################################################### cpio itcvb F /tmp/$target 2>/dev/null cut -bytes= rm /tmp/$target rm /tmp/linux.backup.save.sets exit Rogue Wave Software, Inc. All Rights Reserved. 61

62 62 File Level Backup to the IFS an Example List.Files Script #!/bin/sh #################################################3 # Script Name: list.files.sh ################################################## # Environment Variables ################################################## HOST= USER= qsecofr PASSWD= packers ############################################################### # The following commands use ftp to build a list of save sets # # available in the IFS # ############################################################### ftp n $HOST << END_SCRIPT quote USER $USER quote PASS $PASSWD cd /home/linux.backup ls quit END_SCRIPT exit Rogue Wave Software, Inc. All Rights Reserved. 62

63 63 File Level Backup to the IFS an Example Restore Script #!/bin/sh ################################################ # Script Name: restore.sh ################################################ # Environment Variables ################################################ HOST= USER= qsecofr PASSWD= packers ####################################################################################### # The following statements test the usage of the command. If the command was called # # with the incorrect number of arguments, a usage statement is output along with a # # list of save sets available in the IFS. # ####################################################################################### If test $# -lt 2 then echo Usage $0 backup-file-name name-of-file-to-restore echo echo Backup files are located in the /home/linux.backup echo directory of the IFS of the OS/400 partition echo echo Files follow the naming convention of linux.backup.ddmmmyy echo Current files available echo list.files.sh 2>/dev/null grep linux.backup tr s cut f6 d exit 0 fi 2016 Rogue Wave Software, Inc. All Rights Reserved. 63

64 64 File Level Backup to the IFS an Example Restore Script ########################################################################## # The following statements retrieve the request backup set from the IFS # ########################################################################## TARGET=$1 FILE=$2 FTPDST=/home/linux.backup/$TARGET echo $TARGET ftp n $HOST << END_SCRIPT quote USER $USER quote PASS $PASSWD bin bet $FTPDST /tmp/$target quit END_SCRIPT ###################################################################################### # The following statement retrieves the specified file from the retrieved backup set # ###################################################################################### cpio ivcbum $FILE < /tmp/$target rm /tmp/$target exit Rogue Wave Software, Inc. All Rights Reserved. 64

65 65 WebMin Can Be Customized to Launch Your Scripts Launchers can be developed in WebMin that execute Linux scripts Creative solutions can be implemented for a number of functions 2016 Rogue Wave Software, Inc. All Rights Reserved. 65

67 More information qsh Read the book bash man bash

com Other useful links http://www.linuxcommand.

68 A Moment of Fun Linux and Open Source Quotes the Linux philosophy is "laugh in the face of danger". Oops. Wrong one. "Do it yourself". That's it (Linus Torvalds) Linux: the choice of a GNU generation ('93 Tshirt) Linux hackers are funny people: They count the time in patch levels (seen in a.sig) What happens when you read some doc and either it doesn't answer your question or is demonstrably wrong? In Linux, you say "Linux sucks" and go read the code. In Windows/Oracle/etc you say "Windows sucks" and start banging your head against the wall" (Denis Vlaskenko in ikml) Those who don't understand Linux are doomed to reinvent it, poorly. (Unidentified source) It's a bird, it's a plane...no, it's KernelMan, faster than a speeding bullet to your rescue. Doing new kernel versions in under 5 seconds flat (Linus, in the announcement for ) Rogue Wave Software, Inc. All Rights Reserved. 68

Review of Fundamentals

Review of Fundamentals 1 The shell vi General shell review 2 http://teaching.idallen.com/cst8207/14f/notes/120_shell_basics.html The shell is a program that is executed for us automatically when we log