Table of Contents. Evaluation Copy

Size: px

Start display at page:

Download "Table of Contents. Evaluation Copy"

Godwin White
6 years ago
Views:

1 Table of Contents Perspective grep sort head and tail tr cut od paste split uniq sed gawk more and less tee lp CHAPTER 2: BASH SHELL PROGRAMMING Shells Scripting Rationale Scripting Prerequisites Creating a bash Script bash Startup Scripts A Script s Environment Exporting Variables Exit Status Programming the bash Shell Parameter Passing Operators Decision Making - if Complex Decisions Arithmetic Looping Constructs - for Input and Output - echo and read Looping Constructs - while Functions Interrupts CHAPTER 3: BASH SHELL ARITHMETIC Introduction expr bash Arithmetic Typed Variables v

2 Arrays CHAPTER 4: NETWORKING TOOLS TCP/IP IP Addresses Network Configuration Files Client / Server Computing telnet ping ftp Non-Interactive ftp ssh scp CHAPTER 5: ADMINISTRATIVE TOOLS date adduser chown sudo df du gzip and gunzip tar at crontab yum CHAPTER 6: SOFTWARE TOOLS Building a Linux Utility Creating a Utility The C Compiler make Libraries Shared Libraries APPENDIX A: THE VISUAL (VI) EDITOR vi Modes...A-2 Starting and Stopping vi...a-3 Last Line Mode Commands...A-4 Cursor Movement Commands...A-5 Delete and Search Commands...A-6 vi

3 Chapter 1: Linux Filters 1) Perspective ) grep ) sort ) head and tail ) tr ) cut ) od ) paste ) split ) uniq ) sed ) gawk ) more and less ) tee ) lp

4 Perspective Good software engineering has always been a trademark of any UNIX system. One of the original UNIX philosophies, present in any Linux system, was to provide a large set of software tools (i.e., component utilities), each of which: is well written does one thing does it efficiently To perform simple tasks on the system, use the appropriate tool. To perform complex tasks on the system, use appropriate tools and combine them properly. pipes scripts This strategy works best if each command is able to read the standard input and also write its output to the standard output. Such commands on UNIX and Linux systems are referred to as filters. This section presents the most commonly used Linux filters. 1-2

5 grep The most heralded Linux command is grep, a patternmatching utility used to display lines of a file that match a pattern. The pattern is given as the first argument. The files to be searched are given as subsequent arguments. If no files are given on the command line, grep reads the standard input. A few simple examples display lines from /etc/passwd that match a pattern. $ grep root /etc/passwd root:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/root:/sbin/nologin $ echo $LOGNAME michael $ grep $LOGNAME /etc/passwd michael:x:527:527::/home/michael:/bin/bash grep supports many useful options. Some of them are listed here. -n displays line numbers -c displays a count of lines -i ignores case -l lists file names containing match -v displays unmatched lines 1-3

6 grep $ grep n root /etc/passwd 1:root:x:0:0:root:/root:/bin/bash 12:operator:x:11:0:/root:/sbin/nologin $ echo $LOGNAME michael $ grep n $LOGNAME /etc/passwd 64:michael:x:527:527::/home/michael:/bin/bash $ wc l /etc/passwd 67 /etc/passwd $ grep c /bin/bash /etc/passwd 35 $ grep cv /bin/bash /etc/passwd 32 $ su root Password: # grep l initdefault /etc/* /etc/inittab # exit 1-4

7 grep In addition to simple patterns, grep also allows the use of patterns containing meta-characters - i.e., characters that have special meanings. Since some of these special characters are also special to the shell, they are typically quoted to prevent the shell from expanding them.. match any character $ match at end of line ^ match at beginning of line \ turn off special meaning [ ] alternatives * zero or more of the preceding character Examples: $ grep '^the' file1 # beginning w/ the $ grep 'mike$' file1 # ending w/ mike $ grep '^the$' file1 # containing just the $ grep '\.$' file1 # ending with a period # lines containing a five-letter pattern # beginning with th and ending with e $ grep 'th..e' file1 # lines beginning with at least one digit and # followed by an upper case character $ grep '^[0-9][0-9]*[A-Z]' file1 1-5

8 grep Suppose that you need to find lines in a file containing a phrase. You must quote the pattern so that it will be parsed by the shell as the first argument. $ grep Run xdm /etc/inittab grep: xdm: No such file or directory /etc/inittab:# Run gettys in standard run levels /etc/intttab:# Run xdm in runlevel 5 $ grep "Run xdm" /etc/inittab # Run xdm in runlevel 5 Once you are thoroughly familiar with filters, you will begin to see how useful they are when used in pipelines. Here is a simple example using grep. $ date grep Thu # Thursday? Thu Dec 21 08:07:12 EST 2008 # Yes $ date grep Tue # Tuesday? # No Commands such as those in the previous session can be used in scripts to conditionally control a process. Here is a preview. while date grep Thu do date >> users-log who >> users-log sleep 60 done # If Thursday # date/time # who is logged in # one minute pause 1-6

9 sort sort sorts lines from text files named on the command line or in their absence sorts the standard input. By default, the entire line is used as the key. The examples will use the file named data (in your datafiles directory). $ cat data Michael Saltzman 58 Instructor Patti Ordonez 35 Instructor David Flanagan 37 Instructor Susan Saltzman 41 President Susan Doddridge 40 Technical Robert Knox 61 Sales Tom Vielandi 46 Sales Alan Baumgarten 52 Instructor Maria Gonzales 46 Coordinator Jason Pearlman 22 Instructor Erin Flanagan 29 Administrator Below is a simple example. $ sort data Alan Baumgarten 52 Instructor David Flanagan 37 Instructor Erin Flanagan 29 Administrator Jason Pearlman 22 Instructor Maria Gonzales 46 Coordinator Michael Saltzman 58 Instructor Patti Ordonez 35 Instructor Robert Knox 61 Sales Susan Doddridge 40 Technical Susan Saltzman 41 President Tom Vielandi 46 Sales 1-7

10 sort Various options control the behavior of sort. For example, to sort the data using the second field (and the rest of the line) as the key, use the following. $ sort -k2 data Alan Baumgarten 52 Instructor Susan Doddridge 40 Technical Erin Flanagan 29 Administrator David Flanagan 37 Instructor Maria Gonzales 46 Coordinator Robert Knox 61 Sales Patti Ordonez 35 Instructor Jason Pearlman 22 Instructor Susan Saltzman 41 President Michael Saltzman 58 Instructor Tom Vielandi 46 Sales To sort numerically, say on the third field, use the following. $ sort -nk3 data Jason Pearlman 22 Instructor Erin Flanagan 29 Administrator Patti Ordonez 35 Instructor David Flanagan 37 Instructor Susan Doddridge 40 Technical Susan Saltzman 41 President Maria Gonzales 46 Coordinator Tom Vielandi 46 Sales Alan Baumgarten 52 Instructor Michael Saltzman 58 Instructor Robert Knox 61 Sales Reverse sorts are accomplished with the r option. $ sort -rk2 data $ sort -rnk3 data 1-8

11 sort There is always the chance that there may be "ties" when you are sorting. These ties are normally settled by using the entire line as a "secondary key." However, you can explicitly specify a secondary key as follows. $ sort -k2,2 -k4,4 data Alan Baumgarten 52 Instructor Susan Doddridge 40 Technical Erin Flanagan 29 Administrator David Flanagan 37 Instructor Maria Gonzales 46 Coordinator Robert Knox 61 Sales Patti Ordonez 35 Instructor Jason Pearlman 22 Instructor Michael Saltzman 58 Instructor Susan Saltzman 41 President Tom Vielandi 46 Sales Other useful options to the sort are shown below. -f ignores case -t separates fields -o names the output file To do a reverse sort on the UID field of /etc/passwd: $ sort t: -nrk3 /etc/passwd To perform an "in place" sort: $ sort o data data Sort is often used in pipelines. Shown below is a common idiom. $ ls l /bin sort -nk5 1-9

12 head and tail The head command displays the first ten lines of a file. The exact number of lines is controlled by the first argument. $ head 4 data Michael Saltzman 58 Instructor Patti Ordonez 35 Instructor David Flanagan 37 Instructor Susan Saltzman 41 President The tail command displays the last ten lines of a file. The exact number of lines is controlled by the first argument. $ tail 4 data Alan Baumgarten 52 Instructor Maria Gonzales 46 Coordinator Jason Pearlman 22 Instructor Erin Flanagan 29 Administrator tail has a very useful option that produces all the lines of a file beginning with a particular line. $ tail -n +10 data Jason Pearlman 22 Instructor Erin Flanagan 29 Administrator Another useful feature of tail is to output data to a file as the file grows. This is useful when you are monitoring log files. $ tail f logfile 1-10

13 tr tr is used to translate one set of characters to another or to delete a set of characters. Input must come from the standard input rather than from files named as command line arguments. $ head 2 data tr aeiou AEIOU MIchAEl SAltzmAn 58 InstrUctOr PAttI OrdOnEz 35 InstrUctOr Some sets are more common than others are. Therefore, tr allows special notations in expressing these sets. These special notations should be quoted to prevent possible shell expansion. [0-9] digits [a-z] lower case characters [A-Z] upper case characters [a-za-z] alphabetic characters To translate to upper case: $ head 2 data tr '[a-z]' '[A-Z]' MICHAEL SALTZMAN 58 INSTRUCTOR PATTI ORDONEZ 35 INSTRUCTOR tr is commonly used in a pipeline to perform a small editing job. For example, suppose you wish to display the PATH directories, with one directory name per line. Compare the output for the following two command lines. $ echo $PATH $ echo $PATH tr : '\n' 1-11

14 tr The most commonly used options are shown below. -d deletes this set of characters -s replaces a sequence of characters with one -c complements of a character set To delete certain characters (say digits): $ head 2 data tr -d '[0-9]' Michael Saltzman Instructor Patti Ordonez Instructor To squeeze certain characters (say s and p ): $ echo mississippi tr -s sp misisipi A common use of tr is to determine the words (tokens) of a file. The strategy is to take each character that is not alphabetic and replace it with a newline, and then squeeze the newlines. The following tr idiom does the job. $ head 1 data tr cs '[a-za-z]' '\n' Michael Saltzman Instructor 1-12

15 cut cut reads the standard input or files named on the command line and displays specified characters or fields. c for files whose fields are fixed length -f for files whose fields are variable length The fields or characters to display are denoted by lists whose formats are specified as follows. List Meaning 3 3 3,10,12 3 and 10 and through through the rest of the line -3 1, 2 and 3 Examples: $ head 3 data cut -c1-10 Michael Sa Patti Ordo David Flan $ date cut -c1-3 Fri $ DAY=`date cut -c1-3` $ echo $DAY Fri $ alias day=`date cut -c1-3` $ day Fri 1-13

16 cut cut assumes that the field separator is the tab character. If this is not the case, then you must use the d option to specify the field separator. For example, if you wanted to display only the first (login name) field of /etc/passwd, you would use: $ cut -d: -f1 /etc/passwd head -3 root bin daemon You might wish to display the first and last fields of the file named data. Since that file had a single blank character separating its fields, you would use: $ head 3 data cut -d' ' f1,4 Michael Instructor Patti Instructor David Instructor Note carefully how the blank character had to be protected from shell expansion. 1-14

17 od od is the octal dump command. It provides different views of a file. Suppose there is a file named mydata. $ cat mydata A b 0 1 X $ wc mydata mydata The wc command reports that there are 10 characters in the mydata file. Here are some examples of od to display the characters reported by wc. $ od -a mydata # character A sp b ht 0 sp 1 ht X nl $ od -x mydata # hex a58 $ od -b mydata # octal $ od -cb mydata A b \t 0 1 \t X \n

18 paste paste pastes lines from separate files together as a single file. The tab character is the default field separator in the resultant file. Commonly used options are: -d specify your own separator -s paste one file at a time - file is standard input Examples: $ cat users usera userb $ cat ids $ paste users ids usera 101 userb 102 $ paste d' ' users ids usera 101 userb 102 $ paste s users ids usera userb $ ls file1 file2 file3 file4 file5 file6 ids users $ ls paste -d' ' file1 file2 file3 file4 file5 file6 ids users 1-16

19 split split splits large files into smaller ones. This is useful for treating parts of a file as solo entities. The resultant files are 1000 lines by default. $ wc -l bigfile 2760 bigfile $ split bigfile $ wc -l xaa xab xac 1000 xaa 1000 xab 760 xac 2760 total You can use arguments to determine the size and the names of the output files. $ split -500 bigfile myname $ wc -l myname* 500 mynameaa 500 mynameab 500 mynameac 500 mynamead 500 mynameae 260 mynameaf 2760 total 1-17

20 uniq uniq reports duplicate adjacent lines. One of each set of adjacent identical lines is sent to the standard output. Commonly used options are: -u one copy of the unique lines -d one copy of adjacent identical lines -c counts occurrences of adjacent identical lines Examples: $ cat uniqdata this this this sample sample data this this $ uniq uniqdata this sample data this $ uniq d uniqdata this sample this $ uniq u uniqdata data $ uniq -c uniqdata 3 this 2 sample 1 data 2 this 1-18

21 sed sed is the stream editor. Each of the lines read by sed is sent to the standard output, possibly edited. Below are several simple sed commands. $ sed 1d file1 # delete line 1 $ sed '$d' file1 # delete last line $ sed e 1d e '$d' file1 # delete first and # last lines $ sed 's/hi/aloha/g' file1 # global substitution # of aloha for hi $ sed '/mike/d' file1 # delete all lines # matching mike $ sed '/mike/!d' file1 # delete all lines # not matching mike When sed commands become complex, you may place them in a file and instruct sed to read from that file. $ sed f sedcmds datafile(s) 1-19

22 sed sed uses some special characters so that editing tasks are made easier. The dot character (.) matches any single character. The asterisk character (*) matches one or more of the characters to its left. Therefore, the combination of.* means "as many characters as possible." In particular, you can surround them with other characters such as: 'A.*B' '.* ' # A anything B # blank anything blank Below are some examples: Strip the day from the date command. $ date Wed Sep 21 11:30:29 EDT 2008 $ date sed 's/... //' Sep 21 11:30:29 EDT 2008 Strip extra output from the id command. $ id uid=501(mike) gid=501(mike) groups=501(mike) $ id sed 's/.*//' uid=501(mike) 1-20

23 gawk awk is a pattern scanning and small data base retrieval language written in the mid 1970s by Alfred Aho, Peter Weinberger, and Brian Kernighan. It has been revised several times and now runs under the name of gawk. Despite its complexity, gawk can be simple enough to be used as a command line utility. The basic strategy is to: process one line at a time from its input apply one (or more) gawk commands to each input line. gawk has several built-in variables. NR record number NF number of fields on the input line $0 input line $1 first field of the input line $2 second field $3 third field, etc. gawk allows any number of spaces and/or tabs as field delimiters. In the examples that follow, we will use the previously seen data file as the input file. 1-21

24 gawk $ gawk '{print NR, $0}' data 1 Michael Saltzman 58 Instructor 2 Patti Ordonez 35 Instructor 3 David Flanagan 37 Instructor 4 Susan Saltzman 41 President 5 Susan Doddridge 40 Technical 6 Robert Knox 61 Sales 7 Tom Vielandi 46 Sales 8 Alan Baumgarten 52 Instructor 9 Maria Gonzales 46 Coordinator 10 Jason Pearlman 22 Instructor 11 Erin Flanagan 29 Administrator $ gawk '{print $2, $4}' data Saltzman Instructor Ordonez Instructor Flanagan Instructor Saltzman President Doddridge Technical Knox Sales Vielandi Sales Baumgarten Instructor Gonzales Coordinator Pearlman Instructor Flanagan Administrator 1-22

25 gawk You can print text as well as data. Simply enclose the text inside double quotes. $ head 2 data gawk '{print $1, "is", $3}' Michael is 58 Patti is 35 Each gawk command actually has the following form. 'condition { action }' If the action is omitted, input lines are printed if the condition is true. If the condition is omitted, the action occurs for all lines. Here is a list of selected conditions: pattern /pattern/ built-in function length($0) relational condition $3 > 30 $3 > 30 && $3 <

26 gawk Examples Relational conditional $ gawk '$3 > 50 {print $1,$2,$4}' data Michael Saltzman Instructor Robert Knox Sales Alan Baumgarten Instructor Using a function $ gawk 'length($1) < 5' data Tom Vielandi 46 Sales Alan Baumgarten 52 Instructor Erin Flanagan 29 Administrator Pattern matching $ gawk '$4 ~ /Ins/' data Michael Saltzman 58 Instructor Patti Ordonez 35 Instructor David Flanagan 37 Instructor Alan Baumgarten 52 Instructor Jason Pearlman 22 Instructor If the field separator is anything other than spaces or tabs, then the F option must be used. $ gawk -F: '$3 < 100 {print $1}' /etc/passwd gawk programs can be arbitrarily complex. Rather than writing long command lines, you can place gawk commands into a file and execute them as shown here. $ gawk -f file_of_awk_commands inputfile(s) 1-24

27 more and less more and less are two utilities that allow you to display a file one page at a time. Each command displays the first page of the file and then awaits instructions. Instructions are given in the form of commands that allow for searching, scrolling, and jumping to various places within the file. less is the more recent of the two and therefore has more functionality. Each is typically used: to scroll through a file one page at a time $ more /etc/profile $ ps -A less as the last command in a sequence of piped commands. $ find. more $ ls -l /usr/bin less When you start either command, you can get help by pressing the 'h' key. 1-25

28 tee tee takes its input and does the following. sends it to a file sends it to the standard output When you have a long sequence of commands in a pipeline, you only see the results of the last command in the pipeline. $ tr cs '[a-za-z]' '\n' < input sort uniq c If you wish to capture intermediate results, use the tee command. $ tr cs '[a-za-z]' '\n' < input sort > tee save uniq c In the above example, the file named save would contain all of the processing up to and including the output from sort. You can use the tee command at each stage in the pipe sequence. $ tr cs '[a-za-z]' '\n' < input sort > tee save uniq c tee savewords sort -n In the above sequence, the intermediate processing has been saved: after the sorting after the counting 1-26

29 lp To prevent simultaneous printing by more than one user, Linux uses a client/server printing system. The printer daemon, cupsd, is typically started when your system is booted. It then waits for jobs to be printed. Users use the lp command to send jobs to a well-known spooling directory. Each job is given an identification string consisting of the printer name and a number. $ lp testfile request id is myprinter-154 $ lp thisfile thatfile request id is myprinter-156 (2 files) Many Linux systems have more than one printer. Each printer is given a symbolic name by the system administrator when the printer system is configured. Users can specify the particular printer with the -d option. $ lp -d printername file1 file2 The lpq command gives the status of various printer queues. 1-27

30 Exercises 1. Use the head and tail commands together in a pipeline to produce only lines 5 through 10 from the input data. (Hint: Recall n +NUMBER option to the tail command.) Use cat n /etc/profile to generate the data. 2. The ls a command lists those files that begin with a dot in addition to all other files. Use this idea along with a grep command to list only the files that begin with a dot in your home directory. (Note that Linux provides the 'l.' alias.) 3. How can you tell if the white space between the ls l output fields are blanks or tabs? (Hint: od) 4. Use tr to squeeze these white space characters from an ls l listing, and then use cut to display only the sizes and filenames. 5. Write a command that lists the sizes and names of the ordinary files in the /bin directory in descending order by size. Ordinary files will have a dash in the first column of the ls l output. 6. Use tr, sort, uniq, and head to list the five most frequently used words in a file and their corresponding frequency. (Do not worry about ties.) 7. In the previous example, use tee to save the output before the final five words are displayed. 1-28

31 Exercises 8. List the filename and owner of the files in the /bin directory which have the permission 755 and whose size is greater than 500,000 bytes. (Hint: You will need gawk here.) 9. Use the output of the id command along with sed to display just the uid number. This will require more than one sed command. 10. The file named fourlines (in the datafiles directory) has four-line "records." Write as many commands as you need in order to sort this file on last names. (Hint: Consider splitting the file into four line files and then pasting together lines from the same file. Now you can use the sort utility). This is a hard problem! 11. Sort the mdzips file (in the datafiles directory) in order by city name (rather than by zipcode). 12. Write commands to perform each of the following actions on the people file (in the datafiles directory). Hint: use grep, gawk, cut, and/or sed. Print all lines that contain "Street" Print all lines where the first name begins with "B" Print all lines where the last name begins with "Ker" Print lines 2 through 4 Print phone numbers that are in the 408 area code Print Lori Gortz's name and address Print Ephram's first and last name in capital letters Print lines that do NOT contain a "4" 1-29

32 Exercises Exercise 12 (continued) Print the line containing Wilhelm Kopf, changing Wilhelm to William Print Tommy's birthday (without the year) Print lines where the last field is exactly 5 digits Print the names of all persons born in March 1-30

Table of contents. Our goal. Notes. Notes. Notes. Summer June 29, Our goal is to see how we can use Unix as a tool for developing programs

Table of contents. Our goal. Notes. Notes. Notes. Summer June 29, Our goal is to see how we can use Unix as a tool for developing programs Summer 2010 Department of Computer Science and Engineering York University Toronto June 29, 2010 1 / 36 Table of contents 1 2 3 4 2 / 36 Our goal Our goal is to see how we can use Unix as a tool for developing