UNIX. Contents. Overview of UNIX Structures File systems

Transcription

1 UNIX Contents Overview of UNIX Structures File systems o Pathname o Security Process: o Exit code o IPC: Pipes Shells UNIX Philosophy UNIX s H11istory. Utilities/commands Basic: cp, rm, mkdir, mv, file, wc, chmod Advanced: grep/egrep, sort, find. Shell and shell scripting language 2 Previous lecture 1

2 Shell and Utilities The rest of the operating system Focus of the last 3+1 lectures of this course Cause of debate in Linux community user shell and utilities kernel hardware 3 Files and Processes A file is a collection of data that is usually stored on disk, although some files are stored on tape. When a program is started, it is loaded from disk into RAM. When a program is running, it is called a process. Most processes read and write data from files. MEMORY DISK 4 Inter-Process Communication 2

3 Unix File System Files are just sequences of bytes No file types (data vs. executable) Example of UNIX philosophy (KISS: keep it simple ) Directories are lists of files and other directories, along with their status: creation date permissions, etc. Each directory entry links (points) to a file on the disk. Moving a file does not actually move any (disk) data around. creates link in new location deletes link in old location UNIX provides services for the creation, modification, and destruction of programs, processes, and files. 5 File Pathnames Two files in different directories can have the same name. We need pathnames to differentiate between files with the same names located in different directories. A pathname is a sequence of directory names that leads you through the hierarchy from a starting directory to a target file. gcc /cs/dept/course/ /s/2031/submit/lab6/cse12345/lab6a.c. cat /home/tim/readme.txt Windows DOS / uses \ For example, here s a small hierarchy that contains three files called readme.txt home in three separate directories. bin huiwang readme.txt 6 tim readme.txt readme.txt 3

4 Absolute Pathnames A pathname relative to the root directory is often termed an absolute, or full pathname. /home/huiwang/readme.txt /home/tim/readme.txt /bin/readme.txt ~/readme.txt / home bin readme.txt 7 huiwang readme.txt tim readme.txt From anywhere, cat /home/tim/readme.txt Relative Pathnames A process may also unambiguously specify a file by using a pathname relative to its current working directory. UNIX file system supports the following special fields that may be used when supplying a relative pathname: Field Meaning. current directory.. parent directory Same in DOS 8 cd.. ls.. a.out <../input.txt fopen(../input.txt, r) choose wisely cd../../../../../../../../lab1??? 4

5 Relative Pathnames Relative Pathnames (from /home/huiwang) readme.txt or./readme.txt../tim/readme.txt../../bin/readme.txt / huiwang readme.txt home tim readme.txt bin readme.txt 9 cat../../bin/readme.txt Links A link is a pointer to a file. In fact, in UNIX all filenames (directory entry) are just links to a file. Most files only have one link. -rw-r--r-- 1 jbond cs 154 Feb 4 15:00 letter3 -rw-r--r-- 1 jbond cs 64 Feb 4 15:00 names drwxr-xr-x 2 jbond cs 512 Feb 4 15:00 secret/ Additional links to a file allow the file to be shared. The ln command creates new links. $ ln names NAMES $ ls -l total 8 -rw-r--r-- 2 jbond cs 64 Feb 6 18:36 NAMES -rw-r--r-- 1 jbond cs 154 Feb 4 15:00 letter3 -rw-r--r-- 2 jbond cs 64 Feb 4 15:00 names drwxr-xr-x 2 jbond cs 512 Feb 4 15:00 secret/ 10 Hard link names Names file (on disk) 5

6 File and Directory Permissions UNIX provides a way to protect files based on users and groups. Three types of permissions: read (r) process may read contents of file write (w) process may write contents of file execute (x) process may execute file Three sets of permissions: permissions for owner permissions for group permissions for other -rwxr-xr-- 1 huiwang Same types and sets of permissions as for files apply to directories: read process may read the directory contents (i.e., list files) write process may add/remove files in the directory 11 execute process may open files in directory or subdirectories File Permissions (Security) File permissions are the basis for file security. They are given in three clusters. In the example, the permission settings are rw-r-xr-- : 1 - rw- r-x r-- 1 huiwang faculty 213 Jan 31 00:12 heart.final User (owner) Group Others clusters r w - r - x r - - Each cluster of three letters has the same format: Read permission Write permission Execute permission r w x 12 6

7 Change File Permissions: chmod Only owner and admin can change chmod -R change{, change}* {filename }+ The chmod utility changes the modes (permissions) of the specified files according to the change parameters, which may take the following forms: clusterselection + newpermissions clusterselection - newpermissions clusterselection = newpermissions (add permissions) (subtract permissions) (assign permissions absolutely) 13 where clusterselection is any combination of: u (user/owner) g (group) o (others) a (all) newpermissions is any combination of r (read) w (write) x (execute) s (set user ID/set group ID) The -R option recursively changes the modes of the files in directories. Changing File Permissions: examples Requirement Add group write permission Remove group write permission Remove other s read and write permission Change parameters g+w g-w o-rw Add execute permission for user, group, a+x u+x,g+x,o+x and others. Give the group read permission only. g=r Add write permission for user, and remove group read and write permission. u+w, g-rw For a web file to be accessible, o must has r permission. $ chmod o+r lab7.pdf 14 7

8 Changing Permissions Using Octal Numbers The chmod utility allows you to specify the new permission setting of a file as an octal number. Each octal digit (0~7) represents a permission triplet. binary 1/0 1/0 1/0 r w x For example, if you wanted a file to have the permission settings of rwx r-x --- then the octal permission setting would be 750, calculated as follows: User Group Others setting rwx r-x binary octal Changing Permissions Using Octal Numbers The chmod utility allows you to specify the new permission setting of a file as an octal number. 16 8

9 Changing File Permissions Using Octal Numbers The octal permission setting would be supplied to chmod as follows: $ chmod 750 file # update permissions. $ ls -l file # confirm. - rwx r-x huiwang faculty 4096 Apr 29 14:35 $ _ rwx r - x chmod u=rwx, g=rx 17 An example: setting up submit directory using chmod

10 19 Processes Each command involves a process ls, gedit Unix can execute many processes simultaneously. When a process ends, there is a exit code (return value) associated with the process outcome a non-negative integer 0 means success anything larger than 0 represents various kinds of failure Opposite to C 20 The return value is passed to the parent process For some shells, stored in system variable $? 10

11 Process: Communication Processes can communicate using a number of means: passing arguments, environment read/write regular disk files exit values $? inter-process communication with shared queues, memory and semaphores signals pipes sockets A pipe is a one-way medium-speed data channel that allows two processes on the same machine to talk. If the processes are on different machines connected by a network, then a mechanism called a socket may be used instead. A socket is a two-way high-speed data channel. 21 Unix Pipes A special mechanism called a pipe built into the heart of UNIX to support cascading utilities. A pipe allows a user to specify that the output of one process is to be used as the input to another process. Two or more processes may be connected in this fashion, resulting in a pipeline of data flowing from the first process through to the last. 22 Process 1 Process 2 Process 3 11

12 Pipeline Example A utility called who that outputs an unsorted list of the users, and another utility called sort that outputs a sorted version of its input. $ who $ sort input.txt Use a file (instead of a pipe) Run first program, save output into file Run second program, using file as input $ who > tmp.txt $ sort tmp.txt process 1 process 2 Disadvantages: Unnecessary use of the disk o Slow o Can take up a lot of space MEMORY DISK Pipeline Example A utility called who that outputs an unsorted list of the users, and another utility called sort that outputs a sorted version of its input. MEMORY DISK $ who Use a pipe instead of a file who sort terminal $ sort input.txt These two utilities may be connected together with a pipe so that the output from who passes directly into sort, resulting in a sorted list of users. $ who sort 24 % who sort wc w s T 12

13 Pipe-Equivalent Communication Using a File process 1 process 2 process 3 who > tmp; grep Wang tmp > tmp2; wc -l tmp2 who grep wc terminal who grep Wang wc -l w g w T More examples Process 1 Process 2 Process 3 who sort head - 5 # list fist 5 people in the list who > tmp; sort tmp > tmp2; head -5 tmp2 wc -l classlist cat classlist wc -l grep Wang classlist wc -l cat classlist grep Wang wc -l a.out grep position: a.out grep speed: 26 13

14 Shells The $ or % prompt that you see when you first log in is displayed by a special kind of program called a shell. A Shell is a program that acts as a middleman between you and the raw UNIX operating system. It lets you run programs, build pipelines of processes, save output to files, and run more that one program at the same time. A shell executes all of the commands that you enter. The four most popular shells are: the Bourne shell (sh) the Korn shell (ksh) the C shell (csh) the Bash Shell (bash) user shell and utilities kernel hardware 27 Basic utilities/commands Introduced the following utilities, listed in in groups: General man clear cal date mail (quota v) 28 Directory mkdir -p rmdir cd pwd ls -S -t -r File File print cat lp more lpr head lprm tail lpq cp -r lpstat mv rm -r -i wc -l -c -w chmod chgrp chown newgrp 14

15 Renaming/Moving a file/dir: mv mv -i oldfilename newfilename // rename mv -i oldfilename existingdirectoryname // move mv -i olddirectoryname newdirectoryname // rename mv -i olddirectoryname existingdirectoryname // move rename in DOS move in DOS The first form renames oldfilename as newfilename. The second form moves a collection of files to (under) a directory. The third form renames a directory to newdirectoryname. The fourth form moves the entire directory to (under ) another directory. Another way of mv fileordirname newlocation/newname understanding if no newname => mv fileordirname newlocation o move to newlocation with same name 29 If no newlocation => mv fileordirname newname o move to same location with newname o Actually rename Deleting files: rm del in DOS The rm utility removes a file s label from the hierarchy. rm -fir {filename}* The -i option prompts the user for confirmation before deleting a filename. It is a very good idea to use -i or create a shell alias that translates from rm to rm -i. some files one day you have been warned! Confirm in login shell, but not in other shells If filename is a directory, the -r option causes all of its contents, including sub-directories, to be recursively deleted. Really used for deleting directories

16 In the login shell (tcsh), to be safe, When you issue cp, it is replaced by cp -i When you issue mv, it is replaced by mv -i 31 When you issue rm, it is replaced by rm -i In other shells, should use -i Utilities II advanced utilities Introduces utilities for power users, grouped into logical sets We introduce about thirty useful utilities. Section Utilities 32 Filtering files egrep, fgrep, grep, uniq Sorting files sort Comparing files cmp, diff Archiving files tar, cpio, dump Searching for files find Scheduling commands at, cron, crontab Programmable text processing awk, perl Hard and soft links ln Switching users su Checking for mail biff Transforming files compress, crypt, gunzip, gzip, sed, tr, cut, ul, uncompress Looking at raw file contents od Mounting file systems mount, umount Identifying shells whoami Document preparation nroff, spell, style, troff Timing execution of commands time 16

17 Filtering Files grep, uniq grep, egrep, fgrep -w -i -v Global/Get Regular Expression and Print Filter out, all lines that do not contain a specified pattern, Giving you the line that contains the specified pattern uniq, which filters out duplicate adjacent lines $ cat inputfile.txt # list the file to be filtered line1 Well you know it s your bedtime, line2 So turn off the light, line3 Say all your prayers and then, line4 Oh you sleepy young heads dream of wonderful things, line5 Beautiful mermaids will swim through the sea, line6 And you will be swimming there too. $ grep the inputfile.txt # search for the word the line2 So turn off the light, line3 Say all your prayers and then, line5 Beautiful mermaids will swim through the sea, 33 line6 And you will be swimming there too. Searching for Regex: grep $ grep -w the inputfile.txt # -w: Whole word only -n line number Line 2 So turn off the light, line 5 Beautiful mermaids will swim through the sea, $ grep -v -w the inputfile.txt # -v: reverse the filter. Line 1 Well you know it s your bedtime, line 3 Say all your prayers and then, Line 4 Oh you sleepy young heads dream of wonderful things, Line 6 And you will be swimming there too. We stopped here last time $ grep -i -w the inputfile.txt # ignore case, default case sensitive $ grep -w Wang EECS2031A # who has same family name as me? 34 $ grep -w Wang EECS2031A wc -l # how many? 17

18 Regular Expressions What is a Regular Expression? A regular expression (regex) describes a pattern to match multiple input strings. Regular expressions descend from a fundamental concept in Computer Science called finite automata theory Regular expressions are endemic to Unix Some utilities/programs that use Regex: vi, ed, sed, and emacs awk, tcl, perl and Python grep, egrep Compilers scanf ( %[^/n]s, str); The simplest regular expression is a string of literal characters to match. The string matches the regular expression if it contains the substring

19 Regular Expressions: Exact Matches regular expression cks grep cks inputfile.txt UNIX Tools rocks. match UNIX Tools sucks. match UNIX Tools is okay. 37 no match Regular Expressions: Multiple Matches A regular expression can match a string in more than one place. regular expression apple Scrapple from the apple. match 1 match

20 Regular Expressions: Matching Any Character The. regular expression can be used to match any character. regular expression o. For me to put on. match 1 match 2 39 Regular Expressions: Alternate Character Classes Character classes [] can be used to match any specific set of characters. regular expression b [eor] a t beat a brat on a boat match 1 match 2 match 3 40 [aeiou] will match any of the characters a, e, i, o, or u [kk]orn will match korn or Korn 20

21 Regular Expressions: Negated Character Classes Character classes can be negated with the [^] syntax. regular expression b [^eo] a t beat a brat on a boat match no match 41 scanf ( %[^/n]s, str); Regular Expressions: Other Character Classes Other examples of character classes: [aeiou] will match any of the characters a, e, i, o, or u [kk]orn will match korn or Korn Ranges can also be specified in character classes [1-9] is the same as [ ] [a-e] is equivalent to [abcde] You can also combine multiple ranges [abcde ] is equivalent to [a-e1-9] [a-za-z] all the letters Note that the - character has a special meaning in a character class but only if it is used within a range 42 [-123] would match the characters -, 1, 2, or 3 21

22 Regular Expressions: Named Character Classes Commonly used character classes can be referred to by name alpha, lower, upper, alnum, digit, punct, cntl Syntax [:name:] [a-za-z] [[:alpha:]] [a-za-z0-9] [[:alnum:]] [45a-z] [45[:lower:]] Important for portability across languages For interested students 43 Regular Expressions: Anchors Anchors are used to match at the beginning or end of a line (or both). ^ means beginning of the line ^ bat begin with bat $ means end of the line bat$ end with bat regular expression beat a brat on a boat match ^ b [eor] a t regular expression b [eor] a t $ beat a brat on a boat match 44 grep ^cse EECS2031A ^word$ ^$ 22

23 Regular Expression: Repetitions Kleene Star The * is used to define zero or more occurrences of the single regular expression preceding it. regular expression y a * y I got mail, yaaaaaaaaaay! zero or more occurrences of a (between y y) match regular expression o a * o For me to loo k on. Take oaao zero or more occurrences of a (between o o) 45 match.* match Regular Expressions: Repetition Ranges, Subexpressions Ranges can also be specified {n,m} notation can specify a range of repetitions for the immediately preceding regex {n} means exactly n occurrences {n,} means at least n occurrences {n,m} means at least n occurrences but no more than m occurrences Example:.{0,} same as.* a{2,} same as aaa* If you want to group part of an expression so that * applies to more than just the previous character, use ( ) notation Subexpresssions are treated like a single character a* matches zero or more occurrences of a abc* matches ab, abc, abcc, abccc, # ab followed by 0 or more c (abc)* matches abc, abcabc, abcabcabc, 46 (abc){2,3} matches abcabc or abcabcabc 23

24 Regular Expressions: Repetition Ranges, Subexpressions Some examples Don t get confused with filename wildcard * ls a*.c 47 ls ba*.c Extended Regular Expressions: Repetition Shorthands The * (star) has already been seen to specify zero or more occurrences of the immediately preceding character The? (question mark) specifies an optional character, the single character that immediately precedes it July? will match Jul or July zero or one occurrence o Equivalent to (Jul July) abc?d will match abd and abcd but will not match abccd The + (plus) means one or more occurrence of the preceding character abc+d will match abcd, abccd, or abccccccd but will not match abd 48 24

25 grep and egrep RE Pattern Maning Example c Non-special, matches itself 'tom' \c Turn off special meaning '\$' ^ Start of line '^ab' $ End of line 'ab$'. Any single character '.nodes' [ ] Any single character in [] '[tt]he' [^ ] Any single character not in [] '[^tt]he' R* Zero or more occurrences of R 'e*' R? Zero or one occurrences of R (egrep) 'e?' R+ One or more occurrences of R (egrep) 'e+' R1R2 R1 followed by R2 '[st][fe]' R1 R2 R1 or R2 (egrep) 'the The' anchored repetition 49 grep and egrep RE Pattern Maning Example c Non-special, matches itself 'tom' \c Turn off special meaning '\$' ^ Start of line Don t get confused '^ab' $ End of line 'ab$'. Any single character '.nodes' [ ] Any single character in [] '[tt]he' [^ ] Any single character not in [] '[^tt]he' R* Zero or more occurrences of R 'e*' R? Zero or one occurrences of R (egrep) 'e?' R+ One or more occurrences of R (egrep) 'e+' R1R2 R1 followed by R2 '[st][fe]' R1 R2 R1 or R2 (egrep) 'the The' anchored repetition 50 25

26 grep and egrep RE 51 Pattern Maning Example c Non-special, matches itself 'tom' \c Turn off special meaning '\$' ^ Start of line Don t get confused '^ab' $ End of line 'ab$'. Any single character '.nodes' [ ] Any single character in [] '[tt]he' [^ ] Any single character not in [] '[^tt]he' R* Zero or more occurrences of R 'e*' R? Zero or one occurrences of R (egrep) 'e?' R+ One or more occurrences of R (egrep) 'e+' R1R2 R1 followed by R2 '[st][fe]' R1 R2 R1 or R2 (egrep) 'the The' Don t get confused with UNIX metacharacter (file name wildcards) ls file*.c *.java cp xfile?.c. anchored repetition Regular expression and extended expression maybe confusing. grep may behave differently in different shells. So for this course Use grep -E or egrep Work on Bourne (again) shell (sh/bash) 52 26

27 Searching for Regex: grep $ grep ^cse inputfile.txt # begins with cse -n line number $ grep [0-9]x inputfile.txt # contains digits followed by x $ grep ^[a-z] inputfile.txt # begin with a lower case letter $ grep.nd inputfile.txt # contains one any character followed by nd $ grep [ab]nd$ inputfile.txt # ends with and or bnd $ grep -w Ch[ae]n EECS2031A # has family name Chen or Chan 53 $ grep -w Ch[ae]n EECS2031A wc -l # how many? Utilities II advanced utilities Introduces utilities for power users, grouped into logical sets We introduce about thirty useful utilities. 54 Section Filtering files Sorting files Extracting fields Comparing files Archiving files Searching for files Scheduling commands Programmable text processing Hard and soft links Switching users Checking for mail Transforming files Looking at raw file contents Mounting file systems Identifying shells Document preparation Timing execution of commands Utilities egrep, fgrep, grep, uniq sort cut cmp, diff tar, cpio, dump find at, cron, crontab awk, perl ln su biff compress, crypt, gunzip, gzip, sed, tr, ul, uncompress od mount, umount whoami nroff, spell, style, troff time 27

28 Removing Duplicate Lines: uniq The uniq utility displays a file with all of its identical adjacent lines replaced by a single occurrence of the repeated line. Here s an example of the use of the uniq utility: $ cat animals # look at the test file. cat snake monkey snake dolphin elephant dolphin elephant goat elephant pig pig pig pig monkey pig pig pig 55 $ uniq animals # filter out duplicate adjacent lines. cat snake monkey snake dolphin elephant goat elephant pig pig monkey pig pig pig Removing Duplicate Lines: uniq The uniq utility displays a file with all of its identical adjacent lines replaced by a single occurrence of the repeated line. Here s an example of the use of the uniq utility: $ cat animals # look at the test file. cat snake monkey snake dolphin elephant dolphin elephant goat elephant pig pig pig pig monkey pig pig pig 56 How about un-adjacent lines? $ uniq -c animals # filter out duplicate adjacent lines. 1 cat snake 1 monkey snake 2 dolphin elephant 1 goat elephant 2 pig pig 1 monkey pig 1 pig pig sort and then uniq 28

29 Sort sorts a file in ascending or descending order based on one or more fields. Individual fields are ordered lexicographically, which means that corresponding characters are compared based on their ASCII values. -t field separator (default is blank or tab) -r descending instead of ascending -n numeric sort -f ignore case -M month sort (3 letter month abbreviation) -k key Sort examples $ cat data.txt John Smith Apr 1956 Tony Jones Mar 1950 John Duncan 2 20 Jan 1966 Larry Jones Dec 1946 Lisa Sue Jul 1980 $ sort data.txt # cat data.txt sort John Duncan 2 20 Jan 1966 John Smith Apr 1956 Larry Jones Dec 1946 Lisa Sue Jul 1980 Tony Jones Mar 1950 $ sort -r data.txt Tony Jones Mar 1950 Lisa Sue Jul 1980 Larry Jones Dec 1946 John Smith Apr 1956 John Duncan 2 20 Jan 1966 Whole lines are ordered lexicographically 29

30 sort -k -n $ sort -k3 data.txt # -k (start 0) Tony Jones Mar 1950 John Duncan 2 20 Jan 1966 Lexicographically Lisa Sue Jul 1980 column 3 not sorted John Smith Apr 1956 correctly Larry Jones Dec 1946 $ sort -k3 -n data.txt # -nk3 -nk start 0 John Duncan 2 20 Jan 1966 Tony Jones Mar 1950 John Smith Apr 1956 Lisa Sue Jul 1980 Larry Jones Dec n enables column 3 to be sorted numerically $ sort -k3 -k4 -n data.txt # John Duncan 2 20 Jan 1966 Tony Jones Mar 1950 Lisa Sue Jul 1980 # Lisa and John further sorted John Smith Apr Larry Jones Dec 1946 sort -M $ sort -k5 data.txt # +4-5 John Smith Apr 1956 Larry Jones Dec 1946 John Duncan 2 20 Jan 1966 Lisa Sue Jul 1980 Tony Jones Mar 1950 $ sort -k5 -M data.txt # +4-5 John Duncan 2 20 Jan 1966 Tony Jones Mar 1950 John Smith Apr 1956 Lisa Sue Jul 1980 Larry Jones Dec 1946 Lexicographically Months not sorted correctly -M enables months to be sorted correctly 30

31 Two more examples who sort aboelaze pts/ :26 (6.dsl.bell.ca) farhanieh pts/ :05 (:20) franck pts/ :28 (gradchair.eecs.yorku.ca) franck pts/ :11 (5.cpe.teksavvy.com) fwei pts/ :35 (net.cable.rogers.com) fwei pts/ :42 (net.cable.rogers.com) who sort -k3 farhanieh pts/ :05 (:20) franck pts/ :28 (gradchair.eecs.yorku.ca) franck pts/ :11 (5.cpe.teksavvy.com) fwei pts/ :35 (net.cable.rogers.com) fwei pts/ :42 (net.cable.rogers.com) aboelaze pts/ :26 (6.dsl.bell.ca) 61 Two more examples sort -t cat /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail:/sbin/nologin operator:x:11:0:operator:/root:/sbin/nologin cat /etc/passwd sort -t : -k4 -n # -t :" halt:x:7:0:halt:/sbin:/sbin/halt operator:x:11:0:operator:/root:/sbin/nologin root:x:0:0:root:/root:/bin/bash shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown sync:x:5:0:sync:/sbin:/bin/sync bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin 62 mail:x:8:12:mail:/var/spool/mail:/sbin/nologin 31

32 sort + uniq uniq is a little limited but we can combine it with sort sort uniq -c cat snake monkey snake dolphin elephant dolphin elephant goat elephant pig pig pig pig monkey pig pig pig sort cat snake monkey snake dolphin elephant dolphin elephant goat elephant pig pig pig pig pig pig monkey pig uniq 1 cat snake 1 monkey snake 2 dolphin elephant 1 goat elephant 3 pig pig 1 monkey pig 63 Comparing Files: cmp, diff There are two utilities that allow you to compare the contents of two files: cmp, which finds the first byte that differs between two files diff, which displays all of the differences and similarities between two files Testing for sameness: cmp The cmp utility determines whether two files are the same. $ cat lady1 # look at the first test file. Lady of the night, I hold you close to me, And all those loving words you say are right. $ cat lady2 # look at the second test file. Lady of the night, I hold you close to me, And everything you say to me is right. $ cmp lady1 lady2 # files differ. lady1 lady2 differ: char 48, line 3 $ _ 64 32

33 File Differences: diff The diff utility compares two files and displays a list of editing changes that would convert the first file into the second file. $ diff lady1 lady2 # compare lady1 and lady2. 3c3 < And all those loving words you say are right. > And everything you say to me is right. $ _ $ gcc yourcode; $ a.out > youroutput; $ cmp youroutput sampleoutput; 65 cut deal with fields (columns) -d -f Used to split lines of a file A line is split into fields Fields are separated by delimiters/separators A common case where a delimiter is a space: Default is tab, (not " " ) need to set it for blank -d " " cut -f3 -d" " hello there world 66 delimiter field 3 33

34 $ cat data.txt # assuming tab as delimiter John Smith Apr 1956 Tony Jones Mar 1950 John Duncan Jan 1966 Larry Jones Dec 1946 Lisa Sue Jul 1980 $ cut -f 1 data.txt # show field 1, tab as delimiter John Tony John Larry Lisa $ cut -f 1,3 data.txt John 1222 Tony 101 John 1111 Larry 1223 Lisa 1222 $ cut -f 1-3 data.txt John Smith 1222 Tony Jones 101 John Duncan 1111 Larry Jones 1223 Lisa Sue 1222 find Utility find pathlist expression finds files starting at pathlist finds files descending from there Allows you to perform certain actions e.g., copying (cp), renaming (mv), deleting (rm) the files e.g., copying, Find all the c files and make a backup of them/rename to.bak find. name "*.c" -exec mv {} {}.bak \; Find all the Java class files and delete them find. name "*.class" -exec rm {} \; 34

35 find Utility -name pattern True if file s name matches pattern, which include shell metacharacters *? [ ] -mtime count True if the content of the file has been modified within count days -atime count True if the file has been accessed within count days -ctime count True if the contents of the file have been modified within count days or any of its file attributes have been modified -exec command True if the exit code = 0 from executing the command. command must be terminated by \; If {} is specified as a command line argument, it is replaced by the file name currently matched find examples $ find / -name x.c # search for file x.c in the entire file system $ find. -mtime 14 # lists files modified in the last 14 days in current and subdirectories $ find. -name '*.bak - # "*.bak" search for all bak files $ find. -name 'a?.c' ex # "a?.c" find all a?.c and a1.c a2.c a2.c.bak a3.c a3.c.bak 35

36 find examples -exec $ find / -name x.c searches for file x.c in the entire file system $ find. -mtime 14 -ls lists files modified in the last 14 days $ find. -name '*.bak' -exec rm {} \; # remove all files that end with.bak $ find. -name 'a2.c' -exec cp {} a2.c.bak \; # find a2.c and make a copy called a2.c.bak $ find. -name 'a?.c' -exec mv {} {}.bak \; # find ax.c and then rename it to ax.c.bak $ find. -name '*.c' -exec cp {} {}.2017Su \; # find all c files xx.c and then cp it to xx.c.2017su Transforming Files: tr tr copies the standard input to the standard output with substitution or deletion of selected characters input characters from string1 are replaced with the corresponding characters in string2 (the last character in string2 is used for padding if necessary) tr -cds string1 string2 the -d option results in deletion from the input string of all characters specified in string1 the -s option condenses all repeated characters in the input string to a single character with the -c option, string1 is substituted with its complement; i.e., all characters that are not in the original string

37 Transforming Files with tr: examples $ tr xyz abc this is a x world y z this is a a world b c ^D $ cat afile a lot of space $tr alo xyz < afile # transfer a to x, l to y # and o to z x yzt zf spxce 73 Transforming Files with tr: examples # transfer all lower case character to upper case $ tr [a-z] [A-Z] < afile [:lower:] [:upper:] A LOT OF SPACE # transfer all spaces into dots $ tr " ". < afile a...lot...of...space # transfer all spaces into dots and compact the # dots $ tr " ". < afile tr -s "." a.lot.of.space $ tr 0-9 "*" < EECS2031A #replace digits with * 74 37

38 Utilities II advanced utilities Introduces utilities for power users, grouped into logical sets We introduce about thirty useful utilities. section Filtering files Sorting files Extract fields Comparing files Archiving files Searching for files Scheduling commands Programmable text processing Hard and soft links Switching users Checking for mail Transforming files Looking at raw file contents Mounting file systems Identifying shells Document preparation Timing execution of commands Utilities egrep, fgrep, grep, uniq sort cut cmp, diff tar, cpio, dump find at, cron, crontab awk, perl ln su biff compress, crypt, gunzip, gzip, sed, tr, ul, uncompress od mount, umount whoami nroff, spell, style, troff time Utilities II advanced utilities Regular Expression grep/egrep sort uniq cut awk tr cmp/diff find Start column 1, default delimiter: blank/tab grep -w i ^[Tt]he file123 sort t : -k4 -r -n/m file uniq -c cut -d" " -f2,3 awk F":" {print $1,$2} tr a-z A-Z < filename cmp/diff Start column 1 Default delimiter: tab Start column 1 Default delimiter blank/tab find. name "*.c" -exec cp {} {}.bak \; 38

39 Contents Overview of UNIX Structures File systems o absolute and relative pathname o security -rwx--x--x Process: o has return value 0 (success) or non 0 (sth wrong) o communication: pipes who sort who grep Wang wc -l who wc whoort more Utilities/commands Basic, cp, rm, mkdir, mv, file, wc, chmod Advanced grep/egrep, sort, cut, find, Shell (common shell functionalities) Bourn (again) Shell scripting language 77 UNIX Shells Ch 4 Unix shells UNIX for Programmers and Users Third Edition, Prentice-Hall, GRAHAM GLASS, KING ABLES 39

40 INTRODUCTION A shell is a program that is an interface between a user and the raw operating system. It makes basic facilities such as multitasking and piping easy to use, and it adds useful file-specific features such as wildcards and I/O redirection. There are four common shells in use: the Korn shell the C shell the Bourne shell the Bash shell (Bourne Again Shell) SHELL FUNCTIONALITY This part describes the common core of functionality that all four shells provide E.g., pipe who sort E.g., filename wildcards ls *.c ls a?.c The relationship among the four shells: Korn shell Bourne shell Common core Bourne Again Shell C shell Common core 40

41 SHELL FUNCTIONALITY A hierarchy diagram to illustrate the features shared by the four shells Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution Local Environment Conditional Unconditional Displaying Information : echo The built-in echo command displays its arguments to standard output and works like this: Shell Command: echo {arg}* echo is a built-in shell command that displays all of its arguments to standard output. By default, it appends a new line to the output. use n to fix sh-3.00$ echo hello world hello world sh-3.00$ echo -n hello hellosh-3.00$ # echo hello world 41

42 SHELL OPERATIONS Commands range from simple utility invocations like: $ ls to complex-looking pipeline sequences like: $ cat xfilecompact123 sort uniq cut f 2 head -3 - a command with a backslash(\) character, and the shell will allow you to continue the command on the next line: $ echo this is a very long shell command and needs to \ be extended with the line-continuation character. Note \ that a single command may be extended for several lines. $ _ METACHARACTERS Some characters are processed specially by a shell and are known as metacharacters. All four shells share a core set of common metacharacters, whose meanings are as follow: Symbol Meaning > Output redirection; writes standard output to a file. >> Output redirection; appends standard output to a file. < Input redirection; reads standard input from a file. * File-substitution wildcard; matches zero or more characters.? File-substitution wildcard; matches any single character. [ ] File-substitution wildcard; matches any character between the brackets. CSE1020 lab tour. Don t confuse with RE 42

43 Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution Local Environment Conditional Unconditional Symbol Meaning `command` Command substitution; replaced by the output from command. $ Variable substitution. Expands the value of a variable. & Runs a command in the background. Pipe symbol; sends the output of one process to the input of another. ; Used to sequence commands. % echo hello; wc lyrics Conditional execution; executes a command if the previous one fails. && Conditional execution; executes a command if the previous one succeeds. ( ) Groups commands. # All characters that follow up to a new line are ignored by the shell and program (i.e., used for a comment) \ Prevents special interpretation of the next character. <<tok Input redirection; reads standard input from script up to tok. When you enter a command, the shell scans it for metacharacters and (if any)processes them specially. When all metacharacters have been processed, the command is finally executed. To turn off the special meaning of a metacharacter, precede it by a backslash(\) character. # Also " " ' ' (later) Here s an example: $ echo hi > file # store output of echo in file. $ cat file # look at the contents of file. hi $ echo hi \> file2 # inhibit > metacharacter. hi > file2 # > is treated like other characters. $ cat file2 # look at the file again. Not written ls: cannot access file2: No such file or directory such a file $ echo 3 \* 4 = 12 $ echo = 5 # this is a comment 43

44 Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution Local Environment Conditional Unconditional Now, lets go through the functionalities, as well as their associated metacharacters Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution Local Envirnment Conditional Unconditional Redirection > >> < << The shell redirection facility allows you to: 1) store the output of a process to a file (output redirection) 2) use the contents of a file as input to a process (input redirection) Output redirection To redirect output, use either the > or >> metacharacters. The sequence $ a.out > filename $ echo new line > filename $ echo new line >> filename Difference? sends the standard output of command to the file with name filename. 44

45 Input Redirection Input redirection is useful because it allows you to prepare a process input beforehand and store it in a file for later use. To redirect input, use either the < or << metacharacters. The sequence $ a.out < filename executes command using the contents of the file filename as its standard input. If the file doesn t exist or doesn t have read permission, an error occurs. Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution Local Envirnment Conditional Unconditional FILENAME SUBSTITUTION ( WILDCARDS ) All shells support a wildcard facility that allows you to select files that satisfy a particular name pattern from the file system. The wildcards and their meanings are as follows: ls *.c cp a?.c Wildcard Meaning * Matches any string, including the empty string. ls *.c? Matches any single character. ls a?.c [..] Matches any one of the characters between the brackets. A range of characters may be specified by separating a pair of characters by a hyphen. [a-d] [0-9] Don t confuse with Regulation Expression grep a*b file123.* grep a?.c file123? 45

46 - Prevent the shell from processing the wildcards in a string by surrounding the string with single quotes(apostrophes) or double quotes. -A backslash(/) character in a filename must be matched explicitly. - Here are some examples of wildcards in action: $ ls *.c # list any text ending in.c. a.c b.c cc.c $ ls?.c # list text for which one character is followed by.c. a.c b.c $ ls a*.c a.c $ ls a?.c #? a.c? $ cp /cs/dept/course/ /s/2031/xfile?. $ cp /cs/dept/course/ /s/2031/xfile*. $ cp /cs/dept/course/ /s/2031/xfile[23]. Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution Local Envirnment Conditional Unconditional PIPES - Shells allow you to use the standard output of one process as the standard input of another process by connecting the processes together using the pipe( ) metacharacter. - The sequence $ command1 command2 causes the standard output of command1 to flow through to the standard input of command2. - Any number of commands may be connected by pipes. A sequence of commands changed together in this way is called a pipeline. 46

47 $ head -4 /etc/passwd root:ej2s10rve8mcg:0:1:operator:/:/bin/csh nobody:*:65534:65534::/: daemon:*:1:1::/: sys:*:2:2::/:/bin/csh Or, awk -F: { print $1 } $ cat /etc/passwd cut d '':'' -f 1 sort audit bin daemon glass ls Pipe cut Pipe sort Terminal ingres news nobody root sync sys tim $ cat /etc/passwd grep w Wang wc -l $ head -4 /etc/passwd root:ej2s10rve8mcg:0:1:operator:/:/bin/csh nobody:*:65534:65534::/: daemon:*:1:1::/: sys:*:2:2::/:/bin/csh Or, awk -F: { print $1 } $ cat /etc/passwd cut d '':'' -f 1 sort head -5 audit bin daemon glass ls Pipe cut Pipe sort head ingres news nobody root sync sys tim $ cat /etc/passwd grep w Wang wc -l 47

48 Pipeline Example abacus abacus bottle sort abacus abacus abacus uniq abacus bottle abacus bottle sort uniq wc terminal sort xfile123 uniq wc -l S U W T Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution Local Envirnment Conditional Unconditional COMMAND SUBSTITUTION used very very heavily in script! A command surrounded by grave accents ( ) - back quote - is executed, and its standard output is inserted in the command s place in the entire command line. Any new lines in the output are replaced by spaces. For example: $ echo the date today is date`, right? the date today is Sun Jul 26 08:57:44 EDT 2015, right? $ _ $ echo there are `who wc -l` users on the system there are 3 users on the system 48

49 Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution Local Envirnment Conditional Unconditional COMMAND SUBSTITUTION used very very heavily in script! A command surrounded by grave accents ( ) - back quote - is executed, and its standard output is inserted in the command s place in the entire command line. Any new lines in the output are replaced by spaces. For example: $echo there are cat calsslist wc l` students in the class $ there who are --->,62 look students at the in output the class of who. posey ttyp0 Jan 22 15:31 ( blackfoot:0.0 ) glass $echo has cat ttyp3 classlist Feb grep 3 w 00:41 Wang ( wc bridge05.utdalla -l`students with ) name Wang huynh has 2,students ttyp5 with name Jan Wang 10 10:39 ( atlas.utdallas.e ) $ echo there are who wc -l` users on the system Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution Local Envirnment Conditional Unconditional COMMAND SUBSTITUTION used very very heavily in script! A command surrounded by grave accents ( ) - back quote - is executed, and its standard output is inserted in the command s place in the entire command line. Any new lines in the output are replaced by spaces. Two more examples: $ which mkdir # man which: show the full pathname of shell command /bin/mkdir $ file `which mkdir` # file /bin/mkdir /bin/mkdir: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux , BuildID[sha1]=8cec890564feb596de5a36b1a5321b05a089079f, stripped $x=`wc l classlist` # x get value 62 (talk later) 49

50 Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution Local Envirnment Conditional Unconditional SEQUENCES ; If you enter a series of simple commands or pipelines separated by semicolons, the shell will execute them in sequence, from left to right. This facility is useful for type-ahead(and think-ahead) addicts who like to specify an entire sequence of actions at once. Here s an example: $ date; pwd; ls # execute three commands in sequence. Mon Feb 2 00:11:10 CST 1998 /home/glass/wild a.c b.c cc.c dir1 dir2 $ _ $ gcc yourcode; a.out > output.txt ; cmp output.txt sampleslu.txt Each command in a sequence may be individually I/O redirected $ date > date.txt; pwd > pwd.txt a.c b.c cc.c date.txt dir1 dir2 Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution Local Envirnment Conditional Unconditional Conditional Sequences && - Every UNIX process terminates with an exit code (return value). By convention, an exit value of 0 means that the process completed successfully, and a nonzero exit value indicates failure. (opposite to C) - All built-in shell commands return a value of >=1 if they fail. You may construct sequences that make use of this exit value: 1) For a series of commands separated by && tokens, the next command is executed only if the previous command returns an exit code of 0. i.e., successful 2) For a series of commands separated by tokens, the next command is executed only if the previous command returns a non-zero exit code. fails 50

51 - For example, if gcc compiles a program without fatal errors, it creates an executable program called a.out and returns an exit code of 0; otherwise, it returns a non-zero exit code. $ gcc myprog.c && a.out # if gcc successful, then run a.out $ gcc myprog.c echo compilation failed. # if gcc is not successful, then echo $ grep w Wang classlist && echo found someone in the list return 0 if match, return 1 otherwise GROUPING COMMANDS ( ) - Commands may be grouped by placing them between parentheses, which causes them to be executed by a child shell(subshell). - The group of commands shares the same standard input, standard output, and standard error channels and may be redirected and piped as if it were a simple command. - Here are some examples: $ date; ls; pwd > out.txt # execute a sequence. Mon Jul 10 23:25:26 EDT 2017 # output from date. a.c b.c # output from ls. $ cat out.txt # only pwd was redirected. /home/huiwang $ ( date; ls; pwd ) > out.txt # group and then redirect. $ cat out.txt # all output was redirected. Mon Jul 10 23:25:26 EDT 2017 a.c b.c /home/huiwang $ _ 51

52 Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution Local Environment Conditional Unconditional Background Processing & - If you follow a simple command, pipeline, sequence of pipelines, or group of commands by the & metacharacter, a subshell is created to execute the commands as a background process - The background process runs concurrently with the parent shell and does not take control of the keyboard. - Background processing is therefore very useful for performing several tasks simultaneously, as long as the background tasks do not require input from the keyboard. Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution SHELL PROGRAMS: SCRIPTS Local Environment Conditional Unconditional Any series of shell commands may be stored inside a regular text file for later execution. A file that contains shell commands is called a script. batch file (.bat)in Windows Before you can run a script, you must give it execute permission by using the chmod utility. chmod u+x filename echo hello world date To run it, you need only to type its name. Scripts are useful for storing commonly used sequences of commands, and they range in complexity from simple one-liners to fully blown programs. 52

53 SHELL PROGRAMS: SCRIPTS one for the Bash shell and the other for the Korn shell. $ cat > script.sh # create the bash script. #! /bin/sh # This is a sample sh script. echo "Hello world" echo -n "The date today is " # in sh, -n omits new line date # output today s date ^D # end of input. $ chmod u+x script.sh # make the scripts executable. $ script.sh # execute the shell script. hello world The date today is Sun Jul 09 19:50:00 CST 2017 SHELL PROGRAMS: SCRIPTS one for the Bash shell and the other for the Korn shell. $ cat > script.sh # create the bash script. #! /bin/sh # This is a sample sh script. echo "Hello world" ` ` command substitution echo The date today is `date`. # use command substitution ^D # end of input. $ chmod u+x script.sh # make the scripts executable. $ script.sh # execute the shell script. hello world The date today is Sun Jul 09 19:50:00 CST

54 Shell functions Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution VARIABLES and variable substitution $ - A shell supports two kinds of variables: local and environment variables. local: user defined, positional Local Environment Conditional Unconditional Both kinds of variables hold data in a string format. The child shell gets a copy of its parent shell s environment variables, but not its local variables. Every shell has a set of predefined environment variables that are usually initialized by the startup files. Environment VARIABLES (for your reference) - Here is a list of the predefined environment variables that are common to all shells: Name $HOME $PATH $MAIL $USER $SHELL $TERM Meaning the full pathname of your home directory a list of directories to search for commands the full pathname of your mailbox your username the full pathname of your login shell the type of your terminal 54

55 Built-in local variables (must know!) -several common built-in local variables that have special meanings: Name Meaning $$ The process ID of the shell. $0 The name of the shell script ( if applicable ). $1..$9 $n refers to the n th command line argument ( if applicable ). $* A list of all the command-line arguments. $ myscript paul ringo george john $0 $1 $2 $3 $4 $* #! /bin/sh echo the name of this script is $0 echo the first argument is $1 echo a list of all the arguments is $*. file. x=5 echo the value of variable x is $x $ variable substitution $ script.sh paul ringo george john # execute the script. the name of this script is script.sh the first argument is paul a list of all the arguments is paul ringo george john value of variable x is 5. $ _ 55

56 QUOTING There are often times when you want to inhibit the shell s wildcard-substitution *? [], variable-substitution $, and/or command-substitution ` ` mechanisms. The shell s quoting system allows you to do just that. - Here s the way that it works: 1) Single quotes (' ') inhibits wildcard substitution, variable substitution, and command substitution. 2) Double quotes(" ") inhibits wildcard substitution only. 3) When quotes are nested, it s only the outer quotes that have any effect. QUOTING - The following example illustrates the difference between the two different kinds of quotes: $ echo 3 * 4 = 12 # remember, * is a wildcard. 3 a.c b b.c c.c 4 = 12 $ echo "3 * 4 = 12" # double quotes inhibit wildcards. 3 * 4 = 12 $ echo ' 3 * 4 = 12 ' # single quotes inhibit wildcards. 3 * 4 = 12 another way? $ echo 3 \* 4 = 12 # backslash inhibit a metacharacter 3 * 4 = 12 $ name=graham 56