Unix Course Notes Part 2

Transcription

1 Unix Course Notes Part 2 Chris Bayliss and Alan Reed (Version 5.05)

2 Chris Bayliss and Alan Reed

3 Table of Contents Part 1: Security - File and Directory Access 5 chmod command 5 chmod by numbers 7 Part 2: Strings, Patterns, Regular Expressions and grep 8 strings 8 patterns 9 regular expressions 9 grep 10 Part 3: More about shells 11 Command line recall and editing 11 ksh Programming 11.profile 13 More redirection 14 Part 4: Finding Files 16 find 16 Part 5: Utilities 18 sort 18 uniq 18 Combining files - cat and paste 19 cut 20 wc 20 Part 6: Topics not covered - sed, awk, perl and vi 21 sed 21 awk 21 perl 21 vi 21 Part 7: Processes and how to kill them 22 Processes 22 ps 22 fg 23 kill 23 Chris Bayliss and Alan Reed

4 Appendix I: ASCII characters, strings, patterns, and regular expressions 24 ASCII characters 24 strings 24 patterns 25 regular expressions 25 Expression Meaning 25 Standards used in the document Where you have to do something is shown in text like this: type the following text However, the text shown in non-bold is the prompt text from Unix. The text in bold is the text you have to type. prompt text text you type When using combination keys (ctrl, shift etc) these are shown as follows: control+h this means that you have to hold the Control key down and then press once the letter H (while still holding the control key). Chris Bayliss and Alan Reed

5 Part 1: Security - File and Directory Access chmod command As stated in the first course, there are three main types of access which can be applied to a file, and these can be applied selectively to yourself, your groups or everyone else. These types of access are called file modes. The ls -l command can be used to display the file mode. The chmod command is used to change the file modes. The chmod command is used to change the modes (access permissions) associated with a file. The change is specified in the form: chmod modes filename The mode is three characters which represent who will have their access changed, the operation (whether the access is to be added, subtracted or set) and what access permission is being referred to. who is affected is specified by one of: u user the owner of the file g group the group to which the owner belongs o other everyone else a all u, g and o operation is specified by one of: + add the specified permission - subtract the specified permission = assign the specified permission, ignoring the current setting and access permission is specified by one or more of: r w x Read Write execute (or list if a directory) Note that in order to have access to a file, the user concerned also needs list access (x) to the directory which contains the file, and all directories above it up to and including root. You will need to be aware of this if you set up files which are to be read by others, for example World Wide Web pages. In the examples below the chmod command is given together with the old and new modes for the files concerned. Chris Bayliss and Alan Reed Page 5

6 Ex.1 Given a file file.one with the default modes (read and write for the user), add read mode for the group: Type ls -l file.one the access modes should be as follows:- user group others Before: rw Type chmod o+r file.one Type ls -l file.one again, the access modes have now changed to:- After: rw- --- r-- Ex. 2 Remove read and write access for everyone else from file file.two. Type ls -l file.two the access modes should be as follows:- user group others Before: rwx rw- rw- chmod o-rw file.two After: rwx rw- --- Ex. 3 Give everyone (owner, group and others) read, write and execute permissions on file file.three Before: user Group others irrrelevant chmod a=rwx file.three After: rwx Rwx rwx thing to do! WARNING: Giving write access to other (everyone) is a risky Chris Bayliss and Alan Reed Page 6

7 chmod by numbers File access can be set using numbers rather than letters. The access is specified by three digits, the first being for the User, the second for the Group, and the third for the Others. Each digit is calculated by adding the numbers associated with read, write and execute access. The associated numbers are as follows: 4 Read 2 Write 1 execute (or search in the case of a directory) Chmod will then take the numbers as an argument instead of letters as above. In this way, a particular mode can be set instead of adding or subtracting from specific modes. For example: chmod 777 file4 would set read, write and execute modes for user, group and others. You would not normally wish to give others write access. chmod 755 file4 would set read, write and execute for user, and just read and execute for group and others. chmod 644 file4 would set read and write for user, and just read for group and others. Chris Bayliss and Alan Reed Page 7

8 Part 2: Strings, Patterns, Regular Expressions and grep strings There is a detailed treatment of strings, patterns and regular expressions in the appendix. This section covers a basic description with a few examples. These are sequences of ASCII characters, not just letters. These are taken as arguments to various commands. As special characters and spaces can be included in strings, it is sometimes necessary to quote them, as spaces would normally separate different arguments and you may or may not wish to preserve special characters. There are three types of quotes recognised by Unix. Single quotes pass the literal string without further processing. Double quotes allow the substitution of variables, and back quotes are used when the output of a command is to be used as a string. The differences are shown using the echo command echo $HOME /usr/ittc/demo0 echo " $HOME" /usr/ittc/demo0 echo '$HOME' $HOME echo 'date' date echo `date` Wed Sep 6 14:57:43 BST 1995 Chris Bayliss and Alan Reed Page 8

9 patterns Patterns match one or more strings and may contain wildcards (*?) or ranges of characters ( [a-z], [1-9] ). Note that quotes will inhibit the expansion of wildcards before the string is passed to the command. Examples using ls ls * ls???? list all files list all files with four characters ls [0-9]* list all files starting with a number ls * list files named * regular expressions These are more advanced forms of pattern used by some Unix commands and contain even more special characters. For example, ^ matches start of line and $ matches end of line. Note that the * has a different meaning than in patterns; it means zero or more repetitions of an expression. Examples fred ^fred fred$ fred anywhere on the line fred at the start of a line fred at the end of a line. any character.* any character repeated zero or more times Chris Bayliss and Alan Reed Page 9

10 grep This command finds all occurrences of a regular expression in a file or from the standard input piped to it, and displays all lines containing the regular expression. Regular expressions are explained in Appendix I. These are used by various different programs to represent particular character strings. The grep command can be useful when trying to find a given word or expression in a file or group of files. For example, if you wished to find every line with the word window in it in the file terminal.txt, you would type grep "window" terminal.txt The lines containing window would then be displayed. By default, the search is case sensitive. If you did not wish the match to be case sensitive you would use the -i flag. grep -i "window" terminal.txt The command can be combined with pipes, wildcards and other commands. For example, grep -i "window" *.txt The regular expression can be used to search for a string at the start or end of a line. For example, grep -i "^window" *.txt grep -i "window$" *.txt If you wished to display a list full details of directories in the current directory, you could pipe the output of ls-l into grep. For example, ls -l grep "^d" As ^ in a regular expression matches beginning of line, this displays only lines starting with a d. Chris Bayliss and Alan Reed Page 10

11 Part 3: More about shells Command line recall and editing The Korn shell supports command line recall and edit. This can save a lot of typing, as it gives you the ability to recall previous command lines, alter them and reissue them. Two modes are supported, emacs mode and vi mode, because of the style of the editor that is copied. In order to enable emacs mode, the built-in set command is used. set -o emacs This is normally done in your.profile file (this is discussed later). The basic functions supported are as follows CONTROL+P CONTROL+N CONTROL+B CONTROL+F recall previous line recall next line (only useful after one or more <ctrl>p move back one character move forward one character ksh Programming There are others, and these are documented in the ksh manual pages. Shells can be used to write programs. The simplest programs are files which contain a sequence of commands and allow the passing of parameters from the command line. This is sufficient for many uses. However, the Korn shell has many features which allow complex programs to be written, such as conditional statements and loops. This course does not cover these, but they are mentioned here in case they are of interest. Full details are contained in the manual pages for the Korn shell. A simple example program is given below #!/bin/ksh # Simple demonstration program # echo "What is your name?" read name echo "Hello, $name" The first line sets the shell to be used for this particular program., as it starts with #! followed by the pathname of a shell. The line indicates that the Korn shell should be used. Any other lines starting with a # are treated as comments. Comments are useful for keeping track of what Chris Bayliss and Alan Reed Page 11

12 parts of the file do, who changed the file, why and what changes were made. In addition, they can be used to put in blank lines to improve readability. The fourth line prompts the user for a name. The quotes ensure that the question is followed by a space. The third line reads the user input and puts it in the variable $name. The fourth line types Hello, followed by the value of $name. The program is in a file called prog1. An example of running it is given below. prog1 What is your name? Chris Hello, Chris Note that before a program can be run in this way, you must make sure that you have read and execute access to the file, and that "working directory" is on your PATH (see below). The other basic feature which is very useful is the passing of parameters from the command line. These are available as variables called $1, $2, $3, etc. All parameters together are available in a variable called $*. For example, the program #!/bin/ksh echo "Hello, $1." would print out Hello, followed by the first parameter on the command line. The file in this example is called prog2. An example of running it follows. prog2 Chris Hello, Chris. Chris Bayliss and Alan Reed Page 12

13 .profile This is a special shell program which is run every time you login, and it resides in your home directory. Some parameters are set in the system profile, which is a special profile run when everyone logs in. A typical.profile is given below. # ACS ksh.profile # Last modified: 18-Feb-94 JWH # # This is a file of commands which is read in by your shell when # it first starts up. You may edit it as you wish. # PATH=$PATH:$HOME/bin:. export PATH # umask 077 set -o ignoreeof set -o emacs # The first few lines start with a # and are comments. Normally, the first line selects the shell to be used with information selecting the shell, however this is not done in the.profile as this is run within the login shell, which is set for each user in a system file. PATH=$PATH:$HOME/bin:. export PATH These set your PATH environment variable. The directories $HOME/bin and. (your working directory) are added to your PATH variable. Note that $PATH is expanded to the original value of PATH. This method of modifying PATH avoids the possibility of omitting some of the directories needed to find the commands. The other method is to specify all directories needed in the final PATH and is not recommended, unless you need to remove directories from the PATH set by the system. Chris Bayliss and Alan Reed Page 13

14 umask 077 set -o ignoreeof set -o emacs More redirection This sets the user file creation mask. The effect of this setting is to ensure that nobody else can read or write files which you create. The access modes can be set for individual files or groups of files using the chmod command, should you need to give other people access to them after their creation. Normally, <ctrl>d signifies end of file. This has the same effect as exit to the shell. This command disables this feature, preventing accidental logouts by use of <ctrl>d. This sets emacs mode for command line recall and editing. If you wish to change any settings or add any commands, you may do this by editing your.profile file. For example, if you wished to display the date and time whenever you login, you could add the command echo It is `date` There is one type of redirection which allows the input of several lines to a command. An example is given below. cat <<XXXX Twinkle twinkle little star, How I wonder what you are, Up above the stars so high, Like a diamond in the sky XXXX The above example simply displays the verse on the terminal. However, in programming the pipe could be into a command to send , for example, and if necessary, variables could be included in the text. The XXXX is a sequence of characters which are given to indicate end of input. Any sequence of characters can be used, but it is wise to use a sequence unlikely to appear in the command input. There is a command which can be useful when redirecting standard output. This is tee and allows output from a command to be directed to a file and the terminal at the same time. For example: ls -l tee dirlist would direct the output from the ls command into the file dirlist as well as displaying it on a terminal. Chris Bayliss and Alan Reed Page 14

15 In addition to standard output, commands also produce error output. This is used for error messages and is treated differently by the system and is not redirected in the same way as standard output. However, error output can be directed to a file. For example ls -l xxxx 2> temperrors would put any errors produced by the ls command into the file temperrors instead of displaying the errors on the terminal. This feature is particularly useful if you issue a command which you know will produce a lot of error messages and you do not wish to display them at the terminal. An example of this occurs later in the course. Chris Bayliss and Alan Reed Page 15

16 Part 4: Finding Files find List will only list files in a directory or those immediately below it. Find is a command which will find files within a directory hierarchy, wherever they are within the tree. For example to find the file treasure the following would be used find. -name "treasure" -print The starting directory in this case is the current directory - all directories below it in the tree are searched. Any directory could be specified at this point. If you wished to search the entire hierarchy, you would use find / -name "treasure" -print Note that this will take some time. If you wish to cancel the command before it has finished, you can do so by typing control+c. You can cancel many other commands in this way. The other problem with running a command like the one above is that you can generate a lot of error messages. In this case, the errors are because you do not have sufficient access to some of the directories in the filestore to examine their contents. If you wish to suppress the error messages, you can use redirection. This is demonstrated in the following example. find / -name "treasure" -print 2> /dev/null The file /dev/null is a special device on Unix and is effectively a black hole. Any unwanted output can be directed to it and it will be discarde. If you wanted to search for a pattern rather than a fixed filename, you need to put the pattern in quotation marks. This is to prevent the shell from expanding the pattern before passing the pattern to the command. If you omitted the quotes, find the shell would find matches for the pattern in the current directory, and pass them as arguments to the find command. This would almost certainly not produce the desired results. find. -name "treasure*" -print In all the above examples, the -print option has been used. The -print displays the pathname when it finds the file. If -print is missing the command will find the file and not display it. This may seem a little bizarre to the uninitiated and volatile fuel for the Unix basher, but find has other options and you do not always want to display the pathname if using some of the other options. Chris Bayliss and Alan Reed Page 16

17 Find can perform other tasks instead of simply displaying the pathname. If desired, a command can be given to operate on each file found which matches the criteria. For example, the command. find. -name "*" -exec grep window {} \; would find all files in all directories including and below the current one, and display all lines in all files containing the word "window". The -exec indicates the command to be run, the curly brackets {} indicate where in the command line the found filename is to be placed and the \; characters indicate the end of command to be executed. In this example the print option has not been used. However, if you wished to print only the names of the files for which the command returned output, you would use find. -name "*" -exec grep window {} \; -print Chris Bayliss and Alan Reed Page 17

18 Part 5: Utilities sort uniq The sort command sorts files according to the ASCII sorting sequence by default - this for many purposes is an alphabetical sort. The output is directed to standard output. For example:- sort unsorted.list By default output is sent to the screen, but the results can be piped into a file. For example:- sort unsorted.list > sorted.list You will notice that the command completes very quickly. This causes problems as many users, when sorting a large file for the first time, wrongly assume that the sort has not worked then waste a lot of time checking that it has. The above example sorts the file according to the ASCII ordering of characters. This is fine for most alphabetic sorts, but not very useful for numeric sorts. If you wish to sort a list of numbers, the -n flag is used. sort -n numbers.txt > sorted.txt It is important to make sure that input and output filenames are different. Unix opens the output file for writing first, deleting any previous contents. If the filenames are the same, the contents of the file is therefore deleted. This is true of many different commands in Unix. There are a variety of other options, which allow reverse sorting, ignoring of case or blanks and sorting on different fields. The manual describes these in detail. This command will find and display or remove consecutive duplicate lines in a file. It is particularly useful in sorted files, and can be extremely useful when finding or eliminating repeated lines in data files, for example. uniq datafile.dat would display the sorted file, eliminating duplicates (the -u flag in sort will also do this). sort -u datafile.dat will display only lines which are not repeated. uniq -d datafile.dat displays only repeated lines. Chris Bayliss and Alan Reed Page 18

19 The output is sent to standard output which, by default, is the terminal. As normal this can be redirected to a file if desired. Combining files - cat and paste There are two main commands to join two files together. The first is cat, which is short for concatenate. The files are joined in order, by line, each file on the command line following the previous file. For example, cat verse1.txt verse2.txt verse3.txt > poem.txt would join the three files, one after the other and put the result in a file called poem.txt. The other way to join files is side by side. the paste command can do this. Line by line, the files are joined. For example if we take two files, firstnames.txt surnames.txt containing the following firstnames.txt Jemima Jeremy surnames.txt Puddleduck Fisher and then joined them with the paste command as follows, paste firstnames.txt surnames.txt > fullnames.txt the file fullnames.txt would contain: Jemima Puddleduck Jeremy Fisher Chris Bayliss and Alan Reed Page 19

20 cut The cut command allows extraction of columns of information from files. For example if the file animals.txt contains the following chimpanzee, rat, giraffe, hedgehog, lion baboon, mouse, camel, porcupine, tiger tamarin, coypu, cow, aardvark, leopard wc and you wished to extract the rodents (column 2) the command would be cut -d, -f 2 animals.txt and the output would be rat mouse coypu This can be used in conjunction with paste in order to move columns between files. The wc (word count) command counts the number of lines, words or characters in a file. Used on its own, all three are counted. For example, wc curry.txt curry.txt The -l option restricts the output to the number of lines. This can be very useful when combined with other commands. For example ls wc -l will show the number of files in a directory. Chris Bayliss and Alan Reed Page 20

21 Part 6: Topics not covered - sed, awk, perl and vi sed awk perl vi These three utilities are not covered by the course, but a brief description is given below. The commands sed and awk are documented in the Unix manual. In addition there is a very good Nutshell handbook ( sed & awk by Dale Dougherty ISBN published by O'Reilley and Associates) which covers both of these commands. The Nutshell handbooks cover specific topics in good detail. For most users, a general Unix textbook may well be helpful, preferably one with a positive attitude towards Unix. Our current recommendation is Open Computing Unix Unbound by Harley Hahn ISBN published by McGraw Hill. sed is a stream editor. This allows editing of a file which is piped into it on a line by line basis. Regular expressions can be used and substitutions can be made. Lines can be selectively added or deleted if necessary. awk is a programming language which allows processing of a file on a line by line basis. The file is scanned line by line, patterns are matched and appropriate processing takes place. In addition to this, actions to be taken both before and after the file is processed may be specified. One major use of this is the gathering and processing of statistics from system log files. perl is a shell like programming language. It is not a standard part of Unix, but is very popular. Whilst the Korn shell can be used to write complex programs, perl is commonly used instead. vi is a screen editor. It is the one supplied with Unix. Most people prefer to use emacs, but vi has its supporters. In addition, if you are installing a new system or do not have emacs available, you may need to use vi. It has a man page and a tutorial. An online tutorial is available called vitutor. Chris Bayliss and Alan Reed Page 21

22 Part 7: Processes and how to kill them Processes Most commands and programs run as separate processes. These have their own set of variables, although environment variables are passed to each process you create. Sometimes, processes are run in the background. This may be done intentionally or accidentally. Some programs will have options to run them in the background (for example control Z on emacs), others can be run in the background by placing an ampersand (&) at the end of the command line. If you have background processes running, it may be necessary to either bring them to the foreground and terminate them cleanly, or destroy them. If you try to logout when you have background jobs running, the system will warn you, giving you the opportunity to take any appropriate action. ps The ps (process status) command displays information about active processes. In order to identify a process, which is essential if the process is to be destroyed, the process number is needed. The ps command has many options and will display all running processes. In order to print basic information about your own processes, the -u option is used. For example, ps -u demo0 PID TTY TIME COMD pts/27 0:02 elm pts/27 0:02 ksh pts/27 0:02 ps pts/27 0:02 emacs pts/27 0:02 emacs The PID is the process ID, a number which at any one time uniquely identifies a process on a particular system. Chris Bayliss and Alan Reed Page 22

23 fg The fg command brings the process into the foreground and under your control. If you have only one background process, the command needs no arguments, and can be given on its own. If you have more than one background process, the most recent is brought to the foreground by default. You may specify a process ID to bring a particular background process into the foreground. Using the above example, fg fg the foreground. brings elm into the foreground brings the older suspended emacs session into kill The kill command send a signal to a another process. There are several options. The one most commonly used is the -9 option, as this terminates processes which can ignore some of the other options. It is a useful command for terminating a n unwanted background process. For example in order to terminate the process in the above example, you could use kill Take care when terminating processes such as elm and emacs in this way, as work in progress can be lost. If you need to save work in progress, bring the processes to foreground and terminated them cleanly. Chris Bayliss and Alan Reed Page 23

24 Appendix I: ASCII characters, strings, patterns, and regular expressions ASCII characters strings When using UNIX you will come across the above terms, this document is designed to explain what they are and how they are used in UNIX. These are the letters, digits, punctuation marks, and special symbols that you can type from a keyboard attached to a UNIX computer system. Some characters you can 'see' printed on the keytops but there are others that are hidden from you but the keyboard is able to send to the UNIX computer. Each character has a number associated with it from 0 to 255, thus there are 256 ASCII characters in all. For example the letter A has the number 65, and the digit 0 has 48. In computing we count in 10's (decimal), 8's (octal), and 16's (hexadecimal). Thus the letter A would be 65 in decimal, 101 in octal, or 41 in hexadecimal. The printable ASCII characters are a 'space' and!"#$%&'()*+,-./ :;<=>?@ ABCEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx yz{ }~ These are a sequence of the above ASCII characters. Thus FRED is a string of the four ASCII characters F R E D. In a UNIX 'shell' such as the Korn shell you can write strings in single quotes ' or in double quotes " or single back quotes `. In single quotes the string is taken literally*, in double quotes it is examined by the shell and all $ names are substituted. When the single back quote is used the shell evaluates the string as a command and returns the result as a string. To understand these behaviours try out these shell commands:- echo '$HOME' produces $HOME echo "$HOME" produces your home directory echo 'date' produces the string d a t e echo `date` produces today s date and time * Not entirely literally because the \ symbol is treated in a special way. To get the non-printing characters you use the \ symbol followed by the three octal digits of the character, thus \007 means the bell, and \014 a newpage. You can also use \n for the linefeed character, and \r for carriage return. Try typing echo 'Hello there\007', and listen for the beep! Chris Bayliss and Alan Reed Page 24

25 patterns A pattern is a string used to match, or stand for, many more strings. In it the character? matches ONE character, * matches ZERO or MORE characters. [a-f] (say)matches the lower case letters a or b or c or d or e or f. The main use you will have for a pattern is for filename generation as follows: Try typing:- ls * ls???? ls x* ls x*f ls [0-9]* ls [abc][xyx] lists all your file names lists all four letter filenames lists all names starting with x ditto starting with x ending in f lists all names starting with a digit lists all 2 letter filenames of all combinations of abc and xyz Note the absence of quote marks, and investigate the difference between typing ls * and ls '*', the former looks for all names while the latter looks for a filename of a single star character. For full details see the man page for the Korn shell (man ksh). regular expressions Expression Meaning A regular expression is an advanced form of a pattern used by some UNIX commands such as grep or the editor sed. The following description is taken from the man page for regular expression (man regexp). A regular expression specifies a set of character strings. A member of this set of strings is said to be matched by the regular expression. Some characters have special meaning when used in a regular expression; other characters stand for themselves. c the character c where c is not a special character. \c the character c where c is any character, except a digit in the range 1-9 ^ the beginning of the line being compared. $ the end of the line being compared.. any character in the input. [s] any character in the set s, where s is a sequence of characters and/or a range of characters, for example, [a-z]. [^s] any character not in the set s, where s is defined as above. Chris Bayliss and Alan Reed Page 25

26 r* zero or more successive occurrences of the regular expression r. The longest leftmost match is chosen. rx the occurrence of regular expression r followed by the occurrence of regular expression x. (Concatenation) r\{m,n\} any number of m through n successive occurrences of the regular expression r. The regular expression r\{m\} matches exactly m occurrences; r\{m,\} matches at least m occurrences. \(r\) the regular expression r. When \n (where n is a number greater than zero) appears in a constructed regular expression, it stands for the regular expression x where x is the nth regular expression enclosed in \( and \) that appeared earlier in the constructed regular expression. For example, \(r\)x\(y\)z\2 is the concatenation of regular expressions rxyzy. Characters that have special meaning except when they appear within square brackets ([ ]) or are preceded by \ are:., *, [, \. Other special characters, such as $ have special meaning in more restricted contexts. The character ^ at the beginning of an expression permits a successful match only immediately after a newline, and the character $ at the end of an expression requires a trailing newline. Two characters have special meaning only when used within square brackets. The character - denotes a range, [c-c], unless it is just after the open bracket or before the closing bracket, [ -c] or [c-] in which case it has no special meaning. When used within brackets, the character ^ has the meaning complement of if it immediately follows the open bracket (example: [^c]); elsewhere between brackets (example: [c^]) it stands for the ordinary character ^. The special meaning of the \ operator can be escaped only by preceding it with another \, for example \\. Chris Bayliss and Alan Reed Page 26

27 After that what can one say! Here are some examples:- regular expression matches fred fred anywhere in a line ^john john at start of a line john$ john at end of a line ^john$ john on a line ONLY ^$ a blank line ONLY =\{14\} exactly fourteen = characters \(.\)\1\1\1 exactly four characters ALL THE SAME. Chris Bayliss and Alan Reed Page 27