The input can also be taken from a file and similarly the output can be redirected to another file.

Transcription

1 Filter A filter is defined as a special program, which takes input from standard input device and sends output to standard output device. The input can also be taken from a file and similarly the output can be redirected to another file. UNIX treats all devices as files and the shell provides three data streams (a stream is a collection of characters): 0 Standard Input stream (the keyboard) or stdin 1 Standard Output stream (the screen) or stdout 2 Standard Error stream (the screen) or stderr Now if we wish to take inputs or send outputs to and from other sources the respective data streams must be redirected. Thus we also have the following redirection symbols: < - Input Redirection > - Output Redirection - Pipe Redirection, which can represent both input as well as output redirections. A few examples of filters used in UNIX are cat, wc, sort, more, grep, cut, etc. Filter cat The command used to concatenate files is cat. So by just issuing the command: $ cat expects input from the keyboard (the standard input) and sends output to VDU (the standard output). As soon as the user presses <Enter>, the contents of the buffer are flushed out and the values are displayed on the console. This process will continue until the user chooses <CTRL + d> to terminate it. 1

2 By default, cat expects input from standard input device, i.e, keyboard and sends output to standard output device i.e, VDU. Variations of cat Some of the variations of cat command with input and output redirections are given below: $cat file1 The above command accepts input from a file called file1 and redirects the output to the standard output device (VDU). This implies that the cat command is accepting the input from a file and sends the output to the standard output file. The above command can also be written in the following way using the input redirection symbol: $cat<file1 2

3 Using the above logic, we may take input from the stdin, and redirect the content to a file, i.e. we shall create our own file using the following syntax: $ cat >file1 It will take input from the keyboard till <Ctrl + d>. The only drawback of creating a file using the above way is that the file cannot be modified (till any text editor is used) after hitting the <Enter> key. In the above command, if the file called file1 already exists, the existing text will be overwritten. To prevent this, the following command can be issued: $cat>>file The command above will append the new text to the existing one. The usage of stderr is given below: $cat hello 2>err $cat err In the above command the file hello does not exist. The error message is redirected to a file called err instead of sending it to the standard output device. The command [cat file1>file2] is similar to [cp file1 file2] Lets take another example $ cat fl1 > fl2 2> err If the file fl1 exists, readable and the read permission is granted then the output will be stored in fl2, other wise the system defined error message will be stored in the file err. This is to be noticed that in either case nothing will be displayed 3

4 Filter wc UNIX provides a filter to keep track of the number of lines, characters and words of a particular file. The filter that is used to perform the task is wc. The usage of wc filter is given below: $wc file1 The output is given below: Lines Characters If one likes to view only one value at a time then the following switches may be used: Switches Description -l Displays number of lines. -w Displays number of words. -c Displays number of characters. Some typical usages of wc is given below: $wc l file1 Words The above command will display only the number of lines in file1. The output is given below: 4

5 $ls wc -l The above command will give a total count of files and directories in the current directory. The ls command redirects the output to wc filter instead of sending it to the standard output device. See the use of Pipe ( ) $wc<file1>count The above command facilitates wc to accept input from the file called file1 and then redirects the output to another file called count, instead of sending the output to standard output device. Now, display the content of count using the cat command. The output is given below: Filter more Sometimes when the output of a particular command is not accommodated within one full screen, the first part of the display scrolls up. To have one page display at a time, there is a useful filter called more. 5

6 The usage of more is as follows: $ more file1 The above command will display the contents of the file one screen full at a time. To move to the next line, press the Enter key. Another usage of more is to display the contents of a directory page wise: $ ls more The above command can also be used with the pg filter. $ls l pg Arranging output more differs from pg in respect that with the help of more, we can navigate back and forward with the help of b and f keys respectively. In case of pg, we can scroll back and forward using the page number. To arrange data sequentially either in ascending or descending order in order, the sort filter is used. The sort filter reorders lines depending on the ASCII values of the characters. The sorting starts from the first character of each line and proceeds to next character only when the characters in two lines are identical. By default, the lines are sorted in ascending order. The usage of sort command is given below: $sort The above command will wait for any input from standard input device. After pressing <Ctrl+d>, it will display the output in a sorted manner in the standard output device. Press <ctrl+d> 6

7 $sort file1 By default, the sort command starts sorting from the first field (field number starts from 0). The field can be explicitly specified from where the sorting should start using +fieldnumber switch where fieldnumber denotes the value added to the default start field. The command sort takes space as default delimiter. The example given below starts sorting from the second field. $sort +1 file1 7

8 Now, to sort the file only depending on the second field, the following command is used: $sort +1 2 file1 In the above command, -2 is restricting the sort for the second field only. Some of the frequently used switches with sort are given below: sort r: By default, sort arranges the output in ascending order. To arrange the output in descending order, sort r option is used. $sort r file1 8

9 sort t : If a file contains fields having delimiter other than a space, sort t can be used to specify that delimiter. $sort t file1 The fields in file1 are separated by a pipe ( ). In the above command, this delimiter has been specified using sort t. sort o : As sort is a filter, sorted output can be redirected to a file using > operator or o option. The usage of sort o is given below: sort o <sortedfile> <originalfile> Now, issue the following command from $ prompt: $sort o sortfile file1 sort n: In UNIX, sorting can be done on numeric fields also. The command below will perform the task. $sort +1n file1 9

10 sort can be used with n and r simultaneously to arrange the output in reverse order based on the numeric field. $sort file1 +1nr Moreover, sort can be used with uniq command to eliminate the redundant lines from a file. $sort file1 uniq 10

11 Filter tee As already discussed, with the help of pipes, the output of one command is sent as input of another command and the ultimate output is achieved. But let us take an example where we want to display the number of users that are currently working in the system and at the same time we want to store the output of who command into a file. In order to get that, UNIX provides the tee filter. The tee command enables the output of a command to be redirected to a file and at the same time passes it to the input of a pipe. $who tee file3 wc -l tee a is used to append the output to a file. Filter grep UNIX provides a filter called grep that searches for string. The grep stands for Globally Search for Regular Expressions and Print out. The type of search expressions that grep filter can handle is enormous and if used with a combination with sort, the result can be obtained efficiently. The usage of grep is given below: $ grep <search_expression> <file> The following example searches a specified expression in a particular file and will display the lines containing the expression: $grep Hello file1 11

12 If more than one word needs to be specified as a grep pattern, the pattern must be enclosed within quotes. Some of the search expressions are given below: CHARACTER MEANING EXAMPLE EX. DESCRIPTION [ ] Matches any one of a set of characters grep lab[abc] Means laba or labb or labc [ ] with hyphen Matches any one in the range grep lab[a-c] Means laba or labb or labc ^ Pattern following it must occur at the beginning of a line grep ^lab The word lab must occur at the beginning of a line [^ ] Must not contain any grep lab[^a-c] Any other pattern but character(s) specified laba or labb or within [ ] $ Pattern preceding it must occur at the end of each line grep lab$ labc. The word lab must occur at the end of the line.(dot) Matches a single character grep lab. The pattern is lab followed by any character. \ Ignores any special meaning of the character following it. grep lab\[abc\] Means literally lab[abc] as the pattern Some frequently used switches with grep command are given in the table below: Option Description -n Prints each line with the matching pattern along with the line number. 12

13 -c Prints the total count of the lines matching the pattern. -v Prints all those lines that do not match the pattern. -i Ignores cases while searching for a pattern. Some typical usages of grep filter are given below: $ls l grep v ^- The above command will display all the directory and special files under the current directory. $grep v ^$ file1 > newfile The above command will delete all the blank lines from a file called file1 and the final output will be redirected to a file called newfile. 13

14 $grep v ^ *$ file > newfile Sometimes a line that looks blank is not really blank. It can contain some spaces. The command given above deletes all those blank lines that contain spaces. $grep n Hello file1 The command given above will display the lines containing the word hello with the line numbers. The output is given in the next page: $grep c Hello file1 The command given above will display the count of lines containing the text Hello in the file called file1. $grep i hello file1 The above command will display the lines containing the text hello irrespective of the case specified during the search. 14

15 Filter tr The tr filter of UNIX has multiple roles. It can be used for the following two purposes: Case conversion The command given below will wait for any input from stdin and then it will convert all the upper case characters to lower case. $tr [A-Z] [a-z] $tr he HE The above command will wait for input from stdin and then it will convert all h and e to H and E. Squeeze characters The tr command can also be used to convert multiple occurrences of column separator to single. The s switch with tr filter helps to achieve the purpose. The command given below squeezes multiple occurrences of space in the output of who command to single space. 15

16 $who tr s $ls tr s The parameter used with -s, must be the same as that of the column separator of the source data The cut filter When it is required to display only a part of the columns from the output of certain commands like ls l, who, or specified columns of a data file, the filter cut comes in handy. The frequently used switches with cut filter are given in the table below: Switch Description -f Field or column number that is to be cut -d Cut upto the column delimiter (must be uniform throughout all the columns). The default delimeter of cut filter is tab. -c The position of the character to be cut Some typical usages of cut filter are given below: 16

17 $cat file cut c1 The above command will redirect the content of the file called file1 to cut filter. The cut filter will extract the first character of each line and will display it. The output of the above command is given in the next page. $cat file1 cut c 1,2 The above command will extract first and second character of each line from file1 and will display it. $cat file1 cut c1-5 The above command will extract all the characters lying between first and fifth position of each line of the file called file1and will display it. 17

18 $cat file1 cut c3- The above command will extract the character from third position of each line till the end of the line. $cat file1 tr s cut d -f1,2 The above command will redirect the content of file1 to tr filter. The tr filter will squeeze the multiple occurrences of spaces between the fields into single space. Ultimately the cut filter will take the column delimiter as space (mentioned as d ) and will extract the first and the second field. 18

19 The /etc/hosts file You start with the most fundamental file that you need for communicating with other computers on your network the hosts file. When you access another host by name on the Internet or any intranet/internet, your computer needs to know the remote host's IP address. You can get remote host addresses from DNS (Domain Name System) or from your computer's hosts file. This file lists the names and addresses of other hosts known by your computer. When you need to know about thousands of hosts on the Internet, maintaining the hosts file is really too cumbersome a mechanism. Imagine having to spend all that time updating it as computers come and go or relocate on the Internet! In that case, you need DNS to locate remote hosts. IP address are used by many TCP/IP utilities like ftp, telnet. For example : ftp or telnet but it s impractical to remember machines by their IP addresses, while network understands only the numbers and not the names. On smaller networks, the name-address mappings are placed in the file /etc/hosts on each and every host of the network. This database allows the applications to look up a name and find out its corresponding IP address. A look at a sample file reveals its flexile structure : localhost # the loopback address Kaushik Suvendu Suvro For each machine in the network, this table contains a line mapping the IP address to its respective host name. It contains at least 2 fields for each line, one for each type of the address. 19

20 Typical Examples Using Filters Count the number of users currently logged in as user1. $who grep user1 wc -l Current Users Count Logged in as user1 Find the largest file in the current directory $ls l tr s cut d f5,9 sort nr head n1 Squeeze Cut (Field No. 5 and 9) Numeric sort (desc) Head 20

21 The example given below will display the name and the designation of an employee who is not a clerk and earns the minimum salary. For this purpose, let us first see the content of the file file1: Now, issue the following command from the $ prompt: $grep iv clerk file1 sort +2n cut d f1-2 head -1 Ignore the case and line matching the pattern Cut (Field No. 1 and 2) Numeric Sort (asc order) Display the line number 2 and 3 of a particular file $cat file1 tail 5 head -2 The above command will first extract the last five lines of the file and then it will display the first two lines from that. The output is given below: 21

22 22