Unix Introduction. André Heck c AMSTEL Institute. September 2003

Transcription

1 Unix Introduction André Heck c AMSTEL Institute September 2003

2 Contents 1 Introduction Learning goals Practical issues for first-time Unix users at FNWI Just in case you want to use a Unix Workstation Working on the students server from a PC Using softpkg Basics of Unix Directories and files Changing the working directory Simple commands to manage files and directories Printing of files Processes Unix documentation Logging out Managing files and directories File attributes Wildcards Locating and comparing files Secure ftp File compression and archives Managing processes Working with more programs at the same time Inquiring about the process status Job control Summary of Unix commands 23 2

3 1 Introduction 1.1 Learning goals Welcome to the Unix Introduction. The goal of this course is to familiarize yourself with the Unix operating system, in particular with the Sun Solaris version and/or with the Linux implementation. Unlike Microsoft Windows and MacOS, Unix is not a graphical environment in which you manoeuvre with the mouse. It is a command-driven environment. In this course you learn the most important Unix commands. On top of Unix you normally use on a workstation a graphical user interface. The X Window System has become the default environment. But in this course we shall not pay attention to the graphical user interface. Unix consists of three parts: The kernel: it handles the use of processors, memory, disk space, and accessibility of files, amongst other things. The shell: this program interprets the commands that you enter and tries to carry them out. A set of auxiliary programs for common tasks like file management, editing and printing of text, and so on. In short, files and processes play a important central role in the work of a Unix user. For Simulating and Programming this holds even more because this work often involves writing a computer program in a file; preparing an input file; computing intermediate results and storing them in files; searching stored data efficiently; manipulating of stored data, e.g., sorting. You will learn how to deal with these issues in a scientific computing setting. The Unix introduction is mainly a practical introduction based on the principle of get started quickly and learn by doing. The idea is that, if you feel the need after a while to know more about a particular topic, you have already become familiar enough with the system to find out yourself, for example by using the on-line help system or written documentation, or by asking a more experienced user. The learning goals of this Unix introduction can be summarized as follows: at the end of the course you have an overview of the auxiliary programs that are available and you know how to get information about their usage; can effectively manipulate data files with the built-in Unix facilities, as far as this concerns common computer tasks. With respect to the knowledge about Unix this means that you have insight in the hierarchical file system; understand what role the shell plays in manipulating files (however, we will not pay attention to reading, adapting, or creating shell scripts); know what possibilities exist in Unix for managing processes and how they can be applied in simulations; can judge or quickly find out whether the Unix system provides built-in solutions of computer tasks or that you must program these yourself. 3

4 The skills that you master are: working with the operating at basic level; organizing and manipulating files; managing processes; interpreting error messages and warnings; reading on-line help pages and Unix documentation. 1.2 Practical issues for first-time Unix users at FNWI If this is the first time that you use a Unix workstation or that you work in the Unix operating system at the Faculty of Science, you may need some practical information to get started. If not, you can skip most of this subsection Just in case you want to use a Unix Workstation Please note that you may skip this section when you do not use a Unix workstation or PC running Linux. If you want to work on a Unix workstation or on a PC running Linux you first have to login. On the machines at the Faculty of Science you can do this with the same user account and password that you use for the PC s running Windows XP. By the way, the central students server, where all your files are stored and where all software available to FNWI students resides, is called gene (if needed, the corresponding cluster name is edu). At the time of logging in you can also specify the graphical user interface that you want to use. If it is the first time that you try to login a default choice is made, otherwise the environment that you used last time is chosen. At any time you can request in the login window a specific desktop (do this before entering your password). Henceforth we shall assume that you are working on a Sun workstation using the Common Desktop Environment (CDE). A PC running Linux looks similar to a PC running Windows when the K Desktop Environment (KDE) is used. This needs no further explanation. More details on logging in on Sun Workstation, using X Windows and CDE can be found on the web site You are advised to familiarize yourself with the CDE, for example by reading the Desktop Introduction that is available in the CDE-Help system. One of the things that you must keep in mind is that there is a difference between clicking on the left, middle, or right mouse button 1, depending on the position of the mouse cursor. Left-clicking on an icon in the Control Panel activates in many cases a program, whereas right-clicking allows you to specify preferences. Right-clicking when the mouse cursor is on the desktop will open the so-called Workspace menu. But don t worry, by practice you will find out the functions of the three mouse buttons. One of the first things you need is a window to enter Unix commands. Such a terminal window can be obtained by selecting from the hosts menu. the item called this host. The menu appears when you right-click on the desktop. In the terminal window you will see a blinking cursor after a prompt, indicating that the system is ready for input. For example, enter the command date to find out the current date and time of the day. The last thing to do in a computer session is to logout. Clean the desktop, right-click on the empty desktop, and choose the menu-item log out... Alternatively, clean the desktop and press the Exit button in the Control Panel of the CDE (normally located below on the screen). 1 on a PC-mouse, the mouse wheel is often the middle button 4

5 1.2.2 Working on the students server from a PC SSH (Secure Shell) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. On all PC s at the faculty two types of SSH-clients are available: those that execute programs on a remote machine and those that do secure file transfer. Make your choice in the SSH Secure Shell item of the Programs menu, which is available under the Start button. You can also login from a PC at home or elsewhere if a SSH-client available. An example of a free MS Windows client is PuTTY, which can be obtained And if no SSH-client is available, you can still start a SSH-client directly from within your (Java capable) web browser: browse to gene.science.uva.nl/ssh/. Logging in via a SSH-client is easy: you specify the machine that you want to work on, enter your user account, and type the corresponding password. For example, you can log in on the central students server called gene, where all your files are stored and where all software available to FNWI students, and use the password for your FNWI edu-account. To logout, it suffices to enter the command exit Using softpkg You are strongly advised to use the software package management facility, e.g., to enable the use of packages like Matlab or Mathematica. Enter in a terminal window or shell client the command softpkg -l to see a long list of available packages and the names under which they are added to the list. (heck@gene 1) softpkg -l package os rev arch description DEFAULT sunos any any default packages DT sunos any any Theorem prover Deep Thought 3.4 TeX sunos any sparc TeX typesetting system (tetex 1.0.7) ace sunos any sparc ACE compilers acroread sunos any sparc Adobe Acroread... You must add the name of the desired software package at the end of the hidden file.pkgrc in your home directory. You can do this with an editor or by the following trick, which enables the use of Matlab 6.5: (heck@gene 2) echo matlab-edu-r13 >>.pkgrc To have effect in the current window you must enter the following command (at least when you are using the C shell), otherwise just start a new session: (heck@gene 3) source.cshrc Now you can start Matlab. For example, in the session below Matlab is started without the Java Virtual Machine and without the Matlab desktop. We leave the software package immediately by entering the command quit. (heck@gene 4) matlab -nojvm < M A T L A B > Copyright The MathWorks, Inc. Version a Release 13 Jun Using Toolbox Path Cache. Type "help toolbox_path_cache" for more info. To get started, type one of these: helpwin, helpdesk, or demo. 5

6 For product information, visit >> quit 5) Exercise 1.1. Running Mathematica in command line mode 1. Open a terminal window and login with your user account on the students server. 2. Enter the command date to find the current date and time. 3. Enable the use of Matlab on the Unix machine, following the above instructions. 4. Enable the use of the computer algebra system Mathematica on the Unix machine. Verify that you can start it with the command math. 2 Basics of Unix 2.1 Directories and files A file is a set of data that has a name. The information can be an ordinary text, a user-written computer program, results of a computation, a picture, and so on. The file name may consist of ordinary characters, digits and special tokens like the underscore ( ) except the forward slash (/). It is permitted to use special tokens like the ampersand (&) or spaces in a file name, but you are strongly advised not to use them because such file names are difficult to handle under Unix 2. Unix organizes files in a tree-like hierarchical structure, with the root directory, indicated by a forward slash /, at the top of the tree. See the figure below, in which part of the hierarchy of files and directories 3 on the computer is shown. / (root) bin tmp home pwd cd ls gastel heck walter latexcourse proefles.zip The path gives the route through the tree to arrive at a certain subdirectory or file. An absolute path (also called full path) starts at the root and lists all the subdirectories that you have to go through to a arrive at a desired spot in the tree structure. For example, /home is the subdirectory of the root that contains the home directories of all users. Your home directory is the starting point when you login to the system and where you have ownership of files and subdirectories. During a computer session you can change the directory where you are at work, and move to another working directory (also called current directory). The command pwd prints the name of the working directory and with the command cd you can change your working directory. If you use a name of a file or directory that does not start with a slash, then you refer to the object relatively with respect to your current directory. One speaks of a relative pathname. 2 For example, you can only refer a file name like strange & difficult name if you put the name between apostrophes. 3 A directory is a special kind of file that holds other files and other directories. 6

7 There are three special notations for directories: The tilde indicates your home directory.. The single dot indicates the working directory... Two dots, one immediately after the other, indicate the directory one level upward in the tree structure that contains the current directory. You will regularly use these special notations in Unix commands. For example, to move in the hierarchical file system one level upward it suffices to enter the command cd Changing the working directory You can always find out what your working directory currently is by typing on a command line pwd ( print working directory). You can change from directory to another with the command cd directory, whereby the name of the directory can be absolute or relative. The command name cd stands for change directory. Example: Suppose that your login name is equal to heck so that your home directory is /home/heck or ~heck. At any moment you can set your working directory equal to this directory with one of the following commands: cd /home/heck cd ~heck cd ~ (Recall that ~ stands in any command for the name of your own home directory) cd (So, cd without any argument always take you to your home directory) Exercise 2.1. Changing your working directory 1. Open a terminal window and verify that you working directory is equal to your home directory. 2. Move in two different ways to the directory that is one level higher in the hierarchy of files and verify that it is equal to /home. 3. Move from your home directory in three different ways to the home directory heck. 4. Suppose your working directory is /home. Find five ways to reset in one command your working directory to your home directory. 2.3 Simple commands to manage files and directories In this subsection you practise working with files and directories. Think of tasks like creating and deleting of files and directories, copying and moving of files, and so on. You obtain the first files to work with from a web site. It is a text document in various formats that contains a list of commonly used Unix commands. Exercise 2.2. Download files via the web browser Open the web browser Mozilla and enter the following web address for a list of frequently used Unix commands: Save this information in a file in your home directory on the Unix server 4. Repeatedly use Save Page As... to save the file in the following two formats: Source, i.e., in the original format of HTML (use the type Web Page, HTML only). 4 On a PC in the FNWI-network, your home directory is the same as your folder My Documents and the mounted network drive N:. 7

8 Text, i.e., as plain text (use the type Text Files and file extension.txt). Also print the web page to the file UnixCommands.ps (check the Print to file checkbox in the print diolog window). If everything has gone well you have at this moment added three files to your home directory, viz., UnixCommands.html, UnixCommands.txt en UnixCommands.ps. Check this and if it is not the case, make sure you get these three files. The downloaded list of Unix commands mentions the following commands for manipulating files and directories 5 : Command pwd ls ls dir ls -a [dir] ls -l [dir] ls -t [dir] mkdir dir rmdir dir cd cd dir cp f1 f2 cp -r dir1 dir2 locate f mv f1 f2 mv f1 [f2...] dir rm f rm -rf dir file f less f more f cat f Explanation print the name of the working directory give a listing of files in the working directory give a listing of files in the directory dir show hidden files, too display all file attributes, too (long listing) sort listing of files on timestamp make a new directory dir remove the empty directory dir go to your home directory go to directory dir copy file f1 to f2 copy (recursively) the whole directory tree under dir1 to dir2 locate file f at file server change file name from f1 into f2 move file f1, f2, etc., to directory dir remove file f remove the directory dir including all files and subdirectories. You are not prompted for confirmation and undoing is not possible. determine the type of file f display contents of file f screen by screen display contents of file f screen by screen display contents of file f in one go You see in the table clearly that Unix commands are acronyms in general: mv instead of move, cp instead of copy, rm instead of remove, mkdir instead make directory, etc. When you use tcsh 6, the following two suggestions make working at the computer easier. Name completion: Use the Tab key to extend names of files and directories as far as possible, while they have only been partially entered. Command line editing: With the Up-arrow key you can get back a previously entered command on the command line. By pressing more than once the up- or down-arrow key you can scroll through previous commands that can be changed and re-entered. Exercise 2.3. Work with files and directories Use the above list of Unix commands for the following tasks: 1. Make a directory called UnixIntro. 2. Copy the text document that you created in the previous exercise, whose name ends with.txt, and with the list of Unix commands as contents into the newly created directory. 3. Make the UnixIntro directory now your working directory. 5 Optional arguments are between brackets. 6 tcsh is the enhanced C shell that is by default given to a new user account. 8

9 4. Move the HTML-file with Unix commands from your home directory to the UnixIntro directory (where you are now). 5. Remove the text document UnixCommandos.txt from your home directory without leaving the current directory UnixIntro. 6. Check what are the contents of the current directory and your home directory, without leaving the current directory. 7. Browse the contents of the text document with Unix commands that is still, if everything went well, in the directory UnixIntro, with the command more. 8. Browse the corresponding HTML-file and PostScript-file in your home directory. 9. Verify with the command file that the PostScript file UnixIntro.ps is (probably) recognized as an ascii document. If you would try to print the file as it is on a PostScript printer (don t do this!) the output would just be a waste of paper containing raw PostScript code. The reason is that the first 5 lines in the file may look like 2345X@PJL SET RESOLUTION = SET BITSPERPIXEL = SET ECONOMODE = ENTER LANGUAGE = POSTSCRIPT because you did not choose ZZ Generic PostScript Printer as the current printer. Officially, the first line in a PostScript file must start with %!PS (but %! is also correct). One remedy is to delete the first 5 lines of the present file with an editor so that it starts with the line %!PS-Adobe-3.0 or something similar. Another remedy for you to try is to print the web page again to a file, but now by selecting first the ZZ Generic PostScript Printer (do not check the Print to file checkbox). Verify that you have now obtained a valid PostScript file 10. Go to the home directory of someone else (your neighbor for example). Can see what kind of files and directories this person has. 11. Return to your home directory and make a copy of the directory UnixIntro using the new name unix. Verify that this directory is indeed an exact copy of the original. 12. Finally remove the directories UnixIntro and unix. 2.4 Printing of files For making a printed copy of a document in a Unix environment the following three commands are relevant: lpr for the actual printing. lpq for inspecting the printer queue. lprm for removing a print job in the printer queue (in case you did something wrong or have no time left to wait for the printed output). For the last two tasks exist alternative commands, viz., lpstat, to display information about the status of the print service, and cancel, to cancel a print request. We refer to the online manual page for details. 9

10 In print requests you must specify the name of the printer. In the Euclides building the student printer is called sl1-1 and it is located on the first floor in room P103, nearby the computer room P127. In the Wibauthuis pr-w3-01 is the printer available to students. We refer to the web page for details about FNWI printers. The specification of the printer is done via the option -P. For example, use lpr -Psl1-1 filename to print the file with given name on the printer sl1-1. Exercise 2.4. Make a printed copy 1. Check with the command file whether you have indeed a PostScript version of the list of Unix commands (this is the file which has most probably the extension.ps). 2. Print the PostScript file to the nearest PostScript printer available. 3. Check with lpq how many print jobs are in the printer queue and see if your own print job is included. 4. Remove your print job via lprm if this is still possible. For this you need the job-id of your print job. This number can be found with lpq. The command lpq is also useful to find out whether your document has already been printed and can be taken from the printer. 5. If you have removed the print job before, re-enter the print job and get the paper from the printer once it is ready with your job. 2.5 Processes Every command that you enter or each menu item that you select in a graphical user interface has the effect that some process is started. Entering the pwd command has the effect that the process with the same name is started. The program code resides in the file /bin/pwd. This file is not human readable, but it is executable computer code. Another command is matlab, which starts the numerical software environment with this name. The instructions to start this program can be found in the file /usr/local/arch/matlab-edu_r13/bin/matlab. You can verify this by (heck@gene 6) which matlab /usr/local/arch/matlab-edu_r13/bin/matlab Exercise 2.5. Using Perl Find out where the Practical Extraction and Report Language (Perl) resides, i.e., where the perl program that allows you to run Perl scripts can be found on your computer. From the examples you see that we hit a problem: which process must actually be started when a command is entered? The file system is so large that a complete search for the requested process is too time-consuming. Besides, what to do when various versions of a program exist on different locations? In order to circumvent both problems, the shell, i.e., the program that allows you to communicate with the operating system of the computer, uses a standard search path for commands entered. In many shells this search path is stored in the so-called environment variable PATH. Although you can specify the search path in a start-up script of the shell in an explicit way, this is not recommended at FNWI. Instead you must use the previously explained software package management facility softpkg. In the Unix world one prefers to speak about processes instead of programs because you can use a program more than once at the same time. Unix is a so-called multi-user, multitasking operating system. During a computer lab many students will use the same text editor or the same computational software program at the same time. A single user may also activate the same program more than once. In order to distinguish activated programs from 10

11 each other we call each started program a process. In short, a process is a running program using its own memory space and attributes. A new process is started each time when you start a program. Later you will learn more about managing processes. 2.6 Unix documentation All standard Unix commands and many other commands are described in so-called man pages. A man page (abbreviation of manual page ) is a clarifying description of a command. You get the explanation via the man command, followed by the name of the command about which you want information. For example, man pwd explains the pwd command. In the terminal window win which you enter this command you get something like: User Commands pwd(1) NAME pwd - return working directory name SYNOPSIS /usr/bin/pwd DESCRIPTION pwd writes an absolute path name of the current working directory to standard output. Both the Bourne shell, sh(1), and the Korn shell, ksh(1), also have a built-in pwd command. ENVIRONMENT See environ(5) for descriptions of the following environment variables that affect the execution of pwd: LC_MESSAGES and NLSPATH. EXIT STATUS The following exit values are returned: --More-- 32% The last line indicates that only 32% of the man page has been displayed yet. You can continue in many ways: with the Return or the usual Enter key you proceed line by line; pressing the space bar brings you a whole page further. When you keep pressing the space bar you will finally see the normal prompt back on the screen. You can finish reading the man page in between by pressing the q key or by entering the ZZ character combination. Entries in the reference manuals are organized into sections. In the above example, you see the information of section 1 about pwd. You can specify in which section to search for information. Exercise 2.6. Searching a man page in a specific section 1. Find out what is the difference between the command man ispell and man -s4 ispell 2. Compare the man page of environ in section 1 and 5. With man -k keyword you can search the on-line help system for the given keyword. For example, man -k HTML gives a list of commands that have to do with HTML. (heck@gene 7) man -k HTML texi2html texi2html (1) - a Texinfo to HTML converter 2.7 Logging out The last thing to do in a login session is to log out. You can use the logout command or simply type exit. Exercise 2.7. Ending a session Start a new login session on the student server and terminate it again. 11

12 3 Managing files and directories 3.1 File attributes Because Unix is a multi-user operating system, it keeps an administration of the ownership and user permissions of files and directories. Suppose ls -l gives the following long listing -rw-r--r-- 1 bvandijk st-wis 21 Aug 20 09:24 voorbeeld drwx bvandijk st-wis 21 Aug 20 09:24 Mail then the user account bvandijk is owner of the file voorbeeld and the directory Mail. The directory can only be visited, read, and written by the user account bvandijk. Each line in the long listing gives columnated the following information: 1. mode: The first 10 special characters are the file attributes The first character denotes what kind of file it is: the hyphen - in the above example shows that voorbeeld is an ordinary file. Directories are special files listed by the character d. You may also see the character l denoting a symbolic link, but we do not pay attention to such files in this introduction nor to other exotic entries. The next 9 characters in the first column are the file permissions. Read them in groups of three characters: the first set shows what the owner can do with the file, the second set of three characters indicates the permissions of group members, and the last set establishes the permissions of other users. For security reasons users are divided into groups: science students are placed in groups like st-wis (mathematics student) and st-schei (chemistry student). In our example it can be seen that bvandijk is owner of the file voorbeeld and that this user is member of the group st-wis. The first set of tokens rw- indicates that the owner can read the file (r permission) and can change its contents (w permission). The third token is in this case -, but many times you will see there the token x when it is an executable file. The next two sets of tokens (r--) indicate that fellow members of the group st-wis and all other users can only read the contents of the file. A second example: ls -l /bin/pwd displays the following mode of this file -r-xr-xr-x, meaning that this file is only readable and executable for each user. 2. number of links: this is not interesting for us. 3. owner: the login name of the owner. 4. group: the name of the group to which the owner belongs. 5. size: the disk space (in bytes) token up by the file. 6. date and time: The time stamp when the file was created or changed for the last time. 7. name: The name of the file. We summarize the rwx permissions of files and directories: files: The r permission means that the file is readable. The w permission means that the file contents may be changed (writable file). The x permission indicates an executable file. directories: The r permission means that a listing of the files in the directory may be displayed. The w permission allows the contents of the directory to be changed. Note: files you may not be the owner of or files of which the write-permission is turned off, but that are in a writable directory, can be removed or given another name, even though you may not be allowed to changed them. The x permission allows you to go through the directory, i.e., to use a name of a file in the directory. Whether you can do something with the file depends on the permissions of the file itself. 12

13 File attributes can be changed in the following ways: chown loginname file... Make the user with the given account name the owner of the file(s) mentioned. Only the current owner can execute this command, but after doing this the ownership has gone to the other user. chgrp groupname file... Change the group of the file(s) mentioned. chmod mode file... Change the permissions of the file(s) of which you are owner with respect to readability / writability / executability for yourself (user), a fellow member of the group (group), or other users (all). The most comprehensible manner to request a mode of a file is as follows: indicator=rwx, where indictor is u (user) for the owner, g (group) for members of the group (minus the owner), and o (others) for the rest of the people having a user account. For example, the permissions rw-r--r-- of the file voorbeeld come into being by the command chmod u=rw,g=r,o=r voorbeeld Combinations are also permitted: go means anyone but the owner. Just before you could have entered chmod u=rw,go=r voorbeeld. Instead of ugo you may also use the token a. For example, assuming that the current permissions of the example file are rw-r--r--, the command chmod a=r voorbeeld makes the file readable and only readable for everyone. Instead of the equal sign (=) you may use the plus sign (+) and the minus sign (-) to add and remove permissions, respectively. A small session to illustrate this: (bvandijk@gene 1) chmod u=rw,g=r,o=r voorbeeld (bvandijk@gene 2) ls -l voorbeeld -rw-r--r-- 1 bvandijk st-wis 21 Aug 20 09:24 voorbeeld (bvandijk@gene 3) chmod a=rw voorbeeld (bvandijk@gene 4) ls -l voorbeeld -rw-rw-rw- 2 bvandijk st-wis 12 Aug 20 09:24 voorbeeld (bvandijk@gene 5) chmod go-w voorbeeld (bvandijk@gene 6) ls -l voorbeeld -rw-r--r-- 2 bvandijk st-wis 12 Aug 20 09:24 voorbeeld (bvandijk@gene 7) chmod g+w voorbeeld (bvandijk@gene 8) ls -l voorbeeld -rw-rw-r-- 2 bvandijk st-wis 12 Aug 20 09:24 voorbeeld By the way, the default permissions of a new file or directory are such that only the owner can change them. We end this section with a listing of the most important options of the ls command. Options may be combined: instead of ls -l -a you may also enter ls -la. Option Meaning -l display all file attributes -a display all files, including the hidden files starting with a dot -t sort listing of files on time stamp -u sort listing of files on last access (use) -d if an argument is a directory, list only its name and not its contents -R for each directory, list all files in this directory and all files in subdirectories, etc. (R stands for recursively ) 13

14 Exercise 3.1. Listing of file permissions 1. Make a long listing of the root directory. 2. In which subdirectory everybody has write permission? Check that you can indeed create a file in this subdirectory. 3. Check in the directory /bin which files are executable. 4. Check which files are executable in /usr/local/bin. Exercise 3.2. Changing permissions of files and directories 1. Create a file so that you can practise the use of the chmod command. 2. Change the permissions of the file such that everybody may do anything. 3. Change the permissions of the file such that the owner can only read the contents of the file. 4. Change the permissions of the file such that anyone in the group but the owner cannot do anything with the file. 5. Change the permissions of the file such that the owner cannot overwrite the file. 6. Remove the file and repeat the above tasks with a directory. Particularly interesting is a directory with only the execute permission. 3.2 Wildcards Suppose that you want to get with ls an overview of all matlab files of which the name ends with.m in the current directory. With ls alone you may see so many files that it is not easy anymore to find the Matlab files. More convenient is in this case the command ls *.m that lists all file names in the current directory that end with.m. The asterisk in this example is called a wildcard and stands for any sequence of tokens. One exception: if the asterisk appears at the beginning or immediately after a slash /, then you may not substitute a dot. as first token. So: if the current directory contains a file with name.voorbeeld.m, then it is not listed with the command ls -a *.m. More examples of wildcards: all file names that begin with opg can be specified with opg* all file names that begin with prog and end with.m can be specified with prog*.m all file names that do not begin with. can be specified with * The question mark is interpreted by the shell as any single character. You can make matching of characters more specific by enclosing a set of characters in square brackets. Examples explain this better: opg? stands for all file names that begin with opg followed by one token. opg[12] stands for all file names that begin with opg followed by the single digits 1 or 2, i.e., the file names opg1 and opg2, but not opg3 or opg12. opg[0-7] stands for all file names that begin with opg followed by one of the digits 0, 1,..., 7. opg?? stands for all file names that begin with opg followed by exactly two tokens. [A-Z]* stands for all file names that begin with an uppercase character. 14

15 Note that you may use more than one wildcard at the same time and compose some pattern. Such patterns can be used in other commands as well. For example, cp *[0-9] /tmp copies all files whose name end with a digits to the directory /tmp and mv *.. moves all files in the current directory up one level in the hierarchy of files. If you do want the shell to take an asterisk or a question mark literally you must enclose it by double quotes. Exercise 3.3. Using wildcards 1. make a listing of all files in your home directory that have a file name beginning with a dot. 2. copy all non-hidden files of your home directory to /tmp and then remove all files in /tmp of which you have ownership. 3.3 Locating and comparing files Sometimes it is difficult to find back a file. Then the locate command may be helpful. With the command locate pat you search one or more databases of file names and display the file names that contain the pattern pat. An example: (heck@gene 1) locate .html /home/gwhakker/opera/help/us .html /home/jberk/opera/help/en-gb/us .html /home/jkottman/public_html/ .html... Such a listing of files can become very large and contain many files that have nothing to do with your own files. In this case it is better to pipe the result of one command as arguments of another command. Use the vertical bar for a pipe. For example, the above output of locate can be piped to the grep command in order to select on the pattern us . (heck@gene 1) locate .html grep us /home/gwhakker/opera/help/us .html /home/jberk/opera/help/en-gb/us .html /home/sputten/opera/help/us .html /home/vdmeer/opera/help/us .html This command grep deserves more explanation. It is a program that reads a text stream and filters out the lines matching some pattern. The command grep FNWI < file1 > file2 reads file1 and writes all lines in which the word FNWI occurs to file2. This is called I/O redirection: input and output have nothing to do anymore with the terminal-window. If we leave out in the above command the part > file2, then the output is send to standard output, i.e., in most cases to the terminal window. You can leave out the less-then sign, too: grep FNWI file1. In this case, the lines containing the pattern FNWI in file1 are listed. Wildcards can be used: with grep FNWI * you search in all files in the current directory for lines containing the word FNWI. Unix displays the file name and the line in which the pattern is found. Sometimes you want to compare the contents of two files. The command diff file1 file2 displays all lines in which the files differ from each other. If no differences are found, Unix is very brief: no message, but only a new prompt to enter a command. Exercise 3.4. I/O redirection Find out what the following two commands do: ls more ls > lijst 15

16 3.4 Secure ftp ftp stands for file transfer protocol. This makes it possible to download and upload files from one computer to another. For example: you can ftp files from your computer at home to the students server gene. Ftp can also be arranged via a web browser such as Mozilla. Enter the web address ftp://username@gene.science.uva.nl (instead of username use your own loginname). An example session of jvdwindt on workstation ow34. We interrupt the session for explanation. First a secure ftp-connection must be established between the workstation ow34 (local computer) and the students server gene (remote computer). You must login. (jvdwindt@ow34 1) sftp gene.science.uva.nl Connecting to gene.science.uva.nl. Warning: Permanently add gene.science.uva.nl (RSA) to the list of known hosts jvdwindt@gene.science.uva.nl s password: sftp> Once the connection is established, many commands can be entered. For example, ls to list the files in the working directory on the remote computer. sftp>ls * drwx--l--- 2 jvdwindt st-inf 1024 Aug 30 15:46 nsmail drwx--x--x 11 jvdwindt st-inf 512 Aug 10 08:55 public_html -rw-r--r-- 1 jvdwindt st-inf 21 Sep 1 10:35 testfile -rw-r--r-- 1 jvdwindt st-inf 21 Sep 1 10:37 testfile2 sftp> There exist two commands to transfer a file: get to transfer a file from the remote computer to the local computer and put for file transfer in the other direction. With get and put only one file can be sent at the same time. Use wildcards if you want more files to be transfer in a single command. sftp> get testfile testfile 100% 8 1.3KB/s 00:00 ftp> get * Fetching /home/jvdwindt/mail to Mail cannot download non-regular file: /home/jvdwindt/mail Fetching /home/jvdwindt/testfile testfile 100% 8 1.3KB/s 00:00 Fetching /home/jvdwindt/testfile2 testfile2 100% 8 1.3KB/s 00:00 Our remote computer is a Unix machine and the command cd works as usual. To change the working directory on the local computer you must use the lcd (l stands for local ) command. sftp> lcd public_html sftp> cd nsmail sftp> put index.html index.html 100% 36KB 622KB/s 00:00 You leave the sftp program by entering bye or quit. ftp> bye (jvdwindt@ow34 2) When you use ftp you must be careful and think at each command at which computer you entered the sftp command (local machine) and to which computer you established the connection (remote machine). The most important commands that you can use inside sftp are: 16

17 sftp command cd dir lcd dir put file get file lls dir bye quit Explanation change directory on remote computer. change directory on local computer. copy file from local to remote computer. copy file from remote to local computer. display local directory listing leave the sftp program leave the sftp program Hint: if you transfer files from Unix machines to Windows PC s and vice versa, then you will soon notice that text files have different format: DOS format on a PC and ISO format in Unix. In order to quickly change from one format into the other you can use the commands dos2unix and unix2dos. Exercise 3.5. File transfer Transfer a file and a small directory from the PC you are working on to the students server gene. Hereafter, transfer the file and directory back to your PC in the directory D:\temp. 3.5 File compression and archives To save disk space you are advised to compress a file that you do not need instant access to anymore or that does not have to be stored in readable format. Especially PostScript file in compressed format much smaller. There exist several command for file compression, the two most popular of which are: compress and uncompress to pack and unpack a file, respectively. A compress file can be recognized by the extension.z gzip and gunzip are commands from the GNU series. A compress file can be recognized by the extension.gz. We recommend the use of the GNU commands gzip and gunzip because gzip creates in general a more compact file than compress does. By the way, recent versions of the WinZip program on a PC recognize a gzipped file. The following computer session illustrates how spectacular the saving of disk space can be. (heck@gene 1) ls -l intro* -rw-r--r-- 1 heck ams Aug 21 11:03 intro.ps (heck@gene 2) compress intro.ps (heck@gene 3) ls -l intro* -rw-r--r-- 1 heck ams Aug 21 11:03 intro.ps.z (heck@gene 4) uncompress intro.ps.z (heck@gene 5) gzip intro.ps (heck@gene 6) ls -l intro* -rw-r--r-- 1 heck ams Aug 21 11:03 intro.ps.gz (heck@gene 7) gzip intro.ps (heck@gene 8) ls -l intro* -rw-r--r-- 1 heck ams Aug 21 11:03 intro.ps.gz (heck@gene 9) gunzip intro.ps.gz (heck@gene 10) ls -l intro* -rw-r--r-- 1 heck ams Aug 21 11:03 intro.ps The above commands only work for a single file. There exist other commands for archiving and compressing a complete directory in a single file: zip and unzip, or tar. To start with the first two commands: suppose that you have in the current directory a subdirectory, say with the name testdir, then the command zip -r testdir testdir creates the archive testdir.zip in which all the files of the directory testdir are stored in compressed format. The command unzip testdir.zip uncompresses the files and creates the directory testdir with the given contents. 17

18 The command tar (tape archive ) can be used to create a so-called tar file, to change it, or to unpack such a file. A tar file is usually a file in which a complete directory structure is stored. The current directory (and the complete directory structure beneath) can be packed into a tar file by tar cvf file.tar. and can be unpacked (for example, in a different directory) by tar xvf file.tar The character c and x stand here for create and extract. The character v stands for verbose and make information about the process of packing and unpacking available to you. You may leave this option out of your commands. The next computer session illustrates the use of zip, unzip and tar. (heck@gene 1) ls -ld test* rwxr-xr-x 2 heck ams 512 Aug 21 12:24 testdir (heck@gene 2) zip -r testdir testdir adding: testdir/ (stored 0%) adding: testdir/intro.ps (deflated 67%) (heck@gene 3) tar cvf testdir.tar testdir a testdir/ 0K a testdir/intro.ps 3117K Now we have created two archives, of which the zip archive is already compressed and the tar archive can be compressed afterwards. (heck@gene 4) ls -l test* -rw-r--r-- 1 heck ams Aug 21 12:25 testdir.tar -rw-r--r-- 1 heck ams Aug 21 12:25 testdir.zip testdir: total rw-r--r-- 1 heck ams Aug 21 11:03 intro.ps (heck@gene 5) gzip testdir.tar (heck@gene 6) ls -l testdir.* -rw-r--r-- 1 heck ams Aug 21 12:25 testdir.tar.gz -rw-r--r-- 1 heck ams Aug 21 12:25 testdir.zip It is possible via options, filters, and I/O redirection to create a tar file and to compress its result in one command: (heck@gene 7) tar cvf - testdir gzip -c > test.tgz creates the compressed tar file test.tgz from the directory testdir. The next command unpacks the archive: (heck@gene 8) gunzip -c test.tgz tar xvf - Exercise 3.6. File compression and archives 1. Use the quota command to find out what disk quota holds for you. 2. Make sure that you have some PostScript files with extension.ps, for example by copying a single PostScript file to other files with extension.ps 3. Zip these PostScript files into one zip archive. Check your quota. Unzip the archive and check again your disk quota. 4. Tar the PostScript files into one tar file and compress the result. Remove the PostScript files and check your disk quota. 5. Remove the PostScript files that you do not need anymore. 18

19 4 Managing processes 4.1 Working with more programs at the same time UNIX is a multi-processing operating system: besides the shell, which is always present, other processes can be active at the same time. Normally the shell is in a waiting status when another process runs, but by ending a command line with an ampersand &, the process is put running in the background and the shell comes immediately with a user prompt. We speak about starting a job in the background. For example, in order to start Matlab in a terminal window and to be able to use the same terminal window for entering other commands, you enter matlab &. You see in the terminal window something like: (heck@gene 1) matlab & [1] 3886 (heck@gene 2) Before the second Unix prompt you see between square brackets the job number (in this case there is only one job) and the process ID number (a unique label of the process). In subsequent commands you can refer to the job or the process by the job number and the process ID number. Let us also start a text editor in the background : (heck@gene 2) emacs & [2] 3894 (heck@gene 3) A new text window appear, but you can still enter commands in the terminal window. This feature makes a Unix system so attractive: you can work smoothly with various programs at the same time. While a heavy computation is running in one window, you can already be busy with writing a report about this work. 4.2 Inquiring about the process status You can ask which processes you have in use and what their status is by the ps command (process status). (heck@gene 3) ps PID TTY TIME CMD 3920 pts/5 0:06 Matlab 3925 pts/5 0:00 emacs 3916 pts/5 0:00 tcsh You get more information with the option -l (heck@gene 4) ps -l F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD 8 S f5ffc f630893e pts/5 0:06 matlab 8 S f5fab f pts/5 0:00 emacs 8 S f5ffcb f5ffcd20 pts/5 0:00 tcsh You get one line of information for each process. The status of each process is stated in the first column under S. The possibilities are: 19

20 Job status O R S T Z Explanation Operational, i.e., the process is currently running on a processor. Runnable, i.e., the process is active or is waiting for its turn in a waiting queue. Sleeping. The non-active process waits for an event. Terminated. For example interrupted by the user or stopped because input was asked from the terminal while the process ran in the background. Zombie process. These are finished process, but this change of status has not been noticed (yet). Other part of the information obtained by ps -l are: UID The user ID number identifies the user that started the process. PID The process ID number is the unique process number that distinguishes this process from others. PPID The parent process ID number indicates the process inside which a given process was started. In our example you see that the commands matlab and emacs have been started in the shell tcsh with process ID number SZ The memory space allocated by the process (SIZE in pages; 1 page = 4Kbytes). TIME The computing time (in seconds) that the process consumed since its start (waiting time excluded). TTY The terminal on which the process runs. CMD The command that started the process. The ps commando has many options. In combinations of options all options must be collected into the first argument. Three much used options of ps (besides -l) are: -a displays information about all processes with terminals, also of other users. -A (identical to -e) displays information about all processes, including those without an associated terminal. -u display information about all processes of a user, also of processes running in a separate terminal. If you only want information about processes that have been started in the current shell, then it is more convenient to use the jobs command. instead of ps. With the option -l the process ID number are also displayed. (heck@gene 5) jobs -l [1] Running matlab [2] Running emacs The plus sign at the matlab job indicates that this is the current job. This has only meaning when controlling multiple jobs. See the next subsection. 4.3 Job control A job running in the foreground or a process that was started in the normal way, i.e., without the ampersand &, can be put into the background by interrupting the process with ^Z (Control-z: When the cursor is in the terminal window in which the job was started, pressing the z key while holding the Control key down), followed by entering bg (background). In the 20

21 opposite direction, you can bring a process from the background into the foreground with fg (foreground). There can only run one process in the foreground. A process in the foreground can often be terminated by ^C (Control-c). For example, enter in a terminal window the command clock. A window with a clock displaying the current time appears. It disappears when you enter Control-c in the terminal window in which you started the clock. A process in the background can be stopped with the stop command. Let us demonstrate this in a continuation of the previously started computer session. (heck@gene 6) emacs ^Z Suspended (heck@gene 7) bg [3] emacs & After we first started the text editor emacs in the foreground in a terminal window, we have interrupted the process and put it into the background. There are now 3 jobs in the background: (heck@gene 8) jobs [1] + Running matlab [2] - Running emacs [3] Running emacs If you want to bring process 3 again into the foreground, then enter: (heck@gene 9) fg %3 emacs ^C (heck@gene 10) jobs [1] + Running matlab [2] - Running emacs If you do not specify a job number in the fg command, then the current job is brought into the foreground. This is the job that is marked in the job listing with a plus sign. Above we have interrupted the text editor by Control-c. There are now only two processes left. Let us interrupt the matlab program, too. Because there is only one process with this name, you can also use the job name instead of the job number. To get rid of the job you can enter the kill command. (heck@gene 11) stop %matlab [1] + Suspended (signal) matlab (heck@gene 12) jobs [1] + Suspended (signal) matlab [2] - Running emacs (heck@gene 13) kill %1 (heck@gene 14) [1] Terminated matlab (heck@gene 14) jobs [2] + Running emacs Individual processes listen to their process ID number: For example kill would have killed the matlab process 3920, too. When you try to log out while there are still suspended jobs, then the shell will tell you so via the message There are stopped jobs. By entering logout once more you can really finish the computer session. 21