CS2204: Introduction to Unix

Transcription

1 What is Unix? CS2204: Introduction to Unix January 19 th, 2004 Class Meeting 1 * Notes adapted by Christian Allgood from previous work by other members of the CS faculty at Virginia Tech A modern computer operating system Operating System a program that acts as an intermediary between a user of the computer and the computer hardware Software that manages your computer s resources (files, programs, disks, network) Examples: Windows, MacOS, Solaris, BSD, Linux (e.g. Mandrake, Red Hat, Slackware) Modern Stable, flexible, configurable, allows multiple users and programs Christian Allgood, Virginia Tech, Why Unix? Used in many scientific and industrial settings Open-source operating system (OS) Huge number of free and well-written software programs Excellent programming environment Internet servers and services run on Unix Roughly 65% of the world s web servers are Linux/Unix machines running Apache Brief History of Unix Ken Thompson and Dennis Richie originally developed the earliest versions of Unix at Bell Labs for internal use in the 1970s Simple and elegant Written in a high-level language instead of assembly language Small portion written in assembly language (kernel) Remaining code written in C on top of the kernel Christian Allgood, Virginia Tech, Christian Allgood, Virginia Tech, Unix Variants Two main threads of development Berkeley software distribution ( Unix System Laboratories ( Sun: SunOS, Solaris SGI: Irix FreeBSD, OpenBSD, NetBSD Hewlett-Packard: HP-UX Apple: OSX (Darwin) Linux (many flavors) Brief History of Linux Andrew Tanenbaum, a Dutch professor developed MINIX to teach the inner workings of operating systems to his students In 1991 at the University of Helsinki, Linus Torvalds, inspired by Richard Stallman s GNU free software project and the knowledge presented in Tanenbaum s operating system, created Linux, an open-source, Unix-based operating system Over the last decade, the effort of thousands of opensource developers has resulted in the establishment of Linux as a stable, functional operating system Christian Allgood, Virginia Tech, Christian Allgood, Virginia Tech,

2 Layers in a Unix-based System Unix Structure User Interface Library Interface System calls Users Standard Utility Programs (shells, editors, compilers, etc.) Standard Library (open, close, read, write, etc.) Unix Operating System (process/memory management, file system, I/O) Hardware (CPU, memory, disks, terminals, etc.) user mode kernel The kernel is the core of the Unix operating system, controlling the system hardware and performing various lowlevel functions. Other parts of a Unix system (including user programs) call on the kernel to perform services for them. The shell accepts user commands and is responsible for seeing that they are carried out. Christian Allgood, Virginia Tech, Christian Allgood, Virginia Tech, Unix Structure (cont.) Getting Started Over two hundred utility programs or tools are supplied with the Unix system. These utilities (or commands) support a variety of tasks such as copying files, editing text, performing calculations, and developing software. This course will introduce a limited number of these utilities and tools, focusing on those that aid in software development. Logging in to a Unix machine requires an account on that system After logging in, some information about the system will be displayed, followed by a shell prompt, where commands may be entered $ % # username@hostname> hostname% Christian Allgood, Virginia Tech, Christian Allgood, Virginia Tech, The Shell Command Syntax The shell is the program you use to send commands to the Unix system Some commands are a single word who date ls Others use additional information cat textfile ls l Commands must be entered exactly command options argument(s) Options modify a command s execution Arguments often indicate upon what a command should act (often filenames) Christian Allgood, Virginia Tech, Christian Allgood, Virginia Tech,

3 Example Command: ls Logging Out ls l ls a ls la ls a; ls l ls al textfile1 ls al textfile1 textfile2 ls al directory Always log out when you are done Use the exit command to log out of a shell Note: If you are running a window system, logging out of a shell only ends that shell. You must also log out of the window system, typically using a graphical menu. Christian Allgood, Virginia Tech, Christian Allgood, Virginia Tech,

4 Unix Filesystem Unix Filesystem January 26 th, 2004 Class Meeting 2 The filesystem is your interface to: physical storage (disks) on your machine storage on other machines output devices Everything in Unix is a file (programs, textfiles, peripheral devices, terminals) There are no drive letters in Unix the filesystem is a logical view of the storage devices * Notes adapted by Christian Allgood from previous work by other members of the CS faculty at Virginia Tech Working Directory Working Directory the current directory in which you are located pwd (print working directory) command outputs the absolute path (more later) of your working directory Unless you specify another directory, commands will assume that you want to operate on the working directory Home Directory Directory for users to store personal files At login, your working directory will be set to your home directory The path (more later) to your home directory can be referred to by the ~ (tilde) symbol The home directory of user1 is represented by ~user1 Unix File Hierarchy Unix Paths Root Directory: / Directories may contain plain files or other directories Result is a tree structure for the filesystem Unix does not recognize any special filename extensions / bin home etc user1 user2 textfile lab1txt cs2204 lab2txt Separate directories by / Absolute Path start at root and follow the tree Example: /home/user1/textfile ~users1/textfile ~/textfile / bin home etc user1 user2 textfile lab1txt cs2204 lab2txt 1

5 Unix Paths (cont) Some Standard Directories Relative Path start at working directory.. level above. working directory / bin home etc user1 user2 textfile lab1txt cs2204 lab2txt / - root directory /bin standard commands and utilities /dev block and character device directory /etc host-specific configuration files and directories /home users directory /lib library directory /sbin system commands and utilities (needed to boot) /tmp temporary files /usr most user utilities and applications /var files that vary as the system runs (logs, spools) Changing Directories cd changes the working directory cd <directory_path> can use absolute or relative path names cd without any arguments is the same as: cd ~ Examples: cd /home/user1 cd../../user1 Output of ls -lf lrwxrwxrwx 1 callgood Grads 20 Jan 24 20:16 home -> /home/grads/callgood/ -rw-r--r-- 1 callgood Grads Sep 22 10:07 atoll.jpg drwxr--r-- 2 callgood Grads 64 Jan 24 18:33 cs2204/ permissions user group modified date filename file type size We ll keep coming back to this slide Types of Files Plain ( - ) Most files, binary or text Directory ( d ) Directory is actually a file Points to another set of files Link ( l ) Pointer to another file or directory Special b block device (disks, CD-ROM) c character device (keyboard, joystick) Manipulating Files touch <file> create a new file or change last modified date mv <file1> <file2> rename file1 as file2 mv <file1> <dir> move file1 into the dir directory mv <file1> <dir/file2> move file1 into dir and rename as file2 cp <file1> [<file2> <dir> <dir/file2>] copy file with new name, into directory, or both rm [-i] <file(s)> remove file or list of files 2

6 Creating and Removing Directories mkdir <directory_name> create a subdirectory of the current directory rmdir <directory_name> remove directory only works for empty directories rm -r <directory_name> remove directory and all of its contents, including subdirectories Creating Links ln -s <existing_file> <link_name> creates a symbolic link link_name will be a pointer to existing_file which may be in another directory or even on another physical machine File Ownership Each file has a single owner chown command can be used to change the owner; usually only root user can use this command Users can also belong to various groups Groups may have different permissions than everyone else File Permissions Permissions are used to allow or disallow access to files or directories Three types of permission: Read ( r ) Write ( w ) Execute ( x ) Permission exists on three levels: User ( u ) Group ( g ) World ( o ) File Permissions (cont) chmod <mode> <file(s)> chmod 700 textfile r w x user r w x group r w x world chmod g+rw textfile ugo +/- rwx File Modification Date Last time a file was changed Useful when... there are many copies of a file many users are working on a file touch command can be used to update the modification date to the current date (or to create a file if it doesn t exist) 3

7 Looking at File Contents Getting Help on Unix Commands cat <filename(s)> short for concatenate output the contents of the file all at once more <filename(s)> output the contents of the file one screen at a time allows forward and backward search man <command_name> shows all of the documentation for a command (more-style output) apropos <keyword> shows you all of the commands with the specified keyword in their description 4

8 Text Editing Text Editing February 2 nd, 2004 Class Meeting 3 Last week, you learned to manipulate files in the file system (cp, mv, rm, ln) and to view their contents (cat, more) How do you modify the contents of files? Unix editors work with plain ASCII text files: vi, emacs, pico Windows systems use their own windowbased editors: Notepad, Wordpad Why vi? Availability included on practically all Unix/Linux systems Command-line editor, allowing use with remote login Simple and powerful vi Basics Invoke with: vi [filename(s)] Editor takes over screen Various modes: command mode characters send commands to editor (saving, quitting, searching, replacing) insert mode append mode typing changes the text change mode Command Mode vi starts in this mode From other modes, typing Esc will enter command mode Commands for: Cursor movement Editing File operations Searching Entering other modes Cursor Movement Up: k Down: j Left: h Right: l Larger Movements n{j k l h} move n characters/lines up/down/left/right Ctrl-F, Ctrl-B page down, page up w move to the beginning of the word :n move to line n 0, $ move to beginning or end of file 1

9 Editing Commands File Operations u undo last typing x delete current character dd delete current line dw delete current word rx replace current character with x yy copy current line p paste copied/deleted items J join two lines can be preceded by a number to perform operation multiple times ZZ, :wq save and quit :w save :w <filename> save as <filename> :q quit :q! quit without saving :e <filename> load another file :n load next file Search /string search for string?string search backwards for string n repeat previous search N repeat search in opposite direction % find match of current (,[, or { Entering Modes from Command Mode i insert text before current character I insert text at beginning of line a append text after current character A append text at end of the line o open line above current line O open line below current line cw change (overwrite) current word C change text after cursor Additional Info More Shell Commands Unix in a Nutshell Chapter 8 lists all vi commands vi uses all lowercase letters except v (and many uppercase characters and punctuation) for commands Be careful! Learn undo commands emacs is covered in Chapter 7 ps list current processes top dynamic display of system s utilization by processes kill terminate a process time keep timing information for a process 2

10 Redirecting stdout I/O Redirection and Regular Expressions February 9 th, 2004 Class Meeting 4 Instead of displaying output to the terminal, you can redirect the output to another file using the > operator >> operator is used to append output to an existing file Examples: man ls > ls_help.txt echo $PWD > current_directory cat file1 >> file2 Redirecting stdin Pipes and Filters Instead of reading from a terminal, you can tell a program to read from another file using the < operator Examples: mail user@domain.com < message interactive_program < command_list Pipe a way to send the output of one command to the input of another Filter a program that takes input and transforms it in some way wc gives a count of words/lines/characters grep searches a line with a given string (regular expression) more sort sorts lines alphabetically or numerically Examples of Filtering What is a Regular Expression? ls -la more cat file wc man ksh grep history ps aux grep user1 wc -l who sort > current_users A regular expression (RE) is a pattern of metacharacters that define a set of strings Each of these strings is said to match the regular expression Pattern matching is useful in many real-world situations: searching for a file on the filesystem finding and replacing text in a file extracting data elements from a database 1

11 Unix programs that use REs Basic vs. Extended REs grep (search within files) egrep (grep with extended REs) vi/emacs (text editors) sed (stream editor) awk (pattern scanning language) perl (scripting language) In basic regular expressions the metacharacters?, +, { }, ( ), and have no special meaning (grep) To give them special meaning, use the escaped versions: \?, \+, \{ \}, \( \), and \ When using extended regular expressions, these metacharacters have special meaning grep -E = egrep Using egrep Metacharacters egrep pattern filename(s) To be safe, put quotation marks around your pattern Examples: egrep abc textfile egrep -i abc textfile egrep -v abc textfile egrep -n abc textfile Period (.): matches any single character a.c matches abc, adc, a&c, a;c, u..x matches unix, uvax, u3(x, Asterisk (*): matches zero or more occurences of the previous RE not the same as wildcards in the shell ab*c matches ac, abc, abbc, abbbc,.* matches any string Metacharacters (cont) Metacharacters (cont) Plus (+): matches one or more occurences of the preceding RE ab+c matches abc, abbc, abbc, but not ac Question Mark (?): matches zero or one occurences of the preceding RE ab?c matches ac or abc, but not abbc Logical Or ( ): matches RE before or RE after abc def matches abc or def Caret (^): beginning of line ^D matches all strings starting with D Dollar Sign ($): end of line d$ matches all strings ending with d Backslash (\): escapes other metacharacters file\.txt matches file.txt, but not fileatxt 2

12 Metacharacters (cont) Metacharacters (cont) Square Brackets ([ ]): indicates a set/range of characters any characters in the set will match ^ before the set negates the set - indicates range Examples: [ff]un matches fun or Fun b[aeiou]g matches bag, beg, big, bog, bug [A-Z].* matches any string beginning with a capital letter [^abc].* matches any string not containing an a, b, or c Parentheses ( ): used to group REs when using other metacharacters a(bc)* matches a, abc, abcbc, abcbcbc, (foot base)ball matches football or baseball Braces ({ }): specify the number of repetitions of an RE [a-z]{3} matches three lowercase letter m.{2,4} matches strings with m followed by between 2 and 4 characters What do these mean? egrep ^B.*s$ file egrep [0-9]{3} file egrep num(ber)? [0-9]+ file egrep word file wc -l egrep [A-Z].*\? file What if grep was used instead? Remember, RE matches largest string --color option illustrates the largest match 3

13 Why Window Systems? Unix Window System February 16 th, 2004 Class Meeting 5 Increased usability due to Visibility Graphical representation of programs Seeing multiple environments at once Direct Manipulation Enables powerful graphics programs Pixar: Window Systems and Unix Most Unix users can be considered experts, and are fiercely protective of the command line All current Unix systems have a built-in window system, due to the obvious advantages of a graphical user interface X Windows Practically all Unix window systems are based on X Windows (XFree86) Standard Version: X11R6 Complex system with many parts X11: Manages the screen space Draws simple graphics Assigns rectangular regions to various programs X s Client-Server Architecture X is actually meant to work over the network X server: software that runs on the machine where the program s output will be displayed X client: program running on the same or another machine Client sends drawing and other X commands to the server, which displays the results Historical Use of X Users sat at X terminals graphical terminals that only knew how to run an X server They logged in to other Unix machines remotely and ran X clients there This gave users the benefits of a window system without the need for a fullfeatured computer on every desk 1

14 Features of X Desktop Environments Transparent remote execution Gives programs their own virtual screen Includes important windowing concepts Window damage Window reveal events Backing store X11 programs are highly portable Purpose is to integrate the various interface components into a cohesive, consistent whole Provides file managers (Nautilus), handles desktop icons, and overall desktop operations Can handle window operations or use a window manager Popular desktop environments GNOME (GTK+) KDE (Qt) XFce (GTK+) Window Managers Not part of X11 itself; run on top of X11 Place borders and decorations on windows Handle input from users There are many, many choices with different look and feel 2

15 Shell Characteristics Unix Shell Environments February 23rd, 2004 Class Meeting 6 Command-line interface between the user and the system Automatically starts when you log in, waits for user to type in commands A Unix shell is both a command interpreter, which provides the user interface to the rich set of utilities, and a programming language, allowing these utilities to be combined. Main Shell Features Various Unix Shells Interactivity Aliases File-name completion Scripting language Allows programming (shell scripting) within the shell environment Uses variables, loops, conditionals, etc. Next lecture sh (Bourne shell, original Unix shell) ksh (Korn shell) csh (C shell, developed at Berkeley) tcsh bash (Bourne again SHell) Differences mostly in level of interactivity support and scripting details Bourne Again SHell Changing Your Shell We will be using bash as the standard shells for this class Superset of the Bourne shell (sh) Borrows features from sh, csh, tcsh, and ksh Created by the Free Software Foundation On most Unix machines (including the lab)... which bash chsh On some machines... Ypchsh 1

16 Environment Variables A set of variables the shell uses for certain operations Variables have a name and a value Current list can be displayed with the env command A particular variable s value can be displayed with echo $<var_name> Environment Variable Examples Some interesting environment variables: $HOME /home/grads/callgood $PATH /usr/local/bin:/bin:/usr/bin:/us r/x11r6/bin $PS1 \u@\h:\w\$ $USER callgood $HOSTNAME mango.cslab.vt.edu $PWD /home/grads/callgood/cs2204 Setting Environment Variables Set a variable with <name>=<value> Examples: PS1=myprompt> PS1=$USER@$HOSTNAME: PS1= multiple word prompt> PATH=$PATH:$HOME/bin PATH=$PATH:~ DATE=`date` Aliases Aliases are used as shorthand for frequently-used commands Syntax: alias <shortcut>=<command> Examples: alias ll= ls lf alias la= ls la alias m=more alias up= cd.. alias prompt= echo $PS1 Repeating Commands Use history command to list previously entered commands Use fc l <m> <n> to list previously typed commands from m through n Editing on the Command Line bash provides a number of line editing commands; many are the same as emacs editing commands M-b Move back one word M-f Move forward one word C-a Move to beginning of line C-e Move to end of line C-k Kill text from cursor to end of line 2

17 Login Scripts You don t want to enter aliases, set environment variables, etc., each time you log in All of these things can be done in a script that is run each time the shell is started Login Scripts (cont) For bash, order is... /etc/profile ~/.bash_profile ~/.bash_login (if no.bash_profile) ~/.profile (if neither are present) ~/.bashrc After logout... ~/.bash_logout Example.bash_profile (partial) Example.bashrc (partial) #.bash_profile # include.bashrc if it exists if [ -f ~/.bashrc ]; then. ~/.bashrc fi # Set variables for a warm fuzzy environment export CVSROOT=~/.cvsroot export EDITOR=/usr/local/bin/emacs export PAGER=/usr/local/bin/less #.bashrc # abbreviations for some common commands alias f=finger alias h=history alias j=jobs alias l='ls -lf' alias la='ls -alf' alias lo=logout alias ls='ls -F' Login Shell Background Processing /etc/profile ~/.bash_profile ~/.bashrc interactive shell ~/.bashrc login shell interactive shell ~/.bashrc interactive shell ~/.bashrc Allows you to run your programs in the background callgood@mango:~/$ emacs textfile& callgood@mango:~/$ 3

18 stdin, stdout, and stderr Each shell (and in fact all programs) automatically open three files when they start up Standard input (stdin): Usually from the keyboard Standard output (stdout): Usually to the terminal Standard error (stderr): Usually to the terminal Programs use these three files when reading (e.g. cin), writing (e.g. cout), or reporting errors/diagnostics 4

19 Shell Script (Program) Shell Scripting March 1st, 2004 Class Meeting 7 What is a shell script? A series of shell commands placed in an ASCII text file Commands include... anything you can type on the command line shell variables control statements (if, while, for,... ) Script Execution Simple Script Provide script as an argument to the shell program (i.e. bash my_script) Or specify which shell to use within the script First line of the script: #!/bin/bash Make sure that the script is executable Run directly from the command line No compilation is necessary #!/bin/bash echo Hello, World! cd ~ pwd Output: Hello, World! /home/grads/callgood Shell Variables Numeric Variables Numeric Strings Arrays var refers to the name of the variable, $var refers to the value var=100 # sets the value to 100 (( var2=1+$var )) Variable names begin with an alphabetic character and can include alpha, numeric, and the underscore Integer variables are the only pure numeric variables that can be used in bash Declaration and setting value declare i var=100 Numeric expressions are enclosed in double parentheses (( var+=1 )) Operators are the same as in C/C++ +, -, *, /, %, &,, <, >, <=, >=, ==,!=, &&, 1

20 String Variables Unless variables are explicitly declared as another type, they are considered to be strings var=100 makes the var the string 100 However, placing the variable within double parentheses will treat it as an integer (( var2=1+$var )) String Variables (cont) Using substrings ${string:n} ${string:5} # first five chars ${string:-2} # last two chars ${string:n:m} ${string:0:4} # first to fifth ${string:1:3} # second to fourth ${#string} # length of string Concatenating strings string_var1= $string_var1 $string_var2 Array Variables Array is a list of values do not have to declare size Reference a value by ${a[index]} ${a[3]} # value in fourth position $a # same as ${a[0]} Use the declare a command to declare an array declare a sports sports=(basketball football soccer) sports[3]=hockey Array Variables (cont) Array initialization sports=(football basketball) moresports=($sports tennis) ${array[@]} or ${array[*]} refers to the entire contents of the array echo ${moresports[*]} Output: football basketball tennis Command Line Arguments If arguments are passed to a script, they can be referenced by $1, $2, $3,... $0 refers to the name of the script $@ refers to all of the arguments not an array space separated group of strings $# refers to the number of arguments Output and Quoting echo command is used to output information to standard out echo n does not print a newline Shell interprets $ and `` within double quotes $ - variable substitution ` - command substitution echo `date +%D` # 03/01/04 Shell does not interpret special charactes within single quotes echo `date +%D` # `date +%D` 2

21 Redirecting Output to File Conditions Redirecting output to a file is the same as on the command line Examples echo $var > $OUTFILE # overwrite echo $var >> $OUTFILE # append String expansion echo $ \n\n\n echo $ \t If using integers: (( condition )) If using strings: [[ condition ]] Examples (( a == 10 )) (( b >= 3 )) [[ $1 = -n ]] [[ ($v!= fun) && ($v!= games) ]] Special conditions for file existence, file permissions, ownership, file type,... Conditions (cont) If Statement [[ -e $file ]] File exists? [[ -f $file ]] Regular file? [[ -d $file ]] Directory? [[ -L $file ]] Symbolic link? [[ -r $file ]] Read permission? [[ -w $file ]] Write permission? [[ -x $file ]] Execute permission? Syntax if condition then statements elif then statements else statements fi optional If Statement (cont) For Loops Example if [[ -r $file ]] then echo $file is readable elif [[ (-w $file) && (-x $file) ]] then echo $file is writeable and exectuable fi Syntax for var [in list] do statements done If list is omitted, $@ is assumed Otherwise ${list[*]}, where list is an array variable 3

22 For Loops (cont) Example colors=(yellow red blue green orange) for color in ${colors[*]} do echo $color done While Loops Syntax while condition do statements done The keywords break, continue, and return have the same meaning as in C/C++ Case Statements Syntax case expression in pattern1) statements ;; pattern2) statements ;;... *) statements ;; esac Case Statements (cont) Example case $1 in -a) # statements related to option a ;; -b) # statements related to option b ;; *) # all other options ;; esac Command Substitution Use the command of a command in a variable, condition,... `command` $(command) Example files=(`ls`) for file in ${files[*]} do echo $file done Functions Syntax function function_name { statements... } function_name() { statements... } Functions can take arguments, in the same way that the script takes arguments (i.e. $1 represents first argument, $2 the second,... ) 4

23 Functions (cont) Assignment Suggestions Example function add_two{ (( $sum=$1+$2 )) return $sum }... # calling a function add_two 1 3 echo $? $? represents the return value of the last function call or command Function definition needs to occur before the function is called in the script One approach to the script: Handle arguments, setting flags for each i.e., if recursive option is given, set a variable that indicates that it was set Write part of script that generates the makefile in a function; call this function recursively, if necessary (i.e. if recursive option was set) Assignment Suggestions (cont) Assignment Suggestions (cont) Within function, store the output of ls command in an array; loop through this array and test filename of each file in the current directory for valid filenames and/or extensions files=(`ls`) for file in ${files[*]} do # no need to check directories if [[ -f $file ]] then # test for valid filename and/or extensions fi done Remember that you need to know all of the valid filenames before you can begin writing any of the makefile; you may need to store parts of the output in temporary files, and then concatenate these files into your makefile; be sure that you remove and temporary files that you create 5

24 Software Development Process Development: g++ and make March 15th, 2004 Class Meeting 8 Creation of source files RCS Compilation and linking g++ and make Running and testing programs gdb Development Tools Compilation Process Creation of source files Text editors vi emacs Revision control systems rcs cvs Compilation and linking Compilers gcc g++ Automatic building tools make Running and testing programs (gdb) Linker-Loader Assembler Optimizer Compiler Preprocessor Links object files and libraries to form one executable file Translates assembly code into machine language code kept in object files Optional: Language-neutral code optimization Translates code into assembly language #include, #define, #ifdef Basic g++ Examples g++ hello.cc compile hello.cc produce executable a.out g++ -o hello hello.cc compile hello.cc produce executable hello g++ -o hello hello.cc util.cc compile hello.cc and util.cc produce exectuable hello Using Intermediate Files From any source file you can produce an object file to be linked in later to an executable g++ -c hello.cc g++ -c util.cc g++ -o hello hello.o util.o 1

25 g++ Options g++ Options (cont) -c compile source files, but do not link output is an object file corresponding to source file -o <file> puts output in file called <file> -g include debugging symbols in the output to be used later by debugging program (gdb) -Wall display all warnings program may still compile -D<macro> defines macro with the string 1 g++ Options (cont) Using make to Compile -l<name> include library called lib<name>.a -I<path> look for include files in the directory provided -L<path> look for libraries in the directory provided There are default directories in which g++ looks for include files and libraries With medium to large software projects containing many files, it is difficult to: Type commands to compile all the files correctly each time Keep track of which files have been changed Keep track of dependencies among files make automates this process Basic Operation of make Basic Makefile Example Reads a file called [Mm]akefile that contains rules for building a program if program is dependent on another file, then that file is built all dependencies are built, working backward through the chain of dependencies programs are only built if they are older than the files they depend u pon program : main.o iodat.o dorun.o g++ -o program main.o iodat.o dorun.o main.o : main.cc g++ -c main.cc iodat.o : iodat.cc g++ -c iodat.cc dorun.o : dorun.cc g++ -c dorun.cc 2

26 Parts of a Makefile Parts of a Makefile (cont) Dependency Lines contain target names and dependencies (optional) dependencies files targets Commands below dependency line always begin with a tab any commands that you can run on a command line program : main.o iodat.o dorun.o g++ -o program main.o iodat.o dorun.o main.o : main.cc g++ -c main.cc tab target command dependencies Macros and Special Variables Use macros to represent text in a makefile saves typing allows easy modification of the makefile Assignment MACRONAME = macro value Usage: ${MACRONAME} Special variables are used in commands $@ represents the target $? represents the dependencies Simplifying the Example OBJS = main.o iodat.o dorun.o CC = /usr/bin/g++ program : ${OBJS} ${CC} -o $@ $? main.o : main.cc ${CC} -c $? iodat.o : iodat.cc ${CC} -c $? dorun.o : dorun.cc ${CC} -c $? Invoking make Be sure that the description file is called makefile or Makefile is in the directory where the source files are make builds the first target in the file make target(s) builds target(s) Other options -n: don t run the commands, just list them -f <file>: use <file> instead of [Mm]akefile Suffix Rules Still tedious to specifically tell make how to build each.o file from.c/.cc file Suffix rules can be used to generalize such situations A default suffix rule turns.c/.cc files into.o files by running the command ${CC} ${CFLAGS} c $< 3

27 Simplest Makefile Example OBJS = main.o iodat.o dorun.o CC = /usr/bin/g++ program : ${OBJS} ${CC} -o $@ $? Makefile Tips Include a way to clean up your mess clean : rm f *.o core Include a target to build multiple programs all : program1 program2 program3 4

28 Software Development Process Development: Revision Control March 22nd, 2004 Class Meeting 9 Creation of source files RCS Compilation and linking g++ and make Running and testing programs gdb What is revision control? Why do you need revision control? A way to ensure that edits on files are: logged and archived consistent Especially applicable when projects include: multiple files multiple developers maintenance (corrective and enhancing) Software development is normally done in teams Multiple people may be responsible for code in a single file You may want to freeze a working version and create a new revision branch You may want to roll back development to a previous point in time Overview of RCS Key RCS Commands Maintains complete revision information for a set of files (not limited to source code) Uses major and minor revision numbers (e.g. 1.1, 2.3) Supports locking files so that only one user can edit at a time Supports merging two edits of the same file Can automatically place revision information within the files themselves rcs (administer project) ci (check in file) co (check out file) rlog (view the log) rcsdiff (see changes since last revision) 1

29 Basic RCS Usage Create a subdirectory RCS of the directory containing the files in question Check in the files to create an initial revision (1.1) Check out and lock files to edit them Check files back in to create a new revision and allow others to edit them Checking Out Files Assume we start with the file main.c checked in: RCS/main.c,v co main.c creates read-only file main.c in current directory co l main.c creates writable file main.c in current directory locks files so others cannot check it out Checking In Files Setting the Revision Number Assume we have locked and edited main.c ci main.c creates a new revision (e.g. 1.2) removes main.c from current directory ci u main.c creates a new revision leaves read-only main.c in current directory ci l main.c creates a new revision (checkpoint) allows you to keep editing Perhaps we have done major revisions to revision 1.4 of main.c and we want to start a new branch (2.x) of the revision tree ci r2 main.c co rr file retrieves revision number R of file instead of most recent version Using Keyword Substitution Keyword Example Each time you check in a new revision, RCS keeps information about the author, date/time, a log message, etc. You can show this information in files by placing special tags within the files when setting up RCS /* * Last Revision: * $Revision$ by * $Author$ on * $Date$ */ int main(){ }... /* * Last Revision: $Revision: 1.1 $ by $Author: callgood $ $Date: 2004/03/12 $ */ int main(){ }... 2

30 Software Development Process Development: Revision Control March 28th, 2004 Class Meeting 10 Creation of source files RCS Compilation and linking g++ and make Running and testing programs gdb Why use a debugger? Common Debugger Functions No one writes perfect code the first time, every time Desk checking code can be tedious and error-prone Putting print statements in the code requires re-compilation and a guess as to the source of the problem Debuggers are powerful and flexible Run program Stop program at breakpoints Execute program one line of code at a time Display values of variables Show sequence of function calls Catch signals The GNU Debugger (gdb) Free command-line debugger Common utilization: gdb executable gdb executable core gdb executable process_id Execution Commands list or l lists source code list list function_name list line_number run or r run program from beginning run run argument1 argument2 next or n execute next line, stepping over function calls step or s execute next line, stepping into function calls 1

31 Breakpoint Commands break or b set a breakpoint break function_name break line_number delete or d delete a breakpoint delete breakpoint_number continue or c continue execution when stopped Program Information Commands print or p prints value of a variable or expression print x print x*y print function(x) display continuously display value of a variable undisplay Stop displaying value of a variable Miscellaneous Commands set change a variable s value set n=3 help or h display help text help help command_name help keyword quit or q quit gdb 2

32 Layers in a Unix-based System System Programming: File Management March 22nd, 2004 Class Meeting 11 User Interface Library Interface System calls Users Standard Utility Programs (shells, editors, compilers, etc.) Standard Library (open, close, read, write, etc.) Unix Operating System (process/memory management, file system, I/O) Hardware (CPU, memory, disks, terminals, etc.) user mode kernel Unix System Programming C versus C++ Programming that uses special features of the Unix system (kernel) Programs make system calls via libraries Types of system calls File IO Process Management Inter-Process Communication (IPC) Signal Handling No string data types Use character arrays instead Use strcpy(), strncpy(), strcmp(), strncmp() to assign and compare character arrays No embedded declarations Must declare all variables at the beginning of a code block Very different File and Standard IO functions printf() versus cout scanf() and fgets() versus cin Basic File IO Basic File IO (cont) Remember everything in Unix is a file Processes keep a list of open files Files can be opened for reading, writing Must include <stdio.h> to use IO system calls Each file is referenced by a file descriptor (integer) Three files are opened automatically FD 0: standard input FD 1: standard output FD 2: standard error When new files are opened, it is assigned the lowest available FD 1

33 open() read() fd = open(path, flags, mode) path: char*, absolute or relative path flags: O_RDONLY open for reading O_WRONLY open for writing O_RDWR open for reading and writing O_CREAT create the file if it doesn t exist O_TRUNC truncate the file it exists (overwrite) O_APPEND only write at the end of the file mode: specify permissions if using O_CREATE Returns newly assigned file descriptor fd = open( myfile, O_CREAT, 00644) bytes = read(fd, buffer, count) Read from file associated with fd; place count bytes into buffer fd: file descriptor to read from buffer: pointer to an array count: number of bytes to read Returns number of bytes read or -1 if an error occurred int fd = open( somefile, O_RDONLY); char buffer[4]; int bytes = read(fd, buffer, 4*sizeof(char)); write() close() bytes = write(fd, buffer, count) Write contents of buffer to file associated with fd fd: file descriptor buffer: pointer to an array count: number of bytes to write Returns the number of bytes written or -1 if an error occurred return_val = close(fd) Closes an open file descriptor Returns 0 on success, -1 on error int fd = open( somefile, O_WRONLY); char buffer[4]; int bytes = write(fd, buffer, 4*sizeof(char)); File IO using FILEs fopen() Most Unix programs use higher-level IO functions fopen() printf() scanf() fclose() These use the FILE data type instead of file descriptors FILE *file_stream = fopen(path, mode) path: char*, absolute or relative path mode: r open file for reading r+ open file for reading and writing w overwrite file or create file for writing w+ open for reading and writing; overwrites file a open file for appending (writing at end of file) a+ open file for appending and reading fclose(file_stream) Closes open file stream 2

34 printf() printf(formatted_string,...) formatted_string: string that describes the output information variable types are escaped with % (see next slide) string is followed by as many expressions as are referenced in the formatted string printf() (cont) int term = 15; printf( Twice %d is %d \n, term, 2*term); formatted string expressions Escaping Variable Types %d, %i decimal integer %u unsigned decimal integer %o unsigned octal integer %x, %X unsigned hexadecimal integer %c - character %s string or character array %f float %e, %E double (scientific notation) %g, %G double or float %% - outputs a % character printf() Examples printf("the sum of %d, %d, and %d is %d\n", 65, 87, 33, ); Output: The sum of 65, 87, and 33 is 185 printf("error %s occurred at line %d \n", emsg, lno); emsg and lno are variables Output: Error invalid variable occurred at line 27 printf("hexadecimal form of %d is %x \n", 59, 59); Output: Hexadecimal form of 59 is 3B scanf() scanf(formatted_string,...) Similar syntax as printf, only the formatted string represents the data that you are reading in Must pass variables by reference Example scanf( %d %c %s, &int_var, &char_var, string_var); printf() and scanf() Families fprintf(file_stream, formatted_string,...) Prints to a file stream instead of stdout sprintf(char_array, formatted_string,...) Prints to a character array instead of stdout fscanf(file_stream, formatted_string,...) Reads from a file stream instead of stdin sscanf(char_array, formatted_string,...) Reads from a string instead of stdin 3

35 Processes in Unix System Programming: Process Management April 12 th, 2004 Class Meeting 12 Process basic unit of execution Executing instance of a program Has a process ID Occurs in a hierarchy of processes (parents and children) Has its own state/context/memory Process Management Process Creation System calls dealing with: Process creation Setting the program a process executes Waiting for a process to terminate Terminating a process Sending signals to a process pid = fork(); Creates a new child process that is an exact copy of the current process Same program running at the same location Same variable values Same open files Only difference: child has a new process ID fork (cont) Setting the Program of Execution Process 1... pid = fork() if(pid == 0){ // child } else{ // parent } DATA Process 2... pid = fork() if(pid == 0){ // child } else{ // parent } DATA exec family of functions execvp(executable_name, args) Others: execl, execlp, execle, execv, execve Replaces the current process with a new process image running the executable specified with the arguments listed New process retains the old process ID and any open files 1

36 Waiting for a Process to Terminate Using fork, exec, wait together pid = wait(&status); Suspends the execution of the calling process until any child process terminates Returns the process ID of the terminating child Puts exit status (int) of child in status pid = waitpid(pid, &status, options) Suspends execution of the calling process until a specific child terminates pid = fork(); if(pid == 0){ execl(./program, program, arg1, NULL); } else{ pid = wait(&status); } Process Termination Sending Signals to a Process exit(status); Terminates the process Closes all open file descriptors Returns status to the parent process retval = kill(pid, signal); Some common signals a user program might send: SIGINT: interrupt (Ctrl-C) SIGKILL: kill SIGUSR1, SIGUSR2: user-defined Inter-Process Communication (IPC) IPC with pipes Information passing between processes Two basic paradigms Message passing: processes send information back and forth in message/packets Shared memory: processes share a chunk of physical memory and read/write data there to share that information Example of message passing int fds[2]; retval = pipe(fds); Creates two file descriptors (a pipe is a file), the first for reading and the second for writing How does another process connect to this pipe? 2

37 Basic Pipe Example int fds[2]; char s[100]; int pid; retval = pipe(fds); pid = fork(); if(pid == 0){ read(fds[0], s, 100); printf( Read: %s\n, s); } else{ write(fds[1], hello, 6); } Notes: Data is written/read in order (FIFO) Reads block until there is data to read Writes block if the pipe is full Closing the writing fd causes EOF to be read on the other end 3

38 sed Intro to sed and awk April 19 th, 2004 Class Meeting 13 Stream editor Originally derived from the ed line editor Used primarily for non-interactive operations Operates on data streams Usage sed options `address action` file(s) Example: sed `1,$s/^bold/BOLD/g` foo Line Addressing Using line numbers Range separated by commas n,m Example: sed 1,3 foo.txt Specify first line and step f~s Example: sed 2~3 foo.txt sed 3,6p foo.txt Print lines 3-6 from foo.txt Is this the output you get? sed automatically prints input Use n option to suppress printing every line sed n 3,6 foo.txt Line Addressing (cont) sed n $p foo.txt $ represents the last line Prints only the last line of the file Reversing line criteria sed n 3,$!p foo.txt Prints every line except 3-$ (last line) Context Addressing Use patterns/regular expressions rather than explicitly specifying line numbers sed n /^From: /p $HOME/mbox Retrieve all the sender lines from the mailbox file Note that / / mark the beginning and end of the pattern to match ls l sed n /^...w/p Prints the line if the fourth character is a w Substitution Strongest feature of sed Syntax [address]s/expression1/subs/flags Example sed s/ /:/ data.txt Substitute the character with : First occurrence in a particular line sed s/ /:/g data.txt All occurrences in the line 1

39 Using Files Tedious to type in commands at the prompt, especially if commands are repetitive Can put commands in a file and send can use them Do not need to enclose commands in quotes sed f cmds.sed data.txt awk Powerful pattern scanning and processing language Named after creators Aho, Weinberger, and Kernighan Most commands operate on entire line; awk operates on fields within each line Usage: awk options [scriptfile] files(s) Example: awk f script.awk foo.txt Processing Model BEGIN{ # Commands executed before input } { # Main execution loop for each line of input } END{ # Commands executed after input } Other Features Formatted printing using C-style printf Conditional statements (if-else) Looping constructs for while do-while Associative Arrays Normal arrays use integers as their indices Associative arrays use strings as their indices Example: age[ John ] = 23 Special Variables FS Field separator Default value is space RS Record separator Default value is newline ( \n ) FNR Record number in the current input file OFS Output field separator Default value is space 2

40 Special Variables (cont) ORS Output record separator Default value is newline ( \n ) NF Number of fields in the current input record NR Number of records seen so far 3

41 Basic UNIX system administration CS 2204 Class meeting 14 *Notes by Doug Bowman and other members of the CS faculty at Virginia Tech. Copyright System administration Thus far, we ve only discussed: the use of UNIX from an end user point of view System programming - accesses the core OS but doesn t change the way it operates System administration is another level: changing the way the system is set up and operates for end users Strongly related to security issues (C) Doug Bowman, Virginia Tech, The superuser Most sys. admin. tasks can only be done by the superuser (also called the root user) Superuser has access to all files/directories on the system can override permissions owner of most system files Shell command: su <username> Set current user to superuser or another user with proper password access Administration through files As you would expect, system settings are stored in files Most of these are stored in /etc We ll look at files related to: Users and groups File systems System initialization System upkeep In section 5 of the man pages (C) Doug Bowman, Virginia Tech, (C) Doug Bowman, Virginia Tech, /etc/passwd Information about system users bowman:x:65:20:d.bowman:/home/bowman:/bin/ksh login name user ID real name group ID [encrypted password] command interpreter home directory (C) Doug Bowman, Virginia Tech, /etc/group Information about system groups faculty:x:23:bowman,ribbens,mcquain list of group members group ID [encrypted group password] group name (C) Doug Bowman, Virginia Tech,

42 /etc/fstab Information about file systems /dev/cdrom /cdrom iso9660 defaults,ro,user,noauto 0 0 file system type mount options mount point file system (local device or remote dir.) other flags /etc/inittab and /etc/init.d/ inittab: configuration file for the init process Defines run levels init.d: Directory containing system initialization scripts Script rc <n> is run at run level n Starts and stops various services (C) Doug Bowman, Virginia Tech, (C) Doug Bowman, Virginia Tech, Configuring the system Often by directly editing files with a text editor In some cases there are programs that modify the files for you Many systems also have nice graphical user interfaces that let you manipulate these files indirectly File System Implementation A possible file system layout (C) Doug Bowman, Virginia Tech, (C) Doug Bowman, Virginia Tech, Internal View of a File System Boot Block: The first block in a UNIX file system, contains the boot program and other initialization information or unused. Super Block Always the second block, contains the complete catalog of specific information about the file system, including lists of free memory Internal View of a File System Inode list blocks List of inodes for the file system, contiguous and always follows the super block. The number of inodes is specified by the system administrator File access and type information, collectively known as the mode. File ownership information. Time stamps for last modification, last access and last mode modification. Link count. File size in bytes. Addresses of physical blocks. (C) Doug Bowman, Virginia Tech, (C) Doug Bowman, Virginia Tech,

43 Internal View of a File System Data blocks OS files, user data and program files etc. File system Commands mount mount a file system umount unmount a file system fsck check and repair a Linux file system sync flush filesystem buffers (C) Doug Bowman, Virginia Tech, (C) Doug Bowman, Virginia Tech, crontab Useful to have a script or command executed without human intervention a script to verify that the networks are working correctly cron daemon reads cron configuration files called crontab short for cron table parse crontabs find the soonest command to be run go to sleep until the command s execution time has arrived What s cron and crontabs? Under UNIX, periodic execution is handled by the cron daemon read one or more configuration files containing as following command lines times at which they are to be invoked (on some systems) login names under which they are to run crontabs /etc/crontab (C) Doug Bowman, Virginia Tech, (C) Doug Bowman, Virginia Tech, Format of Crontab files Seven or six fields minute hour day month weekday [username] command an asterisk matches all possible values, a single integer matches that exact value, a list of integers separated by commas (no spaces) used to match any one of the values two integers separated by a dash (a range) used to match any value within the range (C) Doug Bowman, Virginia Tech, Example crontab SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root HOME=/ 01 * * * * root nice - n19 run- parts /etc/cron.hourly 02 4 * * * root nice - n19 run- parts /etc/cron.daily 22 4 * * 0 root nice - n19 run- parts /etc/cron.weekly * * root nice - n19 run- parts /etc/cron.monthly (C) Doug Bowman, Virginia Tech,