ACSC 372 Systems Programming

Transcription

1 Regular Expressions ACSC 372 Systems Programming A Regular Expression is a pattern that describes a set of strings. Regular expressions are constructed using various operators to combine smaller expressions. Lecture 5: Advanced Unix Commands and System Utilities II 2.? * + A regular expression may be followed by one of several repetition operators (metacharacters): {N} ^ $ \b \B \< \> Regular Expressions (cntd) Operator {N,} {N,M} Effect Matches any single character The preceding item will be matched zero times or once The preceding item will be matched zero or more times The preceding item will be matched one or more times The preceding item is matched exactly N times The preceding item is matched N or more times The preceding item is matched at least N times, but not more than M times Represents the range if it s not first or last in a list or the ending point of a range in a list Matches the empty string at the beginning of a line; also represents the characters not in the range of a list Matches the empty string at the end of a line Matches the empty string at the edge of a word Matches the empty string provided it s not at the edge of a word Matches the empty string at the beginning of word Matches the empty string at the end of word Regular Expressions (cntd) Character Class: A bracket expression is a list of characters enclosed by square brackets "[" and "]". It matches any single character in that list. If the first character of the list is the caret, "^", then it matches any character NOT in the list. For example, the regular expression [ ] matches any single digit. Within a bracket expression, a range expression consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, inclusive. For example, [ ] is equivalent to [0-9]. Another example, [a d] is equivalent to [abcd]. This can also be expressed as: a b c d Further example, [a-cx-z] is equivalent to [abcxyz]. This can also be expressed as: a b c x y z 3 4

2 Regular Expressions (cntd) Regular Expressions (cntd) Two regular expressions may be concatenated; the resulting regular expression matches any string formed by concatenating two substrings that respectively match the concatenated subexpressions. Two regular expressions may be joined by the infix operator (alternation); the resulting regular expression matches any string matching either subexpression. The Repetition operators take precedence over concatenation, which in turn takes precedence over alternation. A whole subexpression may be enclosed in parentheses to override these precedence rules. Basic versus Extended Regular Expressions In basic regular expressions, the metacharacters:? + { ( ) lose their special meaning; instead use the backslash versions: \? \+ \{ \ \( \) 5 6 Regular Expressions (cntd) Single Quotes used in order to maintain the characters that are enclosed in single quotes as are (literally) Double Quotes used in order to maintain the characters that are enclosed in double quotes as are (literally), apart from the $ (dollar sign), the `` (backward single quotes) and the \ (backslash). Example echo '$HOME' $HOME echo "$HOME" /home/com.chc Commandgrep Command grep (options -i, -n, -v, -w, -l ) searches the input files for lines containing a match to a given pattern list. When it finds a match in a line, it copies the line to standard output (by default), or whatever other sort of output you have requested with options. 7 8

3 Commandgrep (cntd) grep <options> <pattern> <filename> Option -i Case-insensitive ignore case distinctions between upper and lower case letters Option -n prefix each line of output with the line number within its input file. Option -v Invert the sense of matching, to select non-matching lines. Option -w Select only those lines containing matches that form whole words. Option -l print the name of each input file from which output would normally have been printed (have line(s) that match the pattern) 9 Commandgrep (cntd) grep command handles two different versions of constructing regular expressions: the basic and extended regular expressions. Basic regular expressions can be used by the command grep. For the metacharacters:? + { ( ) we can use the backslash OTHERWISE Extended regular expressions can be used by the command egrep no need to use the backslash for the metacharacters need to use the backslash for the metacharacters when used as are (literally, and not with their special meaning) 10 Commandgrep (cntd) Commandgrep (cntd) Examples Examples (cntd) grep n root /etc/passwd 1: root:x:0:0:root:/root:/bin/bash 12: operator:x:11:0:operator:/root:/sbin/nologin grep -v bash /etc/passwd grep -v nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt news:x:9:13:news:/etc/news: nx:x:16:16::/var/lib/nxserver/home:/usr/bin/nxserver grep ^root /etc/passwd root:x:0:0:root:/root:/bin/bash grep :$ /etc/passwd news:x:9:13:news:/etc/news: grep export ~/.bash_profile grep '\<PATH' export PATH=.:$NS/bin:$NS/tcl8.4.18/unix:$NS/tk8.4.18/unix:$PATH export PATH 11 12

4 Commandgrep (cntd) Examples (cntd) grep -w / /etc/fstab LABEL=/ / ext3 defaults 1 1 grep [yf] /etc/group sys:x:3:root,bin,adm tty:x:5: mail:x:12:mail,postfix ftp:x:50: nobody:x:99: floppy:x:19: mysql:x:27: postfix:x:89: nfsnobody:x:65534: xfs:x:43: Commandgrep (cntd) Examples (cntd) ls -l grep -i '\<i' -rwx com.chc staff Jan iasted1.pdf -rwx com.chc staff Jan iasted2.pdf -rwx com.chc staff Dec ICIS_chrysostomou.pdf -rwx com.chc staff Dec ICIS_pitsillides.pdf drwxr-x--- 2 com.chc staff 4096 Aug idcc drwxr-x--- 5 com.chc staff 4096 Jul IPE ls -l grep '^d' grep -i '\<i' drwxr-x--- 2 com.chc staff 4096 Aug idcc drwxr-x--- 5 com.chc staff 4096 Jul IPE Commandgrep (cntd) Examples (cntd) Commandgrep (cntd) grep '^c.*h$' /usr/share/dict/words more cabbalah cable-stitch cablish caddish cadish caducibranch cafeneh cafh cailleach cailliach calabash calash calenturish calfish calipash caliph calligraph callipash calvish calycanth camelish cameograph camooch --More-- grep '^c..h$' /usr/share/dict/words more cafh caph cash cath chih coch cosh coth croh csch cush Examples (cntd) ls -l grep '^-' grep '+\ W\+' -rw-r--r-- 1 com.chc staff 0 Jan 31 16:46 file+1.txt -rw-r com.chc staff 371 Feb WS_FTP.LOG ls -l grep '^-' egrep '\+ W+' -rw-r--r-- 1 com.chc staff 0 Jan 31 16:46 file+1.txt -rw-r com.chc staff 371 Feb WS_FTP.LOG ls -l grep '^-' egrep '\<f' -rw-r--r-- 1 com.chc staff 0 Jan 31 16:46 file+1.txt 15 16

5 Process Control Process: How the Unix Operating System refers to a running program Job: How the shell views running programs started from that shell One job is one complete Unix sentence or command line task Can have multiple processes run at the same time. e.g. the pipe ls l sort more invokes three processes: ls, sort and more This specific command line constitutes a job Process ID - PID The Unix operating system kernel assigns a unique number to each running program. Control Number Job ID The shell assigns a unique number to each command line run from that shell. Can stop/suspend (not terminate) a job (command line task) that runs in the foreground, by pressing Ctrl-z. When a job has been stopped, we can resume its execution: Command fg (foreground) Resume a program that has been stopped Refer to the program that has control of the shell One command line is executed at a time 19 20

6 Command bg (background) Tasks that do not need the shell (e.g., no output is displayed in the screen, no input is needed from the standard input stream) can be run in the background, where they run until they terminate. If a program that runs in the background, need to write to the screen, or read from the keyboard, the system will stop the execution automatically. We can use the command fg to bring the program in the foreground to resume the stopped job. A different approach is to run a program in the background, and let the UNIX system handle it. If the program needs some input or output, then it is stopped, Like the jobs we have put in the background with the command bg, after they have already started to run. A program (or pipe) automatically runs in the background, by placing a & at the end of the command line Ctrl-z Job ID The most recent task more test-large.c /* ** GP142.c by Neil McKenzie ** ** RCS $Id: gp142.c,v /07/17 22:33:50 boren Exp $ ** ** Copyright (C) 1994 Neil McKenzie. All rights reserved. ** ** Copyright (c) 1994 Corey Anderson. ** ** Copyright (c) 1995 Casey Anderson. ** Revised version 1.5 of October, 1994 ** Revised version 1.61 of December, 1994 ** Revised version 2.00 of January, 1995 ** conversion from callback fns to await_event ** Revised version 2.01 of July, 1995 ** fixed hotkeys in windows ** Revised version 2.1 of October, 1995 ** transparent text in windows; float pragma; casts; ** animation based on wall time elapsed ** "Fixed" PC window size problem, see FUDGE, below -- brd ** Includes changes made for port to WIN32 by Shuichi Koga ** All additions ifdefed by WIN32 -- brd ** --More--(0%) [1]+ Stopped more test-large.c --More--(0%) [1]+ Stopped more test-large.c fg resume the stopped job more test-large.c ** Revised version 2.2 on July, 1997 ** X11R6 Melissa Johnson mjhnsn@u.washington.edu ** Assembly timer code for Mac replaced with C-based code. (Corey) ** Removed stack of objects. Replaced with offscreen buffer. ** Dramatically improves animation quality. (Josh, Melissa, Corey) ** Set GP142_DATE to indicate when GP142 was last _updated_, not ** merely last compiled. (Corey) ** ** Revised version of February,

7 job ID Process ID - PID pressing RETURN, a completed background job will notify us that it s done. who grep com.chc & [1] 2616 com.chc pts/ :11 (csmachine.cs.school.edu) [1]+ Done who grep com.chc ls -l ~ grep test sort uniq more & [1] 2723 [1]+ Stopped ls -l ~ grep test sort uniq more fg ls -l ~ grep test sort uniq more -rw-r--r-- 1 com.chc staff 0 Feb 1 00:22 test-cut.txt~ -rw-r--r-- 1 com.chc staff0 Feb 1 00:30 test-tr~ -rw-r--r-- 1 com.chc staff0 Jan 24 09:08 test1.txt grep '^c.*h$' /usr/share/dict/words sort -r > output.txt Ctrl-z [1]+ Stopped grep '^c.*h$' /usr/share/dict/words sort -r >output.txt bg resume the stopped job in the background [1]+ grep '^c.*h$' /usr/share/dict/words sort -r >output.txt & [1]+ Done grep '^c.*h$' /usr/share/dict/words sort -r >output.txt pressing RETURN, a completed background job will notify us that it s done. Running multiple processes Sequentially It is possible to specify the UNIX system to execute a series of commands, by inserting a command line using the operator «;» e.g. ls ; more *.txt ; cd Parallel e.g. ls & cat file.txt & grep pattern file.txt 27 28

8 Seeing what jobs are running Command jobs With no arguments, it simply prints out a list of all jobs that are running in the background of that shell Gives the control number (job id), and the command line running grep '^c.*h$' /usr/share/dict/words sort -r > output.txt Ctrl-z [1]+ Stopped grep '^c.*h$' /usr/share/dict/words sort -r >output.txt ls -l ~ grep test sort uniq more & [2] 3035 [2]+ Stopped ls -l ~ grep test sort uniq more jobs [1]- Stopped grep '^c.*h$' /usr/share/dict/words sort -r >output.txt [2]+ Stopped ls -l ~ grep test sort uniq more Switching jobs to the foreground Command fg %<job_id> The number specified here is the control number, NOT the Process ID The number listed by the jobs command fg without any parameters takes the most recent task (listed with a + sign in the jobs command) Switching jobs to the background Command bg %<job_id> The number specified here is the control number, NOT the Process ID bg without any arguments moves the most recent task into the background` After a job is switched to the background, it can continue on its way or fg %1 grep '^c.*h$' /usr/share/dict/words sort -r >output.txt bg %1 [1]- grep '^c.*h$' /usr/share/dict/words sort -r >output.txt & [1]- Done grep '^c.*h$' /usr/share/dict/words sort -r >output.txt jobs [2]+ Stopped ls -l ~ grep test sort uniq more 31 32

9 Learn which processes are running Command ps (process status) Displays currently running processes among other information, it gives the PID of each running process Flag a e f u x Useful options of ps command See the manual (man) Meaning Shows all processes associated with terminals attached to the system Shows all processes on the system Gives full output format Produces user-oriented output Shows all processes in the system see your processes on full-format output ps fu USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND com.chc pts/1 Ss 12:11 0:00 -ksh com.chc pts/1 S 12:11 0:00 \_ bash com.chc pts/1 R+ 18:53 0:00 \_ ps fu see everything on the machine ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root ? Ss Jan30 0:01 init [5] root ? S Jan30 0:00 [migration/0] ug ? Ssl Feb02 0:00 dbus-daemon --f ug ? Ssl Feb02 0:00 dbus-daemon --f ug ? Ssl Feb02 0:00 dbus-daemon --f com.chc pts/1 Ss 12:11 0:00 -ksh com.chc pts/1 S 12:11 0:00 bash com.chc ? Ssl 12:22 0:00 dbus-daemon --f com.chc pts/1 R+ 19:07 0:00 ps aux Field USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Meaning Name of the user Process ID Percent of CPU used Percent of RAM used Virtual memory used Real memory used Associated terminal (terminal to which the process is connected) Status Exact time when the process was started CPU time used command 35 36

10 Status value R S I T Z Meaning Running Sleeping (20 seconds or less) Idle (sleeping more than 20 seconds) Stopped Zompie process (the process has ended but hasn t freed up its resources) Learn which processes are running (cntd) Command top Only shows the processes that are taking up the most CPU time The processes that are causing your computer to run slowly. top top - 19:38:10 up 5 days, 5:10, 2 users, load average: 0.04, 0.01, 0.00 Tasks: 115 total, 1 running, 113 sleeping, 1 stopped, 0 zombie Cpu(s): 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si, Mem: k total, k used, 99520k free, k buffers Swap: k total, 0k used, k free, k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root S :01.31 init 2 root RT S :00.00 migration/0 3 root S :00.00 ksoftirqd/0 4 root RT S :00.00 watchdog/ Terminating Jobs Command kill % <job_id> ls -l ~ grep test sort uniq more & [1] 3650 jobs [1]+ Stopped ls -l ~ grep test sort uniq more ps u USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND com.chc pts/1 Ss 12:11 0:00 -ksh com.chc pts/1 S 12:11 0:00 bash com.chc pts/1 T 19:47 0:00 ls -l /home/com.chc com.chc pts/1 T 19:47 0:00 grep test com.chc pts/1 T 19:47 0:00 sort com.chc pts/1 T 19:47 0:00 uniq com.chc pts/1 T 19:47 0:00 more com.chc pts/1 R+ 19:47 0:00 ps u kill %1 ps u USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND com.chc pts/1 Ss 12:11 0:00 -ksh com.chc pts/1 S 12:11 0:00 bash com.chc pts/1 R+ 19:47 0:00 ps u [1]+ Terminated ls -l ~ grep test sort uniq more jobs Terminating Processes Command kill <options-number> <PID> Will send a signal to a process running By default, sends the termination signal (the same as Ctrl-c) 39 40

11 Number (name) 1 SIGHUP 2 SIGINT 9 SIGKILL 15 SIGTERM Meaning Hang up (sent to every process you are running just before you log out the system) Interrupt (is the signal sent when you press Ctrl-C).. Kill (cannot be caught or ignored) (the process is terminated immediately, without even a chance to clean up after itself) Software termination signal (requests an immediate termination of the program, but it allows the program to remove temporary files it might have created) ls -l ~ grep test sort uniq more & [1] 3771 [1]+ Stopped ls -l ~ grep test sort uniq more ps u USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND com.chc pts/1 Ss 12:11 0:00 -ksh com.chc pts/1 S 12:11 0:00 bash com.chc pts/1 T 20:14 0:00 ls -l /home/com.chc com.chc pts/1 T 20:14 0:00 grep test com.chc pts/1 T 20:14 0:00 sort com.chc pts/1 T 20:14 0:00 uniq com.chc pts/1 T 20:14 0:00 more com.chc pts/1 R+ 20:15 0:00 ps u ki kill ps u USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND com.chc pts/1 Ss 12:11 0:00 -ksh com.chc pts/1 S 12:11 0:00 bash com.chc pts/1 T 20:14 0:00 grep test com.chc pts/1 T 20:14 0:00 sort com.chc pts/1 T 20:14 0:00 uniq com.chc pts/1 T 20:14 0:00 more com.chc pts/1 R+ 20:15 0:00 ps u [1]+ Stopped ls -l ~ grep test sort uniq more Finding Files and Programs Finding Files and Programs (cntd) Command which <ProgramName> Tells you what executable is being run when you run a program Only searches your PATH which more /bin/more which ls /bin/ls which gran which: no gran in (/usr/local/bin:/bin/:/usr/bin:/etc:/usr/sbin:/u sr/ucb:/home/com.chc/bin:/usr/bin/x11:/sbin:.) Command whereis <ProgramName> Locates the source, executable, and man page for a command whereis ls ls: /bin/ls /usr/share/man/man1/ls.1.gz /usr/share/man/man1p/ls.1p.gz 43 44

12 Finding Files and Programs (cntd) Command find <path> <options> <expression> Locate files based on any number of factors (see man find) name Modification time Access permissions File size Finding Files and Programs (cntd) Options -name Matches the name of a file -iname Case-insensitive -type d directory f regular file L symbolic link -atime N File was last accessed N days ago -mtime N File was last changed N days ago Ν+ More than N days ago Ν- Less than N days ago Finding Files and Programs (cntd) Example: find ~ -name "test*.txt" /home/com.chc/test/test1/test1.txt /home/com.chc/test/test2/test2.txt /home/com.chc/test/test.txt Action Commands: -print Simply prints out the name of the file that was found -exec <COMMAND> \; Executes the command (COMMAND) for each file found Finding Files and Programs (cntd) Example: find ~ -name "test*.txt" -print /home/com.chc/test/test1/test1.txt /home/com.chc/test/test2/test2.txt /home/com.chc/test/test.txt find ~ -name "test*.txt" -exec echo {} \; /home/com.chc/test/test1/test1.txt /home/com.chc/test/test2/test2.txt /home/com.chc/test/test.txt {} gets replaced with the name of the file that was found. The exec option allows you to perform some command on all files that match a certain criteria

13 Finding Files and Programs (cntd) Problem: The exec command will run the command once for every file that matches A lot of overhead May take a lot longer than you d like Solution: Command xargs Finding Files and Programs (cntd) Command xargs <COMMAND> Take values from standard input and convert them into command line parameters Used when a program outputs a list that will be used as input to another program Used mainly in pipes with find Example: find. name *.txt xargs grep Markus find. name *.txt -exec grep Markus {} \; xargs calls grep many, many times less than the second call (-exec) and takes less time Further UNIX Commands Command alias <NAME>= COMMAND Making another name for some other command Useful for making sure certain flags are always used alias ll='ls -l' ll test/ total 4 -rw-r--r-- 2 com.chc staff 28 Jan 25 14:35 HardLinkToFile1.txt lrwxrwxrwx 1 com.chc staff 20 Jan 24 10:50 SymbLinkToFile2.txt -> test/test2/test2.txt drwxr-xr-x 4 com.chc staff 50 Jan 28 17:25 test1 drwxr-xr-x 2 com.chc staff 22 Jan 24 10:42 test2 -rwxrwxr-x 1 com.chc staff 0 Jan 30 19:40 test.txt alias alias ll='ls -l unalias ll alias Further UNIX Commands (cntd) Command cut (options -d, -f) More precise control over individual lines Will cut out certain words from each individual line so they can be processed Displays on the output selected fields from the file s lines Option d <delimiter> use a new field delimiter of lines, instead of TAB Option f <field_number(s)> Output certain fields 51 52

14 Further UNIX Commands (cntd) more test-cut.txt Line number 1 Line number 2 Line number 3 Line number 4 cut -d' ' -f3 test-cut.txt cut -d' ' --field=1,3 test-cut.txt Line 1 Line 2 Line 3 Line 4 -f1,3 new specified field delimiter: single empty space equivalent Further UNIX Commands (cntd) Command tr (options -d, -s) Translate, squeeze, and/or delete characters from standard input, writing to standard output. Usage: tr <SET1> <SET2> Option -d Do not translate; delete characters, listed in SET1, from standard input. Option - s replace each input sequence of a repeated character that is listed in SET1 with a single occurrence of that character Further UNIX Commands (cntd) Further UNIX Commands (cntd) more test-tr.txt Thiss is the start of the novel i wrote. tr 'i' 'I' < test-tr.txt ThIss Is the start of the novel I wrote. tr -d 'is' < test-tr.txt Th the tart of the novel wrote tr -s 's' < test-tr.txt This is the start of the novel i wrote. Command tee read from standard input and write to standard output and files, simultaneously Place the command tee anywhere in a pipe to copy the standard input of command tee to a file and also to standard output, or to the next step of the pipe. sort somefile.txt tee sorted_file.txt uniq -c 55 56

15 Further UNIX Commands (cntd) Command mail (options -s, -c) send and receive Option -s Subject Option -cc cc-address mail s ACSC372 cc com.chc@frederick.ac.cy acsc372@frederick.ac.cy < acsc372_syllabus.pdf Further UNIX Commands (cntd) Command comm compare two sorted files line by line With no options, produce three-column output: Column one contains lines unique to FILE1 Column two contains lines unique to FILE2, and Column three contains lines common to both files. Option -1 suppress lines unique to FILE1 (e.g., display the lines of FILE2 that are not contained in FILE1, and the lines common to both files) Option -2 suppress lines unique to FILE2 Option -3 suppress lines that appear in both files File name: «-» means to read from standard input Further UNIX Commands (cntd) Command diff find differences between two files Option -i Case-insensitive Further UNIX Commands (cntd) Command crontab Used for executing scheduled programs/commands, periodically. The commands are saved in a file known as «crontab»; this file is read and the commands inside the file are run in the background from the cron daemon The cron daemon is always running in the background; it starts up every time a UNIX machine boots up; once a minute, it checks certain files to see if anything needs to be run. These commands are called cron jobs. To schedule the jobs you want to be run, you need to send them to the cron daemon using the command crontab. The easiest way is to produce a file with the commands/jobs (e.g., example.crontab) and the jobs are activated by calling the command: crontab example.crontab

16 Further UNIX Commands (cntd) Further UNIX Commands (cntd) or crontab -e Edit your crontab file, or create one if it doesn't already exist. crontab -l Display your crontab file Crontab syntax : A crontab file has five fields to define the day, date, and time, followed by the command that is to be run at the specified interval * * * * * command to be executed day of week (0-6) (Sunday=0) month (1-12) day of month (1-31) hour (0-23) min (0-59) Example Crontab A line in the crontab file: remove the temporary files from the /home/someuser/tmp every day at 6.30 p.m., and at 6.32 p.m. write to a file, with the name myshell_pid.out, the list of contents of the someuser s home directory * * * rm /home/someuser/tmp/* * * * ls al ~ > /home/someuser/tmp$$.out 61 62