Motivation (Scenarios) Topic 4: Grep, Find & Sed. Displaying File Names. grep

Transcription

1 Topic 4: Grep, Find & Sed grep: a tool for searching for strings within files find: a tool for examining a directory tree sed: a tool for "batch editing" Associated topic: regular expressions Motivation (Scenarios) I have a file called (something)220(something).java which contains the phrase "final exam" in a comment. Where is it? I need to find all the files in a directory tree with names containing "2013" and change to "2014". I need to find all the files and directories in a directory tree whose names contain "220" and change to "220A" I want to delete all files in a directory tree whose names end in "~" or start and end with "#" Style changes in large multi-file C program: change header comment and names of some functions. 1 2 grep Displaying File Names Use to search inside files for strings Default behavior: shows file names only if searching >1 file. Name comes from character sequence in old editor: g/re/p: globally search for a regular expression and print grep <pattern> <files> Searches files and prints all lines containing the pattern grep Frost * Flags: -h: never show file names -H: always show file names -l: show only the file names (not the matches) (lower-case L) Flags: -i: case-insensitive match -w: string must appear as an entire word, not part of a word -num: num lines of context around each match 3 4

2 No file arguments: grep reads standard input (useful for pipes!) history grep cd Grep as a filter Using grep in a conditional exit status: 0 = at least one match was found 1 = no matches found 2 = error useful flag: -q: no output (quiet) if grep -q $word $filename then Regular Expressions Pattern to grep in previous examples: literal value For more complex searches: use a regular expression In this course: using extended regular expression syntax much easier than the default syntax! use -E flag (or egrep command) put patterns inside single quotes Regular expressions not to be confused with wildcards in file names! Match A Single Character. in pattern: matches any single character grep -E 'n.t' * Matches: not, nat, nit, etc. Does not match: nt or naut 7 8

3 Optional Items Grouping Items?: Preceeding item is optional grep -Ew 'loved?' * Matches love or loved Can group items into one using parenthesis grep -Ew 'love(ly)?' * Matches love or lovely 9 10 Repeated Items Alternatives *: Match preceding item zero or more times +: Match preceding item one or more times {n}: Match preceding item exactly n times Examples: grep -E 'a(bc)*d' * matches ad, abcd, abcbcd, etc. grep -E 'a(bc)+d' * matches abcd, abcbcd, etc., but not ad grep -E 'a(bc){2}d' * matches abcbcd and nothing else Two regular expressions joined with matches either expression. grep -E 'red black' * Matches red or black Precedence: highest: repetition (* or +) next: concatenation (two items in a row) lowest: alternation ( ) When in doubt, use parenthesis 11 12

4 Bracket Expressions Sequence of characters in brackets: matches any one of those characters. grep -E 'n[aeiou]t' * Maches nat, net, nit, not, nut Use dash to specify a character range: [b-d] matches b, c or d [b-dr-t] matches b,c,d,r,s or t ^ at the beginning of a bracket expression: matches any character except the characters in the bracket [^b-dr-t] matches any character that is not b,c,d,r,s or t Character Classes Special notation for matching common groups of characters [:digit:] [:alpha:] - any letter (upper or lower case) & more on summary sheet These may used only inside bracket expressions Examples: egrep -w '[[:alpha:]]{4}' egrep 'linux[[:digit:]]{2}\.[[:digit:]]+' egrep -w '[[:alpha:]123]+' Matching Word & Line Positions Quoting Special Characters \<: matches beginning of word (letters, digits, underscores) \>: matches end of word ^: matches start of line $: matches end of line These are not characters! They match positions only. \ before any character that normally has a special meaning means match the character literally. matches any single character \. matches a period

5 Reminder For Grep Grep matches any line with with a substring that matches the pattern. grep 'cat' Matches any line containing cat. Not necessary to write egrep '^.*cat.*$' Backreferences Preliminary note: What does the following match? (cat dog){3} Each match of (cat dog) can be different. If that's not what you want, use backreferences. \1: the substring matched by the first parenthesized sub-expression \2, \3, etc... (Jar Nicely)-\1 matches Jar-Jar or Nicely-Nicely does not match Jar-Nicely or Nicely-Jar find (1) Searches directory trees for files having particular properties. General form: find [dir...] [test...] [action...] Default action: print name of file Simplest way to use: find files based on their names. find the file called podcast220.xml find all CSS files (names ending with.css) Searches directory and its sub-directories no limit on depth case-insensitive name match: find web220 iname '*.html' Note: find includes hidden files and directories (names starting with ".") find (2): finding files by type find web220 type f shows names of all regular files in web220 find web220 type d shows names of all directories in web220 find web220 type l shows names of all symbolic links in web220 Can combine tests: all must be true find web220 type d name 's*' shows names of all directories whose names start with s 19 20

6 find (3): finding files by size find web220 size 4157c prints name of all files whose size is exactly 4157 bytes find web220 size +4157c prints name of all files in web220 whose size is > 4157 bytes find web220 size -4157c prints name of all files in web220 whose size is < 4157 bytes find (4): finding files by age find web220 mtime 4 prints names of all files modified 4 days ago (number of days rounded down) find web220 mtime +4 prints names of all files modified more than 4 days ago -10 for less than 10 days ago size modifiers: c = bytes k = kilobytes M = megabytes G = gigabytes Sizes are rounded up! find (5): finding files by age, continued -mtime asks about time in days -mmin asks about time in minutes find web220 mmin -240 shows names of files modified less than 4 hours ago -atime and amin ask about access times, not modification times find web220 amin -240 shows names of files looked at less than 4 hours ago find (6): actions Default action: -print (print the name of the file) Another possible action: -delete Handy for cleaning up. (Careful: no confirmation!) emacs creates backup files ending in ~ find mydir name '*~' delete Deletes all emacs backup files Suppose you'd like to see the names of the files that are deleted. Use print action first. find mydir name '*~' print delete 23 24

7 find (7): Combining with other Programs Goal: use find to identify files and then run other programs on those files. FILES=$(find...) for FILE in $FILES do... commands using $FILE done Caution:file names containing spaces are a problem! Examples clean-up script Use find to write-protect a directory tree Show just the base names of all files in directory Find all files in web site containing the word "quiz" or: for FILE in $(find...) do... commands using $FILE done sed Simple Ways To Run Sed sed = a "stream editor" editing in batch mode automates repetitive edits Scenario: CISC 123 last offered in fall 2006 Offering CISC 123 again in winter 2015, need new web site To start web site: copy all files, make some global changes (fall to winter, 2006 to 2015, instructor's name, etc) Options: 1. Open each file by hand and make the edits. 2. Write a Java/C/Python program to make the edits 3. Write a script using find & sed to run program on all files Which option sounds like less work? Which option is more error-prone? To run a single sed command on one or more files: sed -r -e 'command' filename1 filename2... Output goes to standard output (can redirect or pipe) -r means use extended regular expression syntax (like -E for grep) -e means next argument will be a sed command Can combine, but e must come last: sed -re 's/mary/martha/' myfile > newfile No file name: apply command to standard input (useful for pipes or experiments) 27 28

8 Simple Substitute Command in sed Global Substitutions s/pattern/replacement/ pattern = regular expression replacement: string to substitute for the match s/mary/martha/ First occurrence of Mary in each line replaced by Martha. s/pattern/replacement/g means replace all non-overlapping instances of pattern s/pattern/replacement/2 means replace second instance of pattern Regular Expressions in Sed Which Match? (1) Substitution patterns in sed can be regular expressions. Sometimes input line matches pattern in more than one way. sed -re 's/[0-9]+/number/' Question: What will the following command do sed -re 's/[0-9]*/number/' to this input line: My cat is 16 years old. 1. Non-overlapping matches sed -re 's/one/two/' Input line: one three one sed uses the first match Output: two three one 31 32

9 Which Match? (2) Which Match? (3) Sometimes input line matches pattern in more than one way. Sometimes input line matches pattern in more than one way. 2. Two matches starting at the same spot sed -re 's/a.+b/z/' Input line: axxbxxb Two possible matches: axxbxxb or: axxbxxb sed uses the longer match output is: Z 3. Overlapping matches sed -re 's/a[a-z]{5}/z/' Input line: abcabcdefg Two possible matches: abcabcdefg or: abcabcdefg sed uses the first match output is: Zdefg Advice Using File of Commands Try to make your regular expressions as specific as possible If you want a 6-letter word starting with 't' Possible patterns: t.{5} \<t.{5}\> \<t[[:alpha:]]{5}\> For more than simple one-time edits, put commands in file. Why? only have to type them once easy to include multiple commands sed -rf scriptfile inputfile Applies all commands in scriptfile to inputfile

10 Multiple Commands Suppose script contains these two lines: s/tuesday/wednesday/ s/tues/thurs/ And testfile contains just one line: Today is Tuesday. Command: sed -rf script testfile What will the output be?? Sed's Execution Cycle For every line in the input: 1. Read the next line of input into the "pattern space" 2. Execute all commands in order. Each command operates on the pattern space, not the original line Commands may change the pattern space 3. When all commands are finished for this line, print the pattern space to the standard output. Some commands change this loop we'll see examples later Addresses Put an address or address range at the start of a command to specify which line(s) it applies to Example 1: 5s/Mary/Martha/ Makes the substitution on line 5 of the input only. Example 2: 1,10s/Mary/Martha/ Makes the substitution on lines 1-10 of the input only. Other Forms of Addresses /pattern/: command applies only to lines that match the pattern /Mary/s/little/big/g $: last line of input stream No address on a command: apply command to every line in the file. Caution: line numbers refer to entire input stream! sed -re '10s/a/b/' file1 file2 file3 Command applies to line 10 in the combined stream, not line 10 of each file

11 Deleting Lines Inserting Text d command: delete the current line In other words, skip any later commands that apply and do not copy the pattern space to the output Examples: 1,5d /^$/d /^ *$/d i\ text Text may be more than one line. Each line except the last must end with a backslash. Meaning: send the text to the output immediately (So it appears before the current line.) 1i\ #!/bin/bash 3i\ # comment 1\ # comment Another Example Appending Text sed script: 2i\ Extra line about Mary. s/mary/martha/g What is the effect? a\ text Like i, but text is added to output after the current line is finished. In terms of execution cycle: Text is queued to be output at the end of the current cycle, before the next line is read

12 Using sed to Change a File (1) Using sed to Change a File (2) Common mistake: Better way: sed -re 's/lamb/sheep/g' mary > mary sed -re 's/lamb/sheep/g' mary > temp mv -f temp mary Using sed to Change a File (3) Example (1) Another option: use the -i flag for sed: A sed script to clean up a C program: sed -i -re 's/lamb/sheep/g' mary Be very careful -- syntax of commands with "-i" are tricky! 1i// This program has been modified by Margaret's sed script.\ /^[[:blank:]]*$/d /\<[[:alpha:]]\>/i// One-letter variable name in the\ //following line. Please change it! Difficult to read! Treat sed scripts like programs and think about style

13 Example (2) # A sed script with improved style # Add a comment at the beginning of the file 1i// This program has been modified by Margaret's sed script.\ # Get rid of blank lines /^[[:blank:]]*$/d # Variable name "i" -> "loopcounter" s/\<i\>/loopcounter/g # Insert warning comment about other one-letter # variable names /\<[[:alpha:]]\>/i// One-letter variable name in the\ //following line. Please change it! Using the Preceeding Example sed -rf sedscript myprogram.c Output goes to standard output, myprogram.c unchanged sed -rf sedscript myprogram.c > new.c rm myprogram.c mv new.c myprogram.c New altered version of myprogram.c sed -i -rf sedscript myprogram.c myprogram.c is changed in place sed -irf sedscript myprogram.c confusing error messages! Old Quiz Problem #1 I have several C files in which I want to make two changes: 1. I have used the word "color" in many of the files and I want to correct its spelling to "colour". This should affect "color" used on its own and also inside a word ("colors", "discolor", etc.). 2. I want to remove all lines containing a macro definition -- that is, all lines that start with the character "#", possibly preceeded by spaces. (By "remove" I mean get rid of the line altogether, not replace it with an empty line.) Your job is to write a sed script called change.sed that will accomplish these two changes. I have a directory full of C files and I want to go to that directory and type this command: sed -i -rf change.sed *.c Old Quiz Problem #2 Write a grep (or egrep) command that will find and print any lines in words.txt which contain the same lower-case letter three times in a row. The three occurrences of the letter may appear three times in a row (as in "mooo") or be separated by one or more spaces (as in "x x x") but not by any other characters. For example, if words.txt contains the following three lines: The monster said "grrr"! He lived in an igloo only in the winter. He looked like an aardvark. your command should print just the first two lines 51 52

14 Old Quiz Problem #3 Write a script that takes a directory name as a parameter and prints a list of the base names of all of the files in that directory and its subdirectory tree, in alphabetical order. This list should not include the names of the subdirectories, just regular files. (Hint: besides find, the sort command will help here.) Old Quiz Problem #4 Suppose you have a file called poem.txt. Write a one-line command that will print every line in poem.txt that starts with a or b or c or A or B or C, and ends in a character that is not e or y or E or Y. For example, if poem.txt contains: Bananas are fruit Cherries are nice and apples are yummy cats do not eat grapes Dogs eat apples your command should print: Bananas are fruit cats do not eat grapes Old Exam Problem #1 Old Exam Problem #2 Suppose you have a file called exam.txt. Write a grep (or egrep) command that will print every line in exam.txt that contains exactly two words separated by dashes. For this command, we define a word as a sequence of upper and/or lower case letters. The line must not contain anything except dashes and the two words no spaces, numbers, other punctuation, etc. There may be dashes before the first word and/or after the last word in a line, but there may not be. For example, if exam.txt contains: ----study---study Four-words----in this ----Santa----Claus----- Happy-Hannukah -onelongword---- DONT-PANIC! ---No-more-exams-- -ALL-DONEyour command should print: ----study---study ----Santa----Claus----- Happy-Hannukah -ALL-DONE- 55 Write a shell script that will delete every file which meets all of the following conditions: The name contains the word "bad", in any capitalization as in badfile.txt, verybad.c, NOT_BAD. The file has been modified at some time in the last three days. The file is somewhere in your own directory structure i.e. in your home directory, or a subdirectory of your home directory, or a sub-subdirectory of your home directory, etc. 56