Topic 4: Grep, Find & Sed

Size: px

Start display at page:

Download "Topic 4: Grep, Find & Sed"

Jonas Newman
5 years ago
Views:

1 Topic 4: Grep, Find & Sed grep: a tool for searching for strings within files find: a tool for examining a directory tree sed: a tool for "batch editing" Associated topic: regular expressions 1

2 Motivation (Scenarios) I have a file called (something)220(something).java which contains the phrase "final exam" in a comment. Where is it? I need to find all the files in a directory tree containing "2010" and change to "2011". I need to find all the files and directories in a directory tree whose names contain "220" and change to "220A" I want to delete all files in a directory tree whose names end in "~" or start and end with "#" Style changes in large multi-file C program: change header comment and names of some functions. 2

3 Use to search files for strings grep grep <pattern> <files> Searches files and prints all lines containing the pattern Example: grep Frost * Flags: -i: case-insensitive match -w: string must appear as an entire word, not part of a word -num: num lines of context around each match 3

4 Displaying File Names Default behavior: shows file names only if searching >1 file. Flags: -h: never show file names -H: always show file names -l: show only the file names (not the matches) 4

5 No file arguments: grep reads standard input (useful for pipes!) Example: history grep cd Grep as a filter 5

6 Using grep in a conditional exit status: 0 = at least one match was found 1 = no matches found 2 = error useful flag: -q: no output (quiet) Example: if grep -q $word $filename then... 6

7 Regular Expressions Pattern to grep in previous examples: literal value For more complex searches: use a regular expression In this course: using exended regular expression syntax much easier than the default syntax! use -E flag (or egrep command) put patterns inside single quotes Regular expressions not to be confused with wildcards in file names! 7

8 Match A Single Character. in pattern: matches any single character Example: grep -E 'n.t' * Matches: not, nat, nit, etc. Does not match: nt or naut 8

9 Optional Items?: Preceeding item is optional Example: grep -Ew 'loved?' * Matches love or loved 9

10 Grouping Items Can group items into one using parenthesis Example: grep -Ew 'love(ly)?' * Matches love or lovely 10

11 Repeated Items *: Match preceding item zero or more times +: Match preceding item one or more times {n}: Match preceding item exactly n times Examples: grep -E 'a(bc)*d' * matches ad, abcd, abcbcd, etc. grep -E 'a(bc)+d' * matches abcd, abcbcd, etc., but not ad grep -E 'a(bc){2}d' * matches abcbcd and nothing else 11

12 Alternatives Two regular expressions joined with matches either expression. Example: grep -E 'red black' * Matches red or black Precedence: highest: repetition (* or +) next: concatenation (two items in a row) lowest: alternation ( ) When in doubt, use parenthesis 12

13 Bracket Expressions Sequence of characters in brackets: matches any one of those characters. Example: grep -E 'n[aeiou]t' * Maches nat, net, nit, not, nut Use dash to specify a character range: [b-d] matches b, c or d [b-dr-t] matches b,c,d,r,s or t ^ at the beginning of a bracket expression: matches any character except the characters in the bracket [^b-dr-t] matches any character that is not b,c,d,r,s or t 13

14 Character Classes Special notation for matching common groups of characters [:digit:] [:alpha:] - any letter (upper or lower case) & more on summary sheet These may used only inside bracket expressions Examples: egrep -w '[[:alpha:]]{4}' egrep 'linux[[:digit:]]\.[[:digit:]]+' egrep -w '[[:alpha:]3]+' 14

15 Matching Word & Line Positions \<: matches beginning of word (letters, digits, underscores) \>: matches end of word ^: matches start of line $: matches end of line These are not characters! They match positions only. 15

16 Quoting Special Characters \ before any character that normally has a special meaning means match the character literally Example:. matches any single character \. matches a period. 16

17 Reminder For Grep Grep matches any line with with a substring that matches the pattern. Example: grep 'cat' Matches any line containing cat. Not necessary to write egrep '^.*cat.*$' 17

18 find (1) Searches directory trees for files having particular properties. General form: find [dir...] [test...] [action...] Default action: print name of file Simplest way to use: find files based on their names. find the file called podcast.xml find all CSS files (names ending with.css) Searches directory and its sub-directories no limit on depth case-insensitive name match: find web220 iname '*.html' Note: find includes hidden files and directories (names starting 18

19 find (2): finding files by type find web220 type f shows names of all regular files in web220 find web220 type d shows names of all directories in web220 find web220 type l shows names of all symbolic links in web220 Can combine tests: all must be true find web220 type d name 's*' shows names of all directories whose names start with s 19

20 find (3): finding files by size find web220 size 4157c prints name of all files whose size is exactly 4157 bytes find web220 size +4157c prints name of all files in web220 whose size is > 4157 bytes find web220 size -4157c prints name of all files in web220 whose size is < 4157 bytes size modifiers: c = bytes k = kilobytes M = megabytes G = gigabytes Sizes are rounded up! 20

21 find (4): finding files by age find mydir mtime 4 prints names of all files modified 4 days ago (number of days rounded down) find mydir mtime +4 prints names of all files modified more than 4 days ago -4 for less than 4 days ago 21

22 find (5): finding files by age, continued -mtime asks about time in days -mmin asks about time in minutes Example: find mydir mmin -120 shows names of files modified less than 2 hours ago -atime and amin ask about access times, not modification times Example: find mydir amin -120 shows names of files looked at less than 2 hours ago 22

23 find (6): actions Default action: -print (print the name of the file) Another possible action: -delete Handy for cleaning up. (Careful: no confirmation!) Example: emacs creates backup files ending in ~ find mydir name '*~' delete Deletes all emacs backup files Suppose you'd like to see the names of the files that are deleted. Use print action first. find mydir name '*~' print delete 23

24 find (7): Combining with other Programs Goal: use find to identify files and then run other programs on those files. FILES=$(find...) for FILE in $FILES do... commands using $FILE done or: for FILE in $(find...) do... commands using $FILE done Caution:file names containing spaces are a problem! 24

25 Examples clean-up script Use find to write-protect a directory tree Show just the base names of all files in directory Find all files in web site containing the word "quiz" 25

26 sed sed = a "stream editor" editing in batch mode automates repetitive edits Scenario: CISC 123 last offered in winter 2006 Offering CISC 123 again in fall 2011, need a new web site To start web site: copy all files, make some global changes (winter to fall, 2006 to 2011, instructor's name, etc) Options: 1. Open each file by hand and make the edits. 2. Write a Java/C/Python program to make the edits 3. Write a script using find & sed to run program on all files Which option sounds like less work? Which option is more error-prone? 26

27 Simple Ways To Run Sed To run a single sed command on one or more files: sed -r -e 'command' filename1 filename2... Output goes to standard output (can redirect or pipe) -r means use extended regular expression syntax (like -E for grep) -e means next argument will be a sed command Can combine, but e must come last: sed -re 's/mary/martha/' myfile > newfile No file name: apply command to standard input (useful for pipes or experiments) 27

28 Simple Substitute Command in sed s/pattern/replacement/ pattern = regular expression replacement: string to substitute for the match Example: s/mary/martha/ First occurrence of Mary in each line replaced by Martha. 28

29 Global Substitutions s/pattern/replacement/g means replace all non-overlapping instances of pattern s/pattern/replacement/2 means replace second instance of pattern 29

30 Regular Expressions in Sed Substitution patterns in sed can be regular expressions. sed -re 's/[0-9]+/number/' Question: What will the following command do sed -re 's/[0-9]*/number/' to this input line: My cat is 16 years old. 30

31 Which Match? (1) Sometimes input line matches pattern in more than one way. 1. Non-overlapping matches sed -re 's/one/two/' Input line: one three one sed uses the first match Output: two three one 31

32 Which Match? (2) Sometimes input line matches pattern in more than one way. 2. Two matches starting at the same spot sed -re 's/a.+b/z/' Input line: axxbxxb Two possible matches: axxbxxb or: axxbxxb sed uses the longer match output is: Z 32

33 Which Match? (3) Sometimes input line matches pattern in more than one way. 3. Overlapping matches sed -re 's/a[a-z]{5}/z/' Input line: abcabcdefg Two possible matches: abcabcdefg or: abcabcdefg sed uses the first match output is: Zdefg 33

34 Advice Try to make your regular expressions as specific as possible Example: If you want a 6-letter word starting with 't' Possible patterns: t.{5} \<t.{5}\> \<t[[:alpha:]]{5}\> 34

35 Using File of Commands For more than simple one-time edits, put commands in file. Why? only have to type them once easy to include multiple commands sed -rf scriptfile inputfile Applies all commands in scriptfile to inputfile. 35

36 Multiple Commands Suppose script contains these two lines: s/tuesday/wednesday/ s/tues/thurs/ And testfile contains just one line: Today is Tuesday. Command: sed -rf script testfile What will the output be?? 36

37 Sed's Execution Cycle For every line in the input: 1. Read the next line of input into the "pattern space" 2. Execute all commands in order. Each command operates on the pattern space, not the original line Commands may change the pattern space 3. When all commands are finished for this line, print the pattern space to the standard output. Some commands change this loop we'll see examples later. 37

Motivation (Scenarios) Topic 4: Grep, Find & Sed. Displaying File Names. grep

Motivation (Scenarios) Topic 4: Grep, Find & Sed. Displaying File Names. grep Topic 4: Grep, Find & Sed grep: a tool for searching for strings within files find: a tool for examining a directory tree sed: a tool for "batch editing" Associated topic: regular expressions Motivation