Paolo Santinelli Sistemi e Reti. Regular expressions. Regular expressions aim to facilitate the solution of text manipulation problems

Transcription

1 aim to facilitate the solution of text manipulation problems are symbolic notations used to identify patterns in text; are supported by many command line tools; are supported by most programming languages; grep (means global regular expression print) grep searches text files for the occurrence of a specified regular expression and outputs any line containing a match to standard output. grep usage: grep [options] regex [file...] regex: regular expression ITIS E. Fermi, Modena 1 /26

2 aim to facilitate the solution of text manipulation problems Option Description i... Ignore case. Do not distinguish between upper and lower case characters. May also be specified --ignore-case. -v... Invert match. Normally, grep prints lines that contain a match. This option causes grep to print every line that does not contain a match. May also be specified --invert-match. -c... Print the number of matches (or non-matches if the -v option is also specified) instead of the lines themselves. May also be specified --count. -l... Print the name of each file that contains a match instead of the lines themselves. May also be specified --files-with-matches. -L Like the -l option, but print only the names of files that do not contain matches. May also be specified --files-withoutmatch -n... Display a prompt for input using the string prompt. Prefix each matching line with the number of the line within the file. May also be specified --line-number. ITIS E. Fermi, Modena 2 /26

3 aim to facilitate the solution of text manipulation problems Option Description h... For multi-file searches, suppress the output of filenames. May also be specified --no-filename --ignore-case. some text files to search ls /bin > dirlist-bin.txt ls /usr/bin > dirlist-usr-bin.txt ls /sbin > dirlist-sbin.txt ls /usr/sbin > dirlist-usr-sbin.txt ls dirlist*.txt dirlist-bin.txt dirlist-sbin.txt dirlist-usr-sbin.txt dirlist-usr-bin.txt ITIS E. Fermi, Modena 3 /26

4 grep simple search grep searches all of the listed files for the string bzip and finds two matches, both in the file dirlist-bin.txt: grep bzip dirlist*.txt dirlist-bin.txt:bzip2 dirlist-bin.txt:bzip2recover -l option: only the list of files that contained matches: grep -l bzip dirlist*.txt dirlist-bin.txt -L option: list of the files that did not contain a match paolo@ubuntu-server:~$ grep -L bzip dirlist*.txt dirlist-sbin.txt dirlist-usr-bin.txt dirlist-usr-sbin.txt ITIS E. Fermi, Modena 4 /26

5 Some Definitions literal: A literal is any character we use in a search or matching expression, for example, to find ind in windows the ind is a literal string - each character plays a part in the search, they form the string we want to find. metacharacter: A metacharacter is one or more special characters that have a unique meaning and are NOT used as literals in the search expression, for example, the character ^ (circumflex or caret) is a metacharacter. target string: This term describes the string that we will be searching, that is, the string in which we want to find our match or search pattern. escape sequence: An escape sequence is a way of indicating that we want to use one of our metacharacters as a literal. In a regular expression an escape sequence involves placing the metacharacter \ (backslash) in front of the metacharacter that we want to use as a literal, for example, if we want to find (s) in the target string window(s) then we use the search expression \(s\) and if we want to find \\file in the target string c:\\file then we would need to use the search expression \\\\file (each \ we want to search for as a literal (there are 2) is preceded by an escape sequence \). ITIS E. Fermi, Modena 5 /26

6 Some Definitions Metacharacter: ^ $. [ ] { } -? * + ( ) \ In addition to literals, regular expressions may also include metacharacters that are used to specify more complex matches ITIS E. Fermi, Modena 6 /26

7 the dot or period character: it is used to match any character. The Any Character If it is included it in a regular expression, it will match any character in that character position. Here s an example: paolo@ubuntu-server:~$ grep -h '.zip' dirlist*.txt bunzip2 bzip2 bzip2recover gunzip gzip funzip gpg-zip preunzip prezip prezip-bin unzip unzipsfx search for any line in files that matches the regular expression.zip, (the length of the required match is four characters). ITIS E. Fermi, Modena 7 /26

8 Anchors The caret (^) and dollar sign ($) characters are treated as anchors: they cause the match to occur only if the regular expression is found at the beginning of the line (^) or at the end of the line ($): Here there are some examples: grep -h '^zip' dirlist*.txt zip zipcloak zipgrep zipinfo zipnote Zipsplit grep -h 'zip$' dirlist*.txt gunzip gzip funzip gpg-zip preunzip prezip unzip Zip grep -h '^zip$' dirlist*.txt zip ITIS E. Fermi, Modena 8 /26

9 Bracket Expressions and Character Classes bracket expressions: match a single character from a specified set of characters; [ ] Match anything inside the square brackets for ONE character position once and only once; set of characters may contain any number of characters, and metacharacters lose their special meaning when placed within brackets; paolo@ubuntu-server:~$ grep -h '[bg]zip' dirlist*.txt bzip2 bzip2recover gzip ITIS E. Fermi, Modena 9 /26

10 Negation ^ : negates the expression; Bracket Expressions and Character Classes If the first character in a bracket expression is a caret (^), the remaining characters are taken to be a set of characters that must not be present at the given character position paolo@ubuntu-server:~$ grep -h '[^bg]zip' dirlist*.txt bunzip2 gunzip funzip gpg-zip preunzip prezip prezip-bin unzip unzipsfx ITIS E. Fermi, Modena 10 /26

11 Bracket Expressions and Character Classes Character Ranges: - (dash) inside square brackets is the 'range separator', it allows to define a range; [ ] could be rewritten as [0-9] paolo@ubuntu-server:~$ grep -h '^[ABCDEFGHIJKLMNOPQRSTUVWXZY]' dirlist*.txt paolo@ubuntu-server:~$ grep -h '^[A-Z]' dirlist*.txt MAKEDEV ControlPanel GET HEAD POST X X11 Xorg MAKEFLOPPIES NetworkManager NetworkManagerDispatcher ITIS E. Fermi, Modena 11 /26

12 Bracket Expressions and Character Classes Character Ranges: Any range of characters can be expressed this way including multiple ranges; [0-9A-C] means check for 0 to 9 and A to C matches all filenames starting with letters and numbers: paolo@ubuntu-server:~$ grep -h '^[A-Za-z0-9]' dirlist*.txt by to include a dash character in to a bracket expression, make it the first in the expression: paolo@ubuntu-server:~$ grep -h '[-AZ]' dirlist*.txt will match every filename containing a dash, or a uppercase A or an uppercase Z. ITIS E. Fermi, Modena 12 /26

13 Bracket Expressions and Character Classes POSIX Character Classes: the POSIX standard includes a number of character classes which provide useful ranges of characters Character Class Description [:alnum:]... The alphanumeric characters. In ASCII, equivalent to: [A-Za-z0-9] [:word:]... The same as [:alnum:], with the addition of the Underscore (_) character. [:alpha:]... The alphabetic characters. In ASCII, equivalent to: [A-Za-z] [:blank:]... Includes the space and tab characters [:cntrl:]... The ASCII control codes. Includes the ASCII characters 0 through 31 and 127 [:digit:]... The numerals zero through nine [:graph:]... The visible characters. In ASCII, it includes Characters 33 through 126. [:lower:]... The lowercase letters [:punct:]... The punctuation characters. In ASCII, equivalent to: [-!"#$%&'()*+,./:;<=>?@[\\\]_`{ }~] [:print:]... The printable characters. All the characters in [:graph:] plus the space character ITIS E. Fermi, Modena 13 /26

14 Bracket Expressions and Character Classes POSIX Character Classes: the POSIX standard includes a number of character classes which provide useful ranges of characters Character Class Description [:space:]... The whitespace characters including space, tab, Carriage return, newline, vertical tab, and form feed. In ASCII, equivalent to: [ \t\r\n\v\f] [:upper:]... The uppercase characters [:xdigit:]... Characters used to express hexadecimal numbers. In ASCII, equivalent to: [0-9A-Fa-f] ITIS E. Fermi, Modena 14 /26

15 Bracket Expressions and Character Classes POSIX Character Classes: the POSIX standard includes a number of character classes which provide useful ranges of characters paolo@ubuntu-server:~$ ls /usr/sbin/[[:upper:]]* /usr/sbin/makefloppies /usr/sbin/networkmanagerdispatcher /usr/sbin/networkmanager ITIS E. Fermi, Modena 15 /26

16 POSIX Basic Vs. Extended Regular Expressions POSIX splits regular expression implementations into two kinds: basic regular expressions (BRE) and extended regular expressions (ERE) POSIX regular expression basic regular expressions (BRE) extended regular expressions (ERE) BRE recognize the following metacharacters recognized: ^ $. [ ] * ( ) { } are treated as metacharacters in BRE if they are escaped with a backslash ERE recognize the following metacharacters recognized: ( ) { }? + ERE are supported by the egrep program, and grep when the -E option is used. ITIS E. Fermi, Modena 16 /26

17 Extended Regular Expressions Alternation: is the facility that allows a match to occur from among a set of expressions paolo@ubuntu-server:~$ echo "AAA" grep -E 'AAA BBB' AAA paolo@ubuntu-server:~$ echo "BBB" grep -E 'AAA BBB' BBB paolo@ubuntu-server:~$ echo "CCC" grep -E 'AAA BBB' paolo@ubuntu-server:~$ the regular expression 'AAA BBB' means match either the string AAA or the string BBB. Alternation is not limited to two choices: paolo@ubuntu-server:~$ echo "AAA" grep -E 'AAA BBB CCC' AAA ITIS E. Fermi, Modena 17 /26

18 Extended Regular Expressions Parenthesis (): is the facility that allows a match to occur from among a set of expressions To combine alternation with other regular expression elements, parenthesis () can be used to separate the alternation This expression matches the filenames that start with either bz, gz, or zip. paolo@ubuntu-server:~$ grep -Eh '^(bz gz zip)' dirlist*.txt This expression matches any filename that begins with bz or contains gz or zip paolo@ubuntu-server:~$ grep -Eh '^bz gz zip' dirlist*.txt ITIS E. Fermi, Modena 18 /26

19 Extended Regular Expressions Quantifiers: they are used to specify the number of times an element is matched? - Match An Element Zero Or One Time this means, Make the preceding element optional Let s say we wanted to check a phone number for validity and we considered a phone number to be valid if it matched either of these two forms: (nnn) nnn-nnnn nnn nnn-nnnn where n is a numeral. We could construct a regular expression like this: ^\(?[0-9][0-9][0-9]\)? [0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]$ ITIS E. Fermi, Modena 19 /26

20 Extended Regular Expressions Quantifiers: they are used to specify the number of times an element is matched? - Match An Element Zero Or One Time this means, Make the preceding element optional paolo@ubuntu-server:~$ echo "(555) " grep -E '^\(?[0-9][0-9][0-9] \)? [0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]$' (555) paolo@ubuntu-server:~$ echo " " grep -E '^\(?[0-9][0-9][0-9]\)? [0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]$' paolo@ubuntu-server:~$ echo "AAA " grep -E '^\(?[0-9][0-9][0-9]\)? [0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]$' paolo@ubuntu-server:~$ ITIS E. Fermi, Modena 20 /26

21 Extended Regular Expressions Quantifiers: they are used to specify the number of times an element is matched * - Match An Element Zero Or More Times it is used to denote an optional item; unlike the?, the item may occur any number of times, not just once. Let s say we wanted to see if a string was a sentence; that is, it starts with an uppercase letter, then contains any number of upper and lowercase letters and spaces, and ends with a period. we could use a regular expression like this: character classes [[:upper:]][[:upper:][:lower:] ]*\. two bracket expressions, * metacharacter, period escaped with a backslash ITIS E. Fermi, Modena 21 /26

22 Extended Regular Expressions Quantifiers: they are used to specify the number of times an element is matched * - Match An Element Zero Or More Times paolo@ubuntu-server:~$ echo "This works." grep -E '[[:upper:]][[:upper:][ :lower:] ]*\.' This works. paolo@ubuntu-server:~$ echo "This Works." grep -E '[[:upper:]][[:upper:][ :lower:] ]*\.' This Works. paolo@ubuntu-server:~$ echo "this does not" grep -E '[[:upper:]][[:upper: ][:lower:] ]*\.' paolo@ubuntu-server:~$ ITIS E. Fermi, Modena 22 /26

23 Extended Regular Expressions Quantifiers: they are used to specify the number of times an element is matched + - Match An Element One Or More Times: the + metacharacter works much like the *, except it requires at least one instance of the preceding element to cause a match. Example: regular expression that match lines consisting of groups of one or more alphabetic characters separated by single spaces: ^([[:alpha:]]+?)+$ paolo@ubuntu-server:~$ echo "This that" grep -E '^([[:alpha:]]+?)+$' This that paolo@ubuntu-server:~$ echo "a b c" grep -E '^([[:alpha:]]+?)+$' a b c paolo@ubuntu-server:~$ echo "a b 9" grep -E '^([[:alpha:]]+?)+$' paolo@ubuntu-server:~$ echo "abc d" grep -E '^([[:alpha:]]+?)+$' paolo@ubuntu-server:~$ ITIS E. Fermi, Modena 23 /26

24 Extended Regular Expressions Quantifiers: they are used to specify the number of times an element is matched { } - Match An Element A Specific Number Of Times the { and } metacharacters are used to express minimum and maximum numbers of required matches. They may be specified in four possible ways: Specifier Meaning {n}... Match the preceding element if it occurs exactly n times {n,m}... Match the preceding element if it occurs at least n times, but no more than m times. {n,}... Match the preceding element if it occurs n or more Times {,m}... Match the preceding element if it occurs no more than m times ITIS E. Fermi, Modena 24 /26

25 Extended Regular Expressions Quantifiers: they are used to specify the number of times an element is matched { } - Match An Element A Specific Number Of Times Check a phone number for validity, consider a phone number as valid if it matched either of these two forms: (nnn) nnn-nnnn or nnn nnn-nnnn, where n is a numeral. two equivalent regular expressions: ^\(?[0-9][0-9][0-9]\)? [0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]$ ^\(?[0-9]{3}\)? [0-9]{3}-[0-9]{4}$ ITIS E. Fermi, Modena 25 /26

26 Extended Regular Expressions Quantifiers: they are used to specify the number of times an element is matched { } - Match An Element A Specific Number Of Times Check a phone number for validity, consider a phone number as valid if it matched either of these two forms: (nnn) nnn-nnnn or nnn nnn-nnnn, where n is a numeral. paolo@ubuntu-server:~$ echo "(555) " grep -E '^\(?[0-9]{3}\)? [0-9] {3}-[0-9]{4}$' (555) paolo@ubuntu-server:~$ echo " " grep -E '^\(?[0-9]{3}\)? [0-9] {3}-[0-9]{4}$' paolo@ubuntu-server:~$ echo "AAA " grep -E '^\(?[0-9]{3}\)? [0-9] {3}-[0-9]{4}$' paolo@ubuntu-server:~$ ITIS E. Fermi, Modena 26 /26