Regular expressions. LING78100: Methods in Computational Linguistics I

Transcription

1 Regular expressions LING78100: Methods in Computational Linguistics I

2 String methods Python strings have methods that allow us to determine whether a string: Contains another string; e.g., assert "and" in "vandal" Starts with another string; e.g., assert "vendetta".startswith("vend") Ends with another string; e.g., assert "vigorous".endswith("us") But what if we want to do more?

3 String matching Regular expressions are a domain-specific language (DSL) for representing strings. With a regular expression we can perform three closely-related operations: Matching: determine whether a given string has a certain desired format Parsing: extracting matching subsets (or groups ) of a given string Substitution: replace particular groups of a given string with another string

4 Notation In what follows, regular expressions are written in blue, underlined typewriter font, and strings are written in red typewriter font. (In Python, regular expressions are specified as strings, though Python compiles them into an faster internal representation.)

5 Learning regular expressions......is a life-long process. There are two parts we ll target this week: The domain-specific language used to specify regular expressions, which is common across many tools/programming languages The software interface for matching, parsing, and substitution, which is language-specific.

6 Motivating examples woo+f matches woof, wooof, woooof, wooooof, and so on. San?tan?a? matches Santa, Satan, and Santana. (\+?(\d{1,3}-)((\d{3})-)?(\d{3})-(\d{4}) matches 7-, 10-, and 11-to-digit phone numbers like and can be used to parse them into a tuple of (country code, area code, exchange, base). A substitution can be used to rewrite every instance of heck as h*ck.

7 The regular expression language

8 Printable characters A regular expression may contain printable alphanumeric characters; e.g., apple matches apple and bass matches bass.

9 The universal character class A period. matches any single character (except newline).

10 Predefined character classes \d: a digit character \s: a whitespace character (such as space, tab, newline, etc.) \w: a word character (i.e., an alphanumeric character) And, their complement classes: \D: a non-digit character \S: a non-whitespace character \W: a non- word character (This is not an exhaustive list.)

11 Ad-hoc character classes Consist of single characters in square brackets; e.g., [234] matches 2, 3, or 4.* Predefined character classes can be used in ad-hoc classes, too; e.g., [\dabc] matches one of {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, c}. A hyphen between two characters can be used to indicate a range; e.g., [a-z]: an lowercase ASCII character [A-Z]: an uppercase ASCII character [A-Za-Z]: an alphabetic ASCII character *In set theory terms, it s a union.

12 Boundary symbols ^: left edge (beginning of string) $: right edge (end of string) These can only be sensibly used at the start (resp. end) of a regular expression.

13 Escaping Characters used for other purposes, including. [ and ] ^ and $ \ must be escaped (preceded by a backslash) to receive a literal interpretation; e.g.: \. matches. \[ matches [ \^ matches ^

14 Basic quantification.? matches zero or one characters.* (or Kleene star ) matches zero or more characters.+ (or Kleene plus ) matches one or more characters E.g.: woo+f matches woof, wooof, woooof, wooooof, and so on. colou?r matches color and colour.

15 Bracket quantification.{n} matches N characters.{m,n} matches between M (inclusive) and M (inclusive) characters.{m,} matches M or more characters.{,n} matches 0 to N characters E.g., wo{2,4}f matches woof, wooof, and woooof.

16 Groups Parentheses are used to bound spans of symbols. There are two uses for this: We can use grouping to recover substrings of the match. E.g., for 7-digit phone numbers we can extract the exchange and the base as separate groups with (\d{3})-?(\d{4}). For the string group 1 is then 867 and group 2 is 5309 (the optional - is not part of a group). We can use grouping to indicate the scope of quantifiers: E.g., ba(na)+ matches bana, banana, bananana, etc. Note that groups can be nested.

17 The operator The operator indicates the union ( either-or ) of multiple substrings. E.g., cigar(ette illo)? matches cigar, cigarette, and cigarillo. Here, the parentheses delimit both the scope of the? quantifier and the operator.

18 Quantifier scope Quantifiers take scope over the preceding character, character class, or group. E.g., \d{3} matches any sequence of three digits. [a-z]{4} matches any sequence of four lowercase ASCII characters. ba(na)+ matches bana, banana, bananana, and so on.

19 Non-greedy matching By default, the quantifiers * (zero or more) and + (one or more) are greedy: they match the largest possible string. To make them non-greedy, we use *? and +?. E.g.: ^.*rd (greedy quantifier) matches the whole string reward the Nord. ^.*?rd (non-greedy quantifier) applied to that string just matches the reward part.

20 Review Printable characters Universal character class. Pre-defined character classes \d, \s, \w, \D, \S, \W,... Ad-hoc character classes (e.g., in square brackets) Boundary symbols ^ and $ Escaping (e.g. \[ vs. [) Quantification (e.g.,?, *, +, *?, {N},...) Groups (e.g., in parentheses) The union operator

21 Demo: re

22 Free advice Use regular expressions for moderately-complex string matching problems Use other tools such as finite-state transducers or context-free grammars when: More than a few regular expressions are required One-to-many substitution is required Linguistic insights are required Performance is required Readability is required Or, if things are getting complicated, consider just enumerating all the possible strings, if they re finite, and storing them in a set. Learn about re flags, which modify the behavior of a match; e.g.: re.ignorecase: makes matching case-insensitive re.dotall: makes. also match newline characters...