Regular Expressions in Perl

Transcription

1 Regular Expressions in Perl Marco Baroni Computational skills for text analysis

2 Outline Practical advice Regular expressions

3 Practical advice

4 The programming/testing loop Reading output from a text file Keep source code open in Scite (or other editor) Launch program, save output to file Open output file in a different Scite tab Edit code in Scite and save Re-launch program In Scite, refresh the output with the Revert command (or similar commands of other editors)

5 The Brown corpus The Brown University Standard Corpus of Present-Day American English, assembled by Henry Kučera and Nelson Francis in the sixties The first balanced corpus About 1 million words from various kinds of written sources Made available for teaching purposes by kind permission from Kučera Download from the class website, unzip and place the brown.txt file in your working directory

6 Outline Practical advice Regular expressions

7 Regular expressions I omit their precise mathematical characterization (but they do have one) From our point of view, a regular expression is a string that respects a certain syntax, and can be used to look for patterns inside a text Perl was one of the first languages to implement regexps, and it is still one of the languages where they are better integrated This is part of the reason why Perl is popular among linguists

8 A general frame for the programs we will develop The frame: while (<>) { $input = $_; if ($input =~ /REGEXP/) { # regexp match print "$input"; Usage: perl -w prog.pl brown.txt > out.txt

9 First example A simple string is a simple regexp: while (<>) { $input = $_; if ($input =~ /love/) { print "$input"; This also matches glove, clove and Pullover And it does not match loving nor Love

10 Sets and alternatives Use the square brackets [ ] to refer to a character set, such that any character from the set can occur in the target position Use ˆ to match the negation (complement) of the character set (i.e., any character not in the specified set) [aeiou]: any lower case vowel [ˆaeiou] any character except lower case vowels (e.g., A, t, k,!) With ( ), you can look for sets of alternative character sequences For example: lov(e ing)

11 Character sets Use the square brackets [ ] to refer to a character set, such that any single character from the set can occur in the target position For example, [Ll]ove matches both love and Love: while (<>) { $input = $_; if ($input =~ /[Ll]ove/) { print "$input";

12 Character sets What if we want to avoid glove and Lovelace? (let us ignore loves, loving, etc. for now) A simple solution: while (<>) { $input = $_; if ($input =~ / [Ll]ove /) { print "$input"; Are we going to capture all instances of Love and love in this way?

13 Character sets What if we want to avoid glove and Lovelace? (let us ignore loves, loving, etc. for now) A simple solution: while (<>) { $input = $_; if ($input =~ / [Ll]ove /) { print "$input"; Are we going to capture all instances of Love and love in this way? Not quite: we need to express the constraint that the edges around love can contain anything except an alphabetic symbol

14 Character sets Ranges and negations We need to express the constraint that the edges around love can contain anything except an alphabetic symbol All (English) alphabetic symbols: [a-za-z] (this expression denotes two ranges; for digits, you can use: [0-9]) For Italian, you d also need the accented vowels: [a-za-zàèéìòù] Now we negate the previous expression using ˆ: [ˆa-zA-Z] means: anything, except the alphabetic symbols

15 Character sets Anything non-alphabetical at the edges: if ($input = /[ˆa-zA-Z][Ll]ove[ˆa-zA-Z]/) { print "$input"; NB: no spaces in this regular expression: if you see some, they are due to my L A T E X issues

16 Practice time Create a text file with lines to test the regular expression above, e.g.: some love... should match glove should not match

17 Alternative sequences Square brackets only help when alternatives are limited to single characters For alternative sequences, use ( ), e.g.: if ($input =~ /[Ll]ov(e e[sd] ing)/) { print "$input";

18 Modifiers +, * and? specify how many times the immediately preceding character must be repeated +: at least once ehm+ matches ehm, ehmm, ehmmm..., but not, for example, eh *: as many times as you want, including 0 times loves* matches love, loves, lovess... Note equivalence of ehm+ and ehmm*?: once, optionally loves? matches love and loves, but not, e.g., lovess NB: unlike with wildcards, the modifiers refer to the preceding character (lov* matches lo, lov, lovv..., it does not mean lov followed by any character, that is probably what you meant if you wrote that!) We ll do more practice with modifiers after we tokenize the corpus

19 Solving problems with the modifiers Looking for interesting collocations in a flexible manner... if ($input =~ / solve a problem/) { print "$input"; Problems?

20 Solving problems: inflected forms With (......): if ($input =~ / solv(e e[ds] ing) a problem/) { print "$input"; This approach gets tedious soon

21 The + modifier While it is less precise, we can simply say: search solv followed by one or more alphabetic characters: solv[a-z]+ Insert this pattern in the solving problems regexp

22 Solving problems and the/a/some problem With the * and? modifiers if ($input =~ / solv[a-z]+ [a-z]*?problem/) { print "$input";

23 Practice time Write a simple program that extracts lines that contain proper names from the Brown corpus For the current purposes, a proper name is simply a sequence of two capitalized words Problems? Sketches of solutions?

24 Regexp reference table In part from Kenneth Church: Unix for Poets Example Explanation a letter a [a-z] any lower case English letter [a-zàèéìòù] any lower case Italian letter [A-Z] any upper case English letter [A-ZÀÈÉÌÒÙ] any upper case Italian letter [ ] any digit [0-9] any digit [aeiouaeiou] any English vowel [ˆaeiouAEIOU] any character except the vowels. any character ˆ beginning of string $ end of string x* zero or more x x+ one or more x x? zero or one x (xz yw) xz or yw