An Introduction to Regular Expressions in Python

Size: px

Start display at page:

Download "An Introduction to Regular Expressions in Python"

Alaina Richards
5 years ago
Views:

1 An Introduction to Regular Expressions in Python Fabienne Braune 1 1 LMU Munich May 29, 2017 Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

2 Outline 1 Introductory Exercise 2 Regular Expressions Basics 3 Advanced Operations on Regex and Match Objects 4 Exercise Session Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

3 Introductory Exercise (10 min.) Consider the following variants of the German verb sagen : sagen, sagt, sagte, gesagt, zugesagt Write a function matchsag() that takes a string as argument and returns: true if the string contains one of the variants false if the string does not contain a variant Er sagt mir immer guten Tag true Das hat er nie gesagt true Er hat wieder was interessantes berichtet false Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

4 Introductory Exercise How many lines of code does your program have? Did you use any special module to implement your code? Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

5 Regular Expressions Basics Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

6 Regular Expressions Regular Expressions (REGEX) describe patterns of text \d stands for digit regex for \d\d\d\d-\d\d\d -\d\d\d-\d\d\d \w stands for word regex for toy238 \w\w\w\w\w\w \w\w\w\d\d\d toy\d\d\d Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

7 Creating Regex Objects Regex functions are in re module import re at beggining of script re class implements regex manipulation Creating regex with re use re object to call functions that manipulate regex re.compile() creates a pattern to match argument is string value describing regex returns pattern object corresponding to regex myphone=re.compile(r \d\d\d\d-\d\d\d-\d\d\d-\d\d\d ) r raw string no need to escape backslashes Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

8 Searching for Regex Objects Searching for regex with re myregex.search() searches myregex in given string argument is string containing regex myphone=re.compile(r \d\d\d\d-\d\d\d-\d\d\d-\d\d\d ) returns Match Object or None match if pattern is matched in string match.group() returns matched text mymatch=myphone.search( Meine Telefonnummer lautet ) None=myphone.search( Meine Telefonnummer lautet 0162-WWSR-hh-2 )) Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

9 Recap: Search for Phone Numbers in Strings 1 import re 2 myphone=re.compile(r \d\d\d\d-\d\d\d-\d\d\d-\d\d\d ) 3 match=myphone.search( Meine Telefonnummer lautet )) 4 print(match.group()) Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

10 Interactive Session Let s search for birth dates in strings! Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

11 Grouping Using groups subpatterns of regex can be printed Create groups by inserting parentheses inside regex myphone=re.compile(r (\d\d\d\d)-(\d\d\d)-(\d\d\d)-\d\d\d ) match=myphone.search( Meine Telefonnummer lautet )) print(match.group(1)) 0162 print(match.group(2)) 787 print(match.group(3)) 334 print(match.group(0)) Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

12 Matching Multiple Groups Sometimes we want to match multiple groups Match multiple groups with mysagen=re.compile(r sag\w\w \w\wsag\w ) mysagen.search( Er sagt ja, Sie sagen nein ) sagen mysagen.search( Sie haben nein gesagt ) gesagt Match object contains first occurrence that matches regex mysagen.search( Sie sagen nein, er hat ja gesagt ) sagen findall() returns list with all regex Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

13 Matching Multiple Groups Convenient to match groups with same prefix Use and parentheses mysagen=re.compile(r sag(te en t ten) ) mysagen.search( Er sagt ja, Sie sagen nein ) sagt mysagen.search( Sie sagten nein ) sagten mysagen.search( Sie haben nein gesagt ) sagt mysagen.findall( Er sagte ja, sie sagt nein und die anderen sagten vielleicht ) sagte, sagt, sagten Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

14 Optional Matching Convenient to match groups optionally? matches 0 or 1 occurrences * matches 0 to n occurrences + matches 1 to n occurrences mysagen=re.compile(r (ge)?sag(te en t ten) ) mysagen.search( Er sagt ja, Sie sagen nein ) sagt mysagen.search( Sie haben nein gesagt ) gesagt mysagen=re.compile(r (ge)*sag(te en t ten) ) mysagen.search( Sie haben nein gegegegesagt ) gegegegesagt Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

15 Optional Matching Also works with single characters? matches 0 or 1 occurrences * matches 0 to n occurrences + matches 1 to n occurrences mysagen=re.compile(r sag(te? en ten) ) mysagen.search( Er sagt ja, Sie sagen nein ) sagt mysagen.search( Er sagte ja, Sie sagten nein ) sagte mysagen=re.compile(r sag(te* en ten) ) mysagen.search( Er sagteeeeee nein ) sagteeeeee Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

16 Matching Repetitions Useful to define number of matches {1,4} matches 1 to 4 repetitions mysagen=re.compile(r (ge){1,4}sag(te en t ten) ) mysagen.search( Sie haben nein gesagt ) gesagt mysagen.search( Sie haben nein gegegegesagt ) gegegegesagt mysagen.search( Sie haben nein gegegegegegegesagt ) no match Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

17 Introductory Exercise (5 min.) Consider the following inflexions of the German verb sagen: sagen, sagt, sagte, gesagt, zugesagt, gesagten, zugesagten, abgesagten Using the regex module write a function matchsag() that takes a string as argument and returns: true if the string contains one of the variants false if the string does not contain a variant Er sagt mir immer guten Tag true Das hat er nie gesagt true Er hat wieder was interessantes berichtet false Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

18 Interactive Session (10 min.) Let s try out your solutions! Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

19 Greedy and Non-Greedy Matching Python regex greedy by default mysagen=re.compile(r (ge){1,4}sag(te en t ten) ) mysagen.search( Sie haben nein gesagt ) gesagt mysagen.search( Sie haben nein gegegegesagt ) gegegegesagt could also match gesagt but takes longest Enable non-greedy mode with? mysagen=re.compile(r (ge){1,4}?sag(te en t ten) ) mysagen.search( Sie haben nein gesagt ) gesagt mysagen.search( Sie haben nein gegegegesagt ) gesagt could also match gegegegesagt but takes shortest Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

20 Matching Beginning of String (only) Use match() instead of search() mysagen=re.compile(r (ge){1,4}sag(te en t ten) ) mysagen.match( Sie haben nein gesagt ) no match mysagen.match( gesagt hat er nichts ) gesagt mysagen.search( gegegegesagten ) gegegegesagten Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

21 Returning all Matched Strings Match object returned by search() contains first matched string What if all matched strings should be returned? Use findall() mysagen=re.compile(r (ge){1,4}sag(te en t ten) ) mysagen.search( gesagt oder nicht gegesagt ) gesagt mysagen.findall( gesagt oder nicht gegesagt ) gesagt,gegesagt when multiple groups in regex, returns a list of tuples Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

22 Some Character Classes \d digit from 0 to 9 \D not digit from 0 to 9 \w word: letter, digit or underscore \W not word: letter, digit or underscore \s space: space, tab or newline \S not space: space, tab or newline mystery=re.compile(r \d+\s\w+ ) What does mystery match? Give examples. Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

23 Creating Character Classes The pre-defined character classes may be too broad Define custom character classes with [ ] [aeiouaeiou] any vowel [a-za-z0-9] all lowercased letters [0-5]? Negative classes are defined with ˆ What does [ˆaeiouAEIOU] match? What does [ˆ0-5] match? Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

24 More Special Symbols ˆ : match beginning of string beg=re.compile(r ˆsag(te en t ten)* ) beg.search( sag hallo ) sag $ : match end of string beg=re.compile(r $sag(te en t ten)* ) beg.search( hallo sagen ) sagen. : match everything beg=re.compile(r $sag(.)* ) beg.search( hallo sagen ) sagen beg.search( hallo sagen sagen sagen ) sagen sagen sagen (greedy) Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

25 Regex symbols recap (10 min.) What do the following symbols do?? * + {n} {n,} {,m} {n,m} *? +? {n,m}? ˆ $. [abc] [ˆ abc] Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

26 Advanced Operations on Regex and Match Objects Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

27 String Modifications Useful to modify string containing regex. beg=re.compile(r ˆsag(te en t ten)* ) beg.search( sag hallo ) sag beg.sub( schrei, sag hallo ) schrei hallo re.sub(r ˆsag(te en t ten)*, schrei, sag hallo ) schrei hallo re.escape(p) : escape pattern re.purge() : empty regex cache Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

28 Advanced Grouping Print list containing all groups with groups() Returns list containing all matched groups Non-capturing groups Match a group but don t save its content use?: at beginning of group mymatch = re.match( (?:Asimov)+, Isaac Asimov ) mymatch.groups() is empty Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

29 Advanced Grouping Named groups Match a group and save its name use?p<abc> at beginning of group mymatch = re.match( (?P<abc>abc)+, abc ) mymatch.group( abc ) returns abc Positive and negative Lookahead use?= at beginning of group mymatch = re.match( Isaac (?=Asimov)+, Isaac Asimov ) mymatch.groups() is Isaac only matches Isaac Asimov Negative lookahead with?! at beginning of group Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

30 Advanced Grouping Print dictionary containing all groups with their names use groupdict() mymatch = re.match( (?P<lemma>sag)(?P<suff>(te ten)), sagten ) mymatch.groupdict() returns lemma: sag, suffix: ten Return span of matched group use span(group) mymatch = re.match( (?P<lemma>sag)(?P<suff>(te ten)), sagten ) mymatch.span(1) returns 0,2 mymatch.start(1) returns 0 mymatch.end(1) returns 2 Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

31 Compilation Flags re.compile(regex,flags=0) Possibility to pass compilation flags Flag Short Description re.ignorecase re.i Case insensitive matching re.multiline re.m ˆ $ also match at beginning of newline re.dotall re.s. matches any symbol (also newline) re.verbose re.x allows to insert comments and newlines re.unicode re.u char sequences dependent on unicode Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

32 Exercise Session Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

33 Digit Manipulation Create a function that: 1 Recognizes a date in YYYY-MM-DD format 2 Modifies YYYY-MM-DD into DD MM YYYY Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

34 Digit Manipulation: Solution import re def rewritedate(s): res=re.search(r (\d{4})-(\d{2})-(\d{2}) ) print res.group(3) + + res.group(2) + + res.group(1) Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

35 Digit and Word Manipulation Given this list: l = [ Neu, Allison, Burns, C. Montgomery, Putz, Lionel, Simpson, Homer Jay ] Write a function that transforms the list into: Allison Neu C. Montgomery Burns Lionel Putz Homer Jay Simpson Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

36 Digit and Word Manipulation: Solution import re def rewritelist(l): for i in l: res = re.search(r ([0-9-]*)\s*([A-Za-z]+),)\s+(.*), i) print res.group(3) + + res.group(2) + + res.group(1) Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

37 Word Manipulation Given a list of words write regexes to: 1 Find all words that include four consecutive vowels 2 Find every word with 5 repeat letters 3 Find all words beginning with sag 4 Find all words containing the segment sag 5 Remove repeat letters Make a function that creates an acronym from a phrase. Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

38 Interactive Session Let s try out your solutions! Fabienne Braune (CIS) An Introduction to Regular Expressions in Python May 29,

Algorithmic Approaches for Biological Data, Lecture #8

Algorithmic Approaches for Biological Data, Lecture #8 Katherine St. John City University of New York American Museum of Natural History 17 February 2016 Outline More on Pattern Finding: Regular Expressions