CSE : Python Programming

Size: px

Start display at page:

Download "CSE : Python Programming"

Bennett Ray
5 years ago
Views:

1 CSE : Python Programming Lecture 11: Regular expressions April 2,

2 Announcements About those meeting from last week If I said I was going to look into something or you some information, you should send me and remind me We'll have similar meetings near the end of classes Code and documentation for projects is due on the last day of classes (April 20) 2

3 Regular Expressions

4 Survey time (again) Who here knows about the following? Deterministic finite-state automata (DFAs) Non-deterministic finite-state automata (NFAs) Regular languages Recall (if you've taken CSE 260?) that the proof showing the equivalence of the above three gives an algorithm for translating a regular language into an automaton 4

5 Regular expressions (overview) A regular expression is a compact way of specifying a (potentially large) set of strings Example: Compilers and identifiers Source code: Object, LinkedList, my_int, pagelist Regular expression: [a-za-z][a-za-z0-9]* They are useful for finding a particular kind string within some larger string, but they're not great for everything 5

6 Warning: Code Mathematics Mathematics has an idea of what a regular expression is Programming languages also have ideas for this Shells have yet more ideas for this They are not exactly the same! Somewhat different notations and meanings Some languages provide features which have no correspondence to mathematics 6

7 Regular expression in Python The following characters have special meanings inside a regular expression. ^ $ * +? { } [ ] \ ( ) If you want to refer to literally refer to these characters, prefix them with a backslash For example: \. 7

8 Backslash mayhem and raw strings But, you have to give regular expressions as strings to Python, and backslashes have another meaning there For example: '\n' is a one-character string Suppose we want to match the backslash character: Regular expression we have to use: \\ As a Python string: '\\\\' Raw strings, e.g., r'foo' and r"foo", don't interpret backslash characters in any special way 8

9 !"#$%&'()""*#%'+)",-$%&. /%$0',122)%3'!"#$%&'&'1#'!"#()*%+$. (4"-1%'5*&67)#'809#*::$1%'$:'!"##$%$&'(),-*)$*).*)*. $;'#*9*)"$%&'<'1#'21#*'"$2*:. /0$1'2)",-*:'/1='/01='/001='>>>. /,021-$1'2)",-*:'/1='/0211='/ ='>>>..;'#*9*)"$%&'?'1#'21#*'"$2*:. /0.1'2)",-*:'/01='/001='>>>. /,021-.1'2)",-*:'/0211='/ ='>>>. *;'<'1#'?'"$2*:

10 !"#$#%&'$(!)#** +!"!,'#-*!./$01!#$$"%%&!,#&%"'*!$$!/$!%% + '$%()!,#&%"'*!$2!%2(/$!( + /$!*3,4)5!'$*() + '6738#)'-&(&/(#$"%"(&( + '$%()+!,#&%"'*!$2!%2!(2!$$2!$%2!$(2(%$2!%%2! %(2!($2!(%2!((2(999 + ',-)!,#&%"'*!#-5(%"#$!':%'4&!- + ',.*/)!,#&%"'*!#-5(%"#$!':%'4&!#!;3<3& + $'%(0)1!,#&%"'*(,#-5(,/$'(&"#-!$#%(0&1

11 !"#$%&'()&*)+,--./ 0 1/)23"#$%&'(4)5-)3-"')3"#$%&'()#%-)1-(&''&'() 67,#&7')78)")*#,&'( 0!"#$%&)3"#$%-*)#%-)9'.-,:&'-.)6",#)&')!#$#$' 0 (,--./)*-",$%)5&#%)1"$;#,"$;&'( 0!"#$'%(#)3"#$%-*)!#$'#<)!#$'#$'<)!#$' 0 #,/)3"#$%)#%-)6"##-,')!)#$'*(#)5&#%)*#,&'()!#$#' Step Matched Explanation 1 a The a in the RE matches. 2 abcbd The engine matches [bcd]*, going as far as it can, which is to the end of the string. 3 Failure The engine tries to match b, but the current position is at the end of the string, so it fails. 4 abcb Back up, so that [bcd]* matches one less character. 5 Failure Try b again, but the current position is at the last character, which is a d. 6 abc Back up again, so that [bcd]* is only matching bc. 6 abcb Try b again. This time but the character at the current position is b, so it succeeds.

12 !"#$%&'()$*$#+&*", #)$*$#+&*"'-.+)'"%&#.$/'0&$1.12",!"#"$"%"&"'"(")"*"+","-"."/,",.01,/"0$+#)&"".01/,"!"0$+#)&"'$13'".12/&'#)$*$#+&*,"!%"0$+#)&"'$13'"+*.12,",,"0$+#)&"",,"#"0$+#)&"'+)&'4& '56'$'/.1&'5*'"+*.12, '15+'+)&'#'.1".7&'#)$*8#/$""&"'*#999+,"$"0$+#)&"'+)&'&17'56'$'/.1&'5*'"+*.12, 0*123+%1$'75&"'15+'0$+#)'"+*.12'01213

13 Special character classes \d Matches any decimal digit; this is equivalent to the class [0-9]. \D Matches any non-digit character; this is equivalent to the class [ˆ0-9]. \s Matches any whitespace character; this is equivalent to the class [ \t\n\r\f\v]. \S Matches any non-whitespace character; this is equivalent to the class [ˆ \t\n\r\f\v]. \w Matches any alphanumeric character; this is equivalent to the class [a-za-z0-9 ]. \W Matches any non-alphanumeric character; this is equivalent to the class [ˆa-zA-Z0-9_]. 13

14 Overview of functions Method/Attribute match search split sub subn Purpose Determine if the RE matches at the beginning of the string. Scan through a string, looking for any location where this RE matches. Split the string into a list, splitting it wherever the RE matches Find all substrings where the RE matches, and replace them with a different string Does the same thing as sub(), except you can limit the number of replacements 14

15 !"#$%#&'()*+,-./"0 6 $+(,-.7*#"-5#(0*8&9)*'$*$,'2"37*%#*,*&,-./"3*%89".-!!!"#$%&'("')!!!"')*$+(,-./0+1234/5"667 8&9).%&1'2"3*4"#0'%(* '0*$,0-"#*$%#* #"1",-"3*50"!!!"%":"')*,&$%#;)./0+1234/7!!!"% <=>')*?@A=B+(()'9"&CD),("+("EF,G,HE!!!!"%*$+(,-.667!!!"%'#9("%*$+(, &9)!!!"$":"%*$+(,-."/()$%&/7!!!"%'#9("$ <=>')*?@A=I+(,-"&CD),("+("EF,JKLE!!"

16 Methods on match objects Method/Attribute group() start() end() span() Purpose Return the string matched by the RE Return the starting position of the match Return the ending position of the match Return a tuple containing the (start, end) of the match 16

17 !"#$%&'()&*+",$% -!"#$%&'&.+#+,/01+(&02&3"##+,1&/"#$%+(&"#&#%+& &62&"&(#,015 - ()"*$%&'&($"1(&#%,675%&#%+&(#,015&#6&(++&02&"18& (74(#,015&/"#$%+( +++,-*./#,-0!"#$%&1222,!)(("3)1' 45/) +++,!,6,-0()"*$%&1222,!)(("3)1' +++,-*./#,! 7*)08"#$%9:;)$#,./(#"/$),"#,<=$>?@=+ +++,!03*5A-&' 1!)(("3)1 +++,!0(-"/&' &BC,DD'

18 Matching a "word boundary" >>> p = re.compile(r \bclass\b ) >>> print p.search( no class at all ) <re.matchobject instance at 80c8f28> >>> print p.search( the declassified algorithm ) None >>> print p.search( one subclass is ) None A word is defined as a sequence of alpha-numeric characters Whitespace and punction effectively denote the beginning and end of a word 18

19 !"#$%&&'(!"#$!!"#$")*+'( %!"#$%&&'(!&'()&*#!+!,-#(!./!+,,!#)0#(&-*1#!(2+(!3+(42'# %!"#$")*+'(!&'()&*#!+*!-('&+(.&!./!3+(42'5!.06'4(#,,,-.-/-+*0123."&*'45$64(,,,-7-/-489-$+:33*+7;-88-.".*+7;-8<-&2+$74,,,-.0!"#$%&&'( "#)-3%)1G07.%#'( 000 '<;-9( '8F;-8I( '9D;-9J(

20 !"#$%&&'(!"#$!!"#$")*+'( %!"#$%&&'(!&'()&*#!+!,-#(!./!+,,!#)0#(&-*1#!(2+(!3+(42'# %!"#$")*+'(!&'()&*#!+*!-('&+(.&!./!3+(42'5!.06'4(#,,,-.-/-+*0123."&*'45$64(,,,-7-/-489-$+:33*+7;-88-.".*+7;-8<-&2+$74,,,-.0!"#$%&&'( "#)-3%)1G07.%#'( 000 '<;-9( '8F;-8I( '9D;-9J( Notice the "greedy" matching here.

21 !"#$%& +++"""""%'#<("$+D'&E%.3A"$+D'&E% ",4)":;;"./,4)/A"/:;;/3,#4""=>>"./,#4/A"/=>>/3!!!"$"*"%+4)8',9.43!!!"$+D'&E%.3 /,4)":;;/!!!"$+D'&E%.>3 /,4)":;;/!!!"$+D'&E%.F3 /,4)/!!!"$+D'&E%.G3 /:;;/!!!"$+D'&E%4.GAF3./:;;/A"/,4)/3

22 !"#$%& +++"""""%'#<("$+D'&E%.3A"$+D'&E% ",4)":;;"./,4)/A"/:;;/3,#4""=>>"./,#4/A"/=>>/3 Parentheses define groups. Group n starts at the nth open parenthesis.!!!"$"*"%+4)8',9.43!!!"$+d'&e%.3 /,4)":;;/!!!"$+D'&E%.>3 /,4)":;;/!!!"$+D'&E%.F3 /,4)/!!!"$+D'&E%.G3 /:;;/!!!"$+D'&E%4.GAF3./:;;/A"/,4)/3

23 !"#$%"&%'#"()* + #$,$##-&.%/"%)#$0-"(*%.#"()*%12%063%0B3%444!!!"#"$"%&'()*+,-&.%/ /4!!!"5"$"789,5",5"89&""89&"():%5&7!!!"#';,<=>--.54?/89&/@!!!"#'5&>%(9.54'A%):+.4 /89&""89&/ /2)"%5$/$6/$#%78 #%9$:&*%;#:<=!!!"+"$"%&'()*+,-&./.>.14(4=/4!!!"*"$"+'*>8(9./>1(=/4!!!"*'A%):+.C4 />1(=/!!!"*'A%):+.64 />1(/!!!"*'A%):+.B4 /1/!"

24 !"#$%"&%'#"()* + #$,$##-&.%/"%)#$0-"(*%.#"()*%12%063%0B3%444!!!"#"$"%&'()*+,-&.%/ /4!!!"5"$"789,5",5"89&""89&"():%5&7!!!"#';,<=>--.54?/89&/@!!!"#'5&>%(9.54'A%):+.4 /89&""89&/ These refer to the text that was matched, not the pattern. /2)"%5$/$6/$#%78 #%9$:&*%;#:<=!!!"+"$"%&'()*+,-&./.>.14(4=/4!!!"*"$"+'*>8(9./>1(=/4!!!"*'A%):+.C4 />1(=/!!!"*'A%):+.64 />1(/!!!"*'A%):+.B4 /1/!"

25 Non-capturing group (?:regex) Exactly like a normal group (regex), except that it doesn't count for purposes of counting or returning groups from matches >>> p = re.compile(r'.*[.](.*)([12])') >>> p.match('test.backup1').groups() ('backup', '1') >>> p = re.compile(r'.*[.](?:.*)([12])') >>> p.match('test.backup1').groups() ('1',) >>> p = re.compile(r'.*[.](.*)(?:[12])') >>> p.match('test.backup1').groups() ('backup',) 22

26 Named groups (?P<name>regex) Lets you refer to this group by name in addition to by number Analog of '\number' is '(?P=name)'. >>> p = re.compile(r'(?p<word>\b\w+\b)') >>> m = p.search( '(((( Lots of punctuation )))' ) >>> m.group('word') 'Lots' >>> m.group(1) 'Lots' 23

27 Other qualifiers {n,m} says to match between n and m copies {0,} is the same as * {1,} is the same as + {0,1} is the same as? Missing lower limit treated as 0 Missing upper limit treated as "infinity" Append '?' to a qualifier (*, +,?) to make it non-greedy This means: go for the shortest match, not longest 24

28 !"#$%&''()*+,-#./!'& 0 ('1-,2.*3-.45/#%*/6*%&''() 0,6'*#"#$%&''()*7,-#./!'&6*! """#$#%#&'()*+"'(,-."')/)+,"0/)+,'1)/)+,"& """#23/4)#3,5*-)6(7&'58"&9#$:5;3<=27: '()*+"'(,-."')/)+,"0/)+,'1)/)+," """#23/4)#3,5*-)6(7&'58!"&9#$:5;3<=27: '()*+" """#2#%#3,56<*2/+,7&'-#(3,>%758:"&: """#25*-)6(7?'-#(3,>%@?/4.,A5()*+@?"B-6C'1-"?:5;3<=27D: &?/4.,A5()*+?"B-6C'1-"& """#2#%#3,56<*2/+,7&'-#(3,>%758!:"&: """#25*-)6(7?'-#(3,>%@?/4.,A5()*+@?"B-6C'1-"?:5;3<=27D: &?/4.,A5()*+?&

29 Look-ahead assertions (?=regex) Looks for regex at the current spot. Does not consume characters, so the rest of the pattern starts at the same spot regex did. (?!regex) Like the above, except checks to see that regex does not match at the current spot. 26

30 Look-ahead assertions: Example >>> p = re.compile(r'.*[.](?!bat$ exe$)(.*)$') >>> p.match('sendmail.cf').groups() ('cf',) >>> p.match('sendmail.cf').group() 'sendmail.cf' >>> p.match('sendmail.cf.bak').groups() ('bak',) >>> p.match('sendmail.cf.bak').group() 'sendmail.cf.bak' >>> p.match('sendmail.exe.cf').group() 'sendmail.exe.cf' >>> print p.match('sendmail.exe') None 27

31 re.verbose pat = re.compile(r""" \s* # Skip leading whitespace (?P<header>[^:]+) # Header name \s* : # Whitespace, and a colon (?P<value>.*?) # The header's value -- *? used to # lose the following trailing whitespace \s*$ # Trailing whitespace to end-of-line """, re.verbose) Whitespace outside a character class is ignored Can embed Python-style comments Makes long expressions much more readable 28

32 Resources Used as the basis for this lecture The documentation for the re module The documentation for the shlex module Not exactly related to regular expresions Splits strings based on shell-like syntax 29

33 Two more lectures to go Networking Review of Python Something mind-breaking? 30

Regular Expressions. Steve Renals (based on original notes by Ewan Klein) ICL 12 October Outline Overview of REs REs in Python

Regular Expressions. Steve Renals (based on original notes by Ewan Klein) ICL 12 October Outline Overview of REs REs in Python Regular Expressions Steve Renals s.renals@ed.ac.uk (based on original notes by Ewan Klein) ICL 12 October 2005 Introduction Formal Background to REs Extensions of Basic REs Overview Goals: a basic idea