Pattern Matching. exact pattern matching Knuth-Morris-Pratt RE pattern matching grep

Transcription

1 Pttern Mtching exct pttern mtching Knuth-Morris-Prtt RE pttern mtching grep exct pttern mtching Knuth-Morris-Prtt RE pttern mtching grep References: Algorithms in C (nd edition), Chpter 9 (pdf online) Roert Sedgewick nd Kevin Wyne Copyright 8 Algorithms in Jv, th Edition April, 8 9:8: AM Exct pttern mtching Applictions Gol. Find pttern of length M in text strem of length N. typiclly N >> M pttern n e e d l e text i n h y s t c k n e e d l e i n Computer forensics. Serch memory or disk for signtures, e.g., ll URLs or RSA keys tht the user hs entered. Prsers. Spm filters. Digitl lirries. Screen scrpers. Word processors. We serch engines. Nturl lnguge processing. Computtionl moleculr iology. Feture detection in digitized imges.

2 Spm filtering Screen scrping Identify ptterns indictive of spm. PROFITS AMAZING GUARANTEE LSE WEGHT herl Vigr There is no ctch. LW MRTGAGE RATES This is one-time miling. This messge is sent in complince with spm regultions. You're getting this messge ecuse you registered with one of our mrketing prtners. Gol. Extrct relevnt dt from we pge. Ex. Find string delimited y <> nd </> fter first occurrence of pttern Lst Trde:. <tr> <td clss= "yfnc_tlehed" width= "8%"> Lst Trde: </td> <td clss= "yfnc_tledt"> <ig><>.9</></ig> </td></tr> <td clss= "yfnc_tlehed" width= "8%"> Trde Time: </td> <td clss= "yfnc_tledt">... 6 Exct pttern mtching in Jv Brute-force exct pttern mtch The method s.indexof(pttern, offset) in Jv's String lirry returns the index of the first occurrence of pttern in string s, strting t given offset. Check for pttern strting t ech text position. pulic clss StockQuote pulic sttic void min(string[] rgs) String nme = " In in = new In(nme + rgs[]); String input = in.redall(); int strt = input.indexof("lst Trde:", ); int from = input.indexof("<>", strt); int to = input.indexof("</>", from); String price = input.sustring(from +, to); StdOut.println(price); % jv StockQuote goog.9 h y n e e d s n x % jv StockQuote msft

3 Brute-force exct pttern mtch: Jv implementtion Brute-force exct pttern mtch: worst cse Check for pttern strting t ech text position. Brute-force lgorithm cn e slow if text nd pttern re repetitive. pulic sttic int serch(string pttern, String text) int M = pttern.length(); int N = text.length(); for (int i = ; i < N - M; i++) int j; for (j = ; j < M; j++) if (text.chrat(i+j)!= pttern.chrat(j)) rek; if (j == M) return i; index in text where pttern strts return -; not found Worst cse. ~ MN chr compres. 9 Algorithmic chllenges in pttern mtching Brute-force is not good enough for ll pplictions. Theoreticl chllenge. Liner-time gurntee. fundmentl lgorithmic prolem Prcticl chllenge. Avoid ckup in text strem. often no room or time to sve text Now is the time for ll people to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for mny good people to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for lot of good people to come to the id of their prty. Now is the time for ll of the good people to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for ech good person to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for ll good Repulicns to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for mny or ll good people to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for ll good Democrts to come to the id of their prty. Now is the time for ll people to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for mny good people to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for lot of good people to come to the id of their prty. Now is the time for ll of the good people to come to the id of their prty. Now is the time for ll good people to come to the id of their ttck t dwn prty. Now is the time for ech person to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for ll good Repulicns to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for mny or ll good people to come to the id of their prty. Now is the time for ll good people to come to the id of their prty. Now is the time for ll good Democrts to come to the id of their prty. exct pttern mtching Knuth-Morris-Prtt RE pttern mtching grep

4 Knuth-Morris-Prtt exct pttern-mtching lgorithm Deterministic finite-stte utomt KMP. Clssic lgorithm tht meets oth chllenges. Liner-time gurntee. No ckup in text strem. DFA review. Finite numer of sttes (including strt nd ccept). Exctly one trnsition for ech input symol. Accept if sequence of trnsitions leds to ccept stte. Bsic pln (for inry lphet). Build DFA from pttern. Simulte DFA with text s input. Don Knuth Jim Morris Vughn Prtt text DFA for pttern ccept reject pttern in text pttern NOT in text Q. Which itstrings does this DFA ccept? Knuth-Morris-Prtt DFA exmple Knuth-Morris-Prtt DFA simultion One stte for ech pttern chrcter. Mtch input chrcter: move from i to i+. Mismtch: move to previous stte. DFA for pttern 6

5 Knuth-Morris-Prtt DFA simultion Knuth-Morris-Prtt DFA simultion When in stte i. Mtches in i previous input chrs (nd is longest such mtch). Ex. End in stte iff text ends in. Ex. End in stte iff text ends in (ut not or ). ccept! 7 8 Knuth-Morris-Prtt implementtion Knuth-Morris-Prtt: Jv implementtion DFA representtion. A single stte-indexed rry next[]. Upon chrcter mtch in stte j, go forwrd to stte j+. Upon chrcter mismtch in stte j, go ck to stte next[j]. Two key differences from rute-force implementtion. Text pointer i never decrements. Need to precompute next[] tle (DFA) from pttern. 6 Simultion of KMP DFA next DFA for pttern only need to store mismtches int j = ; for (int i = ; i < N; i++) if (text.chrat(i) == pttern.chrat(j)) j++; // chr mtches else j = next[j]; // chr mismtch if (j == M) return i - M + ; // found pttern return -; // not found 9

6 Knuth-Morris-Prtt: incrementl DFA construction Knuth-Morris-Prtt DFA construction: two cses Key ide. DFA for first i sttes contins info needed to uild stte i+. Let X e the next stte in the simultion nd j the next stte to uild. Ex. Given DFA for pttern, to compute DFA for pttern : On mismtch t 7th chr, need to simulte 6-chr ckup. Previous 6 chrs re known ( in exmple). 6-stte DFA (known) determines next stte! 6-chr ckup Cse. If p[x] nd p[j] mtch, copy nd increment. next[j] = next[x] X = X + stte for X X stte for j Q. How to do efficiently? A. Keep trck of DFA stte for pttern, strting t nd chr. 6 p[] next[] 6 DFA for pttern DFA for pttern Knuth-Morris-Prtt DFA construction: two cses Knuth-Morris-Prtt DFA construction Let X e the next stte in the simultion nd j the next stte to uild. Cse. If p[x] nd p[j] mismtch, do the opposite. next[j] = X + X = next[j] stte for stte for 6 p[] next[] X j X j mtch mismtch mtch X: current stte in simultion compre p[j] with p[x] mtch: copy nd increment next[j] = next[x]; X = X + ; mismtch: do the opposite next[j] = X + ; X = next[x]; DFA for pttern 6 mtch mismtch

7 DFA construction for KMP: Jv implementtion Optimized KMP implementtion int X = ; int[] next = new int[m]; for (int j = ; j < M; j++) if (pttern.chrat(x) == pttern.chrat(j)) // mtch next[j] = next[x]; X = X + ; else // mismtch next[j] = X + ; X = next[x]; DFA Construction for KMP (ssumes inry lphet) Anlysis. Tkes time nd spce proportionl to pttern length. Ultimte serch progrm for ny given pttern. One sttement compring ech pttern chrcter to next. Mtch: proceed to next sttement. Mismtch: go ck s dictted y DFA. Trnsltes to mchine lnguge (three instructions per pttern chr). int kmpserch(chr text[]) int i = ; s: if (text[i++]!= '') goto s; s: if (text[i++]!= '') goto s; s: if (text[i++]!= '') goto s; s: if (text[i++]!= '') goto s; s: if (text[i++]!= '') goto s; s: if (text[i++]!= '') goto s; s6: if (text[i++]!= '') goto s; s7: if (text[i++]!= '') goto s; return i - 8; ssumes pttern is in text (o/w use sentinel) pttern[] next[] 6 Knuth-Morris-Prtt summry Exct pttern mtching: other pproches Generl lphet. More difficult. Esy with next[i][c] indexed y mismtch position i, chrcter c. KMP pper hs ingenious solution tht uses single D next[] rry. [ uild NFA, then prove tht it finishes in N steps ] Rin-Krp: mke digitl signture of the pttern. Hshing without the tle. Liner-time proilistic gurntee. Plus: extends to D ptterns. Minus: rithmetic ops slower thn chr comprisons. Bottom line. Liner-time pttern mtching is possile (nd prcticl). Short history. Inspired y esoteric theorem of Cook. Discovered in 976 independently y two theoreticins nd hcker. - Knuth: discovered liner time lgorithm - Prtt: mde running time independent of lphet - Morris: trying to uild text editor Theory meets prctice. 7 Boyer-Moore: scn from right to left in pttern. Min ide: cn skip M text chrs when finding one not in the pttern. Needs dditionl KMP-like heuristic. Plus: possiility of suliner-time performnce (~ N/M ). Used in Unix, emcs. pttern s y z y g y text s y z y g y s y z y g y s y z y g y 8

8 Exct pttern mtch cost summry Cost of serching for n M-chrcter pttern in n N-chrcter text. lgorithm opertions typicl worst-cse rute-force chr compres. N M N KMP chr compres. N N Krp-Rin rithmetic ops N N Boyer-Moore chr compres N/M N ssumes pproprite model rndomized exct pttern mtching Knuth-Morris-Prtt RE pttern mtching grep 9 Regulr-expression pttern mtching RE pttern mtching: pplictions Exct pttern mtching. Find occurrences of single pttern in text. RE pttern mtching. Find occurrences of one of multiple ptterns in text. Ex. (genomics) Frgile X syndrome is common cuse of mentl retrdtion. Humn genome contins triplet repets of cgg or gg, rcketed y gcg t the eginning nd ctg t the end. Numer of repets is vrile, nd correlted with syndrome. Use RE to specify pttern: gcg(cgg gg)*ctg. Do RE pttern mtch on person s genome to detect Frgile X. pttern (RE) text gcg(cgg gg)*ctg gcggcgtgtgtgcggggtgggtttgctggcgcggggcggctggcgcggggctg Test if string mtches some pttern. Process nturl lnguge. Scn for virus signtures. Serch for informtion using Google. Access informtion in digitl lirries. Retrieve informtion from Lexis/Nexis. Serch-nd-replce in word processors. Filter text (spm, NetNnny, Crnivore, mlwre). Vlidte dt-entry fields (dtes, emil, URL, credit crd). Serch for mrkers in humn genome using PROSITE ptterns. Prse text files. Compile Jv progrm. Crwl nd index the We. Red in dt stored in d hoc input file formt. Automticlly crete Jv documenttion from Jvdoc comments.

9 Regulr expression exmples Regulr expression exmples (continued) A regulr expression is nottion to specify set of strings. Nottion is surprisingly expressive opertion exmple RE in set not in set regulr expression in set not in set conctention every other string wildcrd union.u.u.u. cumulus jugulum succuus tumultuous every other string.*sp.* (contins the trigrph sp) * (****)* (numer of 's is multiple of ) rsperry crispred suspce suspecies closure *.*... (fifth to lst digit is ) prentheses ( ) ()* every other string gcg(cgg gg)*ctg (frgile X syndrome) gcgctg gcgcggctg gcgcggggctg gcgcgg cggcggcggctg gcgcggctg nd plys well-understood role in the theory of computtion. Generlized regulr expressions Regulr expressions in Jv Additionl opertions re often dded for convenience. Ex. [-e]+ is shorthnd for ( c d e)( c d e)* opertion exmple RE in set not in set one or more chrcter clsses (c)+de [A-Z-z][-z]* cde ccde word Cpitlized de cde cmelcse illegl Vlidity checking. Is input in the set descried y the re? Jv string lirry. Use input.mtches(re) for sic RE mtching. pulic clss Vlidte pulic sttic void min(string[] rgs) String re = rgs[]; String input = rgs[]; oolen isvlid = input.mtches(re); StdOut.println(isVlid); exctly k [-9]-[-9] negtions [^eiou]6 rhythm decde % jv Vlidte "..oo..oo." loodroot true % jv Vlidte "[$_A-Z-z][$_A-Z-z-9]*" ident true % jv Vlidte "[-z]+@([-z]+\.)+(edu com)" rs@cs.princeton.edu true need help solving crosswords? legl Jv identifier vlid emil ddress (simplified) Cvet. Need to e lert for non-regulr dditions, e.g., ck reference. % jv Vlidte "[-9]-[-9]-[-9]" 66-- true Socil Security numer 6

10 Regulr expressions in other lnguges Regulr expression cvet Brodly pplicle progrmmer's tool. Originted in Unix in the 97s Mny lnguges support extended regulr expressions. Built into grep, wk, emcs, Perl, PHP, Python, JvScript. % grep NEWLINE */*.jv print ll lines contining NEWLINE which occurs in ny file with.jv extension % egrep '^[qwertyuiop]*[zxcvnm]*$' dict.txt egrep '...' Writing RE is like writing progrm. Need to understnd progrmming model. Cn e esier to write thn red. Cn e difficult to deug. Sometimes you hve progrmming prolem nd it seems like the est solution is to use regulr expressions; now you hve two prolems. PERL. Prcticl Extrction nd Report Lnguge. % perl -p -i -e 's from to g' input.txt replce ll occurrences of from with to in the file input.txt % perl -n -e 'print if /^[A-Z-z][-z]*$/' dict.txt print ll uppercse words do for ech line 7 8 Cn the verge we surfer lern to use REs? Cn the verge TV viewer lern to use REs? Google. Supports * for full word wildcrd nd for union. TiVo. WishList hs very limited pttern mtching. Reference: pge 76, Hughes DirectTV TiVo mnul 9

11 Cn the verge progrmmer lern to use REs? Perl RE for vlid RFC8 emil ddresses (?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?: \r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\ ](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\](?: (?:\r\n)?[ \t])*))* \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])* )(?:\.(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))*) *:(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r \n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t ]))*"(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\](?:(? :\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*) \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z "(?:[^\"\r\\] \\. (?:(?:\r\n)? [ \t]))*"(?:(?:\r\n)?[ \t])*)*:(?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) " (?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\ ".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[ \]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))* (?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n)?[ \t])*(?:@(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) \[([ ^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) \[([^\[\ ]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))*(?:,@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) \[([^\[\]\ r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))*)*:(?:(?:\r\n)?[ \t])*)?(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) "(? :[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\". \[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\] ])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*)(?:,\s*(?:(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\ ".\[\]])) "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[ \["()<>@,;:\\".\[\]])) "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t ])+ \Z (?=[\["()<>@,;:\\".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))* (?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\ ]])) "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n)?[ \t])*(?:@(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\[" ()<>@,;:\\".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))*(?:,@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@, ;:\\".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\ ".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))*)*:(?:(?:\r\n)?[ \t])*)?(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\". \[\]])) "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\[ "()<>@,;:\\".\[\]])) "(?:[^\"\r\\] \\. (?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t]) + \Z (?=[\["()<>@,;:\\".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \-\]+(?:(?:(?:\r\n)?[ \t])+ \Z (?=[\["()<>@,;:\\".\[\]])) \[([^\[\]\r\\] \\.)*\](?:(?:\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*))*)?;\s*) exct pttern mtching Knuth-Morris-Prtt RE pttern mtching grep http GREP implementtion: sic pln Dulity Overview is the sme s for KMP! Liner-time gurntee. No ckup in text strem. Bsic pln for grep (generlized regulr expression print). Build DFA from RE. Simulte DFA with text s input. Ken Thompson RE. Concise wy to descrie set of strings. DFA. Mchine to recognize whether given string is in given set. Kleene's theorem. For ny DFA, there exists RE tht descries the sme set of strings. For ny RE, there exists DFA tht recognizes the sme set of strings. input ctgtgcggggcggcgcggcggggggctggcg DFA for pttern gcg(cgg gg)*ctg ccept reject pttern in text pttern NOT in text RE * (****)* numer of 's is multiple of DFA numer of 's is multiple of Good news. Bsic pln works. Bd news. The DFA cn e exponentilly lrge. Consequence. Need etter strct mchine.

12 Nondeterministic finite-stte utomt GREP implementtion: sic pln (revised) NFA. My hve,, or more trnsitions for ech input symol. My hve ε-trnsitions (move to nother stte without reding input). Accept if ny sequence of trnsitions leds to ccept stte. Bsic pln for GREP. uild NFA from RE. Simulte NFA with text s input. Give up on liner-time gurntee (ut not poly-time gurntee). Ken Thompson convention: unleled rrows re ε-trnsitions in set:,, not in set:,, itstrings tht do not contin polynomil exponentil lowup possile input ctgtgcggggcggcgcggcggggggctggcg NFA for pttern gcg(cgg gg)*ctg ccept reject pttern in text pttern NOT in text Proof of Kleene s theorem. RE NFA DFA RE. 6 Simulting n NFA NFA simultion Q. How to efficiently simulte n NFA? A. Mintin SET of ll possile sttes tht NFA could e in fter reding in the first i symols. Q. How to perform rechility? A. Grph rechility in Digrph (!) 7 8

13 Converting from n RE to n NFA: sic trnsformtions Converting from n RE to n NFA exmple: * * Use generlized NFA with full RE on trnsitions. Strt with one trnsition hving given RE. Remove opertors with trnsformtions given elow. Gol: stndrd NFA (ll single-chrcter or ε-trnsitions). strt from R to conctention closure * * * * * * union from from from R S R from S from c R to c R from c* S to c S * * to to to to 9 Grep running time Industril-strength grep implementtion Input. Text with N chrcters, RE with M chrcters. Clim. The numer of edges in the NFA is t most M. Single chrcter: consumes symol, cretes edge. Wildcrd chrcter: consumes symol, cretes edges. Conctention: consumes symols, cretes edges. Union: consumes symol, cretes edges. Closure: consumes one symol, cretes edges. NFA simultion. O(MN) since NFA hs M trnsitions Bottleneck: grph rechility per input chrcter. Cn e sustntilly fster in prctice if few ε-trnsitions. To complete the implementtion, Del with prentheses. Extend the lphet. Add chrcter clsses. Add cpturing cpilities. Del with met chrcters. Extend the closure opertor. Error checking nd recovery. Greedy vs. reluctnt mtching. NFA construction. Ours is O(M ) ut not hrd to mke O(M).

14 Hrvesting informtion Regulr expressions in Jv (revisited) Gol. Print ll sustrings of input tht mtch RE. RE pttern mtching is implemented in Jv s Pttern nd Mtcher clsses. % jv Hrvester "gcg(cgg gg)*ctg" chromosomex.txt gcgcggcggcggcggcggctg gcgctg gcgctg hrvest ptterns from DNA gcgcggcggcggggcggggcggctg hrvest links from wesite % jv Hrvester " import jv.util.regex.pttern; import jv.util.regex.mtcher; pulic clss Hrvester pulic sttic void min(string[] rgs) String re = rgs[]; In in = new In(rgs[]); String input = in.redall(); Pttern pttern = Pttern.compile(re); Mtcher mtcher = pttern.mtcher(input); while (mtcher.find()) StdOut.println(mtcher.group()); compile() cretes Pttern (NFA) from RE mtcher() cretes Mtcher (NFA simultor) from NFA nd text find() looks for the next mtch group() returns the sustring most recently found y find() Algorithmic complexity ttcks Not-so-regulr expressions Wrning. Typicl implementtions do not gurntee performnce! grep, Jv, Perl % jv Vlidte "( )*" c.6 seconds % jv Vlidte "( )*" c.7 seconds % jv Vlidte "( )*" c 9.7 seconds % jv Vlidte "( )*" c. seconds % jv Vlidte "( )*" c 6. seconds % jv Vlidte "( )*" c 6.6 seconds Bck-references. \ nottion mtches su-expression tht ws mtched erlier. Supported y typicl RE implementtions. % jv Hrvester "\(.+)\\" dictionry.txt erieri couscous word oundry SpmAssssin regulr expression. % jv RE "[-z]+@[-z]+([-z\.]+\.)+[-z]+" spmmer@x... Tkes exponentil time. Spmmer cn use pthologicl emil ddress to DOS mil server. Some non-regulr lnguges. Set of strings of the form ww for some string w: erieri. Set of itstrings with n equl numer of s nd s:. Set of Wtson-Crick complemented plindromes: tttcggt. Remrk. Pttern mtching with ck-references is intrctle. 6

15 Context Summry of pttern-mtching lgorithms Astrct mchines, lnguges, nd nondeterminism. sis of the theory of computtion intensively studied since the 9s sis of progrmming lnguges Compiler. A progrm tht trnsltes progrm to mchine code. KMP string DFA. grep RE NFA. jvc Jv lnguge Jv yte code. Progrmmer. Implement exct pttern mtching vi DFA simultion (KMP). Implement RE pttern mtching vi NFA simultion (grep). Theoreticin. RE is compct description of set of strings. NFA is n strct mchine equivlent in power to RE. DFAs nd REs hve limittions. You. Prcticl ppliction of core CS principles. pttern prser compiler output simultor KMP grep Jv string RE progrm unnecessry check if legl check if legl DFA NFA yte code DFA simultor NFA simultor JVM Exmple of essentil prdigm in computer science. Build intermedite strctions. Pick the right ones! Solve importnt prcticl prolems. 7 8