COS 333: Advanced Programming Techniques

Transcription

1 COS 333: Advnced Progrmming Techniques Brin Kernighn CS Building (ut emil is lwys etter) TA's: Junwen Li, CS 217, Yong CS 103C, Tody course overview dministrtive stuff regulr expressions nd grep Check out the course we pge notes, redings nd ssignments will e posted there Assignment 1 is posted Do the survey if you hven't lredy Themes lnguges C, C++, Jv, AWK, Perl, Visul Bsic,... progrmmle tools, ppliction-specific lnguges tools for progrmming nd progrmmers how to use them (voctionl trining) where did they come from nd why how they hve evolved, mutted, decyed how they work inside how to uild your own uilding progrms design, interfces reuse, theft, prototyping, components progrms tht write progrms portility, stndrds, style deugging, testing performnce ssessment nd improvement tricks of the trde trdeoffs, compromises, engineering versus science history nd culture of progrmming 1

2 (Very) Tenttive Outline Fe 3 Fe 10 Fe 17 Fe 24 Mr 2 Mr 9 Mr 15 regulr expressions; grep scripting lnguges: Awk & Perl more scripting: Perl, PHP(?), CGI Jv; oject-oriented progrmming Jv; networking; dtses; project user interfces, Swing (spring rek) Mr 23 C++ Mr 30 C++, Stndrd Templte Lirry Apr 6 Visul Bsic; COM, components Apr 13 XML, we services;.net, C#? Apr 20 Tcl/Tk?, lnguge tools? Apr 27? My 5-6 project presenttions Some Mechnics prerequisites C, Unix (COS 217) 5 progrmming ssignments in first hlf posted on course we pge dedlines mtter group project in second hlf groups of 3-4; strt identifying potentil temmtes detils in few weeks dedlines mtter monitor the we pge redings for most weeks notes generlly posted hed of time clss ttendnce nd prticiption <=> no midterm or finl spordic unnnounced short quizzes re possile 2

3 Regulrexpressions nd grep regulr expressions nottion mechniztion pervsive in Unix tools not in most generl-purpose lnguges though common in scripting lnguges nd (some) editors sic implementtion is remrkly simple efficient implementtion requires theory nd prctice grepis the prototypicl tool people used to write progrms for serching (or did it y hnd) tools ecme importnt tools re not s much in fshion tody Grepregulr expressions c ny chrcter mtches itself, except for metchrcters. [ ] ^ $ * + \ r 1 r 2 mtches r 1 followed y r 2. mtches ny single chrcter [...] mtches one of the chrcters in set... set like -z or 0-9 includes ny chrcter in the rnge [^...] mtches one of the chrcters not in set set like -z or 0-9 includes ny chr in the rnge ^ mtches eginning of line when ^ egins pttern no specil mening elsewhere in pttern $ mtches end of line when $ ends pttern no specil mening elsewhere in pttern * ny regulr expression followed y * mtches zero or more instnces \c mtches c unless c is ( ) or digit \(...\) tgged regulr expression tht mtches... the mtched strings re ville s \1, \2, etc. 3

4 Exmples of mtching thing ^thing thing$ ^thing$ thing nywhere in string thing t eginning of string thing t end of string string tht contins only thing ^$ empty string. non-empty, i.e., t lest 1 chr ^ mtches ny string, even empty thing.$ thing\.$ \\thing\\ thing plus ny chr t end of string thing. t end of string \thing\ nywhere in string [tt]hing thing or Thing nywhere in string thing[0-9] thing followed y one digit thing[^0-9] thing followed y non-digit thing[0-9][^0-9] thing followed y digit, then non-digit thing1.*thing2 thing1 then ny text then thing2 ^thing1.*thing2$ thing1 t eginning nd thing2 t end egrep: fncier regulr expressions r+ one or more occurrences of r r? zero or one occurrences of r r 1 r 2 r 1 or r 2 (r) r (grouping) ([0-9]+\.?[0-9]* \.[0-9]+)([Ee][-+]?[0-9]+)? 4

5 Grmmr for egrepregulr exprs r: c. ^ $ [ccc] [^ccc] r* r+ r? r 1 r 2 r 1 r 2 (r) Precedence: * +? re higher thn conctention which is higher thn ([0-9]+\.?[0-9]* \.[0-9]+)([Ee][-+]?[0-9]+)? The grepfmily grep sic mtching egrep fncier regulr expressions trdes compile time nd spce for run time fgrep prllel serch for mny fixed strings grep "pproximte" grep: serch with errors permitted reltives tht use similr regulr expressions ed originl unixeditor sed strem editor vi, emcs, sm,... editors lex lexicl nlyzer genertor wk, perl, tcl, python, scripting lnguges Jv, C#... lirries in minstrem lnguges simpler vrints filenme "wild crds" in Unix nd other shells "LIKE" opertor in Visul Bsic, SQL, etc. 5

6 Bsic grep lgorithm while (get line) if mtch(regexpr, line) print line (perhps) compile regexpr into n internl representtion suitle for efficient mtching mtch() slides the regexpr long the input line, looking for mtch t ech point regexpr line Grep (TPOP, p 226) /* grep: serch for regexp in file */ int grep(chr *regexp, FILE *f, chr *nme) { int n, nmtch; chr uf[bufsiz]; nmtch = 0; while (fgets(uf, sizeof uf, f)!= NULL) { n = strlen(uf); if (n > 0 && uf[n-1] == '\n') uf[n-1] = '\0'; if (mtch(regexp, uf)) { nmtch++; if (nme!= NULL) printf("%s:", nme); printf("%s\n", uf); return nmtch; 6

7 Mtch nywhere on line look for mtch t ech position of text in turn /* mtch: serch for regexp nywhere in text */ int mtch(chr *regexp, chr *text) { if (regexp[0] == '^') return mtchhere(regexp+1, text); do { /* must look even if string is empty */ if (mtchhere(regexp, text)) return 1; while (*text++!= '\0'); return 0; Mtch strting t current position /* mtchhere: serch for regexp t eginning of text */ int mtchhere(chr *regexp, chr *text) { if (regexp[0] == '\0') return 1; if (regexp[1] == '*') return mtchstr(regexp[0], regexp+2, text); if (regexp[0] == '$' && regexp[1] == '\0') return *text == '\0'; if (*text!='\0' && (regexp[0]=='.' regexp[0]==*text)) return mtchhere(regexp+1, text+1); return 0; follow the esy cse first: no metchrcters note tht this is recursive mximum depth: one level for ech regexprchrcter tht mtches 7

8 Mtching * (repetitions) mtchstr() clled to mtch c*... mtches if rest of regexpr mtches rest of input null mtches require test t the ottom /* mtchstr: serch for c* regexp t eginning of text */ int mtchstr(intc, chr * regexp, chr *text) { do { /* * mtches zero or more instnces */ if (mtchhere(regexp, text)) return 1; while (*text!= '\0' && (*text++ == c c == '.')); return 0; finds the leftmost shortest mtch just right for pttern mtching in grep NOT usully wht we wnt in text editor null mtches re surprising nd rrely desired Profiling: where does the time go count numer of times ech line is executed mesure how long ech function tkes plus lots of other informtion $ lcc -p grep.c $.out x../i >foo $ prof %Time Seconds Cumsecs #Clls msec/cll Nme mtchhere _mcount mtch _red _memccpy filuf fgets strlen min grep setprognm printf... $ wc../i /i $.out x../i wc _mcount is profiling overhed note consistent counts 8

9 Sttement frequency counts $ lcc - grep.c;.out x../i >/dev/null; print int mtch(chr *regexp, chr *text) <31102>{ if (<31102>regexp[0] == '^') return <0>mtchhere(regexp+1, text); do < >{ /* must look even if string is empty */ if (< >mtchhere(regexp, text)) return <1326>1; < > while (< >*text++!= '\0'); return <29776>0; int mtchhere(chr *regexp, chr *text) < >{ if (< >regexp[0] == '\0') return <1326>1; if (< >regexp[1] == '*') return <0>mtchstr(regexp[0], regexp+2, text); if (< >regexp[0] == '$' && <0>regexp[1] == '\0') return <0>*text == '\0'; if (< >*text!='\0' && < >(regexp[0]=='.' < >regexp[0]==*text)) return <1326>mtchhere(regexp+1, text+1); return < >0; note conservtion lws How to mke grep fster use optimiztion (cc -O) chnge compilers (lcc, gcc, vc++) code tuning e.g., mtch clls mtchhere mny times even though most of them must necessrily fil ecuse the trget string doesn't contin the first chrcter of the pttern lgorithm chnges 9

10 Code tuning vrint checks whether trget contins first chrcter of pttern efore clling mtchhere unless it is x* /* mtch: serch for regexp nywhere in text */ int mtch(chr *regexp, chr *text) { chr *p; if (regexp[0] == '^') return mtchhere(regexp+1, text); if (regexp[0]!= '\0' && regexp[0]!= '.' && regexp[1]!= '*') if ((p=strchr(text, regexp[0])) == NULL) return 0; do { /* must look even if string is empty */ if (mtchhere(regexp, p)) return 1; while (*p++!= '\0'); return 0; is this fster? Sttement frequencies fter chnge /* mtch: serch for regexp nywhere in text */ int mtch(chr *regexp, chr *text) <31102>{ chr *p; if (<31102>regexp[0] == '^') return <0>mtchhere(regexp+1, text); if (<31102>regexp[0]!= '\0' && <31102>regexp[0]!= '.' && <31102>regexp[1]!= '*') if (<31102>(p=strchr(text, regexp[0])) == NULL) return <29776>0; do <1326>{ /* must look even if string is empty */ if (<1326>mtchhere(regexp, p)) return <1326>1; <0> while (<0>*p++!= '\0'); return <0>0; /* mtchhere: serch for regexp t eginning of text */ int mtchhere(chr *regexp, chr *text) <2652>{ if (<2652>regexp[0] == '\0') return <1326>1; if (<1326>regexp[1] == '*') return <0>mtchstr(regexp[0], regexp+2, text); if (<1326>regexp[0] == '$' && <0>regexp[1] == '\0') return <0>*text == '\0'; if (<1326>*text!='\0' && <1326>(regexp[0]=='.' <1326>regexp[0]==*text)) return <1326>mtchhere(regexp+1, text+1); return <0>0; 10

11 Simple greplgorithm est for short simple ptterns e.g.,grep foo *.[ch] most use is like this reflects use in text editor for smll mchine limittions tries the pttern t ech possile strting point e.g., look for in. potentilly O(mn) for pttern of length m complicted ptterns (.*.*.*) require ckup potentilly exponentil cn't do some things, like lterntion (OR) this leds to extensions nd new lgorithms egrep complicted ptterns, lterntion fgrep lots of simple ptterns in prllel oyer-moore long simple ptterns grep pproximte mtches Finite stte mchines/finite utomt finite stte mchine set of sttes n lphet (e.g., scii) trnsition rules: current stte & input chr -> new stte strt stte set of finl "ccepting" sttes regulr expressions re equivlent to finite stte mchines cn go from one to the other mechniclly *c n n, if n < 4 c cn't count: cn't hndle ritrry n in fixed numer of sttes cn't do plindromes: no memory 11

12 Non-deterministic finite utomt (NDFA) RE:.*.* FSM: input: x x stte fter: ? diff seq: if the mchine could guess right every time, it would mtch properly voids "cking up", decides out ech chrcter the first time it's seen NDFA mtches n input if there is ny possile pth from strt stte to finl stte. it rejects/does not mtch if there is no pth from the strt stte to finl stte. how do we mke mchine tht's lwys lucky? mke deterministic finite utomton tht simultes the NDFA Egrep: regexpr -> NDFA -> DFA Exmple: ( ) NDFA: Convert to DFA y inventing sttes tht represent sets of sttes of the NDFA: 0 1,3,5 4,6 Recognition time is O(n) Construction time could e O(2 m ) ecuse there re 2 m susets of the sttes newer versions construct sttes s needed: lzy evlution

13 Importnt ides from regexprs & grep tools: let the mchine do the work good pckging mtters nottion: mkes it esy to sy wht to do my orgnize or define implementtion hcking cn mke progrm fster, sometimes, usully t the price of more complexity etter lgorithm cn mke progrm go lot fster don't worry out performnce if it doesn't mtter (nd it often doesn't) when it does, use the right lgorithm use the compiler's optimiztion code tune, s lst resort 13