Building a Scanner, Part I

Size: px

Start display at page:

Download "Building a Scanner, Part I"

Angelina Elliott
5 years ago
Views:

1 COMP 506 Ric Univrsity Spring 2018 Building a Scannr, Part I sourc cod IR Front End Optimizr Back End IR targt cod Copyright 2018, Kith D. Coopr & Linda Torczon, all rights rsrvd. Studnts nrolld in Comp 506 at Ric Univrsity hav xplicit prmission to mak copis of ths matrials for thir prsonal us. Faculty from othr ducational institutions may us ths matrials for nonprofit ducational purposs, providd this copyright notic is prsrvd Chaptr 2 in EaC3

2 Th Compilr s Front End sourc cod IR Front End Optimizr Back End IR targt cod stram of charactrs stram of tokns Scannr microsyntax Parsr syntax Smantic Elaboration IR annotations Front End Scannr and Parsr collaborat to chck th syntax of th input program. Scannr maps stram of charactrs into words (words ar th fundamntal unit of syntax) Scannr producs a stram of tokns for th parsr Tokn is <part of spch, word> Part of spch is a unit in th grammar Parsr maps stram of words into a sntnc in a grammatical modl of th input languag COMP 506, Spring

3 Th Compilr s Front End sourc cod IR Front End Optimizr Back End IR targt cod stram of charactrs stram of tokns Scannr microsyntax Parsr syntax Smantic Elaboration IR annotations Front End Scannr looks at vry charactr Convrts stram of charactrs to stram of tokns, or classifid words: <part of spch, word> Efficincy & scalability mattr Scannr is only part of compilr that looks at vry charactr Parsr looks at vry tokn Dtrmins if th stram of tokns forms a sntnc in th sourc languag Fits tokns to som syntactic modl, or grammar, for th sourc languag COMP 506, Spring

4 Th Compilr s Front End sourc cod IR Front End Optimizr Back End IR targt cod stram of charactrs stram of tokns Scannr microsyntax Parsr syntax Smantic Elaboration IR annotations Front End Why sparat scanning & parsing? Primary rational is fficincy Scannr idntifis & classifis words by splling (microsyntax) Parsr constructs drivations in a grammar (syntax) Parsing is hardr than scanning Modrn viw (lss widly hld) Scannr-lss parsrs ar gaining popularity, bcaus thy liminat on mor st of tools Mayb w can afford th ovrhad A littl mor involvd (SGLR parsrs) COMP 506, Spring

5 Implmntation Stratgis sourc cod IR Front End Optimizr Back End IR targt cod stram of charactrs stram of tokns Scannr microsyntax Parsr syntax Smantic Elaboration IR annotations Front End Rlvant Qustion: How can w automat th construction of scannrs & parsrs? Scannr Spcify syntax with rgular xprssions (REs) Construct finit automaton & scannr from th RE Parsr Spcify syntax with contxt-fr grammars (CFGs) Construct push-down automaton & parsr from th CFG COMP 506, Spring

6 Why Can t W Just Us A Rgx Library Rgular xprssions (calld REs, or rgxs, or rgx pattrns) ar ssntially a tiny, highly spcializd programming languag mbddd insid Python and mad availabl through th RE modul. Rgular xprssion pattrns ar compild into a sris of bytcods which ar thn xcutd by a matching ngin writtn in C. For advancd us, it may b ncssary to pay carful attntion to how th ngin will xcut a givn RE, and writ th RE in a crtain way in ordr to produc bytcod that runs fastr. Optimization isn t covrd in this documnt, bcaus it rquirs that you hav a good undrstanding of th matching ngin s intrnals. Th rgular xprssion languag is rlativly small and rstrictd, so not all possibl string procssing tasks can b don using rgular xprssions. Thr ar also tasks that can b don with rgular xprssions, but th xprssions turn out to b vry complicatd. In ths cass, you may b bttr off writing Python cod to do th procssing; whil Python cod will b slowr than an laborat rgular xprssion, it will also probably b mor undrstandabl. From Python documntation, mphasis addd Scannr gnrators us th sam principls as rgx libraris Automaton costs O(1) pr rcognizd charactr, indpndnt of th xprssion Scannr gnrators hav a clan intrfac to gnratd parsrs (.g., flx bison) Rad 2.4 in EaC2 if you want th dtails W will skim th surfac lightly in lctur COMP 506, Spring

7 Why Can t W Just Us A Rgx Library Rgular xprssions (calld REs, or rgxs, or rgx pattrns) ar ssntially a tiny, highly spcializd programming languag mbddd insid Python and mad availabl through th r modul. Rgular xprssion pattrns ar compild into a sris of bytcods which ar thn xcutd by a matching ngin writtn in C. For advancd us, it may b ncssary to pay carful attntion to how th ngin will xcut a givn RE, and writ th RE in a crtain way in ordr to produc bytcod that runs fastr. Optimization isn t covrd in this documnt, bcaus it rquirs that you hav a good undrstanding of th matching ngin s intrnals. Th rgular xprssion languag is rlativly small and rstrictd, so not all possibl string procssing tasks can b don using rgular xprssions. Thr ar also tasks that can b don with rgular xprssions, but th xprssions turn out to b vry complicatd. In ths cass, you may b bttr off writing Python cod to do th procssing; whil Python cod will b slowr than an laborat rgular xprssion, it will also probably b mor undrstandabl. From Python documntation, mphasis addd W will com back to this point nxt lctur Scannr gnrators us th sam principls as rgx libraris Automaton costs O(1) pr rcognizd charactr, indpndnt of th xprssion Scannr gnrators hav a clan intrfac to gnratd parsrs (.g., flx bison) Rad 2.4 in EaC2 if you want th dtails W will skim th surfac lightly in lctur COMP 506, Spring

8 Exampl Suppos that w nd to rcogniz th kyword not : How would w look for it? All of your training, to dat, tlls you to call som fancy routin, such as fscanf() in C, th Scannr class in Java, or a rgx library (Python, Java, C++) COMP 506 is a class about implmnting languags, not just about using thm, so w will look a littl dpr In a compilr, w would build a rcognizr How about som cod? Th splling is n followd by o followd by t Th cod, shown to th right, is as simpl as th dscription Cost is O(1) pr charactr Structur allows for prcis rror mssags c nxt charactr If c = n thn { c nxt charactr if c = o thn { c nxt charactr if c = t thn rturn <NOT, not > ls rport rror } ls rport rror } ls rport rror COMP 506, Spring

s 2 t s 3 Excution bgins in th start stat, s 0 On an input of n, it follows th dg labld n, and so on, Transition to an rror stat, s, on any unxpctd

9 Automata W can rprsnt this cod as an automaton c nxt charactr If c = n thn { c nxt charactr if c = o thn { c nxt charactr if c = t thn rturn <NOT, not > ls rport rror } ls rport rror } ls rport rror s 0 s n Any charactr s 1 Transition Diagram for not o O(1) pr input charactr, if branchs ar O(1) s 2 t s 3 Excution bgins in th start stat, s 0 On an input of n, it follows th dg labld n, and so on, Transition to an rror stat, s, on any unxpctd charactr Halts whn out of input Stats drawn with doubl lins indicat succss (a final stat) Cost is O(1) pr charactr COMP 506, Spring 2018 S 2.2 in EaC2 9

10 Automata To rcogniz a largr st of kywords is (rlativly) asy Say, for xampl, th st of words { if, int, not, thn, to, writ } s 2 if Any charactr s 0 i n s 1 s 5 f n o s 3 s 6 t t s 4 s 7 int not s ERROR Transitions to s ar implicit from vry stat t w s 8 h o s 9 s 10 s 11 s 12 to n thn on-to-on map from final stats to words (mayb parts of spch) s 13 r s 14 i s 15 t s 16 s 17 writ Cost COMP is 506, still Spring O(1) pr 2018charactr 10

Spcifying An Automaton Spcifying a collction of words is simpl W list th words Each word s splling is uniqu and concis W can sparat thm with th symbol, rad as or Th spcification if int not thn to

11 Spcifying An Automaton Spcifying a collction of words is simpl W list th words Each word s splling is uniqu and concis W can sparat thm with th symbol, rad as or Th spcification if int not thn to writ is a simpl rgular xprssion for th languag accptd by our automaton Evry rgular xprssion corrsponds to an automaton W can construct, automatically, an automaton from th RE W can construct, automatically, an RE from any automaton W can convrt an automaton asily and dirctly into cod Th automaton and th cod both hav O(1) cost pr input charactr So, an RE spcification lads dirctly to an fficint scannr Thr is som subtlty hr. W introducd notation for concatnation and choic (or altrnation). Concatnation: ab is a followd by b Altrnation: a b is ithr a or b COMP 506, Spring

12 Unsignd Intgrs Notic th cyclic transition dgs in th automata In th xampl, ach word had on splling. What about a singl part of spch with many spllings? Considr spcifying an unsignd intgr An unsignd intgr is any string of digits from th st [0...9] W might want to spcify that no lading zros ar allowd Is 0001 allowd? Or 00? Th Automata 0 9 s 0 s 2 With Lading Zros 0 9 unsignd intgr 0 s 0 s s 2 Without Lading Zros unsignd intgr unsignd intgr How do w writ th corrsponding RE? Any charactr s ERROR COMP 0 9 mans 506, Spring any charactr 2018 from 0 to 9, inclusiv; somtims w will writ it as

Closur x + rprsnts on or mor instancs of x 1 9 0 9 [0-9] + for lading zros; 0 [1-9] [0-9] * for without lading

13 Unsignd Intgrs Th Automata 0 9 s 0 s 2 With Lading Zros 0 9 unsignd intgr 0 s 0 s 1 Th Rgular Exprssions W nd a notation to rprsnt that cyclic dg in th automaton x Th Kln Closur x * rprsnts zro or mor instancs of x Th Positiv Closur x + rprsnts on or mor instancs of x [0-9] + for lading zros; 0 [1-9] [0-9] * for without lading zros Any charactr ERROR COMP 506, Spring s 2 Without Lading Zros unsignd intgr unsignd intgr s 2 x s 0 s 2 x s

14 Unsignd Intgrs Again, th RE corrsponds dirctly to an automaton and an implmntation 0 s 0 s 1 s 2 unsignd intgr unsignd intgr c nxt charactr n 0 if c = 0 thn rturn <CONSTANT,n> ls if ( 1 c 9 ) thn { n atoi(c) c nxt charactr whil ( 0 c 9 ) { t atoi(c) n n * 10 + t } rturn <CONSTANT,n> } ls rport rror Th automaton and th cod can b gnratd automatically Th dtails of th constructions ar givn in 2.4 of EaC2 or th 412 nots COMP Cost is 506, still Spring O(1) 2018 pr charactr 14

15 Rgular Exprssions W nd a mor formal dfinition for a rgular xprssion Rgular Exprssions ovr an Alphabt Σ If x Σ, thn x is an RE dnoting th st { x } or th languag L = { x } If x and y ar REs thn xy is an RE dnoting L(x)L(y) = { pq p L(x) and q L(y) } x y is an RE dnoting L(x) L(y) x * is an RE dnoting L(x) * = 0 k < L(x) k St of all strings that ar zro or mor concatnations of x x + is an RE dnoting L(x) + = 1 k < L(x) k St of all strings that ar on or mor concatnations of x ε is an RE dnoting th mpty st (Kln Closur) (Positiv Closur) Many RE-basd systms support mor complx oprators. Thos oprators ar built on top of altrnation, concatnation, and closur plus, prhaps logical complmnt or ngation. Complmnt is asy and fficint; rvrs th final and non-final stats. COMP 506, Spring

16 Rgular Exprssions How do ths oprators hlp? Rgular Exprssions ovr an Alphabt Σ If x is in Σ, thn x is an RE dnoting th st { x } or th languag L = { x } Th splling of any lttr in th alphabt is an RE If x and y ar REs thn xy is an RE dnoting L(x)L(y) = { pq p L(x) and q L(y) } If w concatnat lttrs, th rsult is an RE (splling of words) x y is an RE dnoting L(x) L(y) Any finit list of words can b writtn as an RE, ( w 0 w 1 w 2 w n ) x * is an RE dnoting L(x) * = 0 k < L(x) k x + is an RE dnoting L(x) + = 1 k < L(x) k W can us closur to writ finit dscriptions of infinit, but countabl, sts ε is an RE dnoting th mpty st In practic, th mpty string is oftn usful COMP 506, Spring

17 Rgular Exprssions Lt th notation [0-9] b shorthand for ( ) Exampls Positiv intgr [0-9] [0-9] * or [0-9] + No lading zros 0 [1-9] [0-9] * Algol-styl Idntifir ([a-z] [A-Z]) ([a-z] [A-Z] [0-9]) * Dcimal numbr 0 [1-9] [0-9] *. [0-9] * Ral numbr ((0 [1-9] [0-9] * ) (0 [1-9] [0-9] *. [0-9] * ) E [0-9] [0-9] * Each of ths REs corrsponds to an automaton and an implmntation. COMP 506, Spring

From RE to Scannr W can us rsults from automata thory to construct scannrs dirctly from REs Thr ar svral ways to prform this construction Classic approach is a two-stp mthod. 1.

18 From RE to Scannr W can us rsults from automata thory to construct scannrs dirctly from REs Thr ar svral ways to prform this construction Classic approach is a two-stp mthod. 1. Build automata for ach pic of th RE using a simpl tmplat-drivn mthod Build a spcific variation on an automaton that has transitions on ε and nondtrministic choic (multipl transitions from a stat on th sam symbol) This construction is calld Thompson s construction 2. Convrt th nwly built automaton into a dtrministic automaton Dtrministic automaton has no ε-transitions and all choics ar singl-valud This construction is calld th subst construction Givn th dtrministic automaton, w can run a minimization algorithm on it to rduc th numbr of stats An NFA A DFA Minimization is a spac optimization. Both th original automaton and th minimal on tak O(1) tim pr charactr COMP 506, Spring ?

19 Kn Thompson, CACM, 1968 Thompson s Construction (in 2 slids) (s 2.4.2) Th Ky Ida For ach RE symbols and oprator, w hav a small tmplat Build thm, in prcdnc ordr, and join thm with ε-transitions a S 0 S 1 a S 0 S 1 b S 3 S 4 NFA for a NFA for ab S 0 a S 1 S 2 b S 3 S 4 S 5 S 0 S 1 a S 3 S 4 NFA for a b NFA for a * COMP Prcdnc 506, Spring REs: 2018Parnthss, thn closur, thn concatnation, thn altrnation 19

20 Thompson s Construction (in 2 slids) Lt s build an NFA for a ( b c ) * 1. a, b, & c 2. b c a S 0 S 1 c S 0 S 1 b S 0 S 1 b S 1 S 2 S 0 S 5 c S 3 S 4 3. ( b c ) * b S 2 S 3 S 0 S 1 S 6 S 7 c S 4 S 5 4. a ( b c ) * a S 0 S 1 b S 4 S 5 S 2 S 3 S 8 S 9 S 6 c S 7 COMP Prcdnc 506, Spring REs: 2018Parnthss, thn closur, thn concatnation, thn altrnation 20

21 Subst Construction (s 2.4.3) Th Concpt Build a simplr automaton (no ε-transitions, no multi-valud transitions) that simulats th bhavior of th mor complx automaton Each stat in th nw automaton rprsnts a st of stats in th original NFA a n 0 n 1 a ( b c )* b n 4 n 5 n n 3 n 2 8 n 9 c n 6 n 7 b DFA NFA DFA b a d 0 d 1 c b d 2 c d 0 n 0 d 1 n 1 n 2 n 3 n 4 n 6 n 9 d 2 n 5 n 8 n 9 n 3 n 4 n 6 d 3 d 3 n 7 n 8 n 9 n 3 n 4 n 6 c COMP 506, Spring 2018 Mapping btwn NFA and DFA stats 21

rplacing multipl quivalnt stats with a singl on Minimization rducs th numbr of stats, but dos

22 Minimization (s & 2.6.3) DFA Minimization algorithms work by discovring stats that ar quivalnt in thir contxts and rplacing multipl quivalnt stats with a singl on Minimization rducs th numbr of stats, but dos not chang th costs b b a d 0 d 1 c b d 2 c minimization a s 0 s 1 b c d 3 c Minimal DFA Stat s 0 s 1 Original DFA Stats d 0 d 1, d 2, d 3 COMP 506, Spring

23 Implmnting an Automaton (s 2.5) A common stratgy is to simulat th DFA s xcution Sklton parsr + a tabl that ncods th automaton stat s 0 char NxtChar( ) whil (char ¹ EOF) { stat d[stat,char] char NxtChar( ) } if (stat is a final stat) thn rport succss ls rport an rror δ a b c s 0 s 1 s s s 1 s s 1 s 1 Transition tabl for our minimal DFA Simpl Sklton Scannr Th scannr gnrator constructs th tabl Th sklton parsr dos not chang (much) COMP 506, Spring

24 Implmnting an Automaton (s 2.5) A common stratgy is to simulat th DFA s xcution Sklton parsr + a tabl that ncods th automaton stat s 0 char NxtChar( ) whil (char ¹ EOF) { stat d[stat,char] char NxtChar( ) } if (stat is a final stat) thn rport succss ls rport an rror Simpl Sklton Scannr Th scannr gnrator constructs th tabl Th sklton scannr dos not chang (much) Notic that th sklton scannr uss a whil loop and an array accss to rplac th if-thn-ls from our arlir xampls. This structur δ translats a b into an c nd-ofloop jump s 0 and sa 1 branch s for th s tst an asily prdictabl branch. (1 cycl vs. 4 or 5?) s 1 s s 1 s 1 Th branchs in th if-thn-ls ar hard to prdict, making that cod much slowr than th Transition sklton tabl scannr. for our minimal DFA Studnts ar oftn surprisd to discovr how much tim scanning and parsing taks. COMP 506, Spring

25 Automatic Scannr Construction sourc cod Sklton Scannr Tabls <part of spch, word> pairs Knowldg ncodd in tabls to driv sklton spcifications (as REs) Scannr Gnrator Scannr Gnrator Taks in a spcification writtn as a collction of rgular xprssions Combins thm into on RE using altrnation ( ) Builds th minimal automaton ( 2.4, 2.6.2) Emits th tabls to driv a sklton scannr ( 2.5) COMP 506, Spring

26 Automatic Scannr Construction sourc cod Scannr <part of spch, word> pairs Knowldg mbddd in gnratd program txt spcifications (as REs) Scannr Gnrator Scannr Gnrator As altrnativ, th gnrator can produc cod rathr than tabls Dirct-codd scannrs ar ugly, but oftn fastr than tabl-drivn scannrs Othr than spd, th two ar quivalnt Us th scannr gnrator availabl to you COMP 506, Spring

Automatic Gnration of Scannrs and Parsrs Thr tim frams At dsign tim, th compilr writr writs

spcifications to cod and compil that cod to produc th actual compilr At compil tim, th usr invoks th

stram of charactrs Parsr Gnrator Scannr Gnrator Scannr Parsr Smantic Elaboration Compilr-build tim

27 Automatic Gnration of Scannrs and Parsrs Thr tim frams At dsign tim, th compilr writr writs spcifications for th microsyntax (splling) and th syntax (grammar) At build tim, th tools convrt spcifications to cod and compil that cod to produc th actual compilr At compil tim, th usr invoks th compilr to translat an application into an xcutabl form spcifications as CFGs spcifications as REs stram of charactrs Parsr Gnrator Scannr Gnrator Scannr Parsr Smantic Elaboration Compilr-build tim Compil tim IR annotations Front End In COMP lab 1, 506, you Spring will us 2018 a scannr gnrator & a parsr gnrator 27

28 Automatic Scannr Construction sourc cod 2. Whn th compilr runs, it uss th gnratd scannr to convrt sourc cod into a stram of tokns. 1. W writ REs and gnrat a scannr whn w build th compilr spcifications writtn as rgular xprssions Scannr Scannr Gnrator stram of <part of spch,word> pairs compil tim scannr gnration tim Scannr and parsr gnrators oprat at compilr-build tim Th compilr writr crats a input fil that dscribs th microsyntax (splling) of th words in th sourc languag * Whn th compilr is built, th scannr gnrator crats th scannr Builds an automaton, convrts it to a tabl or cod * Whn th compilr runs, it invoks th gnratd scannr to tokniz input Scannr rturns a stram of <part of spch,word> pairs to th parsr COMP 506, Spring

29 Automatic Scannr Gnration Th Point (for Scannrs) Th tchnology lts us writ a st of REs and gnrat a good scannr Scannr gnrator builds th NFA, th DFA, th minimal DFA, and thn writs out a tabl-drivn or dirct codd scannr Th tools rliably produc fast, robust scannrs For most modrn languag faturs, this works and works wll You should think twic bfor introducing a languag fatur that dfats a DFA-basd scannr W hav sn som ovr tim; thy hav not bn particularly usful Insignificant blanks in FORTRAN, non-rsrvd kywords in PL/I Of cours, not vrything fits into th rgular languag framwork which is why w nd parsrs COMP 506, Spring

30 Altrnativ (arly) nding COMP 506, Spring

31 Summary Automating Scannr Construction 1. Writ down th REs for th input languag and connct thm with 2. Build a big, simpl NFA 3. Build th DFA that simulats th NFA 4. Minimiz th numbr of stats in th DFA 5. Gnrat an implmntation from th minimal DFA Scannr Gnrators lx, flx, jflx, and such all work along ths lins Algorithms ar wll-known and wll-undrstood Ky issus ar: finding longst match rathr than first match, and nginring th intrfac to th parsr You could build a scannr gnrator in a wknd COMP Find th 506, longst Spring 2018 match is whr O(1) pr charactr can turn into O(n). Mor nxt lctur. 31

Principles of Programming Languages Topic: Formal Languages II

Principles of Programming Languages Topic: Formal Languages II Principls of Programming Languags Topic: Formal Languags II CS 34,LS, LTM, BR: Formal Languags II Rviw A grammar can b ambiguous i.. mor than on pars tr for sam string of trminals in a PL w want to bas