CSCI 3130: Formal Languages and Automata Theory Lecture 12 The Chinese University of Hong Kong, Fall 2011

Transcription

1 CSCI 3130: Forml Lnguges nd utomt Theory Lecture 12 The Chinese University of Hong Kong, Fll 2011 ndrej Bogdnov In progrmming lnguges, uilding prse trees is significnt tsk ecuse prse trees tell us the mening of computer progrms. typicl CFG for progrmming lnguge like jv could hve out nonterminl symols nd terminls. The terminls of progrmming lnguge, clled tokens, re things like oolen nd rithmetic opertors (e.g., =, +, &, ==), prentheses (e.g., (, )), specil symols (e.g.,.,,, ;), specil keywords (e.g., if, return, int, clss), s well s plceholders for identifiers (ID, which stnds for vrile, procedure, nd clss nmes) nd literls (e.g., INT_LIT which stnds for n integer literl like 5). When the jv compiler is invoked on progrm, it first runs lexicl nlyzer whose tsk is to identify the tokens. fter running it, line of code like would e converted to if ( n == 0 ) { return x; } if ( ID == INT_LIT ) { return ID ; } Once we hve the progrm in this form, the jv prser cretes prse tree of the progrm. Here is the (simplified) tree tht descries this line of code: Sttement if PrExpression Sttement ( Expression ) Block Expression ExpressionRest { BlockSttements } Primry Infixop Expression BlockSttement Identifier == Primry Sttement ID Literl return Expression ; INT_LIT Primry Identifier ID 1

2 2 How does the jv prser uild this tree? One possiility is to use the Cocke-Younger-Ksmi lgorithm. However, this lgorithm is too slow even for moderte-size jv progrm. The running time of the CYK lgorithm is on the order of gn 3, where g is the numer of symols in the CFG nd n is the length of the string to e prsed (i.e., the progrm). If the progrm hs 1000 tokens resonle size the CYK prser could well tke over week to complete its tsk. This is totlly indequte for computer progrms, which need to e prsed much fster. However, the CYK prser is the fstest generl-purpose prsing lgorithm the we know of. Yet we know from experience tht when we run the jv compiler, our progrms get prsed within few milliseconds. How is this possile? The nswer is tht the CFG of the jv progrmming lnguge (nd other lnguges) re designed in wy tht llows extremely quick prsing. The jv CFG llows progrms to e prsed in liner time. In fct, the prser only mkes one left-to-right sweep of the input, t the end of which the prse tree is completely uilt. The jv CFG is specil type of CFG clled LR(1). 1 LR(1) grmmrs do not cover ll possile context-free lnguges, ut they re rich enough to descrie the semntics of most progrmming lnguges, while llowing for extremely fst prsing. Before we descrie how LR(1) grmmrs work, we strt with n even simpler kind of grmmr clled LR(0). While LR(0) grmmrs re much less useful in prctice, they will llow us to introduce the min ides ehind LR(1) prsing while voiding some confusing complictions. 1 LR(0) prsing t this point, you my expect definition sying n LR(0) grmmr is [something]. However unlike in previous lectures, tody we will do things it differently. Insted of giving (syntctic) definition of LR(0) grmmrs, we will egin y descriing prsing lgorithm. Unlike the CYK prsing lgorithm, the lgorithm we show tody will not work for every CFG. The CFGs for which the lgorithm hppens to work correctly re the LR(0) grmmrs. But given grmmr G, how do we know if the prsing lgorithm will work or not? Unlike the CYK lgorithm, which is single (generl-purpose) prsing lgorithm which works for ny CFG, LR(0) prsing lgorithms re specificlly tilored for the CFG we wnt to prse. Wht we will descrie is process tht, given ny CFG G, ttempts to uild prsing lgorithm tilored to G. t some point in the construction of this lgorithm, we my oserve tht this lgorithm hs certin kind of ug ; if the ug does not occur, we will sy tht G is n LR(0) grmmr. This process, which tkes grmmr G nd outputs fst lgorithm for prsing strings in G (or outputs n error if the lgorithm hs ug) is clled prser genertor. The prser genertor cn itself e viewed s n lgorithm: It is n lgorithm which tkes s input the description of CFG G nd produces s output the code of n lgorithm for prsing strings in G. 1 LR(1) stnds for Left-to-right red, Rightmost derivtion, with 1 symol of lookhed.

3 3 CFG G prsergenertor LR(0) prser for G error if G is not LR(0) 2 n exmple The intuition ehind the LR(0) lgorithm is very nturl. To explin it, suppose you hve the following CFG G c nd you wnt to prse the string. How would you go out this tsk in systemtic wy? Imgine tht you re DF. You re stnding to the left of the input nd you hve not seen ny of the input symols yet, like this: Now you see tht the first symol coming t you is n. Looking t the productions of your CFG, you notice tht there re two possiilities: The top of the prse tree must e either or, ut certinly not toc. You wnt to keep oth of the first two possiilities in mind. So you consume the first, move one spot to the right nd find yourself in this sitution: You wnt to rememer tht t this stge, fter you hve seen the first symol, the derivtion you re looking t must either strt with or. You hve lredy consumed the first, so you must e in one of these two situtions: or. To keep trck of this, you set up registry where you store oth of these options s possiilities. Before moving on, you notice tht the first option on your registry sys tht wht is coming next is derivtion strting from. To prepre for this possile derivtion, you dd the following three items to your registry:,, c. t this point, the registry looks like this: c You now peek hed gin nd notice tht the next symol is n. This immeditely rules out the second nd fifth production from the registry. The first production is it tricky; the symol fter the is non-terminl, so wht should you do with it? Let s rememer (in seprte plce) tht this is the stte of the production fter reding the first, nd tke out this item from the registry too. Wht is left inside is,. fter moving pst the second,

4 4 the registry is updted to,. Since the first possiility gin indictes tht wht my e coming is derivtion for, we dd the items,, c nd get the following registry: c Peeking gin, we notice the next input symol is, mening tht the only vile item in our registry is. fter moving pst this, the registry is updted to single item: This mens tht we must hve just completed prt of the derivtion, nd we cn construct the corresponding portion of the prse tree: To continue from here, we must cktrck it. We rememer tht the su-tree we just constructed cme from the item which we put side fter we red the second in the input. So we must now e in the sitution. So we cler the registry nd put this item inside: Moving long, we notice the next (nd lst) input symol is, which corresponds exctly to wht is predicted y the only item in the registry. We red this lst nd updte the registry to. The unique item in the registry now tells us how to complete the prse tree: 3 Prsing with the help of DF Let us now go ck to the exmple we just did nd try to pin down the strtegy tht we used in producing the prse tree. The min ide ws to keep trck of ll possile items tht represent the current stte of the prse tree. Before we go on, let s introduce some nottion. s usul, will cpitl letters like, B, C to denote vriles nd lowercse letters like,, c for terminls. The letter X denotes symol tht is either vrile or terminl. We use greek letters α, β, γ for string consisting of vriles nd terminls.

5 5 Definition 1. n item of CFG G is string of the form α β, where αβ is production in G. t ech step of the process, we used registry to keep trck of ll possile vlid items, which we updted long the wy sed on wht we sw in the input. Let us try to formlize the rules we used for updting items. Initilly, the wht we hve in the registry re ll items of the form S α, where S is the strt vrile. We then updted the items ccording to the following rules: 1 If the registry does not contin ny item of the form X 1 X 2... X k : 2 Red the next input symol. 3 Replce every item of the form B α β with B α β. 4 Discrd every item of the form B α Xβ where X. 5 For every item of the form B α Cβ, dd ll items of the form C δ. 6 If the registry contins single item of the form X 1 X 2... X k : 7 Discrd this item. 8 For every item of the form B α β tht ws discrded k steps go, 9 dd the item B α β to the registry. 11 Build portion of the prse tree. 12 For every item of the form B α Cβ, dd ll items of the form C δ. 13 If there re no items left in the registry nd the whole input hs een trversed, ccept. n item of the form X 1 X 2... X k is clled complete item. n NF for the registry updtes It will e convenient to represent some of the ove steps using n NF N G. The purpose of this NF is it different from the NFs we sw in clss so fr: It will e used not merely to decide if its input is in G or not, ut lso help uild prse tree. The NF will serve s tool in constructing prser for G. The NF N G hs one stte for every item of G, plus strt stte q 0. Let us not worry out finl sttes for now. We will ttempt to represent the vlid items registry fter t steps y the collection of sttes tht the NF cn e in t the sme stge. Indeed, the rules for updting items in the registry look much like trnsitions of n NF. Let s try to stte them in NF lnguge: Initil setup: For every production of the form S α in G, where S is the strt vrile, put n -trnsition from q 0 to the stte S α in N G. NF updte rule 1 (line 3) For every terminl nd every rule of the form B αβ in G, put trnsition from B α β to B α β with lel. NF updte rule 2 (lines 5 nd 12) For every item of the form B α Cβ, dd n -trnsition from B α Cβ to C δ. NF updte rule 3 (lines 8-9) For every vrile nd every rule of the form B αβ in G, put trnsition from B α β to B α β with lel.

6 6 The lst updte rule does not quite mke sense, ecuse the vrile will never occur in the input. However, this vrile comes up implicitly t some stge of the prsing. To explin how, let us recll wht hppened t this stge in the exmple we just did: t this point, we hd to cktrck nd figure out which item the lst portion of the prse tree cme from. To do so, let us go ck to the stge we must hve een t just efore we uilt the portion of the prse tree rooted t : t tht stge there ws single item in the registry of the form α β, nmely the item. Using NF updte rule 3, upon seeing the next, we updte this item to s the input ecomes: So lthough the vrile never occurs s n input symol, when we discover prse tree rooted t, it is helpful to replce the corresponding portion of the input y the symol nd move the ck in front. t this point we cn pply one of the trnsitions descried in NF updte rule 3. Now notice tht NF updte rules 1 nd 3 re very similr. We cn merge them into the following single rule: NF updte rule 1 nd 3 For every terminl or vrile X nd every rule of the form B αxβ in G, put trnsition from B α Xβ to B αx β with lel. pplying the construction to the CFG G, we otin the following NF N G : strt q 0 c c c

7 7 The NF we just descried implements ll the steps we took in the prsing exmple, except for the step in line 9: How cn the NF know which items were discrded k steps go? We will tke cre of this lter y ugmenting the NF with stck tht will implicitly keep trck of the registry t different stges of the prsing process. But efore we do so, there is nother prolem we need to solve. Eventully, to produce the prse tree, we will need to run the NF N G. However, n NF is difficult to run ecuse it cn e in severl sttes t once. Let us first convert it to DF. Converting the NF to DF: Shift nd reduce sttes Recll tht when we convert n NF to DF, the sttes of the DF re susets of the sttes in the NF. s the sttes of the NFN G correspond to items of G, the sttes of the corresponding DF D G will represent collections of items. Ech such collection of items is possile stte of the registry we used in our exmple. The stte of D G hs specil mening it mens tht the registry is empty. If n input ends up in this stte, it is not in the lnguge of G, nd so it cnnot e prsed. pplying the NF to DF conversion rules to N G, we otin the following DF D G. For clrity we omit the stte from the digrm. strt c 1 2 c 3 4 c c 5 6 This DF hs two kinds of sttes: Sttes 1, 2, nd 3 ll contin only items tht re not complete. Such sttes re clled shift sttes. Sttes 4, 5, nd 6 only contin single item tht is complete. Such sttes re clled reduce sttes. Our prsing strtegy crucilly relies on the fct tht there re no other kinds of sttes: If we hd shift item nd reduce item inside single stte, we would not know whether our following ction should e to red the next input symol corresponding to the shift item or uild portion of the prse tree corresponding to the reduce item. Definition 2. CFG G is n LR(0) grmmr if ll sttes in the DF D G re either shift sttes or reduce sttes. 4 Trcking previous items in stck Let us now ttempt to use the DF D G to prse the string. Using the mrker to denote our position in the input, we re initilly looking t, nd we re in stte 1, which is shift

8 8 stte. fter reding the input, we move the stte 2, lso shift stte, nd the stte of the input ecomes. We red the next, loop to stte 2, nd the input ecomes. On the next, we move to stte 6, which is reduce stte. We cn now uild the portion of the prse tree: We would now like to go ck to stte 2, with the following picture in mind: But how is the DF D G know tht it is supposed to go to stte 2? We would like to send D G ck to whichever stte it ws in efore we processed the lst two symols. The DF itself cnnot rememer this stte, ut we cn help it store this informtion y using stck. Once we ugment the DF with stck, wht we hve is no more DF, ut PD P G. The stck P G will keep trck of the sequence of sttes tht the prsing process goes through. s usul, the stck will e initilized to $. fter shifting the first, we push the symol 1 onto the stck, in order to rememer where we were efore we processed this. fter the second, we push 2. fter the first, we push nother 2. We now find ourselves in reduce stte 6 with input stte nd stck contents. $122. We now wnt to replce the pttern in the input with the pttern. But we must lso figure out which stte of the DF to go ck to efore we continue the processing. To do so, we look t the stte 6 item nd notice tht it comes from production with two symols on the right. To figure out where we were efore we processed these two symols, we therefore need to pop two items from the stck, nd go to the stte descried y the lst item we popped. Since the stte 6 production hs two symols on the right nd the first one is 2, fter popping these two symols we end up in the following sitution: We re in stte 2, the contents of the stck re $1, nd the input is now, representing the following prtil prse tree: We re now ck into shift stte, so peek t the next input symol which is now n. We move to stte 3, the input ecomes, nd the stck contents re now $12. This is gin shift stte, so we process the next input symol. The input ecomes, the stck ecomes $123, nd we re now in reduce stte 4. Stte 4 hs three symols on the right. fter reding off three symols from the stck nd uilding the prse tree tht corresponds to this item, we re sent ck to stte 1 nd the input is now

9 9 The stck now only contins the ottom of stck symol $, so we hve finished prsing nd re redy to ccept the input. forml description of the PD generl LR(0) grmmr G. 2 We now give forml description of the PD P G for Sttes: The sttes of P G re the sme s the sttes of D G. In ddition, P G hs specil strt stte S nd specil finl stte S. There re no other finl sttes. Trnsitions: The PD D G hs the following trnsitions: For every shift stte q nd every trnsition q X q of D G (except those going to the ded stte of D G ), P G hs trnsition from q to q with the ction red X, push q. If q is reduce stte with item X 1... X k, P G hs trnsitions going out of q with the following sequence of ctions: pop q k,..., pop q 1, go to stte q 1. Replce the portion of the input contining the pttern X 1... X k with nd construct prt of the prse tree y connecting to X 1,..., X k. In ddition, P G hs the specil trnsitions push $ from stte S to the strt stte of D G nd pop $ from the strt stte of D G to the finl stte S. 5 The prser genertor We now hve ll the ingredients necessry to descrie the prser genertor for LR(0) grmmrs. Recll tht on input G, the output of the prser genertor is n lgorithm for prsing G if G is n LR(0) grmmr. If G is not LR(0), the prser genertor produces n error messge. On input G, where G is CFG: Construct the NF N G. Convert N G to DF D G. If D G hs sttes tht re neither shift or reduce sttes, Output the error messge G is not n LR(0) grmmr nd hlt. Otherwise, construct nd output the PD P G. 2 Techniclly, this is not PD like the ones we sw in previous lectures ecuse in ddition to deciding if the input is in the lnguge of G, our PD needs to uild n ctul prse tree.