Evaluating GLR parsing algorithms

Transcription

1 cience of Compter Programming 61 (2006) Evalating GLR parsing algorithms Adrian Johnstone, Elizabeth cott, Giorgios Economopolos Department of Compter cience, Royal Holloway, University of London, Egham, rrey, United Kingdom Available online 19 May 2006 Abstract We describe the behavior of three variants of GLR parsing: (i) Farshi s original correction to Tomita s non-general algorithm; (ii) the Right Nlled GLR algorithm which provides a more efficient generalisation of Tomita and (iii) the inary Right Nlled GLR algorithm, on three types of LR table. We present a gide to the parse-time behavior of these algorithms which illstrates the inefficiencies in conventional Farshi-style GLR parsing. We also describe the tool GT (Grammar Tool ox) which provides a platform for comparative stdies of parsing algorithms; and se GT to exercise the three GLR algorithms rnning with LR(0), LR(1) and LR(1) tables for ANI-C, IO-Pascal and IM V-COOL. We give reslts showing the size of the strctres constrcted by these parsers and the amont of searching reqired dring the parse, which abstracts their rntime. c 2006 Elsevier.V. All rights reserved. Keywords: GLR parsing; Grammar types; Context free langages; LR tables The compting literatre contains plenty of experimental stdies, bt compter science, nlike traditional experimental science, is poor at providing geninely repeatable reslts. This arises partially from the rapid changes in the nderlying technology and partly from a natral desire of researchers to explore new fields rather than comprehensively mapping the territory opened p by pioneers. Frthermore the complexity of compter systems can make it difficlt to define problems in a manner that allows comparative experimental approaches. Compilers and formal langage translators in general are perhaps or best candidate for combining formal rigor with engineering practicality. It is forty years since the links were noted between Chomsky s formalisms and the engineering practice of langage design and translator implementation; and nearly twenty years since the engineering practice for parsing conventional programming langages settled on the se of LR style bottom p parsers, sally based on LALR tables which are atomatically generated, and the simpler LL(1) parsers which are often hand written. LALR techniqes admit a large sbset of the deterministic langages and, with a little jdicios se of prioritisation to disambigate table conflicts, the well known generators sch as YACC [11] and its derivatives GNU ison and TYACC [7] have redced parser generation to a mostly clerical task. Nevertheless, sers of these tools have to grapple with the lack of generality of deterministic parsers, and it is or experience that both neophyte and experienced langage designers experience nasty srprises. ottom p parsers, in particlar, display behavior that can be hard to interpret in terms of the grammar. For example, YACC-like tools allow semantics to be specified within a prodction Corresponding athor. Tel.: ; fax: addresses: a.johnstone@rhl.ac.k (A. Johnstone), e.scott@rhl.ac.k (E. cott) /$ - see front matter c 2006 Elsevier.V. All rights reserved. doi: /j.scico

2 A. Johnstone et al. / cience of Compter Programming 61 (2006) even thogh semantics exection is in reality associated with the redction of an entire rle. YACC will silently split a rle to spport semantics exection, bt in doing so may generate rles with LALR conflicts. The effect from a ser s point of view is that a working parser breaks when semantics are added. In the last decade the compting commnity has shown an increasing interest in parsing techniqes that go beyond the standard approaches. There are a plethora of parser generators that extend both top-down and bottomp approaches with backtracking and lookahead constrcts. As we have noted elsewhere [12] sch parsers can display srprising pathologies: in particlar parser generators sch as PRECC [4], PCCT [22], ANTLR [21] and JAVACC [1] are really matching strings against ordered grammars in which the rle ordering is significant, and it can be hard to specify exactly what langage is matched by sch a parser. In any case, backtracking yields exponential parse times in the worst case. A safer approach is to se one of the trly general context free parsing algorithms sch as Earley [8], CYK [33] or a variant of Tomita s GLR algorithm [30]. Many of these algorithms are primarily recognisers which retrn data strctres from which derivations may be extracted one at a time. For ambigos grammars this may be nacceptable since the nmber of derivations is not necessarily finite, and no garantees can be made abot the order in which they are extracted, hence the practical time bond may be dominated by the examination (and rejection) of semantically nacceptable derivations. Practical general algorithms display at least cbic worst case time behavior bt on modern hardware this need not preclde their se. everal Eiffel compilers se Earley parsers to spport their relaxed se of expression separators. Tools sch as AF+DF [31] stress generality and can parse ambigos langages with spport for shared packed parse forests (PPF) which efficiently encode all possible derivations along with sophisticated techniqes for disambigating the forest. AF+DF ses GLR parsing to accomplish this: as a final indicator that general techniqes are entering the mainstream consider that even GNU ison has recently acqired a general parsing mode. (It trns ot that the present ison GLR implementation merely splits stacks when conflicts are encontered, so it displays exponential growth in memory reqirements which makes it impractical; bt the addition of this mode at least demonstrates the demand for general parsing.) In recent years we have stdied a wide variety of general parsing techniqes along with mechanisms for extracting derivation forests; semi-atomatic generation of abstract syntax trees; and dataflow analysis and code generation techniqes based on the reslting strctres [14]. We ended p with a set of only loosely comparable tools: in particlar it was hard to answer what-if qestions abot parse-time performance and the size of the strctres reqired for particlar techniqes in particlar application niches. The Grammar Tool ox (GT) is a nifying framework into which we can implement new theoretical contribtions beside those already in the literatre so as to allow direct experimental comparisons as well as to act as a prodction qality translator generator. In this paper we shall se GT to describe the behavior of three variants of GLR parsing: (i) Farshi s original correction to Tomita s non-general algorithm; (ii) the Right Nlled GLR algorithm which provides a more efficient generalisation of Tomita and (iii) the inary Right Nlled GLR algorithm which achieves cbic performance on any context free grammar whilst retaining linear time efficiency when parsing deterministic grammars. We present a gide to the parse-time behavior of these algorithms which illstrates the inefficiencies in conventional Farshi-style GLR parsing and exercise the three GLR algorithms rnning with LR(0), LR(1), LALR and LR(1) tables for ANI-C, IO-Pascal and IM V-COOL. We give reslts showing the size of the strctres constrcted by these parsers and the amont of searching reqired dring the parse, which abstracts their rntime. 1. The Grammar Tool ox (GT) and Parser Animation Tool (PAT) GT is an interpreter for a procedral programming langage with facilities for direct maniplation of translator related data strctres. A set of bilt-in fnctions is provided for creating, modifying and displaying these strctres. At the simplest level, the GT langage is sed for scripting a standard process sch as the generation of an LR(1) parse table. Unlike a conventional monolithic parser generator, GT reqires the generation process to be specified as a detailed chain of operations on grammars and atomata. Or goal is to open p the degrees of freedom in translator implementation in a strctred way so that reprodcible experiments can be monted between competing techniqes. Natrally, a pre-written GT script may be sed to get the effect of a conventional parser generator like YACC or RDP, so GT can replace those kinds of tools in a prodction environment.

3 230 A. Johnstone et al. / cience of Compter Programming 61 (2006) The Parser Animation Tool (PAT) [9] is an accompanying visalisation tool that has its own implementations of or parser algorithms which rn from tables generated by GT. PAT is written in Java, and GT in C++ so they necessarily reqire separate implementations, bt we have the added benefit of two athors constrcting independent programs from the theoretical treatment. As might be expected, reconciliation of reslts has sometimes revealed bgs. In se PAT dynamically displays the constrction of parse-time strctres. PAT can also be sed in batch mode to generate statistics on parser behavior: the reslts given in ection 4.3 were obtained in this way. 2. LR parsing fndamentals The standard LR parsing approach was introdced by Knth [17]. The idea is to constrct a deterministic finite state atomaton (DFA) whose transitions are labelled with terminals and non-terminals from a context free grammar and whose states are labelled with sets of grammar rles. The standard LR parser then ses a stack and an inpt string to traverse the DFA. The state on the top of the stack is the crrent state. If there is a transition from this state labelled with the crrent inpt symbol then the target of this transition is pshed onto the top of the stack and the crrent inpt symbol is pdated. This is referred to as a shift action. If the state at the top of the stack is labelled with a grammar rle A ::= x 1... x j then the top j states are popped off the stack, leaving state h say on top, and then the target of the transition labelled A from h is pshed onto the stack (it is a property of the DFA that sch a transition will always exist). This is referred to as a redction action. The inpt is accepted if, when all the inpt has been read, the state on the top of the stack is the accepting state of the DFA. (ee, for example, [2] for a fll discssion of standard LR parsing.) Althogh the DFA itself is deterministic, the LR parsing process may not be. It is possible for there to be both a shift and a redction associated with a DFA state, and a DFA state may be labelled with more than one rle, yielding a choice of redction actions. These choices are called LR-parsing conflicts. There are several classical classes of DFA which can be sed as the basis of an LR parser, each constrcted with different ses of inpt symbol lookahead. The LR(0) DFA does not se lookahead, the LR(1) DFA ses one symbol of lookahead in a global fashion sing the so-called follow sets, and the LR(1) DFA ses the lookahead symbols in a more detailed fashion which takes accont of the local properties of the grammar. The LALR DFA [6] is obtained by merging some of the LR(1) DFA states. The classes of grammar whose LR(0), LR(1), LALR, and LR(1) tables, respectively, are conflict-free form a strict hierarchy nder inclsion. The standard LR parsing algorithm reqires that there be no conflicts, ths sing LR(1) DFAs permits the largest class of context free grammars to be parsed with the techniqe. However, while for a given grammar the LR(0), LR(1) and LALR DFAs are the same size, the LR(1) DFA can be sbstantially larger LR tables in GT We follow Grne s treatment [10] of LR DFA generation, both in or theoretical reslts and in the implementation. Traditional treatments of LR parsers focs on the definition of the LR handle-finding atomaton as a closre over sets of items. In general an item is an ordered pair (, F) where is a dotted position in the grammar and a lookahead set representing those terminals which a parser may legally enconter dring a parse. etting F appropriately gives an LR(0), LR(1) or LR(1) atomaton. In [10] these relationships are emphasised by writing the grammar rles as an NFA in a natral way. The handle-finding DFA is then derived from this NFA by applying the sbset constrction. We have fond this approach sefl both pedagogically and as a basis for theoretical analysis of LR parsers. As an implementation method it is less attractive since the LR(1) NFA s for some langages can be large and in or present scheme mst be co-resident with the DFA. ome ingenity has been reqired to design internal representations for the NFA that optimise the se of memory, and we do not rle ot more direct constrction techniqes in GT for prodction se. The following GT example illstrates the definition of a (tiny) langage. In the GT inpt format grammar rles are written in simple NF, one rle for each non-terminal, terminals are single qoted strings and # represents ɛ. The example shows the generation of all sentential forms in the langage and the constrction of the LR(1) DFA and parse table via the constrction of an NFA followed by the application of the sbset constrction. The NFA and DFA are also otpt in a format which can be viewed sing the VCG tool [24], as shown in Figs. 1 and 2. ::= a A A z. A ::= a.

4 A. Johnstone et al. / cience of Compter Programming 61 (2006) Fig. 1. Nondeterministic LR(1) atomaton. Fig. 2. Deterministic LR(1) atomaton. ::= b #. ( this_grammar := grammar[] this_nfa := nfa[this_grammar lr 1 nllable_redctions] render[open["nfa.vcg"] this_nfa] lr1_dfa := dfa[this_nfa] lalr_dfa := la_merge[lr1_dfa] this_derivation := rnglr_parse[lalr_dfa "a b"] render[open["ssg.vcg"] this_derivation] write["\n" CPU_time " CPU seconds elapsed\n\n"] ) GT can handle mltiple grammars simltaneosly, and extract mltiple grammars from a rle set by sing different start symbols (a facility which is sefl for some techniqes that segment grammars, sch as the Aycock and Horspool trie-based atomaton [3]). The bilt-in fnction grammar constrcts a grammar object from a particlar start rle and as a side effect calclates first and follow sets. Nondeterministic atomata are constrcted sing the nfa fnction. The types presently spported are LL, LR and the nrolled -LR atomata sed in or version of Aycock and Horspool s parsing algorithm [26]. Follow sets are specified with positive integers for right-hand-side (instance-level) follow sets and negative integers for left-hand-side follows, so an nrolled LR(1) table is obtained with nfa[this_gram nrolled -1]. (In practice, we allow some sgared calls so nfa[this_gram slr 1] has the expected effect.) An LR DFA is obtained by rnning the sbset constrction on one of these NFAs. An LALR DFA can be constrcted from an LR(1) DFA sing the la_merge fnction. Once the DFA has been constrcted the reqired parsing algorithm can be rn. For example, rnglr_parse rns an RNGLR parser, and otpts a G in a format that can be viewed sing VCG.

5 232 A. Johnstone et al. / cience of Compter Programming 61 (2006) GLR parsing GLR parsing, an extension of LR parsing, was introdced by Tomita [30] as a means for natral langage parsers to captre all possible derivations when parsing potentially highly ambigos grammars. Tomita-style GLR algorithms extend the LR parsing techniqe to cope with conflicts, ths GLR parsers can se any of the LR DFAs. However, we expect the efficiency of the GLR parser to be different on different types of DFA. The LR(1) parser typically enconters more conflicts and ths may generate more parser activity, bt the LR(1) DFA contains more states and ths the size of the rntime strctres may be larger. In ection 4.3 we se GT and PAT to investigate the effects of different DFA types on GLR parsers. First we give an otline of the GLR algorithms we se. Nondeterminism in an LR parser gives rise to mltiple crrent stacks. Tomita s algorithm constrcts a graph strctred stack (G) that efficiently represents these stacks. Tomita s approach centres arond the processing of states crrently on the tops of the stacks. We think of the G nodes corresponding to these states as the crrent frontier of the G, and the stacks are lock-stepped on shift actions so the G nodes fall into levels associated with each inpt symbol. In detail, as each node on the frontier is processed each redction A ::= x 1... x j in the corresponding state (if there are any) is applied down all paths of length j from. This can case new states to be added to the frontier. When all the nodes on the frontier have been processed, a new frontier is begn by applying the shift actions associated with the next inpt symbol. We describe the GLR algorithm sing as a rnning example the grammar, Γ 1, whose rles are ::= a a ::= b ɛ Here and are non-terminals, a and b are terminals, ɛ is the empty string, and is the start symbol. We shall se the LR(0) DFA for Γ 1, shown below, bt or discssion applies eqally well to the LR(1), LALR and LR(1) DFAs. Traditionally DFA states are labelled with sets of items (see ection 3.5) bt we have only shown the redction items. The start state is state 0 and the accepting state is state 1. a ::= a ::= ::= ::= a a 4 1 b 7 ::= b b The DFA will be traversed in the standard way, reading an inpt string, in this example aab, and pshing states onto a stack. The first three steps of the algorithm, given in ection 3.1, illstrate the basic approach, inclding the stack sharing in the G. The last step generates mltiple pending actions and at that point we discss, ection 3.2, the se of work lists to manage the action application. The example has been chosen to illstrate the basic ideas, bt there are problems with the approach. We shall illstrate these, and their soltions, in ection 3.3 sing the same grammar with a different inpt string, aaa The basic approach We traverse the above DFA for Γ 1 with the inpt string aab. We read the first inpt symbol, a, begin in the start state, 0, and constrct a G node 0 labelled 0. There are no redctions from state 0 so we traverse the a-labelled arrow, perform the shift action and psh onto the stack the state, 2, which is the target of the a-arrow from 0. Ths we create a new G node 1 labelled 2 and an arrow from this to 0 to indicate that 2 is above 0 in the stack. 0 0 a 1 G 2 tack 0 2 (The label a on the arrow is not really part of the G bt it has been inclded to make the exposition clearer.) We then read the next inpt symbol, a, and apply the redction actions. tate 2 incldes a redction sing the rle ::= a. To apply this redction we need to trace back along all paths of length 1 from the node 1, in this case we jst get node 0, and then create a new node 2 labelled 1 and an edge ( 2, 0 ). This corresponds to dplicating the stack and performing the corresponding pop and psh operations.

6 A. Johnstone et al. / cience of Compter Programming 61 (2006) a G tacks 0 0 tate 1 has no redctions so the frontier is complete and we apply the shift actions. There is an a-arrow from DFA state 2 to itself, so we create a corresponding G node. There is no a-arrow from state 1 so the stack whose top is 1 dies. 0 0 a 1 2 a 2 3 G tacks Next we read the last inpt symbol, b. Applying the redction from state 2 reslts in a G node 4 labelled 3. tate 3 incldes a redction sing the rle ::= ɛ, and the path of length 0 ends at 4, so we create a G node 5 labelled 5 and an edge ( 5, 4 ). tate 5 also incldes a redction sing ::= ɛ, so we create a G node 6 labelled 6. tate 6 has a redction sing ::= a, so we find all paths of length 4 from 6. There is one sch path which ends at 0, and so we create a G node 7 labelled a 2 a G tacks We have now applied all the redctions from all of the crrent stack tops, so we apply the shift actions and read the next inpt symbol which is the end-of-file symbol $. When we apply the shift action from state 3 we create a new G node 8 labelled 4. When we apply the shift action from state 5 we find that there is already a G node labelled 4 on the crrent frontier so we rese this node. This is an example of the stack re-merging that we mentioned above a 2 a b 4 8 b G tacks 0 0 o far in this example we have only had one frontier node to be processed at a time and each redction has been applied down only one path, so we have not needed to address the book-keeping aspects of the process. To cope efficiently with mltiple redctions we se a work list Work lists There is an important cost associated with the se of a G: in an LR parser a redction is performed by popping j states from the stack where j is the length of the rle being redced. This can be performed with a single modification of a stack pointer. In the GLR case a redction involves a search down all pathways of length j, and in worst case this can reqire O(2 j ) time. Ths it is important that the searches are only applied once, and this implies that the algorithm mst incorporate work lists to ensre that previosly carried ot searches are not repeated. It is this efficient constrction of the G which is the important contribtion of Tomita s work.

7 234 A. Johnstone et al. / cience of Compter Programming 61 (2006) Redctions of length greater than zero are applied down paths. We cold keep a list of the nodes that have been created and then process each node, applying any redction associated with the node down all the paths from the node. The problem is that, as we shall see later, additional edges can be added to a node after it has been processed. If we simply re-process the node we will retrace all the paths that we have already explored, as well as the new paths, leading to an nnecessary explosion in the searching cost of the algorithm. Tomita s approach is to keep a work list of the redctions together with the first edge down which they are to be applied. Then when a new edge is created the redctions are only applied down the new paths. In fact all we need to record is the length of the rle associated with the redction action, the non-terminal on the left hand side of the rle, and the node at the end of the first edge down which the redction is to be applied. Ths we have a work list R which consists of triples (, A, m) where is a G node, A is the left hand side of a grammar rle and m is the length of the rle. When a node v is created, if there are any rles ::= ɛ associated with this node then we add (v,, 0) to R. When an edge (v, ) is created, if there are any rles A ::= x 1... x m, where m 1, associated with v then we add (, A, m) to R. In addition we also keep a work list Q for the shift actions. The elements of Q are pairs (v, k) where v is a G node and k is a (integer) DFA state nmber. When a node v labelled h is created, if there is an a-transition from h to k, where a is the next inpt symbol, then (v, k) is added to Q. The algorithm operates by creating the start node 0 labelled 0 and then adding any appropriate elements to R and Q. Contining with or rnning example above, when the shift actions are applied which create the edges ( 8, 4 ) and ( 8, 5 ), the elements ( 4,, 1) and ( 5,, 1) are added to R. When we remove ( 5,, 1) from R we find all paths of length 1 1 = 0 from 5. This, of corse, ends at 5 so we create a new G node, 9, labelled 6 and an edge ( 9, 5 ). tate 6 has an associated redction on the rle ::= a so on the creation of the edge ( 9, 5 ) we add ( 5,, 4) to R. imilarly removing ( 4,, 1) we create a new G node, 10, labelled 5, an edge ( 10, 4 ) and, since state 5 has a redction on ::= ɛ, on the creation of 10 we add ( 10,, 0) to R. 0 0 a 1 2 a b 4 b R = ( ( 5,, 4), ( 10,, 0) ) Removing ( 5,, 4) we look for paths of length 3 from 5, and there is one which ends at 0. From this we create a new node 11 labelled 1. ince there are no actions associated with state 1 no elements are added to R. We then remove ( 10,, 0) from R. There is already a node on the frontier labelled 6, 9, so we add an edge ( 9, 10 ) and the creation of this edge cases ( 10,, 4) to be added to R. 0 0 a 1 2 a b 4 b R = ( 10,, 4) When ( 10,, 4) is removed from R we look for paths of length 3 from 10. There is one which ends at 0. ince there is already a frontier node, 11, labelled 1 and an edge ( 11, 0 ) no new nodes or edges are created, and the list R is empty. ince the inpt symbol is the end-of-string symbol, and there is a stack with the accepting state 1 on the top (i.e. there is a node labelled 1 on the crrent frontier), the inpt string is (correctly) accepted.

8 3.3. The failre of Tomita s algorithm A. Johnstone et al. / cience of Compter Programming 61 (2006) Tomita s basic algorithm [30], which we call Tomita-1, can be applied to all ɛ-free grammars. What we have described above is essentially an extension of Tomita-1 to inclde ɛ-rles in the obvios way, and we call this algorithm Tomita-1e. (Or formlation is slightly different from Tomita s in that Tomita s G inclded nodes for the grammar symbols as well as the DFA states. Also, Tomita did not se the optimisation of starting the redction path searches from the second node on the path.) The problem is that Tomita-1e does not always work correctly on grammars with hidden right recrsion. Consider again the above grammar Γ 1, this time with the inpt string aaa. Reading the first three inpt symbols and proceeding as above, after we perform the shift action associated with the third a we have the following G and work list R. 0 0 a 1 2 a 2 3 a R = ( 3,, 1) Removing ( 3,, 1) from R reslts in the creation of a G node 9 labelled 3 and an edge ( 9, 3 ), and ( 9,, 0) is added to R. When this element is removed we create 10 labelled 5 and the element ( 10,, 0), and then in trn we create 11 labelled 6 and the element ( 10,, 4). Processing ( 10,, 4) we find the path of length three from 10 to 1 and we create an edge ( 9, 1 ). 0 0 a 1 2 a 2 3 a ince state 3 has only a redction of length 0, there are no elements added to R as a reslt of the creation of the edge ( 9, 1 ) ths the work lists R and Q are both empty and the process terminates. ince there is no node labelled 1 on the crrent frontier, the inpt string is (incorrectly) rejected. The problem is that the edge ( 9, 1 ) creates a new path of length 4 from 11 and so the redction ::= a shold be applied down this path. The method of storing redctions on a work list with the first edge of the path down which they are to be applied only works if new paths are only created by adding edges at the front of existing paths. Tomita was aware of this and modified his algorithm to try to ensre this property held, bt the modification cased the constrction of an infinite G in certain cases. A different soltion was proposed by Farshi [20] Farshi s algorithm Farshi extended Tomita-1 to inclde ɛ-rles and added a brte force search to ensre that rles are correctly applied down newly created paths. The algorithm ses slightly different work lists from those we have described. There is a work list A of frontier nodes waiting to be processed, and when a node is created it is added to A. When a node v, whose label is h say, is removed from A, if there is a DFA transition from h to k labelled with the crrent inpt symbol then, as for Tomita-1e, the pair (v, k) is added to a work list Q. Redction actions are treated slightly differently. For each rle D ::= x 1... x m in state h (if there are any) all the G nodes at the ends of paths of length m from v are fond and for each sch node w the pair (w, D) is added to R.

9 236 A. Johnstone et al. / cience of Compter Programming 61 (2006) There is a state, t say, which is the target of an D-transition from l, where l is the label of w, and when (w, D) is removed from R, if there is no frontier node labelled t, then one is created and added to A, and an edge is created from this node to w. Farshi s approach reslts in the frontier nodes being created in a different order to Tomita s bt the method is essentially the same and it does not solve the basic problem with Tomita s algorithm. To address this Farshi added an additional step to the processing part of his algorithm. A new edge in the middle of an existing path can only be created when processing an element (w, D) from R. Ths there is an additional instrction when sch an edge, (, w) say, is created. This instrction says: If an edge (, w) is created from an existing node, then for each node v on the crrent frontier which is not in A, for each rle ::= x 1... x j associated with v, and for each path of length j from v which incldes the edge (, w), add (y, ) to R, where y is the node at the end of the path. This does solve the problem with Tomita s algorithm, bt at a very high cost. Farshi s algorithm does not say how paths which inclde a particlar edge are to be fond, and in the limit, all paths of length j mst be re-traced to determine whether or not they contain the edge. This is exactly the potentially explosive repeated searching that Tomita s algorithm was designed to avoid. We illstrate Farshi s approach by retrning to the above example and the string aaa, at the point where the shift on the third a is applied. The node 8 is created and added to A. This node is removed and, for the rle ::= a which is associated with state 4, Farshi s algorithm finds the nodes at the end of paths of length 1 from 8, in this case 3, and adds ( 3, ) to R. Next ( 3, ) is removed from R and processed. This reslts in the creation of 9 labelled 3, which is added to A, and the creation of the edge ( 9, 3 ). Now 9 is removed from A and, becase ::= ɛ is associated with state 3, we add ( 9, ) to R. When this element is removed a node 10 labelled 5 is created. When 10 is processed ( 10, ) is added to R, and then processing this reslts in the creation of 11 labelled 6 and the edge ( 11, 10 ). When 11 is processed, becase ::= a is associated with state 6, we find the nodes at the end of paths of length 4 from 11 and hence add ( 1, ) to R. 0 0 a 1 2 a 2 3 a R = ( 1, ) When ( 1, ) is processed we find that there is already a node 9 labelled 3, so we jst create an edge ( 9, 1 ). At this point the additional step in Farshi s process is triggered. The nodes 8, 9, 10 and 11 are no longer in A, ths paths of length 1 from 8 and of length 4 from 11 are looked for again. The path from 8 and one of the paths from 11 do not contain the new edge, so no elements are added to R. t we also find an additional path of length 4 from 11 which does contain ( 9, 1 ). The end of this path is 0 so ( 0, ) is added to R. Ths we have fond the extra redction application that Tomita s algorithm missed, bt note that in the process we have had to trace one of the paths of length 4 from 11 twice. Finally removing ( 0, ) from R we create a new frontier node 12 labelled 1 and an edge ( 12, 0 ). At this point all of the work lists are empty so the algorithm terminates and, since there is a frontier node labelled 1, the algorithm (correctly) accepts the inpt string. In this example, Farshi s modification only reqired two paths, one of length 4 and one of length 1, to be traversed twice. t it is not hard to see that on the inpt string aaaa a similar pattern will appear bt two extra edges will be added to the node labelled 3 on the last frontier. The creation of the second edge will case a path of length 4 to be retraced, and the creation of the third will case two paths of length 4 to be retraced. Ths we will have a total of for extra length 4 path traversals. For inpt aaaaa the total is 10, and so on. In the above discssion we have described Farshi s algorithm as it is described in [20]. Of corse we can limit the length of the re-searching carried ot sing some observations abot the creation of new edges. As written, Farshi s

10 A. Johnstone et al. / cience of Compter Programming 61 (2006) algorithm reqires s to search for all paths containing the new edge. However, the strctre of the algorithm ensres that any new edge will have its sorce node on the crrent frontier. Ths when searching for new paths which contain the edge we can terminate the search down each path when we reach a node which is not on the crrent frontier. In or experiments we have implemented Farshi s algorithm exactly as it is described in [20], and we have also implemented a more efficient version which trncates the searches as described above. We refer to the former as Farshi-naïve and the latter as Farshi-opt, and we have presented the reslts for both of these algorithms The RNGLR algorithm The RNGLR algorithm [27] provides a different soltion to the problem with Tomita-1e, which involves changing the DFA rather than the algorithm. If an edge (, w) is added to an existing G node, then is on the crrent frontier. o if there is a path j,..., p,, w, p 3,..., 1 in the G then j,..., p are also on the crrent frontier. It is not too hard to see that if a redction A ::= x 1... x j has to be applied down this path then we mst have x r ɛ, for p r j. Ths rather than applying the redction A ::= x 1... x j from the node j we apply the redction A ::= x 1... x p 1 from. One of the beaties of this approach is that it can be achieved simply by adding some extra redctions to the DFA states. The standard DFA constrction process labels the DFA states with items which are grammar rles with a dot on the right hand side. The rles that we have associated with the states in the above description have the dot at the end, A ::= x 1... x j, and or additional rles have the dot at the point where the limited redction is to be applied, A ::= x 1... x p x p... x j. We call the DFAs with these additional redctions right nlled (RN) DFAs. They are constrcted in exactly the same way as traditional LR DFAs, and then additional items are nominated as redction items. The RN LR(0) DFA for the grammar Γ 1 in or rnning example is a ::= a ::= a ::= a ::= ɛ ::= ɛ 3 a 6 1 b b 7 ::= a 4 ::= b To illstrate the effect of the RN DFA we consider again the above example and the string aaa, and take p the process at the point where the shift on the third a is applied. Again the node 8 and edge ( 8, 3 ) are created, casing ( 3,, 1) to be added to R. When this element is removed from R a node 9 labelled 3 and an edge ( 9, 3 ) are created and, becase both ::= ɛ and ::= a are associated with state 3, ( 9,, 0) and ( 3,, 2) are added to R. (It is the addition of the latter element which allows the parser to eventally terminate correctly.) As in the previos cases, processing ( 9,, 0) cases a node 10 labelled 5 to be created, and as a reslt ( 10,, 0) and ( 9,, 3) are added to R. Processing ( 3,, 2) reqires s to look for paths of length one from 3, and there is one which ends at 1. ince there is already a frontier node labelled 3 we jst add a new edge ( 9, 1 ) and then an element ( 1,, 2) to R. 0 0 a 1 2 a 2 3 a R = ( ( 10,, 0), ( 9,, 3), ( 1,, 2) ) Processing ( 10,, 0) creates a node, 11, labelled 6 and cases ( 10,, 4) to be added to R. Processing ( 9,, 3), we look for nodes at the end of paths of length 2 from 9. We find 1 and 0. An edge to 1 of the reqired type already

11 238 A. Johnstone et al. / cience of Compter Programming 61 (2006) exists so we take no frther action. For 0 we create a frontier node 12 labelled 1. Finally, processing ( 1,, 2) and ( 10,, 4) does not reslt in any new G edges, so the process is complete and has constrcted the same final G as Farshi s algorithm did. In fact it trns ot that becase we have added the additional redctions it is not necessary to apply any of the redctions down paths whose first edge lies between two nodes on the crrent frontier. This is becase a limited version of the redction will be applied at an earlier point in the algorithm. This allows s to present a slightly different version of Tomita-1e which does not reqire any additional searching, and in fact is even slightly more efficient than Tomita-1e. (pecifically, in the above example the elements ( 9,, 3) and ( 10,, 4) will not actally be added to R.) We call this algorithm the RNGLR algorithm, and the RNGLR approach is discssed in detail in [27] and [28]. For completeness we inclde, withot frther discssion, the formal version of the RNGLR algorithm. Inpt: an RN DFA written as a table T, an inpt string a 1... a d PARER { if d = 0 { if acc T (0, $) report sccess else report failre } else { create a node v 0 labelled with the start state 0 of the DFA. set U 0 = {v 0 }, R =, Q =, a d+1 = $, U 1 =,..., U d = if pk T (0, a 1 ) add (v 0, k) to Q forall r(x, 0) T (0, a 1 ) add (v 0, X, 0) to R for i = 0 to d while U i { while R { REDUCER(i) } HIFTER(i) } if the DFA final accepting state is in U d report sccess else report failre }} REDUCER(i) { remove (v, X, m) from R find the set χ of nodes which can be reached from v along a path of length (m 1), or length 0 if m = 0 for each node χ do { let k be the label of and let pl T (k, X) if there is a node w U i with label l { if there is not an edge from w to { create an edge from (w, ) if m 0 { forall r(, t) T (l, a i+1 ) where t 0, add (,, t) to R }}} else { create a node w U i labelled l and an edge (w, ) if ph T (l, a i+1 ) add (w, h) to Q forall r(, 0) T (l, a i+1 ) add (w,, 0) to R if m 0 { forall r(, t) T (l, a i+1 ) where t 0, add (,, t) to R }}}} HIFTER(i) { if i d { Q = while Q { remove an element (v, k) from Q if there is w U i+1 with label k { create an edge from (w, v) forall r(, t) T (k, a i+2 ) where t 0 add (v,, t) to R} else { create a node, w U i+1 labelled k and an edge (w, v) if ph T (k, a i+2 ) add (w, h) to Q forall r(, t) T (k, a i+2 ) where t 0 add (v,, t) to R forall r(, 0) T (k, a i+2 ) add (w,, 0) to R}} copy Q into Q }}

12 3.6. The RNGLR algorithm A. Johnstone et al. / cience of Compter Programming 61 (2006) The Tomita-1e and RNGLR algorithms are of worst case order O(n k+1 ) where k is the length of the longest rle in the grammar, and the order of Farshi s algorithm can be even higher. The rn-time costs come from the fact that in the limit there can be m k paths of length k from a node on the mth frontier and these paths mst all be traversed when a redction is carried ot. It is possible to trn the RNGLR algorithm into a cbic algorithm by performing the searches for paths of length k in k 1 steps of length 2. For example, in the basic RNGLR algorithm if an edge is the first edge of m k 1 paths from a node then it can be traversed m k 1 times. Traversing the paths in steps of length two ensres that each edge is only traversed at most O(n) times for an inpt sting of length n. We call this two step version the (inary) RNGLR algorithm, and it is described in detail in [25]. elow, we se GT to compare the performance of the Farshi, RNGLR and RNGLR algorithms From recognition to parsing Tomita designed his algorithms to allow straightforward generation of an PPF, a representation of the (potentially infinite) set of derivation trees for an inpt string. It is possible for derivation tree generation to significantly increase the rn-time cost and space reqirements of a recognition algorithm, bt Tomita designed the PPF generation to be as efficient as the recogniser part of the algorithm. In particlar, Tomita s G nodes have intervening symbol nodes which correspond to nodes in the PPF. Farshi described only a recogniser version of his algorithm. Rekers [23] trned Farshi s algorithm into a parser in which the G edges are labelled with PPF nodes. Visser [32] ses a generally similar approach bt incorporates lexical processing directly into the parser and its associated G and PPF. The RNGLR algorithm ses Rekers approach to PPF generation becase, althogh reqiring a little more searching, Rekers approach reslts in more compact PPFs, see [27] for more discssion of this. In ection 4.3 we report on the both the G and PPF sizes. 4. Experiments The complete tool chain for these experiments comprises ebnf2bnf, a tool for converting extended NF grammars to NF nder the control of annotations that specify the type of expansion to be applied; GT which is sed to analyse grammars and constrct tables and PAT which is sed to animate algorithms and generate statistics on parsetime strctre size. We report reslts for Pascal, C and COOL along with small grammars that display pathological behavior with some algorithms Grammars for IO-Pascal, ANI-C and COOL Pascal and C typify the top down and bottom p schools of langage design. In the folklore at least, Pascal is thoght of as being designed for LL(1) parsing and C for LALR parsing. In practice, Pascal is indeed reasonably close to LL(1) notwithstanding the IF-THEN-ELE ambigity and the need for lexical backtracking to distingish between a real literal (2.3) and an integer range C is essentially parsable by LALR parsers, bt was not initially designed that way. The LALR ANI-C grammar was only written by Tom Penello in abot jorn trostrp described at some length the difficlties involved in attempting to bild parsers for early versions of C++ sing a hybrid of YACC and a lexer containing mch lexical trickery relying on recrsive descent techniqes [29, pp 68 69,103]. Those of s interested in generalised parsing shold perhaps be gratefl for C++ s nondeterministic syntax since it has clearly stimlated the development of tools sch as ANTLR and the GLR mode of ison. COOL s development was contemporary with that of Algol-60 and ths pre-dates the development of deterministic parsing techniqes. The langage has a large vocablary (some 400 terminals depending on variant) which can challenge any table based parsing method. For these experiments we have sed the grammar for IO-7185 Pascal extracted from the standard, the grammar for ANI-C extracted from [15] and a grammar for IM V-COOL developed in Amsterdam. The original extraction of this grammar is described at length in [18] and some interesting work on techniqes to generate tolerant variants is described in [16]. A version of the grammar is available as a hyperlinked browsable HTML file at v.nl/grammarware/browsable/vs-cobol-ii/: we sed a version prepared for AF+DF from which we extracted the context free rles.

13 240 A. Johnstone et al. / cience of Compter Programming 61 (2006) In all three cases we have sppressed lexical level prodctions and sed separate tokenisers to convert sorce programs into strings of terminals from the main grammar, so for instance the Pascal fragment program tree_file(inpt, otpt) ; const MAX_INTET = 200 ; MAX_TREE = 200 ; fnction intset_create(var s : intset) : intset_sig ; is tokenised to program ID ( ID, ID ) ; const ID = INTEGER ; ID = INTEGER ; fnction ID ( var ID : type_id ) : type_id ; For Pascal, or sorce was a tree viewer program which has approximately 5000 tokens; for ANI-C a oolean eqation minimiser of abot 4500 tokens; and for COOL, strings of approximately 2000 tokens extracted from the test set spplied with the grammar ENF to NF conversion isses Or sorce COOL and Pascal grammars are specified sing variants of ENF, that is, NF spplemented by reglar expressions. The browsable COOL grammar also incldes permtation phrases [5] which are sed to express free-order constrcts in which each phrase may appear at most once: in the AF+DF version these are mapped to Kleene closre over the permtation phrases with the assmption that semantic checks will weed ot strings with mltiple occrrences of a phrase. The standard LR table constrction algorithms do not spport reglar expressions within rles, so ENF rles mst be converted to NF. In general there are many ways to do this and not all approaches yield the same size of table or the same amont of parse time stack activity. As a trivial example consider the expansion of Kleene closre to NF. Left recrsive rles are handled efficiently by LR parsers bt right recrsive rles generate extra stack activity. On the other hand, left recrsive rles case nontermination in LL parsers, so in that case right recrsion is mandatory. One sometimes sees discssions on the size of parse tables for langages. To make realistic comparisons we need to know both the fll provenance of the grammar, and the way that it has been massaged into a form acceptable to the tools from which measrements have been taken. The IM V-COOL grammar is saltary. Or first attempt expanded all closres by creating a new left-recrsive nonterminal for each closre. GT ran ot of memory when trying to create an LR(1) NFA for this grammar. imple back-sbstittion for head and tail recrsive rles generated an NFA with 1.4 million nonterminal headers. Applying a broader set of sbstittions yields an NFA with only 21,000 headers. Clearly, a mechanism for applying transformations in a way that can be adited by grammar sers is needed: starting with a standard grammar is helpfl bt not if we then perform a series of ad hoc operations as we remove the ENF syntactic sgar which can case the size of or atomata to vary by nearly two orders of magnitde. We se a separate tool ebnf2bnf to perform these conversions which takes as inpt an (E)NF grammar annotated with expansion operators and otpts an (E)NF grammar. The tool constrcts a rles tree from the original grammar and then performs tree transformations nder the control of the annotations, before writing the grammar back ot. An ambitios environment that spports this kind of operation has been prototyped in AF+DF [19]: the athors describe ENF to NF conversion as Yacc-ification. Or tool emphasises ease-of-se and traceability of the basic operations Reslts We rn or experiments on grammars as described above for Pascal, C, and COOL, and on the grammar Γ 1 from ection 3. The latter grammar displays mch worse behavior on the LR(0) DFA than on the LR(1) and LR(1) DFAs, illstrating the potential effects of sing lookahead to improve efficiency, even when the parsing techniqe does not reqire lookahead to ensre determinism. We also consider two minor variants of Γ 1 which illstrate sitations in which LR(1) is a mch better choice than LR(1), and in which all three DFA types are eqally bad. Γ 2 : ::= T b T a T ::= a T a ::= b ɛ Γ 3 : ::= T a T ::= a T a ::= b ɛ

14 Table 1 The size of LR and RNLR parse tables A. Johnstone et al. / cience of Compter Programming 61 (2006) Pascal ANI-C COOL LR(0)/LR(1)/LALR LR(1) Table 2 The nmber of conflict cells in LR and RNLR parse tables Pascal ANI-C COOL LR(0)/RNLR(0) 768/ / ,506/167,973 LR(1)/RNLR(1) 1/242 88/88 65,913/73,003 LALR/RNLALR 1/233 75/75 LR(1)/RNLR(1) 2/ /421 Table 3 The size of G and PPF strctres RNGLR/Farshi RNGLR RNGLR/Farshi RNGLR G edges G edges PPF edges PPF edges Pascal LR(0) 31,015 34,441 35,350 38,770 Pascal LR(1) 21,258 23,826 23,325 25,891 Pascal LALR 21,258 23,826 23,325 25,891 Pascal LR(1) 21,135 23,655 23,050 25,572 C LR(0) 39,389 41,748 40,454 42,800 C LR(1) 28,604 30,670 29,033 31,092 C LALR 28,476 30,542 28,906 30,956 C LR(1) 28,477 30,512 28,761 30,797 COOL LR(0) 23,002 26,461 24,792 29,342 COOL LR(1) 13,512 14,517 12,204 13,167 We consider first the size of the parse tables reqired for or three large grammars. For a given combination of grammar and LR NFA the deterministic handle finding atomata, and ths the tables, will be the same size for both RN- and conventional Knth-style redctions. In general the RN tables will contain far more conflicts, which might be expected to generate more searching dring G constrction. It trns ot that the RNGLR algorithm short-circits these extra searches, as we shall see. Table 1 shows the sizes of the tables given as the nmber of rows (states) by the nmber of colmns (grammar symbols), and Table 2 shows the nmber of table cells with conflicts (mltiple entries). As an aside, the LR(1) tables sometimes show more conflict cells than the LR(1). This is becase the LR(1) DFA contains mltiple copies of a given LR(1) DFA state, corresponding to different local follow sets. The tables for ANI-C show a perhaps nexpectedly high nmber of conflicts for a grammar that is known to prodce only one conflict with YACC: however that reslt relies on the se of the so-called C lexer hack which maps type names defined by typedef statements to a special token. Or figres here relate to the grammar sed as-is withot any lexical trickery. COOL has a large alphabet and reqires more than seven times as many states as ANI-C even for LR(0) and LR(1) tables. In fact, or LR(1) table generator ran ot of memory when processing COOL, so we leave those entries empty. We can also see that this COOL grammar is highly nondeterministic, reflecting the constrction process described in [18]. We now consider the size of the G and PPF strctres. The Farshi and RNGLR algorithms generate the same strctres. The binary RNGLR algorithm achieves cbic rn times bt at the cost of a worst-case constant factor increase in the size of the strctres. That is, the asymptotic space reqirements of RNGLR and RNGLR algorithms are the same, bt for any grammar with prodctions greater than two symbols long the RNGLR algorithm introdces additional nodes into both the G and PPF. Table 3 shows expansions of between 5 and 20% in the size of the strctres.

15 242 A. Johnstone et al. / cience of Compter Programming 61 (2006) Table 4 G size for pathological grammars Γ 1, a 20 Γ 2, a 20 Γ 3, a 20 Γ 1, a 1000 Γ 2, a 1000 Γ 3, a 1000 LR(0) , , ,496 LR(1) , , ,496 LR(1) ,004 2, ,490 Table 5 Performance G edge visits Farshi-naïve Farshi-opt RNGLR RNGLR Pascal LR(0) 41,460 38,964 8,556 8,550 Pascal LR(1) 25,753 24,100 5,665 5,663 Pascal LALR 25,753 24,100 5,665 5,663 Pascal LR(1) 23,305 22,459 5,572 5,570 C LR(0) 42,707 42,251 5,184 5,180 C LR(1) 30,235 29,940 4,502 4,498 C LALR 30,096 29,801 4,502 4,498 C LR(1) 30,754 30,484 4,450 4,446 COOL LR(0) 139, ,120 10,056 9,554 COOL LR(1) 47,464 38,984 3,581 3,487 Γ 1 LR(0) 334,831, ,664, , ,500 Γ 1 LR(1) 1,001, , Γ 1 LR(1) 1,000, , Γ 2 LR(0) 334,832, ,665, , ,500 Γ 2 LR(1) 334,831, ,664, , ,500 Γ 2 LR(1) 1,000, , Γ 3 LR(0) 335,831, ,165, , ,499 Γ 3 LR(1) 333,829, ,161, , ,502 Γ 3 LR(1) 332,833, ,662, , ,502 As we apply stronger parsing techniqes, there is a potential trade-off between the amont of nondeterminism in the table and the size of the G. ome early reports from the natral langage processing commnity sggest that LR(1) based Gs wold be mch larger than LR(1) ones becase the nmber of states is so mch larger, and in the limit the nmber of nodes in a G is bonded by the prodct of the nmber of table states and the length of the string. However, not all states will be poplated, and or figres show that in practice LR(1) Gs are a little smaller than their LR(1) eqivalents. Of corse, the LR(1) tables themselves are sally mch bigger than LR(1) tables (by a factor of 4.6 for ANI-C) so the rather small redction in G size might only be jstified for very long strings. AF+DF ses LR(1) tables. However, relatively simple grammars can show pathological behaviors. Table 4 shows the different sizes of G for the three types of DFA on the grammars Γ 1, Γ 2 and Γ 3 for two inpt strings a 20 and a The figres are the nmber of edges in the G generated by the RNGLR and Farshi algorithms, and we see that while for Γ 1 the LR(1) DFA is comparable with the LR(1) DFA, for Γ 2 the LR(1) DFA is better, and for Γ 3 the LR(0) DFA based parser is comparable with the other two. Now we trn to performance. As we have already discssed in ection 3, the asymptotic time order of the algorithms is dominated by the search time in the G associated with redctions. We cont the nmber of times each edge is visited dring a search, and present here the sm over all these individal edge conts, which abstracts the total amont of searching performed in each rn. Table 5 presents figres for for algorithms: RNGLR, RNGLR and two variants of Farshi s algorithm described in ection 3.4. The figres for Γ 1, Γ 2 and Γ 3 are for the string a 1000.