RE2C { A More Versatile Scanner Generator. Peter Bumbulis Donald D. Cowan. University of Waterloo. April 15, Abstract

Transcription

1 RE2C { A More Versatile Scanner Generator Peter Bumbulis Donald D. Cowan Computer Science Department and Computer Systems Group University o Waterloo April 15, 1994 Abstract It is usually claimed that lexical analysis routines are still coded by hand, despite the widespread availability o scanner enerators, or eciency reasons. While eciency is a consideration, there exist reely available scanner enerators such as GLA [7] that can enerate scanners that are aster than most hand-coded ones. However, most enerated scanners are tailored or a particular environment, and retarettin these scanners to other environments, i possible, is usually complex enouh to make a hand-coded scanner more appealin. In this paper we describe RE2C, a scanner enerator that not only enerates scanners which are aster (and usually smaller) than those produced by any other scanner enerator known to the authors, includin GLA, but also adapt easily to any environment. Cateories and Subject Descriptors: D.3.2 [Prorammin Lanuaes]: Lanuae Classications { specialized application lanuaes; D.3.4 [Prorammin Lanuaes]: Processors General Terms: Alorithms, Lanuaes, Perormance Additional Key Words and Phrases: Lexical analysis, scanner enerator 1 Introduction Lexical analysis routines are still oten coded by hand despite the widespread availability o scanner enerators. For example, while most Unix systems have a scanner enerator installed (typically LEX [15] or ex [16]), ew Unix applications use a mechanically enerated scanner. One commonly cited reason or not usin LEX-enerated scanners is perormance: they can be 10 times slower than equivalent hand-coded scanners [13]. As a result, there has been considerable research into improvin the perormance o mechanically enerated scanners [16, 7, 9]. GLA [7], one such scanner enerator, can produce scanners that are aster than most hand-coded scanners. However, the use o hand-coded scanners is still prevalent. One possibility is that this is due to the diculty o adaptin the enerated scanners to specic applications. Most scanner enerators are tailored to a particular environment. In act, the trend in recent years has been to interate scanner enerators with compiler toolkits. For example, GLA is part o the Eli compiler construction system [8], and Rex [9] is part o the GMD Toolbox or Compiler Construction 1. Scanners Permission to copy without ee all or part o this material is ranted provided that the copies are not made or distributed or direct commercial advantae, the ACM copyriht notice and the title o the publication and its date appear, and notice is iven that copyin is by permission o the Association or Computin Machinery. To copy otherwise, or to republish, requires a ee and/or specic permission. Copyriht 1994 by the Association or Computin Machinery, Inc. To appear in LOPLAS 2(1{4). 1 Also known as Cocktail (Compiler-Compiler-Toolbox Karlsruhe). 1

2 enerated by these tools assume the existence o a library o support modules or error handlin, input buerin, symbol table manaement, and similar unctions. While these support modules simpliy the task o implementin a compiler or interpreter, they make adaptation to other purposes more dicult. Adaptation to other environments is also made more dicult because oten assumptions are made about the input and restrictions are placed on tokens in order to achieve better perormance. RE2C oes to the other extreme: it concentrates solely on eneratin code or matchin reular expressions. RE2C is successul at its task: not only does it produce scanners which are aster than those created by other scanner enerators but, surprisinly, they are usually smaller as well. Further, RE2C does not make any assumptions about the input or place any restrictions on tokens. To a lare deree, the perormance and exibility o RE2C-enerated scanners is due to a novel method or determinin when to rell a buer which avoids the complications introduced by the sentinel method [1]. The ollowin sections o this paper describe RE2C scanner specications, discuss how these specications are converted into scanners, and ive perormance results achieved by our implementation (includin a comparison with GLA). 2 Scanner Specications An RE2C source le consists o C[14] or C++[4] 2 code interleaved with comments o the orm containin scanner specications. These specications are replaced with enerated code that is invoked simply by \allin into" the comments as illustrated in Fiure 1 and in Appendix A 3. Fiure 1: A simple scanner. A scanner specication takes the orm o a list o rules, each rule consistin o a reular expression [10] and an action expressed in executable code. Fiure 2 illustrates a trivial RE2C scanner specication that will be used as an example throuhout this paper. Each call to the code enerated rom a specication will Fiure 2: Sample specication. a b matches any character between a and b, inclusively. The last rule, or example, will match any eiht bit character. Rules are listed in order o precedence. rst determine the lonest possible prex o the remainin input that matches one o the reular expressions and will then execute the action in the rst applicable rule. 2 Retarettin RE2C to a dierent lanuae is straihtorward. 3 RE2C-enerated scanners require no additional support code. 2

3 [a-z] [\000-\377]\ [0-9a-z] [a-z]\p [a-z]\r [a-z]\i [a-z]\n [a-z]\t [a-z] p 4 r 4 i 4 n 4 t 5 0 [1-9] [0-9] x 7 8 [0-9] 3 3 [0-9a-] 9 [ a-] 2 Fiure 3: A DFA or the sample specication in Fiure 2. State 0 is the start state. Acceptin states are labeled with the number o the rule that they match. For example, state 10 accepts rule 2. Transitions dierin only by label are represented with the same arc. For example, state 0 has transitions to state 6 on all o the ollowin characters:,,,,. RE2C is dierent rom most other scanner enerators in that the user must provide the input buerin mechanism or the scanner; the enerated code simply assumes that the user has dened three pointers:, and, and a routine n. Beore executin the enerated code, and must be set to point to the rst and one past the last character in the buer, respectively. Ater a token is reconized, and beore any action is executed, is set to point to just past the token. will be called as the buer needs llin; at least n additional input characters should be provided. When is called, will point to the next character to be scanned and, i set, will point to a possible backtrackin point in the buer. must update, and possibly and beore returnin. Typically,,, and n will be dened as macros. 2.1 Thins That RE2C Doesn't Provide RE2C doesn't provide many thins available in more conventional scanner enerators includin deault rules, end-o-input pseudo-tokens, and buer manaement routines. All o these must be supplied by the user. Rather than bein a handicap, this allows RE2C-enerated scanners to be tailored to almost any environment. For example, the scanner dened in Fiure 1 compiles into 32 bytes o i486 code (usin Watcom C 9.5); the same size as an equivalent hand-coded routine. Most other scanner enerators cannot produce scanners that are competitive with hand-coded analyzers in this case. Further, it is not overly dicult to implement a more traditional scanner usin RE2C. For example, Appendix A contains the support code or the C scanner benchmarked in Table 1. Note that this code allows or arbitrarily lon contiuous tokens and provides line and column number inormation. 3

4 3 Generatin Directly Executable Scanners As demonstrated by GLA [7] eneratin directly executable code instead o tables can result in much aster scanners. However, to achieve this speed, GLA-enerated scanners make some assumptions about the input and place certain restrictions on tokens 4. In this section we will show how to enerate directly executable scanners which not only avoid such restrictions, but are also aster and usually smaller. The approach presented here has the added benet that even aster scanners can be easily be created, at the expense o increased code size, by usin a technique akin to loop unrollin. 3.1 Constructin a DFA The rst step in eneratin a directly executable scanner is to construct a DFA that reconizes the reular expressions in the specication. Fiure 3 presents a DFA that reconizes the reular expressions in Fiure 2. One possible alorithm or constructin such a DFA can be ound in [1]. Given such a DFA, the task o 4 These assumptions and restrictions are discussed in more detail in Sections and

5 scannin the input can be expressed as ollows: Startin rom the start state, move rom state to state alon transitions labeled with consecutive characters rom the input. When no urther transitions can be made, backtrack to the last acceptin state, say q. The path to q spells the next token and the rule associated with q determines the code to be executed. As a result, the problem o eneratin scanners essentially reduces to the problem o eneratin an executable representation or a DFA. 3.2 Generatin Code I we assume that the input is entirely contained in a sinle buer then eneratin code or the DFA is relatively straihtorward, as is illustrated by the code templates in Fiure 4. Note that the only dierence Proloue start q Code or acceptin state n n action(n) rule(q) code or states q Code or non-acceptin state q c oto(q; c) q c oto(q; c) Fiure 4: Directly executable scanner. The code enerated or a scanner consists o a proloue ollowed by code or each state. start is the start state. action(n) denotes the code associated with rule n, oto(q; c) denotes the state reached rom state q alon the transition labeled with c and rule(q) denotes the rule associated with state q. is used to save backtrackin inormation. The -labels will be used in section between the templates or acceptin and non-acceptin states is that the acceptin states have additional code to save backtrackin inormation. Fiure 5 shows code that miht be enerated or state 1 in Fiure Buerin Complications arise when the input is not contained in a sinle buer: additional code is needed or llin the buer as necessary. 5

6 Fiure 5: Code or state 1. Fiure 6: Code or state The Sentinel Method Most scanner enerators use the sentinel method [1] to determine when the buer needs llin. In the simplest case, a symbol that does not appear in valid input is chosen as the sentinel character. An extra state is added to the DFA and transitions to this state on the sentinel symbol are added to the oriinal states. When the DFA arrives in this new state it is time to rell the buer. Ater the buer is relled, scannin must be restarted in the previous state. Unortunately, this is not possible with the approach outlined in Fiure 4: the necessary inormation is simply not available. Code could be added to each state to save the necessary inormation but this would result in slower and larer scanners. GLA solves this problem by ensurin that the sentinel only ets inserted between tokens: i this is the case, the scanner can always be restarted rom the start state. To ensure that the sentinel only ets inserted between tokens, GLA allows newline (ASCII LF) characters to appear only at the end o a token and disallows the buerin o partial lines Buerin RE2C-enerated scanners check i the buer needs llin simply by comparin and. A method inspired by the mechanism used to uard aainst stack overow in [17] 6 is used to reduce the amount o checkin. Checks are only inserted in certain key states. These checks simply ensure that there is enouh input in the buer or the scan to proceed until the next key state. For example, in the DFA o Fiure 3 it is sucient to check that there are at least 6 characters in the buer when it starts, and that there is at least one character in the buer when the DFA is in states 6, 8, or 10. No other checks are required. The checks inserted in key states are o the orm 6

7 n n where n is the maximum number o characters that can be consumed beore another key state is reached. For example, Fiure 6 shows the code enerated or state 6 in Fiure 3. A set o key states can be determined by discoverin the stronly-connected components (SCCs) o the DFA. An SCC is a maximal subset o states such that there exists a path rom any state in the subset to any other. The set o key states consists o all o the states in non-trivial SCCs, toether with the start state. Note that or each SCC S, we actually only have to include a subset o states o S such that when the subset is removed, S becomes acyclic. Indeed, [17] describes a simple heuristic or choosin such a subset. However, since in practice most o the (non-trivial) SCCs encountered will consist o a sinle state the current version o RE2C simply includes all states in non-trivial SCCs 7. An alorithm iven in [3] was used to compute the SCCs. 3.4 Optimizations Even ood optimizin C compilers can be coaxed into eneratin much smaller and slihtly aster code i some transormations are rst applied to the enerated code Eliminatin Backtrackin Consider state 1 in the DFA in Fiure 3. Note that since all o the transitions rom state 1 reach only acceptin states, backtrackin inormation does not need to be saved i the code or the case is chaned to o directly to the code associated with state 1. The result o this optimization is shown in Fiure 7. More enerally, this optimization can be applied to all acceptin states which have transitions only to acceptin states Optimizin es Fiure 7: Code or state 1 with backtrackin eliminated. Most C compilers will enerate either a jump table or a set o statements or a statement dependin on the distribution o the labels. In many compilers the decision as to which method to use is biased towards eneratin jump tables since in most cases this results in aster albeit larer code. However, experience with directly executable scanners has shown, that replacin many o these jump tables 5 I the input contains no newlines, a GLA scanner will attempt to buer the entire input stream. 6 The problem o detectin stack overow in LR parsers is probably best let to hardware mechanisms [12]. 7 It should be noted that ndin the minimal set o states to remove rom an SCC in order to render it acyclic is equivalent to the FEEDBACK VERTEX SET problem which is NP-complete [6]. 7

8 with statements results in scanners which are much smaller, and surprisinly, in some cases slihtly aster as well 8. As a result, the capability o replacin a statement with statements was added to RE2C. RE2C bases its decision on whether to enerate a statement or to replace it with s solely on the density 9 o the statement. It is surprisin that such a simple heuristic works well. For more esoteric applications in which the input alphabet is not a simple interval RE2C has the advantae in that there is no provision or don't care entries in a statement: i no matches none o the statements in the must be executed. However, or the examples in Table 1 this is not so: RE2C simply does a better job o eneratin code or statements than the compiler. [18], [11], and [2] also address the problem o eneratin ood code or statements. Replacin es with s When replacin a statement with statements, it is useul to sort the s by label and then roup them accordin to rule into subranes, as illustrated in Fiure 8. RE2C replaces a with either a linear or binary search, dependin on the number o subranes in the Fiure 8: or state 0.. I there are only a ew subranes a linear search is enerated; otherwise, a binary search is used. Fiure 9 and Fiure 10 show linear and binary searches, respectively, that could be used to replace the Fiure 9: Linear lookup code sequence or state 0. 8 See Table 1 or examples. 9 The number o distinct subranes divided by the total number o s. 8

9 Fiure 10: Binary lookup code sequence or state 0. in Fiure 8. Note in particular the comparison or the \ " in Fiure 9. This optimization eliminates a comparison each time it is applied. Also note that no comparisons are required at the top and bottom o the rane. Simpliyin es As a eneral rule, better replacement code can be enerated or a i it contains ewer subranes. One way o reducin the number o subranes in a, at the expense o some speed, is to locate a base which is very similar and then replace the code or all cases which appear identically in the base with a to (the code enerated or) the base. RE2C uses this optimization to ood advantae when eneratin code in the transitions o states used or matchin keywords. For example, note that the es or states 1 throuh 4 dier rom the o state 6 only on \ ", \ ", \ ", and \ ", respectively. Fiure 11 shows the code enerated or these states. Another way Fiure 11: Code or states 1{4 ater all optimizations. o implementin this optimization is to construct a tunnel automaton [9] rom the DFA, and then enerate code rom the tunnel automaton. Common Subexpression Elimination Many compilers will miss the act that in Fiures 9 and 10 should be loaded into a reister. Most can be coaxed to do so by rst assinin to a local variable. 4 Experimental Results Table 1 compares two RE2C-enerated C scanners with the (hand-coded) lcc scanner [5] and comparable GLA- and ex-enerated scanners on a variety o platorms. It reports the times in seconds required by the various scanners to scan about 170,000 lines o C source. The 5,607,820 byte source le used essentially 9

10 time space proram user sys total text data bss total R4000 / cc O ex -Cem ex -C lcc la re2c re2c -s R4000 / cc O -Olimit 5000 ex -Cem ex -C lcc la re2c re2c -s SPARC / cc O ex -Cem ex -C lcc la re2c re2c -s i486 / cc O ex -Cem ex -C lcc la re2c re2c -s / cc1.40 -O ex -Cem ex -C lcc la re2c re2c -s Table 1: Comparison o enerated C scanners. 10

11 consists o 10 copies o the source to James Clark's SGML parser, smls 10. The times reported are averaes or 10 trials; the sizes reported include everythin but C library code 11. ex provides a number o table compression options includin or tables optimized or space, and or tables optimized or speed. By deault, RE2C will use a heuristic to decide i a should be replaced with s: the option orces RE2C to always enerate es. To make comparisons more meaninul, all semantic processin code was removed rom the GLAenerated and lcc scanners, and code to provide line and column number inormation was added to the RE2C specication. The remainin dierences o note between the scanners include: The ex-enerated scanners do not provide line or column number inormation. The GLA-enerated scanner assumes 7-bit input. As a eneral rule, the RE2C-enerated scanners were the astest, ollowed by the GLA-enerated scanner and then the lcc scanner. The ex-enerated scanners were sinicantly slower. Only the space-optimized ex scanner was smaller than the deault RE2C scanner, and only by a narrow marin. There are some architectures, notably the IBM 370, on which table driven scanners will probably produce better results: IBM 370 compilers typically enerate poor code or lare routines. The various scanners and input les used or the tests are available or anonymous tp rom cs.- uwaterloo.ca in /pub/peter/re2c/sampler.tar.z. ex is available or anonymous tp rom tp.uu.net as /packaes/- nu/ex tar.z, GLA is available or anonymous tp rom tp.cs.colorado.edu as part o the Eli packae /pub/cs/distribs/eli/eli3.4.2.tar.z, and the lcc ront end is available or anonymous tp rom princeton.edu as /pub/lcc/lcce-1.9.tar.z. An alpha version o RE2C will soon be made available or anonymous tp rom cs.uwaterloo.ca as /pub/peter/re2c/re2c-0.5.tar.z. 5 Related Work The key to the perormance and exibility o an RE2C-enerated scanner is the approach used to determine when the buer needs llin. Interestinly, the lcc scanner [5] uses a similar approach (with certain concessions to keep the bookkeepin manaeable.) 5.1 Comparison With GLA It is natural to compare RE2C to GLA [7] as it also enerates directly executable scanners. RE2C and GLA have many dierences simply because they are tareted or dierent types o users: GLA is intended or people who simply wish to leverae their eorts with existin tools and libraries; RE2C is intended or people that have more specialized needs and are willin to provide their own support routines. For example, GLA provides a ood buerin mechanism, RE2C users must supply their own. These dierences, however, are not unique to GLA and have been addressed or the most part in previous sections. O more interest is the dierences in the code that RE2C and GLA enerate. Scanners enerated by RE2C and GLA dier primarily in two aspects: how they determine when the buer needs llin, and how they enerate code or es. GLA uses the ASCII NUL character as the sentinel to determine when the buer needs llin. To improve the speed and reduce the size o the enerated scanners GLA buers only complete lines and restricts tokens to those that do not contain newline (ASCII LF) characters 12. I a token with an embedded newline character (such as a comment) is required it must be reconized with an auxiliary scanner written in C. This code has to perorm the buerin-related bookkeepin that is done automatically by GLA-enerated code. 10 Available or anonymous tp rom tp.uu.net as /pub/text-processin/sml/smls-1.1.tar.z. 11 The GLA-enerated scanner sizes also do not include the size o an error reportin module. 12 This is discussed in more detail in Section

12 The mechanism RE2C uses to rell the buer eliminates these restrictions and yet allows RE2C to enerate aster and smaller scanners. RE2C also allows both auxiliary and primary scanners to be specied usin reular expressions. For example, Appendix A contains an auxiliary scanner or comments. Like RE2C, GLA usually replaces es with s. Unlike RE2C, GLA does not use a -based heuristic to decide which es to replace: rather, it always enerates a or the start state and uses s or the rest. GLA replaces es with code sequences o the orm: in S 1 1 in Sn n Bit vectors are used or all membership tests involvin sets with more than one element. As an optimization, i a state has a transition to itsel the test as to whether to remain in the same state or not is perormed rst. For example, Fiure 12 shows the GLA-enerated code or state 8 in Fiure Note the use o Fiure 12: GLA code or state 8 in Fiure element entries or the bit vectors to reduce the scanner size: A GLA-enerated scanner will crash or otherwise behave unpredictably i a non-ascii character appears in the source 14. In some sense the results o Section 4 are a bit misleadin: the GLA specication that was used to obtain the ures in Table 1 is not a typical GLA specication. Usually scanners implemented usin GLA will handle keywords as identiers as GLA has been optimized or this [7]. Table 2 presents a more air comparison: the keyword matchin rules were removed rom both the GLA and RE2C specications. The RE2C-enerated scanners were still aster and smaller except on the MIPS R4000, where the cc-compiled GLA scanner was slihtly aster. Note however, that the RE2C specication can be substantially sped up by usin a technique akin to loop unrollin. Replacin the oriinal keyword matchin rule in the RE2C specication 15 with the ollowin rules 13 Actually, GLA would enerate a statement. Most compilers will enerate the same object code or both. 14 No checks are made to ensure that only 7-bit characters appear in the input. 15 = and =. 12

13 time space proram user sys total text data bss total R4000 / cc O la re2c re2c -s re2c -s y R4000 / cc O -Olimit 5000 la re2c re2c -s SPARC / cc O la re2c re2c -s i486 / cc O la re2c re2c -s / cc1.40 -O la re2c re2c -s Table 2: Scanner perormance with keywords treated as identiers. y specication. uses an \unrolled" 13

14 reduces the number o end-o-buer checks and results in a sinicant speed improvement over the GLAenerated scanner. 6 Summary and Further Work This paper has described RE2C, a tool or creatin lexical analyzers. Unlike other such tools, RE2C concentrates solely on eneratin ecient code or matchin reular expressions. Not only does this sinleness o purpose make RE2C more suitable or a wider variety o applications, it allows it to enerate scanners which approach hand-crated scanners in terms o size and speed. Compared to scanners enerated by ex, and GLA, RE2C-enerated scanners are aster and in many cases smaller as well. While RE2C-enerated scanners perorm well, there is still room or improvement. Near term improvements include usin GLA's bit vectors to simpliy some es and addin a state unrollin operator. In the loner term, inline actions will be added to RE2C. For example, a specication like miht be used to obtain the value o a previously scanned inteer. Typically, these sorts o specications would be used as an action in some other specication. 7 Acknowledments The authors thank the reerees or their many valuable comments and suestions. 14

15 A C Scanner more rules 15

16 Reerences [1] Aho, A. V., Sethi, R., and Ullman, J. D. Compilers: principles, techniques, and tools. Addison- Wesley, Reprinted with corrections. [2] Bernstein, R. L. Producin ood code or the case statement. Sotware{Practice and Experience 15, 10 (October 1985), 1021{1024. [3] DeRemer, F., and Pennello, T. Ecient computation o LALR(1) look-ahead sets. ACM Transactions on Prorammin Lanuaes and Systems 4, 4 (October 1982), 615{649. [4] Ellis, M., and Stroustrup, B. The Annotated C++ Reerence Manual. Addison-Wesley, [5] Fraser, C. W., and Hanson, D. R. A retaretable compiler or ANSI C. SIGPLAN Notices 26, 10 (October 1991), 29{43. [6] Garey, M. R., and Johnson, D. S. Computers and Intractability: A Guide to the Theory o NP- Completeness. W. H. Freeman and Company, [7] Gray, R. W. -GLA - A enerator or lexical analyzers that prorammers can use. USENIX Conerence Proceedins (June 1988), 147{160. [8] Gray, R. W., Heurin, V. P., Levi, S. P., Sloane, A. M., and Waite, W. M. Eli: A complete, exible compiler construction system. Communications o the ACM 35, 2 (February 1992), 121{131. [9] Grosch, J. Ecient eneration o lexical analysers. Sotware{Practice and Experience 19, 11 (1989), 1089{1103. [10] Harrison, M. A. Introduction to Formal Lanuae Theory. Addison-Wesley, [11] Hennessy, J. L., and Mendelsohn, N. Compilation o the Pascal case statement. Sotware{Practice and Experience 12, 9 (September 1982), 879{882. [12] Horspool, R. N., and Whitney, M. Even aster LR parsin. Sotware{Practice and Experience 20, 6 (1990), 515{535. [13] Jacobson, V. Tunin UNIX Lex or it's NOT true what they say about Lex. In USENIX Conerence Proceedins (Washinton, DC, Winter 1987), pp. 163{164. Abstract only. [14] Kernihan, B. W., and Ritchie, D. M. The C Prorammin Lanuae, 2nd Ed. Prentice-Hall, Inc., [15] Lesk, M. E. LEX { a lexical analyzer enerator. Computin Science Technical Report 39, Bell Telephone Laboratories, Murray Hill, NJ, [16] Paxson, V. ex { man paes, In ex tar.z. Available or anonymous tp rom tp.uu.net in /packaes/nu. [17] Pennello, T. J. Very ast LR parsin. In Proceedins o the ACM SIGPLAN'86 Symposium on Compiler Construction (July 1986), ACM. [18] Sale, A. The implementation o case statements in Pascal. Sotware{Practice and Experience 11, 9 (September 1981), 929{