Towards efficient, typed LR parsers

Transcription

1 roduction An automaton An ML implementation Beyond ML Conclusion 1 rançois Pottier and Yann Régis-Gianas June 2005 rançois Pottier and Yann Régis-Gianas

2 roduction An automaton An ML implementation Beyond ML Conclusion 2 roduction An automaton An ML implementation Beyond ML Conclusion rançois Pottier and Yann Régis-Gianas

3 roduction An automaton An ML implementation Beyond ML Conclusion 3 In short his talk is meant to illustrate how an expressive type system allows guaranteeing the safety of complex programs. he programs considered here are LR parsers and the type system is an extension of ML with generalized algebraic data types GADs). rançois Pottier and Yann Régis-Gianas

4 roduction An automaton An ML implementation Beyond ML Conclusion 4 LR parsers People like to specify a parser as a context-free grammar, typically in BN format, decorated with semantic actions. People like to implement a parser as a deterministic pushdown automaton DPDA). A grammar is LR if such an implementation is possible. rançois Pottier and Yann Régis-Gianas

5 roduction An automaton An ML implementation Beyond ML Conclusion 5 LR parser generators here are tools that generate, out of an LR grammar, a program that simulates execution of the corresponding automaton. Can one guarantee the safety of the generated program without requiring trust in the tool s correctness? rançois Pottier and Yann Régis-Gianas

6 roduction An automaton An ML implementation Beyond ML Conclusion 6 What do existing tools produce? Yacc, Bison, etc. produce C programs, with no safety guarantee. hey use a union to represent semantic values, and do not protect against stack underflow. ML-Yacc or Happy produce ML or Haskell programs, which are typed. Yet, runtime exceptions still arise when pattern matching fails, so safety isn t quite guaranteed. urthermore, redundant dynamic tests incur a runtime penalty. Before showing any code, let s have a look at a sample grammar and automaton. rançois Pottier and Yann Régis-Gianas

7 roduction An automaton An ML implementation Beyond ML Conclusion 7 roduction An automaton An ML implementation Beyond ML Conclusion rançois Pottier and Yann Régis-Gianas

8 roduction An automaton An ML implementation Beyond ML Conclusion 8 A simple grammar Here is a very simple LR grammar, drawn from the Dragon Book: 1) {x} {y} {x y} 2) {x} {x} 3) {x} {y} {x y} 4) {x} {x} 5) {x} ) {x} 6) int{x} {x} he terminals or tokens are,,, ), and int. he non-terminals are,, and. he first four have no semantic value ; the last four have an integer semantic value. rançois Pottier and Yann Régis-Gianas

9 roduction An automaton An ML implementation Beyond ML Conclusion 9 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Here is a pushdown automaton that accepts this grammar. rançois Pottier and Yann Régis-Gianas

10 roduction An automaton An ML implementation Beyond ML Conclusion 10 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input int ) $ Stack State Next action ɛ S 1 shift S 4 rançois Pottier and Yann Régis-Gianas

11 roduction An automaton An ML implementation Beyond ML Conclusion 11 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input int ) $ Stack State Next action S 1 S 4 shift S 10 rançois Pottier and Yann Régis-Gianas

12 roduction An automaton An ML implementation Beyond ML Conclusion 12 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input ) $ Stack S 1 S 4 int State S 10 Next action reduce int, goto rançois Pottier and Yann Régis-Gianas

13 roduction An automaton An ML implementation Beyond ML Conclusion 13 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input ) $ Stack S 1 S 4 State Next action goto rançois Pottier and Yann Régis-Gianas

14 roduction An automaton An ML implementation Beyond ML Conclusion 14 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input ) $ Stack S 1 S 4 State S 5 Next action reduce, goto rançois Pottier and Yann Régis-Gianas

15 roduction An automaton An ML implementation Beyond ML Conclusion 15 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input ) $ Stack S 1 S 4 State S 6 Next action reduce, goto rançois Pottier and Yann Régis-Gianas

16 roduction An automaton An ML implementation Beyond ML Conclusion 16 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input ) $ Stack State Next action S 1 S 4 S 11 shift S 12 rançois Pottier and Yann Régis-Gianas

17 roduction An automaton An ML implementation Beyond ML Conclusion 17 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input $ Stack S 1 S 4 S 11 ) State S 12 Next action reduce ), goto rançois Pottier and Yann Régis-Gianas

18 roduction An automaton An ML implementation Beyond ML Conclusion 18 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input $ Stack S 1 State S 5 Next action reduce, goto rançois Pottier and Yann Régis-Gianas

19 roduction An automaton An ML implementation Beyond ML Conclusion 19 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input $ Stack S 1 State S 6 Next action reduce, goto rançois Pottier and Yann Régis-Gianas

20 roduction An automaton An ML implementation Beyond ML Conclusion 20 S1 S11 S2 ) S12 S3 S7 S8 S4 S9 S6 S5 S10 <)> :. Input $ Stack S 1 State S 2 Next action accept rançois Pottier and Yann Régis-Gianas

21 roduction An automaton An ML implementation Beyond ML Conclusion 21 roduction An automaton An ML implementation Beyond ML Conclusion rançois Pottier and Yann Régis-Gianas

22 roduction An automaton An ML implementation Beyond ML Conclusion 22 Lexer interface okens are made up of a tag and possibly of a semantic value: type token = KPlus KStar KLeft KRight Knd K of int he lexer provides two functions for looking up and for discarding the current token: val peek : unit token val discard : unit unit rançois Pottier and Yann Régis-Gianas

23 roduction An automaton An ML implementation Beyond ML Conclusion 23 Data structures he type of states is easily defined: type state = S0 S1... S11 rançois Pottier and Yann Régis-Gianas

24 roduction An automaton An ML implementation Beyond ML Conclusion 24 Data structures cont d) he stack is made up of pairs of a state and a semantic value whose type depends on the non-terminal with which it is associated. his is a linked list of tagged cells. type stack = Smpty SPlus of stack state SStar of stack state SLeft of stack state SRight of stack state S of stack state int S of stack state int S of stack state int S of stack state int rançois Pottier and Yann Régis-Gianas

25 roduction An automaton An ML implementation Beyond ML Conclusion 25 Implementation general structure) he automaton is simulated by run. Out of the current state, stack, and implicitly) token stream, this function either produces a semantic value for the entire parse or fails. let rec run s : state) stack : stack) : int = match s, peek) with... shift or reduce transitions ), raise Syntaxrror rançois Pottier and Yann Régis-Gianas

26 roduction An automaton An ML implementation Beyond ML Conclusion 26 Implementation shift) A shift transition pushes the current state and the semantic value for the current token onto the stack, discards the current token, and changes the current state: let rec run s : state) stack : stack) : int = match s, peek) with... S9, KStar shift S 7 ) discard ); run S7 SStar stack, S9))... rançois Pottier and Yann Régis-Gianas

27 roduction An automaton An ML implementation Beyond ML Conclusion 27 Implementation reduce) A reduce transition pops a number of semantic values off the stack and exploits them to compute a new one, which is pushed back onto the stack. let rec run s : state) stack : stack) : int = match s, peek) with... S9, KPlus reduce {x} {y} {x y} ) let S SPlus S stack, s, x ), ),, y) = stack in let stack = S stack, s, x y) in goto s stack goto )... Observe that pattern matching is nonexhaustive. rançois Pottier and Yann Régis-Gianas

28 roduction An automaton An ML implementation Beyond ML Conclusion 28 Implementation end) A goto transition examines the state that was popped off the stack during reduction and changes the current state. and goto s : state) : stack int = match s with S0 run S1 S4 run S8 Again, pattern matching is nonexhaustive. rançois Pottier and Yann Régis-Gianas

29 roduction An automaton An ML implementation Beyond ML Conclusion 29 In short his program is considered well-typed by an ML compiler. Yet, the compiler warns about nonexhaustive pattern matching, which means that the absence of runtime failures is not guaranteed. he problem is to modify the program so that every pattern matching becomes exhaustive. Suppressing redundant dynamic tests will lead to a safety guarantee as well as better efficiency. rançois Pottier and Yann Régis-Gianas

30 roduction An automaton An ML implementation Beyond ML Conclusion 30 roduction An automaton An ML implementation Beyond ML Conclusion rançois Pottier and Yann Régis-Gianas

31 roduction An automaton An ML implementation Beyond ML Conclusion 31 Why are these tests redundant? he dynamic tests performed during the previous reduce transition are redundant because, when the automaton is in state S 9, the stack must be of the form...??? he dynamic tests performed during the previous goto transition are redundant because, when the automaton is in state S 9, the stack must be of the form... S 0 S 4 )????? rançois Pottier and Yann Régis-Gianas

32 roduction An automaton An ML implementation Beyond ML Conclusion 32 he invariant fragment) In fact, one can prove that, when the automaton is in state S 9, the stack must be of the form... S 0 S 4 ) S 1 S 8 ) S 6 More generally, knowledge of the current state determines a suffix of the stack rançois Pottier and Yann Régis-Gianas

33 roduction An automaton An ML implementation Beyond ML Conclusion 33 he full invariant Stack State ɛ S 0 ɛ S 0 S 1... S 0 S 4 ) S 2... S 0 S 4 S 6 ) S 3... S 0 S 4 S 6 S 7 ) S 4... S 0 S 4 S 6 S 7 ) int S 5... S 0 S 4 ) S 1 S 8 ) S 6... S 0 S 4 S 6 ) S 2 S 9 ) S 7... S 0 S 4 S 6 S 7 ) S 4 S 8... S 0 S 4 ) S 1 S 8 ) S 6 S 9... S 0 S 4 S 6 ) S 2 S 9 ) S 7 S S 0 S 4 S 6 S 7 ) S 4 S 8 ) S 11 rançois Pottier and Yann Régis-Gianas

34 roduction An automaton An ML implementation Beyond ML Conclusion 34 owards more precise types It is easy to manually prove, by structural induction over a run of the automaton, that the invariant is sound. or this invariant to be exploited by the compiler, it has to be explicitly provided and mechanically verified. he programming language must come with a type system that is sufficiently expressive to allow encoding the invariant. rançois Pottier and Yann Régis-Gianas

35 roduction An automaton An ML implementation Beyond ML Conclusion 35 he idea On must tell the compiler about the correlation between the current state and the structure of the stack. o this end, one parameterizes the type state with a type variable α. he idea is, if the current state has type α state, then the current stack has type α. rançois Pottier and Yann Régis-Gianas

36 roduction An automaton An ML implementation Beyond ML Conclusion 36 he structure of stacks he type stack disappears. he structure of stacks is defined by a family of parameterized types, which are independent of one another: type empty = Smpty type α cellplus = SPlus of α α state type α cellstar = SStar of α α state type α cellleft = SLeft of α α state type α cellright = SRight of α α state type α cell = S of α α state int type α cell = S of α α state int type α cell = S of α α state int type α cell = S of α α state int Compare to the original definition.) rançois Pottier and Yann Régis-Gianas

37 roduction An automaton An ML implementation Beyond ML Conclusion 37 ncoding the invariant fragment) he fact that, when the automaton is in state S 9, the stack must be of the form...???, is encoded by assigning the data constructor S9 the type and similarly for other states. α.α c cp c state Such a declaration is impossible in ML! he type state is a generalized algebraic data type GAD). rançois Pottier and Yann Régis-Gianas

38 roduction An automaton An ML implementation Beyond ML Conclusion 38 he structure of states type state : where S0 : empty state S1 : empty c state S2 : α.α c state S3 : α.α c state S4 : α.α cl state S5 : α.α ci state S6 : α.α c cp state S7 : α.α c cs state S8 : α.α cl c state S9 : α.α c cp c state S10 : α.α c cs c state S11 : α.α cl c cr state rançois Pottier and Yann Régis-Gianas

39 roduction An automaton An ML implementation Beyond ML Conclusion 39 Implementation general structure) he type of run changes: it now accepts an arbitrary state and a stack whose structure is consistent with respect to that state. let rec run : α.α state α int = fun s stack match s, peek) with..., raise Syntaxrror Compare to the original type.) rançois Pottier and Yann Régis-Gianas

40 roduction An automaton An ML implementation Beyond ML Conclusion 40 Implementation shift) he code for shift transitions is unchanged, but typechecking becomes more subtle. let rec run : α.α state α int = fun s stack match s, peek) with S9, KStar SStar stack, S9) has type α cs ) run S7 has type γ.γ c cs int ) urthermore, α = β c cp c, for an unknown β ) hus α cs = γ c cs, where γ = β c cp ) discard ); run S7 SStar stack, S9)) Consult the definition of the type of states.) rançois Pottier and Yann Régis-Gianas

41 roduction An automaton An ML implementation Beyond ML Conclusion 41 Implementation reduce) he code for reduce transitions is also unchanged, but pattern matching is now exhaustive. let rec run : α.α state α int = fun s stack match s, peek) with S9, KPlus α = β c cp c, for an unknown β ) hus stack : β c cp c ) let S SPlus S stack, s, x ), ),, y) = stack in stack : β, s : β state, x : int, y : int ) let stack = S stack, s, x y) in stack : β c ) goto s stack rançois Pottier and Yann Régis-Gianas

42 roduction An automaton An ML implementation Beyond ML Conclusion 42 Implementation end) he type ascribed to goto states that at the top of the stack is a cell associated with the non-terminal and that the remainder of the stack must be consistent with state s. and goto : α.α state α c int = fun s match s with S0 run S1 S4 run S8 has type β cl c int, for every β ) urthermore, α = β cl, for an unknown β ) run S8 Here, pattern matching remains nonexhaustive.) rançois Pottier and Yann Régis-Gianas

43 roduction An automaton An ML implementation Beyond ML Conclusion 43 In short We have encoded part of the invariant into data type declarations and into the types ascribed to run and goto. In fact, the whole invariant can be encoded. hen, typechecking involves proving the invariant. Pattern matching provides type equations with local scope. Shared type variables allow coordinating data structures. All this is typical of GADs. rançois Pottier and Yann Régis-Gianas

44 roduction An automaton An ML implementation Beyond ML Conclusion 44 roduction An automaton An ML implementation Beyond ML Conclusion rançois Pottier and Yann Régis-Gianas

45 roduction An automaton An ML implementation Beyond ML Conclusion 45 Results We have obtained a safety guarantee about the generated parser, without requiring trust in the generator. he tool that produces the automaton knows the invariant, or thinks it knows, and produces appropriate data type declarations without difficulty. If the tool produces an incorrect program, the latter is rejected by the compiler. rusting the compiler remains necessary, unless of course a certifying compiler is used. rançois Pottier and Yann Régis-Gianas

46 roduction An automaton An ML implementation Beyond ML Conclusion 46 owards more proofs in programs We have exploited a very expressive type system to prove the safety of a program. Proof assistants have allowed this, and more, for a long time. Here, however, we have remained within the framework of a programming language equipped, in particular, with a powerful type inference mechanism and with an extremely efficient compilation scheme. Narrowing the gap between programming and proving is probably a worthy long-term?) research goal. rançois Pottier and Yann Régis-Gianas

47 roduction An automaton An ML implementation Beyond ML Conclusion 47 References Slides, draft paper, and prototype implementations of the typechecker and parser generator are available online: http: // cristal. inria. fr/ ~fpottier/ http: // cristal. inria. fr/ ~regisgia/ rançois Pottier and Yann Régis-Gianas