Functional Parsing. Languages for Lunch 10/14/08 James E. Heliotis & Axel T. Schreiner

Transcription

1 Functional Parsing Languages for Lunch 10/14/08 James E. Heliotis & Axel T. Schreiner

2 why monads? execution sequencing failure handling: preempt, catch stateful computation support do notation

3 do notation Haskell x = do name <- value... return... `mplus` do... x = {{{ name <- value; ; expression sequential execution scope for assignment possibly retry? ~ats/projects/jsm/paper.xml

4 binding and typing x = do name <- value... return... `mplus` do... x = {{{ name <- value; continuation alternative ; ((lambda (name) continuation) value) x = value.andthen( name => continuation ).orelse( alternative );

5 bottom line term: number '(' expression ')'; term = do number `mplus` do symbol "(" expression symbol ")" Parser term () { return {{{ pf.pattern(0); // number literal("("); expression(); literal(")"); ;

6 bottom line term: number '(' expression ')'; term = do number `mplus` do symbol "(" expression symbol ")" Parser term () { return {{{ pf.pattern(0); // number literal("("); expression(); literal(")"); ; do notation and mplus can look like grammar. {{{ even more so.

7 bottom line term: number '(' expression ')'; term = do number `mplus` do symbol "(" expression symbol ")" Parser term () { return {{{ pf.pattern(0); // number literal("("); expression(); literal(")"); ; do notation and mplus can look like grammar. {{{ even more so. infrastructure for parsing? port to other languages?

8 the Haskell classes Monad andthen controls sequencing >>= succeed wraps a success return fail wraps a failure fail MonadPlus fail indicates failure orelse controls failure handling mzero mplus

9 mops.parser interface ParserFunction { Result f (Input i); class Result { boolean failed; Object value; Input input; class Parser { Parser (ParserFunction f) {... Result parse (Input input) { return parserfunction.f(input);... Parser is like State combined with Either. Input is the state, Result is Either.

10 mops.parser constructors static Parser succeed (final Object value) { return new Parser( new ParserFunction() { Result f (Input input) { return new Result(value, input); ); static Parser fail (String message) { return new Parser( new ParserFunction() { Result f (Input input) { return new Result(input, message); );

11 mops.parser combinators static Parser succeed (Object value) {... static Parser fail (String message) {... Parser orelse (Parser b) { Parser a = this; return new Parser( new ParserFunction() { Result f (Input input) { Result result = a.parse(input); return result.failed? b.parse(input) : result; );

12 mops.parser combinators static Parser succeed (Object value) {... static Parser fail (String message) {... Parser orelse (Parser b) {... Parser andthen (Scope b) { Parser a = this; return new Parser( new ParserFunction() { Result f (Input input) { Result result = a.parse(input); return result.failed? result : b.f(result.value).parse(result.input); );

13 mops.parser combinators static Parser succeed (Object value) {... static Parser fail (String message) {... Parser orelse (Parser b) {... Parser andthen (Scope b) { Parser a = this; return new Parser( new ParserFunction() { Result f (Input input) { ); public interface Scope { Parser f (Object value); Result result = a.parse(input); return result.failed? result : b.f(result.value).parse(result.input);

14 mops.parser combinators static Parser succeed (Object value) {... static Parser fail (String message) {... Parser orelse (Parser b) {... Parser andthen (Scope b) {... Parser optional (Object value) { return this.orelse(succeed(value));

15 mops.parser combinators static Parser succeed (Object value) {... static Parser fail (String message) {... Parser orelse (Parser b) {... Parser andthen (Scope b) {... Parser optional (Object value) {... Parser some () { Parser self = this; // closure return self.andthen( new Scope() { Parser f (Object fromself) { return self.many().andthen( new Scope() { Parser f (Object frommany) { return succeed(... ); ); );

16 mops.parser combinators static Parser succeed (Object value) {... static Parser fail (String message) {... Parser orelse (Parser b) {... Parser andthen (Scope b) {... Parser optional (Object value) {... Parser some () {... Parser many () { return this.some().orelse(succeed(... ));

17 mops.parser combinators static Parser succeed (Object value) {... static Parser fail (String message) {... Parser orelse (Parser b) {... Parser andthen (Scope b) {... Parser optional (Object value) {... Parser some () {... Parser many () { return this.some().orelse(succeed(... )); can translate EBNF verbatim into Java. can translate to other host languages.

18 A Simple Example A mops-style parser that gets its input tokens from a "stream" of strings in an array no scanner the language to be parsed is balanced parentheses: (), a, (a), (a b),(a (b a) (d)), etc. In EBNF: S ::= S1 $ S1 ::= ( S1* ) id

19 Language Implementations Java Object-oriented / Statically typed ParserFunction and Scope interfaces Input class w/immutable instances Parser class with the EBNF operations succeed, fail orelse, AndThen, optional, some, many

20 Language Implementations Python Dynamically typed, functional style employed Scope concept is a closure. Kept Input class w/immutable instances Non-method parser functions take one or two parser function arguments: succeed, fail orelse, AndThen, optional, some, many Parser functions are closures.

21 Java Example: Parser.some() public Parser some() { final Parser self = this; return self.andthen( new Scope() { public Parser getparser( final Object valuefromself ) { return self.many().andthen( new Scope() public Parser getparser( final Object valuefrommany ) { ArrayList< Object > result = new ArrayList< Object >(); result.add(valuefromself); result.addall( (ArrayList< Object >)valuefrommany ); return succeed( result ); ); );

22 Python Example: some(aparser) def some( primary ): def scope1( value1 ): def scope2( value2 ): result = [] result.append( value1 ) for val in value2: result.append( val ) return succeed( result ) return andthen( many( primary ), scope2 ) return andthen( primary, scope1 )

23 Experiment Conclusions Language issues Having to build a single function by wrapping it inside a class for the OO approach is annoying and degrades readability. It is possible that one could leverage the advantages of OO with instance variables or utility methods inside those classes (TBD). Debugging the dynamically typed parser "function cannot be indexed." "lambda expects 0 arguments; got 1."

24 Experiment Conclusions Basic parsing operations are only built and debugged once in a language Code for specific parsers (like BalParens) is generated static Parser S = {{{ S1; af.eof(); ; static Parser S1 = {{{ af.literal( "(" ); S1.many(); af.literal( ")" ); af.pattern(0); // id ;

25 EBNF sum: product ('+' product '-' product)*; static Parser sum = {{{ product; {{{ af.literal("+"); product; Parser.succeed(...); af.literal("-"); product; Parser.succeed(...);.many(); Parser.succeed(...); ; preprocess using Web Compiler Service.

26 scanner.scanner java scanner.test 0 if iffy whilest while (?:[white space])* skip patterns (([a-za-z_][a-za-z_0-9]*) pattern 0 ([0-9]+) pattern 1... \Qwhile\E \Qreturn\E etc. literal 0 literal \n ($)) single characters, eof (1) literal 14 "if" (2) pattern 0 "iffy" (3) pattern 0 "whilest" (4) literal 17 "while" (5) [eof] lex-like scanner, constructed from regular expressions

27 mops.parserfactory class ParserFactory extends Scanner { Parser literal (String s) {... Parser pattern (int n) {... Parser eof () {... constructed from regular expressions: skips tokens, e.g., number, identifier,... literals creates parsers which succeed with an expected symbol or fail. should be portable to other languages.

28 do notation: mops.mopsparser Java (or JavaScript, or...?) plus {{{ classname... generates tree classes to represent input. {{{ id <- parser ; translated using andthen and orelse. optional, many, some can be suffixed in code.

29 oops3 One!stop Shopping public class parser { %% <void> Comment = '#.*'; <void> WhiteSpace = '[ \r\t\n]+'; <Integer> Number = '[0-9]+'; <Sum>? sum: term (add subtract)*; <Add> add: '+' term; <Subtract> subtract: '-' term; term: Number '(' sum ')'; %% oops3 oops3 optionally generates a functional parser. javac # input 3 - (4 + 5) parser 2 Sum Integer 3 2 Subtract 2 Sum Integer 4 2 Add Integer 5

30 references chapter 8 in Graham Hutton's Haskell book parsec pysec Monadic Parsing using JavaScript oops3 / mops / scanner on the Web Compiler Service Programming Skills example