Bottom-Up Syntax Anaysis

Transcription

1 Implementation of Parsers Bottom-Up Syntax Anaysis Educational Objectives: General Principles of LR(k) Analysis Resolving Conflicts in Parser Generation Connection between CFGs and push-down automata Ina Schaefer Context-Free Analysis 59

2 Implementation of Parsers Basic Ideas: Bottom-Up Analysis is more powerful than top-down analysis, since production is chosen at the end of the analysis while in top-down analysis the production is selected up front. LR: Read input from left (L) and search for right derivations (R) Ina Schaefer Context-Free Analysis 60

3 Implementation of Parsers Principles of LR Parsing 1. Reduce from sentence to axiom 2. Construct sentential forms from prefixes in (N T ) and input rests in T. Prefixes are right sentential forms of grammar. Such prefixes are called viable prefixes. This prefix property has to hold invariantly to avoid dead ends. 3. Reductions are always made at the left-most possible position. Ina Schaefer Context-Free Analysis 61

4 Viable Prefix Implementation of Parsers Definition Let S rm βau rm βαu a right sentential form of Γ. Then α is called handle or redex of the right sentential for βαu. Each prefix of βα is a viable prefix of Γ. Ina Schaefer Context-Free Analysis 62

5 Implementation of Parsers Regularity of Viable Prefixes Theorem The language of viable prefixes of a grammar Γ is regular. Proof. Cf. Wilhelm, Maurer Thm and Corrollary (pp. 361, 362), Essential proof steps are illustrated in the following by construction of LR DFA(Γ). Ina Schaefer Context-Free Analysis 63

6 Implementation of Parsers LR Parsing: Examples Consider Γ 1 S acd C b D a b Analysis of aba can lead to an dead end. (cf. Lecture). Considering viable prefixes can avoid this. Ina Schaefer Context-Free Analysis 64

7 Implementation of Parsers LR Parsing: Examples (2) Consider Γ 2 S E# E a (E) EE Analysis of ( ( a ) ) ( a) # (cf. Lecture) Stack can manage prefixes already read. Ina Schaefer Context-Free Analysis 65

8 Implementation of Parsers LR Parsing: Examples (3) Consider Γ 3 S E# E E + T T T ID Analysis of ID + ID + ID # (cf. Lecture) Ina Schaefer Context-Free Analysis 66

9 Implementation of Parsers LR Parsing: Shift and Reduce Schematic syntax tree for input xay with a T, x, y T and start symbol S:! x a y Read Lesezeiger Pointer Ina Schaefer Context-Free Analysis 67

10 Implementation of Parsers LR Parsing: Shift and Reduce (2) Shift step: Reduce step: "$!a =>! = "# x a y Lesezeiger Read Pointer x a y Lesezeiger Read Pointer Ina Schaefer Context-Free Analysis 68

11 Implementation of Parsers LR Parsing: Shift and Reduce (3) Main Problems: Reductions can only be performed if remaining prefix is still a viable prefix. When to shift? When to reduce? Which production to use? Solution: For each grammar Γ construct LR DFA(Γ) automaton (also called LR(0) automaton), that describes the viable prefixes. Ina Schaefer Context-Free Analysis 69

12 Implementation of Parsers Construction of LR-DFA Let Γ=( T, N, Π, S) be a CFG. For each non-terminal n N, construct Item Automaton Build union of item automata: Start state is the start state of item automaton for S, Final states are final states of item automata Add ɛ transitions from each state which contains the position point in front for a non-terminal A to the starting state of the item automaton of A If all states of the LR-DFA automaton are considered as final states, the accepted language is the language of viable prefixes. Ina Schaefer Context-Free Analysis 70

13 Implementation of Parsers Construction of LR-DFA: Example Γ 3 : S E#, E E + T T, T ID E # [S.E #] [S E.# ] [S E#.] [E.E+T] [E.T ] " " E + T [E E.+T] [E E+.T] [E E+T.] T [E T.] " " [T.ID ] ID [T ID.] Ina Schaefer Context-Free Analysis 71

14 Implementation of Parsers Construction of LR-DFA: Example (2) Determinisation: q0 [S.E #] [E.E+T] [E.T ] [T.ID ] T ID q1 E [S E.# ] # [E E.+T] + Error Fehler q5 [E T.] [T ID.] q7 ID q6 q2 [S E#.] q3 [E E+.T] [T.ID ] T [E E+T.] q4 Error Transitions bezeichnet Fehlerkanten Viable prefixes of maximal length: E#, T, ID, E + ID, E + T Ina Schaefer Context-Free Analysis 72

15 Implementation of Parsers Construction of LR-DFA: Example (3) Remarks: In the example, each final state contains one completely read production, this is in general not the case. If a final state contains more than one completely read productions, we have a reduce/reduce conflict. If a final state contains a completely read and an uncompletely read production with a terminal after the position point, we have a shift/reduce conflict. Ina Schaefer Context-Free Analysis 73

16 Implementation of Parsers Analysis with LR-DFA Analysis of ID + ID + ID # with LR-DFA (the viable prefix is underlined) ID + ID + ID # <= T + ID + ID # <= E + ID + ID # <= E + T + ID # <= E + ID # <= E + T # <= E # <= S Ina Schaefer Context-Free Analysis 74

17 Implementation of Parsers Analysis with LR-DFA (2) Note: The sentential forms always consist of a viable prefix and a remaining input. If an LR-DFA is used, after each reduction the sentential form has to be read from the beginning. Thus: Use pushdown automaton for analysis. Ina Schaefer Context-Free Analysis 75

18 Implementation of Parsers LR pushdown automaton Definition Let Γ=( N, T, Π, S) be a CFG. The LR-DFA pushdown automaton for Γ contains: a finite set of state Q (the states of the LR-DFA(Γ)) a set of actions Act = {shift, accept, error} red(π), where red(π) contains an action reduce(a α) for each production A α. an action table at : Q Act. a successor table succ : P (N T ) Q with P = {q Q at(q) =shift} Ina Schaefer Context-Free Analysis 76

19 Implementation of Parsers LR pushdown automaton (2) Remarks: The LR-DFA pushdown automaton is a variant of pushdown automata particularly designed for LR parsing. States encode the read left context. If there are no conflicts, the action table can be directly constructed from the LR-DFA: accept: final state of item automaton of start symbol reduce: all other final states error: error state shift: all other states Ina Schaefer Context-Free Analysis 77

20 Implementation of Parsers Execution of Pushdown Automaton Configuration: Q T where variable stack denotes the sequence of states and variable inr denotes the remaining input Start configuration: (q 0, input), where q 0 is the start state of the LR-DFA Interpretation Procedure: (stack, inr) := (q0,input); do { step(stack,inr); } while ( at(top(stack))!= accept and at(top(stack))! = error ); if (( at (top(stack)) == error) return error; with Ina Schaefer Context-Free Analysis 78

21 Implementation of Parsers Execution of Push-Down Automaton (2) void step ( var StateSeq stack, var SymbolSeq inr) { State tk: = top(stack); switch ( at(tk) ) { case shift: stack: = push ( succ (tk,top(inr)), keller); inr := tail(inr); break; case reduce A -> a: stack := mpop( length(a),stack); stack := push (succ(top(stack), A), stack); break; } } Ina Schaefer Context-Free Analysis 79

22 Implementation of Parsers LR push down automaton: Example LR-DFA with states q 0,..., q 7 for grammar Γ 3 Action Table Aktionstabelle: q0 schieben shift q1 schieben shift q2 akzeptieren accept q3 schieben shift q4 reduzieren reduce E E+T q5 reduzieren reduce E T q6 reduzieren reduce T ID q7 fehler error Successor Table ID + # E T q0 q6 q7 q7 q1 q5 q1 q7 q3 q2 q7 q7 q2 q3 q6 q7 q7 q7 q4 q4 q5 q6 q7 Ina Schaefer Context-Free Analysis 80

23 Implementation of Parsers LR push down automaton: Example (2) Computation for Input ID + ID + ID # Stack Input Rest Action Keller Eingaberest Aktion q0 q0 q6 q0 q5 q0 q1 q0 q1 q3 q0 q1 q3 q6 q0 q1 q3 q4 q0 q1 q0 q1 q3 q0 q1 q3 q6 q0 q1 q3 q4 q0 q1 q0 q1 q2 ID + ID + ID # schieben shift + ID + ID # reduce reduzieren T ID + ID + ID # reduce reduzieren E T + ID + ID # shift schieben ID + ID # schieben shift + ID # reduce reduzieren T ID + ID # reduce reduzieren E E+T + ID # shift schieben ID # shift schieben # reduce reduzieren T ID # reduce reduzieren E E+T # shift schieben accept akzeptieren Ina Schaefer Context-Free Analysis 81

24 Implementation of Parsers LR-DFA Construction Questions: Does LR-DFA construction work for all unambiguous grammars? For which grammars does the construction work? How can the construction be generalized / made more expressive? Ina Schaefer Context-Free Analysis 82

25 Example LR-DFA Implementation of Parsers LR-DFA for Γ 6 : S E#, E T + E T, T ID N(), N ID q0 q1 q2 # q3 [S E.# ] [S E#.] [E T+E.] E [S.E #] [E.T+E] [E.T ] [T.ID ] [T [N.N( ) ].ID ] N T ID [E T.+E] [E T. ] [T ID.] [N ID.] q4 q6 + T ID N E q5 [E T+.E] [E.T+E] [E.T ] [T.ID ] [T.N( ) ] [N.ID ] q10 error Fehler [ T N.( ) ] [ T N(.) ] ( ) [ T N( ). ] q7 q8 q9 Error Transitions bezeichnet Fehlerkanten Ina Schaefer Context-Free Analysis 83

26 Implementation of Parsers LR Parsing Conflicts 2 Kinds of Conflicts: Shift/Reduce Conflicts (q 4 in example) Reduce/Reduce Conflicts (q 6 in example) Ina Schaefer Context-Free Analysis 84

27 LR Parsing Theory Implementation of Parsers Definition Let Γ=( N, T, Π, S) be a CFG and k N. Γ is an LR(k) grammar if for any two right derivations S rm αau rm αβu S rm γbv rm αβw it holds that: If prefix(k, u) =prefix(k, w) then α = γ, A = B and v = w Ina Schaefer Context-Free Analysis 85

28 Implementation of Parsers LR Parsing Theory (2) Remarks: While for LL grammars the selection of the production depends on the non-terminal to be derived, for LR grammars it depends on the complete left context. For LL grammars, the look ahead considers the language to be generated from the non-terminal. For LR grammars, the look ahead considers the language generated from not yet read non-terminals. Ina Schaefer Context-Free Analysis 86

29 Implementation of Parsers Characterization of LR(0) Theorem A reduced CFG Γ is LR(0) if-and-only-if the LR-DFA(Γ) contains no conflicts. Proof. cf. Lecture Ina Schaefer Context-Free Analysis 87

30 Implementation of Parsers Characterization of LR(0) (2) Example: Application of LR(0)-Chracterization Show (using the above theorem) that Γ 5 is LR(0). Γ 5 : S A B A aab 0 B abbb 1 Ina Schaefer Context-Free Analysis 88

31 Implementation of Parsers Expressiveness of LR(k) For each context-free language L with prefix property (i.e. v, w L: v is no prefix of w), there exists an LR(0) grammar. Grammar Γ 5 is not LL(k), but LR(0). Methods for LR(1) can be generalized to LR(k), SLR(k) and LALR(k). Ina Schaefer Context-Free Analysis 89

32 Implementation of Parsers Resolving Conflicts by Look Ahead Compute look ahead sets from (N T ) k for items. The look ahead set of an item approximates the set of prefixes of length k with which the input rest at this item can start. If the look ahead sets at an item are disjoint, then the action to be executed (shift, reduce) can be determined by k symbols look ahead. For an item, select the action whose look ahead set contains the prefix of the input rest. Action table has to be extended. For computation of look ahead sets, there are different methods. Ina Schaefer Context-Free Analysis 90

33 Implementation of Parsers Common Methods for Look Ahead Computation SLR(k) uses LR-DFA and FOLLOW k of conflicting items for look ahead LALR(k) - look ahead LR - uses LR-DFA with state-dependent look ahead sets LR(k) integrates computation of look ahead sets in automata construction (LR(k) automaton) Ina Schaefer Context-Free Analysis 91

34 Implementation of Parsers SLR Grammars Definition (SLR(1) grammar) Let Γ=( N, T, Π, S) be a CFG and LA([A α.]) = FOLLOW 1 (A). A state LR-DEA(Γ) has an SLR(1) conflict if there exists two different reduce items with LA([A α.]) LA([B β.]) or two items [A α.] and [B α.aβ] with a LA([A a]). Γ is SLR(1) if there is no SLR(1) conflict. Ina Schaefer Context-Free Analysis 92

35 SLR Grammars (2) Implementation of Parsers Example: Γ 6 is an SLR(1) grammar S E# E T + E T T ID N() N ID Consider the conflicts between [E T.] and [E T. + E] and between [T ID.] and [N ID.] FOLLOW 1 (E) {+} = {#} { +} = FOLLOW 1 (T ) FOLLOW 1 (N) ={#, +} { (} = Ina Schaefer Context-Free Analysis 93

36 SLR Grammars (3) Implementation of Parsers Example: Γ 7 (simplifed C expressions) is not an SLR(1) grammar S E# E L = R R L R ID R L Ina Schaefer Context-Free Analysis 94

37 SLR Grammars (4) Implementation of Parsers LR-DFA for Γ 7 [E R.] [S E.# ] # [S E#. ] R [S.E# ] [E.L=R] [E.R] [L.*R] [L.ID] [R.L] E [L *.R] R [R.L] * [L.*R] [L.ID] * ID L ID [L ID.] [R L.] [L *R.] * L [E L.=R] [R L.] = ID [E L=.R] [R.L] [L.*R] [L.ID] L R [E L=R.] Only conflict in items [E L. = R] and [R L.] with FOLLOW 1 (R) {=} = {=, #} { =} Ina Schaefer Context-Free Analysis 95

38 Implementation of Parsers Construction of LR(1) Automata LR(1) automaton contains items [A α.β, V ] with V T where α is on top of the stack the input rest is derivable from βc with c V, i.e. V FOLLOW 1 (A). Ina Schaefer Context-Free Analysis 96

39 Implementation of Parsers Construction of LR(1) Automata (2) LR(1) automaton for Γ 7. Conflict is resolved, as {=} { #} =. S S E#. E E.# S.E# E.L=R # E.R # L.*R #,= L.ID #,= R.L # * # L *.R #,= R.L #,= L.*R #,= L.ID #,= L R ID ID L R E L.=R # = R L. # L E R. # R L. # L ID. #,= L ID. # ID R L. #,= L *R. #,= L E L=.R # R.L # L.*R # L.ID # R E L=R. # * * L *.R # R.L # L.*R # L.ID # R L *R. # * Ina Schaefer Context-Free Analysis 97

40 LALR(1) Automata Implementation of Parsers LALR(1) Automata are constructed from LR(1) automata by merging states in which items only differ in look ahead sets. Look ahead sets for equal items are conjoint. The resulting automaton has the same states as the LR-DFA. However, LALR(1) automata can be generated more efficiently. Ina Schaefer Context-Free Analysis 98

41 Implementation of Parsers LALR(1) Automata (2) LR(1) automaton for Γ 7. S E#. q3 # S E.# q0 E S.E# E.L=R # E.R # L.*R #,= L.ID #,= R.L # L R ID q8 q4 E L.=R # = E L=.R # R L. # R.L # L.*R # q5 L.ID # E R. # q9 R q1 L ID. #,= ID L E L=R. # q2 * L *.R #,= R.L #,= L.*R #,= L.ID #,= ID L R q6 R L. #,= q7 L *R. #,= * * Ina Schaefer Context-Free Analysis 99

42 Grammar Classes Implementation of Parsers unambiguous eindeutige Grammatiken grammars LL(k) LL(1) LR(k) LR(1) LALR(1) SLR(1) LL(0) LR(0) ambiguous grammars mehrdeutige Grammatiken Ina Schaefer Context-Free Analysis 100

43 Implementation of Parsers Literature Recommended Reading for Bottom-Up Analysis: Wilhelm, Maurer: Chapter 8, Sections , pp Ina Schaefer Context-Free Analysis 101

44 Implementation of Parsers Parser Generators Educational Objectives Usage of Parser Generators Characteristics of Parser Generators Ina Schaefer Context-Free Analysis 102

45 Implementation of Parsers JavaCUP Parser Generator CUP - Constructor of Useful Parsers Java-based Generator for LALR-Parsers JFlex can be used to generate according scanner. Running JavaCUP: java -jar java-cup-11a.jar options inputfile Ina Schaefer Context-Free Analysis 103

46 Implementation of Parsers Structure of JavaCUP Specification package JavaPackageName; import java_cup.runtime.*; /* User supplied code for scanner, actions,... */ /* Terminals (tokens returned by the scanner). */ terminal TerminalDecls; /* Non-terminals */ non terminal NonTerminalDecls; /* Precedences */ precedence [left right nonassoc ] TerminalList; /* Grammar */ start with non-terminalname; non_terminalname :: = prod_1... prod_n ; Ina Schaefer Context-Free Analysis 104

47 Implementation of Parsers Example: JavaCUP Specification for Γ 7 import java_cup.runtime.*; /* Terminals (tokens returned by the scanner). */ terminal ID, EQ, MULT; /* Non terminals */ non terminal S, E, L, R; /* The grammar */ start with S; S ::= E; E ::= L EQ R R; L ::= MULT R ID; R ::= L; Ina Schaefer Context-Free Analysis 105

48 Implementation of Parsers Structure of Generated Parser Code Output Files parser.java and sym.java Tables for LALR Automaton Production table: provides the symbol number of the left hand side non-terminal, along with the length of the right hand side, for each production in the grammar, Action table: indicates what action (shift, reduce, or error) is to be taken on each lookahead symbol when encountered in each state Reduce-goto table: indicates which state to shift to after reduce Ina Schaefer Context-Free Analysis 106

49 Implementation of Parsers Usage of Generated Parser Parser calls scanner with scan() method when a new terminal is needed Initialising Parser with new Scanner parser parser_obj = new parser(new my_scanner()); Usage of Parser: Symbol parse_tree = parser_obj.parse(); Ina Schaefer Context-Free Analysis 107

50 Error Handling Error Handling Educational Objectives: Problems and Principles of Error Handling Techniques of Error Handling for Context-Free Analysis Ina Schaefer Context-Free Analysis 108

51 Error Handling Principles of Error Handling Error handling is required in all analysis phases and at runtime. One distinguishes lexical errors parse errors (in context-free analysis) errors in name and type analysis runtime errors (cannot be avoided in most cases) logical errors (behavioural errors) First 2 (3) kinds of errors are syntactic errors. We only consider error handling in context-free analysis. Specification of error handling results basically from language specification. Ina Schaefer Context-Free Analysis 109

52 Error Handling Requirements for error handling Errors should be localized as exactly as possible. (Problem: Error is not detected at error position.) As many errors at possible should be detected at once, but only real errors and no errors as consequences. Errors are not always unique, i.e. it is not clear in general how to correct an error: class int { Int a;... } or int a = 1-; Error handling should not slow down analysis of correct programs. Therefore, error handling is non-trivial and depends on the source language to be analysed. Ina Schaefer Context-Free Analysis 110

53 Error Handling Error Handling in Context-Free Analysis 1. Panic Error Handling Mark synchronizing terminal symbols, e.g. end or ; If parser reaches error state, all symbols up to next synchronizing symbol are skipped and the stack is corrected as if the production with the synchronizing symbol was read correctly. Pros: easy to implement, termination guaranteed Cons: large parts of the program can be skipped or misinterpreted Example: Incorrect Input a : = b *** c; Read until ; correct stack and reuse as if statement has been accepted Ina Schaefer Context-Free Analysis 111

54 Error Handling Error Handling in Context-Free Analysis (2) 2. Error Productions Extend grammar with productions describing typical error situations, so called error productions. Error messages can be directly associated with error productions. Pros: easy to implement, termination guaranteed Cons: extended grammar can belong to more general grammar class, knowledge of typical error situations is necessary Example: Typical error in PASCAL if... then A := E; else... Error Production: Stmt if Expr then Stmt ; else Stmt Ina Schaefer Context-Free Analysis 112

55 Error Handling Error Handling in Context-Free Analysis (3) 3. Production-Local Error Correction Goal is local correction of input such that analysis can be resumed. Local means that it is tried to correct the input for the current production. Pros: flexible and powerful technique Cons: problematic if errors occur earlier than they can be detected, operations for corrections can lead to nonterminating analysis Ina Schaefer Context-Free Analysis 113

56 Error Handling Error Handling in Context-Free Analysis (4) 4. Global Error Correction Attempt to get a correction that is as good as possible by altering the read input or the look ahead input. Idea: Define distance or quality measure on inputs. For each incorrect input, look for a syntactically correct input that is best according to the used measure. Pros: very powerful technique Cons: analysis effort can be rather high, implementation is complex and poses risk of non-termination. Ina Schaefer Context-Free Analysis 114

57 Error Handling Error Handling in Context-Free Analysis (5) 5. Interactive Error Correction In modern programming languages, syntactic analysis is often already supported by editors. In this case, editor marks error positions. Pros: quick feedback, possible error positions are shown directly, interaction with programmer possible Cons: editing can be disturbed, analysis must be able to handle incomplete programs The presented techniques can also be combined. For selection of technique, programming language syntax is important. Error handling also depends on grammar class and implementation techniques used for parser. Ina Schaefer Context-Free Analysis 115

58 Error Handling Burke-Fischer Error Handling Example of global error correction technique Procedure: Use correction window of n symbols before symbol at which error was detected. Check all possible variations of symbol sequence in correction window that can be obtained by insertion, exchange or modification of a symbol at any position. Quality Measure: Choose variation that allows longest continuation of parsing procedure Implementation: Work with two stack automata, one represents the configuration at the beginning of the correction window, the other one the configuration at the end of the correction window. In an error case, the automaton running behind can be used to resume at the old position and to test the computed variations. Ina Schaefer Context-Free Analysis 116

59 Literature Error Handling Recommended Reading: Wilhelm, Maurer: Chapter 8, Sections and (general understanding sufficient) Ina Schaefer Context-Free Analysis 117

60 Concrete and Abstract Syntax Concrete and Abstract Syntax Educational Objectives Connection of parsing to other phases of program processing and translation Differences between abstract and concrete syntax Language concepts for describing syntax trees Syntax tree construction Ina Schaefer Context-Free Analysis 118

61 Concrete and Abstract Syntax Connection of Parsers to other Phases 1. Parser directly controls following phases 2. Concrete Syntax Tree as Interface 3. Abstract Syntax Tree as Interface Ina Schaefer Context-Free Analysis 119

62 Concrete and Abstract Syntax Direct Control by Parser Example: Recursive Descent: Parser calls other actions after each derivation/reduction step Pros: simple (if realisable) flexible efficient (especially memory efficient) Cons: non-modular, no clear interfaces not suitable for global aspects of translation following phases depend on parsing cannot be used with every parser generator Ina Schaefer Context-Free Analysis 120

63 Concrete and Abstract Syntax Abstract Syntax vs. Concrete Syntax Definition (Concrete Syntax) The concrete syntax of a programming languages determines the actual text representation of the programs (incl. key words, separators). If Γ is the CFG used for parsing a program P in a certain language, the syntax tree of P according to Γ is the concrete syntax tree of P. Definition (Abstract Syntax) The abstract syntax of a programming language describes the tree structure of programs in a form that is sufficient and suitable for further processing. A tree for representing a program P according to the abstract syntax of a language is called abstract syntax tree of P. Ina Schaefer Context-Free Analysis 121

64 Abstract Syntax Concrete and Abstract Syntax abstraction from keywords and separators operator precedences are represented in tree structure (different non-terminals are not necessary) better incorporation of symbol information simplifying transformations Remarks: The abstract syntax of a language is often not specified in the language report. The abstract syntax usually also comprises information about source code positions. Ina Schaefer Context-Free Analysis 122

65 Concrete and Abstract Syntax Example: Concrete vs. Abstract Syntax Concrete Syntax: Γ 2 S E# E T + E T T F T F F (E) ID Abstract Syntax Exp = Add Mult Ident Add (Exp left, Exp right) Mult (Exp left, Exp right) Ina Schaefer Context-Free Analysis 123

66 Concrete and Abstract Syntax Example: Concrete vs. Abstract Syntax (2) Text: (a + b) c Concrete Syntax Tree Abstract Syntax Tree S E # Mult T T F Add c F E a b E T T F F ( ID + ID ) * ID a b c Ina Schaefer Context-Free Analysis 124

67 Concrete and Abstract Syntax Concrete Syntax Tree as Interface Token Stream Parser (with Tree Construction) Concrete Syntax Tree Further Language Processing Counters disadvantages of direct control by parser Advantages over Abstract Syntax No additional specification of abstract syntax required Tree construction does not have to be described. Tree construction can be done automatically by parser generators. Ina Schaefer Context-Free Analysis 125

68 Concrete and Abstract Syntax Abstract Syntax Tree as Interface Token Stream Parser (with Transforming Tree Construction) Abstract Syntax Tree Further Language Processing Advantages over Concrete Syntax Simpler, more compact tree representation Simplifies later phases Often implemented by programming or specification language as mutable data structure Ina Schaefer Context-Free Analysis 126

69 Concrete and Abstract Syntax Abstract Syntax: Specification and Tree Construction For representing abstract syntax trees, we use order-sorted terms. The sets and types of these terms are described by type declarations. Ina Schaefer Context-Free Analysis 127

70 Concrete and Abstract Syntax Order-sorted Data Types Definition Oder-sorted Data Types are specified by declarations of the following form: Variant Type Declarations V = V 0 V 1... V m Tuple Type Declaration T (T 1 sel 1,..., T n sel n ) List Type Declarations L S Example: Exp = Add Mult Ident Add (Exp left, Exp right) Mult (Exp left, Exp right) where Ident is a predefined type. Ina Schaefer Context-Free Analysis 128

71 Concrete and Abstract Syntax Order-sorted Data Types (2) Definition (Order-sorted Types - contd.) Order-sorted Terms are recursively defined as If t is a term of type V i, then it is also of type V. If t i is a term of type T i for each i, then T (t 1,..., t n ) is of type T, T is also the constructor. If s 1,..., s k are terms of type S, then L(s 1,..., s n ) is of type L, L is also the list constructor. Additional Operators the selectors sel k : T T k returns the k-th subterm of a tuple the usual list operations (rest, append, conc,...) Ina Schaefer Context-Free Analysis 129

72 Concrete and Abstract Syntax Order-sorted Data Types (3) Remarks: Order-sorted Data Types generalise data types of functional languages by subtyping, a term can belong to serveral types are used in specification languages, e.g. OBJ3, MAX,... are a very compact form for type declaration can be implemented by OO languages Ina Schaefer Context-Free Analysis 130

73 Concrete and Abstract Syntax Example: Order-sorted Data Types and OO Types Declaration of order-sorted data types: Exp = Add Mult Neg Ident Add (Exp left, Exp right) Mult (Exp left, Exp right) Neg (Exp val) Ina Schaefer Context-Free Analysis 131

74 Concrete and Abstract Syntax Implementation in Java interface Exp { Exp left() throws IllegalSelectException; Exp right() throws IllegalSelectException; Exp val() throws IllegalSelectException; } class Add implements Exp { private Exp left; private Exp right; } Add( Exp l, Exp r ) { left = l; right = r; } Exp left() { return left; } Exp right(){ return right; } Exp val() throws IllegalSelectException { throw new IllegalSelectException(); } Ina Schaefer Context-Free Analysis 132

75 Concrete and Abstract Syntax Implementation in Java (2) class Mult implements Exp { // analog zu Add } class Neg implements Exp { private Exp val; Neg( Exp v ) { val = v; } Exp left() throws IllegalSelectException { throw new IllegalSelectException(); } Exp right() throws IllegalSelectException { throw new IllegalSelectException(); } } Exp val() { return val; } Ina Schaefer Context-Free Analysis 133

76 Concrete and Abstract Syntax Implementation in Java (3) class Ident implements Exp extends PredefIdent { Exp left() throws IllegalSelectException { throw new IllegalSelectException(); } Exp right() throws IllegalSelectException { throw new IllegalSelectException();} } Exp val() throws IllegalSelectException { throw new IllegalSelectException(); } Ina Schaefer Context-Free Analysis 134

77 Concrete and Abstract Syntax Transformation of Concrete to Abstract Syntax S E # T Mult(_,_) T F F E Add(_,_) E T T F F ( ID + ID ) * ID a b c Ina Schaefer Context-Free Analysis 135

78 Concrete and Abstract Syntax Transformation to Abstract Syntax with JavaCUP S ::= E:e # {: RESULT = e; :} ; E ::= E:e + T:t {: RESULT = ADD(e,t); :} T:t {: RESULT = t; :} ; T ::= T:t * F:f {: RESULT = MULT(t,f); :} F:f {: RESULT = f; :} ; F ::= ( E:e ) {: RESULT = e ; :} ID:i {: RESULT = i; :} ; Ina Schaefer Context-Free Analysis 136

79 Concrete and Abstract Syntax Recommended Reading Wilhelm, Maurer: Section 9.1, pp Appel: Chapter 4, pp Ina Schaefer Context-Free Analysis 137