Definition of AST AST and Static Code Analysis Design Patterns. AST Features Applicability of AST JTB JJTree

Transcription

1 Definition of AST AST and Design Patterns Błaej Pietrzak AST Features Applicability of AST JTB JJTree Parse Tree Input: id*(id+id) E -> E + E E -> E * E E -> ( E ) E -> id Abstract Syntax Tree Parse tree (or concrete syntax tree) is a tree that represents the syntactic structure of a string according to some formal grammar. A program that produces such trees is called a parser Abstract syntax tree (AST) is a finite, labeled, directed tree, where the internal nodes are labeled by operators, and the leaf nodes represent the operands of the node operators. Thus, the leaves have nullary operators, i.e., variables or constants A code A code Scanner Scanner tokens Parser 1

2 A code Scanner tokens Abstract Syntax Tree Parser Code Analysis Builder Composite or not? Not necessarily, since clients may want to know about the AST structure Interpreter vs Visitor Defining the Interpret operation. You don't have to define the Interpret operation in the expression classes. If it's common to create a new interpreter, then it's better to use the Visitor pattern to put Interpret in a separate "visitor" object. It will be more likely to use a visitor to avoid defining these operations on every grammar class. Flyweight pattern Sharing terminal symbols Terminal nodes generally don't store information about their position in the AST Parent nodes pass them whatever context they need during interpretation There is a distinction between shared (intrinsic) state and passed-in (extrinsic) stat. AST contrary to parse tree captures the essential structure of the input in a tree form, while omitting unnecessary syntactic details In an AST the grouping of operands is explicit in the tree structure ASTs omit tree nodes that represent unary productions in the grammar. Such information is directly represented in ASTs by the structure of the tree. AST has many similar forms e.g. for, while expressions may have the same AST. Expressions in AST may be complex, nested it doesn t have to be the simplest represent. for analysis For complex grammars parser generators are a better alternative they can interpret expressions without building ASTs The most efficient interpreters are usually not implemented by interpreting parse trees directly, but by first translating them into another form e. g. regular expressions are often transformed into state machines Java Tree Builder (JTB) external library JJTree part of JavaCC 2

3 Simple and easy to use No need for extra code in grammar file Produces big AST trees Every nonterminal generates a node + extra nodes Tree node classes implement the Visitor design pattern Nodes preserve type information by storing references to actual types of its children Operates on standard unmodified JavaCC grammar file Alters generated files Original Grammar file *.jtb Decorated new grammar file *.jj Java source code java -jar jtb.jar *.jtb javacc jtb.out.jj syntaxtree package visitor package A package containing Parser, TokenManager etc. options JDK_VERSION = "1.5"; PARSER_BEGIN(Example) public class Example public static void main(string args[]) new Example(System.in); try Start start = Example.Start(); start.accept(new MyVisitor()); catch (Exception ex) System.err.println(ex.getMessage()); class MyVisitor extends DepthFirstVisitor public void visit(nodetoken n) System.out.println("visited " + n.tokenimage); public void visit(start n) System.out.println("<Start>"); super.visit(n); PARSER_END(Example) SKIP : " " "\t" "\n" "\r" <"//" (~["\n","\r"])* ("\n" "\r" "\r\n")> <"/*" (~["*"])* "*" (~["/"] (~["*"])* "*")* "/"> 3

4 TOKEN : /* IDENTIFIERS */ < IDENTIFIER: <LETTER> (<LETTER> <DIGIT>)* > < #LETTER: ["_","a"-"z","a"-"z"] > < #DIGIT: ["0"-"9"] > void Start(): Expression() ";" void Expression(): AdditiveExpression() void AdditiveExpression(): MultiplicativeExpression() ( "+" MultiplicativeExpression() )* void MultiplicativeExpression(): UnaryExpression() ( "*" UnaryExpression() )* void UnaryExpression(): "(" Expression() ")" Identifier() void Identifier(): <IDENTIFIER> Start Expression AdditiveExpression NodeToken ; MultiplicativeExpression UnaryExpression NodeListOptional NodeChoice NodeToken UnaryExpression Identifier NodeToken * NodeChoice NodeList id NodeToken Expression NodeToken ( ) Greater flexibility for trees generation Added complexity and development time Smaller trees less resources are required Coarse support for Visitor design pattern Nodes don t preserve type information Operates on modified JavaCC grammar file Certain productions can be flagged so as not to generate a node class (by default a node is constructed for each nonterminal) Although JavaCC is a top-down parser, JJTree constructs the parse tree from the bottom up (uses stack) Does not alter generated files Exception handling Makes possible for parsers to implement error recovery and continue with the node stack in a known state. An exception thrown by an expansion within a node scope that is not caught within the node scope is caught by JJTree itself. Any nodes that have been pushed on to the node stack within the node scope are popped and thrown away. Then the exception is rethrown. 4

5 MULTI = false <<interface>> Node SimpleNode int id <<interface>> $ParserVisitor public Object visit( SimpleNode node, Objectdata); Multi = true <<interface>> Node SimpleNode ASTAdditiveExpression <<interface>> $ParserVisitor ASTVariable public Object visit( SimpleNode node, Objectdata); public Object visit( ASTVariable node, Objectdata); public interface Node public interface Node public Called after void the jjtclose(); node has been made the current public node. void jjtsetparent(node n); public It indicates Node that jjtgetparent(); child nodes can now be added to it. public interface Node public This method void is called jjtsetparent(node after all the child nodes n); have been public added. Node jjtgetparent(); public interface Node This pair of methods are used to inform the node of its parent. public interface Node This method tells the node to add its argument to the node's list of children. 5

6 public interface Node public This method int jjtgetnumchildren(); returns a child node. The children are numbered from zero, left to right. public interface Node public Return the void number jjtaddchild(node of children the n, node int has. i); public interface Node public Accept Node the visitor. jjtgetchild(int i); JJTreeState JJTree keeps its state in a parser class field called jjtree. You can use methods in this member to manipulate the node stack. options VISITOR = true; MULTI = true; Vistor design pattern support Generate a multi mode parse tree When Multi = true then for every AST node a distinct class is generated e.g. for Start nonterminal a ASTStart class is generated When MULTI = false then every node in a tree is an instance of SimpleNode no class is generated. PARSER_BEGIN(Example) public class Example public static void main(string args[]) Example app = new Example(System.in); try SimpleNode n = Example.Start(); n.dump(""); n.jjtaccept(app.new MyVisitor(), null); catch (Exception ex) System.out.println(ex.getMessage()); 6

7 class MyVisitor implements ExampleVisitor public Object visit(simplenode node,object data) System.out.println("visited " + node.tostring()); return node.childrenaccept(this, data); public Object visit(aststart node, Object data) System.out.println("Start"); return node.childrenaccept(this, data); PARSER_END(Example) class MyVisitor implements ExampleVisitor public Object visit(simplenode node,object data) System.out.println("visited " + node.tostring()); return node.childrenaccept(this, data); PARSER_END(Example) SKIP : " " "\t" "\n" "\r" <"//" (~["\n","\r"])* ("\n" "\r" "\r\n")> <"/*" (~["*"])* "*" (~["/"] (~["*"])* "*")* "/"> TOKEN : /* IDENTIFIERS */ < IDENTIFIER: <LETTER> (<LETTER> <DIGIT>)* > < #LETTER: ["_","a"-"z","a"-"z"] > < #DIGIT: ["0"-"9"] > SimpleNode Start(): Expression() ";" return jjtthis; void Expression() #void : AdditiveExpression() SimpleNode Start(): Expression() ";" return jjtthis; Each void node Expression() is associated with #void a node : scope. User actions within this scope can access the node under construction by using jjtthis identifier to refer to the node. AdditiveExpression() This identifier is implicitly declared to be of the correct type for the node, so any fields and methods that the node has can be easily accessed. 7

8 If you want to suppress the creation of a node for a production SimpleNode you can Start(): use the # void syntax Any parse tree nodes pushed by nonterminals in the expansion Expression() of will ";" remain on the stack, to be popped return and made jjtthis; children of a production further up the tree void Expression() #void : AdditiveExpression() void AdditiveExpression(): MultiplicativeExpression() ( "+" MultiplicativeExpression() )* void MultiplicativeExpression(): UnaryExpression() ( "*" UnaryExpression() )* void UnaryExpression() #void : "(" Expression() ")" Identifier() void Identifier(): <IDENTIFIER> ASTStart ASTAdditiveExpression ASTMultiplicativeExpression ASTAdditiveExpression ASTMultiplicativeExpression ASTMultiplicativeExpression Still contains unary nodes! There is no information about an value. options VISITOR = true; MULTI = false; NODE_SCOPE_HOOK = true; JJTree generates calls to two user-defined parser methods on the entry and exit of every node scope. PARSER_BEGIN(Example) public static void jjtreeopennodescope(node node) SimpleNode temp = (SimpleNode) node; if (temp.id == ExampleTreeConstants.JJTIDENTIFIER) temp.firsttoken = gettoken(1); public static void jjtreeclosenodescope(node node) SimpleNode temp = (SimpleNode) node; if (temp.id == ExampleTreeConstants.JJTIDENTIFIER) temp.lasttoken = gettoken(0); 8

9 void AdditiveExpression() #AdditiveExpression(jjtree.nodeArity() > 1): MultiplicativeExpression() ( "+" MultiplicativeExpression() )* void MultiplicativeExpression() #MultiplicativeExpression(jjtree.nodeArity() > 1): UnaryExpression() ( "*" UnaryExpression() )* void AdditiveExpression() #AdditiveExpression(jjtree.nodeArity() > 1): MultiplicativeExpression() ( "+" MultiplicativeExpression() )* void MultiplicativeExpression() #MultiplicativeExpression(jjtree.nodeArity() > 1): A conditional node is constructed with all of the children UnaryExpression() that were pushed ( "*" on UnaryExpression() the stack within its )* node scope if and only if its condition evaluates to true. If it evaluates to false, the node is not constructed, and all of the children remain on the node stack. ASTStart ASTMultiplicativeExpression ASTAdditiveExpression SimpleNode id=start SimpleNode id =MultiplicativeExpression id id id JavaCC JJTree Reference Documentation Java Tree Builder (JTB) E. Gamma, R. Helm, R. Johnson, J. Vlissides: Design Patterns: Elements of Reusable Object- Oriented Software JavaCC Eclipse plugin Thank You for your attention What is your general impression (1-6) Was it too slow or too fast? What important did you learn during the lecture? What to improve and how? 9