EDA180: Compiler Construc6on. Top- down parsing. Görel Hedin Revised: a
|
|
- Jack Moses Carson
- 6 years ago
- Views:
Transcription
1 EDA180: Compiler Construc6on Top- down parsing Görel Hedin Revised: a
2 Compiler phases and program representa6ons source code Lexical analysis (scanning) Intermediate code genera6on tokens intermediate code Syntac6c analysis (parsing) Op6miza6on AST APributed AST intermediate code Seman6c analysis Analysis Machine code genera6on Synthesis machine code 2
3 A closer look at the parser text Scanner tokens Defined by: regular expressions Parser Pure parsing concrete parse tree (implicit) This lecture context- free grammar AST building AST abstract grammar 3
4 Different parsing algorithms Unambiguous LR LL Ambiguous This lecture All context- free grammars LL: Left-to-right scan Leftmost derivation Builds tree top-down Simple to understand LR: Left-to-right scan Rightmost derivation Builds tree bottom-up More powerful 4
5 CompoundStmt IfStmt LL and LR parsers, main idea Id Assign Id Assign Id Id if ID then ID = ID ; ID... if ID then ID = ID ; ID... LL(1): decides to build Assign after seeing the first token of its subtree. The tree is built top down. LR(1): decides to build Assign after seeing the first token following its subtree. The tree is built bottom up. The token is called lookahead. LL(k) and LR(k) use k lookahead tokens. 5
6 Recursive- descent parsing A way of programming an LL(1) parser by recursive method calls Assume an EBNF grammar with exactly one produc6on rule for each nonterminal symbol. For each nonterminal, a method is constructed. A nonterminal method matches tokens and calls other nonterminal methods, according to the grammar. If the lookahead token does not match, an error is reported. A -> B C D B -> a C b D C ->... D ->... 6
7 Example Java implementa6on: overview statement -> assignment compoundstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE expr ->... class Parser { private int token; "// current lookahead token void accept(int t) {...} "// accept t and read in next token void error(string str) {...}"// generate error message void statement() {...} void assignment () {...} void compoundstmt () {...}... } 7
8 Example: recursive descent methods statement -> assignment compoundstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE class Parser { void statement() { switch(token) { case ID: assignment(); break; case LBRACE: compoundstmt(); break; default: error("expecting statement, found: " + token); } } void assignment() { accept(id); accept(assign); expr(); accept(semicolon); } void compoundstmt() { accept(lbrace); while (token!=rbrace) { statement(); } accept(rbrace); }... } 8
9 Example: Parser skeleton details statement -> assignment compoundstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE expr ->... class Parser { final static int ID=1, WHILE=2, DO=3, ASSIGN=4,...; private int token; "// current lookahead token void accept(int t) { "// accept t and read in next token if (token==t) { token = nexttoken(); } else { error("expected " + t + ", but found " + token); } } void error(string str) {...}"// generate error message private int nexttoken() {...} // read next token from scanner void statement() } 9
10 Are these grammars LL(1)? expr -> name params name Common prefix expr -> expr "+" term term term -> ID Led recursion What would happen in a recursive- descent parser? Could they be LL(2)? LL(k)? 10
11 Dealing with common prefix of limited length: Local lookahead LL(2) grammar: statement -> assignment compoundstmt callstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE callstmt -> ID LPAR expr RPAR SEMICOLON void statement()... 11
12 Dealing with common prefix of limited length: Local lookahead LL(2) grammar: statement -> assignment compoundstmt callstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE callstmt -> ID LPAR expr RPAR SEMICOLON void statement() { switch(token) { case ID: if (lookahead(2) == ASSIGN) { assignment(); } else { callstmt(); } break; case LBRACE: compoundstmt(); break; default: error("expecting statement, found: " + token); } } 12
13 Common prefix? If two produc6ons can derive a sentence star6ng in the same way, they share a common prefix. A -> a B A -> a C B -> b C -> c A -> B a A -> B b B -> c d A -> a B B -> a C B -> b C C -> c A has two rules that can derive the prefix a The grammar is LL(2) A has two rules that can derive the prefix c d The grammar is LL(3) No problem. The two rules that start the same cannot be derived from the same nonterminal. The grammar is LL(1) Which nonterminals have common prefix produc6ons? How long is the common prefix? Is the grammar LL(1), LL2(),...? 13
14 Common prefix? A -> B A -> C A -> D B -> b a C -> b d D -> e A has two rules that can derive the prefix b The grammar is LL(2) A -> B a A -> B b B -> B c B -> d A has two rules that can derive the prefix d c* So, the prefix can become arbitrarily long. The grammar is not LL(k), no matter what k we use. We need to rewrite the grammar, or use another parsing method. Which nonterminals have common prefix produc6ons? How long is the common prefix? Is the grammar LL(1), LL2(),...? 14
15 Elimina6ng the common prefix Rewrite to an equivalent grammar without the common prefix Exp -> Name Params Exp -> Name With common prefix - not LL(1) 15
16 Elimina6ng the common prefix Rewrite to an equivalent grammar without the common prefix Exp -> Name Params Exp -> Name With common prefix - not LL(1) Exp -> Name OptParams OptParams -> Params OptParams -> ε Without common prefix - LL(1) Eliminating a common prefix this way is called "left factoring". 16
17 Elimina6ng the common prefix Rewrite to an equivalent grammar without the common prefix A -> B A -> C B -> b a B -> e D B -> f C -> b d D -> B C Indirect common prefix 17
18 Elimina6ng the common prefix Rewrite to an equivalent grammar without the common prefix A -> B A -> C B -> b a B -> e D B -> f C -> b d D -> B C Indirect common prefix First, make the common prefix directly visible: Substitute all B right-hand sides into the A -> B rule We can't remove the B rules since B is used in other places. Similarly for the A -> C rule A -> b a A -> e D A -> f B -> b a B -> e D B -> f A -> b d C -> b d D -> B C Direct common prefix Then, eliminate the direct common prefix, as previously. 18
19 Dealing with led recursion in LL parsers Method 1: Rewrite to an equivalent grammar without led recursion (A bit cumbersome) Left-recursive grammar not LL(k) E -> E "+" T E -> T T-> ID Rewrite to right-recursion! But there is now a common prefix! Still not LL(k). E -> T "+" E E -> T T-> ID Eliminate the common prefix. The grammar is now LL(1) E -> T E' E' -> "+" E E' -> ε T-> ID A left-recursive AST can be built during the right-recursive parse. 19
20 Dealing with led recursion in LL parsers Method 2: Rewrite to EBNF (Easy!) Left-recursive grammar not LL(k) E -> E "+" T E -> T T-> ID Rewrite to EBNF! E -> T ( "+" T )* T-> ID A left-recursive AST can be built during the iteration. 20
21 JavaCC: An LL- based parser generator CFG (in Java-like spec langauge) JavaCC Parser (in Java code) 21
22 JavaCC specifica6on CFG: statement -> assignment compoundstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE JavaCC: void statement() : {} { assignment() compoundstmt() } void assignment() : {} { id() <ASSIGN> expr() <SEMICOLON> } void compoundstmt() : {} { <LBRACE> (statement())* <RBRACE> } void id() : {} { <ID> } Place where Java code can be added. You can also add Java code inside the rules. (For semantic actions, e.g. build the AST) Good idea to add a nonterminal id for ID tokens. This way you can avoid code duplication in the semantic actions. 22
23 Using local lookahead in JavaCC Local lookahead can be used to discriminate between an assignment and a procedure call: statement -> assignment callstmt whilestmt assignment -> ID ASSIGN expr SEMICOLON callstmt -> ID LPAR expr RPAR SEMICOLON JavaCC: void statement() : {} { LOOKAHEAD(2) assignment() callstmt() whilestmt() }... A lookahead of 2 tokens will be used before selecting assignment. If that fails, ordinary single-token lookahead will be used in the following alternatives. 23
24 Using EBNF in JavaCC Straight forward! expr -> term (PLUS term)* term -> factor (TIMES factor)* factor -> ID INT LPAR expr RPAR JavaCC: void expr() : {} { term() (<PLUS> term())* } void term() : {} { factor() (<TIMES> factor())* } void factor() : {} { id() intexpr() <LPAR> expr () <RPAR> } 24
25 Algorithm for construc6ng an LL(1) parser Fairly simple. The non-trivial part: how to select the correct production p for X, based on the lookahead token. X p1: X ->... p2: X ->... Which tokens can occur in the FIRST position?... t 1... t n t n+1... FIRST FOLLOW Can one of the productions derive the empty string? I.e., is it "NULLABLE"? If so, which tokens can occur in the FOLLOW position? 25
26 Steps in construc6ng an LL(1) parser 1. Write the grammar on canonical form 2. Analyze the grammar to construct a table. The table shows what production to select, given the current lookahead token. 3. Conflicts in the table? The grammar is not LL(1). 4. No conflicts? Straight forward implementation using table-driven parser or recursive descent. t 1 t 2 t 3 t 4 X 1 p1 p2 X 2 p3 p3 p4 26
27 Example: Construct the LL(1) table for this grammar: p1: statement -> assignment p2: statement -> compoundstmt p3: assignment -> ID "=" expr ";" p4: compoundstmt -> "{" statements "}" p5: statements -> statement statements p6: statements -> ε statement assignment compoundstmt statements ID "=" ";" "{" "}" For each production p: X -> γ, we are interested in: FIRST(γ) the tokens that occur first in a sentence derived from γ. NULLABLE(γ) is it possible to derive ε from γ? And if so: FOLLOW(X) the tokens that can occur immediately after an X-sentence. 27
28 Example: Construct the LL(1) table for this grammar: p1: statement -> assignment p2: statement -> compoundstmt p3: assignment -> ID "=" expr ";" p4: compoundstmt -> "{" statements "}" p5: statements -> statement statements p6: statements -> ε ID "=" ";" "{" "}" statement p1 p2 assignment p3 compoundstmt p4 statements p5 p5 p6 To construct the table, look at each production p: X -> γ. Compute the token set FIRST(γ). Add p to each corresponding entry for X. Then, check if γ is NULLABLE. If so, compute the token set FOLLOW(X), and add p to each corresponding entry for X. 28
29 Example: Dealing with End of File: p1: vardecl -> type ID optinit p2: type -> "integer" p3: type -> "boolean" "=" expr ";" p4: optinit -> "=" INT p5: optinit -> ε ID integer boolean "=" ";" INT vardecl type optinit 29
30 Example: Dealing with End of File: p0: S -> vardecl EOF p1: vardecl -> type ID optinit p2: type -> "integer" p3: type -> "boolean" "=" expr ";" p4: optinit -> "=" INT p5: optinit -> ε ID integer boolean "=" ";" INT EOF S p0 p0 vardecl p1 p1 type p2 p3 optinit p4 p5 30
31 Example: Ambiguous grammar: p1: E -> E "+" E p2: E -> ID p3: E -> INT E "+" ID INT 31
32 Example: Ambiguous grammar: p1: E -> E "+" E p2: E -> ID p3: E -> INT "+" ID INT E p1, p2 p1, p3 Collision in a table entry! The grammar is not LL(1) An ambiguous grammar is not even LL(k) adding more lookahead does not help. 32
33 Example: Unambiguous, but led- recursive grammar: p1: E -> E "*" F p2: E -> F p3: F -> ID p4: F -> INT E F "*" ID INT 33
34 Example: Unambiguous, but led- recursive grammar: p1: E -> E "*" F p2: E -> F p3: F -> ID p4: F -> INT "*" ID INT E p1,p2 p1,p2 F p3 p4 Collision in a table entry! The grammar is not LL(1) A grammar with left-recursion is not even LL(k) adding more lookahead does not help. 34
35 Example: Grammar with common prefix: p1: E -> F "*" E p2: E -> F p3: F -> ID p4: F -> INT p5: F -> "(" E ")" E F "*" ID INT "(" ")" 35
36 Example: Grammar with common prefix: p1: E -> F "*" E p2: E -> F p3: F -> ID p4: F -> INT p5: F -> "(" E ")" "*" ID INT "(" ")" E p1,p2 p1,p2 p1,p2 F p3 p4 p5 Collision in a table entry! The grammar is not LL(1) A grammar with common prefix is not LL(1). Some grammars with common prefix are LL(k), for some k, but not this one. 36
37 Example: Another grammar with common prefix: p1: Stmt -> ID "(" IdList ")" p2: Stmt -> ID "=" Exp Stmt... ID "(" ")" "=" 37
38 Example: Another grammar with common prefix: p1: Stmt -> ID "(" IdList ")" p2: Stmt -> ID "=" Exp Stmt... ID "(" ")" "=" p1, p2 Collision in a table entry! The grammar is not LL(1) A grammar with common prefix is not LL(1) But this grammar is LL(2) 38
39 We could create an LL(2) table! (global lookahead 2) p1: Stmt -> ID "(" IdList ")" p2: Stmt -> ID "=" Exp ID ID ID "(" ID "=" ID ")" "(" ID "(" "("... Stmt p1 p2... No conflicts! The grammar is LL(2)! But k > 1 gives very large tables inefficient! 39
40 A beper alterna6ve: Local lookahead! p1: Stmt -> ID "(" IdList ")" p2: Stmt -> ID "=" Exp Stmt "(" "="... p1 ID "(" ")" "=" p2 No collisions! A slightly more complex table structure for the local lookahead. Will be efficient. JavaCC can generate both LL(k) and local lookahead parsers. Using k > 1 is not recommended. Too slow parsing. Use local lookahead if needed. 40
41 Summary: construc6ng an LL(1) parser 1. Write the grammar on canonical form 2. Analyze the grammar using FIRST, NULLABLE, and FOLLOW. 3. Use the analysis to construct a table. The table shows what production to select, given the current lookahead token. 4. Conflicts in the table? The grammar is not LL(1). 5. No conflicts? Straight forward implementation using table-driven parser or recursive descent. 41
42 Summary ques6ons Construct a CFG for a simple part of a programming language. Construct a recursive descent parser for a simple language. Give typical examples of ambigui6es in CFGs What is the difference between LL(1) and LL(k)? What is a "common prefix", and how can it be eliminated? What is meant by "led factoring"? What is "led recursion" and how can it be eliminated? In what way can an LL syntax tree differ from the desired AST? Construct an EBNF grammar for conven6onal arithme6c expressions that respect standard precedence and associa6vity. What is NULLABLE(X), FIRST(X), and FOLLOW(X)? Construct an LL(1) table for a grammar. What does it mean if there is a collision in an LL(1) table? What is the difference between local lookahead and global lookahead? Why can it be useful to add an end- of- file rule to some grammars? How can we decide if a grammar is LL(1) or not? 42
43 Readings F4: Predic6ve parsing. Recursive descent. LL grammars and parsing. Led recursion and factoriza6on. Appel, chapter
EDA180: Compiler Construc6on Context- free grammars. Görel Hedin Revised:
EDA180: Compiler Construc6on Context- free grammars Görel Hedin Revised: 2013-01- 28 Compiler phases and program representa6ons source code Lexical analysis (scanning) Intermediate code genera6on tokens
More informationLL parsing Nullable, FIRST, and FOLLOW
EDAN65: Compilers LL parsing Nullable, FIRST, and FOLLOW Görel Hedin Revised: 2014-09- 22 Regular expressions Context- free grammar ATribute grammar Lexical analyzer (scanner) SyntacKc analyzer (parser)
More informationEDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing. Görel Hedin Revised:
EDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing Görel Hedin Revised: 2017-09-04 This lecture Regular expressions Context-free grammar Attribute grammar
More informationEDA180: Compiler Construc6on. More Top- Down Parsing Abstract Syntax Trees Görel Hedin Revised:
EDA180: Compiler Construc6on More Top- Down Parsing Abstract Syntax Trees Görel Hedin Revised: 2013-02- 05 Compiler phases and program representa6ons source code Lexical analysis (scanning) Intermediate
More informationEDAN65: Compilers, Lecture 06 A LR parsing. Görel Hedin Revised:
EDAN65: Compilers, Lecture 06 A LR parsing Görel Hedin Revised: 2017-09-11 This lecture Regular expressions Context-free grammar Attribute grammar Lexical analyzer (scanner) Syntactic analyzer (parser)
More informationLL(k) Compiler Construction. Top-down Parsing. LL(1) parsing engine. LL engine ID, $ S 0 E 1 T 2 3
LL(k) Compiler Construction More LL parsing Abstract syntax trees Lennart Andersson Revision 2011 01 31 2010 Related names top-down the parse tree is constructed top-down recursive descent if it is implemented
More informationLL(k) Compiler Construction. Choice points in EBNF grammar. Left recursive grammar
LL(k) Compiler Construction More LL parsing Abstract syntax trees Lennart Andersson Revision 2012 01 31 2012 Related names top-down the parse tree is constructed top-down recursive descent if it is implemented
More informationPart III : Parsing. From Regular to Context-Free Grammars. Deriving a Parser from a Context-Free Grammar. Scanners and Parsers.
Part III : Parsing From Regular to Context-Free Grammars Deriving a Parser from a Context-Free Grammar Scanners and Parsers A Parser for EBNF Left-Parsable Grammars Martin Odersky, LAMP/DI 1 From Regular
More informationParsing. source code. while (k<=n) {sum = sum+k; k=k+1;}
Compiler Construction Grammars Parsing source code scanner tokens regular expressions lexical analysis Lennart Andersson parser context free grammar Revision 2012 01 23 2012 parse tree AST builder (implicit)
More informationCS1622. Today. A Recursive Descent Parser. Preliminaries. Lecture 9 Parsing (4)
CS1622 Lecture 9 Parsing (4) CS 1622 Lecture 9 1 Today Example of a recursive descent parser Predictive & LL(1) parsers Building parse tables CS 1622 Lecture 9 2 A Recursive Descent Parser. Preliminaries
More information8 Parsing. Parsing. Top Down Parsing Methods. Parsing complexity. Top down vs. bottom up parsing. Top down vs. bottom up parsing
8 Parsing Parsing A grammar describes syntactically legal strings in a language A recogniser simply accepts or rejects strings A generator produces strings A parser constructs a parse tree for a string
More informationCourse Overview. Introduction (Chapter 1) Compiler Frontend: Today. Compiler Backend:
Course Overview Introduction (Chapter 1) Compiler Frontend: Today Lexical Analysis & Parsing (Chapter 2,3,4) Semantic Analysis (Chapter 5) Activation Records (Chapter 6) Translation to Intermediate Code
More informationParsing Algorithms. Parsing: continued. Top Down Parsing. Predictive Parser. David Notkin Autumn 2008
Parsing: continued David Notkin Autumn 2008 Parsing Algorithms Earley s algorithm (1970) works for all CFGs O(N 3 ) worst case performance O(N 2 ) for unambiguous grammars Based on dynamic programming,
More informationTop down vs. bottom up parsing
Parsing A grammar describes the strings that are syntactically legal A recogniser simply accepts or rejects strings A generator produces sentences in the language described by the grammar A parser constructs
More informationContext-free grammars (CFG s)
Syntax Analysis/Parsing Purpose: determine if tokens have the right form for the language (right syntactic structure) stream of tokens abstract syntax tree (AST) AST: captures hierarchical structure of
More informationTopic 3: Syntax Analysis I
Topic 3: Syntax Analysis I Compiler Design Prof. Hanjun Kim CoreLab (Compiler Research Lab) POSTECH 1 Back-End Front-End The Front End Source Program Lexical Analysis Syntax Analysis Semantic Analysis
More informationLL(k) Parsing. Predictive Parsers. LL(k) Parser Structure. Sample Parse Table. LL(1) Parsing Algorithm. Push RHS in Reverse Order 10/17/2012
Predictive Parsers LL(k) Parsing Can we avoid backtracking? es, if for a given input symbol and given nonterminal, we can choose the alternative appropriately. his is possible if the first terminal of
More informationDerivations vs Parses. Example. Parse Tree. Ambiguity. Different Parse Trees. Context Free Grammars 9/18/2012
Derivations vs Parses Grammar is used to derive string or construct parser Context ree Grammars A derivation is a sequence of applications of rules Starting from the start symbol S......... (sentence)
More information4 (c) parsing. Parsing. Top down vs. bo5om up parsing
4 (c) parsing Parsing A grammar describes syntac2cally legal strings in a language A recogniser simply accepts or rejects strings A generator produces strings A parser constructs a parse tree for a string
More informationSyntax Analysis/Parsing. Context-free grammars (CFG s) Context-free grammars vs. Regular Expressions. BNF description of PL/0 syntax
Susan Eggers 1 CSE 401 Syntax Analysis/Parsing Context-free grammars (CFG s) Purpose: determine if tokens have the right form for the language (right syntactic structure) stream of tokens abstract syntax
More informationCompila(on (Semester A, 2013/14)
Compila(on 0368-3133 (Semester A, 2013/14) Lecture 4: Syntax Analysis (Top- Down Parsing) Modern Compiler Design: Chapter 2.2 Noam Rinetzky Slides credit: Roman Manevich, Mooly Sagiv, Jeff Ullman, Eran
More informationParsing. Roadmap. > Context-free grammars > Derivations and precedence > Top-down parsing > Left-recursion > Look-ahead > Table-driven parsing
Roadmap > Context-free grammars > Derivations and precedence > Top-down parsing > Left-recursion > Look-ahead > Table-driven parsing The role of the parser > performs context-free syntax analysis > guides
More informationCS 2210 Sample Midterm. 1. Determine if each of the following claims is true (T) or false (F).
CS 2210 Sample Midterm 1. Determine if each of the following claims is true (T) or false (F). F A language consists of a set of strings, its grammar structure, and a set of operations. (Note: a language
More informationSyntax. Syntax. We will study three levels of syntax Lexical Defines the rules for tokens: literals, identifiers, etc.
Syntax Syntax Syntax defines what is grammatically valid in a programming language Set of grammatical rules E.g. in English, a sentence cannot begin with a period Must be formal and exact or there will
More informationCS502: Compilers & Programming Systems
CS502: Compilers & Programming Systems Top-down Parsing Zhiyuan Li Department of Computer Science Purdue University, USA There exist two well-known schemes to construct deterministic top-down parsers:
More informationSyntax/semantics. Program <> program execution Compiler/interpreter Syntax Grammars Syntax diagrams Automata/State Machines Scanning/Parsing
Syntax/semantics Program program execution Compiler/interpreter Syntax Grammars Syntax diagrams Automata/State Machines Scanning/Parsing Meta-models 8/27/10 1 Program program execution Syntax Semantics
More information3. Parsing. Oscar Nierstrasz
3. Parsing Oscar Nierstrasz Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes. http://www.cs.ucla.edu/~palsberg/ http://www.cs.purdue.edu/homes/hosking/
More informationCS 314 Principles of Programming Languages
CS 314 Principles of Programming Languages Lecture 5: Syntax Analysis (Parsing) Zheng (Eddy) Zhang Rutgers University January 31, 2018 Class Information Homework 1 is being graded now. The sample solution
More informationDefining syntax using CFGs
Defining syntax using CFGs Roadmap Last time Defined context-free grammar This time CFGs for specifying a language s syntax Language membership List grammars Resolving ambiguity CFG Review G = (N,Σ,P,S)
More informationBuilding a Parser III. CS164 3:30-5:00 TT 10 Evans. Prof. Bodik CS 164 Lecture 6 1
Building a Parser III CS164 3:30-5:00 TT 10 Evans 1 Overview Finish recursive descent parser when it breaks down and how to fix it eliminating left recursion reordering productions Predictive parsers (aka
More informationCS Parsing 1
CS414-20034-03 Parsing 1 03-0: Parsing Once we have broken an input file into a sequence of tokens, the next step is to determine if that sequence of tokens forms a syntactically correct program parsing
More informationLexical and Syntax Analysis (2)
Lexical and Syntax Analysis (2) In Text: Chapter 4 N. Meng, F. Poursardar Motivating Example Consider the grammar S -> cad A -> ab a Input string: w = cad How to build a parse tree top-down? 2 Recursive-Descent
More informationComp 411 Principles of Programming Languages Lecture 3 Parsing. Corky Cartwright January 11, 2019
Comp 411 Principles of Programming Languages Lecture 3 Parsing Corky Cartwright January 11, 2019 Top Down Parsing What is a context-free grammar (CFG)? A recursive definition of a set of strings; it is
More informationLECTURE 3. Compiler Phases
LECTURE 3 Compiler Phases COMPILER PHASES Compilation of a program proceeds through a fixed series of phases. Each phase uses an (intermediate) form of the program produced by an earlier phase. Subsequent
More informationLecture 14: Parser Conflicts, Using Ambiguity, Error Recovery. Last modified: Mon Feb 23 10:05: CS164: Lecture #14 1
Lecture 14: Parser Conflicts, Using Ambiguity, Error Recovery Last modified: Mon Feb 23 10:05:56 2015 CS164: Lecture #14 1 Shift/Reduce Conflicts If a DFA state contains both [X: α aβ, b] and [Y: γ, a],
More informationSyntax Analysis. COMP 524: Programming Language Concepts Björn B. Brandenburg. The University of North Carolina at Chapel Hill
Syntax Analysis Björn B. Brandenburg The University of North Carolina at Chapel Hill Based on slides and notes by S. Olivier, A. Block, N. Fisher, F. Hernandez-Campos, and D. Stotts. The Big Picture Character
More informationFall Compiler Principles Lecture 2: LL parsing. Roman Manevich Ben-Gurion University of the Negev
Fall 2017-2018 Compiler Principles Lecture 2: LL parsing Roman Manevich Ben-Gurion University of the Negev 1 Books Compilers Principles, Techniques, and Tools Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman
More informationCPS 506 Comparative Programming Languages. Syntax Specification
CPS 506 Comparative Programming Languages Syntax Specification Compiling Process Steps Program Lexical Analysis Convert characters into a stream of tokens Lexical Analysis Syntactic Analysis Send tokens
More informationJavaCC Parser. The Compilation Task. Automated? JavaCC Parser
JavaCC Parser The Compilation Task Input character stream Lexer stream Parser Abstract Syntax Tree Analyser Annotated AST Code Generator Code CC&P 2003 1 CC&P 2003 2 Automated? JavaCC Parser The initial
More informationProgramming Language Specification and Translation. ICOM 4036 Fall Lecture 3
Programming Language Specification and Translation ICOM 4036 Fall 2009 Lecture 3 Some parts are Copyright 2004 Pearson Addison-Wesley. All rights reserved. 3-1 Language Specification and Translation Topics
More informationCSE 401 Midterm Exam Sample Solution 2/11/15
Question 1. (10 points) Regular expression warmup. For regular expression questions, you must restrict yourself to the basic regular expression operations covered in class and on homework assignments:
More informationCOP4020 Programming Languages. Syntax Prof. Robert van Engelen
COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview Tokens and regular expressions Syntax and context-free grammars Grammar derivations More about parse trees Top-down and bottom-up
More informationCSE 130 Programming Language Principles & Paradigms Lecture # 5. Chapter 4 Lexical and Syntax Analysis
Chapter 4 Lexical and Syntax Analysis Introduction - Language implementation systems must analyze source code, regardless of the specific implementation approach - Nearly all syntax analysis is based on
More information1 Introduction. 2 Recursive descent parsing. Predicative parsing. Computer Language Implementation Lecture Note 3 February 4, 2004
CMSC 51086 Winter 2004 Computer Language Implementation Lecture Note 3 February 4, 2004 Predicative parsing 1 Introduction This note continues the discussion of parsing based on context free languages.
More informationParsing III. CS434 Lecture 8 Spring 2005 Department of Computer Science University of Alabama Joel Jones
Parsing III (Top-down parsing: recursive descent & LL(1) ) (Bottom-up parsing) CS434 Lecture 8 Spring 2005 Department of Computer Science University of Alabama Joel Jones Copyright 2003, Keith D. Cooper,
More informationRevisit the example. Transformed DFA 10/1/16 A B C D E. Start
Revisit the example ε 0 ε 1 Start ε a ε 2 3 ε b ε 4 5 ε a b b 6 7 8 9 10 ε-closure(0)={0, 1, 2, 4, 7} = A Trans(A, a) = {1, 2, 3, 4, 6, 7, 8} = B Trans(A, b) = {1, 2, 4, 5, 6, 7} = C Trans(B, a) = {1,
More informationCOP4020 Programming Languages. Syntax Prof. Robert van Engelen
COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview n Tokens and regular expressions n Syntax and context-free grammars n Grammar derivations n More about parse trees n Top-down and
More informationParsers. Xiaokang Qiu Purdue University. August 31, 2018 ECE 468
Parsers Xiaokang Qiu Purdue University ECE 468 August 31, 2018 What is a parser A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure
More informationCA Compiler Construction
CA4003 - Compiler Construction David Sinclair A top-down parser starts with the root of the parse tree, labelled with the goal symbol of the grammar, and repeats the following steps until the fringe of
More informationICOM 4036 Spring 2004
Language Specification and Translation ICOM 4036 Spring 2004 Lecture 3 Copyright 2004 Pearson Addison-Wesley. All rights reserved. 3-1 Language Specification and Translation Topics Structure of a Compiler
More informationSYNTAX ANALYSIS 1. Define parser. Hierarchical analysis is one in which the tokens are grouped hierarchically into nested collections with collective meaning. Also termed as Parsing. 2. Mention the basic
More informationWednesday, September 9, 15. Parsers
Parsers What is a parser A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure of a program (think: diagramming a sentence) Agenda
More informationParsers. What is a parser. Languages. Agenda. Terminology. Languages. A parser has two jobs:
What is a parser Parsers A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure of a program (think: diagramming a sentence) Agenda
More informationSyntactic Analysis. Syntactic analysis, or parsing, is the second phase of compilation: The token file is converted to an abstract syntax tree.
Syntactic Analysis Syntactic analysis, or parsing, is the second phase of compilation: The token file is converted to an abstract syntax tree. Compiler Passes Analysis of input program (front-end) character
More informationFall Compiler Principles Lecture 2: LL parsing. Roman Manevich Ben-Gurion University of the Negev
Fall 2016-2017 Compiler Principles Lecture 2: LL parsing Roman Manevich Ben-Gurion University of the Negev 1 Books Compilers Principles, Techniques, and Tools Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman
More informationSyntax. In Text: Chapter 3
Syntax In Text: Chapter 3 1 Outline Syntax: Recognizer vs. generator BNF EBNF Chapter 3: Syntax and Semantics 2 Basic Definitions Syntax the form or structure of the expressions, statements, and program
More information10/5/17. Lexical and Syntactic Analysis. Lexical and Syntax Analysis. Tokenizing Source. Scanner. Reasons to Separate Lexical and Syntax Analysis
Lexical and Syntactic Analysis Lexical and Syntax Analysis In Text: Chapter 4 Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input characters and output
More informationPrinciples of Programming Languages COMP251: Syntax and Grammars
Principles of Programming Languages COMP251: Syntax and Grammars Prof. Dekai Wu Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong, China Fall 2006
More informationCS 230 Programming Languages
CS 230 Programming Languages 10 / 16 / 2013 Instructor: Michael Eckmann Today s Topics Questions/comments? Top Down / Recursive Descent Parsers Top Down Parsers We have a left sentential form xa Expand
More informationCompiler Passes. Syntactic Analysis. Context-free Grammars. Syntactic Analysis / Parsing. EBNF Syntax of initial MiniJava.
Syntactic Analysis Syntactic analysis, or parsing, is the second phase of compilation: The token file is converted to an abstract syntax tree. Compiler Passes Analysis of input program (front-end) character
More informationAbstract Syntax Trees & Top-Down Parsing
Abstract Syntax Trees & Top-Down Parsing Review of Parsing Given a language L(G), a parser consumes a sequence of tokens s and produces a parse tree Issues: How do we recognize that s L(G)? A parse tree
More informationAbstract Syntax Trees & Top-Down Parsing
Review of Parsing Abstract Syntax Trees & Top-Down Parsing Given a language L(G), a parser consumes a sequence of tokens s and produces a parse tree Issues: How do we recognize that s L(G)? A parse tree
More informationNote that for recursive descent to work, if A ::= B1 B2 is a grammar rule we need First k (B1) disjoint from First k (B2).
LL(k) Grammars We need a bunch of terminology. For any terminal string a we write First k (a) is the prefix of a of length k (or all of a if its length is less than k) For any string g of terminal and
More information4. Lexical and Syntax Analysis
4. Lexical and Syntax Analysis 4.1 Introduction Language implementation systems must analyze source code, regardless of the specific implementation approach Nearly all syntax analysis is based on a formal
More informationThe Parsing Problem (cont d) Recursive-Descent Parsing. Recursive-Descent Parsing (cont d) ICOM 4036 Programming Languages. The Complexity of Parsing
ICOM 4036 Programming Languages Lexical and Syntax Analysis Lexical Analysis The Parsing Problem Recursive-Descent Parsing Bottom-Up Parsing This lecture covers review questions 14-27 This lecture covers
More informationTypes of parsing. CMSC 430 Lecture 4, Page 1
Types of parsing Top-down parsers start at the root of derivation tree and fill in picks a production and tries to match the input may require backtracking some grammars are backtrack-free (predictive)
More informationCS 536 Midterm Exam Spring 2013
CS 536 Midterm Exam Spring 2013 ID: Exam Instructions: Write your student ID (not your name) in the space provided at the top of each page of the exam. Write all your answers on the exam itself. Feel free
More information10/4/18. Lexical and Syntactic Analysis. Lexical and Syntax Analysis. Tokenizing Source. Scanner. Reasons to Separate Lexical and Syntactic Analysis
Lexical and Syntactic Analysis Lexical and Syntax Analysis In Text: Chapter 4 Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input characters and output
More informationAbstract Syntax Trees & Top-Down Parsing
Review of Parsing Abstract Syntax Trees & Top-Down Parsing Given a language L(G), a parser consumes a sequence of tokens s and produces a parse tree Issues: How do we recognize that s L(G)? A parse tree
More informationCSCI312 Principles of Programming Languages
Copyright 2006 The McGraw-Hill Companies, Inc. CSCI312 Principles of Programming Languages! LL Parsing!! Xu Liu Derived from Keith Cooper s COMP 412 at Rice University Recap Copyright 2006 The McGraw-Hill
More informationParsing II Top-down parsing. Comp 412
COMP 412 FALL 2018 Parsing II Top-down parsing Comp 412 source code IR Front End Optimizer Back End IR target code Copyright 2018, Keith D. Cooper & Linda Torczon, all rights reserved. Students enrolled
More information1. Explain the input buffer scheme for scanning the source program. How the use of sentinels can improve its performance? Describe in detail.
Code No: R05320502 Set No. 1 1. Explain the input buffer scheme for scanning the source program. How the use of sentinels can improve its performance? Describe in detail. 2. Construct predictive parsing
More information4. Lexical and Syntax Analysis
4. Lexical and Syntax Analysis 4.1 Introduction Language implementation systems must analyze source code, regardless of the specific implementation approach Nearly all syntax analysis is based on a formal
More informationAdministrativia. WA1 due on Thu PA2 in a week. Building a Parser III. Slides on the web site. CS164 3:30-5:00 TT 10 Evans.
Administrativia Building a Parser III CS164 3:30-5:00 10 vans WA1 due on hu PA2 in a week Slides on the web site I do my best to have slides ready and posted by the end of the preceding logical day yesterday,
More informationSyntax Analysis, III Comp 412
COMP 412 FALL 2017 Syntax Analysis, III Comp 412 source code IR Front End Optimizer Back End IR target code Copyright 2017, Keith D. Cooper & Linda Torczon, all rights reserved. Students enrolled in Comp
More informationSyntax Analysis. The Big Picture. The Big Picture. COMP 524: Programming Languages Srinivas Krishnan January 25, 2011
Syntax Analysis COMP 524: Programming Languages Srinivas Krishnan January 25, 2011 Based in part on slides and notes by Bjoern Brandenburg, S. Olivier and A. Block. 1 The Big Picture Character Stream Token
More informationDefining syntax using CFGs
Defining syntax using CFGs Roadmap Last 8me Defined context-free grammar This 8me CFGs for syntax design Language membership List grammars Resolving ambiguity CFG Review G = (N,Σ,P,S) means derives derives
More informationA Simple Syntax-Directed Translator
Chapter 2 A Simple Syntax-Directed Translator 1-1 Introduction The analysis phase of a compiler breaks up a source program into constituent pieces and produces an internal representation for it, called
More information3. Parsing. 3.1 Context-Free Grammars and Push-Down Automata 3.2 Recursive Descent Parsing 3.3 LL(1) Property 3.4 Error Handling
3. Parsing 3.1 Context-Free Grammars and Push-Down Automata 3.2 Recursive Descent Parsing 3.3 LL(1) Property 3.4 Error Handling 1 Context-Free Grammars Problem Regular Grammars cannot handle central recursion
More informationPart 3. Syntax analysis. Syntax analysis 96
Part 3 Syntax analysis Syntax analysis 96 Outline 1. Introduction 2. Context-free grammar 3. Top-down parsing 4. Bottom-up parsing 5. Conclusion and some practical considerations Syntax analysis 97 Structure
More informationBuilding Compilers with Phoenix
Building Compilers with Phoenix Parser Generators: ANTLR History of ANTLR ANother Tool for Language Recognition Terence Parr's dissertation: Obtaining Practical Variants of LL(k) and LR(k) for k > 1 PCCTS:
More informationCompilers. Predictive Parsing. Alex Aiken
Compilers Like recursive-descent but parser can predict which production to use By looking at the next fewtokens No backtracking Predictive parsers accept LL(k) grammars L means left-to-right scan of input
More informationSyntax Analysis: Context-free Grammars, Pushdown Automata and Parsing Part - 4. Y.N. Srikant
Syntax Analysis: Context-free Grammars, Pushdown Automata and Part - 4 Department of Computer Science and Automation Indian Institute of Science Bangalore 560 012 NPTEL Course on Principles of Compiler
More informationCOMP3131/9102: Programming Languages and Compilers
COMP3131/9102: Programming Languages and Compilers Jingling Xue School of Computer Science and Engineering The University of New South Wales Sydney, NSW 2052, Australia http://www.cse.unsw.edu.au/~cs3131
More informationChapter 3. Parsing #1
Chapter 3 Parsing #1 Parser source file get next character scanner get token parser AST token A parser recognizes sequences of tokens according to some grammar and generates Abstract Syntax Trees (ASTs)
More informationSyntax Analysis, III Comp 412
Updated algorithm for removal of indirect left recursion to match EaC3e (3/2018) COMP 412 FALL 2018 Midterm Exam: Thursday October 18, 7PM Herzstein Amphitheater Syntax Analysis, III Comp 412 source code
More informationChapter 3. Describing Syntax and Semantics ISBN
Chapter 3 Describing Syntax and Semantics ISBN 0-321-49362-1 Chapter 3 Topics Introduction The General Problem of Describing Syntax Formal Methods of Describing Syntax Copyright 2009 Addison-Wesley. All
More informationParsing. Lecture 11: Parsing. Recursive Descent Parser. Arithmetic grammar. - drops irrelevant details from parse tree
Parsing Lecture 11: Parsing CSC 131 Fall, 2014 Kim Bruce Build parse tree from an expression Interested in abstract syntax tree - drops irrelevant details from parse tree Arithmetic grammar ::=
More informationSometimes an ambiguous grammar can be rewritten to eliminate the ambiguity.
Eliminating Ambiguity Sometimes an ambiguous grammar can be rewritten to eliminate the ambiguity. Example: consider the following grammar stat if expr then stat if expr then stat else stat other One can
More informationChapter 4. Lexical and Syntax Analysis
Chapter 4 Lexical and Syntax Analysis Chapter 4 Topics Introduction Lexical Analysis The Parsing Problem Recursive-Descent Parsing Bottom-Up Parsing Copyright 2012 Addison-Wesley. All rights reserved.
More informationFall Compiler Principles Lecture 3: Parsing part 2. Roman Manevich Ben-Gurion University
Fall 2014-2015 Compiler Principles Lecture 3: Parsing part 2 Roman Manevich Ben-Gurion University Tentative syllabus Front End Intermediate Representation Optimizations Code Generation Scanning Lowering
More informationSyntax Analysis. Martin Sulzmann. Martin Sulzmann Syntax Analysis 1 / 38
Syntax Analysis Martin Sulzmann Martin Sulzmann Syntax Analysis 1 / 38 Syntax Analysis Objective Recognize individual tokens as sentences of a language (beyond regular languages). Example 1 (OK) Program
More informationContext-free grammars
Context-free grammars Section 4.2 Formal way of specifying rules about the structure/syntax of a program terminals - tokens non-terminals - represent higher-level structures of a program start symbol,
More informationCSCI 1260: Compilers and Program Analysis Steven Reiss Fall Lecture 4: Syntax Analysis I
CSCI 1260: Compilers and Program Analysis Steven Reiss Fall 2015 Lecture 4: Syntax Analysis I I. Syntax Analysis A. Breaking the program into logical units 1. Input: token stream 2. Output: representation
More informationCSE 3302 Programming Languages Lecture 2: Syntax
CSE 3302 Programming Languages Lecture 2: Syntax (based on slides by Chengkai Li) Leonidas Fegaras University of Texas at Arlington CSE 3302 L2 Spring 2011 1 How do we define a PL? Specifying a PL: Syntax:
More informationOutline. Top Down Parsing. SLL(1) Parsing. Where We Are 1/24/2013
Outline Top Down Parsing Top-down parsing SLL(1) grammars Transforming a grammar into SLL(1) form Recursive-descent parsing 1 CS 412/413 Spring 2008 Introduction to Compilers 2 Where We Are SLL(1) Parsing
More informationTop-Down Parsing and Intro to Bottom-Up Parsing. Lecture 7
Top-Down Parsing and Intro to Bottom-Up Parsing Lecture 7 1 Predictive Parsers Like recursive-descent but parser can predict which production to use Predictive parsers are never wrong Always able to guess
More informationLecture 8: Deterministic Bottom-Up Parsing
Lecture 8: Deterministic Bottom-Up Parsing (From slides by G. Necula & R. Bodik) Last modified: Fri Feb 12 13:02:57 2010 CS164: Lecture #8 1 Avoiding nondeterministic choice: LR We ve been looking at general
More informationLexical and Syntax Analysis. Top-Down Parsing
Lexical and Syntax Analysis Top-Down Parsing Easy for humans to write and understand String of characters Lexemes identified String of tokens Easy for programs to transform Data structure Syntax A syntax
More informationParsing III. (Top-down parsing: recursive descent & LL(1) )
Parsing III (Top-down parsing: recursive descent & LL(1) ) Roadmap (Where are we?) Previously We set out to study parsing Specifying syntax Context-free grammars Ambiguity Top-down parsers Algorithm &
More information