COP4020 Programming Languages. Syntax Prof. Robert van Engelen
|
|
- Ashlyn Palmer
- 5 years ago
- Views:
Transcription
1 COP4020 Programming Languages Syntax Prof. Robert van Engelen
2 Overview n Tokens and regular expressions n Syntax and context-free grammars n Grammar derivations n More about parse trees n Top-down and bottom-up parsing n Recursive descent parsing COP4020 Fall
3 Tokens n n n Tokens are the basic building blocks of a programming language Keywords, identifiers, literal values, operators, punctuation We saw that the first compiler phase (scanning) splits up a character stream into tokens Tokens have a special role with respect to: Free-format languages: source program is a sequence of tokens and horizontal/vertical position of a token on a page is unimportant (e.g. Pascal) Fixed-format languages: indentation and/or position of a token on a page is significant (early Basic, Fortran, Haskell) Case-sensitive languages: upper- and lowercase are distinct (C, C++, Java) Case-insensitive languages: upper- and lowercase are identical (Ada, Fortran, Pascal) COP4020 Fall
4 Defining Token Patterns with Regular Expressions n The makeup of a token is described by a regular expression (RE) n A regular expression r is one of A character (an element of the alphabet Σ), e.g. Σ = {a, b, c} a Empty, denoted by ε Concatenation: a sequence of regular expressions r 1 r 2 r 3 r n Alternation: regular expressions separated by a bar r 1 r 2 Repetition: a regular expression followed by a star (Kleene star) r* COP4020 Fall
5 Example Regular Definitions for Tokens n digit n unsigned_integer digit digit* n signed_integer (+ - ε) unsigned_integer n relop < <= <> > >= = n letter a b z A B Z n id letter (letter digit)* n Cannot use recursive definitions! digits digit digits digit COP4020 Fall
6 Finite State Machines = Regular Expression Recognizers relop < <= <> > >= = id letter ( letter digit ) * start 0 < 1 = 2 return(relop, LE) > 3 return(relop, NE) other 4 * return(relop, LT) = 5 return(relop, EQ) > 6 = 7 return(relop, GE) other 8 * return(relop, GT) letter or digit start 9 letter other * return(gettoken(), install_id()) COP4020 Fall
7 Non-Deterministic Finite State Automata n An NFA is a 5-tuple (S, Σ, δ, s 0, F) where S is a finite set of states Σ is a finite set of symbols, the alphabet δ is a mapping from S Σ to a set of states s 0 S is the start state F S is the set of accepting (or final) states start 0 a b a b b S = {0,1,2,3} Σ = {a,b} s 0 = 0 F = {3} COP4020 Fall
8 From a Regular Expression to an NFA ε start i ε f a start i a f r 1 r 2 r 1 r 2 start i ε ε N(r 1 ) N(r 2 ) start i N(r 1 ) N(r 2 ) f r* start i ε N(r) ε COP4020 Fall ε ε ε ε f f
9 Grammars n Context-free grammar has four components: T is a finite set of tokens (terminal symbols) N is a finite set of nonterminals P is a finite set of productions of the form α β where α (N T)* N (N T)* and β (N T)* S N is a designated start symbol COP4020 Fall
10 Context Free Grammars: BNF Notation n Regular expressions cannot describe nested constructs, but context-free grammars can n Backus-Naur Form (BNF) notation for productions: <nonterminal> ::= sequence of (non)terminals where Each terminal in the grammar is also a token A <nonterminal> defines a syntactic category The symbol denotes alternative forms in a production The special symbol ε denotes empty COP4020 Fall
11 Example <Program> ::= program <id> ( <id> <More_ids> ) ; <Block>. <Block> ::= <Variables> begin <Stmt> <More_Stmts> end <More_ids> ::=, <id> <More_ids> ε <Variables> ::= var <id> <More_ids> : <Type> ; <More_Variables> ε <More_Variables> ::= <id> <More_ids> : <Type> ; <More_Variables> ε <Stmt> ::= <id> := <Exp> if <Exp> then <Stmt> else <Stmt> while <Exp> do <Stmt> begin <Stmt> <More_Stmts> end <More_Stmts> ::= ; <Stmt> <More_Stmts> ε <Exp> ::= <num> <id> <Exp> + <Exp> <Exp> - <Exp> COP4020 Fall
12 Extended BNF n Extended BNF simplifies grammar definitions n Extended BNF adds Optional constructs with [ and ] Repetitions with [ ]* Some EBNF definitions also add [ ] + for non-zero repetitions n Any extended BNF grammar can be rewritten into BNF COP4020 Fall
13 Example <Program> ::= program <id> ( <id> [, <id> ]* ) ; <Block>. <Block> ::= [ <Variables> ] begin <Stmt> [ ; <Stmt> ]* end <Variables> ::= var [ <id> [, <id> ]* : <Type> ; ] + <Stmt> ::= <id> := <Exp> if <Exp> then <Stmt> else <Stmt> while <Exp> do <Stmt> begin <Stmt> [ ; <Stmt> ]* end <Exp> ::= <num> <id> <Exp> + <Exp> <Exp> - <Exp> COP4020 Fall
14 Derivations n From a grammar we can derive strings (= sequences of tokens) The opposite process of parsing n Starting with the grammar s designated start symbol, in each derivation step a nonterminal is replaced by a righthand side of a production for that nonterminal COP4020 Fall
15 Example Derivation Start symbol Sentential forms <expression> ::= identifier unsigned_integer - <expression> ( <expression> ) <expression> <operator> <expression> <operator> ::= + - * / <expression> <expression> <operator> <expression> <expression> <operator> identifier <expression> + identifier <expression> <operator> <expression> + identifier <expression> <operator> identifier + identifier <expression> * identifier + identifier identifier * identifier + identifier Replacement of nonterminal with one of its productions The final string is the yield COP4020 Fall
16 n Rightmost versus Leftmost Derivations When the nonterminal on the far right (left) in a sentential form is replaced in each derivation step the derivation is called right-most (left-most) <expression> <expression> <operator> <expression> <expression> <operator> identifier Replace in rightmost derivation Replace in rightmost derivation Replace in leftmost derivation <expression> <expression> <operator> <expression> identifier <operator> <expression> Replace in leftmost derivation COP4020 Fall
17 A Language Generated by a Grammar n n A context-free grammar is a generator of a context-free language The language defined by a grammar G is the set of all strings w that can be derived from the start symbol S L(G) = { w S * w } <S> ::= a ( <S> ) L(G) = { set of all strings a (a) ((a)) (((a))) } <S> ::= <B> <C> <B> ::= <C> + <C> <C> ::= 0 1 L(G) = { 0+0, 0+1, 1+0, 1+1, 0, 1 } COP4020 Fall
18 Parse Trees n A parse tree depicts the end result of a derivation The internal nodes are the nonterminals The children of a node are the symbols (terminals and nonterminals) on a right-hand side of a production The leaves are the terminals <expression> <expression> <operator> <expression> <expression> <operator> <expression> identifier * identifier + identifier COP4020 Fall
19 Parse Trees <expression> <expression> <operator> <expression> <expression> <operator> identifier <expression> + identifier <expression> <operator> <expression> + identifier <expression> <operator> identifier + identifier <expression> * identifier + identifier identifier * identifier + identifier <expression> <expression> <operator> <expression> <expression> <operator> <expression> identifier * identifier + identifier COP4020 Fall
20 Ambiguity n There is another parse tree for the same grammar and input: the grammar is ambiguous n This parse tree is not desired, since it appears that + has precedence over * <expression> <expression> <operator> <expression> <expression> <operator> <expression> identifier * identifier + identifier COP4020 Fall
21 Ambiguous Grammars n Ambiguous grammar: more than one distinct derivation of a string results in different parse trees n A programming language construct should have only one parse tree to avoid misinterpretation by a compiler n For expression grammars, associativity and precedence of operators is used to disambiguate <expression> ::= <term> <expression> <add_op> <term> <term> ::= <factor> <term> <mult_op> <factor> <factor> ::= identifier unsigned_integer - <factor> ( <expression> ) <add_op> ::= + - <mult_op> ::= * / COP4020 Fall
22 Ambiguous if-then-else: the Dangling Else n A classical example of an ambiguous grammar are the grammar productions for if-then-else: <stmt> ::= if <expr> then <stmt> if <expr> then <stmt> else <stmt> n It is possible to hack this into unambiguous productions for the same syntax, but the fact that it is not easy indicates a problem in the programming language design n Ada uses different syntax to avoid ambiguity: <stmt> ::= if <expr> then <stmt> end if if <expr> then <stmt> else <stmt> end if COP4020 Fall
23 Linear-Time Top-Down and Bottom-Up Parsing n A parser is a recognizer for a context-free language n A string (token sequence) is accepted by the parser and a parse tree can be constructed if the string is in the language n For any arbitrary context-free grammar parsing can take as much as O(n 3 ) time, where n is the size of the input n There are large classes of grammars for which we can construct parsers that take O(n) time: Top-down LL parsers for LL grammars (LL = Left-to-right scanning of input, Left-most derivation) Bottom-up LR parsers for LR grammars (LR = Left-to-right scanning of input, Right-most derivation) COP4020 Fall
24 Top-Down Parsers and LL Grammars n n n Top-down parser is a parser for LL class of grammars Also called predictive parser LL class is a strict subset of the larger LR class of grammars LL grammars cannot contain left-recursive productions (but LR can), for example: <X> ::= <X> <Y> and <X> ::= <Y> <Z> <Y> ::= <X> LL(k) where k is lookahead depth, if k=1 cannot handle alternatives in productions with common prefixes <X> ::= a b a c A top-down parser constructs a parse tree from the root down Not too difficult to implement a predictive parser for an unambiguous LL(1) grammar in BNF by hand using recursive descent COP4020 Fall
25 Top-Down Parser in Action <id_list> ::= id <id_list_tail> <id_list_tail>::=, id <id_list_tail> ; A, B, C; A, B, C; A, B, C; A, B, C; COP4020 Fall
26 Top-Down Predictive Parsing n Top-down parsing is called predictive parsing because parser predicts what it is going to see: 1. As root, the start symbol of the grammar <id_list> is predicted 2. After reading A the parser predicts that <id_list_tail> must follow 3. After reading, and B the parser predicts that <id_list_tail> must follow 4. After reading, and C the parser predicts that <id_list_tail> must follow 5. After reading ; the parser stops COP4020 Fall
27 An Ambiguous Non-LL Grammar for Language E n Consider a language E of simple expressions composed of +, -, *, /, (), id, and num <expr> ::= <expr> + <expr> <expr> - <expr> <expr> * <expr> <expr> / <expr> ( <expr> ) <id> <num> n Need operator precedence rules COP4020 Fall
28 An Unambiguous Non-LL Grammar for Language E <expr> ::= <expr> + <term> <expr> - <term> <term> <term> ::= <term> * <factor> <term> / <factor> <factor> <factor> ::= ( <expr> ) <id> <num> COP4020 Fall
29 An Unambiguous LL(1) Grammar for Language E <expr> ::= <term> <term_tail> <term> ::= <factor> <factor_tail> <term_tail> ::= <add_op> <term> <term_tail> ε <factor> ::= ( <expr> ) <id> <num> <factor_tail> ::= <mult_op> <factor> <factor_tail> ε <add_op> ::= + - <mult_op> ::= * / COP4020 Fall
30 Constructing Recursive Descent Parsers for LL(1) n n Each nonterminal has a function that implements the production(s) for that nonterminal The function parses only the part of the input described by the nonterminal <expr> ::= <term> <term_tail> procedure expr() term(); term_tail(); n When more than one alternative production exists for a nonterminal, the lookahead token should help to decide which production to apply <term_tail> ::= <add_op> <term> <term_tail> procedure term_tail() ε case (input_token()) of '+' or '-': add_op(); term(); term_tail(); otherwise: /* no op = ε */ COP4020 Fall
31 Some Rules to Construct a Recursive Descent Parser n For every nonterminal with more than one production, find all the tokens that each of the right-hand sides can start with (called the FIRST set): <X> ::= a starts with a b a <Z> starts with b <Y> starts with c or d <Z> f starts with e or f <Y> ::= c d <Z> ::= e ε n Empty productions are coded as skip operations (nops) n If a nonterminal does not have an empty production, the function should generate an error if no token matches COP4020 Fall
32 Example for E procedure expr() term(); term_tail(); procedure term_tail() case (input_token()) of '+' or '-': add_op(); term(); term_tail(); otherwise: /* no op = ε */ procedure term() factor(); factor_tail(); procedure factor_tail() case (input_token()) of '*' or '/': mult_op(); factor(); factor_tail(); otherwise: /* no op = ε */ procedure factor() case (input_token()) of '(': match('('); expr(); match(')'); of identifier: match(identifier); of number: match(number); otherwise: error; procedure add_op() case (input_token()) of '+': match('+'); of '-': match('-'); otherwise: error; procedure mult_op() case (input_token()) of '*': match('*'); of '/': match('/'); otherwise: error; COP4020 Fall
33 Recursive Descent Parser s Call Graph = Parse Tree n The dynamic call graph of a recursive descent parser corresponds exactly to the parse tree n Call graph of input string 1+2*3 COP4020 Fall
34 Example <type> ::= <simple> ^ id array [ <simple> ] of <type> <simple> ::= integer char num dotdot num COP4020 Fall
35 Example (cont d) <type> ::= <simple> ^ id array [ <simple> ] of <type> <simple> ::= integer char num dotdot num The FIRST sets {integer, char, num} {^} {array} {integer} {char} {num} COP4020 Fall
36 Example (cont d) procedure match(t : token) if input_token() = t then nexttoken(); else error; procedure type() case (input_token()) of integer or char or num : simple(); of ^ : match( ^ ); match(id); of array : match( array ); match( [ ); simple(); match( ] ); match( of ); type(); otherwise: error; procedure simple() case (input_token()) of integer : match( integer ); of char : match( char ); of num : match( num ); match( dotdot ); match( num ); otherwise: error; COP4020 Fall
37 Step 1 Check lookahead and call match type() match( array ) Input: array [ num dotdot num ] of integer lookahead COP4020 Fall
38 Step 2 type() match( array ) match( [ ) Input: array [ num dotdot num ] of integer lookahead COP4020 Fall
39 Step 3 type() match( array ) match( [ ) simple() match( num ) Input: array [ num dotdot num ] of integer lookahead COP4020 Fall
40 Step 4 type() match( array ) match( [ ) simple() match( num ) match( dotdot ) Input: array [ num dotdot num ] of integer lookahead COP4020 Fall
41 Step 5 type() match( array ) match( [ ) simple() match( num ) match( dotdot ) match( num ) Input: array [ num dotdot num ] of integer lookahead COP4020 Fall
42 Step 6 type() match( array ) match( [ ) simple() match( ] ) match( num ) match( dotdot ) match( num ) Input: array [ num dotdot num ] of integer lookahead COP4020 Fall
43 Step 7 type() match( array ) match( [ ) simple() match( ] ) match( of ) match( num ) match( dotdot ) match( num ) Input: array [ num dotdot num ] of integer lookahead COP4020 Fall
44 Step 8 type() match( array ) match( [ ) simple() match( ] ) match( of ) type() match( num ) match( dotdot ) match( num ) simple() Input: match( integer ) array [ num dotdot num ] of integer lookahead COP4020 Fall
45 Bottom-Up LR Parsing n Bottom-up parser is a parser for LR class of grammars n Also called shift-reduce parsing: shift tokens on a stack until the parser recognizes a right-hand side of a production which it then reduces to a left-hand side (nonterminal) to form a partial parse tree n Difficult to implement by hand n Tools (e.g. Yacc/Bison) exist that generate bottom-up parsers for LALR grammars automatically COP4020 Fall
46 Shift-Reduce Parsing Grammar: <S> a <A> <B> e <A> <A> b c b <B> d These match production right-hand side Reducing a sentence: a b b c d e a <A> b c d e a <A> d e a <A> <B> e <S> Shift-reduce corresponds to a rightmost derivation: <S> rm a <A> <B> e rm a <A> d e rm a <A> b c d e rm a b b c d e <S> <A> <A> <A> <A> <A> <A> <B> <A> <B> a b b c d e 46 a b b c d e a b b c d e a b b c d e
47 Model of an LR Parser input a 1 a 2 a i a n $ stack s m X m s m-1 LR Parsing Program (driver) output 47 X m-1 s 0 action shift reduce accept error goto DFA Constructed with LR(1) or LALR(1)
48 Bottom-Up Parser in Action <id_list> ::= id <id_list_tail> <id_list_tail>::=, id <id_list_tail> ; Shift Shift Shift Shift Shift Shift input A, B, C; A A, B, C; A, A, B, C; A,B stack A, B, C; A,B, A, B, C; A,B,C A, B, C; A,B,C; Stack contains shifted input (DFA states not shown) parse tree Reduce A, B, C; A,B,C Cont d Matches production right-hand side COP4020 Fall
49 Reduce Reduce A, B, C; A,B,C A, B, C; A,B These match production right-hand sides Reduce A, B, C; A A, B, C; Accept COP4020 Fall
50 LR(1) Table s means shift and goto state # r means reduce by production # acc means accept and stop state char on input id * = $ s5 s4 acc s6 r6 goto state S L R Grammar: 1. S S 2. S L = R 3. S R 4. L * R 5. L id 6. R L s5 s4 s12 s11 r5 r4 r6 r3 r5 r4 r6 r r6 11 s12 s r r4
51 I 0 : [S S, $] goto(i 0,S)=I 1 [S L=R, $] goto(i 0,L)=I 2 [S R, $] goto(i 0,R)=I 3 [L *R, =/$] goto(i 0,*)=I 4 [L id, =/$] goto(i 0,id)=I 5 [R L, $] goto(i 0,L)=I 2 I 6 : I 7 : [S L= R, $] goto(i 6,R)=I 9 [R L, $] goto(i 6,L)=I 10 [L *R, $] goto(i 6,*)=I 11 [L id, $] goto(i 6,id)=I 12 [L *R, =/$] I 1 : [S S, $] I 8 : [R L, =/$] I 2 : I 3 : I 4 : I 5 : 51 [S L =R, $] goto(i 0,=)=I 6 [R L, $] [S R, $] [L * R, =/$] goto(i 4,R)=I 7 [R L, =/$] goto(i 4,L)=I 8 [L *R, =/$] goto(i 4,*)=I 4 [L id, =/$] goto(i 4,id)=I 5 [L id, =/$] I 9 : I 10 : I 11 : I 12 : I 13 : [S L=R, $] [R L, $] [L * R, $] goto(i 11,R)=I 13 [R L, $] goto(i 11,L)=I 10 [L *R, $] goto(i 11,*)=I 11 [L id, $] goto(i 11,id)=I 12 [L id, $] [L *R, $] The set of LR(1) items constructed for the grammar
COP4020 Programming Languages. Syntax Prof. Robert van Engelen
COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview Tokens and regular expressions Syntax and context-free grammars Grammar derivations More about parse trees Top-down and bottom-up
More informationSyntax. Syntax. We will study three levels of syntax Lexical Defines the rules for tokens: literals, identifiers, etc.
Syntax Syntax Syntax defines what is grammatically valid in a programming language Set of grammatical rules E.g. in English, a sentence cannot begin with a period Must be formal and exact or there will
More informationProgramming Languages (CS 550) Lecture 4 Summary Scanner and Parser Generators. Jeremy R. Johnson
Programming Languages (CS 550) Lecture 4 Summary Scanner and Parser Generators Jeremy R. Johnson 1 Theme We have now seen how to describe syntax using regular expressions and grammars and how to create
More informationChapter 2 - Programming Language Syntax. September 20, 2017
Chapter 2 - Programming Language Syntax September 20, 2017 Specifying Syntax: Regular expressions and context-free grammars Regular expressions are formed by the use of three mechanisms Concatenation Alternation
More informationCSE 3302 Programming Languages Lecture 2: Syntax
CSE 3302 Programming Languages Lecture 2: Syntax (based on slides by Chengkai Li) Leonidas Fegaras University of Texas at Arlington CSE 3302 L2 Spring 2011 1 How do we define a PL? Specifying a PL: Syntax:
More informationCS415 Compilers. Syntax Analysis. These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University
CS415 Compilers Syntax Analysis These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Limits of Regular Languages Advantages of Regular Expressions
More informationTHE COMPILATION PROCESS EXAMPLE OF TOKENS AND ATTRIBUTES
THE COMPILATION PROCESS Character stream CS 403: Scanning and Parsing Stefan D. Bruda Fall 207 Token stream Parse tree Abstract syntax tree Modified intermediate form Target language Modified target language
More informationCS 315 Programming Languages Syntax. Parser. (Alternatively hand-built) (Alternatively hand-built)
Programming languages must be precise Remember instructions This is unlike natural languages CS 315 Programming Languages Syntax Precision is required for syntax think of this as the format of the language
More information3. Context-free grammars & parsing
3. Context-free grammars & parsing The parsing process sequences of tokens parse tree or syntax tree a / [ / index / ]/= / 4 / + / 2 The parsing process sequences of tokens parse tree or syntax tree a
More informationCS 403: Scanning and Parsing
CS 403: Scanning and Parsing Stefan D. Bruda Fall 2017 THE COMPILATION PROCESS Character stream Scanner (lexical analysis) Token stream Parser (syntax analysis) Parse tree Semantic analysis Abstract syntax
More informationCS 314 Principles of Programming Languages
CS 314 Principles of Programming Languages Lecture 5: Syntax Analysis (Parsing) Zheng (Eddy) Zhang Rutgers University January 31, 2018 Class Information Homework 1 is being graded now. The sample solution
More informationBuilding Compilers with Phoenix
Building Compilers with Phoenix Syntax-Directed Translation Structure of a Compiler Character Stream Intermediate Representation Lexical Analyzer Machine-Independent Optimizer token stream Intermediate
More informationSometimes an ambiguous grammar can be rewritten to eliminate the ambiguity.
Eliminating Ambiguity Sometimes an ambiguous grammar can be rewritten to eliminate the ambiguity. Example: consider the following grammar stat if expr then stat if expr then stat else stat other One can
More informationCPS 506 Comparative Programming Languages. Syntax Specification
CPS 506 Comparative Programming Languages Syntax Specification Compiling Process Steps Program Lexical Analysis Convert characters into a stream of tokens Lexical Analysis Syntactic Analysis Send tokens
More informationCOP 3402 Systems Software Syntax Analysis (Parser)
COP 3402 Systems Software Syntax Analysis (Parser) Syntax Analysis 1 Outline 1. Definition of Parsing 2. Context Free Grammars 3. Ambiguous/Unambiguous Grammars Syntax Analysis 2 Lexical and Syntax Analysis
More informationprogramming languages need to be precise a regular expression is one of the following: tokens are the building blocks of programs
Chapter 2 :: Programming Language Syntax Programming Language Pragmatics Michael L. Scott Introduction programming languages need to be precise natural languages less so both form (syntax) and meaning
More informationParsing. source code. while (k<=n) {sum = sum+k; k=k+1;}
Compiler Construction Grammars Parsing source code scanner tokens regular expressions lexical analysis Lennart Andersson parser context free grammar Revision 2012 01 23 2012 parse tree AST builder (implicit)
More informationSyntax Analysis. COMP 524: Programming Language Concepts Björn B. Brandenburg. The University of North Carolina at Chapel Hill
Syntax Analysis Björn B. Brandenburg The University of North Carolina at Chapel Hill Based on slides and notes by S. Olivier, A. Block, N. Fisher, F. Hernandez-Campos, and D. Stotts. The Big Picture Character
More informationCOP 3402 Systems Software Top Down Parsing (Recursive Descent)
COP 3402 Systems Software Top Down Parsing (Recursive Descent) Top Down Parsing 1 Outline 1. Top down parsing and LL(k) parsing 2. Recursive descent parsing 3. Example of recursive descent parsing of arithmetic
More informationSyntax Analysis. Prof. James L. Frankel Harvard University. Version of 6:43 PM 6-Feb-2018 Copyright 2018, 2015 James L. Frankel. All rights reserved.
Syntax Analysis Prof. James L. Frankel Harvard University Version of 6:43 PM 6-Feb-2018 Copyright 2018, 2015 James L. Frankel. All rights reserved. Context-Free Grammar (CFG) terminals non-terminals start
More informationCompilers. Yannis Smaragdakis, U. Athens (original slides by Sam
Compilers Parsing Yannis Smaragdakis, U. Athens (original slides by Sam Guyer@Tufts) Next step text chars Lexical analyzer tokens Parser IR Errors Parsing: Organize tokens into sentences Do tokens conform
More informationPart 5 Program Analysis Principles and Techniques
1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape
More informationParsing. Roadmap. > Context-free grammars > Derivations and precedence > Top-down parsing > Left-recursion > Look-ahead > Table-driven parsing
Roadmap > Context-free grammars > Derivations and precedence > Top-down parsing > Left-recursion > Look-ahead > Table-driven parsing The role of the parser > performs context-free syntax analysis > guides
More informationLecturer: William W.Y. Hsu. Programming Languages
Lecturer: William W.Y. Hsu Programming Languages Chapter 2 - Programming Language Syntax 3 Scanning The main task of scanning is to identify tokens. 4 Pseudo-Code Scanner We skip any initial white spaces
More informationSyntax Analysis Check syntax and construct abstract syntax tree
Syntax Analysis Check syntax and construct abstract syntax tree if == = ; b 0 a b Error reporting and recovery Model using context free grammars Recognize using Push down automata/table Driven Parsers
More informationLexical Analysis. Introduction
Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies
More informationCompiler Construction 2016/2017 Syntax Analysis
Compiler Construction 2016/2017 Syntax Analysis Peter Thiemann November 2, 2016 Outline 1 Syntax Analysis Recursive top-down parsing Nonrecursive top-down parsing Bottom-up parsing Syntax Analysis tokens
More informationSyntax Analysis. The Big Picture. The Big Picture. COMP 524: Programming Languages Srinivas Krishnan January 25, 2011
Syntax Analysis COMP 524: Programming Languages Srinivas Krishnan January 25, 2011 Based in part on slides and notes by Bjoern Brandenburg, S. Olivier and A. Block. 1 The Big Picture Character Stream Token
More informationChapter 3. Describing Syntax and Semantics ISBN
Chapter 3 Describing Syntax and Semantics ISBN 0-321-49362-1 Chapter 3 Topics Introduction The General Problem of Describing Syntax Formal Methods of Describing Syntax Copyright 2009 Addison-Wesley. All
More informationSyntax Analysis Part I
Syntax Analysis Part I Chapter 4: Context-Free Grammars Slides adapted from : Robert van Engelen, Florida State University Position of a Parser in the Compiler Model Source Program Lexical Analyzer Token,
More informationCMSC 330: Organization of Programming Languages
CMSC 330: Organization of Programming Languages Context Free Grammars and Parsing 1 Recall: Architecture of Compilers, Interpreters Source Parser Static Analyzer Intermediate Representation Front End Back
More informationChapter 4. Lexical and Syntax Analysis
Chapter 4 Lexical and Syntax Analysis Chapter 4 Topics Introduction Lexical Analysis The Parsing Problem Recursive-Descent Parsing Bottom-Up Parsing Copyright 2012 Addison-Wesley. All rights reserved.
More informationCIT Lecture 5 Context-Free Grammars and Parsing 4/2/2003 1
CIT3136 - Lecture 5 Context-Free Grammars and Parsing 4/2/2003 1 Definition of a Context-free Grammar: An alphabet or set of basic symbols (like regular expressions, only now the symbols are whole tokens,
More informationLecture 4: Syntax Specification
The University of North Carolina at Chapel Hill Spring 2002 Lecture 4: Syntax Specification Jan 16 1 Phases of Compilation 2 1 Syntax Analysis Syntax: Webster s definition: 1 a : the way in which linguistic
More informationConcepts Introduced in Chapter 4
Concepts Introduced in Chapter 4 Grammars Context-Free Grammars Derivations and Parse Trees Ambiguity, Precedence, and Associativity Top Down Parsing Recursive Descent, LL Bottom Up Parsing SLR, LR, LALR
More informationA programming language requires two major definitions A simple one pass compiler
A programming language requires two major definitions A simple one pass compiler [Syntax: what the language looks like A context-free grammar written in BNF (Backus-Naur Form) usually suffices. [Semantics:
More informationEECS 6083 Intro to Parsing Context Free Grammars
EECS 6083 Intro to Parsing Context Free Grammars Based on slides from text web site: Copyright 2003, Keith D. Cooper, Ken Kennedy & Linda Torczon, all rights reserved. 1 Parsing sequence of tokens parser
More informationDr. D.M. Akbar Hussain
Syntax Analysis Parsing Syntax Or Structure Given By Determines Grammar Rules Context Free Grammar 1 Context Free Grammars (CFG) Provides the syntactic structure: A grammar is quadruple (V T, V N, S, R)
More informationTheoretical Part. Chapter one:- - What are the Phases of compiler? Answer:
Theoretical Part Chapter one:- - What are the Phases of compiler? Six phases Scanner Parser Semantic Analyzer Source code optimizer Code generator Target Code Optimizer Three auxiliary components Literal
More informationSyntax Analysis. Amitabha Sanyal. (www.cse.iitb.ac.in/ as) Department of Computer Science and Engineering, Indian Institute of Technology, Bombay
Syntax Analysis (www.cse.iitb.ac.in/ as) Department of Computer Science and Engineering, Indian Institute of Technology, Bombay September 2007 College of Engineering, Pune Syntax Analysis: 2/124 Syntax
More informationSyntax. In Text: Chapter 3
Syntax In Text: Chapter 3 1 Outline Syntax: Recognizer vs. generator BNF EBNF Chapter 3: Syntax and Semantics 2 Basic Definitions Syntax the form or structure of the expressions, statements, and program
More informationPART 3 - SYNTAX ANALYSIS. F. Wotawa TU Graz) Compiler Construction Summer term / 309
PART 3 - SYNTAX ANALYSIS F. Wotawa (IST @ TU Graz) Compiler Construction Summer term 2016 64 / 309 Goals Definition of the syntax of a programming language using context free grammars Methods for parsing
More informationCSE302: Compiler Design
CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 20, 2007 Outline Recap
More informationFormal Languages and Grammars. Chapter 2: Sections 2.1 and 2.2
Formal Languages and Grammars Chapter 2: Sections 2.1 and 2.2 Formal Languages Basis for the design and implementation of programming languages Alphabet: finite set Σ of symbols String: finite sequence
More informationLexical and Syntax Analysis (2)
Lexical and Syntax Analysis (2) In Text: Chapter 4 N. Meng, F. Poursardar Motivating Example Consider the grammar S -> cad A -> ab a Input string: w = cad How to build a parse tree top-down? 2 Recursive-Descent
More informationLanguages and Compilers
Principles of Software Engineering and Operational Systems Languages and Compilers SDAGE: Level I 2012-13 3. Formal Languages, Grammars and Automata Dr Valery Adzhiev vadzhiev@bournemouth.ac.uk Office:
More information3. Parsing. Oscar Nierstrasz
3. Parsing Oscar Nierstrasz Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes. http://www.cs.ucla.edu/~palsberg/ http://www.cs.purdue.edu/homes/hosking/
More informationArchitecture of Compilers, Interpreters. CMSC 330: Organization of Programming Languages. Front End Scanner and Parser. Implementing the Front End
Architecture of Compilers, Interpreters : Organization of Programming Languages ource Analyzer Optimizer Code Generator Context Free Grammars Intermediate Representation Front End Back End Compiler / Interpreter
More informationIntroduction to Parsing
Introduction to Parsing The Front End Source code Scanner tokens Parser IR Errors Parser Checks the stream of words and their parts of speech (produced by the scanner) for grammatical correctness Determines
More informationPrinciples of Programming Languages COMP251: Syntax and Grammars
Principles of Programming Languages COMP251: Syntax and Grammars Prof. Dekai Wu Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong, China Fall 2006
More informationCSE 130 Programming Language Principles & Paradigms Lecture # 5. Chapter 4 Lexical and Syntax Analysis
Chapter 4 Lexical and Syntax Analysis Introduction - Language implementation systems must analyze source code, regardless of the specific implementation approach - Nearly all syntax analysis is based on
More informationCMSC 330: Organization of Programming Languages
CMSC 330: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Analyzer Optimizer Code Generator Abstract Syntax Tree Front End Back End Compiler
More informationCMSC 330: Organization of Programming Languages. Context Free Grammars
CMSC 330: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Analyzer Optimizer Code Generator Abstract Syntax Tree Front End Back End Compiler
More informationCMSC 330: Organization of Programming Languages
CMSC 330: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Analyzer Optimizer Code Generator Abstract Syntax Tree Front End Back End Compiler
More informationMIT Specifying Languages with Regular Expressions and Context-Free Grammars
MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology Language Definition Problem How to precisely
More informationWhere We Are. CMSC 330: Organization of Programming Languages. This Lecture. Programming Languages. Motivation for Grammars
CMSC 330: Organization of Programming Languages Context Free Grammars Where We Are Programming languages Ruby OCaml Implementing programming languages Scanner Uses regular expressions Finite automata Parser
More information3. Syntax Analysis. Andrea Polini. Formal Languages and Compilers Master in Computer Science University of Camerino
3. Syntax Analysis Andrea Polini Formal Languages and Compilers Master in Computer Science University of Camerino (Formal Languages and Compilers) 3. Syntax Analysis CS@UNICAM 1 / 54 Syntax Analysis: the
More information4. Lexical and Syntax Analysis
4. Lexical and Syntax Analysis 4.1 Introduction Language implementation systems must analyze source code, regardless of the specific implementation approach Nearly all syntax analysis is based on a formal
More informationEDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing. Görel Hedin Revised:
EDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing Görel Hedin Revised: 2017-09-04 This lecture Regular expressions Context-free grammar Attribute grammar
More informationCMSC 330: Organization of Programming Languages. Architecture of Compilers, Interpreters
: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Scanner Parser Static Analyzer Intermediate Representation Front End Back End Compiler / Interpreter
More informationCMSC 330: Organization of Programming Languages. Context Free Grammars
CMSC 330: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Analyzer Optimizer Code Generator Abstract Syntax Tree Front End Back End Compiler
More information4. Lexical and Syntax Analysis
4. Lexical and Syntax Analysis 4.1 Introduction Language implementation systems must analyze source code, regardless of the specific implementation approach Nearly all syntax analysis is based on a formal
More information2. λ is a regular expression and denotes the set {λ} 4. If r and s are regular expressions denoting the languages R and S, respectively
Regular expressions: a regular expression is built up out of simpler regular expressions using a set of defining rules. Regular expressions allows us to define tokens of programming languages such as identifiers.
More informationChapter 2 :: Programming Language Syntax
Chapter 2 :: Programming Language Syntax Michael L. Scott kkman@sangji.ac.kr, 2015 1 Regular Expressions A regular expression is one of the following: A character The empty string, denoted by Two regular
More informationCS 406/534 Compiler Construction Parsing Part I
CS 406/534 Compiler Construction Parsing Part I Prof. Li Xu Dept. of Computer Science UMass Lowell Fall 2004 Part of the course lecture notes are based on Prof. Keith Cooper, Prof. Ken Kennedy and Dr.
More informationSyntax and Parsing COMS W4115. Prof. Stephen A. Edwards Fall 2003 Columbia University Department of Computer Science
Syntax and Parsing COMS W4115 Prof. Stephen A. Edwards Fall 2003 Columbia University Department of Computer Science Lexical Analysis (Scanning) Lexical Analysis (Scanning) Goal is to translate a stream
More informationThe Parsing Problem (cont d) Recursive-Descent Parsing. Recursive-Descent Parsing (cont d) ICOM 4036 Programming Languages. The Complexity of Parsing
ICOM 4036 Programming Languages Lexical and Syntax Analysis Lexical Analysis The Parsing Problem Recursive-Descent Parsing Bottom-Up Parsing This lecture covers review questions 14-27 This lecture covers
More informationChapter 3: CONTEXT-FREE GRAMMARS AND PARSING Part2 3.3 Parse Trees and Abstract Syntax Trees
Chapter 3: CONTEXT-FREE GRAMMARS AND PARSING Part2 3.3 Parse Trees and Abstract Syntax Trees 3.3.1 Parse trees 1. Derivation V.S. Structure Derivations do not uniquely represent the structure of the strings
More informationSyntax. A. Bellaachia Page: 1
Syntax 1. Objectives & Definitions... 2 2. Definitions... 3 3. Lexical Rules... 4 4. BNF: Formal Syntactic rules... 6 5. Syntax Diagrams... 9 6. EBNF: Extended BNF... 10 7. Example:... 11 8. BNF Statement
More informationRevisit the example. Transformed DFA 10/1/16 A B C D E. Start
Revisit the example ε 0 ε 1 Start ε a ε 2 3 ε b ε 4 5 ε a b b 6 7 8 9 10 ε-closure(0)={0, 1, 2, 4, 7} = A Trans(A, a) = {1, 2, 3, 4, 6, 7, 8} = B Trans(A, b) = {1, 2, 4, 5, 6, 7} = C Trans(B, a) = {1,
More informationMIT Specifying Languages with Regular Expressions and Context-Free Grammars. Martin Rinard Massachusetts Institute of Technology
MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Massachusetts Institute of Technology Language Definition Problem How to precisely define language Layered structure
More informationCMSC 330: Organization of Programming Languages. Context Free Grammars
CMSC 330: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Analyzer Optimizer Code Generator Abstract Syntax Tree Front End Back End Compiler
More informationSYNTAX ANALYSIS 1. Define parser. Hierarchical analysis is one in which the tokens are grouped hierarchically into nested collections with collective meaning. Also termed as Parsing. 2. Mention the basic
More informationLR Parsing, Part 2. Constructing Parse Tables. An NFA Recognizing Viable Prefixes. Computing the Closure. GOTO Function and DFA States
TDDD16 Compilers and Interpreters TDDB44 Compiler Construction LR Parsing, Part 2 Constructing Parse Tables Parse table construction Grammar conflict handling Categories of LR Grammars and Parsers An NFA
More informationHabanero Extreme Scale Software Research Project
Habanero Extreme Scale Software Research Project Comp215: Grammars Zoran Budimlić (Rice University) Grammar, which knows how to control even kings - Moliere So you know everything about regular expressions
More informationBottom-up parsing. Bottom-Up Parsing. Recall. Goal: For a grammar G, withstartsymbols, any string α such that S α is called a sentential form
Bottom-up parsing Bottom-up parsing Recall Goal: For a grammar G, withstartsymbols, any string α such that S α is called a sentential form If α V t,thenα is called a sentence in L(G) Otherwise it is just
More informationCOMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou
COMP-421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3 - Lexical Analysis Sections 3.1-3.4, 3.6-3.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna
More informationCMPS Programming Languages. Dr. Chengwei Lei CEECS California State University, Bakersfield
CMPS 3500 Programming Languages Dr. Chengwei Lei CEECS California State University, Bakersfield Chapter 3 Describing Syntax and Semantics Chapter 3 Topics Introduction The General Problem of Describing
More informationRegular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications
Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata
More informationProgramming Language Specification and Translation. ICOM 4036 Fall Lecture 3
Programming Language Specification and Translation ICOM 4036 Fall 2009 Lecture 3 Some parts are Copyright 2004 Pearson Addison-Wesley. All rights reserved. 3-1 Language Specification and Translation Topics
More informationLECTURE 7. Lex and Intro to Parsing
LECTURE 7 Lex and Intro to Parsing LEX Last lecture, we learned a little bit about how we can take our regular expressions (which specify our valid tokens) and create real programs that can recognize them.
More informationSection A. A grammar that produces more than one parse tree for some sentences is said to be ambiguous.
Section A 1. What do you meant by parser and its types? A parser for grammar G is a program that takes as input a string w and produces as output either a parse tree for w, if w is a sentence of G, or
More informationCSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B-1
CSEP 501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2008 1/8/2008 2002-08 Hal Perkins & UW CSE B-1 Agenda Basic concepts of formal grammars (review) Regular expressions
More informationLexical and Syntax Analysis. Top-Down Parsing
Lexical and Syntax Analysis Top-Down Parsing Easy for humans to write and understand String of characters Lexemes identified String of tokens Easy for programs to transform Data structure Syntax A syntax
More informationDefining syntax using CFGs
Defining syntax using CFGs Roadmap Last time Defined context-free grammar This time CFGs for specifying a language s syntax Language membership List grammars Resolving ambiguity CFG Review G = (N,Σ,P,S)
More informationContext-free grammars
Context-free grammars Section 4.2 Formal way of specifying rules about the structure/syntax of a program terminals - tokens non-terminals - represent higher-level structures of a program start symbol,
More informationICOM 4036 Spring 2004
Language Specification and Translation ICOM 4036 Spring 2004 Lecture 3 Copyright 2004 Pearson Addison-Wesley. All rights reserved. 3-1 Language Specification and Translation Topics Structure of a Compiler
More informationLexical Analysis (ASU Ch 3, Fig 3.1)
Lexical Analysis (ASU Ch 3, Fig 3.1) Implementation by hand automatically ((F)Lex) Lex generates a finite automaton recogniser uses regular expressions Tasks remove white space (ws) display source program
More informationPrinciples of Programming Languages
Principles of Programming Languages h"p://www.di.unipi.it/~andrea/dida2ca/plp- 14/ Prof. Andrea Corradini Department of Computer Science, Pisa Lesson 8! Bo;om- Up Parsing Shi?- Reduce LR(0) automata and
More informationS Y N T A X A N A L Y S I S LR
LR parsing There are three commonly used algorithms to build tables for an LR parser: 1. SLR(1) = LR(0) plus use of FOLLOW set to select between actions smallest class of grammars smallest tables (number
More informationParsing. Handle, viable prefix, items, closures, goto s LR(k): SLR(1), LR(1), LALR(1)
TD parsing - LL(1) Parsing First and Follow sets Parse table construction BU Parsing Handle, viable prefix, items, closures, goto s LR(k): SLR(1), LR(1), LALR(1) Problems with SLR Aho, Sethi, Ullman, Compilers
More informationLexical and Syntax Analysis
Lexical and Syntax Analysis In Text: Chapter 4 N. Meng, F. Poursardar Lexical and Syntactic Analysis Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input
More informationWednesday, September 9, 15. Parsers
Parsers What is a parser A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure of a program (think: diagramming a sentence) Agenda
More informationParsers. What is a parser. Languages. Agenda. Terminology. Languages. A parser has two jobs:
What is a parser Parsers A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure of a program (think: diagramming a sentence) Agenda
More informationParsing Techniques. CS152. Chris Pollett. Sep. 24, 2008.
Parsing Techniques. CS152. Chris Pollett. Sep. 24, 2008. Outline. Top-down versus Bottom-up Parsing. Recursive Descent Parsing. Left Recursion Removal. Left Factoring. Predictive Parsing. Introduction.
More informationCompiler Construction: Parsing
Compiler Construction: Parsing Mandar Mitra Indian Statistical Institute M. Mitra (ISI) Parsing 1 / 33 Context-free grammars. Reference: Section 4.2 Formal way of specifying rules about the structure/syntax
More informationIntroduction to parsers
Syntax Analysis Introduction to parsers Context-free grammars Push-down automata Top-down parsing LL grammars and parsers Bottom-up parsing LR grammars and parsers Bison/Yacc - parser generators Error
More informationParsing Algorithms. Parsing: continued. Top Down Parsing. Predictive Parser. David Notkin Autumn 2008
Parsing: continued David Notkin Autumn 2008 Parsing Algorithms Earley s algorithm (1970) works for all CFGs O(N 3 ) worst case performance O(N 2 ) for unambiguous grammars Based on dynamic programming,
More informationA Simple Syntax-Directed Translator
Chapter 2 A Simple Syntax-Directed Translator 1-1 Introduction The analysis phase of a compiler breaks up a source program into constituent pieces and produces an internal representation for it, called
More informationLR Parsing Techniques
LR Parsing Techniques Introduction Bottom-Up Parsing LR Parsing as Handle Pruning Shift-Reduce Parser LR(k) Parsing Model Parsing Table Construction: SLR, LR, LALR 1 Bottom-UP Parsing A bottom-up parser
More information