Topic 5: Syntax Analysis III
|
|
- Clemence Hardy
- 6 years ago
- Views:
Transcription
1 Topic 5: Syntax Analysis III Compiler Design Prof. Hanjun Kim CoreLab (Compiler Research Lab) POSTECH 1
2 Back-End Front-End The Front End Source Program Lexical Analysis Syntax Analysis Semantic Analysis IR Code Generation Intermediate Representation IR Optimization Target Code Generation Target Code Optimization Target Program Lexical Analysis Break into tokens Think words, punctuation Syntax Analysis Parse phase structure Think document, paragraphs, sentences Semantic Analysis Calculate meaning 2
3 Parser in the Front-End Source Stream of Abstract Lexer Tokens Parser Syntax Tree FE IR Parser Functions: Verify that token stream is valid If it is not valid, report syntax error and recover Build Abstract Syntax Tree (AST) 3
4 Parsing Power Unambiguous Grammar LL(k) LR(k) Ambiguous Grammar LL(1) LR(1) LALR(1) SLR LL(0) LR(0) 4
5 Real-world Parser Generators 5
6 Real-world Parser Generators Context-Free Grammar Parser Generator Stream of Tokens Parser Parser generators yacc, bison: LALR parser generators for C ml-yacc: a LALR parser generator for ML Parser Generator Specification Input: a set of context-free grammars specifying a parser Outputs A parser in target language A description of state machine Rules: consists of a pattern and an action: Pattern is context free grammar Action is a fragment of ordinary target code Examples: exp: exp PLUS exp (exp1 + exp2) Abstract Syntax Tree 6
7 Lexer Generator Example: Bison %{ #include <math.h> %} %token NUM %left '-' '+' %left '*' %left NEG /* negation--unary minus */ %% line: '\n' exp '\n' { printf ("\t%.10g\n", $1); } ; exp: NUM { $$ = $1; } exp '+' exp { $$ = $1 + $3; } exp '-' exp { $$ = $1 - $3; } exp '*' exp { $$ = $1 * $3; } '-' exp %prec NEG { $$ = -$2; } ; %% main () { yyparse (); } Declarations Rules User Codes 7
8 Lexer Generator Example: ML-YACC structure A = struct type id = S.symbol datatype binop = PLUS MINUS TIMES DIV datatype stm = CompoundStm of stm * stm AssignStm of id * exp datatype exp = IDExp of id NUMExp of int OpExp of exp * binop * exp end %% %term INT of int ID of string PLUS MINUS %nonterm exp of A.exp stm of A.stm prog of A.stm %% prog: LPAREN stm RPAREN (stm) User Declaration YACC Definition stm: stm SEMICOLON stm stm: ID ASSIGN exp exp: INT exp: ID exp: exp PLUS exp exp: exp MINUS exp (A.CompoundStm(stm1, stm2)) (A.AssignStm(S.symbol(ID), exp)) (A.IntExp(INT)) (A.IDExp(S.symbol(ID))) (A.OpExp(exp1, A.PLUS, exp2)) (A.OpExp(exp1, A.MINUS, exp2)) Rules 8
9 Lexer Generator Example: ML-YACC User Declaration Define various values that are available to rules section YACC Definition Declare terminal and non-terminal symbols, and their attribute %term IF THEN ELSE NUM of int %nonterm prog stmt exp Declare precedences for terminals that help resolve shift-reduce conflicts Specify the type of the current input file position (%pos int) Optionally specify end-of-parse symbol (%eop EOF) Optionally specify start symbol otherwise, LHS non-terminal of first rule is taken as start symbol %start prog Rules Specify productions of grammar and semantic actions associated with productions symbol 0 symbol 1 symbol 2 symbol n (semantic action) 9
10 Lexer Generator Example: ML-YACC Positions In order to report semantic error, need to annotate each AST node with source file position of characters X < n >: returns attribute of nth occurrence of X X < n > left: returns left-end position of token corresponding to X X < n > right: returns right-end position of token corresponding to X Example: stm: stm SEMICOLON stm (A.PosStm(stm1left, A.CompoundStm(stm1, stm2))) 10
11 AST Example structure A = struct type id = S.symbol datatype binop = PLUS MINUS TIMES DIV datatype stm = CompoundStm of stm * stm AssignStm of id * exp PosStm of int * stm datatype exp = IDExp of id NUMExp of int OpExp of exp * binop * exp PosExp of int * exp end %% %term INT of int ID of string PLUS MINUS %nonterm exp of A.exp stm of A.stm prog of A.stm %% prog: LPARAEN stm RPAREN (stm) stm: stm SEMICOLON stm stm: ID ASSIGN exp exp: INT exp: ID (A.PosStm(stm1left, A.CompoundStm(stm1, stm2))) (A.PosStm(IDleft, A.AssignStm(S.symbol(ID),exp))) (A.PosExp(INTleft, A.IntExp(INT))) (A.PosExp(IDleft, A.IDExp(S.symbol(ID)))) 11
12 AST Example Input Program: (a := 5 ; b := a + 1) Abstract syntax: PosStm[ int = 1, stm = CompoundStm[ stm = PosStm[ int = 2, stm = AssignStm[ ID = PosExp[int = 2, exp = IDExp(S.symbol( a ))], exp = PosExp[int = 7, exp = NUMExp(5)] ] ], stm = PosStm[ int = 11, stm = AssignStm[ ID = PosExp[int = 11, exp = IDExp(S.symbol( b ))], exp = PosExp[ int = 16, exp = OpExp[ exp = PosExp[int = 16, exp = IDExp(S.symbol( b ))], binop = PLUS, exp = PosExp[int = 20, exp = NUMExp(1)] ] ] ] ] ] ] 12
13 YACC & Ambiguous Grammars A grammar is ambiguous if it can derive a string of tokens with two or more different parse tree Consider 4+5*6 * + + NUM(6) NUM(4) * NUM(4) NUM(5) NUM(5) NUM(6) We prefer to bind * tighter than + 13
14 YACC & Ambiguous Grammars Similarly, consider: We prefer to bind left + first NUM(6) NUM(4) + NUM(4) NUM(5) NUM(5) NUM(6) 14
15 YACC & Ambiguous Grammars YACC will report shift-reduce conflicts 4+5*6 When + is on top of stack, parser gets * as the current token Parser can reduce by rule + or shift Prefer shift When + is on top of stack, parser gets + as the current token Parser can reduce by rule + or shift Prefer reduce 15
16 Directives Three Solutions Let YACC complain, but check if the choice (shift) is correct Rewrite grammar to eliminate ambiguity Keep grammar, but add precedence directives which enable conflicts to be resolved Use %left, %right, %nonassoc For this grammar %left PLUS MINUS %left MULT DIV PLUS, MINUS are left associative, bind equally tightly MULT, DIV are left associative, bind equally tightly MULT, DIV bind tighter than PLUS, MINUS 16
17 Directives Given directives, YACC assigns precedence to each terminal and rule Precedence of terminal based on order in which associativity specified Precedence of rule is the precedence of right-most terminal Ex: precedence( + )=precedence(plus) Given shift-reduce conflict, YACC performs the following: Find precedence of rule to be reduced, terminal to be shifted prec(terminal) > prec(rule) : shift prec(rule) > prec(terminal) : reduce prec(terminal) = prec(rule) assoc(terminal) = left : reduce assoc(terminal) = right: shift assoc(terminal) = nonassoc: report error 17
18 Precedence Example Input : * 6 Stack : Action: prec(*) > prec(+) -> shift Input : 4 * Stack : 4 * 5 Action: prec(*) > prec(+) -> reduce Input : Stack : Action: assoc(+) = left -> reduce 18
19 Default Behavior What if directives not specified? shift-reduce: report error, shift by default reduce-reduce: report error, reduce by rule that occurs first What to do: shift-reduce: acceptable in well defined cases (dangling else) reduce-reduce: unacceptable, Rewrite grammar 19
20 %prec directive Commonly used for the unary minus problem %left PLUS MINUS %left MULT DIV Consider -4*6 We prefer to bind left unary minus (-) tighter, but precedence of MINUS is lower than MULT -(4*6) not (-4)*6 Solution: %term NUM PLUS MINUS MULT DIV UMINUS %left PLUS MINUS %left MULT DIV %left UMINUS : MINUS %prec UMINUS () PLUS () 20
21 A parser can support semantic action. Why does a compiler separate semantic action from parsing? 21
22 Precedence Parsing with semantic action E -> E + E E -> E E E -> E * E E -> NUM E -> -NUM %% %term INT of int PLUS MINUS TIMES UMINUS EOF %nonterm exp of int %start exp %eop EOF Left Associativity %left PLUS MINUS %left TIMES %left UMINUS %% exp: INT exp: exp PLUS exp exp: exp MINUS exp exp: exp TIMES exp exp: MINUS exp %prec UMINUS (INT) (exp1 + exp2) (exp1 exp2) (exp1 * exp2) (~exp) 22
23 Parsing with semantic action E -> E + E E -> E E E -> E * E E -> NUM E -> -NUM Input Program: * 3 Stack Input Action * 3 $ shift NUM(1) + 2 * 3 $ reduce E(1) + 2 * 3 $ shift E(1) PLUS 2 * 3 $ shift E(1) PLUS NUM(2) * 3 $ reduce E(1) PLUS E(2) * 3 $ shift E(1) PLUS E(2) TIMES 3 $ shift E(1) PLUS E(2) TIMES NUM(3) $ reduce E(1) PLUS E(2) TIMES E(3) $ reduce E(1) PLUS E(6) $ reduce E(7) $ accept 23
24 Parsing with semantic action Parser with semantic action Disadvantages File becomes too large; difficult to manage Program must be processed in order in which it is parsed; Impossible to do global/inter-procedural optimization Alternative: Separate parsing from remaining compiler phases 24
25 Context-Free Grammars are more powerful than Regular Expressions 25
26 Context-Free Grammar & REs CFGs are More powerful than REs Any language that can be generated using regular essions can be generated by a context-free grammar There are languages that can be generated by a contextfree grammar that cannot be generated by any regular ession Example: Matching parentheses Nested comments 26
27 Proof Given a RE R, we can generate a CFG G such that L(R) == L(G) We can define a grammar G for which there is no FA F such that L(F) == L(G) 27
28 Proof 1 Base Cases: Symbol(a): RE a Epsilon(ε): RE ε Inductive Cases: Alternation (M N): RE M RE N Concatenation (M N): RE M N Repetition (M*): RE M RE RE ε 28
29 Proof 2 S S S ε FAs have a FINITE number of states, N FA must remember the number of (, to generate ) s At or before N+1 ( s, FA will revisit a state that represents two different counts of ) s Both count must now be accepted One count will be invalid Representations Regular, finite-state grammars: FAs Context-free grammars: Push-Down Automata 29
30 Application of a Lexer and Parser 30
31 Applications Compiler & Interpreter Pattern matching Searching an exact word (ex. compiler ) Find and replace with a rule (ex. [a-z][a-z0-9]*) Rendering Rendering a web page of HTML + Content Rendering an image Printing a document Natural language processing Translation Understanding Korean particles Data Analysis Analyze xml files Big data analysis 31
Topic 3: Syntax Analysis I
Topic 3: Syntax Analysis I Compiler Design Prof. Hanjun Kim CoreLab (Compiler Research Lab) POSTECH 1 Back-End Front-End The Front End Source Program Lexical Analysis Syntax Analysis Semantic Analysis
More informationCompilation 2013 Parser Generators, Conflict Management, and ML-Yacc
Compilation 2013 Parser Generators, Conflict Management, and ML-Yacc Erik Ernst Aarhus University Parser generators, ML-Yacc LR parsers are tedious to write, but can be generated, e.g., by ML-Yacc Input:
More informationEDAN65: Compilers, Lecture 06 A LR parsing. Görel Hedin Revised:
EDAN65: Compilers, Lecture 06 A LR parsing Görel Hedin Revised: 2017-09-11 This lecture Regular expressions Context-free grammar Attribute grammar Lexical analyzer (scanner) Syntactic analyzer (parser)
More informationCompilation 2014 Warm-up project
Compilation 2014 Warm-up project Aslan Askarov aslan@cs.au.dk Revised from slides by E. Ernst Straight-line Programming Language Toy programming language: no branching, no loops Skip lexing and parsing
More informationAbstract Syntax. Mooly Sagiv. html://www.cs.tau.ac.il/~msagiv/courses/wcc06.html
Abstract Syntax Mooly Sagiv html://www.cs.tau.ac.il/~msagiv/courses/wcc06.html Outline The general idea Cup Motivating example Interpreter for arithmetic expressions The need for abstract syntax Abstract
More informationCOMPILER (CSE 4120) (Lecture 6: Parsing 4 Bottom-up Parsing )
COMPILR (CS 4120) (Lecture 6: Parsing 4 Bottom-up Parsing ) Sungwon Jung Mobile Computing & Data ngineering Lab Dept. of Computer Science and ngineering Sogang University Seoul, Korea Tel: +82-2-705-8930
More informationPart III : Parsing. From Regular to Context-Free Grammars. Deriving a Parser from a Context-Free Grammar. Scanners and Parsers.
Part III : Parsing From Regular to Context-Free Grammars Deriving a Parser from a Context-Free Grammar Scanners and Parsers A Parser for EBNF Left-Parsable Grammars Martin Odersky, LAMP/DI 1 From Regular
More informationIntroduction to Parsing Ambiguity and Syntax Errors
Introduction to Parsing Ambiguity and Syntax rrors Outline Regular languages revisited Parser overview Context-free grammars (CFG s) Derivations Ambiguity Syntax errors Compiler Design 1 (2011) 2 Languages
More informationIntroduction to Parsing Ambiguity and Syntax Errors
Introduction to Parsing Ambiguity and Syntax rrors Outline Regular languages revisited Parser overview Context-free grammars (CFG s) Derivations Ambiguity Syntax errors 2 Languages and Automata Formal
More informationLecture 14: Parser Conflicts, Using Ambiguity, Error Recovery. Last modified: Mon Feb 23 10:05: CS164: Lecture #14 1
Lecture 14: Parser Conflicts, Using Ambiguity, Error Recovery Last modified: Mon Feb 23 10:05:56 2015 CS164: Lecture #14 1 Shift/Reduce Conflicts If a DFA state contains both [X: α aβ, b] and [Y: γ, a],
More informationIntroduction to Parsing. Lecture 5
Introduction to Parsing Lecture 5 1 Outline Regular languages revisited Parser overview Context-free grammars (CFG s) Derivations Ambiguity 2 Languages and Automata Formal languages are very important
More informationUsing an LALR(1) Parser Generator
Using an LALR(1) Parser Generator Yacc is an LALR(1) parser generator Developed by S.C. Johnson and others at AT&T Bell Labs Yacc is an acronym for Yet another compiler compiler Yacc generates an integrated
More informationIntroduction to Parsing. Lecture 5
Introduction to Parsing Lecture 5 1 Outline Regular languages revisited Parser overview Context-free grammars (CFG s) Derivations Ambiguity 2 Languages and Automata Formal languages are very important
More informationConflicts in LR Parsing and More LR Parsing Types
Conflicts in LR Parsing and More LR Parsing Types Lecture 10 Dr. Sean Peisert ECS 142 Spring 2009 1 Status Project 2 Due Friday, Apr. 24, 11:55pm The usual lecture time is being replaced by a discussion
More informationA Simple Syntax-Directed Translator
Chapter 2 A Simple Syntax-Directed Translator 1-1 Introduction The analysis phase of a compiler breaks up a source program into constituent pieces and produces an internal representation for it, called
More informationFall Compiler Principles Lecture 4: Parsing part 3. Roman Manevich Ben-Gurion University of the Negev
Fall 2016-2017 Compiler Principles Lecture 4: Parsing part 3 Roman Manevich Ben-Gurion University of the Negev Tentative syllabus Front End Intermediate Representation Optimizations Code Generation Scanning
More informationCS 11 Ocaml track: lecture 6
CS 11 Ocaml track: lecture 6 n Today: n Writing a computer language n Parser generators n lexers (ocamllex) n parsers (ocamlyacc) n Abstract syntax trees Problem (1) n We want to implement a computer language
More informationLR Parsing LALR Parser Generators
LR Parsing LALR Parser Generators Outline Review of bottom-up parsing Computing the parsing DFA Using parser generators 2 Bottom-up Parsing (Review) A bottom-up parser rewrites the input string to the
More informationCOP 3402 Systems Software Syntax Analysis (Parser)
COP 3402 Systems Software Syntax Analysis (Parser) Syntax Analysis 1 Outline 1. Definition of Parsing 2. Context Free Grammars 3. Ambiguous/Unambiguous Grammars Syntax Analysis 2 Lexical and Syntax Analysis
More informationParser Tools: lex and yacc-style Parsing
Parser Tools: lex and yacc-style Parsing Version 6.11.0.6 Scott Owens January 6, 2018 This documentation assumes familiarity with lex and yacc style lexer and parser generators. 1 Contents 1 Lexers 3 1.1
More informationTDDD55 - Compilers and Interpreters Lesson 3
TDDD55 - Compilers and Interpreters Lesson 3 November 22 2011 Kristian Stavåker (kristian.stavaker@liu.se) Department of Computer and Information Science Linköping University LESSON SCHEDULE November 1,
More informationPART ONE Fundamentals of Compilation
PART ONE Fundamentals of Compilation 1 Introduction A compiler was originally a program that compiled subroutines [a link-loader]. When in 1954the combination algebraic compiler came into use, or rather
More informationFundamentals of Compilation
PART ONE Fundamentals of Compilation 1 Introduction A compiler was originally a program that compiled subroutines [a link-loader]. When in 1954 the combination algebraic compiler came into use, or rather
More informationLR Parsing LALR Parser Generators
Outline LR Parsing LALR Parser Generators Review of bottom-up parsing Computing the parsing DFA Using parser generators 2 Bottom-up Parsing (Review) A bottom-up parser rewrites the input string to the
More informationFall Compiler Principles Lecture 5: Parsing part 4. Roman Manevich Ben-Gurion University
Fall 2014-2015 Compiler Principles Lecture 5: Parsing part 4 Roman Manevich Ben-Gurion University Tentative syllabus Front End Intermediate Representation Optimizations Code Generation Scanning Lowering
More informationContext-Free Grammars
Context-Free Grammars Lecture 7 http://webwitch.dreamhost.com/grammar.girl/ Outline Scanner vs. parser Why regular expressions are not enough Grammars (context-free grammars) grammar rules derivations
More informationCSE450 Translation of Programming Languages. Lecture 4: Syntax Analysis
CSE450 Translation of Programming Languages Lecture 4: Syntax Analysis http://xkcd.com/859 Structure of a Today! Compiler Source Language Lexical Analyzer Syntax Analyzer Semantic Analyzer Int. Code Generator
More informationLecture 8: Deterministic Bottom-Up Parsing
Lecture 8: Deterministic Bottom-Up Parsing (From slides by G. Necula & R. Bodik) Last modified: Fri Feb 12 13:02:57 2010 CS164: Lecture #8 1 Avoiding nondeterministic choice: LR We ve been looking at general
More informationCS 132 Compiler Construction
CS 132 Compiler Construction 1. Introduction 2 2. Lexical analysis 31 3. LL parsing 58 4. LR parsing 110 5. JavaCC and JTB 127 6. Semantic analysis 150 7. Translation and simplification 165 8. Liveness
More informationCS 2210 Sample Midterm. 1. Determine if each of the following claims is true (T) or false (F).
CS 2210 Sample Midterm 1. Determine if each of the following claims is true (T) or false (F). F A language consists of a set of strings, its grammar structure, and a set of operations. (Note: a language
More informationSyntax Analysis. Tokens ---> Parse Tree. Main Problems. Grammars. Convert the list of tokens into a parse tree ( hierarchical analysis)
Syntax Analysis Convert the list of tokens into a parse tree ( hierarchical analysis) source program lexical analyzer token get next token parser parse tree The syntactic structure is specified using context-free
More informationDefining syntax using CFGs
Defining syntax using CFGs Roadmap Last time Defined context-free grammar This time CFGs for specifying a language s syntax Language membership List grammars Resolving ambiguity CFG Review G = (N,Σ,P,S)
More informationEDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing. Görel Hedin Revised:
EDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing Görel Hedin Revised: 2017-09-04 This lecture Regular expressions Context-free grammar Attribute grammar
More informationParsing. COMP 520: Compiler Design (4 credits) Professor Laurie Hendren.
COMP 520 Winter 2015 Parsing COMP 520: Compiler Design (4 credits) Professor Laurie Hendren hendren@cs.mcgill.ca Parsing (1) COMP 520 Winter 2015 Parsing (2) A parser transforms a string of tokens into
More informationLecture 7: Deterministic Bottom-Up Parsing
Lecture 7: Deterministic Bottom-Up Parsing (From slides by G. Necula & R. Bodik) Last modified: Tue Sep 20 12:50:42 2011 CS164: Lecture #7 1 Avoiding nondeterministic choice: LR We ve been looking at general
More informationCompilers and Language Processing Tools
Compilers and Language Processing Tools Summer Term 2011 Prof. Dr. Arnd Poetzsch-Heffter Software Technology Group TU Kaiserslautern c Prof. Dr. Arnd Poetzsch-Heffter 1 Parser Generators c Prof. Dr. Arnd
More informationSyntax Analysis Check syntax and construct abstract syntax tree
Syntax Analysis Check syntax and construct abstract syntax tree if == = ; b 0 a b Error reporting and recovery Model using context free grammars Recognize using Push down automata/table Driven Parsers
More informationYacc: A Syntactic Analysers Generator
Yacc: A Syntactic Analysers Generator Compiler-Construction Tools The compiler writer uses specialised tools (in addition to those normally used for software development) that produce components that can
More informationOutline. Regular languages revisited. Introduction to Parsing. Parser overview. Context-free grammars (CFG s) Lecture 5. Derivations.
Outline Regular languages revisited Introduction to Parsing Lecture 5 Parser overview Context-free grammars (CFG s) Derivations Prof. Aiken CS 143 Lecture 5 1 Ambiguity Prof. Aiken CS 143 Lecture 5 2 Languages
More informationCSE 401 Midterm Exam Sample Solution 11/4/11
Question 1. (12 points, 2 each) The front end of a compiler consists of three parts: scanner, parser, and (static) semantics. Collectively these need to analyze the input program and decide if it is correctly
More informationBottom-Up Parsing. Lecture 11-12
Bottom-Up Parsing Lecture 11-12 (From slides by G. Necula & R. Bodik) 2/20/08 Prof. Hilfinger CS164 Lecture 11 1 Administrivia Test I during class on 10 March. 2/20/08 Prof. Hilfinger CS164 Lecture 11
More informationSYED AMMAL ENGINEERING COLLEGE (An ISO 9001:2008 Certified Institution) Dr. E.M. Abdullah Campus, Ramanathapuram
CS6660 COMPILER DESIGN Question Bank UNIT I-INTRODUCTION TO COMPILERS 1. Define compiler. 2. Differentiate compiler and interpreter. 3. What is a language processing system? 4. List four software tools
More informationCompilers. Bottom-up Parsing. (original slides by Sam
Compilers Bottom-up Parsing Yannis Smaragdakis U Athens Yannis Smaragdakis, U. Athens (original slides by Sam Guyer@Tufts) Bottom-Up Parsing More general than top-down parsing And just as efficient Builds
More informationParser Tools: lex and yacc-style Parsing
Parser Tools: lex and yacc-style Parsing Version 5.0 Scott Owens June 6, 2010 This documentation assumes familiarity with lex and yacc style lexer and parser generators. 1 Contents 1 Lexers 3 1.1 Creating
More informationCompiler Design Concepts. Syntax Analysis
Compiler Design Concepts Syntax Analysis Introduction First task is to break up the text into meaningful words called tokens. newval=oldval+12 id = id + num Token Stream Lexical Analysis Source Code (High
More informationG53CMP: Lecture 4. Syntactic Analysis: Parser Generators. Henrik Nilsson. University of Nottingham, UK. G53CMP: Lecture 4 p.1/32
G53CMP: Lecture 4 Syntactic Analysis: Parser Generators Henrik Nilsson University of Nottingham, UK G53CMP: Lecture 4 p.1/32 This Lecture Parser generators ( compiler compilers ) The parser generator Happy
More informationCS453 : JavaCUP and error recovery. CS453 Shift-reduce Parsing 1
CS453 : JavaCUP and error recovery CS453 Shift-reduce Parsing 1 Shift-reduce parsing in an LR parser LR(k) parser Left-to-right parse Right-most derivation K-token look ahead LR parsing algorithm using
More informationParser Generators. Aurochs and ANTLR and Yaccs, Oh My. Tuesday, December 1, 2009 Reading: Appel 3.3. Department of Computer Science Wellesley College
Parser Generators Aurochs and ANTLR and Yaccs, Oh My Tuesday, December 1, 2009 Reading: Appel 3.3 CS235 Languages and Automata Department of Computer Science Wellesley College Compiler Structure Source
More informationLast Time. What do we want? When do we want it? An AST. Now!
Java CUP 1 Last Time What do we want? An AST When do we want it? Now! 2 This Time A little review of ASTs The philosophy and use of a Parser Generator 3 Translating Lists CFG IdList -> id IdList comma
More information( ) i 0. Outline. Regular languages revisited. Introduction to Parsing. Parser overview. Context-free grammars (CFG s) Lecture 5.
Outline Regular languages revisited Introduction to Parsing Lecture 5 Parser overview Context-free grammars (CFG s) Derivations Prof. Aiken CS 143 Lecture 5 1 Ambiguity Prof. Aiken CS 143 Lecture 5 2 Languages
More informationIn One Slide. Outline. LR Parsing. Table Construction
LR Parsing Table Construction #1 In One Slide An LR(1) parsing table can be constructed automatically from a CFG. An LR(1) item is a pair made up of a production and a lookahead token; it represents a
More informationA simple syntax-directed
Syntax-directed is a grammaroriented compiling technique Programming languages: Syntax: what its programs look like? Semantic: what its programs mean? 1 A simple syntax-directed Lexical Syntax Character
More informationQUESTIONS RELATED TO UNIT I, II And III
QUESTIONS RELATED TO UNIT I, II And III UNIT I 1. Define the role of input buffer in lexical analysis 2. Write regular expression to generate identifiers give examples. 3. Define the elements of production.
More informationIntroduction to Parsing. Lecture 5. Professor Alex Aiken Lecture #5 (Modified by Professor Vijay Ganesh)
Introduction to Parsing Lecture 5 (Modified by Professor Vijay Ganesh) 1 Outline Regular languages revisited Parser overview Context-free grammars (CFG s) Derivations Ambiguity 2 Languages and Automata
More informationA lexical analyzer generator for Standard ML. Version 1.6.0, October 1994
A lexical analyzer generator for Standard ML. Version 1.6.0, October 1994 Andrew W. Appel 1 James S. Mattson David R. Tarditi 2 1 Department of Computer Science, Princeton University 2 School of Computer
More informationCOMPILERS AND INTERPRETERS Lesson 4 TDDD16
COMPILERS AND INTERPRETERS Lesson 4 TDDD16 Kristian Stavåker (kristian.stavaker@liu.se) Department of Computer and Information Science Linköping University TODAY Introduction to the Bison parser generator
More informationProperties of Regular Expressions and Finite Automata
Properties of Regular Expressions and Finite Automata Some token patterns can t be defined as regular expressions or finite automata. Consider the set of balanced brackets of the form [[[ ]]]. This set
More informationYacc Yet Another Compiler Compiler
LEX and YACC work as a team Yacc Yet Another Compiler Compiler How to work? Some material adapted from slides by Andy D. Pimentel LEX and YACC work as a team Availability call yylex() NUM + NUM next token
More informationSyntax-Directed Translation. Lecture 14
Syntax-Directed Translation Lecture 14 (adapted from slides by R. Bodik) 9/27/2006 Prof. Hilfinger, Lecture 14 1 Motivation: parser as a translator syntax-directed translation stream of tokens parser ASTs,
More informationSyntax and Parsing COMS W4115. Prof. Stephen A. Edwards Fall 2003 Columbia University Department of Computer Science
Syntax and Parsing COMS W4115 Prof. Stephen A. Edwards Fall 2003 Columbia University Department of Computer Science Lexical Analysis (Scanning) Lexical Analysis (Scanning) Goal is to translate a stream
More informationBottom-Up Parsing. Lecture 11-12
Bottom-Up Parsing Lecture 11-12 (From slides by G. Necula & R. Bodik) 9/22/06 Prof. Hilfinger CS164 Lecture 11 1 Bottom-Up Parsing Bottom-up parsing is more general than topdown parsing And just as efficient
More informationSimple LR (SLR) LR(0) Drawbacks LR(1) SLR Parse. LR(1) Start State and Reduce. LR(1) Items 10/3/2012
LR(0) Drawbacks Consider the unambiguous augmented grammar: 0.) S E $ 1.) E T + E 2.) E T 3.) T x If we build the LR(0) DFA table, we find that there is a shift-reduce conflict. This arises because the
More informationWednesday, September 9, 15. Parsers
Parsers What is a parser A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure of a program (think: diagramming a sentence) Agenda
More informationSYNTAX ANALYSIS 1. Define parser. Hierarchical analysis is one in which the tokens are grouped hierarchically into nested collections with collective meaning. Also termed as Parsing. 2. Mention the basic
More informationParsers. What is a parser. Languages. Agenda. Terminology. Languages. A parser has two jobs:
What is a parser Parsers A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure of a program (think: diagramming a sentence) Agenda
More informationSyntax-Directed Translation
Syntax-Directed Translation ALSU Textbook Chapter 5.1 5.4, 4.8, 4.9 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 What is syntax-directed translation? Definition: The compilation
More informationIntroduction to Parsing. Lecture 8
Introduction to Parsing Lecture 8 Adapted from slides by G. Necula Outline Limitations of regular languages Parser overview Context-free grammars (CFG s) Derivations Languages and Automata Formal languages
More informationLexical and Syntax Analysis
Lexical and Syntax Analysis (of Programming Languages) Bison, a Parser Generator Lexical and Syntax Analysis (of Programming Languages) Bison, a Parser Generator Bison: a parser generator Bison Specification
More informationReview main idea syntax-directed evaluation and translation. Recall syntax-directed interpretation in recursive descent parsers
Plan for Today Review main idea syntax-directed evaluation and translation Recall syntax-directed interpretation in recursive descent parsers Syntax-directed evaluation and translation in shift-reduce
More informationA clarification on terminology: Recognizer: accepts or rejects strings in a language. Parser: recognizes and generates parse trees (imminent topic)
A clarification on terminology: Recognizer: accepts or rejects strings in a language Parser: recognizes and generates parse trees (imminent topic) Assignment 3: building a recognizer for the Lake expression
More informationCS2210: Compiler Construction Syntax Analysis Syntax Analysis
Comparison with Lexical Analysis The second phase of compilation Phase Input Output Lexer string of characters string of tokens Parser string of tokens Parse tree/ast What Parse Tree? CS2210: Compiler
More informationCPS 506 Comparative Programming Languages. Syntax Specification
CPS 506 Comparative Programming Languages Syntax Specification Compiling Process Steps Program Lexical Analysis Convert characters into a stream of tokens Lexical Analysis Syntactic Analysis Send tokens
More informationDefining syntax using CFGs
Defining syntax using CFGs Roadmap Last 8me Defined context-free grammar This 8me CFGs for syntax design Language membership List grammars Resolving ambiguity CFG Review G = (N,Σ,P,S) means derives derives
More informationLECTURE 11. Semantic Analysis and Yacc
LECTURE 11 Semantic Analysis and Yacc REVIEW OF LAST LECTURE In the last lecture, we introduced the basic idea behind semantic analysis. Instead of merely specifying valid structures with a context-free
More informationCPSC 411: Introduction to Compiler Construction
CPSC 411: to Compiler Construction Kris De Volder kdvolder@cs.ubc.ca http://www.ugrad.cs.ubc.ca/~cs411/ 1 This Lecture Course Organization and Policies Website Newsgroup Assignments General Information
More informationParsers. Xiaokang Qiu Purdue University. August 31, 2018 ECE 468
Parsers Xiaokang Qiu Purdue University ECE 468 August 31, 2018 What is a parser A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure
More informationCS 315 Programming Languages Syntax. Parser. (Alternatively hand-built) (Alternatively hand-built)
Programming languages must be precise Remember instructions This is unlike natural languages CS 315 Programming Languages Syntax Precision is required for syntax think of this as the format of the language
More informationCOMP 181. Prelude. Prelude. Summary of parsing. A Hierarchy of Grammar Classes. More power? Syntax-directed translation. Analysis
Prelude COMP 8 October, 9 What is triskaidekaphobia? Fear of the number s? No aisle in airplanes, no th floor in buildings Fear of Friday the th? Paraskevidedekatriaphobia or friggatriskaidekaphobia Why
More informationCMSC 330: Organization of Programming Languages. Context Free Grammars
CMSC 330: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Analyzer Optimizer Code Generator Abstract Syntax Tree Front End Back End Compiler
More informationIntro To Parsing. Step By Step
#1 Intro To Parsing Step By Step #2 Self-Test from Last Time Are practical parsers and scanners based on deterministic or non-deterministic automata? How can regular expressions be used to specify nested
More informationMIT Specifying Languages with Regular Expressions and Context-Free Grammars
MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology Language Definition Problem How to precisely
More informationOutline. Limitations of regular languages. Introduction to Parsing. Parser overview. Context-free grammars (CFG s)
Outline Limitations of regular languages Introduction to Parsing Parser overview Lecture 8 Adapted from slides by G. Necula Context-free grammars (CFG s) Derivations Languages and Automata Formal languages
More informationError Detection in LALR Parsers. LALR is More Powerful. { b + c = a; } Eof. Expr Expr + id Expr id we can first match an id:
Error Detection in LALR Parsers In bottom-up, LALR parsers syntax errors are discovered when a blank (error) entry is fetched from the parser action table. Let s again trace how the following illegal CSX-lite
More informationSyntax Analysis Part IV
Syntax Analysis Part IV Chapter 4: Bison Slides adapted from : Robert van Engelen, Florida State University Yacc and Bison Yacc (Yet Another Compiler Compiler) Generates LALR(1) parsers Bison Improved
More informationCompilers Course Lecture 4: Context Free Grammars
Compilers Course Lecture 4: Context Free Grammars Example: attempt to define simple arithmetic expressions using named regular expressions: num = [0-9]+ sum = expr "+" expr expr = "(" sum ")" num Appears
More informationIntroduction to Lexing and Parsing
Introduction to Lexing and Parsing ECE 351: Compilers Jon Eyolfson University of Waterloo June 18, 2012 1 Riddle Me This, Riddle Me That What is a compiler? 1 Riddle Me This, Riddle Me That What is a compiler?
More informationLab 2. Lexing and Parsing with Flex and Bison - 2 labs
Lab 2 Lexing and Parsing with Flex and Bison - 2 labs Objective Understand the software architecture of flex/bison. Be able to write simple grammars in bison. Be able to correct grammar issues in bison.
More informationDerivations vs Parses. Example. Parse Tree. Ambiguity. Different Parse Trees. Context Free Grammars 9/18/2012
Derivations vs Parses Grammar is used to derive string or construct parser Context ree Grammars A derivation is a sequence of applications of rules Starting from the start symbol S......... (sentence)
More informationDownloaded from Page 1. LR Parsing
Downloaded from http://himadri.cmsdu.org Page 1 LR Parsing We first understand Context Free Grammars. Consider the input string: x+2*y When scanned by a scanner, it produces the following stream of tokens:
More informationLexical Analysis. COMP 524, Spring 2014 Bryan Ward
Lexical Analysis COMP 524, Spring 2014 Bryan Ward Based in part on slides and notes by J. Erickson, S. Krishnan, B. Brandenburg, S. Olivier, A. Block and others The Big Picture Character Stream Scanner
More informationAs we have seen, token attribute values are supplied via yylval, as in. More on Yacc s value stack
More on Yacc s value stack As we noted last time, Yacc uses a second stack to store the attribute values of the tokens and terminals in the parse stack. For a token, the attributes are computed by the
More informationLex & Yacc (GNU distribution - flex & bison) Jeonghwan Park
Lex & Yacc (GNU distribution - flex & bison) Jeonghwan Park Prerequisite Ubuntu Version 14.04 or over Virtual machine for Windows user or native OS flex bison gcc Version 4.7 or over Install in Ubuntu
More informationCMSC 330: Organization of Programming Languages
CMSC 330: Organization of Programming Languages Context Free Grammars and Parsing 1 Recall: Architecture of Compilers, Interpreters Source Parser Static Analyzer Intermediate Representation Front End Back
More informationParsing III. CS434 Lecture 8 Spring 2005 Department of Computer Science University of Alabama Joel Jones
Parsing III (Top-down parsing: recursive descent & LL(1) ) (Bottom-up parsing) CS434 Lecture 8 Spring 2005 Department of Computer Science University of Alabama Joel Jones Copyright 2003, Keith D. Cooper,
More informationProgramming Languages & Translators PARSING. Baishakhi Ray. Fall These slides are motivated from Prof. Alex Aiken: Compilers (Stanford)
Programming Languages & Translators PARSING Baishakhi Ray Fall 2018 These slides are motivated from Prof. Alex Aiken: Compilers (Stanford) Languages and Automata Formal languages are very important in
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ws-1819/cc/ Recap: LR(1) Parsing Outline of Lecture 11 Recap: LR(1)
More informationA Bison Manual. You build a text file of the production (format in the next section); traditionally this file ends in.y, although bison doesn t care.
A Bison Manual 1 Overview Bison (and its predecessor yacc) is a tool that take a file of the productions for a context-free grammar and converts them into the tables for an LALR(1) parser. Bison produces
More information1. Explain the input buffer scheme for scanning the source program. How the use of sentinels can improve its performance? Describe in detail.
Code No: R05320502 Set No. 1 1. Explain the input buffer scheme for scanning the source program. How the use of sentinels can improve its performance? Describe in detail. 2. Construct predictive parsing
More informationSyn S t yn a t x a Ana x lysi y s si 1
Syntax Analysis 1 Position of a Parser in the Compiler Model Source Program Lexical Analyzer Token, tokenval Get next token Parser and rest of front-end Intermediate representation Lexical error Syntax
More informationLECTURE 3. Compiler Phases
LECTURE 3 Compiler Phases COMPILER PHASES Compilation of a program proceeds through a fixed series of phases. Each phase uses an (intermediate) form of the program produced by an earlier phase. Subsequent
More information