Syntax and Type Analysis
|
|
- Cornelia McCormick
- 5 years ago
- Views:
Transcription
1 Syntax and Type Analysis Lecture Compilers Summer Term 2011 Prof. Dr. Arnd Poetzsch-Heffter Software Technology Group TU Kaiserslautern Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 1 Content of Lecture 1. Introduction: Overview and Motivation 2. Syntax- and Type Analysis Context-Free Syntax Analysis 2.3 Context-Dependent Syntax Analysis 3. Translation to Target Language 3.1 Translation of Imperative Language Constructs 3.2 Translation of Object-Oriented Language Constructs 4. Selected Aspects of Compilers 4.1 Intermediate Languages 4.2 Optimization 4.3 Data Flow Analysis 4.4 Register Allocation 4.5 Code Generation 5. Garbage Collection 6. XML Processing (DOM, SAX, XSLT) Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 2
2 2. Syntax and Type Analysis Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 3 Educational Objectives Tasks of different syntax analysis phases Interaction of syntax analysis phases Specification techniques for syntax analysis Generation techniques Usage of tools Lexical analysis Context-free analysis (parsing) Context-sensitive analysis Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 4
3 Syntax Analysis Introduction to Syntax and Type Analysis Tasks of Syntax Analysis Check if input is syntactically correct Dependent on result: Error message Generation of appropriate data structure for subsequent processing Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 5 Introduction to Syntax and Type Analysis Syntax and Type Analysis Phases Source Code Lexical analysis: Character stream token stream (or symbol stream) Context-free analysis: Token stream syntax tree Context-sensitive analysis: Syntax tree syntax tree with cross references Syntax and Type Analysis Character Stream Scanner Token Stream Parser Syntax Tree Name and Type Analysis Attributed Syntax Tree Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 6
4 Introduction to Syntax and Type Analysis Reasons for Separation of Phases Lexical and context-free analysis Reduced load for context-free analysis, e.g., whitespaces are not required for context-free analysis Context-free and context-sensitive analysis Context-sensitive analysis uses tree structure instead of token stream Advantages for construction of target data structure For both cases Increased efficiency Natural process (cmp. natural language) More appropriate tool support Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 8
5 Tasks Break input character stream into a token stream wrt. language definition Classify tokens into token classes Representation of tokens Hashing of identifiers Conversion of constants Elimination of whitespaces (spaces, comments...) external constructs (compiler directives...) Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 9 (2) Terminology Token/symbol: a word over an alphabet of characters (often with additional information, e.g. token class, encoding, position..) Token class: a set of tokens (identifier, constants,...); correspond to terminal symbols of a context-free grammar Remark: the terms token and symbol refer to the same concept. The term token is in general used when talking about parsing technology, whereas the term symbol is used when talking about formal languages. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 10
6 : Example Input Line 23: if ( A <= 3.14 ) B = B Token Class String Token Information Col:Row IF if 23:3 OPAR ( 23:5 ID A 72 (Hash) 23:7 RELOP <= 4 (Encoding) 23:9 FLOATCONST ,14 (Constant Value) 23:12 CPAR ) 23:16 ID B 84 (Hash) 23:20... Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 11 Specification Specification of Scanners The specification of the lexical analysis is a part of the language specification. The two parts of lexical analysis specification: Scanning algorithm (often only implicit) Specification of tokens and token classes Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 12
7 Examples: Scanning Specification of Scanners 1. Statement in C B = B --- A; Problem: Separation ( - - and - are tokens) Solution: Longest token is chosen, i.e, B = B -- - A; 2. Java Fragment class public { public m() {...} } Problem: Ambiguity (keyword, identifier) Solution: Precedence rules Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 13 Specification of Scanners Standard Scan Algorithm (Concept) Scanning is often implemented as a procedure: Procedure returns next token State is remainder of input In error cases, returns the UNDEF token and updates the input Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 14
8 Specification of Scanners Standard Scan Algorithm (Pseudo Code) CharStream inputrest := input; Token nexttoken() { Token curtoken := longesttokenprefix(inputrest); inputrest:= cut(curtoken, inputrest); return curtoken; } where cut is defined as if curtoken UNDEF, curtoken is removed from inputrest else inputrest remains unchanged. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 15 Specification of Scanners Standard Scan Algorithm (2) Token longesttokenprefix(charstream ir) { require availablechar(ir) > 0 int curlength = 1; String curprefix := prefix(curlength,ir); Token longesttoken := UNDEF; } while( curlength <= availablechar(ir) && istokenprefix(curprefix) ) { if (istoken(curprefix) { longesttoken := curprefix; } curlength++; curprefix := prefix(curlength,ir); } return longesttoken; Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 16
9 Standard Scan Algorithm (3) Specification of Scanners Predicates to be defined: istokenprefix: String boolean istoken: String boolean Remarks: Standard scan algorithm is used in many modern languages, but not, e.g., in FORTRAN because blanks are not special, except in literal tokens, e.g. DO 7 I = 1.25 DO 7 I is an identifier. DO 7 I = 1,25 DO is a keyword. Error cases are not handled Complete realization of longesttokenprefix is discussed later. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 17 Specification of Scanners Specification of Token Classes Token classes are defined by regular expressions (REs). REs specify the set of strings, which belong to a certain token class. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 18
10 Regular Expressions Specification of Scanners Let Σ be an alphabet, i.e. an non-empty set of characters. Σ is the set of all words over Σ, ɛ is the empty word. Definition (Regular expressions, regular languages) ε is a RE and specifies the language L = {ɛ}. Each a Σ is a RE and specifies the language L = {a}. Let r and s be two RE specifying the languages R and S, resp. Then the following are RE and specify the language L: (r s) with L = R S (union) rs with L = {vw v R, w S} (concatenation) r with {v 1... v n v i R, 0 i n} (Kleene star) The language L Σ is called regular if there exists RE r defining L. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 19 Regular Expressions (2) Specification of Scanners Remarks: L = is not regular according to the definition, but is often considered regular. Other Operators, e.g. +,?,., [] can be defined using the basic operators, e.g. r + (r r ) r \ {ɛ} [abd] a B d [a g] a b c d e f g Caution: Regular expressions only define valid tokens and do not specify the program or translation units of a programming language. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 20
11 sequence of regular expressions and actions (input language of scanner generator) Scanner Generator scanner program (usually in a programming language) Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 21 Scanner Generator: JFlex Typical use of JFlex: java -jar JFlex.jar Example.jflex javac Yylex.java Actions are written in Java Examples : 1. Regular expression in JFlex [a-za-z_0-9] [a-za-z_0-9] * 2. JFlex input with abbreviations ZI = [0-9] BU = [a-za-z_] BUZI = [a-za-z_0-9] %% {BU}{BUZI}* { anaction(); } Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 22
12 A Complete JFlex Example enum Token { DO, DOUBLE, IDENT, FLOATCONST, STRING;} %% %line %column %debug %type Token // declare token type ZI = [0-9] BU = [a-za-z_] BUZI = [a-za-z_0-9] ZE = [a-za-z_0-9?\]\[\. \t...] WhiteSpace = [ \t\n] %% {WhiteSpace} { } "double" { return Token.DOUBLE; } "do" { return Token.DO; } {BU}{BUZI}* { return Token.IDENT; } {ZI}+\.{ZI}+ { return Token.FLOATCONST; } \"({ZE} \\\")*\" { return Token.STRING; } <<EOF>> { System.out.println("FINISHED"); return null; } Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 23 Scanner Generators Scanner generation uses the equivalence between Regular expressions Non-deterministic finite automata (NFA) Deterministic finite automata (DFA) Construction methods is based in two steps: Regular expressions NFA NFA DFA Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 24
13 Definition of NFA Definition (Non-deterministic Finite Automaton) A non-deterministic finite automaton is defined as a 5-tuple where Σ is the input alphabet Q is the set of states q 0 Q is the initial state F Q is the set of final states M = (Σ, Q,, q 0, F) Q Σ {ɛ} Q is the transition relation. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis Schritt: Reguläre Implementation of Ausdrücke Scanners NEA Regular Expressions NFA Übersetzungsschema: Prinzip: Konstruiere für jeden regulären Teilausdruck NEA mit genau einem Start- und Endzustand der die gleiche Sprache akzeptiert. Principle: For each regular sub-expression, construct NFA with one start and end state that accepts the same language. s 0 a s a 0 f 0 (r s) s 1 R f 1 s 0 f 0 s 2 S f 2 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 26
14 a a Regular Expressions a NFA (2) (r s) (r s) a (r s) der 0 die gleiche Sprache akzeptiert. s a 0 s 0 s a f 0 0 f 0 s 0 f 0 s 1 R f s 1 1 R f 1 s 0 f s 0 0 s 1 R f 1 s 0 s f 2 s S f 0 2 S 2 f s 2 2 f 2 (rs) (rs) (rs) s s 1 R f 1 s 1 2 S f 1 RR f 1 f 2 1 s 2 S f 2 r* r* r* s 0 s s 0 0 s 1 R f 1 s 1 s 1 R f 1 f f 0 f A. Poetzsch-Heffter, TU Kaiserslautern A. Poetzsch-Heffter, TU Kaiserslautern A. Poetzsch-Heffter, TU Kaiserslautern Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 27 Example: Construction of NFA LZ, TAB Übersetzung am Beispiel von Folie 41: s 1 s d 2 s o 3 s 4 s 0 s d 5 s o 6 s u 7 s b 8 s l 9 s e 10 s 11 s 12 BU s 13 BUZI ZI ZI s ZI. 14 s 15 s ZI 16 s 17 s 18 s 19 ZE s 20 s 21 \ s 22 s 23 s 24 s 25 s 26 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 28
15 ɛ-closure Function closure computes the ɛ-closure of a set of states s 1,..., s n. Definition (ɛ-closure) For an NFA M = (Σ, Q,, q 0, F) and a state q Q, the ɛ-closure of q is defined by ɛ-closure(q) = {p Q p reachable from q via ɛ-transitions} For S Q, the ɛ-closure of S is defined by ɛ-closure(s) = s S ɛ-closure(s) Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 29 Longest Token Prefix with NFA Token longesttokenprefix(char[] ir) { // length(ir) > 0 StateSet curstate := closure( {s0} ); int curlength := 0; int tokenlength := undef; } while (curlength <= length(ir) && isemptyset(curstate) ) { if (contains(curstate,finalstate)) { tokenlength := curlength; } curlength++; curstate := closure(successor(curstate,ir[curlength])); } return token(prefix(ir,tokenlength)); Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 30
16 Longest Token Prefix with NFA (2) Remark: Problem of ambiguity: If there are more than one token matching the longest input prefix, procedure token nondeterministically returns one of them. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 31 NFA DFA Principle: For each NFA, a DFA can be constructed that accepts the same language. (In general, this does not hold for NFA with output.) Properties of DFA: No ɛ-transitions Transitions are deterministic given the input char Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 32
17 NFA DFA (2) Definition (Deterministic Finite State Automaton) A deterministic finite automaton is defined as a 5-tuple where Σ is the input alphabet Q is the set of states q 0 Q is the initial state F Q is the set of final states M = (Σ, Q,, q 0, F) : Q Σ Q is the transition function. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 33 NFA DFA (3) Construction: (according to John Myhill) The States of the DFA are subsets of NFA states (powerset construction). Subsets of finite sets are also finite. The start state of the DFA is the ɛ-closure of the NFA start state The final states of the DFA are the sets of states that contain an NFA final state. The successor state of a state S in the DFA under input a is obtained by computing all successors p of q S under a in the NFA and adding the ɛ-closure of p Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 34
18 NFA DFA (4) If working with character classes (e.g. [a-f]), characters and character classes at outgoing transitions must be disjoint. Completion of automaton for error handling: Insert additional (final) state (nt) For each state, add a transition for each character for which no outgoing transition exists to the nontoken state. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 35 NFA DFA (5) Definition (DFA for NFA) Let M = (Σ, Q,, q 0, F) be a NFA. Then, the DFA M corresponding to the NFA M is defined as M = (Σ, Q,, q 0, F ) where the set of states is Q P(Q), power set of Q the initial state q 0 is the ɛ-closure of q 0 the final states are F = {S Q S F } (S, a) = ɛ-closure({p (q, a, p), q S}) for all a Σ. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 36
19 Example: DFA s 11,13 e Wg. Übersichtlichkeit Kanten zu ks nur angedeutet. s 10,13 Transitions to nt sketched. nt ks l LZ, TAB s 1 BUZI\{e} s 9,13 b BUZI BUZI\{l} s 8,13 u BUZI\{b} s 4,7,13 o LZ, TAB s 13 BUZI\{u} s3,6,13 BUZI BUZI\{o} BU\{d} d ZI ZI s 0,1,2,5,12,14,18 s 17 s 26 ZI. ZI s 16 s 15 ZE s 19,20,21,22,25 s ZE 19,20,22,25 \ ZE \ ZE s19,20,21,22,23,25 s 19,20,22,24,25,26 \ \ Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 37 Longest Token Prefix with DFA Token longesttokenprefix(char[] ir) { // length(ir) > 0 State curstate : = StartState; int curlength := 0; int tokenlength := undef; } while (curlength <= length(ir) && curstate = nt) if (curstate is FinalState) { tokenlength := curlength; } curlength++; curstate := successor(curstate,ir[curlength])); } return token(prefix(ir,tokenlength)); Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 38
20 Longest Token Prefix with DFA (2) Remarks: Computation of closure at construction time, not at runtime. (Principle: Do as much statically as you can) Problem of ambiguity still not solved. However, many scanner generators allows the user to control which token is returned. For example, JFlex returns the token of the first rule in the JFlex file that matches the longest input prefix. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 39 Longest Token Prefix with DFA (3) Implementation Aspects: Constructed DFA can be minimized. Input buffering is important: often use of cyclic arrays (caution with maximal token length, e.g. in case of comments) Encode DFA in table Choose suitable partitioning of alphabet in order to reduce number of transitions (i.e. size of table) Interface with parser: usually parser asks proactively for next token Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 40
21 Recommended Reading Wilhelm, Maurer: Chap. 7, pp (More theoretical) Appel: Chap 2, pp (More practical) Additional Reading: Aho, Sethi, Ullman: Chap. 3 (very detailed) Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 41
2. Syntax and Type Analysis
Content of Lecture Syntax and Type Analysis Lecture Compilers Summer Term 2011 Prof. Dr. Arnd Poetzsch-Heffter Software Technology Group TU Kaiserslautern Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type
More informationCompiler phases. Non-tokens
Compiler phases Compiler Construction Scanning Lexical Analysis source code scanner tokens regular expressions lexical analysis Lennart Andersson parser context free grammar Revision 2011 01 21 parse tree
More informationIntroduction to Lexical Analysis
Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular
More informationLexical Analysis. Chapter 2
Lexical Analysis Chapter 2 1 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples
More informationLexical Analysis. Introduction
Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies
More informationLexical Analysis - 2
Lexical Analysis - 2 More regular expressions Finite Automata NFAs and DFAs Scanners JLex - a scanner generator 1 Regular Expressions in JLex Symbol - Meaning. Matches a single character (not newline)
More informationLexical Analysis. Lecture 2-4
Lexical Analysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 2 1 Administrivia Moving to 60 Evans on Wednesday HW1 available Pyth manual available on line.
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation
More informationThe Front End. The purpose of the front end is to deal with the input language. Perform a membership test: code source language?
The Front End Source code Front End IR Back End Machine code Errors The purpose of the front end is to deal with the input language Perform a membership test: code source language? Is the program well-formed
More informationAnnouncements! P1 part 1 due next Tuesday P1 part 2 due next Friday
Announcements! P1 part 1 due next Tuesday P1 part 2 due next Friday 1 Finite-state machines CS 536 Last time! A compiler is a recognizer of language S (Source) a translator from S to T (Target) a program
More information2010: Compilers REVIEW: REGULAR EXPRESSIONS HOW TO USE REGULAR EXPRESSIONS
2010: Compilers Lexical Analysis: Finite State Automata Dr. Licia Capra UCL/CS REVIEW: REGULAR EXPRESSIONS a Character in A Empty string R S Alternation (either R or S) RS Concatenation (R followed by
More informationLexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast!
Lexical Analysis Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Compiler Passes Analysis of input program (front-end) character stream
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-16/cc/ Conceptual Structure of a Compiler Source code x1 := y2
More informationOutline. 1 Scanning Tokens. 2 Regular Expresssions. 3 Finite State Automata
Outline 1 2 Regular Expresssions Lexical Analysis 3 Finite State Automata 4 Non-deterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA 6 NFA to DFA 7 8 JavaCC:
More informationLexical Analysis 1 / 52
Lexical Analysis 1 / 52 Outline 1 Scanning Tokens 2 Regular Expresssions 3 Finite State Automata 4 Non-deterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA
More informationLexical Analysis. Lecture 3-4
Lexical Analysis Lecture 3-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 3-4 1 Administrivia I suggest you start looking at Python (see link on class home page). Please
More informationCSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B-1
CSEP 501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2008 1/8/2008 2002-08 Hal Perkins & UW CSE B-1 Agenda Basic concepts of formal grammars (review) Regular expressions
More informationCSc 453 Lexical Analysis (Scanning)
CSc 453 Lexical Analysis (Scanning) Saumya Debray The University of Arizona Tucson Overview source program lexical analyzer (scanner) tokens syntax analyzer (parser) symbol table manager Main task: to
More informationChapter 3 Lexical Analysis
Chapter 3 Lexical Analysis Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator The role of lexical
More informationEDAN65: Compilers, Lecture 02 Regular expressions and scanning. Görel Hedin Revised:
EDAN65: Compilers, Lecture 02 Regular expressions and scanning Görel Hedin Revised: 2014-09- 01 Course overview Regular expressions Context- free grammar ARribute grammar Lexical analyzer (scanner) SyntacIc
More informationRegular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications
Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata
More informationCompiler course. Chapter 3 Lexical Analysis
Compiler course Chapter 3 Lexical Analysis 1 A. A. Pourhaji Kazem, Spring 2009 Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata
More informationUNIT -2 LEXICAL ANALYSIS
OVER VIEW OF LEXICAL ANALYSIS UNIT -2 LEXICAL ANALYSIS o To identify the tokens we need some method of describing the possible tokens that can appear in the input stream. For this purpose we introduce
More informationCS412/413. Introduction to Compilers Tim Teitelbaum. Lecture 2: Lexical Analysis 23 Jan 08
CS412/413 Introduction to Compilers Tim Teitelbaum Lecture 2: Lexical Analysis 23 Jan 08 Outline Review compiler structure What is lexical analysis? Writing a lexer Specifying tokens: regular expressions
More informationLexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata
Lexical Analysis Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Phase Ordering of Front-Ends Lexical analysis (lexer) Break input string
More informationCompiler Construction
Compiler Construction Lecture 2: Lexical Analysis I (Introduction) Thomas Noll Lehrstuhl für Informatik 2 (Software Modeling and Verification) noll@cs.rwth-aachen.de http://moves.rwth-aachen.de/teaching/ss-14/cc14/
More informationLexical Analysis. Chapter 1, Section Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual
Lexical Analysis Chapter 1, Section 1.2.1 Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual Inside the Compiler: Front End Lexical analyzer (aka scanner) Converts ASCII or Unicode to a stream of tokens
More informationLexical Analysis. Implementation: Finite Automata
Lexical Analysis Implementation: Finite Automata Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs)
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Lecture 4 (Modified by Professor Vijay Ganesh) Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested
More information5. Garbage Collection
Content of Lecture Compilers and Language Processing Tools Summer Term 2011 Prof. Dr. Arnd Poetzsch-Heffter Software Technology Group TU Kaiserslautern c Prof. Dr. Arnd Poetzsch-Heffter 1 1. Introduction
More informationFigure 2.1: Role of Lexical Analyzer
Chapter 2 Lexical Analysis Lexical analysis or scanning is the process which reads the stream of characters making up the source program from left-to-right and groups them into tokens. The lexical analyzer
More informationCSE 413 Programming Languages & Implementation. Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions
CSE 413 Programming Languages & Implementation Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory
More informationDavid Griol Barres Computer Science Department Carlos III University of Madrid Leganés (Spain)
David Griol Barres dgriol@inf.uc3m.es Computer Science Department Carlos III University of Madrid Leganés (Spain) OUTLINE Introduction: Definitions The role of the Lexical Analyzer Scanner Implementation
More informationFormal Languages and Compilers Lecture IV: Regular Languages and Finite. Finite Automata
Formal Languages and Compilers Lecture IV: Regular Languages and Finite Automata Free University of Bozen-Bolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/
More informationCompiler Construction I
TECHNISCHE UNIVERSITÄT MÜNCHEN FAKULTÄT FÜR INFORMATIK Compiler Construction I Dr. Michael Petter SoSe 2015 1 / 58 Organizing Master or Bachelor in the 6th Semester with 5 ECTS Prerequisites Informatik
More informationOutline CS4120/4121. Compilation in a Nutshell 1. Administration. Introduction to Compilers Andrew Myers. HW1 out later today due next Monday.
CS4120/4121 Introduction to Compilers Andrew Myers Lecture 2: Lexical Analysis 31 August 2009 Outline Administration Compilation in a nutshell (or two) What is lexical analysis? Writing a lexer Specifying
More informationComputer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres
Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres dgriol@inf.uc3m.es Introduction: Definitions Lexical analysis or scanning: To read from left-to-right a source
More informationCSE302: Compiler Design
CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 13, 2007 Outline Recap
More informationCSE 413 Programming Languages & Implementation. Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions
CSE 413 Programming Languages & Implementation Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory
More informationAdministrivia. Lexical Analysis. Lecture 2-4. Outline. The Structure of a Compiler. Informal sketch of lexical analysis. Issues in lexical analysis
dministrivia Lexical nalysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Moving to 6 Evans on Wednesday HW available Pyth manual available on line. Please log into your account and electronically
More informationCompiler Construction I
TECHNISCHE UNIVERSITÄT MÜNCHEN FAKULTÄT FÜR INFORMATIK Compiler Construction I Dr. Michael Petter, Dr. Axel Simon SoSe 2014 1 / 59 Organizing Master or Bachelor in the 6th Semester with 5 ECTS Prerequisites
More informationCS321 Languages and Compiler Design I. Winter 2012 Lecture 4
CS321 Languages and Compiler Design I Winter 2012 Lecture 4 1 LEXICAL ANALYSIS Convert source file characters into token stream. Remove content-free characters (comments, whitespace,...) Detect lexical
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-16/cc/ Recap: First-Longest-Match Analysis Outline of Lecture
More informationImplementation of Lexical Analysis
Outline Implementation of Lexical nalysis Specifying lexical structure using regular expressions Finite automata Deterministic Finite utomata (DFs) Non-deterministic Finite utomata (NFs) Implementation
More informationFormal Languages and Compilers Lecture VI: Lexical Analysis
Formal Languages and Compilers Lecture VI: Lexical Analysis Free University of Bozen-Bolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/ artale/ Formal
More informationConcepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective
Concepts Lexical scanning Regular expressions DFAs and FSAs Lex CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 2 Lexical analysis
More informationLexical Analysis - An Introduction. Lecture 4 Spring 2005 Department of Computer Science University of Alabama Joel Jones
Lexical Analysis - An Introduction Lecture 4 Spring 2005 Department of Computer Science University of Alabama Joel Jones Copyright 2003, Keith D. Cooper, Ken Kennedy & Linda Torczon, all rights reserved.
More informationCS 310: State Transition Diagrams
CS 30: State Transition Diagrams Stefan D. Bruda Winter 207 STATE TRANSITION DIAGRAMS Finite directed graph Edges (transitions) labeled with symbols from an alphabet Nodes (states) labeled only for convenience
More informationChapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective
Chapter 4 Lexical analysis Lexical scanning Regular expressions DFAs and FSAs Lex Concepts CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley
More informationLexical Analysis. Lecture 3. January 10, 2018
Lexical Analysis Lecture 3 January 10, 2018 Announcements PA1c due tonight at 11:50pm! Don t forget about PA1, the Cool implementation! Use Monday s lecture, the video guides and Cool examples if you re
More informationFinite automata. We have looked at using Lex to build a scanner on the basis of regular expressions.
Finite automata We have looked at using Lex to build a scanner on the basis of regular expressions. Now we begin to consider the results from automata theory that make Lex possible. Recall: An alphabet
More informationWriting a Lexical Analyzer in Haskell (part II)
Writing a Lexical Analyzer in Haskell (part II) Today Regular languages and lexicographical analysis part II Some of the slides today are from Dr. Saumya Debray and Dr. Christian Colberg This week PA1:
More informationIntroduction to Lexical Analysis
Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-17/cc/ Recap: First-Longest-Match Analysis The Extended Matching
More informationNondeterministic Finite Automata (NFA): Nondeterministic Finite Automata (NFA) states of an automaton of this kind may or may not have a transition for each symbol in the alphabet, or can even have multiple
More informationLecture 4: Syntax Specification
The University of North Carolina at Chapel Hill Spring 2002 Lecture 4: Syntax Specification Jan 16 1 Phases of Compilation 2 1 Syntax Analysis Syntax: Webster s definition: 1 a : the way in which linguistic
More informationCS Lecture 2. The Front End. Lecture 2 Lexical Analysis
CS 1622 Lecture 2 Lexical Analysis CS 1622 Lecture 2 1 Lecture 2 Review of last lecture and finish up overview The first compiler phase: lexical analysis Reading: Chapter 2 in text (by 1/18) CS 1622 Lecture
More informationZhizheng Zhang. Southeast University
Zhizheng Zhang Southeast University 2016/10/5 Lexical Analysis 1 1. The Role of Lexical Analyzer 2016/10/5 Lexical Analysis 2 2016/10/5 Lexical Analysis 3 Example. position = initial + rate * 60 2016/10/5
More informationCMSC 350: COMPILER DESIGN
Lecture 11 CMSC 350: COMPILER DESIGN see HW3 LLVMLITE SPECIFICATION Eisenberg CMSC 350: Compilers 2 Discussion: Defining a Language Premise: programming languages are purely formal objects We (as language
More informationMIT Specifying Languages with Regular Expressions and Context-Free Grammars
MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology Language Definition Problem How to precisely
More informationConcepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens
Concepts Introduced in Chapter 3 Lexical Analysis Regular Expressions (REs) Nondeterministic Finite Automata (NFA) Converting an RE to an NFA Deterministic Finite Automatic (DFA) Lexical Analysis Why separate
More informationCSE Lecture 4: Scanning and parsing 28 Jan Nate Nystrom University of Texas at Arlington
CSE 5317 Lecture 4: Scanning and parsing 28 Jan 2010 Nate Nystrom University of Texas at Arlington Administrivia hcp://groups.google.com/group/uta- cse- 3302 I will add you to the group soon TA Derek White
More informationTheoretical Part. Chapter one:- - What are the Phases of compiler? Answer:
Theoretical Part Chapter one:- - What are the Phases of compiler? Six phases Scanner Parser Semantic Analyzer Source code optimizer Code generator Target Code Optimizer Three auxiliary components Literal
More informationFormal Languages and Grammars. Chapter 2: Sections 2.1 and 2.2
Formal Languages and Grammars Chapter 2: Sections 2.1 and 2.2 Formal Languages Basis for the design and implementation of programming languages Alphabet: finite set Σ of symbols String: finite sequence
More informationCS 403: Scanning and Parsing
CS 403: Scanning and Parsing Stefan D. Bruda Fall 2017 THE COMPILATION PROCESS Character stream Scanner (lexical analysis) Token stream Parser (syntax analysis) Parse tree Semantic analysis Abstract syntax
More informationProf. Mohamed Hamada Software Engineering Lab. The University of Aizu Japan
Compilers Prof. Mohamed Hamada Software Engineering Lab. The University of Aizu Japan Lexical Analyzer (Scanner) 1. Uses Regular Expressions to define tokens 2. Uses Finite Automata to recognize tokens
More informationLecture 3: Lexical Analysis
Lecture 3: Lexical Analysis COMP 524 Programming Language Concepts tephen Olivier January 2, 29 Based on notes by A. Block, N. Fisher, F. Hernandez-Campos, J. Prins and D. totts Goal of Lecture Character
More informationCS308 Compiler Principles Lexical Analyzer Li Jiang
CS308 Lexical Analyzer Li Jiang Department of Computer Science and Engineering Shanghai Jiao Tong University Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular
More informationCS 314 Principles of Programming Languages. Lecture 3
CS 314 Principles of Programming Languages Lecture 3 Zheng Zhang Department of Computer Science Rutgers University Wednesday 14 th September, 2016 Zheng Zhang 1 CS@Rutgers University Class Information
More informationCOMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou
COMP-421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3 - Lexical Analysis Sections 3.1-3.4, 3.6-3.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna
More information[Lexical Analysis] Bikash Balami
1 [Lexical Analysis] Compiler Design and Construction (CSc 352) Compiled By Central Department of Computer Science and Information Technology (CDCSIT) Tribhuvan University, Kirtipur Kathmandu, Nepal 2
More informationMIT Specifying Languages with Regular Expressions and Context-Free Grammars. Martin Rinard Massachusetts Institute of Technology
MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Massachusetts Institute of Technology Language Definition Problem How to precisely define language Layered structure
More informationCS415 Compilers. Lexical Analysis
CS415 Compilers Lexical Analysis These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Lecture 7 1 Announcements First project and second homework
More informationLanguages, Automata, Regular Expressions & Scanners. Winter /8/ Hal Perkins & UW CSE B-1
CSE 401 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2010 1/8/2010 2002-10 Hal Perkins & UW CSE B-1 Agenda Quick review of basic concepts of formal grammars Regular
More informationLexical Analyzer Scanner
Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce
More informationTHE COMPILATION PROCESS EXAMPLE OF TOKENS AND ATTRIBUTES
THE COMPILATION PROCESS Character stream CS 403: Scanning and Parsing Stefan D. Bruda Fall 207 Token stream Parse tree Abstract syntax tree Modified intermediate form Target language Modified target language
More information1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System can be represented as shown figure below.
UNIT I Translator: It is a program that translates one language to another Language. Examples of translator are compiler, assembler, interpreter, linker, loader and preprocessor. Source Code Translator
More informationLexical Analyzer Scanner
Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce
More informationRegular Languages. MACM 300 Formal Languages and Automata. Formal Languages: Recap. Regular Languages
Regular Languages MACM 3 Formal Languages and Automata Anoop Sarkar http://www.cs.sfu.ca/~anoop The set of regular languages: each element is a regular language Each regular language is an example of a
More information2. Lexical Analysis! Prof. O. Nierstrasz!
2. Lexical Analysis! Prof. O. Nierstrasz! Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes.! http://www.cs.ucla.edu/~palsberg/! http://www.cs.purdue.edu/homes/hosking/!
More informationCPSC 434 Lecture 3, Page 1
Front end source code tokens scanner parser il errors Responsibilities: recognize legal procedure report errors produce il preliminary storage map shape the code for the back end Much of front end construction
More informationLexical Analysis. COMP 524, Spring 2014 Bryan Ward
Lexical Analysis COMP 524, Spring 2014 Bryan Ward Based in part on slides and notes by J. Erickson, S. Krishnan, B. Brandenburg, S. Olivier, A. Block and others The Big Picture Character Stream Scanner
More informationPart 5 Program Analysis Principles and Techniques
1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape
More informationCS 536 Introduction to Programming Languages and Compilers Charles N. Fischer Lecture 5
CS 536 Introduction to Programming Languages and Compilers Charles N. Fischer Lecture 5 CS 536 Spring 2015 1 Multi Character Lookahead We may allow finite automata to look beyond the next input character.
More informationCompiler Construction LECTURE # 3
Compiler Construction LECTURE # 3 The Course Course Code: CS-4141 Course Title: Compiler Construction Instructor: JAWAD AHMAD Email Address: jawadahmad@uoslahore.edu.pk Web Address: http://csandituoslahore.weebly.com/cc.html
More informationNon-deterministic Finite Automata (NFA)
Non-deterministic Finite Automata (NFA) CAN have transitions on the same input to different states Can include a ε or λ transition (i.e. move to new state without reading input) Often easier to design
More informationAbout the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. Compiler Design
i About the Tutorial A compiler translates the codes written in one language to some other language without changing the meaning of the program. It is also expected that a compiler should make the target
More informationLexical Analysis. Finite Automata
#1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical
More informationInterpreter. Scanner. Parser. Tree Walker. read. request token. send token. send AST I/O. Console
Scanning 1 read Interpreter Scanner request token Parser send token Console I/O send AST Tree Walker 2 Scanner This process is known as: Scanning, lexing (lexical analysis), and tokenizing This is the
More informationCS 4240: Compilers and Interpreters Project Phase 1: Scanner and Parser Due Date: October 4 th 2015 (11:59 pm) (via T-square)
CS 4240: Compilers and Interpreters Project Phase 1: Scanner and Parser Due Date: October 4 th 2015 (11:59 pm) (via T-square) Introduction This semester, through a project split into 3 phases, we are going
More informationGarbage Collection. Lecture Compilers SS Dr.-Ing. Ina Schaefer. Software Technology Group TU Kaiserslautern. Ina Schaefer Garbage Collection 1
Garbage Collection Lecture Compilers SS 2009 Dr.-Ing. Ina Schaefer Software Technology Group TU Kaiserslautern Ina Schaefer Garbage Collection 1 Content of Lecture 1. Introduction: Overview and Motivation
More informationCunning Plan. Informal Sketch of Lexical Analysis. Issues in Lexical Analysis. Specifying Lexers
Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical Analysis Lookahead Ambiguity Specifying Lexers Regular Expressions
More informationPRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer
PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer As the first phase of a compiler, the main task of the lexical analyzer is to read the input
More informationCOMPILER DESIGN UNIT I LEXICAL ANALYSIS. Translator: It is a program that translates one language to another Language.
UNIT I LEXICAL ANALYSIS Translator: It is a program that translates one language to another Language. Source Code Translator Target Code 1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System
More informationCompiler Construction D7011E
Compiler Construction D7011E Lecture 2: Lexical analysis Viktor Leijon Slides largely by Johan Nordlander with material generously provided by Mark P. Jones. 1 Basics of Lexical Analysis: 2 Some definitions:
More informationCSCI312 Principles of Programming Languages!
CSCI312 Principles of Programming Languages!! Chapter 3 Regular Expression and Lexer Xu Liu Recap! Copyright 2006 The McGraw-Hill Companies, Inc. Clite: Lexical Syntax! Input: a stream of characters from
More informationDr. D.M. Akbar Hussain
1 2 Compiler Construction F6S Lecture - 2 1 3 4 Compiler Construction F6S Lecture - 2 2 5 #include.. #include main() { char in; in = getch ( ); if ( isalpha (in) ) in = getch ( ); else error (); while
More informationScanners. Xiaokang Qiu Purdue University. August 24, ECE 468 Adapted from Kulkarni 2012
Scanners Xiaokang Qiu Purdue University ECE 468 Adapted from Kulkarni 2012 August 24, 2016 Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved
More informationCOP4020 Programming Languages. Syntax Prof. Robert van Engelen
COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview n Tokens and regular expressions n Syntax and context-free grammars n Grammar derivations n More about parse trees n Top-down and
More information