CS Lecture 2. The Front End. Lecture 2 Lexical Analysis


 Martin Henderson
 2 years ago
 Views:
Transcription
1 CS 1622 Lecture 2 Lexical Analysis CS 1622 Lecture 2 1 Lecture 2 Review of last lecture and finish up overview The first compiler phase: lexical analysis Reading: Chapter 2 in text (by 1/18) CS 1622 Lecture 2 2 The Front End Source code Scanner tokens Parser Responsibilites: Recognize legal and illegal programs Report errors meaningfully Produce and initial storage map Shape the code for the backend Typically automatically constructed From a lexical specification Based on finite automata (meet theory) Very well understood CS 1622 Lecture 2 3 1
2 Source code Scanner tokens Parser Maps characters into tokens  basic lexical units x = y + z becomes <id> <assign> <id> <binop> <id> Lexeme = string that matches the token x, y, and z are lexemes that match <id> Some tokens have attributes <id, x> or <binop, plus> Eliminates whitespace In some languages performs preprocessing (in C done by the preprocessor) CS 1622 Lecture 2 4 Source code Scanner tokens Parser Recognizes syntactic structure & errors Directs semantic analysis (type checking) Builds for source program For some languages (more precisely: grammars) can be easily built by hand More flexible: use parser generators Can change language more easily Typically very fast Well undestood theory ( Pushdown automata CS 1622 Lecture 2 5 Grammars A concise and precise way to specify languages For contextfree grammars can build efficient parsers Can typically write a CFG for a programming language Tool of choice for specifying syntactic structure CS 1622 Lecture 2 6 2
3 Grammars Formally, a grammar G = (S,N,T,P) S is the start symbol N is a set of nonterminal symbols T is a set of terminal symbols or words P is a set of productions or rewrite rules (P : N N T ) CS 1622 Lecture 2 7 CFG Example 1. goal expr 2. expr expr op term 3. term 4. term number 5. id 6. op S = goal T = { number, id, +,  } N = { goal, expr, term, op } P = { 1, 2, 3, 4, 5, 6, 7} CS 1622 Lecture 2 8 Deriving Sentences Production Result goal 1 expr 2 expr op term 5 expr op y 7 expr  y 2 expr op term  y 4 expr op 2  y 6 expr y 3 term y 5 x y To recognize a valid sentence for some CFG, we reverse this process and build up a parse CS 1622 Lecture 2 9 3
4 Parse Tree x y goal expr expr op term expr op term  <id,y> term <id,x> + <number,2> Lots of superfluous detail. 1. goal expr 2. expr expr op term 3. term 4. term number 5. id 6. op CS 1622 Lecture 2 10 Abstract Syntax Tree (AST)  <id,x> + <number,2> <id,y> The AST summarizes grammatical structure, without including detail about the derivation This is much more concise ASTs are one form of intermediate representation () CS 1622 Lecture 2 11 The Back End  instruction selection Instruction Selection Instruction Scheduling Register Allocation Machine code Responsibilities: Translates to target code Selects target instructions for (trivial for RISC) Allocates machine resources (registers, memory) Typically implemented manually For CISC some automated pattern matching approaches Lots of handcrafting done for good backends  must know target architecture well! CS 1622 Lecture
5 Back end  instruction scheduling Instruction Selection Instruction Scheduling Register Allocation Machine code Avoid hardware stalls and interlocks Use all functional units productively Can increase lifetime of variables Optimal scheduling is NPComplete in nearly all cases but good heuristic techniques are well understood CS 1622 Lecture 2 13 Back end  register allocation Instruction Selection Instruction Scheduling Register Allocation Machine code Have each value in a register when it is used Manage a limited set of resources Can change instruction choices & insert LOADs & STOREs Optimal allocation is NPComplete approximate CS 1622 Lecture 2 14 Traditional Threepass Compiler Source Code Front End Middle End Back End Machine code Analyzes and rewrites (or transforms) Primary goal is to reduce running time of the compiled code May also improve space, power consumption, Must preserve meaning of the code CS 1622 Lecture
6 The Optimizer Opt Opt Opt... Opt n Discover & propagate some constant value Move a computation to a less frequently executed place Specialize some computation based on context Discover a redundant computation & remove it Remove useless or unreachable code Encode an idiom in some particularly efficient form CS 1622 Lecture 2 16 The Scanner: Overview Task: translate the sequence of characters to a corresponding sequence of tokens  essentially grouping characters into words removing irrevelant characters  e.g., white space Each time the scanner is called, it should find the longest sequence of characters in the input starting with the current character that corresponds to a token, and return that token. CS 1622 Lecture 2 17 How to write a scanner? write it from scratch, or automatically generate it with a scanner generator lex or flex (produce C code), or jlex (produces Java code). input to a scanner generator: one regular expression for each token output of a scanner generator: a finite state machine so, you need to understand: regular expressions finite automata. CS 1622 Lecture
7 Lexical analyzers Goals: To simplify specification & implementation of scanners To understand the underlying techniques and technologies source code specifications Scanner Scanner Generator parts of speech tables or code CS 1622 Lecture 2 19 Regular Expressions to Finite Automata Generating a scanner NFA Regular expressions DFA Lexical Specification Tabledriven Implementation of DFA CS 1622 Lecture 2 20 Recognizing words Example  begin b e g i n s s 0 1 s 2 s 3 s 4 s 5 c= next char; if c!= b then error c = next char; if c!= e the error; c = next char; if c!= g then error;. Transition diagrams  serve as abstractions for code that would be written  finite automata CS 1622 Lecture
8 Finite Automata A compiler recognizes legal programs in some (source) language. A finitestate machine recognizes legal strings in some language. Example: Identifiers sequences of one or more letters or digits, starting with a letter: letter digit S letter A CS 1622 Lecture 2 22 FiniteAutomata State Graphs A state The start state An accepting/final state A transition a CS 1622 Lecture 2 23 Finite Automata Transition s 1 a s 2 Is read In state s 1 on input a go to state s 2 If end of input or no transition possible If in accepting state => accept Otherwise => reject CS 1622 Lecture
9 Language defined by FSM The language defined by a FSM is the set of strings accepted by the FSM. in the language of the FSM on previous slide: x, tmp2, XyZzy, position27. not in the language of the FSM on previous slide: 123, a?, 13apples. CS 1622 Lecture 2 25 Example: Integer Literals FA that accepts integer literals with an optional + or  sign: digit digit S +  B A digit CS 1622 Lecture 2 26 Formal FSA Definition A finite automaton is a 5tuple (Σ, S, δ, s 0, S F ) where: An input alphabet Σ ν A set of states S ν A start state s 0 ν A set of accepting states S F S ν δ is the state transition function: S x Σ S (i.e., encodes transitions state input state) CS 1622 Lecture
10 FA for the integerliteral example Σ = {digit, +,  ) A set of states S = {S, A and B} A start state S 0 = S A set of accepting states S F S = {B} δ is the state transition function = (S,digit) > B (S, + ) > A (S,  ) > A (B, digit) > B (A, digit) > B CS 1622 Lecture 2 28 Two kinds of Automata Deterministic (DFA): No state has more than one outgoing edge with the same label. NonDeterministic (NFA): States may have more than one outgoing edge with same label. Edges may be labeled with ε (epsilon), the empty string. The automaton can take an ε epsilon transition without looking at the current input character. CS 1622 Lecture 2 29 Example of NFA integerliteral example: digit S ε +  B A digit CS 1622 Lecture
11 Nondeterministic automata (NFA) often simpler (e.g. smaller) than DFA can be in multiple states at the same time NFA accepts a string is if there exists a sequence of moves starting in the start state, ending in a final state, that consumes the entire string. Think about it as pursuing all choices in parallel or having an oracle that says what to do. Example: the integerliteral NFA on input "+75": CS 1622 Lecture 2 31 Equivalence of DFA and NFA Theorem: For every nondeterministic finitestate machine M, there exists a deterministic machine M' such that M and M' accept the same language. Why is the theorem important for scanner generation? Theorem is not enough: what do we need for automatic scanner generation? CS 1622 Lecture 2 32 How to Implement a FSM A tabledriven approach: table: one row for each state in the machine, and one column for each possible character. Table[j][k] which state to go to from state j on character k, an empty entry corresponds to the machine getting stuck. CS 1622 Lecture
12 The tabledriven program for a DFA state = S // S is the start state repeat { } k = next character from the input if k == EOF the // end of input if state is a final state then accept else reject state = T[state,k] if state = empty then reject // got stuck CS 1622 Lecture 2 34 Generating a scanner NFA Regular expressions DFA Lexical Specification Tabledriven Implementation of DFA CS 1622 Lecture 2 35 Regular Expressions FA s not good way to specify tokens  diagrams hard to write down regular expressions are another specification technique a compact way to define a language that can be accepted by an automaton. used as the input to a scanner generator define each token, and define whitespace, comments, etc these do not correspond to tokens, but must be recognized and ignored. CS 1622 Lecture
13 Example: Simple identifier English: A letter, followed by zero or more letters or digits. RE: letter. (letter digit)* Operators: means "or". means "followed by (usually just use position) * means zero or more instances () are used for grouping CS 1622 Lecture 2 37 Operands of a regular expression Operands are same as labels on the edges of an FSM single characters, or the special character ε (the empty string) "letter" is a shorthand for a b c... z A... Z "digit is a shorthand for sometimes we put the characters in quotes necessary when denoting characters:. * CS 1622 Lecture 2 38 Precedence of. * operators. Regular Expression Operator Analogous Arithmetic Operator Precedence plus lowest. times middle * exponentiation highest Consider regular expressions: letter.letter digit* letter.(letter digit)* CS 1622 Lecture
14 Examples Describe (in English) the language defined by each of the following regular expressions: letter (letter digit*) digit digit* "." digit digit* CS 1622 Lecture 2 40 Example: Integer Literals An integer literal with an optional sign can be defined in English as: (nothing or + or ) followed by one or more digits The corresponding regular expression is: (+  epsilon).(digit.digit*) A new convenient operator + digit.digit* is the same as digit+ which means "one or more digits CS 1622 Lecture 2 41 Language Defined by a Regular Expression Recall: language = set of strings Language defined by an automaton / RE Regular Exp. the set of strings accepted by the automaton the set of strings that match the expression. epsilon {""} a a.b.c a b c Corresponding Set of Strings {"a"} {"abc"} {"a", "b", "c"} (a b c)* {"", "a", "b", "c", "aa", "ab",..., "bccabb"...} CS 1622 Lecture
15 REs describe regular languages Patterns form a regular language *** any finite language is regular *** Regular Expression (RE) (over alphabet Σ) ε is a RE denoting the set {ε} If a is in Σ, then a is a RE denoting {a} If x and y are REs denoting L(x) and L(y) then x is an RE denoting L(x); y is a RE denoting L(y); x y is an RE denoting L(x) L(y) xy is an RE denoting L(x)L(y) x * is an RE denoting L(x)* Can combine RE to form other REs CS 1622 Lecture
Highlevel View of a Compiler
Highlevel View of a Compiler Source Compiler Implications Must recognize legal (and illegal) programs Must generate correct Must manage storage of all variables (and ) Must agree with OS & linker on format
More informationCompilers and Interpreters
Overview Roadmap Language Translators: Interpreters & Compilers Context of a compiler Phases of a compiler Compiler Construction tools Terminology How related to other CS Goals of a good compiler 1 Compilers
More informationLexical Analysis. Introduction
Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies
More informationIntroduction to Compiler
Formal Languages and Compiler (CSE322) Introduction to Compiler Jungsik Choi chjs@khu.ac.kr 2018. 3. 8 Traditional Twopass Compiler Source Front End Back End Compiler Target High level functions Recognize
More informationCompiling Techniques
Lecture 2: The view from 35000 feet 19 September 2017 Table of contents 1 2 Passes Representations 3 Instruction Selection Register Allocation Instruction Scheduling 4 of a compiler Source Compiler Machine
More informationCS415 Compilers. Lexical Analysis
CS415 Compilers Lexical Analysis These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Lecture 7 1 Announcements First project and second homework
More informationFront End. Hwansoo Han
Front nd Hwansoo Han Traditional Twopass Compiler Source code Front nd IR Back nd Machine code rrors High level functions Recognize legal program, generate correct code (OS & linker can accept) Manage
More informationAdministrativia. Extra credit for bugs in project assignments. Building a Scanner. CS164, Fall Recall: The Structure of a Compiler
dministrativia Building a canner Extra credit for bugs in project assignments in starter kits and handouts Ts are final arbiters of what s a bug only the first student to report the bug gets credit C64,
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs) Implementation
More informationThe Front End. The purpose of the front end is to deal with the input language. Perform a membership test: code source language?
The Front End Source code Front End IR Back End Machine code Errors The purpose of the front end is to deal with the input language Perform a membership test: code source language? Is the program wellformed
More informationRegular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications
Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs) Implementation
More informationAnnouncements! P1 part 1 due next Tuesday P1 part 2 due next Friday
Announcements! P1 part 1 due next Tuesday P1 part 2 due next Friday 1 Finitestate machines CS 536 Last time! A compiler is a recognizer of language S (Source) a translator from S to T (Target) a program
More informationIntroduction to Lexical Analysis
Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular
More informationLexical Analysis  An Introduction. Lecture 4 Spring 2005 Department of Computer Science University of Alabama Joel Jones
Lexical Analysis  An Introduction Lecture 4 Spring 2005 Department of Computer Science University of Alabama Joel Jones Copyright 2003, Keith D. Cooper, Ken Kennedy & Linda Torczon, all rights reserved.
More informationCSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B1
CSEP 501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2008 1/8/2008 200208 Hal Perkins & UW CSE B1 Agenda Basic concepts of formal grammars (review) Regular expressions
More informationCS 314 Principles of Programming Languages
CS 314 Principles of Programming Languages Lecture 2: Syntax Analysis Zheng (Eddy) Zhang Rutgers University January 22, 2018 Announcement First recitation starts this Wednesday Homework 1 will be release
More informationFormal Languages and Compilers Lecture VI: Lexical Analysis
Formal Languages and Compilers Lecture VI: Lexical Analysis Free University of BozenBolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/ artale/ Formal
More informationCompiler course. Chapter 3 Lexical Analysis
Compiler course Chapter 3 Lexical Analysis 1 A. A. Pourhaji Kazem, Spring 2009 Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata
More informationLexical Analysis. Chapter 2
Lexical Analysis Chapter 2 1 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples
More informationThe View from 35,000 Feet
The View from 35,000 Feet This lecture is taken directly from the Engineering a Compiler web site with only minor adaptations for EECS 6083 at University of Cincinnati Copyright 2003, Keith D. Cooper,
More informationCSE 413 Programming Languages & Implementation. Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions
CSE 413 Programming Languages & Implementation Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory
More informationPart 5 Program Analysis Principles and Techniques
1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape
More informationCS415 Compilers Overview of the Course. These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University
CS415 Compilers Overview of the Course These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Critical Facts Welcome to CS415 Compilers Topics in the
More informationLexical Analysis. Lecture 24
Lexical Analysis Lecture 24 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 2 1 Administrivia Moving to 60 Evans on Wednesday HW1 available Pyth manual available on line.
More informationLexical Analysis. Note by Baris Aktemur: Our slides are adapted from Cooper and Torczon s slides that they prepared for COMP 412 at Rice.
Lexical Analysis Note by Baris Aktemur: Our slides are adapted from Cooper and Torczon s slides that they prepared for COMP 412 at Rice. Copyright 2010, Keith D. Cooper & Linda Torczon, all rights reserved.
More informationChapter 3 Lexical Analysis
Chapter 3 Lexical Analysis Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator The role of lexical
More informationCS 314 Principles of Programming Languages. Lecture 3
CS 314 Principles of Programming Languages Lecture 3 Zheng Zhang Department of Computer Science Rutgers University Wednesday 14 th September, 2016 Zheng Zhang 1 CS@Rutgers University Class Information
More informationCSE 413 Programming Languages & Implementation. Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions
CSE 413 Programming Languages & Implementation Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory
More informationConcepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens
Concepts Introduced in Chapter 3 Lexical Analysis Regular Expressions (REs) Nondeterministic Finite Automata (NFA) Converting an RE to an NFA Deterministic Finite Automatic (DFA) Lexical Analysis Why separate
More informationLexical Analysis. Lecture 34
Lexical Analysis Lecture 34 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 34 1 Administrivia I suggest you start looking at Python (see link on class home page). Please
More informationLanguages and Compilers
Principles of Software Engineering and Operational Systems Languages and Compilers SDAGE: Level I 201213 3. Formal Languages, Grammars and Automata Dr Valery Adzhiev vadzhiev@bournemouth.ac.uk Office:
More informationCOMP421 Compiler Design. Presented by Dr Ioanna Dionysiou
COMP421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3  Lexical Analysis Sections 3.13.4, 3.63.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna
More informationLexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast!
Lexical Analysis Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Compiler Passes Analysis of input program (frontend) character stream
More informationCSE450 Translation of Programming Languages. Lecture 4: Syntax Analysis
CSE450 Translation of Programming Languages Lecture 4: Syntax Analysis http://xkcd.com/859 Structure of a Today! Compiler Source Language Lexical Analyzer Syntax Analyzer Semantic Analyzer Int. Code Generator
More informationCSCI312 Principles of Programming Languages!
CSCI312 Principles of Programming Languages!! Chapter 3 Regular Expression and Lexer Xu Liu Recap! Copyright 2006 The McGrawHill Companies, Inc. Clite: Lexical Syntax! Input: a stream of characters from
More informationOptimizing Finite Automata
Optimizing Finite Automata We can improve the DFA created by MakeDeterministic. Sometimes a DFA will have more states than necessary. For every DFA there is a unique smallest equivalent DFA (fewest states
More informationLecture 3: CUDA Programming & Compiler Front End
CS 515 Programming Language and Compilers I Lecture 3: CUDA Programming & Compiler Front End Zheng (Eddy) Zhang Rutgers University Fall 2017, 9/19/2017 1 Lecture 3A: GPU Programming 2 3 Review: Performance
More informationGoals for course What is a compiler and briefly how does it work? Review everything you should know from 330 and before
CMSC 430 Today Goals for course What is a compiler and briefly how does it work? Review everything you should know from 330 and before > This looks like a lot of material (and it is), but most of this
More informationChapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective
Chapter 4 Lexical analysis Lexical scanning Regular expressions DFAs and FSAs Lex Concepts CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley
More informationMidTerm Papers Solved MCQS with Reference (1 to 22 lectures)
CS606 Compiler Construction MidTerm Papers Solved MCQS with Reference (1 to 22 lectures) by Arslan Arshad (Zain) FEB 21,2016 03002462284 http://lmshelp.blogspot.com/ Arslan.arshad01@gmail.com AKMP01
More informationOverview of a Compiler
Highlevel View of a Compiler Overview of a Compiler Compiler Copyright 2010, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have
More informationLexical Analysis. Implementation: Finite Automata
Lexical Analysis Implementation: Finite Automata Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs)
More informationCS606 compiler instruction Solved MCQS From Midterm Papers
CS606 compiler instruction Solved MCQS From Midterm Papers March 06,2014 MC100401285 Moaaz.pk@gmail.com Mc100401285@gmail.com PSMD01 Final Term MCQ s and Quizzes CS606 compiler instruction If X is a
More informationImplementation of Lexical Analysis
Outline Implementation of Lexical nalysis Specifying lexical structure using regular expressions Finite automata Deterministic Finite utomata (DFs) Nondeterministic Finite utomata (NFs) Implementation
More informationImplementation of Lexical Analysis. Lecture 4
Implementation of Lexical Analysis Lecture 4 1 Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested It is easier to modify a working
More informationConcepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective
Concepts Lexical scanning Regular expressions DFAs and FSAs Lex CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 2 Lexical analysis
More informationLexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata
Lexical Analysis Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Phase Ordering of FrontEnds Lexical analysis (lexer) Break input string
More informationCompiler phases. Nontokens
Compiler phases Compiler Construction Scanning Lexical Analysis source code scanner tokens regular expressions lexical analysis Lennart Andersson parser context free grammar Revision 2011 01 21 parse tree
More informationCSE450. Translation of Programming Languages. Lecture 20: Automata and Regular Expressions
CSE45 Translation of Programming Languages Lecture 2: Automata and Regular Expressions Finite Automata Regular Expression = Specification Finite Automata = Implementation A finite automaton consists of:
More informationMIT Specifying Languages with Regular Expressions and ContextFree Grammars
MIT 6.035 Specifying Languages with Regular essions and ContextFree Grammars Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology Language Definition Problem How to precisely
More informationLexical Analysis. COMP 524, Spring 2014 Bryan Ward
Lexical Analysis COMP 524, Spring 2014 Bryan Ward Based in part on slides and notes by J. Erickson, S. Krishnan, B. Brandenburg, S. Olivier, A. Block and others The Big Picture Character Stream Scanner
More informationWeek 2: Syntax Specification, Grammars
CS320 Principles of Programming Languages Week 2: Syntax Specification, Grammars Jingke Li Portland State University Fall 2017 PSU CS320 Fall 17 Week 2: Syntax Specification, Grammars 1/ 62 Words and Sentences
More informationAbout the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. Compiler Design
i About the Tutorial A compiler translates the codes written in one language to some other language without changing the meaning of the program. It is also expected that a compiler should make the target
More informationCOLLEGE OF ENGINEERING, NASHIK. LANGUAGE TRANSLATOR
Pune Vidyarthi Griha s COLLEGE OF ENGINEERING, NASHIK. LANGUAGE TRANSLATOR By Prof. Anand N. Gharu (Assistant Professor) PVGCOE Computer Dept.. 22nd Jan 2018 CONTENTS : 1. Role of lexical analysis 2.
More informationZhizheng Zhang. Southeast University
Zhizheng Zhang Southeast University 2016/10/5 Lexical Analysis 1 1. The Role of Lexical Analyzer 2016/10/5 Lexical Analysis 2 2016/10/5 Lexical Analysis 3 Example. position = initial + rate * 60 2016/10/5
More informationStructure of Programming Languages Lecture 3
Structure of Programming Languages Lecture 3 CSCI 6636 4536 Spring 2017 CSCI 6636 4536 Lecture 3... 1/25 Spring 2017 1 / 25 Outline 1 Finite Languages Deterministic Finite State Machines Lexical Analysis
More informationCPS 506 Comparative Programming Languages. Syntax Specification
CPS 506 Comparative Programming Languages Syntax Specification Compiling Process Steps Program Lexical Analysis Convert characters into a stream of tokens Lexical Analysis Syntactic Analysis Send tokens
More informationCOMP 181 Compilers. Administrative. Last time. Prelude. Compilation strategy. Translation strategy. Lecture 2 Overview
COMP 181 Compilers Lecture 2 Overview September 7, 2006 Administrative Book? Hopefully: Compilers by Aho, Lam, Sethi, Ullman Mailing list Handouts? Programming assignments For next time, write a hello,
More informationIntroduction to Lexing and Parsing
Introduction to Lexing and Parsing ECE 351: Compilers Jon Eyolfson University of Waterloo June 18, 2012 1 Riddle Me This, Riddle Me That What is a compiler? 1 Riddle Me This, Riddle Me That What is a compiler?
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Lecture 4 (Modified by Professor Vijay Ganesh) Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested
More informationCompilers. Lecture 2 Overview. (original slides by Sam
Compilers Lecture 2 Overview Yannis Smaragdakis, U. Athens Yannis Smaragdakis, U. Athens (original slides by Sam Guyer@Tufts) Last time The compilation problem Source language Highlevel abstractions Easy
More information2. Lexical Analysis! Prof. O. Nierstrasz!
2. Lexical Analysis! Prof. O. Nierstrasz! Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes.! http://www.cs.ucla.edu/~palsberg/! http://www.cs.purdue.edu/homes/hosking/!
More informationLast lecture CMSC330. This lecture. Finite Automata: States. Finite Automata. Implementing Regular Expressions. Languages. Regular expressions
Last lecture CMSC330 Finite Automata Languages Sets of strings Operations on languages Regular expressions Constants Operators Precedence 1 2 Finite automata States Transitions Examples Types This lecture
More informationMIT Specifying Languages with Regular Expressions and ContextFree Grammars. Martin Rinard Massachusetts Institute of Technology
MIT 6.035 Specifying Languages with Regular essions and ContextFree Grammars Martin Rinard Massachusetts Institute of Technology Language Definition Problem How to precisely define language Layered structure
More informationCS 132 Compiler Construction
CS 132 Compiler Construction 1. Introduction 2 2. Lexical analysis 31 3. LL parsing 58 4. LR parsing 110 5. JavaCC and JTB 127 6. Semantic analysis 150 7. Translation and simplification 165 8. Liveness
More informationAdministrivia. Lexical Analysis. Lecture 24. Outline. The Structure of a Compiler. Informal sketch of lexical analysis. Issues in lexical analysis
dministrivia Lexical nalysis Lecture 24 Notes by G. Necula, with additions by P. Hilfinger Moving to 6 Evans on Wednesday HW available Pyth manual available on line. Please log into your account and electronically
More informationUNIT 2 LEXICAL ANALYSIS
OVER VIEW OF LEXICAL ANALYSIS UNIT 2 LEXICAL ANALYSIS o To identify the tokens we need some method of describing the possible tokens that can appear in the input stream. For this purpose we introduce
More informationCS321 Languages and Compiler Design I. Winter 2012 Lecture 4
CS321 Languages and Compiler Design I Winter 2012 Lecture 4 1 LEXICAL ANALYSIS Convert source file characters into token stream. Remove contentfree characters (comments, whitespace,...) Detect lexical
More informationCSE 3302 Programming Languages Lecture 2: Syntax
CSE 3302 Programming Languages Lecture 2: Syntax (based on slides by Chengkai Li) Leonidas Fegaras University of Texas at Arlington CSE 3302 L2 Spring 2011 1 How do we define a PL? Specifying a PL: Syntax:
More informationCOMPILER DESIGN UNIT I LEXICAL ANALYSIS. Translator: It is a program that translates one language to another Language.
UNIT I LEXICAL ANALYSIS Translator: It is a program that translates one language to another Language. Source Code Translator Target Code 1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System
More informationLexical and Syntax Analysis
Lexical and Syntax Analysis In Text: Chapter 4 N. Meng, F. Poursardar Lexical and Syntactic Analysis Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input
More informationCS 403 Compiler Construction Lecture 3 Lexical Analysis [Based on Chapter 1, 2, 3 of Aho2]
CS 403 Compiler Construction Lecture 3 Lexical Analysis [Based on Chapter 1, 2, 3 of Aho2] 1 What is Lexical Analysis? First step of a compiler. Reads/scans/identify the characters in the program and groups
More informationSyntactic Analysis. The Big Picture Again. Grammar. ICS312 MachineLevel and Systems Programming
The Big Picture Again Syntactic Analysis source code Scanner Parser Opt1 Opt2... Optn Instruction Selection Register Allocation Instruction Scheduling machine code ICS312 MachineLevel and Systems Programming
More informationNondeterministic Finite Automata (NFA): Nondeterministic Finite Automata (NFA) states of an automaton of this kind may or may not have a transition for each symbol in the alphabet, or can even have multiple
More informationG52LAC Languages and Computation Lecture 6
G52LAC Languages and Computation Lecture 6 Equivalence of Regular Expression and Finite Automata Henrik Nilsson University of Nottingham G52LACLanguages and ComputationLecture 6 p.1/28 This Lecture (1)
More informationCSc 453 Lexical Analysis (Scanning)
CSc 453 Lexical Analysis (Scanning) Saumya Debray The University of Arizona Tucson Overview source program lexical analyzer (scanner) tokens syntax analyzer (parser) symbol table manager Main task: to
More informationfor (i=1; i<=100000; i++) { x = sqrt (y); // square root function cout << x+i << endl; }
Ex: The difference between Compiler and Interpreter The interpreter actually carries out the computations specified in the source program. In other words, the output of a compiler is a program, whereas
More information1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System can be represented as shown figure below.
UNIT I Translator: It is a program that translates one language to another Language. Examples of translator are compiler, assembler, interpreter, linker, loader and preprocessor. Source Code Translator
More informationWhere We Are. CMSC 330: Organization of Programming Languages. This Lecture. Programming Languages. Motivation for Grammars
CMSC 330: Organization of Programming Languages Context Free Grammars Where We Are Programming languages Ruby OCaml Implementing programming languages Scanner Uses regular expressions Finite automata Parser
More informationCMSC 330: Organization of Programming Languages
CMSC 330: Organization of Programming Languages Context Free Grammars and Parsing 1 Recall: Architecture of Compilers, Interpreters Source Parser Static Analyzer Intermediate Representation Front End Back
More information10/4/18. Lexical and Syntactic Analysis. Lexical and Syntax Analysis. Tokenizing Source. Scanner. Reasons to Separate Lexical and Syntactic Analysis
Lexical and Syntactic Analysis Lexical and Syntax Analysis In Text: Chapter 4 Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input characters and output
More informationFigure 2.1: Role of Lexical Analyzer
Chapter 2 Lexical Analysis Lexical analysis or scanning is the process which reads the stream of characters making up the source program from lefttoright and groups them into tokens. The lexical analyzer
More informationLexical Analysis. Lecture 3. January 10, 2018
Lexical Analysis Lecture 3 January 10, 2018 Announcements PA1c due tonight at 11:50pm! Don t forget about PA1, the Cool implementation! Use Monday s lecture, the video guides and Cool examples if you re
More informationCSE302: Compiler Design
CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 01, 2007 Outline Recap
More informationLanguages, Automata, Regular Expressions & Scanners. Winter /8/ Hal Perkins & UW CSE B1
CSE 401 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2010 1/8/2010 200210 Hal Perkins & UW CSE B1 Agenda Quick review of basic concepts of formal grammars Regular
More informationFormal Languages and Compilers Lecture IV: Regular Languages and Finite. Finite Automata
Formal Languages and Compilers Lecture IV: Regular Languages and Finite Automata Free University of BozenBolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/
More informationQuestion Bank. 10CS63:Compiler Design
Question Bank 10CS63:Compiler Design 1.Determine whether the following regular expressions define the same language? (ab)* and a*b* 2.List the properties of an operator grammar 3. Is macro processing a
More informationDr. D.M. Akbar Hussain
1 2 Compiler Construction F6S Lecture  2 1 3 4 Compiler Construction F6S Lecture  2 2 5 #include.. #include main() { char in; in = getch ( ); if ( isalpha (in) ) in = getch ( ); else error (); while
More informationCSC 467 Lecture 3: Regular Expressions
CSC 467 Lecture 3: Regular Expressions Recall How we build a lexer by hand o Use fgetc/mmap to read input o Use a big switch to match patterns Homework exercise static TokenKind identifier( TokenKind token
More informationCMSC 330: Organization of Programming Languages. Architecture of Compilers, Interpreters
: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Scanner Parser Static Analyzer Intermediate Representation Front End Back End Compiler / Interpreter
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwthaachen.de/teaching/ss16/cc/ Conceptual Structure of a Compiler Source code x1 := y2
More informationCSCE 314 Programming Languages
CSCE 314 Programming Languages Syntactic Analysis Dr. Hyunyoung Lee 1 What Is a Programming Language? Language = syntax + semantics The syntax of a language is concerned with the form of a program: how
More informationprogramming languages need to be precise a regular expression is one of the following: tokens are the building blocks of programs
Chapter 2 :: Programming Language Syntax Programming Language Pragmatics Michael L. Scott Introduction programming languages need to be precise natural languages less so both form (syntax) and meaning
More informationCS 536 Introduction to Programming Languages and Compilers Charles N. Fischer Lecture 2
CS 536 Introduction to Programming Languages and Compilers Charles N. Fischer Lecture 2 CS 536 Spring 2015 1 Reading Assignment Read Chapter 3 of Crafting a Com piler. CS 536 Spring 2015 21 The Structure
More informationCMSC 350: COMPILER DESIGN
Lecture 11 CMSC 350: COMPILER DESIGN see HW3 LLVMLITE SPECIFICATION Eisenberg CMSC 350: Compilers 2 Discussion: Defining a Language Premise: programming languages are purely formal objects We (as language
More informationCMSC 330: Organization of Programming Languages. Context Free Grammars
CMSC 330: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Analyzer Optimizer Code Generator Abstract Syntax Tree Front End Back End Compiler
More informationCS5371 Theory of Computation. Lecture 8: Automata Theory VI (PDA, PDA = CFG)
CS5371 Theory of Computation Lecture 8: Automata Theory VI (PDA, PDA = CFG) Objectives Introduce Pushdown Automaton (PDA) Show that PDA = CFG In terms of descriptive power Pushdown Automaton (PDA) Roughly
More information10/5/17. Lexical and Syntactic Analysis. Lexical and Syntax Analysis. Tokenizing Source. Scanner. Reasons to Separate Lexical and Syntax Analysis
Lexical and Syntactic Analysis Lexical and Syntax Analysis In Text: Chapter 4 Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input characters and output
More informationCS415 Compilers. Syntax Analysis. These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University
CS415 Compilers Syntax Analysis These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Limits of Regular Languages Advantages of Regular Expressions
More information