# Introduction to Lexical Analysis

Size: px
Start display at page:

Transcription

1 Introduction to Lexical Analysis

2 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples of regular expressions Compiler Design 1 (2011) 2

3 Lexical Analysis What do we want to do? Example: if (i == j) then Z = 0; else Z = 1; The input is just a string of characters: \tif (i == j)\nthen\n\tz = 0;\n\telse\n\t\tz = 1; Goal: Partition input string into substrings Where the substrings are tokens Compiler Design 1 (2011) 3

4 What s a Token? A syntactic category In English: noun, verb, adjective, In a programming language: Identifier, Integer, Keyword, Whitespace, Compiler Design 1 (2011) 4

5 Tokens Tokens correspond to sets of strings. Identifier: strings of letters or digits, starting with a letter Integer: a non-empty string of digits Keyword: else or if or begin or Whitespace: a non-empty sequence of blanks, newlines, and tabs Compiler Design 1 (2011) 5

6 What are Tokens used for? Classify program substrings according to role Output of lexical analysis is a stream of tokens which is input to the parser Parser relies on token distinctions An identifier is treated differently than a keyword Compiler Design 1 (2011) 6

7 Designing a Lexical Analyzer: Step 1 Define a finite set of tokens Tokens describe all items of interest Choice of tokens depends on language, design of parser Recall \tif (i == j)\nthen\n\tz = 0;\n\telse\n\t\tz = 1; Useful tokens for this expression: Integer, Keyword, Relation, Identifier, Whitespace, (, ), =, ; Compiler Design 1 (2011) 7

8 Designing a Lexical Analyzer: Step 2 Describe which strings belong to each token Recall: Identifier: strings of letters or digits, starting with a letter Integer: a non-empty string of digits Keyword: else or if or begin or Whitespace: a non-empty sequence of blanks, newlines, and tabs Compiler Design 1 (2011) 8

9 Lexical Analyzer: Implementation An implementation must do two things: 1. Recognize substrings corresponding to tokens 2. Return the value or lexeme of the token The lexeme is the substring Compiler Design 1 (2011) 9

10 Example Recall: \tif (i == j)\nthen\n\tz = 0;\n\telse\n\t\tz = 1; Token-lexeme groupings: Identifier: i, j, z Keyword: if, then, else Relation: == Integer: 0, 1 (, ), =, ; single character of the same name Compiler Design 1 (2011) 10

11 Why do Lexical Analysis? Dramatically simplify parsing The lexer usually discards uninteresting tokens that don t contribute to parsing E.g. Whitespace, Comments Converts data early Separate out logic to read source files Potentially an issue on multiple platforms Can optimize reading code independently of parser Compiler Design 1 (2011) 11

12 True Crimes of Lexical Analysis Is it as easy as it sounds? Not quite! Look at some programming language history... Compiler Design 1 (2011) 12

13 Lexical Analysis in FORTRAN FORTRAN rule: Whitespace is insignificant E.g., VAR1 is the same as VA R1 Footnote: FORTRAN whitespace rule was motivated by inaccuracy of punch card operators Compiler Design 1 (2011) 13

14 A terrible design! Example Consider DO 5 I = 1,25 DO 5 I = 1.25 The first is DO 5 I = 1, 25 The second is DO5I = 1.25 Reading left-to-right, cannot tell if DO5I is a variable or DO stmt. until after, is reached Compiler Design 1 (2011) 14

15 Lexical Analysis in FORTRAN. Lookahead. Two important points: 1. The goal is to partition the string. This is implemented by reading left-to-write, recognizing one token at a time 2. Lookahead may be required to decide where one token ends and the next token begins Even our simple example has lookahead issues i vs. if = vs. == Compiler Design 1 (2011) 15

16 Another Great Moment in Scanning PL/1: Keywords can be used as identifiers: IF THEN THEN THEN = ELSE; ELSE ELSE = IF can be difficult to determine how to label lexemes Compiler Design 1 (2011) 16

17 More Modern True Crimes in Scanning Nested template declarations in C++ vector<vector<int>> myvector vector < vector < int >> myvector (vector < (vector < (int >> myvector))) Compiler Design 1 (2011) 17

18 Review The goal of lexical analysis is to Partition the input string into lexemes (the smallest program units that are individually meaningful) Identify the token of each lexeme Left-to-right scan lookahead sometimes required Compiler Design 1 (2011) 18

19 Next We still need A way to describe the lexemes of each token A way to resolve ambiguities Is if two variables i and f? Is == two equal signs = =? Compiler Design 1 (2011) 19

20 Regular Languages There are several formalisms for specifying tokens Regular languages are the most popular Simple and useful theory Easy to understand Efficient implementations Compiler Design 1 (2011) 20

21 Languages Def. Let Σ be a set of characters. A language Λ over Σ is a set of strings of characters drawn from Σ (Σ is called the alphabet of Λ) Compiler Design 1 (2011) 21

22 Examples of Languages Alphabet = English characters Language = English sentences Alphabet = ASCII Language = C programs Not every string on English characters is an English sentence Note: ASCII character set is different from English character set Compiler Design 1 (2011) 22

23 Notation Languages are sets of strings Need some notation for specifying which sets of strings we want our language to contain The standard notation for regular languages is regular expressions Compiler Design 1 (2011) 23

24 Atomic Regular Expressions Single character Epsilon ' c' = {" c" } ε = {""} Compiler Design 1 (2011) 24

25 Compound Regular Expressions Union Concatenation Iteration * { or } A+ B= s s A s B { and } AB= ab a A b B = U i i = i 0 A A where A A... i times... A Compiler Design 1 (2011) 25

26 Regular Expressions Def. The regular expressions over Σ are the smallest set of expressions including ε ' c' where c A+ B where A, B are rexp over AB " " " * A where A is a rexp over Compiler Design 1 (2011) 26

27 Syntax vs. Semantics To be careful, we should distinguish syntax and semantics (meaning) of regular expressions L( ε) = {""} L(' c') = {" c"} LA ( + B) = LA ( ) LB ( ) L( AB) = { ab a L( A) and b L( B)} LA = U LA * ( ) ( i ) i 0 Compiler Design 1 (2011) 27

28 Example: Keyword Keyword: else or if or begin or ' else' + 'if' + 'begin' + L Note: 'else' abbreviates 'e''l''s''e' Compiler Design 1 (2011) 28

29 Example: Integers Integer: a non-empty string of digits digit = '0' + '1' + '2' + '3' + '4' + '5' + '6' + '7' + '8' + '9' integer = digit digit * Abbreviation: A + = AA * Compiler Design 1 (2011) 29

30 Example: Identifier Identifier: strings of letters or digits, starting with a letter letter = 'A' + K+ 'Z' + 'a' + K+ 'z' identifier = letter (letter + digit) * * * Is (letter + digit ) the same? Compiler Design 1 (2011) 30

31 Example: Whitespace Whitespace: a non-empty sequence of blanks, newlines, and tabs (' ' + '\n' + '\t') + Compiler Design 1 (2011) 31

32 Example 1: Phone Numbers Regular expressions are all around you! Consider +46(0) Σ = digits {+,,(,)} country = digit digit city = digit digit univ = digit digit digit extension = digit digit digit digit phone_num = + country ( 0 ) city univ extension Compiler Design 1 (2011) 32

33 Example 2: Addresses Consider = letters name = + letter address = name name '.' name '.' name Compiler Design 1 (2011) 33

34 Summary Regular expressions describe many useful languages Regular languages are a language specification We still need an implementation Next time: Given a string s and a regular expression R, is s L( R)? Compiler Design 1 (2011) 34

### Introduction to Lexical Analysis

Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular

### Syntactic Analysis. CS345H: Programming Languages. Lecture 3: Lexical Analysis. Outline. Lexical Analysis. What is a Token? Tokens

Syntactic Analysis CS45H: Programming Languages Lecture : Lexical Analysis Thomas Dillig Main Question: How to give structure to strings Analogy: Understanding an English sentence First, we separate a

### Lexical Analysis. Chapter 2

Lexical Analysis Chapter 2 1 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples

### Lexical Analysis. Lecture 2-4

Lexical Analysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 2 1 Administrivia Moving to 60 Evans on Wednesday HW1 available Pyth manual available on line.

### Lexical Analysis. Lecture 3. January 10, 2018

Lexical Analysis Lecture 3 January 10, 2018 Announcements PA1c due tonight at 11:50pm! Don t forget about PA1, the Cool implementation! Use Monday s lecture, the video guides and Cool examples if you re

### Lexical Analysis. Lecture 3-4

Lexical Analysis Lecture 3-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 3-4 1 Administrivia I suggest you start looking at Python (see link on class home page). Please

### Administrivia. Lexical Analysis. Lecture 2-4. Outline. The Structure of a Compiler. Informal sketch of lexical analysis. Issues in lexical analysis

dministrivia Lexical nalysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Moving to 6 Evans on Wednesday HW available Pyth manual available on line. Please log into your account and electronically

### Lexical Analysis. Finite Automata

#1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical

### Lexical Analysis. Finite Automata

#1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical

### Cunning Plan. Informal Sketch of Lexical Analysis. Issues in Lexical Analysis. Specifying Lexers

Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical Analysis Lookahead Ambiguity Specifying Lexers Regular Expressions

### Lexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata

Lexical Analysis Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Phase Ordering of Front-Ends Lexical analysis (lexer) Break input string

### Implementation of Lexical Analysis

Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation

### Implementation of Lexical Analysis

Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation

### CSE 3302 Programming Languages Lecture 2: Syntax

CSE 3302 Programming Languages Lecture 2: Syntax (based on slides by Chengkai Li) Leonidas Fegaras University of Texas at Arlington CSE 3302 L2 Spring 2011 1 How do we define a PL? Specifying a PL: Syntax:

### Formal Languages and Compilers Lecture VI: Lexical Analysis

Formal Languages and Compilers Lecture VI: Lexical Analysis Free University of Bozen-Bolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/ artale/ Formal

### Defining Program Syntax. Chapter Two Modern Programming Languages, 2nd ed. 1

Defining Program Syntax Chapter Two Modern Programming Languages, 2nd ed. 1 Syntax And Semantics Programming language syntax: how programs look, their form and structure Syntax is defined using a kind

### Implementation of Lexical Analysis

Outline Implementation of Lexical nalysis Specifying lexical structure using regular expressions Finite automata Deterministic Finite utomata (DFs) Non-deterministic Finite utomata (NFs) Implementation

### Lexical Analysis. Introduction

Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies

### Part 5 Program Analysis Principles and Techniques

1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape

### 1. Lexical Analysis Phase

1. Lexical Analysis Phase The purpose of the lexical analyzer is to read the source program, one character at time, and to translate it into a sequence of primitive units called tokens. Keywords, identifiers,

### A simple syntax-directed

Syntax-directed is a grammaroriented compiling technique Programming languages: Syntax: what its programs look like? Semantic: what its programs mean? 1 A simple syntax-directed Lexical Syntax Character

### Week 2: Syntax Specification, Grammars

CS320 Principles of Programming Languages Week 2: Syntax Specification, Grammars Jingke Li Portland State University Fall 2017 PSU CS320 Fall 17 Week 2: Syntax Specification, Grammars 1/ 62 Words and Sentences

### CS412/413. Introduction to Compilers Tim Teitelbaum. Lecture 2: Lexical Analysis 23 Jan 08

CS412/413 Introduction to Compilers Tim Teitelbaum Lecture 2: Lexical Analysis 23 Jan 08 Outline Review compiler structure What is lexical analysis? Writing a lexer Specifying tokens: regular expressions

### Specifying Syntax. An English Grammar. Components of a Grammar. Language Specification. Types of Grammars. 1. Terminal symbols or terminals, Σ

Specifying Syntax Language Specification Components of a Grammar 1. Terminal symbols or terminals, Σ Syntax Form of phrases Physical arrangement of symbols 2. Nonterminal symbols or syntactic categories,

### The Front End. The purpose of the front end is to deal with the input language. Perform a membership test: code source language?

The Front End Source code Front End IR Back End Machine code Errors The purpose of the front end is to deal with the input language Perform a membership test: code source language? Is the program well-formed

### Compiler course. Chapter 3 Lexical Analysis

Compiler course Chapter 3 Lexical Analysis 1 A. A. Pourhaji Kazem, Spring 2009 Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata

### Administrativia. Extra credit for bugs in project assignments. Building a Scanner. CS164, Fall Recall: The Structure of a Compiler

dministrativia Building a canner Extra credit for bugs in project assignments in starter kits and handouts Ts are final arbiters of what s a bug only the first student to report the bug gets credit C64,

### CSC 467 Lecture 3: Regular Expressions

CSC 467 Lecture 3: Regular Expressions Recall How we build a lexer by hand o Use fgetc/mmap to read input o Use a big switch to match patterns Homework exercise static TokenKind identifier( TokenKind token

### Implementation of Lexical Analysis. Lecture 4

Implementation of Lexical Analysis Lecture 4 1 Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested It is easier to modify a working

### Chapter 3 Lexical Analysis

Chapter 3 Lexical Analysis Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator The role of lexical

### Implementation of Lexical Analysis

Implementation of Lexical Analysis Lecture 4 (Modified by Professor Vijay Ganesh) Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested

i About the Tutorial A compiler translates the codes written in one language to some other language without changing the meaning of the program. It is also expected that a compiler should make the target

### Regular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications

Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata

### Compiler Construction

Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-16/cc/ Conceptual Structure of a Compiler Source code x1 := y2

### UNIT -2 LEXICAL ANALYSIS

OVER VIEW OF LEXICAL ANALYSIS UNIT -2 LEXICAL ANALYSIS o To identify the tokens we need some method of describing the possible tokens that can appear in the input stream. For this purpose we introduce

### Lexical Analysis. Implementation: Finite Automata

Lexical Analysis Implementation: Finite Automata Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs)

### Topic 3: Syntax Analysis I

Topic 3: Syntax Analysis I Compiler Design Prof. Hanjun Kim CoreLab (Compiler Research Lab) POSTECH 1 Back-End Front-End The Front End Source Program Lexical Analysis Syntax Analysis Semantic Analysis

### Compiler Construction D7011E

Compiler Construction D7011E Lecture 2: Lexical analysis Viktor Leijon Slides largely by Johan Nordlander with material generously provided by Mark P. Jones. 1 Basics of Lexical Analysis: 2 Some definitions:

### CSCE 314 Programming Languages

CSCE 314 Programming Languages Syntactic Analysis Dr. Hyunyoung Lee 1 What Is a Programming Language? Language = syntax + semantics The syntax of a language is concerned with the form of a program: how

### A Pascal program. Input from the file is read to a buffer program buffer. program xyz(input, output) --- begin A := B + C * 2 end.

A Pascal program program xyz(input, output); var A, B, C: integer; begin A := B + C * 2 end. Input from the file is read to a buffer program buffer program xyz(input, output) --- begin A := B + C * 2 end.

### COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou

COMP-421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3 - Lexical Analysis Sections 3.1-3.4, 3.6-3.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna

### A Simple Syntax-Directed Translator

Chapter 2 A Simple Syntax-Directed Translator 1-1 Introduction The analysis phase of a compiler breaks up a source program into constituent pieces and produces an internal representation for it, called

### for (i=1; i<=100000; i++) { x = sqrt (y); // square root function cout << x+i << endl; }

Ex: The difference between Compiler and Interpreter The interpreter actually carries out the computations specified in the source program. In other words, the output of a compiler is a program, whereas

### David Griol Barres Computer Science Department Carlos III University of Madrid Leganés (Spain)

David Griol Barres dgriol@inf.uc3m.es Computer Science Department Carlos III University of Madrid Leganés (Spain) OUTLINE Introduction: Definitions The role of the Lexical Analyzer Scanner Implementation

### Lexical analysis. Syntactical analysis. Semantical analysis. Intermediate code generation. Optimization. Code generation. Target specific optimization

Second round: the scanner Lexical analysis Syntactical analysis Semantical analysis Intermediate code generation Optimization Code generation Target specific optimization Lexical analysis (Chapter 3) Why

### CPS 506 Comparative Programming Languages. Syntax Specification

CPS 506 Comparative Programming Languages Syntax Specification Compiling Process Steps Program Lexical Analysis Convert characters into a stream of tokens Lexical Analysis Syntactic Analysis Send tokens

### Lexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast!

Lexical Analysis Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Compiler Passes Analysis of input program (front-end) character stream

### for (i=1; i<=100000; i++) { x = sqrt (y); // square root function cout << x+i << endl; }

Ex: The difference between Compiler and Interpreter The interpreter actually carries out the computations specified in the source program. In other words, the output of a compiler is a program, whereas

### CSE302: Compiler Design

CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 20, 2007 Outline Recap

### CSE302: Compiler Design

CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 01, 2007 Outline Recap

### CS321 Languages and Compiler Design I. Winter 2012 Lecture 4

CS321 Languages and Compiler Design I Winter 2012 Lecture 4 1 LEXICAL ANALYSIS Convert source file characters into token stream. Remove content-free characters (comments, whitespace,...) Detect lexical

### Formal Languages and Grammars. Chapter 2: Sections 2.1 and 2.2

Formal Languages and Grammars Chapter 2: Sections 2.1 and 2.2 Formal Languages Basis for the design and implementation of programming languages Alphabet: finite set Σ of symbols String: finite sequence

### Lexical Analysis. COMP 524, Spring 2014 Bryan Ward

Lexical Analysis COMP 524, Spring 2014 Bryan Ward Based in part on slides and notes by J. Erickson, S. Krishnan, B. Brandenburg, S. Olivier, A. Block and others The Big Picture Character Stream Scanner

### COMPILER DESIGN LECTURE NOTES

COMPILER DESIGN LECTURE NOTES UNIT -1 1.1 OVERVIEW OF LANGUAGE PROCESSING SYSTEM 1.2 Preprocessor A preprocessor produce input to compilers. They may perform the following functions. 1. Macro processing:

### Non-deterministic Finite Automata (NFA)

Non-deterministic Finite Automata (NFA) CAN have transitions on the same input to different states Can include a ε or λ transition (i.e. move to new state without reading input) Often easier to design

### CMPT 755 Compilers. Anoop Sarkar.

CMPT 755 Compilers Anoop Sarkar http://www.cs.sfu.ca/~anoop Parsing source program Lexical Analyzer token next() Parser parse tree Later Stages Lexical Errors Syntax Errors Context-free Grammars Set of

### Lexical Analysis (ASU Ch 3, Fig 3.1)

Lexical Analysis (ASU Ch 3, Fig 3.1) Implementation by hand automatically ((F)Lex) Lex generates a finite automaton recogniser uses regular expressions Tasks remove white space (ws) display source program

### Concepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens

Concepts Introduced in Chapter 3 Lexical Analysis Regular Expressions (REs) Nondeterministic Finite Automata (NFA) Converting an RE to an NFA Deterministic Finite Automatic (DFA) Lexical Analysis Why separate

### Interpreter. Scanner. Parser. Tree Walker. read. request token. send token. send AST I/O. Console

Scanning 1 read Interpreter Scanner request token Parser send token Console I/O send AST Tree Walker 2 Scanner This process is known as: Scanning, lexing (lexical analysis), and tokenizing This is the

### Syntax. In Text: Chapter 3

Syntax In Text: Chapter 3 1 Outline Syntax: Recognizer vs. generator BNF EBNF Chapter 3: Syntax and Semantics 2 Basic Definitions Syntax the form or structure of the expressions, statements, and program

### Compiler Construction

Compiler Construction Lecture 2: Lexical Analysis I (Introduction) Thomas Noll Lehrstuhl für Informatik 2 (Software Modeling and Verification) noll@cs.rwth-aachen.de http://moves.rwth-aachen.de/teaching/ss-14/cc14/

### CSc 453 Compilers and Systems Software

CSc 453 Compilers and Systems Software 3 : Lexical Analysis I Christian Collberg Department of Computer Science University of Arizona collberg@gmail.com Copyright c 2009 Christian Collberg August 23, 2009

### Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres

Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres dgriol@inf.uc3m.es Introduction: Definitions Lexical analysis or scanning: To read from left-to-right a source

### LANGUAGE PROCESSORS. Presented By: Prof. S.J. Soni, SPCE Visnagar.

LANGUAGE PROCESSORS Presented By: Prof. S.J. Soni, SPCE Visnagar. Introduction Language Processing activities arise due to the differences between the manner in which a software designer describes the

### Compiler Construction

Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-16/cc/ Recap: First-Longest-Match Analysis Outline of Lecture

### Chapter 3: CONTEXT-FREE GRAMMARS AND PARSING Part 1

Chapter 3: CONTEXT-FREE GRAMMARS AND PARSING Part 1 1. Introduction Parsing is the task of Syntax Analysis Determining the syntax, or structure, of a program. The syntax is defined by the grammar rules

### Today. Assignments. Lecture Notes CPSC 326 (Spring 2019) Quiz 2. Lexer design. Syntax Analysis: Context-Free Grammars. HW2 (out, due Tues)

Today Quiz 2 Lexer design Syntax Analysis: Context-Free Grammars Assignments HW2 (out, due Tues) S. Bowers 1 of 15 Implementing a Lexer for MyPL (HW 2) Similar in spirit to HW 1 We ll create three classes:

### Syntactic Analysis. The Big Picture Again. Grammar. ICS312 Machine-Level and Systems Programming

The Big Picture Again Syntactic Analysis source code Scanner Parser Opt1 Opt2... Optn Instruction Selection Register Allocation Instruction Scheduling machine code ICS312 Machine-Level and Systems Programming

### Principles of Programming Languages COMP251: Syntax and Grammars

Principles of Programming Languages COMP251: Syntax and Grammars Prof. Dekai Wu Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong, China Fall 2006

### Where We Are. CMSC 330: Organization of Programming Languages. This Lecture. Programming Languages. Motivation for Grammars

CMSC 330: Organization of Programming Languages Context Free Grammars Where We Are Programming languages Ruby OCaml Implementing programming languages Scanner Uses regular expressions Finite automata Parser

### Programming Languages and Compilers (CS 421)

Programming Languages and Compilers (CS 421) Elsa L Gunter 2112 SC, UIUC http://courses.engr.illinois.edu/cs421 Based in part on slides by Mattox Beckman, as updated by Vikram Adve and Gul Agha 10/30/17

### UNIT II LEXICAL ANALYSIS

UNIT II LEXICAL ANALYSIS 2 Marks 1. What are the issues in lexical analysis? Simpler design Compiler efficiency is improved Compiler portability is enhanced. 2. Define patterns/lexeme/tokens? This set

### Programming Languages & Compilers. Programming Languages and Compilers (CS 421) I. Major Phases of a Compiler. Programming Languages & Compilers

Programming Languages & Compilers Programming Languages and Compilers (CS 421) I Three Main Topics of the Course II III Elsa L Gunter 2112 SC, UIUC http://courses.engr.illinois.edu/cs421 New Programming

### CSE 413 Programming Languages & Implementation. Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions

CSE 413 Programming Languages & Implementation Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory

### Lexical Analyzer Scanner

Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce

### Principles of Programming Languages COMP251: Syntax and Grammars

Principles of Programming Languages COMP251: Syntax and Grammars Prof. Dekai Wu Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong, China Fall 2007

### CSCI312 Principles of Programming Languages!

CSCI312 Principles of Programming Languages!! Chapter 3 Regular Expression and Lexer Xu Liu Recap! Copyright 2006 The McGraw-Hill Companies, Inc. Clite: Lexical Syntax! Input: a stream of characters from

### Lexical Analyzer Scanner

Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce

### Chapter 3: Lexical Analysis

Chapter 3: Lexical Analysis A simple way to build a lexical analyzer is to construct a diagram that illustrates the structure of tokens of the source language, and then to hand translate the diagram into

### CMSC 330: Organization of Programming Languages. Architecture of Compilers, Interpreters

: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Scanner Parser Static Analyzer Intermediate Representation Front End Back End Compiler / Interpreter

### CSE 413 Programming Languages & Implementation. Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions

CSE 413 Programming Languages & Implementation Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory

### Chapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective

Chapter 4 Lexical analysis Lexical scanning Regular expressions DFAs and FSAs Lex Concepts CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley

### Languages and Compilers

Principles of Software Engineering and Operational Systems Languages and Compilers SDAGE: Level I 2012-13 4. Lexical Analysis (Scanning) Dr Valery Adzhiev vadzhiev@bournemouth.ac.uk Office: TA-121 For

### Implementation of Lexical Analysis

Written ssignments W assigned today Implementation of Lexical nalysis Lecture 4 Due in one week :59pm Electronic hand-in Prof. iken CS 43 Lecture 4 Prof. iken CS 43 Lecture 4 2 Tips on uilding Large Systems

### CMSC 330: Organization of Programming Languages

CMSC 330: Organization of Programming Languages Context Free Grammars and Parsing 1 Recall: Architecture of Compilers, Interpreters Source Parser Static Analyzer Intermediate Representation Front End Back

### Scanners. Xiaokang Qiu Purdue University. August 24, ECE 468 Adapted from Kulkarni 2012

Scanners Xiaokang Qiu Purdue University ECE 468 Adapted from Kulkarni 2012 August 24, 2016 Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved

### Compiler Construction

Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-17/cc/ Recap: First-Longest-Match Analysis The Extended Matching

### Related Course Objec6ves

Syntax 9/18/17 1 Related Course Objec6ves Develop grammars and parsers of programming languages 9/18/17 2 Syntax And Seman6cs Programming language syntax: how programs look, their form and structure Syntax

### Course Overview. Introduction (Chapter 1) Compiler Frontend: Today. Compiler Backend:

Course Overview Introduction (Chapter 1) Compiler Frontend: Today Lexical Analysis & Parsing (Chapter 2,3,4) Semantic Analysis (Chapter 5) Activation Records (Chapter 6) Translation to Intermediate Code

### CSE450 Translation of Programming Languages. Lecture 4: Syntax Analysis

CSE450 Translation of Programming Languages Lecture 4: Syntax Analysis http://xkcd.com/859 Structure of a Today! Compiler Source Language Lexical Analyzer Syntax Analyzer Semantic Analyzer Int. Code Generator

### Syntax. A. Bellaachia Page: 1

Syntax 1. Objectives & Definitions... 2 2. Definitions... 3 3. Lexical Rules... 4 4. BNF: Formal Syntactic rules... 6 5. Syntax Diagrams... 9 6. EBNF: Extended BNF... 10 7. Example:... 11 8. BNF Statement

### COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou

COMP-421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! Any questions about the syllabus?! Course Material available at www.cs.unic.ac.cy/ioanna! Next time reading assignment [ALSU07]

### CS 314 Principles of Programming Languages. Lecture 3

CS 314 Principles of Programming Languages Lecture 3 Zheng Zhang Department of Computer Science Rutgers University Wednesday 14 th September, 2016 Zheng Zhang 1 CS@Rutgers University Class Information

### The Structure of a Syntax-Directed Compiler

Source Program (Character Stream) Scanner Tokens Parser Abstract Syntax Tree Type Checker (AST) Decorated AST Translator Intermediate Representation Symbol Tables Optimizer (IR) IR Code Generator Target

### PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer

PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer As the first phase of a compiler, the main task of the lexical analyzer is to read the input

### Outline CS4120/4121. Compilation in a Nutshell 1. Administration. Introduction to Compilers Andrew Myers. HW1 out later today due next Monday.

CS4120/4121 Introduction to Compilers Andrew Myers Lecture 2: Lexical Analysis 31 August 2009 Outline Administration Compilation in a nutshell (or two) What is lexical analysis? Writing a lexer Specifying

### Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective

Concepts Lexical scanning Regular expressions DFAs and FSAs Lex CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 2 Lexical analysis