# Introduction to Lexical Analysis

Size: px
Start display at page:

Transcription

1 Introduction to Lexical Analysis

2 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular expressions Examples of regular expressions 2

3 Lexical Analysis What do we want to do? Example: if (i == j) then z = 0; else z = 1; The input is just a string of characters: if (i == j)\nthen\n\tz = 0;\n\telse\n\t\tz = 1; Goal: Partition input string into substrings where the substrings are tokens and classify them according to their role 3

4 What s a Token? A syntactic category In English: noun, verb, adjective, In a programming language: Identifier, Integer, Keyword, Whitespace, 4

5 Tokens Tokens correspond to sets of strings these sets depend on the programming language Identifier: strings of letters or digits, starting with a letter Integer: a non-empty string of digits Keyword: else or if or begin or Whitespace: a non-empty sequence of blanks, newlines, and tabs 5

6 What are Tokens Used for? Classify program substrings according to role Output of lexical analysis is a stream of tokens which is input to the parser Parser relies on token distinctions An identifier is treated differently than a keyword 6

7 Designing a Lexical Analyzer: Step 1 Define a finite set of tokens Tokens describe all items of interest Choice of tokens depends on language, design of parser Recall if (i == j)\nthen\n\tz = 0;\n\telse\n\t\tz = 1; Useful tokens for this expression: Integer, Keyword, Relation, Identifier, Whitespace, (, ), =, ; 7

8 Designing a Lexical Analyzer: Step 2 Describe which strings belong to each token Recall: Identifier: strings of letters or digits, starting with a letter Integer: a non-empty string of digits Keyword: else or if or begin or Whitespace: a non-empty sequence of blanks, newlines, and tabs 8

9 Lexical Analyzer: Implementation An implementation must do two things: 1. Recognize substrings corresponding to tokens 2. Return the value or lexeme of the token The lexeme is the substring 9

10 Example Recall: if (i == j)\nthen\n\tz = 0;\n\telse\n\t\tz = 1; Token-lexeme groupings: Identifier: i, j, z Keyword: if, then, else Relation: == Integer: 0, 1 (, ), =, ; single character of the same name 10

11 Why do Lexical Analysis? Dramatically simplify parsing The lexer usually discards uninteresting tokens that don t contribute to parsing E.g. Whitespace, Comments Converts data early Separate out logic to read source files Potentially an issue on multiple platforms Can optimize reading code independently of parser 11

12 True Crimes of Lexical Analysis Is it as easy as it sounds? Not quite! Look at some programming language history... 12

13 Lexical Analysis in FORTRAN FORTRAN rule: Whitespace is insignificant E.g., VAR1 is the same as VA R1 FORTRAN whitespace rule was motivated by inaccuracy of punch card operators 13

14 A terrible design! Example Consider DO 5 I = 1,25 DO 5 I = 1.25 The first is DO 5 I = 1, 25 The second is DO5I = 1.25 Reading left-to-right, the lexical analyzer cannot tell if DO5I is a variable or a DO statement until after, is reached 14

15 Lexical Analysis in FORTRAN. Lookahead. Two important points: 1. The goal is to partition the string This is implemented by reading left-to-right, recognizing one token at a time 2. Lookahead may be required to decide where one token ends and the next token begins Even our simple example has lookahead issues i vs. if = vs. == 15

16 Another Great Moment in Scanning History PL/1: Keywords can be used as identifiers: IF THEN THEN THEN = ELSE; ELSE ELSE = IF can be difficult to determine how to label lexemes 16

17 More Modern True Crimes in Scanning Nested template declarations in C++ vector<vector<int>> myvector vector < vector < int >> myvector (vector < (vector < (int >> myvector))) 17

18 Review The goal of lexical analysis is to Partition the input string into lexemes (the smallest program units that are individually meaningful) Identify the token of each lexeme Left-to-right scan lookahead sometimes required 18

19 Next We still need A way to describe the lexemes of each token A way to resolve ambiguities Is if two variables i and f? Is == two equal signs = =? 19

20 Regular Languages There are several formalisms for specifying tokens Regular languages are the most popular Simple and useful theory Easy to understand Efficient implementations 20

21 Languages Def. Let Σ be a set of characters. A language Λ over Σ is a set of strings of characters drawn from Σ (Σ is called the alphabet of Λ) 21

22 Examples of Languages Alphabet = English characters Language = English sentences Alphabet = ASCII Language = C programs Not every string on English characters is an English sentence Note: ASCII character set is different from English character set 22

23 Notation Languages are sets of strings Need some notation for specifying which sets of strings we want our language to contain The standard notation for regular languages is regular expressions 23

24 Atomic Regular Expressions Single character Epsilon ' c' = {" c" } ε = {""} 24

25 Compound Regular Expressions Union Concatenation Iteration * { or } A+ B= s s A s B { and } AB= ab a A b B = U i i = i 0 A A where A A... i times... A 25

26 Regular Expressions Def. The regular expressions over Σ are the smallest set of expressions including ε ' c' where c A+ B where A, B are rexp over AB " " " * A where A is a rexp over 26

27 Syntax vs. Semantics To be careful, we should distinguish syntax and semantics (meaning) of regular expressions L( ε) = {""} L(' c') = {" c"} LA ( + B) = LA ( ) LB ( ) L( AB) = { ab a L( A) and b L( B)} LA = U LA * ( ) ( i ) i 0 27

28 Example: Keyword Keyword: else or if or begin or ' else' + 'if' + 'begin' + L Note: 'else' abbreviates 'e''l''s''e' 28

29 Example: Integers Integer: a non-empty string of digits digit = '0' + '1' + '2' + '3' + '4' + '5' + '6' + '7' + '8' + '9' integer = digit digit * Abbreviation: A + = AA * 29

30 Example: Identifier Identifier: strings of letters or digits, starting with a letter letter = 'A' + K+ 'Z' + 'a' + K+ 'z' identifier = letter (letter + digit) * * * Is (letter + digit ) the same? 30

31 Example: Whitespace Whitespace: a non-empty sequence of blanks, newlines, and tabs (' ' + '\n' + '\t') + 31

32 Example 1: Phone Numbers Regular expressions are all around you! Consider +46(0) Σ = digits {+,,(,)} country = digit digit city = digit digit univ = digit digit digit extension = digit digit digit digit phone_num = + country ( 0 ) city univ extension 32

33 Example 2: Addresses Consider = letters name = + letter address = name name '.' name '.' name 33

34 Summary Regular expressions describe many useful languages Regular languages are a language specification We still need an implementation Next: Given a string s and a regular expression R, is s L( R)? A yes/no answer is not enough! Instead: partition the input into tokens We will adapt regular expressions to this goal 34

35 Implementation of Lexical Analysis

36 Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation of regular expressions RegExp NFA DFA Tables 36

37 Notation For convenience, we will use a variation (we will allow user-defined abbreviations) in regular expression notation Union: A + B A B Option: A + ε A? Range: a + b + + z [a-z] Excluded range: complement of [a-z] [^a-z] 37

38 Regular Expressions Lexical Specifications 1. Select a set of tokens Integer, Keyword, Identifier, LeftPar, Write a regular expression (pattern) for the lexemes of each token Integer = digit + Keyword = if + else + Identifier = letter (letter + digit)* LeftPar = ( 38

39 Regular Expressions Lexical Specifications 3. Construct R, a regular expression matching all lexemes for all tokens R = Keyword + Identifier + Integer + = R 1 + R 2 + R 3 + Facts: If s L(R) then s is a lexeme Furthermore s L(R i ) for some i This i determines the token that is reported 39

40 Regular Expressions Lexical Specifications 4. Let input be x 1 x n (x 1... x n are characters in the language alphabet) For 1 i n check x 1 x i L(R)? 5. It must be that x 1 x i L(R j ) for some i and j (if there is a choice, pick a smallest such j) 6. Report token j, remove x1 x i from input and go to step 4 40

41 How to Handle Spaces and Comments? 1. We could create a token Whitespace Whitespace = ( + \n + \t ) + We could also add comments in there An input " \t\n 555 " is transformed into Whitespace Integer Whitespace 2. Lexical analyzer skips spaces (preferred) Modify step 5 from before as follows: It must be that x k... x i L(R j ) for some j such that x 1... x k-1 L(Whitespace) Parser is not bothered with spaces 41

42 Ambiguities (1) There are ambiguities in the algorithm How much input is used? What if x 1 x i L(R) and also x 1 x K L(R) The maximal munch rule: Pick the longest possible substring that matches R 42

43 Ambiguities (2) Which token is used? What if x 1 x i L(R j ) and also x 1 x i L(R k ) Rule: use rule listed first (j if j < k) Example: R 1 = Keyword and R 2 = Identifier if matches both Treats if as a keyword not an identifier 43

44 Error Handling What if No rule matches a prefix of input? Problem: Can t just get stuck Solution: Write a rule matching all bad strings Put it last Lexical analysis tools allow the writing of: R = R R n + Error Token Error matches if nothing else matches 44

45 Summary Regular expressions provide a concise notation for string patterns Use in lexical analysis requires small extensions To resolve ambiguities To handle errors Good algorithms known (next) Require only single pass over the input Few operations per character (table lookup) 45

46 Regular Languages & Finite Automata Basic formal language theory result: Regular expressions and finite automata both define the class of regular languages. Thus, we are going to use: Regular expressions for specification Finite automata for implementation (automatic generation of lexical analyzers) 46

47 Finite Automata A finite automaton is a recognizer for the strings of a regular language A finite automaton consists of A finite input alphabet Σ A set of states S A start state n A set of accepting states F S A set of transitions state input state 47

48 Finite Automata Transition Is read s 1 a s 2 In state s 1 on input a go to state s 2 If end of input If in accepting state accept Otherwise If no transition possible reject 48

49 Finite Automata State Graphs A state The start state An accepting state A transition a 49

50 A Simple Example A finite automaton that accepts only 1 1 A finite automaton accepts a string if we can follow transitions labeled with the characters in the string from the start to some accepting state 50

51 Another Simple Example A finite automaton accepting any number of 1 s followed by a single 0 Alphabet: {0,1}

52 And Another Example Alphabet {0,1} What language does this recognize?

53 And Another Example Alphabet still { 0, 1 } 1 1 The operation of the automaton is not completely defined by the input On input 11 the automaton could be in either state 53

54 Epsilon Moves Another kind of transition: ε-moves A ε B Machine can move from state A to state B without reading input 54

55 Deterministic and Non-Deterministic Automata Deterministic Finite Automata (DFA) One transition per input per state No ε-moves Non-deterministic Finite Automata (NFA) Can have multiple transitions for one input in a given state Can have ε-moves Finite automata have finite memory Enough to only encode the current state 55

56 Execution of Finite Automata A DFA can take only one path through the state graph Completely determined by input NFAs can choose Whether to make ε-moves Which of multiple transitions for a single input to take 56

57 Acceptance of NFAs An NFA can get into multiple states Input: Rule: NFA accepts an input if it can get in a final state 57

58 NFA vs. DFA (1) NFAs and DFAs recognize the same set of languages (regular languages) DFAs are easier to implement There are no choices to consider 58

59 NFA vs. DFA (2) For a given language the NFA can be simpler than the DFA NFA DFA DFA can be exponentially larger than NFA (contrary to what is shown in the above example)

60 Regular Expressions to Finite Automata High-level sketch NFA Regular expressions DFA Lexical Specification Table-driven Implementation of DFA 60

61 Regular Expressions to NFA (1) For each kind of reg. expr, define an NFA Notation: NFA for regular expression M M i.e. our automata have one start and one accepting state For ε For input a ε a 61

62 Regular Expressions to NFA (2) For AB A ε B For A + B ε ε B A ε ε 62

63 Regular Expressions to NFA (3) For A* ε ε A ε 63

64 Example of Regular Expression NFA conversion Consider the regular expression (1+0)*1 The NFA is ε A ε B ε ε C D 1 0 E F ε ε G H ε I 1 J ε 64

65 NFA to DFA. The Trick Simulate the NFA Each state of DFA = a non-empty subset of states of the NFA Start state = the set of NFA states reachable through ε-moves from NFA start state Add a transition S a S to DFA iff S is the set of NFA states reachable from any state in S after seeing the input a considering ε-moves as well 65

66 NFA to DFA. Remark An NFA may be in many states at any time How many different states? If there are N states, the NFA must be in some subset of those N states How many subsets are there? 2 N - 1 = finitely many 66

67 NFA to DFA Example A ε B ABCDHI ε ε 0 1 C D ε ε 1 0 E F ε ε FGABCDHI 0 1 EJGABCDHI ε 1 G H I J

68 Implementation A DFA can be implemented by a 2D table T One dimension is states Other dimension is input symbols For every transition S i a S k define T[i,a] = k DFA execution If in state S i and input a, read T[i,a] = k and skip to state S k Very efficient 68

69 Table Implementation of a DFA S 0 1 T 0 1 U S T U T T U U T U 69

70 Implementation (Cont.) NFA DFA conversion is at the heart of tools such as lex, ML-Lex, flex or jlex But, DFAs can be huge In practice, flex-like tools trade off speed for space in the choice of NFA and DFA representations 70

71 Theory vs. Practice Two differences: DFAs recognize lexemes. A lexer must return a type of acceptance (token type) rather than simply an accept/reject indication. DFAs consume the complete string and accept or reject it. A lexer must find the end of the lexeme in the input stream and then find the next one, etc. 71

### Introduction to Lexical Analysis

Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples

### Implementation of Lexical Analysis

Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation

### Implementation of Lexical Analysis

Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation

### Lexical Analysis. Chapter 2

Lexical Analysis Chapter 2 1 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples

### Lexical Analysis. Lecture 2-4

Lexical Analysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 2 1 Administrivia Moving to 60 Evans on Wednesday HW1 available Pyth manual available on line.

### Implementation of Lexical Analysis

Outline Implementation of Lexical nalysis Specifying lexical structure using regular expressions Finite automata Deterministic Finite utomata (DFs) Non-deterministic Finite utomata (NFs) Implementation

### Lexical Analysis. Lecture 3-4

Lexical Analysis Lecture 3-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 3-4 1 Administrivia I suggest you start looking at Python (see link on class home page). Please

### Administrivia. Lexical Analysis. Lecture 2-4. Outline. The Structure of a Compiler. Informal sketch of lexical analysis. Issues in lexical analysis

dministrivia Lexical nalysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Moving to 6 Evans on Wednesday HW available Pyth manual available on line. Please log into your account and electronically

### Syntactic Analysis. CS345H: Programming Languages. Lecture 3: Lexical Analysis. Outline. Lexical Analysis. What is a Token? Tokens

Syntactic Analysis CS45H: Programming Languages Lecture : Lexical Analysis Thomas Dillig Main Question: How to give structure to strings Analogy: Understanding an English sentence First, we separate a

### Implementation of Lexical Analysis

Implementation of Lexical Analysis Lecture 4 (Modified by Professor Vijay Ganesh) Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested

### Lexical Analysis. Lecture 3. January 10, 2018

Lexical Analysis Lecture 3 January 10, 2018 Announcements PA1c due tonight at 11:50pm! Don t forget about PA1, the Cool implementation! Use Monday s lecture, the video guides and Cool examples if you re

### Implementation of Lexical Analysis. Lecture 4

Implementation of Lexical Analysis Lecture 4 1 Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested It is easier to modify a working

### Lexical Analysis. Finite Automata

#1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical

### Lexical Analysis. Implementation: Finite Automata

Lexical Analysis Implementation: Finite Automata Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs)

### Implementation of Lexical Analysis

Written ssignments W assigned today Implementation of Lexical nalysis Lecture 4 Due in one week :59pm Electronic hand-in Prof. iken CS 43 Lecture 4 Prof. iken CS 43 Lecture 4 2 Tips on uilding Large Systems

### Implementation of Lexical Analysis

Written ssignments W assigned today Implementation of Lexical nalysis Lecture 4 Due in one week y 5pm Turn in In class In box outside 4 Gates Electronically Prof. iken CS 43 Lecture 4 Prof. iken CS 43

### Lexical Analysis. Finite Automata

#1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical

### Cunning Plan. Informal Sketch of Lexical Analysis. Issues in Lexical Analysis. Specifying Lexers

Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical Analysis Lookahead Ambiguity Specifying Lexers Regular Expressions

### CSE450. Translation of Programming Languages. Lecture 20: Automata and Regular Expressions

CSE45 Translation of Programming Languages Lecture 2: Automata and Regular Expressions Finite Automata Regular Expression = Specification Finite Automata = Implementation A finite automaton consists of:

### Compiler course. Chapter 3 Lexical Analysis

Compiler course Chapter 3 Lexical Analysis 1 A. A. Pourhaji Kazem, Spring 2009 Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata

### Chapter 3 Lexical Analysis

Chapter 3 Lexical Analysis Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator The role of lexical

### Lexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata

Lexical Analysis Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Phase Ordering of Front-Ends Lexical analysis (lexer) Break input string

### Lexical Analysis. Finite Automata. (Part 2 of 2)

# Lexical Analysis Finite Automata (Part 2 of 2) PA0, PA Although we have included the tricky file ends without a newline testcases in previous years, students made good cases against them (e.g., they

### Kinder, Gentler Nation

Lexical Analysis Finite Automata (Part 2 of 2) # Kinder, Gentler Nation In our post drop-deadline world things get easier. While we re here: reading quiz. #2 Summary Regular expressions provide a concise

### CSE450. Translation of Programming Languages. Automata, Simple Language Design Principles

CSE45 Translation of Programming Languages Automata, Simple Language Design Principles Finite Automata State Graphs A state: The start state: An accepting state: A transition: a A Simple Example A finite

### Administrativia. Extra credit for bugs in project assignments. Building a Scanner. CS164, Fall Recall: The Structure of a Compiler

dministrativia Building a canner Extra credit for bugs in project assignments in starter kits and handouts Ts are final arbiters of what s a bug only the first student to report the bug gets credit C64,

### Formal Languages and Compilers Lecture VI: Lexical Analysis

Formal Languages and Compilers Lecture VI: Lexical Analysis Free University of Bozen-Bolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/ artale/ Formal

### Regular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications

Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata

### CS Lecture 2. The Front End. Lecture 2 Lexical Analysis

CS 1622 Lecture 2 Lexical Analysis CS 1622 Lecture 2 1 Lecture 2 Review of last lecture and finish up overview The first compiler phase: lexical analysis Reading: Chapter 2 in text (by 1/18) CS 1622 Lecture

### Concepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens

Concepts Introduced in Chapter 3 Lexical Analysis Regular Expressions (REs) Nondeterministic Finite Automata (NFA) Converting an RE to an NFA Deterministic Finite Automatic (DFA) Lexical Analysis Why separate

### UNIT -2 LEXICAL ANALYSIS

OVER VIEW OF LEXICAL ANALYSIS UNIT -2 LEXICAL ANALYSIS o To identify the tokens we need some method of describing the possible tokens that can appear in the input stream. For this purpose we introduce

### Lexical Analysis. Introduction

Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies

### CSE 413 Programming Languages & Implementation. Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions

CSE 413 Programming Languages & Implementation Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory

### CSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B-1

CSEP 501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2008 1/8/2008 2002-08 Hal Perkins & UW CSE B-1 Agenda Basic concepts of formal grammars (review) Regular expressions

### Compiler Construction

Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-16/cc/ Conceptual Structure of a Compiler Source code x1 := y2

### Week 2: Syntax Specification, Grammars

CS320 Principles of Programming Languages Week 2: Syntax Specification, Grammars Jingke Li Portland State University Fall 2017 PSU CS320 Fall 17 Week 2: Syntax Specification, Grammars 1/ 62 Words and Sentences

### Compiler Construction D7011E

Compiler Construction D7011E Lecture 2: Lexical analysis Viktor Leijon Slides largely by Johan Nordlander with material generously provided by Mark P. Jones. 1 Basics of Lexical Analysis: 2 Some definitions:

### Lexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast!

Lexical Analysis Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Compiler Passes Analysis of input program (front-end) character stream

### CSE 3302 Programming Languages Lecture 2: Syntax

CSE 3302 Programming Languages Lecture 2: Syntax (based on slides by Chengkai Li) Leonidas Fegaras University of Texas at Arlington CSE 3302 L2 Spring 2011 1 How do we define a PL? Specifying a PL: Syntax:

### CSE 413 Programming Languages & Implementation. Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions

CSE 413 Programming Languages & Implementation Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory

### COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou

COMP-421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3 - Lexical Analysis Sections 3.1-3.4, 3.6-3.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna

### Lexical Analyzer Scanner

Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce

### Lexical Analyzer Scanner

Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce

### CS412/413. Introduction to Compilers Tim Teitelbaum. Lecture 2: Lexical Analysis 23 Jan 08

CS412/413 Introduction to Compilers Tim Teitelbaum Lecture 2: Lexical Analysis 23 Jan 08 Outline Review compiler structure What is lexical analysis? Writing a lexer Specifying tokens: regular expressions

### David Griol Barres Computer Science Department Carlos III University of Madrid Leganés (Spain)

David Griol Barres dgriol@inf.uc3m.es Computer Science Department Carlos III University of Madrid Leganés (Spain) OUTLINE Introduction: Definitions The role of the Lexical Analyzer Scanner Implementation

### Compiler phases. Non-tokens

Compiler phases Compiler Construction Scanning Lexical Analysis source code scanner tokens regular expressions lexical analysis Lennart Andersson parser context free grammar Revision 2011 01 21 parse tree

### Defining Program Syntax. Chapter Two Modern Programming Languages, 2nd ed. 1

Defining Program Syntax Chapter Two Modern Programming Languages, 2nd ed. 1 Syntax And Semantics Programming language syntax: how programs look, their form and structure Syntax is defined using a kind

### Lexical Analysis. COMP 524, Spring 2014 Bryan Ward

Lexical Analysis COMP 524, Spring 2014 Bryan Ward Based in part on slides and notes by J. Erickson, S. Krishnan, B. Brandenburg, S. Olivier, A. Block and others The Big Picture Character Stream Scanner

### Part 5 Program Analysis Principles and Techniques

1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape

### Scanners. Xiaokang Qiu Purdue University. August 24, ECE 468 Adapted from Kulkarni 2012

Scanners Xiaokang Qiu Purdue University ECE 468 Adapted from Kulkarni 2012 August 24, 2016 Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved

### 1. Lexical Analysis Phase

1. Lexical Analysis Phase The purpose of the lexical analyzer is to read the source program, one character at time, and to translate it into a sequence of primitive units called tokens. Keywords, identifiers,

### Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective

Concepts Lexical scanning Regular expressions DFAs and FSAs Lex CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 2 Lexical analysis

### Chapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective

Chapter 4 Lexical analysis Lexical scanning Regular expressions DFAs and FSAs Lex Concepts CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley

### Compiler Construction

Compiler Construction Lecture 2: Lexical Analysis I (Introduction) Thomas Noll Lehrstuhl für Informatik 2 (Software Modeling and Verification) noll@cs.rwth-aachen.de http://moves.rwth-aachen.de/teaching/ss-14/cc14/

### Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres

Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres dgriol@inf.uc3m.es Introduction: Definitions Lexical analysis or scanning: To read from left-to-right a source

### CSc 453 Lexical Analysis (Scanning)

CSc 453 Lexical Analysis (Scanning) Saumya Debray The University of Arizona Tucson Overview source program lexical analyzer (scanner) tokens syntax analyzer (parser) symbol table manager Main task: to

### Front End: Lexical Analysis. The Structure of a Compiler

Front End: Lexical Analysis The Structure of a Compiler Constructing a Lexical Analyser By hand: Identify lexemes in input and return tokens Automatically: Lexical-Analyser generator We will learn about

### Interpreter. Scanner. Parser. Tree Walker. read. request token. send token. send AST I/O. Console

Scanning 1 read Interpreter Scanner request token Parser send token Console I/O send AST Tree Walker 2 Scanner This process is known as: Scanning, lexing (lexical analysis), and tokenizing This is the

### CS308 Compiler Principles Lexical Analyzer Li Jiang

CS308 Lexical Analyzer Li Jiang Department of Computer Science and Engineering Shanghai Jiao Tong University Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular

### Lexical analysis. Syntactical analysis. Semantical analysis. Intermediate code generation. Optimization. Code generation. Target specific optimization

Second round: the scanner Lexical analysis Syntactical analysis Semantical analysis Intermediate code generation Optimization Code generation Target specific optimization Lexical analysis (Chapter 3) Why

### Monday, August 26, 13. Scanners

Scanners Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved words, literals, etc. What do we need to know? How do we define tokens? How can

### Compiler Construction

Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-16/cc/ Recap: First-Longest-Match Analysis Outline of Lecture

### Compiler Construction

Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-17/cc/ Recap: First-Longest-Match Analysis The Extended Matching

### Languages, Automata, Regular Expressions & Scanners. Winter /8/ Hal Perkins & UW CSE B-1

CSE 401 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2010 1/8/2010 2002-10 Hal Perkins & UW CSE B-1 Agenda Quick review of basic concepts of formal grammars Regular

### CS 314 Principles of Programming Languages. Lecture 3

CS 314 Principles of Programming Languages Lecture 3 Zheng Zhang Department of Computer Science Rutgers University Wednesday 14 th September, 2016 Zheng Zhang 1 CS@Rutgers University Class Information

### Wednesday, September 3, 14. Scanners

Scanners Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved words, literals, etc. What do we need to know? How do we define tokens? How can

### Lexical Analysis. Chapter 1, Section Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual

Lexical Analysis Chapter 1, Section 1.2.1 Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual Inside the Compiler: Front End Lexical analyzer (aka scanner) Converts ASCII or Unicode to a stream of tokens

### CSCE 314 Programming Languages

CSCE 314 Programming Languages Syntactic Analysis Dr. Hyunyoung Lee 1 What Is a Programming Language? Language = syntax + semantics The syntax of a language is concerned with the form of a program: how

### ECS 120 Lesson 7 Regular Expressions, Pt. 1

ECS 120 Lesson 7 Regular Expressions, Pt. 1 Oliver Kreylos Friday, April 13th, 2001 1 Outline Thus far, we have been discussing one way to specify a (regular) language: Giving a machine that reads a word

### Zhizheng Zhang. Southeast University

Zhizheng Zhang Southeast University 2016/10/5 Lexical Analysis 1 1. The Role of Lexical Analyzer 2016/10/5 Lexical Analysis 2 2016/10/5 Lexical Analysis 3 Example. position = initial + rate * 60 2016/10/5

### Non-deterministic Finite Automata (NFA)

Non-deterministic Finite Automata (NFA) CAN have transitions on the same input to different states Can include a ε or λ transition (i.e. move to new state without reading input) Often easier to design

### MIT Specifying Languages with Regular Expressions and Context-Free Grammars

MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology Language Definition Problem How to precisely

### CS164: Programming Assignment 2 Dlex Lexer Generator and Decaf Lexer

CS164: Programming Assignment 2 Dlex Lexer Generator and Decaf Lexer Assigned: Thursday, September 16, 2004 Due: Tuesday, September 28, 2004, at 11:59pm September 16, 2004 1 Introduction Overview In this

### CMSC 350: COMPILER DESIGN

Lecture 11 CMSC 350: COMPILER DESIGN see HW3 LLVMLITE SPECIFICATION Eisenberg CMSC 350: Compilers 2 Discussion: Defining a Language Premise: programming languages are purely formal objects We (as language

### Regular Languages. MACM 300 Formal Languages and Automata. Formal Languages: Recap. Regular Languages

Regular Languages MACM 3 Formal Languages and Automata Anoop Sarkar http://www.cs.sfu.ca/~anoop The set of regular languages: each element is a regular language Each regular language is an example of a

### The Front End. The purpose of the front end is to deal with the input language. Perform a membership test: code source language?

The Front End Source code Front End IR Back End Machine code Errors The purpose of the front end is to deal with the input language Perform a membership test: code source language? Is the program well-formed

### CSC 467 Lecture 3: Regular Expressions

CSC 467 Lecture 3: Regular Expressions Recall How we build a lexer by hand o Use fgetc/mmap to read input o Use a big switch to match patterns Homework exercise static TokenKind identifier( TokenKind token

### Figure 2.1: Role of Lexical Analyzer

Chapter 2 Lexical Analysis Lexical analysis or scanning is the process which reads the stream of characters making up the source program from left-to-right and groups them into tokens. The lexical analyzer

### MIT Specifying Languages with Regular Expressions and Context-Free Grammars. Martin Rinard Massachusetts Institute of Technology

MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Massachusetts Institute of Technology Language Definition Problem How to precisely define language Layered structure

### CS415 Compilers. Lexical Analysis

CS415 Compilers Lexical Analysis These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Lecture 7 1 Announcements First project and second homework

### Languages and Compilers

Principles of Software Engineering and Operational Systems Languages and Compilers SDAGE: Level I 2012-13 4. Lexical Analysis (Scanning) Dr Valery Adzhiev vadzhiev@bournemouth.ac.uk Office: TA-121 For

### CSCI312 Principles of Programming Languages!

CSCI312 Principles of Programming Languages!! Chapter 3 Regular Expression and Lexer Xu Liu Recap! Copyright 2006 The McGraw-Hill Companies, Inc. Clite: Lexical Syntax! Input: a stream of characters from

### Formal Languages and Grammars. Chapter 2: Sections 2.1 and 2.2

Formal Languages and Grammars Chapter 2: Sections 2.1 and 2.2 Formal Languages Basis for the design and implementation of programming languages Alphabet: finite set Σ of symbols String: finite sequence

### Lex Spec Example. Int installid() {/* code to put id lexeme into string table*/}

Class 5 Lex Spec Example delim [ \t\n] ws {delim}+ letter [A-Aa-z] digit [0-9] id {letter}({letter} {digit})* number {digit}+(\.{digit}+)?(e[+-]?{digit}+)? %% {ws} {/*no action and no return*?} if {return(if);}

### Outline. 1 Scanning Tokens. 2 Regular Expresssions. 3 Finite State Automata

Outline 1 2 Regular Expresssions Lexical Analysis 3 Finite State Automata 4 Non-deterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA 6 NFA to DFA 7 8 JavaCC:

### Structure of Programming Languages Lecture 3

Structure of Programming Languages Lecture 3 CSCI 6636 4536 Spring 2017 CSCI 6636 4536 Lecture 3... 1/25 Spring 2017 1 / 25 Outline 1 Finite Languages Deterministic Finite State Machines Lexical Analysis

### PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer

PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer As the first phase of a compiler, the main task of the lexical analyzer is to read the input

### CSc 453 Compilers and Systems Software

CSc 453 Compilers and Systems Software 3 : Lexical Analysis I Christian Collberg Department of Computer Science University of Arizona collberg@gmail.com Copyright c 2009 Christian Collberg August 23, 2009

### Lexical Analysis 1 / 52

Lexical Analysis 1 / 52 Outline 1 Scanning Tokens 2 Regular Expresssions 3 Finite State Automata 4 Non-deterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA

### CS 432 Fall Mike Lam, Professor. Finite Automata Conversions and Lexing

CS 432 Fall 2017 Mike Lam, Professor Finite Automata Conversions and Lexing Finite Automata Key result: all of the following have the same expressive power (i.e., they all describe regular languages):

### Chapter 3: Lexing and Parsing

Chapter 3: Lexing and Parsing Aarne Ranta Slides for the book Implementing Programming Languages. An Introduction to Compilers and Interpreters, College Publications, 2012. Lexing and Parsing* Deeper understanding

### Programming Languages and Compilers (CS 421)

Programming Languages and Compilers (CS 421) Elsa L Gunter 2112 SC, UIUC http://courses.engr.illinois.edu/cs421 Based in part on slides by Mattox Beckman, as updated by Vikram Adve and Gul Agha 10/30/17

### 2. Lexical Analysis! Prof. O. Nierstrasz!

2. Lexical Analysis! Prof. O. Nierstrasz! Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes.! http://www.cs.ucla.edu/~palsberg/! http://www.cs.purdue.edu/homes/hosking/!

### Programming Languages & Compilers. Programming Languages and Compilers (CS 421) I. Major Phases of a Compiler. Programming Languages & Compilers

Programming Languages & Compilers Programming Languages and Compilers (CS 421) I Three Main Topics of the Course II III Elsa L Gunter 2112 SC, UIUC http://courses.engr.illinois.edu/cs421 New Programming

### Automating Construction of Lexers

Automating Construction of Lexers Regular Expression to Programs Not all regular expressions are simple. How can we write a lexer for (a*b aaa)? Tokenizing aaaab Vs aaaaaa Regular Expression Finite state

### Writing a Lexical Analyzer in Haskell (part II)

Writing a Lexical Analyzer in Haskell (part II) Today Regular languages and lexicographical analysis part II Some of the slides today are from Dr. Saumya Debray and Dr. Christian Colberg This week PA1:

### Last lecture CMSC330. This lecture. Finite Automata: States. Finite Automata. Implementing Regular Expressions. Languages. Regular expressions

Last lecture CMSC330 Finite Automata Languages Sets of strings Operations on languages Regular expressions Constants Operators Precedence 1 2 Finite automata States Transitions Examples Types This lecture