CSC 467 Lecture 3: Regular Expressions

Size: px
Start display at page:

Download "CSC 467 Lecture 3: Regular Expressions"

Transcription

1 CSC 467 Lecture 3: Regular Expressions Recall How we build a lexer by hand o Use fgetc/mmap to read input o Use a big switch to match patterns Homework exercise static TokenKind identifier( TokenKind token ) { /* exercise for you did you do it?*/ int len; char* start = cursor; while( isletter(*cursor) isdigit(*cursor) ) cursor ++; token->kind = KIND_TOKEN_IDENTIFIER; len = cursor start; token->u.stringval = malloc( len +1 ); strncpy( token->u.stringval, start, len ); token->u.stringval[len] = \0 ; return token->kind; TokenKind gettoken( Token token ) { for( ; ; ) { c = *curser ++; switch( c ) { case a : case A : if( c[-1] == f && c[0] == o && c[1] == r && isblank(c[2]) ) { cursor += 2; return KIND_TOKEN_FOR; return identifier( token ); Today How can we build lexer systematically

2 Start by how to describe token patterns Regular Expressions The notation we use to precisely capture all the variations that a given category of token may take are called "regular expressions" (or, less formally, "patterns". The word "pattern" is really vague and there are lots of other notations for patterns besides regular expressions). Regular expressions are a shorthand notation for sets of strings. In order to even talk about "strings" you have to first define an alphabet, the set of characters which can appear. 1. Epsilon: 1. Notation: (ε) 2. Definition: { : is a regular expression denoting the set containing the empty string 2. Symbol: 1. a 2. { a : Any letter in the alphabet is also a regular expression denoting the set containing a one-letter string consisting of that letter. 3. Alteration: For regular expressions r and s, 1. r s 2. is a regular expression denoting the union of r and s 4. Concatenation: For regular expressions r and s, 1. r s 2. is a regular expression denoting the set of strings consisting of a member of r followed by a member of s 5. Repetition: For regular expression r, 1. r* 2. is a regular expression denoting the set of strings consisting of zero or more occurrences of r. Notation Sugar Although these operators are sufficient to describe all regular languages, in practice everybody uses extensions: You can parenthesize a regular expression to specify operator precedence (otherwise, alternation is like plus, concatenation is like times, and closure is like exponentiation) For regular expression r, r+ is a regular expression denoting the set of strings consisting of one or more occurrences of r. Equivalent to rr*

3 For regular expression r, r? is a regular expression denoting the set of strings consisting of zero or one occurrence of r. Equivalent to r ε The notation [abc] is short for a b c. [a-z] is short for a b... z. [^abc] is short for: any character other than a, b, or c. Example for (keyword) for letter [a-za-z] digit [0-9] identifier letter (letter digit)* sign + - ε integer sign (0 [1-9] digit*) decimal integer.digit* real (integer decimal) E sign digit* There is some ambiguity though: If the input includes the characters for8, then the first rule (for for-keyword) matches 3 characters (for), the fourth rule (for identifier) can match 1, 2, 3, or 4 characters, the longest being for8. To resolve this type of ambiguities, when there is a choice of rules, scanner generators choose the one that matches the maximum number of characters. In this case, the chosen rule is the one for identifier that matches 4 characters (for8). This disambiguation rule is called the longest match rule. If there are more than one rules that match the same maximum number of characters, the rule listed first is chosen. This is the rule priority disambiguation rule. For example, the lexical word for is For example, the lexical word for is taken as a for-keyword even though it uses the same number of characters as an identifier. lex(1) and flex(1) These programs generally take a lexical specification given in a.l file and create a corresponding C language lexical analyzer in a file named lex.yy.c. The lexical analyzer is then linked with the rest of your compiler. The C code generated by lex has the following public interface. Note the use of global variables instead of parameters, and the use of the prefix yy to distinguish scanner names from your program names. This prefix is also used in the YACC parser generator. FILE *yyin; /* set this variable prior to calling yylex() */ int yylex(); /* call this function once for each token */ char yytext[]; /* yylex() writes the token's lexeme to an array */ /* note: with flex, I believe extern declarations must read extern char *yytext; */ int yywrap(); /* called by lex when it hits end-of-file; see below */

4 The.l file format consists of a mixture of lex syntax and C code fragments. The percent sign (%) is used to signify lex elements. The whole file is divided into three sections separated by %%: %% %% header body helper functions The header consists of C code fragments enclosed in %{ and % as well as macro definitions consisting of a name and a regular expression denoted by that name. lex macros are invoked explicitly by enclosing the macro name in curly braces. Following are some example lex macros. letter [a-za-z] digit [0-9] ident {letter({letter {digit)* The body consists of of a sequence of regular expressions for different token categories and other lexical entities. Each regular expression can have a C code fragment enclosed in curly braces that executes when that regular expression is matched. For most of the regular expressions this code fragment (also called a semantic action consists of returning an integer that identifies the token category to the rest of the compiler, particularly for use by the parser to check syntax. Some typical regular expressions and semantic actions might include: " " { /* no-op, discard whitespace */ {ident { return IDENTIFIER; "*" { return ASTERISK; "." { return PERIOD; You also need regular expressions for lexical errors such as unterminated character constants, or illegal characters. The helper functions in a lex file typically compute lexical attributes, such as the actual integer or string values denoted by literals. One helper function you have to write is yywrap(), which is called when lex hits end of file. If you just want lex to quit, have yywrap() return 1. If your yywrap() switches yyin to a different file and you want lex to continue processing, have yywrap() return 0. The lex or flex library (-ll or -lfl) have default yywrap() function which return a 1, and flex has the directive %option noyywrap which allows you to skip writing this function. A Short Comment on Lexing C Reals C float and double constants have to have at least one digit, either before or after the required decimal. This is a pain: ([0-9]+.[0-9]* [0-9]*.[0-9]+)... You might almost be happier if you wrote ([0-9]*.[0-9]*) { return (strcmp(yytext,"."))? REAL : PERIOD;

5 You-all know C's ternary e1? e2 : e3 operator, don't ya? Its an if-then-else expression, very slick. Lex extended regular expressions Lex further extends the regular expressions with several helpful operators. Lex's regular expressions include: c normal characters mean themselves \c backslash escapes remove the meaning from most operator characters. Inside character sets and quotes, backslash performs C-style escapes. "s" Double quotes mean to match the C string given as itself. This is particularly useful for multi-byte operators and may be more readable than using backslash multiple times. [s] This character set operator matches any one character among those in s. [^s] A negated-set matches any one character not among those in s.. The dot operator matches any one character except newline: [^\n] r* match r 0 or more times. r+ match r 1 or more times. r? match r 0 or 1 time. r{m,n match r between m and n times. r 1 r 2 concatenation. match r 1 followed by r 2 r 1 r 2 alternation. match r 1 or r 2 (r) parentheses specify precedence but do not match anything r 1 /r 2 lookahead. match r 1 when r 2 follows, without consuming r 2 ^r match r only when it occurs at the beginning of a line r$ match r only when it occurs at the end of a line

Using Lex or Flex. Prof. James L. Frankel Harvard University

Using Lex or Flex. Prof. James L. Frankel Harvard University Using Lex or Flex Prof. James L. Frankel Harvard University Version of 1:07 PM 26-Sep-2016 Copyright 2016, 2015 James L. Frankel. All rights reserved. Lex Regular Expressions (1 of 4) Special characters

More information

Lecture Outline. COMP-421 Compiler Design. What is Lex? Lex Specification. ! Lexical Analyzer Lex. ! Lex Examples. Presented by Dr Ioanna Dionysiou

Lecture Outline. COMP-421 Compiler Design. What is Lex? Lex Specification. ! Lexical Analyzer Lex. ! Lex Examples. Presented by Dr Ioanna Dionysiou Lecture Outline COMP-421 Compiler Design! Lexical Analyzer Lex! Lex Examples Presented by Dr Ioanna Dionysiou Figures and part of the lecture notes taken from A compact guide to lex&yacc, epaperpress.com

More information

EXPERIMENT NO : M/C Lenovo Think center M700 Ci3,6100,6th Gen. H81, 4GB RAM,500GB HDD

EXPERIMENT NO : M/C Lenovo Think center M700 Ci3,6100,6th Gen. H81, 4GB RAM,500GB HDD GROUP - B EXPERIMENT NO : 07 1. Title: Write a program using Lex specifications to implement lexical analysis phase of compiler to total nos of words, chars and line etc of given file. 2. Objectives :

More information

EXPERIMENT NO : M/C Lenovo Think center M700 Ci3,6100,6th Gen. H81, 4GB RAM,500GB HDD

EXPERIMENT NO : M/C Lenovo Think center M700 Ci3,6100,6th Gen. H81, 4GB RAM,500GB HDD GROUP - B EXPERIMENT NO : 06 1. Title: Write a program using Lex specifications to implement lexical analysis phase of compiler to generate tokens of subset of Java program 2. Objectives : - To understand

More information

Module 8 - Lexical Analyzer Generator. 8.1 Need for a Tool. 8.2 Lexical Analyzer Generator Tool

Module 8 - Lexical Analyzer Generator. 8.1 Need for a Tool. 8.2 Lexical Analyzer Generator Tool Module 8 - Lexical Analyzer Generator This module discusses the core issues in designing a lexical analyzer generator from basis or using a tool. The basics of LEX tool are also discussed. 8.1 Need for

More information

Ulex: A Lexical Analyzer Generator for Unicon

Ulex: A Lexical Analyzer Generator for Unicon Ulex: A Lexical Analyzer Generator for Unicon Katrina Ray, Ray Pereda, and Clinton Jeffery Unicon Technical Report UTR 02a May 21, 2003 Abstract Ulex is a software tool for building language processors.

More information

Chapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective

Chapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective Chapter 4 Lexical analysis Lexical scanning Regular expressions DFAs and FSAs Lex Concepts CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley

More information

CS412/413. Introduction to Compilers Tim Teitelbaum. Lecture 2: Lexical Analysis 23 Jan 08

CS412/413. Introduction to Compilers Tim Teitelbaum. Lecture 2: Lexical Analysis 23 Jan 08 CS412/413 Introduction to Compilers Tim Teitelbaum Lecture 2: Lexical Analysis 23 Jan 08 Outline Review compiler structure What is lexical analysis? Writing a lexer Specifying tokens: regular expressions

More information

CS321 Languages and Compiler Design I. Winter 2012 Lecture 4

CS321 Languages and Compiler Design I. Winter 2012 Lecture 4 CS321 Languages and Compiler Design I Winter 2012 Lecture 4 1 LEXICAL ANALYSIS Convert source file characters into token stream. Remove content-free characters (comments, whitespace,...) Detect lexical

More information

Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective

Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective Concepts Lexical scanning Regular expressions DFAs and FSAs Lex CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 2 Lexical analysis

More information

Concepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens

Concepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens Concepts Introduced in Chapter 3 Lexical Analysis Regular Expressions (REs) Nondeterministic Finite Automata (NFA) Converting an RE to an NFA Deterministic Finite Automatic (DFA) Lexical Analysis Why separate

More information

The structure of a compiler

The structure of a compiler The structure of a compiler Source code front-end Intermediate front-end representation compiler back-end machine code Front-end & Back-end C front-end Pascal front-end C front-end Intel x86 back-end Motorola

More information

LECTURE 7. Lex and Intro to Parsing

LECTURE 7. Lex and Intro to Parsing LECTURE 7 Lex and Intro to Parsing LEX Last lecture, we learned a little bit about how we can take our regular expressions (which specify our valid tokens) and create real programs that can recognize them.

More information

A lexical analyzer generator for Standard ML. Version 1.6.0, October 1994

A lexical analyzer generator for Standard ML. Version 1.6.0, October 1994 A lexical analyzer generator for Standard ML. Version 1.6.0, October 1994 Andrew W. Appel 1 James S. Mattson David R. Tarditi 2 1 Department of Computer Science, Princeton University 2 School of Computer

More information

CSc 453 Lexical Analysis (Scanning)

CSc 453 Lexical Analysis (Scanning) CSc 453 Lexical Analysis (Scanning) Saumya Debray The University of Arizona Tucson Overview source program lexical analyzer (scanner) tokens syntax analyzer (parser) symbol table manager Main task: to

More information

Edited by Himanshu Mittal. Lexical Analysis Phase

Edited by Himanshu Mittal. Lexical Analysis Phase Edited by Himanshu Mittal Lexical Analysis Phase Lexical Analyzer The main task of Lexical analysis is to read input characters of source program and group them into lexemes and produce as output a sequence

More information

Flex and lexical analysis. October 25, 2016

Flex and lexical analysis. October 25, 2016 Flex and lexical analysis October 25, 2016 Flex and lexical analysis From the area of compilers, we get a host of tools to convert text files into programs. The first part of that process is often called

More information

Languages and Compilers

Languages and Compilers Principles of Software Engineering and Operational Systems Languages and Compilers SDAGE: Level I 2012-13 4. Lexical Analysis (Scanning) Dr Valery Adzhiev vadzhiev@bournemouth.ac.uk Office: TA-121 For

More information

Structure of Programming Languages Lecture 3

Structure of Programming Languages Lecture 3 Structure of Programming Languages Lecture 3 CSCI 6636 4536 Spring 2017 CSCI 6636 4536 Lecture 3... 1/25 Spring 2017 1 / 25 Outline 1 Finite Languages Deterministic Finite State Machines Lexical Analysis

More information

Figure 2.1: Role of Lexical Analyzer

Figure 2.1: Role of Lexical Analyzer Chapter 2 Lexical Analysis Lexical analysis or scanning is the process which reads the stream of characters making up the source program from left-to-right and groups them into tokens. The lexical analyzer

More information

Syntactic Analysis. CS345H: Programming Languages. Lecture 3: Lexical Analysis. Outline. Lexical Analysis. What is a Token? Tokens

Syntactic Analysis. CS345H: Programming Languages. Lecture 3: Lexical Analysis. Outline. Lexical Analysis. What is a Token? Tokens Syntactic Analysis CS45H: Programming Languages Lecture : Lexical Analysis Thomas Dillig Main Question: How to give structure to strings Analogy: Understanding an English sentence First, we separate a

More information

Introduction to Lexical Analysis

Introduction to Lexical Analysis Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples

More information

Big Picture: Compilation Process. CSCI: 4500/6500 Programming Languages. Big Picture: Compilation Process. Big Picture: Compilation Process.

Big Picture: Compilation Process. CSCI: 4500/6500 Programming Languages. Big Picture: Compilation Process. Big Picture: Compilation Process. Big Picture: Compilation Process Source program CSCI: 4500/6500 Programming Languages Lex & Yacc Scanner Lexical Lexical units, token stream Parser Syntax Intermediate Parse tree Code Generator Semantic

More information

CS4850 SummerII Lex Primer. Usage Paradigm of Lex. Lex is a tool for creating lexical analyzers. Lexical analyzers tokenize input streams.

CS4850 SummerII Lex Primer. Usage Paradigm of Lex. Lex is a tool for creating lexical analyzers. Lexical analyzers tokenize input streams. CS4850 SummerII 2006 Lexical Analysis and Lex (contd) 4.1 Lex Primer Lex is a tool for creating lexical analyzers. Lexical analyzers tokenize input streams. Tokens are the terminals of a language. Regular

More information

CS143 Handout 04 Summer 2011 June 22, 2011 flex In A Nutshell

CS143 Handout 04 Summer 2011 June 22, 2011 flex In A Nutshell CS143 Handout 04 Summer 2011 June 22, 2011 flex In A Nutshell Handout written by Julie Zelenski with minor edits by Keith. flex is a fast lexical analyzer generator. You specify the scanner you want in

More information

An introduction to Flex

An introduction to Flex An introduction to Flex 1 Introduction 1.1 What is Flex? Flex takes a set of descriptions of possible tokens and produces a scanner. 1.2 A short history Lex was developed at Bell Laboratories in the 1970s.

More information

Introduction to Lexical Analysis

Introduction to Lexical Analysis Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular

More information

Lexical Analysis. Lecture 3-4

Lexical Analysis. Lecture 3-4 Lexical Analysis Lecture 3-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 3-4 1 Administrivia I suggest you start looking at Python (see link on class home page). Please

More information

L L G E N. Generator of syntax analyzier (parser)

L L G E N. Generator of syntax analyzier (parser) L L G E N Generator of syntax analyzier (parser) GENERATOR L LGEN The main task of generator LLGEN is generate a parser (in C), which use the recursive go down method without recurrence The source code

More information

CS Lecture 2. The Front End. Lecture 2 Lexical Analysis

CS Lecture 2. The Front End. Lecture 2 Lexical Analysis CS 1622 Lecture 2 Lexical Analysis CS 1622 Lecture 2 1 Lecture 2 Review of last lecture and finish up overview The first compiler phase: lexical analysis Reading: Chapter 2 in text (by 1/18) CS 1622 Lecture

More information

Formal Languages and Compilers Lecture VI: Lexical Analysis

Formal Languages and Compilers Lecture VI: Lexical Analysis Formal Languages and Compilers Lecture VI: Lexical Analysis Free University of Bozen-Bolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/ artale/ Formal

More information

Flex and lexical analysis

Flex and lexical analysis Flex and lexical analysis From the area of compilers, we get a host of tools to convert text files into programs. The first part of that process is often called lexical analysis, particularly for such

More information

Lexical Analysis. Lecture 2-4

Lexical Analysis. Lecture 2-4 Lexical Analysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 2 1 Administrivia Moving to 60 Evans on Wednesday HW1 available Pyth manual available on line.

More information

Lexical Analysis. Implementing Scanners & LEX: A Lexical Analyzer Tool

Lexical Analysis. Implementing Scanners & LEX: A Lexical Analyzer Tool Lexical Analysis Implementing Scanners & LEX: A Lexical Analyzer Tool Copyright 2016, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California

More information

CSE302: Compiler Design

CSE302: Compiler Design CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 13, 2007 Outline Recap

More information

PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer

PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer As the first phase of a compiler, the main task of the lexical analyzer is to read the input

More information

Gechstudentszone.wordpress.com

Gechstudentszone.wordpress.com Unit 7 LEX AND YACC-1 Lex is a program generator designed for lexical processing of character input streams. It accepts a high-level, problem oriented specification for character string matching, and produces

More information

Outline CS4120/4121. Compilation in a Nutshell 1. Administration. Introduction to Compilers Andrew Myers. HW1 out later today due next Monday.

Outline CS4120/4121. Compilation in a Nutshell 1. Administration. Introduction to Compilers Andrew Myers. HW1 out later today due next Monday. CS4120/4121 Introduction to Compilers Andrew Myers Lecture 2: Lexical Analysis 31 August 2009 Outline Administration Compilation in a nutshell (or two) What is lexical analysis? Writing a lexer Specifying

More information

Lexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast!

Lexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Lexical Analysis Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Compiler Passes Analysis of input program (front-end) character stream

More information

Using an LALR(1) Parser Generator

Using an LALR(1) Parser Generator Using an LALR(1) Parser Generator Yacc is an LALR(1) parser generator Developed by S.C. Johnson and others at AT&T Bell Labs Yacc is an acronym for Yet another compiler compiler Yacc generates an integrated

More information

Big Picture: Compilation Process. CSCI: 4500/6500 Programming Languages. Big Picture: Compilation Process. Big Picture: Compilation Process

Big Picture: Compilation Process. CSCI: 4500/6500 Programming Languages. Big Picture: Compilation Process. Big Picture: Compilation Process Big Picture: Compilation Process Source program CSCI: 4500/6500 Programming Languages Lex & Yacc Symbol Table Scanner Lexical Parser Syntax Intermediate Code Generator Semantic Lexical units, token stream

More information

Parsing and Pattern Recognition

Parsing and Pattern Recognition Topics in IT 1 Parsing and Pattern Recognition Week 10 Lexical analysis College of Information Science and Engineering Ritsumeikan University 1 this week mid-term evaluation review lexical analysis its

More information

G Compiler Construction Lecture 4: Lexical Analysis. Mohamed Zahran (aka Z)

G Compiler Construction Lecture 4: Lexical Analysis. Mohamed Zahran (aka Z) G22.2130-001 Compiler Construction Lecture 4: Lexical Analysis Mohamed Zahran (aka Z) mzahran@cs.nyu.edu Role of the Lexical Analyzer Remove comments and white spaces (aka scanning) Macros expansion Read

More information

CS 403: Scanning and Parsing

CS 403: Scanning and Parsing CS 403: Scanning and Parsing Stefan D. Bruda Fall 2017 THE COMPILATION PROCESS Character stream Scanner (lexical analysis) Token stream Parser (syntax analysis) Parse tree Semantic analysis Abstract syntax

More information

THE COMPILATION PROCESS EXAMPLE OF TOKENS AND ATTRIBUTES

THE COMPILATION PROCESS EXAMPLE OF TOKENS AND ATTRIBUTES THE COMPILATION PROCESS Character stream CS 403: Scanning and Parsing Stefan D. Bruda Fall 207 Token stream Parse tree Abstract syntax tree Modified intermediate form Target language Modified target language

More information

LECTURE 6 Scanning Part 2

LECTURE 6 Scanning Part 2 LECTURE 6 Scanning Part 2 FROM DFA TO SCANNER In the previous lectures, we discussed how one might specify valid tokens in a language using regular expressions. We then discussed how we can create a recognizer

More information

Lexical Analysis. Chapter 2

Lexical Analysis. Chapter 2 Lexical Analysis Chapter 2 1 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples

More information

Handout 7, Lex (5/30/2001)

Handout 7, Lex (5/30/2001) Handout 7, Lex (5/30/2001) Lex is a venerable Unix tool that generates scanners. Input to lex is a text file that specifies the scanner; more precisely: specifying tokens, a yet to be made scanner must

More information

Lexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata

Lexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Lexical Analysis Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Phase Ordering of Front-Ends Lexical analysis (lexer) Break input string

More information

flex is not a bad tool to use for doing modest text transformations and for programs that collect statistics on input.

flex is not a bad tool to use for doing modest text transformations and for programs that collect statistics on input. flex is not a bad tool to use for doing modest text transformations and for programs that collect statistics on input. More often than not, though, you ll want to use flex to generate a scanner that divides

More information

LECTURE 11. Semantic Analysis and Yacc

LECTURE 11. Semantic Analysis and Yacc LECTURE 11 Semantic Analysis and Yacc REVIEW OF LAST LECTURE In the last lecture, we introduced the basic idea behind semantic analysis. Instead of merely specifying valid structures with a context-free

More information

1 Lexical Considerations

1 Lexical Considerations Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.035, Spring 2013 Handout Decaf Language Thursday, Feb 7 The project for the course is to write a compiler

More information

Lexical Analysis. Introduction

Lexical Analysis. Introduction Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies

More information

CS164: Programming Assignment 2 Dlex Lexer Generator and Decaf Lexer

CS164: Programming Assignment 2 Dlex Lexer Generator and Decaf Lexer CS164: Programming Assignment 2 Dlex Lexer Generator and Decaf Lexer Assigned: Thursday, September 16, 2004 Due: Tuesday, September 28, 2004, at 11:59pm September 16, 2004 1 Introduction Overview In this

More information

Lesson 10. CDT301 Compiler Theory, Spring 2011 Teacher: Linus Källberg

Lesson 10. CDT301 Compiler Theory, Spring 2011 Teacher: Linus Källberg Lesson 10 CDT301 Compiler Theory, Spring 2011 Teacher: Linus Källberg Outline Flex Bison Abstract syntax trees 2 FLEX 3 Flex Tool for automatic generation of scanners Open-source version of Lex Takes regular

More information

PRACTICAL CLASS: Flex & Bison

PRACTICAL CLASS: Flex & Bison Master s Degree Course in Computer Engineering Formal Languages FORMAL LANGUAGES AND COMPILERS PRACTICAL CLASS: Flex & Bison Eliana Bove eliana.bove@poliba.it Install On Linux: install with the package

More information

JFlex Regular Expressions

JFlex Regular Expressions JFlex Regular Expressions Lecture 17 Section 3.5, JFlex Manual Robb T. Koether Hampden-Sydney College Wed, Feb 25, 2015 Robb T. Koether (Hampden-Sydney College) JFlex Regular Expressions Wed, Feb 25, 2015

More information

Implementation of Lexical Analysis

Implementation of Lexical Analysis Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation

More information

Part 5 Program Analysis Principles and Techniques

Part 5 Program Analysis Principles and Techniques 1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape

More information

UNIT -2 LEXICAL ANALYSIS

UNIT -2 LEXICAL ANALYSIS OVER VIEW OF LEXICAL ANALYSIS UNIT -2 LEXICAL ANALYSIS o To identify the tokens we need some method of describing the possible tokens that can appear in the input stream. For this purpose we introduce

More information

Ray Pereda Unicon Technical Report UTR-02. February 25, Abstract

Ray Pereda Unicon Technical Report UTR-02. February 25, Abstract iflex: A Lexical Analyzer Generator for Icon Ray Pereda Unicon Technical Report UTR-02 February 25, 2000 Abstract iflex is software tool for building language processors. It is based on flex, a well-known

More information

Lexical Analysis - Flex

Lexical Analysis - Flex Lexical Analysis - Flex CMPSC 470 Lecture 03 Topics: Flex / JFlex A. Lex/Flex Lex and flex (fast lex) are programs that 1. Take, as input, a program containing regular expressions (describing patterns

More information

Lexical Analyzer Scanner

Lexical Analyzer Scanner Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce

More information

Introduction to Lex & Yacc. (flex & bison)

Introduction to Lex & Yacc. (flex & bison) Introduction to Lex & Yacc (flex & bison) Lex & Yacc (flex & bison) lexical rules (regular expression) lexical rules (context-free grammar) lex (flex) yacc (bison) Input yylex() yyparse() Processed output

More information

CMSC445 Compiler design Blaheta. Project 2: Lexer. Due: 15 February 2012

CMSC445 Compiler design Blaheta. Project 2: Lexer. Due: 15 February 2012 CMSC445 Compiler design Blaheta Project 2: Lexer Due: 15 February 2012 In this project we ll actually get started on our C compiler; we ll use flex to process a C program into a stream of tokens (not just

More information

Lex & Yacc. By H. Altay Güvenir. A compiler or an interpreter performs its task in 3 stages:

Lex & Yacc. By H. Altay Güvenir. A compiler or an interpreter performs its task in 3 stages: Lex & Yacc By H. Altay Güvenir A compiler or an interpreter performs its task in 3 stages: 1) Lexical Analysis: Lexical analyzer: scans the input stream and converts sequences of characters into tokens.

More information

Lexical Analysis. Lecture 3. January 10, 2018

Lexical Analysis. Lecture 3. January 10, 2018 Lexical Analysis Lecture 3 January 10, 2018 Announcements PA1c due tonight at 11:50pm! Don t forget about PA1, the Cool implementation! Use Monday s lecture, the video guides and Cool examples if you re

More information

Lexical Considerations

Lexical Considerations Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.035, Fall 2005 Handout 6 Decaf Language Wednesday, September 7 The project for the course is to write a

More information

Lexical Analyzer Scanner

Lexical Analyzer Scanner Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce

More information

Administrivia. Lexical Analysis. Lecture 2-4. Outline. The Structure of a Compiler. Informal sketch of lexical analysis. Issues in lexical analysis

Administrivia. Lexical Analysis. Lecture 2-4. Outline. The Structure of a Compiler. Informal sketch of lexical analysis. Issues in lexical analysis dministrivia Lexical nalysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Moving to 6 Evans on Wednesday HW available Pyth manual available on line. Please log into your account and electronically

More information

The Front End. The purpose of the front end is to deal with the input language. Perform a membership test: code source language?

The Front End. The purpose of the front end is to deal with the input language. Perform a membership test: code source language? The Front End Source code Front End IR Back End Machine code Errors The purpose of the front end is to deal with the input language Perform a membership test: code source language? Is the program well-formed

More information

Lex & Yacc. by H. Altay Güvenir. A compiler or an interpreter performs its task in 3 stages:

Lex & Yacc. by H. Altay Güvenir. A compiler or an interpreter performs its task in 3 stages: Lex & Yacc by H. Altay Güvenir A compiler or an interpreter performs its task in 3 stages: 1) Lexical Analysis: Lexical analyzer: scans the input stream and converts sequences of characters into tokens.

More information

UNIT - 7 LEX AND YACC - 1

UNIT - 7 LEX AND YACC - 1 COURSE MATERIAL UNIT 7: LEX AND YACC Subject: System software Subject code : Faculty: Chetankumar Naik Department of Computer Science and Engineering, JIT Davangere UNIT - 7 LEX AND YACC - 1 7.1.INTRODUCTION:

More information

1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System can be represented as shown figure below.

1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System can be represented as shown figure below. UNIT I Translator: It is a program that translates one language to another Language. Examples of translator are compiler, assembler, interpreter, linker, loader and preprocessor. Source Code Translator

More information

Implementation of Lexical Analysis

Implementation of Lexical Analysis Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation

More information

Regular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications

Regular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata

More information

CSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B-1

CSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B-1 CSEP 501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2008 1/8/2008 2002-08 Hal Perkins & UW CSE B-1 Agenda Basic concepts of formal grammars (review) Regular expressions

More information

Compiler Construction D7011E

Compiler Construction D7011E Compiler Construction D7011E Lecture 2: Lexical analysis Viktor Leijon Slides largely by Johan Nordlander with material generously provided by Mark P. Jones. 1 Basics of Lexical Analysis: 2 Some definitions:

More information

CPS 506 Comparative Programming Languages. Syntax Specification

CPS 506 Comparative Programming Languages. Syntax Specification CPS 506 Comparative Programming Languages Syntax Specification Compiling Process Steps Program Lexical Analysis Convert characters into a stream of tokens Lexical Analysis Syntactic Analysis Send tokens

More information

CSE 3302 Programming Languages Lecture 2: Syntax

CSE 3302 Programming Languages Lecture 2: Syntax CSE 3302 Programming Languages Lecture 2: Syntax (based on slides by Chengkai Li) Leonidas Fegaras University of Texas at Arlington CSE 3302 L2 Spring 2011 1 How do we define a PL? Specifying a PL: Syntax:

More information

Preparing for the ACW Languages & Compilers

Preparing for the ACW Languages & Compilers Preparing for the ACW 08348 Languages & Compilers Introductory Lab There is an Introductory Lab Just involves copying the lab task See separate Lab slides Language Roadmaps Convenient way of showing syntax

More information

CSCI-GA Compiler Construction Lecture 4: Lexical Analysis I. Hubertus Franke

CSCI-GA Compiler Construction Lecture 4: Lexical Analysis I. Hubertus Franke CSCI-GA.2130-001 Compiler Construction Lecture 4: Lexical Analysis I Hubertus Franke frankeh@cs.nyu.edu Role of the Lexical Analyzer Remove comments and white spaces (aka scanning) Macros expansion Read

More information

Lexical Considerations

Lexical Considerations Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.035, Spring 2010 Handout Decaf Language Tuesday, Feb 2 The project for the course is to write a compiler

More information

About the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. Compiler Design

About the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. Compiler Design i About the Tutorial A compiler translates the codes written in one language to some other language without changing the meaning of the program. It is also expected that a compiler should make the target

More information

TDDD55- Compilers and Interpreters Lesson 2

TDDD55- Compilers and Interpreters Lesson 2 TDDD55- Compilers and Interpreters Lesson 2 November 11 2011 Kristian Stavåker (kristian.stavaker@liu.se) Department of Computer and Information Science Linköping University PURPOSE OF LESSONS The purpose

More information

Parser Tools: lex and yacc-style Parsing

Parser Tools: lex and yacc-style Parsing Parser Tools: lex and yacc-style Parsing Version 6.11.0.6 Scott Owens January 6, 2018 This documentation assumes familiarity with lex and yacc style lexer and parser generators. 1 Contents 1 Lexers 3 1.1

More information

COMPILER DESIGN UNIT I LEXICAL ANALYSIS. Translator: It is a program that translates one language to another Language.

COMPILER DESIGN UNIT I LEXICAL ANALYSIS. Translator: It is a program that translates one language to another Language. UNIT I LEXICAL ANALYSIS Translator: It is a program that translates one language to another Language. Source Code Translator Target Code 1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System

More information

Programming in C++ 4. The lexical basis of C++

Programming in C++ 4. The lexical basis of C++ Programming in C++ 4. The lexical basis of C++! Characters and tokens! Permissible characters! Comments & white spaces! Identifiers! Keywords! Constants! Operators! Summary 1 Characters and tokens A C++

More information

Lexical Analysis and jflex

Lexical Analysis and jflex Lecture 6: Lexical Analysis and jflex Dr Kieran T. Herley Department of Computer Science University College Cork 2017-2018 KH (03/10/17) Lecture 6: Lexical Analysis and jflex 2017-2018 1 / 1 Summary Lexical

More information

UNIT II LEXICAL ANALYSIS

UNIT II LEXICAL ANALYSIS UNIT II LEXICAL ANALYSIS 2 Marks 1. What are the issues in lexical analysis? Simpler design Compiler efficiency is improved Compiler portability is enhanced. 2. Define patterns/lexeme/tokens? This set

More information

Lexical Analysis. Chapter 1, Section Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual

Lexical Analysis. Chapter 1, Section Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual Lexical Analysis Chapter 1, Section 1.2.1 Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual Inside the Compiler: Front End Lexical analyzer (aka scanner) Converts ASCII or Unicode to a stream of tokens

More information

COMPILER CONSTRUCTION LAB 2 THE SYMBOL TABLE. Tutorial 2 LABS. PHASES OF A COMPILER Source Program. Lab 2 Symbol table

COMPILER CONSTRUCTION LAB 2 THE SYMBOL TABLE. Tutorial 2 LABS. PHASES OF A COMPILER Source Program. Lab 2 Symbol table COMPILER CONSTRUCTION Lab 2 Symbol table LABS Lab 3 LR parsing and abstract syntax tree construction using ''bison' Lab 4 Semantic analysis (type checking) PHASES OF A COMPILER Source Program Lab 2 Symtab

More information

Chapter 3 -- Scanner (Lexical Analyzer)

Chapter 3 -- Scanner (Lexical Analyzer) Chapter 3 -- Scanner (Lexical Analyzer) Job: Translate input character stream into a token stream (terminals) Most programs with structured input have to deal with this problem Need precise definition

More information

Full file at

Full file at Java Programming: From Problem Analysis to Program Design, 3 rd Edition 2-1 Chapter 2 Basic Elements of Java At a Glance Instructor s Manual Table of Contents Overview Objectives s Quick Quizzes Class

More information

Scanners. Xiaokang Qiu Purdue University. August 24, ECE 468 Adapted from Kulkarni 2012

Scanners. Xiaokang Qiu Purdue University. August 24, ECE 468 Adapted from Kulkarni 2012 Scanners Xiaokang Qiu Purdue University ECE 468 Adapted from Kulkarni 2012 August 24, 2016 Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved

More information

Program Development Tools. Lexical Analyzers. Lexical Analysis Terms. Attributes for Tokens

Program Development Tools. Lexical Analyzers. Lexical Analysis Terms. Attributes for Tokens Program Development Tools lex makefiles vi and gvim ctags source level debugging diff and cmp Lexical Analyzers A lexical analyzer reads in a stream of characters as input and produces a sequence of symbols

More information

Yacc: A Syntactic Analysers Generator

Yacc: A Syntactic Analysers Generator Yacc: A Syntactic Analysers Generator Compiler-Construction Tools The compiler writer uses specialised tools (in addition to those normally used for software development) that produce components that can

More information

Syntax Analysis Part IV

Syntax Analysis Part IV Syntax Analysis Part IV Chapter 4: Bison Slides adapted from : Robert van Engelen, Florida State University Yacc and Bison Yacc (Yet Another Compiler Compiler) Generates LALR(1) parsers Bison Improved

More information

A program that performs lexical analysis may be termed a lexer, tokenizer, or scanner, though scanner is also a term for the first stage of a lexer.

A program that performs lexical analysis may be termed a lexer, tokenizer, or scanner, though scanner is also a term for the first stage of a lexer. Compiler Design A compiler is computer software that transforms computer code written in one programming language (the source language) into another programming language (the target language). The name

More information

Marcello Bersani Ed. 22, via Golgi 42, 3 piano 3769

Marcello Bersani  Ed. 22, via Golgi 42, 3 piano 3769 Marcello Bersani bersani@elet.polimi.it http://home.dei.polimi.it/bersani/ Ed. 22, via Golgi 42, 3 piano 3769 Flex, Bison and the ACSE compiler suite Marcello M. Bersani LFC Politecnico di Milano Schedule

More information