Interpreter. Scanner. Parser. Tree Walker. read. request token. send token. send AST I/O. Console

Transcription

1 Scanning 1

2 read Interpreter Scanner request token Parser send token Console I/O send AST Tree Walker 2

3 Scanner This process is known as: Scanning, lexing (lexical analysis), and tokenizing This is the first step for any compiler or interpreter A scanner takes in a raw source and turns it into lexemes A lexeme is the smallest grouping of characters that represents something useful/meaningful Generating/Designing a Scanner Look at your language Find all of your lexemes Lexemes are the smallest meaningful grouping of characters You also must respect the maximal munch rule Write a regular expression for each lexeme All of these expressions together create a lexical grammar Generate or Design a scanner Use Lex or Flex or JFlex or to automatically generate a scanner Write a scanner by hand (known as the Ad-hoc method) 3

4 Lexeme categories Single-character punctuators (+, ; - ( } etc.) Multi-character punctuators (== <= etc.) Comments Literals (strings and numbers) Reserved keywords (while, for, let, int, etc.) Identifiers Ad-Hoc Scanners For this class, we will be writing an ad-hoc scanner This has the benefit of not hiding any of the details This has the drawback of not hiding any of the details An ad-hoc scanner is basically a match expression on steroids Or a switch statement if your in C++ Or branching if-statements if your in some other language pretty much a switch statement with delusions of grandeur. 4

5 Ad-Hoc Scanner // Read two values read A read B // Add them sum := A + B assign --> ':' '= plus --> '+ minus --> '- times --> '* // Print stuff write sum div --> '/ write sum / 2 lparen --> '( rparen --> ') id --> letter (letter digit)* except for 'read' and 'write number --> digit digit* digit* ('.' digit digit '.') digit* letter --> a..z A..Z digit --> 0..9 Writing a scanner: 1. Find all of your lexemes 2. Write a regular expression for each lexeme 3. Write a large match expression Ad-Hoc Scanner assign --> ':' '= plus --> '+ minus --> '- comment --> '/' '*' (non-* non-/)* '*'+ '/ times --> '* // (non-newline)* newline div --> '/ lparen --> '( rparen --> ') id --> letter (letter digit)* except for 'read' and 'write number --> digit digit* digit* ('.' digit digit '.') digit* letter --> a..z A..Z digit -->

6 Writing a scanner: 1. Find all of your lexemes 2. Write a regular expression for each lexeme 3. Write a large match expression 6

7 Generating a Scanner (using tools) 1. Write REs by hand 2. Pass REs to a generator 1. Build NFA from REs 2. Build DFA from NFA 3. Minimize DFA 3. Add any special logic Distinguish reserved keywords from identifiers Save text of some tokens (which ones?) Save location in text file (why?) Regular Expressions Part of formal language theory Make sure you know the difference between theory and implementation! A sequence of characters that defines a search pattern Regular expressions describe regular languages A bunch of regular expressions describe a regular grammar 7

8 Language Theory Rule/Expression: a single rule Grammar: a set of rules Language: a set of strings of symbols that is defined by a grammar A grammar can be used to define a language A program that implements a language can be called a recognizer Recognizers are implemented as automatons The type of automaton needed depends on the type of language Chomsky Hierarchy Type-0: Turing machine Type-1: Linear bounded automaton Type-2: Pushdown automaton Type-3: Finite state automaton 8

9 First Step: Write REs by hand What is an RE? A single character The empty string, denoted by e Two REs can be combined with concatenation Choose between two REs with alternation A regular expression followed by the Kleene star * Build the REs so that you can identify your language's tokens REs a b a OR b a b a followed by b (a b) c a OR b followed by c a* zero of more a a? zero of one of a a+ one or more a [a..z] (or [a-z]) all lowercase letters in range [A..Z] all capital letters in range [0..9] all digits in range 9

10 Examples a b* (a b)(a b) a a*b What are the possible strings in these languages? Example (what are the REs) The language of all strings of a s and b s that start and end with an a. Language of all lowercase letters (zero of more of each letter) in alphabetical order. 10