Outline Implementation of Lexical nalysis Specifying lexical structure using regular expressions Finite automata Deterministic Finite utomata (DFs) Non-deterministic Finite utomata (NFs) Implementation of regular expressions RegExp NF DF Tables 2 Notation For convenience, we will use a variation (we will allow user-defined abbreviations) in regular expression notation Union: + B B Option: +? Range: a + b + + z [a-z] Excluded range: complement of [a-z] [^a-z] Regular Expressions in Lexical Specification Last lecture: a specification for the predicate s L(R) But a yes/no answer is not enough! Instead: partition the input into tokens We will adapt regular expressions to this goal 3 4
Regular Expressions Lexical Specifications. Select a set of tokens Integer, Keyword, Identifier, LeftPar,... Regular Expressions Lexical Specifications 3. Construct R, a regular expression matching all lexemes for all tokens 2. Write a regular expression (pattern) for the lexemes of each token Integer = digit + Keyword = if + else + Identifier = letter (letter + digit)* LeftPar = ( R = Keyword + Identifier + Integer + = R + R 2 + R 3 + Facts: If s L(R) then s is a lexeme Furthermore s L(R i ) for some i This i determines the token that is reported 5 6 Regular Expressions Lexical Specifications 4. Let input be x x n (x... x n are characters) For i n check x x i L(R)? 5. It must be that x x i L(R j ) for some j (if there is a choice, pick a smallest such j) 6. Remove x x i from input and go to previous step How to Handle Spaces and Comments?. We could create a token Whitespace Whitespace = ( + \n + \t ) + We could also add comments in there n input " \t\n 555 " is transformed into Whitespace Integer Whitespace 2. Lexical analyzer skips spaces (preferred) Modify step 5 from before as follows: It must be that x k... x i L(R j ) for some j such that x... x k- L(Whitespace) Parser is not bothered with spaces 7 8
mbiguities () There are ambiguities in the algorithm How much input is used? What if x x i L(R) and also x x K L(R) Rule: Pick the longest possible substring The maximal munch mbiguities (2) Which token is used? What if x x i L(R j ) and also x x i L(R k ) Rule: use rule listed first (j if j < k) Example: R = Keyword and R 2 = Identifier if matches both Treats if as a keyword not an identifier 9 Error Handling What if No rule matches a prefix of input? Problem: Can t just get stuck Solution: Write a rule matching all bad strings Put it last Lexical analysis tools allow the writing of: R = R +... + R n + Error Token Error matches if nothing else matches Summary Regular expressions provide a concise notation for string patterns Use in lexical analysis requires small extensions To resolve ambiguities To handle errors Good algorithms known (next) Require only single pass over the input Few operations per character (table lookup) 2
Regular Languages & Finite utomata Basic formal language theory result: Regular expressions and finite automata both define the class of regular languages. Thus, we are going to use: Regular expressions for specification Finite automata for implementation (automatic generation of lexical analyzers) Finite utomata finite automaton is a recognizer for the strings of a regular language finite automaton consists of finite input alphabet Σ set of states S start state n set of accepting states F S set of transitions state input state 3 4 Finite utomata Transition s a s 2 Is read In state s on input a go to state s 2 Finite utomata State Graphs state The start state If end of input (or no transition possible) If in accepting state accept Otherwise reject n accepting state transition a 5 6
Simple Example finite automaton that accepts only nother Simple Example finite automaton accepting any number of s followed by a single lphabet: {,} 7 8 nd nother Example lphabet {,} What language does this recognize? nd nother Example lphabet still {, } The operation of the automaton is not completely defined by the input On input the automaton could be in either state 9 2
Epsilon Moves nother kind of transition: -moves Machine can move from state to state B without reading input B Deterministic and Non-Deterministic utomata Deterministic Finite utomata (DF) One transition per input per state No -moves Non-deterministic Finite utomata (NF) Can have multiple transitions for one input in a given state Can have -moves Finite automata have finite memory Enough to only encode the current state 2 22 Execution of Finite utomata DF can take only one path through the state graph Completely determined by input NFs can choose Whether to make -moves Which of multiple transitions for a single input to take cceptance of NFs n NF can get into multiple states Input: Rule: NF accepts an input if it can get in a final state 23 24
NF vs. DF () NFs and DFs recognize the same set of languages (regular languages) NF vs. DF (2) For a given language the NF can be simpler than the DF DFs are easier to implement There are no choices to consider NF DF DF can be exponentially larger than NF (contrary to what is shown in the above example) 25 26 Regular Expressions to Finite utomata High-level sketch NF Regular Expressions to NF () For each kind of reg. expr, define an NF Notation: NF for regular expression M M Regular expressions Lexical Specification DF Table-driven Implementation of DF i.e. our automata have one start and one accepting state For For input a a 27 28
Regular Expressions to NF (2) For B Regular Expressions to NF (3) For * B For + B B 29 3 Example of Regular Expression NF conversion NF to DF. The Trick Consider the regular expression Simulate the NF (+)* Each state of DF The NF is = a non-empty subset of states of the NF Start state = the set of NF states reachable through -moves from NF start state B C D E F G H I J dd a transition S a S to DF iff S is the set of NF states reachable from any state in S after seeing the input a considering -moves as well 3 32
NF to DF. Remark NF to DF Example n NF may be in many states at any time How many different states? If there are N states, the NF must be in some subset of those N states How many subsets are there? 2 N - = finitely many B BCDHI C D E F FGBCDHI EJGBCDHI G H I J 33 34 Implementation Table Implementation of a DF DF can be implemented by a 2D table T One dimension is states Other dimension is input symbols For every transition S i a S k define T[i,a] = k S T U DF execution If in state S i and input a, read T[i,a] = k and skip to state S k Very efficient S T U T T U U T U 35 36
Implementation (Cont.) NF DF conversion is at the heart of tools such as lex, ML-Lex or flex But, DFs can be huge In practice, lex/ml-lex/flex-like tools trade off speed for space in the choice of NF and DF representations Theory vs. Practice Two differences: DFs recognize lexemes. lexer must return a type of acceptance (token type) rather than simply an accept/reject indication. DFs consume the complete string and accept or reject it. lexer must find the end of the lexeme in the input stream and then find the next one, etc. 37 38