Compilers: table driven scanning Spring 2008

Transcription

1 Compilers: table driven scanning Spring

2 Contents Transition diagrams (3.4.1) Sketch implementation (3.4.4) Table-driven scanner (3.8) Object-oriented implementation 2

3 Transition diagrams (ALSU, 3.4.1) Scanners for many languages can be expressed as deterministic finite automatons (DFA) A transition diagram has a collection of nodes (states) and edges (transitions). Certain states are said to be accepting, which indicates that a lexeme token has been found. 3

4 Sketch implementation (Fig. 3.18) TOKEN getrelop() { TOKEN rettoken = new(relop); while(1) { switch(state) { case 0: c = nextchar(); if(c == < ) state = 1; else fail(); break; case 1: c = nextchar(); if(c == = ) state = 2; else if(c == > ) state = 3; else state = 4; break; case 2:...; case 3:... case 4: retract(); rettoken.attribute = LT; return rettoken; 4

5 Evaluation of the implementation Manually translating all transitions to a big switchcase -construct is straightforward Negative impact on readability and modifiability Think about lexemes with common prefix, e.g. > vs. >> vs. >>> ALSU Figure 3.13 handles this problem with retracting accepting states (marked with an asterisk (*) near the state action) A commonly used solution is to automatically generate the states and transitions The name lexer comes from the tool Lex A driver executes the transition table, which contains the states and transitions in a data structure. 5

6 Object-oriented scanner table Let s define an object-oriented scanner We start with an abstract class TableAction The scanner data structure is composed of objects which subclass TableAction The driver reads the next character from the input stream Depending on current state number, a TableAction object is retrieved from the scanner table structure The read character is passed with the current scanner state to the TableAction object, which modifies the scanner state 6

7 Scanner table with a driver A scantable<character, TableAction[]> The driver algorithm: while(true) { Token token = null; while(token == null) { int curchar = input.read(); TableAction[] tbl = scantable.get(curchar); token = tbl[state.currentstate].action(state, curchar); Subclasses of TableAction could be e.g. MoveAction; to change to a new state RecognizeAction; to recognize a lexeme SkipAction; to signal that the character shouldn t be considered 7

8 Scanner state Scanner state object carries state information between the driver and the actions class ScannerState { PushbackInputStream in = null; List<Character> pushback = new LinkedList<Character>(); StringBuffer image = new StringBuffer(); int currentstate, rememberedstate = 0; public void reset() { currentstate = rememberedstate = 0; pushback.clear(); image = new StringBuffer(); 8

9 Move action static class MoveAction extends TableAction { int newstateindex; // new state which we'll change upon this action public MoveAction(int outstate) { this.newstateindex = outstate; public Token action(scannerstate state, int curchar) { if(tokentable[newstateindex]!= null) { // this could be a final state state.rememberedstate = newstateindex; state.pushback.clear(); else { state.pushback.add( (char) curchar ); state.image.append( (char) curchar ); state.currentstate = newstateindex; return null; 9

10 Recognize action static class RecognizeAction extends TableAction { public RecognizeAction() { public Token action(scannerstate state, int curchar) throws IOException { state.in.unread(curchar); return tokentable[state.currentstate]; 10

11 Skip action static class SkipAction extends TableAction { public SkipAction() { public Token action(scannerstate state, int curchar) { return null; 11

12 Static initialization of the scanner tables (1/2) Depending on the scanner definition, we fill in a static data structure. Define an enumeration of different token types: For each accepting state, define the corresponding token: static Token[] tokentable = new Token[]{ // state number to token map ; enum Token { LE, NE, LT, EQ, GE, GT null, // state 0 Token.LT, Token.LE, Token.NE, // states 1-3 Token.EQ, Token.GT, Token.GE, // states

13 Static initialization of the scanner tables (2/2) tbl = new TableAction[MAX_STATE]; scantable.put(' ', tbl); scantable.put('\n', tbl); scantable.put('\t', tbl); tbl[0] = new SkipAction(); tbl[1] = new RecognizeAction(); tbl[2] = new RecognizeAction(); tbl[3] = new RecognizeAction(); tbl[4] = new RecognizeAction(); tbl[5] = new RecognizeAction(); tbl[6] = new RecognizeAction(); // actions for character '<' tbl = new TableAction[MAX_STATE]; scantable.put('<', tbl); tbl[0] = new MoveAction(1); tbl[1] = new RecognizeAction(); tbl[2] = new RecognizeAction(); tbl[3] = new RecognizeAction(); tbl[4] = new RecognizeAction(); tbl[5] = new RecognizeAction(); tbl[6] = new RecognizeAction(); // actions for character '>' tbl = new TableAction[MAX_STATE]; scantable.put('>', tbl); tbl[0] = new MoveAction(5); tbl[1] = new MoveAction(3); tbl[2] = new RecognizeAction(); tbl[3] = new RecognizeAction(); tbl[4] = new RecognizeAction(); tbl[5] = new RecognizeAction(); tbl[6] = new RecognizeAction(); // actions for character '=' tbl = new TableAction[MAX_STATE]; scantable.put('=', tbl); tbl[0] = new MoveAction(4); tbl[1] = new MoveAction(2); tbl[2] = new RecognizeAction(); tbl[3] = new RecognizeAction(); tbl[4] = new RecognizeAction(); tbl[5] = new MoveAction(6); tbl[6] = new RecognizeAction(); 13

14 Closing This scanner can recognize the relational operators <, <=, <>, =, > and >= Full Java source is available at Modifiable by changing the data structures that define the scanner A data structure can also be analysed by other tools e.g. to generate a picture of the corresponding DFA 14