2. Syntax and Type Analysis

Size: px

Start display at page:

Download "2. Syntax and Type Analysis"

Malcolm Holmes
5 years ago
Views:

1 Content of Lecture Syntax and Type Analysis Lecture Compilers Summer Term 2011 Prof. Dr. Arnd Poetzsch-Heffter Software Technology Group TU Kaiserslautern Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 1 1. Introduction: Overview and Motivation 2. Syntax- and Type Analysis Context-Free Syntax Analysis 2.3 Context-Dependent Syntax Analysis 3. Translation to Target Language 3.1 Translation of Imperative Language Constructs 3.2 Translation of Object-Oriented Language Constructs 4. Selected Aspects of Compilers 4.1 Intermediate Languages 4.2 Optimization 4.3 Data Flow Analysis 4.4 Register Allocation 4.5 Code Generation 5. Garbage Collection 6. XML Processing (DOM, SAX, XSLT) Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 2 Educational Objectives 2. Syntax and Type Analysis Tasks of different syntax analysis phases Interaction of syntax analysis phases Specification techniques for syntax analysis Generation techniques Usage of tools Lexical analysis Context-free analysis (parsing) Context-sensitive analysis Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 3 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 4 Syntax Analysis Introduction to Syntax and Type Analysis Introduction to Syntax and Type Analysis Syntax and Type Analysis Phases Source Code Tasks of Syntax Analysis Check if input is syntactically correct Dependent on result: Error message Generation of appropriate data structure for subsequent processing Lexical analysis: Character stream token stream (or symbol stream) Context-free analysis: Token stream syntax tree Context-sensitive analysis: Syntax tree syntax tree with cross references Syntax and Type Analysis Character Stream Scanner Token Stream Parser Syntax Tree Name and Type Analysis Attributed Syntax Tree Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 5 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 6 Introduction to Syntax and Type Analysis Reasons for Separation of Phases Lexical and context-free analysis Reduced load for context-free analysis, e.g., whitespaces are not required for context-free analysis Context-free and context-sensitive analysis Context-sensitive analysis uses tree structure instead of token stream Advantages for construction of target data structure For both cases Increased efficiency Natural process (cmp. natural language) More appropriate tool support 2.1. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 7 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 8

2 (2) Tasks Break input character stream into a token stream wrt. language definition Classify tokens into token classes Representation of tokens Hashing of identifiers Conversion of constants Elimination of whitespaces (spaces, comments...) external constructs (compiler directives...) Terminology Token/symbol: a word over an alphabet of characters (often with additional information, e.g. token class, encoding, position..) Token class: a set of tokens (identifier, constants,...); correspond to terminal symbols of a context-free grammar Remark: the terms token and symbol refer to the same concept. The term token is in general used when talking about parsing technology, whereas the term symbol is used when talking about formal languages. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 9 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 10 : Example Specification Input Line 23: if ( A <= 3.14 ) B = B Token Class String Token Information Col:Row IF if 23:3 OPAR ( 23:5 ID A 72 (Hash) 23:7 RELOP <= 4 (Encoding) 23:9 FLOATCONST ,14 (Constant Value) 23:12 CPAR ) 23:16 ID B 84 (Hash) 23:20... The specification of the lexical analysis is a part of the language specification. The two parts of lexical analysis specification: Scanning algorithm (often only implicit) Specification of tokens and token classes Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 11 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 12 Examples: Scanning Standard Scan Algorithm (Concept) 1. Statement in C B = B --- A; Problem: Separation ( - - and - are tokens) Solution: Longest token is chosen, i.e, B = B -- - A; 2. Java Fragment class public { public m() {... Problem: Ambiguity (keyword, identifier) Solution: Precedence rules Scanning is often implemented as a procedure: Procedure returns next token State is remainder of input In error cases, returns the UNDEF token and updates the input Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 13 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 14 Standard Scan Algorithm (Pseudo Code) Standard Scan Algorithm (2) CharStream inputrest := input; Token nexttoken() { Token curtoken := longesttokenprefix(inputrest); inputrest:= cut(curtoken, inputrest); return curtoken; where cut is defined as if curtoken UNDEF, curtoken is removed from inputrest else inputrest remains unchanged. Token longesttokenprefix(charstream ir) { require availablechar(ir) > 0 int curlength = 1; String curprefix := prefix(curlength,ir); Token longesttoken := UNDEF; while( curlength <= availablechar(ir) && istokenprefix(curprefix) ) { if (istoken(curprefix) { longesttoken := curprefix; curlength++; curprefix := prefix(curlength,ir); return longesttoken; Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 15 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 16

3 Standard Scan Algorithm (3) Specification of Token Classes Predicates to be defined: istokenprefix: String boolean istoken: String boolean Remarks: Standard scan algorithm is used in many modern languages, but not, e.g., in FORTRAN because blanks are not special, except in literal tokens, e.g. DO 7 I = 1.25 DO 7 I is an identifier. DO 7 I = 1,25 DO is a keyword. Error cases are not handled Complete realization of longesttokenprefix is discussed later. Token classes are defined by regular expressions (REs). REs specify the set of strings, which belong to a certain token class. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 17 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 18 Regular Expressions Regular Expressions (2) Let Σ be an alphabet, i.e. an non-empty set of characters. Σ is the set of all words over Σ, ɛ is the empty word. Definition (Regular expressions, regular languages) ε is a RE and specifies the language L = {ɛ. Each a Σ is a RE and specifies the language L = {a. Let r and s be two RE specifying the languages R and S, resp. Then the following are RE and specify the language L: (r s) with L = R S (union) rs with L = {vw v R, w S (concatenation) r with {v 1... v n v i R, 0 i n (Kleene star) The language L Σ is called regular if there exists RE r defining L. Remarks: L = is not regular according to the definition, but is often considered regular. Other Operators, e.g. +,?,., [] can be defined using the basic operators, e.g. r + (r r ) r {ɛ [abd] a B d [a g] a b c d e f g Caution: Regular expressions only define valid tokens and do not specify the program or translation units of a programming language. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 19 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 20 Scanner Generator: JFlex Typical use of JFlex: sequence of regular expressions and actions (input language of scanner generator) Scanner Generator scanner program (usually in a programming language) java -jar JFlex.jar Example.jflex javac Yylex.java Actions are written in Java Examples : 1. Regular expression in JFlex [a-za-z_0-9] [a-za-z_0-9] * 2. JFlex input with abbreviations = [0-9] BU = [a-za-z_] BU = [a-za-z_0-9] %% {BU{BU* { anaction(); Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 21 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 22 A Complete JFlex Example Scanner Generators enum Token { DO, DOUBLE, IDENT, FLOATCONST, STRING; %% %line %column %debug %type Token // declare token type = [0-9] BU = [a-za-z_] BU = [a-za-z_0-9] = [a-za-z_0-9?][. t...] WhiteSpace = [ tn] %% {WhiteSpace { "double" { return Token.DOUBLE; "do" { return Token.DO; {BU{BU* { return Token.IDENT; {+.{+ { return Token.FLOATCONST; "({ ")*" { return Token.STRING; <<EOF>> { System.out.println("FINISHED"); return null; Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 23 Scanner generation uses the equivalence between Regular expressions Non-deterministic finite automata (NFA) Deterministic finite automata (DFA) Construction methods is based in two steps: Regular expressions NFA NFA DFA Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 24

4 l Definition of NFA Definition (Non-deterministic Finite Automaton) A non-deterministic finite automaton is defined as a 5-tuple 1. Schritt: Reguläre Ausdrücke NEA M = (Σ, Q,, q 0, F) where 1. Schritt: Reguläre Ausdrücke NEA Übersetzungsschema: Σ is the input alphabet QPrinzip: is the Übersetzungsschema: Konstruiere of states für jeden regulären Teilausdruck q 0 Prinzip: Q is the 1. NEA Schritt: Konstruiere initial mit state genau Reguläre für einem jeden Ausdrücke regulären Start- und NEA Teilausdruck Endzustand, der F Q is the NEA die set ofmit gleiche final genau Sprache states einem Start- akzeptiert. und Endzustand, Übersetzungsschema: Q Σ der {ɛ die Q gleiche is the transition Sprache akzeptiert. relation. Prinzip: Konstruiere für jeden regulären Teilausdruck NEA mit genau einem Start- und Endzustand, s der 0 die gleiche Sprache akzeptiert. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 25 a a a Regular Expressions a s a 0 NFA f (2) 1 R f s 1 1 R f 1 (r s) s (r s) s 0 1 R f 1 s 0 (r s) s 2 S f 2 r* r* r* a s 2 S f s 2 2 f 2 s s 1 R f 1 2 S f 1 R R f 2 1 s 2 S f 2 s 1 f 1 s 1 R f 1 s 1 R f 1 s 1 f 1 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis Schritt: Reguläre Implementation of Ausdrücke Scanners NEA Regular Expressions NFA Übersetzungsschema: Prinzip: Konstruiere für jeden regulären Teilausdruck Principle: For each regular sub-expression, construct NFA with one NEA mit genau einem Start- und Endzustand, start and end state that accepts the same language. der die gleiche Sprache akzeptiert. a r* s a 0 s 1 R f 1 (r s) Implementation s 2 of Scanners S f 2 Example: Construction of NFA Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 26 Übersetzung am Beispiel von Folie 41: LZ, TAB s 1 s d 2 s o 3 s 4 s 5 s 6 s 7 s 8 s 9 s 10 s 11 e b u o d s 1 R f 1 s 2 S f 2 BU s 13 s 1 R f 1 BU s 12 s 17 s. 14 s 15 s 16 s 20 s 21 s 26 s 25 s 18 s 19 s 22 s 23 s 24 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 28 ɛ-closure Longest Token Prefix with NFA Function closure computes the ɛ-closure of a set of states s 1,..., s n. Definition (ɛ-closure) For an NFA M = (Σ, Q,, q 0, F) and a state q Q, the ɛ-closure of q is defined by ɛ-closure(q) = {p Q p reachable from q via ɛ-transitions For S Q, the ɛ-closure of S is defined by ɛ-closure(s) = s S ɛ-closure(s) Token longesttokenprefix(char[] ir) { // length(ir) > 0 StateSet curstate := closure( {s0 ); int curlength := 0; int tokenlength := undef; while (curlength <= length(ir) && isemptyset(curstate) ) { if (contains(curstate,finalstate)) { tokenlength := curlength; curlength++; curstate := closure(successor(curstate,ir[curlength])); return token(prefix(ir,tokenlength)); Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 29 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 30 Longest Token Prefix with NFA (2) NFA DFA Remark: Problem of ambiguity: If there are more than one token matching the longest input prefix, procedure token nondeterministically returns one of them. Principle: For each NFA, a DFA can be constructed that accepts the same language. (In general, this does not hold for NFA with output.) Properties of DFA: No ɛ-transitions Transitions are deterministic given the input char Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 31 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 32

5 l NFA DFA (2) NFA DFA (3) Definition (Deterministic Finite State Automaton) A deterministic finite automaton is defined as a 5-tuple where Σ is the input alphabet Q is the set of states q 0 Q is the initial state F Q is the set of final states M = (Σ, Q,, q 0, F) : Q Σ Q is the transition function. Construction: (according to John Myhill) The States of the DFA are subsets of NFA states (powerset construction). Subsets of finite sets are also finite. The start state of the DFA is the ɛ-closure of the NFA start state The final states of the DFA are the sets of states that contain an NFA final state. The successor state of a state S in the DFA under input a is obtained by computing all successors p of q S under a in the NFA and adding the ɛ-closure of p Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 33 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 34 NFA DFA (4) NFA DFA (5) If working with character classes (e.g. [a-f]), characters and character classes at outgoing transitions must be disjoint. Completion of automaton for error handling: Insert additional (final) state (nt) For each state, add a transition for each character for which no outgoing transition exists to the nontoken state. Definition (DFA for NFA) Let M = (Σ, Q,, q 0, F) be a NFA. Then, the DFA M corresponding to the NFA M is defined as M = (Σ, Q,, q 0, F ) where the set of states is Q P(Q), power set of Q the initial state q 0 is the ɛ-closure of q 0 the final states are F = {S Q S F (S, a) = ɛ-closure({p (q, a, p), q S) for all a Σ. Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 35 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 36 Example: DFA s 11,13 e s 10,13 LZ, TAB s 1 BU{e s 9,13 b BU BU{l s 8,13 u BU{b s 4,7,13 Longest Token Prefix with DFA Wg. Übersichtlichkeit Kanten zu ks nur angedeutet. Transitions to nt sketched. nt ks o LZ, TAB s 13 BU{u s3,6,13 BU BU{o BU{d d,1,2,5,12,14,18 s 17 s 26. s 15 s 16 s 19,20,21,22,25 s 19,20,22,25 s19,20,21,22,23,25 s 19,20,22,24,25,26 Token longesttokenprefix(char[] ir) { // length(ir) > 0 State curstate : = StartState; int curlength := 0; int tokenlength := undef; while (curlength <= length(ir) && curstate = nt) if (curstate is FinalState) { tokenlength := curlength; curlength++; curstate := successor(curstate,ir[curlength])); return token(prefix(ir,tokenlength)); Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 37 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 38 Longest Token Prefix with DFA (2) Longest Token Prefix with DFA (3) Remarks: Computation of closure at construction time, not at runtime. (Principle: Do as much statically as you can) Problem of ambiguity still not solved. However, many scanner generators allows the user to control which token is returned. For example, JFlex returns the token of the first rule in the JFlex file that matches the longest input prefix. Implementation Aspects: Constructed DFA can be minimized. Input buffering is important: often use of cyclic arrays (caution with maximal token length, e.g. in case of comments) Encode DFA in table Choose suitable partitioning of alphabet in order to reduce number of transitions (i.e. size of table) Interface with parser: usually parser asks proactively for next token Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 39 Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 40

6 Recommended Reading Wilhelm, Maurer: Chap. 7, pp (More theoretical) Appel: Chap 2, pp (More practical) Additional Reading: Aho, Sethi, Ullman: Chap. 3 (very detailed) Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 41

Syntax and Type Analysis

Syntax and Type Analysis Lecture Compilers Summer Term 2011 Prof. Dr. Arnd Poetzsch-Heffter Software Technology Group TU Kaiserslautern Prof. Dr. Arnd Poetzsch-Heffter Syntax and Type Analysis 1 Content