The Lexical Structure of Verdi TR Mark Saaltink. Release date: July 1994

Transcription

1 The Lexical Structure of Verdi TR Mark Saaltink Release date: July 1994 ORA Canada 267 Richmond Road, Suite 100 Ottawa, Ontario K1Z 6X3 CANADA

2 Verdi Compiler Project TR This report formally denes the lexical structure of Verdi [1, 3], using the Z notation [4]. This formulation of lexical structure is similar to the denition in the formal denition of Turing [2]. The grammar of Verdi, like that of most programming languages, is most conveniently described using two distinct phases: lexical analysis and parsing. Verdi programs are composed of characters. Lexical analysis transforms this sequence of characters to a sequence of tokens. This sequence of tokens is then parsed according to a context-free grammar. 1 Changes to Verdi The original description of Verdi had a lexical structure that was ambiguous; the string "\141" could be interpreted as "a" or as "141", depending on whether the escape was interpreted as a numeric representation or as a single character escape. This ambiguity has been removed by restricting the characters that may appear in a single character escape. 2 Characters Verdi programs are written in the ASCII character set. The set CHAR comprises some representation of these characters. [CHAR] Function char code is a bijection between CHAR and small numbers; for a character c, char code(c) is the ASCII code of c. char code : CHAR! char val : ! CHAR char val = char code 01 We will write characters inside single quotes, for example `a' denotes the lower case letter \a", which is also the value of char val(97). We use the usual names for the common formatting characters: cr; lf ; ; sp; tab : CHAR cr = char val(13) lf = char val(10) = char val(12) sp = char val(32) tab = char val(9) Several sets of characters are used in the denitions below. The visible characters, with codes between 33 and 126 inclusive, have associated glyphs; the graphic characters also include space. The blank characters are formatting characters that leave \white space". Carriage return and line feed characters are used to end lines. Visible; Graphic; Blank; EndLine : P CHAR Visible = char val(j j) Graphic = char val(j j) = Visible [ fspg Blank = fcr; lf ; ; sp; tabg EndLine = fcr; lf g

3 Verdi Compiler Project TR Several kinds of digits are used in the description of numerals: BinaryDigit; OctalDigit; Digit; HexDigit : P CHAR BinaryDigit = f`0'; `1'g = char val(j j) OctalDigit = f`0'; `1'; `2'; `3'; `4'; `5'; `6'; `7'g = char val(j j) Digit = OctalDigit [ f`8'; `9'g = char val(j j) HexDigit = Digit [ f`a'; `b'; `c'; `d'; `e'; `f'; `A'; `B'; `C'; `D'; `E'; `F'g The escape characters have special uses in character literals and strings: EscapeChar : P CHAR EscapeTable : CHAR 7! CHAR EscapeChar = dom EscapeTable EscapeTable = f`b' 7! char val(8); `d' 7! char val(127); `l' 7! lf ; `n' 7! lf ; `p' 7! ; `r' 7! cr; `s' 7! sp; `t' 7! tab; `"' 7! `"'; `\' 7! `\'g 3 Lexical Units Two kinds of lexical units are used: tokens and separators. LexicalUnit b= Token [ Separator 3.1 Tokens Tokens are certain sequences of characters. There are four classes of tokens in Verdi: numerals, identiers, character literals, and strings; in addition, the left and right parentheses are special tokens. Token : P(seq CHAR) Token = Numeral [ Identier [ Character literal [ String literal [ fh`('i; h`)'ig 3.2 Attributes We will associate an \attribute" with each token. This attribute is used in the description of the semantics of Verdi, or else is used to dene the abstract syntax corresponding to the concrete syntax. The attribute of a numeral is the number it represents. The attribute of an identier is a name, which has been normalized (by converting upper case letters to lower case). The attribute of a character literal is a character. The attribute of a string is a sequence of characters. Parentheses have no attributes. Attribute :== numhhzii j identhhseq CHARii j charhhcharii j stringhhseq CHARii j openparen j closeparen Function attr gives the attribute of a token. This function is composed in the obvious way from functions determining the attribute value for each type of token. These individual functions will be dened below.

4 Verdi Compiler Project TR attr : Token! Attribute attr = (num num attr) [ (ident ident attr) [ (char char attr) [ (string string attr) [ fh`('i 7! openparen; h`)'i 7! closepareng 3.3 Numerals Numeral : P(seq CHAR) Numeral = seq 1 Digit [ f s : seq 1 Digit h`-'i a s g [ f s : seq 1 BinaryDigit; r : f`b'; `B'g h`#'; ri a s g [ f s : seq 1 OctalDigit; r : f`o'; `O'g h`#'; ri a s g [ f s : seq 1 HexDigit; r : f`h'; `H'g h`#'; ri a s g The value of a numeral is calculated in the obvious way. value in radix : Z 2 seq HexDigit! Z digit value : HexDigit! Z value in radix(r; hi) = 0 value in radix(r; s a hdi) = r 3 value in radix(r; s) + digit value(d) digit value = f`0' 7! 0;... ; `9' 7! 9; `a' 7! 10; `A' 7! 10;... ; `f' 7! 15; `F' 7! 15g num attr : Numeral! Z 8 s : seq 1 Digit num attr(s) = value in radix(10; s) 8 s : seq 1 Digit num attr(h`-'i a s) = 0value in radix(10; s) 8 s : seq 1 BinaryDigit; r : f`b'; `B'g num attr(h`#'; ri a s g = value in radix(2; s) 8 s : seq 1 OctalDigit; r : f`o'; `O'g num attr(h`#'; ri a s g = value in radix(8; s) 8 s : seq 1 HexDigit; r : f`h'; `H'g num attr(h`#'; ri a s g = value in radix(16; s) 3.4 Identiers Identier : P(seq CHAR) Identier = (seq 1 X ) n Numeral where X = Visible n f`('; `)'; `"'; `''; ``'; `;'; `#'; g It is not immediately obvious that this lexical class can be dened by a regular expression. Clearly, though, seq 1 X is denable by a regular expression, as is Numeral. Furthermore, the set of regular languages is closed under complementation and intersection. So, we can conclude that Identier can be specied by a regular expression. Once we think to look for it, it is easy to nd: Identier ::= `-' j XV 3 j (D j `-')D 3 NV 3 where V = Visible n f`('; `)'; `"'; `''; ``'; `;'; `#'; g D = Digit N = V n Digit X = N n f`-'g

5 Verdi Compiler Project TR The attribute associated with an identier is derived by converting all letters in the identier to lower case. (We could equally well convert them to upper case; all that matters is that case is normalized.) ident attr : Identier! Identier lower : CHAR! CHAR ident attr = i : Identier (lower i) lower = (id CHAR) 8 f`a' 7! `a';...; `Z' 7! `z'g 3.5 Character Literals Character literal : P(seq CHAR) Escape : P(seq Char) Character literal = f c : Graphic j c 6= `\' h`''; c; `''i g [ f s : Escape h`''i a s a h`''i g Escape = f c : EscapeChar h`\'; ci g [ f s : seq OctalDigit j #s = 3 h`\'i a s g char attr : Character literal! CHAR escape value : Escape! CHAR 8 c : Graphic char attr(h`''; c; `''i) = c 8 s : Escape char attr(h`''i a s a h`''i) = escape value(s) 8 c : EscapeChar escape value(h`\'; ci) = EscapeTable(c) 8 s : seq OctalDigit j #s = 3 escape value(h`\'i a s) = char val(value in radix(8; s)) 3.6 String Literals String literal : P(seq CHAR) String element : P(seq CHAR) String literal = fs : seq String element h`"'i a ( a = s) a h`"'i g String element = f c : (Graphic n f`"'; `\'g) hci g [ Escape A given string literal can be divided into elements in only one way: Lemma 1 Suppose e; e 0 : seq String element, and suppose a = e = a = e 0. Then e = e 0. This property makes it easy to nd the attribute of a string literal. Each element is interpreted as (the body of) a character literal: string attr : String literal! seq CHAR element char : String element! CHAR 8 s : String literal; x : seq String element j s = h`"'i a ( a = x) a h`"'i string attr(s) = element char x 8 e : String element element char(e) = char attr(h`''i a e a h`''i)

6 Verdi Compiler Project TR Separators Separators are either whitespace or comments. Separator : P(seq CHAR) Whitespace : P(seq CHAR) Comment : P(seq CHAR) Separator = Whitespace [ Comment Whitespace = f c : Blank hci g Comment = f c : EndLine; s : seq(char n EndLine) h`;'i a s a hci g 4 Tokenization A sequence of characters is tokenized by rst dividing it into tokens and separators, then throwing away the separators. The rst stage introduces possible ambiguities: the sequence h`1'; `2'i can be divided into two one-digit numbers, or into a single number; similarly, the sequence h`a'; `b'i can be divided into a single identier, or two identiers. A simple principle (called maximum munch in [2]) resolves such problems: each lexical unit must be as long as possible. To put it another way, a sequence of characters is divided into two or more tokens only if necessary. This principle can be formalized as a property of a sequence of lexical units: each unit in the sequence must be the longest possible lexical unit that is a prex of the input. This is easily described formally (note that the input is obtained by concatenating the lexical units in the sequence): Maximal : P(seq LexicalUnit) 8 u : seq LexicalUnit Maximal u, (#u > 0 ) Maximal (tail(u)) ^ 8 t : LexicalUnit j t a = u t head(u)) Tokenization is a relation between an input sequence of characters and an output sequence of tokens, dened according to the above description: Tokenize : seq CHAR $ seq Token Tokenize = f u : seq LexicalUnit j Maximal u ( a = u; u TOKEN ) g Tokenize is, in fact, a partial function. We rst show that the maximal munch principle uniquely determines the lexical tokens comprising a sequence (this is also proven in [2]): Lemma 2 Suppose u; u 0 : seq LexicalUnit are both maximal, and a = u = a = u 0. Then u = u 0. We can prove this by inducting on the length of u. Suppose rst that a = u is empty. Since <>62 LexicalUnit, u 0 must be empty as well. Therefore, we may assume that both u and u 0 are nonempty. Put h = head(u) and h 0 = head(u 0 ). Then h and h 0 are lexical units, and By the maximality of u 0, h a = u = a = u 0 : 8 t : LexicalUnit j t a = u 0 t head(u 0 ):

7 Verdi Compiler Project TR Therefore, instantiating t to h, we have h head(u 0 ) = h 0 : Similarly, since u is maximal, we can show h 0 h. Therefore h = h 0. By the denition of maximality, both tail(u) and tail(u 0 ) are maximal. Moreover, since a = u = a = u 0 and head(u) = head(u 0 ), we have a = tail(u) = a = tail(u 0 ). So by the induction hypothesis, tail(u) = tail(u 0 ), and thus u = u 0. Lemma 3 Tokenize is a function. Suppose (s; t) 2 Tokenize and (s; t 0 ) 2 Tokenize. We must show t = t 0. By the denition of Tokenize and the hypotheses, we have and 9 u : seq LexicalUnit j Maximal u s = a = u ^ t = u Token 9 u 0 : seq LexicalUnit j Maximal u 0 s = a = u 0 ^ t 0 = u 0 Token But Lemma 2 shows that u = u 0, so clearly t = t 0. We should make sure that the maximum munch principle does not resolve too much ambiguity. In fact, this is so: the only possible ambiguities it resolves are as in the above examples, where identiers or numerals are arbitrarily divided: Lemma 4 Suppose t and t 0 are tokens, t 6= t 0, and t t 0. Then t and t 0 are both in the set Identier [ Numeral. The proof is a rather tedious consideration of cases. References [1] Dan Craigen. Reference manual for the language Verdi. Technical Report TR , Odyssey Research Associates, February [2] R. C. Holt, P. A. Matthews, J. A. Rosselet, and J. R. Cordy. The Turing Programming Language: Design and Denition. University of Toronto (Draft of June 1985). [3] Mark Saaltink. A formal description of Verdi. Technical Report TR a, Odyssey Research Associates, November [4] J. M. Spivey. The Z Notation: A Reference Manual. Prentice Hall, 1989.