Case study: leical analysis Leical analysis converts u sequences of characters u into u sequences of tokens u tokens are also called words or leemes F

Transcription

1 Lecture 5 0

2 Case study: leical analysis Leical analysis converts u sequences of characters u into u sequences of tokens u tokens are also called words or leemes For us, a token will be one of: u a number (sequence of digits) u an identier (sequence of letters or digits starting with a letter) u a `special symbol' such as +, *, <, ==> or ++ u special symbols are specied by a table { see later 1

3 Numbers and letters A number is a sequence of digits <= is overloaded and can be applied to strings u suppose and y are single-character strings u then <=y just tests whether the ASCII code of is less then or equal to the ASCII code of y fun IsDigit = "0" <= andalso <= "9"; > val IsDigit = fn : string -> bool ASCII codes of lower case letters are adjacent ASCII codes of upper case letters are adjacent fun IsLetter = ("a" <= andalso <= "z") orelse ("A" <= andalso <= "Z"); > val IsLetter = fn : string -> bool 2

4 Separators Separators are spaces, newlines and tabs fun IsSeparator = ( = " " orelse = "\n" orelse = "\t"); > val IsSeparator = fn : string -> bool Characters that are not digits, letters or separators are assumed to be special symbols u multi-character special symbols are considered later Input a list of single-charater strings u leical analysis converts input to a token list 3

5 Special case: only numbers Suppose input just consists of numbers separated by separators Leical analysis for just this case needs to: u repeatedly remove digits until a non-number is reached u then implode the removed characters into a token u and add that to the list of tokens GetNumber takes a list, l say, of single-character strings and returns a pair consisting of u a string representing a number consisting of all the digits in l up to the rst non-digit u the remainder of l after these digits have been removed GetNum uses an auiliary function GetNumAu u GetNumAu has an etra argument buf for accumulating a (reversed) list of characters making up the number 4

6 GetNumAu and GetNum fun GetNumAu buf [] = (implode(rev buf), []) GetNumAu buf (l as (::l')) = if IsDigit then GetNumAu (::buf) l' else (implode(rev buf),l); > val GetNumAu = > fn > : string list -> string list -> string * string list GetNumAu ["a","b","c"] ["1","2","3"," ","4","5"]; > val it = > ("cba123",[" ","4","5"]) : string * string list Then GetNum is simply dened by: val GetNum = GetNumAu []; > val GetNum = fn : string list -> string * string list GetNum ["1","2","3"," ","4","5"]; > val it = ("123",[" ","4","5"]) : string * string list GetNum ["a","0","1"]; > val it = ("",["a","0","1"]) : string * string list Anomalous return of "" ed later Could localise denition of GetNumAu using local in end 5

7 Special case: only identiers Analysis of identiers similar to numbers fun GetIdentAu buf [] = (implode(rev buf), []) GetIdentAu buf (l as (::l')) = if IsLetter orelse IsDigit then GetIdentAu (::buf) l' else (implode(rev buf),l); > val GetIdentAu = > fn > : string list -> string list -> string * string list GetIdentAu ["a","b","c"] ["e","f","g","4","5"," ","6","7"]; > val it = > ("cbaefg45",[" ","6","7"]) : string * string list An identier must start with a letter eception GetIdentErr; > eception GetIdentErr fun GetIdent (::l) = if IsLetter then GetIdentAu [] l else raise GetIdentErr; > val GetIdent = > fn : string list -> string * string list 6

8 Unied treatment Can unify Analysis of numbers and identiers u single general function GetTail u takes a predicate as argument u then uses this to test whether to keep accumulating characters or to terminate u GetNumAu corresponds to GetTail IsDigit u GetIdentAu corresponds to GetTail (fn => IsLetter orelse IsDigit ) fun GetTail p buf [] = (implode(rev buf),[]) GetTail p buf (l as ::l') = if p then GetTail p (::buf) l' else (implode(rev buf),l); > val GetTail = fn > : (string->bool) > -> string list > -> string list -> string * string list 7

9 GetNetToken and Tokenise fun GetNetToken [] = (,[]) GetNetToken (::l) = if IsLetter then GetTail (fn => IsLetter orelse IsDigit ) [] l else if IsDigit then GetTail IsDigit [] l else (,l); > val GetNetToken = > fn : string list -> string * string list To leically analyse a list of characters: u repeat GetNetToken & discard separators fun Tokenise [] = [] Tokenise (l as ::l') = if IsSeparator then Tokenise l' else let val (t,l'') = GetNetToken l in t::(tokenise l'') end; > val Tokenise = fn : string list -> string list Tokenise (eplode "123abcde1][ ] 56a"); > val it = > ["123","abcde1","]","[","]","56","a"] : string list 8

10 Multi-character special symbols Tokenise doesn't handle multi-character special symbols u these will be specied by a table u represented as a list of pairs u that shows which characters can follow each initial segment of each special symbol u such a table represents a FSM transition function For eample, suppose the special symbols are <=, <<, =>, =, ==>, -> then the table would be: [("<", ["=","<"]), ("=", [">","="]), ("-", [">"]), ("==", [">"])] Not fully general u if ==> is a special symbol u then == must be also 9

11 Utility functions Test for membership fun Mem [] = false Mem ('::l) = (=') orelse Mem l; > val Mem = fn : ''a -> ''a list -> bool Get looks up the list of possible successors of a given string in a special-symbol table fun Get [] = [] Get ((',l)::rest) = if =' then l else Get rest; > val Get = fn : ''a -> (''a * 'b list) list -> 'b list Get "=" [("<",["=","<"]), ("=",[">","="]), ("-",[">"]), ("==",[">"])]; > val it = [">","="] : string list Get "?" [("<", ["=","<"]), ("=", [">","="]), ("-", [">"]), ("==",[">"])]; > val it = [] : string list 10

12 GetSymbol GetSymbol takes u a special-symbol table u and a token It etends the token by u removing characters from the input u until table says no further etension is possible fun GetSymbol spectab tok [] = (tok,[]) GetSymbol spectab tok (l as ::l') = if Mem (Get tok spectab) then GetSymbol spectab (tok^) l' else (tok,l); > val GetSymbol = fn > : (string * string list) list > -> string -> string list -> string * string list 11

13 GetNetToken GetNetToken can be enhanced to handle special symbols Special-symbol table supplied as an argument fun GetNetToken spectab [] = (,[]) GetNetToken spectab (::(l as '::l')) = if IsLetter then GetTail (fn => IsLetter orelse IsDigit ) [] l else if IsDigit then GetTail IsDigit [] l else if Mem ' (Get spectab) then GetSymbol spectab (implode[,']) l' else (,l); > val GetNetToken = fn > : (string * string list) list > -> string list -> string * string list 12

14 Tokenise Tokenise can be enhanced to use the new GetNetToken fun Tokenise spectab [] = [] Tokenise spectab (l as ::l') = if IsSeparator then Tokenise spectab l' else let val (t,l'') = GetNetToken spectab l in t::(tokenise spectab l'') end; > val GetNetToken = fn > : (string * string list) list > -> string list -> string * string list 13

15 Eample val SpecTab = [("=", ["<",">","="]), ("<", ["<",">"]), (">", ["<",">"]), ("==", [">"])]; > val SpecTab = > [("=",["<",">","="]), > ("<",["<",">"]), > (">",["<",">"]), > ("==",[">"])] > : (string * string list) list Tokenise SpecTab (eplode "a==>b c5 d5==ff+gg7"); > val it = > ["a","==>","b","c5","d5","==","ff","+","gg7"] > : string list Le is a leical analyser val Le = Tokenise SpecTab o eplode; > val Le = fn : string -> string list Le "a==>b c5 d5==ff+gg7"; > val it = > ["a","==>","b","c5","d5","==","ff","+","gg7"] > : string list 14

16 The -calculus The -calculus is a theory of functions u originally developed by Alonzo Church u as a foundation for mathematics u in the 1930s, several years before digital computers were invented In the 1920s Moses Schonnkel developed combinators In the 1930s, Haskell Curry rediscovered and etended Schonnkel's theory u and showed it equivalent to the -calculus. About this time Kleene showed that the - calculus was a universal computing system u it was one of the rst such systems to be rigorously analysed 15

17 Enter Computer Science In the 1950s John McCarthy was inspired by the -calculus to invent the programming language LISP In the early 1960s Peter Landin showed how the meaning of imperative programming languages could be specied by translating them into the -calculus u he also invented an inuential prototype programming language called ISWIM u ISWIM introduced the main notations of functional programming u and inuenced the design of both functional and imperative languages u ML was inspired by ISWIM 16

18 Strachey & Turner Building on this work, Christopher Strachey laid the foundations for the important area of denotational semantics Technical questions concerning Strachey's work inspired the mathematical logician Dana Scott to invent the theory of domains u an important part of theoretical computer science During the 1970s Peter Henderson and Jim Morris took up Landin's work and wrote a number of inuential papers arguing that functional programming had important advantages for software engineering At about the same time David Turner proposed that Schonnkel and Curry's combinators could be used as the machine code of computers for eecuting functional programming languages 17

19 Theory can be useful! -calculus is an obscure branch of mathematical logic that underlies important developments in programming language theory, such as the: u study of fundamental questions of computation u design of programming languages u semantics of programming languages u architecture of computers 18

20 Synta and semantics of the -calculus -calculus is a notation for dening functions u each -epression denotes a function u functions can represent data and data-structures u details later u eamples include numbers, pairs, lists Just three kinds of -epressions u Variables u Function applications or Combinations u Abstractions 19

21 Variables Functions denoted by variables are determined by what the variables are bound to u binding is done by abstractions V, V 1, V 2 etc. range over arbitrary variables 20

22 Function applications (combinations) If E 1 and E 2 are -epressions u then so is (E1 E 2 ) u it denotes the result of applying the function denoted by E 1 to the function denoted by E 2 u E 1 is called the rator (from `operator') u E 2 is called the rand (from `operand') 21

23 Abstractions If V is a variable and E is a -epression u then V: E is an abstraction u with bound variable V u and body E Such an abstraction denotes the function that takes an argument a and returns as result the function denoted by E when V denotes a More specically, the abstraction V: E denotes a function which u takes an argument E 0 u and transforms it into E[E 0=V ] u the result of substituting E 0 for V in E u substitution dened later Compare V: E with fn V => E 22

24 Summary of -epressions < -epression> ::= <variable> j j (< -epression> < -epression>) ( <variable> : < -epression>) If V ranges over < variable > And E, E 1, E 2, : : : etc. range over < -epression > Then: E ::= V 6 j (E 1 E 2 ) {z } 6 j V: E {z } 6 variables abstractions applications (combinations) 23