Lexical and Syntax Analysis - PDF Free Download

Lexical and Syntax Analysis (of Programming Languages) Abstract Syntax

What is Parsing? Parser String of characters Data structure Easy for humans to write Easy for programs to process A parser also checks that the input string is well-formed, and if not, rejects it.

Example 1 Parser Charlton, 49 Lineker, 48 Beckham, 17 49 Charlton 17 Beckham 48 Lineker CSV (Comma Separated Value) Array of pairs

Concrete and Abstract Syntax The concrete syntax is a set of rules that describe valid inputs to the parser. The abstract syntax is a set of rules that describe valid outputs from the parser. The data structure produced by a parser is commonly termed the abstract syntax tree.

Concrete and Abstract Syntax Parser String of characters Data structure Conforms to the Concrete Syntax of the language Conforms to the Abstract Syntax of the language

Abstract syntax The abstract syntax is usually specified as a data type in the programming language being used, in our case C. Example: typedef struct { char* name; int goals; } Player; typedef struct { Player* players; int size; } Squad; An abstract syntax tree is a value of this type.

This Chapter How: to define the abstract syntax to construct abstract syntax trees in the programming language C. Also revisits some important C programming techniques. If you need a C tutorial then the following books are recommended.

POINTERS Pointer: a variable that holds the address of a core storage location. [The Free Dictionary]

Pointers Declare a variable x of type int and initialise it to the value 10. x: int x = 10; 10 Declare a variable p of type int* (read: int pointer). int* p; Make p point to x (or assign the address of x to p). p = &x; p: p: x: 10

Pointers Print p (here, the address of x). printf("%i\n", p ); p: x: Print the value pointed to by p (here, the value of x). printf("%i\n", *p ); p: 10 x: 10 Assign 20 to the location pointed to by p. *p = 20; p: x: 20

Exercise 1 What is printed by the following program? void swap(int* x, int* y) { int tmp; tmp = *x; *x = *y; *y = tmp; } void main() { int a = 1; int b = 2; swap(&a, &b); printf("a=%i, b=%i\n", a, b); }

DYNAMIC ALLOCATION Dynamic Allocation: the allocation of memory storage for use in a computer program. [The Free Dictionary]

Array allocation Declare a variable p of type int*. int* p; p: Allocate memory for an array of 4 int values and let point p to it. p = malloc(4 * sizeof(int)); p:

Array indexing Assign 10 to the location pointed to by p. *p = 10; Assign 20 to the first element of the array pointed to by p. p[0] = 20; Copy the first element of the array to the third element. p[2] = p[0]; p: p: p: 10 20 20 20

Array deallocation When finished with an array allocated by malloc, call free to release the space, otherwise your program may run out of memory. free(p); p: Space released, so it can be reused by future calls to malloc.

STRINGS String: a series of consecutive characters. [The Free Dictionary]

Strings Declare a variable s, initialised to point to the string hi. char* s = hi ; s: h i \0 Let s point to the next character. s = s + 1; s: h i \0 And let s point to the previous character again. s = s - 1; s: h i \0

Exercise 2 What is printed by the following program? int f(char* s) { int i = 0; while (s[i]!= '\0') i++; return i; } void main() { char* x = Hello ; printf( %i\n, f(x)); }

USER-DEFINED TYPES Type: the general character or structure held in common by a number of things. [The Free Dictionary]

Type definitions A typedef declaration allows a new name to be given to a type. typedef int Integer; typedef char* String; Existing type A new name Example use: String s; /* Declare a string s */ Integer i; /* and an integer i */ i = 0; s = hello ;

Enumerations An enum declaration introduces a new type whose values are members of a given set. enum colour {RED, GREEN, BLUE}; New type Possible values Example use: enum colour c; c = RED; if (c == RED) printf( Red\n ); Give it a shorter name: typedef enum colour Colour;

Structures An struct declaration introduces a new type that is a conjunction of one or more existing types. New type struct rectangle { float width; float height; }; Example use: struct rectangle r; r.width = 10; r.height = 20; A width and a height A circle: struct circle { float radius; }

Unions An union declaration introduces a new type that is a disjunction of one or more existing types. New type union shape { struct circle circ; struct rectangle rect; }; A circle or a rectangle Example use: struct shape s; s.circ.radius = 10;

Tagged unions Often a tag is used to denote the active disjunct of a union. Another definition of shape: struct shape { enum { CIRCLE, RECTANGLE } tag; union { struct circle circ; struct rectangle rect; }; };

Tagged unions Example: s is a circle and t is a rectangle, and both are of type struct shape. struct shape s, t; s.tag = CIRCLE; s.circ.radius = 10; t.tag = RECTANGLE; t.rect.width = 5; t.rect.height = 15;

Tagged unions Example: compute the area of any given shape s. float area(struct shape s) { if (s.tag == CIRCLE) { float r = s.circ.radius; return (3.14 * r * r); } if (s.tag == RECTANGLE) { return (s.rect.width * s.rect.height); } }

Recursive structures A value of type struct t may contain a value of type struct t*. struct list { int head; struct list* tail; }; typedef struct list List; Suppose x is a value of type List*. (*xs).head xs->head (*(*xs).tail).head xs->tail->head

Recursive structures Example: inserting an item onto the front of a linked list. List* insert(list* xs, int x) { List* ys = malloc(sizeof(list)); ys->head = x; ys->tail = xs; return ys; }

CASE STUDY A simplifier for arithmetic expressions.

Concrete syntax Consider the following concrete syntax for arithmetic expressions, where v ranges over variable names and n over integers. e v n e + e e * e ( e ) Example expression: x * y + (z * 10)

Simplification Consider the algebraic law: x. x * 1 = x This law can be used to simplify expressions by using it as a rewrite rule from left to right. Example simplification: x * (y * 1) x * y

Problem 1. Define an abstract syntax, in C, for arithmetic expressions. 2. Show how to construct abstract syntax trees that represent arithmetic expressions. 3. Implement the simplification as a C function that takes and returns an abstract syntax tree.

Abstract syntax typedef enum { ADD, MUL } Op; struct expr { enum { VAR, NUM, APP } tag; union { char* var; }; }; int num; struct { struct expr* e1; Op op; struct expr* e2; } app; typedef struct expr Expr; A variable or a number or an op and two sub-expressions

Constructors Expr* mkvar(char* v) { Expr* e = malloc(sizeof(expr)); e->tag = VAR; e->var = v; return e; } Expr* mknum(int n) { Expr* e = malloc(sizeof(expr)); e->tag = NUM; e->num = n; return e; } Expr* mkapp(expr* e1, Op op, Expr* e2) { Expr* e = malloc(sizeof(expr)); e->tag = APP; e->app.op = op; e->app.e1 = e1; e->app.e2 = e2; return e; }

Abstract syntax trees An abstract syntax tree that represents the expression x + y * 2 can be constructed by the following C expression mkapp( mkvar("x"), ADD, mkapp( mkvar("y"), MUL, mknum(2)))

Simplification x. x * 1 = x is implemented by void simplify(expr* e) { if (e->tag == APP && e->app.op == MUL && e->app.e2->tag == NUM && e->app.e2->num == 1) { *e = *(e->app.e1); } if (e->tag == APP) { simplify(e->app.e1); simplify(e->app.e2); } }

Homework exercises Implement a pretty printer that prints an abstract syntax tree in a concrete form. void print(expr* e) {... } Extend the simplifier to exploit the following algebraic law. x. x * 0 = 0

Motivation for LSA In LSA, we are interested in how to implement the following kind of function Expr* parse(char* string) {... } It takes a string conforming to the concrete syntax and returns an abstract syntax tree.