Chapter 10: Compiler I: Syntax Analysis

Size: px

Start display at page:

Download "Chapter 10: Compiler I: Syntax Analysis"

Britton Jackson
6 years ago
Views:

1 Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005 Chapter 10: Compiler I: Syntax Analysis Usage and Copyright Notice: Copyright 2005 Noam Nisan and Shimon Schocken This presentation contains lecture materials that accompany the textbook The Elements of Computing Systems by Noam Nisan & Shimon Schocken, MIT Press, The book web site, features 13 such presentations, one for each book chapter. Each presentation is designed to support about 3 hours of classroom or self-study instruction. You are welcome to use or edit this presentation for instructional and non-commercial purposes. If you use our materials, we will appreciate it you will include in them a reference to the book s web site. And, you have any comments, you can reach us at tecs.ta@gmail.com Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 1

2 Course map Human Thought Abstract design Chapters 9, 12 abstract interface H.L. Language & Operating Sys. Compiler Chapters abstract interface Virtual Machine Software hierarchy VM Translator Chapters 7-8 abstract interface Assembly Language Assembler Chapter 6 abstract interface Machine Language Computer Architecture Chapters 4-5 Hardware hierarchy abstract interface Hardware Platform Gate Logic Chapters 1-3 abstract interface Chips & Logic Gates Electrical Engineering Physics Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 2

3 The big picture Modern compilers: Front-end: from high-level to intermediate Some... Some compiler Some Other Some Other compiler... Jack Jack compiler Compiler lectures Back-end: from the intermediate to binary code. VM implementation over CISC platforms VM imp. over RISC platforms Intermediate code VM emulator VM imp. over the Hack platform VM lectures CISC machine RISC machine... written in a high-level Hack machine CISC machine RISC machine other digital platforms, each equipped with its VM implementation Any computer Hack computer Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 3

4 Compiler architecture (front end) Some... Some compiler Some Other Some Other compiler... Jack compiler Jack Intermediate code Jack Compiler (Chapter 10) XML code VM implementation VM imp. over CISC over RISC platforms platforms CISC RISC machine machine VM emulator... VM imp. over the Hack platform written in a high-level Hack machine Syntax Analyzer Jack Program Tokenizer Parser Code Gene -ration (Chapter 11) VM code Front-end: Syntax analysis: understanding the semantics implied by the source code Code Tokenizing: generation: creating reconstructing a list of the atoms semantics using a target syntax Parsing: matching the atom list with the grammar XML output = proof that the syntax analyzer is parsing correctly Code generation: reconstructing the semantics using the target syntax. Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 4

Tokenizing / Lexical analysis Remove white space Construct a token list ( atoms) Things to worry about: Language specic rules: e.g. how to treat ++ Language specic token types: keyword, identier, operator, constant,.

5 Tokenizing / Lexical analysis Remove white space Construct a token list ( atoms) Things to worry about: Language specic rules: e.g. how to treat ++ Language specic token types: keyword, identier, operator, constant,... Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 5

6 Jack Tokenizer Source code (x (x < 153) let city = Paris ; Tokenizer s output <tokens> <keyword> </keyword> <symbol> ( </symbol> <identier> x </identier> <symbol> < </symbol> <integerconstant> </integerconstant> <symbol> ) </symbol> <symbol> </symbol> <keyword> let let </keyword> <identier> city </identier> <symbol> = </symbol> <stringconstant> Paris </stringconstant> <symbol> ; </symbol> <symbol> </symbol> </tokens> Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 6

7 Parsing Each is characterized by a grammar A text is given: The parser, using the grammar, can either accept or reject the text In the process, the parser performs a complete analysis of the text The can be: Context-dependent (English, ) Context-free (Jack, ). Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 7

8 Examples context free context dependent (5+3)*2 sqrt(9*4) she discussed sex with her doctor - parse 1 parse 2 + * 2 sqrt * discussed with discussed she sex her doctor she with sex her doctor Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 8

9 A typical grammar (C/Java-like) program: program: : : Statement Statement Statement Statement other other possibilities possibilities '' '' Sequence Sequence '' '' Statement: Statement: '' '' '(' '(' ')' ')' Statement: Statement: simpleif simpleif Else Else simpleif: simpleif: '' '' '(' '(' ')' ')' code sample Else: Else: '' '' '(' '(' ')' ')' 'else' 'else' Sequence: Sequence: '' '' null, null, i.e. i.e. the the empty empty sequence sequence ';' ';' Sequence Sequence : : definition definition of of an an comes comes here here more more definitions definitions follow follow Simple (terminal) forms / complex (non-terminal) forms Grammar = set of rules on how to construct complex forms from simpler forms Highly recursive. code sample Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 9

10 Parse tree Input Text: (count<=100) /** demonstration */ count++;... Tokenized: ( count <= 100 ) count ++ ;... Statement program: program: : : Statement Statement Statement Statement other other possibilities possibilities '' '' Sequence Sequence '' '' Statement: Statement: '' '' '(' '(' ')' ')' Sequence Sequence ( count <= 100 ) count ++ ;... Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 10

Recursive descent parsing code sample Highly recursive LL(0) grammars: the first token determines in which rule we are In other grammars you have to look ahead 1 or more tokens Jack is almost LL(0).

11 Recursive descent parsing code sample Highly recursive LL(0) grammars: the first token determines in which rule we are In other grammars you have to look ahead 1 or more tokens Jack is almost LL(0). parsestatement() parsewhilestatement() parseifstatement() parsestatementsequence() parseexpression(). Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 11

12 A linguist view on parsing Parsing: One of the mental processes involved in sentence comprehension, in which the listener determines the syntactic categories of the words, joins them up in a tree, and identies the subject, object, and predicate, a prerequisite to determining who did what to whom from the information in the sentence. (Steven Pinker, The Language Instinct) Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 12

13 The Jack grammar x : x : x appears verbatim x: x: x is is a construct x?: x?: x appears 0 or or 1 times x*: x*: x appears 0 or or more more times xy: xy: either x or or y appears (x,y): x appears, then then y. y. Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 13

14 The Jack grammar (cont.) x : x : x appears verbatim x: x: x is is a construct x?: x?: x appears 0 or or 1 times x*: x*: x appears 0 or or more more times xy: xy: either x or or y appears (x,y): x appears, then then y. y. Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 14

15 Jack syntax analyzer in action Class Class Bar Bar method method Fraction Fraction foo(int foo(int y) y) var var int int temp; temp; a variable variable let let temp temp = (xxx+12)*-63; Syntax analyzer Syntax analyzer Using the grammar, a programmer can write a syntax analyzer program The syntax analyzer takes a source text file and attempts to match it on the grammar If successful, it generates a parse tree in structured format, e.g. XML. This syntax analyzer s algorithm: If xxx is non-terminal, output: <xxx> Recursive code for the body of xxx </xxx> If xxx is terminal (keyword, symbol, constant, or identier), output: <xxx> xxx value </xxx> <vardec> <vardec> <keyword> var var </keyword> <keyword> int int </keyword> <identier> temp temp </identier> <symbol> <symbol> ; </symbol> </vardec> <s> <letstatement> <keyword> let let </keyword> <identier> temp temp </identier> <symbol> <symbol> = </symbol> <> <term> <term> <symbol> <symbol> ( </symbol> <> <term> <term> <identier> xxx xxx </identier> </term> </term> <symbol> <symbol> + </symbol> <term> <term> <int.const.> </int.const.> </term> </term> </> Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 15

16 Summary and next step Syntax analysis: understanding syntax Code generation: constructing semantics Jack Compiler (Chapter 10) XML code Syntax Analyzer Jack Program Tokenizer Parser Code Gene -ration (Chapter 11) VM code The code generation challenge: Extend the syntax analyzer into a full-blown compiler that, instead of generating passive XML code, generates executable VM code Two challenges: (a) handling data, and (b) handling commands. Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 16

17 Perspective The parse tree can be constructed on the fly Bottom up compilation is harder and more powerful Syntax analyzers are typically built using tools like: Lex for tokenizing Yacc for parsing The Jack is intentionally simple: Statement prefixes: let, do,... No operator priority No error checking Basic data types, etc. Typical s are richer, requiring more powerful compilers The Jack compiler: designed to illustrate the key ideas that underlie modern compilers, leaving advanced features to more advanced courses. Elements of Computing Systems, Nisan & Schocken, MIT Press, 2005, Chapter 10: Compiler I: Syntax Analysis slide 17

Compiler I: Sytnax Analysis

Elements of Computing Systems, Nisan & Schocken, MIT Press www.idc.ac.il/tecs Compiler I: Sytnax Analysis Usage and Copyright Notice: Copyright 2005 Noam Nisan and Shimon Schocken This presentation contains