The use of symbol-state tables

Transcription

1 The use of symbol-state tables A. C. Day Computer Centre, University College London, Gordon Street, London WC This paper describes how a certain kind of table may be used to check the syntax of a string of symbols. In its simplest form, a symbol-state table closely resembles afinitestate machine, and requires a large amount of space, often to store duplicated information. By means of the subroutine principle such duplication may be avoided, and recursive use of the table achieved, making possible the checking of recursive structures. When syntax checking is not the only aim, or when checking needs to be performed which is beyond the scope of the table, sections of program coding may be called from the table. Applications have included the writing in FORTRAN of a syntax checker for FORTRAN, and the automatic proof reading of dictionary entries. Received January Arrays are often found to be useful as an alternative way of storing programming information, rather than coding a great many conditional instructions. An example of this is the use of decision tables, in which the fulfilment of a set of conditions leads to the execution of a set of operations, the conditions and the operations being specified by the table. Another example is that of transition matrices, as used in compiler writing. Usually in this case the next symbol in the input stream and the symbol at the top of a stack are used to denote the row and column of the matrix. Their intersection gives the place in the matrix at which is stored the address of a subroutine to handle the incoming symbol. This is shown diagrammatically in Fig.. The tables described in this paper are also used to store supplementary programming information, and are useful stack symbol Input symbol aids when any kind of syntax checking needs to be performed. In a symbol-state table the next symbol in the input stream denotes the column, and the current state gives the row. The intersection of these in the table yields a datum which may be the new state, an error indication, or certain other kinds of information see Fig.. If the datum in the table is always a new state, then the symbol-state table is equivalent to a finite state machine with no output. If the datum is always a subroutine address, then the table is a transition matrix. However, there are many other possibilities. I will deal here with the practical possibilities rather than the theoretical classifications. Examples of simple symbol-state tables An elementary example should make clear the way in which a symbol-state may be used for syntax checking. Let us suppose that it is necessary to check the syntax of FORTRAN GOTO statements. We will further suppose that the keyword GOTO has already been identified, so that all that needs to be done is to check the remaining non-blank characters of the statement. These will be Input symbol Downloaded from by guest on October ;urrent state Subroutine Fig.. Transition matrix The Computer Journal Volume Number November Fig.. Symbol-state table

2 FOLLOWS: GO TO part of name EXPECTS: /label \{label, / \ label, / \> label f alph. char. < ALPH. NUM. Fig.. Checking simple and computed GO TO our input symbols, which we will consider to be terminated by a further symbol called end of statement. A subroutine which we will call NEXT is needed to deliver the type of the next symbol. The types will be:. Alphabetic characters.. Numeric s.. Left bracket.. Right bracket.. Comma... For simplicity we will first assume that the only permittcu forms are the simple GOTO and the computed GOTO, i.e. the forms: GOTO label GOTO label, label..., name where label represents a string of to decimal s, and name must begin with an alphabetic character, which is then followed by to alphabetic or numeric characters. A symbol-state table which will check the syntax of these GOTO statements is shown in Fig.. Certain cells are left blank for greater readability. These would normally be filled with negative numbers, indicating an error condition. A zero cell indicates that the whole statement has been checked, with no errors detected. A positive cell is to be interpreted as the new state. On the left-hand side of the table in Fig. there are two columns of comments which have been found useful when constructing large or complex tables. The problems which arise then are similar to those encountered i Tlie use of symbol-state tables when programming in absolute non-symbolic machine code. The left-hand column of comments gives some indication of the symbols which have been found immediately before reaching that state. The right-hand column shows which symbols are to be expected and therefore permitted. The table in Fig. must be used in conjunction with a driving program, of course. A suitable section of FORTRAN coding for performing this task is given in Prog.. It is assumed that the table in Fig. has been loaded into the array called TABLE in Prog.. This array has been dimensioned with rows to allow for expansion as we deal with more complex examples. DIMENSION TABLE, INTEGER, TYPE, TABLE C SET INITIAL = C GET THE TYPE OF THE NEXT INPUT CALL NEXT TYPE C RETRIEVE THE NEXT FROM THE TABLE = TABLE, TYPE C TEST FOR ERROR, EXIT OR CONTINUE IF,, C SUCCESSFUL EXIT - NO ERRORS FOUND CONTINUE * C ERROR EXIT CONTINUE Prog. We will now see how the table in Fig., driven by the coding above, checks the FORTRAN statement: GOTO,,, IPT The table is entered with an initial state of when the next symbol in the input stream is the one immediately following the GOTO. When the cell in the table is positive, it is taken to be the state for examining the next symbol. The sequence of states may be shown as follows: Downloaded from by guest on October Next symbol Symbol State Cell of table ^ X» "", I ^ A U -^j -^ P T

3 A. C. Day FOLLOWS : GOTO name, EXPECTS: label I {label, alph. char, { label, / label f ALPH. NUM. Fig.. Amendment to Fig. for checking assigned GO TO in addition to simple and computed GO TO In Fig. there are cells left blank, to be filled with negative error flags. By choosing different negative integers for different error conditions, we are able to diagnose up to different types of error. Of course, such detail is hardly necessary, and several cells may be given the same error number. At the error exit, we may print out different messages according to the error number, or we may print out a standard diagnostic together with the error number itself. The checking performed by the table in Fig. is not adequate for many purposes. For instance, we do not test to see that names and labels do not exceed their maximum permitted lengths. Nor do we ensure that the name is integer in type. However, this is intended merely as a simple example to show the workings of a symbol-state table. Note that the driving loop in this example is only three FORTRAN statements in length. It is true that one of these three is a call to a subroutine, but this subroutine NEXT need not be very complex, and it would be needed whatever method of checking was used. FOR- TRAN code is conserved because the table is in effect interpretive code. This concept of the table as code is a useful one, as it may be developed to cover subroutines within the body of the table, which may in fact be called recursively, and also calls from the table to sections of program coding in order to perform special actions which are beyond the scope of the table. These ideas will be developed in the remainder of the article. Table subroutines A further type of GOTO statement exists in FOR- TRAN, namely, the assigned GOTO. This has the format: GOTO name, {label, label... j Fig. shows the amendments which need to be made to Fig. in order for the table to test the syntax of all three types of GOTO. Rows of the table in Fig. which are not mentioned in Fig. are assumed to remain the same. At this point there is considerable redundancy in the table, as rows - are identical with rows -. These two sections are both checking for the structure: label, label... We cannot directly combine these two sections, because after rows - we must look for, name whereas after rows - only the end of statement is permitted. The redundancy and waste of space which is apparent here becomes progressively more of a problem as the syntax which is tested becomes more detailed. This is obviously the kind of problem which, when coding, can be handled by a subroutine which is called from two different places. The redundancy in the symbol-state table under consideration can be overcome by means of a table analogue for a subroutine. Throughout the rest of this paper such an analogue will be written as subroutine in italics to distinguish it from a coded subroutine. Instead of having two blocks of rows which are identical, only one version of these rows is kept, and when this is entered, the return address is stored. The return address is here the row which should be used as the old state on leaving the subroutine. FOLLOWS: GOTO {label..., part of name part of name name, {label... label, Fig.. EXPECTS: label j {label, [name, / -, L \, name name falph. char. -< [ falph. char. I {label, label, label f label J, ' anything ALPH. Checking simple, assigned and computed GOTOs using a subroutine rows - NUM : Downloaded from by guest on October

4 DIMENSION TABLE, INTEGER, TYPE, TABLE, RET C SET INITIAL AND RETURN ADDRESS = RET = C GET THE TYPE OF THE NEXT INPUT CALL NEXT TYPE C LOOK UP THE NEXT IN THE TABLE = TABLE, TYPE C TEST FOR ERROR, EXIT OR CONTINUE IF,, C POSITIVE - IS IT MORE THAN IF.LT. GO TO C SUBROUTINE CALL - UNPACK RETURN ADDR. RET = MOD, = / GO TO CIF RETURN ADDR. IS NOT ZERO, GO TO IT k IF RET.EQ. GO TO = RET RET = GO TO C SUCCESSFUL EXIT - NO ERRORS FOUND CONTINUE C ERROR EXIT CONTINUE Prog. The table in Fig. is the equivalent of those in Figs. and in the checking it performs, but rows - of the former take the place of the duplicated rows of the latter. Rows - of Fig. are a subroutine for checking two or more labels, separated by commas, and terminated by a right bracket. This subroutine is called from two cells of the table, rows and, column in each case. In order to call a subroutine two row addresses are needed, one to indicate the starting row of the subroutine, and one for the return row. This information is packed into one cell in Fig. by the arbitrary method of multiplying The use of symbol-state tables the subroutine address by and adding to it the return address. Of course, two tables could be used, one of which would contain only the subroutine addresses for calls, but this would be very wasteful of space. A zero cell in the table now has two meanings. When executing a subroutine it means 'go to the return address' as in row of Fig.. Outside a subroutine it means 'exit from the table' as in row, column of Fig.. The FORTRAN coding needed to drive the table will now have to be amended. A variable RET will be used to store the return address. Each time a positive datum is obtained from the table it is now necessary to test whether it is greater than. The coding will now be that of Prog.. In FORTRAN the assigned and computed GOTO statements have in common the string label, label... It will be seen from Fig. that the subroutine is not called until the first common symbol the left bracket is encountered. It usually happens that we do not know whether the subroutine needs to be called until we reach the first symbol which could have been checked by that subroutine. In Fig. the left bracket is checked in rows and, so that it is pointless to have the subroutine check it over again. Therefore I adopt the convention that on a call to a subroutine the next symbol is obtained from the input stream ready for the subroutine to check it. Thus in Fig. the first symbol to be checked by the subroutine is the first numeric inside the brackets. A similar convention needs to be adopted on returning from a subroutine. In the example used here, we know that control should pass from the subroutine as soon as the right bracket is found, so we could return from row, column of Fig., obtaining the next symbol from the input stream as we do so. However, the more usual case is that we only realise that we ought to have returned from the subroutine when examining the first symbol which is not to be checked by that subroutine. Therefore the convention I adopt is to return on the first symbol not handled by the subroutine, and not to get the next symbol from the input stream as control is passed back. This is why, in the coding of Prog., when a return address is used as the next state, control is passed to statement rather than to statement. This means that in Fig. row appears to be wasted. Its purpose is really to get the next symbol beyond the right bracket, in order to return with this symbol to the return address either row or row, where this symbol will be checked. It is not necessary to put error codes in row, e.g. to trap an erroneous left bracket. All such checking can be done when the return address is reached, i.e. in this case, row or row. Once again, this is merely an example to show what can be done by means of symbol-state tables. In this case we have saved rows out of by using a subroutine. When the syntax to be checked is more complex, the savings which accrue by using subroutines become very considerable indeed. The FORTRAN loop which is executed when there are no calls, errors or returns is still only four statements in length. Nested and recursive calls The FORTRAN coding for use with the table in Downloaded from by guest on October

5 Fig. was written with the assumption that only one subroutine would be called at a time. It is very useful to be able to have nested calls. As a call to a symbol-state table subroutine does not change the table in any way, these calls may in fact be recursive with no additional machinery. FOLLOWS: EXPECTS: fbeginning exprn fop < t exprn Fig.. ^ NAME OP Checking expressions by recursive calls OTHER Recursive techniques are particularly useful for checking recursive structures. As an example, let us consider expressions which are defined by the following BNF grammar: <name> ::= A B C D E F G H I J K L M N O P Q R S T U V W X Y Z <op> ::= + - * / <exprn> ::= <name> «exprn» <exprn><op><exprn> The third rule is a recursive definition of <exprn> in terms of itself. Examples of expressions defined by this grammar are: F A +B «G/D - E*M + P The syntax of all expressions defined by this grammar may be checked by the symbol-state table in Fig.. For both nested and recursive calls the return addresses must be kept in a stack. FORTRAN coding to drive Fig. is therefore a little more complicated, but Prog. is sufficient. Subroutine NEXT also needs to be amended in order to deliver the types of symbols required by Fig.. An extra column column has been added to Fig. in order to cope with invalid symbols which are encountered by NEXT. When NEXT finds a symbol which it cannot assign to any of the five types, it makes it type. All of the cells in column contain error codes, so that the invalid symbol is trapped by the same mechanism which finds symbols in invalid positions. One weakness of Fig. is that both a return from a call, and exit from the table, can occur when either a right bracket or the end of statement is reached. This means that excess right brackets could cause exit from the table before the whole statement has been checked. This could have been avoided by adding extra rows to the table, so that the first call to the subroutine is separated from all other recursive calls. For interest, this table is given in Fig.. However, the economy of Fig. may be preserved simply by the addition of the statement labelled in Prog., to check on final exit from the table that the end of statement has in fact been reached. A. C. Day As an example, let us take the expression G/D - E*M + P and trace the sequence of states as this expression is DIMENSION TABLE,, STACK INTEGER, TYPE, TABLE, STACK * C SET INITIAL AND LEVEL OF CALLS = LEVEL = C GET THE TYPE OF THE NEXT INPUT CALL NEXT TYPE C LOOK UP THE NEXT IN THE TABLE = TABLE, TYPE C TEST FOR ERROR, EXIT OR CONTINUE IF, if, C POSITIVE - IS IT MORE THAN IF.LT. GO TO C SUBROUTINE CALL - INCREMENT LEVEL LEVEL = LEVEL + IF LEVEL.GT. GO TO C UNPACK RETURN AND SUBROUTINE ADDR. STACK LEVEL = MOD, = / GO TO C IF STACK IS NOT EMPTY, RETURN k IF LEVEL.EQ. GO TO = STACK LEVEL LEVEL = LEVEL - GO TO C EXIT - NO ERRORS FOUND - TEST IF IF TYPE.NE. GO TO * C ERROR EXIT CONTINUE. C CALLS ARE NESTED TOO DEEPLY CONTINUE Prog. Downloaded from by guest on October

6 The use of symbol-state tables Next symbol Symbol type G / D - E * M + P Old state New state Top of stack "' ^ ^ ^ ^ / ^ ^ X ^ Level checked. In each state the stack contains the number of values given by the variable LEVEL, but here only the last value placed in the stack will be shown. The table being used is that in Fig.. Note that on a call to a subroutine the new state is used to give both the next old state, and also the return address which is pushed down on the stack. On return from a subroutine no new symbol is obtained from the input stream, but the next old state is derived by 'popping' the top of the stack. The sequence of states will be as in the table above. The power of symbol-state tables is seen in the fact that only three rows are sufficient to check recursive structures of considerable complexity. Rows - of Fig. may themselves be called as a subroutine if such expressions occur at different places in higher level structures. In that case the FORTRAN coding given in Prog. is sufficient for both the higher level structures and the expression subroutine; no more coding is needed for the driver program than that given. Action calls Just as writers of programs in high level languages find it convenient at times to call subroutines written in assembly language, so writers of symbol-state tables sometimes find it necessary to perform part of the syntax checking by means of specially written program statements rather than by means of rows in the matrix. In either case the motivation may be economy of space or time, or in order to do something which is not possible in the basic medium. It is possible for a symbol-state table to call sections of program coding, control usually being handed back to the table again. A convenient name for such a process is action call. The need for this may be illustrated using Fig.. beginning { name op exprn exprn exprn exprn /name op op fop \ NAME OP OTHER Fig.. Expanded to check expressions by recursive calls Column of rows and, and columns and of rows and, are used to check for a FORTRAN variable name, which must begin with an alphabetic character, which may then be followed by up to five alphabetic or numeric characters. All that the table in Fig. does is to check that the first character is alphabetic, and that any subsequent characters are alphabetic or numeric. No check is made on the length of the name, or its type which for an assigned or computed GOTO must be integer. The length of the name could be checked by means of a subroutine, but this would use up an extra five rows in the symbol-state table. At the expense of an extra column in the table a test could be made as to whether the first character of the name was between I and N, but the type may have been set by an explicit type statement. The only way to test the type for sure is to keep a table of names and their attributes, which a symbol-state table is unable to do. What is needed is a section of program coding which can be called from the table, and which will process a name, checking its length and type. This could be done by modifying Fig., row, column, to the number. The convention would be that the ten-thousands indicates that this is an action call. The next two s specify that action is required. The final s give the row to which control should be returned after the action call. The contents of columns and in row are now irrelevant, as the coding for the action call will process all alphabetic and numeric characters belonging to the name. The same action call will be required from row, column, so this cell in the table will be changed to. In the same way, Fig. does not check labels sufficiently, and could be supplemented with action coding for this purpose. No FORTRAN label may be more than five decimal s in length. Also any thorough checking of a FORTRAN program will need to keep a table of labels, and must amend the table for all labels referenced in GOTO statements to ensure that the program segment contains these labels, and that they are attached to executable statements. To check a label in these ways could be the purpose of action. Then row, column of Fig. would be changed to, meaning 'action call, return address '. All other checking of labels in Fig. would similarly use action. The revised form of Fig. is given in Fig.. Obviously a FORTRAN driving program for use with the table in Fig. must be different from that used with the table of Fig.. If the next state retrieved from the table is greater than, then a further test must be made to see whether it is greater than, in which case the three numbers must be unpacked from the cell. A computed GOTO based on the required action here or leads to the appropriate action coding. Downloaded from by guest on October

7 A. C. Day ALPH. NUM. Fig.. Fig. modified to include action calls for checking names and labels. Note that rows, and could now be condensed into one row. This has not been done in order to preserve similarity with Fig.. Z id UH z < Q U s«z" CO ftze Sis. o< ii oa ow za! S ft w gaction P <: Multiple symbol-state tables There are certain problems for which more than one symbol-state table may be needed. For example, when checking FORTRAN statements, the FORMAT statement will need a different symbol-state table and a different subroutine to deliver the type of the next input symbol from all other statements. This is because in most FORTRAN statements there is no significant difference in type between such characters as A, Fand X, whereas in the FORMAT statement they must be treated as different types of symbols. The number of columns for a table checking FORMAT statements will normally be greater than for one checking other statements. The two tables could only be combined at the expense of a considerable wastage of space. It may even be necessary at times to have a hierarchy of symbol-state tables, the higher tables checking the order of items whose internal structure has been checked by lower tables. For example, let us consider a hierarchy of tables for checking the syntax of FORTRAN. The lowest level tables will check the syntax of statements. There will probably be two of these, one to check FORMAT statements, and the other to check all other statements. For both these tables the symbols will be characters in the input stream. A higher level table will be needed to check for the ordering of statements within program segments O u. Z ENT L O a: D gl!i a < ATEMEN YPE W O M a Downloaded from by guest on October Action calls: Error message statement out of order. Error message END line missing. Error message statement is illegal in BLOCK DATA. Error message null subprogram. Error message if the statement is not labelled. Error message END not preceded by transfer of control. Error message subprogram contains no executable statement. Error message unrecognised statement. Fig.. Checking for the order of FORTRAN statements

8 The use of symbol-state tables A suitable table for checking that this conforms to the specification of ASI FORTRAN is shown in Fig.. In this case the symbols are FORTRAN statements which have already been checked using the lowest level tables. The table in Fig. has no subroutines, so the hundreds is used to indicate action calls. The actions required are shown beneath the table. A higher level table still could be used in which the symbols are program segments, and in which a test is made that one and only one main program, and not more than one BLOCK DATA subprogram, occur. However, this can be done more economically by keeping flags which are set by action calls from the table in Fig.. The point is that complex structures may be checked by using a hierarchy of tables, thereby increasing their power. Applications From the examples given here it will be apparent that symbol-state tables are a great help in designing syntax checkers for a programming language. Compilers do not do all the syntax checking which may be desired, as they are almost always biased towards some manufacturer's version of the language, and hardly ever check for an internationally agreed standard. Consequently there arises a need for syntax checkers written in a high level language in order to be machine independent which test the syntax of programs in a high level language, e.g. a FORTRAN program to check the syntax of a FORTRAN program and flag every statement which does not conform to the ASI standard. This could be a great help in establishing whether or not a particular program is machine independent. The size of such a syntax checker can be greatly reduced by using symbolstate tables, without impairing the speed. Another application has been that of processing text material in a natural language. A project was undertaken at the Computer Centre of University College London under a grant from the School of Oriental and African Studies to produce the final volume of Sir Ralph Turner's Comparative Dictionary of Indo-Aryan Languages. This was to be a phonetic analysis, listing all those headwords which contained examples of certain sequences of symbols. The, Sanskrit headwords from the dictionary were punched on to paper tape, then copied on to magnetic tape. The Sanskrit is represented in the dictionary by a Romanised transliteration which includes many 'diacritics', i.e. marks above and below the letters. For computer input these composite symbols had to be coded as the letter followed by special characters representing the diacritics, e.g. t became T became S/ a became A/= A magnetic tape had to be produced for automatic typesetting on which every printable symbol composite or not had to be represented by an integer, e.g. t was represented as, a as, and so on. The problem was to recognise groups of characters in the input stream as the corresponding printable symbols, and to do it efficiently. A symbol-state table was written with a subroutine which built up the printable symbols. Each time a letter was found in the input stream, the subroutine was called. Action calls from the subroutine as the letter and any following diacritics were processed, built up the resulting single integer. In fact, as the subroutine was called, a count was initialised to a certain value depending on the letter which had been encountered. When a diacritic was found, the count was incremented by an amount depending on the diacritic. When a character was encountered which could not be part of the printable symbol, control was handed back from the subroutine, and at the same time the resulting count was placed in the output stream. This process enabled some proofreading to be performed, as certain invalid sequences of characters were trapped by the symbol-state table. Other possible uses of symbol-state tables which have suggested themselves are pattern-matching, parsing, or even with suitable action calls as the core of an interpreter. The great drawback to the tables presented here is the labour of constructing them by hand. However, it should not be too difficult to write programs which will accept rules in an appropriate form, and which will build a table to check statements according to those rules. These tables provide a powerful technique which should not be confined to compiler writers, or to those who program in assembly languages. Acknowledgements My thanks are due to Paul A. Samet, who first introduced me to simple symbol-state tables. Alan Shaw has assisted me under grants from the National Computing Centre for work on syntax checkers and from the School of Oriental and African Studies for work mentioned above. Downloaded from by guest on October References CHAPIN, N.. Parsing of Decision Tables, CACM, Vol., No., pp. -. GRIES, D.. Use of Transition Matrices in Compiling, CACM, Vol., No., pp. -. FELDMAN, J., and GRIES, D.. Translator Writing Systems, CACM, Vol., No., pp. -.