CYK Algorithm Adapted to the Penttonen Normal Form

Transcription

1 CYK Algorithm Adapted to the Pettoe Normal Form Domiika Klobučíková* Abstract This paper deals with the topic of cotext-sesitive grammars as special cases of urestricted grammars, their ormal forms ad applicatios of these ormal forms. It presets a algorithm desiged usig the priciples of Cocke-Youger-Kasami algorithm to make a decisio, whether a iput strig is a setece of a cotext-sesitive grammar. The fial applicatio, which implemets this algorithm, works with cotext-sesitive grammars i Pettoe ormal form. Keywords: formal laguages ormal forms Kuroda Pettoe sytax aalysis CYK CKY Cocke-Youger-Kasami Supplemetary Material: Theoretical Example ( *xklobu0@stud.fit.vutbr.cz, Faculty of Iformatio Techology, Bro Uiversity of Techology. Itroductio Sytax aalysis is a essetial compoet of may disciplies focusig o strig processig oe such field is program iterpretatio. I practice, it has always bee limited to uambiguous, cotext-free grammars, as the less restricted families of grammars had prove both hard to use ad too difficult to process. However, as the aforemetioed area of study has bee progressig, the cotext-freeess offered by this set of grammars became o loger satisfactory, ad a eed to decompose the iput strig i a way reflectig atural laguage has arise. Cotext-sesitive grammars, ad therefore laguages geerated by them, preset a suitable iterlik betwee these sets, because they allow for cotext, which is a importat aspect of huma speech. Curretly, very few parsers capable of detectig cotext, ad therefore of processig grammars of this family, exist. The aim of this paper is to desig ad implemet such a algorithm that ca determiistically decide, whether the iput strig is a setece geerated by some grammar. To allow for maximum flexibility, the grammar is to be specified by the user. The algorithm itroduced i this paper is based o Cocke-Youger-Kasami (CYK) parsig algorithm for cotext-free grammars [], which works with grammars i Chomsky ormal form. To decide whether a strig is a member of some grammar, it cosiders all derivatio sequeces possibly leadig to the desired strig. It works i a bottom-up way. The preseted algorithm works with grammars i Pettoe ormal form, as a expasio of Chomsky ormal form. Apart from the way CYK works, it also applies a set of restrictios to prevet icosistecies of the derivatio tree. If some otermial was used to apply both a cotext-sesitive ad a cotext-free rule, it could lead to occureces of otermials that would ot appear i the derivatio tree uder ormal coditios. I evet of such rule collisio, the sytax tree is split ito two versios, of which each propagates a differet rule. I the evet of failure of oe such tree, a substitute is chose ad the process is repeated util the strig is either accepted, or there are o other possible sytax trees to be examied ad the strig is rejected. This algorithm offers the ability to parse a iput strig accordig to ay user-specified cotext-sesitive grammar i Pettoe ormal form [2]. Sice every cotext-sesitive grammar ca be coverted to a equivalet grammar i Pettoe ormal form, the algorithm therefore works for ay such grammar. This algorithm always halts eve for cyclic ad ambiguous

2 grammars. Details of its fuctioality will be further described i the followig sectios. 2. Urestricted Grammars Grammar is a ordered pair, G = (Σ,R), where: Σ is the alphabet composed of disjoit sets of otermials, N, ad termials,, R is a set of productio rules. N cotais the start symbol, deoted by S []. Urestricted grammars are the geeral family of grammars they properly iclude cotext-sesitive, cotext-free, ad regular grammars. Grammars of this family ca be recogized by a Turig machie or two-pushdow automato. Neither side of rules belogig to these grammars is restricted i ay way it ca cosist of ay strig of otermial ad termial symbols. The family of laguages geerated by urestricted grammars is closed uder the operatios of itersectio, uio, cocateatio ad iteratio, but ot uder set differece. Urestricted grammars geerally use Kuroda ormal form [3], which ormalizes all grammar rules ito oe of the followig forms: A BC, AB DC, A a, A ε. This form is similar to Chomsky ormal form for cotext-free laguages it adds the cotext-sesitiveess of AB DC. If A = D, the grammar is also i Pettoe ormal form, a special case of Kuroda ormal form. Ay grammar ca be algorithmically coverted ito a equivalet grammar i ormal form [2] the algorithm works as follows. It modifies the origial grammar, ad after each made chage, it moves the set of productio rules that adhere to the ormal form to the ew grammar. These chages are made as follows:. I all rules, replace every termial by a otermial ad iclude these productios i the ew grammar. 2. I all rules whose left-had side is loger tha the right-had side, exted the left-had side by a otermial that fially derives ito ε. 3. For each rule, i which a otermial derives ito a differet otermial, add a rule whose right-had side is composed of the derived otermial followed by the left-had side otermial of every rule the derived otermial is used i. 4. For every cotext-free productio rule whose right-had side is loger tha two, split it ito several coectig rules. 5. For every cotext-sesitive productio rule, pick the first pair of otermials o the left-had side of the rule ad esure that the pair derives ito the first otermial o the right-had side of the rule followed by a supplemetary otermial. Subsequetly, add a rule whose left-had side cosists of the supplemetary otermial ad the rest of the origial rule. Right-had side of this rule cosists of the remaiig otermials of the right-had side of the origial rule. I case the right-had side of the origial rule was of a legth greater tha three, further modify the rule util it coforms to the ormal form. 3. Algorithm Descriptio Give a grammar, G = ( G Σ, G R), i Pettoe ormal form, ad a strig w = a a 2...a with a i G, i, for some, this algorithm decides, whether w is a setece of L(G) i a bottom-up way. For its work, it uses a matrix of sets, CV CurretVersio, where CV [i, j], i j. Each such versio represets a versio of the sytax tree. Other tha this, the algorithm uses several sets that hold additioal iformatio the most importat oe beig V Versios, whose members are versios of the CV matrix the algorithm has geerated so far. Therefore, it cotais differet versios of the derivatio tree, of which all are mutually exclusive. Every member of CV [i, j] has its ow BLQ set every BlackListQueue cotais coordiates of all cells of the correspodig matrix that cotai cotext-sesitive otermials, but whose igored coordiates have ot bee set yet. As metioed above, every cell of the CV matrix cotais a set of otermials that are likely i this phase of the derivatio process. For every such otermial A, there exists a Predecessor set P A, cotaiig refereces to all of its predecessors. Ay otermial ca be marked as cotext-sesitive, which idicates that it origiated from a cotext-sesitive rule. The algorithm starts the parsig process by scaig the iput strig, addig A to CV [i,i], if A a i G R. For every such otermial, a empty set of its predecessors is costructed. Afterwards, the algorithm iterates through the matrix ad costructs the sets CV [i, j]. For every pair CV [i, j] ad CV [ j +,k], it checks whether CV [i, j] is a member of BLQ, which idicates that the set cotais cotext-sesitive oter-

3 mials, but its igored coordiates have ot bee appoited yet. These are the coordiates of the first right eighbour CV [i, j] is compared with, whose otermials must ot be used i combiatio with the cotextsesitive otermials of the examied set this is to esure the cosistecy of the derivatio tree. If the igored coordiates have ot bee set yet, those of CV [ j +,k] are used, ad CV [i, j] is removed from BLQ. The behaviour of the followig step is depedet o whether the igored coordiates of CV [i, j] are equal to CV [ j +,k]. If they are uequal, the algorithm proceeds as follows. Ay otermial B that satisfies A CV [i, j], C CV [ j +,k], AB AC G R, is added to CV [ j +,k]. This implies that C a j+... a k, ad therefore B a j+...a k. A empty set P B, used to refer to predecessors of the otermial, is costructed. A referece to otermial B is the added to sets P A ad P C, i case either of the origial otermials is to be deleted i the evet of versio splittig. This process is repeated util CV [ j +, k] caot be exteded aymore. If ay cotextsesitive otermials were added ad the set s igored coordiates have ot bee set yet, a referece to this set is added to BLQ. Ay otermial A is the added to the set CV [i,k], if it satisfies B CV [i, j], C CV [ j +,k], A BC G R, because B a i...a j, C a j+...a k, ad therefore A a i...a k. A empty set of predecessors, P A, is created for each added otermial A. A referece to otermial A is added to sets P B ad P C. If the igored coordiates are equal to CV [ j +,k], a copy of the matrix is created before the first matchig rule is applied. All cotext-sestive otermials of CV [i, j], icludig their predecessors, are deleted from the copy usig predecessor sets, P A for a otermial A, to recursively detect all of a otermial s predecessors. All of the subsequetly added otermials are the saved to the copy istead of CV. After the sca is completed, this copy is added to V. Oce o set ca be exteded, it is examied whether S CV [,], so S a...a. If the check is passed, the algorithm aouces ACCEPT. Otherwise, CV is removed from V, a member is appoited as the ew CV, ad parsig is cotiued for this versio of the matrix. If V is empty, ad therefore o ew CV ca be appoited, there are o alterative derivatio trees to be costructed. I such case, the algorithm aouces REJECT. The mai part of the algorithm, which describes otermial set matrix traversal, is described i Algorithm. This part also describes the fial testig for presece of the startig otermial. Algorithm Matrix traversal Iput: a grammar, G = ( G E, G R), i Pettoe ormal form; w = a a 2...a with a i G, i, for some. Output: ACCEPT if w L(G); REJECT if w / L(G). : itroduce set V = ; 2: itroduce matrix of sets CV, where CV [i, j] = for i j, ad add CV to V ; 3: itroduce set BLQ = ; 4: for i = to do 5: if A a i G R the 6: add A to CV [i, i]; 7: itroduce set P A = holdig refereces to A s predecessors; 8: parse loop: 9: repeat 0: for level = to do : for i = to level do 2: set k to i + level; 3: for offset = 0 to level do 4: set j to i + offset; 5: ApplyRules(i, j,k); 6: util o chage; 7: if S CV [,] the 8: ACCEPT 9: else 20: remove CV from V ; 2: if V the 22: pick a elemet of V ad set it as CV ; 23: else REJECT; 24: goto parse loop; The idices acquired i Algorithm are used to locate sets represetig substrigs of the iput strig, ad to subsequetly appoit coordiates of the otermial that jois these substrigs. The process is further described by Algorithm 2. Because the grammar, G, is fiite, the algorithm always fiishes. The proof of soudess is aalogical to that of the origial CYK algorithm [] (p. 20 2). Time complexity of the algorithm depeds o both size of the grammar, G, ad legth of the iput strig,. I the worst case sceario, each iteratio of the outermost loop reduces a sigle rule that triggers versio splittig. I this case, the complexity reaches

4 Algorithm 2 Rule Applicatio : procedure APPLYRULES(i, j, k) 2: if CV [i, j] BLQ the 3: set its igored coordiates to [ j +,k]; 4: remove CV [i, j] from BLQ; 5: if A CV [i, j], C CV [ j +,k], rhs(p) = AC for some A, C G N, p G R the 6: if coordiates igored by CV [i, j] are uequal to [ j +,k] the 7: if AB AC G R for some B G N the 8: add B to CV [ j+,k] ad mark it as cotext-sesitive; 9: if igored coordiates of CV [ j +, k] are ot set the 0: add CV [ j +,k] to BLQ; : if B AC for some B G N the 2: add B to CV [i, k]; 3: itroduce set P B = holdig refereces to B s predecessors; 4: add a referece to the otermial B to sets P A ad P C ; 5: else 6: if this istace of A is cotext-sesitive the cotiue to ext otermial; 7: create a copy of CV, its BLQ, ad all sets of predecessors; 8: remove all predecessors of cotextsesitive otermials i copy[i, j]; 9: remove all cotext-sesitive otermials from copy[i, j]; 20: if AB AC G R for some B G N the 2: add B to copy[ j +,k] ad mark it as cotext-sesitive; 22: if igored coordiates of copy [ j +,k] are ot set the 23: add copy[ j +,k] to the copy s BLQ; 24: if B AC G R for some B G N the 25: add B to copy[i,k]; 26: itroduce set P B = holdig refereces to B s predecessors; 27: add a referece to the otermial B to sets P A ad P C ; 28: add the copy to V ; O( 3 G 3 ), as described by Equatio. I that case, the algorithm creates a total of G additioal matrices, of which each cell ca hold G reduced otermials. That also meas the space complexity is O( 2 G 3 ), as described by Equatio 2. t = ( G + ) 2 i= i j= i k=0 G t = (2 ) G ( G + ) 2 6 O(t) = O( 3 G 3 ) s = ( G + ) i= G i ( G + )( G + )( G + ) s = 2 O(s) = O( 2 G 3 ) () (2) The experimetal results support these assumptios, as they show a steady growth whe icreasig oe of the parameters, ad a expoetial growth whe values of both are high. The values listed i Table are the average of a total of three rus with the specified parameters. The data was acquired usig Memcheck ad Callgrid utilities available as a part of the Valgrid memory checker suite. This data is further visualized ad compared to the aalytical values i Figure. Table. Time ad space complexity test results for a total of 42 iput grammars ad strigs with a overall success rate of 40.5 %. G Istructios Memory [B] Prototype Implemetatio The workig prototype, which implemets the algorithm, is programmed i C++. The laguage was chose because it allows object-orieted programmig, as well as its stadard cotaiers ad operatios available o them.

5 (a) Time complexity of the preseted algorithm estimated at O( 3 G 3 ). (b) Space complexity of algorithm reaches O( 2 G 3 ) i worst case sceario. Figure. Time ad space of the preseted algorithm both reach at least cubic values because of the cotextsesitive ature of the discussed topic. The purple grids represet aalytical values of estimated complexities, while the red poits represet average experimetal evalues from Table. The program is divided ito several classes, of which each has a distict role i the process of parsig user settigs, versioig, ad parsig the iput strig to fially decide membership of the iput strig: Grammar Adapter This module parses rules of a u- ser grammar accordig to the predefied format. Before savig a rule, it checks if the rule adheres to restrictios of Pettoe ormal form. Processed rules are subsequetly added to Grammar, as described below. Iput Strig Adapter This module works i a way similar to the previous oe. It scas the iput file ad coverts it ito a vector of termials that are later parsed accordig to Grammar. The tokes are divided by a white space, therefore the iput strig ca be of ulimited legth. Grammar Structure This module stores grammar rules used to detect derivatios. It cosists of three separate lists of rules, of which each set is used durig a differet phase of the parsig process termial rules, which are used to costruct the startig sets of otermials, cotext-free rules ad cotext-sesitive rules, which are used durig the later phases of the process. This module is resposible for verifyig right-had side match of the appropriate rule durig the parsig, as well as retrievig the left-had side otermials i case of a positive match. Versioig System This module is resposible for storig ad maagig viable versios of the otermial matrix the program has geerated so far. Each versio of the matrix has a correspodig blacklist queue of cotext-sesitive sets, as described i Sectio 3. The module keeps a referece to the curret versio of the matrix, which is modified durig the applicatio of productio rules. This module aouces the fial rejectio of the iput strig, if uable to appoit a ew versio of the parsig matrix for processig. Parsig Matrix This class presets a two-dimesioal map of matrix cells. Each such cell cotais a set of otermials used durig the parsig process, ad lists of rules that have bee used to geerate the member otermials of the correspodig set. These lists are used to prevet both edless loopig of cotext-sesitive rule applicatio, ad applicatio of rules that would lead to addig a otermial that had previously bee deleted from the set because of a cotext collisio. These modules are ecapsulated by a sigle Parser object. It parses the iput files, ad saves the adapters output. As described i algorithm, it creates the first versio of the matrix, iitializes it by creatig the startig sets of otermials accordig to the previously obtaied grammar, ad sets it as curret versio. Afterwards, the secod phase of the parsig process begis. Oce a matrix caot be chaged ay further, it checks for the presece of the startig otermial. I case of failure, the versioig system attempts to appoit a ew curret versio. 5. Coclusios This paper deals with the topic of cotext-sesitive grammars as special cases of urestricted grammars, ad applicatios of their ormal forms. It presets a algorithm capable of parsig these grammars ad a software implemetatio of this algorithm that implemets this algorithm. The algorithm is based o Cocke-Youger-Kasami parsig algorithm ad therefore works i a bottom-

6 Parser +parse() uses IputStrigAdapter -tokes: vector<strig> implemets Adapter <<Eumeratio>> RuleType #iput_file: ifstream +parseiputfile() TermialRule CotextFreeRule CotextSesitiveRule cosists of uses GrammarAdapter Grammar ProductioRule uses parses #iput_file: ifstream -applychages() -applycotextfreerule() -applycotextsesitiverule() -ruletype: RuleType +matchesrightside() +getleftsidenotermial() cosists of VersioSet ParsigMatrix Termial Coordiates +colum: Iteger +row: Iteger stores uses curret versio has a +traversematrix() cosists of cosists of -ame: strig exteds has blacklist queue VersioCofiguratio +has_versio_chaged: bool +isiblacklistqueue() +isversiosuccessful() is located at MatrixCell -wascfruleused: vector<bool> -wascsruleused: vector<bool> +addnotermialtoset() Notermial cotais a set of -is_cotext_sesitive: bool +addpredecessor() Figure 2. Diagram of classes as implemeted i the workig prototype. A istace of Parser class ecapsulates the two mai modules, of which each has a distict role Adapter scas the iput files holdig iput strig ad grammar, ad VersioSet stores the cosidered versios of the parsig matrix. up way. It is capable of parsig ay cotext-sesitive grammar i Pettoe ormal form, icludig ambiguous grammars. As this work focuses o a topic that has ot bee thoroughly researched yet, it aims to serve maily as a base for subsequet research. I the future, I would like to expad the algorithm to accept grammars i additioal ormal forms, which would allow for more extesive applicatio. Ackowledgemets I would like to thak my supervisor Prof. Alexader Medua, CSc. for his help. RNDr. Refereces [] MEDUNA, Alexader, Elemets of compiler desig. Boca Rato: Auerbach Publicatios, xiii, 286 p. ISBN [2] MEDUNA, Alexader, Automata ad laguages: theory ad applicatios editio. Lodo: Spriger, xv, 96 p. ISBN [3] ROZENBERG, Grzegorz ad Arto SALOMAA, 997. Hadbook of formal laguages: word, laguage, grammar. Berli: Spriger, xvii, 873 pages; 23 cm. ISBN