COMS W4705, Spring 2015: Problem Set 2 Total points: 140

Transcription

1 COM W4705, pring 2015: Problem et 2 Total points: 140 Analytic Problems (due March 2nd) Question 1 (20 points) A probabilistic context-ree grammar G = (N, Σ, R,, q) in Chomsky Normal Form is deined as ollows: N is a set o non-terminal symbols (eg,,, etc) Σ is a set o terminal symbols (eg, cat, dog, the, etc) R is a set o rules which take one o two orms: X Y 1 Y 2 or X N, and Y 1, Y 2 N X Y or X N, and Y Σ N is a distinguished start symbol q is a unction that maps every rule in R to a probability, which satisies the ollowing conditions: r R, q(r) 0 X N, X α R q(x α) = 1 Now assume we have a probabilistic CFG G, which has a set o rules R which take one o the two ollowing orms: X Y 1 Y 2 Y n or X N, n 2, and i, Y i N X Y or X N, and Y Σ Note that this is a more permissive deinition than Chomsky normal orm, as some rules in the grammar may have more than 2 non-terminals on the right-hand side An example o a grammar that satisies this more permissive deinition is as ollows: Vt 08 Vt PP 02 DT NN NN 03 PP 07 PP IN 10 Vt saw 10 NN man 07 NN woman 02 NN telescope 01 DT the 10 IN with 05 IN in 05

2 Question 1(a): Describe how to transorm a PCFG G, in this more permissive orm, into an equivalent PCFG G in Chomsky normal orm By equivalent, we mean that there is a one-to-one unction between derivations in G and derivations in G, such that or any derivation T under G which has probability p, (T ) also has probability p (Note: one major motivation or this transormation is that we can then apply the dynamic programming parsing algorithm, described in lecture, to the transormed grammar) Hint: think about adding new rules with new non-terminals to the grammar Question 1(b): how the resulting grammar G ater applying your transormation to the example PCFG shown above Question 2 (20 points) Nathan L Pedant decides to build a treebank He inally produces a corpus which contains the ollowing three parse trees: John V1 BAR ally V1 BAR Fred V1 BAR said COMP declared COMP pronounced COMP that that that ally AD Bill AD Je AD V2 loudly V2 quickly V2 elegantly snored ran swam Clarissa Lexica then purchases the treebank, and decides to build a PCFG, and a parser, using Nathan s data Question 2(a): how the PCFG that Clarissa would derive rom this treebank Question 2(b): how two parse trees or the string Je pronounced that Fred snored loudly, and calculate their probabilities under the PCFG Question 2(c): Clarissa is shocked and dismayed, (see 2(b)), that Je pronounced that Fred snored loudly has two possible parses, and that one o them that Je is doing the pronouncing loudly has relatively high probability, in spite o it having the AD loudly modiiying the higher verb, pronounced This type o high attachment is never seen in the corpus, so the PCFG is clearly missing something Clarissa decides to ix the treebank, by altering some non-terminal labels in the corpus how one such transormation that results in a PCFG that gives zero probability to parse trees with high attachments (Your solution should systematically reine some non-terminals in the treebank, in a way that slightly increases the number o non-terminals in the grammar, but allows the grammar to capture the distinction between high and low attachment to s)

3 Question 3 (20 points) Consider the CKY algorithm or inding the maximum probability or any tree when given as input a sequence o words x 1, x 2,, x n As usual, we use N to denote the set o non-terminals in the grammar, and to denote the start symbol The base case in the recursive deinition is as ollows: or all i = 1 n, or all X N, π(i, i, X) = { q(x xi ) i X x i R 0 otherwise and the recursive deinition is as ollows: or all (i, j) such that 1 i < j n, or all X N, π(i, j, X) = max (q(x Y Z) π(i, s, Y ) π(s + 1, j, Z)) X Y Z R, s {i(j 1)} Finally, we return π(1, n, ) = max t T G (s) p(t) Now assume that we want to ind the maximum probability or any balanced tree or a sentence Here are some example balanced trees: B A D C B A e D C F G B A B A e g D C F G D C F G e g e g It can be seen that in balanced trees, whenever a rule o the orm X -> Y Z is seen in the tree, then the non-terminals Y and Z dominate the same number o words NOTE: we will assume that the length o the sentence, n, is n = 2 x or some integer x For example n = 2, or 4, or 8, or 16, etc

4 Question 3(a): Complete the recursive deinition below, so that the algorithm returns the maximum probability or any balanced tree underlying a sentence x 1, x 2,, x n Base case: or all i = 1 n, or all X N, π(i, i, X) = { q(x xi ) i X x i R 0 otherwise Recursive case: (Complete) Return: π(1, n, ) = max t T G (s) p(t) Programming Problems (due March 12th) Please read the submission instructions, policy and hints on the course webpage In this programming assignment, you will train a probabilistic context-ree grammar (PCFG) or syntactic parsing In class we deined a PCFG as a tuple G = (N, Σ,, R, q) where N is the set o non-terminals, Σ is the vocabulary, is a root symbol, R is a set o rules, and q is a probabilistic model We will assume that the grammar is in Chomsky normal orm (CNF) Recall rom class that this means all rules in R will be either binary rules X Y 1 Y 2 where X, Y 1, Y 2 N or unary rules X Z where X N and Z Σ You will estimate the parameters o the grammar rom a corpus o annotated sentences rom the Wall treet Journal Beore diving into the details o the corpus ormat, we will discuss the rule structure o the grammar You will not have to implement this section but it will be important to know when building your parser Rule tructure Consider a typical tagged sentence rom the Wall treet Journal There/ is/verb no/ asbestos/ in/adp our/pron products/ now/adv / The correct parse tree or the sentence (as annotated by linguists) is as ollows

5 VERB PP AD There is ADP ADV no asbestos in PRON now our products Note that nodes at the ringe o the tree are words, the nodes one level up are part-o-speech tags, and the internal nodes are syntactic tags For the deinition o the syntactic tags used in this corpus, see The Penn Treebank: An Overview The part-o-speech tags should be sel-explanatory, or urther reerence see A Universal Part-o-peech Tagset Both are linked to on the website Unortunately the annotated parse tree is not in Chomsky normal orm For instance, the rule has three children and the rule has a single non-terminal child We will need to work around this problem by applying a transomation to the grammar We will apply two transormations to the tree The irst is to collapse illegal unary rules o the orm X Y where X, Y N into new non-terminals X + Y, or example + There There The second is to split n-ary rules X Y 1 Y 2 Y 3 where X, Y 1, Y 2, Y 3 N into right branching binary rules o the orm X Y 1 X and X Y 2 Y 3 or example With these transormation to the grammar, the new parse correct parse or this sentence is

6 + There VERB is PP AD+ADV no asbestos ADP now in PRON our products Converting the grammar to CNF will greatly simpliy the algorithm or decoding; however, as you saw in Question 1, there are other transormations into CNF that better preserve properties o the original grammar All the parses we provide or the assignment will be in the transormed ormat, you will not need to implement this transormation, but you should understand how it works Corpus Format We provide a training data set with parses parse traindat and a development set o sentences parse devdat along with the correct parses parse devkey All the data comes or the Wall treet Journal corpus and consists o sentences o less than 15 words or eiciency In the training set, each line o the ile consists o a single parse tree in CNF The trees are represented as nested multi-dimensional arrays conorming to the JON standard JON is general data encoding standard (similar to XML) JON is supported in pretty much every language (see Binary rules are represented as arrays o length 3 DT ["", ["DT", ], ["", ]] Unary rules are represented as arrays o length 2 asbestos

7 ["", "asbestos"] For example, the ile treeexample has the tree given above ["", ["", ["", "There"]], ["", ["", ["VERB", "is"], ["", [" ", ["", "no"], ["", "asbestos"]], ["", ["PP", ["ADP", "in "], ["", ["PRON", "our"], ["", "products"]]], ["AD", ["ADV", "now"]]]]], ["", ""]]] The tree is represented as a recursive, multi-dimensional JON array I the array is length 3 it is a binary rule, i it is length 2 it is a unary rule as shown above As an example, assume we want to access the node or is by walking down the tree as in the ollowing diagram There 1 VERB 1 is PP AD+ADV no asbestos ADP now in PRON our products In Python, we can read and access this node using the ollowing code import json tree = jsonloads(open("treeexample")readline()) print tree[2][1][1][1] The irst line reads the tree into an array, and the second line walks down the tree to the node or is Other languages have similar (although possibly more verbose) libraries or reading in these arrays Consult the TAs i you are unable to ind a parser or your avorite language Tree Counts In order to estimate the parameters o the model you will need the counts o the rules used in the corpus The script count cg reqspy reads in a training ile and produces these counts Run the script on the

8 training data and pipe the output into some ile: python count cg reqspy parse traindat > cgcounts Each line in the output contains the count or one event There are three types o counts: Lines where the second token is NONTERMINAL contain counts o non-terminals Count(X) 17 NONTERMINAL indicates that the non-terminal was used 17 times Lines where the second token is BINARYRULE contain emission counts Count(X Y 1 Y 2 ), or example 13 BINARYRULE indicates that binary rule was used 13 times in the training data Lines where the second token is UNARYRULE contain unigram counts Count(X Y ) 14 UNARYRULE + products indicates that the rule + products was seen 14 times in the training data Question 4 (20 points) As in the tagging model, we need to predict emission probabilities or words in the test data that do not occur in the training data Recall our approach is to map inrequent words in the training data to a common class and to treat unseen words as members o this class Replace inrequent words (Count(x) < 5) in the original training data ile with a common symbol RARE Note that this is more diicult than the previous assignment because you will need to read the training ile, ind the words at the ringe o the tree, and rewrite the tree out in the correct ormat We provide a script python pretty print treepy parser devkey which you can use to view your trees in a more readable ormat The script will also throw an error i you output is not in the correct ormat Re-run count cg reqspy to produce new counts ubmit your code and the new training ile it produces Question 5 (40 points) Using the counts produced by count cg reqspy, write a unction that computes rule parameters q(x Y 1 Y 2 ) = Count(X Y 1 Y 2 ) Count(X) q(x w) = Count(X w) Count(X)

9 Using the maximum likelihood estimates or the rule parameters, implement the CKY algorithm to compute arg max t T G (s) p(t) Your parser should read in the counts ile (with rare words) and the ile parse devdat (which is just the sentence rom parse devkey without the parse) and produce output in the ollowing ormat: the tree or one sentence per line represented in the JON tree ormat speciied above Each line should correspond to a parse or a sentence rom parse devdat You can view your output using pretty print parsepy and check your perormance with eval parserpy by running python eval parserpy parser devkey prediction ile The evaluator takes two iles where each line is a matching parse in JON ormat and outputs an evaluation score Note: There is an slight change rom the class notes ome o these sentences are ragments, and they do not have as the root For the purpose o this assignment you should change the last line o the CKY algorithm to This should help perormance i π[1, n, ] 0 return π[1, n, ] else return max π[1, n, X] X N ubmit your program and report on the perormance o your model In addition, write up any observations you made Question 6 (20 Points) Consider the ollowing subtree o the original parse VERB PP AD The tree structure makes the implicit independence assumption that the rule DT is generated independently o the parent non-terminal, ie the probability o generating this subtree in context is p( VERB PP AD )p( ) However, oten this independence assumption is too strong It can be inormative or the lower rule to condition on the parent non-terminal as well, ie model generating this subtree as p( VERB PP AD )p(, parent=)

10 This is a similar idea to using a trigram language model, except with parent non-terminals instead o words We can implement this model without changing our parsing algorithm by applying a tree transormation known as vertical markovization We simply augment each non-terminal with the symbol o its parent VERB PP AD We apply vertical markovization beore binarization; the new non-terminals are augmented with their original parent Ater applying both transormations the inal tree will look like VERB PP AD There is a downside o vertical markovization The transormation greatly increases the number o rules N in the grammar potentially causing data sparsity issues when estimating the probability model q In practice though, using one level o vertical markovization (conditioning on the parent), has been shown to increase accuracy Note: This problem looks straightorward, but the diiculty is that we now have a much larger set o nonterminals N It is likely that a simple implementation o CKY that worked or the last question will not scale to this problem The challenge is to improve the basic ineiciencies o the implementation in order to handle the extra non-terminals The ile parse train vertdat contains the original training sentence with vertical markovization applied to the parse trees Train with the new ile and reparse the development set Run python eval parserpy parser devkey prediction ile to evaluate perormance Report on the perormance o your parser with the new trees In what ways did vertical markovization help or hurt parser perormance? Find an example where it improved your parser s output ubmit your program and any urther observations