Ling/CMSC 773 Take-Home Midterm Spring 2007

Transcription

1 Ling/CMSC 773 Take-Home Midterm Spring 2007 Ground rules. This is a take-home exam. Each person should be working completely independently no communication of any kind. There will be no communication with the instructor or TA either. If you have questions or believe that there is an error of some kind, do your best to answer, making whatever assumptions you feel are appropriate and necessary and saying explicitly what they are. Exams must be turned in either by to Philip, or by hardcopy under Philip s office door, AVW3143; in both cases the deadline for receipt is Monday, April 2, at 9am. (Philip or an assistant will be picking up the hardcopies at 9am sharp.) Use of notes and other materials. This exam is open book: you are free to use the textbook, your notes, and any other hardcopy materials (even, say, another textbook). However, you may not use the Web. Once the exam is handed out, and until it is turned in, you are forbidden to search the Web for explanations, notes, tutorials, or any other materials. The only Web pages you are permitted to access in relation to this examination are which contains errata for the textbook, and any page explicitly linked from the course Web page or syllabus. (But not the transitive closure of that linking relationship!) Programming. This exam is to be done entirely without computer programming. The only permitted use of software is for word processing (e.g. MS Word, LaTeX, etc.) or related creation of charts/graphs. Use of spreadsheet software (e.g. Excel) to do numerical calculations is not permitted, although you can use Excel to create graphs and charts if so desired. (The use of calculators is, of course, allowed.) Honor pledge. On the first page of your examination, in addition to your name, you are expected to include and sign (or type, for e-submissions) the following: I pledge on my honor that I have not given or received any unauthorized assistance on this examination. For purposes of this exam unauthorized assistance includes access to non-permitted notes, computer programs, or other materials as specified above. Other notes. Plesae turn in your answers in the same order that questions appear on the exam. Make sure to clearly identify which answers go with which questions, even for sub-parts of questions. (I don t want to have to guess where you make the transition from part (a) to part (b), etc.) You are encouraged to show all your work, so that you can be awarded partial credit. You are welcome to turn in material that includes scratch paper or scratch space, if you like, but if you do so then you need to clearly indicate what is intended to be graded. Unless the question says otherwise, for numerical calculations, you do not need to do the arithmetic for example, in most cases, if you re computing a product of probabilities, you can give your answer in the form , or whatever. Just make sure it s completely clear where the numbers come from. Please return this document to the instructor at the next class, with your name clearly written at the top. 1

2 1 Short Answer Questions [30 points total] Answer each question clearly and briefly. (Even if you make sure to be pretty thorough which you should, in order to obtain partial credit these are short answers, not long essays.) (a) Give a constituency tree for Many students said they enjoyed the interesting exam (using the usual sort of notation from class, please, nothing linguistically fancy). Give a head table for the context-free rules needed to construct the tree, i.e. context-free rules annotated to show which constituent is the head. Then show explicitly, step-by-step, how to convert your constituency tree into an unlabeled dependency tree. (b) For this question, assume data sparseness is not a problem, i.e. assume all probabilities can be estimated accurately. Consider the difficulty that a simple probabilistic CFG has in distinguishing the likelihood of the sentences people eat roasted peanuts and peanuts eat roasted people. Briefly describe one solution to this problem and its advantages and disadvantages. Now consider the sentence Children sometimes munch chocolate soldiers on Christmas morning. With the solution you just described, is the probability of this sentence likely to have a value that accords with your intuition? Explain why or why not. (c) Suppose you have an unfair coin that comes up heads 7/8 of the time when you flip it. Suppose you flip the coin 1000 times and report the outcome each time, paying one penny per bit for your communication. If you re thrifty (you want to pay as little as possible) and smart (you ve taken this class), what s the lowest amount you could expect to get away with paying? Justify your answer. (d) A former Amazon employee once said: We sold more books today that didn t sell at all yesterday than we sold today of all the books that did sell yesterday. (To state this in plainer English, If you look at our infrequently-sold books, the total number we sell (let s call it N) is very large, even if each individual book is only bought by a few people. And if you look at N, it s bigger than the total sales of the frequently sold books! ) George Zipf would be unsurprised. Briefly explain why. (e) Briefly explain/illustrate how a measure of uncertainty can also be considered a measure of surprise and a measure of quantity of information. (f) Here s an argument: Consider the sentences (i) Hold the newsreader s nose squarely, waiter, or friendly milk will countermand my trousers, and (ii) Countermand friendly, hold milk my, newsreader s nose or squarely the trousers waiter will. Neither (it s fair to assume) has ever occurred in the prior experience of a speaker of English, and yet any such speaker would readily identify the former as grammatical and the latter as ungrammatical. Therefore one cannot associate the notion grammaticality of a sentence in English with the notion likelihood of a sentence in English. Briefly but convincingly demolish this argument. 2

3 2 EM [20 points] Consider the following hidden model variation on a bigram model for word sequences. As in the usual bigram model, we express the probability of an entire sequence w 1 w 2... w T by (1) T p(w 1 w 2... w T ) = p(w 1 ) p(w t w t 1 ). t=2 However, the parameters used in the product are defined as follows: (2) p(w t w t 1 ) = C p(w t c)p(c w t 1 ) i=1 In plain English, the generative story for this model is the following. Instead of generating the next word w t based on the previous word w t 1, as in a usual bigram model, we generate a class c based on w t 1, and then we generate w t based on c. 1 So the probability of choosing the next word w t is a sum of probabilities, one for each hidden class. The probability contributed to the choice of w t for each class c, namely p(w t c)p(c w t 1 ), represents the joint probability of two events: picking c based on w t 1, and then picking w t based on c. Since c is hidden, we have to sum up over the different possibilities. Intuitively, the hidden class c can be viewed as capturing the general properties of w t 1 that are relevant for generating the next word. Or you can think of going through hidden class c to get from w t 1 to w t. As you can see, there are two sets of parameters in this model. The first set is the word-to-class probabilities p(c w i ), and the second set is the class-to-word probabilities p(w j c). (Where both w i and w j range over the entire vocabulary, and c ranges over the set of C hidden classes.) Because of the way the model is structured, there s an EM algorithm for estimating these parameters that is much simpler than the Forward-Backward algorithm. In particular, there s no need at all for dynamic programming. 2 For the sake of consistent notation, please use N(x, y) as your notation for counts. E.g. N(w i, w j ) would be the number of times w i is followed by w j. (a) (2 points) In addition to capturing generalizations (by associating words with abstract classes that are learned automatically), an advantage of this model is that in practice it has far fewer parameters than a usual (non-hidden) bigram model. Let the size of the vocabulary be V and the number of classes be C. How many classes can you use in the hidden model before the total number of parameters exceeds the number of parameters in the usual (non-hidden) bigram model? If you re comparing the total number of parameters in the two models, What does the comparison look like for typical values of C = 32 and V = 50000? (b) (2 points) Suppose the classes c were observable rather than hidden. Express the maximum likelihood estimate for the probability p(w j c) in terms of counts. (c) (2 points) Suppose the classes c were observable rather than hidden. Express the maximum likelihood estimate for the probability p(c w i ) in terms of counts. (d) (7 points) Recall that for many cases of EM algorithms, the basic structure of the algorithm can be described as follows: 1. Set initial values for parameters µ 1 Also, as usual, we can assume w 1 is always a special start word that always starts an observed sequence with probability 1. 2 Intuitively, this is because the choice of the hidden class at every step depends only on what s observable, i.e. hidden classes are independent of each other given the intervening word. 3

4 2. E-step: Figure out expected counts of relevant events, where those events typically involve both observable and hidden values, using the current parameters µ to determine what s expected. 3. M-step: Use the expected counts to compute µ new, a new set of parameters. In the EM algorithms we ve studied, this is a maximum likelihood estimate, i.e. just normalizing (expected) counts. 4. If the probabilities have converged (or if we ve done some maximum number of iterations), stop; otherwise let µ = µ new and go back to the E-step for the next iteration. If you read about the EM algorithm for this model, the updating of the parameters is described as having the following M-step for the two sets of parameters. w p new (c w i ) = j N(w i, w j )p(c w i, w j ) (3) c w j N(w i, w j )p(c w i, w j ) w p new (w j c) = i N(w i, w j )p(c w i, w j ) (4) w i N(w i, w j )p(c w i, w j ) The E-step is described as: (5) w j p(c w i, w j ) = p(w j c)p(c w i ) c p(w j c )p(c w i ) In words, p(c w i, w j ) can be thought of as the probability that c was the hidden state that got used when going from w i to w j. Explain why equation 4 is the way to calculate the new value for p new (w j c) given the previous guesses for the parameters. Partial credit will be assigned for good explanations, but a perfect answer will show via equations (along with written explanation) how to derive equation 4 using the non-hidden maximum likelihood estimate (part b) as the starting point. (I.e. the form of the explanation will follow the same general schema we used for deriving the update for HMM transition probabilities a ij, starting with the maximum likelihood estimate you d use if all the state transitions were visible. Though, as was mentioned, no dynamic program is needed.) (e) (7 points) Same as part (d), but explain why equation 3 is the way to update p new (c w i ). For those who have ever looked more formally at EM: you will not receive credit for deriving the EM updates from the model s likelihood function, using Lagrange multipliers to optimize, etc. I don t want to see a Q function. I m looking for an understanding of the updates in relation to the non-hidden maximum likelihood estimate, as was our focus in discussions of the Forward-Backward and Inside-Outside algorithms. 4

5 3 N-gram models [15 points] Consider the following problem, known as language identificiation : given a previously unseen string of natural language text, what language is it written in? This is often solved using n-gram models. Assume you have training samples of 20,000 words in each of k languages, {L 1,..., L k }. Furthermore, to keep things simple, assume that all of these languages are alphabetic (i.e. not ideographic like many Asian languages) and that all data is in Unicode, and that all k languages use significantly overlapping alphabets (e.g. you could suppose they all use a roman alphabet with accents/diacritics).. If you need to for concreteness, you can assume the genre is newswire, and that there is no markup. Now, imagine that you are asked to design a solution to the language identification problem based on n-grams. (a) (5 points) Would you use n-grams composed of words or characters? Justify your answer in quantitative terms, clearly stating any assumptions you make (e.g. about sizes of vocabularies, sizes of alphabets, or any other relevant quantities). (b) (5 points) Explain in detail how, given a new piece of text in one of the k languages, you would take a Bayesian approach to identifying which language L i it is written in. (c) (5 points) Suppose you have read about two different kinds of n-gram model (e.g. a bigram model and a trigram model, or a standard bigram model and the aggregate bigram model of the previous question) and you want to know which kind performs better on this problem. Under the assumption that you have only the small amount of data in each language as described above, explain in detail how you would conduct an evaluation to assess which of the two kinds of models is better for this problem. 5

6 4 Relative Entropy [20 points] Selectional preferences are semantic constraints that a predicate places on its arguments. For example, the verb drink prefers that its objects be in the semantic class beverage, which is why John drank the wine sounds a lot better than, say, John drank the toaster. In the early 1990s, the following was proposed as a probabilistic model of selectional preferences: S object (v) = D(p(c v) p(c)) = Pr(object is in class c verb is v) log c Classes A object (v, c) = Pr(object is in class c verb is v) log Pr(object is in class c verb is v). Pr(object is in class c) Pr(object is in class c verb is v). Pr(object is in class c) The value S object (v) was referred to as the selectional preference strength of verb v, and the value A object (v, c) was called the selectional association between the verb v and a particular semantic category c. (a) Assuming probabilities are estimated accurately (for example, pretend the classes are observable), and assuming that verb-object frequencies in the corpus are a reasonably accurate reflection of the real world, would the model correctly predict that A object (drink, beverage) should be higher than A object (drink, appliance)? Explain why or why not. (b) Below, the graph on the left shows A object (v 1, c) and the graph on the right shows A object (v 2, c), for two verbs v 1 and v 2. (The x-axis ranges over possible semantic categories c, and each bar gives the value of A object (v, c).) As you can see, one of the verbs is eat and the other one is find. Even without the labels, you would have been able to identify which was which, based on what you know about the model (and the assumptions we ve made). Explain how, with explicit reference to the formal definition of the model. A(v,c) x Selectional Profile: find profile.find Class ID x 10 6 A(v,c) x Selectional Profile: eat profile.eat Class ID x (c) The person who proposed this model claimed that S object (v) models, quantitively, how much information the verb carries about the semantic category of its direct object. Explain why this claim is true. 6

7 5 Parsing and Evaluation [15 points] You have been hired by a AwesomeSearchEngine.com to make recommendations concerning language technology. They have been approached by Whizdee, Inc., a startup company, and in a sales presentation Whizdee s sales representative says the following: We re doing very exciting work on parsing, and our results are very impressive. In one experiment, we trained one of our parsers on 80% of the Penn Treebank and tested on the other 20%, and its labeled recall on constituent boundaries was 98.2%. We also did an experiment, training on the same data, where the labeled precision for constituent boundaries was 97.6%. With numbers like that, how could you lose? (a) Consider the true parse T, and the system parse P, below: [T] (S (NP the prof) (VP (VPRT looked up) (NP the grade)) (ADVP today)) [P] (S (NP the prof) (VP looked (PP up (NP the grade)) (ADVP today))) What are the values for labeled precision and labeled recall? Note that I ve omitted all part of speech labels because they re not used in constituent recall/precision calculations. Every uppercase symbol in the parse is a constituent label. (b) Should people be impressed by Whizdee s numbers? Explain why or why not. (c) Whizdee s decided you re so smart that they, too, want to pay you as a consultant. They ve got a questionanswering search engine that uses a parser, and a contract from the New York Times. They paid minimum wage to impoverished linguistics grad students in New York City to create parse trees for 20,000 New York Times articles written between March 15, 1997 and March 15, Whizdee plans to make their system available to the public starting April 1, but the New York Times insists on a formal parser evaluation first. One of Whizdee s scientists says that they should evaluate their parser by doing 10-fold cross validation. Another of their scientists says that they should evaluate it by training on the data up to March 15, 2006 and testing on the rest. Explain what the two competing evaluation approaches are and discuss the advantages and disadvantages of each approach. 7