Elements of Language Processing and Learning lecture 3

Transcription

1 Elements of Language Processing and Learning lecture 3 Ivan Titov TA: Milos Stanojevic Institute for Logic, Language and Computation

2 Today Parsing algorithms for CFGs Recap, Chomsky Normal Form (CNF) A dynamic programming algorithm for parsing (CKY) Extension of CKY to support unary inner rules Parsing for PCFGs Extension of CKY for parsing with PCFGs Language Modeling Parser evaluation (if we have time) After this lecture you should be able to start working on the assignment step 1.2 2

3 Syntactic parsing Syntax The study of the patterns of formation of sentences and phrases from words Borders with semantics and morphology sometimes blurred Parsing The process of predicting syntactic representations Syntactic Representations Afyonkarahisarlılaştırabildiklerimizdenmişsinizcesinee in Turkish mean "as if you are one of the people that we thought to be originating from Afyonkarahisar" [wikipedia] PN My Different types of syntactic representations are possible, for example: N dog S V ate VP D a N sausage Each internal node correspond to a phrase root poss nsubj dobj det root My dog ate a sausage PN N V D N Constuent (a.k.a. phrasestructure) tree Dependency tree 3

4 Constituent trees S Internal nodes correspond to phrases VP S a sentence PN My N dog V ate D a N sausage (Noun Phrase): My dog, a sandwich, lakes,.. VP (Verb Phrase): ate a sausage, barked, PP (Prepositional phrases): with a friend, in a car, Nodes immediately above words are PoS tags (aka preterminals) PN pronoun D determiner V verb N noun P preposition 4

5 Context-Free Grammar Context-free grammar is a tuple of 4 elements G =(V,,R,S) V - the set of non-terminals In our case: phrase categories (VP,,..) and PoS tags (N, V,.. aka preterminals) - the set of terminals Words - the set of rules of the form, where, R X! Y 1,Y 2,...,Y n n 0 X 2 V S, Y i 2 V [ is a dedicated start symbol S! VP N! girl! D N N! telescope! PN V! saw! PP V! eat PP! P

6 An example grammar V = {S, V P,, P P, N, V, P N, P } = {girl, telescope, sandwich, I, saw, ate, with, in, a, the} S = {S} R : Inner rules Preterminal rules (correspond to the HMM emission model aka task model) S! VP VP! V VP! V VP! VP PP! PP! D N! PN PP! P ( A girl) (VP ate a sandwich) (V ate) ( a sandwich) (VP saw a girl) (PP with a telescope) ( a girl) (PP with a sandwich) (D a) (N sandwich) (P with) ( with a sandwich) N! girl N! telescope N! sandwich PN! I V! saw V! ate P! with P! in D! a D! the 6

7 Why parsing is hard? Ambiguity Prepositional phrase attachment ambiguity S S VP PN I V saw VP D N a girl P with PP D a N telescope PN I V saw D N a girl P with PP D a N telescope Both a generated by the grammar 7

8 An example probabilistic CFG S! VP 1.0 ( A girl) (VP ate a sandwich) N! girl N! telescope VP! V 0.2 N! sandwich 0.1 VP! V VP! VP PP (VP ate) ( a sandwich) (VP saw a girl) (PP with ) PN! I V! saw ! PP! D N! PN ( a girl) (PP with.) (D a) (N sandwich) V! ate P! with P! in D! a 0.3 PP! P 1.0 (P with) ( with a sandwich) D! the 0.7 8

9 Parsing Parsing is search through the space of all possible parses e.g., we may want either any parse, all parses or the highest scoring parse (if PCFG): The probability by the arg max P (T ) PCFG model T 2G(x) Bottom-up: Set of all trees given by the grammar for the sentence x One starts from words and attempt to construct the full tree Top-down Start from the start symbol and attempt to expand to get the sentence 9

10 CKY algorithm (aka CYK) Cocke-Kasami-Younger algorithm Independently discovered in late 60s / early 70s An efficient bottom up parsing algorithm for (P)CFGs can be used both for the recognition and parsing problems Very important in NLP (and beyond) We will start with the non-probabilistic version 10

11 Constraints on the grammar The basic CKY algorithm supports only rules in the Chomsky Normal Form (CNF): C! x Unary preterminal rules (generation of words given PoS tags N! telescope, D! the, ) C! C 1 C 2 Binary inner rules (e.g., S! VP,! D N) 11

12 Constraints on the grammar The basic CKY algorithm supports only rules in the Chomsky Normal Form (CNF): C! x C! C 1 C 2 Unary preterminal rules (generation of words given PoS tags N! telescope, D! the, ) Binary inner rules (e.g., S! VP,! D N) Any CFG can be converted to an equivalent CNF Equivalent means that they define the same language However (syntactic) trees will look differently Makes linguists unhappy It is possible to address it but defining such transformations that allows for easy reverse transformation 12

13 Transformation to CNF form What one need to do to convert to CNF form Get rid of empty (aka epsilon) productions: C! Get rid of unary rules: C! C 1 N-ary rules: C! C 1 C 2...C n (n>2) Generally not a problem as there are not empty production in the standard (postprocessed) treebanks Not a problem, as our CKY algorithm will support unary rules Crucial to process them, as required for efficient parsing 13

14 Transformation to CNF form: binarization Consider! DT N V BG NN DT N VBG NN the Dutch publishing group How do we get a set of binary rules which are equivalent? 14

15 Transformation to CNF form: binarization Consider! DT N V BG NN DT N VBG NN the Dutch publishing group How do we get a set of binary rules which are equivalent?! DT X X! N Y Y! VBG NN 15

16 Transformation to CNF form: binarization Consider! DT N V BG NN DT N VBG NN the Dutch publishing group How do we get a set of binary rules which are equivalent?! DT X X! N Y Y! VBG NN A more systematic way to refer to new non-terminals! DT! DT DT N! VBG NN 16

17 Transformation to CNF form: binarization Instead of binarizing tules we can binarize trees on preprocessing: DT the N Dutch VBG publishing NN group Also known as lossless Markovization in the context of PCFGs DT Can be easily reversed on postprocessing the DT N Dutch VBG publishing NN group 17

18 Transformation to CNF form: binarization Instead of binarizing tules we can binarize trees on preprocessing: DT N VBG NN the Dutch publishing group DT the DT N This is exactly the transformation used in the code of the assignment Dutch VBG DT N VBG NN group 18

19 CKY: Parsing task We a given start symbol a grammar G =(V,,R,S) a sequence of words w =(w 1,w 2,...,w n ) Our goal is to produce a parse tree for w 19

20 CKY: Parsing task We a given start symbol [Some illustrations and slides in this lecture are from Marco Kuhlmann] a grammar G =(V,,R,S) a sequence of words w =(w 1,w 2,...,w n ) Our goal is to produce a parse tree for w We need an easy way to refer to substrings of w indices refer to fenceposts span (i, j) refers to words between fenceposts i and j 20

21 Key problems Recognition problem: does the sentence belong to the language defined by CFG? Is there a derivation which yields the sentence? Parsing problem: what is a derivation (tree) corresponding the sentence? Probabilistic parsing: what is the most probable tree for the sentence? 21

22 Parsing one word C! w i 22

23 Parsing one word C! w i 23

24 Parsing one word C! w i 24

25 Parsing longer spans C! C 1 C 2 Check through all C 1,C 2, mid 25

26 Parsing longer spans C! C 1 C 2 Check through all C 1,C 2, mid 26

27 Parsing longer spans 27

28 Signatures Applications of rules is independent of inner structure of a parse tree We only need to know the corresponding span and the root label of the tree Its signature [min, max, C] Also known as an edge 28

29 CKY idea Compute for every span a set of admissible labels (may be empty for some spans) Start from small trees (single words) and proceed to larger ones 29

30 CKY idea Compute for every span a set of admissible labels (may be empty for some spans) Start from small trees (single words) and proceed to larger ones When done, how do we detect that the sentence is grammatical? 30

31 CKY idea Compute for every span a set of admissible labels (may be empty for some spans) Start from small trees (single words) and proceed to larger ones When done, check if S is among admissible labels for the whole sentence, if yes the sentence belong to the language That is if a tree with signature [0, n, S] exists 31

32 CKY idea Compute for every span a set of admissible labels (may be empty for some spans) Start from small trees (single words) and proceed to larger ones When done, check if S is among admissible labels for the whole sentence, if yes the sentence belong to the language That is if a tree with signature [0, n, S] exists Unary rules? 32

33 CKY in action lead can poison S! VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

34 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 max = 1 max = 2 max = 3 S? Chart (aka parsing triangle) VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

35 CKY in action lead can poison max = 1 max = 2 max = 3 S? min = 1 min = 2 min = 3 lead can poison S! VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

36 CKY in action lead can poison max = 1 max = 2 max = 3 S? min = 1 min = 2 min = 3 lead can poison S! VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

37 CKY in action lead can poison max = 1 max = 2 max = 3 S? min = 1 min = 2 min = 3 lead can poison S! VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

38 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 max = 1 max = 2 max = 3 S? VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

39 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 max = 1 max = 2 max = S? VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

40 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3? 2? 3? VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

41 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3? 2? 3? VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

42 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V 2 N,M 3 N,V VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

43 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 2 N,M 3 N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

44 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2? N,M 3 N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

45 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2? N,M 3 N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

46 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 3 N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

47 CKY in action lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 3 Check about unary rules: no unary rules here N,V,VP S! VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

48 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 5 3? N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

49 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 5 3 S, V P, N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

50 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 5 3 S, V P, N,V,VP Check about unary rules: no unary rules here VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

51 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 6 5 3? S, V P, N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

52 CKY in action S! VP lead can poison min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 6 5 3? S, V P, N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

53 CKY in action S! VP lead can poison mid = 1 min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 6 S, 5 3 S, V P, N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

54 CKY in action S! VP lead can poison mid = 2 min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 6 S, S(?!) 5 3 S, V P, N,V,VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

55 CKY in action lead can poison mid = 2 min = 0 min = 1 min = 2 1 max = 1 max = 2 max = 3 N,V,VP 4 2 N,M 6 S, S(?!) S, V P, N,V Apparently the sentence is ambiguous,vpfor the grammar: (as the grammar overgenerates) 5 3 S! VP VP! VP VP! M V VP! V! N! N N! can N! lead N! poison M! can M! must V! poison V! lead Preterminal rules Inner rules

56 Ambiguity S S N lead M can VP V poison N lead N can VP V poison No subject-verb agreement, and poison used as an intransitive verb Apparently the sentence is ambiguous for the grammar: (as the grammar overgenerates)

57 CKY more formally Chart can be represented by a Boolean array chart[min][max][label] chart[min][max][c] Relevant entries have 0 < min < max apple n Here we assume that labels (C) are integer indices chart[min][max][c] = true if the signature (min, max, C) is already added to the chart; false otherwise. max = 1 max = 2 max = 3 min = 0 1 N,V,VP 4 6 S,V P, min = 1 2 N,M 5 S, V P, 3 N,V min = 2,VP 57

58 CKY more formally Chart can be represented by a Boolean array chart[min][max][label] chart[min][max][c] Relevant entries have 0 < min < max apple n Here we assume that labels (C) are integer indices chart[min][max][c] = true if the signature (min, max, C) is already added to the chart; false otherwise. max = 1 max = 2 max = 3 min = 0 min = 1 1 N,V,VP 4 2 N,M 6 5 S,V P, S, V P, In the assignment code we use a class Chart but its access methods are similar 3 N,V min = 2,VP 58

59 Implementation: preterminal rules 59

60 Implementation: binary rules 60

61 Unary rules How to integrate unary rules C! C 1? 61

62 Implementation: unary rules 62

63 Implementation: unary rules But we forgot something! 63

64 Unary closure What if the grammar contained 2 rules: A! B B! C But C can be derived from A by a chain of rules: A! B! C One could support chains in the algorithm but it is easier to extend the grammar, to get the transitive closure A! B B! C ) A! B B! C A! C 64

65 Unary closure What if the grammar contained 2 rules: A! B B! C But C can be derived from A by a chain of rules: A! B! C One could support chains in the algorithm but it is easier to extend the grammar, to get the reflexive transitive closure A! B B! C ) A! B B! C A! C A! A B! B C! C Convenient for programming reasons in the PCFG case 65

66 Implementation: skeleton 66

67 Algorithm analysis Time complexity? (n 3 R ) R, where is the number of rules in the grammar There exist algorithms with better asymptotical time complexity but the `constant' makes them slower in practice (in general) 67

68 Algorithm analysis Time complexity? (n 3 R ) R, where is the number of rules in the grammar There exist algorithms with better asymptotical time complexity but the `constant' makes them slower in practice (in general) 68

69 Algorithm analysis Time complexity? A few seconds for sentences under < 20 words for a nonoptimized parser (n 3 R ) R, where is the number of rules in the grammar There exist algorithms with better asymptotical time complexity but the `constant' makes them slower in practice (in general) 69

70 Algorithm analysis Time complexity? A few seconds for sentences under < 20 words for a nonoptimized parser (n 3 R ) R, where is the number of rules in the grammar There exist algorithms with better asymptotical time complexity but the `constant' makes them slower in practice (in general) 70

71 Practical time complexity Time complexity? (for the PCFG version) n 3.6 [Plot by Dan Klein] 71

72 Today Parsing algorithms for CFGs Recap, Chomsky Normal Form (CNF) A dynamic programming algorithm for parsing (CKY) Extension of CKY to support unary inner rules Parsing for PCFGs Extension of CKY for parsing with PCFGs Language Modeling Parser evaluation (if we have time) 72

73 Probabilistic parsing We discussed the recognition problem: check if a sentence is parsable with a CFG Now we consider parsing with PCFGs Recognition with PCFGs: what is the probability of the most probable parse tree? Parsing with PCFGs: What is the most probable parse tree? 73

74 CFGs PN I S V 0.5 saw 1.0 D VP N P a girl with PP D N S! VP VP! V VP! V VP! VP PP! PP! D N! PN PP! P N! girl N! telescope N! sandwich PN! I V! saw V! ate P! with P! in D! a D! the a telescope p(t )= =

75 Distribution over trees Let us denote by G(x) the set of derivations for the sentence x The probability distribution defines the scoring P (T ) over the trees T 2 G(x) Finding the best parse for the sentence according to PCFG: arg max T 2G(x) P (T ) 75

76 CKY with PCFGs Chart is represented by a double array chart[min][max][label] chart[min][max][c] It stores probabilities for the most probable subtree with a given signature chart[0][n][s] parse tree will store the probability of the most probable full 76

77 Intuition C! C 1 C 2 For every C choose C 1, C 2 and mid such that P (T 1 ) P (T 2 ) P (C! C 1 C 2 ) is maximal, where T 1 and T 2 are left and right subtrees. 77

78 Implementation: preterminal rules 78

79 Implementation: binary rules 79

80 Unary rules Similarly to CFGs: after producing scores for signatures (c, i, j), try to improve the scores by applying unary rules (and rule chains) If improved, update the scores 80

81 Unary (reflexive transitive) closure A! B B! C ) A! B B! C A! C A! A B! B C! C Note that this is not a PCFG anymore as the rules do not sum to 1 for each parent

82 Unary (reflexive transitive) closure The fact that the rule is composite needs to be stored to recover the true tree A! B B! C ) A! B B! C A! C A! A B! B C! C Note that this is not a PCFG anymore as the rules do not sum to 1 for each parent

83 Unary (reflexive transitive) closure The fact that the rule is composite needs to be stored to recover the true tree A! B B! C ) A! B B! C A! C A! A B! B C! C Note that this is not a PCFG anymore as the rules do not sum to 1 for each parent A! B B! C A! C e 5 ) A! B B! C A! C A! A B! B C! C

84 Unary (reflexive transitive) closure The fact that the rule is composite needs to be stored to recover the true tree A! B B! C ) A! B B! C A! C A! A B! B C! C Note that this is not a PCFG anymore as the rules do not sum to 1 for each parent A! B B! C A! C e 5 ) A! B B! C A! C A! A B! B C! C What about loops, like: A! B! A! C? 84

85 Recovery of the tree For each signature we store backpointers to the elements from which it was built (e.g., rule and, for binary rules, midpoint) start recovering from [0, n, S] Be careful with unary rules Basically you can assume that you always used an unary rule from the closure (but it could be the trivial one ) C! C 85

86 Speeding up the algorithm (approximate search) Any ideas? 86

87 Speeding up the algorithm (approximate search) Basic pruning (roughly): For every span (i,j) store only labels which have the probability at most N times smaller than the probability of the most probable label for this span Check not all rules but only rules yielding subtree labels having non-zero probability Coarse-to-fine pruning Parse with a smaller (simpler) grammar, and precompute (posterior) probabilities for each spans, and use only the ones with non-negligible probability from the previous grammar 87

88 Today Parsing algorithms for CFGs Recap, Chomsky Normal Form (CNF) A dynamic programming algorithm for parsing (CKY) Extension of CKY to support unary inner rules Parsing for PCFGs Extension of CKY for parsing with PCFGs Language Modeling Parser evaluation (if we have time) 88

89 Language modeling The goal of language modeling is estimation how likely is the next word given previous words (in a sentence) Many of you are familiar with ngram language models They are an important component in, for example, in machine translation and speech recognition Estimate the probability of the next word p(w k w k 1,...,w 1 ) Ngram language models make the nth-order Markovian assumption p(w k w k 1,...,w 1 ) p(w k w k 1,...,w k n ) Maximum likelihood estimates are just frequencies but in practice various forms of smoothing are used The probability of a word sequence is p(w 1,...,w m )= my p(w k w k 1,...,w k n ) k=1 89

90 Language modeling Ngram models are not very good at capturing long dependencies Availability of Web-scale data means that they are not as bad as long (up to 5 words) ngrams estimated from over a trillion of tokens is available for English. Still it can easily lead to ungrammatical sentence John has not even reached his/her destination Ankommen is a verb with a separable prefix Er kam am Freitagabend nach einem harten Arbeitstag und dem üblichen Ärger, der ihn schon seit Jahren immer wieder an seinem Arbeitsplatz plagt, mit fraglicher Freude auf ein Mahl, das seine Frau ihm, wie er hoffte, bereits aufgetischt hatte, endlich zu Hause an. [Wikipedia] Syntactic information should help us in modeling these long dependencies 90

91 Language modeling Note, equivalently, to modeling the probability of the next word, we can model the probability of a sentence prefix, as p(w k w k 1,...,w 1 )= p(w 1,...,w k 1,w k ) p(w 1,...,w k 1 ) Instead of looking into modeling probability of any sentence prefix, we will consider probabilities of whole sentences It is simpler and also important Dynamic programming algorithms for sentence prefixes are considered in Jelinek and Lafferty [91] and Stolcke [94]. 91

92 Syntactic language models To get the probability P (x) of a word sequence x we need to sum probabilities P (T ) of all its syntactic derivations T 2 G(x) P (x) = X P (T ) T 2G(x) S S VP VP P (lead can poison) =P ( N M V )+P ( N V ) lead can poison lead N can poison 92

93 Syntactic language models To get the probability P (x) of a word sequence x we need to sum probabilities P (T ) of all its syntactic derivations T 2 G(x) P (x) = X P (T ) T 2G(x) 93

94 Syntactic language models To get the probability P (x) of a word sequence x we need to sum probabilities P (T ) of all its syntactic derivations T 2 G(x) P (x) = X P (T ) T 2G(x) How can we do this efficiently (for PCFGs)? (Recall we know how to parse efficiently: arg max T 2G(x) P (T ) ) 94

95 Modifying CKY In CKY we maintained the probability of the most probable subtree for each signature Here, we will maintain the sum of probabilities of all the subtrees for each signature The CKY algorithm is an instance of the Viterbi algorithm, also known as Max-Product We will look into the algorithm called Sum-Product 95

96 CKY (Max-Product) 96

97 Sum-Product The probability of the sentence can be read from chart[0][n][s] 97

98 Sum-Product The probabilities stored in the chart are called "inside probabilities" and play an important role in induction of trees from unlabeled data The probability of the sentence can be read from chart[0][n][s] 98

99 Language modeling Though these syntactic models encode non-local dependences, their weakness as language models is that they are not rich enough to capture lexical dependencies In practice, syntactic language models use syntactic information to extract syntactically relevant previous words (instead of just previous words) See e.g., Chelba (1997) 99

100 Where are we?... Today we learnt how to parse with (P)CFGs Efficient bottom-up dynamic algorithm Earley algorithm is an effective top-down algorithm (but harder to understand + does not trivially generalize to PCFGs) Touched on language modeling Assignment 1.2: Ready to start working on it Add support of unary rules to a CKY implementation 100

101 Today Parsing algorithms for CFGs Recap, Chomsky Normal Form (CNF) A dynamic programming algorithm for parsing (CKY) Extension of CKY to support unary inner rules Parsing for PCFGs Extension of CKY for parsing with PCFGs Language Modeling Parser evaluation (if we have time) 101

102 Parser evaluation Though has many drawbacks it is easier and allows to track stateof-the-art across many years Intrinsic evaluation: Automatic: evaluate against annotation provided by human experts (gold standard) according to some predefined measure Manual: according to human judgment Extrinsic evaluation: score syntactic representation by comparing how well a system using this representation performs on some task E.g., use syntactic representation as input for a semantic analyzer and compare results of the analyzer using syntax predicted by different parsers. 102

103 Standard evaluation setting in parsing Automatic intrinsic evaluation is used: parsers are evaluated against gold standard by provided by linguists There is a standard split into the parts: training set: used for estimation of model parameters development set: used for tuning the model (initial experiments) test set: final experiments to compare against previous work 103

104 Automatic evaluation of constituent parsers Exact match: percentage of trees predicted correctly Bracket score: scores how well individual phrases (and their boundaries) are identified Crossing brackets: percentage of phrases boundaries crossing The most standard measure; we will focus on it Dependency metrics: scores dependency structure corresponding to the constituent tree (percentage of correctly identified heads) 104

105 Brackets scores The most standard score is bracket score It regards a tree as a collection of brackets: The set of brackets predicted by a parser is compared against the set of brackets in the tree annotated by a linguist Precision, recall and F1 are used as scores Subtree signatures for CKY [min, max, C] 105

106 Bracketing notation S VP PN N V My dog ate D N a sausage The same tree as a bracketed sequence (S ( (PN My) (N Dog) ) (VP (V ate) ( (D a ) (N sausage) ) ) ) 106

107 Brackets scores Pr = Re = number of brackets the parser and annotation agree on number of brackets predicted by the parser number of brackets the parser and annotation agree on number of brackets in annotation F 1= 2 Pr Re Pr+ Re Harmonic mean of precision and recall 107

108 Preview: F1 bracket score Treebank PCFG Unlexicalized Lexicalized PCFG PCFG (Klein and (Collins, 1999) Manning, 2003) Automatically Induced PCFG (Petrov et al., 2006) The best results reported (as of 2012) 108

109 Preview: F1 bracket score Treebank PCFG Unlexicalized Lexicalized PCFG PCFG (Klein and (Collins, 1999) Manning, 2003) Automatically Induced PCFG (Petrov et al., 2006) The best results reported (as of 2012) We will introduce these models next time 109