Natural Language Processing (CSE 517): Dependency Syntax and Parsing

Transcription

1 Natural Language Processing (CSE 517): Depenency Syntax an Parsing Noah A. Smith Swabha Swayamipta c 2018 University of Washington {nasmith,swabha}@cs.washington.eu May 11, / 93

2 Recap: Phrase Structure S NP NP VP DT The NN luxury NN auto NN maker JJ last NN year VBD sol CD NP NN IN PP NP 1,214 cars in DT NNP the U.S. 2 / 93

3 Parent Annotation (Johnson, 1998) S ROOT NP S NP S VP S DT NP The NN NP luxury NN NP auto NN NP maker JJ NP last NN NP year VBD VP NP VP PP VP sol CD NP NN NP IN PP NP PP 1,214 cars in DT NP NNP NP the U.S. Increases the vertical Markov orer: p(chilren parent, granparent) 3 / 93

4 Heaeness S NP NP VP DT The NN luxury NN auto NN maker JJ last NN year VBD NP PP sol CD NN IN NP 1,214 cars in DT the NNP U.S. Suggests horizontal markovization: p(chilren parent) = p(hea parent) p(ith sibling hea, parent) i 4 / 93

5 Lexicalization S sol NP maker NP year VP sol JJ last NN year DT The The NN luxury luxury NN auto auto NN maker maker last year VBD sol NP cars PP in sol CD 1,214 NN cars IN in NP U.S. 1,214 cars in DT the NNP U.S. the U.S. Each noe shares a lexical hea with its hea chil. 5 / 93

6 Depenencies Informally, you can think of epenency structures as a transformation of phrase-structures that maintains the wor-to-wor relationships inuce by lexicalization, as labels to them, an eliminates the phrase categories. There are also linguistic theories built on epenencies (Tesnière, 1959; Mel čuk, 1987), as well as treebanks corresponing to those. Free(r)-wor orer languages (e.g., Czech) 6 / 93

7 Depenency Tree: Definition Let x = x 1,..., x n be a sentence. A a special root symbol as x 0. A epenency tree consists of a set of tuples p, c, l, where p {0,..., n} is the inex of a parent c {1,..., n} is the inex of a chil l L is a label Different annotation schemes efine ifferent label sets L, an ifferent constraints on the set of tuples. Most commonly: The tuple is represente as a irecte ege from x p to x c with label l. The irecte eges form an arborescence (irecte tree) with x 0 as the root (sometimes enote root). 7 / 93

8 Example S NP VP Pronoun we Verb wash NP Determiner Noun our cats Phrase-structure tree. 8 / 93

9 Example S NP VP Pronoun we Verb wash NP Determiner Noun our cats Phrase-structure tree with heas. 9 / 93

10 Example S wash NP we VP wash Pronoun we we Verb wash wash Determiner our NP cats Noun cats our cats Phrase-structure tree with heas, lexicalize. 10 / 93

11 Example we wash our cats Bare bones epenency tree. 11 / 93

12 Example we wash our cats who stink 12 / 93

13 Example we vigorously wash our cats who stink 13 / 93

14 Content Heas vs. Function Heas Creit: Nathan Schneier little kis were always watching birs with fish little kis were always watching birs with fish 14 / 93

15 Labels root pobj sbj obj prep kis saw birs with fish Key epenency relations capture in the labels inclue: subject, irect object, preposition object, ajectival moifier, averbial moifier. In this lecture, I will mostly not iscuss labels, to keep the algorithms simpler. 15 / 93

16 Coorination Structures we vigorously wash our cats an ogs who stink The bugbear of epenency syntax. 16 / 93

17 Example we vigorously wash our cats an ogs who stink Make the first conjunct the hea? 17 / 93

18 Example we vigorously wash our cats an ogs who stink Make the coorinating conjunction the hea? 18 / 93

19 Example we vigorously wash our cats an ogs who stink Make the secon conjunct the hea? 19 / 93

20 Nonprojective Example ROOT ATT PU TMP ATT SBJ VC PC ATT A hearing is scheule on the issue toay. 20 / 93

21 Depenency Schemes Direct annotation. Transform the treebank: efine hea rules that can select the hea chil of any noe in a phrase-structure tree an label the epenencies. More powerful, less local rule sets, possibly collapsing some wors into arc labels. Stanfor epenencies are a popular example (e Marneffe et al., 2006). Only results in projective trees. Rule base epenencies, followe by manual correction. 21 / 93

22 Approaches to Depenency Parsing 1. Transition-base parsing with a stack. 2. Chu-Liu-Emons algorithm for arborescences (irecte graphs). 3. Dynamic programming with the Eisner algorithm. 22 / 93

23 Transition-Base Parsing Depenency tree represente as a linear sequence of transitions. 23 / 93

24 Transition-Base Parsing Depenency tree represente as a linear sequence of transitions. Transitions: Simple operations to be execute on a parser configuration 24 / 93

25 Transition-Base Parsing Depenency tree represente as a linear sequence of transitions. Transitions: Simple operations to be execute on a parser configuration Parser Configuration: stack S an a buffer B. 25 / 93

26 Transition-Base Parsing Depenency tree represente as a linear sequence of transitions. Transitions: Simple operations to be execute on a parser configuration Parser Configuration: stack S an a buffer B. During parsing, apply a classifier to ecie which transition to take next, greeily. No backtracking. 26 / 93

27 Transition-Base Parsing: Transitions Parser Configuration: Initial Configuration: buffer contains x an the stack contains the root. Final Configuration: Empty buffer an the stack contains the entire tree. 27 / 93

28 Transition-Base Parsing: Transitions Parser Configuration: Initial Configuration: buffer contains x an the stack contains the root. Final Configuration: Empty buffer an the stack contains the entire tree. Transitions : arc-stanar transition set (Nivre, 2004): shift the wor at the front of the buffer B onto the stack S. right-arc: u = pop(s); v = pop(s); push(s, v u). left-arc: u = pop(s); v = pop(s); push(s, v u). 28 / 93

29 Transition-Base Parsing: Transitions Parser Configuration: Initial Configuration: buffer contains x an the stack contains the root. Final Configuration: Empty buffer an the stack contains the entire tree. Transitions : arc-stanar transition set (Nivre, 2004): shift the wor at the front of the buffer B onto the stack S. right-arc: u = pop(s); v = pop(s); push(s, v u). left-arc: u = pop(s); v = pop(s); push(s, v u). (For labele parsing, a labels to the right-arc an left-arc transitions.) 29 / 93

30 Transition-Base Parsing: Example Buffer B: Stack S: root we vigorously wash our cats who stink Actions: 30 / 93

31 Transition-Base Parsing: Example Buffer B: Stack S: we root vigorously wash our cats who stink Actions: shift 31 / 93

32 Transition-Base Parsing: Example Buffer B: Stack S: vigorously we root wash our cats who stink Actions: shift shift 32 / 93

33 Transition-Base Parsing: Example Stack S: wash vigorously we root Buffer B: our cats who stink Actions: shift shift shift 33 / 93

34 Transition-Base Parsing: Example Stack S: vigorously we root wash Buffer B: our cats who stink Actions: shift shift shift left-arc 34 / 93

35 Transition-Base Parsing: Example Stack S: we vigorously wash root Buffer B: our cats who stink Actions: shift shift shift left-arc left-arc 35 / 93

36 Transition-Base Parsing: Example Stack S: our we vigorously wash Buffer B: cats who stink root Actions: shift shift shift left-arc left-arc shift 36 / 93

37 Transition-Base Parsing: Example Stack S: cats our we vigorously wash Buffer B: who stink root Actions: shift shift shift left-arc left-arc shift shift 37 / 93

38 Transition-Base Parsing: Example Stack S: our cats Buffer B: who stink we vigorously wash root Actions: shift shift shift left-arc left-arc shift shift left-arc 38 / 93

39 Transition-Base Parsing: Example Stack S: who our cats Buffer B: stink we vigorously wash root Actions: shift shift shift shift left-arc left-arc shift shift left-arc 39 / 93

40 Transition-Base Parsing: Example Stack S: stink who our cats Buffer B: we vigorously wash root Actions: shift shift shift left-arc left-arc shift shift left-arc shift shift 40 / 93

41 Transition-Base Parsing: Example Stack S: who our stink cats Buffer B: we vigorously wash root Actions: shift shift shift left-arc left-arc shift shift left-arc shift shift right-arc 41 / 93

42 Transition-Base Parsing: Example Stack S: our cats who stink Buffer B: we vigorously wash root Actions: shift shift shift left-arc left-arc shift shift left-arc shift shift right-arc right-arc 42 / 93

43 Transition-Base Parsing: Example Stack S: Buffer B: we vigorously wash our cats who stink root Actions: shift shift shift left-arc left-arc shift shift left-arc shift shift right-arc right-arc right-arc 43 / 93

44 Transition-Base Parsing: Example Stack S: root Buffer B: we vigorously wash our cats who stink Actions: shift shift shift left-arc left-arc shift shift left-arc shift shift right-arc right-arc right-arc right-arc 44 / 93

45 The Core of Transition-Base Parsing: Classification At each iteration, choose among {shift, right-arc, left-arc}. (Actually, among all L-labele variants of right- an left-arc.) 45 / 93

46 The Core of Transition-Base Parsing: Classification At each iteration, choose among {shift, right-arc, left-arc}. (Actually, among all L-labele variants of right- an left-arc.) Features can look S, B, an the history of past actions usually there is no ecomposition into local structures. 46 / 93

47 The Core of Transition-Base Parsing: Classification At each iteration, choose among {shift, right-arc, left-arc}. (Actually, among all L-labele variants of right- an left-arc.) Features can look S, B, an the history of past actions usually there is no ecomposition into local structures. Training ata: oracle transition sequence that gives the right tree converts into 2 n pairs: state, correct transition. 47 / 93

48 The Core of Transition-Base Parsing: Classification At each iteration, choose among {shift, right-arc, left-arc}. (Actually, among all L-labele variants of right- an left-arc.) Features can look S, B, an the history of past actions usually there is no ecomposition into local structures. Training ata: oracle transition sequence that gives the right tree converts into 2 n pairs: state, correct transition. Each wor gets shifte once an participates as a chil in one arc. Linear time. 48 / 93

49 Transition-Base Parsing: Remarks Can also be applie to phrase-structure parsing (e.g., Sagae an Lavie, 2006). Keywor: shift-reuce parsing. The algorithm for making ecisions oesn t nee to be greey; can maintain multiple hypotheses. E.g., beam search, which we ll iscuss in the context of machine translation later. Potential flaw: the classifier is typically traine uner the assumption that previous classification ecisions were all correct. As yet, no principle solution to this problem, but see ynamic oracles (Golberg an Nivre, 2012). 49 / 93

50 Approaches to Depenency Parsing 1. Transition-base parsing with a stack. 2. Chu-Liu-Emons algorithm for arborescences (irecte graphs). 3. Dynamic programming with the Eisner algorithm. 50 / 93

51 Acknowlegment Slies are mostly aapte from those by Swabha Swayamipta an Sam Thomson. 51 / 93

52 Features in Depenency Parsing For transition-base parsing, we coul use any past ecisions to score the current ecision: a s global (y) = s(a) = s(a i a 0:i 1 ) We gave up on any guarantee of fining the best possible y in favor of arbitrary features. For a neural network-base moel that fully exploits this, see Dyer et al. (2015). i=1 52 / 93

53 Graph-Base Depenency Parsing Selects structures which are globally optimal. 53 / 93

54 Graph-Base Depenency Parsing Selects structures which are globally optimal. Start with a fully connecte graph. Set of O(n 2 ) eges, E. 54 / 93

55 Graph-Base Depenency Parsing Selects structures which are globally optimal. Start with a fully connecte graph. Set of O(n 2 ) eges, E. No incoming eges to x 0, ensuring that it will be the root. 55 / 93

56 First-Orer Graph-Base (FOG) Depenency Parsing (McDonal et al., 2005) Every possible irecte ege e between a parent p an a chil c gets a local score, s(e). y = argmax y E s global (y) = argmax y E subject to the constraint that y is an arborescence s(e) e y Classical algorithm to efficiently solve this problem: Chu an Liu (1965), Emons (1967) 56 / 93

57 Chu-Liu-Emons Intuitions Every non-root noe nees exactly one incoming ege. 57 / 93

58 Chu-Liu-Emons Intuitions Every non-root noe nees exactly one incoming ege. In fact, every connecte component that oesn t contain x 0 nees exactly one incoming ege. 58 / 93

59 Chu-Liu-Emons Intuitions Every non-root noe nees exactly one incoming ege. In fact, every connecte component that oesn t contain x 0 nees exactly one incoming ege. Maximum spanning tree. 59 / 93

60 Chu-Liu-Emons Intuitions Every non-root noe nees exactly one incoming ege. In fact, every connecte component that oesn t contain x 0 nees exactly one incoming ege. Maximum spanning tree. High-level view of the algorithm: 1. For every c, pick an incoming ege (i.e., pick a parent) greeily. 2. If this forms an arborescence, you are one! 3. Otherwise, it s because there s a cycle, C. Arborescences can t have cycles, so some ege in C nees to be kicke out. We also nee to fin an incoming ege for C. Choosing the incoming ege for C etermines which ege to kick out. 60 / 93

61 Chu-Liu-Emons: Recursive (Inefficient) Definition ef maxarborescence(v, E, root): # returns best arborescence as a map from each noe to its parent for c in V \ root: bestinege[c] argmax e E:e= p,c e.s # i.e., s(e) if bestinege contains a cycle C: # buil a new graph where C is contracte into a single noe v C new Noe() V V {v C } \ C E {ajust(e, v C ) for e E \ C} A maxarborescence(v, E, root) return {e.original for e A} C \ {A[v C ].kicksout} # each noe got a parent without creating any cycles return bestinege 61 / 93

62 Unerstaning Chu-Liu-Emons There are two stages: Contraction (the stuff before the recursive call) Expansion (the stuff after the recursive call) 62 / 93

63 Chu-Liu-Emons: Contraction For each non-root noe v, set bestinege[v] to be its highest scoring incoming ege. If a cycle C is forme: contract the noes in C into a new noe v C ajust subroutine on next slie performs the following: Eges incoming to any noe in C now get estination vc For each noe v in C, an for each ege e incoming to v from outsie of C: Set e.kicksout to bestinege[v], an Set e.s to be e.s e.kicksout.s Eges outgoing from any noe in C now get source vc Repeat until every non-root noe has an incoming ege an no cycles are forme 63 / 93

64 Chu-Liu-Emons: Ege Ajustment Subroutine ef ajust(e, v C ): e copy(e) e.original e if e.est C: e.est v C e.kicksout bestinege[e.est] e.s e.s e.kicksout.s elif e.src C: e.src v C return e 64 / 93

65 Contraction Example ROOT V1 V2 V3 bestinege V1 a : 5 b : 1 c : 1 : 11 V2 f : 5 g : 10 i : 8 e : 4 h : 9 V3 a b c e f g h i kicksout 65 / 93

66 Contraction Example ROOT V1 V2 V3 bestinege g V1 a : 5 b : 1 c : 1 : 11 V2 f : 5 g : 10 i : 8 e : 4 h : 9 V3 a b c e f g h i kicksout 66 / 93

67 Contraction Example ROOT V1 V2 V3 bestinege g V1 a : 5 b : 1 c : 1 : 11 V2 f : 5 g : 10 i : 8 e : 4 h : 9 V3 a b c e f g h i kicksout 67 / 93

68 Contraction Example ROOT V1 V2 V3 bestinege g V1 a : 5 10 b : 1 11 c : 1 V4 : 11 V2 f : 5 g : 10 i : 8 11 e : 4 h : 9 10 V3 a b c e f g h i kicksout g g 68 / 93

69 Contraction Example a : 5 ROOT V1 V2 V3 V4 bestinege g V4 b : 10 c : 1 f : 5 i : 3 e : 4 h : 1 V3 a b c e f g h i kicksout g g 69 / 93

70 Contraction Example a : 5 ROOT V1 V2 V3 V4 bestinege g f V4 b : 10 c : 1 f : 5 i : 3 e : 4 h : 1 V3 a b c e f g h i kicksout g g 70 / 93

71 Contraction Example a : 5 ROOT V1 V2 V3 V4 bestinege g f h V4 b : 10 c : 1 f : 5 i : 3 e : 4 h : 1 V3 a b c e f g h i kicksout g g 71 / 93

72 Contraction Example ROOT a : 5 1 b : 10 1 c : 1 5 V5 V4 f : 5 i : 3 e : 4 h : 1 V3 V1 V2 V3 V4 V5 a b c e f g h i bestinege g f h kicksout g, h, h f g 72 / 93

73 Contraction Example ROOT b : 9 a : 4 c : 4 V5 V1 V2 V3 V4 V5 a b c e f g h i bestinege g f h kicksout g, h, h f f g 73 / 93

74 Contraction Example ROOT b : 9 a : 4 c : 4 V5 V1 V2 V3 V4 V5 a b c e f g h i bestinege g f h a kicksout g, h, h f f g 74 / 93

75 Chu-Liu-Emons: Expansion After the contraction stage, every contracte noe will have exactly one bestinege. This ege will kick out one ege insie the contracte noe, breaking the cycle. Go through each bestinege e in the reverse orer that we ae them Lock own e, an remove every ege in kicksout(e) from bestinege. 75 / 93

76 Expansion Example ROOT b : 9 a : 4 c : 4 V5 V1 V2 V3 V4 V5 a b c e f g h i bestinege g f h a kicksout g, h, h f f g 76 / 93

77 Expansion Example ROOT b : 9 a : 4 c : 4 V5 V1 V2 V3 V4 V5 a b c e f g h i bestinege a g f a h a kicksout g, h, h f f g 77 / 93

78 Expansion Example ROOT a : 5 b : 10 c : 1 V4 f : 5 i : 3 e : 4 h : 1 V3 V1 V2 V3 V4 V5 a b c e f g h i bestinege a g f a h a kicksout g, h, h f f g 78 / 93

79 Expansion Example ROOT a : 5 b : 10 c : 1 V4 f : 5 i : 3 e : 4 h : 1 V3 V1 V2 V3 V4 V5 a b c e f g h i bestinege a g f a h a kicksout g, h, h f f g 79 / 93

80 Expansion Example V1 ROOT a : 5 b : 1 c : 1 : 11 V2 f : 5 g : 10 i : 8 e : 4 h : 9 V3 V1 V2 V3 V4 V5 a b c e f g h i bestinege a g f a h a kicksout g, h, h f f g 80 / 93

81 Expansion Example V1 ROOT a : 5 b : 1 c : 1 : 11 V2 f : 5 g : 10 i : 8 e : 4 h : 9 V3 V1 V2 V3 V4 V5 a b c e f g h i bestinege a g f a h a kicksout g, h, h f f g 81 / 93

82 Observation The set of arborescences strictly inclues the set of projective epenency trees. Is this a goo thing or a ba thing? 82 / 93

83 Chu-Liu-Emons: Notes This is a greey algorithm with a clever form of elaye backtracking to recover from inconsistent ecisions (cycles). 83 / 93

84 Chu-Liu-Emons: Notes This is a greey algorithm with a clever form of elaye backtracking to recover from inconsistent ecisions (cycles). CLE is exact: it always recovers an optimal arborescence. 84 / 93

85 Chu-Liu-Emons: Notes This is a greey algorithm with a clever form of elaye backtracking to recover from inconsistent ecisions (cycles). CLE is exact: it always recovers an optimal arborescence. What about labele epenencies? As a matter of preprocessing, for each p, c, keep only the top-scoring labele ege. 85 / 93

86 Chu-Liu-Emons: Notes This is a greey algorithm with a clever form of elaye backtracking to recover from inconsistent ecisions (cycles). CLE is exact: it always recovers an optimal arborescence. What about labele epenencies? As a matter of preprocessing, for each p, c, keep only the top-scoring labele ege. Tarjan (1977) offere a more efficient, but unfortunately incorrect, implementation. Camerini et al. (1979) correcte it. The approach is not recursive; instea using a isjoint set ata structure to keep track of collapse noes. Even better: Gabow et al. (1986) use a Fibonacci heap to keep incoming eges sorte, an fins cycles in a more sensible way. Also constrains root to have only one outgoing ege. With these tricks, O(n 2 + n log n) runtime. 86 / 93

87 More Details on Statistical Depenency Parsing What about the scores? McDonal et al. (2005) use carefully-esigne features an (something close to) the structure perceptron; Kiperwasser an Golberg (2016) use biirectional recurrent neural networks. 87 / 93

88 More Details on Statistical Depenency Parsing What about the scores? McDonal et al. (2005) use carefully-esigne features an (something close to) the structure perceptron; Kiperwasser an Golberg (2016) use biirectional recurrent neural networks. What about higher-orer parsing? Requires approximate inference, e.g., ual ecomposition (Martins et al., 2013). 88 / 93

89 Important Traeoffs (an Not Just in NLP) 1. Two extremes: Specialize algorithm that efficiently solves your problem, uner your assumptions. E.g., Chu-Liu-Emons for FOG epenency parsing. General-purpose metho that solves many problems, allowing you to test the effect of ifferent assumptions. E.g., ynamic programming, transition-base methos, some forms of approximate inference. 89 / 93

90 Important Traeoffs (an Not Just in NLP) 1. Two extremes: Specialize algorithm that efficiently solves your problem, uner your assumptions. E.g., Chu-Liu-Emons for FOG epenency parsing. General-purpose metho that solves many problems, allowing you to test the effect of ifferent assumptions. E.g., ynamic programming, transition-base methos, some forms of approximate inference. 2. Two extremes: Fast (linear-time) but greey Moel-optimal but slow 90 / 93

91 Important Traeoffs (an Not Just in NLP) 1. Two extremes: Specialize algorithm that efficiently solves your problem, uner your assumptions. E.g., Chu-Liu-Emons for FOG epenency parsing. General-purpose metho that solves many problems, allowing you to test the effect of ifferent assumptions. E.g., ynamic programming, transition-base methos, some forms of approximate inference. 2. Two extremes: Fast (linear-time) but greey Moel-optimal but slow Dirty secret: the best way to get (English) epenency trees is to run phrase-structure parsing, then convert. 91 / 93

92 References I Paolo M. Camerini, Luigi Fratta, an Francesco Maffioli. A note on fining optimum branchings. Networks, 9 (4): , Y. J. Chu an T. H. Liu. On the shortest arborescence of a irecte graph. Science Sinica, 14: , Marie-Catherine e Marneffe, Bill MacCartney, an Christopher D. Manning. Generating type epenency parses from phrase structure parses. In Proc. of LREC, Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, an Noah A. Smith. Transition-base epenency parsing with stack long short-term memory. In Proc. of ACL, Jack Emons. Optimum branchings. Journal of Research of the National Bureau of Stanars, 71B: , Harol N. Gabow, Zvi Galil, Thomas Spencer, an Robert E. Tarjan. Efficient algorithms for fining minimum spanning trees in unirecte an irecte graphs. Combinatorica, 6(2): , Yoav Golberg an Joakim Nivre. A ynamic oracle for arc-eager epenency parsing. In Proc. of COLING, Mark Johnson. PCFG moels of linguistic tree representations. Computational Linguistics, 24(4):613 32, Eliyahu Kiperwasser an Yoav Golberg. Simple an accurate epenency parsing using biirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4: , Anré F. T. Martins, Miguel Almeia, an Noah A. Smith. Turning on the turbo: Fast thir-orer non-projective turbo parsers. In Proc. of ACL, / 93

93 References II Ryan McDonal, Fernano Pereira, Kiril Ribarov, an Jan Hajic. Non-projective epenency parsing using spanning tree algorithms. In Proceeings of HLT-EMNLP, URL Igor A. Mel čuk. Depenency Syntax: Theory an Practice. State University Press of New York, Joakim Nivre. Incrementality in eterministic epenency parsing. In Proc. of ACL, Kenji Sagae an Alon Lavie. A best-first probabilistic shift-reuce parser. In Proc. of COLING-ACL, Robert E. Tarjan. Fining optimum branchings. Networks, 7:25 35, L. Tesnière. Éléments e Syntaxe Structurale. Klincksieck, / 93