Learning Latent Linguistic Structure to Optimize End Tasks. David A. Smith with Jason Naradowsky and Xiaoye Tiger Wu

Transcription

1 Learning Latent Linguistic Structure to Optimize End Tasks David A. Smith with Jason Naradowsky and Xiaoye Tiger Wu 12 October 2012

2 Learning Latent Linguistic Structure to Optimize End Tasks David A. Smith with Jason Naradowsky and Xiaoye Tiger Wu 12 October 2012

3 Why Use NLP? oh, we need consecutive we need 6 and 7 i have why do n t you get 7 2

4 Why Use NLP? K S S C oh, we need consecutive we need 6 and 7 i have why do n t you get 7 2

5 Why Use NLP? K 2 oh, we need consecutive S str 1 we need 6 and 7 S hnd 2 i have C pup 2 why do n t you get 7 2

6 Why Use NLP? K 2 oh, we need consecutive S str 1 we need 6 and 7 S hnd 2 i have C pup 2 why do n t you get 7 3

7 Why Use NLP? K 2 oh, we need consecutive S str 1 we need 6 and 7 S hnd 2 i have C pup 2 why do n t you get 7 4

8 Task-directed Learning What does the cow say? The cow says mooo

9 Language Learning learning feedback KB say(cow, moo ) = 0.87 REL:say the cow says moo the cow says moo utterance

10 How to Use NLP? K 2 oh, we need consecutive S str 1 we need 6 and 7 S hnd 2 i have C pup 2 why do n t you get 7 7

11 How to Use NLP? K 2 oh, we need consecutive S str 1 we need 6 and 7 S hnd 2 i have C pup 2 why do n t you get 7 Can we parse these as coordinations? 7

12 How to Use NLP? Do we need training data for tagging, parsing, entity detection, coref, etc.? K 2 oh, we need consecutive S str 1 we need 6 and 7 S hnd 2 i have C pup 2 why do n t you get 7 Can we parse these as coordinations? 7

13 How to Use NLP? Isn t predicting this, or other game states, what we care about? Do we need training data for tagging, parsing, entity detection, coref, etc.? K 2 oh, we need consecutive S str 1 we need 6 and 7 S hnd 2 i have C pup 2 why do n t you get 7 Can we parse these as coordinations? 7

14 The Story So Far Joint inference prevents propagating errors But... Out-of-domain data plays havoc with NLP Joint training requires everything to be annotated

15 Learning Latent Representations Require less training data for......each stage in the pipeline...adapting to a new domain Efficient inference and learning for coupling structured observed and latent variables Features inspired by pipelined models 9

16 This Talk Motivation Latent representations from auxiliary tasks Latent representations from structured end tasks

17 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Start state: ([ ], [1,...,n], {}) Final state: (S, [], A) Shift: (S, i B, A) ) (S i, B, A) Reduce: (S i, B, A) ) (S, B, A) Right-Arc: (S i, j B, A) ) (S i j, B, A [ {i! j}) Left-Arc: (S i, j B, A) ) (S, j B, A [ {i j})

18 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Stack Buffer Arcs [] S [who, did, you, see] B {}

19 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Stack Buffer Arcs [who] S [did, you, see] B {} Shift

20 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Stack Buffer Arcs [] S [did, you, see] B { who OBJ did } Left-arc OBJ

21 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Stack Buffer Arcs [did] S [you, see] B { who OBJ did } Shift

22 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Right-arc SBJ Stack Buffer Arcs [did, you] S [see] B { who OBJ did, did SBJ! you }

23 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Reduce Stack Buffer Arcs [did] S [see] B { who OBJ did, did SBJ! you }

24 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Right-arc VG Stack Buffer Arcs [did, see] S [] B { who OBJ did, did SBJ! you, did VG! see }

25 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Right-arc SBJ Stack Buffer Arcs [did, you] S [see] B { who OBJ did, did SBJ! you }

26 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Right-arc SBJ Stack Buffer Arcs [did, you] S [see] B { who OBJ did, did SBJ! you } Choose action w/best classifier score 100k - 1M features

27 Transition-Based Parsing Arc-eager shift-reduce parsing (Nivre, 2003) Right-arc SBJ Stack Buffer Arcs Very fast linear-time performance WSJ 23 (2k sentences) in 3 s [did, you] S [see] B { who OBJ did, did SBJ! you } Choose action w/best classifier score 100k - 1M features

28 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext % Accuracy Unlabeled

29 Classifying Actions Stack Buffer Arcs [did] S [you, see] B { who OBJ did } Shift

30 Classifying Actions Stack Buffer Arcs [did] S [you, see] B { who OBJ did } Shift Output: Action [+ Label]

31 Classifying Actions POS, Form in buffer; lookahead 3 Stack Buffer Arcs [did] S [you, see] B { who OBJ did } Shift Output: Action [+ Label]

32 Classifying Actions POS, Form of top 2 stack items POS, Form in buffer; lookahead 3 Stack Buffer Arcs [did] S [you, see] B { who OBJ did } Shift Output: Action [+ Label]

33 Classifying Actions POS, Form of top 2 stack items POS, Form in buffer; lookahead 3 Stack Buffer Arcs [did] S [you, see] B { who OBJ did } Shift Dep. label of words in stack, buffer Output: Action [+ Label]

34 Classifying Actions POS, Form of top 2 stack items POS, Form in buffer; lookahead 3 Stack Buffer Arcs [did] S [you, see] B { who OBJ did } Shift Dep. label of words in stack, buffer Can these be Output: Action [+ Label] made less sparse?

35 Distributional Embeddings Keyword in 7-gram context among them the winner in Paris in becoming a Snavely winner, I was but onto the 77 winner mark with Bishops The recent Newbury winner has been installed to be the winner for the second We've got the winner on the phone

36 Distributional Embedding became the first female winner of the prestigious Nobel Prize 23

37 Distributional Embedding became the first female winner of the prestigious Nobel Prize 23

38 Distributional Embedding Keyword in 7-dependency context became the first female winner of the prestigious Nobel Prize 24

39 Distributional Embedding Discriminative language model training became the first female winner of the prestigious Nobel Prize capital 25

40 Distributional Embedding Discriminative language model training became the first female winner of the prestigious Nobel Prize capital Learn to predict observed word 25

41 Distributional Embedding Discriminative language model training became the first female winner of the prestigious Nobel Prize 26

42 Distributional Embedding Discriminative language model training became the first female winner of the prestigious Nobel Prize Lookup in codebook 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 26

43 Distributional Embedding Discriminative language model training became the first female winner of the prestigious Nobel Prize Lookup in codebook 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 3-layer perceptron Correct word? 26

44 Distributional Embedding Discriminative language model training became the first female winner of the prestigious Nobel Prize Lookup in codebook Backprop updates codes 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 3-layer perceptron Correct word? 26

45 Distributional Embedding Discriminative language model training Words=points in 50d space became the first female winner of the prestigious Nobel Prize Lookup in codebook Backprop updates codes 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 50 dim. 3-layer perceptron Correct word? 26

46 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext 100 % Accuracy Unlabeled

47 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext % Accuracy Unlabeled

48 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext % Accuracy Unlabeled

49 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext % Accuracy Unlabeled

50 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext % Accuracy Unlabeled

51 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext % Accuracy Unlabeled

52 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext % Accuracy Unlabeled

53 Better Features for Parsing Baseline 7-gram +7-gram +7-gram +ext +7-dep clusters 7-dep +7-dep +7-dep +ext % Accuracy Unlabeled

54 Better Features for Parsing ROOT % Accuracy Baseline dep

55 Unsupervised Domain Adaptation Baseline W embed B embed W+B embed % Accuracy WSJ Brown

56 This Talk Motivation Latent representations from auxiliary tasks Latent representations from structured end tasks

57 Semantic Role Labeling Pierre Vinken, 61 years old, will join the board as a nonexecutive director Vinken s joining the board as director is imminent 31

58 Semantic Role Labeling Pierre Vinken, 61 years old, will join the board as a nonexecutive director Vinken s joining the board as director is imminent PRED ARG0 ARG1 ARG-PRD join Vinken board director 31

59 Semantic Role Labeling Pierre Vinken, 61 years old, will join the board as a nonexecutive director Vinken s joining the board as director is imminent PRED ARG0 ARG1 ARG-PRD join Vinken board director 31

60 Coupling Structured Variables Outputs are complex combinatorial objects E.g., sequences, trees, graphs, strings Intermediate representations are complex combinatorial objects E.g., syntax helps predict semantics Developed efficient inference and learning 32

61 Composable Syntax Semantic Role Labeling Relation Extraction employs(organization, PERSON) Dependency and constituency models Named-entity recognition Couple constituency model with semi-crf No grammar composition

62 Global Constraints Dependency trees Cf. Projective & nonprojective (MST) parsing (Nested) bracketings Cf. CKY & semi-crfs Matchings and alignments Weighted bipartite matching & network flow; ITG Intersecting specialized grammars (CFG, TAG, etc.) Other finite-state, context-free, etc., constraints... Compare TurboParsers and Dual Decomposition 34

63 Semantic Role Labeling a.) Syntactic Combinatorial Constraint DEP-Tree b.) Syntactic Layer Link 5, 1 Link 5, 2 Link 5, 3... Link 5, n D-Connect 5, 1 D-Connect 5, 2 D-Connect 5, 3 D-Connect 5, n c.) Argument Prediction Arg 5, 1 Arg 5, 2 Arg 5, 3 d.) Sense Prediction sense. 01 role A0 role A1 At Most 1 role A2... role A-TM At Most 1 Pred 5 sense sense. S 35

64 Semantic Role Labeling 100 Baseline Oracle syntax Hidden syntax Median SRL Max SRL Labeled F Chinese Czech English German 36

65 Hidden Syntax $ Alles bleibt voller Widersprüche. Everything remains full of contradictions. 37

66 Hidden Syntax Gold $ Alles bleibt voller Widersprüche. Everything remains full of contradictions. 37

67 Hidden Syntax Gold $ Alles bleibt voller Widersprüche. Hidden Everything remains full of contradictions. 37

68 Hidden Syntax Gold $ Alles bleibt voller Widersprüche. Hidden Everything remains full of contradictions. Heading sentence with punctuation doesn t affect SRL. 37

69 Hidden Syntax Gold $ Es bleibt nur die Hoffnung auf einen erneuten Urknall. Hidden There remains only the hope of a renewed Big Bang. 38

70 Hidden Syntax Gold $ Es bleibt nur die Hoffnung auf einen erneuten Urknall. Hidden It s useful to have NPs consistent inside and outside of PPs. There remains only the hope of a renewed Big Bang. 38

71 Relation Extraction 39

72 Relation Extraction Labeled F Baseline Parser Deps. Hidden Deps. Parser Constit. Hidden Constit. 40

73 In Summary Representations trained on more appropriate auxiliary tasks support better learning Models with latent syntax models near performance of those with oracle syntax Assertion (see papers): Latent structured variables without asymptotic slowdown

74 Thank you

75 Hidden Distribution 43