Density-Driven Cross-Lingual Transfer of Dependency Parsers

Transcription

1 Density-Driven Cross-Lingual Transfer of Dependency Parsers Mohammad Sadegh Rasooli Michael Collins Presented by Owen Rambow EMNLP 2015

2 Motivation Availability of treebanks Accurate parsers use annotated treebanks. There are no gold-standard treebanks for many languages. Annotated treebanks are very expensive to create. Common approach: using universal linguistic information Without parallel data; e.g [Zhang and Barzilay, 2015] With parallel data; e.g [Ma and Xia, 2014] The best results but still lags behind supervised parsing

3 Motivation Availability of treebanks Accurate parsers use annotated treebanks. There are no gold-standard treebanks for many languages. Annotated treebanks are very expensive to create. Common approach: using universal linguistic information Without parallel data; e.g [Zhang and Barzilay, 2015] With parallel data; e.g [Ma and Xia, 2014] The best results but still lags behind supervised parsing

4 Projecting Dependencies from Parallel Data Bitext Prepare bitext The political priorities must be set by this House and the MEPs. ROOT Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

5 Projecting Dependencies from Parallel Data Align Align bitext (e.g. via Giza++) The political priorities must be set by this House and the MEPs. ROOT Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

6 Projecting Dependencies from Parallel Data Parse Parse source sentence with a supervised parser. The political priorities must be set by this House and the MEPs. ROOT Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

7 Projecting Dependencies from Parallel Data Project Project dependencies. The political priorities must be set by this House and the MEPs. ROOT Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

8 Projecting Dependencies from Parallel Data Train Train on the projected dependencies. Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

9 Practical Problems Most translations are not word-to-word. Alignment errors! Supervised parsers are not perfect. Difference in syntactic behavior across languages.

10 Previous Results 95 dependency accuracy (avg. over 6 EU languages) [McDonald et al., 2011] [Zhang and Barzilay, 2015] [Ma and Xia, 2014] MPH11 ZB15 Previous work MX14 1st-ord Sh-R

11 Previous Results 95 [McDonald et al., 2011] dependency accuracy (avg. over 6 EU languages) [Zhang and Barzilay, 2015] [Ma and Xia, 2014] [McDonald et al., 2005] [Rasooli and Tetreault, 2015] % lower than a first-order supervised model MPH11 ZB15 MX14 1st-ord Sh-R Previous work Supervised models

12 Our Approach We define different sets of dense structures Full trees Dense partial trees

13 Dense Structures A projected full tree t P 100 is: A projective dependency tree All words have one parent Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

14 Dense Structures A partial tree t P 80 is: A projective dependency tree (a collection of projective trees) At least 80% of words have one parent Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

15 Dense Structures A partial tree t P k is: A projective dependency tree (a collection of projective trees) There is at least one span of length k where all words in that span have one parent Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT k=7 in the above tree

16 Our Contributions We demonstrate the utility of dense projected structures. We describe a training algorithm that builds on dense structures.

17 Our Contributions 95 [McDonald et al., 2011] [Zhang and Barzilay, 2015] dependency accuracy (avg. over 6 EU languages) [Ma and Xia, 2014] [McDonald et al., 2005] [Rasooli and Tetreault, 2015] MPH11 ZB15 MX14 Our Model 1st-ord Sh-R Previous work Supervised models

18 Our Contributions 95 dependency accuracy (avg. over 6 EU languages) [McDonald et al., 2011] [Zhang and Barzilay, 2015] 5.5% absolute improvement [Ma and Xia, 2014] Our model 87.5 [McDonald et al., 2005] [Rasooli and Tetreault, 2015] MPH11 ZB15 MX14 Our Model 1st-ord Sh-R Previous work Supervised models

19 Overview The learning algorithm Results Analysis

20 Projecting Dependencies Languages from Google universal treebank: English (only as source), German, Spanish, French, Italian, Portuguese, and Swedish. English to German transfer data for developing our models. We use Giza++ intersected alignments on EuroParl data We use the Yara parser [Rasooli and Tetreault, 2015], a shift-reduce beam parser.

21 Functions Used in Our Algorithm We use the following function definitions: Train(D) CDECODE(P, θ) TOP(D, θ)

22 Train(D) Input D A set of dependency trees (full trees) Output θ A parsing model

23 CDECODE(P, θ) Input P Input θ A set of partial dependency structures Parsing model Output D A set of full trees that are completely consistent with the dependencies in P. Filling in partial trees with dynamic oracles [Goldberg and Nivre, 2013]. Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

24 CDECODE(P, θ) Input P Input θ A set of partial dependency structures Parsing model Output D A set of full trees that are completely consistent with the dependencies in P. Filling in partial trees with dynamic oracles [Goldberg and Nivre, 2013]. Die politischen Prioritäten müssen von diesem Parlament und den Europaabgeordneten abgesteckt werden. ROOT

25 TOP(D, θ) Input D Input θ A set of full dependency trees Parsing model Output A Top m highest scoring trees in D We use m=200,000 in our experiments. Score: Perceptron-based parse score normalized by sentence length

26 Definitions A 0 = P 100 A 1 = P 7 P 80 A 2 = P 5 P 80 A 3 = P 1 P 80 Note A 1 A 2 A 3

27 Learning Algorithm Train on full trees θ 0 = Train(A 0 ) for i = do D i = CDECODE(A i, θ i 1 ) A i = TOP(D i, θ i 1 ) θ i = Train(A 0 A i ) end for Return θ 3 Given definitions: A 0 = P 100 A 1 = P 7 P 80 A 2 = P 5 P 80 A 3 = P 1 P 80 Note A 1 A 2 A 3

28 Learning Algorithm Gradually decrease density θ 0 = Train(A 0 ) for i = do D i = CDECODE(A i, θ i 1 ) A i = TOP(D i, θ i 1 ) θ i = Train(A 0 A i ) end for Return θ 3 Given definitions: A 0 = P 100 A 1 = P 7 P 80 A 2 = P 5 P 80 A 3 = P 1 P 80 Note A 1 A 2 A 3

29 Learning Algorithm Fill in partial trees θ 0 = Train(A 0 ) for i = do D i = CDECODE(A i, θ i 1 ) A i = TOP(D i, θ i 1 ) θ i = Train(A 0 A i ) end for Return θ 3 Given definitions: A 0 = P 100 A 1 = P 7 P 80 A 2 = P 5 P 80 A 3 = P 1 P 80 Note A 1 A 2 A 3

30 Learning Algorithm Select high-scoring trees θ 0 = Train(A 0 ) for i = do D i = CDECODE(A i, θ i 1 ) A i = TOP(D i, θ i 1 ) θ i = Train(A 0 A i ) end for Return θ 3 Given definitions: A 0 = P 100 A 1 = P 7 P 80 A 2 = P 5 P 80 A 3 = P 1 P 80 Note A 1 A 2 A 3

31 Learning Algorithm Train on the new set θ 0 = Train(A 0 ) for i = do D i = CDECODE(A i, θ i 1 ) A i = TOP(D i, θ i 1 ) θ i = Train(A 0 A i ) end for Return θ 3 Given definitions: A 0 = P 100 A 1 = P 7 P 80 A 2 = P 5 P 80 A 3 = P 1 P 80 Note A 1 A 2 A 3

32 Learning Algorithm Return the final model θ 0 = Train(A 0 ) for i = do D i = CDECODE(A i, θ i 1 ) A i = TOP(D i, θ i 1 ) θ i = Train(A 0 A i ) end for Return θ 3 Given definitions: A 0 = P 100 A 1 = P 7 P 80 A 2 = P 5 P 80 A 3 = P 1 P 80 Note A 1 A 2 A 3

33 Overview The learning algorithm Results Analysis

34 Two Settings Scenario 1 Transfer from English. Scenario 2 (voting) The different languages vote on dependencies. This scenario is true for cases such as Europarl.

35 Results on European Languages UAS German Spanish French Italian Portuguese Swedish θ 0 (Full trees)

36 Results on European Languages UAS German Spanish French Italian Portuguese Swedish θ 0 (Full trees) θ 3 (Full+dense)

37 Results on European Languages UAS German Spanish French Italian Portuguese Swedish θ 0 (Full trees) θ 3 (Full+dense) θ 3 (voting)

38 Results on European Languages (Comparison) UAS German Spanish French Italian Portuguese Swedish [Ma and Xia, 2014] θ 3 (voting) Sup. MST-1st

39 Comparison to Previous Work 95 [McDonald et al., 2011] [Zhang and Barzilay, 2015] dependency accuracy (avg. over 6 EU languages) [Ma and Xia, 2014] [McDonald et al., 2005] [Rasooli and Tetreault, 2015] MPH11 ZB15 MX14 θ 0 θ 3 θ 3 (voting) 1st-ord Sh-R Previous work Supervised models

40 Comparison to Previous Work 95 dependency accuracy (avg. over 6 EU languages) [McDonald et al., 2011] [Zhang and Barzilay, 2015] [Ma and Xia, 2014] θ 0 (full trees) θ 3 (full+dense) [McDonald et al., 2005] [Rasooli and Tetreault, 2015] % absolute improvement MPH11 ZB15 MX14 θ 0 θ 3 θ 3 (voting) 1st-ord Sh-R Previous work Supervised models

41 Comparison to Previous Work 95 dependency accuracy (avg. over 6 EU languages) [McDonald et al., 2011] [Zhang and Barzilay, 2015] [Ma and Xia, 2014] θ 0 (full trees) θ 3 (full+dense) θ 3 (voting) [McDonald et al., 2005] [Rasooli and Tetreault, 2015] % absolute improvement MPH11 ZB15 MX14 θ 0 θ 3 θ 3 (voting) 1st-ord Sh-R Previous work Supervised models

42 Overview The learning algorithm Results Analysis

43 Accuracy of Full Trees The accuracy of full trees is high. Voting increases the number of words per sentence, number of sentences and accuracy of full trees. Setting English target Voting Sen# 17K 77K Word/sen Prec. vs supervised

44 Density of Partial Trees in Voting The length and number of sentences are increased in partial dense trees. The accuracy of partial trees are lower than full trees. Setting P 100 P 80 P 7 Sen# 77K 243K Deps# Words/sen Density 100% 50% Prec. vs supervised

45 Accuracy across Different Languages Language P 100 P 80 P 7 Sup. #sen words/sen #dep Prec. #sen words/sen #dep Prec. German 47K K Spanish 109K K French 78K K Italian 101K K Portuguese 39K K Swedish 86K K Average 77K K

46 Conclusion We showed the utility of dense structures in projected dependencies. We showed a simple and effective learning method to utilize dense structures. Our performance is very close to a supervised parser. Future work: Applying to a broader set of languages. Using this model to improve machine translation.

47 Thanks

48 References I Goldberg, Y. and Nivre, J. (2013). Training deterministic parsers with non-deterministic oracles. TACL, 1: Ma, X. and Xia, F. (2014). Unsupervised dependency parsing with transferring distribution via parallel guidance and entropy regularization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages , Baltimore, Maryland. Association for Computational Linguistics. McDonald, R., Pereira, F., Ribarov, K., and Hajič, J. (2005). Non-projective dependency parsing using spanning tree algorithms. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT 05, pages , Stroudsburg, PA, USA. Association for Computational Linguistics. McDonald, R., Petrov, S., and Hall, K. (2011). Multi-source transfer of delexicalized dependency parsers. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 62 72, Edinburgh, Scotland, UK. Association for Computational Linguistics.

49 References II Rasooli, M. S. and Tetreault, J. (2015). Yara parser: A fast and accurate dependency parser. arxiv preprint arxiv: Zhang, Y. and Barzilay, R. (2015). Hierarchical low-rank tensors for multilingual transfer parsing. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon, Portugal.