Projective Dependency Parsing with Perceptron

Size: px

Start display at page:

Download "Projective Dependency Parsing with Perceptron"

Dominic Reed
5 years ago
Views:

1 Projective Dependency Parsing with Perceptron Xavier Carreras, Mihai Surdeanu, and Lluís Màrquez Technical University of Catalonia 8th June 2006

2 Outline Introduction Parsing and Learning Parsing Model Parsing Algorithm Global Perceptron Learning Algorithm Features Experiments and Results Results Discussion

3 Outline Introduction Parsing and Learning Parsing Model Parsing Algorithm Global Perceptron Learning Algorithm Features Experiments and Results Results Discussion

4 Introduction Motivation Blind treatment of multilingual data Use well-known components Our Dependency Parsing Learning Architecture: Eisner dep-parsing algorithm, for projective structures Perceptron learning algorithm, run globally Features: state-of-the-art, with some new ones In CoNLL-X data, we achieve moderate performance: of overall labeled attachment score 10th position in the ranking

5 Outline Introduction Parsing and Learning Parsing Model Parsing Algorithm Global Perceptron Learning Algorithm Features Experiments and Results Results Discussion

6 Parsing Model A dependency tree is decomposed into labeled dependencies, each of the form [h, m, l] where : h is the position of the head word m is the position of the modifier word l is the label of the dependency Given a sentence x the parser computes: dparser(x, w) = arg max score(x, y, w) y Y(x) = arg max score([h, m, l], x, y, w) y Y(x) [h,m,l] y = arg max w l φ([h, m], x, y) y Y(x) [h,m,l] y w = (w 1,..., w l,..., w L ) is the learned weight vector φ is the feature extraction function, given a priori

7 The Parsing Algorithm of Eisner (1996) Assumes that dependency structures are projective; in CoNLL data, this only holds for Chinese Bottom-up dynamic programming algorithm In a given span from word s to word e : 1. Look for the optimal point giving internal structures: s r r+1 e 2. Look for the best label to connect the structures:??

8 The Parsing Algorithm of Eisner (1996) (II) A third step assembles two dependency structures without using learning s r r e s r e

9 Perceptron Learning Global Perceptron (Collins 2002): trains the weight vector dependently of the parsing algorithm. A very simple online learning algorithm: it corrects the mistakes seen after a training sentence is parsed. w = 0 for t = 1 to T foreach training example (x, y) do ŷ = dparser(x, w) foreach [h, m, l] y \ŷ do /* missed deps */ w l = w l + φ(h, m, x, ŷ) foreach [h, m, l] ŷ \y do /* over-predicted deps */ w l = w l φ(h, m, x, ŷ) return w

10 Outline Introduction Parsing and Learning Parsing Model Parsing Algorithm Global Perceptron Learning Algorithm Features Experiments and Results Results Discussion

11 Feature Extraction Function φ(h, m, x, y): represents in a feature vector a dependency from word positions m to h, in the context of a sentence x and a dependency tree y φ(h, m, x, y) = φ token (x, h, head ) + φ tctx (x, h, head ) + φ token (x, m, mod ) + φ tctx (x, m, mod ) + φ dep (x, mm h,m, d h,m ) + φ dctx (x, mm h,m, d h,m ) + φ dist (x, mm h,m, d h,m ) + φ runtime (x, y, h, m, d h,m ) where mm h,m is a shorthand for the tuple min(h, m), max(h, m) d h,m indicates the direction of the dependency

12 Context-Independent Token Features Represent a token i type indicates the type of token being represented, i.e. head or mod Novel features are in red. φ token (x, i, type) type word(x i ) type lemma(x i ) type cpos(x i ) type fpos(x i ) foreach f morphosynt(x i ) : type f type word(x i ) cpos(x i ) foreach f morphosynt(x i ) : type word(x i ) f

13 Context-Dependent Token Features Represent the context of a token x i The function extracts token features of surrounding tokens It also conjoins some selected features along the window φ tctx (x, i, type) φ token (x, i 1, type string( 1)) φ token (x, i 2, type string( 2)) φ token (x, i + 1, type string( 1)) φ token (x, i + 2, type string( 2)) type cpos(x i ) cpos(x i 1 ) type cpos(x i ) cpos(x i 1 ) cpos(x i 2 ) type cpos(x i ) cpos(x i+1 ) type cpos(x i ) cpos(x i+1 ) cpos(x i+2 )

14 Context-Independent Dependency Features Features of the two tokens involved in a dependency relation dir indicates whether the relation is left-to-right or right-to-left φ dep (x, i, j, dir) dir word(x i ) cpos(x i ) word(x j ) cpos(x j ) dir cpos(x i ) word(x j ) cpos(x j ) dir word(x i ) word(x j ) cpos(x j ) dir word(x i ) cpos(x i ) cpos(x j ) dir word(x i ) cpos(x i ) word(x j ) dir word(x i ) word(x j ) dir cpos(x i ) cpos(x j )

15 Context-Dependent Dependency Features Capture the context of the two tokens involved in a relation dir indicates whether the relation is left-to-right or right-to-left φ dctx (x, i, j, dir) dir cpos(x i ) cpos(x i+1 ) cpos(x j 1 ) cpos(x j ) dir cpos(x i 1 ) cpos(x i ) cpos(x j 1 ) cpos(x j ) dir cpos(x i ) cpos(x i+1 ) cpos(x j ) cpos(x j+1 ) dir cpos(x i 1 ) cpos(x i ) cpos(x j ) cpos(x j+1 )

16 Surface Distance Features Features on the surface tokens found within a dependency relation Numeric features are discretized using binning to a small number of intervals φ dist (x, i, j, dir) foreach(k (i, j)): dir cpos(x i ) cpos(x k ) cpos(x j ) number of tokens between i and j number of verbs between i and j number of coordinations between i and j number of punctuations signs between i and j

17 Runtime Features Capture the labels of the dependencies that attach to the head word This information is available in the dynamic programming matrix of the parsing algorithm? h m l 1 l 2 l φ runtime (x, y, h, m, dir) foreach i, 1 i S : dir cpos(x h ) cpos(x m ) l i dir cpos(x h ) cpos(x m ) l 1 dir cpos(x h ) cpos(x m ) l 1 l 2 dir cpos(x h ) cpos(x m ) l 1 l 2 l 3 dir cpos(x h ) cpos(x m ) l 1 l 2 l 3 l 4 3 l... S

18 Outline Introduction Parsing and Learning Parsing Model Parsing Algorithm Global Perceptron Learning Algorithm Features Experiments and Results Results Discussion

19 Results GOLD UAS LAS Japanese Chinese Portuguese Bulgarian German Danish Swedish Spanish Czech Slovene Dutch Arabic Turkish Overall

20 Feature Analysis φ token +φ dep +φ tctx +φ dist +φ runtime +φ dctx Japanese Portuguese Spanish Turkish This table shows LAS at increasing feature configurations All families of feature patterns help significantly

21 Errors Caused by 4 Factors 1. Size of training sets: accuracy below 70% for languages with small training sets: Turkish, Arabic, and Slovene. 2. Modeling large distance dependencies: our distance features (φ dist ) are insufficient to model well large-distance dependencies: to root >= 7 Spanish Portuguese Modeling context: our context features (φ dctx, φ tctx, and φ runtime ) do not capture complex dependencies. Top 5 focus words with most errors: Spanish: y, de, a, en, and que Portuguese: em, de, a, e, and para 4. Projectivity assumption: Dutch is the language with most crossing dependencies in this evaluation, and the accuracy we obtain is below 70%.

22 Thanks!

Agenda for today. Homework questions, issues? Non-projective dependencies Spanning tree algorithm for non-projective parsing

Agenda for today. Homework questions, issues? Non-projective dependencies Spanning tree algorithm for non-projective parsing Agenda for today Homework questions, issues? Non-projective dependencies Spanning tree algorithm for non-projective parsing 1 Projective vs non-projective dependencies If we extract dependencies from trees,