Combining Neural Networks and Log-linear Models to Improve Relation Extraction

Transcription

1 Combining Neural Networks and Log-linear Models to Improve Relation Extraction Thien Huu Nguyen and Ralph Grishman Computer Science Department, New York University

2 Outline Relation Extraction Dataset Previous Work Models Experiments and conclusions

3 Relation Extraction (RE) EMPLOYMENT PERSON ORGANIZATION Fred Flintstone was named CTO of Time Bank Inc. in Relation Mention = Entity Mention Pair + Sentence Classify relation mentions (X) into some predefined semantic relation classes (Y) : P

4 Dataset The ACE (Automatic Content Extraction) 2005 dataset 600 documents manually annotated for entity mentions and relations between them Only focus on the relation mentions that have two entity mentions within the same sentences There are 6 classes of relations a 7-class classification problem (6 annotated relation classes plus one class for NONE ) The total number of relation mentions: 43,497

5 Experimental Scenario Documents belong to 6 different domains: broadcast news (bn), newswire (nw), broadcast conversation (bc), conversational telephone speech (cts), web blog (wl), usenet (un) The domains are very different in vocabulary and styles. Split bc into two parts, use one part for development data and the other half for testing (same split with previous work for comparison) Train models on bn+nw and evaluate the models on the test part of bc, cts and wl

6 Previous Work Feature-based Models - Represent relation mentions by various binary features based on linguistic analysis and knowledge resources Lexical features: words, n-grams etc Syntactic features: part of speech tagging, dependency paths, constituent parse trees Semantic features: word clusters, gazetteers etc Fred Flintstone was named CTO of Time Bank Inc. in Feed the features into some log-linear classifiers (mostly logistic classifiers) to perform classification Capture the discrete structures within sentences for RE

7 Convolutional Neural Networks (CNN) for for RE RE P CNN Pooling In the morning, the President traveled to Detroit. e1 e2 Words are transformed into vectors using word embeddings and word feature vectors (i.e, relative positions, dependency types etc) Capture the hidden representations of the consecutive and short k-grams in the sentences

8 Recurrent Neural Networks (RNN) for for RE RE P RNN Pooling Concatenation In the morning, the President traveled to Detroit Using GRU (gated recurrent units, a simplified version of LSTM) Variants: Forward, Backward, Bidirect Capture long and possibly non-consecutive patterns for RE

9 Some Examples CNN The Iraqi unit in possession of those guns The al Qaeda chief operations officer RNN Some of the 40,000 British troops are kicking up a lot of dust in the Iraqi desert making sure that nothing is left behind them that could hurt them

10 Models The Log-linear, CNN and RNN models focus on different angles for relation extraction Combine the three models to improve the performance for RE

11 Combing CNN and RNN for for RE RE Ensembling P E ~ P CNN x P RNN Voting -Majority Voting on the outputs of CNN and RNN -If two models disagree, pick the prediction with higher probability

12 Combing CNN and RNN for for RE RE Stacking: 2 variants: CNN-RNN and RNN-CNN In the morning, the President traveled to Detroit

13 Evaluation of of the the Combined Models Performance on the development data Separate Models F1 BIDIRECT FORWARD BACKWARD CNN Ensembling CNN- BIDIRECT CNN- FORWARD CNN- BACKWARD Voting CNN- BIDIRECT CNN- FORWARD F F Stacking F1 CNN-BIDIRECT CNN- FORWARD CNN- BACKWARD BIDIRECT-CNN FORWARD- CNN BACKWARD- CNN CNN- BACKWARD 65.52

14 Combing the the Neural Networks with the the Log-Linear Modes Let P login be the probability distribution of the log-linear model P login uses the state-of-the-art feature set in the literature for RE Combine P login with a neural network with probability P N by the element-wise multiplication: P hybrid-n ~ P login x P N Combine P login with a voting model of the neural networks P CNN (Y X) and P RNN : P hybrid-cnn ~ P login x P CNN P hybrid-rnn ~ P login x PR NN Vote on P hybrid-cnn and P hybrid-rnn

15 The Hybrid-Voting Models The hybrid model corresponding to the neural network N might cover a different relation mention set than N perform voting on the outputs of N and its corresponding hybrid model For a neural network model N: Vote on P hybrid-n and P N For a voting model of the neural networks CNN and RNN: Vote on P CNN, P RNN, P hybrid-cnn and P hybrid-rnn

16 Evaluation of of the the Combined Models Neural Networks Separate Models CNN BIDIRECT FORWARD BACKWARD Combined Models VOTE-BIDIRECT STACK-FORWARD VOTE-BACKWARD VOTE-BIDIRECT = Voting on CNN and BIDIRECT STACK-FORWARD = Stacking CNN and FORWARD VOTE-BACKWARD = Voting on CNN and BACKWARD

17 Evaluation of of the the Combined Models Separate Models Neural Networks Hybrid Models CNN BIDIRECT FORWARD BACKWARD Combined Models + P login VOTE-BIDIRECT STACK-FORWARD VOTE-BACKWARD VOTE-BIDIRECT = Voting on CNN and BIDIRECT STACK-FORWARD = Stacking CNN and FORWARD VOTE-BACKWARD = Voting on CNN and BACKWARD

18 Evaluation of of the the Combined Models + P login + Voting Neural Networks Hybrid Models Hybrid-Voting Models Separate Models CNN BIDIRECT FORWARD BACKWARD Combined Models VOTE-BIDIRECT STACK-FORWARD VOTE-BACKWARD VOTE-BIDIRECT = Voting on CNN and BIDIRECT STACK-FORWARD = Stacking CNN and FORWARD VOTE-BACKWARD = Voting on CNN and BACKWARD

19 Comparing the the State of of the the Art Model bc cts wl Ave The state-of-the-art Systems (Gormley et al., 2015) FCM Hybrid FCM Hybrid-Voting Systems VOTE-BIDIRECT STACK-FORWARD VOTE-BACKWARD

20 Conclusions Feature-based Log-Linear Models are good to capture the discrete structures for RE Convolutional Neural Networks are good to capture the hidden short and consecutive patterns for RE Recurrent Neural Networks are good to capture the hidden long and nonconsecutive patterns for RE Combining them helps to improve the performance for RE significantly Thien Huu Nguyen Computer Science Department, New York University

21 Comparing the the State of of the the Art Model bc cts wl Ave The state-of-the-art Systems FCM Hybrid FCM Separate Systems Log-Linear CNN BIDIRECT FORWARD BACKWARD Hybrid-Voting Systems VOTE-BIDIRECT STACK-FORWARD VOTE-BACKWARD