Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank text

Transcription

1 Philosophische Fakultät Seminar für Sprachwissenschaft Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank text 06 July 2017, Patricia Fischer & Neele Witte

2 Overview Sentiment Analysis Sentiment Treebank Neural Network Architecture Recursive Neural Network Matrix Vector RNN Recursive Neural Tensor Network Experiments Fine-grained Sentiment for All Phrases Full Sentence Binary Sentiment Contrastive Conjunction High Level Negation Most Positive/Negative Phrases 2 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

3 Sentiment Analysis 3 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

4 Sentiment Analysis Sentiment analysis is the measurement of positive and negative language. Sentiment Analysis is the process of determining whether a piece of writing is positive, negative or neutral. It s also known as opinion mining, deriving the opinion or attitude of a speaker. Using NLP, statistics, or machine learning methods to extract, identify, or otherwise characterize the sentiment content of a text unit. 4 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

5 Sentiment Analysis Classification of users, texts, phrases, words Ratings - Binary: or or - Scales: - Open category: 5 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

6 Sentiment Analysis Challenges Opinions expressed in complex ways Stylistic devices such as sarcasm, irony etc. 6 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

7 Sentiment Analysis Examples 7 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

8 Sentiment Analysis Examples 8 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

9 From To 9 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

10 Motivation for New Model and Database Not only want to represent sentiment by the sum of the sentiments of their components, but by the composition of them Word order is important, especially for detecting negation No database with annotated single sentences (usually documents) good results for long texts but not for short texts (e.g. Twitter Data), phrases, segments Accuracy for three classes on short texts: below 60% Aim: construct a database to train and evaluate compositional models 10 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

11 Sentiment Treebank 11 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

12 Sentiment Treebank Normalized histogram of sentiment annotations at each n-gram length. 11,855 single sentences, 215,154 unique phrases Movie review excerpts from rottentomatoes.com Stanford parser Labeling: amazon mechanical turk Fine-grained sentiment classification 12 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

13 Semantic Representation of Words Map words into vector space to represent their meaning (semantic) Similar words are close to each other How can we represent meaning of longer phrases? 13 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

14 Semantic Representation of Sentences Can we find a semantic representation for sentences (of arbitrary length) as well? Map phrases into the same vector space as well How? 14 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

15 Semantic Representation of Sentences Bag of Words: represent sentence as Bag of words and create one vector per sentence Problem: word order ignored Sentence embeddings: create embeddings for n-grams (e.g. 7-gram represents a sentence embedding) Problem: cannot create so many embeddings, sentences can be very long 15 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

16 Recursive Neural Network Principle of Compositionality The meaning (vector) of a sentence is defined by 1. the meaning of its words 2. the rules that combine them Recursive Neural Nets can jointly learn compositional vector representations and parse trees 16 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

17 Recursive Structure 1. Extract a binary syntactic tree 2. Recursively merge smaller segments to get representation of bigger segments / the whole sentence 17 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

18 Building Blocks for Neural Network Composition function for merging two children f (W [c1; c2] + b) (1) Classification function for assigning a label to each node y a = softmax(w s a) (2) Loss function: the cross-entropy error between the predicted distribution and the target distribution 18 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

19 Recursive NN Structure 19 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

20 MV-RNN Structure 20 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

21 Recursive Neural Tensor Network a p 2 p 1 p 1 = f ( [b c ] T [ ] b V [1:d] c + W [ ] ) b c not b very c good ( [ ] T [ ] a a p 2 = f V [1:d] p 1 p 1 + W [ ] ) a p 1 21 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

22 Experiments 22 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

23 Fine-grained Sentiment for All Phrases Model Fine-grained Positive/Negative All Root All Root NB SVM BiNB VecAvg RNN MV-RNN RNTN Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

24 Full Sentence Binary Sentiment Model Fine-grained Positive/Negative All Root All Root NB SVM BiNB VecAvg RNN MV-RNN RNTN Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

25 Contrastive Conjunction There are slow and repetitive parts but it has just enough spice to keep it interesting. 25 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

26 High Level Negation Can the Model correctly classify the reversal from positive to negative sentiment? 26 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

27 Negating Sentence with Negative Sentiment How often did the model increase positive activation in the sentiment? 27 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

28 Negating Sentence with Negative Sentiment Sentiment of the sentence will become less negative (not necessarily positive) 28 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

29 Most Positive/Negative Phrases 29 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

30 References Bo Pang and Lillian Lee. (2008) Opinion mining and sentiment analysis. In Foundations and Trends in Information Retrieval. R. Socher, C. D. Manning, and A. Y. Ng. (2010) Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Fea- ture Learning Workshop. R. Socher, C. Lin, A. Y. Ng, and C.D. Manning. (2011a) Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In ICML. R. Socher, B. Huval, C. D. Manning, and A. Y. Ng. (2012) Semantic compositionality through recursive matrix-vector spaces. In EMNLP. 30 Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen

31 Thank you! Contact: Philosophische Fakultät Seminar für Sprachwissenschaft Wilhelmstraße 19, Tübingen Phone: +49 (0) Fax: +49 (0) Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c 2017 Universität Tübingen