CSE 250B Assignment 4 Report

Transcription

1 CSE 250B Aignment 4 Report March 24, 2012 Yuncong Chen yuncong@c.ucd.edu Pengfei Chen pec008@ucd.edu Yang Liu yal060@c.ucd.edu Abtract In thi project, we implemented the recurive autoencoder (RAE) a decribed in Socher paper to dicover the entiment of entence. We train and tet our RAE with a dataet of over entence from movie review, and achieve 75.4% accuracy. 1 Introduction The paper of Socher2011 decribe a framework that ue recurive autoencoder to analye the meaning of text. They repreent each word in the vocabulary with a fixed-length vector that encode the meaning of that word. By contructing a recurive autoencoder, the meaning vector of word are propagated forward to a top node that encapulate the meaning of the entire text egment. Many application can be built uing thee meaning vector, one of which i to detect the entiment underlying a piece of text. In thi project we implement Socher approach to analye the entiment of entence from movie review. The data i a et of entence from the webite Rotten Tomato, each of 5-50 word long and labelled with an either poitive or negative entiment. We aim to learn an RAE that predict the entiment for thee entence. 2 Backpropagation in RAE Recurive autoencoder i an intance of a feedforward neural network. We aim to learn the weight on the edge that minimize ome meaure of error for example in the training et. The error gradient with repect to thee parameter can be computed via backpropagation. 2.1 Structure of RAE In our RAE, a phrae node i a node that i fed from two other phrae/word node. Each phrae node ha two recontruction node that trie to recover the meaning vector of that phrae node two children. Each phrae node, a well a each word node, ha a label node, that contain the predicted label for the meaning vector of that phrae/word node. Suppoe the dimenion of the meaning vector i d. The parameter θ of an RAE are the weight on the edge, including the extended combination-weight matrix W = [W l, W r, b w ] R d (2d+1), the extended recontruction-weight matrix U = [U l, b l ; U r, b r ] R 2d (d+1) and the extended label matrix V = [V 0, b v ] R p (d+1), where p i the dimenion of label. We alo learn the meaning vector of each word in the vocabulary, L R d C, where C i the ize of the vocabulary. Suppoe an internal node receive activation a, and it value r = tanh(a). The two children of are [c l ] and [c r ], with value c l and c r. The parent node of i p. The two recontruction node are [y l ] and [y r ], with value y l and y r. The label node i v with value v. The feedforward formula of the value in the RAE are, 1

2 r = tanh(w [c l ; c r ; 1]) (1) [y l ; y r ] = tanh(u[r; 1]) (2) v = V [r; 1] (3) 2.2 Error and error gradient For each non-terminal node, the recontruction error i defined a a quare lo E rec, = 1 ( [yl ] [c l ] 2 + [y r ] [c r ] 2) (4) 2 For each non-output node, the label error can be defined uing the cro-entropy a in the coure note: or a quare lo a in Socher code E label, = t T log oftmax(v [r ; 1]) (5) E label, = 1 2 t v 2 (6) where t i the target label for. For the ake of computational implicity, we ue quared lo in our implemetation. The error E (m) for the m th training example i defined a the um of thee two kind of error over all aociated node in the RAE of that example. For the m th example, denote the et of non-terminal node and the et of non-output node (including leave) in it RAE a NT (m) and NO (m) repectively. E (m) = α E rec, + (1 α) E label, (7) NT (m) NO (m) The total error for all training example i J = 1 E (m) + λ N 2 θ 2 (8) m where N i the total number of non-terminal node. It follow that the error gradient with repect to the parameter θ i 2.3 Compute δ for each node J θ = 1 + λθ (9) N θ m For each node in the RAE of one example, we define a vector δ = / a, i.e. the error gradient with repect to the input vector a of that node. We ditinguih between output node and non-output node. The output node in an RAE are the two recontruction node and one label node for each non-terminal node. For all recontruction node [y l ] and [y l ], For all label node v, with quared lo δ [yl ] = α(y l c l ) (10) δ [yr] = α(y r c r ) (11) δ v = (1 α)(r t )σ (a v ) (12) 2

3 The non-output node in an RAE are the phrae node and leaf node. For each uch node, we compute it δ vector by backpropagation from the δ vector of connected output node. We firt define [W lr ], If i not a leaf node, δ = tanh (a ) { Wl if i a left child of p [W lr ] = W r if i a right child of p ( ) δ[y T l ] U l + δ[y T r] U r + δp T [W lr ] α([y r ] p r ) + δv T V (13) If i a leaf node, δ = [W lr ] T δ p α([y l ] p r ) + V T 0 δ v (14) With thee δ vector, we can eaily compute the error gradient with repect to the parameter. 2.4 Compute the gradient for each node The total error gradient i the cumulation of the error gradient obtained at each node. The change of U propagate to the total error through recontruction node, o for each recontruction node [y l ] and [y r ] we compute the gradient with repect to U: E (m) U = [δ [y l ] ; δ [yr] ][r; 1] T (15) The change of W propagate to the total error through phrae node, o for each phrae node, we compute the gradient with repect to W. W = δ [c l ; c r ; 1] T (16) The change of V propagate to the total error through each label node v, o for each label node v, we compute the gradient with repect to V. V = δ v [r ; 1] T (17) The change of L propagate to the total error through each leaf node. Becaue there i no nonlinearity or ummation over input at a leaf node, the gradient i imply 3 Implementation 3.1 Parameter L = δ (18) In our experiment, the input vector dimenion d = 50, the label dimenion p = 1. We et α = Compute gradient via backpropagation We firt do a forward propagation tarting from the leave, during which we compute the value of all node, a well a the δ vector for all output node. Then we do a backpropagation to compute the δ vector for all non-output node. Starting from the root node, we perform a depth-firt travere of all non-ouput node, and compute the δ vector for each node viited. In thi way, we guarantee that when we compute the δ for a node, the δ vector of it parent non-output node i already available. 3

4 When the δ vector of a node i obtained, computing the contribution to the total gradient through that node i traightforward by equation (12)-(15). We then add up all the contribution from each node, and obtain the total gradient. The computed gradient are compared to the reult of numerical differentiation of the lo function 1. We tet the gradient tarting from a minimal RAE with only two input and each input i imply a two-dimenional vector. When the gradient match, we then increae the number and dimenion of input, and add nonlinearity and normalization to the value. We alo try to determine whether the problem come from the recontruction part or the label part, by teting with α = 0 and α = 1. After tep-by-tep debugging, we manage to make the ratio between computed gradient for the complete RAE and the reult of numerical differentiation extremely cloe to unity, which confirm the correctne of our gradient evaluation. 3.3 Optimization For optimization, we ued L-BFGS 2. We originially intended to compare it performance againt SGD, but due to time contraint, we did not manage to do the experiment. We gue that a a batch optimization technique, L-BFGS will be lower than SGD, which make an update to the weight after proceing each example. 3.4 Prediction After we obtain the parameter, we build an RAE for each tet example, and then compute label value for the top node uing the trained V matrix. In Socher code, they take the top node vector and the average of all vector in the tree a a concatenated feature vector, and train a eperate claifier uing logitic regreion. After three iteration, our method give an accuracy of 68.9%, while Socher method give an accuracy of 75.4%. Thi how that Socher method i empirically better becaue it conider the average meaning in the phrae level and i more robut to noie at the top node. 4 Dicuion 4.1 I normalization neceary? Theoretically, normalization i neceary for a node value to be a proper meaning vector which lie on an embedded d-dimenional phere. However, we did not normalize the node value in our implementation, a it make the gradient harder to calculate. We etimate that normalization doe not play an important role in the prediction accuracy ince ultimately it i the proportion of element in the meaning vector that determine the prediction, not the magnitude of the meaning vector. In addition, the value of each node will not be too large due to lack of normalization a it i retricted by a igmoid function. 4.2 I full backup neceary? Dividing a RAE into eperate triplet training example make the gradient computation eaier, but the obtained gradient i jut an approximation to the true gradient. An advantage of thi approximation may be the poibility of more parallel computation. 5 Concluion In thi project we implemented the recurive autoencoder (RAE) a decribed in Socher paper to detect the entiment of movie review. We detailed the algorithm for uing backpropagation to compute error gradient with repect to the parameter. We train and mchmidt/software/minfunc.html 4

5 tet our RAE with entence from movie review, and achieve 75.4% accuracy. We alo dicued ome deign choice and algorithmic iue in our implementation. 5