Transformation Networks for Target-Oriented Sentiment Classification ACL / 25

Transcription

1 Transformaton Networks for Target-Orented Sentment Classfcaton 1 Xn L 1, Ldong Bng 2, Wa Lam 1, Be Sh 1 1 The Chnese Unversty of Hong Kong 2 Tencent AI Lab ACL Jont work wth Tencent AI Lab Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

2 Outlne 1 Target-Orented Sentment Classfcaton Introducton Problem Formulaton 2 Transformaton Networks for Target-Orented Sentment Classfcaton Motvaton The proposed model 3 Experment Settngs Comparatve Study Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

3 Outlne 1 Target-Orented Sentment Classfcaton Introducton Problem Formulaton 2 Transformaton Networks for Target-Orented Sentment Classfcaton Motvaton The proposed model 3 Experment Settngs Comparatve Study Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

4 Introducton Target-Orented Sentment Classfcaton (TOSC) s to detect the overall opnons / sentments of the user revew towards the gven opnon target. TOSC s a supportng task of Target / Aspect-based Sentment Analyss [5]. TOSC has been nvestgated extensvely n other names: Aspect-level Sentment Classfcaton [1, 7, 10, 11, 12]. Targeted Sentment Predcton [6, 14]. Target-Dependent Sentment Classfcaton [2, 9]. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

5 Outlne 1 Target-Orented Sentment Classfcaton Introducton Problem Formulaton 2 Transformaton Networks for Target-Orented Sentment Classfcaton Motvaton The proposed model 3 Experment Settngs Comparatve Study Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

6 Problem Formulaton TOSC s a typcal classfcaton task but the nput texts come from two sources: 1 Target: explctly mentoned phrase of opnon target, also called aspect term or aspect. 2 Context: the orgnal revew sentence or the sentence wthout target phrase. TOSC s to predct the overall sentment of the context towards the target. Example [Boot tme] s super fast, around anywhere from 35 seconds to 1 mnute. Ths revew conveys postve sentment over the nput Boot tme. Great [food] but the [servce] s dreadful. Gven the target food, the sentment polarty s postve whle f the nput target s servce, t becomes negatve. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

7 Outlne 1 Target-Orented Sentment Classfcaton Introducton Problem Formulaton 2 Transformaton Networks for Target-Orented Sentment Classfcaton Motvaton The proposed model 3 Experment Settngs Comparatve Study Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

8 Motvaton 1 Convolutonal Neural Network (CNN) s more sutable for ths task than Attenton-based Models [1, 6, 7, 10, 11, 12, 13]. Sentments towards the targets are usually determned by key phrases. Example: Ths [dsh] s my favorte and I always get t and never get tred of t. CNN whose am s to capture the most nformatve n-grams (e.g., s my favorte ) n the sentence should be a sutable model. Attenton-based weghted combnaton of the entre word-level features may ntroduce some noses (e.g., never and tred n above sentence). We employ proxmty-based CNN rather than attenton-based RNN as the top-most feature extractor. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

9 Motvaton 2 CNN lkely fals n cases where a sentence expresses dfferent sentments over multple targets. Example: great [food] but the [servce] was dreadful! CNN cannot fully explore the target nformaton va vector concatenaton. Combnng context nformaton and word embeddng s an effectve way to represent a word n the convoluton-based archtecture [4] Our Soluton: () We propose a Target-Specfc Transformaton (TST) component to better consoldate the target nformaton wth word representatons. () We desgn two context-preservng mechansms Adaptve Scalng (AS) and Loseless Forwardng (LF) to combne the contextualzed representatons and the transformed representatons. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

10 Motvaton 3 Most of the exstng works do not dscrmnate dfferent words n the same target phrase In the target phrase, dfferent words would not contrbute equally to the target representaton. For example, n amd turn processor, phrase head processor s more mportant than amd and turn. Our TST solves ths problem n two steps: () Explctly calculatng the mportance scores of the target words. () Conductng word-level assocaton between the target and ts context. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

11 Outlne 1 Target-Orented Sentment Classfcaton Introducton Problem Formulaton 2 Transformaton Networks for Target-Orented Sentment Classfcaton Motvaton The proposed model 3 Experment Settngs Comparatve Study Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

12 B-drectonal LSTM Transformaton Archtecture Convoluton Layer Model Overvew y Conv2d (L) h 1 (L) h 2 (L) h n CPT CPT CPT CPT h n (l+1) LF/AS TST (l) h n fully-connected (1) h 1 (1) h 2 (1) h n r τ h n (l) CPT CPT CPT h 1 (0) h 2 (0) h n (0) h 1 τ h 2 τ h m τ h n (l) x 1 τ x 2 τ x m τ x 1 x 2 x n Fgure: Archtecture of TNet. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

13 Model Overvew The proposed TNet conssts of the followng three components: 1 (BOTTOM) B-drectonal LSTM for memory buldng Generatng contextualzed word representatons. 2 (MIDDLE) Deep Transformaton archtecture for learnng target-specfc word representatons Refnng word-level representatons wth the nput target and the contextual nformaton. 3 (TOP) Proxmty-based convolutonal feature extractor. Introducng poston nformaton to detect the most salent features more accurately. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

14 Deep Transformaton Archtecture Deep Transformaton Archtecture stacks multple Context-Preservng Transformaton (CPT) layers Deeper network helps to learn more abstract features (He et al., CVPR 2016; Lecun et al., Nature 2015). Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

15 CPT Layer The functons of the CPT layer are two folds: 1 Incorporatng opnon target nformaton nto the word-level representatons. Generatng context-aware target representatons r τ condtoned on the -th word representaton h (l) fed to the l-th layer: r τ = m j=1 h τ j F(h (l), h τ j ), (l) h TST fully-connected τ (l) r h τ h τ 1 h τ 2 h m h (l) F(h (l), h τ j ) = exp (h (l) h τ j ) m k=1 exp (h(l) hk τ ), Obtanng target-specfc word representatons h (l) : h (l) = g(w τ [h (l) : r τ ] + b τ ), x 1 τ x 2 τ x m τ Fgure: Target-Specfc Transformaton (TST) component Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

16 CPT Layer 2 Preservng context nformaton for the upper layers We desgn two Context-Preservng Mechansms to add context nformaton back to the transformed word features h (l) () Adaptve Scalng (AS) (Smlar to Hghway Connecton [8]): t (l) = σ(w trans h (l) + b trans ), h (l+1) = t (l) h (l) + (1 t (l) ) h (l). () Lossless Forwardng (LF) (Smlar to Resdual Connecton [3]): h (l+1) = h (l) + h (l). Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

17 Proxmty-based Convolutonal Feature Extractor Ths component ams to capture the most salent feature w.r.t. the current target for sentment predcton. As observed n (Chen et al., 2017; L and Lam, 2017), dstance nformaton s effectve for better locatng the salent features. Basc dea: Up-weghtng the words close to the target and down-weghtng those far away from the target. Convolutonal neural network (Km, 2014) s used to extract features from the weghted word representatons. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

18 Outlne 1 Target-Orented Sentment Classfcaton Introducton Problem Formulaton 2 Transformaton Networks for Target-Orented Sentment Classfcaton Motvaton The proposed model 3 Experment Settngs Comparatve Study Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

19 Settngs Datasets LAPTOP, REST: datasets from SemEval14 ABSA challenge, contanng the user revews from laptop doman and restaurant doman respectvely. TWITTER: a dataset bult n (Dong et al., 2014), contanng twtter posts and the opnon targets are annotated. Compared Models Tradtonal Models: SVM (Krtchenko et al., 2014). Attenton-based Models: ATAE-LSTM (Wang et al., 2016), MemNet (Tang et al., 2016), IAN (Ma et al., 2017), BILSTM-ATT-G (Lu and Zhang, 2017), RAM (Chen et al., 2017). Other Neural Models: AdaRNN (Dong et al., 2014), TD-LSTM (Tang et al., 2016), AE-LSTM (Wang et al., 2016), CNN-ASP Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

20 Outlne 1 Target-Orented Sentment Classfcaton Introducton Problem Formulaton 2 Transformaton Networks for Target-Orented Sentment Classfcaton Motvaton The proposed model 3 Experment Settngs Comparatve Study Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

21 Man Results TNet varants Baselnes Models LAPTOP REST TWITTER ACC Macro-F1 ACC Macro-F1 ACC Macro-F1 TNet-LF 76.01, 71.47, 80.79, , 73.36, TNet-AS 76.54, 71.75, 80.69, 71.27, 74.97, 73.60, SVM AdaRNN AE-LSTM ATAE-LSTM IAN CNN-ASP TD-LSTM MemNet BILSTM-ATT-G RAM The proposed TNet-LF and TNet-AS consstently outperform the baselnes. TNet varants perform well on both user revews (LAPTOP & REST) and twtter posts (TWITTER). Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

22 Ablaton Experment TNet varants CPT Alternatves Ablated TNet Models LAPTOP REST TWITTER ACC Macro-F1 ACC Macro-F1 ACC Macro-F1 TNet-LF 76.01, 71.47, 80.79, , 73.36, TNet-AS 76.54, 71.75, 80.69, 71.27, 74.97, 73.60, LSTM-ATT-CNN LSTM-FC-CNN-LF LSTM-FC-CNN-AS TNet w/o transformaton TNet w/o context TNet-LF w/o poston TNet-AS w/o poston Usng attenton (ATT) and fully-connected layer (FC) to replace CPT layer makes the performance worse. Each component / element n TNet contrbutes to the overall performance mprovement. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

23 Impact of CPT layer number We conduct experments on the held-out tranng data of LAPTOP and vary layer number L from 2 to 10, ncreased by Accuracy (%) Macro-F1 (%) TNet-LF TNet-AS TNet-LF TNet-AS Increasng the layer number can ncrease the performance but the results wll go down when L 4 due to the lmted tranng data. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

24 Case Study Sentence BILSTM-ATT-G RAM TNet-LF TNet-AS 1. Ar has hgher [resoluton] P but the [fonts] N are small. (N, N) (N, N) (P, N) (P, N) 2. Great [food] P but the [servce] N s dreadful. (P, N) (P, N) (P, N) (P, N) 3. Sure t s not lght and slm but the [features] P make up for t 100%. N N P P 4. Not only dd they have amazng, [sandwches] P, [soup] P, [pzza] P etc, but ther [homemade sorbets] P (P, O, O, P) (P, P, O, P) (P, P, P, P) (P, P, P, P) are out of ths world! 5. [startup tmes] N are ncredbly long : over two mnutes. P P N N 6. I am pleased wth the fast [log on] P, speedy [wf connecton] P and the long [battery lfe] P ( > 6 hrs ). (P, P, P) (P, P, P) (P, P, P) (P, P, P) 7. The [staff] N should be a bt more frendly. P P P P Our TNet can make correct predctons when the opnon s target specfc, e.g., long n the 5th and the 6th example. TNet can capture the salent features for target sentment predcton accurately. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

25 Summary Our TNet employs CNN as feature extractor to detect the salent features, avodng ntroducng the noses. Armed wth target-specfc word representaton and proxmty nformaton, the TNet varants can predct the sentment w.r.t. the target more accurately. Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

26 References: [1] P. Chen, Z. Sun, L. Bng, and W. Yang. Recurrent attenton network on memory for aspect sentment analyss. In Proceedngs of EMNLP, pages , [2] L. Dong, F. We, C. Tan, D. Tang, M. Zhou, and K. Xu. Adaptve recursve neural network for target-dependent twtter sentment classfcaton. In Proceedngs of ACL, pages 49 54, [3] K. He, X. Zhang, S. Ren, and J. Sun. Deep resdual learnng for mage recognton. In Proceedngs of CVPR, pages , [4] S. La, L. Xu, K. Lu, and J. Zhao. Recurrent convolutonal neural networks for text classfcaton. In Proceedngs of AAAI, volume 333, pages , [5] B. Lu. Sentment analyss and opnon mnng. Synthess Lectures on Human Language Technologes, 5(1):1 167, [6] J. Lu and Y. Zhang. Attenton modelng for targeted sentment. In Proceedngs of EACL, pages , [7] D. Ma, S. L, X. Zhang, and H. Wang. Interactve attenton networks for aspect-level sentment classfcaton. In Proceedngs of IJCAI, pages , Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

27 [8] R. K. Srvastava, K. Greff, and J. Schmdhuber. Hghway networks. arxv preprnt arxv: , [9] D. Tang, B. Qn, X. Feng, and T. Lu. Effectve lstms for target-dependent sentment classfcaton. In Proceedngs of COLING, pages , 2016a. [10] D. Tang, B. Qn, and T. Lu. Aspect level sentment classfcaton wth deep memory network. In Proceedngs of EMNLP, pages , 2016b. [11] Y. Tay, A. T. Luu, and S. C. Hu. Learnng to attend va word-aspect assocatve fuson for aspect-based sentment analyss. arxv preprnt arxv: , [12] Y. Wang, M. Huang, x. zhu, and L. Zhao. Attenton-based lstm for aspect-level sentment classfcaton. In Proceedngs of EMNLP, pages , [13] M. Yang, W. Tu, J. Wang, F. Xu, and X. Chen. Attenton based lstm for target dependent sentment classfcaton. In Proceedngs of AAAI, pages , Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25

28 [14] M. Zhang, Y. Zhang, and D.-T. Vo. Gated neural networks for targeted sentment analyss. In Proceedngs of AAAI, pages , Transformaton Networks for Target-Orented Sentment Classfcaton ACL / 25