A Comprehensive Method for Text Summarization Based on Latent Semantic Analysis

Transcription

1 A Comprehesive Method for Text Summarizatio Based o Latet Sematic Aalysis Yigjie Wag ad Ju Ma School of Computer Sciece ad Techology, Shadog Uiversity, Jia, Chia yigj_wag@hotmail.com, maju@sdu.edu.c Abstract. Text summarizatio aims at gettig the most importat cotet i a codesed form from a give documet while retais the sematic iformatio of the text to a large extet. It is cosidered to be a effective way of tacklig iformatio overload. There exist lots of text summarizatio approaches which are based o Latet Sematic Aalysis (LSA). However, oe of the previous methods cosider the term descriptio of the topic. I this paper, we propose a comprehesive LSA-based text summarizatio algorithm that combies term descriptio with setece descriptio for each topic. We also put forward a ew way to create the term by setece matrix. The effectiveess of our method is proved by experimetal results. O the summarizatio performace, our approach obtais higher ROUGE scores tha several well kow methods. Keywords: Text Summarizatio, Latet Sematic Aalysis, Sigular Value Decompositio. 1 Itroductio The widespread use of the Iteret has dramatically icreased the amout of accessible iformatio ad it becomes difficult for users to sift through the multitude of sources to fid out the right documet. With the help of search egie, the majority of irrelevat documets are filtered out, however, users still hesitate to determie which particular search result should avigate to. Automated summarizig system ca be used as a istrumet for decidig whether a documet is related to their eeds. The summary of a documet is defied as: a text that is produced from the documet that coveys importat iformatio i the origial text, ad that is o loger tha half of the origial text ad usually sigificatly less tha that [1]. Automatically geeratig the summary of a documet has log bee studied sice 1950s ad it is still a research hotpot util ow [2, 3, 4]. Oe of the most famous approaches is usig LSA [5, 6, 7, 8, 9, 10] to get the ideal summary. The foudatioal work that uses LSA for text summarizatio selects oe setece for each topic accordig to topic importace [6]. The work i [7] starts with calculatio of the legth of each setece vector ad the chooses the logest seteces as the summary. I the work [9], the legth strategy proposed i [7] is improved ad a cross method is proposed. I [8], for each topic, the umber of seteces to be collected is determied by gettig the percetage of the related sigular values over the sum of all sigular values. G. Zhou et al. (Eds.): NLPCC 2013, CCIS 400, pp , Spriger-Verlag Berli Heidelberg 2013

2 A Comprehesive Method for Text Summarizatio Based o LSA 395 However, there are some disadvatages of the previous algorithms. The mai drawback is that seteces that are closely related to the chose topic somehow but do ot have the highest idex value will ot be selected. Also, all chose topics are composed of oly oe setece [6], whereas the sigle setece fails to fully express the topic. The legth strategy [7, 9] requires a method of decidig how may LSA dimesios to iclude i the latet space. For the work i [8], if there is a wide gap betwee the curret sigular value ad the ext oe, the there is little chace to iclude the topics whose correspodig sigular values are less tha the curret oe. I our work, we propose a comprehesive method that combies term descriptio with setece descriptio for each topic. We edeavor to select a set of seteces that ot oly have the best represetatio of the topic but also iclude the terms that ca best represet this topic. Also, i order to utilize the mutual reiforcemet betwee eighbor seteces, we put forward a ew way to create the term by setece matrix. This paper is orgaized as follows: i Sectio 2, we itroduce LSA briefly. Sectio 3 progresses to preset our method i detail. I Sectio 4, the effectiveess of our method is cofirmed by experimetal results. Fially, we coclude this paper i Sectio 5. 2 Latet Sematic Aalysis LSA uses Sigular Value Decompositio (SVD) to fid out the sematic meaig of seteces. The SVD of a matrix A with the dimesio of m (m>) ca be defied T as: A = UΣV, where U = [ u1, u 2,, u ] is a m colum-orthogoal matrix whose left sigular vector u i is a m-dimesioal colum vector, V = [ v1, v 2,, v ] is a colum-orthogoal matrix whose right sigular vector v j is a -dimesioal colum vector. Σ = diag ( σ 1, σ 2,, σ ) is a diagoal matrix whose diagoal elemets are o-egative sigular values sorted i descedig order. From sematic perspective, we assume that SVD geerates the cocept dimesio [11]. Each triplet (left sigular vector ad right sigular vector) ca be viewed as represetig such a cocept, the magitude of its sigular value represets the degree of importace of this cocept. 3 Text Summarizatio Based o Latet Sematic Aalysis 3.1 Documet Aalysis This step cotais two tasks: Documet Represetatio ad Sigular Value Decompositio. First, each documet eeds to be represeted by a matrix. The matrix is costructed by terms (words with stop words elimiated) that occurred i the documet represetig rows ad seteces of the documet represetig colums, thus it is called term by setece matrix. For a text with m terms ad seteces where without loss of geerality m>, it ca be represeted by = [ ]. The cell a ca be A a m

3 396 Y. Wag ad J. Ma filled out with differet approaches. We will elaborate o the weightig schemes i sectio 4. Oce the term by setece matrix is costructed, SVD will be employed to break it ito three parts: U, ad V T. Based o the discussio i sectio 2, we take U as term by cocept matrix, V T as cocept by setece matrix while the magitude of sigular values i suggests the degree of importace of the cocepts. 3.2 Setece Selectio As with [6], a cocept ca be represeted by the setece that has the largest idex value i the correspodig right sigular vector, we make aother hypothesis: a cocept ca also be represeted by a few of terms, ad these terms should have the largest idex values i the correspodig left sigular vector. The two forms of descriptio of a cocept are called setece descriptio ad term descriptio. Here each cocept is treated as a idepedet topic. Sice seteces are composed of terms, it is hoped that the most represetative seteces of the curret cocept should iclude the terms that best represet this cocept. Therefore, each topic i the summary ca be recostructed by selectig seteces accordig to the magitude of the idex values i the right sigular vector util a few of most represetative terms that have the largest idex values i the left sigular vector are fully icluded. The process of selectig summary seteces ca be illustrated as follows. Formulatio. For a documet D with m terms ad seteces, suppose term i (1 i m) deotes the i-th term, ad set j (1 j ) deotes the j-th setece, the D={set 1, set 2,, set }. M is the maximum umber of seteces to be selected, k is the umber of cocepts that ca be selected ad N k is the umber of seteces for the k-th cocept, k ad N k are iitialized to 1 ad 0 respectively. Let set S cotai the summary seteces ad iitialize S to ull. Setece Selectio ad Term Selectio. While S <M, for the k-th cocept, select the setece that has the largest idex value from the k-th right sigular vector v k. Get l that l satisfies v kl =Argmax(v ki ), iclude the l-th setece set l ito S ad delete the l-th elemet v il for v i (1 i ), update V T ad icrease N k. The select three terms u kp, u kq, u ks that are represeted by the Top3 largest idex values from the k- th left sigular vector u k, ad let set T={term p, term q, term s }. Combiatio. Delete terms that appear both i T ad set l from T. While T is ot ull, if N k <3 ad S <M, cotiue to select seteces for this cocept, update V T ad T, icrease N k, else set T to ull. The icrease k ad begi to select seteces for the ext cocept. Based o the above discussio, we give the formal descriptio of our Setece Selectio method i Algorithm 1.

4 A Comprehesive Method for Text Summarizatio Based o LSA 397 Algorithm 1. Setece Selectio based o LSA Iput: Documet D, Matrix U, Matrix V T, M Output: Set S 1 Iitialize S=ϕ, k=1 2 while S <M 3 get l i v k, S=S { set l }, update V T, N k =1 4 get p, q, s i u k, T={ term p, term q, term s } 5 T 0 =T set l, T=T-T 0 6 while (T ϕ) 7 if (N k <3 ad S <M) 8 get l i v k, S=S { set l }, update V T, N k = N k +1 9 T 0 =T set l, T=T-T 0 10 else T=ϕ 11 ed while 12 k=k+1 13 ed while 14 Retur S 4 Experimets ad Evaluatio 4.1 Weightig Schemes I order to elaborate o the weightig schemeswe defie a = L t ) * G ( t ) + N ( t ), (1) ( where L(t ) is the Local Weight for term i i set j, G(t ) is the Global Weight for term i i the whole documet, N(t ) is the Neighbor Weight of term i i set j. I the followig, we use tf deotes the umber of times that term i occurs i set j, tf max deotes the frequecy of the most frequetly occurrig term i set j, is the total umber of seteces, i is the umber of seteces that cotai term i, gf i is the umber of times that term i occurs i the whole documet.. For Local Weight, we choose to use the followig four alterative strategies: Biary Represetatio (BR): If term i appears i set j, L ( ) = 1, otherwise 0. Term Frequecy (TF): L ( t ) = tf. Augmet weight (AW): L t ) = * ( tf / tf ). ( + max L ( t ) = log( 1 + tf. Logarithm Weight (LW): ) For Global Weight, possible weightig schemes ca be: No Global Weight (NG): G ( ) = 1. Iverse Setece Frequecy (ISF): G t ) = 1 + log( / ). t t ( i

5 398 Y. Wag ad J. Ma p log p Etropy Frequecy (EF): G ( t ) = 1 +, where log j tf p =. gf i I order to make use of terms that occur i the eighbor seteces, we put forward the cocept of Neighbor Weight ad defie Neighbor Weight as N ( t ) = λ [ L( ti, j 1 ) * G ( t i, j 1 ) + L( t i, j+ 1 ) * G ( ti, j+ 1 )], where λ is a parameter which we will explore i the followig experimets. So i the weightig schemes, we may add Neighbor Weight () or just let Neighbor weight equals to 0 (). Neighbor Weight is added maily by the followig three otable cosideratios: (1) Neighbor seteces ca be affected by each other thus form clusters to make the topics more covice. (2) It helps to resolve aaphora resolutio, sice most of the time a proou ad what it demostrates appear i the adjacet seteces. (3) With eighbor weight added, it helps to resolve the issue of data sparsity. 4.2 Datasets ad Evaluatio Methods The datasets that are used for the evaluatio of our LSA-based summarizatio approach are DUC2002 dataset ad DUC2004 dataset 1. DUC2002 dataset cotais 567 documets, each documet is provided with two 100-word huma summaries. The dataset of DUC2004 icludes 5 tasks, while i our work, we oly use task 2. I this task, documets are clustered ito 50 topics of 10 documets each. Two kids of metrics that F score ad ROUGE toolkit [12] are adopted. S cad S ref S cad S ref (1 + β 2 ) PR P =, R =, F =, (2) 2 S S β P + R ROUGE cad N = S S ref S S ref ref gram S gram S Cout match ( gram Cout ( gram ) ), (3) where S cad deotes the cadidate summary ad S ref deotes the referece summary, stads for the legth of the -gram, Cout(gram ) is the umber of -grams i the referece summaries, Cout match (gram ) is the maximum umber of -grams cooccurrig i a cadidate summary ad the referece summaries. I our experimets, Logest Commo Subsequece ROUGE-L together with ROUGE-SU4 [12] are also beig used. 4.3 Experimetal Results ad Aalysis First, i order to compare the differet weightig schemes we coduct experimets o DUC2002 dataset. We set λ i the Neighbor Weight to iitially. 1

6 A Comprehesive Method for Text Summarizatio Based o LSA BR * NG BR*ISF BR*EF TF*NG TF*ISF TF*EF AW*NG AW*ISF AW*EF LW*NG LW*ISF LW*EF BR * NG BR*ISF BR*EF TF*NG TF*ISF TF*EF AW*NG AW*ISF AW*EF LW*NG LW*ISF LW*EF (a) (b) BR*NG BR*ISF BR*EF TF*NG TF*ISF TF*EF AW*NG AW*ISF AW*EF LW*NG LW*ISF LW*EF BR*NG BR*ISF BR*EF TF*NG TF*ISF TF*EF AW*NG AW*ISF AW*EF LW*NG LW*ISF LW*EF (c) Fig. 1. Compariso of differet weightig schemes (Local Weight*Global Weight +Neighbor Weight) for (a) F-1 score, (b) ROUGE-1, (c) ROUGE-2 ad (d) ROUGE-L From Figure 1 we ca tell: the best combiatio of Local Weight ad Global Weight is BR*EF, it performs better tha other combiatios at large. With Neighbor Weight added, early the results of all combiatios acquire a improvemet. (d) (a) (b) (c) Fig. 2. Relatioship betwee λ ad (a) F-1 score, (b) ROUGE-1, (c) ROUGE-2 ad (d) ROUGE-L (d) We apply the weightig scheme of BR*EF+ i our experimets to show the impact that λ makes o the performace ad get Figure 2.

7 400 Y. Wag ad J. Ma I Figure 2, x-axis deotes the rage of λ from 0 to 1, y-axis deotes the correspodig metrics value. From this figure, we ca tell: with the Neighbor Weight added, the correspodig metric value icreases firstly ad the decreases with the raise of λ. Geerally it is beeficial for λ i a small iterval betwee 0 ad. I order to get the most satisfyig performace, we assig 5 to λ to make a compromise. I the followig, we take the weightig scheme of BR*EF+, ad set the parameter λ i the Neighbor Weight to 5 to coduct experimets o DUC2002 dataset ad DUC2004 dataset. Four LSA-based methods: GLLSA [6], SJLSA [7], MRCLSA [8] ad OCALSA [9] together with three other latest models: DSDR-o [13], SATS [14] ad MCMR [15] are adopted for compariso with our method. The ROUGE metrics of ROUGE-1(R-1), ROUGE-2 (R-2), ROUGE-SU4 (SU4) ad ROUGE-L (R-L) are used for evaluatio. Table 1 shows differet ROUGE scores o DUC2002 dataset ad DUC2004 dataset. It ca be observed that our LSA-based method achieves higher ROUGE scores ad outperforms the other oes. As see from this table, o DUC2002 dataset, ROUGE-1 score of our method is close to DUC-best, the scores of other three metrics are competitive with the DUC-best. O DUC2004 dataset, the scores of ROUGE-1, ROUGE-2 ad ROUGE-SU4 of our method are higher tha the DUC best. More importatly, our approach, early i terms of all ROUGE scores, outperforms the other methods that are based o LSA ad is better tha the three other latest modes. Table 1. ROUGE results o datasets of DUC2002 ad DUC2004 Algorithm DUC2002 dataset DUC2004 dataset R-1 R-2 SU4 R-L R-1 R-2 SU4 R-L Baselie DUC-best GLLSA SJLSA MRCLSA OCALSA DSDR-o SATS MCMR Ours Coclusio I this paper, we propose a improved LSA-based summarizatio algorithm that combies term descriptio with setece descriptio for each topic. We select three seteces at most for each topic ad the seteces selected ot oly have the best represetatio of the topic but also iclude the terms that ca best represet this topic. We also put forward the cocept of Neighbor Weight ad propose a ovel way that tries to utilize the mutual reiforcemet betwee eighbor seteces to create the term by setece matrix. Experimetal results prove that our method achieve higher ROUGE scores tha several well kow methods.

8 A Comprehesive Method for Text Summarizatio Based o LSA 401 Refereces 1. Radev, D.R., Hovy, E., McKeow, K.: Itroductio to the special issue o summarizatio. Computatioal Liguistics-Summarizatio 28(4), (2002) 2. Vodolazova, T.: The role of statistical ad sematic features i sigle-documet extractive summarizatio. Artificial Itelligece Research 2(3), (2013) 3. Gupta, V., Lehal, G.S.: A survey of Text Summarizatio Extractive Techiques. Joural of Emergig Techologies i Web Itelligece 2(3) (2010) 4. Das, D., Martis, A.: A Survey o Automatic Text Summarizatio. I: Literature Survey for the Laguage ad Statistics II Course at CMU (2007) 5. Deerwester, S.C., Dumais, S.T., Ladauer, T.K., Furas, G.W., Harshma, R.A.: Idexig by latet sematic aalysis. Joural of the America Society for Iformatio Sciece ad Techology, (1990) 6. Gog, Y.H., Liu, X.: Geeric text summarizatio usig relevace measure ad latet sematic aalysis. I: Proceedigs of the 24th Aual Iteratioal ACM SIGIR Coferece o Research ad Developmet i Iformatio Retrieval, pp ACM, New York (2002) 7. Steiberger, J., Ježek, K.: Text Summarizatio ad Sigular Value Decompositio. I: Yakho, T. (ed.) ADVIS LNCS, vol. 3261, pp Spriger, Heidelberg (2004) 8. Murray, S.: Reals, ad J. Carletta. Extractive Summarizatio of Meetig Recordigs. I: Proceedigs of the 9th Europea Coferece o Speech Commuicatio ad Techology, pp (2005) 9. Ozsoy, M.G., Clicekli, I., Alpasla, F.N.: Text summarizatio of Turkish Texts usig Latet sematic aalysis. I: Proceedigs of the 23rd Iteratioal Coferece o Computatioal Liguistics (Colig 2010), Beig, pp (2010) 10. Ai, D., Zheg, Y., Zhag, D.: Automatic text summarizatio based o latet sematic idexig. Artif. Life Robotics 15, (2010) 11. Berry, M.W., Dumais, S.T., O Brie, G.W.: Usig liear algebra for itelliget iformatio retrieval. SIAM Review 37(4), (1995) 12. Li, C.Y.: Rouge: a package for automatic evaluatio of summaries. I: Proceedigs of the ACL Text Summarizatio Workshop, pp (2004) 13. He, Z., Che, C., Bu, J., Wag, C., Zhag, L.: Documet Summarizatio Based o Data Recostructio. I: Proceedig of the Twety-Sixth AAAI Coferece o Artificial Itelligece, pp (2012) 14. Chadra, M., Gupta, V., Paul, S.K.: A statistical approach for Automatic Text Summarizatio by Extractio. I: 2011 Iteratioal Coferece o Commuicatio Systems ad Network Techologies, pp (2011) 15. Alguliev, R.M., Aliguliyev, R.M., Hajirahimova, M.S., Mehdiyev, C.A.: MCMR: Maximum coverage ad miimum redudat text summarizatio model. Expert Systems with Applicatios 38(12), (2011)