A Study of Semi-Discrete Matrix Decomposition for LSI in Automated Text Categorization

Transcription

1 A Study of Semi-Discete Matix Decomposition fo LSI in Automated ext Categoization Wang Qiang Wang XiaoLong Guan Yi School of Compute Science and echnology, Habin Institute of echnology, Habin {qwang, wangxl, guanyi Abstact. his pape poposes the use of Latent Semantic Indexing (LSI) techniques, decomposed with semi-discete matix decomposition (SDD) method, fo text categoization. he SDD algoithm is a ecent solution to LSI, which can achieve simila pefomance at a much lowe stoage cost. In this pape, LSI is used fo text categoization by constucting new featues of categoy as combinations o tansfomations of the oiginal featues. In the expeiments on data set of Chinese Libay Classification we compae accuacy to a classifie based on -Neaest Neighbo (-NN) and the esult shows that -NN based on LSI is sometimes significantly bette. Much futue wo emains, but the esults indicate that LSI is a pomising technique fo text categoization. 1 Intoduction ext categoization, the assignment of fee text documents to one o moe pedefined categoies based on thei content, is an impotant component in many infomation management tass [1]. Now a vaiety of text categoization fo supevised leaning algoithms is based on vecto space model (VSM). In this model documents ae epesented as a set of index tems that ae weighted accoding to thei impotance fo a paticula document and fo the geneal collection. But it can be misleading, because a document can be elevant fo a test document without having any tems in common with it. his pape exploes the use of Latent Semantic Indexing (LSI) fo text categoization as an impovement to VSM. he idea behind the LSI is to map each document and test vecto into a lowe dimensional space which is associated with concepts and compae the documents in this space [2]. We pefomed expeiments using -NN LSI, a new combination of the standad -NN method on top of LSI, and applying a new matix decomposition algoithm, Semi-Discete Matix Decomposition, to decompose the vecto matix. he Expeimental esults show that text categoization effectiveness in this space will be bette and it will also be computationally less costly, because it needs a lowe dimensional space. 1 his investigation was suppoted by the National Natual Science Foundation (Habin ) and he high echnology Reseach and Development Pogamme (Habin 2002AA )

2 his pape poceeds as follows. Section 2 pesents the geneal famewo fo a ecent LSI method, Semi-Discete Matix Decomposition (SDD). hen, the specific application of LSI to text classification is discussed in Section 3. Related wo is pesented in Section 4. Expeimental esult is shown in Section 5. Finally, Section 6 maes out plans fo futue wo. 2 Latent Semantic Indexing Cuent methods of VSM in indexing and etieving documents fom databases usually depend on a lexical match between quey tems and eywods extacted fom documents in a database. hese methods can poduce incomplete o ielevant esults due to the use of synonyms and polysemous wods. In fact the association of tems with documents o implicit semantic stuctue can be deived using lage spase tem by document matices. So both tems and documents can be matched using epesentations in -space deived fom of the lagest appoximate singula vectos of these tems by document matices. his completely automated appoach called Latent Semantic Indexing (LSI), which uses subspaces spanned by the appoximate singula vectos to encode impotant associative elationships between tems and documents in -space [3]. Using LSI, two o moe documents may be close to each othe in -space yet shae no common tems. 2.1 Singula Value Decomposition (SVD) fo LSI SVD is the most common method to LSI which decompose a tem-by-document ectangula matix X into the poduct of thee othe matices: A U V, Whee U (M R) and V (R N) have othonomal columns and Σ (R R) is the diagonal matix of singula values. R min(m,n) is the an of A. If the singula values of Σ ae odeed by size, the K lagest may be ept and the emaining smalle ones set to zeo. he poduct of the esulting matices is a matix A which is an appoximation to A with an K: A U V (1) K Whee Σ (K K) is obtained by deleting the zeo ows and columns of Σ, and U (M K) and V (N K) ae obtained by deleting the coesponding ows and columns of U and V [4]. A in one sense captues most of the undelying stuctue in A, yet at the same time emoves the noise o vaiability in wod usage. Since the numbe of dimensions K is much smalle than the numbe of unique wods M, mino diffeences in teminology will be ignoed.

3 2.2 Semi-Discete Matix Decomposition (SDD) fo LSI A semi-discete decomposition (SDD) appoximates a matix as a weighted sum of oute poducts fomed by vectos with enties constained to be in the set S {-1, 0, 1}. O'Leay and Peleg intoduced the SDD in the context of image compession, and Kolda and O'Leay (1998, 1999) used the SDD fo latent semantic indexing (LSI) in infomation etieval. he pimay advantage of the SDD ove othe types of matix appoximations such as the tuncated singula value decomposition (SVD) is that, it typically povides a moe accuate appoximation fo fa less stoage. An SDD of an m n matix A is a decomposition of the fom: e1 0 L 0 v 1 e v A u u u L 4243 L [ 1 2 ] M M O M M u 0 0 L e v { Hee each u i is an m-vecto with enties fom the set S {-1, 0, 1}, each v i is an n- vecto with enties fom the set S, and each e i is a positive scala. We call this a - tem SDD. Although evey matix can be expessed as a mn-tem SDD: v (2) A m n i 1 1 a i e e i (3) Whee e is the -th unit vecto, the usefulness of the SDD is in developing appoximations that have fa fewe tems. An SDD appoximation can be fomed iteatively via a geedy algoithm. Let A denote the -tem appoximation (A 0 0). Let R be the esidual at the th step, that is R A - A -1. hen the optimal choice of the next tiplet (d, x, y ) is the solution to the sub poblem: min F ( d, x, y ) m 2 R dxy s.t. x S, y S, d > 0 (4) F his is a mixed intege pogamming poblem. Assuming that a fixed numbe of inne iteations ae set pe step, the complexity of the algoithm is O ( 2 ( m + n) + mlog m + nlog n). At the meanwhile, since the stoage equiement fo a -tem SDD is floating point numbes plus (m+n) enties fom S, it is also inexpensive to stoe quite a lage numbe of tems [5]. In evaluating queies, a document vecto can be teated as ~ 1 α A (5) V his is a -dimension vecto. And the test vecto is poected into the same - dimensional space by: q~ α U q (6) n

4 he similaity between a document and test vecto can be calculated by ~ ~ ~ q A ~ S cos( q, A ) ~ 2 ~ 2 ( q ) ( A ) i 1 (7) In this study, the value of the splitting paamete α in equation has left at the default 0. As in the Vecto Model, documents can now be aned accoding to thei similaity to the test document, and the categoy of the quey is the categoy of the most simila document. 3 LSI fo ext Categoization he -NN method is a vey simple appoach that has peviously shown vey good pefomance on text categoization tass [6][7]. Hence, we decided to use this method fo classification. In this pape, we apply LSI model to the -NN algoithm to veify the impovement on text categoization. o classify an unnown document vecto d, the -neaest neighbo (-NN) algoithm ans the document s neighbos among the taining document vectos, and uses the class labels of the most simila neighbos to pedict the class of the input document. he classes of these neighbos ae weighted by the similaity of each neighbo to d, whee similaity may be measued by such as the Euclidean distance o the cosine between the two document vectos. he -NN LSI algoithm has fou steps: 1. Index the taining set 2. Use SDD to map each document vecto into a lowe dimensional space which is associated with concepts and pocess the documents in this space. 3. Fo each document x to be classified, etieve its most simila documents v fom the taining set (whee is a paamete of the algoithm). Call this set R (X ). 4. Fo each categoy C, compute its elevance to x as: S ( c, x ) sim ( d, x ) (8) d R ( x, c ) Whee R ( x, C) is the subset of documents in R (x) that ae elevant to C. hee ae many ways to tansfom the scoes S( c, x) fo a paticula categoydocument pai into a YES/NO decision on whethe to assign that document to that categoy. In this pape, we use the methods called SCut. SCut assigns to each categoy a theshold t(c) and assigns a document x to categoy C if S( c, x) t(c). he choice method of t(c) is explained in section 5.1.

5 4 Related Wo Anothe study has been pefomed using LSI fo text classification. It is made by Ana Cadoso-Cachopo and Alindo Limede Oliveia(2000). In a compaison between - NN LSI and vecto model, and using the Mean Recipocal Ran (MRR) as a measue of oveall pefomance, this study poved that -NN LSI pefomed almost as well as the best pefoming methods, such as Suppot Vecto Machine (SVM), fo text categoization. But his study is confined to English copus and SVD technique is adopted. 5 Expeiments his section povides some empiical evidence to show that LSI is a competitive solution to text classification. he esults ae examined on the data sets of Chinese Libay Classification. 5.1 Data Sets and Potocol he expeiment opeates Chinese text as pocessing obect and uses the Chinese Libay Classification 4 (Simplified Vesion) as citeia (able 1), which is a compehensive one in common use in China's most libaies, infomation institutes and centes. All the Data Sets ae gatheed fom digital libay and Intenet. he web pages, all togethe containing 10,857 pages, ae divided into thity-seven categoies. A Maxism, Leninism, Maoism & Deng Xiao ping's heoy B Philosophy and Religion C Social Science D Politics and law E Militay Science F Economics G Cultue, Science, Education and Athletics H Linguistics able 1. Chinese Libay Classification K Histoy and Geogaphy N Natual Science O Mathematics, Physics and Chemisty P Astonomy and Geosciences Q Bioscience R Medicine and Hygiene S Agicultual Science B Industial echnology F Metallugy Industy G Metal and Metalwoing ech- H Machine and Mete J Weapony K Kinetics Industy L Atomic Enegy M Electo technician N Radio electonics ele-technology S Light industy and Handicaft U Achitectue Science V Wate Consevancy U anspotation V Aviation X Envionmental Science Z Compehensive Boos

6 I Liteatue J At D Mining Engineeing E Petoleum and Natual gas Industy P Automation technology Compute Science Q Chemisty Industy he copus is boen into wods with Wod-Lattice algoithm and afte emoving toens that occu only once o ae on a stoplist, a vocabulay of size 57,040 is left. We set 9,115 pages as the tain set and othes 1,742 as test set. In the stage of taining, the Expected Coss Entopy (ECE) is used on tain set fo featue selection, which is defined as p ( c i ) ECE ( ) p ( c i ) log (9) i p ( c i ) Whee p( c i ) is the pobability of tem and categoy c i co-occuence and p ( c i ) is the pobability of categoy c i. In the pocess of tuning documents into vectos, the tem weights ae computed using a vaiation of the Oapi tem-weighting fomula [8]: tf( t, d) N n( t) wt (, d) log( ) lend ( ) n( t) (10) tf( t, d) avg_ len Whee w( t, d ) is the weight of tem t in document d ; tf ( t, d ) is the withindocument fequency of tem t; N is the numbe of documents in the taining set; n(t) is the numbe of taining documents in which t occus; len(d ) is the numbe of toens in document d afte stop-wod emoval; avg_len is the aveage numbe of toens pe document in the taining set. he values of N, n(t), and avg_len wee computed fom the entie taining set. Befoe the -NN algoithm can be used, the value of must be set. We used standad m-way coss-validation to set this value; the taining data was split into m patitions, with documents assigned andomly to each patition. Fo each coss-validation un, m t of these patitions fomed the taining subset and m va ( m - m t ) patitions the validation subset. Patitions wee otated between the taining and validation subsets so that each patition was used m t times fo taining and m va times fo validation. Pefomance was aveaged ove all m uns to poduce a final value used fo compaison between diffeent values of. Setting the values of t(c) fo the SCut method is though a Modified Leave-oneout Coss-validation Algoithm. Fo each document d in the taining set, use evey othe document in the taining set to assign scoes S( c, d ) via the -NN algoithm. hen set the values of t(c) to be those which poduce optimal pefomance ove this

7 set of scoes. his method has the advantage of deiving the values of t(c) fom a data set that is as close as possible to the actual taining data. At last, we use the standad cosine-similaity metic to compute similaity between the taining and test documents. e.g. sim( d, x) cos( d, x d x ) (11) d x 5.2 Evaluation In expeiment, each document belongs to no moe than two categoies. But the evaluation only consults the fist categoy with highest scoe. Fo text categoization evaluation, the effectiveness measues of pecision, ecall and F1 ae defined espectively. Fo categoy : pecision l m 100 % ecall l n 100 % (12) F 1 ecall ecall pecision + pecision 2 Whee l is the numbe of test set categoy membes assigned to categoy and m is the total numbe of test set membes assigned to categoy.whee n is the numbe of categoy membes in test set. hus fo all categoies, Maco-ecall, Maco-pecision and Maco aveaged F1 scoe ae espectively defined. In ou expeiment, diffeent featue-set sizes wee tested, and the size optimized by the global F1 scoe fo classifie. Finally, 5362 featues wee selected fo -NN. And the in -NN was set to 50. hough the tansfomations of SDD, the 5362 featue tems become a lowe dimensional space of an- appoximation. Affimatively applying SDD in tain phase upgade a little taining cost, but it is negligible to pefomance-pomoting in test phase. he expeiment shows that -NN (SDD) pomotes executive efficiency geatly with stoage cost educed fom 96.6M to 5.62M and executive time condensed fom 3437s to 435s. he choice of -value is an empiical method. he tem-document matix was fed into SDD tansfomation with a vaiety of -values. he esults of these levels ae displayed in Figue 1. hey showed the best aveage F1 with 140.

8 maco-aveage F value Fig. 1. Result of cuent LSI (SDD) system with vaious -values able 2 shows the pecision, ecall and F1 scoe fo each categoy on the full taining set. Note that in all categoies, -NN LSI method made evaluation measue bette in a diffeent degee. otally using -NN LSI against -NN VSM, 143 eo documents ae eliminated and F 1 scoe is 9.48 pecent highe than befoe. able 2. Compaison of the esults on P, R and F 1 between -NN LSI and -NN VSM -NN LSI K-NN VSM n m l Pecsion Recall F 1 m l Pecsion Recall F 1 A % 83.67% % 75.51% B % 92.68% % 92.68% C % 29.17% % 4.17% D % 52.00% % 38.00% E % 68.00% % 58.00% F % 96.00% % 96.00% G % 95.74% % 90.43% H % 92.00% % 84.00% I % 98.00% % 96.00% J % 86.00% % 88.00% K % 32.00% % 32.00% N % 75.00% % 25.00% O % 79.17% % 54.17% P % 96.00% % 94.00% Q % 84.00% % 60.00% R % 96.00% % 76.00% S % 80.85% % 72.34% % 28.00% % 4.00% % 83.33% % 79.17% % 95.56% % 93.33% % 73.47% % 61.22% % 50.00% % 58.33% % 52.08% % 33.33%

9 J % 88.00% % 90.00% % 78.72% % 68.09% % 89.36% % 87.23% % 75.51% % 67.35% % 76.09% % 82.61% % 94.00% % 96.00% % 77.08% % 83.33% % 58.33% % 16.67% % 74.00% % 66.00% % 88.00% % 88.00% U % 85.71% % 83.67% V % 50.00% % 32.00% X % 32.00% % 30.00% Z % % 22.22% 0.33 M 77.32% 75.03% % 64.29% Figue 2 compae the pefomance cuves of the impoved and the oiginal classifies on F 1 scoe with espect to the 37 categoies. hese cuves ae obtained by egading categoies as the hoizontal axis and plotting the pe categoy F 1 scoes fo each classifie % 80.0% 60.0% 40.0% 20.0% 0.0% K-NN LSI K-NN VSM A B C D E F G H I J K N O P Q R S B D E F G H J K L M N P Q S U V U V X Z Fig. 2. he maco-aveaged F 1 cuves fo each categoy using -NN VSM vesus -NN LSI hough expeiments we find that the efficiency of automated text categoization is dependent on tain set quality seiously, theefoe the paamete value pesented in this pape should be adusted to copus in othes application. 6 Conclusion In this pape we pesented a study with significance analyses on text categoization methods based on LSI (SDD). Ou main conclusions ae: LSI (SDD) can achieve simila o highe pefomance at a much lowe stoage cost and little executive time in automated text categoization. As the volume of infomation available on the Intenet continues to incease, online text categoization is equied seiously. So LSI (SDD) is a pomising technique fo text categoization. LSI (SDD) is a technique that waants futhe investigation fo text classification.

10 7 Futue Wo Much wo emains. Fistly, we can impove the method to compute the distance between two document vectos[9].with cosine computing documents similaity, the technique neglects the diffeent among featues and all index tems ae of equal impotance in the pocess of computing, which lead to the inexact esults and dop down the pecision of text categoization. Secondly, lexical analysis can be futhe studied by teating some phases as tems. hidly, the easons fo the poo pefomance of a numbe of categoization should be investigated. Acnowledgements We would lie to than Sun ChenJie and Mao YongQuan fo gatheing web pages and system obust tests, which made the significant test caied out smoothly. Refeences 1. Kesti Aas and Line Eivil. June (1999). ext Categoisation: A Suvey. Nowegian Computing Cente. 2. Ana Cadoso-Cachopo and Alindo Limede Oliveia, (2000). An Empiical Compaison of ext Categoization Methods, Instituto Supeio _ecnico Depatamento de Engenhaia Infom_atica Av. Rovisco Pais. 3. Michael W. Bey and Ricado D.Fieo. (1996). Low_Ran Othogonal Decompositions fo Infomation Retieval Applications. Numeical Linea Algeba with Applications, Vol1(1), Scott C. Deeweste, Susan. Dumais, homas K. Landaue, Geoge W. Funas, and Richad A. Hashman. (1990). Indexing by latent semantic analysis. Jounal of the Ameican Society fo Infomation Science. 5. Kolda,. G. and O'Leay, D. P. (2000). Algoithm 805: Computation and uses of the semidiscete matix decomposition, ACM ansactions on Mathematical Softwae 26(3): 415{ Yiming Yang and Xin Liu. August (1999). A e-examination of text categoization methods. Poceedings of the 22nd Annual Intenational ACM SIGIR Confeence on Reseach and Development in Infomation Retieval, pages 42-49, Beeley, CA, USA. 7. S. Dumais, J. Platt, D. Heceman, and M. Sahami. (1998). Inductive Leaning Algoithms and Repesentations fo ext Categoization, echnical Repot, Micosoft Reseach. 8. om Ault and Yiming Yang. (1999) NN at REC-9. Language echnologies Institute and Compute Science Depatment Newell Simon Hall 3612C, Canegie Mellon Univesity Pittsbugh, PA 15213?213, USA 9. Caon, J. (2000). Expeiments with lsa scoing: Optimal an and basis, echnical epot,siam Computatio-nal IR Woshop.URL: citesee.n.nec.com