Text Categorization Based on a Similarity Approach

Transcription

1 Text Categorzato Based o a Smlarty Approach Cha Yag Ju We School of Computer Scece & Egeerg, Uversty of Electroc Scece ad Techology of Cha, Chegdu 60054, P.R. Cha Abstract Text classfcato ca effcetly ehace the text processg capablty by automatcally sortg out them accordg to defed collecto of categores. Ths paper uses TFIDF method to represet documets, ad set the NGramSze value to be 6. Word Frequecy vector s used to measure ad dstgush dfferet features o documets. The Smlarty Approach uses Cose fucto to costruct the classfer. The expermet results dcate that proposed algorthm yelds good performace wth the accuracy up to 98%. Keywords: Text classfcato, TFIDF, Vector space model, Word frequecy, Smlarty. Itroducto The fast growth ad dyamc chage of ole formato have provded us a very large amout of formato ad lead to formato overload. Text (Documet) categorzato (TC) s a mportat tool for orgazg documets to categorzatos by applyg statstcal methods or artfcal tellgece techques. By ths way, the utlzato of the documets ca be expected to be more effectve. As a result, the stuato of formato overload may be allevated. The am of documet categorzato s to assg a umber of approprate categores to a textual documet based o the cotet. Ths categorzato process has may applcatos such as documet routg, dssemato, or flterg. A large umber of techques have bee developed for text classfcato, cludg Nave Bayes, Nearest Neghbor, eural etworks, regresso, rule ducto, ad Support Vector Maches []. Formally, TC cossts of determg whether a documet d (from a set of documets D) belogs or ot to a category C (from a set of categores C), cosstetly wth the kowledge of the correct categores for a set of trag documets. To perform ths task, two ma processes eed to be carred out. Frst, the documets must be trasformed to a form sutable for automatc processg. Ths chore cludes the removal of tags, the elmato of o-formatve words. The, each mportace stem the documet or relevat words are selected ad used to represet the documets. Ths reducto of the vocabulary cosdered to represet the documets s a crucal step the process, sce t has bee oted that t ca greatly mprove the overall performace. Secod, oce the represetato of the documets s fxed the assgmet of the documets to the categores s the ext step. Usually, a automatc classfer s duced from a set of correctly labeled examples usg statstcal methods or mache learg algorthms. The ths classfer s used to assg each ew documet to oe or more categores. The goal of a text classfcato system s to determe whether a gve documet belogs to ay of the predefed categores. Sce the documet ca belog to zero, oe, or more categores, the system ca be a collecto of bary classfers, whch oe classfer classfes for oe category. These are uversal bary classfers able to fd lear or o-lear threshold fuctos to separate the examples of a certa category from the rest, whch are based o the Structural Mmzato Rsk prcple from computatoal learg theory. Ths paper uses the popular feature selecto Word Frequecy ad the effectve classfer model for TC. The model evaluato usg classfcato accuracy method dcate that the Smlarty Approach take good effcecy o TC. The followg chapters below: Chapter troduces the TC s hstory from '80s to ow ad how ths paper to deal wth TC more effcetly. Chapter 3 gves the key techologes TFIDF, Word Frequecy ad Smlarty Approach the TC process. I Chapter 4 the expermets was performed, the the results was aalyzed. Chapter 5 offered the cocluso of ths paper ad the future work to mprove the TC method.. Related works I the '80s, order to create the automatc documet classfers ther maual costructo, kowledge egeerg (KE) techques are used. Example to buld maually, a expert system requred set of maually defed rules uder the followg type:

2 f (DNF' Boolea formula) the (category) else ot(category) It meas that the documet was classfed uder (category) f s satsfed (DNF Boolea formula). Ad the costrue system was bult by Carege Group for the Reuters ews agecy, s the typcal example for ths approach. Sce the early '90s, the more effectve ad powerful approach whch has bee bult ad replaced for the KE approach, was mache learg (ML).By extractg the characterstcs of a set of documets whch have bee pre-classfed maually uder c by a doma expert, a geeral ductve process (also called the learer) automatcally bulds a classfer for a category c. The advatages of ths approach are that costructo of a classfer based o a automatc bulder of classfer from a set of maually classfed documets (learg), ot of a classfer. Recet research o mache learg ad data mg has provded developed methods ad algorthms to costruct statstcal models of etwork data cludg socal etworks, web-page etworks, emal tracks, ctato etworks, ad so o. The models ca be costructed ether drectly from data usg formato extracto algorthms, whch are appled mostly o relatoal database, ad sem-structured text, or drectly from ustructured textual data usg text mg techques. The VSM (Vector Space Model) s a regular model to represet text documets. It s a method of bag of words, whch s used wdely IR, TC, TM ad WM. Ad the TFIDF algorthm s ofte combed wth VSM TC applcatos. Ths represetato gores the sequece whch the words occur ad s based o the statstc about sgle words solato. There are may text represetatos that am to stad for the features of documets dfferet domas, such as -grams represeted that employs word sequeces of legth-grams up to, documet cocept categores ad so o. Ths paper s to preprocess the documets by Feature Extracto ad Feature Selecto. The methods TFIDF ad Word Frequecy were used. Ad the 00 documets were used for the expermet. The process of TC cossts of trag ad testg. 3. Classfcato methods 3.. TFIDF We use feature-vector to represet documets, that s, take oe documet as a set of Term Sequeces, cludg term t ad term weght w. The the documet wll be made up of the pars of <t, w>. t, t, t 3 t represet the features whch express the documet cotet. We could treat them as a N-dmeso coordate. w, w, w 3 w represet the value relevat to coordate. So every documet (d) s mapped to the target space as a feature-vector V (d) (t, w, t, w, t 3, w 3... t, w ) [7]. The ma purpose of data preprocessg s to deal wth the data resource ad buld up the feature-vectors. We use weght as the crtero of feature selecto. The values of the vector elemets w for a documet d are calculated as a combato of the statstcs TF (t, d) ad DF (t). The term frequecy TF (t, d) s the umber of tmes word t occurs documet d. The documet frequecy DF (t) s the umber of documets whch the word t occurs at least oce. The verse documet frequecy IDF (t) ca be calculated from the documet frequecy. D IDF ( t) log ( ) DF t () D s the total umber of documets. The verse documet frequecy of a word s low f t occurs may documets ad s hghest f the word occurs oly oe. The value w of features t for documet d s the calculated as the product W TF t, d) ID t ) ( () w s called the weght of word t documet d. Ths word weghtg heurstcally says that a word t s a mportat dexg term for documet d f t occurs frequetly t (the term frequecy s hgh). O the other had, words whch occur may documets are rated less mportat dexg terms due to ther low verse documet frequecy. We could fd above that IDF servers as a adustg fucto to modulate the term frequecy. 3.. Feature selecto Feature selecto studes how to select a subset or lst of attrbutes or varables that are used to costruct models descrbg data. Its purposes clude reducg dmesoalty, removg rrelevat ad redudat features, reducg the amout of data eeded for learg, mprovg algorthms predctve accuracy, ad creasg the costructed models comprehesblty. Dmeso reducto techques ca geerally be classfed to Feature Extracto (FE) approaches ad Feature Selecto (FS). FS algorthms select a subset of the most represetatve features from the orgal feature space. FE algorthms trasform the orgal feature space to a smaller feature space to reduce the dmeso. Though the FE algorthms have bee proved to be effectve for dmeso of

3 data sets the text doma reducto, hgh dmeso ofte fals may FE algorthms due to ther hgh computatoal cost. Thus FS algorthms are more popular for practcal text data dmeso reducto problems. The performace of a feature subset should be evaluated based o a certa groud, whch s acheved by Evaluators. Evaluato fucto s to measure ad dstgush the classfcato capabltes of dfferet features o documets. I fact varous evaluato fuctos are already appled, such as: MI (Mutual Iformato), IG (Iformato Ga), ECE (Expected Cross Etropy), OR (Odds Rato), WET (the Weght of Evdece for Text), WF (Word Frequecy) []. The evaluato fucto of Word Frequecy s: Freg ( F) T W ) (3) I ths paper, we mplemeted Word Frequecy method whch s smple ad effcet for Chese documet classfcato Text categorzato va smlarty classfer There are two maor purposes of text smlarty measure: the frst oe s to fd out all of the smlar (or related) documets from a large documet collecto, such as IR (Iformato Retreval), TM (Text Mg), WM (Web Mg) ad TC (Text Classfcato/Clusterg); the other s to fd out the copes of a documet from the collecto, such as (Documet Copy Detecto) DCD. That causes dfferet requests for smlarty measure. The former wats to retur those related documets that are far away from other categores. The latter oe eeds to dstgush the almost same documets (copes) from other smlar documets []. Cose fucto, dot product ad proporto fucto are commoly used smlarty measures. Usually, we defe the smlarty value [0, ] so that cose ad proporto fuctos are wdely used. Let F ( ad F ( be documet A ad B word frequecy vectors, the the smlarty of A ad B cose fucto s Scos ( : S ( cos α F α F F ( α F ( (4) where α s the word weght vector, F (, F ( are the respectve umber of occurreces of the th word A ad B. Obvously Scos ( Scos (B, called symmetrc smlarty. The smlarty of A ad B proporto fucto s S% ( : I S% ( U, α ( F α F + ( ) α F ( where F ( meas that: F ( F + F ( 0,,,3,..., w w w w (5) (6) For smlarty, the copes (same documets) value s ad the more overlapped words betwee documets the hgher score. But they caot dstgush the subset copes from partly overlapped documets. As we kow that A s cluded B s dfferet from B s cluded.e. A B B A. So the measuremet of A B should be dfferet from that of B A. However the smlarty does ot satsfy that. I order to fd out subset documet copy, Shvakumar ad Garca-Mola (995) proposed RFM (Relatve Frequecy Model). The subset measure of documet A to be a subset of documet B to be: Subset( w c ( α F F ( α F (7) It s obvous that Subset ( Subset (B, ad Subset ( Ac) f Ac s a copy of A. Hece we call ths type as smlarty measure. The fal RFM smlarty measure betwee two documets A ad B s: S RFM ( max { Subset(, Subset( B, } (8) The Subset ( may be greater tha. I order to regularze the smlarty value [0, ], the fal RFM smlarty of documets A ad B s: S RFM ( m {,max{ Subset(,Subset(B, } (9) The RFM s derved from cose fucto. Smlar to that, we defe aother asymmetrc smlarty that derved from proporto fucto. We

4 call t IPM (Icluso Proporto Model). The cluso proporto of A B s: I Icl(, α ( F α F ( ) (0) Icl ( Icl (B, ad Zcl ( Ac) f Ac s a copy of A. The same as RFM, the fal IPM smlarty of documets A ad B s: S IPM ( m {,max{ Icl(, Icl( B, } () From expermets we beleve the RFM ad IPM are both excellet for subset copy detecto. 4. Expermets Fgure s a basc Text Categorzato model. TC s composed of two parts trag ad testg. I the trag process, the frst thg s to buld the feature of the trag text sets ad the get the feature sets. The feature extracto algorthm s TF-IDF ad the feature selecto algorthm s Word Frequecy. Durg the classfcato, we use the smlarty approach to costruct the classfer to get the category of the text sets for testg. Fg. : Text Categorzato Model. 4.. Data set For ths expermet we used 00 documets dowloaded from sa.com. 0 documets of them are for trag ad others for testg. These categores are: socety, ecoomy, musc, ad computer. 4.. Performace measuremets Classfcato performace s measured usg both recall ad precso. I ths case, recall s the proporto of the correct documets that are assged to a category by the algorthm. Precso s the proporto of documets assged to a category that belog to that category. Text categorzato s essetally a seres of dchotomous results ad so both mcro ad macro averagg ca be used to geerate a overall performace over the set of categores used. The classfcato model s evaluato fuctos are: Classfcato Accuracy: Accuracy(M ) ex P(C(ex) C(ex)) Accuracy(M, ex) ; 0; Precso: Recall: 4.3. Results P(ex)Accuracy(M, ex) C(ex) C(ex) () other Precso(M, targetc) P(targetC tar getc) Recall(M,targetC) P(tar getc targetc) (3) (4) The performace preseted ths subsecto s evaluated from the accuracy of a approach to predct the category of test documets. The predcto accuracy s defed usg the percetage of test documets that arc correctly categorzed. For a documet the data set that was assged to several categores, the predcto s cosdered to be correct f t was oe of the gve categores. I addto to the alteratves or test documet represetato at descrbed above, we also performed expermets where ormalzed term frequecy (TF) ad TF-IDF weghtg schemes were used to estmate the membershp degree of words occurrg documets. TF-IDF s a well-studed weghtg scheme from formato retreval that assgs the weght of a term proportoal to the occurrece Frequecy of the term each documet ad versely proportoal to the total umber of documets to whch the term occurs a gve documet collecto. However, the performace obtaed by employg these two weghtg schemes s eve worse. Fe-tug the membershp degrees

5 of words documets degrades the performace of the smlarty approach. 0 documets were traed to costruct the classfer. Ad the 80 documets got the best category of C were tested. Every testg documet had ts result wth the rak of every category by the classfcato model s evaluato fucto. The smaller the rak value s, the better the testg text gets the category. Ad sce we kow what the text came from, we ca evaluate how well the classfer s dog wth ths method. 5. Coclusos Ths paper uses a smlarty approach for text categorzato. Wth TFIDF ad Word Frequecy, the expermetal results dcate that proposed algorthm yelds good performace o TC. There are some works to do to optmze the text categorzato method the future. Frst, we wll try to fd the best feature extracto ad selecto method hadlg polysemy ad syoymy for the Chese text classfcato. Secod, a mproved smlarty approach s requred to take much effcecy of TC. Ackowledgemet The work s partally fuded by Youg Fud of Electroc Scece & Techology of Cha. Refereces [] J.W. Ha, Mchele Kamber, Data Mg Cocepts ad Techques, Secod Edto, Cha Mache Press, 007. [] Z.Z. Sh, Kowledge Dscovery, Tsghua Uversty Press, 005. [3] G. Forma, a Extesve Emprcal Study of Feature Selecto Metrcs for Text Classfcato [4] F. Sebasta, Mache learg automated text categorzato. ACM Computg Surveys, 34, 00. [5] V. Vapk, the Nature of Statstcal Learg Theory, New York: Sprger-Verlag, 995. [6] X.H. Wag, Retreval-based Chese Text Mg Techology Study ad Desg. Nov [7] X. Luo, D.L. Xa, P. Ya, Improved feature selecto method ad TF-IDF formula based o word frequecy dffereta. Computer Applcatos, 5(9), 005. [8] T. Joachms, a Proballstc Aalyss of the Toccho Algorthm wth TF-IDF for Text Categorzato. Prof. of the 4th Iteratoal Coferece o Mache Learg, ICML97, 997. [9] Y. Yag, Pederse Jo, a Comparatve Study o Feature Selecto Text Categorzato. [0] X.Y. Che, Y Che, L. Wag ad Y.F. Wag, Text Categorzato Based o Frequet Patters Wth Term Frequecy. Proceedgs of the Thrd Iteratoal Coferece o Mache Learg ad Cyberetcs, Shagha, 004. [] G. Yu, Y.J. Pe, Z.Y. Zhu ad H.Y. Che, Research of text smlarty based o word smlarty computg. Computer Egeerg ad Desg, 7 (), 006. [] Y.T. Zhag, L. Gog, Y.C. Wag, a Improved TF-IDF Approach for Text Classfcato [3] Y. Jag, Z.H. Zhou, a Text Classfcato Method Based o Term Frequecy Classfer Esemble. 006 [4] X.Y. Che, the Key Techques Research o Text Mg [5] JaBakus, Mohamed S. Kamel, Hgher order feature selecto for text classfcato, 006.