An Improvement to Naive Bayes for Text Classification

Transcription

1 Avalable onlne at Proceda Engneerng 15 (2011) Advancen Control Engneerngand Informaton Scence An Improvement to Nave Bayes for Text Classfcaton We Zhang a, Feng Gao a, a* a MOE KLINNS Lab, X an Jaotong Unversty, X an, Shaanx Provnce, ,Chna Abstract Naïve Bayes classfers whch are wdely used for text classfcaton n machne learnng are based on the condtonal probablty of features belongng to a class, whch the features are selected by feature selecton methods. In ths paper, an auxlary feature methos proposed. It determnes features by an exstng feature selecton method, and selects an auxlary feature whch can reclassfy the text space amed at the chosen features. Then the correspondng condtonal probablty s adusten order to mprove classfcaton accuracy. Illustratve examples show that the proposed meth-ondeemproves the performance of naïve Bayes classfer Publshed by Elsever Ltd. Open access under CC BY-NC-ND lcense. Selecton and/or peer-revew under responsblty of [CEIS 2011] Keywords:Text classfcaton; Feature selecton; Machne learnng; Naïve Bayes. 1. Introducton Nave Bayes s based on Bayes theorem and an attrbute ndependence assumpton [1],[2]. Its compettve performance n classfcaton s surprsng, because the condtonal ndependence assumpton on whch t s based, s rarely true n real world applcatons. Naïve Bayes have been studed extensvely by some researchers n text classfcaton task [3],[4]. The exstng lteratures about text classfcaton wth naïve Bayes have focused on three aspects. One s to construct anmprove naïve Bayes model[5],[6],[7], another s to dscuss the naïve hypothess enable or not, and analyze the effect on classfy performance, then present the correspondng mprovement[8]. The other s to mprove feature selecton because naïve Bayes s hghly senstve to feature selecton * Correspondng author. Tel.: E-mal address: wzhang@se.xtu.edu.cn Publshed by Elsever Ltd. do: /.proeng Open access under CC BY-NC-ND lcense.

2 We Zhang and Feng Gao / Proceda Engneerng 15 (2011) [9],[10],[11],[12]. In theses lteratures, the naïve Bayes classfers are based on the condtonal probablty of the features belongng to one class after features selecton usng the exstng methods. An auxlary feature methos proposen ths paper. It determnes features by an exstng feature selecton method, and selects an auxlary feature whch can reclassfy the class space amed at the chosen features. Then the correspondng condtonal probablty s adusten order to mprove classfcaton accuracy. The experments wth data sets obtaned from CCERT shows that the auxlary feature methomproves the classfcaton accuracy of naïve Bayes because of reclassfcaton of the class space. In ths paper, we propose the auxlary feature method. The feature wth auxlary feature was found and the probablty of the feature wth auxlary feature was adusted after feature selecton. Ths paper s structured as follows. Secton 2 ntroduces Naïve Bayes model. In Secton 3 we present our approach for enhancng nave Bayes by usng auxlary feature adust probablty. Secton 4 contans expermental results demonstratng that the predctve accuracy of nave Bayes can be mproved by auxlary feature method. Secton 5 dscusses related work and future work. 2. Naïve Bayes Model n Text Classfcaton Denote a vector of varables D = d, = 1,2,..., n, represent document, where d s correspondng to a letter, a word, or other attrbutes about some text n realty, and a set of C = { c1, c 2,..., ck } s predefned classes. Text classfcaton s to assgn a class label c, = 1,2,..., k from C to a document. Bayes classfer s a hybrd parameter probablty model n essence: ) P( D c ) = P( D) (1) P c s pror nformaton of the appearng probablty of class, P D s the nformaton Where ( ) c ( ) from observatons, whch s the knowledge from the text tself to be classfed, and dstrbuton probablty of document D n classes space. Bayes classfer s to ntegrate these nformaton and compute separately the posteror of document D fallng nto each class c, and assgn the document to the class wth the hghest probablty, that s * c ( D) = arg max P( c (2) P( D c ) s the Assume the components of D are ndependent wth each other snce condtonal probablty P( D c ) cannot be computed drectly n practce. Thus P( D c ) = P( d c) (3) The model wth the above assumpton s called natve Bayes model, and equaton (1) becomes ) P( d c) = (4) P( D) Because the sample nformaton P( D) s dentcal to each class c, = 1,2,..., k, equaton (2) becomes * c D = arg max P c P d c (5) ( ) ( ) ( )

3 2162 We Zhang and Feng Gao / Proceda Engneerng 15 (2011) Ths paper wll deal wth two knds of classfcaton problems for smplcty, and denote class set C = { +, }. We choose mult-varable Bernoull model as our Bayes model[13], that s, f correspondng to feature presence n the document, assgn ts value 1, otherwse, assgn the value Auxlary Feature Method There are lots of feature selecton methods n text classfcaton, such as DF (Document Frequency) IG (Informaton Gan) MI (Mutual Informaton) OR(Odds Rato) χ 2 (CHI Squared Statstc) ECE (Expected Cross Entropy) WET (the Weght of Evdence for Text)[14],[15],[16],[17], and plentful works have been researched on these methods. Ths paper won t evaluate the pros and cons of them here, and we ust adust the probablty of feature wth the auxlary property to mprove the performance of orgnal naïve Bayes classfer, after feature selecton usng exstng algorthm. Denote the feature correspondng to the component d as, and denote p w + and p w as p + p an separately for smplcty. For every ndependent feature, f p+ > p, Bayes classfer wll assgn the document to class +. The dea here s to fnd a auxlary feature for feature,whch satsfes p p > p p + + p +, and ans the probablty of document belongng to class + and - w p respectvely when feature and auxlary feature presence smultaneously n the document. The geometrc llustraton for the methos as follows: w w ( ) w ( ). (b) The text space. (d) The msclass- Fg.1. The geometrc llustraton of auxlary feature method. (a) The text space dvson by the feature dvson by the feature w and the auxlary feature w. (c) The msclassfcaton nduced by the feature fcaton nduced by the feature and the auxlary feature w w. w w In fgure 1, (a) denotes all the document ncludng the features ans a subspace of the whole text space, and the dagonal separate dfferent classes; The shadow part of (b) denotes the text subspace smultaneously ncludng auxlary feature w ; The shadow part of (c) denotes the classfcaton fault of orgnal naïve Bayes method. The shadow part of (d) denotes classfcaton fault of auxlary feature method. The constrant condton p p+ > p+ p when fndng auxlary feature s to guarantee the top part of the shadow s smaller than the blank porton below n (d). Algorthm of auxlary feature: Step1:make feature selecton to determne wth text vector of the dmenson n. Step2:fnd the set Θ composed of all document n whch presence. Step3:fnd the auxlary feature for d n Θ. Step4:repeat step2 and step3 for = 1, 2,..., n. w

4 We Zhang and Feng Gao / Proceda Engneerng 15 (2011) To classfy a new document wth a feature vector d1, d2,..., dn, hypothess the auxlary feature of the feature correspondng to the component presence n the document make p = p, p = p exst. Whle the auxlary feature and orgnal feature all, otherwse same wth naïve Bayes. Note that not all of the features have correspondng auxlary one. If one feature exst more than a auxlary feature, we choce auxlary feature whch s subect to max( ( p p ) ( p p )) Experments The data sets we used here s two mal sets wth unk mals and normal mals from CCERT[18]. We choose unk mals and normal mals as the data sets for our experment. There are three cases here: (a) choose 1000 features to represent the document, ant turns out that 96 features have auxlary feature; (b) choose 1500 features to represent the document, ant turns out that 152 features have auxlary feature; (c) choose 2000 features to represent the document, ant turns out that 217 features have auxlary feature. We conduct 5 tmes 10-fold cross valdaton usng naïve Bayes method and the proposed method respectvely for the three cases. The average results are as follows. Table1.The classfcaton precse of auxlary feature method vs. nave Bayes. Naïve Bayes Auxlary Feature Method 1000 features features features Concluson After feature selecton n text classfcaton, nave Bayes classfer partton the text subspace composed of all document n whch d present based on each. Because naïve Bayes classfer assgns the document to the class wth the hghest probablty, naïve Bayes classfer s the optmal n probablty sense. The auxlary feature method proposed here partton the text subspace agan, so t outperforms the tradtonal way, anllustratve examples show that the proposed methondeemproves the performance of nave Bayes classfer. Snce the auxlary feature method need choose features twce, how to gve the auxlary drectly s meanngful and can reduce the computaton complexty. Meanwhle, the relatonshp between exstng choosng features methods and our methos promsng. In addton,due to the sparsty problem n text classfcaton, whether to take the feature total of document nto account when adustng the probablty s worth to work other than substtuton n ths paper. Acknowledgements The research s supporten part by the Natonal Natural Scence Foundaton ( , , , ), Natonal Scence Fund for Dstngushed Young Scholars ( ), Key

5 2164 We Zhang and Feng Gao / Proceda Engneerng 15 (2011) Proects n the Natonal Scence &Technology Pllar Program (2011BAK08B02), 863 Hgh Tech Development Plan (2007AA01Z480, 2008AA01Z415). References [1] Duda,R.O.and Hart,P.E.:Pattern Classfcaton and Scene Analyss.New York:John Wley [2] Lews, D.D. Nave (Bayes) at forty: The ndependence assumpton n nformaton retreval. MachneLearnng: ECML-98, Tenth European Conference on Machne Learnng [3] D.D. Lews, Representaton and Learnng n Informaton Retreval, PhD dssertaton, Dept. of Computer Scence, Unv.of Massachusetts, Amherst, [4] A.K. McCallum and K. Ngam, Employng EM n Pool-Based Actve Learnng for Text Classfcaton, Proc. ICML-98, 15th Int l Conf. Machne Learnng, J.W. Shavlk, ed., pp , [5] McCallum, A., Ngam, K.: A comparson of event models for Nave Bayes text classfcaton. In: Learnng for Text Categorzaton: Papers from the AAAI Workshop, AAAI Press Techncal Report WS [6] Eyheramendy, S., Lews, D.D., Madgan, D.: On the Nave Bayes model for text categorzaton. In Bshop, C.M., Frey, B.J., eds.: AI & Statstcs 2003: Proceedngs of the Nnth InternatonalWorkshop on Artfcal Intellgence and Statstcs [7] D. Pavlov, R. Balasubramanyan, B. Dom, S. Kapur, and J. Parkh. Document preprocessng for naïve bayes classfcaton and clusterng wth mxture of multnomals. In Proceedngs of the ACM SIGKDD Internatonal Conference on Knowledge Dscovery and Data Mnng (KDD-2004), [8] Eyheramendy, S., Lews, D.D., Madgan, D.: On the Nave Bayes model for text categorzaton. In Bshop, C.M., Frey, B.J., eds.: AI & Statstcs 2003: Proceedngs of the Nnth InternatonalWorkshop on Artfcal Intellgence and Statstcs. 2003, [9] Y. Yang and J. O. Pedersen. A comparatve study on feature selecton n text categorzaton. In Internatonal Conference on Machne Learnng, pages , [10] Rogat, M.; Yang, Y. Hgh-performng feature selecton for text classfcaton. CIKM 02, 2002, pp [11] J. Chen, H. Huang, S. Tan, and Y. Qu, "Feature selecton for text classfcaton wth Naïve Bayes," Expert Systems wth Applcatons, vol. 36, no. 3, pp , Aprl [12] M. Srnvas, K. P. Supreeth, E. V. Prasad, and S. A. Kumar, "Effcent Text Classfcaton Usng Best Feature Selecton and Combnaton of Methods," n Proceedngs of the Symposum on Human Interface 2009 on ConferenceUnversal Access n Human-Computer Interacton. Part I: Held as Part of HCI Internatonal Sprnger-Verlag, 2009, pp [13] McCallum, A., Ngam, K.: A comparson of event models for Nave Bayes text classfcaton. In: Learnng for Text Categorzaton: Papers from the AAAI Workshop,AAAI Press (1998) Techncal Report WS [14] D. Koller and M. Saham. Toward optmal feature selecton. In Internatonal Conference on Machne Learnng, pages , [15] Y. Yang and J. O. Pedersen. A comparatve study on feature selecton n text categorzaton. In Internatonal Conference on Machne Learnng, pages , [16] S. Das. Flters, wrappers and a boostng-based hybrd for feature selecton. In Internatonal Conference on Machne Learnng, [17] E. P. Xng, M. I. Jordan, and R. M. Karp. Feature selecton for hgh-dmensonal genomc mcroarray data. In Proc. 18th Internatonal Conf. on Machne Learnng, pages Morgan Kaufmann, San Francsco, CA, [18]