Improved Mutual Information Based on Relative Frequency. Factor and Degree of Difference among Classes

Transcription

1 2nd Informaon Technology and Mechatroncs Engneerng Conference (ITOEC 2016 Improved Mutual Informaon Based on Relave Frequency Factor and Degree of Dfference among Classes Janwen Gao a*, X Yangb,Wen Wenc and Jhua Yangd Department of Informaon Engneerng,Engneerng Unversty of CAPF, X an ,Chna @qq.com, @qq.com, c @qq.com, d @qq.com a b Keywords: feature selecon; relave frequency factor; ; degree of dfference among classes Abstract. By ntroducng degree of dfference among classes, an mproved mutual nformaon feature selecon method s proposed to effecvely mprove the accuracy of feature selecon, accuracy and effcency of classfcaon. At the same me, relave frequency factor s appled to solve the tendency of tradonal methods to choose the shortage of low frequency words. The expermental results show that the mproved method can reasonably mprove the performance of mutual nformaon feature selecon. 1 Introducon The rapd development of Internet technology enables onlne data to have explosve growth, and all of these data manly exsts n the form of text. In the text automac classfcaon, documents are often converted nto models so as to better computer processng. But the hgh dmenson of the feature space and data sparseness lead to ncrease of compung me and lowered effcency n the process of text representaon, whch may have mpacts on the accuracy of classfcaon. Therefore, to reduce the dmenson of orgnal feature space and ncrease accuracy of classfcaon become the dffcules of text automac classfcaon. At present, methods that are used n dmenson reducon are feature extracon an feature selecon[1]. Feature selecon means to select out those features sets wth strong communcave ablty and greater contrbuon rate of classfcaon n the ntegraon of orgnal feature tems[2]. Currently, the common used selecon methods are TF-IDF, Informaon Gan(IG[3], Mutual Informaon([4], Ch-square Test(CHI[5], Weght of Evdence for Text(WET, Expected Cross Entropy(ECE, etc. Lterature[6] through expermental research shows that methods have relavely low effects n feature selecon, because tradonal methods do not take feature tems of document frequency of dfferent classfcaons nto consderaon, nor dd t consder the word frequency of dfferent documents n the same classfcaon, and s prone to select low-frequency words n the selecon process. In ths paper, through ntroducng degree of dfference among classes, puts forward an mproved feature selecon method of, and ntroduces relave The authors - Publshed by Atlans Press 194

2 frequency factor to deal wth feature selecon method whch s prone to select the defcency of low word frequency. 2 Studes on Method 2.1 Feature Selecon Methods of Mutual Informaon Accordng to the appearance possblty of classfcaon c and feature tem t, means the measurement of relevant degree between them[7]. In text classfcaon, f the total number of document sets s N, the classfcaon ntegraon wll be {c 1, c 2,..., c,..., c m }, the feature tems ntegraon s {t 1,t 2,,t,,t n }, and the computaonal formula of mutual nformaon of classfcaon c and feature tem t s[8] : c, (, c log log c t t, (1 n the formula: c means the appearance possblty of classfcaon c ; t means appearance possblty of feature tem t n the document ntegraon; c,t means the smultaneous appearance possblty classfcaon c and feature tem t ; c /t means the appearance possblty of feature tem t n the classfcaon c. From formula (1, the lower frequency of feature tem t n the document sets and the hgher frequency of classfcaon c, the bgger mutual nformaon value, whch means stronger relevant degree between them. If feature tem t dd not appear n classfcaon c, the mutual nformaon value would be 0. Consderng that feature tem may be dstrbuted n other classfcaon, n order to acqure feature tem t n the average mutual nformaon value of the whole text, the computaonal formula wll be[9]: m (, c c log, (2 t 1 n the formula: m means the number of classfcaon. 2.2 Analyss on The Mutual Informaon Methods From analyss formula (1 and (2, elements that decde the sze of the mutual nformaon value only reles on the frequency number of feature tem and the appearance frequency number of the whole text, so the results of the computaon exst followng defcences. Tendng to select low frequency words. In classfcaon c, when feature tem t 1 >t 2 and t 1 /c <t 2 /c, we could acqure (t 1,c <(t 2,c,from formula (1. Accordng to the rankng of mutual nformaon value, t 1 was lsted n the last. In the fnal threshold value selecon, t 1 was elmnated. But for classfcaon c, hgh-frequency word t 1 carres more nformaon, and s more good at expressng text content. 2 There exsts some dencal mutual nformaon value of some feature tems, but feature tems lsted behnd are easly elmnated, whch may cause the loss of some valuable nformaon. 3 In dfferent classfcaon, feature tem t 1 appears n one or several classfcaons, and feature tem t 2 s unformly dstrbuted accordng to dfferent classfcaon. When (t 1, c <(t 2, c appears n computaonal mutual nformaon value, feature tem t 1 has more representaon ablty n classfcaon. 4 In the same classfcaon, feature tem t 1 appears mostly n rare documents, 195

3 and feature tem t 2 s unformly dstrbuted n contaned documents. But for ths classfcaon, t 2 has more representaon ablty and hgher contrbuon rate. In compung mutual nformaon value, the value of t 1 may be bgger than t 2, whch makes t 2 leave behnd, and t 2 may be elmnated n the fnal threshold value. Ths thess puts forward a feature selecon method based on degree of dfference among classes, so as to enhance precson rate of feature selecon method, whle takng relave frequency factor nto consderaon. 3 Improved Feature Selecon Method The thess puts forward mproved feature selecon method based on the two elements of degree of dfference among classes and relave frequency factor. 3.1 Degree of Dfference among Classes The deology of degree of dfference among classes s feature tems that have strong representaon ablty whch should focus on one or several classfcaons, and the contaned documents of all these classfcaons are unformly dstrbuted[10]. Degree of dfference among classes method ntegrates between-class scatter AC wth couplng wth the classfcaon DC, namely consderng the frequency number of feature tem n dfferent classfcaon and dstrbuted dfference of feature tems n dfferent documents of same classfcaon. (1Introducng between-class scatter AC to descrbe dstrbuon condon of feature tems. One feature tem wth strong classfcaon ablty should focus on one or several classfcaons rather than unformly dstrbuted. The computaonal formula of between-class scatter s: m AC 1 ( ( df ( df ( 2, (3 m 1 1 n the formula: df (t represents classfcaon c whch contans documents number of feature tems; df t means average documents number of every classfcaon ( contaned feature tem t, and m s classfcaon number. The greater the between-class scatter, the greater classfcaon ablty of feature tem. (2Introducng classfcaon DC to descrbe dstrbuon condons of classfcaon text. A strong feature tem wth representaon ablty should be unformly dstrbuted n the documents rather than focusng on several documents. The computaonal formula of classfcaon s: 1 n ( fk( f ( t, (4 DC n k n the formula: n represents the total documents number of c ; f k (t represents the appearance number of the k documents of feature tem t ; f t represents the average number of classfcaon c of feature tem t. The bgger the couplng wth the classfcaon, the stronger ablty of feature tem. At last degree of dfference among classes s ntroduced: AC DC. (5 After ntroducng degree of dfference among classes nto formula (1, we ( 196

4 acqure: log t. (6 3.2 Relave Word Frequency Factor relave frequency factor Introducng relave frequency factor s manly to solve defcences that are prone to select low-frequency words of feature selecon n mutual nformaon method. Word frequency s based on the frequency number of feature tem n classfcaon. If f (t represents frequency number of feature tem t n classfcaon c, the relave word frequency degree of feature tem t s and represents: f ( t, (7 f ( t n the formula: f t represents the average value of appearance frequency of feature ( tem t n all classfcaons. Introducng relave frequency factor :, (8 2 from above defnons, we could know that for one certan feature tem, the bgger the relave word frequency, the bgger the classfcaon dfference, and the contrbuon rate to text classfcaon wll be hgher. Therefore, ntroducng relave word frequency factor and classfcaon dfference can acqure the computaonal formula of new feature selecon method: log t. (9 4 Expermental Results and Analyss 4.1 Expermental Preparaon The experment of the thess selects the Chnese corpus data of Fudan Unversty, and carres out the experments of fve classfcaon : envronment, polcs, economy, mltary, and sports. Varous types of documents are selected as shown n Table 1. Table1. Data Sets Classfcaon Tranng Sets Test Sets Envronment Polcs Economy Mltary Sports The selected proporon of tranng sets and test sets s 5:1. The experment wll be carred out accordng to the pre-process of text data, feature selecon, classfer tranng, data test, and results analyss. The hardware envronment s CPU 5, 2.6G Hz, RAM 4 GB; appled software envronment s Chnese word segmentaon system of Chnese Academy of Scences(ICTCLAS, Eclpse; the classfer selects Support Vector Machne(SVM; programmng language based on JAVA. The experment s dvded nto two parts, one makes a comparson of mproved method and tradonal method under the same dmenson; another makes a comparson of 197

5 mproved method and tradonal method under dfferent dmenson. 4.2 Evaluaon Indcator Evaluaon ndcator of classfcaon effects adopts R(Recall and P(Precson that are unversally recognzed, and the computaonal formula of the two ndcators s : Text Number of Correct Classfcaon R, (10 Total Number of Tesng Docunents Text Number of Correct Classfcaon P, (11 Text Number of Actual Classfcaon R represents the ablty to measure the text number of correct arthmec classfcaon; P represents the ablty to measure the text number of declnng arthmec errors. Both of them reflects text qualty from dfferent aspects, and they are ndspensable. In actual condons, what s generally adopted s the harmonc mean of the two ndcators, namely, comprehensve evaluaon ndcator F 1, and the computaonal formula s: P R F 2. (12 1 P R 4.3 Experment Results and Analyss (1 Experment 1 To verfy the effects of the method, the thess makes a comparson of tradonal mutual nformaon method and mproved method. The feature dmenson selecon s 300 wth SVM classfer test, and ts stascal classfcaon effects are as Table 2. Table2. Results from Experment 1 Classfcaon Tradonal Method Improved Method R/% P/% R/% P/% Envronment Polcs Economy Mltary Sports Table 2 shows the comparson results of varous recall rao and precson rao adopted the two methods. After classfcaon, the recall rao and precson rao of all classfcaons are: envronment classfcaon ncreased 12.67% and 9.99%; polcal classfcaon ncreased 0.56% and 12.04%; economc classfcaon ncreased 5.75% and 4.8%; mltary classfcaon ncreased 14.79% and 13.31%; sports classfcaon ncreased 22.54% and 0.04%. The bggest rse of recall rao s 8.04%, and the hghest precson rao s mltary. The average recall rao ncreased 11.26%, and the average precson rao 8.04%. The data results show that the recall rao and precson rao results that adopted the method have a certan degree of rse compared wth tradonal nformaon methods. Accordng to the above results, F 1 value of comprehensve apprasal ndcator s as Table

6 Table3. Result 2 from Experment 1 Classfcaon Increased Tradonal Improved rao/(% Envronment Polcs Economy Mltary Sports From Table 3, every classfcaon of F 1 value adopted the method n the thess has a certan degree of rse, wth envronment classfcaon the hghest and economc classfcaon the lowest. The average F 1 ncreases 18.55%, whch means that the method put forward n the thess s better than the tradonal mutual nformaon methods n feature selecon. (2 Experment 2 In order to further verfy the enhanced effects of feature selecon n the method, the thess makes a comparson of the method put forward n the thess and tradonal mutual nformaon methods n dfferent dmensons. The experment adopts the SVM, and the results are as the Fg.1. F 1 The average value of F1 Improved Tradonal Dmenson Fg.1. Effects Comparson of Text Classfcaon of Dfferent Dmenson From Fg.1, we could see that F 1 value of the two methods would ncrease wth the rse of dmenson number. When the dmenson number reaches 4,000, the curve tends to be horzontal. The F 1 value of results acqured from tradonal mutual nformaon methods n low dmensonalty s relavely low, and wth the ncrease of dmensonalty number, the growth rate may fluctuate. The thess adopts the mproved method. When the F 1 value reaches the number of 3,000, there s a turnng pont; when reachng 4,000 and 5,000, there s a declne. Generally speakng, all the F 1 values that are put forward n the thess are hgher than tradonal mutual nformaon methods n dfferent dmenson numbers. 199

7 4 Concluson The good or the bad of the feature selecon methods has drect mpact on the accuracy of the results of text classfcaon. Targeted at the exsng defcences of the tradonal mutual nformaon methods, the thess puts forward an mproved mutual nformaon method based on degree of dfference among classes, and ntroduces relave frequency factor whch are prone to select hgh-frequency words. The experment on text classfcaon shows that the methods put forward n the thess are helpful to enhance the recall rate, precson rate and F 1 value compared wth the tradonal mutual nformaon methods. Feature dmensonalty reducon s the key and the dffculty of the text classfcaon research. Feature selecon and feature extracon are two methods of feature dmensonalty reducon whch have ther own advantages and dsadvantages. The stress of the next-step research should be attracng the advantages of the two methods, putng forward a comprehensve method to feature selecon so as to sasfy the demand of text classfcaon. References [1] Yan.J, Lu.N, Yan.S.C, et al. Trace-orented feature analyss for large-scale text data dmenson reducon. IEEE Transacons on Knowledge and Data Engneerng, 2011, 23(7: [2] Zhao.Z, Wang.L, Lu.H, et al. On smlarty preservng feature selecon. IEEE Transacons on Knowledge and Data Engneerng, 2013, 25(3: [3] Harun.U. A two-stage feature selecon method for text categorzaon by usng nformaon gan, prncpal component analyss and genec algorthm. Knowledge-Based Systems, 2011, 24(7: [4] Amr.F, Rezae.M. Mutual nformaon-based feature selecon for ntruson detecon systems. Journal of Network and Computer Applcaons, 2011, 34: [5] Forman.G. An extensve emprcal study of feature selecon metrcs for text classfcaon. Journal of Machne Learnng Research, 2003, 3(1: [6] Yang.Y.H, J.O.Pedersen. A comparave study on feature selecon n text categorzaon // Proceedngs of the 14th Internaonal Conference on Machne Learnng. Nashvlle: Morgan Kaufmann, 1997: [7] Yang.J.M, Wang.J, Qu.Z.Y. Feature selecon method based on the relave contrbuon. Journal of Northeast Danl Unversty, 2014, 34(4: [8] Bakus.J, Kamel.M.S. Hgher order feature selecon for text classfcaon. Knowledge and Informaon Systems, 2006, 9(4: [9] Cheng.W.Q, Tang.X. A text feature selecon method usng the mproved mutual nformaon and nformaon entropy. Journal of Nanng Unversty of Posts and Telecommuncaons, 2013, 33(5: [10] Zhou.Q.N, Zhang.Z.H, Xu.D.C. Feature selecon method for Chnese text categorsaon based on class dscrmnang words. Computer Applcaons and Software, 2013, 33(7: