Text Feature Selection based on Feature Dispersion Degree and Feature Concentration Degree

Transcription

1 Available olie at vol. 13, o. 7, November 017, pp DOI: /ijpe p Text Feature Selectio based o Feature Dispersio Degree ad Feature Cocetratio Degree Zhifeg Zhag a, Yuhua Li a, Haodog Zhu b,* a School of Software, Zhegzhou Uiversity of Light Idustry, Zhegzhou, Hea, 45000, P. R. Chia b School of Computer ad Commuicatio Egieerig, Zhegzhou Uiversity of Light Idustry Zhegzhou, Hea, 45000, P. R. Chia Abstract Text feature selectio is oe of the key steps i text classificatio, ad thus ca affect performace of text classificatio. I this paper, the feature dispersio degree of betwee-class documets is first put forward to measure the feature dispersio betwee categories (the greater its value, the larger the ifluece of the feature has). The feature cocetratio degree of withi-class documets is the proposed to measure feature cocetratio i the text of a category (the greater its value, the larger the ifluece of feature has). Subsequetly, a text feature selectio method is preseted, which uses both of the proposed degrees comprehesively to measure the importace of features. Experimetal compariso results show that the proposed feature selectio method ca ofte get more represetative feature subsets ad improve performace of text classificatio. Keywords: feature selectio; text classificatio; vector space model; text feature vector (Submitted o July 17, 017; First Revised o October 7, 017; Secod Revised o October 15, 017; Accepted o October 17, 017) 017 Totem Publisher, Ic. All rights reserved. 1. Itroductio With the popularizatio of computer ad the developmet of moder iformatio techology, text iformatio icreases sharply. Automatic text classificatio as a effective core techology to process a large amout of text iformatio has attracted great attetio to academia. Automatic text classificatio assigs a documet to oe or more categories accordig to the documet cotet as well as its properties [8]. At preset, vector space model is geerally used to represet a documet i text classificatio model [3]. I this model, the traiig documets are usually composed of a large umber of features, ad each feature has a certai degree of effect to text classificatio. However, these features also ivolve may related ad redudat features. The existece of those redudat features ot oly icreases the cost of time ad space of the system, but also greatly limits the choice of classificatio algorithm ad reduces the performace of automatic text classificatio. So, before the implemetatio of automatic text classificatio, redudat features have to be elimiated to reduce operatio cost ad improve the accuracy ad the efficiecy of text classificatio [1,5]. I terms of elimiatig redudat features, feature selectio is a more effective method. It selects some represetative features, which have a larger cotributio to classificatio algorithm, to compose feature subset [10,11]. The geeral process is to evaluate every feature with scores usig a evaluatio fuctio, rak them accordig to these scores, ad take those features with high scores to form a feature subset to represet relevat texts. Curretly, the feature selectio methods maily cotai documet frequecy, mutual iformatio, cross etropy, iformatio gai, x statistics, ad so o. However, these methods oly focus o oe aspect of the feature i evaluatio phase, which makes the selected feature subset ot represetative [4,9]. So, i this paper, a ew feature selectio method is proposed. It cosiders the feature dispersio degree of betwee-class documets ad the feature cocetratio degree of * Correspodig author. Tel.: ; fax: address: zhuhaodog80@163.com.

2 1160 Zhifeg Zhag, Yuhua Li, Haodog Zhu withi-class documets so that it ca comprehesively evaluate the selected features; the selected feature subset is more represetative to improve the performace of text classificatio.. Problem Descriptio I text classificatio, text is ofte represeted by a vector space model, which ivolves some basic defiitios as follows [6,7]: Defiitio 1: Text Feature. Text is usually composed of some basic laguage uits, such as words, phrases or a set of these basic laguage uits. These basic laguage uits are collectively referred to as text features. Give a text T, which ca be represeted by a set of features, i.e. T( t1, t,, t ), where is feature ad 1k. t k Defiitio : Feature Weights. For a text T( t1, t,, t ) cotais features, where tk is a feature, 1k. Usually t k is edowed with a weight to describe its importace, i.e. T( t1, w1 ; t, w ; ; t, w ). Whe features are determiate, it is ofte writte T( w1, w,, w ) briefly. Defiitio 3: Vector Space Model (VSM). Give a textt( t1, w1 ; t, w ; ; t, w ), feature (1k) ca geerally be repeated several times with a certai order, which greatly icreases the difficulty of text aalysis. I order to simplify the text aalysis, we do t cosider the order of i the text, ad oly thik that each feature is differet from others. So that t1, t,, t ca be cosidered as a -dimesioal coordiate system ad w1 w t k,,, w are correspodig to coordiate values. Cosequetly, T( w1, w,, w ) is regarded as a vector of dimesio to represet the text T. Defiitio 4: Text Feature Vector.I the vector space model, each text is represeted by a vector, whose elemets are formed by the weights of features ad the vector is called the text feature vector. Text feature vector is a feature descriptio of the text, i a sese it ca fully represet the documet. For example, there is a text T of features, ad its feature vector ca be formulated as T( w1, w,, w ). Defiitio 5. Give a set of m texts, where Ti ( i 1,,, m) represets the i th text feature vector. The set possesses p categories, ad the the average vector for the etire text set is as follows: t k m C (1/ m) T i1 i (1) The average vector for k categories: m C (1/ m ) k T k k i i1 () Where p m k C (1/ m) mk Ck k 1. deotes the text umber belog to category k. These two average vectors have the relatio 3. Feature Dispersio Degree ad Feature Cocetratio Degree The purpose of feature selectio is to fid a more represetative feature subset, ad the use the feature subset to represet the origial set of texts. I classical liear discrimiat aalysis, betwee-class matrix ad withi-class matrix are two commoly used objective fuctios, wherei betwee-class matrix describes the dispersio degree betwee various categories of documets, ad withi-class matrix describes the dispersio degree of withi-class documets. Geerally speakig, the higher dispersio degree betwee various categories of documets is, the smaller dispersio degree of withiclass documets is, the more beeficial it is for classificatio system. I this case, the task of the feature selectio is to fid such a feature subset. If the subset is to be used to represet the traiig set, the the greater the dispersio degree betwee various categories of documets is ad the greater the cocetratio degree of withi-class documets is, the larger the

3 Text Feature Selectio based o Feature Dispersio Degree ad Feature Cocetratio Degree 1161 importace of feature is. So, the greater a feature s cotributio to the dispersio degree of betwee-class documets is, the more represetative the feature is. The greater its cotributio to the cocetratio degree of withi-class documets is, the more represetative the feature is. Based o this idea, i this paper, two kids of cotributios of feature are defied. Defiitio 6. Feature Dispersio Degree of Betwee-Class Documets: It expresses the cotributio of a feature to dispersio degree of betwee-class documets, the greater its value is, the larger the ifluece of the feature to discrimiate the categories of documets is. It ca be described by a formula as follows: Dispersio-degree( i) (1/ p) ( C( i) C ( i)) p j (3) j1 text set, Where p deotes the umber of categories, ad Ci () deotes the average vector weight of the i th feature i the whole Cj () i deotes the average vector weight of the i th feature i the category j. Defiitio 7. Feature Cocetratio Degree of Withi-Class Documets: It expresses the cotributio of a feature to cocetratio degree of withi-class documets, the greater its value is, the larger the ifluece of the feature to represet documet class is. It ca be described by a formula as follows: Coc etrat io-degree( i) (1/ m) ( C ( i) T ( i)) p m j j jk (4) j1 k1 Where m deotes the umber of total documets, ad p deotes the umber of categories, vector weight of the i th feature i the category j. Tjk () i Cj () i deotes the average deotes the weight of the i th feature of k th text feature vector i the category j. O the basis of above two cotributios of a feature, we ca get the whole importace of a feature, which ca be described by: 4. Proposed Feature Selectio Method Importace-degree( i) Dispersio-d egree( i) Cocetratio -degree( i) (5) Accordig to formula (5), we ca desig a ew feature selectio method, which is described as follows: Iput: Origial feature vector set ad categories set. Output: a feature subset Q. Step1: accordig to formula (1), we calculate the average vector C of text set. Step: accordig to formula (), we calculate the average vectors of various categories. Step3: accordig to formula (3), the cotributio of a feature to the dispersio degree of betwee-class documets Dispersio-degree( i ) is obtaied. Step4: accordig to formula (4), the cotributio of a feature to the cocetratio degree of withi-class documets Cocetratio-degree(i) is obtaied. Step5: accordig to formula (5), we get the whole importace Importace-degree( i) of the feature i. Step6: we select the top P features accordig to importace scores to form a subset Q ad output. 5. Experimetal Verificatio 5.1. Experimetal Corpus I this paper, we select the Chiese text classificatio corpus of Fuda Uiversity as the experimetal corpus, which is costructed by the team of atural laguage processig of computer iformatio ad techology departmet, ad ca be dowloaded from cat_id=16. The experimetal corpus cotais 0 categories, ad is divided ito the traiig set ad the test set, ad each part cotais 0 subdirectories. The same categories of documets are stored i a correspodig subdirectory, meawhile each storage file cotais oly a documet. All documets are uiquely umbered with the file ame. After removig some repeated ad damaged documets, oly 14378

4 116 Zhifeg Zhag, Yuhua Li, Haodog Zhu documets remai, wherei the traiig set cotais 814 articles, ad test set cotais 6164 articles, without the repeated cross-category documets, i.e. oe documet oly belogs to oe category. The distributio of documets of the corpus is ueve. Amog them, there are 1369 traiig documets for the class Ecoomy of the traiig set, which has the largest umber of documets, ad 5 traiig documets for the miimal class. Meawhile, there are 11 categories whose umber of traiig documets of categories are less tha 100, ad the traiig set ad the test set are ot overlapped. 5.. Experimetal Eviromet Settigs We employ the Chiese lexical aalysis system (ICTCLAS) developed by Istitute of Computig Techology of Chiese Academy of Scieces to carry out word segmetatio, ad select Weka tool as the experimet platform, which is developed by the uiversity of Waikato, New Zealad. Weka tool icludes a rage of machie learig algorithms i the field of data miig, such as data preprocessig, classificatio ad regressio aalysis, clusterig, associatio rules, ad visualizatio, ad ca be dowloaded from the followig url: We adopt MATLAB 7.0 to implemet umerical calculatio Classifier ad Evaluatio Stadards The proposed method is maily compared with followig three feature selectio methods []: iformatio gai (IG), statistics (CHI), mutual iformatio (MI). We use KNN classifier (K is set to 0, ad cosie distace is adopted to calculate similarity) implemet classificatio experimet. I order to evaluate performace of these four methods with the chagig umber of features, we select the micro average ad the macro average as the performace evaluatio stadards Experimetal Results ad Aalysis Micro-average are calculated uder differet umbers of features, ad experimetal results are show i Figure 1 ad Figure, respectively. Note that i Figure 1 ad Figure, the umbers from 1 to 15 represet the umber of features respectively, i.e. 50, 100, 00, 500, 800, 900, 1000, 1500, 000, 500, 3000, 3500, 4000, 4500, The y-axis represets the correspodig average micro ad Macro-average ad macro average i uit %. x Micro-average F1 \% The proposed method IG MI The differet umber of features /Null Figure 1. Micro-average uder differet umber of features With the chagig umber of features, the performace chage of the classificatio model ca reflect the sesitivity of classificatio model. As see from the Figure 1, with the icrease of umber of features, the micro-average icreases also icreases with the gradually, ad achieves a relatively stable level. As see from the Figure, the macro-average growth of umber of features but with relatively large fluctuatios because the distributios of categories i this selected corpus are extremely ueve. From Figure 1 ad Figure, it ca be see that KNN classifier has the best performace i top CHI

5 Text Feature Selectio based o Feature Dispersio Degree ad Feature Cocetratio Degree features for the proposed method; the micro-average ad the macro-average are about 90% ad 80% respectively. KNN classifier for IG has the best performace o selected top 000 features; the micro-average macro-average the micro-average ad the are about 67% ad 6%. KNN classifier for CHI has the best performace o selected top 1500 features; ad the macro-average are about 86% ad 74%. KNN classifier for MI has the best performace o selected top 1500 features; the micro-average result, the overall performaces of these four feature selectio methods from large to small are the proposed method > CHI > MI > IG. The reaso lies i that the proposed method ot oly cosiders the ifluece of the features to the dispersio degree of betwee-class documets, but also cosiders the ifluece of the features to the cocetratio degree of withiclass documets. Therefore, this proposed method is ot affected by the distributio of the selected documet corpus, ad ca comprehesively cosider the selected features. So, the selected features have better represetativeess. At the same time, MI oly examies the existig situatio of the selected features i selectig feature phase, ad CHI cosiders tow situatios of the feature occurrig ad ot occurrig, so CHI is superior to MI. Because IG is extremely sesitive to the distributio of the samples, if it is used i the coditio of the ueve distributio of samples, the represetative of the selected feature set is poorer. I this paper, the distributio of the categories of selected corpus is extremely ueve, so the performace of IG is the worst ad the macro-average are about 84% ad 71% respectively. As a Macro-average F1 \% The proposed method 45 IG 40 CHI MI The differet umber of features /Null Figure. Macro-average uder differet umber of features 6. Coclusios This paper defies the feature dispersio degree of betwee-class documets ad the feature cocetratio degree of withiclass documets. Based o them, a ew feature selectio method is proposed. As demostrated through experimetatios o the corpus of Fuda Uiversity ad comparisos with three classical methods (MI, CHI ad IG), the proposed method has better capability to select the most represetative features, ad ca be used i some kowledge discovery algorithm to reduce time ad space complexity. The proposed method ot oly has applicatio i the text classificatio, but also provides a idea for other text feature selectio methods. Ackowledgmets The authors would like to thak the editors ad the aoymous reviewers for their helpful commets ad suggestios, which have improved the presetatio. This work is supported i part by the Sciece ad Techology Pla Projects of Hea Provice of Chia uder grat No ad No , the Youth Backboe Teachers Fudig Plaig Project of Colleges ad Uiversities i Hea Provice of Chia uder grat No.014GGJS-084, the Key Sciece Research Project of Colleges ad Uiversities i Hea Provice of Chia uder grat No. 16A50030, the Youth Backboe Teachers Traiig Targets Fuded Project of Zhegzhou Uiversity of Light Idustry of Hea Provice of Chia uder grat No.XGGJS0, the Ph.D. Research Fuded Project of Zhegzhou Uiversity of Light Idustry of Hea Provice of Chia uder grat No.010BSJJ038 ad No.014BSJJ080, ad the Natioal Sciece Foudatio of Chia uder grat No

6 1164 Zhifeg Zhag, Yuhua Li, Haodog Zhu Refereces 1. J. Cai, J. Luo, C. Liag, S. Yag, " A Novel Iformatio Theory-Based Esemble Feature Selectio Framework for High- Dimesioal Microarray Data", Iteratioal Joural of Performability Egieerig, vol. 13, o. 5, pp , A. Destrero, S. Mosci, C. D. Mol,A. Verri, F. Odoe, "Feature selectio for high-dimesioal data", Computatioal maagemet sciece, vol. 6, o. 1, pp. 5-40, F. Jiméez, G. Sáchez, J. M. García, et al, "Multi-objective evolutioary feature selectio for olie sales forecastig", Neurocomputig, vol. 34, pp. 75-9, S. R. Y. Leela, V. Sucharita, B. Debath, H. J. Kim, "Performace evaluatio of feature selectio methods o large dimesioal databases ", Iteratioal Joural of Database Theory ad Applicatio, vol. 9, o. 9, pp. 75-8, J. H. Liu, Y. J. Li, M. L. Li, "Feature selectio based o quality of iformatio", Neurocomputig, vol. 5, pp. 11-, J. N. Meg, H. F. Li, Y. H. Yu, "A two-stage feature selectio method for text categorizatio", Computers & Mathematics with Applicatios, vol. 6, o. 7, pp , M. H. Nguye, D. F. Torre, "Optimal feature selectio for support vector machies", Patter Recogitio, vol. 43, o. 3, pp , A. Rehma, K. Javed, H. A. Babri, "Feature selectio based o a ormalized differece measure for text classificatio", Iformatio Processig & Maagemet, vol. 53, o., pp , T. Su, S. Y. Qia, H. D. Zhu, "Feature selectio method based o category correlatio ad discerible sets", Joural of Computatioal Iformatio Systems, vol.11, o., pp , S. Q. Wag, J. M. Wei, "Feature selectio based o measuremet of ability to classify subproblems", Neurocomputig, vol. 4, pp , H. D. Zhu, H. C. Li, D. Wu, D. S. Huag, B. Wag, "Feature selectio method based o feature distiguishability ad fractal dimesio", Joural of Iformatio ad Computatioal Sciece, vol. 36, o. 5, pp , 015. Biography Zhifeg Zhag was bor i 1978 i Hea Provice, Chia. He received his B.S. degree from Xi'a Uiversity of Electroic Sciece ad Techology, Xi'a, Shaxi Provice, Chia, i 001, ad his M.S. degree from Xi'a Uiversity of Techology, Xi'a, Shaxi Provice, Chia, i 006. Sice 006, he has bee i the School of Software, Zhegzhou Uiversity of Light Idustry, Zhegzhou, Hea Provice, Chia, where he is curretly a Associate Professor. His major research iterests iclude Cloud Computatio, Itelligece Iformatio Processig, ad Data Miig. Yuhua Li is curretly a lecturer at School of Software, Zheg Zhou Uiversity of Light Idustry, Zhegzhou, Hea Provice, Chia. He received his Ph.D. degree i Computer Software ad Theory from Su Yat-se Uiversity, Guagzhou, Guagdog Provice, Chia,i 014. His curret research iterests iclude multimedia iformatio retrieval, machie learig ad computatioal itelligece. Haodog Zhu was bor i Hea Provice, Chia, i He received the B.S. degree from Lazhou Jiaotog Uiversity, Lazhou,Gasu Provice,Chia, i 004, ad the M.S. degree from Sichua Uiversity of Sciece & Egieerig, Zigog, Sichua Provice,Chia,i 008, ad the Ph.D. degree from Graduate Uiversity of Chiese Academy of Scieces i 011. Sice 010, he has bee with the faculty of the School of Computer ad Commuicatio Egieerig, Zhegzhou Uiversity of Light Idustry, Zhegzhou, Hea Provice, Chia, where he is curretly a Associate Professor ad a master Tutor. His major research iterests iclude Cloud Computatio, Itelligece Iformatio Processig, Computig Itelligece ad Data Miig.