Classifier Selection Based on Data Complexity Measures *

Transcription

1 Classfer Selecton Based on Data Complexty Measures * Edth Hernández-Reyes, J.A. Carrasco-Ochoa, and J.Fco. Martínez-Trndad Natonal Insttute for Astrophyscs, Optcs and Electroncs, Lus Enrque Erro No.1 Sta. Ma. Tonantzntla, Puebla, Méxco C. P {ereyes, arel, fmartne}@naoep.mx Abstract. Tn Kam Ho and Ester Bernardò Manslla n 004 proposed to use data complexty measures to determne the doman of competton of the classfers. They appled dfferent classfers over a set of problems of two classes and determned the best classfer for each one. Then for each classfer they analyzed how the values of some pars of complexty measures were, and based on ths analyss they determne the doman of competton of the classfers. In ths work, we propose a new method for selectng the best classfer for a gven problem, based n the complexty measures. Some experments were made wth dfferent classfers and the results are presented. 1 Introducton Selectng an optmal classfer for a pattern recognton applcaton s a dffcult task. Few efforts have been made n ths drecton; for example STATLOG [1] where several classfcaton algorthms were compared based on some emprcal data sets and a metal-level machne learnng rule on the algorthm selecton was provded. Other example s Meta Analyss of Classfcaton Algorthms [] where a statstcal metamodel to predct the expected classfcaton performance of each algorthm as a functon of data characterstcs was proposed. They used ths nformaton to fnd the relatve rankng of classfcaton algorthms. In ths work we propose an alternatve method usng the geometry of data dstrbutons and ts relatonshp to classfer behavor. Followng [3] the classfer selecton depends on the problem complexty, whch can be measured based on data dstrbuton. In [3] some data complexty measures were ntroduced. These measures characterze the complexty of a classfcaton problem, focusng on the geometrcal complexty of the class boundary. In [4] some problems were characterzed by nne measures taken from [3] to determne the doman of competton of sx classfers. They made the comparson of ther results between two measures. Based on ths comparson, they determned the doman of competton of the classfers. However they dd not present the results f more than two measures were compared together. In ths work, we propose a new method for selectng the best classfer for a gven problem wth two classes (-class problem). Our method descrbes problems wth * Ths work was fnancally supported by CONACyT (Mexco) through the project J38707-A. M. Lazo and A. Sanfelu (Eds.): CIARP 005, LNCS 3773, pp , 005. Sprnger-Verlag Berln Hedelberg 005

2 Classfer Selecton Based on Data Complexty Measures 587 complexty measures and labels them wth the classfer that gets the best accuracy among fve classfers. After, other classfers were used to make the selecton. Ths paper s organzed as follows: n secton the complexty measures used n ths work are descrbed. In secton 3 the proposed method s explaned, n secton 4 some experments are shown and n secton 5 we present our conclusons and future work. Complexty Measures We selected 9 complexty measures from those defned n [3] whch descrbe the most mportant aspects of boundary complexty of -class problems. The selected measures are shown n table 1. Table 1. Complexty measures F1 F F3 L L3 N N3 N4 T Fsher s dscrmnant Volume of overlap regon Maxmum feature effcency Error rate of lnear classfer Nonlnearty of lnear classfer Rato of average ntra/inter class NN dstance Error rate of 1nn classfer Nonlnearty of 1nn classfer Average number of ponts per dmenson These measures are defned as follows: F1: Fsher s Dscrmnant Fsher s dscrmnant was defned for only one feature. Ths s measured by calculatng, for each class, the mean ( µ ) and the varances ( σ ) of the feature; and evaluat- ng the next expresson: ( µ F1 = σ 1 µ ) 1 + σ For a multdmensonal problem, the maxmum F1 over all the features s used to descrbe the problem. F: Volume of Overlap Regon Ths measure takes nto account how the dscrmnatory nformaton s dstrbuted across the features. Ths can be measured by fndng, for each feature (f ), the maxmum max(f,c j ) and the mnmum mn(f,c j ) values for each class (c j ), and then calculatng the length of the overlap regon defned as: (1) MIN(max( f, c1 ),max( f, c)) MAX (mn( f, c1 ),mn( f, c)) F = MAX (max( f, c ),max( f, c ) MIN(mn( f, c ),mn( f, c ))) 1 1 ()

3 588 E. Hernández-Reyes, J.A. Carrasco-Ochoa, and J.Fco. Martínez-Trndad F3: Maxmum Feature Effcency F3 s a measure that descrbes how much each feature contrbutes to the separaton of the two classes. For each feature, all ponts (p) of the same class have values fallng n between the maxmum and the mnmum of that class. If there s an overlap n the feature values, the classes are ambguous n that regon along that dmenson. The effcency of each feature s defned as the fracton, of all remanng ponts, whch are separable by that feature. For a multdmensonal problem we use the maxmum feature effcency. =. F3 separable ( p) p where (3) 1 f p s separable by the feature separable(p) = 0 otherwse L: Nonlnearty of the Lnear Classfer Many algorthms have been proposed to determne lnear separablty. L uses the error rate of the classfer on the tranng set to descrbe the nonlnearty of the lnear classfer. L = error _ rate( lnear _ classfer( tranng _ set)) (4) L3: Nonlnearty of Lnear Classfer L3 descrbes the nonlnearty of the lnear classfer. Ths metrc measures the error rate of the classfer on a test set. L3 = error _ rate( lnear _ classfer( test _ set)) (5) N: Rato of Average Intra/Inter Class NN Dstance Ths metrc s measured as follows: frst compute the average (x) of the Eucldean dstances from each pont to ts nearest neghbour of the same class, and the average (y) of all dstances to nter-class nearest neghbors. The rato of these two averages s the metrc N. Ths measure compares the dsperson wthn the classes aganst the separaton between the classes. x N = (6) y N3: The Nearest Neghbor Error Rate The proxmty of ponts n opposte classes obvously affects the error rate of the nearest neghbor classfer. Thus N3 descrbes the nonlnearty of the K-nn classfer and t measures the error rate of the K-nn classfer on a test set. N3 = error _ rate( K _ nn( test _ set)) (7)

4 Classfer Selecton Based on Data Complexty Measures 589 N4: Nonlnearty of the K-nn Gven a tranng set, a test set s created by lnear nterpolaton between randomly drawn pars of ponts from the same class. Then the error rate of the K-nn on ths test set s measured. Thus N4 uses the error rate of K-nn wth the tranng set to descrbe the nonlnearty of the K-nn classfer. N4 = error _ rate( k nn( tranng _ set)) (8) T: Average Number of Ponts Per Dmenson Ths metrc s measured by calculatng the average number of samples per features. samples T = (9) features 3 Proposed Method In ths secton we descrbe the proposed method based on data complexty measures to select the best classfer for -class. The dea of our method s to descrbe the -class problem by some complexty measures. The label of each -class problem s ts best classfer, whch s determned testng a set of classfers, n ths way; we wll obtan a tranng set of a supervsed classfcaton problem. Therefore a classfer could be used to select the best classfer for a new -class problem. Our method works as follow: 1. Gven a database set, for each problem wth n classes, two or more, C(n,) - class problems are created, takng all possble pars of classes. Ths s done because as t was mentoned n secton 3, the complexty measures were desgned to descrbe the complexty of -class problems.. For each -class problem created n the prevous step a) Calculate the nne complexty measures. b) Apply the set of classfers and assgn a label that ndcates whch was the classfer wth the lowest error for the -class problem. Thus, each problem s characterzed by ts nne complexty measures and labeled wth the class of ts best classfer. These data conform the tranng set. 3. Apply a classfer on the tranng set to make the selecton of the best classfer for a new -class problem. Ths method s depcted n fgure 1.

5 590 E. Hernández-Reyes, J.A. Carrasco-Ochoa, and J.Fco. Martínez-Trndad Generatng -class problem -class problems Apply the classfers on each -class problem and label t. Calculate the 9 complexty metrc for each -class problem New -class problem Tranng set Apply a classfer on the tranng set Fg. 1. Proposed method Best Classfer for the new -class problem 4 Expermental Results In order to test our method we selected 5 data sets from the UC-Irvng repostory [5] (Abalone, Setter, Irs, Pma, Yeast). Followng the proposed method, n the frst step, for each database, wth n classes, C(n,) -class problems were created; thus we had 75 -class problems (see table ). Table. -class problems for each used database Databases Classes -class Problems Abalone Irs 3 3 Setter 6 35 Pma 1 Yeast Total 75 In the second step, for each -class problem, the nne complexty measures were calculated. Then, each problem was evaluated wth fve classfers. The used classfers were: 1. K-nn. Nave Bayes 3. Lneal regresson 4. RBFNetwork 5. J48

6 Classfer Selecton Based on Data Complexty Measures 591 RBFNetwork s a normalzed Gaussan radal bass functon network and J48 s a verson of C4.5, both mplemented n weka [6]. In our method, we consdered the classfer wth the lowest error on a -class problem as the best method, and then we assgn ths classfer as the label of the -class problem. Table 3 shows how the problems were dstrbuted accordng ther best classfer. Table 3. Dstrbuton of the problems Classfer Problems K-nn 41 Nave Bayes 08 J48 13 The problems were only dstrbuted n 3 classes (K-nn, Nave Bayes and j48), because the other two classfers dd not obtan a better classfcaton rate for any of the -class problems. Thus, we obtaned the problems characterzed by ther nne measures of complexty and labeled wth the class of ther best classfer. These data form a tranng set of 75 objects wth 9 varables and separated n 3 classes. Fnally, to select the best classfer for a new -class problem, we appled three dfferent classfers (1-nn, J48, RBFNetwork) on the tranng set. We used ten-fold cross valdaton to evaluate the accuracy of our method. From the used classfers (1-nn, j48 and RBFNetwork). The best was 1-nn, whch obtaned a classfcaton accuracy of 83.5 %. In table 4 we can apprecate the results. Table 4. Results for best classfer selecton Classfer Selecton accuracy 1-nn 83.5 % RBFNetwork 71.6 % J % 5 Conclusons In ths paper, a new method based on complexty measures for selectng the best classfer of a gven -class problem was ntroduced. Our method descrbes -class problems wth complexty measures and labels them wth the class of ther best classfer. After, for makng the selecton a classfer was used. We found that the complexty measures are a good set of features to characterze the problems and make the selecton of the best classfer. As future work, we wll compare our method aganst other methods. Also, we propose to extend the proposed method for problems wth more than two classes by mean of redefnng the complexty measures, n order to allow applyng them on multple class problems.

7 59 E. Hernández-Reyes, J.A. Carrasco-Ochoa, and J.Fco. Martínez-Trndad References 1. D. Mche, D. J. Spegelhalter, and C. C. Taylor: Machne Learnng, Neural and Statcal Classfcaton. New York: Ells Horwood, So Young Sohn: Meta Analyss of Classfcaton Algorthms for Pattern Recognton. IEEE Trans. on PAMI,1,11,Noveber 1999, T.K. Ho, M. Basu: Complexty measures of supervsed classfcaton problem. IEEE Trans. on PAMI, 4, 3, March 00, Ester Bernadó Manslla, Tn Kam Ho: On Classfer Domans of Competence. ICPR (1) 004: C.L. Blake, C.J. Merz: UCI Repostory of machne learnng databases. [ Irvne, CA: Unversty of Calforna, Department of nformaton and Computer Scence. 6. Weka: Data Mnng Software n Java. [