Classification of binary vectors by using DSC distance to minimize stochastic complexity

Transcription

1 Patter Recogitio Letters 24 (2003) Classificatio of biary vectors by usig DSC distace to miimize stochastic complexity Pasi Fr ati *, Matao Xu, Ismo K arkk aie Departmet of Computer Sciece, Uiversity of Joesuu, P.O. Box 111, Fi Joesuu, Filad Received 17 October 2001; received i revised form 28 February 2002 Abstract Stochastic complexity (SC) has bee employed as a cost fuctio for solvig biary clusterig problem usig Shao code legth (CL distace) as the distace fuctio. The CL distace, however, is defied for a give static clusterig oly, ad it does ot take ito accout of the chages i the class distributio durig the clusterig process. We propose a ew DSC distace fuctio, which is derived directly from the differece of the cost fuctio value before ad after the classificatio. The effect of the ew distace fuctio is demostrated by implemetig it with two clusterig algorithms. Ó 2002 Elsevier Sciece B.V. All rights reserved. Keywords: Clusterig; Stochastic complexity; Algorithms; Distace fuctios 1. Itroductio Biary vector classificatio has bee widely used i DNA computig ad huma chromosome study, ad i solvig taxoomy problems from biomedical area. Statistical models are usually applied to describe ad solve predictio ad taxoomy problems. For example, Rissae (1987, 1996) has itroduced a model kow as stochastic complexity (SC), which is a extesible explaatio for Shao iformatio theory (Kotkae et al., 1999). To be a cost fuctio of classificatio, SC eeds to be approximated by a simple model (Rissae, * Correspodig author. Tel.: ; fax: address: frati@cs.joesuu.fi (P. Fr ati). 1987). Gylleberg et al. (1994, 1997, 2000) has give a simple ad practical approximatio of SC for biary vector classificatios. Thereafter, SC has bee employed as a geeric evaluatio fuctio i solvig biary clusterig problems as follows. The clusterig problem is first formulated as a optimizatio problem. Approximatio solutios are the foud for every reasoable umber of groups. SC is applied for measurig the goodess of the various clusterig results. Idividual clusterig ca be geerated usig ay algorithms such as the geeralized Lloyd algorithm (GLA) (Lide et al., 1980), ad the radomized local search (RLS) Fr ati ad Kivij arvi, A better idea is to itegrate the SC cost fuctio directly i the clusterig algorithm as doe i Gylleberg et al. (1997) ad Fr ati et al. (2000). The vector-to-cluster distace for the classificatio /03/$ - see frot matter Ó 2002 Elsevier Sciece B.V. All rights reserved. PII: S (02)

2 66 P. Fr ati et al. / Patter Recogitio Letters 24 (2003) of the vectors must the be re-defied correspodigly. Euclidea distace (L 2 -orm) provides the optimal classificatio of the data vectors for the miimizatio of the MSE, but ot for the SC. The optimal classificatio for SC is give by the Shao code legth (CL) fuctio (Gylleberg et al., 1997). It represets the etropy of the biary vector whe coded by the probability model of the particular cluster. Surprisigly, the CL distace itroduces a ew problem that ever arises with the L 2 -distace. This is illustrated i Fig. 1, i which we classify the black poit accordig the two existig clusters. The probability distributio of the leftmost cluster idicates the poit belogs to this class with a low probability. Nevertheless, the probability distributio of the rightmost cluster have zero variace i the horizotal dimesio ðr x ¼ 0Þ resultig i zero probability ad ifiite etropy. As a cosequece, the poit will be classified to the leftmost cluster. This ifiite etropy problem happes ofte i the classificatio of multi-dimesioal biary data vectors. It has therefore bee ecessary to make modificatios to the existig clusterig algorithms whe SC has bee applied as a cost fuctio. Previously, the problem has bee solved i Gylleberg et al. (1997) ad Fr ati et al. (2000) by applyig the clusterig algorithms first usig the sub-optimal but less problematic L 2 -distace. The CL distace is the applied i the last stage of the algorithm whe the global clusterig structure has already settled dow ad oly fie-tuig of the solutio takes place. The drawback of this approach is that similar patch should be made for every clusterig algorithm that is to be applied with the SC. I this paper, we propose a more geeral solutio to the ifiity problem by proposig a ew Fig. 1. Illustrative example of the problem i the CL distace. DSC distace fuctio. The distace fuctio is derived directly from the differece of the cost fuctio value before ad after the classificatio. It therefore implicitly takes ito accout the chage i the class distributio caused by the re-classificatio of the data vector, ad i this way, avoids the ifiity problem. The DSC is geeral i the sese that it applies to ay clusterig algorithm ad o more patches are therefore eeded. The effect of the ew distace fuctio is demostrated by implemetig it with two clusterig algorithms. The rest of the paper is orgaized as follows. I Sectio 2, we defie the clusterig problem of biary vectors, ad give the simplified formalizatio of the SC. The SC fuctio is the applied withi two clusterig algorithms as the cost fuctio, ad the CL distace is employed i the RLS ad GLA algorithms as a practical vector-to-cluster distace. I Sectio 3, we itroduce the ew DSC fuctio derived from the SC differece of the old ad ew classificatio whe a data vector is moved from oe class to aother. I Sectio 4, we make performace comparisos of the differet variats icludig the RLS ad GLA algorithms, ad the DSC, CL ad L 2 distace fuctios. 2. Clusterig by miimizig SC We use the followig otatios: N: umber of data vectors, M: umber of groups, D: dimesio of vectors, X: set of N data objects X ¼fx 1 ; x 2 ;...; x N g, P: partitio idices of x i : P ¼fp i ji ¼ 1;...; Ng, C: set of cluster cetroids C ¼fc j jj ¼ 1;...; Mg. The goal of the clusterig is to partitio a give set of N data vectors ito a umber of groups so that a give cost fuctio is miimized. I the clusterig process, we must solve both the umber of clusters (M) ad their locatio (c i ). The clusterig result is described by the partitio (P) of the data set by givig for each vector (x i ) the cluster idex (p i ) of the group, which it belogs to. We

3 P. Fr ati et al. / Patter Recogitio Letters 24 (2003) cosider a set of d-dimesioal biary data vectors Stochastic complexity SC ca be applied to the clusterig by fidig the miimum descriptio of the data via usig a clusterig model. SC measures the iformatio cotet of the data, ad it is defied as the shortest possible code legth for the data obtaiable by usig a set of class distributios. SC icludes both the model parameters ad the codig of the data i the measuremet. Suppose that we have classified the data vectors ito M groups described by the partitio of the data. The model of the class j ca the be described by the probability distributio withi the class i each dimesio: c ij ¼ ij = j ð1þ where j is the umber of biary vectors i the class j, ad ij is the umber of vectors havig the ith coordiate value 1. The probability vector c j of the class j is also the cetroid (average vector) of the cluster. The simplified approximatio of the SC fuctio i Gylleberg et al. (1997) ca be described usig the class distributio models as: SC ¼ XM X d j h ij þ XM j log j j N j¼1 þ d 2 X M j 1 j¼1 log maxð1; j Þ ð2þ where h measures the etropy of a biary distributio hðpþ ¼ p logðpþ ð1 pþlogð1 pþ ð3þ Sice every vector is classified to some group, it is kow that P j ¼ N. Moreover, N log N i the middle term is a costat ad, therefore, the equatio ca be simplified as: SC XM X d j h ij þ XM j log j þ d j¼1 j 2 j¼1 However, the simplified Eq. (4) above could make SC egative. The first part of the SC fuctio measures the itra-class iformatio as the code legth whe every data vector is coded accordig to the class probability model. The code legth is calculated by multiplyig the umber of vectors i each cluster ( j ) by the average etropy (h) ofthe cluster. The secod part measures the iter-class iformatio as the code legth of the partitio. It ca be calculated by the umber of vectors i each cluster ( j ) multiplied by the average etropy of the correspodig cluster idex. The third part measures the iformatio of the model as the code legth of the class distributio whe described by a series umbers betwee ½1:: j Š Clusterig algorithm The SC ca be applied for the clusterig problem as follows. We first fid approximatio solutios for every reasoable umber of groups usig ay clusterig algorithm. The solutios are the evaluated, ad the oe that miimizes the SC is the fial result of the clusterig. This search strategy ca use ay clusterig algorithm to fid the idividual solutios. I the followig, we recall two clusterig algorithms: the GLA by Lide et al. (1980), ad the RLS by Fr ati ad Kivij arvi (2000). The pseudocode of the GLA is show i Fig. 2. The algorithm takes ay iitial solutio (here the partitio P) as a iput, ad iteratively fie-tues the solutio by repeatig two operatios i tur. The first operatio calculates the cetroids of the clusters, ad the secod operatio re-partitio the data vectors accordig to the ew set of cetroids. XM j¼1 log maxð1; j Þ ð4þ Fig. 2. Pseudocode for the GLA.

4 68 P. Fr ati et al. / Patter Recogitio Letters 24 (2003) Fig. 3. Pseudocode for the RLS. The algorithm is iterated util o more improvemet appears i the solutio. The method is simple to implemet, ad has bee widely used for the clusterig problem as such, or itegrated with more complicated methods. The pseudocode of the RLS is show i Fig. 3. The method takes ay iitial solutio, which is the improved by a sequece of operatios. At each iteratio phase, the algorithm creates a ew cadidate solutio by makig a small chage to the curret clusterig structure. First, a radomly chose cluster cetroid (c j ) is replaced by a radomly chose data vector (x i ). This moves the cluster locatio to aother part of the vector space. The partitio is the adjusted by a local repartitio operatio, which cosists of the two steps as show i Fig. 4. I the first step, the old cluster is removed by re-partitioig its data vectors to other clusters. I the secod step, the ewly created cluster is populated by attractig data vectors from the eighborig clusters. The modified clusterig is fie-tued by the applicatio of the GLA. The ew cadidate solutio is the evaluated ad accepted oly if it improves the previous solutio. Fig. 4. Pseudocode for the local repartitio operatio. Otherwise, the cadidate solutio is discarded ad the previous solutio remais as the startig poit for the ext iteratio. The GLA ad the RLS are both applicable for the clusterig task ad also rather simple to implemet. The RLS is less sesitive to the iitializatio because it is capable of makig global chages i the clusterig structure (by radom swappig of the clusters), ad therefore, correct the icorrect settlemet of the iitial clusterig. If the GLA is to be used, it should be repeated several times i order to reduce the depedecy o the iitializatio Shao code-legth distace The clusterig algorithms employ a distace fuctio d, which measures the vector-to-cluster distace, ad is used for the classificatio of the vectors durig the clusterig process. Usually the distace fuctio is defied as the Euclidea distace (L 2 -orm) betwee the data vector x i ad the particular cluster cetroid c j : vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ux d d E ðx i ; c j Þ¼t kx ik c jk k 2 ð5þ k¼1 This gives optimal classificatio for the miimizatio of the MSE but ot for the SC. The optimal classificatio for the SC is give by the Shao code-legth fuctio CLðx i ; c j Þ i Gylleberg et al. (1997). d CL ðx i ; c j Þ¼ Xd ð1 x i Þ logð1 c ij Þþx i log c ij log j ð6þ N It measures the code legth whe the data vector (the summatio term i the equatio) ad its class idex (secod term i the equatio) are coded usig the give model. I priciple, the CL distace is well defied, but i practice, it has a fudametal problem i its defiitio, which will be explaied by the followig example. Cosider a sigle cluster c 1 cosistig of the followig three data vectors: x 1 ¼ð0; 0; 0Þ, x 2 ¼ ð0; 1; 0Þ ad x 3 ¼ð1; 0; 0Þ. The correspodig class

5 P. Fr ati et al. / Patter Recogitio Letters 24 (2003) probability distributio of the cluster is c 1 ¼ð0:33; 0:33; 0:00Þ. The distaces of the vectors ca ow be calculated usig the CL distace: d CL ðx 1 ; c 1 Þ¼0:58 þ 0:58 þ 0:00 ¼ 1:17 d CL ðx 2 ; c 1 Þ¼0:58 þ 1:58 þ 0:00 ¼ 2:17 d CL ðx 3 ; c 1 Þ¼1:58 þ 0:58 þ 0:00 ¼ 2:17 The secod term of Eq. (6) is omitted i this example for simplicity. Let us the cosider a fourth vector x 4 ¼ð0; 0; 1Þ, which is equally close to the cluster accordig to the Euclidea distace, but the CL distace gives the followig result: d CL ðx 4 ; c 1 Þ¼0:58 þ 0:58 þ1¼udefied The problem ca appear whe there is a uiform bit distributio i ay dimesio, ad the data vector has differet value i the same positio. The homogeous bit distributio idicates that there is o ucertaity ad the etropy of the cotradictig value would therefore approach ifiite. This is a serious flaw especially i the local re-partitio procedure of the RLS. It creates ew clusters startig from a sigular cluster, which evidetly has uiform bit distributio. As a cosequece, o other data vectors (except equal oes) ca ever be classified to this cluster. The problem of the CL distace is that eve though it measures the ucertaity of the classificatio, it does ot take ito accout the ucertaity of the model itself. I other words, the bit distributio of the class model is ideed homogeeous, but the model is oly a approximatio ad subject to chage durig the clusterig process. Zero-probability is therefore ot a feasible approximatio of the classificatio. The ifiite values could be avoided by prevetig the cetroids to take values 0 ad 1. This ca be achieved, for example, give biary data vector x, by takig the cetroid values to be mea vector of x ad c j. If some coordiate of c j equals to 0 or 1 value, the umber of vectors i jth cluster, j ca be take as a parameter of CL distace fuctio. Obviously, c j ca be replaced with a ew vector, which is the cetroid of jth cluster after vector x is put ito cluster j. c ij ¼ jc ij þ x i j þ 1 c ij 6¼ x i ð7þ If c j ¼ x, CL distace is adopted as log 2 ðnþ. Aother coditio o CL distace value is cosidered as follows: x i log c ij þð1 x i Þ logð1 c ij Þ¼0 c ij ¼ x i ; c j 6¼ x ð8þ This patch, however, does ot remove the problem itself as it merely assigs a low probability istead of a zero value. Additioal modificatios have therefore bee ecessary for the clusterig algorithms so that the CL distace could have bee used properly i the GLA ad i the RLS. For example, i the algorithms preseted i Gylleberg et al. (1997) ad Fr ati et al. (2000) the CL distace is applied oly i the last step of the clusterig process whe the global clusterig structure has already settled dow, ad oly fie-tuig of the solutio takes place. The problem of this approach is that it is ot trivial to determie the stage of the clusterig process, whe it would be safe eough to start to use the CL distace. To sum up, the problem with the CL distace is fudametal i its ature. It is therefore better to fix it tha to fid patch for every clusterig algorithm that is to be applied. 3. DSC distace fuctio We itroduce a ew vector-to-cluster distace fuctio deoted as DSC distace. It is based o a desig paradigm, i which the distace fuctio is derived directly from the differece of the cost fuctio value before ad after the classificatio of a data vector. The mai advatage of this desig philosophy is that it implicitly takes ito accout of the chages i the clusterig model caused by the classificatio. It is also geeral i the sese that it does ot deped o the chose clusterig algorithm ad should therefore be applicable with ay distace-based clusterig method. The DSC distace fuctio is always defied relative to a give model. We ca therefore assume that we have a model, for which we ca calculate the SC value. If we the cosider the distace

6 70 P. Fr ati et al. / Patter Recogitio Letters 24 (2003) calculatio as a movemet of the data vector from oe group to aother, we ca defie the distace fuctio as the differece i the SC of the clusterig before ad after the movemet of the data vector. Give two classes j 1, j 2 ad a biary vector x, which we cosider to move from the class j 1 to class j 2, the SC fuctio i (2) value after the movemet is: SC XM X d j j6¼j 1 ;j 2 þ d 2 X M j6¼j 1 ;j 2 h ij j þ X j6¼j 1 ;j 2 ð j log 2ð j ÞÞ log maxð1; j Þ ð j1 1Þ logð j1 1Þ ð j2 þ 1Þ logð j2 þ 1Þ þ Xd 1Þh x i ð j1 j1 1 þð j2 þ 1Þh þ x i j2 þ 1 þ d 2 ðlog maxð1; j 1 1Þ þ log maxð1; j2 þ 1ÞÞ þ N log N ð9þ We ca the calculate the differece betwee the SC fuctio values of the old clusterig (before the movemet) ad the ew oe (after the movemet) as: SC-diffðx; j 1 ; j 2 Þ¼ Xd j1 1 h x i j1 1 j1 h j1 þð j2 þ 1Þh þ x i j2 þ 1 j2 h þð j1 d=2þ j2 log j1 þð j2 d=2þ log j2 ð j2 þ 1 d=2þ logð j2 þ 1Þ ð j1 1 d=2þ logð j1 1Þ j1 > 1 SC-diffðx; j 1 ; j 2 Þ¼ Xd j2 þ 1 h x i j2 1 j2 h j2 þð j2 d=2þ log j2 þðd=2 j2 1Þ logð j2 þ 1Þ j1 ¼ 1 ð10þ The SC-diff takes zero value if j 1 ¼ j 2. Negative values are obtaied whe the movemet of the vector improves the solutio, ad positive values otherwise. The SC-diff could ow be applied as such i the cases whe we re-classify a vector i a existig solutio. I the SC-diff fuctio we assume that the give vector is already classified ito some class. I geeral, however, this is ot the case but we must be able to defie a more geeral distace fuctio that depeds oly o the vector x i ad o the cadidate cluster c j. For example, i the repartitio procedure of the RLS algorithm, we classify vectors whose previous class has bee removed. More geeral DSC fuctio ca be derived from (10) as follows. The classificatio ca be cosidered as a twostep procedure, i which we first remove the vector x i from the class j 1 ad the add it to the class j 2 For a give vector x i the cost of the removal is costat. This meas that the parameters N, j1, ij1 are fixed i the classificatio, ad as a cosequece, we ca cosider oly the cost of addig the vector i class j 2 ad igore the removal part i the formula. Thus, the DSC ðj 1 6¼ j 2 Þ ca be defied merely as the cost of the additio DSCðx; C j2 Þ¼ Xd j2 þ 1 h þ x i j2 þ 1 j2 h j2 þð j2 d=2þ log j2 þðd=2 j2 1Þ logð j2 þ 1Þ þ log N ð11þ

7 P. Fr ati et al. / Patter Recogitio Letters 24 (2003) This gives the same result as the SC-diff with the differece of a costat. The oly exceptio is whe we measure the distace of x i to the cluster, i which it is already icluded ðj 1 ¼ j 2 Þ. I this case, we should use ð j1 1Þ as the class size istead of j1 because the class size does ot icrease due to the classificatio. Thus, if the previous classificatio is kow, we should apply the followig equatio for this special case: DSCðx; C j2 Þ¼ Xd j1 h j1 ð j1 1Þh x i j1 1 Fig. 5. Clusterig results by the RLS algorithm for DNA-1. þðd=2 j1 Þ log j1 þð j1 1 d=2þ logð j1 1Þ þ log N j 1 ¼ j 2 ; j1 > 1 DSCðx; C j2 Þ¼log N j 1 ¼ j 2 ; j1 ¼ 1 ð12þ Hece, DSC distace as i Eq. (11) is applicable as vector-to-cluster distace i all cases, although it uderestimates the distace i the case whe the vector is already icluded i the class. The special case of (12) should therefore be used whe applicable to give more exact value. 4. Test results We use three biary data sets to test the ew method: DNA-1, DNA-2, Normal. The features of the first two sets (DNA-1 ad DNA-2) were extracted from aalysis of DNA samples of fishes (presece or absece of give DNA fragmet) i biological research experimets. There are dimesioal biary vectors i (DNA-1) ad dimesioal vectors i DNA-2. The third set (Normal) was artificially created by geeratig 265 biary vectors ito 10-dimesioal vector space with 12 clusters. We study first the DNA-1 ad DNA-2 sets whe clustered usig the RLS ad GLA methods. The RLS was performed 80 iteratios. The results i Figs. 5 8 show that the DSC distace ad L 2 -distace Fig. 6. Clusterig results by the GLA algorithm for DNA-1. Fig. 7. Clusterig results by the RLS algorithm for DNA-2.

8 72 P. Fr ati et al. / Patter Recogitio Letters 24 (2003) Table 1 summarizes the clusterig ad classificatio results for the Normal data set. It is the oly data set for which the real classificatio is kow, ad thus, classificatio rate could be calculated. The results show that employig RLS by DSC distace gives the best performaces both i terms of best clusterig result (smallest SC values), ad the highest classificatio rate. The RLS algorithm foud the correct umber of clusters also with the L 2 -distace ad CL distace but the correspodig classificatio rates were smaller. The GLA, however, was able to fid the correct result (12 clusters) oly by usig the DSC distace. Fig. 8. Clusterig results by the GLA algorithm for DNA-2. come up with much better results tha CL distace i RLS algorithm. It seems that there is o big differece betwee DSC distace ad L 2 - distace whe they are employed i RLS. The differece, however, ca be sigificat whe the correct umber is to be determied i the stepwise search. The result with the GLA is quite differet from that of the RLS, maily because the variace of the results is much greater. The CL distace still performs worse tha the L 2 -distace, but the DSC distace is ow clearly better tha the L 2 -distace almost with respect to every umber of clusters. It is expected that the correct result would be reached more reliably usig the DSC distace. The drawback of the DSC distace is takes much more time to compute tha the L 2 -distace. 5. Coclusios We proposed a ew vector-to-cluster distace i the classificatio problems of biary vectors by miimizig SC. The distace fuctio was applied both i the GLA ad RLS algorithms. Experimets show that RLS by DSC distace gives the best clusterig performace i miimizig SC amog all the variats cosidered, ad the highest classificatio rate. I most cases, the modified CL distace performs eve worse tha the L 2 -distace. It is somehow difficult for the stepwise GLA to deliver satisfactory results i solvig the correct umber of clusters, eve by usig the DSC distace. The L 2 -distace is moderately effective to classify simple data. Amog the three distaces, the DSC distace is the most precise to miimize stochastic complexity. Our approach by usig DSC distace is geeral i its ature as the same desig paradigm ca be applied with ay other cost fuctio too. Table 1 The classificatio results of the RLS ad GLA algorithms with the L 2 -distace, CL-distace ad DSC-distace RLS algorithm GLA algorithm Real classificatio L 2 CL DSC L 2 CL DSC SC Number of a 6 a clusters Classificatio rate 96.98% 92.83% 98.87% 74.34% 87.55% 91.32% 100% a The classificatio rates are calculated from the clusterig of 12 clusters eve though smaller SC-value was foud with 6 clusters.

9 P. Fr ati et al. / Patter Recogitio Letters 24 (2003) Refereces Fr ati, P., Kivij arvi, J., Radomized local search algorithm for the clusterig problem. Patter Aalysis ad Applicatios 3 (4), Fr ati, P., Gylleberg, H.G., Gylleberg, M., Kivij arvi, J., Koski, T., Lud, T., Nevalaie, O., Miimizatio stochastic complexity usig GLA ad local search with applicatios to classificatios of bacteria. Biosystems 57 (1), Gylleberg, M., Koski, T., Probabilistic Models for Bacterial Taxoomy, TUCS Techical Report, No. 325, Turku Ceter for Computer Sciece, Filad, February Gylleberg, M., Koski, M., Verlaa, M., Clusterig ad quatizatio of biary vectors with stochastic complexity. Proc. IEEE Iterat. Symposium o Iformatio Theory, Trodheim, Germay, Gylleberg, M., Koski, T., Verlaa, M., Classificatio of biary vectors by stochastic complexity. Joural of Multivariate Aalysis 63, Kotkae, P., Myllym aki, P., Silader, T., Tirri, H., O the accuracy of stochastic complexity approximatios. I: Gammerma, A. (Ed.), Causal Models ad Itelliget Data Maagemet, Chapter 9. Spriger-Verlag. Lide, Y., Buzo, A., Gray, R., A algorithm for vector quatizer desig. IEEE Tras. Commuicatios 28 (1), Rissae, J., Stochastic Complexity ad the MDL Priciple. Ecoometric Reviews 6, Rissae, J., Stochastic complexity. Joural of Statistics Society 4 (10), Rissae, J., Fisher iformatio ad stochastic complexity. IEEE Tras. o Iformatio Theory 42 (1),