Fuzzy clustering ensemble based on mutual information

Transcription

1 Poceedngs of the 6th WSEAS Intenatonal Confeence on Smulaton, Modellng and Optmzaton, Lsbon, Potugal, Septembe 22-24, Fuzz clusteng ensemble based on mutual nfomaton YAN GAO SHIWEN GU LIMING XIA ZHINING NIAO 2 Facult of Infomaton Scence and Engneeng, Cental South Unvest 475, Hunan, P.R.Chna Chna 2 Depatment of Compute Scence, Loughboough Unvest Lecs, UK, LE3TU {gaoan,swgu,xlm}@csu.edu.cn {laozn98}@ahoo.com Abstact: -Clusteng ensemble s a new topc n machne leanng. It can fnd a combned clusteng wth bette ualt fom multple pattons. But how to fnd the combned clusteng s a dffcult poblem. In ths pape, we extend the object functon poposed b Stehl & Ghosh whch s based on mutual nfomaton and we pesent a new algothm smla to nfomaton bottleneck to solve the object functon. Ths algothm can combne soft pattons and need not establsh label coespondence between dffeent pattons. We conducted expements on fou eal-wold data sets to compae ou algothm wth othe fve ensemble algothms, ncludng,,,. The esults ndcate that ou algothm povdes solutons of mpoved ualt. Ke-Wods: - Clusteng Ensemble, Mutual Infomaton, soft pattons Intoducton Clusteng s the poblem of pattonng a fnte set of ponts n a mult-dmensonal space nto classes (called clustes) so that () the ponts belongng to the same class ae smla and () the ponts belongng to dffeent classes ae dssmla. Clusteng has been extensvel studed n machne leanng, databases, and statstcs fom vaous pespectves. Man applcatons of clusteng have been dscussed and man clusteng technues have been developed. Ensemble leanng efes to a collecton of methods that lean a taget functon b tanng a numbe of ndvdual leanes and combnng the pedctons. In ensemble leanng, a moe elable esult can be obtaned b combnng the output of multple expets, and a complex poblem can be decomposed nto multple sub-poblems that ae ease to undestand and solve (dvde-and-conue appoach). Ensemble leanng s a hot topc n machne leanng, and s egaded as one of fou man dectons n machne leanng []. Most famous ensemble leanng methods s desgned fo supevsed leanng, such as boostng [2], baggng [3]. Recentl, people focus on usng ensemble leanng to mpove pefomance of clusteng. Fed & Jan [4], Fen & Bodle [5], Mont et al [6] establshed the co-assocaton matx based on smlates between dffeent clusteng solutons, and then use agglomeatve heachcal clusteng. Topch et al. [7][8] poposed a mxtue model n ode to obtan a consensus functon. The basc dea s to consde the labels of the ndvdual pattons as featues chaactezng the objects and the consensus patton s obtaned b goupng ths data set. The also establshed a uadatc mutual nfomaton cteon fo clusteng ensemble and the appoxmate esults fo ths cteon can be obtaned b unnng k-means [8]. W. Tang and Z.H.Zhou [9] poposed baggng-based selectve cluste ensemble algothm n whch the mutual nfomaton between the clusteng esult and othe esults can be egad as the weght of clusteng esult n baggng. Fossnots [] appled boostng to clusteng ensemble. Stehl & Ghosh[] poposed thee dffeent appoaches to geneatng consensus functons, most of them based on hpegaph pattonng. The also ponted out that clusteng ensemble can be egaded as the optmal poblem based on mutual nfomaton, but not pont out how to solve t and t s onl to combne had patton. In ths pape, we extend the object functon based on mutual nfomaton ntoduced b Stehl & Ghosh, and popose a new ensemble algothm to combne soft pattons. Ths algothm need not consde the poblem of label coespondence between dffeent clusteng esults. 2 Clusteng Ensemble Defnton (Clusteng ensemble): Gven a data set of n nstances X = {X, X 2,,X n }, a set of pattons poduced b base clusteng algothm on ths data set can be epesented b Π={,, }, whee ={c, c k }, U k c k =X. Clusteng ensemble s to deduce the fnal patton fom ths set of pattons Π.

2 Poceedngs of the 6th WSEAS Intenatonal Confeence on Smulaton, Modellng and Optmzaton, Lsbon, Potugal, Septembe 22-24, Fo clusteng ensemble, thee ae two mpotant components: ensemble constucto and consensus functon. Gven a dataset, ensemble constucto geneates a set of dvese pattons. Dvest guaantees that all the ndvdual leanes do not make the same eos. Consensus functon s a good method to combne the dffeent pattons and poduce a sngle bette patton on the data set. So choosng the appopate consensus functon s the ke poblem fo clusteng ensemble. In ths pape, we poposed a new consensus functon: ensemble algothm based on mutual nfomaton. 3 The object functon based on mutual nfomaton fo clusteng ensemble In [], Stehl and Ghosh thought that combned clusteng should shae the most nfomaton wth the ognal clustengs: Π={,. }. But how do we measue shaed nfomaton between clustengs? Stehl and Ghosh used mutual nfomaton to measue t. In nfomaton theo, mutual nfomaton s a smmetc measue to uantf the statstcal nfomaton shaed between two dstbutons. Suppose thee ae two pattons: a and b. Let n h s the numbe of nstance whch label s C l n patton b, n l s the numbe of nstance whch label s C h n patton a, n l h s the numbe of nstance whch label not onl s C l n patton b but also s C h n patton a. n s the total numbe of nstances n data set. In [], Stehl & Ghosh used the nomalzed mutual nfomaton. So the object functon s: K K h NMI 2 h n ln φ (, ) = n l log 2 ( ) k h l n k= k = n n opt NMI = ag max φ (, ) = () Although Stehl & Ghosh poposed the object functon based on mutual nfomaton fo ensemble clusteng, the also ponted out that fo fnte populatons, the tval soluton s to exhaustvel seach though all possble clustengs wth k labels (appoxmatel k n /k! fo n>> k) fo the one wth the maxmum ANMI whch s computatonall pohbtve. And ths object functon s onl appled to combne the had pattons. We also thought that clusteng ensemble should extact the combned clusteng shang the most nfomaton wth the ognal clustengs. But n ths pape, we modfed the object functon to cluste soft pattons. Fst, we thnk that extactng clustes stuctue fom the data can be vewed as data compesson. So besdes pesevng moe nfomaton of ognal clustengs, the data should be compessed as much as possble. In the nfomaton theo [2], compessng data means mnmzng the mutual nfomaton between the clusteng and data. Fgue depcts ou ensemble model. X 2... a. The compessed nfomaton b. The peseved nfomaton Fgue. The compessed nfomaton and the peseved nfomaton fo clusteng ensemble So the new object functon can be avalable b subtactng an tem fom ognal object functon (): η I(X, ). Fo computng ths functon convenentl, we eplaced the nomalzed mutual nfomaton n () wth standad mutual nfomaton. opt I( = ag max = I(, ) ηi(x; ) p(c,cj ) p(c,cj )log p(c )p(c ) ( 2 ) k k, ) = ( 3 ) = j= j Second, we extended the object functon to combnng soft pattons. Suppose thee ae a set of pattons on data set: Π={, }, whee ={c, c k }. Fo an two patton a, b, the nfomaton of the cluste c a n patton a should be tansfeed to cluste c b j n patton b though x (c b j ->x->c a ). So the p(c a,c b j ) n (3) can be defned as: p (c, c j ) = x p(c x)p(c j x)p(x) ( 4 ) a a p(c ) = p(c x)p(x) x

3 Poceedngs of the 6th WSEAS Intenatonal Confeence on Smulaton, Modellng and Optmzaton, Lsbon, Potugal, Septembe 22-24, p(c ) = p(c b b x)p(x) x ( 5 ) 4 The soluton fo object functon Supposng that the labels of ognal clusteng ae used to the new featue fo object, labels n eve patton can fom the new featue space. So eve object can be epesented b featue spaces. If we use to epesent the label n patton and use c to epesent label n the combned clusteng. The Lagangan functon fo (2) s: I(, ) ηi(x; ) (x) ( p(c x)) = λ = x c p(c x) (6) Poposton: The local optmal soluton of (3) can be obtaned b teatng the followng euatons: p t (c) p t(c x) = exp( βd[p( x) pt ( c)]) Z = p t(c) =x p(x)p t(c x) p t( c) = x X p( x)p t(c x)p(x) p t(c) ( 7 ) whee: Z = c pt ( c)exp( βd = [ p( x) p t ( c)]), β =, D s elatve entop of p wth espect to, η t s also called Kullback Lebles dstance, p(x) D (p ) = x p(x) log (x). Poof: Supposng that the labels n patton ae pesented b, the label n combned patton s epesented b c. The mutual nfomaton between and s: I( ; ) = c p(,c) p(,c)log p( ) Consdeng the nfomaton about must tansfe to c though x, a Makov chan can consst of, x and c: ->x->c. So: p(,c) = x p( x)p(c x)p(x) Fo I( ; ), the patal devatve wth espect to the vaable p(c x) s: I( ; ) p(c x) = p(t = p(x) + p(x) x) = p(x) I(X; ) = p( x)p(c x)p(x) log c x p( ) p( p( D D x) log [p( [p( xc p( c)p( x) x) p( x)p( ) x) p( p( )] c)] p(c x) p(c x)p(x)log c) (8) So the patal devatve of I(X; ) wth espect to the vaable p(c x) s: I(X; ) p(c x) = p(c x) x c p(c p(c x) x)p(x) log p(c x) = p(x)log (9) So accodng to (8), (9), the patal devatve of the functon (6) wth espect to the vaable p(c x) s: p(c x) = p(x) λ(x,,... ) p(x)log p(c x) = β{p(x)d [p( x)] p( c)]} () whee: λ(x) λ( x,,..., ) = + βd[p( x) p( )] p(x) =. When () s eual to, the teatng euatons (7) can be obtaned, and functon (2) has local optmal soluton. 5 Algothm mplementaton Accodng to the above poposton, we knew that gven the ntal dstbuton, the local optmal soluton of (2) can be obtaned b computng (7) teatvel. When the value of η s ve small, the soluton of (2) s the appoxmate soluton of (). So n ou ensemble algothm, η s.. Obsevng (2) and (7), we found that when wee eual to, (2) s eual to the object functon of

4 Poceedngs of the 6th WSEAS Intenatonal Confeence on Smulaton, Modellng and Optmzaton, Lsbon, Potugal, Septembe 22-24, nfomaton bottleneck (IB) [3] and (7) became the teatng euatons fo IB. So the mplementaton of ou algothms s smla to IB. Thee ae man mplementaton algothms fo IB, such as seuence IB (sib), agglomeatve IB (aib), teatve IB (IB). Slonm [3] compaed seveal IB algothms, concludng that best had clusteng esults ae obtaned wth a seuental method (sib), n whch elements ae fst assgned to a fxed numbe of clustes and then ndvduall moved fom cluste to cluste whle calculatng a -step lookahead scoe, untl the scoe conveges. Hee we adopted sib. But we must make a lttle change n sib. Fo clusteng ensemble, we change the object functon and teatng euatons n sib accodng to (2), (7). 6 Expements Data set Table The detal of fou data sets Num. of featues Num. of classes Num. of objects Fuzzkmeans (eo ate) Wne Glass Ionosp hee Spambase We expement on fou data sets fom UCI [4]. The attbutes n fou data sets ae numbecal. The detal of fou data sets s descbed n Table. In expements, we use two ensemble algothms based on mutual nfomaton: and h-mi. s used to combne soft pattons and h-mi s used to combne had pattons. We compaed h-mi, wth othe ensemble algothms, ncludng,, and. fo pattonng of hpegaphs nduced fom the co-assocaton values. establshes a hpegaph fo cluste ensemble and pattons the hpegaph b cuttng a mnmal numbe of hpe-edges. modfes va extended set of hpeedge opeatons and addtonal heustcs, s based on uadatc mutual nfomaton cteon and use k-means to obtan the appoxmatel combned clusteng. The,, code s avalable n [5]. In ode to poduce dvese pattons, we used andom subspace method [6][7], whee each base clusteng s geneated on a andoml selected subset of the ognal dmensons. Fuzz k-means s used on new subspace to poduce a soft patton. In ode to obtanng the had pattons, eve object s assgned to the cluste n whch ts condtonal pobablt s maxmum. The maxmum teatve tme n fuzz k-means s. The dmenson of sub space s d / 4. The numbe of clustes fo eve data set: k, s the actual numbe of classes n data set. Ou algothm s susceptble to the pesence of local mnma of the objectve functons. To educe the sk of convegence to a lowe ualt soluton, we used a smple heustc affoded b low computatonal complextes of these algothms. The fnal patton was pcked fom the esults of thee uns (wth andom ntalzatons) accodng to the value of objectve functon. The hghest value of MI functon (2) seved as a cteon fo ou algothm. We andoml choose fve value [, 5, 2, 3, 4 ] fo the sze of cluste ensemble. In expements, the mean clusteng eo ate of clusteng ensemble pocedues s used to measue the pefomance of clusteng ensemble. Let C tue epesent the tue (gven) clusteng and C epesent the ensemble clusteng, Confuson epesent the confuson matx of two clustengs. Confuson(ktue, k) = (C tue ktue C k ),.e. numbe of ponts x that ae cluste ktue n tue clusteng and cluste k n the clusteng poduced, then eo _ ate = ( Confuson( ktue, k)) / n () n ktue k ktue Whee n s the total numbe of objects. Clusteng eo ate s defned as the numbe of msclassfcaton". The low value of eo ndcates good ualt of clusteng.fgue, 2, 3, 4 shows the mean eo ate on fou data sets. Fg. The mean eo ate fo wne data set

5 Poceedngs of the 6th WSEAS Intenatonal Confeence on Smulaton, Modellng and Optmzaton, Lsbon, Potugal, Septembe 22-24, mean e o at e ensembl e s ze Fg. 2 The mean eo ate fo glass data set mean eo ate ensembl e s ze Fg. 3 The mean eo ate fo spambase data set mean e o at e ensembl e s ze Fg. 4 The mean eo ate fo onosphee data set mean e o at e ensembl e s ze Fom Fgue, 2, 3, 4, we fnd that among the fou data sets, the pefomance of clusteng ensemble n wne s best. The second s spambase. When combnng the pattons n glass o n onosphee, the pefomance of the fve algothms ae all not good, the mean eo ate s hghe than that of fuzz k-means on whole dataset. Fom Fgue,2,3 4, we also fnd that thee s no algothm whch pefomance s best fo all fou data set wth dffeent ensemble s sze. But on the whole, the pefomance of s best among sx algothms. Especall when the sze of ensemble s small, the mean eo ate of s lowe than that of othe algothms, because ognal soft pattons contan much nfomaton than had pattons. In expements, although h-mi and ae both based on mutual nfomaton cteon fo had patton ensemble, h-mi povdes clustes wth bette ualt than. 7 Concluson In ths pape, we have poposed a clusteng ensemble method to combne soft pattons. Ths method extends the object functon based on mutual nfomaton poposed b Stehl & Ghosh. And the soluton of the new object functon can be obtaned b usng the algothm whch s smla to nfomaton bottleneck. Ths method s not onl to combne soft and had pattons, but also has an advantage that t need not establsh label coespondence between dffeent pattons. Expements on 4 data sets fom UCI ndcate that usng ou algothm to combne soft pattons can povde clustes wth bette ualt than,, and.. Refeences:

6 Poceedngs of the 6th WSEAS Intenatonal Confeence on Smulaton, Modellng and Optmzaton, Lsbon, Potugal, Septembe 22-24, [] Dettech TG, Machne leanng eseach: Fou cuent dectons, AI Magazne, 8 (4), 997, pp [2] Robet E. Shape, A bef ntoducton to boostng, In Poceedngs of the Sxteenth Intenatonal Jont Confeence on Atfcal Intellgence, Mogan Kaufmann, 999, pp [3] Beman, L, Baggng Pedctos, Machne Leanng, Vol. 24, No. 2, 996, pp [4] A.L.N. Fed and A.K. Jan, Data Clusteng Usng Evdence Accumulaton, In Poc. of the 6th Intenatonal Confeence on Patten Recognton, ICPR 22, Quebec Ct, pp [5] Fen, X. Z., & Bodle, C. E. Random pojecton fo hgh dmensonal data clusteng: A cluste ensemble appoach, ICML 23. [6] Mont, S. Tamao, P, Mesov, J, & Golub, T. Consensus clusteng: A esamplng-based method fo class dscove and vsualzaton of gene expesson mcoaa data, Machne Leanng, vol. 52, no. -2, 23, pp.9 8. [7] A. Topch, A. Jan, and W. Punch, A mxtue model fo clusteng ensembles, In Poc. SIAM Data Mnng, 24, pp [8] A. Topch, A. Jan, and W. Punch, Combnng multple weak clustengs, In Poc. Thd IEEE Intenatonal Confeence on Data Mnng (ICDM'3), Novembe 23. [9] We Tang, ZhHua Zhou, Baggng-Based Selectve Clustee Ensemble. Jounal of softwae, Vol.6, No.4, 25, pp [] D. Fossnots, A. Lkas, A. Staflopats, A clusteng method based on boostng, Patten Recognton Lettes 25 (24), pp [] A. Stehl and J. Ghosh, Cluste ensembles-a knowledge euse famewok fo combnng pattonngs, In Poc. Confeence on Atfcal Intellgence (AAAI 22), Edmonton, pp [2] T. M. Cove and J. A. Thomas, Elements of Infomaton Theo, John Wle&Sons,New Yok, 99. [3] Noam Slonm, The Infomaton Bottleneck: Theo and Applcatons, PhD thess, Hebew Unvest, Jeusalem, Isael, 22. [4] Blake C, Keogh E, Mez CJ, UCI Reposto of machne leanng databases, Ivne: Depatment of Infomaton and Compute Scence, Unvest of Calfona, uc.edu /~mlean/mlreposto.html [5],, code, [6] D. Geene, A. Tsmbal, N. Bolshakova, P. Cunnngham, Ensemble clusteng n medcal dagnostcs, n: R. Long et al. (Eds.), Poc. 7th IEEE Smp. on Compute-Based Medcal Sstems CBMS 24, Bethesda, MD, Natonal Lba of Medcne/Natonal Insttutes of Health, IEEE CS Pess, 24, pp [7] T. K. Ho,The andom subspace method fo constuctng decson foests, IEEE Tansactons on Patten Analss and Machne Intellgence, 2(8),998, pp