Supervised Learning in Parallel Universes Using Neighborgrams

Transcription

1 Supervsed Learnng n Parallel Unverses Usng Neghborgrams Bernd Wswedel and Mchael R. Berthold Department of Computer and Informaton Scence, Unversty of Konstanz, Germany Abstract. We present a supervsed method for Learnng n Parallel Unverses,.e. problems gven n multple descrptor spaces. The goal s the constructon of local models n ndvdual unverses and ther fuson to a superor global model that comprses all the avalable nformaton from the gven unverses. We employ a predctve clusterng approach usng Neghborgrams, a one-dmensonal data structure for the neghborhood of a sngle object n a unverse. We also present an ntutve vsualzaton, whch allows for nteractve model constructon and vsual comparson of cluster neghborhoods across unverses. 1 Introducton Computer-drven learnng technques are based on a sutable data-representaton of the objects beng analyzed. The classcal concepts and technques that are typcally appled are all based on the assumpton of one approprate unque representaton. Ths representaton, typcally a vector of numerc or nomnal attrbutes, s assumed to suffcently descrbe the underlyng objects. In many applcaton domans, however, varous dfferent descrptons for an object are avalable. These dfferent descrptor spaces for the same object doman typcally reflect dfferent characterstcs of the underlyng objects and as such often even have ther own, unque semantcs and can and should therefore not be merged nto one descrptor. As most classcal learnng technques are restrcted to learnng n exactly one descrptor space, the learnng n the presence of several object representatons s n practce often solved by ether reducng the analyss to one descrptor, gnorng the others; by constructng a jont descrptor space (whch s often mpossble); or by performng ndependent analyses on each space. All these strateges have lmtatons because they ether gnore or obscure the multple facets of the objects (gven by the descrptor spaces) or do not respect overlaps. Ths often makes them napproprate for practcal problems. As a result Learnng n Parallel Unverses [13] has emerged as a novel learnng scheme that encompasses the smultaneous analyss of all gven descrptor spaces,.e. unverses. It deals wth the concurrent generaton of local models for each unverse, whereby these models cover local structures that are unque for ndvdual unverses and at the same tme can also cover other structures that span multple unverses. The resultng global model outperforms the above mentoned schemes and often also provdes new nsghts wth regard to overlappng as well as unverse-specfc structures. Although the learnng task tself can be both supervsed (e.g. buldng a classfer) and J. Gama, E. Bradley, and J. Hollmén (Eds.): IDA 2011, LNCS 7014, pp , c Sprnger-Verlag Berln Hedelberg 2011

2 Supervsed Learnng n Parallel Unverses Usng Neghborgrams 389 unsupervsed (e.g. clusterng), we concentrate n the followng on supervsed problem scenaros. In ths paper we dscuss an extenson to the Neghborgram algorthm [4,5] for learnng n parallel unverses. It s a supervsed learnng method that has been desgned to model small or medum szed data sets or to model a (set of) mnorty class(es). The latter s commonly encountered n the context of actvty predcton for drug dscovery. The key dea s to represent an object s neghborhoods, whch are gven by the varous smlarty measures n the dfferent unverses, by usng neghborhood dagrams (so-called neghborgrams). The learnng algorthm derves cluster canddates from each neghborgram and ranks these based on ther qualtes (e.g. coverage). The model constructon s carred out n a sequental coverng-lke manner,.e. startng wth all neghborgrams and ther cluster canddates, takng the numercally best one, addng t as cluster and proceedng wth the remanng neghborgrams whle gnorng the already covered objects. We wll present and dscuss extensons to ths algorthm that reward clusters whch group n dfferent unverses smultaneously, and thus respect overlaps. As we wll see ths can sgnfcantly mprove the classfcaton accuracy over just consderng the best cluster at a tme. Another nterestng usage scenaro of the neghborgram data structure s the possblty of dsplayng them and thus nvolvng the user n the learnng process. Usng the vsualzaton technques descrbed n [4] n a grd vew, whch algns the dfferent unverses column by column, the user can nspect the dfferent neghborhoods across the avalable unverses and assess possble structural overlaps. 2 Related Work Subspace Clusterng. Subspace clusterng methods [11] operate on a sngle, typcally very hgh-dmensonal nput space. The goal s to dentfy regons of the nput space (a set of features), whch exhbt a hgh smlarty on a subset of the objects, or, more precsely, the objects data representatons. Subspace clusterng methods are commonly categorzed nto bottom-up and top-down approaches. Bottom-up starts by consderng densty on ndvdual features and then subsequently mergng features/subspaces to subspaces of hgher dmensonalty, whch stll retan a hgh densty. The most promnent example s CLIQUE [2], whch parttons the nput space usng a statc grd and then merges grd elements f they meet a gven densty constrant. Top-down technques ntally consder the entre feature space and remove rrelevant features from clusters to compact the resultng subspace clusters. One example n ths category s COSA [10], a general framework for ths type of subspace clusterng; t uses weghts to encode the mportance of features to subspace clusters. These weghts nfluence the dstance calculaton and are optmzed as part of the clusterng procedure. Subspace clusterng methods share wth learnng n parallel unverses that they try to respect localty also n terms of the features space, although they do t on a dfferent scale. They refer to ndvdual features whereas we consder unverses,.e. semantcally meanngful descrptor spaces, whch are gven a-pror. Mult-Vew-Learnng. In mult-vew-learnng [12] one assumes a smlar setup as n learnng n parallel unverses,.e. the avalablty of multple descrptor spaces

3 390 B. Wswedel and M.R. Berthold (unverses or vews). Ths learnng concept has a dfferent learnng scope snce t expects all unverses/vews to share the same structure. Therefore each ndvdual unverse would suffce for learnng f enough tranng data were avalable. One of the frst works n the mult-vew area was done by Blum and Mtchell [6], who concentrate on the classfcaton of web pages. They ntroduce co-tranng n a sem-supervsed settng,.e. they have a relatvely small set of labeled and a rather large set of unlabeled nstances. The web pages are descrbed usng two dfferent vews, one based on the actual content of the web ste (bag of words), the other one based on anchor texts of hyperlnks pontng to that ste both vews are assumed to be condtonally ndependent. Co-tranng creates a model on each vew, whereby the tranng set s augmented wth data that was labeled wth hgh confdence by the model of the respectvely other vew. Ths way the two classfers bootstrap each other. There s also a theoretcal foundaton for the co-tranng setup: It was shown that the dsagreement rate between two ndependent vews s an upper bound for the error rate of ether hypothess [9]. Ths prncple already hghlghts the dentcal structure property of all vews as the base assumpton of mult-vew learners, whch s contrary to the setup of learnng n parallel unverses, where some nformaton may only be avalable n a subset of unverses. There exst a number of other smlar learnng setups. These nclude ensemble learnng, mult-nstance-learnng and sensor fuson. We do not dscuss these methods n detal but all have ether a dfferent learnng nput (no a-pror defned unverses) or a dstnct learnng focus (no partally overlappng structures across unverses whle retanng unverse semantcs). 3 Learnng n Parallel Unverses Learnng n parallel unverses refers to the problem settng of multple descrptor spaces, whereby sngle descrptors are not suffcent for learnng. Instead we assume that the nformaton s dstrbuted among the avalable unverses,.e. ndvdual unverses may only explan a part of the data. We propose a learnng concept, whch overcomes the lmtatons outlned above by a smultaneous analyss of all avalable unverses. The learnng objectve s twofold. Frst, we am to dentfy structures that occur n only one or few unverses (for nstance groups of objects that cluster well n one unverse but not n others). Secondly we want to detect and facltate overlappng structures between multple, not necessarly all, unverses (for nstance clusters that group n multple unverses). The frst am addresses the fact that a unverse s not a complete representaton of the underlyng objects, that s, t does not suffce for learnng. The task of descrptor generaton s n many applcaton domans a scence by tself, whereby sngle descrptors (.e. unverses) carry a semantc and mrror only certan propertes of the objects. The second am descrbes cases, n whch structures overlap across dfferent unverses. For clusterng tasks, for nstance, ths would translate to the dentfcaton of groups of objects that are self-smlar n a subset of the unverses. Note that n order to detect these overlaps and to support ther formaton t s necessary for nformaton to be exchanged between the ndvdual models durng ther constructon. Ths s a major characterstc of learnng n parallel unverses, whch cannot be realzed wth any of the other schemes outlned n the prevous secton.

4 3.1 Applcaton Scenaros Supervsed Learnng n Parallel Unverses Usng Neghborgrams 391 The challenge of learnng n parallel unverses can be found n almost all applcatons that deal wth the analyss of complex objects. We outlne a few of these below. Molecular data. A typcal example s the actvty predcton of drug canddates, typcally small molecules. Molecules can be descrbed n varous ways, whch potentally focus on dfferent aspects [3]. It can be as smple as a vector of numercal propertes such as molecular weght or number of rotatable bonds. Other descrptors concentrate on more specfc propertes such as charge dstrbuton, 3D conformaton or structural nformaton. Apart from such semantc dfferences, descrptors can also be of a dfferent type ncludng vector of scalars, chemcal fngerprnts (bt vectors) or graph representatons. These dverse representatons make t mpossble to smply jon the descrptors and construct a jont feature space. 3D object data. Another nterestng applcaton s the mnng of 3D objects [8]. The lterature lsts three man categores of descrptors: (1) mage-based (features descrbng projectons of the object, e.g. slhouettes and contours), (2) shape-based (lke curvature measures) and (3) volume-based (parttonng the space nto small volume elements and then consderng elements that are completely occuped by the object or contan parts of ts surface area, for nstance). Whch of these descrptons s approprate for learnng, mostly depends on the class of object beng consdered. For nstance mageor volume-based descrptors fal on modelng a class of humans, takng dfferent poses, snce ther projectons or volumes dffer, whereas shape-based descrptors have proven to cover ths class ncely. Image data. There are also manfold technques avalable to descrbe mages. Descrptors can be based on propertes of the color or gray value dstrbuton; they can encode texture nformaton or propertes of the edges. Other unverses can reflect user annotatons lke ttles that are assocated wth the mage. Also n ths doman t depends sometmes on the descrptor, whether two mages are smlar or not. 4 Neghborgrams n Parallel Unverses In the followng we descrbe the neghborgram data structure [4], dscuss the fully automated clusterng algorthm and hghlght the advantages of the vsualzaton. We consder a set of objects, each object beng descrbed n U,1 u U, parallel unverses. The object representaton for object n unverse u s x,u. Each object s assgned a class c(). We further assume approprate defntons of dstance functons for each unverse d (u) (x,u,x j,u ). 4.1 Neghborgram Data Structure We defne a neghborgram as a lst of R-nearest neghbors to an object n a unverse u: NG (u) = x (u) l,u,...,x,1 l (u).,r,u

5 392 B. Wswedel and M.R. Berthold The subscrpt l (u),r centrod. For the sake of smplcty we abbrevate l (u) reflects the orderng of the neghbors accordng to the dstance to the,r wth l r and also omt the second subscrpt u, always accountng for the fact that the lst contans objects referrng to a centrod object n a specfc unverse u. The orderng mples for any neghborgram l (u),1 = l 1 = snce the centrod s closest to tself. Ths lst s not necessarly a unque representaton of the neghborhood as objects may have an equal dstance to the centrod. However, ths s not a lmtaton to the algorthm. Neghborgrams are constructed for all objects of nterest,.e. ether for all objects of a small- or medum-sze data set or the objects belongng to one or more target classes, n all unverses. As an llustratve example consder the data set n fgure 1, whch s gven n a sngle unverse. Ths unverse s 2-dmensonal (shown on the left) wth two dfferent classes (empty and flled crcles). The neghborgrams for the three numbered empty objects are shown on the rght. We use the Manhattan norm as dstance functon,.e. the dstance between two objects s the number of steps on the grd (for nstance the dstance between 1 and 3 s 3, two steps n horzontal and one n vertcal drecton). The correspondng lst representaton s then NG 1 = 1,, 2, 3, NG 2 = 2, 3, 1,, NG 3 = 3, 2,, 1, We wll use these neghborgrams n the followng to ntroduce basc measures that help us to derve cluster canddates and to determne numerc qualty measures. 6 y x d d d Fg. 1. Sample nput space wth neghborgrams for the objects 1, 2 and Neghborgram Clusterng Algorthm The basc dea of the learnng algorthm s to consder each neghborgram n each unverse as a potental (labeled) cluster canddate. These canddates are derved from the lst representaton, are based on the class dstrbuton around the centrod, and are assgned the class of ther respectve centrods. Good clusters wll have a hgh coverage,.e. many objects of the same class as the centrod n the close vcnty, whereby cluster boundares are represented by the dstance of the farthest neghbor satsfyng some purty constrant. The basc clusterng algorthm wll teratvely add the cluster canddate wth the hghest coverage to the set of accepted clusters, remove the covered objects

6 Supervsed Learnng n Parallel Unverses Usng Neghborgrams 393 from consderaton, and start over by re-computng the coverage of the remanng canddates. The nterestng pont here s that ths basc algorthm s not restrcted to learn n a sngle unverse. In fact, ths basc algorthm s free to choose the best cluster canddate from all unverses, thereby drectly buldng a cluster set wth cluster from dfferent orgns and hence a model for parallel unverses. Before we provde the algorthm n pseudo-code let us defne some measures that are used to defne a cluster canddate and ts numercal qualty. Each of the followng values s assgned to a sngle neghborgram (but computed for all): Coverage Γ (u) : The coverage descrbes how many objects of the same class as the centrod c() are wthn a certan length r n the neghborgram for object n unverse u. These objects are covered by the neghborgram. { Γ (u) (r)= x lr NG (u) 1 r r c(l r )=c()} For example, the coverages for NG 1 n fgure 1 are: Γ 1 (1)=1, Γ 1 (2)=1, Γ 1 (3)= 2andΓ 1 (4)=3. Purty Π (u) (r): The purty denotes the rato of objects of the correct class (.e. same class as the centrod) to all objects that are contaned wthn a certan neghborhood of length r: { Π (u) x lr NG (u) 1 r r c(l r )=c()} (r)= { } x lr NG (u). 1 r r For nstance, object 1 n fgure 1 would have purtes: Π 1 (1) =1, Π 1 (2) = 1 2, Π 1 (3)= 2 3 and Π 1 (5)= 3 5. Optmal Length Λ (u) : The optmal length s the maxmum length for whch a gven purty threshold p mn s stll vald. In practcal applcatons t has shown to be reasonable to constran the optmal length to be at least some mnmum length value r mn n order to avod clusters that cover only few objects. Λ (u) (p mn,r mn )= { } max r r mn r r Π (u) (r ) p mn. Note, ths term may be undefned and practcally often s for nosy data. In the example n fgure 1 the optmal length values are for a mnmum length r mn = 1 (unconstraned): Λ 2 (1,1)=2andΛ 2 ( 2 3,1)=3. Apart from the specfcaton of a mnmum purty p mn and a mnmum cluster sze r mn, the user has to specfy a mnmum overall coverage ψ mn. If the sum of coverage values of all accepted clusters s greater than ths value, the algorthm termnates. The basc algorthm s outlned n algorthm 1. The ntalzaton starts wth the constructon of neghborgrams for all objects of certan target classes (mnorty class(es) or all classes) n lne 1. Pror to the learnng process tself, t determnes a cluster canddate for each

7 394 B. Wswedel and M.R. Berthold Algorthm 1. Basc Neghborgram Clusterng Algorthm 1:,u: c() s target class construct NG (u) 2: NG (u) : compute Λ (u) = Λ (u) (p mn,r mn ) 3: s 0 /* accumulated coverage */ 4: NG /0 /* (empty) result set */ 5: î undefned /* current best Neghborgram */ 6: repeat ( ) 7: NG (u) : compute Γ (u) Λ (u) { ( 8: (î,û) argmax (,u) )} Γ (u) Λ (u) 9: f î s undefned then 10: ˆω Λ (û) î { ( ) 11: M covered l (û) 1 r ˆω c l (û) = c ( î )} î,r î,r 12: NG (u) 13: s s +Γ (û) î 14: NG NG :NG (u) NG (u) ( ˆω) {( NG (û) î 15: end f 16: untl (s ψ) (î s undefned) 17: return NG \ { x j,u j M covered } )}, dˆ /* optmal length best Neghborgram */ neghborgram, whch s based on user defned values for the mnmum purty and the mnmum sze (lne 2). The learnng phase s shown n lne 6 to 16: It teratvely adds the cluster canddate wth the hghest (remanng) coverage (lne 8) to the cluster set NG, whle removng the covered objects from all remanng neghborgrams. The algorthm termnates f ether no more cluster canddates satsfy the search constrants (mss the mnmum sze constrant) or the accumulated coverage s larger than the requred coverage ψ mn. Ths basc algorthm learns a set of clusters dstrbuted over dfferent unverses, whereby each cluster s assocated wth a class (the class of ts centrod). The predcton of unlabeled data s carred out by a best-matchng approach,.e. by dentfyng the cluster that covers the query object best. In case more than one cluster covers the query and these clusters are assgned dfferent classes, the fnal class can be determned usng a majorty vote of the clusters. Unverse Interacton. The basc algorthm nherently enables smultaneous learnng n dfferent unverses by consderng all neghborgrams as potental cluster canddates and removng covered objects n all unverses. However, ths s a rather weak nteracton as t does not respect overlaps between unverses. These are often of specal nterest for manly two reasons: Frstly, they gve new nsghts regardng recurrent structures n dfferent unverses and help the expert to better understand relatons between unverses. Secondly, they may help to buld a more robust model. The basc algorthm descrbed above penalzes such overlappng structures snce t removes covered objects from all remanng neghborgrams. If two clusters group equally n two unverses, the algorthm would choose any one of the two clusters and remove the objects. The non-accepted

8 Supervsed Learnng n Parallel Unverses Usng Neghborgrams 395 cluster wll never be consdered as a good cluster canddate agan, because all ts supportng objects were removed already. In order to take these overlaps nto account, we modfy the algorthm to detect overlappng clusters durng the clusterng process. More formally, we defne the relatve overlap υ :NG NG [0,1] of two neghborgrams as ( υ NG (u),ng (y) j ) = M (u) M (u) M (y) j M (y), whereby M (u) denotes the set of objects of the same class as the centrod that are covered by the cluster canddate derved from the Neghborgram NG (u) accordng to the user settngs of mnmum purty and sze. Ths overlap s the cardnalty of the ntersecton of these two sets dvded by the sze of the unon. Two neghborgrams n dfferent unverses representng the same cluster wll therefore have a hgh overlap. Usng the above formula we can numercally express the overlap between cluster canddates. We modfy the basc algorthm to also add those canddates to the cluster set that have the hghest overlap wth the currently best cluster canddate n each unverse unless ther overlap s below a gven threshold υ mn. That s, n each teraton (refer to lne 6 16 n algorthm 1) we determne the cluster wth the hghest remanng coverage n all unverses. Before removng the covered objects from further consderaton we dentfy the cluster canddates n all remanng (.e. non-wnner) unverses wth the hghest overlap wh respect to the current wnner. For each of these overlappng cluster canddates we test whether the overlap s at least υ mn and, f so, also accept t as cluster. Ths strategy of course ncreases the number of clusters n the fnal model but t also mproves the predcton qualty on test data consderably as we wll see later. Weghted Coverage. Another benefcal modfcaton to the algorthm s the usage of a dfferent cluster representaton by means of gradual degrees of coverage. In the prevous sectons we used sharp boundares to descrbe a cluster. An object s ether covered by a cluster or not, ndependent of ts dstance to the cluster centrod. By usng a weghted coverage scheme that assgns hgh coverage ( 1) to the objects n the cluster center and low values ( 0) to those at the cluster boundares, we have a much more natural cluster representaton [5]. As weghtng scheme we use n the followng a lnear functon; the degree of coverage μ (u) for a neghborgram NG (u) μ (u) ( j)= ( j) for an object j to the cluster canddate wth ts centrod x,u n unverse u s: { d d ˆ (u) (x,u,x j,u) dˆ 0 else, j f d (u) (x,u,x j,u ) ˆ d whereby dˆ represents the dstance of the farthest object covered by the neghborgram accordng to the optmal length Λ (u). Note that due to the gradual coverage the algorthm needs to be slghtly adapted: The coverage of a cluster s no longer a smple count of the contaned objects but an accumulaton of ther weghted coverage. Also the removal of objects from consderaton needs to be changed n order to reflect the

9 396 B. Wswedel and M.R. Berthold partal coverage of objects. These changes are straghtforward and therefore we omt a repeated lstng of the modfed algorthm here. 5 A k-nearest Neghbor Classfer for Parallel Unverses Before we present our results, we shall brefly dscuss an extenson to the k-nearest neghbor approach for parallel unverses, whch we wll use to compare our results aganst. It s based on the dea presented n [7], who use a modfed dstance measure to perform 3D object retrevals. Smlar to learnng n parallel unverses they are gven data sets n multple descrptor spaces. The dstance between a query object and an object n the tranng set s composed of ther dstances n the dfferent unverses. The ndvdual dstance values are weghted by the κ-entropy mpurty, whch s the class entropy of the κ nearest neghbors n the (labeled) tranng set n a unverse around the query object,.e. the object to be classfed. The entropy mpurty wll be 0 f all κ neghbors n the tranng set have an equal class attrbute and accordngly larger f the neghborhood contans objects of dfferent classes. If mp (u) (q,κ) denotes the κ-entropy mpurty n unverse u for a query object q, then the accumulated overall dstance δ(q,) between q and and object s [7]: δ(q,)= U u= mp (u) (q,κ) d(u) (x q,u,x,u ) d max (u). q Note the term d (u) max q n the denomnator of the dstance coeffcent: t s the maxmum dstance between q and the objects n the tranng set. It s used to overcome normalzaton problems that arse when accumulatng dstances from dfferent domans/unverses a problem that the neghborgram approach fortunately does not suffer from as t does not compare dstances across unverses. We use the above dstance functon n a k-nearest neghbor algorthm n the followng secton as t has proven to be approprate for the 3D data set beng analyzed [7]. However, ts general applcablty for parallel unverse problems s questonable for manly three reasons: (1) t does not scale for larger problem szes (specfcally because of query-based nearest neghbor searches n all unverses), (2) t has the above-mentoned normalzaton problem (the normalzaton factor depends heavly on the data set), and (3) t does not produce an nterpretable model n the form of rules or labeled clusters (as n the Neghborgram approach). 6 Results We use a data set of 3D objects to demonstrate the practcal usefulness of the presented neghborgram approach. The data set contans descrptons of 3D objects gven n dfferent unverses, whch cover mage-, volume- and shape based propertes [1,7]. There are a total of 292 objects, whch were manually classfed nto 17 dfferent classes such as arplanes, humans, swords, etc. The objects are descrbed by means of 16 dfferent unverses, whose dmensons vary from 31 to more than 500 dmensons. The number of

10 Supervsed Learnng n Parallel Unverses Usng Neghborgrams msclassfcaton count (υ mn = 0.8) 0/1 coverage weghted coverage mn purty p mn Fg. 2. Msclassfcaton counts for sharp (0/1 coverage) vs. soft cluster boundares (weghted coverage) objects per class ranges from 9 to 56. The descrptors manly fall nto three categores. The mage-based descrptors extract propertes from an object s projectons (typcally after normalzng and algnng the object). They typcally encode nformaton regardng an object s slhouette or depth nformaton. There are 6 unverses n ths category. Volume-based descrptors reflect volumetrc propertes of an object, for nstance a dstrbuton of voxels (small volume elements) n a unt cube beng occuped by the object. There are 5 unverses of ths type. Fnally, shape-based descrptors encode surface and curvature propertes, e.g. based on the center ponts of the objects faces (center of the dfferent polygons descrbng an object). The remanng 5 unverses are of ths type. Smlar to the results presented n [7] we use the Manhattan norm on the unnormalzed attrbutes n each unverse and perform 2-fold cross valdaton to determne error rates (we lst absolute error counts for the 292 objects to be classfed below). We frst ran a k-nearest neghbor approach to determne a reference value for the achevable error rate. We used the aggregated dstance measure presented n secton 5 and tested dfferent settngs. The smallest error we could acheve was 40 msclassfcatons for κ = 3 (to calculate a unverse s κ-entropy mpurty) and k = 2 (nearest neghbor parameter), whch matches the fndngs of [7]. In a frst experment wth the neghborgram clusterng approach we tested the effect of the weghted coverage approach and compared t to the error rates when usng a sharp cluster representaton. We vared the mnmum purty parameter p mn from 0.7 to 1.0 and set an overlap threshold υ mn = 0.8. A mnmum sze r mn foraclusterwas not set to respect underrepresented classes. Fgure 2 shows the results. The 0/1 coverage curve ndcates the error rates when usng a sharp cluster representaton,.e. even objects at the cluster boundares are fully covered. The weghted coverage approach mproves the predcton qualty consderably and yelds error rates comparable to the k-nearest neghbor method. Note, usng the weghted coverng approach also ncreases the total number of clusters n the fnal model. If there are no further constrants regardng termnaton crteron or mnmum cluster sze, the cluster count can reach up to 600 clusters (from a total of = cluster canddates). In another experment we evaluated the mpact when dentfyng overlappng clusters across unverses and addng those to the cluster set. Ths experment compares the

11 398 B. Wswedel and M.R. Berthold 80 msclassfcaton count (υ mn = 0.6) SIL DBF ParUn (basc) ParUn(overlap) mn purty p mn cluster count (υ mn = 0.6) SIL DBF ParUn (basc) mn purty p mn Fg. 3. Msclassfcaton rates and cluster counts usng the Parallel Unverse extenson compared to models bult on ndvdual unverses basc algorthm shown n algorthm 1 wth the extenson dscussed n secton 4.2 (usng weghted coverage n ether case). The results are summarzed n fgure 3 along wth the results of the two best sngle-unverse classfers. These were bult n the unverses DBF (depth buffer) and SIL (slhouette), both of whch are mage-based descrptors. The basc algorthm n parallel unverses already outperforms the sngle-unverse classfers n terms of both classfcaton accuracy and numbers of clusters used. When addtonally takng overlappng clusters nto account (represented by the curve labeled Parun (overlap) ), the error rate drops consderably, showng the advantage of havng some redundancy across unverses n the cluster set. However, these re-occurrng structures come at a prce of an ncreased number of cluster, whch s n the order of (omtted n the graph). Ths suggests that the basc algorthm s sutable f the focus s on buldng an understandable model wth possbly only a few clusters, whereas the enhanced algorthm wth overlap detecton may be approprate when the focus les on buldng a classfer wth good predcton performance.

12 Neghborgram Vsualzaton Supervsed Learnng n Parallel Unverses Usng Neghborgrams 399 Another advantage of the neghborgram data structure s the ablty to vsualze them and thus allow the user to vsually assess a cluster s qualty and potental overlaps across unverses. Fgure 4 shows an example. A row represents the neghborgrams for one object n four dfferent unverses (two mage- (DBF & SIL), one shape- (SSD) and one volume-based (VOX)). We use the same vsualzaton technque as n the llustratve example n fgure 1,.e. a dstance plot, whereby the vertcal stackng s only used to allow an ndvdual selecton of ponts. The plots show the 100 nearest neghbors; objects of the same class as the centrod are shown n dark grey and objects of all other classes n lght grey. Objects of a (sem-automatcally) selected cluster are addtonally hghlghted to see ther occurrence also n the respectve other neghborgrams/unverses. 2D depctons of these objects are also shown at the bottom of the fgures, whereby we manually expanded the cluster to also cover some conflctng objects (shown on the bottom rght). Fgure 4 shows the largest cluster n the DBF unverse, whch covers the class swords. Note ths cluster also groups n the SIL unverse, whch s also mage-based, though there seems to be no groupng n the shape- and volume-based descrptors SSD and VOX. In contrast, clusters of the class humans form ncely n the shape-based unverse SSD. There are more nterestng clusters n ths data set, for nstance larger groups of cars, whch cluster well n the DBF and VOX unverse or a group of weeds, whch only clusters n the volume-based descrptor space. We do not show these clusters n separate fgures due to space constrants. However, ths demonstrates ncely that dependng on the type of object certan descrptors,.e. unverses, are better suted to group them. These results emphasze that learnng n parallel unverses s an mportant and well suted concept when dealng wth such types of nformaton. Fg. 4. The largest cluster n unverse DBF covers objects of class sword (hghlghted by border). 2D depctons of the covered objects are shown at the bottom. The cluster was manually expanded to cover also two conflct objects of class spoon (shown at bottom rght).

13 400 B. Wswedel and M.R. Berthold 7 Summary We presented a supervsed method for the learnng n parallel unverses,.e. for the smultaneous analyss of multple descrptor spaces usng neghborgrams. We showed that by usng an overlap crteron for clusters between unverses we can consderably mprove the predcton accuracy. Apart from usng neghborgrams as an underlyng bass to learn a classfcaton model, they can also be used as vsualzaton technque to ether nvolve the user n the clusterng process or to allow for a vsual comparson of dfferent unverses. References 1. Konstanz 3D model search engne (2008), (last access March 20, 2009) 2. Agrawal, R., Gehrke, J., Gunopulos, D., Raghavan, P.: Automatc subspace clusterng of hgh dmensonal data for data mnng applcatons. In: Proc. ACM SIGMOD Internatonal Conference on Management of Data, pp ACM Press, New York (1998) 3. Bender, A., Glen, R.C.: Molecular smlarty: a key technque n molecular nformatcs. Organc and Bomolecular Chemstry 2(22), (2004) 4. Berthold, M.R., Wswedel, B., Patterson, D.E.: Neghborgram clusterng: Interactve exploraton of cluster neghborhoods. In: IEEE Data Mnng, pp IEEE Press, Los Alamtos (2002) 5. Berthold, M.R., Wswedel, B., Patterson, D.E.: Interactve exploraton of fuzzy clusters usng neghborgrams. Fuzzy Sets and Systems 149(1), (2005) 6. Blum, A., Mtchell, T.: Combnng labeled and unlabeled data wth co-tranng. In: Proceedngs of the Eleventh Annual Conference on Computatonal Learnng Theory (COLT 1998), pp ACM Press, New York (1998) 7. Bustos, B., Kem, D.A., Saupe, D., Schreck, T., Vranc, D.V.: Usng entropy mpurty for mproved 3D object smlarty search. In: Proceedngs of IEEE Internatonal Conference on Multmeda and Expo (ICME 2004), pp (2004) 8. Bustos, B., Kem, D.A., Saupe, D., Schreck, T., Vranć, D.V.: An expermental effectveness comparson of methods for 3D smlarty search. Internatonal Journal on Dgtal Lbrares, Specal ssue on Multmeda Contents and Management n Dgtal Lbrares 6(1), (2006) 9. Dasgupta, S., Lttman, M.L., McAllester, D.A.: PAC generalzaton bounds for co-tranng. In: NIPS, pp (2001) 10. Fredman, J.H., Meulmany, J.J.: Clusterng objects on subsets of attrbutes. Journal of the Royal Statstcal Socety 66(4) (2004) 11. Parsons, L., Haque, E., Lu, H.: Subspace clusterng for hgh dmensonal data: a revew. SIGKDD Explor. Newsl. 6(1), (2004) 12. Rüpng, S., Scheffer, T. (eds.): Proceedngs of the ICML 2005 Workshop on Learnng Wth Multple Vews (2005) 13. Wswedel, B., Berthold, M.R.: Fuzzy clusterng n parallel unverses. Internatonal Journal of Approxmate Reasonng 45(3), (2007)