Semi-Supervised Affinity Propagation with Instance-Level Constraints

Transcription

1 Semi-Supervised Affinity Propagation with Instane-Level Constraints Inmar E. Givoni, Brendan J. Frey Probabilisti and Statistial Inferene Group University of Toronto 10 King s College Road, Toronto, Ontario, Canada, M5S 3G4 Abstrat Reently, affinity propagation (AP) was introdued as an unsupervised learning algorithm for exemplar based lustering. Here we extend the AP model to aount for semisupervised lustering. AP, whih is formulated as inferene in a fator-graph, an be naturally extended to aount for instanelevel onstraints: pairs of data points that annot belong to the same luster (annotlink), or must belong to the same luster (must-link). We present a semi-supervised AP algorithm (SSAP) that an use instanelevel onstraints to guide the lustering. We demonstrate the appliability of SSAP to interative image segmentation by using SSAP to luster superpixels while taking into aount user instrutions regarding whih superpixels belong to the same objet. We demonstrate SSAP an ahieve better performane ompared to other semi-supervised methods. 1 Introdution Affinity propagation (AP) (Frey & Duek, 2007) is an exemplar-based lustering method that takes as input similarities between data points. It outputs a set of data points that best represent the data (exemplars), and assignments of eah non-exemplar point to its most appropriate exemplar, thereby partitioning the data-set into lusters. The objetive of AP is to maximize the sum of similarities between the data points and their exemplars. The AP algorithm is based on asting this NP-hard optimization problem in terms of Appearing in Proeedings of the 12 th International Conferene on Artifiial Intelligene and Statistis (AISTATS) 2009, Clearwater Beah, Florida, USA. Volume 5 of JMLR: W&CP 5. Copyright 2009 by the authors. a fator-graph, and performing approximate MAP inferene using the max-produt algorithm(kshishang et al., 2001). The fator-graph used in AP an be modified to allow onstraints and additional information to be aounted for in a prinipled way, like introduing flexible priors on luster size (Tarlow et al., 2008). In this work we are interested in extending the AP framework to semi-supervised lustering. Semi-supervised lustering algorithms are onerned with finding good partitions of data in the presene of side information. Two popular forms of side information are partial labels and instane-level onstraints. We onsider the ase where side information is given in the form of instane-level onstraints on pairs of data points. Cannot-link onstraints indiate the two data points annot be in the same luster while mustlink onstraints indiate the data points must be in the same luster (Wagstaff & Cardie, 2000). There is a distint differene between side information given in the form of partial labels and that given in the form of instane-level onstraints. The two are not equivalent sine labeled data an always be used to onstrut instane-level onstraints while the onverse does not hold in general, making instane-level onstraints weaker in terms of the amount of information they arry. However, instane-level onstraints are often faster or heaper to obtain than labels, and an sometimes be automatially olleted (Wagstaff et al., 2001; Klein et al., 2002; Shental et al., 2003). Another differene between partial labels and instanelevel onstraints is that instane-level onstraints do not diretly provide information about the total number of lusters or lasses in the data. It is always possible to onstrut the transitive losure for any set of instane-level onstraints: if points i and j are onstrained to be in the same luster, and points j and k are likewise onstrained, then it follows that i,j, and k must all be in the same luster. Similarly, if i and j must be in the same luster, but i and k annot be in the same luster then j and k annot be in the same luster. However, suh grouping of onstraints does not indiate the total number of groups is equal 161

2 Semi-Supervised Affinity Propagation with Instane-Level Constraints to the total number of lusters in the data, as it is only a lower bound. Thus, many algorithms that inorporates instane-level onstraints, suh as lustering algorithms that attempt to partition the data subjet to the onstraints, an detet lusters omposed of points for whih no side information is given (Wagstaff et al., 2001; Klein et al., 2002; Shental et al., 2003). When, on the other hand, side information is provided as partial labels, it is often assumed that the unique number of lass labels should be the number of identified lasses (Xiao et al., 2007; Leone et al., 2008). In order for suh algorithms to sueed, there must be at least one labeled example from eah lass in the data. The property of being able to detet lusters for whih no side information is given is desirable for ases where some lusters are easy to detet, while others may require human intervention. The motivating example in this work is the ase of user-guided image-segmentation, where segmentation results an be sometimes improved with quite limited user input, and where it should not be neessary to expliitly provide side-information for every objet in the image. One prominent approah for devising semi-supervised algorithms for instane-level onstraints is to modify standard unsupervised lustering algorithms so that they expliitly aount for the onstraints. This was done for k-means lustering (Wagstaff et al., 2001), and Mixture of Gaussians (Shental et al., 2003). A missing aspet of this approah is that it does not expliitly propagate onstraints. Intuitively, if we look at the set of points that are very lose to either point in a must-link onstraint, it is likely these points should also be in the same luster as the must-link onstraint points. This property is sometimes referred to as spae-level onstraints (Klein et al., 2002), and is often imposed by adapting the similarities (or distanes) between data points to better align with the onstraints; a good distane metri for the data should intuitively assign a small distane to a pair of points with a mustlink onstraint, and a large distane to a pair of data points with a annot-link onstraint. Then, the adjusted metri an be used as input to an unsupervised lustering algorithm (Xing et al., 2003). Naturally, the approah of adapting similarities an be ombined with the approah of expliitly aounting for onstraints (Klein et al., 2002; Basu et al., 2004). The solution we propose here is to adapt the underlying fator-graph used in affinity propagation suh that onstraints are expliitly added, but also propagated. We add meta-points to the underlying model and appropriate funtion nodes to govern the allowed set of solutions. The meta-points link together the must-link pairs and prevent annot-link pairs from being in the same luster. Similarities of points to meta-points are onstruted in a way that allows the model to also aount for spae-level onstraints: points that are very lose to points that must be in the same luster are likely to also be in the same luster. Although, to the best of our knowledge, there are no other AP-based methods that inorporate instanelevel onstraints, Xiao et al. desribe an adaption of AP to inlude partial labels by ontrating all similarly labeled points to a new point and adjusting similarities between unlabeled points to the new ontrated points. The set of potential exemplars is restrited to that of the ontrated points, and standard AP is then run on the modified data-set. This restrition prevents differently labeled points from being put in the same luster and fores the number of lusters to be equal to the number of unique labels. Leone et al. also ontrat all labeled points and adjust the similarities, but they do not restrit only ontrated labeled points to be exemplars. However, they do not attempt to luster the ontrated labeled points, whih only serve as potential exemplars. Both algorithms do not aount for must-link or annot-link onstraints. In addition, the goal of AP is not only to partition the data but to also find the most representative data points, or exemplars. In appliations where the atual exemplars are meaningful the ontrated points may not be useful sine they do not represent real data points. 2 Semi-Supervised AP 2.1 Unsupervised AP I 1 I i IN s 11 si1 sn1 11 i1 N1 E 1 s 1 j sij snj 1 j ij Nj E j s 1N s in snn 1N in NN E N Figure 1: A binary variable model for AP We begin with a brief review of the unsupervised affinity propagation (AP) model for lustering data points, where we use the binary grid fator-graph desribed in (Givoni & Frey, 2009) (Fig. 1). The input to the algorithm is pairwise similarities s(i, j) between N data points i (1... N), j (1... N), where we assume similarities are negative, and the maximal possible similarity between two points is 0. We define N 2 hidden binary variables ij. Setting ij = 1 denotes that i s exemplar is j, and ii = 1 indiates i is its own 162

3 Givoni, Frey exemplar. The I funtion nodes introdue the 1-of-N onstraint: eah point an be assigned to at most one exemplar (that exemplar an be the point itself, meaning the point is hoosing itself as an exemplar). The E funtion nodes introdue the exemplar onsisteny onstraint: in order for any point i, i j to hoose j as its exemplar, j must be its own exemplar. Finally, the S ij funtion nodes inorporate the user-defined input similarities s(i, j) between data points and their potential exemplars and evaluate to the similarity s(i, j) when ij = 1. Formally, the funtion definitions are: { if I i ( i1,..., in ) = j ij 1, (1) 0 otherwise. { if jj = 0 and E j ( 1j,..., Nj ) = i ij > 0, 0 otherwise. (2) { s(i, j) if ij = 1, S ij ( ij ) = (3) 0 otherwise. The graphial model in Fig. 1 together with (1)-(3) result in the following objetive funtion 1 : S( 11,..., NN ) = i,j + j S ij ( ij ) + i E j ( 1j,..., Nj ), I i ( i1,..., in ) stating that we wish to find the onfiguration of the ij variables that maximizes the similarity of the data points to their exemplars. The tendeny of a partiular point to be an exemplar is given by the preferenes s(i, i). Like the similarities, it is assumed to be at most 0. A small negative preferene indiates a point is most suitable to be an exemplar and a big negative preferene indiates the opposite. Usually, all data points share the same preferene p, and this number is a free parameter that ontrols the amount of lusters found by the algorithm. An intuitive interpretation for the preferene is to view it as a ost assoiated with reating a luster (with an assoiated exemplar). Given luster set-up osts, and similarities between points, the heaper it is to reate lusters, the more lusters the algorithm an find. Therefore the term i,j S ij( ij ) in the objetive funtion an be broken into a that represents the sum of similarities of data points to their luster exemplar and a term representing the total ost assoiated with setting the lusters. The algorithm attempts to find the best trade-off for a partiular setting of the ost. The approximate MAP setting for the ij variables is inferred by the max-sum algorithm(kshishang et 1 The formulation used here is the log-domain maxprodut, or max-sum algorithm al., 2001). It is shown in (Givoni & Frey, 2009) that the different types of messages that need to be propagated in the Fig. 1 graph an be redued to two simple sets of messages that are iteratively updated until onvergene. These messages are the ones exhanged between the hidden variables and the olumn funtion nodes E, where the other messages are subsumed into them for simpliity. The AP update messages are as follows: max[0, r(k, j)] k j a(i, j) = min [0, r(j, j) + ] max[0, r(k, j)] k j,i i = j i j (4) r(i, j) = s(i, j) max(s(i, k) + a(i, k)) (5) k j The messages have an intuitive interpretation; The responsibilities r are indiators of how muh data points think other data points are suited to be their exemplars. The availabilities a indiate to what extent data points onsider themselves fit to serve as exemplars for other data points. After onvergene, the exemplars are found by alulating the set of positive a(i, i) + r(i, i) messages for eah i {1... N}. Nonexemplars are assigned their respetive exemplars by hoosing max j J (a(i, j)+r(i, j)), where J denotes the set of exemplars. 2.2 From Unsupervised to Semi-supervised AP Now, suppose we obtain instane-level onstraints for some input data points, and we wish to endow our model with the ability to use this side information. The first intuitive approah might be to diretly onnet the hidden variables orresponding to data points that must be in the same lusters via a funtion that enfores this onstraint, and similarly, onneting the hidden variables orresponding to annot-link data points with an appropriate funtion node as well. It turns out, however, that running AP on suh a graphial model yields solutions that satisfy the onstraints but are otherwise meaningless; the transitive losure of data points with must-link onstraints get grouped together in their own lusters, while non-onstrained data points that are similar to the must-link onstrained ones are not neessarily in the same luster with the must-link points, unlike what ommon sense would ditate. The reason for this is diret onstraints do not indue propagation of information from onstrained points to non-onstrained ones. Another intuitive idea for inorporating onstraints is to alter the similarities between data points. For example, by making the similarity between must-link 163

4 Semi-Supervised Affinity Propagation with Instane-Level Constraints points be maximal, the similarity between annot-link points minimal and the similarity between any other two points the shortest-path between them. This way, if i is similar to j, s is similar to t, and j and s must be in the same luster, the similarity between i and j an be inreased if s(i, j) + s(t, s) > s(i, s). However, as noted in (Klein et al., 2002) this approah aptures the property that must-link points and their neighbors should be in the same luster but it does not enfore the desired property that data points similar to annot-link points are more likely to be put in different lusters. Indeed, it is also noted in (Klein et al., 2002) that enforing annot-link onstraints is oneptually harder than must-link onstraints; even determining if the set of annot-link onstraints has a satisfying assignment is NP-omplete. A similar observation is made in (Shental et al., 2003) where the must-link onstraints are inorporated by onstruting their transitive losure, and onstraining them to be in the same luster, using a relatively simple modifiation of the EM algorithm update equations, while the annot-link onstraints require the inlusion of a hidden MRF over the data points, resulting in a onsiderably more involved inferene proedure, and an inreased degree of approximation. The solution we propose here is to augment the data points with fititious meta-points or M T P s. We ompute the transitive losure of the must link onstraints, and add one MT P for eah resulting group as well as to eah point in a annot-link onstraint if it is not also part of a must-link group. The MT P s allow us to expliitly enfore the must-link onstraints and annot-link onstraints, as well as to propagate must-link onstraints and onstrut a mehanism for annot-link onstraints to be propagated. As expeted from the disussion above, the effetiveness of propagating annot-link onstraints is more limited but it is inorporated and inferred using the same simple formulation as the rest of the model, and is shown to yield good results in pratie. We now desribe how to augment the model with the MT P s. Let M be the number of MT P s, and let P m be the set of data points assoiated with MT P m, m {1..., M}. We define symmetri similarities between MT P m and the input data points as: { 0 if i Pm, S(i, MT P m ) = max j Pm s(i, j) otherwise. Note that S(MT P m, i) = S(i, MT P m ). The intuition behind the onstrution of the metapoints is that data points will now be able to hoose either real exemplars or one of the MT P s, if it is more suitable than a real exemplar. The MT P s in turn will have to hoose a real exemplar. Sine all points in a must-link group will neessarily hoose the MT P assoiated with them, this will also result in a lustering that respets must-link onstraints. Other data points are also free to hoose MT P s and so points that are similar to a group of must-link points are likely to also hoose that group s MT P as an exemplar, leading to the spae-level propagation of onstraints. Furthermore, if there is a annot-link onstraint between some i P m and a P n we introdue a diret inequality onstraints between MT P m and MT P n by inluding funtion nodes that prevent the MT P s from hoosing the same exemplar. This is a ompat representation of annot-link onstraints: let L m = {i, j, k} be one set of points that must be in the same luster, and L n = {a, b, } be another set of points that must be in the same luster, if we also know that i annot be with a, then all pairs in the ross produt L m L n also have a annot-link onstraint,whether it was expliitly speified or not. However, one onstraint between the MT P s of eah set is all we need in order to enfore all these onstraints, as opposed to the representation in (Shental et al., 2003) that involved MRF onnetions among all suh onstraints. Furthermore, any point that will hoose some MT P m as an exemplar, most likely beause it is similar to one of the points in the must-link group assoiated with MT P m, will be in a different luster than any point that has a annot-link onstraint with the must-link group assoiated with MT P m, if suh exists. This an allow spae-level propagation of annot-link onstraints. I 1 I N I N+1 I N + M 11 N1 N+1,1 N + M,1 E 1 1N NN N +1, N CL + 1 N + 1, N M N+ M, N E N 1, N + 1 N, N +1 CL + N N + 1, N M 1, N + M N, N+ M Figure 2: Semi-supervised affinity propagation. The fator-graph inside the dotted line is the original AP model, and the similarity funtions s have been removed for larity Fig. 2 shows the graphial model of SSAP that inludes the MT P s (the similarity funtion nodes have been omitted for larity) and the annot-link onstraints. The I funtion nodes, that enfore the property that eah point must hoose exatly one exemplar, remain the same as in the standard AP model. For the in- 164

5 Givoni, Frey put data points, the hoie of an exemplar is over all data points and all MT P s. Sine MT P s should only be allowed to hoose exemplars from the set of input data points, the domain of their assoiated I funtions is only over the input data points. The E funtion nodes, that enfore the onstraint that a point an only hoose an exemplar if that exemplar hooses itself as an exemplar, is not required for the MT P s, as in fat, by definition they should not hoose themselves as exemplars but instead pik an exemplar from the set of input data points. The CL funtion nodes are added to enfore the annot-link onstraints between pairs of MT P s for whih suh onstraints are given, as desribed above. In Fig. 2 we show for larity only two MT P s and a annot-link onstraint between them, but in the general ase there is not neessarily a onstraint between every two MT P s and the exat nature of annot-link onnetions between MT P s depend on the given onstraints. Formally the annot-link funtion nodes are given by CL k N+m,N+n ( N+m,k, N+n,k ) = { N+m,k = N+n,k 0 otherwise The message updates for the new model are similar to the original AP messages. In fat, the a(i, j) messages (4) remain the same, save for the indexing domain over k in the sum that hanges from k {1,..., N} to k {1,..., N + M}. The r(i, j) messages (5) remain as before for i < N with a similar indexing hange over the maximization. In order to express r(m, j) for m > N, we need to inlude the messages arriving from and sent to the CL onstraint nodes. In order to keep the notation simplified, let us define the following two messages: q j (m, mn) = µ CN+m,j CL j N+m,N+n (6) q j (mn, m) = µ CL j N+m,N+n C N+m,j, (7) The update rule for (6), the message from a variable node to the CL funtion node, is: q j (m, mn) = a(m, j) + r(m, j) q j (mn, m) (8) And the update rule for the message from the CL funtion node (7) has the form q j (mn, m) = max[0, q j (n, mn)] (9) Now we an express r(m, j) for m > N: r(m, j) = s(m, j) + q j (mn, m) n CL m max k j (s(m, k) + n CL m q k (mn, m) + a(m, k)), Where CL m denotes the set of all annot-link onstraints assoiated with MT P m. Note that if we substitute in the expression ŝ(m, ) = s(m, ) + n CL m q (mn, m) we reover the message update rule of the standard r message (5): r(m, j) = ŝ(m, j) max(ŝ(m, k) + a(m, k)) k j Therefore, the influene of messages oming from all the annot-link onstraints an be seen as a modifiation of similarities to aount for these onstraints. The message sheduling we have hosen alulates iteratively q, a, and r messages until onvergene. One the algorithm terminates we assign to all the points whih hose an MT P as their exemplar the exemplar hosen by that MT P. 3 Experimental evaluation 3.1 User Interative Image Segmentation The partiular appliation we onsider here is user interative image segmentation. There exist many algorithms for unsupervised image segmentation, but in many ases they fail to provide a segmentation that is lose to what a human would onsider appropriate. One reason behind their shortomings an be failing to group together different parts of an objet, if the parts are very different under any reasonable similarity measure between the elements on whih the segmentation is arried out. 2. For example, a human might onsider an image of a person wearing multi-olored and multi-textured lothes as one objet but an automati segmentation will most likely put them in different segments sine olor, texture, and edge ues will all indiate they should be different objets. Another soure of error an be putting together objets that should be separated. For example, if two very similar animals are present in the same image, so that their bodies partially overlap, many segmentation algorithms will group them into one segment. Although it is yet an open question how to overome these errors in a ompletely unsupervised manner, often only a small amount of user intervention is needed in order to orret these types of errors. We are interested in evaluating the usefulness of semisupervised AP for the task of user interative image segmentation. Although the term interative image segmentation usually refers to algorithms geared towards fine separation of bakground from foreground given a user marked ontour of the objet or an area known to ontain the foreground objet, e.g. (Rother 2 These elements an be the image pixels or superpixels - small pixel groups of oherent image regions, obtained by some method of over-segmentation. 165

6 Semi-Supervised Affinity Propagation with Instane-Level Constraints et al., 2004), here we are interested in orretly grouping superpixels, segments of over-segmented images, into possibly several objets. In order to obtain quantitative results for a range of experimental settings, we simulate user interation by randomly seleting a subset of superpixels for whih the algorithm is given the instane-level onstraints. This is repeated 10 times to obtain results for different subsets of training data for eah image. We test the performane aross a range of perentage of points for whih onstraints are provided to the algorithm. Our image data onsists of 23 images. We first obtained 200 superpixels for eah image (Mori, 2005). Eah superpixel was hand-labeled, with eah image ontaining between 2 to 11 distint labels. We then onstruted a similarity measure between superpixels that has a 2D distane omponent, and an equally weighted olor omponent, similar to (Xiao et al., 2007). The first omponent is alulated as the negative squared Eulidean distane between the enters of mass of every pair of superpixels, normalized by the sum of all suh distanes. The olor omponent is the negative squared Eulidean distane between average superpixel olor in Cielab spae, similarly normalized. 3.2 Evaluation Criterion We report the modified Rand index (Rand, 1971; Wagstaff & Cardie, 2000) ahieved by eah method. The Rand index alulates the agreement between two lustering solutions C, Ĉ, where usually one is a lustering algorithm solution (Ĉ), and the other is the true lass labels (C). The index is in the range [0, 1] where 1 indiates a perfet agreement between the lusterings. It an also be interpreted as the probability the two lustering solutions agree on whether two randomly drawn points belong to the same luster or to different lusters. For every pair of points lustered, the points are either in the same luster in both solutions, not in the same luster in both solutions, or the pair of points an be in the same luster aording to one solution and not in the same luster aording to the other solution. The first and seond ases above represent the agreement events between the lustering solutions. The total sum of these events is normalized by the total number of events (the number of all N(N 1) pairs of points, 2 ) and the result is the Rand index. Following (Wagstaff & Cardie, 2000) we alulate this quantity only for pairs for whih no supervised information was given, either diretly or by transdution. Furthermore, as observed by (Xing et al., 2003), this measure tends to give inflated sores when there are many lusters, sine there are many more pairs of points that are not in the same luster than there are pairs of points that are in the same luster, and most algorithms will orretly predit that most pairs are not in the same luster. This an be remedied by giving the same weight to the points that are in the same luster (aording to Ĉ) and those that are not in the same luster. The probabilisti interpretation is the hane of two data points to have agreeing lustering solutions, where the data points are drawn uniformly at random from the same luster (aording to Ĉ) with hane 0.5 and from different lusters with hane 0.5. The following expression is used for alulating the modified Rand index: R(C, Ĉ) = i>j,{i,j} / L [ i = j ĉ i = ĉ j ] 2 i>j,{i,j} / L [ĉ i = ĉ j ] + i>j,{i,j} / L [ i j ĉ i ĉ j ] 2 i>j,{i,j} / L [ĉ. i ĉ j ] L is the set of data-point pairs for whih sideinformation was given to the algorithm. i (ĉ i ) indiates the luster index of point i aording to C (Ĉ). 3.3 Results We ompare our results against Constrained EM (CEM) 3 (Shental et al., 2003). CEM performs expetation maximization (EM) in a Gaussian Mixture Model, where the must-link onstraints are enfored via a modified form of the EM update equations, and the annot-link onstraints are enfored via a Markov Random Field net imposed over the hidden luster assignment variables. It was shown to outperform similar methods (Wagstaff et al., 2001) and (Klein et al., 2002) on a variety of tasks. We also ompare SSAP to standard AP in order to validate that SSAP an be used to improve the results of standard AP. Similarly to AP, SSAP does not take as input the number of lusters to find. Rather, it uses preferenes as a tuning parameter. In order to perform the omparison we first run SSAP, and then use the disovered number of lusters as our input to CEM. Fig. 5 demonstrates an example of segmentation result for some of the images in the data set. Fig. 3 desribes a quantitative analysis of the 3 algorithms. Eah point represents, for a partiular amount of instane-level onstraints, the average modified Rand index aross all 23 images. Eah image was subjeted to SSAP and CEM with 10 different sets of instane-level onstraints. Standard AP does not use the instane-level onstraints, and therefore its Rand index is alulated aross all point pairs, while that of SSAP and CEM is omputed only over data for whih no onstraints were given, as detailed in setion 3.2. Sine the number of lusters found by SSAP hanges as the amount of onstrained information hanges, the orresponding AP solutions with the same number of lusters also varies, 3 ode obtained from 166

7 Givoni, Frey Rand Index AP SSAP CEM % Constrained Data Figure 3: Clustering auray as measured by the modified Rand Index for image segmentation. and therefore we observe different lustering auray for AP for different amounts of onstraints although AP does not use these onstraints. We note that omparison between SSAP and CEM is not straight-forward. Although both are lustering algorithms that utilize instane-level onstraints, and therefore are most similar in terms of their approah, the similarity measure used by SSAP is given as part of the input and is for the user to deide, while CEM assumes a multi-variate Normal distribution of eah luster and performs maximum-likelihood fitting of the distribution parameters. However, the similarities given to SSAP are based on normalized negative Eulidean distane, and are therefore omparable to a normal distribution assumption. Sine every set of labeled data points an be transformed to a list of instane-level onstraints, a natural question is whether SSAP is omparable semisupervised methods that require labeled data. In partiular, we are interested in omparing SSAP to (Xiao et al., 2007) (SSAP-X), whih modified the AP algorithm to aount for partial labels by ontrating all similarly labeled points to a new point, with adjusted similarities, and allowed only the ontrated points to serve as exemplars. When making suh a omparison, it is important to reall that SSAP may still find lusters that have no labeled information, unlike SSAP-X. The modified Rand index may still be biased for a solution with more lusters. Therefore, we report results only for the subset of experiments where the number of lusters found by SSAP was no more than 1.5 times the lusters found by SSAP-X. This redues the number of images for whih the omparison an be fairly made to (1,2,3,6,10,13,19) images for (5%,7%,9%,11%,13%,15%,20%) of labeled data, respetively. The results are shown in Fig. 4. As an be Rand Index AP SSAP CEM SSAP X % Constrained Data Figure 4: Comparison to SSAP-X for a limited subset of the data, for whih omparison an be made, sine SSAP-X requires labeled data and assigns labels to all data-points, unlike SSAP and CEM. seen, at the presene of very little labeled data, the instane-level onstraints appear to be more helpful then the atual labels, possibly sine the number of distint input labels is less than the number of objets in the image, as well as the fat that the set of potential exemplars is extremely limited and not likely representative of the data. 4 Disussion We presented a semi-supervised version of affinity propagation for instane-level onstraints and demonstrated its appliability to user-interative image segmentation. This work was motivated by the observation that segmentation results an be improved with very little user interation. In partiular, this interation an often be restrited to only a subset of the lusters in the image if some lusters are easy to detet in an unsupervised fashion. Therefore, the ombination of instane-level onstraints with a lustering algorithm apable of finding lusters using side information as well as lusters that are not supervised seems appropriate for this task. The SSAP algorithm we have developed for this task performs well in omparison to similar methods. The ultimate goal would be to build a user-interative tool that an iteratively refine results. We believe that introduing more struture to the lustering problem, by reating hierarhies for example, an further improve lustering results. In addition, hierarhies an enable a more natural way to define the user-interation by allowing the user a simple way of indiating how segments should be ombined and split apart. We plan to pursue this diretion in future work. In addition, using the onstraints to 167

8 Semi-Supervised Affinity Propagation with Instane-Level Constraints (a) (b) () (d) (e) (f) Figure 5: Some examples of image segmentation based on instane-level onstraints. Columns from left to right: (a) The original image, (b) superpixels omputed for the image, () hand-labeling of the super-pixels, (d) SSAP results, (e) AP results, (f) Constrained EM results. learn a better similarity matrix for AP is also a natural extension of this work. Referenes Basu, S., Bilenko, M., & Mooney, R. J. (2004). A probabilisti framework for semi-supervised lustering. In Am sigkdd. ACM. Frey, B. J., & Duek, D. (2007). Clustering by passing messages between data points. Siene, 305 (5814), Givoni, I., & Frey, B. J. (2009). A binary variable model for affinity propagation. Neural Computation. Klein, D., Kamvar, S. D., & Manning, C. D. (2002). From instane-level onstraints to spae-level onstraints: Making the most of prior knowledge in data lustering. In ICML. Kshishang, F., Frey, B. J., & Loeliger, H.-A. (2001). Fator Graphs and the Sum-Produt Algorithm. IEEE Transa. Info. Theory, 47 (2), Leone, M., Sumedha, & Weigt, M. (2008). Unspervised and semi-supervised lustering by message passing: soft onstraint affinity propagation. European Phys.J. B. Mori, G. (2005). Guiding model searh using segmentation. In ICCV. Rand, W. M. (1971). Objetive riteria for the evaluation of lustering methods. J. Amerian Stat. Asso., 6 (336), Rother, C., Kolmogorov, V., & Blake, A. (2004). GrabCut : interative foreground extration using iterated graph uts. ACM Trans. Graph., 23 (3), Shental, N., Bar-hillel, A., Hertz, T., & Weinshall, D. (2003). Computing gaussian mixture models with EM using equivalene onstraints. In NIPS. Tarlow, D., Zemel, R., & Frey, B. J. (2008). Flexible priors for exemplar-based lustering. In UAI. Wagstaff, K., & Cardie, C. (2000). Clustering with instane-level onstraints. In ICML. Wagstaff, K., Cardie, C., Rogers, S., & Shroedl, S. (2001). Constrained k-means lustering with bakground knowledge. In ICML. Xiao, J., Wang, J., Tan, P., & Quan, L. (2007). Joint affinity propagation for multiple view segmentation. In ICCV. Xing, E. P., Ng, A. Y., Jordan, M. I., & Russell, S. (2003). Distane metri learning, with appliation to lustering with side-information. In NIPS. 168