Co-segmentation of Non-homogeneous Image Sets

Transcription

1 1 Co-segmentation of Non-homogeneous Image Sets Avik Hati Subhasis Chaudhuri Rajbabu Velmurugan Abstract In this paper, we formulate image co-segmentation as a classification problem in an unsupervised framework with the classes being the common foreground and the remaining regions in the image set. We first find a set of superpixels across all images with high feature similarity such that the constituent superpixels in individual images are spatially compact and label them as seed for the common foreground. Those superpixels with high background probability are labeled as respective seeds for multiple background classes. Seed computation here is unsupervised and automated unlike some semisupervised methods. Then, we compute discriminative features that separate the initially labeled classes using linear discriminant analysis. We use these features to perform spatially constrained label propagation and obtain labels for the unlabeled regions, and iterate this process till the seed regions grow to the common object. Experimental results demonstrate excellent robustness properties even while processing non-homogeneous image sets where the common object is present only in majority of the images. Co-segmentation, outliers, superpixel Index Terms I. INTRODUCTION Image co-segmentation [1], [2] is the process of finding the common object(s) with similar features from a set of two or more images by simultaneously segmenting all the images. See Fig. 1 for illustration. The images to be co-segmented are typically captured under different camera illumination and compositional context (e.g., crowd-sourced images). These variations make co-segmentation extremely difficult. Further in a non-homogeneous dataset, there may be some totally irrelevant images (outliers) that do not contain the common object. This makes co-segmentation even more challenging. Rother et al. [1] first introduced and solved the co-segmentation problem for an image pair and it was followed by the methods in [3], [4], [5] that proposed different optimization techniques for more efficient solutions. Subsequently, the focus A. Hati is the corresponding author ( avik.hati17@gmail.com). A. Hati, S. Chaudhuri and R. Velmurugan are with the Department of Electrical Engineering, Indian Institute of Technology Bombay, India.

2 2 Fig. 1: Co-segmentation result obtained using the proposed method. Row 1 shows a set of six inputs images to be co-segmented. Row 2 shows the co-segmentation output. shifted to co-segmentation of more than two images because of its many practical applications. Some of these works include the methods [6], [7], [8], [9], [10], [11], [12] that jointly segment the images into multiple classes to find foreground objects of different classes. But the methods in [6], [7], [9] require the common object class to be identified manually. Lee et al. [13] and Collins et al. [14] employed random walk for co-segmentation. The method in [15] co-segments different sized common foreground objects by constraining their histograms to have low entropy and to be linearly dependent. Tao et al. [16] use shape consistency among common foreground regions which may be invalid for natural images due to changes in pose and viewpoint. Supervised methods in [17], [18], [19] co-segment a set of similar images using scribbles drawn by users. The methods in [20], [21], [22], [23] first compute image saliency and use it as a key component in their co-segmentation methods. Meng et al. [24] and Chen et al. [21] split the input image set into simple and complex subsets. Subsequently they use results obtained from the simple subset for co-segmentation of the complex one. However, the common object may not always be salient in all constituent images. Learning based co-segmentation algorithms have been proposed in [25], [26], [27]. In order to obtain high-level features, semi-supervised methods in [28], [29], [30] compute region proposals from images using pre-trained networks, whereas Quan et al. [31] use CNN features. As mentioned earlier, the set of crowd-sourced images to be co-segmented may include outlier images that do not at all share the common object present in majority of the images in the set. Most existing methods [6], [7], [9], [11], [12], [13] do not consider this scenario ([32] and [20] do consider this). But the method of [20] being a saliency based approach cannot detect all non-salient co-segments. Wang et al. [32] first learn an object segmentation model from a training image set of same category. Consequently during co-segmentation, the outlier, if any, is rejected due to its non-conformability to the learned model. In this paper, we solve robust co-segmentation of an image set containing outlier images in a fully automated unsupervised framework, yet yielding excellent accuracy.

3 3 Input images Initial BG label & CFG label Compact cluster Superpixel clustering Discriminative features (LDA) Label propagation Common object in each image Fig. 2: Block diagram of the proposed co-segmentation algorithm (BG is background, CFG is common foreground). A. Problem definition In a set of images to be co-segmented, typically (i) the common object regions in different images are concentrated in the feature space since they are similar in feature and (ii) the background varies across images. But, variations in ambient imaging conditions across different images make feature selection difficult which in turn makes it very difficult to extract the common object(s) accurately. Hence, we propose to compute discriminative features (which better distinguish between background and the common foreground) using linear discriminant analysis (LDA) and formulate co-segmentation as a foregroundbackground classification problem. We first find a set of seed superpixels belonging to the common foreground class (C F ) and K B number of background classes (C Bk ) using an unsupervised method. Then we find labels for the remaining superpixels using a feature iterated label propagation method. The key contributions in this paper are The proposed method can handle outlier images. We show that discriminative feature computation through LDA in an unsupervised manner helps to reduce the limitations of low-level and mid-level features. We propose a label propagation technique with spatial constraints, that achieves robust co-segmentation. II. CO-SEGMENTATION ALGORITHM Let I 1, I 2,..., I N be the set of images to be co-segmented and the common object is present in M(< N) of them (M is unknown). First, we segment them into superpixels {s} using the simple linear iterative clustering (SLIC) method [33]. Then we compute low-level and mid-level features (Lab color, SIFT and bag-of-words feature) and define an object to be a set of contiguous superpixels with similar features. The proposed algorithm uses seed superpixel selection, discriminative feature computation and label propagation (LP) as shown in the block diagram in Fig. 2 and illustrated using Fig. 3, Fig. 4, Fig. 5.

4 4 Fig. 3: Illustration of superpixel clustering of six images including one outlier image. All superpixels from all images are clustered into R = 8 clusters based on their features. Row 1: Input image set to be co-segmented with I 1 being the outlier image. Rows 2-9: Clusters 1-8 and the sub-images constituted by the superpixels in every cluster. Cluster 5 (shown in Row 6) has the highest compactness value and superpixels from this cluster are chosen as seeds for the common foreground C F. A. Selection of superpixel seeds Typically, the set of superpixels constituting the common object in multiple images have similar features and they are closer to each other in the feature space compared to the background superpixels. Under this assumption, we first cluster all superpixels from all images based on their features. Let S ij be the superpixel subset of I i belonging to the j th cluster, assuming R clusters. Figure 3 shows an example considering R = 8 clusters. Since the common object superpixels in every image I i are spatially compact, we find the seeds for the common foreground C F as the set of superpixels belonging to the most spatially compact cluster. We define compactness of cluster-j as Γ j = ( N i=1 σ 2 ij A ij ) 1, where σ 2 ij is the spatial variance of centroids of superpixels in S ij and area A ij is the total number of pixels constituting them. For an image with a segmentable object, the superpixels should be spatially close, indicated by high Γ. Hence, we define C F = {s N i=1 S ip} where p = arg max j Γ j is the most compact cluster. The

5 5 Fig. 4: Illustration of seed labeling. Row 1: Input image set to be co-segmented. Row 2: Sub-images containing regions constituted by the common foreground (C F ) seed superpixels (in every image) that are obtained from the most compact cluster after superpixel feature clustering. Rows 3-6: Sub-images containing regions in every image constituted by the background (C Bk ) seed superpixels, ensuring P(s) > 0.99, in every image. sub-images constituted by C F is shown in Row 6 of Fig. 3 (and also in Row 2 of Fig. 4). We do not use information of remaining clusters further. To obtain seeds for the background, we use the background probability measure P( ), obtained from geodesic distance of every superpixel from image boundary [34]. We find the set of background seed superpixels with high confidence as S B = {s N i=1 I i : P(s) > 0.99}. As the superpixels in S B belong to different images, they usually have highly varying features. Grouping feature points with large variance together may lead to poor convergence during the LP stage. So, we cluster them into K B clusters (different from the clusters used of Γ computation) and obtain C B1, C B2,.... Hence, the total number of classes for labeling is K = 1 + K B. Rows 3-6 in Fig. 4 show background seed regions in four clusters. B. Feature iterated label propagation The set of seed superpixels are used to label the remaining superpixels. To achieve a more accurate labeling and robust co-segmentation, we first project the computed low-level and mid-level features (x R d ) onto a space that better discriminates foreground from background and obtain discriminative features. We use LDA to find the optimal discriminants and use them for feature projection [35] as

6 6 described next. For simplicity, let C 1 = C F, C 2 = C B1, C 3 = C B2,... Let X R d nt contains feature vectors of all (say n T ) labeled superpixels. Let (n k, m k ) be the cardinality of C k and the mean of feature vectors in C k, n T = K k=1 n k, and m be the mean of all feature vectors in all classes. We define the inter-class (or between-class) scatter matrix Q b R d d and the intra-class (or within-class) scatter matrix Q w R d d as in [35] and Q b = K k=1 n k n T (m k m)(m k m) T (1) Q w = K k=1 n k n T Q k, (2) where Q k is the covariance matrix of feature vectors in C k. We compute an appropriate projection matrix W = [w 1 w 2... w dr ], with w i R d and d r < d, by maximizing the following cost function (see [35] for details). cost(w) = tr{(v w ) 1 V b } such that w T i w i = 1, (3) where V w = W T Q w W and V b = W T Q b W are the intra-class and inter-class scatter matrices in the projected domain and tr{ } denotes trace. In Eqn (3), a large value of tr{v b } ensures good separation among classes and a small value of tr{v w } ensures less overlap among classes in the projected domain. The solution W, that contains the discriminants w i, is determined by the eigenvectors corresponding to the d r largest eigenvalues of Q 1 w Q b [35]. Here, max(d r ) = K 1 as Q 1 w Q b has rank at most equal to K 1. We project every feature vector x (labeled as well as unlabeled) as z = W T x. In Sec. II-A, we obtained labels of seed superpixels and our aim is to grow them to find the common object by assigning labels to the remaining superpixels. We perform region growing in two stages. First, we perform LP considering all superpixels (say n A ) from all images simultaneously using the discriminative features (z). Then we prune the updated superpixel labels in every image independently using spatial constraints as described next. We assign every seed superpixel s i belonging class C j, a label L = j and define a binary seed label matrix Y R na K Y(i, j) = { 1 if superpixel si has label L = j (4a) 0 otherwise. (4b) Here, Y has class information of only the seed superpixels. Our aim is to obtain an optimal label matrix L R na K with class information of all n A superpixels from all images.

7 7 First, we compute the feature similarity matrix S R na na where S ij is the negative exponential of the normalized distance between z i and z j and D is a diagonal matrix with D ii = j S ij. To obtain the optimal L, we minimize the following cost function n A i,j=1 1 S ij L i 1 L j 2 + α Dii Djj n A i=1 L i Y i 2 (5) where α > 0 is a regularization parameter and L i is the i th row of L. The first term (smoothness term) updates L using the similarity matrix S. Thus, labels are assigned to unlabeled superpixels through label propagation from the labeled superpixels. The second term (data fitting term) minimizes the difference between L and Y. It has been shown in [36], that minimization with respect to L yields where β 1 = L = β 1 (I β 2 D 1/2 SD 1/2 ) 1 Y (6) α 1+α, β 2 = 1 1+α. We obtain the label of s i as L = arg max j L (i, j), under constraints J1, J2. (7) Every row and column of Y correspond to a superpixel and a class, respectively. If the number of superpixels in one class C j is significantly large compared to the remaining classes (C k, k j), the columns of Y corresponding to C k s will be sparse. In such scenario, the solution to Eqn (7) will be biased towards C j. Hence, we normalize every column of Y by its L 0 -norm. Next, we update this solution using two constraints. J1: L (i, j) is a measure of similarity of superpixel s i to the set of superpixels with label j. If L (i, j) is small for all j = 1, 2,..., K (i.e., max j L (i, j) < t l ), these similarity values may lead to wrong label assignment; so we do not label that superpixel s i. We set t l to be median(l ). J2: The label update formulation in Eqn (5) does not use any spatial information of superpixels. Hence, every newly labeled superpixel (based only on feature similarity) may not be a neighbor of the seed regions in that sub-image belonging to a certain class and that sub-image may contain many discontiguous regions. But typically objects and background regions, e.g., sky, field and water body are contiguous regions. Hence we add a spatial constraint to Eqn (5) so that an unlabeled superpixel s i will be considered for assignment of label L = j using Eqn (7) only if it belongs to the first order spatial neighborhood of an already labeled region (with L = j) in that sub-image. Due to the above two constraints, only a limited number of superpixels in the spatial neighborhood of already labeled superpixels are assigned labels. After this label updating, we consider all labeled superpixels to again compute discriminative features from original feature vectors and perform label propagation again. We iterate these two stages alternately until convergence as shown in the block diagram in Fig. 2. The iteration converges if

8 8 Fig. 5: Co-segmentation output. Row 1: Input image set to be co-segmented. Row 2: Co-segmented objects constituted by the common foreground superpixels (in every image) that are obtained after convergence of feature iterated label propagation. Rows 3-6: the updated background clusters after convergence. either there is no more unlabeled superpixel left or labels no longer get updated. Row 2 in Fig. 5 shows the final co-segmentation result after convergence. As α in Eqn (5) is non-zero, initial labels of the labeled superpixels also get updated. We set α = 0.2. The proposed algorithm is given as a pseudocode in Algorithm 1. III. RESULTS Choice of features and datasets: We compute dense SIFT and CSIFT features from all images and encode them using locality-constrained linear coding [37], with the codebook size being 100, to obtain mid-level features. We have used L a b mean color feature (length 3) as low-level features. Hence, the feature dimension D = = 203. Unlike other methods [7], [13], we process all images in their original sizes and process all images simultaneously. We use the commonly used icoseg dataset [17] that contains images of 38 homogeneous classes and create a much larger dataset by embedding each class with several outlier images randomly chosen from other classes. Each set may have upto 30% of the data as outlier images. This dataset has an overwhelmingly large 570 sets containing a total of 9,563 non-unique images (given in the supplementary material). Similarly, we have created another non-homogeneous dataset from the MIT object discovery

9 9 TABLE I: Comparison of Jaccard similarity (J) of the proposed method (PM) with [6], [7], [9], [11], [13], [20] on the non-homogeneous datasets created using the icoseg dataset and the MIT dataset. J \ Methods PM [13] [11] [9] [20] [6] [7] icoseg MIT dataset internet dataset [20] for experiments. We have computed results for other methods using the codes provided by the authors. A. Quantitative results We have used Jaccard similarity (J) [20] as the metric to quantitatively evaluate the performance of the proposed method (PM) and compare with state-of-the-art methods in [6], [7], [9], [11], [13], [20], which fall into the same class of unsupervised co-segmentation. Jaccard similarity is defined as the intersection over union between the ground-truth and the binary mask of the co-segmentation output. We compare the values of J obtained from 570 sets of images, with each set containing 5-50 images, using PM and the existing methods in Table I. For the methods in [6], [7], [9], output segmentation class has to be manually chosen for each dataset. The poor performance of the method in [20] is due to the use of saliency as a cue as discussed in Sec. I. We found the number of initial clusters R [6, 8] and background classes K B [3, 4] yield similar results. The quantitative analysis clearly shows the superiority of PM. These results are obtained without performing any post-processing (e.g., GrabCut). B. Qualitative comparison In Fig. 6, we show visual comparison of the co-segmentation outputs obtained using PM with that of [20], [9], [6], [13], [11] on the pyramid images from the icoseg dataset that also includes one outlier image (Image 1) from goose subset. The proposed method correctly co-segments only pyramids (Row 7), whereas the methods in [9], [6], [13], [11] (Rows 3-6) incorrectly detect some background regions and horse in I 4, I 5, I 7, and the method in [20] (Row 2) incorrectly detects the sky as the common object. Moreover, they cannot handle the presence of outlier image and wrongly co-segment part of the goose (regions of significant size in case of [13]) from them unlike PM. More results are shown in Fig. 8, Fig. 9, Fig. 10, Fig. 11. In Fig. 7, we show that the proposed method can detect common objects of multiple classes by selecting foreground seeds from the second most compact cluster defined in Sec. II-A, after having detected the common object belonging to the previous class.

10 10 Fig. 6: Visual comparison of co-segmentation results on a set of 7 images (Row 1) from the icoseg dataset with I1 being the outlier. Results obtained using methods in [20], [9], [6], [13], [11] and PM are shown in Rows 2-7, respectively. Row 8: ground-truth. IV. C ONCLUSIONS We propose a co-segmentation algorithm that finds the common objects in an image set using a spatially constrained label propagation (LP) method that grows initially computed seed regions by assigning labels to the remaining regions. We do not use high level information like region proposals, saliency or CNN features that are used in semi-supervised methods. Instead we used only low and mid-level features to find seeds, which are used to obtain discriminative features for better feature representation in a completely unsupervised manner. LP in conjunction with LDA results in a more accurate labeling, thus yielding robust segmentation. The spatial constraint at the LP stage ensures spatially compact co-segmented objects.

11 11 Fig. 7: Illustration of multiple class co-segmentation. Row 1: Input image set. Rows 2, 3: Common objects in class 1 (helicopter) and class 2 (balloon), respectively. Fig. 8: Row 1: Input image set to be co-segmented with I6 being the outlier image. Rows 2-6: Cosegmentation output of methods in [20], [9], [6], [13], [11], respectively. Row 7: Co-segmentation output of the proposed method. Row 8: Ground-truth. R EFERENCES [1] C. Rother, T. Minka, A. Blake, and V. Kolmogorov, Cosegmentation of image pairs by histogram matching - incorporating a global constraint into MRFs, in Proc. CVPR, June 2006, vol. 1, pp

12 12 Fig. 9: Row 1: Input image set to be co-segmented with I7 being the outlier image. Rows 2-6: Cosegmentation output of methods in [20], [9], [6], [13], [11], respectively. Row 7: Co-segmentation output of the proposed method. Row 8: Ground-truth. [2] A. Hati, S. Chaudhuri, and R. Velmurugan, Image co-segmentation using maximum common subgraph matching and region co-growing, in Proc. ECCV, 2016, pp [3] L. Mukherjee, V. Singh, and C. Dyer, Half-integrality based algorithms for cosegmentation of images, in Proc. CVPR, June 2009, pp [4] D. Hochbaum and V. Singh, An efficient algorithm for co-segmentation, in Proc. ICCV, Sept 2009, pp [5] S. Vicente, V. Kolmogorov, and C. Rother, Cosegmentation revisited: Models and optimization, in Proc. ECCV, 2010, pp [6] A. Joulin, F. Bach, and J. Ponce, Discriminative clustering for image co-segmentation, in Proc. CVPR, June 2010, pp [7] G. Kim, E. Xing, L. Fei-Fei, and T. Kanade, Distributed cosegmentation via submodular optimization on anisotropic diffusion, in Proc. ICCV, Nov 2011, pp

13 13 Fig. 10: Row 1: Input image set to be co-segmented with I1 being the outlier image. Rows 2-6: Cosegmentation output of methods in [20], [9], [6], [13], [11], respectively. Row 7: Co-segmentation output of the proposed method. Row 8: Ground-truth. [8] G. Kim and E. Xing, On multiple foreground cosegmentation, in Proc. CVPR, June 2012, pp [9] A. Joulin, F. Bach, and J. Ponce, Multi-class cosegmentation, in Proc. CVPR, June 2012, pp [10] F. Wang, Q. Huang, M. Ovsjanikov, and L. J. Guibas, Unsupervised multi-class joint image segmentation, in Proc. CVPR, 2014, pp [11] H.-S. Chang and Y.-C. F. Wang, Optimizing the decomposition for multiple foreground cosegmentation, Elsevier Comput. Vis. Image Understand., vol. 141, pp , [12] J. Ma, S. Li, H. Qin, and A. Hao, Unsupervised multi-class co-segmentation via joint-cut over L1 -manifold hyper-graph of discriminative image regions, IEEE Trans. Image Process., vol. 26, no. 3, pp , [13] C. Lee, W.-D. Jang, J.-Y. Sim, and C.-S. Kim, Multiple random walkers and their application to image cosegmentation, in Proc. CVPR, June 2015, pp [14] M. Collins, J. Xu, L. Grady, and V. Singh, Random walks based multi-image segmentation: Quasiconvexity results and

14 14 Fig. 11: Row 1: Input image set to be co-segmented with I1 being the outlier image. Rows 2-6: Cosegmentation output of methods in [20], [9], [6], [13], [11], respectively. Row 7: Co-segmentation output of the proposed method. Row 8: Ground-truth. GPU-based solutions, in Proc. CVPR, June 2012, pp [15] L. Mukherjee, V. Singh, and J. Peng, Scale invariant cosegmentation for image groups, in Proc. CVPR, June 2011, pp [16] W. Tao, K. Li, and K. Sun, Sacoseg: Object cosegmentation by shape conformability, IEEE Trans. Image Process., vol. 24, no. 3, pp , [17] D. Batra, A. Kowdle, D. Parikh, J. Luo, and T. Chen, icoseg: Interactive co-segmentation with intelligent scribble guidance, in Proc. CVPR, June 2010, pp [18] X. Dong, J. Shen, L. Shao, and M.-H. Yang, Interactive cosegmentation using global and local energy optimization, IEEE Trans. Image Process., vol. 24, no. 11, pp , [19] W. Wang and J. Shen, Higher-order image co-segmentation, IEEE Trans. Multimedia, vol. 18, no. 6, pp , 2016.

15 15 [20] M. Rubinstein, A. Joulin, J. Kopf, and C. Liu, Unsupervised joint object discovery and segmentation in internet images, in Proc. CVPR, June 2013, pp [21] M. Chen, S. Velasco-Forero, I. Tsang, and T.-J. Cham, Objects co-segmentation: Propagated from simpler images, in Proc. ICASSP, 2015, pp [22] L. Lattari, A. Montenegro, and C. Vasconcelos, Unsupervised cosegmentation based on global clustering and saliency, in Proc. ICIP, 2015, pp [23] K. R. Jerripothula, J. Cai, and J. Yuan, Image co-segmentation via saliency co-fusion, IEEE Trans. Multimedia, vol. 18, no. 9, pp , [24] F. Meng, J. Cai, and H. Li, Cosegmentation of multiple image groups, Elsevier Comput. Vis. Image Understand., vol. 146, pp , [25] J. Sun and J. Ponce, Learning dictionary of discriminative part detectors for image categorization and cosegmentation, Intl. Journal Comput. Vis., vol. 120, no. 2, pp , [26] J. C. Rubio, J. Serrat, A. López, and N. Paragios, Unsupervised co-segmentation through region matching, in Proc. CVPR, 2012, pp [27] Z. Yuan, T. Lu, and Y. Wu, Deep-dense conditional random fields for object co-segmentation, in Proc. IJCAI, 2017, pp [28] S. Vicente, C. Rother, and V. Kolmogorov, Object cosegmentation, in Proc. CVPR, June 2011, pp [29] K. Li, J. Zhang, and W. Tao, Unsupervised co-segmentation for indefinite number of common foreground objects, IEEE Trans. Image Process., vol. 25, no. 4, pp , [30] Y. Li, L. Liu, C. Shen, and A. van den Hengel, Image co-localization by mimicking a good detectors confidence score distribution, in Proc. ECCV, 2016, pp [31] R. Quan, J. Han, D. Zhang, and F. Nie, Object co-segmentation via graph optimized-flexible manifold ranking, in Proc. CVPR, 2016, pp [32] L. Wang, G. Hua, J. Xue, Z. Gao, and N. Zheng, Joint segmentation and recognition of categorized objects from noisy web image collection, IEEE Trans. Image Process., vol. 23, no. 9, pp , [33] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, SLIC superpixels compared to state-of-the-art superpixel methods, IEEE Trans. PAMI, vol. 34, no. 11, pp , [34] W. Zhu, S. Liang, Y. Wei, and J. Sun, Saliency optimization from robust background detection, in Proc. CVPR, 2014, pp [35] C. M. Bishop, Pattern recognition and machine learning, Springer, [36] D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf, Learning with local and global consistency, in Advances in NIPS, 2004, pp [37] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, Locality-constrained linear coding for image classification, in Proc. CVPR, 2010, pp

16 Algorithm 1 Proposed co-segmentation algorithm Input: Set of images I 1, I 2,..., I N Output: Set of superpixels in all images belonging to the common objects 1: for i = 1 to N do 2: Superpixel segmentation of every image I i 3: Compute background probability P(r) of every superpixel r I i 4: end for 5: Compute feature x for every suprepixel r using LLC from SIFT, CSIFT and L a b mean color 6: // Initial labeling 7: Cluster all superpixels {x N i=1 I i} from all images into R clusters based on their features 8: for j = 1 to R do 9: for i = 1 to N do 10: S ij superpixel subset of I i belonging to the j th cluster 11: A ij total number of pixels constituting the superpixels in S ij 12: σij 2 spatial variance of centroids of superpixels in S ij 13: end for 14: Compute compactness of every cluster Γ j 15: end for ( N i=1 ) σ 2 1 ij A ij 16: Find the most compact cluster p arg max j Γ j 17: Common foreground seed C F {r N i=1 S ip} 18: Cluster {r N i=1 I i : P(r) > 0.99} in K B clusters and find background seeds as clusters C B1, C B2, C B3,... 19: K K B : Initialize t 0, C (0) 1 C F, C (0) 2 C B1, C (0) 3 C B2, C (0) 4 C B3,... 21: For each k = 1, 2,..., K, assign label L (0) r k, r C k 22: // Label propagation 23: while no convergence do 24: Find discriminant matrix W from features in C (t) 1, C(t) 2, C(t) 3,... using LDA 25: Project every feature vector x (labeled as well as unlabeled) as z W T x 26: Compute similarity matrix S where S(i, j) similarity(z i, z j ) 27: Compute diagonal matrix D where D(i, i) j S(i, j) 28: for k = 1 to K do 29: for r i C (t) k do 30: Y(i, k) 1 C (t) k 31: end for 32: end for 33: L (I αd 1/2 SD 1/2 ) 1 Y 34: t l median(y ) 35: // Label update 36: for all r i do 37: if max k 38: L (t+1) r i L (i, k) > t l r i N s ( K arg max L (i, k) k 39: end if 40: end for 41: For each k = 1, 2,..., K, C (t+1) k k=1 C(t) k ) then {r : L (t+1) r = k} 42: t t : end while 44: C F = C 1 at convergence is the set of superpixels in all images constituting the common object 16