On Clustering Images of Objects

Transcription

1 On Clustering Images of Objects Jongwoo Lim University of Illinois at Urbana-Champaign Honda Research Institute

2 Personal Introduction Education B.S. Seoul National University, 1993~1997 Ph.D. student at University of Illinois at Urbana-Champaign, 2000 ~ M.S. University of Illinois at Urbana-Champaign, 2003 Visiting, University of California at San Diego, 2002~2005 Experience Honda Research Institute, 2003, 2005~ Summer Intern : ASIMO face detection/recognition/tracking system Senior Research Scientist UIUC Active Space project, 2001~2005 Research Assistant : people tracking system Triton Tech, 1997~2000 Assistant Manager / Programmer : electronic document management system [2]

3 Publications Beyond Pairwise Clustering S. Agarwal, J. Lim, L. Zelnik, P. Perona, D. Kriegman, S. Belongie / CVPR 2005, vol. 2, pp CVPR05 Image Clustering with Metric, Local Linear Structure and Affinity Symmetry J. Lim, J. Ho, M-H. Yang, K-C. Lee, D. Kriegman / ECCV 2004, vol. 1, pp Clustering Appearances of Objects Under Varying Illumination Conditions J. Ho, M-H. Yang, J. Lim, K-C. Lee, D. Kriegman / CVPR 2003, vol 1, pp PhD Thesis Passive Photometric Stereo from Motion J. Lim, J. Ho, M-H. Yang, D. Kriegman / ICCV 2005, to appear ICCV05, accepted A Direct Method for Modeling Non-rigid Motion with Thin Plate Spline J. Lim, M-H. Yang / CVPR 2005, vol. 1, pp CVPR05 Incremental Learning for Visual Tracking J. Lim, D. Ross, R-S. Lin, M-H. Yang / NIPS 2004 Adaptive Discriminative Generative Model and Its Applications R-S. Lin, D. Ross, J. Lim, M-H. Yang / NIPS 2004 Probabilistic Visual Tracking with Incremental Subspace Update D. Ross, J. Lim, M-H. Yang / ECCV 2004, vol. 2, pp Tracking Humans Using Prior and Learned Representations of Shape and Appearance J. Lim, D. Kriegman / FGR 2004, pp Master Thesis [3]

4 Outline Introduction Generic Clustering Algorithms Normalized Cut / Spectral Clustering Hypergraph Clustering Clustering Images When illumination condition changes. When pose (viewing direction) changes. When pose and illumination changes simultaneously. Conclusion [4]

5 What is the Clustering Problem? Grouping data elements with similar characteristics. E.g., Perceptual grouping of points in 2D Position or distance in the space. [Zelnik,Perona NIPS04] [5]

6 Another Clustering Example Clustering news articles TORONTO, ONTARIO: Air France crash pilot to be quizzed 2005/08/04 BEIJING, CHINA: NK: Nuclear talks 'at a stalemate' 2005/08/04 SAN JOSE, COSTA RICA: Swimming with dolphins, sharks barred 2005/08/04 LONDON, ENGLAND: Blair in anti-hatred crackdown 2005/08/05 ADDIS ABABA, ETHIOPIA: African leaders discuss U.N. role 2005/08/04 WASHINGTON: Poll: Bush's Iraq rating at low point 2005/08/05 MOSCOW, RUSSIA: Trapped sub surfaces, crew safe 2005/08/06 Various ways to cluster articles : region, date, topic, etc. Each field has different characteristics. Text string, Time, Geographic Location, Number, etc. [ [6]

7 Clustering Algorithms Parametric Model-based Approaches Graph-based Approaches [7]

8 Graph-based Clustering Similarity = Edge Weight in a Graph Clustering = Partitioning a complete graph Spectral clustering, Normalized cut, etc. Pros and Cons No metric space assumption. Discrete or categorical fields can be handled. Ability to handle all possible partitions. Computationally expensive. [8]

9 Measuring Similarity How is the Similarity determined? Affinity in [0,1] : from totally different to identical. It depends on the type of data and the goal of clustering. (Pairwise) Affinity Matrix (i,j) element represents the similarity between data i and j. A symmetric and non-negative matrix. [9]

10 From Distance To Affintiy Often the distance is easier to measure from data. Conversion function : A D [10]

11 Spectral Clustering How to assign each point into a group? Use the eigenvectors of the normalized affinity matrix associated with the k largest eigenvalues. Run k-means algorithm on the embedded points. [Ng,Jordan,Weiss NIPS01] [11]

12 Partitioning Graphs : Normalized Cut Minimum Cut vs. Normalized Cut A B cut A B assoc [WuLeahy PAMI93] [Shi,Malik PAMI00] [12]

13 K-way Normalized Cut K-way Normalized Cut S 2 Use the eigenvectors of L associated with the 2 nd ~(k+1) th eigenvalues. Treat each row of the eigenvector matrix as a point in K. Run k-means algorithm on the embedded points. S 1 R S 3 [Bach,Jordan NIPS03] [13]

14 Multi-adic Similarity Measure What if a measure of similarity is not available for pairs of data elements, but one is available for larger groups of elements? For example, consider k-lines clustering problem. Every pair of points trivially defines a line. At least three points are needed to measure the fitting error. [14]

15 Graph vs. Hypergraph Pairwise similarity : Graph edge = Multi-adic similarity : Hyperedge The similarity value = the (hyper-)edge weight. Hyperedges can contain any number of vertices. Hypergraph A set of vertices (i.e. data elements). A set of hyperedges { z ; z = {i, j, k, l} }. D h(z) i j k l z = { i, j, k, l } A weight function h(z). [15]

16 Working with Hypergraphs Work directly on a hypergraph? It is not computationally feasible. ( # of hyperedges = ) Few analysis methods are available. Approximate with a graph. Some information will be lost. However, we only need to preserve the cluster structure. [16]

17 Approximation of a Hypergraph How to convert hyperedges to edges? Star Expansion Replace each hyperedge with one additional vertex and k edges. h(z) d c b a d c e a b g(a,e) h(z) Clique Expansion Create a clique on the vertices with edges. h(z) d c b a d c a b g(a,b) g(a,b) + h(z) [Hu,Morder VLSICircuitLayout 85] [Hadley DiscreteApplMath 95] [17]

18 Clique Averaging Problems in previous approaches Star Expansion creates too many new vertices. Clique Expansion just accumulates the hyperedge weights to the corresponding graph edges. Optimally distribute a hyperedge weight to its clique edges. h(z) d c b a d c a b [Agarwal,Lim,Zelnik,Perona,Kriegman,Belongie CVPR05, to appear] [18]

19 Clique Averaging Solve a constrained linear system of the equations. Solving where is a zero-one matrix s.t. ij is 1 if edge j is incident on hyperedge i, and 0 o.w. A highly over-determined, sparse and low-rank linear system. is huge : subsample the hypergraph. [19]

20 Hypergraph Clustering Algorithm [20]

21 Experiment: k-curves problem Synthetic Data: Cluster points sampled from k curves. Curves are in 5D space and pass the origin. Curves are gently curved and points are sampled with noise. 350 points from 5 curves. Similarity measure Pick 3 points. Compute the 2 nd singular value s 2 after subtracting the mean. [21]

22 Experiment: k-curves problem Pilot Experiment : parameter sweep Experimental Result Algorithms Error CliqueAverage 12.6 CliqueExpansion 12.9 GibsonSum 17.3 GibsonProd 55.1 KHMETIS 18.0 CRANSAC 23.4 [22]

23 Experiment: Yale Face Database Cluster images over illumination variation. The dataset contains 450 images of 10 people. Images are cropped and resized into Similarity measure Pick 4 images. Compute 4 th singular value s 4. [23]

24 Experiment: Yale Face Database Pilot Experiment : parameter sweep (7 people) [24]

25 Experiment: Yale Face Database # people CliqueAverage 4.2/ / / /3.0 CliequeExpansion 11.8/ / / /4.3 CRANSAC 16.2/ / / /6.6 KHMETIS 21.5/ / / /3.3 GibsonSum 25.9/ / / /2.1 GibsonProd 67.4/ / / /0.7 Average error rate % / standard deviation over 30 randomly sampled sub-datasets. [25]

26 Discussion: Hypergraph Clustering Multi-adic similarity measure In many problems, this is a useful affinity measure. Find an approximate graph, then cluster it. Clique Averaging It generates better and more robust approximate graphs. The relation to other algorithms is presented. [26]

27 Images of an Object Images of an object may look very different, due to External / Environmental causes When pose (or viewing direction) changes. When lighting changes. Intrinsic causes Articulation, deformation. Human face Facial expression changes. Glasses, facial hair. Aging. [27]

28 Image as a Point in the Image Space Image of size (w h) Point in (wh)-dim Space w h wh [28]

29 Image Clustering over Illumination Objects under different lightings but in a fixed pose. [Ho,Yang,Lim,Lee,Kriegman CVPR03] [29]

30 Illumination Cone Illumination Cone Theorem The set of n-pixel images of any object, seen under all possible lighting conditions, is a convex cone in n. Derived from the superposition property of illumination. An image in the cone can be written as a convex combination of other images in the cone. R = α 1 + α 2 + α 3, α 1, α 2, α 3 0 [Belhumeur,Kriegman IJCV98] [30]

31 Similarity Measure #0 Multi-adic Subspace measure The cone is approximated by a low-dimensional subspace. Images of a Lambertian object under a distant point light source without any shadow : 3D subspace. For 4 images, h(z) = e( /σ) h(z) = e( /σ) h(z) = e( /σ) [31]

32 Similarity Measure #1 Conic Affinity If I = α 1 I α k I k, α i 0, then α i (I I i ) / I i 2 and α i I / I i. Each α i encodes the similarity of I i to I. The conic affinity of an image shows its relation to other images in the set. Global measure conic linear affinity [32]

33 Gradient Images and Illumination Statistical Image Gradient Analysis Image gradient depends on both object geometry and surface reflectance property (albedo). The magnitude and orientation of image gradient form a joint probability distribution. [Chen,Belhumeur,Jacobs IJCV,CVPR00] [33]

34 Similarity Measure #2 Gradient Affinity Compare the difference in the direction and strength of the image gradient vector at each pixel in two images. Pairwise measure [34]

35 Experiments: YaleB and PIE database [Georghiades,Belhumeur,Kriegman PAMI01] [Sim,Baker,Bsat CMUTR01] [35]

36 Experiments: YaleB and PIE database Yale B Face Database 450 images of 10 people under 45 different lighting. Single directional light sources. Two poses : frontal and profile views. CMU PIE Database 1386 images of 66 people under 21 different lighting. Directional light + ambient background light. PIE20 : 10 datasets of randomly selected 20 people. [36]

37 Experiments: YaleB and PIE database Methods Yale B (frontal) Yale B (non-frontal) PIE66 (frontal) Conic Affinity 0.44 % 4.22 % 4.18 % Gradient Affinity 1.78 % 2.22 % 3.97 % L 2 + k-means % % % L 2 + Spectral % % % Minimum error rate % [37]

38 Conic Affinity Conversion Function More control on the effective affinity range Original conic affinity : A = (W + W T ) / 2 Modified affinity : A A σ = 10 σ = 100 [38]

39 Parameter Sweep Figure Error Rate Error Rate % # Nearest Neighbor log 10 (σ) [39]

40 Parameter Sweep on Conic Affinity Yale10 YalePose10 PIE20 [40]

41 Parameter Sweep on Gradient Affinity Yale10 YalePose10 PIE20 [41]

42 Image Clustering over Pose Variation Objects seen from different Viewing Directions (under same Illumination condition). [Lim,Ho,Yang,Lee,Kriegman ECCV04] [42]

43 How are the images related? Appearance Manifold Images of an object taken from a moving camera (equivalently a moving object from a fixed camera) form a complex but locally-connected structure in the image space. Geodesic distance/relation is more important than Euclidean distance. [Murase,Nayar IJCV95] [43]

44 Secant Chord Approximation x can be approximated by its neighbors y 1 and y 2. x In general, x can be approximated with y s as y 1 y 2 and weights are recovered by solving Secant Chord y 1 x y 3 y 2 [44]

45 Handling Small Pose Variation 2D Affine warp : translation, rotation, scale, skew Preserves parallelism. 6 parameters : (t x, t y, θ, s, α, φ) Why do we use the 2D affine warp? Α Appearance changes due to small 3D motions can be well approximated by 2D affine warps. = A = [45]

46 Local Linear Structure LLS Affinity Measure Each a i can be used as an affinity measure between an image x and affine-corrected y i. Only y i s close to x are considered : Local measure [46]

47 Experiments: COIL20, VEH10 COIL20 VEH10 [47]

48 Experiments: COIL100 [48]

49 Experiments: COIL Pose Variation Top: 5 degrees apart, Middle : 10 degrees apart, Boottom : 20 degrees apart [49]

50 Experiments: FACE10 Top: 10 subjects in the dataset, Bottom: typical pose variation [50]

51 Experiments: Clustering Result Datasets Error rate (%) FACE images 0.00 COIL20 COIL20.2 COIL20.4 5, 1440 images 10, 720 images 20, 360 images COIL20s 10 random, 500 images (min.avg.) VEH10.2 COIL100.2 COIL , 360 images 10, 3600 images 20, 1800 images [51]

52 Experiments: Comparison Datasets COIL20.2 COIL20.4 FACE10 VEH10.2 Proposed Algorithm 0.00 % % 0.00 % % Affine+KNN+Spectral 7.36 % % % % Affine + Spectral % % % % Euclidean + Spectral % % % % Euclidean + k-means % % % % [52]

53 Parameter Sweep on LLS COIL20.2 COIL20.4 COIL20s [53]

54 Images with Light & Pose Variation Pose and Illumination changes simultaneously. [54]

55 YCOIL dataset Yale B face database + COIL database 18 Objects on a turntable (24 poses) Yale lighting dome : 64 lights per pose Preprocessing Resize and crop tightly (64x64 pixels) Add simulated ambient light (optional) Exclude extreme lights : use subset 1~3. Flash Positions [Personal Communication with A. Georghiades at Yale univ.] [55]

56 Experiments: YCOIL05a vs. YCOIL05b Simulated Ambient Lighting No ambient light in the original YCOIL. Hard to have correct object bounding boxes. Strong cast and attached shadows. Average images with same pose over lighting variations. YCOIL05b : 0.6 * OrgImg * AmbImg. Simulated Ambient Images [56]

57 Pose-Light Datasets Simulated Ambient Light : YCOIL05a / YCOIL05b Original YCOIL dataset is collected without any ambient lighting. 300 randomly picked images of 5 randomly chosen objects. Larger Experiment : YCOIL10 With simulated ambient lights (as YCOIL05b) 500 randomly picked images of 10 randomly chosen objects. [57]

58 Geodesic Expansion of Affinity Matrix Geodesic distance Sparsely connected graph Fill the distance matrix with geodesic distances between points. Affinity Distance : D ij = 1 / A ij. Sparse Affinity Sparse Distance Full Distance Full Affinity Geodesic Distance [Image from Tenebaum et. al. Science00] [58]

59 Empirical Joint PDF on Image Gradients Build an estimate of a probability distribution from real data. Image gradient vectors at each pixel in image pairs of same object, same pose, different lighting. Yale B face database JPDF over 2 variables : angular difference, magnitude difference JPDF over 3 variables : angular difference, two magnitudes 2D Joint PDF from Yale B database [59]

60 Local Affine-corrected Gradient Affinity Affine correction on nearby images Similarity is computed by Image Gradient JPDF 2D JPDF : LGA2D 3D JPDF : LGA3D [60]

61 Experiments: YCOIL Results YCOIL05a YCOIL05b YCOIL10 LLS 15.13% 11.13% 29.93% LGA2D 34.47% 10.20% 19.24% LGA3D After Geodesic Expansion 27.50% 6.77% 17.99% Spectral on YCOIL10 : 51.92% YCOIL05a YCOIL05b YCOIL10 geodesic - LLS 12.90% 9.57% 24.36% geodesic - LGA2D 32.00% 9.93% 19.70% geodesic - LGA3D 23.40% 6.77% 17.81% [61]

62 Parameter Scan: YCOIL05a / YCOIL05b YCOIL05a LLS LGAcb2 LGAcb3 YCOIL05b [62]

63 Parameter Scan on YCOIL10 Dataset LLS LGAcb2 LGAcb3 geolls geolgacb2 geolgacb3 [63]

64 Discussion: Image Clustering Affinity measure must reflect less-variant features. Conic affinity captures global relations among images in the set. Gradient affinity is a pairwise measure based on the probabilistic distribution of the gradient vectors under changing illumination. Local Linear Structure measure uses 2D affine warped neighbor images to compute weights of secant chord approximation. Geodesic expansion of affinity improves clustering results. Histograms may be used as an approximate prob. distribution of image gradient vectors. [64]

65 Discussion: Comparing Cluster Results Clustering algorithms have free parameters. Simply reporting the best result may not be enough. Is it possible to find good parameters for next similar trial? Two approaches Run pilot experiments, fix parameters, then run real experiments. Take average of many experiments, and plot over parameters. [65]

66 Future Direction Pose-Light Image Clustering Alignment of two images. Feature/Part-based approaches. Evaluation of Clustering Result Large dataset of images. Object detection / segmentation from real-world images. [66]

67 Thank You! Hypergraph Clustering Clique Averaging Image Clustering over Illumination and Pose Variation Conic Affinity Gradient Affinity Local Linear Structure + 2D Affine Image Alignment Gradient Affinity with 2D Affine Alignment Conclusion & Future Work Image Clustering needs study on less-variant Image characteristics. Fair Comparison of Clustering Algorithms [67]