Clustering appearance and shape by learning jigsaws Anitha Kannan, John Winn, Carsten Rother

Transcription

1 Clustering appearance and shape by learning jigsaws Anitha Kannan, John Winn, Carsten Rother

2 Models for Appearance and Shape Histograms Templates discard spatial info articulation, deformation, variation Patch-based approaches a happy medium size/shape of the patches is fixed

3 Jigsaw Intended as a replacement for fixed patch model Learn a jigsaw image such that: Pieces are similar in appearance and shape to multiple regions in training image(s) All training images can be ~reconstructed using only pieces from the jigsaw Pieces are as large as possible for a particular reconstruction accuracy

4 Jigsaw Model μ(z) intensity value at pixel z λ-1(z) variance at z l(i) offset between image pixel i and corresp. jigsaw pixel

5 Generative Model

6 Generative Model Each offset map entry is a 2D offset mapping point i in the image to point z = (i l(i)) mod J in the jigsaw, where J = (jigsaw width, jigsaw height) Product is over image pixels

7 Generative Model E is the set of edges in a 4-connected grid, with nodes representing offset map values γ influences the typical jigsaw piece size; set to 5 per channel δ( true ) = 1, δ( false ) = 0

8 Generative Model μ0 = 0.5, β = 1, b = 3 times data 2 precision, a = b Normal-Gamma prior allows for unused portions of the jigsaw to be well-defined

9 MAP Learning Image set is known Find J, Ls to maximize joint probability Initialize jigsaw Set precisions λ to expected value under the prior Set means μ to Gaussian noise with same mean and variance as the data

10 MAP Learning Iteration step 1: Given J, I1..N, update L1..N using α-expansion graph-cut algorithm Iteration step 2: Repeat until convergence

11 α-expansion Graph-Cut Start with arbitrary labeling f Loop: For each label α: Find f' = arg min E(f') among f' within one αexpansion of f If E(f') < E(f), set f := f' Else return f α-expansion defined in detail in Fast Approximate Energy Minimization via Graph Cuts. Yuri Boykov, Olga Veksler, Ramin Zabih.

12 Determining Jigsaw Pieces For each image, define region boundaries as the places where the offset map changes value. Each region thus maps to a contiguous area of the jigsaw. Cluster regions based on overlap: Ratio of intersection to union of the jigsaw pixels mapped to by the two regions Each cluster corresponds to a jigsaw piece.

13 Toy Example

14 Epitome Another unfixed patch-based generative model Patches have fixed size and shape, but not location Patches can be subdivided (24x24, 12x12, 8x8) Patches can overlap (average value taken) Cannot capture occlusion w/o a shape model Epitome model defined in detail in Epitomic analysis of appearance and shape. Nebojsa Jojic, Brendan J. Frey, Anitha Kannan.

15 Jigsaw vs. Epitome

16 Jigsaw for Multiple Images 100 face images from AT&T's Olivetti database

17 Unsupervised Part Learning

18 The Good Jigsaw allows automatically sized patches Occlusion is modeled implicitly, i.e. patch shape is variable Image segmentation is automatic Unsupervised part learning an easy next step Jigsaw reconstructions more accurate and better looking than equivalently sized Epitome model reconstructions

19 The Bad At each iteration, must solve a binary graph cut for each jigsaw pixel 30 minutes to learn 36x36 jigsaw from 150x150 toy image No patch transformation Can add specific transformations with linear cost increase Can favor similar neighboring offsets in addition to identical ones

20 The Questions?

21 Normalized Cuts and Image Segmentation Jianbo Shi and Jitendra Malix

22 Recursive Partitioning Segmentation/partitioning inherently hierarchical Image segmentation from low-level cues should sequentially build hierarchical partitions Partitioning done big-picture downward Mid- and high-level knowledge can confirm groups are identify repartitioning candidates

23 Graph Theoretic Approach Set of points represented as a weighted undirected graph G = (V,E) Each point is a node; G is fully-connected w(i,j) is a function of the similarity between i and j Find a partition of vertices into disjoint sets where by some measure in-set similarity is high, but cross-set similarity is low.

24 Minimum Graph Cut Dissimilarity between two disjoint sets of vertices can be measured as total weight of edges removed: The minimum cut defines an optimal bipartitioning Can use minimum cut for point clustering Proposed in An Optimal Graph Theoretic Approach to Data Clustering: Theory and Its Application to Image Segmentation. Z. Wu and R. Leahy.

25 Minimum Cut Bias Minimum cut favors small partitions cut(a,b) increases with the number of edges between A and B With w(i,j) inversely proportional to dist(i,j), B = n1 is the minimum cut.

26 Normalized Cut Measure cut cost as a fraction of total edge connections to all nodes Any cut that partitions small isolated points will have cut(a,b) close to assoc(a,b)

27 Normalized Association Can also use assoc to measure similarity within groups Minimizing Ncut equivalent to maximizing Nassoc Makes minimizing Ncut a very good partitioning criterion

28 Minimizing Ncut is NP-Complete Proof by Papadimitriou (1997) an appendix to the paper Reformulate problem: For i in V, xi = 1 if i is in A, -1 otherwise di = sumj w(i,j)

29 Reformulation (cont.) Let D be an NxN diagonal matrix with d on the diagonal Let W be an NxN symmetrical matrix with W(i,j) = wij Let 1 be an Nx1 vector of ones b = k/(1-k) y = (1 + x) b(1 - x)

30 Reformulation (cont.) with the condition ytd1 = 0 and yi in {1, -b}. This is a Rayleigh quotient By allowing y to take on real values, can minimize this by solving the generalized eigenvalue system (D W)y = λdy. But what about the two constraints on y?

31 First Constraint Transform the previous into a standard eigensystem: D-1/2(D W)D-1/2z = λz, where z = D1/2y z0 = D1/21 is an eigenvector with eigenvalue 0. Since D-1/2(D W)D-1/2 is symmetric positive semidefinite, z0 is the smallest eigenvector and all eigenvectors are perpendicular to each other.

32 First Constraint (cont.) Translating this back to the general eigensystem: y0 = 1 is the smallest eigenvector, with eigenvalue 0 0 = z1tz0 = y1td1, where y1 is the second smallest eigenvector

33 First Constraint (cont.) Since we are minimizing a Rayleigh quotient with a symmetric matrix, we use the following property under the constraint that x is orthogonal to the j-1 smallest eigenvectors x1,...,xj-1, the quotient is minimized by xj with the eigenvalue λj being the minimum value. For more details see Matrix Computations, G.H. Golub and C.F. Van Loan, and Partitioning Sparse Matrices with Eigenvectors of Graphs, A. Pothen, H.D. Simon, and K.P. Liou.

34 Real-valued Solution y1 is thus the real valued solution for a minimal Ncut. We cannot force a discrete solution relaxing the second constraint makes this problem tractable. Can transform y1 into a discrete solution by finding the splitting point such that the resulting partition has the best Ncut(A,B) value.

35 Lanczos Method Graphs are often only locally connected resulting eigensystem are very sparse Only the top few eigenvectors are needed for graph partitioning Need very little precision in resulting eigenvectors These properties exploited by using Lanczos method; running time approximately O(n3/2)

36 Recursive Partitioning redux After partitioning, the algorithm can be run recursively on each partitioned part Recursion stops once the Ncut value exceeds a certain limit, or result is unstable When subdividing an image with no clear way of breaking it, eigenvector will resemble a continuous function Construct a histogram of eigenvector values if the ratio of minimum to maximum bin size exceeds 0.06, reject partitioning

37 Simultaneous K-Way Cut Since all eigenvectors will be perpendicular, can use third, fourth, etc. smallest to immediately subdivide partitions Some such eigenvectors would have failed the stability criteria Can use top n eigenvectors to partition, then iteratively merge segments Mentioned by the paper, but no experimental results presented

38 Recursive Two-Way Ncut Algorithm Given a set of features, construct weighted graph G, summarize information into W and D Solve (D W)x = λdx for the eigenvectors with the smallest eigenvalues Find the splitting point in x1 and bipartition the graph Check the stability of the cut and the value of Ncut Recursively repartition segmented parts if necessary

39 Weighting Schemes X(i) is the spatial location of node i F(i) is a feature vector defined as F(i) = 1, for point sets F(i) = I(i), the intensity value, for brightness F(i) = [v, v*s*sin(h), v*s*cos(h)](i), for color segmentation F(i) = [ I*f1,..., I*fn ](i), where fi are DOOG filters, in the case of texture segmentation DOOG filters were used to discriminate textures in Preattentive Texture Discrimination with Early Vision Mechanisms, J. Malik and P. Perona.

40 Brightness Segmentation Image sized 80x100, intensity normalized to lie in [0,1]. Partitions with Ncut value less than 0.04.

41 Brightness Segmentation 126x106 weather radar image. Ncut value less than 0.08.

42 Color Segmentation 77x107 color image (reproduced in grayscale in the paper). Ncut value less than 0.04.

43 Texture Segmentation Texture features correspond to DOOG filters at six orientations and fix scales.

44 Motion Segmentation Treat the image sequence as spatiotemporal data set. Weighted graph is constructed by taking all pixels as nodes and connecting spatiotemporal neighbors. d(i,j) represents motion distance between pixels i and j.

45 Motion Distance Defined as one minus the cross correlation of motion profiles, where the motion profile estimates the probability distribution of image velocity at each pixel.

46 Motion Segmentation Results Above: two consecutive frames The head and body have similar motion but dissimilar motion profiles due to 2D textures.

47 Questions?