Announcements, schedule. Lecture 8: Fitting. Weighted graph representation. Outline. Segmentation by Graph Cuts. Images as graphs

Transcription

1 Announcements, schedule Lecture 8: Fitting Tuesday, Sept 25 Grad student etensions Due of term Data sets, suggestions Reminder: Midterm Tuesday 10/9 Problem set 2 out Thursday, due 10/11 Outline Review from Thursday (affinity, cuts) Local scale and affinity computation Hough transform Generalized Hough transform Shape matching applications Fitting lines Least squares Incremental fitting, k-means Weighted graph representation ( Affinity matri ) Images as graphs Segmentation by Graph Cuts q p C pq c w A B C Fully-connected graph node for every piel link between every pair of piels, p,q similarity c pq for each link» similarity is inversely proportional to difference in color and position Break Graph into Segments Delete links that cross between segments Easiest to break links that have low similarity similar piels should be in the same segments dissimilar piels should be in different segments Slide by Steve Seitz

2 Eample Distance matri D(:,1)= 0.01 D(1,:)= (0) Dimension of points : d = 2 Number of points : N = 4 for i=1:n for j=1:n D(i,j) = i - j 2 Distance matri D(:,1)= Distance matri D(1,:)= (0) (0) for i=1:n for j=1:n D(i,j) = i - j 2 for i=1:n for j=1:n D(i,j) = i - j 2 N N matri Measuring affinity One possibility: Map distances to similarities, use as edge weights on graph D Distances affinities A for i=1:n for j=1:n D(i,j) = i - j 2 for i=1:n for j=1:n A(i,j) = ep(-1/(2*σ^2)* i - j 2 )

3 One possibility: Measuring affinity D= Scale affects affinity Essentially, affinity drops off after distance gets past some threshold Small sigma: group only nearby points Increasing sigma Large sigma: group distant points Shuffling the affinity matri Scale affects affinity Permute the order of the vertices, in terms of how they are associated with the matri rows/cols Points 1 10 Data points σ=.2 Points σ=.1 σ=.2 σ=1 Affinity matrices Eigenvectors and graph cuts w Eigenvectors and graph cuts Want a vector w giving the association between each element and a cluster Want elements within this cluster to have strong affinity with one another Maimize wa T Aaw w Aw = ΣΣ w i A ij w i j j A w subject to the constraint wa T a w= 1... Aw = λw Eigenvalue problem : choose the eigenvector of A with largest eigenvalue

4 Rayleigh Quotient Eample Data points Affinity matri eigenvector Check out this tutorial by Martial Hebert Eigenvectors and multiple cuts Scale affects affinity, number of clusters Use eigenvectors associated with k largest eigenvalues as cluster weights Or re-solve recursively Multiscale data Scale really affects clusters [Self-Tuning Spectral Clustering, L. Zelnik-Manor and P. Perona, NIPS 2004] Local scale selection Possible solution: choose sigma per point Local scale selection Distance to Kth neighbor for point s_i [Self-Tuning Spectral Clustering, L. Zelnik-Manor and P. Perona, NIPS 2004] [Self-Tuning Spectral Clustering, L. Zelnik-Manor and P. Perona, NIPS 2004]

5 Local scale selection, synthetic data [Self-Tuning Spectral Clustering, L. Zelnik-Manor and P. Perona, NIPS 2004] Fitting Want to associate a model with observed features Fitting lines Given points that belong to a line, what is the line? How many lines are there? Which points belong to which lines? [Fig from Marszalek & Schmid, 2007] Difficulty of fitting lines Hough transform Etraneous data: clutter, multiple models Missing data: only some parts of model are present Noise in the measured edge points, orientations Cost: infeasible to check all combinations of features by fitting a model to each possible subset Enter: Voting schemes Maps model (pattern) detection problem to simple peak detection problem Record all the structures on which each point lies, then look for structures that get many votes Useful for line fitting

6 Finding lines in an image Finding lines in an image y b y b b 0 y 0 image space m 0 m Hough (parameter) space 0 image space m Hough (parameter) space Connection between image (,y) and Hough (m,b) spaces A line in the image corresponds to a point in Hough space To go from image space to Hough space: given a set of points (,y), find all (m,b) such that y = m + b Slide credit: Steve Seitz Connection between image (,y) and Hough (m,b) spaces A line in the image corresponds to a point in Hough space To go from image space to Hough space: given a set of points (,y), find all (m,b) such that y = m + b What does a point ( 0, y 0 ) in the image space map to? Answer: the solutions of b = - 0 m + y 0 this is a line in Hough space Finding lines in an image y b y 0 0 image space m Hough (parameter) space Polar representation for lines Issues with (m,b) parameter space: Can take on infinite values Undefined for vertical lines (=constant) Connection between image (,y) and Hough (m,b) spaces A line in the image corresponds to a point in Hough space To go from image space to Hough space: given a set of points (,y), find all (m,b) such that y = m + b What does a point ( 0, y 0 ) in the image space map to? Answer: the solutions of b = - 0 m + y 0 this is a line in Hough space Polar representation for lines Hough transform algorithm Using the polar parameterization: H: accumulator array (votes) [0,0] y d ө cos ө + y sin ө + d = 0 d: perpicular distance from line to origin ө : angle the perpicular makes with the -ais (0 <= ө <= 2π) Point in image space sinusoid segment in Hough space Basic Hough transform algorithm 1. Initialize H[d, θ]=0 2. for each edge point I[,y] in the image for θ = 0 to 180 // some quantization H[d, θ] += 1 3. Find the value(s) of (d, θ) where H[d, θ] is maimum 4. The detected line in the image is given by Hough line demo Time compleity (in terms of number of votes)? θ d

7 Eample: Hough transform for straight lines Eample: Hough transform for straight lines Square : Circle : y d edge coordinates θ Votes Eample: Hough transform for straight lines Eample with noise y d edge coordinates θ Votes Eample with noise / random points Etensions Etension 1: Use the image gradient 1. same 2. for each edge point I[,y] in the image compute unique (d, θ) based on image gradient at (,y) H[d, θ] += 1 3. same 4. same (Reduces degrees of freedom) edge coordinates Votes Etension 2 give more votes for stronger edges Etension 3 change the sampling of (d, θ) to give more/less resolution Etension 4 The same procedure can be used with circles, squares, or any other shape Adapted from Steve Seitz

8 Recall: Image gradient The gradient of an image: The gradient points in the direction of most rapid change in intensity The gradient direction (orientation of edge normal) is given by: The edge strength is given by the gradient magnitude Etensions Etension 1: Use the image gradient 1. same 2. for each edge point I[,y] in the image compute unique (d, θ) based on image gradient at (,y) H[d, θ] += 1 3. same 4. same (Reduces degrees of freedom) Etension 2 give more votes for stronger edges (use magnitude of gradient) Etension 3 change the sampling of (d, θ) to give more/less resolution Etension 4 The same procedure can be used with circles, squares, or any other shape Slide credit: Steve Seitz Hough transform for circles Circle: center (a,b) and radius r ( i a) + ( yi b) = r For a fied radius r, unknown gradient direction Hough transform for circles Circle: center (a,b) and radius r ( i a) + ( yi b) = r For unknown radius r, unknown gradient direction Hough space Hough space Hough transform for circles Circle: center (a,b) and radius r ( i a) + ( yi b) = r For unknown radius r, known gradient direction θ Hough transform for circles For every edge piel (,y) : For each possible radius value r: θ = gradient direction, from,y to center a = r cos(θ) b = y + r sin(θ) H[a,b,r] += 1 Hough space

9 Real World Circle Eamples Original Finding Coins Edges (note noise) Slide credit: S. Narasimhan Crosshair indicates results of Hough transform, bounding bo found via motion differencing. Slide credit: S. Narasimhan Penny Finding Coins Quarters Finding Coins Note: a different Hough transform (with separate accumulators) was used for each circle radius (quarters vs. penny). Slide credit: S. Narasimhan Coin finding sample images from: Vivek Kwatra Hough transform for parameterized curves For any curve with analytic form f(,a) = 0, where : edge piel in image coordinates a : vector of parameters 1. Initialize accumulator array: H[a] = 0 2. For each edge piel: determine each a such that f(,a) = 0, and increment H[a] 3. Local maima in H correspond to curves Practical tips Minimize irrelevant tokens first (take edge points with significant gradient magnitude) Choose a good grid / discretization (trial and error) Vote for neighbors, also (smoothing in accumulator array) Utilize direction of edge to reduce free parameters by 1

10 Hough transform Pros All points are processed indepently, so can cope with occlusion Some robustness to noise: noise points unlikely to contribute consistently to any single bin Can detect multiple instances of a model in a single pass Hough transform Cons Compleity of search time increases eponentially with the number of model parameters Non-target shapes can produce spurious peaks in parameter space Quantization: hard to pick a good grid size Too coarse large votes obtained when too many different lines correspond to a single bucket Too fine miss lines because some points that are not eactly collinear cast votes for different buckets Generalized Hough transform What if we still know direction to some reference point (a), but allow arbitrary shapes defined by their boundary points? θ [Dana H. Ballard, Generalizing the Hough Transform to Detect Arbitrary Shapes, 1980] a p 1 p 2 r 1 = a p 1 r 2 = a p 2 Generalized Hough transform For model shape: construct a table storing these r vectors as function of gradient direction To detect model shape: for each edge point Inde into table with θ Use indeed r vectors to vote for (,y) position of reference point Peak in this Hough space is reference point with most supporting edges Assuming translation is the only transformation here, i.e., orientation and scale are fied. Generalized Hough transform Generalized Hough transform a Model shape New test shape : observed edges

11 Generalized Hough Transform Find Object Center ( c, yc ) given edges ( i, y i, φi ) Create Accumulator Array A( c, yc ) Initialize: A( c, yc ) = 0 ( c, yc ) For each edge point ( i, y i, φi ) For each r vector entry indeed in table, compute: Find peaks in Increment Accumulator: A c, y ) ( c = + r cosα c c i i i k A(, y ) A(, y ) + 1 c i k i k y = y + r sinα i k c = c c With modifications to table can generalize to add scale, orientation increases size of accumulator array Eample in recognition: implicit shape model Build codebook of local appearance for each category using agglomerative clustering [B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, 2004] Codebook Patch descriptors etracted from lots of car images Eample in recognition: implicit shape model Build codebook of local appearance for each category using agglomerative clustering In all training images, match codebook entries to images with cross correlation (activate all entries with similarity > t) For each codebook entry, store all positions it was found, relative to object center [B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, 2004] Implicit shape model Implicit shape model Given test image, etract patches, match to codebook entry (or entries) Cast votes for possible positions of object center Search for maima in voting space using Mean-Shift (Etract weighted segmentation mask based on stored masks for the codebook occurrences) [B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, 2004] [B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, 2004]

12 Grouping and fitting Grouping, segmentation: make a compact representation that merges similar features Relevant algorithms: K-means, hierarchical clustering, Mean Shift, Graph cuts Fitting: fit a model to your observed features Relevant algorithms: Hough transform for lines, circles (parameterized curves), generalized Hough transform for arbitrary boundaries; least squares; assigning points to lines incrementally or with k- means