Fitting a transformation: Feature-based alignment April 30 th, Yong Jae Lee UC Davis

Transcription

1 Fitting a transformation: Feature-based alignment April 3 th, 25 Yong Jae Lee UC Davis

2 Announcements PS2 out toda; due 5/5 Frida at :59 pm Color quantization with k-means Circle detection with the Hough transform 2

3 Last time: Deformable contours a.k.a. active contours, snakes Given: initial contour (model) near desired object Goal: evolve the contour to fit eact object boundar Main idea: elastic band is iterativel adjusted so as to be near image positions with high gradients, and satisf shape preferences or contour priors [Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV987] 3 Slide credit: Kristen Grauman Figure credit: Yuri Bokov

4 Last time: Deformable contours Image from 4 Kristen Grauman

5 Last time: Deformable contours Pros: Useful to track and fit non-rigid shapes Contour remains connected Possible to fill in subjective contours Fleibilit in how energ function is defined, weighted. Cons: Must have decent initialization near true boundar, ma get stuck in local minimum Parameters of energ function must be set well based on prior information 5 Kristen Grauman

6 Toda Interactive segmentation Feature-based alignment 2D transformations Affine fit RANSAC 6

7 Interactive forces How can we implement such an interactive force with deformable contours? 7 Kristen Grauman

8 Interactive forces An energ function can be altered online based on user input use the cursor to push or pull the initial snake awa from a point. Modif eternal energ term to include a term such that E push n r = = ν i i 2 p Nearb points get pushed hardest 2 Adapted b Devi Parikh from Kristen Grauman 8

9 Intelligent scissors Another form of interactive segmentation: Compute optimal paths from ever point to the seed based on edge-related costs. Adapted b Devi Parikh from Kristen Grauman [Mortensen & Barrett, SIGGRAPH 995, CVPR 999] 9

10 Intelligent scissors Slide credit: Kristen Grauman

11 Intelligent scissors Slide credit: Kristen Grauman

12 Beond boundar snapping Another form of interactive guidance: specif regions Usuall taken to suggest foreground/background color distributions User Input Result Bokov and Joll (2) How to use this information? 2 Kristen Grauman

13 Recall: Images as graphs q p w pq w Full-connected graph node for ever piel link between ever pair of piels, p,q similarit w pq for each link» similarit is inversel proportional to difference in color and position 3 Steve Seitz

14 Recall: Segmentation b Graph Cuts w A B C Break graph into segments Delete links that cross between segments Easiest to break links that have low similarit similar piels should be in the same segments dissimilar piels should be in different segments 4 Steve Seitz

15 Graph cuts for interactive segmentation Adding hard constraints: Add two additional nodes, object and background terminals Link each piel To both terminals To its neighboring piels 6 Yuri Bokov

16 Graph cuts for interactive segmentation Adding hard constraints: Let the edge weight to object or background terminal reflect similarit to the respective seed piels. 7 Yuri Bokov

17 Graph cuts for interactive segmentation Bokov and Joll (2) 8 Yuri Bokov

18 Graph cuts for interactive segmentation Another interaction modalit: specif bounding bo Slide credit: Kristen Grauman Intelligent Scissors Mortensen and Barrett (995) Graph Cuts Bokov and Joll (2) GrabCut Rother et al. (24) 9

19 Grab Cut Loosel specif foreground region Iterated graph cut? User initialization User Initialisation K-means for learning colour distributions Rother et al (24) Graph cuts to infer the segmentation 2

20 Grab Cut Loosel specif foreground region Iterated graph cut R Foreground & Background Iterated graph cut R Foreground Background G Background G Gaussian Miture Model (tpicall 5-8 components) Rother et al (24) 2

21 Grab Cut Rother et al (24) 22

22 Toda Interactive segmentation Feature-based alignment 2D transformations Affine fit RANSAC 23 Slide credit: Kristen Grauman

23 Motivation: Recognition Slide credit: Kristen Grauman 24 Figures from David Lowe

24 Motivation: medical image registration Slide credit: Kristen Grauman 25

25 Motivation: mosaics (In detail net week) Slide credit: Kristen Grauman Image from 26

26 Alignment problem We have previousl considered how to fit a model to image evidence e.g., a line to edge points, or a snake to a deforming contour In alignment, we will fit the parameters of some transformation according to a set of matching feature pairs ( correspondences ). i T i Slide credit: Kristen Grauman 27

27 Parametric (global) warping Eamples of parametric warps: translation rotation aspect affine perspective 28 Source: Alosha Efros

28 Parametric (global) warping T p = (,) p = (, ) Transformation T is a coordinate-changing machine: p = T(p) What does it mean that T is global? Is the same for an point p can be described b just a few numbers (parameters) Let s represent T as a matri: p = Mp = M 29 Source: Alosha Efros

29 Source: Alosha Efros Scaling Scaling a coordinate means multipling each of its components b a scalar Uniform scaling means this scalar is the same for all components: 2 3

30 Source: Alosha Efros Scaling Non-uniform scaling: different scalars per component: X 2, Y.5 3

31 Scaling Scaling operation: Or, in matri form: b a = = = b a scaling matri S Source: Alosha Efros 32

32 What transformations can be represented with a 22 matri? 2D Rotate around (,)? * cos * sin * sin * cos Θ + Θ = Θ Θ = Θ Θ Θ Θ = cos sin sin cos 2D Shear? sh sh + = + = * * = sh sh Source: Alosha Efros 2D Scaling? s s * * = = = s s 33

33 What transformations can be represented with a 22 matri? 2D Mirror about Y ais? = = = 2D Mirror over (,)? = = = 2D Translation? = + t = + t NO! 34 Source: Alosha Efros

34 2D Linear Transformations = a c b d Onl linear 2D transformations can be represented with a 22 matri. Linear transformations are combinations of Scale, Rotation, Shear, and Mirror 35 Source: Alosha Efros

35 Homogeneous coordinates Convenient coordinate sstem to represent man useful transformations To convert to homogeneous coordinates: homogeneous image coordinates Converting from homogeneous coordinates: Slide credit: Kristen Grauman 36

36 Homogeneous Coordinates Q: How can we represent 2d translation as a 33 matri using homogeneous coordinates? A: Using the rightmost column: = t t Translation t t + = + = Source: Alosha Efros 37

37 Translation + + = = t t t t t = 2 t = Homogeneous Coordinates Source: Alosha Efros 38

38 Basic 2D Transformations Basic 2D transformations as 33 matrices Θ Θ Θ Θ = cos sin sin cos = t t = sh sh Translate Rotate Shear = s s Scale Source: Alosha Efros 39

39 2D Affine Transformations Affine transformations are combinations of Linear transformations, and Translations Parallel lines remain parallel = w f e d c b a w Slide credit: Kristen Grauman 4

40 Toda Interactive segmentation Feature-based alignment 2D transformations Affine fit RANSAC 4

41 Alignment problem We have previousl considered how to fit a model to image evidence e.g., a line to edge points, or a snake to a deforming contour In alignment, we will fit the parameters of some transformation according to a set of matching feature pairs ( correspondences ). i T i 42 Kristen Grauman

42 Image alignment Two broad approaches: Direct (piel-based) alignment Search for alignment where most piels agree Feature-based alignment Search for alignment where etracted features agree Can be verified using piel-based alignment 43 Slide credit: Kristen Grauman

43 Fitting an affine transformation Assuming we know the correspondences, how do we get the transformation? ), ( i i ), ( i i + = t t m m m m i i i i Slide credit: Kristen Grauman 44

44 An aside: Least Squares Eample Sa we have a set of data points (X,X ), (X2,X2 ), (X3,X3 ), etc. (e.g. person s height vs. weight) We want a nice compact formula (a line) to predict X s from Xs: Xa + b = X We want to find a and b How man (X,X ) pairs do we need? What if the data is nois? 2 2 X b a X X b a X = + = + = 2 2 X X b a X X A=B = X X X b a X X X overconstrained 2 min B A Source: Alosha Efros 45

45 Fitting an affine transformation Assuming we know the correspondences, how do we get the transformation? ), ( i i ), ( i i + = t t m m m m i i i i = i i i i i i t t m m m m Slide credit: Kristen Grauman 46

46 Fitting an affine transformation How man matches (correspondence pairs) do we need to solve for the transformation parameters? = i i i i i i t t m m m m Kristen Grauman 47

47 Affine: # correspondences? T(,)? How man correspondences needed for affine? Alosha Efros

48 Fitting an affine transformation How man matches (correspondence pairs) do we need to solve for the transformation parameters? Once we have solved for the parameters, how do we compute the coordinates of the corresponding point for? Where do the matches come from? = i i i i i i t t m m m m ), ( new new Kristen Grauman 49

49 What are the correspondences?? Compare content in local patches, find best matches. e.g., simplest approach: scan with template, and compute SSD or correlation between list of piel intensities in the patch Later in the course: how to select regions using more robust descriptors. 5 Kristen Grauman

50 Fitting an affine transformation Affine model approimates perspective projection of planar objects. Figures from David Lowe, ICCV 999 5

51 Fitting an affine transformation 52

52 Fitting an affine transformation 53

53 Fitting an affine transformation Affine model approimates perspective projection of planar objects. 54 Figures from David Lowe, ICCV 999

54 Toda Interactive segmentation Feature-based alignment 2D transformations Affine fit RANSAC 55

55 Outliers Outliers can hurt the qualit of our parameter estimates, e.g., an erroneous pair of matching points from two images an edge point that is noise, or doesn t belong to the line we are fitting. 56 Kristen Grauman

56 Outliers affect least squares fit Slide credit: Kristen Grauman 57

57 Outliers affect least squares fit Slide credit: Kristen Grauman 58

58 RANSAC RANdom Sample Consensus Approach: we want to avoid the impact of outliers, so let s look for inliers, and use those onl. Intuition: if an outlier is chosen to compute the current fit, then the resulting line won t have much support from rest of the points. Slide credit: Kristen Grauman 59

59 RANSAC loop: RANSAC: General form. Randoml select a seed group of points on which to base transformation estimate (e.g., a group of match) 2. Compute transformation from seed group 3. Find inliers to this transformation 4. If the number of inliers is sufficientl large, re-compute estimate of transformation on all of the inliers Keep the transformation with the largest number of inliers Slide credit: Kristen Grauman 6

60 RANSAC for line fitting eample Source: R. Raguram 6 Lana Lazebnik

61 RANSAC for line fitting eample Least-squares fit Source: R. Raguram 62 Lana Lazebnik

62 RANSAC for line fitting eample. Randoml select minimal subset of points Source: R. Raguram 63 Lana Lazebnik

63 RANSAC for line fitting eample. Randoml select minimal subset of points 2. Hpothesize a model Source: R. Raguram 64 Lana Lazebnik

64 RANSAC for line fitting eample. Randoml select minimal subset of points 2. Hpothesize a model 3. Compute error function Source: R. Raguram 65 Lana Lazebnik

65 RANSAC for line fitting eample. Randoml select minimal subset of points 2. Hpothesize a model 3. Compute error function 4. Select points consistent with model Source: R. Raguram 66 Lana Lazebnik

66 RANSAC for line fitting eample Source: R. Raguram. Randoml select minimal subset of points 2. Hpothesize a model 3. Compute error function 4. Select points consistent with model 5. Repeat hpothesize-andverif loop 67 Lana Lazebnik

67 RANSAC for line fitting eample Source: R. Raguram. Randoml select minimal subset of points 2. Hpothesize a model 3. Compute error function 4. Select points consistent with model 5. Repeat hpothesize-andverif loop 68 Lana Lazebnik

68 RANSAC for line fitting eample Uncontaminated sample Source: R. Raguram. Randoml select minimal subset of points 2. Hpothesize a model 3. Compute error function 4. Select points consistent with model 5. Repeat hpothesize-andverif loop 69 Lana Lazebnik

69 RANSAC for line fitting eample Source: R. Raguram. Randoml select minimal subset of points 2. Hpothesize a model 3. Compute error function 4. Select points consistent with model 5. Repeat hpothesize-andverif loop 7 Lana Lazebnik

70 RANSAC for line fitting Repeat N times: Draw s points uniforml at random Fit line to these s points Find inliers to this line among the remaining points (i.e., points whose distance from the line is less than t) If there are d or more inliers, accept the line and refit using all inliers 7 Lana Lazebnik

71 RANSAC pros and cons Pros Simple and general Applicable to man different problems Often works well in practice Cons Lots of parameters to tune Doesn t work well for low inlier ratios (too man iterations, or can fail completel) Can t alwas get a good initialization of the model based on the minimum number of samples 72 Lana Lazebnik

72 Toda Interactive segmentation Feature-based alignment 2D transformations Affine fit RANSAC 73

73 Coming up: alignment and image stitching 74

74 Questions? See ou Tuesda! 75