Dense Correspondence and Its Applications

Transcription

1 Dense Correspondence and Its Applications NTHU 28 May 2015

2 Outline Dense Correspondence Optical ow Epipolar Geometry Stereo Correspondence

3 Why dense correspondence? Features are sparse Some applications of dense correspondence Frame-rate conversion View morphing, view synthesis Stereo Denition Compute a vector eld (u(x, y), v(x, y)) over the pixels of image I 1 so that the pixels I 1 (x, y) and I 2 (x + u(x, y), y + v(x, y)) correspond.

4 Related topics Motion vector Optical ow estimation Registration Stereo correspondence, epipolar geometry, rectication Video matching Image morphing, view synthesis

5 Ane transformation I 1 (x, y) and I 2 (x, y ) x = a 11 x + a 12 y + b 1 y = a 21 x + a 22 y + b 2 (1) Rigid motions (translations and rotations) Similarity transformations (rigid motions plus a uniform change in scale) Shape-changing transformations (shears)

6 Projective transformation Homography x = h 11x + h 12 y + h 13 h 31 x + h 32 y + h 33 y = h 21x + h 22 y + h 23 h 31 x + h 32 y + h 33 (2) Related to perspective projection camera undergoes pure rotation (panning, tilting, and zooming only) scene is entirely plannar Image registration Estimating the parameters

7 Homogeneous coordinates Representing an image location (x, y) as a triple (x, y, 1) ( any triple (x, y, z) can be converted backt to an image coordinate x, ) y z z x y 1 h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 x y 1 (3) The simbol means that the two vectors are equivalent up to a scalar multiple

8 Normalized direct linear transform (Hartley and Zisserman) Initial estimation of the parameters of a projective transformation given a set of feature matches Two sets of features {(x 1, y 1 ),..., (x n, y n )} and {(x 1, y 1 ),..., (x n, y n)} Normalize each set to have zero mean and average distance from the origin 2 Construct a 2n 9 matrix A, where each feature match generates two rows of A [ ] xi y A i = i 1 y x i i y y i i y i x i y i x x i i x y i i x (4) i (A i [h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 ] T = 0) Singular value decomposition A = UDV T. Let h be the last column of V. Reshape h into a 3 3 matrix Ĥ Recover the nal projective transformation estimate as H = T 1 Ĥ T

9 Scattered data interpolation Find a continuous function f (x, y) dened over the rst image plane so that f (x i, y i ) = (x, y ), i = 1,..., n. i i given a set of feature matches unevenly distributed in each image the goal is to generate a dense correspondence for every point in the rst image plane.

10 Thin-plate spline interpolation f (x, y) = [ n ] i=1 w 1iφ(r i ) + a 11 x + a 12 y + b n 1 i=1 w 2iφ(r i ) + a 21 x + a 22 y + b 2 where φ(r) is called a radial bassis function and [ ] [ ] r i = x xi y y i 2 (5) (6)

11 Solving a linear system where r ij = [ xi y i ] [ xj y j ] 2 (7)

12 Thin-plate spline interpolation n correspondences, 2(n + 3) parameters including 6 global ane motion parameters and 2n coecients for rbfs

13 Choice of rbf Smallest bending energy Minimizingthe integral ( 2 f x 2 ) 2 ( ) 2 2 ( ) f 2 2 f + + dx dy (8) x y y 2 φ(r) = r 2 log r (9)

14 B-spline interpolation Basis spline Not at the feature point locations but at control points on a lattice overlaid on the rst image plane 3 3 f (x, y) = w kl (x, y)ψ kl (x, y) (10) ψ: cubic polynomials k=0 l=0

15 Dieomorphisms `ow' the rst image I 1 to the second image I 2 over time interval t [0, 1] the ow is represented by (u(x, y, t), v(x, y, t)) a mapping S(x, y, t) specifying the owed location of each point (x, y) at time t, t = 0 for I 1 and t = 1 for I 2 S(x, y, t) = t [ u(x, y, t) v(x, y, t) ], t [0, 1] (11)

16 Optimization The result is a dieomorphism that does not suer from the grid line self-intersection problem

17 As-rigid-as-possible deformation f (x, y) = T x,y (x, y), where T x,y is a rigid transformation dened at (x, y) each T x,y is estimated by minimizing n T x,y (x i, y i ) (x, y i i ) 2 2 (x, y) (x i, y i ) 2α 2 i=1 (12)

18 Problems with scattered data interpolation techniques Hard to control (e.g. to keep straight lines straight) Independent of the underlying image intensities Might reuqire signicant user interaction

19 Optical ow Estimating a motion vector (u, v) at every point (x, y) such that I 1 (x, y) and I 2 (x + u, y + v) correspond: Horn-Schunck method Lucas-Kanade method

20 The Horn-Schunck method Brightness constancy assumption I (x + u, y + v, t + 1) = I (x, y, t) (13) Lambertian assumption same brightness regardless of viewing direction Ideal image formation process no vignetting

21 The Horn-Schunck method Taylor expansion I x u + I y v + I t = 0 (14) ] [ u I v = I t The component of the ow vector in the direction of the gradient is I t (15) I 2 I (16) 2

22 The Horn-Schunck method Considering smoothness E HS (u, v) = E data (u, v) + λe smoothness (u, v) (17) λ is a regularization parameter E data (u, v) = x,y ( I x u + I y v + I ) 2 (18) t E smoothness = x,y ( ) 2 ( ) 2 ( ) 2 ( ) 2 u u v v (19) x y x y

23 Euler-Lagrange ( ) 2 I λ 2 u = u + I I x x y v + I I x t ( ) 2 I λ 2 v = v + I I y x y u + I I y t (20) (21)

24 Finite dierences

25 Strength and weakness Advantage difussion equation, creates good estimates of the dense correspondence eld even when the local gradient is nearly zero smoothly lling in reasonable ow vectors based on nearby locations Disadvantage Taylor series approximation is noly valid when (u, v) is close to zero I 1 and I 2 are already very similar Hierarchical or multiresolution approach

26 Hierarchical approach Laplacian pyramid Flow is computed for the coarset level of the pyramid Warp the second image plane to the rst using the estimated ow the ow between the rst image and the warped second image is expected to be nearly zero

27 Improvements Applying a median lter to the incremental ow eld during the warping process D. Sun, S. Roth, and M. Black. Secrets of optical ow estimation and their principles. CVPR 2010.

28 The Lucas-Kanade method Optical ow vector at a point (x 0, y 0 ) is dened as the minimizer (u, v) of E LK (u, v) = (x,y) w(x, y)(i (x + u, y + v, t + 1) I (x, y, t)) 2 (22) where w(x, y) is a window function centered at (x 0, y 0 ).

29 Solution of the linear system H = ( ) 2 (x,y) w(x, y) I (x, y) x ( ) (x,y) w(x, y) I I (x, y) (x, y) x y H [ u v ( ) (x,y) w(x, y) I I (x, y) (x, y) x y ( ) 2 (x,y) w(x, y) I (x, y) y ] [ (x,y) = w(x, y) ( ] I I (x, y) (x, y)) ( x t ) (x,y) w(x, y) I I (x, y) (x, y) y t (23) (24)

30 Comparisons the Lucas-Kanade algorithm is local the ow vector can be computed at each pixel independently easier to solve the Horn-Schunck algorithm is global all the ow vectors depend on each other through the dierential equation Both assume that the ow vectors are small pyramidal implementation

31 Renements and extensions Changing the data term Changing the smoothness term Changing the form of cost functions

32 Changing the data term Horn-Schunck brightness constancy assumtion Uras et al. gradient constancy assumption I (x + u, y + v, t + 1) = I (x, y, t) (25) Uras et al. A computational approach to motion perception. Biological Cybernetics, 60( 2): 79 87, Dec allows some local variation to illumination changes

33 Changing the data term Brox et al. assume both brightness and gradient constancy E data (u, v) = x,y (I 2 (x+u, y+v) I 1 (x, y)) 2 +γ I 2 (x+u, y+v) I 1 (x, y) 2 2 (26) Brox et al. High accuracy optical ow estimation based on a theory for warping. ECCV γ weights the contribution of the terms (typically γ 100) directly expresses the deviation from the constancy assumptions, instead of using the Taylor approximation that is only valid for small u and v Xu et al. introduce a binary swithch variable choosing either the brightness constancy or gradient constancy assumption

34 Changing the data term LK + HS E data (u, v) = (x,y) w(x, y)(i 2 (x + u, y + v) I 1 (x, y)) 2 (27) A. Bruhn, J. Weickert, and C. Schnöorr. Lucas/ Kanade meets Horn/ Schunck: Combining local and global optic ow methods. IJCV, 61( 3): , Feb small window size, local spatial smoothing, which makes the data term robust to noise global regularization, which makes the ow eld smooth and dense Probabilistic data term (Sun et al.) learned distribution of I 2 (x + u, y + v) I 1 (x, y) mixture of Gaussian

35 Changing the smoothnes term The smoothness term should be downweighted perpendicular to image edges correspond to depth discontinuities Nagel and Enkelmann. An investigation of smoothness constraints for the estimation of displacement vector elds from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8( 5): , Sept E smoothness (u, v) = trace ( [ u x v y v x v y D : R R 2 2 is the anisotropic diusion tensor D(g) = ] T D( I 1 (x, y)) [ u 1 ( ) g 2 + g g T + β 2 I β2 x v y v x v y ] ) (28) (29)

36 Anisotropic diusion and eigenvalues the major and minor axes of each ellipse are aligned with the tensor's eigenvectors and weighted by the corresponding eigenvalues

37 Changing the smoothness term as a prior term on the optical ow eld reects the assumptions about `good' ows S. Roth and M. Black. On the spatial statistics of optical ow. IJCV, 74( 1): 33 50, Aug Field-of-Experts model

38 Changing the cost functions Horn and Schunck: quadratic functions Robust estimation M. J. Black and P. Anandan. The robust estimation of multiple motions: Parametric and piecewise-smooth ow elds. CVIU, 63( 1): , Jan Robust data term E data (u, v) = x,y ( I ρ x u + I y v + I ) t (30) Robust smoothness term E smoothness (u, v) = ρ ( [ ] u x, u y, v x, v ) y 2 (31) ρ(z) = z 2 : Horn-Schunck other choices

39 Robust penalty functions typically used for optical ow

40 Occlusion Implicit assumption: every pixel in I 2 corresponds to some pixel in I 1. This assumption is violated at occlusions. Cross checking, or left-right checking (u, v) fwd (x, y) = (u, v) bwd (I 2 (x + u fwd (x, y), y + v fwd (x, y))) (32) Robust cost functions can partially alleviate the occlusion problem

41 Layered Flow Layered motion for video matting Post-converting a monocular lm into stereo M. J. Black and P. Anandan. The robust estimation of multiple motions: Parametric and piecewise-smooth ow elds. CVIU, 63( 1): , Jan (using robust penalty functions to estimate a dominant motion in the scene, classify inlier pixels into a solved layer, and re-apply the process to outlier pixels) M. Black and A. Jepson. Estimating optical ow in segmented images using variable-order parametric models with local deformations. PAMI, 18( 10): , Oct (ow eld in each region can be represented by a low-dimensinal parametric transformation (e.g. ane), coarse to ne) Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. CVPR (the non-parametric ow eld is smooth, using EM to estimate layer memberships and estimate the ow eld in each layer)

42 Large-displacement optical ow T. Brox, C. Bregler, and J. Malik. Large displacement optical ow. CVPR small structure, fast motion, e.g. hand waving starts with the segmentation of each image into roughly constant-texture patches, each of which is described by a SIFT-like descriptor matching descriptors, as additional constraints C. Liu, J. Yuen, and A. Torralba. SIFT ow: Dense correspondence across scenes and its applications. PAMI, 33( 5): , May replaces the brightness constancy assumption with a SIFT-descriptor constancy assumption, S 2 (x + u, y + v) S 1 (x, y) dense SIFT designed to match between dierent scenes

43 Human-assisted motion annotation C. Liu, W. Freeman, E. Adelson, and Y. Weiss. Human-assisted motion annotation. CVPR semi-automatically created layers

44 Evaluation

45 Epipolar Geometry Two images taken at the same time, in dierent positions, by dierent cameras don't need to search across the entire image to estimate the motion vector (u, v) only one degree of freedom for the possible correspondence

46 Epipolar line and epipole all of the epipolar lines in one image intersect at the epipole, the projection of the camera center of the other image every point in the scene has to lie on one of those planes

47 The fundamental matrix the epipolar line in one image corresponding to a point in the other image can be computed from the fundamental matrix a 3 3 matrix F that concisely expresses the relationship between any two matching points any correspondence {(x, y) I 1, (x, y ) I 2 } must satisfy x T x y 1 F y 1 = 0 (33) the fundamental matrix F is dened up to scale F has rank 2 what are the coecients on x, y, and 1?

48 Geometric argument to see why F has rank 2 all the epipolar lines in I 1 intersect at the epipole e = (x e, y e ) for any (x, y ) I 2, e lies on the corresponding epipolar line x T x e y F y e = 0 (34) 1 1 holds for every (x, y ) it means that F so [x e, y e, 1] T is an eigenvector of F with eigenvalue 0. x e y e 1 = 0 (35) similarly, [x e, y e, 1 ] T is an eigenvector of F T with eigenvalue 0.

49 Representating F as a factorization based on the epipole in the second image x e F = y e M (36) 1 where M is a full-rank 3 3 matrix. The notation [e] : 0 e 3 e 2 [e] = e 3 0 e 1 (37) e 2 e 1 0 rank? the fundamental matrix is not dened for an image pair that shares the same camera center a project transformation that directly species each pair of corresponding points; the same type of relationship holds when the scene contains only a single plane the fundamental matrix for an image pair taken by a pair of separated cameras of a real-world scene is dened and unique (up to scale)

50 Estimating the fundamental matrix like a projective transformation, the fundamental matrix can be estimated from a set of feature matches [x i x i, x i y i, x i, y i x i, y i y i, y i, x i y i 1][f 11 f 12 f 13 f 21 f 22 f 23 f 31 f 32 f 33 ] T = 0 (38) collecting the linear equations for each point yields an n 9 linear system Af = 0 basic algorithm: normalized eight-point algorithm

51 Normalized eight-point algorithm 1. The input is two sets of features {(x 1, y 1 ),..., (x n, y n )} and {(x 1, y 1 ),..., (x n, y n)}. Normalize each set of feature matches to have zero mean and average distance from the origin 2. Done by a pair of similarity transformations S and S. 2. Construct the n 9 matrix A, where each feature match generates a row. 3. Compute the singular value decomposition of A, A = UDV T. Let f be the last column of V (corresponding to the smallest sigular value). 4. Reshape f into a 3 3 matrix ˆF, ling in the elements from left to right in each row. 5. Compute the singular value decomposition of ˆF, ˆF = UDV T. Set the lower right (3, 3) entry of D equal to zero to create ˆD and replace ˆF with U ˆDV T. 6. Recover the nal fundamental matrix estimate as F = S T ˆF S.

52 Example

53 Extensions Maximum likelihood estimate under the assumptions that the measurement errors in each feature location are Gaussian Non-linear optimization and RANSAC When each camera's location, orientation, and internal conguration are known, the funcamental matrix can be computed directly

54 Image rectication epipolar lines are typically slanted, which can make estimating correspondences along conjugate epipolar lines complicated due to repeated image resampling operations rectication: making conjugate epipolar lines coincide with scanlines epipolar lines are parallel and the epipoles are `at innity', represented by the homogeneous coordinate [1, 0, 0] the fundamental matrix for a rectied image pair is given by F = (39) for a correspondence in the rectied pair of images, y = y, corresponding to the denition of rectication.

55 Rectication method proposed by Hartley the idea is to estimate a projective transformation H 2 for the second image that moves the epipole to the homogeneous coordinate [1, 0, 0] while resembling a rigid transformation as much as possible Estimate the fundamental matrix F from the feature matches using the eight-point algorithm. Factor F in the form of F = [e ] M, where e = [x e, y e, 1] T is the homogeneous coordinate of the epipole in the second image. Choose a location (x, 0 y 0 ) in the second image, e.g., the center of the image, and determine a 3 3 homogeneous translation matrix T that moves (x, 0 y 0 ) to the origin. Determine a rotation matrix R that movels the epipole onto the x-axis; let its new location be (x, 0). Compute H 2 as H 2 = /x 0 1 RT (40)

56 Rectication method proposed by Hartley (cont.) 5.(cont.) The rst matrix moves the epipole to innity along the x-axis. 6.Apply the projective transformation H 2 M to the features in I 1 and the projective transformation H 2 to the features in I 2 to get a transformed set of feature matches. 7.At this point, the two images are rectied, but applying H 2 M to I 1 may result in an unacceptably distorted image. The next step is to nd a horizontal shear and translation that bring the feature matches as close together as possible. n (aˆx i + bŷ i + c ˆx i ) 2 (41) i=1 8.Compute H 1 as H 1 = a b c H2 M (42)

57 Example

58 Stereo correspondence Estimating the disparity map Quantized disparity values Occlusions are explicitly modeled Monotonicity assumption Calibration is assumed known Images are acquired simultaneously