Agenda. Rotations. Camera models. Camera calibration. Homographies

Transcription

1 Agenda Rotations Camera models Camera calibration Homographies

2 D Rotations R Y = Z r r r r r r r r r Y Z Think of as change of basis where ri = r(i,:) are orthonormal basis vectors r rotated coordinate frame r r

3 D Rotations Lots of parameterizations that try to capture DOFs Helpful ones for vision: orthonormal matrix, axis-angle, exponential maps Represent a D rotation with a unit vector pointed along the axis of rotation, and an angle of rotation about that vector -vs- D D

4 Background: euler s rotation theorm Any rotation of a rigid body in a three-dimensional space is equivalent to a pure rotation about a single fixed axis

5 Review: dot and cross products Dot product: a b = a b cos Cross product: a b = a b a b b a a b a b a b Cross product matrix: a b = âb = 0 a a b a 0 a b a a 0 b â T = â (skew-symmetric matrix)

6 Approach v R, v = x

7 Rodrigues' rotation formula v R, v = x k x? x. Write as x as sum of parallel and perpindicular component to omega. Rotate perpindicular component by D rotation of theta in plane orthogonal to omega R = I +ˆv sin +ˆvˆv( cos ) [Rx can simplify to cross and dot product computations]

8 Agenda Rotations Camera models Camera calibration Homographies

9 Perspective projection revisited y COP x (x,y,f) z (,Y,Z) x = f Z y = f Z Y

10 Perspective projection revisited x y = f f Y Z Given (,Y,Z) and f, compute (x,y) and lambda: x = f = Z x = x = f Z

11 Special case: f = (normalized image plane) (x,y,) (,Y,Z) COP x y = f f 0 Y Z x y = 0 0 Y Z Z D point is obtained by scaling ray pointed at image coordinate Scale factor = true depth of point [Aside: given an image with a focal length f, resize by /f to obtain unit-focal-length image]

12 Homogenous notation For now, think of above as shorthand notation for x y z Y Z x y z Y Z 9 s.t. x y z = Y Z

13 D rigid-body transformations D translations Y + T = Z + t x Y + t y R Z + t z Y = Z D rotations r r r r r r r r r Y Z R Y + T = Z r r r t x r r r t y r r r t z 6Y 7 Z

14 Aside: D affine transformations D translations Y + T = Z + t x Y + t y R Z + t z any invertible matrix R Y = Z r r r r r r Y r r r Z R Y + T = Z r r r t x r r r t y r r r t z 6Y 7 Z

15 Alternative perspective: change of coordinate system think of camera moving through world coordinate frame R Y + T = Z r r r t x r r r t y r r r t z 6Y 7 Z r r camera r T world coordinate frame

16 Camera projection x y = f 0 0 r r r t x 0 f 0 r r r t y 0 0 Camera instrinsic matrix K (can include skew & non-square pixel size) r r r t z Camera extrinsics (rotation and translation) 6Y Z 7 D point in world coordinates r r camera r T world coordinate frame Aside: homogenous notation is shorthand for x = x

17 Fancier intrinsics x s = s x x y s = s y y x 0 = x s + o x y 0 = y s + o y x =x 0 + s y 0 } } non-square pixels shifted origin y skewed image axes x K = s x s o x 0 s y o y 0 0 f f 0 = 0 0 fs x fs o x 0 fs y o y 0 0

18 Notation x y = fs x fs o x 0 fs y o y 0 0 r r r t x r r r t y r r r t z 6 6 Y Z 7 7 = K R T 6 6 Y Z 7 7 = M 6 6 Y Z 7 7 [Using Matlab s rows x columns] = A b 6 6 Y Z 7 7 = A Y Z + b

19 Notation & claims x y = M 6Y 7 Z = A Y + b Z Claims:. A x matrix M can be a camera matrix iff det(a) is not zero (geometric intution later). M is determined only up to a scale factor (easy to show) M = m T m T m T, A = a T a T a T, b = b b b

20 Applying the projection matrix x = ( Y Z a + b ) y = ( Y Z a + b ) = Y Z a + b Set of D points that project to x = 0: Set of D points that project to y = 0: Y Z a + b =0 Y Z a + b =0 Set of D points that project to x = inf or y = inf: Y Z a + b =0 What do these sets look like in D world?

21 Convention of most of today s slides (x,y) COP (,Y,Z) Draw image plane behind pinhole

22 Geometric intuition Rows of the camera matrix describe the planes defined by the image coordinate system a y a COP a x image plane

23 Other geometric properties y COP x (x,y,f) z (,Y,Z) What s set of (,Y,Z) points that project to same (x,y)? x y = A Y + b Z Solve above expression for (,Y,Z) as a function of (x,y)

24 Other geometric properties y COP x (x,y,f) z (,Y,Z) What s set of (,Y,Z) points that project to same (x,y)? Y = w + b 0 Z Direction of ray: w = A x y COP: b 0 = A b

25 Affine cameras perspective weak perspective Crucial constraint on camera matrix: m T = 0 0 0

26 Affine cameras Captures D affine transformation + orthographic projection + D affine transformation apple x y = = = apple a a a b a a a b 6Y 7 Z apple apple a a a Y b + a a a b Z x = A + b 6 7 6Y 7 Z Projection defined by 8 parameters Parallel lines project to parallel lines D points = linear projection of D points (+ D translation)

27 Geometric Transformations Point out differences in D transformations and D transformations Euclidean (trans + rot) preserves lengths + angles Affine: preserves parallel lines Projective: preserves lines Projective Affine Euclidean

28 Agenda Rotations Camera models Camera calibration Homographies

29 PnP = Perspective n-point Calibration: Recover M from scene points P,..,P N and the corresponding projections in the image plane p,..,p N p i = x i y i y COP x z P i = i 6Y i 7 Z i

30 The math for the calibration procedure follows a recipe that is used in many (most?) problems involving camera geometry, so it s worth remembering: Write relation between image point, projection matrix, and point in space: x i y i = M i 6Y i 7 Z i Write non-linear relations between coordinates: x i = mt P i m T P i, y i = mt P i m T P i

31 Estimating a camera matrix y x z Write constraints as linear unknowns M m T P i x i = m T P i. LM(:) = Homogenous linear system How many corresponding points needed? How many degrees of freedom in M? In noise-free case, L will have a non-zero null-space

32 What about noisy case? min M(:) = LM(:) Min right singular vector of L (or eigenvector of L T L) Aside: this technique of linearizing a nonlinear equation resulting from perspective projection is known as a direct linear transform Is this the right error to minimize? If not, what is?

33 Ideal error Err(M) = i (x i m T P i m T P i ) +(y i m T P i m T P i ) Initialize nonlinear optimization with algebraic solution

34 Radial Lens Distortions

35 Radial Lens Distortions No Distortion Barrel Distortion Pincushion Distortion

36

37 Correcting Radial Lens Distortions Before After

38 Overall approach Minimize reprojection error: Error(M,k s) Initialize with algebraic solution (approaches in literature based on various assumptions)

39 Agenda Rotations Camera models Camera calibration Homographies

40 Homography transformations. Models perspective effects for a planar scene. Models perspective effects from camera rotations

41 Let s analyze the projection of a D plane Place world coordinate frame on object plane x y = f 0 0 r r r t x 0 f 0 r r r t y 0 0 r r r t z 6Y 0 7

42 Projection of a D plane x y = f 0 0 r r r t x 0 f 0 r r r t y 0 0 r r r t z f 0 0 r r t x = 0 f 0 r r t y Y 0 0 r r t z fr fr ft x = fr fr ft y Y r r t z Convert between D location on object plane and image coordinate with a matrix H (Above holds for any instrinc matrix K) 6Y 0 7

43 Two-views of a plane Image correspondences x y x y = H Y = H Y x y = H H x y [LHS and RHS are related by a scale factor] [Aside: H usually invertible]

44 Computing homography projections Given (x,y) and H, how do we compute (x,y)? x y = a b c d e f g h i x y x = x = ax + by + c gx + hy + i

45 How many corresponding points needed? How many degrees of freedom in H? Estimating homographies Given corresponding D points in left and right image, estimate H Image correspondences x (gx + hy + i) =ax + by + c... AH(:) = Homogenous linear system

46 Estimating homographies Given corresponding D points in left and right image, estimate H Image correspondences min H(:) = AH(:) Minimum right singular vector of A (eigenvector of A T A)

47 Frontalizing planes using homographies Estimate homography on (at least) pairs of corresponding points (e.g., corners of quad/rect) Apply homography on all (x,y) coordinates inside target rectangle to compute source pixel location

48 Frontalizing planes using homographies

49 hies are derived from the corresponding points, forming a mosaic cally is shaped like a bow-tie, as images farther away from the are warped outward to fit the homography. The figure below is efeys and Hartley & Zisserman. Special case of views: rotations about camera center Can be modeled as planar transformations, regardless of scene geometry! (a) incline L.jpg (img) (b) incline R.jpg (img) (c) img warped to img s frame Figure : Example output for Q6.: Original images img and img (left and center) and img warped to fit img (right). Notice that the warped image clips out of the image. We will fix this in Q6. Figure 6: Final panorama view. With homography estimated with

50 Derivation Relation between D camera coordinates: D->D projection: Combining both: Y Z x y x y = R = Y Z f f = K RK K x y Y Z

51 Take-home points for homographies x y = a b c d e f g h i x y If camera rotates about its center, then the images are related by a homography irrespective of scene depth. If the scene is planar, then images from any two cameras are related by a homography. Homography mapping is a x matrix with 8 degrees of freedom.

52 Agenda Rotations Camera models Camera calibration Homographies Putting it all together

53 RANSAC for alignment

54 RANSAC for alignment

55 RANSAC for alignment

56 RANSAC for estimating transformation RANSAC loop:. Select feature pairs (at random). Compute transformation T (exact). Compute inliers (point matches where p i - T p i < ε). Keep largest set of inliers Because transformation is fit using minimal # of points, algebraic solution often suffices (e.g., noise-free soln exists when estimating H from pts)

57 Planar object recognition (what is transformation used; how many pairs must be selected in initial step?