CS231M Mobile Computer Vision Structure from motion

Transcription

1 CS231M Mobile Computer Vision Structure from motion - Cameras - Epipolar geometry - Structure from motion

2 Pinhole camera Pinhole perspective projection f o f = focal length o = center of the camera

3 z y f y' z f ' y P z y P Pinhole camera Derived using similar triangles ) z y,f z (f z) y,, (

4 From retina plane to images Piels, bottom-left coordinate systems

5 From retina plane to images y c c

6 Converting to piels y y c 1. Off set C=[c, c y ] c y (, y, z) (f c, f c y ) z z

7 Converting to piels y y c c 1. Off set 2. From metric to piels y (, y, z) (f k c, f l c y ) z z C=[c, c y ] Units: k,l : piel/m f : m Non-square piels, : piel

8 Camera Matri y y c y (, y, z) ( c, c y z z ) c Matri form? C=[c, c y ]

9 Homogeneous coordinates For details see lecture on transformations in CS131A homogeneous image coordinates homogeneous scene coordinates Converting from homogeneous coordinates

10 Camera Matri ) c z y, c z ( z) y,, ( y ' z y c c z z c y z c P y y y c y c C=[c, c y ] Camera matri K

11 Camera Skew sin 0 0 cot ' z y c c P y y c y c C = [c, c y ] P P M ' P I K 0 K has 5 degrees of freedom!

12 World reference system R,T j w k w O w i w The mapping so far is defined within the camera reference system What if an object is represented in the world reference system

13 World reference system R,T j w P k w O w P i w In 4D homogeneous coordinates: P R T P w I P P K 0 Internal parameters R 0 T 1 ' K I 0 Pw 44 K Eternal parameters R M T P w

14 Properties of Projection Points project to points Lines project to lines Distant objects look smaller

15 Properties of Projection Angles are not preserved Parallel lines meet! Vanishing point

16 Camera Calibration Calibration rig j C P' KI 0 P K R T P w P 1 P n with known positions in [O w,i w,j w,k w ] p 1, p n known positions in the image Goal: compute intrinsic and etrinsic parameters

17 Camera Calibration Calibration rig image j C P' KI 0 P K R T P w P 1 P n with known positions in [O w,i w,j w,k w ] p 1, p n known positions in the image Goal: compute intrinsic and etrinsic parameters

18 CS231M Mobile Computer Vision Structure from motion - Cameras - Epipolar geometry - Structure from motion

19 Can we recover the structure from a single view? Pinhole perspective projection P p O w Calibration rig Scene C Camera K Why is it so difficult? Intrinsic ambiguity of the mapping from 3D to image (2D)

20 Can we recover the structure from a single view? Intrinsic ambiguity of the mapping from 3D to image (2D) Courtesy slide S. Lazebnik

21 Two eyes help! X? K =known 2 1 R, T O 2 O 1 This is called triangulation K =known

22 Structure from motion problem X j 1j M m M 1 mj 2j M 2 Given m images of n fied 3D points ij = M i X j, i = 1,, m, j = 1,, n

23 Structure from motion problem X j 1j M m M 1 mj 2j M 2 From the mn correspondences ij, can we estimate: m projection matrices M i n 3D points X j motion structure

24 Study relationship between X, 1 and 2 X K =known 2 1 R, T O 2 O 1 Epipolar geometry! K =known

25 Epipolar geometry X 1 2 e 1 e 2 Epipolar Plane Epipoles e 1, e 2 Baseline Epipolar Lines O 1 O 2 = intersections of baseline with image planes = projections of the other camera center

26 Epipolar geometry X For details see CS131A Lecture e 1 e 2 Epipolar Plane Epipoles e 1, e 2 Baseline Epipolar Lines O 1 O 2 = intersections of baseline with image planes = projections of the other camera center

27 Eample: Converging image planes e e

28 Eample: Parallel image planes X e e 2 O 1 O 2 Baseline intersects the image plane at infinity Epipoles are at infinity Epipolar lines are parallel to ais

29 Eample: Parallel Image Planes e at infinity e at infinity

30 Why are epipolar constraints useful? - Two views of the same object - Suppose I know the camera positions and camera matrices - Given a point on left image, how can I find the corresponding point on right image?

31 Why are epipolar constraints useful? X O 1 O 2

32 Why are epipolar constraints useful? - Two views of the same object - Suppose I know the camera positions and camera matrices - Given a point on left image, how can I find the corresponding point on right image?

33 Essential matri P p p O R, T O Assume camera matrices are known p T E E p T R 0 E = essential matri (Longuet-Higgins, 1981)

34 b a b a ] [ z y y z y z b b b a a a a a a Cross product as matri multiplication

35 Epipolar Constraint P p p O R, T O p T F K F p 0 T 1 T R K F = Fundamental Matri (Faugeras and Luong, 1992)

36 Epipolar Constraint p T F p P p 1 p 2 e 1 e 2 O 1 O 2 F p 2 is the epipolar line associated with p 2 (l 1 = F p 2 ) F T p 1 is the epipolar line associated with 1 (l 2 = F T p 1 ) F e 2 = 0 and F T e 1 = 0 F is 33 matri; 7 DOF F is singular (rank two)

37 Why F is useful? l = F T - Suppose F is known - No additional information about the scene and camera is given - Given a point on left image, how can I find the corresponding point on right image?

38 Why F is useful? F captures information about the epipolar geometry of 2 views + camera parameters MORE IMPORTANTLY: F gives constraints on how the scene changes under view point transformation (without reconstructing the scene!) Powerful tool in: 3D reconstruction Multi-view object/scene matching

39 O O p p P Estimating F The Eight-Point Algorithm 1 v u p P 1 v u p P 0 F p p T (Hartley, 1995) (Longuet-Higgins, 1981)

40 Estimating F OPENCV: findfundamentalmat

41 CS231M Mobile Computer Vision Structure from motion - Cameras - Epipolar geometry - Structure from motion

42 Structure from motion problem X j 1j M m M 1 mj 2j M 2 From the mn correspondences ij, can we estimate: m projection matrices M i n 3D points X j motion structure

43 Similarity Ambiguity The scene is determined by the images only up a similarity transformation (rotation, translation and scaling) This is called metric reconstruction Similarity The ambiguity eists even for (intrinsically) calibrated cameras For calibrated cameras, the similarity ambiguity is the only ambiguity [Longuet-Higgins 81]

44 Similarity Ambiguity It is impossible based on the images alone to estimate the absolute scale of the scene (i.e. house height)

45 Structure from Motion Ambiguities Projective In the general case (nothing is known) the ambiguity is epressed by an arbitrary affine or projective transformation M j i X j M K i i R i T i H X j M H 1 j j M i X j -1 M H H X i j

46 Projective Ambiguity R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd edition, 2003

47 Metric reconstruction (upgrade) The problem of recovering the metric reconstruction from the perspective one is called self-calibration Stratified reconstruction: from perspective to affine from affine to metric

48 Mobile SFM Intrinsic camera parameters are known or can be calibrated. For calibrated cameras, the similarity ambiguity is the only ambiguity [Longuet-Higgins 81] No need for stratified solution or auto-calibration Similarity Metric reconstruction can be determined if a calibration pattern is used or the absolute size of an known object is given.

49 Structure-from-Motion Algorithms Algebraic approach (by fundamental matri) Factorization method (by SVD) Bundle adjustment

50 Algebraic approach (2-view case) 1j M 1 2j M i j i X j M 2 Apply a projective transformation H such that: M H 1 I 0 1 M H A b 1 Canonical perspective cameras 2

51 Algebraic approach (2-view case) 1. Compute the fundamental matri F from two views (eg. 8 point algorithm) 2. Compute b and A from F Compute b as least sq. solution of F b = 0, with b =1 using SVD; b is an epipole A= [b ] F 3. Use b and A to estimate projective cameras M 0 M [b ]F b 1 I 2 4. Use these cameras to triangulate and estimate points in 3D For details, see CS231A, lecture 7

52 Structure-from-Motion Algorithms Algebraic approach (by fundamental matri) Factorization method (by SVD) Bundle adjustment

53 Affine structure from motion (simpler problem) Image World Image From the mn correspondences ij, estimate: m projection matrices M i (affine cameras) n 3D points X j

54 Affine cameras X Camera matri M for the affine case u v M X 1 AX b; M A b

55 Centering the data X j ˆ ij Normalize points w.r.t. centroids of measurements from each image ij AX j b ˆ A ij i X j

56 mn m m n n D ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ cameras (2m ) points (n ) A factorization method - factorization Let s create a 2m n data (measurement) matri:

57 Let s create a 2m n data (measurement) matri: n m mn m m n n X X X A A A D ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ cameras (2m 3) points (3 n ) The measurement matri D = M S has rank 3 (it s a product of a 2m3 matri and 3n matri) A factorization method - factorization (2m n) M S

58 2m Factorizing the Measurement Matri = Measurements Motion Structure 3 n 3 n D = MS

59 2m Factorizing the Measurement Matri Singular value decomposition of D: = D U W V T n n n n n

60 Factorizing the Measurement Matri Since rank (D)=3, there are only 3 non-zero singular values

61 Factorizing the Measurement Matri S = structure M = Motion (cameras)

62 Factorizing the Measurement Matri What is the issue here? D has rank>3 because of: measurement noise affine approimation T Theorem: When D has a rank greater than p, U pwpvp is the best possible rank- p approimation of A in the sense of the Frobenius norm. T A0 U3 D U3W3V3 P W V T 0 3 3

63 Affine Ambiguity = D M C C -1 S The decomposition is not unique. We get the same D by using any 3 3 matri C and applying the transformations: M MC S C -1 S M S Additional constraints must be enforced to resolve this ambiguity

64 Reconstruction results C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2): , November 1992.

65 Structure-from-Motion Algorithms Algebraic approach (by fundamental matri) Factorization method (by SVD) Bundle adjustment

66 Bundle adjustment Non-linear method for refining structure and motion Minimizing re-projection error E(M, X) m i1 n j1 D 2, X ij M i j X j? M 1 X j 1j 3j P 1 M 2 X j 2j M 3 X j P 2 P 3

67 Bundle adjustment Non-linear method for refining structure and motion Minimizing re-projection error E(M, X) m i1 n j1 D 2, X ij M i Advantages Handle large number of views Handle missing data Can leverage standard optimization packaged such as Levenberg-Marquardt Limitations Large minimization problem (parameters grow with number of views) Requires good initial condition Used as the final step of SFM j

68 Results and applications Courtesy of Oford Visual Geometry Group Lucas & Kanade, 81 Chen & Medioni, 92 Debevec et al., 96 Levoy & Hanrahan, 96 Fitzgibbon & Zisserman, 98 Triggs et al., 99 Pollefeys et al., 99 Kutulakos & Seitz, 99 Levoy et al., 00 Hartley & Zisserman, 00 Dellaert et al., 00 Rusinkiewic et al., 02 Nistér, 04 Brown & Lowe, 04 Schindler et al, 04 Lourakis & Argyros, 04 Colombo et al. 05 Golparvar-Fard, et al. JAEI 10 Pandey et al. IFAC, 2010 Pandey et al. ICRA 2011 Microsoft s PhotoSynth Snavely et al., Schindler et al., 08 Agarwal et al., 09 Frahm et al., 10

69 CS231M Mobile Computer Vision Net lecture: Eample of a SFM pipeline for mobile devices: the VSLAM pipeline

70