Fundamental Matrix & Structure from Motion

Transcription

1 Fundamental Matrix & Structure from Motion Instructor - Simon Lucey Designing Computer Vision Apps

2 Today Review of Assignment 0. Transformations between images Structure from Motion The Essential Matrix The Fundamental Matrix

3 Assignment 1 is now released!!! Slight delay in the release. Due date is now midnight Monday October 6th.

4 Assignment 0 Everyone successfully uploaded their assignments!!!! Thanks for your patience :). Reminder of late policy. Each student is allotted a total of five late-day points for the semester. Late-day points are for use on assignments only (They cannot be used for midterm or final projects). Late-day points work as follows: A person can extend an assignment deadline by one day using one point. If a person does not have remaining late day points, late hand-ins will incur a 10% penalty per day (up to three days per assignment). No assignments will be accepted more than three days after the deadline.

5 Assignment 0 Question 1a) - Scaling of Lena Question 2a) - Accessing.xml files Question 2a) - Color conversion More detailed solutions will be placed in your AFS dropbox!!

6 AFS Dropbox Your AFS dropbox is here, /afs/cs.cmu.edu/academic/class/16423-f15-users/andrew-id Current list of drop boxes is, If you do not see your andrew ID and you are enrolled in the class, please contact Chen-Hsuan through Piazza.

7 Today Review of Assignment 0. Transformations between images Structure from Motion The Essential Matrix The Fundamental Matrix

8 Transformations between images So far we have considered transformations between the image and a plane in the world Now consider two cameras viewing the same plane There is a homography between camera 1 and the plane and a second homography between camera 2 and the plane It follows that the relation between the two images is also a homography

9 Camera under pure rotation Special case is camera under pure rotation. Homography can be showed to be Why is this?

10 Panorama Example a) b) Models for transformations c) d) Figure Computing visual panoramas. a-c) Three images of the same scene where the camera has rotated but not translated. Five matching points

11 Today Review of Assignment 0. Transformations between images Structure from Motion The Essential Matrix The Fundamental Matrix

12 Taken from Agarwal et al. Building Rome in a Day, ICCV 2009.

13 Structure from Motion For simplicity, we ll start with simpler problem Just J=2 images Known intrinsic matrix Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

14 Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince Epipolar lines

15 Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince Epipole

16 Special configurations Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

17 Today Review of Assignment 0. Transformations between images Structure from Motion The Essential Matrix The Fundamental Matrix

18 The Essential Matrix First camera: Second camera: Substituting: This is a mathematical relationship between the points in the two images, but it s not in the most convenient form. Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

19 Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince The Essential Matrix

20 The Essential Matrix The cross product term can be expressed as a matrix Defining: We now have the essential matrix relation Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

21 Properties of the Essential Matrix Rank 2: 5 degrees of freedom Non-linear constraints between elements Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

22 Recovering Epipolar Lines Equation of a line: or or Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

23 Recovering Epipolar Lines Equation of a line: Now consider This has the form where So the epipolar lines are Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

24 Recovering Epipolar Lines Every epipolar line in image 1 passes through the epipole e 1. In other words for ALL This can only be true if e 1 is in the nullspace of E. Similarly: We find the null spaces by computing, and taking the last column of and the last row of. Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

25 Decomposition of E Essential matrix: To recover translation and rotation use the matrix: We take the SVD and then we set Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

26 Four Interpretations To get the different solutions, we mutliply τ by -1 and substitute

27 Today Review of Assignment 0. Transformations between images Structure from Motion The Essential Matrix The Fundamental Matrix

28 The Fundamental Matrix Song Taken from Daniel Wedge s home page -

29 The Fundamental Matrix Now consider two cameras that are not normalised By a similar procedure to before, we get the relation or where Relation between essential and fundamental Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

30 Estimation of the Fundamental Matrix

31 Estimation of Fundamental Matrix When the fundamental matrix is correct, the epipolar line induced by a point in the first image should pass through the matching point in the second image and vice-versa. This suggests the criterion If and then Unfortunately, there is no closed form solution for this quantity. Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

32 The 8 Point Algorithm Approach: solve for fundamental matrix using homogeneous coordinates closed form solution (but to wrong problem!) Known as the 8 point algorithm Start with fundamental matrix relation Writing out in full: or Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

33 The 8 Point Algorithm Can be written as: where Stacking together constraints from at least 8 pairs of points, we get the system of equations Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

34 The 8 Point Algorithm Minimum direction problem of the form Find minimum of subject to. To solve, compute the SVD and then set to the last column of. Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

35 Fitting Concerns This procedure does not ensure that solution is rank 2. Solution: set last singular value to zero. Can be unreliable because of numerical problems to do with the data scaling better to re-scale the data first Needs 8 points in general positions (cannot all be planar). Fails if there is not sufficient translation between the views Use this solution to start non-linear optimisation of true criterion (must ensure non-linear constraints obeyed). There is also a 7 point algorithm.

36 More to read Prince et al. Chapter 16, Sections 1-3.