Metric Structure from Motion

Transcription

1 CS443 Final Project Metric Structure from Motion Peng Cheng 1

2 Objective of the Project Given: 1. A static object with n feature points and unknown shape. 2. A camera with unknown intrinsic parameters takes m images at several unknown position and orientation. 3. Correspondence between points in different images is known. Task: Reconstruct the metric structure of the shape. Construct 3D shape of an object from its 2D images. 2

3 Main Ideas of the Methods Construction of the projective structure 1. S. Mahamud, Y. Omori, K. McHenry, J. Ponce. Provably-convergent Iterative methods for projective structure from motion 2. P. Sturm, B. Triggs. A factorization based algorithm for multi-image projective structure and motion Construction of the metric structure 3. J. Ponce. Metric upgrade of a projective reconstruction under the rectangular pixel assumption 4. M. Pollefeys, R. Koch, L. Good. Self-calibration and metric reconstruction in spite of varying and unknown intrinsic camera parameters 5. D. Forsyth and J. Ponce, Computer vision: a modern approach 3

4 Projective Structure Reconstruction ρ ij p ij = M i P j {p ij } M i, P j, i = 1,, m, j = 1,, n 4

5 Projective Structure Reconstruction Methods - I Iterative Factorization Methods [1, 2] Extension of the factorization method for affine case. Require that all points are visible in every image. D = {ρ ij p ij } = MP, M = [M T 1,, M T m] T, P = [P 1,, P n ] Given ρ ij, estimate M and P by SVD decomposition of D. Update ρ ij. [1] does it by solving a generalized eigenvalue problem and the method is provably-convergent. [2] does it by using least square linear optimization without convergence guarantee. 5

6 Projective Structure Reconstruction Methods - II Iterative Bilinear Methods [1] Avoid the calculation of the projective depth. Do not require that all points are visible in every image, i.e., it solves the missing point problem. Compute an initial estimate of P j and normalize these vectors. Compute M i from visible points in ith image. Compute P i from projection matrices of images in which P i is visible. Both steps are solved by calculating eigenvalues. 6

7 Metric Structure Reconstruction M i, P j Q, i = 1,, m; j = 1,, n 7

8 Metric Structure Reconstruction Methods [3,4,5] Constraints on intrinsic parameters of the camera, such as zero skew, known principle point, generate invariance between projection matrices between images. ˆM = MQ = M[Q 3 q 4 ] = ρk[r t] MLM = MQ 3 Q T 3 M T = ρ KK T Different parameterization and calculation methods [3] parameterizes L by 20 parameters and solves Q 3 by calculating a linear problem first and refining it with a nonlinear optimization. [4] parameterizes L by minimal 8 parameters and solves it by nonlinear optimization. 8

9 Implementation Obtain the projective structure using the iterative factorization method [1]. Construct the metric structure using the method using the method [5]. The 1st and 10th image in the image sequence 9

10 Constructed Projective Structure 10

11 Reprojection Error in the Calculation Error = 1 nm ij (x ij m i1p j m i3 P j ) 2 + (y ij m 21P j m i3 P j ) 2 11

12 Constructed Metric Structure - I Known principle point, zero skew, unit aspect ratio 12

13 Angle between perpendicular lines: (1, 4) (6, 8): (1, 4) (5, 7): (15, 23) (5, 7): (6, 11) (13, 21): Angle between perpendicular planes: (6, 10, 11) (13, 21, 24): (9, 13, 21) (13, 16, 21): (9, 13, 21) (6, 10, 11): Distance between points: (6, 9): 0.18, (8, 11): 0.17 (6, 8): 0.098, (9, 11): (16, 24): 0.30, (15, 23): 0.29 (16, 15): 0.081, (24, 23):

14 Texture Mapped Structure 14

15 Reconstructed Metric Structure - II Known principle points 15

16 Angle between perpendicular lines: (1, 4) (6, 8): (1, 4) (5, 7): (15, 23) (5, 7): (6, 11) (13, 21): Angle between perpendicular planes: ( ) ( ): ( ) ( ): ( ) ( ): Distance between points (6 9): 0.45, (8 11): 0.44 (6 8): 0.19, (9 11): 0.16 (16 24): 0.46, (15 23): 0.45 (16 15): 0.22, (24 23):

17 Conclusion and Discussion Implemented the projective structure construction using iterative factorization algorithm. Used camera parameter constraints to upgrade the projective structure to metric structure. Since there are a lot of numerical matrix calculation in the algorithm, it is important to avoid generating ill conditional matrices. Remember normalize point coordinates before calculation. Avoid using nonlinear optimization since the initial value estimation seems another difficult problem. The number of images required depends on which kind of parameterization and optimization is used. If nonlinear optimization is used, a minimum number of images are necessary. If a linear optimization is used, more images might be necessary. For example, in the metric reconstruction, Q 3 has 8 minimal parameters. If we directly calculate Q 3 using nonlinear optimization with the known 17

18 principle point of the camera, then 4 images are enough to reconstruct the metric structure. If L = Q 3 Q T 3 is used to solve it by a linear optimization and L is parameterized as a symmetric matrix with 10 parameters, 5 images are needed. How to deal with the symmetric decomposition of L if L has less than 3 positive eigenvalues? Use nonlinear optimization to directly calculate Q 3. What is the good way to choose the initial value? What is the reason that L has less than three positive eigenvalues? Is it related to parameterization? 18

19 Acknowledgement Thank Prof. Ponce for many suggestions, Drew Gilliam for useful discussions, Kenton McHenry for the castle data. 19