Computer Vision I - Algorithms and Applications: Multi-View 3D reconstruction

Transcription

1 Computer Vision I - Algorithms and Applications: Multi-View 3D reconstruction Carsten Rother 09/12/2013 Computer Vision I: Multi-View 3D reconstruction

2 Roadmap this lecture Computer Vision I: Multi-View 3D reconstruction 09/12/ Two-view reconstruction From projective to metric space (e.g. self-calibration) Multi-view reconstruction Iterative projective cameras Closed form: affine cameras Closed form: reference plane Next lecture: dense labeling problems in computer vision: stereo matching ICP, KinectFusion

3 3D reconstruction - Definitions Computer Vision I: Multi-View 3D reconstruction 09/12/ Sparse Structure from Motion (SfM) SLAM in robotics: Simultaneous Localization and Mapping: Place a robot in an unknown location and in an unknown environment and have the robot incrementally build a map of this environment while simultaneously using this map to compute its location Dense Multi-view reconstruction

4 Roadmap this lecture Computer Vision I: Multi-View 3D reconstruction 09/12/ Two-view reconstruction From projective to metric space (e.g. self-calibration) Multi-view reconstruction Iterative projective cameras Closed form: affine cameras Closed form: reference plane Next lecture: dense labeling problems in computer vision: stereo matching ICP, KinectFusion

5 First: Two-View reconstruction Computer Vision I: Multi-View 3D reconstruction 09/12/2013 5

6 Epipolar Geometry (Reminder) Computer Vision I: Robust Two-View Geometry 09/12/ Fundamental matrix F: p 2 T Fp 1 = 0

7 2-view reconstruction Computer Vision I: Multi-View 3D reconstruction 09/12/ We have x T Fx = 0 Can we get P, P such that: x = PX ; x = P X and x T Fx = 0 for all 3D points X Derivation (blackboard) see HZ page 256 P = I ]; P = e F e ]

8 Derivation Computer Vision I: Multi-View 3D reconstruction 09/12/ T

9 Derivation Computer Vision I: Multi-View 3D reconstruction 09/12/2013 9

10 Triangulation - algebraic Computer Vision I: Multi-View 3D reconstruction 09/12/ Input: x, x, P, P Output: X s Triangulation is also called intersection Simple algebraic solution: 1) λx = P X and λ x = P X 2) Eliminate λ, λ by taking ratios. This gives 4 linear independent equations for 3 unknowns: X = (X 1, X 2, X 3, X 4 ) where X = 1. An example ratio is: x 1 x 2 = p 1 X 1 +p 2 X 2 +p 3 X 3 +p 4 X 4 p 5 X 1 +p 6 X 2 +p 7 X 3 +p 8 X 4 3) This gives (as usual) a least square optimization problem: A X = 0 with X = 1 where A is of size 4 4. This can be solved in closed-form using SVD.

11 Triangulation - geometric Computer Vision I: Multi-View 3D reconstruction 09/12/ Minize re-projection error with Fixed fundamental matrix. ^ ^ ^ ^ ^ ^ x, x = argmin d x, x 2 + d x, x 2 subject to x Fx = 0 x, ^ x ^

12 Triangulation - geometric Computer Vision I: Multi-View 3D reconstruction 09/12/ Minimize re-projection error with fixed fundamental matrix. ^ ^ ^ ^ ^ ^ x, x = argmin d x, x 2 + d x, x 2 subject to x Fx = 0 x, ^ x ^ Solution can be expressed as a 6-degree polynomial in t This has up to 6 solutions and can computed (roots of polynomial)

13 Triangulation - uncertainty Computer Vision I: Multi-View 3D reconstruction 09/12/ Large baseline Smaller area smaller baseline Larger area Very small baseline Very larger area

14 Roadmap this lecture Computer Vision I: Multi-View 3D reconstruction 09/12/ Two-view reconstruction From projective to metric space (e.g. self-calibration) Multi-view reconstruction Iterative projective cameras Closed form: affine cameras Closed form: reference plane Next lecture: dense labeling problems in computer vision: stereo matching ICP, KinectFusion

15 3D reconstruction Computer Vision I: Multi-View 3D reconstruction 09/12/ Given: m cameras and n static 3D points Formally: x ij = P j X i for j = 1 m; i = 1 n Important in practice we do not have all points visible in all views, i.e. Number x ij mn Goal: find all P j s and X i s Example: Visibility matrix

16 Iterative multi-view reconstruction Method 1 Computer Vision I: Multi-View 3D reconstruction 09/12/ Three views of an un-calibrated or calibrated camera

17 Iterative multi-view reconstruction Method 1 Computer Vision I: Multi-View 3D reconstruction 09/12/ Three views of an un-calibrated or calibrated camera Similiar problem as intersection (and same as camera calibration): λx = P X and λ x = PX with 6 points you get 12 constraints and we have 11 unknowns.

18 Iterative multi-view reconstruction Method 1 Computer Vision I: Multi-View 3D reconstruction 09/12/ Three views of an un-calibrated or calibrated camera Similiar problem as intersection (and same as camera calibration): λx = P X and λ x = PX with 6 points you get 12 constraints and we have 11 unknowns. You can iterate between: intersection and re-sectioning to get all points and cameras reconstructed (in projective or metric space)

19 Iterative multi-view reconstruction Method 2 Computer Vision I: Multi-View 3D reconstruction 09/12/ Three views of an un-calibrated or calibrated camera Reconstruct Points and Camera 1 and 2

20 Iterative multi-view reconstruction Method 2 Computer Vision I: Multi-View 3D reconstruction 09/12/ Three views of an un-calibrated or calibrated camera Zipp the 2 reconstructions together. 1. They share a camera and 7+ points (needed for F-matrix). 2. Get H 4 4 from X 1 7 = HX 1 7 (here points X 1 7 are in second reconstruction and X 1 7 in first reconstruction) Reconstruct Points and Camera 2 and 3 3. Zipp them together: P 3 = P 3 H 1 ; X i = HX i (Here P 3 is camera in second reconstruction) Note:

21 The full pipeline Computer Vision I: Multi-View 3D reconstruction 09/12/ Comment: Between 3 and 4 views there exists Trifocal and Quadrifocal tensor (as Fundamental matrix for 2 views) not discussed in this lecture [See page 453 HZ]

22 Bundle adjustment Computer Vision I: Multi-View 3D reconstruction 09/12/ Global refinement of jointly structure (points) and cameras Minimize geometric error: argmin {P s,x s} α ij d(p j X i, x ij ) j i here α ij is 1 if X j visible in view P j (otherwise 0) Non-linear optimization with e.g. Levenberg-Marquard

23 Example Computer Vision I: Multi-View 3D reconstruction 09/12/

24 Main Problem of iterative methods is Drift Computer Vision I: Multi-View 3D reconstruction 09/12/ Solution: 1) look for Loop closure if possible 2) global methods (next)

25 Roadmap this lecture Computer Vision I: Multi-View 3D reconstruction 09/12/ Two-view reconstruction From projective to metric space (e.g. self-calibration) Multi-view reconstruction Iterative projective cameras Closed form: affine cameras Closed form: reference plane Next lecture: dense labeling problems in computer vision: stereo matching ICP, KinectFusion

26 Reminder: affine cameras (from lecture 4) Computer Vision I: Multi-View 3D reconstruction 09/12/ Affine camera has 8 DoF: x y 1 = a b c d e f g h X Y Z 1 In short: x = MX + t ~ ~

27 Reminder: affine cameras (from lecture 4) Computer Vision I: Multi-View 3D reconstruction 09/12/ (normal focal length) Parallel 3D lines map to parallel 2D lines (since points stay at infinity) (very large focal length) Close to parallel projection

28 Multi-View Reconstruction for affine cameras Computer Vision I: Multi-View 3D reconstruction 09/12/ (derivation on blackboard)

29 Multi-View Reconstruction for affine cameras Computer Vision I: Multi-View 3D reconstruction 09/12/ (derivation on blackboard) W MX = 0 W = MX

30 Comments / Extensions Computer Vision I: Multi-View 3D reconstruction 09/12/ Main restriction is that all points have to be visible in all views. (can be used for a subset of views and then zipping subviews together) Extensions to missing data have been done (see HZ chapter 18) Extensions to projective cameras have been done (ch. 18.4) Extensions to non-rigidly moving scenes (ch. 18.3)

31 Roadmap this lecture Computer Vision I: Multi-View 3D reconstruction 09/12/ Two-view reconstruction From projective to metric space (e.g. self-calibration) Multi-view reconstruction Iterative projective cameras Closed form: affine cameras Closed form: reference plane Next lecture: dense labeling problems in computer vision: stereo matching ICP, KinectFusion

32 Direct reference plane approach (DRP) Computer Vision I: Multi-View 3D reconstruction 09/12/ H = KR is called infinity homography since it is the mapping from plane at infinity to image: x y x x x = KR (I C) ~ z =KR y z = H y z 0 Idea: simply define an arbitrary plane as π = 0,0,0,1 T (this can be done in projective space) [Rother PhD Thesis 2003]

33 Direct reference plane approach (DRP) Computer Vision I: Multi-View 3D reconstruction 09/12/ Derivation on blackboard [Rother PhD Thesis 2003]

34 Result Computer Vision I: Multi-View 3D reconstruction 09/12/

35 How to get infinite Homographies Computer Vision I: Multi-View 3D reconstruction 09/12/ Real Plane in the scene: Fixed / known K and R, e.g. translating camera with fixed camera intrinsics Orthogonal scene directions and a square pixel camera see above we get out: K, R (up to a small ambiguity)

36 Result: University Stockholm Computer Vision I: Multi-View 3D reconstruction 09/12/ (Show video)

37 Half-way slide Computer Vision I: Multi-View 3D reconstruction 09/12/ Minutes break

38 Roadmap this lecture Computer Vision I: Multi-View 3D reconstruction 09/12/ Two-view reconstruction From projective to metric space (e.g. self-calibration) Multi-view reconstruction Iterative projective cameras Closed form: affine cameras Closed form: reference plane Next lecture: dense labeling problems in computer vision: stereo matching ICP, KinectFusion

39 Scale ambiguity Computer Vision I: Multi-View 3D reconstruction 09/12/ Is the pumpkin 5 meters or 30 cm high?

40 Scale ambiguity Computer Vision I: Multi-View 3D reconstruction 09/12/

41 Structure from Motion Ambiguity Computer Vision I: Multi-View 3D reconstruction 09/12/ We can always write: x = P k k X = 1 k P (kx) It is impossible to recover the absolute scale of the scene!

42 Projective ambiguity Computer Vision I: Multi-View 3D reconstruction 09/12/ We can write (most general): x ij = P j X i = P j Q 1 QX i = P j X i Q has 15 DoF (projective ambiguity) If we do not have any additional information about the cameras or points then we cannot recover Q. Possible information (we will see details later) known calibration matrix calibration matrix is same for all cameras external constraints: given 3D points

43 Projective ambiguity Computer Vision I: Multi-View 3D reconstruction 09/12/ D points map to image points This is a protectively correct reconstruction but not a nice looking one

44 Affine ambiguity Computer Vision I: Multi-View 3D reconstruction 09/12/ We can write (most general): x ij = P j X i = P j Q 1 QX i = P j X i Q has now 12 DoF (affine ambiguity) With affine cameras we get only affine ambiguity Q leaves the plane at infinity π = 0,0,0,1 T in place: any point on π moves like: Q a, b, c, 0 T = a, b, c, 0 Therefore parallel 3D lines stay parallel for any Q

45 From Projective to Affine Computer Vision I: Multi-View 3D reconstruction 09/12/ X 3 X 2 The red directions point to places where the projection of a vanishing point lies. X 1 Step: Take points X 1 3 = x 1 3, y 1 3, z 1 3, 1 T and move to a point at infinity: x 1 3, y 1 3, z 1 3, 0 T. Same as having plane at infinity in its canonical position π = 0,0,0,1 T

46 Affine ambiguity Computer Vision I: Multi-View 3D reconstruction 09/12/ D Points at infinity stay at infinity

47 Similarity Ambiguity (Metric space) Computer Vision I: Multi-View 3D reconstruction 09/12/ We can write (most general): x ij = P j X i = P j Q 1 QX i = P j X i Q has now 7 DoF (similarity ambiguity) Q preserves angles, ratios of lengths, etc. For visualization purpose this ambiguity is sufficient. We don t need to know what 1m, 1cm, 1mm, etc. means Note, if we don t care about the choice of Q we can set for instance the camera center of first camera to 0.

48 From Projective/Affine to Metric Computer Vision I: Multi-View 3D reconstruction 09/12/ Essentially, we need the true camera calibration matrices K (see details later) Known K Steps: (see HZ page 277) One option: with known K operate with essential matrix and get out R,C ~ of all cameras

49 Similarity Ambiguity Computer Vision I: Multi-View 3D reconstruction 09/12/

50 How to upgrade a reconstruction Computer Vision I: Multi-View 3D reconstruction 09/12/ Illustrating some ways to upgrade from Projective to Affine and Metric (see details in HZ page 270ff and chapter 19) Camera is calibrated Calibration from external constraints Calibration from a mix of in- and external constraints x ij = P j X i = P j Q 1 QX i = P j X i Find plane at infinity and move in canonical position: One of the cameras is affine see HZ page 271) 3 non-collinear 3D vanishing points Calibration from internal constraints, called self/auto calibration Translational motion (HZ page 268)

51 Projective to Affine Three 3D vanishing points Computer Vision I: Multi-View 3D reconstruction 09/12/ Goal: Find plane at infinity and move to π = 0,0,0,1 T Possible method 1. Identify three pairs of lines in the 3D reconstruction that have to be parallel, respectively. 2. Compute the three 3D intersection points X Move X 1 3 somewhere on the plane at infinity π = 0,0,0,1 T. Define the equation: X 1 3 = Q X 1 3 where X 1 3 = 1,0,0,0 T, 0,1,0,0 T, 0,0,1,0 T The red directions point to places where the projection of a vanishing point lies. 4. Compute an Q using SVD 5. Update all points and cameras P j = Q 1 P j ; X i = QX i

52 Projective to Metric Direct Method Computer Vision I: Multi-View 3D reconstruction 09/12/ Given: five known 3D points Compute Q: 1) QX i = X i (each 3D point gives 3 linear independent equations) 2) 5 points give 15 equations, enough to compute Q using SVD Upgrade cameras and points: P j = P j Q 1 and X i = QX i

53 But without external knowledge? Computer Vision I: Multi-View 3D reconstruction 09/12/ For a camera P = K [I 0] the ray outwards is: x = P X hence X = K 1 x The angle Θ is computed as the normalized rays d 1, d 2 : cos Θ = d 1 T d 2 d 1 T d 1 d 2 T d 2 = K 1 x 1 T K 1 x 2 K 1 x 1 T K 1 x 1 K 1 x 2 T K 1 x 2 = x 1 T ω x 2 x 1 T ωx 1 x 2 T ωx 2 We define the matrix: ω = K T K 1 Comment: (K 1 ) T = (K T ) 1 =: K T

54 But without external knowledge? Computer Vision I: Multi-View 3D reconstruction 09/12/ We have: cos Θ = x 1 T ω x 2 x 1 T ωx 1 x 2 T ωx 2 If we were to know ω then we can compute angle Θ K can be derived from ω = K T K 1 using Cholesky decomposition (see HZ page 582) Comment, if Θ = 90 o then we have x 1 T ω x 2 = 0 How do we get ω?

55 ω is the Image of the absolute Conic (IAC) Computer Vision I: Multi-View 3D reconstruction 09/12/ ω is called the image of the absolute conic Ω = I 3 3 on the plane at infinity π = 0,0,0,1 T Remember a conic C is defined as: x T C x = 0. Here it is: x, y, 1 Ω x, y, 1 T = 0 That is: x 2 + y 2 = 1 an imaginary circle with radius i the points (i, 0) and (0, i) lie on the conic. H ω image

56 ω is the Image of the absolute Conic (IAC) Computer Vision I: Multi-View 3D reconstruction 09/12/ ω is called the image of the absolute conic Ω = I 3 3 on the plane at infinity π = 0,0,0,1 T Proof. H 1. The homography H = KR is the mapping from the pane at infinity to the image plane. Since: ~ x = KR I C] x, y, z, 0 T is x = KR x, y, z T ω image 2. The conic C = Ω = I 3 3 maps from the plane at infinity to π to the image as: ω = H T CH 1 = KR T I KR 1 = K T R T R 1 K 1 = K T K 1 Note, ω depends on K only and not on R, C. Hence it plays a central role in auto-calibration. ~

57 Degrees of Freedom of ω Computer Vision I: Multi-View 3D reconstruction 09/12/ We have: K = f s p x 0 mf p y then K 1 = a b c 0 d e where a, b, c, d, e are some values that depend on: f, m, s, p x, p y Then it is: ω = (K 1 ) T K 1 = a 0 0 b d 0 c e 1 a b c 0 d e = a 2 ab ac ab b 2 + d 2 bc + de ac bc + de c 2 + e = ω 1 ω 2 ω 3 ω 2 ω 4 ω 5 ω 3 ω 5 ω 6 This means that ω has 5 DoF (scale is not unique)

58 How to compute the IAC? Computer Vision I: Multi-View 3D reconstruction 09/12/ External constraints: orthogonal directions, etc. f s p x K = 0 mf p y camera matrix Internal constraints: Square pixels (i.e. m = 1, s = 0 in K) Camera matrix is the same in two or more views (e.g. video sequence with no zooming) If only internal constraints are used then it is called auto/self-calibration.

59 How to compute the IAC? Computer Vision I: Multi-View 3D reconstruction 09/12/ We need 5 constraints on ω to determine it uniquely

60 Example: internal + external constraints Computer Vision I: Multi-View 3D reconstruction 09/12/ Square pixel cameras (i.e. m = 1, s = 0 in K) gives two constraints: ω 1 = ω 4 and ω 2 = 0 f s p x K = 0 mf p y camera matrix Then ω = ω 1 0 ω 3 0 ω 1 ω 5 ω 3 ω 5 w 6 with only 3 DoF v 1 Given 3 image points v 1 3 that point to orthogonal directions, respectively: v 1 T ω v 2 = 0; v 1 T ω v 3 = 0; v 2 T ω v 3 = 0 v 2 v 3 This gives an equation system Aω = 0 with A of size 3 4. Hence ω can be obtained with SVD K can be derived from ω using Cholesky decomposition

61 Example: internal constraints (practically important) Computer Vision I: Multi-View 3D reconstruction 09/12/ Assume two cameras with: s = 0, and m, p x, p y known The only unknowns are the two focal lengths: f 0, f 1 Then you get the so-called Kruppa equations: u T 1 ω 1 0 u 1 = u 0 T ω 1 0 u 1 = u 0 T ω 1 0 u 0 σ 2 0 v T 0 ω 1 1 v 0 σ 0 σ 1 v T 0 ω 1 1 v 1 σ 2 1 v T 1 ω 1 1 v 1 f s p x K = 0 mf p y camera matrix where SVD of F = u 0 u 1 e 1 0 σ 1 0 σ v 0 T v 1 T e 0 T 1 and ω i 1 = (K i T K i 1 ) = K i K i T = diag(f i 2, f i 2, 1) This can be solved in closed form (see next slide)

62 Sketch of the solution for f 0, f 1 Computer Vision I: Multi-View 3D reconstruction 09/12/

63 Example: internal constraints (sketch) Computer Vision I: Multi-View 3D reconstruction 09/12/ Assume K is constant over 3+ frames then K can be computed We know (lecture 6) we can get K, R, C from P = K R (I 3 3 C) ~ ~ We have already P 1, P 2, P 3 and it is x i1 = P 1 X i = P 1 Q 1 QX i = P 1 X i x i2 = P 2 X i = P 2 Q 1 QX i = P 2 X i x i3 = P 3 X i = P 3 Q 1 QX i = P 3 X i Try to find a Q such that all P 1, P 2, P 3 have the same K but different ~ R 1 3 and C 1 3 See details in chapter 19 HZ

64 How to upgrade a reconstruction Computer Vision I: Multi-View 3D reconstruction 09/12/ Illustrating some ways to upgrade from Projective to Affine and Metric (see details in HZ page 270ff and chapter 19) Camera is calibrated Calibration from external constraints (example: 5 known 3D points) Calibration from a mix of in- and external constraints (example: single camera: 3 orthogonal vanishing points and a square pixel camera) Calibration from internal constraints, called self/auto calibration (example: 3+ views with same K; or two cameras with unknown focal length) Find plane at infinity and move in canonical position: One of the cameras is affine. See HZ page 271) 3 non-collinear 3D vanishing points Translational motion (HZ page 268)

65 Roadmap this lecture Computer Vision I: Multi-View 3D reconstruction 09/12/ Two-view reconstruction From projective to metric space (e.g. self-calibration) Multi-view reconstruction Iterative projective cameras Closed form: affine cameras Closed form: reference plane Next lecture: dense labeling problems in computer vision: stereo matching ICP, KinectFusion