3D Photography: Epipolar geometry

Transcription

1 3D Photograph: Epipolar geometr Kalin Kolev, Marc Pollefes Spring 203

2 Schedule (tentative) Feb 8 Feb 25 Mar 4 Mar Mar 8 Mar 25 Apr Apr 8 Apr 5 Apr 22 Apr 29 Ma 6 Ma 3 Ma 20 Ma 27 Introduction Lecture: Geometr, Camera Model, Calibration Lecture: Features, Tracking/Matching Project Proposals b Students Lecture: Epipolar Geometr Lecture: Stereo Vision Easter Short lecture SfM / SLAM + 2 papers Project Updates Short lecture Active Ranging, Structured Light + 2 papers Short lecture Volumetric Modeling + 2 papers Short lecture Mesh-based Modeling + 2 papers Short lecture Shape-from-X + 2 papers Pentecost / White Monda Final Demos

3 Two-view geometr Three questions: (i) Correspondence geometr: Given an image point in the first image, how does this constrain the position of the corresponding point in the second image? (ii) Camera geometr (motion): Given a set of corresponding image points { i i }, i,,n, what are the cameras P and P for the two views? (iii) Scene geometr (structure): Given corresponding image points i i and cameras P, P, what is the position of (their pre-image) X in space?

4 The epipolar geometr C,C,, and X are coplanar

5 The epipolar geometr What if onl C,C, are known?

6 The epipolar geometr All points on π project on l and l

7 The epipolar geometr Famil of planes π and lines l and l Intersection in e and e

8 The epipolar geometr epipoles e,e intersection of baseline with image plane projection of projection center in other image vanishing point of camera motion direction an epipolar plane plane containing baseline (-D famil) an epipolar line intersection of epipolar plane with image (alwas come in corresponding pairs)

9 Eample: converging cameras

10 Eample: motion parallel with image plane (simple for stereo rectification)

11 Eample: forward motion e e

12 The fundamental matri F algebraic representation of epipolar geometr l' we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented b the fundamental matri F

13 The fundamental matri F geometric derivation ' H π [ e' ] H π F l' e' ' mapping from 2-D to -D famil (rank 2)

14 The fundamental matri F algebraic derivation ( λ) P λc X + + ( PP + I) l' P' C P' P + P + X( λ) F [ ] + e' P' P (note: doesn t work for CC F0)

15 The fundamental matri F correspondence condition The fundamental matri satisfies the condition that for an pair of corresponding points in the two images ' T F 0 ( ' T l' 0)

16 The fundamental matri F F is the unique 33 rank 2 matri that satisfies T F0 for all (i) Transpose: if F is fundamental matri for (P,P ), then F T is fundamental matri for (P,P) (ii) Epipolar lines: l F & lf T (iii) Epipoles: on all epipolar lines, thus e T F0, e T F0, similarl Fe0 (iv) F has 7 d.o.f., i.e. 33-(homogeneous)-(rank2) (v) F is a correlation, projective mapping from a point to a line l F (not a proper correlation, i.e. not invertible)

17 Fundamental matri for pure translation

18 Fundamental matri for pure translation

19 Fundamental matri for pure translation General motion F [ ] + e' P' P Pure translation P K[I 0] P' K[I t] P + K 0 - F [ e' ] 0 e z e e z 0 e e e 0 for pure translation F onl has 2 degrees of freedom

20 The fundamental matri F relation to homographies [ e' ] H F π l' H -T π l e' H e π valid for all plane homographies

21 The fundamental matri F relation to homographies π l π ' H requires [ l ] F π π l T e' π 0 H [ e' ] F e.g. e' T e' 0 ( )

22 Projective transformation and invariance Derivation based purel on projective concepts ˆ H, ˆ' H' ' Fˆ PX ( )( - PH H X) Pˆ Xˆ ( )( - P' H H X) Pˆ' Xˆ H' -T FH - F invariant to transformations of projective 3-space ' P' X ( P, P' ) F F ( P, P' ) canonical form P' [I 0] [M m] unique not unique P [ m] M F + ( P' P ) F [ e' ]

23 Projective ambiguit of cameras given F previous slide: at least projective ambiguit this slide: not more! ~ ~ Show that if F is same for (P,P ) and (P,P ), there eists a projective transformation H so that ~ ~ PHP and P HP lemma: P [I 0] P' [A ~ ~ P [I 0] P' [A ~ a] a~ ] T a~ ka and k ( A + av ) [ ] ~ rank 2 a A 0 af ~ a ka A ~ [ a] A [ a~ ] A ~ F af a a A ~ a A ~ a ka ~ - A 0 ka ~ - A [ ] [ ] [ ] ( ) ( ) T av H k k I v T 0 k (22-57, ok) P'H [A a] k k [ k v I T 0 k ~ T ( A - av ) ka] P'

24 The projective reconstruction theorem If a set of point correspondences in two views determine the fundamental matri uniquel, then the scene and cameras ma be reconstructed from these correspondences alone, and an two such reconstructions from these correspondences are projectivel equivalent allows reconstruction from pair of uncalibrated images!

25 Canonical cameras given F Possible choice: P [I 0] P' [[e'] F e'] + F [e'] P'P [e'] [[e'] F e'] I 0 T T ([e'] [e'] e'.e' ( e'.e') I) ( T ( T e'.e' e'.e')) F λf Canonical representation: P [I 0] P' [[e'] F + e' v T λe']

26 Epipolar geometr π Underling structure in set of matches for rigid scenes C m l M L2 L l T l 2 T m F m 2 0 e e2 Fundamental matri (33 rank 2 matri) m2 C2 l2 Canonical representation: P [I 0] P' [[e'] F + e' v T λe']. Computable from corresponding points 2. Simplifies matching 3. Allows to detect wrong matches 4. Related to calibration

27 Epipolar geometr? courtes Frank Dellaert

28 Other entities besides points? Lines give no constraint for two view geometr (but will for three and more views) Curves and surfaces ield some constraints related to tangenc (e.g. Sinha et al. CVPR 04)

29 Computation of F (and E) Linear (8-point) Minimal (7-point) Robust (RANSAC) Non-linear refinement (MLE, ) Practical approach Calibrated 5-point Calibrated + know vertical 3-point

30 Epipolar geometr: basic equation ' T F 0 ' f + ' f2 + ' f3 + ' f2 + ' f 22 + ' f23 + f3 + f32 + f33 0 separate known from unknown T [ ', ', ', ', ', ',,,][ f, f, f, f, f, f, f, f, f ] 0 (data) (unknowns) (linear) ' ' ' ' ' ' ' n n ' n n ' n ' n n ' n n ' n n n f 0 Af 0

31 f f f f f f f f f n n n n n n n n n n n n ~0000 ~0000 ~0000 ~0000 ~00 ~00 ~00 ~00! Orders of magnitude difference between column of data matri least-squares ields poor results the NOT normalized 8-point algorithm

32 the normalized 8-point algorithm Transform image to ~[-,][-,] (0,500) (700,500) (-,) (0,0) (,) (0,0) (700,0) (-,-) (,-) normalized least squares ields good results (Hartle, PAMI 97)

33 the singularit constraint 0 F e' T 0 Fe 0 detf 2 F rank T T T T 3 2 V U σ V σ U V U σ V σ σ σ U F + + SVD from linearl computed F matri (rank 3) T T T 2 V σ U V U σ V 0 σ σ U F' + F F - F' min Compute closest rank-2 approimation

34

35 the minimum case 7 point correspondences 0 f ' ' ' ' ' ' ' ' ' ' ' ' ( ) T V,0,0,...,σ σ diag A U V ] A[V ( ) T 8 T [ ] V e.g.v...7 0, ) λf (F 2 T + i i i one parameter famil of solutions but F +λf 2 not automaticall rank 2

36 the minimum case impose rank 2 σ 3 (obtain or 3 solutions) F 7pts F F F det( F + λf ) a λ + a2λ + aλ + a (cubic equation) - det( F + λf2 ) det F2 det(f2 F + λi) 0 ( det ( AB) det( A ).det ( B) ) - F 2 F Compute possible λ as eigenvalues of (onl real solutions are potential solutions)

37 RANSAC Automatic computation of F Step. Etract features Step 2. Compute a set of potential matches Step 3. do Step 3. select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers (verif hpothesis) until Γ(#inliers,#samples)<95% Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches Γ ( ( ) #inliers 7 # samples ) #matches } (generate hpothesis) #inliers 90% 80% 70% 60% 50% #samples

38 Finding more matches restrict search range to neighborhood of epipolar line (e.g. ±.5 piels) rela disparit restriction (along epipolar line)

39 Issues: (Mostl) planar scene (see net slide) Absence of sufficient features (no teture) Repeated structure ambiguit Robust matcher also finds support for wrong hpothesis solution: detect repetition (Schaffalitzk and Zisserman, BMVC 98)

40 Computing F for quasi-planar scenes QDEGSAC 337 matches on plane, off plane #inliers %inclusion of out-of-plane inliers data rank 7% success for RANSAC 00% for QDEGSAC

41 5-point relative motion (Nister, CVPR03) Linear equations for 5 points Linear solution space E X + Y + zz + ww Non-linear constraints dete 0 scale does not matter, choose w 0 cubic polnomials

42 5-point relative motion (Nister, CVPR03) Perform Gauss-Jordan elimination on polnomials [n] represents polnomial of degree n in z -z -z -z

43 Minimal relative pose with know vertical Fraundorfer, Tanskanen and Pollefes, ECCV200 -g Vertical direction can often be estimated inertial sensor vanishing point 43 5 linear unknowns linear 5 point algorithm 3 unknowns quartic 3 point algorithm

44 two-view geometr geometric relations between two views is full described b recovered 33 matri F

45 Triangulation L2 C X L 2 C2 Triangulation - calibration - correspondences

46 Backprojection Triangulation C L2 X L 2 C 2 Triangulation Iterative least-squares Maimum Likelihood Triangulation

47 Optimal 3D point in epipolar plane Given an epipolar plane, find best 3D point for (m,m 2 ) m l m 2 l 2 m m 2 l m m 2 l 2 Select closest points (m,m 2 ) on epipolar lines Obtain 3D point through eact triangulation Guarantees minimal reprojection error (given this epipolar plane)

48 Non-iterative optimal solution Reconstruct matches in projective frame b minimizing the reprojection error ( ) 2 m, PM D( m P M) 2 D + 2, Non-iterative method Determine the epipolar plane for reconstruction ( ( )) 2 m, l α + D( m l ( α) ) 2 Reconstruct optimal point from selected epipolar plane Note: onl works for two views 2 (Hartle and Sturm, CVIU 97) D 2, (polnomial of degree 6) 2 l (α) m m 2 l 2 (α) 3DOF DOF

49 Backprojection Represent point as intersection of row and column Condition for solution? Useful presentation for deriving and understanding multiple view geometr (notice 3D planes are linear in 2D point coordinates)

50 Multi-view geometr (intersection constraint) (multi-linearit of determinants) ( epipolar constraint!) (counting argument: 2-57)

51 Multi-view geometr (multi-linearit of determinants) ( trifocal constraint!) (33327 coefficients) (counting argument: 3-58)

52 Multi-view geometr (multi-linearit of determinants) ( quadrifocal constraint!) (33338 coefficients) (counting argument: 4-529)