Computing F class 13. Multiple View Geometry. Comp Marc Pollefeys

Transcription

1 Computing F class 3 Multiple View Geometr Comp Marc Pollefes

2 Multiple View Geometr course schedule (subject to change) Jan. 7, 9 Intro & motivation Projective D Geometr Jan. 4, 6 (no class) Projective D Geometr Jan., 3 Projective 3D Geometr (no class) Jan. 8, 30 Parameter Estimation Parameter Estimation Feb. 4, 6 Algorithm Evaluation Camera Models Feb., 3 Camera Calibration Single View Geometr Feb. 8, 0 Epipolar Geometr 3D reconstruction Feb. 5, 7 Fund. Matri Comp. Structure Comp. Mar. 4, 6 Planes & Homographies Trifocal Tensor Mar. 8, 0 Three View Reconstruction Multiple View Geometr Mar. 5, 7 MultipleView Reconstruction Bundle adjustment Apr., 3 Auto-Calibration Papers Apr. 8, 0 Dnamic SfM Papers Apr. 5, 7 Cheiralit Papers Apr., 4 Dualit Project Demos

3 Two-view geometr Epipolar geometr 3D reconstruction F-matri comp. Structure comp.

4 Underling structure in set of matches for rigid scenes Epipolar geometr C m π = P T l l [ e] l = π M L L l T l T m F m = 0 T + T T P P [e] Fundamental matri (33 rank matri) e = 0 e T = l m C 0 l l + T T = P P l Canonical representation: P = [I 0] P' = [[e'] F + e' v T λe']. Computable from corresponding points. Simplifies matching 3. Allows to detect wrong matches 4. Related to calibration

5 The projective reconstruction theorem If a set of point correspondences in two views determine the fundamental matri uniquel, then the scene and cameras ma be reconstructed from these correspondences alone, and an two such reconstructions from these correspondences are projectivel equivalent allows reconstruction from pair of uncalibrated images!

6 Objective Given two uncalibrated images compute (P M,P M,{X Mi }) (i.e. within similarit of original scene and cameras) Algorithm (i) Compute projective reconstruction (P,P,{X i }) (a) Compute F from i i (b) Compute P,P from F (c) Triangulate X i from i i (ii) Rectif reconstruction from projective to metric Direct method: compute H from control points P = M PH - - M = P H X M i = HXi P Stratified method: (a) Affine reconstruction: compute π I 0 H = π (b) Metric reconstruction: compute IAC ω H = A 0-0 AA T = X E = ( T ) M ωm i HX i

7 Image information provided View relations and projective objects 3-space objects reconstruction ambiguit point correspondences F projective point correspondences including vanishing points F,H π affine Points correspondences and internal camera calibration F,H ω,ω π Ω metric

8 Epipolar geometr: basic equation ' T F = 0 ' f + ' f + ' f3 + ' f + ' f + ' f3 + f3 + f3 + f33 = 0 separate known from unknown T [ ', ', ', ', ', ',,,][ f, f, f, f, f, f, f, f, f ] 0 (data) = (unknowns) (linear) ' ' ' ' ' ' M M M M M M M M ' n n ' n n ' n ' n n ' n n ' n n n M f = 0 Af = 0

9 the singularit constraint 0 F e' T = 0 Fe = 0 detf = F rank = T T T T 3 V U σ V U σ U σ V V σ σ σ U F + + = = SVD from linearl computed F matri (rank 3) T T T V U σ U σ V V 0 σ σ U F' + = = F min F- F' Compute closest rank- approimation

10

11 the minimum case 7 point correspondences 0 f ' ' ' ' ' ' ' ' ' ' ' ' = M M M M M M M M M ( ) T V,0,0,...,σ diag σ A = U V ] A[V = ( ) T 8 T [ ] V e.g.v =...7 0, λf ) (F T = = + i i i one parameter famil of solutions but F +λf not automaticall rank

12 the minimum case impose rank σ 3 (obtain or 3 solutions) F 7pts F F F 3 det( F + λf ) = a3λ + aλ + aλ + a0 = 0 (cubic equation) - det( F + λf ) = det F det(f F + λi) = 0 - F F Compute possible λ as eigenvalues of (onl real solutions are potential solutions)

13 = f f f f f f f f f n n n n n n n n n n n n M M M M M M M M M ~0000 ~0000 ~0000 ~0000 ~00 ~00 ~00 ~00! Orders of magnitude difference Between column of data matri least-squares ields poor results the NOT normalized 8-point algorithm

14 the normalized 8-point algorithm Transform image to ~[-,][-,] (0,500) (700,500) (-,) (0,0) (,) (0,0) (700,0) (-,-) (,-) Least squares ields good results (Hartle, PAMI 97)

15

16 algebraic minimization possible to iterativel minimize algebraic distance subject to det F=0 (see book if interested)

17

18 Geometric distance Gold standard Sampson error Smmetric epipolar distance

19 Gold standard Maimum Likelihood Estimation (, ˆ ) d( ', ˆ' ) d i i + i i i (= least-squares for Gaussian noise) subject to ˆ' T Fˆ = 0 Initialize: normalized 8-point, (P,P ) from F, reconstruct X i Parameterize: P = [I 0],P' = [M t],x i ˆ = PX, ˆ = i i i P' X Minimize cost using Levenberg-Marquardt (preferabl sparse LM, see book) i (overparametrized)

20

21 Gold standard Alternative, minimal parametrization (with a=) (note (,,) and (,,) are epipoles) problems: a=0 pick largest of a,b,c,d to fi epipole at infinit pick largest of,,w and of,,w 433=36 parametrizations! reparametrize at ever iteration, to be sure

22 Zhang&Loop s approach CVIU 0

23 First-order geometric error (Sampson error) ( T ) e T JJ e T e e JJ T (one eq./point JJ T scalar) e = ' T F = 0 e i = ' T F0 0 T T ( ' F) + ( ' F) + ( F) ( ) F T JJ = + T e e JJ T T T ( ' F) + ( ' F) + ( F) + ( F) ' T F (problem if some is located at epipole) advantage: no subsidiar variables required

24 Smmetric epipolar error ( F ) d( T ',, F ' ) d i i + i i T = ' F + T T ( ' F) + ( ) ( F) + ( F) ' F i

25 Some eperiments:

26 Some eperiments:

27 Some eperiments:

28 Some eperiments: Residual error: T ( ', F ) d(,f ' ) d i i + i i i (for all points!)

29 Recommendations:. Do not use unnormalized algorithms. Quick and eas to implement: 8-point normalized 3. Better: enforce rank- constraint during minimization 4. Best: Maimum Likelihood Estimation (minimal parameterization, sparse implementation)

30 Automatic computation of F (i) Interest points (ii) Putative correspondences (iii) RANSAC (iv) Non-linear re-estimation of F (v) Guided matching (repeat (iv) and (v) until stable)

31 Feature points Etract feature points to relate images Required properties: Well-defined (i.e. neigboring points should all be different) Stable across views (i.e. same 3D point should be etracted as feature for neighboring viewpoints)

32 Feature points (e.g.harris&stephens 88; Shi&Tomasi 94) Find points that differ as much as possible from all neighboring points homogeneous edge SSD M = W Δ I I T M [ ] I I w(, )dd Δ M should have large eigenvalues Feature = local maima (subpiel) of F(λ, λ ) corner

33 Feature points Select strongest features (e.g. 000/image)

34 Feature matching Evaluate NCC for all features with similar coordinates e.g. w w h h (, ) [, + ] [ + ] 0 0 0, 0 Keep mutual best matches Still man wrong matches!?

35 Feature eample Gives satisfing results for small image motions

36 Wide-baseline matching Requirement to cope with larger variations between images Translation, rotation, scaling Foreshortening Non-diffuse reflections } Illumination } geometric transformations photometric changes

37 Wide-baseline matching (Tutelaars and Van Gool BMVC 000) Wide baseline matching for two different region tpes

38 Finding more matches restrict search range to neighborhood of epipolar line (±.5 piels) rela disparit restriction (along epipolar line)

39 RANSAC Step. Etract features Step. Compute a set of potential matches Step 3. do Step 3. select minimal sample (i.e. 7 matches) Step 3. compute solution(s) for F Step 3.3 determine inliers(verif hpothesis) until Γ(#inliers,#samples)<95% } (generate hpothesis) Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches Γ = ( ( ) #inliers 7 # samples ) #matches #inliers 90% 80% 70% 60% 50% #samples

40 How man samples? Choose N so that, with probabilit p, at least one random sample is free from outliers. e.g. p=0.99 ( ( ) ) s N e = p ( ) ( ( ) ) s N = log p / log e proportion of outliers e s 5% 0% 0% 5% 30% 40% 50%

41

42

43 More problems: Absence of sufficient features (no teture) Repeated structure ambiguit Robust matcher also finds support for wrong hpothesis solution: detect repetition (Schaffalitzk and Zisserman, BMVC 98)

44 two-view geometr geometric relations between two views is full described b recovered 33 matri F

45 Special case: Enforce constraints for optimal results: Pure translation (dof), Planar motion (6dof), Calibrated case (5dof) compute essential matri; but for E, two singular values are equal.

46 Other entities? Lines give no constraint for two view geometr (but will for three and more views) Curves and surfaces ield some constraints related to tangenc

47 Degenerate cases: Degenerate cases Planar scene Pure rotation No unique solution Remaining DOF filled b noise Use simpler model (e.g. homograph) Model selection (Torr et al., ICCV 98, Kanatani, Akaike) Compare H and F according to epected residual error (compensate for model compleit)

48 The envelope of epipolar lines What happens to an epipolar line if there is noise? Monte Carlo

49 T C = ll k Σ l n=0 n=5 n=5 n=50

50 Net class: image pair rectification reconstructing points and lines