Multi-View Geometry Part II (Ch7 New book. Ch 10/11 old book)

Transcription

1 Multi-View Geometry Part II (Ch7 New book. Ch 10/11 old book) Guido Gerig CS-GY 6643, Spring 2016 Credits: M. Shah, UCF CAP5415, lecture 23 Trevor Darrell, Berkeley, C280, Marc Pollefeys

2 Relates Multi-View Geometry

3 Multi-View Geometry Relates 3D World Points

4 Multi-View Geometry Relates 3D World Points Camera Centers

5 Multi-View Geometry Relates 3D World Points Camera Centers Camera Orientations

6 Multi-View Geometry Relates 3D World Points Camera Centers Camera Orientations Camera Intrinsic Parameters

7 Multi-View Geometry Relates 3D World Points Camera Centers Camera Orientations Camera Intrinsic Parameters Image Points

8 Stereo scene point image plane optical center

9 Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays

10 Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays

11 Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays

12 Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays

13 Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays Requires calibration point correspondence

14 Stereo Constraints p p? Given p in left image, where can the corresponding point p in right image be?

15 Stereo Constraints Image plane M Y 1 p Z 1 O 1 X 1 Focal plane

16 Stereo Constraints Image plane M Y 1 p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

17 Stereo Constraints Image plane M Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

18 Stereo Constraints Image plane M Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

19 Stereo Constraints Image plane M Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

20 Stereo Constraints Image plane M Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

21 Stereo Constraints Image plane M Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

22 Stereo Constraints Image plane M Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

23 Stereo Constraints Image plane M Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

24 Stereo Constraints Image plane M Epipolar Line Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane O 2 Z 2

25 Stereo Constraints Image plane M Epipolar Line Y 1 p p Y 2 X 2 Z 1 O 1 X 1 Focal plane Epipole O 2 Z 2

26 Demo Epipolar Geometry Java Applet credit to: Quang-Tuan Luong SRI Int. Sylvain Bougnoux

27 Epipolar constraint Source: M. Pollefeys

28 Epipolar constraint Potential matches for p have to lie on the corresponding epipolar line l. Source: M. Pollefeys

29 Epipolar constraint Potential matches for p have to lie on the corresponding epipolar line l. Source: M. Pollefeys

30 Epipolar constraint Potential matches for p have to lie on the corresponding epipolar line l. Source: M. Pollefeys

31 Epipolar constraint Potential matches for p have to lie on the corresponding epipolar line l. Source: M. Pollefeys

32 Epipolar constraint Source: M. Pollefeys Potential matches for p have to lie on the corresponding epipolar line l. Potential matches for p have to lie on the corresponding epipolar line l.

33 Finding Correspondences Strong constraints for searching for corresponding points! Andrea Fusiello, CVonline

34 Example Parallel Cameras: Corresponding points on horizontal lines.

35 Epipolar Constraint

36 From Geometry to Algebra

37 From Geometry to Algebra P p p O O

38 From Geometry to Algebra P p p O O

39 From Geometry to Algebra P p p O O

40 From Geometry to Algebra P p p O O

41 From Geometry to Algebra P p p O O

42 From Geometry to Algebra P p p O O

43 From Geometry to Algebra P p p O O

44 From Geometry to Algebra P p p O O

45

46

47

48 Linear Constraint: Should be able to express as matrix multiplication.

49 Review: Matrix Form of Cross Product

50 Review: Matrix Form of Cross Product

51 Matrix Form

52 The Essential Matrix

53 The Essential Matrix Based on the Relative Geometry of the Cameras Assumes Cameras are calibrated (i.e., intrinsic parameters are known) Relates image of point in one camera to a second camera (points in camera coordinate system). Is defined up to scale 5 independent parameters

54 The Essential Matrix What is εp?

55 The Essential Matrix p Similarly ε T p is the epipolar line corresponding to p in the right camera.

56 The Essential Matrix e T εe? What is εe? εe line εp converges to epipole e e (center of camera C 2 expressed in frame C 1) e

57 Pencil of epipolar lines

58 Pencil of epipolar lines

59 Epipoles The epipolar line l = εx to each point x (except e) intersects the epipole e 0. Thus e satisfies e T (εx) = (e T ε)x = 0 for all x. This implies that e T ε = 0 T or ε T e = 0. The epipole e is thus a null vector to ε T (in the left null-space of ε). Similarly, εe = 0, i.e. e is a null-vector to ε (in the right null-space of ε). MVG Hartley & Zisserman

60 MVG Hartley & Zisserman

61 Epipoles MVG Hartley & Zisserman

62 The Essential Matrix [ t ] εe = R e = 0 Similarly, ε T T T T e R [ t ] e = R [ t ] = 0 = e The essential matrix ε = [t] R has 5 degrees of freedom; 3 rotation angles in R, 3 elements in t, but arbitrary scale. Essential Matrix is singular with rank 2.

63 What if Camera Calibration is not known

64 Review: Intrinsic Camera Parameters X Y Z C Image plane Focal plane M m ( ) ( ) ( ) ( ) C C C Z Y X,, u v i j I J ( ) ( ) ( ) I I u,v ( ) ( ) ( ) ( ) ( ) = C C C v u I I Z Y X v f u f S v u Θ = = = = = 90 β α v v u u fk f fk f Қ

65 Fundamental Matrix Tε p p = 0 p and p are in camera coordinate system If u and u are corresponding image coordinates then we have: 1 T u = K p ( 1 ) T T T 1 p = K1 u p = K1 u = u K1 u = K p 1 p = K u u T K 2 T ε 1 1 K 2 u = 0 2 u T Fu = 0 F K T ε 1 = 1 K2

66 Fundamental Matrix u T Fu = 0 F K ε K T 1 = 1 2 Fundamental Matrix is singular with rank 2. The fundamental matrix F has 7 degrees of freedom: A 3 3 homogenous matrix has 8 degrees of freedom. The constraint rank(f) = 2 or det(f) = 0 reduces the number to 7. In principal F has 7 parameters up to scale and can be estimated from 7 point correspondences. Direct Simpler Method requires 8 correspondences (Olivier Faugeras, Computer Vision textbook).

67

68

69 Example II: compute F for a forward translating camera f X Y f Z

70 X Y Z f f first image second image

71 Example: forward motion 71 courtesy of Andrew Zisserman

72 Example: forward motion 72 courtesy of Andrew Zisserman

73 e e

74 Estimating Fundamental Matrix u T Fu = 0 The 8-point algorithm (Faugeras) Each point correspondence can be expressed as a linear equation: F F 11 u 1 [ u v 1] F F F v = F F F F F F F F [ uu uv u u v vv v u v 1] F = 0 F F F F

75 The 8-point Algorithm Scaling: Set F 33 to 1 -> Solve for 8 parameters.

76 Example

77 Example ctd. u T Fu = 0 Fu =l, where l is epipolar line associated to u. * *refers to normal form of line: rho = x cos(phi) + y sin(phi)

78 Example: Left to Right

79 Example: Right to Left

80 Example: Epipoles?

81 Example: Epipoles? e

82 Summary: Properties of the Fundamental matrix