Lecture 5 Epipolar Geometry

Lecture 5 Epipolar Geometry Professor Silvio Savarese Computational Vision and Geometry Lab Silvio Savarese Lecture 5-24-Jan-18

Lecture 5 Epipolar Geometry Why is stereo useful? Epipolar constraints Essential and fundamental matrix Estimating F Examples Reading: [AZ] Chapter: 4 Estimation 2D perspective transformations Chapter: 9 Epipolar Geometry and the Fundamental Matrix Transformation Chapter: 11 Computation of the Fundamental Matrix F [FP] Chapter: 7 Stereopsis Chapter: 8 Structure from Motion Silvio Savarese Lecture 5-24-Jan-18

Recovering structure from a single view Pinhole perspective projection P p O w Calibration rig Scene C Camera K From calibration rig location/pose of the rig, K From points and lines at infinity + orthogonal lines and planes structure of the scene, K Knowledge about scene (point correspondences, geometry of lines & planes, etc

Recovering structure from a single view Pinhole perspective projection P p O w Calibration rig Scene C Camera K Why is it so difficult? Intrinsic ambiguity of the mapping from 3D to image (2D)

Recovering structure from a single view Intrinsic ambiguity of the mapping from 3D to image (2D) Courtesy slide S. Lazebnik

Two eyes help!

Two eyes help! P = l l" [Eq. 1] P l l K =known p p K =known R, T O 1 O 2 This is called triangulation

Triangulation Find P* that minimizes d(p, M P*)+ d(p, M P*) [Eq. 2] P* P MP* p M P* p O 1 O 2

Multi (stereo)-view geometry Camera geometry: Given corresponding points in two images, find camera matrices, position and pose. Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images. Correspondence: Given a point p in one image, how can I find the corresponding point p in another one?

Epipolar geometry P p p e e Epipolar Plane Epipoles e, e Baseline Epipolar Lines O 1 O 2 = intersections of baseline with image planes = projections of the other camera center

Example of epipolar lines

Example: Parallel image planes P e v p p v e u u O 1 O 2 Baseline intersects the image plane at infinity Epipoles are at infinity Epipolar lines are parallel to u axis

Example: Parallel Image Planes

Example: Forward translation e P O 2 e O 1 The epipoles have same position in both images Epipole called FOE (focus of expansion)

Epipolar Constraint p Where is p? - Two views of the same object - Given a point on left image, how can I find the corresponding point on right image?

Epipolar geometry P Epipolar line 2 p p O 1 O 2

Epipolar Constraint p p Epipolar line 2

v Epipolar Constraint P p p u O 1 R, T O 2 M = K! " I 0! # M P = # # " u v 1 $ & & = p & % # $ " M = K R T R T T #$ " u! % $ [Eq. 3] M! P = $ v! = p [Eq. 4] $ # 1 & % &

Epipolar Constraint P! # K canonical = # # " 1 0 0 0 1 0 0 0 1 $ & & & % p p O 1 R, T O 2 M = K! " I 0 # $ K = K are known (canonical cameras) " M = K R T R T T #$ % & M = [ I 0] [Eq. 5] " M = R T R T T #$ % & [Eq. 6]

The cameras are related by R, T Camera 2 p T O 2 Camera 1 O 1 T = O 2 in the camera 1 reference system R is the rotation matrix such that a vector p in the camera 2 is equal to R p in camera 1.

Epipolar Constraint P p p O 1 R, T O 2 p in first camera reference system is T ((R p ")+T) = T (R p ") = R p! +T is perpendicular to epipolar plane [ ( )] = 0 T R p p T [Eq. 7]

b a b a ] [ 0 0 0 = ú ú ú û ù ê ê ê ë é ú ú ú û ù ê ê ê ë é - - - = z y x x y x z y z b b b a a a a a a Cross product as matrix multiplication

Epipolar Constraint P p p O 1 R, T O 2 [ T (R p )] 0 p T = [ T ] R p 0 p T = [Eq. 8] [Eq. 9] E = Essential matrix (Longuet-Higgins, 1981)

Epipolar Constraint p T E p = 0 P [Eq. 10] p l l e e p O 1 O 2 l = E p is the epipolar line associated with p l = E T p is the epipolar line associated with p E e = 0 and E T e = 0 E is 3x3 matrix; 5 DOF E is singular (rank two)

Epipolar Constraint P p p O 1 O 2 M = K[ I 0] R, T " M = K R T R T T #$ p c = K 1 p p c! = K 1 p! [Eq. 11] [Eq. 12] % &

p c = K 1 p [Eq. 11] Epipolar Constraint P p c! = K 1 p! [Eq. 12] p p [Eq.9] O 1 O 2 p T c [ T ] R # -1 T p = 0 (K p) [ T ] R K p = 0 c -1 p T K [ T ] R K -1 p = 0 p T = - T F p 0 [Eq. 13]

Epipolar Constraint P p p O 1 O 2 [Eq. 13] p T F p = 0 F F = Fundamental Matrix (Faugeras and Luong, 1992) [ ] -1 = K - T T R K [Eq. 14]

Epipolar Constraint p T F p = 0 P p l l p e e O 1 O 2 l = F p is the epipolar line associated with p l = F T p is the epipolar line associated with p F e = 0 and F T e = 0 F is 3x3 matrix; 7 DOF F is singular (rank two)

Why F is useful? p p l = F T p - Suppose F is known - No additional information about the scene and camera is given - Given a point on left image, we can compute the corresponding epipolar line in the second imag

Why F is useful? F captures information about the epipolar geometry of 2 views + camera parameters MORE IMPORTANTLY: F gives constraints on how the scene changes under view point transformation (without reconstructing the scene!) Powerful tool in: 3D reconstruction Multi-view object/scene matching

The Eight-Point Algorithm Estimating F (Longuet-Higgins, 1981) (Hartley, 1995) P p p O 1 O 2 p T F p = 0

[Eq. 13] ( ) p T F p = Estimating F 0 æf11 F12 F13 öæu ö ç ç uv,,1 F F F v = 0 21 22 23 ç ç F31 F32 F ç 33 1 è øè ø ( uu uv u vu vv v u v ) Let s take 8 corresponding points! # p = # # " æf11 ö ç ç F12 ç F 13 ç F ç 21,,,,,,,,1 ç F = 0 u v 1 22 ç $ & & & % ç F23 ç ç F31 ç F 32 ç F è 33 ø " $ p! = $ $ # u! v! 1 % & [Eq. 14]

Estimating F

Estimating F! # # # # # # ( u i u i,u i v i,u i,v i u i,v i v i,v i,u i,v i,1) # # # # # # # " F 11 F 12 F 13 F 21 F 22 F 23 F 31 F 32 F 33 $ & & & & & & & = 0 [Eq. 14] & & & & & & %

W Homogeneous system Rank 8 If N>8 Estimating F æf11 ö æuu uv u vu vv v u v 1öç F 1 1 1 1 1 1 1 1 1 1 1 1 ç uu 2 2 uv 2 2 u2 vu 2 2 vv 2 2 v2 u 2 v 2 1 ç 12 ç ç F 13 ç uu 3 3 uv 3 3 u3 vu 3 3 vv 3 3 v3 u 3 v 3 1 ç ç ç F21 uu 4 4 uv 4 4 u4 vu 4 4 vv 4 4 v4 u 4 v 4 1 F ç 22 uu 5 5 uv 5 5 u5 vu 5 5 vv 5 5 v5 u 5 v 5 1 ç ç ç F23 ç uu 6 6 uv 6 6 u6 vu 6 6 vv 6 6 v6 u 6 v 6 1 ç ç F31 uu 7 7 uv 7 7 u7 v7u 7 v7v 7 v7 u 7 v 7 1 ç ç ç F 32 uu 8 8 uv 8 8 u8 vu 8 8 vv 8 8 v8 u 8 v 8 1 ç ç ç = 0 è øç F è 33 ø A non-zero solution exists (unique) Lsq. solution by SVD! f =1 W f = 0 f [Eqs. 15] Fˆ

Fˆ satisfies: ^ p T Fˆ p = and estimated F may have full rank (det(f) 0) But remember: fundamental matrix is Rank2 0 ^ Find F that minimizes F - Fˆ = 0 Frobenius norm (*) Subject to det(f)=0 SVD (again!) can be used to solve this problem (*) Sq. root of the sum of squares of all entries

Find F that minimizes F - Fˆ = 0 Frobenius norm (*) Subject to det(f)=0! # F =U# # "# s 1 0 0 0 s 2 0 0 0 0 $ & & & %& V T [HZ] pag 281, chapter 11, Computation of F Where:! # U# # "# s 1 0 0 0 s 2 0 0 0 s 3 $ & & V T = SVD( ˆF) & %&

Data courtesy of R. Mohr and B. Boufama.

Mean errors: 10.0pixel 9.1pixel

Problems with the 8-Point Algorithm W f = 0, Lsq solution by SVD f =1 F - Recall the structure of W: - do we see any potential (numerical) issue?

Problems with the 8-Point Algorithm Wf = 0 æf11 ö æuu uv u vu vv v u v 1öç F 1 1 1 1 1 1 1 1 1 1 1 1 ç uu 2 2 uv 2 2 u2 vu 2 2 vv 2 2 v2 u 2 v 2 1 ç 12 ç ç F 13 ç uu 3 3 uv 3 3 u3 vu 3 3 vv 3 3 v3 u 3 v 3 1 ç ç ç F21 uu 4 4 uv 4 4 u4 vu 4 4 vv 4 4 v4 u 4 v 4 1 F ç 22 uu 5 5 uv 5 5 u5 vu 5 5 vv 5 5 v5 u 5 v 5 1 ç ç ç F23 ç uu 6 6 uv 6 6 u6 vu 6 6 vv 6 6 v6 u 6 v 6 1 ç ç F31 uu 7 7 uv 7 7 u7 v7u 7 v7v 7 v7 u 7 v 7 1 ç ç ç F 32 uu 8 8 uv 8 8 u8 vu 8 8 vv 8 8 v8 u 8 v 8 1 ç ç ç = è øç F è 33 ø Highly un-balanced (not well conditioned) Values of W must have similar magnitude This creates problems during the SVD decomposition 0 HZ pag 108

Normalization IDEA: Transform image coordinates such that the matrix W becomes better conditioned (pre-conditioning) For each image, apply a transformation T (translation and scaling) acting on image coordinates such that: Origin = centroid of image points Mean square distance of the image points from origin is ~ 2 pixels

Example of normalization T Coordinate system of the image before applying T Coordinate system of the image after applying T Origin = centroid of image points Mean square distance of the image points from origin is ~ 2 pixels

Normalization q i = T p i q = T p i i

The Normalized Eight-Point Algorithm 0. Compute T and T for image 1 and 2, respectively 1. Normalize coordinates in images 1 and 2: q = T i p i 2. Use the eight-point algorithm to compute from the corresponding points q and q. i q = T p i i i ˆF q 1. Enforce the rank-2 constraint: 2. De-normalize F q : F F q such that: q T F q q = 0 T = T Fq T det( Fq ) = 0

With normalization Without normalization Mean errors: 10.0pixel 9.1pixel Mean errors: 1.0pixel 0.9pixel

The Fundamental Matrix Song http://danielwedge.com/fmatrix/

Next lecture: Stereo systems Silvio Savarese Lecture 5-24-Jan-18

Example: Parallel image planes P e v p p e y u z x O 1 O 2 Hint : K E=? 1 =K 2 = known R = I T = (T, 0, 0) x parallel to O 1 O 2

Essential matrix for parallel images E = [ T ] R " $ E = $ $ # $ 0 T z T y T z 0 T x T y T x 0 T = [ T 0 0 ] R = I % " $ R = $ $ & # 0 0 0 0 0 T 0 T 0 % & [Eq. 20]

Example: Parallel image planes P v v e l p p l e y u u z O 1 What are the directions of epipolar lines? x " $ l = E p = $ $ # 0 0 0 0 0 T 0 T 0 %" $ $ $ &# u v 1 % " $ = $ $ & # 0 T T v % & horizontal!

Example: Parallel image planes P v v e p p e y u u z O 1 x How are p and p related? p T E p = 0

Example: Parallel image planes P v v e p p e y u u z O 1 x How are p and p related? é0 0 0 ùæu ö æ 0 ö ( u v ê ) T úç ç = Þ v ( u v ) T Tv Tv ê - ú = Þ - = Þ = ê 0 T 0 úç 1 ç Tv ë ûè ø è ø v = v" T pep 0 1 0 0 0 1 0

Example: Parallel image planes P v v e p p e y u u z O 1 x Rectification: making two images parallel Why it is useful? Epipolar constraint v = v New views can be synthesized by linear interpolation

Application: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30

Rectification P H 1 I 1 I s î 1 v u I 2 î S v u C 1 v u î 2 H 2 C s C 2

From its reflection!

Deep view morphing D. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017

Deep view morphing D. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017

Deep view morphing D. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017