Undergrad HTAs / TAs. Help me make the course better! HTA deadline today (! sorry) TA deadline March 21 st, opens March 15th

Transcription

1

2 Undergrad HTAs / TAs Help me make the course better! HTA deadline today (! sorry) TA deadline March 2 st, opens March 5th

3 Project 2 Well done. Open ended parts, lots of opportunity for mistakes. Real implementation experience of a tricky vision system.

4 Episcopal Gaudi the haunted palace

5 Harder to mark Part 2 is somewhat open ended. Many of you came up with different solutions. -> We may have a few issues in the marking. Let us know if you think we ve made an error.

6 MATLAB tip - thresholding No need to iterate. img = im2double( imread( a.jpg ) ); imgt = img. double(img >.5);

7 Average Accuracy Across Notre Dame, Mt. Rushmore, and Gaudi s Episcopal Palace. 76% - Katya Schwiegershausen 2. 72% -Prasetya Utama % - Jessica Fu % Tiffany Chen 5. Gaudi s choice award: 34% - Spencer Boyum (st in Episcopal Palace)

8 Outline Recap camera calibration Epipolar Geometry

9 Oriented and Translated Camera R j w t k w O w i w

10 Degrees of freedom t x K R z y x t r r r t r r r t r r r v u s v u w z y x 5 6

11 How to calibrate the camera? Z Y s sv su t x K R

12 How do we calibrate a camera? su sv s Y Z

13 m m m m m m m m m m m m v Z v Y v v Z Y u Z u Y u u Z Y v Z v v Y v Z Y u Z u u Y u Z Y n n n n n n n n n n n n n n n n n n n n Method homogeneous linear system Solve for m s entries using linear least squares Ax= form Z Y m m m m m m m m m m m m s sv su [U, S, V] = svd(a); M = V(:,end); M = reshape(m,[],3)';

14 For project 3, we want the camera center

15 Estimate of camera center

16 Oriented and Translated Camera R j w t k w O w i w

17 Recovering the camera center Z Y s sv su t x K R z y x t r r r t r r r t r r r v u s v u w z y x This is not the camera center C. It is RC (because a point will be rotated before t x, t y, and t z are added) This is t K Q So K - m 4 is t So we need -R - K - m 4 to get C Q is K R. So we just need -Q - m 4

18 Estimate of camera center

19 Epipolar Geometry and Stereo Vision Many slides adapted from James Hays, Derek Hoiem, Lana Lazebnik, Silvio Saverese, Steve Seitz, Hartley & Zisserman

20 Depth from disparity image I(x,y) Disparity map D(x,y) image I (x,y ) (x,y )=(x+d(x,y), y) If we could find the corresponding points in two images, we could estimate relative depth James Hays

21 What do we need to know?. Calibration for the two cameras.. Camera projection matrix 2. Correspondence for every pixel. Like project 2, but project 2 is sparse. We need dense correspondence!

22 2. Correspondence for every pixel. Where do we need to search?

23 Depth from Stereo Goal: recover depth by finding image coordinate x that corresponds to x x x x z x' f f C Baseline C B

24 Depth from Stereo Goal: recover depth by finding image coordinate x that corresponds to x Sub-Problems. Calibration: How do we recover the relation of the cameras (if not already known)? 2. Correspondence: How do we search for the matching point x? x x'

25 Epipolar geometry Relates cameras from two positions

26 Wouldn t it be nice to know where matches can live? To constrain our 2d search to d?

27 Key idea: Epipolar constraint x x x x Potential matches for x have to lie on the corresponding line l. Potential matches for x have to lie on the corresponding line l.

28 VLFeat s 8 most confident matches among,+ local features.

29 Epipolar lines

30 Keep only the matches at are inliers with respect to the best fundamental matrix

31 Epipolar geometry: notation x x Baseline line connecting the two camera centers Epipoles = intersections of baseline with image planes = projections of the other camera center Epipolar Plane plane containing baseline (D family)

32 Epipolar geometry: notation x x Baseline line connecting the two camera centers Epipoles = intersections of baseline with image planes = projections of the other camera center Epipolar Plane plane containing baseline (D family) Epipolar Lines - intersections of epipolar plane with image planes (always come in corresponding pairs)

33 Think Pair Share Where are the epipoles? What do the epipolar lines look like? = camera center a) b) c) d)

34 Example: Converging cameras

35 Example: Motion parallel to image plane

36 Example: Forward motion e e Epipole has same coordinates in both images. Points move along lines radiating from e: Focus of expansion

37 What is this useful for? x x Find : If I know x, and have calibrated cameras (known intrinsics K,K and extrinsic relationship), I can restrict x to be along l. Discover disparity for stereo.

38 What is this useful for? x x Given candidate x, x correspondences, estimate relative position and orientation between the cameras and the 3D position of corresponding image points.

39 What is this useful for? x x Model fitting: see if candidate x, x correspondences fit estimated projection models of cameras and 2.

40 VLFeat s 8 most confident matches among,+ local features.

41 Epipolar lines

42 Keep only the matches at are inliers with respect to the best fundamental matrix

43 Epipolar constraint: Calibrated case x x x x xˆ K x Homogeneous 2d point (3D ray towards ) 2D pixel coordinate (homogeneous) 3D scene point xˆ K x 3D scene point in 2 nd camera s 3D coordinates

44 Epipolar constraint: Calibrated case x x x x t xˆ K Homogeneous 2d point (3D ray towards ) 2D pixel coordinate (homogeneous) x xˆ [ t 3D scene point ( Rxˆ )] (because x, R x, and t are co-planar) xˆ K x 3D scene point in 2 nd camera s 3D coordinates

45 Essential matrix x x xˆ [ t ( Rxˆ )] ˆ x T E xˆ with E t R E is a 3x3 matrix which relates corresponding pairs of normalized homogeneous image points across pairs of images for K calibrated cameras. Essential Matrix (Longuet-Higgins, 98) Estimates relative position/orientation. Note: [t] is matrix representation of cross product

46 Epipolar constraint: Uncalibrated case x x If we don t know K and K, then we can write the epipolar constraint in terms of unknown normalized coordinates: ˆ x T Ex ˆ x K xˆ, x Kx ˆ

47 The Fundamental Matrix Without knowing K and K, we can define a similar relation using unknown normalized coordinates xˆ T Ex ˆ T T xˆ K xˆ K x x x Fx with F K EK Fundamental Matrix (Faugeras and Luong, 992)

48 Properties of the Fundamental matrix x x x T Fx with F K EK T F x = is the epipolar line l associated with x F T x = is the epipolar line l associated with x F is singular (rank two): det(f)= F e = and F T e = (nullspaces of F = e ; nullspace of F T = e ) F has seven degrees of freedom: 9 entries but defined up to scale, det(f)=

49 F in more detail F is a 3x3 matrix Rank 2 -> projection; one column is a linear combination of the other two. Determined up to scale. 7 degrees of freedom a b αa + βb c d αc + βd e f αe + βf where a is scalar; e.g., can normalize out. Given x projected from into image, F constrains the projection of x into image 2 to an epipolar line.

50 Estimating the Fundamental Matrix 8-point algorithm Least squares solution using SVD on equations from 8 pairs of correspondences Enforce det(f)= constraint using SVD on F Note: estimation of F (or E) is degenerate for a planar scene.

51 8-point algorithm. Solve a system of homogeneous linear equations a. Write down the system of equations x T Fx uu f + uv f 2 + uf 3 + vu f 2 + vv f 22 + vf 23 + u f 3 + v f 32 + f 33 = Af = u u u v u v u v v v u v u n u v u n v n u n v n u n v n v n v n u n v n f f 2 f 3 f 2 f 33 =

52 8-point algorithm. Solve a system of homogeneous linear equations a. Write down the system of equations b. Solve f from Af= using SVD Matlab: [U, S, V] = svd(a); f = V(:, end); F = reshape(f, [3 3]) ;

53 Need to enforce singularity constraint

54 8-point algorithm. Solve a system of homogeneous linear equations a. Write down the system of equations b. Solve f from Af= using SVD Matlab: [U, S, V] = svd(a); f = V(:, end); F = reshape(f, [3 3]) ; 2. Resolve det(f) = constraint using SVD Matlab: [U, S, V] = svd(f); S(3,3) = ; F = USV ;

55 8-point algorithm. Solve a system of homogeneous linear equations a. Write down the system of equations b. Solve f from Af= using SVD 2. Resolve det(f) = constraint by SVD Notes: Use RANSAC to deal with outliers (sample 8 points) How to test for outliers?

56 Problem with eight-point algorithm f f f f f f f 2 uu uv u vu vv v u v f

57 Problem with eight-point algorithm f f f f f f f 2 uu uv u vu vv v u v Poor numerical conditioning Can be fixed by rescaling the data f

58 The normalized eight-point algorithm (Hartley, 995) Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels Use the eight-point algorithm to compute F from the normalized points Enforce the rank-2 constraint (for example, take SVD of F and throw out the smallest singular value) Transform fundamental matrix back to original units: if T and T are the normalizing transformations in the two images, than the fundamental matrix in original coordinates is T T F T

59 VLFeat s 8 most confident matches among,+ local features.

60 Epipolar lines

61 Keep only the matches at are inliers with respect to the best fundamental matrix

62 Comparison of estimation algorithms 8-point Normalized 8-point Nonlinear least squares Av. Dist pixels.92 pixel.86 pixel Av. Dist pixels.85 pixel.8 pixel

63 Let s recap Fundamental matrix song