Dr. Ulas Bagci

Transcription

1 CAP5415-Computer Vision Lecture 17-Fundamental Matrix Dr. Ulas Bagci 1

2 Reminders PA#4 is due on 22 nd October (extended to next Monday) PA#5 is due on 5 th of November (no extension). Mini-project #1 is on due 23 rd of November (no extension) Mini-project #2 is on due 10 th of December (no extension) Reading Materials Shah, M., Chapters 6.2.1, 6.2.4, and Szeliski, R., Chapter 11. 2

3 DefiniWon & ApplicaWons DefiniWon: The fundamental matrix is a relawonship between any two images of the same scene that constrains where the projecwon of points from the scene can occur in both images. Briefly, fundamental matrix relates corresponding points in stereo images. Stereo Structure from mowon View invariant acwon recogniwon 3

4 Two view-general ConfiguraWon Motion between the two views is not known Given two views of the scene recover the unknown camera displacement and 3D scene structure 4

5 Recap from Dr. Shah s Lecture 3D points Image points Perspective Projection Rigid Body Motion Rigid Body Motion + Persp. projection 5

6 Stereo-Photometric Constraint Same world point has same intensity in both images. Lambertian fronto-parallel Issues (noise, specularities, foreshortening) Difficulties ambiguities, large changes of appearance, due to change Of viewpoint, non-uniquess 6

7 Stereo Matching What if? For each scanline, for each pixel in the left image Compare with every pixel on same epipolar line in right image? pick pixel with minimum match cost This will never work, so: improvement match windows 7

8 Comparing Windows? = f g For each window, match to closest window on epipolar line in other image. Most popular 8

9 Example: stereo pair (translawon) 9

10 Preliminary (Matrix Algebra) a. Rank of a matrix b. Linear independence c. Matrix norm d. SVD (singular value decomposiwon) e. Vector/Cross product, Matrix mulwplicawon f. RANSAC 10

11 a. Rank of a matrix Rank of a matrix is the dimension of the vector space generated (or spanned) by its columns It is a measure of nondegenerateness 11

12 A = Example (rank of a matrix)

13 A = Example (rank of a matrix)

14 A = Example (rank of a matrix)

15 A = Example (rank of a matrix) Basic columns Row echelon form Rank (A) = 2 15

16 b. Linear independence A set of vectors S = {v 1,...,v n } is said to be linearly independent set whenever the only soluwon for the scalars i in the homogeneous equawon 1 v v n v n =0 is the trivial soluwon 1 = 2 =... = n =0 Nontrivial soln: at least one vector is a combinawon of other vectors. 16

17 Example (linear independence)

18 Example (linear independence)

19 Example (linear independence) =0 19

20 Example (linear independence) Row echelon form Rank = = Non-trivial soluwon exists! 20

21 c. Matrix Norm 21

22 d. SVD Singular values can tell you something about geometry of linear transformawons because singular values indicate directly how much distorwon occurs under the transformawon A. (How matrix A distort the unit circle) Many applica@ons: pseudo-inverse, least square soluwon, low-rank approximawon, separable models, PCA,.. 22

23 e. Vector Cross Product Given two linearly independent vectors u and v, the cross product, u v, is a vector that is perpendicular to both and therefore normal to the plane containing them. It has many applicawons in mathemawcs, physics, engineering, and computer programming. 23

24 f. RANSAC RANSAC -> random sample consensus 24

25 f. RANSAC RANSAC -> random sample consensus an iterawve method to eswmate parameters of a mathemawcal model from a set of observed data which contains outliers. 25

26 f. RANSAC RANSAC -> random sample consensus an iterawve method to eswmate parameters of a mathemawcal model from a set of observed data which contains outliers. It is a non-determiniswc algorithm in the sense that it produces a reasonable result only with a certain probability, with this probability increasing as more iterawons are allowed. 26

27 f. RANSAC RANSAC -> random sample consensus an iterawve method to eswmate parameters of a mathemawcal model from a set of observed data which contains outliers. It is a non-determiniswc algorithm in the sense that it produces a reasonable result only with a certain probability, with this probability increasing as more iterawons are allowed. Step 1. Randomly select two points to fit a line, Step 2. Find the error between eswmated soluwon on all other points, if error >a tolerance, repeat step 1. 27

28 RANSAC Example: Finng a line hops:// 1. Select a random subset of the original data. Call this subset the hypothewcal inliers. 2. A model is fioed to the set of hypothewcal inliers. 3. All other data are then tested against the fioed model. Those points that fit the eswmated model well, according to some model-specific loss funcwon, are considered as part of the consensus set. 4. The eswmated model is reasonably good if sufficiently many points have been classified as part of the consensus set. 5. Auerwards, the model may be improved by reeswmawng it using all members of the consensus set. 28

29 Lecture 17: Fundamental Matrix RANSAC Example: Finng a line 29

30 DerivaWon of Fundamental Matrix Epipolar geometry Defined for two stawc cameras 30

31 Epipolar geometry C,C,x,x, and X are coplanar. 31

32 Converging cameras 32

33 MoWon parallel with image plane 33

34 Epipolar Geometry 34

35 Epipolar Geometry (geometry of stereo vision) Three non-collinear points in the 3D define a plane. The points O 1, O 2, and the unknown point P define the epipolar plane. 35

36 EssenWal Matrix 36

37 EssenWal Matrix 37

38 Fundamental Matrix 38

39 Another look x' = H π x l' e' x' e' H π x = = [ ] Fx = mapping from 2-D to 1-D family (rank 2) 39

40 Fundamental Matrix 40

41 Fundamental Matrix 41

42 Fundamental Matrix Captures the relawonship between corresponding points in two views 42

43 Fundamental Matrix 43

44 ComputaWon of Fundamental Matrix (Hartley, normalized 8-point alg.) 44

45 ComputaWon of Fundamental Matrix (Hartley, normalized 8-point alg.) 3X i=1 3X p ri F ij p lj =0 j=1 45

46 ComputaWon of Fundamental Matrix i=1 j=1 (Hartley, normalized 8-point alg.) 3X 3X p ri F ij p lj =0 q T f = r = 9X q i f i =0 i=1 46

47 ComputaWon of Fundamental Matrix i=1 j=1 (Hartley, normalized 8-point alg.) 3X 3X p ri F ij p lj =0 q T f = r = 9X q i f i =0 i=1 q =[p l1 p r1,p l2 p r1,p r1,p l1 p r2,p l2 p r2,p r2,p l1,p l2, 1] T 47

48 ComputaWon of Fundamental Matrix i=1 j=1 (Hartley, normalized 8-point alg.) 3X 3X p ri F ij p lj =0 q T f = r = 9X q i f i =0 i=1 q =[p l1 p r1,p l2 p r1,p r1,p l1 p r2,p l2 p r2,p r2,p l1,p l2, 1] T f =[F 11,F 12,F 13,F 21,F 22,F 23,F 31,F 32,F 33 ] T 48

49 ComputaWon of Fundamental Matrix (Hartley, normalized 8-point alg.) To find F, we need to use least-square fashion as follows, min f =1 Qf 2 49

50 ComputaWon of Fundamental Matrix (Hartley, normalized 8-point alg.) To find F, we need to use least-square fashion as follows, min f =1 Qf 2 Q is Kx9 matrix, Qf 2 =(Qf) T (Qf) =f T (Q T Q)f 50

51 ComputaWon of Fundamental Matrix (Hartley, normalized 8-point alg.) To find F, we need to use least-square fashion as follows, min f =1 Qf 2 Q is Kx9 matrix, Qf 2 =(Qf) T (Qf) =f T (Q T Q)f M = Q T Q 51

52 ComputaWon of Fundamental Matrix (Hartley, normalized 8-point alg.) To find F, we need to use least-square fashion as follows, min f =1 Qf 2 Q is Kx9 matrix, Qf 2 =(Qf) T (Qf) =f T (Q T Q)f M = Q T Q Now, the system can be solved via SVD (i.e., eigenvalue distribuwon), M is 9x9 matrix. The matrix F is found as an eigenvector w of M which corresponds to the lowest eigenvalue of M. 52

53 ComputaWon of Fundamental Matrix (Hartley, normalized 8-point alg.) 53

54 ProperWes of Fundamental Matrix F (i) Transpose: if F is fundamental matrix for (P,P ), then F T is fundamental matrix for (P,P) (ii) Epipolar lines: l =Fx & l=f T x (iii) Epipoles: on all epipolar lines, thus e T Fx=0, for all x àe T F=0, similarly Fe=0 (iv) F has 7 d.o.f., i.e. 3x3-1(homogeneous)-1(rank2) (v) F is a correlation, projective mapping from a point x to a line l =Fx (not a proper correlation, i.e. not invertible) 54

55 Fundamental Matrix of Pure TranslaWon 55

56 Epipolar Lines 56

57 Epipolar Lines 57

58 Lecture 17: Fundamental Matrix EsWmaWon of F and comparison Classical stereo calibrawon (red) and nonlinear method (green) 58

59 CompuWng F in PracWce 59

60 References and Slice Credits Shah, M., Chapters 6.2.1, 6.2.4, and Szeliski, R., Chapter 11. Wikipedia, Epipolar Geometry Marc Pollefeys, Slices. MulWple View Geometry in Computer Vision, Hartley and Zisserman, Cambridge Press. E.R Davies, Computer and Machine Vision, AP Press. 60