Transformations Between Two Images. Translation Rotation Rigid Similarity (scaled rotation) Affine Projective Pseudo Perspective Bi linear

Transcription

1 Transformations etween Two Images Translation Rotation Rigid Similarit (scaled rotation) ffine Projective Pseudo Perspective i linear

2 Fundamental Matri Lecture 13

3 pplications Stereo Structure from Motion View Invariant ction Recognition..

4 Stereo Pairs and Depth Maps (from Szeliski s book)

5 Image Rectification For Stereo

6 Photosnth: Structure From Motion

7 Fundamental Matri Longuet Higgins (1981) Hartle (1992) Faugeras (1992) Zhang (1995)

8 Fundamental Matri Song

9

10 Fundamental Matri Song

11 Preliminaries Linear Independence Rank of a Matri Matri Norm Singular Value Decomposition Vector Cross product to Matri Multiplication RNSC

12 Linearl Independence finite subset of n vectors, v 1, v 2,..., v n, from the vector space V, is linearl dependent if and onl if there eists a set of n scalars, a 1, a 2,..., a n, not all zero, such that

13 Rank of a Matri The column rank of a matri is the maimum number of linearl independent column vectors of. The row rank of a matri is the maimum number of linearl independent row vectors of. The column rank of is the dimension of the column space of The row rank of is the dimension of the row space of. Copright Mubarak Shah 2003

14 Eample (Row Echelon) Rank is 2

15 Singular Value Decomposition (SVD) Theorem: n m b n matri, for which,can be written as is diagonal mn mn nn nn are orthogonal Copright Mubarak Shah 2003

16 Singular Value Decomposition (SVD) If is square, then are all square. nn nn nn nn Copright Mubarak Shah 2003

17 Matri Norm L1 matri norm is maimum of absolute column sum. L infinit norm is maimum of sum of absolute of row sum.

18 Vector Cross Product to Matrivector multiplication z z z z z z z S z z z z z z k j i

19 How to Fit Line? Least squares Fit (over constraint) RNSC (constraint) Hough Transform (under constraint) lper Yilmaz, Mubarak Shah, Fall 2011 UCF

20 RNSC Song

21 How to Fit Line? m c lper Yilmaz, Mubarak Shah, Fall 2011 UCF

22 Least Squares Fit Standard linear solution to estimating unknowns. If we know which points belong to which line Or if there is onl one line m c f, m, Minimize E c i i f, m, c Take derivative wrt m and c set to 0 i 2 lper Yilmaz, Mubarak Shah, Fall 2011 UCF

23 Line Fitting c m c m c m c m n n D n n c m D D D D T T T T T T T T lper Yilmaz, Mubarak Shah, Fall 2011 UCF

24 RNSC: Random Sampling and Consensus lper Yilmaz, Mubarak Shah, Fall 2011 UCF

25 RNSC Song E&feature=relmfu

26 RNSC: Random Sampling and Consensus 1. Randoml select two points to fit a line 2. Find the error between the estimated solution an all other points. If the error is less than tolerance, then quit, else go to step (1). lper Yilmaz, Mubarak Shah, Fall 2011 UCF

27 Derivation of Fundamental Matri

28 Epipolar Geometr Defined for two static cameras Left camera 3D World Right camera Left camera image plane Right camera image plane

29 Epipolar Geometr C l l e l T P e r r C r Epipolar line: set of world points that project to the same point in left image, when projected to right image forms a line. Epipole: intersection of image plane with line connecting camera centers. Image of a left camera center in the right, and vice versa. Epipolar plane: plane defined b the camera centers and world point.

30 Essential Matri P l P P r l r C l e l T e r C r Coplanarit constraint between vectors (P l -T), T, P l. Pl T T P l 0 P RP T r l P r RT P l 0 R P T T r P r R P P l l T T T

31 Vector Cross Product to Matrivector multiplication z z z z z z z S z z z z z z k j i

32 Essential Matri P l P P r l r C l e l T e r C r P r RT P l 0 P r R S P l 0 P r ΕP l 0 essential matri 0 Tz T T 0 T z T T 0 E R S

33 Fundamental Matri P l P P r l r C l e l T e r C r ppl Camera model M l M T r 1 1 r M l r T r P P l r P T r l r M P P r ΕP l l r l M P r 0 r r M 1 M EM 0 r r F r l EM l 1 l 0 fundamental matri l l 0

34 Fundamental Matri P l P P r l r C l e l T e r C r relative rotation P r P P l r R P T l R P R P T r l T l r R R r Rl T T l R T r () () l r M P P r ΕP l l r l M P r 0 r 1 M EM 0 r F r l l 0 fundamental matri l relative translation

35 Fundamental Matri Given a point in left camera, epipolar line in right camera is: u r =F 0 ' ' m h g f e d c b a F

36 Fundamental Matri 33 matri with 9 components Rank 2 matri (due to S) 7 degrees of freedom Given a point in left camera, epipolar line in right camera is: u r =F 0 r l r l m h g f e d c b a F

37 Fundamental Matri Longuet Higgins (1981) Hartle (1992) Faugeras (1992) Zhang (1995)

38 Fundamental Matri Fundamental matri captures the relationship between the corresponding points in two views.

39 Fundamental Matri One equation for one point correspondence To solve the equation, the rank(m) must be 8.

40 Computation of Fundamental Matri

41 Normalized 8-point algorithm (Hartle) Objective: Compute fundamental matri F such that lgorithm Normalize the image Find centroid of points in each image, determine the range, and normalize all points between 0 and 1 Linear solution determining the eigen vector corresponding to the smallest eigen value of,

42 Normalized 8-point algorithm Construct (Hartle) Normalize L1 matri norm is maimum of absolute column sum. Constraint enforcement SVD decomposition L infinit norm is maimum of sum of absolute of row sum. Rank enforcement De-normalization:

43 Robust Fundamental Matri Estimation (b Zhang) Uniforml divide the image into 8 8 grid. Randoml select 8 grid cells and pick one pair of corresponding points from each grid cell, then use Hartle s 8-point algorithm to compute Fundamental Matri F i. For each F i, compute the median of the squared residuals R i. R i = median k [d(p 1k,F i p 2k ) + d(p 2k,F i p 1k )] Select the best F i according to R i. Determine outliers if R k >Th. Using the remaining points compute the fundamental Matri F b weighted least square method.

44 Epi-polar Lines

45 Epi-polar lines

46

47 Stereo

48 Stereo Pairs and Depth Maps (from Szeliski s book)

49 Image Rectification For Stereo

50 Correlation ased Stereo Methods Once disparit is available compute depth using Z f d lper Yilmaz, Mubarak Shah, Fall UCF

51 Correspondence using Search

52 Correlation ased Stereo Methods Depth is computed onl at tokens and interpolated/etrapolated to remaining piel Disparit map is constructed based on a correlation measure SSD I left I D I left I CC t I 1 I t right right 2 NC MC 1 64 left lper Yilmaz, Mubarak shah, Fall 2011 UCF I left I right left. I. I right right Ileft right I left right

53 lper Yilmaz, Mubarak Shah, Fall 2011 UCF arnard s Stereo Method Similar intensit Similar to brightness constraint Smoothness of disparit ), ( ) ),, ( ( ), ( E i j right left D j D i I j i I ), ( ), ( ), ( i j D j i D D

54 arnard s Stereo Method Energ can be minimized using brute force search Let ma allowed disparit is 10 piels For image for 10 possible levels of disparit There possible disparit values We can select an minimization technique arnard choose simulated annealing lper Yilmaz, Mubarak Shah, Fall 2011 UCF

55 Simulated nnealing Select a random state S (disparities) Select a high temperature Select random S Compute E=E(S )-E(S) If (E<0) SS Else Pep(-E/T) Xrandom(0,1) If X<P then SS If no decrease in several iterations lower T lper Yilmaz, Mubarak Shah, Fall 2011 UCF

56 Eamples bread to apple lper Yilmaz, Mubarak Shah, Fall 2011 UCF

57 Eamples Left Image Right Image Depth Map lper Yilmaz, Fall 2004 UCF

58 Stereo results Data from Universit of Tsukuba Similar results on other images without ground truth Scene Ground truth

59 Results with window correlation Window-based matching (best window size) Ground truth

60 Results with better method State of the art method okov et al., Fast pproimate Energ Minimization via Graph Cuts, International Conference on Computer Vision, September Ground truth

61 pplications of Stereo (from Szeliski s book)

62 Reading Material Fundamental of Computer Vision 6.2.1, and Computer Vision: lgorithms and pplications, Richard Szeliski Chapter 11