Computer Vision I - Robust Geometry Estimation from two Cameras

Transcription

1 Computer Vision I - Robust Geometry Estimation from two Cameras Carsten Rother 16/01/2015 Computer Vision I: Image Formation Process

2 FYI Computer Vision I: Image Formation Process 16/01/ Microsoft Research Cambridge excellent 8 week paid summer internship Deadline: 16 th January 2015 If interested, please contact me The last lecture on will mainly be a Q&A session. Please look through all lectures again and prepare questions!

3 Computer Vision I: Image Formation Process 16/01/ Roadmap for next five lecture Appearance based matching (sec. 4.1) How do we get an RGB image? (sec ) The Human eye The Camera and Image formation model Projective Geometry - Basics (sec ) Geometry of a single camera (sec 2.1.5, 2.1.6) Pinhole camera Lens effects Geometry of two cameras (sec. 7.2) Robust Geometry estimation for two cameras (sec ) Multi-View 3D reconstruction (sec )

4 Camera parameters - Summary Computer Vision I: Image Formation Process 16/01/ Camera matrix P has 11 DoF: Image x = P X 3D world point x = K R (I 3 3 C) X f s p x K = 0 mf p y f Intrinsic parameters Focal length f Pixel y-direction magnification factors m Skew (non-rectangular pixels) s Principal point coordinates (p x, p y ) Extrinsic parameters Rotation R (3DoF) and translation C (3DoF) relative to world coordinate system

5 Reminder: The upcoming tasks Computer Vision I: Two-View Geometry 16/01/ Two-view transformations we look at: Homography H: between two views Camera matrix P (mapping from 3D to 2D) Fundamental matrix F between two un-calibrated views Essential matrix E between two calibrated views Derive geometrically: H, P, F, E, i.e. what do they mean? Calibration: Take primitives (points, lines, planes, cones, ) to compute H, P, F, E : What is the minimal number of points to compute them (very important for next lecture on robust methods) If we have many points with noise: what is the best way to computer them: algebraic error versus geometric error? Can we derive the intrinsic (K) an extrinsic (R, C) parameters from H, P, F, E? What can we do with H, P, F, E? (e.g. Panoramic Stitching)

6 Homography H: Summary Computer Vision I: Two-View Geometry 16/01/ Derive geometrically H Calibration: Take measurements (points) to compute H Minimum of 4 points. Solution: right null space of Ah = 0 Many points. Use SVD to solve h = argmin h Ah Can we derive the intrinsic (K) an extrinsic (R, C) parameters from H? -> hard. Not discussed much What can we do with H? -> augmented reality on planes, panoramic stitching

7 Camera Matrix P: Summary Computer Vision I: Two-View Geometry 16/01/ Derive geometrically P x = P X x = K R (I 3 3 C) X Calibration: Take measurements (points) to compute P 6 or more points. Use SVD to solve p = argmin h Ap Can we derive the intrinsic (K) an extrinsic (R, C) parameters from H? -> yes, use SVD and RQ decomposition What can we do with P? -> very many things (robotic, photogrammetry, augmented reality, )

8 Topic 3: Fundamental/Essential Matrix F/E Computer Vision I: Two-View Geometry 16/01/ Derive geometrically F/E Calibration: Take measurements (points) to compute F/E How do we do that with a minimal number of points? How do we do that with many points? Can we derive the intrinsic (K) an extrinsic (R, C) parameters from F/E? What can we do with F/E?

9 Reminder: Matching two Images Computer Vision I: Image Formation Process 16/01/ Find interest points Find orientated patches around interest points to capture appearance Encode patch appearance in a descriptor Find matching patches according to appearance (similar descriptors) Verify matching patches according to geometry (later lecture) v We will discover in next slides: Seven 3D points define how other 3D points match between 2 views!

10 Reminder: 3D Geometry Computer Vision I: Two-View Geometry 16/01/ Non-moving scene Rigidly (6D) moving scene P Both cases are equivalent for the following derivations

11 Reminder: Epipolar Geometry Computer Vision I: Two-View Geometry 16/01/ Epipole: Image location of the optical center of the other camera. (Can be outside of the visible area)

12 Reminder: Epipolar Geometry Computer Vision I: Two-View Geometry 16/01/ Epipolar plane: Plane through both camera centers and world point.

13 Reminder: Epipolar Geometry Computer Vision I: Two-View Geometry 16/01/ Epipolar line: Constrains the location where a particular point (here p 1 ) from one view can be found in the other.

14 Reminder: Epipolar Geometry Computer Vision I: Two-View Geometry 16/01/ Epipolar lines: Intersect at the epipoles In general not parallel

15 Reminder: Example: Converging Cameras Computer Vision I: Two-View Geometry 16/01/

16 Reminder: Example: Motion Parallel to Camera Computer Vision I: Two-View Geometry 16/01/ We will use this idea when it comes to stereo matching

17 Reminder: Example: Forward Motion Computer Vision I: Two-View Geometry 16/01/ Epipoles have same coordinate in both images Points move along lines radiating from epipole focus of expansion

18 The maths behind it: Fundamental/Essential Matrix Computer Vision I: Robst Two-View Geometry 16/01/ derivation on black board X Camera 0 Camera 1 (R, T)

19 Computer Vision I: Two-View Geometry 16/01/ The maths behind it: Fundamental/Essential Matrix The 3 vectors are in same plane (co-planar): 1) T = C 1 C 0 2) X C 0 3) X C 1 X Set camera matrix: x 0 = K 0 I 0 X and x 1 = K 1 R 1 I C 1 The three vectors can be re-writting using x 0,1, K: 1) T 2) K 1 0 x 0 3) RK 1 1 x 1 + C 1 C 1 = RK 1 1 x 1 We know that: K 1 T 0 x 0 T R(K 1 1 x 1 ) = 0 which gives: x T 0 K T 0 T R(K 1 1 x 1 ) = 0 X

20 The maths behind it: Fundamental/Essential Matrix Computer Vision I: Two-View Geometry 16/01/ In an un-calibrated setting (K s not known): x 0 T K 0 T T R(K 1 1 x 1 ) = 0 In short: x 0 T Fx 1 = 0 where F is called the Fundamental Matrix (discovered by Faugeras and Luong 1992, Hartley 1992) In an calibrated setting (K s are known): we use rays: x i = K 1 i x i then we get: x T 0 T Rx 1 = 0 In short: x T 0 Ex 1 = 0 where E is called the Essential Matrix (discovered by Longuet-Higgins 1981)

21 Fundamental Matrix: Properties Computer Vision I: Two-View Geometry 16/01/ We have x 0 T Fx 1 = 0 where F is called the Fundamental Matrix It is det F = 0. Hence F has 7 DoF Proof: F = K T 1 0 T RK 1 F has Rank 2 since T has Rank 2 x = 0 x 3 x 2 x 3 0 x 1 x 2 x 1 0 Check: det( x ) = x 3 x 3 0 x 1 x 2 + x 2 x 1 x 3 + x 2 0 = 0

22 Fundamental Matrix: Properties Computer Vision I: Two-View Geometry 16/01/ For any two matching points (i.e. they have the same 3D point) we have: x 0 T Fx 1 = 0 Compute epipolar line in camera 1 of a point x 0 : l 1 T = x 0 T F (since l 1 T x 1 = x 0 T Fx 1 = 0) Compute epipolar line in camera 0 of a point x 1 : l 0 = Fx 1 (since x 0 T l 0 = x 0 T Fx 1 = 0) Camera 0 Camera 1

23 Fundamental Matrix: Properties Computer Vision I: Robust Two-View Geometry 16/01/ For any two matching points (i.e. have the same 3D point) we have: x 0 T Fx 1 = 0 Epipole e 0 is the left nullspace of F (can be computed with SVD) It is: e 0 T Fx i = 0 for all points x i (since all lines l 0 = Fx i go through e 0 T ) This means: e 0 T F = 0 Epipole e 1 is right null space of F (Fe 1 = 0) X Camera 0 (R, T) Camera 1

24 How can we compute F (2-view calibration)? Computer Vision I: Two-View Geometry 16/01/ Each pair of matching points gives one linear constraint x T Fx = 0 in F. For x, x we get: x 1 x 1 x 1 x 2 x 1 x 3 x 2 x 1 x 2 x 2 x 2 x 3 x 3 x 1 x 3 x 2 x 3 x 3... f 11 f 12 f 13 f 21 f 22 f 23 f 31 f 32 f 33 = 0 Given m 8 matching points (x, x) we can compute the F in a simple way.

25 How can we compute F (2-view calibration)? Computer Vision I: Two-View Geometry 16/01/ Method (normalized 8-point algorithm): 1) Take m 8 points 2) Compute T, and condition points: x = Tx; x = T x 3) Assemble A with Af = 0, here A is of size m 9, and f vectorized F 4) Compute f = argmin f Af subject to f = 1. Use SVD to do this. 5) Get F of unconditioned points: T T FT (note: (Tx) T F T x = 0) 4) Make rank F = 2 [See HZ page 282]

26 How to make F Rank 2 Computer Vision I: Two-View Geometry 16/01/ (Again) use SVD: Set last singular value σ p 1 to 0 then A has Rank p 1 and not p (assuming A has originally full Rank) Proof: diagonal matrix has Rank p 1 hence A has Rank p 1

27 Can we compute F with just 7 points? Computer Vision I: Robust Two-View Geometry 16/01/ Method (7-point algorithm): 1) Take m = 7 points 2) Assemble A with Af = 0, here A is of size 7 9, and f vectorized F 3) Compute 2D right null space: F 1 and F 2 from last two rows in V T (use the SVD decomposition: A = UDV T ) 4) Choose: F = αf α F 2 (see comments next slide) 5) Determine α s (either 1 or 3 solutions for α) (see comments next slide) by using the constraint: det(αf α F 2 ) = 0. (This is a cubic polynomial equation for α which has one or three real-value solutions for α) Note an 8 th point would determine which of these 3 solutions is the correct one. We will see later that the 7-point algorithm is the best choice for the robust case.

28 Comments to previous slide Computer Vision I: Image Formation Process 16/01/ Step 4) Choose: F = αf α F 2 The full null-space is given by: F = αf 1 + βf 2 We are free to say that we want: αf 1 + βf 2 1 (here F 1, F 2 are in vectorised form. Note that this is the same as having F 1, F 2 in matrix form and using the Frobenius norm for matrices) It is: α F 1 + β F 2 = αf 1 + βf 2 αf 1 + βf 2 (triangulation inequality) Hence we want: α F 1 + β F 2 1 Hence we want: α + β 1 (since F 1, F 2 are rows in V T ) Hence we can choose: β = 1 α

29 Comments to previous slide Computer Vision I: Image Formation Process 16/01/ Step 5) Compute det(αf α F 2 ) = 0.

30 Computer Vision I: Robust Two-View Geometry 16/01/ Can we get K s, R, T from F? Assume we have can we get out K 1, R, K 0, T? F = x 0 T K 0 T T RK 1 1 F has 7 DoF K 1, R, K 0, T have together 16 DoF Not directly possible. Only with assumptions such as: External constraints Camera does not change over several frames This is an important topic (more than 10 years of research!) called auto-calibration or self-calibration. We look at it in detail in next lecture.

31 Coming back to Essential Matrix Computer Vision I: Robust Two-View Geometry 16/01/ In a calibrated setting (K s are known): we use rays: x i = K 1 i x i then we get: x T 0 T Rx 1 = 0 In short: x T 0 Ex 1 = 0 where E is called the Essential Matrix E has 5 DoF, since T has 3DoF, R 3DoF (note overall scale of T is unknown) X E has also Rank 2 (R, T)

32 Computer Vision I: Robust Two-View Geometry 16/01/ How to compute E We have: x 0 T Ex 1 = 0 (reminder: x 0 T Fx 1 = 0) Given m 8 matching run 8-point algorithm (as for F) Given m = 7 run 7-point algorithm and get 1 or 3 solutions Given m = 5 run 5-point algorithm to get up to 10 solutions. This is the minimal case since E has 5 DoF. 5-point algorithm history: Kruppa, Zur Ermittlung eines Objektes aus zwei Perspektiven mit innere Orientierung, Sitz.-Ber. Akad. Wiss., Wien, Math.-Naturw. Kl., Abt. IIa, (122): , found 11 solutions M. Demazure, Sur deux problemes de reconstruction, Technical Report Rep. 882, INRIA, Les Chesnay, France, 1988 showed that only 10 valid solutions exist D. Nister, An Efficient Solution to the Five-Point Relative Pose Problem, IEEE Conference on Computer Vision and Pattern Recognition, Volume 2, pp , 2003 fast method which gives out 10 solutions of a 10 degree polynomial

33 Computer Vision I: Robust Two-View Geometry 16/01/ Can we get R, T from E? Assume we have E = T R, can we get out R, T? E has 5 DoF R, T have together 6 DoF We can get T up to scale, and a unique R X (R, T)

34 Computer Vision I: Image Formation Process 16/01/ How to get a unique T, R? 1) Compute T Note: E has rank 2, and T is in the left nullspace of E since This means that an SVD of E must look like: T v 0 E = UDV T = u 0 u 1 T T v ) Compute 4 possible solutions for R T t T = (0,0,0) This fixes the norm of T to 1, but correct sign (+/ T) is done in step 3 R 1,2 =+/ UR T 90 o where E = UDV T, R 90 = 3) Derive the unique solution for R and sign for T: 1) det(r) = 1 2) Reconstruct a 3D point and choose the solution where it lies in front of the two cameras. (In robust case: Take solution where most ( 5) points lie in front of the cameras) v 2 T V T T ; R 3,4 =+/ UR 90 ov T (see derivation HZ page 259; Szeliski page 310) , R 90 =

35 Visualization of the 4 solutions for R, T Computer Vision I: Robust Two-View Geometry 16/01/ T This is the correct solution, since point is in front of cameras! T T T The property that points must lie in front of the camera is known as Chirality (Hartley 1998)

36 What can we do with F, E? Computer Vision I: Robust Two-View Geometry 16/01/ F/E encode the geometry of 2 cameras Can be used to find matching points (dense or sparse) between two views F/E encodes the essential information to do 3D reconstruction

37 Fundamental and Essential Matrix: Summary Computer Vision I: Robust Two-View Geometry 16/01/ Derive geometrically F, E F for un-calibrated cameras E for calibrated cameras Calibration: Take measurements (points) to compute F, E F minimum of 7 points -> 1 or 3 real solutions. F many points -> least square solution with SVD E minimum of 5 points -> 10 solutions E many points -> least square solution with SVD Can we derive the intrinsic (K) an extrinsic (R, T) parameters from F, E? -> F next lecture -> E yes can be done (translation up to scale) What can we do with F, E? -> essential tool for 3D reconstruction

38 Half-way slide 3 Minutes break Computer Vision I: Multi-View 3D 16/01/

39 In last lecture we asked (for rotating camera) Computer Vision I: Robust Two-View Geometry 16/01/ Question 1: If a match is completely wrong then argmin h Ah is a bad idea Question 2: If a match is slightly wrong then argmin h Ah might not be perfect. Better might be a geometric error: argmin h Hx x

40 Robust model fitting Computer Vision I: Robust Two-View Geometry 16/01/ RANSAC: Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography Martin A. Fischler and Robert C. Bolles (June 1981). [Side credits: Dimitri Schlesinger]

41 Computer Vision I: Robust Two-View Geometry 16/01/ Example Tasks Search for a straight line in a clutter of points y x i.e. search for parameters and for the model given a training set

42 Example Tasks Computer Vision I: Robust Two-View Geometry 16/01/ Estimate the fundamental matrix i.e. parameters satisfying given a training set of correspondent pairs For Homography of rotating camera we have: x l i H = x r i

43 Two sources of errors Computer Vision I: Robust Two-View Geometry 16/01/ Noise: the coordinates deviate from the true ones according to some rule (probability) the father away the less confident 2. Outliers: the data have nothing in common with the model to be estimated Ignoring outliers can lead to a wrong estimation. The way out find outliers explicitly, estimate the model from inliers only

44 Computer Vision I: Robust Two-View Geometry 16/01/ Task formulation Let be the input space and be the parameter space. The training data consist of data points Let an evaluation function of a point with a model. Straight line Fundamental matrix be given that checks the consistency f x 1, x 2, a, b = 0 if ax 1 + bx 2 1 t (e. g. 0.1) 1 otherwise f x l, x r, F = 0 if x l t Fx r t (e. g. 0.1) 1 otherwise Inlier Outlier The task is to find the parameter that is consistent with the majority of the data points: y = argmin y i f(x i, y) x 2 x 1

45 First Idea: 2D Line estimation Computer Vision I: Robust Two-View Geometry 16/01/ Question: How to compute: y = argmin y A naïve approach: enumerate all parameter values know as Hough Transform (very time consuming and not possible at all for many free parameters (i.e. high dimensional parameter space) i f(x i, y) Image with points y Θ r x Encode all line with two parameters (r, Θ) Hough transform Goal: Find point in the figure where most lines meet Image with just 3 points All lines that go through these 3 points

46 First Idea: 2D Line estimation Computer Vision I: Robust Two-View Geometry 16/01/ Hough transform r Θ Accumulator space number of inliers (sketched) Observation: The parameter space have very low counts Idea: do not try all values but only some of them. Which ones?

47 Data-driven Oracle Computer Vision I: Robust Two-View Geometry 16/01/ An Oracle is a function that predicts a parameter given the minimum amount of data points (d-tuple): Examples: Line can be estimated from d = 2 points Fundamental matrix from d = 7 or 8 points correspondences Homography can be computed from d = 4 points correspondences First Idea: Do not enumerate all parameter values but all d-tuples of data points That is then n d number of tests, e.g. n 2 for lines (with n points) The optimization is performed over a discrete domain. y = argmin y i f(x i, y) Second Idea: Do not try all subsets, but sample them randomly

48 Computer Vision I: Robust Two-View Geometry 16/01/ RANSAC [Random Sample Consensus, Fischler and Bolles 1981] Basic RANSAC method: Can be done in parallel! Repeat many times select d-tuple, e.g. (x 1, x 2 ) for lines compute parameter(s) y, e.g. line y = g x 1, x 2 evaluate f y = i f(x i, y) If f y f y set y = y and keep value f y Sometimes we get a discrete set of intermediate solutions y. For example for F-matrix computation from 7 points we have up to 3 solutions. The we simply evaluate f y for all solutions. How many times do you have to sample in order to reliable estimate the true model?

49 Computer Vision I: Robust Two-View Geometry 16/01/ Convergence Observation: it is necessary to sample any in order to estimate the model correctly. -tuple of inliers just once Let ε be the probability of outliers. The probability to sample d inliers is 1 ε d (here = 0.64) The probability of a wrong d-tuple is (here 0.36) The probability to sample n times only wrong tuples is (here = ) 1000 points overall ε 0.2 The probability to sample the right tuple at least once during the process (i.e. to estimate the correct model according to assumptions) (here %)

50 Convergence probabity Computer Vision I: Robust Two-View Geometry 16/01/ ε (outliers) n

51 Comment Computer Vision I: Image Formation Process 16/01/ In our derivation for p = we were slightly optimistic since degenerate inliers may give rise to bad lines + + However, these bad lines have little support wrt number of inliers We also define later a refinement procedure which can correct such bad lines

52 The choice of the oracle is crucial Computer Vision I: Robust Two-View Geometry 16/01/ Example the fundamental matrix: a) 8-point algorithm Probability: 70% (n = 300; ε = 0.5; d = 8) b) 7-point algorithm Probability: 90% (n = 300; ε = 0.5; d = 7) Number of trials to get p% accuracy (here 99%) d ε p = ε d n n = log 1 p log(1 1 ε d )

53 The choice of evaluation function is crucial Computer Vision I: Image Formation Process 16/01/ Evaluation function: f x 1, x 2, a, b = 1 if ax 1 + bx 2 1 t (e. g. 0.1) 0 otherwise Algebraic error: Is a measure that has no geometric meaning Example: For a line: d x 1, x 2, a, b = ax 1 + bx 2 1 For a homograpy: d x 1, x 2, a, b = Ah (where A is 1 8 matrix derived as above For F-matrix: d x l, x r, F = x l t Fx r error function Geometric error: Is a measure that considers a distance in image plane Example: For a line: d x 1, x 2, a, b = d( x 1, x 2, l a, b ) Line: l a, b (d is Euclidean distance between point to line) d( x 1, x 2, l a, b ) x 1, x 2 Geometric error: for homography and F-matrix to come

54 The choice of confidence interval is crucial Computer Vision I: Robust Two-View Geometry 16/01/ Examples: Large confidence, right model, 2 outliers Large confidence, wrong model, 2 outliers Small confidence, Almost all points are outliers (independent of the model)

55 Extension: Adaptive number of samples n Computer Vision I: Robst Two-View Geometry 16/01/ Choose n in an adaptive way: 1) Fix p = 99.9% (very large value) 2) Set n = and ε = 0.9 (large value for outlier) 3) During RANSAC adapt n, ε : 1) Re-compute ε from current best solution ε = outliers / all points 2) Re-Compute new n: n = log 1 p log(1 1 ε d )

56 MSAC (M-Estimator SAmple Consensus) Computer Vision I: Robust Two-View Geometry 16/01/ If a data point is an inlier the penalty is not 0, but it depends on the distance to the model. Example for the fundamental matrix: becomes f x l, x r, F = 0 if x l t Fx r t (e. g. 0.1) 1 otherwise f x l, x r, F = x l t Fx r if x t l Fx r t (e. g. 0.1) t otherwise the task is to find the model with the minimum average penalty robust function t t) [P.H.S. Torr und A. Zisserman 1996]

57 Randomized RANSAC Computer Vision I: Robust Two-View Geometry 16/01/ Evaluation of a hypothesis, i.e. is often time consuming Randomized RANSAC: instead of checking all data points 1. Sample m points from 2. If all of them are good, check all others as before 3. If there is at least one bad point, among m, reject the hypothesis It is possible that good hypotheses are rejected. However it saves time (bad hypotheses are recognized fast) one can sample more often overall often profitable (depends on application).

58 Refinement after RANSAC Computer Vision I: Image Formation Process 16/01/ Typical procedure: 1. RASNAC: compute model y in a robust way 2. Find all inliers x inliers 3. Refine model y from inliers x inliers 4. Go to Step 2. (until numbers of inliers or model does not change much)

59 In last lecture we asked (for rotating camera) Computer Vision I: Robust Two-View Geometry 16/01/ Question 1: If a match is completly wrong then Answer: RANSAC with d = 4 argmin h is a bad idea Question 2: If a match is slighly wrong then argmin h Ah might not be perfect. Better might be a geometric error: argmin h Hx x Answer: see next slides Ah

60 Reminder from last Lecture: Homography for rotating camera Computer Vision I: Robust Two-View Geometry 16/01/ image Algorithm: 1) Take m 4 point matches (x, x ) 2) Assemble A with Ah = 0 3) compute h = argmin h Ah subject to h = 1, use SVD to do this.

61 Refine Hypothesis H with inliers Computer Vision I: Robust Two-View Geometry 16/01/ Algebraic error: argmin h Ah 2. First geometric error: H = argmin H i d(x i, Hx i ) where d(a, b) is 2D geometric distance a b 2 x i Hx i x i This is not symmetric

62 Refine Hypothesis H with inliers Computer Vision I: Robust Two-View Geometry 16/01/ Algebraic error: argmin h Ah 2. First geometric error: H = argmin H i d(x i, Hx i ) where d(a, b) is 2D geometric distance a b 2 3. Second, symmetric geometric error: H = argmin H i d x i, Hx i + d x i, H 1 x i x i H 1 x i Hx i x i

63 Refine Hypothesis H with inliers Computer Vision I: Robust Two-View Geometry 16/01/ Algebraic error: argmin h Ah 2. First geometric error: H = argmin H i d(x i, Hx i ) where d(a, b) is 2D geometric distance a b 2 3. Second, symmetric geometric error: H = argmin H i d x i, Hx i + d x i, H 1 x i 4. Third, optimal geometric error (gold standard error): {H, x^ i, x ^ i } = argmin i d(x i, x^ i ) + d(x i, x ^ i ) subject to ^x i = Hx ^ i H, x ^ ^ i, x i H x i ^ ^ x i x i x i the true 3D points ^ X are searched for Comment: This is optimal in the sense that it is the maximum-likelihood (ML) estimation under isotropic Gaussian noise assumption for x ^ (see page 103 HZ)

64 Full Homography Method (HZ page 123) Computer Vision I: Robust Two-View Geometry 16/01/ we discussed : Harries corner detector we discussed : Kd-tree to make it fast This is a geometric error (for fixed H, see next slides). Depending on runtime one can choose different once. See next slide This is the optimal geometric error See next slides [see details on page 114ff in HZ]

65 Example Computer Vision I: Robust Two-View Geometry 16/01/ Input images 500 interest points

66 Example Computer Vision I: Robust Two-View Geometry 16/01/ putative matches 117 outliers found 151 inliers found 262 inliers after guided matching Guided matching Variant: use given H and look for new inliers. Here we also double the threshold on appearance feature matches to get more inliers.

67 Geometric derivation of confidence interval Computer Vision I: Image Formation Process 16/01/ Assume Gaussian noise for a point with σ standard deviation and 0 mean: To have a 95% chance that an inlier is inside the confidence interval, we require: 1. For a 2D line: d x, l σ 3.84 = t 2. For a Homography: d x l, x r, H σ 5.99 = t 3. For an F-matrix: d x l, x r, F σ 3.84 = t (see page 119 HZ)

68 Methods for F/E/H Matrix computation - Summary Computer Vision I: Image Formation Process 16/01/ Procedure (as mentioned above): 1. RASNAC: compute model F/E/H in a robust way 2. Find all inliers x inliers (with potential relaxed criteria) 3. Refine model F/E/H from inliers x inliers 4. Go to Step 2. (until numbers of inliers or model does not change much) We need geometric error for a fixed model F/E/H : 1. For a Homography: d x, x, H = min d x, x ^ + d x, x ^ subject to x ^ = Hx^ x, ^ x ^ 2. For an F/E-matrix: d x, x, F/E = min d x, x ^ + d x, x ^ subject to x ^ t F/Ex ^ = 0 ^x, x ^ We need geometric error for model refinement F/E/H : 1. For a Homography: {H, x^ i, x^ i } = argmin H, x^ i, x ^ i i d(x i, x^ i ) + d(x i, ^ x i ) subject to ^ x i = Hx ^ i 2. For an F/E-matrix: {F /E, x^ i, x^ i }= argmin d(x i, x^ i ) + d(x i, ^x i ) sbj. to x ^ t i F/Ex^ i = 0 F/E, x^ i, ^x i i we see in next lecture that this one can be computed in closed-form

69 A few word on iterative continuous optimization So far we had linear (least square) optimization problems: x = argmin x Ax For non-linear (arbitrary) optimization problems: x = argmin x f(x) Iterative Estimation methods (see Appendix 6 in HZ; page 597ff) Gradient Descent Method (good to get roughly to solution) Newton Methods (e.g. Gauss-Newton): second order Method (Hessian). Good to find accurate result Levenberg Marquardt Method: mix of Newton method and Gradient descent Red Newton s method; green gradient descent Computer Vision I: Robust Two-View Geometry 16/01/

70 Application: Automatic Panoramic Stitching Computer Vision I: Robust Two-View Geometry 16/01/ An unordered set of images: Run Homography search between all pairs of images

71 Application: Automatic Panoramic Stitching Computer Vision I: Robust Two-View Geometry 16/01/ automatically create a panorama

72 Application: Automatic Panoramic Stitching Computer Vision I: Robust Two-View Geometry 16/01/ automatically create a panorama

73 Application: Automatic Panoramic Stitching Computer Vision I: Robust Two-View Geometry 16/01/ automatically create a panorama