Computational Optical Imaging - Optique Numerique. -- Single and Multiple View Geometry, Stereo matching --

Transcription

1 Computational Optical Imaging - Optique Numerique -- Single and Multiple View Geometry, Stereo matching -- Autumn 2015 Ivo Ihrke with slides by Thorsten Thormaehlen

2 Reminder: Feature Detection and Matching Wide-Baseline-Matching

3 Wide-baseline Matching with SIFT SIFT = Scale Invariant Feature Transform David G. Lowe: Distinctive image features from scale-invariant keypoints (IJCV 2004) Suited for wide-baseline matching Invariance to changes in illumination, scale, and rotation No gradient approach, but compares feature description of all candidates Many applications

4 Wide-baseline Matching with SIFT Application example: Mosaic generation

5 Wide-baseline Matching with SIFT Application example: Object recognition

6 SIFT: Keypoint localization Stereo pair with super-imposed SIFT features Arrow size indicates scale 729 out of 832 feature are left after contrast thresholding (try cross-eye fusion to identify the outliers :)

7 SIFT: Descriptor matching

8 Software VLFeat ( Implements SIFT and other modern features C-implementation with MATLAB bindings Win/Linux Good tutorials on parameter selection

9 Towards Multiple Views and Self- Calibration -- Two-View Geometry and Basic Stereo --

10 Fundamental Matrix Fundamental Matrix (F-Matrix):

11 Epipolar Geometry in our example

12 Fundamental Matrix F is a rank 2 homogeneous matrix with 7 degrees of freedom Epipolar lines Epipoles

13 Fundamental Matrix F can be computed from camera matrices General projective cameras: with and Canonical cameras not at infinity and

14 Use Case Stereo Matching Typical left and right image with parallel optical axes and only horizontally displaced Left image Right image [implementation by Rohit Singh and Mitul Sara]

15 Epipolar Geometry of a Stereo Pair Epipolar lines are parallel lines if optical axes are parallel

16 Epipolar Geometry of a Stereo Pair Simplest: Search a moving window and perform correlation E.g. SSD (sum of squared differences) Alternatively, SAD (sum of absolute differences), CC (cross correlation), etc. Usually restricted search range

17 Correlation-Based Stereo Matching Example, SSD, win=5, search range=15 [implementation by Rohit Singh and Mitul Saha]

18 Correlation-Based Stereo Matching Example, SSD, win=5, search range=8 [implementation by Rohit Singh and Mitul Saha]

19 Correlation-Based Stereo Matching Example, SAD, win=5, search range=8 [implementation by Rohit Singh and Mitul Saha]

20 Correlation-Based Stereo Matching Example, SSD, win=20, search range=8 [implementation by Rohit Singh and Mitul Saha]

21 A Modern Technique ADCensus [Mei 11] (#1 Middlebury stereo benchmark, Sept. 2013)

22 Towards Multiple Views and Self- Calibration -- The Structure-from-motion Pipeline --

23 Voodoo Camera Tracker - Steps of camera tracking image sequence Feature detection and correspondence analysis Outlier Elimination Incremental Bundle Adjustment Self-Calibration camera parameters

24 Correspondence Analysis KLT Tracker Correspondence Analysis For video KLT-Tracker (Kanade, Lucas and Tomasi) Minimization of the SSD between two windows in subsequent camera images to find the displacement vector window image K-1 image K

25 KLT Tracker Linearization of the image signal with a Taylor series Finding the minimum by setting the derivative to zero

26 KLT Tracker with

27 Optical Flow Horn & Schunck 81 Continuous version of tracking: optical flow Apparent motion of brightness patterns in an image sequence (typically two frames) For images: u( x): R 2 R 2, is a vector valued fct. Often visualized as vector field or color coded Optical Flow as seen by a person at the back of a train

28 right left Example Yosemite sequence Flow field (middlebury coding) middlebury coding IPOL coding Flow field (IPOL coding)

29 Optical Flow- Derivation Assume a video I( x, t): R 3 R Brightness constancy implies I x + u( x, t), t = I( x, t + 1) Look at one particular time step with flow vectors u = (u x, u y ) perform Taylor expansion of I x, t + 1 : I x, t + 1 I x, t + I x u x + I y u y + I t + O( 2 ) implies I x u x + I y u y + I t = 0 Alternative form: I u + I t = 0

30 KLT-Tracker: Resolution pyramids KLT and optical flow descriptions are differential in nature only suitable for small displacements (up to 2 pixels) Large displacements are small on a coarser scale compute displacements on coarse scale Upsample Iteratively compute residuals on finer scales resolution

31 KLT-Tracker: Affine mapping for longer sequences

32 KLT-Tracker summary For video (small displacements) Minimization of the SSD between two windows in subsequent camera images Linearization of the image signal with a Taylor series linear equation system Linearization only valid for small displacements Resolution Pyramids

33 Voodoo Camera Tracker - Steps of camera tracking image sequence Feature detection and correspondence analysis Outlier Elimination Incremental Bundle Adjustment Self-Calibration camera parameters

34 Outlier Elimination RANSAC (Random Sample Consensus) method

35 RANSAC example line fit LS-fit after RANSAC Least-squares fit

36 Outlier Elimination - Fundamental Matrix Fundamental Matrix (F-Matrix):

37 Fundamental Matrix Estimation Estimating the fundamental matrix from feature correspondence: multiple of these equations gives a linear equation system

38 Outlier Elimination - 2D Homography 2D Homography (H-Matrix):

39 2D Homography Matrix Estimation Estimating the 2D homography matrix from feature correspondence: multiple of these equations gives a linear equation system

40 Outlier Elimination - Camera Matrix Camera matrix (A-Matrix):

41 Incremental Bundle Adjustment

42 Bundle Adjustment Bundle Adjustment

43 Levenberg Marquardt Non-linear Least Squares Measurement vector Parameter vector Taylor approximation: with N x M Jacobian Matrix

44 Levenberg Marquardt Non-linear Least Squares transformed to linear least squares problem for each iteration

45 Levenberg Marquardt Non-linear Least Squares Linear least squares problem can be solved with normal equations use to update solution iteratively

46 Levenberg Marquardt Non-linear Least Squares Levenberg Marquardt uses slightly different normal equations Original normal equations Modified normal equations Lambda is changed during optimization successful iteration failed iteration small large ~ Newton style (quadratic convergence) ~ Gradient descent style (guaranteed decrease)

47 Levenberg Marquardt Requirements for Levenberg Marquardt minimization Function to compute f Start value X 0 Optionally, function to compute J (but numerical derivation works as well)

48 Sparse Levenberg Marquardt For bundle adjustment the problem becomes to large (100 cameras D object points = parameters) Can achieve huge speed-up by exploiting sparse structure of Jacobian matrix Partition parameters partition A partition B (only dependent on A and itself) (typically A contains camera parameters, and B contains 3D points)

49 Sparse Levenberg Marquardt Jacobian becomes Normal equations become

50 Sparse Levenberg Marquardt Corresponding block structure is where denotes augmented by multiplying its diagonal entries by a factor of 1 +, and likewise. Left multiplication with yields which can be used to find with which may be back-substituted to get with

51 Sparse Levenberg Marquardt Jacobian for bundle adjustment has sparse block structure im.pts. A 1 J A 1 A 2 A 3 P J T J U 1 U 2 W U 3 12 x m 3 x n (in general much larger) W T Needed for non-linear minimization V

52 Voodoo camera tracker demo session

53 Self-calibration Why self-calibration? Allows flexible acquisition No prior calibration necessary Possibility to vary intrinsic camera parameters Use archive footage

54 Self-calibration We want to find a 4x4 transformation matrix that transforms all projective cameras into metric cameras This does not change the back-projections onto the feature points

55 Voodoo Camera Tracker Some applications Virtual advertising

56 Voodoo Camera Tracker More applications Car navigation 3D Endoscopy UAV terrain reconstruction Architectural visualisation

57 Matchmoving in Cloverfield

58 Matchmoving in Cloverfield

59 References 1. Multiple View Geometry in Computer Vision, Richard Hartley and Andrew Zisserman, 2 nd Edition 2. Triggs, B.: Autocalibration and the absolute quadric. In IEEE Conference on Computer Vision and Pattern Recognition, S Pollefeys, M., Koch, R., Gool, L. V.: Self-Calibration and Metric Reconstruction in Spite of Varying and Unknown Internal Camera Parameters. In EEE International Conference on Computer Vision, S