3D Photography. Marc Pollefeys, Torsten Sattler. Spring 2015

Transcription

1 3D Photography Marc Pollefeys, Torsten Sattler Spring 2015

2 Schedule (tentative) Feb 16 Feb 23 Mar 2 Mar 9 Mar 16 Mar 23 Mar 30 Apr 6 Apr 13 Apr 20 Apr 27 May 4 May 11 May 18 May 25 Introduction Geometry, Camera Model, Calibration Features, Tracking / Matching Project Proposals by Students Structure from Motion (SfM) + 2 papers Dense Correspondence (stereo / optical flow) + 2 papers Bundle Adjustment & SLAM + 2 papers Easter Project Updates Multi-View Stereo & Volumetric Modeling + 2 papers 3D Modeling with Depth Sensors + 2 papers 3D Scene Understanding + 2 papers 4D Video & Dynamic Scenes + 2 papers Final Demos Pentecost

3 3D Photography Class 2 Features & Correspondences feature extraction, image descriptors, feature matching, feature tracking Chapters 4, 8 in Szeliski Book

4 Overview Local Features Intro Invariant Feature Extraction & Matching Feature Tracking

5 Today: Features & Correspondences M 2D-3D 2D-3D m i 2D-2D m i+1 Correspondences are at the heart of 3D reconstruction from images

6

7 Feature Matching vs. Tracking Image-to-image correspondences are key to passive triangulation-based 3D reconstruction Extract features independently and then match by comparing descriptors Extract features in first images and then try to find same feature back in next view What is a good feature?

8 Comparing Image Regions Compare intensities pixel-by-pixel I(x,y) I (x,y) Dissimilarity measures Sum of Square Differences SSD x y I (x,y) I(x,y) 2

9 Feature Points / Keypoints Required properties: Well-localized Stable across views, repeatable (i.e. same 3D point should be extracted as feature for neighboring viewpoints)

10 Feature Point Extraction Find points (local image patches) that differ as much as possible from all neighboring points homogeneous edge corner

11 Finding Stable Features Measure uniqueness of candidate Approximate SSD for small displacement Δ SSD w(x i ) I(x i ) I(x i ) 2 i w(x i ) I(x i ) I I x y i 2 I w(x i ) T x I x I y 2 I x I y I i y possible weights 2 I(x i ) T M I x I x

12 Finding Stable Features homogeneous edge corner Suitable feature positions should maximize i.e. maximize smallest eigenvalue of M

13 Harris Corner Detector Use small local window: Maximize cornerness : Only use local maxima, subpixel accuracy through second order surface fitting Select strongest features over whole image and over each tile (e.g. 1000/image, 2/tile)

14 Simple Matching For each corner in image 1 find the corner in image 2 that is most similar and vice-versa Only compare geometrically compatible points (epipolar constraints, ) Keep mutual best matches What transformations does this work for?

15 Comparing Image Regions Compare intensities pixel-by-pixel I(x,y) I (x,y) Dissimilarity measures Sum of Square Differences SSD x y I (x,y) I(x,y) 2

16 Comparing Image Regions Compare intensities pixel-by-pixel I(x,y) I (x,y) Similarity measures Zero-mean Normalized Cross Correlation NCC N I,I N I',I N I',I' N I,I I(x,y) I x y I (x,y) I

17 Feature Matching Example What transformations does this work for? What level of transformation do we need?

18 Overview Local Features Intro Invariant Feature Extraction & Matching Feature Tracking

19 Wide Baseline Matching Requirement to cope with larger variations between images Translation, rotation, scaling Perspective foreshortening Non-diffuse reflections Illumination geometric photometric changes transformations

20 Invariant Detectors Rotation invariant Scale invariant Affine invariant (approximately invariant w.r.t. perspective/viewpoint)

21 2D Transformations of a Local Patch e.g. VIP e.g. Harris corners & NCC e.g. MSER e.g. SIFT

22 David Lowe s SIFT Features (Lowe, ICCV99) Detector + descriptor (later) Recover features with position, orientation and scale

23 Position Look for strong responses of DOG filter (Difference-Of-Gaussian) Only consider local maxima

24 Scale Look for strong responses of DOG filter (Difference-Of-Gaussian) over scale space Only consider local maxima in both position and scale Fit quadratic around maxima for subpixel

25 Minimum Contrast and Cornerness

26 Invariants So Far Additive and multiplicative changes in brightness Difference-Of-Gaussian! Translation Scale Rotation is missing!

27 Orientation Create histogram of local gradient directions computed at selected scale Assign canonical orientation at peak of smoothed histogram Each key specifies stable 2D coordinates (x, y, scale, orientation)

28 2D Transformations of a Local Patch e.g. VIP e.g. Harris corners & NCC e.g. MSER e.g. SIFT

29 Affine Invariant Features Perspective effects can locally be approximated by affine transformation

30 Extreme Wide Baseline Matching Find Correspondences between these images using the MSER Detector [Matas 02]

31 Maximally Stable Extremal Regions Local regions, not points!

32 Maximally Stable Extremal Regions Extremal regions: Much brighter than surrounding Use intensity threshold

33 Maximally Stable Extremal Regions Extremal regions: OR: Much darker than surrounding Use intensity threshold

34 Maximally Stable Extremal Regions Regions: Connected pixels at some threshold Region size = #pixels Maximally stable: Region constant near some threshold

35 A Sample Feature

36 A Sample Feature T is maximally stable with respect to surrounding

37 From Regions To Ellipses Compute center of gravity Compute Scatter (PCA / Ellipsoid)

38 From Regions To Ellipses Ellipse abstracts from pixels! Geometric representation: position/size/shape

39 Comparing Features Idea: Normalize to default position/size/shape e.g. circle of radius 16 pixels

40 Normalize ellipse to circle 2D rotation still unresolved

41 Same approach as for SIFT: Compute histogram of local gradients Find dominant orientation in histogram Rotate local patch into dominant orientation

42 Each normalized patch obtained from single image!

43 MSER Features Detect sets of pixels brighter/darker than surr. Fit elliptical shape to pixel set Warp image so that ellipse becomes circle Rotate to dominant gradient direction [other constructions possible as well]

44 MSER Features - Invariants Constant brightness changes (additive and multiplicative) Rotation, translation, scale Affine transformations Affine normalization of feature leads to similar patches in different views!

45 Local Patch Descriptors Small misalignments Brightness change More tolerant comparison needed!

46 Lowe s SIFT Descriptor Gradient Orientation/Magnitude Gradient Magnitude Ignore pixel values, use only local gradients Gradient direction more important than positions Partition into sectors to retain spatial information

47 Lowe s SIFT Descriptor Thresholded image gradients are sampled over 16x16 array of locations in scale space Create array of orientation histograms 8 orientations x 4x4 histogram array = 128D

48 Descriptor Computation Gradient Orientation/Magnitude Orientation Histogram per Sector Quantize gradient orientations in 45 steps Bin gradients into histogram Weight of gradient = gradient magnitude Concatenate histograms

49 Descriptor Computation Quantization errors: Small differences can lead to different bins! 22 quantized/rounded to 0 23 quantized/rounded to 45 Can be caused by Small errors in feature position, size, shape, or orientation Image noise Descriptor must be robust against this!

50 Hard Binning Classical (closest bin) If orientation is 3 different, all measurements go to second bin! Sudden change in histogram from ( ) to ( )

51 Soft Binning Soft weights: Bin Correctness Soft Binning If orientation is 3 different, descriptor changes only gradually!

52 Descriptor Invariance? Translation, scale, affine deformations: Inherited from detector Rotation: Align bins / histograms with dominant orientation of patch Uniform intensity / illumination changes? Adding constant value does not affect gradient Normalize vector to handle multiplicative changes Robustness to non-uniform changes Idea: Change affects gradient magnitude but not direction After normalization: Clip descriptor entries to be 0.2 Renormalize! But no invariance!

53 Memory Consumption After normalization: floating point values Need 128 x 4 = 512 bytes per descriptor Reduce memory consumption by quantization Use only 1 byte per descriptor entry 128 bytes Simple quantization: Multiply with 512 and round to integer value Little impact on descriptor quality For comparison: Storing raw grey values for 11x11 patch requires 121 bytes

54 For each region: 128-dim. descriptor How to find correspondences?

55 Descriptor Matching - Scenario I Two images in a dense image sequence: Think about maximum movement d (e.g. 50 pixel) Search in a window +/- d of old position Compare descriptors (Euclidean distance), choose nearest neighbor

56 Descriptor Matching - Scenario II Two arbitrary images / wide baseline Brute force search (e.g. GPU) OR: Approximate nearest neighbor search in descriptor space (kd-tree) OR: Find small set of matches, predict others

57 kd-tree-based Matching Descriptor Space kd-tree Iteratively split dimension with largest variance Matching: Traverse tree based on splits Depth 30 1B descriptors (~122MB) Curse of dimensionality: Need to visit all leaves to guarantee finding nearest neighbor Approximate search: Visit N leaf nodes

58 Searching Descriptor Space Learn important dimensions of 128D space for a given scene, e.g. PCA or LDA Project descriptors to important dimensions, use kd-tree

59 Descriptor Matching Spatial Search Window: Requires/exploits good prediction Can avoid far away similar-looking features Good for sequences Descriptor Space: Initial tree setup Fast lookup for huge amounts of features More sophisticated outlier detection required Good for asymmetric (offline/online) problems, registration, initialization, object recognition, wide baseline matching

60 Correspondence Verification Features have only very local view Mismatches How to detect? Discard matches with low similarity Discard non-distinctive matches through Lowe s ratio test / SIFT ratio test Check for bi-directional consistency Geometric verification, e.g., RANSAC

61 Object Detection / Pose Estimation using single MSER Affine feature: position(2x), shape(3x), orientation(1x) 6 DOF, more than 2D simple point!

62 2D Transformations of a Local Patch e.g. VIP e.g. Harris corners & NCC In practice hardly observable for small patches! e.g. MSER e.g. SIFT

63 Viewpoint Invariant Patches (VIP) (Wu, CVPR 08) Use known planar geometry to remove perspective distortion Alternative: Use vanishing points to rectify patch

64 Feature Detectors And Descriptors Detector Find interesting regions (position/size/shape) Assign dominant gradient orientation Normalize regions Descriptor Compute signature upon normalized region Behave smoothly in presence of distortions: brightness changes / normalization inaccuracies

65 Recent Variants and Accelerations (much faster, but this usually comes at a price) BRIEF[Calonder10]: binary descriptor (tests=position a darker than b), compare descriptors by XOR (Hamming) + POPCNT RIFF[Takacs10]: CENSURE + gradients tangential/radial ORB[Rublee11] FAST+orientation BRISK[Leutenegger11] FAST+scale+BRIEF FREAK[Alahi12] FAST + daisy -BRIEF Lucid[Ziegler12]: sort intensities D-BRIEF[Trzcinski12]:Box-Filter+learned projection+brief LDA HASH[Strecha12]: binary tests on descriptor

66 Some Feature Resources Affine feature evaluation + binaries: SIFT, MSER & much more (mostly Matlab): SURF: GPU-SIFT: DAISY (dense descriptors) FAST[er] corner detector (simple but ) OpenCV (MSER, binary descriptors, matching, )

67 Require Features: Local Normalization + Robust Descriptors (Nearly) planarity accross feature in 3D Detectable regions, enough structure / texture Allow us to handle Changing perspective/illumination/scale Scenarios with many occlusions Different reasoning (descriptor vectors)

68 Features: Local Normalization + Robust Descriptors But... Not all is perfect No true invariance against viewpoint changes SIFT: ~30 degrees Affine covariant, e.g., MSER, up to maybe 60 degrees Higher invariance = slower computation, fewer features Invariance vs. Descriptiveness: Similar descriptors even when regions are not! What level of invariance / speed / properties do YOU really need?

69 Overview Local Features Intro Invariant Feature Extraction & Matching Feature Tracking

70 Feature Tracking Identify features and track them over video Small difference between frames potential large difference overall Standard approach: KLT (Kanade-Lukas-Tomasi)

71 Tracking Corners Through Video

72 Good Features to Track Use same window in feature selection as for tracking itself (see first part of lecture) Compute motion assuming it is small differentiate: linear system Affine is also possible, but a bit harder (6x6 instead of 2x2)

73 Example affine translation Simple displacement is sufficient between consecutive frames, but not to compare to reference template

74 Example translation affine

75 Good features to keep tracking Perform affine alignment between first and last frame Stop tracking features with too large errors

76 Intensity Linearization Brightness constancy assumption (small motion) 1D example possibility for iterative refinement

77 Intensity Linearization Brightness constancy assumption (small motion) 2D example (1 constraint) (2 unknowns)? the aperture problem isophote I(t+1)=I isophote I(t)=I Barberpole illusion (image source: Wikipedia)

78 Intensity Linearization How to deal with aperture problem? (3 constraints if color gradients are different) Assume neighbors have same displacement

79 Lucas-Kanade Assume neighbors have same displacement least-squares:

80 Revisiting the Small Motion Assumption Is this motion small enough? Probably not it s much larger than one pixel (1 st order Taylor not sufficient) How might we solve this problem? * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

81 Reduce the Resolution! * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

82 Gaussian pyramid of image I t-1 Gaussian pyramid of image I Coarse-to-Fine Optical Flow Estimation slides from Bradsky and Thrun u=1.25 pixels u=2.5 pixels u=5 pixels image I t-1 u=10 pixels image I

83 Coarse-to-Fine Optical Flow Estimation slides from Bradsky and Thrun run iterative L-K run iterative L-K warp & upsample... image J I t-1 Gaussian pyramid of image I t-1 image I Gaussian pyramid of image I

84 Next week: Project Proposals Reminder: Submit your proposal until Friday!