Step-by-Step Model Buidling

Review Feature selection Feature selection Feature correspondence Camera Calibration Euclidean Reconstruction Landing Augmented Reality Vision Based Control Sparse Structure and camera motion

Review Feature selection Feature selection Feature correspondence Camera Calibration Epipolar Rectification Dense Correspondence Euclidean Reconstruction Sparse Structure and motion Texture mapping 3-D Model

Review Feature selection Feature selection Feature correspondence Projective Reconstruction Partial Scene Knowledge Partial Motion Knowledge Partial Calibration Knowledge Camera Self-Calibration Epipolar Rectification Dense Correspondence Euclidean Reconstruction Texture mapping 3-D Model

Examples

Feature Selection Compute Image Gradient Compute Feature Quality The image cannot be displayed. Your measure for each pixel Search for local maxima Feature Quality Function Local maxima of feature quality function

Feature Tracking Translational motion model Closed form solution 1. Build an image pyramid 2. Start from coarsest level 3. Estimate the displacement at the coarsest level 4. Iterate until finest level

Coarse to fine feature tracking 0 1 2 1. compute 2. warp the window in the second image by 3. update the displacement 4. go to finer level 5. At the finest level repeat for several iterations The image cannot be displayed. Your

Optical Flow Integrate around over image patch Solve

Affine feature tracking Contrast change Intensity offset

Tracked Features

Wide baseline matching Point features detected by Harris Corner detector

Wide baseline Feature Matching 1. Select the features in two views 2. For each feature in the first view 3. Find the feature in the second view that maximizes 4. Normalized cross-correlation measure Select the candidate with the similarity above selected threshold

More correspondences and Robust matching Select set of putative correspondences Repeat 1. Select at random a set of 8 successful matches 2. Compute fundamental matrix 3. Determine the subset of inliers, compute distance to epipolar line The ima ge cann 4. Count the number of points in the consensus set

RANSAC in action Inliers Outliers

Epipolar Geometry Epipolar geometry in two views Refined epipolar geometry using nonlinear estimation of F

Two view initialization calibrated Recover epipolar geometry Compute (Euclidean) projection matrices and 3-D struct. uncalibrated unknown Compute (Projective) projection matrices and 3-D struct.

Nonlinear Refinement Euclidean Bundle adjustment Initial estimates of are available Final refinement, nonlinear minimization with respect to all unknowns

Example - Euclidean multi-view reconstruction

Example Original sequence Tracked Features

Recovered model

Euclidean Reconstruction

Epipolar rectification Make the epipolar lines parallel Dense correspondences along image scanlines Computation of warping homographies 1. Map the epipole to infinity Translate the image center to the origin Rotate around z-axis for the epipole lie on the x-axis Transform the epipole from x-axis to infinity The imag e The imag e 2. Find a matching transformation is compatible with the epipolar geometry is chosen to minimize overall disparity T h e

Epipolar rectification Rectified Image Pair

Dense Matching Establish dense correspondences along scan-lines Standard stereo configuration Constraints to guide the search 1. ordering constraint 2. disparity constraint limit on disparity 3. uniqueness constraint each point has a unique match in the second view

Dense Matching

Dense Reconstruction

Texture mapping, hole filling

Texture mapping

Steps + = Images! Points: Structure from Motion Points! More points: Multiple View Stereo Points! Meshes: Model Fitting Meshes! Models: Texture Mapping Images! Models: Image-based Modeling + + + =

Bundle adjustment Theory: The Levenberg Marquardt algorithm Practice: The Ceres-Solver from Google

Multiple View Stereo State-of-the-art: PMVS: http://grail.cs.washington.edu/software/pmvs/ Accurate, Dense, and Robust Multi-View Stereopsis, Y Furukawa and J Ponce, 2 Benchmark: http://vision.middlebury.edu/mview/ A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithm SM Seitz, B Curless, J Diebel, D Scharstein, R Szeliski. 2006. Baseline: Multi-view stereo revisited. M Goesele, B Curless, SM Seitz. 2006.

How to get the intrinsic parameters? Auto-calibration Self-Calibration and Metric Reconstruction in spite of Varying and Unknown Internal Camera Parameters, M Pollefeys, R Koch and L Van Gool, 1998. http://mit.edu/jxiao/public/software/autocalibrate/autocalibration_lin.m Grid Search to look for the solution with minimal reprojection error for f=min_f:max_f do everything, then obtain reprojection error after bundle adjustment Optimize for this value in bundle adjustment Camera Calibration (with checkerboard) http://www.vision.caltech.edu/bouguetj/calib_doc/ EXIF of JPEG file recorded from digital camera Read the code of Bundler to understand how to convert EXIF into focal length value http://phototour.cs.washington.edu/bundler/

Real World Applications Streetview Reconstruction and Recognition http://vision.princeton.edu/projects/2009/iccv/ http://vision.princeton.edu/projects/2009/tog/ Photo Tourism http://phototour.cs.washington.edu/ Microsoft Photosynth http://photosynth.net/ 2d3, boujor (Matchmovers) and movies http://www.2d3.com/ http://www.vicon.com/boujou/ Robotics: SLAM http://openslam.org/ Simultaneous Localization And Mapping