Flow Estimation. Min Bai. February 8, University of Toronto. Min Bai (UofT) Flow Estimation February 8, / 47
|
|
|
- Bernadette Simmons
- 6 years ago
- Views:
Transcription
1 Flow Estimation Min Bai University of Toronto February 8, 2016 Min Bai (UofT) Flow Estimation February 8, / 47
2 Outline Optical Flow - Continued Min Bai (UofT) Flow Estimation February 8, / 47
3 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Min Bai (UofT) Flow Estimation February 8, / 47
4 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Better dense optical ow generation - EpicFlow (CVPR 2015) Min Bai (UofT) Flow Estimation February 8, / 47
5 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Better dense optical ow generation - EpicFlow (CVPR 2015) Scene Flow Min Bai (UofT) Flow Estimation February 8, / 47
6 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Better dense optical ow generation - EpicFlow (CVPR 2015) Scene Flow Min Bai (UofT) Flow Estimation February 8, / 47
7 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Better dense optical ow generation - EpicFlow (CVPR 2015) Scene Flow Optic Flow vs Scene Flow Min Bai (UofT) Flow Estimation February 8, / 47
8 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Better dense optical ow generation - EpicFlow (CVPR 2015) Scene Flow Optic Flow vs Scene Flow 3D to 2D Projection Review Min Bai (UofT) Flow Estimation February 8, / 47
9 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Better dense optical ow generation - EpicFlow (CVPR 2015) Scene Flow Optic Flow vs Scene Flow 3D to 2D Projection Review Epipolar Constraint Review Min Bai (UofT) Flow Estimation February 8, / 47
10 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Better dense optical ow generation - EpicFlow (CVPR 2015) Scene Flow Optic Flow vs Scene Flow 3D to 2D Projection Review Epipolar Constraint Review Static scene structure from monocular vision - Robust Monocular Epipolar Flow Estimation (CVPR13) Min Bai (UofT) Flow Estimation February 8, / 47
11 Outline Optical Flow - Continued Better descriptors for matching - DeepFlow (ICCV 2013) Better dense optical ow generation - EpicFlow (CVPR 2015) Scene Flow Optic Flow vs Scene Flow 3D to 2D Projection Review Epipolar Constraint Review Static scene structure from monocular vision - Robust Monocular Epipolar Flow Estimation (CVPR13) Dynamic scene structure from stereo + time sequence - 3D Scene Flow Estimation with a Piecewise Rigid Scene Model (CVPR15) Min Bai (UofT) Flow Estimation February 8, / 47
12 DeepFlow - Motivation DeepFlow: Large Displacement Optical Flow with Deep Matching Weinzaepfel et al, ICCV 2013 Challenges in nding correct matching points in image pairs: Distortions due to perspective changes Distortions due to large displacements of object between image captures Occlusion, scale, texture, etc DeepFlow addresses rst two items Min Bai (UofT) Flow Estimation February 8, / 47
13 DeepFlow - Comparison with SIFT Creates descriptive feature unique to one point based on context patch Source: codeproject.com Min Bai (UofT) Flow Estimation February 8, / 47
14 DeepFlow - Comparison with SIFT Left and middle - traditional SIFT matching Based on rigid patch Right - more optimal matching Non rigid, deformable patch Source: Weinzaepfel et al, ICCV 2013 Min Bai (UofT) Flow Estimation February 8, / 47
15 DeepFlow - Architecture Inspired by deep convolutional neural networks Source: Weinzaepfel et al, ICCV 2013 Min Bai (UofT) Flow Estimation February 8, / 47
16 DeepFlow - Architecture Progression of image data through architecture Source: Min Bai (UofT) Flow Estimation Weinzaepfel et al, ICCV 2013 February 8, / 47
17 DeepFlow - Energy optimization Traditionally, ow eld is estimated by minimizing the following energy over the image: E(w) = E data + αe smoothness data term encourages color and gradient consistency smoothness term discourages abrupt changes in ow eld DeepFlow adds one more term to encourage ow estimation to equal displacements from matching E(w) = E data + αe smoothness + βe matching matching term also takes quality of matches into account Min Bai (UofT) Flow Estimation February 8, / 47
18 EpicFlow EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow Revaud et al, CVPR2015 Based in part on Deep Matching (as developed in DeepFlow) Also incorporates knowledge about visual edges in images Min Bai (UofT) Flow Estimation February 8, / 47
19 EpicFlow - Motivation Commonly used method - coarse-to-ne interpolation / energy minimization Start with smoothed or down-sampled image, estimate ow Use estimated ow for a less down-sampled image for further renement E smooth = Ψ( u 2 + v 2 ) Ψ is a robust penalization function Min Bai (UofT) Flow Estimation February 8, / 47
20 EpicFlow - Motivation Question: Should smoothness in ow eld be uniformly encouraged? What e ect does this have on natural motion discontinuities in 3D? What e ect does this have on small but fast moving objects? Source: Min Bai (UofT) Flow Estimation February 8, 2016 KITTI 11 / 47
21 EpicFlow - Motivation Question: Should smoothness in ow eld be uniformly encouraged? What e ect does this have on natural motion discontinuities in 3D? What e ect does this have on small but fast moving objects? Source: Min Bai (UofT) Flow Estimation February 8, 2016 KITTI 12 / 47
22 EpicFlow - Key Insight Motion discontinuities tend to occur at image edges In the natural world, the same objects tend to be coherent in color (compared to background) Abrupt changes in true optical ow tends to be due to 3D points not belonging to the same rigidly moving body or abrupt changes in depth Min Bai (UofT) Flow Estimation February 8, / 47
23 EpicFlow - Architecture Run edge detection algorithm (based on Structured Forests for Fast Edge Detection, ICCV 2013) Run Deep Matching (same algorithm as discussed previously) Flow eld interpolation through edge-aware distances This skips the coarse-to-ne pyramid! One level energy minimization Source: Revaud et al, CVPR 2015 Min Bai (UofT) Flow Estimation February 8, / 47
24 EpicFlow - Sparse to Dense Flow Field Interpolation Start with sparse ow eld from key point matching Calculate "edge aware distance" between two pixels p and q Then perform weighted average interpolation for ow at pixel p Finally smooth over ow image using one level variational method E(w) = E data + αe smoothness Min Bai (UofT) Flow Estimation February 8, / 47
25 EpicFlow - KITTI Table : KITTI Results Inputs Matching Edges Result Source: Min Bai (UofT) Flow Estimation Revaud et al, CVPR 2015 February 8, / 47
26 Dierence between Scene Flow and Optical FLow Optical ow reasons in 2D Output is only ow eld showing 2D pixel correspondences Scene ow reasons in 3D Recognizes that changes in the 2D image are due to object motion and/or change in camera position in 3D Optical ow = projection of 3D scene ow onto image Output can include: 3D structure (depth) 3D Camera motion 3D Object motion Min Bai (UofT) Flow Estimation February 8, / 47
27 Why bother with scene ow? Optical ow reasons about things like matching points, edges, interpolation, regional smoothing, etc in 2D Can only incorporate intuition about 2D But we know much about the world in 3D! For example, we know that in general... Many (most) objects are rigid Many (most) surfaces are planar Objects have continuous motion - no teleportation (in classical physics) Plus some domain specic knowledge (eg for driving): Most of a scene is likely static (road, trees, buildings) Most object motion is conned to road plane (for now) Camera motion is conned to road plane (if car is driven well) How do we incorporate these beliefs? Reason in 3D! Min Bai (UofT) Flow Estimation February 8, / 47
28 Why bother with scene ow? Source: Min Bai (UofT) Flow Estimation February 8, 2016 KITTI 19 / 47
29 Why is Scene Flow Hard? Inherent ambiguity: Need to infer 3D information from 2D images 2D observed ow can be due to any combination of object motion, camera motion, depth Similar problems as optical ow Sparseness of matches Errors in matching Occlusions, displacement, view point changes, etc Min Bai (UofT) Flow Estimation February 8, / 47
30 3D to 2D Projection Review Given a 3D point P = [X, Y, Z, 1] in homogeneous coordinates Projection into image is p = [x, y, 1] p = [K 0][R t]p (1) For canonical camera at world center: K = 0 1 0, R = I, t = f and thus x = f X Z, y = f Y Z Min Bai (UofT) Flow Estimation February 8, / 47
31 Epipolar Geometry Review Given two images I and I' of a static scene, p' corresponding to p must lie on epipolar line: p F p' = 0 F R 3x3 = fundamental matrix, rank = 2 Source: Min Bai (UofT) Flow Estimation February 8, / 47
32 Robust Monocular Epipolar Flow Estimation Robust Monocular Epipolar Flow Estimation Yamaguchi, McAllester, Urtasun, CVPR2013 Scene ow estimation from monocular (not stereo) image pairs at dierent times Assumes static scene Based on Epipolar ow Planar superpixel assumption Smoothing Min Bai (UofT) Flow Estimation February 8, / 47
33 Epipolar Flow Optical ow assumed to consist of two components p and p are corresponding points in I and I Flow u = p p u = u w (p) + u v (p) has two components: Idea: u w (p) due to camera rotation u v (p) due to camera translation Undo camera rotation rst (by estimating F ) Treat problem as epipolar ow Min Bai (UofT) Flow Estimation February 8, / 47
34 Epipolar Flow - Rectication Stereo rectication Epipolar rectication Source: S. Savarese Source: Yamaguchi et al Min Bai (UofT) Flow Estimation February 8, / 47
35 Epipolar Flow - Rectication Epipolar expansion (or contraction) Only valid when camera motion is purely translational! Flow is conned to epipolar lines coming out of epipole Epipole is found by right nullspace of F Fo = 0 Rectied image pair have epipole at SAME location Knowing F and o we can nd epipolar line in I on which p must lie Source: Yamaguchi et al Min Bai (UofT) Flow Estimation February 8, / 47
36 Epipolar Flow - Searching for Matches Search along epipolar lines to nd matching point pairs Generate dense matches (ow eld) Minimize matching cost using Semi Global Smoothing (for more info: publications/phowo11/180hirschmueller.pdf) Source: neurology.org Min Bai (UofT) Flow Estimation February 8, / 47
37 Epipolar Flow - Segmentation and Smoothing Given motion is only translation + static scene constant v z for all 3D points relative to camera Dene a ratio ω p = vp Zp called the vz-index for each point p Source: Yamaguchi et al Min Bai (UofT) Flow Estimation February 8, / 47
38 Epipolar Flow - Segmentation and Smoothing Assume each piecewise planar world - superpixels represent 3D planes Jointly segment image into slanted plane superpixels using Pixel appearance + location Flow information (use the vz-index for each p) Regularizers Planar assumption constrains vz-ratios for all p = (u, v) belonging to superpixel i centered at c i to disparity plane: ω i = α i (u c iv ) + β i (v c iv ) + γ i Goal - t one such disparity plane described by α i, β i, γ i to each superpixel Source: Yamaguchi et al Min Bai (UofT) Flow Estimation February 8, / 47
39 Question What about scenes with moving objects? Points in general no longer obey the same epipolar constraint How can we estimate both world geometry AND object motion? Min Bai (UofT) Flow Estimation February 8, / 47
40 3D Piecewise Rigid Scene Model 3D Scene Flow Estimation with a Piecewise Rigid Scene Model Vogel et al, IJCV 2015 Scene ow estimation from stereo image pairs + time series of images Handles dynamic environment Based on Homographies Piecewise planar assumption of 3D world Single reference view energy minimization Multi-frame motion continuity Min Bai (UofT) Flow Estimation February 8, / 47
41 3D Piecewise Rigid Scene Model - Homography Review Similar idea to epipolar geometry and fundamental matrix, but dierent Fundamental Matrix: F R 3x3, rank 2 maps point p in I to line l = F p in I Valid for general 3D points in world Homography Matrix H R 3x3, rank 3 maps point p in I to point p = Hp in I Valid for 3D points in world belonging to same plane n P = 0 Min Bai (UofT) Flow Estimation February 8, / 47
42 3D Piecewise Rigid Scene Model - Homography Review "Homography" means "similar drawing" Source: Min Bai (UofT) Flow Estimation February 8, / 47
43 3D Piecewise Rigid Scene Model - Homography Review Let π = π(r, t, n) be a moving plane in 3D with 9 total degrees of freedom 3 degrees in each of R, t, n Two observing cameras l and r taking two pictures each at t = 0 and t = 1 Camera l at t = 0 considered canonical, aka p 0 l = [K 0]P 0 Camera r at t = 0 has projection matrix M, or p 0 r = [M m]p 0 Thus view of plane can be transformed between view points and time: Source: Vogel et al Min Bai (UofT) Flow Estimation February 8, / 47
44 3D Piecewise Rigid Scene Model - Technique Method assumes scene is piecewise planar Fits parameters to each plane by energy minimization involving: Single reference image (2 cameras, 2 times), or more generally Entire image sequence terms (2 cameras, consistent motion through time) Source: Vogel et al Min Bai (UofT) Flow Estimation February 8, / 47
45 3D Piecewise Rigid Scene Model - Technique Single reference image (2 cameras, 2 times) Let P be the assignment of pixels to segments Let S be the assignment of segments to 3D moving plane E(P, S) = E D (P, S) + λe R (P, S) + µe S (P) + E V (P, S) To initialize inference, use output of other optical / stereo / scene ow algorithms Energy here essentially further renes ow by adding additional priors Min Bai (UofT) Flow Estimation February 8, / 47
46 3D Piecewise Rigid Scene Model - Technique Single reference image (2 cameras, 2 times) Let P be the assignment of pixels to segments Let S be the assignment of segments to 3D moving plane E(P, S) = E D (P, S) + λe R (P, S) + µe S (P) + E V (P, S) E D (P, S) encourages view consistency of planes between cameras and time Source: Vogel et al Min Bai (UofT) Flow Estimation February 8, / 47
47 3D Piecewise Rigid Scene Model - Technique Single reference image (2 cameras, 2 times) Let P be the assignment of pixels to segments Let S be the assignment of segments to 3D moving plane E(P, S) = E D (P, S) + λe R (P, S) + µe S (P) + E V (P, S) E R (P, S) builds in a prior favoring simpler scene geometry and motion Source: Vogel et al Min Bai (UofT) Flow Estimation February 8, / 47
48 3D Piecewise Rigid Scene Model - Technique Single reference image (2 cameras, 2 times) Let P be the assignment of pixels to segments Let S be the assignment of segments to 3D moving plane E(P, S) = E D (P, S) + λe R (P, S) + µe S (P) + E V (P, S) E S (P) sets maximum superpixel size so that scene model doesn't become overly simple Min Bai (UofT) Flow Estimation February 8, / 47
49 3D Piecewise Rigid Scene Model - Technique Single reference image (2 cameras, 2 times) Let P be the assignment of pixels to segments Let S be the assignment of segments to 3D moving plane E(P, S) = E D (P, S) + λe R (P, S) + µe S (P) + E V (P, S) E V (S) - based on estimation for changes in visibility of areas of image, penalize inconsistent motion proposals Min Bai (UofT) Flow Estimation February 8, / 47
50 3D Piecewise Rigid Scene Model - Technique Extension to View-Consistent Model (2 cameras, more than 2 times) Prior: objects don't suddenly teleport within sequence of frames Prior: objects have non-zero mass (no instantaneous acceleration, smooth velocity) Let P be the assignment of pixels to segments Let S be the assignment of segments to 3D moving plane E(P, S) = E VC D VC VC (P, S) + λe (P, S) + µe (P) R S Min Bai (UofT) Flow Estimation February 8, / 47
51 3D Piecewise Rigid Scene Model - Technique Extension to View-Consistent Model (2 cameras, more than 2 times) Let P be the assignment of pixels to segments Let S be the assignment of segments to 3D moving plane E(P, S) = E VC D VC VC (P, S) + λe (P, S) + µe (P) R E VC (P, S) now encourages: D View consistency between cameras + times (same as before) Plausibility of occlusions Fewer cases of moving out of bounds Fewer moving plane violations S Min Bai (UofT) Flow Estimation February 8, / 47
52 3D Piecewise Rigid Scene Model - Technique Extension to View-Consistent Model (2 cameras, more than 2 times) Let P be the assignment of pixels to segments Let S be the assignment of segments to 3D moving plane E VC (P, S) D E(P, S) = E VC D VC VC (P, S) + λe (P, S) + µe (P) R S Source: Vogel et al Min Bai (UofT) Flow Estimation February 8, / 47
53 3D Piecewise Rigid Scene Model - Sample Result Source: Min Bai (UofT) Flow Estimation Vogel et al February 8, / 47
54 Wrap-up Thanks! Questions? Suggestions for future directions? Min Bai (UofT) Flow Estimation February 8, / 47
Dense Correspondence and Its Applications
Dense Correspondence and Its Applications CVFX @ NTHU 28 May 2015 Outline Dense Correspondence Optical ow Epipolar Geometry Stereo Correspondence Why dense correspondence? Features are sparse Some applications
Computer Vision Lecture 17
Computer Vision Lecture 17 Epipolar Geometry & Stereo Basics 13.01.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar in the summer semester
Computer Vision Lecture 17
Announcements Computer Vision Lecture 17 Epipolar Geometry & Stereo Basics Seminar in the summer semester Current Topics in Computer Vision and Machine Learning Block seminar, presentations in 1 st week
Recap: Features and filters. Recap: Grouping & fitting. Now: Multiple views 10/29/2008. Epipolar geometry & stereo vision. Why multiple views?
Recap: Features and filters Epipolar geometry & stereo vision Tuesday, Oct 21 Kristen Grauman UT-Austin Transforming and describing images; textures, colors, edges Recap: Grouping & fitting Now: Multiple
EE795: Computer Vision and Intelligent Systems
EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 14 130307 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Stereo Dense Motion Estimation Translational
Depth from Stereo. Dominic Cheng February 7, 2018
Depth from Stereo Dominic Cheng February 7, 2018 Agenda 1. Introduction to stereo 2. Efficient Deep Learning for Stereo Matching (W. Luo, A. Schwing, and R. Urtasun. In CVPR 2016.) 3. Cascade Residual
Exploiting Semantic Information and Deep Matching for Optical Flow
Exploiting Semantic Information and Deep Matching for Optical Flow Min Bai, Wenjie Luo, Kaustav Kundu, Raquel Urtasun Department of Computer Science, University of Toronto {mbai, wenjie, kkundu, urtasun}@cs.toronto.edu
Depth from two cameras: stereopsis
Depth from two cameras: stereopsis Epipolar Geometry Canonical Configuration Correspondence Matching School of Computer Science & Statistics Trinity College Dublin Dublin 2 Ireland www.scss.tcd.ie Lecture
Dense 3D Reconstruction. Christiano Gava
Dense 3D Reconstruction Christiano Gava christiano.gava@dfki.de Outline Previous lecture: structure and motion II Structure and motion loop Triangulation Today: dense 3D reconstruction The matching problem
Lecture 14: Basic Multi-View Geometry
Lecture 14: Basic Multi-View Geometry Stereo If I needed to find out how far point is away from me, I could use triangulation and two views scene point image plane optical center (Graphic from Khurram
Depth from two cameras: stereopsis
Depth from two cameras: stereopsis Epipolar Geometry Canonical Configuration Correspondence Matching School of Computer Science & Statistics Trinity College Dublin Dublin 2 Ireland www.scss.tcd.ie Lecture
Stereo Wrap + Motion. Computer Vision I. CSE252A Lecture 17
Stereo Wrap + Motion CSE252A Lecture 17 Some Issues Ambiguity Window size Window shape Lighting Half occluded regions Problem of Occlusion Stereo Constraints CONSTRAINT BRIEF DESCRIPTION 1-D Epipolar Search
CS 4495 Computer Vision A. Bobick. Motion and Optic Flow. Stereo Matching
Stereo Matching Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix
Structure from Motion. Prof. Marco Marcon
Structure from Motion Prof. Marco Marcon Summing-up 2 Stereo is the most powerful clue for determining the structure of a scene Another important clue is the relative motion between the scene and (mono)
Today. Stereo (two view) reconstruction. Multiview geometry. Today. Multiview geometry. Computational Photography
Computational Photography Matthias Zwicker University of Bern Fall 2009 Today From 2D to 3D using multiple views Introduction Geometry of two views Stereo matching Other applications Multiview geometry
Fast Guided Global Interpolation for Depth and. Yu Li, Dongbo Min, Minh N. Do, Jiangbo Lu
Fast Guided Global Interpolation for Depth and Yu Li, Dongbo Min, Minh N. Do, Jiangbo Lu Introduction Depth upsampling and motion interpolation are often required to generate a dense, high-quality, and
Dense 3D Reconstruction. Christiano Gava
Dense 3D Reconstruction Christiano Gava christiano.gava@dfki.de Outline Previous lecture: structure and motion II Structure and motion loop Triangulation Wide baseline matching (SIFT) Today: dense 3D reconstruction
Vision 3D articielle Disparity maps, correlation
Vision 3D articielle Disparity maps, correlation Pascal Monasse monasse@imagine.enpc.fr IMAGINE, École des Ponts ParisTech http://imagine.enpc.fr/~monasse/stereo/ Contents Triangulation Epipolar rectication
There are many cues in monocular vision which suggests that vision in stereo starts very early from two similar 2D images. Lets see a few...
STEREO VISION The slides are from several sources through James Hays (Brown); Srinivasa Narasimhan (CMU); Silvio Savarese (U. of Michigan); Bill Freeman and Antonio Torralba (MIT), including their own
Stereo Vision. MAN-522 Computer Vision
Stereo Vision MAN-522 Computer Vision What is the goal of stereo vision? The recovery of the 3D structure of a scene using two or more images of the 3D scene, each acquired from a different viewpoint in
3D Computer Vision. Dense 3D Reconstruction II. Prof. Didier Stricker. Christiano Gava
3D Computer Vision Dense 3D Reconstruction II Prof. Didier Stricker Christiano Gava Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de
calibrated coordinates Linear transformation pixel coordinates
1 calibrated coordinates Linear transformation pixel coordinates 2 Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration with partial
Using Subspace Constraints to Improve Feature Tracking Presented by Bryan Poling. Based on work by Bryan Poling, Gilad Lerman, and Arthur Szlam
Presented by Based on work by, Gilad Lerman, and Arthur Szlam What is Tracking? Broad Definition Tracking, or Object tracking, is a general term for following some thing through multiple frames of a video
Stereo. 11/02/2012 CS129, Brown James Hays. Slides by Kristen Grauman
Stereo 11/02/2012 CS129, Brown James Hays Slides by Kristen Grauman Multiple views Multi-view geometry, matching, invariant features, stereo vision Lowe Hartley and Zisserman Why multiple views? Structure
Feature Transfer and Matching in Disparate Stereo Views through the use of Plane Homographies
Feature Transfer and Matching in Disparate Stereo Views through the use of Plane Homographies M. Lourakis, S. Tzurbakis, A. Argyros, S. Orphanoudakis Computer Vision and Robotics Lab (CVRL) Institute of
Multi-view stereo. Many slides adapted from S. Seitz
Multi-view stereo Many slides adapted from S. Seitz Beyond two-view stereo The third eye can be used for verification Multiple-baseline stereo Pick a reference image, and slide the corresponding window
Optical flow. Cordelia Schmid
Optical flow Cordelia Schmid Motion field The motion field is the projection of the 3D scene motion into the image Optical flow Definition: optical flow is the apparent motion of brightness patterns in
Multiple View Geometry
Multiple View Geometry CS 6320, Spring 2013 Guest Lecture Marcel Prastawa adapted from Pollefeys, Shah, and Zisserman Single view computer vision Projective actions of cameras Camera callibration Photometric
Epipolar Geometry and Stereo Vision
Epipolar Geometry and Stereo Vision Computer Vision Jia-Bin Huang, Virginia Tech Many slides from S. Seitz and D. Hoiem Last class: Image Stitching Two images with rotation/zoom but no translation. X x
Stereo. Outline. Multiple views 3/29/2017. Thurs Mar 30 Kristen Grauman UT Austin. Multi-view geometry, matching, invariant features, stereo vision
Stereo Thurs Mar 30 Kristen Grauman UT Austin Outline Last time: Human stereopsis Epipolar geometry and the epipolar constraint Case example with parallel optical axes General case with calibrated cameras
Motion Analysis. Motion analysis. Now we will talk about. Differential Motion Analysis. Motion analysis. Difference Pictures
Now we will talk about Motion Analysis Motion analysis Motion analysis is dealing with three main groups of motionrelated problems: Motion detection Moving object detection and location. Derivation of
The SIFT (Scale Invariant Feature
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor developed by David Lowe University of British Columbia Initial paper ICCV 1999 Newer journal paper IJCV 2004 Review: Matt Brown s Canonical
CS 4495 Computer Vision A. Bobick. Motion and Optic Flow. Stereo Matching
Stereo Matching Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix
CSE 252B: Computer Vision II
CSE 252B: Computer Vision II Lecturer: Serge Belongie Scribes: Jeremy Pollock and Neil Alldrin LECTURE 14 Robust Feature Matching 14.1. Introduction Last lecture we learned how to find interest points
Multi-stable Perception. Necker Cube
Multi-stable Perception Necker Cube Spinning dancer illusion, Nobuyuki Kayahara Multiple view geometry Stereo vision Epipolar geometry Lowe Hartley and Zisserman Depth map extraction Essential matrix
MAPI Computer Vision. Multiple View Geometry
MAPI Computer Vision Multiple View Geometry Geometry o Multiple Views 2- and 3- view geometry p p Kpˆ [ K R t]p Geometry o Multiple Views 2- and 3- view geometry Epipolar Geometry The epipolar geometry
CS664 Lecture #19: Layers, RANSAC, panoramas, epipolar geometry
CS664 Lecture #19: Layers, RANSAC, panoramas, epipolar geometry Some material taken from: David Lowe, UBC Jiri Matas, CMP Prague http://cmp.felk.cvut.cz/~matas/papers/presentations/matas_beyondransac_cvprac05.ppt
Depth. Common Classification Tasks. Example: AlexNet. Another Example: Inception. Another Example: Inception. Depth
Common Classification Tasks Recognition of individual objects/faces Analyze object-specific features (e.g., key points) Train with images from different viewing angles Recognition of object classes Analyze
Two-View Geometry (Course 23, Lecture D)
Two-View Geometry (Course 23, Lecture D) Jana Kosecka Department of Computer Science George Mason University http://www.cs.gmu.edu/~kosecka General Formulation Given two views of the scene recover the
But First: Multi-View Projective Geometry
View Morphing (Seitz & Dyer, SIGGRAPH 96) Virtual Camera Photograph Morphed View View interpolation (ala McMillan) but no depth no camera information Photograph But First: Multi-View Projective Geometry
arxiv: v1 [cs.cv] 28 Sep 2018
![arxiv: v1 [cs.cv] 28 Sep 2018](/thumbs/93/113542646.jpg "arxiv: v1 [cs.cv] 28 Sep 2018")
Camera Pose Estimation from Sequence of Calibrated Images arxiv:1809.11066v1 [cs.cv] 28 Sep 2018 Jacek Komorowski 1 and Przemyslaw Rokita 2 1 Maria Curie-Sklodowska University, Institute of Computer Science,
Reminder: Lecture 20: The Eight-Point Algorithm. Essential/Fundamental Matrix. E/F Matrix Summary. Computing F. Computing F from Point Matches
Reminder: Lecture 20: The Eight-Point Algorithm F = -0.00310695-0.0025646 2.96584-0.028094-0.00771621 56.3813 13.1905-29.2007-9999.79 Readings T&V 7.3 and 7.4 Essential/Fundamental Matrix E/F Matrix Summary
Lecture 10 Multi-view Stereo (3D Dense Reconstruction) Davide Scaramuzza
Lecture 10 Multi-view Stereo (3D Dense Reconstruction) Davide Scaramuzza REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time, ICRA 14, by Pizzoli, Forster, Scaramuzza [M. Pizzoli, C. Forster,
Stereo Epipolar Geometry for General Cameras. Sanja Fidler CSC420: Intro to Image Understanding 1 / 33
Stereo Epipolar Geometry for General Cameras Sanja Fidler CSC420: Intro to Image Understanding 1 / 33 Stereo Epipolar geometry Case with two cameras with parallel optical axes General case Now this Sanja
Learning Two-View Stereo Matching
Learning Two-View Stereo Matching Jianxiong Xiao Jingni Chen Dit-Yan Yeung Long Quan Department of Computer Science and Engineering The Hong Kong University of Science and Technology The 10th European
Fundamental Matrix & Structure from Motion
Fundamental Matrix & Structure from Motion Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps Today Transformations between images Structure from Motion The Essential Matrix The Fundamental
Massachusetts Institute of Technology Department of Computer Science and Electrical Engineering 6.801/6.866 Machine Vision QUIZ II
Massachusetts Institute of Technology Department of Computer Science and Electrical Engineering 6.801/6.866 Machine Vision QUIZ II Handed out: 001 Nov. 30th Due on: 001 Dec. 10th Problem 1: (a (b Interior
EE795: Computer Vision and Intelligent Systems
EE795: Computer Vision and Intelligent Systems Spring 2012 TTh 17:30-18:45 FDH 204 Lecture 12 130228 http://www.ee.unlv.edu/~b1morris/ecg795/ 2 Outline Review Panoramas, Mosaics, Stitching Two View Geometry
Chaplin, Modern Times, 1936
Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection set] Multi-view geometry problems Structure: Given projections
C18 Computer vision. C18 Computer Vision. This time... Introduction. Outline.
C18 Computer Vision. This time... 1. Introduction; imaging geometry; camera calibration. 2. Salient feature detection edges, line and corners. 3. Recovering 3D from two images I: epipolar geometry. C18
Lecture 6 Stereo Systems Multi-view geometry
Lecture 6 Stereo Systems Multi-view geometry Professor Silvio Savarese Computational Vision and Geometry Lab Silvio Savarese Lecture 6-5-Feb-4 Lecture 6 Stereo Systems Multi-view geometry Stereo systems
Range Imaging Through Triangulation. Range Imaging Through Triangulation. Range Imaging Through Triangulation. Range Imaging Through Triangulation
Obviously, this is a very slow process and not suitable for dynamic scenes. To speed things up, we can use a laser that projects a vertical line of light onto the scene. This laser rotates around its vertical
Machine vision. Summary # 11: Stereo vision and epipolar geometry. u l = λx. v l = λy
1 Machine vision Summary # 11: Stereo vision and epipolar geometry STEREO VISION The goal of stereo vision is to use two cameras to capture 3D scenes. There are two important problems in stereo vision:
What have we leaned so far?
What have we leaned so far? Camera structure Eye structure Project 1: High Dynamic Range Imaging What have we learned so far? Image Filtering Image Warping Camera Projection Model Project 2: Panoramic
3D Sensing and Reconstruction Readings: Ch 12: , Ch 13: ,
3D Sensing and Reconstruction Readings: Ch 12: 12.5-6, Ch 13: 13.1-3, 13.9.4 Perspective Geometry Camera Model Stereo Triangulation 3D Reconstruction by Space Carving 3D Shape from X means getting 3D coordinates
Structure from Motion. Introduction to Computer Vision CSE 152 Lecture 10
Structure from Motion CSE 152 Lecture 10 Announcements Homework 3 is due May 9, 11:59 PM Reading: Chapter 8: Structure from Motion Optional: Multiple View Geometry in Computer Vision, 2nd edition, Hartley
Learning and Inferring Depth from Monocular Images. Jiyan Pan April 1, 2009
Learning and Inferring Depth from Monocular Images Jiyan Pan April 1, 2009 Traditional ways of inferring depth Binocular disparity Structure from motion Defocus Given a single monocular image, how to infer
Stereo CSE 576. Ali Farhadi. Several slides from Larry Zitnick and Steve Seitz
Stereo CSE 576 Ali Farhadi Several slides from Larry Zitnick and Steve Seitz Why do we perceive depth? What do humans use as depth cues? Motion Convergence When watching an object close to us, our eyes
Augmented Reality VU. Computer Vision 3D Registration (2) Prof. Vincent Lepetit
Augmented Reality VU Computer Vision 3D Registration (2) Prof. Vincent Lepetit Feature Point-Based 3D Tracking Feature Points for 3D Tracking Much less ambiguous than edges; Point-to-point reprojection
Fundamental matrix. Let p be a point in left image, p in right image. Epipolar relation. Epipolar mapping described by a 3x3 matrix F
Fundamental matrix Let p be a point in left image, p in right image l l Epipolar relation p maps to epipolar line l p maps to epipolar line l p p Epipolar mapping described by a 3x3 matrix F Fundamental
first order approx. u+d second order approx. (S)
Computing Dierential Properties of 3-D Shapes from Stereoscopic Images without 3-D Models F. Devernay and O. D. Faugeras INRIA. 2004, route des Lucioles. B.P. 93. 06902 Sophia-Antipolis. FRANCE. Abstract
Application questions. Theoretical questions
The oral exam will last 30 minutes and will consist of one application question followed by two theoretical questions. Please find below a non exhaustive list of possible application questions. The list
1 (5 max) 2 (10 max) 3 (20 max) 4 (30 max) 5 (10 max) 6 (15 extra max) total (75 max + 15 extra)
Mierm Exam CS223b Stanford CS223b Computer Vision, Winter 2004 Feb. 18, 2004 Full Name: Email: This exam has 7 pages. Make sure your exam is not missing any sheets, and write your name on every page. The
Final project bits and pieces
Final project bits and pieces The project is expected to take four weeks of time for up to four people. At 12 hours per week per person that comes out to: ~192 hours of work for a four person team. Capstone:
EXAM SOLUTIONS. Image Processing and Computer Vision Course 2D1421 Monday, 13 th of March 2006,
School of Computer Science and Communication, KTH Danica Kragic EXAM SOLUTIONS Image Processing and Computer Vision Course 2D1421 Monday, 13 th of March 2006, 14.00 19.00 Grade table 0-25 U 26-35 3 36-45
Object Scene Flow. Moritz Menze a,, Christian Heipke a, Andreas Geiger b,c. Abstract
Object Scene Flow Moritz Menze a,, Christian Heipke a, Andreas Geiger b,c a Institute of Photogrammetry and GeoInformation, Leibniz Universität Hannover, Nienburger Str. 1, D-30167 Hannover, Germany b
Fundamental Matrix & Structure from Motion
Fundamental Matrix & Structure from Motion Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps Today Review of Assignment 0. Transformations between images Structure from Motion The Essential
Stereo imaging ideal geometry
Stereo imaging ideal geometry (X,Y,Z) Z f (x L,y L ) f (x R,y R ) Optical axes are parallel Optical axes separated by baseline, b. Line connecting lens centers is perpendicular to the optical axis, and
Stereo Vision II: Dense Stereo Matching
Stereo Vision II: Dense Stereo Matching Nassir Navab Slides prepared by Christian Unger Outline. Hardware. Challenges. Taxonomy of Stereo Matching. Analysis of Different Problems. Practical Considerations.
Image Rectification (Stereo) (New book: 7.2.1, old book: 11.1)
Image Rectification (Stereo) (New book: 7.2.1, old book: 11.1) Guido Gerig CS 6320 Spring 2013 Credits: Prof. Mubarak Shah, Course notes modified from: http://www.cs.ucf.edu/courses/cap6411/cap5415/, Lecture
Lecture 10 Dense 3D Reconstruction
Institute of Informatics Institute of Neuroinformatics Lecture 10 Dense 3D Reconstruction Davide Scaramuzza 1 REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time M. Pizzoli, C. Forster,
Epipolar Geometry CSE P576. Dr. Matthew Brown
Epipolar Geometry CSE P576 Dr. Matthew Brown Epipolar Geometry Epipolar Lines, Plane Constraint Fundamental Matrix, Linear solution + RANSAC Applications: Structure from Motion, Stereo [ Szeliski 11] 2
The end of affine cameras
The end of affine cameras Affine SFM revisited Epipolar geometry Two-view structure from motion Multi-view structure from motion Planches : http://www.di.ens.fr/~ponce/geomvis/lect3.pptx http://www.di.ens.fr/~ponce/geomvis/lect3.pdf
Lecture'9'&'10:'' Stereo'Vision'
Lecture'9'&'10:'' Stereo'Vision' Dr.'Juan'Carlos'Niebles' Stanford'AI'Lab' ' Professor'FeiAFei'Li' Stanford'Vision'Lab' 1' Dimensionality'ReducIon'Machine'(3D'to'2D)' 3D world 2D image Point of observation
Visual Odometry. Features, Tracking, Essential Matrix, and RANSAC. Stephan Weiss Computer Vision Group NASA-JPL / CalTech
Visual Odometry Features, Tracking, Essential Matrix, and RANSAC Stephan Weiss Computer Vision Group NASA-JPL / CalTech Stephan.Weiss@ieee.org (c) 2013. Government sponsorship acknowledged. Outline The
Rectification and Disparity
Rectification and Disparity Nassir Navab Slides prepared by Christian Unger What is Stereo Vision? Introduction A technique aimed at inferring dense depth measurements efficiently using two cameras. Wide
Manhattan-world Stereo and Surface Reconstruction
Manhattan-world Stereo and Surface Reconstruction Kuan-Ting Yu CSAIL MIT Cambridge, USA peterkty@csail.mit.edu Abstract Depth estimation from 2D images has been extensively studied in the computer vision
Feature Tracking and Optical Flow
Feature Tracking and Optical Flow Prof. D. Stricker Doz. G. Bleser Many slides adapted from James Hays, Derek Hoeim, Lana Lazebnik, Silvio Saverse, who 1 in turn adapted slides from Steve Seitz, Rick Szeliski,
Stereo and Epipolar geometry
Previously Image Primitives (feature points, lines, contours) Today: Stereo and Epipolar geometry How to match primitives between two (multiple) views) Goals: 3D reconstruction, recognition Jana Kosecka
CS223b Midterm Exam, Computer Vision. Monday February 25th, Winter 2008, Prof. Jana Kosecka
CS223b Midterm Exam, Computer Vision Monday February 25th, Winter 2008, Prof. Jana Kosecka Your name email This exam is 8 pages long including cover page. Make sure your exam is not missing any pages.
Epipolar Geometry and Stereo Vision
Epipolar Geometry and Stereo Vision Computer Vision Shiv Ram Dubey, IIIT Sri City Many slides from S. Seitz and D. Hoiem Last class: Image Stitching Two images with rotation/zoom but no translation. X
Fundamentals of Stereo Vision Michael Bleyer LVA Stereo Vision
Fundamentals of Stereo Vision Michael Bleyer LVA Stereo Vision What Happened Last Time? Human 3D perception (3D cinema) Computational stereo Intuitive explanation of what is meant by disparity Stereo matching
3D Geometry and Camera Calibration
3D Geometry and Camera Calibration 3D Coordinate Systems Right-handed vs. left-handed x x y z z y 2D Coordinate Systems 3D Geometry Basics y axis up vs. y axis down Origin at center vs. corner Will often
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry Topics: Basics of projective geometry Points and hyperplanes in projective space Homography Estimating homography from point correspondence
Real-Time Simultaneous 3D Reconstruction and Optical Flow Estimation
Real-Time Simultaneous 3D Reconstruction and Optical Flow Estimation Menandro Roxas Takeshi Oishi Institute of Industrial Science, The University of Tokyo roxas, oishi @cvl.iis.u-tokyo.ac.jp Abstract We
SPM-BP: Sped-up PatchMatch Belief Propagation for Continuous MRFs. Yu Li, Dongbo Min, Michael S. Brown, Minh N. Do, Jiangbo Lu
SPM-BP: Sped-up PatchMatch Belief Propagation for Continuous MRFs Yu Li, Dongbo Min, Michael S. Brown, Minh N. Do, Jiangbo Lu Discrete Pixel-Labeling Optimization on MRF 2/37 Many computer vision tasks
On-line and Off-line 3D Reconstruction for Crisis Management Applications
On-line and Off-line 3D Reconstruction for Crisis Management Applications Geert De Cubber Royal Military Academy, Department of Mechanical Engineering (MSTA) Av. de la Renaissance 30, 1000 Brussels geert.de.cubber@rma.ac.be
Step-by-Step Model Buidling
Step-by-Step Model Buidling Review Feature selection Feature selection Feature correspondence Camera Calibration Euclidean Reconstruction Landing Augmented Reality Vision Based Control Sparse Structure
Lecture 19: Motion. Effect of window size 11/20/2007. Sources of error in correspondences. Review Problem set 3. Tuesday, Nov 20
Lecture 19: Motion Review Problem set 3 Dense stereo matching Sparse stereo matching Indexing scenes Tuesda, Nov 0 Effect of window size W = 3 W = 0 Want window large enough to have sufficient intensit
Final Review CMSC 733 Fall 2014
Final Review CMSC 733 Fall 2014 We have covered a lot of material in this course. One way to organize this material is around a set of key equations and algorithms. You should be familiar with all of these,
A Patch Prior for Dense 3D Reconstruction in Man-Made Environments
A Patch Prior for Dense 3D Reconstruction in Man-Made Environments Christian Häne 1, Christopher Zach 2, Bernhard Zeisl 1, Marc Pollefeys 1 1 ETH Zürich 2 MSR Cambridge October 14, 2012 A Patch Prior for
Chapter 9 Object Tracking an Overview
Chapter 9 Object Tracking an Overview The output of the background subtraction algorithm, described in the previous chapter, is a classification (segmentation) of pixels into foreground pixels (those belonging
BIL Computer Vision Apr 16, 2014
BIL 719 - Computer Vision Apr 16, 2014 Binocular Stereo (cont d.), Structure from Motion Aykut Erdem Dept. of Computer Engineering Hacettepe University Slide credit: S. Lazebnik Basic stereo matching algorithm
Computational Optical Imaging - Optique Numerique. -- Single and Multiple View Geometry, Stereo matching --
Computational Optical Imaging - Optique Numerique -- Single and Multiple View Geometry, Stereo matching -- Autumn 2015 Ivo Ihrke with slides by Thorsten Thormaehlen Reminder: Feature Detection and Matching
Previously. Part-based and local feature models for generic object recognition. Bag-of-words model 4/20/2011
Previously Part-based and local feature models for generic object recognition Wed, April 20 UT-Austin Discriminative classifiers Boosting Nearest neighbors Support vector machines Useful for object recognition
Local qualitative shape from stereo. without detailed correspondence. Extended Abstract. Shimon Edelman. Internet:
Local qualitative shape from stereo without detailed correspondence Extended Abstract Shimon Edelman Center for Biological Information Processing MIT E25-201, Cambridge MA 02139 Internet: edelman@ai.mit.edu
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry Topics: Basics of projective geometry Points and hyperplanes in projective space Homography Estimating homography from point correspondence
Rectification and Distortion Correction
Rectification and Distortion Correction Hagen Spies March 12, 2003 Computer Vision Laboratory Department of Electrical Engineering Linköping University, Sweden Contents Distortion Correction Rectification
Mosaics. Today s Readings
Mosaics VR Seattle: http://www.vrseattle.com/ Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html Today s Readings Szeliski and Shum paper (sections
Multi-View Stereo for Static and Dynamic Scenes
Multi-View Stereo for Static and Dynamic Scenes Wolfgang Burgard Jan 6, 2010 Main references Yasutaka Furukawa and Jean Ponce, Accurate, Dense and Robust Multi-View Stereopsis, 2007 C.L. Zitnick, S.B.