Depth from two cameras: stereopsis Epipolar Geometry Canonical Configuration Correspondence Matching School of Computer Science & Statistics Trinity College Dublin Dublin 2 Ireland www.scss.tcd.ie Lecture Name Course Name 1 1
Human Stereo Vision Faking Stereo Vision Stereoscope Random Dot Stereograms Stereo Glasses Head mounted VR display Emulating Stereo Vision Two cameras Active robot head Lecture Name Computer Vision Lecture 4 2 2
Basics of depth extraction Single Camera calibrated intrinsics extrinsics inverse perspective transform Two Cameras Calibration extrinsics matching points correspondence problem intersection of rays (maybe) depth / disparity calculation Lecture Name Computer Vision Lecture 4 3 3
Geometry of Stereo Baseline Connecting Camera Centers Epipolar constraint a point in one image must lie along a line in the other: l and l Image point and camera centers define a plane All planes pass through the epipoles Canonical configuration places epipoles at infinity Lecture Name Computer Vision Lecture 4 4 4
Epipolar Constraint Lecture Name Computer Vision Lecture 4 5 5
Essential Matrix Translation and Rotation between two cameras when we know the camera calibration Essential Matrix 9 coefficients Parameterised by 3 DoF of R and 2 DoF of t Relationship between points p[t (Rp )] p Ep =0where E =[t x ]R and [a x ]b = a b Lecture Name Computer Vision Lecture 4 6 6
Fundamental Matrix Unknown camera calibration Fundamental Matrix Has 7 parameters Parameterised by 4 numbers (a,b,c,d) p = K p, p = K p p T Fp =0, where F = K T EK 1 e =(α, β) T and e =(α, β ) T Lecture Name Computer Vision Lecture 4 7 7
Canonical configuration baseline parallel to image plane epipolar lines parallel image rectification to place images into canonical configuration search for correspondences along raster lines Andrea Fusiello, University of Verona (Reproduced with permission) Lecture Name Computer Vision Lecture 4 8 8
Calculating Depth Point in 3D space P(x,y,z) Clear disparity in the views in both cameras: Pl and Pr simple geometry can be used to give depth Z Lecture Name Computer Vision Lecture 4 9 9
Problem Solved? Stereo Assumptions see point in both images unique solution to correspondence problem Problems (Self) Occlusion Non-unique mappings Lecture Name Computer Vision Lecture 4 10 10
Correspondence problem A key problem in computer vision one of these things looks a lot like the other Why should this be hard differences of view point specularities subtle scale changes Applying Constraints make this easier epipolar uniqueness pixel compatibility Lecture Name Computer Vision Lecture 4 11 8 11
Other Constraints Disparity Smoothness Constraint with two scene points close to each other p & q Threshold = ( pl-pr - ql-qr ) Figural Disparity Constraint if an edge element: must be on one in both images Disparity limit constraint human visual system can only resolve stereo images if disparity is less than a threshold: sets a limit to the disparity search Lecture Name Computer Vision Lecture 4 12 9 12
Paradigms to solve correspondence Correlation based - bottom up approaches Psychology > humans do not use monocular features to match Block Matching: 5X5, 7x7,... Matching Criteria: SSD, NCC, etc Other constraints must be used to produce good matching pyramidal approaches; use of edge info at finer resolutions Lecture Name Computer Vision Lecture 4 13 10 13
Correlation based approaches Lecture Name Computer Vision Lecture 4 14 14
Feature Based Approaches Lecture Name Computer Vision Lecture 4 15 15
One Feature Based Approach PMF Stereo Algorithm Pollard, Mayhew and Frisby assumes features have been extracted correspondence pairs are generated 3 constraints used epipolar uniqueness disparity gradient limit Lecture Name Computer Vision Lecture 4 16 13 16
Separation S(), disparity difference D() Lecture Name Computer Vision Lecture 4 17 17
Definitions from PMF paper http:// homepages.inf.ed.ac.uk/rbf/books/ MAYHEW/scan/ 11-24.pdf Lecture Name Computer Vision Lecture 4 18 18
PMF Algorithm from Sonka Lecture Name Computer Vision Lecture 4 19 19
Disparity Map / Depth Image Lecture Name Computer Vision Lecture 4 20 20
Disparity Map / Depth Image Lecture Name Computer Vision Lecture 4 21 21