3D Sensing and Reconstruction Readings: Ch 12: 12.5-6, Ch 13: 13.1-3, 13.9.4 Perspective Geometry Camera Model Stereo Triangulation 3D Reconstruction by Space Carving
3D Shape from X means getting 3D coordinates from different methods shading silhouette texture stereo light striping motion mainly research used in practice
Perspective Imaging Model: 1D real image point z c E x i O f x f camera lens D image of point B in front image This is the axis of the real image plane. O is the center of projection. This is the axis of the front image plane, which we use. x i x = x c c B f 3D object point z c
Perspective in 2D Y c 3D object point P=(x c,y c, z c ) =(x w,y w,z w ) (Simplified) ray x i F P =(x i,y i,f) y i f camera X c y c optical axis z w =z c Z c x c Here camera coordinates equal world coordinates. x i f y i f = = x c z c y c z c x i = (f/z c )x c yi = (f/z c )y c
3D from Stereo 3D point left image right image disparity: the difference in image location of the same 3D point when projected under perspective to two different cameras. d = x left - x right
Depth Perception from Stereo Simple Model: Parallel Optic Axes camera L f image plane z Z x l baseline camera R b f x r x-b X P=(x,z) z f = x x l z f = x-b x r z = y = y f y l y r y-axis is perpendicular to the page.
Resultant Depth Calculation For stereo cameras with parallel optical axes, focal length f, baseline b, corresponding image points (x l,y l ) and (x r,y r ) with disparity d: z = f*b / (x l - x r ) = f*b/d x = x l *z/f or b + x r *z/f y = y l *z/f or y r *z/f This method of determining depth from disparity is called triangulation.
Finding Correspondences If the correspondence is correct, triangulation works VERY well. But correspondence finding is not perfectly solved. (What methods have we studied?) For some very specific applications, it can be solved for those specific kind of images, e.g. windshield of a car.
3 Main Matching Methods 1. Cross correlation using small windows. dense 2. Symbolic feature matching, usually using segments/corners. sparse 3. Use the newer interest operators, ie. SIFT. sparse
Epipolar Geometry Constraint: 1. Normal Pair of Images The epipolar plane cuts through the image plane(s) forming 2 epipolar lines. P y 1 y 2 P 1 P 2 C 1 b C 2 z 1 z 2 The match for P 1 (or P 2 ) in the other image, must lie on the same epipolar line. x epipolar plane
Epipolar Geometry: General Case P y 1 y 2 P P 2 1 e 2 x 2 e 1 x 1 C 1 C2
Constraints 1. Epipolar Constraint: Matching points lie on corresponding epipolar lines. P Q 2. Ordering Constraint: Usually in the same order across the lines. e 1 e 2 C 1 C2
Structured Light 3D data can also be derived using light stripe a single camera a light source that can produce stripe(s) on the 3D object light source camera
Structured Light 3D Computation 3D data can also be derived using a single camera 3D point (x, y, z) a light source that can produce stripe(s) on the 3D object b [x y z] = --------------- [x y f] 3D f cot θ - x image light source θ b f (x,y,f) (0,0,0) x axis
Depth from Multiple Light Stripes What are these objects?
Our (former) System 4-camera light-striping stereo cameras projector rotation table 3D object
Camera Model: Recall there are 5 Different Frames of Reference y c Object C y f x c z w World image a x f Camera Real Image Pixel Image z c W A z p pyramid y p object y w x w x p
The Camera Model How do we get an image point IP from a world point P? s IP r s IP c s = c 11 c 12 c 13 c 14 c 21 c 22 c 23 c 24 c 31 c 32 c 33 1 P x P y P z 1 image point camera matrix C What s in C? world point
The camera model handles the rigid body transformation from world coordinates to camera coordinates plus the perspective transformation to image coordinates. 1. CP = T R WP 2. FP = π(f) CP Why is there not a scale factor here? s FP x s FP y s FP z s image point = 1 0 0 0 0 1 0 0 0 0 1 0 0 0 1/f 0 perspective transformation CP x CP y CP z 1 3D point in camera coordinates
Camera Calibration In order work in 3D, we need to know the parameters of the particular camera setup. Solving for the camera parameters is called calibration. y w C y c x c intrinsic parameters are of the camera device z w W x w z c extrinsic parameters are where the camera sits in the world
Intrinsic Parameters principal point (u 0,v 0 ) scale factors (d x,d y ) C f aspect ratio distortion factor γ (u 0,v 0 ) focal length f lens distortion factor κ (models radial lens distortion)
Extrinsic Parameters translation parameters t = [t x t y t z ] rotation matrix R = r 11 r 12 r 13 0 r 21 r 22 r 23 0 r 31 r 32 r 33 0 0 0 0 1 Are there really nine parameters?
Calibration Object The idea is to snap images at different depths and get a lot of 2D-3D point correspondences.
The Tsai Procedure The Tsai procedure was developed by Roger Tsai at IBM Research and is most widely used. Several images are taken of the calibration object yielding point correspondences at different distances. Tsai s algorithm requires n > 5 correspondences {(x i, y i, z i ), (u i, v i )) i = 1,,n} between (real) image points and 3D points. Lots of details in Chapter 13.
We use the camera parameters of each camera for general stereo. P y 1 y 2 P 1 =(r 1,c 1 ) B P 2 =(r 2,c 2 ) e 2 x 2 e 1 x 1 C
For a correspondence (r 1,c 1 ) in image 1 to (r2,c2) in image 2: 1. Both cameras were calibrated. Both camera matrices are then known. From the two camera equations B and C we get 4 linear equations in 3 unknowns. r 1 = (b 11 - b 31 *r 1 )x + (b 12 - b 32 *r 1 )y + (b 13 -b 33 *r 1 )z c 1 = (b 21 - b 31 *c1)x + (b 22 - b 32 *c 1 )y + (b 23 -b 33 *c 1 )z r 2 = (c 11 - c 31 *r 2) x + (c 12 - c 32 *r 2 )y + (c 13 - c 33 *r 2 )z c 2 = (c 21 - c 31 *c 2 )x + (c 22 - c 32 *c 2 )y + (c 23 - c 33 *c 2 )z Direct solution uses 3 equations, won t give reliable results.
Solve by computing the closest approach of the two skew rays. V = (P 1 + a 1 *u 1 ) (Q 1 + a 2 *u 2 ) P 1 u 1 (P 1 + a 1 *u 1 ) (Q 1 + a 2 *u 2 ) u 1 = 0 (P 1 + a 1 *u 1 ) (Q 1 + a 2 *u 2 ) u 2 = 0 solve for shortest V P Q 1 u 2 If the rays intersected perfectly in 3D, the intersection would be P. Instead, we solve for the shortest line segment connecting the two rays and let P be its midpoint.