Camera Calibration Jesus J Caban Schedule! Today:! Camera calibration! Wednesday:! Lecture: Motion & Optical Flow! Monday:! Lecture: Medical Imaging! Final presentations:! Nov 29 th : W. Griffin! Dec 1 st : F. Zafar, Y. Wang, and N. Chhaya! Dec 6 th : J. Dandois! Dec 8 th :! Dec 13 th :! Dec 20 th : +5 bonus points +3 bonus points +2 bonus points +1 bonus points +0 bonus points +0 bonus points Note: You have until next Monday to let me know
Camera Calibration: Motivation! Images are obtained from 3D scenes! The exact position and orientation of the camera sensing device are often unknown! Camera needs to be related to some global frame / reference to be able to:! Get accurate measurements! Inspect materials! Etc Camera Model! Camera model:! from world to camera (coordinate systems)! from camera (3D) to image frame (2D) x im (x im,y im ) O y x y im p P P w Z w X w Y w
Camera Calibration! Goal: build the geometric relations between a 3D scene and its 2D images.! 3D transformation that represents the viewpoints and viewing directions of the camera! The perspective projection that maps 3D points into 2D images Camera Model: Parameters! Camera Parameters! Intrinsic Parameters (of the camera): link the frame coordinates of an image point with its corresponding camera coordinates! Extrinsic parameters: define the location and orientation of the camera coordinate system with respect to the world coordinate system x im (x im,y im ) O y x y im p P P w Z w X w Y w
Transformations 2-D Rotation (x, y ) (x, y)! x = x cos(!) - y sin(!) y = x sin(!) + y cos(!)
2D Rotation! Rotation by angle! (about the origin)!"#"%!"#"%!% &()%*+%),%*-.,/+,0% 12/%/2)(32-+4% Scaling! Scaling operation:! Or, in matrix form: " x % " = a 0 %" # y & # 0 b& x % # y & scaling matrix S
Scaling! Scaling a coordinate means multiplying each of its components by a scalar! Uniform scaling means this scalar is the same for all components: " 2 Scaling! Uniform scaling by s:!"#"%!"#"%
Non-uniform Scaling! Non-uniform scaling: different scalars per component: X " 2, Y " 0.5 2x2 Matrices! What types of transformations can be represented with a 2x2 matrix? 2D Mirror about Y axis? 2D Mirror over (0,0)?
All 2D Linear Transformations! Linear transformations are combinations of! Scale,! Rotation,! Shear, and! Mirror! Properties of linear transformations:! Origin maps to origin! Lines map to lines! Parallel lines remain parallel! Ratios are preserved! Closed under composition What about 2D Translation? 56%7/(-+8(32-0%! Translation is not a liner operation on 2D coordinates! Homogeneous coordinates
Homogeneous coordinates 7/*9:4%%(;;%2-,%<2/,%922/;*-(),4% B%!%#%&#%% 2<2=,-,2>+%F8(-,% 2<2=,-,2>+%*<(=,%% 922/;*-(),+% A% %C%D%!%E#%&E#%(% @%?2-.,/3-=%!"#%2<2=,-,2>+%922/;*-(),+% Homogeneous Coordinates! Add a 3rd coordinate to every 2D point! (x, y, w) represents a point at location (x/w, y/w)! (x, y, 0) represents a point at infinity! (0, 0, 0) is not allowed y 2 (2,1,1) or (4,2,2) or (6,3,3) Convenient coordinate 1 system to represent many useful x transformations 1 2
2D Translation?! Solution: homogeneous coordinates to the rescue Translation! Example of translation Homogeneous Coordinates " x % " 1 0 t x %" x% y = 0 1 t y y # 1 & # 0 0 1 & # 1& " x + t x % = y + t y # 1 & t x = 2 t y = 1
Affine transformations (-A%)/(-+G2/<(32-%B*)%% 8(+)%/2B%H%"%"%D%I%B,%9(88%(-%% )*+,%)/(-+G2/<(32-- Basic affine transformations Translate Scale 2D in-plane rotation Shear
3D Rotations 1 0 0 & ) X " = & 0 cos" #sin") %& 0 sin" cos" () cos" 0 sin" & ) Y " = & 0 1 0 ) %& #sin" 0 cos" () cos" #sin" 0 & ) Z " = & sin" cos" 0) %& 0 0 1( ) Where do we go from here? (J-,%)/(-+G2/<(32-% B()%(FF,-+%B,-%B,% 9(-=,%)*+%/2B0-
Projective Transformations?(88,;%(%.##/")0.&%%!2/%01)+)"-0,"20,345,-)0- Image warping with homographies image plane in front image plane below black area where no pixel maps to
Camera Calibration Camera Model: Parameters! Camera Parameters! Intrinsic Parameters (of the camera): link the frame coordinates of an image point with its corresponding camera coordinates! Extrinsic parameters: define the location and orientation of the camera coordinate system with respect to the world coordinate system x im (x im,y im ) O y x y im p P P w Z w X w Y w
Camera coordinate system Principal axis: line from the camera center perpendicular to the image plane Principal point (p): point where principal axis intersects the image plane (origin of normalized coordinate system) Perspective Projection Equations (X-Z Plane) X x f Z "x f = X Z "x = f X Z Notice: Position on the image plane is related to depth. (u,v) are scaled equally based on focal length and depth
Equivalent Image Geometries! Consider case with object on optical axis! More convenient geometry, with an upright image! Both are equivalent mathematically x = f X Z (X,Y,Z)! ( f X Z, f Y Z ) X Y Z 1 " # % &! f X f Y Z " # % & Pinhole camera model X Y Z 1 " # % & f 0 f 0 1 0 " # % & =
Principal point offset principal point: Camera coordinate system: origin is at the principal point Image coordinate system: origin is in the corner Principal point offset principal point: (X,Y,Z)! ( f X Z + p x, f Y Z + p y )
Pixel size Pixel size:! m x pixels per meter in horizontal direction, m y pixels per meter in vertical direction pixels/m m pixels Skew The camera coordinate system may be skewed due to some manufacturing error.
Lens distortion! Two main lens distortion:! Radial distortion: arise as a result of the shape of lens (e.g. fish-eye)! Tangential distortion: arise from the assembly process of the camera as a whole. Radial Distortion! Zero at the optical center of the image! Increases as we move toward the periphery! Rays farther from the center of the lens are bent more than those closer in! Terms: k1, k2, k3
Intrinsic parameters! Principal point coordinates! Focal length! Pixel magnification factors! Skew (non-rectangular pixels)! Radial distortion Camera parameters Extrinsic parameters! Rotation and translation relative to world coordinate system Camera rotation and translation In general, the camera coordinate frame will be related to the world coordinate frame by a rotation and a translation ( ) X cam = R X - C coords. of point in camera frame coords. of a point in world frame (nonhomogeneous) coords. of camera center in world frame
Intrinsic parameters! Principal point coordinates! Focal length! Pixel magnification factors! Skew (non-rectangular pixels)! Radial distortion Camera parameters Extrinsic parameters! Rotation and translation relative to world coordinate system Camera Calibration Basic matrix: " 1 0 0 0% " 1 0 0 t 1 %" R 11 R 12 R 13 0% " 1 0 0 T 1 % 0 1 0 0 G = PLRT = 0 1 0 t 2 R 21 R 22 R 23 0 0 1 0 T 2 0 0 1 0 0 0 1 t 3 R 31 R 32 R 33 0 0 0 1 T 3 # 0 0 1/ f 0& # 0 0 0 1 &# 0 0 0 1& # 0 0 0 1 & Intrinsic Parameters Extrinsic Parameters How to estimate those parameters?
Camera calibration Given n points with known 3D coordinates X i and known image projections x i, estimate the camera parameters X i x i Example Calibration Pattern Harris Corner Detector From Sebastian Thrun 44
Camera calibration Calibration: 2 steps! Step 1: Transform into camera coordinates " X C % " X W % Y C # Z C = f ( Y W & Z W,(,),*,T) # &! Step 2: Transform into image coordinates 46
Calibration Model (extrinsic) 47 Advantage of Homogeneous C s i-th data point 48
Camera Calibration! Algorithm 1. Measure N 3D coordinates (Xi, Yi,Zi) 2. Locate their corresponding image points (xi,yi) - Edge, Corner, Hough 3. Build matrix A of a homogeneous system Av = 0 4. Compute SVD of A, solution v 5. Determine aspect ratio # and scale 6. Recover the first two rows of R and the first two components of T up to a sign 7. Determine sign s of by checking the projection equation 8. Compute the 3 rd row of R by vector product, and enforce orthogonality constraint by SVD 9. Solve Tz and fx using Least Square and SVD, then fy = fx / #% Y w Z w X w How Many Images Do We Need?! Assumption: K images with M corners each! 4+6K parameters! 2KM constraints! 2KM & 4+6K M>3 and K &2/(M-3)! 2 images with 4 points, but will 1 images with 5 points work?! No, since points cannot be co-planar! 50
Calibration tools? Multi-view Geometry
Two-view geometry Scene geometry (structure): Given corresponding points in two or more images, where is the pre-image of these points in 3D? Correspondence (stereo matching): Given a point in just one image, how does it constrain the position of the corresponding point x in another image? Camera geometry (motion): Given a set of corresponding points in two images, what are the cameras for the two views? Parameters of a Stereo System! Intrinsic Parameters! Characterize the transformation from camera to pixel coordinate systems of each camera! Focal length, image center, aspect ratio P l P P r! Extrinsic parameters! Describe the relative position and orientation of the two cameras X l! Rotation matrix R and translation vector T p l Y l f l O l Z l R, T Z r X r p r Y r f r O r Bahadir K. Gunturk
Triangulation Given projections of a 3D point in two or more images (with known camera matrices), find the coordinates of the point X? x 1 x 2 O 1 O 2 Triangulation We want to intersect the two visual rays corresponding to x 1 and x 2, but because of noise and numerical errors, they don t meet exactly R 2 X? R 1 x 1 x 2 O 1 O 2
Triangulation: Geometric approach Find shortest segment connecting the two viewing rays and let X be the midpoint of that segment X x 1 x 2 O 1 O 2 Epipolar geometry X x x Baseline line connecting the two camera centers Epipolar Plane plane containing baseline (1D family) Epipoles = intersections of baseline with image planes = projections of the other camera center = vanishing points of camera motion direction Epipolar Lines - intersections of epipolar plane with image planes (always come in corresponding pairs)
Epipolar constraint X x x If we observe a point x in one image, where can the corresponding point x be in the other image? Epipolar constraint X X X x x x x Potential matches for x have to lie on the corresponding epipolar line l. Potential matches for x have to lie on the corresponding epipolar line l.
Calibrated Camera Essential matrix Bahadir K. Gunturk Example: Converging cameras
Rectification! Problem: Epipolar lines not parallel to scan lines P P l P r Epipolar Plane p l Epipolar Lines p r O l e l e r Or Epipoles From Sebastian Thrun/Jana Kosecka Rectification! Problem: Epipolar lines not parallel to scan lines P Epipolar Plane P l P r Rectified Images p l Epipolar Lines p r O l Or Epipoles at infinity From Sebastian Thrun/Jana Kosecka
Epipolar rectification Rectified Image Pair Epipolar rectification Rectified Image Pair
Stereo results Scene Ground truth (from Seitz) Results with window correlation Window-based matching (best window size) Ground truth (from Seitz)
Results with better method State of the art Ground truth Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. (from Seitz) How can We Improve Stereo? Space-time stereo scanner uses unstructured light to aid in correspondence Result: Dense 3D mesh (noisy) From Sebastian Thrun/Jana Kosecka
Unstructured Light 3D Scanner OpenCV! cvfindchessboardcorners()! Calibrate:! cvcalibratecamera2()! cvstereocalibrate()! Rectify / Epipolar! cvinitundistortmap()! cvinitundistortrectifymap()! cvcomputecorrespondepilines()! cvfindfundamentalmatrix()! cvstereorectify()! Stereo! cvfindstereocorrespondencebm()! Depth maps! cvreprojectimageto3d()
Assignment #4! Download the Matlab Camera Calibration Toolbox! Read Documentation! First calibration example! Download data! Run camera calibration software! Submit results Conclusion! There s a mathematical model to describe cameras! Parameters can be estimated using a camera calibration approach! Camera calibration is key for accurate stereo reconstruction