Camera models and calibration Read tutorial chapter 2 and 3. http://www.cs.unc.edu/~marc/tutorial/ Szeliski s book pp.29-73
Schedule (tentative) 2 # date topic Sep.8 Introduction and geometry 2 Sep.25 Camera models and calibration 3 Oct.2 Multiple-view geometry 4 Oct.9 Invariant features 5 Oct.6 Model fitting (RANSAC, EM, ) 6 Oct.23 Stereo Matching 7 Oct.3 Segmentation 8 Nov.6 Shape from X (silhouettes, ) 9 Nov.3 Structure from motion Nov.2 Optical flow Nov.27 Tracking (Kalman, particle filter) 2 Dec.4 Object category recognition 3 Dec. Specific object recognition 4 Dec.8 Research overview
Brief geometry reminder x = l l' l T x = 2D line-point coincidence relation: Point from lines: Line from points: π T X = 3D plane-point coincidence relation: T π X Point from planes: T π X X 2 = Plane from points: T π X 3 l = x x' T T 2 T 3 π = 3D line representation: [ ] (as two planes or two points) T 2x2 2D Ideal points ( x, x, ) T 2 l,, 2D line at infinity ( ) T = P Q T A B = T 3D Ideal points ( X, X2, X3,) 3D plane at infinity Π = (,,,) T 3
Conics and quadrics Conics l x T Cx = l T C * l = * - C = C x C l=cx Quadrics X T T * QX = π Q π = 4
Hierarchy of 2D transformations Projective 8dof Affine 6dof Similarity 4dof Euclidean 3dof h h h a a sr sr r r2 2 3 2 2 h h h a a r r 2 22 32 2 22 sr sr 2 22 2 22 h3 h 23 h 33 tx t y tx t y tx t y transformed squares invariants Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l Ratios of lengths, angles. The circular points I,J lengths, areas.
Hierarchy of 3D transformations Projective 5dof A T v t v Intersection and tangency Affine 2dof A T t Parallellism of planes, Volume ratios, centroids, The plane at infinity π Similarity 7dof s R T t Angles, ratios of length The absolute conic Ω Euclidean 6dof R T t Volume
Camera model Relation between pixels and rays in space?
Pinhole camera Gemma Frisius, 544
23 Distant objects appear smaller
vanishing point Parallel lines meet 24
Vanishing points H VPL VPR VP VP 2 25 To different directions correspond different vanishing points VP 3
Geometric properties of projection Points go to points Lines go to lines Planes go to whole image or half-plane Polygons go to polygons Degenerate cases: line through focal point yields point plane through focal point yields line 26
Pinhole camera model T T Z fy Z fx Z Y X ) /, / ( ),, ( = Z Y X f f Z fy fx Z Y X linear projection in homogeneous coordinates!
Pinhole camera model = Z Y X f f Z fy fx = Z Y X f f Z fy fx x = PX [ ] I,), diag( P f f =
Principal point offset T y x T p Z fy p Z fx Z Y X ) + /, + / ( ),, ( principal point T x p y p ), ( = + + Z Y X p f p f Z Zp fy Zp fx Z Y X y x x x
Principal point offset = + + Z Y X p f p f Z Zp fy Zp fx y x x x [ ] cam X x = K I = y x p f p f K calibration matrix
Camera rotation and translation = R( X ~ -C ~ ) X ~ cam - RC ~ X ~ R Y R - RC X cam = = X Z [ ] x = KRK I [ I -C X ~ cam ]X x = PX P = K[ R t] t = -RC ~
CCD camera = y y x x p p K α α = y x y x p f p f m m K
General projective camera = y x x x p p s K α α = y x x x p p K α α [ ] C ~ P = KR I non-singular dof (5+3+3) [ ] t P = K R intrinsic camera parameters extrinsic camera parameters
Radial distortion Due to spherical lenses (cheap) Model: R R 2 2 2 2 2 ( xy, ) ( K( x y) K2( x y)...) x = + + + + + y straight lines are not straight anymore http://foto.hut.fi/opetus/26/luennot//atkinson_6-_radial_distortion_zoom_lenses.jpg
Camera model Relation between pixels and rays in space?
Projector model Relation between pixels and rays in space (dual of camera)? (main geometric difference is vertical principal point offset to reduce keystone effect)
Meydenbauer camera vertical lens shift to allow direct ortho-photographs
Affine cameras
Action of projective camera on points and lines projection of point x = PX forward projection of line ( µ ) = P(A+ µb) = PA + µpb= a µb X + back-projection of line Π = P T l Π T X = l T PX ( l T x = ; x = PX)
Action of projective camera on conics and quadrics back-projection to cone Q = T T T T co P CP x Cx = X P CPX = ( x = PX) projection of quadric C * = * T T * T * T Π Q Π = l PQ P l = PQ P ( Π = P T l)
Resectioning Xi x i P?
Direct Linear Transform (DLT) x = [ i ] i i PX i x PX rank-2 matrix
Direct Linear Transform (DLT) Minimal solution P has dof, 2 independent eq./points 5½ correspondences needed (say 6) Over-determined solution n 6 points minimize p = Ap subject to constraint use SVD
Degenerate configurations (i) Points lie on plane or single line passing through projection center (ii) Camera and points on a twisted cubic
Data normalization Scale data to values of order. move center of mass to origin 2. scale to yield order values σ 2 D σ 3 D
Line correspondences Extend DLT to lines Π = P T l i (back-project line) l T i PXi l T i PX2i (2 independent eq.)
Geometric error
Gold Standard algorithm Objective Given n 6 2D to 3D point correspondences {X i x i }, determine the Maximum Likelyhood Estimation of P Algorithm (i) Linear solution: (a) Normalization: (b) DLT X ~ i = UX i x~ = Tx (ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: ~ ~ ~ i i (iii) Denormalization: P = T - ~ PU
Calibration example (i) Canny edge detection (ii) Straight line fitting to the detected edges (iii) Intersecting the lines to obtain the images corners typically precision </ (H&Z rule of thumb: 5n constraints for n unknowns)
Errors in the image (standard case) xˆ i = PX i Errors in the world i X i Errors in the image and in the world x = P X i
Restricted camera estimation Find best fit that satisfies skew s is zero pixels are square principal point is known complete camera matrix K is known Minimize geometric error impose constraint through parametrization Minimize algebraic error assume map from param q P=K[R -RC], i.e. p=g(q) minimize Ag(q)
Restricted camera estimation Initialization Use general DLT Clamp values to desired values, e.g. s=, α x = α y Note: can sometimes cause big jump in error Alternative initialization Use general DLT Impose soft constraints gradually increase weights
Image of absolute conic
A simple calibration device (i) compute H for each square (corners (,),(,),(,),(,)) (ii) compute the imaged circular points H(,±i,) T (iii) fit a conic to 6 circular points (iv) compute K from ω through cholesky factorization ( Zhang s calibration method)
Some typical calibration algorithms Tsai calibration Zhangs calibration http://research.microsoft.com/~zhang/calib/ Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(): 33-334, 2. Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 999. http://www.vision.caltech.edu/bouguetj/calib_doc/
8 from Szeliski s book
Next week: Multiple-view geometry 82