3D Photography Marc Pollefeys, Torsten Sattler Spring 2015
Schedule (tentative) Feb 16 Feb 23 Mar 2 Mar 9 Mar 16 Mar 23 Mar 30 Apr 6 Apr 13 Apr 20 Apr 27 May 4 May 11 May 18 Apr 6 Introduction Geometry, Camera Model, Calibration Features, Tracking / Matching Project Proposals by Students Structure from Motion (SfM) + 2 papers Dense Correspondence (stereo / optical flow) + 2 papers Bundle Adjustment & SLAM + 2 papers Easter Multi-View Stereo & Volumetric Modeling + 2 papers Project Updates 3D Modeling with Depth Sensors + 2 papers 3D Scene Understanding + 2 papers 4D Video & Dynamic Scenes + 2 papers Final Demos Pentecost
Important Dates Decide for a project by Feb. 27 Form groups and send us your topics Proposal submission by March 6 Talk to your supervisor and send proposals Proposal presentation on March 9 5min per group
3D Photography Class 2 Projective Geometry and Camera model points, lines, planes conics and quadrics transformations camera model Read tutorial chapter 2 and 3.1 http://www.cs.unc.edu/~marc/tutorial/ Chapters 1, 2 and 5 in Hartley and Zisserman Chapters 2, 3 and 6 in 2 nd edition
Overview 2D Projective Geometry 3D Projective Geometry Camera Models & Calibration
2D Projective Geometry? Projections of planar surfaces
2D Projective Geometry? Measure distances A. Criminisi. Accurate Visual Metrology from Single and Multiple Uncalibrated Images. PhD Thesis 1999.
2D Projective Geometry? Discovering details Piero della Francesca, La Flagellazione di Cristo (1460) A. Criminisi. Accurate Visual Metrology from Single and Multiple Uncalibrated Images. PhD Thesis 1999.
2D Projective Geometry? Image Stitching
2D Projective Geometry? Image Stitching
Homogeneous coordinates y t x t y x' y' x y As matrix multiplication x Homogeneous coordinates: y x' cos sin t x x y' sin cos t y y 1 0 0 1 1 Equivalence class of vectors, any vector is representative Projective Space P 2 = R 3 (0,0,0) T x,y,1 T ~ k x,y,1 z 1 x Projective transformation = matrix multiplication k x,y,1 T T,k 0 z
Homogeneous coordinates l a b (Homogeneous) representation of 2D line: ax by c 0 a,b,c T x,y,1 0 c / a 2 b 2 The point x lies on the line l if and only if l T x 0 Note that scale is unimportant for incidence relation a,b,c T ~ ka,b,c T,k 0 x,y,1 T ~ k x,y,1 T x 1 x 3, x 2 x 3 T,k 0 Homogeneous coordinates x 1,x 2,x 3 T but only 2DOF Inhomogeneous coordinates x, y T
Points from lines and vice-versa Intersections of lines The intersection of two lines l and l' is Line joining two points The line through two points x and x' is x l l' l x x' Example Note: x x' x x' x (0,1,-1) y 1 y 1 x 1 x (1,0,-1) y 1 with x 0 z -y -z 0 x y -x 0
Ideal points and the line at infinity Intersections of parallel lines l a,b,c Example T and l' a,b,c' T l l' b,a,0 T b,-a a,b tangent vector normal direction Ideal points x =1 x = 2 x 1,x 2,0 Line at infinity l 0,0,1 T T P 2 R 2 l Note that in P 2 there is no distinction between ideal points and others
Conics Curve described by 2 nd -degree equation in the plane ax 2 bxy cy 2 dx ey f 0 or homogenized x a x 1 x3,y a x 2 x3 ax 2 1 bx 1 x 2 cx 2 2 dx 1 x 3 ex 2 x 3 fx 2 3 0 or in matrix form a b/2 d /2 x T Cx 0 with C b/2 c e /2 d /2 e /2 f 5DOF: a :b :c : d :e : f (defined up to scale)
Five points define a conic For each point the conic passes through ax i 2 bx i y i cy i 2 dx i ey i f 0 or x 2 i,x i y i,y 2 i,x i,y i,1.c 0 c a,b,c,d,e, f T stacking constraints yields x 1 2 x 2 2 x 3 2 x 4 2 x 5 2 x 1 y 1 y 1 2 x 2 y 2 y 2 2 x 3 y 3 y 3 2 x 4 y 4 y 4 2 x 5 y 5 y 5 2 x 1 y 1 1 x 2 y 2 1 x 3 y 3 1 c 0 x 4 y 4 1 x 5 y 5 1
Tangent lines to conics The line l tangent to C at point x on C is given by l=cx x l C
Dual conics A line tangent to the conic C satisfies l T * C l = 0 In general (C full rank): * -1 C C Dual conics = line conics = conic envelopes
Degenerate conics A conic is degenerate if matrix C is not of full rank e.g. two lines (rank 2) C = lm T + ml e.g. repeated line (rank 1) T m l C ll T l Degenerate line conics: 2 points (rank 2), double point (rank1) Note that for degenerate conics ( * C ) * C
2D projective transformations Definition: A projectivity is an invertible mapping h from P 2 to itself such that three points x 1,x 2,x 3 lie on the same line if and only if h(x 1 ),h(x 2 ),h(x 3 ) do. Theorem: A mapping h:p 2 P 2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P 2 reprented by a vector x it is true that h(x)=hx Definition: Projective transformation x' 1 x' 2 x' 3 h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 h 33 x 1 x 2 x 3 or x' Hx 8DOF projectivity=collineation=projective transformation=homography
Transformation of 2D points, lines and conics For a point transformation x' Hx Transformation for lines l' H -T l Transformation for conics C' H -T CH -1 Transformation for dual conics C' * HC * H T
Fixed points and lines He = λe (eigenvectors H = fixed points) ( 1 = 2 pointwise fixed line) H -T l l (eigenvectors H - T = fixed lines)
Hierarchy of 2D transformations Projective 8dof Affine 6dof Similarity 4dof Euclidean 3dof h h h a a 0 sr sr 0 r11 r21 0 11 21 31 11 21 11 21 h h h a a r r 12 22 32 12 22 0 sr sr 12 22 0 12 0 22 h13 h 23 h 33 tx t y 1 tx t y 1 tx t y 1 transformed squares invariants Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l Ratios of lengths, angles. The circular points I,J Lengths, areas.
Application: Removing Perspective Two stages: From perspective to affine transformation via the line at infinitiy From affine to similarity transformation via the circular points
The line at infinity 0 T T A 0 l H A l 0 l T T t 1 A 1 The line at infinity l is a fixed line under a projective transformation H if and only if H is an affinity Note: not fixed pointwise
Affine properties from images projection rectification 1 0 0 H P H A 0 1 0 l 1 l 2 l 3 l l 1 l 2 l 3 T,l 3 0
Affine Rectification v 1 v 2 l l 1 l 3 l 2 l 4 v v 2 = l3 l4 1 l1 l2 l v 1 v 2
The circular points 1 I i 0 1 J i 0 I H S I scos ssin t x 1 1 ssin scos t y i 0 0 1 sei i I 0 0 The circular points I, J are fixed points under the projective transformation H iff H is a similarity
The circular points circular points l x 1 2 x 2 2 dx 1 x 3 ex 2 x 3 fx 3 2 0 I 1,0,0 x 3 0 T i0,1,0 T x 1 2 x 2 2 0 I 1,i,0 T J 1,-i,0 T Algebraically, encodes orthogonal directions
Conic dual to the circular points C C * * H IJ T C JI T * T S H S * C 1 0 0 0 1 0 0 0 0 J I l * C The dual conic is fixed conic under the projective transformation H iff H is a similarity * C Note: has 4DOF (det = 0) l is the nullvector
Angles Euclidean: l l 1,l 2,l 3 cos T m m 1,m 2,m 3 l 1 m 1 l 2 m 2 l 2 2 1 l 2 m 2 2 1 m 2 T Projective: cos l T C * m l T C * l m T C * m l T C * m 0 (orthogonal) Once we know dual conic on projective plane, we can measure Euclidean angles!
Metric Rectification Dual conic under affinity C * A t 0 T I 0 1 0 T AT 0 AA T 0 1 t T 1 0 T 0 S=AA -T symmetric, estimate from two pairs of orthogonal lines: l 1 m 1,l 1 m 2 l 2 m 1,l 2 m 2 s 0 Note: Result defined up to similarity
Overview 2D Projective Geometry 3D Projective Geometry Camera Models & Calibration
3D points and planes Homogeneous representation of 3D points and planes π1x1 + π2x 2 + π3x3 + π4x 4 = 0 The point X lies on the plane π if and only if π T X = 0 The plane π goes through the point X if and only if π T X = 0
Planes from points 0 π X X X 3 2 1 T T T π = 0 X π = 0 X π = 0, X π 3 2 1 T T T and from Solve (solve as right nullspace of ) π T T T 3 2 1 X X X
Solve π π π T 1 T 2 T 3 X Points from planes T T T X fromπ1 X = 0, π2x = 0 and π3x = 0 0 X = Mx M X X π T M = 0 X (solve as right nullspace of T ) Representing a plane by its span 1 2X3 π π π T 1 2 T 3
Lines (4dof) Representing a line by its span A W B T T Dual representation W * P Q T T λa μb λp μq * T W W = WW * T = 0 2 2 Example: X-axis 0 W 1 0 0 0 0 1 0 W * 0 0 0 1 1 0 0 0 (Alternative: Plücker representation, details see e.g. H&Z)
Points, lines and planes W M T M π = 0 X X W W * M T MX 0 π W π
Quadrics and dual quadrics X T QX 0 (Q : 4x4 symmetric matrix) 9 DOF (up to scale) In general, 9 points define quadric det(q)=0 degenerate quadric tangent plane Dual quartic: ( adjoint) relation to quadric (non-degenerate) QX T Q * 0 Q * Q -1 Q * o Q o o o o o
Transformation of 3D points, planes and quadrics For a point transformation X' HX (cfr. 2D equivalent) x' Hx Transformation for lines ' H -T Transformation for quadrics Q' H -T QH -1 l' H -T l C' H -T CH -1 Transformation for dual quadrics Q' * HQ * H T C' * HC * H T
Hierarchy of 3D transformations Projective 15dof A v T t v Intersection and tangency Affine 12dof Similarity 7dof Euclidean 6dof A t 0 T 1 sr t 0 T 1 R t 0 T 1 Parallellism of planes, Volume ratios, centroids, The plane at infinity π Angles, ratios of length The absolute conic Ω Volume
Hierarchy of 3D transformations projective Plane at infinity affine Absolute conic similarity
The plane at infinity 0 T T A 0 0 π H A π π T T - t A 1 0 1 The plane at infinity π is a fixed plane under a projective transformation H iff H is an affinity 1. canonical position 2. contains directions D X 1,X 2,X 3,0 T 3. two planes are parallel line of intersection in π 4. line // line (or plane) point of intersection in π 0,0,0,1 T
The absolute conic The absolute conic Ω is a (point) conic on π. In a metric frame: or conic for directions: (with no real points) X 1 2 X 2 2 X 3 2 X 4 0 X 1,X 2,X 3 I X 1,X 2,X 3 The absolute conic Ω is a fixed conic under the projective transformation H iff H is a similarity T 1. Ω is only fixed as a set 2. Circle intersect Ω in two circular points 3. Spheres intersect π in Ω
The absolute dual quadric * I 0 0 T 0 The absolute dual quadric Ω * is a fixed quadric under the projective transformation H iff H is a similarity 1. 8 dof 2. plane at infinity π is the nullvector of Ω 3. Angles: cos 1 T * 2 T 1 * 1 T 2 * 2
Overview 2D Projective Geometry 3D Projective Geometry Camera Models & Calibration
Camera Model Relation between pixels and rays in space?
Pinhole Camera
Pinhole Camera Model (X,Y,Z) T a ( fx /Z, fy /Z) T X Y a Z 1 fx fy Z X f 0 f 0 Y Z 1 0 1 linear projection in homogeneous coordinates!
Pinhole Camera Model X X fx f 10 0 Y 0 fy Y x PX f 1 0 Z Z Z 1 0 1 0 1 1 P diag( f, f,1) I 0
Principal Point Offset (X,Y,Z) T a ( fx /Z p x, fy /Z p y ) T X Y a Z 1 (p x, p y ) T fx Zp x fy Zp x Z principal point X f p x 0 f p y 0 Y Z 1 0 1
Principal Point Offset X fx Zp x f p x 0 fy Zp Y x KI 0f X p y 0 cam Z Z 1 0 1 f p x K f p y 1 calibration matrix
Camera Rotation and Translation C X cam RX - C X X cam R -R C Y R -R C X 0 1 Z 0 1 1 x KR I 0 I X -C cam X x PX P KR t t -RC C -R T t
CCD Camera K m x m y p x f p y 1 f p x p y 1
General Projective Camera K x s p x y p y 1 P KRI -C non-singular P KR t In practice: 11 dof (5+3+3) x y, s 0 p x, p y intrinsic camera parameters extrinsic camera parameters = image center
Affine Cameras
Camera Model Relation between pixels and rays in space?
Projector Model Relation between pixels and rays in space (dual of camera)? (main geometric difference is vertical principal point offset to reduce keystone effect)
Meydenbauer Camera vertical lens shift to allow direct ortho-photographs
Projector-Camera Systems IR projector IR camera
Action of Projective Camera on Points and Lines forward projection of point x PX image point back-projects to line / ray forward projection of line (not pointwise!) X P(A B) PA PB a b back-projection of line P T l X R T K 1 x T X l T PX l T x 0; x PX
Action of Projective Camera on Conics and Quadrics back-projection to cone Q co P T CP x T Cx X T P T CPX 0 x PX projection of quadric C * PQ * P T T Q * l T PQ * P T l 0 P T l
Image of Absolute Conic * P * P T I 0 K R t KK T K -T K 1 0 0 RT t T K T
A Simple Calibration Device (i) compute H for each square (corners @ (0,0),(1,0),(0,1),(1,1)) (ii) compute the imaged circular points H(1,±i,0) T (iii) fit a conic to 6 circular points (iv) compute K from through Cholesky factorization ( Zhang s calibration method)
Practical Camera Calibration Method and Pictures from Zhang (ICCV 99): Flexible Camera Calibration By Viewing a Plane From Unknown Orientations Unknown: constant camera intrinsics K (varying) camera pose R,t Known: 3D coordinates of chessboard corners => Define to be the z=0 plane ( X=[X 1 X 2 0 1] T ) Point is mapped as λx = K (r 1 r 2 r 3 t) X x = K (r 1 r 2 t) [X 1 X 2 1] K Homography H between image and chess coordinates, estimate from known X i and measured x i f x f y p x p y 1
Direct Linear Transformation (DLT) x i Hx i 0 x i x i, y i, w i y i h 3T x i w i h 2T x i x i Hx i w i h 1T x i x i h 3T x i x i h 2T x i y i h 1T x i T w i x i T y i x i 0 T T w i x i T y i x i 0 T T x i x i T x i x i 0 T A i h 0 T h 1 h 2 h 3 h 1T x i Hx i h 2T x i h 3T x i 0
Direct Linear Transformation (DLT) Equations are linear in h A i h 0 Only 2 out of 3 are linearly independent (indeed, 2 eq/pt) 0 T T T i x i y i x i 0 T T T T w w i x i 0 T i x i y T x i x i x i T i w T T y i x i i 0 T T x i x i x i 0 T i x i h 1 h 2 h 3 (only x i Adrop 1 i y third i A 2 i row w i Aif 3 i w i 0 0) Holds for any homogeneous representation, e.g. (x i,y i,1) 0
Direct Linear Transformation (DLT) Solving for H A 1 A 2 A 3 A 4 Ah h 0 size A is 8x9 or 12x9, but rank 8 Trivial solution is h=0 9 T is not interesting 1-D null-space yields solution of interest pick for example the one with h 1
Direct Linear Transformation (DLT) Over-determined solution No exact solution because of inexact measurement i.e. noise Ah 0 A 1 A 2 Ah h 0 M A n Find approximate solution - Additional constraint needed to avoid 0, e.g. - not possible, so minimize Ah h 1
DLT algorithm Objective Given n 4 2D to 2D point correspondences {x i x i }, determine the 2D homography matrix H such that x i =Hx i Algorithm (i) (ii) For each correspondence x i x i compute A i. Usually only two first rows needed. Assemble n 2x9 matrices A i into a single 2nx9 matrix A (iii) Obtain SVD of A. Solution for h is last column of V (iv) Determine H from h
Importance of Normalization 0 0 0 x i y i 1 y i x i x i y i 1 0 0 0 x i x i y i y i x i y i y i x i ~10 2 ~10 2 1 ~10 2 ~10 2 1 ~10 4 ~10 4 ~10 2 orders of magnitude difference! h 1 h 2 h 3 0 Monte Carlo simulation for identity computation based on 5 points (not normalized normalized)
Normalized DLT Algorithm Objective Given n 4 2D to 2D point correspondences {x i x i }, determine the 2D homography matrix H such that x i =Hx i Algorithm (i) (ii) Normalize points Apply DLT algorithm to (iii) Denormalize solution x i T norm x i, x i x i x i, -1 H T norm H T norm Normalization (independently per image): T norm x i Translate points such that centroid is at origin Isotropic scaling such that mean distance to origin is 2
x xˆ x Geometric Distance measured coordinates estimated coordinates true coordinates d(.,.) Euclidean distance (in image) Error in one image H ˆ argmin d H H i i 2 x i,hx i e.g. calibration pattern Symmetric transfer error H ˆ argmin dx i,h -1 x 2 i d x i,hx i Reprojection error H ˆ, x ˆ i, x ˆ i argmin d x i, ˆ H ˆ, x ˆ i, ˆ x i x' i 2 x i 2 d x ˆ i x i, 2 subject to x ˆ i H ˆ ˆ x i
Reprojection Error 2 d d x,h -1 x x,hx 2 2 dx, x ˆ 2 d x, ˆ x
Statistical Cost Function and Maximum Likelihood Estimation Optimal cost function related to noise model Assume zero-mean isotropic Gaussian noise (assume outliers removed) 2 / 2 2 Prx 1 x,x 2 2 ed Error in one image Pr x 1 H i i log Pr x H 1 i 2 2 ed x i,hx i 2 / 2 2 Maximum Likelihood Estimate d 2 x i,hx i 2 2 i d( x i,hx i ) 2 const
Statistical Cost Function and Maximum Likelihood Estimation Optimal cost function related to noise model Assume zero-mean isotropic Gaussian noise (assume outliers removed) 2 / 2 2 Prx 1 x,x 2 2 ed Error in both images Pr x 1 H i i / 2 2 d x 2 2 e i,x i 2 d x,hx 2 i i Maximum Likelihood Estimate x i 2 x i 2 d x i, ˆ d x i, ˆ
Objective Gold Standard Algorithm Given n 4 2D to 2D point correspondences {x i x i }, determine the Maximum Likelihood Estimation of H (this also implies computing optimal x i =Hx i ) Algorithm (i) Initialization: compute an initial estimate using normalized DLT or RANSAC (ii) Geometric minimization of symmetric transfer error: Minimize using Levenberg-Marquardt over 9 entries of h or reprojection error: compute initial estimate dx for optimal {x i } i, x ˆ i 2 dx i, x ˆ i 2 minimize cost over {H,x 1,x 2,,x n } if many points, use sparse method
Radial Distortion Due to spherical lenses (cheap) Model: R R 2 2 2 2 2 ( x, y) (1 K1( x y ) K2( x y )...) x y straight lines are not straight anymore http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpg
Calibration with Radial Low radial distortion: Distortion Ignore radial distortion during initial calibration Estimate distortion parameters, refine full calibration High radial distortion: Simultaneous estimation Fitzgibbon, Simultaneous linear estimation of multiple view geometry and lens distortion, CVPR 2001 Kukelova et al., Real-Time Solution to the Absolute Pose Problem with Unknown Radial Distortion and Focal Length, ICCV 2013
Bouguet Toolbox http://www.vision.caltech.edu/bouguetj/calib_doc/
Rolling Shutter Cameras Image build row by row Distortions based on depth and speed Many mobile phone cameras have rolling shutter Video credit: Olivier Saurer
Rolling Shutter Cameras Calibration more complex: Internal calibration + shutter speed Oth et al., Rolling Shutter Camera Calibration, CVPR 2013 Non-central projection under motion Google StreetView Camera Setup Klinger et al., Street View Motion-from-Structurefrom-Motion, ICCV 2013
Event Cameras Kim et al., Simultaneous Mosaicing and Tracking with an Event Camera, BMVC 2014
Reminder Project presentation in 2 weeks Find team mate / decide topic soon Submit proposal
Next class: Features, Tracking / Matching