Camera model and calibration

Transcription

1 and calibration AVIO Laboratoire de Traitement du Signal et des Images (LTSI) Université de Rennes 1. Mardi 21 janvier 1 AVIO tristan.moreau@univ-rennes1.fr and calibration

2 Plan : Introduction 1 Introduction 2 AVIO tristan.moreau@univ-rennes1.fr and calibration

3 Overview Figure: Overview in digital image processing. 3 AVIO tristan.moreau@univ-rennes1.fr and calibration

4 Plan global 1 Introduction AVIO tristan.moreau@univ-rennes1.fr and calibration

5 Planar geometry A hierarchy of transformations Plan : 2 Planar geometry A hierarchy of transformations 5 AVIO tristan.moreau@univ-rennes1.fr and calibration

6 Introduction Planar geometry A hierarchy of transformations Vanishing point 6 AVIO tristan.moreau@univ-rennes1.fr and calibration

7 Planar geometry A hierarchy of transformations Homogeneous representation of lines in 2D A line in plane can be represented as ax + by + c = 0. A line l can be represented by the vector (a, b, c) T. The same line l can be represented by multiples of a, b and c since kax + kby + kc = 0 = ax + by + c for k 0. Thus (a, b, c) T and k(a, b, c) T represents the same line for k 0. This equivalence set of vectors is known as homogeneous vector. The set of equivalence classes of vectors in R 3 (0, 0, 0) T forms the projective space P 2. 7 AVIO tristan.moreau@univ-rennes1.fr and calibration

8 Planar geometry A hierarchy of transformations Homogeneous coordinates : points in 2D A point x = (x, y) T lies on the line l = (a, b, c) T if if ax + by + c = 0. This may be written in terms of an inner product (x, y, 1)(a, b, c) T = (x, y, 1)l = 0. The set of vectors k(x, y, 1) T is a representation of a point (x, y) T in R 2. Just as lines, points are represented by homogeneous vectors. In P 2 the point x = (x 1, x 2, x 3 ) T represents the point (x 1 /x 3, x 2 /x 3 ) T. Result The point x lies on the line l if and only x T l = 0. 8 AVIO tristan.moreau@univ-rennes1.fr and calibration

9 Intersection of lines in 2D Planar geometry A hierarchy of transformations The intersection point x of two lines l 1 and l 2 satifies x T l 1 = x T l 2 = 0. Thus x = l 1 xl 2 (x :cross-product). 9 AVIO tristan.moreau@univ-rennes1.fr and calibration

10 Planar geometry A hierarchy of transformations Intersection of lines in 2D : an example Intersection of lines x = 1 and y = 1? 10 AVIO tristan.moreau@univ-rennes1.fr and calibration

11 Planar geometry A hierarchy of transformations Intersection of lines in 2D : an example Intersection of lines x = 1 and y = 1 x = 1 has homogeneous representation l1 = ( ) T. y = 1 has homogeneous representation l2 = ( ) T. x = l1xl2 = AVIO tristan.moreau@univ-rennes1.fr and calibration

12 Line joining points Planar geometry A hierarchy of transformations The line trough two points x 1 and x 2 is l = x 1 xx AVIO tristan.moreau@univ-rennes1.fr and calibration

13 Intersection of paralell lines Planar geometry A hierarchy of transformations 2 parallel lines l 1 = ( a b c 1 ) T and l2 = ( a b c 2 ) T. Intersection l 1 xl 2 is the point ( b a 0 ) T. Inhomogeneous representation of the intersection ( b/0 a/0 ) T, which makes no sense, except to suggest that the point of intersection has infinitely large coordinates. This observation agrees with the usual idea that parallel lines meet at infinity. 13 AVIO tristan.moreau@univ-rennes1.fr and calibration

14 Planar geometry A hierarchy of transformations Intersection of lines in 2D : an example Intersection of lines x = 1 and x = 2 x = 1 has homogeneous representation l1 = ( ) T. x = 2 has homogeneous representation l2 = ( ) T. x = l1xl2 = which is the point at infinity in direction ot the y-axis. 14 AVIO tristan.moreau@univ-rennes1.fr and calibration

15 Isometries Introduction Planar geometry A hierarchy of transformations x cos(θ) sin(θ) t x x y = sin(θ) cos(θ) t y y AVIO tristan.moreau@univ-rennes1.fr and calibration

16 Similarity transformations Planar geometry A hierarchy of transformations x scos(θ) ssin(θ) t x x y = ssin(θ) scos(θ) t y y AVIO tristan.moreau@univ-rennes1.fr and calibration

17 Affine transformations Planar geometry A hierarchy of transformations x a 11 a 12 t x x y = a 21 a 22 t y y AVIO tristan.moreau@univ-rennes1.fr and calibration

18 Projective transformations Planar geometry A hierarchy of transformations x a 11 a 12 a 13 x y = a 21 a 22 a 23 y. 1 a 31 a 32 a AVIO tristan.moreau@univ-rennes1.fr and calibration

19 Plan : Finite cameras The projective camera 3 Finite cameras The projective camera 19 AVIO tristan.moreau@univ-rennes1.fr and calibration

20 History Introduction Finite cameras The projective camera Figure: Camera obscura published in De Radio Astronomica et Geometrica AVIO and calibration

21 The basic pinhole camera model 1 Finite cameras The projective camera 21 AVIO tristan.moreau@univ-rennes1.fr and calibration

22 The basic pinhole camera model 2 Finite cameras The projective camera Central projection of points space (X, Y, Z) T onto image plane (x, y) T. 22 AVIO tristan.moreau@univ-rennes1.fr and calibration

23 The basic pinhole camera model 3 Finite cameras The projective camera Central projection mapping from world to image coordinates : (X, Y, Z) T is mapped (fx /Z, fy /Z) T. 23 AVIO tristan.moreau@univ-rennes1.fr and calibration

24 The basic pinhole camera model 4 Central projection using homogeneous coordinates X fx f fy = 0 f 0 0 Y Z Z x = PX x : image point (3-vector). X : world point (4-vector). P : camera projection matrix (3x4).

25 The basic pinhole camera model 5 Principal point offset X fx + Zp x f 0 p x 0 fy + Zp y = 0 f p y 0 Y Z Z f 0 p x K = 0 f p y and x = K ( Id 0 ) X cam (camera coordinate) K : camera calibration matrix.

26 The basic pinhole camera model 6 Camera rotation and translation (inhomogeneous) X : 3-vector in the world coordinate frame. X cam : same point in camera coordinate frame. C : represents the coordinates of the camera center in the world coordinate frame. R rotation matrix (3x3) representing the orientation of the camera coordinate frame.

27 The basic pinhole camera model 7 Camera rotation and translation X cam = R( X C) (inhomogeneous). ( ) X ( ) R R C X cam = Y R R 0 1 Z = C X x = KR ( Id C ) X. X in world coordinate.

28 CCD cameras α x 0 x 0 K = 0 α y y 0 (4 DOF) α x = fm x and α y = fm y : focal length in pixel dimensions. x 0 = (x 0, y 0 ) : principal point in pixel dimensions. x 0 = m x p x and y 0 = m y p y. P = KR ( Id C ) (10 DOF).

29 Finite projective camera Finite cameras The projective camera Skew parameter : s α x s x 0 K = 0 α y y 0 (5 DOF) P = KR ( Id C ) (11 DOF). A camera with this type of K is called finite projective camera. A finite projective camera has 11 DOF. 29 AVIO tristan.moreau@univ-rennes1.fr and calibration

30 Finite projective camera : summary Finite cameras The projective camera Finite projective camera x = PX. α x s p x ( ) P = 0 α y p y R R C P = KR ( Id C ) and KR invertible. (11 DOF = 5+3+3). 30 AVIO tristan.moreau@univ-rennes1.fr and calibration

31 Finite cameras The projective camera Finite projective camera : decomposition of P Finite projective camera P = KR ( Id C ). P = M ( Id C ) where M = KR. P = ( M p 4 ) where p4 last column of P. [K, R] = RQ(M). RQ matrix decomposition where R is upper-triangular and Q is a rotation matrix. C = M 1 p AVIO tristan.moreau@univ-rennes1.fr and calibration

32 Plan : Finite cameras The projective camera 3 Finite cameras The projective camera 32 AVIO tristan.moreau@univ-rennes1.fr and calibration

33 Camera anatomy Anatomy Camera center. Column vectors. Principal plane. Axis plane. Principal point. Principal ray.

34 Camera anatomy : camera centre Finite cameras The projective camera Camera centre P has a 1D null space. Let C a 4-vector such that PC = 0. Let show that C is camera center X(λ) = λa + (1 λ)c. Line containing C and any point A. Points on this line are projected : x = PX(λ). x = PX(λ) = λpa + (1 λ)pc = λpa. All points on the line are mapped to the same image point PA, which means that the line is a ray through the camera center. 34 AVIO tristan.moreau@univ-rennes1.fr and calibration

35 Finite cameras The projective camera Camera anatomy : column vectors of P Image of column vectors p 1, p 2, p 3, p 4 Column vectors are the image points, which project the axis directions (X,Y,Z) and the origin. 35 AVIO tristan.moreau@univ-rennes1.fr and calibration

36 Camera anatomy : principal plane Principal plane Plane trough the camera center and parallel to the image plane. PX = (x, y, 0) T. x P 1T X y = Y if P3T X = 0. 0 P 2T P 3T Z 1 P 3T is the vector representing the principal plane.

37 Camera anatomy : principal point Principal point Principal axe : line through C and perpendicular to principal plane. Principal point X 0 : intersection of principal axe with image plane. ˆP 3 = (p 31, p 32, p 33, 0) T : normal direction to principal plane. P = (M, p 4 ) and x 0 = P ˆP 3 = Mm 3. m 3T : third row of M.

38 Finite cameras The projective camera Decomposition of the camera matrix Finding the camera centre C = (X, Y, Z, T ) PC = 0. Numerically : SVD of P. Algebrically : X = det((p 2, p 3, p 4 )). Y = det((p 1, p 3, p 4 )). Z = det((p 1, p 2, p 4 )). T = det((p 1, p 2, p 3 )). 38 AVIO tristan.moreau@univ-rennes1.fr and calibration

39 Finite cameras The projective camera Decomposition of the camera matrix Finding the camera orientation and internal parameters P = KR ( Id C ) = M ( Id α x s x 0 C ) where M = KR. K = 0 α y y s is generally AVIO tristan.moreau@univ-rennes1.fr and calibration

40 Plan : Basic equations DLT algorithm Radial distortion with opencv 4 Basic equations DLT algorithm Radial distortion with opencv 40 AVIO tristan.moreau@univ-rennes1.fr and calibration

41 Basic equations DLT algorithm Radial distortion with opencv 41 AVIO and calibration

42 Resectioning Basic equations DLT algorithm Radial distortion with opencv Numerical methods for estimating the camera projection matrix P from corresponding world and image entities. Simplest entities are the correspondance between a 3D point X i and its image x i. P will be determined for sufficient many correpondances. 42 AVIO tristan.moreau@univ-rennes1.fr and calibration

43 Basic equations Find P (3x4) so that i{1,...n}, x i = PX i. Cross product : i{1,...n}, x i xpx i = 0. p 1T X i If the j-th row of P is denoted p jt then PX i = p 2T X i. p 3T X i y i p 3T X i w i p 2T X i Let x i = (x i, y i, w i ) then x i xx i = w i p 1T X i x i p 3T X i. x i p 2T X i y i p 1T X i 0 T w i X T i y i X T i p 1 w i X T i 0 T x i X T i p 2 = 0 since y i X T i x i X T i 0 T p 3 p jt X i = X T i p j. ( 0 T w i X T i y i X T i w i X T i 0 T x i X T i ) p 1 p 2 = 0. p 3

44 Basic equations Basic equations DLT algorithm Radial distortion with opencv ( 0 T w i X T i y i X T ) i p 1 w i X T i 0 T x i X T p 2 = 0. i p 3 A i p = 0 where A i is 2x12 matrix and p 12-vector. From a set of n point correspondances, we obtain a 2n x 12 matrix A by stacking equations. Solve linear equation Ap = AVIO tristan.moreau@univ-rennes1.fr and calibration

45 Solving for p Basic equations DLT algorithm Radial distortion with opencv Minimal solution P has 12 entries (11 DOF ignoring scale). Each points correspondance : 2 independant equations. Need 5,5 correspondences to solve P. In this case, the solution is exact (ie. the space points are projected exactly onto their measured images). 45 AVIO tristan.moreau@univ-rennes1.fr and calibration

46 Solving for p Basic equations DLT algorithm Radial distortion with opencv Over-determined solution Noisy data : n > 6. Minimizing an algebraic error Ap subject to normalization constraint. For example p = 1. One algorithm to soleve this : DLT (Direct Linear Transformation). 46 AVIO tristan.moreau@univ-rennes1.fr and calibration

47 Basic DLT algorithm Basic equations DLT algorithm Radial distortion with opencv 47 AVIO and calibration

48 Data normalization Need of normalization DLT algortihm is not invariant to similarity transformations of the image. Is there some coordinate systems better than others? Yes. Isotropic scaling Points are translated so that their centroid is at the origin. The points are scaled so that the average distance from the origin is 2 for 2D or 3 for 3D.

49 Normalized DLT algorithm Objective Given n 6 3D to 2D point correspondances, determine the projection matrix P such that x i = PX i. Algorithm Normalization of x i : compute similarity t (translation and scaling) that takes points x i to a new point x i such that the centroid of the points x i is the coordinate origin and their average distance from the origin is 2. Normalisation of X i : compute similarity T (translation and scaling) that takes points X i to a new point X i such that the centroid of the points X i is the coordinate origin and their average distance from the origin is 3. Basic DLT algorithm : P. Denormalization : P = T 1 Pt.

50 Limits of the Pinhole camera model Basic equations DLT algorithm Radial distortion with opencv Lens One ray in the aperture? Need of lens to get more light and focus. But lens introduces distorsions : radial distorsion. tangential distorsion. Need to correct these distortions AVIO tristan.moreau@univ-rennes1.fr and calibration

51 Radial distortion Basic equations DLT algorithm Radial distortion with opencv 51 AVIO and calibration

52 Radial distortion Basic equations DLT algorithm Radial distortion with opencv ) ( x ỹ : image coordinates of a point under ideal (non-distorded) pinhole projection. ) T ( ) ( x ỹ 1 = Id 0 Xcam where X cam is a 3D point in camera coordinates. The radial distorsion is modelled as ( xd y d ) ) ( x = L( r). ỹ r is the radial distance x 2 + ỹ 2 from center of distortion. L( r) : distortion factor. 52 AVIO tristan.moreau@univ-rennes1.fr and calibration

53 Radial distortion Basic equations DLT algorithm Radial distortion with opencv Correction of distortion x = x c + L(r)(x x c ). ỹ = y c + L(r)(y y c ), where x c and y c center of radial distortion. L(r) is defined for r 0. L(r) = 1 + k 1 r + K 2 r 2 + k 3 r (Taylor expansion). The coefficient k 1, k 2... are considered part of the interior calibration of the camera. 53 AVIO tristan.moreau@univ-rennes1.fr and calibration

54 Introduction to opencv Basic equations DLT algorithm Radial distortion with opencv AVIO and calibration

55 with opencv Basic equations DLT algorithm Radial distortion with opencv Demo AVIO and calibration