Camera calibration. Robotic vision. Ville Kyrki

Transcription

1 Camera calibration Robotic vision

2 Where are we? Images, imaging Image enhancement Feature extraction and matching Image-based tracking Camera models and calibration Pose estimation Motion analysis and visual servoing Stereo vision 3-D reconstruction

3 This week Camera calibration Needed for Euclidean measurements Problem: Estimate Intrinsic parameters, Extrinsic parameters, or Both Use a calibration target with known structure

4 Calibration Idea 3-D locations of points on a calibration target known 2-D locations of the same points can be measured from an image Use these correspondences to find camera parameters

5 Calibration options Alternative possibilities: Direct parameter calibration Estimate single parameter at a time Projection matrix calibration Estimate the projection matrix Parameters can be found later Direct parameter calibration by optimization Estimate all parameters simultaneously by iterative optimization Needs an initial guess

6 Projection matrix x=m X Contains all necessary information (extrinsic + intrinsic parameters) about a camera. except lens distortion Projection matrix can be solved, instead of individual parameters. In many cases, knowledge of projection matrix is sufficient. Projection matrix can also be decomposed to intrinsic and extrinsic parts. camera location in world M=K [ R C t W ]=K [ R R W t C ]

7 Solving projection matrix Assume: are set of homogeneous 3-D points, x i are corresponding homogeneous 2-D points How many points?

8 Solving projection matrix Assume: are set of homogeneous 3-D points, x i are corresponding homogeneous 2-D points How many points? 11 degrees of freedom in M (3x4 matrix, with unknown scale) -> at least 6 points in practice, there are errors in measurements many more points needed for good result Solution: write x i =M for all of the points

9 Solving projection matrix (cont'd) Projection can be written x i = u i w i = m 11 +m 12 Y i +m 13 Z i +m 14 m 31 +m 32 Y i +m 33 Z i +m 34 y i = v i w i = m 21 +m 22 Y i +m 23 Z i +m 24 m 31 +m 32 Y i +m 33 Z i +m 34 Method called Direct linear transform (DLT)

10 Solving projection matrix (cont'd) Projection can be written x i = u i w i = m 11 +m 12 Y i +m 13 Z i +m 14 m 31 +m 32 Y i +m 33 Z i +m 34 y i = v i w i = m 21 +m 22 Y i +m 23 Z i +m 24 m 31 +m 32 Y i +m 33 Z i +m 34 Method called Direct linear transform (DLT) (m 31 +m 32 Y i +m 33 Z i +m 34 )x i =m 11 +m 12 Y i +m 13 Z i +m 14 (m 31 +m 32 Y i +m 33 Z i +m 34 )y i =m 21 +m 22 Y i +m 23 Z i +m 24

11 Solving projection matrix (cont'd) Projection can be written x i = u i w i = m 11 +m 12 Y i +m 13 Z i +m 14 m 31 +m 32 Y i +m 33 Z i +m 34 y i = v i w i = m 21 +m 22 Y i +m 23 Z i +m 24 m 31 +m 32 Y i +m 33 Z i +m 34 Method called Direct linear transform (DLT) (m 31 +m 32 Y i +m 33 Z i +m 34 )x i =m 11 +m 12 Y i +m 13 Z i +m 14 (m 31 +m 32 Y i +m 33 Z i +m 34 )y i =m 21 +m 22 Y i +m 23 Z i +m 24 m 11 +Y i m 12 + Z i m 13 +m 14 x i m 31 x i Y i m 32 x i Z i m 33 x i m 34 =0 m 21 +Y i m 22 +Z i m 23 +m 24 y i m 31 y i Y i m 32 y i Z i m 33 y i m 34 =0

12 Solving projection matrix (cont'd) We can write the previous as A m=0 where X1 Y 1 Z x1 X 1 x1 Y 1 x1 Z 1 x X 1 Y 1 Z 1 1 y 1 X 1 y 1 Y 1 y 1 Z 1 y A=( X N Y N Z N x N X N x N Y N x N Z N x N X N Y N Z N 1 y N X N y N Y N y N Z N y N ) m=(m 11, m 12,, m 33, m 34 ) T How to solve?

13 Solving projection matrix (cont'd) Minimize: min m A m 2

14 Solving projection matrix (cont'd) Minimize: min m A m 2 Solution: Null-space of A Can be found by singular value decomposition Calculate the SVD A=UDV T Solution is the column of V corresponding to the smallest singular value of A Problems: not possible to estimate lens distortion parameters sensitive to errors (considers algebraic error) Trivial solution m=0. The other solution homogeneous. Matlab: [U,D,V]=svd(A);

15 Decreasing error sensitivity Data normalization increases numerical stability also called pre-conditioning required because units are different for image and model homogeneous coordinate has little effect, e.g. (100,100,1) arithmetic has limited precision

16 Normalization in DLT Idea: Compute similarity transformation (translation and scaling) which makes the centroid of points zero, and average absolute coordinate value 1 Avg abs coordinate value 1 corresponds to average distance from origin: for 2-D (image) points for 3-D (model) points 2 3 Normalizing transformation can be represented by a matrix acting on homogeneous coordinates

17 Normalization for image points The normalizing transformation T for image is =( 2 / d 0 2 x/ d ) T 0 2 / d 2 y/ d x=1/ N i x i y=1 /N i y i d=1/n i ( x i x) 2 +(y i y) 2 Point after normalization is then got from: normalized x i =T x i original

18 Normalization for model points In a similar fashion, normalizing transformation U for model is D X / D U=( 3/ ) 0 3/ D 0 3 Ȳ / D / D 3 Z / D where X,Ȳ, Z are the average coordinates, and D is the average length similar to previous slide =U

19 Normalization in nutshell 1) Normalization: Determine and use similarity transform T to normalize image points. Determine and use similarity transform U to normalize model points. 2) Perform DLT: Use SVD to solve the projection matrix, using the normalized points. 3) Denormalization: Determine the projection matrix for unnormalized coordinates from M=T 1 M U M

20 So far How to estimate M Next How to decompose M Decomposition = Finding out the calibration parameters from the matrix camera center orientation intrinsic parameters

21 Camera center Camera center C=(X,Y,Z,W) T is point for which MC=0. Can be solved either from SVD of M or from equations: homogeneous coordinates X =det( m 2 m 3 m 4 ) Y = det (m 1 m 3 m 4 ) Z =det( m 1 m 2 m 4 ) W = det (m 1 m 2 m 3 ) determinant m i i:th column of M

22 Orientation and intrinsics 3x3 matrix M=K [ R R C ]=[KR KR C] M=K [ R R C ]=[KR KR C] KR can be decomposed using RQ-decomposition Any non-singular matrix can be decomposed into product RQ where R is upper triangular, Q is orthogonal, similar to QR-decomposition with reversed order Q is then the camera rotation (R) R is the intrinsic calibration matrix (K) Note: R,Q not unique (mirror image)

23 Intrinsics revisited scale factor in x skew K =(α x s o x 0 α y o y ) origin (principal point)

24 Degenerate configurations Projection matrix can only be uniquely determined when the points are in general configuration Degenerate configuration = projection matrix is not unique several (2, infinite) solutions Two most important degenerate configurations camera and all points on a twisted cubic points on union of plane and a line containing camera 3-D analogue of 2-D conic

25 2-D examples of degeneracy points camera centers CAMERA AND POINTS ON CONIC CAMERA AND POINTS ON TWO LINES

26 3-D example of degeneracy NOTE: ALL POINTS ON A PLANE IS A DEGENERATE CONFIGURATION.

27 Geometric error Assume that model points are known very accurately compared to image measurements Geometric error function can then be defined as E= i d ( x i, ; p) 2 d ( x i, ; p) 2 = x i f (, p) 2 d(.,.) is the image-space Euclidean distance between the 2-D point x and 3-D point X projected using parameters p also known as reprojection error Minimization of geometric error requires iterative methods such as gradient descent or Gauss-Newtonmethod. Can you explain gradient descent?

28 Gradient descent Problem: Find the vector p for which a given function E(.) reaches a minimum, or min p E ( p) Solution: Starting with an initial guess p 0, iterate the following update until convergence p t + 1 = p t ρ E ( p ) t p ρ where is the step length (constant or variable) Gradient of E In practice, other methods such as conjugate gradient method are superior in multi-dimensional problems. Can we do better?

29 Minimization for sums of squares Error functions often sums of squares. Structure can be exploited. Write problem using residuals E ( p)= i d (x i, ; p) 2 = i d i ( p) 2 = d ( p) 2 Problem is then min d ( p) 2

30 Gauss-Newton method Approximate residual d by 1st order Taylor series d ( p t + 1 ) d ( p t )+ d p p= p t ( p t + 1 p t ) =d ( p t )+ J Δ J is the Jacobian of d, delta is the step from current to next p Minimize the length of residual vector arg min d ( p t +1 ) 2 arg min d ( p t )+ J Δ 2 Right-hand-side is now a linear least squares problem, with solution Δ = (J T J ) 1 J T Gauss-Newton d ( p i ) solution p t + 1 = p t +Δ

31 Gauss-Newton method Approximate residual d by 1st order Taylor series d ( p t + 1 )=d ( p t )+ d p p= p t (d( p t + 1 ) d ( p t )) =d ( p t )+ J Δ J is the Jacobian of d, delta is the step from current to next p Minimize the length of residual vector arg min d ( p t +1 ) 2 arg min d ( p t )+ J Δ 2 Right-hand-side is now a linear least squares problem, with solution Δ = (J T J ) 1 J T Gauss-Newton d ( p i ) solution p t + 1 = p t +Δ That's the theory for now, we will see applications later.

32 Lens distortion parameters Optimization of lens distortion parameters requires iterative methods (numeric minimization) e.g. Gauss-Newton Geometric error is usually minimized Good news: Good camera calibration routines are publicly available (Matlab camera calibration toolbox, etc.)

33 Summary You should know: What is calibration and why it is necessary? What are the parameters that need to be calibrated? How the projection matrix can be estimated? How the parameters can be found from projection matrix? How to create a calibration pattern?

34 References Trucco & Verri, chapter 6 Hartley & Zisserman, chapters 4 and 7 Software OpenCV calibration module

35 Next time: Pose estimation How to determine the pose of a 3-D target?