Camera Projection Models We will introduce different camera projection models that relate the location of an image point to the coordinates of the

Transcription

1 Camera Projection Models We will introduce different camera projection models that relate the location of an image point to the coordinates of the corresponding 3D points. The projection models include: full perspective projection model, weak perspective projection model, affine projection model, and orthographic projection model. 1

2 The Pinhole Camera Model Based on simply trigonometry (or using 3D line equations), we can derive u = fx c z c v = fy c z c 2

3 The Computer Vision Camera Model P p optical axis Image plane f optical center 3

4 u = fx c z c v = fy c z c where f z c is referred to as isotropic scaling.the full perspective projection is non-linear. 4

5 Weak Perspective Projection If the relative distance δz c (scene depth) between two points of a 3D object along the optical axis is much smaller than the average distance z c to the camera (δz < z 20 ), i.e, z c z c then u = f x c z c fx c z c v = f y c z c fy c z c We have linear equations since all projections have the same scaling factor. 5

6 Orthographic Projection As a special case of the weak perspective projection, when f z c factor equals 1, we have u = x c and v = y c, i.e., the lins (rays) of projection are parallel to the optical axis, i.e., the projection rays meet in the infinite instead of lens center. This leads to the sizes of image and the object are the same. This is called orthgraphic projection. 6

7 7

8 Perspective projection geometry y c camera frame C c x c image plane perspective center row-column 01 frame C p focal length c v f r P i u principal point image frame C i optical axis z c P z y x object frame C o Figure 1: Perspective projection geometry 8

9 Notations Let P = (x y z) t be a 3D point in object frame and U = (u v) t the corresponding image point in the image frame before digitization. Let X c = (x c y c z c ) t be the coordinates of P in the camera frame and p = (c r) t be the coordinates of U in the row-column frame after digitization. 9

10 Projection Process Our goal is to go through the projection process to understand how an image point (c,r) is generated from the 3D point (x,y,z). 3D point in object frame 10 Affine transformation 3D point in camera frame Perspective projection Image point in image frame Spatial Sampling Image point in row-column frame

11 Relationships between different frames Between camera frame (C c ) and object frame (C o ) X c = RX +T (1) X is the 3D coordinates of P w.r.t the object frame. R is the rotation matrix and T is the translation vector. R and T specify the orientation and position of the object frame relative to the camera frame. 11

12 R and T can be parameterized as R = r 11 r 12 r 13 r 21 r 22 r 23 T = t x t y r 31 r 32 r 33 t z r i = (r i1,r i2,r i3 ) be a 1 x 3 row vector, R can be written as R = r 1 r 2 r 3 12

13 Substituting the parameterized T and R into equation 1 yields x c y x z c = r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 x y z + t x t y t z (2) 13

14 Between image frame (C i ) and camera frame (C c ) Perspective Projection: Hence, u = fx c z c v = fy c z c X c = x c y c z c = λ u v f (3) 14

15 where λ = z c f is a scalar and f is the camera focal length. 15

16 Relationships between different frames (cont d) Between image frame (C i ) and row-col frame (C p ) (spatial quantization process) c r = s x 0 0 s y u v + c 0 r 0 (4) where s x and s y are scale factors (pixels/mm) due to spatial quantization. c 0 and r 0 are the coordinates of the principal point in pixels relative to C p 16

17 Collinearity Equations Combining equations 1 to 4 yields c = s x f r 11x+r 12 y +r 13 z +t x r 31 x+r 32 y +r 33 z +t z +c 0 r = s y f r 21x+r 22 y +r 23 z +t y r 31 x+r 32 y +r 33 z +t z +r 0 17

18 Homogeneous system: perspective projection In homogeneous coordinate system, equation 3 may be rewritten as λ Note λ = z c. u v 1 = f f x c y c z c (5) 18

19 Homogeneous System: Spatial Quantization Similarly, in homogeneous system, equation 4 may be rewritten as c r 1 = s x 0 c 0 0 s y r u v 1 (6) 19

20 Homogeneous system: quantization + projection Substituting equation 5 into equation 6 yields λ c r 1 = s x f 0 c s y f r x c y c z c 1 (7) where λ = z c. 20

21 Homogeneous system: Affine Transformation In homogeneous coordinate system, equation 2 can be expressed as x c y c z c 1 = r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z x y z 1 (8) 21

22 Homogeneous system: full perspective Combining equation 8 with equation 7 yields λ c r 1 = s x fr 1 +c 0 r 3 s x ft x +c 0 t z s y fr 2 +r 0 r 3 s y ft y +r 0 t z r 3 t z } {{ } P where r 1, r 2, and r 3 are the row vectors of the rotation matrix R, λ = z c is a scaler and matrix P is called the homogeneous projection matrix. x y z 1 (9) 22

23 P = WM where W = M = ( fs x 0 c 0 0 fs y r R T ) W is often referred to as the intrinsic matrix and M as exterior matrix. Since P = WM = [WR WT], for P to be a projection 23

24 matrix, Det(WR) 0, i.e., Det(W) 0. 24

25 Weak Perspective Camera Model For weak perspective projection, we have z c z c, i.e., z c r t 3 X +t z Hence, u = fx c z c v = fy c z c Hence, u v = f z c x c y c 25

26 Or u v 1 = f z c x c y c z c f Since c r 1 = s x 0 c 0 0 s y r u v 1 26

27 We have c r 1 = f z c s x 0 c 0 0 s y r x c y c z c f Since x c y c = [R 2 T 2 ] x y z 1 where R 2 is the first two rows of R and T 2 is the first two 27

28 elements of T. Or x c y c z c f = R 2 T z c f x y z 1 Hence, c r 1 = f z c s x 0 c 0 0 s y r R 2 T z c f x y z 1 28

29 = 1 z c s x 0 c 0 0 s y r fr 2 ft z c x y z 1 = 1 z c fs x 0 c 0 0 fs y r R 2 T z c x y z 1 29

30 Weak Perspective Camera Model The weak perspective projection matrix is P weak = fs x r 1 fs x t x +c 0 z c fs y r 2 fs y t y +r 0 z c (10) z c where r 1 and r 2 are the first two rows of R 2 and z c = r 3 X +tz. 30

31 Weak Projection Camera Model Another possible solution is as follows 31

32 32

33 Affine Camera Model A further simplification from weak perspective camera model is the affine camera model, which is often assumed by computer vision researchers due to its simplicity. The affine camera model assumes that the object frame is located on the centroid of the object being observed. As a result, we have z c t z, the affine perspective projection matrix is P affine = s x fr 1 s x ft x +c 0 t z s y fr 2 s y ft y +r 0 t z (11) 0 t z 33

34 Affine camera model represents the first order approximation of the full perspective projection camera model. It still only gives an approximation and is no longer useful when the object is close to the camera or the camera has a wide angle of view. 34

35 Orthographic Projection Camera Model Under orthographic projection, projection is parallel to the camera optical axis. therefore we have u = x c v = y c which can be approxmiated by f z c 1. The orthographic projection matrix can therefore be 35

36 obtained as P orth = s x r 1 s x t x +c 0 s y r 2 s y t y +r (12) 36

37 Non-full perspective Projection Camera Model The weak perspective projection, affine, and orthographic camera model can be collectively classified as non-perspective projection camera model. In general, the projection matrix for the non-perspective projection camera model λ c r 1 = p 11 p 12 p 13 p 14 p 21 p 22 p 23 p p 34 x y z 1 37

38 Dividing both sides by p 34 (note λ = p 34 ) yields c r = M 2 3 x y z + v x v y where m ij = p ij /p 34 and v x = p 14 /p 34, v y = p 24 /p 34 For any given reference point (c r,r r ) in image and (x 0,y 0,z 0 ) in space, the relative coordinates ( c, r) in image and ( x,ȳ, z) in space are 38

39 c r = c c r r r r and x ȳ z = x x r y y r z z r It follows that the basic projection equation for the affine and weak perspective model in terms of relative coordinates is c r = M 2 3 An non-perspective projection camera M 2 3 has 3 39 x ȳ z

40 independent parameters. The reference point is often chosen as the centroid since centroid is preserved under either affine or weak perspective projection. Given the weak projection matrix P, P = fs x r 1 fs x t x +c 0 z c fs y r 2 fs y t y +r 0 z c 0 z c The M matrix is M = fs x r 1 z c fs y r 2 z c 40

41 = f z c = f z c s xr 1 s y r 2 s x 0 0 s y r 1 r 2 For affine projection, z c = t z, for orthographic projection, f z c = 1. If we assume s x = s y, then M = fs x z c r 1 r 2 Then, we have only four parameters: three rotation angles and a scale factor. 41

42 Rotation Matrix Representation: Euler angles Assume rotation matrix R results from successive Euler rotations of the camera frame around its X axis by ω, its once rotated Y axis by φ, and its twice rotated Z axis by κ, then R(ω,φ, κ) = R X (ω)r Y (φ)r Z (κ) where ω, φ, and κ are often referred to as pan, tilt, and swing angles respectively. 42

43 Rotation Matrix Representation: Euler angles X pan angle swing angle Z (optical axis) tilt angle Y 43

44 R x (ω) = R y (φ) = R z (κ) = cosω sinω 0 sinω cosω cosφ 0 sinφ sinφ 0 cosφ cosκ sinκ 0 sinκ cosκ

45 Rotation Matrix: Rotation by a general axis Let the general axis be ω = (ω x,ω y,ω z ) and the rotation angle be θ. The rotation matrix R resulting from rotating around ω by θ can be expressed as Rodrigues rotation formula gives an efficient method for computing the rotation matrix. 45

46 Quaternion Representation of R The relationship between a quaternion q = [q 0,q 1,q 2,q 3 ] and the equivalent rotation matrix is Here the quaternion is assumed to have been scaled to unit length, i.e., q = 1. The axis/angle representation ω/θ is strongly related to a quaternion, according to the formula 46

47 cos(θ/2) ω x sin(θ/2) ω y sin(θ/2) ω z sin(θ/2) where ω = (ω x,ω y,ω z ) and ω = 1. 47

48 R s Orthnormality The rotation matrix is an orthnormal matrix, which means its rows (columns) are normalized to one and they are orthonal to each other. The orthnormality property produces R t = R 1 48

49 Interior Camera Parameters Parameters (c 0,r 0 ), s x, s y, and f are collectively referred to as interior camera parameters. They do not depend on the position and orientation of the camera. Interior camera parameters allow us to perform metric measurements, i.e., to convert pixel measurements to inch or mm. 49

50 Exterior Camera Parameters Parameters like Euler angles ω, φ, κ, t x, t y, and t z are collectively referred to as exterior camera parameters. They determine the position and orientation of the camera. 50

51 Camera Calibration and Pose Estimation The purpose of camera calibration is to determine intrinsic camera parameters: c 0,r 0, s x, s y, and f. Camera calibration is also referred to as interior orientation problem in photogrammetry. The goal of pose estimation is to determine exterior camera parameters: ω, φ, κ, t x, t y, and t z. In other words, pose estimation is to determine the position and orientation of the object coordinate frame relative to the camera coordinate frame or vice versus. 51

52 Perspective Projection Invariants Distances and angles are invariant with respect to Euclidian transformation (rotation and translation). The most important invariant with respect to perspective projection is called cross ratio. It is defined as follows: 52

53 D C B A τ( A,B,C,D)= AC BC AD BD Cross-ratio is preserved under perspective projection. 53

54 Projective Invariant for non-collinear points Cross ratio of intersection points between a set of pencil of 4 lines and another line are only function of the angles among the pencil lines, independent of the intersection points on the lines. cross-ratio may be used for ground plane detection from multiple image frames. Chasles theorem: 54

55 Let A, B, C, D be distinct points on a (non-singular) conic (ellipse, circle,..). If P is another point on the conic then the cross-ratio of intersections points on the pencil PA, PB, PC, PD does not depend on the point P. This means given A,B,C, and D, all points P on the same ellipse should satisfy Chasles s theorem. This theorem may be used for ellipse detection. See section 19.3 and 19.4 of Daves book. 55