Single View Geometry. Camera model & Orientation + Position estimation. What am I?

Transcription

1 Single View Geometry Camera model & Orientation + Position estimation What am I?

2 Vanishing point

3 Mapping from 3D to 2D Point & Line Goal:

4 Point Homogeneous coordinates represent coordinates in 2 dimensions with a 3-vector & $ % x y #! " & $ '' homogeneous '''' coords $ $ % x y 1 #!!!"

5 The projective plane Why do we need homogeneous coordinates? represent points at infinity, homographies, perspective projection, multi-view relationships What is the geometric intuition? a point in the image is a ray in projective space (0,0,0) z y x (x,y,1) (sx,sy,s) image plane Each point (x,y) on the plane is represented by a ray (sx,sy,s) all points on the ray are equivalent: (x, y, 1) (sx, sy, s)

6 Projective Lines

7 Projective lines What does a line in the image correspond to in projective space? A line is a plane of rays through origin all rays (x,y,z) satisfying: ax + by + cz = 0 in vector notation : 0 = [ a b c] x y z p A line is also represented as a homogeneous 3-vector l l

8 Line Representation a line is is the distance from the origin to the line is the norm direction of the line It can also be written as

9 Example of Line

10 0.42*pi Example of Line (2)

11 Homogeneous representation Line in Is represented by a point in : But correspondence of line to point is not unique We define set of equivalence class of vectors in R^3 - (0,0,0) As projective space

12 Projective lines from two points

13 Line passing through two points Two points: x Define a line l is the line passing two points Proof:

14 Line passing through two points More specifically,

15 Matlab codes function l = get_line_by_two_points(x, y) x1 = [x(1), y(1), 1]'; x2 = [x(2), y(2), 1]'; l = cross(x1, x2); l = l / sqrt(l(1)*l(1)+l(2)*l(2));

16 Verify p2 lies on the line >> 218*0.6+13*0.8 ans = (close) Example of Line P1 P2

17 Test line: >> p1 = [42,142,1]; >> p2=[218,13,1]; >> l = cross(p1,p2) l = Example of Line P1 P2 >> l = l/sqrt(l(1)^2+l(2)^2) l =

18 >> x = [221;23]; >> y = [218;268]; >> l = get_line_by_two_points(x,y); >> l l =

19 Projective lines from two points When does the line has the form (a, b, 0)? -all lines passing through the optical center When does the line has the form (0, 0, 1)? -line at infinity

20 Points from two lines

21 Intersection of lines Given two lines:, Define a point l X is the intersection of the two lines

22 More specifically, Intersection of lines

23 Matlab codes function x0 = get_point_by_two_line(l, l1) x0 = cross(l, l1); x0 = [x0(1)/x0(3); x0(2)/x0(3)];

24 Example of Lines Intersection

25 Point and line duality A line l is a homogeneous 3-vector It is to every point (ray) p on the line: l p=0 l p 1 p 2 l 2 l 1 p What is the line l spanned by rays p 1 and p 2? l is to p 1 and p 2 l = p 1 p 2 l is the plane normal What is the intersection of two lines l 1 and l 2? p is to l 1 and l 2 p = l 1 l 2 Points and lines are dual in projective space given any formula, can switch the meanings of points and lines to get another formula

26 Points from two lines When P has the form (x,y,0)?

27 Point at infinity Example: Consider two parallel horizontal lines: Intersection = Point at infinity in the direction of y

28 Point at infinity, Ideal points Intersection: Any point (x 1,x 2,0) is intersection of lines at infinity

29 Points at infinity Under projective transformation, All parallel lines intersects at the point at infinity One point at infinity ó one parallel line direction Where are the points at infinity in the image plane?

30 Line at infinity A line passing all points at infinity: Because :

31 Ideal points and lines y (sx,sy,0) (a,b,0) y z x image plane Ideal point ( point at infinity ) p (x, y, 0) parallel to image plane It has infinite image coordinates Ideal line l (a, b, 0) parallel to image plane z x image plane Corresponds to a line in the image (finite coordinates)

32 Comparing heights Vanishing Point

33 Measuring height Camera height

34 Measuring height v z r vanishing line (horizon) v x v t 0 H t R H v y b 0 b

35 Focal Length

36 A Lighting Reproduction Approach to Live-Action Compositing, P. Debevec 2002

37 FOV depends of Focal Length f f Smaller FOV = larger Focal Length

38 Field of View (Zoom)

39 Field of View (Zoom)

40 Large Focal Length compresses depth 400 mm 200 mm 100 mm 50 mm 28 mm 17 mm Michael Reichmann

41

42 Nikon s mm f/5.6-8p lens.

43 Nikon s mm f/5.6-8p lens.

44

45 " /012'+$3+/4$&'()5$2 35'0:+$3 +>5'?+!6 ;<=6 $%&'()*+,+-+. ;<=66!! # 36+!+3$(10+0'27)8 366+!+3$(10+0'27)8 9,+!+:5*)12('+)$+$%&'()+, 9.+!+:5*)12('+)$+$%&'()+.

46 Canon D30 takes image with 2160x1440 pixels. What is the pixel size?

47 What f to make tower appear half the size of image?

48 Fix f=50mm, D2 =? in order to make the tower 1/3 size of image?

49 D1=? to make human the same height as tower?

50 What f to make tower appear twice as big as before (2/3 the height)?

51 Setting f =100mm, D1=? in order to keep the original size of human? (1/3 the height)

52 Lens distortion with focal length

53

54

55 Lens distortion with focal length

56

57

58 Camera projection model The overall goal is to compute 3D geometry of the scene from just 2D images. We will first study 3D->2D camera projection using linear algebra.

59 The Pinhole Camera Light enters a darkened chamber through a pinhole opening and forms an image on the further surface

60 The Pinhole Camera Model

61 Ideal case: y c Projection equation: x = f X / Z y = f Y / Z y z c z y p f C x c Zx = f X Zy = f Y Z = Z

62 Step 1: Camera projection matrix Zx = f X Zy = f Y Z = Z y c f f X c Y c Z c 1 = x y 1 Z c y z c z y p x c f C P 0 X = x

63 Step 2: Intrinsic camera parameters: map camera coordinate to pixel coordinate y c K α x s p x 0 α y p y (3x3 submatrix) x y z Optical world x y z = y Pixel world z c z y p x c f C α x, α y is pixel scaling factor p x, p y is the principle point (where optical axis hits image plane) s is the slant factor, when the image plane is not normal to the optical axis

64 Combine the Intrinsic camera parameters α x s p x 0 α y p y f f X c Y c Z c 1 = x y 1 K (Calibration matrix) α x f s p x 0 0 α y f p y P 0 X = x x c y c z c 1 = x y z P = [K, 0] = K [I, 0]

65 Step 3: External parameters: rotation and translation map the world to camera coordinates 3x3 3x1 (R,t) Camera coordinate y c World coordinate y z c z y p f C x c

66 Combining Internal and External parameters 1) (R,t) Camera coordinate y c World coordinate y z c z y p f C 2) x c 1) Translate the world coordinate into the camera coordinate 2) Translate the camera coordinate into the pixel coordinate

67 Combining Internal and External parameters (R,t) Camera coordinate y c World coordinate After simplication: y z c z y p f C x c x = K [R, t] X pixel world

68 Special case, planar world, homograph X= (xw,yw,zw=0,1) x = K [R, t] X Expand: x = K r1 r2 r3 t xw yw zw=0 1 World coordinate y z c z (R,t) y c y p C f x c Camera coordinate

69 Special case, planar world, homograph X= (xw,yw,zw=0,1) (R,t) After simplication: x = K r1 r2 t xw yw World coordinate y z c z y c y p C f x c Camera coordinate 1 We have the homographic mapping: X = K H3x3 X

70 Special case: rotating camera x = K [R, t] X with t = 0 Expand: x = K [R] X World coordinate y z c z (R,t) y p f x c y c C Camera coordinate X = xw yw zw This is also a homography with H = R

71 Panoramas x = K [R] X

72 Visual Odometry: Overview 1) The transformation (R,t) tells us how to rotate + move the world coordinate to the camera coordinate. 2) In the following slides, we will see 1) how to get the world reference s frame orientation+position from (R,t), 2) how to get the pan/tilt/yaw angles from R 3) If we want to know the camera orientation+position, we need to look at its inverse transform, which we should see is, 1) The computation of the camera pan/tilt/yaw angle is the same as in the previous case World coordinate y z c z (R,t) y p f x c y c C

73 Recover Camera orientation Case 1: single vanishing point (zdirection)

74 Pan/tilt/yaw angles Pan : rotation around y-axis Yaw :rotation around z-axis Tilt : rotation around x-axis z x y x y z

75 Rotation: Pan/Tilt/Yaw Given the 3x3 rotation matrix R, we can recover, pan/tilt/yaw angle, and vise. vica. y cam z world, (0,0,1) T x cam,

76 The basic idea is that if we can see the north star, we know how to oriented ourselves (minus yaw angle) (why?) And the ``north star is just the point of infinity in some direction (R,t) Camera coordinate y c World coordinate y z c z y p f C x c

77 Seeing the vanishing point can tell us: Camera projection equation: K R t x=k [R,t] X, or x w y w z w t w = x c y c z c [R,t]= t r 1 r 2 r 3 Columns of R matrix = image of vanishing points of world-space axes! r 1,r 2,r 3 == image of x,y,z axis vanishing points r 1 r 2 r 3 t = r 3 z-direction (need to normalize to norm =1) vanishing point

78 Recover Pan/Tilt angle from z-vanishing point Geometric explanation: Pan Tilt The alignment between z-axis of the world and image is defined only by the pan/tilt angle: y cam z world, (0,0,1) T r3 = x cam, z cam Note: tilt angle with rotation around x-axis is positive when pointing down in this figure

79 Sanity check Vanishing point in z

80 In this example, we are walking down a track, the ground is tilted up(we are going up the hill). Assume we are b meter tall. We see two parallel lines to our left and right, with x = 1,-1. A line on the ground is described by equation y = az - b, where a is the slope the ground. Image plane y z x x = 1 y = az -b x = -1 Ground plane So we know: y y = az -b Tilt angle z

81 As a line gets far away, z -> infinity. If (1,y,z) is a point on this line, its image is f(1/z,y/z). As z -> infinity, 1/z -> 0. What about y/z? y/z = (az-b)/z = a - b/z -> a. So a point on the line appears at: (0,a). Image plane x VanishingPoint (0,a) y z y = az - b Ground plane

82 z world, (0,0,1) T r z = = Now, we check x (0,a) y z y = az - b

83 Recover Pan/Tilt angle from z-vanishing point Geometric explanation: Pan Tilt The alignment between z-axis of the world and image is defined only by the pan/tilt angle: y cam z world, (0,0,1) T r 3 = x cam, z cam