Single View Geometry. Camera model & Orientation + Position estimation. What am I?

Transcription

1 Single View Geometr Camera model & Orientation + Position estimation What am I?

2 Ideal case: c Projection equation: x = f X / Z = f Y / Z c p f C x c Zx = f X Z = f Y Z = Z

3 Step 1: Camera projection matrix Zx = f X Z = f Y Z = Z c f f X c Y c Z c 1 = x 1 Z c c p x c f C P 0 X = x

4 Step 2: Intrinsic camera parameters: map camera coordinate to pixel coordinate c K α x s p x 0 α p (3x3 submatrix) x Optical world x = Pixel world c p x c f C α x, α is pixel scaling factor p x, p is the principle point (where optical axis hits image plane) s is the slant factor, when the image plane is not normal to the optical axis

5 Combine the Intrinsic camera parameters α x s p x 0 α p f f X c Y c Z c 1 = x 1 K (Calibration matrix) α x f s p x 0 0 α f p P 0 X = x x c c c 1 = x P = [K, 0] = K [I, 0]

6 Step 3: External parameters: rotation and translation map the world to camera coordinates 3x3 3x1 (R,t) Camera coordinate c World coordinate c p f C x c

7 Combining Internal and External parameters 1) (R,t) Camera coordinate c World coordinate c p f C 2) x c 1) Translate the world coordinate into the camera coordinate 2) Translate the camera coordinate into the pixel coordinate

8 Combining Internal and External parameters (R,t) Camera coordinate c World coordinate After simplication: c p f C x c x = K [R, t] X pixel world

9 Special case, planar world, homograph X= (xw,w,w=0,1) x = K [R, t] X Expand: x = K r1 r2 r3 t xw w w=0 1 World coordinate c (R,t) c p C f x c Camera coordinate

10 Special case, planar world, homograph X= (xw,w,w=0,1) (R,t) After simplication: x = K r1 r2 t xw w World coordinate c c p C f x c Camera coordinate 1 We have the homographic mapping: X = K H3x3 X

11 Special case: rotating camera x = K [R, t] X with t = 0 Expand: x = K [R] X World coordinate c (R,t) p f x c c C Camera coordinate X = xw w w This is also a homograph with H = R

12 Panoramas x = K [R] X

13 Visual Odometr: Overview 1) The transformation (R,t) tells us how to rotate + move the world coordinate to the camera coordinate. 2) In the following slides, we will see 1) how to get the world reference s frame orientation+position from (R,t), 2) how to get the pan/tilt/aw angles from R 3) If we want to know the camera orientation+position, we need to look at its inverse transform, which we should see is, 1) The computation of the camera pan/tilt/aw angle is the same as in the previous case World coordinate c (R,t) p f x c c C

14 Recover Camera orientation Case 1: single vanishing point (direction)

15 Pan/tilt/aw angles Pan : rotation around -axis Yaw :rotation around -axis Tilt : rotation around x-axis x x

16 Rotation: Pan/Tilt/Yaw Given the 3x3 rotation matrix R, we can recover, pan/tilt/aw angle, and vise. vica. cam world, (0,0,1) T x cam,

17 The basic idea is that if we can see the north star, we know how to oriented ourselves (minus aw angle) (wh?) And the ``north star is just the point of infinit in some direction (R,t) Camera coordinate c World coordinate c p f C x c

18 Seeing the vanishing point can tell us: Camera projection equation: K R t x=k [R,t] X, or x w w w t w = x c c c [R,t]= t r 1 r 2 r 3 Columns of R matrix = image of vanishing points of world-space axes! r 1,r 2,r 3 == image of x,, axis vanishing points r 1 r 2 r 3 t = r 3 -direction (need to normalie to norm =1) vanishing point

19 Recover Pan/Tilt angle from -vanishing point Geometric explanation: Pan Tilt The alignment between -axis of the world and image is defined onl b the pan/tilt angle: cam world, (0,0,1) T r3 = x cam, cam Note: tilt angle with rotation around x-axis is positive when pointing down in this figure

20 Sanit check Vanishing point in

21 In this example, we are walking down a track, the ground is tilted up(we are going up the hill). Assume we are b meter tall. We see two parallel lines to our left and right, with x = 1,-1. A line on the ground is described b equation = a - b, where a is the slope the ground. Image plane x x = 1 = a -b x = -1 Ground plane So we know: = a -b Tilt angle

22 As a line gets far awa, -> infinit. If (1,,) is a point on this line, its image is f(1/,/). As -> infinit, 1/ -> 0. What about /? / = (a-b)/ = a - b/ -> a. So a point on the line appears at: (0,a). Image plane x VanishingPoint (0,a) = a - b Ground plane

23 world, (0,0,1) T r = = Now, we check x (0,a) = a - b

24 Recover Pan/Tilt angle from -vanishing point Geometric explanation: Pan Tilt The alignment between -axis of the world and image is defined onl b the pan/tilt angle: cam world, (0,0,1) T r 3 = x cam, cam

25 Recover Camera orientation Case 2: two vanishing points, x- direction

26 In some case, we can t measure the -vanishing point directl, but if we have x- vanishing points, we can imagine what the -vanishing point

27 Basic idea: 1) x- vanishing point gives us the first two columns of R, r1 and r2 2) -vanishing point, the third columns of R, can be computed b r3 = r1 x r2 Recall: vanishing points of x, direction, give us the, r1 and r2 r 1 r 2 r 3 t first two columns of R

28 Steps for Recovering R from Two vanishing Point 1. Measure X / Y direction vanishing point

29 Recovering R from Two vanishing Point (2) 2. Translate to the camera coordinates b intrinsic matrix K

30 Recovering R from Two vanishing Point (3) 3. Get r 1 and r 2 from vanishing point x vx and x vy vanishing points of x, direction, give us the, r1 and r2 r 1 r 2 r 3 t first two columns of R

31 4. Get r 3 Recovering R from Two vanishing Point (4) 5. Get R r3 = r1 x r2 r1 r2 r3

32 Example Recover Camera Pan /Tilt From r3

33 Recover Camera Pan /Tilt From r 3 (2) Examples of Camera Pan/Tilt angle - - x α x

34 R= Recovering all three angles: aw + pan/tilt r 1 r 2 r 3 Depends on pan/tilt Depends on aw/pan/tilt 1) Write out equation in aw/pan/tilt angles for r1 r2 2) Solve for aw angle given pan/tilt angles

35 Summar so far 1) If we have the vanishing point in -direction, we can recover pan/tilt angle 1) This is when the planar target is on the ground 2) If don t see the -direction vanishing point directl, ou would need two vanishing points of x- directions 3) Given x- directions vanishing point, we can cover the full rotation matrix, therefore its -direction (pan/tilt) 4) Given pan/tilt + x-direction vanishing point, we can recover aw angle

36 Recover Camera orientation + position Case 3: homograph matrix

37 Recall for planar surface (R,t) H3x3 x = K r1 r2 t xw w World coordinate c c p C f x c Camera coordinate 1 We have the homographic mapping: X = K H3x3 X

38 Recall for planar surface (R,t) H3x3 x = K r1 r2 t xw w World coordinate c c p C f x c Camera coordinate 1 This implies if we have H, 1) We can recover the full rotation matrix, R, 2) We can recover the position vector t

39 Example Recover [R,t] from H (1) 1. Get H from 4 points 2. Computer the norm of first column

40 3. Computer r 1, r 2, t Example Recover [R,t] from H (2) 4. Computer r 3

41 Procedure of recover R, and t from H 1) Compute the rotation of x-axis, set x = K 2) Compute the rotation of the -axis, set r1 r2 t xw w 1 3) Compute the translation vector t (the position of the robot) 4) Compute the rotation of the -axis, set 5) Use the previous case(2), to compute the pan/tilt/aw angle

42 Summar so far 1) Each vanishing point in the image gives ou one rotation vector 2) Given the -vanishing point, we can detect the pan/tilt angle. 1) This is the case when the planar target is on the ground. 3) Given x- vanishing point, we can compute the -vanishing point(therefore pan/tilt angle). 1) This is the case when we are facing the planar surface 2) Given x-rotation vector, pan-tilt angle, we can determine the aw angle 3) But we can t determine the position of the robot 4) If we are given the homograph, (using 4 corresponding points), we can recover both the rotational angles, and the position of the robot.

43 How to estimate the rotation and translation of the robot from the world point of view? In the case of moving robot(rather than moving target), we need to know the orientation/position of the robot in the world ==> we need to how to pan/tilt the world oriented to the robot. Note: pan/tilt of the camera is ver different from the pan/tilt of the world!

44 Converting rotation/translation from (world->camera) to (camera->world) (R,t) (world-> camera) Camera coordinate c World coordinate c p f C x c = With the fact: (camera->world)

45 Converting rotation/translation from (world->camera) to (camera->world) What does it mean: 1) The position of the camera in the world reference frame is: 2) The pan/tilt/aw of the world to align with camera is computed from the rows of the R instead of columns of R World coordinate c (R,t) (world-> camera) Camera coordinate c p C f x c (camera-> world)

46 Recovering World pan/tilt from Rotation matrix R^T 1. Compute the (world->camera) rotation R and translation t 2. Take the last row of the R, and recover pan/tilt angle of camera 3) Compute the aw angle if needed, from first two rows of R 4) Compute the position of the robot, b

47 Example Recover Pan/Tilt in World Frame From World-> Camera: [R, t] èfrom Camera -> World : [R T, -R T t] Tilt x Pan x

48 Example Recover Translation in World Frame (2) target x

49 Final notes 1) There are two rotation/translation we care about: 1) (R,t): world->camera transformation. It tells us parameters of the camera. The columns of R: rotation of camera so it aligns with the world. The vector t = coordinate of the world center in camera frame. 2) : camera-> world. It tells us parameters of the world. Columns of R^t: rotation of the world so it aligns with the camera. The vector R^t(-t) = coordinate of the camera in the world frame. 3) For the visual odometr task used in project 2, ou need to use the transformation. 4) The pan angle of the world, is the robot orientation in the world frame.