Single View Geometry. Camera model & Orientation + Position estimation. What am I?

Single View Geometry Camera model & Orientation + Position estimation What am I?

Vanishing points & line

http://www.wetcanvas.com/

http://pennpaint.blogspot.com/

http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-thesame-direction-as-the-lines-in-order-to-find-their-vanishing-point.html

http://dd.salgoodsam.com/

Focal Length

The Pinhole Camera Light enters a darkened chamber through a pinhole opening and forms an image on the further surface

The Pinhole Camera Model

FOV depends of Focal Length f f Smaller FOV = larger Focal Length

Field of View (Zoom)

Nikon s 1200-1700mm f/5.6-8p lens.

Large Focal Length compresses depth 400 mm 200 mm 100 mm 50 mm 28 mm 17 mm 1995-2005 Michael Reichmann

http://www.creativepro.com/img/story/102604_fg3b.jpg

" /012'+$3+/4$&'()5$2 35'0:+$3 +>5'?+!6 ;<=6 $%&'()*+,+-+. ;<=66!! # 36+!+3$(10+0'27)8 366+!+3$(10+0'27)8 9,+!+:5*)12('+)$+$%&'()+, 9.+!+:5*)12('+)$+$%&'()+.

Canon D30 takes image with 2160x1440 pixels. What is the pixel size?

What f to make tower appear half the size of image?

Fix f=50mm, D2 =? in order to make the tower 1/3 size of image?

D1=? to make human the same height as tower?

What f to make tower appear twice as big as before (2/3 the height)?

Setting f =100mm, D1=? in order to keep the original size of human? (1/3 the height)

Moment: re-experience

Camera projection model The overall goal is to compute 3D geometry of the scene from just 2D images. We will first study 3D->2D camera projection using linear algebra.

The Pinhole Camera Light enters a darkened chamber through a pinhole opening and forms an image on the further surface

The Pinhole Camera Model

Ideal case: y c Projection equation: x = f X / Z y = f Y / Z y z c z y p f C x c Zx = f X Zy = f Y Z = Z

Step 1: Camera projection matrix Zx = f X Zy = f Y Z = Z y c f 0 0 0 0 f 0 0 0 0 1 0 X c Y c Z c 1 = x y 1 Z c y z c z y p x c f C P 0 X = x

Step 2: Intrinsic camera parameters: map camera coordinate to pixel coordinate y c K α x s p x 0 α y p y 0 0 1 (3x3 submatrix) x y z Optical world x y z = y Pixel world z c z y p x c f C α x, α y is pixel scaling factor p x, p y is the principle point (where optical axis hits image plane) s is the slant factor, when the image plane is not normal to the optical axis

Combine the Intrinsic camera parameters α x s p x 0 α y p y 0 0 1 f 0 0 0 0 f 0 0 0 0 1 0 X c Y c Z c 1 = x y 1 K (Calibration matrix) α x f s p x 0 0 α y f p y 0 0 0 1 0 P 0 X = x x c y c z c 1 = x y z P = [K, 0] = K [I, 0]

Step 3: External parameters: rotation and translation map the world to camera coordinates 3x3 3x1 (R,t) Camera coordinate y c World coordinate y z c z y p f C x c

Combining Internal and External parameters 1) (R,t) Camera coordinate y c World coordinate y z c z y p f C 2) x c 1) Translate the world coordinate into the camera coordinate 2) Translate the camera coordinate into the pixel coordinate

Combining Internal and External parameters (R,t) Camera coordinate y c World coordinate After simplication: y z c z y p f C x c x = K [R, t] X pixel world

Special case, planar world, homograph X= (xw,yw,zw=0,1) x = K [R, t] X Expand: x = K r1 r2 r3 t xw yw zw=0 1 World coordinate y z c z (R,t) y c y p C f x c Camera coordinate

Special case, planar world, homograph X= (xw,yw,zw=0,1) (R,t) After simplication: x = K r1 r2 t xw yw World coordinate y z c z y c y p C f x c Camera coordinate 1 We have the homographic mapping: X = K H3x3 X

Special case: rotating camera x = K [R, t] X with t = 0 Expand: x = K [R] X World coordinate y z c z (R,t) y p f x c y c C Camera coordinate X = xw yw zw This is also a homography with H = R

Panoramas x = K [R] X

Visual Odometry: Overview 1) The transformation (R,t) tells us how to rotate + move the world coordinate to the camera coordinate. 2) In the following slides, we will see 1) how to get the world reference s frame orientation+position from (R,t), 2) how to get the pan/tilt/yaw angles from R 3) If we want to know the camera orientation+position, we need to look at its inverse transform, which we should see is, 1) The computation of the camera pan/tilt/yaw angle is the same as in the previous case World coordinate y z c z (R,t) y p f x c y c C

Recover Camera orientation Case 1: single vanishing point (zdirection)

Pan/tilt/yaw angles Pan : rotation around y-axis Yaw :rotation around z-axis Tilt : rotation around x-axis z x y x y z

Rotation: Pan/Tilt/Yaw Given the 3x3 rotation matrix R, we can recover, pan/tilt/yaw angle, and vise. vica. y cam z world, (0,0,1) T x cam,

The basic idea is that if we can see the north star, we know how to oriented ourselves (minus yaw angle) (why?) And the ``north star is just the point of infinity in some direction (R,t) Camera coordinate y c World coordinate y z c z y p f C x c

Seeing the vanishing point can tell us: Camera projection equation: K R t x=k [R,t] X, or x w y w z w t w = x c y c z c [R,t]= t r 1 r 2 r 3 Columns of R matrix = image of vanishing points of world-space axes! r 1,r 2,r 3 == image of x,y,z axis vanishing points r 1 r 2 r 3 t 0 0 1 0 = r 3 z-direction (need to normalize to norm =1) vanishing point

Recover Pan/Tilt angle from z-vanishing point Geometric explanation: Pan Tilt The alignment between z-axis of the world and image is defined only by the pan/tilt angle: y cam z world, (0,0,1) T r3 = x cam, z cam Note: tilt angle with rotation around x-axis is positive when pointing down in this figure

Sanity check Vanishing point in z

In this example, we are walking down a track, the ground is tilted up(we are going up the hill). Assume we are b meter tall. We see two parallel lines to our left and right, with x = 1,-1. A line on the ground is described by equation y = az - b, where a is the slope the ground. Image plane y z x x = 1 y = az -b x = -1 Ground plane So we know: y y = az -b Tilt angle z

As a line gets far away, z -> infinity. If (1,y,z) is a point on this line, its image is f(1/z,y/z). As z -> infinity, 1/z -> 0. What about y/z? y/z = (az-b)/z = a - b/z -> a. So a point on the line appears at: (0,a). Image plane x VanishingPoint (0,a) y z y = az - b Ground plane

z world, (0,0,1) T r z = = Now, we check x (0,a) y z y = az - b

Recover Camera orientation Case 2: two vanishing points, x-y direction

In some case, we can t measure the z-vanishing point directly, but if we have x-y vanishing points, we can imagine what the z-vanishing point

Basic idea: 1) x-y vanishing point gives us the first two columns of R, r1 and r2 2) z-vanishing point, the third columns of R, can be computed by r3 = r1 x r2 Recall: vanishing points of x,y direction, give us the, r1 and r2 r 1 r 2 r 3 t first two columns of R

Steps for Recovering R from Two vanishing Point 1. Measure X / Y direction vanishing point

Recovering R from Two vanishing Point (2) 2. Translate to the camera coordinates by intrinsic matrix K

Recovering R from Two vanishing Point (3) 3. Get r 1 and r 2 from vanishing point x vx and x vy vanishing points of x,y direction, give us the, r1 and r2 r 1 r 2 r 3 t first two columns of R

4. Get r 3 Recovering R from Two vanishing Point (4) 5. Get R r3 = r1 x r2 r1 r2 r3

Example Recover Camera Pan /Tilt From r3

Recover Camera Pan /Tilt From r 3 (2) Examples of Camera Pan/Tilt angle - - y x α z x

R= Recovering all three angles: yaw + pan/tilt r 1 r 2 r 3 Depends on pan/tilt Depends on yaw/pan/tilt 1) Write out equation in yaw/pan/tilt angles for r1 r2 2) Solve for yaw angle given pan/tilt angles

Summary so far 1) If we have the vanishing point in z-direction, we can recover pan/tilt angle 1) This is when the planar target is on the ground 2) If don t see the z-direction vanishing point directly, you would need two vanishing points of x-y directions 3) Given x-y directions vanishing point, we can cover the full rotation matrix, therefore its z-direction (pan/tilt) 4) Given pan/tilt + x-direction vanishing point, we can recover yaw angle