MAN-522: COMPUTER VISION SET-2 Projections and Camera Calibration

Image formation How are objects in the world captured in an image?

Phsical parameters of image formation Geometric Tpe of projection Camera pose Optical Sensor s lens tpe focal length, field of view, aperture Photometric Tpe, direction, intensit of light reaching sensor Surfaces reflectance properties

Image formation Let s design a camera Idea : put a piece of film in front of an object Do we get a reasonable image? Slide b Steve Seitz

Pinhole camera Add a barrier to block off most of the ras This reduces blurring The opening is known as the aperture How does this transform the image? Slide b Steve Seitz

Pinhole camera Pinhole camera is a simple model to approimate imaging process, perspective projection. Image plane Virtual image pinhole If we treat pinhole as a point, onl one ra from an given point can enter the camera. Fig from Forsth and Ponce

Camera obscura In Latin, means dark room "Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on Januar 24, 544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle http://www.acmi.net.au/aic/camera_obscura.html

Camera obscura Jett at Margate England, 898. An attraction in the late 9 th centur Around 87s http://brightbtes.com/cosite/collection2.html Adapted from R. Duraiswami

Camera obscura at home Sketch from http://www.funsci.com/fun3_en/sk/sk.htm http://blog.makezine.com/archive/26/2/how_to_room_ sized_camera_obscu.html

Perspective effects

Perspective effects Far awa objects appear smaller Forsth and Ponce

Perspective effects

Perspective effects Parallel lines in the scene intersect in the image Converge in image on horizon line Image plane (virtual) pinhole Scene

Projection properties Man-to-one: an points along same ra map to same point in image Points points Lines lines (collinearit preserved) Distances and angles are not preserved Degenerate cases: Line through focal point projects to a point. Plane through focal point projects to line Plane perpendicular to image plane projects to part of the image.

Perspective and art Use of correct perspective projection indicated in st centur B.C. frescoes Skill resurfaces in Renaissance: artists develop sstematic methods to determine perspective projection (around 48-55) Raphael Durer, 525

Perspective projection equations 3d world mapped to 2d projection in image plane Image plane Focal length Camera frame Optical ais Scene / world points Scene point Image coordinates Forsth and Ponce

Homogeneous coordinates Is this a linear transformation? no division b z is nonlinear Trick: add one more coordinate: homogeneous image coordinates homogeneous scene coordinates Converting from homogeneous coordinates Slide b Steve Seitz

Perspective Projection Matri divide b the third coordinate to convert back to non-homogeneous coordinates Projection is a matri multiplication using homogeneous coordinates: ' / ' / f z z f ) ', ' ( z f z f Slide b Steve Seitz Complete mapping from world points to image piel positions?

Perspective projection & calibration Perspective equations so far in terms of camera s reference frame. Camera s intrinsic and etrinsic parameters needed to calibrate geometr. Camera frame

Perspective projection & calibration World frame Etrinsic: Camera frame World frame Camera frame Intrinsic: Image coordinates relative to camera Piel coordinates 2D point (3) = Camera to piel coord. trans. matri (33) Perspective projection matri (34) World to camera coord. trans. matri (44) 3D point (4)

Weak perspective Approimation: treat magnification as constant Assumes scene depth << average distance to camera Image plane World points:

Orthographic projection Given camera at constant distance from scene World points projected along ras parallel to optical access

Pinhole size / aperture How does the size of the aperture affect the image we d get? Larger Smaller

Adding a lens focal point f A lens focuses light onto the film Ras passing through the center are not deviated All parallel ras converge to one point on a plane located at the focal length f Slide b Steve Seitz

Pinhole vs. lens

Cameras with lenses F optical center (Center Of Projection) focal point A lens focuses parallel ras onto a single focal point Gather more light, while keeping focus; make pinhole perspective projection practical

Camera Parameters

Imaging Geometr Object of Interest in World Coordinate Sstem (U,V,W) W U V

Imaging Geometr Camera Coordinate Sstem (,,). f is optic ais Image plane located f units out along optic ais f is called focal length

Imaging Geometr W U V Forward Projection onto image plane. 3D (,,) projected to 2D (,)

Imaging Geometr W u U V v Our image gets digitized into piel coordinates (u,v)

Camera Coordinates Imaging Geometr Image (film) Coordinates World Coordinates W u U V v Piel Coordinates

Forward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v We want a mathematical model to describe how 3D World points get projected into 2D Piel coordinates. Our goal: describe this sequence of transformations b a big matri equation!

Backward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v Note, much of vision concerns tring to derive backward projection equations to recover 3D scene structure from images (via stereo or motion) But first, we have to understand forward projection

Forward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v 3D-to-2D Projection perspective projection We will start here in the middle, since we ve alread talked about this when discussing stereo.

Basic Perspective Projection Image Point Scene Point p = (,,f) O P = (,,) f Perspective Projection Eqns f f O.Camps, PSU

Basic Perspective Projection Image Point Scene Point p = (,,f) O P = (,,) f Perspective Projection Eqns f f derived via similar triangles rule O.Camps, PSU f

Basic Perspective Projection Image Point Scene Point p = (,,f) O P = (,,) f Perspective Projection Eqns f f derived via similar triangles rule O.Camps, PSU f f

Basic Perspective Projection Image Point Scene Point p = (,,f) O P = (,,) f Perspective Projection Eqns f f So how do we represent this as a matri equation? We need to introduce homogeneous coordinates. O.Camps, PSU

Homogeneous Coordinates Represent a 2D point (,) b a 3D point (,,z ) b adding a fictitious third coordinate. B convention, we specif that given (,,z ) we can recover the 2D point (,) as ' z' ' z' Note: (,) = (,,) = (2, 2, 2) = (k, k, k) for an nonzero k (can be negative as well as positive)

Perspective Matri Equation (in Camera Coordinates) ' ' ' f f z f f

Forward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v Rigid Transformation (rotation+translation) between world and camera coordinate sstems

World to Camera Transformation P C P W W U V Avoid confusion: Pw and Pc are not two different points. The are the same phsical point, described in two different coordinate sstems.

World to Camera Transformation P C P W W R Rotate to align aes C U Translate b - C (align origins) V P C = R ( P W - C )

Matri Form, Homogeneous Coords P C = R ( P W - C ) r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c z U V W

left camera located at (,,) Eample: Simple Stereo Sstem T z (, ) Left camera located at world origin (,,) and camera aes aligned with world coord aes. (, ) (,,) z right camera located at (T,,)

Simple Stereo, Left Camera r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c z U V W camera aes aligned with world aes located at world position (,,) =

Simple Stereo Projection Equations Left camera

left camera located at (,,) Eample: Simple Stereo Sstem T z (, ) Right camera located at world location (T,,) and camera aes aligned with world coord aes. (, ) (,,) z right camera located at (T,,)

Simple Stereo, Right Camera r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c -T z U V W camera aes aligned with world aes located at world position (T,,) =

Simple Stereo Projection Equations Left camera Right camera

Bob s sure-fire wa(s) to figure out the rotation r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c forget about this while thinking about rotations z U V W P C = R P W This equation sas how vectors in the world coordinate sstem (including the coordinate aes) get transformed into the camera coordinate sstem.

Figuring out Rotations r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 U V W P C = R P W what if world ais (,,) corresponds to camera ais (a,b,c)? a r r 2 r 3 U a ra r 2 r 3 U b r 2 r 22 r 23 V b rb 2 r 22 r 23 V c r 3 r 32 r 33 W c rc 3 r 32 r 33 W we can immediatel write down the first column of R!

Figuring out Rotations and likewise with world ais and world ais... same ais in camera coords r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 ais is world coords U V W world ais (,,) in camera coords world ais (,,) in camera coords world ais (,,) in camera coords

Figuring out Rotations Alternative approach: sometimes it is easier to specif what camera,,or ais is in world coordinates. Then do rearrange the equation as follows. P C = R P W R - P C = P W R T P C = P W r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 U V W

Figuring out Rotations r 2 r 3 r r 2 r 22 r 32 r 3 r 23 r 33 U V W R T P C = P W what if camera ais (,,) corresponds to world ais (a,b,c)? r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 Ua Vb Wc a rb 2 r 22 r 32 rc 3 r 23 r 33 Ua Vb Wc we can immediatel write down the first column of R T, (which is the first row of R). r r 2 r 3

Figuring out Rotations and likewise with camera ais and camera ais... same ais in camera coords camera ais (,,) in world coords r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 ais is world coords U V W camera ais (,,) in world coords camera ais (,,) in world coords

Eample z - R train - R fl -

Note: Eternal Parameters also often written as R,T r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c z U V W R ( P W - C ) = R P W - R C = R P W + T r 2 r 3 r 2 r 23 r 22 r r 3 r 32 r 33 t t t z

Summar World Coords Camera Coords Film Coords Piel Coords U V W u v We now know how to transform 3D world coordinate points into camera coords, and then do perspective project to get 2D points in the film plane. Net time: piel coordinates

Recall: Imaging Geometr Object of Interest in World Coordinate Sstem (U,V,W) W U V

Imaging Geometr Camera Coordinate Sstem (,,). f is optic ais Image plane located f units out along optic ais f is called focal length

Imaging Geometr W U V Forward Projection onto image plane. 3D (,,) projected to 2D (,)

Imaging Geometr W u U V v Our image gets digitized into piel coordinates (u,v)

Camera Coordinates Imaging Geometr Image (film) Coordinates World Coordinates W u U V v Piel Coordinates

Intrinsic Camera Parameters World Coords Camera Coords Film Coords Piel Coords U V W u v Affine Transformation

Intrinsic parameters Describes coordinate transformation between film coordinates (projected image) and piel arra Film cameras: scanning/digitization CCD cameras: grid of photosensors still in T&V section 2.4

Intrinsic parameters (offsets) film plane (projected image) o (,) piel arra u (col) o (,) v (row) u f o o and o called image center or principle point v f o

film plane Intrinsic parameters sometimes one or more coordinate aes are flipped (e.g. T&V section 2.4) piel arra o o (,) (,) v (row) u (col) u f o v f o

Intrinsic parameters (scales) sampling determines how man rows/cols in the image film scanning resolution piel arra CCD analog C cols R rows resample

Effective Scales: s and s u f o v s s f o Note, since we have different scale factors in and, we don t necessaril have square piels! Aspect ratio is s / s O.Camps, PSU

Perspective projection matri / / ' ' ' o o s f s f z O.Camps, PSU Adding the intrinsic parameters into the perspective projection matri: o u f s o v f s u z v z To verif:

Note: Sometimes, the image and the camera coordinate sstems Sometimes, the image and the camera coordinate sstems have opposite orientations: [the book does it this wa] have opposite orientations: [the book does it this wa] / / ' ' ' o o s f s f z s o v f s o u f ) ( ) (

u v w Note 2 In general, I like to think of the conversion as a separate 2D affine transformation from film coords (,) to piel coordinates (u,v): a' 3 ' z' a a 2 a 2 a 22 a 23 M aff f f M proj u = M int P C = M aff M proj P C

Huh? Did he just sa it was a fine transformation? No, it was affine transformation, a tpe of 2D to 2D mapping defined b 6 parameters. More on this in a moment...

Summar : Forward Projection World Coords U V W Camera Coords Film Coords M et M proj M aff Piel Coords u v U V W M et M int u v U V W m m m 2 3 m m m 2 22 3 M m m m 3 23 33 m m m 4 24 34 u v

Summar: Projection Equation Film plane to piels Perspective projection World to camera M aff M proj M et M int M

Intro to Image Mappings

Image Mappings Overview from R.Szeliski

Geometric Image Mappings image Geometric transformation transformed image (,) (, ) = f(,, {parameters}) = g(,, {parameters})

Linear Transformations (Can be written as matrices) image Geometric transformation transformed image (,) (, ) = M(params)

Translation transform t t equations matri form

Scale transform S S equations matri form

Rotation transform ) equations matri form

Euclidean (Rigid) transform ) t t equations matri form

Partitioned Matrices http://planetmath.org/encclopedia/partitionedmatri.html

Partitioned Matrices 2 22 2 2 2 matri form equation form

Another Eample (from last time) r 2 r 3 r 2 r 23 r 22 r r 3 r 32 r 33 t t t z U V W 3 P C 33 3 = R T 3 P W 3 P C = R P W + T

Similarit (scaled Euclidean) transform ) S t t equations matri form

Affine transform equations matri form

Projective transform Note! equations matri form

Summar of 2D Transformations

Euclidean Summar of 2D Transformations

Similarit Summar of 2D Transformations

Affine Summar of 2D Transformations

Projective Summar of 2D Transformations

from R.Szeliski Summar of 2D Transformations