MAN-522: COMPUTER VISION SET-2 Projections and Camera Calibration

Transcription

1 MAN-522: COMPUTER VISION SET-2 Projections and Camera Calibration

2 Image formation How are objects in the world captured in an image?

3 Phsical parameters of image formation Geometric Tpe of projection Camera pose Optical Sensor s lens tpe focal length, field of view, aperture Photometric Tpe, direction, intensit of light reaching sensor Surfaces reflectance properties

4 Image formation Let s design a camera Idea : put a piece of film in front of an object Do we get a reasonable image? Slide b Steve Seitz

5 Pinhole camera Add a barrier to block off most of the ras This reduces blurring The opening is known as the aperture How does this transform the image? Slide b Steve Seitz

6 Pinhole camera Pinhole camera is a simple model to approimate imaging process, perspective projection. Image plane Virtual image pinhole If we treat pinhole as a point, onl one ra from an given point can enter the camera. Fig from Forsth and Ponce

7 Camera obscura In Latin, means dark room "Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on Januar 24, 544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle

8 Camera obscura Jett at Margate England, 898. An attraction in the late 9 th centur Around 87s Adapted from R. Duraiswami

9 Camera obscura at home Sketch from sized_camera_obscu.html

10 Perspective effects

11 Perspective effects Far awa objects appear smaller Forsth and Ponce

12 Perspective effects

13 Perspective effects Parallel lines in the scene intersect in the image Converge in image on horizon line Image plane (virtual) pinhole Scene

14 Projection properties Man-to-one: an points along same ra map to same point in image Points points Lines lines (collinearit preserved) Distances and angles are not preserved Degenerate cases: Line through focal point projects to a point. Plane through focal point projects to line Plane perpendicular to image plane projects to part of the image.

15 Perspective and art Use of correct perspective projection indicated in st centur B.C. frescoes Skill resurfaces in Renaissance: artists develop sstematic methods to determine perspective projection (around 48-55) Raphael Durer, 525

16 Perspective projection equations 3d world mapped to 2d projection in image plane Image plane Focal length Camera frame Optical ais Scene / world points Scene point Image coordinates Forsth and Ponce

17 Homogeneous coordinates Is this a linear transformation? no division b z is nonlinear Trick: add one more coordinate: homogeneous image coordinates homogeneous scene coordinates Converting from homogeneous coordinates Slide b Steve Seitz

18 Perspective Projection Matri divide b the third coordinate to convert back to non-homogeneous coordinates Projection is a matri multiplication using homogeneous coordinates: ' / ' / f z z f ) ', ' ( z f z f Slide b Steve Seitz Complete mapping from world points to image piel positions?

19 Perspective projection & calibration Perspective equations so far in terms of camera s reference frame. Camera s intrinsic and etrinsic parameters needed to calibrate geometr. Camera frame

20 Perspective projection & calibration World frame Etrinsic: Camera frame World frame Camera frame Intrinsic: Image coordinates relative to camera Piel coordinates 2D point (3) = Camera to piel coord. trans. matri (33) Perspective projection matri (34) World to camera coord. trans. matri (44) 3D point (4)

21 Weak perspective Approimation: treat magnification as constant Assumes scene depth << average distance to camera Image plane World points:

22 Orthographic projection Given camera at constant distance from scene World points projected along ras parallel to optical access

23 Pinhole size / aperture How does the size of the aperture affect the image we d get? Larger Smaller

24 Adding a lens focal point f A lens focuses light onto the film Ras passing through the center are not deviated All parallel ras converge to one point on a plane located at the focal length f Slide b Steve Seitz

25 Pinhole vs. lens

26 Cameras with lenses F optical center (Center Of Projection) focal point A lens focuses parallel ras onto a single focal point Gather more light, while keeping focus; make pinhole perspective projection practical

27 Camera Parameters

28 Imaging Geometr Object of Interest in World Coordinate Sstem (U,V,W) W U V

29 Imaging Geometr Camera Coordinate Sstem (,,). f is optic ais Image plane located f units out along optic ais f is called focal length

30 Imaging Geometr W U V Forward Projection onto image plane. 3D (,,) projected to 2D (,)

31 Imaging Geometr W u U V v Our image gets digitized into piel coordinates (u,v)

32 Camera Coordinates Imaging Geometr Image (film) Coordinates World Coordinates W u U V v Piel Coordinates

33 Forward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v We want a mathematical model to describe how 3D World points get projected into 2D Piel coordinates. Our goal: describe this sequence of transformations b a big matri equation!

34 Backward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v Note, much of vision concerns tring to derive backward projection equations to recover 3D scene structure from images (via stereo or motion) But first, we have to understand forward projection

35 Forward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v 3D-to-2D Projection perspective projection We will start here in the middle, since we ve alread talked about this when discussing stereo.

36 Basic Perspective Projection Image Point Scene Point p = (,,f) O P = (,,) f Perspective Projection Eqns f f O.Camps, PSU

37 Basic Perspective Projection Image Point Scene Point p = (,,f) O P = (,,) f Perspective Projection Eqns f f derived via similar triangles rule O.Camps, PSU f

38 Basic Perspective Projection Image Point Scene Point p = (,,f) O P = (,,) f Perspective Projection Eqns f f derived via similar triangles rule O.Camps, PSU f f

39 Basic Perspective Projection Image Point Scene Point p = (,,f) O P = (,,) f Perspective Projection Eqns f f So how do we represent this as a matri equation? We need to introduce homogeneous coordinates. O.Camps, PSU

40 Homogeneous Coordinates Represent a 2D point (,) b a 3D point (,,z ) b adding a fictitious third coordinate. B convention, we specif that given (,,z ) we can recover the 2D point (,) as ' z' ' z' Note: (,) = (,,) = (2, 2, 2) = (k, k, k) for an nonzero k (can be negative as well as positive)

41 Perspective Matri Equation (in Camera Coordinates) ' ' ' f f z f f

42 Forward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v Rigid Transformation (rotation+translation) between world and camera coordinate sstems

43 World to Camera Transformation P C P W W U V Avoid confusion: Pw and Pc are not two different points. The are the same phsical point, described in two different coordinate sstems.

44 World to Camera Transformation P C P W W R Rotate to align aes C U Translate b - C (align origins) V P C = R ( P W - C )

45 Matri Form, Homogeneous Coords P C = R ( P W - C ) r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c z U V W

46 left camera located at (,,) Eample: Simple Stereo Sstem T z (, ) Left camera located at world origin (,,) and camera aes aligned with world coord aes. (, ) (,,) z right camera located at (T,,)

47 Simple Stereo, Left Camera r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c z U V W camera aes aligned with world aes located at world position (,,) =

48 Simple Stereo Projection Equations Left camera

49 left camera located at (,,) Eample: Simple Stereo Sstem T z (, ) Right camera located at world location (T,,) and camera aes aligned with world coord aes. (, ) (,,) z right camera located at (T,,)

50 Simple Stereo, Right Camera r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c -T z U V W camera aes aligned with world aes located at world position (T,,) =

51 Simple Stereo Projection Equations Left camera Right camera

52 Bob s sure-fire wa(s) to figure out the rotation r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c forget about this while thinking about rotations z U V W P C = R P W This equation sas how vectors in the world coordinate sstem (including the coordinate aes) get transformed into the camera coordinate sstem.

53 Figuring out Rotations r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 U V W P C = R P W what if world ais (,,) corresponds to camera ais (a,b,c)? a r r 2 r 3 U a ra r 2 r 3 U b r 2 r 22 r 23 V b rb 2 r 22 r 23 V c r 3 r 32 r 33 W c rc 3 r 32 r 33 W we can immediatel write down the first column of R!

54 Figuring out Rotations and likewise with world ais and world ais... same ais in camera coords r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 ais is world coords U V W world ais (,,) in camera coords world ais (,,) in camera coords world ais (,,) in camera coords

55 Figuring out Rotations Alternative approach: sometimes it is easier to specif what camera,,or ais is in world coordinates. Then do rearrange the equation as follows. P C = R P W R - P C = P W R T P C = P W r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 U V W

56 Figuring out Rotations r 2 r 3 r r 2 r 22 r 32 r 3 r 23 r 33 U V W R T P C = P W what if camera ais (,,) corresponds to world ais (a,b,c)? r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 Ua Vb Wc a rb 2 r 22 r 32 rc 3 r 23 r 33 Ua Vb Wc we can immediatel write down the first column of R T, (which is the first row of R). r r 2 r 3

57 Figuring out Rotations and likewise with camera ais and camera ais... same ais in camera coords camera ais (,,) in world coords r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 ais is world coords U V W camera ais (,,) in world coords camera ais (,,) in world coords

58 Eample z - R train - R fl -

59 Note: Eternal Parameters also often written as R,T r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c z U V W R ( P W - C ) = R P W - R C = R P W + T r 2 r 3 r 2 r 23 r 22 r r 3 r 32 r 33 t t t z

60 Summar World Coords Camera Coords Film Coords Piel Coords U V W u v We now know how to transform 3D world coordinate points into camera coords, and then do perspective project to get 2D points in the film plane. Net time: piel coordinates

61 Recall: Imaging Geometr Object of Interest in World Coordinate Sstem (U,V,W) W U V

62 Imaging Geometr Camera Coordinate Sstem (,,). f is optic ais Image plane located f units out along optic ais f is called focal length

63 Imaging Geometr W U V Forward Projection onto image plane. 3D (,,) projected to 2D (,)

64 Imaging Geometr W u U V v Our image gets digitized into piel coordinates (u,v)

65 Camera Coordinates Imaging Geometr Image (film) Coordinates World Coordinates W u U V v Piel Coordinates

66 Forward Projection World Coords Camera Coords Film Coords Piel Coords U V W u v We want a mathematical model to describe how 3D World points get projected into 2D Piel coordinates. Our goal: describe this sequence of transformations b a big matri equation!

67 Intrinsic Camera Parameters World Coords Camera Coords Film Coords Piel Coords U V W u v Affine Transformation

68 Intrinsic parameters Describes coordinate transformation between film coordinates (projected image) and piel arra Film cameras: scanning/digitization CCD cameras: grid of photosensors still in T&V section 2.4

69 Intrinsic parameters (offsets) film plane (projected image) o (,) piel arra u (col) o (,) v (row) u f o o and o called image center or principle point v f o

70 film plane Intrinsic parameters sometimes one or more coordinate aes are flipped (e.g. T&V section 2.4) piel arra o o (,) (,) v (row) u (col) u f o v f o

71 Intrinsic parameters (scales) sampling determines how man rows/cols in the image film scanning resolution piel arra CCD analog C cols R rows resample

72 Effective Scales: s and s u f o v s s f o Note, since we have different scale factors in and, we don t necessaril have square piels! Aspect ratio is s / s O.Camps, PSU

73 Perspective projection matri / / ' ' ' o o s f s f z O.Camps, PSU Adding the intrinsic parameters into the perspective projection matri: o u f s o v f s u z v z To verif:

74 Note: Sometimes, the image and the camera coordinate sstems Sometimes, the image and the camera coordinate sstems have opposite orientations: [the book does it this wa] have opposite orientations: [the book does it this wa] / / ' ' ' o o s f s f z s o v f s o u f ) ( ) (

75 u v w Note 2 In general, I like to think of the conversion as a separate 2D affine transformation from film coords (,) to piel coordinates (u,v): a' 3 ' z' a a 2 a 2 a 22 a 23 M aff f f M proj u = M int P C = M aff M proj P C

76 Huh? Did he just sa it was a fine transformation? No, it was affine transformation, a tpe of 2D to 2D mapping defined b 6 parameters. More on this in a moment...

77 Summar : Forward Projection World Coords U V W Camera Coords Film Coords M et M proj M aff Piel Coords u v U V W M et M int u v U V W m m m 2 3 m m m M m m m m m m u v

78 Summar: Projection Equation Film plane to piels Perspective projection World to camera M aff M proj M et M int M

79 Intro to Image Mappings

80 Image Mappings Overview from R.Szeliski

81 Geometric Image Mappings image Geometric transformation transformed image (,) (, ) = f(,, {parameters}) = g(,, {parameters})

82 Linear Transformations (Can be written as matrices) image Geometric transformation transformed image (,) (, ) = M(params)

83 Translation transform t t equations matri form

84 Scale transform S S equations matri form

85 Rotation transform ) equations matri form

86 Euclidean (Rigid) transform ) t t equations matri form

87 Partitioned Matrices

88 Partitioned Matrices matri form equation form

89 Another Eample (from last time) r 2 r 3 r 2 r 23 r 22 r r 3 r 32 r 33 t t t z U V W 3 P C 33 3 = R T 3 P W 3 P C = R P W + T

90 Similarit (scaled Euclidean) transform ) S t t equations matri form

91 Affine transform equations matri form

92 Projective transform Note! equations matri form

93 Summar of 2D Transformations

94 Euclidean Summar of 2D Transformations

95 Similarit Summar of 2D Transformations

96 Affine Summar of 2D Transformations

97 Projective Summar of 2D Transformations

98 from R.Szeliski Summar of 2D Transformations