ECE-161C Cameras Nuno Vasconcelos ECE Department, UCSD
Image formation all image understanding starts with understanding of image formation: projection of a scene from 3D world into image on 2D plane 2
Image formation first of all, why do we care about this? 1) allows us to create ( render ) imaginary scenes special effects, games, architecture/visualization, etc. build a CAD model of the scene and then render from different views 3
Image formation e.g. a fly-through camera that allows you to see a sports event from new angles we know where the camera is knowing the projection equations allows us to recreate the image from a 3D model of the scene computer graphics is mostly about this 4
Image formation 2) even better, we could reconstruct the 3D model from images rendering: 3D world to 2D image scene reconstruction: 2D images to 3D model this turns out to be much harder because a 2D projection is consistent with many 3D scenes one image is usually not enough, but can be done from a collection of images 5
Image formation when multiple images are available, it is possible to register them deduce the mapping from 2D to 3D extract a 3D model of the scene render from different angles 6
Image formation even when we do not care about the 3D scene per se knowing the geometry is important for many tasks note the appearence changes as the cars move this is due to perspective has to be accounted for even though the goal is tracking not reconstruction 7
Image formation even for rigid scenes that do not change that much a change of perspective will create massive pixel changes to compensate for this, one has to understand the projection equations this turns out to be quite complicated as usual in science, we simplify as much as we can for example, we adopt the pinhole camera model 8
Pinhole camera we assume that a camera is a black box with an infinitesimally small hole on one face the hole is so small that only one ray of light passes through it and hits the other side real pinhole camera made by Kodak for schools, circa 1930 9
Pinhole camera by placing photo-sensitive material in the back wall you will get an upside-down replica of the scene this is the image plane to avoid the mathematical inconvenience of this inversion we consider a plane outside of the camera this is called the virtual image plane 10
Pinhole camera the virtual image plane it is an abstraction exactly the same as the image plane, with the exception that there is no inversion 11
Pinhole camera one important property: objects that are far away become smaller in the image plane we suspect that distance to the camera plays an important role in perspective projection in particular we would expect image size proportional to 1/d 12
Coordinates to relate world point P to image point P we need a coordinate system the 1/d dependence suggests using pinhole as origin we also make two coordinate axes (i,j) a basis of the image plane and the third (k) orthogonal to it (measures depth) pinhole 13
Pinhole camera definitions: line perpendicular to image plane, through pinhole, is the optical axis point where optical axis intersects image plane is the image center distance f between image plane and pinhole is the focal length image plane focal length image center pinhole optical axis 14
15 Projection equations note that P, O, P are on the same line this implies that there is a λ such that OP = λ OP, and = = = = = z y z x f y x z f z f y y x x ' ' ' ' λ λ λ λ
Perspective projection this is the basic equation of perspective projection x ' y ' x = f y z z note that there is indeed an inverse dependence on the depth Z far objects become small 16
Perspective projection this is a very powerful cue for scene understanding and fun too! note that the visual system infers all sorts of properties from perspective cues e.g. size 17
Perspective projection or shape or proximity 18
Perspective projection is conceptually very simple x ' y ' x = f y z z but is highly non-linear and usually hard to work with e.g. assume you have a big plane on the scene, e.g. a wall then z = ax + by x x ' = f y y ' ( ax + by ) ( ax + by ) the image coordinates depend highly non-linearly on the world coordinates 19
Prospective projection this is the reason why we see this 20
Prospective projection instead of this 21
Projective projection since the size of the wall is constant far away (large z) distances appear to shrunk in the image in many cases, this non-linearity is too much to handle we look for approximations 22
23 Affine projection consider a plane parallel to the image plane this plane has equation z = C and the projection equation becomes image coordinates are simply a re-scaling of the 3D coordinates C f m y x m y x C f y x = = =, ' '
24 Affine projection scaling: if m <1 image points are closer than 3D points, else they are further away this can be seen by noting that, for P=(x p,y p ), Q=(x q,y q ) this is also captured by the relation through a scaling matrix ( ) ( ) ( ) ( ) ), ( ' ' ' ' ') ', ( 2 2 2 2 2 2 Q P d m y y m x x m y y x x Q P d q p q p q p q p = + = + = = y x m m y x 0 0 ' '
Affine projection when can we use this approximation? we are assuming z constant this is acceptable if the variation of depth in the scene is much smaller than the average depth e.g. an airplane taking aerial photos z >> z z 25
Orthographic projection if the camera is always at (approximately) the same distance from the scene: m only contributes a change of scale that we do not care much about (e.g. measure in centimeters vs meters) it is common to normalize to m = 1 this is orthographic projection x ' x = y ' y 26
Orthographic projection this is, of course, very convenient: image coordinates = world coordinates there are not that many scenarios in which it is a good approximation nevertheless can be a good model for a preliminary solution which is then refined with a more complicated model x ' y ' = x y 27
Lenses so far we have assumed the pinhole camera in practice we cannot really build such a camera and obtain decent quality problems: when pinhole is too big many directions are averaged, blurring the image 28
Lenses pinhole problems: if the pinhole is too small we have diffraction effects which also blur the image there is a correct pinhole size from an image distortion point of view, but that introduces other problems 29
Lenses pinhole problems: for the correct pinhole size we cannot get enough light in the camera to sufficiently excite the recording material generally, pinhole cameras are dark, a very small set of rays from a particular point hits the screen this is the reason why we need camera lenses 30
Lenses the basic idea is: lets make the aperture bigger so that we can have many rays of light into the camera to avoid blurring we need to concentrate all the rays that start in the same 3D point so that they end up on the same image plane point 31
Lenses the geometry is as follows image plane light ray R P P lens d 2 d 1 R: radius of curvature of the lens d 1 : distance from 3D point to lens d 2 : distance to image plane 32
Lenses we assume all angles are small (d 2, h are in microns): α = sin α = tg α image plane P R h R h α 1 h d 1 P lens d 2 d 1 α 1 h h + R d 1 33
Lenses we assume all angles are small (d 2, h are in microns): α = sin α = tg α image plane h d 2 α 2 R h R h h R P P lens d 2 d 1 α 2 h h R d 2 34
Lenses Snell s law for light propagating between two media of indexes of refraction n 1 and n 2 image plane α 2 R h α 1 n = 1 sinα1 n 2 sin h R α 2 P P lens d 2 d 1 α 1 h h + R d 1 α 2 h h R d 2 1 1 n 1h + R d 1 1 1 n 2h R d 2 35
Lenses which means that 1 d 1 1 n2 n1 n1 R n 2 d 2 given distance d 2, we can compute distance d 1 of the 3D point image plane R h P P lens d 2 d 1 36
Lenses which means that 1 d 1 1 n2 n1 n1 R n 2 d 2 image plane note that it does not depend on the vertical position of P we can show that it holds for all rays that start in the plane of P R P P lens d 2 d 1 37
Lenses note that, in general, we can only have in focus objects that are in a certain depth range this is why the background is sometimes out of focus on photographs by controlling the focus you are effectively changing the plane of the rays that converge on the image plane without blur 38
39