Lecture 02 Image Formation

Size: px

Start display at page:

Download "Lecture 02 Image Formation"

Laurence Hutchinson
6 years ago
Views:

1 Institute of Informatis Institute of Neuroinformatis Leture 2 Image Formation Davide Saramuzza

2 Outline of this leture Image Formation Other amera parameters Digital amera Perspetive amera model Lens distortion

Historial ontext Pinhole model: Mozi (47-39 BCE), Aristotle (384-322 BCE) Priniples of optis (inluding lenses): Alhaen (965-139 CE) Camera

film (Eastman, 1889) Cinema (Lumière Brothers, 1895) Color Photography (Lumière Brothers, 198) Television (Baird, Farnsworth, Zworykin,

3 Historial ontext Pinhole model: Mozi (47-39 BCE), Aristotle ( BCE) Priniples of optis (inluding lenses): Alhaen ( CE) Camera obsura: Leonardo da Vini ( ), Johann Zahn ( ) First photo: Joseph Niephore Niepe (1822) Daguerréotypes (1839) Photographi film (Eastman, 1889) Cinema (Lumière Brothers, 1895) Color Photography (Lumière Brothers, 198) Television (Baird, Farnsworth, Zworykin, 192s) First onsumer amera with CCD: Sony Mavia (1981) First fully digital amera: Kodak DCS1 (199) Alhaen s notes Niepe, La Table Servie, 1822 CCD hip

4 Image formation How are objets in the world aptured in an image?

5 How to form an image objet film Plae a piee of film in front of an objet Do we get a reasonable image?

6 Pinhole amera objet barrier film Add a barrier to blok off most of the rays This redues blurring The opening is known as the aperture

Camera obsura In Latin, means dark room Basi priniple known to Mozi (47-39 BC), Aristotle (384-322 BC) Drawing aid for artists: desribed by Leonardo da Vini (1452-1519) Image is inverted Depth of the

7 Camera obsura In Latin, means dark room Basi priniple known to Mozi (47-39 BC), Aristotle ( BC) Drawing aid for artists: desribed by Leonardo da Vini ( ) Image is inverted Depth of the room (box) is the effetive foal length "Reinerus Gemma-Frisius, observed an elipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomia et Geometria, It is thought to be the first published illustration of a amera obsura..." Hammond, John H., The Camera Obsura, A Chronile

Camera obsura at home Sketh from http://www.funsi.

8 Camera obsura at home Sketh from

9 Home-made pinhole amera What an we do to redue the blur?

10 Effets of the Aperture Size In an ideal pinhole, only one ray of light reahes eah point on the film the image an be very dim Making aperture bigger makes the image blurry

11 Shrinking the aperture Why not make the aperture as small as possible?

12 Shrinking the aperture Why not make the aperture as small as possible? Less light gets through (must inrease the exposure) Diffration effets

13 Image formation using a onverging lens objet Lens film A lens fouses light onto the film Rays passing through the Optial Center are not deviated

14 Image formation using a onverging lens objet Lens Optial Axis Foal Point Foal Length: f All rays parallel to the Optial Axis onverge at the Foal Point

15 Thin lens equation Objet Lens A f Foal Point B Image z e Similar Triangles: B A e z Find a relationship between f, z, and e

16 Thin lens equation Objet Lens A A Foal Point B f Image z e Similar Triangles: B A B A e z e f f f e 1 Thin lens equation e e f z f z e Any objet point satisfying this equation is in fous

17 In fous objet Lens film Optial Axis Foal Point f Cirle of Confusion or Blur Cirle There is a speifi distane from the lens, at whih world points are in fous in the image Other points projet to a blur irle in the image

18 Blur Cirle Lens Foal Plane L Objet f Image Plane z e Blur Cirle of radius R Objet is out of fous Blur Cirle has radius: A minimal L (pinhole) gives minimal R To apture a good image:adjust amera settings, suh that R remains smaller than the image resolution R L 2e

19 The Pin-hole approximation What happens if z f? Objet Lens A A Foal Point B f Image z We need to adjust the image plane suh that objets at infinity are in fous 1 f 1 1 z e 1 f e 1 f e e

20 The Pin-hole approximation What happens if z f? Foal Plane Objet Lens h Foal Point C Optial Center or Center of Projetion f h' We need to adjust the image plane suh that objets at infinity are in fous 1 f z 1 1 z e 1 f 1 f e e

21 The Pin-hole approximation What happens if z f? Objet Lens h Foal Point C Optial Center or Center or Projetion f h' z We need to adjust the image plane suh that objets at infinity are in fous h' f f h' h h z z The dependene of the apparent size of an objet on its depth (i.e. distane from the amera) is known as perspetive

22 Perspetive effets Far away objets appear smaller

23 Perspetive effets

time, artists developped systemati methods to

24 Perspetive and art Use of orret perspetive projetion indiated in 1 st entury BCE fresoes During Renaissane time, artists developped systemati methods to determine perspetive projetion (around ) Raphael Durer

25 Playing with Perspetive Perspetive gives us very strong depth ues hene we an pereive a 3D sene by viewing its 2D representation (i.e. image) An example where pereption of 3D senes is misleading is the Ames room Ames room A lip from "The omputer that ate Hollywood" doumentary. Dr. Vilayanur S. Ramahandran.

26 Projetive Geometry What is preserved? Straight lines are still straight

27 Projetive Geometry What is lost? Length Angles Parallel? Perpendiular?

28 Vanishing points and lines Parallel lines in the world interset in the image at a vanishing point

29 Vanishing points and lines Parallel lines in the world interset in the image at a vanishing point Vanishing line Vertial vanishing point (at infinity) Vanishing point Vanishing point

30 Vanishing points and lines Parallel planes in the world interset in the image at a vanishing line Vanishing Line Vanishing Point o Vanishing Point o

31 Outline of this leture Image Formation Other amera parameters Digital amera Perspetive amera model Lens distortion

Although a lens an preisely fous at only one distane at a time, the derease in sharpness is

32 Fous and depth of field Depth of field (DOF) is the distane between the nearest and farthest objets in a sene that appear aeptably sharp in an image. Although a lens an preisely fous at only one distane at a time, the derease in sharpness is gradual on eah side of the foused distane, so that within the DOF, the unsharpness is impereptible under normal viewing onditions Depth of field

33 Fous and depth of field How does the aperture affet the depth of field? A smaller aperture inreases the range in whih the objet appears approximately in fous but redues the amount of light into the amera

34 Field of view depends on foal length As f gets smaller, image beomes more wide angle more world points projet onto the finite image plane As f gets larger, image beomes more narrow angle smaller part of the world projets onto the finite image plane

35 Field of view Smaller FOV = larger Foal Length

36 Field of view Angular measure of portion of 3D spae seen by the amera

37 Outline of this leture Image Formation Other amera parameters Digital amera Perspetive amera model Lens distortion

38 Digital ameras Film sensor array Often an array of harge oupled devies Eah CCD/CMOS is light sensitive diode that onverts photons (light energy) to eletrons

39 Pixel Intensity with 8 bits ranges between [,255] Digital images j=1 width 5 i=1 height 3 im[176][21] has value 164 im[194][23] has value 37 NB. Matlab oordinates: [rows, ols]; C/C++ [ols, rows]

40 Color sensing in digital ameras Bayer grid The Bayer pattern (Bayer 1976) plaes green filters over half of the sensors (in a hekerboard pattern), and red and blue filters over the remaining ones. This is beause the luminane signal is mostly determined by green values and the human visual system is muh more sensitive to high frequeny detail in luminane than in hrominane.

41 Color sensing in digital ameras Bayer grid Estimate missing omponents from neighboring values (demosaiing) Foveon hip design ( staks the red, green, and blue sensors beneath eah other but has not gained widespread adoption.

42 Color images: RGB olor spae but there are also many other olor spaes (e.g., YUV) R G B

43 An example amera datasheet

44 Outline of this leture Image Formation Other amera parameters Digital amera Perspetive amera model Lens distortion

45 Perspetive Camera Z = optial axis u Z P O = prinipal point v f O p Image plane C Y X C = optial enter = enter of the lens For onveniene, the image plane is usually represented in front of C suh that the image preserves the same orientation (i.e. not flipped) Note: a amera does not measure distanes but angles! a amera is a bearing sensor

46 From World to Pixel oordinates Find pixel oordinates (u,v) of point P w in the world frame:. Convert world point P w to u P P w amera point P v O Find pixel oordinates (u,v) of point P in the amera frame: y p x 1. Convert P to image-plane oordinates (x,y) C Z X W Z w X w Y Y w 2. Convert P to (disretised) pixel oordinates (u,v) [R T]

47 Perspetive Projetion (1) From the Camera frame to the image plane P =( X,, Z ) T X Z C f p x O Image Plane X The Camera point P =( X,, Z ) T projets to p=(x, y) onto the image plane From similar triangles: x f Similarly, in the general ase: y f Y Z y fy Z X Z x fx Z 1. Convert P to image-plane oordinates (x,y) 2. Convert P to (disretised) pixel oordinates (u,v)

48 Perspetive Projetion (2) From the Camera frame to pixel oordinates To onvert p from the loal image plane oords (x,y) to the pixel oords (u,v), we need to aount for: the pixel oords of the amera optial enter O u, v (,) ) u Image plane Sale fators So: u u v v k u, k v k k v ( for the pixel-size in both dimensions u x u u y v v ku fx Z kv fy Z Use Homogeneous Coordinates for linear mapping from 3D to 2D, by introduing an extra element (sale): u p v u ~ u ~ p v~ v ~ w 1 v O y (u,v ) x p

49 So: Expressed in matrix form and homogenerous oordinates: v u Z Y X v f k u f k v u 1 v u Z Y X K Z Y X v u v u 1 Or alternatively v u Z fy k v v Z fx k u u Image plane (CCD) P C O u v p Z f X Y Foal length in pixels K is alled Calibration matrix or Matrix of Intrinsi Parameters Sometimes, it is ommon to assume a skew fator (K 12 ) to aount for possible misalignments between CCD and lens. However, the amera manufaturing proess today is so good that we an safely assume K 12 = and α u = α v. Perspetive Projetion (3)

50 Exerise 1 Determine the Intrinsi Parameter Matrix (K) for a digital amera with image size pixels and horizontal field of view equal to 9 Assume the prinipal point in the enter of the image and squared pixels What is the vertial field of view?

51 Exerise 1 Determine the Intrinsi Parameter Matrix (K) for a digital amera with image size pixels and horizontal field of view equal to 9 Assume the prinipal point in the enter of the image and squared pixels f = 64 = 32 pixels 2 tan θ 2 K What is the vertial field of view? θ V = 2 tan 1 H 48 = 2 tan 1 2f 2 32 = 73.74

52 Slide from Efros, Photo from Criminisi Exerise 2 Prove that world s parallel lines interset at a vanishing point in the amera image Vanishing line Vertial vanishing point (at infinity) Vanishing point Vanishing point

53 Exerise 2 Prove that world s parallel lines interset at a vanishing point in the amera image Let s onsider the perspetive projetion equation in alibrated oordinates: Two parallel 3D lines have parametri equations: Now substitute this into the amera perspetive projetion equation and ompute the limit for s The result solely depends on the diretion vetor of the line. These are the image oordinates of the vanishing point (VP). What is the intuitive interpretation of this? n m l s Z Y X Z Y X Z Y y Z X x, n m l s Z Y X Z Y X VP i i s VP i i s y n m sn Z sm Y x n l sn Z sl X lim, lim

54 Perspetive Projetion (4) Z Y X K v u 1 From the World frame to the Camera frame Projetion Matrix (M) 1 1 w w w Z Y X RT K v u t t t Z Y X r r r r r r r r r Z Y X w w w w w w w w w Z Y X T R Z Y X t r r r t r r r t r r r Z Y X P O u v p X C Z Y [R T] Extrinsi Parameters W Z w Y w X w = P w Perspetive Projetion Equation

55 Outline of this leture Image Formation Other amera parameters Digital amera Perspetive amera model Lens distortion

56 Radial Distortion No distortion Barrel distortion Pinushion

57 Radial Distortion The standard model of radial distortion is a transformation from the ideal oordinates (u, v) (i.e., undistorted) to the real observable oordinates (distorted) (u d, v d ) The amount of distortion of the oordinates of the observed image is a nonlinear funtion of their radial distane. For most lenses, a simple quadrati model of distortion produes good results u d v d = 1 + k 1 r 2 u u v v + u v where r 2 = u u 2 + v v 2 Depending on the amount of distortion (an thus on the amera field of view), higher order terms an be introdued: u d v d = 1 + k 1 r 2 + k 2 r 4 + k 3 r 6 u u v v + u v r 2 = u u 2 + v v 2

58 To reap, a 3D world point P = X w, Y w, Z w projets into the image point p = u, v where and λ is the depth (λ = Z C ) of the sene point If we want to take into aount the radial distortion, then the distorted oordinates u d, v d (in pixels) an be obtained as where See also the OpenCV doumentation: Summary: Perspetive projetion equations 1 1 ~ ~ ~ ~ w w w Z Y X T R K v u w v u p 1 v u K u d v d = 1 + k 1 r 2 u u v v + u v r 2 = u u 2 + v v 2

59 Summary (things to remember) Perspetive Projetion Equation Intrinsi and extrinsi parameters (K, R, t) Homogeneous oordinates Normalized image oordinates Image formation equations (inluding radial distortion) Chapter 4 of Autonomous Mobile Robot book

Supplementary Material: Geometric Calibration of Micro-Lens-Based Light-Field Cameras using Line Features

Supplementary Material: Geometri Calibration of Miro-Lens-Based Light-Field Cameras using Line Features Yunsu Bok, Hae-Gon Jeon and In So Kweon KAIST, Korea As the supplementary material, we provide detailed