Computer Vision and Applications. Prof. Trevor. Darrell. Class overview Administrivia & Policies Lecture 1

Transcription

1 6.89 Computer Vision and pplications Class overview dministrivia & olicies Lecture rof. Trevor. Darrell erspective projection (review) Rigid motions (review) Camera Calibration Readings: Forsythe & once,., 2., 2.2, 2.3, 3., 3.2

2 Vision What does it mean, to see? to know what is where by looking. How to discover from images what is present in the world, where things are, what actions are taking place. from Marr, 982

3 Why study Computer Vision? One can see the future (and avoid bad things )! Images and movies are everywhere; fast-growing collection of useful applications building representations of the 3D world from pictures automated surveillance (who s doing what) movie post-processing face finding Greater understanding of human vision Various deep and attractive scientific mysteries how does object recognition work?

4 Why study Computer Vision? eople draw distinctions between what is seen Object recognition This could mean is this a fish or a bicycle? It could mean is this George Washington? It could mean is this poisonous or not? It could mean is this slippery or not? It could mean will this support my weight? Great mystery How to build programs that can draw useful distinctions based on image properties.

5 Computer vision class, fast-forward

6 Cameras, lenses, and sensors inhole cameras Lenses rojection models Geometric camera parameters From Computer Vision, Forsyth and once, rentice-hall, 22.

7 Image filtering Review of linear systems, convolution andpass filter-based image representations robabilistic models for images Oriented, multi-scale representation Image

8 Color From Foundations of Vision, by rian Wandell, Sinauer ssoc., 995

9 Models of texture arametric model Non-parametric model arametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients J. ortilla and E. Simoncelli, International Journal of Computer Vision 4(): 49-7, October 2. Kluwer cademic ublishers.. Efros and W. T Freeman, Image quilting for texture synthesis and transfer, SIGGRH 2

10 Statistical classifiers MIT Media Lab face localiation results. pplications: database search, human machine interaction, video conferencing.

11 Multi-view Geometry What are the relationships between images of point features in more than one view? Given a point feature in one camera view, predict it s location in a second (or third) camera?

12 Ego-Motion / Match-move Where are the cameras? Track points, estimate consistent poses Render synthetic objects in real world!

13 Ego-Motion / Match-move Video See Harts War and other examples in Gallery of examples for Matchmove program at

14 Structure from Motion What is the shape of the scene?

15 Segmentation How many ways can you segment six points? (or curves)

16

17

18 Segmentation Which image components belong together? elong together=lie on the same object Cues similar colour similar texture not separated by contour form a suggestive shape when assembled

19

20

21

22 Tracking Follow objects and estimate location.. radar / planes pedestrians cars face features / expressions Many ad-hoc approaches General probabilistic formulation: model density over time.

23 Tracking Use a model to predict next position and refine using next image Model: simple dynamic models (second order dynamics) kinematic models etc. Face tracking and eye tracking now work rather well

24 rticulated Models Find most likely model consistent with observations.(and previous configuration)

25 rticulated tracking Constrained optimiation Coarse-to-fine part iteration ropagate joint constraints through each limb Real-time on Gh pentium

26 Video

27 nd Visual Category Learning Image Databases Image-based Rendering Visual Speechreading Medical Imaging

28 dministrivia Syllabus Grading Collaboration olicy roject

29

30

31 Grading Two take-home exams Five problem sets with lab exercises in Matlab No final exam Final project

32 Collaboration olicy roblem sets may be discussed, but all written work and coding must be done individually. Take-home exams may not be discussed. Individuals found submitting duplicate or substantially similar materials due to inappropriate collaboration may get an F in this class and other sanctions.

33 roject The final project may be n original implementation of a new or published idea detailed empirical evaluation of an existing implementation of one or more methods paper comparing three or more papers not covered in class, or surveying recent literature in a particular area project proposal not longer than two pages must be submitted and approved by pril st.

34 Out today, due 2/2 roblem Set Matlab image exercises load, display images pixel manipulation RG color interpolation image warping / morphing with interp2 simple background subtraction ll psets graded loosely: check, check-,. (Outstanding solutions get extra credit.)

35

36 Cameras, lenses, and calibration Today: Camera models (review) rojection equations (review) You should have been exposed to this material in previous courses; this lecture is just a (quick) review. Calibration methods (new)

37 7-year old s question Why is there no image on a white piece of paper?

38 Virtual image, perspective bstract camera model - box with a small hole in it projection Forsyth&once

39 They are formed by the projection of 3D objects. Figure from US Navy Manual of asic Optics and Optical Instruments, prepared by ureau of Naval ersonnel. Reprinted by Dover ublications, Inc., 969. Images are two-dimensional patterns of brightness values.

40 Reproduced by permission, the merican Society of hotogrammetry and Remote Sensing..L. Nowicki, Stereoscopy. Manual of hotogrammetry, Thompson, Radlinski, and Speert (eds.), third edition, 966. Figure from US Navy Manual of asic Optics and Optical Instruments, prepared by ureau of Naval ersonnel. Reprinted by Dover ublications, Inc., 969. nimal eye: a looonnng time ago. hotographic camera: Niepce, 86. inhole perspective projection: runelleschi, XV th Century. Camera obscura: XVI th Century.

41 The equation of projection x' y' f f x ' y '

42 Forsyth&once Distant objects are smaller

43 Geometric properties of projection oints go to points Lines go to lines lanes go to whole the whole image image or half-planes. or a half-plane olygons go to polygons Degenerate cases line through focal point to point plane through focal point to line x' y' x f ' y f '

44 arallel lines meet Common to draw film plane in front of the focal point. Moving the film plane merely scales the image. Forsyth&once

45 Vanishing points Each set of parallel lines (=direction) meets at a different point The vanishing point for this direction Sets of parallel lines on the same plane lead to collinear vanishing points. The line is called the horion for that plane

46 What if you photograph a brick wall head-on?

47 Two-point perspective

48

49

50

51 Weak perspective Issue perspective effects, but not over the scale of individual objects collect points into a group at about the same depth, then divide each point by the depth of its group dv: easy Disadv: wrong

52 Orthographic projection

53 How large a pinhole?

54 Wandell, Foundations of Vision, Sinauer, 995

55 Wandell, Foundations of Vision, Sinauer, 995

56 The reason for lenses

57 Water glass refraction xhibits/naturalscience/cat-black-andwhite-domestic-shorthair-dsh-with-nose-inglass-of-water-on-bedsidetable-tweaked-mono-- JHD.jpg

58 The thin lens, first order optics ' - = R f f 2( n ) ll rays through also pass through, but only for points at -: depth of field. Forsyth&once

59 More accurate models of real lenses Finite lens thickness Higher order approximation to Chromatic aberration Vignetting sin(6 )

60 Forsyth&once Thick lens

61 Lens systems Lens systems can be designed to correct for aberrations described by 3 rd order optics Forsyth&once

62 Forsyth&once Vignetting

63 Chromatic aberration (great for prisms, bad for lenses)

64 Other (possibly annoying) phenomena Chromatic aberration Light at different wavelengths follows different paths; hence, some wavelengths are defocussed Machines: coat the lens Humans: live with it Scattering at the lens surface Some light entering the lens system is reflected off each surface it encounters (Fresnel s law gives details) Machines: coat the lens, interior Humans: live with it (various scattering phenomena are visible in the human eye)

65 Summary so far Want to make images inhole camera models the geometry of perspective projection Lenses make it work in practice Models for lenses Thin lens, spherical surfaces, first order optics Thick lens, higher-order optics, vignetting.

66 Some background material Rigid motion: translation and rotation Homogenous coordinates

67 Translation Z Y x Z Y x î k ĵ î k ĵ x Y Z O How does relate to? O Г

68 Rotation Z Y x Z Y x î k ĵ x Y Z How does relate to? R

69 Find the rotation matrix roject onto the frame s coordinate axes. Z Y X k j i O î k ĵ x Y Z N N N N N N N N N Z Y X Z Y X Z Y X Z Y X k k j k i k k j j j i j k i j i i i N N N N N N N N N Z Y X Z Y X Z Y X Z Y X k k j k i k k j j j i j k i j i i i N N N N N N N N N Z Y X Z Y X Z Y X Z Y X k k j k i k k j j j i j k i j i i i

70 Rotation matrix N N N N N N N N N Z Y X Z Y X Z Y X Z Y X k k j k i k k j j j i j k i j i i i this R implies N N N N N N N N N k k j k i k k j j j i j k i j i i i R where

71 Translation and rotation Let s write as a single matrix equation: O R Г Z Y X Z Y X O R

72 Homogenous coordinates dd an extra coordinate and use an equivalence relation for 3D equivalence relation k*(x,y,z,t) is the same as (X,Y,Z,T) Motivation ossible to write the action of a perspective camera as a matrix

73 Homogenous/non-homogenous transformations for a 3-d point From non-homogenous to homogenous coordinates: add as the 4 th coordinate, ie From homogenous to non-homogenous coordinates: divide st 3 coordinates by the 4 th, ie y x y x y x T T y x

74 Homogenous/non-homogenous transformations for a 2-d point From non-homogenous to homogenous coordinates: add as the 3 rd coordinate, ie From homogenous to non-homogenous coordinates: divide st 2 coordinates by the 3 rd, ie y x y x y x y x

75 The camera matrix, in homogenous coordinates Turn previous expression into HC s HC s for 3D point are (X,Y,Z,T) HC s for point in image are (U,V,W) T Z Y X f Y X f Z Y X Y X Z f f Z HC Non-HC What about an orthographic camera?

76 The projection matrix for orthographic projection, homogenous coordinates U V W X Y Z T X Y T T X Y HC Non-HC

77 Camera calibration Use the camera to tell you things about the world: Relationship between coordinates in the world and coordinates in the image: geometric camera calibration. (Relationship between intensities in the world and intensities in the image: photometric camera calibration, not covered in this course, see 6.8 or text)

78 Intrinsic parameters Forsyth&once erspective projection u v f f x y

79 Intrinsic parameters ut pixels are in some arbitrary spatial units u v f f x y

80 Intrinsic parameters ut pixels are in some arbitrary spatial units u v,, x y

81 Intrinsic parameters Maybe pixels are not square u v,, x y

82 Intrinsic parameters Maybe pixels are not square u v, - x y

83 Intrinsic parameters We don t know the origin of our camera pixel coordinates u v, - x y

84 Intrinsic parameters We don t know the origin of our camera pixel coordinates u v, - x y Г u Г v

85 Intrinsic parameters May be skew between camera pixel axes u v, - x y Г u Г v

86 Intrinsic parameters May be skew between camera pixel axes u v x,, cot( 6 - y Г v sin( 6 ) ) y Г u

87 Intrinsic parameters ) sin( ) cot( v y v u y x u Г Г 6-6,, ) sin( ) cot( y x v u v u 6-6,, Using homogenous coordinates, we can write this as: or: K p

88 Extrinsic parameters: translation and rotation of camera frame W C W C W C O R Г Non-homogeneous coordinates Z Y X W C C W Z Y X W W W O R C C C Homogeneous coordinates lock matrix form

89 Combining extrinsic and intrinsic calibration parameters Forsyth&once W C W C W C O R Г O R K p W C C W K p Μ p Intrinsic Extrinsic

90 Other ways to write the same equation y x T T T W W W m m m v u M p m m v m m u b b b b pixel coordinates world coordinates is in the camera coordinate system, but we can solve for that, since, leading to: m b 3

91 Calibration target

92 Camera calibration ) ( ) ( b b i i i i v m m u m m So for each feature point, i, we have: m m v m m u b b b b From before, we had these equations relating image positions, u,v, to points at 3-d positions (in homogeneous coordinates):

93 Camera calibration ) ( ) ( b b i i i i m v m u m m Stack all these measurements of i= n points into a big matrix: 3 2 m m m v u v u T n n T n T T n n T T n T T T T T T

94 3 2 m m m v u v u T n n T n T T n n T T n T T T T T T m m m m m m m m m m m m v v v v u u u u v v v v u u u u n n n ny n nx n n ny nx n n n ny n nx n n ny nx y x y x y x y x Showing all the elements: In vector form: Camera calibration

95 We want to solve for the unit vector m (the stacked one) that minimies 2 m m m m m m m m m m m m m v v v v u u u u v v v v u u u u n n n ny n nx n n ny nx n n n ny n nx n n ny nx y x y x y x y x m = The minimum eigenvector of the matrix T gives us that (see Forsyth&once, 3.) Camera calibration

96 Once you have the M matrix, can recover the intrinsic and extrinsic parameters as in Forsyth&once, sect Camera calibration