Lecture 11 MRF s (conbnued), cameras and lenses.

Transcription

1 6.869 Advances in Computer Vision Bill Freeman and Antonio Torralba Spring 2011 Lecture 11 MRF s (conbnued), cameras and lenses. remember correction on Gibbs sampling

2 Motion application image patches image scene patches scene 2

3 What behavior should we see in a motion algorithm? Aperture problem Resolution through propagation of information Figure/ground discrimination 3

4 The aperture problem 4

5 The aperture problem 4

6 The aperture problem 5

7 The aperture problem 5

8 motion program demo 6

9 Inference: Motion estimation results (maxima of scene probability distributions displayed) Image data Iterations 0 and 1 Initial guesses only show motion at edges. 7

10 Motion estimation results (maxima of scene probability distributions displayed) Figure/ground still unresolved here. Iterations 2 and 3 8

11 Motion estimation results (maxima of scene probability distributions displayed) Iterations 4 and 5 Final result compares well with vector quantized true (uniform) velocities. 9

12 Vision applications of MRF s Stereo Motion estimation Labelling shading and reflectance Segmentation Many others 10

13 Random Fields for segmentation I = Image pixels (observed) h = foreground/background labels (hidden) one label per pixel θ = Parameters Posterior Joint Likelihood Prior 11

14 Random Fields for segmentation I = Image pixels (observed) h = foreground/background labels (hidden) one label per pixel θ = Parameters Posterior Joint Likelihood Prior 11

15 Random Fields for segmentation I = Image pixels (observed) h = foreground/background labels (hidden) one label per pixel θ = Parameters Posterior Joint Likelihood Prior 1. Generative approach models joint Markov random field (MRF) 2. Discriminative approach models posterior directly Conditional random field (CRF) 11

16 Generative Markov Random Field i I (pixels) j Image Plane 12

17 Generative Markov Random Field h (labels) {foreground,b ackground} h j h i i I (pixels) j Image Plane 12

18 Generative Markov Random Field h (labels) {foreground,b ackground} h j h i Pairwise Potential (MRF) ψ ij (h i, h j θ ij ) i I (pixels) j Image Plane 12

19 Generative Markov Random Field h (labels) {foreground,b ackground} h j h i i Pairwise Potential (MRF) ψ ij (h i, h j θ ij ) Unary Potential φ i (I h i,θ i ) I (pixels) j Image Plane 12

20 Generative Markov Random Field Likelihood MRF Prior h (labels) {foreground,b ackground} h j h i i Pairwise Potential (MRF) ψ ij (h i, h j θ ij ) Unary Potential φ i (I h i,θ i ) I (pixels) j Image Plane 12

21 Generative Markov Random Field Likelihood MRF Prior h (labels) {foreground,b ackground} h j h i i Pairwise Potential (MRF) ψ ij (h i, h j θ ij ) Unary Potential φ i (I h i,θ i ) I (pixels) j Image Plane Prior has no dependency 12 on I

22 Conditional Random Field Discriminative approach Lafferty, McCallum and Pereira 2001 Unary Pairwise 13

23 Conditional Random Field Discriminative approach Lafferty, McCallum and Pereira 2001 Unary Pairwise h i h j i I (pixels) j Image Plane 13

24 Conditional Random Field Discriminative approach Lafferty, McCallum and Pereira 2001 Dependency on I allows introduction of pairwise terms that make use of image. For example, neighboring labels should be similar only if pixel colors are similar Contrast term I (pixels) Unary Pairwise Image Plane h i h j i j 13

25 Conditional Random Field Discriminative approach Lafferty, McCallum and Pereira 2001 Dependency on I allows introduction of pairwise terms that make use of image. For example, neighboring labels should be similar only if pixel colors are similar Contrast term e.g Kumar and Hebert 2003 I (pixels) Unary Pairwise Image Plane h i h j i j 13

26 OBJCUT Unary Kumar, Torr & Zisserman 2005 Pairwise Color Likelihood Distance from Ω Label smoothness Contrast Ω is a shape prior on the labels from a Layered Pictorial Structure (LPS) model Ω (shape parameter) Segmentation by: - Match LPS model to image (get number of samples, each with a different pose -Marginalize over the samples using a single graph cut [Boykov & Jolly, 2001] 14

27 OBJCUT Unary Kumar, Torr & Zisserman 2005 Pairwise Color Likelihood Distance from Ω Label smoothness Contrast Ω is a shape prior on the labels from a Layered Pictorial Structure (LPS) model Segmentation by: Ω (shape parameter) h i - Match LPS model to image (get number of samples, each with a different pose h j i -Marginalize over the samples using a single graph cut [Boykov & Jolly, 2001] I (pixels) j Image Plane 14 Figure from Kumar et al., CVPR 2005

28 OBJCUT: Shape prior - Ω - Layered Pictorial Structures (LPS) Generative model Composition of parts + spatial layout Layer 2 Layer 1 Spatial Layout (Pairwise Configuration) Parts in Layer 2 can occlude parts in Layer 1 15 Kumar, et al. 2004, 2005

29 Using LPS Model for Cow In the absence of a clear boundary between object and background Image OBJCUT: Results Segmentation 16

30 Outline of MRF section Inference in MRF s. Gibbs sampling, simulated annealing Iterated conditional modes (ICM) Belief propagation Application example super-resolution Graph cuts Variational methods Learning MRF parameters. Iterative proportional fitting (IPF) 17

31 True joint probability 18

32 True joint probability Observed marginal distributions 18

33 Initial guess at joint probability 19

34 IPF update equation, for maximum likelihood estimate of joint probability Scale the previous iteration s estimate for the joint probability by the ratio of the true to the predicted marginals. Gives gradient ascent in the likelihood of the joint probability, given the observations of the marginals. See: Michael Jordan s book on graphical models 20

35 Convergence to correct marginals by IPF algorithm 21

36 Convergence of to correct marginals by IPF algorithm 22

37 IPF results for this example: comparison of joint probabilities True joint probability Initial guess Final maximum entropy estimate 23

38 Application to MRF parameter estimation Can show that for the ML estimate of the clique potentials, φ c (x c ), the empirical marginals equal the model marginals, Because the model marginals are proportional to the clique potentials, we have the IPF update rule for φ c (x c ), which scales the model marginals to equal the observed marginals: Performs coordinate ascent in the likelihood of the MRF parameters, given the observed data. Reference: unpublished notes by Michael Jordan, and by Roweis:

39 Learning MRF parameters, labeled data Iterative proportional fitting lets you make a maximum likelihood estimate a joint distribution from observations of various marginal distributions. Applied to learning MRF clique potentials: (1) measure the pairwise marginal statistics (histogram state co-occurrences in labeled training data). (2) guess the clique potentials (use measured marginals to start). (3) do inference to calculate the model s marginals for every node pair. (4) scale each clique potential for each state pair by the empirical over model marginal 25

40 Image formation 26

41 The structure of ambient light

42 The structure of ambient light

43 The structure of ambient light

44 The structure of ambient light

45 The structure of ambient light

46 The structure of ambient light

47 The structure of ambient light

48 The structure of ambient light

49 The PlenopBc FuncBon Adelson & Bergen, 91

50 Measuring the PlenopBc funcbon Why is there no picture appearing on the paper?

51 Light rays from many different parts of the scene strike the same point on the paper. Forsyth & Ponce

52 Measuring the PlenopBc funcbon The camera obscura The pinhole camera

53 Light rays from many different parts of the scene strike the same point on the paper. The pinhole camera only allows rays from one point in the scene to strike each point of the paper. Forsyth & Ponce

54 pinhole camera demos 34

55 Problem Set 7

56 Problem Set 7

57 Effect of pinhole size Wandell, Foundations of Vision, Sinauer, 1995

58 Wandell, Foundations of Vision, Sinauer, 1995

59 Playing with pinholes

60 Two pinholes

61 Anaglyph pinhole camera

62 Anaglyph pinhole camera

63 Anaglyph pinhole camera

64 Synthesis of new views Anaglyph

65 Problem set 7 Build the device Take some pictures and put them in the report Take anaglyph images Calibrate camera Recover depth for some points in the image

66 Cameras, lenses, and calibration Camera models Projections Calibration Lenses

67 Perspective projection camera world Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) Ignore the third coordinate, and get (x,y,z) ( f x z, f y z )

68 Perspective projection camera world y Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) Ignore the third coordinate, and get (x,y,z) ( f x z, f y z )

69 Perspective projection camera world z y Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) Ignore the third coordinate, and get (x,y,z) ( f x z, f y z )

70 Perspective projection camera world z y y Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) Ignore the third coordinate, and get (x,y,z) ( f x z, f y z )

71 Perspective projection camera world f z y y Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) Ignore the third coordinate, and get (x,y,z) ( f x z, f y z )

72 Perspective projection camera world f z y y Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) Ignore the third coordinate, and get (x,y,z) ( f x z, f y z )

73 Common to draw film plane in front of the focal point. Moving the film plane merely scales the image. Forsyth&Ponce

74 Geometric properties of projection Points go to points Lines go to lines Planes go to whole image or half-planes. Polygons go to polygons Degenerate cases line through focal point to point plane through focal point to line

75 Line in 3-space

76 Line in 3-space Perspective projection of that line x'(t) = fx z = f (x 0 + at) z 0 + ct y'(t) = fy z = f (y 0 + bt) z 0 + ct

77 Line in 3-space Perspective projection of that line x'(t) = fx z = f (x 0 + at) z 0 + ct y'(t) = fy z = f (y 0 + bt) z 0 + ct In the limit as we have (for ):

78 Line in 3-space Perspective projection of that line x'(t) = fx z = f (x 0 + at) z 0 + ct y'(t) = fy z = f (y 0 + bt) z 0 + ct In the limit as we have (for ): x'(t) y'(t) fa c fb c

79 Line in 3-space Perspective projection of that line x'(t) = fx z = f (x 0 + at) z 0 + ct y'(t) = fy z = f (y 0 + bt) z 0 + ct In the limit as we have (for ): This tells us that any set of parallel lines (same a, b, c parameters) project to the same point (called the vanishing point). x'(t) y'(t) fa c fb c

80 Vanishing point y camera z

81 Vanishing point y camera z

82 Vanishing point y camera z

83 Vanishing point y camera z

84 Vanishing point y camera z

85 Vanishing point y camera z

86 Vanishing point y camera z

87 Vanishing point y camera z

88 graphics/two_point_perspective.html

89 Vanishing points Each set of parallel lines (=direction) meets at a different point The vanishing point for this direction Sets of parallel lines on the same plane lead to collinear vanishing points. The line is called the horizon for that plane

90 What if you photograph a brick wall head-on? y x

91 What if you photograph a brick wall head-on? y x

92 Brick wall line in 3-space

93 Brick wall line in 3-space Perspective projection of that line x'(t) = f (x 0 + at) z 0 y'(t) = f y 0 z 0

94 Brick wall line in 3-space Perspective projection of that line x'(t) = f (x 0 + at) z 0 y'(t) = f y 0 z 0 All bricks have same z 0. Those in same row have same y 0 Thus, a brick wall, photographed head-on, gets rendered as set of parallel lines in the image plane.

95 Other projection models: Orthographic projection

96 Other projection models: Weak perspective Issue perspective effects, but not over the scale of individual objects collect points into a group at about the same depth, then divide each point by the depth of its group Adv: easy Disadv: only approximate

97 Three camera projections 3-d point 2-d image position (1) Perspective: (2) Weak perspective: (3) Orthographic:

98 Homogeneous coordinates Is the perspective projection a linear transformation? Slide by Steve Seitz

99 Homogeneous coordinates Is the perspective projection a linear transformation? no division by z is nonlinear Slide by Steve Seitz

100 Homogeneous coordinates Is the perspective projection a linear transformation? no division by z is nonlinear Trick: add one more coordinate: homogeneous image coordinates homogeneous scene coordinates Converting from homogeneous coordinates Slide by Steve Seitz

101 Perspective Projection Projection is a matrix multiply using homogeneous coordinates: x y z 0 0 1/ f 0 1 x = y z / f f x z, f y z This is known as perspective projection The matrix is the projection matrix Slide by Steve Seitz

102 Perspective Projection How does scaling the projection matrix change the transformation? Slide by Steve Seitz / f 0 x y z 1 = x y z / f f x z, f y z

103 Perspective Projection How does scaling the projection matrix change the transformation? Slide by Steve Seitz / f 0 x y z 1 = x y z / f f x z, f y z f f x y z 1

104 Perspective Projection How does scaling the projection matrix change the transformation? Slide by Steve Seitz / f 0 x y z 1 = x y z / f f x z, f y z f f x y z 1 = fx fy z

105 Perspective Projection How does scaling the projection matrix change the transformation? Slide by Steve Seitz / f 0 x y z 1 = x y z / f f x z, f y z f f x y z 1 = fx fy z f x z, f y z

106 Orthographic Projection Special case of perspective projection Distance from the COP to the PP is infinite Image World Also called parallel projection What s the projection matrix?? Slide by Steve Seitz

107 Orthographic Projection Special case of perspective projection Distance from the COP to the PP is infinite Image World Also called parallel projection What s the projection matrix?? Slide by Steve Seitz

108 Orthographic Projection Special case of perspective projection Distance from the COP to the PP is infinite Image World Also called parallel projection What s the projection matrix? Slide by Steve Seitz

109 Orthographic Projection Special case of perspective projection Distance from the COP to the PP is infinite Image World Also called parallel projection What s the projection matrix? Slide by Steve Seitz

110 Camera calibration Use the camera to tell you things about the world: Relationship between coordinates in the world and coordinates in the image: geometric camera calibration, see Szeliski, section 5.2, 5.3 for references (Relationship between intensities in the world and intensities in the image: photometric image formation, see Szeliski, sect. 2.2.)

111 One reason to calibrate a camera Measuring height v r vanishing line (horizon) v v t 0 H t R H v b 0 b

112 Another reason to calibrate a camera World image point Depth of image point Foca optical center (left) baseline optical center (right) Slide credit: Kristen

113 Translation and rotation A p x p A p y A p z B A t

114 Translation and rotation B p = B R A p + B t A A A p y A p x A p z B A t p

115 Translation and rotation as described in the coordinates of frame B B p = B R A p + B t A A A p y A p x A p z B A t p

116 Translation and rotation as described in the coordinates of frame B Let s write as a single matrix equation: B p = B R A p + B t A A A p y A p x A p z B A t p

117 Translation and rotation as described in the coordinates of frame B Let s write as a single matrix equation: B p = B R A p + B t A A A p y A p x A p z B A t p B p x B p y B p z 1 = B A R B A t A p x A p y A p z 1

118 Translation and rotation, written in each set of coordinates Non-homogeneous coordinates B p = B R A p + B t A A

119 Translation and rotation, written in each set of coordinates Non-homogeneous coordinates B p = B R A p + B t A A Homogeneous coordinates B p = B C A p A where B A C = B A R B A t

120 Intrinsic parameters: from idealized world coordinates to pixel values Forsyth&Ponce Perspective projection

121 Intrinsic parameters But pixels are in some arbitrary spatial units

122 Intrinsic parameters Maybe pixels are not square

123 Intrinsic parameters We don t know the origin of our camera pixel coordinates

124 Intrinsic parameters May be skew between camera pixel axes

125 Intrinsic parameters May be skew between camera pixel axes

126 Intrinsic parameters May be skew between camera pixel axes

127 Intrinsic parameters, homogeneous coordinates

128 Intrinsic parameters, homogeneous coordinates Using homogenous coordinates, we can write this as:

129 Intrinsic parameters, homogeneous coordinates u v 1 = α α cot(θ) u 0 0 β sin(θ) v x y z 1 Using homogenous coordinates, we can write this as:

130 Intrinsic parameters, homogeneous coordinates u v 1 = α α cot(θ) u 0 0 β sin(θ) v x y z 1 Using homogenous coordinates, we can write this as: or:

131 Intrinsic parameters, homogeneous coordinates or: Using homogenous coordinates, we can write this as: u v = 1 α α cot(θ) u 0 β 0 v 0 sin(θ) p = K 0 x y 0 0 z 1 C p

132 Intrinsic parameters, homogeneous coordinates or: Using homogenous coordinates, we can write this as: In pixels u v = 1 α α cot(θ) u 0 β 0 v 0 sin(θ) p = K In camera-based coords 0 x y 0 0 z 1 C p

133 Extrinsic parameters: translation and rotation of camera frame

134 Extrinsic parameters: translation and rotation of camera frame C p = C R W p + C t W W Non-homogeneous coordinates

135 Extrinsic parameters: translation and rotation of camera frame C p = C R W p + C t W W Non-homogeneous coordinates C p C = W R C W t W p Homogeneous coordinates

136 Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates C p = K C p p C = W R C W t W p Intrinsic Extrinsic Forsyth&Ponce

137 Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates C p = K C p p C = W R C W t W p Intrinsic World coordinate Extrinsic Forsyth&Ponce

138 Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates Camera coordinates C p = K C p p C = W R C W t W p Intrinsic World coordinate Extrinsic Forsyth&Ponce

139 Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates pixels Camera coordinates C p = K C p p C = W R C W t W p Intrinsic World coordinate Extrinsic Forsyth&Ponce

140 Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates pixels Camera coordinates C p = K C p p C = W R C W t p = K C W R ( ) W C W t W p p Intrinsic World coordinate Extrinsic Forsyth&Ponce

141 Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates pixels Camera coordinates Forsyth&Ponce C p = K C p p C = W R C W t p = K C W R p = M W p ( ) W C W t W p p Intrinsic World coordinate Extrinsic

142 Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates pixels Camera coordinates Forsyth&Ponce C p = K C p p C = W R C W t p = K C W R p = M W p ( ) W C W t W p p Intrinsic World coordinate Extrinsic

143 Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates pixels Camera coordinates Forsyth&Ponce C p = K C p p C = W R C W t p = K C W R p = M W p ( ) W C W t W p p Intrinsic World coordinate Extrinsic

144 Other ways to write the same equation pixel coordinates p = M W p world coordinates u v = 1 T. m 1 T. m 2 T. m W p x W p y W p z 1 Conversion back from homogeneous coordinates leads to: u = m T 1 P m T 3 P v = m 2 P m T 3 P T

145 Calibration target Find the position, u i and v i, in pixels, of each calibration object feature point.

146 Camera calibration

147 Camera calibration From before, we had these equations relating image positions, u,v, to points at 3-d positions P (in homogeneous coordinates):

148 Camera calibration From before, we had these equations relating image positions, u,v, to points at 3-d positions P (in homogeneous coordinates): So for each feature point, i, we have:

149 Camera calibration Stack all these measurements of i=1 n points into a big matrix:

150 Camera calibration Stack all these measurements of i=1 n points into a big matrix:

151 In vector form: Camera calibration Showing all the elements:

152 Camera calibration Q m = 0 We want to solve for the unit vector m (the stacked one) that minimizes The minimum eigenvector of the matrix Q T Q gives us that (see Forsyth&Ponce, 3.1), because it is the unit vector x that minimizes x T Q T Q x.

153 Camera calibration Once you have the M matrix, can recover the intrinsic and extrinsic parameters.

154 Why do we need lenses?

155 The reason for lenses: light gathering ability Hidden assumption in using a lens: the BRDF of the surfaces you look at will be well-behaved

156 Animal Eyes Animal Eyes. Land & Nilsson. Oxford Univ. Press

157 λ 1 = c ω n 1 λ 1 = Lsin(α 1 ) n 1 sin(α 1 ) = Derivation of Snell s law c ω L = n 2 sin(α 2 ) α 1 α 2 α 1 L Wavelength of light waves scales inversely with n. This requires that plane waves bend, according to n 1 sin(α 1 ) = n 2 sin(α 2 ) n 1 n 2 α 2

158 Refraction: Snell s law For small angles,

159 Spherical lens

160 Forsyth and Ponce

161 First order optics f D/2

162 Paraxial refraction equation Relates distance from sphere and sphere radius to bending angles of light ray, for lens with one spherical surface.

163 Paraxial refraction equation Relates distance from sphere and sphere radius to bending angles of light ray, for lens with one spherical surface.

164 Paraxial refraction equation Relates distance from sphere and sphere radius to bending angles of light ray, for lens with one spherical surface.

165 Deriving the lensmaker s formula

166 The thin lens, first order optics The lensmaker s equation: Forsyth&Ponce

167 What camera projection model applies for a thin lens?

168 What camera projection model applies for a thin lens? The perspective projection of a pinhole camera. But note that many more of the rays leaving from P arrive at P

169 Lens demonstration Verify: Focusing property Lens maker s equation

170 More accurate models of real lenses Finite lens thickness Higher order approximation to Chromatic aberration Vignetting

171 Thick lens Forsyth&Ponce

172 Third order optics f D/2

173 Paraxial refraction equation, 3 rd order optics Forsyth&Ponce

174 Spherical aberration (from 3 rd order optics Transverse spherical aberration Longitudinal spherical aberration Forsyth&Ponce

175 Other 3 rd order effects Coma, astigmatism, field curvature, distortion. Forsyth&Ponce no distortion pincushion distortion barrel distortion

176 Lens systems Lens systems can be designed to correct for aberrations described by 3 rd order optics Forsyth&Ponce

177 Vignetting Forsyth&Ponce

178 Chromatic aberration (desirable for prisms, bad for lenses)

179 Other (possibly annoying) phenomena Chromatic aberration Light at different wavelengths follows different paths; hence, some wavelengths are defocussed Machines: coat the lens Humans: live with it Scattering at the lens surface Some light entering the lens system is reflected off each surface it encounters (Fresnel s law gives details) Machines: coat the lens, interior Humans: live with it (various scattering phenomena are visible in the human eye)

180 Summary Want to make images Pinhole camera models the geometry of perspective projection Lenses make it work in practice Models for lenses Thin lens, spherical surfaces, first order optics Thick lens, higher-order optics, vignetting.