Computational Photograph and Video: Warping, Morphing and Mosaics Prof. Marc Pollefes Dr. Gabriel Brostow
Toda s schedule Last week s recap Warping Morphing Mosaics
Toda s schedule Last week s recap Warping Mosaics Morphing
Eposure: shutter vs. aperture Trade off motion blur vs. depth of field (also gain/iso sensitivit)
Imperfect lenses: aberrations, etc.
Sensors and color
Schedule Computational Photograph and Video 2 Feb Introduction to Computational Photograph 27 Feb More on Cameras, Sensors and Color Assignment 1: Color 5 Mar Warping, morphing and mosaics Assignment 2: Alignment 12 Mar Image pramids, Graphcuts Assignment 3: Blending 19 Mar Dnamic Range, HDR imaging, tone mapping Assignment 4: HDR 26 Mar Easter holida no classes 2 Apr TBD Project proposals 9 Apr TBD Papers 16 Apr TBD Papers 23 Apr TBD Papers 3 Apr TBD Project update 7 Ma TBD Papers 14 Ma TBD Papers 21 Ma TBD Papers 28 Ma TBD Final project presentation
Toda s schedule Last week s recap Warping Mosaics Morphing
Image Warping image filtering: change range of image g() = T(f()) f T f image warping: change domain of image g() = f(t()) f f T toda lots of slides from Aleei Efros
Image Warping image filtering: change range of image g() = T(f()) f T g image warping: change domain of image f g() = f(t()) T g
Parametric (global) warping Eamples of parametric warps: translation rotation aspect affine perspective clindrical
Parametric (global) warping T p = (,) p = (, ) Transformation T is a coordinate changing machine: p = T(p) What does it mean that T is global? Is the same for an point p can be described b just a few numbers (parameters) Let s represent T as a matri: p = Mp ' = M '
Scaling Scaling a coordinate means multipling each of its components b a scalar Uniform scaling means this scalar is the same for all components: 2
Scaling Non uniform scaling: different scalars per component: X 2, Y.5
Scaling operation: Or, in matri form: b a = = ' ' = b a ' ' scaling matri S What s inverse of S? Scaling
2 D Rotation (, ) (, ) θ = cos(θ) - sin(θ) = sin(θ) + cos(θ)
2 D Rotation This is eas to capture in matri form: ' ' = cos sin θ ( θ ) sin( θ ) ( ) ( ) cos θ Even though sin(θ) and cos(θ) are nonlinear functions of θ, is a linear combination of and is a linear combination of and What is the inverse transformation? Rotation b θ For rotation matrices R 1 T R = R
What tpes of transformations can be represented with a 22 matri? 2D Identit? = = ' ' = 1 1 ' ' 2D Scale around (,)? s s * ' * ' = = = s s ' ' 22 Matrices
What tpes of transformations can be represented with a 22 matri? 2D Rotate around (,)? * cos * sin ' * sin * cos ' Θ + Θ = Θ Θ = Θ Θ Θ Θ = cos sin sin cos ' ' 2D Shear? sh sh + = + = * ' * ' = sh sh 1 1 ' ' 22 Matrices
22 Matrices What tpes of transformations can be represented with a 22 matri? 2D Mirror about Y ais? ' = ' = ' ' = 1 1 2D Mirror over (,)? ' ' = = ' ' = 1 1
22 Matrices What tpes of transformations can be represented with a 22 matri? 2D Translation? ' = + t ' = + t NO! Onl linear 2D transformations can be represented with a 22 matri
All 2D Linear Transformations Linear transformations are combinations of Scale, Rotation, Shear, and Mirror Properties of linear transformations: Origin maps to origin Lines map to lines Parallel lines remain parallel Ratios are preserved Closed under composition ' ' = a c b e d g ' a = ' c f i h k b d j l
Homogeneous Coordinates Q: How can we represent translation as a 33 matri? ' = + t ' = + t
Homogeneous Coordinates Homogeneous coordinates represent coordinates in 2 dimensions with a 3 vector homogeneous coords 1
Homogeneous Coordinates Add a 3rd coordinate to ever 2D point (,, w) represents a point at location (/w, /w) (,, ) represents a point at infinit (,, ) is not allowed 2 1 (2,1,1) or (4,2,2) or (6,3,3) Convenient coordinate sstem to represent man useful transformations 1 2
Q: How can we represent translation as a 33 matri? A: Using the rightmost column: = 1 1 1 t t Translation t t + = + = ' ' Homogeneous Coordinates
Translation Eample of translation + + = = 1 1 1 1 1 1 ' ' t t t t t = 2 t = 1 Homogeneous Coordinates
Basic 2D Transformations Basic 2D transformations as 33 matrices Θ Θ Θ Θ = 1 1 cos sin sin cos 1 ' ' = 1 1 1 1 1 ' ' t t = 1 1 1 1 1 ' ' sh sh Translate Rotate Shear = 1 1 1 ' ' s s Scale
Affine Transformations Affine transformations are combinations of Linear transformations, and Translations Properties of affine transformations: Origin does not necessaril map to origin Lines map to lines Parallel lines remain parallel Ratios are preserved Closed under composition Models change of basis ' ' w = a d b e c f 1 w
Projective Transformations Projective transformations Affine transformations, and Projective warps Properties of projective transformations: Origin does not necessaril map to origin Lines map to lines Parallel lines do not necessaril remain parallel Ratios are not preserved Closed under composition Models change of basis ' ' w' = a d g b e h c f i w
Transformations can be combined b matri multiplication Θ Θ Θ Θ = w s s t t w 1 1 cos sin sin cos 1 1 1 ' ' ' p = T(t,t ) R(Θ) S(s,s ) p Matri Composition
2D image transformations These transformations are a nested set of groups Closed under composition and inverse is a member
Image Warping in Biolog D'Arc Thompson http://www groups.dcs.st and.ac.uk/~histor/miscellaneous/darc.html http://en.wikipedia.org/wiki/d'arc_thompson Importance of shape and structure in evolution Slide b Durand and Freeman
Recovering Transformations? T(,) f(,) g(, ) What if we know f and g and want to recover the transform T? willing to let user provide correspondences How man do we need?
Translation: # correspondences? How man correspondences needed for translation? How man Degrees of Freedom? What is the transformation matri? T(,)? = 1 ' 1 ' 1 p p p p M
Euclidian: # correspondences? How man correspondences needed for translation+rotation? How man DOF?? T(,)
Affine: # correspondences?? T(,) How man correspondences needed for affine? How man DOF?
Projective: # correspondences?? T(,) How man correspondences needed for projective? How man DOF?
Image warping T(,) f(,) g(, ) Given a coordinate transform (, ) = T(,) and a source image f(,), how do we compute a transformed image g(, ) = f(t(,))?
Forward warping T(,) f(,) g(, ) Send each piel f(,) to its corresponding location (, ) = T(,) in the second image Q: what if piel lands between two piels?
Forward warping T(,) f(,) g(, ) Send each piel f(,) to its corresponding location (, ) = T(,) in the second image Q: what if piel lands between two piels? A: distribute color among neighboring piels (, ) Known as splatting Check out griddata in Matlab
Inverse warping T -1 (,) f(,) g(, ) Get each piel g(, ) from its corresponding location (,)=T 1 (, ) in the first image Q: what if piel comes from between two piels?
Inverse warping T -1 (,) f(,) g(, ) Get each piel g(, ) from its corresponding location (,)=T 1 (, ) in the first image Q: what if piel comes from between two piels? A: Interpolate color value from neighbors nearest neighbor, bilinear, Gaussian, bicubic Check out interp2 in Matlab
Forward vs. inverse warping Q: which is better? A: usuall inverse eliminates holes however, it requires an invertible warp function not alwas possible...
Toda s schedule Last week s recap Warping Mosaics Morphing
Wh Mosaic? Are ou getting the whole picture? Compact Camera FOV = 5 35 Slide from Brown & Lowe
Wh Mosaic? Are ou getting the whole picture? Compact Camera FOV = 5 35 Human FOV = 2 135 Slide from Brown & Lowe
Wh Mosaic? Are ou getting the whole picture? Compact Camera FOV = 5 35 Human FOV = 2 135 Panoramic Mosaic = 36 18 Slide from Brown & Lowe
Mosaics: stitching images together virtual wide-angle camera
A pencil of ras contains all views real camera snthetic camera Can generate an snthetic camera view as long as it has the same center of projection!
How to do it? Basic Procedure Take a sequence of images from the same position Rotate the camera about its optical center Compute transformation between second image and first Transform the second image to overlap with the first Blend the two together to create a mosaic If there are more images, repeat but wait, wh should this work at all? What about the 3D geometr of the scene? Wh aren t we using it?
Aligning images left on top right on top Translations are not enough to align the images
Image reprojection mosaic PP The mosaic has a natural interpretation in 3D The images are reprojected onto a common plane The mosaic is formed on this plane Mosaic is a snthetic wide angle camera
Image reprojection Basic question How to relate two images from the same camera center? how to map a piel from PP1 to PP2 PP2 Answer Cast a ra through each piel in PP1 Draw the piel where that ra intersects PP2 But don t we need to know the geometr of the two planes in respect to the ee? PP1 Observation: Rather than thinking of this as a 3D reprojection, think of it as a 2D image warp from one image to another
Back to Image Warping Which t-form is the right one for warping PP1 into PP2? e.g. translation, Euclidean, affine, projective Translation Affine Perspective 2 unknowns 6 unknowns 8 unknowns
Homograph A: Projective mapping between an two PPs with the same center of projection rectangle should map to arbitrar quadrilateral parallel lines aren t but must preserve straight lines same as: project, rotate, reproject called Homograph w' * * * w' = * * * w * * * 1 p H p To appl a homograph H Compute p = Hp (regular matri multipl) Convert p from homogeneous to image coordinates PP1 PP2
Image warping with homographies image plane in front black area where no piel maps to image plane below
Image rectification p p To unwarp (rectif) an image Find the homograph H given a set of p and p pairs How man correspondences are needed? Trick to write H analticall, but we can solve for it! Find such H that best transforms points p into p Use least-squares!
Computing a homograph λ 1 = h 1 h 2 h 3 > > > > 1 h > 1 h > 2 h > 3 = λ = h 3 h 3 = h 1 h 3 = h 2 = 1 2 equations/point Stack matri representing equations for 4 or more points Solve through SVD, least-square solution is given b last right singular vector USV >.V 9 = σ 9 (i.e. smallest singular value) Numericall better if image coordinates scaled to [-1,1]
Panoramas 1. Pick one image (red) 2. Warp the other images towards it (usuall, one b one) 3. blend
changing camera center Does it still work? PP1 snthetic PP PP2
Planar scene (or far awa) PP3 PP1 PP2 PP3 is a projection plane of both centers of projection, so we are OK! This is how big aerial photographs are made
Planar mosaic
Toda s schedule Last week s recap Warping Mosaics Morphing
Morphing = Object Averaging The aim is to find an average between two objects Not an average of two images of objects but an image of the average object! How can we make a smooth transition in time? Do a weighted average over time t How do we know what the average object looks like? We haven t a clue! But we can often fake something reasonable Usuall required user/artist input
Averaging Points What s the average of P and Q? v = Q - P Q P Linear Interpolation (Affine Combination): New point ap + bq, defined onl when a+b = 1 So ap+bq = ap+(1-a)q P +.5v = P +.5(Q P) =.5P +.5 Q P and Q can be anthing: P + 1.5v = P + 1.5(Q P) = -.5P + 1.5 Q (etrapolation) points on a plane (2D) or in space (3D) Colors in RGB or HSV (3D) Whole images (m b n D) etc.
Idea #1: Cross Dissolve Interpolate whole images: Image halfwa = (1 t)*image 1 + t*image 2 This is called cross dissolve in film industr But what is the images are not aligned?
Idea #2: Align, then cross disolve Align first, then cross dissolve Alignment using global warp picture still valid
Dog Averaging What to do? Cross dissolve doesn t work Global alignment doesn t work Cannot be done with a global transformation (e.g. affine) An ideas? Feature matching! Nose to nose, tail to tail, etc. This is a local (non parametric) warp
Idea #3: Local warp, then cross dissolve Morphing procedure: for ever t, 1. Find the average shape (the mean dog ) local warping 2. Find the average color Cross dissolve the warped images
Local (non parametric) Image Warping Need to specif a more detailed warp function Global warps were functions of a few (2,4,8) parameters Non parametric warps u(,) and v(,) can be defined independentl for ever single location,! Once we know vector field u,v we can easil warp each piel (use backward warping with interpolation)
Image Warping non parametric Move control points to specif a spline warp Spline produces a smooth vector field
Warp specification dense How can we specif the warp? Specif corresponding spline control points interpolate to a complete warping function But we want to specif onl a few points, not a grid
Warp specification sparse How can we specif the warp? Specif corresponding points interpolate to a complete warping function How do we do it? How do we go from feature points to piels?
Triangular Mesh 1. Input correspondences at ke feature points 2. Define a triangular mesh over the points Same mesh in both images! Now we have triangle to triangle correspondences 3. Warp each triangle separatel from source to destination How do we warp a triangle? 3 points = affine warp! Just like teture mapping
Image Morphing We know how to warp one image into the other, but how do we create a morphing sequence? 1. Create an intermediate shape (b interpolation) 2. Warp both images towards it 3. Cross dissolve the colors in the newl warped images
Warp interpolation How do we create an intermediate warp at time t? Assume t = [,1] Simple linear interpolation of each feature pair (1 t)*p1+t*p for corresponding features p and p1
Slide b Durand and Freeman Morphing & matting Etract foreground first to avoid artifacts in the background
Women in Art video http://outube.com/watch?v=nudion _Hs
Problem with morphing So far, we have performed linear interpolation of feature point positions But what happens if we tr to morph between two views of the same object?
View morphing Seitz & Der http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm Interpolation consistent with 3D view interpolation
Main trick Prewarp with a homograph to "prealign" images So that the two views are parallel Because linear interpolation works when views are parallel
morph morph prewarp prewarp input output input homographies
Net week Image pramids and graph cuts