Domain. operations. Image Warping and Morphing. Domain transform. Transformation. BIL721: Computational Photography!

Transcription

1 Image Warping and Morphing Domain Photo by Jeffrey Martin BIL721: Computational Photography! Aykut Erdem! Spring 2015, Lecture 7! Hacettepe University! Computer Vision Lab (HUCVL)! operations Domain transform Apply a function f from R 2 to R 2 to the image domain if (y, x) had color c in the input, f(y, x) has color c in the output Transformation Simple parametric transformations linear, affine, perspective, etc translation! rotation! aspect! affine! perspective! cylindrical!

2 Warping Imagine your image is made of rubber Warp the rubber Application of warping: weight loss Liquify in photoshop

3 Domain transform issues Apply a function f from R2 to R2 to the image domain looks easy enough But two big issues: which direction do we transform how do we deal with non-integer coordinates? Basic resampling And for warping: how do we specify f? Naïve scaling Take all the pixels in input and transform them to their output location im[y, x] => out[k*y, k*x] Use the inverse transform!!!!! Main loop on output pixels out[y, x] <=im[y/k, x/k] def scalebad(im, k): out = constantim(im.shape[0]*k, im.shape[1]*k, 0) for y, x in imiter(im): out[k*y, k*x]= im[y, x] return out def scalewithinverse(im, k): out = constantim(im.shape[0]*k, im.shape[1]*k, 0.0) for y, x in imiter(out): out[y,x]=im[y/k, x/k] return out

4 Remaining problem A little too blocky Because we round to the nearest integer pixel coordinates called nearest neighbor reconstruction Linear reconstruction im is now a 1D array along x reconstruct im[1.3] =0.7*im[1]+0.3*im[2] Bilinear reconstruction Recall nearest neighbor Take 4 nearest neighbors Weight according to x & y fractional coordinates Can be done using two 1D linear reconstructions along x then y (or y then x)

5 Bilinear Take home messages Main loop over OUTPUT pixels Makes sure you cover all of them Use inverse transform Reconstruction makes a difference Linear much better than nearest neighbor Lookup Better reconstruction Bilinear Consider more than 4 pixels: bicubic, Lanczos, etc. Try to sharpen edges Use training database of low-res/high-res pairs

6 Bicubic (Photoshop) Ignore small color issues Bill Freeman s super-resolution Important scientific question How to turn Dr. Jekyll into Mr. Hyde? How to turn a man into a werewolf? Warping & Powerpoint cross-fading? Morphing

7 Important scientific question How to turn Dr. Jekyll into Mr. Hyde? How to turn a man into a werewolf? Powerpoint cross-fading? Digression: old metamorphoses Unless I m mistaken, both employ the trick of making already-applied makeup turn visible via changes in the color of the lighting, something that works only in black-and-white cinematography. It s an interesting alternative to the more familiar Wolf Man time-lapse dissolves. This technique was used to great effect on Fredric March in Rouben Mamoulian s 1932 film of Dr. Jekyll and Mr. Hyde, although Spencer Tracy eschewed extreme makeup for his 1941 portrayal. or Image Warping and Morphing? From An American Werewolf in London Demo of morphing Jekyll & Hide 1932: 35:13 ch18 1:06:45 ch19 1:17:50 Jekyll & Hide 1941: ch20 1:25:13

8 Challenge Smoothly transform a face into another Related: slow motion interpolation interpolate between key frames Averaging images Cross-fading Pretty much the compositing equation C=t F +(1-t) B Problem with cross fading Features (eyes, mouth, etc) are not aligned It is probably not possible to get a global alignment We need to interpolate the LOCATION of features domain transform! Averaging vectors (location) P & Q are two 2D points (in the domain ) V= t P + (1-t) Q P V Q

9 Warping Move pixel spatially: C (x,y)=c(f(x,y)) Leave colors unchanged Warping Deform the domain of images (not range) Central to morphing Also useful for Optical aberration correction Video stabilization Slimming people down Recap Color (range) interpolation: C=t F +(1-t) B Location (domain) interpolation: V= t P + (1-t) Q Warping: C (x,y)=c(f(x,y)) Morphing combine both For each pixel Transform its location like a vector (domain) Then linearly interpolate like an image (range)

10 Morphing Input: two images I 0 and I N Expected output: image sequence I i, with i 1..N-1 User specifies sparse correspondences on the images Pairs of vectors {(P 0 j, P N j)} Morphing For each intermediate frame I t Interpolate feature locations P t i= (1- t) P 0 i + t P 1 i Perform two warps: one for I 0, one for I 1 Deduce a dense warp field from the pairs of features Warp the pixels Linearly interpolate the two warped images t=0 t=1 t=0.5 Warping Imagine your image is made of rubber warp the rubber Warping No prairie dogs were armed when creating this image

11 Warping Assume we are given the spatial mapping How do we compute the warped image? Careful: warp vs. inverse warp How do you perform a given warp: Forward warp For each input pixel, compute output location and copy color there Forward warp and gaps Careful: warp vs. inverse warp Gap How do you perform a given warp: Forward warp Potential gap problems Inverse lookup the most useful For each output pixel Lookup color at inverse-warped location in input

12 Warping in action Warping in action! +" +" Target! Source! Target! Source! Slide by Tamara Berg =! Result! Slide by Tamara Berg User clicks on 4 points in target, 4 points in source.! Find best affine transformation between source subimg and target subimg.! Transform the source subimg accordingly.! Insert the transformed source subimg into the target image.!! Warping in action! Warping in action +" Target! Source! Slide by Tamara Berg =! Result! Slide by Tamara Berg HP commercial!

13 Image Morphing - Women in Art video Morphing = Object Averaging Slide by Tamara Berg The aim is to find an average between two objects! Not an average of two images of objects! but an image of the average object!! How can we make a smooth transition in time?! Do a weighted average over time t! How do we know what the average object looks like?! We haven t a clue! But we can often fake something reasonable! Usually required user/artist input! Idea #1: Cross-Dissolve Idea #2: Align, then cross-disolve Interpolate whole images:! Image halfway = (1 - t)*image 1 + t*image 2! This is called cross-dissolve in film industry! But what is the images are not aligned?! Align first, then cross-dissolve! Alignment using global warp picture still valid!

14 Jason Salavon: The Late Night Triad! Jason Salavon: 100 Special Moments! Slide by Charles R. Dyer Slide by Charles R. Dyer Dog Averaging Idea #3: Local warp, then cross-dissolve What to do?! Cross-dissolve doesn t work! Global alignment doesn t work! Cannot be done with a global transformation (e.g. affine)! Any ideas?! Feature matching! Nose to nose, tail to tail, etc.! This is a local (non-parametric) warp! Morphing procedure:! For every t,! 1. Find the average shape (the mean dog!)! local warping! 2. Find the average color! Cross-dissolve the warped images!

15 Local (non-parametric) Image Warping Image Warping non-parametric Move control points to specify a spline warp! Spline produces a smooth vector field! Need to specify a more detailed warp function! Global warps were functions of a few (2, 4, 8) parameters! Non-parametric warps u(x, y) and v(x, y) can be defined independently for every single location x, y! Once we know vector field u, v we can easily warp each pixel (use backward warping with interpolation)! Warp specification - dense Warp specification - sparse How can we specify the warp?! Specify corresponding spline control points! interpolate to a complete warping function! How can we specify the warp?! Specify corresponding points! interpolate to a complete warping function! How do we do it?!! But we want to specify only a few points, not a grid! How do we go from feature points to pixels?!

16 Warp as interpolation We are looking for a warping field A function that given a 2D point, returns a warped 2D point We have a sparse number of correspondences These specify values of the warping field This is an interpolation problem Given sparse data, find smooth function Interpolation in 1D You know the function at a sparse set of points Simplest interpolation? Piecewise linear! Interpolation in 1D What is that function we interpolate? Warp (absolute or relative displacement) In 1D, pairs Now what about 2D What is the equivalent of piecewise linear in 2D? What is the equivalent of a segment? Triangles and triangulation! Coordinate in image 1 Coordinate in image 0

17 Triangular Mesh Triangulations A triangulation of set of points in the plane is a partition of the convex hull to triangles whose vertices are the points, and do not contain other points.! 1. Input correspondences at key feature points! 2. Define a triangular mesh over the points! Same mesh in both images! Now we have triangle-to-triangle correspondences! 3. Warp each triangle separately from source to destination! How do we warp a triangle?! 3 points = affine warp! Just like texture mapping! There are an exponential number of triangulations of a point set.! An O(n 3 ) Triangulation Algorithm Quality Triangulations Repeat until impossible:! Select two sites.! Let α(t) = (α 1, α 2,.., α 3t ) be the vector of angles in the triangulation T in increasing order.! If the edge connecting them does not intersect previous edges, keep it.! A triangulation T 1 will be better than T 2 if the smallest angle of T 1 is larger than the smallest angle of T 2! The Delaunay triangulation is the best (maximizes the smallest angles)! good bad

18 ! Improving a Triangulation In any convex quadrangle, an edge flip is possible. If this flip improves the triangulation locally, it also improves the global triangulation.! Illegal Edges Lemma: An edge pq is illegal iff one of its opposite vertices is inside the circle defined by the other three vertices.! Proof: By Thales theorem.! p q If an edge flip improves the triangulation, the first edge is called illegal.! Theorem: A Delaunay triangulation does not contain illegal edges.! Corollary: A triangle is Delaunay iff the circle through its vertices is empty of other sites.! Corollary: The Delaunay triangulation is not unique if more than three sites are co-circular.! Naïve Delaunay Algorithm Delaunay Triangulation by Duality Start with an arbitrary triangulation. Flip any illegal edge until no more exist.! Could take a long time to terminate.! Draw the dual to the Voronoi diagram by connecting each two neighboring sites in the Voronoi diagram.! The DT may be constructed in O(nlogn) time! This is what Matlab s delaunay function uses!

19 ! Triangular Mesh Example: warping triangles B B?# 1. Input correspondences at key feature points! 2. Define a triangular mesh over the points! Same mesh in both images! Now we have triangle-to-triangle correspondences! 3. Warp each triangle separately from source to destination! How do we warp a triangle?! 3 points = affine warp! A Source T(x,y)# C A Destination Given two triangles: ABC and A B C in 2D (12 numbers)! Need to find transform T to transfer all pixels from one to the other.! How can we compute the transformation matrix:! & x' # & a $ y'! = $ d $! $ $ % 1!" $ % 0 b e 0 c #& x# f! $ y!! $! 1! " $ % 1! " C Image Morphing Warp interpolation We know how to warp one image into the other, but how do we create a morphing sequence?! 1. Create an intermediate shape (by interpolation)! 2. Warp both images towards it! 3. Cross-dissolve the colors in the newly warped images! How do we create an intermediate warp at time t?! Assume t = [0,1]! Simple linear interpolation of each feature pair! (1-t)*p 1 +t*p 0 for corresponding features p 0 and p 1!

20 Slide by Fredo Durand and Bill Freeman Morphing & matting Extract foreground first to avoid artifacts in the background! Slide by Tamara Berg Summary of morphing 1. Define corresponding points! 2. Define triangulation on points! Use same triangulation for both images! 3. For each t = 0:step:1! a. Compute the average shape (weighted average of points)! b. For each triangle in the average shape! Get the affine projection to the corresponding triangles in each image! For each pixel in the triangle, find the corresponding points in each image and set value to weighted average! c. Save the image as the next frame of the sequence! Beier and Neely Specify warp based on pairs of segments Feature-Based Metamorphosis, SIGGRAPH 1992 Used in Michael Jackson Black and White Music Video Pset 2!! Segment-based warping

21 Problem statement Inputs: One image, two lists of segments before and after, in the image domain Goal: warp the image following the displacement of the segments Idea Each before/after pair of segment implies a planar transformation simple and linear Single line transforms Idea Test Each before/after pair of segment implies a planar transformation simple and linear Single line transforms

22 Idea Transform wrt 1 segment Each before/after pair of segment implies a planar transformation Then take weighted average of transformations Define a coordinate system with respect to segment 1 dimension, u, along segment 1 dimension, v, orthogonal to segment Compute u, v in one image tation The after one, because we use the inverse transform find the corresponding transformations based on a single l find the X in the source image for that transformations are possible. Compute destinationlmage(x) point corresponding to u, v in secondaffine image shears are not possible to specify. = sourcelmage(x ) Comp For each pixel X in the destination image but different positions specifie U,V U,V Q Q b v Single line transforms Transform wrt 2 lines x u 1V x I P Destination Imw-ze Source Image Figure 1: Single line ptiir In Figure 1, X is the location to sample the source image for the pixel at X in the destination image. The location is at a distance v (the distance from the line to the pixel in the source image) from the line P Q. and at a proportion u along that line. Computing u, v 3.3 Transformation with Mult Multiple pairs of lines specify more weighting of the coordinate transfor formed. A position Xi is calculated displacement Di=Xi - Xis the differ inthesource and destination image those displacements is calculated. T distance from X to the line, This av the current pixel location X to deter in the source image. The single Iine of the multiple Iinecase, assuming anywhere in the image. The weight strongest when the pixel is exactly on the pixel is from it. The equation w Transforming a point given u, v The algorithm transforms each pikel coordinate by a rotation, translation, itnd/or a scale, thereby transforming the whole image. All of the pixels along the line in the source image are copied on top of the line in the destination image. Because the u coordinate is normalized try the length of the line, and the v coordinate is not (it is always distance in pixels), the images is scaled along the direction of the lines by the ratio of the lengths of the lines, The scale is only along the direction of the line. We have tried scaling the v coordinate by the length of the line, so that the scaling is always uniform, but found that the given formulation is more useful. X =P +up Q+v perpendicular(p Q )/ P Q u = PX.PQ/ PQ 2 is the length of a line. where The u component is scaled according to segment scaling this way u is 0 at P and 1 at Q pixel to the line, and a, b, and p are change the relative effect of the line But v is absolute (see output3) v=px.perpendicular(pq)/ PQ If a is barely greater than zero, then They say they tried to scale v as well but it didn t work the pixel is zero, the strength is nea where perpendicular(pq) is PQ rotated by 90 degrees, a, the user knows that pixels on the as well tation tation of an image. All of a wants them. Values larger than that and has length PQ find the corresponding transformations based on a single line pair are affine, but not all find the corresponding transformations based on a with singlelessline pair are affine,thb in,g,but precise control. find the X in the source image for that affine transformations are possible. In particular, uniform scales and find the X in the source image for that affine transformationsrelative are possible. uniform strength In of particular, different lines Fa unlike u which is normalized, v is in distance units shears are not possible to specify. shears are not possible specify. destinationlmage(x) = sourcelmage(x ) destinationlmage(x) = sourcelmage(x ) then toevery pixel will be affected on Computer For each pixel X in the destination image but different positions specifies Graphics, 26, 2, July 1992 a translation For each pixel X in the destination U,V u 1V x I P Destination Imw-ze specifies 26, a translation Source Image Figure 1: Single line ptiir In Figure 1, X is the location to sample the source image for the pixel at X in the destination image. The location is at a distance v zero, then each pixel will be affected Q Q 3.3 Transformation with Pairs of Lin 3.3 Transformation with Multiple Pairs of Lines bin the rangemultiple [().5, 2] are the most u in the range more [0, 1];complex if it is zero, then Multiple pairs of lines specify transform Multiple pairs of lines specify more complex transformations. A if it is one,transformations then longer lines V x weighting of the coordinate for have each al weighting of the coordinate transformations for each line is perv x lines, formed. A position shorter Xi is calculated for each pair of lin formed. A position Xi is calculated for each pair of lines. The I I I I I I I I 1 Ill I I I I I I I 1I I I displacement Di=Xi - Xis the difference between the p displacement Di=Xi - Xis the difference between the pixel location I I I I ]ihtlhtli!llllllllllll lli\l]ll1/11 inthesource and destination The multiple line algorithm is as foa images, anda weighted inthesource and destination images, anda weighted average of u those displacements is calculated. The weight is determ those displacements is calculated. The weight is determined by the For line, each This pixel average X in thedisplacement destination distance from X to the distance from X to the line, This average displacement is added to = (0,0) the position X the current pixel locationdsum X to determine the current pixel location X to determine the position X to sample P () falls out as a in the source image. Therveightsunr single Iine =case in the source image. The single Iine case falls out as a special case Source Image Destination Imw-ze each line Qi of the multiple Iinecase, For assuming thepiweight never g of the multiple Iinecase, assuming the weight never goes to zero U,V based on Pilin calculate anywhere in the image. The weight aisigned to each anywhere in the image. The weight aisigned to each line should be FigureI 1: Single line ptiir U,V calculate based strongest when the pixel is exactly on thex i line, and on weake strongest when the pixel is exactly on the line, and weaker the further calculate displacement Di In Figure 1, X is the location to sample the source image for the the pixel is from it. The equation wc use is the pixel is from it. The equation wc use is,, rfist = shortest distance fr pixel at X in the destination image. The location is at a distance v b x positions Graphics, U,V Q v but different Computer U,V U,V Q image length b 1- Single line transforms

23 Multiple segments For each point X For each segment pair sbefore[i], safter[i] Transform X into X i Compute weighted average of all transformed X i weight according to distance to segments Morphing length p b weight = a + dist where a, b, p control the influence Transform wrt 2 lines Input images Segments slide by Fredo Duran slide by Fredo Duran

24 Interpolate segments Warp images to segments[t] t=0.5 The red segments are at the same location in both images Image features such as eyes are aligned Warp images to segments[t] Interpolate color The red segments are at the same location in both images Image features such as eyes are aligned

25 Interpolate color Result Recap For each intermediate frame I t Interpolate segment locations y t i= (1- t) x 0 i + t x 1 i Michael Jackson BW Uses the very technique we just studied! Perform two warps: one for I 0, one for I 1 Deduce a dense warp field from the pairs of features Warp the pixels Linearly interpolate the two warped images