Image Metamorphosis B Affine Transformations Tim Mers and Peter Spiegel December 16, 2005 Abstract Among the man was to manipulate an image is a technique known as morphing. Image morphing is a special effect that transforms or morphs one image into another. In this paper we show how this can be done using affine transformations. Introduction Traditionall morphing would involve cross-fading the images. That is, the first image would be displaed and then would begin to fade out while the second image is fading in simultaneousl. The end result would be that of the second image. Since the earl 1990 s, however, this technique has become almost obsolete. Man new techniques for image morphing can produce more realistic looking results than the old cross-fading technique did. Affine transformations is an ecellent eample of one of the new techniques. Page 1 of 32
(a) begin (b) halfwa (c) end Figure 1: Morph b Affine Transformations Page 2 of 32 Objective Using affine transformations, the begin image and end image would be triangulated similarl and each triangle from the begin image would be transformed into the corresponding triangle in the end image through matri multiplication. The color densit would then be mapped to its new value. After the image is successfull warped from the begin image to the end image the steps are reversed
and a weighted average of each frame is taken to get a more realistic result for each transition frame of the image. The result is a smooth transition that can be shown frame b frame. This approach provides an advantage over man other morphing routines in that each frame of the morph produces a realistic looking image. Triangulation The images are divided up b selecting verte points in each image. In our program the verte points are free to be chosen b the user with a simple mouse click. The program assumes the four corners of each image are chosen and the number of verte points can be easil modified to accommodate a large number of triangles. We begin b sending the verte points in the begin image into Matlab s built-in command delauna to obtain a triangulation of the image. We then use the same triangulation on the end image to form similar triangles. When the whole picture is broken up into triangles each triangle can be dealt with individuall. To keep things as simple as possible, we will eplain the process for one triangle. Let us begin b describing a simple warp of a triangle in R 2. Let v 1, v 2, and v 3 be the vertices of a given triangle. We impose the restriction that the verte points be noncollinear. We will refer to this triangle as the begin-triangle (Figure 2a). Let us now establish a second triangle with vertices w 1, w 2, and w 3 which we will call the end-triangle (Figure 2b). We need to map each verte point in the begin-triangle to its new location in the end-triangle, where v 1 w 1,v 2 w 2, and v 3 w 3. This can be done easil using linear transformations. Page 3 of 32
v 2 w 1 w 2 v 1 v w v 3 w 3 (a) begin triangle (b) end triangle Figure 2: Verte Point Displacement However, because we need to obtain several frames between the begin and end images we chose to map the verte points along a straight line path. This method clearl defines the location of each point at each frame. We need to find the location at certain times t between 0 and 1, where 0 is the position in the begin image and 1 is the position in the end image. For each verte point v i w i where {i = 1, 2, 3} the location can be found b u i (t) = (1 t)v i + tw i. Page 4 of 32 Once the verte positions of a given triangle are known, for each individual
frame of the image, we can transform the shape of the triangle using a linear transformation. The product of a 2-b-2 matri and a vector in R 2 can ield ver desirable results for transforming triangles. This 2-b-2 matri we will refer to as the standard matri. If the standard matri is [ ] a b A =, c d then T ([ ]) = [ ] [ ] a b = c d [ ] a + b. c + d There is a number of different effects these linear operations can produce, namel, reflections, rotations, compressions, epansions and shears. When discussing the different tpes of operations we will observe the effect of each on the unit-square (Figure 3). Note that the standard matri multiplies ever vector in the square. Therefore it changes the shape of the square. Reflections and rotations are two of the effects a standard matri can have on a vector (Figure 4). The matri [ ] 1 0 A = 0 1 Page 5 of 32
Figure 3: Unit Square creates a reflection about the -ais (Figure 4a), whereas the matri [ ] cos(θ) sin(θ) A = sin(θ) cos(θ) creates a rotation of θ about the origin (Figure 4b). If the -coordinate of a vector is multiplied b k, where k > 1, we obtain an epansion in the -direction (Figure 5a). If the -coordinate is multiplied, the epansion is in the -direction. In these cases the standard matrices are A = [ ] k 0 0 1 [ ] 1 0 and A =, 0 k respectivel. If 0 < k < 1 the result is a compression on the vectors (Figure 5b). Page 6 of 32
(a) reflection about the -ais (b) rotation b π/6 Figure 4: Reflection and Rotation Page 7 of 32 (a) epansion Figure 5: Epansion and Compression (b) compression
(a) shear in the -direction (b) shear in the -direction Figure 6: Shears A shear in the -direction is defined as a transformation that moves each point (, ) parallel to the -ais b an amount k to the new position (+k, ). This means that the further points get from the -ais, the further the get moved in the -direction (Figure 6a). A shear in the -direction moves each point (, ) parallel to the -ais b an amount k to the new position (, +k) (Figure 6b). The standard matrices for these operations are A = [ 1 0 k 1 ] and A = [ 1 k 0 1 B using the product of multiple standard matrices we can obtain more comple transformations. For eample, to transform the unit square into the ]. Page 8 of 32
shape shown in (Figure 7c) we would use the standard matri [ ] 0 2 A 1 =, 1 2 which is the product of three linear operations. This can be broken down into elementar steps b calculating a shear in the -direction (Figure 7a) with [ ] 1 2 A =, 0 1 an epansion in the -direction (Figure 7b) with [ ] 1 0 A 2 =, 0 2 and a reflection across the line = (Figure 7c) with [ ] 0 1 A 3 =. 1 0 The final result is the same whether we perform three calculations or a single one with the matri A =A 3 A 2 A 1 [ ] [ ] [ ] 0 1 1 0 1 2 A = 1 0 0 2 0 1 [ ] 0 2 A =. 1 2 Page 9 of 32
(a) shear in the -direction (b) epansion in the - direction Figure 7: Combination of Basic Linear Operations (c) reflection about the line = Page 10 of 32
There are a variet of different was the shape of the unit square can be changed through matri multiplication. In the program we calculate triangles instead of squares, but this is not a problem because the vectors we are multipling are indices into each triangle on a point b point basis. Each point is represented b a vector in R 2. This means that an point not bound b the enclosing triangle is ecluded from the calculation. Another thing to notice is that each one of these operations can be reversed. The standard matrices are all invertible, with the eception of the zero matri which shrinks everthing down to a point in the origin, and projection matrices that flatten everthing onto a line. In order to reverse the operation we need to multipl b the inverse of the standard matri that was used and we are back to what we started with (note A 1 A = I ). Affine Transformations To get all the indices of the piels in the end triangle, we need to perform an affine transformation. In other words, multipl all the indices b a standard matri that performs a linear operation on each of the coordinate points in the begin triangle then add a vector b to the new points. Let v represent an coordinate point in our begin triangle and w the corresponding point in the end triangle, then w = Mv + b. Page 11 of 32
v 2 w 1 w 2 v 1 v w v 3 w 3 (a) begin triangle (b) end triangle It can be shown that Figure 8: Verte Point Displacement v v 3 =c 1 (v 1 v 3 ) + c 2 (v 2 v 3 ) (1) v =c 1 v 1 c 1 v 3 + c 2 v 2 c 2 v 3 + v 3 (2) =c 1 v 1 + c 2 v 2 + (1 c 1 c 2 )v 3. (3) Page 12 of 32 Setting the vector v becomes Let c 3 = 1 c 1 c 2, v = c 1 v 1 + c 2 v 2 + c 3 v 3. w = c 1 w 1 + c 2 w 2 + c 3 w 3, (4)
and for w i = Mv i + b (5) i = 1, 2, 3. Substitute (5) into (4) and multipl it out; w =c 1 (Mv 1 + b) + c 2 (Mv 2 + b) + c 3 (Mv 3 + b) =Mc 1 v 1 + c 1 b + Mc 2 v 2 + c 2 b + Mc 3 v 3 + c 3 b. The matri M and vector b can be factored out w = M(c 1 v 1 + c 2 v 2 + c 3 v 3 ) + (c1 + c2 + c3)b. To obtain M and b we set up a sstem of equations, using the verte points that define the triangles we are working with. Let [ ] [ ] vi wi v i =, w i = (6) v i be the i-th verte points of the begin image and end image, respectivel, where i = 1, 2, 3 for the three verte points. Then w i = Mv i + b. w i Page 13 of 32 Let b = [ b b ] [ ] M1 M and M = 2 M 3 M 4
Then we can write equation (6) as [ ] [ ] [ wi M1 M = 2 vi w i M 3 M 4 v i [ ] [[ ] wi M1 = v w i M i + 3 [ ] [ wi M1 v = i + M 2 v i w i M 3 v i + M 4 v i ] + [ M2 M 4 [ b b ] ] ] v i + From this epression it is trivial to set up a sstem of equations where v i and w i are the verte points of the triangles and are known. The matri M with its 4 elements and the two elements of b are the unknowns. w i = M 1 v i + M 2 v i ] + [ b b ]. [ b b ] + b w i = M 3 v i + M 4 v i + b For each one of the verte points we obtain two equations, take three verte points, and ou have si equations with si unknowns. w 1 = M 1 v 1 + M 2 v 1 + b w 1 = M 3 v 1 + M 4 v 1 + b w 2 = M 1 v 2 + M 2 v 2 + b w 2 = M 3 v 2 + M 4 v 2 + b w 3 = M 1 v 3 + M 2 v 3 + b w 3 = M 3 v 3 + M 4 v 3 + b Page 14 of 32
Note that the si variables are lined up above each other. The net step is to put this sstem of equations in matri form v 1 v 1 1 M 1 w 1 v 1 v 1 1 M 2 w 1 v 2 v 2 1 M 3 v 2 v 2 1 M 4 = w 2 w 2 v 3 v 3 1 b w 3 v 3 v 3 1 b w 3 and set up the augmented matri. v 1 v 1 1 w 1 v 1 v 1 1 w 1 A = v 2 v 2 1 w 2 v 2 v 2 1 w 2 v 3 v 3 1 w 3 v 3 v 3 1 w 3 This matri can be row reduced to find the solution for the elements of M and b. Page 15 of 32 An Eample Let s do an eample to make the concept clear. The goal is to transform the triangle with verte points v 1, v 2 and v 3 (Figure 9a), where v 1 = (1, 1), v 2 = (2, 2) and v 3 = (3, 1),
w 2 w 3 w 1 v 2 v 1 v 3 (a) Begin Triangle (b) End Triangle or in vector representation, [ ] 1 v 1 =, v 1 2 = Figure 9: Transforming Triangles [ ] 2 2 and v 3 = [ ] 3. 1 The objective is to transform this triangle into a triangle with verte points w 1, w 2 and w 3 (Figure 9b) where w 1 = (4, 5), w 2 = (6, 9) and w 3 = (7, 8). Page 16 of 32 Put this in vector representation, then [ ] [ ] 4 6 w 1 =, w 5 2 = 9 and w 3 = [ ] 7. 8
Now it is possible to set up the sstem of equations 4 = M 1 (1) + M 2 (1) + b 5 = M 3 (1) + M 4 (1) + b 6 = M 1 (2) + M 2 (2) + b 9 = M 3 (2) + M 3 (2) + b 7 = M 1 (3) + M 2 (1) + b 8 = M 3 (3) + M 4 (1) + b and put it into an augmented matri. 1 1 1 4 1 1 1 5 A = 2 2 1 6 2 2 1 9 3 1 1 7 3 1 1 8 Using linear algebra this matri can be row reduced. 1 3/2 1 1/2 R = 1 3/2 1 5/2 1 2 1 1 Page 17 of 32
v 2 w 2 p v 3 p w 3 w 1 v 1 v 2 v 1 p v 3 (a) this is the triangle we are starting with (b) triangle multiplied b M (c) finall, the vector b is added Figure 10: Warping a Triangle From R we obtain M = [ 3/2 ] 1/2 3/2 5/2 and b = [ ] 2. 1 Now let s see what this does to our triangle v 1 v 2 v 3 and a point p in the triangle (Figure 10a), where [ ] 2.3 p =. 1.2 First let s multipl our triangle and our point p b M and observe the result (Figure 10b). Page 18 of 32
[ ] [ ] [ ] v 1 3/2 1/2 1 2 =Mv 1 = = 3/2 5/2 1 4 [ ] [ ] [ ] v 2 3/2 1/2 2 4 =Mv 2 = = 3/2 5/2 2 8 [ ] [ ] [ ] v 3 3/2 1/2 3 5 =Mv 2 = = 3/2 5/2 1 7 [ ] [ ] [ ] p 3/2 1/2 2.3 4.05 =Mp = = 3/2 5/2 1.2 6.45 As we can see the triangle alread has the shape of our final triangle but it needs to be moved. Recall that some transformations can t be performed onl b matri multiplication. The last step will be to add the vector b (Figure 10c). [ ] [ ] [ ] w 1 =v 1 2 2 4 + b = + = 4 1 5 [ ] [ ] [ ] w 2 =v 2 4 2 6 + b = + = 8 1 9 [ ] [ ] [ ] w 3 =v 3 5 2 7 + b = + = 7 1 [ ] p t =p 4.05 + b = + 6.45 8 [ ] 2 = 1 [ ] 6.05 7.45 Page 19 of 32
Alternativel, we can use homogeneous coordinates to perform our affine transformation through matri multiplication in one step. This means representing a vector in R [ ] v v v 2 as a vector v in R 3. Using this sstem, translation can be epressed with matri multiplication, thereb resulting in an affine 1 transformation. This is easil accomplished once we have obtained the matri M and the vector b b placing them into an augmented matri and adding a row of zeros, with the eception of the last pivot location which we set equal to 1, at the bottom. The standard matri then becomes M 1 M 2 b M 3 M 4 b. 0 0 1 We can then represent the verte points and all the points within the triangle as points in R 3 b simpl adding a 1 in the third row of each vector. This will allow us to assemble an augmented matri with each verte point and each individual point within the enclosing region for matri multiplication. The end result is w 1 w 2 w 3 p... M 1 M 2 b v 1 v 2 v 3 p... w 1 w 2 w 3 p... = M 3 M 4 b v 1 v 2 v 3 p... (7) 1 1 1 1... 0 0 1 1 1 1 1... Page 20 of 32
w 1 w 2 w 3 p M 1 M 2 b v 1 v 2 v 3 p w 1 w 2 w 3 p = M 3 M 4 b v 1 v 2 v 3 p 1 1 1 1 0 0 1 1 1 1 1 3/2 1/2 2 1 2 3 2.3 = 3/2 5/2 1 1 2 1 1.2 0 0 1 1 1 1 1 4 6 7 6.05 = 5 9 8 7.45. 1 1 1 1 After discarding the last row, we have [ ] 4 6 7 6.05. 5 9 8 7.45 Page 21 of 32 This triangle was successfull warped from one shape and position into another. Also all the points of the triangle are warped if the affine transformation is applied to ever one of them. This is the main idea of warping an image, where the ke is that some amount of triangles are laid above the image. The affine transformation maps ever piel in the image to a new shape and therefore
(a) begin image (b) end image Figure 11: Triangulation of a Pair of Images distorts the image. In the previous section we successfull warped one triangle. What can be done with one triangle can be done with man triangles. Figure 11 shows a pair of images with 82 triangles in each image and photographs of two faces. Each photo can be thought of as 82 triangles which make up the entire picture. Note that each triangle in the end image corresponds to one triangle in the begin image and that the triangles enclosing boundar contains specific facial features that match the features of the corresponding triangle. The task at hand is to perform an affine transformation of ever triangle in the begin image to warp it into the shape of the corresponding triangle in the end image (Figure 12). This is done with the method described earlier b finding a unique 2-b-2 matri M and a two-dimensional vector b for each triangle. In the Page 22 of 32
(a) begin image (b) end image Figure 12: Warp of an Image eample (Figure 11) we have 82 triangles and we need 82 affine transformations to accomplish a warp of the image. The result is an image that is in the shape of the end image, but still has the same gra scale values as the begin image. Before we can do a morph we still need to know how to do one more thing. If a series of intermediate frames are captured as the image warps a time-varing warp can be produced. This creates the illusion of a motion picture that graduall changes from the begin image into the end image. The technique behind this is to let the verte points of the triangles continuousl move from their Page 23 of 32
starting point, which is their location in the begin image to their destination, which is their location in the end image. Their motion along a line is a function with respect to time. Let our time units range from t = 0 to t = 1, where t = 0 corresponds to the begin image and t = 1 corresponds to the end image. Suppose u i (t) denotes the position of the i-th verte point at time t. The point travels along the line at a constant velocit. This means that u i (0) = v i (its position in the begin image) and u i (1) = w i (its position in the end image). For an time in between the position is given b u i (t) = (1 t)v i + t(w i ). Equipped with this knowledge it is possible to set up a triangulation for an given time time t and perform a warp from the begin image to this new intermediate position. Figure 13 is the result of computing the warps at five different times. When put together these frames create a motion picture of the face slowl being distorted. If this procedure is applied to the end image as well and if the results are reversed, in other words, we start with the warped image and work towards the original image (Figure 14) we have the basis for a morph. Page 24 of 32 Picture Densit The grascale component of the image is stored in matri form and the concentration of color of a given piel is referred to as picture densit. The densit of the image, which we will define as ρ, must also be mapped for each point. The values range from 0 to 255 and go from white to black for the colormap
t = 0 t = 0.25 t = 0.50 t = 0.75 t = 1 Figure 13: Time-Varing Warp of Begin Image Page 25 of 32 t = 0 t = 0.25 t = 0.50 t = 0.75 t = 1 Figure 14: Reversed Time-Varing Warp of End Image
that we are using. If we are mapping the point v to w and the corresponding gra values are ρ v = 100 and ρ w = 200 respectivel, then we need to obtain a weighted value as the point changes color. The weight depends on the distance that the point has traveled at each step. For eample, if we are constructing eleven frames for the morphed image, then the value of ρ(u) will change b 1/10 for each frame, where ρ t (u) is the densit at time t in frame u. The equation to obtain the value at each step is ρ t (u) = (1 t)ρ 0 + tρ 1 where t is the distance the point has transversed along the path from v to w and u is the location at that distance. This will provide a smooth transition from the color of v to the color of w. The transition from v to w would produce ρ v = 100 110 120... 190 200 = ρ w. Page 26 of 32
t = 0 t = 0.25 t = 0.50 t = 0.75 t = 1 References Figure 15: Morph with 5 frames [1] D. Arnold. For all his help on this project. [2] G. Strang. Introduction to Linear Algebra, Third Ed. [3] H. Anton, C. Rorres. Elementar Linear Algebra with Applications, 8th Ed. [4] S. Johansson. Computing in Matlab Page 27 of 32
begin triangle end triangle Figure 16: Morph of a Triangle with a finite Number of Points Trouble with Discrete Values When writing a Matlab program that morphs two images, we ran into some problems. First, the idea of multipling a triangle with a matri works fine if there are an infinite number of points in the triangle, and if we are multipling each one of them with our matri. Unfortunatel when handling an image we are dealing with discrete values. The picture onl has a finite number of piels that we can multipl with. Imagine a triangle that is being stretched from Page 28 of 32
Page 29 of 32 Figure 17: A Warp that went wrong
its begin shape into a larger end shape (Figure 16). Because there are onl a certain amount of piels in the begin triangle, and the end triangle consists of more piels than the begin triangle, the result of the warp is an image that has gaps in it. The amount of piels that where enough to cover the area of the begin triangle are no longer sufficient to cover the area of the end triangle. Figure 17 is an eample of a warp that went wrong due to this issue. The wa to resolve this issue is to reverse the process of the warp. The conventional wa to warp the image would be to take the inde of each piel, multipl it b the standard matri M and add the vector b (Equation 8) and the piel with the resulting inde must have the same gra scale value as the one we started with. If we start with the warped shape of the triangle and go reverse to retrieve the picture densit of ever piel in the end triangle we will have a value for ever one of those piels and there will be no missing piels. We know that w = Mv + b is the equation for warping an object. It is eas to reverse the process. w = Mv + b (8) w b = Mv (9) M 1 (w b) = v (10) The first step will be to subtract the vector b from all of our points (Equation 9), then we need to multipl the result b the inverse of the standard matri M Page 30 of 32
Figure 18: Overlapping Triangles (Equation 10) and now we have an equation where we can plug in w to obtain it s corresponding gra scale value, which is equal to the gra scale value at v. Page 31 of 32 Overlapping Triangles Another issue with warps are overlapping triangles. If a verte point in the end triangulation is moved across the line of an other triangle (Figure 18) we are going to run into troubles. The problem is that now some points are defined to belong to more than one triangle. Since the standard matri and the vector b are different for ever triangle it is not clear which matri and which vector to appl.
The onl wa we could resolve this issue was b carefull choosing our verte points to make sure the triangles don t overlap. Eperiment on Your Own If ou want to eperiment with the program the instructions are as follows: 1. Go to the url provided below 2. Download and save the zip to a local director in the MatLab path. 3. Etract all the files. 4. Open MatLab and set the current director to the folder with the etracted files. 5. Go to MatLab s command line prompt and tpe metamorphosis. 6. When asked to choose a begin image tpe in the name as shown on screen. 7. Choices should look like choice with quotes. The MatLab files that generate the morph can be found at http://online.redwoods.edu/instruct/darnold/laproj/fall2005/ petertim/archive.zip Page 32 of 32