Feature Article Polymorph: Morphig Amog Multiple Images Image metamorphosis has prove to be a powerful visual effects tool. May breathtakig examples ow appear i film ad televisio, depictig the fluid trasformatio of oe digital image ito aother. This process, commoly kow as morphig, couples image warpig with color iterpolatio. Image warpig applies 2D geometric trasformatios o images to alig their features geometrically, while color iterpolatio bleds their colors. Details Polymorph exteds of various image morphig techiques ca be foud i several covetioal morphig to recet papers. -4 derive morphed images Traditioal image morphig cosiders oly two iput images at a from more tha two images time the source ad target images. I that case, morphig amog multiple images ivolves a series of at oce, effectively trasformatios from oe image to itegratig geometric aother. This limits ay morphed image to the features ad colors maipulatios ad color bleded from just two iput images. Give morphig s success usig this bledig. paradigm, it seems reasoable to cosider the beefits possible from a bled of more tha two images at a time. For istace, cosider geeratig a facial image with bleded characteristics of eyes, ose, ad mouth from several iput faces. I this case, morphig amog multiple images ivolves a bled of several images at oce a process we call polymorphig. Rowlad ad Perrett cosidered a special case of polymorphig to obtai a prototype face from several tes of sample faces. 5 They superimposed feature poits o iput images to specify the differet positios of features i sample faces. Averagig the specified feature positios determied the shape of a prototype face. A prototype face resulted from image warpig each iput Seugyog Lee Postech, Korea George Wolberg City College of New York Sug Yog Shi KAIST, Korea image ad the performig a cross-dissolve operatio amog the warped images. I performig predictive geder ad age trasformatios, they used the shape ad color differeces betwee prototypes from differet geders ad ages to maipulate a facial image. I this article, we preset a geeral framework for polymorphig by extedig the traditioal image morphig paradigm that applies to two images. We formulate each iput image as a vertex of a ( )-dimesioal simplex, where equals the umber of iput images. Note that a ( )-dimesioal simplex is a covex polyhedro havig vertices i ( )-dimesioal space, such as a triagle i 2D or a tetrahedro i 3D. A arbitrary ibetwee (morphed) image ca be specified by a poit i the simplex. The barycetric coordiates of that poit determie the weights used to bled the iput images ito the i-betwee image. Whe cosiderig oly two images, the simplex degeerates ito a lie. Poits alog the lie correspod to i-betwee images i a morph sequece. This case is idetical to covetioal image morphig. Whe cosiderig more tha two images, a path lyig aywhere i the simplex costitutes the ibetwee images i a morph sequece. I morphig betwee two images, ouiform bledig was itroduced to derive a i-betwee image i which bledig rates differ across the image. 3,4 This lets us geerate more iterestig aimatios, such as a trasformatio of the source image to the target from top to bottom. Nouiform bledig was also cosidered i volume metamorphosis to cotrol bledig schedules. 6,7 I this article, the framework for polymorphig icludes ouiform bledig of features i several iput images. For istace, a facial image ca be geerated to have its eyes, ose, mouth, ad ears derived from four differet iput faces. Polymorph is ideally suited for image compositio applicatios. It treats a composite image as a metamorphosis of selected regios i several iput images. The 58 Jauary/February 998 0272-76/98/$0.00 998 IEEE
regios seamlessly bled together with respect to geometry ad color. The techique produces high-quality composites with cosiderably less effort tha covetioal image compositio techiques. I this regard, polymorphig brigs to image compositio what image warpig has brought to cross-dissolve i derivig morphig: a richer, more sophisticated class of visual effects achieved with ituitive ad miimal user iteractio. First we ll look at the mathematical framework for polymorph, followed by warp fuctio geeratio ad propagatio, bledig fuctio geeratio, ad the implemeted polymorph system. Metamorphosis examples demostrate the use of polymorph for image compositio. Mathematical framework This sectio presets the mathematical framework for polymorph. We exted the metamorphosis framework for two images 3,4 to geerate a i-betwee image from several images. The framework is further optimized by itroducig the otio of a cetral image. Fially, we itroduce preprocessig ad postprocessig steps to ehace the usefuless of the polymorphig techique. Image represetatio Cosider iput images I, I 2,, I. We formulate each iput image to be a vertex of a ( )-dimesioal simplex. A i-betwee image is cosidered a poit i the simplex. All poits are give i barycetric coordiates i R by b = (b, b 2,, b ), subject to the costraits b i 0 ad Σ i = b i=. Each iput image I i correspods to the ith vertex of the simplex, where oly the ith barycetric coordiate is ad all the others are 0. A i-betwee image I is specified by a poit b, where each coordiate b i determies the relative ifluece of iput image I i o I. I covetioal morphig betwee two images, trasitio rates 0 ad imply the source ad target images, respectively. 3,4 A i-betwee image is the represeted by a real umber betwee 0 ad, which determies a poit i 2D barycetric coordiates. The image represetatio i polymorph ca be cosidered a geeralizatio of that used for morphig betwee two images. I the covetioal approach, morphig amog iput images implies a sequece of aimatios betwee two images, for example, I 0 I I. The aimatio sequece correspods to a path visitig all vertices alog the edges of the simplex. I cotrast, polymorphig ca geerate a aimatio correspodig to a arbitrary path iside the simplex. The aimatio cotais a sequece of i-betwee images that bleds all iput images at a time. I the followig, we cosider the procedure to geerate the i-betwee image associated with a poit alog the path. The procedure ca be readily applied to all other poits alog the path to geerate a aimatio. W 0 W W 0 Basic metamorphosis framework Suppose that we wat to geerate a i-betwee image I at poit b = (b, b 2,, b ) from iput images I, I 2,, I. Let W ij be the warp fuctio from image I i to image I j. W ij specifies the correspodig poit i I j for each poit i I i. Whe applied to I i, W ij geerates a warped image whereby the features i I i coicide with their correspodig features i I j. Note that W ii is the idetity warp fuctio, ad W ji is the iverse fuctio of W ij. To geerate a i-betwee image I, we first derive a warp fuctio W i by liearly iterpolatig W ij for each i. Each image I i is the distorted by W i to geerate a itermediate image I i. Images I i have the same i-betwee positios ad shapes of correspodig features for all i. I-betwee image I is fially obtaied by liearly iterpolatig the pixel colors amog I i. Each coordiate b i of I is used as the relative weight for I i i the liear iterpolatio of warps ad colors. We call b a bledig vector. It determies the bledig of geometry ad color amog the iput images to geerate a i-betwee image. For simplicity, we treat the bledig vectors for both geometry ad color as idetical, although they may differ i practice. Figure shows the warp fuctios used to geerate a i-betwee image from three iput images. Each warp fuctio i the figure distorts oe image toward the other so that the correspodig features coicide i their shapes ad positios. Note that warp fuctio W ij is idepedet of the specified bledig vector b, while W i is determied by b ad W ij. Sice o geometric distortios exist betwee itermediate image I i ad the fial i-betwee image I, it is sufficiet for Figure to depict warps directly from I i to I, omittig ay referece to Īi. I this maer, the figure cosiders oly the warp fuctios ad eglects color bledig. Give images I i ad warp fuctios W ij, the followig equatios summarize the steps for geeratig a ibetwee image I from a bledig vector b. W i I i deotes the applicatio of warp fuctio W i to image I i. p ad r represet poits i I i ad I, respectively, related by r= W i(p). Color iterpolatio is achieved by atteuatig the pixel colors of the iput images ad addig the warped images. i ( ) = j ij( ) j= W p b W p ( ) = ( ) ( ) I r W p b I p i i i i ( ) = i ( ) i= I r I r I 0 I W 2 I W 0 W 02 W 2 W 20 W 2 I 2 Warp fuctios for three iput images. IEEE Computer Graphics ad Applicatios 59
Feature Article W 0 [b] I 0 I 0 2 A crossdissolve ad uiform ibetwee image from three iput images. W [b] Cross-dissolve I I I W 2 [b] I 2 I 2 The image o the left i Figure 2 results from ordiary cross-dissolve of iput images I i. Notice that the image appears triple-exposed due to the bledig of misaliged features. The images o the right of I i illustrate the process to geerate a i-betwee image I usig the proposed framework. Although the itesities of itermediate images I i should appear atteuated, we show the images i full itesity to clearly demostrate the distortios. The bledig vector used for the figure is b = (/3, /3, /3). The resultig image I equally bleds the shapes, positios, ad colors of the eyes, ose, ad mouth of the iput faces. Geeral metamorphosis framework The framework preseted above geerates a uiform i-betwee image o which we use the same bledig vector across the image. We ca obtai a more visually compellig i-betwee image by applyig differet bledig vectors to its various parts. For example, cosider a facial image that has its eyes, ears, ose, ad mouth derived from four differet iput images. We itroduce a bledig fuctio to facilitate a ouiform i-betwee image that has differet bledig vectors over its poits. A bledig fuctio specifies a bledig vector for each poit i a image. Let B i be a bledig fuctio defied o image I i. For each poit p i I i, B i(p) is a bledig vector that determies the weights used for liearly iterpolatig W ij(p) to derive W i (p). Also, the ith coordiate of B i(p) determies the color cotributio of poit p to the correspodig poit i i-betwee image I. The metamorphosis characteristics of a ouiform i-betwee image are fully specified by oe bledig fuctio B i defied o ay iput image I i. This is aalogous to usig oe bledig vector to specify a uiform i-betwee image. From the correspodece betwee poits i iput images, the bledig iformatio specified by B i ca be shared amog all iput images. The bledig fuctios B j for the other images I j ca be derived by composig B i ad warp fuctios W ji. That is, B j= B i W ji, or equivaletly, B j(p)= B i(w ji (p)). For all correspodig poits i the iput images, the resultig bledig fuctios specify the same bledig vector. Give images I i, warp fuctios W ij, ad bledig fuctios B i, the followig equatios summarize the steps for geeratig a ouiform i-betwee image I. b j i(p) deotes the jth coordiate i bledig vector B i(p). Wi ( p) = b j i ( p) Wij( p) j= i Ii ( r) = Wi ( p) bi ( p) Ii ( p) ( ) = i ( ) i= I r I r Figure 3 illustrates the above framework. We have chose bledig fuctios B i that make i-betwee image I retai the hair, eyes ad ose, ad mouth ad chi from iput images I 0, I, ad I 2, respectively. The B i 60 Jauary/February 998
determie warp fuctios W i, which geerate distortios of I i whereby the parts of iterest remai itact. Here agai itermediate images I i appear i full itesity for clarity. I practice, iput images I i are ouiformly atteuated by applyig B i before they are distorted. The atteuatio maitais those parts to be retaied i I at their full itesity. The strict requiremet to retai specified features of the iput images has produced a uatural result aroud the mouth ad chi i Figure 3. Additioal processig might be ecessary to address the artifacts iheret i the curret process. First, we will cosider how to reduce rutime computatio ad memory overhead i geeratig a i-betwee image. Optimizatio with a cetral image The evaluatio of warp fuctios W ij is geerally expesive. To reduce rutime computatio, we compute W ij oly oce, whe feature correspodeces are established amog iput images I i. We sample each W ij over all source pixels ad store the resultig target coordiates i a array. The arrays for all W ij are the used to compute itermediate warp fuctios W i, which deped o bledig fuctios B i. For large, these 2 warp fuctios require sigificat memory overhead, especially for large iput images. To avoid this memory overhead, we defie a cetral image ad use it to reduce the umber of stored warp fuctios. A cetral image I C is a uiform i-betwee image correspodig to the cetroid of the simplex that cosists of iput images. The bledig vector (/, /,, /) is associated with this image. Istead of keepig 2 warp fuctios W ij, we maitai 2 warp fuctios, W ic ad W Ci, ad compute W i from a bledig fuctio specified o I C. W ic is a warp fuctio from iput image I i to cetral image I C. Coversely, W Ci is a warp fuctio from I C to I i. W Ci is the iverse fuctio of W ic. Let a bledig fuctio B C be defied o cetral image I C. B C determies the metamorphosis characteristics of a i-betwee image I. B C has the same role as B i except that it operates o I C istead of I i. Therefore, B C(q) gives a bledig vector that specifies the relative iflueces of correspodig poits i I i oto I for each poit q i I C. The equivalet bledig fuctios B i for iput images I i ca be derived by fuctio compositios B C W ic. To obtai warp fuctios W i from I i to I, we first geerate a warp fuctio W C from I C to I. For each poit i I C, warp fuctios W Ci give a set of correspodig poits i iput images I i. Hece, W C ca be derived by liearly I 0 I 0 I I I I 2 I 2 I W 0 [B 0 ] W [B ] W 2 [B 2 ] W C W C I 0 W 0C W C0 W C I C W C2 W 2C iterpolatig W Ci with the weights of B C. The W i are the obtaied by fuctio compositios W C W ic. Figure 4 shows the relatioship of warp fuctios W ic, W Ci, ad W C with images I i, I C, ad I. Note that W ic ad W Ci are idepedet of B C, whereas W C is determied by B C from W Ci. Give images I i ad warp fuctios W ic ad W Ci, the followig equatios summarize the steps for geeratig a i-betwee image I from a bledig fuctio B C defied o cetral image I C. Let p, q, ad r represet poits i I i, I C, ad I, respectively. They are related by q = W ic(p) ad r= W C(q). bi(p) j ad bc(q) j deote the jth coordiates i bledig vectors B i(p) ad B C(q), respectively. I I 2 3 A ouiform ibetwee image from three iput images. 4 Warp fuctios with a cetral image. IEEE Computer Graphics ad Applicatios 6
Feature Article 5 Warp fuctios with preprocessig ad postprocessig. C ( ) = i C ( ) Ci ( ) i= W q b q W q i ( ) = ( C ic )( ) = C ic ( ) ( ) = ( )( ) = ( ) W p W o W p W W p B p B o W p B W p i C ic C ic i Ii ( r) = Wi ( p) bi ( p) Ii ( p) ( ) = i ( ) i= I r I r ( ) ( ) P 0 I 0 I 0 W 0C W C0 W C I Note that the ad operators deote forward ad iverse mappig, respectively. For a image I ad warp fuctio W, W I maps all pixels i I oto the distorted image I, while I W maps all pixels i I oto I, assumig that the iverse fuctio W exists. Although the operator could have bee applied to compute I i above, Q I we use the operator because W i is ot readily available. I Figure 4, cetral image I C has dashed borders because it is ot actually costructed i the process of geeratig a i-betwee image. We itroduced it to provide a coceptual itermediate step to derive the ecessary warp ad bledig fuctios. Ay image, icludig a iput image, ca be made to play the role of the cetral image. However, we have defied the cetral image to lie at the cetroid of the simplex to establish symmetry i the metamorphosis framework. I most cases, a cetral image relates to a atural ibetwee image amog the iput images. It equally bleds the features i the iput images, such as a prototype face amog iput faces. W C W C I C W C2 W 2C P P 2 I I I 2 I 2 Preprocessig ad postprocessig Polymorphig proves useful i feature-based image compositio, where selected features from iput P 0 W 0 [B C ] 6 I-betwee image geeratio with preprocessig ad postprocessig. I 0 I 0 I 0 P W [B C ] Q I I I I I P 2 W 2 [B C ] I 2 I 2 I 2 62 Jauary/February 998
images bled seamlessly i a ibetwee image. I that case, if the shapes ad positios of the selected features do ot match amog the iput images, the i-betwee image might ot have features i appropriate shapes ad positios. For example, the i-betwee face i Figure 3 retais hair, eyes ad ose, ad mouth ad chi features from the iput faces, yet it appears uatural. This results from the rigid placemet of selected iput regios ito a patchwork of iappropriately scaled ad positioed elemets aki to a cut-ad-paste operatio where smooth bledig was limited to areas betwee these regios. We ca overcome this problem by addig a preprocessig step to the metamorphosis framework. Before usig the framework o iput images I i, we ca apply warp fuctios P i to I i to geerate distorted iput images I i. The distortios chage the shapes ad positios of the selected features i I i so that a i-betwee image from I i has appropriate feature shapes ad positios. I that case, we apply the framework to I i istead of I i, treatig I i as the iput images. After derivig a i-betwee image through the framework, we sometimes eed image postprocessig to ehace the result, eve though the preprocessig step has already bee applied. For example, we might wat to reduce the size of the i-betwee face i Figure 3 ot readily doe by preprocessig iput images. To postprocess a i-betwee image I, we apply a warp fuctio Q to I ad geerate the fial image I. The postprocessig step is useful for local refiemet ad global maipulatio of a i-betwee image. Figure 5 shows the warp fuctios used i the metamorphosis framework, icludig the preprocessig ad postprocessig steps. Warp fuctios P i distort iput images I i toward I i, from which a i-betwee image I is geerated. Applyig warp fuctio Q to I derives the fial image I. Figure 6 illustrates the process with itermediate images. I preprocessig, the hair of iput image I 0 ad the mouth ad chi of I 2 move upwards ad to the lower right, respectively. The ose i I arrows slightly. The face i i-betwee image I ow appears more atural tha that i Figure 3. I postprocessig, the face i I is scaled dow horizotally to geerate the fial image I. Whe addig the preprocessig step to the metamorphosis framework, distorted iput images I i determie the cetral image I C ad warp fuctios W ic ad W Ci. However, i that case, wheever we apply differet warp fuctios P i to iput images I i, we must recompute W ic ad W Ci to apply the framework to I i. This ca become very cumbersome, especially sice several preprocessig iteratios might be ecessary to derive a satisfactory i-betwee image. To overcome this drawback, we recofigure Figure 5 to Figure 7 so that I C, W ic, ad W Ci deped oly o I i, regardless of P i. I the ew cofiguratio, we ca derive the correspodig poits i the I i for each poit i I C by fuctio compositios P i W Ci. Give a bledig fuctio B C defied o I C, warp fuctio W C is computed by liearly iterpolatig P i W Ci with the weights of B C. The resultig W C is equivalet to W C used i Figure 5. The, the warp fuctios from I i to I ca be obtaied by W C W ic, properly reflectig the effects of preprocessig. Warp fuctios W i from I i to the fial i-betwee image I, icludig the postprocessig step, ca be derived by Q W C W ic. Cosider iput images I i, warp fuctios W ic ad W Ci, ad a bledig fuctio B C defied o cetral image I C. The followig equatios summarize the process to geerate a i-betwee image I from B C ad warp fuctios P i ad Q, which specify the preprocessig ad postprocessig steps. Let p, q, ad r represet poits i I i, I C, ad I, respectively. They are related by q = W ic(p) ad r=(q W C)(q). C i i ( ) = C ( )( i Ci )( ) = C ( ) i Ci ( ) i= i= W q b q P ow q b q P W q Wi p Q o WC o W p Q ic WC WiC p B p B o W p B W p i C ic C ic i Ii ( r) = Wi ( p bi p Ii p ( ) ( ) ( ) ( ) = ( )( ) = ( ) ( ) = ( )( ) = ( ) ) ( ) ( ) ( ) = i ( ) I r I r i= P 0 I C W C2 P P 2 I I 2 I 2 I W C W C W 0C I 0 I 0 W C0 W C W 2C The differeces betwee this framework ad that i the sectio Optimizatio with a cetral image lie oly i the computatio of warp fuctios W C ad W i. We icluded additioal fuctio compositios to icorporate the preprocessig ad postprocessig effects. I Figure 7, images I i ad I appear with dashed borders because they are ot actually costructed i geeratig I. As i the 7 Warp fuctios with optimal preprocessig ad postprocessig. I Q I IEEE Computer Graphics ad Applicatios 63
Feature Article W 0 [B C ] I 0 I 0 8 I-betwee image geeratio with optimal preprocessig ad postprocessig. W [B C ] I I I W 2 [B C ] I 2 I 2 case of cetral image I C, images I i ad I provide coceptual itermediate steps to derive W C ad W i. Figure 8 shows a example of i-betwee image geeratio with the framework. Itermediate images I i are the same as the distorted images geerated if we apply warp fuctio Q to images _ I i i Figure 6. I other words, I i reflects the preprocessig ad postprocessig effects i additio to the distortios defied by bledig fuctio B C. Hece, oly three itermediate images are eeded to obtai the fial i-betwee image I, as i Figure 3, rather tha seve as i Figure 6. Notice that I is the same as i Figure 6. Give iput images, geeratig a i-betwee image through the framework requires a solutio to each of the followig three problems: how to fid 2 warp fuctios W ic ad W Ci, how to specify a bledig fuctio B C o I C, ad how to specify warp fuctios P i ad Q for preprocessig ad postprocessig. Warp fuctio geeratio This sectio addresses the problem of derivig warp fuctios W ic ad W Ci, ad P i ad Q. A covetioal image morphig techique computes warp fuctios for ( ) pairs of iput images. We propagate these warp fuctios to obtai W ij for all pairs of iput images. Averagig W ij for each i produces W ic. We compute W Ci as the iverse fuctio of W ic by usig a warp geeratio algorithm. P i ad Q are derived from the user iput specified by primitives such as lie segmets overlaid oto images. Warp fuctios betwee two images We ca derive the warp fuctios betwee two iput images usig a covetioal image morphig techique. Traditioally, image morphig betwee two images begis with establishig their correspodece with pairs of feature primitives such as mesh odes, lie segmets, curves, or poits. Each primitive specifies a image feature, or ladmark. A algorithm the computes a warp fuctio that iterpolates the feature correspodece across the iput images. The several image morphig algorithms i commo use differ i the maer i which features are specified ad warp fuctios are geerated. I mesh warpig, bicubic splie iterpolatio computes a warp fuctio from the correspodece of mesh poits. I field morphig, 2 pairs of lie segmets specify feature correspodeces, ad weighted averagig determies a warp fuctio. More recetly, thi-plate splies 4,8 ad multilevel free-form deformatios 3 have bee used to compute warp fuctios from selected poit-to-poit correspodeces. I this article, we use the multilevel free-form deformatio algorithm 3 to geerate warp fuctios betwee two iput images. We selected this algorithm because it efficietly geerates C 2 -cotiuous ad oe-to-oe 64 Jauary/February 998
warp fuctios. The oe-to-oe property guaratees that the distorted image does ot fold back upo itself. A warp fuctio represets a mappig betwee all poits i two images. I this work, we save the warp fuctios i biary files to reduce rutime computatio. A warp fuctio for a M N image is stored as a M N array of target x- ad y-coordiates for all source pixels. Oce feature correspodece betwee two images I i ad I j is established, we ca derive warp fuctios i both directios, W ij ad W ji. Therefore, we store two M N arrays of target coordiates for each pair of iput images for which feature correspodece is established. Figure 9 depicts the warp geeratio process. I Figures 9a ad 9b, the specified features are overlaid o iput images I 0 ad I. Figures 9c ad 9d illustrate warp fuctios W 0 ad W 0 geerated from the features by the multilevel free-form deformatio. (a) (b) 9 Specified features ad the resultig warp fuctios. Warp fuctio propagatio We ca derive the warp fuctios betwee two images by specifyig their feature correspodece. Multiple images, however, require determiig warp fuctios for each image to every other image. This exacerbates the already tedious ad cumbersome operatio of specifyig feature correspodece. For istace, iput images require establishig feature correspodece amog ( )/2 pairs of images. We ow address the problem of miimizig this feature specificatio effort. Cosider a directed graph G with vertices. Each vertex v i correspods to iput image I i, ad a edge e ij coects v i to v j if warp fuctio W ij from I i to I j has bee derived. To miimize the feature specificatio effort, we first select ( ) pairs of images so that the associated edges costitute a coected graph G. We specify the feature correspodece betwee the selected image pairs to obtai warp fuctios betwee them. These warp fuctios ca the be propagated to derive the remaiig warp fuctios for all other image pairs. The propagatio occurs i the same maer as that used for computig the trasitive closure of graph G. The coectivity of G guaratees that warp fuctios are determied for all pairs of images after propagatio. To determie a ukow warp fuctio W ij, we traverse G to fid ay vertex v k shared by existig edges e ik ad e kj. If we ca fid such a vertex v k, we update G to iclude edge e ij ad defie W ij as the composite fuctio W kj W ik. Whe there exist several such v k, the composed warp fuctios through those v k are computed ad averaged. If o v k coects v i ad v j, W ij remais ukow ad e ij is ot added to G. This procedure iterates for all ukow warp fuctios, ad the iteratio repeats util all warp fuctios are determied. If W ij remais ukow i a iteratio, it will be determied i a followig iteratio as G gets updated. From the coectivity of G, at most ( 2) iteratios are required to resolve every ukow warp fuctio. With the warp propagatio approach, the user must specify feature correspodeces for ( ) pairs of images. This is far less effort tha cosiderig all ( )/2 pairs. Figure 0 shows a example of warp propagatio. Figure 0a illustrates warp fuctio W 02, which was derived by specifyig feature correspodece (c) (a) (d) (b) betwee iput images I 0 ad I 2. To determie warp fuctio W 2 from I to I 2, we compose W 0 ad W 02, show i Figure 9d ad Figure 0a, respectively. Figure 0b illustrates the resultig W 2 = W 02 W 0. Notice that the left ad right sides of the hair have wideed i W 2 ad arrowed i W 02. Warp fuctios for cetral image We ow cosider how to derive the warp fuctios amog the cetral image I C ad all iput images I i. I C is the uiform i-betwee image correspodig to a bledig vector (/, /,, /). Hece, from the basic metamorphosis framework, warp fuctios W ic from I i to I C are straightforward to compute. That is, W ic(p)=σ j= W i j(p)/ for each poit p i I i. Computig warp fuctio W Ci from I C back to I i, however, is ot straightforward. We determie W Ci as the iverse fuctio of W ic. Each poit p i I i is mapped to a poit q i I C by W ic. The iverse of W ic should map each q back to p. Hece, the correspodig poits q for pixels p i I i provide W Ci with scattered positioal costraits, W Ci(q) = p. The multilevel free-form deformatio techique, used to derive warp fuctios, ca also be applied to the costraits to compute W Ci. Whe warp fuctios W ij are oe-to-oe, it is ot mathematically clear that their average fuctio W ic is also oe-to-oe. Coceptually, though, we ca expect W ic to be oe-to-oe because it is the warp fuctio that might be geerated by movig the features i I i to the 0 Warp propagatio example: (a) warp W 02 ad (b) warp W 2 = W 02 W0. IEEE Computer Graphics ad Applicatios 65
Feature Article Warp fuctios betwee I 0 ad I C: (a) warp W 0C ad (b) warp W C0. 2 Specified primitives for preprocessig: (a) origial positios ad (b) ew positios. (a) (a) (b) (b) Figure 6. The primitive has bee moved to the lower right to chage the positios of the mouth ad chi. We ca derive warp fuctio Q for postprocessig i the same maer as P i. I this case, primitives overlaid o i-betwee image I i Figure 7 select parts to distort toward the fial i-betwee image I. The overlaid primitives are moved to defie Q, ad multilevel free-form deformatio computes Q from the movemets. We costruct I oly to allow for the specificatio of features ad their movemets, ot for the process of i-betwee image geeratio. The postprocessig operatio illustrated i Figure 3 horizotally scales dow the primitive specified o I to make the face arrower. Bledig fuctio geeratio This sectio addresses the problem of geeratig a bledig fuctio B C defied o the cetral image I C. A bledig vector suffices to determie a B C that geerates a uiform i-betwee image. A ouiform ibetwee image, however, ca be specified by assigig differet bledig rates to selected parts of various iput images. We derive the correspodig B C by gatherig all the bledig rates oto I C ad applyig scattered data iterpolatio to them. 3 Specified primitives for postprocessig: (a) origial positios ad (b) ew positios. (a) (b) Uiform i-betwee image To geerate a uiform i-betwee image from iput images, a user must specify a bledig vector b = (b, b 2,, b ), subject to the costraits b i 0 ad Σ i=0 b i=. If these costraits are violated, we eforce them by clippig egative values to zero ad dividig each b i by Σ i=0 b i. The bledig fuctio B C is the a costat fuctio havig the resultig bledig vector as its value at every poit i I C. averaged positios. I the worst case that W ic is ot oeto-oe, the positioal costraits, W Ci(q) = p, may have two differet positios for a poit q. I that case, we igore oe of the two positios ad apply the multilevel free-form deformatio to the remaiig costraits. Figure shows examples of warp fuctios betwee iput images ad a ceter image. Figure a illustrates warp fuctio W 0C from I 0 to I C, which is the average of the idetity fuctio, W 0 i Figure 9c, ad W 02 i Figure 0a. I Figure b, W C0 has bee derived as the iverse fuctio of W 0C by the multilevel free-form deformatio techique. Warp fuctios for preprocessig ad postprocessig Warp fuctios P i for preprocessig are derived i a similar maer to those betwee two images we overlay primitives such as lie segmets ad curves oto image I i. However, i this case, the primitives select the parts i I i to distort, istead of specifyig feature correspodece with aother image. P i is defied by movig the primitives to the desired distorted positios of the selected parts ad computed by applyig the multilevel free-form deformatio techique to the displacemets. Figure 2 shows the primitive ad its movemet specified o iput image I 2 to defie warp fuctio P 2 i Nouiform i-betwee image To geerate a ouiform i-betwee image I, a user assigs a real value b i [0,] to a selected regio R i of iput image I i for some i. The value b i assiged to R i determies the ifluece of I i oto the correspodig part R of i-betwee image I. Whe b i approaches, the colors ad shapes of the features i R i domiate those i R. Coversely, whe b i approaches 0, the ifluece of R i o R dimiishes. Figures 4a, 4b, ad 4c show the polygos used to select regios i I 0, I, ad I 2 for geeratig I i Figure 3. All poits iside the polygos i Figures 4a, 4b, ad 4c have bee assiged the value.0. We geerate a bledig fuctio B C by first projectig the values b i oto I C. We ca do this by mappig poits i R i oto I C usig warp fuctios W ic. Figure 4d shows the projectio of the selected parts i Figures 4a, 4b, ad 4c oto I C. Let (b, b 2,, b ) be a -tuple represetig the projected values of b i oto a poit i I C. This -tuple is defied oly i the projectio of R i o I C. Sice the user does ot have to specify b i for all I i, some of the b i may be udefied for the -tuple. Let D ad U deote the sets of defied ad udefied elemets b i i the -tuple, respectively. Further, let s be the sum of the defied values i D. There are three cases to cosider: s >, s ad U is empty, ad s ad U is ot empty. If s >, the we assig zero to the udefied values ad scale dow the elemets i D to satisfy s =. 66 Jauary/February 998
If s ad U is empty, the we scale up the elemets i D to satisfy s =. Otherwise, we assig ( s)/k to each elemet i U, where k is the umber of elemets i U. Normalizig the -tuples of the projected values lets us obtai bledig vectors for the poits i I C that correspod to the selected parts R i i I i. These bledig vectors ca the be propagated to all poits i I C by scattered data iterpolatio. We costruct a iterpolatig surface through the ith coordiates of these vectors to determie b i of a bledig vector b at all poits i I C. For the scattered data iterpolatio, we use multilevel B-splies, 9 a fast techique for geeratig a C 2 - cotiuous surface. After costructig surfaces, we have a -tuple at each poit i I C. Sice each surface is geerated idepedetly, the sum of the coordiates i the -tuple does ot ecessarily equal oe. I that case, we scale them to force their sum to oe. The resultig -tuples at all poits i I C defie a bledig fuctio B C that satisfies the user-specified bledig costraits. Figure 5 illustrates the costructed surfaces to determie B C used i Figure 3. The heights of the surfaces i Figures 5a, 5b, ad 5c represet b 0, b, ad b 2 of bledig vector b at the poits i I C, respectively. I Figure 4b, b is.0 at the poits correspodig to the eyes ad ose i I C, satisfyig the requiremet specified i Figure 4b. They are 0.0 aroud the hair, mouth, ad chi due to the value.0 assiged to those parts i Figures 4a ad 4c. Figures 5a ad 5c also reflect the requiremets for b 0 ad b 2 specified i Figure 4. Implemetatio This sectio describes the implemetatio of the polymorph framework. We also preset the implemeted system s performace. (a) (c) (a) (b) (b) (d) 5 Costructed surfaces for a bledig fuctio. 4 Selected parts o iput images ad their projectios oto the cetral image. Polymorph system The polymorph system cosists of three modules. The first module is a image morphig system that cosiders two iput images at a time. It requires the user to establish feature correspodece amog two iput images ad geerates warp fuctios betwee them. We adopted the morph system preseted by Lee at al. 3 because it facilitates flexible poit-to-poit correspodeces ad produces oe-to-oe warp fuctios that avoid udesirable foldovers. The first module is applied to ( ) pairs of iput images, which correspod to the edges selected to costitute a coected graph G. The geerated warp fuctios are sampled at each pixel ad stored i biary files. The secod module is the warp propagatio system. It first reads the biary files of the warp fuctios associated with the selected edges i graph G. Those warp fuctios are the propagated to derive ( ) warp fuctios W ij amog all iput images I i. Fially, the secod module computes 2 warp fuctios i both directios betwee the cetral image I C ad all I i. The resultig warp fuctios, W ic ad W Ci, are stored i biary files ad used as the iput of the third module, together with all I i. The third module is the i-betwee image geeratio system. It lets the user cotrol the bledig characteristics ad the preprocessig ad postprocessig effects i (c) a i-betwee image. To determie the bledig characteristics of a uiform i-betwee image, the user must provide a bledig vector. For a ouiform i-betwee image, the user selects regios i I i ad assigs them bledig values. If preprocessig ad postprocessig are desired, the user must specify primitives o images I i ad I, respectively, ad move them to ew positios. Oce the user iput is give, the third module first computes bledig fuctio B C ad warp fuctios P i ad Q. The module the geerates a i-betwee image by applyig the polymorph framework to I i, W ic, ad W Ci. The polymorph system modules are idepedet of each other ad commuicate by way of biary files storig warp fuctios. Ay image morphig system ca serve as the first module if it ca save a derived warp fuctio to a biary file. The secod module does ot eed iput images ad maipulates oly the biary files passed from the first module. The first ad secod modules together serve to compute warp fuctios W ic ad W Ci. Give iput images, these modules ru oly oce, IEEE Computer Graphics ad Applicatios 67
Feature Article 6 Iput images (top row) ad combiatios of the hair, eyes ad ose, ad mouth ad chi regios (middle ad bottom rows). 7 Effects of varyig bledig values with the same selected regios. ad the derived W ic ad W Ci pass to the third module. The user the rus the third module repeatedly to geerate several i-betwee images by specifyig differet B C, P i, ad Q. Performace We ow discuss the performace of the polymorph system i terms of the examples already show. The iput images resolutio i Figure 2 is 300 360, ad we measured the rutime o a Su Sparc0 workstatio. The first module, whe applied to iput image pairs (I 0, I ) ad (I 0, I 2), derived warp fuctios W 0, W 0, W 02, ad W 20. The secod module, which rus without user iteractio, computed W ic ad W Ci i 59 secods. The third module took seve secods to obtai the uiform ibetwee image i Figure 2 from bledig vector b = (/3, /3, /3). A ouiform i-betwee image requires more computatio tha a uiform i-betwee image because three surfaces must be costructed to determie bledig fuctio B C. It took 38 secods to derive the ibetwee image i Figure 3 from the user iput show i Figure 4. To use preprocessig ad postprocessig, we must compute warp fuctios P i ad Q. It took 43 secods to derive the i-betwee image i Figure 8, which icludes the preprocessig ad postprocessig effects defied by Figure 2 ad Figure 3. Metamorphosis examples The top row of Figure 6 shows the iput images, I 0, I, ad I 2. We selected three groups of features i these images ad assiged them bledig value b i = to geerate i-betwee images. The feature groups cosisted of the hair, eyes ad ose, ad mouth ad chi. Each feature group was selected i a differet iput image. For istace, Figure 4 shows the feature groups selected to geerate the leftmost ibetwee image i the middle row of Figure 6. Notice that the i-betwee image is the same as I i Figure 6. The middle ad bottom rows of Figure 6 show the ibetwee images resultig from all possible combiatios i selectig those feature groups from the iput images. Figure 7 shows the chages i i-betwee images whe we vary the bledig values assiged to selected feature regios i the iput images. For example, cosider the middle image i the bottom row of Figure 6. We derived that image by selectig the mouth ad chi, hair, ad eyes ad ose from iput images I 0, I, ad I 2, respectively. Bledig value b i = was assiged to each selected feature group. I Figure 7, we geerated ibetwee images by chagig b i to 0.75, 0.5, 0.25, ad 0.0, from left to right ad top to bottom. Note that decreasig a assiged bledig value dimiishes the ifluece of the selected feature i the i-betwee image. For istace, with b i = 0, the selected features i all the iput images vaish i the lower right i-betwee image i Figure 7. I polymorph, a i-betwee image is represeted by a poit i the simplex whose vertices correspod to iput images. We ca geerate a aimatio amog the iput images by derivig i-betwee images at poits that costitute a path i the simplex. Figure 8 shows a metamorphosis sequece amog the iput images i Figure 6. We obtaied the i-betwee images at the sample 68 Jauary/February 998
8 A metamorphosis sequece amog three images. I 0 p 0 I I 2 9 The path ad sample poits for the metamorphosis sequece. poits o the circle iside the triagle show i Figure 9. The top left image i Figure 8 correspods to poit p 0. The other images, read from left to right ad top to bottom, correspod to the poits o the circle i couterclockwise order. Every i-betwee image bleds the characteristics of all the iput images at oce. The bledig is determied by the positio of the correspodig poit relative to the vertices i the triagle. For example, iput image I 0 domiates the features i the top left i-betwee image i Figure 8 because poit p 0 lies close to the vertex of the triagle correspodig to I 0. Figure 20 shows metamorphosis examples from four iput images. The iput images appear i the top row. Figure 20e is the cetral image, a uiform i-betwee (a) (b) (c) (d) 20 Iput images (top row) ad ibetwee images from them (bottom row). (e) (f) (g) (h) IEEE Computer Graphics ad Applicatios 69
Feature Article image geerated by bledig vector (/4, /4, /4, /4). I Figure 20f, the hair, eyes ad ose, mouth ad chi, ad clothig were derived from Figures 20d, 20a, 20b, ad 20c, respectively. We rotated ad elarged the eyes ad ose i Figure 20a i the preprocessig step to match them with the rest of the face i Figure 20f. The eyes ad ose i Figure 20g resulted from selectig those i Figures 20b ad 20d ad assigig them bledig value b i = 0.5. The hair ad clothig i Figure 20g were derived from Figure 20c ad 20d, respectively. I Figure 20h, we retaied the hair ad clothig from Figure 20a. The eyes, ose, ad mouth were bleded from Figures 20b, 20c, ad 20d with b i = /3. The resultig image resembles a ma with a woma s hairstyle ad clothig. Discussio I this sectio, we discuss the applicatio of polymorph i feature-based image compositio ad extesios of the implemeted system. Applicatio Polymorph ideally suits image compositio applicatios that seamlessly bled elemets from two or more images. The traditioal view of image compositio is essetially oe of geeralized cut-ad-paste. That is, we cut out a regio of iterest i a foregroud image ad idetify where to paste it i the backgroud image. Compositio theory permits several variatios for bledig, particularly for removig hard edges. Although image compositio is widely used to embellish images, curret methods are limited i several respects. First, compositio is geerally a biary operatio restricted to oly two images at a time the foregroud ad backgroud elemets. I additio, geometric maipulatio of the elemets is ot effectively itegrated. Istead, it is geerally hadled idepedetly of the bledig operatios. Our examples demostrated the use of polymorphig for image compositio. We exteded the traditioal cut-ad-paste approach to effectively itegrate geometric maipulatio ad bledig. A composite image is treated as a metamorphosis betwee selected regios of several iput images. For example, cosider the regios selected i Figure 4. Those regios seamlessly bled together with respect to geometry ad color to produce the i-betwee image i Figure 6. That image would otherwise be cosiderably more cumbersome to produce usig covetioal image compositio techiques. Extesios I the polymorph system, we use warp propagatio to obtai warp fuctios W ij for all pairs of iput images. To eable the propagatio, we eed to derive warp fuctios associated with the edges selected to make graph G coected. The selected edges i G are idepedet of each other i computig the associated warp fuctios. We ca specify differet feature sets for differet pairs of iput images to apply differet warp geeratio algorithms. This permits the reuse of feature correspodece previously established for a differet applicatio, such as morphig betwee two images. Also, simple trasformatios like a affie mappig may serve to represet warp fuctios betwee a image pair whe appropriate. We derive warp fuctios W ij betwee iput images to compute warp fuctios W ic ad W Ci. Suppose that we specified the same set of features for all iput images. For example, give face images, we ca specify the eyes, ose, mouth, ears, ad profile of each iput face I i as its feature set F i. I this case, W ic ad W Ci ca be computed directly without derivig W ij. That is, we compute the cetral feature set F C by averagig the positios of feature primitives i F i. We ca the derive W ic ad W Ci by applyig a warp geeratio algorithm to the correspodece betwee F i ad F C i both directios. With the same set of features o iput images, we ca derive warp fuctios W i for a uiform i-betwee image i the same maer as W ic. Give a bledig vector, we derive a i-betwee feature set F by weighted averagig feature sets F i o iput images. W i ca the be computed by a warp geeratio algorithm applied to F i ad F. With this approach, we do ot eed to maitai W ic ad W Ci to compute W i for a uiform i-betwee image. However, this approach requires a warp geeratio algorithm to ru times wheever a i-betwee image is geerated. This takes more time tha the approach we described usig W ic ad W Ci. I the polymorph system, oce we have derived W ic ad W Ci, we ca quickly compute W i by liearly iterpolatig warp fuctios ad applyig fuctio compositios. Coclusios Polymorph provides a geeral framework for morphig amog multiple images. We exteded covetioal morphig to derive i-betwee images from more tha two images at oce. This paradigm requires feature specificatio amog oly ( ) pairs of iput images, a sigificat savigs over all ( )/2 pairs. The use of preprocessig ad postprocessig stages accommodates fie cotrol over the scalig ad positioig of selected iput regios. I this maer we resolve coflictig positios of selected features i iput images whe they are bleded to geerate a ouiform i-betwee image. Polymorph is ideally suited for image compositio applicatios where elemets from multiple images are bleded seamlessly. A composite image is treated as a metamorphosis betwee selected regios of iput images. Future work remais i simplifyig the feature specificatio process through the use of sakes 3 ad itelliget scissors. 0 Ackowledgmets Seugyog Lee performed work described i this article while he was a visitig scholar i the Departmet of Computer Sciece at the City College of New York. This work was supported i part by NSF Presidetial Youg Ivestigator Award IRI-957260, PSC-CUNY grat RF- 66735, ad Pohag Uiversity of Sciece ad Techology grat RB970060. 70 Jauary/February 998
Refereces. G. Wolberg, Digital Image Warpig, IEEE Computer Society Press, Los Alamitos, Calif., 990. 2. T. Beier ad S. Neely, Feature-Based Image Metamorphosis, Computer Graphics (Proc. Siggraph 92), Vol. 26, No. 2, 992, pp. 35-42. 3. S.-Y. Lee et al., Image Metamorphosis Usig Sakes ad Free-Form Deformatios, Proc. Siggraph 95, ACM Press, New York, 995, pp. 439-448. 4. S. Lee et al., Image Morphig Usig Deformatio Techiques, J. Visualizatio ad Computer Aimatio, Vol. 7, No., 996, pp. 3-23. 5. D.A. Rowlad ad D.I. Perrett, Maipulatig Facial Appearace through Shape ad Color, IEEE Computer Graphics ad Applicatios, Vol. 5, No. 5, 995, pp. 70-76. 6. J.F. Hughes, Scheduled Fourier Volume Morphig, Computer Graphics (Proc. Siggraph 92), Vol. 26, No. 2, 992, pp. 43-46. 7. A. Lerios, C.D. Garfikle, ad M. Levoy, Feature-Based Volume Metamorphosis, Proc. Siggraph 95, ACM Press, New York, 995, pp. 449-456. 8. P. Litwiowicz ad L. Williams, Aimatig Images with Drawigs, Proc. Siggraph 94, ACM Press, New York, 994, pp. 409-42. 9. S. Lee, G. Wolberg, ad S.Y. Shi, Scattered Data Iterpolatio with Multilevel B-splies, IEEE Tras. Visualizatio ad Computer Graphics, Vol. 3, No. 3, 997, pp. 228-244. 0. E.N. Mortese ad W.A. Barrett, Itelliget Scissors for Image Compositio, Proc. Siggraph 95, ACM Press, New York, 995, pp. 9-98. Seugyog Lee is a assistat professor i the Departmet of Computer Sciece ad Egieerig at Pohag Uiversity of Sciece ad Techology (Postech), Korea. He received his BS i computer sciece ad statistics from Seoul Natioal Uiversity, Korea, i 988, ad his MS ad PhD i computer sciece from Korea Advaced Istitute of Sciece ad Techology (KAIST) i 990 ad 995, respectively. His research iterests iclude computer graphics, computer aimatio, ad image processig. George Wolberg is a associate professor i the Departmet of Computer Sciece at the City College of New York. He received his BS ad MS i electrical egieerig from Cooper Uio i 985, ad his PhD i computer sciece from Columbia Uiversity i 990. His research iterests iclude image processig, computer graphics, ad computer visio. Wolberg received a 99 NSF Presidetial Youg Ivestigator Award ad the 997 CCNY Outstadig Teachig Award. He is the author of Digital Image Warpig (IEEE Computer Society Press, 990). Sug Yog Shi is a full professor i the Departmet of Computer Sciece at Korea Advaced Istitute of Sciece ad Techology (KAIST), Taejo, Korea. He received his BS i idustrial egieerig i 970 from Hayag Uiversity, Seoul, Korea, ad his MS ad PhD i idustrial ad operatios egieerig from the Uiversity of Michiga, A Arbor, USA, i 983 ad 986, respectively. His research iterests iclude computer graphics ad computatioal geometry. Cotact Lee at the Departmet of Computer Sciece ad Egieerig, Postech, Pohag, 790-784, S. Korea, e-mail leesy@postech.ac.kr. Cotact Wolberg at the Departmet of Computer Sciece, City College of New York, 38th St. at Covet Ave., New York, NY 003, e-mail wolberg@cs-mail.egr. ccy.cuy.edu. Cotact Shi at the Departmet of Computer Sciece, KAIST, Taejo, 305-70, S. Korea, e-mail syshi@ jupiter.kaist.ac.kr. IEEE Computer Graphics ad Applicatios 7