Automatic Generation of Caricatures with Multiple Expressions Using Transformative Approach

Transcription

1 Automatc Generaton of Carcatures wth Multple Expressons Usng Transformatve Approach Wen-Hung Lao and Chen-An La Department of Computer Scence, Natonal Chengch Unversty, Tape, Tawan Abstract. The prolferaton of dgtal cameras has changed the way we create and share photos. Novel forms of photo composton and reproducton have surfaced n recent years. In ths paper, we present an automatc carcature generaton system usng transformatve approaches. By combng facal feature detecton, mage segmentaton and mage warpng/morphng technques, the system s able to generate stylzed carcature usng only one reference mage. When more than one reference sample are avalable, the system can ether choose the best ft based on shape matchng, or synthesze a composte style usng polymorph technque. The system can also produce multple expressons by controllng a subset of MPEG-4 facal anmaton parameters (FAP). Fnally, to enable flexble manpulaton of the synthetc carcature, we also nvestgate ssues such as color quantzaton and raster-to-vector converson. A major strength of our method s that the syntheszed carcature bears a hgher degree of resemblance to the real person than tradtonal component-based approaches. Keywords: Carcature generaton, facal anmaton parameters, mage morphng, facal feature localzaton. 1 Introducton Carcatures are exaggerated, cartoon-lke portrats that attempt to capture the essence of the subject wth a bt of humor or sarcasm. These portrats are usually drawn by professonal cartoonsts/panters based on ther personal judgment or representatve style. Recently, nterests n creatng systems that can automatcally generate 2D carcatures usng photo as the nput have surged due to the prolferaton of mage capturng devces as well as advances n facal feature analyss. Some system (e.g., M edtor n W) employs a component-based approach, n whch a collecton of facal components are pre-defned and then selected to assemble a cartoon face ether manually or by smlarty analyss of ndvdual facal parts. A major ssue wth componentbased approach s that the composte cartoons exhbt lmted varetes n ther styles due to the nature of template selecton. Even worse, the resultng carcature sometmes bear lttle resemble to the person even when the ndvdual parts are matched flawlessly. In example-based approaches [1][2], a shape exaggeraton model s learned from a labeled example set. At runtme, the facal shape s extracted usng F. Huang and R.-C. Wang (Eds.): ArtsIT 2009, LNICST 30, pp , Insttute for Computer Scences, Socal-Informatcs and Telecommuncatons Engneerng 2010

2 264 W.-H. Lao and C.-A. La actve shape model (ASM) and then classfed nto one of the exaggeraton prototypes for further deformaton. It appears that shape plays a prmary role (compared to texture) n ths type of approach. As a result, the computer-generated carcatures are usually monochrome, lackng detals n textural elements. To address the ssues descrbed above, we have developed and mplemented an automatc carcature generaton system usng a transformatve framework [3]. In ths type of approach, a new carcature s syntheszed by transferrng the style from a reference cartoon drawng va matchng of the two sets of pre-defned control ponts n the nput photo and the reference mage, as depcted n Fg. 1. In [4], we further mprove the precson of the feature localzaton result usng actve appearance model (AAM). We also propose to create a 3D carcature model by mergng two planar carcatures usng prncples of bnocular vson. + = Carcature cartoonst by Input photo Transformed mage Fg. 1. Transformatve approach for generatng carcature from a sngle reference drawng [3] The research descrbed n ths paper marks an extenson to our prevous efforts. Specfcally, we ncorporate the MPEG-4 facal anmaton parameters nto the shape model to enable of automatc generaton of carcature wth multple expressons usng a sngle reference drawng. We also explot the dea of creatng new drawng styles by blendng features from multple sources usng polymorph technque [5]. As the collecton of reference drawngs becomes larger, t wll be convenent to select/recommend a proper template based on shape smlarty measure. Fnally, to facltate effcent storage and flexble manpulaton of the syntheszed carcature, we look nto ssues regardng color reducton and raster-to-vector converson. The rest of ths paper s organzed as follows. In Secton 2, we frst revew the facal mesh model desgned for our applcaton. We then present two general approaches for creatng carcatures from an nput photo and a reference cartoon. The second part of Secton 2 s concerned wth template selecton usng shape nformaton, as well as style blendng usng polymorph technque. Secton 3 defnes the 2D FAP set employed n our system and the assocated facal component actons, whch are further combned to derve archetypal expresson profles,.e., dfferent emoton types. Results of syntheszed mages wth multple expressons are presented as Emotcons. Secton 4 concludes ths paper wth a bref summary.

3 Automatc Generaton of Carcatures wth Multple Expressons Carcature Generaton The overall procedure for generatng carcature from nput photos s llustrated n Fg. 2. In the frst step, a frontal-vew photo s presented as the source mage. Usng actve appearance model (AAM), we can locate mportant facal components and the correspondng mesh, as shown n Fg. 3. The mesh s comprsed of 8 components, wth a total of 66 control ponts and 114 constraned conformng Delaunay trangles (CCDT) for defnng the facal confguraton. The carcature can then be generated usng two dfferent approaches: mage morphng or rotoscopng. In mage morphng, a collecton of reference cartoons wth dfferent styles needs to be gathered. Control ponts as well as the correspondng mesh of these cartoons wll be labeled manually. The user wll choose a template to transfer the style from. We then proceed to dentfy the correspondng trangular meshes between the reference cartoon and the nput mage. After the correspondence has been establshed, t s a smple matter to apply mage morphng technque to transfer the texture from the reference drawng to the nput photo. In our current mplementaton, barycentrc coordnate s employed to speed up the morphng process. Detals regardng the coordnate transformaton on CCDT can be found n [6]. Typcal morphng results are shown n Fg. 4. Input Photos Face Detecton Feature Ponts Detecton Edge Detecton Image Segmentaton Color Quantzaton Cartoon Samples Mesh Warpng Rotoscoped Image Stylsh Carcature Expressons Profle wth FAP Angry Happy Sad Fg. 2. Proposed framework for generatng carcatures wth multple expressons

4 266 W.-H. Lao and C.-A. La Fg. 3. (a) Control ponts for the 2D face model, (b) trangular facal mesh Fg. 4. Carcatures generated from dfferent reference drawngs The results depcted n Fg. 4 (bottom row) are generated usng dfferent reference cartoons (top row). Generally speakng, each syntheszed mage bears certan degree of lkeness to the orgnal photo, yet at the same tme exhbts the drawng style orgnated from the reference cartoon. As the number of avalable templates ncreases, t becomes convenent for the system to select or recommend best references based on shape smlarty. Snce each component n our face model forms a closed contour, Fourer descrptor can be utlzed to represent the ndvdual parts. Let (x (n),y (n)) denote the n-th contour pont of the -th facal component, and let u (n)=x (n)+jy (n), the facal shape descrptors (FSD) can then be defned accordng to:

5 Automatc Generaton of Carcatures wth Multple Expressons 267 a ( k) = F{ u ( n)} = N 1 n= 0 u ( n) e j 2 kn / N π (1) where N ndcates the total number of control ponts n the -th component. Usng Eq. (1), the dfference between two facal shapes P (extracted from photo) and Q s (template n the database) can be expressed accordng to: where D M ( P Qs ) =, Dst( FSD, FSD ) (2) = 1 P Qs P QS ( a ( k) a ( k) ) 0.5 L 1 2 P Q S ( FSD, FSD ) = L k= 0 Dst (3) M denotes the number of facal components, and L denotes the number of Fourer coeffcents retaned to represent the -th component. Fg. 5 presents the top 5 choces for photos of dfferent gender sung FSD dstance. An addtonal beneft of defnng FSD arses from the blendng process. It s known that blendng of styles can be easly accomplshed usng polymorph technque. Wth FSD, the weght for each template can be set to be nversely proportonal to the dstance between the nput shapes and the respectve reference carcatures, as shown n Fg. 6. The second method to generate carcature s rotoscopng. In rotoscopng, we combned contour detecton, mesh generaton, hstogram specfcaton and mean-shft algorthm to obtan a smplfed regon-based representaton of the face wth user-defned Shape Dst Shape Dst. 0.7 d Dataset Order Dataset Order Fg. 5. Top template recommendatons usng shape smlarty analyss

6 268 W.-H. Lao and C.-A. La D(I 0,I 1 ) W1 D(I 0,I 2 ) W2 D(I 0,I 3 ) I W3 Fg. 6. Blendng of styles usng polymorph wth FSD-controlled weghts Fg. 7. Rotoscoped mages color palette. Detals regardng the rotoscopng process can be found n [6]. Fg. 7 llustrates some rotoscoped mages. Due to the nature of mean-shft segmentaton, some trangular meshes are merged and contan the same color. These smplfed regons are readly subject to raster-to-vector converson to obtan vector-based representaton such as Scalar Vector Graphcs (SVG). It should be noted that the color quantzaton stage wll effectvely reduce the number of colors used n the resultng pcture, where as the hstogram specfcaton stage wll attempt to preserve color nformaton from the orgnal photo. These steps can also be ncorporated nto the mage morphng approach to retan the color tone from the nput photo.

7 Automatc Generaton of Carcatures wth Multple Expressons Synthess of Multple Expressons The FAP n MPEG-4 defnes the set of parameters for 3D facal anmaton. Here we are manly concerned wth the synthess on a 2D model. Consequently, we decompose the archetypal expresson profle defned n [7] to arrve at a reduced set of facal anmaton parameters (FAP) for 2D carcatures, as shown n Table 1. Usng facal component acton defnton, the FAP can be set to manpulate the 66 facal defnton ponts (FDP) n our model drectly to synthesze dfferent expressons. We have also defned 12 profles accordng to popular emotcons. An example s llustrated n Table 2, where the emoton type (surprse) and ts correspondng facal component acton defnton (FCAD), appled FDP and adjusted FAP are tabulated. Fg. 8 depcts the cartoons generated usng the emotcon settngs. Table 1. Extracted 2D FAP Set Group Descrpton FAP No. No. of Ponts No. of Ponts Included n Reduced 2D Set 1 Vsemes, Expressons 1~ Jaw, chn, nner-lowerlp, corner-lps, 3~18 mdlp (69%) 3 Eyeballs, pupls, eyelds 19~ (33%) 4 Eyebrow 31~ (100%) 5 Cheeks 39~ Tongue 43~ Head rotaton 48~ Outer-lp postons 51~ (80%) 9 Nose 61~ Ears 65~ Total (45.6%) Table 2. An example of emotcon profle settng Emotcon FCAD Appled FDP Related FAP Surprse Brow: up Eye: wde Jaw: oh F 31,33,35, 32,34,36 (1), F 37,38 (1.2) F 19,21 (1.2), F 20,22 (1.2) F 3 (0.6), F 4,8,9 (0), F 5,10,11 (-0.6)F 12,59 (-0.8), F 13,60 (-0.8)F 6,53 (0.5), F 7,54 (0.5)F 55,56,57,58 (1.3)

8 270 W.-H. Lao and C.-A. La Multple Expressons Generaton Neutral Angry Annoyed Smle Sadness Happness Surprse Blnk Kss Laugh Sleepy Snster Smle Fg. 8. Syntheszed cartoons wth multple expressons 4 Conclusons We have demonstrated an automatc carcature generator that s capable of syntheszng personalzed cartoons wth multple expressons usng only one reference mage. New styles can be easly created by blendng exstng samples. The ncorporaton of FSD facltates selecton of templates n a cartoon database, as well as weght calculaton n polymorph process. Compact representaton of the syntheszed carcatures remans an ssue for further nvestgaton. Current model cannot strke a balance between fdelty and color quantzaton/regon segmentaton factors. Raster-to-vector converson usually generates over-segmented patches. Novel types of vector representaton such as dffuson curves may be appled to tackle ths problem. References 1. Lang, L., Chen, H., Xu, Y.Q., Shum, H.Y.: Example-Based Carcature Generaton wth Exaggeraton. In: 10th Pacfc Conference on Computer Graphcs and Applcatons (PG 2002), pp (2002)

9 Automatc Generaton of Carcatures wth Multple Expressons Mo, Z., Lews, J.P., Neumann, U.: Improved Automatc Carcature by Feature. Normalzaton and Exaggeraton. In: Proceedngs of ACM SIGGRAPH Conference on Computer Graphcs and Interactve Technque (2004) 3. Chang, P.Y., Lao, W.H., L, T.Y.: Automatc Carcature Generaton by Analyzng Facal Features. In: Proc. of 2004 Asan Conference on Computer Vson (2004) 4. Chen, Y.L., Lao, W.H., Chang, P.Y.: Generaton of 3D Carcature by Fusng Carcature Images. In: IEEE Internatonal Conference on Systems, Man and Cybernetcs, vol. 1, pp IEEE Press, Los Alamtos (2006) 5. Lee, S., Wolberg, G., Shn, S.Y.: Polymorph: Morphng Among Multple Images. IEEE Computer Graphcs and Applcatons 18, (1998) 6. La, C.-A.: Automatc Generaton of Carcatures wth Multple Expressons Usng. Transformatve Approach. Master s Thess, Natonal Chengch Unversty (2008) 7. Perln, K.: Layered Compostng of Facal Expresson. In: ACM SIGGRAPH Techncal Sketch (1997)