Compuer Games Technology Volume 2008, Aricle ID 35398, 7 pages doi:0.55/2008/35398 Research Aricle Auo Coloring wih Enhanced Characer Regisraion Jie Qiu, Hock Soon Seah, Feng Tian, Quan Chen, Zhongke Wu, 2 and Konsanin Melikhov Ineracion and Enerainmen Research Cener, School of Compuer Engineering, Nanyang Technological Universiy, 50 Nanyang Drive, Singapore 637553 2 College of Informaion Science and Technology, Beijing Normal Universiy, Beijing 00875, China Correspondence should be addressed o Jie Qiu, jqiu@nu.edu.sg Received 27 July 2007; Acceped 2 Ocober 2007 RecommendedbyKevinKokWaiWong An enhanced characer regisraion mehod is proposed in his paper o assis he auo coloring for 2D animaion characers. Afer skeleons are exraced, he skeleon of he characer in a arge frame is relocaed based on a sable branch in a reference frame. Subsequenly he characers among a sequence are auomaically mached and regisered. Occlusion are hen deeced and locaed in cerain componens segmened from he characer. Two differen approaches are applied o color regions in componens wihou and wih occlusion respecively. The approach has been esed for coloring a pracical animaion sequence and achieved high coloring accuracy, showing is applicabiliy in commercial animaion producion. Copyrigh 2008 Jie Qiu e al. This is an open access aricle disribued under he Creaive Commons Aribuion License, which permis unresriced use, disribuion, and reproducion in any medium, provided he original work is properly cied.. INTRODUCTION Inking/coloring of he individual animaed characers for every frame is one of he mos ime-consuming and laborinensive procedures in cel animaion producion. Many sysems and some algorihms have been proposed o assis he coloring of cel animaion producion. However, hese sysems or algorihms have significan limiaion, in erms of no being able o auomaically esablish he correc region correspondences when here is a big change or occlusion of he characer, as reviewed in []. In summary, compuerassised auo coloring (CAAC) remains a ough issue in research. To deec and handle occlusion, a novel characer regisraion mehod was proposed in our previous work [2], as he firs aemp for auo coloring 2D characers for some special cases. As indicaed in [2], he auo coloring approach may be challenged if he characer s posiion changes grealy in a sequence of frames. In his paper, a relocaion mehod is proposed o enhance he characer regisraion approach in [2], making he auo coloring approach more robus. The res of he paper is organized as follows. Firs, he enhanced characer regisraion mehod is inroduced in deail. Subsequenly, our occlusion deecion and auo coloring process is inroduced. Experimens are hen designed o es he algorihm and resuls analyzed. Conclusion and fuure work can be found a he end of his paper. 2. CHARACTER REGISTRATION 2.. Skeleon exracion A skeleon is a useful shape absracion ha capures he essenial opology of an objec in boh wo and hree dimensions. I represens a hinned version of he original objec, which sill reains he shape properies of he original objec [3]. The skeleon of a 2D characer can preserve he mos essenial geomeric and opological informaion of i, which makes i suiable for our characer regisraion purpose. In pracical animaion producion, some characers are complex feauring sharp convex or concave deails in heir ouline, which inroduce redundan branches when he skeleon exracion approach based on acive conours [, 5] is applied. As shown in Figure, he branches in red and green do no relae o he physical srucure of he characer. These are viewed as noise in his paper. The shor redundan branches in Figure can be easily pruned based on a hreshold, bu i is difficul o remove he long ones. To reduce he noise, Gaussian filering is applied o smooh he ouline and remove he deails. The Gaussian funcion in 2D form is G(x, y) = 2πσ 2 e(x2 +y 2 )/ 2σ 2. ()
2 Compuer Games Technology Figure 3: Exraced skeleon and opology graph. σ = 0.5 σ = 2.5 (e) Ouline Skeleon Redundan shor branches Redundan long branches Figure : Exraced skeleons. σ = σ = 3 (f) σ =.5 σ = 3.5 (g) σ = 2 σ = (h) Figure 2: Skeleon exracion afer Gaussian smoohing. er pars usually exiss for expressing he local moion like he movemens of ariculaed pars. Wih hese uncerainies, i is hard o find an accurae ransform o locae he wo characers in he same posiion wih he mos similariy. When drawing inbeweens, animaors locae characers of wo key frames by overlapping he mos similar pars of he characer in he same posiion. Mimicking his mehod, wo skeleons can be relocaed according o a sable branch which does no change much among he whole animaion sequence. The sable branch of each frame can be ineracively seleced by he user. Alernaively, user only needs o indicae he sable branch in he firs reference frame o be regisered. Given a reference frame and a arge frame, a predicion and relocaion mehod is proposed as follows o reduce user inervenion. Given a populaion of random vecors x, he principal componens analysis (PCA) is defined as y = A ( x m x ), (2) The smoohed oulines and corresponding skeleons wih differen σ are illusraed in Figure 2.Theσ is empirically seleced as 3 for he examples used in his paper. Afer he ouline is smoohed, he skeleon of each characer is exraced using acive conours approach [, 5], and furher hinned under SUSAN hinning rules [6], and pruned. Finally, i is segmened ino L-branches and J- branches asinroducedin[2]. Topology graphs of each characer is also exraced using he approach inroduced in [5]. Figure 3 shows he exraced skeleon and opology graph for he characer in Frame of he firs es used in he paper. Juncion nodes are illusraed in red, leaf nodes in blue, L-branches in green, and J-branches in black, respecively. 2.2. Skeleon relocaion Animaion is an ar of capuring a series of individual movemens, wheher on film or in digial form, and replaying hem in rapid succession o give he illusion of movemen [7]. Accordingly, even in wo successive frames, he posiions of he wo characers may differ much. Besides affine ransforms which are ofen used for depicing he characer s global moion, deformaion of he charac- where A is he marix whose rows are formed from he eigenvecors of x s covariance marix C x, m x is x s mean vecor [8]. (i) Compue he PCA ransforming marices T i and M i for he sable branch B i in he reference frame, where T i and M i, respecively, correspond o A and m x in (2). (ii) Transform he skeleon S in he reference frame o he ransforming coordinae sysem R 2 according o T i and M i : S i = T i ( S M i ). (3) (iii) For each branch B i in he arge frame, compue he dissimilariy value D (i, i ). Compue he PCA ransforming marices T i and M i for a branch B i in he arge frame, where T i and M i, respecively correspond o A and m x in (2). Transform he skeleon S in he arge frame o he ransforming coordinae sysem R 2 according o T i and M i : S i = T i ( S M i ). ()
Jie Qiu e al. 3 60 00 80 20 0 60 80 200 220 20 260 0 50 00 50 200 20 00 80 60 0 20 0 20 0 60 80 50 00 50 0 50 00 Head & neck 2 3 Righ arm Lef arm Torso Righ leg 5 6 Lef leg Human and human-like characers Head & neck 2 3 Righ arm Lef arm Torso Tail Righ leg 5 6 Lef leg Back leg 2 Tail 7 6 Head & neck Torso Back Fron Fron 5leg leg 2 3 2 leg Before relocaion Afer relocaion Figure : Skeleon relocaion. Tailed human-like animals quadruped Figure 5: General opology model. Compue he Hausdorff disance H (S i, S i )beweens i and S i : 2 3 2 3 (xa ) 2 ( ) 2, d(a, b) = x b + ya y b (x, y) R 2, ( h S i, S ) i = sup inf d(a, b), H ( S i, S i a S i b S i ) = max { h ( S i, S i ) (, h S i, S )} i. (5) Compue he Hausdorff disance H (S i, S i ) beween S i and S i. (e) The dissimilariy value D (i, i )beweenbranchesb i and B i is he minimum of H (S i, S i )andh (S i, S i ): D (i, i ) = min ( ( H S i, S ) ( i, H S i, S i )). (6) (iv) If D (i, i ) is he minimum, B i and B i are corresponded, he skeleon S in he arge frame is ransformed and normalized o be S i or S i according o T i and M i. The ransforming coordinae sysem R 2 for S i and S i or S i is reaed as he global coordinae sysem R 2 g. Figure shows he relocaed skeleons and frames. Frames and 2 are represened in blue and red, respecively. 2.3. Skeleon regisraion General opology models are predefined based on he characer s physical srucure.figure 5 illusraes some common opology models, which represen he mos essenial opological informaion of he characers. 5 6 5 6 Figure 6: Skeleon regisraion. To assure regisraion accuracy, a frame is seleced as he firs reference frame from he sequence o be pained, which conains a sable branch as inroduced in Secion 2.2 and he maximum number of branches among he sequence. The skeleon and opology graph of he characer in he seleced frame is hen regisered o he general model, as illusraed in Figure 6. The opology graph is adjused accordingly. For characers in he oher frames in he sequence, heir skeleons and branches are regisered based on he branch correspondences esablished by skeleon maching inroduced in he nex secion. 2.. Skeleon maching As inroduced in [2], our skeleon maching algorihm is based on boh geomeric and opological informaion of he skeleon, conaining he following wo seps. 2... Geomeric maching Global dissimilariy due o he moion of branches is represened by he global Hausdorff disance H g (B i, B i )and compued using he global coordinae sysem R 2 g of he wo frames.
Compuer Games Technology 2 3 2 3 5 6 5 6 Frame Frame 5 Frame Frame 5 (Sep ) Figure 7: Segmenaion resul. The local deformaion of each branch is represened by he local Hausdorff disance H l (B i, B i ) and compued using he local coordinae sysem R 2 l of he wo frames esablished based on PCA [2]. Wih he global and local Hausdorff disances beween wo branches, he dissimilariy value DV is defined as Frame 5 (Sep 2) Frame 5 (Sep 3) Figure 8: Auo coloring process. ( ( ( ( )) DV(i, i ) = H g Bi, B i ) +min Hl Bi, B i ), Hl Bi, B i. (7) 3. ENHANCED AUTO COLORING 3.. Occlusion deecion Each branch B i in he reference frame is mached wih all branches in he arge frame firs. If DV(i, i ) is he minimum among all dissimilariy values, a maching (B i B i ) is obained. Subsequenly, each branch B i obains is besmached branch in he reference frame. If a bidirecional maching is achieved, (B i B i ) (B i B i ), B i and B i are corresponded (B i B i ), and B i is regisered as he same branch wih B i in he general opology model. The unregisered branches will be readjused in laer process. 2..2. Topological readjusmen Afer geomeric maching, some branches wih big moion or deformaion may be lef unregisered. To mach and regiser hese branches and deec occlusion, a opological readjusmen mehod is applied. For each seleced branch Bi in he arge frame, is merging candidaes are obained, and i is merged wih he one relaed mos closely [2]. If no merging candidae is found, geomeric maching is applied o hose unregisered branches in Bi s subgraph. 2.5. Componen segmenaion Afer skeleon regisraion, characers are segmened ino several componens corresponding o he branches in skeleon and opology graph, based on he explained area of each branch [2]. The segmened and regisered componens for Frames and 5 are illusraed in Figure 7, where corresponding componens are in he same color. To deec and locae occlusion in componens, a mehod based on he variaion of region areas is advanced in [2]. (i) Each region in he reference frame is quanized and coded as an English characer based on is area. (ii) All he regions in he arge frame are quanized and coded based on he scale poins compued in he reference frame. (iii) Regions in a componen C i in he reference frame are coded as a sring S i and compared wih he sring S i of is corresponding componen C i in he arge frame, and he leas conversion cos beween hem is compued as γ(s i S i ). (iv) Occlusion is locaed in C i if γ(s i S i ) is bigger han an empirically defined hreshold. 3.2. Auo coloring Wih occlusion deecion, regions can be divided ino wo caegories: he firs caegory conains all regions which are in he componens wihou occlusion, and he second consiss of hose in he componens wih occlusion. The former is supposed o be more sable han he laer and differen maching mehods are applied o he wo caegories. The coloring process consiss of hree seps, and he coloring resul afer each sep is illusraed in Figure 8. (i) For he firs caegory, he hierarchical feaure-based region maching approach as proposed in [, 9] is applied. (ii) For he second caegory, he coninuiy of some region conours is broken because of occlusion. Hence, feaures such as area, curve lengh, characer poins, and relaions wih neighboring regions change grealy. Bu
Jie Qiu e al. he res of regions sill have relaively sable feaures of area, curve lengh and characer poins. So a maching mehod similar o ha for he firs caegory is applied o mach hese regions. Each region is mached wih hose in corresponding componens in he reference frame. The only difference is ha he feaure of relaions wih neighboring regions is ignored, so ha he coded characer sring [] is only composed of characer poins. (iii) Finally, each of he remaining regions in he relocaed arge frame inheris he color of he region ha i overlaps mosly in he reference frame. If no such region is obained, which means ha he region fully overlaps he background, i is pained in he color which fills he majoriy of he componen (major color) ha i belongs o. Afer skeleon regisraion for he firs reference frame, he prior and subsequen uncolored frames are seleced as new arge frames ieraively. The neares colored frame which conains a sable branch is seleced as he arge frame s reference frame.. 5 Frame Frame 2 Frame 3 Frame (he firs reference frame) RESULTS AND ANALYSIS To es he possibiliy of applying our approach in pracical animaion producion, we use a Japanese-syle sequence in pracical producion of he railer of Juseen animaion. The 8 frames as shown in Figure 9 compose 2/3 of a cu in he railer, showing a boy jumping down ino he conrol room of his robo. I can be noiced ha occlusion arises due o he moion of he characer s arms. To minimize he informaion loss due o occlusion, we selec Frame as he firs reference frame. Then he prior and subsequen frames are colored one by one according o he color informaion of is nex and prior frame, respecively. From he resuls we can see ha mos of he regions are correcly colored. The regions wrongly colored are mainly because no corresponding regions in he reference frame can be obained due o he occlusion or informaion loss. Figure 0 illusraes he main errors among he sequence and he correc resuls. As illusraed in Figure 0, he righ arm in Frame 5 changes grealy compared wih he one in Frame, and he occluded par of he bracele in Frame is visible in Frame 5. Accordingly, he regions indicaed in blue circles are wrongly mached as no correc correspondences exis in Frame. Wih more reference frames (which conain he corresponding regions) provided, his kind of error can be avoided. In Figure 0, he regions in blue circles in Frame 5 are creaed due o he occlusion, and hey inheri he color of he regions hey overlap mosly in Frame as no corresponding regions are obained. In Figure 0(e), he region in blue circle in Frame 7 is creaed as he hair is swaying o he posiion overlapping wih he eyebrow. I is hus wrongly mached o a hair branch as hey have grea similariy. These errors can be solved if he hair is drawn in a separae layer. The mouh in Frame is widely open so ha he ongue appears only in his frame. I is no mached o any region, and hus inheris he color of Frame 5 (e) Frame 7 (g) Frame 6 (f) Frame 8 (h) Figure 9: Tes (resoluion: 220 220 pixels) (couresy of Anime Inernaional Co., Inc., Japan). he face in Frame 2. This kind of error due o informaion loss is hard o be avoided. In summary, he coloring accuracy for he oal 7 uncolored frames is over 93%, which shows ha our approach is applicable o pracical animaion producion. Figure shows he oher es, wih he firs frame seleced as he firs reference frames. The characer is simple, bu i is noiceable ha large moions and deformaions exis among he sequence. Wih he relocaion mehod proposed in his paper, skeleons are accuraely mached, ensuring he high coloring accuracy for he es.
6 Compuer Games Technology Figure 2: Oher frames in he cu used in Tes (couresy of Anime Inernaional Co., Inc., Japan). 5. (e) Errors (g) (f) Correc resuls (h) Figure 0: Error Analysis for Tes. CONCLUSION AND FUTURE WORK In his paper, our previous work on characer regisraion is enhanced wih a refined skeleon exracion procedure and a novel characer relocaion mehod. Wih he enhanced characer regisraion, occlusion can be correcly deeced and locaed in componens segmened from he characer. Subsequenly, wo differen maching mehods are applied o regions in componens wihou and wih occlusion, respecively. A pracical animaion sequence from a cu of Juseen animaion railer is applied o validae he approach. A high coloring accuracy is achieved. I shows ha he proposed auo coloring approach wih enhanced characer regisraion is applicable o pracical animaion producion. Neverheless, here are limiaions. The proposed approach requires ha each frame has a sable branch, which is no always rue among a long sequence. For example, he cu we used in Tes conains 2 frames in oal, and he firs frames are illusraed in Figure 2. Due o he informaion loss, no sable branch exiss in hese frames. So hey canno be auomaically colored using our approach. Fuure research work will be focusing on he auo coloring for hese kinds of special cases and new relocaion mehods. ACKNOWLEDGMENT Frame (he firs reference frame) Frame 2 This work has been parially suppored by he Science and Engineering Research Council (SERC) Gran no. 052 05 002 Ref:, which is awarded by he Agency for Science and Technology Research (A STAR) and adminisered hrough he Singapore Naional Grid Office. REFERENCES Frame 3 Frame Figure : Tes 2 (resoluion: 550 00 pixels). [] J. Qiu, H. S. Seah, F. Tian, Z. Wu, and Q. Chen, Feaure- and region-based auo paining for 2D animaion, The Visual Compuer, vol. 2, no., pp. 928 9, 2005. [2] J. Qiu, H. S. Seah, F. Tian, Q. Chen, Z. Wu, and M. Konsanin, Auo coloring wih characer regisraion, in Proceedings of he Inernaional Conference on Game Research and Developmen
Jie Qiu e al. 7 (CyberGames 06), vol. 223, pp. 25 32, Perh, Ausralia, December 2006. [3] N. D. Cornea, D. Silver, X. Yuan, and R. Balasubramanian, Compuing hierarchical curve-skeleons of 3D objecs, The Visual Compuer, vol. 2, no., pp. 95 955, 2005. [] P. Golland and W. E. L. Grimson, Fixed opology skeleons, in Proceedings of he IEEE Compuer Sociey Conference on Compuer Vision and Paern Recogniion (CVPR 00), vol., pp. 0 7, Hilon Head Island, SC, USA, June 2000. [5] J.Qiu,H.S.Seah,F.Tian,Q.Chen,andK.Melikhov, Topology enhanced componen segmenaion for 2D animaion characer, in Proceedings of Inernaional Workshop on Advanced Imaging Technology, pp. 30 35, Okinawa, Japan, January 2006. [6] S. M. Smih, Edge hinning used in he SUSAN edge deecor, Inernal Technical Repor TR95SMS5, Defence Research Agency, Surrey, UK, 995. [7] C. Pamore, The Complee Animaion Course: The Principles, Pracice and Techniques of Successful Animaion, Thames & Hudson, London, UK, 2003. [8] R. C. Gonzalez and R. E. Woods, Digial Image Processing, Addison-Wesley, Reading, Mass, USA, 992. [9] J.Qiu,H.S.Seah,F.Tian,Q.Chen,andZ.Wu, Enhancedauo coloring wih hierarchical region maching, Compuer Animaion and Virual Worlds, vol. 6, no. 3-, pp. 63 73, 2005.
Roaing Machinery Engineering The Scienific World Journal Disribued Sensor Neworks Sensors Conrol Science and Engineering Advances in Civil Engineering Submi your manuscrips a Elecrical and Compuer Engineering Roboics VLSI Design Advances in OpoElecronics Navigaion and Observaion Chemical Engineering Acive and Passive Elecronic Componens Anennas and Propagaion Aerospace Engineering Volume 200 Modelling & Simulaion in Engineering Shock and Vibraion Advances in Acousics and Vibraion