Resolvng Ambguty n Depth Extracton for Moton Capture usng Genetc Algorthm Yn Yee Wa, Ch Kn Chow, Tong Lee Computer Vson and Image Processng Laboratory Dept. of Electronc Engneerng The Chnese Unversty of Hong Kong, Shatn, Hong Kong {yywa, ckchow1, tlee}@ee.cuhk.edu.hk Abstract - There are consderable nterests n moton capture from an mage sequence taken from a vdeo camera. However, snce the mages only consst of D nformaton, the dstance of an object from the mage plane cannot be determned unquely unless some constrants are mposed. Coplanar moton s a popular constrant adopted for ths purpose. If the captured moton s not coplanar, only two possble solutons can be obtaned at the best even the actual se of the object s gven. Therefore, the moton cannot be captured wthout resolvng the dstances of each control pont from the mage plane. By assumng smooth transton, the problem s formulated as one mnmng the transton between mage frames and applcaton of GA to solve ths optmaton problem s proposed. I. INTRODUCTION Moton capture makes anmated moton realstc. It s an effcent way to record any human arbtrary moton []. Applcatons nclude creatng 3D humanod anmaton, usng a pre-generated humanod 3D model wth captured moton, n games, moves, vrtual human, smulatons and even for human moton analyss. Tradtonal ways of moton capture nclude magnetc sensors or markers on real human model to record the moton by multple cameras from dfferent drectons. Herda, Fua, Plankers, Boulc and Thalmann [7] suggested optcal moton capture whle Moe, Ta and Gerard [6] suggested capturng moton by 17 cameras surroundng. However, snce the optcal and magnetc components are very expensve, such systems are not sutable for low cost applcatons. Recently, consderable attenton has been gven to moton capture from monocular sequence of D mages. Ths could largely reduce the producton cost. Whle the D coordnates (x and y) are readly avalable from the D mages wth ths approach, a major dffculty s to extract depth nformaton, whch s the dstance of the object from the mage plane. There are some trcks to extract the depths n order to reconstruct the 3D moton. Holt, Netraval, Huang and Qan [3] advocated an approach of decomposton wth assumptons of coplanar to determne artculated moton from perspectve vews. Zhuang, Lu and Pan [5] proposed an approach for vdeo moton capture usng feature trackng, such as jont and color model of body part, by Kalmen flter and morph-block smlarty algorthm. Taylor [4] presented an algorthm of recoverng nformaton about the confguraton of an artculated object from pont correspondences n a sngle mage. As many as 048 possble solutons for a model wth 11 jonts were obtaned for human moton capture and the best soluton was selected by the user. The unque soluton can only be obtaned when the coplanar constrants are gven. Pan and Ma [] too adopt coplanar constrant and artculated model assumptons to estmate 3D moton of a human walkng. However, f the actual se of an object s known, the number of solutons for a pont can be reduced to two. Ths s llustrated n fg. 1. Gven two control ponts J 0 and J 1 on an object and the lnk between them, the depth nformaton can be obtaned n a sngle D mage from ts projecton on the mage plane because x + y + = d where x and y are the projecton coordnates of the lnk whle d s the physcal dstance of the lnk. However, there are two solutons and f an object has N lnks, the number of possble solutons s N. y x d (x,y,-) -θ J 0 J 1 +θ x (x,y,) Fg. 1 Top vew of a jont s projecton Image plane Ths paper descrbes an applcaton of Genetc Algorthm (GA) to determne the moton from a monocular mage sequence wthout coplanar constrant, by resolvng the ambguty n depth extracton for each lnk. Usng a camera wth hgh samplng rate, the captured moton s expected to be smooth and movng n one drecton rather than havng sudden change of movng drecton. Therefore, the absolute values of rotatng angles are expected to be small and hence the change of rotatng angles from one frame to the next should also be small. Thus we propose to determne the depth nformaton of each control pont n each mage frame by 0-7803-78-4/0/$10.00 00 IEEE
fndng the best combnaton that gves a moton wth the smoothest transton. Snce the depth of each control pont at each mage frame can have two possble solutons, determnng the best combnaton becomes a combnatoral optmaton problem that s most sutably solved usng GA. The remanng of ths paper s organed as follows. Frst, the problem of moton capture s brefly descrbed n the next secton. Then the depth extracton problem for moton captured s formulated as an optmaton problem and ts soluton s proposed n secton III. Experments applyng the proposed GA for moton captured on two mage sequences are reported n secton IV. Fnally, dscusson and concluson are gven n secton V and VI respectvely. II. MOTION ANALYSIS Frstly, we defne an artculated humanod stck model as shown n fgure. Hence moton capture becomes matchng the artculated model wth the human posture dentfed n the mage. An assumpton s also made that the correspondence between the jonts of the model and the feature ponts n each mage are provded. Secondly, the D nformaton and possble depths of each jont are derved from each D mage. The derved depths are then nput to the GA formulated n secton III to determne the most sutable combnaton for a moton sequence wth smooth transton. A. Artculated Model e 3 e 5 J 7 J 8 e 5. The trunk physcally has a constant orentaton wth respect to the lmbs J 7 -J 9 -J 11 -e 3 and J 8 -J 10 -J 1 -e4 that the lmbs rotate about J 7 and J 8 wth the trunk n statonary relatvely. J 0 plays the role as the centrod of the model, about whch J 0 - J 7 -J 8 and J 0 -J 1 -J rotate. There exst planes formed by J 1,,7,8 - J 3,4,9,10 -J 5,6,11,1 and J 3,5 -J 5,6 -e 1, such that lnk J 11 -J 9 rotates about the normal of plane J 7 -J 9 -J 11 at J 11 and smlar for others. Besdes, each jont has ts own lmts on the range of rotatng angles. These defne the set of feasble solutons. B. Reference Length When applyng the stck model for moton capturng, the reference length of each lnk needed to be determned n advance because dfferent persons would have dfferent ses. They can be measured when all the lnks are parallel to the mage plane. For example, fgure 6 shows a stand stll human model and the reference lengths are measured n ths frame. Generally, t s hard to have the feet absolutely parallel to mage plane, so the reference lengths of the feet can only be measured approxmately. C. Perspectve Scale Although the reference lengths of each lnk can be obtaned usng a reference frame, the lnks would appear to have dfferent lengths at dfferent frames because the object may move away or closer to the camera. However, the lengths of each lnk should only dffer by a scale factor. Ths scale factor can be found wth the ad of the trangular-lnked jonts J 0 -J 1 -J. As the reference lengths of the 3 lnks are known, the scale factor, s, n a partcular frame can be calculated by trgonometry. The steps are shown as follow: a) Measure the reference lengths of lnks J 0 -J 1 (R 01 ), J 1 -J (R 1 ), and J 0 -J (R 0 ). See fgure 3 (a). b) Smlarly, from one partcular frame, measure the projected lengths of the 3 lnks as r 01, r 1 and r 0 n the D mage. See fgure 3 (b). J 11 J 0 J 10 J 9 J1 J c) r s n fact the projecton of scaled R. See fgure 3 (c). Holt, Netraval, Huang and Qan [3] mentoned that human bodes can be approxmately modeled by artculated objects. So, a stck model shown n fgure s adopted n our work. The model descrbes the man jonts of a real human body and can be decomposed nto lmbs: J 1 -J 3 -J 5 -e 1, J -J 4 -J 6 - e, J 7 -J 9 -J 11 -e 3 and J 8 -J 10 -J 1 -e 4, trunk: J 7 -J 8 -J 0 -J 1 -J and head: J 1 e 4 (a) and (b) show two possble depths of J 1, one s behnd and one s n front of J 0. J 3 J 4 Let J 0 = (x 0, y 0, o ), J 1 = (x 1, y 1, 1 ), J = (x, y, ). e 1 In (a) and (b) J 5 J 6 e r 01 1 = 0 ± R01 (1) Fg. The artculated stck model for human moton analyss defned wth 13 jonts and 5 endponts. 0-7803-78-4/0/$10.00 00 IEEE s where s s the scale due to perspectve projecton. s obtaned n the smlar way and to be r0 = 0 ± R0 () s In order to elmnate 0, R 1 s consdered. Smlar to (a) and (b), but wth J 0 replaced by J, the followng relaton can be obtaned.
r1 R 1 = ( 1 ) + (3) s Then, s can be derved by solvng equatons (1) to (3) snce R are the reference lengths and r can be obtaned from each frame. y J 1 x (a) J 0 R 01 R 0 J 1 R 1 D. Depth Extracton R 01 J Fg. 3 (a)reference, (b)scaled projecton of the trangularlnked J 0 -J 1 -J, (c) re-scaled projecton of lnk J 01 When a 3D object s projected onto a D mage, as shown n fg. 4, the dsplacement of the object on the mage plane can be measured from the D coordnates (x, y) on the mage plane but the depth cannot be determned unquely. There are two possbltes correspondng to swng forward poston (x front, y front ) or backward poston (x back, y back ). Fgure 4 shows possble depths of jont 10 where the reference length of lnk J 8 -J 10 s known. The possble depths can be found by trgonometry, j 1 (c) 1-0 J 0 0-1 j 0 r 01 r 0 r 1 (b) r 01 s 1 R r 01 01 0 J s 1 0 1 j J 0 Image plane moton to be captured. We propose to solve ths problem by a GA n the followng secton. y x sr e5 J8 - -θ r +θ J10 (x back,y back ) III. OPTIMIZATION BY GENETIC ALGORITHM We ntend to derve the requred soluton by fndng the set of θ for each lnk n each frame such that a smooth moton can be obtaned. Therefore, the change of angles should not be too much along the mage sequence. Followng s an example llustratng the assumpton. Consderng a sample human moton shown n fgure 5. Assume the model standng wth hs hands pontng down parallel to the mage plane n the frst frame. In the second frame, possbly, he may swng hs hands upward n front of or behnd the body, and the same for the thrd frame. However, the moton s supposed to be smooth, such that, the hands should swng ether n front of or behnd the body all the way. It s not expected to have the hands to swng n front of the body now but behnd after a few mn-seconds then back to front mmedately. If so, the resultng absolute value of change of rotatng angle wll be very large and n dfferent drectons compared to the adjacent frames. + J10 (x front,y front ) Image plane Fg. 4 Top vew of the projecton of jont 10 = ( sr) θ = cos 1 r r sr (4) (5) where r represents the projected length of body part (measured from the D mage) and sr represents the scaled reference length. ± s the depth relatve to the head and ±θ s the relatve rotatng angle. Therefore the depth of lnk can be defned by θ whch could take ether postve or negatve values. So for N lnks n the model, there wll be N θ and each could take ether postve or negatve values. Therefore, gven each value of θ for each lnk n each frame, the objectve of our work s to determne the most sutable set of sgns of θ descrbng the Fg. 5 Example of an mage sequence Therefore, we have the followng total change of rotatng angle crteron: F = M 1N 1 = 1 j = 1 α j + 1θ j + 1 α θ j j (6) 0-7803-78-4/0/$10.00 00 IEEE
M and N are the total numbers of jonts and frames. stands for the relatve rotatng angle of jont n frame j. s the scale factor of the rotatng drecton whch s ether 1 for +ve rotaton and 1 for ve rotaton. So we seek the values of to mnme Eq. (6). Wth Eq. (6) as the ftness functon, the drectons of rotaton ( ) can be derved usng GA such that F s mnmed. Therefore, after all have been obtaned from the mage sequence, we defne the chromosome wth θ j θ j for all and j. Table I shows an example of the chromosome wth 1 or -1 representng one gene. TABLE I DEFINITION OF A CHROMOSOME Jont 1 Jont Jont Frame 1 1-1 -1 Frame 1 1-1 Frame j -1-1 -1 A. Reproducton Wth the chromosomes defned above, new offsprng s generated usng the followng cross-over and mutaton operatons. 1) Cross-Over A mult-pont cross-over operator s adopted here. Gven two chromosomes, the number of genes go through the crossover operaton s randomly generated. The locaton of the cross-over pont s also randomly generated such that all genes have the same probablty of gong through cross-over. Wth the human model we defned n fgure, whch has 18 jonts, the expected number of genes that wll go through the cross-over operaton s 9.5. ) Mutaton After cross-over, the chromosome wll go through the mutaton process. Agan, the number of genes that wll go through the mutaton s randomly generated wth equal probablty. Therefore the expected number of genes to be mutated s also 9.5. Then the locaton of genes that wll go through mutaton s randomly generated next. As shown n Table I, each gene takes on the value of ether 1 or -1. The gene to be mutated wll have ts value change from -1 to 1 or vce versa. B. Selecton After reproducton, the new populaton s doubled (from n to n) n se ncludng the orgnal populaton. A new generaton s formed by selectng the best n chromosomes from the new populaton. To do the selecton, we examne the ftness of each chromosome defned n Eq. (6). Snce our objectve s to mnme the change of angles, chromosome wth smaller ftness s the preferred soluton. Therefore, chromosomes wth large F are omtted so that the populaton s reduced to the orgnal n. C. Termnaton The process (reproducton ftness functon selecton reproducton ) wll contnue untl the soluton converges. Ths s done by checkng the resultng F n Eq. (6); f t reaches stable state, t converges. IV. EXPERIMENTAL RESULTS The algorthm has been appled to two real mage sequences. One s about a human pretendng to pck up somethng on the ground and another one s about a snger walkng freely on the stage. Both of them were recorded by a sngle monocular camera whle the frst one was taken from the front and the second from the sde. The mages were captured wth 0.3 seconds dfference n-between every two frames. 6 frames of the mage sequences are shown n fgures 8 and 10 whle the estmated motons n fgures 9 and 11 respectvely. Snce we assumed that the jont locatons are avalable, they are marked manually. Fg. 6 Approxmaton of reference lengths from frame 0 of the frst moton sequence Fg. 7 Approxmaton of reference lengths from the last frame of the second moton sequence In our experments, the populaton se was set to be 1000 and the maxmum number of generaton to be 40. In both experments, 10 trals were taken, and 8 converged to the same soluton whle the other converged to solutons wth hgher ftness values. A. Pckng Up Somethng On The Ground At Front Vew Fgure 6 shows the stand stll model, whch provdes suffcent nformaton on the reference length of every part of the human model, except for the feet because they are not exactly parallel to the mage plane. Subsequently, the reference lengths of the feet have to be estmated. 0-7803-78-4/0/$10.00 00 IEEE
Comparng fgures 8 and 9, the estmated poses were very much lke the real mage sequence. B. Snger Walkng Freely On Stage Captured At Sde Vew For ths sequence, no stand-stll reference frame s avalable. The reference lengths were estmated from the last frame because the upper part of the human model was approxmately parallel to the mage plane, just smlar to the stand stll pose. The results were shown n fgure 7. The frame also showed the sde vew of the feet, from whch, the reference length of the feet could be estmated. Ths approxmaton step may ntroduce error to the jont locaton and degrade the result but good results were also obtaned as depcted n fg. 11. Comparng fgures 10 and 11, the estmated poses were also very much smlar to the real mage sequence. V. DISCUSSION The current mplementaton assumes the correspondence between the jonts of the real human body and the feature ponts on the D mage are provded. In practce, ths s not applcable and automatc feature pont mappng should be appled. As the performance of the algorthm hghly depends on how accurate the feature ponts are beng marked n the mages, an accurate automatc feature extracton system should be developed to mark the jonts. Alternatvely, colored markers or Morph-block algorthm n [5] may be appled to track the jonts. More constrants may be consdered to obtan more accurate soluton, lke the knetc constrants appled n [8]. VI. CONCLUSION Ths paper descrbes an applcaton of GA for moton capture. Snce the chromosome conssts of bnary genes, from the Schema Theorem [9], such problems can be solved usng GA. Our mplementaton showed that the proposed procedure was effectve. When mplemented on a 450MH Pentum III PC, the average operaton tme for the two motons s mnutes. However, each mage sequence only conssts of 6 frames. In practce, an mage sequence would consst of many frames. Subsequently, the number of parameters to be tuned by the GA s huge. Extenson of the proposed approach for handlng long mage sequence s under consderaton. ACKNOWLEDGEMENT The project s partally supported by the CUHK Drect Grant 01/0. REFERENCE [1] C.K. Chow, H.T. Tsu, and T. Lee, Optmaton on Unbounded Soluton Space usng Dynamc Genetc Algorthms, n Proc. IJCNN 001, Washngton, July 001, Vol. 4, pp.349-354. [] C. Pan and S. Ma, 3D Moton Estmaton of Human By Genetc Algorthms, n Proc. 15 th Internatonal Conference on Pattern Recognton, Vol. 1, pp. 159-163, 000. [3] R. J. Holt, A. N. Netraval, T. S. Huang, R. J. Qan, Determnng Artculated Moton from Perspectve Vews: A Decomposton Approach, n Proc. of the IEEE Workshop on Moton of Non-Rgd and Artculated Objects, pp. 16-137, 1994. [4] C. J. Taylor, Reconstructon of Artculated Objects from Pont Correspondences n a Sngle Uncalbrated Image, n Proc. of IEEE Conference on Computer Vson and Pattern Recognton, Vol.1, pp. 677-368, 000. [5] Y. Zhuang, X. Lu and Y. Pan, Vdeo Moton Capture Usng Feature Trackng and Skeleton Reconstructon, 1999 Internatonal Conference on Image Processng, Vol. 4, pp. 3-36, 1999. [6] S. Moe, L. C. Ta and P. Gerard, Vrtual Vew Generaton for 3D Dgtal Vdeo, IEEE Multmeda, Vol. 4, Issue 1, pp.18-6, Jan-March 1997. [7] L. Herda, P. Fua, R. Plankers, R. Boulc and D. Thalmann, Skeleton- Based Moton Capture for Robust Reconstructon of Human Moton, n Proc. on Computer Anmaton, pp. 77-83, 000. [8] K. Yamane and Y. Nakamura, Dynamcs Flter Concept and Implementaton of On-Lne Moton Generator and Human Fgures, n Proc. of IEEE Internatonal Conference on Robotcs & Automaton, pp. 688-695, 000. [9] D.E. Goldberg, Genetc Algorthms n Search Optmaton & Machne Learnng. New York: Addson-Wesley, 1989 Frame 1 Frame Frame 3 Frame 4 Frame 5 Frame 6 Fg. 8 The frst real mage sequence. 0-7803-78-4/0/$10.00 00 IEEE
Fg. 9 Estmated moton of the frst mage sequence Frame 1 Frame Frame 3 Frame 4 Frame 5 Frame 6 Fg. 10 Image sequence of snger walkng onto stage. Fg. 11 Estmated moton of the second mage sequence 0-7803-78-4/0/$10.00 00 IEEE