The Online Gait Measurement for the Audience-Participant Digital Entertainment

Transcription

1 The Onlne Gat Measurement for the Audence-Partcpant Dgtal Entertanment Mayu Okumura 1, Yasush Makhara 1, Shnsuke Nakamura 2, Shgeo Morshma 2, and Yasush Yag 1 1 Osaka Unversty, {okumura, makhara, yag}@am.sanken.osaka-u.ac.jp 2 Waseda Unversty, {nakamura, morshma}@mlab.phys.waseda.ac.jp Abstract. Ths paper presents a method to measure onlne the gat features from the gat slhouette mages and reflect the gat features to CG characters for an audence-partcpaton dgtal entertanment. Frst, both statc and dynamc gat features are extracted from the slhouette mages captured by the onlne gat measurement system wth two cameras and a chroma-key background. Then, Standard Gat Models (SGMs) wth varous types of gat features are constructed and stored, whch are composed of a par of CG characters renderng parameters and syntheszed slhouette mages. Fnally, blend ratos of the SGMs are estmated to mnmze gat feature errors between the blended model and the onlne measurement. In an experment, a gat database wth 100 subjects s used for gat feature analyss and t s confrmed that the measured gat features are reflected to the CG character effectually. 1 Introducton Recently, audence-partcpant dgtal entertanment has ganed more attenton, where the ndvdual features of partcpants or users are reflected to computer games and Computer Graphc (CG) cnemas. In EXPO 2005 AICHI JAPAN[1], the Future Cast System (FCS) [2] n the Mtsu-Toshba pavlon have been performed as a one of large-scale audence-partcpant dgtal entertanments. The system captures an audence s face shape and texture onlne, and a CG character s face n the CG cnema s replaced by the captured audence s face. In addton, as an evolutonal verson of the FCS, Dve Into the Move (DIM) Project [3] tres to reflect not only the ndvdual features of face shape and texture but also those of voce, facal expresson, face skn, body type, and gat. Among these, body type and gat are expected to attract more attenton of the audence because there are reflected to the whole body of the CG character. For the purpose of gat measurement, acceleraton sensors [4] and Moton Capture (MoCap) systems have been wdely used. These systems are, however, not sutable for an onlne gat measurement system because t takes much tme for the audences to wear the acceleraton sensors or MoCap markers. In a computer vson-based gat analyss area, both model-based and appearancebased approaches [5][6] were proposed, whch can measure gat feature wthout any wearable sensors or attached markers.

2 In the model-based methods, a human body s expressed as artculated lnks or generc cylnders and they are ft to the captured mage to obtan both statc features lke lnk length and dynamc features lke jont angle sequence. Although these features can be used for the gat feature measurement, the method s not sutable for onlne measurement because of hgh computatonal cost and dffcultes of model fttng. The appearance-based methods extract gat features drectly from the captured mages wthout troublesome model fttng. In general, extract features are composte of both statc and dynamc components. Although the composte features are stll useful for gat dentfcaton, they are not sutable for separate measurement of statc and dynamc components. Therefore, we propose a method of onlne measurement of ntutve statc and dynamc gat features from the slhouette sequences and also a method of gat feature reflecton to the CG character n the CG cnema. Sde-vew and frontvew cameras capture mage sequences of target subject s straght walk, and Gat Slhouette Volume (GSV) s constructed by slhouette extracton, slhouette sze normalzaton, and regstraton. Then, the both statc and dynamc components are measured separately from the GSV. Because the proposed method extracts the gat features drectly from the slhouette sequence wthout model fttng, ts computatonal cost get much lower than that of the model-based methods, whch enables onlne measurement of the audence s gat. 2 The Gat Database 2.1 Mult-vew Treadmll Gat Database For a prelmnary gat analyss, we have constructed a mult-vew gat capturng system. Our system conssts manly of a treadmll, 25 synchronous cameras (2 layers of 12 surroundng cameras and an overhead-vew camera usng a mrror), and sx screens surroundng the treadmll for chroma-key background. The frame rate and resoluton of all the cameras are set to 60 fps and VGA, respectvely. Our database ncludes 454 subjects (271 male, 183 female), wth ages rangng from 4 to 75 years old, (see Fg. 1(a) for dstrbuton). Subjects were collected by open recrutment and sgned a statement of consent for the usage to research purposes, whch s ssued by Yag laboratory at ISIR (The Insttute of Scentfc and Industral Research) n Osaka Unversty. After walkng practce on the treadmll, subjects walked at 4 km/h or slower f necessary for chldren and the elderly. They wore standard clothng (long-sleeved shrts and long pants, otherwse ther own casual clothes). 2.2 Large-scale Ground Gat Database The onlne gat measurement system contans front- and sde-vew cameras and lght green chroma-key background lt wth LED lghts. The length of walkng

3 (a) Mult-vew Treadmll Gat Database (b) Large-scale Ground Gat Database Fg. 1. The dstrbuton of subjects gender and age. 3m LED 4m 1m chroma-key back 3m chroma-key back front camera LED deceleraton capturng acceleraton area area area 3.5m walkng end walkng start sde camera (a) Overhead vew LED LED board (b) Sde vew Fg. 2. The outlne of the onlne gat feature measurement system course s decded so as to capture more than two stable gat cycles 3. Assumng the step s 1.0 m at most, the necessary length for capturng s 4.0 m, and addtonal 3.0 m ntervals are set up for the suffcent acceleraton and deceleraton respectvely. As a result, the total length s 11.0 m as shown n Fg. 2. The sdevew camera s set 3.5m far from the subjects and the front-vew camera s set so as to capture the whole body of the subjects. Vstors at the outreach actvty of DIM project 4 experenced the onlne gat measurement system and they sgned a statement of consent for the usage to research purposes, whch s ssued by the DIM project. As a result, the database ncludes 1610 subjects (830 male, 780 female), wth ages rangng from 3 to 94 years old (see Fg. 1(b) for dstrbuton). 3 A gat cycle contans two steps of left and rght legs. 4 Natonal museum of emergng scence and nnovaton durng 23rd to 25th March n 2009.

4 3 Gat Feature Measurement 3.1 GSV Constructon The frst step n gat feature measurement s the constructon of a spatotemporal Gat Slhouette Volume (GSV). Frst, gat slhouettes are extracted by background subtracton and a slhouette mage s defned by a bnary mage whose pxel value s 1 f t s n the slhouette and s 0 otherwse. Second, the heght and center values of the slhouette regon are obtaned for each frame. Thrd, the slhouette s scaled so that the heght s a pre-determned sze, whlst mantanng the aspect rato. In ths paper, the sze s set to a heght of h g = 60 pxels and a wdth of w g = 40 pxels. Fourth, each slhouette s regstered such that ts center corresponds to the mage center. Fnally, a spato-temporal GSV s produced by stackng the slhouettes on the temporal axs. Let f(x, y, n) be a slhouette value of the GSV at poston (x, y) of the nth frame. 3.2 Estmaton of Key Gat Phase The next step s estmaton of two types of key gat phases: Sngle Support Phase (SSP) and Double Support Phase (DSP) where subject s legs and arms are the closest and the most spraddle respectvely. The SSP and DSP are estmated as local mnmum and maxmum of the second-order moment around the central vertcal axs of the GSV n the half gat cycle. The second-order moment at nth frame wthn the vertcal range of [h t, h b ] s defned as S [ht,h b ](n) = h b y=h t w g 1 (x x c ) 2 f(x, y, n). (1) x=0 Where x c s the horzontal poston of the central vertcal axs. Because the SSP and DSP occur once per a half gat cycle alternately, those at th half gat cycle are obtaned as follows. n SSP = arg mn n DSP j = { 1 n [n DSP j,n DSP j +g p /2] (n DSP 1 (n DSP 1 = arg { max n [n SSP k 1 k = < n SSP,n SSP k +g p /2] (n SSP 1 (n SSP 1 S [ht,h b ](n) (2) 1 ) n SSP 1 ) S [ht,h b ](n) (3) < ndsp 1 ) ndsp 1 ), where n SSP 1 and n DSP 1 are ntalzed to zero, and gat perod g p s detected by maxmzng the normalzed autocorrelaton of the GSV for the temporal axs. In the followng secton, N SSP and N DSP represent the number of the SSP and DSP respectvely. Note that SSP and DSP focused on arms, legs, and entre

5 (a) Body (b) Leg (c) Arm Fg. 3. Example of GSV Fg. 4. SSP (left) and DSP (rght) body are obtaned by defnng the vertcal range [h t, h b ] approprately. In ths paper, the ranges of arms and legs are defned as [0.33h g, 0.55h g ] and [0.55h g, h g ] respectvely. Fgure 4 shows the result of the estmaton of SSP and DSP. 3.3 Measurement of the Statc Features Statc features contans the heght, the wast wdth, and the wast bulge. When the statc features are measured, GSV at SSP s used to reduce the nfluence of the swng arm and the bend of the legs. Frst, the number of the slhouette pxels at heght y of the sde and front GSV are defned as the wdth w(y) and the bulge b(y) respectvely as shown n Fg. 5. Then, ther averages W GSV and B GSV wthn the wast range of heghts [y W t, y W b ] and [y Bt, y Bb ] are calculated respectvely. W GSV = yw b ybb y=y W t w(y) (y W b y W t + 1), D y=y GSV = Bt b(y) (y Bb y Bt + 1) (4) Then, gven the heght on the orgnal sze mage H p, the wdth and the bulge on the mage W p and B p are obtaned as p = H p h g, W p = pw GSV, B p = pb GSV. (5) Fnally, gven the dstance l from the camera to the subject and the focal length f of the camera, the real heght H, the wdth W, and the bulge B are H = H p l f, W = W l p f, B = B l p f. (6) For statstcal analyss of the statc features, 100 subjects were chosen at random from the mult-vew treadmll gat database and front- and left sde-vew cameras were used for measurement. Fgure 6 shows the relaton between the measured heght and the questonnare result range of the heght. The measured heghts almost dstrbuted wthn the questonnare result range. The measured heghts of some short subjects (chldren) are, however, out of lower bound of the questonnare result. These errors may result from self-enumeraton errors due to the rapd growth rate of the chldren. Fgure 7 shows the relaton between the measured bulge and the questonnare result of the weght and t ndcates that the bulge correlates wth the weght to some extent. For example, the bulges

6 of lght Subject A and heavy Subject B are small and large as shown n Fg. 7 respectvely. As an exceptonal example, the bulge of lght Subject C becomes large by mstake because he/she wears a down jacket. 3.4 Measurement of the Dynamc Features Step Sde-vew slhouettes are used for step estmaton. Frst, walkng speed v s obtaned by dstance between slhouette postons at the frst and the last frames and elapsed tme. Then, the averaged steps length are obtaned by multplyng the walkng speed v and the gat cycle g p. Arm swng The sde-vew GSV from SSP to DSP s used for arm swng measurement. Frst, the body front and back boundary lne (let them be l f and l b respectvely) are extracted from a GSV at SSP, and then front and back arm swng canddate areas R Af and R Ab are set respectvely as shown n Fg. 8. Next, the slhouette sweep mage F (x, y) are calculated for th nterval from SSP to the next DSP as F (x, y) = { 0 1 else n DSP f(x, y, n) = 0 n=n SSP, j = { + 1 (n SSP (n SSP > n DSP ) < n DSP ). (7) Fnally, the front and back arm swng areas are obtaned as areas of the swept pxels of F (x, y) n the R Af and R Ab respectvely. A f = (x,y) R Af F (x, y), A b = (x,y) R Ab F (x, y) (8) For statstcal analyss of the arm swng, the same 100 subjects as the statc feature analyss are used. Fgure 10 shows the result of the measured the front swng arm area. We can see that the measured arm swngs are wdely dstrbuted, that s, they are useful cues for ndvdual feature reflecton to the CG character. For example, degrees of the arm swng for Subject A (small), B (mddle), and C (large) can be confrmed n the correspondng GSV at DSP on the graph. In addton, though the asymmetry of the arm swng s not so outstandng, some subjects have the asymmetry such as Subject D. y = y y = w y y Wt Wb y = y = b y y Bt y Bb (a) Wdth (b) Bulge Fg. 5. Measurement of the body sze

7 Measured heght[cm] heght Lower n enquete Upper n enquete SubjectA SubjectB SubjectC Questonnare result of the heght [cm] Measured bulge [cm] SubjectC SubjectB SubjectA Questonnare result of the weght [kg] Fg. 6. The measured heght and the questonnare result Fg. 7. The measured bulge and the questonnare result of the weght front arm object area R Af y = y At back arm object area R Ab y = y Ab Fg. 8. Front lne(red) Bacl lne(green) Fg. 9. Swng srm area Red:Front swng, Blue:Back swng Stoop We propose two methods of stoop measurement: slope-based and curvaturebased methods. In the both methods, a sde-vew GSV at a SSP s used to reduce the nfluence of the arm swng, and a back contour s extracted from the GSV. Then, the slope of the back lne s obtaned by fttng the lne l b to the back contour, and also the curvature s obtaned as maxmum k-curvatures of the back contour (k s set to 8 emprcally). For statstcal analyss of the stoop, the same 100 subjects are used. Fgure 11 shows the result of the measured the stoop. By measurng both the slope and the curvature of the back contour, varous knds of the stoops are measured such as large slope and small curvature (e.g. SubjectA), small slope and large curvature (e.g. Subject C), and large slope and curvature (e.g. Subject C). 4 Reflectng the gat features to the CG character 4.1 Standard Gat Models Generally speakng, the parameters of lnk length and jont angle sequence are needed n order to render the CG characters, whch are dfferent from the appearance-based gat features n the proposed framework. Therefore we should defne a mappng from the appearance-based gat features to the renderng parameters. In addton, the mapped CG characters moton should le n the natural gat doman. To satsfy these two requrements, we ntroduce Standard Gat Models (SGM): MoCap data of varous types of walkng sequence of a tranng subject, and defne the mappng problem as blend rato estmaton of the SGM.

8 4.2 Blend Rato Estmaton Frst, the appearance-based gat feature of the SGMs are extracted n the same way as the onlne measurement from a walkng slhouette sequence syntheszed wth ther renderng parameters. Then, let s assume the gat feature vector ˆv of the blended model s approxmated as a weghted lnear sum of those of the SGMs. n ˆv α v = V α (9) =1 where v and α represent the m dmensonal gat feature vector and the blend rato of the th SGM respectvely, and n s the number of the SGMs. Then, the blend rato α s estmated so as to mnmze errors between the gat features ˆv of the blended model and the onlne measured gat features v (call t an nput vector). The mnmzaton problem s formulated as the followng convex quadratc programmng. α CQP = arg mn (V α α v)t (V α v) (10) n subject to α = 1, 0 α 1 =1 The mnmzaton problems s solved wth the actve set method. Moreover, when the number of the SGMs n s larger than the dmenson of the gat features m, the soluton to eq. (10) s ndetermnate. On the other hand, eq. (9) s the approxmate expresson because the mappng from the renderng parameter doman to the appearance-based gat feature doman s generally nonlnear. Therefore, t s desrable to choose closer gat features to the nput feature. Thus, another cost functon s defned as an nner product of the blend rato α and the cost weght vector w of the Eucldean dstance from the features of each Left arm sweep area AL 900 Symmetry lne SubjectD AL AR SubjectC 400 SubjectB 300 SubjectA Rght arm sweep area AR Slope slope S -4 curvature L -6 SubjectA slope L curvature S SubjectB slope L curvature L Curvature SubjectC Fg. 10. The measured front arm swng area and the asymmetry Fg. 11. The measured stoop

9 Table 1. The SGMs and nput gat features Table 2. The blendng rato of the SGMs SGM Arm Swng Step Stoop Arm Swng L Arm Swng S Step L Step S Stoop Recurvature Average Subject Arm Swng Step Stoop A B C SGM \Subject A B C Arm Swng L Arm Swng S Step L Step S Stoop Recurvature Average SGMs to the nput feature, whch s defned as w = [ v 1 v,..., v n v ] T. Gven one of the soluton to eq. (10) as α CQP and the resultant blended model v = V α CQP, the mnmzaton problem can be wrtten as a lnear programmng. α = arg mn w T α (11) α n subject to V α = v, α = 1, 0 α 1 Fnally, ths lnear programmng s solved wth the smplex method. =1 4.3 Expermental Results In ths experment, three gat features: arm swng, step, and stoop are reflected to the CG characters, and Seven SGMs and three test subjects are used as shown n Tab. 1 respectvely. Note that the gat features are z-normalzed so as to adjust scales among them. The resultant blend ratos are shown n Tab. 2. For example, Subject A wth large arm swng gans the largest blend rato of the Arm Swng L and t results n the syntheszed slhouette of the blended model wth large arm swng (Fg. 12). The gat features of Subject B (large step and small arm swng) and C (large stoop and large arm swng) are also successfully reflected n both terms of the blend ratos and the syntheszed slhouettes as shown n Tab. 2 and Fg Concluson Ths paper presents a method to measure onlne the statc and dynamc gat features separately from the gat slhouette mages. The suffcent dstrbuton of the gat features were observed n the statstcal analyss wth large-scale gat

10 SubjectA The result of SubjectA SubjectB The result of SubjectC SubjectC The result of SubjectC Fg. 12. The synthetc result database. Moreover a method of gat feature reflecton to CG characters was proposed wth the blend rato estmaton of the SGMs. The expermental results show the gat feature lke arm swng, step, and stoop are effectvely reflected to the syntheszed blended model. Acknowledgements Ths work s supported by the Specal Coordnaton Funds for Promotng Scence and Technology of Mnstry of Educaton, Culture, Sports, Scence and Technology, and Grant-n-Ad for Young Scentsts (B) References 1. : EXPO 2005 AICHI JAPAN 2. : The Future Cast System 3. : Dve nto the move 4. Gafurov, D., Helkala, K., Sondrol, T.: Bometrc gat authentcaton usng accelerometer sensor. Journal of Computer 1(7) (Aprl 2006) Bobck, A., Johnson, A.: Gat recognton usng statc actvty-specfc parameters. In: Proceedngs of IEEE Conference on Computer Vson and Pattern Recognton. Volume 1. (2001) Kale, A., Cuntoor, N., Yengnanarayana, B., Rajagopalan, A., Chellappa, R.: Gat analyss for human dentfcaton. In: Audo- and Vdeo-Based Bometrc Person Authentcaton. Volume 3. (Aprl 2003)