Motion-Capture-Based Avatar Control Framework in Third-Person View Virtual Environments

Transcription

1 ACM SIGCHI Internatonal Conference on Advances n Computer Entertanment Technology 26 (ACE 26), Hollywood, U.S.A., June 26. Moton-Capture-Based Avatar Control Framework n Thrd-Person Vew Vrtual Envronments Masak Oshta Kyushu Insttute of Technology 68-4 Kawazu, Izuka, Fukuoka, Japan oshta@ces.kyutech.ac.jp ABSTRACT Ths paper presents a moton-capture-based control framework for thrd-person vew vrtual realty applcatons. Usng moton capture devces, a user can drectly control the full body moton of an avatar n vrtual envronments. In addton, usng a thrdperson vew, n whch the user watches hmself as an avatar on the screen, the user can sense hs own movements and nteractons wth other characters and objects vsually. However, there are stll a few fundamental problems. Frst, t s dffcult to realze physcal nteractons from the envronment to the avatar. Second, t s also dffcult for the user to walk around vrtual envronments because the moton capture area s very small compared to the vrtual envronments. Ths paper proposes a novel framework to solve these problems. We propose a trackng control framework n whch the avatar s controlled so as to track nput moton from a moton capture devce as well as system generated moton. When an mpact s appled to the avatar, the system fnds an approprate reactve moton and controls the weghts of two trackng controllers n order to realze realstc and also controllable reactons. In addton, when the user walks n poston, the system generates a walkng moton for the controller to track. The walkng speed and turn angle are also controlled through the user s walkng gestures. Usng our framework, the system generates seamless transtons between user controlled motons and system generated motons. In ths paper, we also ntroduce a prototype applcaton ncludng a smplfed optcal moton capture system. Categores and Subject Descrptors I.3.6 [Computer Graphcs]: Methodology and Technques - Interacton Technques; I.3.7 [Computer Graphcs]: Threedmensonal Graphcs and Realsm - Anmaton; I.3.7 [Computer Graphcs]: Three-dmensonal Graphcs and Realsm - Vrtual Realty. Keywords Avatar, Moton Control, Interface, Moton Capture, Vrtual Realty.. INTRODUCTION Moton capture technology has been used n many areas, ncludng move makng, moton analyss, and computer games. Although the most current man uses are for offlne processes, moton capture s expected to become a major control devce for computer entertanment. Usng moton capture devces, a user can drectly control the full body moton of a character n vrtual envronments. For such knds of applcatons, we beleve that the thrd-person vew s most sutable (Fgure ). Most current vrtual realty applcatons employ the frst-person vew wth whch the users can feel as f they are n the vrtual envronments. However, ths vew s not qute suted for some knds of applcatons, such as fghtng games n whch the movements of the characters are mportant. In the real world, we can feel the movements of our own body and physcal nteractons wth other people and objects through our senses. However, we cannot do such thngs n current vrtual realty systems snce we can only obtan vsual nformaton or very lmted physcal nformaton from some haptc devce. By usng the thrd-person vew, however, n whch the user watches hmself as an avatar, the user can sense such movements and nteractons wth other objects vsually. Ths s the reason why we consder the thrd-person vew to be more sutable for some entertanment applcatons. However, there are stll a few fundamental ssues to be dealt wth n order to realze such knds of applcatons. One problem s that t s dffcult to (a) Frst-Person Vew (HMD) (b) Frst-Person Vew (c) Thrd-Person Vew Head Mount Dsplay Large Screen or CAVE, etc Large Screen or CAVE, etc Character Character Avatar User + Mocap Devce User + Mocap Devce User + Mocap Devce Fgure. Frst-person vew (a) (b) and thrd person vew (c).

2 Moton Capture Devces Moton Database Input Moton Generated Moton (Reacton, Walkng, etc) Trackng Control (Input Moton) Normal Condton Trackng Control (Generated Moton and Input Moton) Specal Condton Avatar State Fgure 4. Pctures of the prototype system. Fgure 2. Trackng control approach. Under normal condtons the avatar tracks a captured moton. Under condtons n whch multple system generated motons are requred, the avatar can track each of them seamlessly. (a) Normal Movng (b) Walkng Interface Fgure 3. Walkng nterface. Bascally the user can freely move around n the mocap area. If the user performs a walkng gesture wthout horzontal movement, the system then generates a walkng moton to be tracked. realze physcal nteracton from the envronment to the avatar because the avatar s controlled based on nput from a moton capture devce and because physcal nfluences cannot be appled to the user n the real world. Another problem s that t s dffcult for the user to walk around vrtual envronments because moton capture areas are very small compared to the sze of vrtual envronments. Some exstng vrtual realty systems use an addtonal devce such as a gamepad or a 3D pontng nterface for navgaton, whch can be very dffcult to use n moton-capturebased systems. Ths paper presents a moton-capture-based control framework that s sutable for thrd-person vew vrtual realty applcatons. Our method solves the avatar control problems descrbed above. In our system, an avatar s controlled so as to track an nput moton from a moton capture devce nstead of just playng the nput moton (Fgure 2). When a physcal nteracton s appled to the avatar (e.g., a collson wth other characters), the system automatcally generates a reactve moton (e.g., fallng down to the ground), and the avatar then tracks the reactve moton. Even durng the reactve moton, the user can control the avatar slghtly. Wth ths trackng approach, the system can generate seamless transtons between user controlled motons and system generated reactve motons. The user can experence the physcal nteractons vsually through the reactons of the avatar n the thrd-person-vew vrtual envronments wthout any specal forcefeedback devces. In addton, we propose a walkng-gesturebased navgaton nterface (Fgure 3). In ths nterface, when the user walks n poston, n other words, performs a walkng moton n one place wthout movng horzontally, the system generates a walkng moton. The walkng moton s controlled through the user s walkng gestures. For example, f the user moves hs legs and arms quckly, the system wll generate runnng moton based on the moton speed. The user can control both the speed and drecton of the moton very easly and ntutvely. We developed a prototype of ths system ncludng an optcal moton capture system (Fgure 4). The rest of ths paper s organzed as follows. Secton 2 ntroduces related work and dscuses the advantages of our work. Secton 3 provdes an overvew of the proposed system framework and the basc data representatons. Sectons 4, 5, and 6 explan the system s trackng control, reactve moton control, and walkng control, respectvely. Secton 7 then ntroduces the expermental results and evaluates the proposed method. Fnally, Secton 8 concludes ths paper and descrbes future drectons of research. 2. RELATED WORK A number of studes have nvestgated moton-capture-based character control [4]. However, n most of these systems, the nput moton s drectly used to control the character s moton. Unfortunately, physcal nteracton between the character and the envronment s therefore not consdered. Hasegawa et al. [4] proposed a system smlar to ours. They also used the thrd-person vew and a trackng control system. However, they always tracked the nput moton and dd not generate automatc reactons. Therefore, the character may move unnaturally when a large mpact s appled to the character. Hämälänen et al. [3] used a profle thrd-person vew n a martal arts combat game. They also used vdeo mages of the users captured n real-tme wth a camera nstead of a combnaton of a moton capture system and computer generated human fgure. Whle ths s an nterestng method, because of ts approach the avatar cannot react to mpacts from ts enemes. Ther system also addressed the moton area problem and exaggerated the horzontal movements of the users by smply scalng the horzontal poston of the user s mage.

3 Although ths approach expands the user s reach, the users stll cannot move around wde vrtual envronments. Generatng realstc reactve moton to a gven physcal mpact s one of the most dffcult challenges n computer anmaton. Snce the human control mechansm s very complex and has yet to be well modeled, t s very dffcult to generate realstc reactons. Some researchers have tackled ths problem by usng a trackng control approach and/or a moton database n a way smlar to our methods. A method proposed by Zordan et al. [8] searched for approprate reactve motons from a database based on the results of physcs-based smulatons after mpacts were appled to a vrtual character, and a method proposed by Komura et al. [7] searched for reactve motons based on the amount of mpact that generated resultng reactons usng a trackng control system that took nto account the searched moton data and momentum varatons durng the reactve motons. Oshta et al. [2] generated reactve motons that dd not use any reacton data and nstead used a heurstc dynamc control that altered the output of smple trackng control whle takng nto account balance and stress controls. These methods were desgned for the generaton of realstc reactons wthout any user control. On the other hand, the control method proposed n ths paper controls the avatar based on both nput moton from the moton capture devce and searched reactve moton from a database usng an approach smlar to those n [7] and [8]. Although we currently use searched moton from the database wthout any modfcaton as the target moton of the trackng control, more advanced methods such as those descrbed n [7] and [8] can also be used wth our system. There are many control nterfaces that use an ntutve nput devce dfferent from moton capture [6]. However, such nterfaces do not consder nteractons between the character and the envronment, and t s also dffcult to use them n our system as addtonal nterfaces for navgaton, etc., as our system needs the full body control of the user. There are many approaches avalable for moton recognton [2]. However, the exstng methods smply recognze what knd of moton s beng executed. In our system, therefore, because walkng moton should be controlled through the user s movements, we employ a fuzzy based state detecton and state tree for computng walkng speed. 3. SYSTEM FRAMEWORK An overvew of the proposed control framework s shown n Fgure 6. The avatar state s updated based on the nput moton from the moton capture n each step of the smulaton. A trackng controller computes control acceleratons based on the current state and the gven moton usng a PD servo so that the avatar tracks the nput moton. When a large mpact s appled to the avatar or when the user performs a walkng gesture, the coordnator generates a target moton and controls the blendng weghts of the output of the trackng controllers. The ntegrator updates the avatar state based on the acceleratons and the blendng weghts. The ntegrator also changes the avatar state when t colldes wth another character or an object. The avatar bascally tracks the nput moton from the moton capture devce. When a large mpact s appled to the avatar, t manly tracks a reactve moton that s searched for from a reacton database. When the user performs a walkng gesture, a Moton Database Moton Capture Coordnator Input Moton Control Weghts Generated Moton Trackng Control Trackng Control walkng moton s generated by usng a moton blendng technque, and the avatar accordngly tracks the walkng moton. Even durng reactve or walkng moton, dependng on the condtons the avatar s stll controllable through the nput moton. The detals of ths process are explaned n the followng sectons. The coordnator determnes the blendng weghts of the output of the two trackng controllers. In our system, the trackng controllers compute acceleratons based on the current state of the avatar and the target moton. Although n standard physcs-based smulaton systems [5][8] a trackng controller computes jont torques and a smulator then computes the resultng jont acceleratons from the jont torques, we chose to use angular acceleratons for controllablty. In torque based control, the gan parameters of the trackng controller should be carefully tuned for the ndvdual characters and target motons. They produce unstable moton especally when the system mxes multple outputs from a number of trackng controllers. Therefore, we decded to control angular acceleratons n a way smlar to that descrbed n [2] or [5]. Although torque based controls are generally thought to produce more physcally vald motons, ths s not necessary n our system snce the nput moton capture data and reactve motons n the database are based on human moton and therefore are consdered to be already physcally vald. Thus we use trackng control for blendng multple controls but not for producng physcally correct motons. 3. Human Model Vrtual characters, ncludng avatars and autonomous characters, are modeled as artculated fgures. The human model of our prototype system has 2 segments and 9 jonts (Fgure 5), whch s a typcal model used for computer games. The posture of a human model s represented as follows. M = p, q, q, L, q, L, q. () { } root root n Ths conssts of the poston p root Control Acceleratons Control Acceleratons Fgure 6. System overvew. and orentaton q root Fgure 5. Human fgure model. Integraton Avatar State of the

4 pelvs (root) and the rotaton of all jonts ( = ) q K n, where n s the number of jonts. In our method, pelvs orentaton and jont rotaton are represented by rotaton vectors. The orentaton of the vector represents the rotaton axs n the local coordnates system of the parent segment, and the length of the vector represents the rotaton angle around the rotaton axs. The veloctes and acceleratons of a human model are also represented n the same way as that descrbed n (). In our framework, the avatar state conssts of posture M and velocty M &. The trackng controller outputs angular acceleratons M &&. Changes n the posture and velocty of the avatar are computed from the angular acceleratons usng numercal ntegraton on each smulaton step. When the avatar colldes wth other characters or objects, the changes of velocty M & are computed by solvng a lnear system [9]. 3.2 Optcal Moton Capture System Any knd of moton capture system can be used wth our framework. For our prototype system, we have developed a smplfed optcal moton capture system. Current moton capture systems requre some markers or sensors to be attached to the user. Although markerless moton capture [] s desrable for our target applcatons, t s stll dffcult to capture human moton precsely and n real-tme wthout usng markers or sensors. Currently the most popular moton capture systems are optcal-based systems. However, they are very expensve and requre a lot of tme to set up the requred number of markers. Therefore, we have developed a smplfed optcal system whch uses a relatvely small number of markers. We use three OptTrack FLEX3 nfrared cameras [] from Natural Pont. Our system acqures only the poston or both the poston and orentaton of the prmary body parts. Inverse knematcs [4] s then used to compute the full body posture. For example, we computed the jont rotatons of an arm from the poston and orentaton of the shoulder and the poston of the hand. The orentaton of the hand can be estmated from the elbow angle. In addton, the rotatons of the elbow around the axs from the shoulder to the hand can be determned from an expermentally determned constant value [7]. 4. AVATAR CONTROL As explaned n the prevous secton, we use a PD controller to compute trackng acceleratons based on the current state of the avatar and the target state from the target moton. ( target, ) d( target, ) q&& = k q q + q& q&, (2) where q&& s the output jont rotatonal acceleraton of the -th jont, q, q& are the current jont rotaton and rotatonal velocty, respectvely, and qtarget,, q& target, are the target rotaton and rotatonal velocty, respectvely, that are acqured from the target moton. The gan parameter k and dampng parameter d are set manually. Snce we use angular acceleratons nstead of jont torques, we can use the same k and d for all jonts. To control jont rotatons n the PD controllers, we treat rotatons and rotatonal veloctes as rotaton vectors. In addton to jont rotatons, the pelvs and end-effectors that come nto contact wth the ground are also controlled. The system controls the poston and orentaton of the body parts and apples nverse knematcs to correct the jont rotatons on the lmb. For the nverse knematcs, we use the same analytcal method [4] used n our optcal moton system (Secton 3.2). The poston of the pelvs (end-effectors) p s controlled as follows. j ( ) d ( ) p&& = k p p + p& p&. (3) j pelvs target, j j pelvs target, j j The orentaton of the pelvs s controlled n the same way used for jont control shown n equaton (2). 5. REACTIVE MOTIONS FOR IMPACTS To realze realstc reacton to varous knds of mpacts, we ntroduced a few types of control schemes. The controllable factors n the proposed framework nclude the choce of reactve moton and the weghts for the two trackng controls of nput moton and reactve moton. Our method s based on the observaton of real human movements, takng nto consderaton the fact that reactve human motons can be dvded nto three phases: actve control, actve control wth protectve steps, and passve control wth fallng down (Fgure 7). When an mpact s appled to a human, he frst attempts to mantan hs balance by movng hs arms and upper body (actve control phase). If the mpact s small, he can recover to a stable state. If the mpact s relatvely large and he cannot recover durng ths phase, protectve steps must be taken to keep from fallng down (actve control wth protectve steps phase). He may then recover after a few steps, or he may fall down. Whle fallng down, he can barely control hs body because the fallng moton s bascally caused by gravty (passve control phase). To sum up, there are three types of reactve motons dependng on the number of phases nvolved (Fgure 7 (a)-(c)). Our control scheme realzes these reactve motons. Frst, the type of reacton s determned based on the avatar s state after an mpact. Then, the weght values for the nput moton and reactve moton are controlled durng each phase based on the determned reacton type. Fgure 7 shows the trajectores of the weght values for each reacton type. When the changes of the avatar state are small after an mpact s appled to the avatar (Fgure 7 (a)), no reactve moton s used and only a small weght s used for the trackng control of the nput moton n order to realze the reacton of the actve control phase. When a large mpact s appled to the avatar, however, an approprate reactve moton s searched for from wthn the reacton database. Based on the chosen reactve moton, the system then determnes whether the avatar wll recover at the end of the moton (Fgure 7 (b)) or fall down (Fgure 7 (c)). At frst a smaller weght s gven to the trackng control of the nput moton n the same way as n the case of a small mpact. If the avatar s to recover, then the weght becomes larger durng the second phase and the avatar goes back to the nput moton. On the other hand, f the avatar s to fall down, then the weght becomes smaller durng the next phase whle the weght for the reactve moton becomes larger. Durng passve control, the weght for nput moton s set to and the avatar tracks the reactve moton.

5 (a) Small Impact Case (b) Mddle Impact Case (c) Large Impact Case Actve Control Actve Control wth Protectve Steps Passve Control Fgure 7. Reactve motons. We assume that reactve motons consst of three phases. Reactve motons are categorzed based on n whch phase the avatar recovers hs balance. The red and blue trajectores show the weghts of reactve moton and nput moton, respectvely. After the avatar falls down, the avatar s posture and the user s posture may dffer. If the avatar tracks the user s pose, an unnatural standng up moton s generated. In such a case, after a reactve moton has fnshed, a transton moton s executed to allow the avatar to reach a posture that s close to the user s posture. For ths reason the avatar agan reverts to the trackng control of the nput moton. The executon of such transton moton s explaned n Secton Reacton Database A number of reactve motons are stored n a database. When a large mpact s appled to an avatar and the avatar loses hs balance, the system searches for an approprate reactve moton from the database and uses t as a target moton for trackng control. To fnd the best reacton, each reacton moton has an ndex key whch s automatcally computed n advance. The ndex key contans the balance and posture states of the avatar at the tme when the reacton began. The drecton and speed at whch the fgure s fallng (balance state) have frst prorty here snce these values determne the drecton of the reacton. To take both the poston and velocty of the center of mass nto account, we used the dea of extrapolated center of mass (XcoM) proposed by Hof et al. [6] to evaluate the balance condtons of human movements. The XcoM s based on an nverted pendulum model. The poston of XcoM s computed by

6 p = p + l g p&, (4) XcoM CoM CoM where pcom, p& CoM are the poston and velocty of the center of mass, respectvely, l s the leg length, and g s the acceleraton of gravty. Though here we could use the poston and velocty of the center of mass as separate varables and evaluate them wth some arbtrary weghts, n ths case we would have to tune the most approprate set of weghts. Thus we decded to use an establshed predcton model nstead of gong through ths process. In addton to the center of mass (XcoM), the posture and veloctes of the avatar are also mportant factors when choosng a reactve moton. However, the use of the poston and velocty of all jonts or segments s redundant. Therefore, the postons and veloctes of only the end-effectors are consdered. Snce the veloctes of a human fgure tend to be large when the fgure s losng ts balance, we cannot gnore the affect of these veloctes. Based on the same concept of XcoM, we use equaton (4) for the end-effectors also n order to smultaneously take the postons and veloctes nto account. We call these values the extrapolated end-effectors (XEE), and they are represented n the local coordnates wth ther orgns n the root of the lmbs. Further, the front vector s the fgure orentaton, and the up vector s the world Y-axs. As a result, 5 dmensonal vectors are used for the ndex keys (Fgure 8). The ndex key of a reactve moton s computed n advance based on the character s state at the tme when the magntude of XcoX becomes largest just after the mpact s appled to the character. The system automatcally searches the tme around the specfed frst key tme wthn a certan tme range (.2 sec). 5.2 Searchng for Reactve Moton When an mpact s appled to the avatar, the system frst supposes that the mpact s small and smply starts to control the weght of the trackng control wthout any reacton data, as shown n Fgure 7 (a). Next, the balance condton (XcoM and XEEs) s computed at each frame. If the XcoM exceeds the support polygon, then the fgure s consdered to be startng to lose ts balance [6]. The reactve moton that has the closest ndex key to the balance condton s then used. The support polygon s computed from the faces of the fgure model that are n contact wth the ground usng a convex hull computaton []. When the fgure starts to lose ts balance, meanng that the XcoM moves outsde the support polygon, a reactve moton s searched for from wthn the database based on the dstance between the ndex key of each reacton data set and the search key computed from the state of the fgure at that partcular tme. The dstance of the -th data n the database s computed as E = w p p + w p p, (5) XcoM XcoM XcoM j j j where, pxcom p j are the poston of XcoM and the j-th endeffectors (hands, foots and head), respectvely, and w, XcoM w j are the weghts for XcoM and the end-effectors, respectvely. We currently use w XcoM = 5., w j =. n our mplementaton. Snce the XcoM dstance may be small at frst even when a large mpact s appled to the fgure, the system keeps computng the balance condton after the XcoM exceeds the support polygon durng a fxed duraton. Then the condtons of the largest balance values are used for searchng the reacton data. 5.3 Control of Weghts The weght values for the nput moton and the reactve moton are determned from the reacton type, the current phase, and the current tme, as shown n Fgure 7. For each reactve moton, we assgn keytmes that ndcate the respectve segment of each phase n advance. Durng the frst phase, the weghts change lnearly. Durng the second phase, the weghts change lnearly before a fxed duraton from the keytme. In addton to these lnear functons, the weghts for nput moton are changed based on the veloctes of the upper body of the nput moton. If only the weghts n Fgure 7 were used and the user dd not move hs upper body much, the produced moton would become less reactve compared to the orgnal reactve moton. To address ths, we multply the normalzed veloctes of the hands and head wth the weght functon when the sum of the veloctes s smaller than threshold v. w nput () t XEE(r_head XEE(r_hand) XEE(r_foot) XEE(l_hand) XcoM XEE(l_foot) Fgure 8. Index keys of a reactve moton. The approprate reactve moton s searched for based on the extrapolated Center of Mass (XcoM) and the Extended End Effectors (XEE). () ( vrght_hand vleft_hand vhead ) () ( v + v + v ) v f t f + + > v = (6) f t rght_hand left_hand head / otherwse. After the reacton type and reactve moton are determned, the system smply controls the weghts. After the executon of the reactve moton and transton moton (f requred) have fnshed, the system then goes back to normal control. 5.4 Transton to Input Moton As explaned n Secton 5, after the avatar falls down, the system executes a transton moton n order to prevent an unnatural transton. We assume that the user s always standng, and therefore a standng up moton s executed for transton when the avatar falls down. Further, though we can choose the approprate moton based on the nput state (the user s state) and the current state (the avatar s state), n our current mplementaton we smply assgn a standng up moton to each reactve moton manually.

7 Rght Leg Up Rght Leg Down Left Leg Up Left Leg Down Not Walkng Fgure 9. Walkng states. Whle walkng the avatar transts through the above four states. Rght Hand Rght Foot Pelvs Left Hand Left Foot Fgure. Veloctes used for detectng walkng gestures. 6. WALKING CONTROL The system generates walkng or runnng moton as a target moton for the trackng controllers when the user performs a walkng gesture (Fgure 3). In order to make the avatar start to walk, the user has to move hs legs and arms n a synchronzed way as we normally do durng walkng. Once a walkng moton starts, the avatar keeps walkng whle the user keeps movng hs legs. The user can move hs upper body freely after the walkng moton starts. For example, the avatar can wave ts hand or ht other character whle walkng. To realze ths knd of nterface, the system recognzes both the full body gesture (startng condton) and the lower body gesture (mantanng condton). In addton, the trackng weghts for the nput moton and walkng moton are controlled based on the recognton results. 6. Generaton of Walkng Motons Currently we use a moton blendng technque [3] for the generaton of walkng moton so that the speed and turn angle of walkng are controlled based on the user s moton. Of course, any other knd of moton generaton technque (e.g., moton graphs, dynamc smulatons, etc.) can also be used wth our nterface. To use moton blendng, we frst prepare example walkng motons. The keytmes are set to the example motons by hand. Ths keytme nformaton s essental to the synchronzaton of the example motons and to makng the moton blendng work. In addton, the speed and turn angles of each example moton are computed n advance. Usng ths nformaton, the blendng weghts for the example motons are computed when a set of walkng parameters (speed and turn angle) are gven. A resultng walkng moton s then computed by blendng the example motons wth the blendng weghts. More detals regardng the moton blendng method can be found n [3]. 6.2 Detecton of Walkng Gestures To control the parameters of walkng moton, n addton to whether or not the user s walkng, the speed and turn angle parameters are also requred. The walkng orentaton (turn angle) s easly computed from the orentaton of the user s pelvs. To compute the walkng speed, we dvde walkng moton nto four phases (rght leg up, rght leg down, left leg up, left leg down), as shown n Fgure 9, and detect n whch state the user s currently. Based on the state transton speed, the walkng speed s then estmated. The walkng state s determned based on the velocty of the pelvs, feet, and hands (Fgure ). We use a heurstc method based on fuzzy rules. To detect the full body walkng states, we use the velocty of the pelvs, feet, and hands. For example, durng the rght-leg-up state, the vertcal velocty of the rght foot v rght_foot y must be a large postve value. The velocty of the left hand v left_hand z must be frontward, and the velocty of the rght hand v rght_hand z must be backward. In addton, the horzontal velocty of the pelvs v pelvs x + z must be always a small value durng walkng. We evaluate the scores of these condtons based on a membershp functon (Fgure ), whch s represented by the average and dsperson. For example, the evaluaton ( to ) for the rght foot v rght_foot y s computed as follows. rght_foot rght_foot rght_foot rght_foot rght_foot rght_foot rght_foot ( vy a )/ d f v a < d y ey = (7) rght_foot rght_foot rght_foot f v a d. y Based on evaluatons computed n the same way, the probablty of the state s then computed by ( ) mn {,,, } y x+ z z z rght_leg_up full rght_foot pelvs rght_hand left_hand e = e e e e. (8) To detect the lower body walkng state, we need consder only the veloctes of the pelvs and feet. ( ) mn {,, } y y x z rght_leg_up lower rght_foot left_foot pelvs e = e e e +. (9) For the other three states, we use the same functons. rght_foot e y pelvs e x+ z left_hand e z rght_hand e z Fgure. Membershp functons for evaluatng the condtons of rght leg up.

8 rght_leg_up ( full) left _leg_up ( full) When the probabltes of e or e exceed a threshold, the avatar starts to walk. Whle walkng, f the rght_leg_ down ( lower) probablty of the next state ( e n the case of rght leg up, for example) exceeds the threshold, t then transts to the next state. When the walkng state transts from one state to the next, the velocty of the end-effectors crosses zero, and the state s not recognzed correctly n the nstance. To address ths, we wat for a fxed duraton untl a stop n the walkng moton. When the next state s detected, the walkng state then transts to the next state. The state transton speed s computed based on the prevous transton tmes. In addton, based on the state transton speed, the generc tme ( to ) of the walkng moton s also computed. 6.3 Control of Walkng Motons Durng walkng moton, the weght for the lower body ncludng the pelvs s set to. For the upper body, the weght s set to when ether the user moves hs arms n the walkng way or when the user doesn t move hs arms. On the other hand, a larger value s set to the upper body when the user moves hs arms n the same manner as gven n equaton (6). As a result, after a walkng moton starts, the user can control hs arm movements. 7. EXPERIMENTS AND DISCUSSION We developed the proposed framework and controllers as well as a prototype system n whch the user can control the avatar through a moton capture system and nteract wth an autonomous character. Fgure 4 shows some pctures of the prototype system. Currently, the motons of the character are ether randomly chosen from a moton set or are controlled by another user through a keyboard nterface. Although our optcal moton capture system s nfluenced by camera noses and the captured moton s somewhat unstable snce we use a heurstc labelng algorthm smlar to that descrbed n [7], the detecton of walkng gestures bascally works well. We have not experenced any errors, for example, n whch the avatar starts walkng based on a msjudgment. We dd, however, experence walkng motons that sometmes stopped contrary to the user s ntenton. Currently, f one walkng step s passed over due to moton capture nose or some other factor, the walkng moton stops even f the user mantans hs walkng moton. To avod such stuatons, we need to refne our method to be more stable. Another problem of the current walkng nterface s that t s dffcult to walk backward n our system. Ths s because f the user turns back he cannot look at the screen. Actually, ths dffculty of changng one s orentaton s a common problem of the thrd-person vew. Usng multple screens that show an dentcally rendered mage s one possble soluton [3]. Other problems related to usng the thrd-person vew nclude the dffcultly for users to see small objects or the faces of characters clearly n front of the avatar because of the camera dstance. To solve such problems, applcaton-dependent automatc camera postonng and orentaton control or dynamc swtchng vews between the frst-person vew and the thrdperson vew may be useful. In addton, we expect that the walkng-gesture-based nterface can be appled to other knds of actons, such as clmbng, swmmng, or usng props. However, snce some actons requre more complex gestures, a more generc and robust method may be necessary. Moreover, our algorthm for reactve motons works well. However, some users dd not recognze the dfferences between reactve motons and the effects of the user s movements. Currently, the user s movements durng reactve moton only affect the vsual appearance of the avatar but not the result of the moton. For example, no matter how the user tres to recover hs balance by movng hs arms, t does not affect the results of the moton (fallng or recoverng). If we can change the results of the motons based on the user s control, the reactve motons wll be more nterestng to the users. Moreover, currently we only handle nstant collsons between characters, and therefore the system cannot handle constant contact stuatons, such as wrestlng. To address these problems, we may need to ntroduce physcs-based algorthms nto the controllers. 8. CONCLUSION AND FUTURE WORK We have proposed a control framework for thrd-person vew vrtual envronments. We combne multple trackng controllers for both captured moton and system generated motons. Although the current prototype system s a very smple applcaton, t s very nterestng to be able to control an avatar through a moton capture system and to nteract wth a vrtual character. In the near future, t s expected that computer entertanment ncorporatng ths knd of control framework wll be avalable for us to play. 9. REFERENCES [] Cheung, G., Baker, S., Kanade, T. Shape-From-Slhouette of Artculated Objects and ts Use for Human Body Knematcs Estmaton and Moton Capture. In Proc. of Computer Vson and Pattern Recognton 23. [2] Emerng, L., Boulc, R., Thalmann, D. Lve Partcpant's Acton Recognton for Vrtual Realty Interactons. In Proc. of Pacfc Graphcs, 5-2, 997. [3] Hämälänen, P., Ilmonen, T., Höysnem, J., Lndholm, M., Nykänen, A. Martal Arts n Artfcal Realty, Proceedngs of ACM Conference on Human Factors n Computng Systems (CHI 5), 78-79, 25. [4] Hasegawa, S., Ishkawa, T., Naok Hashmoto, N. Human Scale Haptc Interacton wth a Reactve Vrtual Human n a Realtme Physcs Smulator. In Proc. of Advances n Computer Entertanment Technology 25. [5] Hodgns,, J. K., Wooten, W. L., Brogan, D. C. and O'Bren, J. F. Anmatng Human Athletes. SIGGRAPH '95 Proceedngs, 7-78, 995. [6] Hof A. L., Gazendam, M. G. J., Snke, W. E. The condton for dynamc stablty. Journal of Bomechancs, 38, -8, 25. [7] Komura T., Ho E. S.L., Lau R. W.H. Lau. Anmatng Reactve Moton Usng Momentum-based Inverse Knematcs. The Journal of Computer Anmaton and Vrtual Worlds, 6(3-4), , Wley, 25. [8] Laszlo J., van de Panne M., Fume E. Lmt Cycle Control and Its Applcaton to the Anmaton of Balancng and Walkng. In Proc. of SIGGRAPH 996, 55-62, 996.

9 [9] Moore, M., and Wlhelms, J. Collson Detecton and Response for Computer Anmaton. Computer Graphcs (SIGGRAPH '88 Proceedngs), 22(3), , 988. [] Natural Pont Inc. [] O'Rourke, J. Computatonal Geometry n C (Cambrdge Tracts n Theoretcal Computer Scence), Cam-brdge Unversty Press, 998. [2] Oshta, M., Maknouch, A. A Dynamc Moton Control Technque for Human-lke Artculated Fgures. Computer Graphcs Forum (EUROGRAPHICS 2), 2(3), 92-22, 2. [3] Park S. I., Shn H. J., Shn S. Y. On-lne Locomoton Generaton Based on Moton Blendng. In Proc. of ACM SIGGRAPH Symposum on Computer Anmaton 22, 3-2, 22. [4] Shn, H.J., Lee, J., Glecher, M., Shn, S.Y. Computer Puppetry: An Importance-Based Approach, ACM Transactons on Graphcs. 2 (2), 67-94, 2. [5] Yamane, K., Nakamura, Y. Dynamcs Flter - Concept and Implementaton of Onlne Moton Generator for Human Fgures. IEEE Transactons on Robotcs and Automaton, 9(3), 23. [6] Yn, K. K., Pa, D. K. FootSee: an Interactve Anmaton System. In Proc. of ACM SIGGRAPH / Eurapraphcs Symposum on Computer Anmaton 23, , 23. [7] Yonemoto, S., Arta, D., Tanguch, R. Real-tme Human Moton Analyss and IK-based Human Fgure Control. In Proc of Workshop of Human Moton 2, 49-54, 2. [8] Zordan, V. B., Chu, B., Majkowska, A., Fast, M. Dynamc Response for Moton Capture Anmaton. ACM Transactons of Graphcs (Proc. of SIGGRAPH 25), 24(3), 697-7, 25.