Motion History Volume for Spatiotemporal Editing of 3D Video in Multi-party Interaction Scenes

Transcription

1 IPSJ SIG Techncal Report Moton Hstory Volume for Spatotemporal Edtng of 3D Vdeo n Mult-party Interacton Scenes Qun Sh,a) Shohe Nobuhara,b) Takash Matsuyama,c) Abstract: In ths paper we present a novel method that performs spatotemporal edtng of 3D mult-party nteracton scenes for free-vewpont browsng, from separately captured data. The man dea s to frst propose the augmented Moton Hstory Volume (MHV) for moton representaton. Then by modelng the correlatons between dfferent MHVs we can defne a mult-party nteracton dctonary, descrbng the spatotemporal constrants for dfferent types of multparty nteracton events. Fnally, constrant satsfacton and global optmzaton methods are proposed to synthesze natural and contnuous mult-party nteracton scenes. Experments wth real data llustrate the effectveness of our method.. Introducton Ths paper s amed at presentng a novel method to synthesze spatotemporally synchronzed 3D mult-party nteracton scenes from separately captured data, whle preservng the orgnal moton dynamcs of each object as much as possble. In the lterature of vrtual character moton edtng, most conventonal methods assume to have knematc models of the objects. Such methods are known to work robustly for dverse moton edtng tasks, but the modelng of mult-party nteractons s stll an open problem. In addton, these knematc model-based methods are not drectly applcable for certan knd of 3D data wthout a unfed mesh model. In computer vson and computer graphcs felds, n order to avod occluson problems or to ncrease the reusablty of data, mult-party nteracton scenes could be created by frst capturng each object ndependently and then, syntheszng them together. Snce separate capture wll nevtably result n spatal and temporal msmatches, the synthess of mult-party nteracton scenes becomes a non-trval task and requres large amount of manual work. Our research proposes to ntroduce the dea of augmented Moton Hstory Volume (MHV) to represent both the sngle object moton and mult-party nteracton event. Ths approach allows the edtor to realze natural 3D edtng of multparty nteracton scenes from separately captured data wth spatal and temporal msmatches sem-automatcally, no matter knematc models are avalable or not. The techncal problem we address for ths research s how we can perform spatotemporal algnments on separately captured data, whch s reconstructed nto temporal sequences of 3D meshes. Generally, the word nteracton has dfferent levels of meanngs, so n ths research work we narrow our focus onto the Kyoto Unversty a) sek@vson.kuee.kyoto-u.ac.jp b) nob@vson.kuee.kyoto-u.ac.jp c) tm@vson.kuee.kyoto-u.ac.jp Fg. Mult-party nteracton scenes edtng from separately captured data. explct nteracton event conducted by spatotemporally correlated body motons. The deas behnd our method are as follows. Bascally, an nteracton event can be consdered as a par of spatotemporally synchronzed motons performed by dfferent objects. By representng the motons wth tme recorded volume data, we wll be able to ntroduce varous spatotemporal constrants for multple objects, based on whch a mult-party nteracton dctonary can be defned. Then for each nteracton scene, rather than computng an ndvdual frame, the MHV consders a duraton of moton smultaneously n computaton. Ths makes t possble to consder spatotemporal constrants on the entre moton, and to have objectve crtera that descrbe entre motons. Thus for each nteracton scene we can acqure a soluton group wth multple reasonable relatve locatons of the two nteractng objects by performng MHV-based constrant satsfacton, whle preservng the orgnal moton of each object. The overall processng scheme of the proposed method s as follows. Gven two separately captured moton sequences, we frst perform data segmentaton and temporal algnment. Then we buld up the MHV for each moton segment n a mult-party nteracton scene. Next, we decde the spatotemporal constrants c 03 Informaton Processng Socety of Japan

2 IPSJ SIG Techncal Report Fg. Computatonal processes for mult-party nteracton edtng. for each nteractve moton segment par and compute the soluton group to realze the edtng of each sngle nteracton scene. Fnally all the edted nteracton scenes wll be ntegrated together under a global optmzaton strategy. Fgure llustrates the computatonal processes of our method. It should be noted that step 3, 5, 6, 7 can be computed automatcally usng the proposed method, whle step 4 requres some manual work from the edtor. The two pre-processng steps are out of the scope of ths artcle, and they could be done usng conventonal methods. The man contrbutons of ths paper consst of () we have augmented the orgnal Moton Hstory Volume by encodng more temporal nformaton and assgnng labels onto ts surfaces, () we have desgned a novel method that models mult-party nteracton events nto an nteracton dctonary, based on the MHV. The possble applcatons nclude the makng and edtng of moves, computer anmatons and vdeo games. The rest of ths paper s organzed as follows. Frst, we revew related studes to clarfy the novelty of ths work n Secton. We then ntroduce our MHV-based mult-party nteracton representaton n Secton 3, and the ntra-key segment edtng and nter-key segment optmzaton algorthm n Secton 4. Evaluaton of our method wth real data s gven n Secton 5. Fnally, we summarze the proposed method n Secton 6.. Related Work. Moton edtng work for computer anmaton The prolferaton of varous technques for creatng motons for vsual meda products, coupled wth the needs to best utlze these artstc assets, has made moton edtng a rapdly developng area of research and development. The past few years have wtnessed an exploson of new technques for varous moton edtng tasks, as well as deployment of commercal tools. For edtng a sngle object, moton warpng [], [] has been wdely appled to preserve the orgnal features of nput moton data whle satsfyng a new set of constrants as much as possble. Usng structurally smlar moton database, Muka[3] have proposed a moton nterpolaton approach to obtan contnuous spans of hgh qualty moton. Hlton [4] proposed a shape constraned Laplacan mesh deformaton approach that performs nteractve edtng of mesh sequences whle preservng the natural dynamcs of the captured surfaces. Some other researchers combned an nterpolaton technque wth moton graphs to allow parametrc nterpolaton of moton whle traversng the graph [5], [6]. Whle these works are proved to work robustly for edtng sngle object motons, they dd not provde any constrants for edtng multparty nteracton events. Besdes, Zordan [7] have ntroduced a technque for ncorporatng unexpected mpacts nto a moton capture-drven anmaton system through the combnaton of a physcal smulaton and a specalzed search routne. Ye [8] have proposed a fully automatc method that learns a nonlnear probablstc model of dynamc responses from very few perturbed sequences. Ther model s able to synthesze responses and recovery motons under new perturbatons dfferent from those n the tranng examples. Although these methods can somehow, work for very short nteractve events, they are performng the edtng by changng the orgnal moton dynamcs nstead of keepng t, and they are not applcable for edtng long and contnuous mult-party nteracton scenes ether. In ths paper, we are presentng a novel spatotemporal edtng framework that synthesze well synchronzed mult-party nteracton scenes from separately captured data, n whch objects are performng nteractve motons that should match wth each other, respectvely.. Moton representaton approaches In computer graphcs and computer vson area, researchers have nvented varous approaches for descrbng motons. Postons and veloctes of human body parts have been used by Green [9] for human movement recognton. Optcal flows [0], moton templates [] and space-tme volumes [], [3] are also wdely used for solvng trackng and recognton problems. Such methods are manly used for descrbng the objects moton features for recognton tasks, and they do not offer any controllable factors for moton edtng task. Besdes, knematc models [7], [4] are wdely used n robotcs, moton capture system and computer anmaton edtng. They can represent varety knd of motons, and as well they provde easly controllable factors for the edtors. However, the knematc models are lackng n the ablty of representng mult-party nteracton events. Few spatotemporal constrants can be drectly defned between multple objects based on them. Not to menton that for certan type of c 03 Informaton Processng Socety of Japan

3 IPSJ SIG Techncal Report (a) MHV of the attacker. (b) MHV of the dodger. Fg. 3 Examples of MHV. unstructured data [5] wthout a unfed mesh model, knematc structures are not applcable at all. On the other hand, the dea of Moton Hstory Volume has been proposed by Wenland [6], for solvng free vewpont acton recognton task. It carres out a concept of consderng the entre moton as a whole nstead of descrbng t for each sngle frame, by encodng the hstory of moton occurrences n the volume. Voxels are therefore multple-values encodng how recently moton occurred at a voxel. Mathematcally, consder the bnary-valued functon D(x, y, z, t) ndcatng moton at tme t and locaton (x, y, z), then ther MHV functon s defned as follows: τ f D(x, y, z, t) v τ (x, y, z, t) = max(0, v τ (x, y, z, t ) ) otherwse () where τ s the maxmum duraton a moton s stored. Our work proposed an augmentaton of the orgnal MHV by recordng precsely the full temporal nformaton of moton, makng t more sutable for mult-party nteracton edtng tasks. 3. Defnton of MHV and mult-party nteracton dctonary In ths secton, we frst propose an augmentaton of the orgnal Moton Hstory Volume by addng n full temporal nformaton of the moton. Then an MHV based mult-party nteracton representaton method wll be gven as well. 3. MHV of ndvdual moton As s wrtten n Secton., the dea of moton hstory volume was nvented for acton recognton, and has been proved to be effectve. However, the orgnal defnton of MHV only cares about the occurrence of the moton, whch may not be enough for performng mult-party nteracton edtng. Snce we need to do spatal and temporal algnments of multple separately captured moton sequences, full temporal nformaton ncludng both the startng and endng moments of the moton s necessary for representng the spatotemporal correlatons between multple objects motons. Therefore, we extend the defnton of MHV by addng n more temporal nformaton of the moton as follows: (t start, t end ) f τ =0 D(x, y, z, ) v τ (x, y, z, t) = () Null otherwse where t start and t end are the startng and endng tme of the moton. And we denote the center of gravty of MHV vτ A (x, y, z, t) as C A (x, y, z), whch s consdered as the root node of an MHV. Fgure 3 llustrates the MHVs of the attacker and the dodger Fg. 4 Lables of MHV surfaces. n a fghtng scene, respectvely. Here dfferent colors represent dfferent endng tme of the moton on the voxels. It should be noted that MHV needs not necessarly contan the moton of the entre body. Instead t can be smplfed by only countng n the partal volume of nterest on the objects body. For example, n a sword fghtng scene we may only care about the weapon of the attacker, so that an MHV of the sword would be enough for further edtng work. 3. Mult-party nteracton dctonary usng MHVs Usng the MHV as s descrbed n Secton 3., we can represent the spatotemporal structure of a sngle objects moton. However, n order to perform effectve edtng of mult-party nteracton events, we stll need to fnd a way to model the spatotemporal correlatons of multple objects motons (whch s represented wth MHV). To realze that, we assgn certan labels onto the MHV surfaces accordng to the moton drectons to help descrbe the spatotemporal relatonshp of multple MHVs. As s shown n Fgure 4, we denote the startng surface of MHV wth +, the endng surface wth -, and the lateral surface wth 0. Then, let a par of labels denote the contactng relatonshp of two MHVs. For example, + & + means the two objects MHVs contact wth each other at the endng surfaces. Based on the combnaton of those labels, we can descrbe all the possble spatotemporal correlatons of multple MHVs, as s llustrated n Table. We name the sum up of them a mult-party nteracton dctonary, whch represents all the possble multparty nteracton types that could be modeled usng MHV based representaton. In detals, the spatotemporal constrants for each nteracton type are descrbed as follows: Frst of all, for all nteracton types there s a spatal constrant that the 3D volume of two MHVs should contact wth each other: V A τ (x, y, z) V B τ (x, y, z) φ (3) here V A τ (x, y, z) represents a collecton of voxels n v A τ (x, y, z, t). Note that the two pared MHVs should have the same maxmum duraton τ, we wll explan how we ensure ths n Secton 4. As well, the recorded tmng should be scaled nto the segment orented tmng by subtractng ts real frame numbers from the orgnal moton sequence. Second, each nteracton type has ts own temporal constrant on all the contactng voxels as follows: + & - : t A end = T end, t B start = T start (4) c 03 Informaton Processng Socety of Japan 3

4 IPSJ SIG Techncal Report Table Interacton Dctonary wth examples. Interacton Dctonary + & - 0 & 0 + & + - & - + & 0 - & 0 Hand shakng Shake Hands Rase Arm Arm Down Fghtng Push&Pull Attack&Dodge Attack&Guard After A&G Punch After Punch 0 & 0 : (a) If nterest volumes contact wth each other at every moment wthn τ τ t=0 (V A t (x, y, z) V B t (x, y, z) φ) (5) (b) If nterest volumes contact wth each other at certan moments wthn τ τ (V t A (x, y, z) V t B (x, y, z) φ) (6) t=0 (c) If nterest volumes have no contacts throughout τ + & + : - & - : + & 0 : - & 0 : t A start > t B end (7) t A end = tb end = T end (8) t A start = t B start = T start (9) t A end = T end (0) t A start = T start () here T start = 0 and T end = τ. V t A (x, y, z) represents a subset of Vτ A (x, y, z) at tme t. Ths mult-party nteracton dctonary provdes us varous spatotemporal constrants for performng the edtng work. For example, an attack and guard scene n sword fghtng can be counted as type + & +. Then on the nterested part of the two MHVs, we requre tend A = tb end = T end, meanng that the two swords contact and only contact at the end of the whole moton duraton. 4. Spatotemporal 3D edtng algorthm In ths secton we present the spatotemporal 3D edtng algorthm based on MHV. 4. Overvew and defnton of terms In ths research work, the expected nput data s a seres of 3D mesh surfaces of the objects, be that a unfed mesh model or frame-wse unstructured meshes. And we suppose a moton sequence can be consdered as a collecton of key segments and transtonal segments. For each short acton scene formed up by a par of key segments, there should exst multple reasonable solutons. Whle a unque optmal soluton for the entre nteracton event could fnally be found n the ntegraton throughout the whole sequence. Generally, our mult-party nteracton edtng method conssts of three steps as () data segmentaton and temporal algnment, () ntra-key segment edtng by MHV-based constrant satsfacton and (3) nter-key segment global optmzaton. In order to make t more understandable, we frst gve clear defntons of these edtng-related terms as follows: ( ) Moton sequence: The whole sequence of the orgnal captured objects, n the form of 3D surface meshes. Let M = {V, E} denote the 3D mesh, then a moton sequence can be denoted as S = {M k k =,..., n k }. For our mult-party nteracton edtng, moton sequences of dfferent objects wll serve as the nput. ( ) Key segment: In each moton sequence, frames where nteractve motons happen are consdered to be key segment K = {M k k =,..., n k }. Snce the objects are supposed to perform motons that match wth each other, key segments should appear n pars from dfferent moton sequences. ( 3 ) Transtonal segment: Intermedate frames between key segments n the moton sequence, n whch no nteractve moton s stored. We use T = {M k k =,..., n k } to denote transtonal segments, and t should be noted that there can be no transtonal segment between two key segments. ( 4 ) Acton scene: Spatotemporally synchronzed short multparty nteracton scene syntheszed from a sngle par of key segments. ( 5 ) Interacton sequence: Spatotemporally synchronzed long mult-party nteracton sequence syntheszed by ntegratng all the key segment pars and transtonal segments n the orgnal moton sequences. 4. Data segmentaton and temporal algnment In ths research work, snce our man focus s on edtng multparty nteracton scenes nstead of recognzng them, the data segmentaton s performed manually by the edtor. Below are three crtera for selectng key segments: () Interactve motons should happen n each key segment, so that spatal and temporal constrants as defned n the nteracton dctonary exst wthn each key segment par. () The moton trend of the nterested volume should be undrectonal, whch wll ensure the smplcty of MHV n each key segment. (3) Multple key segments n one moton sequence should have no overlaps, and they need not to be adjacent. On the other hand, snce the potental nteractvely correspondng motons n the orgnal captured data would nevtably have temporal msmatches, the canddates n each selected moton segment par may also have dfferent lengths. Therefore, a temporal normalzaton s needed for buldng up comparable MHVs as follows: For a par of key segment K A and K B, f τ A > τ B, else, M A m = M A n (0 < m < τ B, n = m τa τ B ) () M B m = M B n (0 < m < τ A, n = m τa τ B ) (3) c 03 Informaton Processng Socety of Japan

5 IPSJ SIG Techncal Report τ (nv t A (x, y, z) nv t B (x, y, z)) = φ (5) t =0 Fg. 5 Mutual vsblty constrant. where τ A and τ B represent the duraton of K A and K B, MA n represents the nth frame n K A, and M A m represents the mth frame of the normalzed K A. x s the floor functon that returns the largest nteger not greater than x. Note that the same processng wll be appled to the transtonal segment pars as well. 4.3 Intra-key segment edtng 4.3. Edtor defned constrants After data segmentaton and temporal algnment, the nteractve motons of each object are organzed nto key segment pars, and nsde each par the two motons are scaled nto the same length. Then the MHVs of each object n each segment par can be computed as vτ A (x, y, z, t) and vτ B (x, y, z, t). For each nteracton event nsde a key segment par, the two nteractng objects should follow certan spatotemporal constrants based on () nteracton type, () mutual vsblty and (3) fdelty requrement. Frst for each key segment par the nteracton type should be decded by the user based on hs/her edtng ntenson, producng the spatotemporal constrant as s descrbed n Secton 3.. Second, for mult-party nteracton scenes, naturally the objects should be vsble for each other. We use the method of Sh [7] to compute the objects average gazng drecton durng the edtor-assgned frames. Then a vsual cone s generated for each object as VS A(x, y, z) and VS B (x, y, z), lookng from the average 3D poston of the center of the eyes nto the average gazng drecton. The detaled parameters (e.g. the cone angles) can be adjusted by the edtor as approprate. Then the mutual vsblty constrant can be descrbed as follows: (VS A (x, y, z) V B τ (x, y, z)) (VS B (x, y, z) V A τ (x, y, z)) φ (4) It should be noted that the vsblty constrant s an optonal constrant for the edtor, meanng that t s not always requred for each moment n the key segment par. Thrd, the fdelty requrement avods the unnatural fakeness, lke two object merge nto each other s body, from happenng. Let nv A τ (x, y, z, t) and nv B τ (x, y, z, t) denote the MHVs of the objects that should not contact n the nteracton event, then the fdelty constrant can be defned as follows: 4.3. Acton scene edtng usng constrant satsfacton Wth the edtor defned constrants, we compute the proper relatve poston of the objects by constrant satsfacton. Frst, the two objects are put nto the same world coordnate system. We fx the poston of vτ A (x, y, z, y, t) and then, sample the xy plane nto grd ponts, and each grd pont wll be further sampled by surroundng angles. Next, translate and rotate vτ B (x, y, z, t) onto each sampled poston and drecton, then test wth the edtor defned constrants. If for a sampled poston and drecton all the constrants are fulflled, then the dstance between the two objects L AB = C AC B wth the two reference angles θ A and θ B can be counted as one soluton for ths key segment par. Here C B denotes the relocated poston of C B n one sampled poston, and θ A, θb represents the ncluded angles between the average vewng drecton of each MHV and lne C AC B, respectvely. By testng all the sampled postons and drectons, for each key segment par a soluton group R AB = (L AB j, θ A j, θ B j ) wll be computed. 4.4 Inter-key segment optmzaton Havng computed the soluton groups for each key segment par, we wll combne them together wth the transtonal segments to synthesze a complete mult-party nteracton sequence, by () fndng the optmal soluton n each key segment par and () addng n the transtonal segments and performng path optmzaton Optmal soluton searchng for key segments Before edtng, suppose the postons of the MHVs root node n the world coordnate system are C A, CA,..., CA I, CB, CB,..., CB I, and the facng drectons of the MHVs n the world coordnate system are F A, F A,..., FI A, F B, F B,..., FI B. In order to make natural and smooth edtng, we should mantan the vectors C A CA + and C BCB + as much as possble to mnmze the artfcal offsets. As well, the facng drectons F A, F B should also be preserved. If we denote the edted deal postons of the MHVs root node as C A, C A,..., C I A, C B, C B,...,, the edted facng drectons C B I of the MHVs as F A, F A,..., FI A, F B, F B,..., FI B, and the angles between F A and C A C B as θ A, then we should search for the optmal soluton for each key segment par by fulfllng: and [ C A CA + C + A + C B CB + I mn = C A +λ( F A C B = LĀ B C A θ A F A + F B C B C+ B F B )] (6) = L A j B = C A j C B j (7) = θ A, θ B = θ B (8) here λ s a weghtng factor to balance the translaton and rotaton transformatons. The optmal soluton for all the key segments c 03 Informaton Processng Socety of Japan 5

6 IPSJ SIG Techncal Report can be computed by solvng functon (6), (7) and (8) usng dynamc programmng approach Path optmzaton for transtonal segments Havng found the optmal solutons for all the key segments, we then add n the transtonal segments and perform path optmzaton. Suppose nsde a transtonal segment, the orgnal postons of the object s center of masses for all the frames are C, C,..., Cm and the edted postons are denoted as. Whle orgnal and edted facng drectons are C, C,..., Cm. denoted respectvely as F, F,..., Fm and F, F,..., Fm (a) Image data of object A. Fg. 6 (b) Image data of object B. Separately captured mage data of two objects. Then we defne our objectve functon for performng path optmzaton as follows: f = m X [δ fa (C ) + ( δ)ωc fv (C ) = (a) (b) (c) (d) (e) (f) (g) (h) (9) +λ(δ fa (F ) + ( δ)ωf fv (F )] Specfcally, fa (C ) s defned as follows to preserve the orgnal acceleratons for each frame: C + C+ ) fa (C ) = (C C + C+ ) (C Fg. 7 Moton edtng results for short acton scenes. (0) and fv (C ) s defned as follows to preserve the orgnal speed for each frame: fv (C ) = C+ C C+ C () Besdes, ωc = + exp{α(v vk )} () ωf = + exp{α(r rk )} (3) Fg. 8 C C + C C+ v = (4) F F + F F+ r = (5) here vk and rk are constant parameters to be set by edtors. By mnmzng ths objectve functon we can perform path optmzaton onto all the transtonal segments and fnally, acqure a natural lookng spatotemporally synchronzed mult-party nteracton 3D Vdeo sequence for free-vewpont browsng. 5. Experment and evaluaton In ths secton we present an evaluaton of the proposed multparty nteracton edtng method wth real data. 5. Experment setup and pre-processng To prove the effectveness of our method, we prepared the 3D vdeo data of two professonal actors, separately captured by 5 calbrated XGA cameras runnng at 5 Hz wth msec shutter speed, and reconstructed wth frame-wse 3D shape reconstructon method. In the data, the two separately captured actors are performng a pre-desgned sword fghtng moton sequence, respectvely. Note that the reconstructed 3D shapes do not have a unfed 3D mesh model. Body part segmentaton and knematc models are not avalable ether. As a pre-processng, the data s manually segmented nto 9 c 03 Informaton Processng Socety of Japan (a) Frst-person-vew mage of ob- (b) Frst-person-vew mage of object A. ject B. Frst-person-vew mages rendered from the edtng result. key-segment pars and nsde each par the motons of the two objects are algned nto the same temporal length. After the preprocessng, the lengths of both moton sequences are resampled nto 45 frames. 5. Mult-party nteracton edtng results Wth the separately captured data, f we smply put them together nto the same coordnate system and match up ther begnnng frames manually, n the followng moton sequences varous spatotemporal msmatches wll occur, as s shown n Fgure 7(a)(d). On the other hand, the edtng results usng the proposed method are llustrated n Fgure 7(e)-(h). It can be clarfed that n each syntheszed nteracton scene, the relatve locaton of the two objects looks reasonable and they well qualfy the edtor defned spatotemporal constrants. In addton, Fgure 8 llustrate the frst-person-vew mages rendered from the edtng results, usng Sh s method[7]. It can be seen clearly that the two objects are nsde each other s vew, fulfllng the mutual vsblty constrant. 5.3 Processng cost of the proposed method The experment s conducted usng a PC wth Intel(R) Core(TM) Duo CPU, 3.00GHz. Under the samplng resoluton of 0mm, degree, the average computaton tme for each automatc process s as follows: ( ) MHV computaton for each object n a sngle segment:

7 IPSJ SIG Techncal Report mn ( ) Constrant satsfacton for each short acton scene: 5 mns ( 3 ) Optmal soluton searchng for each key segment par: 98 mns ( 4 ) Path optmzaton for each object n a sngle transtonal segment: 7 mns ( 5 ) Total tme cost: 87 mns Note that the computatons for each segment can be conducted parallelly nsde process,, Concluson In ths paper we presented a novel moton edtng method that syntheszes spatotemporally synchronzed mult-party nteracton 3D Vdeo scenes from separately captured data. By proposng the dea of MHV we can effectvely model the nteracton events of multple objects as well as representng the motons of a sngle object, based on whch the constrant satsfacton method can be appled to perform ntra-key segment edtng. An optmal soluton search and path optmzaton scheme s also desgned to mnmze the artfacts generated n the edtng and mantan the orgnal moton dynamcs as much as possble. Experments wth real data proved the effectveness and robustness of the proposed method. In our work, the data beng used s the unstructured 3D Vdeo data. However, snce our method only requres a sequence of 3D meshes as nput, t wll work for (Moton Capture Drven) CG data, whch has a unfed mesh model, as well. For further studes, we are plannng to perform the same edtng job usng CG data wth unfed mesh models, to examne how we can combne our spatotemporal edtng method wth conventonal knematc structure based methods together to acqure better results. Acknowledgement acton events from multple vewponts. EventVdeo0, pages 647, 00. [3] Ylmaz, A. and Shah, M.. Actons sketch: A novel acton representaton. CVPR05, pages I: , 005. [4] Hlton, A.. A Survey of Advances n Vson-Based Human Moton Capture and Analyss. Internatonal Journal on Computer Vson and Image Understandng, Vol.04, No.-3 :90-7, 006. [5] Nobuhara, S. and Matsuyama, T.. Deformable Mesh Model for Complex Mult-Object 3D Moton Estmaton from Mult-Vewpont Vdeo. The Thrd Internatonal Symposum on 3D Data Processng, Vsualzaton and Transmsson (3DPVT006), [6] Wenland, D., Ronfard, R. and Boyer E.. Free vew-pont acton recognton usng moton hstory volumes. COMPUTER VISION AND IM- AGE UNDERSTANDING (006). [7] Sh, Q., Nobuhara, S. and Matsuyama, T.. 3D Face Reconstructon and Gaze Estmaton from Mult-vew Vdeo usng Symmetry Pror. IPSJ Transactons on Computer Vson and Applcatons, Vol.4, pp.49-60, Ths study s supported by JSPS Ayame project Vsual Gesture Percepton. References [] Glecher, M.. Retargettng moton to new characters. SIGGRAPH 98: Proceedngs of the 5th annual conferenceon Computer graphcs and nteractve technques (998), ACM, pp [] Lee, J., Shn, S.Y.. A herarchcal approach to nteractve moton edtng for human-lke fgures. Proceedngs of SIGGRAPH 99 (999), pp [3] Muka, T., Kuryama, S.. Geostatstcal moton nterpolaton. ACM Transactons on Graphcs 4, 3 (005), [4] Tejera, A. and Hlton, A.. Space-tme edtng of 3D vdeo sequences. CVMP, 0 [5] Heck, R., Glecher, M.. Parametrc moton graphs. Proceedngs of Symposum on Interactve 3D Graphcs and Games (007), pp [6] Safonova, A., Hodgns, J.K.. Constructon and optmal search of nterpolated moton graphs. ACM Transactons on Graphcs 6, 3 (007), 06.. [7] Zordan, V.B.. Dynamc response for moton capture anmaton. ACM Trans. Graph., 4, 3, pp (005). [8] Ye, Y. and Lu, C.K.. Synthess of responsve moton usng a dynamc model. Comput. Graph. Forum, 9,, pp (00). [9] Green, R. and Guan, L.. Quantfyng and recognzng human movement patterns from monocular vdeo mages. IEEE transacton on Crcuts and Systems for Vdeo Technology, 4, February 004. [0] Alexe, A.E., Alexander, B., Greg, M., and Jtendra, M.. Recognzng acton at a dstance. ICCV, 003. [] Aaron, F.B. and James, W.D.. The recognton of human movement usng temporal templates. IEEE Trans. Pattern Anal. Mach. Intell., 3(3):57-67, 00. [] Syeda-Mahmood, T.F., Vaslescu, M.A.O. and Seth, S.. Recognton c 03 Informaton Processng Socety of Japan 7