Video-based Characters Creating New Human Performances from a Multi-view Video Database

Transcription

1 Vieo-base Characters Creating New Human Performances from a Multi-view Vieo Database Feng Xu Yebin Liu? Carsten Stoll? James Tompkin Gaurav Bharaj? Qionghai Dai Hans-Peter Seiel? Jan Kautz Christian Theobalt? Tsinghua University, China University College Lonon, UK? MPI Informatik, Germany Figure 1: An animation of an actor create with our metho from a multi-view vieo atabase. The motion was esigne by an animator an the camera was tracke from the backgroun with a commercial camera tracker. In the composite scene of animation an backgroun, the synthesize character an her spatio-temporal appearance look close to lifelike. Abstract 1 We present a metho to synthesize plausible vieo sequences of humans accoring to user-efine boy motions an viewpoints. We first capture a small atabase of multi-view vieo sequences of an actor performing various basic motions. This atabase nees to be capture only once an serves as the input to our synthesis algorithm. We then apply a marker-less moel-base performance capture approach to the entire atabase to obtain pose an geometry of the actor in each atabase frame. To create novel vieo sequences of the actor from the atabase, a user animates a 3D human skeleton with novel motion an viewpoints. Our technique then synthesizes a realistic vieo sequence of the actor performing the specifie motion base only on the initial atabase. The first key component of our approach is a new efficient retrieval strategy to fin appropriate spatio-temporally coherent atabase frames from which to synthesize target vieo frames. The secon key component is a warping-base texture synthesis approach that uses the retrieve most-similar atabase frames to synthesize spatio-temporally coherent target vieo frames. For instance, this enables us to easily create vieo sequences of actors performing angerous stunts without them being place in harm s way. We show through a variety of result vieos an a user stuy that we can synthesize realistic vieos of people, even if the target motions an camera views are ifferent from the atabase content. There is still a substantial quality gap between photo-realistic vieo sequences an fully animate human characters. In current vieo games, animation techniques are highly evelope an intricate motions can be create. However, the realism of the renere animation sequences still oes not match a capture vieo. In contrast, acquire vieo sequences for movie prouctions are realistic because they are irectly capture through high-quality cameras. However, all require motions an actions nee to be performe by actors, an it is very ifficult to make any kin of motion eits later on even changing boy appearance in vieos with the same motion is alreay a challenge [Jain et al. 2010]. This means that actors nee to repeat their performance many times until the esire motion/action is achieve to a sufficient quality. Consequently, generating photo-realistic sequences of human beings is highly esirable for both computer games an movie prouction. Vieo-texture methos first attempte to synthesize new vieo footage by recombining existing vieo clips [Schöl et al. 2000]. However, they o not allow moification of the camera viewpoint an they face a big challenge in resynthesizing vieos showing plausible articulate boy motion [Flagg et al. 2009]. To our knowlege, our propose metho is the first technique that enables the synthesis of photo-realistic vieos of human characters performing user-efine motions, observe from user-efine viewpoints. CR Categories: I.4.8 [Computer Graphics]: Scene Analysis Time-varying imagery; Introuction One of the main ifficulties in synthesizing photo-realistic vieos of animate characters lies in the creation of realistic textures. When people perform ifferent kins of motions, their appearance continually changes accoring to their motion. For example, highlights appear an isappear, fols an wrinkles form an move on clothes, an skin color varies with motion. As the appearance of human characters is affecte by various physical conitions, simulating realistic texture is a ifficult problem [Jimenez et al. 2010]. In our scheme, we evelop an image-base metho to overcome this ifficulty. We buil an search a multi-view multi-motion atabase for vieo frames with appropriate texture to synthesize frames of a novel target animation. We o not run any kin of simulation but rather perform retrieval an image-base warping to achieve realistic textures. Image-warping is guie by a etaile moel of

2 skeletal motion an ynamic shape that we capture for each multiview sequence in the atabase with a state-of-the-art marker-free performance capture approach. Our re-animation approach also allows us to support novel viewpoints by jointly consiering the pose an viewpoint. This guarantees sufficient view an pose information to synthesize target frames. Specifying a target animation is easy: In a stanar animation tool, a new motion for the actor s skeleton is specifie, applie to the boy moel, an renere into the esire camera path. We then use a warping-base metho to integrate information from multiple atabase frames to generate a renering of the texture sequence of the target poses an views. 2 Relate Work We capitalize on recent performance capture approaches that reconstruct etaile ynamic shape an motion moels of humans from multi-view vieo without using optical markers [e Aguiar et al. 2008; Vlasic et al. 2008; Braley et al. 2008; Vlasic et al. 2009; Cagniart et al. 2010]. In particular, we use the joint skeleton an surface tracking approach of Gall et al. [2009] to automatically capture performance moels of all multi-view sequences in the atabase without optical markers. Several previous image-base approaches can rener novel footage of humans from input vieo, but none can simultaneously alter the motion an the camera viewpoint an still achieve photo-realistic results. 3D vieo methos capture a ynamic scene with multiple vieo cameras (see Theobalt et al. [2007] for an overview), reconstruct [Zitnick et al. 2004; Matusik et al. 2000; Tung et al. 2009] or fit [Carranza et al. 2003; Ballan an Cortelazzo 2008] a geometry moel, or use light fiel renering to create novel viewpoints [Wilburn et al. 2005]. While these approaches enable renering a moving actor from novel viewpoints, large changes in motion are not possible. Waschbüsch et al. [2006] presente a 3D vieo processing framework that extens 2D vieo operations such as segmentation an compositing to the 3D omain; eiting the motion of a 3D vieo object is not possible. Weyrich et al. [2005] capture the reflectance fiel of a static object an use a geometric warping metho to rener the appearance of the object uner new lighting after surface eformation. Scaling such an approach to vieos of full humans woul be a major engineering effort, an realistic surface eformations in the target animation woul have to be entirely simulate. This is still a major problem in itself. Recently, Stoll et al. [2010] moifie animation moels of humans capture from multi-view vieo with estimate physics-base cloth moels. However, they cannot generate realistic textures for the new animations. Some previous work tries to create novel images or vieos in which certain appearance aspects of a person are moifie. Leyvan et al. [2008] propose a learning base metho to change the face shape of an input image. Kemelmacher-Shlizerman et al. [2010] transferre source facial expressions to a target character by fining similar frames in a vieo of the target s face. Zhou et al. [2010] reshape a human boy in one image base on user interaction. Jain et al. [2010] eite boy shape in vieo sequences. All these approaches eliver realistic novel image or vieo footage, but none prouce new camera views an new motions ifferent from the input. In Hornung et al. s [2007] work, a person in a single picture is animate by fitting to it a kinematic skeleton, an warping the image base on motion capture ata. Viewpoint an texture cannot be change. Cobzas et al. [2002] synthesize non-rigi motions of a moving arm by learning ynamic texture from multiple images. However, full boy motion an varying viewpoints are not consiere. The vieo textures concept [Schöl et al. 2000] is also relate to our work. The original iea is to analyze vieo sequences for possible transitions between any two frames. A new vieo can be assemble by rearranging the vieo frames an passing through transitions in a ranom orer. As an extension, Schöl an Essa [2002] built a motion graph of input vieo sequences an use repeate subsequence replacement to support the control of transitions by users. Applying vieo textures to synthesize new vieos of moving humans is challenging since ientifying correct transitions without any knowlege of 3D pose an shape is ifficult. Celly an Zoran [2004] took on that challenge by ientifying transition regions between a restricte set of filme human motions an performing purely image-base warping at transitions. Mori et al. [2004] followe a similar strategy but manually marke skeletons on images to support the creation of correct transitions. Flagg et al. [2009] use marker-base motion capture on the vieo footage to buil a motion graph structure for synthesizing new footage. All these methos use some form of internal motion graph, i.e., they create new motion by re-orering entire sub-sequences of specific input motions such as walking, jogging, or running. In other wors, no novel motions can be synthesize an viewpoint changes are not feasible. The work by Starck et al. [2005] extens some of the above ieas to 3D vieo sequences, i.e., it creates a motion graph base on shape-from-silhouette meshes an textures reconstructe from multi-view vieo. For a few basic motion types, 3D blens are pre-compute. This enables creating new animations by concatenating the basic motions through the transitions. Similarly, Huang et al. [2009] built a motion graph to transition between 3D meshes reconstructe from a hanful of multi-view vieo sequences showing basic motions. New 3D mesh sequences can be concatenate from the original mesh sequences. However, new sequences are restricte to the space of basic motions in the atabase, an texture synthesis is not consiere. In contrast, our approach can synthesize new motions an textures even if none of the poses an camera views are part of the atabase. When creating textures for the frames of a target animation we nee to warp textures from the input vieo frames both across pose an viewpoints. One way of synthesizing textures in novel views or poses woul be to use some form of projective texturing an blening, e.g., [Debevec et al. 1996; Narayanan et al. 1998; Buehler et al. 2001; Carranza et al. 2003; Cheung 2003; Hornung an Kobbelt 2009]. However, these approaches are known to prouce texture ghosting when there are even the slightest of inaccuracies in scene geometry. For our work this is even more of an issue since we perform texture transfer across view an moel pose. An alternative to texturing approaches are view synthesis methos that warp the source images into the target view using image-base corresponences, sometimes supporte by a 3D scene moel [Einarsson et al. 2006; Stich et al. 2008]. Warping approaches often prevent ghosting artifacts even if none or only approximate scene geometry is at han. Ballan et al. [2010] use a billboar to transition from one vieo sequence to another at some optimize temporal frame using a warping approach without consiering human geometry. Thus, we use an image-base warping metho guie by our reconstructe performance moels to warp textures across pose an viewpoint. 3 System Overview Input to our system is a user-efine query: a skeletal animation seen from a user-efine camera, corresponing to a rigge surface mesh of an actor. The pose mesh in each animation time step is use to retrieve appropriate images from our multi-view vieo atabase of the capture actor in orer to synthesize a realistic, animate output vieo (Fig. 2).

3 Database Retrieval Warp Moel Query Matche Frames Results Figure 2: Overview. The system maintains a atabase of multi-view vieo sequences of an actor performing simple motions. Skeleton motion an surface geometry were capture for each sequence using marker-less performance capture. The user is given a skeleton an surface mesh moel of an actor. By specifying skeletal motion an camera settings, he can create a query animation. Our algorithm synthesizes a photo-realistic vieo of the actor performing the user-esigne animation by selecting appropriate poses an their respective images from the atabase an warping these images to the target view. Database To guarantee sufficient images for synthesizing novel pose an viewpoint, we capture multi-view images of a character performing various basic motions. We call these atabase sequences, containing atabase frames. We then perform a skeletonbase performance capture algorithm [Gall et al. 2009] on the atabase sequences to obtain a 3D bone skeleton an a eforme mesh surface for each atabase frame. The obtaine skeletons an meshes form the atabase skeletons an atabase meshes. The atabase sequences only nee to be acquire an tracke once to synthesize novel target animations of the same character. Query The skeleton an rigge surface mesh of the actor, or boy moel, is presente to the user for creating an animation. The user can also animate the camera to synthesize vieo sequences from novel viewpoints. For creation of a target animation, henceforth calle a query, the user can employ any stanar 3D animation package that supports character animation. Once the query is create, we eform the mesh surface of the character accoring to the skeleton. After that, for each query frame we have a query skeleton, a query camera an a query mesh. The same mesh an skeleton moels of the actor are use to specify the queries an to capture the atabase performances. Retrieval In this step we fin appropriate images for synthesizing each target frame in our final output sequence (a so-calle target sequence, frames of which are calle target frames). We efine a similarity to measure the istance between a atabase frame an a query frame from joint an camera positions. We fin atabase frames matching the whole query sequence with this similarity, an call them source frames. Warping-base Synthesis We synthesize the target sequence with the retrieval results. In this step, image-base warping ajusts the pose of the source character an the viewpoint of the source camera to satisfy the query. This is necessary as our atabase frames are not exactly the same as our target frames. We use vertex corresponence between the query mesh an the atabase mesh to guie the warping. By using the projecte vertex corresponences in the image omain, the propose warping metho warps source images to obtain the final target sequence. 4 Multi-view Database an Motion Query We first capture an actor performing multiple basic motions in our multi-view capture system. Our system has 12 synchronize an calibrate vieo cameras that capture the actor from ifferent viewpoints at 45 fps with a frame resolution of pixels. Our basic motions are walking, running, jumping, kicking, punching, rolling, twisting an waving sequences. The atabase coul be extene by aing more complex motions; however, we foun that these basic motion types were enough to realistically synthesize a range of complex input animations. We foun it unnecessary to esign atabase motions such that the range of poses is maximize. To acquire sufficient view-epenent information, each basic motion is performe several times facing ifferent irections. After acquiring multi-view vieo sequences, skeleton-base marker-less performance capture [Gall et al. 2009] is applie to all capture sequences. This metho requires an initial mesh surface of the capture character an a corresponing skeleton. The initial mesh surface is obtaine by a static full-boy laser scan. The resolution of the mesh surface is low, containing only 5000 vertices for one human character. The unerlying skeleton of the mesh is create by manually marking joint positions. We use a skeleton with 16 joints as shown in Fig. 2. The skinning weight between each mesh vertex an each skeleton joint is calculate using the metho of [Baran an Popovic 2007]. Motion tracking provies a skeleton an a mesh surface for the actor for each atabase frame. The skeletons, moels, an capture multi-view sequences will be use in the following steps. The same skeleton an surface moel are provie to the user to create new queries. A query sequence is a user efine sequence that specifies the target vieo sequence using the same skeleton an mesh moel as are in the atabase (Fig. 3, top row). It comprises a query skeleton sequence which efines the pose of the character at all time instances an a query camera sequence which specifies the camera parameters. The skeleton motion is use to animate the surface mesh from the query camera view. Animation tools allow the whole animation sequence to be efine, incluing key frame-base skeletal animation an camera motion. Alternatively, motion retargeting techniques [Gleicher 1998] can apply motion capture ata from existing atabases to the given skeleton, an match-moving software can extract camera paths from backgroun vieos, e.g., Voooo 1. 5 Retrieval In this step we take the query sequence an perform a atabase retrieval to fin source frames for synthesis. The retrieve source frames nee to contain enough information to synthesize the target frame. Our retrieval metho accounts for pose an view similarity, an temporal consistency. We base our retrieval metho on the following observations: Renering the target animation with a static texture creates implausible results as the natural expecte motion an appearance changes on the surface of the character are not reprouce, see Sec. 7. It is crucial that the target animation features a spatio-temporally coherent 1

4 S1-C6-T234 S3-C4-T071 S3-C4-T072 S3-C5-T049 S2-C7-T148 S3-C4-T080 S3-C5-T052 S3-C5-T054 S3-C5-T055 S2-C7-T148 S4-C0-T123 S3-C5-T052 S1-C6-T232 S2-C7-T158 S2-C7- T159 S1-C6-T232 S2-C7-T161 S3-C4-T083 S3-C4-T085 S2-C7- T159 S3-C4-T070 S2-C7-T156 S2-C7-T157 S3-C4-T071 S5-C2-T130 S2-C7-T160 S3-C4-T071 S2-C7-T161 S3-C3-T283 S5-C2-T130 S2-C2-T310 S1-C6-T233 S2-C7-T160 S3-C4-T073 S3-C5-T051 S2-C5-T051 S3-C4-T081 S3-C4-T222 S1-C4-T073 S3-C5-T059 Figure 3: Illustration of fining matching caniate sequences using l = 2. This figure shows 4 caniate frames for each query skeleton. Here SX-CY-TZ enotes the image frame from sequence X, camera view Y an time inex Z. Caniate sequences are color-coe. No sequence of suitable length can be foun for query frame 10. time-varying textural appearance that shows plausible ynamics for the given pose sequence an as little spatial an temporal aliasing as possible. Thus, the first goal is to fin a plausible texture for each query pose. It is clear that for most poses an camera views in the query sequence we will not fin an exact matching frame in the atabase. However, in most cases the appearance of the surface in a atabase frame will be very similar to the esire target if the camera angle an posture in the atabase are close to the query. The appearance is also influence by lighting ifferences, but most of these can be equalize through color-base post-processing, as escribe in Sec. 6. The secon goal is to favor temporally coherent texture patterns from the atabase rather than rapily switching between atabase frames that are not temporally ajacent. This leas to naturally evolving an temporally coherent surface etail motion. The two goals are often conflicting, an the retrieval an synthesis strategy shoul fin a balance that leas to high-fielity output. In esigning the best retrieval strategy we can capitalize on aitional qualitative observations, such as that texture patterns usually change more rapily if the motion in the scene is very fast. In the following paragraphs we escribe how we translate the above concepts into a practical retrieval algorithm. We nee a frame-to-frame istance metric that measures the similarity between a query frame an a atabase frame (Sec. 5.1). To guarantee temporal consistency in the output, we prefer to fin sub-sequences in the atabase in which every frame is sequentially matche to a query sub-sequence. We call this step sequence-tosequence matching (Sec. 5.2). Matching only iniviual frames woul often lea to temporal flickering artifacts in the result. Sequence-to-sequence matching can only be establishe when there are some sub-sequences in the atabase similar to the query. However, as we use a atabase which contains only a hanful of basic motion sequences to synthesize all possible motions, this similarity is ifficult to guarantee. Therefore, we also allow the algorithm to fall back to frame-to-sequence matching, i.e., matching successive query frames to a single common atabase frame if no appropriate sequence-to-sequence match can be foun (Sec. 5.3). The face region unergoes special processing, as we explain in Sec Frame-to-frame Distance Metric Let us first efine a frame-to-frame istance metric which we will then exten to fin sequence-to-sequence matches. Our istance efinition is base on pose an camera ifferences. Both the tracke atabase skeletons an the query skeletons are first registere to the same worl coorinate system by translating all root joints to the same 3D position an rotating all skeletons to face to the same irection. The camera is move using the same global transformation as the skeleton in each frame to preserve its relative position an orientation to the skeleton. Given two frames, F q from the query an F from the atabase, we first measure the camera ifference: D cam (C q, C ) = (C q C ) 2 (1) between the registere camera centers C in the registere query an atabase frame. Consiering that all cameras are facing the actor at similar istances, this is a reasonable measurement. We also measure the skeletal istance: D skel (S q, S ) = 16 (Sq j S j )2 j=1 σ 2 (2) j where S j q is the position of the j-th joint of the skeleton in F q (an S j for F ) an σ j is the variance of the position of joint j in the atabase. We weight the iniviual joint position istances accoring to the variance in 3D pose of the respective joint in the atabase. Without such a weighting, joints that naturally unergo larger motions (such as joints towars the tips of en effectors like the wrist or ankle joints) woul ominate the istance measure when compare to joints that are closer to the torso. We finally combine the two istance measures: D(F q, F ) = D skel (S q, S ) + λ D cam (C q, C ) (3) where λ weights the camera an joint-similarity contributions. In practice, D skel (S q, S ) an D cam (C q, C ) are normalize, an we set λ = 2 which we etermine through experiments. Using this istance measure, we search the atabase an collect the 100 bestmatching frames F c (c = ) as caniates for synthesizing the texture in each query image. Ranking Caniates Our final aim is to synthesize a realistic image for each query F q, so we wish the chosen atabase frames to contain as much information as possible. The measure D(, ) fins similar poses an views reliably an very quickly, which is crucial to ensure that overall retrieval run times are reasonable. However, it is base on the assumption that pose an view similarity correspons to the amount of information available in a caniate frame. While this assumption generally hols, the measure is not base on caniate frame information content an oes not allow for an accurate ranking of caniates F c. Therefore, we have erive a ifferent measure to rank caniate frames F c. As escribe in Sec. 4, we have a 3D moel an camera parameters for each frame in both the query an the atabase sequences. We use as a measure a visibility score V (F q, F c ), efine as the number of mesh facets that are visible from both the view of the query an the view of the tracke mesh of a atabase frame. While it may be possible to use this visibility score to fin the 100 best caniate frames as well (instea of ranking them), it woul be inefficient as only similar poses an views are likely to contain reasonable matches. 5.2 Sequence-to-sequence Matching After collecting the best-matching frames for the whole query sequence, we try to exten the matches to generate sequence-tosequence corresponence as follows: For each caniate of the first query frame, we check whether its temporally successive frame in the atabase is also a caniate of the secon query frame. If so,

5 we link the later temporal caniate to the former caniate. This process is continue to form a caniate sequence until no temporally consistent caniate can be foun. In this case, we start a new caniate sequence at the frame where the sequence is interrupte. We repeat this for each caniate frame, builing all possible caniate sequences S i which may cover ifferent parts of the query sequence an may have ifferent lengths (see Fig. 3). The caniate sequences can start at any point in the animation. We now select the best caniate sequences by consiering three ifferent factors. First, sequences shoul be as long as possible, since short sequences cannot guarantee temporal consistency. Secon, the frames in a caniate sequence shoul be highly ranke (among the best matching frames for each query frame), as the ranking orer correspons to the similarity between the target frame an the atabase frames. Thir, the amount of motion in a sequence shoul be similar to the query sequence. This ensures that the etaile surface motion (i.e., fols an wrinkles) oes not vary infrequently because a slow motion from the atabase was use to synthesize a fast motion query sequence. To escribe the amount of motion, we efine a motion energy between consecutive frames F I an F I+1 as: E motion ( F I, F I+1) = D skel (S I, S I+1 ), (4) where S I represents the skeleton corresponing to frame F I. The sum of motion energies in all frames in a sequence efines the amount of motion in the sequence. The combine score Λ( ) for a caniate sequence S i, is: Λ(S i ) = exp ( α is E is (S i ) α rank Rank(S i ) 1/L(S i )), (5) E is (S i ) = E motion (F I, F I+1 ) E motion (Fq I, Fq I+1 ), (6) I S i I Q Si Rank(S i ) = 1 L(S i ) I S i Rank(F I ), (7) where L(S i ) is the number of frames in caniate sequence S i, α is an α rank weight the three factors an Q Si is the query subsequence corresponing to the caniate sequence S i. We usually set α Eis = min I QSi 1/E motion (Fq I, Fq I+1 ) an α rank = (200 is twice the number of caniates for a single query). This achieves goo performance in our experience. A caniate sequence will not be use when its sequence length is smaller than a threshol l (with l = 10 for all our results) or its corresponing query sequence is covere by another sequence element with a higher score. Accoring to this mechanism, long sequence elements compose of high ranking caniates an with similar amounts of motion to the query will be chosen as the final matching result F match(i) for frames Fq I. 5.3 Frame-to-sequence Matching After the sequence-to-sequence matching there may still be some query frames with no matches, because all caniate sequences were less than l frames long. For example, the 10th query in Fig. 3 is a frame with no match. If the query sequence motion an viewpoint is quite ifferent from the atabase then the number of unmatche query frames may be large. In this step, we ensure consistent matches are foun from the 100 caniate frames. We coul reuce the length l to guarantee no unmatche query frames. However, this woul introuce many short sequences to the result an so a temporal jitter (see Sec. 7 valiation II). For an unmatche query sub-sequence, F U1 q to Fq U2, we coul simply fin the top ranke caniate F best(u) (U [U 1,U 2 ]) as F match(u) for each query frame Fq U. However, top caniates may change frequently in a sub-sequence an therefore cause consierable temporal jittering in the final synthesize sequence (see user stuy, Sec. 7). Consiering that consecutive frames in a query sequence are likely to change slowly, we instea use the single best atabase frame for as many consecutive query frames as possible. Here, we trae reuce texture motion (moving cloth fols) in the synthesize result for reuce temporal jitter. Given that for frame Fq J (initially J = U 1 ) we have foun the best matching atabase frame F best(j), we must ecie how many consecutive query frames Fq J+n can use this atabase frame match. We apply insight as before: The query sequence parts with fast motion will require more atabase frames to represent them. Conversely, slow motions will require fewer frames. With this observation, we keep the best atabase frame F best(j) until the visibility score between the current query frame Fq J+n an the atabase frame falls below a threshol: V (Fq J+n, F best(j) ) < θ vis. (8) This threshol is base on accumulate motion energy over consecutive frames: θ vis = 0.9 V (F J+n q, F best(j+n) J+n 1 ) exp( I=J ( α mot E motion F I q, Fq I+1 ) ), We require an increasingly large visibility score as the number of frames increases (we set α mot = α is /200). This ensures that for fast motions we switch to the current best caniate F best(j+n) quickly an a transition occurs. 5.4 Retrieval for Face Region The escribe pipeline retrieves atabase entries for the whole boy except for the hea. The hea is processe separately by a simplifie version of the retrieval pipeline. The complete pipeline retrieves enough information to reconstruct complete spatio-temporally coherent target views (see Sec. 6), which is essential for the boy region. The quality criteria for the hea is slightly ifferent. Since human perception is tune to recognize faces, any form of visual iscontinuity in the renere target face woul be highly isturbing (such as a visible seam across the face). Spatial coherence is significantly more relevant here, an other forms of small errors such as missing texture are less visually objectionable. We o not use the full retrieval pipeline because even subtle errors in tracke hea poses in the atabase lea to starkly visible iscontinuities when several source images are combine to form the target face. Instea, in each query frame we fin the single best atabase frame F hea from which to copy over the entire hea texture. Since we want to copy the face region from a atabase frame where the face is completely visible, we choose the frame for which the visibility score V between target face facets an atabase face facets is highest. Similar to Sec. 5.3, we use the same F hea for as many subsequent query frames as possible. 6 Warping-base Vieo Synthesis We now want to synthesize a realistic image for each of the target frames. After retrieval (for the boy), we have the best matching frame F match from the atabase (an the corresponing camera view an pose / surface mesh as seen from that camera) for every query frame F q of the target sequence (an the query mesh an query camera at every time step). In aition, we also employ the frames F k (k C \ {match}, C being the set of atabase camera views) from all other camera views onto the same atabase pose. For every query frame, we also have the single best matching frame F hea for synthesizing the hea texture in the target view. For every query

6 Warping (a) (b) (c) Figure 4: (a) Surface mesh of a person use in query an atabase. (b) Vertices of the moel that are use as MLS warping constraints. (c) Vertices are labele accoring to the boy part upon which they lie (color=label). In MLS warping each vertex only affects the appearance of the respective boy part, except for the bounary regions between boy parts where warps are blene. frame, we also synthesize an initial guess for the appearance of the target frame by projectively texturing the query mesh with a static surface texture taken from all camera views at the first time step in the atabase, henceforth calle F static. The final texture for the query frame is then synthesize by moel-guie warping of texture information from the best matching atabase frames to the target. In practice, we first synthesize the target view for the hea, an subsequently for the rest of the boy (Fig. 5). For the hea, only a single source image F hea is warpe. In the case of the boy, frame F match contains most of the texture information neee to synthesize the target texture. However, ue to slight pose an camera ifferences between atabase an target, we warp aitional image ata F k to the target an blen with the existing result (Fig. 5). If any holes remain we blen with F static. Imagebase warping with this ata creates complete target textures even with visibility ifferences, completely invisible regions an geometry inaccuracies in the query an atabase meshes. We capitalize on the fact that we are given the same surface mesh in both the query an atabase poses (i.e., the same connectivity but ifferent poses). We use vertex corresponences between image projections of the same vertex in the source an target to etermine the warp. Using these corresponences as guiance, we use moving least square (MLS) [Schaefer et al. 2006] to obtain corresponing pixels in the source frame for all pixels in the target frame; i.e., to calculate motions of pixels between source an target frame. MLS warping weights every pixel neighborhoo in an image equally, an thus through a local coherence assumption creates smoothly interpolate warps even if actual warping constraints are sparser than the pixel gri (see Fig. 4(b)). In some regions, such as at occlusion bounaries between ifferent parts of the boy (e.g., an arm in front of the torso) this smooth interpolation may be unwante. However, if two neighboring boy parts are actually connecte, the warp shoul be smooth. The problem is that the original MLS metho is agnostic to scene geometry. Therefore, we evelop a strategy to enable smooth warping where it is wante an to prevent it across occlusion eges. First, we group human boy pixels into 16 segments base on the boy part information of the query 3D mesh (see Fig. 4(c)). The motions of pixels in each boy part are compute only from vertex corresponences in the same boy part. For every pixel lying on the bounary between connecte boy parts A an B, we compute two warp hypothesis: one uner the assumption that it belongs to A, an one that it belongs to B. The final warp is compute by weighte blening accoring to the istance to the boy part bounary. Finally, we warp the source frame to the target. There are specific implementation etails which Figure 5: Step-by-step warping-base synthesis of the result using MLS. First row: source frames use in warping (F match an F k). Secon row: results after warping the respective frame (above) to the target. Holes in the target view (blue) are continuously fille in. nee to be aresse uring the warping, an these are escribe in the following paragraphs. Missing Regions The best-matching atabase frame F match oes not usually contain all the information require to synthesize the target frame, an missing regions may occur in the target image after warping. As a consequence, F k are also use in our warping process. The warping algorithm performe on these frames is the same as the warping of F match. However, only missing pixels in the target frames are consiere for warping. A complete target image is obtaine after spatial blening of the warping results from ifferent images. This proceure is shown in Fig. 5. Any pixel to which the query mesh projects that is still untexture after all warps is colore from the statically texture result F static using Poisson image blening for compositing. Ghosting Due to small inaccuracies in the tracke 3D mesh, its projection will never align exactly with the atabase frames, an pixels may be matche with incorrect boy parts. These mismatches will lea to ghosting artifacts (see Fig. 6(b)). We etect the occurrence of these artifacts using epth information. Ghosting in the target frame is likely to originate from areas aroun epth iscontinuities in the atabase frame. In these regions, occlusion bounaries may not be correctly represente by the 3D geometry, e.g., pixels on an arm may be treate as pixels on the torso an warpe to torso pixels in the target frame. Base on this observation, we first efine occlusion bounaries in atabase frames by analyzing epth iscontinuities. We then ignore pixels near occlusion bounaries in the warping proceure. Detecting epth iscontinuities alone is not sufficient since the epth ifference may be very small if the arm is close to the boy, but ghosting may still occur. Thus, we ignore pixels if they are close to a epth an boy part label iscontinuity. Consequently, all pixels use are far from occlusion bounaries an have correct boy part labels, an ghosting artifacts are eliminate from our final results. Temporal Jumps Noticeable temporal jumps in appearance can occur when consecutive target images are synthesize from very ifferent atabase images. While we ensure that this happens as little as possible (see Sec. 5), it can still occur. We overcome this problem by temporally blening between consecutive sequences S r, S s. Namely, each target frame is synthesize twice from two

7 Transition Results (a) (b) (c) () (e) Figure 6: Ghosting is minimize by ignoring pixels near epth iscontinuities in the atabase frame. (a) the best matching atabase frame F match ignore pixels are shown in re. (b,c) are the warping results from F match without an with the eghosting metho. (,e) are the final results corresponing to (b) an (c). ifferent caniate sequences. A smooth transition is achieve by blening the two results (after spatial alignment with optical flow), weighte by their istances to the centers of the two sequences (see Fig. 7). The time uration of the blen can be ajuste to suit each sequence (see Sec. 7). Temporal blening is only triggere for segments of the target animation for which caniate sequences in the atabase were foun. Lighting Ajustment When compositing a target image, atabase frames are warpe an combine into a single view that may exhibit noticeable lighting changes. To compensate, we color ajust warpe frames to the alreay existing target texture by means of a linear color transform. Since the new texture to be warpe will usually have some overlap with the existing target texture, this transform can be estimate from overlapping regions. Color ajustment also nees to be applie over time. During a transition from one target frame to another, whenever a transition happens to a new best matching atabase sequence a similar color ajustment is compute to match the new atabase sequence to the color of the previous target frame. 7 Results We have evaluate our approach with three actor atabases: a male wearing a black t-shirt with a logo pattern an jeans (s1, Fig. 8 top), a male subject wearing a blue t-shirt with a strongly texture logo an normal pants (s2, Fig. 8 bottom), an a female subject wearing a re sweater an normal pants (s3, Fig. 8 mile). For each subject we recore a atabase of 10 multi-view vieo sequences, each one showing a basic motion such as walking, running, punching (one han punch, two han punch, up-own punch, left-right punch), jumping (one foot an two feet) or waving. Please refer to the supplemental vieo for a visualization. The atabase sequences are between 400 an 1000 frames long. Skeleton an mesh motion for each sequence were capture with the camera system an the algorithm escribe in Sec. 4. Also, as escribe in that section, we create a skeleton moel an a laser scan of each subject an rig the skeleton to the latter. We use 8 ifferent skeleton motion sequences as queries. Six of these were capture with the marker-less capture system (but are not part of the multi-view atabase): KungFu, Fight1, Fight2, March, Run1, an Run2. We also use two skeletal motion sequences that we ownloae from an online motion capture repository 2 : MJDance an Catwalk. These sequences are between 600 an 1000 frames long. All motions are quite ifferent from the 2 Figure 7: We avoi temporal appearance jumps by smoothly blening between two consecutive sequences S r (top) an S s (bottom). Re an blue are use to color-coe respective blening weights use for the final result in the mile. contents of the atabases. Using these ata, we create 13 new vieo animations by applying the query motions to the ifferent subjects, an by specifying ifferent camera paths. In the paper an the vieo we will refer to a synthesize sequence by the name s_m_c, meaning subject s performing motion m seen from camera c. We use Autoesk Maya TM to set up cameras an preview the queries, an also use the built-in retargeting tool if the motion ata came from a ifferent subject than the one in the target animation. The size of the synthesize characters in our sequences varie from to pixels. The figures in the paper an the accompanying vieo show a variety of synthesize animations. In aition to the synthesize character, we also rener synthetic shaows in the backgroun using the 3D moel from the target animation an 8 light sources. In all results, the appearance of the final vieo result looks plausible since the motions an the ynamic textural appearance are synthesize in a realistic an spatio-temporally coherent way. In sequences s2_march_c9 an s2_mjwalk_c6 one can see that plausible coherent patterns of fols are create on the clothes, an the ynamics of the texture patterns reflects the ynamics of the overall character motion. We also successfully synthesize the same target motion from two ifferent camera views (s3_catwalk_c7 an s3_catwalk_c8, as well as s1_run1_c2 an s1_run1_c3) showing the flexibility an freeom in animation synthesis that our approach provies. Similarly, we can generate in high quality the same target motion from the same camera view for two ifferent characters (s1_kungfu_c5 an s3_kungfu_c5). The vieo an Fig. 1 also show a composite of the sequence s3_catwalk_c11 with a real backgroun. Camera c11 was tracke with a commercial camera tracker an shaows of the person on the groun were renere in Maya. The final result looks convincing an emonstrates that it is feasible to insert vieo-base animations with user-esigne motions into real vieos. Our approach also succees if the frame rates of target animation an atabase vieos are not the same, but equal frame rates alleviates some problems. Tracking accuracies of the atabase sequences may vary, an thus some parameters of the metho may nee ajustment: for instance, the number of pixels aroun epth iscontinuities that are ignore for ghosting removal (4 for s2 an 8 for other subjects); also, the temporal winow in which transitions are blene varies between 6 an 10 frames. User Stuy We performe a user stuy with 32 participants to valiate the visual fielity of our synthesize vieo animations.

8 Each of the participants was shown a web page containing four vieos: one (vieo B) was a real vieo recore in our stuio, while the other three vieos were generate all from the same camera view with our system. To generate these animations, a test sequence, s1_run2_i2, was recore in our stuio, which was not part of the atabase use for synthesis. We then use the capture performance of that sequence from one of the input camera views, i2, as query to synthesize new results. For the synthetic sequences we create synthetic shaows an composite the results with a backgroun frame of the empty stuio to match the backgroun of the real vieo. Vieo C was synthesize by applying a static projective texture to the moel (F static from Sec. 6). Vieo A uses the best matching atabase frame (accoring to the visibility score from Sec. 5.1) plus six closest camera views from the same atabase sequence at the same time step to synthesize each target frame inepenently. Vieo D shows the result of our full pipeline. In the first experiment, participants were aske to evaluate how realistic an convincing the character in each vieo looks with respect to its motion an appearance on a scale from 1 (very realistic) to 5 (not realistic at all). All participants correctly ientifie vieo B showing the real footage as the most realistic character with average score 1.4. Our result was ranke as secon best (vieo D: 3.1), an is clearly preferre over the static texture (vieo C: 4.15) an the best match per frame of animation (vieo D: 4.12). This result is statistically significant with a one-way ANOVA p-value < A similar result was achieve in our secon experiment, where we aske the subjects to rank the vieos accoring to realism (1=most realistic, 4=least realistic). All subjects ranke the real sequence (vieo B) highest. 66% of the subjects ranke our result (vieo D) as the secon most realistic character. Valiation The user stuy confirms that all components of the retrieval an renering pipelines are crucial to the success of our metho. Not surprisingly, the user stuy shows that using a static texture for creating the final vieo is not consiere plausible, as any variation of surface appearance characteristic of real vieos is missing entirely. Using the best matching atabase frames at each query frame inepenently oes not take spatio-temporal coherence into account an leas to objectionable high-frequency brightness variations an incoherent texture patterns. Thus, retrieval of matching sequences is an important feature. A crucial renering component is temporal blening at sequence transitions, as a comparison on sequence s1_run2_i2 (labele valiation I in the secon supplementary vieo) shows. We show our full pipeline versus the pipeline without temporal blening. The vieo without blening contains objectionable jumps. Another comparison on s1_run2_i2 (labele valiation II in secon supplemental vieo) shows that simply lowering parameter l (see Sec. 5.3) to minimize the number of frames not matche to a caniate sequence from the atabase is not avisable. As oppose to our metho, the latter strategy creates too many temporal artifacts. This is why we resorte to the best frame matching from Sec. 5.3 only as a last resort after as-long-as-possible sequences from the atabase were matche. In all our experiments, the ratio of the target frames from frame-to-sequence matching to the target frames from sequence-to-sequence matching range from 1:9.99 to 1:2.39. Fig. 6 shows that eghosting is key to achieving plausible results. It effectively removes artifacts stemming from subtle geometry-toimage misalignments that woul otherwise spoil the impression. Timings We evaluate our metho on an Intel Core 2 with 2.66 GHz. The skeleton is manually embee in the mesh for each character which takes aroun 5 minutes. This is the only manual processing step in our pipeline (apart from generating query animations an viewpoints), an it only nees to be performe once for each actor. The atabase tracking takes aroun 10s per multi-view vieo time step an nees to be performe only once for each actor atabase. Given a atabase an a target animation, synthesizing the output vieo takes between 40s an 50s per frame. This inclues neighbor search (about 10s per frame), an warping+composition (about 30s per frame). Per-frame run times are mostly epenent on the resolution of the person in the target frame, as well as the number of frames for which temporal blening is triggere. All steps are performe automatically without any user intervention. Manual intervention is possible to inspect results an rerun computations with change parameters if esire. 7.1 Discussion Even though we cannot guarantee that our final results look exactly like the real person performing a specifie motion, our results are plausible. Our textures are spatio-temporally coherent, exhibit little aliasing, an reprouce plausible time-varying appearance patterns that match the ynamics of the target motion. This enable us to convey a very realistic impression of a real person. Our approach is subject to limitations. To a certain extent, the quality of the final results epens on the performance capture accuracy of the atabase. As a specific example, even subtle inaccuracies in tracke hea pose sometimes lea to artifacts in the final warping result. These artifacts coul be mitigate through an improve pose tracking of the hea, or through optional manual correction of the pose in the atabase. Currently, we o not emonstrate results with people wearing loose apparel, but we believe this is feasible. The performance capture approach coul track such atabase sequences, but we woul nee to ientify cloth on the boy moel an simulate its motion when generating the query. The latter is a challenging problem in itself. [Stoll et al. 2010] show an approach to ientify cloth in 3D performances an simulate new motions, an combining it with our approach may be an avenue worth taking. Our metho is geare towars minimizing objectionable spatial an temporal artifacts. However, they cannot be entirely prevente. Sometimes the result vieos have subtle brightness variations in some regions, which are ue to lighting ifferences between atabase frames. Our strategy to minimize these artifacts is to create uniform lighting in the atabase recoring environment. In future, more avance lighting ajustment strategies coul be researche. Some subtle implausible texture shifting is ue to nonoptimal skinning of the moel. In professional prouctions, it is very common that skinning weights are ajuste for every animation sequence iniviually. For time reasons we i not o this, an some artifacts may be lessene by this. While we can synthesize previously unseen motions, the synthesize animations may contain artifacts if the query motion is too issimilar to all atabase motions. In this case, one shoul augment the atabase with aitional sequences closer to the query. However, our examples suggest that with a atabase of moerate size, a ecent range of cameras an motions can be generate. Finally, we purposefully ecie to use an image-base MLS warping scheme to synthesize the final vieo rather than a projective texturing variant. A purely image-base warping scheme conceptually provies more flexibility. For instance, with MLS warping it is possible to warp pixel information to the target from areas close to the projecte moel bounaries, even if they fall outsie of the projecte moel in the atabase frames. This way, inaccuracies in the shape moel will be less noticeable in the target image. Also,

9 Figure 8: Iniviual frames from several of our result vieos for actors s1 (top), s3 (mile), an s2 (bottom). The inlays show the respective query skeleton poses use to synthesize the target frame. the warping scheme gives us more control over how many warping constraints to use in target texture generation an we expect the result to be smoother in the target for a large range of triangle mesh resolutions. However, in principle it is feasible to implement source-to-target frame warping with projective texturing. 8 Conclusion We have presente a new ata-riven approach for image-base synthesis of realistic vieo animations containing user-efine human motions seen from user-efine camera views. One key component of our approach is a retrieval algorithm to fin suitable images for synthesis of target textures from a multi-view vieo atabase of simple motions. Our approach exploits skeleton an shape knowlege for each input sequence, obtaine via marker-less performance capture. The retrieval component is esigne to assemble spatio-temporally coherent texture patterns in the target that are plausible epictions of the actor performing the user-esigne motion. An image-base warping approach composites the textural appearance in the target frame an is esigne to minimize temporal an spatial appearance artifacts, as well as lighting mismatches an ghosting. We have emonstrate the success of our metho with numerous examples an valiate it with a user stuy. Acknowlegements This work was complete when the first author was a visitor at MPI Informatik. Financial support came from the Intel Visual Computing Institute, Saarbrücken, Germany. Support in China came from the National Basic Research Project (No.2010CB731800), NSFC grants No & , an the 863 Program No. 2009AA01Z327. Support in the UK came from the EngD VEIV center at UCL an from BBC R&D. References BALLAN, L., AND CORTELAZZO, G. M Marker-less motion capture of skinne moels in a four camera set-up using optical flow an silhouettes. In 3DPVT. BALLAN, L., BROSTOW, G. J., PUWEIN, J., AND POLLEFEYS, M Unstructure vieo-base renering: Interactive exploration of casually capture vieos. ACM TOG (Proc. SIG- GRAPH), BARAN, I., AND POPOVIC, J Automatic rigging an animation of 3 characters. ACM TOG (SIGGRAPH) 26, 3, 72. BRADLEY, D., POPA, T., SHEFFER, A., HEIDRICH, W., AND BOUBEKEUR, T Markerless garment capture. ACM TOG (Proc. SIGGRAPH) 27, 3, 99. BUEHLER, C., BOSSE, M., MCMILLAN, L., GORTLER, S., AND COHEN, M Unstructure lumigraph renering. In SIG- GRAPH, CAGNIART, C., BOYER, E., AND ILIC, S Free-form mesh tracking: a patch-base approach. In Proc. IEEE CVPR, 1 8. CARRANZA, J., THEOBALT, C., MAGNOR, M., AND SEIDEL, H.- P Free-viewpoint vieo of human actors. In ACM TOG (Proc. SIGGRAPH).