Skinning Mesh Animations

Transcription

1 Skinning Mesh Animations Doug L. James Christopher D. Twigg Carnegie Mellon University Figure 1: Stampede! Ten thousand skinned mesh animations (SMAs) synthesized in graphics hardware at interactive rates. All SMAs are deformed using only traditional matrix palette skinning with well-chosen nonrigid bone transforms. Distant SMAs are simplified. Abstract We extend approaches for skinning characters to the general setting of skinning deformable mesh animations. We provide an automatic algorithm for generating progressive skinning approximations, that is particularly efficient for pseudo-articulated motions. Our contributions include the use of nonparametric mean shift clustering of high-dimensional mesh rotation sequences to automatically identify statistically relevant bones, and robust least squares methods to determine bone transformations, bone-vertex influence sets, and vertex weight values. We use a low-rank data reduction model defined in the undeformed mesh configuration to provide progressive convergence with a fixed number of bones. We show that the resulting skinned animations enable efficient hardware rendering, rest pose editing, and deformable collision detection. Finally, we present numerous examples where skins were automatically generated using a single set of parameter values. CR Categories: I.6.8 [SIMULATION AND MODELING]: Types of Simulation Animation; Keywords: deformation, skin, compression, mean shift, collision 1 Introduction Animating articulated characters such as virtual humans is a fundamental operation in computer graphics and interactive applications. Techniques for rigging character skins by weighting vertices to an Copyright 25 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) or permissions@acm.org. 25 ACM 73-31/5/7-399 $5. associated skeleton, or by interpolating example deformations, are widely used in video games and the computer animation industry. There are numerous reasons for their popularity: most skinning approaches are conceptually easy to understand and apply; they are capable of approximating interesting character shapes; and skinning can be hardware-accelerated on almost every commodity graphics card. However, the application of character skinning approaches has been almost entirely limited to objects with userdefined skeletons and rigid bones. In this paper, we show that skinning techniques can also be used to automatically skin deformable mesh animations, without any need for specifying skeletons or bones. By estimating proxy bone transformations and vertex weights for deformable shape sequences, we produce skinned animations that approximate the original deformable animation. Note that we do not estimate a hierarchical kinematic skeleton, nor are there reconfigurable joint angles. Skinned meshes provide an approximate articulated mapping from undeformed to deformed geometry at all animation frames, and we further encode displacement corrections to provide progressive convergence. The resulting skinned animations support hardware-accelerated rendering on commodity graphics hardware. We present several examples of mesh animations that can be compiled to run on existing graphics hardware, including interactive examples involving millions of animated mesh triangles (see Figure 1). In addition to fast hardware rendering, skinned mesh animations provide a deformable shape parameterization that can aid subsequent processing: displacement mapping the skin s reference mesh results in modifications to the entire animation; the animation skin can be used for straight-forward level-of-detail (LOD) generation; and the skinned models define local reference frames that can be used to orient bounding volume hierarchies for reducedcoordinate collision detection algorithms [James and Pai 24]. In summary, we show that a simple animation skinning technique can provide a familiar hardware-accelerated rendering format for mesh animations, as well as enable efficient animation processing. 399

2 2 Related Work Character skinning has a long history in graphics, and related techniques have been used extensively to provide intuitive animation controls. We refer the reader to the related work section of [Lewis et al. 2] for a nice summary. Most closely related to our work are several skinning by example approaches, that use input character poses to generate approximate and/or progressive skinning approximations. A key difference is that our input mesh examples are not required to have any skeletons, nor associated bone transforms. Pose space deformation [Lewis et al. 2] comprises one such family of approaches wherein character shapes (or local frame corrections) are interpolated as a function of character pose (see also [Allen et al. 22]). For hardware rendering, EigenSkin is an approach for generating a compact pose space deformation model for complex shapes, such as a hand [Kry et al. 22]. In that work, the reduced basis is generated by employing sensitivity analysis to move each joint in the hand independently, about a number of key hand poses, in order to infer the spatial structure of the hand displacement fields as a function of hand pose. A displacement correction model was then compressed in the rest pose to exploit redundancy. The resulting articulated reduced model included displacements and normals, and exhibited significant rank-reduction (much better than PCA on articulated data sets), but required a priori knowledge of joints and skinning weights. In comparison, our approach applies to general mesh animations as opposed to poseable articulated characters (see Figure 2). Figure 2: One of 27 hand poses with core bone triangles for 17 estimated bones (ε =.5). Although our method can skin mesh sequences with pre-existing skeletons ( 5), it s strength lies in optimizing non-hierarchical bone transforms for each frame ( 4), and avoiding pose space interpolation of progressive skin corrections ( 6). In contrast to pose space methods, simpler kinematic models, such as common linear blend skinning (c.f. single-weight enveloping (SWE), skeletal subspace deformation (SSD), etc.), can be fit to mesh examples, and do not require runtime data interpolation. Mohr and Gleicher [23] estimated single-weight skins with rigid character bones, with provisions made for adding additional bones, and they also estimated vertex weights, but without nonnegativity constraints. Multi-weight enveloping (MWE) [Wang and Phillips 22] provides better approximations than SWE, but at the cost of 12 weights/vertex/bone, instead of 1 weight/vertex/bone in SWE. Although we only use 1 weight/bone/vertex, our nonrigid affine bone transformations allow bones to squash and stretch to better approximate the mesh sequence. Like other approaches, MWE requires manual intervention to paint on bone influences, whereas our skin estimation approach is automatic. Animation compression approaches have been explored in the last half decade [Lengyel 1999; Briceño et al. 23; Guskov and Khodakovsky 24], including PCA compression of shapes [Sloan et al. 21], animations [Alexa and Müller 2; Karni and Gotsman 24], and parameterized animation compression [Hakura et al. 2]. The goal of our approach is not animation compression per se, but rather hardware-accelerated rendering using a very simple and common vertex shader technique called matrix palette skinning [Lindholm et al. 21]. Our skinned mesh animations are intended to produce skinned approximations with a large number of bone transforms (to support efficient hardware rendering). Nevertheless, our approximations can be quite good (even without using our articulated data reduction), so that our animation compression is quite competitive. Nevertheless, incremental coding approaches, such as [Gupta et al. 22; Ibarria and Rossignac 23], can be better suited to compression, and could be combined with our approach. Multi-resolution methods [Guskov and Khodakovsky 24] provide powerful decorrelation tools, but involve multi-resolution mesh reparameterization that can be difficult for models with complex base domains, e.g., our high-genus bridge example. Finally, special data structures exist for representing and displaying mesh animations [Shamir et al. 2], however for real-time rendering, the bottleneck is often data transfer to the graphics card. Our approach avoids this bottleneck by having a single-weighted mesh resident in graphics hardware, and only sending a few hundred floats of bone transforms to hardware each frame. 3 Skinning Mesh Sequences We seek to construct a vertex skinning transform, T = (T i ), that approximates a sequence of input meshes. A key point is that the mesh sequences we approximate do not have predefined bone transformations, and therefore we must first estimate proxy bone transforms before estimating vertex weights. Let the sequence of S meshes have deformed vertex positions P = (p 1,p 2,...,p S ), where p t R 3N for N vertices. For convenience, we will refer to the index t as time, even though the sequence may not have a time interpretation. Given these meshes, we seek a skinning transformation T t at each sequence step, t, that transforms the undeformed rest pose points, p, to approximate p t : p t T t p, t = 1...S, (1) with similar equations possible for vertex normals. We use linear blend skinning, where the transform for vertex i is T t i = b B i w ib T t b, (2) but where the transforms T t b need not be rigid. In the following sections, we first show how to compute robust estimates of proxy bone transformations, T t b, for our otherwise skeleton-free meshes ( 4), and then (in 5) we estimate the skin s vertex-bone dependencies, B i, and corresponding vertex bone weights, w ib. Note that if T t b are already given, e.g., by a kinematic skeleton model, one may proceed directly to Section 5. 4 Identifying Near-Rigid Structures using Mean Shift on Rotation Sequences Our first goal is to estimate proxy bone transformations { T t b } for the mesh sequence. Our insight is that clustering triangles with similar rotation sequences reveals the near-rigid structure of the mesh animation. For reasons discussed in detail in 4.2, we use a nonparametric clustering approach based on the mean shift algorithm (see [Cheng 1995; Comaniciu and Meer 22] for details). Note that the goal of this bone estimation section is not to partition the mesh into near-rigid components, but rather to estimate bone transformations { T t b } for the mesh sequence. 4.1 Triangle Rotation Sequences We estimate the near-rigid structure by identifying statistically significant rotation sequences of the mesh triangles. The rotation of triangle j at instant t relative to its rest configuration is computed 4

3 using the Polar Decomposition [Shoemake and Duff 1992; Golub and Loan 1996], which is also common in large deformation kinematics (see [Etzmuss et al. 23; Müller and Gross 24]). Let the indices of the triangle s vertices be i 1, i 2, and i 3, then the 3 3 nonorthogonal orientation matrix of triangle j at time t is O t j = [ v 21 v 31 ˆn] where v ab = (p t i a p t i b ) and ˆn = ( v 21 v 31 )/ v 21 v Similarly, let the nonorthogonal orientation matrix of rest triangle j be Õ j. From the triangle s deformation gradient F t j = Ot j (Õ j ) 1 R 3 3 (at time t), we extract the intrinsic triangle rotation, R t j, using the polar decomposition, F t j = R t j W t j (see Figure 3). Here W t j R 3 3 is the symmetric right stretch matrix that stretches the triangle before it is rotated. Figure 3: Triangle rotation sequences as estimated using the polar decomposition. Rotation sequences represent each triangle motion as a highdimensional point for subsequent mean shift clustering to estimate nearrigid components. Given the sequence of 3 3 relative rotation matrices for triangle j, we construct a point in d = 9S dimensions, z j R 9S, composed of concatenated rotation sequence vectors, ) z j = (vec(r 1 j ),..., vec(rs j ) (3) where vec(r) : R 3x3 R 9 converts the row-ordered 3 3 rotation matrix, R, to a row-major 9-vector. We now estimate the statistically important near-rigid components by carefully clustering the triangle rotation sequence points, {z j }. 4.2 Mean Shift Clustering of Rotation Sequences To estimate proxy bone transformations we seek to find (a) triangles that have similar rotation sequence points z, and also to discover (b) the number of different proxy bones required to resolve rotation sequence differences to some tolerance. Triangles can be partitioned using any number of clustering algorithms, but the quality of the resulting clusters for arbitrary skinning problems is unclear. Also, finding the right number of clusters for a certain skinning error can be cumbersome, e.g., using k-means clustering where k must be specified by the user. Given that we want to automatically skin a wide variety of input mesh sequences, we desire a robust clustering approach for estimating bone components. Background on Mean Shift Clustering: Given a list of n points, z j R d, the fixed bandwidth mean shift algorithm constructs the sample point estimator (density function), f (z) = 1 n n j=1 k ( z z j h 2), (4) where h is the bandwidth of some spherically symmetric kernel function, k( ). The gradient of this density function f (z) is then efficiently computed and used in a hill climbing process to map each input point, z j, to the nearest stationary point, z j, of the density function (see Figure 4). For robust nonparametric clustering, these resulting modes, z j, can be used to select cluster shapes using basins of attraction, and can therefore have very nontrivial shapes unlike k-means clustering where points are simply assigned to the nearest cluster center. The single bandwidth parameter, h, allows the number of clusters to be chosen in terms of a length scale in the input point space. Figure 4: Mean shift clustering of 2D points (images courtesy of [Comaniciu and Meer 22] c IEEE 22) (Left) Input 2D points, with color-coded cluster output; note the interesting oblong cluster shapes. (Right) Related density field, with trajectories from the mean shift gradient ascent algorithm. Red dots indicate the final mode centers used for proximity-based classification of mean-shifted points. Why Choose Mean Shift for Bone Estimation? The mean shift clustering algorithm (see [Cheng 1995; Comaniciu and Meer 22]) has many attractive features for bone estimation for skinning. First, mean shift clustering is robust to outliers, unlike k- means clustering which can be corrupted by a few bad points when computing each cluster s mean position. Robust approaches to compute averages are common when handling noisy real-world data, e.g., [Jones et al. 23]. For bone estimation, robustness is important because mesh sequences can have many triangles with independent and/or atypical motions, such that their rotation sequence z points are statistical outliers. Second, mean shift allows the number of clusters to be specified indirectly in terms of a physical scaling parameter, h, related to how different triangle rotations must be in order to belong to different bones. Specifically, we define a single intuitive rotation tolerance parameter, ε, such that h 9Sε for an S-mesh sequence, and mean shift automatically determines the correct number of clusters. In fact, except where stated explicitly, we skinned all examples in this paper using a single parameter value: ε =.5. With this single parameter value, we found that mean shift identified characteristic clusters of various number, size and shape for all examples (see Figure 6), supporting our claim that the method automatically produces reasonable results without excessive parameter tuning. Furthermore, we found that values around ε =.5 tend to extract the most bones that would be useful, and decreasing ε does not tend to produce statistically significant clusters. In practice, fewer bones may be desired, e.g., for real-time rendering, and ε can be increased to reduce bones, at the cost of increasing skinning error. Figure 5: Number of bones, B, versus ε (horse example) shows the tendency to reduce bones with increasing rotation tolerance, ε. We observe that ε =.5 identifies useful near-rigid bones structure to skin almost all examples. #BONES, B EPSILON One practical complication for mean shift clustering of rotation sequences represented as points in d = 9S dimensions, is that efficient multidimensional range searching is required to find closest neighbors of a point to evaluate (4). Consequently, we recommend the approach taken by [Georgescu et al. 23], wherein localitysensitive hashing (LSH) is used for fast approximate nearest neighbor searches, and we use their publicly available mean shift implementation (see reference for website). 41

4 Associating Triangles to Core Bones: Given the input points {z j }, and h, the output of the mean shift clustering algorithm is each triangle s mean-shifted point, z j, and B statistically significant modes, { z b } B b=1 associated with B discovered bones. Triangle j can then be associated with the bone whose mode z b is closest to their mean-shifted point, z j, with filtering performed for outlier points that are very far from statistically significant modes. In our implementation, we only assign triangles to their closest bone if D j z b z j 1 < h/4, and let T b denote the core bone triangles strongly associated with bone b. The total fraction of triangles associated with bones is denoted as the near-rigid fraction, η (,1] (ε =.5 assumed hereafter), and are displayed as colored (nonblack) triangles (see Figure 6). Finally, mesh sequences with higher η values tend to yield better skinning approximations. 4.3 Estimating Rigid Bones Mean shift helps identify the number of bones, B, and the core triangles {T b } B b=1 most strongly associated with each of the B bones. It remains to estimate the average rotation of the core bone triangles, T b, at each mesh sequence, t. Unfortunately, the cluster mode centers, { z b } B b=1, are just vectors in R9S, and the extracted 3 3 matrix sequences are not even rotation matrices of SO(3), and are therefore unsuitable for direct use as proxy bone rotations. To estimate bone rotations, we use a convenient result by Moakher [22], who proves that a meaningful average rotation can be computed as the arithmetic mean R of triangle rotations, followed by the unique projection onto SO(3) given by the unique polar factor in the polar decomposition of R [Golub and Loan 1996]. This average rotation is meaningful provided that det( R) >. In practice, we observe that this latter condition is satisfied when ε 1 for h = ε9s. For very large ε choices, det( R) > is not guaranteed, e.g., if R is the average of all mesh triangle rotations when B = 1. We use the area-weighted average triangle rotation, which is also easy to compute [Moakher 22]. Finally, we note that averages of spherical quantities, such as quaternions, have been discussed in the graphics community for some time (see [Buss and Fillmore 21]). Lastly, given the rotation matrix for bone b at time, t, the translation component of the bone transformation T t b is estimated using an area-weighted least squares fit to the motion of the centroids of the core bone triangles, T b. 4.4 Estimating Flexible Bones Unlike in character animation where bones are clearly rigid skeletal components, there is no reason that our bones can not flex at each frame. Since much greater accuracy can be achieved with flexible bones, and mesh sequences are free to define bones as they wish, we advocate flexible bones in our implementation. Mathematically, instead of representing each bone transform T by a rotation/translation pair, (R,v), we instead use (F,v) where F = RW is a 3 3 matrix involving a stretch in addition to a rotation, and v is a (different) translation. Given the core bone triangles, T b, we use least squares to estimate the flexible bone transform sequence (F t,v t ) that minimizes the distance from triangle centroids of the mesh sequence, and the reference model transformed using (F, v). This minimization can be done independently for each frame, and each bone, and the details are provided in Appendix A. Discussion: Given that we prefer flexible bones, one might wonder why we cluster R when F is ultimately required. Why not cluster F sequences? The answer is that when a triangle is used to estimate an F = RW matrix, the right stretch tensor, W, involves an incomplete observation of pre-stretch normal to the undeformed triangle. Consequently, triangles of different orientation will estimate different F matrices, even if they undergo the same spatial stretch and rotation. Therefore, rotations are a more suitable quantity to use to estimate near-rigid components with clustering. 5 Skin Estimation Given a mesh sequence and matching proxy bone transformations, { T t b }, we now show how to estimate which bones can influence each vertex, and what the corresponding vertex weights are. We stress that the skin estimation procedure (in this section) accepts as input arbitrary bone transforms, such as those of a kinematic skeleton model, and is in no way restricted to mean shift bone estimation. 5.1 Estimating Bone Influences Let B i be the vertex-bone influence set containing bone indices that influence vertex i. In our implementation the maximum number of per-vertex bones, β, is specified by the user, since it depends on the intended application. For example, in hardware rendering it is favorable to keep the number of bones per-vertex relatively small, e.g., β = 4. Specifying the bones that influence each vertex is traditionally a task often performed by 3D artists, e.g., by socalled painting bone weights on a mesh. Assigning bones to each B i influence set can be a difficult discrete model selection problem since it is often constrained by the maximum number of bones per vertex, β, or conditions on smoothness of the resulting mesh. We use a simple but effective model selection approach that picks the best β bones for vertex i that have the smallest square errors when used to individually predict deformed positions. Specifically, bone b s sum of squared errors, when predicting vertex i s position sequence, is γ bi = t=1...s p t i T t b p i 2 2, b = 1...B. (5) Because γ is only dependent on spatial quantities, bone influence sets also tend to have coherent spatial variations. 5.2 Estimating Vertex Weights Given vertex-bone influence sets, {B i }, the associated weights are computed using a least squares approach. Least squares is often used for vertex weight estimation in skinning by example approaches, e.g., [Wang and Phillips 22; Mohr and Gleicher 23]. Weights are constrained by the mesh sequence approximation equations, T t p = p t, which results in the over-constrained system of S equations with B = B i unknowns, which is of the form ( T t b p i)w ib = p t i, t = 1...S, (6) b B i A (i) w (i) = b (i), i = 1...N, (7) where for N vertices we have N different matrix problems. Additionally, we would like to enforce the affine constraint, w ib = 1, (8) b 42

5 dance snake horse camel elephant bridge Figure 6: Estimation of near-rigid bones and vertex weights: (Top) Reference triangle mesh of mesh sequence. (Middle) Estimated bones using mean shift classification of triangle rotation sequences into clusters. All examples use mean shift parameter, ε =.5. Here black denotes statistical outliers that are typically nonrigid triangles, e.g., shoulders (dance), floppy tails (camel), or bending joints. (Bottom) Vertex weighting to bones (TSVD weights). which could be done using constrained least squares. However, for reasons that will become clear shortly, in our implementation we will just consider the augmented system, [ ] ( ) ca (i) w (i) = cb (i) à (i) w (i) = b (i), (9) where c = 1/ b (i) 2 is a scaling parameter, and the last row of ones has been added to account for (8). Weight Over-Fitting: Direct solution of (7) (when A is nonsingular) can result in weights with potentially large positive and negative values. This is often referred to as over-fitting. Over-fitting can produce the best weights for the mesh sequence, and is adequate if only that mesh sequence is to be displayed. However, if the skinned model is to be able to approximate new poses (using new transforms), or support progressive corrections (as in 6), or allow rest pose editing (see Figure 15), or be simplified, then large (negative) weights should be avoided. Computing Vertex Weights: types of least squares methods: We solve (9) using two different 1. TSVD: For cases where over-fitting is allowed, we solve (9) using Truncated Singular Value Decomposition (TSVD), truncating singular values, e.g., at 1 5 à (i) 2, to avoid severe ill-conditioning. Higher thresholds can reduce overfitting, but negative weights still occur at practical thresholds. 2. NNLS: To limit over-fitting and avoid regularization parameters entirely, we recommend positive vertex weights obtained by solving the nonnegative least squares (NNLS) problem, Figure 7: Plot of sorted vertex weights computed using TSVD and NNLS (chicken example). TSVD allows negative weights that can lead to over-fitting. NNLS optimization provides a simple, efficient, and parameter-free way to get positive vertex weights that avoid over-fitting. VERTEX WEIGHT TSVD NNLS Finally, we always normalize (w ib ) b Bi by its sum to ensure (8), with the resulting correction being negligible in all of our examples. Note that by not using the affine constraint (8) to eliminate one weight variable from the matrix equation (7) (as in [Mohr and Gleicher 23]), we avoid preferential treatment of individual bone weights during TSVD or NNLS solves. These weights are compared in Figures 7 and 8. Figure 8: Over-fitting comparison of TSVD and NNLS vertex weights: (Left) Core triangles; (Middle) Weights computed using NNLS have a similar coloration to the core triangles since their weights are nonnegative; (Right) Significant over-fitting is observed with TSVD weights since they can use large negative values to better approximate the data. Note: bone colors are weighted by vertex weights, so exaggerated colors indicate over-fitting (also see TSVD weights in Figure 6). Solve à (i) w (i) = b (i) subject to w (i). (1) We use the efficient NNLS iterative solution approach of [Lawson and Hanson 1974]. Because of the nonnegativity constraint, positive weights tend to be sparse. In general, we can bound negative weights, w (i) δ by substituting w (i) = ( w (i) δ) into (9), and solving the modified problem Solve à (i) w (i) = b (i) + à (i) δ subject to w (i). (11) 6 Progressive Skin Corrections Skinned animations may provide sufficient accuracy for many applications, however, they can always be improved by increasing the number of bones (same as decreasing ε). Another way of improving accuracy is by using data reduction to progressively add corrective displacements and normals. These two approaches, increasing 43

6 bones or corrections, provide complementary improvements, with the latter being possible at runtime. T p t T Tp ~ Figure 9: Skin corrections are defined in the rest pose. Uq ~ ~ We use an approach analogous to EigenSkin [Kry et al. 22] where the displacement and normal corrections are added in the rest configuration of the mesh prior to applying the skin transformation (see Figure 9). Related approaches also appear in the literature (see [Lewis et al. 2]). Given the sequence of mesh points p t, and their skinned approximations, T t p, we obtain a sequence of S reference-frame displacement corrections, ((T t ) 1 p t p), that we perform data reduction on to obtain the representation, ((T t ) 1 p t p) = Ũ q t. Although any data reduction can be used, we use TSVD to generate a progressive approximation that converges to the exact mesh sequence as the rank of the displacement basis Ũ is increased (see Figure 1). This amounts to the articulated shape model, p t = T t ( p + Ũ q t ), t = 1...S. (12) where a tilde denotes pre-transform quantities defined in the undeformed configuration of the mesh (see Figure 9). Similar equations exist for normals (c.f. [Kry et al. 22]), n = F T (ñ + Ñ q n ), or n = F(ñ + Ñ q n). (13) As an alternative to progressive normal approximations, we can also use progressive approximations to other deformable appearance models. For example, in our implementation, we generate diffuse Precomputed Radiance Transfer (PRT) [Sloan et al. 22] for SMAs (and translating ground planes). This is done by computing PRT transfer vectors for each mesh, then either computing TSVD of transfer vectors (as in [James and Fatahalian 23]), or non-negative matrix factorization (NMF) [Lee and Seung 2] of pre-lit vertex colors, to produce a low-rank, deformable, illumination approximation that can be efficiently computed in a vertex shader. Results are shown in Figure 1 for NMF-factorized, deformable, monochromatic PRT. 7 Results Results for our examples are summarized in Table 1. Preprocess timings for are unoptimized, and were generated on an Intel Pen- p ~ Exact Rank 7 NNLS Rank 7 TSVD {T b } Figure 11: Over-fitting impedes corrections: (Left) An extreme frame (#273) from the Chicken Crossing animation; (MidLeft) our approximation (B = 22, β = 4, affine bones) with NNLS weights and only 7 displacement corrections is almost indistinguishable; (MidRight) approximation with TSVD weights suffers from over-fitting, and exhibits unpredictable displacement artifacts (see spike near beak). (Right) Core bone triangles. PERCENT POSITION ERROR, E Rank NNLS Rank 5 NNLS #BONES, B RMS UNIT NORMAL ERROR #BONES, B Figure 12: Animation error with increasing bones, B for horse model (flexible bones, NNLS weights, β = 4). (Left) Percent relative position error, E, as a function of bones, B, shown for Rank and Rank 5 corrections. While increasing the number of bones can improve accuracy, we observe that skin corrections are typically more effective when the near-rigid structure is resolved, e.g., at B=29 bones (ε =.5), or left alone at B=1. Note that the B=1 case is similar to the PCA compression approach of Alexa and Müller [2]. (Right) The RMS unit normal error as a function of B. tium 4 (2. GHz) processor with 1 GB RAM. Mean shift clustering clearly dominates all preprocess times, but is still under an hour in the worst case (ironically, the elephant example). Although the computational complexity of the approach is O(nSB) for n triangles, S frames, and B bones, the memory complexity is O(nS) for naïve in-core implementations, and can be inconvenient for large animations. Animation compression is not our ultimate goal, however SMAs can provide compact approximations for quasi-articulated models. Results are shown in Figures 12 and 13, and Table 1. Hardware implementation of SMAs is trivial given that any existing vertex program for matrix palette skinning can be used. Interactive performance can be achieved for very large scenes on an NVIDIA GeForce 68 GT (see Figure 1). Mesh levels of detail are produced using mesh simplification with additional vertex attributes [Garland and Heckbert 1998] for skinning weights, displacement corrections, PRT coefficients, etc. (see Figure 14). Exact Rank NNLS Rank 1 NNLS Figure 1: Progressive displacement corrections: (Left) Exact camel mesh in a non-reference pose; (Middle) The uncorrected NNLS skinned model already provides a good resemblance (B = 29, β = 4, affine bones); (Right) One displacement correction removes the majority of distortion. Comparison to vertex buffer objects: We compare the performance of our hardware-accelerated SMAs to an optimized OpenGL mesh sequence renderer based on Vertex Buffer Objects (VBO); each frame, the array of vertex positions and normals are transferred to static VBO memory, and then drawn as indexed geometry. Our test scene is composed of 15 SMAs (499 camels, 483 elephants, and 518 horses) and involves 71.5 million triangles. The brute-force VBO approach (.35 Hz) is more than six times slower than hardware-skinned SMAs with display lists (2.1 Hz). Also, SMAs can avoid excessive mesh storage, and the bottleneck of sending millions of triangles across our (8X) AGP bus. 44

7 Model Frames, S Vertices, N Triangles, n t meanshi ft t skin Bones, B η E (Rigid,TSV D) E (Rigid,NNLS) Compress E (Flex,TSV D) E (Flex,NNLS) Compress ball min.6 min (.).78 (.) ().2 () 71.2 snake min 1.3 min (.1).24 (.1) (.6).17 (.6) 32.2 dance min 1.4 min (.29) 1.34 (.29) (.17).77 (.18) 4.3 bridge min 4.6 min (.16) 27.1 (.15) (.1) 17.6 (.5) 12.8 camel min 1.2 min (.98) 8.17 (.93) (.54) 4.43 (.54) 18.9 elephant min 2.4 min (2.81) 12.5 (1.78) (1.1) 6.22 (.85) 19.6 horse min.6 min (1.46) 8.9 (1.38) (.59) 4.53 (.53) 15.9 chicken min 1.3 min (.2).84 (.17) (.57).45 (.14) 28.7 hands min.1 min (.64) 2.78 (.62) (.59) 2.63 (.56) 7.9 elasticcow min.7 min (1.54) 3.27 (1.55) (1.94) 3.18 (2.2) 27.6 clothhorse min.6 min (15.3) 44.1 (.88) ( ) 3.3 (.74) 21.3 flag (ε=.5) min 1.6 min (7.12) 21.5 (5.93) (147) 26.6 (38.9) 33.1 flag (ε=.1) min 2.4 min 1 (.92) 2.26 (1.26) 2.39 (1.25) (1.67) 1.58 (1.78) 14.4 Table 1: SMA statistics for models sorted by near-rigid fraction, η. Includes precomputation time for mean shift (t meanshi ft ) and model skinning (t skin ) with β = min(4,b). We report approximation errors in terms of percent distortion, E = 1% P P approx F / P P timeaverage F (as in [Karni and Gotsman 24]) for the four possible combinations of Rigid/Flexible bones with TSVD/NNLS weights. In addition to uncorrected skin error, we also report error for a rank-1 displacement correction model (in brackets); we underline the optimal model for the uncorrected and rank-1 cases. The general trend is that for models with a lot of near-rigid structure (high η), flexible bones are always better, with TSVD weights typically better when no corrections are used (rank ), but NNLS weights better with corrections a sign of TSVD over-fitting. For highly deformable models (η <.5), rigid bones become more favorable than flexible bones, probably because the latter over-fit at low η values. Finally, overall compression (Compress) of uncorrected skins tends to favor 7-float rigid bones (quaternion + translation) instead of 12-float flexible bones for sequences with larger SB values. Percent Position Error, E Flexible, NNLS Flexible, TSVD Rigid, NNLS Rigid, TSVD [Alexa and Mueller 2] Rest pose editing: SMAs allow displacement edits applied to the reference mesh to be automatically applied to all animation meshes. The displacement edits are defined in the local frame of reference so that they orient correctly as the mesh is animated. See Figure 15 and our video for an example. We use weights computed with NNLS solves to avoid over-fitting that would make rest pose editing less intuitive. Figure 15: Rest Pose Editing: (Left) Reference mesh before edit; (Middle) Displacement edited reference mesh; (Right) Animated pose using skinned mesh with displacement edit applied (flexible bones, NNLS weights, β =4) Rank of Displacement Correction Figure 13: Progressive approximations: Chicken Crossing animation (B = 22 bones at ε =.5) shown for increasing displacement correction rank. Flexible bones with NNLS weights provide the best overall SMA approximation, with passable low-rank (even zero-rank) approximations that make them suitable for efficient hardware rendering. Their stable displacement correction behavior is in contrast to flexible bones with TSVD weights, which has irregular convergence (also see Figure 11). For comparison, we plot (blue curve) the PCA approximation of Alexa and Müller [2] (B=1 flexible bone) which requires much higher rank to achieve similar accuracy, e.g., rank 37 (B = 1) is comparable to the rank 7 (B = 22), NNLS, flexible bone approximation shown in Figure 11. Note that the B=1 approach can provide better overall compression for long low-rank animations than the frame-heavy B=22 case, but it is inconvenient for hardware rendering. Figure 14: SMA Simplification: (Left) Rest pose horse (16843 triangles) and simplification (21 triangles); (Right) horses in skinned pose (flexible bones, NNLS weights, β =4). Highly deformable models: We observe that the near-rigid triangle fraction, η, (ε =.5) provided by mean shift reliably indicates models with sufficient near-rigid structure for skinning to be successful. Datasets with low η values tend to indicate highly deformable models for which skinned approximations tend to be inefficient. Examples include a collapsing cloth horse (η =.12; see Figure 16), or a flag flapping in the wind (η =.42; see Figure 16). One characteristic problem, is that highly deformable regions, such as the flapping flag edge, lack sufficient near-rigid structure for skinning (see Figure 17) unless extremely many bones are used. It appears that datasets with η(ε =.5) values below.5 are non-robust candidates for skinning, and that progressive skin corrections can be expected to perform no better (and often much worse) than direct data reduction methods. Deformable collision detection can also be accelerated for SMAs by exploiting the articulated mesh parameterization. By partitioning mesh triangles into regions most strongly associated with each rigid bone, we can build rigid local frames of reference for each bone s mesh region. By computing the displacement correction model U (b) q (b) for bone b s triangle s vertices, we can construct a Bounded Deformation Tree [James and Pai 24] on each region (see Figure 18). The method is effective because building BD-Trees on local mesh regions can exploit the local frame of reference and simpler resulting deformation models. Such approaches 45

8 {T b } Weights Rank NNLS Rank 5 NNLS Exact Figure 16: Skinning highly deformable animations, such as a collapsing cloth horse, are particularly hard since the distribution of estimated core bone triangles (far left) can be too sparse (small η =.12). In general, highly deformable animations yield unpredictable SMA approximations often no better than direct data reduction. ε =.5 ε =.1 Rank NNLS (flat shading) Rank NNLS (Rank 3 normals) Exact Figure 17: A flapping flag has deficient bone structure at ε =.5, but is reasonably approximated by 1 bones (ε =.1). Non-smooth SMA reconstructions are typical of highly deformable animations, but can be partly hidden using normal corrections, or data-driven appearance models. can reduce bounding volume hierarchy updating costs, e.g., in physical simulations (see Figure 18). models tend to avoid over-fitting, making their reconstructions stable under small displacement modifications introduced by progressive displacement corrections, SMA simplification, and rest pose editing. Finally, the method is well-suited to real-time hardware rendering of mesh animations of quasi-articulated phenomena, such as those shown in Figure 1, and skinned animations can be used with output-sensitive collision detection methods for deformable models. Acknowledgments: We would like to thank the NSF (CAREER-43528), Pixar, and NVIDIA for generous support; the anonymous reviewers for helpful suggestions; the Rutgers Robust Image Understanding Laboratory for providing their mean shift implementation; Bob Sumner, Jernej Barbič, Hector M. Briceño, Matthias Müller, and co-authors for mesh data. The chicken character was created by Andrew Glassner, Tom McClure, Scott Benza, and Mark Van Langeveld. This short sequence of connectivity and vertex position data is distributed solely for the purpose of comparison of geometry compression techniques. Appendix A Flexible Bones using Least Squares For completeness, we provide equations to estimate bone b s affine transform, T b = (F,v), for a given mesh sequence instance. We define T as the transform that minimizes the triangle area-weighted sum over all core bone triangles, T b, of the squared position errors between transformed reference centroids, T b c, and the deformed centroids, c. Specifically, we minimize Φ = a i c i F c i v 2 2 = a i (c ix F xy c iy v x ) 2 (14) i T b i,x y where x,y = 1,2,3. Taking the derivatives of Φ w.r.t the 12 unknowns, F xy and v x, and setting each to zero, leads to 12 sparse equations in 12 unknowns, iz a i c iz c iy F xz + ( i a i c iz F xz + ( iz i a i c iy )v x = a i c ix c iy, x,y = 1,2,3; i a i )v x = a i c ix, x = 1,2,3, i where z is summed over the three components. The (F,v) solution is the optimal flexible bone in an area-weighted least-squares sense. Figure 18: SMA Collision Processing: (Top) Bounded Deformation Trees can be precomputed on near-rigid components (or any desirable mesh partitioning) to provide output-sensitive collision detection for SMAs; (Bottom) SMA collisions driving a physical simulation. 8 Summary We have presented an automatic method for constructing skinned mesh approximations to parametrically coherent mesh sequences. Our experiments indicate that (a) mean shift clustering of triangle rotation sequences is a robust estimator of near-rigid mesh structure, and (b) nonrigid affine bone transformations combined with positive vertex weights (computed using nonnegative least squares (NNLS)) lead to accurate approximations for examples with sufficient near-rigid structure (e.g., η >.7). The resulting skinned References ALEXA, M., AND MÜLLER, W. 2. Representing Animations by Principal Components. Computer Graphics Forum 19, 3 (Aug.), ALLEN, B., CURLESS, B., AND POPOVIĆ, Z. 22. Articulated Body Deformation From Range Scan Data. ACM Transactions on Graphics 21, 3 (July), BRICEÑO, H. M., SANDER, P. V., MCMILLAN, L., GORTLER, S., AND HOPPE, H. 23. Geometry Videos: A New Representation for 3D Animations. In SCA 3: Proceedings of the 23 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association, BUSS, S. R., AND FILLMORE, J. P. 21. Spherical Averages and Applications to Spherical Splines and Interpolation. ACM Transactions on Graphics 2, 2 (Apr.),

9 CHENG, Y Mean Shift, Mode Seeking, and Clustering. IEEE Trans. Pattern Anal. Mach. Intell. 17, 8, COMANICIU, D., AND MEER, P. 22. Mean Shift: A Robust Approach Toward Feature Space Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24, 5, ETZMUSS, O., KECKEISEN, M., AND STRASSER, W. 23. A fast finite element solution for cloth modelling. In Proceedings of Pacific Graphics, GARLAND, M., AND HECKBERT, P. S Simplifying Surfaces with Color and Texture using Quadric Error Metrics. In IEEE Visualization 98, GEORGESCU, B., SHIMSHONI, I., AND MEER, P. 23. Mean Shift Based Clustering in High Dimensions: A Texture Classification Example. In Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV 23). Adaptive mean shift code available at GOLUB, G. H., AND LOAN, C. F. V Matrix Computations, third ed. Johns Hopkins University Press, Baltimore. GUPTA, S., SENGUPTA, K., AND KASSIM, A. A. 22. Compression of dynamic 3D geometry data using iterative closest point algorithm. Comput. Vis. Image Underst. 87, 1-3, GUSKOV, I., AND KHODAKOVSKY, A. 24. Wavelet Compression of Parametrically Coherent Mesh Sequences. In SCA 4: Proceedings of the 24 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, ACM Press, HAKURA, Z. S., LENGYEL, J. E., AND SNYDER, J. M. 2. Parameterized Animation Compression. In Rendering Techniques 2: 11th Eurographics Workshop on Rendering, IBARRIA, L., AND ROSSIGNAC, J. 23. Dynapack: Space-Time compression of the 3D animations of triangle meshes with fixed connectivity. In SCA 3: Proceedings of the 23 ACM SIG- GRAPH/Eurographics Symposium on Computer animation, Eurographics Association, JAMES, D. L., AND FATAHALIAN, K. 23. Precomputing Interactive Dynamic Deformable Scenes. ACM Transactions on Graphics 22, 3 (July), JAMES, D. L., AND PAI, D. K. 24. BD-Tree: Output-Sensitive Collision Detection for Reduced Deformable Models. ACM Transactions on Graphics (SIGGRAPH 24) 23, 3 (Aug.). JONES, T. R., DURAND, F., AND DESBRUN, M. 23. Non- Iterative, Feature-Preserving Mesh Smoothing. ACM Transactions on Graphics 22, 3 (July), KARNI, Z., AND GOTSMAN, C. 24. Compression of soft-body animation sequences. Computers & Graphics 28, 1, KRY, P. G., JAMES, D. L., AND PAI, D. K. 22. EigenSkin: Real Time Large Deformation Character Skinning in Hardware. In ACM SIGGRAPH Symposium on Computer Animation, LAWSON, C. L., AND HANSON, R. J Solving Least Square Problems. Prentice Hall, Englewood Cliffs, NJ. LEE, D. D., AND SEUNG, H. S. 2. Algorithms for nonnegative matrix factorization. In NIPS, LENGYEL, J. E Compression of Time-dependent Geometry. In SI3D 99: Proceedings of the 1999 Symposium on Interactive 3D Graphics, ACM Press, LEWIS, J. P., CORDNER, M., AND FONG, N. 2. Pose Space Deformations: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation. In Proceedings of ACM SIG- GRAPH 2, Computer Graphics Proceedings, Annual Conference Series, LINDHOLM, E., KILGARD, M. J., AND MORETON, H. 21. A User-Programmable Vertex Engine. In Proceedings of ACM SIGGRAPH 21, Computer Graphics Proceedings, Annual Conference Series, MOAKHER, M. 22. Means and Averaging in the Group of Rotations. SIAM J. Matrix Anal. Appl. 24, MOHR, A., AND GLEICHER, M. 23. Building Efficient, Accurate Character Skins From Examples. ACM Transactions on Graphics 22, 3 (July), MÜLLER, M., AND GROSS, M. 24. Interactive Virtual Materials. In Proceedings of Graphics Interface. SHAMIR, A., BAJAJ, C., AND PASCUCCI, V. 2. Multiresolution dynamic meshes with arbitrary deformations. In Proc. of Visualization, IEEE Computer Society Press, SHOEMAKE, K., AND DUFF, T Matrix Animation and Polar Decomposition. In Graphics Interface 92, SLOAN, P.-P. J., III, C. F. R., AND COHEN, M. F. 21. Shape by Example. In ACM Symp. on Interactive 3D Graphics, SLOAN, P.-P., KAUTZ, J., AND SNYDER, J. 22. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments. ACM Transactions on Graphics 21, 3 (July), WANG, X. C., AND PHILLIPS, C. 22. Multi-Weight Enveloping: Least-Squares Approximation Techniques for Skin Animation. In ACM SIGGRAPH Symp. on Computer Animation,