Overview on Mocap Data Compression

Transcription

1 Overview on Mocap Data Compression May-chen Kuo, Pei-Ying Chiang and C.-C. Jay Kuo University of Southern California, Los Angeles, CA , USA Abstract The motion capture (mocap) data have been widely used in motion synthesis applications for education, medical diagnosis, entertainment, etc. The richness of a mocap database is essential to motion synthesis applications. The limitation on the network bandwidth or storage capacity leads to constraints on the size of the mocap collection to be used, thus it is desirable to develop an effective compression scheme to accommodate a larger mocap data collection for higher quality motion synthesis. Two compression schemes recently developed in our lab will be discussed and presented in this paper. They are designed to address different objective functions with different application purposes. The first one aims at preserving nearly lossless content. With the new temporal and spatial information handling, we can now achieve 50:1 compression, which is 2.5 times better than the state-of-the-art techniques. The second one is perception based compression, which aims for further compression with visually pleasant quality. We achieve on average 150:1 compression ratio in this framework. I. INTRODUCTION AND PREVIOUS WORK The motion capture (mocap) data are obtained by recording the temporal trajectories of position sensors mounted on subjects. The 3-dimensional (3D) position of each sensor is tracked, and as the information is mapped to the skeleton of the subject, we can calculate temporal variation of the rotation of each joint in the skeleton. In general, the Euler representation is used to describe the rotation of each joint, i.e., the rotated angle against the X-, Y-, and Z-axes. The temporal trajectory of each parameter defines a degree of freedom (dof) curve. As a result, the mocap data can be presented in two formats: i) marker positions (.c3d format) and ii) joint rotations (.amc format). The size of a mocap clip is proportional to the number of markers or the number of joints. To uniquely define the rotation of a joint, three markers are required. Thus, the size of a mocap file in marker format is three times of that in joint format. The mocap data have been widely used in many motion synthesis applications for education, medical diagnosis, entertainment, etc. In the entertainment business, the synthesized motion can be easily ported to different models to animate virtual creatures. Although quite a few physical based (as opposed to the data-driven) motion synthesis methods have been proposed, the naturalness of synthesized motion is still not yet well quantified. It is easier to ensure the naturalness of motion using mocap data. As a result, most physical based motion synthesis methods actually adopt a hybrid approach, which still demands some mocap data to assist. The richness of a mocap database is essential to motion synthesis applications. In general, the richer the collection, the higher quality the synthesized motion. Since there is limitation on the network bandwidth or storage capacity, there are constraints on the size of the mocap collection to be used. It is desirable to develop an effective compression scheme to accommodate a larger mocap data collection for higher quality motion synthesis. Previous research on mocap data compression [1]-[5] consists of the following 3 ingredients. Motion categorization With proper categorization, one can cluster similar motion clips together so that the PCA application is more effective. However, it is difficult to automate the categorization process so that it still demands human intervention nowadays. The categorization step in previous research is considered part of the pre-processing and thus not counted in the total execution time. Principal Component Analysis (PCA) [4] The PCA technique has been used to represent and encode animation sequences before. For mocap data compression, the PCA technique is used for dimension reduction. For the PCA to be effective, it requires a reasonable amount of similar data in the training set. Inverse kinematics (IK) in post-processing [6] An inverse kinematics mechanism can be adopted to encode the environmental contacts as a post-processing step to ensure clean contact motion. However, this mechanism might over-stretch some joints to result in an artifact known as the knee-popping artifact. There are two main shortcomings in the existing mocap data coding algorithms. First, as automatic motion clustering is difficult, it is usually a semi-automatic process that demands human guidance. Second, the amount of similar mocap data required for PCA training may not be available in a general context. In this paper, two compression schemes recently developed in our lab will be discussed and presented. They are designed to address different objective functions with different application purposes. The first one aims at preserving nearly lossless content. With the new temporal and spatial information handling, we can now achieve 50:1 compression, which is 2.5 times better than the state-of-the-art techniques. The second one is perception based compression, which aims for further compression with visually pleasant quality. We achieve on average 150:1 compression ratio in this framework. To overcome these two shortcomings, the proposed methodology in this research does not demand any prior knowledge on motion category, and can be applied to generic mocap data. Being similar to previous work, we include the IK technique APSIPA. All rights reserved. Proceedings of the Second APSIPA Annual Summit and Conference, pages , Biopolis, Singapore, December 2010.

2 in the post-processing step. The rest of the paper is organized as follows. An overview of the problem and our proposed framework are presented in Section II. The two proposed compression schemes are introduced in Section III and IV. Experimental results are illustrated in Section V. Finally, conclusion and future work are summarized in Section VI. (a) Human skeleton II. FRAMEWORK OVERVIEW dof value time (second) (b) Human mocap data Fig. 1. Mocap data generation: (a) human skeleton, (b) exemplary mocap data A standard human model used for mocap data is shown in Fig. 1(a). In the standard mocap data format, each dof is captured at 120 frames per second (fps). The data size of each dof, q i (t), at each frame is four bytes so that its precision can be as high as out of the 60 dofs are Euler angles, whose ranges are in [ π, π). The three dofs representing the global position have no range limit in principle. However, in practice, they are bounded by the area of the motion capture environment. For a motion clip containing N dofs, and is of L frames, the full representation of this motion is q 1 (1) q 1 (2) q 1 (L). q 2 (1) q 2 (2).. q2 (L) q N (1) q N (2) q N (L) where q i (k) is the value of dof i at frame k. Some exemplary dof curves are shown in Fig. 1(b). In the context of lossy data compression, the rate-distortion (R-D) trade-off is often expressed in form of J = αd + λr, where R denotes the coding bit rate and D denotes the distortion. A good compression method is designed to minimize the above objective function. While it is straight-forward to evaluate R, a proper representation of D is still an open problem, which depends on the application. In some applications say medical imaging, we may demand that the compressed result to be as close to ground truth as possible. As to the mocap database management, it is desired to have a preview feature so that users can have a quick view on the data in the database. The coded result does not have to be close to the ground truth data on the frame-byframe basis, but perceptually similar. To meet this demand, we develop two motion compression schemes. The precisionbased scheme provides a coding result which is close to the ground truth in terms of low mean-squared errors (MSE). The other, the perception-based scheme, offers a coding result that is perceptually similar to the original motion and achieves a very high compression ratio. With this perception-based compression scheme, users can have an efficient preview of various motion clips stored in the database. III. PRECISION-BASED COMPRESSION Mocap data compression can be achieved by reducing their spatial and temporal redundancies. The N-dimensional dof curves at the same time instance define a N-dimensional vector, which is called a frame. Each frame is either directly coded or predicted by a reference frame. By borrowing the term from video coding, the former is called an I-frame while the latter is called a B-frame. The accuracy of I-frames will affect that of B-frames since B-frames are predicted by I- frames. Generally speaking, it is worthwhile to spend more bits on the I-frames than B-frames. Representative frames are selected to serve as the I-frame. The selection of proper I- frames is important since it affects the compression ratio and quality of the motion. As for bit allocation, assigning some bits to residual coding may bound the error more effectively than assigning all bits to the quantized dof vector. B-frames are also preserved with residual coding. For B-frames, the bits spent on residual coding not only enhance the quality, but also allow a simpler interpolation algorithm, which may be advantageous for its low complexity. With residual coding, errors introduced in the quantization and prediction stages can be controlled, thus guaranteeing the quality of compressed mocap data to some degree. Our proposed coding system consists of the following three modules. 1) Temporal Sampling We will develop a rule to select proper frames to serve as I-frames. 2) Vector quantization in selected I-frames Each I-frame is vector quantized. For human mocap data, we can exploit the symmetry of human skeleton to break down a pose, and apply different codebooks to different parts of the human body. 3) Interpolation and residual coding We will describe a scheme to predict the B-frames based on their adjacent I-frames via interpolation. Furthermore, the prediction error of B-frames will be encoded, which is called the residual coding. The above three coding modules are examined in detail in Sections III-A, III-B, and III-C, respectively. A. Temporal Sampling To sample a curve at a fixed interval is straightforward. It is however difficult to pre-select an optimal interval length. If the interval is too long, it is difficult to interpolate the dof 854

3 values accurately based on sampled points. If the interval is too short, one may sacrifice the coding performance gain. Consider the example in Fig. 2, where T is a pre-selected interval length, circles denote sampled points and dotted curves are predicted dof curves by interpolating sampled points. The prediction in the last interval of Fig. 2(a) is far from the ground truth. The interval in Fig. 2(b) is shorter than T, but the prediction is still not good in some intervals. The predicted curve in Fig. 2(c) is satisfactory. However, as compared with Fig. 2(d), we see that six out of the thirteen points are redundant (a) Maximum Error = Fig. 3. The data structure of a partitioned interval can be stored in form of a binary tree in (a), which corresponds to the case shown in (b). (b) T T T (a) (b) (c) (d) Fig. 2. The solid line is the ground truth, the dotted line is the approximated curve, and circles are sampled points. The predictions in (a) and (b) are worse than (c) and (d). Fixed-interval sampling is used in (a)-(c) and adaptive sampling is used in (d). The above example suggests the use of adaptive sampling. Comparing to fixed-interval sampling, extra bits has to be spent on storing the temporal information of I-frames. We adopt dyadic sampling to minimize the requirement of the bits to be spent. The dyadic sampling algorithm aims to record the positions of I-frames where the maximum error of the coded dof curve is bound by a certain threshold T. Let the I-frame index set be S and the motion duration be L frames. First, we take the first and the last frames as the initial I-frame index set, i.e., S = {1, L}. Then, we interpolate the coded curve using values at the two end points and evaluate the maximum error of the entire interval. If it is higher than T, we take the middle point as the new sample, i.e., S = {1, L/2, L} so that there are two sub-intervals. For the sub-interval that has its maximum error higher than T, we add its middle point to the sample point set. Generally speaking, given k samples in S, there are k 1 regions. For regions with the maximum error less than T, they will remain the same. For regions with the maximum error larger than T, they will take the center of the sub-interval as a new temporal sampling position. All information in a dyadic temporal sampling scheme can be stored in form of a binary tree. To give an example, the binary tree in Fig. 3(a) is obtained from the example shown in Fig. 3(b). In the memory size is constrained, we can divide the whole tree into multiple sub-trees and handle each sub-tree separately. For example, if the memory can accommodate one half of the duration of the motion in Fig. 3 only, we can drop the root node and work on two sub-trees one by one. This configuration can be handled by the program automatically. Each region in the algorithm is represented by a node. If the maximum error of the region is within the threshold value T, it will not be split furthermore and the corresponding node will have no children node. It is a leaf node and labeled by 0. The non-leaf nodes are labeled by 1. After labeling, we have (a) histogram of the dof (b) histogram of the dof differences Fig. 4. (a) The histogram of dofs and (b) the histogram of dof differences. a binary tree contain 0s and 1s only. Giving the property that nodes of value 1 have two children, and nodes of value 0 have 0 children, we can represent this tree using the string result of the depth first search. The binary string can be encoded using the QM coder. B. I-Frame Coding: Vector Quantization Human motion is constrained by the skeleton-muscle system of the captured subject. We observe that some dofs are highly correlated while others are less correlated. Thus, we divide dofs into multiple groups and handle them separately according to the group characteristics. Specifically, we adopt the notion of Labanotation and decompose a pose into five subposes. Furthermore, we apply the vector quantization (VQ) technique to the space formed by each sub-pose. We perform the forward differencing operation on the dof values to enhance the coding performance. Fig. 4 shows the histogram of dofs in (a) and dof differences in (b). Clearly, the distribution in (b) is more skewed than (a). Thus, the entropy of dof differences is lower, which indicates a higher compression ratio if we encode dof differences rather than dof values directly. 855

4 C. B-Frame Coding: Interpolation and Residual Coding The dofs of natural human motions are continuous and often smooth in the time domain. This property can be exploited to interpolate B-frames based on sampled I-frames. In this section, we will discuss the interpolation techniques, which should be easy to implement, robust and close to the ground truth. Besides, in contrast with traditional motion interpolation methods which demands a decent approximation accuracy, we introduce a post-processing step to reduce the residual errors. That is, we allow the coding of interpolation errors to control the degree of interpolation errors at different time instances. This procedure is called the residual coding. In other words, we can use higher bit rates to trade for better coded dof curves. 1) Interpolation Techniques: While high order (> 3) polynomials usually do not fit the context of our interest since they might introduce extra bumps in the motion curve. Two interpolation techniques are considered below. Linear interpolation Linear interpolation is a method of curve fitting using linear polynomials. It connects two sample points with a straight line. It is the easiest and sometimes used to interpolate short period of motion in previous work. Spline interpolation Spline interpolation is the most common interpolation strategy applied to the mocap data. A spline is a special function defined piecewise by polynomials. Typical choices include the B-spline and the Hermite spline. 2) Residual Coding: The residual error is the difference between the interpolated and the actual values. There are two types of residual errors: errors in I-frames and B-frames. We use VQ to encode these residual errors with two different codebooks as described below. I-frame residual coding For a sub-pose codebook of I-frames, we may train a sub-pose residual codebook correspondingly since the use of sub-pose codebooks will fit each group of dofs better. However, we observe that all sub-pose residuals can share the same codebook. This can be explained by the following arguments. The limb-dependent property is primarily captured by the I-frame codebook so that the residual becomes limb independent. B-frame residual coding Errors are introduced in the interpolation stage. B-frame residuals can be encoded to compensate these errors. Another tree-structured codebook can be trained based on the B-frame residuals. IV. PERCEPTION-BASED COMPRESSION It was observed in [7] that two animated motion sequences can be perceptually indifferent although their MSE is large. Actually, if there is no abrupt change in energy, human perception can accommodate minor differences. When the maximum error is bounded, it is often that two motions are perceptually similar (even their MSE is high). Based on the above observation, we propose a perception-based mocap data Pre-Processing Segmentation & Normalization normalized segments temporal info spatial info Coding Vector Quantization Dyadic Encoding Scalar Quantization Compressed Data Fig. 5. The block-diagram of the perception-based mocap data coding algorithm. coding algorithm. Although the coded dof values may have a larger MSE value, the motion is perceptually similar to the original one. The following two properties can be exploited to develop the perceptual-based mocap data coding algorithm. If the timing information of the key-poses is shifted slightly, the resulting motion is still perceptually similar to the original one. If the poses in a frame are modified slightly, the resulting motion is still perceptually similar to the original one. The second property was exploited to justify the coding algorithms developed in section III, too. In this section, we explore the above two properties to achieve a higher compression ratio yet achieving the perceptual similarity. The block-diagram of the perception-based mocap data coding algorithm is illustrated in Fig. 5. The functions of the three major modules in the block-diagram are described below: Pre-processing module The first module performs filtering on the input dof data to produce smooth curves so that they will not be oversegmented due to small noise in the next module. Segmentation and normalization module The second module partitions a dof curve into multiple segments by the sign of the velocity. In other words, the motion is segmented at local extrema and each segment is normalized to a block of height 2π (i.e., the dof range is normalized to 2π.) and width 120 frames (i.e., 1 second.). Coding module The third module consists of three operations which can be executed in parallel. They are: VQ applied to the normalized curve; SQ applied to the normalization range parameter; Dyadic coding applied to the temporal information of sampled frames. Finally, entropy coding is applied to VQ and SQ results. To compress a curve like the one in Figure 6, the procedure works as follows. First, the segmentation module partitions this curve based on local extrema, where each segment can be covered by a bounding box. The width of the bounding box gives the temporal information, and its value will be rounded 856

5 TABLE I SUBJECTIVE TEST RESULTS ON PERCEPTION-BASED MOCAP DATA COMPRESSION. time playground follow path run/jog walk jump Similarity Naturalness basketball dance cartwheel Tai-Chi boxing Similarity Naturalness MSE bound Fig. 6. An example to illustrate the perception-based algorithm. 0.3/Enhanced 0.1/Enhanced 0.3/Basic 0.1/Basic RD comparison 0% 20% 40% 60% 80% 100% % of Benchmark Database Fig. 7. Percentages of the data in CMU mocap database which are able to achieve designated MSE in each compression ratio level. to the closest dyadic number. The height of the bounding box is the range of the dof, and its value can be quantized with a SQ. Second, we normalize all bounding boxes to a unit square with their height and width set to 1 and encode all dof curves inside the unit square with the VQ scheme. In other words, each dof curve will be approximated by another curve, which is the closest codeword, from the codebook. Finally, the outputs from SQ and VQ can be further compressed using the entropy coding. A. Precision-based Scheme V. EXPERIMENTAL RESULTS We compare the our proposed algorithm (labeled as enhanced) to previous works (labeled as basic) and show their coding performance at two different error levels (i.e., 0.1 and 0.3) in Fig. 7. Darker color implies higher compression ratio. The experiments were conducted for all mocap data in the CMU database. It is clear that the enhanced algorithm is able to achieve a higher coding ratio while maintaining the same level of distortion. B. Perception-based Scheme The coding performance of the perception-based coding algorithm depends on the duration of partitioned segments. The final size of the coded file is roughly in proportion to the number of segments. As a result, a slow motion clip can give 50:1 30:1 25:1 20:1 15:1 10:1 a higher compression ratio while a fast motion clip allows a lower compression ratio. The average compression ratios of several motion categories are shown in Fig. 8. the jumping motion can be compressed aggressively with a ratio of 300:1 since the movement is very simple. The motion in the category of acrobatics involves a lot of balancing. However, the pose is similar so that the compression ratio is still high. The motion in martial arts contains repetitive sword plays and Tai-Chi, both of which allow a high compression ratio. The motion clips in the categories of Basketball and Boxing have a lower compression ratio since the speed of most dofs changes frequently. However, even in the worst cases, the compression ratio is close to 100:1, which is twice of the precision-based scheme. The averaged compression ratio of the perception-based coding algorithm applied to the CMU database is 150:1. To further justify the perception-based coding result, we conduct a subjective test on perceptual quality evaluation. Ten motion patterns from different categories were selected for this test. For each motion pattern, there were two associated questions. In the first question, each user was asked to compare the similarity of two synthesized motion patterns using the original mocap data and the compressed mocap dat side by side. The user can give a score from 1 (nothing alike) to 10 (indistinguishable). In the second question, the user is also asked to give a score on motion naturalness from 1 (extremely unnatural) to 10 (very natural). The reported statistics in Table I were collected from 20 users. As shown in Table I, every coded motion is still natural (score 10 out of 10) regardless whether it is similar to the original motion or not. For similarity comparison, the score is also very high (above 9.4 out of 10). The motion with the lowest similarity score is the follow path motion. This motion involves a lot of turning, which makes its temporal advances or delay more visible than those in other motions patterns. VI. CONCLUSIONS AND FUTURE WORK In this research, we explored the characteristics of mocap data and proposed two compression schemes that allow a flexible rate-quality trade-off. The first one is a precisionbased method. It aims to offer a compressed result which is loyal to the ground truth. The second one is a perception-based method. It aims to preserve the perceptual experience while providing a very high compression ratio. 857

6 Compression Ra-o running walking jumping basketball dance gymnas:cs acroba:cs Mo-on Category mar:al arts paddle sports soccer boxing Fig. 8. The averaged compression ratios of different motion categories in the CMU database. Both proposed schemes are automatic and generic. They both operate in the domain of dofs. The dof-based coding scheme has three major advantages. First, it is a more compact representation than the marker position domain. The use of the dof format cuts down the size of the marker position format by two thirds. Second, it is easier to compare two poses using the dof representation. The precision-based scheme achieved compression by reducing the time-domain redundancy with temporal sampling, and the space-domain redundancy by applying vector quantization to sampled frames (or I-frames). The dimension of a human pose is 60, which is still high. Consequently, we decomposed a full poses to 5 sub-poses, and encoded them separately. The remaining frames (or B-frames) were predicted by interpolating I-frames, and the prediction error was coded. On average, we can achieve 50:1 compression. The perception-based scheme achieved compression by segmenting each dof at extremes and quantizing each block. Since minor temporal offset or spatial stretch is not obvious to human eyes, we leverage this property to achieve a higher compression ratio. The encoder consists of three modules: 1) pre-processing, 2) segmentation and normalization, and 3) coding. In the second module, the encoder partitions a dof into segments of positive and negative velocities, normalizes each segment and applies vector/scalar quantization and dyadic encoding. The performance of the proposed algorithm depends on the motion speed and the existence of pose repetition in the motion. Dynamic and fast-changing motion such as playing basketball can reach a compression of ratio of about 100:1, while cyclic motion such as walking may reach a compression ratio of 250:1. In the future, we will look into the application of the proposed scheme to multiple character mocap data. In the current design, each character in the mocap data will be encode separately. For precision-based compression, leveraging the correlation between the movement of the characters may help for further compression. For perceptual-based compression, the synchronization of timing might be visually-significant, thus it has to be preserved. ACKNOWLEDGMENT The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA REFERENCES [1] O. Arikan, Compression of motion capture databases, ACM Transactions on Graphics, 25(3): , July [2] S. Chattopadhyay, S. M. Bhandarkar, and K. Li, Human motion capture data compression by model-based indexing: a power aware approach, IEEE Transactions on Visualization and Computer Graphics, 13:5-14, [3] P. Beaudoin, P. Poulin, and M. van de Panne, Adapting wavelet compression to human motion capture clips, Graphics Interface, pp , May [4] M. Alexa and W. Muller, Representing animations by principal components, Computer Graphics Forum, 19(3): , August [5] M. Sattler, R. Sarlette, and R. Klein, Simple and efficient compression of animation sequences, ACM SIGGRAPH/ Eurographics Symposium on Computer Animation, pp , July [6] L. Kovar, J. Schreiner, and M. Gleicher, Footskate clean-up for motion capture editing, ACM SIGGRAPH Symposium on Computer Animation, pp , July [7] T. Y. Yeh, G. Reinman, S. J. Patel, and P. Faloutsos, Fool me twice: Exploring and exploiting error tolerance in physics-based animation, ACM Transactions on Graphics, 29(1):1-11,