Motion Style Transfer in Correlated Motion Spaces

Transcription

1 Motion Style Transfer in Correlated Motion Spaces Alex Kilias 1 and Christos Mousas 2(B) 1 School of Engineering and Digital Arts, University of Kent, Canterbury CT2 7NT, UK alexk@kent.ac.uk 2 Department of Computer Science, Southern Illinois University, Carbondale, IL 62901, USA christos@cs.siu.edu Abstract. This paper presents a methodology for transferring different motion style behaviors to virtual characters. Instead of learning the differences between two motion styles and then synthesizing the new motion, the presented methodology assigns to the style transformation the motion s distribution transformation process. Specifically, in this paper, the joint angle values of motion are considered as a threedimensional stochastic variable and as a set of samples respectively. Thus, the correlation between three components can be computed by the covariance. The presented method imports covariance between three components of joint angle values, while calculating the mean along each of the three axes. Then, by decomposing the covariance matrix using the singular value decomposition (SVD) algorithm, it is possible to retrieve a rotation matrix. For fitting the motion style of an input to a reference motion style, the joint angle orientation of the input motion is scaled, rotated and transformed to the reference style motion, therefore enabling the motion transfer process. The results obtained from such a methodology indicate that quite reasonable motion sequences can be synthesized while keeping the required style content. Keywords: Character animation Motion style Style transfer 1 Introduction Animated virtual characters can be met in various applications of virtual reality, such as in video games, in films and so on. Those characters should be able to provide realistically each different motion that is performed, such as enabling a human like way for representing each action. Hence, even if the well-known key framing techniques are able to provide an intuitive way for animating a virtual character, since these are always dependent on the developer s technical and artistic skills, it still may not be possible to generate highly realistic character motion. Due to these shortcomings, motion capture technologies were developed in order to enhance the naturalness of the virtual character s motion. With c Springer International Publishing AG 2017 L.T. De Paolis et al. (Eds.): AVR 2017, Part I, LNCS 10324, pp , DOI: /

2 Motion Style Transfer in Correlated Motion Spaces 243 these technologies, it is possible to capture the required motion sequences, by simply capturing real humans performing the required motions. Then, by using motion retargeting techniques [1], those motion sequences can be transferred to any character. Moreover, these motion sequences can be interpolated [2], blended [3] and so on, therefore providing to the developer the ability to reuse the motion data, as well as input into the motion data the new spatial and temporal characteristics of a new animated sequence. Among others, the requirement of transferring the style behavior of a motion sequence to another has attracted the research community. Generally, the techniques that have been proposed during the past years are based on the ability to learn the style content of a motion by using various methodologies related to statistical analyses and syntheses of human motion data. However, less attention has been given to methodologies that are able to transfer the required style by simply transferring the distribution of character s joint angles to a reference distribution that represents the motion style. The main advantage of such a methodology is its ability to treat each character s joint separately, allowing a partial motion style to be mapped to the original motion. Yet, it is required that the reference style motion be similar in content to the input motion. For example, it is not possible to transfer a motion style from locomotion to a non-locomotion sequence, and vice versa. Hence, in conjunction with the presented statics-based motion style transfer methodology, a simple extension that provides the partial motion style transfer is introduced. Based on the aforementioned explanation, this paper presents a novel methodology for transferring a motion style of a motion sequence to any other motion sequences. The presented methodology is achieved by assigning the motion distribution transfer process to a linear transformation methodology. Based on this methodology, different examples are implemented and presented in this paper where either the whole body or the partial body motion of a character is enhanced with style content. The remainder of the paper is organized in the following sections. In Sect. 2 related work on motion synthesis techniques for transferring the motion styles is presented. The problem statement and the methodology s overview of the proposed methodology are presented in Sect. 3. The proposed methodology that is used for transferring the motion style of an input to a reference motion is presented in Sect. 4. The results obtained from the implementation of the proposed methodology are presented in Sect. 5. Finally, conclusions are drawn and potential future work is discussed in Sect Related Work Among others, data-driven techniques for animating virtual characters are the most popular. Those techniques are responsible for synthesizing new motion sequences by using existing motion data. The most popular approach is the motion graphs [4 7], which allows transitions between poses or footprints [8 10] where the locomotion of a character is synthesized by following the footprints placed in the 3D environment. However, the main drawback of those techniques

3 244 A. Kilias and C. Mousas is that it does not allow generalization of the motion data, such as allowing new styles to be synthesized. While the ability to edit or synthesize new motion sequences by keeping the style variations of existing sequences is required, methodologies that are related to transferring the motion styles of one motion to any other have been previously proposed. The parameterization process for the motion data can be quite powerful in cases that require the prediction of a new motion style from existing motion data. In general, dimensionality reduction techniques, such as principal component analysis (PCA) [11, 12] and Gaussian process models (GPMs) [13], as well as probabilistic models, such as the hidden Markov model [14,15], or other machine learning methods, such as the well-known radial basis function (RBF) [16,17], can be quite beneficial in cases that require a learning process to distinguish separate different motion styles. In the following paragraphs, methodologies that are responsible for synthesizing style variations from human motions are presented. Urtasun et al. [12] used PCA to train a large motion capture dataset with variations in locomotion style sequences. Using PCA coefficients, they synthesized new motions with different heights and speeds. Cao et al. [18] used independent component analysis (ICA) to automatically determine the emotional aspects of facial motions and edit the styles of motion data. In addition, Shapiro et al. [19] used ICA to decompose each single motion into components, providing the ability to select the style component manually. Torresani et al. [20] proposed a motion blending-based controller, where the blending weights were acquired from a large dataset of motion sequences of which the motions styles had been labelled by specialists. Liu et al. [21] constructed a physical model using an optimization approach to generate motions with learned physical parameters that contained the style aspects. Another solution proposed by Elgammal and Lee [22] assigned style properties to time-invariant parameters and used a decomposable generative model that explicitly decomposed the style in a walking motion video. On the other hand, statistics have been used extensively in character animation [23 25]. Generally statistical models are also able to provide quite reasonable results. Specifically, Hsu et al. [26] proposed a style transfer methodology that learns linear time-invariant models by comparing the input and output motions in order to perform style translation. Brand and Hertzmann [15] used Markov models to capture the style of training motion, which were archived in order to synthesize new motion sequences while the style variations of the motion were transferred. Moreover, Gaussian process latent variable models (GPLVM) have been used to synthesize stylistic variations of human motion data. Generally, GPLVM can enable the probabilistic mapping of non-linear structure in human motion data from the embedded space to the data space. Methodologies such as those proposed by Grochow et al. [27] can be used for motion editing while maintaining the original style by adapting a Gaussian process latent variable models with the shared latent spaces (SGPLVM) model. Methodologies such as those proposed by Wang et al. [28] can be used to separate different style variations.

4 Motion Style Transfer in Correlated Motion Spaces 245 Interpolation and motion blending methodologies can also provide the desirable results. Therefore, Kovar and Gleicher [29] proposed a denser sampling methodology for the parameter space and applied blending techniques in order to generate new motion sequences. Rose et al. [16] used an interpolation method based on RBF to generate motions based on verb and adverb parameters. They constructed a verb graph to create smooth transitions between different actions. By using signal processing related techniques it is possible to transfer a motion style to another motion sequence. Specifically, Unuma et al. [30] proposed a method that uses Fourier techniques to change the style of human gaits in a Fourier domain. Using this method, the motion characteristics based on the Fourier data could be extracted. Bruderlin and Williams [31] edited the stylistic motions by varying the frequency bands of the signal. Perlin [32] added rhythmic and stochastic noise functions to a character s skeletal joints in order to synthesize motion sequences with personality variations. In the proposed methodology we try to transfer a motion style by aligning the distribution of two corresponding motion styles for each of the character s joint angle orientations. Based on this linear transformation process, the presented methodology succeed in synthesizing a style motion in a well manner. 3 Overview In the following two subsections we present the problem statement of motion style transfer, and the methodology that is used in this paper for approaching this problem. 3.1 Problem Statement This motion style transfer process problem is the requirement for finding a continuous mapping such as the input motion m in to be represented as t(m in ), fulfilling the form m in t(m in ), where t(m in ) denotes the target distribution of the reference style motion m ref. Finally, having aligned the corresponding motion sequences, a decomposition process that is responsible for synthesizing the input motion and fulfilling the target style content is required. This transformation process in mathematic literature is known as the mass preserving transport problem [33 35]. Figure 1 shows a simple example of this problem. 3.2 Motion Representation In the presented methodology, each joints angle value of the character s motion is considered as a three-dimensional stochastic variable, and the motion data as a set of sample (postures), and therefore the correlation between three components can be measured by covariance. The presented approach changes motion style through a series of transformations of the mean and covariance related to

5 246 A. Kilias and C. Mousas Fig. 1. The distribution transfer problem requires finding a mapping m in t(m in) between the input motion m in and a reference motion style m ref. Fig. 2. The pipeline of the presented methodology. translation, scaling, and rotation. Hence, the resulted motion succeeds in keeping the required components of the reference style motion. A simple graphical explanation of the presented methodology is shown in Fig. 2. It should be noted that in the presented methodology the motion sequences are represented as M = {P (t),q(t),r 1 (t),..., r n (t)} where P (t) andq(t) represents the position and the orientation of the character s root in the t th frame, and r i (t) fori =1,..., n is the orientation of the i th joint of the character in the t th frame. It should also be noted that the character s root position is executed by this process. Based on this representation, the necessary components of the character s joint angles are computed. Then by using the aforementioned methodology the transfer of the required style content to an input motion is achieved.

6 Motion Style Transfer in Correlated Motion Spaces Methodology In this section we present the methodology that was used for transferring a reference motion style to an input motion. In the presented methodology this is achieved by developing a statistics-based method that provides the feasibility of the transformation process in the joint angles orientation space by just utilizing the mean and the covariance matrix of the input motion. In the remainder of this section we introduce the methodology that was used for achieving this transformation. 4.1 Motion Style Transfer Firstly, for both the input motion and the reference motion style, the mean angle orientation of joint angles along the three axes as well as the covariance matrix between the three components in the Euler space are computed. Thus for the joint angle values of the input motion M in we have M in =( X in, Ȳin, Z in )and for the reference motion style M ref we have M ref =( X ref, Ȳref, Z ref ), while we also have the covariance matrices represented as C in and C ref respectively. Now it is possible to decompose the covariance matrixes using the singular value decomposition (SVD) methodology as presented in Konstantinides and Yao [36]: C = UΛV T (1) where U and V are orthogonal matrices and are compressed by the eigenvectors of C. Moreover,Λ = diag(λ X,λ Y,λ Z ) where λ X,andλ Z are the eigenvectors of C. U is employed in the next step as a rotation matrix to manipulate the joint angles of the style motion. Finally, the following transformation is used: M = T ref R ref S ref S in R in T in M in (2) where M =(X, Y, Z, 1) T and M in =(X in,y in,z in, 1) T denote the homogeneous coordinates of a joint angle orientation in Euler space for the output and the input motion respectively. Moreover, T ref, T in, R ref, R in, S ref,ands in denote the matrices of translation, rotation and scaling derived from the reference style and the input motion respectively. The definition of each aforementioned component is given in Appendix. Based on this methodology, the key component of this transformation is its ability to transform an ellipsoid in order to fit another one. Thus, the two ellipsoids fit separately the joint angle orientation of the reference style and the input motion in the Euler angle space. The fitting of the ellipsoid functions as an extension to the method of fitting an ellipse in the two-dimensional space as proposed by Lee [37] and involves computing the mean and the covariance matrix. While the mean denotes the center coordinates of an ellipsoid, the eigenvalues and eigenvectors of the covariance matrix indicate the length and orientation of the three aces of the ellipsoid. The transformations act on all of the character s joint angles in the input motion and move them in the appropriate position in the Euler space. Results of the presented motion style transfer process are shown in Fig. 3 as well as at the accompanying video.

7 248 A. Kilias and C. Mousas Fig. 3. Examples of motions s with the proposed methodology. The input motion (a), and the synthesized styles (b) (e). 5 Implementation and Results For the implementation of the proposed solution we asked an experienced designer to provide motion sequences with style content which are related to happy, angry, sad, tiered, proud, sneaky, cat walking, crab walking, lame walking and many more (see Fig. 4). On average, those motion sequences are not greater that 100 frames. Hence, based on the aforementioned methodology we Fig. 4. Example postures of the different reference motion styles that used in the presented methodology.

8 Motion Style Transfer in Correlated Motion Spaces 249 use various input motion-sequence themes provided by the CMU motion capture database [38] along with various synthesized example motions. Additional examples are presented in the accompanying video. It should be noted that the aforementioned methodology that maps the input motion to the reference motion style was generated off-line. However, the motion synthesis process is generated in real-time. Hence, by using an Intel i7 at 2.2 GHz with 8 GB memory, the presented methodology succeeds in providing the new motion on average of 45 frames per second. 6 Conclusions and Future Work In this paper, a novel methodology for transferring the motion style of a reference motion to an input one was presented. The proposed methodology succeeds in transferring a motion style by aligning the corresponding motion spaces and by aligning the distribution and the centre of each joint angle orientation values. However, the presented method can provide quite reasonable results only while the motion sequences that are used correspond to the same content (i.e., to transferring a locomotion style to another locomotion sequence). Hence, for enhancing the motion style transfer process we assumed that a partial motion style transfer could be quite beneficial. Thus, we implemented a simple methodology that allows the partial motion transfer. On the other hand, the motion style transfer is quite a complex process. We assume that methodologies that synthesize motion sequences to contain specific stylistic content by aligning the distributions are a quite promising research area. Therefore, in our future work we would like to extend the presented methodology by developing enhanced methodologies that will enable a generalized motion style transfer process. Appendix Here the definition of the components used in Eq. 2 is presented. Specifically, the matrices of T ref, T in, R ref, R in, S ref,ands in denote the translation, rotation and scaling derived from the reference style and the input motion respectively. They are solved as: 100 X ref T ref = 010Ȳref 001 Z ref (3) X in T in = 010 Ȳin 001 Z in (4) 000 1

9 250 A. Kilias and C. Mousas References λ X ref S ref = 0 λy ref λz ref 0 (5) / λx in / λy S in = in 0 0 (6) 0 0 1/ λz in R ref = U ref (7) R in = U 1 in (8) 1. Gleicher, M.: Retargetting motion to new characters. In: Annual Conference on Computer Graphics and Interactive Techniques, pp (1998) 2. Mukai, T., Kuriyama, S.: Geostatistical motion interpolation. ACM Trans. Graph. 24(3), (2005) 3. Kovar, L., Gleicher, M.: Flexible automatic motion blending with registration curves. In: ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp (2003) 4. Arikan, O., Forsyth, D.A.: Interactive motion generation from examples. ACM Trans. Graph. 21(3), (2002) 5. Kovar, L., Gleicher, M., Pighin, F.: Motion graphs. ACM Trans. Graph. 21(3), (2002) 6. Lee, J., Chai, J., Reitsma, P.S., Hodgins, J.K., Pollard, N.S.: Interactive control of avatars animated with human motion data. ACM Trans. Graph. 21(3), (2002) 7. Safonova, A., Hodgins, J.K.: Construction and optimal search of interpolated motion graphs. ACM Trans. Graph. 26(3), 106 (2007) 8. van Basten, B.J., Peeters, P.W.A.M., Egges, A.: The step space: example-based footprint-driven motion synthesis. Comput. Animat. Virtual Worlds 21(3 4), (2010) 9. Mousas, C., Newbury, P., Anagnostopoulos, C.: Footprint-driven locomotion composition. Int. J. Comput. Graph. Animat. 4(4), (2014) 10. Mousas, C., Newbury, P., Anagnostopoulos, C.: Measuring the steps: generating action transitions between locomotion behaviours. In: International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games, pp (2013) 11. Chien, Y.-R., Liu, J.-S.: Learning the stylistic similarity between human motions. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Remagnino, P., Nefian, A., Meenakshisundaram, G., Pascucci, V., Zara, J., Molineros, J., Theisel, H., Malzbender, T. (eds.) ISVC LNCS, vol. 4291, pp Springer, Heidelberg (2006). doi: / Urtasun, R., Glardon, R., Boulic, R., Thalmann, D., Fua, P.: Style-based motion synthesis. Comput. Graph. Forum 23(4), (2004)

10 Motion Style Transfer in Correlated Motion Spaces Ma, X., Le, B.H., Deng, Z.: Style learning and transferring for facial animation editing. In: ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp (2009) 14. Tilmanne, J., Moinet, A., Dutoit, T.: Stylistic gait synthesis based on hidden markov models. EURASIP J. Adv. Sig. Process. 1, 1 14 (2012) 15. Brand, M., Hertzmann, A.: Style machines. In: 27th Annual Conference on Computer Graphics and Interactive Techniques, pp (2000) 16. Rose, C., Bodenheimer, B., Cohen, M.F.: Verbs and adverbs: multidimensional motion interpolation. IEEE Comput. Graph. Appl. 18(5), (1998) 17. Song, J., Choi, B., Seol, Y., Noh, J.: Characteristic facial retargeting. Comput. Animat. Virtual Worlds 22(2 3), (2011) 18. Cao, Y., Faloutsos, P., Pighin, F.: Unsupervised learning for speech motion editing. In: ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp (2003) 19. Shapiro, A., Cao, Y., Faloutsos, P.: Style components. In: Graphics Interface, pp (2006) 20. Torresani, L., Hackney, R., Bregler, C.: Learning motion style synthesis from perceptual observations. In: Advances in Neural Information Processing Systems, pp (2007) 21. Liu, C., Hertzmann, A., Popović, Z.: Learning physics-based motion style with nonlinear inverse optimization. ACM Trans. Graph. 24(3), (2005) 22. Elgammal, A., Lee, C.: Separating style and content on a nonlinear manifold. In: IEEE Conference on Computer Vision and Pattern Recognition, pp (2004) 23. Mousas, C., Newbury, P., Anagnostopoulos, C.N.: Evaluating the covariance matrix constraints for data-driven statistical human motion reconstruction. In: Spring Conference on Computer Graphics, pp (2014) 24. Mousas, C., Newbury, P., Anagnostopoulos, C.-N.: Data-driven motion reconstruction using local regression models. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H. (eds.) AIAI IAICT, vol. 436, pp Springer, Heidelberg (2014). doi: / Mousas, C., Newbury, P., Anagnostopoulos, C.-N.: Efficient hand-over motion reconstruction. In: International Conference on Computer Graphics, Visualization and Computer Vision, pp (2014) 26. Hsu, E., Pulli, K., Popović, J.: Style translation for human motion. ACM Trans. Graph. 24(3), (2005) 27. Grochow, K., Martin, S.L., Hertzmann, A., Popović, Z.: Style-based inverse kinematics. ACM Trans. Graph. 23(3), (2004) 28. Wang, J.M., Fleet, D.J., Hertzmann, A.: Multifactor Gaussian process models for style-content separation. In: International Conference on Machine learning, pp (2007) 29. Kovar, L., Gleicher, M.: Automated extraction and parameterization of motions in large data sets. ACM Trans. Graph. 23(3), (2004) 30. Unuma, M., Anjyo, K., Takeuchi, R.: Fourier principles for emotion-based human figure animation. In: Annual Conference on Computer Graphics and Interactive Techniques, pp (1995) 31. Bruderlin, A., Williams, L.: Motion signal processing. In: Annual Conference on Computer Graphics and Interactive Techniques, pp (1995) 32. Perlin, K.: Real time responsive animation with personality. IEEE Trans. Vis. Comput. Graph. 1(1), 5 15 (1995)

11 252 A. Kilias and C. Mousas 33. Evans, L.C.: Partial differential equations and monge-kantorovich mass transfer. In: Current Developments in Mathematics, pp (1999) 34. Gangbo, W., McCann, R.J.: The geometry of optimal transportation. Acta Mathe. 177(2), (1996) 35. Villani, C.: Topics in Optimal Transportation, vol. 58. American Mathematical Society (2003) 36. Konstantinides, K., Yao, K.: Statistical analysis of effective singular values in matrix rank determinatio. IEEE Trans. Acoust. Speech Sig. Process. 36(5), (1988) 37. Lee, L.: Gait analysis for classification. Ph.d. thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology (2002) 38. Carnegie Mellon University, Motion capture database.