3D Regularization of Animated Surfaces

Transcription

1 D Regularization of Animated Surfaces Simon Courtemanche Ensimag, Grenoble-INP Introduction Understanding the motion of living beings is a major research subject in computer animation. This kind of study is often based on real data, obtained by experiments. The process of digitization of these experiences provides data with unwanted noise. A regularization of these data is then needed. The present work handles the issue of regularization. It is based on the Laplacian Mesh Processing as explained in [S05]. Laplacian Mesh Processing is a method for static meshes. These methods are based on the Laplacian operator and the differential surface representation. It also describes several ways of application, such as shape approximation and compact representation, mesh editing, watermarking and morphing. We will tackle the implementation issue of the Laplacian processing framework. We will also mention some reflections to extend these static methods to animated surfaces. On a practical point of view, this work is included in the project ANR Kameleon. The aim of this project is to improve the techniques of animation of vertebrate motion. The global idea is to add the knowledge of internal structure of the skeleton to the current techniques of motion capture. This project is a collaboration of four research groups specialized into complementary domains : MNHN for anatomy, LPBEM for biomechanics, LNRS for clinical research, and INRIA for animation and video analysis. The project ANR Kameleon is lead by the team EVASION of INRIA and Laboratoire Jean Kuntzmann research lab.. Mesh A mesh is made of three sets : a set of vertices, a set of edges and a set of faces. Vertices are points in the space of study - e.g. dimension space. Vertices are linked to each other by edges, which are straight lines. Faces are planar polygons shaped by edges. In our case, faces are triangles. Thus, our meshes are called triangular-meshes. There are several file formats to store a mesh. We use the Object File Format. Files with this format have the extension.off, and are made up of three parts : - The first line, with the word OFF in upper-case letters, followed by the number of vertices, the number of faces, and the number of edges respectively. - The list of all vertices. Each line of this list contains the three coordinates of a vertex. - The list of all faces. Each line of this list begins by the number of vertices of a face, followed by the index of these vertices in the previous list, where indexes start at 0 ; eventually followed by a hash mark # and a comment. See figure for an example of an off file and its associated mesh. During this internship, several libraries and software have been used : - OpenGL is the basic Graphic Library used by most of the following libraries and softwares. It is installed by default on many operating systems. - GLEW is a library for OpenGL which provides Type of data In this section, we will review the notion of mesh, describe how it can be programmed and present the animated surfaces we will deal with. Figure : An example of an off file and its corresponding mesh.

2 mechanism for determining which OpenGL extensions are supported on the target platform. - Qt is a library developed by Trolltech providing a graphic user interface. Versions and 4 of Qt are not compatible, but extensions exist to use programs based on Qt with Qt4. Qt is no longer available for downloading. - libqglviewer is a library providing D viewer functionalities. It uses the Qt framework. - Cortona d Viewer is an interactive d viewer and API. - Matlab is a high level language developed by The MathWorks. Matlab provides a function called trimesh to display a triangular mesh. We can read off files in Matlab with the functions fopen, fgetl, fscan and fclose. We will return to the limitations of Matlab later. - Openmesh is a generic data structure for representing and manipulating polygonal meshes. This library has been founded by the German Ministry for Research and Education. It uses GLEW, and Qt4 libraries. It can be compiled under Microsoft Visual Studio 008, and gcc 4. - Graphite is an interactive software for mesh manipulation using numerical geometry. This research platform has been developed by the research team ALICE, which is one of the four teams of the Image Geometry and Computation group in INRIA Nancy Grand-Est / Loria.. Animated surface An animated surface is represented by a sequence of meshes. Each mesh corresponds to a frame of the animation. In our case, sequences are extracted from outline images. This implies that two different meshes can have a totaly different number of vertices. Thus there is no direct link between a mesh and its successor. The extracting process have been done with the Grimage platform at INRIA. This process consists on filming a moving subject with several cameras, from different points of view. Cameras are calibrated such that the position of one with respect to an other is precisely known. The capture is done with a homogeneous background. Thus the outline of the subject in each frame of each video can be computed. This outline is then projected through the space along the direction of the corresponding camera to shape a volume. The intersection of these volumes provides a mesh. We experimented this technique with a set of cameras and obtained the results shown figure. Figure : Experimental results of mesh construction from outline images with Grimage platform - INRIA. The blue image in the middle is the constructed mesh. The two images on both sides are outline images from two different cameras. The textures have been added. Noise comes from the fact that with this process, we can only obtain a mesh surounding the captured object, and not the object itself. A regularization is then needed on these obtained meshes. Laplacian operator One way to regularize these meshes is to use the Laplacian operator. This operator is described in detail in [S05]. We will present an overview of this method, review tools with which it can be implemented, and discuss the limitations of its application in our case.. Definitions The Laplacian operator is a symmetric matrix that transforms the Cartesian coordinates to relative coordinates. This matrix is sparse. This major property is use to program efficient computation methods. The relative coordinates, or δ-coordinates, of a vertex is the difference between the Cartesian coordinates of this vertex and the coordinates of the centre of mass of its immediate neighbours in the mesh, as illustrated in figure. Thus δ-coordinates represent the gradient of the divergence of the mesh, which describes the curvature of the represented object.

3 Figure : In yellow, the vector of the δ- coordinates of a vertex in D discrete space and D continuous space. These two images have been extracted from [S05]. The symmetric form of the Laplacian operator results from the difference between the degree matrix and the adjacency matrix of the mesh. The degree matrix is a diagonal matrix containing, at index (i,i), the number of neighbours of the vertex i. The adjacency matrix contains, at index (i,j), if vertex i and j are linked by an edge, 0 otherwise. For details, see the beginning of section. in [S05]. We notice that the Laplacian matrix is sparse. This property is very useful to reduce computation time while manipulating this matrix. Several libraries provide a specific sparse format for matrix : Blas [blas], Lapack [lapack], Eigen [eigen], Taucs [taucs], OpenCV [opencv] and Matlab [matlab]. All these packages provide utilities for linear algebra computing. But each of them have its specificity : - The BLAS (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. - LAPACK (Linear Algebra PACKage) provides routines for solving linear systems. This package is built from BLAS and is written in Fortran90. Dense and banded matrices are handled, but not general sparse matrices. - Eigen is a recent project, started in 00. It provides a SparseMatrix class. Several linear solvers are also provided, but currently, the SelfAdjointEigen- Solver class does not support the sparse matrix format. The other solvers have not been tested. - Taucs is a library built especially for sparse matrix. It has been developed between 00 and 00 and is no longer maintained. However this library is currently used by Graphite, described previously. - OpenCV (Open Source Computer Vision) is a library of programming functions mainly aimed at real time computer vision. This library provides a wide range of procedures for linear algebra, and a sparse matrix structure with the class CvSparse- Mat. Unfortunately, some of these procedures also do not support the sparse structure. - Matlab also provides a sparse structure. In our case, Matlab can t be used without using this structure : We handle meshes made up with around 0000 vertices. The size of the corresponding Laplacian matrix is thus If we assume that we need to store floats in this matrix for its manipulation, we obtain storage cost of 400Mo. When we tried to duplicate this matrix under Matlab to compute its transpose for instance, we obtained a memory overflow error. This can be explained by the fact that Matlab allocates continuous memory space to store matrices.. Basic smoothing The basic idea for smoothing a mesh is to consider the spectral properties of the Laplacian matrix, because this matrix directly determines the differential coordinates, and thus, the shape of the mesh. Since the Laplacian matrix is symmetric positive semi-definite, it has an orthonormal eigenbasis, with associated eigenvalues. The smallest eigenvalues correspond to the rough shape of the mesh. The highest eigenvalues correspond to the details of the mesh. Assuming this, we can decompose the Cartesian coordinates on the eigenbasis, and reduce or remove the coordinates on this base corresponding to the high frequency components. The figure 4 shows, in two dimension space, the projection of the coordinates of the vertices of a simple mesh on the eigenbasis of his Laplacian matrix. To compute eigenvectors of large meshes (around one hundred thousand vertices), we tested several libraries : - Eigen : The SelfAdjointEigenSolver is a class which provide methods for computing eigenvectors of a selfadjoint matrix. As it has been said previously, this method does not support the sparse matrix format. However, this method has been tested on the Laplacian matrix of a mesh of 94 vertices, viewed as a full matrix. After two hours and thirty minutes of computation, the computation has been stopped. - OpenCV : The cveigenvv is another procedure to compute eigenvectors for a symmetric matrix. The fourth parameter of this procedure allows us to specify the accuracy of the diagonalization. With an accuracy of 0, we obtained a result after 0 minutes of computation. The accuracy of the eigenvectors computed hasn t been checked. We can read on OpenCV literature, that an accuracy of 0 5 is generally enough. - Taucs : This library has not been directly tested, but via Graphite, for the reasons explained previ-

4 (x,y) (e, e ) (e, e ) (e, e ) Figure 4 : A simple mesh of three vertices of coordinates (x,y) and their decomposition on the eigenbasis (e,e,e ) of the corresponding Laplacian matrix L. 0 ( ) ( ) ( ) L - - A e = e = e = x = ( 0 0 ) = e + e + e y = ( 0 0 ) = e e + e viously. This software offers interactive methods for smoothing a mesh in an efficient time. Unfortunately, linear algebra methods are not directly available.. Mesh reconstruction from δ- coordinates and smoothing In [S05], another technique is presented to avoid the explicit computation of the eigenbasis of the Laplacian matrix. This method is applied on the reconstruction of the Cartesian coordinates from the differential coordinates. To understand the reconstruction process, we need to notice that the Laplacian matrix is not a full rank matrix, because the sum of its rows is the null vector. Thus this matrix is not invertible. On an intuitive point of view, this is explained by the fact that a set of δ-coordinates of a mesh can also be the set of δ-coordinates of the same mesh translated in any direction in the d space. To locate the mesh in this space, we then need to set the positions of at least one vertex. These positions are called constraints. Then the idea is to find a set of Cartesian coordinates that both best approximate these constraints and represent a smooth mesh. The smoothness is evaluated by computing the norm of the δ-coordinates of this mesh. Thus we obtain a full-rank system which we can solve in the leastsquares sense. This method is presented in [S05] as a shape representation technique. It assumes that the constraints are provided with the mesh to rebuild it. The strength of this procedure is that these constraints are determined according to the original shape of the mesh. But in our case, we do not have any original mesh, and thus these constraints are not provided. To choose these constraints, the first idea is to hand-select them. But this idea would be very costly in time and tedious when dealing with mesh sequences. A random selection of these vertices avoids this hand-task and has a very low time cost. But the result is uncertain. A third idea is to use a clustering algorithm, and take an approximation of the computed cluster centres as constraints. Thus we obtain constraints homogeneously distributed along the mesh. The drawback of this method is the computation time cost. It is important to note that all of these techniques are applicable to static meshes only. In the following section, we will discuss how to extend these methods to sequences of meshes. 4 Extension to mesh sequences The starting point to extend the Laplacian methods to mesh sequences is to consider the time as a fourth dimension. If we assume that a sequence of D meshes is a large mesh in four dimensions, the Laplacian operator can then be applied on this mesh as above. But this intuitive method raises 4

5 several issues. Building such a four dimension mesh is not obvious. Each frame of the sequence as a different number of vertices. This issue can be solved by remeshing each frame. Techniques exists for this task. For instance, Graphite allows to specify a new number of vertices for a mesh and compute the corresponding mesh. The drawback of this method is the losts of informations. Another question to build our very large mesh based on several frames is how to choose the edges between frames. A possible answer to this question is to link a given vertex with the closest vertex in the next frame of the animation. By adding the property that each vertex of a given frame can be linked with one and only one vertex in the previous frame, we obtain a mesh sequence corresponding to the animation of a unique mesh. Then we can combine d static techniques, such as the Laplacian operator, for smoothing each frame, with d dynamic techniques, such as the d Fourier transforms, for smoothing the motion of each vertex. To avoid the remeshing step, let us consider that we do not have the previous property of the uniqueness of the edge between a vertex and one of its two neighbouring frames. The linking part can then be done by choosing edges between a vertex and the closest vertices in its two neighbouring frames. The differential coordinates are thus computed as the difference between a vertex and the centre of mass of its neighbouring vertices in a set of three frames. Noticing that time is, in our case, on a rough discrete scale, we will consider that the time component of each δ-coordinate is zero. Thus, after computation, no vertex will be between two frames. Then we can build the Laplacian matrix as previously, and apply the methods seen in section on the three space coordinates of all vertices of our mesh sequence. It is important to note that these techniques has not been experimented in this study. Thus we currently do not know their efficiency. 5 Conclusion Differential coordinates are an essential representation to deal with the shape of a mesh. The associated Laplacian operator has strong properties to theoretically smooth a static mesh, such as its eigenbasis. However, no efficient libraries have been found in this study to compute this basis. The Laplacian operator could be extended to sequences of meshes by regarding these sequences as very large meshes in a four dimensions space. This study has allowed me to discover, in a short time, what research laboratory is. I especially notice during this internship that research is not only to imagine new solutions to a problem, but is most of the time to take into account of what other researchers have already done and how this can be used. Especially in computer science research, a lot of time can be spent on the installation of packages. Actually, I spent around fifty percent of my research time on this part. However, I am glad to have imagined my own ideas on the issue of the extension of the Laplacian operator to mesh sequencies, and to have presented them in section 4. In the short time allows to this study, only an overview of a possible implementation of the Laplacian operator and its extension to mesh sequences has been provided. The first step of a future work could be to go deeper into the study of the enumerated libraries to verify if any implementation of the eigen problem for sparse matrix exists. Then an extension of these libraries for this problem could be searched. The second step could be to continue the research of the extension to mesh sequences, beginning by reading existing computer graphics literature on this subject. Acknowledgements I would like to express my deepest gratitude to Estelle Duveau, Franck Hetroy, Lionel Reveret and Maxime Tournier for their help. 5

6 References [S05] Olga Sorkine, Laplacian Mesh Processing, Stat of The Art Report, EUROGRAPHICS 005 [blas] http :// [cortona] http :// [eigen] http ://eigen.tuxfamily.org/dox/ [glew] http ://glew.sourceforge.net/ [graphite] http ://alice.loria.fr/software/graphite/ [lapack] http :// [matlab] http :// product page.html [off] http ://shape.cs.princeton.edu/benchmark/documentation/off format.html [opencv] http ://opencv.willowgarage.com/wiki/ [opengl] http :// [openmesh] http ://openmesh.org/ [qt] http :// [taucs] http :// stoledo/taucs/