An Animation Synthesis System based on 2D Skeleton Structures of Images Lieu-Hen Chen Department of Computer Science and Information Engineering, National Chi Nan University Tel: 886-49-2910960 ext. 4861 lhchen@ncnu.edu.tw Chi-Jr Yu Department of Computer Science and Information Engineering, National Chi Nan University Tel: 886-49-2910960 ext. 4861 s0321531@ncnu.edu.tw Tzu-Chien Lo Department of Computer Science and Information Engineering, National Chi Nan University Tel: 886-49-2910960 ext. 4861 s0321029@ncnu.edu.tw Abstract Because the popularizations of digital image capture devices, there are enormous photograph images in our daily lives. In this paper, we propose a simple animation synthesis system to produce interesting visual effects by further utilizing these image materials in an intuitive and simple way. The basic concept can be divided into three steps. 1) First, skeletonize the object in the picture and set its kinematical joints. 2) Apply motion patterns and animate the objects according to the motion scripts. 3) Use image morphing techniques to synthesize image frames of animation. The integration of skeletons and the motion pattern controlled by forward kinematics scripts provides flexibility to synthesize interesting lifelike animation. Keywords Computer animation, Skeletonization, Image morphing, Kinematics. 1. Introduction Together with the miniaturization of digital image capture devices and the integration of digit camera into cellular phones, there are enormous photograph images around us. The research purpose of this paper is developing an animation synthesis system to further utilize the above-mentioned image materials effectively, and produce interesting and vivid animation effects in a natural and simple way. However, reconstructing the 3D models by using only 2D images is still a very difficult task. Also, building the virtual reality world by the traditional computer graphics methods is a highly time-consuming task. For the above problems, the image-based rendering technology provides an elegant approximate solution. Basically, the required image of arbitrary viewing position and direction is synthesized by morphing two or multiple pre-captured images at some determined camera positions. Applying the image morphing technology does not only reduce the expensive cost of world construction, but also overcome the discontinuity drawback of the pre-captured image sequence. However, if the morphing results fail to preserve the original shape of objects, it will cause a fake construction, especially when the target object is artifact. In this paper, we propose a simple shape-preserving image morphing method to produce frames of animation of images. First, the skeleton structures of the pre-captured images are extracted. The skeletonization is a process for reducing foreground regions in a digital image to a skeletal remnant that largely preserves the properties of the original region while discarding most of the original foreground pixels. We use the Discrete Voronoi Skeletons [1] proposed by Robert L. Ogniewigz to build object s skeleton. After the pre-captured images are analyzed, the skeletons data structure of the required image are applied some forward kinematics motion patterns like walking or running. Finally, the reasonable result images are synthesized by feature-based image morphing technique. 1.1 System Configuration Our system configuration is shown as figure 1. First, we load two images and then gain boundary B with depth information through the user s setting. Next, the system computes the skeletons S of an object. With the application of the motion pattern provided by the system, the skeletons S are altered according to the setting of the pattern, and we will get skeletons S. Then we utilize S to inverse compute the corresponding Boundary B. Finally, feature-based image morphing technique is employed to conduct texture mapping, and a consequence of the image is obtained. Figure 1. System Configuration.
1.2 System Manipulation According to the system configuration concept, we implement an animation synthesis system, following the steps below: 1) Input images and bounding the object Load two bitmap images and bound the shape with the object depth. 2) Shape skeletonization Compute skeletons of the shape. 3) Set the joints of the skeletons Setting the motion joints of skeletons according to the object image. 4) Merge skeletons Merge trivial skeletons and combine joints with the whole skeletons. 5) Select a motion pattern Select a motion pattern designed with forward kinematics scripts by the system. 6) Image morphing Use feature-based image morphing to do texture mapping, and thus gain a result image. Skeletons form Voronoi diagrams. Medial axis extraction from a distance map. Topological thinning. Since thinning method does not generate perfectly thinned output, and medial axis extraction from a distance map cannot guarantee the connectivity, we decide to implement Discrete Voronoi Skeletons proposed by Robert L. Ogniewicz. There are many advantages using Discrete Voronoi Skeletons technique to implement skeletonization. It has topological and geometrical properties, and multi-resolution by adjustable threshold. The most important property is that there exists the correlation between skeletons and boundary points. Figure 3. Voronoi diagram of 50 random points. 2. Algorithms Figure 2. System Manipulation. In this system, we implement the most important three techniques: skeletonization, image morphing and forward kinematics. We will introduce some algorithms in the above-mentioned techniques and explain why we choose one of the algorithms to conduct our implementation. 2.1 Skeletonization Skeletonization (in the plane) denotes a procedure that translates a two-dimensional shape into a one-dimensional graph-like object, similar to a stick figure. There are several different skeletonization algorithms: 2.1.1 Discrete Voronoi Skeletons 2.1.1.1 The Discrete Voronoi Medial Axis In Computational Geometry, the Voronoi diagram (VD) is a well-known tool. A VD of 50 points is shown in Figure 3. Given two points P i and P j in a set S of n points, H (P i, P j ) denotes the half plane containing the set of points closer to P i. The Voronoi polygon associated with P i is a convex polygon region with no more than n-1 sides. The Voronoi diagram of S are consists of all the Voronoi polygons of these points. Definition 2.1 (Voronoi Diagram) def V(p i ) H(p i, p j ). i j The VD of S is the collection of all V (pi): def Vor(S) V(pi). i j The Voronoi diagram of boundary points B (S) raster cracked from shape S was called the discrete Voronoi medial axis (DVMA): def DVMA(B(S)) Vor(B(S)) S
Definition 2.2 (Discrete Voronoi Medial Axis) The Discrete Voronoi Skeletons (DVSK for short) has three major operations: Computing the DVMA, Regularization, and the Skeleton Pyramid. 2.1.1.2 Computing the Discrete Voronoi Medial Axis There are several ways to implement Voronoi diagram algorithm. They can be roughly categorized into four classes: incremental algorithms, divide-and-conquer techniques, scan-line based methods, and hybrid techniques. Our implementation is a hybrid of the incremental approach and the divide-and-conquer technique. Hence, the advantages are the simplicity of the incremental construction of the VD and the clarity of the divide-and-conquer method. And a disadvantage is that if every scan-line exists exactly one point, algorithm will show a worst-case complexity of O (n 2 ). Since the points are extracted by scan-line from a raster image, there is quite low probability, with only one point in scan-line. 2.1.1.3 Regularization To pruning of the DVMA, there are four regularization methods, also called residual functions. They can assign each Voronoi edge a residual value, and through user-defined threshold, we can remove Voronoi edges that the residual value is below the threshold. We implement the simplest regularization function, the potential residual Rp, which will get the shortest length between its anchor point sites P A, P B along the boundary: Definition 2.3 (Potential Residual) Rp(e) w AB dist B (e), e belongs to DVMA(S). Figure 4. Establishing a hierarchy of skeleton branches. 2.1.1.4 The Skeleton Pyramid The threshold-based pruning method will not remove the numerous branches that seem less relevant to the main outline of the shape, especially the jagged boundary. When we increase the threshold value, the meaningful branch will be cut back but the spurious branch still appears. The skeleton pyramid clusters a hierarchical of skeleton edges, so it will get a multi-resolution skeleton and extract the most salient feature from the DVMA by a user-defined threshold. The implementation is mainly to set a residual table that starts with the largest skeleton edge based on residual value. After sorting the table by descent, we select skeleton branches according to the resolution of the threshold. Figure 4 is a Skeleton Pyramid concept. There is a example in Figure 5(a)-(c) using Discrete Voronoi Skeletons technique. Figure 5(a). The Discrete Voronoi Medial Axis of Heart Shape. Figure 5(b). The Discrete Voronoi Skeletons, potential residual, threshold = 10.
Figure 6(a). The concept of One Pairs of Line Transformation. Figure 5(c). The Skeleton Pyramid also called first order skeleton, threshold = 0.7. 2.1.1.5 Skeleton depth While users draw the boundary of a region in the image frame, our system provides users the ability to assign the average depth of this region on the z axis depth in 3D space. This creates the so-called 2.5D skeleton structure. With the 2.5D skeleton information, we can imitate and solve the problem of angle variation between the object and the viewpoint. The default depth distance between each layer is determined by the average type of build of the Orientals nowadays. For example, the average width of male shoulders is 42 cm, and that of female shoulders is 38.7 cm. Figure 6(b). Using OPL Transform to compute relations of boundary points and skeletons. 2.2 Image Morphing Image Morphing has long been recognized as a powerful tool for visual effects. Amazing effects can be seen in fluently switching one image to another in many movies and television programs nowadays. Its procedure combines image warping, color blending, and interpolating. An animator defines the corresponding features between two images, such as mesh nodes, line segments, curves or points. Then the animator uses corresponding features to calculate mapping functions that define the spatial relationship among all points in the two images. Because the main work is to warp, it is also called warp function. When the both images complete warp function and generate middle feature shapes, the colors in the original images will be blended and interpolated into the middle images. The work is done. Feature-based image morphing is easy to implement, and is a morphing technique that fits for our system property. Using the One Pair of Lines Transformation provided by this morphing technique can compute relations of skeletons and boundary points, and by using Multiple Pairs of Lines Transformation, we can texture mapping the animated shape of our objects. Using the boundary as the line segments of feature is seen in Figure 6. Figure 6(c). The concept of Multiple Pairs of Lines Transformation. Figure 6(d). Texture mapping using Multiple Pairs of Lines Transformation.
2.2.1 Transformation with One Pair of Lines There are a coordinate mapping (u, v) defines destination image pixel coordinate X to the source image pixel coordinate X : Figure 8. Hip rotation script. 2.2.2 Transformation with Multiple Pairs of Lines In Figure 6(c), there is a coordinate X i that is calculated for each pairs of lines. The displacements D i = X i - X is the difference between the pixel location in the source and destination images, then calculate weighted average of these displacements. Finally, we obtain all of pixel color in destination image. Algorithm is shown in Figure 7. 2.3.2 Knee rotation script Figure 9 is the knee rotation script. 2B represents foot placement and 3B stands for foot lift-off. However, there is always a constraint: knee rotation can never be positive. Figure 9. Knee rotation script. 2.3.3 Ankle rotation script Finally, in order to make the moving actions look natural and avoid the angle problem that makes feet get through the ground or walk without pressing on the ground, we make an ankle rotation script, as shown in Figure 10. Figure 7. The algorithm of Multiple Pairs of Lines Transformation. 2.3 Forward kinematics We implement a simple forward kinematics system that can be used for walk motion, and the following is a brief introduction of the legged locomotion: 2.3.1 Hip rotation script The upper leg in a walk cycle rotates backwards less than it does forwards. Thus we define an animation for the hip rotation as a curve between the limits of 35 o and 45 o. This is shown in Figure 8. 2.3.4 Motion Pattern Figure 10. Ankle rotation script. Our system offers users several different motion patterns in the format of motion scripts. Users can apply and combine these motion patterns to perform a series of movement. The result is also stored as a motion script. During this design stage, users are
prompted to assign the rotation joints to the relative skeleton to modulate each pattern. In the motion script, the duration of each motion cycle is set, and the joints are defined and organized hierarchically. By using script, we can directly set the angle variation of joints between two frames, and also determine and interpolate other frames. 3. Result and Conclusion Our system successfully synthesized photorealistic animation from 2D image(s). The skeleton structure of image extracted by the discrete Voronoi skeletons method preserves the geometrical properties of shapes and provides an approximative solution to animate the images without violating the kinematical correctness. The integration of skeletons and motion patterns controlled by kinematics scripts provides flexibility to synthesize interesting lifelike animations. The animation synthesis system is developed on the platform of Windows2000, Visual C++ 98 using MFC framework and STL library. Figure 13. Extracted skeletons of object shape and setting skeleton joints. Figure 14. Result image of animation frames. Figure 11. The system interface and two origin images. (a) Figure 15. There are two origin images with the 2.5D layered boundaries and skeletons. (b) Figure 12. Computing DVSK for all of layered boundaries.
[14] R.C.T. Lee, Introduction to the Design and Analysis of Algorithms, second edition, FLAGE publishing company. [15] Paul S. Heckbert, Pixar, Survey of Texture Mapping, IEEE Computer Graphics and Application, 1986, 56-67 Figure 16. If there are no appropriate motion patterns to apply your images, this system still can interpolate the result images between two origin images. This is a result image by interpolate value 0.5. 4. Reference [1] R.L. Ogniewicz, Discrete Voronoi Skeletons, Hartung-Gorre Verlag, Konstanz, Germany, (1993), Revised and extended version of Ph.D. thesis No. 9876, ETH-Zurich, Switzerland. [2] R. L. Ogniewicz, and O.Kubler, Hierarchic Voronoi Skeletons, Pattern Recognition, Vol. 28, no. 3, pp. 343-359, 1995. [3] H. Blum, A transformation for extracting new descriptors of shape, in W. Wathen-Dunn, editor, Models for the Perception of Speech and Visual Form, MIT Press, Cambridge MA, (1967) [4] Sitez S. M., Image-based Transformation of Viewpoint and Scene Appearance, A Dissertation Submitted in Partial Fulfillment of The Requirements For The Degree of Doctor of Philosophy, University of Wisconsin Madison, 1997. [5] Seitz S. M. and Dyer C. R., View Morphing, Proc. SIGGRAPH 1996, 21-30 [6] T. Beier and S. Neely. Feature-based Image Metamorphosis. SIGGRAPH 92 Proceedings, pp. 33-42, 1992 [7] G. Wolberg. Image Morphing Survey. The Visual Computer, 14, 8/9, 1998 [8] S. J. Fortune, A sweepline algorithm for Voronoi diagrams, Algorithmica, 2:153{174, 1987. [9] C. Roman, Construction of Voronoi diagrams using Fortune's method : A look on an Implementation, http://www.cg.tuwien.ac.at/studentwork/cescg/cescg99/ RCuk/, 1999 [10] Alan Watt, Mark Watt. Advanced Animation and Rendering Techniques, ACM Press, 1992 [11] Alan Watt, 3D Computer Graphics 3 rd Edition, Addison-Wesley, 2000 [12] Chen S. E. (1995). QuickTime VR-An Image based approach to virtual environment navigation. Proc. SIGGRAPH 1995, 29-38 [13] Williams L. and Chen S. E., View Interpolation for Image Synthesis, Proc. SIGGRAPH 1993, 279-88
Figure 17. There are result images of interpolate sequences, interpolate value 0.1, 0.3, 0.5, 0.7, and 0.9. Figure 18. The sequence of images applies the walking motion pattern provided by the system.