Render. Predicted Pose. Pose Estimator. Feature Position. Match Templates

Transcription

1 3D Model-Based Head Tracking Antonio Colmenarez Ricardo Lopez Thomas S. Huang University of Illinois at Urbana-Champaign 405 N. Mathews Ave., Urbana, IL ABSTRACT This paper introduces a new approach to feature-based head tracking and pose estimation. Head tracking and pose estimation nd their most important applications in motion analysis for model-based video coding. The proposed algorithm employs an underlying 3D head model, feature-based pose estimation, and texture mapping to produce accurate templates for the feature tracking. In this way, the set of templates used for the matching is constantly updated with the pose changes, allowing the algorithm to track the features over a large range of head motion without loss of precision and error accumulation. Given a rough estimate of the head scale, the initial feature identication is performed automatically and the tracking is successful over a large number of video frames. Computational complexity is also considered with the aim towards creating a real-time end-to-end model-based video coding system. Keywords: head tracking, feature tracking, model-based coding, feature extraction 1 INTRODUCTION Object tracking and motion analysis on video sequences are two aspects of computer vision that connect and relate to a number of applications ranging from video annotation to video coding to human computer interfaces. In most scenarios, objects are not rigid in that their motion with respect to the scene is mixed with deformations and changes in light conditions. A framework for automatic detection and tracking of moving objects in such complex conditions is very important in computer vision, and mandatory in model-based video coding in which the bandwidth is reduced by sending only motion information. Much of the work in automatic object tracking has sought to relax the constraints under which such systems will produce accurate results. In early work, detection and tracking of moving objects was carried out in limited scenarios and only supercial understanding of the scene was achieved. 1{4 Low level modeling such as snakes, active contours, and deformable templates introduced some improvements. 5{7 However, to obtain a robust and complete analysis of the objects and their motion, high level modeling is required. Note that in approaches such as analysis by synthesis, the scene understanding is limited by the model's capability to represent the scene. In this paper we present a scheme for model-based facial feature tracking and head pose estimation that is robust, accurate, and can be implemented in real-time. The system consists of three modules acting in a feedback loop, see Fig. 1: (i) 3D Object Modeling, (ii) 2D-3D Pose Estimation, and (iii) Synthesis-based Template

2 Matching. Because the algorithm employs analysis-by-synthesis using a 3D model, the system overcomes many common problems such as error accumulation over long sequences, changing light conditions, and occlusions. Video Stream 1st Frame Locate Initial Features Cyberscan Texture Compute Initial Pose Cyberscan Range Map Texture texture mapped model initial features Compute Pose Track Features Render Templates 3D Motion Estimation 2D Feature tracking Initialization Stage Tracking Loop Figure 1: Head tracking system overview 2 SYSTEM OVERVIEW As indicated in the block diagram in Fig. 1, the system consist of three modules: (i) 3D head modeling, (ii) feature detection, and (iii) pose estimation. A more detailed block diagram of the main loop is shown in Fig. 2. Given a predicted head pose, the 3D head model provides a synthetic view from which templates are made. Features are detected via template matching with such templates. Finally, the 2D feature positions and their correspondent 3D positions in the head model are used to estimate the new head pose. Kalman lters are used to predict the head pose and the 2D feature locations using the previous frames. Note that if this procedure is applied repeatedly on the same frame, it becomes an iterative approach to rene the current head pose estimation. The system is initialized assuming a front view in the rst frame, and the rough location of the facial features; the visual pattern recognition technique in 13 provides these initial eye corner locations. Cyberscan Data 3D Features Render Synthetic View, and Feature Positions Synthesized Images Kalman Predictor Predicted Pose 3D Motion Estimated Pose Pose Estimator 2D Feature Tracking Kalman Predictor Feature Position Template Maker Predicted Feature Position Match Templates Templates Video Sequence Figure 2: Block diagram of the main tracking loop. 3 HEAD POSE ESTIMATION One of the main steps in the proposed system is the update of the head pose from the corresponding 2D-3D facial features. A large set of 3D features are extracted manually from the initial head scan and a smaller subset

3 of 2D features are obtained from the feature tracking module. Using these correspondences, we can compute an alignment transform that maps the 3D features to their 2D locations, 14, This transform is then applied to the entire head model to create a new image from which the template database is obtained. One diculty is that, since only 3 feature points are used, the pose estimation results in 2 mathematically equivalent transforms, only one of which is correct for our purposes. We limit ourselves to 3 features (2 eye corners and nose tip) because of rigidity constraints and also to reduce complexity. However, a 4th feature point (mouth center) can be roughly approximated and used to resolve the ambiguity in the transforms. Since we know the 3D location of this 4th feature, the two resulting transforms can be applied to this point and compared to the estimated 2D position. The correct transform will result in a signicantly smaller mapping error. The problem setup is as follows. We are given three pairs of points in correspondence: (a m ; a i ), (b m ; b i ), (c m ; c i ). The model points are measured with respect to the 3D model coordinate system and the image points are measured with respect to the 2D sensor coordinates (image plane). We can describe the algorithm with the following steps: Step 1 Rotate and translate the model points so that the new a m is at the origin (0,0,0) and the new b m and c m are in the x-y plane. First we translate the model points so that a m is at the origin: a 0 m = 0; b 0 m = b m? a m ; c 0 m = c m? a m : (1) Next, we rotate about the X axis until b 0 m is in the XY plane: where, a 00 m = 0; b 00 m = R b 0 m ; c00 m = R c 0 m ; (2) tan() = b0 z ; sin() = b0 z b 0 y kb 0 mk ; cos() = b0 y kb 0 mk : Finally, we rotate the model points about b 00 m so that c 00 m is in the XY plane: where, a 000 m = 0; b 000 m = b 00 m; c 000 m = R c 00 m (3) tan() =? c00 z c b? m ; and c b? m = (c00 xb 00 y)? (c 00 yb 00 x) kb 00 m k Step 2 Translate the image points so that a i is at the origin. b i is at b i? a i and c i is at c i? a i Step 3 Solve for the 2x2 transformation matrix mapping the model points to the image points. Use Lb m = b i and Lc m = c i to solve for L. L = l11 l 12 ) L = [b l 21 l i jc i ][b m jc m ]?1 (4) 22 Step 4 Solve for the remaining elements of the 3x3 Ane matrix and for the scale to bring the points into alignment. 16 Call this matrix sr where s is the scale transformation. sr = 2 4 l 11 l 12 (c 2 l 21? c 1 l 22 )=s l 21 l 22 (c 1 l 12? c 2 l 11 )=s c 1 c 2 (l 11 l 22? l 21 l 12 )=s p where s = l l c2 1 and we solve for c 1 and c 2 using: r 1 p c 1 = 2 (w + w 2 + 4q 2 ); and c 2 =?q (5) c 1 3 5

4 where w = l l 2 22? (l l 2 21); and q = l 11 l 12 + l 21 l 22 Step 5 Combine the transform in Step 1 with the resulting transformation Q (from steps 2 and 4) to obtain the nal, rotation and translation needed to align the 3D model points with the image points. x 0 m = (R R T m )x m, and x i = O(sRx 0 m) + a i (6) where the O indicates taking the (x,y) coordinates of the resulting 3D vector. 4 SYNTHESIS BASED FEATURE TRACKING A typical feature-based motion estimation system relies on the precise detection of the features. Small errors in the feature locations can produce large errors in the estimated homogeneous transformation matrix representing the motion. Classical template-based feature tracking accumulates error over long sequences, and full search feature detection at every frame is too computationally expensive. With these issues in mind, our algorithm uses a combination of template-based detection and feature tracking that overcomes the error accumulation problem and can be eciently implemented. In addition, the system can also be used as an iterative approach to estimate and rene the global head pose transformation for a single frame. The system tracks the features at each frame using templates obtained from a synthetic image of the previous frame, and predicts the search areas using the previous feature positions. Shown in Fig. 3 are the actual images regions with the corresponding synthesized templates used in the correlation stage. One major advantage of our approach is that, because the facial feature locations on the synthetic images are known from the range data, no error is accumulated over the sequence during the tracking. Additionally, good spatial localization is achieved by using weighted-template matching where the match error is weighted with a Gaussian function centered at the position of the features. Problems with lighting conditions are dealt with by using the texture obtained from the rst frame of the sequence (or in general, any previous near-frontal view frame). (a) (b) Figure 3: Templates used in the matching stage. (a) Actual image regions, (b) synthesized templates using the recovered pose and the 3D model. As mentioned earlier, the proposed system is intended for real time analysis of video sequences. To achieve such performance we implement a weighted correlation using a set of look-up tables with pre-computed distance functions. If I 2 (k; l) is the template image, and I 1 (k; l) is the frame image, the weighted-error at position (n; m) is: X d(n; m) = w(k; l) f(i 1 (k? n; l? m)? I 2 (k; l)) (k;l)2wr where W r dene the size of the template, w(k; l) is the weight map, and f(e) is the distance function. In classical template matching, every position is equally weighted, and the square error is used; i.e. w(k; l) = 1 and f(e) = e 2.

5 By quantizing the weights to N levels and the gray-level image to M levels, the weighted-error can be implemented with a 2M N-entry-look-up table, and its computation reduces to a number xed point additions, and indexations: X d(n; m) = h[w(k; l)][i 1 (k? n; l? m)? I 2 (k; l)] where h is a Nx2M matrix. (k;l)2wr 5 EXPERIMENTAL RESULTS Several sequences were tested with the proposed system, three of which are presented here. The rst two were composed of greyscale, 200x200 images captured at 15 fps, while the third was 320x240 greyscale at 30 fps. The head models for each subject were created with the range scanner and the necessary 3D features were extracted. Results of the automatic feature initialization and tracking are shown in Fig. 4, Fig. 5, and Fig. 6. The top row of the rst 2 gures shows the results of the automatic feature tracking. The bottom row represents the synthesized images created by applying the head pose estimates to the texture-mapped 3D head model. The initial frontal image for each frame has been used to provide the texture map. While the main focus of this paper has been on accurate and automatic feature tracking, we can see in these images the possibility of creating extremely low bit rate video streams using an analysis-synthesis approach. The results of tracking both rigid and non-rigid points is shown in Fig. 6. Finally, the images in Fig. 7 show the results of wireframe tracking. The 3D wireframe at the computed pose is overlayed over the original image. Figure 4: Tracking Results: (top) The original sequences with tracked features. sequence using recovered head motion. (bottom) The synthesized

6 Figure 5: Tracking Results: (top) The original sequences with tracked features. sequence using recovered head motion. (bottom) The synthesized 6 CONCLUSIONS AND FUTURE WORK In this paper we have presented a robust and novel approach to real-time feature tracking using a 3D modelbased framework. A small set of facial features were tracked successfully over a large range of head motion. The combination of the 3D model, head pose estimation and texture mapping avoids the error accumulation problem and allows better localization of the features. Future work includes using generic head models tted to the person automatically, instead of subject specic head scans. Also, for use in video coding, local motion estimation as well as periodic texture updates need to be implemented. Finally, the speed of the matching algorithm can be signicantly improved using hierarchical methods. 7 ACKNOWLEDGMENTS This work was supported in part by Joint Services Electronics Program ONR N , the Army Research Laboratory under Cooperative Agreement No. DAAL , a grant from Rockwell International, and an AT&T Bell Laboratories Fellowship. 8 PRINCIPAL AUTHOR BIO Antonio Colmenarez received the B.S. and M.S. degrees from the Simon Bolivar University, Caracas, Venezuela in 1991 and 1993 respectively. He is currently with the Beckman Institute at the University of Illinois at Urbana-

7 Champaign pursuing a Ph.D. degree in computer vision and image processing. His current research interests include model-based video coding and pattern recognition. 9 REFERENCES [1] I. K. Sethi and R. Jain, Finding Trajectories of Feature Points in a Monocular Image Sequences, PAMI, pages 56{73,Jan [2] D. Huttenlocher, J. Noh, W. Rucklidge, Tracking Non-rigid Objects in Complex Scenes, ICCV [3] A Framework for Real-Time Window-Based Tracking Using O-The-Shelf Hardware,Technical Report for Version 0.95 Alpha, August 25, [4] Y. Yao and R. Chellappa, Dynamic Feature Point Tracking in an image sequence, IEEE Int. Conf. Pattern Recognition, Oct [5] F. Leymarie and M. D. Levine, Tracking Deformable Objects in the Plane Using an Active Contour Model, PAMI Jun [6] C. Kervrann and F. Heitz, Robust Tracking of Stochastic Deformable Models in Long Image Sequences, IEEE Int. Conf. Mach. Intel. Jun [7] F. G. Meyer and P. Bouthemy, Region-Based Tracking Using Ane Motion Models in Long Image Sequences, CVGIP:Image Understanding, Sep [8] K. Aizawa and T. S. Huang, Model-Based Image Coding: Advanced Video Coding Techniques for Very Low Bit-Rate Applications, Proceedings of the IEEE Vol. 83, pages 259{271, Feb [9] Y. Altunbasak, A. m. Tekalp, and G. Bozdagi, Two-Dimensional Object-Based Coding Using a Content-Based Mesh And Ane Motion Parameterization, Proc. IEEE Int. Conf. on Image Proccessing, Washington DC, [10] Y. Wang and O Lee, Active Mesh - A Feature Seeking and Tracking Image Sequence Representation Scheme, IEEE Tran. Image Processing, pages 610{624, Sep [11] I. A. Essa and A. Pentland, A Vision System for Observing and Extracting Facial Action Parameters, CVPR [12] D. Stork and M. Hennecke, Speechreading: An Overview of Image Processing, Feature Extraction, Sensory Integration and Pattern Recognition Techniques. Int. Conf. Automatic Face and Gesture Recognition [13] A. Colmenarez and T. S. Huang, Maximum Likelihood Face Detection, Int. Conf. Automatic Face and Gesture Recognition, pages 307{309, October [14] Ricardo Lopez and Thomas Huang. Head pose computation for very low bit-rate video coding. In Vaclav Hlavac and Radim Sara, editors, Computer Analysis of Images and Patterns, pages 440{447, Prague, Czech Republic, September Springer. [15] Ricardo Lopez and Thomas Huang. 3d head pose computation from 2d images: Templates versus features. In IEEE International Conference in Image Processing, pages 220{224, Washington DC, USA, October IEEE, IEEE Press. [16] Shimon Ullman and D. P. Huttenlocher. Recognizing solid objects by alignment with an image. International Journal of Computer Vision, 5(2):195{212, 1990.

8 Figure 6: Tracking Results for rigid and non-rigid points.

9 Figure 7: Wireframe Tracking Results