Information and Communication Engineering July 9th Survey of Techniques for Head Pose Estimation from Low Resolution Images

Transcription

1 Information and Communication Engineering July 9th 2010 Survey of Techniques for Head Pose Estimation from Low Resolution Images Sato Laboratory D1, , Teera SIRITEERAKUL Abstract Head pose estimation has become popular research topic during recent years. While majority of them focus on high-resolution images, some of the techniques are applicable to low-resolution images as well. Furthermore, with increase of surveillant systems which produce head images in very low resolution, there are several methods proposed specifically for the low resolution images as well. This paper discusses some of the methods applicable to low-resolution images as well as methods which are focusing on this particular area. The methods will be categorized into four general approaches couple with their advantage, limitation, and their potential to function in practical applications. Furthermore, this paper discusses some potential research topics as well as some of our research progress in head pose estimation from low resolution images. 1 Introduction Figure 1: In work by Benfold and Reid [2], surveillance system can be guided to focus on areas of attention. Head orientation provides crucial information for human behavior analysis which can be useful for many applications. There are several reported systems utilized head pose estimation in many areas such as 1. Surveillance: Benfold and Reid [2] proposed a system that can guide a surveillance system to focus on areas of attention. ( see figure 1. ) 2. Human Interaction: Zhang et al. [19] proposed a system for analyzing human interaction in a seminar room. 3. Advertisement: Smith et al. [12] proposed a system for tracking the visual focus of attention for a varying number of wandering people. The system can be used to count how many people look at a particular outdoor advertisement. ( see figure 2. ) Although there are several proposed applications utilizing technique for head pose estimation from Figure 2: In work by Smith et al. [12], the system can be used to count how many people have looked at a particular outdoor advertisement. 1

2 Figure 3: Examples of head pose images in low resolution. Figure 4: Example of head pose classes. low resolution images, all of them are not yet reliable enough for practical usage. Work by Benfold and Reid as well as work by Zhang et al. require higher head pose classification accuracy. Work by Smith et al. was mostly used to detect a single head pose angle which might as well use some face detector techniques. There are many challenges in head pose estimation. Firstly, there are a number of appearance variation of individuals such as skin color, hair color, or location of facial features ( e.g. location of eyes ). Secondly, there can be occlusion such as eyeglasses, hat, or other objects within the scene. Thirdly, scenes can be very different subject to day time, night time, indoor, outdoor, and etc. Lastly, image noise has much more impact since there are only few information available in low resolution images. See examples of low resolution images in figure 3. In the context of computer vision, head pose estimation is most commonly interpreted as the ability to infer the orientation of a person s head relative to the view of a camera [5]. At the finest level, head poses can be measured continuously up to three degrees of freedom (DOF). These DOF s are usually referred to as pitch, roll, and yaw as illustrate by figure 4. However, it usually not enough information for such refined estimation in low resolution images. Therefore, head pose are usually discretized into multiple pose classes ( e.g. eight classes of yaw direction with 45 of separation - see figure 5 for head pose classes example ). Figure 5: Example of head pose classes. 2

3 This paper presents a survey of the head pose estimation methods for low resolution images. The survey begins by organizing methods by their common approaches. Then, advantages and disadvantages as well as their potential for practical application are discussed. Thereafter, this paper discusses some research opportunities in each approaches as well as in the area in general. Finally, our research progress will be discussed along with the results. 2 Methods for Head Pose Estimation from Low Resolution Images Figure 6: Appearance template approach takes an input image and finds the best match specific to a pose. This paper presents the estimation methods based on the fundamental approach that underlies their implementations. This is done similar to the survey by Murphy and Trivedi [5]. Table 1 shows four general approaches for head pose estimation from low resolution, example works for each approaches, as well as their brief explanation. Details of each approaches are described next. In each approach, functional requirements are discussed along with their advantages and disadvantages. 2.1 Appearance Templace Methods Appearance template matches head images into one of the discrete head poses using image-based comparison metrics as illustrated in figure 6. Work by Robertson and Reid [9] matches 10x10 pixel head images into one of the eight poses using descriptor based on hand-select skin histogram, as illustrate by figure 5. Hand skin color selection requirements were later dissolved by Benfold and Reid [1] using randomized fern proposed by Ozuysal et al. [7]. On the other hand, Orozco et al. [6] suspected that explicit skin/hair separation by Benfold and Reid may not be feasible in low resolution images. In stead, Orozco et al. proposed another image descriptor based on pseudo Kullback-Leibler divergence between the input image and the mean images of each head pose classes computed from the training images. This techniques has been reported to work well for low-resolution images. One limitation of this approach is the fact that two images of the same person with different poses may appear more similarly than two images of the different persons with the same pose. Therefore, in order for the appearance template approach to work, images must be represented in such a way that the head pose images from the same orientation class are grouped together. Furthermore, these techniques require Figure 7: Detector array approach uses multiple face detectors and/or multiple cameras to detect a specific head pose. good head localization which can be another research area by itself. 2.2 Detector Array Methods This technique stems from face tracking where there are multiple face detectors, each for different discrete poses, to detect face in images, as illustrate in figure 7. One example is proposed by Rowley et al. [10] where a router network can resolves roll angle after a patch is assumed to be a face image. Another example is proposed by Zhang et al. [19] where five FloatBoost classifiers were trains using multiple far-field multi-camera. This setting can detect up to five different frontal yaw poses. Since detector array derives from face detection, the technique can perform both face and pose detection which can be considered as a great advantage. However, Since a face detector is trained for 3

4 Table 1: Approaches of methods for head pose estimation from low resolution images. Approach Example Works Brief Description Tem- Appearance plate Robertson 06[9] Benfold 08[1] Orozco 09[6] Detector Array Rowley 98[10] Zhang 06[19] Regres- Nonlinear sion Rae 98[8] Stiefelhagen 02[14, 13] Tian 03[15] Seemann 04[11] Gourier 06[4] Voit 06[17] Trackig Wu 00[18] Tu 06[16] Compare input images to a set of templates specific to each discrete pose. Using multiple face detectors, one for each poses, to estimate head pose from images. Create a mapping from the image space to the head pose space by using nonlinear regression. Estimate change in head pose from series of images taken from consecutive video frames. specific head orientation, multiple detectors or multiple cameras are required for head pose estimation. Thus, this results in increase of computational cost or system requirements. Also, to train each detector, both positive and negative training are requires. Furthermore, this techniques are limited to frontal face images. 2.3 Nonlinear Regression Methods Head pose can be estimated by using nonlinear function to map from the image space to one or more pose angles, as illustrate in figure 8. Most widely used tool for nonlinear regression is neural networks. Three multilayer perceptron (MLP), one for each degree of freedoms (DOF), were used in Tian et al. [15] and Seemann et al. [11]. The nonlinear regression approach can be very fast and accurate in practice. Furthermore, it only needs cropped head images for training. However, in order for the nonlinear regression approach to work, two head pose images with the same orientation should be mapped to the same point, or at least close to each others in the space of head orientation. Unfortunately, in the case of low resolution images, two images of the same person with different poses may be mapped closer to each other than images of different persons with the same pose. Therefore, this approach requires a robust image representation that is only, or at least more, sensitive to changes in head orientations. Another limitation of this approach is the fact that the refined mapping may not work in low resolution images due Figure 8: Nonlinear regression approach estimates functional mapping from image space to head pose space. 4

5 Table 2: Mean angular errors and correct classification rates of each works. Publication Mean error Rate Dimension Yaw Pit Figure 9: Tracking methods tracks change in head pose from images taken from consecutive video frames. to limited information. Furthermore, the nonlinear regression approach requires accurate head localization, which is similar to the appearance template approach. 2.4 Tracking Methods Tracking method utilizes temporal continuity and smooth motion of head poses in consecutive video frames ( see figure 9 ). One way to use tracking method is employing a model-based approach. Work by Wu and Toyama [18] uses an ellipsoid with a set of points on the surface where its north-pole is at the nose tip. Then, each points were train to match local image features. Further improvement of model-tracking approach can be achieved by utilizes particle filtering. Method proposed by Tu et al. [16] works well in low resolution video with adaptive PCA subspaces. While this technique can have fairly high accuracy, it usually requires an initial step such as frontal view or manual initialization which would limit practical use. Furthermore, it is only applicable to continuous video. Also, model mapping may not be feasible in very low resolution images. 3 Head Pose Estimation Comparison 3.1 Comparison of Published Results Due to lack of standard dataset of head pose images in low resolution, this paper will rely on published results of each works. In these works, absolute angular error is a common information metrics for evaluating head pose estimation. Another common information metrics for the head pose estimation is the rate of correct template matching. These two information metrics are summarized in table 2. By correct matching rate, detector array method proposed by Zhang et al. [19] seems to yield better Y. Wu [18] Y. Tian [15] % 16x16 E. Seemann [11] % 24x32 J. Tu [16] x13 Z. Zhang [19] % - N. Robertson [9] x20 B. Benfold [1] x10 J. Orozco [6] % 20x20 result than nonlinear regression methods [11, 15], but their absolute errors suggested otherwise. The reason being Zhang et al. considers neighboring poses to be correct classification as well. It is rather hard to draw any conclusion about which technique performed better than the others since the data set used were diverse. One conclusion we can draw is both tracking [16] and appearance template [9] techniques can yield decent accuracy results. Although both works require further improvements which will be discussed in the next section. 3.2 Comparison of Real-World Applicability These following assumptions are used in works mentioned in this paper. This list is a subset of assumptions used in a survey by Murphy et al. [5]: A Continuous video assumption. Only small change of head pose allow in subsequence video frame. B Initialization assumption. Initial pose must be known before a system can begin the estimation. C Antidrift assumption. Head pose will only be calculated for a short time, during which there will be no significant abnormalities from the visual information. D Stereo vision assumption. Subject is visible by two or more cameras at a small enough distance to discriminate depth information across the face. E Single DOF assumption. The head is only allowed to rotate around one axis. 5

6 These assumptions limits applicability of any system, regardless of accuracy reported. List of papers and their limitations can be found table 3. Although work by Wu and Toyama requires no assumption, its accuracy is too low to be used in any practical application. 4 Research Opportunity This section discusses research opportunities specific to each approach as well as in the head pose estimation from low resolution images in general. These research opportunities are mostly based on limitation of each specific approaches. 4.1 Appearance Template Methods There are several rooms for improvement regarding appearance template approach. First, in order to match an input image a correct template, there must be a robust image representation. An improvement from Robertson and Reid [9] to Benfold and Reid [1] to Orozco et al. [6] have included the improvement in image descriptors. The second point for improvement which is relate to the image descriptor is the distance metrics. In the end, if there is no image descriptor with robustness to the location of head in the input image, we need to develop a system that can crop head images automatically and precisely. 4.2 Detector Array Methods There is not much room for technical research in detector array approach. This is due to the fact that this approach make use of face detector which has been well develop for number of years. Although, one can try to make the detector array faster in general. On the other hand, one can try to make use of frontal head pose estimation result from this approach for some practical application as it was done by Smith et al. [12]. 4.3 Nonlinear Regression Methods Similar to appearance template methods, the nonlinear regression approach requires robust image feature extraction as well as a robust functional mapping. Although it should be noted that both works reported in this paper [11, 15] require multiple cameras to achieve the presented accuracies. 4.4 Tracking Methods With reported performances of tracking methods, one expected research opportunity would be to try to hybrid these methods to remove initialization and antidrift assumptions. These assumptions can be remove by detector array techniques but it would still be limited to frontal poses initialization. 4.5 General Opportunity Generally, all the reported work only consider head pose images as an only mean of gathering information. In case of low resolution images, this can be troublesome. To eliminate such limitation, one can consider information from other source, especially cues from environment. This can be head pose of other persons in the scene, objects in line of sight, or even saliency map of the scene. Another great opportunity for research is to developing some practical application that make use available head pose estimation techniques. Although there are limitations in existed works, it may be applicable to some less demanding applications. 5 Discussion This section provides summaries of previous works mentioned in the previous sections as well as some concluding remarks of our progress. First, from the reported results, the tracking methods can provide the highest head pose estimation accuracy while requiring some initialization. This can be solved by couple techniques in this approach by techniques from another approach such as the detector array approach. On the other hand, the nonlinear regression might be good for practical use if reasonable amount of training data can be obtained. The appearance template approach, which has been popular in the recent years, can be improved with better image descriptor and distance metrics as well as head localization. The last statement is applicable to the nonlinear regression approach as well. Lastly, the detector array methods can be very practical in limited usage where only frontal poses are required. While much improvement is still in need for lowresolution head pose estimation, some of the system shows great potential. The system proposed by Tian et al. [15] and Seemann et al. [11] performed considerably well in small room environment while the system proposed by Robertson and Reid [9] and Benfold and Reid [2] did similarly in unconstrained field. As for our progress, we have developed and tested two descriptors for low resolution images to be used for head pose classification. The preliminary result is promising. we aim to resolve the head localiza- 6

7 Table 3: Comparison of real-world applicability. Year Publication Approach Assumption DOF Domain 2000 Y. Wu [18] Tracking poses 2003 Y. Tian [15] Nonlinear Regression D, E 1 Up to 12 poses 2004 E. Seemann [11] Nonlinear Regression D 2 Continuous 2006 J. Tu [16] Tracking A, B, C 3 Continuous 2006 Z. Zhang [19] Detector Array F 1 5 poses 2006 N. Robertson [9] Appearance Template A, E 1 8 poses 2008 B. Benfold [1] Appearance Template A, E 1 8 poses 2009 J. Orozco [6] Appearance Template E 1 8 poses tion limitation in the appearance template model. Furthermore, our final goal is to develop a system that can analyze some human behavior based on available head pose information. 6 Summary This paper provide general observation of the head pose estimation techniques from low resolution images as well as some specific works in this area. Generally speaking, there are many areas within this research field that can be improve. Although, some practical application can be developed utilizing limited usages of available techniques. References [1] B. Benfold and I. Reid. Colour invariant head pose classification in low resolution video. In Proceeding of the British Machine Vision Conference, [2] B. Benfold and I. Reid. Guiding visual surveillance by tracking human attention. In Proceeding of the British Machine Vision Conference, [3] C.-C. Chang and C.-J. Lin. LIB- SVM: a library for support vector machines, Software available at cjlin/libsvm. [4] N. Gourier, J. Maisonnasse, D. Hall, and J. L. Crowley. Head pose estimation on low resolution images. In Proceeding of the 1st International Evaluation Conference on Classification of Events, Activities and Relationships, pages , [5] E. Murphy-Chutorian and M. M. Trivedi. Head pose estimation in computer vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31: , [6] J. Orozco, S. Gong, and T. Xiang. Head pose classification in crowded scenes. In Proceeding of the British Machine Vision Conference, [7] M. Ozuysal, M. Calonder, V. Lepetit, and P. Fua. Fast keypoint recognition using random ferns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32: , [8] R. Rae and H. J. Ritter. Recognition of human head orientation based on articial neural networks. IEEE Transactions on Neural Networks, 9(2), [9] N. Robertson and I. Reid. Estimating gaze direction from low-resolution faces in video. In Proceeding of the 9th European Conference on Computer Vision, volume 3952/2006, pages , [10] H. A. Rowley, S. Baluja, and T. Kanade. Rotation invariant neural network-based face detection. In Proceeding of the Conference on Computer Vision and Pattern Recognition, pages 38 44, [11] E. Seemann, K. Nickel, and R. Stiefelhagen. Head pose estimation using stereo vision for human-robot interaction. In Proceeding of the International Conference on Automatic Face and Gesture Recognition, pages , [12] K. Smith, S. Ba, J. Odobez, and D. Gatica Perez. Tracking the visual focus of attention for a varying number of wandering people. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(7): , [13] R. Stiefelhagen. Estimating head pose with neural networks-results on the pointing04 icpr workshop evaluation data. In Proceedings of 7

8 the ICPR Workshop on Visual Observation of Deictic Gestures, [14] R. Stiefelhagen, J. Yang, and A. Waibel. Modeling focus of attention for meeting indexing based on multiple cues. IEEE Transaction on Neural Networks, 13(4), [15] Y.-L. Tian, L. Brown, J. Connell, S. Pankanti, A. Hampapur, A. Senior, and R. Bolle. Absolute head pose estimation from overhead wideangle cameras. In Proceeding of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures, pages 92 99, [16] J. Tu, T. Huang, and H. Tao. Accurate head pose tracking in low resolution video. In Proceeding of the 7th International Conference on Automatic Face and Gesture Recognition, pages , [17] M. Voit, K. Nickel, and R. Stiefelhagen. A bayesian approach for multi-view head pose estimation. In Proceedings of the IEEE International Conference Multi-sensor Fusion and Integration for Intelligent Systems, [18] Y. Wu and K. Toyama. Wide-range, personand illumination-insensitive head orientation estimation. In Proceeding of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition, pages , [19] Z. Zhang, Y. Hu, M. Liu, and T. Huang. Head pose estimation in seminar room using multi view face detectors. In Proceeding of the 1st International Evaluation Conference on Classification of Events, Activities and Relationships, pages ,