Construction of Frontal Face from Side-view Images using Face Mosaicing

Transcription

1 Construction of Frontal Face from Side-view Images using Face Mosaicing Hiranmoy Roy 1, Debotosh Bhattacherjee 2, Mita Nasipuri 2, Dipak Kumar Basu 2* and Mahantapas Kundu 2 1 Department of Information Technology, RCC Institute of Information Technology, Kolkata , India 2 Department of Computer Science and Engineering, Jadavpur University, Kolkata, , India {hiranmoy_roy17@yahoo.co.in, debotosh@indiatimes.com, mita_nasipuri@gmail.com, dipakkbasu@gmail.com, mkundu@cse.jdvu.ac.in} * AICTE Emeritus Fellow Abstract In this paper, a method for fully automatic construction of frontal 2D face image of an individual is described. This frontal image is constructed from his/her available side-view images using face mosaicing. Firstly an automatic eye-brow detection algorithm is used to detect the eye-brow along with its position i.e. left or right. By this result it is easier to detect whether the given image is left- or rightside view of a face. After that an automatic face cutting algorithm is used to extract the facial areas and then alignment mechanism using eye-brow leveling is used to mosaic them to construct the frontal face Lastly eyes are shifted using geometrical transformation to make it almost perfect frontal view of the face. Also an approximation of the frontal face from a single side-view image can be generated by cutting it according to the previous method and copying the same half into the second half. Experimental results show that frontal face images may be easily constructed and those images can be used by simple face recognition techniques, which performs with low computational effort. Index Terms Eye-brow detection, face alignment, eye shift, face mosaicing. I. INTRODUCTION Image processing, pattern recognition, and computer vision are currently very active area of research. These are the sub-disciplines of biometry. The main objective of biometry is to build special systems which can identify human from some observable characteristics such as their face, fingerprints, iris, etc. Among them faces have the strongest appeal for the human users. Different techniques have been used to track faces [1]. Recently some test showed, most systems utilize frontal facial images as their input patterns. So we need to get the frontal face image but sometime frontal face image may not be available, instead of that we have some side-view images or single side-view If we can generate the frontal face from these side-view images then we can easily process them. Again it can be helpful in case of footage captured from CCTV. We can also generate the frontal face of the terrorist from CCTV footage. On 26/11 the terrorist attack in Mumbai if we can generate the frontal face of the terrorists from the CCTV footage, then it will be helpful for the investigating agencies. But most of the biometry methods are susceptible to pose and lighting conditions. X. Lu, D. Colbry and A.K. Jain [2] proposed to override these 55 limitations by combining modalities like color, depth, 3D facial surface. Almost all 3D acquisition systems use professional devices like traveling camera or a 3D scanner. Usually, these systems need that the subject remains motionless during several seconds in order to obtain a 3D scan, and therefore these systems may not be suitable for some applications like categorization of human expression using movement estimation or real-time applications. Another factor is their cost which can make these systems unaffordable for normal and routine applications. In order to avoid these costly and time consuming 3D acquisition devices, some face recognition systems generate 3D information from stereo-vision. Fan Yang, Michel Paindavoine, Hervé Abdi, Dominique Arnoult[3] proposed another possible approach by deriving some 3D information from a set of face images, but without trying to reconstitute the complete 3D structure of the face. The goal of the paper done by them is to describe a system which can potentially process 3D faces in real-time. They describe the method for creating panoramic face mosaics using successive linear transformations. Several panoramic image construction algorithms have already been introduced. For example, A. Jain and A. Ross [4] have developed an image mosaicing procedure that creates a more complete fingerprint template using two impressions of the same finger. In the proposed algorithm, they at first aligned the two impressions and for that they used the corresponding minutiae points. Then, this alignment was used by a modified version of the Iterative Closest Point (ICP) algorithm in order to compute a transformation matrix. This matrix represents the spatial relationship between the two impressions. Modeling 3D human faces has been a challenging topic in computer graphics and computer vision in the past decades. Shuicheng Yan, Changhu Wang, Hongjiang Zhang and Weiying represent a fully automatic and efficient algorithm for realistic 3D face reconstruction by fushing multiple 2D face images [5]. Since the pioneering work of Parke [6], many different algorithms have been proposed for modeling the geometry of faces. The 2Dbased methods do not consider the specific structure of human faces. In the work of Lam et al. [7], face samples with out-of-plane rotation are warped into frontal faces based on a cylinder face model, but it requires heavy

2 manual labeling work. Shape from shading has been explored to extract 3D face geometry information and generate virtual samples by rotating the generated 3D face models. Among the most popular works on 3D face modeling and analysis, one is the morphable 3D face model proposed by S. Romdhani, V. Blanz, and T. Vetter [8] and another is the artificial 3D shape model proposed by Liu, Z., Zhang, Z., Jacobs, C. and Cohen, M. [9]. The former presented a 3D reconstruction algorithm to pick up the shape and texture parameters, and the latter developed a system which can construct textured 3D face model from video sequences. Y. X. Hu, D. L. Jiang, S. C. Yan, Lei Zhang, H.J. Zhang. [10] presented an automatic 2D-to-3D integrated face reconstruction method to recover the 3D face model based on a frontal face However, there are still limitations in these works: 1) In both Vetter and Zhang s works they have to manually initialize the face recognition systems; 2) In Zhang s work the two images should be close to the frontal view which are impractical for real applications; and 3) Hu and Yan s work assumed fixed pose parameters which limited its extension to side view images.. In this paper, a fully automatic and efficient framework is proposed for frontal face construction from two 2D face images in side views. It not only uses the advantages of the works mentioned above, but successfully generates the frontal 2D face image also with ease. Firstly, a simple but efficient eye-brow detection algorithm is developed. This eye-brow s information is very helpful in face detection, eye detection, face cutting and face mosaicing. Then sideview faces are cut accordingly and then eye ball is detected. Eye is shifted using geometrical transformation. Lastly faces are aligned depending on eye-brow levels and then mosaic to get the full frontal view of human face. II. PRESENT METHOD In this present method the main goal is to detect the eye-brow of human face and depending on that the face has been cut and then joined. After that eye shifting is performed based on the eye-brow detected. The present method is described through a block diagram, shown in figure 1. Firstly, color image is converted to grayscale image and then into bi-tonal From the bi-tonal image eye-brow is detected and then side-view images are cut accordingly. Finally, eyes are shifted and side-view images are joined to get the final frontal face The algorithm designed for this purpose is described next. A. Algorithm Step1. Color image is converted to grayscale image by setting the maximum value from R, G, and B values. Step2. Face edge is detected by checking the difference between two consecutive pixel values (whether sharp change in difference or not). Face edge is detected once from moving left to right horizontally and again from right to left. From the two different edge points face horizontal length is calculated. This step is shown in Fig. 2. Step3. Grayscale image is converted into bi-tonal Step4. Eye-brow of human face is nearly 1/3 rd of the face length. So from bi-tonal image we search for continuous black pixels having continuous white pixels above it with almost same length of eye-brow. We move from both the edge points towards the face. Detected eyebrow is shown in Fig. 3(a). Step5. The left and right profile images are cut by straight line from few columns after the eye-brow, shown in Fig. 3(b). Then this image is stored in a matrix. Input two color sideview face images of same human Convert the images into gray-level images Detect the edges of the face and the face length Figure 1. Block diagram of the method. Convert the images into binary images Assuming the ¼ length of face length as the eye-brow length detect the continuous black pixels in the binary image and thus detect the eye-brow Detect whether left side-view or right side-view according to the eye-brow presence Cut the images according to the eye-brow Mosaic the cut images to get the frontal face image Eyes are shifted into the mosaic face and finally background is removed 56

3 Figure 5. After the eye shifting the resultant Figure 2. Edge Detection (a. Color image, b. Gray scale image, c. Binary image, d. Edge detected image). Step 9. Then the left and right profile images are aligned depending on the presence of eye-brows and the two images are mosaic to get the frontal face image, shown in Fig. 6. Figure 3. a. Eye-brow detection, b. Cut Step 6. Whether the face image is a left profile image or right profile image can be easily determined. If the eye-brow is detected when moving from left to right for eye-brow detection then it is left profile image and similarly when moving right to left then right profile Step 7. Now the most important part is eye detection and eye shifting, which is shown in Fig. 4. Eye is present just below the eye-brow, so very easy to determine. For frontal face the eyes must be towards left for left profile image and towards right for right profile image respectively. Eye ball s mid point is detected from finding the maximum length black pixels present vertically in the bitonal Step 8. From the mid point of eye ball the eye is cut and shifted by reflecting it in other part, shown in Fig. 5. Figure 6. a. Mosaic image without eye shifting, b. Mosaic image after eye shifting. At last the background is removed to get the final frontal face, shown in Fig. 7. Figure 7. a. Mosaic image, b. Final image after removal of background. Then the R, G and B all images are cut and mosaic in similar way to get the final color frontal image, shown in Fig = Figure 4. Step by step eye shifting and frontal eye creation. Figure 8. Frontal color image generated after combining the R, G and B matrices. 57

4 III. EXPERIMENTAL RESULTS In order to test the validity of the algorithm designed here, several experiments are conducted to obtain frontal face images from two side-view images. One such experiment is shown through Fig. 9 to Fig. 12. Figure 11. a. Left profile image, b. Right profile image, c. Without eye Figure 9. a. Left profile image, b. Right profile image, c. Without eye Figure 10. a. Left profile image, b. Right profile image, c. Without eye 58 Figure 12. a. Left profile image, b. Right profile image, c. Without eye This method has some demerits also. It finds difficulty in mosaicing when larger variations in pose exist, shown in Fig. 13. It also finds difficulties in eye-brow detection in situations where hair and eyebrow overlap and if there

5 is any gap in the eyebrows, which is shown in Fig. 14(a) and Fig. 14(b) respectively. infrastructural facilities during progress of the work. Dr. D. K. Basu acknowledges the Emeritus fellowship with thanks to AICTE, New Delhi. REFERENCES Figure 13. Left profile image with different pose, b. Right profile image with different pose, c. Left profile image with too much rotation, d. Right profile image with too much rotation. Figure 14. a. Left profile image where hair falls on eye-brow, b. Right profile image, where a big cut mark on eye-brow. IV. CONCLUSION In this work a method for construction of frontal face images from side-view face images is thoroughly described. The reconstructed frontal view face images can be effectively used to yield better face recognition performance. Frontal face images for unknown persons can be easily generated when a single face image with any one side view is available. [1] M. Turk and A.Pentland, Eigenfaces for recognition, Journal Cognitive Neuroscience, Vol.3, pp.71-86, [2] X. Lu, D. Colbry and A.K. Jain, Three-Dimensional model based Face Recognition, Proc. International Conference on Pattern Recognition, Cambridge, UK, August, [3] Fast Image Mosaicing for Panoramic Face Recognition Fan Yang, Michel Paindavoine, Hervé Abdi, Dominique Arnoult Journal of Multimedia, Vol. 1, No. 2, May [4] A.K.Jain and A.Ross, Fingerprint Mosaicing, IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Orlando, Florida, May, [5] Realistic 3D Face Modeling by Fusing Multiple 2D Images Model. Changhu Wang, Shuicheng Yan, Hongjiang Zhang, Weiying Ma Proceedings of the 11th International Multimedia Modelling Conference (MMM 05) /05 $ IEEE. [6] F.I. Parke. Computer generated animation of faces, in ACM National Conference. ACM, November [7] Kin-Man Lam and Hong Yan, An Analytic-to-Holistic Approach for Face Recognition Based on a Single Frontal View, PAMI98, Vol2, No7, page [8] S.Romdhani, V.Blanz, and T.Vetter. Face identification by fitting a 3D Morphable model using linear shape and texture error functions. In Computer Vision ECCV 02, volume 4, pages 3-19, [9] Liu, Z., Zhang, Z., Jacobs, C. and Cohen, M. (2000). Rapid modeling of animated faces from video, Proc. 3rd International Conference on Visual Computing, Mexico City, pp Also in the special issue of The Journal of Visualization and Computer Animation, Vol.12, [10] Y. X. Hu, D. L. Jiang, S. C. Yan, Lei Zhang, H.J. Zhang. "Automatic 3D Reconstruction for Face Recognition", In FG2004 Proceedings, pages , In criminal investigation it can be a real good tool to generate the approximate face image of the criminal when a side view is available from CCTV footage or other sources. There are some difficulties in construction of frontal face when faces are rotated with larger degrees or have different poses. Acknowledgment Authors are thankful to the "Center for Microprocessor Application for Training Education and Research" of Computer Science & Engineering Department, Jadavpur University, for providing 59