A Tool for Detection and Analysis of a Human Face for Aesthetical Quality Using Mobile Devices

Transcription

1 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV' A Tool for Detection and Analysis of a Human Face for Aesthetical Quality Using Mobile Devices Shuyi Zhao, Scott King, and Dulal Kar Department of Computer Sciences, Texas A&M University-Corpus Christi, Corpus Christi, Texas, USA. Abstract - A tool for mobile devices is presented to detect a human face in an image and analyze the detected face for its aesthetical quality. For the purpose of face detection, the tool uses two well-known classifiers: Haar feature-based cascade classifier and Local Binary Patterns (LBP) cascade classifier. Once a face is detected, the tool uses the Active Shape Model (ASM) algorithm to identify facial feature points and obtain raw data accordingly for aesthetical quality analysis. The analysis is carried out on the notion of Golden Ratio by finding ratios on the facial feature points obtained by using the ASM algorithm. To determine its effectiveness and limitations the tool is tested on animal faces, beautiful faces, ordinary faces, and unattractive faces. The statistics of results exposes distinctions between unattractive faces and beautiful/ordinary faces. However, no such distinction exists to separate beautiful faces from ordinary faces. Keywords: Face Detection, Cascade Classifier, Active Shape Model, Golden Ratio, Mobile Devices 1 Introduction Face detection and facial features tracking in images have many applications. Face detection involves finding a face or multiple faces in a picture or in a video clip. Due to improvement of hardware and software technology, the mobile devices now have the processing power and capacity to process images for many immediate or real time applications. In this work, we focus on face detection and analysis of a face for aesthetical quality using a mobile computing device. There are three major problems are addressed in this connection: face detection, facial feature point detection, and human face analysis. For human face analysis on aesthetical quality, Golden Ratio analysis of faces is implemented and tested for its effectiveness and suitability on mobile devices. The Android tool developed for the purpose provides the user with results upon analyzing of a human face. After detecting a face, the tool extracts facial features of the face. The raw data on facial features is used for Golden Ratio analysis to determine its esthetical quality. The tool allows users to input a photo and obtain the analysis results immediately and gives a measure of beauty on a detected face. This can be useful to determine ratings of attractiveness or unattractiveness of human faces. The tool can distinguish beautiful faces from unattractive faces clearly. However, we observe some limitations on using the Golden Ratio analysis approach when it comes to identify a beautiful face from an ordinary face. This paper proceeds as follows. Section 1.1 discusses earlier works and their usefulness to the development of the tool. Section 1.2 defines aesthetical parameters for measurement on the notion of Golden Ratio. We describe the methodology for the implementation of the tool in section 2.1 and 2.2. Section 2.3 presents the test results on effectiveness of the tool under different scenarios. Section 2.4 concludes the paper. 1.1 Related works There are many techniques found on face detection reported in literature. Among them are the techniques on analysis of facial motion, model-based techniques, featurebased techniques, and holistic analysis techniques [1, 3, 4]. To analyze facial motion, Mase proposes a technique that allocates an axis for each muscle to estimate its movement. Similarly, Cohn et al. define several facial feature points on a face to track them. However, model-based techniques map the images of the faces onto a predefined physical model [3]. Feature-based approaches are based on facial feature measurements such as nose length, chin shape, and so on. A holistic analysis technique analyzes the gray-scale level of a photo for face detection entirely. The Local Binary Patterns (LBP) Cascade classifier, a model-based technique is used for face detection [8]. The LBP Cascade uses pattern recognition that makes the checking pixel as center to capture a 3 by 3 table for analyzing the eight neighbors, as shown Figure 1. The eight neighbors are compared to a threshold and accordingly set to either 0 or 1. The Haar Cascade classifier is another model-based face detection method used in this work. For the purpose of training, it needs a large number of images of two types, positive images with a human face in them and negative images without any human face in them [2]. However, compared to Haar Cascade classifier, the LBP Cascade classifier is faster and performs reasonably well in less expensive hardware. For mobile devices, we find the LBP classifier very efficient and useful for our proposed face detection and analysis tool.

2 160 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV'18 Figure 1. Example of basic LBP operator. They are many algorithms reported in literature on facial features tracking. Heisele, Serre, and Poggio present a double layer classifier system in [6]. First, the image is scanned through a fixed size window to find a pyramid that is similar to a module composed of the eyes, nose, and mouth. Once the system detects a pyramid module, the module window is passed through a second layer classifier. In this level, background and hair pixels are removed from the sub-image. It uses a14 reference points on a face for component learning. Before component learning, the system is trained with data for identification. Farajzadeh, Faez, and Pan use a pre-processing step after detecting the sub-image [5]. They place the center of the eyes in a line to locate the face, instead of using reference points. In this work, the Active Shape Models (ASM) algorithm is chosen. This algorithm was originally proposed by Cootes, Edwards, and Taylor in 1995 [4]. The algorithm uses the target image to match a statistical model in order to find out the facial feature points. To implement this algorithm, machine learning must be involved, which requires a dataset with images with human faces in them, annotations, symmetry indices, and connectivity indices. The annotations means to annotate the facial features points. The symmetry indices means the indices for the mirrored points. The connectivity indices define the connectivity of the structure of points. In this work, XM2VTSDB multi-modal face database is used, which contains four different images of 295 people taken over a four months [10]. 1.2 Concepts of Aesthetics In Aesthetics, there are ways to judge the beauty of a human face based on geometrical measurement. Generally for facial beauty, more symmetric a person s face is, more beautiful the person is considered [7]. In this regard, the measurement of Golden Ratio is often considered in Aesthetics as a measure of symmetry [8]. As shown in Figure 2, there are vertical and horizontal dimensions for calculating Golden Ratios. Each largest rectangle should be times bigger than that of the smaller rectangle of the same color. Figure 2. Golden Ratio modules. The following list defines the rectangle of each color and size. Vertical golden ratios are shown in Figure 2 as: White - Hairline : Eyebrow top : Eye top Gold - Eyebrow top : Eyebrow bottom : Eye top : Eye bottom Blue - Eye pupil : Nose flair : Nose bottom Green - Eye pupil : Nose bottom : Mouth Green - Eye pupil : Nose bottom : Chin Green - Eye pupil : Mouth : Chin Similarly, horizontal golden ratios are shown in Figure 2 as: Gold - Face side : Eyebrows : Face side Gold - Face side : Eye inside : Face side Gold - Face side : Nose width : Face side White - Face side : Eye outside : Nose center Blue - Eye outside : Eye inside : Nose center Green - Mouth outside : Lip cupids bow : Mouth outside In this work, we investigate whether the preceding ratios can be used for analysis of faces for aesthetical purposes meaningfully or not. 2 Methodology The tool allows analysis of an image from a gallery or a picture taken with an inbuilt camera on a mobile device. First, the tool checks if a human face exists in the image. If a human face is detected, it shows the region of the face in the given image on the screen. Meanwhile, analysis of the detected face continues in the background. Two sets of results on a face can be obtained by using the tool, one set on horizontal Golden ratio analysis and the other set on vertical Golden Ratio analysis. The tool also provides some suggestive feedback along with the results. Figure 3 shows results on both horizontal and vertical Golden Ratio analyses.

3 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV' face: the symmetry of the face, the horizontal Golden Ratio results, the vertical Golden Ratio results, and the variance of the Golden Ratio results. Using the coordinates of the facial feature points of two bitmaps, the algorithm analyzes the image on the four aspects. The formula of calculating the symmetry grade is: (1),where is a full score of 100, point (x 67, y 67) is the center of the face, (x i,y i) is a point on the original bitmap image and (x i, y i ) is a corresponding point on the mirror bitmap image, as shown in Figure 5. Point (x 67, y 67) is considered the standard point, from which distances to points 0 to 66 are calculated to determine the symmetry grade. Figure 3. Views on Golden Ratio analyses. 2.1 Facial Feature Point Detection To detect a face in a given image, the tool uses the Haar Cascade classifier and the LBP Cascade classifier. To detect facial feature points in a face, the tool uses the ASM library developed by Wei Yao [9]. The ASM library comes with training images. The following pseudo code describes the steps involved in the process: 1. Convert the given image into bitmap format and obtain a mirror copy of the bitmap image. 2. Convert both the bitmap image and the mirror bitmap image from their RGBA format to grayscale. 3. Detect face in the grayscale image, if any using the LBP algorithm and the Haar Cascade algorithm. 4. If a face is detected, find and map the facial feature points (67 designated points) on the RGBA bitmap image and on its mirror bitmap image. For our tool implementation, we use OpenCV library. The bitmap of the processing image is converted into gray scale because the facial feature points can only be detected in gray scale using the algorithms provided in OpenCV. The LBP cascade algorithm is less accurate than the Haar cascade algorithm. However, the LBP cascade algorithm is faster and costs less hardware resource. Because of the limitations of mobile devices, using LBP cascade to detect face makes the tool more efficient. The feature points are marked by red dots in RGBA bitmap as shown in Figure 4(a). In the mirrored bitmap, the points are also marked as shown Figure 4(b). The mirror copy bitmap is for the Golden Ratio analysis. Both copies are useful for our analysis, especially to find symmetry in a given face. 2.2 Face Grading Algorithm The feature points on a detected face are used to perform Golden Ratio analysis. A face is judged on four aspects of the Figure 5. Labeled facial feature points The point in the mirror bitmap with the same label is the symmetric point of the point in the original bitmap under the same label. In the mirror bitmap, the face is mirrored, so point 0 in the mirror bitmap is point 14 in the original bitmap. The absolute value of the difference of the two distances is divided by the product of and where is the number of points and = 5 is the minimum unit difference used for our analysis. The correlation table of Horizontal Golden Ratio is shown in Table 1. Table 2. Correlation Table of Vertical Golden Ratio (slope = k') White - Hairline : Eyebrow top : Eye top (ignored) Gold - Eyebrow top : Eyebrow bottom : line 22/23/24 : line 26/25/24 : Eye top : Eye bottom line 28 : line 30 Blue - Eye pupil : Nose flair : Nose bottom line 31 : line 67 : line 41 Green - Eye pupil : Nose bottom : Mouth chin line 31 : line 41 : line 66 Green - Eye pupil : Mouth : Chin line 31 : line 66 : line 7 In Table 1, line x means a line through point with slope. Slope is the slope of the midline of the face. Then the ratio is calculated by the distance of two lines divided by the other distance of two lines. Theoretically the midline should be through points 7, 41, 51, 57, 61, 64, 66, and 67, which means any two points should determine the midline. However, it is not ideal. Therefore, the regression algorithm is used to find the midline determined by points 7, 41, 51, 57, 61, 64, 66, and

4 162 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV' The result is shown in Figure 6. The lines are drawn on the bitmap by the color defined in the correlation table (Table 1). face is detected in the photo of a cat, dog, rabbit, or gorilla. However, the tool mistakenly recognizes the face of a monkey in a photo as a human face. This is because a monkey has a facial structure similar to a human face (Figure 7). Figure 6. Images processed with and without regression. To calculate the Vertical Golden Ratio, the following correlation table as shown Table 2 is used. Slope k' is the slope of a horizontal line of the face. The horizontal line is determined by points 31 and 36, which are the centers of two pupils. Since the hairline can be arbitrary as it depends on hair style, we ignore in our implementation the ratio of hair line : eyebrow top : eye top. The eyebrow top is determined by comparing the distances of points 22, 23, and 24 to the pupil at point 31. The eyebrow bottom is determined by comparing the distances of points 26, 25, and 24 to the pupil at point 31. The lines are drawn on the bitmap by the color as shown in the correlation table (Table 2). The grades and the corresponding suggestions based on the resultant ratio are reported to the user after computation. Table 1. Correlation Table of Horizontal Golden Ratio (slope = k') Gold - Face side : Eyebrows : Face side line 1 : line 24 : line 13 line 1 : line 29 : line 34 : Gold - Face side : Eye inside : Face side line 13 Gold - Face side : Nose width : Face side line 1 : line 39 : line 13 White - Face side : Eye outside : Eye inside line 1 : line 27 : line 29 Blue - Eye outside : Eye inside : Nose center line 27 : line 29 : line 67 Green - Mouth outside : Lip cupids bow : line 48 : line 50 : line 52 : Mouth outside line Results and Discussions Figure 7. Animal faces Multiple Faces Five images, each containing more than one human face, are tested. As shown in Figure 8, the tool can detect only one human face out of many human faces in every image. Figure 8. Images with multiple faces Beautiful Human Faces As shown in Figure 9, five beautiful human faces (Caucasian, Indian, Asian, Latino, and African) are tested with the tool. The ratio results are shown in Table 3. The symmetry grades of them are 99, 100, 97, 98, and 99 respectively. Each column average shows the average of the absolute values for all ratios minus the Golden Ratio. We call this a residual average. To determine the accuracy and effectiveness of the face analysis tool, we test the tool on five different scenarios or cases: 1) animal faces, 2) multiple faces in a photo, 3) analysis of beautiful faces, 4) analysis of ordinary faces, and 5) analysis of unattractive faces Animal Faces As shown in Figure 7, five different animal photos (cat, dog, rabbit, gorilla, and monkey) are tested with the tool. No Figure 9. Samples of beautiful faces.

5 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV' Table 3. Ratio Results on Beautiful Faces Caucasian Indian Asian Latino African Average FS:EB:FS* FS:EI:FS FS:NW:FS FS:EO:NC EO:EI:NC MO:LCB:MO EP:NF:NB EBT:ET:EB EBT:EBB:ET EP:NB:MC EP:M:C Residual Average *Acronyms used: FS Face Side, EB Eyebrow, EI Eye Inside, NW Nose Width, EO Eye Outside, MO Mouth Outside, LCB Lip Cupids Bow, NC Nose Center, EBT Eyebrow Top, ET Eye Top, EB Eye Bottom, EBB Eyebrow Bottom, EP Eye Pupil, M Mouth, and C Chin) Ordinary Human Faces Five ordinary human faces (Caucasian, Indian, Asian, Latino, and African) as shown in Figure 10 are tested. The ratio results are shown in Table 4. The symmetry grades of them are 97, 92, 98, 97, and 97 respectively. Figure 10. Samples of ordinary human faces. Table 4. Ratio Results on Ordinary Faces Caucasian Indian Asian Latino African Average FS:EB:FS FS:EI:FS FS:NW:FS FS:EO:NC EO:EI:NC MO:LCB:MO EP:NF:NB EBT:ET:EB EBT:EBB:ET EP:NB:MC EP:M:C Residual Average Figure 11 are given in Table 5. The symmetry grades of them are 92, 91, 95, 94, and null respectively. The sample of the unattractive African face was not recognized by the tool. Figure 11. Samples of unattractive faces. Table 5. Ratio Results on Unattriactive Faces Caucasian Indian Asian Latino African Average FS:EB:FS N/A FS:EI:FS N/A FS:NW:FS N/A FS:EO:NC N/A EO:EI:NC N/A MO:LCB:MO N/A EP:NF:NB N/A EBT:ET:EB N/A EBT:EBB:ET N/A EP:NB:MC N/A EP:M:C N/A Residual Average N/A Conclusions In this work, we present a tool for Android devices that can be used to analyze and judge aesthetical quality of a human face in an image. The Haar feature-based cascade classifier and the Local Binary Patterns cascade classifier are used in the development of the tool to detect a human face. By adopting the idea of the Golden Ratio as a measure of beauty for human faces, we propose a tool that can be used to gauge aesthetic quality of a given human face. Particularly, the tool produces distinctive results on Golden Ratio analysis for unattractive human faces. However, no such clear distinctive results on Golden Ratio analysis are found by the tool to separate beautiful faces from ordinary faces. That is, the test results on the ordinary human group slightly differ from the test results of the beautiful human group. Further work is needed to demarcate these groups using a better approach Unattractive Human Faces The ratio results of the unattractive human faces (Caucasian, Indian, Asian, Latino, and African) as shown in

6 164 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV'18 4 References [1] M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski, Measuring facial expressions by computer image analysis, Psychophysiology, vol. 36, pp , [2] M. Castrillon, O. Deniz, D. Hernandez, and J. Lorenzo, A comparison of face and facial feature detectors based on the Viola-Jones general object detection framework, Machine Vision and Applications, pp , [3] T. Cohn, D. Cooper, C. Taylor, and J. Graham, Active shape models - their training and application, Computer Vision and Image Understanding, vol. 61, no. 1, pp , [4] T. Cootes, G. Edwards, and C. Taylor, Active appearance models, Proceedings of the European Conference, vol. 2, pp , [5] N. Farajzadeh, K. Faez, and G. Pan, Study on the performance of moments as invariant descriptors for practical face recognition systems, IET Computer Vision, vol. 4, no. 4, pp , December [6] K. Pulli, A. Baksheev, K. Kornyakov, and V. Eruhimov, Real-time computer vision with OpenCV, Communications of the ACM, vol. 55, pp , 2012). [7] D. Zaidel and M. Hessamian, Asymmetry and symmetry in the beauty of human faces, Symmetry, vol. 2, pp , [8] D. Huang, C. Shan, M. Ardabilian, Y. Wang, and L. Chen, Local binary patterns and its application to facial image analysis: a survey, IEEE Transactions on Systems, man, and cybernetics, vol. 41, no. 6, pp , [9] Y. Wei, Research on facial expression recognition and synthesis, Master Thesis Dep. Comput. Technol. Nanjing, [10] K. Messer, J. Matas, J. Kittler, and K. Jonsson, XM2VTSDB: The extended M2VTS database, Proceedings of the Second International Conference on Audio and Videobased Biometric Person Authentication (AVBPA 99), Washington D.C., 1999.