UMB-DB: A Database of Partially Occluded 3D Faces

Transcription

1 UMB-DB: A Database of Partially Occluded 3D Faces Alessandro Colombo colomboal@disco.unimib.it Claudio Cusano claudio.cusano@disco.unimib.it Raimondo Schettini schettini@disco.unimib.it Università degli Studi di Milano-Bicocca Viale Sarca 336, Milano, Italy Abstract InthispaperwepresenttheUMB-DB3Dfacedatabase. The database has been built to test algorithms and systems for 3D face analysis in uncontrolled and challenging scenarios, in particular in those cases where faces are occluded. The database is composed of 1473 pairs of depth and color images of 143 subjects. Each subject has been acquired with different facial expressions, and with the face partially occluded by various objects such as eyeglasses, hats, scarves and hands. The total number of occluded acquisitions is 578. The database, that is freely available for research purposes, could be used for various investigations, some of which are suggested in the paper. For the sake of comparison, we report the results of some of the 3D face detection and recognition algorithms in the state of the art. 1. Introduction In recent years, 3D face analysis have become topics of great interest in the research community [7, 8, 9, 2, 5]. The increasing diffusion of 3D imaging technologies is one of the main reason of attractiveness. The use of 3D imagery has been also encouraged by the difficulties reported in solving face-related tasks on 2D images [6]. In fact, 3D data provides invariance from scale, pose, and lighting conditions, which are sources of undesired intra-class variability(even though certain lighting conditions may deteriorate the accuracy of some acquisition devices). However, realworld applications are forced to deal with uncontrolled scenarios which present other problematic issues. In particular, subjects may have a non neutral facial expression and the face may be partially occluded by hair, hands, phones and other kind of accessories. The majority of existing approaches to 3D face analysis, do not deal with these scenarios. This fact is also motivated by the lack of a public database featuring a large number of acquisitions taken under unconstrained conditions. To cope with this problem, Colomboetal.[10,11]proposedamethodtosynthetizeartificially occluded 3D faces. This approach, while reasonable, cannot guarantee that the tested methods will actually work, with comparable performance, in real scenarios. InthispaperwepresenttheUMB-DB(UniversityofMilano Bicocca Database) 3D face database, which has been built to test 3D face analysis algorithms and systems, such as face detectors and recognizers, in scenarios where the acquisition conditions are not optimal. In particular, it is suitable to design and test methods dealing with partially occluded faces. The database is freely available for research purposes. 1 Thepaperisorganizedasfollows: insection2webriefly review the state of the art focusing on detection and recognition of partially occluded faces; Section 3 describes and compares publicly available 3D face databases; the UMB- DB Database and its content are presented in Section 4; Section 5 reports the results obtained by face detection and recognition algorithms in the state of the art, and Section 6 concludes the paper with our final considerations. 2. Occlusions in 2D and 3D face detection and recognition The greater part of the state of the art in face detection and recognition does not take into account occlusions; only a few approaches which consider 2D images can be found in the literature, and even less methods exist, as far as we know, for 3D images D approaches Several methods for the detection of faces in images (suchasthemethodproposedbyviolaandjones[34])show some degree of robustness with respect to occlusions, even IEEE International Conference on Computer Vision Workshops /11/$26.00 c 2011 IEEE 2113

2 without a specific training. On the contrary, recognition of partially occluded faces seems to require specially designed methods. For example, Park et al. have proposed a method for removing glasses from a frontal image of the human face [29]. They first detect the regions occluded by the glasses and then generate a natural looking facial image without glasses using Principal Component Analysis(PCA) reconstruction. A more general solution is needed, however, for the more challenging scenario where the occlusions are unforeseen and the characteristics of the occluding objects are unconstrained. Local approaches represent a possible solution for this problem. For instance, Martinez [23, 24] divided the face into parts which are independently compared. Kim et al. have proposed a part-based local representation method based on Locally Salient Independent Component Analysis (LS-ICA) [21]. Gökberk et al. [15] proposed a distance called Asymmetric Trimmed Distance Measure that is used to select non occluded regionsfor the global similarity computation between the probe and the gallery faces. Instead of searching for local non-occluded regions, Tarrés designed a method which eliminates those parts, that may correspond to occlusions, and that may hinder recognition accuracy[33]. A similar approach has been proposed by Zhang et al. [36] which divided the faces into rectangular regions and, on the basis of their gray level histograms, estimated the probability of occlusion. Park et al. [28] presented a method for face recognition tolerant to occlusions, based on an attributed relational graph. Complex face models have been used to describe the shape and the texture of 2D faces. Occlusions can thereforebe detectedbecausetheydonotfit theface model. For instance,moetal.[25]usedanappearancemodelstorepresent shape and texture variations, while De Smet et al.[13], and Hwang and Lee [20] investigated the use of morphable models. Lin and Tang[22] derived a Bayesian formulation unifying the occlusion detection and recovery stages. They also developed a quality assessment model to drive the process D approaches With the exception of the method presented by Colombo et al. [10] and, possibly, of the method proposed by Nair and Cavallaro [27] we are not aware of any method for the detection of 3D occluded faces. The problem of occlusions has been already addressed in thecontextof3dobjectretrieval[32],butfewmethodsexist, as far as we know, for the three dimensional recognition of partially occluded 3D faces. Alyüz et al. proposed a part based recognition method, where the parts are independently matched by ICP against a set of average regional models[1]. They experimented various fusion techniques to integrate the similarity between the face parts and the corresponding average regional models. They reported a significant improvement in recognition rates with respect to global ICP matching in the cases of occlusions(hand, hair, or eyeglasses). However, they relied on manually annotated feature points. Colombo et al. [10, 11] presented a strategy which approaches the occlusion problem by performing a restoration of the faces: the three dimensional occluded regions are detected, and the non-occluded regions are then used to recover the missing information. The occlusion detection method, inspired by the work of Skocaj and Leonardis[32], considers three dimensional occlusions as local deformations of the face that correspond to perturbations in a face space designed to represent non-occluded faces. Once detected, occlusions represent missing information which can be seen as holes in the faces. The restoration module recovers the whole face by exploiting the information provided by the non-occludedpart of the face, and using a basis that is appropriate for the face space in which the nonoccluded faces lie. 3. 3D Face Databases Overview A fair comparison of the 2D and 3D approaches above described, and of any other approach available is not possible without a public available database. Several databases of 2D face images are already available and some of them, designed for uncontrolled scenarios, contain occluded faces[16]. Publicly available 3D face databases are mainly aimed to analyze systems and algorithms for face recognition in controlled scenarios. In Table 1 we report a brief analysis of currently available databases. As it can be seen, facial expressions is the most considered source of variability. Only the Bosphorus database [31] provides a set of occluded faces. More precisely, it provides 381 occluded acquisitions of 105 subjects with three types of occluding objects (hand, eyeglasses, and hair). The locations of the occluding objects are usually the mouth, one eye(for hand), botheyes(foreyeglasses)and,inthecaseofhair,partofthe foreheadorhalfoftheface. In light of the previous considerations, the UMB Database is aimed to further improve these databases in terms of occlusions; in particular the UMB-DB:(i) includes a larger number of occluded faces;(ii) increases the number of occluding objects (iii); provides an increased variability in terms of location and extent of the occluded face regions; and(iv) keeps a larger number of subjects. 4. The UMB-DB Database TheUMB-DBiscomposedof14732Dcolorimagesand 3D depth images of 143 subjects. The subjects include 98 males and 45 females; most of them have an age ranging 2114

3 Database Ref. Subjects Acquisitions Expressions Poses Occlusions total/per subj. total/objects FRGCv.2 [30] / BU3DFE [35] / intensities 1 0 ND2006 [14] / York [18] /15 5 3? 0 CASIA [37] / FRAV3D [12] / BJUT-3D [19] / GavabDB [26] / DRMA [4] / Texas 3D [17] /1 89 unknown 1 0 Bosphorus [31] / /3 UMB-DB / /6+ Table 1. Summary of the major 3D face databases currently available. from19to50;apairoftwinsandafouryearsoldbabyhave been also acquired. Almost all the subjects are Caucasian with just a few exceptions. For each subject we captured a minimum of 9 acquisitions(see Figure 1): three with a neutral expression; three with non neutral facial expressions: smiling, angry, and bored; three with face occluded by different objects: scarf, hat and hands in random positions. Most of the subjects have been also captured with eyeglasses, holding phones, partially occluded by hair and other miscellaneous objects. The total number of occluded faces is 578. During the acquisitions subjects were free to cover different parts of their faces; therefore, occlusions do not hide the same parts of the face. This characteristics of the data make it possible a reliable test of the robustness of face analysis algorithms. Occlusions cover, on average, 42% of the face area, with a maximum of about 84%. In Figure 2 are depicted some examples of acquisitions with occlusions. For the sake of comparison, database features are also summarized in Table 1. For each acquisition, 7 feature points have been manually annotated: the corner of the eyes, the tip of the nose and the corners of the mouth (see Figure 3). In case of occluded acquisitions, only the non occluded feature points are given. Each acquisition is also associated with a set of labels describing it (occluded/non-occluded, occluding object if present, facial expression etc...). Two- and three-dimensional images have been simultaneously captured using the Minolta Vivid 900 laser scanner (in slow mode, with the 25mm lens). During the acquisitions, subjects have been asked to keep their eyes closed for safety reasons. Faces have been acquired in various indoor locations with uncontrolled lighting conditions. The 2D/3D images are released in their original form; i.e samples, without any further processing such as noise reduction or holes filling. The UMB Database can be adopted for various kind of investigations about face analysis in unconstrained scenarios,andsomeofthemhavenotbeenaddressedsofarinthe literature. For example: is 3D better than 2D when faces are partially occluded by unknown objects? Are, int this case, multimodal approaches better than single modality approaches? Are part-based approaches better than reconstruction based approaches? Occlusions introduce some degree of uncertainty in the localization of facial features: how this uncertainty influences face recognition performances? 5. Some experimental results In order to demonstrate the challenges posed by the UMB-DB, we report here some results obtained by applying algorithms for 3D face detection, normalization and recognition D Face Detection We report the results of a 3D face detection algorithm which is able to deal with partially occluded faces[10]. Briefly, the algorithm uses curvature segmentation in order to detect possible eyes and noses. A set of candidate faces is generated grouping pairs or triplets of facial features. At this point, each candidate face is registered to the mean face through a variant of the ICP algorithm. In this phase, the parts of the candidate face which is too far in depth from the mean face are considered occluded. Once registered, 2115

4 (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) Figure 1. Anexample of the minimal set of images taken for each subject: neutral (a, h); smiling(b, i); angry (c,j); bored (d, k); occluded byscarf (e,l);occluded byhand (f,m); occluded by hat (g,n). Total False False Detected Faces Positives Negatives Faces (96.5%) Table 2. Face detection results obtained on all the acquisitions from the UMB-DB. Acquisition type Number of faces Detected Faces Neutrals (99.1%) Non-neutral (97.5%) Occluded (95.7%) Scarf (93.4%) Glasses 75 71(94.7%) Hair 33 30(90.9%) Hand (90.9%) Hat (97.8%) Misc 28 26(92.85%) Table 3. Detailed results obtained by the face detector on the UMB-DB database. the non occluded part of the candidate is classified as face or non-face using a gappy-pca based classifier. Detection results are reported in Table 2. While in Table 3 we detail the results categorized by acquisition type. As it can be seen, the detector performs reasonably well on both neutral and non-neutral faces. When faces are occluded, the number of failures increases. The detection performances for the different types of occluding objects are also reported D Face Recognition For 3D face recognition we report the results obtained using a holistic approach based on the face restoration procedure presented in [11]. Each test has been performed enrolling only one neutral acquisition per subject. We considered the identity verification scenario, where the subject is supposed to claim an identity and the system must accept or rejects such a claim. During the evaluation, we considered each combination of test acquisition and claimed identity. Two widely used performance indices have been computed: the False Acceptance Rate (FAR, fraction of cases erroneously accepted in which the claimed identity differs from the actual identity) and the False Rejection Rate(FRR, fraction of cases erroneously rejected in which the claimed identity matches the actual identity). By varying the acceptance threshold, a Receiver Operator Characteristic (ROC) curve is obtained, and used to report the results. Being the faces acquired in an unconstrained scenario, a normalization procedure is required: in our experiments faces have been normalized in two ways: manual and automatic. In the latter case, the detector above described is used. In the manual case, we used the following procedure: a rough registration is first computed by registering the available feature points with those of the mean face model, as described in[3]. Once the rough registration has been computed, fine registration is achieved using the ICP algorithm betweentheinputfaceandthe meanfacemodel. The restoration module exploits the information provided by the non-occluded part of the face to recover the whole face, using an appropriate basis for the space in which non-occluded faces lie. Final classification is performed on the restored face using PCA. For details, see[11]. Figure 4 reports the performance obtained by the method, using the automatic and the manual normalization procedures. As it can be seen, there is not a significant difference in performance for these two procedures therefore, we detail only the results obtained using the automatic normalization(see Figure5). In Figure 6 we report the perfor- 2116

5 (a) (b) Figure 3. The seven annotated landmarks. 1 (c) (d) FRR (e) (f) PCA manual PCA auto Restoration manual Restoration auto 1 FAR Figure 4. ROC curves, in logarithmic scale, obtained on the entire database. Results using restoration with manual and automatic normalization are shown. (g) (h) Figure 2. Some examples of occluded acquisitions from the UMBDB: 2D color images (a, c, e, g); corresponding 3D depth images (b, d, f, h). mances detailed by object type. Scarf and hands seems to be the more problematic objects. Table 4 summarizes the results obtained by the algorithms considered in terms of Equal Error Rate (EER) for the identity verification scenario. We also considered the identification scenario, where the subjects make no claims about their identity, and the system must assign the correct one. This scenario is evaluated by computing the Identification Rate (IR) which measures the fraction of cases in which the correct identity is selected (i.e. the correct identity corresponds to the most similar acquisition in the gallery). Considering the results of the restoration proce- Normalization Test set EER IR Manual All cases All cases Neutral Expressions Occlusions % 69.6% 98.0% 66.7% 56.5% Table 4. Summary of face recognition performances obtained on the UMB-DB. dure obtained on artificial occlusions and on the Bosphorus Database reported in [11] (equal error rate of 11.5% and 10.9% respectively, on occluded faces), the UMB-DB represents a challenging test set even for a pipeline specifically designed for occluded acquisitions. 2117

6 1 using our database are, for example: verifyandcomparetherobustnessof2d,3dandmultimodal(2d + 3D) face detection algorithms; FRR verify and compare 2D, 3D and multimodal feature points localization algorithms; verify and compare part-based, holistic and reconstruction based approaches for the recognition of occluded faces(2d, 3D and multimodal); expressions neutrals occluded FAR Figure 5. ROC curves, in logarithmic scale, obtained using restoration on the entire database, subdivided by category(automatic normalization). FRR 1 hat scarf hand glasses misc 1 FAR Figure 6. ROC curves, in logarithmic scale, obtained using restoration on the set of occluded acquisitions. Results are displayed by object type (automatic normalization). 6. Conclusions and Future work We presented the UMB-DB, a database for the evaluation of face related algorithms in unconstrained scenarios. Thedatabaseisfocusedonfacialocclusionsandithasbeen collected with the aim to provide a challenging test bed to the research community. The results of the baseline algorithms show that the UMB-DB acquisitions represent a real challenge for current algorithms. Some investigations and open issues concerning face analysis in unconstrained scenarios that could be addressed explore the robustness of 3D normalization procedures; explore multimodal face normalization procedures. The database, including all the acquisitions and annotations, is publicly available at the address References [1] N.Alyüz,B.Gökberk, andl.akarun. A3dfacerecognition system for expression and occlusion invariance. In Proceedings of 2nd IEEE International Conference on Biometrics: Theory, Applications and Systems, pages 1 7, [2] S. Berretti, A. Del Bimbo, and P. Pala. Description and retrieval of 3d face models using iso-geodesic stripes. In Proc. 8th ACM Int l Workshop Multimedia information Retrieval, pages 13 22, [3] P.J.BeslandN.D.McKay. Amethodforregistrationof3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2): , [4] C. Beumier and M. Acheroy. Face verification from 3D and grey level clues. Pattern Recognition Letters, 22(12): , [5] A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Expression-invariant 3d face recognition. In Proceedings of Audio- and Video-Based Biometric Person Authentication, pages 62 70, [6] K. I. Chang, K. W. Bowyer, and P. J. Flynn. An evaluation of multimodal 2d+3d face biometrics. IEEE Trans. Pattern Analysis and Machine Intelligence, 27(4): , [7] A. Colombo, C. Cusano, and R. Schettini. A 3d face recognition system using curvature-based detection and holistic multimodal classification. In Proc. 4th Int l Symp. on Image and Signal Processing and Analysis, pages , [8] A. Colombo, C. Cusano, and R. Schettini. 3d face detection using curvature analysis. Pattern Recognition, 39(3): , [9] A. Colombo, C. Cusano, and R. Schettini. Face 3 a 2d+3d robust face recognition system. In Proceedings of 14th IEEE International Conference on Image Analysis and Processing, pages , [10] A. Colombo, C. Cusano, and R. Schettini. Gappy pca classification for occlusion tolerant 3d face detection. Journal of Mathematical Imaging and Vision, 35(3): ,

7 [11] A. Colombo, C. Cusano, and R. Schettini. Threedimensional occlusion detection and restoration of partially occluded faces. Journal of Mathematical Imaging and Vision, 40(1): , [12] C. Conde and A. Serrano. 3d facial normalization with spin images and influence of range data calculation over face verification. In CVPR 05: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR 05)- Workshops, page 115, Washington, DC, USA, IEEE Computer Society. [13] M. De Smet, R. Franses, and L. Van Gool. A generalized em approach for 3d model based face recognition under occlusions. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages , [14] T. Faltemier, K. Bowyer, and P. Flynn. Using a multiinstance enrollment representation to improve 3d face recognition. In First IEEE International Conference on Biometrics: Theory, Applications, and Systems., pages 1 6, [15] B. Gökberk, L. Akarun, and B. Aksan. How to deceive a face recognizer? In Proceedings of the Biometrics: Challenges arising from Theory to Practice Workshop, pages 35 38, [16] R. Gross. Face databases. In Handbook of Face Recognition, chapter 13, pages [17] S.Gupta,K.Castleman,M.Markey,andA.Bovik. Texas3D Face Recognition Database. In IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI), pages , [18] T. Heseltine, N. Pears, and J. Austin. Three-dimensional face recognition using combinations of surface feature map subspace components. Image and Vision Computing, 26(3): , [19] Y.Hu,Z.Zhang,X.Xu,Y.Fu,andT.Huang. Buildinglarge scale 3D face database for face analysis. In Proceedings of the 2007 international conference on Multimedia content analysis and mining, pages Springer-Verlag, [20] B. Hwang and S. Lee. Reconstruction of partially damaged face images based on a morphable face model. IEEE Trans. on Pattern Analysis and Machine Intelligence, 25(3): , [21] J. Kim, J. Choi, J. Yi, and M. Turk. Effective representation using ica for face recognition robust to local distortion and partial occlusion. IEEE Trans. Pattern Analysis and Machine Intelligence, 27(12): , [22] D. Lin and X. Tang. Quality-driven face occlusion detection and recovery. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages 1 7, [23] A. M. Martinez. Recognition of partially occluded and/or imprecisely localized faces using a probabilistic approach. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, volume 1, pages , [24] A. M. Martinez. Recognizing imprecisely localized, partially occluded and expression variant faces from a single sample per class. IEEE Trans. Pattern Analysis and Machine Intelligence, 24(6): , [25] Z. Mo, J. P. Lewis, and U. Neumann. Face inpainting with local linear representations. In British Machine Vision Conference, volume 1, pages , [26] A.MorenoandA.Sanchez. Gavabdb: A3dfacedatabase. In Proc. 2nd COST275 Workshop on Biometrics on the Internet, Vigo(Spain), pages 75 80, [27] P. Nair and A. Cavallaro. 3-d face detection, landmark localization, and registration using a point distribution model. IEEE Transactions on Multimedia, 11(4): , [28] B. Park, K. Lee, and S. Lee. Face recognition using facearg matching. IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(12): , [29] J. S. Park, Y. H. Oh, S. C. Ahn, and S. W. Lee. Glasses removal from facial image using recursive error compensation. IEEE Trans. Pattern Analysis and Machine Intelligence, 27(5): , [30] P. Phillips, P. Flynn, T. Scruggs, K. Bowyer, J. Chang, K.Hoffman,J.Marques,J.Min,andW.Worek. Overviewof the face recognition grand challenge. In IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pages , [31] A. Savran, N. Alyüz, H. Dibeklioglu, O. Çeliktutan, B. Gökberk, L. Akarun, and B. Sankur. Bosphorus database for 3d face analysis. In The First COST 2101 Workshop on Biometrics and Identity Management, pages 47 56, [32] D. Skocaj and A. Leonardis. Robust recognition and pose determination of 3-d objects using range images in eigenspace. In Third International Conference on 3-D Digital Imaging and Modeling, pages , [33] F.Tarrés anda. Rama. A novel method for face recognition under partial occlusion or facial expression variations. In Proc. 47th Int l Symp. ELMAR, pages , [34] P. Viola and M. Jones. Robust real-time object detection. International Journal of Computer Vision, 57(2): , [35] L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato. A 3d facial expression database for facial behavior research. In FGR 06: Proceedings of the 7th International Conference on Face and Gesture Recognition, pages , Washington, DC, USA, IEEE Computer Society. [36] W.Zhang,S.Shan,X.Chen,andW.Gao. Localgaborbinary patterns based on kullbackleibler divergence for partially occluded face recognition. IEEE Signal Processing Letters, 14(11): , [37] C.Zhong,Z.Sun,andT.Tan. Robust3dfacerecognitionusing learned visual codebook. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1 6,