Performance Enhancement of 2D Face Recognition via Mosaicing

Transcription

1 Performance Enhancement of D Face Recognition via Mosaicing Richa Singh, Mayank Vatsa, Arun Ross, Afze Noore West Virginia University, Morgantown, WV 6506 {richas, mayankv, ross, noore}@csee.wvu.edu Abstract We describe a face mosaicing scheme that generates a composite face image during enroment based on the evidence provided by fronta and semi-profie face images of an individua. Face mosaicing obviates the need to store mutipe face tempates representing mutipe poses of a user s face image. In the proposed scheme, the side profie images are aigned with the fronta image using a terrain transform that expoits neighborhood properties to determine the transformation reating the images. Mutiresoution spining is then used to bend the side profies with the fronta image thereby generating a composite face image of the user. A oca binary pattern matching agorithm is used to compare an input face image with the tempate face mosaic. Experiments conducted on a sma dataset of 7 users indicate that face mosaicing as described in this paper offers significant benefits by accounting for the pose variations commony observed in face images. 1. Introduction The probem of D face recognition continues to pose severa chaenges even after 30 years of research in this fied [1]. Most D face recognition agorithms described in the iterature are sensitive to changes in iumination, pose, facia expressions/accessories, etc. Designing pose-invariant agorithms is especiay very chaenging as discussed in the FRVT (Face Recognition Vendor Test) 00 report []. Severa methods have been suggested to address the probem of pose variations incuding the use of morphabe modes [15], 3D facia imaging [16], mutipe tempates [18], and muticassifier fusion [17]. In this work we propose the use of a mosaicing scheme to generate a D face mosaic of an individua during enroment that can be successfuy used to match various poses of a person s face. Mosaicing uses the fronta and side-profie face images (D) of a user to generate an extended D image. The goa is to adequatey summarize the 3D surface of an individua s face in a D pane, without attempting to compute the 3D structure of the face. This avoids the compexity of generating 3D structure information from mutipe (registered) D images. Mosaicing aso obviates the need to store mutipe tempates of a user during enroment thereby optimizing storage demands and processing time. The potentia of mosaicing facia images has been examined by a few researchers. Yang et a. [5] proposed an agorithm to create panoramic face mosaics. Their acquisition system consists of five cameras that simutaneousy obtain five different views of a subject s face. In order to determine the corresponding points in mutipe face views, the authors pace ten coored markers on the face. Based on these contro points, their agorithm uses a series of fast inear transformations on component images to generate a face mosaic. Finay, a oca smoothing process is carried out to smooth the mosaiced image. Two different schemes were used to represent the panoramic image: one in the spatia domain and the other in the frequency domain. The frequency representation resuted in an identification accuracy of 97.46% whie the spatia representation provided 93.1% accuracy on a database of 1 individuas. Liu and Chen [6] proposed a face mosaicing technique that uses a statistica mode to represent the mosaic. Given a sequence of face images captured under an orthographic camera mode, each frame is unwrapped onto a certain portion of the surface of a sphere via a spherica projection. A minimization procedure using the Levenberg-Marquardt agorithm is empoyed to optimize the distance between an unwrapped image and the sphere. The representationa mode (statistica) comprises of a mean image and a number of eigen-images. The novety of this technique is (a) the use of spherica projection, as opposed to cyindrica projection, which works better when there is head motion in both the horizonta and vertica directions, and (b) the computation of a representationa mode using both the mean image and the eigen-images rather than a singe tempate image. Athough the authors state that this method can be used for face recognition, no experimenta resuts have been presented in the paper. In [7], the authors have proposed another agorithm in which the human head is approximated with a 3D eipsoida mode. The face, at a certain pose, is viewed as a D projection of this 3D

2 eipsoid. A D face images of a subject are projected onto this eipsoid via geometrica mapping to form a texture map which is represented by an array of oca patches. Matching is accompished by adopting a probabiistic mode to compute the distance of patches from an input face image. The authors report an identification accuracy of 90% on the CMU PIE database. Face mosaicing has aso been used in other fieds such as facia animation and rendering [7], [9], 3D face image generation [10], etc. However, these agorithms generate the face mosaic using compex modes which do not necessariy preserve the biometric features of the face. The primary focus of this work is the impementation of a simpe mosaicing and matching scheme for face recognition that is robust to variations in pose. It is assumed that mutipe shots of a user s face image representing various poses are avaiabe at the time of enroment. First, the face is segmented from each image using the Gradient Vector Fow technique [1] and then normaized using aignment techniques proposed in the foowing sections. Mosaicing is accompished using muti-resoution spines based on Gaussian and Lapacian pyramids [11]. Mutiresoution spines aso perform bending as an integra part of mosaicing thereby offering some inherent advantages. The performance of the mosaiced image is evauated using a texture-based face recognition agorithm ([13], [14]). Experiments indicate that mutiresoution spines preserve the textura features of the face necessary for recognition whie enhancing the performance of face recognition in terms of matching accuracy, and time and memory requirements.. Face Mosaicing Face mosaicing, as described in this paper, consists of three major steps: (1) Determining the pair-wise transformation necessary to aign the faces obtained during enroment; () Generating a mask; and (3) Stitching and bending..1. Transformation and Aignment of Faces Before mosaicing the images, it is necessary to transform the mutipe images obtained during enroment into a common pane. Here, we assume that 3 images are made avaiabe during enroment: (a) fronta, (b) semi-eft and (c) semi-right. Aso, the side profie images are assumed to have a rotation of at east 30 0 with respect to the fronta image in order to ensure that sufficient information about the face is avaiabe. Since the camera-to-face distance is fixed in a 3 images, a transformation to hande rotation, aignment and deformation due to various facia expressions is computed. Step 1. Three bank images of size 56 x 56 are first created. Step. The coordinates of the eyes and nose objects are determined in the fronta face image (this can be done using any standard eye-nose finding agorithm). The fronta image is paced on one of the bank image spaces such that its nose coordinates are at the center, i.e., (18, 18) (Figure 1(a)). Step 3. Simiary, the three coordinates (two eyes and one nose) are ocated in the side-profie face images. Based on their respective nose coordinates, these images are paced in the two remaining bank image spaces (Figure 1(b), 1 (c)). Step 4. Based on the positions of the eyes and the nose, a terrain transform is used to aign each of the side profie images with the corresponding fronta face image as shown in Figures and 3. (a) (b) (c) Figure 1: Image initiaization. (a) Fronta image paced on the bank space (nose coordinates are at (18, 18)); (b) and (c) The two profie images in the 56x56 image space. Terrain transform has been extensivey used in the computer graphics community. This transformation was adopted in this work since the neighborhoods of two corresponding points are never rigorousy identica. Typicay, in face images there are differences in geometry and reative distortions caused by the oca reief of the object, i.e. the terrain effect. Furthermore, even in the absence of geometrica distortions, subpixe shifting can occur. This probem can be addressed by geometricay transforming the neighborhood area of one image with respect to the other. Such a transform is not known a priori and depends on the unknown shape of the terrain (neighborhood). Let F 1 and F be the two facia profies; the goa is to transform F as per the oca geometry of F 1. Let f 1 k and f be the oca points (i.e., eyes and nose) on F 1 and F, respectivey, where k is the number of oca features (i.e., 3). Next, a oca rectanguar region is associated with each of these features. The oca eye region is centered about the eye and is defined based on its distance (both horizonta and k

3 vertica) from the edge of the face as iustrated in Figure. Thus, it varies across individuas and poses. The nose region, on the other hand, is 50 pixes wide and 50 pixes ong (38 pixes above the nose and 1 pixes beow it). The dimensions of these regions have been empiricay determined based on experiments conducted on a sma dataset. The terrain transform, between two images, operates by determining the transformation that resuts in the maximum correation between two corresponding oca regions. Ony rotation (R), transation (T) and scaing (S) have been considered in this work. Thus, the foowing criterion function, C(R, T, S), is used to derive the optima transformation parameters. C( R r T t S i= 1 =, =, = ) = k s k i= 1 p ρ i i pi. (1) Here, i denotes the correation of the corresponding oca regions at a particuar transformation (r, t, s), and p i = 1 if the corresponding features are avaiabe in both images (0, otherwise). A sma range of (r, t, s) vaues were used in the search process and, hence, an exhaustive set of possibiities were considered to derive the optima (r*, t*, s*) vaues. Figure 3 shows the transformed images with respect to the fronta face image. Figure : The oca regions considered in a pair of images. (a) (b) (c) Figure 3: Effect of appying the terrain transform. (a) Fronta Image; (b) and (c) Transformed profie images with respect to the fronta image... Mask Generation Once the aignment between an image pair is determined, a bending procedure is used to create a composite image. The chaenge in face mosaicing is to ensure that saient facia features (viz., eye, nose, chin, face contour, etc.) are not distorted during the bending process. In this regard we make the foowing two observations: (a) the seam in the upper region of the face is mosty vertica; (b) the seam in the ower portion of the face has to accommodate the user s chin which is sighty santed. Thus, we deveop a binary mask that defines the regions pertaining to the fronta and side profies to be retained in the composite image. The shape and size of the mask varies across individuas and depends on the rotation of the face. Thus, this is a dynamic mask generated at runtime. Figure 4 shows an exampe of the mask generated from the eft and right profies of a user. Note that the mask presents a strict boundary between the images. In order to soften this, we subject the mask to a Gaussian weighting function. Figure 4: The masks generated from two profie images..3. Stitching and Bending For combining two images we use Mutiresoution Spines [11]. Image spining can be done based on simpe spine weighting function, but the quaity of the stitched image depends on the step size that is chosen. A arge step size may ead to burring whereas a sma step size might resut in discontinuities at the boundary. To overcome this probem, Burt and Adeson [11] used mutiresoution spines to determine different step sizes for various frequency components. In this technique, a sequence of ow pass fitered images is obtained by iterativey convoving each image with a D Gaussian fiter kerne. The resoution of the image between successive iterations is reduced and, therefore, the Gaussian kerne operates on a reduced version of the origina image each time. The resutant images G 0, G 1,,G N may be viewed as a pyramid structure with G 0 having the highest resoution (owermost eve) and G N having the owest resoution (uppermost eve). Let {w(m,n)}, m,n = 1..5, represent the Gaussian kerne. Then this process can be summarized by the foowing equation: G = Reduce G ], 0 < < N, () 5 [ 1 i.e., G = w( m, n) G (i + m, j n. (3) m, n= )

4 Figure 5: Leves in the Gaussian Pyramid expanded to the origina size to see the effects of the ow pass fiter. The effect of convoution (Figure 5) is to bur the image, thereby reducing the fiter band imit by an octave from eve to eve whist reducing the sampe density by the same factor. The mutiresoution spine as described in [11] requires band pass images (as opposed to ow pass images). Band pass images are computed by interpoating the image in each eve of the pyramid and then subtracting it from the next owest eve. This resuts in a sequence of band pass images that can be viewed as a Lapacian pyramid (L 0,L 1,,L N ). The term Lapacian is used since the Lapacian operator resembes the difference of Gaussian-ike functions. These band-pass images are reay a resut of convoving the difference of two Gaussians with the origina image. The steps used to construct this pyramid can aso be used to exacty recover the origina image. The process described above may be summarized as foows: L G Expand G ], 0 < < N, (4) = [ + 1 i + m j + n G ( i, = 4 G (,. (5), k i, k 1 ) m, n= Here, the Expand[.] operator interpoates a owresoution image to the next highest eve. Note that G,k in (5) denotes expanding G k number of times. Various features of the face are segregated by scae in the different eves of the pyramid. Hence, the textura features of the face are preserved over mutipe eves of the pyramid. Let LA and LB represent the Lapacian pyramids of the two images that are being spined. Let GR be the pyramid associated with the Gaussianweighted mask discussed in Section.. The mutiresoution spine, LS, is then computed as, LS ( i, = GR ( i, LA ( i, + (1 GR ( i, LB ( i,, (6) where is the eve of the pyramid. The spined images at various eves are expanded and summed together to obtain the fina face mosaic as shown in Figure Experimenta Resuts The performance of the proposed scheme was tested on a subject set from West Virginia University (WVU) as we as the database used in [5]. The WVU database consists of images of fifteen individuas (9 images per individua) obtained across three different sessions without any constraint on iumination and pose. The face database used in [5] consists of images from tweve individuas (1 images per individua) across four different sessions 1. Figure 7 shows a few exampes of face mosaics obtained using the proposed agorithm. Figure 6: The eft, fronta and right profie face images aong with the mosaiced face image. To vaidate the performance of mosaicing we used the face matching agorithm described in [14]. For each of the 7 users, a set of 3 images (1 fronta and semiprofie) were used as tempates (S1) whie the remaining images (S) were used to test the matching performance of the system. The images in S1 were used to generate the face mosaic for an individua. The foowing experiments were conducted: (a) Each image in S was compared against each of the fronta tempates in S1 to generate match scores (b) Each image in S was compared against a set of 3 images in S1 (pertaining to a singe user) to generate 3 different match scores. These scores were combined using the sum rue and the min rue. (c) Each image in S was compared against the face mosaic generated using the 3 tempate images of a user in S1 in order to generate match scores. A mosaiced face image contains a the features in a face whie the fronta and the side profie images have ony a imited number of features. Therefore, existing face recognition agorithms such as appearance-based or feature-based may not be abe to operate on mosaiced images reiaby. So we use a oca representation of the textura features in order to account for variations in the number of features across different images and individuas. The query image (D non-mosaiced) is registered on a 56x56 pane using the nose coordinate. The oca features are then extracted from the face, converted into poar coordinates and convoved with a D og poar Gabor 1 The origina dataset consisted of 0 images per individua; however, we used ony a subset of this in our experiments.

5 waveet [14]. The phase information of these oca textura facia features is encoded to form a binary tempate which is used in a siding weighted matching scheme. This scheme gives more emphasis to features that are matched with higher confidence. For exampe, if a fronta face is matched with a mosaiced face, the siding weighting scheme assigns more weight to the eyes, nose, and mouth, whereas if a profie face is used, the ear, eye and nose are given prominence. Figure 8 iustrates the steps invoved in comparing the mosaiced face image with a non-mosaiced face image. Figure 9 shows the ROC pot corresponding to these experiments; it is observed that the performance of face recognition is significanty improved with the use of mosaicing technique. We aso evauated the time compexity of the agorithm. Recognition with mosaiced image took 1.11 CPU-time whie the sum/min rue based approach took 1.74 CPU time. The verification time using the mosaiced image is ess because the mosaicing agorithm takes 0.53 CPU-time whie verification takes 0.58 CPU-time; however, in the sum/min rue case we have to execute the verification agorithm three times before invoking the sum/min rue. Memory requirement without mosaicing is 3 x m bytes where as it is approximatey 1.1 x m bytes for the mosaiced image (m is the number of bytes needed to store 1 profie image). These resuts show that the mosaicing process enhances the performance of face recognition agorithm whist reducing the memory requirement and the matching time. In Tabe 1 we compare the proposed agorithm with two other agorithms described in the iterature. Tabe 1: A brief comparison of three different mosaicing schemes. Yang et.a. [5] Liu & Chen [7] Proposed agorithm Subjects Affine using Affine using Affine using Registration points trianges regions Mutiresoution Geometrica Mosaicing Concatenation mapping spines Representation Spatia: PCA Frequency: FFT ampitude Statistica mode 5. Acknowedgements Authors are gratefu to Prof. Herve Abdi and Dr. Fan Yang for providing the face database [5]. Thanks to a the vounteers who participated in this project. Loca binary pattern 4. Summary and Future Work We have described a face mosaicing procedure that creates a composite image of an individua s face by bending the fronta and side profie views. The proposed scheme utiizes terrain transforms to register the views and mutiresoution spines to bend the views together. The bending process is observed to be robust, i.e., it does not distort the saient features of the face. Experimenta resuts suggest that face mosaicing is a good aternative to storing mutipe views of a user s face. Currenty, we are examining ways to improve the registration process. We are aso ooking at nove agorithms to perform matching in the mosaic domain. Experiments wi then be conducted on arger datasets in order to demonstrate the efficacy of the technique. Figure 7: Exampes of mosaiced face tempates. For two of the subjects mutipe mosaics are shown. The CPU-time was cacuated in a MATLAB environment.

6 6. References [1] W. Zhao, R. Cheappa, A. Rosenfed, and P.J. Phiips, Face Recognition: A Literature Survey, ACM Computing Surveys, 35(4): , December 003. [] P.J. Phiips, P. Grother, R.J Micheas, D.M. Backburn, E. Tabassi, and J.M. Bone, FRVT 00: Evauation Report, 003. [3] N.K. Ratha, J.H. Conne and R.M. Boe, Image mosaicing for roed fingerprint construction, Proceedings of ICPR, : , August [4] A. Jain, A. Ross, Fingerprint mosaicking, Proceedings of ICASSP, 4: IV-4064 IV-4067, May 00. [5] F. Yang, M. Paindavoine, H. Abdi, and A. Monopoy, Deveopment of a fast panoramic face mosaicing and recognition system, Optica Engineering, 44, 005, (in press). [6] X. Liu and T. Chen, Geometry-assisted statistica modeing for face mosaicing, Proceeding of IEEE ICIP, : , September 003. [7] X. Liu and T. Chen, Pose-Robust Face Recognition Using Geometry Assisted Probabiistic Modeing, Proceedings of CVPR, 1:50 509, June 005. [8] G. Borshukov and J.P. Lewis, Reaistic human face rendering for the matrix reoaded, cinematography.org/pubications/acrobat/face-s003.pdf). [9] S.B. Kang, A Survey of Image-based Rendering Techniques, Cambridge Research Laboratory Technica report - CRL 97/4, [10] K.W. Bowyer, K. Chang, and P. Fynn, A survey of 3d and muti-moda 3d+d face recognition, Notre Dame Department of Computer Science and Engineering Technica Report, 004. [11] P.J. Burt and E.H. Adeson, A mutiresoution spine with appication to image mosaics, ACM Transaction on Graphics, : 17-36, [1] C. Xu and J.L. Prince, Snakes, Shapes, and Gradient Vector Fow, IEEE Transactions on Image Processing, 7(3): , March [13] R. Singh., M. Vatsa, and A. Noore, Textura feature based face recognition for singe training images, IEE Eectronics Letters, 41(11): 3 4, May 005. [14] R. Singh, M. Vatsa, and A. Noore, Face Recognition using Scanned Images, Pattern Recognition, 005 (Submitted). [15] V. Banz, S. Romdhami, and T. Vetter, Face identification across different poses and iuminations with a 3D morphabe mode, Proceedings of Internationa Conference on Automatic Face and Gesture Recognition, 0-07, May 00. [16] K.I. Chang, K.W. Bowyer, and P.J. Fynn, An evauation of muti-moda D+3D face biometrics, IEEE Transactions on Pattern Anaysis and Machine Inteigence, 7(4): , 005. [17] X. Lu, Y. Wang, and A.K. Jain, Combining Cassifiers for Face Recognition, Proceedings of IEEE Internationa Conference on Mutimedia & Expo, III:13-16, Juy 003. [18] U. Uudag, A. Ross, and A.K. Jain, Biometric Tempate Seection and Update: A Case Study in Fingerprints, Pattern Recognition, 37(7): , Juy 004. Mosaiced Face Loca Feature Extraction (a features) Texture Encoding with D og poar Gabor Loca Feature Extraction (b features) Texture Encoding with D og poar Gabor Siding Weighted Matching Scheme Matching Score Registered Non Mosaiced Face Figure 8: Bock diagram of the matching scheme that was used to compare face images (mosaiced and profie). Figure 9: Performance evauation using the proposed scheme. The min fusion rue is observed to perform we aso. However, the mosaiced image resuted in the best performance.