Improving Alignment of Faces for Recognition

Transcription

1 Improving Alignment o Faces or Recognition Md. Kamrul Hasan Département de génie inormatique et génie logiciel École Polytechnique de Montréal, Québec, Canada md-kamrul.hasan@polymtl.ca Christopher J. Pal Département de génie inormatique et génie logiciel École Polytechnique de Montréal, Québec, Canada christopher.pal@polymtl.ca Abstract Face recognition systems or uncontrolled environments oten work through an alignment, eature extraction, and recognition pipeline. Eective alignment o aces is thus crucial as can be an entry point in the process and poor alignments can greatly aect recognition perormance. The task o alignment is particularly diicult when a ace comes rom highly unconstrained environments or so called aces in the wild. A lot o recent research activity has ocused on aces in the wild and even simple similarity or aine transormations have proven both eective and essential to achieving state o the art perormance. In this paper we explore a straightorward, ast and eective approach to aligning aces based on detecting acial landmarks using Haarlike image eatures and a cascade o boosted classiiers. Our approach is reminiscent o widely used ace detection approaches, but ocused on much more detailed eatures o a ace such eye centres, the nose tip and corners o the mouth. This process generates multiple candidates or each landmark and we present a ast and eective iltering strategy allowing us to ind sets o landmarks that are consistent. Our experiments show that this approach can outperorm contemporary methods and easily its into popular processing pipelines or aces in the wild. I. INTRODUCTION The human ace is an important source o biometric inormation. Face recognition is a well studied problem with many practical applications including: document control (digital chip in passports, drivers licenses), transactional authentication (credit cards, ATMs), physical access control (smart doors), law enorcement and police investigations (identiying suspects in security camera video) and security (user access veriication). Given a query or test ace (rom a still image or a video rame) and a database o stored aces, the ace recognition problem can be ormulated in or two dierent scenarios: Identiication: Identiy the person (i they are present) within a database o aces, or Veriication: For a second ace image (rom inside or outside o a database), decide whether these two aces represent the same person or not. The experiments we present in Section III ocus on veriication, the second scenario above; however, our approach should be well suited to improve identiication results as well. In the long history o ace recognition research, numerous evaluation benchmarks have been produced - some o the most important ones are compiled in table I. O particular note is the Face Recognition Vendor Tests (FRVT) [1], which were a set o independent evaluations o commercially available TABLE I IMPORTANT FACE DATABASES Name no o subjects Variations and images FERET 1199; p,il,e,i,o,t AR Face 126; 4000 il,e,o,t Database CMU-PIE 68; 61,368 p,il,e FRGC Database 466; 50,000 il,e,i Caltech ; natural web aces LFW 5749; 13,233 natural PubFig 200; 59,476 natural p: pose, il: illumination, e: expression, i: indoor, o: outdoor, t: time, -: unknown TABLE II THE REDUCTION OF ERROR RATE FOR FERET, AND THE FRVT [1] Year o Evaluation FRR at FAR= (Turk and Pentland 0.79 [2](Partially automatic)) 1997 (FERET 1996) (FRVT 2002) (FRVT 2006) 0.01 and prototype ace recognition technologies conducted by the US government in the years 2000, 2002 and The FRVT is considered to be a very important milestone in ace recognition evaluations; it also includes three previous ace recognition evaluations the FERET evaluations o 1994, 1995 and Table II quantiies the improvement at our key milestones, where, or each milestone, the FRR (False Rejection Rate) at a FAR (False Acceptance Rate) o (1 in 1000) is given or a representative state-o-the-art algorithm [1]. The lowest error rate o 0.01 was achieved by the NV1-norm algorithm (Neven Vision Corporation) on the very high-resolution still images and by V-3D-n algorithm (L-1 Identity Solutions) on the 3D images among. These results were impressive, however, the problem is that the FRVT benchmarks were generated rom controlled environments, and hence are limited in their applicability to natural environments.

2 One might imagine two dierent strategies or obtaining a more realistic assessment o the perormance o a ace recognition system: (i) Test the models independently on dierent subtasks like pose, illumination, expression recognitions, and deduce some collective perormance metric as a sum ; or (ii) Build a dataset that accumulates the possible maximal variability o (pose, expression, aging, illumination, race, occlusion etc.) in the same benchmark, and perormance evaluations are done on it. Labeled Faces in the Wild (LFW)[3] is a dataset that was constructed rom the second view point. Table III(a) shows a set o LFW George W. Bush images rom this benchmark. The inherent natural variability o aces can be veriied rom the examples there. Generally, a ace recognition pipeline or the in the wild setting [4], [5], [6] works through an (1) alignment, (2) eature extraction, and (3) recognition pipeline. Table III (b,c) shows the aligned images o table III (a) with two popular ace aligner: (b) the congealing and unneling [4], and (c) a iducial point based commercial aligner (LFWa dataset) by L. Wol [7]. It can be veriied rom the LFW results site [8] that the best recognition results were achieved on the aligned dataset, compared to the raw images. The eature extraction step o a aces in the wild recognition pipeline typically selects (i) the whole image, or (ii) the ace detection bounding box area (table IV(a)), or (iii) a smaller central patch cut our o aligned images (table IV (b) and (c)). Features are then computed using one o these underlying representations. Examples o the best ace veriication results reported so ar include Descriptor Based Methods in the Wild (DBMW) [9], Cosine Similarity Metric Learning (CSML)[5], and Probabilistic LDA [6]. Most o these methods use the center patch strategy. Top perorming methods on the LFW benchmark have used simple image transormations based on aine or similarity transorms. For example, the LFW benchmark website itsel provides aces that have been aligned using Learned-Miller s congealing algorithm [10]. Most groups posting results to the LFW evaluation have been using aces that have been aligned using congealing as it produces higher perormance results. Some recent work such as [7], has replace the congealing based alignment step with a commercial system and other work such as the CSML approach in [5] use this commercially aligned variant o the LFW data known as LFWa. Irrespective o the alignment method it is common to then simply cut out a central rectangle rom the aligned image orm a rectangular patch used to build eature descriptors. In particular, the DBMW strategy takes LFW-unneled (or congealed) images and crops out a pixel rom the center o the image. The CSML strategy crops a pixel image rom the LFWa commercially aligned imagery. Some example patches or the DBWM and CSML setup are shown in table IV (b,c). We also show the ace bounding box produced by the OpenCV 2.1 ace detector in table IV (a) and some results using our complete method in (c). It is evident TABLE III A SET OF LFW ORIGINAL, CONGEALLED, AND ALIGNED IMAGES (a) (b) (c) (d) LFW image LFW LFWa ours (congealled) (aligned) rom these examples that the center patches sometime miss important ace inormation; one can think o this as noise maniesting as translation and scaling artiacts. The concrete impact o these errors in the alignment is that the central patch o the image can be poorly centred on the ace. In this paper we test our hypothesis that i the alignment step prior to cutting out a central rectangle could be improved, a better eature descriptor could be computed, and thus the recognition perormance could be improved. We also explore what happens we simply do a better job o cutting out the inal rectangle. The remainder o this paper is structured as ollows. In section II, we will briely describe our proposed model. Section III is a compilation o some experiments and the corresponding results. Finally, section IV concludes the paper with some uture research ideas. II. OUR APPROACH Our approach begins by detecting a set o landmarks or iducial points on a ace image using a Viola-Jones style boosted cascade o Haar-like image eatures. This procedure

3 TABLE IV IMAGE PATCHES, USE BY DIFFERENT ALGORITHMS (a) (b) (c) (d) bbox DBMW CSML ours the detected ace is then deined and used to crop the ace out o the original image into a slightly smaller image. We then search within this smaller image or ive acial landmarks : the let eye, the right eye, the nose tip and the two corners o the mouth. To detect these landmarks we have trained ive dierent boosted cascade based classiiers using Haar-like eatures (again using the Viola-Jones inspired implementation ound in OpenCV) using two data sets o labelled iducial points: the BioID database and the PUT [11] database. Previous work using Haar cascade classiiers trained on PUT was able to properly detect and localize eyes 94% o the time with a alse positive rate o 13% using a heuristic procedure or inding eye pairs [12]. In our approach as well the Haar cascade classiiers produce a number o candidates or each o our ive dierent landmarks. We ilter these candidate landmarks and identiy a set o two to three geometrically consistent acial landmarks using the ollowing procedure. A. Fiducial point iltering The iltering pipeline works in two steps. First, a list o easy alse positives are removed through a heuristic rule ilter. A set o simple rules, or example: (i) points within the border area o certain width are discarded, (ii) the nose must not be within the upper 1/3 area o the ace, (iii) The let eye must be within a certain region in the upper let corner, right eye in the top right, (iv) The mouth should be in the lower hal o the ace region are used a rules or this iltering step. The points that make it past the heuristic iltering become the candidate points that which are evaluated under simple probabilistic model or the spatial positions o all points. We estimate the parameters o a diagonal covariance Gaussian model in 2D or the spatial iducial point distributions o points in the PUT database. Figure 1 illustrates the positions o our ive landmarks within the PUT database. Images were scaled to a size o pixels ollowing the LFW ace benchmarking process[13]. will produce multiple candidates or each landmark. We thus need to ilter detections and identiying a set o landmarks that are consistent with a ace. We describe our procedure or doing this in more detail below. Once we have a set o consistent landmarks we can transorm or register the ace into a common coordinate system using a simple aine, similarity or simpler transormation such that when a central patch is cut out o a larger image, this patch is well centred on the ace. When processing new images we use the Viola-Jones ace detector ound in OpenCV to ind aces. In our experiments here multiple ace detections are iltered by selecting the largest ace. A border o approximately one third the height o Fig. 1. The spatial distribution o the ive landmarks used here within the aces o the PUT database. The let eye, right eye, nose tip, let mouth corner and right mouth corner x,y coordinates are shown as black, blue, green, red and yellow markers respectively. Let, the shape o a ace, x (j) be deined through a set o points {x i, y i } n i=1, where (x i, y i ) is the location o a iducial point, and n is the number o points deining the ace. Let, a subset o m points rom those n points constitute a spatial coniguration, (A 1 A 2... A m ). The probability o observing

4 this coniguration is P (A 1 A 2... A m ) = m P (A i ), (1) i=1 where, P (A i ) = N(µ i, Σ i ). I only one category o landmark is detected, our system does nothing. I we detect candidates or our or ewer categories o landmark, we enumerate all candidates and ind the coniguration with maximal probability under our model. I candidates are detected or 5 categories o landmarks, we use a procedure o randomly selecting three dierent landmark categories and enumerate all possible combinations. For each combination o points in the coniguration we compute its probability under our model. The process is repeated a number o times depending on our time constraints (or our experiments here we randomly identiy 10 combinations o three points and enumerate all possible candidates), and identiy the most probable combination. Table V shows the noisy output points, while table VI shows the points that passed the rule ilter, and the white dots are the model outputs. B. Similarity Transormation We use a similarity transormation to register aces and we thus need at least two landmarks to register a ace to a common coordinate rame. Let, [x y] T be the coordinates o a iducial point detected on a ace, and [u v] T be the corresponding reerence point position on a reerence ace that we have computed by taking the average or each landmarks coordinates over the BioID database. A similarity transormation can be represented as [ [ ] [ ] [ ] u m1 m = 2 x tx + (2) v] m 1 y t y m 2 where, [t x t y ] T is the translation parameter vector, m 1 = s cos θ and m 2 = s sin θ, are two other parameters which contain the traditional parameters o rotation, θ and scale, s. This deines the transormation parameter vector, T = [m 1, m 2, t x, t y ] T. To solve or the transormation parameters the problem can be re-ormulated as a system o linear equations x 1 y u 1 y 1 x m 1 v 1 x 2 y y 2 x m 2 t x ty = u 2 v where, (x i, y i ) is a eature point on the observed ace, {x i, y i } n i=1, while (u i, v i ) is the corresponding target point on the latent ace {u i, v i } n i=1. This re-ormulation o a similarity transormation is similar to the commonly used reormulation o an aine transormation as discussed in [14]. We can write the system in matrix notation as Ax = b, such that x = A 1 b. This approach provides the transormation parameters or a correspondance iducial point set between an (3) observed ace and our average or latent ace. These parameters are used to register the unaligned ace to a common coordinate rame. For two reerence points placed in correspondence with a reerence ace one can solve or x exactly; however, or three points (or more), one can obtain a least squares estimate through computing a pseudo inverse, x = (A T A) 1 A T b. Ater transorming the image the inal pixel values in the aligned image are calculated through bi-linear interpolation. Table III (d) shows the aligned George W. Bush images through this methodology. C. Cutting out the central patch This is the inal step o the piple line where, irst, the ace detector is run again on the aligned image. The eye pair locations are used to estimate a reerence point, (x c, y c ). Let, (x le, y le ) and (x re, y re ) be the let and the right eye locations, and W and H be the width and height o the ace window, returned by the ace detector. Then, the ace patch, selected by the model is: {(x c r w W, y c r hu H), (x c + r w W, y c + r hb H)}, where, x c = (x re x le )/2 is the x axis location o the reerence point and y c = (y re y le )/2 is the y axis value, r w is the width actor and r hu and r hb are the height actors. Finally, the selected area is rescaled to a constant size, say, m n. III. EXPERIMENTS AND RESULTS We used Bio-id [15] and PUT[11] datasets or local iducial point classiiers learning, while spatial point distributions were learned rom the PUT[11] dataset. The models were tested or the ace veriication task on the challenging LFW dataset[13]. In addition, the iducial point localization accuracies were tested on an unseen Bio-id dataset. A. Dataset 1) Bio-Id:: This dataset consists o 1521 gray level images o resolution or 23 dierent subjects. The dataset also includes 20 manually marked iducial points on each ace. 2) LFW: LFW dataset comes with three dierent versions: LFW (ununneled): The raw images, as collected. LFW (unneled) : The images were aligned through a pipeline called congealing and unnelling [4]. LFWa : A dataset by L. Wold [7], aligned through a commercial aligner. Each version has two settings: View 1: This data set is or model development, and it comes with a a precise training test split. The training data contains 1100 positive pairs and 1100 negative pairs, while the test data is with 500 positive and 500 negative pairs. Training and test data are mutually exclusive. Although provided in this strict setting, this data could be used as one likes, however the perormance reporting should strictly ollow the ollowing view 2 set up. View 2: This data is is compiled or any perormance reporting. The data contains 10 olded splits, each with 300 positive and 300 negative pairs.

5 TABLE V INITIALLY DETECTED LANDMARKS TABLE VII LOCALIZATION ACCURACIES point accuracy Light eye.081 ±.04 Right eye.086 ±.05 Nose tip.104 ±.06 Let mouth corner.28 ±.2 Right mouth corner.28 ±.2 TABLE VI DETECTED LANDMARKS AFTER FILTERING B. Fiducial point localization For each o the ive iducial points, 1000 positive patches rom the Bio-id dataset and 9000 rom the PUT dataset (centered on each point) were extracted. For each training image, three negative patches rom random locations, exclusive to the positive patch area were extracted rom the same training set. For each o the cases the patch size was set to The iducial point detection model was tested on an unseen dataset o 200 Bio-id images. The euclidean distance between the model output and the actual point normalized by the eye pair distance (reerred to as normalized distance) was used as the eiciency measure, and the results are compiled in table VII. For table III(a) George W. Bush images table VI shows the iducial point outputs (white dots) o the model. The output patches, computed rom these points are shown in table IV(d). For a patch selection, the parameters were used as: {r w = W/4, r hu = W/4, r hb = W/1.75}. C. Face veriication task For the ace veriication task, we implemented the Cosine Similarity Metric Learning Model (CSML)[5]. The basic idea o the CSML model is to calculate the cosine distance (equation 4) between a given pair o aces (x, y ) by irst projecting them to a lower dimensional space through a linear map A, and using a pre-learned threshold, θ to decide whether they are a positive pair or not. CS(x, y, A) = (Ax ) T (Ay ) Ax Ay To learn A, rom n labeled examples, {x (i), y(i), l i} n i=1, where (x (i), y(i) ) is data instance with label l i {+1, 1}, CSML ormulated it as a maximization problem, arg A max (A), and the objective unction (A) is deined as equation (5). The basic idea is to push the positive and negative samples towards the direction +1 and 1 respectively, and maximize the class distance in the Cosine space. The model also added a quadratic regularizer A A 0 2 to control the over itting artiacts. (A) = i P os CS(x (i), y(i), A) α i Neg (4) CS(x (i), y(i), A) β A A 0 2 (5) Table VIII compiles the veriication results or our dierent patch selection strategies: DBMW, the selection strategy rom [9]; CSML, the selection strategy rom[5]; our-lfwa approach, which corresponds to applying our inal central patch cutting strategy on the commercially aligned LFWa images; and ours-aligned, which corresponds to using our complete processing pipeline or alignment and inal patch selection. The DBMW strategy takes LFW-unneled (or congealed) images and crops out a pixel rom the center o the image. The CSML strategy crops a pixel image rom the LFWa commercially aligned imagery. The irst two columns are or two dierent A 0 initializations { PCA, and WPCA} as used in [5], while the third column is or the inal CSML model. We used stochastic gradient ascent with a mini batch size o 20 or A learning, while the parameter θ was chosen to be at p(θ C + ) = p(θ C ), where C + and C are the positive (the same) and negative (not same) classes, modeled through two Gaussian distributions. The test results are or the LFW view-2 test-set, and we used Local Binary Patterns (LBP) as the eature deinition or a ace patch. The LBP descriptors were extracted or a non overlapping block o size 10 10, and concatenated to orm a global LBP descriptor

6 TABLE VIII VERIFICATION RESULTS FOR THREE PATCH SELECTIONS PCA WPCA CSML DBMW.622 ± ± ±.007 CSML.693 ± ± ±.005 ours-lfwa.700 ± ± ±.005 ours-aligned.688 ± ± ±.007 vector. The LBP eatures were irst reduced to a dimension o 500 by Principal Component Analysis(PCA), beore putting into the CSML ramework. From table VIII we see that our iducial point based patch selection methodology worked better than the conventional centralized patch based ace veriiers. Qualitatively the ace alignment pipeline had close perormance to the commercially available aligner. However, our approach and implementation here ailed to align 10% o the images due to 0 or 1 local point class detections. This may have caused the slightly poorer perormance o ours-aligned vs. ours-lfwa in table VIII. We hypothesize that an improvement in the local classiier and/or deining more iducial point classes may solve this problem, and thus might boost alignment pipeline. Generally, Haar-cascades output a window containing an object o interest (or example, a ace). For iducial point localizations, we hypothesized that the center o a output patch is the output point. We believe that this resulted in some localizations (or example, the mouth corners) to be more error prone than others (or example, the eyes) that have distinctive localization details. These results can be veriied rom table VII. IV. CONCLUSIONS In this paper we have presented a model to localize a consistent set o landmarks or iducial points on a ace image. The primary goal o these localizations was to then align aces to a common coordinate rame and reduce translation, rotation and scaling artiacts that impact the quality o subsequent descriptor based representations o the ace. Accordingly, ive Haar-cascades classiiers were trained and a simple probabilistic model was used to clean away alse positives and determine a similarity transormation based alignment. Based on our analysis we believe a promising direction or uture research would be to increase the number o landmarks or iducial points that could be detected in the irst phase o our ramework. However, as more landmark classes are added, the task o searching or a set o consistent landmarks becomes more time consuming. Thus a more sophisticated search strategy would also be needed. Further, or more precise spatial localization o such landmarks we believe some sort o spatially localized search within the Haar-cascade s output region could be a promising direction. Finally, we would like to improve the iltering pipeline by a more sophisticated probabilistic model. ACKNOWLEDGMENT This work was supported in part by grants rom the Natural Sciences and Engineering Research Council (NSERC) o Canada and through a Google Research award to CJP. The authors would like to thank David Rim or his ruitul suggestions and discussions. REFERENCES [1] Face recognition vendor test (rvt), 2007, [2] M. Turk and A. Pentland, Eigenaces or recognition, Journal o Cognitive Neuroscience, vol. 3, no. 1, pp , [3] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, Labeled aces in the wild: A database or studying ace recognition in unconstrained environments, University o Massachusetts, Amherst, Tech. Rep , October [4] G. B. Huang, V. Jain, and E. Learned-Miller., Unsupervised joint alignment o complex images, in International Conerence on Computer Vision (ICCV), Rio de Janeiro, Brazil, 2007, pp [5] H. V. Nguyen and L. Bai, Cosine similarity metric learning or ace veriication, in ACCV (2), 2010, pp [6] S. Prince, P. Li, Y. Fu, U. Mohammed, and J. Elder, Probabilistic models or inerence about identity, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 99, no. PrePrints, [7] L. Wol, T. Hassner, and Y. Taigman, Similarity scores based on background samples, in Proc. o ACCV, [8] Labeled aces in the wild (lw) [9] L. Wol, T. Hassner, and Y. Taigman, Descriptor based methods in the wild, in Real-Lie Images workshop at the European Conerence on Computer Vision (ECCV), Marseille, France, October [Online]. Available: [10] E. Learned-Miller, Data driven image models through continuous joint alignment, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 28, no. 2, pp , [11] A. Kasinski, A. Florek, and A. Schmidt, The put ace database, Image Processing and Communications, vol. 13, no. 3, pp , [12] A. Kasinski and A. Schmidt, The architecture and perormance o the ace and eyes detection system based on the haar cascade classiiers, Pattern Analysis and Applications, vol. 13, pp , [13] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, Labeled aces in the wild: A database or studying ace recognition in unconstrained environments, University o Massachusetts, Amherst, Tech. Rep , October [14] D. G. Lowe, Distinctive image eatures rom scale-invariant keypoints, International Journal o Computer Vision, vol. 60, no. 2, pp , [15] The bioid database, 2010,