1996 European Conference on Computer Vision. Recognition Using Class Specic Linear Projection

Transcription

1 1996 Euopean Confeence on Compute Vision Eigenfaces vs. Fishefaces: Recognition Using Class Specic Linea Pojection Pete N. Belhumeu? Jo~ao P. Hespanha?? David J. Kiegman??? Dept. of Electical Engineeing, Yale Univesity, New Haven, CT Abstact. We develop a face ecognition algoithm which is insensitive to goss vaiation in lighting diection and facial expession. Taking a patten classication appoach, we conside each pixel in an image as a coodinate in a high-dimensional space. We take advantage of the obsevation that the images of a paticula face unde vaying illumination diection lie in a 3-D linea subspace of the high dimensional featue space { if the face is a Lambetian suface without self-shadowing. Howeve, since faces ae not tuly Lambetian sufaces and do indeed poduce self-shadowing, images will deviate fom this linea subspace. Rathe than explicitly modeling this deviation, we poject the image into a subspace in a manne which discounts those egions of the face with lage deviation. Ou pojection method is based on Fishe's Linea Disciminant and poduces well sepaated classes in a low-dimensional subspace even unde sevee vaiation in lighting and facial expessions. The Eigenface technique, anothe method based on linealy pojecting the image space to a low dimensional subspace, has simila computational equiements. Yet, extensive expeimental esults demonstate that the poposed \Fisheface" method has eo ates that ae signicantly lowe than those of the Eigenface technique when tested on the same database. 1 Intoduction Within the last seveal yeas, numeous algoithms have been poposed fo face ecognition; fo detailed suveys see [4, 24]. While much pogess has been made towad ecognizing faces unde small vaiations in lighting, facial expession and pose, eliable techniques fo ecognition unde moe exteme vaiations have poven elusive. In this pape we outline a new appoach fo face ecognition { one that is insensitive to exteme vaiations in lighting and facial expessions. Note that lighting vaiability includes not only intensity, but also diection and numbe of? P. Belhumeu was suppoted by ARO Gant DAAH ?? J. Hespanha was suppoted by NSF Gant ECS , AFOSR Gant F , and ARO Gant DAAH ??? D. Kiegman was suppoted by NSF unde an NYI, IRI and by ONR N

2 Subset 1 Subset 2 Subset 3 Subset 4 Subset 5 Fig. 1. The same peson seen unde vaying lighting conditions can appea damatically dieent. These images ae taken fom the Havad database which is descibed in Section 3.1. light souces. As seen in Fig. 1, the same peson, with the same facial expession, seen fom the same viewpoint can appea damatically dieent when light souces illuminate the face fom dieent diections. Ou appoach to face ecognition exploits two obsevations: 1. Fo a Lambetian suface without self-shadowing, all of the images of a paticula face fom a xed viewpoint will lie in a 3-D linea subspace of the high-dimensional image space [25]. 2. Because of expessions, egions of self-shadowing and speculaity, the above obsevation does not exactly apply to faces. In pactice, cetain egions of the face may have a vaiability fom image to image that often deviates dastically fom the linea subspace and, consequently, ae less eliable fo ecognition. We make use of these obsevations by nding a linea pojection of the faces fom the high-dimensional image space to a signicantly lowe dimensional featue space which is insensitive both to vaiation in lighting diection and facial expession. We choose pojection diections that ae nealy othogonal to the within-class scatte, pojecting away vaiations in lighting and facial expession while maintaining disciminability. Ou method Fishefaces, a deivative of Fishe's Linea Disciminant (FLD) [9, 10], maximizes the atio of between-class scatte to that of within-class scatte.

3 The Eigenface method is also based on linealy pojecting the image space to a low dimensional featue space [27, 28, 29]. Howeve, the Eigenface method, which uses pincipal components analysis (PCA) fo dimensionality eduction, yields pojection diections that maximize the total scatte acoss all classes, i.e. all images of all faces. In choosing the pojection which maximizes total scatte, PCA etains some of the unwanted vaiations due to lighting and facial expession. As illustated in Fig. 1 and stated by Moses, Adini, and Ullman, \the vaiations between the images of the same face due to illumination and viewing diection ae almost always lage than image vaiations due to change in face identity" [21]. Thus, while the PCA pojections ae optimal fo econstuction fom a low dimensional basis, they may not be optimal fom a discimination standpoint. We should point out that Fishe's Linea Disciminant [10] is a \classical" technique in patten ecognition [9] that was developed by Robet Fishe in 1936 fo taxonomic classication. Depending upon the featues being used, it has been applied in dieent ways in compute vision and even in face ecognition. Cheng et al. pesented a method that used Fishe's disciminato fo face ecognition whee featues wee obtained by a pola quantization of the shape [6]. Contempoaneous with ou wok [15], Cui, Swets, and Weng applied Fishe's disciminato (using dieent teminology, they call it the Most Disciminating Featue { MDF) in a method fo ecognizing hand gestues [8]. Though no implementation is epoted, they also suggest that the method can be applied to face ecognition unde vaiable illumination. In the sections to follow, we will compae fou methods fo face ecognition unde vaiation in lighting and facial expession: coelation, a vaiant of the linea subspace method suggested by [25], the Eigenface method [27, 28, 29], and the Fisheface method developed hee. The compaisons ae done on a database of 500 images ceated extenally by Hallinan [13, 14] and a database of 176 images ceated at Yale. The esults of the tests on both databases shows that the Fisheface method pefoms signicantly bette than any of the othe thee methods. Yet, no claim is made about the elative pefomance of these algoithms on much lage databases. We should also point out that we have made no attempt to deal with vaiation in pose. An appeaance-based method such as ous can be easily extended to handle limited pose vaiation using eithe a multiple-view epesentation such as Pentland, Moghaddam, and Stane's View-based Eigenspace [23] o Muase and Naya's Appeaance Manifolds [22]. Othe appoaches to face ecognition that accommodate pose vaiation include [2, 11]. Futhemoe, we assume that the face has been located and aligned within the image, as thee ae numeous methods fo nding faces in scenes [5, 7, 17, 18, 19, 20, 28]. 2 Methods The poblem can be simply stated: Given a set of face images labeled with the peson's identity (the leaning set) and an unlabeled set of face images fom the same goup of people (the test set), identify the name of each peson in the test images. In this section, we examine fou patten classication techniques fo solving the face ecognition poblem, compaing methods that have become quite popula in the face ecognition liteatue, i.e. coelation [3] and Eigenface methods

4 [27, 28, 29], with altenative methods developed by the authos. We appoach this poblem within the patten classication paadigm, consideing each of the pixel values in a sample image as a coodinate in a high-dimensional space (the image space).

5 2.1 Coelation Pehaps, the simplest classication scheme is a neaest neighbo classie in the image space [3]. Unde this scheme, an image in the test set is ecognized by assigning to it the label of the closest point in the leaning set, whee distances ae measued in the image space. If all of the images have been nomalized to be zeo mean and have unit vaiance, then this pocedue is equivalent to choosing the image in the leaning set that best coelates with the test image. Because of the nomalization pocess, the esult is independent of light souce intensity and the eects of a video camea's automatic gain contol. This pocedue, which will subsequently be efeed to as coelation, has seveal well-known disadvantages. Fist, if the images in the leaning set and test set ae gatheed unde vaying lighting conditions, then the coesponding points in the image space will not be tightly clusteed. So in ode fo this method to wok eliably unde vaiations in lighting, we would need a leaning set which densely sampled the continuum of possible lighting conditions. Second, coelation is computationally expensive. Fo ecognition, we must coelate the image of the test face with each image in the leaning set; in an eot to educe the computation time, implementos [12] of the algoithm descibed in [3] developed special pupose VLSI hadwae. Thid, it equies lage amounts of stoage { the leaning set must contain numeous images of each peson. 2.2 Eigenfaces As coelation methods ae computationally expensive and equie geat amounts of stoage, it is natual to pusue dimensionality eduction schemes. A technique now commonly used fo dimensionality eduction in compute vision { paticulaly in face ecognition { is pincipal components analysis (PCA) [13, 22, 27, 28, 29]. PCA techniques, also known as Kahunen-Loeve methods, choose a dimensionality educing linea pojection that maximizes the scatte of all pojected samples. Moe fomally, let us conside a set of N sample images fx1; x2; : : : ; xng taking values in an n-dimensional featue space, and assume that each image belongs to one of c classes f1; 2; : : : ; cg. Let us also conside a linea tansfomation mapping the oiginal n-dimensional featue space into an m-dimensional featue space, whee m < n. Denoting by W 2 IR nm a matix with othonomal columns, the new featue vectos yk 2 IR m ae dened by the following linea tansfomation: yk = W T xk ; k = 1; 2; : : : ; N: Let the total scatte matix ST be dened as ST = (xk )(xk ) T k=1 whee 2 IR n is the mean image of all samples. NX Note that afte applying the linea tansfomation, the scatte of the tansfomed featue vectos fy1; y2; : : : ; yng is W T SW. In PCA, the optimal pojection Wopt is chosen to maximize the deteminant of the total scatte matix of the pojected samples, i.e. Wopt = ag max W jw T ST W j = [ w1 w2 : : : wm ] (1)

6 whee fwi j i = 1; 2; : : :; mg is the set of n-dimensional eigenvectos of ST coesponding to the set of deceasing eigenvalues. Since these eigenvectos have the same dimension as the oiginal images, they ae efeed to as Eigenpictues in [27] and Eigenfaces in [28, 29]. A dawback of this appoach is that the scatte being maximized is not only due to the between-class scatte that is useful fo classication, but also the within-class scatte that, fo classication puposes, is unwanted infomation. Recall the comment by Moses, Adini and Ullman [21]: Much of the vaiation fom one image to the next is due to illumination changes. Thus if PCA is pesented with images of faces unde vaying illumination, the pojection matix Wopt will contain pincipal components (i.e. Eigenfaces) which etain, in the pojected featue space, the vaiation due lighting. Consequently, the points in pojected space will not be well clusteed, and wose, the classes may be smeaed togethe. It has been suggested that by thowing out the st seveal pincipal components, the vaiation due to lighting is educed. The hope is that if the st pincipal components captue the vaiation due to lighting, then bette clusteing of pojected samples is achieved by ignoing them. Yet it is unlikely that the st seveal pincipal components coespond solely to vaiation in lighting; as a consequence, infomation that is useful fo discimination may be lost. 2.3 Linea Subspaces Both coelation and the Eigenface method ae expected to sue unde vaiation in lighting diection. Neithe method exploits the obsevation that fo a Lambetian suface without self-shadowing, the images of a paticula face lie in a 3-D linea subspace. Conside a point p in a Lambetian suface and a collimated light souce chaacteized by a vecto s 2 IR 3, such that the diection of s gives the diection of the light ays and ksk gives the intensity of the light souce. The iadiance at the point p is given by E(p) = a(p) < n(p); s > (2) whee n(p) is the unit inwad nomal vecto to the suface at the point p, and a(p) is the albedo of the suface at p [16]. This shows that the iadiance at the point p, and hence the gay level seen by a camea, is linea on s 2 IR 3. Theefoe, in the absence of self-shadowing, given thee images of a Lambetian suface fom the same viewpoint taken unde thee known, linealy independent light souce diections, the albedo and suface nomal can be ecoveed; this is the well known method of photometic steeo [26, 30]. Altenatively, one can econstuct the image of the suface unde an abitay lighting diection by a linea combination of the thee oiginal images, see [25]. Fo classication, this fact has geat impotance: It shows that fo a xed viewpoint, all images of a Lambetian suface lie in a 3-D linea subspace embedded in the high-dimensional image space. This obsevation suggests a simple classication algoithm to ecognize Lambetian sufaces { invaiant unde lighting conditions. Fo each face, use thee o moe images taken unde dieent lighting diections to constuct a 3-D basis fo the linea subspace. Note that the thee basis vectos have the same dimensionality as the taining images and can be thought

7 of as basis images. To pefom ecognition, we simply compute the distance of a new image to each linea subspace and choose the face coesponding to the shotest distance. We call this ecognition scheme the Linea Subspace method. We should point out that this method is a vaiant of the photometic alignment method poposed in [25] and, although it is not yet in pess, the Linea Subspace method can be thought of as special case of the moe elaboate ecognition method descibed in [14]. If thee is no noise o self-shadowing, the Linea Subspace algoithm would achieve eo fee classication unde any lighting conditions, povided the sufaces obey the Lambetian eectance model. Nevetheless, thee ae seveal compelling easons to look elsewhee. Fist, due to self-shadowing, speculaities, and facial expessions, some egions of the face have vaiability that does not agee with the linea subspace model. Given enough images of faces, we should be able to lean which egions ae good fo ecognition and which egions ae not. Second, to ecognize a test image we must measue the distance to the linea subspace fo each peson. While this in an impovement ove a coelation scheme that needs a lage numbe of images fo each class, it is still too computationally expensive. Finally, fom a stoage standpoint, the Linea Subspace algoithm must keep thee images in memoy fo evey peson. 2.4 Fishefaces The Linea Subspace algoithm takes advantage of the fact that unde ideal conditions the classes ae linealy sepaable. Yet, one can pefom dimensionality eduction using linea pojection and still peseve linea sepaability; eo fee classication unde any lighting conditions is still possible in the lowe dimensional featue space using linea decision boundaies. This is a stong agument in favo of using linea methods fo dimensionality eduction in the face ecognition poblem, at least when one seeks insensitivity to lighting conditions. Hee we ague that using class specic linea methods fo dimensionality eduction and simple classies in the educed featue space one gets bette ecognition ates in substantially less time than with the Linea Subspace method. Since the leaning set is labeled, it makes sense to use this infomation to build a moe eliable method fo educing the dimensionality of the featue space. Fishe's Linea Disciminant (FLD) [10] is an example of a class specic method, in the sense that it ties to \shape" the scatte in ode to make it moe eliable fo classication. This method selects W in such a way that the atio of the between-class scatte and the within-class scatte is maximized. Let the between-class scatte matix be dened as SB = cx jij (i )(i ) T i=1 and the within-class scatte matix be dened as SW = cx X (xk i)(xk i) T i=1 x k2 i whee i is the mean image of class i, and jij is the numbe of samples in class i. If SW is nonsingula, the optimal pojection Wopt is chosen as that which maximizes the atio of the deteminant of the between-class scatte matix of

8 PCA featue 2 0 FLD class 1 class 2 0 featue 1 Fig. 2. A compaison of pincipal component analysis (PCA) and Fishe's linea disciminant (FLD) fo a two class poblem whee data fo each class lies nea a linea subspace. the pojected samples to the deteminant of the within-class scatte matix of the pojected samples, i.e. Wopt = ag max W jw T SB W j jw T SW W j = [ w 1 w2 : : : wm ] (3) whee fwi j i = 1; 2; : : : ; mg is the set of genealized eigenvectos of SB and SW coesponding to set of deceasing genealized eigenvalues fi j i = 1; 2; : : : ; mg, i.e. SB wi = isw wi ; i = 1; 2; : : : ; m: Note that an uppe bound on m is c 1 whee c is the numbe of classes. See [9]. To illustate the benets of the class specic linea pojections, we constucted a low dimensional analogue to the classication poblem in which the samples fom each class lie nea a linea subspace. Figue 2 is a compaison of PCA and FLD fo a two-class poblem in which the samples fom each class ae andomly petubed in a diection pependicula to the linea subspace. Fo this example N = 20, n = 2, and m = 1. So the samples fom each class lie nea a line in the 2-D featue space. Both PCA and FLD have been used to poject the points fom 2-D down to 1-D. Compaing the two pojections in the gue, PCA actually smeas the classes togethe so that they ae no longe linealy sepaable in the pojected space. It is clea that although PCA achieves lage total scatte, FLD achieves geate between-class scatte, and consequently classication becomes easie. In the face ecognition poblem one is confonted with the diculty that the within-class scatte matix SW 2 IR nn is always singula. This stems fom the fact that the ank of SW is less than N c, and in geneal, the numbe of pixels in each image (n) is much lage than the numbe of images in the leaning set (N). This means that it is possible to chose the matix W such that the within-class scatte of the pojected samples can be made exactly zeo. In ode to ovecome the complication of a singula SW, we popose an altenative to the citeion in Eq. 3. This method, which we call Fishefaces, avoids

9 this poblem by pojecting the image set to a lowe dimensional space so that the esulting within-class scatte matix SW is nonsingula. This is achieved by using PCA to educe the dimension of the featue space to N c and then, applying the standad FLD dened by Eq. 3 to educe the dimension to c 1. Moe fomally, Wopt is given by whee Wopt = Wfld Wpca (4) Wpca = ag max W jw T ST W j Wfld = ag max W jw T Wpca T S B WpcaW j jw T Wpca T S W WpcaW j : Note that in computing Wpca we have thown away only the smallest c pincipal components. Thee ae cetainly othe ways of educing the within-class scatte while peseving between-class scatte. Fo example, a second method which we ae cuently investigating chooses W to maximize the between-class scatte of the pojected samples afte having st educed the within-class scatte. Taken to an exteme, we can maximize the between-class scatte of the pojected samples subject to the constaint that the within-class scatte is zeo, i.e. Wopt = ag max W2W jw T SB W j (5) whee W is the set of n m matices contained in the kenel of SW. 3 Expeimental Results In this section we will pesent and discuss each of the afoementioned face ecognition techniques using two dieent databases. Because of the specic hypotheses that we wanted to test about the elative pefomance of the consideed algoithms, many of the standad databases wee inappopiate. So we have used a database fom the Havad Robotics Laboatoy in which lighting has been systematically vaied. Secondly, we have constucted a database at Yale that includes vaiation in both facial expession and lighting Vaiation in Lighting The st expeiment was designed to test the hypothesis that unde vaiable illumination, face ecognition algoithms will pefom bette if they exploit the fact that images of a Lambetian suface lie in a linea subspace. Moe specifically, the ecognition eo ates fo all fou algoithms descibed in Section 2 will be compaed using an image database constucted by Hallinan at the Havad Robotics Laboatoy [13, 14]. In each image in this database, a subject held his/he head steady while being illuminated by a dominant light souce. The space of light souce diections, which can be paameteized by spheical angles, was then sampled in 15 incements. Fom a subset of 225 images of ve people in this database, we extacted ve subsets to quantify the eects of vaying lighting. Sample images fom each subset ae shown in Fig The Yale database is available by anonymous ftp fom daneel.eng.yale.edu.

10 Subset 1 contains 30 images fo which both of the longitudinal and latitudinal angles of light souce diection ae within 15 of the camea axis. Subset 2 contains 45 images fo which the geate of the longitudinal and latitudinal angles of light souce diection ae 30 fom the camea axis. Subset 3 contains 65 images fo which the geate of the longitudinal and latitudinal angles of light souce diection ae 45 fom the camea axis. Subset 4 contains 85 images fo which the geate of the longitudinal and latitudinal angles of light souce diection ae 60 fom the camea axis. Subset 5 contains 105 images fo which the geate of the longitudinal and latitudinal angles of light souce diection ae 75 fom the camea axis. Fo all expeiments, classication was pefomed using a neaest neighbo classie. All taining images of an individual wee pojected into the featue space. The images wee copped within the face so that the contou of the head was excluded. 5 Fo the Eigenface and coelation tests, the images wee nomalized to have zeo mean and unit vaiance, as this impoved the pefomance of these methods. Fo the Eigenface method, esults ae shown when ten pincipal components ae used. Since it has been suggested that the st thee pincipal components ae pimaily due to lighting vaiation and that ecognition ates can be impoved by eliminating them, eo ates ae also pesented using pincipal components fou though thiteen. Since thee ae 30 images in the taining set, coelation is equivalent to the Eigenface method using 29 pincipal components. We pefomed two expeiments on the Havad Database: extapolation and intepolation. In the extapolation expeiment, each method was tained on samples fom Subset 1 and then tested using samples fom Subsets 1, 2 and 3. 6 Figue 3 shows the esult fom this expeiment. In the intepolation expeiment, each method was tained on Subsets 1 and 5 and then tested the methods on Subsets 2, 3 and 4. Figue 4 shows the esult fom this expeiment. These two expeiments eveal a numbe of inteesting points: 1. All of the algoithms pefom pefectly when lighting is nealy fontal. Howeve as lighting is moved o axis, thee is a signicant pefomance dieence between the two class-specic methods and the Eigenface method. 2. It has also been noted that the Eigenface method is equivalent to coelation when the numbe of Eigenfaces equals the size of the taining set [22], and since pefomance inceases with the dimension of the Eigenspace, the Eigenface method should do no bette than coelation [3]. This is empiically demonstated as well. 3. In the Eigenface method, emoving the st thee pincipal components esults in bette pefomance unde vaiable lighting conditions. 4. While the Linea Subspace method has eo ates that ae competitive with the Fisheface method, it equies stoing moe than thee times as much infomation and takes thee times as long. 5 We have obseved that the eo ates ae educed fo all methods when the contou is included and the subject is in font of a unifom backgound. Howeve, all methods pefomed wose when the backgound vaies. 6 To test the methods with an image fom Subset 1, that image was emoved fom the taining set, i.e. we employed the \leaving-one-out" stategy [9].

11 E o a t e Eigenface (10) Eigenface (10) w/o fist 3 Coelation Linea Subspace Fisheface Eigenface (10) Coelation Eigenface (10) w/o fist (%) 0 Subset 1 Subset 2 Subset 3 Lighting Diection Subset Linea Subspace Fisheface Fig. 3. Extapolation: When each of the methods is tained on images with nea fontal illumination (Subset 1), the gaph and coesponding table show thei elative pefomance unde exteme light souce conditions E o Eigenface (10) a t e Eigenface (10) w/o fist 3 Coelation 5 (%) 0 Subset 2 Subset 3 Subset 4 Lighting Diection Subset Fisheface Linea Subspace Fig. 4. Intepolation: When each of the methods is tained on images fom both nea fontal and exteme lighting (Subsets 1 and 5), the gaph and coesponding table show the methods' elative pefomance unde intemediate lighting conditions. 5. The Fisheface method had eo ates many times lowe than the Eigenface method and equied less computation time. 3.2 Vaiation in Facial Expession, Eyewea, and Lighting Using a second database constucted at the Yale Vision and Robotics Lab, we constucted tests to detemine how the methods compaed unde a dieent ange of conditions. Fo sixteen subjects, ten images wee acquied duing one session in font of a simple backgound. Subjects included females and males (some with facial hai), and some woe glasses. Figue 5 shows ten images of one subject. The st image was taken unde ambient lighting in a neutal facial expession, and the peson woe his/he glasses when appopiate. In the second image, the glasses wee emoved if glasses wee not nomally won; othewise,

12 Fig. 5. The Yale database contains 160 fontal face images coveing sixteen individuals taken unde ten dieent conditions: A nomal image unde ambient lighting, one with o without glasses, thee images taken with dieent point light souces, and ve dieent facial expessions. 35 E o Eigenface R a t e (%) Numbe of Pincipal Components Eigenface w/o fist thee components Fisheface (7.3%) Fig. 6. As demonstated on the Yale Database, the vaiation in pefomance of the Eigenface method depends on the numbe of pincipal components etained. Dopping the st thee appeas to impove pefomance. a pai of boowed glasses wee won. Images 3-5 wee acquied by illuminating the face in a neutal expession with a Luxolamp in thee position. The last ve images wee acquied unde ambient lighting with dieent expessions (happy, sad, winking, sleepy, and supised). Fo the Eigenface and coelation tests, the images wee nomalized to have zeo mean and unit vaiance, as this impoved the pefomance of these methods. The images wee manually centeed and copped to two dieent scales: The lage images included the full face and pat of the backgound while the closely copped ones included intenal stuctues such as the bow, eyes, nose, mouth and chin but did not extend to the occluding contou. In this test, eo ates wee detemined by the \leaving-one-out" stategy [9]: To classify an image of a peson, that image was emoved fom the data set and the dimensionality eduction matix W was computed. All images in the database, excluding the test image, wee then pojected down into the educed space to be used fo classication. Recognition was pefomed using a neaest neighbo classie. Note that fo this test, each peson in the leaning set is epesented by the pojection of ten images, except fo the test peson who is epesented by only nine. In geneal, the pefomance of the Eigenface method vaies with the numbe of pincipal components. So, befoe compaing the Linea Subspace and Fishe-

13 25 E o R a t e (%) Close Cop Full Face 0 Eigenface(30) Coelation Linea Subspace Recognition Algoithm Eigenface(30) w/o fist 3 Fisheface Fig. 7. The gaph and coesponding table show the elative pefomance of the algoithms when applied to the Yale database which contains vaiation in facial expession and lighting. face methods with the Eigenface method, we st pefomed an expeiment to detemine the numbe of pincipal components that esults in the lowest eo ate. Figue 6 shows a plot of eo ate vs. the numbe of pincipal components when the initial thee pincipal components wee etained and when they wee dopped. The elative pefomance of the algoithms is self evident in Fig. 7. The Fisheface method had eo ates that wee bette than half that of any othe method. It seems that the Fisheface method leans the set of pojections which pefoms well ove a ange of lighting vaiation, facial expession vaiation, and pesence of glasses. Note that the Linea Subspace method faied compaatively wose in this expeiment than in the lighting expeiments in the pevious section. Because of vaiation in facial expession, the images no longe lie in a linea subspace. Since the Fisheface method tends to discount those potions of the image that ae not signicant fo ecognizing an individual, the esulting pojections W tend to mask the egions of the face that ae highly vaiable. Fo example, the aea aound the mouth is discounted since it vaies quite a bit fo dieent facial expessions. On the othe hand, the nose, cheeks and bow ae stable ove the within-class vaiation and ae moe signicant fo ecognition. Thus, we conjectue that Fisheface methods, which tend to educe within-class scatte fo all classes, should poduce pojection diections that ae good fo ecognizing othe faces besides the taining set. Thus, once the pojection diections ae detemined, each peson can be modeled by a single image. All of the algoithms pefomed bette on the images of the full face. Note that thee is a damatic impovement in the Fisheface method whee the eo ate was educed fom 7.3% to 0.6%. When the method is tained on the entie face, the pixels coesponding to the occluding contou of the face ae chosen as good featues fo disciminating between individuals. i.e., the oveall shape of the face is a poweful featue in face identication. As a pactical note howeve, it is expected that ecognition ates would have been much lowe fo the full face images if the backgound o haistyles had vaied and may even have been wose than the closely copped images.

14 4 Conclusion The expeiments suggest a numbe of conclusions: 1. All methods pefom well if pesented with an image in the test set which is simila to an image in the taining set. 2. The Fisheface method appeas to be the best at extapolating and intepolating ove vaiation in lighting, although the Linea Subspace method is a close second. 3. Removing the initial thee pincipal components does impove the pefomance of the Eigenface method in the pesence of lighting vaiation, but does not alleviate the poblem. 4. In the limit as moe pincipal components ae used in the Eigenface method, pefomance appoaches that of coelation. Similaly, when the st thee pincipal components have been emoved, pefomance impoves as the dimensionality of the featue space is inceased. Note howeve, that pefomance seems to level o at about 45 pincipal components. Siovitch and Kiby found a simila point of diminishing etuns when using Eigenfaces to epesent face images [27]. 5. The Fisheface method appeas to be the best at simultaneously handling vaiation in lighting and expession. As expected, the Linea Subspace method sues when confonted with vaiation in facial expession. Even with this extensive expeimentation, inteesting questions emain: How well does the Fisheface method extend to lage databases. Can vaiation in lighting conditions be accommodated if some of the individuals ae only obseved unde one lighting condition? i.e., how can infomation about the class of faces be exploited? Additionally, cuent face detection methods ae likely to beak down unde exteme lighting conditions such as Subsets 4 and 5 in Fig. 1, and so new detection methods will be needed to suppot this algoithm. Finally, when shadowing dominates, pefomance degades fo all of the pesented ecognition methods, and techniques that eithe model o mask the shadowed egions may be needed. We ae cuently investigating models fo epesenting the set of images of an object unde all possible illumination conditions; details will appea in [1]. Acknowledgements We would like to thank Pete Hallinan fo poviding the Havad Database, and Alan Yuille and David Mumfod fo many useful discussions. Refeences 1. P. Belhumeu and D. Kiegman. What is the set of images of an object unde all possible lighting conditions? In IEEE Poc. Conf. Compute Vision and Patten Recognition, D. Beyme. Face ecognition unde vaying pose. In Poc. Conf. Compute Vision and Patten Recognition, pages 756{761, R. Bunelli and T. Poggio. Face ecognition: Featues vs templates. IEEE Tans. Patten Anal. Mach. Intelligence, 15(10):1042{1053, R. Chellappa, C. Wilson, and S. Siohey. Human and machine ecognition of faces: A suvey. Poceedings of the IEEE, 83(5):705{740, Q. Chen, H. Wu, and M. Yachida. Face detection by fuzzy patten matching. In Int. Conf. on Compute Vision, pages 591{596, 1995.

15 6. Y. Cheng, K. Liu, J. Yang, Y. Zhuang, and N. Gu. Human face ecognition method based on the statistical model of small sample size. In SPIE Poc.: Intelligent Robots and Compute Vision X: Algoithms and Techn., pages 85{95, I. Caw, D. Tock, and A. Bennet. Finding face featues. In Poc. Euopean Conf. on Compute Vision, pages 92{96, Y. Cui, D. Swets, and J. Weng. Leaning-based hand sign ecognition using SHOSLIF-M. In Int. Conf. on Compute Vision, pages 631{636, R. Duda and P. Hat. Patten Classication and Scene Analysis. Wiley, New Yok, R. Fishe. The use of multiple measues in taxonomic poblems. Ann. Eugenics, 7:179{188, A. Gee and R. Cipolla. Detemining the gaze of faces in images. Image and Vision Computing, 12:639{648, J. Gilbet and W. Yang. A Real{Time Face Recognition System Using Custom VLSI Hadwae. In Poceedings of IEEE Wokshop on Compute Achitectues fo Machine Peception, pages 58{66, P. Hallinan. A low-dimensional epesentation of human faces fo abitay lighting conditions. In Poc. IEEE Conf. on Comp. Vision and Patt. Recog., pages 995{999, P. Hallinan. A Defomable Model fo Face Recognition Unde Abitay Lighting Conditions. PhD thesis, Havad Univesity, J. Hespanha, P. Belhumeu, and D. Kiegman. Eigenfaces vs. Fishefaces: Recognition using class specic linea pojection. Cente fo Systems Science 9506, Yale Univesity, PO Box , New Haven, CT 06520, May B. Hon. Compute Vision. MIT Pess, Cambidge, Mass., A. Lanitis, C. Taylo, and T. Cootes. A unied appoach to coding and intepeting face images. In Int. Conf. on Compute Vision, pages 368{373, T. Leung, M. Bul, and P. Peona. Finding faces in clutteed scenes using labeled andom gaph matching. In Int. Conf. on Compute Vision, pages 637{644, K. Matsuno, C. Lee, S. Kimua, and S. Tsuji. Automatic ecognition of human facial expessions. In Int. Conf. on Compute Vision, pages 352{359, Moghaddam and Pentland. Pobabilistic visual leaning fo object detection. In Int. Conf. on Compute Vision, pages 786{793, Y. Moses, Y. Adini, and S. Ullman. Face ecognition: The poblem of compensating fo changes in illumination diection. In Euopean Conf. on Compute Vision, pages 286{296, H. Muase and S. Naya. Visual leaning and ecognition of 3-D objects fom appeaence. Int. J. Compute Vision, 14(5{24), A. Pentland, B. Moghaddam, and Stane. View-based and modula eigenspaces fo face ecognition. In Poc. Conf. Compute Vision and Patten Recognition, pages 84{91, A. Samal and P. Iyenga. Automatic ecognition and analysis of human faces and facial expessions: A suvey. Patten Recognition, 25:65{77, A. Shashua. Geomety and Photomety in 3D Visual Recognition. PhD thesis, MIT, W. Silve. Detemining Shape and Reectance Using Multiple Images. PhD thesis, MIT, Cambidge, MA, Siovitch, L. and Kiby, M. Low-dimensional pocedue fo the chaacteization of human faces. J. Optical Soc. of Ameica A, 2:519{524, M. Tuk and A. Pentland. Eigenfaces fo ecognition. J. of Cognitive Neuoscience, 3(1), M. Tuk and A. Pentland. Face ecognition using eigenfaces. In Poc. IEEE Conf. on Comp. Vision and Patt. Recog., pages 586{591, R. Woodham. Analysing images of cuved sufaces. Aticial Intelligence, 17:117{ 140, 1981.