Computer Science Technical Report

Transcription

1 Computer Scence echncal Report NLYSIS OF PCSED ND FISHER DISCRIMINNSED IMGE RECOGNIION LGORIHMS Wendy S. Yambor July echncal Report CS3 Computer Scence Department Colorado State Unversty Fort Collns, CO Phone: (97) Fa: (97) WWW:

2 HESIS NLYSIS OF PCSED ND FISHER DISCRIMINNSED IMGE RECOGNIION LGORIHMS Submtted by Wendy S. Yambor Department of Computer Scence In Partal Fulfllment of the Requrements For the Degree of Master of Scence Colorado State Unversty Fort Collns, Colorado Summer

3 COLORDO SE UNIVERSIY July 6, WE HEREY RECOMMEND H HE HESIS PREPRED UNDER OUR SUPERVISION Y WENDY S. YMOR ENILED NLYSIS OF PCSED ND FISHER DISCRIMINNSED IMGE RECOGNIION LGORIHMS E CCEPED S FULFILLING IN PR REQUIREMENS FOR HE DEGREE OF MSER OF SCIENCE. Commttee on Graduate Work dvsor Codvsor Department Head

4 SRC OF HESIS NLYSIS OF PCSED ND FISHER DISCRIMINNSED IMGE RECOGNIION LGORIHMS One method of dentfyng mages s to measure the smlarty between mages. hs s accomplshed by usng measures such as the L norm, L norm, covarance, Mahalanobs dstance, and correlaton. hese smlarty measures can be calculated on the mages n ther orgnal space or on the mages projected nto a new space. I dscuss two alternatve spaces n whch these smlarty measures may be calculated, the subspace created by the egenvectors of the covarance matr of the tranng data and the subspace created by the Fsher bass vectors of the data. Varatons of these spaces wll be dscussed as well as the behavor of smlarty measures wthn these spaces. Eperments are presented comparng recognton rates for dfferent smlarty measures and spaces usng hand labeled magery from two domans: human face recognton and classfyng an mage as a cat or a dog. Wendy S. Yambor Computer Scence Department Colorado State Unversty Fort Collns, CO 853 Summer

5 cknowledgments I thank my commttee, Ross everdge, ruce Draper, Mcheal Krby, and dele Howe, for ther support and knowledge over the past two years. Every member of my commttee has been nvolved n some aspect of ths thess. It s through ther nterest and persuason that I ganed knowledge n ths feld. I thank Jonathon Phllps for provdng me wth the results and mages from the FERE evaluaton. Furthermore, I thank Jonathon for patently answerng numerous questons. v

6 able of Contents. Introducton. Prevous Work.. General lgorthm....3 Why Study hese Subspaces?. 3.4 Organzaton of Followng Sectons 4. Egenspace Projecton 5. Recognzng Images Usng Egenspace, utoral on Orgnal Method utoral for Snapshot Method of Egenspace Projecton Varatons Fsher Dscrmnants 5 3. Fsher Dscrmnants utoral (Orgnal Method) Fsher Dscrmnants utoral (Orthonormal ass Method) Varatons 9 4. Egenvector Selecton Orderng Egenvectors by LkeImage Dfference Smlarty & Dstance Measures re smlarty measures the same nsde and outsde of egenspace? Eperments Datasets he Cat & Dog Dataset he FERE Dataset he Restructured FERE Dataset aggng and Combnng Smlarty Measures ddng Dstance Measures Dstance Measure ggregaton Correlatng Dstance Metrcs LkeImage Dfference on the FERE dataset Cat & Dog Eperments FERE Eperments Concluson Eperment Summary Future Work.. 66 ppend I 68 References 69 v

7 . Introducton wo mage recognton systems are eamned, egenspace projecton and Fsher dscrmnants. Each of these systems eamnes mages n a subspace. he egenvectors of the covarance matr of the tranng data create the egenspace. he bass vectors calculated by Fsher dscrmnants create the Fsher dscrmnants subspace. Varatons of these subspaces are eamned. he frst varaton s the selecton of vectors used to create the subspaces. he second varaton s the measurement used to calculate the dfference between mages projected nto these subspaces. Eperments are performed to test hypotheses regardng the relatve performance of subspace and dfference measures. Nether egenspace projecton nor Fsher dscrmnants are new deas. oth have been eamned by researches for many years. It s the work of these researches that has helped to revolutonze mage recognton and brng face recognton to the pont where t s now usable n ndustry.. Prevous Work Projectng mages nto egenspace s a standard procedure for many appearancebased object recognton algorthms. basc eplanaton of egenspace projecton s provded by []. Mchael Krby was the frst to ntroduce the dea of the lowdmensonal characterzaton of faces. Eamples of hs use of egenspace projecton can be found n [7,8,6]. urk & Pentland worked wth egenspace projecton for face recognton [].

8 More recently Shree Nayar used egenspace projecton to dentfy objects usng a turntable to vew objects at dfferent angles as eplaned n []. R.. Fsher developed Fsher s lnear dscrmnant n the 93 s [5]. Not untl recently have Fsher dscrmnants been utlzed for object recognton. n eplanaton of Fsher dscrmnants can be found n [4]. Swets and Weng used Fsher dscrmnants to cluster mages for the purpose of dentfcaton n 996 [8,9,3]. elhumeur, Hespanha, and Kregman also used Fsher dscrmnants to dentfy faces, by tranng and testng wth several faces under dfferent lghtng [].. General lgorthm n mage may be vewed as a vector of pels where the value of each entry n the vector s the grayscale value of the correspondng pel. For eample, an 88 mage may be unwrapped and treated as a vector of length 64. he mage s sad to st n Ndmensonal space, where N s the number of pels (and the length of the vector). hs vector representaton of the mage s consdered to be the orgnal space of the mage. he orgnal space of an mage s just one of nfntely many spaces n whch the mage can be eamned. wo specfc subspaces are the subspace created by the egenvectors of the covarance matr of the tranng data and the bass vectors calculated by Fsher dscrmnants. he majorty of subspaces, ncludng egenspace, do not optmze dscrmnaton characterstcs. Egenspace optmzes varance among the mages. he

9 ecepton to ths statement s Fsher dscrmnants, whch does optmze dscrmnaton characterstcs. lthough some of the detals may vary, there s a basc algorthm for dentfyng mages by projectng them nto a subspace. Frst one selects a subspace on whch to project the mages. Once ths subspace s selected, all tranng mages are projected nto ths subspace. Net each test mage s projected nto ths subspace. Each test mage s compared to all the tranng mages by a smlarty or dstance measure, the tranng mage found to be most smlar or closest to the test mage s used to dentfy the test mage..3 Why Study hese Subspaces? Projectng mages nto subspaces has been studed for many years as dscussed n the prevous work secton. he research nto these subspaces has helped to revolutonze mage recognton algorthms, specfcally face recognton. When studyng these subspaces an nterestng queston arses: under what condtons does projectng an mage nto a subspace mprove performance. he answer to ths queston s not an easy one. What specfc subspace (f any at all) mproves performance depends on the specfc problem. Furthermore, varatons wthn the subspace also effect performance. For eample, the selecton of vectors to create the subspace and measures to decde whch mages are a closest match, both effect performance. 3

10 .4 Organzaton of Followng Sectons I dscuss two alternatve spaces commonly used to dentfy mages. In chapter, I dscuss egenspaces. Egenspace projecton, also know as KarhunenLoeve (KL) and Prncpal Component nalyss (PC), projects mages nto a subspace such that the frst orthogonal dmenson of ths subspace captures the greatest amount of varance among the mages and the last dmenson of ths subspace captures the least amount of varance among the mages. wo methods of creatng an egenspace are eamned, the orgnal method and a method desgned for hghresoluton mages know as the snapshot method. In chapter 3, Fsher dscrmnants s dscussed. Fsher dscrmnants project mages such that mages of the same class are close to each other whle mages of dfferent classes are far apart. wo methods of calculatng Fsher dscrmnants are eamned. One method s the orgnal method and the other method frst projects the mages nto an orthonormal bass defnng a subspace spanned by the tranng set. Once mages are projected nto one of these spaces, a smlarty measure s used to decde whch mages are closest matches. Chapter 4 dscusses varatons of these two methods, such as methods of selectng specfc egenvectors to create the subspace and smlarty measures. In chapter 5, I dscuss eperments performed on both these methods on two datasets. he frst dataset s the Cat & Dog dataset, whch was developed at Colorado State Unversty. he second dataset s the FERE dataset, whch was made avalable to me by Jonathan Phllps at the Natonal Insttute of Standard and echnology [,,3]. 4

11 . Egenspace Projecton Egenspace s calculated by dentfyng the egenvectors of the covarance matr derved from a set of tranng mages. he egenvectors correspondng to nonzero egenvalues of the covarance matr form an orthonormal bass that rotates and/or reflects the mages n the Ndmensonal space. Specfcally, each mage s stored n a vector of sze N. [ ]... () he mages are mean centered by subtractng the mean mage from each mage vector. N P m, where m P () hese vectors are combned, sdebysde, to create a data matr of sze NP (where P s the number of mages). P [... ] X (3) he data matr X s multpled by ts transpose to calculate the covarance matr. Ω XX (4) hs covarance matr has up to P egenvectors assocated wth nonzero egenvalues, assumng P<N. he egenvectors are sorted, hgh to low, accordng to ther assocated egenvalues. he egenvector assocated wth the largest egenvalue s the egenvector he bar notaton here s slghtly nonstandard, but s ntended to suggest the relatonshp to the mean. complete glossary of symbols appears n ppend I. 5

12 that fnds the greatest varance n the mages. he egenvector assocated wth the second largest egenvalue s the egenvector that fnds the second most varance n the mages. hs trend contnues untl the smallest egenvalue s assocated wth the egenvector that fnds the least varance n the mages.. Recognzng Images Usng Egenspace, utoral on Orgnal Method Identfyng mages through egenspace projecton takes three basc steps. Frst the egenspace must be created usng tranng mages. Net, the tranng mages are projected nto the egenspace. Fnally, the test mages are dentfed by projectng them nto the egenspace and comparng them to the projected tranng mages.. Create Egenspace he followng steps create an egenspace.. Center data: Each of the tranng mages must be centered. Subtractng the mean mage from each of the tranng mages centers the tranng mages as shown n equaton (). he mean mage s a column vector such that each entry s the mean of all correspondng pels of the tranng mages.. Create data matr: Once the tranng mages are centered, they are combned nto a data matr of sze NP, where P s the number of tranng mages and each column s a sngle mage as shown n equaton (3). 3. Create covarance matr: he data matr s multpled by ts transpose to create a covarance matr as shown n equaton (4). 6

13 4. Compute the egenvalues and egenvectors: he egenvalues and correspondng egenvectors are computed for the covarance matr. Ω V ΛV (5) here V s the set of egenvectors assocated wth the egenvalues Λ. 5. Order egenvectors: Order the egenvectors v V accordng to ther correspondng egenvalues λ Λ from hgh to low. Keep only the egenvectors assocated wth nonzero egenvalues. hs matr of egenvectors s the egenspace V, where each column of V s an egenvector.. Project tranng mages [ v v... ] V (6) Each of the centered tranng mages ( ) s projected nto the egenspace. o project an mage nto the egenspace, calculate the dot product of the mage wth each of the ordered egenvectors. v P ~ V (7) herefore, the dot product of the mage and the frst egenvector wll be the frst value n the new vector. he new vector of the projected mage wll contan as many values as egenvectors. 3. Identfy test mages Each test mage s frst mean centered by subtractng the mean mage, and s then projected nto the same egenspace defned by V. y p y m, where m P (8) and 7

14 ~ y V y (9) he projected test mage s compared to every projected tranng mage and the tranng mage that s found to be closest to the test mage s used to dentfy the tranng mage. he mages can be compared usng any number of smlarty measures; the most common s the L norm. I wll dscuss the dfferent smlarty measures n secton 4.3. he followng s an eample of dentfyng mages through egenspace projecton. Let the four mages n Fgure be tranng mages and let the addtonal mage n Fgure be a test mage. he four tranng mages and the mean mage are: m he centered mages are:

15 Fgure. Four tranng mages and one test mage. Combne all the centered tranng mages nto one data matr: Χ Calculate the covarance matr: Ω ΧΧ he ordered nonzero egenvectors of the covarance matr and the correspondng egenvalues are: 9

16 v v v λ 535 λ 5696 λ 3 78 he egenspace s defned by the projecton matr V he four centered tranng mages projected nto egenspace are: ~ V ~ V ~ 3 3 V ~ 4 4 V he test mage vewed as a vector and the centered test mage are:

17 y y he projected test mage s: ~ y V y he L norms are 96, 8, 58 and 449 of the test mage, 3 and y and the tranng mages 4 respectvely. y comparng the L norms, the second tranng mage s found to be closest to the test mage y, therefore the test mage belongng to the same class of mages as the second tranng mage orgnal mages, one sees mage y s most lke. y s dentfed as,. y vewng the. utoral for Snapshot Method of Egenspace Projecton he method outlned above can lead to etremely large covarance matrces. For eample, mages of sze 6464 combne to create a data matr of sze 496P and a covarance matr of sze hs s a problem because calculatng the covarance matr and the egenvectors/egenvalues of the covarance s computatonally demandng. It s known that for a NM matr the mamum number of nonzero egenvectors the matr can have s mn(n,m) [6,7,]. Snce the number of tranng

18 mages ( P ) s usually less than the number of pels ( N ), the most egenvectors/egenvalues that can be found are P. common theorem n lnear algebra states that the egenvalues of X X and X X are the same. Furthermore, the egenvectors of X X are the same as the egenvectors of X X multpled by the matr X and normalzed [6,7,]. Usng ths theorem, the Snapshot method can be used to create the egenspace from a PP matr rather than a NN covarance matr. he followng steps should be followed.. Center data: (Same as orgnal method). Create data matr: (Same as orgnal method) 3. Create covarance matr: he data matr s transpose s multpled by the data matr to create a covarance matr. Ω X X () 4. Compute the egenvalues and egenvectors of O : he egenvalues and correspondng egenvectors are computed for Ω. 5. Compute the egenvectors of egenvectors. Dvde the egenvectors by ther norm. Ω V Λ V () X X V ˆ XV () : Multply the data matr by the v vˆ vˆ (3) 6. Order egenvectors: (Same as orgnal method)

19 he followng s the same eample as used prevously, but the egenspace s calculated usng the Snapshot method. he same tranng and test mages wll be used as shown n Fgure. he revsed covarance matr s: Ω Χ Χ he ordered egenvectors and correspondng nonzero egenvalues of the revsed covarance matr are: v v v λ 535 λ 5696 λ 3 78 he data matr multpled by the egenvectors are: v ˆ v ˆ vˆ elow are the normalzed egenvectors. Note that they are the same egenvectors that were calculated usng the orgnal method. 3

20 v v v Varatons Centerng the mages by subtractng the mean mage s one common method of modfyng the orgnal mages. nother varant s to subtract the mean of each mage from all of the pel values for that mage []. hs varaton smplfes the correlaton calculaton, snce the mages are already mean subtracted. Yet another varaton s to normalze each mage by dvdng each pel value by the norm of the mage, so that the vector has a length of one []. hs varaton smplfes the covarance calculaton to a dot product. n mage cannot be both centered and normalzed, snce these actons counteract the one another. ut an mage can be centered and mean subtracted or mean subtracted and normalzed. For all my work, I use only centered mages. 4

21 3. Fsher Dscrmnants Fsher dscrmnants group mages of the same class and separates mages of dfferent classes. Images are projected from Ndmensonal space (where N s the number of pels n the mage) to C dmensonal space (where C s the number of classes of mages). For eample, consder two sets of ponts n dmensonal space that are projected onto a sngle lne (Fgure a). Dependng on the drecton of the lne, the ponts can ether be med together (Fgure b) or separated (Fgure c). Fsher dscrmnants fnd the lne that best separates the ponts. o dentfy a test mage, the projected test mage s compared to each projected tranng mage, and the test mage s dentfed as the closest tranng mage. 3. Fsher Dscrmnants utoral (Orgnal Method) s wth egenspace projecton, tranng mages are projected nto a subspace. he test mages are projected nto the same subspace and dentfed usng a smlarty measure. What dffers s how the subspace s calculated. Followng are the steps to follow to fnd the Fsher dscrmnants for a set of mages.. Calculate the wthn class scatter matr: he wthn class scatter matr measures the amount of scatter between tems n the same class. For the th class, a scatter matr ( S ) s calculated as the sum of the covarance matrces of the centered mages n that class. 5

22 Fgure. (a) Ponts n dmensonal space. (b) Ponts med when projected onto a lne. (c) Ponts separated when projected onto a lne. where S ( m )( m ) (4) X m s the mean of the mages n the class. he wthn class scatter matr ( S W ) s the sum of all the scatter matrces. C S W S (5) where C s the number of classes.. Calculate the between class scatter matr: he between class scatter matr ( S ) measures the amount of scatter between classes. It s calculated as the sum of the covarance matrces of the dfference between the total mean and the mean of each class. C S n ( m m)( m m) (6) where n s the number of mages n the class, class and m s the mean of all the mages. m s the mean of the mages n the 6

23 3. Solve the generalzed egenvalue problem: Solve for the generalzed egenvectors (V ) and egenvalues ( Λ ) of the wthn class and between class scatter matrces. S V ΛS V (7) W 4. Keep frst Cl egenvectors: Sort the egenvectors by ther assocated egenvalues from hgh to low and keep the frst C egenvectors. hese egenvectors form the Fsher bass vectors. 5. Project mages onto Fsher bass vectors: Project all the orgnal (.e. not centered) mages onto the Fsher bass vectors by calculatng the dot product of the mage wth each of the Fsher bass vectors. he orgnal mages are projected onto ths lne because these are the ponts that the lne has been created to dscrmnate, not the centered mages. Followng s an eample of calculatng the Fsher dscrmnants for a set of mages. Let the twelve mages n Fgure 3 be tranng mages. here are two classes; mages 6 are n the frst class and mages 7 are n the second class. he tranng mages vewed as vectors are:

24 8 Fgure 3. welve tranng mages he scatter matrces are: S

25 S he wthn class scatter matr s: S S S W he mean of each class and the total mean are: m m m

26 he between class scatter matr s: S Snce there are two classes, only one egenvector s kept. he nonzero egenvector and correspondng egenvalue of S V λs V are: W v λ 9.45 he values of the mages projected onto the frst egenvector are shown n able. Fgure 4 shows a plot of the ponts; clearly llustratng the separaton between the two classes. 3. Fsher Dscrmnants utoral (Orthonormal ass Method) wo problems arse when usng Fsher dscrmnants. Frst, the matrces needed for computaton are very large, causng slow computaton tme and possble problems wth

27 able. he values of the mages projected onto the frst egenvector Class Class Fgure 4. Plot of the mages projected onto Fsher bass vectors. numerc precson. Second, snce there are fewer tranng mages than pels, the data matr s rank defcent. It s possble to solve the egenvectors and egenvalues of a rank defcent matr by usng a generalze sngular value decomposton routne, but a smpler soluton ests. smpler soluton s to project the data matr of tranng mages nto an orthonormal bass of sze PP (where P s the number of tranng mages). hs projecton produces a data matr of full rank that s much smaller and therefore decreases computaton tme. he projecton also preserves nformaton so the fnal outcome of Fsher dscrmnants s not affected. Followng are the steps to follow to fnd the Fsher dscrmnants of a set of mages by frst projectng the mages nto any orthonormal bass.. Compute means: Compute the mean of the mages n each class( m )and the total mean of all mages ( m ).. Center the mages n each class: Subtract the mean of each class from the mages n that class. X, X X, ˆ m (8)

28 3. Center the class means: Subtract the total mean from the class means. mˆ m m (9) 4. Create a data matr: Combne the all mages, sdebysde, nto one data matr. 5. Fnd an orthonormal bass for ths data matr: hs can be accomplshed by usng a QR Orthogonaltrangular decomposton or by calculatng the full set of egenvectors of the covarance matr of the tranng data. Let the orthonormal bass be U. 6. Project all centered mages nto the orthonormal bass: Create vectors that are the dot product of the mage and the vectors n the orthonormal bass. ~ U ˆ () 7. Project the centered means nto the orthonormal bass: m ~ U mˆ () 8. Calculate the wthn class scatter matr: he wthn class scatter matr measures the amount of scatter between tems wthn the same class. For the th class a scatter matr ( S ) s calculated as the sum of the covarance matrces of the projected centered mages for that class. S ~ ~ () X he wthn class scatter matr ( S W ) s the sum of all the scatter matrces. C S W S (3) where C s the number of classes.

29 9. Calculate the between class scatter matr: he between class scatter matr ( S ) measures the amount scatter between classes. It s calculated as the sum of the covarance matrces of the projected centered means of the classes, weghted by the number of mages n each class. S C n m ~ m ~ (4) where n s the number of mages n the class.. Solve the generalzed egenvalue problem: Solve for the generalzed egenvectors (V ) and egenvalues ( Λ ) of the wthn class and between class scatter matrces. S V λs V (5) W. Keep the frst Cl egenvectors: Sort the egenvectors by ther assocated egenvalues from hgh to low and keep the frst C egenvectors. hese are the Fsher bass vectors.. Project mages onto egenvectors: Project all the rotated orgnal (.e. Not centered) mages onto the Fsher bass vectors. Frst project the orgnal mages nto the orthonormal bass, and then project these projected mages onto the Fsher bass vectors. he orgnal rotated mages are projected onto ths lne because these are the ponts that the lne has been created to dscrmnate, not the centered mages. he same eample as before wll be calculated usng the orthonormal bass. Let the twelve mages n Fgure 3 be tranng mages. he tranng mages vewed as vectors, the means of each class and the total mean are the same as n the prevous eample. 3

30 4 he centered mages are: ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ he centered class means are: ˆ ˆ m m

31 5 he orthonormal bass calculated by egenspace projecton s: U he centered mages projected nto the orthonormal bass are: ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ 9 8 7

32 6 he centered means projected nto the orthonormal bass are: ~ ~ m m he wthn class scatter matr s: S W Notce that the wthn class scatter matr s a dagonal matr and the values along the dagonal are the egenvalues assocated wth the egenvectors used to create the orthonormal bass. hs occurs because the mages are projected nto ths orthonormal bass before calculatng the wthn class scatter matr. herefore each projected mage s orthogonal to all other projected mages. he between class scatter matr s:

33 S Snce there are two classes, only one egenvector s kept. he nonzero egenvector and correspondng egenvalue of S V λs V are: W v λ he values of the rotated mages projected onto the frst egenvector are shown n able. Fgure 5 shows a plot of the ponts; you can clearly see the separaton between the two classes. 7

34 able. he values of the mages projected onto the frst egenvector Class Class Fgure 5. Plot of the mages projected onto Fsher bass vectors. 8

35 4. Varatons 4. Egenvector Selecton Untl ths pont, when creatng a subspace usng egenspace projecton we use all egenvectors assocated wth nonzero egenvalues. he computaton tme of egenspace projecton s drectly proportonal to the number of egenvectors used to create the egenspace. herefore by removng some porton of the egenvectors computaton tme s decrease. Furthermore, by removng addtonal egenvectors that do not contrbute to the classfcaton of the mage, performance can be mproved. Many varatons of egenvector selecton have been consdered; I wll dscuss fve. hese may be appled ether alone or as part of Fsher dscrmnants.. Standard egenspace projecton: ll egenvectors correspondng to nonzero egenvalues are used to create the subspace.. Remove the last 4% of the egenvectors: Snce the egenvectors are sorted by the correspondng descendng egenvalues, ths method removes the egenvectors that fnd the least amount of varance among the mages. Specfcally, 4% of the egenvectors that fnd the least amount of varance are removed []. 3. Energy dmenson: Rather than use a standard cutoff for all subspaces, ths method uses the mnmum number of egenvectors to guarantee that energy (e) s greater than 9

36 a threshold. typcal threshold s.9. he energy of the th egenvector s the rato of the sum of the frst egenvalues over the sum of all the egenvalues [7] e j k j λ λ j j (3) 4. Stretchng dmenson: nother method of selectng egenvectors based on the nformaton provded by the egenvalues s to calculate the stretch (s) of an egenvector. he stretch of the th egenvector s the rato of the th egenvalue ( λ ) over the mamum egenvalue ( λ ) [7]. common threshold for the stretchng dmenson s.. λ s (3) λ 5. Removng the frst egenvector: he prevous three methods assume that the nformaton n the last egenvectors work aganst classfcaton. hs method assumes that nformaton n the frst egenvector works aganst classfcaton. For eample, lghtng causes consderable varaton n otherwse dentcal mages. Hence, ths method removes the frst egenvector []. Fgure 6 shows the values for energy and stretchng on the FERE dataset. 4. Orderng Egenvectors by LkeImage Dfference Ideally, two mages of the same person should project to the same pont n egenspace. ny dfference between the ponts s unwanted varaton. On the other hand, two mages of dfferent subjects should project to ponts that are as wdely separated as possble. o 3

37 Images Correctly Classfed e67.7% e55.4% e73.79% s.964 e4.76% s.3 e8.4% e77.64% s.555 s.738 e85.% e88.5% e9.9% e97.5% e.% e94.56% e99.3% s.933 s.46 s.6 s.35 s.7 s.35 s.3 s. s Number of Egenvectors Fgure 6. Eample of Energy (e) and Stretchng (s) dmenson of a specfc dataset. capture ths ntuton and use t to order egenvectors, we defne a lkemage dfference (ω ) for each of the k egenvectors []. o defne ω, we wll work wth pars of mages of the same people projected nto egenspace. Let X be tranng mages and Y mages of the correspondng people n the test set ordered such that as follows: j X and y j Y are mages of the same person. Defne ω δ k ω where λ j δ j y j (8) 3

38 When a dfference between mages that ought to match s large relatve to the varance for the dmenson λ then ω s large. Conversely, when the dfference between mages that ought to match s small relatve to the varance, ω s small. Snce the goal s to select egenvectors that brng smlar mages close to each other, we rank the egenvectors n order of ascendng ω and remove some number of the last egenvectors. 4.3 Smlarty & Dstance Measures Once mages are projected nto a subspace, there s the task of determnng whch mages are most lke one another. here are two ways n general to determne how alke mages are. One s to measure the dstance between the mages n Ndmensonal space. he second way s to measure how smlar two mages are. When measurng dstance, one wshes to mnmze dstance, so two mages that are alke produce a small dstance. When measurng smlarty, one wshes to mamze smlarty, so that two lke mages produce a hgh smlarty value. here are many possble smlarty and dstance measures; I wll dscuss fve. L norm: he L norm s also known as the cty block norm or the sum norm. It sums up the absolute dfference between pels[6,]. he L norm of an mage and an mage s: N L (, ) (9) he L norm s a dstance measure. Fgure 7 shows the L dstance between two vectors. 3

39 L norm: he L norm s also known as the Eucldean norm or the Eucldean dstance when ts square root s calculated. It sums up the squared dfference between pels [6,,7]. he L norm of an mage and an mage s: N (, ) ( ) L (3) he L norm s a dstance measure. Fgure 7 shows the L dstance between two vectors. Covarance: Covarance s also known as the angle measure. It calculates the angle between two normalzed vectors. akng the dot product of the normalzed vectors performs ths calculaton [,7]. he covarance between mages and s: cov(, ) (3) Covarance s a smlarty measure. y negatng the covarance value, t becomes a dstance measure []. Fgure 7 shows the covarance between two vectors. Mahalanobs dstance: he Mahalanobs dstance calculates the product of the pels and the egenvalue of a specfc dmenson and sums all these products []. he Mahalanobs dstance between an mage and an mage s: N Mah(, ) C (3) 33

40 Fgure 7. L dstance, L dstance and covarance between two vectors Fgure 8. wo mages wth a negatve correlaton and two that correlate well where C (33) λ Mahalanobs dstance s a dstance measure. Correlaton: Correlaton measures the rate of change between the pels of two mages. It produces a value rangng from to, where a value of ndcates the mages are oppostes of each other and a value of ndcates that the mages are dentcal [7]. he correlaton between an mage and an mage s: corr(, ) N ( µ )( σ σ µ ) (34) where µ s the mean of and σ s the standard devaton of. Fgure 8 shows an eample of two mages wth a negatve correlaton and two that correlate well. 34

41 4.4 re Smlarty Measures the Same Insde and Outsde of Egenspace? n egenspace consstng of all egenvectors assocated wth nonzero egenvalues s an orthonormal bass. n orthonormal bass s a set of vectors where the dot product of any two dstnct vectors s zero and the length of every vector s one. Orthonormal bases have the property that any mage that was used to create the orthonormal bass can be projected nto the full orthonormal bass wth no loss of nformaton. hs means that the mage can be projected nto the orthonormal bass and then converted back to the orgnal mage. For eample, let U be an orthonormal bass and let be an mage used to create U. hen U, where s the mage projected nto U. can be recovered by multplyng by U, U. Gven the fact that no nformaton s lost when projectng specfc mages nto an orthonormal bass, do the values of the smlarty measures change? he answer s that t depends on the smlarty measure. he L norm and correlaton produce dfferent values n the two spaces. Mahalanobs dstance s typcally only used n conjuncton wth egenspace. he L norm and covarance do produce the same value n both spaces; I wll prove ths. heorem 4.: he L norm produces the same value on a par of unprojected vectors and on a par of projected vectors. L (, ) L ( U, U ) (35) 35

42 36 Proof: Let U be an orthonormal bass. Let be a vector such that U. Let be a vector such that U. Now the L norm of s defned n equaton (3) and s the same as: ) ( ) ( (36) he L norm of ) ( s defned as: ), ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ), ( L UU UU UU UU U U U U U U U U L N N Hence, the L norm produces the same value on unprojected vectors and on projected vectors. heorem 4.: Covarance produces the same value on a par of unprojected vectors and on a par of projected vectors. ), cov( ), cov( U U (37) Proof: Let U be an orthonormal bass. Let be a vector such that U and U. Let be a vector such that U and U. he covarance of and s defned n equaton (3) and the covarance of and s defned as: ( ) ( ) U U U U U U U U U ), cov( (38) It s known that U, so

43 cov(, ) (39) U U y theorem 4. U and U. So, cov(, ) cov(, ) (4) Hence, covarance produces the same value on unprojected vectors and on projected vectors. I wll llustrate how each measure behaves wth an eample. Consder two vectors, 7 5, 5. Project these two ponts nto the orthonormal bass U U U Fgure 9 shows ponts,, and. L norm: he L norm produces the same value on unprojected vectors and on projected vectors. Eamne the eample. 37

44 Fgure 9. Plot of ponts,, and. he L norm of L ( ) he L norm of L ( ) s: ( ) ( 7) + ( 5 5) + ( 4 ) s: ( ) ( ) + ( (.44)) + ((.4497) (.3) ) hs eample shows that for ths specfc case they are the same, and the above proof covers the general case. 38

45 Covarance: Covarance produces the same value on unprojected vectors and on projected vectors. Eamne the same eample. he covarance between and s: cov(, ) [ ] he covarance between and s: cov(, ) [ ] hs eample shows that for ths specfc case they are the same and the above proof covers the general case. L norm: he L norm does not produce the same value on unprojected vectors and on projected vectors. Intutvely, two ponts that are located dagonal to each other wll 39

46 produce a larger L dstance than the same ponts rotated to be horzontal to each other. hs can be proven by an eample. he L norm of s: L ( ) he L norm of s: L ( ) (.44) (.4497) (.3) s you can see, these two norms are not the same; hence the L norm does not produce the same value on unprojected vectors and on projected vectors. Correlaton: Correlaton does not produce the same value on unprojected vectors and on projected vectors. Intutvely, Correlaton s effected by the mean of the mages. s mages are rotated ther mean changes, therefore ther correlaton changes. hs can be proven by an eample. he correlaton between and s: µ , µ , σ.87,.566 σ 4

47 N ( )( (, ) µ µ corr σ σ ( )( ) ( )( ) ( )( ) * *.566 (.3333) * * * * he correlaton between and s: ) µ.8643, µ.3675, σ.9485, σ.87 * N ( )( (, ) µ µ corr σ σ ( )( ) ( )( ).9485*5.53 ( )( ) * * *.9485* ) ( 3.775) + ( 3.34).9485*5.53 *.5779 hese two correlatons are not the same; hence the correlaton does not produce the same value on an unprojected vector and on a projected vector. Mahalanobs dstance: ypcally the Mahalanobs dstance s calculated on vectors projected nto egenspace, snce t uses the egenvalues to weght the contrbuton along each as. o llustrate Mahalanobs dstance as measured n egenspace, I wll calculate the Mahalanobs dstance between the projected vectors and. C λ

48 C λ C Mah(, ) C N λ (.668) (5.973*8.769 *.88) + (3.8453*.44*.976) + (.4497*.3*.93) 4

49 5. Eperments I performed four eperments, each on one of two datasets. he frst eperment combnes smlarty measures n an attempt to mprove performance. he second eperment tests whether performance mproves when egenvectors are ordered by lkemage dfference n egenspace projecton. he thrd eperment compares many varatons of egenspace projecton and Fsher dscrmnants on the Cat & Dog dataset. he fourth eperment compares several varatons of egenspace projecton and Fsher dscrmnants on the FERE dataset. 5. Datasets he Cat & Dog dataset and the FERE dataset are used for the eperments. he FERE dataset s restructured for some of the eperments. 5.. he Cat & Dog Dataset he Cat & Dog dataset was created by Mark Stevens and s avalable through the Computer Vson group at Colorado State Unversty. he orgnal dataset of, 6464 pel, grayscale mages conssted of 5 cat mages and 5 golden retrever mages. I collected 5 addtonal cat mages and 5 addtonal mages of dfferent types of dogs. Each mage shows the anmal s head and s taken drectly facng the anmal s face. Fgure shows eample mages from the Cat & Dog dataset. hs s a class classfcaton problem, snce each of the test mages s ether a cat or a dog. 43

50 Fgure. Sample mages from the Cat & Dog dataset Fgure. Sample mages from the FERE dataset 5.. he FERE Dataset Jonathan Phllps at the Natonal Insttute of Standards and echnology made the FERE dataset avalable to our department [,,3]. he FERE database contans mages of 96 ndvduals, wth up to 5 dfferent mages captured for each ndvdual. he mages are separated nto two sets: gallery mages and probes mages. Gallery mages are mages wth known labels, whle probe mages are matched to gallery mages for dentfcaton. he database s broken nto four categores: F: wo mages were taken of an ndvdual, one after the other. One mage s of the ndvdual wth a neutral facal epresson, whle the other s of the ndvdual wth a dfferent epresson. One of the mages s placed nto the gallery fle whle the other s used as a probe. In ths category, the gallery contans 96 mages, and the probe set has 95 mages. 44

51 Duplcate I: he only restrcton of ths category s that the gallery and probe mages are dfferent. he mages could have been taken on the same day or ½ years apart. In ths category, the gallery conssts of the same 96 mages as the F gallery whle the probe set contans 7 mages. fc: Images n the probe set are taken wth a dfferent camera and under dfferent lghtng than the mages n the gallery set. he gallery contans the same 96 mages as the F & Duplcate I galleres, whle the probe set contans 94 mages. Duplcate II: Images n the probe set were taken at least year after the mages n the gallery. he gallery contans 864 mages, whle the probe set has 34 mages. Fgure shows eample mages from the FERE dataset he Restructured FERE Dataset I restructured a porton of the FERE dataset so that there are four mages for each of 6 ndvduals. wo of the pctures are taken on the same day, where one pcture s of the ndvdual wth a neutral facal epresson and the other s wth a dfferent epresson. he other two pctures are taken on a dfferent day wth the same characterstcs. he purpose of ths restructurng s to create a dataset wth more than one tranng mage of each ndvdual to allow testng of Fsher dscrmnants. 5. aggng and Combnng Smlarty Measures Dfferent smlarty measures have been dscussed, but up untl ths pont they have been eamned separately. I wll now eamne combnng some of the smlarty measures 45

52 together n the hopes of mprovng performance. he followng four smlarty measures are eamned: L norm (9), the L norm (3), covarance (3), and the Mahalanobs dstance (3). I test both smple combnatons of the dstance measures and baggng the results of two or more measures usng a votng scheme [,3,9]. 5.. ddng Dstance Measures smple way to combne dstance measures s to add them. In other words, the dstance between two mages s defned as the sum S of the dstances accordng to two or more tradtonal measures: S ( a,..., ah ) a a h (4) Usng S, all combnatons of base metrcs ( L, L, covarance, Mahalonobs) are used to select the nearest gallery mage to each probe mage. he percentage of mages correctly recognzed usng each combnaton on the Duplcate I probe set, s shown n able 5, along wth the recognton rates for the base measures themselves. Of the four base measures, there appears to be a sgnfcant mprovement wth the Mahalanobs dstance. On the surface, 4% seems much better than 33%, 34% or 35%. he best performance of any combned measure s 43% for the S( L, Mahalanobs) and S( L,covarance, Mahalanobs) combnatons. Whle hgher, the dfference does not appear sgnfcant. I used McNemar s test, whch smplfes to the sgn test [,4], to calculate the sgnfcance of dfferences n these results. he McNemar s test calculates how often one algorthm succeeds whle the other algorthm fals. I formulated the followng hypotheses to test sgnfcant dfference n the prevous results. 46

53 able 3. Results of McNemar s test among base measures. lgorthms lgorthm Success/Success Success/Falure Falure/Success P< L L L ngle L Mahalanobs L ngle L Mahalanobs ngle Mahalanobs 5 8. able 4. Results of McNemar s test of the S( L, Mahalanobs) and S( L,covarance, Mahalanobs) combnatons compared to the Mahalanobs dstance lgorthms lgorthm Success/Success Success/Falure Falure/Success P< Mahalanobs S(L +Mahalanobs) Mahalanobs S (L, ng, Mah) Of the four base measures, Mahalanobs dstance outperforms all others, 4% versus 33%, 34% or 35%.. he performance of any combned measures s not statstcally better than the performance of the base measures. 43% for the S( L, Mahalanobs) and S( L,covarance, Mahalanobs) combnatons versus 4% for Mahalanobs dstance. able 3 shows the results of McNemar s test performed for each par of base measures. Note that Mahalanobs dstance always fals less often than the other smlarty measures, ndcated by P <.. Yet, no other measure s found to be dfferent. able 4 shows the results of McNemar s test of the S( L, Mahalanobs) and S( L,covarance, Mahalanobs) combnatons compared to the Mahalanobs dstance. Here no sgnfcant dfference s found between these algorthms, ndcatng that when they do dffer on a partcular mage each s equally lkely to dentfy the mage correctly. 47

54 able 5. Results of addng smlarty measures Classfer Duplcate I L.35 L.33 Covarance.34 Mahalanobs.4 S (L, L ).35 S (L, Covarance).39 S (L, Mahalanobs).43 able 6. Results of baggng smlarty measures Classfer Dup I F L L.33.7 Cov.34.7 Mah.4.74 aggng aggng (best 5) aggng (Weghted) S (L, Covarance).33 S (L, Mahalanobs).4 S (ngle, Covarance).4 S (L, L, Covarance).35 S (L, L, Mahalanobs).4 S (L, Cov, Mah).43 S (L, Cov, Mah).4 S (L, L,Cov, Mah).4 Interestngly, the performance of the combned measures s never less than the performance of ther components evaluated separately. For eample, the performance of S( L, L ) s 35%; ths s better than the performance of L (33%) and the same as L (35%). hese results suggest that L and L are dentfyng the same mages correctly; hence combnng measures does not dentfy any addtonal mages correctly. 5.. Dstance Measure ggregaton he eperment above tested only a smple summaton of dstance measures; one can magne many weghtng schemes for combnng dstance measures that mght outperform smple summaton. Rather than search the space of possble dstance measure combnatons, however, I took a cue from recent work n machne learnng that suggests the best way to combne multple estmators s to apply each estmator ndependently and combne the results by votng [,3,9]. 48

55 For face recognton, ths mples that each dstance measure s allowed to vote for the mage that t beleves s the closest match for a probe. he mage wth the most votes s chosen as the matchng gallery mage. Votng s performed three dfferent ways. aggng: Each classfer s gven one vote as eplaned above. aggng, best of 5: Each classfer votes for the fve gallery mages that most closely match the probe mage. aggng, weghted: Classfers cast fve votes for the closest gallery mage, four votes for the second closest gallery mage, and so on, castng just one vote for the ffth closest mage. able 6 shows the performance of votng for the Duplcate I and F probe sets. On the Duplcate I data, Mahalanobs dstance alone does better than any of the bagged classfers: 4% versus 37% and 38%. On the smpler F probe set, the best performance for a separate classfer s 77% (for L ), and the best performance for the bagged classfers s 78%. he McNemar s test confrms that ths s not a sgnfcant mprovement. In the net secton, I eplore one possble eplanaton for ths lack of mprovement when usng baggng Correlatng Dstance Metrcs s descrbed n [], the falure of votng to mprove performance suggests that the four dstance measures share the same bas. o test ths theory, I correlate the dstances calculated by the four measures over the Duplcate I probe set. Snce each measure s 49

56 defned over a dfferent range, Spearman rank correlaton s used [4]. For each probe mage, the gallery mages are ranked by ncreasng dstance to the probe. hs s done for each par of dstance measures. he result s two rank vectors, one for each dstance measure. Spearman s Rank Correlaton s the correlaton coeffcent for these two vectors. able 7 shows the average correlaton scores. L, covarance and Mahalanobs all correlate very closely to each other, although L correlates less well to covarance and Mahalanobs. hs suggests that there mght be some advantage to combnng L wth covarance or Mahalanobs, but that no combnaton of L, covarance or Mahalanobs s very promsng. hs s consstent wth the scores n able 5, whch show that the combnatons S( L, covarance) and S( L, Mahalanobs) outperform these classfers ndvdually. I also constructed a lst of mages n the F probe set that were grossly msclassfed, n the sense that the matchng gallery mage s not one of the ten closest mages accordng to one or more dstance measures. total of 79 mages are poorly dentfed by at least one dstance measure. able 8 shows the number of mages that are poorly dentfed by all four dstance measures, three dstance measures, two dstance measures, and just one dstance measure. hs table shows that there s shared bas among the classfers, n that they seem to make gross mstakes on the same mages. On the other hand, the errors do not overlap 5

57 able 7. Correlaton between smlarty measures. L L cov Mah L L cov Mah able 8. Number of mages smlarly dentfed poorly Images commonly poorly dentfed # of mages out of 79 % mages by 4 classfers by 3 classfers by classfers by classfer completely, suggestng that some mprovement mght stll be acheved by some combnaton of these dstance measures. 5.3 LkeImage Dfference on the FERE dataset In order to test the performance of lkemage orderng of egenvectors compared to orderng by correspondng egenvalue, I performed two eperments. he frst eperment s on the orgnal FERE dataset and compares results to those of Moon & Phllps. he second eperment s on the restructured dataset, where trals are run and tranng/test data s clearly separated. For each of the 95 probe/gallery matches n the F probe set of the orgnal FERE dataset, I calculate the dfference between the probe and gallery mage n egenspace. hese dfferences are summed together and then dvded by the egenvalue to calculate the lkemage dfference. he smaller ths number s, the better the egenvector should be at matchng mages. he top N egenvalues are selected accordng to the lkemage dfference measure, and the F probe set s reevaluated usng the L norm. Fgure shows the performance scores of the reordered egenvalues compared to the performance of the egenvalues ordered by egenvalue, as performed by Moon & Phllps. able 9 shows the number of mages correctly dentfed by each orderng method and the results 5