SEI 007 4 rth Internatonal Conference: Scences of Electronc, echnologes of Informaton and elecommuncatons March 5-9, 007 UNISIA Feature Selecton By KDDA For SVM-Based MultVew Face ecognton Seyyed Majd Valollahzadeh, Abolghasem Sayadyan, Mohammad Nazar Electrcal Engneerng Department, Amrkabr Unversty of echnology, ehran, Iran, 594 valollahzadeh@yahoo.com eea35@aut.ac.r mohnazar@aut.ac.r Abstract: Applcatons such as Face ecognton (F that deal wth hgh-dmensonal data need a mappng technque that ntroduces representaton of low-dmensonal features wth enhanced dscrmnatory power and a proper classfer, able to classfy those complex features.most of tradtonal Lnear Dscrmnant Analyss (LDA suffer from the dsadvantage that ther optmalty crtera are not drectly related to the classfcaton ablty of the obtaned feature representaton. Moreover, ther classfcaton accuracy s affected by the small sample sze (SSS problem whch s often encountered n F tasks. In ths short paper, we combne nonlnear kernel based mappng of data called KDDA wth Support Vector machne (SVM classfer to deal wth both of the shortcomngs n an effcent and cost effectve manner. he proposed here method s compared, n terms of classfcaton accuracy, to other commonly used F methods on UMIS face database. esults ndcate that the performance of the proposed method s overall superor to those of tradtonal F approaches, such as the Egenfaces, Fsherfaces, and D-LDA methods and tradtonal lnear classfers. Keywords: Face ecognton, Kernel Drect Dscrmnant Analyss (KDDA, small sample sze problem (SSS, Support Vector Machne (SVM. INODUCION Selectng approprate features to represent faces and proper classfcaton of these features are two central ssues to face recognton (F systems. For feature selecton, successful solutons seem to be appearance-based approaches, (see [3], [] for a survey, whch drectly operate on mages or appearances of face objects and process the mages as two-dmensonal (-D holstc patterns, to avod dffcultes assocated wth hree-dmensonal (3-D modellng, and shape or landmark detecton []. For the purpose of data reducton and feature extracton n the appearance-based approaches, Prncple component analyss (PCA and lnear dscrmnant analyss (LDA are ntroduced as two powerful tools. Egenfaces [4] and Fsherfaces [5] bult on the two technques, respectvely, are two state-of-the-art F methods, proved to be very successful. It s generally beleved that, LDA based algorthms outperform PCA based ones n solvng problems of pattern classfcaton, snce the former optmzes the lowdmensonal representaton of the objects wth focus on the most dscrmnant feature extracton whle the latter acheves smply object reconstructon. However, many LDA based algorthms suffer from the so-called small sample sze problem (SSS whch exsts n hgh-dmensonal pattern recognton tasks where the number of avalable samples s smaller than the dmensonalty of the samples. he tradtonal soluton to the SSS problem s to utlze PCA concepts n conjuncton wth LDA (PCA+LDA as t was done for example n Fsherfaces []. ecently, more effectve solutons, called Drect LDA (D-LDA methods, have been presented [], [3]. Although successful n many cases, lnear methods fal to delver good performance when face patterns are subject to large varatons n vewponts, whch results n a hghly non-convex and complex dstrbuton. he lmted success of these methods should be attrbuted to ther lnear nature [4]. Kernel dscrmnant analyss algorthm, (KDDA generalzes the strengths of the recently presented D-LDA [] and the kernel technques whle at the same tme overcomes many of ther shortcomngs and lmtatons. In ths work, we frst nonlnearly map the orgnal nput space to an mplct hgh-dmensonal feature space, where the dstrbuton of face patterns s hoped - -
SEI007 to be lnearzed and smplfed. hen, KDDA method s ntroduced to effectvely solve the SSS problem and derve a set of optmal dscrmnant bass vectors n the feature space. And then SVM approach s used for classfcaton. he rest of the paper s organzed as follows. In Secton tow, we start the analyss by brefly revewng KDDA method. Followng that n secton three, SVM s ntroduced and analyzed as a powerful classfer. In Secton four, a set of experments are presented to demonstrate the effectveness of the KDDA algorthm together wth SVM classfer on hghly nonlnear, hghly complex face pattern dstrbutons. he proposed method s compared, n terms of the classfcaton error rate performance, to KPCA (kernel based PCA, GDA (Generalzed Dscrmnant Analyss and KDDA algorthm wth nearest neghbour classfer on the mult-vew UMIS face database. Conclusons are summarzed n Secton fve. Kernel Drect Dscrm-nant Analyss (KDDA. Lnear Dscrmnant Analyss In the statstcal pattern recognton tasks, the problem of feature extracton can be stated as follows: L Z s Assume that we have a tranng set, { } avalable. Each mage s defned as a vector of ( length N I I face mage sze and space []. w h N,.e. Z where N w I h s the denotes a N-dmensonal real It s further assumed that each mage belongs to C Z one of C classes{ }. he objectve s to fnd a transformatonϕ, based on optmzaton of certan separablty crtera, whch produces a mappng, wth N y that leads to an enhanced separablty of dfferent face objects. Let SBW and SWH be the between- and wthnclass scatter matrces n the feature space F respectvely, expressed as follows: S S C BW C L WH L ( φ φ ( φ φ C C ( j φ ( φj φ j I ( φ ( Where φ φ( Z, φ s the mean of class Zj j j andφ s the average of the ensemble. φ C C j φ L j ( Z j ( Z (3 C C φ φ (4 he maxmzaton can be acheved by solvng the j followng egenvalue problem: Φ S BW Φ Φ arg max (5 Φ Φ S Φ WH he feature space F could be consdered as a lnearzaton space [6], however, ts dmensonalty could be arbtrarly large, and possbly nfnte. Solvng ths problem lead us to LDA[]. Assumng that s S WH nonsngular and Φ the bass vectors correspond to the M frst egenvectors wth the largest egenvalues of the dscrmnant crteron: J (6 tr ( S S WH BtWΦ he M-dmensonal representaton s then obtaned by projectng the orgnal face mages onto the subspace spanned by the egenvectors.. Kernel Drect Dscrmnant Analyss (KDDA he maxmzaton process n (3 s not drectly lnked to the classfcaton error whch s the crteron of performance used to measure the success of the F procedure. Modfed versons of the method, such as the Drect LDA (D-LDA approach, use a weghtng functon n the nput space, to penalze those classes that are close and can potentally lead to msclassfcatons n the output space. Most LDA based algorthms ncludng Fsherfaces [7] and D-LDA [9] utlze the conventonal Fsher s crteron denoted by (3. he ntroducton of the kernel functon allows us to avod the explct evaluaton of the mappng. Any functon satsfyng Mercer s condton can be used as a kernel, and typcal kernel functons nclude polynomal functon, radal bass functon (BF and mult-layer perceptrons [0]. Φ S BW Φ Φ arg max (7 Φ ( Φ S Φ + ( Φ S Φ BW WH he KDDA method mplements an mproved D- LDA n a hgh-dmensonal feature space usng a kernel approach. KDDA ntroduces a nonlnear mappng from the nput space to an mplct hgh dmensonal feature space, where the nonlnear and complex dstrbuton of patterns n the nput space s lnearzed and smplfed so that conventonal LDA can be appled and t effectvely solves the small sample sze (SSS problem n the hgh-dmensonal feature space by employng an mproved D-LDA algorthm. Unlke the orgnal D-LDA method of [0] zero egenvalues of the wthn-class scatter matrx are never used as dvsors n the mproved one. In ths way, the optmal dscrmnant features can be exactly S WH extracted from both of nsde and outsde of s null space. - -
SEI007 In GDA, to remove the null space of S WH, t s requred to compute the pseudo nverse of the kernel matrx K, whch could be extremely ll-condtoned when certan kernels or kernel parameters are used. Pseudo nverson s based on nverson of the nonzero egenvalues. 3 SVM Based Approach for Classfcaton he prncple of Support Vector Machne (SVM reles on a lnear separaton n a hgh dmenson feature space where the data have been prevously mapped, n order to take nto account the eventual non-lneartes of the problem. 3. Support Vector Machnes (SVM If we assume that, the tranng set l X ( x where l s the number of tranng vectors, stands for the real lne and s the number of modaltes, s labelled wth two class targets Y ( y l, where : y {, + } Φ : F (8 Maps the data nto a feature space F. Vapnk has proved that maxmzng the mnmum dstance n space F between Φ( X and the separatng hyper plane Hw (, b s a good means of reducng the generalzaton rsk. Where: H ( w, b { f F < w, f > ( <> s nner product F + b 0 Vapnk also proved that the optmal hyper plane can be obtaned solvng the convex quadratc programmng (QP problem: Mnmze wth l w + c ξ y( < w, Φ( X > + b ξ },,..., l (9 (0 Where constant C and slack varables x are ntroduced to take nto account the eventual nonseparablty of Φ ( X nto F. In practce ths crteron s softened to the mnmzaton of a cost factor nvolvng both the complexty of the classfer and the degree to whch margnal ponts are msclassfed, and the tradeoff between these factors s managed through a margn of error parameter (usually desgnated C whch s tuned through cross-valdaton procedures.although the SVM s based upon a lnear dscrmnator, t s not restrcted to makng lnear hypotheses. Non-lnear decsons are made possble by a non-lnear mappng of the data to a hgher dmensonal space. he phenomenon s analogous to foldng a flat sheet of paper nto any three-dmensonal shape and then cuttng t nto two halves, the resultant non-lnear boundary n the two-dmensonal space s revealed by unfoldng the peces. he SVM s non-parametrc mathematcal formulaton allows these transformatons to be appled effcently and mplctly: the SVM s objectve s a functon of the dot product between pars of vectors; the substtuton of the orgnal dot products wth those computed n another space elmnates the need to transform the orgnal data ponts explctly to the hgher space. he computaton of dot products between vectors wthout explctly mappng to another space s performed by a kernel functon. he nonlnear projecton of the data s performed by ths kernel functons. here are several common kernel functons that are used such as the lnear, d polynomal kernel ( K( x, y ( < x, y > + and the sgmodal kernel ( K( xy, tanh( < xy, > + a, where x and y are feature vectors n the nput space. he other popular kernel s the Gaussan (or "radal bass functon" kernel, defned as: x y K( x, y exp( (σ ( Where σ s a scale parameter, and x and y are feature-vectors n the nput space. he Gaussan kernel has two hyper parameters to control performance C and the scale parameterσ. In ths paper we used radal bass functon (BF. 3. Mult-class SVM he standard Support Vector Machnes (SVM s desgned for dchotomc classfcaton problem (two classes, called also bnary classfcaton. Several dfferent schemes can be appled to the basc SVM algorthm to handle the K-class pattern classfcaton problem. hese schemes wll be dscussed n ths secton. he K-class pattern classfcaton problem s posted as follow: Gven l..d. sample: ( x, y,..., ( xl, y l where x, for,...,l s a feature vector of length d and y {,..., k} s the class label for data pont x. Fnd a classfer wth the decson functon, f ( x such that y f ( x where y s the class label for x. he mult-class classfcaton problem s commonly solved by decomposton to several bnary problems for whch the standard SVM can be used. For solvng the mult-class problem are as lsted below: Usng K one-to-rest classfers (oneaganst-all - 3 -
SEI007 Usng k( k / par wse classfers Extendng the formulaton of SVM to support the k-class problem. 3... Combnaton of one-to-rest classfers hs scheme s the smplest, and t does gve reasonable results. K classfers wll be constructed, one for each class. he K-th classfer wll be traned to classfy the tranng data of class k aganst all other tranng data. he decson functon for each of the classfer wll be combned to gve the fnal classfcaton decson on the K-class classfcaton problem. In ths case the classfcaton problem to k classes s decomposed to k dchotomy decsons m( f x, m K,..., k where the rule fm( x separates tranng data of the m-th class from the other tranng patterns. he classfcaton of a pattern x s performed accordng to maxmal value of functons m( f x, m K, K,..., k.e. the label of x s computed as: f ( x arg(max( fm( x ( m k 3... Par wse Couplng classfers he schemes requre a bnary classfer for each possble par of classes. he decson functon of the SVM classfer for y -to- y and y -to- y has reflectonal symmetry n the zero planes. Hence only one of these pars of classfer s needed. he total number of classfers for a K-class problem wll then be kk ( /. he tranng data for each classfer s a subset of the avalable tranng data, and t wll only contan the data for the two nvolved classes. he data wll be relable accordngly,.e. one wll be labeled as + whle the other as -. hese classfers wll now be combned wth some votng scheme to gve the fnal classfcaton results. he votng schemes need the par wse probablty,.e. the probablty of x belong to class gven that t can be only belong to class or j. he output value of the decson functon of an SVM s not an estmate of the p.d.f. of a class or the par wse probablty. One way to estmate the requred nformaton from the output of the SVM decson functon s proposed by (Haste and bshran, 996 he Gaussan p.d.f. of a partcular class s estmated from the output values of the decson functon, f ( x, for all x n that class. he centrod and radus of the Gaussan s the mean and standard devaton of f ( x respectvely. 4 EXPEIMENS AND ESULS 4. Database In our work, we used a popular face databases (he UMIS [3], for demonstratng the effectveness of our combned KDDA and SVM proposed method. It s compared wth KPCA, GDA and KDDA algorthm wth nearest neghbor classfer. We use a radal bass functon (BF kernel functon: x y K( x, y exp( (3 (σ Where σ s a scale parameter, and x and y are feature-vectors n the nput space. he BF functon s selected for the proposed SVM method and KDDA n the experments. he selecton of scale parameterσ s emprcal. In addton, n the experments the tranng set s selected randomly each tme, so there exsts some fluctuaton among the results. In order to reduce the fluctuaton, we do each experment more than 0 tmes and use the average of them. 4. UMIS Database he UMIS repostory s a mult-vew database, consstng of 575 mages of 0 people, each coverng a wde range of poses from profle to frontal vews. Fgure depcts some samples contaned n the two databases, where each mage s scaled nto ( 9, resultng n an nput dmensonalty of N 0304. For the face recognton experments, n UMIS database s randomly parttoned nto a tranng set and a test set wth no overlap between the two set. We used ten mages per person randomly chosen for tranng, and the other ten for testng. hus, tranng set of 00 mages and the remanng 375 mages are used to form the test set. It s worthy to menton here that both expermental setups ntroduce SSS condtons snce the number of tranng samples are n both cases much smaller than the dmensonalty of the nput space []. Fgure : Some sample mages of four persons randomly chosen from the UMIS database. On ths database, we test the methods wth dfferent tranng samples and testng samples correspondng the tranng number k, 3, 4, 5,6,7,8 of each subject. Each tme randomly select k samples from each subject to tran and the other 0 K to test. he expermental results are gven n the table. - 4 -
SEI007 K able. ecognton rate (% on the UMIS database. KPC GD A A Our method (KDDA+SVM KDDA +NN * 8.8 8.9 75.5 7.5 3 83.5 83.4 76. 7.8 4 87.3 85.4 77. 74.5 5 90.4 87.9 79.8 75. 6 94. 89. 83.4 79.0 7 96.0 93.9 87. 8. 0 96.5 95. 89. 83.0 * Nearest Neghbour Fgure depcts the frst two most dscrmnant features extracted by utlzng KDDA respectvely and we show the decson boundary for frst 6 classes for tranng data n Combnaton of one-torest classfer SVM. Fgure : he decson boundary for frst 6 classes for tranng data (Combnaton of one-to-rest classfer SVM he only kernel parameter for BF s the scale valueσ for SVM classfer. Fgure.4 shows the error rates as functons ofσ, when the optmal number of feature vectors (M s optmum s used. As such, the average error rates of our method wth BF kernel are shown n Fgure 5. It shows the error rates as functons of M wthn the range from to 9 ( σ s optmum. 5 Dscussons and Conclusons A new F method has been ntroduced n ths paper. he proposed method combnes kernel-based methodologes wth dscrmnant analyss technques and SVM classfer. he kernel functon s utlzed to map the orgnal face patterns to a hgh-dmensonal feature space, where the hghly non-convex and complex dstrbuton of face patterns s smplfed, so that lnear dscrmnant technques can be used for feature extracton. he small sample sze problem caused by hgh dmensonalty of mapped patterns s addressed by a kernel-based D-LDA technque (KDDA whch exactly fnds the optmal dscrmnant subspace of the feature space wthout any loss of sgnfcant dscrmnant nformaton. hen feature space wll be fed to SVM classfer. Expermental results ndcate that the performance of the KDDA algorthm together wth SVM s overall superor to those obtaned by the KPCA or GDA approaches. In concluson, the KDDA mappng and SVM classfer s a general pattern recognton method for nonlnearly feature extracton from hghdmensonal nput patterns wthout sufferng from the SSS problem. We expect that n addton to face recognton, KDDA wll provde excellent performance n applcatons where classfcaton tasks are routnely performed, such as content-based mage ndexng and retreval, vdeo and audo classfcaton. Acknowledgements he authors would lke to acknowledge the Iran elecommuncaton esearch Center (IC for fnancally supportng ths work. Fgure 3: error rates as functons σ of SVM. 6 ( σ 5 0 [] KDDA Fgure 4: Comparson of error rates based on BF kernel functon. - 5 -
SEI007 We would also lke to thank Dr. Danel Graham and Dr. Ngel Allnson for provdng the UMIS face database. eferences [6] 0. Duda,., E. Han. P., and G. Stork, D. Parrern ecognron. John Wley & Sons, 000. [7] L. Mangasaran. 0. and. Muscant. D. Successve over relaxaton for support vector machnes, IEEE ransacrons on Neural Nerworks, l0(5, 999. [] J. Lu, K. N. Platanots, A. N. Venetsanopoulos, Face ecognton Usng LDA-Based Algorthms IEEE rans. ON Neural Networks, vol. 4, no., Jan.003. [] M.urk, A random walk through egenspace, IEICE rans. Inform.Syst., vol. E84-D, pp. 586 695, Dec. 00. [3]. Chellappa, C. L.Wlson, and S. Srohey, Human and machne recognton of faces: A survey, Proc. IEEE, vol. 83, pp. 705 740, May 995. [4] M. urk and A. P. Pentland, Egenfaces for recognton, J. Cogntve Neurosc., vol. 3, no., pp. 7 86, 99. [5] P. N. Belhumeur, J. P. Hespanha, and D. J. Kregman, Egenfaces vs. Fsherfaces: ecognton usng class specfc lnear projecton, IEEE rans. Pattern Anal. Machne Intell., vol. 9, pp. 7 70, May 997. [6] M. A. Azerman, E. M. Braverman, and L. I. ozonoer, heoretcal foundatons of the potental functon method n pattern recognton learnng, Automaton and emote Control, vol. 5, pp. 8 837, 964. [7] L.-F.Chen, H.-Y. Mark Lao, M-.. Ko, J.-C. Ln, and G.-J. Yu, A new LDA-based face recognton system whch can solve the small sample sze problem, Pattern ecognton, vol. 33, pp. 73 76, 000. [9] H. Yu and J. Yang, A drect LDA algorthm for hgh-dmensonal data wth applcaton to face recognton, Pattern ecognton, vol. 34, pp. 067 070, 00. [0] V. N. Vapnk, he Nature of Statstcal Learnng heory, Sprnger-Verlag, New York, 995. [] D. B. Graham and N. M. Allnson, Characterzng vrtual egensgnatures for general purpose face recognton, n Face ecognton: From heory to Applcatons, H. Wechsler, P. J. Phllps, V. Bruce, F. Fogelman- Soule, and. S. Huang, Eds., 998, vol. 63, NAO ASI Seres F, Computer and Systems Scences, pp. 446 456. [] D. L. Swets and J. Weng, Usng dscrmnant egenfeatures for mage retreval, IEEE rans. Pattern Anal. Machne Intell., vol. 8, pp. 83 836, Aug. 996. [3] Q. Lu,. Huang, H. Lu, S. Ma, Kernel-Based Optmzed Feature Vectors Selecton and Dscrmnant Analyss for Face ecognton 00 IEEE [4] C. Lu and H.Wechsler, Evolutonary pursut and ts applcaton to face recognton, IEEE rans. Pattern Anal. Machne Intell., vol., pp. 570 58, June 000. [5] K. Lu, Y. Q. Cheng, J. Y. Yang, and X. Lu, An effcent algorthm for Foley Sammon optmal set of dscrmnant vectors by algebrac method, Int. J. Pattern ecog. Artfcal Intell., vol. 6, pp. 87 89, 99. - 6 -