Regularized Orthogonal Local Fisher Discriminant Analysis

Transcription

1 Regularized Orthogoal Local Fisher Discrimiat Aalysis Shuhua Xu Departmet of Maths Uiversity of Shaoxig City South Road, Shaoxig P.R Chia Joural of Digital Iformatio Maagemet ABSRAC: Aimig at deficiecies of the ability for preservig local oliear structure of recetly proposed Regularized Orthogoal Liear Discrimiat Aalysis () for dimesioality reductio, a kid of dimesioality reductio algorithm amed Regularized Orthogoal Local Fisher Discrimiat Aalysis () is proposed i the paper, which is origiated from. he algorithm itroduce the idea of local structure preservig i Local Fisher Discrimiat Aalysis (LFDA) o the basic of, followig properties of ad stregtheig the ability for capturig local structure iformatio of data with oliear structures. Experimets o real face datasets demostrate the effectiveess of our proposed algorithm. Categories ad Subject Descriptors: G.4: [Mathematical aalysis]: Noliear equatios; I..2: [Symbolic ad algebraic algorithms] Geeral erms: Fisher Discrimiat Aalysis, Noliear equatios Keywords: Dimesioality Reductio, Regularized Orthogoal Liear Discrimiat Aalysis, Local Noliear Structure, Local Fisher Discrimiat Aalysis Received: 8 October 2, Revised 2 December 2, Accepted 2 December 2. Itroductio I applicatios of data miig, high-dimesioal data lead to too much redudat feature iformatio ad icrease the computatioal complexity of disposig. Hece dimesioality reductio is ecessary. he purpose of dimesioality reductio is to fid a low-dimesioal represetatio of the high-dimesioal data at miimum cost i the process of preservig certai features. he most well-kow iitial dimesioality reductio methods are Pricipal Compoet Aalysis (PCA) [] ad Liear Discrimiat Aalysis (LDA) [2]. As supervised dimesioality reductio, LDA searches the projectio axes o which the ot-same-class poits are far from each other while requirig the same-class poits to be close to each other. herefore, LDA ecodes discrimiatig iformatio i a liearly separable space ad is successfully applied ito face recogitio. However, it is much expesive to attai plety of class labeled samples i practical applicatios of face recogitio, which cause that the total scatter matrix ca be sigular i LDA sice the data poits are from a very high-dimesioal space ad thus usually the umber of the data samples is much smaller tha the data dimesio. his is kow as the udersampled problem. o solve the udersampled problem i LDA applicatio, extesios of LDA ca be proposed i the literature. he two-stage LDA is proposed i [3-5], which is to apply a itermediate dimesioality reductio stage to reduce the data dimesioality. Although this method is simple, some importat iformatio is removed i the itermediate dimesioality reductio stage. he regularized LDA () is proposed i [6-], which is to cosider regularizatio by addig the perturbatio to the total scatter matrix. he methods based 54 Joural of Digital Iformatio Maagemet Volume Number 2 April 3

2 o pseudoiverse [] are applied to avoid the sigularity problem, icludig orthogoal LDA (OLDA) [2-3], ull space LDA (NLDA) [4-6], ucorrelated LDA [7], QR/ GSVD-based LDA [8-]. OLDA obtais easily the optimal trasformatio matrix by oly orthogoal trasformatios without computig ay eige-decompositio ad matrix iverse. Moreover, OLDA is implemeted by usig several QR factorizatios ad is a fast oe. I above exteds of LDA for the udersampled problem, is a efficiet method with little computatio. he major problem of the is to choose a appropriate regularizatio parameter. Usually these algorithms select the regularizatio parameter from a give parameter cadidate set by usig cross-validatio for classificatio. But how to choose a appropriate cadidate set is ot clear. herefore, up to ow, there is o cocrete mathematical theory available i selectig a appropriate regularizatio parameter i practical applicatios of the. For fillig this gap, Chig et al proposed Regularized Orthogoal Liear Discrimiat Aalysis () [2]. derives from the mathematical relatioship betwee OLDA ad ad fids a mathematical criterio for selectig the regularizatio parameter from the relatio. However, the performace of teds to be degraded o o-liear data because that is origiated from LDA which works very well if the samples i each class follow Gaussia distributios with a liear covariace structure. o overcome this drawback, Local FDA (LFDA) [22-23] has bee proposed, which combies the ideas of FDA ad LPP to maximize betwee-class reparability while preserve withi-class local structure. LFDA ca provide more separate embeddig tha FDA. Ispired by LFDA, a kid of Regularized Orthogoal Local Fisher Discrimiat Aalysis () is proposed for dimesioality reductio. he algorithm itroduces the idea of LFDA ito, preservig local structure iformatio of samples ad calculatig the regular parameter automatically. herefore ca be applied i highdimesioal data with o-liear structure. Experimets o YaleB ad AR demostrate the proposed algorithm is valid. he rest of the paper is orgaized as follows: Sectio 2 reviews LDA, LFDA ad. Our is itroduced i Sectio 3. I Sectio 4, we compare with some related works. he experimetal results ad aalyses are preseted. Fially, we provide some cocludig remarks ad future work i Sectio Related works 2. Liear Discrimiat Aalysis (LDA) Give samples X = {x i x i R d } i =, cotaiig C classes. deotes the umber of samples. he goal of LDA is to seek a projectio matrix such that the betwee-class scatter is maximized ad the withi-class scatter is miimized i projected data. he objective fuctio of LDA is as follows: = arg max tr (( S w ) ( S b )) Where S b deotes the betwee-class scatter matrix ad S w deotes the withi-class scatter matrix S b ad S w are defied as follow: S b = (x ij x ) (x i x ) C i = j= S w = (x ij x i ) (x ij x i ) Where x ij deotes the j th sample i the i th class. x = x i = i deotes the average of all samples X, i x = x ij deotes the average of samples i the i th j = i i class. i ( i C) deotes the umber of samples i the i th class. Equatio () may be further trasformed ito followig geeralized eigevalue problem: () (2) (3) S b t m = (4) λ m S w t m, m =,..., l (l < d ) Get the projectig matrix = [t, t 2,..., t l ]. 2.2 Local Fisher Discrimiat Aalysis (LFDA) LFDA is a localized variat of Fisher discrimiat aalysis. LFDA takes local structure of the data ito accout so the multimodaldata ca be embedded appropriately. LFDA redefie S (lb) ad S (lw) as follows: where S (lb) = S (lw) = 2 2 i, j = i, j = (lb) (x i )(x i ) (lw) (x i )(x i ) (lb) = Ai, j (/ / ) if y = y y i i j / if y i y j (lw) = A i, j (/ y ) if y i = y i j if y i y j (5) (6) (7) (8) x A i, j = exp i 2 ( ) (9) δ i δ j k δ i = x i x i () Where x k i is the k th earest eighbor of x i, A i, j deotes the affiity weight of x i ad x j, the parameter represets the local scalig aroud x i. he objective fuctio of LFDA is defied as follows: = arg max tr (( S (lw) ) ( S (lb) )) () Joural of Digital Iformatio Maagemet Volume Number 2 April 3 55

3 2.3 Regularized Orthogoal Liear Discrimiat Aalysis () O the base of [7], the objective fuctio of is as follows: = arg max tr (( S w + λi) ( S b )) (2) s.t. = I Where, the appropriate regularizatio parameter λ is selected accordig to [2]. 3. Regularized Orthogoal Local Fisher Discrimiat Aalysis () 3. Basic idea () is a efficiet algorithm for the udersampled problem. However, the mai task of the regularized LDA is to choose a appropriate regularizatio parameter λ. If is λ large, iformatio o the scatter matrix is lost. While if it is too small, the regularizatio may ot be sufficietly effective. Usually existig methods select the regularizatio parameterfrom a cadidate set of the regularizatio parameter give with the cross-validatio method. It has bee show i [] that the matrix computatios ivolved i the regularized LDA ca be simplified so that the cross-validatio procedure ca be performed efficietly. However, for methods, it is ot clear how to choose a appropriate cadidate set. makes the aalysis of the relatioship betwee OLDA ad.by meas of this relatioship, fid a mathematical criterio for selectig the regularizatio parameter i. Cosequetly develops a ew regularized orthogoal liear discrimiat aalysis method, i which o cadidate set of regularizatio parameter is eeded. (2) I may practical applicatios of data miig, itrisical low dimesioal data ofte lie i a very high dimesioal space. It is importat to explicitly take ito accout the structure of the maifold o which the data may possibly reside. LPP is to fid the optimal liear approximatios to the eige fuctios of the Laplace Beltrami operator o the maifold, sharig may of the data represetatio properties of oliear dimesioality reductio such as Laplacia Eige maps or Locally Liear Embeddig. LPP keeps earby data pairs i the origial space close i the embeddig space, by which oliear data ca be embedded without losig its local structure. LFDA combies the idea of LPP ad FDA, iheritig merits of them. As above aalyses, o the base of, the idea of local preservig i LFDA is itroduced. his will overcome the poor ability of for capturig the local structure ad iherit characteristics of them. 3.2 Objective fuctio O the base of Equatio (2), the objective fuctio of is defied as follows: = arg max tr ((λ ( S (lb) ) + ( S (lw) )) ( S (lb) )) (3) s.t. = I Where, accordig to [], the regularizatio parameter λ is solved with followig steps: () Give samples X = {x i x i R d } i =, cotaiig k class face. Firstly, R ad Q are gotte via QR-decompositio as follows: X = Q R (4) Where Q R d is colum orthogoal. (2) X R, X 2 R (k ) ad X 2 R ( k) are computed as follows: [X X 2 X 3 ] = R ϑ ϑ ϑ ρ k Where ρ be the permutatio matrix obtaied by exchagig the ( ) i = j = j + colum (for i = 2,., k), but otherwise leavig the order of the remaiig colums uchaged. ϑ i (i = 2,., k) ad ϑ are calculated as follows: ϑ i = i I 2 k (3) calculate th-colum of I ad i th i / i i / i i i =,., k ϑ = I / i [X X 2 ] = Q 2 k R, / i (8) Where R, R q (k ) ad R (γ q ) ( k) are of full row rak. (4) calculate λ λ = ε (6) (9) (+) (+) (+) 2 (+ R, 2 2 [ε + 2 ( + 2 ) ( q + R, 2 F )]) (+) Where ε deotes error, deotes the pseudo iverse of. 3.3 Algorithm steps Iput: samples X = {x i x i R d } i = with C class, error ε >. Output:the colum orthogoal projectio matrix R d l. Steps: () calculate W (lb) W (lw) usig Equatio (7) ad Equatio (8) (7) 56 Joural of Digital Iformatio Maagemet Volume Number 2 April 3

4 (2) calculate Q usig Equatio (4). (3) calculate X, X 2 ad X 3 usig Equatio (5). (4) calculate R,, R, 2 ad usig Equatio (8). (5) calculate λ usig Equatio (9). (6) trasform Equatio (2) ito followig geeralized matrix problem:, S (lb) t m = β m (λ S (lb) + S (lw) ) t m, m =,..., l ad calculate = [t, t 2,..., t l ]. 4. Experimetal results ad aalysis 4. Experimetal Datasets ad Settigs Face datasets are typical high-dimesioal data with oliear structures. I the experimet, YaleB ad AR are selected ad are describe as follows: ) YaleB cotais 244 frot-view face images of 38 idividuals. For each idividual, about 64 pictures were take uder various laboratory-cotrolled lightig coditios. I our experimets, he cropped images with the resolutio of are used. A group of samples i YaleB are show i Figure. 2) AR database cotais over face images of 26 idividuals. For each idividual, 26 pictures were take i two sessios (separated by two weeks) ad each sectio cotais 3 images. hese images iclude frot view of faces with differet expressios, illumiatios ad occlusios. I the experimet, them are resized to A group of samples i AR are show i Figure 2. ad are compared with ad the Nearest Neighbor Classifier is adopted i order to verify the performace of our algorithm. images are selected radomly from a group face ad remais for test samples. Moreover, the regularizatio parameter λ i is set to. ad the eighbor parameter k i is set to Experimetal Results Experimets are made uder differet ad differet reduced dimesios. All experimets are repeated twety times ad average recogitio accuracy rates are gotte. Reduced dimesios are selected with the certai icremet ad correspodig average recogitio accuracies are calculated. Experimetal results are show i Figure 3- Figure 4 Recogitio accuracy (%) Recogitio accuracy (%) Dimesio (a) = Dimesio (b) = Figure 3. Experimetal results o YaleB with Figure. A group of samples i YaleB Figure 2. A group of samples i AR Joural of Digital Iformatio Maagemet Volume Number 2 April 3 57

5 Recogitio accuracy (%) Recogitio accuracy (%) Dimesio (a) = 5 Dimesio (a) = Figure 4. Experimetal results o AR with From above Figure 3- Figure 4, i the term of the relatio of recogitio accuracy ad reduced dimesios, coclusios ca be draw as follows: () With icrease i reduced dimesio, the recogitio accuracy rate of all algorithms promote rapidly firstly ad go smoothly, which shows that these algorithms ca get most recogitio accuracy rates i low dimesio.he reasos is that they is origiated from LDA with the merit. (2) is superior to ad i AR ad YaleB with oliear structures, which is illustrated by that ifuses efficietly the characteristics of preservig local structure i LFDA while ad fail to capture local structures. Moreover, i the term of the relatio of recogitio accuracy ad the umber of traiig samples, some followig coclusios are draw: ) As experimets o YaleB show, the most recogitio accuracy rate of is respectively higher tha that of ad whe is set to. he gap betwee them becomes larger whe is set to. his illumiate that more traiig samples is helpful for to capture the local structure feature. 2) he performace of is better uder = tha that uder =, which shows that more traiig samples make it for with the power ability for capturig local structure features to be more applied o AR with power exteral disturb, icludig illumiatios ad occlusios. 5. Coclusio I the paper, Regularized Orthogoal Local Fisher Discrimiat Aalysis () for dimesioality reductio is proposed. Aimig at the deficiecy o the ability for capturig local structures iformatio, itroduce the idea of preservig local structures i Local Fisher Discrimiat Aalysis (LFDA) o the base of, iheritig merits of ad promote the ability for local learig. Experimetal results o YaleB ad AR show the proposed algorithm fuse efficietly LFDA ad. However, for, the local structure preservig is based o the approximatio of liearizatio istead of istict geometrical structure. How to fuse other oliear algorithm i is our ext work. 6. Ackowledgemets he research is supported by NSF of Zhejiag provice (grat No. LQ2F7, Y22544). Refereces [] Fukuaga, K. (9). Itroductio to Statistical Patter Recogitio, 2 d ed., Academic Press, Bosto, USA. [2] Duda, R. O., Hart, P. E., Stork. D. G. ().Patter Classificatio, secod ed., Joh Wiley & Sos, New York. [3] Howlad, P., Park, H. (4). Geeralizig discrimiat aalysis usig the geeralized sigular value decompositio. IEEE rasactios o Patter Aalysis ad Machie Itelligece, 26 (8) [4] Swets, D. L.,Weg, J. J. (996). Usig discrimiat eigefeatures for image retrieval, IEEE rasactios o Patter Aalysis ad Machie Itelligece, 8 (8) [5] Ye, J. P., Li, Q. (5). A two-stage liear discrimiat aalysis via QR-decompositio. IEEE rasactios o Patter Aalysis ad Machie Itelligece, 27(6) [6] Jerome, H. F. (989). Regularized discrimiat aalysis. Joural of the America Statistical Associatio, 84 (5) [7] Dai, D. Q., Yue, P. C. (3). Regularized discrimiat aalysis ad its applicatio to face recogitio. Patter Recogitio, 36 (3) [8] Guo,Y. Q., Hastie,. (7). Regularized liear discrimiat aalysis ad its applicatio i microarray. Biostatistics, 8 () Joural of Digital Iformatio Maagemet Volume Number 2 April 3

6 [9] Ye, P., Xiog,., Li, Q., Jaarda,R., Bi, B., Cherkassky V., Kambhamettu, C. (6). Efficiet model selectio for regularized liear discrimiat aalysis. I: Proc. of the 5 th ACM iteratioal coferece o Iformatio ad kowledge maagemet (CIKM 6), p ACM, November. [] Jiepig, Y., ie, W. (6). Regularized discrimiat aalysis for high dimesioal, low sample size data. I: Proc. of Proceedigs of the 2 th ACM SIGKDD Iteratioal Coferece o Kowledge discovery ad Data Miig (KDD 6), p ACM, August. [] Golub, G. H., VaLoa, C. F. (996). Matrix Computatios, third ed.. he Johs Hopkis Uiversity Press, Baltimore, MD. [2] Chu, D. L, Goh, S.. (). A ew ad fast orthogoal liear discrimiat aalysis o udersampled problems. SIAM Joural o Scietific Computig, 32 (4) [3] Ye, J. P. (5). Characterizatio of a family of algorithms for geeralized discrimiat aalysis o udersampled problems. Joural of Machie Learig Research, 6 (4) [4] Che, L. F., Liao, H. Y., Ko, M.., Li, J. C., Yu, G. J. (). A ew LDA-based face recogitio system which ca solve the small sample size problem. Patter Recogitio, 33 () [5] Chu, D., hye, G. S. (). A ew ad fast implemetatio for ull space based liear discrimiat aalysis. Patter Recogitio, 43 (4) [6] Rui, H., Qigsha, L., Haqig, L., Sogde, M. (2). Solvig the small sample size problem of LDA. I: Proc. of the Iteratioal Coferece o Patter Recogitio (ICPR 2), p IEEE Computer Society, April. [7] Chu, D. L., Goh, S.., Hug, Y. S. (). Characterizatio of all solutios for udersampled ucorrelated liear discrimiat aalysis problems. SIAM Joural o Matrix Aalysis ad Applicatios, 32 (2) [8] Peg, H., Moogu, J., Haesu, P. (3). Structure preservig dimesio reductio for clustered text data based o the geeralized sigular value decompositio. SIAM Joural o Matrix Aalysis ad Applicatios 25 () [9] Haesu, P., Drake, B. L., Sagmi, L., Park, C. H. (7). Fast Liear Discrimiat Aalysis Usig QR Decompositio ad Regularizatio. echical Report G- CSE-7-2, Georgia Istitute of echology. [] Ye, J. P., Jaarda, R., Park, C. H., Park, H. (4). A optimizatio criterio for geeralized discrimiat aalysis o udersampled problems. IEEE rasactios o Patter Aalysis ad Machie Itelligece, 26 (8) [2] Chiga, W. K, Chub, D., Liaoc, L. Z, Wagb, X. Y (2). Regularized Orthogoal Liear Discrimiat Aalysis. Patter Recogitio, 45 (7) [22] Masashi, S. (6). Local Fisher discrimiat aalysis for supervised dimesioality reductio. I: Proc. of the 23 rd iteratioal coferece o Machie learig (ICML 6), p ACM, Apr. [23] Masashi, S. (7). Dimesioality reductio of multimodal labeled data by local Fisher discrimiat aalysis. Joural of Machie Learig Research, 8 (5) Author Biography Shuhua XU was bor i Jiagxi, Chia, i 977. She received her M.S. degree from Guizhou Uiversity, Guizhou, Chia, i 4. Now she is a istructor of the School of Mathematics Sciece ad physics Sciece. Her research iterests iclude optimizatio theory ad applicatio, machie learig theory ad patter recogitio. Joural of Digital Iformatio Maagemet Volume Number 2 April 3 59