Neurocomputing. Kernel sparse representation based classification. Jun Yin a,n, Zhonghua Liu a, Zhong Jin a, Wankou Yang b. abstract.

Transcription

1 Neurocomputing 77 (2012) Contents lists ville t SciVerse ScienceDirect Neurocomputing journl homepge: Kernel sprse representtion sed clssifiction Jun Yin,n, Zhonghu Liu, Zhong Jin, Wnkou Yng School of Computer Science nd Technology, Nnjing University of Science nd Technology, Nnjing , Chin School of Automtion, Southest University, Nnjing , Chin rticle info Article history: Received 22 Decemer 2010 Received in revised form 23 August 2011 Accepted 28 August 2011 Communicted y Y. Fu Aville online 22 Septemer 2011 Keywords: Clssifiction Sprse representtion Kernel strct Sprse representtion hs ttrcted gret ttention in the pst few yers. Sprse representtion sed clssifiction (SRC) lgorithm ws developed nd successfully used for clssifiction. In this pper, kernel sprse representtion sed clssifiction (KSRC) lgorithm is proposed. Smples re mpped into high dimensionl feture spce first nd then SRC is performed in this new feture spce y utilizing kernel trick. Since smples in the high dimensionl feture spce re unknown, we cnnot perform KSRC directly. In order to overcome this difficulty, we give the method to solve the prolem of sprse representtion in the high dimensionl feture spce. If n pproprite kernel is selected, in the high dimensionl feture spce, test smple is proly represented s the liner comintion of trining smples of the sme clss more ccurtely. Therefore, KSRC hs more powerful clssifiction ility thn SRC. Experiments of fce recognition, plmprint recognition nd finger-knuckle-print recognition demonstrte the effectiveness of KSRC. & 2011 Elsevier B.V. All rights reserved. 1. Introduction In pttern recognition, clssifiction is n indispensle step nd clssifier design is one of the most populr technologies. Severl clssifiction pproches hve een proposed over the pst severl decdes [1,2]. Among them, the nerest-neighor (NN) clssifier nd the nerest-men (NM) clssifier re most widely used ecuse of their simpleness nd vilility. The NN clssifier ssigns test smple to the ctegory of its nerest neighor from the leled trining set. Insted of serching the nerest trining smple, the NM clssifier ssigns test smple to the ctegory of its nerest clss men. Over the lst more thn 10 yers, the kernel sed lgorithms [3] such s kernel principl component nlysis (KPCA) [4] nd kernel fisher discriminnt nlysis (KFD) [5,6] hve roused considerle interest in pttern recognition nd mchine lerning. These kernel sed lgorithms improve the computtionl power of the liner lgorithms. They mp the dt into high dimensionl feture spce y nonliner mpping nd perform liner lgorithms in the high dimensionl feture spce using the inner products. In the high dimensionl feture spce, the inner products cn e computed y kernel function. For clssifiction, utilizing kernel pproch, Yu et l. present the kernel nerest neighor (KERNEL-NN) clssifier [7]. KERNEL-NN pplies the nerest neighor clssifiction method in the high dimensionl n Corresponding uthor. E-mil ddress: yinjun8429@163.com (J. Yin). feture spce. Kernel pproch could chnge the distriution of smples y the nonliner mpping. Some linerly inseprle smples in the originl feture spce cn ecome linerly seprle in the high dimensionl feture spce. If n pproprite kernel is chosen to reshpe the distriution of smples, the KERNEL-NN clssifier could perform etter thn the NN clssifier. Recently, sprse representtion ecomes hot topic of pttern recognition nd computer vision. It is pplied to imge superresolution [8], motion segmenttion [9] nd supervised denoising [10]. Wright et l. pply sprse representtion to clssifiction nd exploit the sprse representtion sed clssifiction (SRC) lgorithm [11]. For SRC, test smple is represented s sprse comintion of trining smples, nd its sprse representtion coefficient is otined y solving the prolem of sprse representtion. The test smple is ssigned to the clss tht minimizes the residul etween itself nd the reconstruction constructed y trining smples of this clss. SRC shows its effectiveness in fce recognition experiments. As well known, fter mpping smples into high dimensionl feture spce y nonliner mpping, kernel pproch cn chnge the distriution of smples. If n pproprite kernel function is utilized, for test smple, more neighors proly hve the sme clss lel s itself in the high dimensionl feture spce. Here, in the high dimensionl feture spce, the test smple cn e represented more ccurtely s the comintion of trining smples from the sme clss. Then the nonzero entries of sprse representtion coefficient vector of the test smple will e more ssocited with trining smples from the sme clss s itself. Nmely, sprse representtion coefficient in the high dimensionl /$ - see front mtter & 2011 Elsevier B.V. All rights reserved. doi: /j.neucom

2 J. Yin et l. / Neurocomputing 77 (2012) feture spce cn denote the ctegory of the test smple more ccurtely nd it hs more powerful discriminting ility. To use sprse representtion coefficient in the high dimensionl feture spce, we propose the kernel sprse representtion sed clssifiction (KSRC) lgorithm in this pper. For KSRC, smples re mpped into high dimensionl feture spce first nd then SRC is performed in this new feture spce. We prove tht SRC in the high dimensionl feture spce cn e formulted in terms of the inner products, while the inner products could e computed y kernel function. Comprehensive comprisons etween KSRC nd NM, NN, KERNEL-NN nd SRC revel the superior chrcteristics of KSRC. The rest of the pper is orgnized s follows: Section 2 descries SRC lgorithm nd proposes KSRC lgorithm. Section 3 descries experiments on severl populr dtses. Finlly, the conclusions re summrized in Section The proposed lgorithm 2.1. Sprse representtion sed clssifiction (SRC) [11] Suppose tht we hve n trining smples for c clsses nd sufficient trining smples elong to the kth clss, A k ¼½x k,1, x k,2,...,x k,nk ŠAR mn k, where m is the dimension of smples nd n k is the numer of trining smples of the kth clss. Any test smple yar m from the kth clss cn e pproximtely represented s the liner comintion of trining smples of this clss: y ¼ k,1 x k,1 þ k,2 x k,2 þþ k,nk x k,nk ð1þ Since the lel of y is unknown initilly, we represent y s the liner comintion of ll the trining smples: y ¼ A 0 ð2þ where A ¼½A 1,A 2,...,A c Š¼½x 1,1,x 1,2,...,x c,nc ŠAR mn is mtrix composed of ll the n trining smples of c clsses nd 0 ¼ ½0,...,0, k,1, k,2,..., k,nk,0,...,0š T AR n is the coefficient vector whose nonzero entries re only ssocited with the kth clss. When c is lrge, 0 will e sprse. If mon, Eq. (2) is underdetermined. The prolem of sprse representtion is to serch vector such tht Eq. (2) is stisfied nd :: 0 is minimized, where :: 0 is the l 0 -norm of. This cn e descried s ^ 0 ¼ rg min :: 0 suject to y ¼ A ð3þ However, finding the sprse solution of Eq.(3) is NP-hrd [12]: nmely, there is no known procedure for otining the sprsest solution, which is significntly more efficient thn exhusting ll susets of the entries for. The theory of sprse representtion nd compressive sensing [13 15] revels tht we cn solve the following convex relxed optimiztion to otin pproximte solution: ^ 1 ¼ rg min :: 1 suject to y ¼ A ð4þ where :: 1 is the l 1 -norm of. Thisprolemcnesolvedy stndrd liner progrmming methods [16]. Furthermore, the oservtions re often inccurte, then we should relx the constrint in Eq. (4) nd get the following optimiztion prolem: ^ 1 ¼ rg min :: 1 sujectto :A y: 2 re ð5þ where e is the tolernce of the reconstruction error. This convex optimiztion prolem cn e solved vi second-order cone progrmming [16]. The optimiztion prolem (5) is minly used to del with smll noise. In prctice, the oservtions possily contin ig noise. For exmple, the imges re corrupted or occluded. Here, the errors cnnot e ignored or solved y the optimiztion prolem (5). The constrint should e modified s y ¼ Aþe ¼½AIŠ ð6þ e where ear m is vector of errors, IAR mm is the identity mtrix. Now, we get the following optimiztion prolem: ^g 1 ¼ rg min :g: 1 suject to y ¼ Pg ð7þ g where P ¼½A IŠAR mðn þ mþ,g ¼ " AR n þ m nd ^g ^ # 1 e 1 ¼ AR n þ m ^e 1 Let ^ 1 denote the solution of sprse representtion prolem (7) otined y l 1 -minimiztion. Idelly, the nonzero entries in ^ 1 will e ssocited with the columns of A from single oject clss, nd we cn esily ssign the test smple y to tht clss. However, noise nd modeling error my cuse smll nonzero entries ssocited with multiple clsses. Simple heuristics such s ssigning y to the clss with the lrgest entry re not dependle. Insted, we define new vector ^ k 1ðk ¼ 1,2,...,cÞ whose only nonzero entries re the entries in ^ 1 tht re ssocited with clss k. The reconstruction with the trining smples of the kth clss is ^y k ¼ A^ k 1ðk ¼ 1,2,...,cÞ. Then y cn e ssigned to the clss tht minimizes the residul etween y nd ^y k : min r k ðyþ¼:y A^ k 1 : 2 k The SRC lgorithm is summrized s follows: Algorithm 1. Sprse representtion sed clssifiction (SRC) 1. Input: the mtrix of trining smples AAR mn, test smple yar m. 2. Normlize the columns of A to hve unit l 2 -norm. 3. Solve the l 1 -minimiztion prolem defined in Eq. (4) or (5) or (7). 4. Compute the residuls r k ðyþ ðk ¼ 1,2,...,cÞ defined in Eq. (8). 5. Output : identityðyþ¼rg minðr k ðyþþ k 2.2. Kernel sprse representtion sed clssifiction (KSRC) As we know, kernel pproch cn chnge the distriution of smples y mpping smples into high dimensionl feture spce [7]. This chnge possily hs two effects if n pproprite kernel function is selected. On the one hnd, some liner inseprle smples in the originl feture spce ecome liner seprle in the high dimensionl feture spce. This leds to superiority of the KERNEL-NN clssifier over the NN clssifier. On the other hnd, test smple cn e represented s the liner comintion of trining smples from the sme clss s itself more ccurtely in the high dimensionl feture spce thn originl. Then the nonzero entries of sprse representtion coefficient vector of the test smple re more ssocited with trining smples of the sme clss. This results in etter clssifiction ility of SRC. So we perform SRC in the high dimensionl feture spce nd propose kernel sprse representtion sed clssifiction (KSRC). Becuse the explicit mpping from the originl feture spce to the high dimensionl feture spce is unknown, KSRC cnnot e performed directly. However, we successfully solve this prolem y Theorem 1. Suppose tht smples re mpped from originl feture spce R m into high dimensionl feture spce H y nonliner ð8þ

3 122 J. Yin et l. / Neurocomputing 77 (2012) mpping f: R m -H, x-fðxþ ð9þ Let B ¼½fðx 1,1 Þ,fðx 1,2 Þ,...,fðx c,nc ÞŠ represent the mtrix composed of ll the trining smples fter the nonliner mpping f. The prolem of sprse representtion in H cn e descried s [11] ^ 0 ¼ rg min :: 0 suject to fðyþ¼b ð10þ where f(y) is ny test smple in the high dimensionl feture spce, which corresponds to y in the originl feture spce. Similrly, the pproximte solution of Eq. (10) cn e otined through the following convex relxed optimiztion [13 15]: ^ 1 ¼ rg min :: 1 suject to fðyþ¼b ð11þ When the oservtions re not ccurte, the constrint in Eq. (11) should e relxed nd the following optimiztion prolem is otined: ^ 1 ¼ rg min :: 1 suject to :B fðyþ: 2 re ð12þ If we set e¼0, Eq. (12) is equivlent to Eq. (11). So Eq. (11) cn e seen s specil cse of Eq. (12) nd we cn only consider the optimiztion prolem (12). Since B nd f(y) re unknown, Eq. (12) cnnot e solved directly. But ccording to Theorem 1, Eq. (12) cn e trnsformed to ^ 1 ¼ rg min :: 1 suject to :B T B B T fðyþ: 2 rd ð13þ Theorem 1. For ny ez0, there must exist dz0 such tht we hve :B fðyþ: 2 re, s long s :B T B B T fðyþ: 2 rd is stisfied. The inner product of smples in the high dimensionl feture spce cn e computed y kernel function. Nmely, for ny smples x nd y, we hve fðxþ T fðyþ¼kðx,yþ, where kðx,yþ is kernel function. Then B T B ¼½fðx 1,1 Þ,fðx 1,2 Þ,...,fðx c,nc ÞŠ T ½fðx 1,1 Þ,fðx 1,2 Þ,...,fðx c,nc ÞŠ 2 3 kðx 1,1,x 1,1 Þ kðx 1,1,x 1,2 Þ kðx 1,1,x c,nc Þ kðx 1,2,x 1,1 Þ kðx 1,2,x 1,2 Þ kðx 1,2,x c,nc Þ ¼ 6 ^ ^ & ^ kðx c,nc,x 1,1 Þ kðx c,nc,x 1,2 Þ kðx c,nc,x c,nc Þ nd 2 3 kðx 1,1,yÞ B T kðx fðyþ¼½fðx 1,1 Þ,fðx 1,2 Þ,...,fðx c,nc ÞŠ T 1,2,yÞ fðyþ¼ 6 4 ^ 7 5 kðx c,nc,yþ ð14þ ð15þ When the kernel function kðx,yþ is given, B T B nd B T fðyþ re otined. Now we could solve the convex optimiztion prolem (13) vi second-order cone progrmming [16]. If the oservtions contin ig noise, s SRC, the constrint in Eq. (13) should e modified s B T fðyþ¼b T BþE ¼½B T B IŠ ~ ð16þ E where EAR n is vector of errors, I ~ AR nn is the identity mtrix. Utilizing constrint (16), the following optimiztion prolem is otined: ^Z 1 ¼ rg min :Z: 1 suject to B T fðyþ¼qz ð17þ Z where Q ¼½B T B IŠAR ~ n2n,z ¼ " # ^ AR 2n nd ^Z 1 E 1 ¼ ^E 1 AR 2n Let ^ 1 denote the solution of optimiztion prolem (17). Similr to SRC, we define new vector ^ k 1ðk ¼ 1,2,...,cÞ y setting only those entries in ^ 1 ssocited with clss k nonzero nd ssigning zero to other entries. Then y cn e ssigned to the clss tht minimizes the residul etween B T fðyþ nd B T B ^ k 1 : min R k ðyþ¼:b T fðyþ B T B ^ k 1 : 2 k ð18þ Algorithm 2. Kernel sprse representtion sed clssifiction (KSRC) 1. Input: the mtrix of trining smples AAR mn, test smple yar m nd kernel function. 2. Normlize the columns of A to hve unit l 2 -norm. 3. Clculte B T B nd B T fðyþ y Eqs. (14) nd (15). 4. Solve the l 1 -minimiztion prolem defined in Eqs. (13) or (17). 5. Compute the residuls R k ðyþ ðk ¼ 1,2,...,cÞ defined in Eq. (18). 6. Output : identityðyþ¼rg minðr k ðyþþ k For smples contining smll noise, the computtionl cost of SRC nd KSRC is minly cused y solving the convex optimiztion prolem (5) nd (13), respectively. According to AAR mn nd B T BAR nn, the computtionl complexity of solving Eqs. (5) nd (13) re oth Oðn 3 Þ. Here, SRC nd KSRC hve the sme computtionl cost. For smples contining ig noise, the computtionl cost of SRC nd KSRC is minly cused y solving the convex optimiztion prolems (7) nd (17) seprtely. We know tht the size of P is m ðnþmþ nd the size of Q is n 2n. Then the computtionl complexity of solving Eq. (7) is OððnþmÞ 3 Þ nd the computtionl complexity of solving Eq. (17) is Oðð2nÞ 3 Þ [23]. At this time, if the numer of trining smple size n is smller thn the dimension m, the computtionl cost of KSRC is shorter thn SRC. Otherwise, the computtionl cost of KSRC is longer thn SRC. 3. Experiments In this section, the effectiveness of KSRC lgorithm is evluted y experiments. We do experiments on FERET, ORL, Yle nd AR fce dtses nd the PolyU plmprint nd finger-knuckle-print (FKP) dtses. Principl component nlysis (PCA) [17] nd rndom projection (RP) [18] re used for feture extrction. We compre the clssifiction ility of KSRC lgorithm with NM, NN, KERNEL-NN nd SRC lgorithms fter feture extrction. Two populr kernels re involved in our experiments. One is polynomil kernel kðx,yþ¼ð1þx T yþ d nd the other is Gussin kernel kðx,yþ¼ expð :x y: 2 =tþ. For KSRC, we use these two kernels, respectively. Since KERNEL-NN [7] using Gussin kernel is equivlent to NN, only polynomil kernel is used for KERNEL-NN. The optiml kernel prmeters re selected Dt corpor FERET fce dtse The FERET fce dtse [19] ws sponsored y the US Deprtment of Defense through the DARPA Progrm. It hs ecome stndrd dtse for testing nd evluting fce recognition lgorithms. We perform lgorithms on suset of

4 J. Yin et l. / Neurocomputing 77 (2012) the FERET fce dtse. The suset is composed of 1400 imges of 200 individuls, nd ech individul hs seven imges. It involves vritions in fce expression, pose nd illumintion. In the experiment, the fcil portion of the originl imge ws cropped sed on the loction of eyes nd mouth. Then we resized the cropped imges to pixels nd preprocess them y histogrm equliztion. Seven smple imges of one person re shown in Fig ORL fce dtse ORL fce dtse contins 400 fce imges of 40 individuls. The imge size is with 256 gry levels per pixel. The fce imges re centrlized. There re vritions in pose, illumintion nd fcil expression. Fig. 2 shows smple imges of one person Yle fce dtse The Yle fce dtse ws constructed t the Yle Center for Computtionl Vision nd Control. It contins 165 gry-scle imges of 15 individuls. The imges demonstrte vritions in lighting, fcil expression nd with/without glsses. In our experiment, every imge ws mnully cropped nd resized to pixels. Fig. 3 shows 11 imges of one people AR fce dtse The AR fce dtse [20] contins over 4000 color fce imges of 126 people, including 26 frontl views of fces with different fcil expressions, lighting conditions, nd occlusions for ech people. The pictures of 120 individuls were collected in two sessions (14 dys prt) nd ech session contins 13 color imges. Fourteen fce imges (ech session contining 7) of these 120 individuls re selected in our experiment. The imges re converted to gryscle. The fce portion of ech imge is mnully cropped nd normlized to pixels. Fig. 4 shows smple imges of one person. These imges vry s follows: neutrl expression, smiling, ngry, screming, left light on, right light on, ll sides light on PolyU FKP dtse FKP imges on the PolyU FKP dtse were collected from 165 volunteers. These imges re collected in two seprte sessions. In ech session, the suject ws sked to provide six imges for ech of the left index finger, the left middle finger, the right index finger nd the right middle finger. The imges were processed y ROI extrction lgorithm descried in [21]. In the experiment, we select 1200 FKP imges of the right index finger of 100 sujects. These selected imges were resized to pixels nd preprocessed y histogrm equliztion. Fig. 5 shows 12 smple imges of one right index finger PolyU plmprint dtse The PolyU plmprint dtse contins 600 imges of 100 different plms with six smples for ech plm. Six smples from ech of these plms were collected in two sessions, where the first three were cptured in the first session nd the other three in Fig. 1. Smple imges of one person on FERET fce dtse. Fig. 2. Smple imges of one person on ORL fce dtse.

5 124 J. Yin et l. / Neurocomputing 77 (2012) Fig. 3. Smple imges of one person on Yle fce dtse. Fig. 4. Smple imges of one person on AR fce dtse. the second session. The centrl prt of ech originl imge ws utomticlly cropped using the lgorithm mentioned in [22]. The cropped imges were resized to pixels nd preprocessed y histogrm equliztion. Fig. 6 shows six smple imges of one plm Experimentl results On FERET fce dtse, first we try to find the optiml kernel prmeters for KSRC using glol-to-locl serch strtegy [3]. Three imges per person re rndomly selected for trining nd the remining four imges re used for vlidtion. After feture extrction y PCA, the dimension of the smples is fixed t 150. Through glolly serching over wide rnge of the prmeter spce, we find cndidte intervl where the optiml prmeters my exist. Here, for the prmeter d of polynomil kernel, the cndidte intervl is from 1 to 10, for the prmeter t of Gussin kernel, the cndidte intervl is lso from 1 to 10. Now, we try to find the optiml kernel prmeters within these intervls. Fig. 7 shows the recognition rtes of KSRC with polynomil kernel versus the vrition of the prmeter d. Fig. 7 shows the recognition rtes of KSRC with Gussin kernel versus the vrition of the prmeter t. From Fig. 7 nd, we cn see tht the optiml prmeter d is 2 nd the optiml prmeter t is lso 2. After determining the optiml kernel prmeters, we compre KSRC with NM, NN, KERNEL-NN nd SRC. The first three imges per person re used for trining nd the rest four imges re used for testing. Tle 1 shows the mximl recognition rtes of five methods nd the corresponding dimensions nd prmeters. From Tle 1, it cn e seen tht KSRC outperforms other four methods, whether polynomil kernel is used or Gussin kernel is used. In the first experiment on ORL fce dtse, three imges per individul re rndomly chosen for trining nd the remining seven imges re used for testing. PCA is used for feture extrction. The experiment is repeted for 20 times. The first 10 times re used

6 J. Yin et l. / Neurocomputing 77 (2012) for tuning kernel prmeters nd the other 10 times for compring the performnce of NM, NN, KERNEL-NN, SRC nd KSRC. The optiml kernel prmeters re lso determined y glol-to-locl serch strtegy. Fig. 8 shows the verge recognition rtes versus the dimensions. Tle 2 lists the mximl verge recognition rte nd the stndrd devition of ech method cross 10 runs nd the corresponding dimension nd prmeter. From Fig. 8 nd Tle 2, we cn see four min points. First, no mtter which kernel is used, our KSRC consistently outperforms other four lgorithms irrespective of the vrition of dimensions. Second, SRC performs etter thn NM, NN nd KERNEL-NN lgorithms when the dimension is over out 20. Third, KERNEL-NN lmost outperforms NN. Lst, NM hs the worst performnce in this experiment. From the first nd the third points, we cn see tht kernel pproch indeed improve the clssifiction ility. We know SRC hs good performnce for recognition under occlusion. In the second experiment on ORL fce dtse, we test the ility of KSRC for hndling occlusion. For the lst imge of every individul, the region from 30 to 60 in width nd from 30 to 60 in length ws replced y lck lock. Fig. 9 shows the imge of one person under occlusion. We use the first nine imges per person for trining nd the lst imge under occlusion for testing. SRC nd KSRC re used for clssifiction fter PCA trnsformtion. The mximl recognition rtes nd the corresponding dimensions of two clssifiers re given in Tle 3. From Tle 3, it cn e seen tht KSRC outperforms SRC. Especilly for Tle 1 The mximl recognition rtes (percent) of NM, NN, KERNEL-NN, SRC nd KSRC on FERET fce dtse nd the corresponding dimensions nd prmeters. Method NM NN KERNEL-NN SRC KSRC (polynomil) KSRC (Gussin) Recognition rte Dimension Prmeter d ¼ 0:8 d ¼ 2 t ¼ 2 Fig. 5. Smple imges of one right index finger on PolyU FKP dtse Recognition rte NM 0.8 NN KNN 0.78 SRC KSRC(Polynomil) KSRC(Gussin) Dimension Fig. 6. Smple imges of one plm on PolyU plmprint dtse. Fig. 8. The verge recognition rtes of NM, NN, KERNEL-NN, SRC nd KSRC versus the dimensions on ORL fce dtse cross 10 runs. Recognition rte KSRC(Polynomil) prmeter:d Recognition rte KSRC(Gussin) prmeter:t Fig. 7. The recognition rtes of KSRC on FERET fce dtse versus the vrition of the kernel prmeters: () Recognition rtes versus the prmeter d of polynomil kernel nd () recognition rtes versus the prmeter t of Gussin kernel.

7 126 J. Yin et l. / Neurocomputing 77 (2012) Tle 2 The mximl verge recognition rtes (percent) nd stndrd devitions of NM, NN, KERNEL-NN, SRC nd KSRC on ORL fce dtse cross 10 runs nd the corresponding dimensions nd prmeters. Method NM NN KERNEL-NN SRC KSRC (polynomil) KSRC (Gussin) Recognition rte 87:572:8 88:673:0 89:673:0 90:072:7 91:672:8 91:072:8 Dimension Prmeter d ¼ 0:5 d ¼ 2 t ¼ Fig. 9. Smple imge of one person under occlusion on ORL fce dtse. Tle 3 The mximl recognition rtes (percent) of SRC nd KSRC under occlusion on ORL fce dtse nd the corresponding dimensions nd prmeters. Method SRC KSRC (polynomil) KSRC (Gussin) Recognition rte NM NN 0.86 KNN SRC KSRC(Polynomil) 0.84 KSRC(Gussin) Trining smple size Recognition rte Dimension Prmeter d ¼ 2 t ¼ 3 Fig. 10. The mximl verge recognition rtes of NM, NN, KERNEL-NN, SRC nd KSRC versus the vrition of the trining smple size on Yle fce dtse. KSRC with polynomil kernel, its recognition rte is 7.5 percent more thn SRC. Therefore, KSRC is good clssifier to hndle occlusion. On Yle fce dtse, l imges per individul (l vries from 3 to 5) re rndomly selected for trining nd the remining 11 l imges re used for testing. We run the system 20 times. The first 10 times re used for prmeter selection nd the rest 10 times re used for performnce evlution of NM, NN, KERNEL-NN, SRC nd KSRC. Here RP is used for feture extrction. For RP, smples re projected into lower dimensionl feture spce using rndom mtrix whose column hs unit lengths. It hs een found to e sufficiently ccurte method for extrcting feture of high dimensionl dt. The optiml kernel prmeters re required y glol-to-locl serch strtegy. The optiml d of polynomil kernel is set s 5 for KSRC nd 0.3 for KERNEL-NN, respectively. The optiml t of Gussin kernel for KSRC is set s 1. Fig. 10 illustrtes the mximl verge recognition rtes of five methods versus the vrition of trining smple size. Fig. 10 shows tht KSRC still performs est irrespective of the vrition of trining smple size, whether polynomil kernel is used or Gussin kernel is used. SRC performs second est. Moreover, KERNEL-NN outperforms NN nd NM. These re ll consistent with the experiments on FERET nd ORL fce dtses. However, there is lso one inconsistent point tht NM performs etter thn NN in this experiment. For AR fce dtse, the first seven imges, which were tken in the first session, re used for trining while the rest seven tken in the second session re used for testing. For the PolyU FKP dtse, we use the first six FKP imges collected in the first session for trining nd the rest six collected in the second session for testing. For the PolyU plmprint dtse, the first three plmprint imges cptured in the first session re chosen for Tle 4 The mximl recognition rtes (percent) of NM, NN, KERNEL-NN, SRC nd KSRC on AR fce, the PolyU FKP nd the PolyU plmprint dtses nd the corresponding prmeters (in prentheses). trining nd the remining three cptured in the second session for testing. PCA is first performed for feture extrction nd dimension reduction. Then the dimension reduced smples re clssified y NM, NN, KERNEL-NN, SRC nd KSRC seprtely. Tle 4 lists the mximl recognition rtes nd the corresponding dimensions of ech clssifiction method on three dtses. From Tle 4, we cn see tht SRC nd KSRC still outperform other three clssifiers nd KSRC performs etter thn SRC. This demonstrtes tht KSRC is more effective thn SRC for clssifiction gin. 4. Conclusions NM NN KERNEL-NN SRC KSRC KSRC (polynomil) (Gussin) AR (d ¼ 0:5) PolyU FKP (d ¼ 0:5) PolyU plmprint (d ¼ 0:8) (d ¼ 2) 73.7 (t ¼ 5) (d ¼ 1) 71.8 (t ¼ 9) (d ¼ 1) 96.3 (t ¼ 3) SRC pplies sprse representtion coefficient to clssifiction. Sprse representtion coefficient contins very importnt discriminting informtion, so SRC hs more powerful discriminting ility thn clssifiction methods such s NM, NN nd KERNEL-NN. In this pper, we develop kernel sprse representtion sed clssifiction (KSRC) lgorithm. For KSRC, smples re mpped from originl

8 J. Yin et l. / Neurocomputing 77 (2012) feture spce into high dimensionl feture spce first, nd then SRC is performed in the high dimensionl feture spce. Although the explicit smples in the high dimensionl feture spce re unknown, we prove tht SRC could e implemented successfully using kernel function. If n pproprite kernel is utilized, sprse representtion coefficient of the test smple in the high dimensionl feture spce will reflect its lel informtion more ccurtely. Nmely, sprse representtion coefficient in the high dimensionl feture spce contins more effective discriminting informtion thn sprse representtion coefficient in the originl feture spce. Hence, KSRC could otin higher recognition rte thn SRC. Experimentl results on FERET, ORL, Yle nd AR fce dtses nd the PolyU plmprint nd FKP dtses indicte the effectiveness of KSRC. For smples contining smll noise, SRC nd KSRC hve the sme computtionl cost. For smples contining ig noise, if the numer of trining smple size is smller thn the dimension of smple, KSRC is more efficient thn SRC. Contrrily, SRC is more efficient thn KSRC. Besides, when performing KSRC, we should find the optiml prmeters. This process will increse its computtionl cost. Acknowledgments This work is supported y the Ntionl Science Foundtion of Chin under Grnt nos , , nd Appendix A The Proof of Theorem 1. Let GðÞ¼ðB fðyþþ T ðb fðyþþ The derivtive of GðÞ with respect to the vrile is G 0 ðþ¼2b T B 2B T fðyþ Since GðÞZ0, GðÞ chieves the minimum vlue when B T B B T fðyþ¼0 If B T B B T fðyþ is close to 0, GðÞ will pproch its minimum vlue nd qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi p :B fðyþ: 2 ¼ ðb fðyþþ T ðb fðyþþ ¼ GðÞ ffiffiffiffiffiffiffiffiffiffi Society Workshop Neurl Networks for Signl Processing, vol. IX, 1999, pp [6] S. Mike, G. Rätsch, B. Schölkopf, A. Smol, J. Weston, K.R. Müller, Invrint feture extrction nd clssifiction in kernel spces, in: Proceedings of the 13th Annul Neurl Informtion Processing Systems Conference, 1999, pp [7] K. Yu, L. Ji, X.G. Zhng, Kernel nerest-neighor lgorithm, Neurl Process. Lett. 15 (2002) [8] J. Yng, J. Wright, T. Hung, Y. M, Imge super-resolution s sprse representtion of rw ptches, in: Proceedings of the IEEE Interntionl Conference on Computer Vision nd Pttern Recognition, [9] S. Ro, R. Tron, R. Vidl, Y. M, Motion segmenttion vi roust suspce seprtion in the presence of outlying, incomplete, nd corrupted trjectories, in: Proceedings of the IEEE Interntionl Conference on Computer Vision nd Pttern Recognition, [10] J. Mirl, G. Spiro, M. Eld, Lerning multiscle sprse representtions for imge nd video restortion, SIAM MMS 7 (1) (2008) [11] J. Wright, A.Y. Yng, A. Gnesh, S.S. Sstry, Y. M, Roust fce recognition vi sprse representtion, IEEE Trns. Pttern Anl. Mch. Intell. 31 (2) (2009) [12] E. Amldi, V. Knn, On the pproximility of minimizing nonzero vriles or unstisfied reltions in liner systems, Theor. Comput. Sci. 209 (1998) [13] E. Cnde, J. Romerg, T. To, Stle signl recovery from incomplete nd inccurte mesurements, Commun. Pure Appl. Mth. 59 (8) (2006) [14] D. Donoho, For most lrge underdetermined systems of liner equtions the miniml l 1 -norm solution is lso the sprest solution, Commun. Pure Appl. Mth. 59 (6) (2006) [15] E. Cnde, T. To, Nerest-optiml signl recovery from rndom projections: universl encoding strtegies, IEEE Trns. Inf. Theory 52 (12) (2006) [16] S. Chen, D. Donoho, M. Sunders, Atomic decomposition y sis pursuit, SIAM Rev. 43 (1) (2001) [17] M. Turk, A.P. Pentlnd, Fce recognition using eigenfces, in: Proceeding of the IEEE interntionl Conference on Computer Vision nd Pttern Recognition, 1991, pp [18] E. Binhm, H. Mnnil, Rndom projection in dimension reduction: ppliction to imge nd text dt, in: Proceeding of the Seventh ACM SIGKDD Interntionl Conference on Knowledge Discovery nd Dt Mining, 2001, pp [19] P.J. Phillips, H. Moon, S.A. Rizvi, P.,.J. Russ, The FERET evlution methodology for fce-recognition lgorithms, IEEE Trns. Pttern Anl. Mch. Intell. 22 (10) (2000) [20] A. Mrtinez, R. Benvente, The AR fce dtse, CVC Tech. Rep. 24 (1998). [21] L. Zhng, L. Zhng, D. Zhng, H.L. Zhu, On-line finger-knuckle-print verifiction for personl uthentiction, Pttern Recognition 43 (7) (2010) [22] D. Zhng, Plmprint Authentiction, Kluwer Acdemic, [23] D. Donoho, Y. Tsig, Fst solution of l 1 -norm minimiztion prolems when the solution my e sprse, Preprint, / reserch.htmls, Jun Yin received BS degree in Mthemtics nd PhD degree in Pttern Recognition nd Intelligence System from Nnjing University of Science nd Technology in 2006 nd 2011, respectively. His current reserch interest includes pttern recognition, fce recognition nd mchine lerning. will lso pproch its minimum vlue. Therefore, for ny ez0, there must exist dz0 such tht if :B T B B T fðyþ: 2 rd :B fðyþ: 2 re will e stisfied. & References [1] A.K. Jin, P.W. Duin, J.C. Mo, Sttisticl pttern recogniton: review, IEEE Trns. Pttern Anl. Mch. Intell. 22 (1) (2000) [2] S.B. Kotsintis, Supervised mchine lerning: review of clssifiction techniques, Informtic 31 (2007) [3] K.R. Müller, S. Mike, G. Rätsch, K. Tsud, B. Schölkopf, An introduction to kernel-sed lerning lgorithms, IEEE Trns. Neurl Networks 12 (2) (2001) [4] B. Schölkopf, A. Smol, K.R. Müller, Nonliner component nlysis s kernel eigenvlue prolem, Neurl Comput. 10 (5) (1998) [5] S. Mike, G. Rätsch, J. Weston, B. Schölkopf, K.R. Müller, Fisher discriminnt nlysis with kernels, in: Proceedings of the 1999 IEEE Signl Processing Zhonghu Liu received BS degree in Computer Engineering from the First Aeronuticl Institute of the Air Force, MS degree in Computer Softwre nd Theory from Xihu University nd PhD degree in Pttern Recognition nd Intelligence System from Nnjing University of Science nd Technology in 1998, 2005 nd 2011, respectively. His current reserch interest includes pttern recognition, fce recognition nd imge processing.

9 128 J. Yin et l. / Neurocomputing 77 (2012) Zhong Jin received BS degree in Mthemtics, MS degree in Applied Mthemtics nd PhD degree in Pttern Recognition nd Intelligence System from Nnjing University of Science nd Technology in 1982, 1984 nd 1999, respectively. Now he is professor in the Deprtment of Computer Science nd Technology t Nnjing University of Science nd Technology. His current interest includes pttern recognition, imge processing nd fce recognition. Wnkou Yng received BS degree in Computer Science nd Technology, MS degree nd PhD degree in Pttern Recognition nd Intelligence System from Nnjing University of Science nd Technology in 2002, 2004 nd 2009, respectively. Now he is Postdoctorl Fellow in the school of utomtion t Southest University. His reserch interest includes pttern recognition, computer vision nd digitl imge processing.