A Semantic Model for Video Based Face Recognition

Transcription

1 Proceedng of the IEEE Internatonal Conference on Inforaton and Autoaton Ynchuan, Chna, August 2013 A Seantc Model for Vdeo Based Face Recognton Dhong Gong, Ka Zhu, Zhfeng L, and Yu Qao Shenzhen Key Lab of Coputer Vson and Pattern Recognton Shenzhen Insttutes of Advanced echnology, Chnese Acadey of Scences he Chnese Unversty of Hong Kong, Hong Kong SAR {dh.gong, ka.zhu, zhfeng.l, yu.qao}@sat.ac.cn Abstract - Vdeo-based face recognton has attracted a great deal of attenton n recent years due to ts wde applcatons. he challenge of vdeo-based face recognton coes fro several aspects. Frst, vdeo data nvolves any fraes, whch ncreases data sze and processng coplexty. Second, key fraes extracted fro vdeos are usually of hgh ntra-personal dscrepancy due to varatons n expressons, poses, and llunatons. In order to address these probles, we propose a novel seantc based subspace odel to prove the perforance of vdeo based face recognton. he basc dea s to construct an approprate low-densonal subspace for each person, upon whch a seantc odel s bult to classfy the key fraes of the person nto specfc class. After the seantc classfcaton, the key fraes belongng to the sae classes,.e. the sae seantcs, are used to tran the lnear classfers for recognton. Extensve experents on a large face vdeo database (XM2VS) clearly show that our approach obtans a sgnfcant perforance proveent over the tradtonal approaches. Index ers - face recognton, vdeo based face recognton, seantc odel. 1. INRODUCION Vdeo-based face recognton has attracted a great deal of research nterests n recent years. he ajor advantage of vdeo-based face recognton over age-based face recognton les n the fact that a vdeo sequence contans uch ore nforaton than n a stll age. Proper utlzaton of such addtonal nforaton ay help prove face recognton perforance. A ajor research challenge n vdeo-based face recognton s that vdeo data often nvolves any fraes, leadng to the probles of large data sze and hgh processng coplexty. A popular way to solve ths proble s to extract a sall set of key fraes for copact feature representaton, and then apply subspace analyss technques to learn a dscrnant subspace for effectve atchng. ypcal subspace analyss ethods nclude prncpal coponent analyss (PCA) [1], lnear dscrnant analyss (LDA) [3], and soe proveents of LDA [9][10][12]. Recently, several effectve ethods [11][13][14][15] are further proposed to address the large data sze proble and prove the recognton perforance n vdeo based face recognton. However, a ltaton wth these ethods les n the fact that the key fraes extracted fro vdeos are usually of hgh dscrepancy due to dfferent llunaton, poses and expressons whch we refer to as seantc varatons. Such varatons ntroduce great ntra-person dscrepancy and thus lead to perforance degradaton. In order to address ths proble and prove the recognton perforance, we propose a seantc based approach for vdeo based face recognton. he prary dea s to classfy the key fraes extracted fro vdeo streas nto approprate seantcs. he key fraes of the sae seantcs are then used to tran the classfers. In atchng process, the probe key fraes are also atched seantcally to reduce potental satch. We nvestgated the effectveness of ths ethod by experentaton wth the XM2VS vdeo face database [4]. Fro the experents, we can clearly see that our new approach s able to obtan sgnfcant perforance proveent over the state-of-the-art algorths. 2. Related Work hs secton descrbes soe prevous works n face feature representaton and subspace analyss that are related to our proposed fraework Local Bnary Patterns (LBP) Aong the exstng descrptors for face recognton, local bnary patterns (LBP) [8] s one of the bestperforng ones [16]. It has been shown to be very successful n face feature representaton [8]. he orgnal LBP operator assgns a label to each age pxel by thresholdng the 3x3 neghborhood of /13/$ IEEE 1369

2 each pxel wth the center pxel value and consderng the result as a bnary nuber. hen the hstogra of the labels s used as a texture descrptor for feature representaton. In face recognton, the basc process for LBP-based feature representaton s to use unfor bnary patterns for face texture descrpton. Each unfor bnary pattern s labeled wth a separate label whle all the non-unfor bnary patterns are labeled wth a sngle label, whch yelds n 59 dfferent labels. hen the hstogra of these 59 labels s used as a texture descrptor to descrbe the cro-structure of the facal ages. o deal wth the textures at dfferent scales, the LBP operator can be extended to use ultple scales. In our study, we use the extended LBP to descrbe the face at ultple scales, by coputng the LBP descrptors coputed at four dfferent rad {1, 3, 5, 7} Lnear Dscrnant Analyss (LDA) LDA s one of the ost popular subspace analyss ethods for face recognton [3][10][12][17]. he basc dea of LDA s to use the wthn-class scatter atrx and the between-class scatter atrx to defne a crteron functon to easure the class separablty. he wthn-class and between-class scatter atrces are defned as S c w = ( X j μ )( X j μ ) (1) = 1 X j C c S b = ( μ μ )( μ μ ), (2) = 1 where μ denotes the ean of the class C, μ denotes the overall ean, and c denotes the nuber of classes. he objectve of LDA s to obtan the optal projecton W opt, whch axzes the rato of the deternant of the between-class atrx to that of the wthn-class atrx, defned as: W SbW. Wopt = argax (3) W W Sw W Matheatcally, t s equvalent to coputng the leadng egenvectors of S w 1 S b. 3. Our approach 3.1. Feature Representaton Our approach begns by extractng a sall set of key fraes fro each vdeo sequence by eans of the spato-teporal synchronzaton ethod, as developed n our prevous work [5][11]. he spatoteporal synchronzaton ethod uses the wavefor of the audo sgnal to allocate desred fraes n each vdeo for further analyss. he objectve of ths key frae extracton procedure s to locate a sall set of dstnct vdeo fraes to represent the characterstcs of the vdeo sequence. In our experents, 21 key vdeo fraes are extracted fro each vdeo sequence usng the spato-teporal synchronzaton ethod. In order to cobat llunaton varatons, all the key fraes are fltered by DoG flters wth σ 1 = 1 and σ 2 = 2, and turned nto Local Bnary Pattern (LBP) [8] representatons. hese key fraes need to be cobned for subsequent analyss and classfcaton of the vdeo. A straghtforward approach s to aggregate C1,1 C1,2 C1,3 K-eans PCA+LDA C2,1 C2,2 C2,3... ranng Cn,1 Cn,2 Cn,3 estng Seantc Classfcaton Subspace Projecton Matchng Score 1370

3 Fgure 1. he llustraton for the ppelne of the proposed syste. At the tranng stage, tranng faces fro the sae person are frst clustered seantcally wth the K-eans algorth, and then the clustered faces fro dfferent persons are further grouped together accordng to ther seantcs. Fnally, face features fro each group are slced nto n slces, and PCA+LDA algorth s appled ndependent on these slces, yeldng n*3 classfers. At the testng stage, the atchng score of the gven par of testng vdeo sequences s coputed by frst seantcally assgnng each ndvdual frae nto one of the groups at the tranng stage, followed by the coputaton of the atchng score n the accordng subspace. he fnal atchng score s the su of the best atchng scores fro each group. the key vdeo fraes nto a sngle large vector, and then conduct regular subspace analyss for feature extracton. Although ths approach of feature level fuson utlzes all the data n vdeo, there are an overly large nuber of feature densons. he hgh densonalty leads to costly coputatons and overfttng probles. hese ssues are coon n facal age recognton, but are vastly aggravated n vdeo-based face recognton. For the rest of ths secton, we ntroduce a seantc odel for vdeo based face recognton. 3.2 Seantc Classfcaton Suppose we have a set of features fro tranng key fraes of N dfferent persons χ = { X = 1,..., N}, j X = x j = 1,..., K (K=21*4 n our syste where { } snce we use all the four sectons as tranng and n each secton there are 21 tranng fraes for each person) s the tranng features of the -th person (each person has K tranng features here). Snce the features of the key fraes fro the sae person should be contaned wthn a low-densonal subspace, we can construct a subspace wth PCA [6] to captures the an varaton for that person. he projecton n the subspace s coputed by j j y = W x μ (4) ( ) Where μ s the ean of the -th tranng person, and W s the projecton atrx whose coluns consst of egenvectors of the covarance atrx of the -th tranng person. Snce the key fraes of the tranng person can contan dfferent expressons and poses, whch we refer to as dfferent seantcs, we partton the tranng features of a person nto M groups (M s eprcally set to be 3 n our syste), wth each correspondng to one seantc. he partton of the tranng features s based on the K-eans [7] algorth, descrbed as below. a) For the -th person, we convert ts tranng features X nto subspace representaton wth equaton (4), j Y = W x μ j = 1,..., K. resultng n { ( ) } b) Apply the K-eans algorth on Y, clusterng the K saples nto M groups. hen the X s dvded nto M groups accordng to the partton schee of the Y, the resultng M groups of tranng features are denoted as: ( ) j X = x j C, = 1,.., M (5) ( ) { } where C s the -th cluster of the -th person. hus, the orgnal tranng set χ s seantcally dvded nto M tranng sets: χ = { X = 1,..., N}, = 1,.., M (6) One thng to note s that the clustered faces fro dfferent persons are further grouped together accordng to ther seantcs,.e. accordng to the Eucldean dstances between the clusters. hus, for each group of face features, t conssts of features fro dfferent persons, but they are sharng the sae seantc,.e. the sae pose or expresson. 3.3 Learnng Seantc-based Classfers Wth the ntroduced seantc classfcaton ethod, the tranng set χ s seantcally parttoned nto M tranng subsets accordng to (6). hen we learn the classfers on these M tranng subsets ndependently. For each of the tranng subset, otvated by the success of local feature dscrnant analyss (LFDA) [9], we frst slce each tranng feature evenly nto P slces (P s set to be 10 eprcally n our syste), followed by the PCA+LDA [10] ethod whch frst apples PCA on the tranng data to reove redundant nforaton or noses, and then uses LDA to construct a subspace where the ntra-personal varatons of the tranng features can be nzed whle the nter-personal varatons can be axzed. he densons of PCA and LDA are set to be 120 and 100, respectvely. In ths way, we have M dfferent tranng subsets and for each of the we can obtan P (the nuber of slces) PCA+LDA classfers. So we have totally (M*P) ndependent classfers. 3.4 Matchng Seantcally 1371

4 At ths pont, we have (M*P) classfers traned, and the atchng echans based on the proposed fraework s ntroduced n ths subsecton. In the atchng process, we have L unseen persons n the gallery set and each person has 21 key fraes extracted fro one vdeo sequence. Our task s to atch a vdeo sequence of a person (probe) to the gallery ones. For the probe vdeo sequence, we also extract 21 key fraes usng the sae technques n the tranng process. he atchng score between the probe person and the -th gallery person s obtaned as follows: a) Assgn each face feature f to one of the M seantc groups accordng to ther Eucldean dstances to the eans of the groups: where k = arg n h c = 1.. M c s the ean n the PCA subspace of the - th group for the tranng features, and h s the subspace representaton for the feature f. b) After the seantc assgnent, the gallery and probe face features are scattered nto M groups. For the -th group, convert the features g and p to subspace representatons usng the sae approach n Secton 3.3 (frst slcng, followed by PCA+LDA), and then copute the par-wse atchng scores fro p to g n the subspace. he atchng score wth regards to the -th group wll be the hghest parwse atchng scores (the hgher atchng score ndcates the better atchng). Fnally, the fnal atchng score fro the probe person to the -th gallery person wll be the su of the atchng scores fro the M groups. 4. Experent We use the XM2VS vdeo database for experental evaluaton. hs database contans face vdeos fro 295 dfferent persons wth each one havng four vdeo sequences captured n dfferent te. A detaled descrpton can be found n [4]. In ths secton, we conduct extensve experents on the XM2VS face vdeo dataset. In our experents, we use all the 295*4 vdeo sequences of 295 dstnct persons fro the four dfferent sessons. he persons n the vdeo are asked to read two sequence of nubers, and Fgure 2 shows one person s 20 age saples. Fgure saples fro a person's vdeo sequence. Fro each vdeo sequence, 21 fraes are selected usng the Audo-Vdeo eporal Synchronzaton ethod [11]. Each frae corresponds to the wavefor peak of a dgt, as llustrated n Fgure 3. An addtonal frae s located at the dpont of the end of the frst sentence and the start of the second sentence. (a) Audo Sgnal one two three four fve sx (b) Face Vdeo Fraes Fgure 3. Exaple vdeo sequence and correspondng speech sgnal. o better evaluate the recognton perforance wth geoetrc and photoetrc nterference fltered out, we preprocess the face ages through the followng steps: Scale the face age so that the dstance between two eyes s a constant. Crop the face fro the orgnal face age accordng to the locaton of the dpont of the two eyes. Perfor hstogra equalzaton on the cropped age. After preprocessng, each face age s noralzed and algned by sze. Fgure 4 shows the noralzed 1372

5 saples of the face ages n Fgure 2. By coparng the noralzed face ages shown n Fgure 4 wth the orgnal face ages shown n Fgure 2, we can see that after preprocessng only the regon whch contans the face nforaton are reaned n the noralzed face age, whch not only lowers the denson of the noralzed face vector but also reduces the large ntra-personal varatons to soe extent. Fgure noralzed saples of one person fro XM2VS database. In our experents, the face vdeo dataset s dvded nto two parts: 145 persons are used for tranng, and the rest 150 persons are used for testng. here s no overlap between the tranng people and the testng people. In the tranng process, the four sectons of 145 persons are used as the tranng set; and n the testng process, the frst secton of the other 150 testng person (unseen) s used as gallery data, and the rest three sectons of the correspondng 150 persons (450 saples) are used as probe set. We frst copare our approach aganst the exstng face atchng algorths desgned for large data sze. he coparson results for both face dentfcaton and verfcaton are shown n able 1 and able 2. Fro these results, we can see that our approach has a notable proveent over the exstng algorths n both face dentfcaton and verfcaton tasks. hs shows the effectveness of the proposed approach n handlng the large data sze proble. able 1. Identfcaton Perforance Coparson Algorths Rank-1 recognton accuracy PCA [1] 68.63% LDA [3] 87.56% Unfed Subspace Analyss [16] 89.19% NDA [18] 90.33% Mult-level Subspace Analyss [17] 91.93% LFDA [15] 92.16% Our Approach 95.33% able 2. Verfcaton Perforance Coparson Algorths Verfcaton FAR=0.1% PCA [1] 73.25% LDA [3] 89.13% Unfed Subspace Analyss [16] 92.03% NDA [18] 92.20% Mult-level Subspace Analyss [17] 94.09% LFDA [15] 94.11% Our Approach 97.82% In the second experent, we copare our approach wth several newly developed vdeo based face recognton n the lterature. he coparatve results are reported n able 3. Fro these results we can see that our approach also has superor perforance over the exstng ethods. hs shows the effectveness of the proposed ethod n addressng the vdeo based face recognton proble. able 3. Coparson of our approach wth the exstng ethods n the lterature. Algorths Rank-1 recognton accuracy [17] 91.93% [19] 91.97% [20] 93.52% [21] 92.27% Our approach 95.33% 5. Concluson hs paper presents a seantc approach for vdeobased face recognton. By clusterng vdeo key fraes nto seantc groups, we show that the proposed algorth acheves a great proveent over the tradtonal face recognton approaches whch treat all the key fraes equvalently. In the future work, we would lke to further prove the groupng accuracy by applyng a supervsed learnng approach to classfy the key fraes. ACKNOWLEDGEMEN hs work was supported by grants fro Natonal Natural Scence Foundaton of Chna ( , ), Shenzhen Basc Research Progra 1373

6 (JC A, JCYJ , JCYJ ), and Guangdong Innovatve Research ea Progra (No D ). REFERENCE [1] M. urk and A. Pentland, "Face recognton usng egenfaces", CVPR [2] K. Fukunnaga, Introducton to statstcal pattern recognton, Acadec Press, [3] V. Belhueur, J. Hespanda, and D. Kregean, Egenfaces vs. Fsherfaces: Recognton Usng Class Specfc Lnear Projecton, IEEE rans. on PAMI, Vol. 19, No. 7, pp , July [4] K. Messer, J. Matas, J. Kttler, J. Luettn, and G. Mattre, XM2VSDB: he Extended M2VS Database, Second Internatonal Conference on AVBPA, March [5] X. ang and Z. L, Frae synchronzaton and ult-level subspace analyss for vdeo based face recognton, Proc. CVPR, June [6] Moore B., Prncpal coponent analyss n lnear systes: Controllablty, observablty, and odel reducton, IEEE ransactons on Autoatc Control, Page(s): 17-32, [7] Krshna K., Genetc K-eans algorth, IEEE ransactons on Systes, Man, and Cybernetcs, Page(s): , [8]. Ahonen, A. Hadd and M. Petkanen, Face Recognton wth Local Bnary Patterns, Proceedngs of ECCV, Page(s): , [9] B. Klare, Z. L and A. K. Jan, Matchng forensc sketches to ugshot photos, IEEE ransactons on Pattern Analyss and Machne Intellgence, Page(s): , [10] X. Wang and X. ang. A unfed fraework for subspace face recognton. IEEE PAMI, 26(9): , [11] X. ang and Z. L, "Audo-guded Vdeo Based Face Recognton", IEEE rans. on CSV, [12] Z. L, D. Ln, and X. ang, "Nonparaetrc Dscrnant Analyss for Face Recognton," IEEE rans. on PAMI, vol. 31, no. 4, pp , [13] H. Cevkalp and B. rggs, "Face recognton based on age sets," n CVPR [14] Y. Chen, V. Patel, S. Shekhar, R. Chellappa, and P. Phllps, "Vdeo-based Face Recognton va Jont Sparse Representaton," FG, [15] M. Wbowo, D. jondronegoro, and V. Chandran, "Probablstc Matchng of Iage Sets for Vdeo- Based Face Recognton", Internatonal Conference on DICA, Dec [16] K. Mkolajczyk and C. Schd, A perforance evaluaton of local descrptors, IEEE rans. Pattern Analyss and Machne Intellgence, vol. 27, no. 10, pp , Oct [17] D. L. Swets and J. Weng, Usng dscrnant egenfeatures for age retreval, IEEE ransactons on Pattern Analyss and Machne Intellgence, vol. 18, pp , Aug