Real-Time Multi-View Face Detection

Transcription

1 Real-Time Multi-View Face Detectin ZhenQiu Zhang 1*, Lng Zhu 2, Stan Z. Li 2, HngJiang Zhang 2 1. Institute f Autmatin, Chinese Academy f Science, Beijing, China 2. Micrsft Research Asia, Beijing Sigma Center, Beijing , China Abstract In this paper, we present a detectr-pyramid architecture fr real-time multi-view face detectin. Using a carse t fine strategy, the full view is partitined int finer and finer views. Each face detectr in the pyramid detects faces f its respective view range. Its training is perfrmed by using a new meta bting learning algrithm. This results in the first real-time multi-view face detectin system which runs at 5 frames per secnd fr 320x240 image sequence. 1. Intrductin Statistics shw that apprximately 75% f the faces in hme phts are nn-frntal [1], and therefre ability t deal with multi-view faces is imprtant fr many face-related applicatins. Multi-view face detectin has been a challenging prblem. The challenge is firstly due t large amunt f variatin and cmplexity brught abut by the changes in facial appearance, lighting and expressin [17]. Changes in facial view (pse) further cmplicate the situatin because the distributin f multi-view faces in a feature space is mre dispersed and mre cmplicated than that f frntal faces. The learning based apprach has been mst effective fr face detectin. Sung and Pggi [14] divided the frntal face image space and nn-face image space each int several prbability clusters. PCA is perfrmed n each cluster s that face/nn-face classificatin is perfrmed in terms f bth the Mahalanbis distance frm the cluster center in the PCA space and the Euclidean distance frm the PCA space are used as features. Rwley et al [9] presented a face detectin system based n retinally cnnected neural netwrks. The input t the NN is the preprcessed image pixel values directly. Pst-prcessing f the neural netwrks are perfrmed by either ANDing/ORing the utputs r using an additinal neural netwrk t arbitrate between the utputs. Osuna el al [5] applied the supprt vectr machines algrithm t train an NN t classify face and nn-face patterns. Yang el al [3] uses a netwrk f linear units. The SNW learning architecture is specifically tailred fr learning in the presence f a very large number f features. * The wrk presented in the paper was carried ut at Micrsft Research Asia. Recently, Vila and Jnes [19] prpse a very fast apprach fr frntal face detectin. Simple Haar-like feature are extracted, face/nn-face classificatin is dne by using a cascade f successively mre cmplex classifiers which are trained by using AdaBst [24] learning algrithm. The cascade structure is supprted by an argument made in [4] that cascading classifiers is a better apprach than multiexpert methds like vting and stacking. Over past years, prgress has been made fr nn-frntal faces detectin and recgnitin. Feraud et al [11] adpt the view-based representatin fr face detectin, and use an array f 5 detectrs with each detectr respnsible fr ne view. Wisktt et al [16] build elastic bunch graph templates fr multi-view face detectin and recgnitin. Gng and clleagues [21] study the trajectries f faces in linear PCA feature spaces as they rtate, and use kernel supprt vectr machines (SVMs) fr multi-pse face detectin and pse estimatin. Huang et al [13] use an SVM t classify three facial pses at , 0, degrees. T deal with cmplexity due t multi-view, a natural treatment is t divide face images int several subsets accrding t the facial view and mdel each view subspace respectively [2], by which explicit 3D mdeling is avided. The system f Schneiderman and Kanade is claimed t be the first ne in the wrld fr multi-view face detectin [10]. The algrithm cnsists f an array f 5 face detectrs each f which is specialized fr a specific pse f face and accmmdates small amunt f variatin arund the designated pse. A detectr classifies a sub-windw int face/nn-face based n statistics f prducts f histgrams cmputed frm examples f the respective view. The results frm all the detectrs are merged such that they are spatially cnsistent. The detectr is claimed t be the first algrithm in the wrld fr multi-view face detectin. Hwever, it is very slw and takes 1 min t wrk n a 320x240 image ver nly 4 ctaves f candidate size [10]. In this paper, we present a nvel framewrk fr real-time multi-view face detectin. A detectr-pyramid architecture is designed t detect multi-view faces efficiently. The detectrpyramid adpts an integrated strategy f carse-t-fine view decmpsitin [18,19], and simple-t-cmplex face/nnface classificatin Vila and Jnes [19]; a sub-windw is prcessed frm the tp t bttm f the pyramid by a sequence f increasingly mre cmplex face/nn-face classifiers designed fr increasingly finer ranges f facial view. The detectr-pyramid ges beynd the straightfrward view decmpsitin methd [2] in that using the carse-tfine and simple-t-cmplex strategy, a vast number f

2 nnface sub-windws can be discarded very quickly with very little lse f face sub-windws. This is very imprtant fr fast face detectin because nly a tiny prprtin f subwindws are f faces. We devise simple image features fr efficient face/nnface classificatin. These features are extensins f thse used in [19] fr frntal face detectin in that the frmer is mre suitable t cater t nn-symmetry f nnfrntal faces. Every detectr in the pyramid is learned frm face/nnface examples using a new learning algrithm called FlatBst [22]. FlatBst incrprates the idea f Flating Search [18] int AdaBst t slve the nnmntnicity prblem encuntered in the sequential algrithm f AdaBst. While the Vila-Jnes detectr [19] is the first real-time frntal face detectr and Schneiderman-Kanade detectr is the first (nn real-time) multi-view face detectr, the algrithm presented in this paper results in the first real-time multi-view face detectin system which runs at 5 frames per secnd fr 320x240 image sequence n a cnventinal 700 MHz Pentium III PC. The rest f the paper is rganized as fllws: Sectin 2 intrduces the detectr-pyramid architecture fr multi-view face detectin. The design and training f individual detectr are presented in sectin 3 and 4. Methd t arbitrate amng nine view channels is presented in sectin 5.Sectin 6 prvides the experimental results and cnclusin is drawn in sectin Detectr-Pyramid Architecture The present multi-view face detectin system is distinguished frm previus systems in its ability t detect multi-view faces in real-time. It is designed based n the fllwing thughts: While it is extremely difficult t distinguish multi-view faces frm nn-face images clearly using a single classifier, it is less difficult t classify between frntal faces and nn-faces and als less difficult t d between multi-view faces and part f nn-faces. Therefre, narrwing dwn the range f view will make face detectin easier and mre accurate fr that view. On the ther hand, a vast number f sub-windws (e.g. 70,401 square sub-windws can result frm the scan f a 320x240 image, frm the size f 20x20 pixels t 240x240 fr the size increment factr f 1.25) result frm scan f the input image; amng these nly a tiny prprtin (say, up t a few dzens) f them are faces. It can save the cmputatin tremendusly if a sequence f detectrs f increasing cmplexity and face/nn-face discriminating pwer are applied t quickly discard nn-faces at the earliest pssible stage using the simplest pssible features. The detectr-pyramid architecture (see Figure 1) is mtivated by the abve reasns. It adpts the carse t fine (tp-dwn in the pyramid) strategy [18,19] in that the full range f facial view is partitined int increasingly narrwer ranges, and thereby the whle face space is partitined int increasingly smaller subspaces. Als it adpts the simple-tcmplex strategy (Vila-Jnes detectr [19]) in that the earlier nes are simpler and s are able t reject a vast number f nn-face sub-windws q1uickly whereas the nes in the later stage are mre cmplex and invlved and spend mre time t scrutinize nly a relatively tiny number f remaining sub-windws. Figure 1: Detectr-pyramid. Our current implementatin cnsists f three levels. The first level cnsists f a single detectr, respnsible fr the full range f [-90,90] degree (0 degree being the frntal; view). There are three detectrs in the secnd level, respnsible fr the three view ranges [-90, -40], [-30,+30], [+40,+90], respectively. The third level cnsists f 9 detectrs f [-90, -80],[-70,-60],, [60,70], [80,90] degrees. Therefre, there are a ttal f 13 detectrs. Fr a sub-windw, if it is rejected by detectr at the tp level, it will be seen as nn-face regin and will nt be prcessed by later levels. If it ges thrugh first level, it will be prcessed by secnd level. If any detectr in secnd level classifies it as face, it will be prcessed by last level, r it will be rejected as nn-face. There are much mre detectrs n the bttm f ur framewrk, and it help us fcus ur attentin n thse pssible face regin, while paying much less time n impssible face regin. At the last level, each detectr nly dues with 20 degree ranges f view and each detectr has high detectin rate fr that view. This pyramidlike framewrk makes ur system have bth high detectin rate and rapid detectin speed fr multi-view face detectin. The full-view detectr in the implementatin is able t reject abut 50% f nn-face sub-windws scanned in the perfrming stage, while retaining 99% f training face examples in the training stage. Only retained sub-windws pssibly cntaining faces are further prcessed in the subsequent levels f finer detectrs. The results frm the detectrs in the bttm level are merged t make a final decisin regarding the input sub-windw. 3. Design f Individual Detectrs The high speed and detectin rate f the algrithm depend nt nly n the detectr-pyramid architecture but als individual detectr. Each detectr classifies a subwindw int face/nn-face. Tw types f simple features,

3 which are blck differences similar t steerable filters, are cmputed as shwn in Figure 2. Each such feature has a scalar value which can be cmputed very efficiently frm the summed-area table [6] r integral image [19]. These features are nn-symmetrical t cater t nnsymmetrical characteristics f nn-frntal faces. They have mre degrees f freedm than thse f [19] in their cnfiguratins: 6 (x, y, delta x, delta y, dx, dy) in the tw blck features and 7 (x, y, delta x, delta y, dx, dx, dy) in the three blck features. There are a ttal number f 102,979 tw-blck features fr a sub-windw f size 20x20 pixels. There are a ttal number f 188,366 three-blck features (with sme restrict t their freedm). x -2 dx y dy +2 Figure 2: The tw types f simple Harr wavelet like features defined in a sub-windw. The rectangles are f size x by y and are at distances f (dx,dy) apart. Each feature takes a value calculated by the weighted +-1,2 sum f the pixels in the rectangles. A face/nnface strng classifier is cnstructed based n a number f weak classifiers where a weak classifier perfrms face/nn-face classificatin using a different single feature, e.g. by threshlding the scalar value f the feature accrding the face/nn-face histgrams f the feature. A detectr can be ne r a cascade f such face/nnface strng classifiers, as in [19]. 4. Training f Individual Detectrs Hw t chse a gd cmbinatin f weak classifiers frm tens f thusands f features t cnstruct a pwerful detectr is a challenging prblem f feature selectin [8][15] and classifier design. We have devised a new bsting algrithm, called FlatBst [22], fr learning face detectrs fr the detectr-pyramid. Similar r better perfrmance than AdaBst is achieved with fewer weak classifiers. +1 dx' dx dy -2 y +1 x The detectrs in the pyramid are trained separately, using different training sets. An individual detectr is respnsible fr ne view, with pssible partial verlapping with its neighbring detectrs. Due t the symmetry f faces, we need t train side view detectrs fr ne-side nly, and mirrr the trained mdels fr the ther side. Fr ne feature used in left-side view, we mirrr its structure (See Figure 3) t cnstruct a new feature used fr right-side view. Each left-side view feature is mirrred by this way, and these new features are cmbined t cnstruct right side view detectrs. 5. Arbitrate amng Individual Outputs In ur framewrk, we have nine channels at the last layer; each channel represents ne facial view. T arbitrate amng these nine detectrs we use sme heuristic methds. Firstly, we cmbine the utput f sme view ranges int ne class. After cmbinatin, nine channels f view are cnverted t five channels (left prfile, left half-prfile, frntal, right half-prfile and right prfile). Fr example, we cmbine [ 90, 60 ] as left half-prfile. Then, we arbitrate utputs within these five view pses. We use Rwly's heuristic methd. We clean-up utputs f each detectr. See figure 4, A is the last utput f frnt face channel, and nly frntal faces are detected by this channel. B is the last utput f half-prfile channel. This channel in fact includes tw channels: right half-prfile channel and left half-prfile channel. Sme frntal faces will be detected by this channel because half-prfile detectrs will detect part f frntal face as half-prfile face (See Figure 5-B). C represents the last utput f prfile channel, and this channel includes tw channels: right prfile, left prfile t. A B C D Figure 4: Output f fntal (A), half-side (B), and full-side (C) view channels, and the final result (D) after pst-prcessing. Figure 3: Mirrring feature. On the left is a feature learned fr a left view detectr. On the right is the crrespnding feature mirrred fr the right view cunterpart. T arbitrate amng five channels, we present a nvel heuristic methd. In practice, we find half-prfile detectrs and prfile detectrs ften detect part f the frntal face as half-prfile r prfile face. S we prescribe that if a particular lcatin is identified as a frntal face, then all ther lcatins detected by prfile r half prfile face detectrs which verlap it are likely t be errrs, and can

4 therefre be eliminated. Similarly, if a particular lcatin is identified as half-prfile face, then all ther lcatins detected by prfile face detectrs are eliminated. D is the last utput arbitrating amng five channels (see figure 5). We can find that sme faces in B (detected by half-prfile channel) verlap part f faces in A (detected by frntal channel). We identify these faces in B which verlaps with faces in A as errrs, and eliminate them in the last utput (in D). By the same, half-prfile and prfile channels have sme verlaps t, and we eliminate faces detected by prfile channel, which verlap with faces by half-prfile channel. 6. Experimental Results This sectin describes the final face detectin system including training data preparatin, training prcedure, and the perfrmance cmparisn with previus view-based multi-view face detectin system. 6.1 Training Data Set Mre than 6,000 face samples are cllected by crpping frm varius surces (mstly frm vide). The view is in the range f > 90, 90 ] with 90 representing the left-side view and 0 representing the frntal view. A ttal number f abut 25,000 multi-view face images are generating frm the 6,000 samples by artificially shifting r rtatin. In ur system, we partitin multi-view face space int smaller and smaller (tp-dwn in the pyramid) subspaces f narrwer view ranges. At the tp layer, there is nly ne detectr. S all face sample are gruped int ne class. At the secnd layer, there are three detectrs, and face samples are gruped int three view classes (frntal, left-prfile and right-prfile). Face samples labeled with 20, 10, 0, 10, 20 are gruped as frntal faces, thse with > 90, 30 ] are gruped as left-prfile face and the faceswith > 30, 90 ] are gruped as right-prfile faces. At the third layer, there are nine detectrs, and face samples are gruped int nine view classes f [-90, -80], [-70, -60],, [80,90] degrees. 6.2 Training phase Figure 5: Multi-view face examples There are 13 detectrs in ur system, but we nly need train eight detectrs. The right view detectrs at the secnd and third levels can be cnstructed by mirrring features used in left view detectrs. This methd saves abut half training time fr ur system. These detectrs are trained separately, using their wn training data. Nn-face images used fr training these detectrs are cllected frm 12,000 images which dn t cntain face. Every detectr can be a cascade f strng classifiers and this guarantees high detectin speed. At the tp level, the detectr is trained using all the faces frm 90 t 90. It has a cascade f three strng classifiers structure. The number f features in these three strng classifiers is 5, 13 and 20 respectively. It can reject abut 50% nn-face training data, while retaining 99% face train data in training stage. At secnd level, there are three detectrs, each f which is trained t detect part range f the full-view faces. Training faces are separated int three classes t train these detectrs. At this level, each detectr has a cascade f six strng classifiers structure. In ur system, this level can ttally rejects abut 97% nn-face training data which g thrugh tp level, and retain 98% face train data in training stage. At bttm level, face training data is separated int nine classes. At this level, each detectr is a cascade f abut twenty strng classifiers structure. Each detectr has a detectin rate f abut 94%, and achieves a false psitive 6 rate f abut Detectin Results The final detectr is scanned acrss the image at multiple scales and lcatins. Scaling is achieved by scaling the detectrs themselves, rather than scaling the image. This prcess makes sense because the features can be evaluated at any scale with the same cst. We scale the detectrs using a factr f In Figure 4, the image is 320 by 240 pixel size. There are a ttal f 70,401 sub-windws t be verified in this image. The full-view detectr at the tp level needs 110 ms t prcess all these sub-windws. Abut 40% subwindws frm test image are rejected by this carse classifier, and nly 41,114 sub-windws can pass thrugh this classifier. At the secnd level, there are three detectrs. They ttally need 77 ms t prcess all the rest sub-windws. Abut 97% sub-windws f the 41,114 sub-windws are rejected by this level, and nly 1298 sub-windws pass thrugh this level. At the third level, there are nine detectrs. They prcess all these 1298 sub-windws. But they nly need 15 ms t d it, because mst sub-windws are rejected at first and secnd levels. The timing is summarized in Table 1. Level First Secnd Third Ttal Time 110ms 77ms 15ms 202ms Table 1: Times needed fr each level t run the 320*240 image. Because spend 15 ms is needed fr the third level, s it will nt affect the efficiency much f the whle system if we partitin multi-view face space int smaller subspaces f narrwer view ranges at the third level. That it t say (nw we have nine detectrs n the third level), if we decmpse multi-view face space int smaller subspaces (fr example:

5 19 view ranges), this system will still has high detectin speed, but the detectin rate will prbably be increased. Methd View-based Detectr-Pyramid Time 976ms 202ms Table 2: Cmparisn between the view-based and detectrpyramid architecture in speed fr multi-view face detectin. If we had nt adpted the pyramid-like framewrk presented in this paper, we can apply all these nine detectrs at the third level directly n all sub-windws withut carse classificatin at the tp and secnd levels. This methd will (we call it view-based) cst much time fr multi-view face detectin (see Table 2). Our system is tested n CMU prfile face test set. This test set cnsists f 208 images with 441 faces f which 347 were prfile views frm varius news web sites. These images were nt restricted in terms f subject matter r backgrund scenery. They were cllected frm varius news web sites. The database can be dwnladed at ml. We present sme results shwn in Fig 6. We als prvide a vide clip shwing multi-view face detectin at Figure 6: Examples f Detectin Results 7. Cnclusins In this paper, we have presented a detectr-pyramid architecture fr multi-view face detectin. Using a carse-tfine and simple-t-cmplex scheme, ur system slves the prblem effectively and efficiently by discarding mst f nn-face sub-windws using the simplest pssible features at the earliest pssible stage. This leads t the first real-time multi-view face detectin system. Given this framewrk demnstrates gd perfrmance in multi-view face detectin, we stress that the underlying architecture is fairly general and can be applied t ther appearance based bject detectin prblem. REFERENCE [1] A. Kuchinsky, C. Pering, M.L. Creech, D. Freeze, B. Serra and J. Gwizdka. Cnsumer Multimedia Organizatin and Retrieval System. Prceedings f ACM SIG CHI'99 Cnference. [2] A. P. Pentland, B. Mghaddam and T. Starner. Viewbased and mdular eigensapces fr face recgnitin. In CVPR, pages 84-91, [3] D. Rth, M.-H. Yang and N.Ahuja. A SNW-Based Face Detectr. NIPS'00. [4] E Alpaydin and C Kaynak. Cascading Classifiers. Kykernetika, 34(4), [5] E. Osuna, R. Freund, and F. Girsi. Training supprt vectr machines: An applicatin t face detectin. In CVPR, pages , [6] F. Crw. Summed-area tables fr texture mapping. In Prcessings f SIGGGRAPH, vlume 18(3), pages , [7] F. Fleuret and D. Geman. Carse-t-fine face detectin. Inter. Jurnal f Cmputer Visin, [8] G.H. Jhn, R. Khavi, and K. Pfleger. Irrelevant features and the subset selectin prblem. In Prcessings f the Eleventh Internatinal Cnference n Machine Learing, Pages ,1994. [9] H.A. Rwley, S.Baluja, and T.Kanade. Neural netwrk-based face detectin. IEEE Transactins n Pattern Analysis and Machine Intelligence, 20(1):23--28, [10] H. Schneiderman and T. Kanade. A statistical methd fr 3d bject detectin applied t faces and cars. In Prceedings f IEEE Cmputer Sciety Cnference n Cmputer Visin and Pattern Recgnitin, [11] J. Feraud, O. Bernier, and M. cllbert. A fast and accurate face detectr fr indexatin f face images. In Prc. Furth IEEE Cmputer Sciety Cnference n Cmputer Visin and Pattern Recgnitin, pages 52-59, [12] J. Friedman, T. Hastie, and R. Tibshirani. Lgistic regressin: a statistical view f bsting. Technical reprt, Department f Statistics, Sequia Hall, Stanfrd, Univerity, July [13] J. Huang, X. Sha, and H. Wechsler. Face pse discriminatin using supprt vectr machines (SVM). In

6 Prceedings f Internatinal Cnference Pattern Recgnitin, Brisbane, Queensland, Australia, [14] K.K. Sung and T. Pggi. Example-based learning fr view-based human face detectin. IEEE Transactins n Pattern Analysis and Machine Intelligence, 20(1):39--51, [15] K. Tieu and P. Vila. Bsting image retrival. In Prceedings f the IEEE Cnference n Cmputer Visin and Pattern Recgnitin, [16] L. Wisktt, J. Fellus, N. Kruger, and C. V. malsburg. Face recgnitin by elastic bunch graph matching. IEEE Transactins n Pattern Analysis and Machine Intelligence,19(7): , [17] M. Bichsel and A.P. Pentland. Human face recgnitin and the face image set s tplgy. In Image Understanding, Vlume 59, pages , [18] P. Pudil, J. Nvvicva, and J. Kittler. Flating search methds in feature selectin. Pattern Recgnitin Letters, , [19] P. Vila and M.J. Jnes, Rbust real-time bject detectin, IEEE ICCV Wrkshp n Statistical and Cmputatinal Theries f Visin. Vancuver, Canada. July 13, [20] R.E. Schapire and Y. Singer. Bsting algrithms using cnfidence-rated predictins. Prceedings f the Eleventh Annual Cnference n Cmputatinal Learning Thery, pages , [21] S. Gng, S.McKenna, and J.Cllins. An investigatin int face pse distributin. In Prc. IEEE Internatinal Cnference n Face and Gesture Recgnitin, Vermnt, [22] S.Z. Li, L. Zhu, Z.Q. Zhang, and H.J. Zhang. Statistical Learning f Multi-View Face Detectin. In Prc. 7 th Eurpean Cnference n Cmputer Visin. Cpenhagen, Demark. May 2002 [23] Y. Amit, D. Geman and K. Wilder. Jint inductin f shape features and tree classifiers. In IEEE Trans. Pattern Analysis. Mach. Intell, 19, , [24] Y. Freund and R.E. Schapire. A decisin-theretic generalizatin f nline learning and an applicatin t bsting. In cmputatin Learning Thery: Eurclt 95, pages 23-37, Springer-Verlag, 1995.