Deformable Part-based Robust Face Detection under Occlusion by Using Face Decomposition into Face Components

Transcription

1 Deformable Part-based Robust Face Detecton under Occluson by Usng Face Decomposton nto Face Components Darjan Marčetć, Slobodan Rbarć Unversty of Zagreb, Faculty of Electrcal Engneerng and Computng, Croata {darjan.marcetc, Abstract - In ths paper, we propose modfcatons of deformable part-based models n order to ncrease the robustness of face detecton under occluson. The modfcatons are: ) the tree, representng the deformable part-based model of the frontal face, whch s parttoned nto 11 subtrees representng face components; ) the weght of each face component whch s obtaned based on the results of psychologcal experments; ) the ntroducton of new scorng functons and thresholds; and v) a new procedure for robust face detecton based on the valuaton of scorng functons and thresholds. The experment was performed only for frontal face mages, and thus ths wor s used only as a proof of concept. Based on the encouragng expermental results, we conclude that the proposed method s sutable for extenson to detect faces wth dfferent poses under occlusons. Keywords: Robust Face Detecton, Occluson, Deformable Partbased Model, Face Components I. INTRODUCTION Face detecton s the process of fndng and localzng faces n an mage. Fndng and analysng faces n unconstraned real-world mages s a very dffcult computer vson tas. Constranng factors are the ambgutes of the face, such as poses, occlusons, expressons, llumnaton, scale, colours and textures. Dverse methods have been developed based on scannng wndow classfers [1], deep neural networs [2], partbased models [3-6] and regresson [7]. Several papers have been publshed related to face detecton under occluson [8-11], but they are constraned to detect only frontal faces under occluson. In [8], a support vector machne (SVM) wth local ernels was used for robust frontal face detecton under partal occluson. A test was performed on HOIP and CMU face databases. The proposed method was demonstrated for the detecton of faces wearng sunglasses or a scarf. It was confrmed that the proposed method was better than a method based on conventonal SVM wth a global ernel. In [9], a generalzaton of the wor of Vola and Jones for the effcent detecton of faces wth occlusons was descrbed. The man dea s based on boostng algorthms and cascade structures to effcently detect faces wth occlusons. Several mprovements were presented: () a robust boostng scheme; () renforcement tranng; and () cascadng wth evdence to extend the system to handle occlusons. Expermental results for detectng frontal faces wth artfcal occlusons were obtaned on CMU and MIT face databases. The authors n [1] dvded a face regon nto multple patches, and mapped those wea classfers to the patches. The wea classfers, nspred by the Vola and Jones approach, belongng to each patch, were combned to create a new classfer to determne f ths was a vald face patch wthout occluson. All of the vald face patches, wth dfferent weghts, were used to mae a fnal decson whether the nput subwndow was a face. The expermental results showed that the proposed method s promsng for the detecton of occluded faces. In [11], a novel approach for automatc and robust face detecton was presented. It used a component-based approach that combned technques from both the statstcal and structural pattern recognton doman. The method reled on Haar-le features and an AdaBoost traned classfer cascade, and the topology verfcaton was based on graph matchng technques. The system was appled to frontal face detecton and the experments showed mproved performance compared to conventonal face detecton n the presence of partal occlusons, uneven llumnaton, and out-of-plane rotatons, thus yeldng hgher robustness. Intally, face detecton, pose estmaton, and landmar localzaton were handled separately. These three tass can be smultaneously handled wth a unfed part-based approach that uses mxtures of trees wth a shared pool of part templates for face landmars, and utlzng an encodng scheme for modes of elastc deformaton, and a vew-based topology for face detecton and pose estmaton [5]. The parameters of part-based models are dscrmnatvely traned n a max-margn framewor. Facal landmars are vsble n multple vews and thus the correspondng parts can be shared among dfferent vews, whch reduces the complexty of the learnng procedure for the model. Most part-based models are senstve to the presence of occlusons because they are learned on nonoccluded faces. Facal landmar scores are processed one by one n a cascade, whch also decreases robustness. In ths paper, we propose robust frontal face detecton obtaned by modfyng the method presented n [5]. In our approach, a face s decomposed nto dstnct face components. These face components are selected by psychologcal experments related to face recognton by humans [12]. Robustness s acheved by fuson wth a

2 weghted votng scheme where the relatve relatons among weghts are selected based on recommendatons gven n [12]. In ths paper, we present a proof of concept for the detecton of frontal faces under occlusons whch s sutable for extenson to mult-vewpont occluded face detecton. II. DEFORMABLE PART-BASED MODELS Deformable part-based models (DPMs) are one of the most popular face detecton, pose estmaton and landmar localzaton methods [5]. The DPM was used to fnd a vsual object n an actual mage n [4]. The authors descrbe the approach of breang down vsual objects nto a number of prmtve parts and specfy a permtted range of spatal relatons whch these prmtve parts must satsfy for the object to be present n an mage. The object model s represented by prmtve parts held together by sprngs. The sprngs jonng the prmtve parts serve both to constran relatve movement and measure the cost of the movement by how much they are stretched [3]. The maxmal matchng between an object model and a vsual object n the scene s expressed by the total cost of embeddng parts at ther locatons. For total cost optmzaton, the authors proposed a method based on dynamc programmng. Ths ntal dea was extended to a mxture of multscale deformable part models, represented by HOG features, and methods for dscrmnatve tranng wth partally labelled data [5]. In [6], the authors proposed a method that solved the speed bottlenec of DPMs wthout mparng the accuracy of detecton. The method proposed n [5] uses a model based on mxtures of trees wth a shared pool of parts represented by HOG descrptors. Facal landmars are modelled as parts. A global mxture of parts s used to capture topologcal changes due to face vewponts. A set of 13 trees was used to represent dfferent face vewponts (Fg. 1). Each tree has only one root node. From 13 trees whch represent dfferent face vewponts, we selected only the tree assocated wth the frontal face, thus smplfyng the model descrpton. Ths smplfcaton was used for a proof of concept, related to the robust detecton of faces under occluson. The tree assocated wth the frontal face T = (V, E) has V = 68 vertces (or parts) and E = 67 edges (Fg. 1). Let us wrte I for an mage and l = (x, y ) for the pxel locaton of part V. A confguraton of parts L ={l : = 1,..., 68} s scored as follows [5]: Score App Shape I,L App I,L 68 I,L w I,l L 1 j E j Shape (L) α 2 2 a dx b dx c dy j j d dy. Equaton 2 sums the appearance evdence for placng a HOG-based template w for part V at locaton l. A HOGbased feature vector I,l s extracted from pxel locaton l n mage I. Equaton 3 scores the mxturespecfc spatal arrangement of parts L, where dx = x x j and dy = y y j are the dsplacement of the V part relatve to the V j part. Each term n the sum can be nterpreted as a sprng that ntroduces spatal constrants between a par of parts, where a, b, c and d represent sprng parameters (rest locaton and rgdty) and were learned on a set of tranng mages [3]. The term α s a scalar bas assocated wth the front face vew. j (1) (2) (3) Inference corresponds to maxmzng Score I,L Eq.1 over L: Score I,L max Score I,L. (4) L Snce T = (V, E) s a tree, the nner maxmzaton can be done effcently wth dynamc programmng [3]. Let us denote L for whch the Score (I, L) s maxmal wth L,.e. Score ( I,L) = Score ( I,L ). If Score ( I,L ) θ 1, then t s assumed that a face s detected n the mage (.e. locatons of facal landmars are gven by L ). θ 1 s a threshold value appled n [5]. III. ROBUST FACE DETECTION In the context of ths paper, robustness s related to the ablty to detect frontal faces n an mage I wth dfferent types of occlusons n cases where the orgnal approach [5] fals due to false negatve detecton (.e. a face has Score ( I,L ) < θ 1, Fg. 2). In order to ncrease the robustness of frontal face detecton, we modfed the orgnal approach descrbed n Secton 2 by ntroducng a new algorthm based on modfcatons as follows: n Fgure 1. Mxture of trees that encode topologcal change due to vewpont [5]. The red lnes denote sprngs between pars of parts.

3 ScoreThreshold(I,L ) INPUT: Score(I,L ), θ 1 INPUT: Score(I,L ), ScoreAppearance(I,L ), ScoreThreshold(I,L ), θ 1, θ 2, θ 3, θ 4 Score(I,L ) θ 1 Score (I,L ) θ 1 Face n I at L Face not n I at L Fgure 2. Flowchart of the decson step for the orgnal face detecton algorthm [3]. ) The frst modfcaton parttons the tree T = (V, E), whch corresponds to the frontal face (seventh tree n Fg. 1), nto 11 subtrees T = (V, E ); = 1, 2,...11, by removng the edges that connect dfferent face components (Fg. 3). Each subtree s assocated wth one face component: left upper edge, left lower edge, rght upper edge, rght lower edge, left sde of the lps, rght sde of the lps, nose, left eye, left eyebrow, rght eye, rght eyebrow. Face n I at L Face n I at L Score (I,L ) θ 2 Face not n I at L ScoreAppearance (I,L ) θ 3 OR ScoreThreshold (I,L ) θ 4 Face not n I at L a) I -mage L - parts locatons Face not n Face may be n I at L θ 2 I at L θ 1 Face n I at L Score(I,L ) < θ Score(I,L 2 Score(I,L ) [θ 2, θ 1> ) θ 1 θ 4 (θ 3, θ 4) Face not n I at L Face n I at L θ 4 θ 3 ScoreAppearance(I,L ) b) Fgure 4. a) Flowchart of the decson step for the proposed face detecton algorthm; b) Intervals Fgure 3. The face regon s parttoned nto 11 face components contanng parts correspondng to the face components. Removed edges are mared wth short yellow lnes. ) The second modfcaton ntroduces three new thresholds (θ 2, θ 3 and θ 4 ) and three new scorng functons ( Appearance ( I,L ), ScoreAppea rance ( I,L ) and ScoreThres hold ( I,L ) ). The orgnal threshold θ 1 proposed n [5] s.65, and the threshold θ 2 s determned based on experments (descrbed n Secton 4). These two thresholds defne three ntervals for Score ( I,L ). Interpretaton of these three ntervals s clarfed n Fg. 4 b). The thresholds θ 3 and θ 4 dvde the 2-dmensonal decson space nto face and non-face space (Fg. 4 b)). Fg. 2 llustrates the orgnal decson step [5]. Fg. 4 a) depcts a flowchart for the fnal decson step of our frontal face detecton algorthm. A. Fnal decson step for the face detecton algorthm The fnal decson step for the face detecton algorthm s gven as follows: ) If Score ( I,L ) θ 1, then a face s detected n mage I at locaton L. ) If Score ( I,L ) θ 2, then a face s not detected n mage I at locaton L. ) If Score ( I,L ) falls nto the nterval [θ 2, θ 1 >, then ScoreAppea rance ( I,L ) and ScoreThres hold ( I,L ) are used n conjuncton wth the newly ntroduced thresholds θ 3 and θ 4, respectvely (Fg. 4). If ScoreAppea rance ( I,L ) θ 3 or ScoreThres hold ( I,L ) θ 4, then there s a face n an mage I at locaton L (Fg. 4). The ScoreAppea rance ( I,L ) and ScoreThres hold ( I,L ) are descrbed as follows. A face appearance score s defned as:

4 ScoreAppea rance 11 I, L Appearance I, L, (5) 1 where s the weght of a -th face component selected based on the results of psychologcal experments [12] whch estmate the mportance of the face components n the face recognton process performed by humans. The expermental results [12] ndcate the followng order of mportance of facal components (n decreasng order): eyes, eyebrows, mouth, nose and contour of face (Table I). TABLE I. THE EXPERIMENTALLY DETERMINED IMPORTANCE OF FACE COMPONENTS IN FACE RECOGNITION BY HUMANS Face component Weght µ Left/rght upper contour - µ 1/µ 2 1 Left/rght lower contour - µ 3/µ 4 1 Left/rght sde of the lps - µ 5/µ 6 2 Nose - µ Left/rght eye - µ 8/µ 1 3 Left/rght eyebrow - µ 9/µ For each face component = 1, 2,...11, the appearance score s defned as: I, L w I,l, (6) Appearance where ndex determnes the parts V from the set of parts V assocated wth a -th face component, w s a HOG-based template for part V, and I,l represents the HOG-based feature vector extracted from pxel locaton l n an mage I. For example, = 8 corresponds to a left eye (Table I), and there are sx parts V 8 = {V 21, V 22,..., V 26 }. A face appearance threshold score s defned as: ScoreThres hold where 1 f Appearance c f Appearance 11 I, L (7) c 1 ( I,L ) Threshold ( I,L ) Threshold and Threshold defnes the mnmal value of the appearance score of a -th face component for whch t s supposed that ths face component s correctly detected. Thresholds Threshold ; = 1, 2,, 11 are selected based on the experments conducted n the XM2VTS [13] and INRIAP [14] databases descrbed n Secton 4. The nterpretaton of ScoreThres hold ( I,L ) s as follows. For example, f ScoreThres hold ( I,L ) = 5, ths means that 5 face components are vsble and properly detected, whle the remanng sx (11 5) face components are not detected due to poor qualty of an mage or occluson. The man reason for ntroducng ScoreAppea rance ( I,L ) and ScoreThres hold ( I,L ) s the fact that n the nterval [θ 2, θ 1 > there are many false postve and false negatve faces, so t s used to mnmze the total error rate (TER). The thresholds θ 2, θ 3 and θ 4 defne the operatng pont and they are selected to mnmze the TER. IV. EXPERIMENTAL RESULTS Two databases were used to perform the experments. From the XM2VTS database [13], 21 mages of frontal faces wthout occlusons were selected (frst column of Fg. 5). For each of these 21 mages, sx new mages were created for smulatng sx typcal confguratons of face occlusons by coverng dfferent face regons wth blac rectangles (from the second to seventh column of Fg. 5). (8) Addtonally, 73 mages wthout faces were selected Fgure 5. Results of face detecton for frontal face mages from the XM2VTS database wth three score values S1 for Score(I,L ), S2 for ScoreAppearance(I,L ) and S3 for ScoreThreshold(I,L ). The frst column s faces wthout occlusons and the other columns represent faces wth dfferent modes of occluson. For example, for the 5th face n the top row S1 =.69887, whch means that the orgnal method fals to detect the face, whle S3 = 3, whch means that the modfed method detects the face.

5 Appearance(I,L ) score from the INRIA person database [14] (Fg. 6). In total, 1,48 (.e ) mages were used for testng. Let us now justfy somewhat the unusual confguraton of the testng. Typcally, some n the wld datasets, such as AFW [15], whch contan complex scenes wth face occlusons, are used for testng face detecton performance. All such databases volate our assumpton about the presence of only frontal faces. In order to crcumvent ths shortcomng of the popular datasets used for face detecton testng, we were forced to create an ad hoc face dataset just for the purpose of the proof of concept. Because the XM2VTS database contans only face mages, the INRIA person database was used to enable an evaluaton of false postve face detectons. Fgure 7. The sngle false detectons of faces for the mage from the INRIA person database wth Score(I,L ) Fgure 6. False detectons of faces for an mage from the INRIA person database wth Score(I,L ) > 1. For each detecton frame, the values of Score(I,L ), ScoreApperance(I,L ) and ScoreThreshold(I,L ) are gven n the same lne, respectvely. For example, n the mage at the lower left frame, the values are Score(I,L ) =.8977, ScoreApperance (I,L ) = and ScoreThreshold(I,L ) = 1. Frst, we ran the orgnally proposed algorthm wth the predefned threshold θ 1 =.65 proposed n [5]. The followng results were obtaned: 16 frontal faces were not detected (false negatves) among 1,47 faces (Fg 9.a)), and 1 mage regon was detected as a face false postve among 1,48 mages (one mage from INRIA had one mage regon wth a Score (I,L ) >.65, Fg. 7.), so we concluded that the robustness was relatvely hgh compared wth the state-of-the-art-method [7]. Among 16 false negatves, there were 145 frontal face mages havng the type of occluson depcted n the 5th column n Fg. 5, and the remanng 15 havng the type of occluson depcted n the 2, 3, 6, or 7th column n Fg. 5. Note that there s no false negatve detecton for 21 mages wthout occlusons, and a sngle false postve was n one of the 73 mages from the INRIA person database (Fg 7.). For our method, we searched for the values for thresholds θ 2, θ 3, and θ 4 to fnd an operatng pont that mnmzes the total error rate (TER). The mnmum TER =.87 was acheved for the followng threshold values: θ 2 =.83, θ 3 =.69, and θ 4 = 3. The threshold θ 1 s assumed to be.65, as n the orgnal paper [5]. The values of the thresholds Threshold ; = 1, 2,, 11 are determned as the value of the frst quartle of Apperance (I, L ) obtaned on 21 frontal mages wthout occlusons (Fg. 8). Threshold Face component Face component Fgure 8. Appearance scores of 11 face components for 21 mages of frontal faces wthout occlusons from the XM2VTS database: a) Appearance (I,L ) dstrbuton, the red lne represents the mean value, the box edges the frst and the thrd quntle, the data range s represented wth the dashed lne, and the red crosses represent the data outlers; b) the values of Threshold are obtaned as the frst quartle from the dstrbuton n a). a) b)

6 Number of detectons Number of detectons 1-FPR FN=16 θ 1 =.65 2 FP= FN =99 θ 1 =.65 θ 2 =.83 θ 3 =.69 θ 4 = 3 2 FP= Threshold for Score(I,L ) Threshold θ 3 for ScoreAppearance(I,L ) 1-FNR (a) (b) (c) Fgure 9. Comparson of the orgnal and the proposed method: a) FN and FP for the orgnal method; b) FP and FN for the proposed method where θ 1 = -.65, θ 2 =-.83, and θ 4 = 3; c) 1-FPR and 1-FNR for the orgnal and the proposed method, the green crcle represents the worng pont of the orgnal algorthm for threshold θ 1 =.65, and the red crcle represents the worng pont for the proposed algorthm for θ 1 =.65, θ 2 =.83, θ 3 =.69, and θ 4 = 3 for whch the TER s mnmal ORIGINAL TER=.112 PROPOSED TER= By usng the proposed method and optmal thresholds we acheved a reducton of false negatve face detectons from 16 to 99, whle false postve face detectons ncreased from 1 to 2. The TER was reduced from.112 for the orgnal approach to.87, whch corresponds to an mprovement of 22.32% (Fg. 9 c). V. CONCLUSION In ths paper, we have analysed the nfluence of face occluson on the face detecton method based on deformable part-based models as orgnally proposed by X. Zhu and D. Ramanan [5]. We proposed modfcatons of the method n order to ncrease the robustness of face detecton and to mnmze the total error rate. The man modfcatons are: ) the tree, representng the part-based model of the frontal face, whch s parttoned nto 11 subtrees representng the face components (left upper edge, left lower edge, rght upper edge, rght lower edge, left sde of the lps, rght sde of the lps, nose, left eye, left eyebrow, rght eye, rght eyebrow); ) the weght of each face component whch s obtaned based on the results of psychologcal experments [12]; ) the ntroducton of new scorng functons and thresholds; v) a new procedure for the robust detecton of faces under occluson based on an evaluaton of scorng functons and thresholds. By usng our method on a database consstng of 1,48 mages (21 non-occluded face mages, 1,26 face mages wth sx types of occluson and 73 non-face mages), the TER was mproved by % compared to the orgnal method [5]. Note that the experment was performed only for frontal face mages, and thus ths wor s used only as a proof of concept. Based on the encouragng results, we plan to extend the proposed method by applyng the orgnal mxture trees for 13 dfferent face poses and n a total of 146 parts [5]. ACKWLEDGMENTS Ths wor has been supported by the Croatan Scence Foundaton under project 6733 De-dentfcaton for Prvacy Protecton n Survellance Systems (DePPSS). It s also the result of actvtes n COST Acton IC126 "De-dentfcaton for Prvacy Protecton n Multmeda Content". LITERATURE [1] P. Vola and M. Jones, Rapd object detecton usng a boosted cascade of smple features, nproc. IEEE Comput. Soc. Conf. Comput.Vs. Pattern Recog., 21, pp. I-511 I-518.[ [2] Y. Tagman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closng the gap to human-level performance n face verfcaton, In CVPR, 214, pp [3] M. Fschler and R. Elschlager, The representaton and matchng of pctoral structures, IEEE. Trans. Computers, C-22(1), 1973, pp [4] P. F. Felzenszwalb, R. B. Grshc, D. McAllester, and D. Ramanan, Object detecton wth dscrmnatvely traned part based models, IEEE TPAMI, 32(9), 21, pp [5] X. Zhu and D. Ramanan, Face detecton, pose estmaton, and landmar localzaton n the wld, n Proc. IEEE Conf. Comput. Vs. Pattern Recog., 212, pp [6] J. Yan, Z. Le, L. Wen, and S. Z. L, The fastest deformable part model for object detecton, n Proc. IEEE Conf. Comput. Vs. Pattern Recog., 214, pp [7] S. Lao, A. K. Jan, S. Z. L, A Fast and Accurate Unconstraned Face Detector, IEEE TPAMI, vol. 38, Issue: 2, 216, pp [8] K. Hotta, A robust face detector under partal occluson, n Proc. Int. Conf. Image Process., 24, pp [9] Y. Ln, T. Lu, and C. Fuh, Fast object detecton wth occlusons, n Proc. Eur. Conf. Comput. Vs., 24, pp [1] J. Chen, S. Shan, S. Yang, X. Chen, and W. Gao, Modfcaton of the adaboost-based detector for partally occluded faces, n Proc. 18th Int. Conf. Pattern Recog., 26, pp [11] L. Goldmann, U. Monch, and T. Sora, Components and ther topology for robust face detecton n the presence of partal occlusons, IEEE Trans. Inf. Forenscs Securty, vol. 2, no. 3, 27, pp [12] P. Snha, B. Balas, Y. Ostrovsy, and R. Russel, Face Recognton by Humans: Nneteen Results All Computer Vson Researchers Should Know About, Proc. IEEE, vol. 94, no. 11, Nov 26, pp [13] K. Messer, J. Matas, J. Kttler, J. Luettn and G. Matre; XM2VTSbd: The Extended M2VTS Database, Proceedngs 2nd Conference on Audo and Vdeo-base Bometrc Personal Verfcaton (AVBPA99) Sprnger Verlag, New Yor, [14] INRIA Person Dataset, [15] The Annotated Faces n the Wld (AFW) testset,