Combined Object Detection and Segmentation

Transcription

1 Combned Object Detecton and Segmentaton Jarch Vansteenberge, Masayuk Mukunok, and Mchhko Mnoh Abstract We develop a method for combned object detecton and segmentaton n natural scene. In our approach segmentaton and detecton are consdered as two faces of the same con that should be combned nto a sngle framework. There are two man steps n our strategy. Frst we focus on the learnng of a vsual vocabulary that effcently encompasses objects appearance, spatal confguraton and underlyng segmentaton. Ths vocabulary s used wthn a Hough votng framework to produces object s confguraton. The second step conssts n searchng for vald objects confguratons by nterpretng and scorng them n terms of both detecton and segmentaton. Ths allows us to prune false detectons and hallucnated object-lke segmentaton. Experments show the advantage of the combned approach and the mprovements over recent related methods. Index Terms Object recognton, random forest, hough votes. Fg. 1. Example results of our proposed combned object detecton and segmentaton approach. The combnaton of segmentaton and detecton mproves the accuracy of both detecton and segmentaton processes. I. INTRODUCTION Ramanan [7] has shown that by usng the segmentaton to verfy detecton hypothess, one could sgnfcantly mprove the detecton performances. Nevertheless, n ther approach detecton and segmentaton are performed sequentally thus gnorng the nteractons between them. ObjCut [4] provdes an elegant method allowng detecton and segmentaton process to contnually nteract wth each other. Usng MRFs wth a layered pctoral structure the algorthm can acheve object detecton along wth hgh accuracy segmentaton wthn natural scene. The total number of parts n the pctoral structure and ther parameters are farly lmted whch makes ther model very senstve to vewpont varatons. The Implct Shape Model (ISM) [1] provdes an nterestng way to combne detecton and segmentaton wthn a sngle framework. The dea s to learn a vsual dctonary of local appearance and ts spatal dstrbuton over a star shaped model. At tranng tme the vsual dctonary s enrched wth underlyng segmentaton mask and the matchng locaton n regards to the object center. At run tme the dctonary s used wthn a Hough votng framework to casts votes for the object locaton. A segmentaton mask s nferred from vsual words local segmentaton masks. The ISM framework has several drawbacks. The learnng of the vsual dctonary of local appearances and ts spatal dstrbuton over the star shape model are learned ndependently. Moreover only postve samples can be used to generate the vsual words (VW). Fnally the aggregaton of evdence wthn the Hough accumulator often leads to false detecton on cluttered background. Over the years several modfcatons of the ISM framework have been proposed, manly focusng on learnng a better vsual dctonary and Mmckng the human vson system s ablty to dentfy an object and to solate t from ts envronment s one of the most challengng tasks n the feld of computer vson. A tremendous amount of work has been done over the years, leadng to dfferent formulaton of the problem and dfferent approaches n handlng ths task. One of the major approaches s the object detecton problem n whch objects scales and locatons wthn the mages are to be dscovered. Another major approach s the object segmentaton problem where mages are dvded nto regons wth some of them beng the objects boundares. One can see that whle beng dfferent these two problems are strongly related. Several approaches have been proposed to combne detecton and segmentaton [1]-[4]. A straghtforward approach conssts n performng the segmentaton wthn the object boundng box provded by a strong detector [5], [6]. Whle offerng accurate segmentaton, such method heavly depends on the qualty of the detecton results. Moreover no feedbacks from the segmentaton process are provded to the detector. Manuscrpt receved September 14, 2012; revsed December 27, Ths work was supported n part by the Academc Center for Computng and Meda Studes, Kyoto Unversty, Japan. J. Vansteenberge s wth the Department of Intellgence Scence and Technology, Graduate School of Informatcs, Kyoto Unversty, Yoshda Honmach, Sakyo-ku, Kyoto, Japan. (e-mal: vansteenberge@mm.kyoto-u.ac.jp). M. Mukunok and M. Mnoh are wth the Academc Center for Computng and Meda Studes, Kyoto Unversty, Yoshda-Honmach, Sakyo-ku, Kyoto, , Japan. DOI: /IJMLC.2013.V

2 The frst term s the dstrbuton of the object s center poston for a gven VW. The second term specfes the confdence for a VW to be related to the target object. The last term descrbe the qualty of the match between a feature and the matchng vsual words. So the pseudo probablty of an object O at locaton c s defned as: mprovng the votng procedure [8]-[10]. In our work, we present a method for combned object detecton and segmentaton wthn natural scene based on Random forest (RF) and the ISM framework. Our contrbuton s threefold; frst we propose a way to learn a vsual dctonary optmzed for combned detecton and segmentaton. The appearance of the vsual words, ther dstrbuton over the star shaped model and the underlyng segmentaton mask are jontly learned. Our second contrbuton les n the effcent evaluaton of the qualty of the aggregated evdence wthn the accumulator. Each confguraton maxma from the Hough accumulator are ndependently scored n terms of detecton and segmentaton. These scores are used to estmate the qualty of the evdence combnatons. Our last contrbuton conssts of llustratng the benefts taken from performng segmentaton and detecton wthn the same framework. We show mprovements n detecton and segmentaton performances when combnng both processes. p (O, c) = p (O, c f k, lk ) p ( f k, lk ), wth k the total number of features extracted from the test mage. Settng the accumulator dmensons to be equal to the test mage dmensons, we have a drect relaton between postons n votng space and n the mage space. B. Votng for Segmentaton Parameters Asde from votng for detecton parameters, the support evdences need to vote for the segmentaton parameters m. Beng drectly related to the target object, the support evdences provde local nterpretatons of the mage content. Assemblng the local nterpretatons agreeng on a same object confguraton allows nferrng a global nterpretaton for the target object. Especally one can compute a back-projected segmentaton mask m as explaned n secton IV.A. To do so we need to store addtonal nformaton about the support evdences. A sngle vote from a feature f k stored at c n the accumulator s: II. GENERATING OBJECT CONFIGURATIONS In our approach, object recognton reles on the generaton of a set of canddate object confguratons whch are then scored n terms of detecton and segmentaton (see Fg. 2). We defne an object confguraton h = (c, m) as an assembly of parameters c related to the object detecton.e. the target object poston and scale wth parameters m related to the object segmentaton. Smlarly to [1] we use a vsual dctonary to make assumptons about the objects confguratons. These assumptons, called votes, are collected nto a Hough accumulator H ( x, y, s ), where the canddate confguratons are searched as local maxma. vc, k = ( p (O, c f k, lk ), lk,{w }k ), III. LEARNING A VISUAL DICTIONARY The vsual words are used to cast votes for the objects confguratons whch makes them crucal to the performances of the algorthm. A partcular attenton needs to be pad to ther desgn. A. Votng for Detecton Parameters The votng procedure for the detecton s defned as follow. Let f be an mage feature extracted at locaton l. We match the vsual dctonary aganst f to obtan a set of vald vsual A. Tranng Data Our tranng data s a set of local patches extracted from random locatons wthn postve and negatve tranng mages. Each tranng patch P = ( A, F, g, d ) has a local {W }, hereafter called support evdences. The condtonal probablty of an object O exstng at the poston c wthn the votng space s computed as: appearance patch A, a ground truth local segmentaton mask p (O, c f, l ) = p (O, c W, f, l ) p (W f, l ) = p(c O, W, l ) p(o W ) p (W f ). (4) where p(o, c) s the confdence for the target object O to be found at poston c. The correspondng confguraton h = ( c, m ) s composed of m = ψ ({lk },{{W }k }) the back-projected segmentaton mask and c = (cx, c y, cs ) the 3D locaton wthn the accumulator. Fg. 2. Overvew of our framework. The vsual dctonary s matched aganst a test mage. Each matchng VW casts votes for the object confguraton nto an accumulator. Strongly supported confguratons are scored n terms of detecton and segmentaton. True object confguratons are requred to have hgh scores for both detecton and segmentaton. (3) wth p(o, c f k, lk ) the confdence of the vote and lk the extracted feature locaton. The accumulaton of votes from multple features for a confguraton at locaton c becomes: vc = ( p(o, c),{lk },{{W }k }), words (2) k F, a class label g and an offset to the object center d. For negatve patches, the local segmentaton and offset vector are left undefned. (1) 61

3 B. Random Forest To learn the VW from the tranng patches, we grow a RF to act as a vsual dctonary. Ths dea has been exploted wth success by the past [11],[10], however, to the best of our knowledge, we are the frst to take nto account the class label, the offset vectors and the local segmentatons nto the learnng of the vsual dctonary. The dea s to grow the trees such as to learn dscrmnant local appearances for consstent locatons relatve to the object center and consstent local segmentatons. A RF s composed of a set of trees traned ndvdually as explaned n the followng paragraph. Tranng. Startng from the root, each node of a tree splts the ncomng patches { P } nto two subsets accordng to a Fg. 3. Back-projectons of confdent leaves traned on Wezmann horse dataset. The top, mddle and bottom rows shows sample leaves from RF optmzed for detecton only, segmentaton only, combned detecton and segmentaton. The combned approach takes beneft from both approaches. Votes are comng from compact locatons, whch favor detecton, whle the back-projected masks cover large local areas and preserves boundares. bnary test t ( A) {0,1}. The test smply compares the pxel's values at two random postons wthn the appearance patch A.The two subsets are passed onto chld nodes where When a tree's maxmal depth has been reached or when the number of patches n a subset s too small, we create a leaf node and store the patches' nformaton, that s, the number of postve patches P p = {P g = 1}, the total number of further splttng s performed. Ths recursve splttng leads to a large number of subsets n whch patches share a smlar appearance. The path from the tree's root to a gven leaf descrbes the patches shared appearance. Thus each leaf acts as a VW. In order to product strong votes for the object confguraton, a vsual word should: Be hghly confdent n votng, vote for a precse locaton and vote for a consstent local segmentaton mask. When growng a tree, we need to choose at each node the bnary test whch ncreases the potental vote's qualty. We defne separate uncertanty measurement for our three crterons. Let Z = {P } be a patch P, the offset vectors {d } and the set of local segmentatons {F }. Testng. At run tme patches are extracted from random locatons n the test mages and passed through each tree of the forest. Each tme a sngle patch from locaton l, reach a leaf node W a vote s casted accordng to (1). The locaton dstrbuton p(c O, W, l ) s determned by the set of locaton subset of patches leavng a gven node n a gven tree. The uncertanty over the class labels {g }, offset vectors d and {c} = l {d } weghted by a unform probablty 1/ d. The probablty p(o W ) s estmated by the rato P p / P. Fnally the probablty p(w f ) s set unform over the number of trees n the forest. local segmentaton masks F are defned as: Z U1 ( Z ) = p ( g ) log( p( g )) (5) IV. EVALUATING PROPER OBJECT CONFIGURATIONS =1 U 2 (Z ) = = (d d )2 Generalzed Hough transform s known to be a very robust parameter estmaton method. In our framework, t allows for object detecton under large occlusons and poses changes. However the addtve nature of the accumulaton of evdences s equvalent to assumng ndependence between the support evdences. Ths crude assumpton makes the Hough framework senstve to cluttered background, whch produces falsely confdent confguratons. Ths s where the combned approach shows ts power. Obvously two adjacent support evdences are strongly correlated. When castng votes for the detecton parameters the correlaton of the support evdences s lost. However, the back-projected segmentaton mask keeps the spatal relatonshp between the support evdences. One can score such mask to estmate f the spatal dstrbuton of support evdences s compatble wth a true object confguraton. We end up wth two scores for a sngle object confguraton. Assumng ndependence between c and m allows to smplfy the computaton of a canddate confguraton s fnal score whch s: (6) : g 1 U 3 (Z ) = = ( F F ) 2, (7) : g 1 wth g = 1 f the patch s a postve sample, d and F the mean offset vector and mean segmentaton patch n Z. The frst measurement tends to mprove the VW dscrmnant power. The second and thrd uncertanty measurements mprove the votes locatons and segmentaton mask accuraces. At each non leaf node of a tree, one of the three uncertanty measurements s randomly selected and used to score bnary tests. The test whch mnmzes the uncertanty s kept and the tree s grown up to the next level. Inhbtng some of the uncertanty measurements wll optmze the trees for ether detecton, wth compact votes locatons or segmentaton, wth large back-projected areas. The Fg. 3 show the nfluence of the tranng when U2 or U3 are nhbted. P (O, h j ) = P(O, c j ) P (O, m j ). 62 (8)

4 The probablty P(O, c j ) from (2) s the object V. RESULTS We have tested our algorthm for detecton and segmentaton on three challengng datasets, the Wezmann Horse dataset [13], the TUD pedestran dataset [14] and the PennFudan pedestran dataset [2].For each dataset, the postve tranng mages have been reszed so that each object's boundng box would have ts largest dmenson approxmately equal to 120 pxels. All the trees were traned on postve and negatve patches. The tranng patches of 16 by 16 pxels were extracted at random locatons wthn the boundng box of postve mages and anywhere wthn negatve mages. For both pedestran datasets, we used a subset of 600 mages from the INRIA dataset [12] as negatve tranng set. Each RF was composed of 5 trees. The SVMs used to score the segmentaton masks are usng RBF kernel and have been traned on 50 postves and 50 negatves masks for the Horse dataset and 200 postves and 200 negatves masks for both pedestran dataset. A true detecton should overlap the ground truth boundng box by more than 0.5. To avod multple detectons of the same nstance we use non-maxma suppresson. Due to the lack of standardzed evaluaton measures n the segmentaton communty, we use two measurements to evaluate the segmentaton performances. The Fscore defned as Fscore = (2 precson recall) / (precson + recall) and the foreground accuracy computed as Acc = (ntersecton) / (unon). The Table I show the superorty of the combned approach over specalzed approach. All the segmentaton results are gven for a detecton recall of 97.7%. We can see the combned approach performs the best for both detecton and segmentaton. Ths dataset was orgnally bult for segmentaton whch makes t not very challengng for detecton. To better llustrate the detecton s mprovements, we used the TUD pedestran dataset. Fg. 5 shows the combned approach ncreased both precson and recall when comparng to RF optmzed for detecton only. We also mproves over two Hough-based approach, the 4DISM [15] and the Hough Forest [10] retraned from the code avalable on ther webste. confguraton's detecton score whle P (O, m j ) s the segmentaton score. The fnals objects confguratons satsfyng both detecton and segmentaton are defned as: h* = {h j P(O, h j ) > υ}, (9) wth υ the threshold controllng the strctness of the algorthm. In the followng secton, we show how to compute P (O, m ) from the support evdences. A. Back-projected top-down segmentaton mask. Smlarly to the detecton parameters c, the segmentaton parameters m whch correspondng to the foreground labelng, are also extracted from the support evdences' votes. We start by collectng all the votes for an object confguraton h wthn a crcular regon centered at the confguraton s locaton c h. The collected votes contan a set of support evdences locatons {l}. {W } Remndng but also ther matchng that local ground truth segmentatons F are avalable for each VW word, one can produce a global segmentaton mask by assemblng these local segmentatons. We closely follow the probablstc formulaton of [1] where the backprojected segmentaton mask m s computed as a weghted sum of the local segmentaton masks. Fg. 4 shows samples back-projected segmentaton masks. Fg. 4. Back-projected segmentaton masks. The frst and second rows show the back-projected masks for respectvely, false and true object confguratons. TABLE I: DETECTION AND SEGMENTATION RESULTS ON THE HORSE DATASET. B. Scorng the Segmentaton Mask Once a back-projected segmentaton mask s avalable, t can be used to score the combnaton of support evdence from whch t orgnated. Indeed, we can observe from Fg. 4 that proper combnatons of support evdences lead to object-lke segmentatons whle some of the poor combnatons produce ll shaped segmentatons. To estmate the qualty of the support evdence combnaton we learn a scorng functon on postve and negatve back-projected masks. HoG [12] feature descrptors are extracted from each segmentaton mask and serves as tranng samples for learnng a scorng functon. We used the lbsvm package to learn the segmentaton class probablty P (O, m ). The Det. EER Seg. Fscore model's output gves the probablty for a segmentaton mask to be the target object foreground. RF Det. only 78.2% 79.2% RF Seg. only 97.7% 78.9% Fg. 5. Detecton performances on the TUD pedestran dataset. The combned approach perform the best among Hough based approach. 63

5 VI. CONCLUSION AND DISCUSSION The Table II shows the comparson of performance for the Wezmann horse dataset. At detecton EER, we acheve better detecton results and mproved the segmentaton qualty by 10.1% compared to the recent related method of Torrent et al. [16]. We also get hgher segmentaton qualty than the early bottom-up segmentaton method of Ren et al. [17]. Fnally we acheve worse segmentaton but smlar detecton compared to the state of the art method [18]. We have presented a smple method to effcently combne detecton and segmentaton nto a sngle framework. Both elements have been taken nto account at tranng tme, n the buldng of the vsual vocabulary and the model but also at run tme by scorng object confguratons n term of detecton and segmentaton. The experments have shown the performance mprovements of the combned approach over specalzed approach and clear mprovements n comparson to closely related methods. The algorthm performs on par wth state of the art for some of the tested dataset. As future work, we are plannng to nclude pose estmaton wthn our framework and ntroducng occluson wthn our model. TABLE II: DETECTION AND SEGMENTATION RESULTS ON THE WEIZMANN HORSE DATASET AT DETECTION EER. Methods Zhu [18] Ren et al. [17] Torrent et al. [16] Seg. Fscore 89.2% 80.2% 69.1% 80.7% Det. EER 99.1% 97.0% Image REFERENCES Comparatve results for the PennFudan dataset can be seen n Table III. We compared our results wth two works havng reported detecton and segmentaton results for ths dataset. [1] [2] TABLE III: DETECTION AND SEGMENTATION RESULTS ON THE PENNFUDAN PEDESTRIAN DATASET AT DETECTION EER. Methods Bo [5] Wang et al. [2] Fscore 82.9 % 83.7 % 78.4 % Acc % 72.8 % 64.7 % EER 85.5% 85.4% 59.5 % 80.7 % Image [3] [4] [5] We mprove detecton EER by more than 20% over Wang et al. [2] results. They dd not provde quanttatve results for the segmentaton. However vsual comparsons of the produced masks show the hgher segmentaton accuracy of our approach (see Fg. 6). Recent results [5] have been publshed for a subset of the orgnal database contanng only 101 fully un-occluded pedestrans from the orgnal 345. We acheve very smlar performances for both the detecton and segmentaton. It should be notced that Bo et al. [5] are usng state of the art bottom up segmentaton algorthm, whle our segmentaton s a purely top-down. Furthermore ther segmentaton results are heavly dependng on the detecton's boundng box. Our approach doesn't suffer from ths flaw. When tested on the full dataset, we observe a decrease n our performances due to the heavy occluson that appears wthn the test set. [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] Fg. 6. Vsual comparson of segmentaton results for the PennFudan dataset. (a) Orgnal mages. (b) Wang et al. [2] segmentaton results. (c) Our results. 64 B. Lebe, A. Leonards, and B. Schele. Robust object detecton wth nterleaved categorzaton and segmentaton, Internatonal Journal of Computer Vson, vol. 77, pp , L. Wang, J. Sh, G. Song, and I. fan Shen, Object detecton combnng recognton and segmentaton, n Proceedngs of the 8th Asan Conference on Computer Vson, vol of Lecture Notes n Computer Scence, pp Sprnger, Z. Tu, X. Chen, A. L. Yulle, and S. C. Zhu. Image parsng: Unfyng segmentaton, detecton, and recognton. In Toward Category-Level Object Recognton, pp , M. P. Kumar, P. H. S. Torr, and A. Zsserman, OBJ CUT, n Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, San Dego, vol. 1, pp , Y. Bo and C. Fowlkes, Shape-based pedestran parsng, CVPR, V. S. Lemptsky, P. Kohl, C. Rother, and T. Sharp, Image segmentaton wth a boundng box pror, n Proc. ICCV, pp , D. Ramanan, Usng segmentaton to verfy object hypotheses, n Proc. CVPR, pp. 1 8, S. Maj and J. Malk, Object detecton usng a max-margn hough transform, n CVPR 09, pp , B. Ommer and J. Malk, Mult-scale object detecton by clusterng lnes, n Proc. ICCV J. Gall and V. Lemptsky, Class-specfc hough forests for object detecton, n Proc. IEEE Conference Computer Vson and Pattern Recognton, J. Shotton, M. Johnson, and R. Cpolla, Semantc texton forests for mage categorzaton and segmentaton, n Proc. CVPR, N. Dalal and B. Trggs, Hstograms of orented gradents for human detecton, n Proc. Internatonal Conference on Computer Vson & Pattern Recognton, vol. 2, pp , June E. Borensten, Combnng top-down and bottom-up segmentaton, n Proc. IEEE workshop on Perceptual Organzaton n Computer Vson, CVPR, pp. 46, M. Andrluka, S. Roth, and B. Schele, People trackng by detecton and people detecton by trackng, n Proc. IEEE Conference on Computer Vson and Pattern Recognton (CVPR 08), E. Seemann, B. Lebe, and B. Schele, Mult-aspect detecton of artculated objects, n Proc. CVPR, vol. 2, pp , A. Torrent, X. Llad o, J. Frexenet, and A. Torralba, Smultaneous detecton and segmentaton for generc objects, n Proc. ICIP, pp , X. Ren, C. Fowlkes, and J. Malk, Cue ntegraton for fgure/ground labelng, n Y. Wess, B. Sch olkopf, and J. Platt, edtors, Advances n Neural Informaton Processng Systems, vol. 18, pp MIT Press, Cambrdge, MA, L. Zhu, Y. Chen, and A. L. Yulle, Learnng a herarchcal deformable template for rapd deformable object parsng, IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 6, pp , 2010.