Contour Box: Rejecting Object Proposals Without Explicit Closed Contours

Transcription

1 Contour Box: Rejeting Objet Proposals Without Expliit Closed Contours Cewu Lu, Shu Liu Jiaya Jia Chi-Keung Tang The Hong Kong University of Siene and Tehnology Stanford University The Chinese University of Hong Kong Shanghai Jiao Tong University Abstrat Closed ontour is an important objetness indiator. We propose a new measure subjet to the ompleteness and tightness onstraints, where the optimized losed ontour should be tightly bounded within an objet proposal. The losed ontour measure is defined using losed path integral, and we solve the optimization problem effiiently in polar oordinate system with a global optimum guaranteed. Extensive experiments show that our method an rejet a large number of false proposals, and ahieve over 6% improvement in objet reall at the hallenging overlap threshold 0.8 on the VOC 2007 test dataset.. Introdution Objet detetion benefits from objet proposal generation. A reasonable number of objet proposals an speed up the detetion proess and avoids exhaustive sliding window searh. Closed ontour is a useful ue in generating objet proposals. To find image ontours, in [5], the observation that objets with losed boundaries an be disriminated by the norm of gradients in pathes was made use of. In [26], a simple box objetness sore was proposed to measure the number of edges in the box exluding those who overlap with the box boundary. Albeit effetive, these methods do not expliitly output a losed ontour, beause of the intrinsi hallenge that there exist an exponential number of ways for pixels to onnet to form losed urves. This paper expliitly seeks an optimal losed ontour for objet proposal generation. There are two issues to onsider, i.e., the mathematial definition of a losed ontour in the ontext of objet proposals, and an effiient optimization algorithm to ompute the optimal losed ontour given the large number of andidates within a given image window. This researh is supported by the Researh Grant Counil of the Hong Kong Speial Administrative Region under grant numbers 697 and 429. The sublass of losed ontours we interested in is modeled using the losed path integral subjet to the ompleteness and tightness onstraints. We ompute the optimal ontour by transforming the solution spae to the polar oordinate system. A spatial losed ontour beomes a ontinuous path in polar oordinate system. Interestingly, this problem beomes the shortest path searhing, whih an be solved exatly and effiiently by dynami programming. The result is guaranteed to be a global optimum with the time ost insignifiant ompared to naive searh and omparison. Every optimized ontour is assigned a ontour s- ore to reflet the onfidene. Our seond ontribution is to employ the ontour s- ores in objet proposal generation. Unlike previous work, whih prefers high-soring proposals to indiate objetness, we use the ontour sore to prune non-objet proposals. The reason for not aepting high-sore proposals is that a losed ontour satisfying both the ompleteness and tightness onstraints does not always orrespond to a semanti objet. On the other hand, if a proposal does not have suh a losed ontour, it stands a high probability that this is a non-objet proposal. Figure shows a number of examples. Our empirial study over 50K proposals shows that 99.% of low-sore windows are regarded as non-objet proposals. Based on this finding, we propose a new objetness sheme that rejets proposals unlikely to be an objet. At the time of submitting this paper, our method yields an improvement of 6% over the best one regarding objet reall with the intersetion-over-union (IoU) overlap set at 0.8. Our evaluation is done on the PASCAL VOC dataset [9]. 2. Related Work The goal of generating objet proposals is to produe a set of reliable bounding boxes that likely to ontain objets. With a signifiant redution in the number of andidate boxes, omputationally expensive lassifiers [2, 23, 0] an be employed in objet detetion to ahieve state-of-the-art detetion performane. This pipeline has been widely adopted in objet detetion, whih further motivates researh on improving objet proposal generation. Previous methods an

2 (a) (b) () (d) (e) (f) (g) (h) Figure. If there is no losed ontour near the boundary, the hane that this window ontains a omplete semanti objet is small. A few examples are shown here with their edge maps. Note that () is not an objet window sine it does not tightly enlose the the flower pillow. be lassified into two ategories: produing a large number of proposals in the viewpoint of segmentation and the use of objetness ranking. Seletive searh [2] is a well-known method in the first ategory, whih merges super-pixels in a greedy manner. When two super-pixels are merged, a bounding box is reated to tightly bound them. This proess ontinues until there is only one super-pixel left. In order to generate a number of good bounding boxes, several strategies were proposed for reasonable diversity. The seletive searh work was extended in [25], where more features are added and weights are learned. In [5], randomness is onsidered during the merging proess. In [4, 8, 9, 24], segmentation is employed while different objet proposals are generated by enumerating many foreground-bakground masks. Their omputation ost is generally high. The method related to ours [6] uses a fixation point to define the polar spae for finding the optimal ontour where the fous is ative visual segmentation. A feedbak proess is used to establish the relationship between edges and regions, whereas our ontour box only uses an edge map. In [2], the struture of super-pixel graph is reused to speed up omputation. A fast normalized-ut algorithm was proposed in [2] to generate objet proposals by grouping image segments. Most of these methods an generate high-quality segmentation masks. However, without a good objetness sore, it is diffiult to redue the large number of boxes into a reasonable number to failitate subsequent proesses. For the seond ategory, they seek different ues to desribe objets. In [], a number of objetness ues, suh as objet ontour, olor, ontrast, edge, edge density and salient map are used to sore a bounding box. Then the sampled bounding boxes are ranked by the generated objetness sores. This method was modified in [8] by hanging the features and lassifiers. In [4], proposals were ranked by performing regression between features and their atual overlap with an objet. In [5], a linear lassifier was proposed to alulate the objetness sore based on gradient features, while in [26] edges were used to alulate the objetness sore by ounting the number of ontours that are wholly ontained in a bounding box and removing those who straddle the box boundary. In [7], a ategory-independent proposal generation method was proposed. In [3], Blashko et al. omputed additional objetness features at a low ost based a nonmaximal suppression (NMS) sheme on the asade arhiteture. Still, the method does not exploit the ue of objet ontour expliitly. Differently, we disover objet ontours and sore them in an optimization framework. We also review other methods related to losed urves using dynami programming in polar oordinate system. In [20], a dynami programming-based algorithm was proposed to extrat multiple paths using onstrained expanded trellis (CET) for feature extration and objet segmentation. In [6], Gore et al. proposed dihotomi multiple searh (DMS) to find the global minimum as the irular shortest path. While we share some similarity in terms of dynami programming and polar oordinates, the new ompleteness and tightness onstraints are important and unique to make our method work for proposal generation. 3. Contour Box: Optimizing Closed Contours In this setion, we desribe our method to optimize a losed urve that predits an objet silhouette tightly bounded by an image window. We first desribe the way to model a losed ontour subjet to the ompleteness and tightness onstraints. Then an effiient solution is presented to ompute it. 3.. Closed Path Integral The edge intensity at pixel loationxis denoted bye(x). Our optimized ontour is defined as a losed urve. In essene, edge evidene along the urve should be as omplete as possible while the urve should be lose to the window border. In our implementation, we ompute edges using the method of [7]. Completeness onstraint An intuitive definition of omplete edge evidene requires the mean edge strength along a losed urve to be suffiiently large. However, this definition annot tell whether salient edges along a losed urve

3 (a) objet (b) non-objet Figure 2. Two losed urves (in red) optimized with and without adjusted edge evidene. Without the adjustment, the mean edge intensities of the urves are similar, whih are and respetively. After the adjustment, the mean edge intensities of the right urve drops to while the left remains unhanged. are missing or not. One example is shown in Figure 2(b), where the mean edge strength along the red urve is reasonably strong. But there is a segment missing in order to form the losed urve. We propose a simple adjustment of the edge map as { y y > τ φ[y] = () γ y τ where τ = 0.00 and γ = are onstants. Empirially, γ is in range [0.8, 2]. This serves to penalize edge intensity belowτ or missing edges. The parameterτ is not ritial in our experiments. With the adjusted edge map, we define the ompleteness ondition as maximizing the objetive funtion φ[e(x)]dx dx, (2) where is a losed urve and dx is a path length normalization fator. Without this fator, maximizing Eq. (2) would produe a unreasonably long urve that maximizes the total edge values. Tightness onstraint Aording to the definition of objets, every window should tightly enlose an objet. Therefore, we expet the losed urve is near or on window borders. Put differently, we prefer a losed urve whose pixels are reasonably far from entroid of the window. To this end, we define the normalized distane to the window entroid for every pixel as u(x) = [2/h,2/w] (x x 0 ) 2, where x 0 is the entroid, is the point-wise produt, and w and h are respetively the width and height of the given image window. We adjust u(x) using a nonlinear funtion that penalizes non-losed urves as ϕ[y] = min{y 2,0.7} (3) to update the results. While there exist other non-linear adjustment funtions, Eq. (3) is good enough in our experi- Figure 3. Average map of all objet boundaries in the VOC 202 segmentation dataset. All objet images are resized into before averaging. ments. To impose the tightness onstraint, we maximize ϕ[u(x)]dx dx. (4) Our objet proposal generator is ategory-independent without any presumed shape priors, while the tightness onstraint impliitly prefers a round (or elliptial) shape, whih is a soft onstraint. To justify our hoie, Figure 3 shows the average map of all the objet boundaries in the VOC 202 segmentation dataset. A rough annulus shape near the boundary an be observed. Objetive funtion We ombine the two terms desribed above to form the objetive funtion max C dx{ φ[e(x)]dx+λ ϕ[u(x)]dx}, (5) where C is the set of all losed paths in the window, and λ is a parameter to be set during training. If Eq. (5) outputs a small energy, there is no suitable objet ontour that is omplete and tight Solution in Polar Coordinates We now desribe how to solve the energy funtion (5). Note the solution spae of Eq. (5) is prohibitively large due to the large number of possible losed paths in the set C. Further, due to the normalized fator dx, popular path optimization algorithms annot be applied diretly. To address these issues, we map our onfiguration from the Cartesian oordinate system to the polar oordinate system. An example is illustrated in Figure 4. A losed path in x-y oordinate system is mapped to a ontinuous one-way path in the polar oordinate system, where the radial oordinate r and angular oordinate θ are obtained with respet to entroid of the image window. We use a linear interpolation approah to ahieve disrete mapping. Our polar oordinate map is stored as a m n matrix. That is, we quantize 360 into m bins and radii into n bins. We use m = 90 and n = 50 in our experiments.

4 (a) (b) Figure 4. Mapping a urve from (a) Eulidean oordinates to (b) polar oordinates. We then rewrite Eq. (5) as max C dx{ M(x)dx}, (6) where M(x) = φ[e(x)]+λϕ[u(x)]. (7) In polar oordinates, M(x) is denoted as g(r, θ). Interestingly, the problem of optimizing our omplete and tight losed ontour in Eq. (5) is equivalent to seeking the optimal ontinuous path in polar oordinates from top to bottom in our m n matrix. Thus we optimize a ontinuous path starting at (,k) in polar oordinates and ending at a point near (m, k), whih ensures that the optimized path is losed. We solve the optimization problem of E(k,g) = max { m g(a[i],i)} a i= s.t. a[i] a[i ] δ, i = 2,...,m (8) a[] = k, a[m] k δ, (9) where a denotes a path and every a[i] is the radial oordinate with angle i. Eq. (8) enfores the onstraint that the optimized urve should be ontinuous in polar oordinates. We set δ = 3 in our experiments. The onstraint (9) presribes that a onnets the starting point to the ending one, sineais losed. This problem is a standard shortest path searh one, whih seeks an optimal path from the soure point (,k) to{(m,k δ),...,(m,k+δ)}. It an be effiiently solved by dynami programming where the global optimum exists. The omputation amounts to heking n possible starting points, and then seleting the best path maxe(k,g). (0) k By optimizing Eq. (0), we obtain the required path and its energy the latter is used to rejet non-objet proposals. Disussion on the Two Coordinate Systems In our disrete solution, a urve is omposed of exatlymsegments, represented bymelements in the polar matrixm. Every element in the polar matrix reords the mean intensity of the orresponding segment in the Eulidean oordinate system. All urves are normalized in terms of length. The omputation in the polar oordinates is simple. Cheking n starting points redues the time omplexity to the order of the matrix resolution. Also, all ontinuous paths from top to bottom have the same length, thus naturally aneling out the length normalization fator dx in the original objetive funtion, whih explains why dynami programming an now be applied. The time omplexity of our method is O(mn 2 ), whih is onstant w.r.t. the window size. Notwithstanding, there exist losed urves in the original oordinate system that annot be mapped to a ontinuous path in the polar oordinate system. These exeptions mainly arise when entroid of the image window is not enlosed by the optimal urve. But we found this ase is rare. In the VOC 2007 training data, 99.3% of the objet boundaries enlose entroid of the image window. In our experiments, the differene of solutions respetively produed in the two oordinate systems is insignifiant Non objet Indiation As disussed above, our ontour sore is used as a nonobjet indiator. We determine the non-objet threshold T and label those proposals whose ontour sores are lower than T as non-objets. To this end, we manually label 40,000 proposals as with ontour and without losed objet ontour respetively on pasal VOC 2007 dataset. Then, we seek the bestλfor Eq. (7) andt together in these two ategories. Finally, we label proposals whose ontour sores are smaller thant as non-objets. 4. Experiments Following previous work [5, 5, 26, ], we evaluate our method on the VOC2007 dataset [9]. We ompare our results with those state-of-the-arts [2, 5, 9, 26, 2, 5, ]. The simple method of omputing losed urves using levelset [4] is also ompared with, sine it is a famous ontour generation strategy. We use the author-released implementations with reommended parameter settings. To evaluate the reall of bounding boxes, we use the Intersetion-over-Union (IoU) metri, whih omputes the intersetion of a andidate box and the ground-truth divided by the area of their union. We report the performane at the hallenging IoU threshold Differene in the Two Coordinate Systems We solve the problem in Eq. (5) in the polar oordinate system. We arry out the following experiments to veri-

5 Detetion rate Proposals SS MCG Edgebox70 Edgebox90 Bing Rantalankila Objetness Rand. Prim s Level Set Ours(C only) Ours(C+S+A) Detetion rate Edgebox Bing Objetness Ours Intersetion over union Figure 5. Reall with different IoUs using,000 proposals. ours (C only) means ontour sore only. ours (C+S+A) is the linear ombination of ontour sore, size and aspet ratio. Detetion rate SS MCG Edgebox70 Edgebox90 Bing Rantalankila Objetness Rand. Prim s Ours IoU = Number of proposals Figure 6. Reall with different proposal numbers at IoU 0.8. fy that our solution is a good approximation of the original one. We randomly selet 20,000 proposals and ompute their ontour sores in both the Eulidian and polar oordinate systems. Then, we solve the original Eq. (5) by exhaustively searhing all possible losed urves. This is very time-onsuming and takes about 2 days to finish. We evaluate a ranking distane metri to evaluate the two results, sine we are onerned more with the relative sore for rejeting non-objet proposals. The ranking distane we use is Kendall-tau rank distane, whih is defined as #{onordant pair} #{disordant pair} 2 n(n ). () n is the total number of sores (20,000 here). The Kendalltau rank distane lies in the interval [, ]. Normally, a distane with 0.8 indiates that two sore lists are very lose in terms of element ranking. The distane of the results omputed in the two oordinate systems is 0.97, whih indiates that these two sets of results are almost idential Intersetion over union Figure 7. Reall of non-objet proposals labeled by different methods (false positive rate) Proposal Quality Comparisons In this setion, we empirially show that our Contour Box, whih adopts non-objet proposal rejetion using ontour sores, improves objet reall and ahieves state-ofthe-art performane. Following [5, 5, 26, ], we evaluate our results on the VOC2007 test set, whih onsists of 4,952 images with bounding box annotations for objets in 20 d- ifferent ategories. We evaluate our performane using the standard reallover-iou performane over,000 proposals: every method reports,000 bounding boxes and its reall is tested over d- ifferent IoUs. To produe,000 proposals, we first perform the seletive searh [2] to generate around 0,000 proposals for every image. Then our ontour sore is used to disard proposals labeled as non-objet. The average number of proposals we remove is per image. Finally, we report,000 proposals by randomly sampling from the remaining proposals. To explore the statistis of objet boxes, we also ombine aspet ratio and size of objet boxes by weighting them with the ontour sore. The 2D weight is learned using linear SVM. Here, objet aspet ratio and size are respetively h w, hw HW, wherehandw are respetively height and width of the bounding box, H and W are respetively height and width of the image. Figure 5 shows the statistial omparison between our method and SS [2], MCG [2], BING [5], EdgeBox [26], Rantalankila [9], Objetness [], Prime proposal [5], Rahtu [7], and Blas [3]. We tune the ode of Rahtu [7] and Blas [3] and report urves with the best reall at IoU= 0.8. In [26], a high IoU (0.7) is reommended for objet detetion. Our method works well with an even higher IoU. We observe about 6% improvement at the hallenging overlap threshold 0.8. When evaluated within the IoU range [0.9, ], our method is still omparable to state-of-the-art

6 ( a ) ( b ) ( ) ( d) ( e ) ( f ) ( g ) ( h) Figure 8. Result examples with optimized losed paths. In every example, the left and right maps are respetively the results in the image and polar oordinate systems, while the top and bottom are the original images and proposals with the optimal paths highlighted in red. (a) (d) are four examples with losed ontours where sores are 02.5, 90.4, 89.2 and 0.9 respetively. () (d) are two examples without losed ontours and the sores are 34.9, 25.6, 27.2 and 7.9 respetively. methods [5] [2] [2] [26] ours reall (in%) Table. We randomly pik 200 oluded objets from the VOC 2007 and 202 datasets. The reall at IoU 0.8 for different methods given 000 proposals are reported. methods. Note our method applies the new ontour sore to pik,000 proposals from seletive searh [2]. They are signifiantly better than the,000 proposals reported by seletive searh at almost all IoUs. To show more about the performane, Figure 6 plots the detetion rates under varying numbers (from to 000) of objet proposals for overlap 0.8. Our method runs best s- tarting at number Oluded Objets Partially oluded objets are ommon in images. Our method also works to an extent for these objets sine the non-oluded part still forms a omplete and tight losed path. Empirially, we randomly pik 200 oluded objets from the VOC 2007 and 202 datasets. Table reports the IoU [3] (in%) Ours (in%) Table 2. Reall on applying our ontour rejetion sheme to the proposals generated by [3], where the number of proposals is 000. reall at IoU Soring on [3] To demonstrate that our ontour soring is general, we apply our method to another proposal generator [3] besides seletive searh. For every image in VOC 202, our ontour rejetion method works on the boxes generated by the method of [3]. We report the reall performane on different IoUs in Table 2. The result shows that our ontour soring ahieves about 4.4% reall improvement at IoU Non objet Identifiation Evaluation A number of methods, suh as [, 26, 5] provide objetness sores, whih an also be used to label non-objet proposals. In our experiments, we sort proposals in a desend-

7 Figure 0. Examples of objet proposals. Our generated proposals are shown in red. Figure. Examples of labeled non-objet proposals are shown as green bounding boxes. ing order regarding the objetness sores, and label the last k proposals as non-objets. To make omparison fair, we set k as the number of non-objet proposals we have labeled for every image using our method. The proposals used for soring are provided by the seletive searh [22], whih produes about 0,000 boxes for every image.

8 (a) (b) () (d) (e) (f) (g) (h) (i) Figure 9. (a) () input edge maps; (d) (f) level set results; (g) (i) our results. The level set method [4] falls short to ompute the desired objet without onsidering the ompleteness and tightness onstraints. Note that we an rejet ase () easily due to its low ompleteness sore. We report the reall values of objet proposals that are mistakenly labeled as non-objet proposals. A small reall means that a small number of objet proposals are mistakenly labeled as non-objets. Figure 7 shows results. Our method is better than others using respetively alulated objet sores for non-objet identifiation. Figure 8 shows examples with optimized losed paths and their orresponding ontour sores in the two oordinate systems. Some objets in those examples have ompliated strutures. The level-set method is a standard tehnique for omputing losed paths. It is less optimal in our ase as shown in Figure 5 without exploiting the ompleteness and tightness onstraints. A few failure ases are shown in Figure 9. Figure 0 shows high-sored proposals. While proposals rejeted by our method are shown in Figure. In the lowsored non-objet proposals, the optimized losed urves do not have salient edge evidenes and/or are not tightly bounded by the window. The running time of our method is about seonds on a PC using a single thread. 5. Conlusion and Future Work We have presented a new method to expliitly ompute objet ontours subjet to the ompleteness and tightness onditions. We ompute these ontours in the polar oordinate system where a omplex searh problem beomes an easily tratable dynami programming problem. As every generated ontour has a sore, it an be used to selet good proposals. Our new sheme is not based on the best sores but instead to sample proposals after removing the low-sore proposals beause they are mostly non-objets. We have validated our non-objet proposal rejetion sheme. Our performane is reasonable. Our future work will be to find other reliable features for identifying objets and ombine them with our ontour ues. The exeutable will be publily available in our projet website. Referenes [] B. Alexe, T. Deselaers, and V. Ferrari. Measuring the objetness of image windows. IEEE Transations on Pattern Analysis and Mahine Intelligene (PAMI), 202. [2] P. A. Arbeláez, J. Pont-Tuset, J. T. Barron, F. Marqués, and J. Malik. Multisale ombinatorial grouping. In CVPR, 204. [3] M. B. Blashko, J. Kannala, and E. Rahtu. Non maximal suppression in asaded ranking models. Image Analysis, pages , 203. [4] J. Carreira and C. Sminhisesu. CPMC: automati objet segmentation using onstrained parametri min-uts. IEEE Transations on Pattern Analysis and Mahine Intelligene (PAMI), 202. [5] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P. Torr. Bing: Binarized normed gradients for objetness estimation at 300fps. In CVPR, 204. [6] M. de La Gore and N. Paragios. Fast dihotomi multiple searh algorithm for shortest irular path. In ICPR, 203. [7] P. Dollár and C. L. Zitnik. Strutured forests for fast edge detetion. In ICCV, 203. [8] I. Endres and D. Hoiem. Category-independent objet proposals with diverse ranking. IEEE Transations on Pattern Analysis and Mahine Intelligene (PAMI), 204. [9] M. Everingham, L. J. V. Gool, C. K. I. Williams, J. M. Winn, and A. Zisserman. The pasal visual objet lasses (VOC) hallenge. International Journal of Computer Vision (IJCV), 200. [0] R. B. Girshik, J. Donahue, T. Darrell, and J. Malik. Rih feature hierarhies for aurate objet detetion and semanti segmentation. In CVPR, 204. [] J. Hosang, R. Benenson, and B. Shiele. How good are detetion proposals, really? In BMVC, 204. [2] A. Humayun, F. Li, and J. M. Rehg. RIGOR: reusing inferene in graph uts for generating objet regions. In CVPR, 204. [3] P. Krähenbühl and V. Koltun. Learning to propose objets. In CVPR, 205. [4] C. Li, C. Xu, C. Gui, and M. D. Fox. Distane regularized level set evolution and its appliation to image segmentation. IEEE Transations on Image Proessing (TIP), 200. [5] S. Manen, M. Guillaumin, and L. J. V. Gool. Prime objet proposals with randomized prim s algorithm. In ICCV, 203.

9 [6] A. K. Mishra, Y. Aloimonos, L.-F. Cheong, and A. Kassim. Ative visual segmentation. IEEE Transation on Pattern Analysis and Mahine Intelligene, 34(2): , 202. [7] E. Rahtu, J. Kannala, and M. Blashko. Learning a ategory independent objet detetion asade. In ICCV, 20. [8] E. Rahtu, J. Kannala, and M. B. Blashko. Learning a ategory independent objet detetion asade. In ICCV, 20. [9] P. Rantalankila, J. Kannala, and E. Rahtu. Generating objet segmentation proposals using global and loal searh. In CVPR, 204. [20] C. Sun and B. Appleton. Multiple paths extration in images using a onstrained expanded trellis. IEEE Transations on Pattern Analysis and Mahine Intelligene (PAMI), 27(2): , [2] J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders. Seletive searh for objet reognition. International Journal of Computer Vision (IJCV), 04(2):54 7, 203. [22] K. E. Van de Sande, J. R. Uijlings, T. Gevers, and A. W. S- meulders. Segmentation as seletive searh for objet reognition. In ICCV, 20. [23] X. Wang, M. Yang, S. Zhu, and Y. Lin. Regionlets for generi objet detetion. In ICCV, 203. [24] Y. Xiao, C. Lu, E. Tsougenis, Y. Lu, and C.-K. Tang. Complexity-adaptive distane metri for objet proposals generation. In CVPR, 205. [25] V. Yanulevskaya, J. R. R. Uijlings, and N. Sebe. Learning to group objets. In CVPR, 204. [26] L. Zitnik and P. Dollar. Edge boxes: Loating objet proposals from edges. In ECCV, 204.