A Benchmark for Interactive Image Segmentation Algorithms

Transcription

1 A Benchmark for Interactive Image Segmentation Algorithms Yibiao Zhao 1,3, Xiaohan Nie 2,3, Yanbiao Duan 2,3, Yaping Huang 1, Siwei Luo 1 1 Beijing Jiaotong University, 2 Beijing Institute of Technology, 3 Lotus Hill Institute {ybzhao.lhi,ybduan.lhi,xhnie.lhi}@gmail.com,{yphuang,swluo}@bjtu.edu.cn Abstract This paper proposes a general benchmark for interactive segmentation algorithms. The main contribution can be summarized as follows: (I) A new dataset of fifty images is released. These images are categorized into five groups: animal, artifact, human, building and plant. They cover several major challenges for the interactive image segmentation task, including fuzzy boundary, complex texture, cluttered background, shading effect, sharp corner, and overlapping color. (II) We propose two types of schemes, pointprocess and boundary-process, to generate user scribbles automatically. The point-process simulates the human interaction process that users incrementally draw scribbles to some major components of the image. The boundaryprocess simulates the refining process that users place more scribbles near the segment boundaries to refine the details of result segments. (III) We then apply two precision measures to quantitatively evaluate the result segments of different algorithm. The region precision measures how many pixels are correctly classified, and the boundary precision measures how close is the segment boundary to the real boundary. This benchmark offered a tentative way to guarantee evaluation fairness of person-oriented tasks. Based on the benchmark, five state-of-the-art interactive segmentation algorithms are evaluated. All the images, synthesized user scribbles, running results are publicly available on the webpage Introduction Image segmentation is one of the most essential problems in the field of computer vision. Although this topic has been extensively studied, common segmentation algorithms often serve as an preprocessing method of other algorithms. Automatic segmentation can hardly obtain satisfied results without high level knowledge of interest object. The 1 Figure 1. Images and ground truth labels in the benchmark dataset. There are fifty images categorized into five classes of animal, artifact, building, human and plant. Person-Oriented approach, from another point of view, focus on how we can make the state-of-the-art useable by the majority of ordinary people. The introducing of human interaction contributes to improving the performance of traditional segmentation methods towards real life application /10/$ IEEE 33

2 fuzzy boundary shading effect complex texture cluttered background sharp corners and edges overlapping color Figure 2. Some typical images in the benchmark dataset cover several major challenges for segmentation algorithms Start from Boycov et. al [2], the interactive segmentation algorithms [11] [1] [4] [5] [10] have drawn wide attention of active researchers, and the person-oriented techniques also become a hot topic in the latest decade. However, when the human interference is involved, the comparison between algorithms can be hardly objective. For automatic segmentation, Martin et. al [8] firstly provided a image database containing wide range of natural scenes and evaluated the precision of result segment boundaries. Unnikrishnan [12] proposed a similarity measure to perform a quantitative comparison between image segmentation algorithms. Recently, Kevin McGuinness [9] developed a software to calculate the feedback as a person is using a interactive segmentation algorithm. In this paper, we strive to propose a general framework to evaluate interactive segmentation algorithms. The contribution of this paper includes: (I) A complete dataset of five categories of images is publicly available. Each image category contains ten representative images, and there is at least one salient object on each image. These images cover some major challenges of image segmentation, including fuzzy boundary, complex texture, cluttered background, shading effect, sharp corner, and overlapping color. Groundtruths are precisely hand-labeled for each image. (II) Two schemes of human interaction, point-process and boundaryprocess, are proposed to objectively simulate interactive process of drawing scribbles. The point-process draws points on key components of foreground/background. The boundary-process simulates the process of the boundary refinement after the point-process. By applying these two schemes, one can automatically generate scribbles, and objectively evaluate the performance of interactive algorithms without human bias. (III) Two criteria of region and boundary precision are further applied to evaluate the region coverage and boundary proximity for those two processes. An overview of our benchmark is shown in Fig.1. The reminder of this paper is organized as follows: In section.2, we start from introducing the design of the dataset, and section.3 presents the idea of interaction simulation. In section.4 we describe the details of our evaluation methodology, and analyze the performance of algorithms based on the quantitative results on our benchmark. And the paper concludes in section Dataset design The dataset contains fifty images from LHI database [13]. These images are selected from the categories of animal, artifact, building, human and plant. The animal category is the most challenging one, which contains some wild animal images with fuzzy boundaries and the complex textures. Some animals also have very similar color appearance to overlapped with background color. The overlapping color between object and environment make a simple foreground color distribution is hard to be distinguished from the background. Opposite to animal category, the artifact category has relative clear boundaries and smooth appearance, while the shading effect also make the color distributions to widely spread even overlap with each other. The building category contains some textured background and structured foreground, it looks easy to be classified. However, some algorithms have problems to deal with the sharp edges and corners on the building. The cluttered background usually appears in the human category. Besides that, the color distributions for human clothes are sometimes multimodal, it is challenging for some parametric color models. In the images of plant category, boundaries are very smooth, so the zigzag effect for some discrete optimization method will be visually apparent. The ground truths of all images are human labeled. Our database is publicly available on the project website. 34

3 Figure 3. The two types of simulation named point-process(top row) and boundary-process(bottom row). The level of point-process increases from left to right to test the algorithms ability to cover region. In the bottom row the gap near boundary become bigger from left to right make locating boundary harder. With the increase of level from one to four, it become more difficult to get a satisfy segmentation result. 3. Interaction simulation In person-oriented applications, person is in the loop of algorithm iteration. The algorithm is running based on the response of users. Therefore, the experiment results can be easily manipulated by human interference. In order to offer a fair benchmark to evaluate different algorithms, we propose two schemes to automatically simulate the interactive process of person drawing scribbles, as illustrated in Fig Point-process simulation Users label some key components of an image to indicate the major features of the foreground and background, and expect that the computer can predict desired labels to the remaining parts of the image. Point-process simulation generates several points to represent key features in the image. In our method, we use k-means algorithm to capture several clusters on color space, each cluster is an important color in the image. We then sample a pixel according to the most representative color in each cluster. The 10*10 area around the pixel is labeled to correspondent label as an initial scribbles. In order to evaluate the performance of algorithms progressively, we define four levels with the increasing numbers of clusters in GMM. Either foreground region or background region has only three clusters in level one. On the level four, there are up to 50 clusters for each label Boundary-process simulation We provide another simulation named boundaryprocess. It simulates the refining process that users place more scribbles near the segment boundaries to refine the details of result segments. In this process, we give the inner part of foreground and background with known labels, and only leave a band near the boundary to be proceed. This input is used to evaluate how precise is the result segment with major part labeled. We also defined four levels of input with the decreasing width of unlabeled area. In level one the width is 40 pixels and in level four it decrease to 10 pixels. 35

4 Method Animal Artifact Building Human Plant bp rp bp rp bp rp bp rp bp rp Boycov et al. [2] ICCV Bai et al. [1] IJCV Couprie et al. [4] ICCV Grady [5] PAMI Noma et al. [10] CoRR Table 1. The segmentation precision on five image category of five algorithms. Method boundary-process level point-process level Boycov et al. [2] ICCV Bai et al. [1] IJCV Couprie et al. [4] ICCV Grady [5] PAMI Noma et al. [10] CoRR Table 2. The segmentation precision on four different simulation level of five algorithms. 4. Quantitative evaluation With a dataset containing some challenging images and two simulation scribbles, we then need quantitative evaluation criteria for the two simulation. For the results generate by point-process simulation we apply a criterion to evaluate region coverage. For boundary-process the most region are pre-labeled leaving the gap near boundary to algorithms to handle, so here we evaluate the proximity between result boundary and desired boundary Region coverage We denote the overlapping ratio of the foreground object as the region segmentation precision, RP (Λ R, Λ G R) = Λ R Λ G R / Λ R Λ G R. (1) where Λ R and Λ G R are the foreground region of segment result and ground-truth respectively. The region coverage ratio RP (Λ R, Λ G R ) is the ratio of intersection to the union of Λ R and Λ G R, and the output is a real value range from 0 to 1 where one means every pixel is labeled correctly Boundary proximity With more areas marked by point-process, almost all methods can obtain a course results without major failure. In this time, users will focus more on the boundary. The input scheme of boundary-process is designed to evaluate ability of precise boundary locating. By introducing an undirected chamfer distance, we define the boundary locating accuracy as : BP (Λ B, Λ G B) = 1/ u min vd(u, v) + v min ud(v, u) v + u where u Λ B and v Λ G B are the pixels on result boundary and ground-truth boundary, v and u denotes the number of pixels on the corresponding boundaries. This is a more rigorous metric of amplifying the subtle difference between result and ground-truth Results and analysis In order to provide a set of baseline results and evaluate the state-of-art algorithms on the new dataset, we test five representative algorithms: Graph cuts [2], Geodesic matting [1], Random walker [5], Power watersheds [4] and Structural Interactive Segmentation [10]. The region coverage precision (rp) and boundary proximity precision (bp) of the five algorithms on each image category are shown in Table.1. The animal category have the lowest boundary proximity precision, while the artifact category is on the contrary. The human category get the poor region coverage precision because of the clustered background, while the algorithms can easily extract the foreground in plant category due to the nearly independent color distribution. From the experimental results, one can easily see that the Graph cuts [2] performs good on the human category, while Power watersheds [4] can handle the sharp edges exist in artifact object and building images. Table.2 shows the segmentation precisions of these algorithms on four simulation levels. The highest region coverage precision of Power watersheds [4] on every simulation level reveals its excellent region extracting ability. It (2) 36

5 is interesting to notice that with the increase of the simulation level the performance of Random walker [5] is close to Power watersheds. The Structural Interactive Segmentation [10] beat all others with boundary based interaction. This reveal this algorithm possess strong ability to attract the result boundary towards the real one, once the major components of the image are labeled. 5. Conclusion In this paper, we propose a general benchmark to evaluate interactive segmentation algorithms. We collect a diverse dataset of natural images. The dataset is composed of five categories. Two interactive simulation schemes are proposed to simulate the user interaction process. Two criteria are applied to evaluate the region coverage and boundary proximity of the two schemes. Five state-of-art algorithms are evaluated and all the experimental results with the dataset are available at website 2. In the future, we are planning to extend the dataset to contain more images and more categories e.g. medical images and aerial images. Other interactive techniques, like bounding-box-based interaction [11] or boundary-based interaction [6] are also interesting to us. The goal of our work is pushing the boundaries of algorithm performance, and enlighten new idea for person-oriented tasks. 6. Acknowledgments This work at Beijing Jiaotong University is supported by China 863 Program 2007AA01Z168, NSF China grants , , , , , Beijing Natural Science Foundation , and Doctoral Foundations of Ministry of Education of China And the work at the Lotus Hill Institute is supported by China 863 Program 2007AA01Z340, 2009AA01Z331 and NSF China grants , [3] V. Caselles, R. Kimmel, and G. Sapiro. Geodesic active contours. International Journal of Computer Vision, 22:61 79, [4] C. Couprie, L. Grady, L. Najman, and H. Talbot. Power watersheds: a new image segmentation framework extending graph cuts, random walker and optimal spanning forest. In In Proceeding of International Conference on Computer Vision., [5] L. Grady. Random walks for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11), [6] M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. International Journal of Computer Vision, 1: , [7] V. Lempitsky, P. Kohli, C. Rother, and T. Sharp. Image segmentation with a bounding box prior. In In Proceeding of International Conference on Computer Vision., [8] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In In Proceeding of International Conference on Computer Vision., volume 2, [9] K. McGuinness and N. E. O Connor. A comparative evaluation of interactive segmentation algorithms. Pattern Recognition, 43, [10] A. Noma, A. B. V. Graciano, L. A. Consularo, R. M. J. Cesar, and I. Bloch. A new algorithm for interactive structural image segmentation. Technical report, [11] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics, 23: , [12] R. Unnikrishnan, C. Pantofaru, and M. Hebert. Toward objective evaluation of image segmentation algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6): , April [13] B. Yao, X. Yang, and S.-C. Zhu. Introduction to a largescale general purpose ground truth database: Methodology, annotation tool and benchmarks. In In Proceeding of Energy Minimization Methods in Computer Vision and Pattern Recognition., pages , References [1] X. Bai and G. Sapiro. Geodesic matting: A framework for fast interactive image and video segmentation and matting. International Journal of Computer Vision, 82, [2] Y. Y. Boykov and M. P. Jolly. Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images. In In Proceeding of International Conference on Computer Vision., volume 1, pages vol.1,

6 Figure 4. A screenshot of two browse mode on our web pages. The left one shows the segmentation results obtained from different algorithms. The bar plot shows the region precision for each algorithms. The right page shows several results by Graph Cuts. These images are from the category of human, and input is selected as point-process level 3. 38