Robust Frequency-tuned Salient Region Detection

Transcription

1 Robust Frequency-tuned Salient Region Detection 1 Yong Zhang, 2 Yihua Lan, 3 Haozheng Ren, 4 Ming Li 1 School of Computer Engineering, Huaihai Institute of Technology, Lianyungang, China, zhyhglyg@126.com *2,Corresponding Author School of Computer Engineering, Huaihai Institute of Technology, Lianyungang, China,lanhua_2000@sina.com 3 School of Computer Engineering, Huaihai Institute of Technology, Lianyungang, China, renhaozheng666@163.com 4 Imaging Processing Business Dept of Beijing E-COM Technology CO., LTD., Beijing , China Beijing, China, @qq.com Abstract Detection of visually salient image regions has been of great research interest in recent years. It is useful for a wide range of applications such as object detection, video summarization and object segmentation. In this paper, we propose a modified method based on frequency-tuned (FT) model for salient region detection that outputs full resolution saliency maps with well-defined boundaries of salient objects. For FT model, saliency is considered as a distance to the mean spatial frequency content that is too simple for complex scenes and inconsistent with our usual sense. Our modified model, which called as Robust FT, following the assumptions of salient region, can get better saliency maps. We consider the mean spatial frequency content as background and formulate a much more robust model to descript it. By practical experiments, it is verified that our model obtains better results than original methods. 1. Introduction Keywords: Visual Saliency; Salient Region; Image Analysis Salient regions are generally regarded as the candidate of attention focus in human eyes. Each pixel in a salient region describes an important point of the object in the image or scene that is projected into the human visual system in a special location. The purpose of detecting salient region is to direct eye movements to the most relevant parts of the visual field [1-3]. It is also very useful for computer vision as they offer an efficient approach in processing complex scenes by locating those regions of interest for directing analysis in a similar way that is compared with human visual system. Therefore, detecting salient regions is a very important stage in object detection, which is widely used in object recognition, object tracing and image compression, to name a few [4-5]. Thus a number of researchers are attracted in this field for developing and investigating efficient techniques to detect salient regions. For the time being, many algorithms have been proposed to simulate this process. Among them, a model for generating salience maps which proposed by Koch and Ullman is recognized as the most influential approach [6]. In their studies, they proposed several viewpoints: first of all, they thought the different topographical maps which are called as early representation by them can be represented in parallel by a number of elementary features, such as color, orientation and so on. Secondly, they thought there is a selective mapping from the early topographical representation into a more central non-topographic representation. At last they proposed certain selection rules to determine which locations can be mapped into the central representation [6]. By using those viewpoints and methods, they attempted to address the question that how simply networks of neuron-like elements can account for a variety of phenomena associated with this shift of selective visual attention. In the end of their literature, Koch and Ullman suggest a possible way which contains three different stages for selective visual attention based on the discussions in their study. Since then, the model proposed by Koch and Ullman has been further researched by many researchers over three decades [7-8]. L. Itti, C. Koch and E. Niebur proposed a model of saliency-based visual attention for rapid scene analysis. Based on the International Journal of Digital Content Technology and its Applications(JDCTA) Volume6,Number20,November 2012 doi: /jdcta.vol6.issue

2 behavior and early primate visual system proposed by Koch and Ullman, L. Itti, C. Koch and E. Niebur combined multi-scale image features, such as colors, intensity and orientations etc., into a single topographical saliency map, then they used a dynamical neural network to select attended locations. The experimental results in their literature showed that their system can select a salient region to be analyzed detailedly by using a computationally efficient manner [7]. D. Walther and C. Koch propose a biologically plausible model of forming and attending to proto-objects, the experimental results showed that the model proposed by them can enable a model of object recognition in cortex to expand from recognizing individual objects in isolation to sequentially recognizing all objects in a more complex scene [8]. Paul L.Rosin proposed a simple method for detecting salient regions [1], the model only use edge detection, threshold decomposition, the distance transform and threshold and avoid the need to set any parameter. Experiments show that their model is effective and easy to be implemented. Although so many methods have been developed, most of them generate regions having low resolution and poorly defined borders. Recently, Achanta et al. [9] illustrated that most techniques in this area primarily operate using extremely low-frequency content in the image, and then they introduced a fast frequency-tuned (FT) approach for salient region detection that outputs full resolution maps with well-defined boundaries. The experimental resulting saliency maps got both higher precision and better recall than five state-of-the-art algorithms (including Itti et al. [10], Ma and Zhang [11], Harel et al. [12], Hou and Zhang [13] and Achanta et al. [14]) on object segmentation. In this paper, we first cover associated shortcomings of FT method, and discuss preliminary solutions of them. And then we present our novel method, which modifies the original one to achieve better performance. The experiments and results demonstrate that our method is a promising way. The rest of this paper as follows: in section 2, the problems of FT method are analyzed. Then the proposed robust FT method by us is introduced in section 3. Section 4 given the experimental and corresponding results. Discussion based on the experimental in section 3 is also presented. At last, conclusions will be given in the end of section Problems of FT method To develop the FT method, Achanta et al. [14] examine five state-of-the-art saliency detectors from a frequency domain perspective as mentioned above. IT method produces very blurry version of the original image because most of the high frequencies are removed. This means the saliency map is un-correct enough for segmentation. The MZ method is a block average method. Even for a good choice of block size, MZ can not give a clear saliency map. GB has the same problem as IT that discards most of the high frequencies, and produces a blurry image. The SR method is better than IT and GB for retaining high frequencies, which can get a similar result as MZ methods. The AC method is quite different from the above four methods. It retains the entire range, and all the high frequencies are retained but low frequencies are sometimes discarded. So its saliency map is quite sharp. For the analysis above, Achanta et al.[14] fined some requirements for a saliency detector: Emphasize the largest salient objects. Uniformly highlight whole salient regions. Establish well-defined boundaries of salient objects. Disregard high frequencies arising from texture, noise and blocking artifacts. Efficiently output full resolution saliency maps. The FT method is base on a DoG filter with large standard deviation ratio of Gaussian kernels. Because the great standard deviation is selected to be infinity, the saliency map is formulated as: S( x, y) = I - I ( x, y) (1) m whc where I m is the mean image feature vector (corresponding to the great standard deviation), Iwhc ( xy, ) is the corresponding image pixel vector value in the Gaussian blurred version of the original image (corresponding to the small standard deviation), and is the L 2 norm. A basic principle in visual system is to suppress the frequently occurring features, while at the same time keeps sensitive to features that deviate from the norm [15]. From this point of view, we consider 362

3 I m as a model of frequently occurring features the background. But there are at least two shortcomings of this model. Firstly, it is not a real background. In fact, it s the mean of background and foreground with different weights which even does not appear in the image. Secondly, just one point in the feature space will be too simple to be a background. Consider a tiny image with black and white background, and a small gray square at the center of the image. Then the background color will be gray (Fig.1). (a) a tiny image example (b) background computed by FT (c) the salient ratio image (d) the real background 1 (e) the real background 2 (f) the real salient ratio image Figure 1. The situation when FT fails. The salient region detection is essentially an unbalance classification problem. When the background is an isotropic Gaussian distribution and occupies a large proportion, the above model will work well. But when the background is complex, such as multi-gaussian distribution, the above model fails. To solve these problems, we need to form a relatively more complex model that describes the real background features without adding foreground factors. 3. The proposed Robust FT method In order to raise FT method s performance on complex background, while keeping the advantages of its efficiency, full resolution maps and well-defined boundaries, we suppose to extend this method as follow: 363

4 S( x, y) = min I -I ( x, y), iî N + (2) i g where I i is the i th background feature vector, I g is the Gaussian blurred image to avoid noise. This formulation model the background with a feature point set and the saliency of each pixel is measured by its shortest distance to the set. How can we get a series of background representatives without using a time-consuming sophisticated classification or clustering method? When we check the assumptions of salient region, we can find that salient region take the character of relatively small size (scale property), spatial concentration (for single object), and usually near the center of the image when the object is huge. Then any part of the image which contains a great portion of background pixels could be a candidate background modeling region (BMR). By this way, the FT model exploits the whole image as the only BMR. But it is not imperative. For simplicity and efficiency, we take four regions as BMRs in this paper: 1/3 top image I top, 1/3 bottom image I bottom, 1/3 right image I right and 1/3 left image I left. These regions are usually unbalanced, contain more background pixels than foreground pixels following the characters of salient region, and they are abundant enough to model backgrounds in common scenes. Another shortcoming of FT model is that the background point I m is usually not a real pixel value in the image. It s a mixture of background and foreground, so it can t suppress background pixels well. For example, consider a bi-valued image which has n 1 pixels of value v 1 and n 2 pixels of value v 2. The saliency of v 1 and v 2 are n2 Sv = v 1 1-v2 (3) n + n 1 2 n 1 Sv = v 2 1-v2 n1+ n2 ( 4 ) and the saliency ratio is Sv / S 1 v = n 2 2 / n1. So a color salient region, without any change of color value, will become not such salient as it grows a little bigger. This is inconsistent with our usual sense. For the pursuit of robust as human vision system, we employ the median point instead of the FT s mean point of the BMR to model the background region. Then the formulation becomes S( x, y) = min{ I - I ( x, y) } (5) median where I = median( I ), Î { top, bottom, right, left}. median The proposed salient region detection is in fact a cluster algorithm in the color space. The image color is sampled by the region we defined as the top, bottom, right and left particle of the image. The background color becomes several center points of the color space. All other pixels are gathered by the shortest distance from these centers. The FT method has only one center, so it s very hard to descript the distribution of image color space. Our method with several centers does a litter better. There is no doubt that some more complex methods which can descript the color distribution well will get much more better results, but suffer from high computing complex, as we can see in Fig.2. g 364

5 (a) the FT s description (b) the proposed description Figure 2. Color space description by FT and the proposed method. (bg: background; fg: foreground) 4. Experiments and results In this section, we present comparisons with our method against the FT method. In order to obtain an objective comparison result, we use a ground truth image database which used by Achanta et al. [9]. This accurate object-contour based ground truth database, which is publicly available, contains masks of 1000 images selected from another publicly available database used by Liu et al. [16] (example in Fig.3). The MATLAB implementation of FT method can be downloaded from Radhakrishna s homepage. The Lab color space is used as a feature space in our experiments. Each pixel is a [,, ] T L ab vector. A few results of saliency map generated by the two methods are shown in Fig.4. The test images contain objects of different sizes, positions and colors, and the backgrounds are also of great variety. Our method clearly outperforms the FT method of suppressing the background. To quantify the different performance of the two methods, we employ Foreground Saliency Ratio (FSR) to measure the saliency map of each image. The FSR can be defined as the formula (6): mean( S( x, y) Î Foreground) FSR = mean( S( x, y) Î Foreground) + mean( S( x, y) ÎBackground) (6) This criterion states that an optimal saliency map should response high in foreground regions and low in background regions. Fig. 5 shows the mean intensity of foreground map and background map of 1000 images, that ordered by their foreground map value. For FT method the average foreground and background map values of the whole 1000 images are and , while for our method

6 and This means our method suppresses the background much better than FT method. Fig. 6 shows FSR s of the database. The images are order by their FSR s by FT method. There are 958 of 1000 images on which our method get high FSR s, and the average FSR of the whole database is for our method and for FT method. (a) original image from [16] (b) ground truth mask from [9] Figure 3. Ground truth examples 366

7 Figure 4. Some results of our method in comparison with FT method. In each group, we present 1) the input image, 2) saliency map generated by FT method, 3) saliency map generated by our method. (a) mean intensity of FT method 367

8 (b) mean intensity of our method Figure 5. Plot of the mean intensity of the saliency map of foreground and background for 1000 images. The images are ordered by their mean intensity of the foreground. 5. Summary Figure 6. Foreground Saliency Ratio of 1000 images In this paper, a novel algorithm for salient region detection is proposed. The proposed method, which is based on FT method, is called as Robust-FT method. It extends the original FT method by modeling the background with a feature set rather than a single feature point, and employing a robust median operator to computer this set from candidate background modeling regions. The new method not only inherits the advantages of the original method, such as well-defined boundaries, full resolution, and computational efficiency, it performances much better suppressing of background, especially when the background is complex. 368

9 Acknowledgement This study was supported in part by the National Natural Science Foundation of China (Grant No ), and by the Science Foundation of Jiangsu Province (Grant No.BK ), and by the Huaihai Institute of Technology Natural Science Foundation (Grant No. Z ), and by the Construction Found of JiangSu for Key Disciplines in Applications of Computing. The authors are grateful to Dr. Zhifang Min, a senior engineer in Department of system, Huazhong Institute of Optoelectronics, for providing theoretical supports on this research. References [1] Hao Jiang, Jie Xu, "Matching objects in multi-camera surveillance without geometric cnstraints", International Journal of Digital Content Technology and its Applications, vol.5, no.6, pp.79-86, [2] Qiaorong Zhang, yongqiang Zhang, "Salient region detection in video using spatiotemporal visual attention model", International Journal of Digital Content Technology and its Applications, Vol.6, no.11, [3] M. Jian, J. Dong and J. Ma, "Image retrieval using wavelet-based salient regions," Imaging Science Journal, The, vol. 59, pp , [4] Songhe Feng, De Xu, Xu Yang, "Attention-driven salient edge (s) and region (s) extraction with application to CBIR," Signal Processing, vol. 90, pp. 1-15, [5] Yang Liu, Xueqing Li, Lei Wang, Yuzhen Niu, Feng Liu, "Oscillation analysis for salient object detection," Multimedia Tools and Applications, pp. 1-21, [6] C. Koch and S. Ullman, "Shifts in selective visual attention: towards the underlying neural circuitry," Hum Neurobiol, vol. 4, no.1, pp , [7] Laurent Itti, Christof Koch and Ernst Niebur, "A model of saliency-based visual attention for rapid scene analysis," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 20, pp , [8] Dirk Bernhardt-Walther, Christof Koch, "Modeling attention to salient proto-objects," Neural Networks, vol. 19, pp , [9] Radhakrishna Achanta, Sheila Hemami, Francisco Estrada, Susstrunk, Frequency-tuned Salient Region Detection, IEEE Conference on Computer Vision and Pattern Recognition, [10] L. Itti, C. Koch, and E. Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vo.20, no.11, pp.: , [11] YuFei Ma, HongJiang Zhang, Contrast-based image attention analysis by using fuzzy growing, In ACM International Conference on Multimedia, [12] J. Harel, C. Koch, and P. Perona, Graph-base visual saliency, Advances in Neural Information Processing Systems, Vol.19,no.1, pp: , [13] Xiaodi Hou, Liqing Zhang, Saliency detection: A spectral residual approach, IEEE Conference on Computer Vision and Pattern Recognition, [14] Radhakrishna Achanta, Francisco Estrada, Patricia Wils, Sabine Susstrunk, Salient region detection and segmentation. International Conference on Computer Vision System, [15] C. Koch, and T. Poggio, Predicting the visual world: silence is golden. Nature Neuroscience, Vol.2, no.1, pp:9-10, [16] Tie Liu, Jian Sun, Nanning Zheng, Xiaoou Tang, Heung-Yeung Shum, Learning to detect a salient object. IEEE Conference on Computer Vision and Pattern Recognition,