A Survey on Detecting Image Visual Saliency

Transcription

1 1/29 A Survey on Detecting Image Visual Saliency Hsin-Ho Yeh Institute of Information Science, Acamedic Sinica, Taiwan {hhyeh}@iis.sinica.edu.tw 2010/12/09

2 2/29 Outline 1 Conclusions

3 3/29 What is visual saliency? Saliency: some regions in an image, and these regions motivate most of the visual attention when people see it at a first glance. (a) (b)

4 4/29 Outline 1 Conclusions

5 1 Depending on the salient region detector, saliency maps have ill-defined object boundary. For example, the range of spatial frequencies in the original image are reduced when it has downsized severely. On the other hand, some methods highlight the salient object boundaries, but they fail to map the entire salient region uniformly. These shortcomings result from the limited range of spatial frequencies retained from the original image in computing the final saliency map. 1 Radhakrishna Achanta et al.. In: CVPR /29

6 The author examines 5 state-of-the-art methods from a frequency domain perspective: IT 2, MZ 3, GB 4, SP 5, and AC 6. 2 Laurent Itti, Christof Koch, and Ernst Niebur. A Model of Saliency-Based Visual Attention for Rapid Scene Analysis. In: IEEE Trans. Pattern Anal. Mach. Intell (1998). 3 Yu-Fei Ma and Hong-Jiang Zhang. Contrast-based image attention analysis by using fuzzy growing. In: ACM Multimedia Jonathan Harel, Christof Koch, and Pietro Perona. Graph-based visual saliency. In: NIPS Xiaodi Hou and Liqing Zhang. Saliency detection: A spectral residual approach. In: CVPR Radhakrishna Achanta et al. Salient region detection and segmentation. In: Tsotsos (Eds.), Computer Vision Systems (2008). 6/29

7 Figure 3. Visual comparison of saliency maps. (a) original image, (b) saliency maps using the method presented by, Itti [16], (c) Ma and 7/29 Spatial Frequency Observation Method Frequency range Resolution Complexity IT [π/256, π/16] S/256 (k IT N) MZ [0, π/10] S/100 (k MZ N) GB [π/128, π/8] S/64 (k GB N 4 K ) SR [0, π/5] (k SR N) AC (0, π] S (k AC N) IG (0, π/2.75] S (k IG N) (a) Original (b) IT [16] (c) MZ [22] (d) GB [10] (e) SR [12] (f) AC [1] (g) IG

8 8/29 Requirement for A Saliency Map Emphasize the largest salient object. Uniformly highlight whole salient regions. Establish well-defined boundaries of salient objects. Disregard high frequency arising from texture, noise, and blocking artifacts. Efficiency output full resolution saliency map.

9 A Band-pass Filter DoG(x, y, σ 1, σ 2 ) = 1 2π [ 1 σ 2 1 exp x 2 +y 2 2σ σ2 2 = G(x, y, σ 1 ) G(x, y, σ 2 ), exp x 2 +y 2 2σ 2 2 ] where σ 1 and σ 2 are the standard deviation (σ 1 > σ 2 ). By combining several narrow band-pass DoG filters ( σ 1 = ρσ and σ 2 = σ). A summation over DoG results in: N 1 n=0 G(x, y, ρ n+1 σ) G(x, y, ρ n σ) = G(x, y, σρ N ) G(x, y, σ), where ρ = 1.6. ω lc = σ 1 = inf and ω hc = σ 2 = π/2.75 for removing high frequency noise and textures and retaining more than twice high-frequency content than SR. 9/29

10 10/29 A Band-pass Filter S(x, y) = I u (x, y) I whc (x, y) l2, where I u is the mean image, I whc is the corresponding image in the gaussian blurred version by 5 5 separable binomial kernel, that is [1, 4, 6, 4, 1]. 1 16

11 Experimental Results Data: Refine the MSRA image saliency dataset 7 into 1,000 images in a precise way. prec = ST GT, rec = ST ST GT, GT where ST and ST stand for the thresholded foreground and background regions of the detected saliency, GT and GT are the foreground and background saliency maps. regions in ground truth, and. indicates the number of pixels in the region. Figure 4. Ground truth examples. Left to Right, original image, ground truth rectangles from [28], and our ground truth, which is both more accurate and treats multiple objects separately. a fixed threshold to binarize the saliency maps. In the second experiment, we perform image-adaptive binarization of In order to obtain an objective comparison of segmentation results, we use a ground truth image database. We derived the database from the publicly available database used by Liu et al. [20]. This database provides bounding boxes drawn around salient regions by nine users. However, a bounding box-based ground truth is far from accurate, as also stated by Wang and Li [28]. Thus, we created an accurate object-contour based ground truth database 2 of 1000 images (examples in Fig. 4). Figure 5. Precision-recall curve for naïve thresholding of saliency maps. Our method IG is compared against the five methods of IT [16], MZ [22], GB [10], SR [12], and AC [1] on 1000 images. methods than simple thresholding. Saliency maps produced by Itti s approach have been used in unsupervised object segmentation. Han et al. [9] use a Markov random field to 7 T. Liu et al. Learning to Detect A Salient Object. In: TPAMI (2010). 11/29

12 12/29 Outline 1 Conclusions

13 8 W. Wang et al. Measuring visual saliency by Site Entropy Rate. In: CVPR /29 8 Intuition Beginning from the information maximization principle via running a random walks on the fully-connected graph to simulate the information transmission among the interconnected neurous.

14 14/29 Framework in Steps 1. Filter the input image with a number of sparse coding bases because sparse coding as an efficient coding strategy for optimal information transmission. 2. Adopt a fully-connected graph representation to capture the long range relation between two sites in an image. 3. Adopt random walker on each sub-band graph to model the information transmission among the neurons, and propose site-entropy rate (SER) which describes the accumulative effects of all the interactions between neurons. 4. Sum all the sub-band SER into the final saliency map. 5. (Option) In video, the novel signal and the change of the signal at a site make the site salient.(accounting the temporal change of neuron response.)

15 neuron connectivity we adopt a fully-connected graph representation for the feature maps. The full connectedness is able to capture the long range relation between two sites in an image. (3) A random walk is adopted on each sub-band feature responses from the corresponding sub-band feature map of the previous frames. Then we still run random walks on the fully-connected graphs of the updated feature maps to obtain the SER maps and finally the saliency map.15/29 Framework in Figure Figure 1. The proposed framework. An input image is filtered by sparse coding basis functions to obtain the corresponding sub-band feature maps. A fully-connected graph is constructed for each feature map, and a random walk is run on each graph to compute a SER map of each channel. Finally, the saliency map is generated by summing over all the SER maps.

16 where G k is the inverse/pseudoinverse of B k. In this paper, ICA 9 is adopted for learning bases. The filter response of G k form the k th sub-band feature map F k. 9 J. H. Van Hateren and A. Van Der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. In: R. Soc. Lond. B /29 The Sparse Coding Bases An image I is a linear superposition of a number of image bases B k, where k indexes for the location, orientation and scale. I = k a k B k, where p(a k ) e α a k is the high-order statistics prior to enforce the sparsity. The coefficient is computed by its corresponding filter function G k a k = x,y G k (x, y)i(x, y),

17 Sub-band Graph Representation A full-connected graph G k = {V k, E k } for feature map F k, where V k = {v k1,..., v kn }, v ki = (x i, y i, f k (x i, y i )) 10. E k = {e kij, i, j = 1,..., n}, e kij = (i, j, w kij ) is the set of weighted edges connecting every pair of nodes, where w kij = φ kij d ij. 11 φ kij = exp{ f k (x i, y i ) f k (x j, y j ) /M k } d ij = exp{ λ (x i, y i ) (x j, y j ) l2 }, D where M k is the largest feature difference, D = max(hgt, wdt), and λ = location and feature parts 11 φ kij denotes feature dissimilarity and d ij represents the spatial distance. 17/29

18 Site-Entropy Rate - Markov Assumption 12 The transition probability from site i to site j is defined as: P ij = w ij j w ij Stochastic assumption: πp = π, where π is the stationary distribution that π i = W i 2W W i = w ij, W = j i,j:j>i w ij 12 Random walk is a stochastic process of a sequence of RVs 18/29

19 19/29 Site-Entropy Rate The author assumes that the information transmission among each site is determined by : the transmission frequency and the amount of information at each transmission. That is site-entropy rate at site i is defined as: SER i = π i P ij log P ij, j where π i defines the frequency at which a random walker visit node i, and j P ij log P ij measures the uncertainty of node i to others at one step. Finally, the saliency value at pixel i is defined: S i = k SER ki, where k is sub-band index.

20 Experimental Results Dataset: Color image 13 : 20 subjects on the 120 color images Area Under the roc Curve (AUC) Itti et al Bruce et al Gao et al Hou et al The author proposed Neil Bruce and John Tsotsos. Saliency Based on Information Maximization. In: NIPS Itti, Koch, and Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis. 15 Bruce and Tsotsos, Saliency Based on Information Maximization. 16 Dashan Gao, Vijay Mahadevan, and Nuno Vasconcelos. The discriminant center-surround hypothesis for bottom-up saliency. In: NIPS Xiaodi Hou and Liqing Zhang. Dynamic visual attention: searching for coding length increments. In: NIPS /29

21 21/29 Outline 1 Conclusions

22 Segmenting Salient Objects from Images and Videos 18 In the previous works, band-pass filtering and sliding-window approach have resulted in the best performance; hence, the author proposes a sliding-window saliency detection method Besides, a segmentation method is proposed by incorporating the detected saliency and Conditional Random Field (CRF) framework. (Option) This method is directly applicable to both still image and videos. 18 E. Rahtu et al. Segmenting salient objects from images and Videos. In: ECCV /29

23 23/29 Intuitions in Figure Intuition: a pixel x is salient (close to 1) if the feature at x is similar to 36 the features at points of the inner window. W B K Fig. 2. Illustration of saliency map computation

24 Intuitions in Mathematics Consider a rectangle window W divided into two disjoint parts: K (kernel window, salient) and B(border window, background). H 0, H 1, and F(x) for the events Z K, Z B, and F(Z ) Q F (x), respectively 19. S 0 (x) = P(Z K F(Z ) Q F (x) ) [0, 1], x K P(F(x) H 0 )P(H 0 ) = P(F(x) H 0 )P(H 0 ) + P(F(x) H 1 )P(H 1 ) h K (x)p 0 = h K (x)p 0 + h B (x)(1 p 0 ), h K (x) = P(F (x) H 0 ) = 1 p(w)dw P(H 0 ) h B (x) = P(F(x) H 1 ), K F 1 (Q F(x) ) where Q F (x) denotes the bin which contains F(x) and 0 < p 0 < Z is a RV describing the distribution of pixels in W. 24/29

25 The saliency map is achieved by sliding the window W with different scales over the image, and the final saliency value is taken as the maximum over all windows containing a particular pixel x. In practice, the author uses a regular grid with step size equal to 1% max(hgt, wdt) and four scales of grid size {25%, 10%; 30%, 30%; 50%, 50%; 70%, 40%} max(hgt, wdt), respectively. 20 For frames in videos, the author combine CIELab and motion information into feature map. That is F (x) = (L(x), a(x), b(x), Y (x)). 21 h K (x) = h L K (L(x)) h a K (a(x)) h b K (b(x)) h Y K (Y (x)) for frames in video. 25/29 Intuitions in Mathematics The feature map 20 at a point x in CIELab color space F(x) = (L(x), a(x), b(x)) Based on the independent assumption, h k (x) 21 is defined as: h K (x) = h L K (L(x)) ha K (a(x)) hb K (b(x)).

26 26/29 Intuitions in Mathematics 375 Mean precision soft CRF trh CRF proposed measure Rahtu VS09 Achanta CVPR09 Guo CVPR Mean recall Precision Recall F beta Fig. 3. Left: Mean precision-recall curves using comparison methods and the proposed approach. Right: Mean precision, recall, and F-measure values for comparison method [26] (1), our method with thresholding (2), and our method with soft assignments (3). Notice that β =0.3 (used according to [26]) strongly emphasizes precision.

27 27/29 Outline Conclusions 1 Conclusions

28 28/29 Conclusions Conclusions They retain a reasonable range of spatial frequency by the proposed band-pass filter to highlight the whole salient object. However, they inaccurately analyze the object saliency since they cannot significantly highlight local contrasts. SER achieses a nice salient detection performance in images. However, The learnt bases cannot fit to all images because they are content-dependent. They observe that a distinctive-colour rectangle contains high contrast information; hence, the salient degree of a pixel in a rectangle is determined by the number of colour-similar pixels. However, without knowing the object size, the sliding-window approaches must vary the window size to locate the object; hence, the problem of object-size variation degrades their performance.

29 29/29 Conclusions Conclusions Thank you