VISUAL ATTENTION FOR CONTENT BASED IMAGE RETRIEVAL. Alex Papushoy and Adrian G. Bors

Transcription

1 VISUAL ATTENTION FOR CONTENT BASED IMAGE RETRIEVAL Alex Papushoy and Adrian G. Bors Department of Computer Science, University of York, York YO10 5GH, UK ABSTRACT A new image retrieval method, based on human visual attention models, called query by saliency content retrieval (QSCR) is presented in this paper. Each image is segmented and a set of characteristic features is evaluated for each region. The saliency for each image region, as it would be perceived by a human observer, is estimated for each region and then used for image retrieval. Images displaying similar features and characterized by similar saliency are then retrieved from the database. Both local and global saliency are considered in the retrieval process. The proposed method ranks the similarity between the query and the a set of given images using the Earth Mover Distance algorithm. Index Terms Content based image retrieval, human visual attention model, saliency, salient edges. 1. INTRODUCTION Content-based image retrieval (CBIR) considers a user-provided image as a query, whose visual information is processed and used in a content-based search [1, 2]. CBIR is based on the notion that visual similarity implies semantic similarity, which is not always the case, but is a valid assumption. The main challenge in CBIR systems is the ambiguity in the high-level (semantic) concepts extracted from the low-level (pixels) features of the image [3, 4, 5]. The second obstacle is the sensory gap which can be interpreted as the incompleteness of the object information captured by an imaging device. Generally, it is difficult for CBIR systems to search for broad semantic concepts because it is hard to limit the feature space without broadening the semantic gap. There are four categories of CBIR methods, [2]: bottom-up, top-down, relevance feedback and based on image classification. Those that rely purely on the information contained in the image are bottom-up approaches such as [6], while top-down approaches consider the prior knowledge. In image classification approaches, the system is presented with training data from which it learns a query [7]. Systems involving the user in the retrieval process via relevance feedback mechanisms are a mixture of bottom-up and top-down approaches [8]. Blobworld [5] is a region-based image retrieval system (RBIR) that segments the image into regions using the Expectation- Maximization (EM) algorithm and then prompts the user to specify the query regions of interest (ROI) and their importance. Certain top-down approaches employ machine learning techniques for the relevance feedback such as support vector machine (SVM) [9] or multiple instance learning (MIL) [10]. In our paper we propose a new query by similarity content retrieval (QSCR) method which is based on human visual attention models [11, 12]. During the first processing stage, the image is segmented into regions and feature vectors characterizing color, contrast, texture, region centroid, and the contrast with respect to the neighborhood are extracted for each region. Saliency is considered at too levels: localized, for defining the saliency of each segmented region, and global for the entire image, estimated based on the statistics of salient edges [13]. An optimization procedure called the Earth Mover s Distance (EMD) [14] is used together with the energy of salient edges for evaluating the similarity between the query image and a given retrieved image. The proposed QSCR methodology and the visual attention model is described in Section 2 while the similarity-based ranking procedure is detailed in Section 3. Experimental results are provided in Section 4 and the conclusions of this research study are outlined in Section QUERY BY SIMILARITY CONTENT RETRIEVAL BASED ON VISUAL ATTENTION MODELS Local saliency is represented as a weight for a distances in the feature space between salient regions, while globally is determined by the statistics of salient edges. The proposed methodology consists of three main processing stages as shown in the diagram from Fig. 1. Image segmentation is performed using the mean-shift algorithm [15]. For each segmented region we calculate a characteristic feature vector with entries corresponding to: color, contrast, texture information, the region neighbourhood information and region centroid. These features are then used into a inter-region distance measure between two images weighted according to their similarity. The human visual system is attracted to specific image regions coinciding with the fixation points chosen by saccades (random eye movements) at the pre-attentive stage for foveation (conscious acquisition of detail). Such image regions are characterized by local discontinuities and relevant features that make them stand out from the rest of the image. There are two ways to identify salient regions in images: bottom-up and top-down attention. Bottom-up attention is instinctive and involuntary corresponding to a human reflex. It is entirely driven by the image, usually by the colors, regions size, orientation, position, and motion. Top-down attention on the other hand, is driven by memory and prior experiences. The Itti-Koch saliency model [11, 16] is a well-known biologically plausible method designed for fast bottom-up scene analysis which employs the model of neurological processes occurring in primates brains. Graph-Based Visual Saliency (GBVS) [12] is the graph-based normalization of the Itti-Koch model. Other approaches, loosely based on the visual attention model of Itti-Koch, are the Saliency Using Natural statistics (SUN) [17] and the Frequency-Tuned Saliency (FTS) [18]. In [18] orientation features are obtained from filtering the image with a bank of Gabor filters at different orientations. SUN is a hybrid biologically-plausible and computational attention model based on the Bayesian framework using the a priori information from natural images collected off-line [17]. GBVS [12] was chosen in this study for calculating the saliency map in images due to its ability to clearly identify the most salient objects and image regions. Generally, saliency maps are created in three steps: feature vectors are extracted for every pixel to create feature maps, then activation maps are computed, and finally the activation maps are normalized and combined. GBVS employs dyadic /15/$ IEEE 971 ICIP 2015

2 Fig. 1: The system used for query by similarity content retrieval (QSCR). Gaussian pyramids for the image scales of 2:1, 3:1 and 4:1. These are applied for each channel of the physiologically derived D*K*L* color space. Orientation maps are created after applying Gabor filters at {0,45,90,135} degrees at every scale of each color channel. Unlike the Itti-Koch model that computes activation maps by centre-surround differences of image features [11], GBVS applies a graph-based approach [12]. The adjacency matrix is constructed by connecting each pixel of the map to all the other pixels, excluding itself, by using the following similarity function w(m x,m y) between feature vectors corresponding to the pixels located at x and y: [ ] Mx w 1(M x,m y) = log exp x y 2 (1) 2σ 2 M y where σ is the scale, in the range of [0.1,0,2] from the given map width. A Markov chain is defined over this adjacency matrix, where the weights of outbound edges are normalized to 1, by assuming that graph nodes are states, and edges are transition probabilities. By computing the equilibrium distribution yields an activation map A, where large values are concentrated in areas of high activation and thus are salient. The resulting activation map is smoothed and normalized. A new graph is constructed on the activation map, with each node connected to all the others including itself and which has the edge weights given by : ] w 2(M x,m y) = A(y)exp [ x y 2 (2) 2σ 2 where A(y) corresponds to the activation at location y. The normalization of the activation maps leads to emphasizing the areas of true dissimilarity, while suppressing non-salient regions. The resulting saliency map for the entire image is denoted as S are represents the sum of the normalized activation maps for each color and local orientation channel as provided by the Gabor filters. The salient regions are decided, following extensive experimentation, as those that have more than 20 % salient pixels in their segmented areas. For evaluating the global saliency we consider the edges extracted by the Canny edge detector [19]. Edge saliency is determined by the edge length and the saliency in the edge surrounding area as given by : S e = λ LL e +λ NN e (3) where S e represents the saliency of edge e, L e represents the length of the edge,n e is the saliency in the edge neighbourhood, and where λ L = 0.3 and λ N = 0.7 are weighting factors. The saliency in the neighbourhood of the edge is calculated by averaging over a small neighbourhood. Salient edges are those that have their saliency above as 25% from the maximum saliency, which was assessed following empirical evaluation. The statistics of the edges are grouped in edge histogram descriptors (EHD) as in the representation of the edge information by the MPEG-7 image compression standard [20]. Salient edges are counted for each orientation from the entire image, resulting in a vector EHD(θ,I), where θ = {0,π/2,π,3π/4,nondir}, where non-dir stands for non-directional. 3. SIMILARITY-BASED IMAGE RANKING The procedure of segmentation, feature extraction and saliency extraction described in the previous section is applied to all the retrieval candidate images as well as to the query image. Given a query image, we rank all the available candidate images according to their similarity with the query image. Let us consider that both the query imageqand a given imagei from the database are segmented into several regions Q i, i = 1,...,M and I j, j = 1,...,N, respectively. The similarity ranking becomes a many-to-many region matching problem which takes into account the salient edges as well. Examples of many-to-many region matching distances are the Integrated Region Matching (IRM) algorithm [4] and the Earth Mover s Distances (EMD) metric [14]. The similarity-based image ranking procedure proposed in this paper is outlined in the lower part of the diagram from Fig. 1. We propose a similarity measure between the query imageqand a given image I, represented as the weighted sum of the EMD matching cost function and the global image saliency measure as given by its salient edges : S(Q,I) = 0.7 EMD(Q,I) θ +0.3 EHD(θ,Q) EHD(θ,I) α EMD 5 α E (4) where EMD(Q,I) is the EMD metric between images Q and I, EHD(θ, Q) represents the average salient edge energy in the direction of θ = {0,π/2,π,3π/4,non-dir} for the image Q. α EMD and α E represent the robustness factors which are set as the 95th percentile of the cumulative distribution function of the EMD and the EHD measures, respectively, calculated using a statistically significant image sample set. The robustness factors are used for normalizing whilst eliminating outliers. EMD represents a measure of similarity in the local salient regions between the images Q and I, while EHD represents the similarity in the salient edge distributions and is extracted as described in the previous section. A inter-region distance measure D(Q i,i j) is calculated between each region i = 1,...,M from the query image Q and each region j = 1,...,N from the candidate retrieval image I as: D(Q i,i j) = ψ(s i,s j)(0.45d 2 cl +2.34d 2 te +0.45d 2 co +0.04d 2 nn +0.04d 2 cd) 1/2 (5) where ψ(s i,s j) denotes the joint saliency weight for Q i and I j, d cl, d te and d co are the Euclidean distances between the primary features of color, texture and contrast vectors, whiled nn andd cd are the Euclidean distances between the secondary features characterizing the colors of the nearest neighboring regions and the centroids of the regions Q i and I j. Each distance component is normalized to the interval[0,1] and is weighted according to its significance for 972

3 retrieval with an empirically estimated weight. The color distance d cl is calculated as the average distance of the normalized E2000 distance, which conforms to the minimal perceptual color difference according to CIE. This distance is calculated as the sum of the differences between the median L*a*b* color coordinates of the regions Q i and I j and the normalized distances between their corresponding standard deviations. The texture distance d te corresponds to the Euclidean distance between the average of the horizontal, vertical and oblique absolute value of the Discrete Wavelet Transform coefficients for the regions Q i and I j divided by their corresponding standard deviations calculated across the entire image database. The contrast difference d co is represented by the normalized Euclidean difference of the contrast features corresponding to the two regions. The neighbourhood characteristic differenced nn is calculated as the average of the resulting twelve color space distances to the four nearest neighboring regions from above, below, left and right. The centroid distance d cd is the Euclidean distance between the coordinates of the regions centers. These inter-region distances from (5) are weighted by the following weight corresponding to the joint saliency of the image regions Q i and I j : ψ(s i,s j) = max ( 1 Si +Sj,0.1 2 where S i is the saliency of the query image region Q i and S j is the saliency of a candidate image region I j, where the region saliency is given by the ratio of salient pixels, calculated as described in the previous section. It can be observed that the distance D(Q i,i j) is smaller when the two regions Q i and I j are both salient. The resulting inter-region matrix D acts as the ground matrix for the EMD algorithm. EMD algorithm [14] is an optimization algorithm with constraints, representing the normalized cost of transforming the query image signature into the signature of the candidate retrieval image. The corresponding weights only add up to unity when all image regions are used. We are filtering out non-salient regions and consequently the weights will add up to a value less than one. Such signatures enable partial matching which is essential in image retrieval where there is a high likelihood of occlusion in the salient regions. 4. EXPERIMENTAL RESULTS We have applied the proposed query by saliency content retrieval (QSCR) methodology to various databases such as Corel 1000, SIVAL and Flickr. Corel 1000 is well known for its medium-to-high image complexity. This database consists of 10 semantic categories of natural scenes, each containing 100 images showing distinct objects. Flickr database consists of 20 categories with 100 highly diverse images in each, while another 2000 images do not have any specific concept. Images are ranked based on their similarity to the query, thus producing an ordered set of results. The rank-weighted average precision (WPR) is given by, [4]: WPR = 1 N N k=1 ) (6) n k k, (7) where N is the total number of retrieved images and n k is the number of matches in the first k retrieved images. Receiver Operating Characteristic (ROC) plots the true positive rate versus false positive rate (false alarm). For assessing the retrieval performance we consider the area under the ROC curve (AUC), which corresponds to the Wilcoxon-Mann-Whitney statistics [6]. Quantitative tests are performed by evaluating the average performance of the QSCR system across the whole of each database considering 300 queries for COREL 1000, 600 queries for Flickr. Fig. 2: Comparison of saliency maps. Original images are in the first row, Itti-Koch saliency maps are in the second row, GBVS maps are in the third row, SUN maps are in the fourth row, and FTS maps are in the fifth row. Saliency maps, pseudo-colored from red to blue depending on the visual attention intensity, are overlaid. In Fig. 2 we provide a comparison of saliency maps produced by four saliency algorithms: Itti-Koch (IK) [11], graph-based visual saliency (GBVS) [12], saliency using natural statistics (SUN) [17], and frequency-tuned saliency (FTS), [18]. From these results it can be seen that IK produces small highly focused peaks in saliency that tend to concentrate on small areas of the object which are insufficient for representing the semantic concept of the image. The large amount of false positives, bias, and lack of precision makes SUN an unsuitable choice for retrieval in the broad image domain, but perhaps it could prove itself useful in specialized applications. FTS algorithm works well when there is a salient color object in the image. GBVS produces the most useful maps for the purposes of image retrieval as it can be observed from Fig. 2. Fig. 3: Retrieving a translucent bowl image from SIVAL database. The query image is the first right-top image on each row followed by the retrieved images without and with modelling the visual attention. In Fig. 3 we present an example of retrieving the translucent bowl (TB) image in SIVAL database without visual attention models and with visual attention models. As it can be observed in this figure when using visual attention models, all first six retrieved images and eight out of the total of nine correspond to the TB category, while when not using the visual attention models only the seventh image is from the correct category but none of the other eight images. Salient edges are extracted and used in the final image ranking evaluation measure from (4). Unlike in [21], the method proposed in this paper decouples the edge histogram from its spatial domain 973

4 Fig. 4: Retrieval by salient edges on COREL 1000 database subset using AUC measure. µ indicates the average area under ROC curve. by considering the edge energy, corresponding to a specific image feature orientation, calculated from the entire image. In Fig. 4 we compare the proposed salient edge retrieval approach when using only the global image saliency as provided by edges and not the local saliency, with SEHD image retrieval method [21]. The mean AUC value for SEHD is while for the proposed method, the mean is Thus, performing a two-tailed Students t-test at the highly significant 1% level with 598 degrees of freedom yields ap-value of which shows that the difference is statistically significant. In Fig.?? we provide the results for segmentation, saliency map extraction, salient edges and the salient region selection for three images from three distinct image categories of COREL 1000 database. Fig. 6: Retrieval of 30 images from the architecture category from COREL database. The first line shows the original image, its saliency, selected salient regions and salient edge images while the subsequent lines displays the ordered retrieved images. The semantic gap is most evident in Flickr database, because its images and ground truths were obtained by performing keyword search. However, retrieval system features, such as edges, that perform well on the Corel database, have a reduced performance in the Flickr database. Fig. 5: Extracting query information from images. The image columns indicate from left to right: original image, segmented regions, the GBVS saliency map, salient edges and the salient regions. Images are ranked according to the similarity measure S(Q, I) from (4), between the query image Q and a candidate retrieval image I. In Fig. 6 we provide the first 30 images retrieved for a query image from the Architecture (ARC) category of Corel The selected region is a false positive in this case because the true object occupies most of the image and the most dissimilar area is a patch of sky in the middle. Nevertheless, the retrieval succeeds because the salient region selection method considers some of the surrounding areas as well. In Fig. 7a we provide the results for the average rank the proposed QSCR algorithm when compared with with SIMPLIcity from [4] on the entire COREL 1000 database, using the average rank or retrieval. QSCR unlike SIMPLIcity provides rather uniform results for all image categories. In Fig. 7b we compare the results of QSCR, with the two retrieval methods proposed in [6], on Flickr database when using AUC. The results of QSCR and ACCIO are broadly similar while outperforming the ACCIO segmentation with neighbors method from [6]. Moreover, ACCIO involves feedback from human intervention, while the proposed approach is completely automatic. The salient edge information provides better performance when the category has limited variation among its salient objects. (a) Fig. 7: Comparison results with: (a) SIMPLIcity on COREL database; (b) ACCIO on Flickr database. 5. CONCLUSION In this paper we present a novel image query by saliency content retrieval using visual attention models. The proposed method is bottom-up and unsupervised and we consider the saliency both locally and global for the entire image. After segmenting each image, we calculate the feature vectors for each region characterizing color, contrast, texture, the region centroid, and the contrast with respect to the neighbourhood. Those regions from the image which attract the human attention are defined as salient by using the graph-based visual saliency algorithm. The global image saliency is assessed using the statistics of salient edges. The final ranking is performed by means of the Earth Mover s Distance (EMD), between image regions, used together with a distance measure in the global saliency. Image query by saliency content retrieval has several applications including image database search, image classification based on content and image characterization. (b) 974

5 6. REFERENCES [1] A. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain, Content-based image retrieval at the end of the early years, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp , [2] R. Datta, D. Joshi, J. Li, and J. Z. Wan, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys, vol. 40, no. 2, pp. 5:1 5:60, [3] J. Z. Wang, G. Wiederhold, O. Firschein, and X. W. Sha, content-based image indexing and searching using daubechies wavelets, International Journal on Digital Libraries, vol. 1, no. 4, pp , [4] J. Wang, J. Li, and G. Wiederhold, Simplicity: Semanticssensitive integrated matching for picture libraries, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 23, no. 9, pp , [5] C. Carson, S. Belongie, H. Greenspan, and J. Malik, Blobworld: Image segmentation using expectation-maximization and its application to image querying, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 24, no. 8, pp , [6] R. Rahmani, S. A. Goldman, H. Zhang, S. R. Cholleti, and J. E. Fritts, Localized content-based image retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 11, pp , [7] Y. Chen and Wang, Image categorization by learning and reasoning with images, J. of Machine Learning Research, pp , [8] Y. Rui, T. Huang, M. Ortega, and S. Mehrotra, Relevance feedback: A power tool in interactive content-based image retrieval, IEEE Trans. on Circuits and Systems for Video Technology, vol. 8, no. 5, pp , [9] J. Bi, Y. Chen, and J. Wang, A sparse support vector machine approach to region-based image categorization, in Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition, vol. 19, 2005, pp [10] R. Rahmani, S. A. Goldman, H. Zhang, S. R. Cholleti, and J. E. Fritts, Localized content-based image retrieval, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 30, no. 11, pp , [11] L. Itti, C. Koch, and E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp , [12] J. Harel, C. Koch, and P. Perona, Graph-based visual saliency, in Proc. Advances in Neural Information Processing Systems (NIPS), vol. 19, 2007, pp [13] A. Papushoy and A. G. Bors, Image retrieval based on query by saliency content, Digital Signal Processing, vol. 36, pp , [14] Y. Rubner, C. Tomasi, and L. J. Guibas, The earth mover distance as a metric for image retrieval, Int. Journal of Computer Vision, vol. 40, no. 2, pp , [15] D. Comaniciu and P. Meer, Mean shift: a robust approach toward feature space analysis, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp , [16] L. Itti and S. Ullman, Shifts in selective visual attention: Towards the underlying neural circuitry, Human Neurobiology, vol. 4, no. 4, pp , [17] L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W. Cottrell, SUN: A Bayesian framework for saliency using natural statistics, Jour. of Vision, vol. 8, no. 7, pp , [18] R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, Frequency-tuned salient region detection, in Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition, 2009, pp [19] J. Canny, A computational approach to edge detection, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp , [20] G. Stamou and S. Kollias, Multimedia Content and the Semantic Web: Standards, Methods and Tools. Wiley & Sons, [21] S. Feng, D. Xu, and X. Yang, Attention-driven salient edge(s) and region(s) extraction with application to CBIR, Signal Processing, vol. 90, no. 1, pp. 1 15,