Semantics-based Image Retrieval by Region Saliency

Transcription

1 Semantics-based Image Retrieval by Region Saliency Wei Wang, Yuqing Song and Aidong Zhang Department of Computer Science and Engineering, State University of New York at Buffalo, Buffalo, NY 14260, USA wwang3, ys2, Abstract. We propose a new approach for semantics-based image retrieval. We use color-texture classification to generate the codebook which is used to segment images into regions. The content of a region is characterized by its self-saliency and the lower-level features of the region, including color and texture. The context of regions in an image describes their relationships, which are related to their relative-saliencies. High-level (semantics-based) querying and query-by-example are supported on the basis of the content and context of image regions. The experimental results demonstrate the effectiveness of our approach. 1 Introduction Although the content-based image retrieval (CBIR) techniques based on low-level features such as color, texture, and shape have been extensively explored, their effectiveness and efficiency are not satisfactory. The ultimate goal of image retrieval is to provide the users with the facility to manage large image databases in an automatic, flexible and efficient way. Therefore, image retrieval systems should be armed to support high-level (semantics-based) querying and browsing of images. The basic elements to carry semantic information are the image regions which correspond to semantic objects, if the image segmentation is effective. Most approaches proposed for region-based image retrieval, although successful in some aspects, could not integrate the semantic descriptions into the regions, therefore cannot support the high-level querying of images. After the regions are obtained, proper representation of the content and context remains a challenge. In our previous work [7], we used color-texture classification to generate the semantic codebook which is used to segment images into regions. The content and context of regions were then extracted in a probabilistic manner and used to perform the retrieval. The computation of the content and context needed some heuristic weight functions, which can be improved to combine visual features. In addition, when retrieving images, users may be interested in both semantics and some specific visual features of images. Thus the content and context of image regions should be refined to incorporate different visual features. The saliencies of the image regions [2], [3], [6] represent the important visual cues of regions, therefore can be used to improve our previous method. In this paper, we will introduce an improved image-retrieval method, which incorporates region saliency to the definition of content and context of image regions.

2 The saliencies of regions have been used to detect the region of interest (ROI). In [6], saliency was further categorized as self-saliency and relative-saliency. Selfsaliency was defined as what determines how conspicuous a region is on its own, while relative-saliency was used to measure how distinctive the region appears among other regions. Apparently, saliencies of regions represent the intrinsic properties and relationships for image regions. Our approach consists of three levels. At the pixel level, color-texture classification is used to form the semantic codebook. At the region level, the semantic codebook is used to segment the images into regions. At the image level, content and context of image regions are defined and represented on the basis of their saliencies to support the semantics retrieval from images. The three levels are illustrated in Figure 1. Image Pixels Semantic Codebook Regions Content & Context Semantic Retrieval Pixel Level Region Level Image Level Fig. 1. System design of our approach. The remainder of the paper is organized as follows. From Section 2 to Section 5, each step of our approach is elaborated. In Section 6 experimental results will be presented, and we will conclude in Section 7. 2 Image segmentation 2.1 Generation of the semantic codebook We generate the semantic codebook by using color-texture classification to classify the color-texture feature vectors for pixels in the images into cells in the color-texture space. About the color-texture feature vectors for each pixel, the three color features we choose are the averaged RGB values in a neighborhood and the three texture features are anisotropy, contrast and floworedge. Anisotropy measures the energy comparison between the dominant orientation and its orthogonal direction. Contrast reflects the contrast, or harshness of the neighborhood. FlowOrEdge can distinguish whether an 1D texture is a flow or an edge. (See [7] for detailed description of these values). The color and texture feature vectors are denoted as Color FV and Texture FV, respectively. Therefore for each pixel, we have a six-dimensional feature vector where three dimensions are for color, and three for texture. The color-texture classification is performed in the following way: by Color FV, the color space is uniformly quantized into classes; by Texture FV, texture space is classified into 7 classes: one class for low contrast and edge, respectively; two classes for flow and three for 2D texture. Formal definition of these classes can be found in [7]. Because edges cannot reflect the color-texture characteristics of image regions semantically, we classify all the pixels, except those pixels corresponding to edges, to 6 texture classes

3 LowContrast 2D 0 2D 1 2D 2 Flow 0 Flow 1. Corresponding to T T 6 texture classes and C C 64 color classes, we now split the color-texture space (denoted as CT S, excluding the pixels on the edges) to C T cells. We then define the major semantic categories based on the semantic constraints of the images in the database: SC sc 1 sc M For each semantic category SC i, certain number of images are chosen to be the training images such that the semantic codebook generated from them can represent as much as possible the color-texture characteristics of all images in the database belonging to the SC. The set of all pixels in the training images, except those in class Edge, is denoted as TrainingPixels. For each pixel in TrainingPixels, its feature vector will fall into one cell of CTS. After all the pixels in TrainingPixels are classified, for each cell in CT S we count the number of pixels in it. Only those cells whose number of pixels exceeds a threshold will be one entry of the semantic codebook of the that database. Therefore the size of semantic codebook will be less or equal to C T. In the following discussion, we use SCDB to denote the semantic codebook for color-texture space, and suppose its size is N. 2.2 Image segmentation based on semantic codebook We segment images by SCDB in this way: for each image I, we extract the color-texture feature vector for each pixel q of I. For each feature vector, we find the cell in CT S where the pixel belongs to. If the cell is one of SCDB s entries, q is replaced with the index of that entry; otherwise its value is set to C T 1 to distinguish it from any valid entry of the semantic codebook. After the process, I becomes a matrix of indices corresponding to entries in SCDB. Because the number of valuable regions in an image is usually less than 5 in our experiments, we only choose 5 most dominant indices (referred as DOMINANT ) and use DOMINANT to re-index the pixels with indices not present in DOMINANT 1. Finally we run the encoded images through a connectedcomponents algorithm and remove the small regions (with area less than 200 pixels). 3 Representation of saliency for regions and semantic categories As stated before, saliency features of the regions can be either self-saliency or relativesaliency. Self-saliency features are computed by the visual features of the regions. Relative-saliency features are computed by taking the value of a feature for the current region and comparing it with the average value of that feature in the neighboring regions. Gaussian and Sigmoid functions are used to map the feature values to the interval [0, 1] for the purpose of normalization and integration of different saliency features. Self and relative-saliency features we are using include: (1) Self-saliency features of image regions: Size (S size ): S size R i max A R i A max 1 0. Here A R i is the area of region R i and A max is a constant used prevent excessive saliency being given to very large regions. It 1 re-index means to find the closest codebook entry in DOMINANT, not in the whole SCDB.

4 was shown in [3] that larger regions are more likely to have larger saliency. However a saturation points exists, after which the size saliency levels off. A max is set to the 5% of the total image area. The value is in the interval [0, 1]. ma jor Eccentricity (S ecce, for shape): S ecce R i axis R i minor axis R i. A Gaussian function maps the value to the interval [0, 1]. Circularity (S circ, for shape): S circ R i Peri R i 2 A R i. Here Peri R i is the perimeter of the region R i and a Sigmoid function maps the value to the interval [0, 1]. Perpendicularity (S perp, for shape): S perp R i angle ma jor axis R i π 2. A Sigmoid function maps the value to the interval [0, 1]. Location (S loca ): S loca R i center R i A R i. Here center R i is the number of pixels in region R i which are also in the center 25% of the image. (2) Relative-saliency features of image regions: similar to [2], relative-saliency features are computed as: f eature f eature R0 f eature Ri Relative saliency R i R i NR i 0 f eature Ri Here NR refers to the neighboring regions. We use Brightness and Location as the features and compute Relative brightness and relative location as the relative-saliency features in the system. The values are mapped to interval [0, 1] using Sigmoid functions. (3) Combining self and relative-saliency to generate the saliency of the regions and semantic categories: It is not necessary that all the saliency features will be used for each semantic categories. We choose particular saliency feature(s) to represent the most dominant visual features of each semantic categories, if possible. For instance, assume we have a category water falls. We will use Eccentricity and Perpendicularity as well as Relative brightness for the saliency features because usually water falls will be long and narrow, and can be approximately described as perpendicular to the ground. Another example is the category flower, the Circularity and Location as well as Relative brightness are used since usually flower will be round and in the middle of the images and brighter compared with surrounding scenes. A table in Section 6 lists the selection of the saliency features for all the semantic categories we have in the system. Because all the self and relative-saliency values are normalized to the interval [0, 1], we can simply add them together as the Saliency of the region with regard to a certain semantic category, denoted as Sal R i SC j, for the region R i and the semantic category SC j. For all the regions in training images that represent the semantic category SC j, we calculate the mean and variance of their saliency and store them as the Saliency of the semantic category, denoted as µ j and σ j for the SC j. 4 Representation of content and context of regions By collecting the statistical data from the training images, we derive the logic to represent the content and context for all the images in the database for the semantics retrieval. We first generate the statistical data from the training images. For each entry e i in the

5 semantic codebook SCDB, and each semantic category SC j, we count the number of regions R in the training images such that (i) the pixels in R belong to e i (e i is a cell in the CT S); and (ii) the region R represents an object belong to the category SC j. The result is stored in a table called cell-category statistics table. In addition, we count the times that two or three different categories present in the same training images. The result is stored in a table, called category-category statistics table. Based on the cell-category statistics table, for each codebook entry of SCDB, we can calculate its probability of representing a certain semantic category. Let N be the size of SCDB, M be the number of SC, i SC j i! 0 N 1"# j! 0 M 1" denote the event that index of SCDB i represents semantics described in SC j. The probability of T i% SC the event i SC j is P i SC j $ j T i where T e represents event e s presence times in the training images. Based on the category-category statistics table, we define the Bi-correlation factor and Tri-correlation factor, for representing the context of regions. Definition 1 For any two different semantic categories SC i and SC j, we define the Bicorrelation factor B i j as: B i j i& j n n i' n j( n i& j Definition 2 For any three different semantic categories SC i, SC j and SC k, we define n the Tri-correlation factor T i j k as: T i j k i& j& k n i' n j' n k( n i& j( n i& k( n j& k' n i& j& Here n i, n k i j and n i j k are entries in the category-category statistics table. Bi and Tri-correlation factors reflect the correlation between two or three different categories within the scope of training images. If the training images are selected suitably, they reflect the relationship of pre-defined semantic categories we are interested in. Armed with the statistical data we have generated from training images, we can define and extract content and context of regions for all the images in the database. Assume we have an image I with number of Q regions. The SCDB s codebook indices of regions are C i i ) 0 Q 1", and regions are represented by R i i! 0 Q 1", and each C i is associated with i N possible semantic categories SC i j* i+ i j i, 0 N 1"# j i 0 i N 1" 2. For I, let P all be the set of all possible combinations of indices of semantic categories represented by regions R i i - 0 Q 1". We have P all. / 0 j 0 0 i j i 0 Q 1 j Q( 1 21 i P C i SC i j* i+ 3 0 i - 0 Q 1"4 i j i 5 0 N 1"# j i 6-0 i N 1" Note there are totally Q ( 1 i 0 i N possible combinations for P all, therefore P all has Q ( 1 i 0 i N tuples of semantic categories indices, with each tuple having Q fields. Corresponding to each tuple κ P all, for Q 37 2 we have B κ 8 p q κ p9 q B p q κ P all and for Q 37 3 we have T κ : p q r κ p9 q9 r T p q r κ P all Here p q r are the indices belonging to tuple κ, B κ represents the sum of Bi-correlation factors of different semantic categories with regard to tuple κ, and T κ represents the sum of Tri-correlation factors of different semantic categories with regard to tuple κ. 2 N is the size of SCDB

6 C 1 Definition 3 We define Context C Score κ of I as: C Score κ 8 Norm κ B κ ; βt κ 0 < κ P all here Norm κ is the normalization function with tuple κ, β is the weight for T κ, since T κ will be more effective in distinguishing contexts of images than B κ 3. We normalize the C Score because several region indices may point to the same semantic category, we need to guarantee that removal the redundant semantic category will not influence the effectiveness of C Score. Definition 4 We define ProbScore P Score κ of I as: P Score κ = Q ( 1 i 0 w R i SC i j P C i SC i j < i j κ κ P all P Score κ represents the probability score corresponding to tuple κ, here w R i SC i j is the weight function with regard to region R i and semantic category SC i j, it can be determined by using the saliency of the region and the semantic category: w R i SC i j $ σ i j > 1 2π e( Sal R i SC i j?( µ i j 2 2σ2 i j where Sal R i SC i j is the saliency of region R i with regard to the semantic category SC i j, µ i j and σ i j are the saliency of the semantic category SC i j. Definition 5 We define the TotalScore T Score of image I as T Score Max P Score γc Score κ κ P all where Max t represents the maximum value of value t. Definition 6 We define the Content of image I as the semantic categories corresponding to T Score. By computing the maximum value of P Score κ A γc Score κ over all tuples in P all, we find the semantic categories that best interpret the semantics of the regions in image I as the Content of I. We store the Content, T Score and the each region s SCDB codebook index and saliency as the final features for I. Note that for those regions whose codebook indices are invalid corresponding to semantic codebook, we will mark its semantic category as UNKNOWN. 5 Semantics retrieval Both semantic keyword query and query-by-example are supported in our approach. According to the submitted semantic keywords (corresponding to semantic categories), the system will first find out those images that contain all of the categories (denoted as set RI), then rank the documents by sorting the T Score stored for each image in the database. For those images belonging to RI that has UNKNOWN category, its T Score will be multiplied by a diminishing factor. When user submits the query-by-example, if the query image is in the database, its Content will be used as query keywords to perform the retrieval, otherwise it will first go through the above steps to obtain the Content and T Score. When users are interested in retrieving images with not only same semantics, but also the similar visual features as the query, Euclidean distance of saliency value will be calculated between the query image Q and image I in RI: d I Q B W ( 1 i 0 3 In our experiments, we select β D 10. Sal I Ri SC j 8 Sal Q Ri SC j 2

7 Here assume Q has W regions and I s region I Ri has same semantic category SC j with Q s region Q Ri. d I Q indicates the distance of saliency between I and Q and therefore can be used to fine tune T Score. 6 Experiments To test the effectiveness of our approach in retrieving the images, we conduct experiments to compare the performance between our approach (with and without using the saliency features) and the traditional CBIR techniques including Keyblock [1], color histogram [5], color coherent vector [4]. The comparison is made by the precisions and recalls of each method on all the semantic categories. In the following table, we define 10 semantic categories and list the saliency features adopted for each semantic category as Y, otherwise as N. Abbreviations represent the saliency features: S Size, E Eccentricity, C Circularity, P Perpendicularity, L-Location, RB Relative brightness, RL Relative location. Category Explanation No. in db S E C P L RB RL SKY Sky (no sunset) 1323 N N N N Y N Y WATER Water 474 N N N N Y N Y TREE GRASS Tree or Grass 778 Y N N N N Y N FALLS RIVER Falls or Rivers 144 N Y N Y N Y N FLOWER Flower 107 N N Y N Y Y N EARTH ROCK Earth or Rocks or MOUNTAIN Mountain composed of 998 Y N N N N N Y ICE SNOW Ice or Snow or MOUNTAIN Mountain composed of 204 Y N N N N N Y SUNSET Sunset scene 619 N N N N N Y N NIGHT Night scene 171 Y N N N Y N N SHADING Shading 1709 N N N N N N N We use an image database with name COREL and size The COREL images can be considered as scenery and non-scenery. Scenery part has 2639 images consisting images containing the defined semantic categories, while non-scenery part has 1226 images including different kinds of textures, indoor, animals, molecules, etc. We choose 251 training images from scenery images as training images and form the semantic codebook with size 149. For each semantic category SC i, we calculate and plot the precision-recall of our approach in the following way. Let RET RIEV ELIST denote the images retrieved with SC i. Suppose RETRIEVELIST has n images. We calculate the n precisions and recalls of first,..., and n images in RETRIEVELIST, respectively. Since traditional CBIR approaches accepts only query-by-example, we have to solve the problem of comparing the approach of query-by-semantics with query-by-example. Let us take Keyblock as the example of traditional CBIR to show how we choose query sets and calculate the precision-recall for these methods. Suppose user submits a semantic keyword query of semantic category of Sky. There are total of 1323 images in 30, 2n 30

8 COREL containing Sky. For each image containing Sky, we use it as a query on COREL to select top 100 images by Keyblock, and count the number of images containing Sky in the retrieved set. Then we sort the 1323 images descendingly by the numbers of Sky images in their corresponding retrieved sets. Let the sorted list be SKY LIST. Then we select the first 5% of SKYLIST as query set, denoted as QUERYSET. Then for each COREL image I, we calculate shortest distance to QUERYSET! I by Keyblock 4. The COREL images are sorted ascendantly by this distance. Top 1323 COREL images are retrieved and we calculate and plot the precision-recall of Keyblock on Sky, as we did for our approach. The average precision-recall on all semantic categories is shown in Figure 2. We can see our method outperforms traditional approaches and retrieval performance improves when saliency features are used Average Precision Recall on COREL Our Approach (New result) Our Approach (Old result) Keyblock Color Coherent Vector Color Histogram 0.8 PRECISION RECALL Fig. 2. Average Precision-recall on all the SCs 7 Conclusion The saliency of image regions, which describes the perceptual importance of the regions, is used for the semantics-based image retrieval. It helps refine the content and context of regions to represent the semantics of regions more precisely. The experimental results show that our approach outperforms the traditional CBIR approaches we compared with. 4 images in the QUERY SET will have distance zero to QUERY SET. Thus the query images will be automatically be retrieved as the top 5% images, which is unfair when making comparison.

9 8 Acknowledgment This research is supported by the NSF Digital Government Grant EIA The authors thank the anonymous reviewers for their valuable comments to the paper. References 1. L. Zhu, A. Zhang, A. Rao and R. Srihari. Keyblock: An approach for content-based image retrieval. In Proceedings of ACM Multimedia 2000, pages , Los Angeles, California, USA, Oct 30 - Nov J. Luo and A. Singhal. On measuring low-level saliency in photographic images. In Proc. IEEE Comp. Vision and Pattern Recognition, pages Vol. 1 pp 84 89, W. Osberger and A. J. Maeder. Automatic identification of perceptually important regions in an image. In Proc. IEEE Int. Conf. Pattern Recognition, Greg Pass, Ramin Zabih, and Justin Miller. Comparing images using color coherence vectors. In Proceedings of ACM Multimedia 96, pages 65 73, Boston MA USA, M.J. Swain and D. Ballard. Color Indexing. Int Journal of Computer Vision, 7(1):11 32, T. F. Syeda-Mahmood. Data and model-driven selection using color regions. In Int. J. Comp. Vision, pages Vol. 21 No. 1. pp 9 36, W Wang, Y Song, and A Zhang. Semantics retrieval by content and context of image regions. In Proc. of the 15th International Conference on Vision Interface (VI 2002), Calgary, Canada, May 27-29, 2002.