Semantic Approach to Image Database Classification and Retrieval

Transcription

1 NII Journal No. 7 (2003.9) Research Paper Semantic Approach to Image Database Classification and Retrieval Zaher AGHBARI Graduate School of Information Science and E.E., Kyushu University Akifumi MAKINOUCHI Graduate School of Information Science and E.E., Kyushu University ABSTRACT This paper demonstrates an approach to image retrieval founded on classifying image regions hierarchically based on their semantics (e.g., sky, snow, rocks, etc.) that resemble peoples perception rather than on low-level features (e.g., color, texture, shape, etc.). Particularly, we consider to automatically classify regions of outdoor images based on their semantics using the support vector machines (SVMs) tool. First, image regions are segmented using the hill-climbing method. Then, those regions are classified by the SVMs. The SVMs learn the semantics of specified classes from a test database of image regions. Such semantic classification allows the implementation of intuitive query interface. As we show in our experiments, the high precision of semantic classification justifies the feasibility of our approach. [Keywords] semantic classification, image region classification, image segmentation, support vector machines, keyword-based indexing. 1 Introduction Advances in technologies such as image digitization, storage, and transmission along with the growth of the World Wide Web have led to the proliferation of online digital images. Therefore, content-based image classification and retrieval systems have been the subject of many research works in recent years. These systems, such as QBIC [1] and VisualSEEK [2], facilitate classification, indexing, and retrieval of images, mainly, based on low-level features of images, such as color, texture and shape. Although such systems enjoy the advantage of being fully automatic, they are hard to use by non-expert users due to the semantic gap that exists between user needs and system requirements. Naturally, non-expert users prefer to formulate their queries using image semantic elements, such as sky, snow and rock, which are closer to their conceptions than lowlevel features. This gap between the low-level descriptions of image content and the semantic needs of users can be reduced if image search systems adopt a semantic based approach to organize and index the image content. However, the automatic extraction of semantic elements of images is not reliable as apposed to the lowlevel features. For example, the color distribution in terms of color histogram can be simply extracted for any image, but the automatic identification of, say, trees or sky in the image may be uncertain. For reliable identification of image elements, the work in [3] manually identifies (labels) image elements (called visual keyword) of sample images. Then, every image in the database is compared against those visual keywords to detect the specific visual keywords of the image. Later the visual keywords approach was integrated with the conceptual graph based representation in [4]; as a result, according to the paper, retrieval precision of family photographs is improved as compared to either approach alone. On the other hand, automatic labeling of images, or image regions, may result in some uncertainty, leading to multiple potential labels for image regions. The approach in [5] proposes a probabilistic approach to label small areas of one set of images as either man-made images or natural, and the local areas of another set were labeled as either 1

2 Semantic Approach to Image Database Classification and Retrieval inside or outside. Such semantic labeling is based on estimating the class likelihoods from color-texture features of the local areas. Another approach [6] proposes a Bayesian framework for classifying vacation images. Unlike the approach of [5], which labels local areas of an image, in [6], the labeling is based on the global feature vector of the image. It classifies images into city versus landscape and then further classifies the landscape images into sunset, forest, and mountain classes. A different approach to organize the images into semantic categories is presented in [7], in which training sample regions of images are flat-clustered, rather than hierarchally classified, by a neural network into 11 clusters (e.g. Brick, Cloud, Grass, etc.). A different classifier was created for every cluster. Similarly, the approach in [8] clusters the image regions into 10 flat clusters (e.g. sky, water, night, etc.). However, the approach in [8] uses a probabilistic approach to define a semantic codebook of every cluster. In this paper, we propose to extend the approaches of the automatic hierarchal classification of image regions into a more detailed classification hierarchy based on the semantics of the regions contents. This paper demonstrates that we can reliably classify regions of outdoor images into semantic classes. We use the hillclimbing algorithm to segment images. Learning the semantics of a predefined hierarchy of classes is performed using support vector machines (SVMs) based on heterogeneous features of the training sample regions of each class. Then, the created conservative classifiers P (classified notmemberofclass) achieves good class separation under the constraint that each region belongs to only one, or none, of the classes. We also represent an image by the classes, in which the image regions are classified. Consequently, an intuitive query interface that utilizes the represented classes in the image is also presented. The rest of this paper is organized as follows: Section 2 presents a simple and fast nonparametric segmentation method based on a hill-climbing algorithm to determine the clusters of an image. In Section 3, we introduce the lowlevel features extracted from the image regions. Then, the support vector machines tool is briefly explained in Section 4. In Section 5, we discuss how the image regions are semantically classified. The keyword- based indexing is discussed in Section 6. Then, we present our experiments in Section 7. Finally, we conclude the paper in Section 8. 2 Image Segmentation In this paper, we use a simple and fast nonparametric segmentation method based on a hill-climbing algorithm [9] to determine the clusters of an image. Our algorithm detects the peaks of clusters in the global threedimensional color histogram of an image. We utilize the histogram bins rather than the pixels themselves to find the peaks of clusters; thus, our algorithm can find the peaks efficiently. Then, the algorithm associates the pixels with the detected clusters based on the local structures of the clusters. The hill-climbing algorithm is outlined as follows: 1 Compute the global 3-dimensional HSV color histogram of the image. 2 Start at a non-zero bin of the color histogram and make uphill moves until reaching a peak (local maximum) as follows: 2.1 Compare the number of pixels of the current histogram bin with the number of pixels of neighboring bins. Note that the feature space is three-dimensional, thus each bin in the color histogram has 3 d -1 = 26 neighbors, where d is the dimensionality of the feature space (d=3). 2.2 If the neighboring bins have different numbers of pixels, the algorithm makes an uphill move towards the neighboring bin with larger number of pixels. 2.3 If the immediate neighboring bins have the same number of pixels, the algorithm checks the next neighboring bins, and so on, until a set of neighboring bins with different number of pixels are found. Then, an uphill move is made towards the bin with larger number of pixels. 2.4 The uphill climbing is continued (repeat steps ) until reaching a bin from where there is no possible uphill movement. That is the case when the neighboring bins have smaller numbers of pixels than the current bin. Hence, the current bin is identified as a peak. 3 Select another unclimbed bin as a starting bin and perform step 2 to find another peak. This step is continued until all non-zero bins of the color histogram 2

3 NII Journal No. 7 (2003.9) are climbed (associated with a peak). 4 The identified peaks represent the initial number of clusters of the input image; therefore, these peaks are saved. 5 Finally, neighboring pixels that lead to the same peak are grouped together. This associates every pixel with one of the identified peaks. Consequently, the segments of the input image are formed. Although we use a global color histogram to find the peaks, step 5 takes into account the spatial information of the pixels when forming the segments, i.e., only spatially close (neighboring) pixels that lead to the same peak are grouped into one segment. An example of segmentation by the hill-climbing algorithm is shown in Figure 1. 3 Extracted Features Four features were extracted from the image regions as we discuss in following subsections. 3.1 Color Histogram A good color space is one in which the perceived color differences should correspond to their Euclidean distances in this chosen color space. The HSV color model is known to satisfy this property. Furthermore, the HSV model simulates the human perception of colors, where the H, S, and V components represent hue, saturation, and value (brightness), respectively. Thus, we represent the color content of each region by its HSV color histogram. 3.2 Edge Direction Histogram There are many different types of edge detector operators. We use the popular Canny edge detector [10]. The Canny edge detector is a first derivative Gaussian. It optimizes the trade-off between noise suppression and localization. The edges are much cleaner and allow better image analysis. It is therefore a very popular edge detection method for high-end applications. 3.3 Higher-Order Autocorrelation Edge Vector The higher order autocorrelation features are the primitive edge features that are shift-invariant (irrelevant to where the objects are located in the image), which is a useful property in image querying [11]. Before extracting these features, an edge image is constructed using the Canny method (Section 3.2). 3.4 Dual-Tree Complex Wavelet Transform The discrete wavelet transform (DWT) is most commonly used in its maximally decimated form in signal compression [12]. In order to have directional selectivity, it is necessary to use complex coefficient filters. Further, the shift invariance is obtained by the use of the undecimated form of filters. Therefore, we use a dualtree complex wavelet transform (DT CWT), which solves the above problems. 4 Support Vector Machines (SVMs) Binary classifiers construct a hyperplane separating members of one class from those of the other. However, real data are non-separable in most cases for which there does not exist a hyperplane that can successfully separate the members of the two classes. One solution to this Figure 1. Examples of segmentation of Natural images by the hill-climbing algorithm. 3

4 Semantic Approach to Image Database Classification and Retrieval problem is to map the data into a higher-dimensional space, where the members of the two classes can be separated by a hyperplane. However, this mapping process incurs two problems: (1) representing the input data with highdimensional vectors can be extremely expensive in terms of memory and time, and (2) separating the data in this way may lead to trivial solutions that overfit the data. Support Vector Machines (SVMs) method [13] solves the above two problems. SVMs method avoids overfitting the data by choosing a hyperplane that can separate the data maximizing the minimum distance from the hyperplane to the closest training point. Such a hyperplane is called the maximum margin hyperplane (Figure 2). Another advantage of the SVMs is the compact representation of the decision boundary. Only the training points that lie on, or between, hyperplanes H 1 and H 2 are considered as the support vectors and give nonzero real values. The other points are ignored. Thus, the number of support vectors is small as compared to the number of points in the training set. A kernel function acts as a similarity function between points. A simple form of similarity metric is the dot product between two vectors. In a space in which the positive members of a class form more than one cluster as in image databases, a Gaussian classifier is known to be more accurate [14]. 2 x y Kxy (, ) = exp 2 2σ Where σ is the width of the Gaussian. Thus, in this paper, we use a Gaussian kernel (shown above), which is a technique used by most radial basis function classifiers. Optimal hyperplane H 1 H 2 W margin 5 Classification of Image Regions To classify image regions into semantic classes, which humans understand, we first manually defined a hierarchy that reflects the semantics in the Nature images, as shown in Figure 3. Learning the semantics of each class is performed using SVMs based on heterogeneous features of training sample regions of each class. We built the conservative SVMs classifiers: P (classified notmemberofclass) These binary classifiers achieve good class separation under the constraint that each region belongs to only one, or none, of the classes. Due to the fact that different semantic classes may have similar visual features (such as OceanSunset and SkySunset classes), it is difficult to build a precise binary classifier using the class organization of Figure 3. After extensive experimentation with our system, we found that for the purpose of identifying the image regions, it is essential to reorganize the defined classes as shown in Figure 4. This organization reflects the similarity/dissimilarity of visual features of the semantic classes. Using the organization of Figure 4, a SVMs binary classifier determines whether a certain image region is a member of some class C i or not. If the image region is identified as a member of class C i it is mapped into C i in the semantic class hierarchy of Figure 3. For example, a binary SVMs classifier P (Sand&Rock notmemberof Sand&Rock) differentiates regions that belong to Sand&Rock collection from others. Then, the regions that are classified into the Sand&Rock collection undergo further identification test to be classified into the Sand class by the P(Sand notmemberof Sand) classifier or into the Rock class by the P(Rock notmemberof Rock) classifier. For each classifier in Figure 4, we prepared a training set of sample image regions. We selected 800 region samples to train the binary SVMs classifiers; on the average, 60 training samples per class. Note that our proposed method shows that semantic classification with high precision can be achieved with such a small number of training samples for every class. Figure 2. A linear hyperplane separating the members of two classes. Here, the support vectors are circled. 4

5 NII Journal No. 7 (2003.9) Scenery Objects NatureObject ArtificialObject Water Sky Land SunsetOcean Waterfall Night BlueSky Rock Sand BlueOcean Sunset Cloud Vegetation Snow GreenForest RedForest Figure 3. A class hierarchy that reflects the semantics of Nature image regions. CWT NatureObject All Regions CWT ArtificialObject CLR CLR CLR CLR CLR Blue&White Night Sand&Rock GreenForest DarkOrange CLR RedForest BlueSky BlueOcean HAC Sand Rock HAC SkySunset OceanSunset HAC Cloud Snow CWT Waterfall Classes to be mapped Figure 4. A class organization used by the SVMs to distinguish the members of a class from others. The feature used to distinguish the members of a class is written on top of a box representing the class, where CLR = color, CWT = dual-tree complex wavelet transform, = edge direction histogram, and HAC = higher-order autocorrelation edge feature 5

6 Semantic Approach to Image Database Classification and Retrieval Figure 5. Keyword-based query interface. Figure 6. Result of a query: keyword = BlueSky, type of query = Single Match. 6 Keyword-Based Indexing Due to the semantic classification of image regions, we are able to represent an image by the classes, in which the image regions are classified. That is each image is represented by a set of keywords (classes names). The choice of keyword-based approach allow for highly intuitive query interface. The keywords in Figure 5 are organized hierarchically similar to that of the semantic class hierarchy (Figure 3). The user is allowed to search by selecting one, or more, keywords. This keywordbased query interface supports Single Match queries, which returns images that contain one, or more, of the selected classes. For example, Figure 6 shows the result of a query that selected the keyword BlueSky and a Single 6

7 NII Journal No. 7 (2003.9) Match type of query. The intention of this query is to find all images that contain a blue sky region. 7 Experiments We implemented the system using Java on a PC running Windows 2000 with Pentium 1GHz CPU and 512MB RAM. The 6000 images used in the experiments were selected from two collections: (1) 4000 images from h2soft ( and (2) 2000 images from the image collection of Stanford University (wwwdb.stanford.edu/image/). At first, we performed the classification Nature regions vs. Artificial regions using different features as shown in Table 1. Edge features show good classification results since edges in Artificial regions tend to be straighter than those of Nature regions. We chose the CWT feature since it gives slightly better precision than the feature. Note that the HAC and CLR features gave low precision for this experiment and thus it is not possible to use them for classification. Table 1: Classification of Nature regions vs. Artificial regions. Nature Artificial 90.7 % 90.7 % CWT 90.8 % 91.0 % HAC Not possible Not possible Then, for the next level (see Figure 4) the best classification results were obtained using the color feature due to the nature of class grouping that reflect the color similarity/dissimilarity between the classes at this level. The high precision values (91.4% 97.2%) of classifications are given in Table 2. Table 2: Classification after regrouping classes as shown in Figure 4 using Color Class Name Precision Subclasses Blue&White 93.2 % BlueSky, Cloud, Snow, Waterfall, BlueOcean Night 95.8 % Sand&Rock 91.4 % Sand, Rock GreenForest 97.2 % Sunset 91.8 % SkySunset, OceanSunset RedForest 95.7 % Table 3: Classification of lower-level classes (Figure 4) using different features Class Feature Precision BlueSky 89.6 Cloud HAC 84.4 Snow CWT 93.3 Waterfall 88.1 BlueOcean 85.9 Sand HAC 89.3 Rock HAC 89.3 SkySunset 87.4 OceanSunset 86.7 Different lower-lever classes are distinguished by different features. Thus, we selected the features that give the highest precision of classification (Figure 4). As shown in Table 3, the subclasses of Blue&White class were classified using the following features: for BlueSky, HAC for Cloud, CWT for Snow, for Waterfall and for BlueOcean. Similarly, each of the Sand&Rock and Sunset classes were divided into 2 subclasses each. The HAC feature leads to the best classification precision for both Sand and Rock subclasses. On the other hand, the feature leads to the best classification precision for the SkySunset and OceanSunset subclasses. 8 Conclusion This paper presented an approach to classify image regions based on their semantics rather than their low-level features. The pre-defined semantic class hierarchy reflects the semantics in the Nature image regions that humans understand. Hence, flexible and intuitive query compositions can be based on such semantic hierarchy. As each image is represented by the classes in which its regions were classified, these semantic classes (labels of an image) are high-level descriptors that can be easily understood by a novice user. The use of the binary SVMs classifiers to classify image regions using different features at different levels in the hierarchy were the main reasons behind the high classification precision that we achieved in our experiments. Currently, we are looking into adding more classifiers to include more classes into the system. A final goal is to generate semantic indices into image databases. 7

8 Semantic Approach to Image Database Classification and Retrieval References [1] M.Flickner, H.Sawhney, J.Ashley, Q.Huang, B.Dom, M.Gorkani, J.Hafner, D.Lee, D.Petkovic, D.Steele, P.Yanker Query By Image and Video Content: The QBIC System. IEEE Computer Magazine, Sep [2] J.R.Smith, S-F.Chang. VisualSEEK: A fully automated content-based image query system. ACM Multimedia Conf., Nov [3] J-H.Lim Explicit Query Formulation with Visual Keywords. ACM Multimedia, Oct [4] P.Mulhem, J-H.Lim. Symbolic Photograph Content-Based Retrieval. ACM Int l Conf. On Information and Knowledge Management (CIKM), Nov [5] B.Bradshaw. Semantic Based Image Retrieval: A Probabilistic Approach. ACM Multimedia, Oct [6] A.Vailaya, A.Jain, M.Figueiredo, H.J.Zhang. Content-Based Hierarchial Classification of Vacation Images. IEEE Int l Conf. On Multimedia Computing and Systems, Jun [7] C.P.Town, D.Sinclair. Content Based Image Retrieval Using Semantic Visual Categories. Technical Report TR , AT&T Laboratories Cambridge, [8] W.Wang, Y.Song, A.Zhang. Semantic-Based Image Retrieval By Region Saliency. Int l Conf. On Image and Video Retrieval, July, [9] T.Ohashi, Z.Aghbari, A.Makinouchi. Hill- Climbing Algorithm for Efficient Color-Based Image Segmentation. IASTED Int l Conf. On Signal Processing, Pattern Recognition, and Applications (SPPRA 2003), Rhodes, Greece, June, [10] J. Canny. A Computational Approach to Edge Detection. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.8, pp , [11] M.Kubo, Z.Aghbari, K-S. Oh, A.Makinouchi. Image Retrieval by Edge Features Using Higher Order Autocorrelation in a SOM Environment. IEICE Trans. on Information and Systems. IEICE Transactions on Information and Systems, vol. E86-D, no. 8, Aug. 2003, pp [12] N.Kingsbury. Complex Wavelets for Shift Invariant Analysis and Filtering of Signals. Applied and Computational Harmonic Analysis, Elsevier Science, vol. 10, no. 3, pp , [13] V. Vapnik. Statistical Learning Theory. Wiley, [14] C.J.C. Burges, "A Tutorial on Support Vector Machines for Pattern Recognition", Data Mining and Knowledge Discovery, vol. 2, no. 2, p ,