Digital Image Retrieval Using Intermediate Semantic Features and Multistep Search

Transcription

1 Digital Image Retrieval Using Intermediate Semantic Features and Multistep Search Dengsheng Zhang 1, Ying Liu 1, and Jin Hou 2 1 Gippsland School of IT, Monash University, 2 School of I.S.T., Southwest Jiaotong University Churchill, VIC 3842, Australia Chengdu, Sichuan , China dengsheng.zhang, ying.liu@infotech.monash.edu.au jhou@home.swjtu.edu.cn Abstract Recently, semantic image retrieval has attracted large amount of interest due to the rapid growth of digital image storage. However, existing approaches have severe limitations. In this paper, a new approach to digital image retrieval using intermediate semantic features and multistep search has been proposed. Instead of looking for human level semantics which is too challenging at this stage, the research looks for heuristic information and intermediate semantic features which can describe image content objectively. Different from the conventional approaches, the intermediate features are used as filters to eliminate large amount of irrelevant images. Conventional content based image retrieval techniques and relevance feedback (RF) are applied following the filtering to improve the retrieval accuracy. The proposed system has the power of capturing both regional features and global features, and making use of both semantic features and low level features. The proposed system also uses a powerful user interface to provide users with convenient retrieval mechanisms including SQL, RF and query by example. Results show the system has a significant gain over existing region based and global image retrieval approaches. 1. Introduction Digital images have been growing in explosive speed. Unfortunately, it has been found that the more the images, the harder to make use of them. The current demand for digital images is like sitting over a gold vault crying for money. On the one hand there are billions or even trillions of images in the storage, on the other hand, we keep creating duplicate images for production. Literally, an image is consisted of a number of individual physical objects, however, the content of an image tells much more than adding individual objects together. While human being is good at describing individual objects, it becomes much more complicated and difficult when a number of objects are put together into an image. Human interpretation of an image can be very subjective and ambiguous depending on the user s gender, age, profession, education level and cultural background. Researches have been done to learn how human being understands an image and to achieve automatic and objective understanding of image content, this is called content based image retrieval (CBIR). Eakins and Graham [1] suggested that CBIR can be divided into three levels and can be achieved in three phases accordingly. The first level is the low level image retrieval, while the second and third level is the higher level or semantic level image retrieval. Currently, there is a significant gap between low level and higher level image retrieval, this is called semantic gap in literature. Recently, a number of approaches have been attempted to narrow down the semantic gap. The best known technique in this aspect is the relevance feedback (RF) [2, 3]. However, RF alone works on the low level features, it relies on the user s feedback to refine weights given to different low level features, it does not provide semantic retrieval functionality for users. Furthermore, the problem with the RF is that once the initial low level retrieval fails, the user will not have any chance to provide relevant feedback. Therefore, an acceptable initial retrieval result is crucial to RF success. The second type of approach is the concept based techniques, including the use of ontology to define high-level concepts [4, 5], machine learning to associate low-level features with query concepts to achieve image classifications [6~8], using semantic templates or dictionary to interpret low level features [9, 10]. However, the problem with these techniques is that the defined concepts or terms are limited, arbitrary and subjective. Another approach is the thesaurus based technique which combines both the

2 visual content of images and the textual information obtained from the Web for WWW image retrieval [11]. Because each keyword acquired from web documents has many synonyms, a thesaurus has to be used to group synonyms. Due to the very high dimension of visual and textual features and the intensive handling of large amount of thesaurus information online, it can drastically reduce the retrieval performance and is even impractical. In this paper, we propose a new approach to obtain both heuristic information and intermediate features to allow user retrieving images as simple as the conventional way. The proposed system employs heuristic information and intermediate features to select relevant images or eliminate irrelevant images. By narrowing down the search list with these powerful information, conventional low level and relevance feedback analysis can be more efficiently and effectively applied to improve the retrieval performance. For example, human face, background plants, linear and circular objects are all powerful heuristic information or intermediate features useful in finding relevant images or eliminating irrelevant images. The system employs reasonable knowledge from users, and makes use of state of the art image retrieval techniques to significantly improve retrieval efficiency and accuracy. The rest of the paper is organized as following. Section 2 provides an overview of the proposed system. Section 3 describes the proposed heuristic information and intermediate features respectively. Retrieval user interface is described in Section 4. We present our experimental results in Section 5. Finally, the paper is concluded in Section Overview of the System It is well known that the low level approach completely ignores human knowledge role in the retrieval, while the higher level concepts defined are insufficient and arbitrary to represent general collections of images [12]. Heuristic information and intermediate features have been suggested in literature, but they were only used for retrieving classified images, and have not attracted appropriate attention [13]. In this paper, we propose to use heuristic information and intermediate features from images to develop a general semantic image retrieval system with similar functionality to the conventional SQL system. The proposed system is consisted of two parts: offline processing units and online processing units as illustrated in Figure 1. The offline process starts by segmenting input crude images into regions; it is followed by extraction of primitive image features; a converting machine then interprets the regions into heuristic information or translates the primitive features into intermediate features; finally, images are represented as heuristics and intermediate features to be indexed into the database. The offline processing is typically done in a backend server. The online process is initiated by a query submitted by the user, the system then applies three stages of retrieval processing to return all images similar to the query. The three stages of retrieval consist of heuristic filter, intermediate filter and relevance feedback. The online processing is typically conducted on a client terminal using a web browser. Figure 1. Block diagram of the proposed system.

3 3. Use Semantic Features as Filter to Select Relevant Images 3.1 Heuristic Features Heuristic information is very powerful for image retrieval, and many of this type of information are ubiquitously available in images. Take human face as an example, large number of images includes human faces. Therefore, we can employ this heuristic information to narrow down the search. For instance, if a user submits a query searching for Clinton, then the search engine can narrow down the search by finding images only containing human face. On the contrary, if a user wants to search for the moon, then the search engine can eliminate those images containing human face. Many advanced face detection techniques are available [14, 15]. However, for image retrieval purpose, there is no need for complex recognition of facial expression, pose, eyes and mouth, accurate boundary etc. As the result, the technique of model based feature localization on skin-tone color map coupled with light compensation is used in our system [14]. Image background is also a powerful heuristic information like green plants (trees/grass), water, sky, land, rock, sand, snow, building etc. Other heuristic information includes animals, clouds, rain, firework, bricks, stone etc. These types of heuristic information can be detected by two steps. The first is to do segmentation to separate them from the image. The second step is to recognize them as particular objects by using color and texture features of the objects/regions [18]. These objects/regions usually have distinct color and texture features which can be used as heuristic information. Spatial information in the image can also be employed to detect background. The feasibility of the approach is that we don t need a very accurate segmentation of the image. Rather, a rough segmentation of image into distinct regions is sufficient for our purpose, and this is achievable using the JSEG segmentation technique [16]. Some examples of segmentation are shown in Figure 2. Figure 2. Examples of region segmentation. Another heuristic is cluttered versus clustered image. A cluttered image has abundant colors and small regions, perceptually, it looks very busy. While a clustered image has only a small number of colors and comprises a few large regions, perceptually, it looks peaceful. Cluttered and clustered images can be detected by using the segmentation technique shown in Figure 2. The difference between the proposed heuristic approach and the conventional concept based techniques is that we use the identified heuristics as filters. While the concept based techniques attempt to use a few concepts to define categories which are insufficient and arbitrary for general collections. In the filter system, it simply eliminates those irrelevant images to shortlist the search, the shortlist is then put into further analysis using intermediate filter and relevance feedback. The proposed heuristic information is the auxiliary mechanism (filter) or component to the system, a few types of ubiquitous heuristic information can significantly narrow down the search, increase the efficiency and effectiveness of the retrieval. As more types of heuristic information is available, the system becomes more robust and effective Intermediate Features It has been shown that it is impossible to define sufficient semantic concepts for large collection of images [12]. However, certain intermediate level of semantic information can be extracted from images, like linear or circular objects. The lines and circles in images are important information to distinguish images containing man made objects and images comprising pure natural scenes. Many man made objects have linear or circular edges, while natural scenes do not. This key characteristic can be used to drastically narrow down the search. The linear and circular objects in an image can be detected using edge detection followed by the generalized Hough transform. An example of Hough transform detection of man made objects in an image is shown in the Figure 3. Many other intermediate features can be found from primitive image features like color, texture and shape. Color histogram can be used to derive common color hues/tones such as pink, burgundy, deep blue, gray, charcoal, cream etc. which are familiar to ordinary users. These intermediate color features of an image can be obtained based on the dominant color tone of those significant image regions [17]. Common texture terms familiar to human such as coarseness, directionality, contrast, line-likeness, regularity, and roughness etc. can also be obtained from images using statistics. Similar to the heuristic information, the acquired intermediate image features can be used as filters to select the relevant images for further analysis or eliminate the irrelevant images from the list. It will significantly improve the retrieval efficiency and accuracy. Compared with the concepts used in concept

4 based techniques, these heuristics and intermediate features are objective and can be specified by ordinary users. (a) (b) (d) (c) Figure 3. Example of Hough transform. (a) original image; (b) Hough transform; (c) peaks detected on the Hough transform map; (d) lines detected based on the peaks in (c). 4. Structured Query Language Interface Contrast to existing query interface, which is either query by example image or query by keyword, we provide user with structured query interface similar to the SQL interface in conventional DBMS. Since all the above described heuristic information and intermediate features can be translated into appropriate semantic keywords, by joining these semantic words using logical operators, we are able to provide structured query. Typical structured queries are like these: find images with linear objects which selects images with man made objects, find images without green plants which discards large number of outdoor scenery images, find images with linear objects AND human face which selects images with town activities or indoor activities, etc. The query and retrieval interface of the proposed system is illustrated in Figure 4. The retrieval of similar images from the database is divided into three stages. The first stage is a filter process, the user submitted a structured query, the system will select the relevant images and eliminate irrelevant images from the database using both the heuristic and intermediate filter. The result is an unranked retrieval list. In the second stage, the user can conduct the conventional query by example to rank the initial retrieval list based on primitive region features of the query image. In the third stage, the user will label the retrieved images from the second stage as relevant or irrelevant, the system then refine and rerank the retrieval result. The RF in the third stage can be done for a few times to further improve the retrieval accuracy. Currently, the RF is yet to be implemented. However, the RF function has been shown in the figure for illustration purpose. Figure 4. User interface of the retrieval system.

5 5. Experimental Results In this section, we conduct image retrieval experiments on the proposed techniques and system. Corel Photo Gallery is used as the image database. As an affiliated member of the SCHEMA project [19], we have 5,000 Corel images obtained from SCHEMA as the test set. SCHEMA is a project funded by the European Community under the Information Society Technology programme. The Corel Photo Gallery has a large amount of images of various contents and is often used to evaluate the performance of image retrieval systems in literature. The images in the Gallery have been pre-classified into different categories (each containing 100 images) by domain professionals. It contains natural scenery images from 50 variant semantic categories such as Alaska, Apes, Australia, Beaches, Deserts, Fields, Firework, Hiking, Sunsets, Tigers and Waves. Each image in the database is segmented using JSEG technique. Regions with less than 5% of the image size are discarded. In total, regions are obtained, with 5.84 regions per image on average. For each region, 35 common colors are extracted based on dominant color in HSV color space. Six Tamura texture features are extracted for each image: coarseness, directionality, contrast, line-likeness, regularity, and roughness. Man made objects in an image are detected using the general hough transform. 20 heuristic features such as sky, water, plants etc. have been learned from 600 sample regions. Two other heuristic features have also been extracted, busyness feature is extracted by using thresholding the number of regions in a image, and face feature is extracted by two class classification based on skin-tone color. In total, 65 heuristic and intermediate features are obtained for each image. For each region in an image, its low level features are represented using 3 dimensional dominant color vector (H, S, V) and 24 dimensional Gabor feature vector (μ 00, σ 00,., μ 35, σ 35 ) (where μ ij and σ ij are the mean and standard deviation at scale i and orientation j). To compare with existing image retrieval techniques, the precision-recall curve is used to measure the retrieval performance. In a precision-recall curve, recall is defined as the ratio of the number of images retrieved (N r ) to the total number of relevant images available in the database (N t ), while precision is defined as the ratio of the number of relevant images retrieved (N r ) to the number of retrieved images K. That is, recall=n r /N t and precision=n r /K. The proposed multi-step search technique is compared with the two region based methods used in [17] and the conventional global image histogram method. The two region based methods are RBIR-CN which uses semantic color names of regions to retrieve images and RBIR-C which uses low level dominant color of regions to retrieve images. 100 images are randomly selected from the test database as queries, their retrieval performance are averaged to obtain the final precision-recall curve which is shown in Figure 5. From Figure 5, it is clearly shown that the proposed multi-step search (multi-search) method using intermediate semantic features shows significant advantage over existing region based methods and conventional histogram method. Existing region based methods usually fail to capture the global features of an image while the global histogram method cannot capture local and regional features in an image. The proposed system has overcome the disadvantages in both the region based and global histogram approaches. Furthermore, the proposed multi-step search makes use of both semantic and low level features, this significantly enhance the retrieval capability of the system. It should be indicated that although the multi-step search using intermediate semantic features shows significant advantage over the region based color method, in some cases, region based color method can have better performance especially when an category has large homogenous background. Considering this factor, the low level retrieval using query by example is provided as a useful alternative in the proposed system. 6. Conclusions In this paper, a new approach to digital image retrieval using intermediate semantic features and multi-step search has been proposed. The approach indicates a new direction from existing image retrieval approaches which works on either higher level semantic features or low level features. Contrast to existing systems in literature, the proposed system has the capability of capturing both regional features and global features, and making use of both semantic features and low level features. Results show that the proposed system has significant advantage over existing techniques and is promising. The proposed system also has a powerful SQL based retrieval interface to support both semantic retrieval and low level retrieval. The system is highly extensible, new semantic features can be easily incorporated into the system as they are available. In the future, the system will be improved to include the RF functionality. More semantic features will also be learnt to expand the present heuristic categories.

6 Precision PR-Chart RBIR-CN RBIR-C Global-C Multi-Search Recall Figure 5. Precision-recall chart of the average retrieval result. 7. References [1] J. P. Eakins and M. E. Graham, Content-based Image Retrieval, Technical Report, University of Northumbria at Newcastle, [2] Z. Su, et al, Relevance Feedback in Content-Based Image Retrieval: Bayesian Framework, Feature Subspaces, and Progressive Learning, IEEE Trans. on Image Proc., 12(8): , 2003 [3] D. Tao, et al, Asymmetric Bagging and Random Subspace for Support Vector Machines-based Relevance Feedback in Image Retrieval, IEEE PAMI, 28(7): , [4] E. Hyvönen and S. Saarela, A. Styrman and K. Viljanen, Ontology-Based Image Retrieval, In Proc of WWW2003, Budapest, Hungary, May, [5] S. Jiang, T. Huang and W. Gao, An Ontology-based Approach to Retrieve Digitized Art Images, In Proc. of IEEE/WIC/ACM Conference on Web Intelligence, pp , [6] H. Feng, and T. Chua, A Bootstrapping Approach to Annotating Large Image Collection, Workshop on Multimedia Information Retrieval in ACM Multimedia, pp , [7] S. Tong, and E.Chang, Support Vector Machine Active Learning for Image Retrieval, Proc. of ACM Inter. Conf. on Multimedia, Ottawa, Canada, pp , [8] L. Cao and Li Fei-Fei, Spatially Coherent Latent Topic Model for Concurrent Object Segmentation and Classification, In Proc. of IEEE Inter. Conf. in Computer Vision (ICCV), [9] J. R. Smith and C. Li, Decoding Image Semantics Using Composite Region Templates, In Proc. of IEEE Workshop on Content-Based Access of Image and Video Libraries, pp. 9-13, [10] Y. Zhuang, X. Liu and Y. Pan, Apply Semantic Template to Support Content-based Image Retrieval, SPIE, Storage and Retrieval for Media Databases, Vol.3972: , [11] Y. Lu et al, A Unified Framework for Semantics and Feature Based Relevance Feedback in Image Retrieval Systems, ACM Inter. Conf. on Multimedia, pp , [12] Y. Liu, D. S. Zhang and G. Lu, Narrowing Down The Semantic Gap in Content-Based Image Retrieval A Survey, Pattern Recognition, 40(1): , [13] M. Obeid, B. Jedynak and M. Daoudi, Image Indexing and Retrieval Using Intermediate Features, In Proc. of the 9th ACM International Conference on Multimedia, pp , Ottawa, Canada, Sept.30-Oct.5, [14] R. Hsu, M. Mottaleb, and A. K. Jain, Face Detection in Color Images, IEEE Trans. PAMI, 24(5): , [15] M. Yang, D. Kriegman, N. Ahuja, Detecting Faces in Images: A Survey, IEEE Trans. PAMI, 24(1):34-58, [16] Y. Deng, B. S. Manjunath, Unsupervised Segmentation of Color-Texture Regions in Images and Video, IEEE Trans. on Pattern Analysis and Machine Learning (PAMI), 23(8): , [17] Y. Liu, D. S. Zhang, G. Lu and W.-Y. Ma, Region-based Image Retrieval with Perceptual Colors, In Proc. of 5th Pacific-Rim Conference on Multimedia (PCM04), Tokyo, Japan, Nov.30-Dec.3, LNCS3332, pp , [18] Y. Liu, D. S. Zhang, and G. Lu, Deriving High-Level Concepts Using Fuzzy-ID3 Decision Tree for Image Retrieval, In Proc. of IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing (ICASSP05), pp , PA, USA, [19] accessed in Dec., 2007.