FRACTAL DIMENSION BASED TECHNIQUE FOR DATABASE IMAGE RETRIEVAL

Transcription

1 FRACTAL DIMENSION BASED TECHNIQUE FOR DATABASE IMAGE RETRIEVAL Radu DOBRESCU*, Florin IONESCU** *POLITEHNICA University, Bucharest, Romania, **Technische Hochschule Konstanz, Abstract: The objective of the present work is to evaluate the potential of the use of the image fractal dimension as textural feature in a content-based image retrieval (CBIR) system. An interactive CBIR system that uses for the classification of images a cummulative feature including the fractal dimension is presented, together with the performances of the algorithm that determines the local and connected fractality of an image. A procedure to take decisions on the image relevance is presented also. Keywords: content-based image retrieval, fractal dimension, local fractality, textural features, features integration, feature relevance 1. INTRODUCTION Advances in modern computer and telecommunication technologies have led to huge archives of multimedia data in diverse application areas such as medicine, remote sensing, entertainment, education and on-line information services. To use this widely available multimedia information effectively, efficient methods for storage, browsing, indexing, and retrieval must be developed. Different multimedia data types may require specific indexing and retrieval tools and methodologies. In the early 1990s, due to the emergence of large-scale image collections, content-based image retrieval (CBIR) was proposed as a way to overcome datbase access difficulties. In CBIR, images are automatically indexed by summarizing their visual contents through automatically extracted quantities, or features, such as color, texture or shape. Thus, low-level numerical features, extracted by a computer, are substituted for higher-level, text-based, manual annotations or keywords. Since the inception of CBIR, many techniques have been developed along this direction and many retrieval systems, both research and commercial, have been built. Note that ideally CBIR systems should automatically extract (and index) the semantic content of images to meet the requirements of specific application areas. Low-level features such as colors, textures and shapes of objects are widely used for CBIR. However, in specific applications, such as medical imaging, low-level features play a substantial role in defining the content of the data. A typical contentbased image retrieval system is depicted in Figure 1. Fig.1. An image retrieval system architecture

2 The image collection database contains raw images for the purpose of visual display. The visual feature repository stores visual features extracted from images needed to support content-based image retrieval. The text annotation repository contains keywords and free-text descriptions of images. Multi-dimensional indexing is used to achieve fast retrieval and make the system scalable to large image collections. The retrieval engine includes a query interface and a query-processing unit. The query interface, typically employing graphical displays and direct manipulation techniques, collects information from users and displays retrieval results. The query-processing unit is used to translate user queries into an internal form, which is then submitted to the DBMS. Moreover, inorder to gap the bridge between low-level visual features and high-level semantic meanings,users are usually allowed to communicate with the search engine in an interactive way. 2. TEXTURE INFORMATION AS INTEGRATED IMAGE FEATURES In computer vision, texture is defined as all what is left after color and local shape have been considered or it is defined by such terms as structure and randomness. Many common textures are composed of small textons usually too great in number to be perceived as isolated objects. The elements can be placed more or less regularly or randomly. They can be almost identical or subject to large variations in their appearance and pose. Texture based approaches in extracting relevant feature sets from images in databases have been shown to give very encouraging results in addressing this problem. Textures are replications, symmetries and combinations of various basic patterns or local functions, usually with some random variation. Textures have the implicit strength that they are based on intuitive notions of visual similarity. This means that they are particularly useful for searching visual databases and other human computer interaction applications. However, since the notion of texture is tied to the human semantic meaning, computational descriptions have been broad, vague and something conflicting. Many methods have been proposed to extract texture features either directly from the image statistics, e.g., cooccurrence matrix, or from the spatial frequency domain. In our researches we have studied the performance of four types of features: Markov Random Fields parameters, Gabor multi-channel features, fractal-based features and co-occurrence features, but in this paper the retrieval technique is based on the fractal dimension estimation of a lot of image features which are generated using various data transformation techniques. Experience shows that the use of a single class of descriptors to index an image database does not generally produce results that are adequate for real applications, and retrieval results are often unsatisfactory even for a research prototype. A strategy to potentially improve image retrieval, both in terms of speed and quality of results, is to combine multiple heterogeneous features. We can categorize feature integration as either sequential or parallel. Sequential feature integration, also called feature filtering, is a multistage process in which different features are sequentially used to prune a candidate image set. In the parallel feature integration approach, several features are used concurrently in the retrieval process. In the latter case, different weights need to be appropriately assigned to different features, since different features have different discriminating power depending on the application and specific task. The feature integration approach appears to be superior to using individual features and, as a consequence, it is implemented in most current CBIR systems. The main limitation of feature integration in most existing CBIR systems is the heavy involvement of the user, who not only must select the features to be used for each individual query, but also must specify their relative weights. An interactive CBIR system designed to simplify this problem will be discussed in the final section of the paper. This system uses the concept of integrated (or cumulative) features. The general class of accumulating features aggregate the spatial information of a partitioning irrespective of the image data. A special type of accumulative features are the global features which are calculated from the entire image. Accumulating features are symbolized by: ( x) F = h ο f (1) j T j where represents an aggregations operation (the sum in this case, but it may be a more complex operator), F j is the set of accumulative features or a set of accumulative features ranked in a histogram. F j is part of feature space F. T j is the partitioning over which the value of F j is computed. The operator h may hold relative weights, for example, to compute transform coefficients. A simple but very effective approach to accumulating features is

3 to use the histogram, that is, the set of features F(m) ordered by histogram index m. Joint histograms add local texture or local shape, directed edges and local higher order structures. Some of the most known textural features used in image database retrieval were described in Aksoy and Haralick works [1, 2]. From these proposals, a set of features is computed from the line-angle-ratio statistics which is a texture histogram method [3] that uses the spatial relationships between lines as well as the properties of their surroundings. Spatial relationships are represented by the angles between intersecting line pairs and properties of the surroundings are represented by the ratio of the mean grey levels inside and outside the regions spanned by those angles. This solution was already tested in a previous work [4] and lead to the idea to use the fractal dimension of a bitmap image representation as texture histogram method, considering that an efficient retrieval algorithm should be able to retrieve images that are not only close (based on similarity) to the query image but also close to each other. The objective of this paper is to precise the measure of confidence in the fractal dimension of an image as alternative to the Haralick's textural features for evaluate the characteristics of different digital images. A classification system prototype built for evaluating purposes are also analysed. 3. THE DETERMINATION OF AN IMAGE FRACTAL DIMENSION The dimension of a fractal curve is a number that characterizes the way in which the measured length between given points increases as scale decreases. Whilst the topological dimension of a line is always 1 and that of a surface always 2, the fractal dimension may be any real number between 1 and 2. The fractal dimension D is defined by: ( L2 / L1 ) ( S / S ) log D = (1) log 1 where L 1, L 2 are the measured lengths of two curves (in units), and S 1, S 2 are the sizes of the units (i.e. the scales) used in the measurements. As an example, if the measurements of the same curve with S 1 = 1 and S 2 = 0.5 give the lengths of L 1 = 7 and L 2 = 20, respectively, we have: D = log (20/7) / log (2) = In similar fashion, the transition from S = 1 to S = 2 yields the slightly smaller estimate of D = 1.22 and from S = 2 to S = 3, D ~ A fractal surface exhibits roughness in the third dimension whose variance is independent of scale. The magnitude of this variance is related to the surface's fractal dimension. The latter can be computed under the assumption of a statistically self-similar roughness and a two-dimensional motion generating surface heights. The fractal dimension can be used to classify two-dimensional digital images, if interpreted as a threedimensional terrain whose height is given by the pixel values. Such images, however, do not exhibit selfsimilarity at arbitrary scales. At a large scale, segment membership constitutes a counteracting deterministic component, and at small scales the image resolution poses a limitation to the computation of the fractal dimension. On medium scales, however, the surface roughness may show a segment-dependent self-similarity. A local fractal dimension (LFD), solely based on the evaluation of this scale range, should yield a parameter for every pixel whose value is related to local surface roughness. Edges add a deterministic element to surface height degrading locally the 'surface fractality' and leading to an underestimation of the LFD. A different local fractal dimension will be determined at edges even if the two segments being incident to the edge do have the same LFD. Moreover, the local fractal dimension makes it possible to discriminate between high frequency components as generated by noise - leading to an increased LFD - and those as generated by an edge - leading to a decreased LFD due to a degraded 'fractality'. A software application realised in the Delphi 5 environment allows to compute the fractal dimension according to the local dimensions and to compute the histogram of the fractal dimension distributions. The application analyzes digital image files of bitmap type. In order to determine the local fractality, for each pixel P the algorithm build a square of side e with the center in P. After counting all the pixels M(e) from this square, the factor α is determined as: α =Log( M (e) )/ Log (e) ) (3) The slope of the graphical representation of function α represents the local fractality. For the determination of the local connected fractality the algorithm consider the number M(e) as the total number of pixels local connected in a square of variable side e centered in point P, i.e. all pixels from the greatest square used in the analysis. We define also a subset as connected if for any pairs of points x and y of the subset the distance belongs to this subset. The algorithm computes the fractal dimension for a single threshold level as follows: a graph of curvature is constructed in which the abscissa is a sequential count of pixels as the 2

4 zone boundary is followed, and the ordinate is the curvature at the corresponding pixel. The curvature graph is then searched for segments that are approximately self-affine. Each self-affine segment is then modeled with multiple fractal interpolation functions. For each model, a fractal dimension is computed and thus a family of fractal-dimension estimates is calculated. An overall fractal-dimension feature is derived as the mean of this family. 4. INTERACTIVE CONTENT-BASED IMAGE RETRIEVAL In early stages of the development of CBIR, research was primarily focused on exploring various feature representations, hoping to find a best representation for each feature. In these systems, users first select some visual features of interest, and then specify the weight for each representation [5], [6]. This burdens the user by requiring a comprehensive knowledge of low-level feature representations in the retrieval system. There are two more important reasons why these systems are limited: the difficulty of representing semantics by means of lowlevel features, and the subjectivity of the human visual system. In our approach an interactive process that involves the human as a part of the retrieval process having the general scheme shown in Fig.2 was considered. There are 4 main parts in this system: the image database, the feature database, the similarity metric selection part and the feature relevance computation part. At the start of processing a query, the system has no a priori knowledge about the query, thus all features are considered equally important and used to compute the similarity measure. The top N similar images will be returned to the user as retrieval results and the user can further refine it by sending relevance feedback to the system until he or she is satisfied [7]. The learning-based feature relevance computation part is used to re-compute the weights of each feature based on user s feedback, and the metric selection part is used to choose the best similarity metric for the input feature weight vector based on reinforcement learning. Fig.2. Interactive CBIR system architecture Therefore, by considering user s feedback, the system can automatically adjust the query and provide a better approximation to the user s expectation. Moreover, by adopting relevance feedback, burdens of concept mapping and specification of weights are removed. Now, the user only needs to mark images that are relevant to the query, and the weight of each feature used for relevance computation is dynamically updated to model high level concepts and subjective perception. The implemented retrieval agorithm consists of the following steps:

5 Extraction of visual features from a database of images and provide links to the images from these features. Extraction of visual features from the query image using the same procedure used for feature extraction of the database images Comparing the features of the query image and features of images from the database and return the best matches as the retrieved images using a distance measure for comparison and a given threshold. The feature extraction procedure was carried out in the following manner: a. Determination of the histogram peak for the local fractality of the selected image F1 b. Determination of the histogram peak for the local connected fractality of the same image F2 c. Determination of the Wavelet based subband coding of the image [8] and estimation of the fractal dimension of each subband [9]. The vector of fractal dimensions constituted the feature vector for each image in the database F3 The last step in the retrieval procedure is the decision on the image relevance. Given an image, we have to decide which images in the database are relevant to it, and we have to retrieve the most relevant ones as the results of the query. In our experiments we use a likelihood ratio approach which is a Gaussian classifier, which define two classes, namely the relevance class A and the irrelevance class B. Given feature vectors of a pair of images, if these images are similar, they should be assigned to the relevance class, if not, they should be assigned to the irrelevance class. In order to evaluate the retrieval performance, two traditional measures: precision and recall were used. Precision is the percentage of retrieved images that are correct and recall is the percentage of correct images that are retrieved. Note that computation of these measures requires prior groundtruthing of the database. Since our automatically generated groundtruths are not the ones required for precision and recall, these measures can not be used directly. We use modified versions of them to evaluate the performance of our algorithm. After manually grouping a smaller set of sub-images in our database, we will evaluate the performance using precision and recall too. To test the retrieval performance, we use the following procedure. Given an input query image of size KxK, images are retrieved in descending order of likelihood ratio or ascending order of distance for nearest neighbor rule. If the correct image is retrieved as one of the k best matches, it is considered a success. Average rank of the correct image is also computed. This can also be stated as a nearest neighbor classification problem where the relevance class is defined to be the k best matches and the irrelevance class is the rest of the images. For this experiment, we use the non-shifted sub-images to compute the best case performance and the shifted sub-images to compute the worst case performance. We call this the worst case because the shifted sub-images overlap a sub-image in the database by only half the area. All other possible sub-images have a sub-image in the database which they overlap by more than half the area. This experimental procedure is appropriate to our problem of retrieving images which have some section in them that is like the user input image. 5. EXPERIMENTAL RESULTS An textural features based retrieval procedure is presented relative to images taken in the fortified saxon church of Homorod (Hamruden), near Brasov. We have created a database with 18 images with the size set at 128x128 gray tones preserving the properties of the material. Several actions can be performed using this data base: Selecting zones of original painting Selecting zones of painting from the same period Underlining of the most damaged zones Matching of images from the data base according to textural features Retrieval of the most appropriate image with a given context. Fig. 3 shows a possibility to analyse an image according to its textural features. Fig. 3. Interpretation of zones with different textural aspect

6 As an example of the retrieval method efectiveness, we have classified images according to the three global features F1, F2 and F3 in order to group the images having the same symbol a cross on a church roof. For each feature three window dimensions (A: 64x64 pixels; B:128x128 pixels; C: 256x256 pixels) where used. The final results are summarized in table 1. Table 1. Results summary Features/ Window size F1/A F1/B F1/C F2/A F2/B F2/C F3/A F3/B F3/C % of correct classification It seams obvious that the accumulative feature F3 give the best classification results. In the same time, a small window size allow to obtain a better accuracy. 6. CONCLUSIONS In this paper current technical achievements in visual feature extraction and CBIR system design are analysed. A new interactive CBIR system architecture is discused, together with the possibillity to use a cummulative integrated feature including the fractal dimension of the image. The algorithm used for the determination of the fractal dimension allows to compute both local and local-connected fractality. These properties are included in a retrieval algorithm that ensures the comparison between a feature vector of the query image and the feature vector of images from the database and return the best matches as retrieved images. Three kinds of feature vector are proposed, all having in common the use of the image fractality. The experiments proved that a classification of images into classes according to the image relevance allows effective high dimensional indexing and presents atractivity for applications where image collection sizes continue to expand rapidly. Results of the classification effectiveness tests showed that the algorithm assigned 80% of the sub-image pairs we were sure were relevant to the relevance class correctly. The obtained results encourage a further investigation involving larger databases. REFERENCES [1] S. Aksoy and R.M Haralick, Textural Features for Image Database Retrieval, Proc. of the IEEE Workshop on Context-Based Access of Image, vol. CVPR 98, Santa Barbara, 1998, [2] S. Aksoy and R.M Haralick, Content-based image database retrieval using variances of gray level spatial dependencies, Proc. of IAPR Int. Workshop on Multimedia Information Analysis and Retrieval, 8, 1998, [3] G. Pass and R. Zabith, Comparing Images Using Joint Histograms, Multimedia Systems, vol. 7, pp , 1999 [4] R. Dobrescu and M. Dobrescu, Using textural features in analysis of digital images from a cultural heritage database, Int. Workshop on Romanian Cultural Heritage Preservation, Bucharest, 2001, [5] Z. Yang, X. Wan and C.-C. J. Kuo, Interactive Image Retrieval: Concept, procedure and tools, IEEE 32nd Asilomar Conference, Montery, CA, pp , Nov [6] A.M. Smeulders, M. Worring, S. Santini, A. Gupta and R. Jain - Content-Based Image Retrieval at the end of the early years, IEEE Trans. on Pattern Analysis and Machine Intell., vol. 22, no. 12, Dec. 2000, pp [7] B. Bhanu, J. Peng and S. Qing, Learning feature relevance and similarity metrics in image databases, CBAVIL, pp , [8] G. C.-H. Chuang and C.-C. J. Kuo, Wavelet descriptor of planar curves: Theory and applications, IEEE Trans. Image Proc., 5(1), pp , [9] A. V. Nevel, Texture synthesis via matching first and second order statistics of a wavelet frame decomposition, IEEE International Conference on Image Processing, 1998.