FIRST: Fractal Indexing and Retrieval SysTem for Image Databases

Transcription

1 Image and Vision Computing 16 (1998) FIRST: Fractal Indexing and Retrieval SysTem for Image Databases M. Nappi*, G. Polese, G. Tortora Dipartimento di Matematica ed Informatica Università degli Studi di Salerno, Baronissi, Italy Received 15 May 1997; accepted 16 January 1998 Abstract We present an image indexing method and a system to perform content-based retrieval in heterogeneous image databases (IDB). The method is based upon the fractal framework of the iterated function systems (IFS) widely used for image compression. The image index is represented through a vector of numeric features, corresponding to contractive functions (CF) of the IFS framework. The construction of the index vector requires a preliminary processing of the images to select an appropriate set of indexing features (i.e. contractive functions). The latter will be successively used to fill in the vector components, computed as frequencies by which the selected contractive functions appear inside the images. In order to manipulate the index vectors efficiently we use discrete Fourier transform (DFT) to reduce their cardinalities and use a spatial access method (SAM), like R*-tree, to improve search performances. The sound theoretical framework underlying the method enabled us to formally prove some properties of the index. However, for a complete validation of the indexing method, also in terms of effectiveness and efficacy, we performed several experiments on a large collection of images from different domains, which revealed good system performances with a low percentage of false alarms and false dismissals Elsevier Science B.V. All rights reserved. Keywords: Content-based retrieval; Iterated functions systems; Contractive function; Discrete Fourier transform; R*-tree 1. Introduction The integration of multimedia processing within computing systems has increased research interests in multimedia database management systems. Particular attention has been devoted to image database issues and to the problem of retrieval by image contents, which has led to the development of new search data structures and strategies, query paradigms, and indexing techniques. Among the search data structures we find tree-based methods like k d-trees and R*-trees [1,2], and linear methods, like quadtree-based [3] or space-filling curve-based [4]. Widely used query paradigms are the query by example (QBE) and the query by sketch (QBS), both allowing visual specification of the contents for the images to be retrieved. Several image indexing techniques have been proposed [5 11]. Most of them provide an iconic representation for the query images (examples or sketches). Then, in order to retrieve images based upon their contents, an approximate match is performed between the iconic representations of the query and of the database images. * Corresponding author. Tel.: ; Fax: ; E- mail: micnap@dia.unisa.it In this paper we present a new image indexing technique for true colour (24 bpp) images, based on the fractal theory of contractive functions (CFs) [12], using both QBE and QBS paradigms. The index structure for the generic image is a vector of numeric features, representing the frequencies by which sets of selected CFs appear in the iterated function systems (IFS) representation of the image. In order to improve performances during index construction and image retrieval, index dimensionality is reduced by using discrete Fourier transform (DFT) [13]. Moreover, index vectors are organized in an R*-tree to speed up navigation through the index structures. The method has been embedded in the system FIRST (Fractal Indexing and Retrieval SysTem). It also allows indexing fractalcompressed images, which turns out to be very useful in distributed environments. We proved the validity of the method by exploiting the theoretical framework of IFS and by running several experiments on a large collection of heterogeneous images. In fact, the IFS property stating the continuous dependence of the image on the coefficients of the CFs guarantees that small variations of the IFS coefficients yield small variations on the associated images. However, for a complete validation of the indexing method we have run several /98/$ - see front matter 1998 Elsevier Science B.V. All rights reserved. PII: S ( 98)

2 1020 M. Nappi et al./image and Vision Computing 16 (1998) experiments with FIRST on different types of images, including medical images (CT scans), scenes with animals (fishes and rabbits), tools, women portraits, etc. We have observed a limited number of false alarms and a low percentage of false dismissals [6]. Physicians for what concerns medical images have evaluated the FIRST. System usability has been judged adequate, since the interface enables visual management of the main aspects of a traditional tomograph. The paper is organized as follows. In Section 2 we review the basic concepts of IFS theory and describe a block indexing technique to speed up IFS coding. In Section 3 we explain how we used IFS principles to perform image indexing and introduce basic definitions. Section 4 describes the indexing method, whereas in Section 5 we show experimental results and provide performance evaluation according to well-known evaluation methods. Finally, conclusions and future developments are given in Section Principles of IFS theory IFS Theory [12] is based upon the concept of contractive function. The latter can be defined as a function f:s S on a complete metric space (S, h) such that there exists a constant c for which the following condition is satisfied: h(f (x), f (y)) c h(x, y) x, y S (1) where h is the metric function and the real number c [0,1[ is the contractive factor for f. From Banach s fixed point theorem we know that each CF converges to a point x f S independently from the starting point to which it is applied. On the other hand, the result of the collage theorem provides a strategy to choose the set of functions that best approximates a given set of points, which are the ones producing the points closest to the original ones [12]. These theoretical results have been exploited to solve the problem of finding the set of CFs that best approximate the points of a given image, leading to a technique for image coding. Such a set of CFs is called IFS. The IFS of an image can be constructed using Jaquin s algorithm [14], which partitions images into regions and encodes them by finding a set of CFs mapping image regions into similar image regions. The former, e.g. the transformed image regions are called domains, whereas the latter are called ranges. The range partition can only contain disjoint image regions, whereas the domain partition can contain overlapping domains. In general, transformations of an IFS have the following form: x a i b i 0 x e i w i 6 4 y 7 5 ¼ 6 4 c i d i y 7 5 þ 6 s i (2) z 0 0 a i z b i where w i is the transformation, (x,y,z) is a region point to be encoded (x, y being the spatial coordinates and z the grey Fig. 1. Original Sierpinsky s triangle. level), the coefficient a i represents the greyscale factor, e i and s i translate the point in the spatial domain, whereas b i translates it in the greyscale domain. The coefficients a i,b i, c i, and d i represent both the geometric contraction and the isometric transformation. The latter includes identity, orthogonal reflection (about axis and diagonal), and rotation around the centre of block through different angles [14]. IFS coding provides a compact but still rich representation of an image. We will show that IFS is also a suitable framework to build an image index. Several arguments can be used to prove this assertion. In fact, IFS is a stable system because of the continuous dependence of the image contents on the coefficients of the IFS representation, meaning that small variations of the coefficients result in small variations of the image contents [12]. As an example, consider Figs 1 and 2, where two similar images, namely Sierpinsky s triangle and its distorted version, have associated similar IFS representatives, as illustrated in Tables 1 and 2. Moreover, scaling and isometric transformations cause slight changes to the IFS representation of the original image. Thus, the IFS representations for an image and its transformed versions can all be computed by using similar IFS codes. Fig. 2. Distorted version of Sierpinsky s triangle.

3 M. Nappi et al./image and Vision Computing 16 (1998) Table 1 IFS for Sierpinsky s triangle CF a b c d e f w w ¹80 1 w IFS coding is a computationally complex process. In fact, for each range we have to explore a very large domain pool to find the domain that can be most effectively transformed into the given range by using Eq. (2). In the following subsection we present a block indexing strategy that will be used to reduce the candidate domains from the domain pool that can be transformed into a given range. Therefore, in Section 3 we show how to build an image index based on IFS and show how this preserves the basic properties of image indexing, as discussed above The block indexing technique The primary goal in devising a block indexing strategy is to derive necessary conditions for similitude among image domains and ranges, both referred to in this paper by the term block. This means establishing an analogy between grey values and weights. If two blocks have the same shape, then they have the same weight distribution; our aim is to determine a minimal number of features that are sufficiently representative of the weight distribution. The first feature that comes to mind is the position of the mass centre mc, since similar weight distributions have their mcs close to each other. The coordinates of the mc for an N N block B 0 are given by: x 0 ¼ 1 M 0 0 i N 0 j N ib 0 (i, j) y 0 ¼ 1 M 0 0 i N 0 j N jb 0 (i, j) (3) 0 i N where M 0 ¼ 0 j N B 0(i, j) is the mass of the block. However, mc proximity is only a necessary condition for block similarity: the position of the mc alone does not adequately describe the mass distribution of the block. In order to characterize the mass distribution further, we consider the mcs (x k, y k ) of all the transformed blocks: 2 B k (i, j) ¼ B k ¹ 1 (i, j) ¹ m k ¹ 1 for 1 k n (4) where m k ¼ M k / N2 is the average mass per pixel in the block B k. this operation yields a transformation from the space of pixels to a 2n-dimensional space of features. The value 3 is a suitable choice for k since higher values add more overhead than useful information. As a result, we describe the block through a feature vector (x 0, y 0, x 1, y 1, x 2, y 2 ) that has a considerably low dimension [15]. The blocks in the space of features must now be organized to allow the application of a spatial access method (SAM). Among the possible choices for a SAM there are tree-based methods (k d-trees and R*-trees are the most efficient [1,2]), linear methods, which can be based on quadtree [3] or space-filling curves [4], and finally, grid-based methods such as the cell technique [3]. In particular, we used a variation of the n-dimensional cell method where the cells are unevenly spaced: as shown in Fig. 3, each (x k, y k )-plane is divided into planecells. The central stripes are thinner so as to minimize collisions, since the features are more likely to fall near the centre. The search strategy is as follows: each range becomes the query point r for the domain that will encode it. The search is limited to a hypersphere H(r) of radius d centred in r, comprising all the points x such that d(x,r) d, where the distance d between two feature points v and w is defined as: d v 0,,v 2n ¹ 1, w0,,w 2n ¹ 1 ¼ maxj v 0 i 2n ¹ w i j i The cells that have a non-empty intersection with H(r) are then searched for the optimum match. Obviously, the radius d affects the search time. When d ¼ we have ordinary exhaustive search, which is very costly but guarantees the best quality for a fixed bit rate [16]. For d ¼ 0 we have a method that falls in the classification category, since for each range only one cell is searched; for 0 d we have a true feature vector technique. Since we have used nine planecells, the number of classes on which we can distribute the blocks is 9 3 ¼ 243. ð5þ Table 2 IFS for distorted Sierpinsky s triangle CF a b c d e f w w ¹80 1 w

4 1022 M. Nappi et al./image and Vision Computing 16 (1998) Fig. 3. Subdivision of the (x k, y k )-plane in nine planecells. The Cartesian product of n planecells on different planes yields a cell. 3. IFS as a framework for image indexing There are some important properties that an image index should satisfy [6]. In particular, compactness, stability, scale and isometric transformation invariance are very critical for many applications. We aimed to meet them through an appropriate employment of the CFS from the IFS framework. It has already been stated that the IFS representation of an image is strongly related to that of a generic scale or isometric transformation of the image, so that one can be easily inferred from the other [17]. Our aim is to build an image index, based on the IFS framework, which is compact, stable, and invariant to the transformations mentioned above. In order to describe our indexing method we will now introduce its basic entities and the notation that will be used. A primary concern in lossy image coding is the quality of the image, in terms of the ratio between the bit rate and the signal to noise ratio (SNR). Using a quadtree-based fractal coding [18] affects the image quality in terms of both the size of the domain pool and the quantization coefficients a and b. The former depends upon two parameters, namely the quadtree level and the domain shift. Let N be the maximum quadtree level used during fractal coding, the domain pool D is defined as: D ¼ c N D i, D i D j ¼ for i j (6) i ¼ 1 where each D i is the domain pool derived using the i-th quadtree level. Frequently used levels correspond to the following block sizes: 32 32, 16 16, 8 8, 4 4: The cardinality of each D i is given by the following formula: img_size ¹ dom_size þ 1 jd i j¼ 2 (7) shift where img_size represents the size of the image, dom_size is the size of the domain, and shift is the displacement of the domain window with respect to the two cartesian coordinates. Let COE be the set of all the possible pairs (cartesian product) of coefficients (a, b), with a and b appropriately quantized, and 8 the number of the possible isometric transformations (derived through the combination of the coefficients a i, b i, c i, d i, e i and f i ), the cardinality of the set of CFs for a fixed domain shift is expressed through the following formula: jcfj¼jdj jcoej 8 (8) Depicted in Fig. 4 is a hypothetical coding scenario: the given image Q has img_size ¼ 16, shift ¼ 8, and two quadtree levels corresponding to domain sizes of and 8 8, respectively. We can compute the cardinality of the domain pool starting from Eq. (6), and the number of Fig. 4. A hypothetical scenario of IFS coding.

5 M. Nappi et al./image and Vision Computing 16 (1998) Fig. 5. IFS code for the image Q of Fig. 4(a); and structure of a generic CF (b). domains for each domain size by using Eq. (7). Thus, the number of domains of size is 16 ¹ 16 þ 1 2 ¼ 1, 8 the number of domains of size 8 8is 16 ¹ 8 þ 1 2 ¼ (2) 2 ¼ 4 8 and the cardinality of the domain pool is D ¼ 4 þ 1 ¼ 5. The IFS code for the image Q is shown in Fig. 5. Obviously, the number of CFs is equal to the number of ranges in the partition. The structure of the generic contractive function CF i associated with the range R i is shown in Fig. 5b: the first component (x,y) contains the spatial coordinates of the domain D i chosen for encoding R i through CF i ; the second component (Dom_size) represents the size of D i ; the third component (a, b) contains the parameters of the transformation; the fourth component (iso) is the isometric transformation chosen for D i ; finally, the last component (Lev) is the quadtree level. As we will see, the basic idea underlying our indexing method is the representation of an image through a histogram of CFs, exploiting the characteristic that a single CF is often associated with multiple ranges [17]. One naive way to perform image indexing through CFs could be to use a vector v of CF components as an index for the image. In order to preserve quality in terms of SNR and other appropriate parameters, we could select the same number of CFs used for the IFS coding of the original image. Here each vector component v[i], i [0, CF-1 ], represents the frequency by which the i-th CF occurs inside the image. The values of the vector components are set during the coding phase. In particular, the value of the single vector component is increased every time we find the optimal transformation for its associated range block. The index vector is built for each of the images in the database and for each of the query images entered by the user when querying the database. The similarity comparison between query images and database images is done by applying the Euclidean distance function to their index vectors, since these can be viewed as points in the n-dimensional space. The query image can be considered as the centre of an hypersphere which includes the index vectors of all the images in the answer set for the given query. In other words, given a query q and its index vector x, for each image I in the database we apply the following formula:! n 1=2 d ~x, ~y ¼ jx t ¹ y t j 2 (9) t ¼ 1 where y is the index vector associated with the image I, and n ¼ CF is the number of components of the index vector. Let DIS be a maximum similarity threshold, the answer set for the query q will contain all the vectors y satisfying the following property: d ~x, ~y DIS (10) The goodness of the query results depends upon the percentage of relevant images falling in the answer set for the query. The index vector for the hypothetical image Q of Fig. 4 is shown in Fig. 6. The vector represents the histogram of the frequency by which the selected CFs appear inside the image Q. Without loss of generality, in this vector CF 0 is assigned to position (i), CF 1 to (j), CF 3 to (h), CF 4 to (k), CF 5 to (l), CF 6 to (r), CF 9 to (t). Notice that these CFs have the following properties: CF 0 ¼ CF 2 CF 5 ¼ CF 7 ¼ CF 8 : According to the CF structure shown in Fig. 5b, this means that (xy) 0 ¼ (xy) 2 (xy) 5 ¼ (xy) 7 ¼ (xy) 8 Fig. 6. The index vector for the image Q of Fig. 4.

6 1024 M. Nappi et al./image and Vision Computing 16 (1998) Dim_size 0 ¼ Dim_size 2 Dim_size 5 ¼ Dim_size 7 ¼ Dim_size 8 (ab) 0 ¼ (ab) 2 (ab) 5 ¼ (ab) 7 ¼ (ab) 8 iso 0 ¼ iso 2 iso 5 ¼ iso 7 ¼ iso 8 The cardinality of the vector is computed according to Eq. (8). The association between vector components and CFs is performed through an appropriate computation on the parameters of the CFs. Other than computationally inefficient, this naive method would not provide an effective image index. In fact, in order to discriminate images based upon their index representations, an index feature (a CF in our case) should represent a salient characteristic of the image, that is, it should appear with a sufficiently high frequency inside the image [19]. Instead, this method would often produce flat frequency histograms, meaning that most of the CFs are referenced, but each with a low frequency. Thus, we needed to exploit the CFs in a more efficient way. For this purpose we provided a hierarchical structure on the set of CFs so that only a subset of their parameters is used during index construction and during image retrieval, as shown in the following section. 4. The CF-based indexing technique In this section we show how we exploited the IFS framework to derive a method for image indexing. Then, we will prove the validity of the method by using some properties of the IFS framework together with experimental results observed by running the FIRST on a large collection of heterogeneous images. In the interests of simplicity the method is described for square images, the extension to the general case of rectangular images is straightforward and will be omitted. In subsections that follow we will first describe an indexing technique for monochromatic images (8 bpp), and true colour images (24 bpp), which considers the only luminance component in the YIQ model. Then we present an extension of the technique to include the representation of the chrominance component, which will be very helpful to reduce the answer set further Indexing with the luminance component In Section 3 we introduced a naive method to perform image indexing through IFS and highlighted its poor performances in terms of both computational complexity and efficacy. Early attempts to employ CFs for building an image index have been made in refs. [17,20]. In spite of the encouraging experimental results, there were several problems due to the elevated number of indexing features, which caused computational inefficiencies in some cases and often produced flat histograms. In fact, it is possible that an image used most of the available CFs during index construction, each with a low frequency. This would reduce the ability to discriminate among images at index level. Thus, we have revised the indexing method by providing a taxonomic organization of its parameters and by creating a separate frequency histogram for each of them, yielding a hierarchical index structure. At the first level of the index structure we have the frequency histogram of the classes used for classifying the domains. As shown in Section 2.1, the selection of the domain that best approximates a given range is performed through a process in which both domains and ranges are first classified and then associated, based upon their morphological and geometric characteristics. Let n be the cardinality of the feature vector for the classes of domains and cl[] the vector representing the first index level, we obtain the following histogram: cl[i] ¼ f_cl i, 0 i n, 0 f_cl i max_range (11) where f_cl i represents the frequency by which class i is referenced during the IFS coding of an image, and max_- range is the maximum number of ranges to be encoded for an image. In other words, f_cl i represents the number of times the domains in class i are exploited to encode the ranges of an image. The histogram defined by Eq. (11) represents the primary index level we scan for similarity search. In this way we build a primary answer set S cl computed by using Eq. (9) as a similarity function and including all the feature vectors satisfying Eq. (10). Let the variables contained in Table 3 represent the coefficients of the CFs used for the IFS coding of the prototype image Q of Fig. 4. In this table the domains are also characterized by the class c i they belong. In Fig. 7 we show the vector cl Q [] of the frequencies by which each class is used for encoding the image Q. This vector is used to create the primary answer set S cl, generated by selecting the images I for which the euclidean distance between cl I [] and cl Q [] satisfies Eq. (10). Once the primary answer set S cl is generated we might need to restrict its magnitude by using the other components of the CF to reduce the possible false alarms. In order to do that, we perform a search on the second index level, corresponding to the frequency histogram of the domains DOM i and their associated coefficients (a, b) li, where the subscript l indicates that the pair (a, b) is computed upon the luminance component. We know that each domain DOM i can be mapped on a number of different ranges by varying the coefficients (a, b) li. Moreover, from Eq. (6) we infer the number of possible domains D. The number of coefficients (a b) i is given by COE, which leads to D COE (cartesian product) possible associations. These associations are

7 M. Nappi et al./image and Vision Computing 16 (1998) Table 3 Coefficients of the contractive functions used for encoding the image Q CF Dom_Coord Dom_Size iso COE Class CF 0 (xy) 0 Dom_Size 0 isom 0 (ab) 0 c 0 CF 1 (xy) 0 Dom_Size 0 isom 0 (ab) 2 c 2 CF 2 (xy) 0 Dom_Size 0 isom 0 (ab) 0 c 0 CF 3 (xy) 1 Dom_Size 1 isom 1 (ab) 3 c 3 CF 4 (xy) 4 Dom_Size 4 isom 4 (ab) 4 c 4 CF 5 (xy) 5 Dom_Size 2 isom 5 (ab) 5 c 5 CF 6 (xy) 2 Dom_Size 2 isom 7 (ab) 1 c 0 CF 7 (xy) 5 Dom_Size 2 isom 5 (ab) 5 c 5 CF 8 (xy) 5 Dom_Size 2 isom 5 (ab) 5 c 5 CF 9 (xy) 4 Dom_Size 4 isom 3 (ab) 6 c 5 represented through the vector dc[], having D COE components. Thus, the answer set S cl is narrowed by performing a secondary search on its elements, according to their dc[] vector. As a result, a secondary answer set S dc is produced and it satisfies the following property: S dc S cl (12) In Fig. 8 we show the vector dc[35] corresponding to the example of Fig. 4: its cardinality is 35 and it is computed as the product between D ¼ 5 and COE ¼ Indexing with the chrominance component The algorithm for IFS compression can also process colour images in RGB format. The image is converted into luminance and chrominance components, one for luminance and two for chrominance, by using the conversion standard YIQ. By exploiting some intrinsic morphologic characteristics, the two chrominance components can be encoded by using the same quadtree partitioning scheme computed for the luminance component and re-computing the coefficients (a, b). After these have been appropriately quantized, these coefficients are combined through a cartesian product and are organized in a vector cr[], in which each component cr[i] represents the frequency by which the pair (a, b) cj appears in the chrominance component. Search with cr[] is performed on the elements of the secondary answer set S dc and leads to the generation of the answer set S cr satisfying the following property: S cr S dc S cl (13) The basic steps executed by our retrieval algorithm are depicted in Fig. 9. We can reassume them as follows: Step 1. A primary answer set S cl is generated as a result of the comparison between the vectors cl[], representing the frequencies by which classes of domains are used during the coding of images based upon the luminance component; the vectors cl[] are organized in an R*-tree; Step 2. A restricted answer set S dc is generated by comparing the vectors dc[] of the luminance component on the answer set S cl computed in the previous step; Step 3. A final answer set S cr is computed by comparing the vectors cr[] of the chrominance component on the set S dc. The aim here is to detect similar chromatic areas. Notice that the invocation of Step 3 causes the chrominance component to be taken into consideration, hence it will only be executed in the presence of true colour query images. In this case, among the images that were included in the answer set S dc in Step 2, the monochromatic images will be considered more distant from the query image than the true colour images. Thus, they will be listed at the bottom of the answer set S cr. Also notice that the employment of a SAM has been limited to the first level cl[] vector of the index. We could have applied it also to the remaining levels. However, since most of the candidate images are eliminated during the search on the first index level, it is sufficient to use sequential search on the two other levels to achieve acceptable performances. We will now show how FIRST is capable of discriminating images. For this sake, let us examine Figs 10 and 11, in Fig. 7. The class frequency vector cl Q [].

8 1026 M. Nappi et al./image and Vision Computing 16 (1998) Fig. 8. The vector dc Q [] for the image Q of Fig. 4. which there are represented luminance components of four images at 24 bpp, namely Redconey, Rockhind, Sky T and Sky T , which represent two fishes and two landscapes. For each component we have represented the quadtree produced during fractal coding, which highlights the similarity between the two pairs of luminance components. In order to analyse the complexity of this indexing method, we need to consider three components, that is, the complexity to build the IFS coding, to compute the DFT, and to perform similarity search. The construction of the IFS coding depends upon the granularity of the domains used for partitioning the original image. Its complexity has been considerably reduced with the recent introduction of classification schemes [15,19], hybrid systems [21], and parallel algorithms [15]. Moreover, we can apply DFT [13] to map cl[], dc[] and cr[] feature vectors on the domain of frequencies, in which it is sufficient to keep few coefficients without loss of significant information (cut off of high frequencies) from the original signal [13]. However, such transformations have to preserve the results of possible queries. For this reason, we can use Parseval s theorem [13], proving that the application of DFT to transform a pair of points from n-dimensional to k-dimensional space does not increase their euclidean distance [22]. The computational complexity for DFT is constant, since the cardinality of the feature vector is constant. The complexity of similarity search is composed by the time for computing the euclidean distance plus the time for scanning the database. The former is constant because DFT considerably reduces the dimension of the space for feature vectors. Database scanning is linear in the number of images populating the database if we use sequential scanning. However, we have observed a considerable improvement using R*-trees on cl[] vectors [23,24]. Fig. 9. Search taxonomy. 5. Experimental results and performance evaluation The FIRST runs on box AIX and Windows 95 operating systems. The version for AIX has been implemented using C language, whereas the one for Windows has been implemented using Java. This section presents experimental results and performance evaluation observed by running the system on an IBM RS/6000 workstation, with databases of different size, containing images from different application domains. We gained acceptable performances both in terms of IFS coding and indexing using three quadtree levels (block size 32, 16, and 8), setting a value for the domain shift of 4 pixels, and quantizing a and b with 3 and 5 bits, respectively. The evaluation method we have used is the AVRR [9], an extension of the normalized recall [25]. The extension allows us to deal with an approximate match in order to make the evaluation method suitable for image retrieval. In order to assess performances of FIRST with respect to human perceptual similarity, we have selected a sample of 15 heterogeneous images and for each of them we have manually selected the 20 most similar images from the database. Then, we have contrasted them with the images retrieved by FIRST automatically. For each query we have computed the following measures [9]: AVRR: the average rank over the set of relevant images retrieved by the FIRST; IAVRR: the ideal AVRR. In our case this coincides with the AVRR when the system perfectly retrieves the 20 most relevant images. We have run experiments on image databases containing and images, progressively selected from our complete set of images by using random selection. Some results are shown in Table 4. The first column represents the size of the database, whereas the second and third columns report the AVRR and the IAVRR, respectively, averaged over the 15 test queries. The ratio and the difference of AVRR and IAVRR give a measure of the effectiveness of the indexing method. In Table 5 we provide a different type of experimental result observed on a sample of 10 queries. The first column represents the feature used to perform the search, the second column represents the minimum euclidean distance necessary to achieve 0 false dismissals, expressed as the average percentage of the radius of the hypersphere for a given feature. The third column represents the average percentage of false alarms with respect to the answer set, whereas the

9 M. Nappi et al./image and Vision Computing 16 (1998) Fig. 10. Examples of quadtree partition on luminance components using fractal coding. last column indicates the average euclidean distance necessary to achieve 0 false alarms. Examples of retrieval using both QBE and QBS paradigms are given in Figs Notice that in Figs 12, and 14 the monochromatic images are listed only after the coloured images in the answer set S cr. In fact, as explained above, this always happens when we extend the search to the chrominance component on the last index level. In this experiment we used DFT to reduce the cardinality of the feature vector. We always used four coefficients to cut off high frequencies, which required the storage of 8 values since each coefficient is a complex number. Experimental Fig. 11. Examples of quadtree partition on luminance components using fractal coding.

10 1028 M. Nappi et al./image and Vision Computing 16 (1998) Table 4 Performance of FIRST using AVRR Size of DB AVRR IAVRR results have shown that this cut-off frequency is fairly adequate to achieve good performances, both in terms of computing time and number of false dismissals. The experiments revealed the superiority of the cl[] R*- tree search over sequential scanning. Fig. 15 plots the response time averaged over a set of 30 queries. We notice that R*-tree search always outperforms sequential scanning independently from the tolerance factor we choose. Fig. 16 plots the average retrieval response time for sequential and R*-tree search using a tolerance between 0 and 1 and varying the number of records in the answer set. Finally, in Fig. 17 we show the trend for the response time when changing the size of the database. Obviously, the performance gap between the two different scanning techniques widens as the number of stored images grows. 6. Conclusion and further research In this paper we propose FIRST, an IFS-based image indexing technique and a system for content-based retrieval in image databases. We have provided experimental results and evaluated system performances in terms of computing time and retrieval precision. In future we aim to further investigate the following issues: 1. Devise visual language generation approaches to enable the construction of the index and of the query interface through visual definition and manipulation languages [26]. This will allow an easier customization of the database for new collections of images and will speed up the generation of an appropriate query interface. For this Table 5 Effectiveness of FIRST in terms of false alarms and false dismissals Features 0 false dismissals No. false alarms First false alarm class d ¼ 25% 13% d ¼ 18% domain and (a, b) d ¼ 12% 0 d ¼ 13% on luminance (a, b) d ¼ 5% 0 d ¼ 8% on crominance Fig. 12. Visual query results using FIRST with the QBE paradigm.

11 M. Nappi et al./image and Vision Computing 16 (1998) Fig. 13. Visual query results using FIRST with the QBE paradigm. reason we are developing a visual language-based approach for index construction, in which special iconic operators [27] are applied to visual objects to perform the different stages required for data definition, index construction and for constructing special views for specific users. 2. Allow the system to run over the World Wide Web. In particular, we want to exploit the fact that images can be Fig. 14. Visual query results using FIRST with the QBS paradigm.

12 1030 M. Nappi et al./image and Vision Computing 16 (1998) Fig. 15. Time performances running the system with different tolerances (1 ¼ exact match): sequential scanning vs R*-tree search. Fig. 16. Time performances obtained by varying the number of records in the answer set (the tolerance ranges between 0 and 1): sequential scanning vs R*-tree search. Fig. 17. Time performances running the system with image databases of different size: sequential scanning vs R*-tree search. queried while they are in the fractal compressed format. 3. Further investigation of the IFS theoretical framework to prove properties of the index other than the continuous dependence of images on their IFS representations. In particular, we are interested in finding relationships among the CFs, in order to predict the most appropriate ordering of the vector components, so as to produce the most suitable signal to be processed by a DFT. 4. Examination of other orthonormal transformations, in addition to DFT.

13 M. Nappi et al./image and Vision Computing 16 (1998) Acknowledgements Part of the database has been populated with the implicit cooperation of Prof. Stan Sclaroff from the Image and Video Computing Group of the Computer Science Department at Boston University. Part of the images we have used, namely the ones with animals and tools, have been downloaded from his site. As requested, we hereby confirm that the images with animals have been taken from the following childrens books: Greenberg, Guide to Corals and Fishes of Florida, the Bahamas, and the Caribbean, Seahawk Press, Miami, FL; Alden, Peterson First Guides: Mammals, Houghton Mifflin, Boston, MA. The images with tools have been acquired by Prof. Stan Sclaroff from MIT Media Laboratory.The images have a resolution of 256*256 and 24/bits per pixel. Each of them has been encoded with the IFS-based algorithm, using several combinations of parameters. References [1] N. Beckmann, H.P. Kriegel, R. Schneider, B. Seeger, The R*-tree: An efficient and robust access method for points and rectangles, Proc. of ACM SIGMOD, May, 1990, pp [2] J.D. Ullman, Principles of Database and Knowledge-based Systems, Computer Science Press, Rockville, MD, USA, [3] H. Samet, The Design and Analysis of Spatial Data Structures, Addison Wesley, New York, [4] H.V. Jagadish, Linear clustering of objects with multiple attributes, Proc. of ACM SIGMOD, Atlantic City, NJ, May 1990, pp [5] S.K. Chang, Q.Y. Shi, C.W. Yan, Iconic indexing by 2D-strings, IEE Trans. on Pattern. Analysis. Machine. Intell. 3 (1987) [6] IEEE Computer, Finding the right image, Special Issue on Content Based Image Retrieval Systems, 28, No. 9, Sept [7] A. Del Bimbo, P. Pala, Visual image retrieval by elastic matching of user sketches, IEEE Trans. On PAMI, Feb [8] M. De Marsico, L. Cinque, S. Levialdi, Indexing pictorial document by their content: a survey of current techniques, Image and Vision Computing 15 (1997) [9] C. Faloutsos, W. Equitz, M. Flickner, W. Niblack, D. Petkovic, R. Barber, Efficient and effective querying by image content, Journal of Intell. Inf. Systems 3 (3/4) (1994) [10] W.I. Grosky, P. Neo, R. Mehrotra, A pictorial index mechanism for model-based matching, Data and Knowledge Eng. 8 (1992) [11] S.Y. Lee, F.J. Hsu, Spatial reasoning and similarity retrieval of image using 2D C-String knowledge representation, Pattern Recognition 25 (1992) [12] M. Barnsley, Fracatlas Everywhere, Academic Press, New York, [13] A.V. Oppenheim, R.W. Schafer, Digital Signal Processing, Prentice- Hall, Englewood Cliffs, NJ, [14] A.E. Jacquin, Image coding based on a fractal theory of iterated contractive image transformations, IEEE Trans. Image Processing 1 (1992) [15] R. Distasi, M. Nappi. S. Vitulano, Speeding up fractal encoding of images using block indexing technique, in: A. Del Bimbo (Ed.), Proc. of ICIAP 97, 9th Int. Conf. On Image Analysis and Processing, Lecture Notes in Computer Science, Springer, 1997, Vol. 1311, pp [16] D. Saupe, The futility of square isometries in fractal image compression, Proc. of ICIP 96, IEEE Int. Conf. on Image Processing, Sept [17] M. Nappi, G. Polese, G. Tortora, An image indexing technique based on contractive functions, Journal of Computing and Information (JCI) 2 (1996) [18] Y. Fisher, Fractal Compression: Theory and Application to Digital Images, Springer, New York, [19] U. Glavitsch, P. Schauble, M. Wechsler, Metadata for integrating speech documents in a text retrieval system, Sigmod Record, vol. 23, No. 4, Dec [20] M. Nappi, G. Polese, G. Tortora, Fractal based indexing in image databases, in: Proc. of the first IAPR Int. Workshop on Image Databases and Multi Media Search, Amsterdam, The Netherlands, 1996, pp [21] M. Nappi, D. Vitulano, S. Vitulano, L. Moltedo, Color image coding combining linear prediction and iterated function systems, Signal Processing 63 (1997) [22] R. Agrawal, C. Faloutsos, A. Swami, Efficient similarity search in sequence databases, Foundation of Data Organization and Algorithm (FODO) Conference, Evanston, IL, Oct [23] A. Guttman, R-Tree: A dynamic index structure for spatial searching, Proc. of ACM SIGMOD, Boston, MA, June 1984, pp [24] I. Kamel, C. Faloutsos, On packing R-Trees, Proc. of CIKM, Second Int. Conf. On Information Knowledge Management, Nov [25] G. Salton, M.J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, [26] S.K. Chang, Principles of Pictorial Information Systems, Prentice Hall, NJ, [27] S.K. Chang, G. Polese, S. Orefice, M. Tucci, A methodology and interactive environment for iconic language design, International Journal of Human Computer Interaction 41 (1994)