Fuzzy Hamming Distance in a Content-Based Image Retrieval System

Transcription

1 Fuzzy Hamming Distance in a Content-Based Image Retrieval System Mircea Ionescu Department of ECECS, University of Cincinnati, Cincinnati, OH 51-3, USA ionescmm@ececs.uc.edu Anca Ralescu Department of ECECS, University of Cincinnati, Cincinnati, OH 51-3, USA aralescu@ececs.uc.edu Abstract The performance of Content-Based Image Retrieval (CBIR) systems is mainly depending on the image similarity measure it use. The Fuzzy Hamming Distance ( À ) is an extension of Hamming Distance for real-valued vectors. Because the feature space of each image is real-valued the Fuzzy Hamming Distance can be successfully used as image similarity measure. The current study reports on the results of applying À as a similarity measure between the color histograms of two images. The Fuzzy Hamming Distance is suitable for this application because it can take into account not only the number of different colors but also the magnitude of this difference. I. INTRODUCTION Large collections of images are accumulated every day. Finding a relevant set of images close to an example image is a major issue of Content-Based Image Retrieval (CBIR) systems. CBIR are the next evolution step of keyword-based systems in which images are retrieved based on the information of their contents. A survey of the functionality of current CBIR systems can be found in []. A. Current CBIR systems CBIR systems are designed to allow users to query the image database in a natural way, by the image content. To achieve this goal, various features such as sketches, layout or structural description, texture, colors, are extracted from each image. A query might be: Find all images with a pattern similar to this one (the query pattern), and then the system finds a subset of images similar to the query image. Ä Ô measures or weighted combination of ÄÔ measures, are mostly used to measure the similarity between two images. More precisely, for two images with n features each, Ü ½ Ü Ò µ Ý ½ Ý Ò µ the Ä Ô measure is defined as: Ä Ô µ Ü Ý ¾ µ Ô (1) One of the earliest content-based systems is QBIC, Query By Image Content, from IBM [?]. For each image a set of features are extracted. These include: color, texture, sketch and optionally, objects defined inside the image. Each object is determined manually by a contour and is described by its color, shape and texture. The color similarity is assessed computing the distance between the color histograms using [5]: ¾ µ µ Ø µ () where in matrix A, is a similarity between colors i and j. It can be noticed that µ is the Euclidean distance when A is the identity matrix. For shapes, a weighted Euclidean distance between shapes attributes (area, circularity, eccentricity), as described in [], is used. The main limitation of current CBIR systems is that they cannot deal with semantic-level image queries effectively. An image always contains some semantic information (for example, an image containing a seascape ). Such semantic features can only be represented by some primitive features in present CBIR systems. This paper proposes to use a fuzzy similarity measure based on a generalization of the Hamming distance(fhd) introduced in [?], as a basis for content based image retrieval, and to show how this measure can be used to capture implicitly some semantic contents of the image in addition to other image features (here color histograms). B. The Fuzzy Hamming Distance 1) Degree of difference: Given the real values Ü and Ý, the degree of the difference between Ü and Ý, modulated by «¼, denoted by «Ü ݵ, is defined as [?]: «Ü ݵ ½ «Ü ݵ¾ (3) The parameter «¼ modulates the degree of difference in the sense that for the same value of Ü Ý different values of «will result in different values of «Ü ݵ. The (membership) function «defined in (3) has the following properties: 1) ¼ «Ü ݵ ½ with equality Ü Ý; ) «Ü ݵ «Ý ܵ 3) for Ü, «Ü µ ¾ ; ) «Ü ݵ «¼ Ü Ý µ. Figure 1 illustrates «Üµ when Ü ¾ ¾ for various values of «. ) Difference fuzzy set for two vectors: Using the notion of degree of difference defined above, the difference fuzzy set Ü Ýµ for two vectors x and y, is defined as follows [?]: Let Ü and Ý be two Ò dimensional real vectors let Ü, Ý denote their corresponding th component. The degree of difference between Ü and Ý along the component, modulated by the parameter «, is «Ü Ý µ.

2 alpha=1 alpha=. alpha=.1 alpha= Fig. 1. The membership function «Üµ with Ü ¾ and «varying from.1 to (some graphs for intermediate values of «are removed for clarity). [?] The difference fuzzy set corresponding to «Ü Ý µ is «Ü ݵ with membership function «Ü ݵ ½ Ò ¼ ½ given by: «Ü ݵ µ «Ü Ý µ () In words, «Ü ݵ µ is the degree to which the vectors Ü and Ý are different along their th component. The properties of «Ü ݵ are determined from the properties of «Ü Ý µ. 3) The fuzzy Hamming distance as cardinality of a fuzzy set: Since the Hamming distance is the number of components along which two bit vectors are different (which is also in fact, the square of the Euclidean distance between two bit vectors) a fuzzy extension of this concept must necessarily be based on the concept of cardinality of a (discrete) fuzzy set. This concept has been studied by several authors including [?], [?], [?], [?], etc. Here the definition put forward in [?] and further developed in [?] is used. Accordingly, the cardinality of the fuzzy set is the fuzzy set where Ö Ò ½ Ò ¼ Ü Ö µ Ö µ µ ½ ½µ µ (5) In (5) µ denotes the th largest value of ; the values ¼µ ½ and Ò ½µ ¼ are introduced for convenience, and denotes the Ñ Ò operation. By analogy with the definition of the classical Hamming distance, and using the cardinality of a fuzzy set [?], [?], [?], the Fuzzy Hamming Distance ( À ) is defined as follows [?]: Definition 1: Given two real Ò dimensional vectors, Ü and Ý, for which the difference fuzzy set «Ü ݵ, with membership function «Ü ݵ defined in (), the fuzzy Hamming distance between Ü and Ý, denoted by À «Ü ݵ is the fuzzy cardinality of the difference fuzzy set, «Ü ݵ. À Ü Ýµ «µ ¼ Ò ¼ ½ denotes the membership function for À «Ü ݵ corresponding to the parameter «. More precisely, À Ü Ýµ «µ Ö «Ü ݵ µ () for ¾ ¼ Ò where Ò ËÙÔÔÓÖØ «Ü ݵ. In words, () means that for a given value, À Ü Ýµ «µ is the degree to which the vectors Ü and Ý are different on exactly components (with the modulation constant «). Example 1: Considering two vectors Ü and Ý with Ü Ý ¾ ½ µ, their À is the same as that between the vector, ¼ and Ü Ý ¾ ½ µ (by property () of «Ü ݵ). Therefore, their À is the cardinality of the fuzzy set Ü Ý ¼µ ½ ¼ ½ ¾ ¼ ¾½ ½ ¼ ½ obtained according to (5). For «½ this is obtained to be À ¼ ¼ ½ ¼ ¾ ¼ ¼¼¼½ ¼ ¼½ ¼ ¼ ¾½ In words, the meaning of À is as follows: the degree to which Ü and Ý differ along exactly ¼ components (that is, do not differ) is ¼, along exactly one component is ¼, along exactly two components is ¼ ¼¼¼½, along exactly three components is ¼ ¼½, and so on. ) Defuzzification of the Fuzzy Hamming Distance: As it is often the case with many fuzzy concepts, a non-fuzzy (crisp) approximation for them is useful. The non-fuzzy cardinality Ò Ö µ, of a fuzzy set is defined in [?] where it is also shown that this approximation of the fuzzy cardinality is equal to the cardinality (in classical sense) of ¼ the -level set of with ¼. More precisely, Ò Ö µ Ü Üµ ¼ (7) where for a set Ë, Ë denotes the closure of Ë. Since the fuzzy Hamming distance is the (fuzzy) cardinality of the difference fuzzy set «Ü ݵ, its defuzzification leads to a non fuzzy approximation for it. It can be easily verified [?], that with «½ the defuzzification of À by taking its.5 level set is the standard Hamming distance. In other words the defuzzification of À, Ò À is the number of elements of Ü Ýµ with membership greater or equal to.5. In the remainder of the paper Ò À will denote the crisp quantity obtained according to (7) when is the fuzzy set À defined in (). That is, for the real vectors Ü and Ý, Ò À «Ü ݵ Ò Ö «Ü ݵµ () II. ADAPTING THE FUZZY HAMMING DISTANCE In this section a closer look is devoted to the modulator parameter «. In particular, it is shown that «can be tuned, to include a scaling factor which controls the À sensitivity to the extent of variation. Tuning of the parameter «can be done

3 so as to include context (other components of the two vectors) or to capture only local (current component) information. One way to set the value for «is to impose a lower bound on the membership function subject to constraints on the difference between vector components, such as described in (9) and (). 1.. membership function «Ü ݵ µ «Ü Ý µ ½ (9). subject to. Ü Ý Å () for Ü Ý µ a generic component pair, ½ a desired lower bound on the membership value to À (as defined in ()), and Å some positive constant. In particular, Å can be set as Å Å where ¾ ¼ ½ and Å denotes the maximum value in the column domain, which leads to Fig.. Membership function for the distance. 1 membership function 1 from which it follows that ½ «Ü Ý µ¾ ½ ½ «ÐÒ Ü Ý µ ¾ ½ (11) This leads to the the following formula for defining a global value for the parameter «: membership degree.... «ÐÒ ½ ½ (1) Å ¾ ¾ where Å and have the meaning stated above. can be viewed as the percentage from Å that is considered as a change for that column. For example, for ¼ ½ (%), Å ¾ and ¼, if the difference between compared components of the vectors is greater or equal to 5 then the degree of change will be greater than ½ ¼, and therefore it will be counted in defuzzification of À. Since can be different for each column, À sensitivity can be controlled differently for each column (feature). Figures, 3, and 5 show the membership function of distance from the point (5,5) and the point Ü Ýµ ¾ ¼ ½¼ ¼ ½¼, when Ü =.1 and Ý =.3, for all possibles values of the actual distance:, 1 or. Figure 5 shows the distance distribution over the D space Ü Ýµ. It can be seen from these figures that À measures how many components are changed with a certain degree. In the middle, for points close to the center, µ, the distance is zero with high degree, while for points close to the margins (determined by the values for Ü and Ý ) the degree for zero decreases and the degree of 1 is increases (passing ½ level). Beyond this margin, along both Ü and Ý, the degree for the distance to be equal to is increasing passing ½ and the degrees for ¼ and ½ are again low. The distance distribution is represented in D space by rectangles and in ÒD by ÒCubes. Y X Fig. 3. Membership function for the distance 1. Using different for each feature means, in effect, weighing each feature thereby allowing a larger or smaller variation between feature values to be considered as a change. III. FHD FOR CBIR: SYSTEM DESIGN The proposed CBIR system consists of the three modules as shown in Figure??. As it can be seen, the approach consists of three simple steps: 1) The preprocessing module extracts the information of interest (in this study the color histograms) from each of the images in the data base and the query image. The output of this module is a collection of color histograms. ) The similarity assessment module takes as input the information from the preprocessing module and computes the similarity (actually the FHD), between the query image and each image in the database. The output of this module is a collection of fuzzy sets (FHD). 3) The ranking module returns the outcome of similarity assessment ranked in decreasing order. A detailed description of these modules follows.

4 Membership degree membership function Y X Fig.. Membership function for the distance. B. The Similarity Assessment Module The collection of histograms corresponding to the image data base, and the color histogram of the query image, are the input to this module. Each image is represented as a matrix of histograms, more precisely, image is represented as À еµ Ð ½ Ñ. The query image Õ is represented as Õ À Õ Ðµµ Ð ½ Ñ. The similarity function, Ë ÑÁÑ, maps two images into a matrix whose entries are the FHD between corresponding regions of the argument images. More precisely, for two images, and Õ, Ë ÑÁÑ Õµ À À е À Õ Ðµµµ Ð ½ Ñ (13) where the À denotes the Fuzzy Hamming Distance parameterized by (introduced in equation (1)), the percentage of difference in two colors (relative to the possible range) needed in order to consider these colors different. For the current experiments, the value ¼ ½, that is ½¼± of the maximum range, is used. It is important to note here, that segmentation of the image into regions and evaluation of the similarity between two images as a function of the similarities of their corresponding regions, assures that position information is also considered, along with color information. This way two images which have the same color histogram but different semantic content (e.g. an image and the one obtained by rotation) will not necessarily be identical. Fig. 5. Distance distribution over Ü Ýµ ¾ ¼ ½¼ ¼ ½¼ A. The Preprocessing Module The images of interest (data based and query image) are preprocessed according to the following steps: 1) Segmentation: Each image is segmented in a Ñ Ñ matrix of regions. For the experiments reported here two values, Ñ and Ñ are used; ) Compute color histograms: The color histogram for each region is computed. Because the RGB color space can have 1M colors, quantization to 9 colors, giving a color histogram of 9 bins, is used. The number of pixels from each region that have the color corresponding to each of these bins is computed. The output of the preprocessing module is the collection of Ñ Ñ color histograms. C. The Ranking Module This module computes a score from the output of the similarity module and ranks images in order of the values of this score. To achieve this, in the current implementation ranking is done in two steps: 1) Defuzzification: The fuzzy sets returned by the similarity module are defuzzified. For comparison purpose three defuzzification methods were used: the center of gravity(cog), extracting the crisp version of the Fuzzy Hamming Distance, nfhd (defined in section I-B), and the area under the curve (AUC). ) Aggregation and Ranking: The results of the defuzzification step are aggregated into a final score using a weighted aggregation. Scores are ranked in nondecreasing order (images closest to the query have smallest number of different different colors). The first Ò images, where Ò is a user-defined value, are returned as relevant to the query image. IV. RESULTS The approach described in the preceding sections is applied to 5 images in JPEG format from the Washington database [?] (which contains images, the remaining 15 are in GIF). Images are represented as Ñ Ñ regions for both Ñ. For comparison purpose, a non-fuzzy similarity measure, based on the Euclidean distance, is also used.

5 For the similarity method using FHD, each of the three defuzzification methods are applied for each query. In the ranking module two weighting schemes are used for aggregation: (1) equal weights, and () regions in the top portion of an image were assigned higher weights. The latter is important to enforce additional positional requirements (such as make sure the retrieved image has a sky like the one in the query image ) for the retrieval procedure. Figure shows the top five results returned for a query. The first three columns represent, in order, the results obtained using FHD, COG, nfhd, AUC, for similarity assessment, with equal aggregation weights. The last column represents the top five results returned by the Euclidean distance. The query image appears in the first row. To better analyze the results returned by the four methods, it is helpful to notice that the (semantic) category for the query image could be described as seascape (the sea in the bottom part of the image, an island across the middle of the image, and the sky in the top part of the image). Above each image is its score. Because of the different defuzzification methods these scores appear in different units. Only the score for the second column (nfhd) has a direct meaning, namely, the number of colors by which the image and the query image differ. As it can be seen from figure each of the four methods retrieves the query image as the best candidate and indeed, their results also coincide for the second and third image returned as well. From the fourth image on, results differ somewhat. Among the FHD based methods, COG gives the best result (this is quite reasonable since the fuzzy set for which it is applied is convex), followed by nfhd and by AUC. Relative difference between scores decreases when moving down each column. The last image in second column, which also appears as the fourth image in the third column is interesting. This image has a building with glass walls behind what appears to be flowering trees. The walls appear blue due to the reflection of the sky (unseen in the image), therefore, from color point of view and, in fact, from the spatial distribution of the color, this image is similar to the query image. However, this image is in a totally different semantic category than the query image. It appears that the Euclidean distance does essentially as well as the COG as it preserves to almost the same extent the semantic category of the query image. A difference emerges in row four: the image returned as the third candidate (which is the same as the fourth candidate returned by COG) contains some artifacts in the lower part of the image which puts it in a slightly different semantic category (e.g. seascape with a small bridge or pier ). Therefore, one can conclude that COG preserves the best the image content both as color and semantic category. Indeed this can be even better seen when the next five candidates for the same query are considered as shown in figure 7. All the images retrieved by the COG method preserve the semantic category. This method is followed in order by the Euclidean Distance, nfhd and AUC. Fig Top five query result-set using equal weights for each image region. Images have been divided into regions. First three columns correspond to COG, nfhd, AUC respectively; the last column uses Euclidean distance. Fig Next five query result-set using equal weight for each image region. Images have been divided into regions. First three columns correspond to COG, nfhd, AUC respectively; the last column uses Euclidean distance. V. CONCLUSIONS AND FUTURE WORK The relevance of the result-set of an CBIR system is hard to assess. It is based on what is relevant for a user, given the query image in a particular context. That is, relevance is both subjective and for the same user, it may be context dependent. This study presented initial results on a new approach to measure similarity between images using the notion of Fuzzy Hamming Distance and its use to content based image retrieval. Even with the remark at the beginning of this section, results seem promising: The main advantage of the FHD is that the extent to which two different images are considered indeed different can be tuned to become more context dependent and to capture (implicit) semantic image information.

6 Several mechanisms can be used to rank and aggregate the similarity measures in addition to the simple point-valued mechanisms illustrated here. These would include mechanisms that rank and aggregate directly fuzzy numbers without a (too early) defuzzification. Developing such mechanisms, experiments with other image features (HSV histograms, textures), and further experiments with the proposed methods belong to the future work in this direction. ACKNOWLEDGMENT Mircea Ionescu s work for this study was supported by a Graduate Fellowship from the Ohio Board of Regents. Anca Ralescu s work for this study was partially supported by a JSPS Senior Research Fellowship at the Brain Science Institute, RIKEN, Japan, and grant N137 from the Department of the Navy. REFERENCES [1] A. G. Barto, R. S. Sutton, and C. W. Anderson, Neuronlike adaptive elements that can solve difficult learning control problems, IEEE Trans. Systems, Man, and Cybernetics, vol. SMC-13, pp. 3, Sept./Oct [] A. N. Michel and R. K. Miller, Qualitative Analysis of Large Scale Dynamical Systems, New York: Academic Press, [3] P. J. Werbos, Neural networks & the human mind: New mathematics fits humanistic insight, Proceedings of the 199 IEEE Trans. Systems, Man, and Cybernetics, where, 199, vol. 1, pp. 7 3.