FReBIR : Fuzzy Region-Based Image Retrieval

Transcription

1 FReBIR : Fuzzy Region-Based Image Retrieval Sylvie Philipp-Foliguet ETIS, ENSEA/UCP/CNRS ENSEA, 6 av. du Ponceau, Cergy, France philipp@ensea.fr Julien Gony ETIS, ENSEA/UCP/CNRS ENSEA, 6 av. du Ponceau, Cergy, France gony@ensea.fr Abstract FReBIR is a region-based image retrieval system, in which images are represented as adjacency graphs of fuzzy regions. An algorithm of fuzzy segmentation is first described. An algorithm to match subgraphs of fuzzy regions is then applied in order to retrieve images from partial queries, taking into account the image composition. Results are improved thanks to a relevance feedback which performs region classification by Support Vector Machine. Keywords: indexing, fuzzy segmentation, graph matching, relevance feedback, partial query. 1 Introduction In this paper, we address the problem of retrieving images containing a specific type of object in a general database of photographs. The main problem is that the searched object can have various shapes and scales according to the shooting. Moreover they can be found in various environments. What we perceive when looking at a scene or at a photograph is coarse zones, with their colours, their textures, and their relative positions. In order to model this perception, coarse zones, without precise edges are extracted from the image and these zones are described by colour and texture features. As zones need not to be accurate, they can be modelled by fuzzy sets. Images are then represented by a set of fuzzy regions, with their features and the composition of the image is stored into an attributed relational graph (ARG) of regions, aiming at representing the relative positions of regions. Then the problem of image retrieval from a partial query can be seen as a problem of inexact graph matching of ARGs. The need for retrieving more than one region arises when the semantic object (for example a person) is split into two or more regions which often happens, because of contrasting colours or textures (for example face, hair and clothing), but also when the user is interested in a set of several objects possibly scattered in the image. We have proposed in [17] an algorithm of fuzzy segmentation, of which we give the broad outline in section 2. Then we present a complete system of image retrieval including a relevance feedback. It is based on an algorithm of inexact graph matching (section 4), compatible with the image features (section 3) and with the image composition. Moreover it is compatible with a real-time use. Furthermore, a relevance feedback improves the system by building semantic classes of regions (section 5). 2 Fuzzy segmentation In order to be compatible with the semantic entities, the regions must be colour uniform. They must be limited by high gradient norms and uncertain when two (or more) regions encounter. Thus we propose to build regions which are not any more crisp sets as in the classical segmentation but fuzzy sets. The term fuzzy segmentation can be found in several papers, with various meanings. Fuzzy thresholding aims at defining membership functions to regions, based on a set of thresholds [4]. Methods of fuzzy pixel classification are generally based on the fuzzy c-means algorithm [1], but they do not lead properly to segmentation, since they only classify the pixels

2 into fuzzy classes and do not create contiguous fuzzy regions. The number of classes is often a priori fixed. In the region growing methods, the issue is to find the seeds of the regions, and the function linking region homogeneity, and membership grades. A complex algorithm of region growing limited by edges has been proposed in [16]. In [23], region growing is carried out by fuzzy rules involving fuzzy criteria such as region homogeneity, region size or gradient sharpness. Sanchez et all propose in [22] to model semantic concepts by means of association rules involving visual features. The algorithm we present below performs a cooperative contour / region approach. Uniform colour regions are extracted by a region growing algorithm performed on the gradient norm image. Membership degrees to a region are linked to the distances to region seeds. Let Ω be a finite referential (set of N pixels). A fuzzy region R j is a fuzzy set of Ω defined by a mapping µ j from Ω to [0, 1]. Definition: A fuzzy segmentation of Ω is a set of M fuzzy regions R j whose supports are included in Ω and defined by the two following axioms. If µ j (s) is the membership degree of pixel s to region R j, then: 1. s Ω, j, µ j (s ) [0, 1] 2. j 0 < µ j ( s ) < N. s Ω Membership degrees are included between 0 and 1, they equal 1 for the core s pixels and 0 for the pixels that do not belong to the fuzzy region. The second axiom means that a fuzzy region must neither be empty nor complete (equal to Ω ). This definition is based on Ruspini s definition of a fuzzy partition [21], but without the third axiom that imposes normalisation : for each pixel, membership degrees to all regions should sum up to 1. This axiom is not desirable for segmentation, as explained in [11]. The algorithm first computes a watershed algorithm on the colour gradient norm image [25]. Every local minimum of the gradient norm is a seed of a basin. This leads to a big number of basins. Thus in a second step, basins are recursively merged according to their areas and depths : a basin too small or not deep enough is absorbed by a bigger neighbour. In this case, the difference between the bottom levels of both basins is applied as a penalty on membership degrees of the absorbed one. Each set of merged basins gives rise to a fuzzy region. Membership degrees are calculated from the topographic distance defined in [17]. For an easier computation, considering common images of about a few hundreds of pixels in each dimension and a standard contrast, membership degrees take values between 0 and 255. They are initialized to 255 for pixels belonging to basin seeds (perhaps minus the difference of basin bottom levels). They decrease as pixels are going away from seeds : 1 for each spatial step (in 4-connectivity), and a value proportional to the difference between gradient norms, until they reach 0. Membership degrees to region R (build from gradient norm image g) For each basin B of R for each pixel s seed (B) µ B (s) penalty put s into Q B while Q B is not empty extract s of Q B for each pixel v neighbour of s µ = µ B (s) ( λ g(v) - g(s) + 1 ) if µ > µ B (v) then µ B (v) µ put v into Q B end for end while End for So pixels belonging to areas of a homogeneous colour, and thereby of a small gradient norm have large membership degrees in the corresponding fuzzy region (cf. Figure 1, 4, 6). These degrees slowly decrease according to the spatial distance to the seed and strongly decrease when meeting an edge, area of a high gradient norm. Moreover the impulse noise is bypassed, because a shorter path is found around it. A crisp segmentation can be obtained by affecting every pixel to the region for which it has the largest membership degree. This "defuzzification" is only used to display simultaneously all the fuzzy regions (cf. Figure 7).

3 An image of our general database Figure 1 : Two fuzzy regions obtained from the top image (the lighter, the larger the membership degree). Parameter λ aims at balancing spatial distance with the closest region seed and difference of gradient. It will influence the spreading of regions It has been set to 2 for all the following tests. The area threshold for basin merging induces the level of detail of the result. It is not beforehand set, but will increase or decrease from an initial value, until an expected number of regions, defined by an interval, is reached. In our general database (see Figure 5 and 7), the number of regions was to lie between 10 and Image signature and dissimilarity Each image of the database is indexed by a signature, which is composed of two parts : the first one is the set of features of all fuzzy regions, and the second one is a representation of the topology of the regions within the image. In this section, membership degrees take values between 0 and 1, as usual. For indexing regions, we used colour and texture distributions. The distribution of feature a for fuzzy region R, defined by its membership function µ, is defined by the probability for any real t [9] : P a ( t) = µ (s) µ (s) (1) s R,a(s) = t s R Distributions are thus obtained by adding the membership degrees of the pixels belonging to each class ; a normalisation is then performed. Thus pixels with small membership degrees belonging to transitions or outliers inside a region have little influence on the distribution shape. CIE L*a*b* space is used for colour, and twelve Gabor filters in 3 different scales and 4 orientations are used for texture analysis. Both spaces are quantified using an enhanced version of k-means algorithm [18]. Each image is first quantified in 256 classes for colour and 256 classes for texture. From previous tests made with our CBIR system [6] we know that 25 is a good compromise between efficiency and rapidity for a general database of about thousand images. Thus the 256 colour (resp. texture) classes for each image are quantified in 25 colour (resp. texture) classes. From the many measures of dissimilarity between distributions, we have chosen a simple measure derived from distance L 1. For two regions R and S respectively represented by the c normalised vectors ( R, c = 1,, k) and c ( S, c = 1,,k ), the distance is : k c c d(r,s) = R S 2 c = 1 1 (2) Spatial relationship between regions are characterized by two ways. First their barycentre are computed as in [20] and normalised by the dimensions of the image, in order to take values between 0 and 1. Secondly adjacencies are stored in an adjacency matrix of regions with a value of 1 if both regions have at least one pixel in common, otherwise 0. A more precise solution, not yet implemented, is to store the region intersection. 4 Graph matching 4.1 State-of-the-art and new proposal We are interested in a query constituted by a set of regions, adjacent or not. The retrieval consists in searching for each image of the database the set of regions which best matches this query (cf. Figure 2). R 1 R 2 R 3 S 1 Figure 2 : Example of an inexact graph match with an under-segmented target image The query is a sub-graph of the graph representing the query image. It may be composed of one or several connected S 2

4 components. Search consists in looking for the best matching sub-graph from graphs representing the images of the database. Most of region-based systems [2][13][19][26] do not take into account spatial relationship of regions. Indeed, the region position is the more often described by the barycentre [8] or the outer rectangle [14], in order to constraint and to reduce the retrieval, but explicit relationship between regions, such as adjacency is not considered. In Picasso [8] images are represented by a multiresolution pyramid ; a query region is then matched with a node of any level of the pyramid. SIMPLIcity [26] allows matching one region with several regions. Segmented images are often represented by graphs, but graph matching has been essentially used for describing rigid objects such as logos [12][19]. The matching algorithm presented in this paper is of the inexact type, since node attributes may differ and is not limited to graph isomorphisms, since one node of a graph may match several nodes of the other graph. Two matching subgraphs do not have to be identical in terms of node number, node attributes or edge number but only similar for node attributes (measured by an appropriate similarity measure expounded below) and consistent for the edges. To solve this NP-complete problem in a reasonable time, we use a tree inspired from [5]. The advantage of such a structure is that with appropriate heuristics, all possibilities of matches of region pairs have not to be explored. Search concentrates on branches which are susceptible to lead to the solution. Heuristics concerns the order of examination of the region pairs, and takes into account dissimilarity between regions and topological consistency. For each image of the database (called target image) the distance between all pairs of query/target regions are computed. At level i of the tree (see Figure 3), a node represents a match between query region R i and one region of the target image S j. A node is developed at a lower level only if the dissimilarity between both regions is lower than a given threshold τ. Edges between nodes represent topological consistency between pairs of regions. (R 1,S 1 ) (R 1,S 2 ) (R 1,S 3 ) (R 1,S 4 ) (R 2,S 1 ) (R 2,S 2 ) (R 2,S 3 ) (R 2,S 4 ) (R 3,S 1 ) (R 3,S 3 ) (R 3,S 1 ) (R 3,S 3 ) Figure 3 : Search tree: each node corresponds to a match of two regions (R i : query region, S j : target region), each arrow corresponds to a topological compatibility The consistency between (R i, S j ) and (R i+1, S k ) takes into account : the adjacency of (R i, R i+1 ) and (S j, S k ) which must be of the same type (adjacent or not); the relative positions of (R i, R i+1 ) and (S j, S k ), for example if R i is above R i+1, S j must be above S k. 4.2 Heuristics for tree building A match between two sub-graphs corresponds to a path from the root to a leaf of the last level. For example in Figure 3, {(R 1, S 2 ), (R 2, S 2 ), (R 3, S 1 )} is one of the four possible matches. It corresponds to the example of Fig. 2. The dissimilarity between sub-graphs is the average dissimilarity of the nodes of a path : m d(r i,s j ) m 1 (3) i = 1 where d represents the dissimilarity between region features (see section 3) and m the number of query regions. So it is worthwhile to put at the highest levels of the tree the pairs which do not match, the tree will thus be more rapidly pruned. For this purpose, all distances d ij = d(r i, S j ) are ranked in decreasing order. Pairs (R i, S j ) for which d ij is higher than τ are discarded. The remaining pairs are stored in a ranked list L.

5 For example, after thresholding, the tree of Figure 3 corresponds to list L : L = { (R 1, S 3 ), (R 2, S 3 ), (R 1, S 4 ), (R 3, S 3 ), (R 1, S 1 ), (R 2, S 1 ), (R 2, S 4 ), (R 3, S 1 ), (R 2, S 2 ), (R 1, S 2 )} The query regions are put in the tree according to their order in this list. The query region of the first pair (that is to say the region the least similar to any target region) is put at the first level of the tree (R 1 in the example). There are as many nodes at the first level as region R 1 appears in list L. The first pair of L with a query region different from R 1 is put at the second level of the tree. There are again as many nodes in level two as this region (R 2 in the example) appears in the list, and so on. The properties of this tree are the following : an image with no similarity with the query is immediately discarded ; the query region which matches the least with the target image (but above threshold τ) is examined at first ; as soon as the global dissimilarity between graphs exceeds a given value, the current node is not developed, there are at most as many levels in the tree as query regions, one region of the target image can match several query regions. The last property allows to manage an undersegmentation of the target image with regard to the query image (cf. Figure 2), but not an oversegmentation. Thus, after this first step of tree building, a second step consists in matching a query region with several target regions. The neighbours of every target region are checked in order to retrieve regions likely to match the query region. More precisely, for each match (R i, S j ), all neighbours S k of S j are examined. If d(r i, S j S k ) is lower than d(r i, S j ) then R i is also matched to S k. A match between two ARGs is then represented by a set of pairs of similar query region - target region whose adjacencies and relative positions are the same in both sub-graphs. If the graph match is finally composed of n query/target pairs (R i, S j ), the dissimilarity for n this match is : 1 d(r i,s j ) n i = 1 The dissimilarity measure for each target image is the minimum of the dissimilarities over all possible matches of the target image with the sub-graph constituting the query. 4.3 Results In the following tests, we only used the vertical consistency, which is suitable for general databases, composed of landscapes, persons, animals, etc. Only the vertical position is of importance to retrieve similar images, symmetry over a vertical axis does not change our perception of the image, while symmetry over a horizontal axis changes it a lot. We first used a database of 3000 images composed of two objects of Columbia database [27] which contains 100 different objects. The problem is to retrieve images containing an object, whatever its position in the image. Figure 4 displays a result with a query made of three fuzzy regions composing the cup of the top left image. The 16 first retrieved images contain this cup in various orientations and small variations of scale. Figure 4 : The query is made of three regions (first row) making the cup of the Columbia database. Images ranked by increasing dissimilarity from top to bottom, left to right. The other results were obtained with our general database which includes photographs of varied types, including animals, landscapes, portraits, cars. A result is shown Figure 5, where we are looking for bears near water. The query consist of the two regions of Figure 1, which are not adjacent and with the bear over the water. The

6 first 16 retrieved images include 11 images of bear near water, but retrieved regions may take varied shapes and disposal within the image provided that the bear is over the water. semantic entity. In the last years, a lot of papers presented interactive systems, most of them based on classifiers. We have used a statistical classifier, but working with regions. Actually the system uses a classifier for each query regions ; it is trained from varied examples. We built a user friendly interface where regions of the returned images are displayed with edges of the same colour as the corresponding query region (cf. Figure 7). Hence, if all returned regions (those with coloured edges) correspond to the query regions and respect the spatial disposition, the user annotates the image as relevant and all returned regions are considered as positive examples. On the contrary if no region corresponds to a query region, the image is annotated as irrelevant and all its regions are negative examples for all classes. Figure 5 : Retrieval result from a query made of two regions (the bear and a part of the water displayed Figure 1). 11 correct images out of 16. Figure 6 : The first two images retrieved for the query made of the two fuzzy regions of Figure 1 and for each of them, the regions which best match the two query regions. 5 Relevance feedback If first retrieved results often include images of the searched category, the system is unable to manage the large variability of appearance of animals, persons or non rigid objects. Whatever the features used to represent a region, a simple system which works with a single query is condemned to remain in a local neighbourhood of the query. Only the user is able to conduct the search towards all modalities of the same Figure 7 : Result from a query made of two fuzzy regions covering the elephant : upper region is light grey, lower region is dark grey. On the right, the query image with the contours of the defuzzified segmented image. Edges are coloured in red and green. Below some of the fuzzy regions, first row the two query regions. On the left, the ranked result images. The feedback loop performs a two-class classification per query region, using all negative examples taken from all irrelevant images and the only positive examples of the class of the query region. The classification is updated at each iteration with the new annotated images, which gives new positive and negative examples. For each class, there are much more negative examples than positive ones, which properly models the asymmetry between the class representing the object (or a part) and the rest of the regions.

7 After each step of annotation, the graph matching is performed, by the way of a tree for each image of the database. The dissimilarity computed for each node of the tree now equals the distance between the region and the class of the query region. Any two-class classifier can be used. We tried bayesian [24], k-nearest neighbours, Fisher Disciminant [15] and SVM [3], all of them with kernels of various types [7]. We have compared the 4 classifiers and various kernels with the general database. A result of precision/recall curves, computed on all categories (Figure 8) shows that the best classifier is the SVM with a 2 d( x,y) Gaussian kernel exp with d = χ 2. 2 accurate, these imprecise regions can be used for pattern recognition. Moreover this algorithm is able to segment a whole database without parameter tuning, so it is well adapted to image indexing. An algorithm of inexact matching, adapted to image retrieval from partial query, made of several regions of an image is then proposed. It includes solutions to reduce the combinatorial complexity of graph matching. Inexact matching overcomes the issue of over- or undersegmentation, since one query region can match several target regions and vice-versa. The algorithm takes into account the similarity between regions, as well as their relative positions. We have showed on examples that our system is able to retrieve images containing a type of object (or animal) with a great variability of position (from the front, in profile, etc.) and of scale, whatever the environment. We showed results with a simple set of features (colour and texture distribution and barycentre position), and a simple distance. Spatial relationship between regions are composed of the adjacency and the relative position of region barycentre. At last a relevance feedback loop strengthens the sets of relevant regions, allowing the system to learn from varied examples leading to a better generalisation capability. Figure 8 : Precision/recall curves for 4 classifiers after 30 iterations (average on all categories) Then we have compared a global approach (RETIN) [10] and our partial query method (FReBIR) with the SVM classifiers. Signatures are computed with the same features, but they are computed on the whole image for RETIN (Figure 9). First one can see the improvement induced by the iterations of relevance feedback for FReBIR. Secondly the comparison between both approaches clearly indicates that, to retrieve images containing a specific object or animal (here white bears), a partial query provides better results (which is not true when semantics is defined in the whole image). 6 Conclusion We have proposed an image representation, with fuzzy regions, their features and their spatial relationship. Although region edges are not Figure 9 : Precision/recall curves with 8 iterations of relevance feedback, for white bear retrieval ; in red RETIN, in green FReBIR References [1] J.C. Bezdek, Pattern recognition with fuzzy objective function algorithms, Plenum Press New York, 1981.

8 [2] C. Carson, S. Belongie, H. Greenspan, J. Malik, Blobworld: image segmentation using Expectation-Maximisation and its application to image querying, IEEE Trans. on PAMI, 24 (8), , 2002 [3] E Chang, B Li, G Wu, KS Goh, Statistical learning for effective visual information retrieval, IEEE Int. Conf. on Image Processing, Barcelona, 2003 [4] H.D. Cheng, Y-H. Chen, Fuzzy partition of two-dimensional histogram and its application to thresholding, Pattern Recognition 32, , 1999 [5] L. P. Cordella, P. Foggia, C. Sansone, M. Vento, Subgraph Transformation for the inexact Matching of Attributed Relational Graphs, Computing, 12, 43-52, [6] M Cord, S Philipp-Foliguet, P-H. Gosselin, J Fournier, Interactive exploration to image retrieval, J. of Applied Signal Processing, vol 2005 (14), Special Issue on Advances in Intelligent Vision Systems: Methods and Applications Part II, , 2005 [7] M. Cord, P-H. Gosselin, S Philipp-Foliguet, Stochastic exploration and active learning for image retrieval, Image and Vision Computing, 2006 (in press) [8] A. Del Bimbo, M. Mugnaini, P. Pala, F. Turco, Visual Querying by colour perceptive regions, Pattern Recognition, 31 (9), , 1998 [9] D. Dubois, M.C. Jaulent, A general approach to parameter evaluation in fuzzy digital pictures, Pattern Recognition Letters 6, , 1987 [10] P.H. Gosselin, M. Cord, RETIN AL : an active learning strategy for image category retrieval, IEEE Int. Conf. on Image Processing, Singapore, 2004 [11] R. Krishnapuram, J. M. Keller, A probabilistic approach to clustering, IEEE Trans. on Fuzzy Systems, 1 (2), , 1993 [12] P. Hong, T. S. Huang, Spatial pattern discovering by learning the isomorphic subgraph from multiple attributed relation graphs, Electronic Notes in Theoretical Computer Science, 46, 2001 [13] F. Jing, M. Li, H-J. Zhang, B. Zhang, An efficient and effective region-based image retrieval framework, IEEE Trans. on PAMI, 13 (5), , 2004 [14] W. Y. Ma, B. S. Manjunath, NeTra: a toolbox for navigating large image databases, ACM Multimedia Systems, 7 (3), , 1999 [15] S Mika, G Ratsch, J Weston, B Scholkopf, KR Muller, Fisher discriminant analysis with kernels, Advances in Neural networks for signal processing IX, 41-48, 1999 [16] A. Moghaddamzadeh, N. Bourbakis, A fuzzy region growing approach for segmentation of colour images. Pattern Recognition, 30 (6), , [17] S. Philipp-Foliguet, M. B. Vieira, A. de A. Araújo, Segmentation into fuzzy regions using topographic distance, 14 th SIBGRAPI, , Florianopolis, Brazil, 2001 [18] S. Philipp-Foliguet, G. Logerot, P. Constant, PH. Gosselin, Multimedia indexing and fast retrieval based on a vote system, ICME 06, Toronto, 2006 (to appear) [19] A. Robles-Kelly, E. R. Hancock, Graph matching using adjacency matrix Markov chains, Proc. 3 rd Int. Workshop EMMCVPR, Sophia-Antipolis, France, 2001 [20] A. Rosenfeld, The fuzzy geometry of image subsets. Pattern Recognition Letters 2, , [21] E.H. Ruspini, A new approach to clustering. Information and Control, 15 (1), 22-32, [22] D Sanchez, J Chamorro-Martinez, Learning Imprecise Semantic Concepts from Image Databases, Mathware and Soft Computing, 9 (1), 59-73,2002 [23] A. Steudel, M. Glesner, Fuzzy segmented image coding using orthonormal bases and derivative chain coding. Pattern Recognition 32, , [24] N. Vasconcelos, Bayesian Models for Visual Information Retrieval, PhD thesis, Massachusetts Institute of Technology, 2000 [25] L. Vincent, P. Soille, Watersheds in digital spaces: an efficient algorithm based on immersion simulation, IEEE Trans on PAMI, 13 (6), , [26] J. Z. Wang, J. Li, G. Wiederhold, SIMPLIcity: semantics-sensitive integrated matching for picture libraries, IEEE Trans. on PAMI, 23 (9), 1-17, 2001 [27] h/softlib/coil-100.html