NOVEL APPROACH FOR COMPARING SIMILARITY VECTORS IN IMAGE RETRIEVAL

Transcription

1 NOVEL APPROACH FOR COMPARING SIMILARITY VECTORS IN IMAGE RETRIEVAL Abstract In this paper we will present new results in image database retrieval, which is a developing field with growing interest. In case of the images in a database many features can be considered like color, shape, texture. These can be considered to find similar images in the database. The investigated procedure is to compose similarity vectors from the similarity values according to these features. Then some measurement is applied to calculate the norm of the similarity vectors (what is between the query image and the other images in the database). Such a norm can be the weighted Euclidean one (applied in Oracle9i). Our new approach is based on neighborhood sequences as a new method to calculate the norm, and we will show that for some purposes it allows more flexibility for the user in composing the queries than the classic norm-computing methods. 1. Introduction Image database retrieval is an extensively investigated area [8,14]. The main task in VIR (Visual Information Retrieval) is to find similar images in databases for an input (query) image. Such applications have great importance in architecture, digital catalog, remote sensing, medicine etc. The general retrieval paradigm has three steps. The first one is the feature selection (which image properties should be considered in finding matches), second is similarity measure selection (to determine the difference of the selected features and derive a similarity value), and the last one is the ranking (based on similarity values obtained in the previous step). The features for the vectors can be e.g. color, texture or position of segments, see [7,13]. After fixing p observed features, a p-dimensional feature vector can be assigned to both the images in the database and also to the query image. After extracting the feature vector for an image, a real value can be assigned to its components to show their similarity to the query. In such a way a p- (or lower) dimensional real valued similarity vector can be assigned to every image. Naturally, the query image is the most similar to itself with similarity values 0, and thus with the similarity feature vector the origin O.

2 There exist several similarity measures to calculate a decisive real value for ranking from the similarity vectors. They are usually some classic real vector norms. One can use e.g. the wellknown L p norms, or more generally some other weighted norms like the Oracle9i [9] does. Consequently, images in the database having feature vectors close to the query one (or equivalently with small norm) are considered to match it. In this paper we propose a new approach to derive these decisive similarity values (that is to calculate the distance from O). Our approach is based on neighborhood sequences (NS) and was motivated by the idea that in this way we can explicitly prescribe which features should be compared in individual steps, because it cannot be expressed in the Oracle s system. Besides considering the applicability of classic neighborhood sequences, we investigate a more general family having a natural load regarding to the logic of image retrieval. We also introduce some practical methodology to let the theoretical results on neighborhood sequences be applied efficiently, and explain illustratively how one can take advantage of the flexibility of our approach.. 2. Similarities in Oracle The Oracle has a special cartridge for storing and managing multimedia content called intermedia. For retrieving image data a special VIR tool has been developed [10]. It is a standard feature of Oracle [9]. It uses color histograms (c), texture descriptors (t), shape representations (s) and their spatial placement (l) for extracting feature vectors. This one by itself is not a meaningful search parameter, but in conjunction with one of the three visual attributes, it provides a search where the visual attribute and its location are both important. So we will omit (l) as a separate feature. The Oracle refers to feature vectors as signatures. To compare two signatures, W c, W s, W t weight values should be assigned to the above features, respectively, indicating the relative importance of the corresponding feature. A feature weighted by 0 is not considered at all, while weight 1 gives highest importance. At the end of the comparison, the 3D similarity vector (c,s,t) contains real values between 0 and 100. The distance from the query O is calculated as c*w c + s*w s + t*w t. To compare Oracle s distance derivation with our technique, in the following we assign the x, y and z coordinates to the color, shape and texture similarity values, respectively. 3. Neighborhood Sequences (NS) In this section we recall some concepts regarding to neighborhood sequences. It is only a narrow slice of the whole theory, for a comprehensive study see [1,2,3,4,5,7,15]. Besides using the known

3 families of neighborhood sequences we also introduce some new ones which can be adopted in image retrieval. We also give a descriptive grammar to define neighborhood sequences to allow them to be easily handled in applications Basic concepts and notation Let R, Z, N denote the sets of real, integer, and positive integer numbers, and write R +, Z + for the subsets containing their non-negative elements, respectively. The general theory of neighborhood sequences is established for arbitrary dimensions, but as we will consider 3D feature vectors later on, we restrict our attention to the 3D domain. By following [4], a neighborhood is a pair (P,w), where P is a finite subset of Z 3, w: P R + is a weight function, and w(q) is the weight of q P. Let Λ be a finite set of neighborhoods. A 3Dneighborhood sequence over Λ is defined as an infinite sequence N=(N i ) i N, where N i Λ for all i N. Let S 3 denote the set of all 3D-neighborhood sequences. If for some j N, N i = N i+j for all i N then N is called periodic with period j. Then we write N=(N 1 N 2 N j ) for the sequence, and P 3 for the set of periodic sequences. The set P 3 (also in higher dimensions) was deeply investigated in [15]. If we can obtain a periodic sequence from an N S 3 by removing its first finitely many elements, then N is called an ultimately periodic sequence. In this case we use the notation N=N 1 N 2 N k (N k+1 N k+2 N l ), that is after removing the first k elements, we have a sequence with period l-k. The set of ultimately periodic sequences will be denoted by UP 3. The importance of UP sequences in applications lays in the fact (see [4]) that we can give them by finite data. Note that P 3 is a strict subset of UP 3. We can measure distance by the help of neighborhood sequences in a natural way (see [4,6,8,19]). Let q,r Z 3 and N=(N i ) i N S 3, with N i =(P i, w i ). The point sequence s=[q=q 0,q 1,,q m =r], where q i - q i-1 P i, is called an N-path between q and r. The length of s is defined as w 0 (q 1 -q 0 )+ w 1 (q 2 - q 1 )+ +w m-1 (q m -q m-1 ). The N-distance w(q,r;n) of q and r is defined as the length of a shortest N- path between them, if such a path exists (for a short notation we write w(n)). We put w(q,q;n)=0 for any q Z 3, and set w(q,r;n)= if there is no path between q and r NS families for image retrieval For image retrieval purposes we will consider two subsets of UP 3, and also their "mixture". The first family contains sequences consisting of the neighborhoods N i =(P i, w i ) with i =1,2,3, and P 1 ={O, (0,0,±1), (0,±1,0), (±1,0,0)},

4 P 2 =P 1 {(0,±1,±1), (±1,0,±1), (±1,±1,0)}, and P 3 =P 2 {(±1, ±1,±1)}, where w i can be any weight function (i =1,2,3). Note that N 1, N 2, and N 3 are based on the well-known 6-, 18-, and 26-neighborhood [13], respectively. For theoretical investigations on such sequences with unit weights see [1,2,3,5]. We denote this classic subset of neighborhood sequences by CNS 3. Dedicated to similarity detection we also focus on that subset of UP 3 which contains sequences consisting of the neighborhoods N i =(P i, w i ) with i {x,y,z,xy,yz,xz,xyz}and P x ={(±1,0,0)}, P y ={(0,±1,0)}, P z ={(0,0,±1)}, P xy ={(±1,±1,0)}, P yz ={(0,±1,±1)}, P xz ={(±1,0,±1)}, P xyz ={(±1,±1,±1)}, where w i can be any weight function (i {x,y,z,xy,yz,xz,xyz}). Each of these neighborhoods spans a 1D, 2D, or 3D subspace of Z 3, respectively. Thus the set of sequences generated by them is denoted by SNS 3. With SNS 3 sequences we can explicitly prescribe which coordinates are allowed to change at a step, while CNS 3 sequences let us to give the number of the changeable coordinates only. Note that neither CNS 3 SNS 3, nor SNS 3 CNS 3, and N xyz =N 3. The third family we will consider, is the subset of the so called mixed neighborhood sequences which consist of the neighborhoods N 1, N 2, N 3, N x, N y, N z, N xy, N yz, N xz. This way we can merge the considerations taken for CNS 3 and SNS 3. The set of such mixed sequences is denoted by MNS 3, and note that MNS 3 is strictly larger than CNS 3 SNS 3. There are different natural possibilities to handle the empty step O and its weight (cost): O has non-zero weight. This is our main approach, with the intuitive meaning that we may hold our position for some cost at every step. We feel this model to be closest to the general way how natural queries are initiated by users. O has zero weight. It means that we can decide to ignore any such elements of neighborhood sequences that cannot help us to reach a given point. This case was investigated deeply in Section 8.2 of [4], and by Remark 4 of [4] we know that the sequence can be permuted freely then. Thus we cannot take advantage of the order of the elements in image retrieval. O is not included at all in N x, N y, N z, N xy, N yz, N xz, N xyz. In other words, we do not let empty steps neither for some cost nor for free, and force to move to some other position at every step. This approach has some properties which we found to be drawbacks for image retrieval NS in applications There are some simple points which are technically important to let the theory of neighborhood sequences be effectively realizable in actual applications. The first step is to convert the (possible

5 real valued) input data onto the cubic grid Z 3. This step can be done e.g. by using the common ways of digitization [12]. In many cases the input real valued data is already digital with some decimal digit accuracy (like the similarity vectors in Oracle9i). Then we can consider this accuracy for scaling to gain the most precise digitization grid. Obviously, we can reduce the size of the data domain with a coarser digitization grid. As the data domain can be quite large, many steps may be needed to take between its values and thus many elements of the neighborhood sequences should be given. Intuitively, it is not a problem to simply list the elements of neighborhood sequences. However, technically it is challenging to give even some thousands of elements in applications. Thus we propose a simple G=<V N,V T,S,H> grammar [11] to prescribe the elements of ultimately periodic sequences in a compact form, where V T ={N 1, N 2, N 3, N x, N y, N z, N xy, N yz, N xz }, H={S {A k }; S SS; A S; A B(q); A B; A a} where k N { }, W R +, a V T. W means the weight (cost) of the given step. If we do not define it in another way, it is considered as 1. From the construction we can see that UP 3 neighborhood sequences are worth considering as they can be given by finite data. For example, let r=(5,3,7), and N=N xz N xz N xz N xz (N y N y N y N 1 N 1 ). Then w(o,r;n)=14, and by the above grammar we can write N={N 4 xz }{{N 3 y }{N 2 1 } }. If N=N x N z N x N z N x N z, and r=(5,1,4), then w(o,r;n)=, and N={N 1 x }{N 1 z } 3. Let N={N 1 x }{N 1 z } 30 {N 1 y }, and r=(5,1,4), then without allowing empty steps w(o,r;n)=31, in case we allow empty steps w(o,r;n)=10. Let r=(1,1,1) and N={N x (5.0) 1 }{N 1 z }{N 1 y }, then w(o,r;n)=7 (sum of the costs). 4. Experimental results To test our approach under real conditions we took a subset of the Hemera PhotoObjects image database with approximately 1300 images. As we focus only on measuring distance between similarity vectors, we used the Oracle9i engine to generate them. Now we present some examples for retrieval, where some intuitively sound queries were formulated by the grammar of neighborhood sequences. To show the flexibility of our approach we compare these results with similar queries supported by the Oracle. Query: Select such images that are quite close in color and texture to the input image. A possible NS answer is N={{N xz } a }{N b y } with e.g. a=3, b=40. In this case we allow 3 steps in the x and z directions first, then the y one can be changed for 40 steps. The periodicity of N guarantees that we do not exclude vectors with greater values than 3 in either their x or z coordinates, however, they will be reached only after applying more periods. See Figure 1 for the first nine matches

6 ordered by their distance from O. A possible Oracle weighting to this question can be W c =W t =7/15, W s =1/15. See Figure 2 for the same query as above. Query Fig. 1. Query results and distance values for {N xz 3 }{N y 40 }. Query Fig. 2. Query results and distance values in Oracle Query Fig. 3. Query results and distance values for {N y 40 }{N x 4 }{N z 4 }. Query Fig. 4. Query results and distance values for {N y 40 }{N z 4 }{N x 4 }.

7 Our next example shows that we have the opportunity to permute the elements of the sequence. Intuitively, in this way we can involve the time factor into our queries, like: Query: Select such images that are quite close first in color then in texture to the input image. NS answer is N={N 40 y }{N 4 x }{N 4 z }, see Figure 3. Query: Select such images that are quite close first in texture then in color to the input image. NS answer is N={N 40 y }{N 4 z }{N 4 x }, see Figure 4. In Oracle we cannot consider the time factor. With allowing costless empty steps, the result for the query sequence {N 40 y }{N 4 z }{N 4 x } is shown in Figure 3: Query Fig. 5. Query result and distance values for {N y 40 }{N x 4 }{N z 4 } allowing empty costless steps. Without further illustration we list some more intuitive queries together with proposals for NS answers (which cannot be described by Oracle weights). Select such images that are quite close in color or in shape to the input image NS answer is N={N 3 2 }{N 1 z }. Select images that are close first in color and texture then in shape within some distance. NS answer is N={N 5 xz }{N 40 y }{N xz }. 5. Conclusions In this paper we focuser only the possible applicability of special 3D neighborhood sequences for image retrieval purposes. We note that our method can be easy extended to be able to manage higher dimensions (e.g. some new features of the images) and the investigated families of neighborhood sequences are expected to be well applicable also in any other field, where similarity vectors are considered. The digitization of the input data can be also improved by by using nonuniform sampling (to separate better the feature space), or an elongated grid (to normalize the features).

8 6. REFERENCES [1] P.E. Danielsson, 3D octagonal metrics, Eighth Scandinavian Conf. Image Process., pp , [2] P.P. Das, P.P. Chakrabarti, and B.N. Chatterji, Generalised distances in digital geometry Inform. Sci. 42, pp , [3] A. Fazekas, A. Hajdu, L. Hajdu, Lattice ofgeneralized neighbourhood sequences in nd and D Publ. Math. Debrecen 60, pp , [4] A. Hajdu, L. Hajdu, R. Tijdeman, General neighborhood sequences in Z n Discrete Appl. Math., submitted. [5] A. Hajdu, B. Nagy, Z. Zörgő, Indexing and segmenting colour images using neighbourhood sequences, IEEE ICIP 2003, Barcelona, Spain, pp. I/ [6] A. Hajdu, T. Tóth, K. Veréb, Neighborhood sequences in image database retrieval IEEE ICIP 2005, Genova, Italia, submitted [7] C. Kiselman, Regularity of distance transformations in image analysis, Computer Vision and Image Understanding 64, pp , [8] Lew, M.S., Principles of Visual Information Retrieval (ed.), Springer, [9] Oracle intermedia User's Guide and Reference, Release Part Number A , [10] Oracle Visual Information Retrieval User's Guide and Reference, Release 8.1.7, Part No. A , [11] Gy. E. Révész, Introduction To Formal Languages McGraw-Hill Book, Singapore, [12] A. Rosenfeld, and R.A. Melter, Digital geometry The Mathematical Intelligencer 11, pp , [13] A. Rosenfeld, and J.L. Pfaltz, Distance functions on digital pictures Pattern Recognition 1, pp , [14] Santini, S., Exploratory Image Databases Academic Press, [15] M. Yamashita, and T. Ibaraki, Distances defined by neighbourhood sequences Pattern Recognition 19, pp , 1986.