Image Database Modeling

Size: px

Start display at page:

Download "Image Database Modeling"

Caren Hodge
5 years ago
Views:

1 Image Database Modeling Aibing Rao, Rohini Srihari Center of Excellence for Document Analysis and Recognition, State University of New York At Buffalo, Amherst, NY14228 arao, Abstract The performance of an indexing technique, namely, a system of feature extraction and similarity measure, is strongly related to the test bed in content-based image retrieval. A theoretical framework is proposed for image database modeling. The goal is to investigate the relationship between a given indexing technique and a given image database. As an analogy to the concept of perplexity of a text corpus, the complexity of an image database is introduced. This is a quantitative measure of the quality of an image database for the task of content access to the database. The complexity of an image database is an internal property of the database and can be used to distinguish the difficulty of different databases on which content access is to be performed. On the other hand, the measure can be used to compare the effectiveness of different indexing techniques with which content access is to be performed. Keywords: Content-Based Image Retrieval, Image Database Modeling, Complexity of Image Databases 1. Introduction There are a tremendous number of techniques for image retrieval nowadays[9, 11, 3, 10, 1, 12]. Lower-level feature extraction of images is extensively explored from the perspective of color, spatial layout, texture, shape, etc. The current state of image retrieval is chaotic in terms of evaluation: different researchers design different algorithms and then test the performance on their own databases. There is no test bed, and there is no theory about how to build such a benchmark. One of the main difficulties is to answer the fundamental question: What is an image database? What is an image database? People tend to call a collection of images an image database. Is it a database? According to Elmasri and Navathe [5], a database is a collection of related data. What is the relationship reflected in a collection of images? Without any indexing and classification being performed, there is no relationship among the images in the collection; hence, it is not a database. In order to make a collection of images look more like a database, the relationships of the images in the collection have to be explored. There are tremendous number of choices to define such relationships. For example, all of the existing content-based image retrieval techniques belong to such choices. In this paper, a much more naive way of exploring the relationship is adopted. In other words, the collection of images is considered as a whole and some properties of the collection ( the database ) are proposed in favor of image retrieval. The properties of the image database is assumed to be internal to the database and can be regarded as a characteristic of the database. This characteristic is a measure of the difficulty of the task of image retrieval to be performed on the database. Here, the difficulty of image retrieval is not with respect to a particular query example but with respect to general queries on average. In other words, the difficulty of image retrieval on the database is the average difficulty of answering all possible reasonable queries on the database. Therefore, it is independent of the selection of queries and is an internal property of the database. Srihari et al. [13] gave an experimental example of the different difficulty of applying the traditional color histogram matching on a database of images of scenery against a database of images with human activities. In the experiment, the technique is the same, but the performance greatly differs with each database. Obviously, some internal difference between those two databases gave rise to the different difficulties of the retrieval task. What are the constituents of a quantitative measure of the difficulty performing the retrieval task on a given database? The following aspects are the major considerations: Homogeneity Images from different clusters of the database are very similar. For example, a rose image database is more homogeneous than a flower image database. Heterogeneity Number of clusters of similar images in the database. Content Variety A description of different levels of the

2 complexity of the semantics of images in the database. For example, a human activity image database is more complex than a purely scenery image database. Cardinality Number of images in the database. A quantitative measure of the difficulty should encompass all these aspects. Such a measure is not a simple summation of these constituents because they are not orthogonal. For example, a database of 1000 copies of an image should bring the same difficulty to the retrieval task as should the database only consisting of the single image. Moreover, the properties of homogeneity and heterogeneity bring the inevitable difficulty in defining such a measure because they are paradoxical. The idea of the quest of a quantitative measure, called the content complexity of an image database, is partially motivated from the concept of perplexity of a test corpus in language modeling[6, 8]. The perplexity of a statistical model M, such as bigram, trigram, etc., on the actual probability distribution P of a text corpus is defined as 2 H(P;M) where H(P; M ) is the cross-entropy of M on P. This concept has two important implications: For a given text corpus, it measures how accurate the model M is in approximating the actual probability distribution P ; For a given language model M, it measures an internal property of different text corpus. In speech recognition, for a given model, the perplexity defined by the model is a crude measure of the average size of the set of words from which a recognizer would choose the next spoken word. Hence it is a measure of the degree of the difficulty of performing speech recognition with the text corpus using the given model. 2. Image Database Modeling: Properties of the Complexity of Image Databases The purpose of image database modeling is to provide a quantitative measure of the difficulties of performing image retrieval task on image databases. Let I be the universal image database which is the set of all images being considered. A model ' of I is a pair consisting of a feature extraction algorithm and an associated similarity measure: ' = (f; d) where f : I?! R n maps an image to a feature vector and d : R n R n?! R + is a distance measure of a pair of feature vectors, here R n is the vector space of dimension n consisting of all n-tuples of reals and R + is the set of all non-negative reals. Let F be the set of all models of I. A measure of the complexity of image databases (CID) is a mapping cid : 2 I F?! R + such that it satisfies (1). For 8' 2 F; 8D 1 ; D I, cid(d 1 [ D 2 ; ') cid(d 1 ; ') + cid(d 2 ; ') if D 1 \ D 2 = ;; (2). cid(d; ') cid(d; ) iff ' is more suitable in describing the semantic content of image database D than is. Condition (1) states that the complexity after merging two disjoint databases together is lower-bounded by the sum of each individual complexities. This assumption is reasonable because the union of two disjoint databases brings new spatial relationship between the components, in addition to the original complexity of each database. Moreover, this assumption implies that a larger image database should be more complex, corresponding to the fact that it is usually more difficult to find the right information from a larger pool of images, no matter what kind of model is used. In fact, suppose D 1 D 2, then D 1 = D 2 [ (D 1? D 2 ) so that cid(d 1 ; ') = cid(d 2 [ (D 1? D 2 ); ') cid(d 2 ; ') + cid(d 1? D 2 ; ') cid(d 2 ; ') Condition (2) states that an less effective algorithm imposes higher complexity on the same database than a more effective one does. Notice that these conditions are ideal and may not be achievable in reality. Since the mapping has two parameters, similar to the properties of the concept of perplexity as stated above, it implies two important aspects: When the model is pre-selected, the mapping is a function of database D and can be regarded as a property of the database. Therefore it can be used to compare two image databases in term of using the chosen model for image retrieval. Moreover, an optimal model can be defined by optimizing the complexity value as follows, cid(d) = min cid(d; ') '2F which suggests that the optimal model is the perfect one used for image retrieval on D. This optimized quantity is independent of the model and is a property of the database. It measures the difficulty of performing image retrieval on D. In reality, due to the limitation of the technology of content access to image data, it is impossible to solve this optimization problem. An approximation is made possible via cid(d) = min cid(d; ' k) 1kK

3 by carefully selecting a list of models f' 1 ; ' 2 ; :::; ' K g as a benchmark for the purpose of measuring the difficulty. On the other hand, for a given image database D, the mapping is a function of model ' and can used as guide to find the optimal model for the given database D. Moreover, it can be used as a measure for comparing the performance of different models. For example, in order to evaluate the performance of a group of algorithms designed for color histograms, assuming there is a benchmark consisting of K color image databases D 1 ; D 2 ; :::; D K, the performance of a color histogram model ' is measured by cid(') = 1 K KX k=1 cid(d k ; ') Next some ideas for implementing the quantitative measure are presented. 3. Query-Based and Clustering-Based Approaches If an image database is regarded as a black box where only query by example is allowed and the output of a list of similar images is guaranteed, then querying the database is extremely important for the outside world to comprehend the content inside the black box. This gives rise to the possibility of computing the content complexity of the black box based on queries. This is called a query-based approach. Second, if the image database is regarded as a database on which clustering can be performed, then the content complexity of the database can be computed by analyzing each of the clusters as well as the relationship among them. This is called a clustering-based approach. Third, if an image is regarded as a list of labels of pre-defined keyword subimages, then the content complexity of the database can be computed in a similar manner as in text domain[6]. This is called a template-based approach. 3.1 Query-Based Approach Given image database D and a model '. The feature extraction algorithm of the model maps each image in D to a feature vector. The structure of D can be understood by carefully designing a set of queries. Suppose Q = fq 1 ; q 2 ; :::; q N g be such a set of queries. For each query q i, using model ' to calculate the feature vector of q i and then calculating the similarity distance between this vector and the feature vector of each image in D. Sort D with respect to the distance values to get a permutation of D which represents the ranking list based on the similarity to the query. The evaluation value of the permutation is Pcalculated by the average precision which is defined as R 1 R i N i where R is the total number of similar images to q i in D and N i is the number of images retrieved at the point while the i th similar image is retrieved, or by the performance area[13] of the precision-recall graph, or by any of the measures discussed in Baeza-Yates and Ribiero-Neto[2]. Denote the value as p i, it represents how good the model performs for the query. In addition, a mapping u : I?! R + is needed to define the difficulty of the question itself. It assigns each image a value representing the content variety of the image. Now the relationship between the content complexity of the database D and the query q i is: if the question is difficult which means u(q i ) is high and the answer is good, then the complexity of the database is relatively lower because hard questions yet gives easy answers. Mathematically, a possible relation is 1? performance difficulty. By a technical modification to avoid nil denominator, take average over all of the queries to define cid(d; ') = 1? 1 N NX p i 1 + u(q i ) This formula somehow catches the properties presented in section 2, at least it is obvious true for condition (2) in the definition of section 2. Currently, the mapping u is defined to map an image to its entropy. The entropy of an image somehow represents the content variety of the image because it is the uncertainty of interpreting the image. In general, the mapping should also take the database on which the query is applied into account because the same image query could be an easy question on one database while it could be a hard question on another database. The great disadvantage of query-based approach is the difficulty of designing the set of queries. To compensate this disadvantage, the database itself is usually used as the required set of queries for the computation. In this case, the mapping u is nil because the measure now is queryindependent. However, this case requires the relevance judgements of the database. 3.2 Clustering-Based Approach Assume the image database D is clustered into M clusters under the model ', namely, D = M[ where D i is a cluster of similar images of D with respect to '. Obviously, the content complexity of D is related to each of D i as well as the relationship of the clusters. Therefore, cid(d; ') = MX D i cid(d i ; ') + ('; D 1 ; D 2 ; :::; D M ) (1)

4 where the second summand ('; D 1 ; D 2 ; :::; D M ) is a non-negative function describing the relationship of the clusters. Now the problem becomes two sub-problems: (1). How to define the content complexity of a similarity cluster; (2). How to define the function ('; D 1 ; D 2 ; :::; D M ), i.e. how to represent the relationship of M clusters. The complexity of one cluster For a given cluster D i, with the simplest model, it can be represented by the centroid C i and the radius. However, this is not enough. We introduce an important concept called the density distribution vector of D i, in a similar manner as in [7] by regarding D i as a pure geometric subset in the feature space. We quantize the enveloping disc of D i into T annular, or angular, or sectoral blocks and then count the number of points in each of the blocks in a specified order to get a vector = ( 1 ; 2 ; :::; T ) of dimension T. For more detail, please refer to [7]. This density distribution vector roughly describes the distribution state of the points of D i. Now the entropy H() of the density distribution [4] can be used to define the complexity of D i. Explicitly, set cid(d i ; ') = H() log 2 T + u(c i) (2) where the entropy H() is given by H() =? TX t=1 t jd i j log t 2 jd i j and u(c i ) is defined in above section. Here jd i j is the cardinality of set D i. Notice that a normalization is performed on the entropy summand because H() achieves maximal value log 2 T at the uniform distribution. The relationship of M clusters: ('; D 1 ; D 2 ; :::; D M ) At first, there is no relationship for only one cluster, hence ('; D) = 0 where D is a database consisting of only one cluster of similar images. Second, the relationship function should be symmetric, namely, for any permutation of f1; 2; :::; M g, ('; D 1 ; D 2 ; :::; D M ) = ('; D (1) ; D (2) ; :::; D (M) ) Third, the relationship among M clusters should be more complex than the relationship among any subset of these clusters, in other words, for any subset fi; j; :::; mg f1; 2; :::; Mg, ('; D 1 ; D 2 ; :::; D M ) ('; D i ; D j ; :::; D m ) The simplest candidate satisfying these two conditions is the average distance of the centroids of the clusters, i.e. P 1i<jM ('; D 1 ; D 2 ; :::; D d(c i; C j ) M ) = = P CM 2 2 1i<jM d(c i; C j ) M (M? 1) (3) where d(c i ; C j ) is the distance between the centroid C i of D i and the centroid C j of D j. Equations (1), (2) and (3) give the complete computation of the complexity of D under model ' by the clusteringbased approach. 3.3 Template-Based Approach A set of keyword subimages (templates) is pre-selected. Suppose all templates are designed as 4 4 subimages. Then there are possible templates if the color images have 24 bits per pixel. Obviously, directly using such a huge templates is not feasible because of the computer power. However, a reasonable set of templates can be selected. Suppose N (usually less than 10000) templates are selected in advance and are labeled with numbers from 1 to N. Now, given an image, compare each 4 subimage with the templates and find the most similar template whose label is recorded. By doing so, the image is now a list of labels of length S where S is the number of pixels of the image. 16 Hence, an image is a document which can be regarded as a list of keywords, just as a text document is a list of words. Then every concept regarding language modeling [6, 8] can be implanted to the image case. For each of the approaches described above, experimental results are to be provided later. 4. Application One important application of the concept introduced above is in the benchmarking of image retrieval systems. The concept of complexity of image database can be used to build a list of image databases on which the difficulty of the image retrieval task goes from the easiest to the most complex. It can also be used as a measure for the evaluation of a particular image retrieval algorithm. In the near future, it is predicted to play an similar role in image retrieval and image understanding as the role the language modeling plays in text retrieval and speech recognition. Furthermore, image database modeling itself has theoretical interest because it proposes a new point of view to manipulate an image database. In a more general sense, the problem itself may not be only restricted in image domain, it may be generalized to the general database. The methodology of regarding a database as an integrity on which some properties are to be derived is itself not nonsense. 5. Conclusion and Future Work We have introduced the important problem of image database modeling in content-based image retrieval and presented a theoretical framework, in parallel to the exist-

5 ing theory for language modeling. We have also investigated Some general properties of a measure of the content complexity of an image database. Two possible approach for the implementation such as query-based approach and clustering-based approach have been analyzed. It is still early to propose a satifiable necessary and sufficient condition for the measure of the complexity of an image database under a given model. In addition, we focused on the mathematical representation of our idea and paid less attention to the mathematical proof of these formulas. Our future work will focus on the modification of the theoretical framework and provide more concrete mathematical proof of the formulation. Moreover, extensive experiments are to be performed for the purpose of verification. [13] R. K. Srihari, Z. Zhang, and A. Rao. Image background search: Combining object detection techniques with content-based image retrieval(cbir) systems. Proceeding of IEEE Workshop on Content-Based Access of Image and Video Libraries (CBAIVL 99), in conjunction with IEEE CVPR 99, pages , June References [1] J. R. Bach, C. Fuller, A. Gupta, A. Hampapur, B. Horowitz, R. Humphrey, R. Jain, and C. Shu. The virage image search engine: An open framework for image management. Symposium on Electronic Imaging: Science and Technology Storage & Retrieval for Image and Video Databases IV, 2670:76 87, [2] R. Baeza-Yates and B. Ribiero-Neto. Modern Information Retrieval. Addison-Wesley, [3] R. Barber, M. Flickner, J. Hafner, W. Niblack, and D. Petkovic. Efficient and effective querying by image content. Journal of Intelligent Information Systems, 3: , [4] T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley & Sons, Inc., New York, [5] R. Elmasri and S. Navathe. Fundamentals of Database Systems, Second Edition. Addison-Wesley, [6] F. Jelinek. Statistical Methods for Speech Recognition. The MIT Press, [7] A. Rao, R. K. Srihari, and Z. Zhang. Spatial color histograms for content-based image retrieval. Proceeding of IEEE International Conference on Tools with Artificial Intelligence, pages , November [8] R. Rosenfeld. Adaptive Statistical Language Modeling: A Maximum Entropy Approach. PhD thesis, Carnegie Mellon University, [9] Y. Rui, T. S. Huang, and S.-F. Chang. Image retrieval: Past, present and future. Journal of Visual Communication and Image Representation, 10:1 23, [10] J. Smith and S.-F. Chang. Visualseek: A fully automated content-based image query system. Proceeding ACM International Conference of Multimedia, pages 87 98, November [11] J. R. Smith. Integrated Spatial and Feature Image Systems: Retrieval, Analysis and Compression. PhD thesis, Columbia Univ., [12] R. K. Srihari and Z. Zhang. Finding pictures in context. Proc. of IAPR International Workshop on Multimedia Information Analysis & Retrieval, Springer-Verlag Press, pages , 1998.

Textural Features for Image Database Retrieval

Textural Features for Image Database Retrieval Selim Aksoy and Robert M. Haralick Intelligent Systems Laboratory Department of Electrical Engineering University of Washington Seattle, WA 98195-2500 {aksoy,haralick}@@isl.ee.washington.edu