Evolutionary Feature Synthesis for Image Databases

Transcription

1 Evolutionary Feature Synthesis for Image Databases Anlei Dong, Bir Bhanu, Yingqiang Lin Center for Researh in Intelligent Systems University of California, Riverside, California 92521, USA {adong, bhanu, Abstrat The high dimensionality of visual features is one of the major hallenges for ontent-based image retrieval (CBIR) systems, and a variety of dimensionality redution approahes have been proposed to find the disriminant features. In this paper, we investigate the effetiveness of oevolutionary geneti programming (CGP) in synthesizing feature vetors for image databases from traditional features that are ommonly used. The transformation for feature dimensionality redution by CGP has two unique harateristis for image retrieval: 1) nonlinearlity: CGP does not assume any lass distribution in the original visual feature spae; 2) expliitness: unlike kernel trik, CGP yields expliit transformation for dimensionality redution so that the images an be searhed in the low-dimensional feature spae. The experimental results on multiple databases show that (a) CGP approah has distint advantage over the linear transformation approah of Multiple Disriminant Analysis (MDA) in the sense of the disrimination ability of the low-dimensional features, and (b) the lassifiation performane using the features synthesized by our CGP approah is omparable to or even superior to that of support vetor mahine (SVM) approah using the original visual features. 1 Introdution Content-based image retrieval (CBIR) systems are designed to automatially extrat various visual features from images, suh as olor, texture, shape, struture, and use them to represent images in the omputer. The olletion of suh visual features usually yields the highdimensionality of the feature spae, whih deteriorates retrieval performanes due to the well-known urse of dimensionality. Thus, a key task in CBIR is to find the most disriminant features from the original feature olletion, so that both retrieval preision and searh speed is improved. Fisher disriminant analysis (also alled Multiple disriminant analysis (MDA) for multiple-lass ase) [1] is a widely-used approah to find disriminating features due to its straight-forward idea of making linear data projetion by maximizing the ratio of the between satter matrix and the within satter matrix. The kernel trik [2] an be ombined with Fisher disriminant analysis (e.g., [3]), so that the nonlinear generalization of MDA is ahieved with lassifiation improvement. Swets and Weng [4] propose a self-organizing hierarhial optimal subspae learning and inferene framework (SHOSLIF) for image retrieval. Suh hierarhial linear analysis is a nonlinear approah in nature. Hastie and Tibshirani [5] present the approah of disriminant adaptive nearest neighbor (DANN) for lassifiation. Based on the analysis of loal dimension information, they also propose a global dimensionality method by pooling loal disriminant information. Wu et al. [6] redue the feature dimensionality for image databases using the approah of weighted multi-dimensional saling (WMDS), whose main harateristi is preserving the loal topology of the high dimensional spae. Su et al. [7] exploit priniple omponent analysis (PCA) for dimensionality redution by using relevane feedbak. In this paper, we investigate the effetiveness of oevolutionary geneti programming (CGP) [8] in generating omposite operator vetors for image databases, so that feature dimensionality is redued to improve retrieval performanes. Geneti programming (GP) is an evolutionary omputational paradigm [1] that is an extension of geneti algorithm and works with a population of individuals. An individual in a population an be any ompliated data struture suh as linked lists, trees and graphs, et. CGP is an extension of GP in whih several populations are maintained and employed to evolve solutions ooperatively. A population maintained by CGP is alled a sub-population and it is responsible for evolving a part of a solution. A omplete solution is obtained by ombining the partial solutions from all the sub-populations. For the task of objet reognition in [9], individuals in sub-populations are omposite operators, whih are the elements of a omposite operator vetor. A omposite operator is represented by a binary tree whose internal nodes are the pre-speified domain-independent primitive opera- 1

2 tors and leaf nodes are original features. It is a way of ombining original features. The CGP approah in this paper follows the algorithm proposed in [9]. The advantage of using a tree struture is that it is powerful in expressing the ways of ombining original features. The original features are visual features (e.g., olor, texture, struture) extrated from the images. With eah element evolved by a sub-population of CGP, a omposite operator vetor is ooperatively evolved by all the sub-populations. By applying omposite operators to the original features extrated from images, omposite feature vetors are obtained. These omposite feature vetors are fed into a lassifier for reognition. The transformation for feature dimensionality redution by CGP has two unique harateristis that are suitable for image retrieval: 1) nonlinearlity: CGP does not assume any lass distribution in the original visual feature spae; 2) expliitness: unlike kernel trik, CGP yields expliit transformation for dimensionality redution so that the images an be searhed the lowdimensional feature spae. The ontributions of this paper are: 1) The oevolutionary geneti programming (CGP) approah in generating omposite operator vetors for feature dimensionality redution and retrieval performane improvement in image databases (Setion 2); 2) the omparisons of the experimental results by CGP, MDA and SVM on multiple image databases, so that the advantage of the CGP approah is demonstrated (Setion 3). 2 Tehnial Approah In this paper, with the purpose of demonstrating the effetiveness of oevolutionary geneti programming approah for image databases, we simply adopt the senario of training the system by diretly providing labelled images in advane. 2.1 Image distribution By intuition the images belonging to the same lass should be lose in some sense in the feature spae of the image database, i.e., these images form a luster with a speifi shape, whih reflets the relevanes of the various visual features for the orresponding lass. However, we annot ignore the possibility that the images in the same lass (as pereived by the user) may form multiple lusters, i.e., the images with different visual features may belong to the same lass. Based on the above observations, some researhers have proposed the Gaussian mixture model for the image distribution in the feature spae of the image databases [10], with eah lass orresponding to a single mixture omponent or a variety of mixture ompo- 2D 1D Class 2 Class 1 dimensionality redution Class 1 Class 3 Class 2 Class 1 by omposite operator vetor Class 3 Class 2 Figure 1: An illustrative example for feature redution (from 2D to 1D) using omposite operation. The original 2D feature spae: three lasses have different distributions, and Class 1 and Class 2 onsists of multiple lusters. The transformed 1D feature spae: eah of the three lasses orresponds to a single Gaussian distribution. nents. Sometimes, mixture model assumption for the image distribution of a database may be too strit. In this paper, we do not have to make any model assumption for the image distribution. Figure 1 provides an example whih illustrates the harateristis of our CGP approah for reduing the feature dimensionality of image databases. The original 2D data belong to three lasses, two of whih (Class 1 and Class 2) form multiple lusters. Different lusters have different distributions (represented by different shapes in Figure 1). The transformation by the CGP approah yield 1D feature spae, in whih eah of the three lasses orresponds to a single Gaussian distribution. 2.2 CGP In CGP approah, eah synthesized feature is derived by implementing a series of operators on the original visual features. Suh operators are alled omposite operators, whih are represented by binary trees with primitive operators as internal nodes and original features as leaf nodes. The goal of these omposite operators is to map the original visual feature spae to the low-dimensional synthesized feature spae, in whih the images belonging to the same lass form a Gaussian omponent no matter how these images are distributed in the original visual spae. The searh spae of all possible omposite operators is so large that it is diffiult to find good omposite operators unless one has a smart searh strategy. How to design suh a smart searh strategy is the task for CGP algorithm. We follow the CGP algorithm proposed in [9], whih attempts to improve the performane of objet reognition. Figure 2 shows the training and test- 2

3 Ground Truth Primitive Operators Training Images Reognition Fitness Evaluator Results Coevolutionary Geneti Programming Feature Extrator Operator Vetor Primitive Features (a) Training module: learning omposite operator vetors and Bayesian lassifier. Bayesian Classifier Feature Vetors Testing Image Fitness Evaluator Primitive Features Operator Vetor Feature Vetors Bayesian Classifier Reognition Results (b) Testing module: applying learned omposite operator vetor and Bayesian lassifier to a test image. Figure 2: System diagram for objet reognition using o-evolutionary geneti programming. ing modules of the system. During training, CGP runs on training images and evolves omposite operators to obtain omposite features. Sine a Bayesian lassifier is derived from the feature vetors obtained from training images, both the omposite operator vetor and the lassifier are learned by CGP The set of primitive operators: A primitive operator takes one or two real numbers, performs a simple operation on them and outputs the result. Table 1 shows the 12 primitive operators being used, where a and b are two real numbers as the inputs to an operator and is a onstant real number stored in an operator Fitness measure: the fitness of a omposite operator vetor is omputed in the following way: apply eah omposite operator of the omposite operator vetor on the original features of training images to obtain omposite feature vetors of training images and feed them to a Bayesian lassifier. Note that not all the original features are neessarily used in feature synthesis. Only the original features that appear in the leaf nodes of the omposite operator are used to generate omposite features. The reognition rate of the lassifier is the fitness of the omposite operator vetor. To evaluate a omposite operator evolved in a sub-population, the omposite operator is ombined with the urrent best omposite operators in other sub-populations to form a omplete omposite operator vetor where omposite operator from the ith sub-population oupies the ith position in the vetor and the fitness of the vetor is defined as the fitness of the omposite operator under evaluation. The fitness values of other omposite operators in the vetor are not affeted. When sub- Table 1: Twelve primitive operators. Primitive Operator Desription ADD (a, b) Add a and b. ADDC (a, ) Add onstant value to a. SUB (a, b) Subtrat b from a. SUBC (a, ) Subtrat onstant value from a. MUL (a, b) Multiply a and b. MULC (a, ) Multiply a with onstant value. DIV (a, b) Divide a by b. DIVC (a, ) Divide a by onstant value. MAX2 (a, b) Get the larger of a and b. MIN2 (a, b) Get the smaller of a and b. (a) LOG (a) Return a if a 0; otherwise, return a. Return log(a) if a 0; Rotherwise, return log( a). populations are initially generated, the omposite operators in eah sub-population are evaluated individually without being ombined with omposite operators from other sub-populations. In eah generation, the omposite operators in the first sub-population are evaluated first, then the omposite operators in the seond subpopulation and so on Parameters and termination: The key parameters are the number of sub-populations N, the population size M, the number of generations G, the rossover and mutation rates, and the fitness threshold. GP stops whenever it finishes the speified number of generations or the performane of the Bayesian lassifier is above the fitness threshold. After termination, CGP selets the best omposite operator of eah sub-population to form the learned omposite operator vetor to be used in testing Seletion, rossover and mutation: The CGP searhes through the spae of omposite operator vetors to generate new omposite operator vetors. The searh is performed by seletion, rossover and mutation operations. The initial sub-populations are randomly generated. Although sub-populations are ooperatively evolved (the fitness of a omposite operator in a sub-population is not solely determined by itself, but affeted by the omposite operators from other sub-populations), seletion is performed only on omposite operators within a sub-population and rossover is not allowed between two omposite operators from different sub-populations. Seletion: The seletion operation involves seleting omposite operators from the urrent sub-population. We use tournament seletion with size = 5. The higher the fitness value, the more likely the omposite operator is seleted to survive. 3

4 Crossover: Two omposite operators, alled parents, are seleted on the basis of their fitness values. The higher the fitness value, the more likely the omposite operator is seleted for rossover. One internal node in eah of these two parents is randomly seleted, and the two subtrees rooted at these two nodes are exhanged between the parents to generate two new omposite operators, alled offspring. It is easy to see that the size of one offspring (i.e., the number of nodes in the binary tree representing the offspring) may be greater than both parents if rossover is implemented in suh a simple way. To prevent ode bloat, we speify a maximum size of a omposite operator (alled max-operator-size). If the size of one offspring exeeds the max-operatorsize, the rossover is performed again. If the size of an offspring still exeeds the max-operator-size after the rossover is performed 10 times, GP selets two subtrees of same size (i.e., the same number nodes) from two parents and swaps the subtrees between the parents. These two subtrees an always be found, sine a leaf node an be viewed as a subtree of size 1. Mutation: To avoid premature onvergene, mutation is introdued to randomly hange the struture of some omposite operators to maintain the diversity of subpopulations. Candidates for mutation are randomly seleted and the mutated omposite operators replae the old ones in the sub-populations. There are three mutations invoked with equal probability: 1) Randomly selet a node of the omposite operator and replae the subtree rooted at this node by another randomly generated binary tree. 2) Randomly selet a node of the omposite operator and replae the primitive operator stored in the node with another primitive operator randomly seleted from the primitive operators of the same number of input as the replaed one. 3) Randomly selet two subtrees of the omposite operator and swap them. The CGP algorithm is shown in Algorithm 1, and more details an be found in [9]. 2.3 Indexing struture For eah lass C i, a Bayesian lassifier is generated based on GP-learned omposite features. In the lowdimensional feature spae, eah lass is orresponding to a single Gaussian omponent, whih is represented by the mean feature vetor and the ovariane matrix of feature vetors of this lass. We onstrut the indexing struture based on the Gaussian omponents obtained by CGP approah. When a query image omes, the system omputes the probabilities that it belongs to those omponents using the omponents parameters (means and ovarianes), Algorithm 1 Coevolutionary geneti programming. 1. Randomly generate N sub-populations of size M and evaluate eah omposite operator in eah sub-population individually. for gen = 1 to generation num do for i =1 to N do 1) Implement standard EM algorithm on X. 2) Keep the best omposite operator in sub-population P i. 3) Perform rossover on the omposite operators in P i until the rossover rate is satisfied and keep all the offspring from rossover. 4) Perform mutation on the omposite operators in P i and the offspring from rossover with the probability of mutation rate. 5) Perform seletion on P i to selet some omposite operators and ombine them with the omposite operators from rossover to get a new sub-population P i of the same size as P i. 6) Evaluate eah omposite operator C j in P i. 7) Perform elitism replaement. 8) Form the urrent best omposite operator vetor onsisting of the best omposite operators from orresponding sub-populations and evaluate it. If its fitness is above the fitness threshold, goto 2. end for end for 2. Selet the best omposite operator from eah subpopulation to form the learned omposite operator vetor and output it. and exeutes the searh within the omponent with the highest probability. The indexing struture implies two advantages for image retrieval: 1) the searh is arried on only in the subset of the whole image database; 2) the searh is arried on in the low-dimensional feature spae. Both of these make the searh faster; furthermore, our CGP approah guarantees that the retrieval preision is high sine the Bayesian lassifiation is good in the low-dimensional feature spae. 3 Experiments To evaluate the effetiveness of the CGP approah for reduing the feature dimensionality of image databases, we implement the CGP algorithm on three different image databases with the purpose to evaluate its effetiveness for different kinds of lass distribution. All the images in these three databases are seleted from Corel Stok Photo Library.We also ompare the results of the CGP approah with other approahes inluding MDA and SVM. 3.1 Image Databases 1) DB1200: We selet 1200 images belonging to 12 lasses, and the images in eah lass have similar visual features, i.e., eah lass in the feature spae forms a luster. These 12 lasses are orresponding to the CDs (series number) in the library inluding Mayan & Azte Ruins (33000), horses (113000), owls (75000), 4

5 ADD ADD (1) CD owls (2) CD hawks and falons Class Bird Class Bird ADD ADDC F12 F1 (1) CD tulips (2) CD wildflowers Class Flower Class Flower (1) CD tigers (2) CD lions Class Beast Class Beast Figure 3: DB1500: sample images from the three lasses ontaining multiple CDs. sunrises & sunsets (1000), North Amerian wildflowers (127000), ski senes (61000, 62000), oasts (5000), auto raing (21000), firework photography (73000), divers & diving (156000), land of the Pyramids (161000) and lions (105000). 2) DB1500: We add other 300 images (from other three CDs in the Library) into DB1200 to obtain DB1500. The three new CDs (series number) are hawks and falons (70000), tigers (108000) and tulips (258000). Eah of these three CDs is merged to one existing CDs in DB1200 to form a lass, so that DB1500 still has 12 lasses. In these 12 lasses, there are three lasses eah of whih onsists of two lusters in visual feature spae: the CD of Hawks and Falons and the CD of owls form the lass of bird, the lass of tulips and the lass of North Amerian wildflowers form the lass of flowers, and the lass of tigers and the lass of lions form the onept of wild beasts. Figure 3 shows the sample images of these three lasses ontaining multiple CDs. Obviously, DB1500 is hallenging for the linear transformation approah suh as MDA, as we will show later in the experiments. 3) DB6600: The 6,600 images are obtained from 66 CDs, whih are assigned to 50 lasses, i.e., eah lass may onsist of a single CD or multiple CDs. For instane, the CDs Arabian horses and horses are assigned to the same lass sine they are both for the same onept of horse. On the other hand, even the images within the same CD may form multiple lusters in sense of visual features. For example, the CD of ity of Italy onsists of images for different objets suh as building, road, people. For most of the lasses, there are many outlier images, eah of whose visual features is far away from the luster(s) its orresponding lass ontains. F5 (a) F20 Figure 4: Sample omposite operator vetors in DB6600 (the leaf nodes are original visual features): (a) a simple omposite operator vetor, (b) a omplex omposite operator vetor. Original visual features: Images are represented by texture features, olor features and struture features. The texture features are derived from 16 Gabor filters [11]. We also extrat means and standard deviations from the three hannels in HSV olor spae. For struture features, we use the water-filling approah [12] to extrat 18 features from eah image. Thus, eah image is represented by 40 visual features. 3.2 Experiments results Experimental parameters: For eah of the three image databases, we randomly selet half of the images as training data and another half as testing data. We implement CGP algorithm on the training images, and the parameters values are: (a) sub-population size: 50; (b) rossover rate: 0.6; () number of generation: 50; (d) mutation rate: 0.05; (e) fitness threshold: 1.0; (f) tournament size: 5. For both CGP approah and MDA approah, we redue the feature dimensionality from 40 to 10, so that it is fair to ompare these two approahes. Figure 4 gives two sample omposite operator vetors in DB6600. The omplex ase in (b) illustrates that the unonventional ombinations of the operators exists in CGP, whih may help to ahieve good lassifiation performane. Classifiation performane: Table 2 shows the lassifiation performanes of the approahes of CGP, MDA, and SVM. The first two attempt to redue the feature dimensionality in the expliit ways, so that the data in the same lass form a single Gaussian omponent in the lower-dimensional feature spae. Thus, we present their Bayesian lassifiation errors in the lowerdimensional (10D) feature spae. Sine SVM exploits kernel trik instead of providing expliit transformation, we present its lassifiation error in the original visual feature spae (40D). From Table 2, we observe that the lassifiation performane of CGP is better than that of MDA on all (b) F12 5

6 Table 2: Classifiation errors: the omparison of the three different approahes on different databases. Database CGP (10D) MDA (10D) SVM (40D) DB % 25.5% 12.6% DB % 31.7% 29.6% DB % 73.3% 52.6% of the three databases. It is easy to understand the signifiant advantage of CGP over MDA on DB1500 and DB6600, both of whih ontain some lasses onsists of multiple lusters in the original visual feature spae. Therefore, the linear approah of MDA annot deal with them well. Although eah of the lass in DB1200 only orresponds to a single luster in the original visual feature spae, CGP still outperforms MDA (14.8% versus 12.6%) due to the reason that the distribution of these lusters may not be Gaussian while MDA assumes the luster distribution as Gaussian and CGP does not make suh assumption. We now ompare CGP and SVM, whih have similar lassifiation errors on DB1200 and DB6600. However, the error (16.5%) by CGP is signifiantly superior to that (29.6%) by SVM on DB1500, whih implies that CGP is more suitable to deal with the lass onsisted of multiple lusters in the original feature spae. With many outliers in DB6600, suh advantage of CGP is not so obvious as in the ase of DB1500 (CGP: 51.8% versus MDA: 52.6%), and neither CGP nor SVM an yield lassifiation performane whih is as good as those on DB1500. Retrieval performane: After reduing the feature spae, we onstrut the indexing struture based on the Gaussian omponents obtained by CGP approah or MDA approah. We use eah of the image the database as query, and obtain average retrieval preisions. Figure 5 shows the retrieval-preision urves on DB6600 by CGP and MDA. We observe that CGP yields better retrieval results than MDA. From above experiments, we onlude that (a) the feature synthesizing by CGP has obvious advantage over that by MDA in the sense of both lassifiation and retrieval. (b) The lassifiation performane in the synthesized feature spae by CGP is omparable to or even superior to that in the original feature spae by SVM. 4 Conlusions This paper presents a oevolutionary geneti programming (CGP) for feature dimensionality redution. The two harateristis of the transformation by CGP, i.e., nonlinearity and expliitness, make it suitable for image retrieval: (a) the nonlinearity yields good lassifia- preision CGP MDA reall Figure 5: DB6600: preision-reall urves: CGP versus MDA. tion performane without any lass distribution model assumption, and (b) the expliitness ahieves the image searh in the low-dimensional feature spae. Experimental results on multiple image databases with different types of distributions have demonstrated the effetiveness of improving image retrieval performane by the CGP approah. Referenes [1] R. Duda, P. Hart, and D. Stork, Pattern Classifiation, John Wiley & Sons, seond edition, [2] B. Shölkopf and A.J. Smola, Learning with Kernels: Support Vetor Mahines, Regularization, Optimization, and Beyond, the MIT Press, [3] G. Baudat and F. Anouar, Generlized disriminant analysis using a kernel approah, Neural Computation, vol. 12, pp , [4] D. Swets and J. Weng, Hierarhial disriminant analysis for image retrieval, IEEE Trans. PAMI, vol. 21, no. 5, pp , May [5] T. Hastie and R. Tibshirani, Disriminant adaptive nearest neighbor lassifiation, IEEE Trans. PAMI, vol. 18, no. 6, pp , [6] P. Wu, B. S. Manjunath, and H. D. Shin, Dimensionality redution for image searh and retrieval, ICIP, [7] Z. Su, S. Li, and H. J. Zhang, Extration of feature subspaes for ontent-based retrieval using relevane feedbak, Pro. ACM MM, pp , [8] J. Koza, Geneti Programming II: Automati Disovery of Reusable Programs, MIT Press, [9] Y. Lin and B. Bhanu, Evolutionary feature synthesis for objet reognition, IEEE Trans. SMC, Part B, to appear. [10] N. Vasonelos, Bayesian Models for Visual Information Retrieval, Ph.D thesis, MIT, [11] B. S. Manjunath and W. Y. Ma, Texture feature for browsing and retrieval of image data, IEEE Trans. PAMI, vol. 18, no. 8, pp , August [12] X.S. Zhou, Y. Rui, and T.S. Huang, Exploration of Visual Data, Kluwer Aademi Publishers,