HBIR: Hypercube-Based Image Retrieval

Transcription

1 Ajorloo H, Lakdashti A. HBIR: Hypercube-based image retrieval. JOURNAL OF COMPUTER SCIENCE AND TECH- NOLOGY 27(1): Jan DOI /s HBIR: Hypercube-Based Image Retrieval Hossein Ajorloo 1 and Abolfazl Lakdashti 2 1 Department of Computer Engineering, Sharif University of Technology, Tehran, Iran 2 University College of Rouzbahan, Sari, Iran ajorloo@ce.sharif.edu; lakdashti@rouzbahan.ac.ir Received April 13, 2010; revised August 29, Abstract In this paper, we propose a mapping from low level feature space to the semantic space drawn by the users through relevance feedback to enhance the performance of current content based image retrieval (CBIR) systems. The proposed approach makes a rule base for its inference and configures it using the feedbacks gathered from users during the life cycle of the system. Each rule makes a hypercube (HC) in the feature space corresponding to a semantic concept in the semantic space. Both short and long term strategies are taken to improve the accuracy of the system in response to each feedback of the user and gradually bridge the semantic gap. A scoring paradigm is designed to determine the effective rules and suppress the inefficient ones. For improving the response time, an HC merging approach and, for reducing the conflicts, an HC splitting method is designed. Our experiments on a set of images from the Corel database show that the proposed approach can better describe the semantic content of images for image retrieval with respect to some existing approaches reported recently in the literature. Moreover, our approach can be better trained and is not saturated in long time, i.e., any feedback improves the precision and recall of the system. Another strength of our method is its ability to address the dynamic nature of the image database such that it can follow the changes occurred instantaneously and permanently by adding and dropping images. Keywords image retrieval, relevance feedback, rule base, hypercube (HC) 1 Introduction Managing pictorial data, such as art galleries, medical image archiving, trademark signs, increase the demand for the development of efficient image retrieval systems. Recently, many content-based image retrieval (CBIR) systems have emerged to address some of the needs [1-6]. The so-called semantic gap in these traditional systems has emerged some new techniques for bridging it. Relevance feedback (RF) [7-15] is an interactive process, which tries to do this. Its general idea is described as follows: the user initializes a query session by submitting an image. The system then returns some images which are inferred as relevant by the underlying algorithm. If the user is not satisfied with the retrieved images, the user can activate an RF process by identifying which retrieved images are relevant and which are irrelevant. The system adapts its internal parameter values to include as many user-desired images as possible in the next retrieved results. The process is repeated until the user is satisfied or the results cannot be further improved. A rule based technique is developed in this paper to make a long-term RF learning which uses hypercubes (HCs) in the feature difference space to make semantic space components. The method is called HC-based image retrieval (HBIR). Beside this long term method, a short term approach is used to improve the accuracy of the system immediately after each feedback to satisfy the user in each query. The remainder of this paper is organized as follows. Section 2 describes the related research on RF and highlights our motivation. Section 3 presents the proposed image retrieval system. Section 4 gives the experimental results and provides comparative performances. The limitations and future work are presented in Section 5. Finally, Section 6 presents the conclusions of the paper. The notations of the variables used in the paper are presented in Table 1. 2 Related Research, Motivation, and Contributions In this section, we look at some major approaches of RF which were published in recent years. Relevance feedback is a technique to train the image retrieval systems from user provided feedbacks. Lin et al. [16] proposed a method called augmented relation Regular Paper 2012 Springer Science + Business Media, LLC & Science Press, China

2 148 J. Comput. Sci. & Technol., Jan. 2012, Vol.27, No.1 Table 1. Notations of Variables in the Paper Notation Meaning Notation Meaning t Number of retrieved images in one display n Feature space dimensionality F q Feature vector of the query image F t Feature vector of the target (database) image R Set of relevant images for a certain query I Set of irrelevant images for a certain query HR i The i-th relevant HC HI i The i-th irrelevant HC h R Number of hit relevant rules h I Number of hit irrelevant rules N R Number of relevant rules N I Number of irrelevant rules c R Number of relevant clusters c I Number of irrelevant clusters FB R Set of feature differences F q F t where FB I Set of feature differences F q F t where the user recognized them as relevant the user recognized them as irrelevant C Set of the cluster centroids M Membership of each point to the clusters p j The i-th parameter of the k-th dimension p j Ri,k I i,k The i-th parameter of the k-th dimension of the j-th relevant HC, of the j-th irrelevant HC, 1 i 2, 1 j N R, 1 k n 1 i 2, 1 j N I, 1 k n s R i Score of the i-th relevant rule, 1 i N R s I i Score of the i-th irrelevant rule, 1 i N I ι Initial score of all rules (= 40 in our σ The value subtracted from missed rules experiments) in each iteration (= 0.1 in our experiments) φ R The value added to correct hit relevant φ I The value added to correct hit relevant rules (= 10 in our experiments) rules (= 3 in our experiments) µ R The value added to incorrect hit relevant µ I The value added to incorrect hit relevant rules (= 0.5 in our experiments) rules (= 1 in our experiments) T o Allowable total overlap between opponent T m Allowable overlap between opponent HCs in optimal-kmeans algorithm HCs in merging alternate ones embedding (ARE) in which the similarity relations (relevances and irrelevances) with respect to the query image given by the feedbacks are stored in two matrices and the low-level similarities are stored in another matrix using a distance measure. An optimum transformation is then obtained to reduce the dimensionality of the feature space using these matrices. The retrieval is done in the transformation domain. A similar approach is proposed by Yu and Tian [17]. They proposed a similarity measure which is a combination of both low level similarity measure and high level semantical similarity calculated from the feedbacks. An optimal semantic subspace projection (SSP) that captures the most important properties of the subspaces with respect to classification is then computed which reduces the dimensionality of the feature space. The similarities are updated in an RF paradigm using the feedbacks. In order to cope with the high dimensionality, a semisupervised method is proposed called maximum margin projection (MMP) aiming at maximizing the margin between positive and negative examples at each local neighborhood [18]. Different from traditional dimensionality reduction algorithms such as principal component analysis (PCA) and linear discriminant analysis (LDA), which effectively see only the global Euclidean structure, MMP is designed for discovering the local manifold structure. Therefore, MMP is likely to be more suitable for image retrieval, where nearest neighbor search is usually involved. After projecting the images into a lower dimensional subspace, the relevant images get closer to the query image; thus, the retrieval performance can be enhanced. In this method, the inter-class distances between images in a class are maximized, while intra-class distances are minimized in the transformation domain. In calculation of the distances, the similarity matrices are utilized which are defining the relation between the images based on the feedbacks. All of these methods suffer from the extremely large matrix computations which will be unmanageable for large databases and when the number of feedbacks grows in the scale of the database size. The difference between the low level features of query and target images is a convenient representative to judge how much two images are similar. In this case, in the n-dimensional space of feature differences, any pair of query-target images makes a unique point. If a user decides to mark this pair of images as relevant, we have a positive point and otherwise, we would have a negative point. Therefore, in the n-dimensional space, these points constitute some positive or negative islands (see Fig.1(a)). Each of these islands can be regarded as a semantic concept indicating the corresponding region as a relevant or irrelevant instance. Therefore, we can fit an HC on each island and make a semantic mapping (see Fig.1(b)). The remaining regions (dashed regions in Fig.1(b)) are regarded as unknown regions and in the retrieval process, if a point is settled in these regions, we can treat it using the underlying CBIR system, until the system learns them in future feedbacks. The main contributions of this paper are as follows: 1) designing a rule based structure for the retrieval process to determine which images in the database are

3 Hossein Ajorloo et al.: Hypercube-Based Image Retrieval 149 effective and ineffective rules; 4) proposing an HC merging procedure for merging adjacent HCs to reduce the system complexity; 5) proposing an HC splitting procedure to split the conflicting HCs. 3 Proposed Approach In this section, we discuss our proposed framework of image retrieval, and its main components containing the RF inference and the RF learning modules. 3.1 Proposed Framework Fig.1. Illustration of the feature relevance mapping approach. (a) Each pair of query-target images constitutes a positive or negative point in the n-dimensional space. (b) For each cluster, an HC is fitted to make a mapping between low level feature difference space and the semantic space driven by users through RF process. relevant to a given query image; 2) designing a training algorithm to make new HCs and rules by gathering feedbacks from the users; 3) providing a scoring paradigm to determine the Fig.2 shows the structure of our proposed image retrieval system. In this system, first, the user sends a query image. The predefined features of the query image are then extracted, and the long-term RF system decides which images in the database are relevant to the query image, based on its rules in the rule base. If the long-term RF system fails to determine the relevance of the query image, it signals the CBIR system to do this. Whether the long-term RF or the CBIR system retrieves the images from the database recognized as relevant by the system, the user is asked to determine which of the images are relevant or irrelevant. The system can also ask the user for approvement of some other images recognized as irrelevant by the system. The relevance feedbacks from the user is accumulated in a feedback base. If the CBIR system did the retrieval, these feedbacks can be used to modify the results by the short-term RF module, and the process is iterated until the user is satisfied. The accumulated feedbacks are utilized periodically to make and modify the rule base. The optimal K- means module makes the best clustering for both positive and negative feedbacks. The result is used by the Fig.2. Overall structure of the proposed image retrieval system.

4 150 J. Comput. Sci. & Technol., Jan. 2012, Vol.27, No.1 rule extraction module to make HCs and thereby new rules. The HC merging and splitting and rule scoring modules modify the HCs and rules to enhance the performance and speed up the system. The accumulated feedbacks are thrown out after each training phase to accommodate for new feedbacks. Really, the old feedbacks are converted to HCs and rules. Each of these modules will be discussed in detail in next subsections. 3.2 RF Inference The RF inference system is responsible for deciding whether a query image is relevant to a target image from database. It uses a rule base for this purpose. The rule base contains If-Then rules of the following form: Rule 1: If F q F t lies in HR 1 Then h R := h R + 1. Rule 2: If F q F t lies in HR 2 Then h R := h R Rule N R : If F q F t lies in H N R R Then h R := h R + 1. Rule N R + 1: If F q F t lies in HI 1 Then h I := h I + 1. Rule N R + 2: If F q F t lies in HI 2 Then h I := h I Rule N R + N I : If F q F t lies in H N R I Then h I := h I + 1, where, h R and h I are two parameters initialized at 0 indicating numbers of hit relevant and irrelevant rules, respectively. Since the HCs may overlap, a point in feature difference space may lie in the overlapped region and hence in several HCs, simultaneously. When a point lies in an HC, we say the corresponding rule is hit. So, for any point, we may have zero, 1, or more rule hits. The inference system checks all rules and therefore sums up the number of hit rules indicating the query image as relevant and irrelevant to the target image via h R and h I. It then assigns a relevance number to this query-target pair as follows: r(q, t) = h R h R + h I. (1) It then sorts all images in the database based on their relevance numbers in descending order and then to their Euclidian distances to the query image in ascending order. It then returns the first t images to the user. In our system, each rule corresponds exactly to a single HC, and thereby, the terms HC and rule are used interchangeably, thereafter. 3.3 RF Learning Fig.3 shows the learning procedure designed for making the rule base. After each feedback, the system checks whether the accumulated feedbacks so far are sufficient or a predefined time is spent from previous update. If so, the optimal K-means (Subsection 3.3.1) algorithm is run. Then, the new HCs are created and the HC merging and splitting procedures are executed. Based on these variations, the rule base is updated accordingly. These steps are discussed in the following subsections. Fig.3. Long-term learning procedure Optimal K-Means After each query, t R images recognized as relevant and t N other images recognized as irrelevant by the system are returned to the user and he/she is asked for approving the results. The feedbacks from the user make a pair of the form ( F q F t, r) where F q and F t denote the query and target image features, respectively and r equals 1 for relevant images and 0, otherwise. We should jointly cluster relevant and irrelevant points to make clusters with minimum overlaps between opponent points. Algorithm 1 finds the best clustering for positive and negative points such that the overlap volume between opponent HCs is minimized. The system initializes c R and c I, numbers of relevant and irrelevant clusters, respectively, to 1. In a repeatuntil loop, the system checks whether the total volume of the overlap between positive and negative HCs created from increasing c R or c I is more than a predefined threshold T o, and it sets the corresponding clusters and HC parameters, appropriately. Kmeans(FB R, c Rn ) runs the K-means algorithm on feedbacks of relevant points with c Rn clusters and returns the matrix C of

5 Hossein Ajorloo et al.: Hypercube-Based Image Retrieval 151 Algorithm 1. Proposed Optimal K-means Algorithm for Finding the Best Clustering for Positive and Negative Points Input: the sets of feedbacks FB R and FB I Output: the sets of hypercubes H R and H I 1 c R 1; 2 c I 1; 3 repeat 4 c Rn c R + 1; 5 [C, M] Kmeans(FB R, c Rn); 6 H Rn Hypercubes(C, M) 7 V R OverlapVolume(H Rn, H I); 8 c In c I + 1; 9 [C, M] Kmeans(FB I, c In); 10 H In Hypercubes(C, M); 11 V I OverlapVolume(H R, H In); 12 if V R < V I then 13 c R c Rn; 14 H R H Rn; 15 V V R; 16 else 17 c I c In; 18 H I H In; 19 V V I; 20 end 21 until V > T o; cluster centroids and matrix M of membership of each point to the clusters. HCs(C, M) computes the parameters of HCs for each cluster (Subsection 3.3.2). OverlapVolume(H Rn, H I ) measures the total volume of overlap between HCs H Rn and H I. The repeat-until loop ends when the total overlap volume reaches T o. The OverlapVolume(.) function computes the total sum of overlap volume between each pair of positive and negative HCs. The overlapped volume between two HCs H i = {p i 1,1, p i 2,1, p i 1,2, p i 2,2,..., p i 1,n, p i 2,n} and H j = {p j 1,1, pj 2,1, pj 1,2, pj 2,2,..., pj 1,n, pj 2,n } is computed as follows: n { ( V (H i, H j )= min(p i 2,k, p j 2,k ) max(pi 1,k, p j 1,k ) ) k=1 U ( min(p i 2,k, p j 2,k ) max(pi 1,k, p j 1,k ) ) }, (2) where U(.) denotes the unit step function defined as { 1, x 0, U(x) = (3) 0, otherwise. Note that in the above formulation, when for a certain dimension l, p i 1,l = pi 2,l = pj 1,l = pj 2,l, the unit step function returns 1, and therefore it removes the effect of the feature element l. This algorithm guarantees that the resulting clusters for positive and negative feedback points have the minimum volume of overlap between opponent clusters, but it does not guarantee to make the minimum number of clusters for each set of positive or negative points. The algorithm is greedy in this sense. However, the number of clusters will be minimized in the cluster merging algorithm (Subsection 3.3.3) Computation of HC Parameters After each clustering step is done with K-means algorithm, it is desirable to convert the clusters to appropriate HCs in the n-dimensional space of the feature vectors. Each HC has 2n parameters, for which we propose the following formulas to compute them from each cluster s points (see also Fig.4): p j 1,i = min{ f q i f t i F q F t C j } δ i, p j 2,i = max{ f q i f t i F q F t C j } + δ i, (4) where δ i is a margin parameter for which we use 0.5% of the dynamic range of each feature element. The parameter p j k,i denotes the k-th parameter of the i-th dimension in the j-th HC depicted in Fig.4. In (4) C j denotes the j-th cluster, and f q i is the i-th element of the vector F q. Fig.4. Computing HC parameters from each cluster. In Fig.4, a cluster is represented by a rectangle (HC in the n-dimensional space for n > 2) surrounding all points of the cluster. This cluster can be splitted in two subclusters and make two smaller rectangles with less improper regions included in the rectangles. Really, the more HCs we made, the more precise system will be created. But, increasing the number of HCs slows down the system and increases the complexity. Note that computation of positive (negative) HC parameters from its corresponding cluster can be done independent of negative (positive) HC parameters HC Merging Procedure After creating new HCs in the evolution of system when it periodically makes new rules from recent

6 152 J. Comput. Sci. & Technol., Jan. 2012, Vol.27, No.1 feedbacks, new HCs may overlap with other congruent HCs created before. Moreover, some HCs made in the same step can be merged, without reducing the system performance, since there does not exist any opponent HC between them. For merging HCs and reducing the number of rules, Algorithm 2 is proposed. It merges positive HCs (merging the negative HCs is done in the same way). Algorithm 2. Proposed Algorithm for Merging Positive HCs Input: the set of hypercubes H R Output: the set of modified hypercubes H R 1 for i 1 to N R 1 do 2 for j i + 1 to N R do 3 if IsDeleted(H j R ) then 4 if IsSurround(H i R, H j R ) then 5 Delete(H j R ); 6 else 7 H m Merge(H i R, H j R ); 8 V OverlapVolume(H m, H I); 9 if V < T m then 10 Delete(H j R ); 11 H i R H m; 12 end 13 end 14 end 15 end 16 end The algorithm checks if it merges each pair of positive HCs, the overlap between the resulting HC and negative HCs is less than a predefined threshold T m. If so, it merges them. The function IsDeleted(.) checks whether its argumented HC is deleted. The sign before it denotes the logical NOT operator. The function issurround(hr, i H j R ) checks whether Hi R surrounds H j R. If so, the algorithm deletes Hj R and continues with other pairs of HCs. The function merge(hr, i H j R ) merges two HCs HR i and H j R in the following way: p m 1,k = min(p i R1,k, p j R1,k ), p m 2,k = max(p i R2,k, p j ), k = 1,..., n, (5) R2,k where p m 1,k denotes the first parameter of the k-th element of the resulting merged HC. The algorithm in its worst case runs the OverlapVolume(.) function N R (N R 1)/2 times. Fig.5 shows two examples for merging HCs. In Fig.5(a), the positive HCs (solid line) can be merged, since the overlap between the resulting HC (dasheddotted line) and the negative HC (dashed line) is less than the threshold T m. In Fig.5(b), these HCs cannot be merged, since the overlap is more than T m. Note that in Fig.5(a) a relatively large unknown area (white region) is also added to the positive HC, i.e., the Fig.5. Illustration of the merging procedure. (a) Two positive HCs can be merged, since the overlap volume V is less than the predefined threshold T m. (b) Two positive HCs cannot be merged, since the overlap volume V is more than the predefined threshold T m. system treats with these areas as a positive area. If this assumption is incorrect, and its opposite is proven in the next feedbacks, the HC is splitted in the HC splitting phase HC Splitting Procedure After each round of feedbacks, the new created rules may conflict with the old ones, i.e., the positive and negative HCs may be overlapped. In this case, the overlapping opponent HCs should be splitted such that each HC is partitioned into some smaller non-overlapping HCs. The overlapping region is left out as unknown region, since, the system cannot divide it into appropriate positive and negative HCs. Algorithm 3 splits both positive and negative HCs in smaller sub-hcs such that the overlap region is kept small enough. The functions AddNewRHC(.) and AddNewIHC(.) add new relevant and irrelevant HCs, respectively, with the specified parameters (and hence, add new rules to the rule base). This algorithm checks all pairs of relevant and irrelevant HCs (lines 1 and 2) to see whether they are overlapping. If so, it splits the overlapping HCs to smaller ones and deletes the overlapping region. The newly created HCs, do not overlap with opposite HCs. In the case of overlap between two opponent HCs, the overlapping rules will be deleted and 2n new rules are

7 Hossein Ajorloo et al.: Hypercube-Based Image Retrieval 153 Algorithm 3. Proposed Algorithm for Splitting the Overlapping Hypercubes Input: the sets of hypercubes H R and H I Output: the sets of modified hypercubes H R and H I 1 for i 1 to N R do 2 for j 1 to N I do 3 for k 1 to n do 4 if p i R1,k p j I 1,k pi R2,k then 5 AddNewRHC(p i R1,1, p i R2,1, p i R1,2, p i R2,2,..., p i R1,k, p j I 1,k, pi R1,k+1, p i R2,k+1,, p i R1,n, p i R2,n); 6 AddNewIHC(p j I 1,1, pj I 2,1, pj I 1,2, pj I 2,2,..., p i R2,k, p j I 2,k, pj I 1,k+1, pj I 2,k+1,..., pj I 1,n, p j I 2,n ); 7 p i R1,k p j I 1,k ; // Shrink Hi R 8 p j I 2,k pi R2,k; // Shrink H j I 9 else if p j I 1,k pi R1,k p j I 2,k then 10 AddNewRHC(p i R1,1, p i R2,1, p i R1,2, p i R2,2,..., p j I 2,k, pi R2,k, p i R1,k+1, p i R2,k+1,..., p i R1,n, p i R2,n); 11 AddNewIHC(p j I 1,1, pj I 2,1, pj I 1,2, pj I 2,2,..., p j I 1,k, pi R1,k, p j I 1,k+1, pj I 2,k+1,..., pj I 1,n, p j I 2,n ); 12 p i R2,k p j I 2,k ; // Shrink Hi R 13 p j I 1,k pi R1,k; // Shrink H j I 14 else if p i R1,k p j I 1,k pj I 2,k pi R2,k then 15 AddNewRHC(p i R1,1, p i R2,1, p i R1,2, p i R2,2,..., p i R1,k, p j I 1,k, pi R1,k+1, p i R2,k+1,..., p i R1,n, p i R2,n); 16 AddNewRHC(p i R1,1, p i R2,1, p i R1,2, p i R2,2,..., p j I 2,k, pi R2,k, p i R1,k+1, p i R2,k+1,..., p i R1,n, p i R2,n); 17 p i R1,k p j I 1,k ; 18 p i R2,k p j I 2,k ; // Shrink Hi R 19 else if p j I 1,k pi R1,k p i R2,k p j I 2,k then 20 AddNewIHC(p j I 1,1, pj I 2,1, pj I 1,2, pj I 2,2,..., p i R2,k, p j I 2,k, pj I 1,k+1, pj I 2,k+1,..., pj I 1,n, p j I 2,n ); 21 AddNewIHC(p j I 1,1, pj I 2,1, pj I 1,2, pj I 2,2,..., p j I 1,k, pi R1,k, p j I 1,k+1, pj I 2,k+1,..., pj I 1,n, p j I 2,n ); 22 p j I 1,k pi R1,k; 23 p j I 2,k pi R2,k; // Shrink H j I 24 else 25 Break // (exit from the inner for loop), Hypercubes do not overlap 26 end 27 end 28 end 29 end added to the rule base. Fig.6 shows four examples of splitting HCs. As it can be seen, in each case the overlapping region is omitted and the remaining regions in each HC is divided into appropriate sub-hcs. In Fig.6(d), the relevant HC (RHC) surrounds the Fig.6. Illustration of the splitting procedure (4 examples). The left plot in each figure shows the overlapping relevant (RHC) and irrelevant (IHC) HCs, and the right plot in each figure shows the splitted HCs. irrelevant HC (IHC). Therefore, the overlapping region is the total volume of IHC and hence, it is omitted after splitting, and the RHC is itself splitted into 4 sub-hcs. Regarding these two algorithms, two questions are arisen which will be answered in the following remarks. Remark 1. The split phase removes some HCs and produces some new smaller HCs. It enhances the precision of the system, but it deteriorates the time efficiency, because the number of rules that should be checked increases. In the next rounds of the training the RF inference module, some of these new HCs may be merged again and produce new HCs. The merging

8 154 J. Comput. Sci. & Technol., Jan. 2012, Vol.27, No.1 phase may reduce the precision of the system, but otherwise it enhances the time efficiency. In long run, the system finds the best partitioning of the feature difference space in almost all scenarios. Remark 2. As stated in Remark 1, the split phase deteriorates the time efficiency. More precisely, by splitting each rule, 2n new rules will be produced. So, for each split, 2n new rules should be checked instead for the new queries. However, most of these rules are merged in the next stages of the RF inference. As stated before, the merging phase enhances the time efficiency of the system. More precisely, by merging two rules, instead of two rules, one rule is checked in the next queries Scoring Rules Another tool for adapting the rules is rule scoring module. It scores the rules in the rule base to determine which rules are effective and which are not. This is done by the following procedure. The system initializes the score of all rules, s i R and sj I (1 i N R and 1 j N I to a predefined positive number ι). After each feedback, if the system has recognized the query as relevant and the user approves it, the system adds a predefined value φ R to all relevant hit rules scores and decreases µ I from all irrelevant hit rules scores. If the user does not confirm it, the system adds a predefined value φ I to all irrelevant hit rules scores and decreases µ R from all irrelevant hit rules scores. The score of all other rules are subtracted by σ. In our experiments, we used the following values for these parameters: ι = 40, φ R = 10, φ I = 3, µ R = 0.5, µ I = 1, and σ = 0.1. For the case of recognizing the query as irrelevant, the procedure is similar. If the score of a rule reaches 0, it will be omitted from the rule base. In this way, the scores of rules that do not hit in a relatively long period reach 0, gradually, and will be omitted from the rule base. Moreover, since the number of relevant rules are typically much smaller than the number of irrelevant rules, we get them more opportunity to live in the system. By this scoring procedure, the malfunctioning and ineffective rules will be omitted from the rule base, and hence, the speed and accuracy of system will be increased gradually by feedbacks from the users. 4 Experimental Results and Comparative Performance Evaluation In this section, we provide our experimental results. We selected images containing 110 folders of images from the Corel database and classified them into 26 classes. We assume each image corresponds to exactly one class and all images in a class are relevant. 4.1 Low-Level Features Two features were extracted from the images discussed in the following subsections Texture We used the pattern orientation histogram (POH) method as the texture features which is based on the pattern orientations in spatial domain [25]. POH represents distribution of five types of patterns from each image and produces 80 bins histogram. Increasing the size of image blocks in the POH method results in an increase of the precision of the image retrieval system up to an optimal value from which the precision deteriorates by more increasing the size. This optimal value depends on the database images. We used a moderate value which is near to the optimal value computed for our database. However, this is not the concern of this paper, since we do not focus on the low-level features, but our main concern is to enhance the performance of the CBIR systems with any low-level feature used Color A good color space is one in which the perceived color differences should correspond to their Euclidean distances in this chosen color space. The HSV color model is known to satisfy this property. We quantize the HSV color space into 162-bin color histogram. These values are achieved by a uniform quantization, which includes 18 levels in H, three levels in S, and three levels in V color space. 4.2 Evolution of Systems vs User Feedbacks We compared our proposed HBIR approach with a simple Euclidian distance CBIR, ARE [16], SSP [17], MMP [18] methods and the ideal system which returns only relevant images and no irrelevant ones. In precision-recall plots which follows, for a single query, the ideal system makes a rectangle in the plot, but in the average precision-recall plots, it does not produce a rectangle. This is because the number of relevant images for each query differs from other ones. So, when averaging the precision-recalls for different numbers of returned images, say for r images, for a query, all first r images may be relevant, but for another query, only the first p images (p < r) may be relevant and the other r p ones are irrelevant. So, after averaging, the precision value becomes a number between 0 and 1, and hence, the resulting plot does not resemble a rectangle. In the following sections, we will discuss the results of image retrieval performed by these approaches. In all of these approaches, we used the same low level features discussed in Subsection 4.1.

9 Hossein Ajorloo et al.: Hypercube-Based Image Retrieval 155 Fig.7 shows the plots of precision-recall values averaged over 110 randomly selected query images each from one of the mentioned Corel classes for different methods after gathering 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 100% of users feedbacks. The precision and recall parameters are defined as: Precision = Recall = number of relevant images retrieved total number of images retrieved, number of relevant images retrieved. total number of relevant images (6) In Fig.7, the green line denotes the simple Euclidian distance CBIR (written as CBIR in diagrams) which does not change by feedbacks. It is used as a lower bound for comparing the methods. The cyan line denotes the average precision-recall values of an ideal system. Note that since the precision and recall values are averaged over 110 images, it does not resemble a rectangle. However, for a single query it will be seen in a rectangle form as it is shown in the following figures. This ideal case is shown as an upper bound on precision and recall values. As it is evident from these plots, by gathering enough feedbacks from the users, the systems evolve and try to reach the ideal system. However, our Fig.7. Precision-recall plots for different percentages of gathered feedbacks averaged over 110 query images. (a) 10%. (b) 20%. (c) 30%. (d) 40%. (e) 50%. (f) 60%. (g) 70%. (h) 80%. (i) 100% of gathered users feedbacks.

10 156 J. Comput. Sci. & Technol., Jan. 2012, Vol.27, No.1 proposed HBIR system is more successful than other approaches and has more capacitance in learning the users feedbacks. The ARE and SSP approaches have similar performances, with SSP having slightly better results. The MMP approach has more ability in learning the users semantics with respect to the previously mentioned methods. On the other hand, the proposed HBIR method resembles almost an ideal system after gathering enough feedbacks from the users. Fig.8 shows the evolution of precision and recall values averaged over the same 110 randomly selected query images as a function of the percentage of gathered feedbacks for different methods mentioned before. These values are computed for retrieval of 60 images by each method. The results of the ideal and simple Euclidian distance CBIR systems are also depicted as upper and lower bounds, respectively. Again, the superiority of the HBIR system over other approaches is evident from these figures. It can approach itself to the ideal system by gathering more feedbacks from the users. That is, it evolves better than the other approaches. Fig.9 shows the average precision-recall plots of different methods for different percentages of gathered Fig.8. Precision (a) and recall (b) values averaged over 110 query images for different methods vs percentage of gathered feedbacks for 60 retrieved images. Fig.9. Precision-recall plots for different methods averaged over 110 query images, for different percentages of gathered feedbacks. (a) ARE. (b) SSP. (c) MMP. (d) HBIR.

11 Hossein Ajorloo et al.: Hypercube-Based Image Retrieval 157 feedbacks. We include these plots to emphasize the evolutionary power of each method. The number written after each method s name, such as ARE10, denotes the percentage of feedbacks utilized. From these plots, we conclude that the proposed HBIR system better evolves with users feedbacks than the other approaches. If we utilize more sophisticated features, we can approach faster to the ideal system by less feedbacks from the users. In Table 2 the average precision and recall values are written for different numbers of returned images (10, 20,..., 60 images in each column) and different percentages of feedbacks (10%, 20%,..., 100% feedbacks in each row) for the considered methods. These also support our claim on effectiveness of the HBIR system compared to other existing approaches. The average Table 2. Precisions and Recalls Computed at Different Numbers of Retrieved Images and Different Percentages of Gathered Feedbacks for Each Method Percentage Precision Recall N Ret of Feedback CBIR Ideal ARE 10% SSP MMP HBIR ARE 20% SSP MMP HBIR ARE 30% SSP MMP HBIR ARE 40% SSP MMP HBIR ARE 50% SSP MMP HBIR ARE 60% SSP MMP HBIR ARE 70% SSP MMP HBIR ARE 80% SSP MMP HBIR ARE 90% SSP MMP HBIR ARE 100% SSP MMP HBIR

12 158 J. Comput. Sci. & Technol., Jan. 2012, Vol.27, No.1 precision and recall values of the simple CBIR and Ideal systems are written on the top row of the table for comparison. In Fig.10, the precision-recall plots of some selected query images are shown for different methods. These values are computed after gathering 30% of users feedbacks. In Figs. 10(a), 10(b), 10(d), 10(f), 10(g) and 10(i), the proposed HBIR system outperforms the other approaches. Especially, in Fig.10(g), the proposed HBIR system has perfectly recognized the query image and acts like an ideal system. This happens because all of the database images constitute an absolute value difference with the query image which lies in the correct non-overlapping relevant and irrelevant HCs. However, in other figures, i.e., Figs.10(c), 10(e), and 10(h), the proposed method is weaker than other approaches. This happens because of incorrect relevant and irrelevant HCs hit, particularly when the number of overlapping HCs are significant. However, these errors will be minimized using the promising individual scoring method by gathering more feedbacks. The MMP approach was more precise than SSP and ARE approaches in most cases, and in some cases, like the one depicted in Fig.10(c), it also outperformed our method. The SSP and ARE approaches treat similarly in most cases. However, their performances were the least in most cases and in some cases like Figs. 10(e) and 10(h), the SSP and ARE were the most powerful approaches, respectively. This shows that in their defined transformation, the relevant and irrelevant images are located far apart in some cases and a good performance is resulted, and in some other cases, they lie in Fig.10. Precision-recall plots of some selected query images for different methods after 30% of gathered feedbacks. (Green: CBIR, Red: ARE, Blue: SSP, Magenta: MMP, Black: HBIR, Cyan: Ideal).

13 Hossein Ajorloo et al.: Hypercube-Based Image Retrieval 159 each other s vicinity and the precision is deteriorated. Also, in any case, one transformation can lead to better results with respect to other defined transformations. 4.3 Comparing the Retrieved Images Figs depict the retrieval results of three sample queries drawn on different RF methods. In the following, we discuss them Experiment 1 In Fig.11(a), the first 48 retrieved images by the ARE method is depicted. Only 22 images are relevant to the query image. This means that in the transformation domain, the relevant and irrelevant images were near. Figs. 11(b) and 11(c) depict the retrieved images using the SSP and MMP approaches, respectively. In this experimentation, the SSP was weaker than ARE and returned only 12 relevant images out of 48 retrieved images, and the MMP approach returned 32 relevant images which is an acceptable result. However, only 2 irrelevant images are returned by the proposed method Experiment 2 In the second experiment (Fig.12), the ARE and MMP approaches treat similarly and better than SSP. However, since only 48 retrieved images are shown, this similarity is not visible in this figure. In this experiment, all 48 retrieved images by ARE and HBIR methods were relevant, but 4 images in SSP and 2 images in MMP methods were irrelevant. Fig.11. Results of Experiment 1: Retrieval results for different methods after gathering 30% of users feedbacks. (a) ARE. (b) SSP. (c) MMP. (d) HBIR. (Images order: row-wise from left to right, i.e., first row is first retrieved, then second row and so on. Note that some images are transposed to fit in the figure.)

14 160 J. Comput. Sci. & Technol., Jan. 2012, Vol.27, No.1 Fig.12. Results of Experiment 2: Retrieval results for different methods after gathering 30% of users feedbacks. (a) ARE. (b) SSP. (c) MMP. (d) HBIR. (Images order: row-wise from left to right, i.e., first row is first retrieved, then second row and so on. Note that some images are transposed to fit in the figure.) Experiment 3 In our third experiment (Fig.13), the MMP was the most precise method with all 48 retrieved images, relevant. But 12 images from ARE, 12 images from SSP, and 16 images from HBIR were irrelevant. The weakness of our method in this experiment was due to the overlaps between HCs, as stated before, which can be reduced by gathering more feedbacks. 5 Limitation and Future Work The main limitation of the proposed HBIR system is its dependency on the low-level features. In the n- dimensional space of feature difference vectors, if the relevant and irrelevant points mix together, the HBIR system creates too many rules. If we prevent the system from this phenomena, the resulting HCs will have a large volume of overlap and hence, the accuracy of the system lowers; otherwise, creating too many rules slows down the system and occupies a large volume of system memory. To overcome this shortcoming, we are going to find appropriate feature vectors which lead to distinct island in the n-dimensional space. This can speed up the system and enhance its accuracy with less rules. 6 Conclusions An HC-based relevance feedback image retrieval system is proposed to bridge the semantic gap of the present CBIR systems. The system learns the user s

15 Hossein Ajorloo et al.: Hypercube-Based Image Retrieval 161 Fig.13. Results of Experiment 3: Retrieval results for different methods after gathering 30% of users feedbacks. (a) ARE. (b) SSP. (c) MMP. (d) HBIR. (Images order: row-wise from left to right, i.e., first row is first retrieved, then second row and so on. Note that some images are transposed to fit in the figure.) semantic and stores them in its rule base using n- dimensional HCs. The main contributions of the paper are on: 1) designing an image retrieval system based on relevance feedback, 2) a training paradigm for making rules and HCs, 3) an HC merging, an HC splitting and a rule scoring methods to enhance the accuracy of the system and speed up it, 4) a long-term relevance feedback inference system, and 5) a short-term RF mechanism to fill the gap of long-term RF system. The experiments on a set of images from the Corel database show the superiority of the proposed HBIR system over some existing short-term and long-term RF systems based on precision and recall parameters. The main advantage of the system is its large capacity in learning the users semantics and ability to resemble an ideal system. Its main shortcoming is the dependency of the number of rules (and hence its speed) and its accuracy on the lowlevel features used which necessitates the use of more sophisticated image features. References [1] Flickner M, Sawhney H et al. Query by image and video content: The QBIC system. Computer, 1995, 28(9): [2] Pentland A, Picard R W, Sclaroff S. Photobook: Contentbased manipulation of image databases. Int. J. Computer Vision, 1996, 18(3): [3] Rui Y, Huang T S, Mehrotra S, Ortega M. Automatic matching tool selection using relevance feedback in MARS. In Proc. the 2nd Int. Conf. Visual Information Systems, Dec. 1997, pp [4] Yoshitaka A, Ichikawa T. A Survey on content-based retrieval

16 162 J. Comput. Sci. & Technol., Jan. 2012, Vol.27, No.1 for multimedia databases. IEEE Transactions on Knowledge and Data Engineering, 1999, 11(1): [5] Smeulders A, Worring M, Santini S, Gupta A, Jain R. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(12): [6] Zhou X, Rui Y, Huang T S. Exploration of Visual Data. Kluwer Academic Publishers, [7] Rocchio J. Relevance feedback in information retrieval. In The SMART System-Experiments in Automatic Document Processing, Salton G (ed.), Prentice Hall, 1971, pp [8] Ciocca G, Schettini R. A relevance feedback mechanism for content-based image retrieval. Information Processing and Management, 1999, 35(6): [9] Rui Y, Huang T S, Ortega M, Mehrotra S. Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 1998, 8(5): [10] Peng J, Bhanu B, Qing S. Probabilistic feature relevance learning for content-based image retrieval. Computer Vision and Image Understanding, 1999, 75(1-2): [11] Meilhac C, Nastar C. Relevance feedback and category search in image database. In Proc. Int. Conf. Multimedia Computing and Systems, June 1999, pp [12] Cox I, Miller M, Minka T, Papathomas T, Yianilos P. The Bayesian image retrieval system, PicHunter: Theory, implementation, and psychophysical experiments. IEEE Transactions on Image Processing, 2000, 9(1): [13] Tong S, Chang E. Support vector machine active learning for image retrieval. In Proc. the 9th ACM Int. Conf. Multimedia (ACM Press), Ottawa, Canada, Sept. 30-Oct. 5, 2001, pp [14] Tieu K, Viola P. Boosting image retrieval. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), June 2000, pp [15] Vasconcelos N, Lippman A. Learning from user feedback in image retrieval systems. In Proc. Neural Information Processing System, [16] Lin Y Y, Liu T L, Chen H T. Semantic manifold learning for image retrieval. In Proc. the 13th ACM Int. Conf. Multimedia (ACM MM), Singapore, Nov. 2005, pp [17] Yu J, Tian Q. Learning image manifolds by semantic subspace projection. In Proc. the 14th ACM Conference on Multimedia (ACM MM), Santa Barbara, USA, Oct. 2006, pp [18] He X F, Cai D, Han J. Learning a maximum margin subspace for image retrieval. IEEE Transactions on Knowledge and Data Engineering, Feb. 2008, 20(2): [19] Lakdashti A, Moin M S. A new content-based image retrieval approach based on pattern orientation histogram. In Proc. the 3rd Int. Conf. Computer Vision/Computer Graphics Collaboration Techniques, March 2007, pp Hossein Ajorloo received two B.Sc. degrees in electrical engineering and computer engineering both from Amirkabir University of Technology, Tehran, Iran, in 2002 and 2005, respectively; and two M.Sc. degrees in telecommunication systems engineering and computer architecture engineering both from Sharif University of Technology, Tehran, in 2004 and 2007, respectively. He is now pursuing his Ph.D. degree in computer architecture at Sharif University of Technology. He had worked as a researcher at the Iran Telecom Research Center (ITRC) from 2002 to 2005 and then he joined the Steady Technology Solutions Company, Tehran, Iran, where he was working as the manager of the Department of Telecommunications from 2005 so far, and has completed various projects. His current research interests include wireless networks, high speed digital communication circuit design, industrial automation, and digital image, video, and speech processing. He has published 4 journal papers, 1 book chapter and 12 conference papers. Abolfazl Lakdashti was born in Sari, Mazandaran, Iran on September 21, He received the B.Sc. degree in computer engineering (software) from Sari University, Mazandaran, Iran, in 1998; the M.Sc. degree in computer engineering (computer architecture) from Research & Science University, Tehran, Iran, in 2002; and the Ph.D. degree in computer engineering (hardware) from Research & Science University, Tehran, Iran in From 1999 to 2003 he worked as a researcher in the Digital Media Lab (DML), Advanced ICT Center (AICTC), Sharif University of Technology, Tehran, Iran. From 2001 to 2006 he worked as a research assistant in ITRC, Tehran, Iran. He is now a professor assistant of the Department of Computer Engineering in Islamic Azed University, Sari Branch. He is the founder and manager of University College of Rouzbahan, Sari from His current research interests include image processing, image/video retrieval, computer vision, multimedia, remote sensing and distributed operating systems. He has published 5 journal papers, 2 book chapters and more than 20 conference papers.