Leveraging Click Completion for Graph-based Image Ranking

Transcription

1 th International Conference on Parallel and Distributed Computing, Applications and Technologies Leveraging Click Completion for Graph-based Image Ranking Xiaohong Qin, Yu He, Jun Wu Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing , China { , , Yingpeng Sang School of Information Science and Technology Sun Yat-Sen University Guangzhou , China Abstract Image ranking is a critical component in the image search systems, and graph-based ranking has become a promising way to enhance the retrieval effectiveness. Leveraging the clickthrough data to facilitate the ranking is one of the current trends. However, the sparse and noisy properties of the click-through data make the exploitation of such resource difficult. To this end, this paper proposes a click completion solution for graphbased image ranking, which consists of two coupled components. The first one is a click completion algorithm to handle the sparseness. Another one is a soft-label graph ranking solution to exploit the completed click-through data noise-tolerantly. We conduct extensive experiments to evaluate the performance of the proposed scheme for image retrieval, in which encouraging results validate the effectiveness of the proposed techniques. I. INTRODUCTION Graph-based ranking has been extensively exploited recently due to its effectiveness in various visual retrieval tasks, such as natural image search [1], shape retrieval [2], cross-media retrieval [3], 3D object retrieval [4], etc. Unlike traditional ranking approaches that consider only pairwise similarity for visual ranking, graph-based methods aim at exploring the intrinsic manifold structure collectively hidden in images, hoping to refine the similarity measure. Despite these successes, the performance of graph-based ranking approaches is still limited by the so-called semantic gap existing between low-level image pixels captured by machines and high-level semantic concepts perceived by humans, especially when the visual targets are dispersed in the feature space. In order to boost the performance of image retrieval and overcome the semantic gap, a relevance feedback mechanism [5] is incorporated into the graph-based ranking framework [1], [6], [7], which encourages the user to label a few images returned as either positive or negative in terms of whether they are relevant to user s query or not. The labeled instances is then used to refine the ranking model towards the user s query intends. Nonetheless, it is not easy to obtain sufficient and explicit user feedback as users are often reluctant to provide enough feedback to search engines. It is noted that search engines can record queries issued by users and the corresponding clicked images. Although the clicked images cannot reflect the explicit user preference on relevance of particular query-image pairs, they statistically indicate the implicit relationship between individual images in the ranked list and the given query [8]. Therefore, we can regard clickthrough data as the implicit user feedback based on an assumption that, in a same query session, most clicked images are relevant to the given query, and the reliability of this assumption has been validated by [9]. We consider a particular ranking scenario where the clickthrough data is incomplete and inaccurate, which makes the exploration of such click-through data difficult. Concretely, incompleteness may lead to an underfitting ranker, while inaccuracy may mislead the ranker. To this end, we propose a Click completion solution for Graph-based Visual Ranking framework, CGVR for short. As shown in Fig. 1, our framework first builds a neighbor graph on the image collection to be ranked, then completes the user-image matrix using a collaborative filter with the visual side-information of the image graph, and lastly applies a soft-label graph ranker to rank the images based on the completed click-through data and the neighborhood graph. The rest of this paper is organized as follows. Section 2 reviews the related works. Section 3 presents the proposed CGVR method. Section 4 reports on the experiments. Finally, section 5 concludes this paper and raises the problem for future works. II. RELATED WORK In this section, we survey related works on graph-based visual ranking and the use of click-through data/implicit feedback. We do not survey the online relevance feedback mechanisms [5] as our method requires no user intervention once the query has been issued. This is motivated by empirical evidence suggesting that few users are willing to perform any form of feedback to improve their search results. A. Graph-based visual ranking Graph-based visual ranking has been extensively studied in the multimedia retrieval area. Its main idea is to describe the dataset as a graph and then decide the importance of each vertex based on local or global structure drawn from the graph. One canonical graph-based ranking technique is the Manifold Ranking (MR) algorithm [10], and He et al.[1] first applied MR to image retrieval. Its limitations are addressed by latter research efforts. For example, Wang et al. [11] improved the /16 $ IEEE DOI /PDCAT

2 Fig. 1. The flowchart of the proposed CGVR framework for image retrieval. Our approach regards the click-through data as a user-image matrix, and applies a click completer to produce a denser user-image matrix, which is taken as the pseudo labels of database images and then cast into a graph ranker to output a ranking list of database images according to the users query. MR accuracy using a k-regular nearest neighbor graph that minimizes the sum of edge weights and balances the edges in the graph as well. Wu et al. [6] proposed a self-immunizing MR algorithm that uses an elastic knn graph to exploit unlabeled images safely. Xu et al. [12] proposed an efficient MR solution based on scalable graph structure to handle largescale image datasets. Wu et al. [7] proposed a multi-view MR method to deal with the scenario that images appear in multiple modalities, such as visual features, surrounding textual features, and query log features, etc. B. Learning with click-through data As a surrogate for relevance feedback, click-through data, as a kind of implicit feedback, has been investigated by researchers. Some researches along this direction are designed for the query-by-example (QBE) paradigm, while others serve for the query-by-keyword (QBK) paradigm. In the first category, both the online feedbacks collected from the current user and the historical feedbacks fulfilled by different users are taken into consideration through a uniform learning framework, so called collaborative image retrieval (CIR). For example, Yin et al. [13] exploited the long-term experiences to select the optimal online ranker from a set of candidates based on reinforcement learning. Hoi et al. [14] regarded the query log as the side information, and then, taking that as constraints, learned a distance metric form a mixture of labeled and unlabeled images. Su et al. [15] suggested discovering the navigation patterns from query logs, and using the patterns to facilitate new searching tasks. Wu et al. [16] proposed a hybrid similarity measure that preserves both visual and semantic resemblance by incorporating shortterm with long-term feedback experiences. Zhang et al. [17] aimed at learning a semantic subspace from similar and dissimilar pairwise constraints generated from query logs. In the second category, an initial ranking list of images is acquired by QBK paradigm, and then reordered using visual information in conjunction with query log, which belongs to the family of the image reranking techniques [18]. For instance, Jain and Varma [19] employed Gaussian Process regression to predict the normalized click count for each image, and combines it with the original ranking score. Zhang et al. [8] leveraged multiple kernels learning to adaptively learn the query-dependent fusion weights for multiple modalities of image collection based on click-through data for identifying user search intention. Pan [20] suggested learning a common latent subspace, which allows direct comparison of text queries and images, by jointly integrating cross-view learning with click-through data. Yu et al. [21] proposed a ranking model that integrates visual features and click-through features through a hypergraph regularizer. III. THE PROPOSED CGVR METHOD Our CGVR method is developed based on two intuitions. At first, a good visual ranker should be able to exploit the implicit feedback (i.e. click-through data) rather than the explicit feedback for the purpose of alleviating the user s labeling burden. Furthermore, the ranker should be able to handle the sparse and noisy properties of the click-through data. A. Preliminaries Let X = {x i,i = 1,,n} denote the image dataset, where each x i R d is a visual feature vector. To discover the geometrical subacute (manifold), we build a neighborhood graph on X, and define W R n n + as the corresponding the 156

3 affinity matrix with element W ij saving the weight of edge between x i and x j. Normally the weight is calculated using a Gaussian kernel ( ) W ij =exp d2 (x i, x j ) σ 2 (1) if i N k (j) or j N k (i), otherwise W ij =0, where N k (i) denotes a k nearest neighbor set of image i. Typically, k is a small number (e.g. a small fraction of n), d(a, b) is a distance metric between two vectors a and b (suggested by [1], L1 distance is considered), and σ is the bandwidth parameter that can be tuned by local scaling technique, the effectiveness of which has been verified in the clustering [22] and ranking [6] tasks. To some degree, the weight W ij can be deemed as the visual similarity between image i and j. The click-through data is represented by a user-image matrix R {0, 1} m n whose rows correspond to users and columns correspond to images, illustrated by Fig. 1. If an image is clicked by a user in a query session, the corresponding cell is assigned to value 1, i.e. R ij = 1, otherwise R ij =0. B. CCV: Click completion with visual side-information Inspired by recommender systems [23] 1, we refer to the problem of click completion as Collaborative Filtering (CF), and more precisely as the item-based CF, which is to predict a user s click on a target image by aggregating the clicks that a few similar images have previously received from that user. Usually, the K nearest neighbors (KNN), the K images with highest similarities to the target image, are selected, and the clicks fulfilled by the given user on those neighboring images are aggregated in order to generate a predicted click for the given user on the target image. Following the notations in Section 3.1, we can formulate the predicted click of user i to image j as follow ˆR ij = Sim(j, k)r ik (2) k Z j where Z j is the set of k neighboring images of image j, and Sim(j, k) designates the similarity between image j and image k. To elevate the accuracy of KNN estimation, we calculate the similarity between two images a and b based on both X and R Sim(a, b) =αsim X (a, b)+(1 α)sim R (a, b) (3) where 0 <α<1 is an adjusting factor, Sim X (a, b) is the visual similarity between image a and image b, which can be calculate using Eq. (1), and Sim R (a, b) is the clicking similarity between image a and image b, which is defined by the Jacquard Coefficient of two clicking vectors 2 Sim R (a, b) = R a R b R a R b 1 The user-image matrix used in this work is very similar to the user-item matrix used in recommender systems. 2 A clicking vector of one image is actually a column of the user-image matrix shown in Fig. 1. (4) where R a {0, 1} m is a column vector of R, which records the clicks fulfilled by different users on image a, and denotes the size of a set. C. GRS: Graph ranking with soft-label Our method is developed based on MR [10], and we slightly modify it for noise-tolerant purpose. Let r : X R be a ranking function that assigns to each image instance x i a ranking score r i. We can view r as a vector r =[r 1,,r n ] T. Also, we define a label vector y =[y 1,,y n ] T, in which y i =1if x i is a query, and y i =0otherwise. The cost function associated with r is defined to be ( ) 2 Q(r) = 1 n r i W ij r j + λ 2 Dii Djj 2 r y 2 (5) i,j=1 where λ is a regularization parameter, and D is a diagonal matrix with D ii = n j=1 W ij. The first term in the cost function is a smoothness constraint, which make the nearby examples in visual space having close ranking scores. The second term is a fitting constraint, which makes the ranking result fitting to the label assignment. By differentiating Q(r) and set it to zero, we can easily get the following closed solution r =(I μs) 1 y (6) where μ =1/(1 + λ), I is an identity matrix, and S is the symmetrical normalization of D, i.e. S = D 1/2 WD 1/2. (7) We can directly use the above closed form solution to compute the ranking scores of examples. However, in large scale problems, we prefer to use the iteration solution r(t +1)=μSr(t)+(1 μ)y. (8) As illustrated by Eq. (6) and (8), it is noted that the label vector plays an important role in image ranking, and a dense y is desirable to derive r. In the regular MR [10], only one labeled instance (i.e. the user s query) is concerned, which is hardly to achieve satisfactory ranking result. A few works, such as [1], [24], [6], and [7], etc., take the online relevance feedback mechanism into consideration for the label vector enrichment, but it is unpractical as mentioned before. Different from previous studies, our idea is to enrich the label vector using the click-through data without any user intervention. The intuition behind our idea is that, when two images are clicked in a same query session, they often share a certain similar meaning and we expect different users to express similar judgments on them. Based on this assumption, the (hidden) correlation between any two images can be inferred by analyzing the judgements (clicks) made by different users on them. Given the user s query q, the correlation of it to a database image i, i.e. Sim R (q, i), can be calculated by Eq. (4). Intuitively, we can directly predict y i =1if Sim R (q, i) is highly positive. Although this idea can be straightly handled by 157

4 MR, it may suffer from performance degradation as erroneous click-through data. In particular, in our scheme, more noises may be introduced by CF. To this end, we treat the labels in two different ways for the fault-tolerance purpose. In details, the suer s query is treated as the hard-labeled instance, while the images predicted by analyzing the query log are treated as soft-labeled instances, i.e. y i = Sim(q, i) [0, 1], where the magnitude of a soft label represents the confidence of the assigned pseudo label. D. Algorithmic framework So far, we can assemble the proposed CCV and GRS methods into a image retrieval framework, which ranks the database images with respect to the user s query based on visual features and click-through data. We outline this algorithmic framework in below. 1) Build a neighborhood graph on X, and compute the corresponding affinity matrix W by Eq. (1) and the normalized one S by Eq. (7); 2) Compute the completed user-image matrix ˆR based on R and S by Eq. (2); 3) Compute the soft-label vector y based on q and ˆR by Eq. (4); 4) Compute the ranking-score vector r based on S and y by Eq. (6) or (8); Note that the step 1 and step 2 can be implemented offline, and therefore CGVR can be quite efficient in processing online queries. Our approach is mainly to process the in-sample queries, but it can be easily extended to the out-of-sample queries. For example, when a completely new query arrives, we can apply a mechanism named one-click query expansion [25] to transform the out-of-sample query into a in-sample query with very little user effort. IV. EXPERIMENTS A. Experimental Setup We employ the 10K Images dataset 3 which is publicly available on the web to make our experiments reproducible. The images are from 100 semantic categories, with 100 images per category. Three kinds of visual features are extracted to represent the images, including a 64-dimensional color histogram, an 18-dimensional wavelet-based texture and a 5- dimensional edge direction histogram [16]. The log dataset consists of 1000 query sessions which are simulated based on the ground truth of image dataset. The average number of clicks in each query session is 20. Further, according to the empirical study conducted by [9], we add 12% noise into the log dataset to approach the real-world search scenario 4 Our CGVR method is compared with two existing approaches which are introduced as follows. 3 The dataset was firstly used in [16]. 4 Experimental result reported by [9] showed that the satisfaction rate of the click-through data is 88.1%, and we use a slightly higher noise level in our experiments. TABLE I TOP 10 UNCLICKED IMAGE RECOMMENDATION. α Precision Recall F TABLE II TOP K UNCLICKED IMAGE RECOMMENDATION. K Precision Recall F Self-Tuning Manifold Ranking (STMR): the regular MR method [1] with self-tuned bandwidth parameter [22]. Hybrid Similarity learning (HySim): a CIR method learning hybrid similarity measure using both explicit and implicit feedbacks [16]. In addition, a degenerated version of CGVR, termed as CGVR-Deg, is also included in comparisons to study whether the CCV solution used by our scheme is effective or not. CGVR-Deg is almost the same with CGVR, except that the former omits the CCV process and directly employs the original click-through data. A query set with 200 image examples is randomly selected to evaluate the average performance of all compared methods. We use the precision-recall-graph (PR-graph), Mean Average Precision (MAP) and the precision at top N retrieval results (P@N) for performance evaluation. B. Performance Evaluation At first, we conduct model selection for our CGVR method, and there are two parameters in total: α used in Eq. (3) and μ used in Eq. (8). For convenience, μ is fixed at 0.01, consistent with the previous experiences [10], [1], [12], [7]. Hence we only need to tune parameter α. Table I summarizes the performance of CCV over different α values, where the best performance has been boldfaced. Based on that, we fix α at 0.7. As illustrated by Table I, although the precision is high, the recall is relatively low. So we further evaluate CCV over different K values (of the TopK recommendation) in order to elevate the recall, described by Table II. We observe that the best F1 value is achieved when K =30, but the highest precision is still attained when K =10. Conservatively, we then evaluate our CGVR method based on two completed log matrices attained when K =10(with highest precision) and K = 30 (with highest F1 value), respectively. Figure 2 print the PR graphs and P@N of three methods, i.e. CGVR-Top10, CGVR-Top30 and CGVR- Deg. By examining all methods, we observe that both CGVR- Top10 and CGVR-Top30 outperform CGVR-Deg, which verifies the effectiveness of the CCV solution used by CGVR. 158

5 (a) PR-graphs (b) curves Fig. 2. The performance of our CGVR method evaluated on different log datasets. Further, by comparing CGVR-Top10 and CGVR-Top30, their performance curves are very similar to each other. Based on this observation, we prefer to Top 10 unclicked images recommendation for the computational efficiency. Furthermore, we compare our CGVR method with MR and HySim, and the performance curves are plotted by Figure 3. By examining all methods, we observe that the performance curves of CGVR and MR are much better than HySim, which verifies the superiority of the graph ranking used by our method in comparison to the pairwise similarity ranking. Also, by comparing CGVR and MR, we find that CGVR outperforms MR significantly, which confirms that leveraging click-through data is beneficial and essential to graph ranking. The performance of HySim is not as good as those reported in [16], which is because when no online feedbacks are provided, HySim is equivalent to a L1 distance based similarity ranking method and therefore its performance degenerates. At last, we compare the three methods again by introducing the online relevance feedback mechanism. For each query, only one round of feedback is performed (the top ten returned images are labeled), since users have no patience to do more. The retrieval performance comparisons are plotted in Figure 4. As expected, the performances of all methods are boosted under the action of online feedbacks, and HySim receives higher improvement than any other methods. It is impressive that the performance of CGVR is still the best after introducing online feedbacks. V. CONCLUSIONS This paper presents a novel CGVR method that aims at noise-resistantly leveraging the click-through data to boost graph-based visual ranking. We first propose a CCV algorithm to enrich the click-through data, and then developing a GRS technique for noise-tolerant image ranking. Experimental s- tudy has validated the superiority of the proposed method in comparison to several existing similarity ranking approaches. To our best knowledge, not much has been known on investigating the click completion problem. In this paper, we tentatively address that using a basic collaborative(cf) technique solution. In the future, we will study more sophisicated CF-like solution to further enhance the graph-based image ranking task. ACKNOWLEDGMENT The authors would like to thank the anonymous reviewers for their constructive suggestions. Jun Wu is the corresponding author of this paper. This work was supported in part by the Natural Science Foundation of China ( , and ), the Fundamental Research Funds for the Central Universities (2015JBM029), and the Research Foundation for Talents of Beijing Jiaotong University (2015RC008). REFERENCES [1] J. He, M. Li, H. Zhang, H. Tong, and C. S. Zhang, Manifold-ranking based image retrieval, in ACM Multimedia, 2004, pp [2] X. Bai, B. Wang, C. Yao, W. Liu, and Z. Tu, Co-transduction for shape retrieval, IEEE Trans. Image Process., vol. 21, no. 5, pp , [3] Y. Yang, D. Xu, F. Nie, J. Luo, and Y. Zhuang, Ranking with local regression and global alignment for cross media retrieval, in ACM Multimedia, 2009, pp [4] S. Bai and X. Bai, Sparse contextual activation for efficient visual reranking, IEEE Trans. Image Process., vol. 25, no. 3, pp , [5] X. S. Zhou and T. S. Huang, Relevance feedback in image retrieval: A comprehensive review, Multimedia Syst., vol. 8, no. 6, pp , [6] J. Wu, Y. Li, S. Feng, and H. Shen, Self-immunizing manifold ranking for image retrieval, in PAKDD, ser. LNAI, vol. 7819, 2013, pp [7] J. Wu, J. Yuan, and J. Luo, Robust multi-view manifold ranking for image retrieval, in PAKDD, ser. LNAI, vol. 9652, 2016, pp [8] Y. Zhang, X. Yang, and T. Mei, Image search reranking with querydependent click-based relevance feedback, IEEE Trans. Image Process., vol. 23, no. 10, pp , [9] G. Smith, C. Brien, and H. Ashman, Evaluating implicit judgments from image search clickthrough data, J. Am. Soc. Inf. Sci. Tec., vol. 63, no. 12, pp , [10] D. Zhou, J. Weston, A. Gretton, O. Bousquet, and B. Scholkopf, Ranking on data manifolds, in NIPS, 2003, pp

6 (a) PR-graphs (b) curves Fig. 3. The performance of our CGVR method compared with several existing approaches using only queries. (a) PR-graphs (b) curves Fig. 4. The performance of our CGVR method compared with several existing approaches after one round of feedback. [11] B. Wang, P. F., K. Hu, and J. Paul, Manifold-ranking based retrieval using k-regular nearest neighbor graph, Pattern Recogn., vol. 45, no. 4, pp , [12] B. Xu, J. Bu, C. Chen, C. Wang, D. Cai, and X. He, Emr: A efficient manifold ranking model for content-based image retrieval, IEEE Trans. Knowl. Data Eng., vol. 27, no. 1, pp , [13] P.-Y. Yin, B. Bhanu, C. K.-C., and A. Dong, Integrating relevance feedback techniques for image retrieval using reinforcement learning, IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10, pp , [14] S. Hoi, W. Liu, and S.-F. Chang, Semi-supervised distance metric learning for collaborative image retrieval, in CVPR, 2008, pp [15] J. Su, W. Huang, P. Yu, and V. Tseng, Efficient relevance feedback for content-based image retrieval by mining user navigation patterns, IEEE Trans. Knowl. Data Eng., vol. 23, no. 3, pp , [16] J. Wu, H. Shen, Y.-D. Li, Z.-B. Xiao, M.-Y. Lu, and C.-L. Wang, Learning a hybrid similarity measure for image retrieval, Pattern Recogn., vol. 46, no. 11, pp , [17] L. Zhang, H. P. H. Shum, and L. Shao, IEEE Trans. Image Process., vol. 25, no. 3, pp , [18] T. Mei, Y. Rui, S. Li, and Q. Tian, Multimedia search reranking: A literature survey, ACM Comput, Surv., vol. 46, no. 3, p. 38, [19] V. Jain and M. Varma, Learning to re-rank: Query-dependent image re-ranking using click data, in WWW, 2011, pp [20] Y. Pan, T. Yao, T. Mei, H. Li, C.-W. Ngo, and Y. Rui, Click-throughbased cross-view learning for image search, in ACM SIGIR, 2014, pp [21] J. Yu, D. Tao, M. Wang, and Y. Rui, Learning to rank using user clicks and visual features for image retrieval, IEEE Trans. Cybern., vol. 45, no. 4, pp , [22] L. Zelnik-Manor and P. Perona, Self-tuning spectral clustering, in NIPS, 2004, pp [23] F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor, Recommender Systems Handbook. Springer, [24] J. He, M. Li, H. Zhang, H. Tong, and C. Zhang, Generalized manifoldranking-based image retrieval, IEEE Trans. Image Process., vol. 15, no. 10, pp , [25] X. Tang, K. Liu, J. Cui, F. Wen, and X. Wang, Intentsearch: Capturing user intention for one-click internet image search, IEEE Trans Pattern Anal.Mach. Intell., vol. 34, no. 7, pp ,