Scalable search-based image annotation

Transcription

1 Multiedia Systes DOI 17/s y REGULAR PAPER Scalable search-based iage annotation Changhu Wang Feng Jing Lei Zhang Hong-Jiang Zhang Received: 18 October 27 / Accepted: 15 May 28 Springer-Verlag 28 Abstract With the popularity of digital caeras, ore and ore people have accuulated considerable digital iages on their personal devices. As a result, there are increasing needs to effectively search these personal iages. Autoatic iage annotation ay serve the goal, for the annotated keywords could facilitate the search processes. Although any iage annotation ethods have been proposed in recent years, their effectiveness on arbitrary personal iages is constrained by their liited scalability, i.e. liited lexicon of sall-scale training set. To be scalable, we propose a search-based iage annotation algorith that is analogous to inforation retrieval. First, content-based iage retrieval technology is used to retrieve a set of visually siilar iages fro a large-scale Web iage set. Second, a text-based keyword search technique is used to obtain a ranked list of can- Counicated by E. Chang. C. Wang (B) Departent of Electronic Engineering and Inforation Science, University of Science and Technology of China, 2327 Hefei, China e-ail: wch@ustc.edu F. Jing Beijing New Sunlight Technologies Co. Ltd, Roo 114, D Building of the place No. 9 of Guanghua Rd, Chaoyang District, 12 Beijing, China e-ail: scenery.jf@gail.co L. Zhang Microsoft Research Asia, 49 Zhichun Road, 119 Beijing, China e-ail: leizhang@icrosoft.co H.-J. Zhang Microsoft Advanced Technology Center, 49 Zhichun Road, 119 Beijing, China e-ail: hjzhang@icrosoft.co didate annotations for each retrieved iage. Third, a fusion algorith is used to cobine the ranked lists into a final candidate annotation list. Finally, the candidate annotations are re-ranked using Rando Walk with Restarts and only the top ones are reserved as the final annotations. The application of both efficient search techniques and Web-scale iage set guarantees the scalability of the proposed algorith. Moreover, we provide an annotation rejection schee to point out the iages that our annotation syste cannot handle well. Experiental results on U. Washington dataset show not only the effectiveness and efficiency of the proposed algorith but also the advantage of iage retrieval using annotation results over that using visual features. 1 Introduction With the popularity of digital caeras, ore and ore people have considerable digital iages on their personal devices. How to effectively index and search these personal iages eerges as a crucial issue. Unlike Web iages that have rich etadata such as filenae, ALT text, URL and surrounding text for indexing and searching, personal iages have little textual inforation. A possible solution is to index and search iages with visual inforation, i.e. content-based iage retrieval (CBIR) [26]. Although CBIR has been extensively studied for ore than a decade, it has three liitations that liit its practicability. First, due to the so-called seantic gap between low level visual features and high level seantic concepts, the precision of CBIR is usually unsatisfactory. Second, due to the high diensionality of visual features and the curse of diensionality, the efficiency and scalability of CBIR are usually low. Finally, the query for of CBIR is unnatural for personal iage search. On the one hand, for query by exaple (QBE), the exaple iage is often absent.

2 C. Wang et al. On the other hand, query by sketch (QBS) is too coplex and adds too uch burden to noral users. To resolve the aforeentioned issues, a straightforward solution is to anually annotate the iages and search the with annotated keywords. However, anually annotating large quantity of iages is too tedious and tie-consuing for coon users. To solve this proble, any autoatic iage annotation ethods have been proposed in recent years. Most of the existing iage annotation approaches can be classified into two categories, i.e. classification-based ethods and probabilistic odeling-based ethods. The classification-based ethods use iage classifiers to represent annotation keywords (concepts). The classifier of a keyword (concept) is trained to separate the training iages with the keyword (concept) fro other keywords (concepts). For a new iage, the outputs of the classifiers will be used to annotate it. Many representative classifiers have been used, such as the two-diensional ulti-resolution hidden Markov odels (2D MHMMs) [19], support vector achine (SVM) [9,14,33], Bayes Point Machine [8], and Mixture Hierarchical Model [6,7]. Since each annotation keyword (concept) has a classifier and all the classifiers should be tested to annotate an iage, the classification-based ethods are unsuitable for iage dataset with unliited lexicon, e.g. personal iage sets. The probabilistic odeling-based ethods attept to infer the correlations or joint probabilities between iages and annotation keywords. As the pioneer work, Mori et al. [23] proposed a ethod for annotating iage grids using co-occurrences in Although the odel is siple, the annotation perforance is relatively low. Another way of capturing co-occurrence inforation is to introduce latent variables that link iage features with keywords. Duygulu et al. [1] proposed a novel approach that treated iage annotation as a achine translation proble. A statistic achine translation odel was used to translate the keywords of an iage to the blob tokens obtained by clustering. The use of EM algorith in the odel constrains its scalability to large iage collections. Other representative work includes the Gaussian Mixture Model, the Latent Dirichlet Allocation Model (LDA) and the correspondence LDA [4]. In spite of the profound statistic odel and strict deduction, the paraetric probabilities used in the odel are too siple to odel real distributions and the paraeter estiation process is too coplex to be scalable. Inspired by the relevance language odels, several relevance odels have been proposed recently, such as, the Cross-Media Relevance Model (CMRM) [15], the Continuous Relevance Model (CRM) [16, 18], and the Multiple Bernoulli Relevance Model (MBRM) [12]. Although these odels are shown to be ore effective than the previous ones, their dependency on sall-scale high quality training set restricts their effectiveness on arbitrary personal iages. That is, only iages that are consistent with the training iages could be annotated with keywords in a liited vocabulary. With the prosperity of the Web, it has becoe a huge deposit of alost all kinds of data. Several probles that were believed to be unsolvable have been successfully solved by leveraging the rich inforation of the Web [11,34]. Copared with the liited nuber of concepts that can be odeled using a relatively sall-scale iage training set, the potentially unliited vocabulary of a Web-scale iage database can be utilized to annotate iages. Motivated by Web search technologies in any coercial systes, we have proposed the AnnoSearch syste [31]. Assuing that an accurate keyword of the iage to be annotated is available, the keyword is used to retrieve several seantically relevant iages. Then, the resulting iages are further re-ranked based on visual features. Finally, a search result clustering (SRC) technique [35] is used to ine the final annotations fro the top iages. Although the initial keyword ight speed up the search process and enhance the relevance of the retrieved iages with the iage to be annotated, it is not always available, especially for the personal iages. Moreover, since SRC was originally designed for general Web search, it ay not be an optial solution to annotation ining. In this paper, we focus on annotation of large personal iage collections. A scalable search-based iage annotation (SBIA) algorith is proposed, which is analogous to inforation retrieval. The algorith basically has four steps. First, CBIR technique is used to retrieve a set of visually siilar iages fro a large-scale Web iage dataset. Second, a text-based keyword search technique is used to obtain a ranked list of candidate annotations for each retrieved iage. Third, a fusion algorith is used to cobine the ranked lists into a final candidate annotation list. Finally, the candidate annotations are re-ranked using Rando Walk with Restarts (RWR) and only the top ones are reserved as the final annotations. The application of both efficient search technologies and Web-scale iage set guarantees the scalability of the proposed algorith. Siilar to [31], the proposed approach enables annotating with alost unliited vocabulary, which is ipossible for ost existing ethods. To be capable of annotating personal iages with little textual inforation, we reove the assuption of an initial keyword and eploy an efficient indexing technique to speed up the search. Furtherore, instead of ining annotations with SRC, we consider this process as a text-based keyword search proble. Instead of treating the retrieved iages equally as in [31], the iages are treated differently according to their visual siilarities to the query iage. Moreover, we also provide an annotation rejection schee to point out the iages that our annotation syste cannot handle well. The scalability of the proposed fraework is reflected in the following aspects. First, the syste leverages a Webscale training set (illions of training iages), which could

3 Scalable search-based iage annotation not be efficiently handled by traditional learning based annotation ethods. Second, the Web-scale training set provides an unliited vocabulary, which is very different fro the sall scale lexicon used by traditional ethods. Third, by leveraging the Web-scale iage and keyword set, the syste has the potential to annotate arbitrary query iage. Fourth, the syste could annotate iages in real tie. Based on the proposed fraework, an online iage annotation service has been deployed. A desktop iage search syste could benefit fro such a service. When indexing desktop iages, the syste can subit iages (or features) one by one to the annotation service and receive autoated annotation results in the background. All the annotation processes are conducted on the server side and do not add any burden to the desktop syste. Then the annotated iages are indexed according to their annotations, and users can easily search iages on desktop by keywords. The rest of the paper is organized as follows. Section 2 presents our otivation and the search view of iage annotation proble. The proposed iage annotation algorith is described in Sect. 3. Experiental results are shown in Sect. 4. We conclude the paper and discuss future work in Sect. 5. The ain results in this paper were first presented in [3]. 2 Iage annotation in the search view 2.1 Motivation Large scale personal iage collections have several special properties, which ake iage anageent and annotation even challenging. First, unlike Web iages, personal iages usually have little text inforation and therefore little training data is available to learn nuerous iage concepts. Second, the visual contents of personal iages in one collection are too diverse, aking it difficult to odel various concepts. Third, since there are usually a large nuber of iages in one s collection, to efficiently anage these iages, all of the need to be autoatically annotated first. The first two properties ake traditional annotation ethods not appropriate in this special environent, and otivate us to seek for other solutions to annotate this kind of iages. The third property necessitates the efficiency of the algorith. Due to the lack of training data in personal iage collections, we need to leverage training data outside personal collections. And to be capable of annotating diverse personal iages, the training set has to be diverse enough, which eans that a large scale training set is crucial. Therefore, leveraging large scale Web iages ay be a good solution, and they can be considered as a low quality and diverse training set. Motivated by text-based search technologies in any coercial systes, we forulate iage annotation as a search proble. That is, given a query iage, how to search for accurate annotation keywords fro a Web-scale training set. Fro this perspective, we further find that search-based iage annotation and inforation retrieval can be treated as dual probles, and therefore we can systeatically leverage those ature inforation retrieval technologies. In the following section we will discuss the correspondences between the two probles. 2.2 Correspondences Inforation retrieval A typical inforation retrieval (IR) syste can be siplified as Fig. 1. Given a text query Bayesian introduction, the syste first locates the inverted lists for each query ter, and then finds relevant docuents that contain all the query ters. If the user is not satisfied with the results, he/she ay change the query, e.g. Bayesian tutorial, and search again to get ore results. Or, he/she can odify his/her query to Bayesian introduction 1 to find ore results in one search session. However, in ost cases, latter searches are ignored, because search results for the original query, e.g. Bayesian introduction, are usually sufficient. Let is assued that the retrieval syste always perfors query synony expansion as a general case. The reaining proble is how to rank the found relevant docuents. Let Dist(t) denote the distance between the original query ter and the synony ter t, which can be easured based on word siilarity as in WordNet [22]. For each query ter t (or synony ter) in a docuent doc_a, a ter frequency inverse docuent frequency (tf-idf) [3] score Score tfidf (doc_a, t) is used to easure the relevance of the docuent to the ter. In a siple case, the relevance score of the docuent can be obtained by suing up all ters tf-idf scores related to that docuent, weighted by a function of Dist(t). The relevance score of docuent doc_a can be calculated as follows: Score relevance (doc_a) = f (Dist(t)) t S Score tfidf (doc_a, t), (1) where S is the set of query ters and synony ters. Although we could rank the docuents according to their relevance scores and return top ones as the search results, the search results could be further refined using pseudorelevance feedback technology [32]. The basic idea of pseudo-relevance feedback is to extract expansion ters fro the top-ranked docuents to forulate a new query. Through 1 is the synony operator used by Google. introduction is siilar to introduction OR tutorial OR FAQ.

4 C. Wang et al. Bayesian ~introduction Synony expansion Inverted doc list Bayesian introduction Dist(t1) doc_a doc_b doc_c Bayesian tutorial Dist(t 2) Bayesian FAQ Dist(t3) Bayesian overview Dist(t4) doc_b doc_e doc_h doc_d doc_f doc_i doc_e doc_g doc_k Doc ranking by TFIDF Final doc list Candidate doc list doc_a doc_e doc_b doc_f doc_c doc_d doc_i doc_h doc_k doc_g Pseudo relevance feedback doc_b doc_e doc_k doc_a doc_c doc_d doc_i doc_h doc_f doc_g Doc erging doc_h doc_i doc_k doc_e doc_f doc_g doc_b doc_d doc_e doc_a doc_b doc_c Fig. 1 Fraework of inforation retrieval CBIR Dist(i1) Text description word_a word_b word_c (ifikf)... (ifikf) (ifikf) Final annotation list Candidate annotation list Dist(i2) Dist(i 3) Dist(i4) word_b word_d word_e (ifikf) (ifikf) (ifikf)... word_e word_f word_g (ifikf) (ifikf) (ifikf)... word_h word_i word_k (ifikf) (ifikf) (ifikf)... Keyword ranking by IFIKF word_a word_e word_b word_f word_c word_d word_i word_h word_k word_g Annotation refineent word_b word_e word_k word_a word_c word_d word_i word_h word_f word_g Keyword erging word_h word_i word_k word_e word_f word_g word_b word_d word_e word_a word_b word_c Fig. 2 Fraework of search-based iage annotation query expansion, soe relevant docuents issed in the initial round can possibly be retrieved and ore relevant docuents could be ranked higher. Pseudo-relevance feedback can be considered as a score refineent process as follows: Score(doc_a) = Ref ine(score relevance (doc_a)). (2) Based on the refined scores for each relevant docuent, we can rank all the relevant docuents and return top ones as the search results to users Search-based iage annotation We re-forulate iage annotation as a search process analogous to the aforeentioned inforation retrieval. The searchbased iage annotation (SBIA) fraework is shown in Fig. 2. Coparing Figs. 2 with 1, we can see that the iages in iage annotation correspond to the textual ters in inforation retrieval, while the annotation keywords in iage annotation correspond to the docuents in inforation retrieval. Given a target iage as the query, our ai is to find the ost

5 Scalable search-based iage annotation related keywords. Siilar to the synonys finding in inforation retrieval, CBIR technique is used to retrieve the ost siilar iages to the target iage fro a large-scale Web iage set. Assue that there are s siilar iages {I i } i=1,...,s. The distance between the target iage and a siilar iage I i is denoted as Dist(i). Since all the siilar iages are Web iages with textual descriptions, these descriptions could be considered as inverted files. For a keyword appearing in the description of an iage, an iage frequency inverse keyword frequency (if-ikf) easure siilar to the tf-idf easure could also be defined to reflect the relevance between the iage and keyword. We will give ore details in Sect Then, the relevance score of a keyword to the target iage is a weighted su of the relevance scores corresponding to the siilar iages. More specifically, the relevance score of akeywordword_a can be calculated as follows: Score relevance (word_a) = f (Dist(i)) I i S Score ifikf (word_a, I i ), (3) where S is the set of siilar iages of the target iage. Analogous to the pseudo-relevance feedback process of inforation retrieval, a novel approach to autoatically refine the candidate annotations of iages is proposed. We reforulate the iage annotation refineent process as a graph ranking proble and solve it with the Rando Walk with Restarts (RWR) algorith. We will give ore details in Sect Siilar to Eq. 2, the refined score of keyword word_a could be calculated as follows: Score(word_a) = Ref ine(score relevance (word_a)) (4) 3 Search-based iage annotation There are four ain coponents of the proposed SBIA algorith: a CBIR stage to retrieve visually siilar iages, a text-based keyword search stage to obtain a ranked list of candidate annotations for each retrieved iage, a fusion stage to cobine the ranked lists into the final candidate annotation list, and an annotation refineent stage to autoatically refine the candidate annotations. We also introduce the proposed annotation rejection ethod at the end of this part. 3.1 Content-based iage retrieval About 2.4 illion iages were collected fro several photo foru sites, e.g. photosig [2]. Iages of such sites have rich textual inforation, such as title and photographer s description. The textual inforation reflects the content of corresponding iage to soe extent. For details of the Web-scale dataset please refer to [36]. For a target iage I t, a typical CBIR technique is used to retrieve a set of visually siilar iages denoted by S. To represent an iage, a 64-diensional feature [37] was extracted. It is a cobination of three features: 6 diensional color oents, 44 diensional banded auto-correlogra and 14 diensional color texture oents. For color oents, the first two oents fro each channel of CIE-LUV color space were extracted. For correlogra, the HSV color space with inhoogeneous quantization into 44 colors is adopted. Note that all existing global or local features and the corresponding distance easures could be used by the algorith. To speed up the siilarity search process, a K-eans-based indexing algorith is used [13]. The rationale of clustering-based high-diensional indexing techniques is to reduce the search space by first finding a few cluster blocks whose centers are nearest to the query point, and then searching through the data points residing in these blocks. Since the data points visited are uch less copared to the whole database, the search procedure can thus significantly speed up. In the ipleentation, 2.4 illion iages are indexed into 2, clusters. Given a query iage, the ost nearest cluster blocks are retrieved to insure that there are at least 1, iages in the retrieved blocks. Several hundreds of iages with largest siilarities will be kept for further use in annotation ining step. With the indexing technique, less than.3 s is needed to retrieve 5 ost siilar iages fro 2.4 illion iages. For each retrieved iage I i S, the distance between I i and the target iage I t is denoted as Dist(i). In this ipleentation, the Euclidean distance is adopted. 3.2 Text-based keyword search For each iage I i S, a text-based keyword search process is used to rank all the related keywords. The related keywords are the ones that appear in title or description of I i except the stop words and soe inappropriate words such as iage and photo. More specifically, for I i, the set of related keywords are denoted as K i. Denote K as the cobination of all K i, i.e. K = Ii SK i. For each keyword K j K, its relevance score to I i is denoted as Score relevance (i, j). Two strategies could be used to calculate Score relevance (i, j). One is to use the proinence score. Proinence score reflects the proinence of a keyword to annotate an iage. The proinence score of a keyword K j to I i is defined as follows: Score Proinence (i, j) = { occurrence(i, j) K k K i occurrence(i,k) K j K i K j / K i (5) where occurrence(i, j) denotes the nuber of K j in title or description of I i.

6 C. Wang et al. Recall that the iages in iage annotation correspond to the textual ters in inforation retrieval, while the annotation keywords in iage annotation correspond to the docuents in inforation retrieval. Thus, in iage annotation proble, a keyword could be considered as a docuent with several related iages being keywords. An iage is deeed as related to a keyword if the keyword appears in the title or description of the iage. Then the aforeentioned occurrence nuber could be considered as iage frequency (IF) that is analogous to ter frequency (TF). Siilar to docuent frequency (DF), the keyword frequency (KF) of an iage could be defined as the nuber of related keywords of the iage. Enlightened by the tf-idf weighted schee, a new scoring schee based on if-ikf is proposed as follows: Score ifikf (i, j)= { occurrence(i, j) log 2 (sizeof (K i )+1) otherwise K i =φ or K j / K i. (6) For efficiency, we index textual descriptions of iages in the database. The textual description of each iage I i in the entire database is represented by a list of keyword-score pairs: (ID_of_K j, Score ifikf (i, j)). Here K j K i. For an iage I i in the database, a text engine could directly return the list of keyword-score pairs to the annotation syste. 3.3 Merging stage The ranked lists of all siilar iages in S are further erged to get the final candidate annotation keywords of the target iage. Considering that the siilar iages have different siilarities with the target iage and ore siilar iage should have ore ipact on the final annotation results, we use the following forula to score and rank the keywords K j K: Score relevance (K j ) = f (Dist(i)) I i S Score relevance (i, j), (7) where f ( ) is a function that transfors distance to siilarity. f ( ) is defined as follows: f (d) = { 1 d 2 } exp 2πσ 2 2σ 2 Once the relevance score of each keyword in K is obtained, we can pick up the top N as the final candidate annotations {w i } i=1,...,n with corresponding confidence score {Score relevance (w i )} i=1...n to be refined in the next annotation refineent stage. (8) 3.4 Annotation refineent with RWR In the candidate annotation set {w i } i=1,...,n, there ay be soe irrelevant annotations. We can explore the relationship aong candidate annotations to try to discard the noisy ones. Jin et al. [17] have done pioneer work on annotation refineent using a generic knowledge-based WordNet. Fro the sall candidate annotation set obtained by an annotation ethod, the irrelevant annotations will be pruned using WordNet [22]. The basic assuption is that highly correlated annotations should be reserved and non-correlated annotations should be reoved. However, in [28,29] we have testified that there are two ain liitations using Word- Net. One is that the siilarity defined using WordNet is soeties not appropriate for the annotation refineent proble. For exaple, ountain and sky usually appear in a scenery photo together, while tree and flag seldo siultaneously appear in an iage. However, with the JCN easure in WordNet, the siilarities of the above two pairs of words are.61 and 48, respectively, which is unreasonable. With the proposed siilarity easure, the two siilarities will be.43 and.95, respectively, which is ore reasonable. The other liitation is that it cannot deal with the annotations that do not exist in the lexicon of WordNet. For exaple, there are 49 out of 374 words of the Corel dataset 2 which either do not exist in WordNet lexicon or have zero siilarity with all other words using the JCN easure [22], while all of the exist in our unliited lexicon. In order to fully utilize the confidence scores of the candidate annotations obtained by forer stages and the Web-scale iage set as aforeentioned in Sect. 3.1, we reforulate the iage annotation refineent process as a graph ranking proble and solve it with the RWR algorith. The basic assuption of the algorith is that highly correlated keywords should be ranked higher. Thus, by odeling the candidate keywords as vertices of a graph and defining certain siilarity as the edges to easure the correlation between keywords, the scores could be propagated aong correlated keywords. By forulating the iage annotation refineent process as a graph ranking proble, two disconnected keywords could also influence each other iteratively, and finally their scores converge to a static state Graph construction Each candidate annotation w i is considered as a vertex of a graph G. All vertices of G are fully connected with proper weights. The weight of an edge is defined based on the co-occurrence siilarity as below. We have built an iage search engine naed as Enjoy- Photo [36] based on the Web-scale iage set as aforeen- 2

7 Scalable search-based iage annotation tioned in Sect Each word w i will be used as a query to query EnjoyPhoto. The nuber of search results is denoted as nu(i). For two different word w i and w j, w i w j will be used as the query. The nuber of search results is denoted as nu(i, j). The weight of the edge between w i and w j is then calculated by the following forula: si(w i,w j ) { nu(w i,w j ) = in (nu(w i ),nu(w j )) + ε nu(w i,w j )> ε nu(w i,w j ) where ɛ is a constant satisfying <ɛ<< The RWR algorith The RWR algorith [24] perfors as follows. Assue that there is a rando walker that starts fro node w i with a certain probability. At each tie-tick, the walker has two choices. One is to randoly choose an available edge to follow. The other choice is to jup to w j with probability c v(j), where v is the restart vector and c is the probability of restarting the rando walk [24] Iage annotation refineent with RWR Assue that G is a graph with N vertices {w i } i=1...n constructed as in Sect Let A be the adjacency atrix of G. A is colun-noralized to ensure that the su of each colun in A is one. The original confidence scores of candidate annotations are considered as the restart vector v. v is noralized to ensure the su of all eleents in v is one. The ai is to estiate the steady-state probability of all vertices, which is denoted by u. Letc be the probability of restarting the rando walk. Then the N-by-1 steady state probability vector u satisfies the following equation: u = (1 c)au + cv (1) The iteration of Eq. 1 can be guaranteed to converge if the transition atrix A is stochastic and priitive, which can be satisfied by our definition to the edge of the graph in Eq. 9. Thus, we have u = c(i (1 c)a) 1 v (11) where I is the N N identity atrix. The ith eleent u(i) of the steady-state vector u is the probability that w i can be the final annotation. We can choose the top annotations with highest probabilities as the final annotations. 3.5 Iage annotation rejection In our syste, we use a 64-diensional global feature to represent an iage. It is worth noting that global features are (9) helpful for iage-level concept annotation, but are not very powerful for object-level annotation. It is thus necessary to develop a schee to estiate the confidence of our syste to annotate the query iages, and thereby reject the ones we cannot handle well. We first construct a consistency ap for the Web-scale iage collection, on which each iage has a consistency value to represent the local seantic consistency between this iage and its visually siilar iages. Then, for a query iage, we use the consistency ap to calculate the annotation confidence of the query iage. Following the notations in Sects. 3.1 and 3.2, weusethe following forula to calculate the consistency value of each iage I i in the collection: Score consistency (I i ) = Si visual (I i, I j ) I j S i Si seantic (I i, I j ), (12) where S i is the siilar iage set of iage I i. Si visual (I i, I j ) represents the visually siilarity between iage I i and I j, which can be calculated by Eq. 8. Si seantic (I i, I j ) represents the seantic consistency between iage I i and I j. Since in the Web-scale iage collection, each iage I i has related keyword descriptions denoted as K i (see Sect.3.2), we can utilize the cosine siilarity [27] between K i and K j to calculate Si seantic (I i, I j ). After the whole consistency ap of the Web-scale iage collection is constructed, we utilize this ap to calculate the annotation confidence of the target iage I t : Score conf idence (I t ) = Si visual (I t, I j ) I j S t Score consistency (I j ). (13) Therefore, for a query iage, besides the annotation list, our syste can offer a confidence score of the annotation list for the query iage. We can infor users that we cannot handle the query iage if this score is below certain threshold. 3.6 Coparison with AnnoSearch In AnnoSearch [31], given a target iage I t and a related keyword w t, the goal is to annotate I t with ore related keywords. The annotation process could be roughly separated into two stages. First, the ost siilar iages S of a largescale Web iage dataset are retrieved using both w t and I t. Then, the search result clustering (SRC) technology [35] is used to ine the annotations w fro text descriptions of S. The two stages can be expressed as the following forula: p(w I t,w t ) = p(w S)p(S I t,w t ). (14) With SRC, the ost representative topics t could be detected based on which w could be learned. Therefore,

8 C. Wang et al. Eq. 14 can be rewritten as follows: p(w I t,w t ) p(w t )p(t S)p(S I t,w t ). (15) As the first work that leverages the Web-scale training corpus, both the scalability and perforance of AnnoSearch are satisfactory. However, it still has spaces for iproveent. The ain liitation is the requireent of an initial accurate keyword which is not always available, especially for personal iages. Moreover, since SRC was initially designed for general Web search, directly using it in annotation ining ay not be an optial solution. Furtherore, the siilar iages are treated equally ignoring their siilarities to the target iage. The proposed four-stage SBIA algorith could be expressed as the following forula: p(w I t ) = RW R p(w I i )p(i i I t ). (16) Ii S First, the ost siilar iages S are retrieved based on CBIR techniques. Meanwhile, the siilarity between target iage I t and an iage I i S is reserved as p(i i I t ). Second, for each iage I i S, the relevance score of the keywords w is obtained by text-based keyword search technology. It is denoted by p(w I i ). Third, all relevance scores fro different iages in S are erged for each keyword in w. Finally, the top candidate keywords in w are refined by the RWR algorith, and the top ones after refined are returned to users as the final annotations of I t. Note that no initial keyword is needed in advance and the siilar iages are used according to their siilarities to the target iage. 4 Experiental results A series of experients were conducted on U. Washington dataset [1] to evaluate the proposed SBIA algorith. First, we use the Web-scale dataset as the training set to test the U. Washington iages in the iage annotation task. Then, to show the effectiveness of the annotations on iage retrieval, the retrieval perforance of query by keyword (QBK) using SBIA results was copared with that of query by exaple (QBE) using visual features. 4.1 Iage annotation The ai of this work is to autoatically annotate arbitrary iages with unliited vocabulary by leveraging a Web-scale iage set. Thus, in this section, we show detailed experiental results of this part. As entioned in Sect. 3.1, we use the 2.4 illion Web iages associated with eaningful descriptions as the Webscale training set, which is consistent with the ipleentation in [31] and [21].We use U. Washington dataset (UW) as the testing set. UW is a CBIR database, which can be downloaded fro the University of Washington. There are about 5 anually labeled ground truth annotations for each iage in UW. Totally, there are 1,19 iages and ore than 35 unique words. Although, for soe iages, not all objects in the are annotated, we strictly use the annotations of UW as the ground truth annotations and the synonys and nonappearing correct annotations are assued incorrect, if there is no other explanation. First, several variations of SBIA algorith were evaluated. Then SBIA was copared with a odified version of Anno- Search [31] that reoved the dependency of initial keywords and LiAnnoSearch [21] Experiental design In AnnoSearch, there is a strong constraint that an accurate query keyword should be provided. The UW folder naes are used as the initial query keyword [31]. Although the initial keyword ight speed up the search process and result in ore relevant iages to the query iage, it is not always available, especially for the personal iages. Due to the unavailability of accurate query keywords for personal iages, we ignore the query keyword when we ipleent AnnoSearch algorith, and only use the query iage. More specifically, the first stage of AnnoSearch is replaced with the first stage of SBIA. Moreover, the average eber iage score criterion [31] is used in the odified version of AnnoSearch (MAnnoSearch) due to its good perforance in [31]. Two strategies are used to evaluate the annotation perforance: phrase-level strategy and ter-level strategy. The original ground truth annotations include both words and phrases. In the phrase-level strategy, an annotation is considered to be correct if and only if it is a ground truth annotation of the target iage. In the ter-level strategy, both the ground truth annotation phrases and the result annotation phrases are divided into separate words. If there is ore than one sae word in the annotations of an iage, only one is reserved. An annotated word is considered to be correct if and only if it appears in the ground truth annotation of the target iage. The precision and recall for the two strategies are defined as follows: n precision phrase = n 1 k=1 n recall phrase = n 1 k=1 n precision ter = n 1 k=1 n recall ter = n 1 k=1 correct p (k) autoatic p (k) correct p (k) groundtruth p (k) correct t (k) autoatic t (k) correct t (k) groundtruth t (k), (17) where correct p (correct t ) is the nuber of correctly annotated phrases (ters) of the testing iage I k. autoatic p

9 Scalable search-based iage annotation (autoatic t ) is the nuber of autoatically annotated phrases (ters) in I k.groundtruth p (groundtruth t )isthe nuber of ground truth phrases (ters) in I k.n is total nuber of testing iages. Although the proposed algorith is able to produce phrase level annotations by using n-gra odel [5], currently only ter-level annotations are considered for siplicity. As a result, the phrase-level evaluation strategy will be disadvantageous for the proposed SBIA algorith. However, the experiental results in the following sections will show that the annotation results of SBIA are still satisfactory even gauged by the phrase-level strategy. Due to the use of alost unliited vocabulary, there will be a long list of annotations for both MAnnoSearch and SBIA. Therefore, the nuber of annotation results is restricted to be no ore than 1. With result annotations, the precision and recall are denoted by precision@ and recall@. First, two variations i.e. size of siilar iage set and scoring strategy were evaluated, without the refineent process (SBIA-N). After the variations were fixed, SBIA-N was copared with MAnnoSearch. Second, we evaluated the proposed iage annotation refineent process in SBIA. After fixing the restart paraeter, we copared the SBIA and MAnnoSearch with the refineent process (MAnnoSearch- Y), followed by the coparison between SBIA and LiAnnoSearch. Then, the experient of annotation rejection was shown. Finally, since our algorith can predict annotations outside the ground truth, we also provided soe results using anual evaluation Size of siilar iage set The size of visually siilar iage set of the first CBIR stage is a coon and crucial paraeter in both MAnnoSearch and SBIA. Let s denote it by s. To facilitate the further evaluations, s is first decided for both algoriths by coparing the annotation perforance with different s. The nuber of annotation results is fixed to 5 and both proinence-based and if-ikf-based scoring strategies are considered. The precision and recall are shown in Fig. 3a and b with s changed fro 2 to 2,5. Five conclusions could be drawn fro Fig. 3a and b. First, the changing trends of precision and recall are siilar. Second, the absolute values of ter-level evaluation are consistently better than that of phrase-level evaluation, which coincides with our intuition. Third, the perforance of MAnnoSearch is best when s is 2. Therefore, s is set to be 2 for MAnnoSearch in the following evaluations. Fourth, the changing trends of both scoring strategies are siilar. Last, the perforance of SBIA is siilar when s is equal or larger than 5. Considering that the saller the value of s, the ore efficient the annotation process will be, we set s to be 5 for SBIA. precision recall s s MAnnoSearch_Phrase SBIA_IFIKF_Phrase SBIA_Proinence_Phrase MAnnoSearch_Ter SBIA_IFIKF_Ter SBIA_Proinence_Ter Fig. 3 a, b Annotation precision and recall of different sizes of siilar iage set Scoring strategy The two scoring strategies of SBIA as discussed in Sect. 3.2 were evaluated and copared. Fro Fig. 3a and b we can see that the if-ikf strategy is consistently better than proinence strategy when is 5. To further copare the two strategies, was varied fro 2 to 1. The coparison results of if-ikf strategy and proinence strategy are shown in Fig. 4a and b. Although the perforance of if-ikf strategy is siilar to that of proinence strategy when is less than 4, both precision and recall of if-ikf strategy outperfor those of proinence strategy when is larger than 4. Therefore, in the following experients, the if-ikf strategy is used for SBIA MAnnoSearch versus SBIA-N Based on the aforeentioned preparations, SBIA without refineent process (SBIA-N) was copared with MAnno- Search using the according paraeters. Figure 5a and b show the coparison results. SBIA-N consistently outperfors MAnnoSearch while is ranging fro 2 to 1.

10 C. Wang et al. precision recall Proinence_Phrase Proinence_Ter IFIKF_Phrase IFIKF_Ter Fig. 4 a, b Annotation precision and recall of different relevance scores Iage annotation refineent In this section, we evaluate the iage annotation refineent process of SBIA. The nuber of candidate annotations is set to be 1. For each query iage, we refine the ten candidate annotations obtained by SBIA-N using the proposed RWR algorith. The restart paraeter c was varied fro to 1. Recall that SBIA degenerates to be SBIA-N when c is 1. The experiental results of annotation refineent based on SBIA-N are shown in Figs. 6a, b and 7a, b. Both results of ter-level and those of phrase-level are shown. Four conclusions could be drawn fro Fig. 7a and b. First, the changing trends of ter-level evaluation and phrase-level evaluation are siilar. Second, both precision and recall rates reach to the lowest value when c is 1 with fixed, which indicates that the refineent process did iprove the iage annotation perforance. Third, the iproveent of precision is ore significant than that of recall. Fourth, since the refineent process is based on the sae candidate annotation list, the perforances with different c tend to be consistent while is approaching precision recall MAnnoSearch_Phrase MAnnoSearch_Ter SBIA-N_Phrase SBIA-N_Ter Fig. 5 a, b Annotation precision and recall coparison of MAnno- Search and SBIA-N The refineent process is orthogonal to the ethod of producing candidate annotations. We have conducted the sae evaluation of annotation refineent for MAnnoSearch. All the aforeentioned conclusions still stand. Since MAnnoSearch with refineent (MAnnoSearch-Y) outperfors MAnnoSearch, we copared SBIA with MAnnoSearch-Y in the next sub-section. We set c to be.3 for both SBIA and MAnnoSearch-Y in the following evaluations, for it is the best paraeter for both of the MAnnoSearch-Y versus SBIA Based on the aforeentioned preparations, SBIA was copared with MAnnoSearch-Y using the according paraeters. Figure 8a and b show the coparison results. SBIA consistently outperfors MAnnoSearch-Y while is ranging fro 2to LiAnnoSearch versus SBIA & MAnnoSearch In [21], Li et al. presented a odified version of Anno- Search (LiAnnoSearch) by reoving the dependency of the initial accurate keyword w t. As the sae as AnnoSearch,

11 Scalable search-based iage annotation ter precision ter recall c c =2 =4 =6 =8 =1 Fig. 6 a, b Annotation precision and recall of different restart paraeters using ter-level evaluation phrase precision phrase recall c c =2 =4 =6 =8 =1 Fig. 7 a, b Annotation precision and recall of different restart paraeters using phrase-level evaluation LiAnnoSearch solved iage annotation proble in a search and ining process. Besides the independency of w t,the ain difference between AnnoSearch and LiAnnoSearch in search stage is that, a novel high-diensional indexing algorith is proposed to index high-diensional visual features in LiAnnoSearch. Table 1 shows the coparisons of SBIA, LiAnnoSearch and MAnnoSearch under ter-level evaluation strategy, which is the only evaluation strategy in LiAnnoSearch. The results of LiAnnoSearch are directly referred to [21], in which the nuber of final annotations is dynaically deterined. To get a fair evaluation, we directly fix to be 1 in MAnnoSearch and SBIA. In Table 1 we can see that LiAnnoSearch is coparable with MAnnoSearch. That is reasonable since both of the can be considered as the odified versions of AnnoSearch by reoving the dependency of initial keywords, and except the dependency of initial accurate keyword, all other probles in AnnoSearch could also be suffered in both of MAnnoSearch and LiAnnoSearch. Moreover, the results also show that the high-diensional indexing algorith proposed in LiAnnoSearch cannot rearkably iprove annotation perforance. Different fro Anno- Search and LiAnnoSearch, we forulate iage annotation as a search proble. Fro this perspective, we further find that search-based iage annotation and inforation retrieval can be treated as dual probles, and therefore we can systeatically leverage those ature inforation retrieval technologies. Therefore, SBIA outperfors both of LiAnno- Search and MAnnoSearch. It took less than.23 s (less than.3 s for iage search process and less than.2 s for annotation ining process) and about 8 M eory (3 M for visual feature and 5 M for textual feature) on average to annotate one iage on a Web server with two 2. GHz CPUs and 2 GB eory, which is coparable with LiAnnoSearch and MAnnoSearch. The rather high efficiency of SBIA guarantees its application to large personal iage collections Iage annotation rejection In Sect. 3.5, we have developed a ethod to calculate the consistency score to represent the local seantic consistency

12 C. Wang et al. precision recall MAnnoSearch-Y_Phrase MAnnoSearch-Y_Ter SBIA_Phrase SBIA_Ter Fig. 8 a, b Annotation precision and recall coparison of MAnnoSearch-Y and SBIA Table 1 Coparison of SBIA, LiAnnoSearch, and MAnnoSearch Models MAnnoSearch LiAnnoSearch SBIA Precision Recall between the query iage and its visually siilar iages in the database. This score could be used to estiate the confidence of our syste to annotate the query iage, and thereby reject the ones we cannot handle well. To show the effectiveness of the rejection schee, we reove the iages of UW with lowest confidence scores, and evaluate other confident iages. Figure 9a and b shows the average precision and recall perforance of reaining query iages after rejecting certain iages with lowest consistency scores (e.g. 2% of the query iage set), using ter-level evaluation. The larger the rejection rate is, the ore confident the left query iages are. The constant iproveent by increasing rejection rate shows that the rejection schee could iprove the annotation perforance by autoatically rejecting the ones we cannot handle well. 1 precision recall =2 =4 =6 =8 =1 1% 2% 3% 4% 5% 6% 7% 8% 9% rejection rate 1% 2% 3% 4% 5% 6% 7% 8% 9% rejection rate Fig. 9 a, b Annotation precision and recall with different rejection rate Results using anual evaluation In the previous evaluations, we strictly used the annotations of UW as the ground truth annotations, and thus the synonys and non-appearing correct annotations were assued incorrect. Although this evaluation ethod can provide a fair and easy coparison between different annotation works, it ay shrink the annotation perforance, for the proposed SBIA annotation syste can predict annotations outside of the ground truth. Since one of the advantages of our ethod is to annotate iages using unliited vocabulary, we also show soe results using anual evaluation instead of strictly using the ground truth of UW dataset. Siilar to the evaluation syste of [2], each iage in 1 rando selected iages of UW dataset is shown together with 1 annotation words assigned by SBIA algorith. A trained person, who did not participate in the developent

13 Scalable search-based iage annotation accuracy coverage rate word rank nuber of words Fig. 1 Annotation perforance based on anual evaluation of 1 rando selected iages of UW dataset. a Percentages of iages correctly annotated by the th word. b Percentages of iages correctly annotated by at least one word aong the top words of the syste, exaines every word against the iage and checks a word if it is judged as correct. Annotation perforance is reported fro two aspects in Fig. 1. Figure 1a shows accuracies, that is, the percentages of iages correctly annotated by the th word. The first word achieves an accuracy of 58%. The accuracy decreases gradually with except for inor fluctuation with the sixth and seventh words. This reflects that the ranking of the words by the SBIA syste is on average consistent with the true level of accuracy. Figure 1b shows the percentages of iages correctly annotated by at least on word aong the top words. Using only six autoatic annotations produced by the syste, we can achieve 9% coverage. Soe exaples in Fig. 11 also show the ability of the proposed annotation syste. Fro Fig. 11 we can see that although soe auto-annotations do not appear at UW ground truth, they are still reasonable to be annotations of the corresponding iage. 4.2 Iage retrieval To show the effectiveness of the annotations of SBIA on iage retrieval, the retrieval perforance of query by keyword (QBK) using SBIA results was copared with that of query by exaple (QBE) using visual features. Ten ters were selected fro the UW dataset as the query concepts. The ters are chosen based on both popularity and diversity. The ten ters with corresponding occurrence nuber in both ground truth and the results of SBIA are listed in Table Evaluation easures Precision and recall are used as the basic easures. Since the queries of QBK and QBE are different, we re-define the precision and recall accordingly. For QBK, the precision and recall of a ter t are defined as follows: precision@ = correct_t() recall@ = correct_t() (18) groundtruth_t, where correct_t() is the nuber of correctly retrieved iages in the first iages and groundtruth_t is the nuber of iages annotated by ter t in ground truth. For QBE, denote the ost siilar iages to an iage I i by S i. Denote the iages in S i that are annotated by a ter t in ground truth by Si t. The precision and recall of the ter t are defined as follows: precision@ = 1 n t recall@ = 1 n t I i S I i S correct t i () correcti t () groundtruth_t, (19) where S t is the set of iages annotated by t in ground truth. n t is the size of S t.correcti t () is the nuber of iages in Si t Retrieval coparison The experiental results of iage retrieval for each selected ter using either QBK or QBE are shown in Figs. 12a, b. is set to be 1. Although for people, snow and rock, QBE is slightly better than QBK, QBK is apparently superior to QBE

14 C. Wang et al. Fig. 11 Annotation results for several UW iages. The second row is the auto-annotation produced by SBIA syste (top 5 results). The third row is the ground truth annotations of UW iages set Auto-annotation: blue, sky, landscape, water, sea Auto-annotation: people, street, children, girl, friend Auto-annotation: landscape, sky, water, white, black Auto-annotation: sunset, sunrise, sun, people, tree UW Ground truth: partly cloudy sky, hills, trees, sall sailboat, water UW Ground truth: husky aluni band, cheerleaders, hec ed pavilion, cobblestones, windows, 2 rounded entryways, dru, horn instruents UW Ground truth: clouds, frozen, lake, ountain, sky UW Ground truth: boats, sailboats, sky, sunrise/sunset Table 2 Selected ters and their corresponding occurrence nuber in ground truth and our annotation results Ter: Sky People Snow Rock Flower 1.9 QBE_CBIR QBK_SBIA Nu (GT) Nu (My) Ter: Street Car Boat Bridge Green Nu (GT) Nu (My) precision for the other 7 concepts. The average retrieval perforances with different are shown in Figs. 13a and b. Both precision and recall of QBK are consistently higher than those of QBE. 5 Conclusions and future work 6 4 In this paper, by forulating iage annotation as a search proble, we have presented a novel search-based iage annotation algorith that is analogous to inforation retrieval. First, CBIR technology is used to retrieve a set of visually siilar iages fro a large-scale Web iage set. Second, a text-based keyword search technique is used to obtain a ranked list of candidate annotations for each retrieved iage. Third, a fusion algorith is used to cobine the ranked lists into the final candidate annotation list. Finally, the candidate annotations are re-ranked using RWR and only the top ones are reserved as the final annotations. The application of both efficient search technologies and Web-scale iage set guarantees the scalability of the proposed algorith. Experiental results on U. Washington datasets show not only the effectiveness and efficiency of the proposed algorith but also the advantage of iage retrieval using annotation results over that using visual features. recall Fig. 12 a, b Retrieval precision and recall of different ters By re-forulating iage annotation as a search process, several effective inforation retrieval techniques could be adaptively used and deeper insight into the annotation