Supervised Coupled Dictionary Learning with Group Structures for Multi-Modal Retrieval

Transcription

1 Poceedings of the Twenty-Seventh AAAI Confeence on Atificial Intelligence Supevised Coupled Dictionay Leaning with Goup Stuctues fo Multi-Modal Retieval Yueting Zhuang and Yanfei Wang and Fei Wu and Yin Zhang and Weig Lu College of Compute Science Zhejiang Univesity, China Abstact A bette similaity mapping function acoss heteogeneous high-dimensional featues is vey desiable fo many applications involving multi-modal data. In this pape, we intoduce coupled dictionay leaning (DL) into supevised spase coding fo multi-modal (cossmedia) etieval. We call this Supevised coupleddictionay leaning with goup stuctues fo Multi- Modal etieval (SliM 2 ). SliM 2 fomulates the multimodal mapping as a constained dictionay leaning poblem. By utilizing the intinsic powe of DL to deal with the heteogeneous featues, SliM 2 extends unimodal DL to multi-modal DL. Moeove, the label infomation is employed in SliM 2 to discove the shaed stuctue inside inta-modality within the same class by a mixed nom (i.e., l 1/l 2-nom). As a esult, the multimodal etieval is conducted via a set of jointly leaned mapping functions acoss multi-modal data. The expeimental esults show the effectiveness of ou poposed model when applied to coss-media etieval. Intoduction Similaity seach, a.k.a. neaest neighbo seach, is a fundamental poblem and has enjoyed success in many applications of data ing, database, and infomation etieval. Nevetheless, most of the similaity seach algoithms ae only conducted in the uni-modal data setting, which ae esticted to etieve the simila data with the same modality as uey data. Nowadays, many eal-wold applications involve multi-modal data, whee infomation inheently consists of data with diffeent modalities, such as a web image with loosely elated naative text desciptions, o a news aticle with paied text and images. Theefoe, it is desiable to suppot similaity seach fo multi-modal data (i.e., cossmedia etieval), e.g., the etieval of textual documents in esponse to a uey image o vice vesa (Wu, Zhang, and Zhuang 2006) (Zhuang, Yang, and Wu 2008). Multi-modal etieval is vey impotant to many applications of pactical inteest, such as finding some detailed textual documents of a touist spot that best match a given image, obtaining a set of images that best visually illustate a given text, o seaching simila esults by a set of combined texts and images. Copyight c 2013, Association fo the Advancement of Atificial Intelligence ( All ights eseved. To the best of ou knowledge, thee ae geneally two kinds of appoaches to boost coss-modal etieval: one is canonical coelation analysis (CCA) (Hotelling 1936) and its vaiants. Fo examples, afte the maximally coelated subspace of text and image featues is obtained by CCA, logistic egession is employed to coss-media etieval in (Rasiwasia et al. 2010). A supevised extension of CCA, efeed as genealized multiview analysis (GMA), was poposed in (Shama et al. 2012) fo coss-media etieval. These existing CCA-based appoaches attempt to enfoce a stong assumption among the multi-modal data, i.e., the diffeent modalities have a common o a shaed subspace. Howeve, this assumption is too esticted to some extent fo analysis of multi-modal data in eal-wold setting. Fo example, given a pai of image and text, the image pobably contains a consideable amount of infomation not elated to its coesponding text, and it is not even guaanteed that the text is elated at all to the visual content of the image. Anothe kind of appoaches fo multi-modal etieval ae extensions of Latent Diichlet Allocation (LDA). Following the seal wok of Blei et al.(blei, Ng, and Jodan 2003), Latent Diichlet Allocation (LDA) has been extended to lean the joint distibution of multi-modal data (e.g., texts and images) such as Coespondence LDA (Co-LDA) (Blei and Jodan 2003), Topic-egession Multi-modal LDA (t-mmlda) (Putthividhy, Attias, and Nagaajan 2010), Multi-field Coelated Topic Modeling (mf-ctm) (Salomatin, Yang, and Lad 2009) and Hieachical Diichlet Pocess(HDP) based LDA (Vitanen et al. 2012). These afoementioned appoaches tend to model the coelations of multi-modal data at latent semantic (topic) level acoss modalities. Theefoe, they eithe assume that all modalities shae same topic popotions, o have one-to-one topic coespondences, o have commonly shaed topics. Nevetheless, those assumptions inheently estain a moe flexible application of coss-media etieval in the setting involved uncontolled multi-modal data. On the othe hand, when the class labels (categoies) of multi-modal data ae available, it is natual to assume that inta-modality data within the same class (categoy) shaes some common aspects. Fo examples, images fom the achitectue categoy have simila low-level visual featues (such as geometic egulaities and patches of unifom colo (Todoovic and Nechyba 2004) ), and textual documents 1070

2 fom biology have ovelapping wods (e.g., cells and genetics). Theefoe, it is appopiate to utilize the class labels to lean the disciately shaed components fo intamodal data fom the same categoy. Motivated by the fact that dictionay leaning (DL) methods have the intinsic powe of dealing with the heteogeneous featues by geneating diffeent dictionaies fo multi-modal data, this pape tends to study on jointly leaning multi-modal dictionaies in a supevised setting, and simultaneously ing the shaed stuctues inside each inta-modality fom the same classes. Thee ae some existing DL appoaches fo multi-modal data. Method was poposed in (Monaci et al. 2007) to lean multi-modal dictionaies fo audiovisual data. This model, howeve, can only deal with synchonous tempoal signals. A dictionay leaning appoach is poposed in (Jia, Salzmann, and Daell 2010) to factoize the latent space acoss modalities into shaed components (to all modalities) and pivate pats (to each modality). The assumption in (Jia, Salzmann, and Daell 2010) that assumes a uniue spase coefficient acoss all the modalities is still too esticted to multi-modal data in eal-wold applications. Inspied by the ecently poposed idea of (semi-)coupled dictionay leaning (CDL) fo image supe-esolution (Jia,Tang, and Wang 2012) and photo-sketch synthesis (Wang et al. 2012), which suggest that one pai of image patches fom diffeent domains (low esolution vs high esolution, o photo vs sketch) has the same dictionay enties o has a mapping function between the econstucted spase coefficients, this pape poposes Supevised coupled dictionay leaning with goup stuctues fo Multi-Modal etieval (SliM 2 ). SliM 2 extends uni-modal DL to multi-modal DL and jointly leans a set of mapping functions acoss diffeent modalities. Futhemoe, SliM 2 utilizes the label infomation to discove the shaed stuctues inside intamodalities fom the same classes. The Model of SliM 2 In this section, we fist biefly eview spase coding and its extensions, then we pesent the fomulation of SliM 2. At last, SliM 2 is conducted fo multi-modal etieval. Dictionay Leaning and Its Extensions The modeling of data with the linea combinations of a few elements fom a leaned dictionay has been the focus of much ecent eseach (Olshausen, Field, and othes 1997) (Wight et al. 2009). The essential challenge to be esolved in spase coding is to develop an efficient appoach with which each sample can be econstucted fom a best dictionay with a spase coefficients. Let X R p n be the data matix to be econstucted, D R p k the leaned dictionay and α R k n the spase econstuction coefficients (also known as spase codes), whee p, n and k ae the dimensions of featue space, the numbe of data samples and the size of the dictionay espectively. The fomulation of spase coding can be expessed as follows: 1 D,α 2 X Dα 2 F + λψ(α) s.t. d i 1, i, whee Ψ(α) epesents the imposed penalty ove spase codes α and d i is one of the dictionay atoms of D. Typically, the l 1 nom is conducted as a penalty to explicitly enfoce spasity on each spase codes α j (α j α(j = 1,..., N)) (Tibshiani 1996) (Jia, Salzmann, and Daell 2010) as follows N Ψ(α) = α j 1. (2) j=1 The above classical data-diven appoach to dictionay leaning is well adapted to econstuction tasks such as estoing a noisy signal. In ode to lean a disciative spase model instead of puely econstuctive one, spase coding is extended into supevised spase coding (Maial et al. 2008). In eal-wod setting, diffeent data can be natually designated into diffeent goups, a mixed-nom egulaization (l 1 /l 2 -nom) can be conducted in spase coding to achieve spasity as well as to encouage the econstuction of samples fom the same goup by the same dictionay atoms, which is named as goup spase coding in (Bengio et al. 2009). If all of the images in one class (categoy) is taken as a goup, as stated befoe, it is appopiate to assume that when a set of dictionay atoms has been selected to epesent one image of a given categoy, the same dictionay atoms could be used to epesent othe images of the same categoy (Bengio et al. 2009). The fomulation of goup spase coding is as follows: 1 J k D,α 2 X Dα 2 F + λ α i,ωl 2 (3) s.t. d i 1, i, l=1 i=1 whee J is the numbe of classes (goups), Ω l epesents the indices of the examples that belong to the l-th class (lth goup), and α :,Ωl is the coefficient matix associated to examples in the l-th goup. The Fomulation of SliM 2 Suppose that we have a labeled taining set of N pais of coespondence data with M modalities fom J classes: {(x (1) i (1),, x (M) i, l i ) : i = 1,..., N} {(X (1),, X (M), L)}. X (m) R Pm N (1 m M) is P m -dimensional data fom the m-th modality, l i = (l i1,..., l ij ) {0, 1} J is the coesponding class label, l ij = 1 if the i-th data x i = (x (1) i,, x (M) i ) belongs to the jth class and l ij = 0 othewise. Hee, the i-th data x i only belongs to a single class: J j=1 l ij = 1. We have seen fom E.(3) that goup spase coding is a way fo uni-modal dictionay leaning when the input signals ae natually assigned into diffeent goups. Of paticula inteest to us in this pape is modeling the elationships 1071

3 between multi-modal data athe than the independent dictionay leaning fom uni-modal data. In ode to esolve this issue, we esot to semi-coupled DL (Wang et al. 2012) fo a mapping between econstuction coefficients. The undelying motivation behind ou SliM 2 has two points: a) jointly lean dictionaies fo each modality data and a elatively simple mapping function acoss modalities; b) discove the shaed stuctues fo each inta-modality data fom the same class via a mixed nom (i.e., l 1 /l 2 -nom). SliM 2 aims to jointly lean a set of dictionaies fo M modality data espectively, i.e., D = {D (1), D (2),, D (M) } with D (m) R Pm K and thei coesponding econstuction coefficients A = {A (1), A (2),, A (M) } with A (m) R K N, whee K is the size of the dictionaies (the numbe of atoms in dictionay). In ode to conduct the multi-modal etieval, we assume thee exists a set of linea mappings W = {W (1), W (2),, W (M) } with W (m) R K K between spase codes. The objective function of ou poposed SliM 2 is fomulated as follows: + β s.t. X (m) D (m) A (m) 2 F + n m J l=1 A (n) W (m) A (m) 2 F + γ d (m) k 1, k, m, λ m A (m) :,Ω l 1,2 W (m) 2 F whee A :,Ωl is the coefficient matix associated to those inta-modality data belonging to the l-th class. Fo an abitay matix A R k n, its l 1 /l 2 -nom is defined as k n A 1,2 = A 2 ij. (5) i=1 j=1 Hee, β,γ and λ m (m = 1,..., M) ae tuning paametes denoting the weights of each tem in E.(4). It is obvious that data in the m-th modality space can be mapped into the n-th modality space by the leaned W (m) accoding to A (n) W (m) A (m) 2 F, theefoe, the computation of multimodal similaity is achieve in SliM 2. The degee of spasity fo data acoss modalities could be diffeent due to thei heteogeneity with high-dimensional settings. As a esult, diffeent λ m (m {1,..., M}) is employed in E.(4) to contol the degee of spasity of the spase codes espectively fo M modality data. It can be obseved fom E.(4) that the poposed SliM 2 not only jointly imizes the econstuction eo of data acoss modalities, but also independently encouages to utilize same dictionay atoms fo the econstuction of the inta-modality data fom the same class. The Optimization of SliM 2 The afoementioned objective function in E.(4) is nonconvex and non-smooth, but it is convex to each set of D = {D (1), D (2),, D (M) }, A = {A (1), A (2),, A (M) } (4) and W = {W (1), W (2),, W (M) } when the othe two ae fixed. Theefoe, in pactice, we can develop an iteative algoithm to optimize the vaiables altenatively. This appoach is called the altenative imization and is widely used in many applications such as (Kang, Gauman, and Sha 2011) and (Jia, Tang, and Wang 2012). Fist, we fix D and W to optimize A. We initialize W as identity matix and D using the dictionay leaning algoithm in (Maial et al. 2010) espectively. With D and W fixed, the optimization of A can be obtained as follows: A + β X (m) D (m) A (m) 2 F + n m A (n) W (m) A (m) 2 F. J l=1 λ m A (m) :,Ω l 1,2 (6) E.(6) is a poblem of multi-modal goup spase coding and we use block-coodinate descent (Qin, Scheinbeg, and Goldfab 2010) (Fiedman, Hastie, and Tibshiani 2010) to solve it. Afte obtaining A, we then update the dictionaies D as follows: D X (m) D (m) A (m) 2 F s.t. k 1, k, m, This is a uadatically constained uadatic pogam (QCQP) poblem which can be solved using the method pesented in (Yang et al. 2010). Finally, we update W as follows: A (n) W (m) A (m) 2 F W n m (8) + (γ/β) W (m) 2 F, d (m) This is a set of idge egession poblem and can be woked out as follows: W (m) = A (n) A (m)t (A (m) A (m)t + (γ/β) I) 1, (9) whee I is the identity matix. The above pocedue iteates until the convegences of A, D and W ae achieved. SliM 2 fo multi-modal etieval Given a uey x (m) R Pm fom m-th modality, suppose we ae looking fo its simila data fom the n-th modality. Now we have jointly leaned the dictionay fo each modality data D = {D (1), D (2),, D (M) } and a set of mapping functions W = {W (1), W (2),, W (M) }. Fo the uey data x (m), we need to map x (m) into the space of n-th modality data. With the initialization as follows: α (m) 1 = α 2 x(m) D (m) α (m) 2 F + λ α (m) 1 α (n) x (n) = W (m) α (m) = D (n) α (n), (7) (10) 1072

4 Algoithm 1 The optimization of SliM 2 Input The labeled taining set of N pais data with M modalities fom J classes {(x (1) i, x (2) i,, x (M) i, l i )} {(X (1), X (2),, X (M), L)}. 1: Initialize D = {D (1), D (2),, D (M) } and W = {W (1), W (2),, W (M) }, 2: Optimize A = {A (1), A (2),, A (M) } by E.(6), 3: Update D = {D (1), D (2),, D (M) } with othe vaiables fixed using E.(7), 4: Update W = {W (1), W (2),, W (M) } with othe vaiables fixed using E.(9), 5: Repeat 2-4 until convegence. Output multi-modal dictionaies D and a set of mapping functions W we then obtain optimized ˆα (n) ˆα (m),ˆα (n) + β α (n) x (m) and ˆα (m) D (m) α (m) 2 F + x (n) W (m) α (m) The uey data x (m) data ˆx (n) as follows: ˆx (n) as follows: D (n) α (n) 2 F 2 F + λ m α (m) 1 + λ n α (n) 1. (11) can be mapped into n-th modality = D (n) ˆα (n). (12) Thus, all of data in the n-th modality which has the least distances to x (n) is anked as the etieved esults of the uey data. We summaize the optimization of SliM 2 in Algoithm 1 and multi-modal etieval by the SliM 2 in Algoithm 2. Expeiments In this section, we evaluate the pefomance of ou poposed SliM 2 when applied to coss-media etieval. We fist intoduce the data sets and evaluation citeions we adopted, then we elaboate paamete setting and tuning in ou expeiments. At last, we compae SliM 2 with othe state-of-the-at algoithms and demonstate the esults. Data Sets One of ou expeimental data sets is the Wiki Text-Image data (Rasiwasia et al. 2010). Wiki Text-Image contains 2173/693(taining/testing) text-image pais fom ten diffeent categoies. Afte SIFT featues (Lowe 1999) ae extacted, k-means clusteing is conducted to obtain the epesentation of bag-of-visual-wods (abbeviated as BoVW) (Fei-Fei, Fegus, and Peona 2004) fo each image. The tem feuency is used to obtain the epesentation of bagof-textual-wods (abbeviated as BoW) fo each text. Since the dimensions of texts and images ae impotant factos fo multi-modal data etieval, we set two kinds of diffeent dimensions fo compaisons: one is 500-dimension BoVW and 1000-dimension BoW, the othe is 1000-dimension BoVW and 5000-dimension BoW. Algoithm 2 The multi-modal etieval by SliM 2 Input The leaned multi-modal dictionaies D = {D (1), D (2),, D (M) } and a set of mapping functions W = {W (1), W (2),, W (M) } fom taining data and uey data x (m) 1: Initialize α (m) using E.(10), 2: Optimize ˆα (m) E.(11), R Pm in the m-th modality,α (n),ˆα (n) and coesponding etieval x (n) with othe vaiables fixed using 3: Update ˆx (n) using E.(12), 4: Repeat 2-3 until convegence. 5: the anked neighbos of ˆx (n). Output The etieved simila data in the n-th modality The othe data set we used is the NUS-WIDE data set. Each image with its annotated tags in NUS-WIDE can be taken as a pai of image-text data. We only select those pais that belong to one of the 10 lagest classes with each pai exclusively belonging to one of the 10 classes. We use the 500-dimension BoVW based on SIFT featues fo the epesentation of each image and 1000-dimension tags fo the epesentation of each text as the authos supplied. Evaluation Methods Thee ae many evaluation citeia fo coss-modal etieval algoithms such as mean aveage pecision (MAP), aea unde cuve (AUC) and pecision ecall cuves. Most of them ae based on the etieved anking list of ueies. Ideally, given labeled pais of image-text, an appopiately coect etieved esult can be one that belongs to the same categoy as the uey data (Shama et al. 2012) o the coesponding uniue one paied with the uey (Jia, Salzmann, and Daell 2011). The fist one epesents the ability of leaning disciative coss-modal mapping functions while the late one eveals the ability of leaning coesponding latent concepts. In this pape, we use both of them as follows: MAP : MAP is defined hee to measue whethe the etieved data belong to the same class as the uey (elevant) o does not belong to the same class (ielevant). Given a uey (one image o one text) and a set of its coesponding R etieved data, the Aveage Pecision is defined as AP = 1 L R pec()δ(), (13) =1 whee L is the numbe of elevant data in the etieved set, pec() epesents the pecision of the etieved data. δ() = 1 if the th etieved datum is elevant to the uey and δ() = 0 othewise. MAP is defined as the aveage AP of all the ueies. Same as (Zhen and Yeung 2012), we set R = 50 in the expeiments. Pecentage: Since thee is only one gound-tuth match fo each image/text, to evaluate the multi-modal pefomance we can esot to the position of the gound-tuth textt/image in the anked list obtained. In geneal, one image (o text) is consideed coectly etieved if it appeas in the fist t pecent of the anked list of its coesponding 1073

5 NUS-WIDE Image Quey Text Text Quey Image CCA GMA SCDL SliM Table 3: The pefomance compaison in tems of MAP scoes on NUS-WIDE data set. The esults shown in boldface ae best esults. NUS-WIDE Image Quey Text Text Quey Image CCA GMA SCDL SliM Table 4: The pefomance compaison in tems of Pecentage scoes on NUS-WIDE data set. The esults shown in boldface ae best esults. etieved texts (o images) accoding to (Jia, Salzmann, and Daell 2011). t is set to eual to 0.2 in ou expeiments. Compaed Methods We devise ou compaed algoithms as follows : compae with one of the popula taditional methods only utilizing the pai-wise infomation, one of ou countepats and the unsupevised dictionay leaning method with a mapping function coss econstuction coefficients. The compaed algoithms with ou poposed SliM 2 ae listed as follows: Canonical Coelational Analysis (CCA): CCA is the classical method in coss modal etieval which leans a common space acoss multi-modal data. Genealized Multiview Analysis (GMA): GMA is a supevised method in coss-modal etieval which utilizes both pai-wised and label infomation of multi-modal data. As stated by authos (Shama et al. 2012), GMA is a supevised kenelizable extension of CCA and maps data in diffeent modality spaces to a single (non) linea subspace. Semi-coupled Dictionay Leaning (SCDL): SCDL (Wang et al. 2012) is an unsupevised dictionay leaning appoach to lean a pai of dictionaies and a mapping function acoss two-views in image domains, hee we conduct SCDL to multi-modal data. Paamete Tuning Fo paamete tuning, we split the taining data sets into 5 folds and test on each fold with the emaining 4 as taining data to do coss validation. β, γ, λ m (m {1, 2}) and K ae tuning paametes in ou expeiments. We pefom gid seach stategy on the fist 4 folds to set λ m (m {1, 2}) and line-seach method fo the othe paametes. The setting of β, γ, λ 1, λ 2 and K on Wiki data set is 1, 0.1, 0.1, 0.01 and 200, espectively while 0.01, 1, 0.01, 0.01 and 128 on NUS- WIDE data set. Hee, λ 1 is the egulaization paamete coesponding to image modality while λ 2 coesponds to text modality. Pefomance Compaisons Fo the Wiki Text-Image data set, the pefomance by each algoithm is given in table 1 and table 2 in tems of MAP and Pecentage espectively. Fo NUS-WIDE data, the pefomance by each algoithm is given in table 3 and table 4 in tems of MAP and Pecentage espectively. In ou expeiments, we can submit one image to etieve texts (Image uey Text), o submit one text to etieve images (Text uey Image). Fom the expeiments, we can make the following obsevations: Fo Image uey Text, in geneal, dictionay leaning based methods (SCDL and SliM 2 ) ae bette than diect mapping-based methods (CCA and GMA) on image uey text case in all of metics fo the two data sets, and moeove SliM 2 achieves the best pefomances. This is due to that SCDL and SliM 2 lean the multi-modal mapping functions fom spase codes instead of BoW/BoVW with spase codes obtaining though the imization of econstuction eos. The intoduction of class label futhe boosts the multi-modal etieval. Fo Text uey Image, the poposed SliM 2 achieves best pefomances in tem of Pecentage metic ove Wiki data set. Since images and texts ae paied in ou expeiments, Pecentage is moe accuate fo tue pefomance. CCA shows a good pefomance ove NUS-WIDE data set fo pecentage because the annotated tags in NUS-WIDE ae manually selected and thee is highly-ualified coelation between images and tags. Fo diffeent algoithms, the algoithms utilize pai-wise infomation pefom bette on Pecentage with algoithms utilized label infomation bette on MAP. Figue 1 illustates one example of image uey text and one example of text uey image ove Wiki image-text data set. The etieved esults by SliM 2 (top ow) and GMA (bottom ow) ae compaed. Fo the example of image uey text, we use the coesponding images of etieved texts to demonstate the esults. Though all of etieved texts come fom the spots categoy same as the uey image, and stongly coespond to the uey image, the esult by SliM 2 is moe visually consistent with the uey image. Fo the example of text uey image, the uey text is about paks fom geogaphy categoy. The etieved images by SliM 2 all come fom geogaphy categoy, while the fist etieved image and the last one by GMA come fom histoy categoy. Fom the undelined wods in the uey text descibing the semantics of this uey text, we can obseve that the etieved images by SliM 2 ae moe semantically coelated with the uey text than that of GMA. Conclusion SliM 2 is poposed in this pape fo multi-modal etieval. SliM 2 can utilize the class infomation to jointly lean disciative multi-modal dictionaies as well as mapping functions between diffeent modalities. We have demonstated the supeio pefomance of SliM 2 in tems of MAP and Pecentage fo two data sets. 1074

6 Wiki BoVW(500D),BoW(1000D) BoVW(1000D),BoW(5000D) Image Quey Text Text uey Image Image Quey Text Text uey Image CCA GMA SCDL SliM Table 1: The pefomance compaison in tems of MAP scoes on Wiki data set. 500-dimensional bag of visual wods (BoVW) and 1000-dimensional bag of textual wods (BoW), as well as 1000-dimensional bag of visual wods (BoVW) and dimensional bag of textual wods (BoW), ae used to epesent each image and text espectively. The esults shown in boldface ae best esults. Wiki BoVW(500D),BoW(1000D) BoVW(1000D),BoW(5000D) Image Quey Text Text Quey Image Image Quey Text Text Quey Image CCA GMA SCDL SliM Table 2: The pefomance compaison in tems of Pecentage scoes on Wiki data set. 500-dimensional bag of visual wods (BoVW) and 1000-dimensional bag of textual wods (BoW), as well as 1000-dimensional bag of visual wods (BoVW) and 5000-dimensional bag of textual wods (BoW), ae used to epesent each image and text espectively. The esults shown in boldface ae best esults. SliM2 GMA Image Quey Text Fanno Ceek passes though o nea 14 paks in seveal juisdictions. The Potland Paks and Receation Depatment manages thee: Hillsdale Pak, with picnic tables and a dog pak nea the headwates; Albet Kelly Pak, with unpaved paths, picnic tables, play aeas, and Wi-Fi noth of the ceek about fom the mouth, and the Fanno Ceek Natual Aea, noth of the ceek about fom the mouth. Text Quey Image Images coesponding to the top etieved texts Top etieved images SliM2 GMA Figue 1: Two examples of image uey text and text uey image ove Wiki data set by SliM 2 (top ow) and GMA (bottom ow). Fo the example of image uey text, we use the coesponding images of etieved texts to demonstate the esults. The uey image comes fom the spots categoy and all of etieved texts (and thei coesponding images) also come fom spots categoy. Fo the example of image uey text, the uey text is about paks fom geogaphy categoy. The undelined wods in the uey text descibe the semantics of the uey text. All of etieved images by SliM 2 come fom geogaphy categoy, and the second and the thid etieved images by GMA come fom geogaphy categoy while the othe two come fom histoy categoy. Acknowledgements This wok is suppoted by 973 Pogam (No. 2012CB316400), NSFC ( , ) and 863 pogam (2012AA012505). 1075

7 Refeences Bengio, S.; Peeia, F.; Singe, Y.; and Stelow, D Goup spase coding. Advances in Neual Infomation Pocessing Systems 22: Blei, D., and Jodan, M Modeling annotated data. In Poceedings of the 26th annual intenational ACM SIGIR confeence on Reseach and development in infomaion etieval, ACM. Blei, D.; Ng, A.; and Jodan, M Latent diichlet allocation. the Jounal of machine Leaning eseach 3: Fei-Fei, L.; Fegus, R.; and Peona, P Leaning geneative visual models fom few taining examples: an incemental bayesian appoach tested on 101 object categoies. In Compute Vision and Patten Recognition Wokshop, CVPRW 04. Confeence on, IEEE. Fiedman, J.; Hastie, T.; and Tibshiani, R A note on the goup lasso and a spase goup lasso. axiv pepint axiv: Hotelling, H Relations between two sets of vaiates. Biometika 28(3/4): Jia, Y.; Salzmann, M.; and Daell, T Factoized latent spaces with stuctued spasity. Advances in Neual Infomation Pocessing Systems 23: Jia, Y.; Salzmann, M.; and Daell, T Leaning coss-modality similaity fo multinomial data. In Compute Vision (ICCV), 2011 IEEE Intenational Confeence on, IEEE. Jia, K.; Tang, X.; and Wang, X Image tansfomation based on leaning dictionaies acoss image spaces. IEEE Tansactions on Patten Analysis and Machine Intelligence. Kang, Z.; Gauman, K.; and Sha, F Leaning with whom to shae in multi-task featue leaning. In Poceedings of the 28th Intenational Confeence on Machine Leaning, Lowe, D. G Object ecognition fom local scaleinvaiant featues. In Compute vision, The poceedings of the seventh IEEE intenational confeence on, volume 2, Ieee. Maial, J.; Bach, F.; Ponce, J.; Sapio, G.; and Zisseman, A Supevised dictionay leaning. axiv pepint axiv: Maial, J.; Bach, F.; Ponce, J.; and Sapio, G Online leaning fo matix factoization and spase coding. The Jounal of Machine Leaning Reseach 11: Monaci, G.; Jost, P.; Vandegheynst, P.; Mailhe, B.; Lesage, S.; and Gibonval, R Leaning multimodal dictionaies. Image Pocessing, IEEE Tansactions on 16(9): Olshausen, B.; Field, D.; et al Spase coding with an ovecomplete basis set: A stategy employed by vi? Vision eseach 37(23): Putthividhy, D.; Attias, H.; and Nagaajan, S Topic egession multi-modal latent diichlet allocation fo image annotation. In Compute Vision and Patten Recognition (CVPR), 2010 IEEE Confeence on, IEEE. Qin, Z.; Scheinbeg, K.; and Goldfab, D Efficient block-coodinate descent algoithms fo the goup lasso. Pepint. Rasiwasia, N.; Costa Peeia, J.; Coviello, E.; Doyle, G.; Lanckiet, G.; Levy, R.; and Vasconcelos, N A new appoach to coss-modal multimedia etieval. In Poceedings of the intenational confeence on Multimedia, ACM. Salomatin, K.; Yang, Y.; and Lad, A Multi-field coelated topic modeling. SDM Shama, A.; Kuma, A.; Daume, H.; and Jacobs, D Genealized multiview analysis: A disciative latent space. In Compute Vision and Patten Recognition (CVPR), 2012 IEEE Confeence on, IEEE. Tibshiani, R Regession shinkage and selection via the lasso. Jounal of the Royal Statistical Society. Seies B (Methodological) Todoovic, S., and Nechyba, M Detection of atificial stuctues in natual-scene images using dynamic tees. In Patten Recognition, ICPR Poceedings of the 17th Intenational Confeence on, volume 1, IEEE. Vitanen, S.; Jia, Y.; Klami, A.; and Daell, T Factoized multi-modal topic model. axiv pepint axiv: Wang, S.; Zhang, L.; Liang, Y.; and Pan, Q Semi-coupled dictionay leaning with applications to image supe-esolution and photo-sketch synthesis. In Compute Vision and Patten Recognition (CVPR), 2012 IEEE Confeence on, IEEE. Wight, J.; Yang, A.; Ganesh, A.; Sasty, S.; and Ma, Y Robust face ecognition via spase epesentation. Patten Analysis and Machine Intelligence, IEEE Tansactions on 31(2): Wu, F.; Zhang, H.; and Zhuang, Y Leaning semantic coelations fo coss-media etieval. In Image Pocessing, 2006 IEEE Intenational Confeence on, IEEE. Yang, M.; Zhang, L.; Yang, J.; and Zhang, D Metaface leaning fo spase epesentation based face ecognition. In Image Pocessing (ICIP), th IEEE Intenational Confeence on, IEEE. Zhen, Y., and Yeung, D.-Y A pobabilistic model fo multimodal hash function leaning. In Poceedings of the 18th ACM SIGKDD intenational confeence on Knowledge discovey and data ing, ACM. Zhuang, Y.-T.; Yang, Y.; and Wu, F Mining semantic coelation of heteogeneous multimedia data fo coss-media etieval. Multimedia, IEEE Tansactions on 10(2):