Cross-lingual Pseudo Relevance Feedback Based on Weak Relevant Topic Alignment

Transcription

1 Cross-lngual Pseudo Relevance Feedback Based on Weak Relevant opc Algnment WANG Xu-wen Insttute of Medcal Informaton & Lbrary, Chnese Academy of Medcal Scences, Beng ZHANG Qang State Grd Electrc Power Research Insttute, Beng pr.sgcc.com.cn WANG Xao-e Beng Unversty of Posts and elecommuncatons, Beng, LI Jun-lan Insttute of Medcal Informaton & Lbrary, Chnese Academy of Medcal Scences, Beng Abstract In ths paper, a cross-lngual pseudo relevance feedback (PRF) model based on weak relevant topc algnment (WRA) s proposed for cross language query expanson on unparallel web pages. opcs n dfferent languages are algned on the bass of translaton. Useful expanson terms are extracted from weak relevant topcs accordng to the blngual term smlarty. Experment results on web-derved unparalell data show the contrbuton of the WRA-based PRF model to cross language nformaton retreval. 1 Introducton he problem of word msmatch between queres and retreved documents s partcularly serous n cross language nformaton retreval (CLIR). he ntegraton of query expanson technques and translaton knowledge s consdered as an effectve way to mprove the CLIR performance (Ballesteros and Croft, 1998; Qu et al., 2000). Pseudo relevance feedback (PRF) s one of the useful query optmzng technologes for monolngual retreval tasks (Roccho, 1971; Ruthven and Lalmas, 2003). As to the CLIR task, researchers lad more efforts on establshng an effectve cross-lngual PRF mechansm on the bass of the relevance and complementary of blngual web pages (Ballesteros and Croft, 1997; Lavrenko et al., 2002). One of the key problems s how to choose useful or relevant blngual expanson terms. ypcal cross-lngual PRF methods assume the top retreved documents are relevant and perform feedback calculatons on the whole pseudo-relevant document level. Hgh-frequency words are often used for expandng orgnal queres. In recent years, topc models were appled to more and more multlngual tasks (Wang et al., 2009; Vulc et al., 2013). Ganguly (2012) proposed an mproved cross-lngual topcal relevance model based on the latent topcs of top ranked documents. Wang (2013) proposed a cross-lngual PRF model based on blngual topcs and showed better results on parallel or comparable corpus. However, the hypothess of common shared blngual topcs s not always sutable for unparallel documents, snce they are often poor n content relevance. In most cases, web pages retreved from dfferent language felds for a specfc query may lack of parallelsm. here may be some common topcs shared by the retreved documents n both languages, but there are also some specfc topcs for source language retreval results or target language retreval results respectvely. Only the former common shared topcs would be helpful to cross-lngual PRF th Pacfc Asa Conference on, Informaton and Computaton pages Shangha, Chna, October 30 - November 1, 2015 Copyrght 2015 by Xuwen Wang, Qang Zhang, Xaoe Wang and Junlan L

2 In ths paper, we assume that retreved results n dfferent languages have ndependent topcal dstrbuton, but there may be some overlappng topcs that descrbe smlar or relevant content. he overlappng content s defned as weak relevant topcs. A cross-lngual PRF model based on weak relevant topc algnment (WRA) s proposed for modelng the weak correlaton between unparallel documents. Relevant topcs n dfferent languages are algned based on translaton equvalent. hen useful expanson terms are extracted from relevant topcs accordng to ther blngual smlarty. he structure of ths paper s organzed as follows: secton 2 ntroduces the structure of the WRA-based cross-lngual PRF model; secton 3 presents the comparson experment of dfferent PRF methods on web-derved data; the fnal secton shows our concluson. 2 Method It s assumed that cross-lngual retreval results of a specfc query, although lack of parallelsm or comparablty, may contan some relevant content. Frstly, we perform monolngual topc modelng for source language documents D S and target language documents D respectvely. A wdely used topc modelng method s the Latent Drchlet Allocaton (LDA) model, whch s proposed by Ble (2003). So the LDA model s employed to generate canddate topc sets. Secondly, topcs n dfferent languages are algned based on translaton equvalence. hrdly, useful expanson terms n algned topcs are selected on the bass of translaton as well as web co-occurrence features. Fgure 1 shows the process of weak relevant topc algnment and expanson terms extracton. opc Modelng Canddate Englsh opcs Blngual Dctonary Web D Z D s opc Modelng Z s Canddate Chnese opcs Algned opcs Relevant Expanson erms Fgure. 1. Weak relevant topc algnment and extracton of relevant expanson terms 2.1 Weak Relevant opc Algnment For a specfc query and ts retreved blngual documents, we use the Gbbs samplng method for LDA nference (Han and Stbor, 2010) and generate two topc sets n dfferent languages. We need some clue for selectng canddate topcs from the two topc sets. opcs that have close relaton wth the query or top-ranked documents are adopted as our canddate topcs. hen relevant blngual topc pars wth better translaton equvalence are collected as the algned topcs. 1. Collectng canddate topcs Query related canddate topcs: opcs ncludng source language query terms Q S or query translaton terms Q are assumed to have drectly correlaton wth users query ntenton, namely query related topcs Z Q, see formula (1) and (2). n S QS ZS 0 pq zs Z p (1) S Q 0 zs zs 1 n Q Z 0 pq z Z p (2) Q 0 z z 1 Alternatve related canddate topcs: he top M retreved documents are supposed to be more relevant wth users query ntenton. So the top k topcs wth hgher probablty n the topc dstrbuton z of the top M documents are adopted as the alternatve related topcs Z E, see as formula (3) and (4). Both of the query related topcs Z Q and the alternatve related topcs Z E are collected as the canddate blngual topc set Z C, see as formula (5). S Z E k arg maxθ d z (3) S dd M z 530

3 Z E dd M Z C k arg maxθ Q z E z d (4) Z Z (5) 2. opc algnment Canddate topcs n dfferent languages are algned accordng to ther translaton equvalence based on the machne-readable dctonary (MRD). For a source language topc z s and a target language topc z t, whch contan N s terms or N t terms respectvely, the topcal algnment rate s computed as formula (6). he m n numerator s the amount of terms n the source language topc z s that have translaton n the target language topc z t, the n s the amount of terms n target language topc z t that have translaton n source language topc z s. m n f zs, z t (6) N N CH40 CH62 CH74 CH99 CH86 Fgure. 2. he algnment rate between the canddate blngual topcs of Informaton retreval Fgure 2 shows the algnment rate between canddate blngual topcs of the query Informaton retreval. he b-drectonal translaton process can be regard as a mutual mult-votng game between topcs n dfferent languages. he hgher rate mples more latent relevance. Z S Z CH40 s CH62 CH74 CH86 CH99 EN21 EN23 EN25 EN27 EN67 Fgure. 3. he algnment of Chnese-Englsh canddate topcs of the query Informaton retreval w, w pw, w t EN23 EN25 EN27 EN67 EN21 Fgure 3 shows the algnment relatonshp between canddate blngual topcs of the query. he sold arrow wth two drectons represents a mutual algnment between two topcs, snce they vote each other wth the hghest rate. In ths case, three couples of topcs are algned successfully. 2.2 Selectng Relevant Expanson erms Cao et al. (2008) analyzed the potental nfluence of dfferent terms to the performance of nformaton retreval tasks, and concluded that useful terms for query expanson n pseudo relevant documents only account for 18% n hgh frequency terms. oo many expanson terms may reduce the effcency of retreval systems (Whte and Marchonn, 2007). In our work, terms from canddate topcs are sorted nto three categores. he frst category contans semantcally relevant terms that have translaton or synonymy wth orgnal queres. erms n the second category have no drect relatonshp wth queres, but they are essental content n descrbng dentcal themes n blngual context. he last category contans rrelevant nosy terms that should be fltered out. o select useful expanson terms effectvely, a blngual term smlarty score s computed based on web-derved data. For each par of algned topcs, a source language term and a target language term are organzed as a conunctve query w s + w t for the real tme web searchng. In the real web searchng, terms n dfferent languages often co-occur n the ttle, snppet or URL of a retreved multlngual webpage. So, the web co-occurrence of each par of terms from algned topcs would be counted, see formula (7). he bnary functon n formula (8) represents the translaton relatonshp between the term w s and wt. he blngual smlarty score of the term par s the lnear combnaton of web co-occurrence and the translaton feature, see as formula (9). he parameter λ s the weghtng coeffcent. In each target language topc, terms are ranked accordng to the smlarty score wth the source lan- s t Sm q, w. erms guage query terms, namely wth smlarty score lower than the threshold μ wll be fltered out. # retreval recordsncludng Nc f C (7) # retreval recordsfrom IR system w, w N 531

4 1, only f w, w w, w rans w, w 0, are mutual translaton f (8) Sm other, w λ f w, w 1 λ f w, w, (0 λ 1) w (9) C 2.3 Cross Pseudo Relevance Feedback Based on WRA Based on the above algorthm, relevant terms are obtaned for cross-lngual query expanson. Fgure 4 shows the CLIR process wth WRA-based PRF mechansm. Source Query Source IR Source Retreval Result Query ranslaton WRAbased PRF arget Query arget IR arget Retreval Result Fgure. 4. CLIR process wth cross language pseudo relevance feedback based on WRA. 3 Experments 3.1 Expermental settng and Data We perform cross-lngual PRF experments on a self-constructed CLIR system, namely CP-CLIR system (Wang et al., 2013). As a prototype system, t contans a text pre-processng module, a query translaton module, a retreval model (Indr 5.2) and the pseudo relevance feedback module, whch ntegrated varous PRF mechansms. he CP-CLIR system could access web pages on lne and retreve local multlngual database automatcally. A Web-derved Chnese-Englsh corpus was collected to smulate the real cross language web search task. he source language query set was selected from the Chnese scence and technology concepts on CNKI. Each query contans 1 to 3 word tokens, totally 54 queres. he target language queres were the Englsh translaton of the Chnese queres, obtaned from the query translaton module. he blngual retreval documents were collected from Google s real tme retreval results. op 10 source language pages were crawled for each Chnese query, snce most web users pay more attenton to the top-ranked results n the retreval lst. he target language pages were retreved va Google s cross-lngual retreval. otally 1080 web pages were collected. hen 20 queres wth poor comparable retreval results were selected as our test set, totally 400 web pages. Other queres were saved as our tranng set, totally 34 queres and 680 web pages. All of the collected web pages were cleaned by the text preprocessng module and then be ndexed by Indr 5.2. Snce the typcal assessment crtera, such as precson or recall, shows no sgnfcant dfference on the relatvely small dataset, we take ndcg (Dscounted Cumulatve Gan) to evaluate the rankng effect of retreval results. 27 volunteers were nvted to udge the relevance of blngual documents. 3.2 Parameters All the parameters were tuned on the bass of our tranng set. It was observed that topcs from the top 1 document as well as the query related topc Z Q contrbuted most to the best rankng results. So the parameters of topc algnment were confgured as follows, the alternatve document number M=1, the alternatve topc number k=2. Each query has 1.5 par of weak relevant topcs on average. he flterng threshold of term probablty n each topc σ= he weghtng coeffcent of the blngual term smlarty score was set as λ=0.05, and ts threshold for flterng terms μ=0.85. he hyper parameters of the LDA model were optmzed based on the tranng set, as follows, 0.1, s 0.01, t he number of tranng teratons was Comparatve Experments o examne the feedback effect of proposed method, we chose the normal CLIR results wthout PRF modulaton as our baselne. 532

5 Varous PRF methods, such as VSM-based PRF framework, LDA-based PRF model, blngual LDA-based PRF model, etc., are also conducted before or after the query translaton stage of CLIR, namely comparatve experments. 3.4 Results Fgure 5 shows the CLIR results employng dfferent PRF methods on unparallel documents. he frst column s the result of CLIR wthout PRF mechansm. he second to the forth column show the results of PRF based on the Vector Space Model (VSM), namely pre-translaton VSM-based PRF, post-translaton VSM-based PRF and combned VSM-based PRF. he ffth to the seventh column show the results of PRF based on monolngual topc model, namely pre-translaton LDA-based PRF, post-translaton LDA-based PRF and combned LDA-based PRF. he eghth column s the result of blngual LDA-based PRF, whch performs ntegrated feedback on the bass of the blngual LDA model. he last column shows the result of proposed WRA-based cross-lngual PRF CLIR_no_PRF VSM_prePRF VSM_postPRF Mean ndcg VSM_combPRF LDA_prePRF LDA_postPRF LDA_combPRF B_opc_PRF WRA_PRF Fgure. 5. Comparson of cross-lngual PRF based on WRA and other PRF methods. It can be observed that the VSM-based PRF methods ntroduced too much nose, snce the feedback calculaton was performed on the entre document level. he LDA-based PRF methods showed a slghtly better performance than former methods, verfyng the fact that a fne-graned topc may ntroduce more relevant terms nto query expanson. However, the PRF method based on blngual LDA model, whch used to acheve better performance than monolngual models on parallel documents, showed no advantage here, snce the poor qualty of the unparallel feedback documents lmted the effectveness of topcal PRF methods. In spte of the nterference from the unparalleled documents, the WRA-based PRF model acheved the hghest mprovement for CLIR. Expanson terms from algned topcs, whch were selected based on the translaton and web co-occurrence features, showed clear relevance wth orgnal queres. On one hand, nosy terms were fltered out effectvely and the amount of expanson terms was reduced sharply. On the other hand, the remaned expanson terms showed postve mpact on the performance of CLIR on unparallel documents. 4 Concluson hs paper descrbes a way to dscover useful nformaton from unparallel retreval results for crosslngual pseudo relevance feedback. A cross language PRF model based on weak-relevant topc algnment s proposed. In comparson wth varous PRF methods, WRA-based PRF model showed better performance and robustness n the CLIR task on less comparable documents. So t s proved to be more sutable for web orented tasks. It s worth notng that the effect of expanson terms for cross-lngual PRF s very complcated. he qualty and quantty of expanson terms, whch are nfluenced by the qualty of translaton as well as feedback documents, should be controlled carefully. oo many expanson terms may drown out valuable nformaton, so the quantty of expanson terms s reduced sharply n our work. Nose terms are removed from canddate expanson terms effectvely, so that useful terms may acheve postve feedback performance. As to the further work, t wll be necessary to ntroduce more multlngual knowledge resources nto the cross-lngual PRF mechansm, such as Wkpeda, multlngual ontology, as well as semantc web knowledge, etc. Rch knowledge resources wll be a helpful supplement for choosng relevant expanson terms, and furthermore, mprovng the performance of PRF model n CLIR tasks. 533

6 References Andrzeewsk D, Buttler D. Latent topc feedback for nformaton retreval [J]. Proceedngs of the 17th Acm Sgkdd Internatonal Conference on Knowledge Dscovery and Data Mnng San Dego Ca Usa August , 2011: Ballesteros L, Croft W. Statstcal Methods for Crosslanguage Informaton Retreval [J] Ballesteros L, Croft W. Phrasal translaton and query expanson technques for cross-language nformaton retreval [J]. Proceedngs of the 20th Annual Internatonal Acm Sgr Conference on Research and Development n Informaton Retreval, 1997, 31(SI): Ble D M, Ng A Y, Jordan M I. Latent drchlet allocaton [J]. he Journal of machne learnng research, 2003, 3: Cao G, Ne J Y, Gao J, et al. Selectng good expanson terms for pseudo-relevance feedback[c]//proceedngs of the 31st annual nternatonal ACM SIGIR conference on Research and development n nformaton retreval. ACM, 2008: Ganguly Debass and Levelng Johannes and Jones Gareth J F Cross-lngual topcal relevance models [C]. 24th Internatonal Conference on Computatonal Lngustcs, Han X, Stbor. Effcent Collapsed Gbbs Samplng for Latent Drchlet Allocaton[J]. Jmlr, J. J. Roccho. Relevance feedback n nformaton retreval. [J]. In the SMAR Retreval System: Experments n Automatc Document Processng, 1971: Lavrenko V, Choquette M, Croft W. Cross-lngual relevance models[j]. Proceedngs of the 25th Annual Internatonal Acm Sgr Conference on Research and Development n Informaton Retreval, Orengo V, Huyck C. Relevance feedback and cross-language nformaton retreval[j]. Informaton Processng and Management an Internatonal Journal, 2006, 42(5): Qu Y, Elerman A, Jn H. he Effect of Pseudo Relevance Feedback on M-Based CLIR[J]. Rao 2000 Content Based Mult Meda Informaton Access Csas, Ruthven I, Lalmas M. A survey on the use of relevance feedback for nformaton access systems[j]. he Knowledge Engneerng Revew, Vulc I, De Smet W, Moens M. Cross-language nformaton retreval models based on latent topc models traned wth document-algned comparable corpora[j]. Informaton Retreval, Wang A, L Y, We W. Cross language nformaton retreval based on LDA [J]. Intellgent Computng and Intellgent Systems. ICIS Wang Xu-wen, Wang Xao-e, Sun Yue-png, Cross-lngual pseudo relevance feedback based on blngual topcs, Journal of Beng Unversty of Posts and elecommuncatons, Volume: 36; Issue 4; (JA) Pages: 81-84, August Wang X, Zhang Q, Wang X, et al. LDA based PSEUDO relevance feedback for cross language nformaton retreval[c]// Cloud Computng and Intellgent Systems (CCIS), 2012 IEEE 2nd Internatonal Conference on- IEEE, 2012: Wang X, Wang X, Zhang Q. A Web-Based CLIR System wth Cross-Lngual opcal Pseudo Relevance Feedback [J]. Lecture Notes n Computer Scence, Volume 8138 LNCS, Whte, R.W., & Marchonn, G. (2007). Examnng the effectveness of real-tme query expanson. Informaton Processng &Management, 43(3),