Merging Results by Using Predicted Retrieval Effectiveness

Transcription

1 Mergng Results by Usng Predcted Retreval Effectveness Introducton Wen-Cheng Ln and Hsn-Hs Chen Departent of Coputer Scence and Inforaton Engneerng Natonal Tawan Unversty Tape, TAIWAN Abstract. In ths paper we proposed several ergng strateges to erge the result lsts of each nteredate runs n dstrbuted MLIR. The predcton of retreval effectveness was used to adjust the slarty scores of docuents n the result lsts. We ntroduced three factors affectng the retreval effectveness,.e., the degree of translaton abguty, the nuber of unknown words and the nuber of relevant docuents n a collecton for a gven query. The results show that the noralzedby-top-k ergng wth translaton penalty and collecton weght outperfors the other ergng strateges except the raw-score ergng. Multlngual Inforaton Retreval abbrevated as MLIR facltates the uses of queres n one language to access docuents n varous languages. Most of the prevous approaches (Oard and Dekea, 998) focused on how to unfy the language usages n queres and docuents. The adaptaton of tradtonal nforaton retreval systes has been consdered. Query translaton and docuent translaton ethods have been ntroduced. The resources used n the translaton have been explored. In real world, ultlngual docuent collectons are dstrbuted fro varous resources, and anaged by nforaton retreval syste of varous archtectures. How to ntegrate the results fro heterogeneous resources s one of the ajor ssues n MLIR. Mergng result lsts of ndvdual languages s a coonly adopted approach. Docuent collectons of each language are ndexed and retreved separately, and the result lsts of each docuent collecton are erged nto a ultlngual result lst. The goal of result lsts ergng s to nclude as ore relevant docuents as possble n the fnal result lst and to ake relevant docuents have hgher ranks. Several attepts were done on ths proble (Peters, 00). The splest ergng ethod s raw-score ergng, whch sorts all the docuents by ther orgnal slarty scores, and then selects the top ranked docuents. The second approach, round-robn ergng, nterleaves the results of each run based on the rank of each docuent. The thrd approach s noralzed-score ergng. For each topc, the slarty score of each docuent s dvded by the axu score n each result lst. After adjustng scores, all results are put nto a pool and sorted by the noralzed score. Ln and Chen (00a, 00b) proposed noralzed-by-top-k ergng to avod the drawback of noralzedscore ergng. Translaton penalty s also consdered durng ergng result lsts. The perforance of noralzed-by-top-k wth translaton penalty s slar to that of raw-score ergng. Moulner and Molnasalgado (00) proposed collecton-weghted noralzed score to erge result lsts. The noralzed collecton score s used to adjust the slarty score between a docuent and a query. Collecton score only reflects the slarty of a (translated) query and a docuent collecton. Ths ethod could fal f a query s not translated well. Savoy (00) used logstc regresson to predct the relevance probablty of docuents accordng to the docuent score and the logarth of the rank. Ths ethod does not consder the qualty of query translaton ether. Furtherore, the relatonshp between the rank and the relevance of a docuent s not strong. Braschler, Göhrng and Schäuble (00) proposed feedback ergng that nterleaves the results accordng to the propostons of the predcted aount of relevant docuents n each docuent collecton. The aount of relevant nforaton was estated by the porton of overlap between the orgnal query and the deal query constructed fro the top ranked docuents. The experental results showed that feedback ergng had lttle pact. In ths paper, we wll explore several ergng strateges. The basc dea of our ergng strateges s adjustng the slarty scores of docuents n each result lst to ake the ore coparable and to reflect the confdence n retreval effectveness. We assue that the portance of each nteredate run depends on ther retreval perforance. We ntroduced three factors affectng the retreval effectveness,.e., the degree of translaton abguty, the nuber of unknown words and the nuber of relevant docuents n a collecton for a gven query. The rest of ths paper s organzed as follows. Secton descrbes our ergng strateges. Secton

2 3 shows the IR odel and query translaton technque. Secton 4 dscusses the experental results. Secton 5 concludes the rearks. Mergng Strateges We a to nclude as ore relevant docuents as possble n the fnal result lst and to ake relevant docuents have hgher ranks durng ergng. If a result lst contans any relevant docuents n the top ranks,.e., t has good perforance, the top ranked docuents should be ncluded n the fnal result lst. On the other hand, f a result lst has few or even no relevant docuents, the fnal result lst should not contan any docuents fro ths lst. Thus, the hgher perforance an ndvdual run has, the ore portant t s. However, wthout the pror knowledge of a query n advance, t s challengng to predct the perforance of an ndvdual run for each docuent collecton. The slarty score between a docuent and a query s one of a few clues that are coon used. A docuent wth hgher slarty score sees to be ore relevant to a specfc query. Because docuent collectons are varous and the underlyng IR systes ay be dfferent, the slarty scores of a query wth dfferent collectons cannot be copared drectly. The basc dea of our ergng strateges s adjustng the slarty scores of docuents n each result lst to ake the ore coparable and to reflect the confdence n retreval effectveness. The characterstcs of the underlyng IR odel, the effects of the query translaton and the statstcs of ndvdual docuent collecton wll be addressed n the followng subsectons.. Noralzed by Top K Slarty scores reported by dfferent nforaton retreval systes ay dffer a lot fro each other. In vectorbased IR odel, the slarty score defned by cosne forula ranges fro 0 to, but the score ay be uch larger than when Okap syste (Robertson, et al., 998) s used. It s obvous that the scores cannot be copared drectly. Thus, slarty scores have to be noralzed to the sae range to ake the coparable at the frst step. The approach of noralzed-score ergng aps the slarty scores of dfferent result lsts to the values wthn the sae range. The ajor drawback s: f the axu score s uch hgher than the second one n the sae result lst, the noralzed-score of the docuent at rank would be ade lower even f ts orgnal score s hgh. Thus, the fnal rank of ths docuent ght be lower than that of the top ranked docuents wth slar orgnal scores n another result lst. A revsed score noralzaton ethod s proposed as follows. The orgnal score of each docuent s dvded by the average score of top k docuents nstead of the axu score. We call ths noralzed-by-top-k approach.. Translaton Penalty Slarty score reflects the degree of slarty between a docuent and a query. A docuent wth hgher slarty score sees to be ore relevant to the desred query. However, f the query s not forulated well, e.g., napproprate translaton of a query, a docuent wth hgh score ay stll not eet users nforaton needs. When the result lsts are erged, those docuents that have hgh, but ncorrect scores should not be ncluded n the fnal result lst. Thus, the effectveness of each ndvdual run has to be consdered n the ergng stage. When query translaton ethod s used to deal wth the unfcaton of language usages n queres and docuents, queres are translated nto target language and then the target language docuents are retreved. We can predct the ultlngual retreval perforance based on the translaton qualty. Intutvely, usng Englsh to access Englsh collecton s expected to have better perforance than usng t to access other collectons. Slarly, usng a blngual dctonary of ore coverage s expected to be better than usng dctonary of less coverage. Less abguous queres have also hgher tendency to acheve better translaton than ore abguous queres. Noralzaton n Secton. just reflects the sae coparson bass, but does not consder the above ssues. Two factors,.e., the degree of translaton abguty and the nuber of unknown words, are used to odel the translaton perforance. For each query, we copute the average nuber of translaton equvalents of query ters and the nuber of unknown words n each language par, and use the to copute the weghts of each cross-lngual run. The followng forula s proposed to deterne the weghts. 5 T U + W = c + c c3 50 n. () where W s the ergng weght of query n a cross-lngual run, T s the average nuber of translaton equvalents of query ters n query, U s the nuber of unknown words n query, n s the nuber of query ters n query, and c, c and c 3 are tunable paraeters, and c +c +c 3 =.

3 .3 Collecton Weght of Indvdual Docuent Collecton How any relevant docuents there are n a collecton for a gven query s also an portant factor for easurng retreval effectveness. If a docuent collecton contans ore relevant docuents, t could have ore contrbuton to the fnal result lst. Callan, et al. (995) proposed CORI net to rank dstrbuted collectons of the sae language for a query. Moulner and Molna-salgado (00) used collecton score to adjust the slarty score between a docuent and a query. In our approach, a collecton weght of a docuent collecton wth a query s defned as follows. A docuent collecton s vewed as a huge docuent and represented as a collecton vector. The th eleent n a collecton vector s the docuent frequency df of the th ndex ter n the collecton. Slarly, the th eleent n a query vector s the frequency of the th ndex ter n the query. Snce docuent collectons are n dfferent languages, we do not use nverse collecton frequency cf, whch s analogous to df. Cosne slarty forula shown as follows s used to copute the collecton weght, whch s ncluded to the ergng weght. W ' CW () = W + c 4 CW = qtf j df j (3) qtf j df j where W s the new ergng weght of query n an nteredate run, W s the ergng weght descrbed n Secton., CW s the collecton weght of a target collecton for query, c 4 s tunable paraeter, qtf j s the ter frequency of ndex ter j n query, df j s the docuent frequency of ndex ter j n a target collecton, and s the nuber of ndex ters..4 Predctng Retreval Effectveness by Lnear Regresson Now three factors,.e., the degree of translaton abguty, the nuber of unknown words and the nuber of relevant docuents for a gven query n a docuent collecton, are proposed to deterne the retreval effectveness. We use lnear regresson to predct the retreval effectveness accordng to the three factors. The orgnal slarty score of a docuent s noralzed by noralzed-by-top-k ethod, and the score of the predcted precson s added to the noralzed score. Docuents fro all collectons are sorted accordng to the adjusted slarty scores and the top ranked docuents are reported. 3 Query Translaton and Docuent Indexng In the experents, Okap IR syste was adopted to ndex and retreve docuents. The weghtng functon was BM5 (Robertson, et al., 998). The docuent set used n CLEF 003 Sall-ultlngual task conssts of Englsh, French, Geran and Spansh. The nubers of docuents n Englsh, French, Geran and Spansh docuent sets are 69,477, 9,806, 94,809 and 454,045, respectvely. The <HEADLINE> and <TEXT> sectons n Englsh docuents were used for ndexng. For Spansh docuents, the <TITLE> and <TEXT> sectons were used. Whle ndexng French and Geran docuents, the <TITLE>, <TEXT>, <TI>, <LD> and <TX> sectons were used. The words n these sectons were steed, and stopwords were reoved. All letters were transfored to the lower cases. We adopted stopword lsts and steers developed by Unversty of Neuchatel (Savoy, 00). Englsh queres were used as the source language queres and translated nto target languages,.e., French, Geran and Spansh. A dctonary-based approach was adopted. For each Englsh query ter, we found ts translaton equvalents by lookng up a dctonary. The frst two translaton equvalents wth the hghest occurrence frequency n the target language docuents were consdered as the target language query ters. If a query ter does not have any translaton equvalents, the orgnal Englsh query ter was reserved n the translated query. The dctonares we used are Ergane Englsh-French, Englsh-Geran and Englsh-Spansh dctonares. They are avalable at

4 4 Experents We subtted four runs n CLEF 003 Sall-ultlngual task. All runs used ttle and descrpton felds. The detals of each run are descrbed n the follows.. NTU4Topn The result lsts were erged by noralzed-by-top-k ergng strategy. The average slarty score of the top 00 docuents were used for noralzaton.. NTU4TopnTp In ths run, translaton penalty was consdered. The slarty scores of each docuent were frst noralzed by the average slarty score of the top 00 docuents and then ultpled a weght deterned by forula (). The values of c, c and c 3 were 0, 0.4 and 0.6, respectvely. In query translaton, an Englsh query ter whch dd not have any translaton equvalents by dctonary lookup was also used to retreve target language docuents. If such an Englsh ter occurs n the target language docuents, t can be vewed as a word slar to the other translated words when the ergng weght s coputed. Table lsts the nuber of Englsh query ters that do not have translaton, but occur n target language collecton. 3. NTU4TopnTpCw In ths run, the collecton weght was also consdered. We used forula () to adjust the slarty score of each docuent. The values of paraeters were the sae as run NTU4TopTp. The value of c 4 was NTU4TopnLnear We used lnear regresson to deterne the weghts of the three varables, ncludng the average nuber of translaton equvalents of query ters, the porton of unknown words n a query and the collecton weght of the target collecton, to predct the perforances of each nteredate run. CLEF 00 and 00 test sets were used as tranng data to estate the paraeters. The orgnal slarty score of a docuent was noralzed by noralzed-by-top-k ethod, and the score of the predcted precson was added to the noralzed score. The results of offcal runs are shown n Table. To copare the effectveness of our approaches wth the past ergng strateges, we also conducted several unoffcal runs that used raw-score ergng, noralzed-score ergng and round-robn ergng strateges. The average precson of optal ergng proposed by Chen (00) was regarded as an upper-bound, whch was used to easure the perforances of our ergng strateges. The perforances of the unoffcal runs are also shown n Table. The perforance relatve to optal ergng s enclosed n parentheses. The results show that the perforances of the ergng strateges we proposed were better than noralzed-score ergng and round-robn ergng, but worse than raw-score ergng. The experental results n CLEF 00 and NTCIR3 (Ln and Chen, 00a, 00b) show that noralzed-by-top-k ergng overcoes the drawback of noralzed-score ergng. In CLEF 003, the perforance of noralzedby-top-k ergng was stll better than noralzed-score ergng. The perforance dropped down slghtly after consderng translaton penalty. Fro Table 3, the perforance of Englsh-Spansh run was worse than the other nteredate runs, but the ergng weghts of three cross-lngual runs were slar. Ths s because that the average nuber of translaton equvalents and the nuber of unknown words of three cross-lngual runs dd not dffer too uch. After consderng the collecton weghts of each docuent collecton, the perforance was proved and was about 7.% ncrease to noralzed-by-top-k ergng. The perforance of usng the ergng weght predcted by lnear regresson was slghtly better than noralzed-by-top-k ergng, but worse than noralzed-by-top-k wth translaton penalty and collecton weght. Table. Nuber of Englsh query ters wthout translaton but n target language corpora Language French Geran Spansh # query ters # query ters wthout translaton equvalents # query ters wthout translaton equvalents but n target language (64.%) (7.43%) (58.57%) corpora

5 Table. Perforances of ergng strateges Run Average precson NTU4Topn (60.97%) NTU4TopnTp (60.5%) NTU4TopnTpCw (65.3%) NTU4TopnLnear 0.56 (6.08%) Raw score ergng 0.69 (69.5%) Noralzed score ergng (55.94%) Round-robn ergng 0.4 (57.8%) Optu ergng 0.44 Table 3. Perforances of nteredate runs Run # Topc Average precson Englsh -> Englsh Englsh -> French Englsh -> Geran Englsh -> Spansh Concluson Mergng proble s crtcal n dstrbuted ultlngual nforaton retreval. In ths paper, we proposed several ergng strateges to ntegrate the result lsts of collectons n dfferent languages. We assue that the portance of each nteredate run depends on ther retreval perforance. We ntroduced three factors affectng the retreval effectveness,.e., the degree of translaton abguty, the nuber of unknown words and the nuber of relevant docuents n a collecton for a gven query. Noralzed-by-top-k avods the drawback of noralzed-score ergng. The experental results show that consderng translaton penalty and collecton weght proves perforance. We also used lnear regresson to predct the retreval effectveness. The perforance of usng the ergng weght predcted by lnear regresson s slar to noralzed-by-top-k. The perforances of our ergng strateges were better than noralzed-score ergng and round-robn ergng, but were worse than raw-score ergng n sngle IR syste envronent. However, raw-scorng ergng s not workable f dfferent nforaton retreval systes are adopted. References Braschler, M., Göhrng, A. & Schäuble, P. (00). Eurospder at CLEF 00. In Peters, C. (Ed.), Workng Notes for the CLEF 00 Workshop, (pp. 7-3). Callan, J.P., Lu, Z. & Croft, W.B. (995). Searchng Dstrbuted Collectons Wth Inference Networks. In Fox, E.A., Ingwersen, P. & Fdel, R. (Eds.), Proceedngs of the 8 th Annual Internatonal ACM SIGIR Conference on Research and Developent n Inforaton Retreval, (pp. -8). Chen, A. (00). Cross-language Retreval Experents at CLEF-00. In Peters, C. (Ed.), Workng Notes for the CLEF 00 Workshop, (pp. 5-0). Ln, W.C. & Chen, H.H. (00a). NTU at NTCIR3 MLIR Task. In Kshda, K. & Ishda, E. (Eds.), Workng Notes of the Thrd NTCIR Workshop Meetng. Part II: Cross Lngual Inforaton Retreval Task, (pp. 0-05). Tokyo, Japan: Natonal Insttute of Inforatcs. Ln, W.C. & Chen, H.H. (00b). Mergng Mechanss n Multlngual Inforaton Retreval. In Peters, C. (Ed.), Workng Notes for the CLEF 00 Workshop, (pp. 97-0). Moulner, I. & Molna-salgado H. (00). Thoson Legal and Regulatory experents for CLEF 00. In Peters, C. (Ed.), Workng Notes for the CLEF 00 Workshop, (pp. 9-96). Oard, D. and Dekea, A. (998) Cross-Language Inforaton Retreval. Annual Revew of Inforaton Scence and Technology, vol. 33, (pp. 3-56). Peters, C. (Ed.) (00). Workng Notes for the CLEF 00 Workshop. Robertson, S.E., Walker, S. & Beauleu, M. (998). Okap at TREC-7: autoatc ad hoc, flterng, VLC and nteractve. In Voorhees, E.M. & Haran, D.K. (Eds.), Proceedngs of the Seventh Text REtreval Conference (TREC-7), (pp ). Gathersburg, MD: Natonal Insttute of Standards and Technology. Savoy, J. (00). Report on CLEF-00 Experents: Effectve Cobned Query-Translaton Approach. In Peters, C., Braschler, M., Gonzalo, J. & Kluck, M. (Eds.), Evaluaton of Cross-Language Inforaton Retreval Systes, (pp. 7-43). Lecture Notes n Coputer Scence, Vol Sprnger. Savoy, J. (00). Report on CLEF-00 Experents: Cobnng Multple Sources of Evdence. In Peters, C. (Ed.), Workng Notes for the CLEF 00 Workshop, (pp. 3-46).