Searching a Russian Document Collection Using English, Chinese and Japanese Queries

Transcription

1 Searchig a Russia Documet Collectio Usig Eglish, Chiese ad Japaese Queries Fredric C. Gey (gey@ucdata.berkeley.edu) UC Data Archive & Techical Assistace Uiversity of Califoria, Berkeley, CA USA ABSTRACT. As i CLEF 2003, Berkeley experimeted with the CLEF Russia Izvestia documet collectio with mooligual ad biligual rus for the Russia collectio. For CLEF 2004 we also experimeted with Chiese ad Japaese as topic laguages, usig Eglish as the pivot laguage. For biligual retrieval our approaches were query traslatio (for Eglish as a topic laguage) ad fast documet traslatio from Russia to Eglish (for Chiese ad Japaese traslated to Eglish as the topic laguage). Chiese ad Japaese topic retrieval sigificatly uder-performed Eglish Russia retrieval because of the double traslatio loss of effectiveess. 1 Itroductio CLEF 2003 was the first time a Russia laguage documet collectio was available i CLEF. We had worked for several years with Russia topics i both the GIRT task ad the CLEF mai tasks, so extesio of our techiques to Russia was straightforward No uusual methodoy was applied to the Russia collectio, however ecodig remaied a issue ad we eded up usig the KOI-8 ecodig scheme for both Russia documets ad topics. 2 Documet rakig Berkeley has used a mooligual documet rakig algorithm which uses statistical clues foud i documets ad queries to predict a dichotomous variable (relevace) based upo istic regressio fittig of prior relevace judgmets. The exact formula is: O ( R = P ( R P ( R D, Q ) = D, Q ) D, Q ) = x 3 + P ( R D, Q ) 1 P ( R D, Q ) x x 4 x 2 where O ( R D, Q ), P( R D, Q ) mea, respectively, odds ad probability of relevace of a documet with respect to a query, ad 1 qtfi x1 = + 1 ql 35 i= 1 +

2 x 2 = i i= 1 dl + dtf 80 1 ctfi x3 = + 1 cl x = 4 i= 1 where is the umber of matchig terms betwee a documet ad a query, ad ql : query legth dl: documet legth cl: collectio legth qtf_i: the withi-query frequecy of the ith matchig term dtf_i: the withi-documet frequecy of the ith matchig term ctf_i: the occurrece frequecy of the ith matchig term i the collectio. This formula has bee used sice the secod TREC coferece ad for all NTCIR ad CLEF cross-laguage evaluatios [1]. 3 Russia Retrieval for the CLEF mai task CLEF 2003 marked the first time a documet collectio was available ad evaluated i the Russia laguage. The CLEF Russia collectio cosists of 16,716 articles from Izvestia ewspaper for This is a small umber of documets by most CLEF measures (the smallest other collectio of CLEF 2003, Fiish, has 55,344 documets; the Spaish collectio has 454,045 documets). We used the Russia ad Eglish idexes geerated for CLEF 2003 for all our CLEF 2004 Russia rus. The collectio is also rich i metadata, icludig specificatio of geography for ews articles; this ca be exploited for mappig ad geotemporal queryig of documets relatig to place ad time [2]. 3.1 Ecodig Issues The Russia documet collectio was supplied i the UTF-8 uicode ecodig, as were the Russia versio of the topics. However, sice the stemmer we employ is i KOI8 format, the etire collectio was coverted ito KOI8 ecodig, as with CLEF 2003 [3]. I idexig the collectio, we coverted upper-case letters to lowercase ad applied Sowball s Russia stemmer ( together with Russia stopword list created by mergig the Sowball list with a traslatio of the Eglish stopword list. I additio the PROMPT traslatio system would also oly work o KOI8 ecodig which meat that our traslatios from Eglish also would come i that ecodig. 3.2 Russia Mooligual Retrieval We submitted two Russia mooligual rus, the results of which are summarized below. As i CLEF 2003, both rus utilized blid feedback, choosig the top 30 terms from the top raked 20 documets of a iitial retrieval ru For BKRUMLRR1 ad BKRUMLRR2 rus we used TITLE ad DESCRIPTION documet fields for idexig. The results of our retrieval are summarized i Table 1. Results were reported by the CLEF orgaizers for 34 topics which had oe or more relevat documets. Ru Name BKRUMLRR1 BKRUMLRR2 Idex Koi Koi Topic fields TD TDN Retrieved

3 Relevat Rel Ret Precisio at at at at at at at at at at at Avg. Precisio Table 1: Berkeley Mooligual Russia rus for CLEF 2004 Addig the Narrative sectio to the query did ot sigificatly improve results because the Narrative sectio did ot cotribute additioal cotet terms beyod those foud i the Title ad Descriptio fields of the topics. 3.3 Biligual Retrieval from Eglish to Russia We submitted eight biligual rus agaist the Russia documet collectio, four with Eglish as topic laguage ad two each with Chiese ad Japaese as topic laguages. These rus used a idex i which oly the TITLE ad TEXT fields of each Russia documet was idexed, so are directly comparable to the mooligual rus BKMLRURR1 ad BKMLRURR2 above. The four Eglish Russia rus utilized query traslatio from Eglish topics ito Russia. We compared two web-available traslatio systems, SYSTRAN at for the first two rus (BKRUBLER1, BKRUBLER2) ad the PROMT system (rus BKRUBLER3, BKRUBLER4) developed i Russia ad foud at Ru Name BKRUBLER1 BKRUBLER2 BKRUBLER3 BKRUBLER4 Traslatio Babelfish Babelfish PROMT PROMT Topic fields TD TDN TD TDN Retrieved Relevat Rel Ret Precisio at at at at at at at at at at at Avg. Prec Table 2. Biligual Eglish Russia rus.

4 The results demostrate clearly the superiority of the PROMT system for this topic set. 3.4 Biligual Retrieval from Chiese ad Japaese to Russia Because Chiese ad Japaese were available as topic laguages, we experimeted with these laguages by traslatig the topics to Eglish (i.e. used Eglish as a pivot laguage). Our approach to traslatio from Chiese or Japaese topics to Eglish was to utilize a widely available software package, the SYSTRAN CJK Persoal system available for less tha $US100. from However, istead of query traslatio a secod time, we utilized a techique (also used for Russia i CLEF 2003) developed by Aitao Che, called Fast Documet Traslatio [4]. Istead of doig complete documet traslatio usig MT software, the MT system is used to traslate the etire vocabulary of the documet collectio o a word-by-word basis without the cotextualizatio of positio i setece with respect to other words. Mooligual retrieval was performed by matchig the Eglish versios of the Chiese or Japaese topics agaist the traslated Eglish documet collectio. More details ca be foud i our CLEF-2003 fial paper [3]. The results, displayed below i Table 3, show that there is cosiderable loss of performace whe usig Eglish as a pivot laguage for these Asia laguage (we have re-displayed the best Eglish Russia rus for compariso). It may be that this performace was hampered by the reduced utility of the Eglish documets traslated from Russia, as was the case for our CLEF 2003 biligual performace which used this method. We did ot try mergig of rus from the two methods to see if it would improvemet performace. Ru Name BKRUBLER3 BKRUBLER4 BKRUMLZE1 BKRUMLZE2BKRUMLJR1 BKRUMLJR2 Laguage Eglish Eglish Chiese Chiese Japaese Japaese Traslatio PROMT PROMT Systra CJK Systra CJK Systra CJK Systra CJK Topic fields TD TDN TD TDN TD TDN Retrieved Relevat Rel Ret Precisio at at at at at at at at at at at Avg. Prec Table 3. Biligual Chiese/Japaese Russia rus 3.5. Brief Aalysis of Retrieval Performace Our mooligual Russia performace was acceptable but certaily ot outstadig. For may topics, Title- Descriptio rus out-performed Title-Descriptio-Narrative rus, because the Narrative sectio added o ew iformatio ad might sometimes add oise terms. For all our rus our biligual retrieval results were worse tha mooligual (Russia-Russia) retrieval i terms of overall precisio. However the traslatio of Eglish to Russia by the PROMT system achieved 82% of mooligual for the TD rus. Oe puzzlig ad iterestig topic was umber 202 ( Nick Leeso's Arrest )

5 where our biligual retrieval out-performed our mooligual rus it seems that the PROMT traslatio ad trasliteratio Арест Ника Лизона came up with a better spellig of the last ame tha the Russia topic creator who used Арест Ника Леесон, which did ot seem to match ay relevat documets. Accordig to the summary results for Russia mooligual, at least oe ru achieved 1.00 precisio for this topic; it would be most iterestig to see how they modified the topic to match to the three relevat documets. A cautioary ote must be made about the CLEF-2004 Russia topic set. The total umber of relevat documets was oly 123 for the etire topic set, with a mea of 3.6 relevat documets per topic. Because of the ature of the retrieval results by query from the Russia collectio (22 of the 34 topics have 2 or fewer relevat documets) oe has to be careful about drawig coclusios from ay submitted results. 4 Summary ad Ackowledgmets For CLEF 2004, we experimeted with the CLEF Russia documet collectio with both mooligual Russia ad biligual to Russia from Eglish, Chiese ad Japaese topics I additio to query traslatio methodoy for biligual retrieval, we tried a fast documet traslatio method of the Russia collectio to Eglish ad performed Eglish-Eglish mooligual retrieval with the traslated topics from Chiese ad Eglish to Japaese. Chiese Russia ad Japaese Russia biligual performace results were sigificatly worse tha query traslatio from Eglish to Russia. We would like to thak Aitao Che for supplyig writig the istic regressio rakig software ad for performig the fast documet traslatio from Russia to Eglish. 5 Refereces [1] A. Che, W. Cooper ad F. Gey. Full text retrieval based o probabilistic equatios with coefficiets fitted by istic regressio. I: D.K. Harma (Ed.), The Secod Text Retrieval Coferece (TREC-2), pages 57-66, March 1994 [2] F. Gey ad K. Carl, Geotemporal Queryig of Multiligual Documets, Proceedigs of the Workshop o Geographic Iformatio Retrieval, available at: [3] V. Petras, N. Perelma ad F. Gey. UC Berkeley at CLEF-2003 Russia Laguage Experimets ad Domai-Specific Retrieval. To appear i: Proceedigs of the CLEF 2003 Workshop, Spriger Computer Sciece Series. [4] A. Che ad F. Gey. Multiligual Iformatio Retrieval Usig Machie Traslatio, Relevace Feedback, ad Decompoudig, Iformatio Retrieval Joural: Special Issue o CLEF, V7 No 1-2, pp , Ja-Apr 2004