School of Engineering and Technology, Department of Computer Science, Pondicherry University Pondicherry-India.

Transcription

1 IEEE-Internatonal Conference on Recent Trends n Informaton Technology, ICRTIT 2 MIT, Anna Unversty, Chenna. June 3-5, 2 Performance Evaluaton of Multlngual Informaton Retreval (MLIR) System over Informaton Retreval (IR) System Raju Korra #, Pothula Sujatha *2, Sdge Chetana *3, Madarapu Naresh Kumar #4 School of Engneerng and Technology, Department of Computer Scence, Pondcherry Unversty Pondcherry-Inda rajuu99@gmal.com 2 spothula@gmal.com 3 sdgechetana@gmal.com 4 madarapu.naresh@gmal.com Abstract Multlngual Informaton Retreval (MLIR) System deals wth the use of queres n one language and retreves the documents n varous languages. Here the Query translaton plays a central role n MLIR research. In ths paper, the language-ndependent ndexng technology s used to process the text collectons of Englsh, Telugu and Hnd languages. We have used multlngual dctonary based word-by-word query translaton. The expermental results are evaluated to analyze and compare the performance of Average Precson (AP IR ) and Mean Average Precson (MAP IR ) metrcs of IR system wth esteem to the Average Precson (AP MLIR ) and Mean Average Precson (MAP MLIR ) metrcs n MLIR system. Expermental result shows that the effectve retreval and performance of MLIR system has mproved by 3.4% over IR system. Index Terms Query translaton, MLIR, Recall, Precson, Average Precson, Mean Average Precson. I. INTRODUCTION Many nformaton worers collect nformaton from the global resources, whch mght be n dfferent languages. MLIR system s consdered as an mportant applcaton n IR. The am of MLIR system s to provde users a way to search documents wrtten n varous languages for a gven query. Several dfferences between monolngual IR and MLIR [3] arse f the user s well-versed wth more than one language. In order to reflect dfferng profcency levels of user language, the user nterface must provde dfferental dsplay capabltes. Translaton nto several languages s needed when more than one user receves the results. Dependng on the user's level of complexty, translaton of dfferent elements at dfferent stages can be provded to users for a range of nformaton access needs, ncludng eyword translaton, term translaton, ttle translaton, abstract translaton, specfc paragraph translaton, capton translaton, full document translaton, etc. Fnally, monolngual IR users can also tae advantage of the results of MLIR, where a query gven by user can retreve a set of valuable documents n other languages wthout any translatons. MLIR system uses a query n one language to retreve documents n varous languages. There exst language translaton ssues n MLIR. In addton to language translaton ssues, producng raned lst that nvolves documents n varous languages from several text collectons s crtcal process. In order to solve ths crtcal ssue there are two possble archtectures n MLIR. They are centralzed and dstrbuted. In the former archtecture, a huge collecton that contans documents n dfferent languages s used. In the later archtecture, documents are n varous languages are ndexed and retreved separately and then all the results are merged nto a multlngual raned lst [8]. For evaluatng the Centralzed MLIR system performance we are comparng some standard metrcs such as Recall, Precson, Average Precson and Mean Average Precson metrcs wth the IR system. MLIR system can help the users to query n ther natve language and retreve nformaton n varous languages. At a basc level, there are two approaches that can be taen when t comes to the desgn of a MLIR system. They are: translaton of query language [3] or translaton of the document language. Snce n MLIR the query language and document language dffer, a query representaton must be compared wth each document representaton n order to determne the degree of smlarty: In MLIR, ether the query must be translated nto the document language, or the document n the query language. Former way s a better way because translatng a query one s much more effcent than translatng each and every document n the collecton nto the query language. The advantage of query translaton [4] rather than document translaton s that a query translaton module may be added to an exstng IR system s easy, when compared wth the cost of modfyng the entre document base and redesgnng the system for multlngual retreval. The remander of ths paper s organzed as follows. Secton 2 dscusses related wor to ths study. Secton 3 Tradtonal measures for evaluatng the performance of IR system, secton 4 descrbes the proposed performance evaluaton method for MLIR system, and secton 5 presents the Expermental Results. Fnally secton 6 concludes the paper. II. RELATED WORK Three dfferent strateges [9] for query translaton were used namely, dctonary-based, thesaurus-based, and corpusbased methods [, 5, 4]. For dctonary-based approaches, //$26. 2 IEEE 722

2 IEEE-ICRTIT 2 Hull and Grefenstette [3] performed experments usng dctonary-based approach wthout ambguty analyss. Informaton Retreval models are weghtng and ndexng process: the three IR models are () Boolean model: In ths model the documents and query are represented as sets of ndex terms; t s set theoretc, () Probablstc model: The framewor for modellng documents and query representaton s based on probablty theory; ths s probablstc, () Vector space model: The query and documents are represents vectors n t-dmensonal space; t s algebrac. They translate nonnames; they used a probablstc word translaton derved from bdrectonal word algnments extracted from GIZA++ [Och and Ney ] [29] by the Machne Translaton members of Defence Advanced Research Projects Agency Global Autonomous Language Explotaton (DARPA GALE) team. Ths teams s provded wth a multlngual corpus, ncludng text and speech, consstng of Englsh, Arabc and Chnese documents. They mplemented a straghtforward probablstc structured query approach [Darwsh and Oard, 3], and translated all word usng ths model. Document Translaton (DT) does sgnfcantly better than Query Translaton (QT). Salton et.al [8] also suggested that the performance of MLIR systems could resemble those monolngual ones provded a correct multlngual thesaurus s establshed. For poor performance of QT havng two problems :() Prevalence of rare names n the queres, t was not covered by the translaton dctonary, () translaton of non-name phrases. A QT module nclude translteraton mght mprove performance of QT for names. For non-name arguments usng full SMT than the typcal approach of word by word translaton mght lead to better QT. Sa and Saa [2] are proposed IR metrcs for the tas of retrevng one hghly relevant documents and opposed to the fndng as many relevant documents as possble. Bucley and Voorhees [22], Voorhees [23] used the stablty and swap method consdered bnary IR metrcs. Sanderson and Zobel [24] and Saa [25] explored a few varatons of the swap method. Saa [26] proposed a method for comparng the senstvty of IR metrcs based on Bootstrap Hypothess Tests. Saa [27] nvestgated the ran correlatons among Ave P, R- Prec, Q-measure and R-measure usng NTCIR data. Vu and Gallnnar [28] generalzed AveP for handlng graded relevance. To promote research and development n the area of IR n Inda, the followng Evaluaton campagns are conducted. ILIR-29 Inda Language IR IRSI-24 IR Socety of Inda FIRE-2 Forum for IR Evaluaton MIRACLE-Maryland Interactve Retreval Advanced Cross-Language Engne NIST- Natonal Insttute of Standards and Technology Evaluaton campagns n other countres ncludes Cranfeld-958 UK TREC-992 Text Retreval Conference USA NTCIR NII-999 Test Collecton for IR Systems Japan CLEF-2Cross-Language Evaluaton Forum Europe The new applcaton of MLIR draws on achevements and technques n several related areas [3] ) Informaton Access: document ndexng (multlngual); retrevng, flterng, clusterng; presentaton and summarzaton of nformaton; multlngual metadata; crosslanguage nformaton retreval. 2) Machne Translaton: comparable and parallel text algnment; language generaton. 3) Computatonal Lngustcs: morphologcal analyss, syntactc parsng, technques for dsambguaton, document segmentaton, corpus analyss, creaton of dervatve lexcons, term recognton and term expanson. 4) Resources: dctonares, thesaur, ndex terms, test collectons, speech data bases. III. TRADITIONAL MEASURES FOR EVALUATING THE PERFORMANCE OF IR SYSTEM To measure s to now, f we cannot measure t, we cannot mprove t, for that purpose we mprove the characterstcs of IR/MLIR systems usng the metrcs. In relevance judgment [4], human evaluators wll read the content of each returned document and judge f t satsfes the nformaton need of that partcular topc. Gradng on returned document s gven as follows. L2 (Relevant): contans nformaton that satsfes the nformaton need. L (Partally relevant): contans nformaton that partally satsfes the nformaton need. L (Not relevant): does not contan nformaton relevant to the nformaton need. Based on relevance judgment the evaluatons nclude: A. Set based evaluaton B. Ran based evaluaton wth explct absolute judgments C. Ran based evaluaton wth explct preference judgments D. Ran based evaluaton wth mplct judgments A. Set based evaluaton ) Precson: Precson [5] [6] s the fracton of the documents retreved that are relevant to the user's nformaton need. () Where t p =relevant tems retreved, t p + f p = relevant retreved tems and t p =true postve, f p =false postve 2) Recall: Recall [5] s the fracton of the documents that are relevant to the query that are successfully retreved. R (2) 723

3 Performance Evaluaton of Multlngual Informaton Retreval (MLIR) System over Informaton Retreval (IR) System Where t p =relevant retreved tems, t p +f n =total number of relevant retreved tems n the database, t p =true postve, f n =false negatve B. Ran based evaluaton wth explct absolute judgments ) Average precson (AP): AP [6] emphaszes ranng relevant documents hgher. It s the average of precsons computed at the pont of each of the relevant documents n the raned sequence. Dscounted gan at Ran r, precson at Ran r = P(r), AP = r Pr j (3) j D j + Where D + denotes the number of relevant documents wth respect to the query. 2) (Voorhees et al. 25) [6] s a measure for evaluatng top postons of a raned lst usng two levels (relevant and rrelevant) of relevance judgment: = r (4) j j = Where = the truncaton poston, r j = f the document n the j th poston s relevant and zero otherwse. C. Ran based evaluaton wth explct preference judgments ) Kendall s ran correlaton: Kendall s ran correlaton [6] s a monotonc functon of the probablty that a randomly chosen par of raned systems s ordered dentcally n the two ranngs. Hence a swap near the top of a raned lst and that near the bottom of the same lst has equal mpact. However, for the purpose of ranng retreval systems, for example, n a competton-style worshop such as NTCIR and TREC [9], the rans near the top of the lst are arguably more mportant than those near the bottom. C D endall = (5) L ( L ) / 2 C = number of system pars that are raned n the same order n both ranngs, and let D= number of system pars that are raned n opposte order n the two ranngs, the sze of the raned lsts be L. D. Ran based evaluaton wth mplct judgments ) Mean Average Precson (MAP): Most standard among the TREC communty s MAP [5] [6], whch provdes a sngle-fgure measure of qualty across recall levels. Among valuaton measures, MAP has been shown to have especally good dscrmnaton and stablty. For a sngle nformaton need, AP s the average of the precson value obtaned for the set of top documents exstng after each relevant document s retreved, and ths value s then averaged over nformaton needs. MAPQ / Q Q PrecsonRj (6) Where Q=set of relevant documents, R=recall IV. PERFORMANCE EVALUATION FOR PROPOSED MLIR METRICS For the MLIR system evaluaton only basc metrcs le Precson, Recall etc. [2] are avalable but there s no standard specfc metrcs for measurng the performance of ths system. A. Dfferent Degrees of MLIR Localzaton of user nterfaces, monolngual n dfferent languages Query n L, all documents n L 2 Query n L, documents n L +L 2...+L n Query n L, documents multlngual The MLIR, both the source and translated queres are used to search a mult-lngual collecton for relevant documents. Then, only retreved documents that are not n the user language are translated nto the user language. In prncple, we need only the engne to realze MLIR n the sense that users can retreve/ browse foregn documents through ther natve language. However, to mprove the qualty of our system, two alternatve post-processng modules can optonally be used. There are two types of techncal ssues must be addressed when dealng wth multlngual data [] () techncal ssues nvolvng data exchange, wth a set of attendant sub-ssues. () natural language questons, also wth a set of attendant research ssues. B. Multple translatons The query translaton process s accomplshed as follow: Let M be the number of query terms, then we defne user s query as: Q = {q } (= M) (7) Then we loo each q up n the dctonary [8] and after fndng translatons of q we splt the translatons nto ts consttuent toens. Weghted Structured Query Translaton[]: taes advantage of multple translatons and translaton probabltes TF and DF of query term e are computed usng TF and DF of ts translatons TF ( e, D ) = p( f e) TF( f, D ) f DF ( e ) = p ( f e ) DF ( f ) f (9) Where p (f e) s the estmated probablty that e would be properly translated to f. ) Precson (P MLIR ): precson s the rate between the relevant documents retreved by the MLIR system n response to a query and the total number of documents retreved. In bnary classfcaton precson s analogous to the postve predctve value. P MLIR () Where r d {, } the relevance of documents d for the user, f d {, } the retreval of documents d n the processng of the current query. (8) 724

4 IEEE-ICRTIT 2 2) Recall (R MLIR ): Recall s the rate between the number of relevant documents retreved and the total number of documents to the query exstng n the database. In bnary classfcaton recall s called as senstvty. R MLIR () Where r d {, } the relevance of documents d for the user, f d {, } the retreval of documents d n the processng of the current query. 3) Average Precson (AP MLIR ): Average of the precson values at the ponts at whch each relevant documents retreved n some user specfc languages. N P MLIR.B D AP MLIR (2) Where r=ran of the query n n languages, N=the number of retreved documents n n languages, B(r) =bnary functon on the relevance of a gven ran r n n languages, D =numbe of relevant documents, P MLIR (r) =Precson at a cut of ran r n n languages. P MLIR r (3) 4) Mean Average Precson (MAP MLIR ): Average of the average precson values for a number of queres n MLIR system. NQ AP MLIR NQ MAP MLIR (4) NQ Where NQ=number of queres The overall desgn of our system, whch conssts of an engne, and two post-processng modules, that s, re-ranng and clusterng modules. The engne retreves documents n response to user queres, and outputs those documents n the source (user) language. The MLIR, both the source and translated queres are used to search a mult-lngual collecton for relevant documents. Then, only retreved documents that are not n the user language are translated nto the user language. In prncple, we need only the engne to realze MLIR n the sense that users can retreve/ browse foregn documents through ther natve language. However, to mprove the qualty of our system, two alternatve postprocessng modules can optonally be used. The two-stage method, the re-ranng module re-rans documents retreved by the engne to mprove the retreval accuracy. In ths case, the engne and re- ranng module correspond to the frst and second stages, respectvely. The post-translaton query expanson component expands the query n the target language n a way smlar to pre-translaton expanson. Fnally document retreval [9] component taes the query n the target language and retrevng the relevant documents from the text collecton. V. EXPERIMENTAL RESULTS In the proposed MLIR metrcs, we used dctonary based query translaton usng word to word translaton. Altogether we have used documents whch nclude Englsh, Telugu and Hnd documents. Fnally MLIR metrcs Precson, Recall, Average Precson and Mean Average Precson are evaluated usng Natural language processng toolt named General Archtecture for Text Engneerng (GATE 6.-beta) that uses Java.6._7 on wndows XP. We used the Google Translate.7 API ( to translate the resources provded n other languages. In evaluaton of MLIR systems, we compared the attrbutes of exstng system that s IR system and we get mproved results of MLIR system n terms of effectve retreval. Table ncludes the calculated precson and Recall values obtaned from 2 Queres on IR and MLIR systems whch are graphcally shown n fg. and fg.2 respectvely. Table 2 contans calculated AP IR and AP MLIR values obtaned from queres appled on IR and MLIR systems. MAP IR and MAP MLIR values are calculated from AP IR and AP MLIR, as tabulated n table 3. Fg 3 gves the comparatve performance evaluaton of AP n IR System over AP n MLIR systems and fg.4 depcts the performance evaluaton of MAP n IR System to that of MLIR system. Expermental results show that the effectve retreval of MLIR system has mproved by 3.4% over IR system. TABLE. CALCULATED PRECISION AND RECALL IN MLIR SYSTEM Query Precson IR Precson MLIR Recall IR Recall MLIR

5 Performance Evaluaton of Multlngual Informaton Retreval (MLIR) System over Informaton Retreval (IR) System.8.4 PrecsonIR PrecsonMLIR Queres Fg. Calculated Precson values n IR and MLIR System.8 RecallIR RecallMLIR Queres TABLE 2.CALCULATED AP IN IR AND MLIR SYSTEMS Fg 2. Calculated Recall values n IR and MLIR System Query AP IR AP MLIR Number of Queres Fg4.comparson of MAP n IR and MLIR systems MAPIR MAPMLIR TABLE 3.CALCULATED MAP IN IR AND MLIR SYSTEMS No. of Queres MAP I IR MAP MLIR APIR APMLIR Query Number Fg 3. Comparson of AP n IR and MLIR systems VI. CONCLUTION Here multlngual dctonary based word-by-word query translaton s used and the text collectons of Englsh, Telugu and Hnd languages are beng processed n IR and MLIR systems. The expermental evaluaton of Precson, Recall, 726

6 IEEE-ICRTIT 2 Average Precson and Mean Average Precson n IR and MLIR systems have been done and the performance s beng compared. Expermental results show that the effectve retreval and performance of MLIR system has mproved by 3.4% over IR system. REFERENCES [] Daqng He, Dan Wu. "Enhancng Query Translaton wth Relevance Feedbac n Translngual Informaton Retreval." Informaton Processng and Management Vol 47, Issue, Pages -7, 2. [2] Dan Wu, Daqng He. A Study of Query Translaton usng Google Machne Translaton System. In Proceedngs of the 2 Internatonal Conference on Computatonal ntellgence and Software Engneerng (CSE 2), Dec. 2, Wuhan, Chna. [3] Dan Wu, Daqng He, Huln Wang. Cross-Language Query Expanson Usng Pseudo Relevance Feedbac. Journal of the Chnese Socety for Scentfc and Techncal Informaton (2): [4] Qang, Pu, Daqng He, Q L. "Query Expanson for Effectve Geographc Informaton Retreval." Evaluatng Systems for Multlngual and Multmodal Informaton Access, 9th Worshop of the Cross-Language Evaluaton Forum, CLEF 28, Aarhus, Denmar, Revsed Selected Papers. Sprnger. 29. [5] Al Dasdan, Kostas Tsoutsoulls, Emre Velpasaoglu Web Search Engne Metrcs for Measurng User Satsfacton {dasdan, Kostas, emrev}@yahoo-nc.com, Yahoo! Inc. 2 Apr 29 8th Internatonal World Wde Web Conference Aprl 2-24, 29. [6] Tetsuya Saa, Stephen Robertson, Modelng a User Populaton for Desgnng Informaton Retreval Metrcs, the Second Internatonal Worshop on Evaluatng Informaton Access (EVIA), December 6, 28, Toyo, Japan. [7] Daqng He, Dan Wu. Translaton Enhancement: A New Relevance Feedbac Method for Cross-Language Informaton Retreval. In the Proceedngs of ACM 7th Conference on Informaton and Knowledge Management (CIKM 28) pages [8] Chen-Hsn Cheng, Reuy-Jye Shue, Hung-Ln Lee, Shu-Yu Hseh, Guann-Cyun Yeh, & Guo-We Ban: AINLP at NTCIR-6: evaluatons for multlngual and cross-lngual nformaton retreval Proceedngs of NTCIR-6 Worshop Meetng, May 5-8, 27, Toyo, Japan. [9] Hsn-Chang Yang and Chung-Hong Lee (28) "Multlngual Informaton Retreval usng GHSOM." In Proceedngs of The Eghth Internatonal Conference on Intellgent Systems Desgn and Applcatons (ISDA-28), Vol., Kaohsung, Tawan, Nov , 28, pp [] D. W. Oard and B. J. Dorr. A survey of multlngual text retreval. Techncal Report UMIACS-TR-96-9, Unversty of Maryland, Insttute for Advanced Computer Studes, College Par, MD, 996. [] Jan-Yun Ne, Fuman Jn: A Multlngual Approach to Multlngual Informaton Retreval. CLEF 22: -. [2] L. Ballesteros and W. B. Croft. Dctonary-based methods for cross-lngual nformaton retreval. In Proceedngs of the 7th Internatonal DEXA Conference on Database and Expert Systems Applcatons, pages 79 8, 996. [3] D. A. Hull and G. Grefenstette. Queryng across languages: a dctonary-based approach to multlngual nformaton retreval. In Proceedngs of the 9th Internatonal Conference on Research and Development n Informaton Retreval, pages 49 57, 996. [4] H. H. Chen, C. C. Ln, and W. C. Ln. Constructon of a chneseenglsh wordnet and ts applcaton to clr. In Proceedngs of 5th Internatonal Worshop on Informaton Retreval wth Asan Languages, pages 89 96, 2. [5] L. Ballesteros and W. B. Croft. Dctonary-based methods for crosslngual nformaton retreval. In Proceedngs of the 7 th Internatonal DEXA Conference on Database and Expert Systems Applcatons, pages 79 8, 996. [6] Olver Chapelle, Quoc V. Le, Alex Smola, and Choon Hu Teo, Optmzaton of Ranng Measures, Journal of Machne Learnng Research (2) -48. [7] Chrstopher D. Mannng, Prabhaar Raghavan & Hnrch Schütze, An Introducton to Informaton Retreval 29 Cambrdge Unversty Press Prnted on Aprl, 29 Webste: [8] G. Salton. Automatc processng of foregn language documents. Journal of the Amercan Socety for Informaton Scence, 2(3):87 94, 97. [9] Alon Lave, Kenj Sagae, Shyamsundar Jayaraman the Sgnfcance of Recall n Automatc Metrcs for MT Evaluaton , In Proceedngs of the 6th Conference of the Assocaton for Machne Translaton n the Amercas (AMTA-24). [2] Mar Davs and Ted Dunnng. A trec evaluaton of query translaton methods for mult-lngual text retreval. In Proceedngs of the Fourth Retreval Conference (TREC-4) Gathersburg, MD: Natonal Insttute of Standards and Technology, Specal Publcaton 5-236, 995. [2] Saa, 26b Saa, T. (26b). Gve me just one hghly relevant document: P-measure. In Proceedngs of the 29th annual nternatonal ACM SIGIR conference on research and development n nformaton retreval (SIGIR 26). [22] Bucley and Voorhees, 24 Bucley, C., & Voorhees, E. M. (24). Retreval evaluaton wth ncomplete nformaton. In Proceedngs of the 27th annual nternatonal ACM SIGIR conference on research and development n nformaton retreval (SIGIR 24) (pp ). [23] Voorhees, 25 Voorhees, E. M. (25). Overvew of the TREC 24 robust retreval trac. In Proceedngs of the 3th text retreval conference (TREC 24). [24] Sanderson and Zobel, 25 Sanderson, M., & Zobel, J. (25). Informaton retreval system evaluaton: effort, senstvty, and relablty. In Proceedngs of the 28th annual nternatonal ACM SIGIR conference on research and development n nformaton retreval (SIGIR 25) (pp ). [25] Saa, 25a Saa, T. (25a). The effect of topc samplng on senstvty comparsons of nformaton retreval metrcs. In Proceedngs of the 5th NTCIR worshop on research n nformaton access technologes (NTCIR-5). [26] Saa, 26a Saa, T. (26a). Evaluatng evaluaton metrcs based on the bootstrap. In Proceedngs of the 29th annual nternatonal ACM SIGIR conference on research and development n nformaton retreval (SIGIR 26). [27] Saa, 24 Saa, T. (24). Ranng the NTCIR systems based on multgrade relevance. In Proceedngs of Asa nformaton retreval symposum 24 (pp. 7 77). [28] Vu and Gallnar, 25 Vu, H.-T., & Gallnar, P. (25). On effectveness measures and relevance functons n ranng INEX systems. In Proceedngs of Asa nformaton retreval symposum 25. Lecture notes n computer scence: Vol (pp ). [29] Krsten Parton, Kathleen McKeown, James Allan, Enrque Henestroza: Smultaneous multlngual search for translngual nformaton retreval. CIKM 28: [3] Contrbutors: Chrstan Fluhr Robert E. Frederng Doug Oard Atosh Oumura, Ka Ishawa, and Kenj Chapter 2 Multlngual (or Cross-lngual) Informaton Retreval,Edtors: Judth Klavans and Eduard Hovy Satoh 727