Multilingual Information Retrieval

Transcription

1 Proposal for Tutorial on Multilingual Information Retrieval Proposed by Arjun Atreya V Shehzaad Dhuliawala ShivaKarthik S Swapnil Chaudhari Under the direction of Prof. Pushpak Bhattacharyya Department of Computer Science & Engineering IIT Bombay

2 Abstract The research in information retrieval is driven by the aim to serve the user's information need. With the continuing boom of information online, one cannot always expect all the information to be present in a single language. With the internet reaching out to several countries/regions, it would be naïve to assume that the need of a user may be limited to English. Multilingual information retrieval aims to allow a search engine to cater to the user's need in multiple languages. The content plan of this tutorial, as listed below, aims to cover the various facets of multilingual information retrieval. The tutorial covers the basics and advances in the field of MLIR. The basic components of an IR system: 1) Crawling, 2) Indexing, 3) Searching and 4) Language processing are discussed along with the additions needed to project them for a multilingual setting. Along with this, we also compare and contrast these components in a monolingual and multilingual system. India, as a country, is the home to a bevy of languages ranging from different families. One major region where the need of a multilingual information retrieval system is felt is in the domain of tourism. The Cross Lingual Information Access (CLIA) project envisioned a search engine for Indian languages to promote tourism. This led to the development of Sandhan: An Indian cross-lingual search engine 1. We also aim to share our experiences garnered during the creation of Sandhan. Sandhan, as a project was conceived after myriad hours of research and development by a consortium of institutes around the country. We end our session by discussing several open research problems in the area. Proposed Duration: Half Day 1

3 Outline Plan 1. Introduction: 10 slides; 15 minutes a. IR components [1] b. MLIR challenges c. The paradigms of Multilingual Information Retrieval [2][3] 2. Crawling: 20 slides; 30 minutes a. Dynamic crawling b. Font transcoding c. Adaptive fetching d. Focused crawling [4] i. Topic modeling ii. Domain identification [5] 3. Indexing: 20 slides; 30 minutes a. Distributed indexing [6] b. Multilingual indexing [7] c. Document to sub document mapping d. Optimization for a multilingual setting 4. Searching: 20 slides; 30 minutes a. Multilingual ranking b. Normalization c. Query processing i. Query expansion [8][9] ii. Query translation [10] 5. Language Processing: 15 slides; 20 minutes a. Morphological analysis [11] b. Language identification [12] c. Multiword and named entity recognition [13] d. Word sense disambiguation [14][15] 6. Evaluation: 10 slides; 15 minutes a. Metrics of evaluation b. Evaluation forums (CLEF 2, TREC 3, FIRE 4, NTCIR 5 ) 7. Sandhan: A Case Study: 15 slides; 20 minutes 8. Conclusion and Open research problems in the area: 10 minutes 9. Question and Answer session: 10 minutes Expected Duration: 180 minutes

4 Presenters: Arjun Atreya V PhD Scholar, Dept of Computer science & Engineering, IIT Bombay Arjun Atreya is pursuing his PhD in the area of multilingual information retrieval at IIT Bombay. His current interests involve cross lingual information retrieval, query expansion and transliteration across Indian languages. He has also co-authored papers in top NLP conferences such as COLING and IJCNLP. Shehzaad Dhuliawala Research Engineer, Dept of Computer science & Engineering, IIT Bombay. Shehzaad is working as a research engineer at CFILT lab IIT-Bombay. He completed his Bachelors in Technology at VNIT Nagpur. He has been working in the domain of cross lingual search for the past 4 months. He is one of the developers of the Sandhan. ShivaKarthik S Technical Officer, Applied AI Group, C-DAC Pune ShivaKarthik S. is working as a Technical officer at the Applied AI Group, CDAC Pune. He has been involved in various search engine projects for the past 7 years. His contributions are in the field of distributed crawling of the web, adaptive fetching, incremental crawling, topic categorization. His interests are in sentiment analysis, cross lingual information retrieval and topic categorization. Swapnil Chaudhari Software Swapnil is currently working as software engineer He has done his Masters from IIT Bombay in cross lingual information retrieval. His current interests include query classification, transliteration and personalization. Arjun and Shehzaad are guided by: Prof. Pushpak Bhattacharyya, Director, IIT Patna Professor, Dept. of Computer science & Engineering, IIT Bombay.

5 References: [1] Manning, Christopher D., Prabhakar Raghavan, and Hinrich Schütze.Introduction to information retrieval. Vol. 1. Cambridge: Cambridge university press, [2] Hull, David A., and Gregory Grefenstette. "Querying across languages: a dictionary-based approach to multilingual information retrieval." Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, [3] Chen, Aitao, and Fredric C. Gey. "Multilingual information retrieval using machine translation, relevance feedback and decompounding." Information Retrieval (2004): [4] Priyatam, Pattisapu Nikhil. Developing Focused Crawlers for Genre Specific Search Engines. Diss. International Institute of Information Technology Hyderabad, [5] Priyatam, Pattisapu Nikhil, et al. "Don t Use a Lot When Little Will Do: Genre Identification Using URLs." Research in Computing Science 70 (2013): [6] Danzig, Peter B., et al. "Distributed indexing: A scalable mechanism for distributed information retrieval." Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, [7] [8] Chinnakotla, Manoj K., Karthik Raman, and Pushpak Bhattacharyya. "Multilingual PRF: english lends a helping hand." Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. ACM, [9] Arjun Atreya V, Yogesh Kakde, Pushpak Bhattacharyya and Ganesh Ramakrishnan, Structure Cognizant Pseudo Relevance Feedback, In Proceedings of IJCNLP 2013, Nagoya, Japan, October, [10] Ballesteros, Lisa, and W. Bruce Croft. "Phrasal translation and query expansion techniques for cross-language information retrieval." ACM SIGIR Forum. Vol. 31. No. SI. ACM, [11] Bapat, Mugdha, Harshada Gune, and Pushpak Bhattacharyya. "A paradigm-based finite state morphological analyzer for Marathi." Proceedings of the 1st Workshop on South and Southeast Asian Natural Language Processing (WSSANLP) [12] Joshi, Gopal Datt, Saurabh Garg, and Jayanthi Sivaswamy. "Script identification from Indian documents." Document Analysis Systems VII. Springer Berlin Heidelberg, [13] de Pablo-Sánchez, César, José Luis, and Paloma Martınez. "Named entity processing for cross-lingual and multilingual IR applications." New Directions in Multilingual Information Access (2006): 15. [14] Stokoe, Christopher, Michael P. Oakes, and John Tait. "Word sense disambiguation in information retrieval revisited." Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, [15] Ng, Hwee Tou. "Does word sense disambiguation improve information retrieval?" Proceedings of the fourth workshop on exploiting semantic annotations in information retrieval. ACM, 2011.