Performance Evaluation of Search Engines for Hindi Language Information Retrieval

Transcription

1 Kumar Sourabh 1 and Vibhakar Mansotra 2 2 Associate Professor, 1,2 Department of Computer Science and IT, University of Jammu, Jammu J&K India 1 kumar9211.sourabh@gmail.com and 2 vibhakar20@yahoo.co.in variations in Hindi are more compared with any other language such as English. [4] ABSTRACT With the internet growing at an exponential rate the web is increasingly hosting web pages in different languages. It is essential for the search engines to be able to search information stored in a specific language. The native users also tend to look for any information on web nowadays. This leads to the need of effective search engines to fulfill native user s needs and provide them information in their native languages. The major population of India use Hindi as a first language. The Indian constitution identifies 22 languages, of which six languages (Hindi, Telugu, Tamil, Bengali, Marathi and Gujarati) are spoken by at least 50 million people within the boundaries of the country there are a large number of them living outside the country. The Hindi language web information retrieval is not in a satisfactory condition. The presence of Hindi on the World Wide Web is still limited and tentative because of attitudinal and technical factors. Besides the other technical setbacks the Hindi language search engines face the problem of morphology, phonetics, word sense disambiguation etc. The performance of search engines is affected by these problems. This paper covers the comprehensive analysis and also the comparison of the affect of language structure related factors (phonetics, WSD, synonyms, influence of foreign words) on the performance of search engines supporting Hindi language. KEYWORDS search engines, morphology, word sense disambiguation, precision, Guruji, Raftaar and Hindi Language.. 1. INTRODUCTION With the web content being written in different languages of the world, it has become important to have tools that can retrieve information from the documents written in different languages. In the context of Indian languages, Hindi language has been given much emphasis leading to the development of significant number of Hindi documents. In fact, of the top 100 languages in the world, English occupies the top position, with Hindi coming fifth. [1].Hindi language information retrieval on the web is still in its nascent stage. The under development of web in Indian regional languages is one of the important reasons behind the limited growth of Internet in India. [2]. It is the language of dozens of major newspapers, magazines, radio and television stations and of other media. This generates the need of the development of the powerful tools for Hindi language information retrieval. [3].Hindi is relatively free word-order and highly inflectional language. Spelling 1. HINDI LANGUAGE AND WEB SEARCHING However, the spread of Internet in India is today constrained by the fact that it is based essentially on the use of the English language. Those who have been benefited by Internet are only in urban areas. Even there only those have been benefited who are able to communicate in English. It is a disappointing situation. When we compare ourselves with China, we find ourselves in a miserable situation. Microsoft's Bhasha India is doing lot of job in Indian regional languages. Delhi based Indicus is also doing small but very promising effort in developing Hindi on the web. The company has started a Hindi search engine and is going to develop it in other regional languages. Many big companies like Google, Yahoo and Sify are also taking big steps in Hindi and other regional languages. Despite the latent demand among Internet users for Hindi, if there is very dismal use of Hindi, it is due to certain constraints. These include technological, attitudinal and economic factors. New search engines like Raftaar and Guruji are now available. They give user the option to type his queries in Hindi and it searches only the Hindi websites. Raftaar and Google also give the user a soft keypad to write in Hindi. The last, but not the least constraint in spread of Hindi over the net is that of limited content. Where there are more than 20 billion pages on web in English, this number is not more than 10 million in Hindi. This poverty of content is partly due to technological factors and partly due to attitudinal. It is quite disappointing to see that most efforts to spread Hindi over the net are done by foreign companies. The same is the situation about content. It is a big dilemma that on the one hand the number of Hindi readers and number of Hindi-speaking people using mobile and computers is so large, and on the other hand the websites are very limited in their content and number. This dilemma should be overcome as soon as possible. For more information please refer to [5] 2. SEARCH ENGINES SUPPORTING HINDI LANGUAGE 3.1 GOOGLE Google is capable of searching information in many Indian languages successfully. Google Docs in Google Apps now supports all popular local languages of India including Hindi, Tamil, Telugu, Gujarati, Kannada, Malayalam, Bengali, Marathi and Oriya. Google also provides virtual keyboard for languages mentioned above.

2 ( 3.2 BING Bing (formerly Live Search, Windows Live Search, and MSN Search) is a web search engine (advertised as a "decision engine") from Microsoft. Bing was unveiled by Microsoft CEO Steve Ballmer on May 28, Bing is available in many languages and has been localized for many countries. Bing supports Hindi Language but do not provide virtual keyboard to enter search queries. 3.3 GURUJI Guruji.com is an Indian Internet search engine that is focused on providing better search results to Indian consumers, by leveraging proprietary algorithms and data in the Indian context. In addition to its tool for searching WebPages, Guruji Search also provides services for searching images, services within a specific city, music, cricket, movie timing, and finance. Guruji.com was launched to the world on October 16, Guruji does not provide virtual keyboard for typing Hindi queries. 3.4 RAFTAAR Raftaar रझत र, a Hindi search engine has been recently launched at the domain Raftaar search features sections on news, photos, blogs, music, sports, movies, education etc. The site also displays an onscreen Hindi keyboard layout for easy typing. The site has been launched by Indicus Netlabs. For conducting experiments we chose Google, Bing and Guruji famous and widely used search engines for Hindi information retrieval. 3. FACTORS AFFECTING PERFORMANCE OF HINDI LANGUAGE SEARCHING ON WEB. There are various factors which affect searching Hindi information on web. 4.1 Morphological Factors 4.2 Phonetic nature of Hindi Language 4.3 Words Synonyms 4.4 Ambiguous Words 4.1 MORPHOLOGICAL FACTORS Morphology is the field of linguistics that studies the internal structure of words. While words are generally accepted as being the smallest units of syntax, it is clear that in most (if not all) languages, words can be related to other words by some grammatical rules. The rules understood by the speaker reflect specific patterns (or regularities) in the way words are formed from smaller units and how those smaller units interact in speech. Morphological analysis is essential for Hindi because it has a fairly rich system of inflectional morphology like other Indian languages. We have taken a sample set of 50 queries to test the affect of the root word. Following table (4.1) is the set of randomly selected queries from the set which throw light on the effect of the root word on the performance of Hindi language search engines. Table shows the results of experiments for effect of morphological factors on Hindi queries. It has been observed that documents returned by all three search engines are more in number when query with root word is submitted. This justifies the searching of documents in the root word because in general we get better results with the keywords in their root form. In case of search engines the quality of results is more important than the quantity. Graph shows the comparison of the precision values of the three search engines. The precision value is calculated by taking the top 10 results of the search engines. On closely observing the results we can say that precision value in case of Google is high in almost all queries. 3.2 PHONETIC NATURE OF HINDI LANGUAGE AND SPELLING VARIATIONS Many different languages are spoken in India, each language being the mother tongue of tens of millions of people. While the languages and scripts are distinct from each other, the grammar and the alphabet are similar to a large extent. One common feature is that all the Indian languages are phonetic in nature. [6] The major reasons for spelling variations in language can be attributed to the phonetic nature of Indian languages and multiple dialects, transliteration of proper names, words borrowed from foreign languages, and the phonetic variety in Indian language alphabet. The variety in the alphabet, different dialects and influence of foreign languages has resulted in spelling variations of the same word [7]. For example; Following are the possible spelling variations for the Hindi word अ म ज (angrējī): (means English) Following table below show the results offered by Google Bing and Guruji. The first column of the table contains the queries. Second column of the table contains the phonetically equivalent words of the word in the keyword/query. Third column shows the number of documents returned by the search engines In the above table it can be clearly seen that search engines return a handful of documents on various phonetically equivalent words of same word in the query. It can be concluded that no particular standard exists for writing the keyword to fetch Hindi web data. It has also been observed that for every phonetically equivalent word variation in results exists. Therefore it becomes obligatory for the search engines to support phonetically equivalent words without letting user to make hard efforts for typing phonetically equivalent keywords. 4.3 WORDS SYNONYMS The researchers observed that, a word can express a myriad of implications, connotations, and attitudes in addition to its basic dictionary meaning. And a word often has near synonyms

3 that differ from it solely in these nuances of meaning. Choosing the right word can be difficult for people, as well as for the information retrieval system. For example the word (Hindi आभ षण) (English Ornaments) has following commonly spoken synonyms गहन, ज वर, अल क र. In our experiments it is justified after analysis that these four issues are needed to be addressed by the search engines to improve information retrieval in Hindi Language. For example the word (Hindi) आभ षण is a standard word against Gold in (English) and its synonyms are गहन, ज वर, अल क र, आभरण. Six random queries are presented in the Table below which are selected from a set of 50 tested queries. A graph has also been presented below which shows the comparison of precision values against three search engines. From the above table and graph it is be observed that documents returned by Google are more in quantity than other two search engines and least number of documents are returned by Guruji search engine the reason behind may be availability of less documents or poor indexing. However we are interested in quality of results than quantity where quality of results is considered it can be clearly seen that Google and Bing provide quality data than Guruji. And in the average case Google still stands first in the row that means precision values by Google are more than that of Bing and Guruji in this case. 4.4 AMBIGUOUS WORDS OR WORD SENSE DISAMBIGUATION Many words are polysemous in nature. Finding the correct sense of the words in a given context is an intricate task. One word has more than one meaning and meaning of word is depends on context of sentence. Ambiguous words deflate the relevancy of the results. The examples mentioned below shows this aspect very clearly. Consider the following query (In English) (Women like gold) (In Hindi) (न र क स न पस द ह ). In this query the word स न (Gold) is ambiguous as it has another meaning i.e. to sleep. In the context of above query the word स न is gold. But it can be also interpreted as women like to sleep. Another query (In English) (The common people's choice) (In Hindi) (आम ल ग क पस द). Here the word आम is ambiguous. The word आम in above query means common. However, In Hindi it also means mango. So the above query can be interpreted as mango is people s choice. Various experiments have been done on this issue on three search engines to check their performance on handling of ambiguous word in a particular context. In table (4.4.1) second column contains five queries in Hindi, third column holds the ambiguous keyword in one context and fifth column holds the same ambiguous keyword in other context. Fourth and sixth columns hold the meaning of queries in English with respect to the ambiguous keyword in context. Ambiguous queries mentioned above in the table are tested for results against three search engines Google, Bing and Guruji. Results are shown below in tables 4.4.1, and as. From the above results obtained in tables it is observed that all three search engines return documents without differentiating between the contexts of keyword in the query. In the above table the last column labeled as other holds the number of results which are not relevant to the query supplied or those documents which contains the keywords in other non required context. E.g. ambiguous keyword क ल can be prefixed with प त as क लप त which means vice chancellor and the query supplied above with क ल as keyword has nothing to with vice chancellor. From the results in column no five, seven and eight it is clear that all search engines return documents in different contexts. Therefore it can be concluded that search engines underperform when supplied with ambiguous queries. DISCUSSION AND CONCLUSION We tested the performance of three search engines Google. Bing and Guruji for the challenges mentioned in above. After the comparative analysis it is concluded that query when supplied with rood word yields maximum results. It is also evident that Google indexes the keyword in its root form and also lists the documents consisting of the morphological variants of the keywords. The other two engines Bing and Guruji do not list any morphological variant and hence they entail to have stemmer. It is apparent from the table (4.2.1) that phonetic variation in Hindi keyword has a great impact on the performance of search engines. All three search engines show variations in the results for the same keyword written in phonetically different form. For each phonetically different word different set of results are obtained. Thus it is concluded that all three search engines Google, Bing and Guruji are not phonetic tolerant. Word synonyms also play a major role in hindi information retrieval process as it has been tabled above that a word with its synonyms when supplied to the search engines results in the retrieval of hand full of documents and none of the search engines is capable of listing a synonym of a keyword in the documents retrieved. However, the precession values of Google are better than other two search engines. Ambiguous words in a query bring down the performance of search engines. None of the search engine is capable enough to handle the problem of ambiguity in query. It was observed that search results were far from relevance and results obtained are out of context in almost all the cases. The above discussion be evidence of that we need a standard Hindi search engine which should implement the structure to overcome the aforesaid challenges

4 REFERENCES [1]. Praveen Kumar, Shrikant Kashyap, Ankush Mittal Indian Institute of Technology, Roorkee, India. A Answering System for E-Learning Hindi Documents SOUTH ASIAN LANGUAGE REVIEW VOL.XIII, Nos 1&2, January-June,2003 [2]. S.K. Dwivedi and Parul Rastogi Babasaheb Bhimrao Ambedkar University (A Central University), Lucknow, Uttar Pradesh, India An Entropy Based Method for Removing Web Ambiguity in Hindi Language Journal of Computer Science 4 (9): , 2008 ISSN Science Publications [3]. S.K. Dwivedi and Parul Rastogi Rajesh kr. Gautam Impact of language morphologies on search engine performance for hindi and English language (IJCSIS) International Journal of Computer Science and Information Security, Vol. 8, No. 3, June [4]. Payas Gupta School of Information System Singapore Management University 80 Stamford Road, Singapore, Stemmer for Indo-Aryan Languages: A survey URL: pdf [5]. Ranjan Srivastava, Chief of Bureau, Prabhat Khabar, (Friday, April 28, 2006) The future of Hindi on the Internet [6]. GANAPATHIRAJU Madhavi, BALAKRISHNAN Mini, BALAKRISHNAN N. REDDY Raj Om: One tool for many (Indian) languages Journal of Zhejiang University SCIENCE ISSN [7]. Vishal Goyal, Ph.D. Development of a Hindi to Punjabi Machine Translation System A Doctoral Dissertation Volume 10: 10 October 2010 ISSN Language in India S. No in Hindi Table 4.1 List of Hindi queries Meaning In English 1 भ रत वष र वन Indian rain forest 1.1 भ रत य वष र वन Indian rain forests 2 हव ई द घर टन क क रण Reason for air crash 2.1 हव ई द घर टन ओ क क रण Reason for air crashes 3 भ रत म ब ल ज न व ल भ ष Language spoken in India भ रत म ब ल ज न व ल भ ष ए भ रत म ब ल ज न व ल भ ष ओ 4 वल प त ह न पर झ ल 4.1 वल प त ह न पर झ ल Languages spoken in India Languages spoken in India Lake on the verge of extinction Lakes on the verge of extinction Graph precision values of the three search engines

5 S. No Root word/s Listing of Keywords Google Bing Guruji Morphological variants Documents Returned Google Bing Guruji 1.1 भ रत, 50,500 4, भ रत, भ रत, वष र वन, वष र वन, भ रत,वष र वष र वन वष र वन वष र वन, वन वन वन, वन 1.2 भ रत य 40, वष र वन 2 द घर टन द घर टन,द घर टन 2.1 द घर टन 133,000 2, द घर टन द घर टन ओ 2.2 द घर टन ओ 117, भ ष 161,000 8, भ ष भ ष, भ ष ओ, भ ष भ ष 3.2 भ ष ए 6, भ ष ओ 6, झ ल झ ल, झ ल, झ ल झ ल 4.1 झ ल 4, झ ल 1, Table Effect of morphological factors on Hindi queries हन द क वत ए खतरन क ध आ करण ब द गण श त य ह र झ स क र न Documents Phonetically equivalent words Returned Google Bing Guruji हन द क वत ए 1,350,000 14,100 3,218 ह द क वत ए 1,120,000 12,100 1,933 हन द क वत ए 153,000 20,100 9,206 ह द क वत ए 112,000 20,200 5,431 ध आ ध आ 18, ध आ 2, करण 101,000 4, करन 16, गण श 30,100 4,320 3,387 गन श झ स क र न 158,000 8, झ स क र न 5,500,000 4, Variations in Results Table results of three search engines on Phonetic nature of Hindi language

6 S. No. Standar d Hindi Word/s Synonyms 1.1 स न क आभ षण Google Documents Returned Bing Per. ji 217, , स न क गहन 188, , स न क आभ षण आभ षण/ गहन 1.3 स न क ज वर 78, , स न क 9, अल क र 1.5 स न क आभरण क ल ब दल 233, , क ल ब दल ब दल 2.2 क ल म घ 40, , क ल जलधर 1, स तर सक दर क अ हक र स तर,न र अ हक र 3.1 स तर 9, , न र 29, , म हल 96, , , औरत 7, सक दर क अ हक र 1, सक दर क अ भम न 2, Table comparison of precision values against three search engines Graph comparison of precision values against three search engines

7 S.No For keyword as In English For keyword as In English 1 न र क स न पस द ह स न (Gold) Women like Gold स न Women like to sleep (To sleep) 2 आम ल ग क पस द आम (common) Common man s choice आम (Mango) Mango is people s choice 3 ब ल वक स और प षण ब ल (Children) Child Development and ब ल(Hair) Hair Nutrition Development and Nutrition 4 सप र क फन फन (Art) Art of snake charmers फन(Snake head) Snake charmer s 5 य द ध म क ल वन श क ल (Aggregate) Aggregate destruction in wars क ल(family) Table List of randomly selected ambiguous queries snake head Destruction of families in war Google Ambiguou Documents Results Found Other s keyword returned 1 स न 50,800 Gold 5 To sleep आम 488,000 Common 3 Mango ब ल 2,900,000 Children 7 Hair फन 184 Art 0 Snake head क ल 17,800 Aggregate 2 Family 3 5 Table Ambiguity test for Google Bing Ambiguous Documents Results Found Other keyword returned 1 स न 2,680 Gold 2 To sleep आम 17,800 Common 3 Mango ब ल 4,030 Children 6 Hair फन 25 Art 0 Snake head क ल 1,900 Aggregate 0 Family 2 8 Table Ambiguity test for Bing Guruji Ambiguous Documents Results Found Other keyword Returned 1 स न 109 Gold 0 To sleep आम 6,756 Common 3 Mango ब ल 635 Children 5 Hair फन No Results Found Art n/a Snake head n/a n/a 5 क ल 84 Aggregate 0 Family 0 10 Table Ambiguity test for Guruji