SEARCH ENGINE MANAGEMENT

Transcription

1 e-issn Volume 2 Issue 5, May Scientific Journal Imact Factor : htt:// SEARCH ENGINE MANAGEMENT Abhinav Sinha Kalinga Institute of Industrial Technology, Bhubaneswar, Odisha abhinav.sinha94m@gmail.com Abstract The Internet is growing at a raid rate; this growth of Internet in ast decade has been influenced by effective data mining techniques, most exressive examles of it being Search Engine giants like Google and Yahoo. We in this aer resent the roblems the search engines today are facing and we roose a model considering the ergonomic and environmental imrovements which could cater to the resent information needs relating to whereabouts in the WWW. Keywords Search engines, thick client, rich client, udation delay, data mining, otimization I. INTRODUCTION Internet has bloomed into youth from a nascent state, and is currently feeding on its enormity. And finally Internet is available to the common man. But since the growing enormity of this web-sace, we are in a race against time to develo the fastest and smartest search engine, to fill the gas in between. Search Engine as defined by Wikiedia [1]: A Web search engine is a tool designed to search for information on the World Wide Web. The search results are usually resented in a list and are commonly called hits. The information may consist of web ages, images, information and other tyes of files. Some search engines also mine data available in databases or oen directories. Unlike Web directories, which are maintained by human editors, search engines oerate algorithmically or are a mixture of algorithmic and human inut. Figure 1. A Simle Search Engine Now for a ractical exlanation, three major rocesses run within any search engine. Firstly the webages are downloaded or Crawled using rograms called web-crawlers. These Crawlers save resourceful data from internet to the storage or web reository. The Semi-structured data from these reositories are used to deduce useful information such as the information value of a webage and the index of keywords of any website. This rocess is called Indexing, the over all efficiency of techniques used for it are directly resonsible for the relevance and quality of the results All rights Reserved 254

2 The third rocess is the search itself which is fired by the user; it is resonsible for the ergonomics of the end-roduct. Figure 2. A Large Scale Search Engine. [2] Large Scale Search engines maintain a gigantic web reository of web ages. This reository is indexed with hel of data mining tools for faster searches. The crawlers reside on high erformance servers sread through out the world and crawling the web 24x7. When they find an un-indexed age they store it in reository and index it. Whereas when a web age is revisited during crawling, if the content is changed an udate is made and the old age in reository is relaced by new age. Commercial search engines have eriodic udate olicies for maintaining a relevant web reository. [3] 1.1 The Good Side 1. The Search Engines today rovide reliable information, in context of the search results roduced. The Search Engines aly NLP to roduce a smarter outut, by considering the etymological relatives and synonyms of the keywords, not to forget, you will observe a Do You Mean Boy link when you missell BOY as BYO or BIY. 2. The Search Engines have landed into domains other than lain text mining. Now you can have an image search, audio search, and other multimedia mining alications. 3. The Seed of resent day Search Engines are enviable, than what is was back in 1992 when Archie saw this world. Google requires fractions of second to load. 4. The results are arranged in an increasing order or relevance of the information from the WebPages, relative to the search query and user exectation. This is done with hel of metrics such as age-rank and tracing user clicks. Thus we get an ergonomically enhanced result. 1.2 The Bad Side 1. Search Engines take a lot of time to udate any website, often web-ages are not modified according to strict deadlines. Thus Periodic Udates have a fat chance of bringing obsoleteness to All rights Reserved 255

3 Figure 3. Google s first roduction server. [4] 2. Servers are devoted for the secific urose of crawling; the servers thus send a lot of time downloading ages redundantly, and ut a load on both the web-server hosting the various websites and the hysical network. This redundancy could be estimated in terms of energy or ower consumtion. From this we might conclude that Environmental effects of search engines can t be negotiated. S Invisible Web Figure 4. Coverage (Left) and the Dee Web. [3] 3. Search Engines still lag behind in coverage, as a major ortion of the WWW is inaccessible to the search engines, due to various reasons, such as Dynamic Content, broken links or absolute gas between the arts of internet within coverage and less referenced ages. Such inaccessible ortion of the web is known as Invisible Web or Dee Web. Only very small no. of web ages is indexed amounting about 6-12 %. [5] This is due to following reasons: 1 Search engines have limited bandwidth 2 Crawlers may not reach imortant ages because they are too far. 3 Deth of crawl is fixed, and age is located too dee. The art of World Wide Web which is not restricted by dynamic content is also amalgamated into dee web due to inability of the crawler to find a link to it from the set of age it is crawling. This may be visualized from given figure All rights Reserved 256

4 4. The ages crawled by the search engine are downloaded and saved into the web reository, and then indexed. If the data already exists online what is the need to save it into reositories maintained by search engines doing this we are unknowingly wasting storage resources? II. MATHEMATICAL MODELLING A Rich client is a client in client-server architecture with rocessing caabilities. Here most of the work is carried out by the clients. In our roosal of a Rich Client based model for search engine, we have user agents which run on the web-servers, do local crawling and submit a XML sitema of the comlete site and its links to the outside world, along with the data useful for indexing such as keywords and frequencies of different tokens. These user agents send sitemas to the Central server of our search engine whenever a change is made to the website. Thus our udate time is instantaneous leaving the network delay in transmission. The central server then reads the sitema and udates its local database, and does the corresonding age-rank comutations at that time. Figure 4. Concetual Rich Client Model for Search Engines. [3] Reduced Time Comlexity The time required by the traditional crawlers to crawl a articular website will be as given below. n S T ( ) * Dn (1) k B Where n is the no. of ages k is a constant defined by the techniques used for concurrency. S is the average web age size B is the bandwidth of the channel D n is the network delay involved. Our time required to crawl the given website will All rights Reserved 257

5 n S T' ( )*( ) Dn (2) k B f Here Bf = bandwidth of local browsing B f B 2 10 Since the XML sitema file size increases with no. of ages using our technique, the D n should be changed with the increase in size of file transferred. But for ease of simlicity we can omit this art, as there is no significant change in our comarison. Comaring these two equations we see that the Network delay is multilied each time, so for a ractical channel the time required by a traditional search engine will be much higher than our roosed crawler. The crawling of the site is said to be fruitful only if the content has changed, therefore the equations 1 & 2 need to be modified to find fruitful time as Where λ is frequency of change n S T n f ( ) * Dn * P( Change) ( t) k B P( Change) e n! n is the no. of arrivals (change events). Assuming the change of age content is a Poisson rocess the crawler will be effective only when the crawling is scheduled so that P (Change) is maximum which is exactly at λ. T f t n S * D * ( ) ( ) 1 n P Change P Change k Bf P (change) in above equation become = 1, as the crawl event is fired by a change of content. These equations can be lot, and then the area covered by each curve gives the successful effort alied by each of them. The traditional crawler s erformance varies with robability of change. [3] III. CONCLUSIONS We have found the drawbacks of the resent search engines, and roosed a new model. The resent Search Engines are time efficient but not energy efficient. The Model though bare-bone is able to reduce redundancy and saves exloitation of recious resources such as ower and time. This Model could be used to make a hybrid search engine, which will assist in the switching of trends in search engines. At a first level imlementation a ractical rich client search engine is ossible using the comutation resources less demanded for internet alications such as web servers; which can be arranged without exhaustively exloiting them. Rich clients can be further modified to collaborate with each other, and thus romote more use of untaed resources. ACKNOWLEDGEMENTS The authors would like to thank everyone, just everyone! REFERENCES [1] Web search engine - Wikiedia, the free encycloedia Wikiedia, 2009 [2] Sergey Brin, L.P. The Anatomy of a Large-Scale Hyertextual Web Search Engine Stanford University, 2000 [3] Praval Kumar Jha, K.N. Rich Client Search Engines 2009,. 4 [4] Wikiedia Google - Wikiedia, the free encycloedia Wikiedia, 2009 [5] Guckian, K. Internet 202: The Invisible Web - Beyond Google All rights Reserved 258

6 [6] Boswell., D. Distributed High Performance Web-Crawler: A Survey of the State of the Art. Dett of Comuter Science, Stanford University, [7] Chowdhry, A. Google Disutes The Times Online UK s Claim About Environmental Imacts On Search queries PULSE2, 2009 [8] Chowdhry, A. Environmental Fellow at Harvard Alex Wissner-Gross Says Times Online Got It Wrong PULSE2, 2009 [9] Margaret H. Dunham, S.S. Introduction to Data Mining [10] Onn Brandman, Junghoo Cho, H.G.-M.N.S. Crawler Friendly Web Servers [11] Wissner, D. Study on Imact of search engines on Global Warming. The Sunday Times, All rights Reserved 259