Web Crawling. Jitali Patel 1, Hardik Jethva 2 Dept. of Computer Science and Engineering, Nirma University, Ahmedabad, Gujarat, India

Transcription

1 Web Crawling Jitali Patel 1, Hardik Jethva 2 Dept. of Computer Science and Engineering, Nirma University, Ahmedabad, Gujarat, India Abstract- A web crawler is a relatively simple automated program that methodically scans or "crawls" through Internet pages to create an index of the data it is looking for. Process is called Web crawling or spidering.basically web crawler provides scope to search engine, for the purpose of finding web pages from the World Wide Web in simply manner. In today s world each search engine has its own crawler and crawling techniques. In this paper we introduce search engines, different types of crawling techniques, crawler architecture, crawler algorithms, diff. types of crawler issues and types of crawler. Need of web crawling is To Test web pages and links for valid syntax and structure and search for copyright permissions. A front-end user interface and a supporting querying engine which queries the database and presents the results of searches so in this way web crawling works with search engine. Keywords: Web crawler, web crawling, search engine, crawler algorithms, crawling techniques. I. INTRODUCTION Web crawling technique is related to search engine so first introducing about search engines, how search engine works and what are the different types of search engine in this world. Web Search Engine: Web Search Engine is a tool enabling document search, with respect to specified keywords, in the Web and returns a list of documents where the keywords were found. Rf. [9] Search engine components: User Interface Parser Web Crawler Database Search Engine = Crawler + Indexer/searcher +GUI Ex. Google, Yahoo, Ask, AltaVista, msn etc. are different search engines available. A web crawler is a relatively simple automated program, or script that methodically scans or "crawls" through Internet pages to create an index of the data it is looking for. Process is called Web crawling. Needs of web crawling: To Test web pages and links for valid syntax and structure. To monitor sites to see when their structure or contents change. To search for copyright permissions. To build a special-purpose index. For example, some content of images stored on the web then that pages index created. The rest of paper contains all about crawler and crawling techniques in that section wise contents II. Types of web crawling, III. Web Crawler, IV. Web crawler algorithms, V. Conclusion mentioned in the paper. 228

2 Fig 1: How crawling works with search engine [11] II. TYPES OF WEB CRAWLING Breadth first crawling, Depth first crawling, Repetitive crawling, Targeted crawling, Deep web crawling are the crawling techniques used for web crawler. Breadth first crawling: In this type of crawling breadth first search algorithm is used. This type of crawling implementing using (queue) data structure. In this type of crawling crawler first start fetching pages from lowest level in graph and then checking neighbor nodes step by step and at the end traverse whole graph of finding pages from internet so in this way breadth first crawling technique used in finding relevant pages from the www. Depth first crawling: In this type of crawling depth first search algorithm is used. This type of crawling implementing using (stack) data structure. In this type of crawling crawler first start fetching pages from highest level in graph and in those level relevant neighbour pages find by crawler and then decrement level count and goes into lower level for fetching pages step by step. Repetitive crawling: In this type of crawling once page have been crawled, some systems require the process to be repeated periodically so that indexes are kept updated. Targeted Crawling: Here main objective is to retrieve the greatest number of pages relating to a particular subject by using the Minimum Bandwidth. Most search engines use crawling process heuristics in order to target certain type of page on specific topic. Deep Web Crawling: The data that which is present in the data base may only be downloaded through the medium of appropriate request or forms this Deep Web name is given to this category of data. For ex. searching hidden pages from databases. [6] Fig 2: BFS and DFS Technique Rf. [11] 229

3 III. WEB CRAWLER Architecture of web crawler: Fig 3: Architecture of web crawler Rf. [11] Maintains a list of unvisited URLs called the frontier, list is initialized with seed URLs which may be provided by a user or another program. In the crawling loop, next url taken from the queue and then fetching the page corresponding to url using http protocol. Each url contains some specific score before adding them into frontier so it helps in priority scheduling of url.when the all pages fetched according to url given in queue then crawling process over and suppose crawler ready for crawl another page but queue is empty then that situation is called dead-end of crawler. [2] Crawler policies: The characteristics of web that make crawling difficult: Its Large Volume Its Fast Rate of Change Dynamic Page generations [1]. 1. Selection policy that states which pages to download. This requires matrices of importance for prioritizing Web pages. 2. Re-visit policy that states when to check for changes to the pages.now in today s world web is very dynamic so pages in www are being updated and some content also delete from that pages so there would be need to revisit that pages through crawling. 3. Politeness policy today s world overloading of websites in www is the biggest issue so how to overcome with that this strategy is included in politeness policy. Needless to say, if a single crawler is performing multiple requests and downloading many files then server load increases so overloading of websites occur. 4. Parallelization policy that states how to coordinate distributed web crawlers.a parallel crawler can run multiple processes in parallel so that downloading rate of pages increases and bandwidth of the data minimize. Crawler implementation issues: Fetching: Client-server mechanism kind of thing in that http client send request to http server for fetching page. In that timeout is the biggest issue so for that no unnecessary time spent on single server. 230

4 Error and exception Handling is also issues in fetching web pages [3]. Robot exclusion protocol: webserver admin access related permission provide by this protocol for the purpose of file access. some file may not access by the crawler so for that file named robot.txt contains list of this files so this files cant access by crawler [3]. Parsing: Parsing contains the task of simply hyperlink/url extraction and contains process of identifying html attributes and from that attributes build HTML tag tree so that url canonicalization can be easily achieved. Remove stop words from the page and new URL added into queue this is the responsibility of parsing [4]. Stop listing: Remove commonly used words or stop words such as it" and can. Process of removing stop words from text is called stoplisting.system recognizes no more than nine words ( an", and", by", for", from", of", the, to", and with") as the stop words. Stemming: Stemming process is used for normalization of words in pages for example connected, connection words are stemmed into connect. URL Extraction and Canonicalization: For URL extraction first thing crawler need to do is find href attributes in pages so that new url can be found. Then crawler converts relative url into absolute url and different url mapped onto single url. HTML tag tree: Crawlers may assess by examining the HTML tag context in which it resides. The crawler only needs the links within a page, and the text or portions of the text in the page by using HTML parsers. Fig 4: HTML tag tree Rf. [11] URL Normalization: More than once URL contains ambiguity so there would be need to do normalization. Also called URL canonicalization, in that conversion of URL from lowers case to upper case so that ambiguity is reduced, then removal of. And, is also part of URL normalization. IV. WEB CRAWLING ALGORITHMS Crawler Basic Algorithm: Remove a url from the unvisited url list Determine the ip address of its host Download the corresponding document Extract any links contained in that document If the link url is new then add into list of unvisited urls Process the downloaded document Back to step1 [3]. 231

5 Fig 5: Crawler basic algorithm[11]. Breadth First Search Algorithm: This algorithm aims in the uniform search across the neighbour nodes. It starts at the root node and searches the all the neighbour nodes at the same level. If the objective is reached, then it is reported as success and the search is terminated. If it is not, then it goes down in the next level and fetching page across neighbor nodes until the objective is not finished [2]. 1. put all the given seeds into the queue; 2. Making list of visited nodes. 3. When queue is not empty then: a. Remove the first node from the queue; b. Append that node to the list of visited nodes c. For each edge starting at that node: i. If the node at the end of the edge already appears on the list of visited nodes or it is already in the queue, then do nothing more with that edge; ii. Otherwise, append the node at the end of the edge to the end of the queue. Fig 6: BFS algorithm [11]. Depth first Search Algorithm: This powerful technique of systematically traverse through the search by starting at the root node and traverse deeper through the child node[2]. Get the 1st link not visited from the start page Visit link and get 1st non-visited link Repeat above step till no non-visited links Go to next non-visited link in the previous level and repeat 2nd step 232

6 Fig 7: DFS algorithm [11] Best first Algorithm: Different Best first strategies of increasing complexity in crawling for avoiding this conflict the best sophisticated criteria is selected according to this criteria link is prioritised and put it into queue.thus the similarity between a page find the topic keywords is used to estimate the relevance of the pages pointed bypath URL with the best estimate is then selected for crawling. The sim() function gives similarity between topic and pages.: [10]. 233

7 Fig 8: Best first algorithm [11]. Where p is the page and q is topic, and fkd is the frequency of term k in d. Page Rank Algorithm: Page rank algorithm determines the importance of the web pages by counting citations or backlinks to a given page. PageRank is an algorithm used by Google Search to rank websites in their search engine results. PageRank is a link analysis algorithm and its assigns score to hyperlinked set of documents so the relative importance is measured. Fig 9: Page rank algorithm [11]. Mathematical PageRank for a simple network, expressed as percentages. (Google uses a logarithmic scale.) PageRank C > PageRank E, even though there are fewer links to C; the one link to C comes from an important and higher page rank Hence the value of page rank is higher. Where, out(d) is the set of links out of d, p is the page being scored, in(p)is the set of pages pointing top, and the constant gamma<1 is a damping factor and it represents probability of random pages [7]. V. CONCLUSION Web crawling processes demanded high performance is the basic components of various Web services. Crawlers are being used for collecting data from web and data mining purpose when a number of crawling processes migrate to different locations and run parallel they make the crawling process fast and they save enormous amount of time in crawling. The documents collected at each 234

8 site are filtered. So only the relevant pages are sent back to the central crawler and this saves network bandwidth. The documents before being sent to the central crawler are compressed locally and then sent to the central crawler which saves a large amount network bandwidth. So for efficient web crawling required efficient algorithm and then that scope is provided to build a robust and effective web crawler so that is beneficial to search engines in this way this web crawling technique is useful in different type of search engines. REFERENCES [1] Douglas E. Comer, The Internet Book, Prentice Hall of India, New Delhi, 2001 [2] Web mining text book by chakrabarti. [3] A. Barabasi and R. Albert. Emergence of scaling in random networks. Science, 286(509), [4] A.K. Sharma, J.P. Gupta, D. P. Aggarwal, PARCAHYDE: An Architecture of a Parallel Crawler based on Augmented Hypertext Documents. [5] S. Brin and L. Page, The anatomy of a large scale hyper textual Web search engine, Technical Report, Stanford University, Stanford, CA, 1997 [6] Maurice de kunder, Size of the world wide web, retrieved from [7] Marc Najork, Web Crawler Architecture retrieved from accessed on 10/8/11 [8] M. Burner. Crawling towards eternity: Building an archive of the World Wide Web [9] [10] [11] 235