Web Clients and Crawlers

Size: px

Start display at page:

Download "Web Clients and Crawlers"

Amy Carter
6 years ago
Views:

1 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser MCS 507 Lecture 21 Mathematical, Statistical and Scientific Software Jan Verschelde, 14 October 2013 Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

2 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

3 Web Clients Usually we download files with a web browser, but we can browse the web using scripts. Why do we want to do this? 1 more efficient: no overhead from a GUI 2 in control: request only what we need update most recent information 3 crawl the web: search recursively operate like a search engine 4 the SymPy HTML documentation is interlinked search the documentaton files off line Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

4 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

5 web pages as files Opening a web page with urllib.urlopen: from urllib import urlopen < object like file > = urlopen( < URL > ) data = < object like file >.read( < size > ) < object like file >.close() we process web pages like we handle files Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

6 opening a web page The SymPy HTML documentation comes with the distribution of the sympy package. from urllib import urlopen URL = file:///users/jan/courses/mcs507/ \ + sympy docs-html/index.html PAGE = urlopen(url) DATA = PAGE.read(80) print 80 characters :, DATA LINES = DATA.split( \n ) print the lines :, LINES PAGE.close() Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

7 function to copy a web page to a file def copypage(url, filename): Given the URL for the web page, a copy of its contents is written to filename. Both url and file are strings. import urllib copyfile = open(filename, w ) webpage = urllib.urlopen(url) while True: data = webpage.read(80) print data if data == : break copyfile.write(data) webpage.close() copyfile.close() Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

8 urlretrieve With urllib.urlretrieve we copy a web page to file. >>> u = file:///users/jan/courses/mcs507/ \... + sympy docs-html/index.html >>> from urllib import urlretrieve >>> urlretrieve(u, mycopy.html ) ( mycopy.html, <mimetools.message instance at 0x1011cb878>) >>> f = open( mycopy.html, r ) >>> d = f.readlines() >>> print d[:4] [ \r\n, \r\n, <!DOCTYPE html PUBLIC "-// W3C//DTD XHTML 1.0 Transitional//EN"\r\n, " ansitional.dtd">\r\n ] Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

9 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

10 scanning HTML files Applications to scan an HTML file: 1 search for particular information, 2 navigate to where the page refers to. Example (1): download all.py files from jan/mcs507/main.html Example (2): retrieve all URLs the page refers to. What is common between these two examples:.py files and URLs appear between " and " scan for all strings between double quotes Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

11 scanning for double quoted strings Input: Output: a file, or object like a file. list of all strings between double quotes. Recall that we read files with fixed size buffer:..."....".."."...."..".".."... For double quoted strings which run across two buffers we need another buffer. Two buffers: Two functions: one for reading strings from file, one for buffering double quoted string. 1 read buffered data from file, 2 scan the data buffer for double quoted strings. Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

12 reading strings from file def quoted_strings(page): Given a file page, this function scans the page and returns a list of all strings on the file enclosed between double quotes. result = [] buff = while True: data = page.read(80) if(data == ): break (result, buff) = update_qstrings(result, \ buff, data) return result Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

13 processing the buffers In L we store the double quoted strings. In b we buffer the double quoted strings...."... L = [], b = "....".."." L = [...,. ], b =... L = [...,. ], b =.".."." L = [...,.,.. ], b = ".."... L = [...,.,..,.. ], b = def update_qstrings(quoted, buff, sdat): quoted is a list of double quoted strings, buff buffers a double quoted string, and sdat is the data string to be processed. Returns an update of (quoted, buff). Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

14 code for update_qstrings def update_qstrings(quoted, buff, sdat): ".." newb = buff for i in range(0, len(sdat)): if(newb == ): if(sdat[i] == \" ): newb = o # o is for opened else: if(sdat[i]!= \" ): newb += sdat[i] else: # do not store o quoted.append(newb[1:len(newb)]) newb = return (quoted, newb) Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

15 the function main() def main(): Prompts the user for a file name and scans the file for double quoted strings. print getting double quoted strings name = raw_input( Give file name : ) datafile = open(name, r ) qstr = quoted_strings(datafile) print qstr datafile.close() if name == " main ": main() Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

16 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

17 scanning web pages for URLs Recall the second example application: list all URLs referred to at def main(): Prompts the user for a web page, and prints all URLs this page refers to. print listing reachable locations page = raw_input( Give URL : ) links = http_links(page) print found %d HTTP links % len(links) show_locations(links) Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

18 filtering a double quoted string from scanquotes import update_qstrings def http_filter(names): Returns from the list names only those strings which begin with http. result = [] for name in names: if(len(name) > 4): if name[0:4] == http : result.append(name) return result Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

19 scanning an HTML file for HTTP strings def http_links(url): Given the URL for the web page, returns the list of all HTTP strings. import urllib page = urllib.urlopen(url) result = [] buff = while True: data = page.read(80) if data == : break (result, buff) = update_qstrings(result, \ buff, data) result = http_filter(result) page.close() return result Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

20 showing only server locations using the module urlparse An URL consists of 6 parts protocol://location/path:parameters?query#frag Given URL u, urlparse.urlparse(u) returns 6-tuple. def show_locations(locs): Shows the locations of the URL in locs. from urlparse import urlparse for location in locs: tupl = urlparse(location) print tupl[1] Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

21 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

22 web crawlers Scanning HTML files and browsing: 1 given a URL, open a web page, 2 compute the list of all URLs in the page, 3 for all URLs in the list do: 1 open the web page defined by location of URL, 2 compute the list of all URLs on that page. continue recursively, crawling the web Things to consider: 1 remove duplicates from list of URLs, 2 do not turn back to pages visited before, 3 limit the levels of recursion, 4 some links will not work. This is similar to finding a path in a maze. Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

23 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

24 modular design of web crawler # L-21 MCS 507 Mon 14 Oct 2013 : webcrawler.py Prompts the user for a URL and the maximal depth of the recursion tree. Lists all locations of web servers that can reached starting from the user given URL. from scanhttplinks import http_links Still left to write: 1 management of list of server locations, 2 recursive function to crawl the web. Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

25 retain only new locations def new_locations(urls, vis): Given the list urls of new URLs and the list of already visited locations in vis, returns the list of new locations, locations not yet visited earlier. from urlparse import urlparse result = [] for name in urls: tup = urlparse(name) loc = tup[1] if not loc in result: if not loc in vis: result.append(loc) return result Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

26 some links will not work... Add an exception handler: def http_links(url): Given the URL for the web page, returns the list of all HTTP strings. import urllib try: print opening + url +... page = urllib.urlopen(url) except IOError: print opening + url + failed return [] Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

27 unparsing URLs Recall that we only store the server locations. To open a web page we also need to specify the protocol. We apply urlparse.urlunparse >>> from urlparse import urlunparse >>> urlunparse(( http, )) We must provide a 6-tuple as argument... Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

28 running the crawler $ python webcrawler.py crawling the web... Give URL : give maximal depth : 2 opening opening opening opening opening opening opening opening opening it takes a while.. total #locations : 538 Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

29 the function main() def main(): Prompts the user for a web page, and prints all URLs this page refers to. print crawling the web... page = raw_input( Give URL : ) depth = input( give maximal depth : ) locs = crawler(page, depth, []) print reachable locations :, locs print total #locations :, len(locs) if name == " main ": main() Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

30 code for the crawler def crawler(url, kst, locs): Returns the list locs updated with the list of locations reachable from the given url using kst steps. from urlparse import urlunparse links = http_links(url) newlocs = new_locations(links, locs) result = locs + newlocs if(kst == 0): return result else: for loc in newlocs: name = urlunparse(( http, loc, \,,, )) result = crawler(name, kst-1, result) return result Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

31 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

32 the HTMLParser module to parse html code In the standard Python distribution: >>> import HTMLParser >>> help(htmlparser) the class HTMLParser allows to override handlers of tags provides a feed method the feed method handles buffering Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

33 using HTMLParser from HTMLParser import HTMLParser from urllib import urlopen class OurHTMLParser(HTMLParser): def init (self): def handle_starttag(self, tag, attrs): def handle_endtag(self, tag): def main(): page = urlopen(url) pars = OurHTMLParser() while True: data = page.read(80) if data == : break pars.feed(data) pars.close() Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

34 tallying the tags using the class HTMLParser Gather basic statistics about a page: 1 what types of tags are used, 2 count number of occurrences for each tag. At end of each tag the tally is updated. Data structure for the tally: dictionary. keys: string with type of tag values: natural number counts #occurrences The tally is an object data attribute. Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

35 Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location 3 Web Crawlers making requests recursively incremental development, modular design of code 4 Parsing HTML the HTMLParser module overriding methods in HTMLParser Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

36 the class Tagtally from HTMLParser import HTMLParser from urllib import urlopen class Tagtally(HTMLParser): Makes a tally of ending tags. def init (self): Initializes the dictionary of tags. def handle_endtag(self, tag): Maintains a tally of the tags. def show_tally(self): Prints the tally to screen. Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

37 the function main() def main(): Opens a web page and parses it. url = print opening %s... % url page = urlopen(url) tags = Tagtally() while True: data = page.read(80) if data == : break tags.feed(data) tags.close() print the tally of tags : tags.show_tally() Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

38 constructor and ShowTally If overriding init, we must initialize parent class. def init (self): Initializes the dictionary of tags. HTMLParser. init (self) self.tally = {} def show_tally(self): Prints the tally to screen. for each in self.tally: print each, :, self.tally[each] Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

39 update of the tally updating a dictionary First check if there is already a tag... def handle_endtag(self, tag): Maintains a tally of the tags. if self.tally.has_key(tag): self.tally[tag] \ = self.tally[tag] + 1 else: self.tally.update({tag:1}) If no tag present, update the dictionary. Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

40 Summary + Exercises Python scripts can read web pages as files. 1 Write a script to download all.py files from jan/mcs507/main.html 2 Limit the search of the crawler so that it only opens pages within the same domain. For example, if we start at a location ending with edu, we only open pages with locations ending with edu. 3 Adjust webcrawler.py to search for a path between two locations. The user is prompted for two URLs. Crawling stops if a path has been found. Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

41 more exercises 4 Use HTMLParser to write a shorter version of the web crawler in the script webcrawler.py. The fourth homework is due on Friday 18 October, 9AM: exercises 3 and 6 of Lecture 13; exercises 2 and 4 of Lecture 14; exercises 1 and 3 of Lecture 15; exercises 2 and 5 of Lecture 16; exercises 3 and 4 of Lecture 17. Scientific Software (MCS 507 L-21) web clients and crawlers 14 October / 41

Web Clients and Crawlers

Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning files looking for strings between double quotes parsing URLs for the server location