Network Programming in Python. What is Web Scraping? Server GET HTML

Size: px

Start display at page:

Download "Network Programming in Python. What is Web Scraping? Server GET HTML"

Blanche Blake
6 years ago
Views:

Network Programming in Python Charles Severance www.dr-chuck.

Attribution 3.0 License. http://creativecommons.org/licenses/by/3.0/.

When a program or script pretends to be a browser and retrieves web pages, looks at those web pages,

1 Network Programming in Python Charles Severance Unless otherwise noted, the content of this course material is licensed under a Creative Commons Attribution 3.0 License. Copyright 2009, Charles Severance What is Web Scraping? When a program or script pretends to be a browser and retrieves web pages, looks at those web pages, extracts information and then looks at more web pages. Search engines scrape web pages - we call this spidering the web or web crawling GET HTML GET Server HTML

2 Why Scrape? Pull data - particularly social data - who links to who? Get your own data back out of some system that has no export capability Monitor a site for new information Spider the web to make a database for a search engine Scraping Web Pages There is some controversy about web page scraping and some sites are a bit snippy about it. Google: facebook scraping block Republishing copyrighted information is not allowed Violating terms of service is not allowed HTML and HTTP in Python Using urllib

Using urllib to retrieve web pages http://docs.python.org/lib/module-urllib.html You get the entire web page when you do f.

3 Using urllib to retrieve web pages You get the entire web page when you do f.read() - lines are separated by a newline character You get the entire web page when you do f.read() - lines are separated by a newline character We can split the contents into lines using the split() function

4 Splitting the contents on the newline character gives use a nice list where each entry is a single line We can easily write a for loop to look through the lines >>> print len(contents) >>> lines = contents.split("") >>> print len(lines) 2244 >>> print lines[3] <style type="text/css"> >>> A Simple Web Browser returl.py for ln in lines: # Do something for each line import urllib url = raw_input("enter a URL: ") print "Retrieving:", url contents = f.read() print "Retrieved",len(contents),"characters" f.close() lines = contents.split(''); print "Retrieved",len(lines),"lines" python returl.py Enter a URL: Retrieving: Retrieved characters Retrieved 2401 lines python returl.py Enter a URL: Retrieving: Retrieved 6769 characters Retrieved 175 lines

python returl.py Enter a URL: www.dr-chuck.com Retrieving: www.dr-chuck.com Traceback (most recent call last): File "returl.py", line 6, in <module> 2.6/lib/python2.6/urllib.

5 python returl.py Enter a URL: Retrieving: Traceback (most recent call last): File "returl.py", line 6, in <module> 2.6/lib/python2.6/urllib.py", line 87, in urlopen 2.6/lib/python2.6/urllib.py", line 203, in open 2.6/lib/python2.6/urllib.py", line 461, in open_file 2.6/lib/python2.6/urllib.py", line 475, in open_local_file IOError: [Errno 2] No such file or directory: ' Parsing HTML Counting Anchor Tags We want to look through a web page and see how many lines have the string <a We don t care if there is more than one per line - we are looking for a rough number

<body> <div id="header"> <h1><a href="index.htm">appenginelearn</a></h1> <ul> <li><a href="/ezlaunch.htm">ez-launch</a></li> <li><a href=http://oreilly.

6 <body> <div id="header"> <h1><a href="index.htm">appenginelearn</a></h1> <ul> <li><a href="/ezlaunch.htm">ez-launch</a></li> <li><a href= target=_new> Book</a></li> <li><a href= target=_new> Author</a></li> <li><a href= <li><a href= target=_new> App Engine</a></li> </ul> </div> import urllib url = raw_input("enter a URL: ") print "Retrieving:", url contents = f.read() print "Retrieved",len(contents),"characters" f.close() lines = contents.split(''); print "Retrieved",len(lines),"lines" count = 0 for line in lines: if line.find("<a") >= 0 : count = count + 1 print count, "lines with <a tag" You get the entire web page when you do f.read() - lines are separated by a newline character We can split the contents into lines using the split() function Splitting the contents on the newline character gives use a nice list where each entry is a single line We can easily write a for loop to look through the lines >>> print len(contents) >>> lines = contents.split("") >>> print len(lines) 2244 >>> print lines[3] <style type="text/css"> >>> for line in lines: # Do something for each line

7 import urllib url = raw_input("enter a URL: ") print "Retrieving:", url contents = f.read() print "Retrieved",len(contents),"characters" f.close() lines = contents.split(''); print "Retrieved",len(lines),"lines" count = 0 for line in lines: if line.find("<a") >= 0 : count = count + 1 print count, "lines with <a tag" python hrefs.py Enter a URL: Retrieving: Retrieved characters Retrieved 2405 lines 63 lines with <a tag python hrefs.py Enter a URL: Retrieving: Retrieved 6769 characters Retrieved 175 lines 33 lines with <a tag Summary Python can easily retrieve data from the web and use its powerful string parsing capabilities to sift through the information and make sense of the information We can build a simple directed web-spider for our own purposes Make sure that we do not violate the terms and conditions of a web seit and make sure not to use copyrighted material improperly

Networked Programs. Getting Material from the Web! Building a Web Browser! (OK, a Very Primitive One )

Networked Programs. Getting Material from the Web! Building a Web Browser! (OK, a Very Primitive One ) Networked Programs Getting Material from the Web! Building a Web Browser! (OK, a Very Primitive One ) 43 So far we ve dealt with files. We ve read data from them, but it s possible to write data to them