Networked Programs. Getting Material from the Web! Building a Web Browser! (OK, a Very Primitive One )

Transcription

1 Networked Programs Getting Material from the Web! Building a Web Browser! (OK, a Very Primitive One ) 43

2 So far we ve dealt with files. We ve read data from them, but it s possible to write data to them too. Servers across the internet are similar to files, except that in addition to getting data from them you also send them requests which may result in data. You do this with sockets. If you write something to a socket, it is sent to the application at the other end of the socket. If you read from the socket, you are given the data that the other application has sent. 44

3 The programs on both ends of this socket need to communicate in a way both understand, so one program isn t waiting in vain for the other to send something. The protocol that the web uses for this is called HTTP: Hyper-Text Transport Protocol. There are many elements to this protocol. We ll be interested in only a few. 45

4 Client Server Internet 46

5 Internet HTML AJAX JavaScript CSS HTTP Request Response GET socket POST Python Templates Data Store memcache 47

6 Network Architecture... 48

7 Transport Control Protocol (TCP) Built on top of IP (Internet Protocol) Assumes IP might lose some data - stores and retransmits data if it seems to be lost Handles flow control using a transmit window Provides a nice reliable pipe Source: org/wiki/internet_protocol_suite 49

8 TCP Connections / Sockets In computer networking, an Internet socket or network socket is an endpoint of a bidirectional inter-process communication flow across an Internet Protocol-based computer network, such as the Internet. Process Internet Socket Process 50

9 TCP Port Numbers A port is an application-specific or process-specific software communications endpoint It allows multiple networked applications to coexist on the same server. There is a list of well-known TCP port numbers 51

10 Incoming 25 Login Web Server blah blah blah blah Personal Mail Box Clipart: Please connect me to the web server (port 80) on 52

11 Common TCP Ports 53

12 Sometimes we see the port number in the URL if the web server is running on a non-standard port. 54

13 Sockets in Python Python has built-in support for TCP Sockets import socket mysock = socket.socket(socket.af_inet, socket.sock_stream) mysock.connect( (' 80) ) Host Port 55

14 Application Protocol Since TCP (and Python) gives us a reliable socket, what do we want to do with the socket? What problem do we want to solve? Application Protocols Mail World Wide Web Source: org/wiki/internet_protocol_suite 57

15 HTTP - Hypertext Transport Protocol The dominant Application Layer Protocol on the Internet Invented for the Web - to Retrieve HTML, Images, Documents, etc Extended to be data in addition to documents - RSS, Web Services, etc.. Basic Concept - Make a Connection - Request a document - Retrieve the Document - Close the Connection 58

16 HTTP The HyperText Transport Protocol is the set of rules to allow browsers to retrieve web documents from servers over the Internet 59

17 What is a Protocol? A set of rules that all parties follow so we can predict each other s behavior And not bump into each other On two-way roads in USA, drive on the right-hand side of the road On two-way roads in the UK, drive on the left-hand side of the road 60

18 protocol host document 1:17-2:19 Robert Cailliau CERN 61

19 Getting Data From The Server Each time the user clicks on an anchor tag with an href= value to switch to a new page, the browser makes a connection to the web server and issues a GET request - to GET the content of the page at the specified URL The server returns the HTML document to the browser, which formats and displays the document to the user For example, let s take a look at the source of ai.isi.edu 62

20 Making an HTTP request Connect to the server like a hand shake Request a document (or the default document) GET GET GET 63

21 64

22 Browser 65

23 Web Server 80 Browser 66

24 Web Server 80 GET Browser 67

25 GET Web Server 80 <h1>the Second Page</h1> <p>if you like, you can switch back to the <a href="page1. htm">first Page</a>.</p> Browser 68

26 GET Web Server 80 <h1>the Second Page</h1> <p>if you like, you can switch back to the <a href="page1. htm">first Page</a>.</p> Browser 69

27 Internet Standards The standards for all of the Internet protocols (inner workings) are developed by an organization Internet Engineering Task Force (IETF) Standards are called RFCs - Request for Comments Source: 70

28 71

29 72

30 Hacking HTTP Web Server HTTP Request $ telnet 80 Trying Connected to Escape character is '^]'. GET HTTP/1.0 Browser <h1>the First Page</h1> <p>if you like, you can switch to the <a href=" Page</a>. </p> % telnet 80 HTTP Response Port 80 is the non-encrypted HTTP port GET 73 HTTP/1.0

31 $ telnet 80 Trying Connected to character is '^]'. GET HTTP/1.0 <h1>the First Page</h1> <p>if you like, you can switch to the <a href=" Page</a>.</p> Connection closed by foreign host. 74

32 More Elaborate Page % telnet 80 GET 75

33 76

34 A browser debugger reveals detail... Most browsers have a developer mode so you can watch it in action It can help explore the HTTP request-response cycle Some simple-looking pages involve lots of requests: HTML page(s) Image files CSS Style Sheets JavaScript files 77

35 Let s Write a Web Browser! A very, very simple one 78

36 An HTTP Request in Python import socket mysock = socket.socket(socket.af_inet, socket.sock_stream) mysock.connect((' 80)) mysock.send('get HTTP/1.0\n\n') while True: data = mysock.recv(512) if ( len(data) < 1 ) : break print data mysock.close() Let s try this out 79

37 HTTP/ OK Date: Sun, 14 Mar :52:41 GMT Server: Apache Last-Modified: Tue, 29 Dec :31:22 GMT ETag: "143c1b33-a7-4b395bea" Accept-Ranges: bytes Content-Length: 167 Connection: close Content-Type: text/plain But soft what light through yonder window breaks It is the east and Juliet is the sun Arise fair sun and kill the envious moon Who is already sick and pale with grief HTTP Header while True: data = mysock.recv(512) if ( len(data) < 1 ) : break print data HTTP Body 80

38 Making HTTP Easier With urllib 81

39 Using urllib in Python Since HTTP is so common, we have a library that does all the socket work for us and makes web pages look like a file import urllib fhand = urllib.urlopen(' for line in fhand: print line.strip() urllib1.py Let s try *that* out! 82

40 import urllib fhand = urllib.urlopen(' for line in fhand: print line.strip() But soft what light through yonder window breaks It is the east and Juliet is the sun Arise fair sun and kill the envious moon Who is already sick and pale with grief urllib1.py 83

41 Like a file... import urllib fhand = urllib.urlopen(' counts = dict() for line in fhand: words = line.split() for word in words: counts[word] = counts.get(word,0) + 1 print counts Run this code 84 urlwords.py

42 Reading Web Pages import urllib fhand = urllib.urlopen(' for line in fhand: print line.strip() <h1>the First Page</h1> <p> If you like, you can switch to the <a href=" Page</a>. </p> urllib2.py 85

43 Going from one page to another... import urllib fhand = urllib.urlopen(' for line in fhand: print line.strip() <h1>the First Page</h1> <p> If you like, you can switch to the <a href=" page2.htm">second Page</a>. </p> 86

44 Parsing HTML (a.k.a. Web Scraping) 87

45 What is Web Scraping? When a program or script pretends to be a browser and retrieves web pages, looks at those web pages, extracts information, and then looks at more web pages. Search engines scrape web pages - we call this spidering the web or web crawling

46 Google import urllib fhand = urllib.urlopen(' for line in fhand: print line.strip() 89

47 Why Scrape? Pull data - particularly social data - who links to who? Get your own data back out of some system that has no export capability Monitor a site for new information Spider the web to make a database for a search engine 90

48 Scraping Web Pages There is some controversy about web page scraping and some sites are a bit snippy about it. Google: facebook scraping block Republishing copyrighted information is not allowed Violating terms of service is not allowed 91

49 92

50 The Easy Way - Beautiful Soup You could do string searches the hard way Or use the free software called BeautifulSoup from www. crummy.com Place the BeautifulSoup.py file in the same folder as your Python code

51 import urllib from BeautifulSoup import * url = raw_input('enter - ') html = urllib.urlopen(url).read() soup = BeautifulSoup(html) # Retrieve a list of the anchor tags # Each tag is like a dictionary of HTML attributes tags = soup('a') for tag in tags: print tag.get('href', None) urllinks.py 94

52 html = urllib.urlopen(url).read() soup = BeautifulSoup(html) tags = soup('a') for tag in tags: print tag.get('href', None) <h1>the First Page</h1> <p>if you like, you can switch to the<a href=" >Second Page</a>.</p> python urllinks.py Enter

53 Summary The TCP/IP gives us pipes / sockets between applications We designed application protocols to make use of these pipes HyperText Transport Protocol (HTTP) is a simple yet powerful protocol Python has good support for sockets, HTTP, and HTML parsing 96