CS 111: Program Design I Lecture 20: Web crawling, HTML, Copyright

Size: px

Start display at page:

Download "CS 111: Program Design I Lecture 20: Web crawling, HTML, Copyright"

Gavin Wiggins
6 years ago
Views:

1 CS 111: Program Desig I Lecture 20: Web crawlig, HTML, Copyright Robert H. Sloa & Richard Warer Uiversity of Illiois at Chicago November 8, 2016

2 Most importat thig If you have ot yet voted (ad are a US citize), GO VOTE!

3 WEB CRAWLER AGAIN

4 Two bits of useful Pytho sytax Do't eed either oe for the web crawler but will make it a bit prettier: break list as coditio for if or while

5 Break Causes immediate termiatio of the loop Typical usage: # We are iside a while or for if <some particular case>: break Use carefully! It ca make code very hard to read

6 Which of these will exit whe x is iitially 9? A while (x%2 == 1 ad x%3 == 0): x = 9 B while True: if (x%2 == 1 ad x%3 == 0): break x = 9 C. Both D. Neither E. I do t kow

7 Idiomatic Pytho: Empty list check If eed to check whether list is empty or oempty Pytho i Boolea cotext after if or while treats empty list as False, ay other list as True So if I wat to keep processig ls as log as it's oempty: while ls: <process ls, remove, apped, etc.>

8 Crawl all pages reachable from start List of pages to visit, iitially start while that list is ot empty: Take a page from the list Get its text # eed to lear how to do this remove that page from to-visit list, add it to already-visited lst Get all the liks i that page for each lik if ot already i visited list add it

9 def crawl(start, limit): to_visit = [start] visited = [ ] while to_visit: address = to_visit.pop() if address ot i visited: cotet = URL address #(Lect. 18) do_the_visit(page, s, etc.) visited.apped(address) if le(visited) >= limit: break

10 Crawl ad Scrape! Crawlers crawl for a purpose Our assigmet: Grab addresses Could just as easily grab all.jpg or all.mp3 or all files Or, a search egie: Or... Build a dictioary showig which words/phrases show up o which web pages

11 A LITTLE ABOUT WHAT'S ON A WEBPAGE: HTML

12 HTML is a markup laguage That has evolved Was simple c. 1996, ad pretty simple c Now, people wat to cotrol the look-ad-feel of the page dow to the pixels ad fots. Plus, we wat to grab iformatio more easily out of Web pages. Leadig to XML, the extesible Markup Laguage. XML allows for ew kids of markup laguages (that, say, explicitly idetify prices or stock ticker codes) for busiess purposes.

13 Three kids of HTML laguages Origial HTML: Simple, what the earliest browsers uderstood. CSS, Cascadig Style Sheets Ways of defiig more of the formattig istructios tha HTML allowed. XHTML: HTML re-defied i terms of XML. A little more complicated to use, but more stadardized, more flexible, more powerful. It's (probably?) future of where the Web is goig.

14 Markup meas addig tags A markup laguage adds tags to regular text to idetify its parts. Tag i HTML eclosed by <agle brackets>. Most tags have a startig tag ad a edig tag. A paragraph is idetified by a <p> at its start ad a </p> at its ed. A headig is idetified by a <h1> at its start ad a </h1> at its ed.

15 HTML is just text i a file Eter text ad tags i just a plai ole ordiary text file. Use a extesio of.html (.htm if your computer oly allows three characters) to idicate HTML. VS Code works just fie for editig ad savig HTML files. Just do't try to load them i termial!

16 Parts of a Web Page

17 Parts of a Web Page Start with a DOCTYPE It tells browsers what kid of laguage you're usig below. Whole documet is eclosed i <html> </html> tags. The headig is eclosed with <head> </head> That's where you put the <title> </title> The body is eclosed with <body> </body> That's where you put <h1> headigs ad <p> paragraphs.

18 Other thigs i there We're simplifyig these tags a bit. More ca go i the <head> Javascript (tos of it) Refereces to documets like cascadig style sheets The <body> tag ca also, e.g., set colors. <body bgcolor="#ffffff" text="#000000" lik="#3300cc" alik="#cc0033" vlik="#550088">

19 HTML is ot a programmig laguage Usig HTML is called codig ad it is about gettig your codes right. But it's ot about codig programs. HTML has o Loops IFs Variables Data types Ability to read ad write files Bottom lie: HTML does ot commuicate process!

20 Ca use programs to geerate HTML def make_page(): file=ope("geerated.html", "w") file.write("""<!doctype html> <html> <head> <title>the Simplest Possible Web Page</title> </head> <body> <h1>a Simple Headig</h1> <p>some simple text.</p> </body> </html>""") file.close()

21 How do Amazo ad Ebay work? By geeratig a catalog as Web pages. Nobody types up all those product HTML pages. Rather, the data is i a database, ad there are programs that put the data ito a database ad programs that use the database to geerate web pages i HTML.

22 OPEN ACCESS: COPYRIGHT AND ACCESS CONTROL

23 Web Crawlers ad Copyright Google, like other search egies, uses a automated program (called the Googlebot ) to cotiuously crawl across the Iteret, to locate ad aalyze available Web pages, ad to catalog those Web pages ito Google s searchable Web idex. As part of this process, Google makes ad aalyzes a copy of each Web page that it fids, ad stores the HTML code from those pages i a temporary repository called a cache. Field v. Google. Whe does makig the copy ifrige copyright?

24 Pytho Web Crawler Code import urllib.reuest as ur coectio= ur.urlope(" #Accesses the website #Issues about uauthorized access, CFAA cotet = coectio.read() #Copies the data ito the computer s memory. #Issues about copyright, copyright statute.

25 Copyright: A Budle of Rights The right to make copies ad distribute copies of the work. make a derivative work. publicly display the work publicly perform the work 17 U. S. C. 106.

26 Copyig Code Suppose you copy some Pytho code off a website, ad the use it i your homework. You a. Create a derivative work. b. Make a copy ad create a derivative work. c. Make a copy

27 How Is The Right Created? Copyright exists whe you create a origial work of authorship fixed i a tagible medium of expressio. A work is fixed i a tagible medium of expressio whe its embodimet i a copy or phoorecord, by ad uder the authority of the author, is sufficietly permaet or stable to permit its to be perceived, reproduced, or otherwise commuicated for a period of more tha trasitory duratio. 17 U. S. C. 101.

28 Why Have Copyright? To promote progress i the arts ad scieces. Assumptios: We wat eough progress i the arts ad scieces. We wo t have eough uless authors ca get paid for their works. They wo t make eough moey if people ca copy their works for free.

29 Copyright Law A Good Fit? Is this a good argumet? We wo t have eough works uless authors ca get paid for their works They wo t make eough moey if people ca copy their works for free. I geeral, is traditioal copyright a good fit for the digital/iteret world?

30 The Need for Cotrol Without Law Idividuals must share some geeral sese that most of the rules goverig the situatio are appropriate. Otherwise, the cost of eforcemet... becomes high eough that it is difficult, if ot impossible, to maitai predictability. Elior Ostrom, Istitutioal Diversity.

31 No-Legal Norms Norms ca ecapsulate a shared geeral sese of appropriate behavior. Thik of the family holiday dier. Web crawler orms cotrol access by search programs to olie iformatio: The orms create a way of opeig or closig the door.

32 Opeig ad Closig: robots.txt User-aget: * [Or bot ames] Disallow: / --Allows othig User-aget: * Disallow: [blak] --Allows all User-aget: * Disallow: [oe or more, each followed by a url]

33 Aother Way To Cotrol The Door <html> <head> <title>... </title> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> </head>

34 <META NAME= ROBOTS CONTENT=... > NOINDEX - prevets idexig. NOFOLLOW - prevets followig liks. NOARCHIVE - cached copy ot available i search results. NOSNIPPET - prevets cachig ad descriptio i search results. NONE - euivalet to "NOINDEX, NOFOLLOW".

35 Coordiatio Norms The use of robots.txt ad metatags ivolves coordiatio orms. This is worth a bit more detail.

36 A Coordiatio Norm A regularity: Web crawlers (a lot of them) obey istructios i robot.txt files ad metatags. Coditioal coformity: web crawlers coform i part because (ad oly as log as) they thik other web crawlers will coform. Elevators ad drivig o the right.

37 The Elevator Norm You eter a elevator with two people i it. Where do you stad? (FRONT = UP) a b c

38 The Nearest Neighbor Norm You maximize the distace from your earest eighbor (while maitaiig peripheral eye cotact with at least oe other perso). Three perso distributio

39 Copyright Law A Good Fit? Now we ca retur to this uestio. How well do the robot orms ad copyright law get alog? The way to aswer is to see how copyright law respods whe someoe violates the orm.

CS 111: Program Design I Lecture 20: Web crawling, HTML, Copyright

CS 111: Program Design I Lecture 20: Web crawling, HTML, Copyright CS 111: Program Desig I Lecture 20: Web crawlig, HTML, Copyright Robert H. Sloa & Richard Warer Uiversity of Illiois at Chicago November 8, 2016 WEB CRAWLER AGAIN Two bits of useful Pytho sytax Do't eed