CS 111: Program Design I Lecture 20: Web crawling, HTML, Copyright

Size: px

Start display at page:

Download "CS 111: Program Design I Lecture 20: Web crawling, HTML, Copyright"

Bathsheba Beatrix Gregory
6 years ago
Views:

1 CS 111: Program Desig I Lecture 20: Web crawlig, HTML, Copyright Robert H. Sloa & Richard Warer Uiversity of Illiois at Chicago November 8, 2016

2 WEB CRAWLER AGAIN

3 Two bits of useful Pytho sytax Do't eed either oe for the web crawler but will make it a bit prettier: break list as coditio for if or while

4 Break Causes immediate termiatio of iermost eclosig while or for loop Typical usage: # We are iside a while or for if <some particular case>: break Use carefully! Ca make code very hard to read

5 Example for i rage(2, 10): for x i rage(2, ): if % x == 0: prit(, 'euals', x, '*', //x) break else: # loop fell through without fidig factor prit(, 'is a prime umber')

6 Which of these will exit whe x is iitially 9? A while (x%2 == 1 ad x%3 == 0): x = 9 B while True: if (x%2 == 1 ad x%3 == 0): break x = 9 C. Both D. Neither E. I do t kow

7 Cotiue Cotiue cotiues with ext iteratio of loop istead of fiishig curret iteratio

8 for um i rage(2, 6): if um % 2 == 0: prit("foud a eve umber", um) cotiue prit("foud a umber", um) Foud a eve umber 2 Foud a umber 3 Foud a eve umber 4 Foud a umber 5

9 Idiomatic Pytho: Empty list check If eed to check whether list is empty or oempty Pytho i Boolea cotext after if or while treats empty list as False, ay other list as True So if I wat to keep processig ls as log as it's oempty: while ls: <process ls, remove, apped, etc.>

10 I.e., Idiomatic test for truth Pythoistas do this NOT this if item_ls: stuff if le(item_ls)!=0: stuff Ad defiitely ot if item_ls!= []: stuff

11 Crawl all pages reachable from start List of pages to visit, iitially start while that list is ot empty: Take a page from the list Get its text # eed to lear how to do this remove that page from to-visit list, add it to already-visited list Get all the liks i that page for each lik if ot already i visited list add it

12 def crawl(start, limit): to_visit = [start] visited = [ ] while to_visit: address = to_visit.pop() if address ot i visited: cotet = URL address #(Lect. 19) do_the_visit(page, s, etc.) visited.apped(address) if le(visited) >= limit: break

13 Crawl ad Scrape! Crawlers crawl for a purpose Our assigmet: Grab addresses Could just as easily grab all.jpg or all.mp3 or all files Or, a search egie: Or... Build a dictioary showig which words/phrases show up o which web pages

14 A LITTLE ABOUT WHAT'S ON A WEBPAGE: HTML

15 HTML is a markup laguage That has evolved Was simple c. 1996, ad pretty simple c Now, people wat to cotrol the look-ad-feel of the page dow to the pixels ad fots. Plus, we wat to grab iformatio more easily out of Web pages. Leadig to XML, the extesible Markup Laguage. XML allows for ew kids of markup laguages (that, say, explicitly idetify prices or stock ticker codes) for busiess purposes.

16 Three kids of HTML laguages Origial HTML: Simple, what the earliest browsers uderstood. CSS, Cascadig Style Sheets Ways of defiig more of the formattig istructios tha HTML allowed. XHTML: HTML re-defied i terms of XML. A little more complicated to use, but more stadardized, more flexible, more powerful. It's (probably?) future of where the Web is goig.

17 Markup meas addig tags A markup laguage adds tags to regular text to idetify its parts Tag i HTML eclosed by <agle brackets> Most tags have startig tag ad edig tag A paragraph is idetified by a <p> at its start ad a </p> at its ed A headig is idetified by a <h1> at its start ad a </h1> at its ed

18 HTML is just text i a file Eter text ad tags i just plai ole ordiary text file. Use extesio.html (.htm if your computer oly allows three characters) to idicate HTML. Ay text or code editor (e.g., WordPad, TextEdit, VS Code) works just fie for editig ad savig HTML files.

19 Parts of a Web Page

20 Parts of a Web Page Start with a DOCTYPE It tells browsers what kid of laguage you're usig below. Whole documet is eclosed i <html> </html> tags. The headig is eclosed with <head> </head> That's where you put the <title> </title> The body is eclosed with <body> </body> That's where you put <h1> headigs ad <p> paragraphs.

21 Other thigs i there We're simplifyig these tags a bit. More ca go i the <head> Javascript (tos of it) Refereces to documets like cascadig style sheets The <body> tag ca also, e.g., set colors. <body bgcolor="#ffffff" text="#000000" lik="#3300cc" alik="#cc0033" vlik="#550088">

22 HTML is ot a programmig laguage Usig HTML is called codig ad it is about gettig your codes right. But it's ot about codig programs. HTML has o Loops IFs Variables Data types Ability to read ad write files Bottom lie: HTML does ot commuicate process!

23 Ca use programs to geerate HTML def make_page(): file=ope("geerated.html", "w") file.write("""<!doctype html> <html> <head> <title>the Simplest Possible Web Page</title> </head> <body> <h1>a Simple Headig</h1> <p>some simple text.</p> </body> </html>""") file.close()

24 How do Amazo (ad Alibaba) work? By geeratig a catalog as Web pages. Nobody types up all those product HTML pages. Rather, the data is i a database, ad there are programs that put the data ito a database ad programs that use the database to geerate web pages i HTML.

25 OPEN ACCESS: COPYRIGHT AND ACCESS CONTROL

26 Web Crawlers ad Copyright Google, like other search egies, uses a automated program (called the Googlebot ) to cotiuously crawl across the Iteret, to locate ad aalyze available Web pages, ad to catalog those Web pages ito Google s searchable Web idex. As part of this process, Google makes ad aalyzes a copy of each Web page that it fids, ad stores the HTML code from those pages i a temporary repository called a cache. Field v. Google. Whe does makig the copy ifrige copyright?

27 Pytho Web Crawler Code import urllib.reuest as ur coectio= ur.urlope(" #Accesses the website #Issues about uauthorized access, CFAA cotet = coectio.read() #Copies the data ito the computer s memory. #Issues about copyright, copyright statute.

28 Copyright: A Budle of Rights The right to make copies ad distribute copies of the work. make a derivative work. publicly display the work publicly perform the work 17 U. S. C. 106.

29 How Is The Right Created? Copyright exists whe you create a origial work of authorship fixed i a tagible medium of expressio. A work is fixed i a tagible medium of expressio whe its embodimet i a copy or phoorecord, by ad uder the authority of the author, is sufficietly permaet or stable to permit its to be perceived, reproduced, or otherwise commuicated for a period of more tha trasitory duratio. 17 U. S. C. 101.

30 Why Have Copyright? To promote progress i the arts ad scieces. Assumptios: We wat eough progress i the arts ad scieces. We wo t have eough uless authors ca get paid for their works. They wo t make eough moey if people ca copy their works for free.

31 Copyig Code Suppose you copy some Pytho code off a website, ad the use it i your homework. You a. Create a derivative work. b. Make a copy ad create a derivative work. c. Make a copy

32 Copyig Code Suppose you copy some Pytho code off a website, ad the use it i your homework. You a. Create a derivative work. b. Make a copy ad create a derivative work. c. Make a copy

CS 111: Program Design I Lecture 20: Web crawling, HTML, Copyright

CS 111: Program Design I Lecture 20: Web crawling, HTML, Copyright CS 111: Program Desig I Lecture 20: Web crawlig, HTML, Copyright Robert H. Sloa & Richard Warer Uiversity of Illiois at Chicago November 8, 2016 Most importat thig If you have ot yet voted (ad are a US