CS 111: Program Design I Lecture # 7: Web Crawler, Functions; Open Access

Transcription

1 CS 111: Program Desig I Lecture # 7: Web Crawler, Fuctios; Ope Access Robert H. Sloa & Richard Warer Uiversity of Illiois at Chicago September 13, 2016

2 Lab Hit/Remider word = "hi" word.upper() à "HI"

3 Questio Sice homeworks ofte due Suday eveig, would Amada office hours Friday 5 6 pm be useful to you? Would you atted? A. Yes B. No

4 Towards Crawlig the Web (MORE ABOUT) FUNCTIONS

5 Web crawler Oe log-term goal of course: build ad uderstad web crawler, program that will visit every page reachable from give start web page Key compoet of, e.g., search egie May pieces, somewhat complicated Need a orgaizig priciple: fuctios! Also eed to do thigs over ad over: iteratio Will retur to crawler from time to time

6 Fuctios: defiitio & use Ca do 2 thigs with fuctio: Defie it; Call it Defiitio: def f_ame(parameters): E.g., def strig_multiply(my_strig, um): Call (use) f_ame(parameters) E.g, strig_multiply("hi", 3) Rus fuctio f_ame o parameters

7 Note: Defiitio must have some ideted code after the def f_ame(): lie This is ot a legal fuctio defiitio: def othigess(): But def othigess(): retur is legal (though useless) fuctio defiitio

8 Iput parameters (1) Most fuctios have 1 or more iput parameters (though legal to create fuctio that takes o iput parameters). Example of a built-i fuctio (techically specifically a method fuctio) that takes zero iputs: Strig method upper(): word = "hi" word.upper() à "HI"

9 Retur values fuctio may or may ot retur a value If eed termiology, call fuctio that returs value fruitful fuctio; fuctio that does't o-fruitful fuctio If (ad oly if) fuctio returs value, legal to assig ame to (retur value of) fuctio call: x = fruitful_f_ame(iputs)

10 Example Say we wat to fid absolute value of a umber (say -3) There is built-i Pytho fuctio called abs that fids absolute value >>> x = abs(-3) >>> x 3

11 Most famous built-i o-fruitful fuctio prit() prit does't retur ay value. We do't use prit (or ay o-fruitful fuctio) for its retur value but for some other reaso

12 fuctios you write that retur somethig Must iclude a lie that begis retur

13 fuctio flow A fuctio's executio eds either whe 1. A retur statemet is executed, or 2. Last lie of code is executed whichever comes first

14 What is wrog with this fuctio? def triple(x): retur 3 * x prit("triple the iput is", 3*x) A. You should ever use a prit statemet i a fuctio B. You must calculate the value of 3*x before you retur it C. You should ot have statemets after the retur D. A fuctio ca retur strig types but ot a umber E. Nothig

15 Fuctio as iput-output box z abs z x triple 3 * x

16 Parameters (1) (actual) parameter >>> y = abs(-3) >>> y 3

17 Parameters (2) Parameters i the ()s i def statemet called formal parameters Formal parameters are (like) variables : they're the thig that chages iside the fuctio Fuctios have 0 or more parameters Value(s) i fuctio call: actual parameter(s) Same umber as formal parameters Could be variables ad/or literal values

18 Formal vs. actual parameters def triple(x): retur 3 * x formal parameter >>> = 17 >>> triple() 51 >>> 4 + triple(20) 64

19 Formal vs. actual parameters def triple(x): retur 3 * x formal parameter (which happes to be x i this example) >>> = 17 >>> triple() 51 >>> 4 + triple(20) 64

20 Formal vs. actual parameters def triple(x): retur 3 * x formal parameter >>> = 17 >>> triple() 51 >>> 4 + triple(20) 64 actual parameter

21 Parameters ad fuctio executio Like fuctios i high-school Algebra 2: At time fuctio called, formal parameter takes o value of actual parameter Algebra 2: if f(x) = 3x, the 4 + f(20) = 64 Pytho: ad implicitly at least, the formal parameter x took o the value triple(x) formal x boud to 3 for legth of ru of triple()

22 I eve more detail def triple(x): retur 3 * x formal parameter >>> 4 + triple(20)

23 I eve more detail def triple(x): retur 3 * x formal parameter x boud to 20 >>> 4 + triple(20) At poit where triple(20) is called, value 20 is assiged to triple's iteral x parameter, multiplicatio is doe gettig value 60, umber 60 is retured (ad triple is doe), ad iterpreter (commad lie) adds 4 ad 60

24 How do I kow which fuctios exist? Pytho documetatio

25 What should I do to make z the smaller of itegers x ad y? x = 10 y = 3 A. z = max(x,y) B. z = mi(x,y) C. z = memoryview(x) D. z = memoryview(y)

26 Additioal built-i fuctios from modules Useful for certai kids of thigs, e.g., math, iteret, makig graphs, are available i modules that must be imported before they ca be used Will discuss a few later as eeded

27 fuctios for strigs Strigs examples of "built-i class" ad Strig class comes with some built-i fuctios (ad class fuctios also called methods). (A little more much later) Same as other built-i fuctios except callig sytax is.f_ame st = "Oly 8 Justices?!" st.upper() à "ONLY 8 JUSTICES?!" st.fid("8") à 5

28 Some Notes o Programmig Style Remember: Code eeds to be uderstood by both computers ad people Should try to make code as easy to read as possible Pro tip: This will make it easier for our TAs ad I to give you partial credit o assigmets, exams, etc

29 Good Pytho Programmig Style Meaigful variable & fuctio ames Geerally startig with lower-case letter Blak lie betwee fuctios Use of docstrig to briefly describe iputoutput behavior of fuctio Ad, of course, be very careful with idetatio

30 Why fuctios istead of e.g., cut & paste same code Code legth (repeatig same thig) Bugs: If there's bug or error, replicated i multiple places If we wat to chage somethig, eed to chage it i every copy

31 Aalogy Imagie you are writig cake cookbook with 17 recipes that use buttercream frostig Do you put the buttercream istructios i each of the 17 recipes?

32 Eve if a fuctio used oly oce Helps modularize code ad make it easier for humas to read ad uderstad

33 fuctios as aid to problem solvig Problem solvig strategy: Describe how to solve your problem assumig wheever you like that you have a fuctio to do some of the work Figure out what iput-output behavior it eeds The write those fuctios This is called fuctioal decompositio Ted to use it o slightly larger problems tha we have worked o so far; will revisit

34 Bous material: retur >1 value (Bous = wo't be o test, ot useful for curret homework AFAIK, might be oe way but ot oly way to do future homework) To retur > 1 value: Wrap retur value i ()s ad assig to same umber of values i ()s Oe built-i fuctio (that I kow): divmod (returs 2 values). Example: (, m) = divmod(17, 3) à 5 (i.e., 17 // 3) m à 2 (i.e., 17 % 3)

35 A Plea Please start homeworks due Suday ight way early We would love to help you Thursday afteroo We will try to check Piazza regularly from 6:30 pm Friday util 11 pm Suday, but wo't be early as reliable as Thursday or Friday at 2:15 pm

36 OPEN ACCESS: COMPUTER FRAUD AND ABUSE ACT

37 Ope Access Ope access typically refers to olie items that are free of restrictios o access ad certai restrictios o use like copyrights, trademarks, ad patets. Digital techologies ca icrease ope access to iformatio. Web crawlers are a good example.

38 Easy to Get Data These four lies yield 26 pages of data: import reuests r = reuests.get(' page = r.text prit (page)

39 Closig Off Access Digital techologies also eable profitorieted firms to extract value from resources previously held i commo ad to establish property rights [ad so cotrol access]. Hess ad Ostrom, Ideas, Artifacts, Ad Facilities. Facebook v. Power Vetures illustrates the ope access/closed access coflict.

40 A Commo Patter The patter begis with iformatio distributed over the Iteret. Facebook v. Power Vetures: the distributed iformatio is social etworkig sites. There are a lot: Facebook, QQ, WeChat, QZoe, Tumblr, Istagram, Twitter, Baidu Tieba, Skype, Viber, Sia Weibo ad so o. Suppose you have accouts o several of those.

41 More of the Patter You would like a way to post o all of them from just oe site. Solvig that problem was the idea behid Power Veture. You could post o all your sites at oce. Power Vetures did what may do: collect scattered iformatio make it coveietly accessible.

42 The Fial Part of the Patter The fial part: the perso or etity i possessio of the data objects to the access ad claims it is illegal. That is what Facebook did. The court agreed with them ad held that Power Vetures access was illegal. Do you thik Power Vetures should be able to access Facebook?

43 Computer Fraud ad Abuse Act The CFAA prohibits acts of computer trespass by those who are ot authorized users. Crimial ad civil liability for whoever (a) itetioally accesses a computer (b) without authorizatio.., ad (c) thereby obtais... iformatio from ay... computer. 18 U.S.C. 1030(a)(2)(C).

44 Uauthorized? Power Vetures itetioally accessed Facebook s computers, ad It obtaied iformatio by doig so. So: was the access uauthorized? Power Vetures did have permissio from Facebook s users to access their accouts. The court still aswers, ot authorized.

45 The Courts Aalogy Suppose that a perso wats to borrow a fried s jewelry that is held i a safe deposit box at a bak. The fried gives permissio for the perso to access the safe deposit box ad leds him a key. The perso decides to visit the bak while carryig a shotgu. The bak ejects the perso from its premises ad bas his reetry. The gu-totig jewelry borrower could ot the reeter the bak, claimig that access to the safe deposit box gave him authority to stride about the bak s property while armed.