CS 111: Program Design I Lecture 14: Encodings & Files concluded; Pandas, Modules, legal data analytics

Transcription

1 CS 111: Program Desig I Lecture 14: Ecodigs & Files cocluded; Padas, Modules, legal data aalytics Robert H. Sloa & Richard Warer Uiversity of Illiois at Chicago October 16, 2018

2 Recall: ASCII Ecodig characters i bytes, 1960s ASCII: Use 1 byte to ecode 95 pritig characters The oes o every computer keyboard to this day Pretty much all ecodigs agree with ASCII o those 95 characters ASCII also has some opritig characters like ewlie ad tab

3 But What about Vigière, Beyocé Kowles, ad Reée Zellweger? А что насчет Aрабского?

4 Ecodig more characters Uicode: over 128,000 characters coverig 135 moder ad historical scripts, ad symbols 2 bytes ot eough for all of it 2 16 is "oly" 65,536 Various character ecodigs for all or a subset Pytho supports Uicode

5 Ecodigs of files i Uicode Two fairly commo ecodigs of Uicode. i.e., bit patter character correspodeces UTF-8 (Pytho's default guess): multi-byte, variable umber of bytes (1 4 bytes/character) ISO : 1 byte, so ca ecode limited # of characters; i fact ecodes Lati-1 ASCII plus most acceted characters plus commo symbols: 195 characters covers all of Eglish ad several commo Europea laguages. (4% of o-ascii web?) ½¼ ÀÁÂÃÄ Æ Ç ÈÉÊË Ø àáâãäå æ ç èé N.B. Oly a issue for files

6 Which ecodig? Fairly uusual to eed to give Pytho optioal ecodig argumet whe opeig a file Pytho defaults to UTF-8, which agrees with ASCII o ASCII characters, ad most text files are ASCII Ad very may of the rest are UTF-8 But some are ISO For websites, about 3 or 4 percet

7 Supreme Court Database that we are workig with is ISO Almost etirely ASCII characters, but uses symbol i places

8 FILES CONTINUED

9 Ope whe uusual characters i file Ope has a optioal third argumet, specifyig a character ecodig Irrelevat most of the time But you may eed it if you are workig with Spaish, Italia, Albaia, Tagalog, etc. Or legal materials usig sectio symbol (But probably ot Arabic, Hebrew, Madari, Russia, etc.) f = ope('scdb_2018_01_justicecetered_citatio.csv', 'r', ecodig='iso ')

10 Ope trivia If o mode is give, ope defaults to read ('r')

11 (Text) File readig, a little more slowly Recall text file = seuece of lies Lie = seuece of characters up to ad icludig the special ewlie character \ (Special case: probably last set of characters at ed of file will work okay eve if text file does't ed with ewlie as it should.) (How could we fid out?)

12 Speakig of text afile.txt: f = ope("afile.txt", "r") lie = f.readlie() 1234 Ca I have a little more? I love you! ABCD Ca I brig my fried to tea? What is le(lie)? A. 0 B. 1 C. 4 D. 5 E. 6

13 Ca iterate over text file referece (ot i book) fileref = ope('afile.txt', 'r') for lie i fileref: # process each lie process lie as we wish i this block rest of program fileref.close() Perhaps easiest way to read text file, all other thigs beig eual

14 Structured text files: CSV I 2018, ofte wat to commuicate betwee all sorts of differet electroic tools CSV (comma-separated values) is format used by Excel, ad very commo for exchagig large collectios of data E.g., SCDB, City of Chicago Data Portal Pytho has a csv module ad it has csv.writer() ad csv.reader() fuctios that could help you We wo t cover those i this course

15 Remider: Files ad programmig with them You eed your executio eviromet, i.e., cosole, i.e., lower right pael of Spyder, to be workig i directory you with file you wat to ope Workig directory butto upper right corer

16 Less likely to make mistake with with with ope('afile.txt', 'r') as fileref: for lie i fileref: # process each lie process lie as we wish i this block rest of program # No eed to remember to close!

17 Also: read fileref.read() returs whole gosh dar text file's cotets as sigle strig fileref.read(umber) reads (the ext) umber characters (ewlie couts) as strig read advaces place i file we're readig from After you've read all the way to ed, read will retur empty strig ''

18 Readig a text file: more methods fileref.readlie(): reads ext lie of file ad returs it as strig (up to ad icludig ewlie) fileref.readlies() returs etire file, as a list of oe-lie strigs

19 A few remarks o the big project 1. Get started! 2.

20 Modules: Oe more thig We all ca make modules for ourselves Modules used to group fuctios Both stadard library or matplotlib ad modules we write ourselves Very useful for clarity ad reuse as overall project sizes get larger Not so much eed for your ow modules i CS 111 Ay file edig i.py ca act as module

21 OBJECTS AND DOT NOTATION

22 Objects (Implicit i Chapter 2, Variables & Expressios, 3.2, Lists basics, & 7.3 Strig methods of Zybook, but ot explicit aywhere: So pay attetio!) Everythig i Pytho is a object Object combies data (e.g., umber, strig, list) with methods that ca act o that object

23 Methods Methods: like (or actually special case of) fuctios but ot globally accessible Caot call method just by givig its ame, the way we call prit(), ope(), abs(), type(), rage(), etc. Method: fuctio that ca oly be accessed through a object Usig dot otatio

24 Dot otatio To call method, use dot otatio: object_ame.method() Strig example: I [1]: test = 'This is my test strig' I [2]: test.upper() Out[2]: 'THIS IS MY TEST STRING'

25 If o is object of type havig method do_it where do_it eeds a iput i additio to o, ad x is defied, what is the proper way to call do_it o iput x? A. do_it(x) B. do_it(o, x) C. o.do_it(x) D. o.do_it(o, x)

26 methods cotiued I [3]: test.fid('my') Out[3]: 8 I [4]: 42.upper() Sytax Error: ivalid sytax I [5]: upper(test) barf

27 Methods deped o type of object scdb.head() prits out 5 rows because head() is a method of objects of type Padas dataframe, which is the type of the scdb object 'test strig'.head() triggers error because head is ot a method of strigs

28 Methods' importace Uderstadig key data types depeds o uderstadig their methods We saw may methods for strigs We have used the apped method for lists, ad will come back to more list methods file referece methods write(), read(), readlie(), readlies() Padas dataframe methods head(), tail(), etc.

29 Whe you get to CS 341 & 342 Or if you kow Java or C++ ow methods are a Object Orieted (OO) cocept I our CS 111 We do eed to kow the basics of dot otatio ad methods We will otherwise be igorig OO, ad takig primarily a procedural approach

30 PANDAS (FROM ANOTHER ANGLE)

31 Padas: What ad Why High performace way to work with large dataframes Dataframe: The 2-d data structure most familiar from Excel spreadsheets, ofte with a header row Padas built to play icely with matplotlib for plottig (ad icidetally NumPy ad Scikit-Lear for machie learig ad works for preprocessig for tesorflow for deep learig)

32 Why Padas ad ot Excel Excel ot desiged for workig with large datasets Large-ish: Previous Chicago Crimes 2008 to mid-2016 file: 1.04 millio rows, 18 colums Ope file i Pytho: Istataeous padas.read_csv(): 8 secs (Sloa s 2013 laptop) Ope file i Excel: several miutes Just resize oe colum for better viewig: 5-30 sec

33 Why Padas ad ot Excel (reaso 1, cot.) Large: Chicago Crimes 2001 to preset file: 7 10 millio rows, ~22 colums Ope file i Pytho: Istataeous padas.read_csv(): ~1 mi (Sloa s 2013 laptop) Ope file i Excel: Surely you gest!

34 Chicago, City of Data! Marvelous data portal Crime:

35 Why Padas ad ot Excel (2) Excel allows you to say/do/compute oly fuctios built ito Excel Pytho is geeral purpose programmig laguage: Ca say/do/compute aythig wat, ot limited to the fuctios Microsoft provides i Excel Geeky fie poit: Aythig that ca be doe with a computer. There are ucomputable problems (theory of computatio CS 301, maybe special lecture i this class if time at ed. Not really issue i data aalytics)

36 Padas data types Most importat: dataframe, which we are gettig from padas.read_csv() 2-d array, with colum headers Series: 1-d array, e.g., oe colum of a dataframe, secod most importat

37 Resource Pytho for Data Sciece Padas Cheat Sheet

38 Dataframe Idexig: Geeral idea overview Sample 3 x 3 dataframe df: A B C Idea is [row][col].iloc with (oly) umbers ("iteger locatio") To get the (red) 1: df.iloc[0][0].loc with labels/colum headers, possibly mixed with umbers To get the 1: df.loc[0]['a']

39 Dataframe idexig: Colums frame[columame] returs series from colum with ame columame Givig the []s list of ames selects those colums i list's order. E.g., scdb[["justicename","chief","docketid"]] Other idexig:.iloc,.loc (also others we wo't cover) Special case: specifically a slice idex to whole frame will slice by rows for coveiece because it's a very commo operatio, but icosistet with overall Padas sytax