CS 111: Program Design I Lecture 15: Modules, Pandas again. Robert H. Sloan & Richard Warner University of Illinois at Chicago March 8, 2018

Transcription

1 CS 111: Program Desig I Lecture 15: Modules, Padas agai Robert H. Sloa & Richard Warer Uiversity of Illiois at Chicago March 8, 2018

2 PYTHON STANDARD LIBRARY & BEYOND: MODULES

3 Extedig Pytho Every moder programmig laguage has way to exted basic fuctios of laguage with ew oes Pytho: importig a module module: Pytho file with ew capabilities defied i it Oe you import module, it's as if you typed it i: you get all fuctios, objects, variables defied it immediately

4 Pytho Stadard Library Pytho always comes with big set of modules List at Examples csv datetime math os radom urllib Read/write csv files Basic date & time types Math stuff (e.g., si(), cos()) E.g., list files i your operatig system radom umber geeratio Ope URLs, parse URLs

5 Usig Modules Use import <module_ame> to make module's fuctio's available Style: Put all import statemets at top of file After import module_ame, access its fuctios (ad variables, etc.) through module_ame.fuctio_ame

6 Module use cotiured If module_ame is log, ca abbreviate i import with as: import module_ame as m m.fuctio_ame There are several commo modules where this is uiversal practice E.g., import matplotlib.pyplot as plt Observe: A few modules come i hierarchical orgiizatio with dots

7 If you prefer to save typig (I mostly do ot do this) To access fuctio_ame without havig to type module_ame prefix, use: from module_ame import fuctio_ame

8 Dot otatio remark Pytho makes two (2.5?) differet uses of dot otatio methods, where as we've see, we call method as obj_ame.method_ame() fuctios i modules module_ame.fuctio_ame Ad a few hierarchically amed modules, like matplotlib

9 Commo but ot stadard matplotlib ad padas are ot part of set of modules that must come with every Pytho 3 matplotlib is very, very widely used, ad padas is widely used Both are amog the may modules that come with the Aacoda distributio of Pytho 3 For may that do't come automatically with Aacoda ca tell Aacoda Navigator to get for you: Eviromets tab (E.g., scrapy)

10 We all ca make modules for ourselves Modules used to group fuctios Both stadard library or matplotlib ad modules we write ourselves Very useful for clarity ad reuse as overall project sizes get larger Not so much eed for your ow modules i CS 111 Ay file edig i.py ca act as module

11 TOWARDS PANDAS

12 Recall from last time: Objects (Implicit i Chapter 2, Variables, & 9.5, Strig Methods of book, but ot explicit aywhere: So pay attetio!) Everythig i Pytho is a object Object combies data (e.g., umber, strig, list) with methods that ca act o that object

13 Why study padas 1. Data sciece is a fasciatig, red-hot subject, ad padas is oe of top two tools for doig it I particular ca aalyze much larger datasets much faster tha i Excel; we'll oly scratch surface 2. Opportuity to work with a large, complex module, ad lear our way aroud 3. Opportuity to work with a complex object with may methods ad lear them

14 PANDAS (FROM ANOTHER ANGLE)

15 Padas: What ad Why High performace way to work with large dataframes Dataframe: The 2-d data structure most familiar from Excel spreadsheets, ofte with a header row Padas built to play icely with matplotlib for plottig (ad icidetally NumPy ad Scikit- Lear for machie learig ad works for preprocessig for tesorflow for deep learig)

16 Why Padas ad ot Excel Excel ot desiged for workig with large datasets Large-ish: Chicago Crimes 2008 to mid-2016 file: 1.04 millio rows, 18 colums Ope file i Pytho: Istataeous padas.read_csv(): 8 secs (Sloa s 2013 laptop) Ope file i Excel: several miutes Just resize oe colum for better viewig: 5-30 sec

17 Why Padas ad ot Excel (1, cot.) Large: Chicago Crimes 2001 to preset file: 6.54 millio rows, 22 colums Ope file i Pytho: Istataeous padas.read_csv(): ~1 mi (Sloa s 2013 laptop) Ope file i Excel: Surely you gest!

18 Chicago, City of Data! Marvelous data portal Crime: Safety/Crimes-2001-to-preset-Dashboard/5cd6- ry5g

19 Why Padas ad ot Excel (2) Excel allows you to say/do/compute whatever is built ito Excel Pytho is geeral purpose programmig laguage: Ca say/do/compute aythig wat, ot limited to the fuctios Microsoft provides i Excel Geeky fie poit: Aythig that ca be doe with a computer. There are ucomputable problems (theory of computatio CS 301, maybe special lecture i this class if time at ed. Not really issue i data aalytics)

20 Padas data types Most importat: dataframe, which we are gettig from padas.read_csv() 2-d array, with colum headers Series: 1-d array, e.g., oe colum of a dataframe, secod most importat

21 Resource Pytho for Data Sciece Padas Cheat Sheet ytho-padas-cheat-sheet

22 Dataframe Idexig: Geeral ideas Sample 3 x 3 dataframe df A B C [row][col] iloc with (oly) umbers ("iteger locatio") To get the 1: df.iloc[0][0] loc with labels/colum headers, possibly mixed with umbers To get the 1: df.loc[0]['a']

23 Dataframe idexig: Colums frame[columame] returs series from colum with ame columame Givig the []s list of ames selects those colums i list order. E.g., scdb[["justicename","chief","docketid"]] Other idexig:.iloc,.loc (also others we wo't cover) Special case is that specifically a slice idex to whole frame will slice by rows for coveiece because it's a commo operatio, but icosistet with overall Padas sytax

24 Dataframe positioal slicig: iloc.iloc for 100% positioal idexig ad slicig with usual Pytho 0 to legth 1 umberig (stads for "iteger locatio") Argumets for both dimesios separated by comma [rows, cols]: frame.iloc[:3, :4] upper left 3 rows/4 cols frame.iloc[:, :3] all rows, first 3 cols Oe argumet: rows (possibly couterituitive) frame.iloc[3:6] secod 3 rows frame.iloc[41] 42 d row

25 Dataframe label idexig:.loc Use.loc to access by labels, or mix selectio list will put colums i list's order; selectio set i {}s origial dataframe order scdb.loc[3:6, {'docketid', 'chief', 'justicename'}] Rows 3 through 6 iclusive, colums i scdb's order scdb.loc[3:6, ['docketid', 'chief', 'justicename']] Rows 3 through 6 iclusive, colums i order ['docketid', 'chief', 'justicename'] Notice loc uses slices iclusive of both eds, ulike all rest of Pytho & Padas (!).loc with oly slices: error (e.g., foo.loc[3:6, 2:4])

26 Dataframe ad series methods head(): returs sub-dataframe (top rows) or for series, first etries tail(): same, bottom rows With o argumet they default to 5 rows; ca give positive iteger argumet for umber of rows cout(): For series, returs umber of values (excludig missig, NaN, etc.) For dataframe, returs series, with cout of each colum, labeled by colum

27 Dataframe ad series methods (cot.) abs, max, mea, media, mi, mode, sum All behave like cout, except will give errors if data types do't support the operatio E.g., a series of strigs does retur good aswer with.max() method (based o alphabetical order), but caot take.media()

28 plottig Both DataFrame ad Series have a plot() method (as do may other Padas types) Must have loaded Pytho's plottig module, because Padas is makig use of it: import matplotlib.pyplot as plt Default is Series makes a lie graph; DataFrame makes oe lie graph per colum, ad labels each lie by colum labels

29 100% Optioal: Aside for graph geeks Optioal for fu: To chage style of your plot: import matplotlib matplotlib.style.use('fivethirtyeight') # OR matplotlib.style.use('ggplot') # R style Out of the box, it's Matlab style, which some folks like a lot