List Comprehensions, Data Analysis & Sorting

Transcription

1 List Comprehensions, Data Analysis & Sorting From last time: remember our functions using list comprehensions? def linesfromfile(filename): '''Returns a list of all lines in the given file. In each line, the terminating newline has been removed.''' with open(filename, 'r') as inputfile: # open the file thelines = [line.strip() for line in inputfile] return thelines CS111 Computer Programming Department of Computer Science Wellesley College def tuplesfromcsvfile(fname): '''Returns a list of tuples, one for each line in fname. ' tuplessofar = [tuple(line.split(', ) for line in linesfromfile(fname)] return tuplessofar 12-2 linesfromfile + tuplesfromcsvfile = tuplesfromfile def linesfromfile(filename): '''Returns a list of all lines in the given file. In each line, the terminating newline has been removed.''' with open(filename, 'r') as inputfile: # open the file thelines = [line.strip() for line in inputfile] return thelines + From last time: allstudents = tuplesfromfile('cs111spring16.csv') [ def tuplesfromcsvfile(fname): '''Returns a list of tuples, one for each line in fname. ' tuplessofar = [tuple(line.split(', ) for line in linesfromfile(fname)] return tuplessofar = def tuplesfromfile(filename): '' Merge linesfromfile and tuplesfromcsvfile into one''' with open(filename, 'r') as inputfile: thetuples = [tuple(line.strip().split(',')) for line in inputfile] return thetuples 12-3 ] In [33]: len(allstudents) Out[33]: 70 In [35]: allstudents[47] Out[35]:('Lia Wang', 'lwang3', '2016', 'Stone Hall', '3 ) In [36]: allstudents[47][0] Out[36]: 'Lia Wang' 12-4

2 for loop mapping pattern vs. list comprehension for loop mapping pattern names = [] names.append(stu[0]) # names contains all names equivalent list comprehension # A list of all names List Comprehensions: Mapping Examples Explain in English before trying them in the notebook. fs = [] fs.append(s[0].split()[0]) # fs contains all first names # A list of all first names [s[3] for s in allstudents] [s[0].split()[-1] for s in allstudents] [s[0].split()[-1] + ' ' + s[2] for s in allstudents] ans = [] for var in somelist: ans.append(expusingvar) # ans is list of results # A list of all results [expusingvar for var in somelist] for loop filtering pattern seniors = [] if s[2] == '2016': seniors.append(s) # seniors has all seniors tows = [] if s[3].startswith('tow'): tows.append(s) # tows has all entries for # students living in Tower ans = [] for var in somelist: if predicateusingvar: ans.append(var) # ans is all elements for # which predicate is true equivalent list comprehension # List of all senior entries # A list of all entries for # students in Tower # A list of all elements for # which predicate is true [var for var in somelist if predicateusingvar] List Comprehensions: Filtering Examples Explain them in English before trying them in the notebook. len([s for s in allstudents if s[2] == '2019']) [s for s in allstudents if s[2] == '2019' and s[3].startswith('tow')] [s for s in allstudents if s[0].split()[0][-1] == 'y']

3 Improving readability with abstractions Define some helpful abstractions: def getcolumn(index, table): return [row[index] for row in table] def firstname(name): return name.split()[0] def lastname(name): return name.split()[-1] What lists are described by these expressions? getcolumn(3, allstudents) List comprehensions with both mapping and filtering Explain in English before trying them in the notebook [s[0] for s in allstudents if s[4] == '2' and s[3].startswith('s')] [firstname(s[0]) for s in allstudents if s[2] == '2016'] [lastname(name) for name in getcolumn(0, allstudents) if firstname(name).endswith('y')] [firstname(name) for name in getcolumn(0, allstudents)] [lastname(name) for name in getcolumn(0, allstudents)] More examples len([name for name in getcolumn(0, allstudents) if firstname(name)[-1] in 'aeiou']) Pie Chart [first for first in [firstname(name) for name in getcolumn(0, allstudents)] if first.endswith('n')] plt.figure(8, figsize=(4,4), facecolor='white') classes = [6,10,19,35] # seniors, juniors, sophs, first-years plt.title('cs111 Class distribution') classcolors = ['red', 'green', 'purple','yellow'] years = ['2016', '2017', '2018','2019'] # pie makes the pie chart plt.pie(classes, labels = years, colors =classcolors) plt.show() # make the plot visible

4 Python built-in functions: sorted( ) and.sort( ) ages = [5,2,8,4,2,0] ages_in_order = sorted(ages) ages_rev_order = sorted(ages, reverse=true) Now let s look at our variables: ages_in_order ages_rev_order ages ages [0,2,2,4,5,8] [8,5,4,2,2,0] [5,2,8,4,2,0] # unchanged # changes the content of ages Another look at our variables: ages [0,2,2,4,5,8] Sorting with tuples # students who live in Caz firstyearscaz = [s for s in allstudents if s[2]=='2019' and s[3].startswith('caz')] [('Chloe Blazey', 'cblazey', '2019', 'Cazenove Hall', '2'), ('Dana Fein-Schaffer', 'dfeinsch', '2019', 'Cazenove Hall', '3'), ('Barakah Quader', 'bquader', '2019', 'Cazenove Hall', '3'), ('Halle Rubera', 'crubera', '2019', 'Cazenove Hall', '3'), ('Noor Pirani', 'npirani', '2019', 'Cazenove Hall', '1'), ('Claire Whitaker', 'cwhitak4', '2019', 'Cazenove Hall', '1'), ('Vivian Zhang', 'vzhang3', '2019', 'Cazenove Hall', '1'), ('Yiting Zhang', 'yzhang16', '2019', 'Cazenove Hall', '1')] sorted(firstyearscaz) [('Barakah Quader', 'bquader', '2019', 'Cazenove Hall', '3'), ('Chloe Blazey', 'cblazey', '2019', 'Cazenove Hall', '2'), ('Claire Whitaker', 'cwhitak4', '2019', 'Cazenove Hall', '1'), ('Dana Fein-Schaffer', 'dfeinsch', '2019', 'Cazenove Hall', '3'), ('Halle Rubera', 'crubera', '2019', 'Cazenove Hall', '3'), ('Noor Pirani', 'npirani', '2019', 'Cazenove Hall', '1'), ('Vivian Zhang', 'vzhang3', '2019', 'Cazenove Hall', '1'), ('Yiting Zhang', 'yzhang16', '2019', 'Cazenove Hall', '1')] sorted([lastname(s[0])for s in firstyearscaz]) ['Blazey', 'Fein-Schaffer', 'Pirani', 'Quader', 'Rubera', 'Whitaker', 'Zhang', 'Zhang'] Sorting with tuples farm = [('lassie',8), ('wilbur', 2),('charlotte',1), ('garfield',10),('pooh', 20),('eeyore',15)] sorted(farm) # sort first by 0 th slot, then 1 st, etc. [('charlotte', 1), ('eeyore', 15), ('garfield', 10), ('lassie', 8), ('pooh', 20), ('wilbur', 2)] sorted([animal[0] for animal in farm]) ['charlotte', 'eeyore', 'garfield', 'lassie', 'pooh', 'wilbur'] Sorting with tuples by a given slot [aka decorate-sort-undecorate] animal_age_tuplist = [(animal[1], animal) for animal in farm] sorted(animal_age_tuplist) [(1, ('charlotte', 1)), (2, ('wilbur', 2)), (8, ('lassie', 8)), (10, ('garfield', 10)), (15, ('eeyore', 15)), (20, ('pooh', 20))] getcolumn(1, sorted(animal_age_tuplist)) [(8, ('lassie', 8)), (2, ('wilbur', 2)), (1, ('charlotte', 1)), (10, ('garfield', 10)), (20, ('pooh', 20)), (15, ('eeyore', 15))] sorted([animal[1] for animal in farm]) [1, 2, 8, 10, 15, 20] [('charlotte', 1), ('wilbur', 2), ('lassie', 8), ('garfield', 10), ('eeyore', 15), ('pooh', 20)] 12-16

5 Another decorate/sort/undecorate example: firsts = ['Ann', Bea', Cat', Dee'] lasts = ['Park', 'Smith', 'Lu'] first_last_tups = zip(firsts,lasts) [('Ann', 'Park'), ('Bea', 'Smith'), ('Cat', 'Lu')] Make new tuple with last name, first_last tuple [('Park', ('Ann', 'Park')), ('Smith', ('Bea', 'Smith')), ('Lu', ('Cat', 'Lu'))] Sort decorated tuples by last name [('Lu', ('Cat', 'Lu')), ('Park', ('Ann', 'Park')), ('Smith', ('Bea', 'Smith'))] Extract the first_last tuple, now sorted by last name [('Cat', 'Lu'), ('Ann', 'Park'), ('Bea', 'Smith')] Sets are collections of items with no duplicates In [155]: dormlist = getcolumn(3, allstudents) In [156]: len(dormlist) Out[156]: 70 In [157]: dormset = # how can you get the unique dorms? In [158]: dormset Out[158]: {'Bates Hall, 'Beebe Hall, 'Cazenove Hall, 'Claflin Hall, 'Davis Hall, 'Dower House, 'Freeman Hall, 'Lake House, 'McAfee Hall, 'Munger Hall, 'Pomeroy Hall, 'Severance Hall', 'Shafer Hall, 'Stone Hall, 'Tower Court East, 'Tower Court West'} In [159]: len(dormset) Out[159]: Sets are collections of items with no duplicates Let s create some new tuples with the number of students in each dorm with the name of the dorm In [160]: studentcountperdorm = # write in notebook In [161]: studentcountperdorm Out[161]: [(2, 'Dower House'), (3, 'Tower Court East'), (3, 'Stone Hall'), (1, 'Lake House'), (6, 'Munger Hall'), (13, 'Cazenove Hall'), (6, 'Freeman Hall'), (6, 'Severance Hall'), (6, 'Shafer Hall'), (9, 'McAfee Hall'), (11, 'Beebe Hall'), (9, 'Tower Court West'), (3, 'Bates Hall'), (4, 'Claflin Hall'), (4, 'Davis Hall'), (6, 'Pomeroy Hall')] Sorting exercises sorted(allstudents) sorted(studentcountperdorm, reverse=true) getcolumn(1, sorted([(lastname(s[0]), s) for s in allstudents])) getcolumn(2, sorted([(s[2], lastname(s[0]), s) for s in allstudents])

6 Make a list of dictionaries, one for each student # create a list of dictionaries, one dict per student # to store the data for each student studentdctinlist = [] for stu in allstudents: stdct = {'name': stu[0], 'section': stu[1], 'year': stu[2], 'dorm': stu[3]} studentdctinlist.append(stdct) print studentdctinlist[14] Now, with this list of dictionaries, we can write more readable code: We can: Find all juniors Find all sophomores in your section And lots of other things Organize our students by year Scenario 1: We know the keys studentsbyyeardct = {'2016': [], '2017': [], '2018': [], '2019': []} Scenario 2: We do not know the keys studentsbyyeardct2 = {} # we start with an empty dict Dictionary Comprehension Very much like list comprehensions: {mykey: myvalue for item in mylist} Example: {x: x**2 for x in (2, 4, 6)} produces {2: 4, 4: 16, 6: 36} Let s write a dictionary comprehension to produce a list of names and length of the names: Hikari Murayama 14 Mina Oh 6 Yu Chi 5 Caroline Martin 14 Stacey Kim 9 Meher Vohra 10 Midori Yang 10 Shaina Ma 8 Anne Shen Dictionary Comprehension len( Anne Shen ) 9 We need to replace the space with an empty string: Anne Shen.replace(, ) AnneShen len( AnneShen ) 8 totallettersinnames = {KEY: VALUE for studct in studentdctinlist} Write the python expressions for KEY and VALUE in the notebook to generate the dictionary comprehension 12-24