Lab 8: File I/O, Mutability vs. Assignment. Ling 1330/2330: Intro to Computational Linguistics Na-Rae Han

Transcription

1 Lab 8: File I/O, Mutability vs. Assignment Ling 1330/2330: Intro to Computational Linguistics Na-Rae Han

2 Objectives File I/O Writing to a file File I/O pitfalls File reference and absolute path Mutability vs. assignment Homework #3 Review Using textstats.py in IDLE shell Shell tips and tricks Tab completion 1/31/2017 2

3 Finally, working with a file How to write a script that reads in a text file and processes it? How to write out the result? File IO (Input/Output) Read in content of a file Write out to a file Write content into an existing file Also: "pickling" a Python data object A list, a dictionary,... NEXT CLASS 1/31/2017 3

4 Recap: Opening a text file for reading f is a file object name of file to read f = open('fox_in_sox.txt', 'r') ## read from f f.close() Closes the file. ALWAYS REMEMBER TO CLOSE YOUR FILE. 'r' for reading. It is default: can be omitted May also need to specify encoding: encoding='utf-8' 1/31/2017 4

5 Opening a text file for writing Just like reading, writing to a file also involves twin operations of creating & closing a file object. f = open('myresults.txt', 'w') ## write to f ## write more to f f.close() Closes the file. ALWAYS REMEMBER TO CLOSE YOUR FILE. 'w' for creating a new file for writing. If file exists, it will be overwritten!! Use 'a' instead for appending to an existing file. May also need to specify encoding: encoding='utf-8' 1/31/2017 5

6 (1) Writing a single string f = open('foo.txt', 'w') f.write('roses are red,\n') f.write('violets are blue.\n') f.close() 'w' option Opens file for writing f.write(str) Writes a single string to file f Creates file foo.txt, and writes a string, twice Roses are red, violets are blue. 1/31/2017 6

7 (2) Writing a list of strings rose = ['Roses are red\n', 'violets are blue.\n'] f = open('foo.txt', 'w') f.writelines(rose) f.close() f.writelines(list) Writes a list of strings to file f Creates file foo.txt, and writes two strings foo.txt Roses are red, violets are blue. 1/31/2017 7

8 .write() works one string at a time f = open('foo.txt', 'w') f.write('roses are red,\n', 'violets are blue.\n') f.close() To write a list of strings in one swoop, use ======================== RESTART =====================.writelines() instead. Traceback (most recent call last): File "F:\foo.py", line 3, in <module> f.write('roses are red,\n', 'violets are blue.\n') TypeError: function takes exactly 1 argument (2 given) >>> 1/31/2017 8

9 Line breaks must be supplied f = open('foo.txt', 'w') f.write('roses are red,') f.write('violets are blue.') f.close() Without \n, these will be printed on a single line. foo.txt Roses are red,violets are blue. print() by default adds a line break..write() and.writelines() DO NOT add a line break. Line breaks '\n' must be explicitly supplied. 1/31/2017 9

10 Can only write string types f = open('foo.txt', 'w') pi = f.write('the value of pi is:\n') f.write(pi) f.close() pi is float type..write() can only take a string argument. =========================== RESTART ========== Traceback (most recent call last): File "F:\foo.py", line 5, in <module> f.write(pi) TypeError: expected a character buffer object >>> 1/31/

11 Can only write string types f = open('foo.txt', 'w') pi = f.write('the value of pi is:\n') f.write(str(pi)) f.close() str() turns pi into a string. foo.txt The value of pi is: write() only takes a string as an argument. Any other data types (integer, float, etc.) must be first converted into string using str() function. 1/31/

12 Don't forget to close file f = open('foo.txt', 'w') f.write('roses are red\n,') f.write('violets are blue\n.') Forgot to specify f.close() foo.txt File writing happens through buffers. Always remember to CLOSE YOUR FILE. Writing out to a file actually happens when your writing buffer is full or the file is closed. So, if you forget to close your file, you might find your output file to be either empty or halfway written. 1/31/

13 Practice 3 minutes Finish the script so it produces the output file. chom = 'Colorless green ideas sleep furiously.' f = open('foo.txt', 'w') for wd in chom.split() : f.write(wd+'\n') f.close()?? foo.txt Colorless green ideas sleep furiously. 1/31/

14 Practice Finish the script so it produces the output file. chom = 'Colorless green ideas sleep furiously.' f = open('foo.txt', 'w') for wd in chom.split() : f.write(wd+'\n') f.close() foo.txt Colorless green ideas sleep furiously. 1/31/

15 Practice 3 minutes Modify the script so it produces the output file. chom = 'Colorless green ideas sleep furiously.' f = open('foo.txt', 'w') for wd in chom.split() : f.write(wd+'\n')?? f.close() foo.txt Colorless is 9 characters long. green is 5 characters long. ideas is 5 characters long. sleep is 5 characters long. furiously. is 10 characters long. 1/31/

16 Practice 3 minutes Modify the script so it produces the output file. chom = 'Colorless green ideas sleep furiously.' f = open('foo.txt', 'w') for wd in chom.split() : f.write(wd+' is '+str(len(wd))+' characters long.\n') f.close() foo.txt Colorless is 9 characters long. green is 5 characters long. ideas is 5 characters long. sleep is 5 characters long. furiously. is 10 characters long. 1/31/

17 Venturing out and into the jungle (aka your computer's hard drive) So far, we have been dealing with files right INSIDE your Python's current WD. We can reference them by their names only, "tale.txt", "fox_in_sox.txt", etc. What about files that are elsewhere on your hard drive? We have to understand and use file and directory path. 1/31/

18 How to find absolute file path Mac Right-click Get Info Windows Right-click Properties 1/31/

19 Absolute file path A reference to a file can include a complete file path starting from the very top of the directory hierarchy: absolute file path OS-X, Linux, Unix: Windows: /Users/narae/Documents/test.txt Starts with the root '/' C:\Users\narae\Documents\test.txt Starts with drive letter: 'C:', 'D:', etc. Uses backslash "\" instead of slash "/" PROBLEMATIC, because backslash has a special meaning in Python!! 1/31/

20 Windows file path in Python Windows uses backslash "\", which is a special character in Python. As a result, Python gives you multiple ways to reference. file = 'C:/Users/narae/Desktop/test.txt' file = 'C:\\Users\\narae\\Desktop\\test.txt' Use slash "/" instead, which Python internally converts to "\" Use backslash "\" but escape every instance file = r'c:\users\narae\desktop\test.txt' r' ' forces the following string ' ' to be interpreted as literal characters Use backslash "\" but instead of escaping use r ('raw string') prefix 1/31/

21 Current working directory Software applications typically operate on the notion of current working directory (WD or CWD) Directory that the application is currently operating in. Typically, "File Open" and "File Save" dialog windows will default to this directory. With Python IDLE, the default WD is: /Users/naraehan/Documents C:\Program Files (x86)\python35-32 (Mac) (Windows) Not convenient at all. We already changed these to our Python script directory!! Windows instruction Mac instruction 1/31/

22 Relative file path, starting from WD A file's location can be specified relative to Working Directory. When a file reference does not start from the top, the starting point is assumed to be the WD. File is located in WD: f = open('fox_in_sox.txt') In a directory called 'data' inside WD: f = open('data/fox_in_sox.txt') Located "one directory up" from WD (WD's parent directory) f = open('../fox_in_sox.txt').. is a short-hand for the parent directory 1/31/

23 File location and path: WD When a file is referred to by its name only, it is assumed to be in the current working directory (WD or CWD). In scripts, WD is the directory where your script is. f = open('fox_in_sox.txt') # read file f.close() outf = open('results.txt', 'w') # write out to file outf.close() foo.py File to read must be in the same directory as the script foo.py File will be created in the directory where foo.py is 1/31/

24 WD in Python shell In IDLE shell or in command-line, WD may be initially set to: Your Python script folder, if you have customized your IDLE environment. If you haven't: /Users/username/Documents C:\Program Files (x86)\python35-32 (Mac) (Windows) >>> f = open('fox_in_sox.txt') Traceback (most recent call last): File "<pyshell#48>", line 1, in <module> f = open('fox_in_sox.txt') IOError: [Errno 2] No such file or directory: 'fox_in_sox.txt' Error: The file is not in your shell's current working directory. 1/31/

25 Discovering and changing your WD >>> import os >>> os.getcwd() 'D:\\Lab' os module must be imported first. os.getcwd() displays "current WD" >>> os.chdir('c:/users/narae/documents') >>> os.getcwd() 'C:\\Users\\narae\\Documents' Windows-internal representation. Directories are separated by '\' os.chdir() changes current WD. It is now set to C:\Users\narae\Documents (Windows) In OS X, directories look like /Users/naraehan/Documents 1/31/

26 File path & WD, a summary Referencing a file with its absolute path: '/Users/naraehan/Documents/fox_in_sox.txt' 'C:/Users/naraehan/Documents/fox_in_sox.txt' always works. OS-X, Linux Windows If referencing with shorthand: 'fox_in_sox.txt', the file has to be in the current WD (working directory). In a SCRIPT: WD is where the script is file and script should be in the same dir. After your script is executed in IDLE shell, shell's WD changes to the script's location. In IDLE shell, your initial WD depends on your setting. Find out WD and change it using the os module: os.getcwd(), os.chdir() Beware: after running a script, your shell's WD changes to the script's location. 1/31/

27 File path & WD, recommended practice If you configured your Python IDLE, it conveniently launches with your own Python script directory as the initial WD. Keep all your scripts,.txt input and output files in there File I/O is relatively hassle free. If you need to work with files somewhere else, be mindful of your WD and the absolute file path. 1/31/

28 Copying a list content, 1 st try >>> >>> sim = ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie'] gran = ['Abe', 'Mona'] How to build a new list of all Simpson family members? 1/31/

29 Copying a list content, 1 st try >>> sim = ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie'] >>> gran = ['Abe', 'Mona'] >>> allsim = sim >>> allsim.extend(gran) >>> allsim ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie', 'Abe', 'Mona'] >>> sim ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie', 'Abe', 'Mona'] >>> Fail!! sim also changed. What's going on?? 1/31/

30 Mutability strikes back Copying a string "Copying" a list >>> x = 'hello' >>> x2 = x >>> x2 += '!!' >>> x2 'hello!!' >>> x 'hello' >>> >>> x = [1, 2, 3] >>> x2 = x >>> x2.append(4) >>> x2 [1, 2, 3, 4] >>> x [1, 2, 3, 4] >>> What's going on?? We need to take a closer look at assignment vs. mutability. 1/31/

31 Assignment: under the hood x = 3 Binding a variable in Python means setting a name to hold a reference to some object. Assignment creates references, not copies. Names in Python do not have an intrinsic type; objects do. Python determines the type of the reference automatically based on the data object assigned to it. You create a name the first time it appears on the left side of an assignment statement: x = 3 A reference is deleted via garbage collection after any names bound to it have passed out of scope. 1/31/

32 Understanding reference semantics Assignment manipulates references: it's all about pointing. var2 = var1 does not make a copy of the object var1 references var2 = var1 makes var2 reference the object var1 references! var1 int 3 var2 1/31/

33 Reference vs. immutable types So, for immutable datatypes (integers, floats, strings) assignment behaves as you would expect: >>> x = 'hi' # Creates 'hi', name x refers to it >>> x2 = x # Creates name x2, refers to 'hi' >>> x2 += '!!' # Creates ref for 'hi!!', changes x2 >>> x2 'hi!!' >>> x # No effect on x, still refers to 'hi' 'hi' x 'hi' int 1/31/

34 Reference vs. immutable types So, for immutable datatypes (integers, floats, strings) assignment behaves as you would expect: >>> x = 'hi' # Creates 'hi', name x refers to it >>> x2 = x # Creates name x2, refers to 'hi' >>> x2 += '!!' # Creates ref for 'hi!!', changes x2 >>> x2 'hi!!' >>> x # No effect on x, still refers to 'hi' 'hi' x 'hi' int x2 1/31/

35 Reference vs. immutable types So, for immutable datatypes (integers, floats, strings) assignment behaves as you would expect: >>> x = 'hi' # Creates 'hi', name x refers to it >>> x2 = x # Creates name x2, refers to 'hi' >>> x2 += '!!' # Creates ref for 'hi!!', changes x2 >>> x2 'hi!!' >>> x # No effect on x, still refers to 'hi' 'hi' x x2 'hi' int 'hi!!' int x and x2 now have different values. 1/31/

36 Reference vs. mutable types Mutable data types (list, dictionaries) behave differently! Method functions change these data in place, so >>> x = [1,2,3] # x references list [1,2,3] >>> x2 = x # x2 now references what x references >>> x2.append(4) # Changes the original list in memory >>> x2 [1, 2, 3, 4] >>> x # x too points to the changed list! [1, 2, 3, 4] x int 1/31/

37 Reference vs. mutable types Mutable data types (list, dictionaries) behave differently! Method functions change these data in place, so >>> x = [1,2,3] # x references list [1,2,3] >>> x2 = x # x2 now references what x references >>> x2.append(4) # Changes the original list in memory >>> x2 [1, 2, 3, 4] >>> x # x too points to the changed list! [1, 2, 3, 4] x x int 1/31/

38 Reference vs. mutable types Mutable data types (list, dictionaries) behave differently! Method functions change these data in place, so >>> x = [1,2,3] # x references list [1,2,3] >>> x2 = x # x2 now references what x references >>> x2.append(4) # Changes the original list in memory >>> x2 [1, 2, 3, 4] >>> x # x too points to the changed list! [1, 2, 3, 4] x x int 1/31/

39 Reference vs. mutable types Mutable data types (list, dictionaries) behave differently! Method functions change these data in place, so >>> x = [1,2,3] # x references list [1,2,3] >>> x2 = x # x2 now references what x references >>> x2.append(4) # Changes the original list in memory >>> x2 [1, 2, 3, 4] >>> x # x too points to the changed list! [1, 2, 3, 4] x x int x and x2 refer to the same object in memory! When x is modified, x2 also changes 1/31/

40 Copying a list content, 1 st try >>> sim = ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie'] >>> gran = ['Abe', 'Mona'] >>> allsim = sim >>> allsim.extend(gran) >>> allsim ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie', 'Abe', 'Mona'] >>> sim ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie', 'Abe', 'Mona'] >>> Fail!! sim also changed. So! How do we create a *new* list object from an existing list? 1/31/

41 Clone a list through [:] >>> >>> >>> >>> >>> ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie', 'Abe', 'Mona'] >>> ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie'] >>> sim = ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie'] gran = ['Abe', 'Mona'] allsim = sim[:] allsim.extend(gran) allsim sim [:] returns a whole slice of sim, as a *new list object* SUCCESS! 1/31/

42 + merges 2 lists into a new one >>> >>> >>> >>> ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie', 'Abe', 'Mona'] >>> ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie'] >>> sim = ['Homer', 'Marge', 'Bart', 'Lisa', 'Maggie'] gran = ['Abe', 'Mona'] allsim = sim + gran allsim sim + merges two lists and returns it SUCCESS! 1/31/

43 Creating a *new* list obj from existing [:] slicing >>> x = [1,2,3,4,5] >>> x[2:] [3, 4, 5] >>> x[:] [1, 2, 3, 4, 5] >>> + operator >>> x = [1,2,3,4] >>> y = [5,6,7] >>> x + y [1, 2, 3, 4, 5, 6, 7] >>> x + [100, 200] [1, 2, 3, 4, 100, 200] >>> The whole slice [:] clones the entire list and returns it as a new object. list1 + list2 returns a new list created by concatenating the contents. 1/31/

44 Try it out 2 minutes >>> x = 'hello' >>> x2 = x >>> x2 += '!!' >>> x2 'hello!!' >>> x 'hello' >>> >>> x = [1,2,3,4] >>> y = [5,6,7] >>> x + y [1, 2, 3, 4, 5, 6, 7] >>> x + [100, 200] [1, 2, 3, 4, 100, 200] >>> >>> x = [1, 2, 3] >>> x2 = x >>> x2.append(4) >>> x2 [1, 2, 3, 4] >>> x [1, 2, 3, 4] >>> x[1:] [2, 3, 4] >>> x[:] [1, 2, 3, 4] >>> x3 = x[:] >>> x3.append(100) >>> x3 [1, 2, 3, 4, 100] >>> x [1, 2, 3, 4] >>> 1/31/

45 Congratulations! You have now learned all of essential Python. From this point on, we will focus on: Applying the knowledge to real-world problems How to process text for basic statistics (already started) How to work with a corpus How to search through text data Learn a few additional tricks List comprehension Pickling Regular expressions 1/31/

46 It is time to REVIEW We have learned a WHOLE LOT so far. First of all, you must KNOW WHAT WE LEARNED. This is a GOOD time to REVIEW and make sure you have a good command of Python basics. Secondly, you must BE ABLE TO SYNTHESIZE from what you know. As you review the slides and tutorials, you will find yourself nodding along. But reading comprehension gets you only so far you should write them yourself! Next level up, you should be able to apply your coding skills to NOVEL PROBLEMS wd = 'penguin' rev = '' for i in wd : rev = i + rev print(rev) print(rev) 1/31/

47 A common misconception Wow. So many commands. How am I going to memorize all these!!! TRUTH: Nobody, even seasoned programmers, codes solely from memory. Rather than trying to commit everything to memory, you should: have a good overall knowledge of programming structure and practices be willing to explore have your references handy know how to look stuff up! 1/31/

48 Be more efficient Do you find coding tedious? Make sure you are: Fully utilizing the SHELL side for testing. Remember the "Rookie vs. Pro ways" video? Utilizing your command-line history. Don't re-type your commands! Change Ctrl+p/n to up arrow ( ) and down arrow ( ) Using dir() and help(). Using TAB completion and tooltips. Will show you in a moment. 1/31/

49 Homework 3 review Online resources The power of functions and modular programming Building up complex data through pipelining n-gram functions How to generalize? Why insist on tuple format? gettypes() revisited.keys() not necessary after all! There are now two ways of building a frequency dictionary. 1/31/

50 Norvig's data: word lists words.js enable1.txt What are they? How big? 1/31/

51 Norvig's data: 1- & 2-grams count_1w.txt count_2w.txt Where do they come from? 1/31/

52 COCA n-gram lists 2-grams 66 a ba 41 a babble 28 a babbling 159 a babe 83 a baboon 9744 a baby 31 a baby-faced 122 a baby-sitter 237 a babysitter 23 a babysitting 95 a baccalaureate 71 a bach 1342 a bachelor 27 a bachelorette 53 a bachelors 1924 a back 38 a back-and-forth 24 a back-door 29 a back-to-basics 27 a back-to-school 100 a back-up 3-grams Anything you noticed? 33 a ba in 35 a babble of 33 a babe in 316 a baby and 25 a baby as 73 a baby at 32 a baby before 53 a baby bird 57 a baby boomer 34 a baby born 146 a baby boy 36 a baby brother 29 a baby by 34 a baby can 45 a baby carriage 45 a baby crying 39 a baby doll 47 a baby for 41 a baby from 224 a baby girl 35 a baby grand 52

53 What did you find? 1/31/

54 Word bigram function def getword2grams(wds) : "Takes a tokenized word list, returns a bigram list" bigrams = [] for i in range(len(txt)-1) : gram = tuple(txt[i:i+2]) bigrams.append(gram) return bigrams Why was it necessary to make the n-grams a tuple type? What is wrong with lists? >>> chomtoks ['colorless', 'green', 'ideas', 'sleep', 'furiously', '.'] >>> getword2grams(chomtoks) [('colorless', 'green'), ('green', 'ideas'), ('ideas', 'sleep'), ('sleep', 'furiously'), ('furiously', '.')] >>> 1/31/

55 It's all about the pipeline getword2grams [('rose', 'is'), ('is', 'a'), ('a', 'rose'), ('rose', 'is'), ('is', 'a'), ('a', 'rose'), ('rose', '.')] Eventually, we want to produce an n-gram frequency dictionary. getfreq {('rose', 'is'): 2, ('a', 'rose'): 2, ('rose', '.'): 1, ('is', 'a'): 2} Tuples (immutable!) can be a dictionary key. Lists and other mutable types cannot. 1/31/

56 Generalized n-gram function def getwordngrams(wds, n) : "Takes a tokenized word list, returns an n-gram list" ngrams = [] for i in range(len(txt)-n+1)? : gram = tuple(txt[i:i+n]) ngrams.append(gram) return ngrams >>> chomtoks ['colorless', 'green', 'ideas', 'sleep', 'furiously', '.'] >>> getwordngrams(chomtoks, 4) [('colorless', 'green', 'ideas', 'sleep'), ('green', 'ideas', 'sleep', 'furiously'), ('ideas', 'sleep', 'furiously', '.')] >>> 1/31/

57 Generalized n-gram function def getwordngrams(wds, n) : "Takes a tokenized word list, returns an n-gram list" ngrams = [] for i in range(len(txt)-n+1) : gram = tuple(txt[i:i+n]) ngrams.append(gram) return ngrams >>> chomtoks ['colorless', 'green', 'ideas', 'sleep', 'furiously', '.'] >>> getwordngrams(chomtoks, 4) [('colorless', 'green', 'ideas', 'sleep'), ('green', 'ideas', 'sleep', 'furiously'), ('ideas', 'sleep', 'furiously', '.')] >>> 1/31/

58 Building various data objects 'Rose is a rose is a rose is a rose.' gettokens gettypefreq gettypes ['rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.']?? {'a': 3, 'is': 3, '.': 1, 'rose': 4} getxfreqwords ['.', 'a', 'is', 'rose'] x = 3 x = 2 getxlengthwords Can we build a type frequency dictionary from a token list? ['a', 'is', 'rose'] ['is', 'rose'] 1/31/

59 More than one way to build 'Rose is a rose is a rose is a rose.' gettokens gettypefreq gettypes ['rose', 'is', 'a', 'rose', 'is', 'a', 'rose', 'is', 'a', 'rose', '.'] getfreq {'a': 3, 'is': 3, '.': 1, 'rose': 4} getxfreqwords ['.', 'a', 'is', 'rose'] x = 3 x = 2 getxlengthwords Yep! Pass it through getfreq(), a general-purpose frequency counter function. ['a', 'is', 'rose'] ['is', 'rose'] 1/31/

60 Simplifying functions def gettypes(txt) : """Takes a piece of text (a single string), returns an alphabetically sorted list of unique word types. """ tfreq = gettypefreq(txt) return sorted(tfreq.keys()) same as: sorted(tfreq) sorted() can take a list, string, and returns a list. When taking a dictionary, it returns a sorted list of dictionary keys. 1/31/

61 Wrapping up Next class: Pickling Handling multiple text files, large text files Exercise 5 File I/O with O. Henry Essentially the same as HW3, but writing results out to a file HW3 solution will be posted on CourseWeb, as an attachment to the original submission link 1/31/