Lab 7: Reading Files, Importing, Bigram Function. Ling 1330/2330: Computational Linguistics Na-Rae Han

Transcription

1 Lab 7: Reading Files, Importing, Bigram Function Ling 1330/2330: Computational Linguistics Na-Rae Han

2 Objectives Importing Reading text files range() Bigram function More sorting with sorted() sorted() + indexing/slicing 9/18/2018 2

3 More libraries through import So far, we have been only using built-in data types and functions: int, str, list, dict, tuple, len(), print(), sum(), *, +, -, But there are tons of additional modules and packages! You gain access to them by importing. import math math.sqrt(64) 8.0 math.factorial(5) 120 Import the math module Call math functions. Must prefix math. 9/18/2018 3

4 math module import math dir(math) [' doc ', ' name ', ' package ', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'hypot', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'modf', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'trunc'] math.sqrt(144) 12.0 help(math.log) Help on built-in function log in module math: log(...) log(x[, base]) dir() and help() come in handy for exploring. Return the logarithm of x to the given base. If the base not specified, returns the natural logarithm (base e) of x. math.log(16, 2) 4.0 9/18/2018 4

5 os module os.getcwd() returns the "current working directory" (CWD) import os os.getcwd() '/Users/narae/Documents' help(os.getcwd) Help on built-in function getcwd in module nt: os provides operatingsystem-dependent functionality. getcwd() Return a unicode string representing the current working directory. os.chdir('pythoncode') os.getcwd() '/Users/narae/Documents/pythoncode' os.listdir() ['pal_loop.py', 'pal_naive.py', 'pg16328.txt', 'pig_latin.solution.py', 'pig_latin.template.py', 'secret.py', 'textstats.template.py', 'v_c_count.py', 'v_c_count_try.py', ' pycache '] 9/18/2018 5

6 os module import os os.getcwd() '/Users/narae/Documents' help(os.getcwd) Help on built-in function getcwd in module nt: getcwd() Return a unicode string representing the current working directory. os.chdir('pythoncode') os.getcwd() '/Users/narae/Documents/pythoncode' os.listdir() os.chdir() changes the "current working directory" ['pal_loop.py', 'pal_naive.py', 'pg16328.txt', 'pig_latin.solution.py', 'pig_latin.template.py', 'secret.py', 'textstats.template.py', 'v_c_count.py', 'v_c_count_try.py', ' pycache '] 9/18/2018 6

7 os module import os os.getcwd() '/Users/narae/Documents' help(os.getcwd) Help on built-in function getcwd in module nt: getcwd() Return a unicode string representing the current working directory. os.chdir('pythoncode') os.getcwd() '/Users/narae/Documents/pythoncode' os.listdir() ['pal_loop.py', 'pal_naive.py', 'pg16328.txt', 'pig_latin.solution.py', 'pig_latin.template.py', 'secret.py', 'textstats.template.py', 'v_c_count.py', 'v_c_count_try.py', ' pycache '] os.listdir() returns the list of files/dirs in the CWD 9/18/2018 7

8 Try it out 1 minute import math dir(math) math.sqrt(144) help(math.log) math.log(16, 2) import os os.getcwd() help(os.getcwd) os.listdir() os.chdir('pythoncode') Depends on your environment. See A10 "File Path and CWD". 9/18/2018 8

9 What is your default CWD? Windows users: You should NOT be seeing this! This should be your own designated Python script directory. Change your default CWD by following: 9/18/2018 9

10 What is your default CWD? Mac users: Ideally, your initial cwd should be your Python script directory Experienced users can change their default CWD by: If you don't want to mess with command-line, you will need to get in the habit of using os.chdir(). os.chdir('pytnoncode') os.getcwd() '/Users/narae/Documents/pythoncode' See A10 "File Path and CWD". 9/18/

11 fox_in_sox.txt The text has 5 lines. 9/18/

12 Opening a text file for reading f is a file object name of file to read f = open('fox_in_sox.txt', 'r') ## read from f f.close() Closes the file. ALWAYS REMEMBER TO CLOSE YOUR FILE. 'r' for reading. It is default: can be omitted May also need to specify encoding: encoding='utf-8' 9/18/

13 (1) Reading entire text as a single string f = open('fox_in_sox.txt', 'r') text = f.read() f.read() Variable text is a string obj whose value is the entire text f.close() print(text) ======================== RESTART ===================== Through three cheese trees three free fleas flew. While these fleas flew, freezy breeze blew. Freezy breeze made these three trees freeze. Freezy trees made these trees' cheese freeze. That's what made these three free fleas sneeze. print added an extra line break 9/18/

14 (2) Reading entire text as a list of lines f = open('fox_in_sox.txt') lines = f.readlines() f.close() print(lines) f.readlines() reads in the text as a list of lines. lines is a list of strings, where each string is a single line. ======================== RESTART ===================== ['Through three cheese trees three free fleas flew.\n', 'While these fleas flew, freezy breeze blew.\n', 'Freezy breeze made these three trees freeze.\n', "Freezy trees made these trees' cheese freeze.\n", "That's what made these three free fleas sneeze.\n"] 9/18/

15 Try it out 3 minutes Download file and save in your usual script directory: And try these commands, in IDLE shell f = open('fox_in_sox.txt') text = f.read() f.close() print(text) print(text.split()) f = open('fox_in_sox.txt') lines = f.readlines() f.close() print(lines) print(lines[0]) Do NOT copy and paste file content! Use "SAVE AS" dialog box in your browser to save the file ITSELF as a text file. for l in lines : if 'tree' in l : print(l) print(l, end='') prevents extra line break 9/18/

16 range() function for x in range(5, 10): print(x) starts at 5 stops BEFORE 10 9/18/

17 range() function for x in range(5, 10): print(x) for x in range(10): print(x) No starting point specified: starts at 0 9/18/

18 Practice 2 minutes Compose a for loop that prints out for every character in a word: its index, the character itself ' t i g e r ' word = 'tiger' for i in range(len(word)): print(i, word[i]) 0 t 1 i 2 g 3 e 4 r The length of the string should be the "stop-before" point. 9/18/

19 Practice 2 minutes Compose a for loop that prints out for every character in a word: its index, the character itself ' t i g e r ' word = 'tiger' for i in range(len(word)): print(i, word[i]) 0 t 1 i 2 g 3 e 4 r len(word) returns 5, range(5) generates 0, 1, 2, 3, 4 9/18/

20 word = 'tiger' for i in range(len(word)): print(i, word[i]) Practice 2 minutes 0 t 1 i 2 g 3 e 4 r word = 'tiger' for i in range(len(word)): print(i, word[i:]) 0 tiger 1 iger 2 ger 3 er 4 r ' t i g e r ' word = 'tiger' for i in range(len(word)): print(i, word[i:i+2]) 0 ti 1 ig 2 ge 3 er 4 r 9/18/

21 word = 'tiger' for i in range(len(word)): print(i, word[i]) Practice 2 minutes 0 t 1 i 2 g 3 e 4 r word = 'tiger' for i in range(len(word)): print(i, word[i:]) 0 tiger 1 iger 2 ger 3 er 4 r ' t i g e r ' word = 'tiger' for i in range(len(word)): print(i, word[i:i+2]) 0 ti 1 ig 2 ge 3 er 4 r 9/18/

22 Producing character-level bigrams txt = 'tiger' txt[0:2] ti txt[1:3] ig for i in range(len(txt)) : print(txt[i:i+2]) ti ig ge er r [i:i+2] returns a 2-character slice 1-char string at the end:* txt[4:6] How to avoid? *Slicing stopping point can be beyond index: 'penguin'[3:100] returns 'guin' just fine. 9/18/

23 Producing character-level bigrams txt = 'tiger' txt[0:2] ti txt[1:3] ig for i in range(len(txt)-1) : print(txt[i:i+2]) ti ig ge er range() input is modified so that i stops at one integer before 9/18/

24 Building a char-level bigram function Where the last gram starts: 2-gram txt = ' t i g e r ' for i in range(len(txt)-1) : gram = txt[i:i+2] Your last 2-gram should be 'er'(txt[3:5]), not 'r' (txt[4:6]) That means i should range from 0 to 3, stopping at 3. That means: len(txt) = 5 range(len(txt)) 0, 1, 2, 3, 4 range(len(txt)-1) 0, 1, 2, 3 9/18/

25 Character-level bigram function def getchar2grams(txt) : bigrams = [] for i in range(len(txt)-1) : gram = txt[i:i+2]?? bigrams.append(gram) return bigrams getchar2grams('penguin') ['pe', 'en', 'ng', 'gu', 'ui', 'in'] getchar2grams('hello, world!') ['He', 'el', 'll', 'lo', 'o,', ', ', ' w', 'wo', 'or', 'rl', 'ld', 'd!'] 9/18/

26 Character-level bigram function def getchar2grams(txt) : bigrams = [] for i in range(len(txt)-1) : gram = txt[i:i+2] bigrams.append(gram) return bigrams getchar2grams('penguin') ['pe', 'en', 'ng', 'gu', 'ui', 'in'] getchar2grams('hello, world!') ['He', 'el', 'll', 'lo', 'o,', ', ', ' w', 'wo', 'or', 'rl', 'ld', 'd!'] Character-level bigrams: tokenization of symbols not necessary 9/18/

27 Try it out 2 minutes def getchar2grams(txt) : bigrams = [] for i in range(len(txt)-1) : gram = txt[i:i+2] bigrams.append(gram) return bigrams getchar2grams('penguin') ['pe', 'en', 'ng', 'gu', 'ui', 'in'] getchar2grams('hello, world!') ['He', 'el', 'll', 'lo', 'o,', ', ', ' w', 'wo', 'or', 'rl', 'ld', 'd!'] 9/18/

28 Sorting + slicing Looking to pick out the very first N items? Returned value of sorted() can be indexed & sliced: li = ['fox', 'dog', 'cat', 'owl', 'ant'] sorted(li) ['ant', 'cat', 'dog', 'fox', 'owl'] sorted(li)[0] 'ant' sorted(li)[:2] ['ant', 'cat'] sorted(li, reverse=true)[:3] ['owl', 'fox', 'dog'] #1 ranked Top 2 Top 3, reverse ranking 9/18/

29 Population by state Find the popul dictionary in text-samples.txt Copy & paste it into IDLE shell 3 minutes Print out the #1 populous state Print out the top 10 most populous states Print out the top 5 least populous states 9/18/

30 for x in sorted(popul, key=popul.get, reverse=true)[:10]: print(x, popul[x]) ca tx ny fl il pa oh ga mi nc for x in sorted(popul, key=popul.get)[:5]: print(x, popul[x]) 10 most populous wy vt nd ak sd least populous 30

31 HW 2 review textstats.py Exercise 5 Update textstats.py with the new character-level bigram function Process how_do_i.txt 9/18/

32 Wrapping up Next class: Writing files, file and directory path, importing user-built module Exercise 5 9/18/