Working with data files; the len() and.split() functions

Transcription

1 Basic File Operations Working with data files; the len() and.split() functions Objectives To use text files to store large data sets To access online data sources To introduce the for loop To introduce ASCII, Unicode, and Latin-1 To use the len() function and.split() method 1

2 Text File Sequence of characters, organized into lines, stored on some external memory device (such as a hard drive) End of line markers (newline character) - Sometimes the last line is missing the newline character Not all files are text files - Some contain binary information instead - We aren't concerned with these right now Files, File Names, and File Handles File a set of data, stored on a disk or flash drive of some sort - Sometimes called a disk file - Text strings or binary data File name the data set s name, used to locate the data set on the drive - Just a text string - Often includes an extension, e.g..txt File handle a Python object, used to refer to and operate on a file 2

3 Things to Do with Files Open a file - open() Read from a file - read()» some number of bytes - readline()» read a line of text - readlines()» read all lines of text Print to a file - print() function, with a keyword argument Write to a file -.write() method» writes one string or a block of bytes Printing Your Output to a File First, open the file: - handle = open( mynewfile.txt, w ) Use the handle in print() statements: - print( example output, file=handle ) - Use as many print() statements as desired; use formatting as desired Finish by closing the file: - handle.close() 3

4 Try: fn = input('new file name?') fh = open(fn, 'w') print('my name is Bob', file=fh) # use your name x = 2**13 print('{} equals {:#x}'.format(x,x), file=fh) print( file=fh ) # print a blank line print('goodbye!', file=fh) Open your new file in a text editor to verify that it was created An Alternative to Printing Writing To a File.write() method writes exactly one string - handle.write( 'This is a string' ) No formatting, no newline at the end - Add your own newlines: handle.write( 'This is a string\n' ) Build a formatted string of desired output, then write it out to the file 4

5 .write( ) example Add your own newlines.write() is a method of the file object Why use.write() instead of print()? More efficient than print() function More general, and allows more precise control of what goes into the file Supports writing of binary / non-text data to a file. 5

6 Organization of Text Files Text files contain sequences of ASCII characters - We ll consider Unicode later Text is grouped into lines print() statements each generate one line of output - (usually) A Detail about Lines in Text Files Lines are separated by newline characters (or end-of-line sequences), and different operating systems use different newlines - MSDOS, Windows: ASCII 0x0d,0x0a known as carriage return, line feed - 0x0d == 13, 0x0a == 10 - Linux: ASCII 0x0a only known as newline or linefeed - MacOS: ASCII 0x0d only known as carriage return Blank lines contain a newline by itself Python handles all three formats the same 6

7 So, How to Read Input From a Text File? First, open the file for reading: - handle = open( mydatafile.txt, r ) Next, read one or more lines into variables: - l0 = handle.readline() # read 1 line into variable - l1 = handle.readline() # read next line - l2 = handle.readline() #...and again Close the file when you re done with it: - handle.close() try: fn = input('what filename did you use before?') fh = open(fn, 'r') l0 = fh.readline() print( '{:3d} -->{}<--'.format(len(l0), l0) ) l1 = fh.readline() print( '{:3d} -->{}<--'.format(len(l1), l1) ) l2 = fh.readline() print( '{:3d} -->{}<--'.format(len(l2), l2) ) 7

8 The len() Function len( 'string' ) provides the length of the string Examples: - len('abcde') # equals 5 - s = 'XYZ' len(s) # equals 3 - len( '' ) # equals 0 len() is used for lists and dictionaries too - we'll see this later The for statement for repeats a logical block of actions, once for each element in a collection of things - A file is a collection of lines of text - The 'lines' list is a collection of text strings General structure: for <variable> in <collection> : statement statement statement... 8

9 for examples for l in 'qwertyuiop': print(l) # print 1 char per line s = '' for w in ['These', 'are', 'words', '.']: s += w print(s) # build a sentence from a word list n = 1 for i in range(5,10): # range( ) generates integers n += i**2 print('sum of squares: ', n) for and text files for statement treats the.readlines() method as a collection of all the textlines in the file - or as many as are left, if part of the file has been read already or if a.seek() has moved to a particular position in the file Typical use looks like this: - for l in handle.readlines(): print( len(l), '-->%s<--' % (l) ) # do other things with l... 9

10 Example Open the file for reading again: - handle = open( mydatafile.txt, r ) Read each of the file's lines into a list: - lines = [] # create empty list for l in handle.readlines(): lines.append(l) # get all lines, 1 at a time, # add to end of list Close the file when you re done with it: - handle.close() Try: fn = input('datafile name? ') fh = open(fn, 'r') for l in fh.readlines(): print('{:3d} ->{}<-'.format(len(l), l)) 10

11 back to text files Once a file is opened for reading, its file handle has methods:.readlines()» read all lines of text, one at a time.readline()» read a single line of text.read( n )» read n characters, default is to read the whole file.seek( n )» find a specific location in the file ( useful with.read() ).close()» close the file to prevent changing it by accident Getting Numbers From Text Files As with the input() statement, all received data consists of text strings As needed, convert the text to numbers - Use int() or float() to convert "number-like" strings into numbers Example: - valuestring = handle.readline() value = float(valuestring) Example, reworked: - value = float( handle.readline() ) 11

12 Try: first set up some random data fn = input('data file name?') fh = open(fn, 'w') import random for line in range(17): print(random.random(), file=fh) Creates a number between 0.0 and 1.0 Look at your file in a text editor Try: read the data back fn = input('data file?') fh = open(fn, 'r') x = 0 n = 0 for line in fh.readlines(): x += float(line) n += 1 print( \ Add number into variable X 'Average of {} values is {}'.format(n, x/n)) 12

13 Getting Numbers From Formatted Files What if more than one value on a line? - Use.split() to form a list of substrings - Convert list elements individually with int() or float() as desired» if in doubt, use float() What if line contains additional information? - Use.split() to form a list of substrings - Choose which element of the list you want - Convert that element with int() or float() Try: store some random integers fn = input('data file name?') fh = open(fn, 'w') import random for line in range(17): for i in range(10): print(random.randrange(-5, 5), \ print(file=fh) end=' ', file=fh) Inspect your file in a text editor Creates numbers from -5 through +4 13

14 Try: read the data back fn = input('data file?') fh = open(fn, 'r') x = 0 n = 0 for line in fh.readlines(): values = line.split() print(values) for v in values: x += int(v) n += 1 Add numbers into variable X print('sum of {} values is {}'.format(n, x)) Try: read from a formatted file Copy 'earthquakes0a.txt from your O: drive or browse to 'montcs.bloomu.edu/~bobmon/datasets/earthquakes0a.txt' - Save file to your P-drive handle = open('earthquakes0a.txt', 'r') maxmagnitude = 0 location = None for line in handle.readlines(): parts = line.split() mag = float(parts[5]) if mag > maxmagnitude: maxmagnitude = mag location = parts[6:] print('strongest: {} at {}'.format(maxmagnitude, \ location)) 14

15 ASCII, Unicode, and Python Text Files Text files contain text, but what is text? Many files use 7-bit ASCII - "Lowest common denominator" Some files use 8-bit extended ASCII - "ISO " encoding, a.k.a. "latin-1" Unicode is becoming more common - "UTF-8" encodes characters into 1, 2, or 4 bytes: Ɱ are examples Python 3 tries to open text files using UTF-8 or latin-1, depending on the host system - Windows: latin-1; Linux/Mac: UTF-8 Try: Copy the files country-population.2016.utf-8.txt and country-population.2016.latin-18.txt from your O: drive Do the following (in the IDLE shell): fh = open('country-population.2016.utf-8.txt', 'r') linesa = fh.readlines() fh = open('country-population.2016.latin-1.txt', 'r') linesb = fh.readlines() 15

16 Continued: Compare line 162 from each file: print(linesa[162]) print(linesb[162]) Do the lines look the same? Bytes Bytes are groups of 8 bits - Computer file and memory sizes are measured in bytes 7-bit ASCII characters take 1 byte each 8-bit extended ASCII also uses 1 byte each - Characters such as: «ñ - Latin-1 encoding is NOT compatible with UTF-8! 8-bit ASCII files must be opened with the correct encoding - or no encoding, as a "binary file" this is messier 16

17 Reading 8-bit ASCII Files Statements like "fh = open('file.txt', 'r')" default to UTF-8 Open 8-bit ASCII files as Latin-1 (or ISO ) - fh = open('file.txt','r',encoding='latin-1') Handle as usual from here on Try: fh = open( 'country-population.2016.txt', \ 'r', encoding='latin-1' ) lines = [] for l in fh.readlines(): lines.append(l) print( len(lines), 'entries in file' ) print(lines[56]) 17

18 What if you don't know how the file was encoded? Most general solution: Treat it as bytes Open the file in binary mode: - fh = open('unknownfile.txt', 'rb') Read byte strings: - bs = fh.readline() Try to decode the line using possible encodings: - try: data = bs.decode('utf-8') except: data = bs.decode('latin-1') try-except: another topic we'll explore later 18