Lecture 17: Files. CS1068+ Introductory Programming in Python. Dr Kieran T. Herley 2018/17. Department of Computer Science University College Cork

Transcription

1 Lecture 17: Files CS1068+ Introductory Programming in Python Dr Kieran T. Herley 2018/17 Department of Computer Science University College Cork

2 Summary Python s sequential file-processing capabilities: Sequential text files. Read, write and append mode. with statement. Binary files. 1

3 Files in Python

4 Files Computer systems use files to store all sorts of stuff: documents, programs (source and executables), images, webpages, lecture slides, etc. Files are persistent: data preserved between executions of program, unlike ephemeral prog. data. 2

5 File types In the Python world files fall into two broad categories: Text files Comprise printable text characters only Convenient and widely used for: programs (source), webpages, Can be read/modified by simple text-editor tools Not system- or application-dependent Binary files e.g. jpg images,.doc files, Typically not human readable (without specific software) Typically system or application dependent 3

6 Sequential Text Files

7 Python s text-file model Python text files contains a linear sequence of characters encoded in ASCII or Unicode separated into lines Sequential file processing: File processed in read-only or write-only mode Progress through a file from beginning to end without backing up (More efficient than hopping around) 4

8 Text files: Unix vs Windows Windows and Unix systems use diff. conventions for marking line breaks: Sym. Char. ASCII (hex) Unix LF \n 0A Windows CR LF \r\n 0A 0D Unix dos2unix and unix2dos useful for conversions 5

9 Python s text-file model Python used open-read-close idiom for seq. file procesing Typical code skeleton: file name = whatever.txt textfile = open(file name, r ) # (1) Must open before use # (2) Read through the file from begining to end textfile. close () # (3) close when done Specify file mode when opening ( r means read only) variable textfile points to file object bundle of housekeeping info relating to file 6

10 Python s file-reading operations Old-fashioned way f.readline() f.read() f.readlines() Better way (usually) Use file iterator (later) readline reads a single line (as a string, including newline); empty string signifies end of file read devours entire file and returns contents as single string readlines devours entire file and returns contents as list of lines (strings) 7

11 Reading file one line at a time readline Code snippet to count the lines in a file; open and close omitted here to declutter num lines = 0 line = textfile. readline () while len( line ) > 0: num lines += 1 line = textfile. readline () print (...) 8

12 Reading file in one go read and readlines The same thing using read and readlines text = textfile.read() num lines = text.count( \n ) Quibble: Assumes last line of file contains a newline. print (...) text = textfile. readlines () num lines = len(text) readlines devours entire file and returns contents as list of lines (strings) print (...) 9

13 File iterators Can use for loop to step through file line by line... for line in textfile : # do stuff... For each iteration line takes on value of a line from file and teh loop body executed 10

14 File iterators cont d The line-counting application again: name = debanks.txt textfile = open(name, r ) num lines = 0 for line in textfile : num lines += 1 print( File '%s' contains %i lines % (name, num lines)) textfile. close () 11

15 File iterators and with statement A slightly neater version line-counting application again: name = debanks.txt with open(name, r ) as textfile : num lines = 0 for line in textfile : num lines += 1 print( File '%s' contains %i lines % (name, num lines)) with statement provides envelope wrapped around file code embodies both open statement and close statement (invisable) 12

16 Critique of solution name = debanks.txt with open(name, r ) as textfile : num lines = 0 for line in textfile : num lines += 1 print( File '%s' contains %i lines % (name, num lines)) Problems can occur if file can t be found file can t be opened (corrupted?) file can t be read (not text?) among others Our code will crash in such situations More robust approach to catch these; need exception handling later 13

17 File Writing

18 Python s other file modes r w a b read only write only append at end of file binary (for non-text files) 14

19 Writing to files Use file mode w when opening to write to a file Previous contents of file are erased Use write function to write strings to file file. write( Hello, World! ) Each write adds to the file after previously written stuff 15

20 Writing to files Application to generate file containing powers of natural numbers file = open( table of powers.txt, w ) LIMIT = 10 header = Table of Powers from 1 to %d\n % LIMIT bar = = len(header) + \n\n file. write(header) file. write(bar) for n in range(1, LIMIT + 1): file. write( %3d %5d %5d\n % (n, n 2, n 3)) 16

21 Writing to files cont d Final contents of table-of-powers.txt: Table of Powers from 1 to 10 =============================

22 Appending to a file Use file mode a to append to file def log(msg): # Append message 'msg' to the log file file = open( logfile. txt, a ) file. write(msg+ \n ) file. close () message = input( Enter message to be added to log: ) log(message) Each write adds an fresh line tacked on to the end of the existing file contents (i.e. does not erase old file contents) 18

23 File Demos

24 Simple file demos Application to print copy of a text file with line numbers prepended. file name = input( Please enter a file name: ) file = open(file name, r ) line num = 0 for line in file : line num = line num + 1 print( %3d %2d %s % (line num, len(line), line), end = ) 19

25 Illustration Text file debanks.txt contains: ===================================== The Banks of My Own Lovely Lee ===================================== How oft do my thoughts in their fancy take flight To the home of my childhood away, To the days when each patriot s vision seem d bright Ere I dreamed that those joys should decay. When my heart was as light as the wild winds that blow Down the Mardyke through each elm tree, Where I sported and play d neath each green leafy shade On the banks of my own lovely Lee. Program generates the following output: ===================================== The Banks of My Own Lovely Lee ===================================== How oft do my thoughts in their fancy take flight To the home of my childhood away, To the days when each patriot s vision seem d bright Ere I dreamed that those joys should decay When my heart was as light as the wild winds that blow Down the Mardyke through each elm tree, Where I sported and play d neath each green leafy shade 20

26 Simple pattern finding tool Develop a tool that hunts through a text file printing only those lines on which a particular pattern occurs. Idea (suppose pattern is P): Open file and read through it line by line For each line check if P occurs and if so print the line 21

27 Illustration Executing grep("debanks.txt", "the") produces: [l4, c26]: How oft do my thoughts in their fancy take flight [l5, c3 ]: To the home of my childhood away, [l6, c3 ]: To the days when each patriot s vision seem d brigh [l8, c30]: When my heart was as light as the wild winds that b [l9, c5 ]: Down the Mardyke through each elm tree, [l11, c3 ]: On the banks of my own lovely Lee. Have included line and column numbers for clarity. 22

28 Code def grep( filename, pattern ): file = open(filename, r ) line num = 0 for line in file : if line. find (pattern)!= 1: col num = line. index(pattern) print( [ l% 3d, c% 2d]: %s % (line num, col num, line[: line num = line num

29 Simple text analysis II

30 Text analysis revisited Write a Python program to read the contents of a text file and to output the number of lines, words and letters found therein. 24

31 Illustration ===================================== The Banks of My Own Lovely Lee ===================================== How oft do my thoughts in their fancy take flight To the home of my childhood away, To the days when each patriot s vision seem d bright Ere I dreamed that those joys should decay. When my heart was as light as the wild winds that blow Down the Mardyke through each elm tree, Where I sported and play d neath each green leafy shade On the banks of my own lovely Lee. Result: Number of lines, words, chars = 12, 80,

32 Text analysis revisited Use following string-manipulating helper functions purge to remove punctuation stats to analyze contents of a string Integrate with file-reading aparatus: Read through file line by line Use stats to calculated number of words and chars on line (Sum these as we go) 26

33 Text analysis revisited def depunctuated(text): # Return filtered version of ' text ' with all punctuation # marks removed.... def stats (text ): # Return the number of letters and words in string parameter # 'text '. Assume no words split across lines.... def word count(filename ): # Return the number of lines, words and letters in # the file named 'filename'... 27

34 Stripping out punctuation def depunctuated(text): # Return filtered version of ' text ' with all punctuation # marks removed. MARKS =.,;:?!\ '() dep = for c in text : if c not in MARKS: dep = dep + c return dep 28

35 Counting letters and words def stats (text ): # Return the number of letters and words in string parameter # 'text '. Assume no words split across lines. 29

36 Counting letters and words def stats (text ): # Return the number of letters and words in string parameter # 'text '. Assume no words split across lines. unpunctuated = depunctuated(text) letters = [x for x in unpunctuated if x. isalpha ()] num letters = len( letters ) 29

37 Counting letters and words def stats (text ): # Return the number of letters and words in string parameter # 'text '. Assume no words split across lines. unpunctuated = depunctuated(text) letters = [x for x in unpunctuated if x. isalpha ()] num letters = len( letters ) words = unpunctuated.split () num words = len(words) return ( num letters, num words) 29

38 Function wordcount def word count(filename ): # Return the number of lines, words and letters in # the file named 'filename' f = open(filename, r ) line count, word count, char count = 0, 0, 0 for line in f : c, w = stats( line ) line count = line count + 1 char count = char count + c word count = word count + w f. close () return line count, word count, char count 30

39 The main program num lines, num words, num chars = word count( debanks.txt ) print( Number of lines, words, chars = %d, %d, %d \ % (num lines, num words, num chars)) 31

40 Text censoring application

41 Censoring text Develop a function that takes the name of a text file and the produces a censored copy in which all the rude words have been bleeped out (with asterisks). 32

42 Illustration curses.txt Damn, blast and double damn! Fiddlesticks! FIDDLESTICKS!!! Fiddle-damn-sticks! Blasted, damned fiddlesticks. clean curses.txt ****, ***** and double ****! ************! ************!!! Fiddle-damn-sticks! Blasted, damned ************. 33

43 Outline def bleeped(word): # Return a bowlderized version of parameter 'word' with # any black listed word replaced by asterisks. Ignore any # non letters at the beginning or end of ' word'.... def censor line ( line ): # Return censored version of the text in string parameter # ' line ' while preserving word breaks and punctuation.... def censor file ( filename ): # Generate a modified copy of the named text file containing # a censored version of the text

44 def censor file ( filename ): # Generate a modified copy of the named text file containing # a censored version of the text. infile = open(filename, r ) outfile = open( clean +filename, w ) for line in infile : clean line = censor line ( line ) outfile. write( clean line ) 35

45 def censor line ( line ): # Return censored version of the text in string parameter # ' line ' while preserving word breaks and punctuation. words = line. split () for i in range(len(words)): words[ i ] = bleeped(words[i ]) return. join (words) + \n 36

46 def bleeped(word): # Return a bowlderized version of parameter 'word' with # any black listed word replaced by asterisks. Ignore any # non letters at the beginning or end of 'word'. # Set ' first ' and ' last ' to indices of leftmost and rightmost letter first = 0 while first < len(word) and not word[ first ]. isalpha (): first = first + 1 last = len(word) 1 while last > first and not word[ last ]. isalpha (): last = last 1 stem = word[ first : last + 1] prestem = word[: first ] poststem = word[last + 1:] if stem.lower() in BLACK LIST: stem = MASK len(stem) return prestem+stem+poststem 37

47 Binary Files

48 Binary files Binary files used for: images, audio, executable,... Python treats binary files as linear sequence of bytes Use same open-(read/write)-close idiom as text files Use f.read(n) for reading (n bytes) f.write(x) for writing (but x must contain byte values) 38

49 Simple GIF Checker Application to check if a file is a GIF. Note: GIF s have a distinctive header in their first four bytes. def is gif (fname): f = open(fname, br ) first4 = tuple(f.read(4)) gif header = (0x47, 0x49, 0x46, 0x38) return ( first4 == gif header) 39

50 Simple file comparer Write application to compare two file to seeof they are identical (byte for byte) Idea: read through files chunk by chunk, checking that the two chunks at each step match (byte for byte); any mismatch signifies that]files are not identical 40

51 Simple file comparer cont d def are equal (fname1, fname2): CHUNKSIZE = 1000 try: f1 = open(fname1, br ) f2 = open(fname2, br ) except IOError: print( Problems opening files '%s' or '%s'. % (fname1, fnam exit () # Read through f1, f2 chunk by chunk, checking that the chunks # match (byte for byte ); set Boolean 'mismatch' is if discrepency f1. close () f1. close () 41

52 Simple file comparer cont d bytes1 = f1.read(chunksize) bytes2 = f2.read(chunksize) mismatch = bytes1!= bytes2 while not mismatch and len(bytes1)!= 0 and len(bytes2)!= 0: bytes1 = f1.read(chunksize) bytes2 = f2.read(chunksize) mismatch = bytes1!= bytes2 42

53 Pickling

54 Pickling Pickling allows contents stored in a Python structure (list, distionary etc) to be stored in a file and to be restored/reconstitued later Useful to preverve data across sucessive prohgrame executions 43

55 Pickling demo To store prog. x, y, z: import pickle data in vars. To re-store prog. data back into vars. x, y, z: import pickle f = open( progstate, w ) pickle. dump(x,f) pickle. dump(y,f) pickle. dump(z,f) f. close () f = open( progstate, r ) x = pickle.load(f) y = pickle.load(f) z = pickle.load(f) f. close () 44

56 Back Material

57 Notes and Acknowledgements Reading Code Acknowledgements 45