streamio Documentation

Transcription

1 streamio Documentation Release dev James Mills April 17, 2014

2

3 Contents 1 About Examples Requirements Installation Supported Platforms Documentation API Documentation TODO Road Map Changes Indices and tables 11 Python Module Index 13 i

4 ii

5 Release dev Date February 26, 2014 Contents 1

6 2 Contents

7 CHAPTER 1 About streamio is a simple library of functions designed to read, write and sort large files using iterators so that the operations will successfully complete on systems with limited RAM. This library has been used extensively at Griffith University whilst developing the TerraNova Climate Change and Adaptation Visualization tool(s) and processing large volumes of data. streamio is written in Python and has extensive documentation and unit tests with 100% coverage. See the API for a list of the available functions. Visit the Project Website Read the Docs Download it from the Downloads Page 1.1 Examples Read a large text file iteratively: from streamio import stream f = stream("large_file.txt") Read a large CSV file iteratively: from streamio import jsonstream f = stream("large_file.json") Merge-sort a large JSON file with the key itemgetter("value"): from operator import itemgetter from streamio import mergesort f = mergesort("large_file.json", key=itemgetter("value")) 1.2 Requirements funcy py streamio also comes with documentation and a full comprehensive unit test suite which require the following: To build the docs: 3

8 sphinx To run the unit tests: pytest 1.3 Installation The simplest and recommended way to install streamio is with pip. You may install the latest stable release from PyPI with pip: > pip install streamio If you do not have pip, you may use easy_install: > easy_install streamio Alternatively, you may download the source package from the PyPI Page or the Downloads page on the Project Website; extract it and install using: > python setup.py install You can also install the latest-development version by using pip or easy_install: > pip install streamio==dev or: > easy_install streamio==dev For further information see the streamio documentation. 1.4 Supported Platforms Linux, FreeBSD, Mac OS X Python 2.7 PyPy 2.2 Windows: We acknowledge that Windows exists and make reasonable efforts to maintain compatibility. Unfortunately we cannot guarantee support at this time. 4 Chapter 1. About

9 CHAPTER 2 Documentation 2.1 API Documentation streamio streamio package Submodules streamio.sort module sort streamio.sort.merge(*iterables, **kwargs) Take a list of ordered iterables; return as a single ordered generator. key function, for each item return key value Directly borrowed from: streamio.sort.mergesort(filename, output=none, key=none, maxitems= , progress=true) Given an input file sort it by performing a merge sort on disk. filename (str or py._path.local.localpath) Either a filename as a str or a py._path.local.localpath instance. output (str or py._path.local.localpath or None) An optional output filename as a str or a py._path.local.localpath instance. key (function or None) An optional key to sort the data on. maxitems (int) Maximum number of items to hold in memory at a time. progress (bool) Whether or not to display a progress bar This uses py._path.local.localpath.make_numbered_dir to create temporry scratch space to work with when splitting the input file into sorted chunks. The mergesort is processed iteratively in-memory using the ~merge function which is almost identical to ~heapq.merge but adds in the support of an optional key function. 5

10 streamio.stat module stat streamio.stat.minmax(xs) Return the min and max values for the given iterable xs (Any iterable of single numerical values.) An iterable of values This function returns both the min and max of the given iterable by computing both at once and iterating/consuming the iterable once. streamio.stream module stream streamio.stream.stream(filename, encoding= utf-8, skipblank=true, strip=true, stripchars= \r\n\t ) Stream every line in the given file. encoding (str or None) A str indicating the charset/encoding to use. filename (str, py._path.local.localpath or file.) A str filename, A py._path.local.localpath instance or open file instnace. skipblank (bool) Whehter to skip blank lines (sometimes undesirable) strip (bool) Whehter to strip lines of surrounding whitespace (sometimes undesirable) stripchars (list, tuple or str) An iterable of characters to strip from the surrounding line. line.strip(...) is used. Each line in the file is read, stripped of surrounding whitespace and returned iteratively. Blank lines are ignored. If they keyword argument encoding is provided and is not None each line in the input strema will be decoded using the given encoding, if None will disable unicode decoding. streamio.stream.csvstream(filename, encoding= utf-8, stripchars= \r\n ) Stream every line in the given file interpreting each line as CSV. filename (str, py._path.local.localpath or file.) A str filename, A py._path.local.localpath instance or open file instnace. encoding (str) A str indicating the charset/encoding to use. stripchars (list, tuple or str) An iterable of characters to strip from the surrounding line. line.strip(...) is used. This is a wrapper around stream where the stream is treated as CSV. streamio.stream.csvdictstream(filename, encoding= utf-8, fields=none, stripchars= \r\n ) Stream every line in the given file interpreting each line as a dictionary of fields to items. filename (str, py._path.local.localpath or file.) A str filename, A py._path.local.localpath instance or open file instnace. encoding (str) A str indicating the charset/encoding to use. 6 Chapter 2. Documentation

11 stripchars (list, tuple or str) An iterable of characters to strip from the surrounding line. line.strip(...) is used. This is a wrapper around csvstream where the stream is treated as dict of field(s) to item(s). streamio.stream.jsonstream(filename, encoding= utf-8 ) Stream every line in the given file interpreting each line as JSON. filename (str, py._path.local.localpath or file.) A str filename, A py._path.local.localpath instance or open file instnace. encoding (str) A str indicating the charset/encoding to use. stripchars (list, tuple or str) An iterable of characters to strip from the surrounding line. line.strip(...) is used. This is a wrappedaround stream except that it wraps each line in a dumps call essentially treating each line as a piece of valid JSON. streamio.stream.compress(iterable, level=9, encoding= utf-8 ) Compress the given iterable of bytes using zlib compressin iterable (An iterable of bytes (If str will be encoded)) An iterable of bytes to compress using zlib (ZIP) level (int (Default: 9)) An optional Compression Level encoding (str (Default: utf-8)) An optional encoding to use when dealing with an iterable of str Returns An iterable compressed with zlib Return type iterable stream of bytes streamio.version module Version Module So we only have to maintain version information in one place! Module contents streamio - reading, writing and sorting large files. streamio is a simple library of functions designed to read, write and sort large files using iterators so that the operations will successfully complete on systems with limited RAM. copyright CopyRight (C) 2013 by James Mills streamio.minmax(xs) Return the min and max values for the given iterable xs (Any iterable of single numerical values.) An iterable of values This function returns both the min and max of the given iterable by computing both at once and iterating/consuming the iterable once. streamio.merge(*iterables, **kwargs) Take a list of ordered iterables; return as a single ordered generator API Documentation 7

12 key function, for each item return key value Directly borrowed from: streamio.mergesort(filename, output=none, key=none, maxitems= , progress=true) Given an input file sort it by performing a merge sort on disk. filename (str or py._path.local.localpath) Either a filename as a str or a py._path.local.localpath instance. output (str or py._path.local.localpath or None) An optional output filename as a str or a py._path.local.localpath instance. key (function or None) An optional key to sort the data on. maxitems (int) Maximum number of items to hold in memory at a time. progress (bool) Whether or not to display a progress bar This uses py._path.local.localpath.make_numbered_dir to create temporry scratch space to work with when splitting the input file into sorted chunks. The mergesort is processed iteratively in-memory using the ~merge function which is almost identical to ~heapq.merge but adds in the support of an optional key function. streamio.stream(filename, encoding= utf-8, skipblank=true, strip=true, stripchars= \r\n\t ) Stream every line in the given file. encoding (str or None) A str indicating the charset/encoding to use. filename (str, py._path.local.localpath or file.) A str filename, A py._path.local.localpath instance or open file instnace. skipblank (bool) Whehter to skip blank lines (sometimes undesirable) strip (bool) Whehter to strip lines of surrounding whitespace (sometimes undesirable) stripchars (list, tuple or str) An iterable of characters to strip from the surrounding line. line.strip(...) is used. Each line in the file is read, stripped of surrounding whitespace and returned iteratively. Blank lines are ignored. If they keyword argument encoding is provided and is not None each line in the input strema will be decoded using the given encoding, if None will disable unicode decoding. streamio.csvstream(filename, encoding= utf-8, stripchars= \r\n ) Stream every line in the given file interpreting each line as CSV. filename (str, py._path.local.localpath or file.) A str filename, A py._path.local.localpath instance or open file instnace. encoding (str) A str indicating the charset/encoding to use. stripchars (list, tuple or str) An iterable of characters to strip from the surrounding line. line.strip(...) is used. This is a wrapper around stream where the stream is treated as CSV. streamio.jsonstream(filename, encoding= utf-8 ) Stream every line in the given file interpreting each line as JSON. 8 Chapter 2. Documentation

13 filename (str, py._path.local.localpath or file.) A str filename, A py._path.local.localpath instance or open file instnace. encoding (str) A str indicating the charset/encoding to use. stripchars (list, tuple or str) An iterable of characters to strip from the surrounding line. line.strip(...) is used. This is a wrappedaround stream except that it wraps each line in a dumps call essentially treating each line as a piece of valid JSON. streamio.csvdictstream(filename, encoding= utf-8, fields=none, stripchars= \r\n ) Stream every line in the given file interpreting each line as a dictionary of fields to items. filename (str, py._path.local.localpath or file.) A str filename, A py._path.local.localpath instance or open file instnace. encoding (str) A str indicating the charset/encoding to use. stripchars (list, tuple or str) An iterable of characters to strip from the surrounding line. line.strip(...) is used. This is a wrapper around csvstream where the stream is treated as dict of field(s) to item(s). streamio.compress(iterable, level=9, encoding= utf-8 ) Compress the given iterable of bytes using zlib compressin iterable (An iterable of bytes (If str will be encoded)) An iterable of bytes to compress using zlib (ZIP) level (int (Default: 9)) An optional Compression Level encoding (str (Default: utf-8)) An optional encoding to use when dealing with an iterable of str Returns An iterable compressed with zlib Return type iterable stream of bytes 2.2 TODO Nothing planned at this stage. See also: Road Map No immediate future plans at this stage... Feel free to Create an Issue TODO 9

14 2.4 Changes streamio dev streamio ( ) Bumped py > streamio ( ) Fixed Unicode support for csvstream adding unicodecsv as a new dependency streamio ( ) Forgot to include required README.rst and CHANGES.rst streamio ( ) Don t include fabfile and tests (for now) in the resulting egg Make mergesort show progress optionally via progress=true streamio ( ) Added encoding parameter support to *stream() function(s) streamio ( ) Added streamio.stream.compress function for compressing an iterable using zlib compression. Loosended requirements on py and progress streamio ( ) Fixed broken link to the API Docs streamio ( ) Added some examples Fixed a few typos streamio ( ) Initial Public Release 10 Chapter 2. Documentation

15 CHAPTER 3 Indices and tables genindex modindex search 11

16 12 Chapter 3. Indices and tables

17 Python Module Index s streamio, 7 streamio.sort, 5 streamio.stat, 6 streamio.stream, 6 streamio.version, 7 13