TH IRD EDITION. Python Cookbook. David Beazley and Brian K. Jones. O'REILLY. Beijing Cambridge Farnham Köln Sebastopol Tokyo

Transcription

1 TH IRD EDITION Python Cookbook David Beazley and Brian K. Jones O'REILLY. Beijing Cambridge Farnham Köln Sebastopol Tokyo

2 Table of Contents Preface xi 1. Data Structures and Algorithms Unpacking a Sequence into Separate Variables Unpacking Elements from Iterables of Arbitrary Length Keeping the Last N Items Finding the Largest or Smallest N Items Implementing a Priority Queue Mapping Keys to Multiple Values in a Dictionary Keeping Dictionaries in Order Calculating with Dictionaries Finding Commonalities in Two Dictionaries Removing Duplicates from a Sequence while Maintaining Order Naming a Slice Determining the Most Frequently Occurring Items in a Sequence Sorting a List of Dictionaries by a Common Key Sorting Objects Without Native Comparison Support Grouping Records Together Based on a Field Filtering Sequence Elements Extracting a Subset of a Dictionary Mapping Names to Sequence Elements Transforming and Reducing Data at the Same Time Combining Multiple Mappings into a Single Mapping Strings and Text Splitting Strings on Any of Multiple Delimiters Matching Text at the Start or End of a String Matching Strings Using Shell Wildcard Patterns Matching and Searching for Text Patterns 42 iii

3 2.5. Searching and Replacing Text Searching and Replacing Case-Insensitive Text Specifying a Regular Expression for the Shortest Match Writing a Regular Expression for Multiline Patterns Normalizing Unicode Text to a Standard Representation Working with Unicode Characters in Regular Expressions Stripping Unwanted Characters from Strings Sanitizing and Cleaning Up Text Aligning Text Strings Combining and Concatenating Strings Interpolating Variables in Strings Reformatting Text to a Fixed Number of Columns Handling HTML and XML Entities in Text Tokenizing Text Writing a Simple Recursive Descent Parser Performing Text Operations on Byte Strings Numbers, Dates, and Times Rounding Numerical Values Performing Accurate Decimal Calculations Formatting Numbers for Output Working with Binary, Octal, and Hexadecimal Integers Packing and Unpacking Large Integers from Bytes Performing Complex-Valued Math Working with Infinity and NaNs Calculating with Fractions Calculating with Large Numerical Arrays Performing Matrix and Linear Algebra Calculations Picking Things at Random Converting Days to Seconds, and Other Basic Time Conversions Determining Last Friday's Date Finding the Date Range for the Current Month Converting Strings into Datetimes Manipulating Dates Involving Time Zones lterators and Generators Manually Consuming an Iterator Delegating Iteration Creating New Iteration Patterns with Generators Implementing the Iterator Protocol Iterating in Reverse Defining Generator Functions with Extra State 120 iv I Table of Contents

4 4.7. Taking a Slice of an Iterator Skipping the First Part of an Iterable Iterating Over All Possible Combinations or Permutations Iterating Over the Index-Value Pairs of a Sequence Iterating Over Multiple Sequences Simultaneously Iterating on Items in Separate Containers Creating Data Processing Pipelines Flattening a Nested Sequence Iterating in Sorted Order Over Merged Sorted Iterables Replacing Infinite while Loops with an Iterator Flies and I/O Reading and Writing Text Data Printing to a File Printing with a Different Separator or Line Ending Reading and Writing Binary Data Writing to a File That Doesn't Already Düst Performing I/O Operations on a String Reading and Writing Compressed Datafiles Iterating Over Fixed-Sized Records Reading Binary Data into a Mutable Buffer Memory Mapping Binary Files Manipulating Pathnames Testing for the Existence of a File Getting a Directory Listing Bypassing Filename Encoding Printing Bad Filenames Adding or Changing the Encoding of an Already Open File Writing Bytes to a Text File Wrapping an Existing File Descriptor As a File Object Making Temporary Files and Directories Communicating with Serial Ports Serializing Python Objects Data Encoding and Processing Reading and Writing CSV Data Reading and Writing JSON Data Parsing Simple XML Data Parsing Huge XML Files Incrementally Turning a Dictionary into XML Parsing, Modifying, and Rewriting XML Parsing XML Documents with Namespaces 193 Table ofcontents 1 v

5 6.8. Interacting with a Relational Database Decoding and Encoding Hexadecimal Digits Decoding and Encoding Base Reading and Writing Binary Arrays of Structures Reading Nested and Variable-Sized Binary Structures Summarizing Data and Performing Statistics Functions Writing Functions That Accept Any Number of Arguments Writing Functions That Only Accept Keyword Arguments Attaching Informational Metadata to Function Arguments Returning Multiple Values from a Function Defining Functions with Default Arguments Defining Anonymous or Inline Functions Capturing Variables in Anonymous Functions Making an N-Argument Callable Work As a Callable with Fewer Arguments Replacing Single Method Classes with Functions Carrying Extra State with Callback Functions Inlining Callback Functions Accessing Variables Defined Inside a Closure Classes and Objects Changing the String Representation of Instances Customizing String Formatting Making Objects Support the Context-Management Protocol Saving Memory When Creating a Large Number of Instances Encapsulating Names in a Class Creating Managed Attributes Calling a Method on a Parent Class Extending a Property in a Subclass Creating a New Kind of Class or Instance Attribute Using Lazily Computed Properties Simplifying the Initialization of Data Structures Defining an Interface or Abstract Base Class Implementing a Data Model or Type System Implementing Custom Containers Delegating Attribute Access Defining More Than One Constructor in a Class Creating an Instance Without Invoking mit Extending Classes with Mixins Implementing Stateful Objects or State Machines 299 vi J Table of Contents

6 8.20. Calling a Method on an Object Given the Name As a String Implementing the Visitor Pattern Implementing the Visitor Pattern Without Recursion Managing Memory in Cyclic Data Structures Making Classes Support Comparison Operations Creating Cached Instances Metaprogramming Putting a Wrapper Around a Function Preserving Function Metadata When Writing Decorators Unwrapping a Decorator Defining a Decorator That Takes Arguments Defining a Decorator with User Adjustable Attributes Defining a Decorator That Takes an Optional Argument Enforcing Type Checicing on a Function Using a Decorator Defining Decorators As Part of a Class Defining Decorators As Classes Applying Decorators to Class and Static Methods Writing Decorators That Add Arguments to Wrapped Functions Using Decorators to Patch Class Definitions Using a Metaclass to Control Instance Creation Capturing Class Attribute Definition Order Defining a Metaclass That Takes Optional Arguments Enforcing an Argument Signature on *args and **kwargs Enforcing Coding Conventions in Classes Defining Classes Programmatically Initializing Class Members at Definition Time Implementing Multiple Dispatch with Function Annotations Avoiding Repetitive Property Methods Defining Context Managers the Easy Way Executing Code with Local Side Effects Parsing and Analyzing Python Source Disassembling Python Byte Code Modules and Packages Making a Hierarchical Package of Modules Controlling the Import of Everything Importing Package Submodules Using Relative Names Splitting a Module into Multiple Files Making Separate Directories of Code Import Under a Common Namespace Reloading Modules 406 Table of Contents I vii

7 10.7. Making a Directory or Zip File Runnable As a Main Script Reading Datafiles Within a Package Adding Directories to sys.path Importing Modules Using a Name Given in a String Loading Modules from a Remote Machine Using Import Hooks Patching Modules on Import Installing Packages Just for Yourself Creating a New Python Environment Distributing Packages Network and Web Programming Interacting with HTTP Services As a Client Creating a TCP Server Creating a UDP Server Generating a Range of IP Addresses from a CIDR Address Creating a Simple REST-Based Interface Implementing a Simple Remote Procedure Call with XML-RPC Communicating Simply Between Interpreters Implementing Remote Procedure Calls Authenticating Clients Simply Adding SSL to Network Services Passing a Socket File Descriptor Between Processes Understanding Event-Driven I/O Sending and Receiving Large Arrays Concurrency Starting and Stopping Threads Determining If a Thread Has Started Communicating Between Threads Locking Critical Sections Locking with Deadlock Avoidance Storing Thread-Specific State Creating a Thread Pool Performing Simple Parallel Programming Dealing with the GIL (and How to Stop Worrying About It) Defining an Actor Task Implementing Publish/Subscribe Messaging Using Generators As an Alternative to Threads Polling Multiple Thread Queues Launching a Daemon Process on Unix Utility Scripting and System Administration 539 viii I Table of Contents

8 13.1. Accepting Script Input via Redirection, Pipes, or Input Files Terminating a Program with an Error Message Parsing Command-Line Options Prompting for a Password at Runtime Getting the Terminal Size Executing an External Command and Getting Its Output Copying or Moving Files and Directories Creating and Unpacking Archives Finding Files by Name Reading Configuration Files Adding Logging to Simple Scripts Adding Logging to Libraries Malcing a Stopwatch Timer Putting Limits on Memory and CPU Usage Launching a Web Browser Testing, Debugging, and Exceptions Testing Output Sent to stdout Patching Objects in Unit Tests Testing for Exceptional Conditions in Unit Tests Logging Test Output to a File Skipping or Anticipating Test Failures Handling Multiple E,xceptions Catching All Exceptions Creating Custom Exceptions Raising an Exception in Response to Another Exception Reraising the Last Exception Issuing Warning Messages Debugging Basic Program Crashes Profiling and Timing Your Program Malcing Your Programs Run Faster C Extensions Accessing C Code Using ctypes Writing a Simple C Extension Module Writing an Extension Function That Operates on Arrays Managing Opaque Pointers in C Extension Modules Defining and Exporting C APIs from Extension Modules Calling Python from C Releasing the GIL in C E,xtensions Mixing Threads from C and Python Wrapping C Code with Swig 627 Table of Contents I ix

9 Wrapping Existing C Code with Cython Using Cython to Write High-Performance Array Operations Turning a Function Pointer into a Callable Passing NULL-Terminated Strings to C Libraries Passing Unicode Strings to C Libraries Converting C Strings to Python Working with C Strings of Dubious Encoding Passing Filenames to C Extensions Passing Open Files to C Extensions Reading File-Like Objects from C Consuming an Iterable from C Diagnosing Segmentation Faults 663 A. Further Reading 665 Index 667 x I Table of Contents