Pro Python. Marty Alchin

Transcription

1 Pro Python Marty Alchin

2 Pro Python Copyright 2010 by Marty Alchin All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13 (pbk): ISBN-13 (electronic): Printed and bound in the United States of America Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. President and Publisher: Paul Manning Lead Editors: Duncan Parkes, Tom Welsh Technical Reviewer: George Vilches Editorial Board: Clay Andres, Steve Anglin, Mark Beckner, Ewan Buckingham, Gary Cornell, Jonathan Gennick, Jonathan Hassell, Michelle Lowman, Matthew Moodie, Duncan Parkes, Jeffrey Pepper, Frank Pohlmann, Douglas Pundick, Ben Renow-Clarke, Dominic Shakeshaft, Matt Wade, Tom Welsh Coordinating Editor: Mary Tobin Copy Editors: Nancy Sixsmith, Angel Alchin Compositor: Bytheway Publishing Services Indexer: John Collin Cover Designer: Anna Ishchenko Distributed to the book trade worldwide by Springer Science+Business Media, LLC., 233 Spring Street, 6th Floor, New York, NY Phone SPRINGER, fax (201) , orders-ny@springersbm.com, or visit For information on translations, please rights@apress.com, or visit Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. ebook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales ebook Licensing web page at The information in this book is distributed on an as is basis, without warranty. Although every precaution has been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work. The source code for this book is available to readers at You will need to answer questions pertaining to this book in order to successfully download the code.

3 Contents at a Glance Contents...iv About the Author... xvi About the Technical Reviewer... xvii Acknowledgments... xviii Introduction... xix Chapter 1: Principles and Philosophy...1 Chapter 2: Advanced Basics...19 Chapter 3: Functions...53 Chapter 4: Classes Chapter 5: Common Protocols Chapter 6: Object Management Chapter 7: Strings Chapter 8: Documentation Chapter 9: Testing Chapter 10: Distribution Chapter 11: Sheets: A CSV Framework PEP 8: Style Guide for Python PEP 10: Voting Guidelines PEP 20: The Zen of Python PEP 257: Docstring Conventions PEP 387: Backwards Compatibility Policy PEP 3000: Python PEP 3003: Python Language Moratorium Index iii

4 Contents Contents...iv About the Author... xvi About the Technical Reviewer... xvii Acknowledgments... xviii Introduction... xix Chapter 1: Principles and Philosophy...1 The Zen of Python...1 Beautiful Is Better Than Ugly... 2 Explicit Is Better Than Implicit... 2 Simple Is Better Than Complex... 3 Complex Is Better Than Complicated... 3 Flat Is Better Than Nested... 4 Sparse Is Better Than Dense... 5 Readability Counts... 5 Special Cases Aren t Special Enough to Break the Rules... 6 Although Practicality Beats Purity... 6 Errors Should Never Pass Silently... 7 Unless Explicitly Silenced... 8 In the Face of Ambiguity, Refuse the Temptation to Guess... 9 There Should Be One and Preferably Only One Obvious Way to Do It Although That Way May Not Be Obvious at First Unless You re Dutch Now Is Better Than Never iv

5 Although Never Is Often Better Than Right Now If the Implementation is Hard to Explain, It s a Bad Idea If the Implementation is Easy to Explain, It May Be a Good Idea Namespaces Are One Honking Great Idea Let s Do More of Those! Don t Repeat Yourself...12 Loose Coupling...13 The Samurai Principle...13 The Pareto Principle...14 The Robustness Principle...14 Backward Compatibility...15 The Road to Python Taking It With You...17 Chapter 2: Advanced Basics...19 General Concepts...19 Iteration Caching Transparency Control Flow...21 Catching Exceptions Exception Chains When Everything Goes Right Proceeding Regardless of Exceptions Optimizing Loops The with Statement Conditional Expressions Iteration...33 Sequence Unpacking List Comprehensions v

6 Generator Expressions Set Comprehensions Dictionary Comprehensions Chaining Iterables Together Zipping Iterables Together Collections...39 Sets Named Tuples Ordered Dictionaries Dictionaries with Defaults Importing Code...45 Fallback Imports Importing from the Future Using all to Customize Imports Relative Imports The import () function The importlib module Taking It With You...52 Chapter 3: Functions...53 Arguments...53 Planning for Flexibility Variable Positional Arguments Variable Keyword Arguments Combining Different Kinds of Arguments Invoking Functions with Variable Arguments Preloading Arguments Introspection Example: Identifying Argument Values Example: A More Concise Version vi

7 Example: Validating Arguments Decorators...67 Closures Wrappers Decorators with Arguments Decorators with or without Arguments...74 Example: Memoization Example: A Decorator to Create Decorators Function Annotations...78 Example: Type Safety Factoring Out the Boilerplate Example: Type Coercion Annotating with Decorators Example: Type Safety as a Decorator Generators...94 Lambdas...96 Introspection...97 Identifying Object Types Modules and Packages Docstrings Taking It With You Chapter 4: Classes Inheritance Multiple Inheritance Method Resolution Order (MRO) Example: C3 Algorithm Using super() to Pass Control to Other Classes Introspection vii

8 How Classes Are Created Creating Classes at Runtime Metaclasses Example: Plugin Framework Controlling the Namespace Attributes Properties Descriptors Methods Unbound Methods Bound Methods Class Methods Static Methods Assigning Functions to Classes and Instances Magic Methods Creating Instances Example: Automatic Subclasses Dealing with Attributes String Representations Taking It With You Chapter 5: Common Protocols Basic Operations Mathematical Operations Bitwise Operations Variations Numbers Sign Operations Comparison Operations viii

9 Iterables Example: Repeatable Generators Sequences Mappings Callables Context Managers Taking It With You Chapter 6: Object Management Namespace Dictionary Example: Borg Pattern Example: Self-caching properties Garbage Collection Reference Counting Cyclical References Weak References Pickling Copying Shallow Copies Deep Copies Taking It With You Chapter 7: Strings Bytes Simple Conversion: chr() and ord() Complex Conversion: The Struct Module Text Unicode Encodings ix

10 Simple Substitution Formatting Looking Up Values Within Objects Distinguishing Types of Strings Standard Format Specification Example: Plain Text Table of Contents Custom Format Specification Taking It With You Chapter 8: Documentation Proper Naming Comments Docstrings Describe What the Function Does Explain the Arguments Don t Forget the Return Value Include Any Expected Exceptions Documentation Outside the Code Installation and Configuration Tutorials Reference Documents Documentation Utilities Formatting Links Sphinx Taking It With You Chapter 9: Testing Test-Driven Development (TDD) Doctests x

11 Formatting Code Representing Output Integrating With Documentation Running Tests The unittest module Setting Up Writing Tests Other Comparisons Testing Strings and Other Sequence Content Testing Exceptions Testing Identity Tearing Down Providing a Custom Test Class Changing Test Behavior Taking It With You Chapter 10: Distribution Licensing GNU General Public License (GPL) Affero General Public License (AGPL) GNU Lesser General Public License (LGPL) Berkeley Software Distribution (BSD) License Other Licenses Packaging setup.py MANIFEST.in The sdist command Distribution Taking It With You xi

12 Chapter 11: Sheets: A CSV Framework Building a Declarative Framework Introducing Declarative Programming To Build or Not to Build? Building the Framework Managing Options Defining Fields Attaching a Field to a Class Adding a Metaclass Bringing It Together Ordering Fields DeclarativeMeta. prepare () Column. init () Column. new () CounterMeta. call () Choosing an Option Building a Field Library StringField IntegerColumn FloatColumn DecimalColumn DateColumn Getting Back to CSV Checking Arguments Populating Values The Reader The Writer Taking It With You xii

13 PEP 8: Style Guide for Python Introduction A Foolish Consistency is the Hobgoblin of Little Minds Code Layout Indentation Tabs or Spaces? Maximum Line Length Blank Lines Encodings (PEP 263) Imports Whitespace in Expressions and Statements Pet Peeves Other Recommendations Comments Block Comments Inline Comments Documentation Strings Version Bookkeeping Naming Conventions Descriptive: Naming Styles Prescriptive: Naming Conventions Programming Recommendations Copyright PEP 10: Voting Guidelines Abstract Rationale Voting Scores xiii

14 Copyright PEP 20: The Zen of Python Abstract The Zen of Python Easter Egg Copyright PEP 257: Docstring Conventions Abstract Rationale Specification What is a Docstring? One-Line Docstrings Multi-Line Docstrings Handling Docstring Indentation Copyright Acknowledgments PEP 387: Backwards Compatibility Policy Abstract Rationale Backwards Compatibility Rules Making Incompatible Changes Copyright PEP 3000: Python Abstract Naming PEP Numbering xiv

15 Timeline Compatibility and Transition Implementation Language Meta-Contributions Copyright PEP 3003: Python Language Moratorium Abstract Rationale Details Cannot Change Case-by-Case Exemptions Allowed to Change Retroactive Extensions Copyright Index xv

16 About the Author Marty Alchin is a professional programmer with a passion for the Web. His work with Django, the popular Web framework, led him to write several articles about Python, to speak at PyCon and even to write his first book, Pro Django, which was published by Apress in December of In addition to writing for print, Marty keeps a blog at where he writes about Python, Django and anything else that strikes his fancy. xvi

17 About the Technical Reviewer George Vilches is a software engineer and systems administrator with an unabashed fondness for Python and the Web in both disciplines. In the last three years, he has made several contributions to Django, with a focus on the ORM and administrative side of things. He is a principal engineer with AOL and builds Django applications with Fortune Cookie Studios ( George s personal time is split between tinkering with open source projects and enjoying the company of his wife Kate, their corgi and their two cats (all of whom would prefer he stop tinkering and attend to them more). xvii

18 INTRODUCTION Acknowledgments I wouldn t have even started this project if not for the endless encouragement from my lovely wife, Angel. She s been my sounding board, my task manager, my copy editor and my own personal cheerleader. There s no way I could do anything like this without her help and support. I d also like to thank my technical reviewer, George, for everything he s done to help me out. He s gone above and beyond the limits of his role, helping with everything from code to grammar and even a good bit of style. After enjoying his help on Pro Django, I wouldn t have even signed on for another book without him by my side. Lastly, I never would ve considered a book like this if not for the wonderful community around Python. The willingness of Python programmers to open their minds and their code is, I believe, unrivaled among our peers. It s this spirit of openness that encourages me every day, leading me to discover new things and push myself beyond the limits of what I knew yesterday. We learn by doing and by seeing what others have done. I hope that you ll take the contents of this book and do more with it than what I ve done. There s no better reward for all this hard work than to see better programmers writing better code. xviii

19 Introduction When I wrote my first book, Pro Django, I didn t have much of an idea what my readers would find interesting. I had gained a lot of information I thought would be useful for others to learn, but I didn t really know what would be the most valuable thing they d take away. As it turned out, in nearly 300 pages, the most popular chapter in the book barely mentioned Django at all. It was about Python. The response was overwhelming. There was clearly a desire to learn more about how to go from a simple Python application to a detailed framework like Django. It s all Python code, but it can be hard to understand based on even a reasonably thorough understanding of the language. The tools and techniques involved require some extra knowledge that you might not run into in general use. This gave me a new goal with Pro Python: to take you from proficient to professional. Being a true professional requires more experience than you can get from a book, but I want to at least give you the tools you ll need. Combined with the rich philosophy of the Python community, you ll find plenty of information to take your code to the next level. Who This Book Is For Because my goal is to bring intermediate programmers to a more advanced level, I wrote this book with the expectation that you ll already be familiar with Python. You should be comfortable using the interactive interpreter, writing control structures and a basic object-oriented approach. That s not a very difficult prerequisite. If you ve tried your hand at writing a Python application even if you haven t released it into the wild, or even finished it you likely have all the necessary knowledge to get started. The rest of the information you ll need is contained in these pages. What You ll Need This book is written with the latest versions of Python in mind, so most of the examples assume that you re already using Python 3.1, which is the latest official release as of the date of publishing. I don t take the jump to Python 3 lightly, though, so there are plenty of compatibility notes along the way, going all the way back to Python 2.5. As long as your copy of Python was released in the last few years, you ll be all set. Nearly all the packages used in this book come from the Python Standard Library, which ships with every Python installation. Some sections will reference third-party libraries that aren t included in that bundle, but those are strictly informative; you won t lose out if you don t have them installed. Source Code The code for all the examples in this book is available at xix