Fluent Python. Luciano Ramalho. Tokyo O'REILLY. Beijing. Sebastopol. Farnham. Boston

Transcription

1 Fluent Python Luciano Ramalho Beijing Boston Farnham Sebastopol Tokyo O'REILLY

2 Table of Contents Preface xv Part I. Prologue 1. The Python Data Model 3 A Pythonic Card Deck 4 How Special Methods Are Used 8 Emulating Numeric Types 9 String Representation 11 Arithmetic Operators 12 Boolean Value of a Custom Type 12 Overview of Special Methods 13 Why len Is Not a Method 14 Chapter Summary 14 Further Reading 15 Part II. Data Structures 2. An Array of Sequences 19 Overview of Built-in Sequences 20 List Comprehensions and Generator Expressions 21 List Comprehensions and Readability 21 Listcomps Versus map and filter 23 Cartesian Products 23 Generator Expressions 25 Tuples Are Not Just Immutable Lists 26 Tuples as Records 26 Tuple Unpacking 27 v

3 Nested Tuple Unpacking 29 Named Tuples 30 Tuples as Immutable Lists 32 Slicing 33 Why Slices and Range Exclude the Last Item 33 Slice Objects 34 Multidimensional Slicing and Ellipsis 35 Assigning to Slices 36 Using + and * with Sequences 36 Building Lists of Lists 37 Augmented Assignment with Sequences 38 A += Assignment Puzzler 40 list.sort and the sorted Built-in Function 42 Managing Ordered Sequences with bisect 44 Searching with bisect 44 Inserting with bisect.insort 47 When a List Is Not the Answer 48 Arrays 48 Memory Views 51 NumPy and SciPy 52 Deques and Other Queues 55 Chapter Summary 57 Further Reading Dictionaries and Sets 63 Generic Mapping Types 64 diet Comprehensions 66 Overview of Common Mapping Methods 66 Handling Missing Keys with setdefault 68 Mappings with Flexible Key Lookup 70 defaultdict: Another Take on Missing Keys 70 The missing Method 72 Variations of diet 75 Subclassing UserDict 76 Immutable Mappings 77 Set Theory 79 set Literals 80 Set Comprehensions 81 Set Operations 82 diet and set Under the Hood 85 A Performance Experiment 85 Hash Tables in Dictionaries 87 vi Table of Contents

4 Practical Consequences of How diet Works 90 How Sets Work Practical Consequences 93 Chapter Summary 93 Further Reading Text versus Bytes 97 Character Issues 98 Byte Essentials 99 Structs and Memory Views 102 Basic Encoders/Decoders 103 Understanding Encode/Decode Problems 105 Coping with UnicodeEncodeError 105 Coping with UnicodeDecodeError 106 SyntaxError When Loading Modules with Unexpected Encoding 108 How to Discover the Encoding of a Byte Sequence 109 BOM: A Useful Gremlin 110 Handling Text Files 111 Encoding Defaults: A Madhouse 114 Normalizing Unicode for Saner Comparisons 117 Case Folding 119 Utility Functions for Normalized Text Matching 120 Extreme "Normalization": Taking Out Diacritics 121 Sorting Unicode Text 124 Sorting with the Unicode Collation Algorithm 126 The Unicode Database 127 Dual-Mode str and bytes APIs 129 str Versus bytes in Regular Expressions 129 str Versus bytes on os Functions 130 Chapter Summary 132 Further Reading 133 Part III. Functions as Objects 5. First-Class Functions 141 Treating a Function Like an Object 142 Higher-Order Functions 143 Modern Replacements for map, filter, and reduce 144 Anonymous Functions 145 The Seven Flavors of Callable Objects 146 User-Defined Callable Types 147 Function Introspection 148 Table ofcontents vii

5 From Positional to Keyword-Only Parameters 150 Retrieving Information About Parameters 152 Function Annotations 156 Packages for Functional Programming 158 The operator Module 158 Freezing Arguments with functools.partial 161 Chapter Summary 163 Further Reading Design Patterns with First-Class Functions 169 Case Study: Refactoring Strategy 170 Classic Strategy 170 Function-Oriented Strategy 174 Choosing the Best Strategy: Simple Approach 177 Finding Strategies in a Module 178 Command 179 Chapter Summary 181 Further Reading Function Decorators and Closures 185 Decorators When Python Executes Decorators 187 Decorator-Enhanced Strategy Pattern 189 Variable Scope Rules 191 Closures 194 The nonlocal Declaration 197 Implementing a Simple Decorator 198 How It Works 200 Decorators in the Standard Library 201 Memoization with functools.lru_cache 202 Generic Functions with Single Dispatch 204 Stacked Decorators 207 Parameterized Decorators 208 A Parameterized Registration Decorator 208 The Parameterized Clock Decorator 211 Chapter Summary 213 Further Reading 214 viii Table of Contents

6 Part IV. Object-Oriented Idioms 8. Object References, Mutability, and Recycling 221 Variables Are Not Boxes 222 Identity, Equality, and Aliases 223 Choosing Between == and is 225 The Relative Immutability oftuples 226 Copies Are Shallow by Default 227 Deep and Shallow Copies of Arbitrary Objects 230 Function Parameters as References 231 Mutable Types as Parameter Defaults: Bad Idea 232 Defensive Programming with Mutable Parameters 234 del and Garbage Collection 236 Weak References 238 The WeakValueDictionary Skit 240 Limitations ofweak References 241 Tricks Python Plays with Immutables 242 Chapter Summary 244 Further Reading A Pythonic Object 249 Object Representations 250 Vector Class Redux 250 An Alternative Constructor 253 classmethod Versus staticmethod 254 Formatted Displays 255 A Hashable Vector2d 259 Private and "Protected" Attributes in Python 264 Saving Space with the slots Class Attribute 267 The Problems with _slots_ 269 Overriding Class Attributes 269 Chapter Summary 272 Further Reading Sequence Hacking, Hashing, and Slicing 277 Vector: A User-Defined Sequence Type 278 Vector Take # 1: Vector2d Compatible 278 Protocols and Duck Typing 281 Vector Take #2: A Sliceable Sequence 282 How Slicing Works 283 A Slice-Aware getitem 285 Vector Take #3: Dynamic Attribute Access 286 Table of Contents ix

7 Vector Take #4: Hashing and a Faster == 290 Vector Take #5: Formatting 296 Chapter Summary 303 Further Reading Interfaces: From Protocols to ABCs 309 Interfaces and Protocols in Python Culture 310 Python Digs Sequences 312 Monkey-Patching to Implement a Protocol at Runtime 314 Alex Martellis Waterfowl 316 Subclassing an ABC 321 ABCs in the Standard Library 323 ABCs in collections.abc 323 The Numbers Tower ofabcs 325 Defining and Using an ABC 326 ABC Syntax Details 330 Subclassing the Tombola ABC 331 A Virtual Subclass oftombola 334 How the Tombola Subclasses Were Tested 337 Usage of register in Practice 340 Geese Can Behave as Ducks 340 Chapter Summary 342 Further Reading Inheritance: For Good or For Worse 351 Subclassing Built-in Types Is Tricky 352 Multiple Inheritance and Method Resolution Order 355 Multiple Inheritance in the Real World 360 Coping with Multiple Inheritance Distinguish Interface Inheritance from Implementation Inheritance Make Interfaces Explicit with ABCs Use Mixins for Code Reuse Make Mixins Explicit by Naming An ABC May Also Be a Mixin; The Reverse Is Not True Don't Subclass from More Than One Concrete Class Provide Aggregate Classes to Users "Favor Object Composition Over Class Inheritance." 365 Tkinter: The Good, the Bad, and the Ugly 365 A Modern Example: Mixins in Django Generic Views 366 Chapter Summary 370 Further Reading 371 x Table ofcontents

8 13. Operator Overloading: Doing It Right 375 Operator Overloading Unary Operators 376 Overloading + for Vector Addition 379 Overloading * for Scalar Multiplication 384 Rich Comparison Operators 388 Augmented Assignment Operators 392 Chapter Summary 397 Further Reading 398 Part V. Control Flow 14. Iterables, Iterators, and Generators 405 Sentence Take # 1: A Sequence ofwords 406 Why Sequences Are Iterable: The iter Function 408 Iterables Versus Iterators 409 Sentence Take #2: A Classic Interior 413 Making Sentence an Iterator: Bad Idea 415 Sentence Take #3: A Generator Function 416 How a Generator Function Works 417 Sentence Take #4: A Lazy Implementation 420 Sentence Take #5: A Generator Expression 421 Generator Expressions: When to Use Them 423 Another Example: Arithmetic Progression Generator 424 Arithmetic Progression with itertools 427 Generator Functions in the Standard Library 428 New Syntax in Python 3.3: yield from 437 Iterable Reducing Functions 438 A Closer Look at the iter Function 440 Case Study: Generators in a Database Conversion Utility 441 Generators as Coroutines 443 Chapter Summary 443 Further Reading Context Managers and else Blocks 451 Do This, Then That: else Blocks Beyond if 452 Context Managers and with Blocks 454 The contextlib Utilities Chapter Summary 463 Further Reading 463 Table of Contents xi

9 16. Coroutines 467 How Coroutines Evolved from Generators 468 Basic Behavior of a Generator Used as a Coroutine 469 Example: Coroutine to Compute a Running Average 472 Decorators for Coroutine Priming 474 Coroutine Termination and Exception Handling 475 Returning a Value from a Coroutine 479 Using yield from 481 The Meaning of yield from 487 Use Case: Coroutines for Discrete Event Simulation 493 About Discrete Event Simulations 493 The Taxi Fleet Simulation 494 Chapter Summary 502 Further Reading Concurrency with Futures 509 Example: Web Downloads in Three Styles 509 A Sequential Download Script 511 Downloading with concurrent.futures 514 Where Are the Futures? 515 Blocking I/O and the GIL 519 Launching Processes with concurrent.futures 519 Experimenting with Executor.map 521 Downloads with Progress Display and Error Handling 524 Error Handling in the flags2 Examples 529 Using futures.as_completed 531 Threading and Multiprocessing Alternatives 534 Chapter Summary 534 Further Reading Concurrency with asyncio 541 Thread Versus Coroutine: A Comparison 543 asyncio.future: Nonblocking by Design 549 Yielding from Futures, Tasks, and Coroutines 550 Downloading with asyncio and aiohttp 552 Running Circling Around Blocking Calls 556 Enhancing the asyncio downloader Script 558 Using asyncio.as_completed 559 Using an Executor to Avoid Blocking the Event Loop 564 From Callbacks to Futures and Coroutines 566 Doing Multiple Requests for Each Download 569 Writing asyncio Servers 571 xii Table of Contents

10 An asyncio TCP Server 572 An aiohttp Web Server 577 Smarter Clients for Better Concurrency 580 Chapter Summary 581 Further Reading 583 Part VI. Metaprogramming 19. Dynamic Attributes and Properties 589 Data Wrangling with Dynamic Attributes 590 Exploring JSON-Like Data with Dynamic Attributes 592 The Invalid Attribute Name Problem 595 Flexible Object Creation with new 597 Restructuring the OSCON Feed with shelve 598 Linked Record Retrieval with Properties 602 Using a Property for Attribute Validation 608 Lineltem Take #1: Class for an Item in an Order 608 Lineltem Take #2: A Validating Property 609 A Proper Look at Properties 611 Properties Override Instance Attributes 612 Property Documentation 614 Coding a Property Factory 615 Handling Attribute Deletion 618 Essential Attributes and Functions for Attribute Handling 620 Special Attributes that Affect Attribute Handling 620 Built-in Functions for Attribute Handling 620 Special Methods for Attribute Handling 621 Chapter Summary 623 Further Reading Attribute Descriptors 629 Descriptor Example: Attribute Validation 629 Lineltem Take #3: A Simple Descriptor 630 Lineltem Take #4: Automatic Storage Attribute Names 635 Lineltem Take #5: A New Descriptor Type 641 Overriding Versus Nonoverriding Descriptors 644 Overriding Descriptor 646 Overriding Descriptor Without get 647 Nonoverriding Descriptor 648 Overwriting a Descriptor in the Class 649 Methods Are Descriptors 650 Table ofcontents xiii

11 Descriptor Usage Tips 652 Descriptor docstring and Overriding Deletion 654 Chapter Summary 655 Further Reading Class Metaprogramming 659 A Class Factory 660 A Class Decorator for Customizing Descriptors 663 What Happens When: Import Time Versus Runtime 665 The Evaluation Time Exercises 666 Metaclasses The Metaclass Evaluation Time Exercise 673 A Metaclass for Customizing Descriptors 677 The Metaclass prepare Special Method 679 Classes as Objects 681 Chapter Summary 682 Further Reading 683 Afterword 687 A. Support Scripts 691 Python Jargon 719 Index 729 xiv Table of Contents