Foundations of Python

Transcription

1 Foundations of Python Network Programming The comprehensive guide to building network applications with Python Second Edition Brandon Rhodes John Goerzen Apress

2 Contents Contents at a Glance About the Authors About the Technical Reviewer Acknowledgments Introduction iv xv xv xvi xvii Chapter 1: Introduction to Client/Server Networking 1 The Building Blocks: Stacks and Libraries 1 Application Layers 4 Speaking a Protocol 5 A Raw Network Conversation 6 Turtles All the Way Down 8 The Internet Protocol 9 IP Addresses 10 Routing 11 Packet Fragmentation 13 Learning More About IP 14 Chapter 2: UDP 15 Should You Read This Chapter? 16 Addresses and Port Numbers 16 Port Number Ranges 17 Sockets 19

3 CONTENTS Unreliability, Backoff, Blocking, Timeouts 22 Connecting UDP Sockets 25 Request IDs: A Good Idea 27 Binding to Interfaces 28 UDP Fragmentation 30 Socket Options 31 Broadcast 32 When to Use UDP SlChapter 3: TCP 35 How TCP Works 35 When to Use TCP 36 What TCP Sockets Mean 37 A Simple TCP Client and Server 38 One Socket per Conversation 41 Address Already in Use 42 Binding to Interfaces 43 Deadlock 44 Closed Connections, Half-Open Connections 48 Using TCP Streams like Files Chapter 4: Socket Names and DNS 51 Hostnames and Domain Names 51 Socket Names 52 Five Socket Coordinates 53 IPv6 54 Modern Address Resolution 55

4 m CONTENTS Asking getaddrinfoo Where to Bind 56 Asking getaddrinfoo About Services 56 Asking getaddrinfoo for Pretty Hostnames 57 Other getaddrinfoo Flags 68 Primitive Name Service Routines 59 Using getsockaddro in Your Own Code 60 Better Living Through Paranoia 61 A Sketch of How DNS Works 63 Why Not to Use DNS 65 Why to Use DNS 66 Resolving Mail Domains Zeroconf and Dynamic DNS 70 Chapter 5: etwork Data and Network Errors 71 Text and Encodings 71 Network Byte Order 73 Framing and Quoting 75 Pickles and Self-Delimiting Formats --79 XML, JSON, Etc 80 Compression 81 Network Exceptions 82 Handling Exceptions Chapter 6: TLS and SSL 87 Computer Security 87 IP Access Rules 88 Cleartext on the Network vii

5 * CONTENTS TLS Encrypts Your Conversations 92 TLS Verifies Identities 93 Supporting TLS in Python 94 The Standard SSL Module 95 Loose Ends Chapter 7: Server Architecture 99 Daemons and Logging 99 Our Example: Sir Launcelot 100 An Elementary Client 102 The Waiting Game 103 Running a Benchmark 106 Event-Driven Servers 109 Poll vs. Select 112 The Semantics of Non-blocking 113 Event-Driven Servers Are Blocking and Synchronous 114 Twisted Python 114 Load Balancing and Proxies 117 Threading and Multi-processing 117 Threading and Multi-processing Frameworks 120 Process and Thread Coordination 122 Running Inside inetd Chapter 8: Caches, Message Queues, and Map-Reduce 125 Using Memcached 126 Memcached and Sharding 128 Message Queues 130 viii

6 If CONTENTS Using Message Queues from Python 131 How Message Queues Change Programming 133 Map-Reduce Chapter 9: HTTP 137 URL Anatomy 138 Relative URLs 141 Instrumenting urllib2 141 The GET Method 142 The Host Header 144 Codes, Errors, and Redirection 144 Payloads and Persistent Connections, 147 POST And Forms 148 Successful Form POSTs Should Always Redirect 150 POST And APIs 151 REST And More HTTP Methods 151 Identifying User Agents and Web Servers 152 Content Type Negotiation 153 Compression 154 HTTP Caching 155 The HEAD Method 156 HTTPS Encryption 156 HTTP Authentication 157 Cookies 158 HTTP Session Hijacking 160 Cross-Site Scripting Attacks 160 WebOb 161

7 i CONTENTS 161 Chapter 10: Screen Scraping 163 Fetching Web Pages 163 Downloading Pages Through Form Submission 164 The Structure of Web Pages 167 Three Axes 168 Diving into an HTML Document 169 Selectors 173 I77 Chapter 11: Web Applications 179 Web Servers and Python 180 Two Tiers 18 Choosing a Web Server 182 WSGI 183 WSGI Middleware 185 Python Web Frameworks 187 URL Dispatch Techniques 189 Templates 190 Final Considerations 191 Pure-Python Web Servers 192 CGI 193 mod_python 194 Chapter 12: Composition and > Decoding Messages Composing Traditional Messages 200 Parsing Traditional Messages 202 X

8 it CONTENTS Parsing Dates 203 Understanding MIME 205 How MIME Works 206 Composing MIME Attachments 206 MIME Alternative Parts 208 Composing Non-English Headers 210 Composing Nested Multiparts 211 Parsing MIME Messages 213 Decoding Headers IChapter13: SMTP 217 Clients, Webmail Services 217 In the Beginning Was the Command Line 218 The Rise of Clients 218 The Move to Webmail 220 How SMTP Is Used 221 Sending 221 Headers and the Envelope Recipient 222 Multiple Hops 223 Introducing the SMTP Library 224 Error Handling and Conversation Debugging 225 Getting Information from EHLO 228 Using Secure Sockets Layer and Transport Layer Security 230 Authenticated SMTP 232 SMTP Tips Chapter 14: POP 235 Compatibility Between POP Servers 235

9 CONTENTS Connecting and Authenticating 235 Obtaining Mailbox Information 238 Downloading and Deleting Messages Chapter 15: IMAP 243 Understanding IMAP in Python 244 IMAPCIient 246 Examining Folders 248 Message Numbers vs. UIDs 248 Message Ranges 249 Information 249 Downloading an Entire Mailbox 250 Downloading Messages Individually 252 Flagging and Deleting Messages 257 Deleting Messages 258 Searching 259 Manipulating Folders and Messages 260 Asynchrony Chapter 16: Telnet and SSH 263 Command-Line Automation 263 Command-Line Expansion and Quoting 265 Unix Has No Special Characters 266 Quoting Characters for Protection 268 The Terrible Windows Command Line 269 Things Are Different in a Terminal 270 Terminals Do Buffering 273 xii

10 11 CONTENTS Telnet 274 SSH:The Secure Shell 278 An Overview of SSH 279 SSH Host Keys 280 SSH Authentication 282 Shell Sessions and Individual Commands 283 SFTP: File Transfer Over SSH 286 Other Features Chapter 17: FTP 291 What to Use Instead of FTP 291 Communication Channels 292 Using FTP in Python 293 ASCII and Binary Files 294 Advanced Binary Downloading 295 Uploading Data 297 Advanced Binary Uploading 298 Handling Errors 299 Detecting Directories and Recursive Download 301 Creating Directories, Deleting Things 302 Doing FTP Securely Chapter 18: RPC 305 Features of RPC 306 XML-RPC 307 JSON-RPC 313 Self-documenting Data 315 xiii

11 II CONTENTS Talking About Objects: Pyro and RPyC 316 An RPyC Example 317 RPC, Web Frameworks, Message Queues 319 Recovering From Network Errors 320 Binary Options: Thrift and Protocol Buffers Index 323 xiv