Network Programming with Go Essential Skills for Using and Securing Networks Jan Newmarch
Network Programming with Go: Essential Skills for Using and Securing Networks Jan Newmarch Oakleigh, Victoria Australia ISBN-13 (pbk): 978-1-4842-2691-9 ISBN-13 (electronic): 978-1-4842-2692-6 DOI 10.1007/978-1-4842-2692-6 Library of Congress Control Number: 2017941517 Copyright 2017 by Jan Newmarch This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director: Welmoed Spahr Editorial Director: Todd Green Acquisitions Editor: Steve Anglin Development Editor: Matthew Moodie Technical Reviewer: Ronald Petty Coordinating Editor: Mark Powers Copy Editor: Kezia Endsley Compositor: SPi Global Indexer: SPi Global Artist: SPi Global Cover image designed by Freepik Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail rights@apress.com, or visit http://www.apress.com/ rights-permissions. Apress titles may be purchased in bulk for academic, corporate, or promotional use. ebook versions and licenses are also available for most titles. For more information, reference our Print and ebook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book's product page, located at www.apress.com/9781484226919. For more detailed information, please visit http://www.apress.com/source-code. Printed on acid-free paper
Contents at a Glance About the Author...xvii About the Technical Reviewer...xix Preface...xxi Chapter 1: Architecture... 1 Chapter 2: Overview of the Go Language... 21 Chapter 3: Socket-Level Programming... 29 Chapter 4: Data Serialization... 57 Chapter 5: Application-Level Protocols... 87 Chapter 6: Managing Character Sets and Encodings... 107 Chapter 7: Security... 121 Chapter 8: HTTP... 137 Chapter 9: Templates... 161 Chapter 10: A Complete Web Server... 175 Chapter 11: HTML... 193 Chapter 12: XML... 199 Chapter 13: Remote Procedure Call... 209 Chapter 14: REST... 221 Chapter 15: WebSockets... 247 Afterword... 267 Index... 269 iii
Contents About the Author...xvii About the Technical Reviewer...xix Preface...xxi Chapter 1: Architecture... 1 Protocol Layers... 1 ISO OSI Protocol... 2 OSI Layers... 2 TCP/IP Protocol... 3 Some Alternative Protocols... 3 Networking... 3 Gateways... 4 Packet Encapsulation... 4 Connection Models... 5 Connection Oriented... 5 Connectionless... 5 Communications Models... 5 Message Passing... 5 Remote Procedure Call... 6 Distributed Computing Models... 7 Client-Server System... 8 Client-Server Application... 8 Server Distribution... 9 v
Communication Flows... 9 Synchronous Communication... 10 Asynchronous Communication... 10 Streaming Communication... 10 Publish/Subscribe... 10 Component Distribution... 10 Gartner Classification... 11 Three-Tier Models... 13 Fat versus Thin... 14 Middleware Model... 14 Middleware Examples... 14 Middleware Functions... 15 Continuum of Processing... 15 Points of Failure... 16 Acceptance Factors... 16 Transparency... 17 Access Transparency... 17 Location Transparency... 17 Migration Transparency... 17 Replication Transparency... 17 Concurrency Transparency... 17 Scalability Transparency... 17 Performance Transparency... 18 Failure Transparency... 18 Eight Fallacies of Distributed Computing... 18 Fallacy: The Network Is Reliable... 18 Fallacy: Latency Is Zero... 19 Fallacy: Bandwidth Is Infinite... 19 Fallacy: The Network Is Secure... 19 vi
Fallacy: Topology Doesn t Change... 19 Fallacy: There Is One Administrator... 19 Fallacy: Transport Cost Is Zero... 20 Fallacy: The Network Is Homogeneous... 20 Conclusion... 20 Chapter 2: Overview of the Go Language... 21 Types... 22 Slices and Arrays... 22 Structures... 22 Pointers... 23 Functions... 23 Maps... 24 Methods... 24 Multi-Threading... 25 Packages... 25 Type Conversion... 25 Statements... 25 GOPATH... 25 Running Go Programs... 26 Standard Libraries... 26 Error Values... 26 Conclusion... 27 Chapter 3: Socket-Level Programming... 29 The TCP/IP Stack... 29 IP Datagrams... 30 UDP... 30 TCP... 30 Internet Addresses... 30 IPv4 Addresses... 31 IPv6 Addresses... 31 vii
IP Address Type... 32 The IPMask Type... 33 The IPAddr Type...36 Host Lookup... 37 Services... 38 Ports... 38 The TCPAddr Type... 39 TCP Sockets... 40 TCP Client... 40 A Daytime Server... 42 Multi-Threaded Server... 44 Controlling TCP Connections... 46 Timeout... 46 Staying Alive... 46 UDP Datagrams... 47 Server Listening on Multiple Sockets... 49 The Conn, PacketConn, and Listener Types... 49 Raw Sockets and the IPConn Type... 52 Conclusion... 55 Chapter 4: Data Serialization... 57 Structured Data... 57 Mutual Agreement... 59 Self-Describing Data... 59 ASN.1... 60 ASN.1 Daytime Client and Server... 66 JSON... 68 A Client and Server... 72 The Gob Package... 75 A Client and Server... 78 viii
Encoding Binary Data as Strings... 81 Protocol Buffers... 83 Installing and Compiling Protocol Buffers... 84 The Compiled personv3.pb.go File...84 Using the Compiled Code... 85 Conclusion... 86 Chapter 5: Application-Level Protocols... 87 Protocol Design... 87 Why Should You Worry?... 88 Version Control... 88 The Web... 89 Message Format... 90 Data Format... 91 Byte Format... 91 Character Format... 92 A Simple Example... 92 A Standalone Application... 93 The Client-Server Application... 94 The Client Side... 94 Alternative Presentation Aspects... 95 The Server Side... 95 Protocol: Informal... 95 Text Protocol... 96 Server Code... 97 Client Code... 99 Textproto Package... 101 State Information... 101 Application State Transition Diagram... 103 Client State Transition Diagrams... 104 ix
x Server State Transition Diagrams... 105 Server Pseudocode... 105 Conclusion... 106 Chapter 6: Managing Character Sets and Encodings... 107 Definitions... 108 Character... 108 Character Repertoire/Character Set... 108 Character Code... 108 Character Encoding... 108 Transport Encoding... 109 ASCII... 109 ISO 8859... 111 Unicode... 111 UTF-8, Go, and Runes... 112 UTF-8 Client and Server... 112 ASCII Client and Server... 113 UTF-16 and Go... 113 Little-Endian and Big-Endian... 113 UTF-16 Client and Server... 114 Unicode Gotchas... 116 ISO 8859 and Go... 117 Other Character Sets and Go... 119 Conclusion... 119 Chapter 7: Security... 121 ISO Security Architecture... 121 Functions and Levels... 122 Mechanisms... 123 Data Integrity... 124 Symmetric Key Encryption... 126 Public Key Encryption... 127
X.509 Certificates... 129 TLS... 132 A Basic Client... 132 Server Using a Self-Signed Certificate... 133 Conclusion... 136 Chapter 8: HTTP... 137 URLs and Resources... 137 I18n... 137 HTTP Characteristics... 138 Versions... 138 HTTP 0.9... 138 HTTP 1.0... 139 HTTP 1.1... 140 HTTP/2... 141 Simple User Agents... 141 The Response Type... 141 The HEAD Method... 142 The GET Method... 143 Configuring HTTP Requests... 145 The Client Object... 147 Proxy Handling... 149 Simple Proxy... 149 Authenticating Proxy... 151 HTTPS Connections by Clients... 153 Servers... 155 File Server... 155 Handler Functions... 156 Bypassing the Default Multiplexer... 158 HTTPS... 159 Conclusion... 160 xi
Chapter 9: Templates... 161 Inserting Object Values... 161 Using Templates... 162 Pipelines... 164 Defining Functions... 165 Variables... 167 Conditional Statements... 168 The HTML / Template Package... 173 Conclusion... 173 Chapter 10: A Complete Web Server... 175 Browser Site Diagram... 175 Browser Files... 177 Basic Server... 177 The listflashcards Function... 179 The manageflashcards Function... 181 The Chinese Dictionary... 181 The Dictionary Type...182 Flashcard Sets... 183 Fixing Accents... 184 The ListWords Function... 187 The showflashcards Function... 189 Presentation on the Browser... 191 Running the Server... 191 Conclusion... 191 Chapter 11: HTML... 193 The Go HTML/Template Package... 194 Tokenizing HTML... 195 XHTML/HTML... 197 xii
JSON... 198 Conclusion... 198 Chapter 12: XML... 199 Parsing XML... 200 The StartElement Type... 200 The EndElement Type... 200 The CharData Type... 200 The Comment Type... 200 The ProcInst Type... 201 The Directive Type... 201 Unmarshalling XML... 203 Marshalling XML... 206 XHTML... 207 HTML... 207 Conclusion... 207 Chapter 13: Remote Procedure Call... 209 Go s RPC... 210 HTTP RPC Server... 212 HTTP RPC Client... 213 TCP RPC Server... 214 TCP RPC Client... 216 Matching Values... 217 JSON... 217 JSON RPC Server... 218 JSON RPC Client... 219 Conclusion... 220 Chapter 14: REST... 221 URIs and Resources... 221 Representations... 222 xiii
REST Verbs... 223 The GET Verb... 223 The PUT Verb... 223 The DELETE Verb... 224 The POST Verb... 224 No Maintained State... 224 HATEOAS... 224 Representing Links... 225 Transactions with REST... 226 The Richardson Maturity Model... 227 Flashcards Revisited... 228 URLs... 228 The Demultiplexer (Demuxer)... 229 Content Negotiation... 230 GET /... 232 POST /... 233 Handling Other URLs... 234 The Complete Server... 234 Client... 240 Using REST or RPC... 245 Conclusion... 245 Chapter 15: WebSockets... 247 WebSockets Server... 248 The Go Sub-Repository Package... 248 The Message Object... 248 The JSON Object... 251 The Codec Type... 254 WebSockets Over TLS... 257 WebSockets in an HTML Page... 259 xiv
The Gorilla Package... 263 Echo Server... 264 Echo Client... 265 Conclusion... 266 Afterword... 267 Index... 269 xv
About the Author Jan Newmarch is head of ICT (higher education) at Box Hill Institute, adjunct professor at Canberra University, and adjunct lecturer in the School of Information Technology, Computing and Mathematics at Charles Sturt University. He is interested in more aspects of computing than he has time to pursue, but the major thrust over the last few years has developed from user interfaces under UNIX into Java, the Web, and then into general distributed systems. Jan developed a number of publicly available software systems in these areas. For the last few years, he has been looking at sound for Linux systems and programming the Raspberry Pi s GPU. He is now exploring aspects of the IoT. He lives in Melbourne, Australia and enjoys the food and culture there, but is not so impressed by the weather. xvii
About the Technical Reviewer Ronald Petty, M.B.A., M.S. is the founder of Minimum Distance LLC, a management consulting firm based in San Francisco. He spends his time helping technology-based startups do the right thing. He is also an instructor at UC Berkeley Extension. xix
Preface It s always fun to learn a new programming language, especially when it turns out to be a major one. Prior to the release of Go in 2009, I was teaching a Master s level subject in network programming at Monash University. It s good to have a goal when learning a new language, but this time, instead of building yet another wine cellar program, I decided to orient my lecture notes around Go instead of my (then) standard delivery vehicle of Java. The experiment worked well: apart from the richness of the Java libraries that Go was yet to match, all the programming examples transferred remarkably well, and in many cases were more elegant than the original Java programs. This book is the result. I have updated it as Go has evolved and as new technologies such as HTTP/2 have arisen. But if it reads like a textbook, well, that is because it is one. There is a large body of theoretical and practical concepts involved in network programming and this book covers some of these as well as the practicalities of building systems in Go. In terms of language popularity, Go is clearly rising. It has climbed to 16th in the TIOBE index, is 18th in the PYPL (Popularity of Programming Language), and is 15th in the RedMonk Programming Language rankings. It is generally rated as one of the fastest growing languages. There is a growing community of developers both of the core language and libraries and of the independent projects. I have tried to limit the scope of this book to the standard libraries only and to the sub-repositories of the Go tree. While this eliminates many excellent projects that no doubt make many programming tasks easier, restricting the book to the official Go libraries provides a clear bound. This book assumes a basic knowledge of Go. The focus is on using Go to build network applications, not on the basics of the language. Network applications are different than command-line applications, are different than applications with a graphical user interface, and so on. So the first chapter discusses architectural aspects of network programs. The second chapter is an overview of the features of Go that we use in this book. The third chapter on sockets covers the Go version of the basics underlying all TCP/IP systems. Chapters 4, 5, and 6 are more unusual in network programming books. They cover the topics of what representations of data will be used, how a network interaction will proceed, and for text, which language formats are used. Then in Chapter 7, we look at the increasingly important topic of security. In Chapter 8, we look at one of the most common application layer protocols in use, HTTP. The next four chapters are about topics related to HTTP and common data formats carried above HTTP HTML and XML. In Chapter 13, we look at an alternative approach to network programming, remote procedure calls. Chapters 14 and 15 consider further aspects of network programming using HTTP. xxi