Network Programming with Go

Transcription

1 Network Programming with Go Essential Skills for Using and Securing Networks Jan Newmarch

2 Network Programming with Go: Essential Skills for Using and Securing Networks Jan Newmarch Oakleigh, Victoria Australia ISBN-13 (pbk): ISBN-13 (electronic): DOI / Library of Congress Control Number: Copyright 2017 by Jan Newmarch This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director: Welmoed Spahr Editorial Director: Todd Green Acquisitions Editor: Steve Anglin Development Editor: Matthew Moodie Technical Reviewer: Ronald Petty Coordinating Editor: Mark Powers Copy Editor: Kezia Endsley Compositor: SPi Global Indexer: SPi Global Artist: SPi Global Cover image designed by Freepik Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY Phone SPRINGER, fax (201) , orders-ny@springer-sbm.com, or visit Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please rights@apress.com, or visit rights-permissions. Apress titles may be purchased in bulk for academic, corporate, or promotional use. ebook versions and licenses are also available for most titles. For more information, reference our Print and ebook Bulk Sales web page at Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book's product page, located at For more detailed information, please visit Printed on acid-free paper

3 Contents at a Glance About the Author...xvii About the Technical Reviewer...xix Preface...xxi Chapter 1: Architecture... 1 Chapter 2: Overview of the Go Language Chapter 3: Socket-Level Programming Chapter 4: Data Serialization Chapter 5: Application-Level Protocols Chapter 6: Managing Character Sets and Encodings Chapter 7: Security Chapter 8: HTTP Chapter 9: Templates Chapter 10: A Complete Web Server Chapter 11: HTML Chapter 12: XML Chapter 13: Remote Procedure Call Chapter 14: REST Chapter 15: WebSockets Afterword Index iii

4 Contents About the Author...xvii About the Technical Reviewer...xix Preface...xxi Chapter 1: Architecture... 1 Protocol Layers... 1 ISO OSI Protocol... 2 OSI Layers... 2 TCP/IP Protocol... 3 Some Alternative Protocols... 3 Networking... 3 Gateways... 4 Packet Encapsulation... 4 Connection Models... 5 Connection Oriented... 5 Connectionless... 5 Communications Models... 5 Message Passing... 5 Remote Procedure Call... 6 Distributed Computing Models... 7 Client-Server System... 8 Client-Server Application... 8 Server Distribution... 9 v

5 Communication Flows... 9 Synchronous Communication Asynchronous Communication Streaming Communication Publish/Subscribe Component Distribution Gartner Classification Three-Tier Models Fat versus Thin Middleware Model Middleware Examples Middleware Functions Continuum of Processing Points of Failure Acceptance Factors Transparency Access Transparency Location Transparency Migration Transparency Replication Transparency Concurrency Transparency Scalability Transparency Performance Transparency Failure Transparency Eight Fallacies of Distributed Computing Fallacy: The Network Is Reliable Fallacy: Latency Is Zero Fallacy: Bandwidth Is Infinite Fallacy: The Network Is Secure vi

6 Fallacy: Topology Doesn t Change Fallacy: There Is One Administrator Fallacy: Transport Cost Is Zero Fallacy: The Network Is Homogeneous Conclusion Chapter 2: Overview of the Go Language Types Slices and Arrays Structures Pointers Functions Maps Methods Multi-Threading Packages Type Conversion Statements GOPATH Running Go Programs Standard Libraries Error Values Conclusion Chapter 3: Socket-Level Programming The TCP/IP Stack IP Datagrams UDP TCP Internet Addresses IPv4 Addresses IPv6 Addresses vii

7 IP Address Type The IPMask Type The IPAddr Type...36 Host Lookup Services Ports The TCPAddr Type TCP Sockets TCP Client A Daytime Server Multi-Threaded Server Controlling TCP Connections Timeout Staying Alive UDP Datagrams Server Listening on Multiple Sockets The Conn, PacketConn, and Listener Types Raw Sockets and the IPConn Type Conclusion Chapter 4: Data Serialization Structured Data Mutual Agreement Self-Describing Data ASN ASN.1 Daytime Client and Server JSON A Client and Server The Gob Package A Client and Server viii

8 Encoding Binary Data as Strings Protocol Buffers Installing and Compiling Protocol Buffers The Compiled personv3.pb.go File...84 Using the Compiled Code Conclusion Chapter 5: Application-Level Protocols Protocol Design Why Should You Worry? Version Control The Web Message Format Data Format Byte Format Character Format A Simple Example A Standalone Application The Client-Server Application The Client Side Alternative Presentation Aspects The Server Side Protocol: Informal Text Protocol Server Code Client Code Textproto Package State Information Application State Transition Diagram Client State Transition Diagrams ix

9 x Server State Transition Diagrams Server Pseudocode Conclusion Chapter 6: Managing Character Sets and Encodings Definitions Character Character Repertoire/Character Set Character Code Character Encoding Transport Encoding ASCII ISO Unicode UTF-8, Go, and Runes UTF-8 Client and Server ASCII Client and Server UTF-16 and Go Little-Endian and Big-Endian UTF-16 Client and Server Unicode Gotchas ISO 8859 and Go Other Character Sets and Go Conclusion Chapter 7: Security ISO Security Architecture Functions and Levels Mechanisms Data Integrity Symmetric Key Encryption Public Key Encryption

10 X.509 Certificates TLS A Basic Client Server Using a Self-Signed Certificate Conclusion Chapter 8: HTTP URLs and Resources I18n HTTP Characteristics Versions HTTP HTTP HTTP HTTP/ Simple User Agents The Response Type The HEAD Method The GET Method Configuring HTTP Requests The Client Object Proxy Handling Simple Proxy Authenticating Proxy HTTPS Connections by Clients Servers File Server Handler Functions Bypassing the Default Multiplexer HTTPS Conclusion xi

11 Chapter 9: Templates Inserting Object Values Using Templates Pipelines Defining Functions Variables Conditional Statements The HTML / Template Package Conclusion Chapter 10: A Complete Web Server Browser Site Diagram Browser Files Basic Server The listflashcards Function The manageflashcards Function The Chinese Dictionary The Dictionary Type Flashcard Sets Fixing Accents The ListWords Function The showflashcards Function Presentation on the Browser Running the Server Conclusion Chapter 11: HTML The Go HTML/Template Package Tokenizing HTML XHTML/HTML xii

12 JSON Conclusion Chapter 12: XML Parsing XML The StartElement Type The EndElement Type The CharData Type The Comment Type The ProcInst Type The Directive Type Unmarshalling XML Marshalling XML XHTML HTML Conclusion Chapter 13: Remote Procedure Call Go s RPC HTTP RPC Server HTTP RPC Client TCP RPC Server TCP RPC Client Matching Values JSON JSON RPC Server JSON RPC Client Conclusion Chapter 14: REST URIs and Resources Representations xiii

13 REST Verbs The GET Verb The PUT Verb The DELETE Verb The POST Verb No Maintained State HATEOAS Representing Links Transactions with REST The Richardson Maturity Model Flashcards Revisited URLs The Demultiplexer (Demuxer) Content Negotiation GET / POST / Handling Other URLs The Complete Server Client Using REST or RPC Conclusion Chapter 15: WebSockets WebSockets Server The Go Sub-Repository Package The Message Object The JSON Object The Codec Type WebSockets Over TLS WebSockets in an HTML Page xiv

14 The Gorilla Package Echo Server Echo Client Conclusion Afterword Index xv

15 About the Author Jan Newmarch is head of ICT (higher education) at Box Hill Institute, adjunct professor at Canberra University, and adjunct lecturer in the School of Information Technology, Computing and Mathematics at Charles Sturt University. He is interested in more aspects of computing than he has time to pursue, but the major thrust over the last few years has developed from user interfaces under UNIX into Java, the Web, and then into general distributed systems. Jan developed a number of publicly available software systems in these areas. For the last few years, he has been looking at sound for Linux systems and programming the Raspberry Pi s GPU. He is now exploring aspects of the IoT. He lives in Melbourne, Australia and enjoys the food and culture there, but is not so impressed by the weather. xvii

16 About the Technical Reviewer Ronald Petty, M.B.A., M.S. is the founder of Minimum Distance LLC, a management consulting firm based in San Francisco. He spends his time helping technology-based startups do the right thing. He is also an instructor at UC Berkeley Extension. xix

17 Preface It s always fun to learn a new programming language, especially when it turns out to be a major one. Prior to the release of Go in 2009, I was teaching a Master s level subject in network programming at Monash University. It s good to have a goal when learning a new language, but this time, instead of building yet another wine cellar program, I decided to orient my lecture notes around Go instead of my (then) standard delivery vehicle of Java. The experiment worked well: apart from the richness of the Java libraries that Go was yet to match, all the programming examples transferred remarkably well, and in many cases were more elegant than the original Java programs. This book is the result. I have updated it as Go has evolved and as new technologies such as HTTP/2 have arisen. But if it reads like a textbook, well, that is because it is one. There is a large body of theoretical and practical concepts involved in network programming and this book covers some of these as well as the practicalities of building systems in Go. In terms of language popularity, Go is clearly rising. It has climbed to 16th in the TIOBE index, is 18th in the PYPL (Popularity of Programming Language), and is 15th in the RedMonk Programming Language rankings. It is generally rated as one of the fastest growing languages. There is a growing community of developers both of the core language and libraries and of the independent projects. I have tried to limit the scope of this book to the standard libraries only and to the sub-repositories of the Go tree. While this eliminates many excellent projects that no doubt make many programming tasks easier, restricting the book to the official Go libraries provides a clear bound. This book assumes a basic knowledge of Go. The focus is on using Go to build network applications, not on the basics of the language. Network applications are different than command-line applications, are different than applications with a graphical user interface, and so on. So the first chapter discusses architectural aspects of network programs. The second chapter is an overview of the features of Go that we use in this book. The third chapter on sockets covers the Go version of the basics underlying all TCP/IP systems. Chapters 4, 5, and 6 are more unusual in network programming books. They cover the topics of what representations of data will be used, how a network interaction will proceed, and for text, which language formats are used. Then in Chapter 7, we look at the increasingly important topic of security. In Chapter 8, we look at one of the most common application layer protocols in use, HTTP. The next four chapters are about topics related to HTTP and common data formats carried above HTTP HTML and XML. In Chapter 13, we look at an alternative approach to network programming, remote procedure calls. Chapters 14 and 15 consider further aspects of network programming using HTTP. xxi