Docker for Data Science

Transcription

1 Docker for Data Science Building Scalable and Extensible Data Infrastructure Around the Jupyter Notebook Server Joshua Cook

2 Docker for Data Science Joshua Cook Santa Monica, California, USA ISBN-13 (pbk): ISBN-13 (electronic): DOI / Library of Congress Control Number: Copyright 2017 by Joshua Cook This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Cover image by Freepik ( Managing Director: Welmoed Spahr Editorial Director: Todd Green Acquisitions Editor: Celestin Suresh John Development Editor: Laura Berendson Technical Reviewer: Jeeva S. Chelladhurai Coordinating Editor: Prachi Mehta Copy Editor: Mary Behr Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY Phone SPRINGER, fax (201) , orders-ny@springer-sbm.com, or visit Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please rights@apress.com, or visit rights-permissions. Apress titles may be purchased in bulk for academic, corporate, or promotional use. ebook versions and licenses are also available for most titles. For more information, reference our Print and ebook Bulk Sales web page at Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book s product page, located at For more detailed information, please visit Printed on acid-free paper

3 To my wife, Aylin.

4 Contents at a Glance About the Author... xv About the Technical Reviewer... xvii Acknowledgments... xix Introduction... xxi Chapter 1: Introduction... 1 Chapter 2: Docker Chapter 3: Interactive Programming Chapter 4: The Docker Engine Chapter 5: The Dockerfile Chapter 6: Docker Hub Chapter 7: The Opinionated Jupyter Stacks Chapter 8: The Data Stores Chapter 9: Docker Compose Chapter 10: Interactive Software Development Index v

5 Contents About the Author... xv About the Technical Reviewer... xvii Acknowledgments... xix Introduction... xxi Chapter 1: Introduction... 1 Big Data... 1 Recommended Practice for Learning... 2 Set up a New AWS Account...2 Configure a Key Pair...3 Infrastructure Limitations on Data Pull the jupyter/scipy-notebook image...15 Run the jupyter/scipy-notebook Image...16 Monitor Memory Usage...17 What Size Data Set Will Cause a Memory Exception? What Size Dataset Is Too Large to Be Used to Fit Different Kinds of Simple Models?...22 Summary Chapter 2: Docker Docker Is Not a Virtual Machine Containerization vii

6 Contents A Containerized Application The Docker Container Ecosystem The Docker Client...33 The Host...34 The Docker Engine...34 The Docker Image and the Docker Container The Docker Registry...35 Get Docker Docker for Linux...36 Docker for Mac...40 Docker for Windows...40 Docker Toolbox...41 Hello, Docker! Basic Networking in Docker...45 Summary Chapter 3: Interactive Programming Jupyter as Persistent Interactive Computing How Not to Program Interactively...49 Setting Up a Minimal Computational Project...50 Writing the Source Code for the Evaluation of a Bessel Function Performing Your Calculation Using Docker Compile Your Source Code Execute Compiled Binary...54 How to Program Interactively Launch IPython Using Docker...55 Persistence...56 Jupyter Notebooks...57 Port Connections...63 viii

7 Contents Data Persistence in Docker...65 Attach a Volume...69 Summary Chapter 4: The Docker Engine Examining the Docker Workstation Hello, World in a Container Run Echo as a Service Isolating the Bootstrap Time...77 A Daemonized Hello World Summary Chapter 5: The Dockerfile Best Practices Stateless Containers...81 Single-Concern Containers...82 Project: A Repo of Docker Images Prepare for Local Development...82 Configure GitHub...83 Building Images Using Dockerfiles...83 Dockerfile Syntax...83 Designing the gsl Image...84 The Docker Build Cache...87 Anaconda...88 Design the miniconda3 Image...88 tini...96 ENTRYPOINT...97 Design the ipython Image...98 Run the ipython Image as a New Container Summary ix

8 Contents Chapter 6: Docker Hub Docker Hub Alternatives to Docker Hub Docker ID and Namespaces Image Repositories Search for Existing Repositories Tagged Images Tags on the Python Image Official Repositories Pushing to Docker Hub Create a New Repository Push an Image Pull the Image from Docker Hub Tagged Image on Docker Hub Summary Chapter 7: The Opinionated Jupyter Stacks High-Level Overview jupyter/base-notebook Notebook Security The Default Environment Managing Python Versions Extending the Jupyter Image Using conda Environments Using joyvan to Install Libraries Ephemeral Container Extension Maintaining Semi-Persistent Changes to Images Summary x

9 Contents Chapter 8: The Data Stores Serialization Serialization Formats and Methods Binary Encoding in Python Redis Pull the redis Image Docker Data Volumes and Persistence Create and View a New Data Volume Launch Redis as a Persistent Service Connecting Containers via Legacy Links Using Redis with Jupyter A Simple Redis Example Track an Iterative Process Across Notebooks Pass a Dictionary via a JSON Dump Pass a Numpy Array as a Bytestring MongoDB Set Up a New AWS t2.micro Configure the New AWS t2.micro for Docker Pull the mongo Image Create and View a New Data Volume Launch MongoDB as a Persistent Service Verify MongoDB Installation Using MongoDB with Jupyter MongoDB Structure pymongo Mongo and Twitter Obtain Twitter Credentials xi

10 Contents Collect Tweets by Geolocation Insert Tweets Into Mongo PostgreSQL Pull the postgres Image Create New Data Volume Launch PostgreSQL as a Persistent Service Verify PostgreSQL Installation Docker Container Networking Minimally Verify the Jupyter-PostgreSQL Connection Connnecting Containers by Name Using PostgreSQL with Jupyter Jupyter, PostgreSQL, Pandas, and psycopg Minimal Verification Loading Data into PostgreSQL PostgreSQL Binary Type and Numpy Summary Chapter 9: Docker Compose Install docker-compose What Is docker-compose? Docker Compose Versions Build a Simple Docker Compose Application Run Your Application with Compose Jupyter and Mongo with Persistence Specifying the Build Context Specify the Environment File Data Persistence Build Your Application with Compose Scaling an AWS Application via Instance Type xii

11 Contents Restart Docker Compose Application Complete the Computation Encode Tweets as Document Vectors Switch AWS Instance Type to t2.micro Retrieve Tweets from MongoDB and Compare Docker Compose Networks Jupyter and Postgres with Persistence Specifying the Build Context Build and Run Your Application with Compose Summary Chapter 10: Interactive Software Development A Quick Guide to Organizing Computational Biology Projects A Project Framework for Interactive Development Project Root Design Pattern Initialize Project Examine Database Requirements Managing the Project via Git Adding a Database to Your Application Interactive Development Create a Python Module Using Jupyter Add Delayed Processing to Your Application Extending the Postgres Module Updating Your Python Module Summary Index xiii

12 About the Author Joshua Cook is a mathematician. He writes code in Bash, C, and Python and has done pure and applied computational work in geo-spatial predictive modeling, quantum mechanics, semantic search, and artificial intelligence. He also has 10 years experience teaching mathematics at the secondary and post-secondary level. His research interests lie in high-performance computing, interactive computing, feature extraction, and reinforcement learning. He is always willing to discuss orthogonality or to explain why Fortran is the language of the future over a warm or cold beverage. xv

13 About the Technical Reviewer Jeeva S. Chelladhurai has been working as a DevOps specialist at the IBM GTS Labs for the last 9 years. He is the co-author of Learning Docker, published by PacktPub, UK. He has more than 20 years of IT industry experience. He has technically managed and mentored diverse teams across the globe in envisaging and building pioneering telecommunication products. He specializes in DevOps, automation, and cloud solution delivery, with a focus on data center optimization, software-defined environments (SDEs), and distributed application development, deployment, and delivery using the newest Docker technology. Jeeva is also a strong proponent of the agile methodologies, DevOps, and IT automation. He holds a master s degree in computer science from Manonmaniam Sundaranar University and a graduation certificate in project management from Boston University, Boston, Massachusetts, USA. Besides his official responsibilities, he writes book chapters and authors research papers. He has been instrumental in crafting reusable technical assets for IBM solution architects and consultants. He speaks in technical forums on DevOps technologies and tools. He hosts one of the largest Open Source communities in Bangalore ( His LinkedIn profile can be found at xvii

14 Acknowledgments Thanks to Mike Frantz, Gilad Gressel, Devon Muraoka, Bharat Ramanathan, Nash Taylor, Matt Zhou, and DSI Santa Monica Cohorts 3 and 4 for talking through some of the more abstract concepts herein with me. Thanks to Chad Arnett for keeping it weird. Thanks to Jim Kruidenier and Jussi Eloranta for teaching an old dog new tricks. Thanks to my father for continually inspiring me and my mother for giving me my infallible belief in goodness. Thanks to Momlo for paving the way and Dablo for his curiosity. Thanks to my wife, Aylin, for her belief in me and tolerance for the word eigenvector. xix

15 Introduction This text is designed to teach the concepts and techniques of Docker and its ecosystem as applied to the field of data science. Besides introducing the core Docker technologies (the container and image, the engine, the Dockerfile), this book contains a discussion on building larger integrated systems using the Jupyter Notebook Server and open source data stores MongoDB, PostgreSQL, and Redis. The first chapter walks the reader through a recommended hardware configuration for working through the text using an AWS t2.micro. Chapters 2 and 3 introduce the core technologies used in the book, Docker and Jupyter, as well as the idea of interactive programming. Chapters 4, 5, 6, and 9 dig deeper into specific areas of the Docker ecosystem. Chapter 7 explores the official Jupyter Docker images developed and maintained by the Jupyter development team. Chapter 8 introduces the Docker images for three open source data stores. Chapters 9 and 10 tie everything together, connecting Jupyter to data stores using Docker Compose. After having completed the book, readers are encouraged to reread Chapter 3 and Chapter 10 to begin to develop their own interactive software development style. The concepts presented herein can be challenging, especially in terms of the abstraction of computer resources and processes. That said, no requisite knowledge is assumed. An attempt has been made to build the discussion from base principles. With this in mind, the reader should be comfortable working at the command line and have an adventurous and inquisitive spirit. We hope that readers with an intermediate to advanced understanding of Docker, Jupyter, or both will gain a deeper understanding of the concepts and learn novel approaches to the solving of computational problems using these tools. xxi