Index. Bessel function, 51 Big data, 1. Cloud-based version-control system, 226 Containerization, 30 application, 32 virtualize processes, 30 31

Transcription

1 Index A Amazon Web Services (AWS), 2 account creation, 2 EC2 instance creation, 9 Docker, 13 IP address, 12 key pair, 12 launch button, 11 security group, 11 stable Ubuntu server, 9 t2.micro type, 9 10 inbound rules, 8 key pair configuration, 3 control panel, 5 EC2 dashboard, 5 import, 6 SSH key, 6 process, 2 security group pane, 7 Anaconda, 88 B Bessel function, 51 Big data, 1 C Cloud-based version-control system, 226 Containerization, 30 application, 32 virtualize processes, D Daemonized Hello World, 78 Data infrastructure DecisionTreeClassifier, 24 jupyter/scipy-notebook image, KNeighborsClassifier, 24 limitations, 14 LogisticRegression, 24 memory exception creation, 20 default values, 19 make_classification import, 18 MemoryError, 21 memory footprint, 21 notebook, 17 Python kernel, 20 running notebook, 18 models, 22 monitor memory usage, 17 T2.micro, 26 Data store technologies, 137 Docker data volumes and persistence, 141 connecting containers via legacy links, create and view, 142 iterative process, 148 launching Redis, pass dictionary via JSON dump, pass numpy array as bytestrings, 150 Redis example, 147 Redis with Jupyter, Joshua Cook 2017 J. Cook, Docker for Data Science, DOI /

2 Data store technologies (cont.) MongoDB, 151 data volume, create and view, insert tweets, 164 installation, verifying, 155 with Jupyter, Mongo and Twitter, 158 new AWS t2.micro, 152 new AWS t2.micro for Docker, 153 as persistent service, pull the mono image, 153 pymongo, structure, 156 tweets by geolocation, Twitter credentials, PostgreSQL, 164 binary type and numpy, connecting containers by name, Docker container networking, installation, verifying, with Jupyter, 174 Jupyter, pandas and psycopg2, 174 Jupyter-PostgreSQL connection, loading data, minimal verification, new data volume, creating, 165 as persistent service, 166 postgres image, pull, 165 Redis, 139 pull the image, serialization, 137 binary encoding in Python, 139 formats and methods, 138 Docker, 1 client, 33 compose, 35 concepts, 32 containerization, 29 container networking, data volumes and persistence, 141 connecting containers via legacy links, create and view, 142 iterative process, 148 launching Redis, pass dictionary via JSON dump, pass numpy array as bytestrings, 150 Redis example, 147 Redis with Jupyter, ecosystem, engine, 34 containers/processes, 76 file server via Docker container, 46 Hello, Docker!, 42 heterogenous infrastructure, 29 host, 34 image and container, 34 Linux, 36 Mac, 40 networking, 45 new AWS t2.micro for, 153 non-root user, 39 registries hold images, 35 repository, SimpleHTTPServer, 45 toolbox, 41 Ubuntu system, 36 Windows, 40 Docker Compose application, 181 AWS application, ch9jupyterredis_default, 183 complete the computation, creation, directory, 182 docker-compose down, 186 docker-compose.yml, 182 docker engine, 183 Docker Hub registry, 182 install, jupyter and postgres, jupyter environment variables, 185 Jupyter Notebook server, 186 jupyter_redis, 183 multi-container systems, 179 networks, Python project, 180 redis, 186 requirements.txt file, 180 restarting, 199 system administrator/operations engineer,

3 this_jupyter service, 182 t2.micro, 186, toolset, 179 versions, 181 Dockerfiles Anaconda, 88 ARG and MAINTAINER, 89 commit changes, 86, 92 creation, 81 Docker build cache, 87 environment variable ENTRYPOINT, 97 ENV, 97 run the build, 97 FROM, gsl image, 84 idempotent, 91 ipython image build, 99 conda, 98 definition, 98 designing, 98 GitHub, 100 new container, 100 runtime command, 99 source directory, 98 joshuacook/gsl image build, 87 local repository, 90, 94 commit changes, 97 run the build, 96 tini, microservices, 81 miniconda3 image, 92 design, 88 local repository, 98 source directory, 89 miniconda installation, 95 repo building images, 83 GitHub, 83 local development, 82 syntax, 83 run build option, RUN instruction, 92 run the build, 95 single-concern containers, 82 stateless containers, 81 Domain-specific language (DSL), 35 E, F Elastic Compute Cloud (EC2), 2 Engine, 71 alpine Docker image, 72 ecosystem bootstrap time, 77 containerized, 76 Hello, World, 74 implicit vs. explicit pulls, 73 workstation, 71 Ephemeral container extension, 130 ephemeral packages, 130 ipython magic shell process command, 131 library installation, package via pip, 133 root environment, 132 semi-persistent changes, 134 Twitter installation, 133 G GitHub, 83 GNU Scientific Library (GSL) apt-get update && apt-get install, 86 build image, 85 creation, 84 definition, 84 FROM gcc, 85 LABEL maintainer=@joshuacook, 86 source directory, 84 H Hub alternative public registries, 103 containers, 115 ID and namespaces, 104 image pushing, 111 joshuacook/numpy repository, 111 jupyter/base-notebook image, 117 local cache, 114 local directory and context subdirectory, 108 numpy-notebook, 108 pull image, 113 push command, 112 repositories,

4 Hub (cont.) repository creation, 110 search existing repositories, 105 server-side application, 103 tagged image, 106, 118 I ID and namespaces, 104 Infrastructure as code (IaC), 179 Interactive application, 49 bessel.py, 57 Jupyter (see Jupyter) launch ipython, 55 persistence, 56 Interactive software development, 213 computational biology projects, organizing, 214 database requirements, managing project via git, database to application, adding, data science-specific, 213 default rails application, 213 delayed processing to application, framework, 214 function s docstring, 233 initialize the project, modules, 232 postgres module, Python module, PostgreSQL, 233 project framework, 215 project root design pattern, , 233 Python module, Jupyter, this_postgres works, 234 J, K joyvan, 130 Jupyter, 49 Bessel function, 51 calculation, 51 container build context, creation, 188 data persistence, 190 docker ps, 195 docker stats, 194 environment file, OAuth object, 193 service and data volume, 187 t2.micro, 193 Twitter authentication, 193 TwitterStream, 194 C programming language, 49 execute compiled binary, 54 jupyter.env, 189 minimal computational project, 50 mongo_data, 190 notebook, 57 data persistence, 65 demo stack, 58 detached mode, 63 file system, 61, 68 foreground mode, 60 jupyter/demo image, 59 port connections, 63 port mappings, 65 Python code, 61 quadratic curve, 66 random time-series data, 62 server, 57 via localhost, 65 volume attachment, 69 opinionated Docker stacks, 57 Python module, source code, 53 Jupyter Notebook server, 137, 217, 218 L Language model, spacy and en, 194 Libraries installation, 130 Linux, 36 Linux Containers (LXC) project, 30 M Mac, 40 MongoDB, 137, 151 container count tweets, 195 docker-compose build, 191 docker-compose.yml, 192 jupyter_mongo, 192 single tweet, 195 data volume, create and view, insert tweets,

5 installation, verifying, 155 with Jupyter, mongo and Twitter, 158 new AWS t2.micro, 152 new AWS t2.micro for Docker, 153 as persistent service, pull the mono image, 153 pymongo, structure, 156 tweets by geolocation, Twitter credentials, N Notebook images conda environment, 125, 127 default environment, 124 high-level overview, 121 jupyter/base-notebook, 122 jupyter/scipy-notebook image, 128 kernel selection, 126 Python versions, 125 security, 122 semantic_analysis image, 127 stack dependency graph, 121 O Opinionated Jupyter stacks, 119 Linux/Mac/Windows, 120 notebook server, 120 port mapping, 120 -P (publish all) flag, 119 scipy-notebook server, 119 toolbox, 120 P, Q PostgreSQL, 137, 164 binary type and numpy, connnecting containers by name, database, 217 Docker container networking, installation, verifying, with Jupyter, 174 Jupyter, pandas and psycopg2, 174 Jupyter-PostgreSQL connection, loading data, minimal verification, new data volume, creating, 165 as persistent service, 166 postgres image, pull, 165 Python modules, 216 demo, 216 lib/ directory, 217 pickle module, 139 program, 216 updating, using Jupyter, R Read-eval-print loop (REPL), 55 Redis, 137, 139 example, 147 with Jupyter, as persistent service, pull the image, S Secure shell (ssh), 2 Serialization, 137 binary encoding in Python, 139 formats and methods, 138 T Tagged images, 106 build command, 107 official repositories, 108 Python image, 107 subtle changes, 106 U Ubuntu system, 36 V Virtualization method, 30 Virtualized hardware, 30 Virtual machine (VM), 30 W, X, Y, Z Windows,