Docker in Appsoma PaaS. Containers FTW

Transcription

1 Docker in Appsoma PaaS Containers FTW

2 Intro Who? Ken Demarest - CTO of Appsoma, 30 yr coder What? Scientific Computation platform

3 What is Appsoma? A PaaS, typically for complex runs on large data Beachhead market is omics Builds on LXC/Docker to support: Reproducible environments Isolated, fast boots

4 Prediction: Containers become the unit of deployment Not executables Not install scripts Not package installs Not VMIs - portability, heavy weight Everything simply works: Container on the kernel But...

5

6 Containers - fast boot, small footprint

7 Why so small? Why fast boot?

8 Under the Hood... Basics LXC - Process Sandboxing Libvert - VM management interface selinux & apparmor for security netlink & netfilter for net mgt & isolate Docker Portable deployment: abstract networking, storage, logging, distro, etc. Application-centric: API, user interface, design philosophy Versioning w/diffs + component re-use Public Registry

9 Future of (most?) Deployments BUT you re probably still wrestling with: - Scripting installs, or - reverting images - Storage

10 Quick Tangent: posix/block/object Object Stores are seeing a new rise Ceph and HDFS (etc) over... Software: leveldb, RocksDB Hardware: Fusion NVMKV, Seagate Kinetic Advantages Faster access by going direct Reduced TCO, due to less maintenance and less hardware Non-relational, but some do allow: indexes metadata

11 Ex: Seagate Kinetic

12 OpenStack unlearns what it has learned Nova is the Cloud Computing Fabric Control System - eg for compute Driver for Docker in OpenStack Compute (Nova) Docker container used as if it were a virtual machine. Learning: Docker is not a VM,

13 OpenStack - Orchestration the new way Not constrained by the OpenStack APIs. May leverage the full power of the Docker Remote API.

14 What does Docker in Heat mean? heat_template_version: description: Simple template to deploy a single compute instance resources: my_instance: type: OS::Nova::Server properties: key_name: my_key image: F18-x86_64-cfntools flavor: m1.small... resources: my_instance: type: OS::Nova::Server properties: key_name: ewindisch_key image: ubuntu-precise flavor: m1.large user_data: #include my_docker_container: type: OS::Heat::Docker docker_endpoint: { get_attr: [my_instance, first_address] } image: cirros

15 The Appsoma Soma Momentary Runs Code Data vs Persistent Runs ENV OS + Packages

16 The Appsoma Soma Linux too old? Easiest with Docker/LXC Code Data ENV OS + Packages No root access? Appsoma supports preambles of Chef, Puppet, or custom script, decorators. Yikes.

17 Pipeline of Somas 3 CODE 1 DATA 2 5 ENV CODE DATA CODE DATA CODE ENV ENV DATA 4 ENV CODE DATA ENV

18 Somas w/ Docker for Run Anywhere App Store Implied Manifests I want to collaborate... use their app on my sys... switch providers... burst to cloud Code + Env (Docker) HPC Cloud

19 Somas w/ Docker for Reproducibility Soma Archives HPC I want to validate a result... tweak a prior run... get through regulatory... publish with high integrity Code + Env (Docker) + Data Cloud

20 Cloud Public Tools Broad Etc. Public Data 1000 Genomes Amazon Appsoma Public - Run Anywhere - Reproduce - Secure - Data Management - Scale Private Google Public Private Etc. HPC 3p YourCo PartnerCo Third Party Users Devs Appsoma Agents Code Data Lab & HPC Developers Data

21 Appsoma is unlearning too... O/S (eg Ubuntu, CoreOS, Appsoma Agent Client (browser) Head Node Docker Container (with IP) If booted : Appsoma Agent (self-updates) (registers in DNS) Code Repo Env Repo Data Repo Otherwise, empty Learning: Use Docker as thin as possible

22 Dev pattern we ve seen Pick Base Container - boot it to shell Install Packages (best guess) Try to run (disciplined install of your code - don t map!) a. Fail? Tweak install & repeat Burn Try with this burn BUT You might slowly corrupt your environment SO Keep your code s reach minimal, if possible Consider Puppet/Chef style setup, for purity

23 Secrets of -v docker run -v /etc/passwd:/etc/passwd -v /etc/shadow:/etc/shadow The user and guest IDs are the SAME in the guest as they are in the host -v /etc/group:/etc/group -u someuser -i -t somecontainerid some_command Without this, your files will all write as root

24 Docker diff - example $ sudo docker diff 7bb0e258aefe C /dev A /dev/kmsg C /etc A /etc/mtab A /go A /go/src A /go/src/github.com A /go/src/github.com/dotcloud A /go/src/github.com/dotcloud/docker D /something

25 Debugging - attach, etc. * Appsoma maps DNS to it * Maps port(s) * Run an agent that shells Client (browser) VT 100 Appsoma Agent (attach) * sudo docker attach $ID * Yes, you can ssh to an sshd * Attaching will end if the master process ends * Remote logs with -v to local or mapped drive * Use -p to map port(s) for live logs

26 Secrets of lxc-attaching To spawn a new shell running inside an existing container, use lxc-attach -n container To restart the cron service of a running Debian container, use lxc-attach -n container -- /etc/init.d/cron restart ** This is for debugging. Don t defeat isolation by re-using a container!

27 Challenges Repo API still rough Requires 3.2 (or 3.8) modern linux kernel Multi-tenancy security relatively new

28 Contact Info Ken Demarest CTO, Appsoma