RStudio in Docker. Mark McCahill OIT. Monday, September 15, 14

Transcription

1 RStudio in Docker Mark McCahill OIT

2 problems Intro to Statistics courses use RStudio (a web-based IDE for the R statistics package) 300+ students, at least one assignment/week Externally hosted RStudio service requires Google Gmail credentials for authentication

3 and more problems Imagine getting GMail credentials from 300 students -- shibboleth authentication is very desirable Negotiations are required to get custom package suites/libraries for RStudio on the shared hosted instance Can we have a Duke instance of RStudio?

4 OK... but how? Monolithic RStudio server instance? scaling issues? load prediction? security consideration: multi-user systems where users can install arbitrary code can be prone to an unfortunate series of events with broad impact

5 personal VMs? One VM per user? resource intensive the only things that really vary are the state of the computation and data the users store

6 Docker to the rescue Many of the virtues of individual VMs, but much more lightweight transparently migrate users across backend infrastructure sequester students as much as possible flexible

7 Architecture shibboleth authentication user RStudio-Host-1 port homedir100 port homedir101 port homedir102...etc... rsync homedirs VM-manage map user to their assigned Docker container redirect them to the container host/port RStudio-Host-2 port homedir200 port homedir201 port homedir202...etc... RStudio-Host-3 port homedir300 port homedir301 port homedir302...etc... backup mule TSM backups

8 RStudio Docker farm 3 servers (8 GB RAM, 2 CPU, 300 GB disk) 110 user RStudio containers/server ~52 MB memory user/container Servers have about 50% free memory Every 5 minutes rsync docker user volume home directories to a backup mule server

9 demo

10

11 just one more thing...

12 RStudio farm: phase II Google has provided a $10,000 grant of Google Compute cycles to this project Containerized workloads mean you can move the computation to where it is the most cost-effective Alternatively, you might use the cloud for handling peak loads/usage spikes

13 Architecture: phase II shibboleth authentication user Ubuntu RStudio-Host-1 RStudio-Host-2 port RStudio-Host homedir200 port homedir100 port homedir101 port port port homedir102 + homedir201 homedir300 port port etc homedir202 homedir301 port etc... + homedir302...etc... rsync homedirs VM-manage map user to their assigned Docker container redirect them to the container host/port CoreOS Google-Host-1 Google-Host-2 Google-Host-3 port homedir100 port port homedir101 homedir200 port port port homedir102 + homedir201 homedir300 port port etc homedir202 homedir301 port etc... + homedir302...etc... backup mule TSM backups

14 managing containers phase I 3 servers (Ubuntu) - a shell script launches containers phase II 3 more servers (CoreOS) servers at Google - a shell script launches containers

15 managing containers phase III single CoreOS cluster spanning onpremises and Google servers - etcd to hold config and status - fleet to manage containers

16 Architecture: phase III shibboleth authentication VM-manage map user to their assigned Docker container redirect them to the container host/port user CoreOS + fleet + etcd RStudio-Host-1 RStudio-Host-2 port RStudio-Host homedir200 port homedir100 port homedir101 port port port homedir102 + homedir201 homedir300 port port etc homedir202 homedir301 port etc... + homedir302...etc... Google-Host-1 Google-Host-2 Google-Host-3 port homedir100 port port homedir101 homedir200 port port port homedir102 + homedir201 homedir300 port port etc homedir202 homedir301 port etc... + homedir302...etc... rsync homedirs backup mule TSM backups

17 next steps Apply what we learned with RStudio to other popular compute environments CoreOS cluster(s) to run Docker containers at scale Dynamic cloud bursting?

18 lessons Dockers incremental builds help manage messy installs with hard-to-fathom dependencies (like R libraries that need random TeX libraries) Containers are ephemeral, persistent data must live outside the container Use Docker volumes to map data directories into the container where needed

19 lessons Don t assume that you can mount users NFS or CIFS home directories - you might want to run in a cloud or somewhere outside Duke Containers are mobile: you will need a way to map/direct users to the current location (i.e. host and port)

20 more info RStudio docker container source at gitorious.oit.duke.edu