Utilizing Container Technology to Streamline Data Science

Transcription

1 Utilizing Container Technology to Streamline Data Science National Laboratory Information Technology May 22, 2018 Dr. Amanda J. Minnich This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA Lawrence Livermore National Security, LLC

2 Who am I? Staff Research Scientist at LLNL Co Lead of ATOM s Data Driven Modeling Team Data Scientist for ~6 years MS and PhD in Computer Science from the University of New Mexico BA in Integrative Biology from UC Berkeley Docker and Kubernetes User/ Administrator for 10 months 2

3 The Rest of the Team Jonathan Allen Computer Scientist leading the machine learning effort on the ATOM project. Works on all aspects of Docker/ Kubernetes/ Data services Ted Stirm DevOps lead on Docker/ Kubernetes deployment Jeff Mast Developer, provides help with deployments and day today DevOps support Mike Goldman Head of LLNL Data Science Institute and advocate for containerized setup Jim Morton sys admin who set up server They are happy to talk about their specific areas of expertise to anyone interested in deploying a similar system! 3

4 Roadmap Motivation Requirements and Tools Containers (Docker) Kubernetes JupyterLab Data services Conclusion 4

5 MOTIVATION Why are we trying to find a new way to enable data science research at LLNL? 5

6 ATOM Consortium Public private consortium whose goal is to innovate the drug discovery pipeline Founding Members: LLNL GlaxoSmithKline Frederick National Lab (National Cancer Institute) UCSF Sit at a central office in San Francisco Bringing together people with very different backgrounds Unique group that imposes unique needs 6

7 Current drug discovery: long, costly, & high failure Is there a better way to get medicines to patients? Molecules Target Human relevant in vitro, in vivo experiments Demonstration of clinical effect and value Preclinical issues: Average time: 5.5 years 33% of total cost of medicine development Millions of molecules tested, 1000s made, and most fail Clinical success rates still only 12%, indicating poor translation in patients Source: 7

8 Patients are waiting A convergence of catalysts points to now as the time to change drug discovery The Opportunity: Ending Cancer as We KnowIt Whilebold,thegoaloftheCancer Moonshot is within our grasp. Today, society has the benefit of decades of scientific understanding and vast amounts of rich data just waiting to be transformed intosolutions. Science Translational Medicine 12 Aug 2015: Vol. 7, Issue 300, pp. 300ps17 DOI: /scitranslmed.aaa moonshot Mol. Inf. 2016,35, 3 14 The exponential increase of computation power enabled with exascalewill fuel a vast range of breakthroughs and accelerate discoveries in national security, medicine, earth sciences and many other fields. 8

9 ATOM Consortium Accelerating Therapeutics for Opportunities in Medicine Mission To accelerate the development of more effective therapies for patients High performance computing Vision How? Diverse biological data Emerging biotech capabilities A new starting point: Transform drug discovery from a slow, sequential, and high failure process into a rapid, integrated, and patient centric model 3 year goal Integrated precompetitive platform 1 year From target to patient ready in <1 year 9

10 The ATOM platform vision Moonshot goals In silico and complex in vitro models reduce reliance on nonhuman models Empirical testing is only a tool to validate and optimize computational predictions Effects can be predicted at the protein, cell, tumor, tissue, organ and organism level Regulatory files can be supported by in vitro data and in silico predictions 10

11 Accelerated drug discovery concept Vision of ATOM workflow in practice ATOM Workflow design synthesize Released to public after a 1 year member benefit Empiric al Patient specific data and samples input to workflow to develop new therapeutics simulate In Silico active learning assay PK Safety Efficacy Open source models and generated data Therapeutic Candidates Members use workflow for internal drug discovery Models of drug behavior in humans Commercialization by members for patient benefit 11

12 REQUIREMENTS AND TOOLS What are ATOM s needs, and what tools satisfy these needs? 12

13 Reproducibility is key ATOM hopes to convince the FDA to accept in silicomodels as a partial replacement for in vitro(bioassay) and in vivo (animal testing) assays for potential cancer drugs Any erroneous results could have serious effects on people s health This means we need to be able to trace the exact processing and modeling paths and reproduce any and all results Must save train/test dataframes, models, parameter metadata One thing that breaks reproducibility: software updates 13

14 However, data science tools change rapidly 14

15 These releases can contain critical new features 15

16 ATOM s end users Data Science Team: 2 Computer Scientists, 2 Statisticians, 1 Bioinformatician ~10 Computational Chemists/Biologists ~10 Pharmacology Researchers Those with Linux/ HPC experience don t necessarily have data domain knowledge Those with data domain knowledge are not familiar with terminals, slurm, ssh, etc. Team members from GSK rotate out after a year So having an easy to access development server is key 16

17 ATOM s data 300 GB of a variety of bioassay and animal toxicology data on ~2 million compounds from GSK Proprietary or sensitive data must only be stored on approved servers Absolutely no laptop storage Hoping to have other pharma partners join Data may remain sequestered from competing pharma members Data is messy and complex, so we want to access it using data services Also have to store and tag output data from models and simulations (~PBs) 17

18 Prior system at LLNL Current system: Livermore Computing (LC) Uses slurmto manage job deployment onto computing nodes Only access via ssh(no GUI access for non tech folks) No data services provided No sudo privileges Potentially long wait time for installation of new libraries Update of libraries with no preservation of old environments 18

19 Summary of Requirements Requirements Environment consistency Easy software updates Orchestration of resources (including GPUs) Role based access to mounted volumes Intuitive GUI Data services (SQL, nosql, metadata annotation) It was clear the existing setup would not support these requirements, so we had to look elsewhere for solutions. 19

20 Requirements and Tools Requirements Environment consistency Easy software updates Orchestration of resources (including GPUs) Role based access to mounted volumes Intuitive GUI Data services (SQL, nosql, metadata annotation) Tools Containers Containers Kubernetes Kubernetes JupyterLab Containers + RESTful API 20

21 Requirements and Tools Requirements Environment consistency Easy software updates Orchestration of resources (including GPUs) Role based access to mounted volumes Intuitive GUI Data services (SQL, nosql, metadata annotation) Tools Containers Containers Kubernetes Kubernetes JupyterLab Containers + RESTful API 21

22 Features of containers Containers allow you to: Package up code and dependencies together Create new environments with different software versions without needing server admin privileges Version a file system via tags, ensuring traceability of work and reproducibility of results Much more lightweight than VMs: can put lots of them on a single host operating system Underlying philosophy: if it works on my machine it will work on yours. 22

23 Containers vs. Virtual Machines VM Stack Container Stack Traits VM Container Virtualization Hardware Level Operating System Level Base Image Size GBs MBs Boot Times Seconds* <100ms* Resource Allocation Static Dynamic 23

24 What is docker? docker is a an open source project based on Linux containers. Basically acts as helpful tool for packing, shipping, and running applications withinthese containers. Adheres to the Open Container Initiative (OCI), which means it is supported by standard container orchestration tools like Kubernetes Caches image layers, saving huge amounts of time for developers Source: for data science 4901f35d7cf9 24

25 Docker terminology Dockerfile Text document with all the commands you would normally use to create an image Image a template/blueprint for creating containers Container Runtime instance of an image Tags Applied to Docker images in a repository. This is how to distinguish between images in a repository. (version numbers) Repository Collection of docker images with same name but different tags Registry Online system that hosts repositories 25

26 Docker registry We use Portusto host and access our registry Open source authorization service and user interface that hosts our private docker registry and handles user authentication and group management 26

27 Containers are ideal for two of ATOM s requirements 1. Environment consistency: Image tags allow tracking of different software version environments For example: atom app:deepchem1.2, atom app:deepchem1.3, atom app:deepchem Easy software updates: To install a new version of a library, simply update dockerfile, rebuild, and push image Allows installation of new software without having root on the remote server 27

28 Requirements and Tools Requirements Environment consistency Easy software updates Orchestration of resources (including GPUs) Role based access to mounted volumes Intuitive GUI Data services (SQL, nosql, metadata annotation) Tools Containers Containers Kubernetes Kubernetes JupyterLab Containers + RESTful API 28

29 Orchestrating containers Once you have your containers, you need to handle: User access Directory access Resource access Docker compose works for local development, but to coordinate many users and many resources, need a more heavy duty orchestrator We chose to use Kubernetes for this task 29

30 What is Kubernetes? Kubernetes is an open source container orchestrator, used to manage containerized workloads and services Combines simplicity of Platform as a Service (PaaS) with flexibility of Infrastructure as a Service (IaaS) Developed by Google, who has been running containers in production for over 10 years 30

31 We use Kubernetes to: Orchestrate the allocation of compute resources: CPU, GPU, and memory Manage access to container based data services Handle Role Based Access Control (RBAC) to various mounted volumes 31

32 Can mount external directories into container 32

33 kubectl Source: commands cheat sheet/ 33

34 Requirements and Tools Requirements Environment consistency Easy software updates Orchestration of resources (including GPUs) Role based access to mounted volumes Intuitive GUI Data services (SQL, nosql, metadata annotation) Tools Containers Containers Kubernetes Kubernetes JupyterLab Containers + RESTful API 34

35 Intuitive GUI Many of ATOM s team members do not have a background in computing However, they are the domain experts Need to be able to view raw data files, generate visualizations, and compute basic statistics Need access to a variety of Python/R libraries Would like to be able to share code/pages with each other Want it to be accessible through a web page, rather than an ssh session 35

36 JupyterLab JupyterLabrunning out of a container: Provides an interactive GUI via a URL Has access to the notebook environment containing the desired packages Has access to a terminal window to push/pull from Git Has text editor window for editing code or viewing files 36

37 Anaconda simplifies installation of data science libraries Anaconda is a freemium open source distribution of the Python and R programming languages for large scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment. We use Anaconda inside docker containers to package up desired libraries Helps to handle any dependency issues for installation of Python and R libraries 37

38 Bokeh allows for interactive plots using Python 38

39 Requirements and Tools Requirements Environment consistency Easy software updates Orchestration of resources (including GPUs) Role based access to mounted volumes Intuitive GUI Data services (SQL, nosql, metadata annotation) Tools Containers Containers Kubernetes Kubernetes JupyterLab Containers + RESTful API 39

40 Data Services Data services are required to organize both raw data and metadata annotations Also need to save model ready train/test datasets, serialized models, and simulation output Containers enable services to be deployed quickly and easily by the data scientists themselves Have data services that use SQL and nosqldatabases as back ends Accessible via a client that uses a RESTful interface Allow the end user to write database agnostic code 40

41 ATOM s data services In house datastore service Allows for association of complex metadata with any type of file mongodb Used as backend for datastore Initially helpful to have more unstructured database option MySQL Much of the GSK data is in CSV files, which fits well with relational framework Probably will expand to other relational dbs 41

42 Overall structure of data services Backend Services deployed via containers NoSQL SQL Object Store Application Services MLMR, CCR, CCDE, MLMT... Web Metadata Application Services Services Server APIs (secure REST interface) Machine Learning Apps Tensorflow,... Application Client APIs (Python, R, etc.) Large Scale Data Science Apps (HPC) Interactive Data Science Apps (Jupyter/ Browser) 42

43 Datastore Maintain controlled data access through user groups Supports basic versioning of data files Allows for direct reference to files to support data stored on physically separated drives Keyword search for files based on metadata tags Accessible both via online GUI for domain experts and a RESTful API for programmatic access 43

44 Enables metadata annotation and search User creates metadata JSON file describing the contents of the file Supports search and retrieval of data files based on metadata file, but not on contents of the file Provides an alternative to maintaining a giant README and pointers to directories, subfolders, and files Key benefit is storing and tracking programmatically created datasets that tend to lead to large heterogeneous file sets 44

45 Datastore uses buckets and collections Buckets Set access control and can be public or private Project or team based via authorization keys Can prevent downloads for certain buckets Can be thought of as high level directories Collections Group together and organize sets of data objects (or files) Useful when operating on a set of data files 45

46 Requirements and Tools Requirements Environment consistency Easy software updates Orchestration of resources (including GPUs) Role based access to mounted volumes Intuitive GUI Data services (SQL, nosql, metadata annotation) Tools Containers Containers Kubernetes Kubernetes JupyterLab Containers + RESTful API 46

47 CONCLUSION Lessons learned, successes, and future work 47

48 Requirements for success DevOps support Team members can train up, but there will be problems that require deep knowledge and more advanced skills Sys admin support Getting this setup to play nice with a server comes with its own set of challenges Team members willing to spend time acting as tech support Need people to help troubleshoot issues or no one will be willing to adopt this workflow 48

49 What our setup looks like: my perspective Get request via Jira, , Slack, Jabber, or in person Container exists? DEBUG Create configuration files for pod Create Dockerfile DEBUG Create and start pod on remote server as root Build image on build server or laptop TROUBLESHOOT Keep an eye on status and follow up with user Push to registry 49

50 Helpful resources Jira Gives users a central place to report issues and request help Separate development and application pages Confluence Documentation saves you from sending out the same instructions over and over Shell scripts Everyone knows this, but putting time into writing automation scripts is worth the effort 50

51 Challenges Had to learn Docker and Kubernetes We were facing a learning curve ourselves while trying to assist others Did not have dedicated DevOps/sys admin help for the first 6 months Supporting users with different levels of IT skills Other projects share (and occasionally break) the server Have to handle LLNL security concerns (certs are a pain) 51

52 Successes Have the entire ATOM Data Driven Modeling Team working out of containers Can get new team members up and running in a day Have generated ~1k deep learning and random forest models Have ~20 notebooks being shared with data intake, processing, curation, and visualization capabilities Domain experts have begun interacting with data through these notebooks Computational biologist is building a Data Explorer webpage with a mongodb backend Have ~10 domain experts putting data in datastore with curated metadata tags 52

53 Future plans Install kubespawnerfor JupyterHubso team members can launch containers without needing to run kubectl Model zoo: formalized pipeline for reading in model ready datasets and outputting serialized models with hyperparameters used and performance metrics calculated Active learning loop that incorporates the datastore Eventually would like to have data science cluster that is separate from the HPC systems, but can sit inside same network 53

54 Thanks for your time! Questions?