globus online The Galaxy Project and Globus Online

Transcription

1 globus online The Galaxy Project and Globus Online Ravi K Madduri Argonne National Lab University of Chicago

2 Outline What is Globus Online? Globus Online and Sequencing Centers What is Galaxy? Integra;ng Galaxy and Globus Online Why? How? Demo Future Work Q & A 2

3 3

4 Benefits of Globus Online Reliable file transfer. Easy fire and forget file transfers Automa;c fault recovery High performance Across mul;ple security domains No IT required. No client sopware installa;on New features automa;cally available Consolidated support and troubleshoo;ng Works with exis;ng GridFTP servers Globus Connect solves last mile problem I moved 400 GB of files and didn t even have to think about it. Lawrence Berkeley Na.onal Lab It s just not a big deal to move big data anymore. Ini.a.ve for Biomedical Informa.cs Fantas.c! I have started using globus connect to transfer data, and it only took me 5 minutes to set up. Thank you! NERSC user 4

5 DNA Sequencing Costs are Declining 13 years $300 million ~1 week $1,000 5

6 Sequencing Data is Increasing Source: Pennisi, E., Science :6018 pp

7 PredicGon of Number of Sequenced Genomes 1,000 Genomes Learning the Ropes 25 Million Genomes A Brave New World 250,000 Genomes Clinical Early Adop;on 5 Million Genomes Consumer Reality Source: Resnick, Richard, Implica;ons of exponen;al growth of global whole genome sequencing capacity. GenomeQuest. July 9, Retrieved April 7,

8 Current CompuGng Infrastructure Won t Support Increasing Data Will Computers Crash Genomics? Pennisi, E., Science :6018 pp. 666 The various members of the genome informa;cs ecosystem are now facing a potengal tsunami of genome data that will swamp our storage systems and crush our compute clusters. Stein, L.D., Genome Biology :5 pp. 207 CREDIT: ALVARO ARTEAGA/ALVAREJO.COM. 8

9 Sequencing Center Workflow Delivery of biological sample to sequencing center by scien;st Data deliver to scien;st. Data produc;on by sequencing center 9

10 ABI SOLiD Sequencing Instrument 1.2 TB of data every 7 to 14 days 2 to 1600 files 1 to 16 customers File Transfer = Product Delivery 10

11 Current File Transfer Methods 11 11

12 Current File Transfer Methods providers and their customers open resort to the sneaker net : overnight shipment of data-laden hard drives. Pennisi, E., Science :6018 pp

13 Globus Online Improves Current Methods providers and their customers open resort to the sneaker net : overnight shipment of data-laden hard drives. Pennisi, E., Science :6018 pp

14 Why Globus Online? Fire- and- forget usage Re- trying failed transfers Logs to iden;fy reasons behind failed transfers Simplicity Simple logon and authen;ca;on Web interface for execu;on and monitoring Globus Connect for sequencing facility endpoint Reliability Checksum op;on (~$10K/genome) Performance Secure enablement Now, with Globus Online, the process is trivial and our scien.sts can move data to the right loca.on with just a few clicks. File size is no longer a barrier to produc.vity. Ini.a.ve for Biomedical Informa.cs 14

15 Globus Online at UC Sequencing Facility Mac using Globus Connect Mount drive Delivery of data to customer ibi File Server Sequencing Instrument 15 ibi General Purpose Compute Cluster Sequencing Specific Compute Cluster

16 Extend to Other Sequencing Centers Over 300 public sequencing centers 16

17 Extend Beyond Sequencing Centers Future Use Case = Biologists Moreover, as so- called third genera;on machines which promise even cheaper, faster produc;on of DNA sequences (Science, 5 March 2010, p. 1190) become available, more, and smaller, labs will start genome projects of their own. Pennisi, E., Science :6018 pp

18 What about Analysis of this Data? 18

19 Enter Galaxy A free (for everyone) web service integra;ng a wealth of tools, compute resources, terabytes of reference data and permanent storage Open source so^ware that makes it easy to integrate your own tools and data and customize your own site Flexible architecture - > Customizable 19

20 Galaxy AdopGon ~50 deployments of Galaxy Galaxy for MicroArray analysis, Machine Learning, Drug Discovery etc ~130,000 jobs a month and growing on the public instance of Galaxy 1 TB/week in user uploads 60TB from China 150+ aqendees in the Galaxy users conference From 6 con;nents 20

21 Globus Online and Galaxy Transferring large quan;;es of data in and out of Galaxy reliably Downloading results to user s laptop Running Galaxy in Produc;on for mul;ple users Running analysis on a cluster in the user space Federated Iden;ty management Flexible group management system (VO) Mechanisms to share data and workflows Parallel execu;on Condor and SwiP 21

22 What works Iden;ty Management Used Globus Provision to bootstrap the provisioned ec2 cluster with GO creden;als so one can put data from GO- enabled endpoints into galaxy and script transferring data from galaxy to any GO- enabled endpoint Data Management Added a GO Transfer task to Galaxy Ability to script the transfer tasks along with the workflow Execu;on Management We know more about this than anything Condor runner for galaxy: The executable along with command line op;ons get passed to a condor runner (runner is galaxy term for execu;on mechanism) Group Management Metadata Management 22

23 Automated deployment of Galaxy clusters on EC2 Uses Globus Provision An NFS server providing global directories for home directories, sopware, and scratch space. An NIS server providing an authen;ca;on domain within the cluster. The list of users must be specified when the cluster is created A GridFTP server, which is automa;cally registered as a GO endpoint. A Condor pool. The number of worker nodes is fixed at the ;me the cluster is created. A Galaxy server with the modifica;ons outlined above. Each user in the cluster has a user cer;ficate signed by the Globus Provision CA (this CA is trusted by GO). If a GO username matches a cluster username, and that user s Globus Provision cer;ficate is uploaded to GO as a trusted cer;ficate, then the user will be able to transfer files to/from the cluster s endpoint (and will also be able to use Galaxy s GO transfer tasks; this further requires that the user s public name in Galaxy matches his GO and cluster username). 23

24 GO Galaxy DescripGon The deployment of clusters on EC2 is completely automated, and we can deploy fully- func;onal Galaxy clusters on EC2 within minutes We created galaxy runners for Condor and SwiP So the applica;ons are run on a worker node instead of the galaxy node Reusable CHEF recipes to provision produc;on galaxy instances on demand on EC2(- ish) clusters 24

25 IntegraGon of Globus Online with Galaxy 25

26 Demo of Current CapabiliGes 26

27 Globus Online: Part of the CompuGng Infrastructure to Support Genomics CREDIT: ALVARO ARTEAGA/ALVAREJO.COM 27 27

28 Globus Online: Part of the CompuGng Infrastructure to Support Genomics CREDIT: ALVARO ARTEAGA/ALVAREJO.COM 28

29 GO Galaxy Future CapabiliGes Integrate flexible Globus Online iden;ty management, group management with Galaxy Ability to create a VO with the above capabili;es (User/Groups management, File system, Transfer, HTC, Parallel execu;on, EC2 Cluster, data sharing) with click of a buqon Integrate with Globus Online on- demand storage solu;on Ability to easily share data, workflows within a Virtual Organiza;on 29

30 References More details about Globus Online can be found here: hqp:// Details on Galaxy: hqp://usegalaxy.org Details on work we did integra;ng Globus Online and Galaxy: bit.ly/qdohhw 30

31 globus online Thanks! Questions? (or on twitter)