Galaxy a community driven platform for accessible, transparent, and reproducible data science

Transcription

1 Galaxy a community driven platform for accessible, transparent, and reproducible data science / #usegalaxy

2 A continuing crisis in genomics research: reproducibility

3 What is reproducibility? (for computational analyses) Reproducibility is not provenance, reusability/ generalizability, or correctness Reproducibility means that an analysis is described/captured in sufficient detail that it can be precisely reproduced (given the data) Yet most published analyses are not reproducible (see e.g. Ioannadis et al /18 microarray experiments reproducible; Nekrutenko and Taylor 2012, 7/50 resequencing experiments reproducible) Missing software, versions, parameters, data

4 Galaxy: accessible analysis system

5 A free (for everyone) web service integrating a wealth of tools, compute resources, terabytes of reference data and permanent storage Open source software that makes integrating your own tools and data and customizing for your own site simple An open extensible platform for sharing tools, datatypes, workflows,...

6 Galaxy s ideological goals: How best can data intensive methods be accessible to scientists? How best to facilitate transparent communication of computational analyses? How best to ensure that analyses are reproducible?

7 Galaxy s practical goals: How to arm researchers with access to powerful compute and latest tools How to build a community of tool developers How to run Galaxy on any HPC

8 Describe analysis tool behavior abstractly Workflow system for complex analysis, constructed explicitly or automatically Analysis environment automatically and transparently tracks details Pervasive sharing, and publication of documents with integrated analysis

9 Visualization and visual analytics

10 To grow Galaxy we needed to actively engage the community Users: support the tools and workflows they want, engage in outreach, training, infrastructure to support communication (e.g. biostar) Developers: make it easy to integrate new tools, to run and test locally, be as flexible as possible, avoid anything specific to a given scientific domain Great value in being able to run anywhere, not building a gateway but a framework to build gateways

11 The Galaxy ecosystem

12 More than 70 known public Galaxy servers 15+ general servers Domain specific servers including: Ballaxy for structure based computational biology, Cistrome for regulatory sequence analysis, Genomic Hyperbrowser: statistical integration of genomic data, GigaGalaxy: integrating workflows published in GigaScience, Pathogen Portal:comparative analysis of host response to pathogens,... Dozens of large scale private Galaxy instances

13

14

15

16

17 Ways to use Galaxy The public web service at Install locally with many compute environments Deploy on a cloud using Cloudman Atmosphere

18 Galaxy can scale: for example Galaxy main Dedicated resources Shared resources SDSC, San Diego Galaxy Cluster 256 cores 2 TB memory Rodeo 128 cores 1 TB memory Corral/Stockyard 20 PB disk Stampede 462,462 cores 205 TB memory PSC, Pittsburgh Blacklight 4,096 cores 32 TB memory Trestles 10,368 cores 20.7 TB memory TACC Austin Nate Coraor

19 Bringing it all together: automate all the things! Unified ansible playbook for Galaxy main, cloud, and local deployments

20 Collaboration among Galaxy instances

21 Galaxies on private clouds Galaxies on public clouds Galaxy Tool Shed Private Tool Sheds private Galaxy installations Greg von Kuster, Dave Bouvier

22 Repositories are owned by the contributor, can contain tools, workflows, etc. Backed by version control, a complete version history is retained for everything that passes through the toolshed Galaxy instance admins can install tools directly from the toolshed using only a web UI Support for recipes for installing the underlying software that tools depend on (also versioned)

23

24 Command line tools to support tool developers Command-line tools to aid development. Test tools quickly without worrying about configuration files. Check tools for common bugs and best practices. Optimized publishing to the ToolShed. Testbed for new dependency management - Homebrew and Homebrew-science

25 git[hub] centric development workflow

26 Tool citations, credit and incentivization Embed DOIs in Tool Configuration, Galaxy resolves and provides a list of citations, with links, which can exported for reference managers

27 Summary Galaxy is an (obsessively) open framework for making data analysis accessible and reproducible Nearly everything in Galaxy is pluggable, allowing it to customized for myriad purposes By supporting and leveraging developers the Galaxy community can collectively keep up with rapid changes in available tools

28 The Core Galaxy Team Engineering Enis Afgan Dannon Baker Dan Blankenberg Dave Bouvier Nate Coraor Martin Čech John Chilton Carl Eberhard Sam Guerler Nitesh Turaga Support and outreach Custodians Dave Clements Jennifer Jackson James Taylor Anton Nekrutenko Jeremy Goecks Supported by the NHGRI (HG005542, HG004909, HG005133, HG006620), NSF (DBI ), Penn State University, Johns Hopkins University, and the Pennsylvania Department of Public Health

29 Extended team and other contributors Björn Grüning Uni Freiburg Peter Cock TJHI Kyle Ellrott UCSC Eric Rasche CPT Nicola Soranzo TGAC Brad Chapman HSPH Nuwan Goonasekera VeRSI Yousef Kowsar VLSCI And many others who have contributed to the main Galaxy code, tools to the ToolShed, participated in discussions, attended the Galaxy conferences,