Building the Genomics Virtual Lab

Transcription

1 Building the Genomics Virtual Lab Ron Horst, Uni QLD Analysis and visualisation platform Community Resources Australian Research Cloud

2 Agenda Objectives Scalable, on demand Latest tools, reproducible Build Architecture Challenges Build Challenges

3 Project Objectives PSU CloudMan Amazon Web Services CloudMan Australian Research Cloud Leverage Australian National Infrastructure On demand Scalable

4 LAUNCH build process image launch

5 Galaxy & CloudBiolinux projects Bioinformatics Tools LAUNCH build process image Galaxy launch Galaxy TUTORIAL Galaxy SERVICE Roll your Roll own your Roll own your own Research Cloud On demand Scalable Biolinux cluster

6 More Objectives Reproducible science Latest tools Maintainable

7 Architecture base image (OS, sys pkgs, R) snap (Galaxy + DB + tools) Indices FS snap (Reference genomes)

8 Architecture base image (OS, sys pkgs, R) snap (Galaxy + DB + tools) Indices FS snap (Reference genomes) Add default tools Add default indices Galaxy Main ToolShed Ref genomes (Penn State, S3) 1 per cloud LEGEND Continuous development Rare updates (~1/yr) Infrequent updates (~1/qtr) Regular updates (as needed) CBL + flavor build scripts External services Manual process Galaxy Reproducible Science Toolshed Latest Tools Architecture Maintainable

9 process v1 base image machine snap Indices FS snap Indices FS Can t share snapshots across zones Can t share s snapshots across Projects Can t resize snapshots Large footprint of every instance machine 51GB Indices FS

10 process v2 Gluster base image Objectstore in objectstore 52GB objectstore Indices FS 200GB gluster

11 process v2 machine Gluster Objectstore base image in objectstore 52GB machine Indices FS 200GB objectstore Any zone Any size Even using default allocation Longer to launch WAN access to Indices machine ephemeral gluster ephemeral

12 Systems Admin => What is Base? CBL Galaxy NeCTAR Users => First build Add tools Toolshed What tools? gvl BUILD BASE Add tools branding HCSVL Add tools Tut EndoVL Cloud imaging

13 Cloudbiolinux CloudMan code NeCTAR OS image Base Image Build process Set up build scripts Run build scripts Tailor target Publish image to Cloud base image (OS, sys pkgs, R) Build Machine target instance

14 Cloudbiolinux CloudMan code NeCTAR OS image Base Image Build process Set up build scripts Run build scripts Tailor target Publish image to Cloud base image (OS, sys pkgs, R) Build Machine target instance Create & Mnt Add Galaxy and tweaks Add Tools Publish to Cloud snap (Galaxy + DB + tools) 52GB Toolshed snap Galaxy

15 Cloudbiolinux CloudMan code NeCTAR OS image Base Image Build process Set up build scripts Run build scripts Tailor target Publish image to Cloud base image (OS, sys pkgs, R) Build Machine target instance Create & Mnt Add Galaxy and tweaks Add Tools Publish to Cloud snap (Galaxy + DB + tools) 52GB Toolshed snap Galaxy Indices Indices snap 250GB

16 Cloudbiolinux CloudMan code NeCTAR OS image Base Image Build process Set up build scripts Run build scripts Tailor target Publish image to Cloud base image (OS, sys pkgs, R) Build Machine target instance Create & Mnt Add Galaxy and tweaks Add Tools Publish to Cloud snap (Galaxy + DB + tools) 52GB Toolshed Publish to objectstore snap snap Galaxy Indices Indices snap 250GB Indices Gluster

17 Galaxy & CloudBiolinux projects Bioinformatics Tools LAUNCH build process image Galaxy launch Galaxy TUTORIAL Galaxy SERVICE Roll your Roll own your Roll own your own Research Cloud On demand Scalable Biolinux cluster Reproducible Science Latest Tools Maintainable

18 Thank you

19 The end

20 process Base Image base image (OS, sys pkgs, R) machine snap (Galaxy + DB + tools) Indices FS snap (Reference genomes) Indices FS Copy to other nodes snap (Galaxy + DB + tools) Copy to Objectstore Indices FS snap (Reference genomes) Indices FS in objectstore machine machine Copy to GLUSTER Indices FS gluster

21 process - NeCTAR ope sta k optio Any NeCTAR project, any resizing of GalaxyFS Any placement zone with Volumes machine v2.09 base image (OS, sys pkgs, R) FSv2.04 in objectstore 51GB machine Indices FS 200GB gluster machine

22 Base Image base image (OS, sys pkgs, R) process - NeCTAR el our e dev optio Must be GenomicsVL project, cannot resize GalaxyFS Must be qh2 (I think) machine FSv2.04 snap (Galaxy + DB + tools) 51GB Indices FS snap (Reference genomes) 200GB 51GB Indices FS 200GB cloudman-dev option machine FSv2.07 snap (Galaxy + DB + tools) 51GB 51GB Indices FS 200GB

23 Galaxy + DB galaxy Manual tweaks (colors, tools, menus Indices FS Add if desired New tool dev galaxy-tut Any number (1 per live instance) Service layer CloudMan ToolShedify process Creation process Component layer base image (OS, sys pkgs, R) snap (Galaxy + DB + tools) Indices FS snap (Reference genomes) Add default tools Add default indices Galaxy Main ToolShed Ref genomes (Penn State, S3) 1 per cloud LEGEND Continuous development Rare updates (~1/yr) Infrequent updates (~1/qtr) Regular updates (as needed) Build layer CBL + flavor build scripts Include dependencies and/or in default config 1 External services Manual process

24

25 Cloudbiolinux CloudMan code NeCTAR OS image Base Image Build process Set up build scripts Run build scripts Tailor target Publish image to Cloud base image (OS, sys pkgs, R) Do this Rarely Build Machine Cloudbiolinux CloudMan code target instance Add Add Galaxy and tweaks Add Tools Create snapshot Create objectstore copy snap (Galaxy + DB + tools) 51GB (Galaxy + DB + tools) 51GB Do this for: - Galaxy release - New tools base instance Galaxy Indices Toolshed List of Tools mnt file systems, tweaks Galaxy main List of Indices base instance Galaxy Indices Create snapshot Create gluster file Indices FS snap (Reference genomes) 200GB Final manual tweaks Indices FS (Reference genomes) 200GB Do this for: - Genome index update Galaxy-TUT base instance Galaxy 2TB Indices 200GB

26 CBL Galaxy NeCTAR What is Base? First build Add tools Custom for Monash gvl Add tools Sim on branding Add tools Qld Tut Custom for QLD