NBIC Cloud. Mattias de Hollander David van Enckevort Leon Mei Rob Hooft

Transcription

1 NBIC Cloud Mattias de Hollander David van Enckevort Leon Mei Rob Hooft

2 SURFsara HPC Cloud 19 nodes, 32 cores and 256 GB RAM each Intel 2.13 GHz 32 cores (Xeon-E7 "Westmere-EX") 400 TB storage in total Nodes are connected by four 10 Gb interfaces to a nonblocking switch So maximum 40 GB access to storage (per node)

3 Cloud Management Console

4 NBIC Galaxy Server NBIC Tools Control panel History panel

5 Share Scripts & Pipelines GAPSS (Variant calling, annotations) DeepSAGE RNA-seq mscompare Chip-seq Circos visualization

6 Portal To Other Resource

7 Strong User Community Galaxy is widely used for analyzing Next Generation Sequencing data PennState University, BSD like license Very active user community (about 200 participants to Galaxy Community Conference in 2012, and 150 in 2011) Galaxy2.nbiceng.net Started in 2010 >240 registered users Used in a number of courses However, only a demo server

8 Migration to Cloud Galaxy2.nbiceng.net 4 CPU, 24G RAM, 1.5T HD, 100Mbit Internet Migration project started in July 2012 Supported by BiGGrid, SURFsara, NBIC, NIOO Production server launched in September 2012 Lightpath connecting to Cloud Will be used as the base for other project specific Galaxy servers in the HPC cloud TraIT NGS Galaxy will contain several CompleteGenomics specific tools NBIC Galaxy VM

9 Architecture Cloudman auto-scaling Dynamic Worker Node Internet Persistent Master Node 4core, 32G NFS mounts Tools, User data, Genomes, 50TB Backup

10 File Structure Master VM CloudMan scripts: /mnt/cm SGE: /opt/sge ProFTP: /opt/galaxy/pkg/proftpd Nginx: /opt/galaxy/pkg/nginx NFS shared /mnt/galaxytools: galaxy, 3rd party tools, 2.4GB /mnt/galaxydata: galaxy user data, postgres DB, 784GB /mnt/galaxyindices: genomes, liftover chains, 72GB

11 Tool Installation Automation CloudMan scripts Developed by the PennState Galaxy team Support Amazon EC2 and OpenStack CloudMan script customized for OpenNebula Developed by Mattias de Hollander (NIOO) Hosted at Fabric installation scripts Galaxy itself PostgreSQL Sun Grid Engine Common NGS tools, e.g. BWA, bowtie, samtools, etc.

12 Genomes Installation Automation Cloud version of BioLinux Developed by a team consists of members from Harvard Univ., J. Craig Venter Institute, the Galaxy team Using Cloud Computing Infrastructure with CloudBioLinux, CloudMan, and Galaxy, June 2012 Fabric installation script Common genome builds, hg18, hg19, mm9, tair10, etc. Tool specific genome indexes for bowtie, BWA, etc.

13 Some Hurdles Installation Fabric scripts use hardcoded versions, some of them are outdated PostgreSQL version conflict Too many layers so hard to tell where is the root cause VM or Cloud Running SGE failed to start

14 Cloud Issues Everything on the NFS share is owned by the same user. However: PostgreSQL requires the ownership of its data directory. CloudMan tries to chown directories. Work-around: disable chown / use idmapd Open Nebula issues VIRTIO network driver bug caused network connection unstable Worker node start up problem

15 DB Migration MySQL to PostgreSQL Many tools are difficult to set up or do not work Fortunately, we found py-mysql2pgsql

16 Remaining Issues Data transfer bottleneck (~40MB/s) between master/worker node and the NFS storage. Installation of additional tools /usr/local/bin => /mnt/galaxytools Enabling test framework NBIC Toolshed

17 Remaining Issues Data transfer bottleneck (~40MB/s) between master/worker node and the NFS storage. Installation of additional tools /usr/local/bin => /mnt/galaxytools Enabling test framework NBIC Toolshed

18 Acknowledgement BiGGrid/SURFsara/NBIC Niek Bosch, Marc van Driel, Irene Nooren, Machiel Jansen, the cloud-support team NBIC Galaxy admin team