HPC in Life Sciences. Bhanu Rekepalli Computational Scientist, JICS Adjunct Assistant Professor, EECS

Transcription

1 HPC in Life Sciences Bhanu Rekepalli Computational Scientist, JICS Adjunct Assistant Professor, EECS

2 Highlights of Talk Problem: Exponential Data Growth in Life Sciences Potential Solutions Scientific Program Development on HPC Arch Petascale Results Science Gateways Projects and Science Impacts

3 Sequencing Technologies 3

4 The $1,000 genome, the $100,000 analysis? Sboner, A., Mu, X., Greenbaum, D., Auerbach, R. & Gerstein, M. Genome biology 12, 125 (2011)

5 Solution Highly-scalable Parallel Informatics Tool Design BLAST, HMMER, MUSCLE, DOCK6, AutoDock, and LINUS Software Modules Tools available as standard software environment modules on HPC architectures Web-based Science Gateway Grant researchers simplified access to HPC resources. (eg. Kraken) Automated Workflows Allow for large-scale knowledge discovery Database Accessible storage for long-term data analysis

6 Kraken: 1 st Academic PetaFLOPS Computer Cray XT PetaFLOPS (Peak) 112,896 Compute cores Each node has 12 cores GHz AMD (Istanbul) Processor 16 GB RAM per node 147 TB of compute memory Scratch disk space, with 2.4PB of usable space

7 Nautilus SGI UltraViolet specs Compute processor type Intel ~2.0 GHz Nehalem Compute cores 1024 Compute sockets (nodes) 128 oct-core Memory per core 4 GB Total memory 4 TB (SMP) Accelerators 16 NVIDIA GPUs Peak system performance 10 TF Interconnect topology NUMAlink5 Parallel file system space 1 PB (GPFS or Lustre) Parallel file system peak performance 30 GB/s

8 WORLD RECORD! Beacon at NICS Intel Xeon + Intel Xeon Phi Cluster First to Deliver GigaFLOPS / Watt 71.4% efficiency #1 on current Green500 Nodes Beacon Cluster 4 service, 6 I/O, 48 compute CPU model Intel Xeon E CPUs per node RAM per node Intel Xeon Phi Coprocessors per node Cores per Intel Xeon Phi coprocessor RAM per Intel Xeon Phi coprocessor 2 8-core, 2.6GHz 256 GB 4 x 5110P 60 8 GB GDDR5 Other brands and names are the property of their respective owners.

9 Darter new Cray Cascade XC30 Latest Cray architecture Intel 2.6GHz x86 Sandy Bridge processors 748 nodes (2 SB each) 1496 sockets 11,968 cores 32 GB per node New Aries interconnect TF peak

10 Highly-Scalable Parallel Wrappers (HSP-Wrap) Reusable solution scales informatics tools for HPC environments. Critical performance optimizations: Efficient database distribution Dynamic hierarchical load balancing High-throughput buffered parallel I/O Robust fault-tolerance Check-pointing and recovery

11 The Wrapper Approach Input Queries Main Memory Query Block 1 Query Block M Database (NR, Pfam, ) Preload Database Master Node Main Memory Database Results 1 Compressed Buffer Output Buffer Database Query Block Results 2 Results N Lustre FS Compression Worker Nodes [1..N] Tool Process 1 (BLAST, DOCK6, HMMER, ) Tool Process P (BLAST, DOCK6, HMMER, )

12 BLAST Scaling Results

13 HMMER Scaling Results

14 NICS Informatics Science Gateway Collaborative effort with SDSC to adapt CIPRES for the NICS Informatics Science Gateway Add handling of data movement for large data sets Incorporate complex computation workflows using Airavata Provide database-supported output analysis tools Features: User registry with fine-grained access and usage control Logging of users and usage Job monitoring and statistics tracking Task-based tool parameter configuration Remote job submission via community account PoPLAR: poplar.nics.tennessee.edu XSEDE project: Science Gateway for Systems Biology Workbench Portal

15 Component Architecture Web Portal Gateway Airavata Parallel modules on HPC resource Subselected Data Raw Data Parser tools (parse, filter, sort) Yes Postprocess? No

16

17 17

18 Automated Workflows 18

19 McGill Arctic Research Station (MARS) is located in an Artic Desert N W Drill site, May 2011 Onstott, Princeton PI

20 Permafrost Regions are Important for Global Environmental Sustainability MARS MARS-McGill Arctic Research Station at Axel Heiberg Island, Nunavut, Canada. Soil that remains at or below 0 C for two or more years Generally located in high latitudes Occupies over 20% of the Earth s land surface Part of the cryosol: Active layer experiences seasonal thawing Permafrost layer does not experience seasonal thaw Active layer and permafrost accounts for 65% of the global soil and below ground organic carbon pool Low biological activity

21 Finding Thermostable Mutations in Cellulase Bhanu Rekepalli 1 *, Amit Upadhyay 1, Nicholas Panasik 2 1 National Institute for Computational Sciences, The University of Tennessee- Knoxville 2 Department of Biology, Claflin University, SC. Goal Compare conformations of thermostabilizing mutations to native conformations using LINUS (Local Independently Nucleated Units of Structure) Aims 1. Get all possible phi/psi for tetrameric polyalanine - no steric clash & no hydrogen bonds 2. Populate conformations from step 1 with side chains and check for steric clash and hydrogen bonding Future Use LINUS with HSP-Wrap for Step 2 Parallelize LINUS C code HSP-Wrap Challenges Amount of computations: Step 1: 3 x 10 8 Step 2: 2 x LINUS written in python Slow Not easy to parallelize Success Port LINUS source code from python to C Step 1 - Total computing time for on Newton cluster (UT) Python code: ~400 hours C code: 33 hours References Srinivasan R, Rose GD., Proteins. 2002;47(4):

22 Highly Scalable Parallel HMMER HMMER Protein Domain Identification tool HMMER compares sequences to a database of hidden Markov models to identify known domains within the sequences New HSP-HMMER code - Excellent performance Currently ~10000x faster than MPI-HMMER for 1K processes Scales up to 98,000 cores very well HSP-HMMER reduces time to identify the Pfam functional domains in 10 millions proteins of the nr (non redundant) database from days on clusters down to less than 10 minutes! using processing cores.. B Rekepalli, C Halloy, IB Jouline. HSP-HMMER: a Tool for Protein Domain Identification on a Large Scale, ACM SAC 2009, A part of an alignment for the Globin family from Pfam 22

23 HPC for Protein Domain Modeling Bhanu Rekepalli Igor Jouline Exploring dark matter of the protein sequence universe Bhanu Rekepalli, Greg Peterson and Igor Jouline. Dynamics of domain coverage of the protein sequence universe, BMC Genomics 2012

24 Future Medicine

25 Ligands / Second / CPU Highly Scalable Parallel Docking to Speedup Drug Discovery Bhanu Rekepalli, University of Tennessee, and Yuri Peterson, Medical University of South Carolina The Challenge Improve speed and scaling of molecular docking tools used to search compounds and reduce the computational time needed for novel drug discovery Libraries such as ZINC and PubChem contain millions of vendor and academic compounds and are growing rapidly The Success Algorithmic and I/O improvements of the MPI version of Dock6 Implications for Future Research The speedups achieved reduce the computational time needed for novel drug discovery from years to months or days, significantly reducing the time to market of new drug Figure 1 Dock6: flexible docking of compounds on Onco protein 0.07 kraken cbrc Speedups achieved on Kraken vs CBRC With Algorithmic, I/O improvements and code optimizations we achieved a 20X speedup per core on NICS Kraken supercomputer compared to MUSC Computational Biology Resource Center (CBRC) cluster.

26 William Mondy and Bhanu Rekepalli

27 Workshops and Training 27

28 Thank you Graduate Students Amit Upadhyay (GST) Yuan Liu (GIS) Lang Lee (GST) Eduardo Ponce (EECS) REU Students Nyalia Lui Jordan Taylor Julian Pierre