Bionimbus: Lessons from a Petabyte-Scale Science Cloud Service Provider (CSP)

Transcription

1 Bionimbus: Lessons from a Petabyte-Scale Science Cloud Service Provider (CSP) Robert Grossman Institute for Genomics & Systems Biology Center for Research Informatics Computation Institute Department of Medicine University of Chicago & Open Data Group September 11, 2012

2 The OSDC & Bionimbus Teams Open Science Data Cloud (OSDC) Team Matt Greenway, Allison Heath, Ray Powell, Rafael Suarez. Major funding for the OSDC is provided by the Gordon and Betty Moore Foundation. Bionimbus Team Elizabeth Bartom, Casey Brown, Jason Grundstad, David Hanley, Nicolas Negre, Tom Stricker, Matt Slattery, Rebecca Spokony & Kevin White. Bionimbus is a joint project between Laboratory for Advanced Computing & White Lab at the University of Chicago and uses in part OSDC infrastructure.

3 Let s Step Back 20 Years : Petabyte Access & Storage Solutions (PASS) Project for SSC. It developed & benchmarked federated relational, OO DB, object stores, & columnoriented data warehouse solutions at the TB-scale.

4 A picture of Cern s Large Hadron Collider (LHC). The LHC took about a decade to construct, and cost about $4.75 billion. Source of picture: Conrad Melvin, Creative Commons BY-SA 2.0,

5 Part 1. Genomics as a Big Data Science

6 Source: Lincoln Stein

7 One Million Genomes Sequencing a million genomes would most likely fundamentally change the way we understand genomic variation. The genomic data for a patient is about 1 TB (including samples from both tumor and normal tissue). One million genomes is about 1000 PB or 1 EB With compression, it may be about 100 PB At $1000/genome, the sequencing would cost about $1B

8 Big data driven discovery on 1,000,000 genomes and 1 EB of data. Genomicdriven diagnosis Improved understanding of genomic science Genomicdriven drug development Precision diagnosis and treatment. Preventive health care.

9 ER+ TNBC With genomics, we can stratify diseases and treat each stratum differently. Source: White Lab, University of Chicago.

10 Clonal Evolution of Tumors Tumors evolve temporally and spatially. Source: Mel Greaves & Carlo C. Maley, Clonal evolution in cancer, Nature, Volume 241, pages , 2012.

11 Combinations of Rare Alleles Penetrance High Intermediate Modest Low Very rare alleles causing Mendelian disease rare variants of small effect very hard to identify by genetic means Rare Low-frequency variants with intermediate penetrance Uncommon rare examples of high-penetrance common variants influencing common disease most common variants implicated in common disease by GWA Common Allele frequency Source: Mark McCarthy

12 TCGA Analysis of Lung Cancer Source: The Cancer Genome Atlas Research Network, Comprehensive genomic characterization of squamous cell lung cancers, Nature, 2012, doi: /nature cases of SQCC (lung cancer) Matched tumor & normal Mean of 360 exonic mutations, 323 CNV, & 165 rearrangements per tumor

13 Some Examples of Big Data Science Discipline Duration Size # Devices HEP - LHC 10 years 15 PB/year* One Astronomy - LSST 10 years 12 PB/year** One Genomics - NGS 2-4 years 0.5 TB/genome 1000 s *At full capacity, the Large Hadron Collider (LHC), the world's largest particle accelerator, is expected to produce more than 15 million Gigabytes of data each year. This ambitious project connects and combines the IT power of more than 140 computer centres in 33 countries. Source: **As it carries out its 10-year survey, LSST will produce over 15 terabytes of raw astronomical data each night (30 terabytes processed), resulting in a database catalog of 22 petabytes and an image archive of 100 petabytes. Source:

14 One large instrument Many smaller instruments

15 Part 2. What Instrument Do we Use to Make Big Data Discoveries? How do we build a datascope?

16 TB? PB? EB? ZB? What is big data?

17 Another way: opencompute.org Think of data as big if you measure it in MW, as in Facebook s Pineville Data Center is 30 MW.

18 An algorithm and computing infrastructure is big-data scalable if adding a rack (or container) of data (and corresponding processors) allows you to do the same computation in the same time but over more data.

19 Commercial Cloud Service Provider (CSP) 15 MW Data Center Monitoring, network security and forensics Automatic provisioning and infrastructure management Accounting and billing 100,000 servers 1 PB DRAM 100 s of PB of disk Customer Facing Portal ~1 Tbps egress bandwidth 25 operators for 15 MW Commercial Cloud Data center network

20 What are some of the important differences between commercial and research-focused CSPs?

21 POV Data & Storage Flows Streams Science CSP Democratize access to data. Integrate data to make discoveries. Long term archive. Data intensive computing & HP storage Science Clouds Large data flows in and out Streaming processing required Commercial CSP As long as you pay the bill; as long as the business model holds. Internet style scale out and object-based storage Lots of small web flows NA Accounting Essential Essential Lock in Moving environment between CSPs essential Lock in is good

22 Part 3. The Open Cloud Consortium s Open Science Data Cloud

23 U.S based not-for-profit corporation. Manages cloud computing infrastructure to support scientific research: Open Science Data Cloud. Manages cloud computing testbeds: Open Cloud Testbed. 23

24 Cloud Services Operations Centers (CSOC) The OSDC operates Cloud Services Operations Center (or CSOC). It is a CSOC focused on supporting Science Clouds for researchers. Compare to Network Operations Center or NOC. Both are an important part of cyber infrastructure for big data science.

25 Different Styles of OSDC Racks 2012 OSDC rack design (draft) 950 TB / rack 600 cores / rack Design 1: Put cores over spindles. Higher cost but easy to compute over all the data. Design 2: separate (some of the )storage from the compute.

26 Open Science Data Cloud Monitoring, compliance, & security Automatic provisioning and infrastructure management Accounting and billing (OSDC) Science Cloud SW & Services 3 PB PB 2012 able to scale to 100 PB? 5-12 operators to operate 1-5 MW Science Cloud Data center network Customer Facing Portal (Tukey) ~100 Gbps bandwidth OSDC Data Stack based upon OpenStack, Hadoop, GlusterFS, UDT,

27 OSDC Philosophy We try to automate as much as possible (we automate the setup & operations of a rack). We try to write as little software as possible. Each project is a bit different, but in general: We assign (permanent) IDs to data managed by the OSDC and manage associated metadata. We assign and enforce permissions for users & groups of users and for files/objects, collections of files/objects, and collections of collections. We Support RESTful interfaces. Do accounting for storage and core-hours.

28 Some Of Our Biggest Mistakes Not charging for services. This resulted in a lot of bad behavior. Trying to support donated equipment without adequate staff. Being too optimistic about when big data software would be ready for prime time. Some problems with big data software doesn t show up at less than the full scale of the OSDC, but we have only one OSDC and it is difficult to test at this scale.

29 Essential Services for a Science CSP Support for data intensive computing Support for big data flows Account management, authentication and authorization services Health and status monitoring Billing and accounting Ability to rapidly provision infrastructure Security services, logging, event reporting Access to large amounts of public data High performance storage Simple data export and import services

30 Number 1000 s Individual scientists & small projects 100 s 10 s Small Public infrastructure Community based science via Science as a Service very large projects Data Size Medium to Large Very Large Shared community infrastructure Dedicated infrastructure

31 Part 4. Bionimbus Bionimbus is a joint project between Laboratory For Advanced Computing & the White Lab at the University of Chicago.

32 Step 1. Prepare a Sample

33 Step 2. Login to Bionimbus and get a Bionimbus Key.

34 Step 3. Send your sample to the sequencing center.

35 Step 4. Login on to Bionimbus and view your data

36 Step 5. Use Bionimbus to perform standard and custom pipelines. Bionimbus can launch multiple virtual machines.

37 Bionimbus Virtual Machine Releases Peak Calling Quality Control Alignment & Genotyping MAT MA2C PeakSeq MACS SPP Various Bowtie TopHat Samtools Picard 37

38 Software Tools: Moving Genomes

39 Bionimbus Community Genomic Cloud researcher 1K genomes PubMed etc. Cloud for Public Data Personal dropbox + compute

40 Bionimbus Private Genomic Cloud researcher 1K genomes PubMed etc. Cloud for Public Data Personal dropbox & compute Cloud for Controlled Data TCGA dbgap

41 Bionimbus Private Biomedical Cloud researcher 1K genomes PubMed etc. Cloud for Public Data Personal dropbox plus compute Cloud for Controlled Data TCGA dbgap Scatter, gather queries Clinical Research Data Warehouse Cloud for PHI data

42 External sequencing partner Step 3b. Return variant calls, CNV, annotation Step 2. Send sample to be sequenced. Step 4. Secure data routing to appropriate cloud based upon BID. Internal Sequencers Step 3a. Return raw reads. Bionimbus Private Cloud UC BID Generator Step 5. Cloud based analysis using IGSB and 3 rd party tools and applications. Step 1. Get Bionimbus ID (BID), assign project, private/community, public cloud, etc. Bionimbus Community Cloud Bionimbus Private Cloud XY dbgap Amazon

43 web2py-based Front End (Eucalyptus, OpenStack) (PostgreSQL) Utility Cloud Services (IDs, etc.) Database Services Data Ingestion Services Analysis Pipelines & Re-analysis Services Data Cloud Services Intercloud Services (UDT, replication) (Hadoop, Sector/Sphere)

44 >300 ChIP datasets -Chromatin/RNA timecourse -CBP -PolII -Pho/silencers -HDACs -Insulators -TFs Predictions 537 silencers 2,307 new promoters 12,285 enhancers 14,145 insulators Negre et al. Nature

45 Part 5. Managing One Million Genomes

46 Relational databases Summary level ( TB) Enrich with clinical data NoSql & scientific databases NoSql, DFS, file overlays? Variation (VCF) Files (1-10 PB) (Genomic variation) Sequence (BAM) Files ( PB) (Sequence data in binary form)

47 Acknowledgements Major funding and support for the Open Science Data Cloud is provided by the Gordon and Betty Moore Foundation, which has provided $2M of funding to the OSDC to launch Phase 1 of the project ( ). Moore Foundation funding is used to support the OSDC-Adler, Sullivan and Root facilities. Additional funding for the OSDC has been provided by the following sponsors: The OCC-Y Hadoop Cluster (approximately 1000 cores and 1 PB of storage) was donated by Yahoo! in Cisco provides the OSDC access to the Cisco C-Wave, which connects OSDC data centers with 10 Gbps wide area networks. NSF awarded the OSDC a 5-year ( ) $3.5M PIRE award to train scientists to use the OSDC and to further develop the underlying technology. OSDC technology for high performance data transport is support in part by NSF Award The StarLight Facility in Chicago enables the OSDC to connect to over 30 high performance research networks around the world at 10 Gbps or higher, with an increasing number of 100 Gbps connections. The OSDC is managed by the Open Cloud Consortium, a 501(c)(3) not-for-profit corporation. If you are interested in providing funding or donating equipment or services, please contact us at info@opensciencedatacloud.org.

48 For more information You can find some more information on my blog: rgrossman.com. Some of my technical papers are also available there. My address is robert.grossman at uchicago dot edu

49 Sources for images The image of the hard disk is from Norlando Pobre, Creative Commons. The image of the Facebook Pineville Data Center is from the Intel Free Press, Creative Commons BY 2.0. The image of the LHC is from Conrad Melvin, Creative Commons BY-SA 2.0,