Cloud computing for genome science and methods

Transcription

1 Cloud computing for genome science and methods Ben Langmead Department of Biostatistics

2 Sequencing throughput GA II 1.6 billion bp per day (2008) GA IIx 5 billion bp per day (2009) HiSeq billion bp per day (2010) Images: Numbers: Dates: Illumina press releases

3 Sequencing throughput End of 2009 Mid 2010 Late 2011/Early 2012 SOLiD 3+ System Up to 50 Gb Source: www3.appliedbiosystems.com/cms/groups/mcb_marketing/documents/generaldocuments/cms_ pdf

4 Sequencing throughput End of 2009 Mid 2010 Late 2011/Early 2012 SOLiD 3+ System Up to 50 Gb Source: www3.appliedbiosystems.com/cms/groups/mcb_marketing/documents/generaldocuments/cms_ pdf

5 Computational throughput Moore s Law: The number of transistors that can be placed inexpensively on an integrated circuit doubles approximately every two years.

6 Computational throughput Source: en.wikipedia.org/wiki/moore%27s_law

7 Computational throughput Core 2 Duo 386 Pentium Source: en.wikipedia.org/wiki/moore%27s_law

8 Throughput growth gap > 4-5x per year 2x per 2 years

9 Throughput growth gap = Idle

10 Throughput growth gap = Faster algorithms

11 Throughput growth gap =

12 Throughput growth gap =

13 Cloud computing

14 Cloud computing 1. Rent, don t buy = :: Cloud vendor :: Electric Company

15 Cloud computing 2. Large, centralized, efficient Columbia river for cheap hydroelectric power, cooling Source: nytimes.com

16 Cloud computing Why? Why not? Cost? Handles demand that grows, shrinks dramatically No hardware maintenance No alternative? Cost? Harder to program Less user-friendly Data movement is inconvenient & can outpace network Privacy (e.g. IRB)

17 Cloud computing Cost? Publications: Why? Handles demand that grows, shrinks dramatically No hardware (2010) maintenance No alternative? Why not? Stein LD. The case for cloud computing in genome informatics. Genome Biology. 2010;11(5):207. Epub 2010 May 5 Cost? Schatz MC, Langmead B, Salzberg SL. Cloud computing and the DNA data race. Nature Biotechnology Jul;28(7): Harder to program Less user-friendly Baker M. Next-generation sequencing: adjusting to data overload. Nature Methods 2010, 7: Data movement is Sansom C. Up in a cloud? Nature Biotechnology 28, Blogs: 1. PolITiGenomics 2. Informatics Iron, 3. business bytes genes molecules inconvenient & can outpace network Privacy (e.g. IRB)

18 Myrna

19 Myrna Sample A Gene 1 GGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT Sample B

20 ATATATATATATATAT Myrna Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC Gene 1 GGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT Sample B GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC

21 ATATATATATATATAT Myrna Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC TGTCGCAGTATCTGTC TATGTCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATATCGCA GAGCACCCTATGTCGC CCGGAGCACCCTATAT CCGGAGCACCCTATAT GCCGGAGCACCCTATG Gene 1 GGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT Sample B GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC TGTCGCAGTATCTGTC GCCGGAGCACCCTATG

22 ATATATATATATATAT Myrna Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC TGTCGCAGTATCTGTC TATGTCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATATCGCA GAGCACCCTATGTCGC Overlap CCGGAGCACCCTATAT CCGGAGCACCCTATAT GCCGGAGCACCCTATG Gene 1 GGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT Sample B GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC TGTCGCAGTATCTGTC GCCGGAGCACCCTATG

23 ATATATATATATATAT Myrna Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC TGTCGCAGTATCTGTC TATGTCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATATCGCA GAGCACCCTATGTCGC Overlap CCGGAGCACCCTATAT CCGGAGCACCCTATAT GCCGGAGCACCCTATG Gene 1 GGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT Sample B GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC TGTCGCAGTATCTGTC GCCGGAGCACCCTATG

24 ATATATATATATATAT Myrna Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC TGTCGCAGTATCTGTC TATGTCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATATCGCA GAGCACCCTATGTCGC Overlap CCGGAGCACCCTATAT CCGGAGCACCCTATAT GCCGGAGCACCCTATG Normalize Gene 1 GGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT Sample B GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC TGTCGCAGTATCTGTC GCCGGAGCACCCTATG Normalize

25 ATATATATATATATAT Myrna Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC TGTCGCAGTATCTGTC TATGTCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATATCGCA GAGCACCCTATGTCGC Overlap CCGGAGCACCCTATAT CCGGAGCACCCTATAT GCCGGAGCACCCTATG Normalize Gene 1 GGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT Sample B GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC TGTCGCAGTATCTGTC GCCGGAGCACCCTATG Normalize

26 ATATATATATATATAT Myrna Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC TGTCGCAGTATCTGTC TATGTCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATATCGCA GAGCACCCTATGTCGC Overlap CCGGAGCACCCTATAT CCGGAGCACCCTATAT GCCGGAGCACCCTATG Normalize Gene 1 differentially expressed?: YES p-value: Statistics Gene 1 GGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT Sample B GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC TGTCGCAGTATCTGTC GCCGGAGCACCCTATG Normalize

27 Myrna Overlap Normalize Statistics Parallel by read

28 Myrna Overlap Normalize Statistics Parallel by read Parallel by genome bin

29 Myrna Overlap Normalize Statistics Parallel by read Parallel by genome bin Parallel by sample

30 Myrna Overlap Normalize Statistics Parallel by read Parallel by genome bin Parallel by sample Parallel by gene

31 Myrna Myrna Runtime, Cost for 1.1 billion reads from Pickrell et al study EC2 Nodes 1 master, 1 master, 1 master, 10 workers 20 workers 40 workers Worker CPU cores Wall clock time 4h:20m 2h:32m 1h:38m Cluster setup 4m 4m 3m 2h:56m 1h:31m 54m Overlap 52m 31m 16m Normalize 6m 7m 6m Statistics 9m 6m 6m Summarize & Postprocess 13m 14m 13m Approximate cost (N. Virginia / Elsewhere) $44.00 / $49.50 $50.40 / $56.70 $65.60 / $73.80 Table 1. Timing and cost for a Myrna experiment with 1.1 billion 35 bp unpaired reads from the Pickrell et al study as input. Costs are approximate and based on the pricing as of this writing, that is, $0.68 per extra-large high-cpu EC2 node per hour in the Northern Virginia zone and $0.78 in other zones, plus a $0.12 per-node-per-hour surcharge for Elastic MapReduce in all zones. Times can vary subject to, for example, congestion and Internet traffic conditions. Data transfer & preprocessing adds 1h:15m and $12

32 Myrna Myrna Runtime, Cost for 1.1 billion reads from Pickrell et al study EC2 Nodes 1 master, 1 master, 1 master, 10 workers 20 workers 40 workers Worker CPU cores Wall clock time 4h:20m 2h:32m 1h:38m Cluster setup 4m 4m 3m 2h:56m 1h:31m 54m Overlap 52m 31m 16m Normalize 6m 7m 6m Statistics 9m 6m 6m Summarize & Postprocess 13m 14m 13m Approximate cost (N. Virginia / Elsewhere) $44.00 / $49.50 $50.40 / $56.70 $65.60 / $73.80 Table 1. Timing and cost for a Myrna experiment with 1.1 billion 35 bp unpaired reads from the Pickrell et al study as input. Costs are approximate and based on the pricing as of this writing, that is, $0.68 per extra-large high-cpu EC2 node per hour in the Northern Virginia zone and $0.78 in other zones, plus a $0.12 per-node-per-hour surcharge for Elastic MapReduce in all zones. Times can vary subject to, for example, congestion and Internet traffic conditions. Data transfer & preprocessing adds 1h:15m and $12

33 Myrna Myrna Runtime, Cost for 1.1 billion reads from Pickrell et al study EC2 Nodes 1 master, 1 master, 1 master, 10 workers 20 workers 40 workers Worker CPU cores Wall clock time 4h:20m 2h:32m 1h:38m Cluster setup 4m 4m 3m 2h:56m 1h:31m 54m Overlap 52m 31m 16m Normalize 6m 7m 6m Statistics 9m 6m 6m Summarize & Postprocess 13m 14m 13m Approximate cost (N. Virginia / Elsewhere) $44.00 / $49.50 $50.40 / $56.70 $65.60 / $73.80 Table 1. Timing and cost for a Myrna experiment with 1.1 billion 35 bp unpaired reads from the Pickrell et al study as input. Costs are approximate and based on the pricing as of this writing, that is, $0.68 per extra-large high-cpu EC2 node per hour in the Northern Virginia zone and $0.78 in other zones, plus a $0.12 per-node-per-hour surcharge for Elastic MapReduce in all zones. Times can vary subject to, for example, congestion and Internet traffic conditions. Data transfer & preprocessing adds 1h:15m and $12

34 Acknowledgements Jeffrey Leek Kasper Hansen Rafael Irizarry Hector Corrada Bravo Margaret Taub Michael Schatz Jimmy Lin Mihai Pop Steven Salzberg Deepak Singh Peter Sirota Myrna website: Paper: