Accelerating Genome Assembly with Power8

Transcription

1 Accelerating enome Assembly with Power8 Seung-Jong Park, Ph.D. School of EECS, CCT, Louisiana State University Revolutionizing the Datacenter Join the Conversation #OpenPOWERSummit

2 Agenda The enome Assembly Problem Accelerating raph Construction with POWER8 Accelerating raph Simplification with CAPI Flash 2

3 The enome Assembly Problem 3

4 Challenges for enome Assemblers NS Technologies Outpaced Moore s Law Software with Extreme Scalability HPC Platform More Compute Cycles Extreme I/O Performance Huge Storage Space NS enome Reads (TBs) HPC 4 Data and Compute Intensive Reconstructed enome (MBs/Bs)

5 MapReduce-based raph Construction 5 TATCA CT CTTTAAT C TACTTTA Map TTTAAACA ATCCATA TATCA CT Map TTTA: TAT:C TTA:A TAA: TCC: A TA: N TCA: AA: A AAC:A ACA: N ATCC: CCA: T CAT: ATA: ATC: CA: AC: T ATC: C AA: C ACA: AT:A TC: A A: C CT: N CT: T TC: A A: C CT: N CTT:T ATC: N A: C CT: T CTT:T ATC: CA: AC: T CTTT:A AAT:C AC: T CTTT:A TAT:C TA: TCA: TTTA: TTA:A TAA:T TTTA: TTA:N Reduce Reduce Reduce TAA:,T TAT:C TCC:A TCA: TA: TTA:A TTTA: ACA:N AAC:A AA:A AAT:C AC:T ATC: ATCC: ATA: CCA:T CA: CAT: CTTT:A ACA: AA:C A:C ATC:C AT:A CTT:T CT:T TC:A

6 Accelerating raph Construction with POWER8 6

7 Experimental Test Beds System Type IBM PKY Cluster LSU SuperMikeII Processor Two 10-core IBM Power8 Two 8-core Intel SandyBridge Xeon Maximum #Nodes used in various experiments #Physical cores/node 20 (8 Simultaneous Multi-Thread) 16 (Hyper threading disabled) #vcores/node RAM/node (B) #Disks/node 5 3 #Disks/node used for shuffled data 3 1 Total Storage space/node used for shuffled data Network 56bps InfiniBand (non-blocking) 40bps InfiniBand (2:1 blockings) 7

8 Datasets enome data set Input size Shuffle data size Rice genome 12B 70B 50B Bumble bee genome 90B 600B 95B Output size Metagenome 3.2TB 20TB 8.6TB 8

9 Hadoop Configurations Hadoop Parameters IBM Power8 SuperMikeII Yarn.nodemanager.cpu.resource.vcore Yarn.nodemanager.memory.mb Mapreduce.map/reduce.cpu.vcore 4 2 Mapreduce.map/reduce.memory.mb Mapreduce.map/reduce.java.opts 6500m 3000m 9

10 Hadoop Scalability with POWER8 SMTs Tested with small size rice genome data on 2 node Almost linear scalability with increasing SMTs 10

11 Rice enome Analyzing small size (12B) data Eliminate the impact of network and disk I/O 7.5X performance improvement per server 11

12 Bumble Bee enome Analyzing Medium size (90B) Bumble Bee genome 7.5x improvement in terms of Performance/server 12

13 Metagenome Analyzing huge (3.2TB) metagenome data Only 6.5 hours on 40-node IBM Power8 cluster More than 9x improvement in terms of performance per server 13

14 raph Simplification with Distributed NoSQL TAA:,T TAT:C ACA:N AAC:A ACA: AA:C CCA:T CA: TCC:A TCA: TA: AA:A AAT:C AC:T A:C ATC:C AT:A CAT: CTTT:A TTA:A ATC: CTT:T TTTA: ATCC: CT:T ATA: TC:A TATCA ACTTTAA 14

15 Accelerating Simplification with IBM CAPI Flash NoSQL I/O Throughput (keys/sec) CAPI Flash I/O Throughput (bytes/sec) Only 20 Power8 Cores + CAPI : 500B raph traversal in 7.5 Hrs 15

16 Computational Challenges The Next Step raph building is the most expensive phase in terms of time and resources The Obvious Solutions: Either use a single machine with LOTS of memory, or run on a cluster. Idea: Use CAPI accelerated flash instead of main memory 16

17 raph Construction on IBM CAPI Flash 17 TATCACT CTTTAATC TACTTTA Map TTTAAACA ATCCATA TATCACT ATC:C AA:C ACA: AT:A TC:A A:C CT:N CT:T TC:A A:C CT:N CTT:T ATC:N A:C CT:T CTT:T AA:A AAC:A ACA:N ATCC: CCA:T CAT: ATA: ATC: CA: AC:T ATC: CA: AC:T CTTT:A AAT:C AC:T CTTT:A TTTA: TAT:C TTA:A TAA: TCC:A TA:N TCA: TAT:C TA: TCA: TTTA: TTA:A TAA:T TTTA: TTA:N Sort ATC:C AA:C ACA: AT:A TC:A A:C CT:N CT:T TC:A A:C CT:N CTT:T ATC:N A:C CT:T CTT:T ACA: AA:C A:C ATC:C AT:A CTT:T CT:T TC:A AA:A AAC:A ACA:N ATCC: CCA:T CAT: ATA: ATC: CA: AC:T ATC: CA: AC:T CTTT:A AAT:C AC:T CTTT:A TTTA: TAT:C TTA:A TAA: TCC:A TA:N TCA: TAT:C TA: TCA: TTTA: TTA:A TAA:T TTTA: TTA:N Sort Sort ACA:N AAC:A AA:A AAT:C AC:T ATC: ATCC: ATA: CCA:T CA: CAT: CTTT:A TAA:,T TAT:C TCC:A TCA: TA: TTA:A TTTA: NoSQL data engine APIs

18 Initial Results of raph Construction Compared 85B bumblebee dataset on 8-node Hadoop cluster vs. a single node with CAPI-accelerated flash. Hadoop Cluster (20 physical cores per node) Peak memory usage of 60B per datanode 1 HDD per datanode 1 hr 56 mins CAPI Accelerated Flash server (20 physical cores) Peak memory usage of 7 B 1 HDD and 1 CAPI card 3 hrs 44 mins Peak memory usage reduced by 60 times. Execution time reduced by 3.5 times per node. 18