Heterogeneous compute in the GATK

Size: px

Start display at page:

Download "Heterogeneous compute in the GATK"

Moses Barnett
6 years ago
Views:

1 Heterogeneous compute in the GATK Mauricio Carneiro GSA Broad Ins<tute Intel Genomic Sequencing Pipeline Workshop Mount Sinai 12/10/2013

2 This is the work of many! Genome sequencing and analysis team Mark DePristo Eric Banks Stacey Gabriel David Altshuler

(e.g. posterior pipeline genotype likelihoods tool, diabetes En<rely new

and recalibra<on sta<s<cal gene<cists and to realignment make be]er use of

non- is successfully GATK analysis processed so^ware) from fastq => analysis

3 Scope and schema of the Best Prac<ces More involved support for A brand downstream new data analysis processing (e.g. posterior pipeline genotype likelihoods tool, diabetes En<rely new portal) alignment pipeline More New tools algorithms for data for analysts and recalibra<on sta<s<cal gene<cists and to realignment make be]er use of the genotyping likelihoods (o^en PCRFree ignored NA12878 in current 80xWGS non- is successfully GATK analysis processed so^ware) from fastq => analysis ready bam in A new under engine 20 minutes framework (plus for data intense alignment repe<<ve ~4-7h). analysis steps over Hadoop

4 We are here in the Best Practices workflow! CALLING VARIANTS

5 GATK s Haplotype Caller is replacing the seasoned Unified Genotyper Haplotype Caller is ready today for a small number of samples, but needs to scale be]er

6 Ar<fact SNPs and small indels caused by large indel is only recovered by local assembly NA12878 original read data Mul<ple caller ar<facts that are hard to filter out, since they are well supported by read data Haplotype Caller (validated) 6

7 Haplotype Caller is more accurate than the Unified Genotyper 1.00 Accuracy of unfiltered mutation calls with 60x coverage, 101bp PCR+ HiSeq NA12878 v2.5 v2.4 v2.6 v2.4 v2.3 v2.6 v2.5 v2.2 v2.1 v2.0 SNPS v2.3 v2.0 v2.1 v True positive rate 0.92 v2.6 v2.4 v2.5 v2.2 v2.0 v2.1 Caller a a HaplotypeCaller UnifiedGenotyper 0.88 INDELS 0.84 v2.3 v2.5 v2.4 v2.6 v2.2 v2.3 v2.0 v False positive rate

8 However the Haplotype Caller is more CPU intensive h R ]] r H 1. Ac#ve region traversal iden<fies the regions that need to be reassembled 3. Pair- Hmm evalua#on of all reads against all haplotypes (scales exponen<ally) 2. Local de- novo assembly builds the most likely haplotypes for evalua<on 4. Genotyping using the exact model

9 Pair- HMM is the biggest culprit for the low performance Stage Time Run#me % Assembly 2,598s 13% Pair- HMM 14,225s 70% Traversal + Genotyping 3,379s 17% NA xWGS chromosome 20 haplotype caller run Chr20 <me: 5.6 hours WGS <me: 7.6 days

10 How we can improve performance? 1. Distributed parallelism: Queue/MapReduce 2. Alterna<ve way to calculate likelihoods. 3. Heterogeneous parallel compute: v GPU, FPGA and Vectoriza<on. 4. Joint- calling with incremental single sample discovery.

11 In memory parallelism + map/reduce parallelism with queue PARALLELISM SUPPORT IN THE GATK TODAY

12 The GATK is actually two different beasts Engine Takes care of the input and output. Preprocess and organizes a traversal system for the walkers Walkers Sees the genome in an organized fashion and applies an algorithm to it

13 The three ways to parallelize the GATK Spawn many Parallelizing at the Parallelizing at the instances of the GATK engine level to walker level to speed to work on separate process different up the processing of (arbitrary) parts of the parts of the genome each individual region genome at the same at the same <me. of the genome. <me. - nt - nct Queue/MapReduce This is not really a solu<on, but a last resort that we use rou<nely to make calls today

14 How we can improve performance? 1. Distributed parallelism: Queue/MapReduce 2. Alterna<ve way to calculate likelihoods. 3. Heterogeneous parallel compute: v GPU, FPGA and Vectoriza<on. 4. Joint- calling with incremental single sample discovery.

15 Reducing the number of <mes we need to run the pair- HMM GRAPH BASED LIKELIHOODS

16 Calcula<ng genotype likelihoods straight from the assembly graph reduces pair- HMM usage Mapping each read to the haplotype assembly graph we can constrain the underlying pair- HMM to avoid quasi- zero likelihood unrealis<c alignments. ALT REF r 3 Lk = 0 ]] r 1 h 1 r 2 ]] ]] r 2 h 2 R r 3 h 3 r 1 ]] Lk = 0 ALT REF h 1 h 2 h 3 H The resul<ng sub- problems become modules that can be reused across haplotypes that share sub- paths in the graph.

17 Speed- up can be quite significant depending on data size and varia<on present. Varia#on Civar PairHMM (ms) GraphBased (ms) Speed- up 1 SNP *1T* x 1 short ins. *3I* x 1 long ins. *30I* x 1 short del. *3D* x 1 long del. *30D* x 1 SNP 1 ins. close by *1T*3=3I* x 1 SNP 1 ins. far away *1T*3I* x 5 close by SNPs *1T8=1T8=1T8=1T8=1T* x 5 far away SNPs *1T*1T*1T*1T*1T* x Invariants: kmersize = 10, readcount = 10000, readlength = 100, regionsize = 300! First implementa<on just 4x speed- up but can (and will) be improved

18 How we can improve performance? 1. Distributed parallelism: Queue/MapReduce 2. Alterna<ve way to calculate likelihoods. 3. Heterogeneous parallel compute: v GPU, FPGA and Vectoriza<on. 4. Joint- calling with incremental single sample discovery.

19 GPUs, FPGA and vectorized implementa<ons of the pair- HMM ALTERNATIVE PLATFORMS

20 Alterna<ve implementa<ons in the GATK FPGA (Convey Computer) Implemented by Sco] Thibault from Green Mountain Compu<ng Systems Fast I/O with bi- direc<onal data bus, somewhat large shared memory Highly parallelizable (hundreds of processing elements) GPU (NVidia CUDA) Implemented in collabora<on with Diego Nehab from IMPA- RJ Fast I/O but limited memory with thousands of cores in one board Especially fast float precision calcula<ons Vectorized (AVX) Implemented in collabora<on with Intel Corp In processor, no extra I/O, already available in most computers up to bit registers per CPU

21 A parallelized version of the Pair- HMM A A C A G C A G T C A G T C C Read Haplotype A A C A G T C C match/inser<on/dele<on

22 PairHMM Performance comparison Data: NA xWGS chromosome 20 TECH Hardware Run#me (seconds) Improvement (fold) AVX Intel Xeon 24- core x GPU NVidia Tesla K x GPU NVidia GeForce GTX Titan x GPU NVidia GeForce GTX x GPU NVidia GeForce GTX x GPU NVidia GeForce GTX x AVX Intel Xeon 1- core x FPGA Convey Computers HC x - C++ (baseline) 1,267 9x - Java (gatk) 10,800 -

23 GATK engine is not ready to leverage the massive parallelism Full Haplotype Caller run on 80xWGS PCR- Free NA12878: 13 days on single CPU (java) 3.5 days on Convey HC- 2 Human Chromosome x improvement instead of the expected 13x fold improvement Jobs per Day 1.5 GATK engine does not keep the pipe full for the alterna<ve implementa<ons of the pair- HMM to shine A different approach is necessary (GATK 3.x target release) C x86 HC- 2

Pair- Hmm evalua#on of all reads against all haplotypes (scales exponen<ally)

24 Synchronous traversal is to blame h R ]] r H 1. Ac#ve region traversal iden<fies the regions that need to be reassembled 3. Pair- Hmm evalua#on of all reads against all haplotypes (scales exponen<ally) 2. Local de- novo assembly builds the most likely haplotypes for evalua<on 4. Genotyping using the exact model

25 Summary The GSA team is con<nuously revising and upda<ng the best prac<ces for variant calling in order to enable the large scale of tomorrow s medical gene<cs needs. Assembly genotyping will work in combina<on with any other accelera<ons made to the pair- HMM. GPUs and AVX are the most promising plaworms for large scale projects or pipelines in need of very fast turnaround (e.g. diagnos<cs) The GATK will need an asynchronous engine to enable the full poten<al of these plaworms Available in the next major version release of the GATK

26 THE GATK TEAM NEEDS YOU Talk to me for more informa<on or

BWT Indexing: Big Data from Next Generation Sequencing and GPU

GPU Technology Conference 2014 BWT Indexing: Big Data from Next Generation Sequencing and GPU Jeanno Cheung HKU-BGI Bioinformatics Algorithms and Core Technology Research Laboratory University of Hong