elprep: a high- performance tool for preparing SAM/BAM files for variant calling Charlo<e Herzeel (Imec) Pascal Costanza (Intel) July 2014

Transcription

1 elprep: a high- performance tool for preparing SAM/BAM files for variant calling Charlo<e Herzeel (Imec) Pascal Costanza (Intel) July 2014

2 Overview Sequencing pipelines in praclce From raw reads to mapped reads to analysis- ready reads Many different tools in use CommunicaLon via files using standardized formats (SAM/BAM) Repeated file I/O for each step, li<le mull- threading in exislng tools elprep: high- performance preparalon of SAM/BAM MulL- threaded, executes enlrely in memory Expresses pipeline as a set of filters on a stream of read data Avoids file I/O, merges computalons of several pipeline steps Open- source, modular, extensible implementalon Benchmarks 2

3 Sequencing pipelines in praclce Goal: from raw reads to analysis Many compelng pipelines and tools in use CommunicaLon via standardized file formats Raw Reads Mapped Reads Analysis Ready Reads Analyzed Reads A"G"G"C C"T"A G"T"T _A"A"T..." "A"G"C"C"T"A"A"T"T"G"A"A"T... 3

4 Sequencing pipelines in praclce Example: GATK best- praclce pipeline (Broad InsLtute): Step Tool File I/O 1. Alignment via bwa in:.fastq, out:.sai 2. SAI to SAM via bwa samse in:.sai, out:.sam 3. SAM to BAM via samtools in:.sam, out:.bam 4. Removing unmapped reads via samtools in:.bam, out:.bam 5. Marking duplicate reads via picard in:.bam, out:.bam 6. Adding RG informalon via picard in:.bam, out:.bam 7. SorLng for conlgs via picard in:.bam, out:.bam 8. SorLng for coordinate order via samtools in:.bam, out:.bam 9. Indel realignment via gatk in:.bam, out:.bam 10. Base recalibralon via gatk in:.bam, out:.bam 11. RR compression via gatk in:.bam, out:.bam 12. Variant calling via gatk in:.bam, out:.vcf 4

5 Sequencing pipelines in praclce Example: GATK best- praclce pipeline (Broad InsLtute): Alternate uses of C and Java programs File I/O, including compression/decompression, at each step Repeated iteralons over the same data set Be<er performing solulon: Keep data in memory throughout enlre pipeline Merge loops where possible by implemenlng pipeline steps as filters that can be composed Make use of big RAM servers We implemented this approach for the preparalon phase as a new tool called elprep 5

6 Pipeline steps as filters Most steps in the pipeline operate on BAM files: SAM/BAM is a standard format for represenlng mapped reads. The pipeline steps can be seen as filters that update or add fields to the BAM representalon of a read, or remove reads all together 6

7 Pipeline steps as filters Removing unmapped reads Check if FLAG has bit 0x4 set (see 2 nd entry) ERR * 0 0 * * 0 0 NCATTCCATTTCATTCCACTAGGGTTCATTCCATTCCGTTCCATTCCATTCCACTCNTGTTGAT TCCATTCCGTNCCTTTCCTTTTCATTCCATTCAATT #1BDFFFFHHGHHJJJJJJJJJJJFGHJIJJJJJJJJJIEIJJJJJJIJJJJJJJI#0? FHIIJJJJJJJJDHI#- 5@EHIEHGHHEDFFEDDDDE@CEE ERR chr M * 0 0 GCTTCTCCTGAGATCATCGTTCCTGNTCCTGGAACACTTTTCCNCCCTAAATTTTACTTTTTAA ATCTTTCTTATTGTTTTTGTTTGCCTTCTGTTGCTN DDDDDDDDDDEDEDDDDDDDDCA;,#FFHHFGHHEHHGIGCB- #JJJJJJJJJIGFJJIJJJJJJIJJJJJJJJJJJJJJJJJIJJJHHHHHFFFDD=1# XT:A:U NM:i:3 X0:i:1 X1:i:0 XM:i:3 XO:i:0 XG:i:0 MD:Z:25C17A55T0 7

8 Pipeline steps as filters Removing unmapped reads Check if FLAG has bit 0x4 set (see 2 nd entry) ERR * 0 0 * * 0 0 NCATTCCATTTCATTCCACTAGGGTTCATTCCATTCCGTTCCATTCCATTCCACTCNTGTTGAT TCCATTCCGTNCCTTTCCTTTTCATTCCATTCAATT #1BDFFFFHHGHHJJJJJJJJJJJFGHJIJJJJJJJJJIEIJJJJJJIJJJJJJJI#0? FHIIJJJJJJJJDHI#- 5@EHIEHGHHEDFFEDDDDE@CEE ERR chr M * 0 0 GCTTCTCCTGAGATCATCGTTCCTGNTCCTGGAACACTTTTCCNCCCTAAATTTTACTTTTTAA ATCTTTCTTATTGTTTTTGTTTGCCTTCTGTTGCTN DDDDDDDDDDEDEDDDDDDDDCA;,#FFHHFGHHEHHGIGCB- #JJJJJJJJJIGFJJIJJJJJJIJJJJJJJJJJJJJJJJJIJJJHHHHHFFFDD=1# XT:A:U NM:i:3 X0:i:1 X1:i:0 XM:i:3 XO:i:0 XG:i:0 MD:Z:25C17A55T0 8

9 Pipeline steps as filters Adding read group informalon Add oplonal tag to each read entry + add header tag ERR chr M * 0 0 GCTTCTCCTGAGATCATCGTTCCTGNTCCTGGAACACTTTTCCNCCCTAAATTTTACTTTTTAA ATCTTTCTTATTGTTTTTGTTTGCCTTCTGTTGCTN DDDDDDDDDDEDEDDDDDDDDCA;,#FFHHFGHHEHHGIGCB- #JJJJJJJJJIGFJJIJJJJJJIJJJJJJJJJJJJJJJJJIJJJHHHHHFFFDD=1# XT:A:U NM:i:3 X0:i:1 X1:i:0 XM:i:3 XO:i:0 XG:i:0 MD:Z:25C17A55T0 9

10 Pipeline steps as filters Adding read group informalon Add oplonal tag to each read entry + add header ID:group1 PL:illumina PU:unit1 LB:lib1 SM:sample1 ERR chr M * 0 0 GCTTCTCCTGAGATCATCGTTCCTGNTCCTGGAACACTTTTCCNCCCTAAATTTTACTTTTTAA ATCTTTCTTATTGTTTTTGTTTGCCTTCTGTTGCTN DDDDDDDDDDEDEDDDDDDDDCA;,#FFHHFGHHEHHGIGCB- #JJJJJJJJJIGFJJIJJJJJJIJJJJJJJJJJJJJJJJJIJJJHHHHHFFFDD=1# XT:A:U NM:i:3 X0:i:1 X1:i:0 XM:i:3 XO:i:0 XG:i:0 MD:Z:25C17A55T0 RG:Z:group1 10

11 Pipeline steps as filters: Marking duplicates PCR duplicates: When mullple reads of the same DNA molecule occur Inherent to the wet lab process of sequencing Hard to idenlfy because PCR duplicates do not necessarily produce the same sequence reads Marking duplicates in Picard: We derive the algorithmic structure from the source code (1117 loc Java) 11

12 Phase 1: Marking duplicates (Picard) sort reads according to mapping coordinates while keeping track of the original order spilled to disk when running out of RAM (manually) pos = 0! sorted-list = []! loop for line in input-file! read = new read(parse(line), pos)! sorted-list.insert-sorted(read, #coordinate-order)! pos++! 12

13 Marking duplicates (Picard) Phase 2: idenlfy groups of potenlal duplicates within the sorted list for those groups of reads, track the file indexes for all but the read with the highest Phred score pos-reads-to-mark = []! duplicates-list = [sorted-list.pop()]! loop for read in sorted-list! dup-read = duplicates-list.peek()! if read.coordinate() == dup-read.coordinate()! duplicates-list.push(read)! else! best-read = duplicates-list.peek()! loop for dup-read in duplicates-list! if best-read.score() < dup-read.score()! best-read = dup-read! loop for dup-read in duplicates-list! if dup-read!= best-read! pos-reads-to-mark.append(dup-read.pos())! duplicates-list = [read]! 13

14 Phase 3: Marking duplicates (Picard) write a new file by copying the reads from the original file, using the file posilon to idenlfy the reads to tag as duplicate pos = 0! read-to-mark = pos-reads-to-mark.pop()! loop for line in input-file! read = new read (parse(line))! if pos == read-to-mark! write read.mark() to output-file! read-to-mark = pos-reads-to-mark.pop()! else! write read to outputfile! pos++! 14

15 Picard algorithm: Marking duplicates (Picard) MulLple loops over the same data The.BAM file is opened and parsed into objects twice: Once for idenlfying the posilons of the duplicate reads in the file Secondly for updalng those reads Intermediate files for spilling sorted reads to disk manually We can improve on this 15

16 Marking duplicates (Picard) Conceptually, this is what marking duplicates does: Two reads are considered duplicates if they map to the same coordinate in the reference Mark the one with the lesser Phred score as duplicate 16

17 Marking duplicates as a filter Algorithmic redesign in elprep: Keep a table of reads checked for duplicate marking so far, hashed on mapping coordinate Only one pass through the reads necessary cache = [] loop for read in reads cached- read = cache.hash(read, #coordinate) if read.score() > cached- read.score() cached- read.mark() cache.remove(cached- read) cache.add(read) else read.mark() 17

18 Composing the filters Define pipeline as a higher- order funclon: One loop to go through read data Pass filters to apply as funclons filters = [#'remove-unmapped, #'mark-duplicates,...]! loop for read in reads! loop for func in filters! apply func read! 18

19 Composing the filters Define pipeline as a higher- order funclon: One loop to go through read data Pass filters to apply as funclons PersisLng to file filters = [#'remove-unmapped, #'mark-duplicates,...]! loop for read in reads! loop for func in filters! if not(apply func read)! return! write read to file! 19

20 Domain- specific filter architecture A filter receives a SAM header It returns a thread- local filter. A thread- local filter returns an alignment filter. An alignment filter receives a SAM alignment. It can modify the alignment. It returns a boolean whether to keep the alignment or not. Alignment filters are executed in parallel 20

21 Pipeline execulon without elprep - - Only some exislng tools are parallelized Most steps are executed sequenlally, with repeated file I/O 21

22 HypotheLcal execulon with parallelized tools - Even if all tools were parallelized, we would slll face sequenlal bo<lenecks and repeated file I/O 22

23 Pipeline execulon with elprep elprep composes all steps and executes them one by one in parallel Parallel threads do not have to meet in between steps, and repeated file I/O is completely avoided (see also: Amdahl s Law, h<ps://en.wikipedia.org/wiki/amdahl%27s_law) 23

24 Command line tool elprep, an open source tool Filtering unmapped reads (strict / non- strict) Replacing reference sequences Replacing read groups Marking / removing duplicates Cleaning SAM SorLng by coordinate order / queryname DocumentaLon for command line and API h<ps://github.com/exascience/elprep 24

25 Benchmarks elprep vs. SAMtools/Picard: single- threaded: 4x faster tested with whole- genome data NA12878 mull- threaded: 25x faster at 40 threads (4x10- core Intel Xeon E7-4870, 512GB RAM) 10.5x faster at 32 threads (2x8- core Intel Xeon E with hyper- threading, 256GB RAM) validated 100% match with SAMtools/Picard output checked with samtools and picard- tools

26 Benchmarks elprep reduces preparalon from 20 hours to 2 hours On a 2x8- core Intel Xeon E with 256GB RAM elprep merges execulon of the different preparalon steps removing unmapped reads reordering contigs sorting marking duplicates adding read groups combined NA12878 SAMtools + Picard elprep samtools/picard Removing unmapped reads 38m12s - Sort conlgs 4h55m55s - SorLng 1h46m45s - elprep Mark duplicates 6h59m44s - 0 2h 4h 6h 8h 10h 12h 14h 16h 18h 20h Read groups 4h58m53s - Total 19h19m29s 1h54m51s 26

27 Advantages of elprep Efficient mull- threaded execulon Operates enlrely in memory, no intermediate files An order of magnitude faster than standard tools 100% equivalent output to SAMtools and Picard CompaLble with exislng tools CompaLble with compressed SAM (BAM/CRAM) Modular soyware architecture, easy to extend Open source 27

28 NoLces and Disclaimers Copyright 2014 Imec and Intel CorporaLon. Intel, the Intel logo, and Xeon are trademarks of Intel CorporaLon in the U.S. and/or other countries. Other names and brands may be claimed as the property of others. Soyware and workloads used in performance tests may have been oplmized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, soyware, operalons and funclons. Any change to any of those factors may cause the results to vary. You should consult other informalon and performance tests to assist you in fully evalualng your contemplated purchases, including the performance of that product when combined with other products. For more informalon go to h<p:// 28