GATB programming day

Transcription

1 GATB programming day G.Rizk, R.Chikhi Genscale, Rennes 15/06/2016 G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

2 GATB INTRODUCTION NGS technologies produce terabytes of data Efficient and fast algorithms are essential to analyze such data >read 1 ACGACGACGTAGACGACTAGC AAAACTACGATCGACTAT >read 2 ACTACTACGATCGATGGTCGC GCTGCTCGCTCTCTCGCT... >read TCTCCTAGCGCGGCGTATACG CTCGCTAGCTACGTAGCT... The Genome Assembly Tool Box (GATB) 1. Open-source software developed by GENSCALE 2. Easy way to develop efficient and fast NGS tools 3. Based on data structure with a very low memory footprint 4. Complex genomes can be processed on desktop computers G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

3 GATB INTRODUCTION The GATB philosophy: a 3-layer construction to analyze NGS datasets 1. GATB-CORE: a C++ library holding all the services needed for developing software dedicated to NGS data 2. GATB-TOOLS: a set of elementary NGS tools mainly built upon the GATB library (k-mer counter, contiger, scaffolder, variant detection, etc.) GATB-PIPELINE GATB-TOOLS GATB-CORE THIRD PARTIES 3. GATB-PIPELINE: a set of NGS pipeline that links together tools from the previous layer G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

4 GATB INTRODUCTION GATB-CORE structure for NGS data Compact de Bruijn graph AGC CGC GCT GAG Encodes most of the information from the sequencing reads CCG TCC CTA CGA AAA GGA TGG TAT Graph with low memory footprint ATC ATT TTG Complex genomes can be processed on a desktop computer A whole human genome handled with 6 GBytes The raspberry genome can be assembled on a Raspberry Pi G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

5 GATB FROM READS TO DE BRUIJN GRAPH Each read is split into words named kmers A kmer has a fixed size K Example for K=11 >read 1 ACGACGACGTAGTAAACTACGATCGACTAT ACGACGACGTA kmer 1 CGACGACGTAG kmer 2 GACGACGTAGT kmer 3 CGACGTAGTA kmer ACGATCGACTA kmer 18 CGATCGACTAT kmer 19 G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

6 GATB FROM READS TO DE BRUIJN GRAPH Count kmers and keep solid kmers (i.e. present at least N times) Each solid kmer is inserted as a node of a de Bruijn graph Nodes A,B are connected <=> suffix(a,k-1) = prefix(b,k-1) Two reads sharing (K-1) overlapping will be connected in the de Bruijn graph >read 1 ACGACGACGTAGTAAACTACGATCGACTAT >read 2 CTACGATCGACTATTAGTGATGATAGATAGAT Kmers specific to read 1 Kmers common to read 1 and read 2 Kmers specific to read 2 G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

7 GATB FROM READS TO DE BRUIJN GRAPH In a nutshell Split the reads into kmers 2. Count kmers and keep the solid ones 3. Insert the solid kmers into a de Bruijn graph >read 1 ACGACGACGTAGACGACTAGCTAGCAATGCTA GCTAGGATCAAAACTACGATCGACTAT >read 2 ACTACTACGATCGATGGTCGAGGGCGAGCTAG CTAGCTGACGCTGCTCGCTCTCTCGCT... >read TCTCCTAGCGCGGCGTATACGCGCTAAGCTAG CTCTCGCTGCTCGCTAGCTACGTAGCT... G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

8 GATB FROM READS TO DE BRUIJN GRAPH GATB-CORE only stores nodes of the de Bruijn graph, the edges can be computed on the fly when needed. The nodes are stored in a Bloom filter, a space-efficient structure Two reasons for a very low memory footprint G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

9 GATB FROM READS TO DE BRUIJN GRAPH So, GATB-CORE transforms a set of reads into a compact de Bruijn graph The graph is saved in HDF5 format Such a graph can be used by tools developed with the GATB-CORE C++ Library Reads (FASTA) >14_G AGTCGGCTAGCAT AGTGCTCAGGAGC TTAAACATGCATG AGAG >14_G ATCGACTTCTCTT CTTTTCGAGCTTA GCTAATCA GATB-CORE API Library Binaries Graph (HDF5) GATB-TOOLS Tool 1 Tool 2 Tool 3 G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

10 GATB THE GATB-CORE API GATB-CORE C++ library Fast development of efficient tools Easy and fast learning curve (full documentation with numerous code samples) From developers point of view don't have to bother with the de Bruijn graph construction focus on their own algorithms GATB-CORE high level API G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

11 GATB THE GATB-CORE API The system package holds all the operations related to the operating system: file management, memory management and thread management. The tools package offers generic operations used throughout user applicative code, but not specific to genomic area. The bank package provides operations related to standard genomic sequence dataset management. Using this package allows to write algorithms independently of the input format. The kmer package is dedicated to fine-grained manipulation of k-mers. The debruijn package provides high-level functions to manipulate the de Bruijn graph data structure G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

12 GATB THE GATB-CORE API Details on the graph API (class Graph) Iterate all the nodes of a graph Iterate branching nodes of a graph Get neighbors nodes/edges from a node Get in/out degree from a node Depth First Search from a node Breadth First Search from a node Two use cases of de Bruijn graph navigation Assembly Detection of patterns in the graph G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

13 G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

14 GATB-core Coding session Basic input/output 15 min break Kmerization / graph API Lunch Mini tool: error correction. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

15 Documentation Open How to use the library? : many code examples. Use top right search field to find doc for a class. Sources in the gatb-core library examples/ General design of library, documentation of API. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

16 GATB input/output GATB can : read files in fasta or fastq format. read gzipped files directly. no need to tell which format it is. read list of files. write fasta/fastq files. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

17 Introduction : code io1.cpp # include <gatb / gatb_core.hpp > int main ( int argc, char * argv []) { IBank * inbank = Bank :: open ( argv [1]) ; Iterator < Sequence >* it = inbank - > iterator (); } for (it -> first ();!it -> isdone (); it -> next ()) { // Shortcut Sequence & seq = it -> item (); // We dump the data size and the comment std :: cout << "[" << seq. getdatasize () << "]" << seq. getcomment () << std :: endl ; // We dump the data std :: cout << seq. tostring () << std :: endl ; } G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

18 First project Go to gatb/devel/samples Compile first example with: make io1 Try with./io1 read.fastq G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

19 IO: Task 1, read a file Open the GATB documentation page Have a look at the IBank interface (type IBank in the search field). Scroll down to find methods estimatenbitems and estimatesequencessize. Modify the code to print some bank information. Should look like this : G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

20 IO: Task 1, read a file Open the GATB documentation page Have a look at the IBank interface (type IBank in the search field). Scroll down to find methods estimatenbitems and estimatesequencessize. Modify the code to print some bank information. Should look like this : int64_t nbseq = inbank - > estimatenbitems (); u_int64_t totalsize = inbank - > estimatesequencessize (); std :: cout << " nbseq : " << nbseq << " totalsize " << totalsize << std :: endl ; G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

21 IO: Task 2, write a file Create a new Bank with new BankFasta constructor: IBank * BankFasta ( const std :: string & filename, bool output_fastq = false, bool output_gz = false ) Insert sequences into this bank with insert method. Do not forget to flush the bank at the end with flush(). = You built a fastq/fasta convertor. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

22 IO: Task 2, solution # include <gatb / gatb_core.hpp > int main ( int argc, char * argv []) { IBank * inbank = Bank :: open ( argv [1]) ; IBank * outbank = new BankFasta (" outbank "); Iterator < Sequence >* it = inbank - > iterator (); for (it -> first ();!it -> isdone (); it -> next ()) { // Shortcut Sequence & seq = it -> item (); } outbank -> insert ( seq ); } outbank -> flush (); G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

23 IO: Task 2 The BankFasta constructor has options to output in fastq or gz, try this out! The input bank can be any of fasta, fastq, fasta.gz, fastq.gz, or a list of files. With comma separated list:./ io1 read. fasta, file2. fastq, file3. fasta.gz Or in a file of files, one filename per line: > echo list_of_files read. fasta file2. fasta >./ io1 list_of_files In that case the iterator on IBank is the concatenation of banks. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

24 IO: Task 3 This template can be used to filter/transform sequences. Explore the Sequence API to trim 20 nt at the end of the sequence, and filter out sequences that have more than 20 N On a larger file, a progress bar would be nice! Go to bank5.cpp code snippet (in the gatb doc) to get an example. Hint: use seq.setdataref( & seq.getdata(),...) to trim and seq.getdatabuffer() to count N s G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

25 IO: Task 3, solution # include <gatb / gatb_core.hpp > int main ( int argc, char * argv []) { IBank * inbank = Bank :: open ( argv [1]) ; IBank * outbank = new BankFasta (" outbank "); } ProgressIterator < Sequence > * it = new ProgressIterator < Sequence > (* inbank, " Iterating sequences "); for (it -> first ();!it -> isdone (); it -> next ()) { Sequence & seq = it -> item (); // trim 20 nt at the end seq. setdataref ( & seq. getdata (),0, seq. getdatasize () -20 ); int nbn = 0; char * data = seq. getdatabuffer (); for ( size_t i =0; i<seq. getdatasize (); i ++) { if ( data [i ]== N ) { nbn ++; } } if(nbn <20) outbank -> insert ( seq ); } outbank -> flush (); G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

26 Kmerization Everything in GATB is kmer based! API for kmer extraction. Open File kmer1.cpp // We declare a kmer model Kmer <span >:: ModelDirect model ( kmersize ); // and a kmer iterator Kmer <span >:: ModelDirect :: Iterator itkmer ( model ); compile: make kmer1 then launch:./kmer1../data/read3.fastq Observe output. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

27 Canonical kmer Reads can be on forward or reverse strand, we do not know: each kmer may appear twice. A widespread concept is to keep only the forward or reverse complement of a kmer. By convention, the minimum (in lexic. order) between a kmer and its reverse complement is the Canonical kmer Accessible through ModelCanonical API. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

28 Kmerization Replace ModelDirect with ModelCanonical See doc KmerCanonical. Try itkmer-> value(), forward(), revcomp() G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

29 Parallelization GATB-CORE provides a way to easily parallelize iterations Concept of Dispatcher that takes an iterator as input, creates N thread ; each thread receives and processes iterated items from the iterator through a functor object. We need to modify previous code a little bit. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

30 Parallelization G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

31 Parallelization G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

32 Graph creation G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

33 Graph creation From command line: Graph created with command./dbgh5 -in read.fasta./dbgh5 to get option list. Use dbginfo -in graph.h5 to print information. Within code: Graph::create or Graph::load. Print graph information with: cout << graph. getinfo (); G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

34 minitool: Read correction Remember, graph nodes is the set of solid kmers. solid kmers are error free. Contains method to query graph for a specific kmer. In this example, the graph is only used as the set of solid kmers (to test membership), edges will not be used here. Warning: GATB structure answers set membership queries exactly only for nodes in the graph. For other kmers, the structure is probabilistic (but this is what makes it great) there might be false positives. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

35 Read correction: theory G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

36 Read correction: in the.tar file data/ : example data solution/ : solutions samples/ : your working directory GATB-core is already installed in the VM. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

37 Read correction: task 1 Open minicorrec1.cpp See documentation for Graph create a Graph with Graph graph = Graph :: create (" - in % s - abundance - min % d - kmer - size %d - debloom original ",argv [1], 3, kmersize ); Iterate over canonical kmers of the reads. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

38 Read correction: task 2 Query for each kmer its solidity. print read profiles in the form of: Print a 1 for a solid kmer, a 0 for weak kmer Hint: Build a graph node with // build node from a kmer Node node ( Node :: Value ( itkmer -> value ())); Query node solidity with graph.contains(node) G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

39 Read correction, task 2 solution Kmer <span >:: ModelCanonical model ( kmersize ); Kmer <span >:: ModelCanonical :: Iterator itkmer ( model ); // We loop over sequences. for (it -> first ();!it -> isdone (); it -> next ()) { itkmer. setdata ((* it) -> getdata ()); // loop over kmers of the sequence for ( itkmer. first ();! itkmer. isdone (); itkmer. next ()) { // build node from a kmer Node node ( Node :: Value ( itkmer -> value ())); if( graph. contains ( node )) { printf ("1"); } else { printf ("0"); } } printf ("\n"); } G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

40 Read correction: task 2 Sample output: TGGCTGCCATCAGGAAGAGTTATAACAGGCATTTTATATCCTTATTTGCAGTGGTGACCCACACGAAAGATCACATACAAAGAAAAATTTGTTTATTAAC GGCACCAATAATTTGCCCATCCACAACAACCGGTACGCCGCCTTCCAGCGACGTTAATTTCGGCGCAGTCACGAACGCGGTACGTCCGTTGTTCACCATC CAGATCAATGTGCCGGAAGATAACGACATGGTGCTCGCAATGTATGAACGCCTGGGATATGAACACGCCGACGAGCTGAGTCTGGGTAAGCGTTTGATTG We ll focus on simple cases: an isolated error generates a gap of k weak kmers. G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

41 Read correction: task 3 Start from solution/minicorrec2.cpp if lost :) Detect stretches of k non solid kmers. infer the position of the error. change the nucleotide at this pos, build a kmer and query it in the graph, to find which nucleotide is the correct one. Hint: You can get the sequence in char * with seq.getdatabuffer(); Build a Kmer from a char * with model.codeseed() Build a Node from a kmer with : Node ( Node :: Value ( putative_corrected_kmer. value ())) You may start form the easier file minicorrec2_easier.cpp G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41

42 Read correction Going further: correction of close snps, snps near begin/end of read. validate correction with multiple overlapping kmers many algorithms / heuristics are possible. Bloocoo (G.Benoit, G.Rizk, C.Lemaitre) > short read correction Lordec (L. Salmela, E. Rivals) > hybrid pacbio correction G.Rizk, R.Chikhi (Genscale, Rennes) GATB workshop 15/06/ / 41