Bioinformatica e analisi dei genomi

Transcription

1 Bioinformatica e analisi dei genomi Anno 2016/2017 Pierpaolo Maisano Delser mail: maisanop@tcd.ie

2 Background Laurea Triennale: Scienze Biologiche, Universita degli Studi di Ferrara, Dr. Silvia Fuselli; Laurea Specialistica: Scienze Biomolecolari e Cellulari, Universita degli Studi di Ferrara, Dr. Silvia Fuselli; PhD in Genetics, University of Leicester, prof. Mark A. Jobling; Post-doctoral fellow EPHE-MNHN, Paris, Dr. Stefano Mona. Research Fellow Trinity College, Dublin, Prof. Daniel Bradley Cusco, Marzo 2009

3 Muséum national d'histoire naturelle - Paris

4 Trinity College Dublin - Ireland

5 Informazioni pratiche Teoria + pratica; Software and tools; Files; Slides on the website; Argomenti nuovi / argomenti gia trattati;

6 Informazioni pratiche Cartella di lavoro (fastq file): /home/bioinfo_file/ File referenza, intevalli per il coverage, genoma per IGV): /home/bioinfo_file/reference/ Ricordatevi i percorsi dei file!! pwd: mostra la vostra posizione

7 Programma next-generation sequencing (NGS) come, quando, perche? un esempio di gestione e analisi dati NGS: tipo di dato; file e formati; programmi; interpretazione dei risultati; stima dell errore; quando fermarsi? Applicazioni e/o progetti su diversi organismi.

8 NGS: come, quando, perché? reads trimming sequencing unaligned reads QC de-novo assembly mapping to a reference genome mapping refinement assembly QC mapping QC DNA-seq whole genome capture: exome/custom/cancer amplicon sequencing Filtering RNA-seq whole transcriptome amplicon sequencing: fixed/custom Validation

9 NGS: come, quando, perché? Domanda: quando? Risposta: quando ha senso! Amplicone 400bp in 100 individui? Sanger sequencing Domanda: perche? Risposta: la vostra idea per un progetto! 50 ampliconi in 100 individui? NGS + target capture Gene conversion, elementi ripetuti, recombination breakpoints? NGS + Sanger sequencing

10 un esempio di gestione e analisi dati NGS Roche 454 Illumina MiSeq/HiSeq Ion torrent PGM/Proton Pacific Bioscience (PacBio) Nanopore minion/gridion

11 un esempio di gestione e analisi dati NGS reads trimming sequencing unaligned reads QC de-novo assembly mapping to a reference genome mapping refinement assembly QC mapping QC DNA-seq whole genome capture: exome/custom/cancer amplicon sequencing Filtering RNA-seq whole transcriptome amplicon sequencing: fixed/custom Validation

12 un esempio di gestione e analisi dati NGS progetto progetto:applicazione (whole genomes? Exomes? Target capture? Amplicon sequencing?) progetto:applicazione:scopo (SNPs, indels, repeated elements, CNVs ) progetto:applicazione:scopo:coverage (SNPs, indels, repeated elements, CNVs )

13 Project: Carcharodon carcharias - the great white shark;

14 Project: Carcharodon carcharias; Diploid organism; 82 chromosomes (41 pairs); Genome size ~5.2 Gb not fully sequenced yet; Target capture experiment; Paired-end reads (250bp).

15 Project: Target capture experiment; Meyerson M et al., 2010, NatRevGenetics

16 Single-end (SE) or paired-end (PE) sequencing. fragment ======================================== fragment + adaptors ~~~========================================~~~ SE read > PE reads R > < R2 unknown gap...

17 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

18 .fa/.fasta.fastq.sam (.sai).bam (.bai).vcf sequences read data mapped reads mapped reads (binary) variant information

19 raw reads 1:N:0:1 AGTCAACAACGGGAACAAAATCCTGAAGGTCATGGTATGTGTANNNNTNTTNNNNNCCNNNNNNA TGTGTCNNNNNNNTNNNNNNTCTGAGTNNNNNNNCTCTCTTNNNNNNNAGTGGGTNNNNNNN GCATCCANNNAGCACGATTTTNNNNNNNTATTCAGGAGACAANNNNNNNGTGGGCANNNNNNN GTGTTGGNNNNNNNNNNNNNNGGAGAGANAAAAAANNNNNNNTGAAGTCNNNNNNNNNNN NAGCGNNANNNNNNNTCNNNNNNNNNNNNNNATCANNNNNNNNNNGGTG + 8ACCFGFGGGCDGGGGCFGGGGGGGFGGGGGFEFFGGGFFEGG####9#::#####:9######::CD@ FG#######:######,:99CF?#######::DBFDE#######4::DFG>#######+9A=D@F###88=+<FFF FGG#######++8@8;EEFG8>DG#######+6@DEFF#######*44D=,:##############*/**2:* #212/8C#######*.*2:/9############)-))##0#######,(##############0((,##########-((-

20 raw reads (.fastq) Terminal: more cc_gn2_r1_trimmed.fastq head cc_gn2_r1_trimmed.fastq

21 raw reads 1:N:0:1 AGTCAACAACGGGAACAAAATCCTGAAGGTCATGGTATGTGTANNNNTNTTNNNNNCCNNNNNNA TGTGTCNNNNNNNTNNNNNNTCTGAGTNNNNNNNCTCTCTTNNNNNNNAGTGGGTNNNNNNN GCATCCANNNAGCACGATTTTNNNNNNNTATTCAGGAGACAANNNNNNNGTGGGCANNNNNNN GTGTTGGNNNNNNNNNNNNNNGGAGAGANAAAAAANNNNNNNTGAAGTCNNNNNNNNNNN NAGCGNNANNNNNNNTCNNNNNNNNNNNNNNATCANNNNNNNNNNGGTG + 8ACCFGFGGGCDGGGGCFGGGGGGGFGGGGGFEFFGGGFFEGG####9#::#####:9######::CD@ FG#######:######,:99CF?#######::DBFDE#######4::DFG>#######+9A=D@F###88=+<FFF FGG#######++8@8;EEFG8>DG#######+6@DEFF#######*44D=,:##############*/**2:* #212/8C#######*.*2:/9############)-))##0#######,(##############0((,##########-((-

22 raw reads (.fastq) Instrument ID Run ID flowcell ID coordinates of the cluster lane tile First mate in the pair (paired-end reads) Is the read filtered? No (N) or Yes (Y) Control included? 0=No index 1:N:0:1 AGTCAACAACGGGAACAAAATCCTGAAGGTCATGGTATGTGTANNNNTNTTNNNNNCCNNNNNNA TGTGTCNNNNNNNTNNNNNNTCTGAGTNNNNNNNCTCTCTTNNNNNNNAGTGGGTNNNNNNN GCATCCANNNAGCACGATTTTNNNNNNNTATTCAGGAGACAANNNNNNNGTGGGCANNNNNNN GTGTTGGNNNNNNNNNNNNNNGGAGAGANAAAAAANNNNNNNTGAAGTCNNNNNNNNNNN NAGCGNNANNNNNNNTCNNNNNNNNNNNNNNATCANNNNNNNNNNGGTG + 8ACCFGFGGGCDGGGGCFGGGGGGGFGGGGGFEFFGGGFFEGG####9#::#####:9######::CD@ FG#######:######,:99CF?#######::DBFDE#######4::DFG>#######+9A=D@F###88=+<FFF FGG#######++8@8;EEFG8>DG#######+6@DEFF#######*44D=,:##############*/**2:* #212/8C#######*.*2:/9############)-))##0#######,(##############0((,##########-((- Quality values for each nucleotide

23 raw reads 1:N:0:1 AGTCAACAACGGGAACAAAATCCTGAAGGTCATGGTATGTGTANNNNTNTTNNNNNCCNNNNNNA TGTGTCNNNNNNNTNNNNNNTCTGAGTNNNNNNNCTCTCTTNNNNNNNAGTGGGTNNNNNNN GCATCCANNNAGCACGATTTTNNNNNNNTATTCAGGAGACAANNNNNNNGTGGGCANNNNNNN GTGTTGGNNNNNNNNNNNNNNGGAGAGANAAAAAANNNNNNNTGAAGTCNNNNNNNNNNN NAGCGNNANNNNNNNTCNNNNNNNNNNNNNNATCANNNNNNNNNNGGTG + 8ACCFGFGGGCDGGGGCFGGGGGGGFGGGGGFEFFGGGFFEGG####9#::#####:9######::CD@ FG#######:######,:99CF?#######::DBFDE#######4::DFG>#######+9A=D@F###88=+<FFF FGG#######++8@8;EEFG8>DG#######+6@DEFF#######*44D=,:##############*/**2:* #212/8C#######*.*2:/9############)-))##0#######,(##############0((,##########-((- Quality values for each nucleotide Lowest ASCII Highest!"#$%&'()*+,-./ :;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{ }~ Illumina 1.8+ Phred+33, raw reads typically (0, 41)

24 raw reads (.fastq) Lowest ASCII }~ Illumina 1.8+ Phred+33, raw reads typically (0, 41) Phred-scale value: Q = -10*log_10P P = 10 -Q/10 Phred Quality Score (Q) Probability of incorrect base call (P) 10 1 in 10 90% 20 1 in % 30 1 in % 40 1 in % 50 1 in % Base call accuracy

25 raw reads (.fastq) Open cc_gn2_r2_trimmed.fastq Terminal: more cc_gn2_r2_trimmed.fastq OR head cc_gn2_r2_trimmed.fastq Are cc_gn2_r1_trimmed.fastq and cc_gn2_r2_trimmed.fastq coming from two different lanes? What s the difference between the two fastq files?

26 raw reads (.fastq) cc_gn2_r1_trimmed.fastq Same lane, different read mate in the 1:N:0:1 AGTCAACAACGGGAACAAAATCCTGAAGGTCATGGTATGTGTANNNNTNTTNNNNNCCNNNNNNA TGTGTCNNNNNNNTNNNNNNTCTGAGTNNNNNNNCTCTCTTNNNNNNNAGTGGGTNNNNNNN GCATCCANNNAGCACGATTTTNNNNNNNTATTCAGGAGACAANNNNNNNGTGGGCANNNNNNN GTGTTGGNNNNNNNNNNNNNNGGAGAGANAAAAAANNNNNNNTGAAGTCNNNNNNNNNNN NAGCGNNANNNNNNNTCNNNNNNNNNNNNNNATCANNNNNNNNNNGGTG 2:N:0:1 CCATTTCTNNNNNNNAGGACCTNNNNNNNAGCCCTNNNNNNNNNNNNNAGNATATGANNNNN NNTCTTATTNANCCANNNTCTAGNNNNNNNCTTTCCTNNNNNNNTCTCTGANNNNNNNNNNNN NNCCCTTCCNNTNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNTTNTCTCNTNNNNNNNN NNNNAAAATCCNNNNNNNNNNNNNNCCACTAANNNNNNNNNNNNNNAAGAAATAACACACNN NNNNNACAAAAANNNNNNNACAACACNNNNNNNGCATAAANNNA

27 raw reads (.fastq) 1. Fastq quality control + trimming Adapters? Low quality bases? 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment GATK base recalibration GATK duplicate removal picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels single/multi-sample samtools raw variants (.vcf) variant score recalibration known SNPs/indels big datasets 6. variant filtering and validation vcftools in silico vs in vitro validation

28 1- Fastq quality control + trimming Fastqc: quality control of the raw data coming out from the sequencer Evaluation of the quality of the generated data; Basic summary statistics of the raw data; Several modules to evaluate different features (i.e. adapters; base quality, etc ) Feedback (green, orange, red): do not fully rely on that, think what does it mean!!

29 1- Fastq quality control + trimming

30 1- Fastq quality control + trimming Per base sequence quality: warning

31 1- Fastq quality control + trimming What can we do to improve the quality at the end of the reads? Read Trimming: removal of lower-quality 3' Ends with Low Quality Scores bp bp

32 1- Fastq quality control + trimming Per sequence quality score: pass

33 1- Fastq quality control + trimming Sequence length: pass

34 1- Fastq quality control + trimming Adapters removal Failed Warning

35 1- Fastq quality control + trimming Adapters removal Pass

36 1- Fastq quality control + trimming Overrepresented sequences Removal of overrepresented sequences (PCR primers).

37 FASTQC references: Software website: Manual:

38 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

39 2- Alignment to a reference genome Alignment : process of determining the most likely location within the genome for the observed DNA read raw reads reference genome

40 2- Alignment to a reference genome the shorter the read the harder is to find its location in the genome big amount of data: computationally challenging for memory and speed trade-off: speed vs sensitivity the higher the accuracy the longer the alignment run two classes of methods: Burrows-Wheeler Fast less robust at high divergence with reference genome e.g. bwa Hashing slow (needs more memory) robust at high divergence with reference genome e.g. stampy BW: Hashing:

41 2- Alignment to a reference genome What if there are several possible places to align your sequencing read? This may be due to: - Repeated elements in the genome - Low complexity sequences - Reference errors and gaps MQ is a phred-score of the quality of the alignment raw reads reference genome low MQ: the probability of mapping to different locations is high, but no perfect multiple matches high MQ: a single match MQ0: a perfect multiple match

42 2- Alignment to a reference genome This may be due to: - Repeated elements in the genome - Low complexity sequences - Reference errors and gaps Reference sequence Element 1 Element 2 Sample_1 Reference sequence 1 copia 1 copia Sample_1 1 copia 1 copia

43 2- Alignment to a reference genome This may be due to: - Repeated elements in the genome - Low complexity sequences - Reference errors and gaps Reference sequence Element 1 Element 1 Element 2 Sample_1 Perfect mul ple matches MQ0 Not a perfect match Low MQ

44 2- Alignment to a reference genome This may be due to: - Repeated elements in the genome - Low complexity sequences - Reference errors and gaps Reference sequence Element 1 Element 1 Element 2 Sample_1 Perfect mul ple matches MQ0 Not a perfect match Low MQ Reference sequence 2copia 1 copia Sample_1 1 copia 1 copia

45 2- Alignment to a reference genome This may be due to: - Repeated elements in the genome - Low complexity sequences - Reference errors and gaps Reference sequence Element 1 Element 2 Sample_1 False heterozygous call Cluster of heterozygotes Reference sequence Sample_1 1 copia 2copia 1 copia 1 copia

46 2- Alignment to a reference genome This may be due to: - Repeated elements in the genome - Low complexity sequences - Reference errors and gaps AluSg7

47 2- Alignment to a reference genome This may be due to: - Repeated elements in the genome - Low complexity sequences - Reference errors and gaps

48 2- Alignment to a reference genome: mapping with bwa-mem Three different algorithm: 1. BWA-backtrack: for illumina reads up to 100bp; 2. BWA-SW: long read support, split alignment; 3. BWA-MEM: long read support, split alignment, faster, more accurate Fastq files are already trimmed adapters removed

49 2- Alignment to a reference genome: mapping with bwa-mem paired-end alignment; it uses the reference genome (.fa) and the reads (.fastq) to create a SAM file; Option to mark shorter split hits as secondary (not supplementary). Split read: Karacok E et al., 2012

50 2- Alignment to a reference genome: mapping with bwa-mem paired-end alignment; it uses the reference genome (.fa) and the reads (.fastq) to create a SAM file; Option to mark shorter split hits as secondary (not supplementary). bwa mem [options] [RefSeq] [fastq1] [fastq2] > cc_gn2_r12.sam Type: bwa mem to check the options

51 2- Alignment to a reference genome: mapping with bwa-mem paired-end alignment; it uses the reference genome (.fa) and the reads (.fastq) to create a SAM file; Option to mark shorter split hits as secondary (not supplementary). Mark shorter split hits as secondary Reference genome Fastq 1 bwa mem -M cc_ref.fa cc_gn2_r1_trimmed.fastq cc_gn2_r2_trimmed.fastq > cc_gn2_r12.sam Fastq 2

52 2- Alignment to a reference genome: from sam to bam Convert sam-to-bam: samtools view input_sam.. input_bam Option to define that the input is a sam file; Option to have output in bam format; Option to define the output;

53 2- Alignment to a reference genome: from sam to bam Input is a sam file sam-to-bam Output in bam format Input file (sam) samtools view -Sb cc_gn2_r12.sam -o cc_gn2_r12.bam Output file (bam)

54 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

55 aligned reads (.sam/.bam) SAM/BAM format SAM sequence alignment map BAM binary alignment map Standard formats for alignment BAM is the binary version of SAM reduced size, easier to store and to access but the full information is not readable by human eye

56 aligned reads (.sam/.bam) more cc_gn2_r12.bam

57 aligned reads (.sam/.bam) BAM format

58 aligned reads (.sam/.bam) more cc_gn2_r12.sam

59 aligned reads (.sam/.bam) SAM format

60 aligned reads (.sam/.bam) SAM sequence alignment map BAM binary alignment map They consist of two parts: 1. Header: contains information about the sample 2. Alignment: contains location and qualities for all the reads You can find a detailed explanation in the sam/bam format specification (

61 aligned reads (.sam/.bam) SAM format Header Alignment

62 aligned reads (.sam/.bam) SAM sequence alignment map BAM binary alignment map They consist of two parts: 1. Header: contains information about the sample Header header Reference sequence dictionary, one per chromosome, SN (reference sequence name) and LN (reference sequence Read Program, ID comment

63 aligned reads (.sam/.bam) They consist of two parts: SAM sequence alignment map BAM binary alignment map 1. Header: contains information about the sample Reference Sequence Name Reference Sequence Length

64 aligned reads (.sam/.bam) SAM format Header Alignment

65 aligned reads (.sam/.bam) They consist of two parts: SAM sequence alignment map BAM binary alignment map 2. Alignment: contains location and qualities for all the reads Alignment contains one line per read, and each line contains 12 columns:

66 aligned reads (.sam/.bam) SAM sequence alignment map BAM binary alignment map 2. Alignment: contains location and qualities for all the reads M00725:28: AJ72K:1:1101:18215: cc_ref M = CTCCTTCACCAGATGGATTCTCGCCTTACAGTCCTGAGGAAACTAACCGCAGAGTCAACAAAG TAATGCGAGNNNNNNNGTACTTGCTACAGCNANNNGGTCCAAATNNNTNNNTTATTGGNNNAGATGTT CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGG#######::CFGGGGGGGGGG#:###:9BFGGGGG###:###::DFGDG###4+

67 aligned reads (.sam/.bam) SAM sequence alignment map BAM binary alignment map 2. Alignment: contains location and qualities for all the reads QNAME M00725:28: AJ72K:1:1101:18215: cc_ref M = FLAG CTCCTTCACCAGATGGATTCTCGCCTTACAGTCCTGAGGAAACTAACCGCAGAGTCAACAAAG TAATGCGAGNNNNNNNGTACTTGCTACAGCNANNNGGTCCAAATNNNTNNNTTATTGGNNNAGATGTT CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGG#######::CFGGGGGGGGGG#:###:9BFGGGGG###:###::DFGDG###4+

68 aligned reads (.sam/.bam) bitwise FLAG It is an integer, but it represents the sum of different values. QNAME FLAG M00725:28: AJ72K:1:1101:18215: Open Firefox > google.co.uk > Type bitwise flag broad There is a tool online which provides a quick translation (

69 aligned reads (.sam/.bam) bitwise FLAG It is an integer, but it represents the sum of different values. QNAME FLAG M00725:28: AJ72K:1:1101:18215: There is a tool online which provides a quick translation (

70 aligned reads (.sam/.bam) SAM sequence alignment map BAM binary alignment map 2. Alignment: contains location and qualities for all the reads QNAME FLAG RNAME POS MAPQ M00725:28: AJ72K:1:1101:18215: cc_ref CIGAR 300M = CTCCTTCACCAGATGGATTCTCGCCTTACAGTCCTGAGGAAACTAACCGCAGAGTCAACAAAG TAATGCGAGNNNNNNNGTACTTGCTACAGCNANNNGGTCCAAATNNNTNNNTTATTGGNNNAGATGTT CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGG#######::CFGGGGGGGGGG#:###:9BFGGGGG###:###::DFGDG###4+

71 aligned reads (.sam/.bam) CIGAR string It is a compact representation of sequence alignment. It includes: M match or mismatch I insertion D deletion read: ACTCA TGCAGT ref: ACTCAGTG GT cigar 5M1D2M2I2M So, what is the cigar line of? read: ACGTCATG CAGT ref: ACG CATGCGGCAGT cigar 3M1I4M3D4M

72 aligned reads (.sam/.bam) SAM sequence alignment map BAM binary alignment map 2. Alignment: contains location and qualities for all the reads QNAME FLAG RNAME POS MAPQ M00725:28: AJ72K:1:1101:18215: cc_ref CIGAR MRNM 300M = CTCCTTCACCAGATGGATTCTCGCCTTACAGTCCTGAGGAAACTAACCGCAGAGTCAACAAAG TAATGCGAGNNNNNNNGTACTTGCTACAGCNANNNGGTCCAAATNNNTNNNTTATTGGNNNAGATGTT CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGG#######::CFGGGGGGGGGG#:###:9BFGGGGG###:###::DFGDG###4+

73 aligned reads (.sam/.bam) SAM sequence alignment map BAM binary alignment map 2. Alignment: contains location and qualities for all the reads QNAME FLAG RNAME POS MAPQ M00725:28: AJ72K:1:1101:18215: cc_ref CIGAR MRNM MPOS ISIZE 300M = SEQ CTCCTTCACCAGATGGATTCTCGCCTTACAGTCCTGAGGAAACTAACCGCAGAGTCAACAAAG TAATGCGAGNNNNNNNGTACTTGCTACAGCNANNNGGTCCAAATNNNTNNNTTATTGGNNNAGATGTT QUAL CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGG#######::CFGGGGGGGGGG#:###:9BFGGGGG###:###::DFGDG###4+

74 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

75 3- Bam refinement before starting BAM missing a RG LINE what is a RG LINE?? You can find a detailed explanation in the sam/bam format specification (

76 3- Bam refinement before starting picard-tools AddOrReplaceReadGroups INPUT=cc_gn2_R12.bam OUTPUT=cc_gn2_R12_rg.bam RGLB=cc_gn2 RGPL=Illumina RGPU=01 RGSM=shark

77 3- Bam refinement before starting sort the bam (this adds the bam extension automatically!) It sorts alignments by coordinates samtools sort cc_gn2_r12_rg.bam cc_gn2_r12_rg_sorted Index the sorted bam file samtools index cc_gn2_r12_rg_sorted.bam

78 aligned reads (.sam/.bam) BAM format

79 aligned reads (.sam/.bam) We can read the header of the BAM file use samtools to check the header of the BAM samtools view -H cc_gn2_r12_rg_sorted.bam 1. How many chromosomes are present in your header? 2. Which version of the BAM is it? 3. Is it sorted?

80 aligned reads (.sam/.bam) Yes, by VN:1.4 SN:cc_ref ID:1 PU:01 LB:cc_gn2 SM:shark PL:Illumina 1 chromosome ( artificial )

81 aligned reads VN:1.3 SN:I SN:II SN:III SN:IV SN:IX SN:Mito SN:V SN:VI SN:VII SN:VIII SN:X SN:XI SN:XII SN:XIII SN:XIV SN:XV SN:XVI ID:bwa PN:bwa VN: r789 CL:bwa mem -M

82 3- Bam refinement Input: BAM Three main steps: 1. Local realignment 2. Base quality recalibration 3. Duplicate removal Output: BAM

83 3- Bam refinement Local realignment Problem: Short indels in the sample relative to the reference sequence can pose difficulties for alignment programs. Indels occuring towards the ends of the reads are often not aligned correctly, introducing an excess of SNPs Ref: ACTTTCGGATGCTGATCGGGATGCTTTAGCTGATGCTGATGGGCTTTCGATCGATTTAAAAGCT ACTTTCGGATGCTGATCGGGATGCTTTAGCTGA TCGGATGCTGATCGGGATGCTTTAGCTGATGCT CTGATCGGGATGCTTTAGCTGATGCTGATGG Ref: ACTTTCGGATGCTGATCGGGATGCTTTAGCTGATGCTGATGGGCTTTCGATCGATTTAAAAGCT ACTTTCGGATGCTGATC ATGCTTTAGCTGA TCGGATGCTGATC T_GCTTTAGCTGATGCT CTGATC ATGCTTTAGCTGATGCTGATGG TC T_GCTTTAGCTGATGCTGATGGGCTT Ref: ACTTTCGGATGCTGATCGGGATGCTTTAGCTGATGCTGATGGGCTTTCGATCGATTTAAAAGCT ACTTTCGGATGCTGATC ATGCTTTAGCTGA TCGGATGCTGATC ATGCTTTAGCTGATGCT CTGATC ATGCTTTAGCTGATGCTGATGG TC ATGCTTTAGCTGATGCTGATGGGCTT

84 3- Bam refinement Local realignment It uses the full alignment context to determine whether the indel exists. Two-step process: 1. RealignerTargetCreator: it determines the small suspicious intervals which are likely in need of realignment 2. IndelRealigner: it runs the realignment on those intervals notes: - having a list of known indels helps

85 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

86 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

87 3- Bam refinement Base Quality Recalibration Each base call has an associated base call quality (phred-scale). Rule of thumb: anything less than Q20 is not useful data. The quality of a call depends on multiple factors (e.g. position in the read, sequence context). In addition, the alignment can provide useful information. Mismatches to the reference are considered errors (unless they are described polymoprhisms). It requires a catalogue of variable sites! How does it work?

88 3- Bam refinement Base Quality Recalibration BAM files with variants List of know variants 123 C/A BQ cov1 cov2 cov3 145 G/A BQ cov1 cov2 cov G/T BQ cov1 cov2 cov3 123 C/A 145 G/A 1345 C/T 1345 C/T BQ cov1 cov2 cov C/G BQ cov1 cov2 cov3 Considered as real variants Recalibrated BQ using different covariates BQ: base quality Covariates: position in the read, sequencing cycle, dinucleotide,

89 3- Bam refinement Base Quality Recalibration BAM files with variants 123 C/A BQ cov1 cov2 cov3 145 G/A BQ cov1 cov2 cov G/T BQ cov1 cov2 cov3 Considered as real variants Recalibrated BQ using different covariates High quality >>>>>>>>>>>>> Low Quality 123 A 1345 C/T BQ cov1 cov2 cov C/G BQ cov1 cov2 cov T 145 A BQ: base quality Covariates: position in the read, end of read with worse calls! 1298 T 1789 G

90 3- Bam refinement Base Quality Recalibration BAM files with variants 123 C/A BQ cov1 cov2 cov3 145 G/A BQ cov1 cov2 cov G/T BQ_1 cov1 cov2 cov3 Considered as real variants Recalibrated BQ using different covariates High quality >>>>>>>>>>>>> Low Quality 123 A 1345 C/T BQ cov1 cov2 cov C/G BQ_1 cov1 cov2 cov T 145 A BQ: base quality Covariates: position in the read, end of read with worse calls! 1298 T 1789 G

91 3- Bam refinement Base Quality Recalibration Covariate: cycle number First cycle: higher quality Last cycles: lower quality

92 3- Bam refinement Base Quality Recalibration We will not run the Base Quality Recalibration because of time and list of variants available. Few more details: It supports several platforms: Illumina, SOLiD, 454, Complete Genomics, Pacific Biosciences (stated on the website) and IonTorrent (stated in the GATK forum). You can find how to do it at: ers_bqsr_baserecalibrator.php

93 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

94 3- Bam refinement Duplicate removal PCR is used during library preparation. This can result in duplicate DNA fragments in the final library prep. What we want: information from independent fragments What we do not want: copies of the same information coming from one fragment Reference Sequence

95 3- Bam refinement Duplicate removal PCR is used during library preparation. This can result in duplicate DNA fragments in the final library prep. What we want: information from independent fragments What we do not want: copies of the same information coming from one fragment Reference Sequence

96 3- Bam refinement Duplicate removal C C C C A Ref call Possible heterozygote, SNP call

97 3- Bam refinement Duplicate removal Why is it important to do it? It can result in false SNPs calls. Duplicates may fake a high coverage thus giving high support to some variants. PCR-free protocols exist but require a large amount of DNA.

98 3- Bam refinement Duplicate removal Number of duplicates varies according to the complexity of the library: whole genome experiments (<5%) custom enrichment ones (<30%) It must be done after alignment and at the library level. How does it work? It identifies read-pairs where the outer ends map to the same position on the genome and removes all but one copy.

99 3- Bam refinement Duplicate removal Duplicate removal Module used picard-tools MarkDuplicates INPUT=cc_gn2_R12_rg_sorted.bam OUTPUT=cc_gn2_R12_rg_sorted_rmdup.bam METRICS_FILE=dupl_metrics.txt Input file Output file Metrics file

100 3- Bam refinement Duplicate removal Duplicate removal picard-tools MarkDuplicates INPUT=cc_gn2_R12_rg_sorted.bam OUTPUT=cc_gn2_R12_rg_sorted_rmdup.bam METRICS_FILE=dupl_metrics.txt What do we have to do after each step??? Sort and Index the newly generated BAM file

101 3- Bam refinement Duplicate removal sort the bam samtools sort cc_gn2_r12_rg_sorted_rmdup.bam cc_gn2_r12_rg_sorted_rmdup_sorted Index the sorted bam file samtools index cc_gn2_r12_rg_sorted_rmdup_sorted.bam

102 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

103 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

104 3- Bam refinement BAM check BAM check gives us answers to several questions: How many duplicates do I have? Is that reasonable for my experiment? Duplicate removal How many of my reads mapped back to the reference? How many of these are paired in mapping? How many pairs are mapped to different chromosomes? Alignment Stats How much average coverage do I have? Is the coverage evenly distributed along my region? Coverage

105 3- Bam refinement BAM check: duplicate removal picard-tools MarkDuplicates INPUT=cc_gn2_R12_rg_sorted.bam OUTPUT=cc_gn2_R12_rg_sorted_rmdup.bam METRICS_FILE=dupl_metrics.txt

106 3- Bam refinement BAM check: duplicate removal Open the file dupl_metrics.txt More/cat/gedit 1) Check the % of duplicates; ~20.3%

107 3- Bam refinement BAM check: duplicate removal ## net.sf.picard.metrics.stringheader # net.sf.picard.sam.markduplicates INPUT=[/media/pier/pierWD/backup/freecom/work/teaching/Unife_bioinfo_122016/material/white_shark_fastq_gn2/whi te_shark/dd10/final_files/cc_gn2_r12_rg_sorted.bam] OUTPUT=/media/pier/pierWD/backup/freecom/work/teaching/Unife_bioinfo_122016/material/white_shark_fastq_gn2/w hite_shark/dd10/final_files/cc_gn2_r12_rg_sorted_rmdup.bam METRICS_FILE=/media/pier/pierWD/backup/freecom/work/teaching/Unife_bioinfo_122016/material/white_shark_fastq_g n2/white_shark/dd10/final_files/dupl_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM= CREATE_INDEX=false CREATE_MD5_FILE=false ## net.sf.picard.metrics.stringheader # Started on: Thu Nov 10 18:37:26 GMT 2016 ## METRICS CLASS net.sf.picard.sam.duplicationmetrics LIBRARY UNPAIRED_READS_EXAMINED READ_PAIRS_EXAMINED UNMAPPED_READS UNPAIRED_READ_DUPLICATES READ_PAIR_DUPLICATES READ_PAIR_OPTICAL_DUPLICATES PERCENT_DUPLICATION ESTIMATED_LIBRARY_SIZE cc_gn We have 20.3% of PCR duplicates in our experiment

108 3- Bam refinement BAM check BAM check gives us answers to several questions: How many duplicates do I have? Is that reasonable for my experiment? Duplicate removal How many of my reads mapped back to the reference? How many of these are paired in mapping? How many pairs are mapped to different chromosomes? Alignment Stats How much average coverage do I have? Is the coverage evenly distributed along my region? Coverage

109 3- Bam refinement BAM check: alignment stats Run flagstat on the BAM file before and after BAM refinement, can you see any difference? BEFORE: samtools flagstat cc_gn2_r12_rg_sorted.bam AFTER: samtools flagstat cc_gn2_r12_rg_sorted_rmdup.bam

110 3- Bam refinement BAM check: alignment stats in total (QC-passed reads + QC-failed reads) duplicates mapped (25.51%:-nan%) paired in sequencing read read properly paired (19.71%:-nan%) with itself and mate mapped singletons (2.91%:-nan%) with mate mapped to a different chr with mate mapped to a different chr (mapq>=5) in total (QC-passed reads + QC-failed reads) duplicates mapped (25.51%:-nan%) paired in sequencing read read properly paired (19.71%:-nan%) with itself and mate mapped singletons (2.91%:-nan%) with mate mapped to a different chr with mate mapped to a different chr (mapq>=5 BEFORE AFTER

111 3- Bam refinement BAM check: alignment stats duplicates LIBRARY UNPAIRED_READ_DUPLICATES READ_PAIR_DUPLICATES PERCENT_DUPLICATION cc_gn (8687*2) = 24400

112 3- Bam refinement BAM check BAM check gives us answers to several questions: How many duplicates do I have? Is that reasonable for my experiment? Duplicate removal How many of my reads mapped back to the reference? How many of these are paired in mapping? How many pairs are mapped to different chromosomes? Alignment Stats How much average coverage do I have? Is the coverage evenly distributed along my region? Coverage

113 3- Bam refinement BAM check: coverage estimation Depth of Coverage The number of reads that spans a given DNA sequence of interest. This is commonly expressed in terms of Yx where Y is the number of reads and x is the unit reflecting the depth of coverage metric (i.e. 5x, 10x, 20x, 100x) 7x 11x 9x

114 3- Bam refinement BAM check: coverage estimation GATK, coverage per base Reference sequence Input file java -jar /opt/gatk-3.5-0/genomeanalysistk.jar -I cc_gn2_r12_rg_sorted_rmdup_sorted.bam -R cc_ref.fa -T DepthOfCoverage Module used -o cc_gn2_r12_rg_sorted_rmdup_coverage -L coverage.intervals List of intervals Output

115 3- Bam refinement BAM check: coverage estimation GATK, coverage per base -L coverage.intervals List of intervals, cc_ref:

116 3- Bam refinement BAM check: coverage estimation gedit cc_gn2_r12_rg_sorted_rmdup_coverage.sample_summ ary OR More/Cat/Head sample_id total mean granular_third_quartile granular_median granular_first_quartile %_bases_above_15 shark Total N/A N/A N/A sample_id total mean granular_third_quartile granular_median granular_first_quartile %_bases_above_15 shark Total N/A N/A N/A

117 3- Bam refinement BAM check: coverage estimation Per base coverage gedit cc_gn2_r12_rg_sorted_rmdup_coverage OR More/Cat/Head Locus Total_Depth Average_Depth_sample Depth_for_shark cc_ref: cc_ref: cc_ref: cc_ref: cc_ref: cc_ref: cc_ref: cc_ref: cc_ref: cc_ref: cc_ref:

118 3- Bam refinement BAM check: coverage estimation Open R: Open a terminal; Type R;

119 3- Bam refinement BAM check: coverage estimation data<read.table("cc_gn2_r12_rg_sorted_rmdup_coverag e", sep="\t", header=t) names(data) old_col<-data$locus new_col<gsub("cc_ref:","",as.character(old_col)) data["pos"]<-new_col plot(data$pos, data$depth_for_shark, type="l")

120 3- Bam refinement BAM check: coverage estimation Coverage Position (bp)

121 3- Bam refinement BAM check: coverage estimation

122 3- Bam refinement BAM check: coverage estimation Why is important to check the coverage? To check how your experiment performed (one of the ways to assess the quality of your experiment); To understand how confident can you be with your data; To decide on filtering after variant calling; To look for structural variation.

123 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

124 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

125 4- BAM check: visualisation Different tools to visualise aligned NGS data: IGV: Tablet: Samtools tview

126 4- BAM check: visualisation Samtools tview: Basic visualization tool; Terminal based; No need of RAM/Java/extra packages;

127 4- BAM check: visualisation samtools tview cc_gn2_r12_rg_sorted_rmdup_sorted.bam -p cc_ref:

128 4- BAM check: visualisation Input: bam file samtools tview cc_gn2_r12_rg_sorted_rmdup_sorted.bam -p cc_ref: Position (Chr:bp)

129 4- BAM check: visualisation Reference sequence Consensus Position Read

130 4- BAM check: visualisation. : base matching positive strand;, : base matching negative strand; underlined: secondary or orphan; Uppercase letters: base matching positive strand; Lowercase letters: base matching negative strand;

131 4- BAM check: visualisation Press.

132 4- BAM check: visualisation Reverse strand Positive strand

133 4- BAM check: visualisation?: menu m: mapping quality n: nucleotide b: base quality.: on/off dots q: exit Secondary or orphan Colours for quality

134 4- BAM check: visualisation Press q, m Press? MQ >=30 0 MQ 9

135 4- BAM check: visualisation Press b Press? 20 BQ BQ 19 BQ 30 0 BQ 9

136 4- BAM check: visualisation Press n What does it happen?? The four nucleotides are highlighted by four different colours, no more mapping or base quality

137 4- BAM check: visualisation

138 4- BAM check: visualisation 1. Is there any polymorphic site between position 1,174,361 and 1,174,391? 2. Is there any polymorphic sites between position 1,233,201 and 1,233,221? If so, state the reference and alterative allele, the average quality of the base (BQ), the average mapping quality of the read (MQ) and how would you call that site (i.e. homozygous reference, heterozygous, homozygous alternative).

139 4- BAM check: visualisation

140 4- BAM check: visualisation 1. Is there any polymorphic site between position 1,174,361 and 1,174,391? No polymorphic sites between position 1,174,361 and 1,174, Is there any polymorphic sites between position 1,233,201 and 1,233,221? If so, state the reference and alterative allele, the average quality of the base (BQ), the average mapping quality of the read (MQ) and how would you call that site (i.e. homozygous reference, heterozygous, homozygous alternative).

141 4- BAM check: visualisation

142 4- BAM check: visualisation Yes, there is one polymorphic sites Reference allele: G Alternative allele: T Call: possible homozygous alternative (T/T) Average BQ: BQ 30 (white) Average MQ: MQ 30 (white)

143 4- BAM check: visualisation 1. Is there any polymorphic site between position 4761 and 4771? No polymorphic sites between position 4761 and Is there any polymorphic sites between position1781 and 1791? Yes, there is one polymorphic sites Reference allele: G Alternative allele: T Call: possible homozygous alternative (T/T) Average BQ: BQ 30 Average MQ: MQ 30 If so, state the reference and alterative allele, the average quality of the base (BQ), the average mapping quality of the read (MQ) and how would you call that site (i.e. homozygous reference, heterozygous, homozygous alternative).

144 4- BAM check: visualisation IGV

145 4- BAM check: visualisation IGV

146 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

147 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

148 5- Variant calling SNPs indels SV samtools GATK: 1. Unified Genotyper 2. Haplotype caller samtools GATK: 1. Unified Genotyper 2. Haplotype caller Dindel SVMerge pipeline combining many different tools SNPs: Single Nucleotide Polymorphisms Indels: insertions/deletions SV: Structural Variation

149 5- Variant calling Variant calling: Examine the bases aligned to position and look for differences What we are looking for: Polymorphic sites / monomorphic sites Factors to consider: - Base call qualities of each supporting base - Proximity to indels and homopolymer run - Mapping qualities of the reads supporting the SNP (increased read length or paired-end help MQ scores) - Sequencing depth - Individual vs multi-sample calling

150 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

151 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

152 5- Variant calling About the variant file: raw variants (.vcf) = filtered variants (.vcf) variants (.vcf)

153 variants (.vcf) Standardised format for storing DNA polymorphism data - SNPs, indels, SV - Rich annotations Can be indexed for fast data retrieval of variants from a range of positions Can store variant information over many samples Record meta-data about the site - dbsnp accession, filter status Very flexible - Tags can be introduced to describe new types of variants - Different VCF files may contain different information/annotations

154 variants (.vcf) Data Header

155 variants (.vcf) Header Something specific to the header lines???

156 variants (.vcf) Header Lines starting with ##: arbitrary number of meta-information lines ##INFO=<ID=AN,Number=1,Type=Integer,Description="Total number of alleles in called genotypes"> ##INFO=<ID=DP4,Number=4,Type=Integer,Description="Number of high-quality ref-forward, ref-reverse, alt-forward and alt-reverse bases"> ##INFO=<ID=MQ,Number=1,Type=Integer,Description="Average mapping quality">

157 variants (.vcf) Header line starting with #: column definition mandatory columns include:

158 variants (.vcf) Header line starting with #: column definition mandatory columns include:

159 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM chromosome

160 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS chromosome position of the start of the variant

161 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs)

162 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID REF chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs) reference allele

163 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID REF ALT chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs) reference allele comma separated list of alternate non-reference alleles

164 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID REF ALT QUAL chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs) reference allele comma separated list of alternate non-reference alleles phred-scaled quality score

165 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID REF ALT QUAL FILTER chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs) reference allele comma separated list of alternate non-reference alleles phred-scaled quality score site filtering information

166 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID REF ALT QUAL FILTER INFO chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs) reference allele comma separated list of alternate non-reference alleles phred-scaled quality score site filtering information user extensible annotation (e.g. samtools and GATK may differ in this)

167 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID REF ALT QUAL FILTER INFO FORMAT chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs) reference allele comma separated list of alternate non-reference alleles phred-scaled quality score site filtering information user extensible annotation (e.g. samtools and GATK may differ in this) how the information for each sample is presented (i.e. GT:DP:DV:SP:DPR)

168 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID REF ALT QUAL FILTER INFO FORMAT chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs) reference allele comma separated list of alternate non-reference alleles phred-scaled quality score site filtering information user extensible annotation (e.g. samtools and GATK may differ in this) how the information for each sample is presented (i.e. GT:DP:DV:SP:DPR) samples follow

169 variants (.vcf) Header line starting with #: column definition mandatory columns include: CHROM POS ID REF ALT QUAL FILTER INFO FORMAT chromosome position of the start of the variant unique identifier of the variant (e.g. rs number for SNPs) reference allele comma separated list of alternate non-reference alleles phred-scaled quality score site filtering information user extensible annotation (e.g. samtools and GATK may differ in this) how the information for each sample is presented (i.e. GT:DP:DV:SP:DPR) samples follow

170 variants (.vcf) Data Header

171 variants (.vcf) Data one line per site (all columns described above per line); useful information per site and per sample;

172 5- Variant calling Software and tools: samtools and bcftools Input: cc_gn2_r12_rg_sorted_rmdup_sorted.bam????? You have to create the command needed depending on what we want to have in the final vcf file.

173 5- Variant calling samtools mpileup cc_gn2_r12_rg_sorted_rmdup_sorted.bam bcftools view > cc_gn2_bq20_mq40.vcf samtools mpileup: No indel calling; Parameter for adjusting mapq (use 50 as value) ; Base quality minimum 20; Mapping quality minimum 40; Output per-sample strand bias P-value (SP); Output per-sample Depth of Coverage (DP); Generate BCF output; Specify reference sequence file.

174 5- Variant calling samtools mpileup cc_gn2_r12_rg_sorted_rmdup_sorted.bam bcftools view > cc_gn2_bq20_mq40.vcf Bcftools view: Call genotypes at variant sites; Variant sites only.

175 5- Variant calling samtools mpileup cc_gn2_r12_rg_sorted_rmdup_sorted.bam bcftools view > cc_gn2_bq20_mq40.vcf How to know all the options corresponding to these requirements??? Type samtools mpileup in the terminal; Type bcftools view in the terminal.

176 5- Variant calling samtools mpileup cc_gn2_r12_rg_sorted_rmdup_sorted.bam bcftools view > cc_gn2_bq20_mq40.vcf samtools mpileup: 1. No indel calling; 2. Parameter for adjusting mapq (use 50 as value) ; 3. Base quality minimum 20; 4. Mapping quality minimum 40; 5. Output per-sample strand bias P-value (SP); 6. Output per-sample Depth of Coverage (DP); 7. Generate BCF output; 8. Specify reference sequence file. Bcftools call: 1. Call genotypes at variant sites; 2. Variant sites only.

177 5- Variant calling samtools mpileup cc_gn2_r12_rg_sorted_rmdup_sorted.bam bcftools view > cc_gn2_bq20_mq40.vcf samtools mpileup: 1. No indel calling; -I 2. Parameter for adjusting mapq (use 50 as value) ; -C Base quality minimum 20; -Q Mapping quality minimum 40; -q Output per-sample strand bias P-value (SP); -S 6. Output per-sample Depth of Coverage (DP); -D 7. Generate BCF output; -g 8. Specify reference sequence file. -f ref_seq Bcftools call: 1. Call genotypes at variant sites; -g 2. Variant sites only. -v

178 5- Variant calling Variant calling command: samtools mpileup -I -C 50 -Q 20 -q 40 -S -D -g -f cc_ref.fa cc_gn2_r12_rg_sorted_rmdup_sorted.bam bcftools view -gv - > cc_gn2_bq20_mq40.vcf

179 5- Variant calling gedit cc_gn2_bq20_mq40.vcf OR More/Cat/Head #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT shark cc_ref G A 140. DP=10;VDB= e- 02;AF1=1;AC1=2;DP4=0,0,6,1;MQ=47;FQ=-48 GT:PL:DP:SP:GQ 1/1:173,21,0:7:0:39

180 5- Variant calling #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT shark cc_ref G A 140. DP=10;VDB= e 02;AF1=1;AC1=2; DP4=0,0,6,1;MQ=47;FQ=-48 GT:PL:DP:SP:GQ 1/1:173,21,0:7:0:39

181 5- Variant calling #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT shark cc_ref G A 140. DP=10;VDB= e 02;AF1=1;AC1=2; DP4=0,0,6,1;MQ=47;FQ=-48 GT:PL:DP:SP:GQ 1/1:173,21,0:7:0:39

182 5- Variant calling #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT shark cc_ref G A 140. DP=10;VDB= e 02;AF1=1;AC1=2; DP4=0,0,6,1;MQ=47;FQ=-48 GT:PL:DP:SP:GQ 1/1:173,21,0:7:0:39 INFO: user extensible annotation (e.g. samtools and GATK may differ in this) INFO: DP=10;VDB= e 02;AF1=1;AC1=2;DP4=0,0,6,1;MQ=47;FQ=-48 DP: DPR: DP4: MQ: "Raw read depth" "Number of high-quality bases observed for each allele" "Number of high-quality ref-forward, ref-reverse, alt-forward and alt-reverse bases" "Average mapping quality"

183 5- Variant calling #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT shark cc_ref G A 140. DP=10;VDB= e 02;AF1=1;AC1=2; DP4=0,0,6,1;MQ=47;FQ=-48 GT:PL:DP:SP:GQ 1/1:173,21,0:7:0:39 INFO field regards the SITE and NOT the specific samples in multisample calling!!!

184 5- Variant calling #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT shark cc_ref G A 140. DP=10;VDB= e 02;AF1=1;AC1=2; DP4=0,0,6,1;MQ=47;FQ=-48 GT:PL:DP:SP:GQ 1/1:173,21,0:7:0:39

185 5- Variant calling #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT shark cc_ref G A 140. DP=10;VDB= e 02;AF1=1;AC1=2; DP4=0,0,6,1;MQ=47;FQ=-48 GT:PL:DP:SP:GQ 1/1:173,21,0:7:0:39 FORMAT: how the information for each sample is presented (i.e. GT:PL:DP:SP:GQ) Each sample (multisample calling) has its own FORMAT field, information are sample specific!! GT:PL:DP:SP:GQ

186 5- Variant calling FORMAT: how the information for each sample is presented (i.e. GT:PL:DP:SP:GQ) GT:PL:DP:SP:GQ GT: genotype 0/0: homozygote reference; 0/1: heterozygote; 1/1: homozygote alternative; 1/1:173,21,0:7:0:39 Homozygote alternative 0: reference allele; 1:alternative allele;

187 5- Variant calling FORMAT: how the information for each sample is presented (i.e. GT:PL:DP:SP:GQ) GT:PL:DP:SP:GQ PL: List of Phred-scaled genotype likelihoods; Approximate likelihood = 10^(-PL/10) 1/1:173,21,0:7:0:39 P(0/0) = 10^(-173/10) = 10^(-17.3) = P(0/1) = 10^(-21/10) = 10^(-2.1) = P(1/1) = 10^(-0/10) = 10^(0) = 1

188 5- Variant calling FORMAT: how the information for each sample is presented (i.e. GT:PL:DP:SP:GQ) GT:PL:DP:SP:GQ DP: Number of high-quality bases; 1/1:173,21,0:7:0:39 DP=7

189 5- Variant calling FORMAT: how the information for each sample is presented (i.e. GT:PL:DP:SP:GQ) GT:PL:DP:SP:GQ SP: Phred-scaled strand bias P-value; Strand bias p value= 10^(- SP /10) 0.05=10^(- SP /10) -10*Log 0.05 = SP SP =13.01 Would you keep sites with SP 13 or sites with SP 13? SP p value there is a sta s cal significant difference in the number of alleles coming from the two strands; SP p value there is NOT a statistical significant difference in the number of alleles coming from the two strands.

190 5- Variant calling FORMAT: how the information for each sample is presented (i.e. GT:PL:DP:SP:GQ) GT:PL:DP:SP:GQ SP: Phred-scaled strand bias P-value; Strand bias p value= 10^(- SP /10) 0.05=10^(- SP /10) -10*Log 0.05 = SP SP =13.01 If we assume a p value threshold of 0.05, we will keep all sites with SP 13 1/1:173,21,0:7:0:39 SP=0

191 5- Variant calling FORMAT: how the information for each sample is presented (i.e. GT:PL:DP:SP:GQ) GT:PL:DP:SP:GQ GQ: conditional genotype quality, encoded as a phred quality; GQ= 10log10 P(genotype call is wrong) P(genotype call is wrong) = 10^(-GQ/10) 1/1:173,21,0:7:0:39 P(genotype call is wrong) = 10^(-39/10)=10^-3.9=

192 5- Variant calling #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT shark cc_ref G A 140. DP=10;VDB= e 02;AF1=1;AC1=2; DP4=0,0,6,1;MQ=47;FQ=-48 GT:PL:DP:SP:GQ 1/1:173,21,0:7:0:39 FORMAT: how the information for each sample is presented (i.e. GT:PL:DP:SP:GQ) GT:PL:DP:SP:GQ GT: PL: DP: SP: DPR: Genotype; List of Phred-scaled genotype likelihoods; Number of high-quality bases; Phred-scaled strand bias P-value; Genotype quality.

193 5- Variant calling Useful references and resources: at-is-a-vcf-and-how-should-i-interpret-it

194 5- Variant calling We have our raw variants but now we need to refine our dataset how? Variant score recalibration; Variant filtering; Validation

195 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

196 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

197 5- Variant calling: Variant quality score recalibration Available in GATK. It aims at producing well-calibrated probabilities for the variants called. It develops a continuous, covarying estimate of the relationship between SNP call annotations ( e.g. MQ, QD ) and the probability that a SNP is a true genetic variant versus a sequencing or data processing artifact. It needs true sites to be trained (i.e. HapMap Phase 3 data, OMNI 2.5 M, etc ). We are not going to use it, because it needs big datasets (either many samples, or whole genome data) to work properly. You can find more information at

198 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

199 raw reads (.fastq) 1. Fastq quality control + trimming 2. alignment to a reference genome close reference? time limited? bwa distant reference? stampy Adapters? Low quality bases? aligned reads (.sam/.bam) 4. bam check visualization 3. bam refinement duplicate metrics (picard) flagstat (samtools) coverage distribution (GATK) samtools IGV/tablet local realignment base recalibration duplicate removal GATK GATK picard aligned reads (.sam/.bam) ready-to-use variants (.vcf) 5. variant calling SNPs/indels raw variants (.vcf) variant score recalibration 6. variant filtering and validation single/multi-sample known SNPs/indels vcftools samtools big datasets in silico vs in vitro validation

200 5- Variant filtering and validation WHY filtering??? Each caller tends to call as many sites it can but it provides useful information on several parameters; Many calls are guesses ; Artifacts; Presence of sequencing error; To create a high quality dataset; WHY validation??? An error-free dataset is unrealistic; To estimate the % of error within our high quality dataset;

201 5- Variant filtering and validation: FILTERING Filtering is all about finding the right balance:

202 5- Variant filtering and validation: FILTERING Filtering is all about finding the right balance: Stringent filters Enough information (sites)

203 5- Variant filtering and validation: FILTERING Less stringent filters Lots of sites but with many possible errors

204 5- Variant filtering and validation: FILTERING Not many sites left Very stringent filters

205 5- Variant filtering and validation: FILTERING General approach: Try different thresholds for each parameter and draw a distribution; If possible, look for sudden changes in the distribution and set a threshold; No fixed rule on which thresholds you should use; It is data and project specific; It is a crucial step to create a high quality dataset.

206 5- Variant filtering and validation: FILTERING Common cautions: - Base quality BQ20 - Depth (min and max) very dependent on your average - Mapping quality MQ40/50 (minimum MQ30) - Strand-bias p-value> SNP density dependent on the genome [e.g. no more than 1 SNP/4bp] - Indel proximity not closer than 10bp to an indel - Missing data? - Some filters may be applied during the variant calling while others are applied afterwards.

207 5- Variant filtering and validation: FILTERING MQ distribution

208 5- Variant filtering and validation: FILTERING MQ distribution

209 5- Variant filtering and validation: FILTERING Coverage: distribution per run

210 5- Variant filtering and validation: FILTERING Coverage: distribution per population

211 5- Variant filtering and validation: FILTERING VCFtools ( --min-meandp <float> --max-meandp <float> Includes only sites with mean depth values (over all included individuals) greater than or equal to the "--min-meandp" value and less than or equal to the "--maxmeandp" value. One of these options may be used without the other. These options require that the "DP" FORMAT tag is included for each site. GT:PL:DP:SP:GQ

212 5- Variant filtering and validation: FILTERING VCFtools ( Input file (vcf) vcftools --vcf cc_gn2_bq20_mq40.vcf DP threshold --min-meandp 20 --recode --recode-info-all Keep all the INFO --out cc_gn2_bq20_mq40_dp20.vcf Create new vcf file (recode in the file name) Output file INFO: DP=10;VDB= e 02;AF1=1;AC1=2;DP4=0,0,6,1;MQ=47;FQ=-48

213 5- Variant filtering and validation: FILTERING Gedit cc_gn2_bq20_mq40_dp20.vcf.recode.vcf OR Cat/more cc_gn2_bq20_mq40_dp20.vcf.recode.vcf

214 5- Variant filtering and validation: FILTERING cc_ref T G 182. DP=26;VDB= e- 02;RPB= e+00;AF1=0.5;AC1=1;DP4=5,6,5,5;MQ=48;FQ=184;PV4=1,1,1,1 GT:PL:DP:SP:GQ 0/1:212,0,217:21:0:99 cc_ref N C 222. DP=24;VDB= e- 01;AF1=1;AC1=2;DP4=0,0,13,10;MQ=48;FQ=-96 GT:PL:DP:SP:GQ 1/1:255,69,0:23:0:99 cc_ref G T 222. DP=24;VDB= e- 01;AF1=1;AC1=2;DP4=0,0,11,12;MQ=46;FQ=-96 GT:PL:DP:SP:GQ 1/1:255,69,0:23:0:99

215 5- Variant filtering and validation: FILTERING cc_ref T G 182. GT:PL:DP:SP:GQ 0/1:212,0,217:21:0:99 cc_ref N C 222. GT:PL:DP:SP:GQ 1/1:255,69,0:23:0:99 cc_ref G T 222. GT:PL:DP:SP:GQ 1/1:255,69,0:23:0:99 1,233,213 G/T familiar??? You discover it in tview!!!

216 5- Variant filtering and validation: FILTERING Visualisation of the filtered variant sites in IGV IGV needs: A genome: reference sequence; BAM files: sorted and indexed.

217 5- Variant filtering and validation: FILTERING Visualisation of the filtered variant sites in IGV./igv.sh Genomes > Load Genome > white_shark.genome Files > Load > cc_gn2_r12_rg_sorted_rmdup_sorted.bam

218 5- Variant filtering and validation: FILTERING Visualisation of the filtered variant sites in IGV cc_ref:39327

219 5- Variant filtering and validation: FILTERING Visualisation of the filtered variant sites in IGV cc_ref:381078

220 5- Variant filtering and validation: FILTERING Visualisation of the filtered variant sites in IGV cc_ref: