Next Generation Sequencing

Transcription

1 Next Generation Sequencing Cavan Reilly November 21, 2014

2 Table of contents Next generation sequencing NGS and microarrays Study design Quality assessment Burrows Wheeler transform BWT example Introduction to UNIX-like operating systems Installing programs Bowtie SAMtools RNA-Seq Aligning reads to the transcriptome TopHat Cufflinks RNA-Seq using TopHat and Cufflinks RNAseq and count models DNA-Seq BCFtools Variant Pattern Analyzer Genome Analysis Toolkit Microbiomics

3 Next generation sequencing Over the last 7 years or so there has been rapid development of methods for next generation sequencing. Here is the process for the Illumina technology (one of the 3 major producers of platforms for next generation sequencing). The biological sample (e.g. a sample of mrna molecules) is first randomly fragmented into short molecules. Then the ends of the fragments are adenylated and adaptor oligos are attached to the ends.

4 Next generation sequencing The fragments are then size selected, purified and put on a flow cell. An Illumina flow cell has 8 lanes and is covered with other oligonucleotides that bind to the adaptors that have been ligated to the fragmented nucleotide molecules from the sample. The bound fragments are then extended to make copies and these copies bind to the surface of the flow cell. This is continued until there are many copies of the original fragment resulting in a collection of hundreds of millions of clusters.

5

6

7 Next generation sequencing The reverse strands are then cleaved off and washed away and sequencing primer is hybridized to the bound DNA. The individual clusters are then sequenced in parallel, base by base, by hybridizing fluorescently labeled nucleotides. After each round of extension of the nucleotides a laser excites all of the clusters and a read is made of the base that was just added at each cluster. If a very short sequence is bound to the flow cell it is possible that the machine will sequence the adaptor sequence-this is referred to as adaptor contamination. There is also a measure of the quality of the read that is saved along with the read itself.

8 Next generation sequencing These quality measures are on the PHRED scale, so if there is an estimated probability of an error of p, the PHRED based score is 10 log 10 p. If we fragment someone s DNA we can then sequence the fragments, and if we can then put the fragments back together we can then get the sequence of that person s genome or transcriptome. This would allow us to determine what alleles this subject has at every locus that displays variation among humans (e.g. SNPs).

9 Next generation sequencing There are a number of popular algorithms for putting all of the fragments back together: BWA, Maq, SOAP, ELAND and Bowtie. We ll discuss Bowtie in some detail later (BWA uses the same ideas as Bowtie). ELAND is a proprietary algorithm from Illumina and Maq and SOAP use hash tables and are considerably slower than Bowtie.

10 Applications There are many applications of this basic idea: 1. resequencing (DNA-seq) 2. gene expression (RNA-seq) 3. mirna discovery and quantitation 4. DNA methylation studies 5. ChIP-seq studies 6. metagenomics 7. ultra-deep sequencing of viral genomes We will focus on resequencing (and SNP calling) and gene expression in this course.

11 NGS and microarrays Some have talked about the death of microarrays due to the use of NGS in the context of gene expression studies, but this hasn t happened yet. Compared to microarrays, RNA-seq 1. higher sensitivity 2. higher dynamic range 3. lower technical variation 4. don t need a sequenced genome (but it helps, a lot) 5. more information about different exon uses (more than 90% of human genes with more than 1 exon have alternative isoforms)

12 NGS and microarrays In one head to head comparison, 30% more genes are found to be differentially expressed with the same FDR (Marioni, Mason, Mane, et al. (2008)). These authors also found that the correlation between normalized intensity values from a microarray and the logarithm of the counts for the same transcript were (this is similar to what others have reported). The largest differences occurred when the read count was low and the microarray intensity was high which probably reflects cross-hybridization in the microarray.

13 NGS and microarrays Others have found slightly higher correlations, here is a figure from a study by Su, Li, Chen, et al. (2011).

14 Study design Before addressing the technical details, we will outline some considerations regarding study design specific to NGS technology. There are several major manufacturers of the technology necessary to generate short reads, and they all have different platforms with resulting differences in the data and processing methods. A few of the major vendors are 1. Illumina, formerly known as Solexa sequencing, they now have MiSeq and HiSeq (the Genome analyzer is the old platform) 2. Roche, which owns 454 Life Sciences, which supports GS FLX+ system 3. Life technologies, which supports Ion-Torrent

15 Study design We will focus on the Illumina technology as the University of Minnesota has invested in this platform (the HiSeq 2000 and MiSeq platforms is available in the UMN Genomics Center). The MiSeq machine produces longer reads than the HiSeq 2000 We also have some capabilities to do 454 sequencing here. The 454 platform produces longer reads (up to 1000 bp now), and is very popular in the microbiomics literature.

16 Study design By sequencing larger numbers of fragments one can estimate the sequence more accurately. The goal of using high sequence coverage is that by using more fragments it is more likely that one fragment will cover any random location in the genome. Clearly one wants to cover the entire genome if the goal is to sequence it, hence the depth of coverage is how many overlapping fragments will cover a random location. By deep sequencing, we mean coverage of 30X to 40X, whereas coverage of 4X or so would be considered low coverage.

17 Study design The rule for determining coverage is: coverage = (length of reads number of reads)/ (length of genome) For a given sequencing budget, there is a tradeoff between the depth of coverage and the number of subjects one can sequence. The objectives of the study should be considered when determining the depth of coverage. If the goal is to accurately sequence a tumor cell then deep coverage is appropriate, however if the goal is to identify rare variants, then lower coverage is acceptable.

18 Study design For example, the 1000 Genomes project is using about 4X coverage. A reasonable rule of thumb is that one probably needs 20-30X coverage to get false negatives less than 1% in nonrepeat regions of a duploid genome (Li, Ruan and Durbin, 2008). If the organism one is sequencing does not have a sequenced genome, the calculations are considerably more complex and one would want greater coverage in this case. Throughout this treatment we will suppose that the organism under investigation has a sequenced genome.

19 Study design Another choice when designing your study regards the use of mate-pairs. With the Illumina system one can sequence both ends of a fragment simultaneously. This greatly improves the accuracy of the method and is highly recommended. If the goal is to characterize copy number variations then getting paired end sequences is crucial.

20 Quality assessment We can use the ShortRead package in R to conduct quality assessment and get rid of data that doesn t meet quality criteria. To explore this we will use some data from dairy cows. You can find a description of the data along with the actual data in sra format at this site. While the NCBI uses the sra format to store data, the standard format for storing data that arises from a sequencing experiment is the fastq format. We can convert these files to fastq files using a number of tools-here is how to use one supported at NCBI called fastq-dump.

21 Quality assessment To use this you need to download the compressed file, unpack it and then enter the directory that holds the executable and issue the command (more on these steps later). sratoolkit centos linux64/bin/fastq-dump SRR sra Then this will generate fastq files which hold the data. If we open the fastq files we see they hold identifiers and information about the reads followed by the actual reads:

22 Quality HWI-EAS283:8:1:9:19 length=36 ACGAGCTGANGCGCCGCGGGAAGAGGGCCTACGGGG +SRR HWI-EAS283:8:1:9:19 length=36 HWI-EAS283:8:1:9:1628 length=36 CCTAATAATNTTTTCCTTTACCCTCTCCACATTAAT +SRR HWI-EAS283:8:1:9:1628 HWI-EAS283:8:1:9:198 length=36 ATTGAATTTGAGTGTAAACATTCACATAAAGAGAGA +SRR HWI-EAS283:8:1:9:198 length=36 The string of letters and symbols encodes quality information for each base in the sequence.

23 Quality assessment Then we define the directory where the data is and read in all of the filenames that are there > library(shortread) > datadir <- "/home/cavan/documents/ngs/bovinerna-seq" > fastqdir <- file.path(datadir, "fastq") > fls <- list.files(fastqdir, "fastq$", full=true) > names(fls) <- sub(".fastq", "", basename(fls))

24 Quality assessment Then we apply the quality assessment function, qa, to each of the files after we read it into R using the readfastq function > qas <- lapply(seq along(fls), function(i, fls) + qa(readfastq(fls[i]), names(fls)[i]), fls) This can take a long time as it has to read all of these files into R, but note that we do not attempt to read the contents of the files into the R session simultaneously (a couple of objects generated by the call to readfastq stored inside R will really bog it down). Then we rbind the quality assessments together and save to an external location. > qa <- do.call(rbind, qas) > save(qa, file=file.path("/home/cavan/documents/ngs/bovinerna-seq","qa.rda")) > report(qa, dest="/home/cavan/documents/ngs/bovinerna-seq/reportdir") [1] "/home/cavan/documents/ngs/bovinerna-seq/reportdir/index.html"

25 Quality assessment This generates a directory called reportdir and this directory will hold images and an html file that you can open with a browser to inspect the quality report. So we now use the browser to do this-we see the data from run 12 is of considerably higher quality than the others but nothing is too bad. One can implement soft trimming of the sequences to get rid of basecalls that are of poor quality. A function to do this is as follows (thanks to Jeremy Leipzig who posted this on his blog). Note that this can require a lot of computational resources for each fastq file.

26 Quality assessment # softtrim # trim first position lower than minquality and all subsequent positions # omit sequences that after trimming are shorter than minlength # left trim to firstbase, (1 implies no left trim) # input: ShortReadQ reads # integer minquality # integer firstbase # integer minlength # output: ShortReadQ trimmed reads library("shortread") softtrim <- function(reads, minquality, firstbase=1, minlength=5){ qualmat <- as(fastqquality(quality(quality(reads))), matrix ) quallist <-split(qualmat, row(qualmat)) ends <- as.integer(lapply(quallist, function(x){which(x < minquality)[1]-1})) # length=end-start+1, so set start to no more than length+1 to avoid # negative-length starts <-as.integer(lapply(ends,function(x){min(x+1,firstbase)})) # use whatever QualityScore subclass is sent newq <-ShortReadQ(sread=subseq(sread(reads),start=starts,end=ends), quality=new(class=class(quality(reads)), quality=subseq(quality(quality(reads)), start=starts,end=ends)),id=id(reads))

27 Quality assessment } # apply minlength using srfilter lengthcutoff <- srfilter(function(x) { width(x)>=minlength}, name="length cutoff") newq[lengthcutoff(newq)] To use: > library("shortread") > source("softtrimfunction.r") > reads <- readfastq("myreads.fq") > trimmedreads <- softtrim(reads=reads,minquality=5,firstbase=4,minlength=3) > writefastq(trimmedreads,file="trimmed.fq")

28 Bowtie background The keys to understanding how the algorithm implemented by Bowtie are the Burrows Wheeler transform (BWT) and the FM index. 1. the FM index: see Ferragina, P. and Manzini, G. (2000), Opportunistic data structures with applications 2. the Burrows Wheeler transform (BWT): see Burrows, M. and Wheeler, D.J. (1994), A Block-sorting lossless data compression algorithm. The Burrows Wheeler transform is an invertible transformation of a string (i.e. a sequence of letters).

29 Computing the BWT To compute this transform, one 1. determines all rotations of the string 2. sorts the rotations lexicographically to generate an array M 3. saves the last letter of each sorted string in addition to the row of M that corresponds to the original string.

30 Computing the BWT: an example Some descriptions of the algorithm introduce a special character (e.g. $) that is lexicographically smaller than all other letters, then append this letter to the string before starting the rest of the transform. This just guarantees that the first row of M is the original string so that one needn t store the row of M that is the original string. To see how the BWT works, let s consider the consensus sequence just upstream of the transcription start site in humans (from Zhang 1998): GCCACC We will use the modification that introduces the special character.

31 BWT example First add the special character to the end of the sequence we want to transform, denoted S with length N. GCCACC$ then get all of the rotations: $GCCACC C$GCCAC CC$GCCA ACC$GCC CACC$GC CCACC$G GCCACC$

32 BWT example then sort them (i.e. alphabetize these sequences) $GCCACC ACC$GCC C$GCCAC CACC$GC CC$GCCA CCACC$G GCCACC$ so the BWT is just the last column, typically denoted L, and we will refer to element i of this array as L[i] for i = 1,..., N and ranges of elements via L[1,..., k]. L=CCCCAG$

33 BWT example Importantly one can invert this transformation of the string to obtain the original string. This was first developed in the context of data compression, so the idea was to transform the data, then compress it (which can be done very efficiently after applying this transformation), then one can invert the transformation after uncompressing.

34 BWT example Suppose we have the BWT of a genome and want to count the number of occurrences of some pattern, e.g. AC, our query sequence. We can do this as follows. First, create a table that for each letter has the count of the number of occurrences of letters that are lexically smaller than that letter. For us this is just x $ A C G C(x)

35 BWT example Next we need to be able to determine the number of times that a letter x occurs in L[1,..., k] for any k, which we will denote O(x, k). Note that all of these calculations can be done once and stored for future use. Now search for the last letter of our query sequence, which is C, over the range of rows given by [C(x) + 1,..., C(x + 1)] where x is C so the range is [3,6]. Note that these are the rows of M that start with a C although we don t actually need to store all of M to determine this (it just follows from the alphabetization).

36 BWT example Then process the second to last letter, which for our example is A. To determine the range of rows of M to search we look at [C(x) + O(x, s 1 1) + 1,..., C(x) + O(x, s 2 )] where x is A and s 1 and s 2 are the endpoints of the interval from the previous step. Now and O(A, 3 1) = O(A, 2) = 0 O(A, 6) = 1 so the interval is [ ,..., 1 + 1] = [2, 2] then since this is the end of the query sequence we determine the number of occurrences by looking at the size of the range of this interval, which is here just 1.

37 BWT example If the range becomes empty or if the lower end exceeds the upper end the query sequence is not present. For example if we query on CCC and proceed as previously, the first step is the same (since this also ends in C), so then searching for C again we find [C(C) + O(C, s 1 1) + 1, C(x) + O(C, s 2 )] but O(C, 2) = 2 and O(C, 6) = 4 so the interval is

38 BWT example [ , 2 + 4] = [5, 6] which has size 2 (as there are 2 instances of CC in the sequence), so then proceed as in step 2 again to get which is [C(C) + O(C, 5 1) + 1, C(C) + O(C, 6)] [ , 2 + 4] = [7, 6] indicating that this sequence is not present.

39 BWT example For aligning short sequences to a genome we need to know where the short segments align, hence we need to determine the location of the start of the sequence in the genome. If the locations of some of the letters are known we can just trace back to those locations as follows. First, note that we know the first column of the array M. We know this due to 2 facts: 1. the first column will be strictly alphabetized 2. every letter in the sequence is represented in the last column L.

40 BWT example For our problem we have L=CCCCAG$ and so the first column is just F =$ACCCCG. As our sequence is of length N, then if we knew the permutation of 1,..., N that mapped F to L we would know the entire sequence (as we will see shortly). That one can determine this mapping is the key ingredient in computing the inverse BWT. Call this map T where we have F ( T (i) ) = L(i).

41 BWT example To find T, first consider M and a new array, call it M, that is obtained from M by taking the last element from each row and making it the first element but otherwise changing nothing. Here is M : C$GCCAC CACC$GC CC$GCCA CCACC$G ACC$GCC GCCACC$ $GCCACC

42 Inverting the BWT Note that every row of M has a corresponding row in M and like M M has all cyclic rotations of the original sequence S. Also note that the relative order of the rows with the same first letter are the same in both arrays: rows 3, 4, 5 and 6 in M become rows 1, 2, 3 and 4 in M. We can use this insight to determine the mapping from F to L, i.e. T. To determine T, we proceed sequentially as follows. L(1) = C and this is the first C in L so T (1) = i where F (i) is the first C in F so looking at F we see that i = 3.

43 BWT example Then continue with this thinking: T (2) = i and L(2) is the second C in L, but then looking at F we see that the second C in F occurs at position 4, so that T (2) = 4. Continuing in this fashion we find that T = (3, 4, 5, 6, 2, 7, 1).

44 BWT example We can then reconstruct the sequence in reverse order. We know the last element of the sequence is the last element of the first row of M which is the first element of L, which is here C. Then we proceed sequentially using the mapping T as follows S(N i) = L ( T i (1) ) where T i (x) = T i 1( T (x) ), i.e. repeatly apply the mapping T to itself.

45 BWT example so S(N 1) = L ( T (1) ) = L(3) = C S(N 2) = L ( T (3) ) = L(5) = A S(N 3) = L ( T (5) ) = L(2) = C S(N 4) = L ( T (2) ) = L(4) = C S(N 5) = L ( T (4) ) = L(6) = G If we were just trying to find the location of a particular match, like AC we could just stop at that point rather than recover the entire sequence.

46 FM index The FM index is a compressed version of L, C and O along with a compressed version of a set of indices with known locations in S. This allows one to trace back to a known location. So one could align short reads to a genome by first computing the BWT and compressing the results. Unfortunately many of the reads would not align due to sequencing errors and polymorphisms. Hence one needs to allow some discrepancies between the genome and the short reads. While this is adjustable, the default specifications for the program Bowtie are that there is a maximum of 2 mismatches in the first 28 bases and a maximum total quality score of 70 for mismatches on the PHRED scale.

47 Introduction to Linux In order to use the latest tools in bioinformatics you need access to a linux based operating system (or Mac OS X). So we will give some background on how to use a linux based system. The linux operating system is based on the UNIX operating system (by a guy named Linus Torvalds), but is freely available (UNIX was developed by a private company). There are a number of implementations of linux that go by different names: e.g. Ubuntu, Debian, Fedora, and OPENsuse.

48 Introduction to Linux Most beginners choose Ubuntu (or one of its offshoots, like Mint). I use Ubuntu or Fedora on my computers and centos is used on the server we will use for this course. You can install it for free on any PC. Such systems are based on a hierarchical structure for organizing files into directories. Much of the time we will use plain text files: these are files that have no formatting, unlike the files you use in a typical word processing program. To edit such files you need to use a text editor. I like vi and vim (which has capabilities to interact with large files).

49 Introduction to Linux The editor pico is simple to use and installed on the server we will use for this class. To use this you type pico followed by the file name-if the file doesn t exist it will create the file otherwise it will open up the file so that you can edit it. For example [user0001@boris ~]$ pico temp1 will open the file temp1 for editing. Commands at the bottom of the window tell you how to conduct various operations.

50 Introduction to Linux You can create subdirectories in your home directory by typing ~]$ mkdir NGS This would create a NGS subdirectory where you can put files related to your NGS project. Then to keep your microarray data separate you could do [user0001@boris ~]$ mkdir microarray To move around hierarchical directory structure you use the command cd. We enter the NGS directory with [user0001@boris ~]$ cd NGS

51 Introduction to Linux To go back to our home location (i.e. the directory you start from when you log on) you just type cd and if you want to go up one level in the hierarchy you type cd... To list all of the files and directories in your current location you type ls. To copy a file you use cp, to move a file from one location to another you use mv and to remove a file you use rm. One can modify the behavior of these commands by using options.

52 Introduction to Linux For example, if I want to see the size of the files (and some additional information) rather than just list them I could issue the command [user0001@boris ~]$ ls -l Other useful commands include more and less for viewing the contents of files and man for information about a command (from the manual). For instance [user0001@boris ~]$ man ls Will open a help file with information about the options that are available for the ls command.

53 Introduction to Linux Wild cards are characters that operating systems use to hold a variable value. These are extremely useful as it lets the user specify many tasks without having to do a bunch of repetitive typing. For example, suppose you had a bunch of files that ended with.txt and you wanted to delete them. You could issue commands like [user0001@boris ~]$ rm file1.txt [user0001@boris ~]$ rm file2.txt [user0001@boris ~]$ rm file3.txt... Or you could just use [user0001@boris ~]$ rm file*txt or even [user0001@boris ~]$ rm *txt

54 Introduction to Linux Most linux configurations will check that you really want to remove the files, but if you are removing many files, the process of typing yes repeatly is unnecessary. Just type ~]$ rm -f *txt When you obtain programs from the internet that run on linux systems these programs will typically install many subdirectories, so if you want to get rid of all the contents of a directory, first enter that directory, [user0001@boris ~]$ cd olddir

55 Introduction to Linux then issue the command olddir]$ rm -rf * then go back up one level [user0001@boris olddir]$ cd.. and get rid of the directory using the rmdir command [user0001@boris ~]$ rmdir olddir If you issue a command to run a program but wish you hadn t and want it to stop you push the Ctrl and C keys simultaneously.

56 Introduction to Linux To connect to a linux server from a Windows machine you need a Windows program that allows you to log on to a terminal and the ability to transfer files to and from the remote server and your Windows machine. WinSCP is a free program that allows one to do this. You can download the data from the WinSCP website and it installs like a regular Windows program: just allow your computer to install it and go with the default configuration. To get a terminal window you can use a program called PuTTY. So download the program PuTTY (which is just the file called PuTTY.exe) and create a subdirectory in your Program Files directory called PuTTY.

57 Introduction to Linux Then save PuTTY.exe in the directory you just created. Once installed start WinSCP and use boris.biostat.umn.edu as the hostname along with your user name and password. You can then get a terminal window using putty: just click on the box that shows 2 computers that are connected right below the session tab. Then disconnect when you are done (using the option in the session tab).

58 Installation of Bowtie Recall: Bowtie is a program that aligns short reads to an indexed genome. Bowtie2 is a more up to date version that allows gapped alignments and Smith Waterman type scoring, Bowtie just allows a couple of nucleotides to disagree between the read and the genome. To use Bowtie we need a set of short reads (in fastq format) and the index of a genome and of course the program itself. We will download a compressed version of the program (an executable) and after we uncompress it, we can use the executable, i.e. a program that does something. Bowtie has the capability to build indexes if we have the genome sequence.

59 Installation of Bowtie First set up a directory to hold all of this work [user0001@boris ~]$ mkdir RNAseq then create a directory to hold all of the programs we will download and set up, we will call it bin (for binaries). [user0001@boris ~]$ mkdir bin Then enter this directory so we can install some programs. [user0001@boris ~]$ cd bin

60 Installation of Bowtie In order to know what version of software you need to install you need to check the architecture of the computer. To do so we issue the following command bin]$ uname -m x86 64 So this is a 64 bit machine (32 bit machines will report something like i386). So get the appropriate version of Bowtie using the wget command [user0001@boris bin]$ wget 64.zip

61 Installation of Bowtie then unpack it bin]$ unzip bowtie linux-x86 64.zip Archive: bowtie linux-x86 64.zip creating: bowtie / creating: bowtie /scripts/ inflating: bowtie /scripts/build test.sh inflating: bowtie /scripts/make a thaliana tair.sh... This will generate a subdirectory with the name bowtie , so enter that subdirectory so that you can have access to the executable called bowtie in this subdirectory.

62 Aligning reads to an indexed genome The program ships with some data from the bacteriophage Enterobacteria phage lambda, so we will check the installation with this. We will first use a fasta file that ships with Bowtie to construct the index files for this organism. To use the program, you type the name of the executable, then give the location and name of the fasta file, then give the index name. So to test the installation type the following [user0001@boris bowtie ]$./bowtie2-build example/reference/lambda virus.fa lambda virus

63 Aligning reads to an indexed genome Then you get a bunch of output written to the screen about the index creation process and it generates 6 new files: lambda virus.1.bt2, lambda virus.2.bt2, lambda virus.3.bt2, lambda virus.4.bt2, lambda virus.rev.1.bt2 and lambda virus.rev.2.bt2. After that we are ready to align reads to the reference genome using the index we constructed. Here the syntax is to give the name of the index after -x, the files with the reads after -U and here we specify that we want the output in the sam file format using -S.

64 Aligning reads to an indexed genome bowtie ]$./bowtie2 -x lambda virus -U example/reads/reads 1.fq -S eg1.sam reads; of these: (100.00%) were unpaired; of these: 596 (5.96%) aligned 0 times 9404 (94.04%) aligned exactly 1 time 0 (0.00%) aligned >1 times 94.04% overall alignment rate We can also look at the output. [user0001@boris bowtie ]$ head eg1.sam

65 Aligning reads to an indexed genome Illumina stores prebuilt indices for many commonly studied organisms at the location software/igenome.ilmn If one has a set of index files one can use the program bowtie2-inspect to recreate the fasta files. The Illumina site has the fasta files for these organisms in the same location.

66 Altering your $PATH Note that we need to be in the same directory level as our executable in order to use that executable thus far-this is not necessary. To have access to an executable anywhere in your system you should alter your PATH variable in the linux environment to include where you keep your executables then write the executables to your path.

67 Altering your $PATH If you look at your path variable you will see that the bin directory we created is already on the path, so we can just copy executables to that location to use them. Here is how to examine the PATH variable. bowtie ]$ echo $PATH /usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin: /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin: /sbin:/export/home/courses/ph7445/user0001/bin

68 Copying executables to bin Many NGS programs use bowtie2, and for this to work they need to find the program in their path. So here we copy all of the executables to the bin directory: bowtie ]$ cp bowtie2 $HOME/bin/bowtie2 bowtie ]$ cp bowtie2-align-l $HOME/bin/bowtie2-align-l bowtie ]$ cp bowtie2-align-s $HOME/bin/bowtie2-align-s bowtie ]$ cp bowtie2-build $HOME/bin/bowtie2-build bowtie ]$ cp bowtie2-build-l $HOME/bin/bowtie2-build-l

69 Copying executables to bin and copy the rest too. bowtie ]$ cp bowtie2-build-s $HOME/bin/bowtie2-build-s bowtie ]$ cp bowtie2-inspect $HOME/bin/bowtie2-inspect bowtie ]$ cp bowtie2-inspect-l $HOME/bin/bowtie2-inspect-l bowtie ]$ cp bowtie2-inspect-s $HOME/bin/bowtie2-inspect-s

70 Altering your $PATH Bowtie 2 uses environmental variables to find the executables-just set these as follows. [user0001@boris bowtie ]$ export BT2 HOME=/export/home/courses/ph7445/user0001/bin/bowtie and check this with [user0001@boris bowtie ]$ echo $BT2 HOME

71 Altering environmental variables To test that this works let s go back to our home directory and issue our Bowtie2 commands again (note that we need to tell Bowtie where to find the fastq and index files) [user0001@boris ]$ bowtie2-build $BT2 HOME/example/reference/lambda virus.fa lambda virus [user0001@boris ]$ bowtie2 -x lambda virus -U $BT2 HOME/example/reads/reads 1.fq -S eg1.sam and it works fine.

72 Producing SAM files As we have seen, Bowtie2 can also produce a special file type that is widely used in the NGS world, a SAM file. If we open our eg1.sam file we see that the first 3 rows are the header (since they start Then there is a row for each read and 11 mandatory tab delimited fields per line that provide information about the alignment, for example the 4th column gives the location where the read maps. For the full specification consult samtools.sourceforge.net/samv1.pdf.

73 SAMtools The SAM (for sequence alignment/map) file type has a very strict format. These files can be processed in a highly efficient manner using a suite of programs called SAMtools. Here is how to setup these tools Download the latest version in your bin directory samtools-1.1.tar.bz2. [user0001@boris bin]$ wget

74 SAMtools then unpack bin]$ tar -jxvf samtools-1.1.tar.bz2 then there is a directory full of C programs called samtools-1.1, so go in there and type [user0001@boris samtools-1.1]$ make and write the executable to your path [user0001@boris samtools-1.1]$ cp samtools $HOME/bin/samtools

75 SAMtools example To use this for variant calling, we need an example with at least 2 subjects, so let s use the example that ships with SAMtools. We will return to this example latter, but for now we will just take a look at the pileup which represents the extent of coverage. Here is how to use SAMtools to index a FASTA file: [user0001@boris samtools-1.1]$ samtools faidx examples/ex1.fa

76 BAM and SAM files then create a BAM file from the SAM file (BAM files are binary, so they are compressed but not readable by eye) [user0001@boris samtools-1.1]$ samtools import examples/ex1.fa.fai examples/ex1.sam.gz ex1.bam should get [sam header read2] 2 sequences loaded

77 BAM and SAM files then index the BAM file samtools-1.1]$ samtools index ex1.bam then we can view the alignment to visually examine the extent of coverage and the depth of coverage samtools-1.1]$ samtools tview ex1.bam examples/ex1.fa and hit q to leave that visualization.

78 Visualization of the pileup CACTAGTGGCTCATTGTAAATGTGTGGTTTAACTCGTCCATGGCCCAGCATTAGGGAGCTGTGGACCCTGCAGCCTGGCT A...C C T.T.T..T...T..TC C T......,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,......T T

79 RNA-Seq As was mentioned previously, we can use next generation sequencing to generate sequences from samples of RNA. The transcriptome is the collection of all transcripts that can be generated from a genome. Current estimates indicate that 92-94% of human transcripts with more than one exon have alternatively spliced isoforms (Wang, Sandberg, Luo, et al. Nature, 2008). This means that the same gene can give rise to multiple mrna molecules. Some will differ because they have have different exons assembled into an mrna, but other possibilities include differences in use of the transcription start site (TSS), and differences in the 5 and 3 untranslated regions (UTRs).

80 Aligning reads to the transcriptome Differences in exon usage will result in mrnas from the same gene leading to different proteins, hence we speak of differential coding DNA sequences (CDS). One of the differences that arises due to differential 3 UTR usage is alternative polyadenelation patterns. These differences in 5 and 3 UTR usage lead to mrna molecules that have differential affinity for proteins that bind to the mature mrna molecule and we are just starting to understand how biological processes are regulated by these subtle mechanisms.

81 RNA-Seq Note that there is a difference in aligning a set of reads from an RNA sample and a DNA sample: the RNA molecules will have undergone removal of introns hence they won t align directly to the genome. If there is an annotated transcriptome for an organism you should start out trying to use this to analyze your data. However most investigations find evidence for transcripts that are not currently annotated, even for well studied organisms like mice and humans.

82 Duplicates One of the differences between using next generation sequencing for resequencing and profiling RNA levels is that with libraries prepared from RNA, one often finds duplicate reads. In fact it is not unusual to find that 60-80% of the reads are duplicates.

83 Duplicates In a resequencing study the chances of finding duplicates is quite low: for example, the human haploid genome is about 3 billion bps so the probability of observing a duplicate is about 1 in 3 billion, so with million reads, duplicate reads seem unlikely. Duplicate reads can arise as a PCR artifact, hence in resequencing studies duplicate reads should probably be treated as artifacts and removed. There is a SAMtools function called rmdup that can be used to do this, but read the manual on its use: for example, it is designed for paired end reads by default.

84 Library complexity However, when we consider RNA-Seq libraries, duplicates might arise just due to low complexity of our sample. By low complexity we mean there are not many distinct RNA molecules. The median length of a human transcript is 2186 nucleotides (according to the UCSC hg19 annotation), so if one has a library with only 1 type of transcript and generates 20 million reads from it one is guaranteed to find duplicates. So for analysis of RNA-Seq experiments it is probably unwise to remove duplicates.

85 Alignment algorithms There are a number of algorithms that are designed to align a set of short reads to the transcriptome, here is a short non-exhaustive list. 1. TopHat 2. GSNAP 3. MapSplice 4. SpliceMap 5. HMMSplicer However it is not clear at this point if any of these really dominate the others in terms of performance. However, TopHat is perhaps the most popular currently, and as it builds off of Bowtie, we will use this program in this course.

86 TopHat The basic idea behind TopHat is that if 2 segments of a read map to different locations on the genome then they are likely mapping to 2 different exons. To use TopHat you need fastq files for all of our samples and the genome sequence for the organism. Note that sometimes the data for a single sample will be in more than one fastq file. While we can use Bowtie to align short reads to a genome, reads that span an intron, or genes that span a splice junction, will not align directly to the genome.

87 TopHat If a read contains an internal segment that should map to an exon, Bowtie will fail to map this read to its correct genomic location as it won t match the genome outside of this exon. For this reason, Tophat initially splits the reads up into 25 base units, does the read mapping, then reassembles reads. Note: if you are using a set of reads that are shorter than 50 bases you should instruct TopHat to break the reads into fragments that are about half the length of your reads. For example, if you have a set of Illumina reads that are 36 bases you should instruct TopHat to break them into fragments of length 18 (using the --segment-length options, like --segment-length 18), otherwise it may not detect any splice junctions.

88 TopHat TopHat starts out by mapping the reads to the genome using Bowtie and sets aside those reads that don t map. It then constructs a set of disjoint covered regions (it no longer uses the program Maq as it did at the time of the publication): these are taken to be the initial set of exons observed in that sample. The program determines a consensus sequence using the reads, however in low coverage regions it will use the reference genome to determine the sequence. Also, as the ends of exons are likely only mapped by reads that span the splice junction, the disjoint regions covered by reads are expanded (45 bases by default) using the reference sequence.

89 TopHat For genes that are transcribed at low levels there could be gaps in the exons after the assembly step, hence exons that are very close are merged into a single exon. Introns of length less than 70 bases are rare in mammalian genomes, hence by default we combine exons that are separated by less than this many bases (this can be adjusted). The resulting set of putative exons (with their extensions from the reference genome) we will call islands. The 5 end of an intron is called a donor site and the 3 end is called the acceptor site.

90 TopHat This donor site almost always has the 2 bases GU at the end while the acceptor site is almost always AG, but these are sometimes GC-AG and AU-AC so current versions of TopHat also look for these acceptor and donor sites. TopHat uses these sequence characteristics to find potential donor and acceptor site pairs in the island sequences (not just adjacent ones, but nearby ones to allow for alternative splicing). It then looks at the set of reads that were initially unmapped and examines if any of these map to the sequence that spans the splice junction. To conduct this mapping TopHat uses the fact that most introns are between 70 and 500,000 bases (the maximum differs from the publication): these intron sizes are based on mammals, so the user should specify values for these if using a different organism.

91 TopHat It also requires that at least one read span at least 8 bases on each side of the splice junction by default, but this value can be adjusted. It also only uses the first 28 bases from the 5 end of the read to ensure that the read is of high quality, but this too can be adjusted. Finally the program reports a splice junction only if it occurs in more than 15% of the depth of coverage of the flanking exons (the 15% value comes from the Wang et al. 2008, Nature paper), but this is also adjustable as this is based on human data. If one supplies a transcriptome to TopHat via a GTF file, the program will first map reads to the transcriptome then try to identify splice junctions for the unmapped reads.

92 TopHat One can disable the subsequent search for splice junctions by specifying the -T option if you have provided a transcriptome. If you are going to use an annotated transcriptome then TopHat will need to construct an index for this and if you are processing multiple samples TopHat will redo exactly the same calculation for each. To save time, you should have TopHat do these calculations for the first sample and save the results. You can do this using the --transcriptome-index option. This option expects a directory and a name to use in this directory.

93 TopHat The exact syntax is like this: note that the --transcriptome-index option must be used right after the -G option. tophat -o out sample1 -G known genes.gtf \ --transcriptome-index=transcriptome data/known \ hg19 sample1 1.fq.z Then for processing the other samples we just tell TopHat where these indices are and don t specify the -G option tophat -o out sample2 \ --transcriptome-index=transcriptome data/known \ hg19 sample2 1.fq.z

94 TopHat To use the program first get the latest version bin]$ wget x86 64.tar.gz Then uncompress the binaries, enter the directory that is generated and copy the executables to your path. bin]$ tar zxvf tophat linux x86 64.tar.gz tophat linux x86 64]$ cp tophat2 $HOME/bin/tophat2 tophat linux x86 64]$ cp * $HOME/bin/

95 TopHat Then you should test the installation, so get some test data. tophat linux x86 64]$ wget data.tar.gz Then uncompress, go into that directory and run a test [user0001@boris tophat linux x86 64]$ tar zxvf test data.tar.gz [user0001@boris tophat linux x86 64]$ cd test data Then test by issuing the command (the -r 20 option indicates that we think there are 20 bases between read pairs) [user0001@boris test data]$ tophat2 -r 20 test ref reads 1.fq reads 2.fq

96 TopHat and if it is working correctly this yields the following [ :54:03] Beginning TopHat run (v2.0.13) [ :54:03] Checking for Bowtie Bowtie version: [ :54:03] Checking for Bowtie index files (genome).. Found both Bowtie1 and Bowtie2 indexes [ :54:08] Run complete: 00:00:05 elapsed

97 Cufflinks While TopHat is useful for aligning a set of reads to the transcriptome, one must use some other program to generate transcript level summaries of gene expression and test for differences between groups. One such program that is supported by the same group of researchers that develop TopHat is Cufflinks. Cufflinks uses the output from TopHat and we will see that one can use functionality of the R package cummerbund to examine the output and generate reports and publication quality graphs. It is designed for paired end reads but can be used for unpaired reads.

98 Cufflinks Cufflinks uses an expression for the likelihood of a set of paired end reads that depends on the length of the fragments: with paired end reads one can estimate the distribution of the lengths. Note that there is a fundamental identifiability problem with transcript assembly that is only mitigated by the presence of reads that span splice junctions. Suppose there are 2 exons and some reads map to both exons but we are unable to identify any reads that span splice junctions. If there are 2 exons there are 3 possible transcripts (2 with one of each exon and a third that uses both) so if we allow for all 3 transcripts there are an infinite collection of combinations of the 3 transcript levels that are consistent with observed read counts that hit both exons.

99 Cufflinks For this reason one must enforce a constraint on the transcripts: for example, we attempt to find the minimal set of transcripts that explain the exon counts. Cufflinks quantifies the level of expression of a transcript via the number of fragments per kilobase of exon per million fragments mapped (FPKM). This attempts to account for the fact that longer transcripts will generate more reads than shorter transcripts due to their length. Cuffdiff is a program that is bundled with Cufflinks that allows one to test for differences between experimental conditions.

100 Cufflinks The log of the ratio of FPKM values are approximately normal random variables: this allows one to define a test statistic once one determines an approximation to the standard error of this log ratio. This standard error depends on the number of isoforms for the transcript: if there is a single isoform there is no uncertainty in the expression estimate produced by the algorithm and the only source of variability is from biological replicates. If there are no replicates within conditions the standard error is computed by examining the variance across the 2 conditions. If there are not many transcripts that differ in their level of expression this will provide a reasonable standard error, but if there are replicates one can obtain better estimates of the variance of the log ratio by using the negative binomial distribution.

101 Cufflinks If there is only a single isoform then an approach similar to that used in the R package DESeq is used, but if there are multiple isoforms then uncertainty in the isoform expression estimates must be taken into account: this is done using a generalization of the negative binomial distribution called the beta negative binomial distribution. Some useful details: While both BAM and SAM files can be used for input the program requires that if chromosome names are present in records in the header of the file then the reads must be sorted in the same order.

102 RNA-Seq using TopHat and Cufflinks We ve already installed and tested the latest version of TopHat, so now let s install Cufflinks using what is now the familiar strategy: download the tarball, unpack it and write the executables to our path by copying them to bin. [user0001@boris bin]$ wget x86 64.tar.gz [user0001@boris bin]$ tar zxvf cufflinks linux x86 64.tar.gz [user0001@boris bin]$ cd cufflinks linux x86 64 [user0001@boris cufflinks linux x86 64]$ cp * $HOME/bin/

103 RNA-Seq using TopHat and Cufflinks Then test the installation using some test data, so download and run. cufflinks linux x86 64]$ wget data.sam cufflinks linux x86 64]$ cufflinks test data.sam and you should see

104 RNA-Seq using TopHat and Cufflinks You are using Cufflinks v2.2.1, which is the most recent release. [bam header read] EOF marker is absent. [bam header read] invalid BAM binary header (this is not a BAM file). File test data.sam doesn t appear to be a valid BAM file, trying SAM [11:36:11] Assembling transcripts and estimating abundances. > Processed 1 loci. [*************************] 100% Which indicates that this is working properly. This generates a GTF file called transcripts.gtf which has information on transcripts that were identified and their level of expression.

105 RNA-Seq using TopHat and Cufflinks As an example of using the TopHat/Cufflinks programs we will analyze the data that was used to illustrate the protocol developed by the authors of the programs (this is from the Nature Protocols paper from March 2012). The data is already loaded on the server and we ve installed all of the software, so we are almost ready to go. We also need a set of Bowtie2 indexes and if we want to use an annotated transcriptome we need a GTF file with that information.

106 RNA-Seq using TopHat and Cufflinks These have already been set up in the subdirectory /export/home/courses/ph7445/data The indexes are in the files with names that start with genome located at /export/home/courses/ph7445/data/drosophila melanogaster/ensembl/bdgp5/sequence/bowtie2index (there are 6 of these and I got them from the Illumina website). The file Drosophila melanogaster.bdgp5.68.gtf was obtained from ensembl.org: ftp://ftp.ensembl.org/pub/release-73/gtf/drosophilia melanogaster.

107 RNA-Seq using TopHat and Cufflinks First we must use TopHat to map the reads in the fastq files to the transcriptome. Note that due to all of the options we are going to use we need to break the command up into multiple lines: we do this by ending each line with a back-slash. [user0001@boris RNAseq]$ tophat2 -p 2 -G \ /export/home/courses/ph7445/data/drosophila melanogaster.bdgp5.68.gtf \ -o C1 R1 thout \ /export/home/courses/ph7445/data/drosophila melanogaster/ensembl/bdgp5/sequence/bowtie2index/genome \ /export/home/courses/ph7445/data/gsm C1 R1 1.fq \ /export/home/courses/ph7445/data/gsm C1 R1 2.fq then you should see something like this