Genome Assembly. 2 Sept. Groups. Wiki. Job files Read cleaning Other cleaning Genome Assembly

Transcription

1 2 Sept Groups Group 5 was down to 3 people so I merged it into the other groups Group 1 is now 6 people anyone want to change? The initial drafter is not the official leader use any management structure you like Wiki Use the wiki as your group notebook Share your job files Need to see your results Job files Read cleaning Other cleaning Genome Assembly

2 RCAC Job files Why? When you log into an RCAC server you are using a special server designed for multiple users. This is called a frontend node ( or sometimes a head node). There are (I think) three front end nodes often they are very busy. Frontend node: edit files, send mail, backup data, compile programs No computing The other nodes are called compute nodes. They are allocated and run by a system called PBS/Torque. The preferred way to use PBS is by submitting a job file using the command qsub When you run a job with qsub, all of the normal output (STDOUT) and error output (STDERR) is sent to files called jobname.o<jobnumber> and jobname.e<jobnumber>, respectively. For example check_clip_man.o check_clip_man.e

3 RCAC Job files Example: Running seqyclean (a module) #!/bin/sh -l #PBS -N seqyclean_monpu1 #PBS -q scholar #PBS -l nodes=1:ppn=16 #PBS -l walltime=168:00:00 module load seqyclean cd $PBS_O_WORKDIR pwd Shebang tell unix this is a shell file. It could be a Perl file Jobname (seen in qstat) Queue use scholar unless otherwise instructed Number of nodes and CPUs (ppn) to reserve. Usually ppn will be 1 or 16 on scholar Maximum CPU time the job will run. The scholar queue is limited to 168 hours cat seqyclean.job date +"%d %B %Y %H:%M:%S" echo " " seqyclean -t 16 \ -1../../data/Monpu1.genome.rawReads.r1.fq \ -2../../data/Monpu1.genome.rawReads.r2.fq \ -v adapter.fa \ -qual \ -minimum_read_length 30 \ -o Monpu1.genome.rawReads.seqyclean.stats \ > seqyclean.log Optional: pwd $PBS_O_WORKDIR is a predefined symbol that means the directory from which you submitted the job with qsub Pwd print the directory after the cd useful for debugging cat <filename> copies the command file to the output The date echo command writes the date into the output The backslash, \, is a line continuation character in unix. It makes it easier to write and understand very long command lines The greater-than symbol, >, redirects output in unix, i.e., everything written to STDOUT is sent to the file seqyclean.log echo " " date +"%d %B %Y %H:%M:%S"

4 RCAC Job files #PBS -N seqyclean_monpu1 #PBS -q scholar #PBS -l nodes=1:ppn=16 #PBS -l walltime=168:00:00 PBS commands can also be entered on the command line when you run qsub qsub N seqyclean_monpu1 -q scholar -l nodes=1:ppn=16 -l walltime=168:00:00 I like the PBS commands in the job file so I have a record Its easy to make a mistake Can save a lot of work by copying from old jobs

5 RCAC job files Job files # (header stuff removed for example) ~/src/btrim/btrim64 \ -3 \ -p adapter2.fa \ -t../monpu1.genome.rawreads.fastq \ -o Monpu1.trimmed \ -s Monpu1.btrim.summary \ >btrim.log This file is in /home/mgribsko/src. In unix, ~ is a symbol for your home directory. ~<username>, for instance ~mgribsko is a symbol for the named user s home directory echo " " date +"%d %B %Y %H:%M:%S" # Btrim64: -q -p <pattern file> -t <fastq file> -o <trim file> [-u 5'-error -v 3'-error -l minlen -b <5'-cut> -e <3'-cut> \ # -w <window> -a <average> -f <5'-trim> -I] # # Required for pattern trimming: # -p <pattern file> each line contains one pair of 5'- and 3'-adaptors; ignored if -q in effect # -t <sequence file> fastq file to be trimmed # -o <output file> fastq file of trimmed sequences # # Required for quality trimming (-q in effect): # -t <sequence file> fastq file to be trimmed # -o <output file> fastq file of trimmed sequences # # Optional: # -q toggle to quality trimming [default=adaptor trimming] # -3 3'-adaptor trimming only [default=off] # -P pass if no adaptor is found [default=off] # -Q do a quality trimming even if adaptor is found [default=off] # -s <summary file> detailed trimming info for each sequence # -u <5'-error> maximum number of errors in 5'-adaptor [default=3] # -v <3'-error> maximum number of errors in 3'-adaptor [default=4] # -l <minimal length> minimal insert size [default=25] # -b <5'-range> the length of sequence to look for 5'-adaptor at the beginning of the sequence [default=1.3 x adaptor length] # I often copy the help for the command into the job file as a comment. Comments begin with #. This makes it much easier to change the command later. Notice that the PBS commands are comments as far as unix is concerned

6 RCAC Job files Time for an Example Job files Grep tricks

7 Adapter trimming Over the summer I tried many methods AdapterRemoval AlienTrimmer Btrim Cutadapt Fastx_clip Fastqmcf Flexbar Reaper Scythe Seqprep Seqyclean Skewer Trimmomatic

8 Adapter trimming Quick and Dirty test: use grep to check for the first 14 bases of the universal and index adapters, and their reverse complement Why 14? Long enough that you don t expect to see (many) matches by chance. Why quick and dirty? Only exact matches will be found Quality not considered Matches may be cut off by end of read This test will UNDERESTIMATE the number of adapters.

9 Adapter trimming index Forward Index Reverse Universal Forward Universal Reverse Total Adapters reads remain adapters remain Monpu1.genome.rawReads.r1.fq Monpu1.genome.rawReads.r2.fq Monpu1.genome.rawReads.both.fq % % Monpu1.genome.filteredReads.fastq % 34.11% adapterremoval % 3.16% alientrimmer % 5.46% cutadapt % 65.96% fastqmcf % 17.39% flexbar % 1.72% reaper % 2.66% scythe % 4.21% seqprep % 3.26% skewer % 2.14% seqyclean all % 0.11% trimmomatic paired.r1.fq trimmomatic unpaired.r1.fq trimmomatic paired.r2.fq trimmomatic unpaired.r2.fq trimmomatic all % 20.45%

10 Adapter trimming Group 1- trimmomatic

11 Adapter trimming

12 Adapter trimming

13 Adapter trimming

14 Adapter trimming

15 Adapter trimming

16 Adapter trimming

17 Other Cleaning Mitochondrial Phi-X174 Match to reads using Bowtie2 (or any other mapper) use local-very-sensitive (matches with small gaps)

18 De Bruijn Graphs (from Homolog.us Bioinformatics)

19 De Bruijn Graph

20 De Bruijn Graph Repeats

21 De Bruijn Graph reads

22 Velvet One of the first De Bruijn assemblers Pruning tips a chain of nodes disconnected on one end caused by sequencing errors OR coverage gaps errors tend to be short (rule trim if < 2 kmer ) errors tend to have low multiplicity at junction bubbles paths that leave and return caused by sequence variation (SNPs) length/multiplicity rule shorter, higher multiplicity paths are preferred Erroneous connections duplicate sequences + errors errors will have low coverage, so will areas with low coverage