Practical Sessions Metagenomics course

Size: px

Start display at page:

Download "Practical Sessions Metagenomics course"

Grace Powers
6 years ago
Views:

1 Practical Sessions Metagenomics course Aitor Gonzaga, Ana-Belen Martin- Cuadrado, Carolina Mizuno, Inmaculada Garcia-Heredia, Mario López-Pérez, Nikole Kimes, Rohit Ghai

Program 9-12 th December Session 1: Introduction to UNIX (Carolina Mizuno and Rohit Ghai) Session 2: Genome assembly and annotation (Mario Lopez-Perez and Aitor Gonzaga)

2 Program 9-12 th December Session 1: Introduction to UNIX (Carolina Mizuno and Rohit Ghai) Session 2: Genome assembly and annotation (Mario Lopez-Perez and Aitor Gonzaga) Session 3: Metagenomic reads (Ana-Belen Martin-Cuadrado and Inmaculada Garcia-Heredia) Session 4: Assembling metagenomes (Rohit Ghai and Carolina Mizuno) Session 5: Sampling Cruise

3 Practical Session 1 December 9 th Introduction to UNIX Carolina Mizuno, Rohit Ghai Mario López-Pérez, Aitor Gonzaga, Ana-Belen Martin-Cuadrado, Inmaculada Garcia-Heredia, Nikole Kimes

4 The Third Generation Microbiology Genomics and Metagenomics (1995-) Introduction to UNIX DNA SEQUENCING ANALYSIS BIOLOGICAL MEANING (pure culture, SAG, environmental, enrichments,...) (Illumina, 454, sanger, pacbio, Ion Torrent, ) (assembly, annotation, recruitment, )

5 Computer power SEQUENCING Illumina 15 Gb data 200 million reads Storage Processing Analysis OUR STRUCTURE: Server 1: 48-Cores, 400 GB RAM Server 2: 48-Cores, 256 GB RAM Server 3: 24-Cores, 128 GB RAM Server 4: 16-Cores, 96 GB RAM Storage systems: 3 storage servers (storage capacity of 100 TB)

6 Tools READS CONTIGS? COMPLETE GENOMES

7 Why UNIX? Introduction to UNIX Many of the most useful programs are designed in and for UNIX. What is UNIX? UNIX is an Operating System (OS). The most popular operating system: Windows Unix is a term often used for a class of Operating Systems: Solaris, Linux, Mac OS. Linux is Unix re-coded under an open-source license.

8 What are we going to learn today? Connect to the servers Basic Linux commands to be able to: manage files (copy, move,..) simple data analysis run the programs we will use in the next days For the practical lessons of the following days we will use and reuse these commands many times, so it is important to understand well the concept of today s lessons!

$Installing programs From the room computer: X:\San_Juan\Curso Metagenomics From the website: http://egg.umh.$

9 Installing programs From the room computer: X:\San_Juan\Curso Metagenomics From the website: Please INSTALL today s programs: -SSH (Secure Shell) -PSPad Important: change language for excel

10 Getting access to the servers Introduction to UNIX

11 Getting access to the servers GROUPS HOST NAME SERVER HERON FLAMINGO User Name: guest Password: 12345

12 Getting access to the servers CONGRATULATIONS! You are connected with the servers

13 Getting access to the servers YOUR DESKTOP SERVER YOUR DESKTOP SERVER DESKTOP

14 Getting access to the servers SSH: Secure Shell - Use a command-line interface (CLI) in a terminal.

Basic navigation guest@flamingo:~$ ENTER guest@flamingo:~$ ls ENTER data group5 group6 group7

15 Basic navigation ENTER ls ENTER data group5 group6 group7 group8 Introduction to UNIX ls: list directory contents options: ls l ls -1 ls lhtr man ls (exit with q)

16 Basic navigation pwd: show current directory pwd /home/guest cd: change directory ls data group5 group6 group7 group8 cd data ls day1 day2 day3 day4 pwd /home/guest/data cd.. ls data group5 group6 group7 group8 cd /home/guest/data/day1

17 Basic navigation *: match anything ls Groups.csv Groups.txt Numbers.txt pch.faa pch.ffn pch.gbk pch.txt ls *.txt Groups.txt Numbers.txt pch.txt ls G* Groups.csv Groups.txt cd /home/guest ls data group5 group6 group7 group8 pwd /home/guest

18 Basic navigation mkdir: make a directory guest@flamingo:~$ ls data group5 group6 group7 group8 guest@flamingo:~$ mkdir EGG guest@flamingo:~$ ls data group5 group6 group7 group8 EGG guest@flamingo:~$ cd EGG guest@flamingo:~/egg$ pwd /home/guest/egg cp: copy files guest@flamingo:~/egg$ cd /home/guest/data/day1 guest@flamingo:~/data/day1$ ls *.txt Groups.txt Numbers.txt pch.txt guest@flamingo:~/data/day1$ cp Groups.txt /home/guest/egg guest@flamingo:~/data/day1$ cd /home/guest/egg guest@flamingo:~/egg$ ls Groups.txt guest@flamingo:~/egg$ cp Groups.txt groups guest@flamingo:~/egg$ ls Groups.txt groups

19 Basic navigation mv: rename a file or move rm: remove a file rmdir: remove a directory guest@flamingo:~/egg$ ls guest@flamingo:~/egg$ mv groups groups.txt guest@flamingo:~/egg$ ls Groups.txt groups.txt guest@flamingo:~/egg$ rm Groups.txt rm: remove regular file `Groups.txt'? guest@flamingo:~/egg$ ls groups.txt guest@flamingo:~/egg$ rm groups.txt rm: remove regular file `groups.txt'? y guest@flamingo:~/egg$ ls guest@flamingo:~/egg$ guest@flamingo:~/egg$ cd.. guest@flamingo:~/$ ls data group5 group6 group7 group8 EGG guest@flamingo:~/$ rmdir EGG guest@flamingo:~/$ ls data group5 group6 group7 group8 y

20 Important tips TIP1: directory=folder TIP2: Do not use spaces in files or folders names. Use - or _ as word separators. TIP3: Do not use special characters (!,,?..). TIP4: You can use tab to complete the file/folder name while using the terminal. TIP5: In Linux, hello is not the same as Hello. TIP6: Files do not need EXTENSION (eg. File.txt ou table.xls). But they are a good way to help in the file organization (and are needed in WINDOWS programs). TIP7: Make sure you are in the right directory (use and abuse of ls and pwd) TIP8: If you want to stop a program, use Ctrl+C

21 Exercise 1 All the data for the practical sessions are in: /home/guest/data/ NEVER WORK INSIDE THE DATA DIRECTORY The data for today s exercises are in: /home/guest/data/day1 Make a folder with yourname (eg. carol) within your group s folder (eg. EGG). Then, copy all files we will use today in that folder. This is an important step that might be repeated in the following days of the practical sessions. From now, you should always work WITHIN your folder! 15 minutes

22 Exercise 1 - result guest@flamingo:~/$ ls data group5 group6 group7 group8 EGG guest@flamingo:~/$ cd EGG guest@flamingo:~/egg$ ls carol rohit mario aitor ana inma nikole guest@flamingo:~/egg$ cd carol guest@flamingo:~/egg/carol$ ls Groups.csv Groups.txt Numbers.txt pch.faa pch.ffn pch.gbk pch.txt

23 Working with files more: show file page by page more Groups.txt group first_name last_name country 1 Blanca Vera-Gargallo Spain 1 Mariane Schmidt Denmark 1 Ricardo Delgado-Santander Spain 1 Steffen Lott Germany 2 Coralis-delM. Rodriguez-Garcia Puerto-Rico 2 Javier Miralles-Lorenzo Spain 2 Lejla Pasic Slovenia 2 Santiago Catala-Garcia Spain 3 Laura Sanguino-Casado France 3 Laura Leite Brazil 3 Mara-F. Cuebas-Irzamy Puerto-Rico 3 Rakel Arrazuria Spain 4 Elena Gomez-Sanz Spain 4 Gerard Muyzer Netherlands 4 Julliane Medeiros Brazil 4 Ylenia Arizaga Spain

24 Working with files group first_name last_name country Blanca Vera-Gargallo Spain 1 Mariane Schmidt Denmark 1 Ricardo Delgado-Santander Spain 1 Steffen Lott Germany 2 Coralis-delM. Rodriguez-Garcia Puerto_Rico 2 Javier Miralles-Lorenzo Spain 2 Lejla Pasic Slovenia 2 Santiago Catala-Garcia Spain 3 Laura Sanguino-Casado France 3 Laura Leite Brazil 3 Mara-F. Cuebas-Irzamy Puerto_Rico 3 Rakel Arrazuria Spain 4 Elena Gomez-Sanz Spain 4 Gerard Muyzer Netherlands 4 Julliane Medeiros Brazil 4 Ylenia Arizaga Spain

25 Working with files wc l: count lines head: show top lines tail: show bottom lines cat: concatenate files. Also show the whole file at once. wc -l Groups.txt 31 Groups.txt head -3 Groups.txt group first_name last_name country 1 Blanca Vera-Gargallo Spain 1 Mariane Schmidt Denmark guest@flamingo:~/egg/carol$ tail -3 Groups.txt 8 Clara Cardoso Netherlands 8 Jose Moya-Cuevas Spain 8 Uljana Hesse South_Africa

26 Making output files Introduction to UNIX In Linux, data output can be put directly to an output file tail -1 Groups.txt 8 Uljana Hesse South_Africa guest@flamingo:~/egg/carol$ tail -1 Groups.txt > tail.txt guest@flamingo:~/egg/carol$ ls *.txt Groups.txt Numbers.txt pch.txt tail.txt guest@flamingo:~/egg/carol$ more tail.txt 8 Uljana Hesse South_Africa guest@flamingo:~/egg/carol$ head -1 Groups.txt group first_name last_name country guest@flamingo:~/egg/carol$ head -1 Groups.txt > tail.txt guest@flamingo:~/egg/carol$ more tail.txt group first_name last_name country guest@flamingo:~/egg/carol$ mv tail.txt header.txt guest@flamingo:~/egg/carol$ ls Groups.txt Numbers.txt pch.txt header.txt guest@flamingo:~/egg/carol$ ls *.txt > alltextfiles guest@flamingo:~/egg/carol$ more alltextfiles Groups.txt Numbers.txt pch.txt header.txt

27 Exercise 2 ls: list directory contents pwd: show current directory cd: change diretory mkdir: make a directory rmdir: remove directory rm: remove files mv: rename or move files cp: copy files man: manual more: show file page by page cat: concatenate files. Also show the whole file at once. head: show top lines tail: show bottom lines wc l: count lines >: redirecting output 20 minutes

28 The search tool: grep grep 7 Groups.txt 7 Allali Imane Morocco 7 Antonio Picazo Spain 7 Lucas Stal Netherlands guest@flamingo:~/egg/carol$ grep -c 7 Groups.txt 3 guest@flamingo:~/egg/carol$ grep antonio Groups.txt guest@flamingo:~/egg/carol$ guest@flamingo:~/egg/carol$ grep Antonio Groups.txt 6 Rafael-Antonio Rojas-Herrera Mexico 7 Antonio Picazo Spain guest@flamingo:~/egg/carol$ grep -i antonio Groups.txt 6 Rafael-Antonio Rojas-Herrera Mexico 7 Antonio Picazo Spain guest@flamingo:~/egg/carol$ grep i -c antonio Groups.txt 2 guest@flamingo:~/egg/carol$ grep 7 Groups.txt > group7.txt guest@flamingo:~/egg/carol$ grep 8 Groups.txt > group8.txt guest@flamingo:~/egg/carol$ cat group8.txt group7.txt > g8-7.txt

29 Cutting columns cut -f: cut a column (tab delimited by deafault) guest@flamingo:~/egg/carol$ more g8-7.txt 8 Clara Cardoso Netherlands 8 Jose Moya-Cuevas Spain 8 Uljana Hesse South_Africa 7 Allali Imane Morocco 7 Antonio Picazo Spain 7 Lucas Stal Netherlands guest@flamingo:~/egg/carol$ cut f2 g8-7.txt Clara Jose Uljana Allali Antonio Lucas guest@flamingo:~/egg/carol$ cut f2,4 g8-7.txt Clara Netherlands Jose Spain Uljana South_Africa Allali Morocco Antonio Spain Lucas Netherlands Introduction to UNIX

30 Sorting sort: sort data more g8-7.txt 8 Clara Cardoso Netherlands 8 Jose Moya-Cuevas Spain 8 Uljana Hesse South_Africa 7 Allali Imane Morocco 7 Antonio Picazo Spain 7 Lucas Stal Netherlands guest@flamingo:~/egg/carol$ sort g8-7.txt 7 Allali Imane Morocco 7 Antonio Picazo Spain 7 Lucas Stal Netherlands 8 Clara Cardoso Netherlands 8 Jose Moya-Cuevas Spain 8 Uljana Hesse South_Africa guest@flamingo:~/egg/carol$ sort k4 g8-7.txt 7 Allali Imane Morocco 7 Lucas Stal Netherlands 8 Clara Cardoso Netherlands 8 Uljana Hesse South_Africa 7 Antonio Picazo Spain 8 Jose Moya-Cuevas Spain

31 Sorting sort: sort data cut f4 g8-7.txt > country_g8-7.txt more country_g8-7.txt Netherlands Spain South_Africa Morocco Spain Netherlands sort country_g8-7.txt > sortedcountry_g8-7.txt more sortedcountry_g8-7.txt Morocco Netherlands Netherlands South_Africa Spain Spain

32 The uniq command uniq: remove duplicates more sortedcountry_g8-7.txt Morocco Netherlands Netherlands South_Africa Spain Spain uniq sortedcountry_g8-7.txt Morocco Netherlands South_Africa Spain uniq c sortedcountry_g8-7.txt 1 Morocco 2 Netherlands 1 South_Africa 2 Spain 6 total

33 Using Linux Pipes cut -f: cut column (tab delimited by default) sort: sort data uniq: remove duplicates Introduction to UNIX cut f4 g8-7.txt > country_g8-7.txt sort country_g8-7.txt > sortedcountry_g8-7.txt uniq c sortedcountry_g8-7.txt > countofcountry_g7-8.txt guest@flamingo:~/egg/carol$ more countofcountry_g8-7.txt 1 Morocco 2 Netherlands 1 South_Africa 2 Spain 6 total guest@flamingo:~/egg/carol$ cat g8-7.txt cut f4 sort uniq -c 1 Morocco 2 Netherlands 1 South_Africa 2 Spain 6 total

34 Using Linux Pipes cat Groups.txt grep 7 7 Allali Imane Morocco 7 Antonio Picazo Spain 7 Lucas Stal Netherlands guest@flamingo:~/egg/carol$ cat Groups.txt grep 7 grep Spain 7 Antonio Picazo Spain guest@flamingo:~/egg/carol$ cat Groups.txt grep 7 grep -v Spain 7 Allali Imane Morocco 7 Lucas Stal Netherlands guest@flamingo:~/egg/carol$ cat Groups.txt grep 7 grep v Spain cut f4 Morocco Netherlands guest@flamingo:~/egg/carol$ cat Groups.txt grep 7 grep Spain cut f2 Antonio

35 Using Linux Pipes cat Groups.txt cut -f4 grep -v "country" sort uniq c 2 Brazil 1 Denmark 2 France 1 Germany 1 Mexico 1 Morocco 4 Netherlands 4 Puerto_Rico 1 Scotland 1 Slovenia 1 South_Africa 11 Spain guest@flamingo:~/egg/carol$ cat Groups.txt cut -f4 grep -v "country" sort uniq c sort nrk1 head Spain 4 Puerto_Rico 4 Netherlands 2 France 2 Brazil

36 Working with fasta files Introduction to UNIX >name ATATCGCTGCATGATGGACCCTATGATAAAGGTAGTAGTAGATGAT GAGGTAGTAGTAGATGAGGTAGTAGTAGATGCATGATGGACCCTTA more single.fasta >seq1 ATATCGCTGCATGATGGACCCTATGATAAAGGTAGTAGTAGATGATGAGGTAGTAGTAGATGAGGTAGT AGTAGATGCATGATGGACCCTTA cat multi.fasta >seq1 ATATCGCTGCATGATGGACCCTATGATAAAGGTAGTAGTAGATGATGAGGTAGTAGTAGA >seq2 ATGATGAGGTAGTAGTAGATGATATAGTAGTATATGCATGATGGACCCTTATAGTAGTAG cat multi.fasta grep c > 2 guest@flamingo:~/egg/carol$ cat smallmeta.fasta grep c >

37 Exercise 3 ls, pwd, cd, mkdir, rmdir, rm, mv, cp, man more, cat, head, tail, wc l, > grep: search data cut -f: cut column (tab delimited by default) sort: sort data uniq: remove duplicates : pipe Tables, fasta and genbank files Use the manual to get the information and many options that each of these commands offer: man grep man cut 30 minutes

38 Text editor pico

39 Compressing files/folders zip file gzip file bzip2 file tar cvf folder.tar folder (make folder.tar) gzip folder.tar (make folder.tar.gz) Decompressing files/folders unzip file.zip gunzip file.gz bunzip2 file.bz2 tar xzvf folder.tar.gz

40 Home made programs convertseq: Biological sequence data can be in several different formats, fasta, genbank etc, and sometimes you need a simple method to convert from one to the other. lenseq:provides a simple way to compute lengths of sequences. It can also deal with a number of different biological sequence formats. gcseq: provides a simple way to compute gc content and lengths of sequences. It can also deal with a number of different biological sequence formats. faslice: provides a simple way to extract a few fasta sequences from a file. It works only with fasta files (not with genbank). lenfilter -h gcfilter -h

Exercise 4 ls, pwd, cd, mkdir, rmdir, rm,

and decompress files Use the help whenever

41 Exercise 4 ls, pwd, cd, mkdir, rmdir, rm, mv, cp, man more, cat, head, tail, wc l, > grep, cut f, sort, uniq convertseq lenseq gcseq faslice lenfilter gcfilter compress and decompress files Use the help whenever is needed: lenseq h lenseq -help Introduction to UNIX

decompress files Simple data analysis Run the programs we will use in the next days

42 We should be able to Connect to the servers Basic Linux commands: cp, mkdir, rmdir, mv cat, more, head, tail, redirect output grep, cut, sort, uniq, pipes compress, decompress files Simple data analysis Run the programs we will use in the next days Introduction to UNIX ASSEMBLY AND ANNOTATION then we are ready for the next session tomorrow!

43 Thank you!

Blast guest@flamingo:~/egg/carol$ formatdb i database.fasta p T database.fasta.phr database.fasta.pin database.fasta.psq guest@flamingo:~/egg/carol$ blastall i single-or-multi.

44 Blast formatdb i database.fasta p T database.fasta.phr database.fasta.pin database.fasta.psq guest@flamingo:~/egg/carol$ blastall i single-or-multi.fasta d /databasepath/database.fasta p blastp a 20 o output.blastp m8 e 1e-5

45 Working with fastq 1:N:0:ACTGAT ANGACGCTGTGTGACTCCCCGCTATCAGAACCGGTGTAAGCTACTGCTAAAACACGATG + C#1ADDFFHFHHHIJJJJJJIJJJJJJIJJJJJJFHGIJIJJJGIHIJJJJJJJHHFFF guest@flamingo:~/egg/carol$ more 1:N:0:ACTGAT ANGACGCTGTGTGACTCCCCGCTATCAGAACCGGTGTAAGCTACTGCTAAAACACGATGACATGTGCTG + C#1ADDFFHFHHHIJJJJJJIJJJJJJIJJJJJJFHGIJIJJJGIHIJJJJJJJHHFFFFFEEDEEEED guest@flamingo:~/egg/carol$ more 2:N:0:ACTGAT TTTGTTCAGCGTCGCTTCGAGAATGGCGCGGAGCCAATTCAGTGGATTCAGCTTTTGTTTCCGGCAGGT + CCCFFFFFHHHGHIJJJJJJJJJEIJJJIGI:AEHEICEA?CCHE;?CDA@CCEEEA?CC;>?B@?>B2

Introduction to UNIX command-line

Introduction to UNIX command-line Boyce Thompson Institute March 17, 2015 Lukas Mueller & Noe Fernandez Class Content Terminal file system navigation Wildcards, shortcuts and special characters File permissions