Anthill User Group Meeting, 2015

Transcription

1 Agenda Anthill User Group Meeting, Introduction to the machines and the networks 2. Accessing the machines 3. Command line introduction 4. Setting up your environment to see the queues 5. The different queues on the system 6. Queues and jobs 7. Policies to ensure equitable availability of computes to all 8. Submitting simple jobs to the queue 9. Submitting array jobs to the queue 10. Local data vs. scratch data 11. Blast, lastal, rapsearch, diamond and similar programs 12. Assembly 13. prinseq 14. focus 15. Moving data on and off the cluster. 16. Backing up data Introduction to the machines and the networks There are only a few machines that you need to worry about: anthill.sdsu.edu, rambox, edwardsdata.sdsu.edu and possibly rohan.sdsu.edu. I'll explain how everything fits together and when to use each. For most things, you are just going to want to use anthill.sdsu.edu for all your work. Accessing the machines SSH Mac OSX / Linux use Terminal Windows use Putty: Accessing from on campus: ssh username@anthill.sdsu.edu or fill in username and anthill.sdsu.edu 1

2 Accessing from off campus, use edwards-data.sdsu.edu or rohan.sdsu.edu: ssh -P 7010 Note: you have to use port 7010 to access edwards-data from off campus. For the more advanced people don't worry about this if it is too confusing screen to run virtual terminals so that you can start a command and go away, come back later and it will be running. Use screen -DR to reconnect to an existing screen. The main screen commands you need to know are: ctrl-a ctrl-c ctrl-a ctrl-n ctrl-a ctrl-p create a new screen move to the next screen move to the previous screen Basic LINUX Commands All our servers run linux (almost all run CentOS version 6), and so you have to get used to moving around in LINUX. We have a couple of cheat sheets to share with you. Here are a couple of resources that it is worth working through: Learn UNIX in 10 minutes: UNIX Tutorials for beginners: Linux Commands: We will work through some of these commands in the workshop:

3 A simple text editor on almost every system is nano Setting up your environment to see the queues We have queues to work on the cluster, and you need to set it up so you can see the queue. The main command to see what is running on the queue is qstat or for a more detailed view you can use qstat -f The different queues on the system Anthill has three queues that you can use: default The default queue has 35 machines, each with 16 processors and 128 GB RAM. Each processor runs an independent job, so you can run 560 jobs simultaneously on these machines. This queue is in eternally friendly mode, and all jobs are run on a first-in first-out basis. important The important queue has 4 machines with 16 processors and 128 GB RAM. This queue is for single jobs only. Do not run array jobs on this queue or they will be terminated! The queue is for testing and running individual programs. smallmem This queue has 9 machines each with 8 processors (72 computes total), Each machine has 14 GB RAM, except node1 that has 24 GB RAM. People often forget about this queue, so sometimes it is worth checking! 3

4 Policies to ensure equitable availability of computes to all Be mellow. Be nice to others. Listen to your mother. Submitting simple jobs to the queue You need a simple shell script (which can be very easy) and then you need to use this command to submit the job: qsub -cwd./scriptname.sh You can also add some other commands. Common commands include: -q use a different queue (e.g. -q important or -q smallmem) -o where to put the output file -e where to put the error file Examples: Submit a job to the cluster using the default queue: qsub -cwd./scriptname.sh Submit a job to the cluster using the important queue: qsub -cwd -q important./scriptname.sh Submitting array jobs to the queue To submit an array job we add the flag -t to our qsub command -t 1-100:1 submit an array job. This will submit jobs from in increments of 1 With an array job, scriptname.sh gets passed a special variable called $SGE_TASK_ID that is the number of the job it is running. I will provide you with some template code to process every file in a directory. Submit an array job to the cluster, and redirect output and error files to a directory: mkdir sge qsub -cwd -e sge -o sge -t 1:540:1./scriptname.sh 4

5 Naming files SGE submission scripts: Please don't use run.sh or job.sh Limit file names to 10 meaningful characters Include some notion of the command (rapsearch, blast, etc) Potentially start everything with s_ Fasta files Use.fna for nucleotide Use.faa for protein Output files Append.blastp,.blastn,.rapsearch2,.lastal etc so that you remember what you did Remembering what you did Make a file called how-to.txt and copy and paste your commands into it. Annotate the file so you remember what the commands are doing. If you forget to do that, you can make one like this history > how-to.txt and then edit the how-to.txt file: nano how-to.txt NFS data vs. scratch data There are two ways of housing your data. Your home directory is on a different file server machine, and so anything in there will need to be transported to the machine where the computations are done, and then the results will need to be moved back again. As an alternative, each of our computers has a local hard drive with space in /scratch. This is common space, that anyone can use, and thus has the problem that it fills up. Opinions vary on the importance of using your home directory versus /scratch. I keep everything on the file server and compute off of that. Kate and Rob S. move their data to the /scratch space and compute off of that. Certainly if you have something large that you routinely compute against (e.g. the nr database) it would be good to house it on /scratch. Also, if you have a lot of jobs running that require network IO, occassionaly it is better to use /scratch to avoid network issues. However, if you don't 5

6 know what you are doing, don't worry about it. The time saving will be very small (I think insignificant in the overall scheme of things), so don't waste your time! Blast, lastal, rapsearch, diamond and similar programs Its time to move away from BLAST, there are some really good alternatives that we have installed on the cluster. For DNA-DNA (instead of blastn) comparisons we (currently) recommend: For protein-protein (instead of blastp) comparisons we (currently) recommend: LASTAL For protein-dna (instead of blastx) comparisons we (currently) recommend: rapsearch2 We will work through examples of all of these. BLAST Even though it is time to move away from it, I suppose people still want to do it. You will need to format your database using the makeblastdb command (unless you use one that is already formatted). The commands that you want to run are: /usr/local/blast+/bin/blastn, /usr/local/blast+/bin/blastp, /usr/local/blast+/bin/blastx, etc. We have a trivial blast solution for you, it is a script that takes your fasta file, splits it up into a series of smaller files, and then runs your specified blast program against that file. split_blast_queries_blastplus If you run the command without any options you get this help output: 6

7 /home3/redwards/bioinformatics/cluster/split_lastal_queries.pl <options> -f file to split -n number to break into -d destination directory (default = ".") -p matrix: BL62 (for protein/protein or BL80 for DNA/protein) -db lastal database -ex lastal executable location (default is /usr/local/last/bin/) -N job name (default is s_lastal) -rev reverse the order of files that are submitted to the queue. (i.e. so you can run twice and start from the end backwards!) -v verbose Other things will be used as lastal options. Unless -db is provided we will just split and stop Basically the main options you need are -f for the fasta file, -n for the number to break it into, -d for a directory to put the results, -p for the blast program to run, -db for the database to compare to. If you want to add options to blast+ you can add them at the end of the command. For example: split_blast_queries_blastplus -f Nudibranch_S7_L001_R1_001.fasta -n 200 -d nudi -p blastx -db /home/db/blast/nr/nr -evalue 1e-5 This creates a directory called nudi and outputs files into there, including the blast output files. Since this is blastx, the blast output files all end with blastx. (That will change if you use blastn or blastp, of course.) Once the blast is complete you can concatenate all the blast output files using the cat command: cat nudi/*.blastx > nudibranch.blastx Now I have a single file called nudibranch.blastx that has all the blast output. LASTAL To use LASTAL you first need to format the database. If you have a protein database you need to specify the the infile is protein with "-p". Then specify the new database name and the location for the fasta formatted input file. lastdb -p nr nr.faa 7

8 This will create several files, and also possibly break up the database into ~20GB blocks. To run LASTAL you need the location of the database, followed by the QUERYFILE, followed by which score matrix to use (I originally was using BL62, but the developer recommended BL80 for short sequences), followed by the output format (0 for tabular). lastal nr QUERYFILE.faa -p BL80 -f 0 A sample SGE script looks like: #!/bin/bash lastal /usr/data/kate/nr/nr QUERYFILE.faa -p BL80 -f 0 We also have a trivial lastal command that is based on the blast command above: split_lastal_queries which has a similar help profile: /home3/redwards/bin/split_lastal_queries <options> -f file to split -n number to break into -d destination directory (default = ".") -p matrix: BL62 (for protein/protein or BL80 for DNA/protein) -db lastal database -ex lastal executable location (default is /usr/local/last/bin/) -N job name (default is s_lastal) -rev reverse the order of files that are submitted to the queue. (i.e. so you can run twice and start from the end backwards!) -v verbose Other things will be used as lastal options. Unless -db is provided we will just split and stop 8

9 You use the command in the same way. although the options are slightly different (-p for the pairwise matrix): split_lastal_queries -f Nudibranch_S7_L001_R1_001.fasta -n 200 -d nudi -p BL80 -db /home/db/lastal/nr-lastal/nr Again, this will result in a directory of output files, and you can concatenate them as before. RAPSearch2, Reduced Alphabet based Protein similarity Search RAPSearch is about 100 times faster than BLAST and in single thread mode requires up to 2G memory. Its a great replacement for blastx, but is slightly less sensitive than blastx (especially in the fast mode) so you may miss some rare matches. There are two steps to running rapsearch2: Formating the database prerapsearch -d Fasta_File -n DatabaseName Running RAPSearch2 rapsearch -a 1 -q FASTA/FASTQ -d DatabaseName -o OUTPUT -v NumberDbSequences -z NumberThreads -e evalue -b 0 -s f -a 1 set the program to its fast mode -a 0 runs the program in its sensitive mode -b 0 sets the program to not write any sequence alignments -s f sets the program to use E value in the same format as BLAST Sequence assembly We currently recommend (and use) the St. Petersburg assembler, SPAdes: This runs fine on the cluster for most sequences, including metagenomes. If you run out of memory we can run it on rambox contact Rob and I'll work with you on that. to run this on the cluster, I put this in my script file and then submit it to the queue: /home3/redwards/bin/spades/spades linux/bin/spades.py -o spades.assembly --careful --dataset files.yaml 9

10 and files.yaml contains: [ ] { } orientation: "fr", type: "paired-end", right reads: [ "/home3/redwards/johnkirby/rob/1291/fastq/sample4-1291wt_s4_l001_r1_001.fastq", /home3/redwards/johnkirby/rob/1291/fastq/1_1291wt_4_cttgta_l001_r1_001.fastq", "/home3/redwards/johnkirby/rob/1291/fastq/2_1291_0177_4_cgatgt_l001_r1_001.fastq", ], left reads: [ "/home3/redwards/johnkirby/rob/1291/fastq/sample4-1291wt_s4_l001_r2_001.fastq", "/home3/redwards/johnkirby/rob/1291/fastq/1_1291wt_4_cttgta_l001_r2_001.fastq", "/home3/redwards/johnkirby/rob/1291/fastq/2_1291_0177_4_cgatgt_l001_r2_001.fastq", ] 10

11 Prinseq You can run prinseq-lite.pl on the cluster. For example, to generate the report, you need to put this in your file: perl prinseq-lite.pl -verbose -fastq test.fq -graph_data test.gd -out_good null -out_bad null You can then upload the test.gd file to the website to see the report. focus Geni will show you! Moving data on and off the cluster Use scp from the command line. For Windows try WinSCP ( SSH Secure ( For Mac try CyberDuck: ( Rbrowser ( Backing Up Data Your data is NOT backed up. It is your responsibility to back it up to an external hard drive or another source. DO NOT RELY ON US TO PRESERVE YOUR DATA!!! Parting thoughts! Be mellow, be nice to others, everyone uses the resources. 11