Using the Galaxy Local Bioinformatics Cloud at CARC

Transcription

1 Using the Galaxy Local Bioinformatics Cloud at CARC Lijing Bu Sr. Research Scientist Bioinformatics Specialist Center for Evolutionary and Theoretical Immunology (CETI) Department of Biology, University of New Mexico CARC Galaxy UNM 1

2 Outline Self- introduction Galaxy Hands on activity Demo 1 Demo 2 Useful information CARC Galaxy UNM 2

3 Hands up if you have Got in touch with NGS Data? Known Fastq format what the 4 lines means? Done RNA- Seq? Run command lines Created a BLAST database Installed tools on Linux/Unix Used online Bioinformatics Platform CARC Galaxy UNM 3

4 Self- Introduction Name Department Project Lijing Bu Biology RNA- Seq, Genome re- sequencing Tools Shell, Perl Blast, ClustaW Tophat, Cufflinks, Trinity R, edger, DESeq Abyss, SOAPdenovo, Velvet CARC Galaxy UNM 4

5 Big Data and Abundant Tools NGS Data: 2~50 GB initial data per project Analysis involves multiple steps and tools Computational challenge Command lines make it easy to construct workflows but it takes time to master them blastx - query Trinity.fasta - db /home/blast/ /swissprot - out blastx.outfmt6 - evalue 1e num_threads 44 - max_target_seqs 1 - outfmt 6 CARC Galaxy UNM 5

6 Bioinformatics Clouds Easy to use Share data, analysis steps Workflows $$$$$ Fixed workflows Few Apps CARC Galaxy UNM 6

7 Open source PSU 700 individual tools in 200 packages, 40 categories Easy to use Highly customizable Local instance Add almost any Bioinformatics tool Use customized reference database Capable to use high- performance computer clusters Developers can publish new tools CARC Galaxy UNM 7

8 Galaxy Interface Tools Panel View Panel History Panel CARC Galaxy UNM 8

9 Example Workflow RNA- Seq CARC Galaxy UNM 9

10 CARC Ulam Cluster 16 nodes x 8 CPUs/32 GB Xena Cluster 1T ~ 3 T shared MEM UNM Local Cloud for Bioinformatics Galaxy Web Server CARC Manager User User User Administrator CARC Galaxy UNM 10

11 Agenda of CARC Phase I - Sputnik: Proof of concept Local galaxy test run. Tools installation. Connect to CARC server, submit PBS jobs. Phase II - Pluto: Internal test. Hardware connection to cluster, install Linux and galaxy, set up to connect to submit PBS jobs, main page design. Continue to add software, separate cluster jobs (60 s lag) versus local jobs. For a few tools, do batch mark test to find best setting to provide best performance. For some tools, extend PBS jobs to be submitted to server of large shared memory (1TB ~ 3TB). Open to few internal users, workshops. Fix and add more tools and local databases based on feedback. Phase III - Pluto: Open to more users Install more tools as requested by users. Build workflows from repeated used tools. Develop tools/workflows for specific purpose, and publish/share them to all Galaxy group. Possible upgrade hardware. CARC Galaxy UNM 11

12 Register CARC Account PI apply a project (approve in 1-2 days) started/request- a- project.html Name, , title Abstract Students apply for an account linked to PI s project (approve in 1-2 days) started/request- an- account.html Name, and project name to link to. Select machines want to use Contact Lijing Bu to create a Galaxy account CARC Galaxy UNM 12

13 Recommend Links about Galaxy All about Galaxy Ask Questions on BioStar Videos of various analysis using Galaxy :alphabetical/format:thumbnail CARC Galaxy UNM 13

14 Galaxy CARC User name: workshop- user# where # is your seat number Password: carcgalaxy Change password after login! Temporary user accounts were created for workshop use only. All data/workflows of temp user accounts will be deleted one month after the workshop. CARC Galaxy UNM 14

15 Hands On Demo 1 Basic dataset management 1. Shared histories 2. NGS Reads QC 3. Workflow Handle Multiple Datasets 1. Select multiple datasets as input 2. Build datasets collection Demo 2 RNA- Seq workflow Copy datasets Upload data with a link View and run workflow Datasets management Manage history Delete/hide datasets Share history CARC Galaxy UNM Detailed instructions PDF file is at outreach/workshops- - training/workshop- materials/index.html Derived from online Galaxy Project s video at 15

16 Demo 1 Basic dataset management 1. Shared histories 2. Reads QC 3. Workflow Handle Multiple Datasets 1. Select multiple datasets as input 2. Build datasets collection CARC Galaxy UNM 16

17 Find Published History CARC Galaxy UNM 17

18 Import History - 1 CARC Galaxy UNM 18

19 Import History - 2 Click to view tools in this category. Click to have brief view Download dataset Eye: view dataset Pencil: change features Cross: delete dataset CARC Galaxy UNM 19

20 Check on the Reads Quality Click on the link to open the tool Single FastQC on fastq read file 1. Delete dataset 3, Click to check deleted data, and undelete dataset 3. CARC Galaxy UNM 20

21 FastQC Results Single Input FastQC generates 2 output files 1. HTML webpage report (shown here data6) 2. Raw text report (data 7) Good or bad Illumina Data? CARC Galaxy UNM 21

22 Reads Quality Filtering Single end data Find Trimmomatic in the tool panel, Click on the link Run with default setting CARC Galaxy UNM 22

23 Trimmomatic Results CARC Galaxy UNM 23

24 FastQC on Filtered Data 2. Switch to filtered dataset 3. Run 1. The re- run button CARC Galaxy UNM 24

25 Improved Reads Quality by Filtering Before After Trimmomatic CARC Galaxy UNM 25

26 Extract Workflow from History Uncheck dataset 2-5, keep the analysis steps on dataset 1 only. CARC Galaxy UNM 26

27 Extract Workflow from History CARC Galaxy UNM 27

28 Extract Workflow from History Save & Run Click tools to add them into current workflow Mark output files to hide the rest in the history. CARC Galaxy UNM 28

29 Select Multiple Files as Input Button to Select multiple files Shift + Select: press the shift key to select a series of files. Control or command key: press to select or deselect multiple files. View from individual tool. CARC Galaxy UNM 29

30 Create Datasets Collection for Multiple Step Analysis Build list for pair- end read files. CARC Galaxy UNM 30

31 Datasets Collection Created CARC Galaxy UNM 31

32 FastQC - Select Datasets Collection CARC Galaxy UNM 32

33 Results of FastQC on Collection Instead of two output files, there are two lists of output files. Each list has 4 files. CARC Galaxy UNM 33

34 Mange the History Share your analysis to another user or to everyone. CARC Galaxy UNM 34

35 Copy Datasets to a New History Select fastq datasets 1 to 5 Name the new history CARC Galaxy UNM 35

36 Demo 2 RNA- Seq workflow Copy datasets Upload data with a link View and run workflow Datasets management Manage history Delete/hide Datasets Share history CARC Galaxy UNM 36

37 RNA- Seq Technology Input 6 files Reads 2 Samples x 2 Replicates Reference Sequences General Feature Format file Tools NGS aligner TopHat2 Reads counter/stats - Cufflinks CARC Galaxy UNM 37

38 Upload Reference Sequence 1. On a new window, open the follow address UCSC FTP site of human reference genome sequences 2. Right click on chromosome 19.fa.gz, and copy link address. Right click: On Mac use two fingers Note: Be careful where you get data! NCBI, UCSC, ENSEMBL databases store data in slightly different format (ID system, chromosome label, GFF). Correct link CARC Galaxy UNM 38

39 Upload Reference Galaxy is sensitive to data type! Most tools require fastqsanger type for fastq files, rather than fastq, fastcssanger, fastqillumina. CARC Galaxy UNM 39

40 Upload Reference Sequence When paste the link, make sure the size is not empty. If empty, type a space after your pasted link address. CARC Galaxy UNM 40

41 Find Published Workflows CARC Galaxy UNM 41

42 CARC Galaxy UNM 42

43 !!! The default input file is the last file that fit the type format. For multiple files with the same format type (here fastq), the input order need to be checked.!!! CARC Galaxy UNM 43

44 !!! The default input file is the last file that fit the type format. For multiple files with the same format type (here fastq), the input order need to be checked.!!! CARC Galaxy UNM 44

45 CARC Galaxy UNM 45

46 Jobs Running If the page didn t reload automatically, but the circle in the tab is circling, the job is running. Be patient. CARC Galaxy UNM 46

47 Grey box Jobs are waiting CARC Galaxy UNM 47

48 Yellow Jobs are running CARC Galaxy UNM 48

49 Red box Error messages CARC Galaxy UNM 49

50 Manage Datasets in the History Click to show deleted files. Click to show hidden files. Click again to hide them. In workflows, you can specify to hide unwanted intermediate files. (more details in workflow build section) CARC Galaxy UNM 50

51 Demo Results BAM files reads aligned to reference. Newly found transcripts in GFF format (two samples merged) Differential Expression Analysis results CARC Galaxy UNM 51