The H3ABioNet GWAS Pipeline

Transcription

1 School of Electrical and UNIVERSITY OF THE WITWATERSRAND, JOHANNESBURG Information Engineering The H3ABioNet GWAS Pipeline Scott Hazelhurst 1 Introduction Need to build pipelines Many bioinformatics workflows are complex multiple steps, software dependancies have multiple parameters Constraints of good scientific practice needs to be re-rerun often in understanding data reproducible by others portable Workflow/Pipeline Packaging of steps in a complex analysis automate the steps not black-box user needs to understand the individual steps Use appropriate software technology to support this Nextflow Containerisation (Docker/Singularity) 1

2 Pan-African Bioinformatics Network for H3Africa Support the work of the Human Heredity and Health in Africa (H3A) Consortium: build informatics and bioinformatics capacity support H3A projects > 30 nodes in 15 African countries Many activities pipelines work is just one of them 2 Pipelines H3ABioNet Pipelines Project Developed set of pipelines to support common workflow next generation sequence variant calling 16S RNA analysis imputation GWAS 3 H3A GWAS H3A GWAS pipeline Three separate workflows conversion from Illumina Top/Bottom format Quality control Basic association study 3.1 Installing Set-up Must install: Java 8 Nextflow Either all the bioinf tools, envs or Docker/Singularity 2

3 Method 1: Using git Especially if you re going to extend. git clone This creates a folder called h3agwas with scripts inside it. Method 2: Using nextflow nextflow pull h3abionet/h3agwas 3.2 Quality control Remove duplicate SNPs Remove SNPs, individuals with high missingness, HWE, MAF Remove outliers on sample heterozygosity Remove relatedness Remove SNPs on HWE, differential missingness Produce reports Running a QC Controlled by a config file params.input_dir = "input" params.input_pat = "sample" params.output = "test-qc" params.output_dir = "output" params.case_control = "/datab/awigengwas/aux/data.phe" params.case_control_col = "bmi_case_control" params.batch = "/datab/awigengwas/aux/awifam.phe" params.batch_col = "batch" params.phenotype = "/datab/awigengwas/aux/data.phe" params.pheno_col = "site" params.sexinfo_available = true params.pi_hat = 0.18 Main components Input directory and file Output directory and file Batch analysis: strongly recommended By phenotype: e.g., site, strongly recommended 3

4 Case-control: binary compulsory QC cut-offs Need phenotype file with headers. Batch analysis is intended for cases where the DNA from the experiment is collected, shipped or genotyped separately. In QC you need to be sure that there aren t significant differences between the different batches. Phenotype analysis allows you to do more detailed batch analysis. Depending on the phenotype chosen, there may be overall genotype difference or not. For example, in the AWI-Gen study, we chose collection site as the phenotype of interest. Here we expect there to be very sigificant overall genotype differences because our sites are in very different parts of Africa. The point of doing phenotype analysis for us is to ensure that there are no (a) batch effects at any sites, and (b) to explore missingness and QC at the site level. You could also choose sex as the phenotype, in which case you would not see overall genotype difference. The case-control column in compulsory you need to choose a phenotype that is binary. We do not expect the genotypes overall to be different between cases and controls and if there is an overall difference there s likely a QC problem. If you don t have a suitable binary case-control make up one even randomly assign participants to different groups. For each of the above, you need to specify the file name where the data can be found and the column (the first line of the file must have headers). Usually, it will be the same file and different column names, but you may have multiple files To get the workflow nextflow pull h3abionet/h3agwas Then to run #either nextflow run -c my.config h3abionet/h3agwas/plink-qc #or nextflow run -c my.config h3abionet/h3agwas/plink-qc nextflow run -c my.config h3abionet/h3agwas/plink-qc N E X T F L O W ~ version Launching../plink-qc.nf [pedantic_kare] - revision: ac5217ecef Sexinfo available command [warm up] executor > local [fe/c0582e] Submitted process > inmd5 (1) [d4/1e3bbb] Submitted process > getduplicatemarkers (1) [f4/ed17d8] Submitted process > removeduplicatesnps (1) [ad/6d480c] Submitted process > getinitmaf (1) [8b/40df13] Submitted process > getx (1) [57/ef097e] Submitted process > identifyindivdiscsexinfo (1) [91/59ea3c] Submitted process > generateindivmissingnessplot (1) [ab/e1643d] Submitted process > generatesnpmissingnessplot (1) [d8/9d32e8] Submitted process > removeqcphase1 (1).... [e5/2817fb] Submitted process > generatemafplot (1) [d4/e6d01c] Submitted process > producereports (1) The output report is called output/test-qc.pdf 4 -profile pbs -profile docker

5 Exercise Set up 1. You should be logged into the head node of the cluster cream.core.wits.ac.za if you can otherwise log on to Create a directory for this exercise. 3. Copy the file /global/gwas-workshop/plinkdata/ga14.config into this directory. 4. Fetch a copy of the workflow. You can choose, either method above but if you are not familiar with git I suggest you do the following nextflow pull h3abionet/h3agwas The first time you run nextflow will require downloadin some software dependancies. The command nextflow pull fetches the workflow software from GitHub. (As an aside, the software is put in a hidden directory ~/.nextflow but you don t need to look at it) Config file Edit the config file I ve set it up for you so for the first time, so you shouldn t need to change anything. You ll get a chance to play around later. But at this stage read the config file carefully to understand what s there. Remember you can read the extensive documentation at https: //github.com/h3abionet/h3agwas. Note that you don t have to copy the input data this can stay where it is in the shared directory. Just specify where it is. I ve suggested that the output directory be qcdata. You don t have to create it as Nextflow will do that for you (though you can if you want). There are many default parameters to view all the parameters that will be applied when you run the job nextflow -c ga14.config config h3abionet/h3agwas/plink-qc.nf Running the workflow To run the workflow you ll need to specify the following: the name of the script to run The configuration file the mode of running if you are on cream.core.wits.ac.za, then specify you want to use the PBS scheduler to run jobs with the -profilepbs option if you are on , don t specify an option You will either use one of the two lines below depending on which machine you are on 5

6 # if you are on cream nextflow run h3abionet/h3agwas/plink-qc.nf -c ga14.config -profile pbs # if you are on nextflow run h3abionet/h3agwas/plink-qc.nf -c ga14.config Report The workflow should complete you will be told where the report can be found. Use scp to copy the file from the cluster to your machine and you can then open it and view. This is a very small example and somewhat artificial so some of the pictures may look a little odd. Experiment Try with different parameters and see what effect the change has. When you run the workflow again use the -resume option then only those parts of the workflow that need to be re-run will be re-run. Association study plink-assoc.nf Association workflow very experiment dependant data, population structure, co-variate, question basic workflow implemented for initial study can be extended Config file Example config file params.input_dir = "/spaces/scott/assoc/agt" params.input_pat = "25" params.output = "allgemma" params.output_dir = "assocresults" params.data = "/data/h3agwas/data.csv" params.covariates = "age,sex" params.pheno="bmi_c/np.log,wst_hip_r_c,standing_height_mm" params.gemma_num_cores = 8 params.gemma = 1 params.linear = 1 nextflow -c assoc.config config h3abionet/h3agwas/plink-assoc.nf 6

7 Running the workflow nextflow run h3abionet/h3agwas/plink-assoc.nf \ -c assoc.config The output will be found in the assocresults configuration file because that s specified in the assoc.config file Note you may need to change the above if you are running on a cluster (e.g. cream.core. wits.ac.za) you need to use the pbs profile nextflow run h3abionet/h3agwas/plink-assoc.nf -c assoc.config -profile pbs And if you are using Docker you will say nextflow run h3abionet/h3agwas/plink-assoc.nf -c assoc.config -profile docker Exercise Copy the file /global/gwas-workshop/plinkdata/assoc.config to your local directory. This has a simple set up for the association testing. Read through the file to understand the basics again, the full documentation can be found on the site. In this example, you will be used the QCd data you produced in the last step. If you changed the output name or directory in the previous exercise, you will have to change input name and/or directory in this exercise. In this example, we will do the basic χ 2 test, with Bonferroni correction and permutation testing (permutation testing is a default you can turn it off). We ll test two phenotypes pheno and sex and use sex as a covariate. Remember this is a very small data set so we are unlikely to find statsitically significant results. Now look at the fulll configuration: 4 Updating the workflow The workflows are updated bug fixes If you got the workflow using Nextflow nextflow pull h3abionet/h3agwas If you used git change directory git pull 7

8 5 Problems If something goes wrong, may be difficult to understand why workflow data As an example, here I deliberatly give the wrong column name for the batch, which causes one of the scripts to fail. We get ERROR ~ Error executing process > batchproc (1) Caused by: Process batchproc (1) terminated with an error exit status (1) Command executed [/spaces/scott/h3abionet/h3agwas/templates/batchreport.py]: #!/usr/bin/env python3 from future import print_function import argparse import sys import re Followed by several hundred lines of text Command exit status: 1 Command output: (empty) Command error: Traceback (most recent call last): File ".command.sh", line 577, in <module> bfrm, btext = getbatchanalysis() File ".command.sh", line 550, in getbatchanalysis result = miss_vals(ifrm,bfrm,args.batch_col,args.sexcheck_report) File ".command.sh", line 188, in miss_vals g = pd.merge(pfrm,ifrm,left_index=true,right_index=true,how= inner ).groupby(pheno_c File "/opt/exp_soft/python36/lib/python3.6/site-packages/pandas/core/generic.py", line **kwargs) File "/opt/exp_soft/python36/lib/python3.6/site-packages/pandas/core/groupby.py", line return klass(obj, by, **kwds) File "/opt/exp_soft/python36/lib/python3.6/site-packages/pandas/core/groupby.py", line 8

9 mutated=self.mutated) File "/opt/exp_soft/python36/lib/python3.6/site-packages/pandas/core/groupby.py", line raise KeyError(gpr) KeyError: batches Work dir: /spaces/scott/h3abionet/h3agwas/test/work/cf/335b6d21ad75841e1e d3d Tip: when you have fixed the problem you can continue the execution appending to the nextfl -- Check.nextflow.log file for details WARN: Killing pending tasks (1) test]$ The error is obscure but if you look you can see the mistake. You are also given the work directory for the process that failed. You can change directory to this directory and explore what happened: All input files will be there The script that failed will be found in.command.sh The output is in.command.out and error in.command.err 9