New High Performance Computing Cluster For Large Scale Multi-omics Data Analysis. 28 February 2018 (Wed) 2:30pm 3:30pm Seminar Room 1A, G/F

Size: px

Start display at page:

Download "New High Performance Computing Cluster For Large Scale Multi-omics Data Analysis. 28 February 2018 (Wed) 2:30pm 3:30pm Seminar Room 1A, G/F"

Emerald Hall
5 years ago
Views:

1 New High Performance Computing Cluster For Large Scale Multi-omics Data Analysis 28 February 2018 (Wed) 2:30pm 3:30pm Seminar Room 1A, G/F

2 The Team (Bioinformatics & Information Technology) Eunice Kelvin Juilian Kevin Nick 2

3 HPCF High Performance Computing Facility Existing HPC (2011) Master / Login Node: hpcf.cgs.hku.hk Hostname: statgenpro 15 Compute Nodes / 376 cores / 1.5TB RAM Operating System: Ubuntu Job Scheduler: Torque / MAUI Tool: None New HPC (2018) Master / Login Node: hpcf2.cgs.hku.hk Hostname: omics 13 (8+5) Compute Nodes / 640 cores / 3.6TB RAM Operating System: CentOS 7.3 Job Scheduler: PBS Pro Tool: Environment Modules 3

What s New Environment Modules a tool to help you manage your shell environment, by allowing groups of related environment-variable settings to be made or removed dynamically What does it mean to you?

4 What s New Environment Modules a tool to help you manage your shell environment, by allowing groups of related environment-variable settings to be made or removed dynamically What does it mean to you? No need to change path or type in full path to access a software and/or a version worry about dependencies of a software Simply load and unload the desired version of a software Minor change of script required but job submission is basically the same Software to be used by all users will be installed by IT using Modules You can continue to install your own software in your /home directory 4

5 What s New Updated Queue parameters Queue name Max. no. of processors per job Max. memory (GB) per job Max. no. of job running per user Max. no. of job queuing per user Max. Walltime (hr) small 2 10 (8) 18 (16) 40 (30) 6 small_ext 2 10 (8) 6 (4) 12 (10) 60 (48) medium 12 (10) 50 (40) 12 (10) 25 (22) 24 (18) medium_ext 12 (10) 50 (40) 6 (4) 8 60 (48) large (90) 3 (2) 4 84 (72) test default NA NA NA NA NA 5

Homo_sapiens mirbase ANNOVAR Mus_musculus ncrna CellRanger

6 What s New Some commonly used genome references / databases have been added Genomes References Database Software Bundle Databases Homo_sapiens mirbase ANNOVAR Mus_musculus ncrna CellRanger Gallus_gallus SILVA_DB GATK_bundle Equus_caballus mutect oncotator igenomes More coming... 6

7 Performance benchmark (HPC 2011 vs HPC 2018) Speed Up GFLOPS x Passmark Performance Test x 1 FLOPS: floating point operations per second 1 GFLOPS:

8 Quick demo Basic use of module module avail module list module load cutadapt module help cutadapt module unload cutadapt module purge Sample script modification Add module load commands/remove full path Log location 8

9 -bash-4.2$ module avail /software/modules/modulefiles ANNOVAR/2017Jul16 HTSeq/0.9.1 (D) STAR/2.5.2a bwa/ BEDTools/ MACS/ STAR/2.5.3a (D) bwa/ (D) BEDTools/ MACS/ (D) TrimGalore/0.4.1 cutadapt/1.8.1 BEDTools/ (D) MUMmer/3.22 TrimGalore/0.4.5 (D) cutadapt/1.15 (D) BioPerl/1.7.2 MUMmer/3.23 (D) Trimmomatic/0.33 idba/1.1.3 Canu/1.5 NCBI-blast/ Trimmomatic/0.36 (D) java/7.0_25 Canu/1.6 (D) NCBI-blast/ (D) VerifyBamID/1.1.2 java/7.0_80 CellRanger/2.0.1 Oncotator/ VerifyBamID/1.1.3 (D) java/8.0_161 (D) CellRanger/2.1.0 (D) PEAR/ bamutil/ java/9.0.4 DESeq2/ PEAR/ (D) bamutil/ (D) miniconda2/ DESeq2/ (D) Perl/ bamtools/2.3.0 mutect/1.1.4 EBSeq/1.9.3 Picard/2.0.1 bamtools/2.5.1 (D) mutect/1.1.5 (D) EBSeq/1.18 (D) Picard/ (D) bcl2fastq/2.19 python2/ FASTX-toolkit/ QIIME/1.9.1 bcl2fastq/2.20 (D) python3/3.6.4 FASTX-toolkit/ (D) QIIME2/ bedgraphtobigwig/4 samtools/ FastQC/ R/3.2.5 bismark/ samtools/1.3 FastQC/ (D) R/3.4.3 (D) bismark/ (D) samtools/1.6 (D) GenomeAnalysisTK/3.5 RNAmmer/1.2 bowtie/1.0.0 strelka/ GenomeAnalysisTK/3.7 RSEM/ bowtie/1.2.2 (D) strelka/2.8.4 (D) GenomeAnalysisTK/3.8 (D) RSEM/1.3.0 (D) bowtie2/2.2.5 trnascan-se/1.3.1 HOMER/4.9 SPAdes/ bowtie2/2.3.4 (D) HTSeq/0.6.1 SPAdes/ (D) bwa/ /opt/lmod/7.7.14/lmod/lmod/modulefiles/core lmod/ settarg/ Where: D: Default Module 9

10 -bash-4.2$ module list No modules loaded -bash-4.2$ -bash-4.2$ module load cutadapt cutadapt/1.15 is loaded -bash-4.2$ -bash-4.2$ module list Currently Loaded Modules: 1) cutadapt/1.15 -bash-4.2$ -bash-4.2$ module load cutadapt/1.8.1 cutadapt/1.15 is unloaded cutadapt/1.8.1 is loaded The following have been reloaded with a version change: 1) cutadapt/1.15 => cutadapt/ bash-4.2$ 10

11 -bash-4.2$ module load GenomeAnalysisTK Lmod has detected the following error: Cannot load module "GenomeAnalysisTK/3.8" without these module(s) loaded: java/8.0_161 While processing the following module(s): Module fullname Module Filename GenomeAnalysisTK/3.8 /software/modules/modulefiles/genomeanalysistk/3.8.lua -bash-4.2$ -bash-4.2$ module list Currently Loaded Modules: 1) cutadapt/ ) java/8.0_161 3) GenomeAnalysisTK/3.8 -bash-4.2$ module unload cutadapt/1.8.1 cutadapt/1.8.1 is unloaded java/8.0_161 is loaded GenomeAnalysisTK/3.8 is loaded -bash-4.2$ module list Currently Loaded Modules: 1) java/8.0_161 2) GenomeAnalysisTK/3.8 11

12 # EXISTING SCRIPT #!/bin/bash #PBS -l nodes=1:ppn=2 #PBS -l mem=10gb #PBS -l walltime=60:00:00 #PBS -m abe #PBS -q test_queue #PBS -N Cutadapt #PBS -e err.$pbs_jobname."$file".$pbs_jobid #PBS -o out.$pbs_jobname."$file".$pbs_jobid echo [MSG] Start filtering adapters... cd $PBS_O_WORKDIR/RAW # UPDATED SCRIPT #!/bin/bash #PBS -l nodes=1:ppn=2 #PBS -l mem=10gb #PBS -l walltime=60:00:00 #PBS -m abe #PBS -q test_queue #PBS -N Cutadapt ##PBS -e err.$pbs_jobname."$file".$pbs_jobid ##PBS -o out.$pbs_jobname."$file".$pbs_jobid echo [MSG] Start filtering adapters... cd $PBS_O_WORKDIR/RAW module load cutadapt/1.8.1 /home/someone/software/cutadapt-1.8.1/bin/cutadapt $(<cutadapt_truseqpe.conf) -o "$file"_1_temp.fastq -p "$file"_2_temp.fastq "$file"_1.fastq "$file"_2.fastq cutadapt $(<cutadapt_truseqpe.conf) -o "$file"_1_temp.fastq -p "$file"_2_temp.fastq "$file"_1.fastq "$file"_2.fastq # run more commands below # run more commands below echo [MSG] Adapters Filtering Completed! echo [MSG] Adapters Filtering Completed 12

13 Potential / Known issues Software currently installed in the existing HPC may have to be recompiled/reinstalled in the new HPC Existing scripts have to be modified gatk 3.5 specifically for gatk in the new environment, you have to add the following parameter in order to prevent error when calling variants java -Xmx80g -jar $GATK_DIR/GenomeAnalysisTK.jar -T HaplotypeCaller \ -R $ref_file \ -I samplea.bam \ -D $GATK_bundle/dbsnp.vcf \ -o $OUT_GVCFDIR/sampleA.g.vcf \ --emitrefconfidence GVCF \ -log $OUT_GVCFDIR/"$file".g.vcf.log \ --pair_hmm_implementation LOGLESS_CACHING \ -nct 12 \ 13

14 Data backup You own the data and are responsible for your own data backup No centralized data backup solution but we re looking into the feasibility of it as a future service 14

15 What happens to the Existing HPC Will be gradually phased out by Dec 2018 Co-located servers will need to migrate to the new HPC by Dec

16 Moving Forward Better visibility of the HPC Job queues and resources status Capacity expansion $$$ More Compute Nodes More disk capacity New user orientation sessions, seminars and workshops Orientation session will be compulsory for new user Introduction to HPCF / PBS Pro / module technical session on 27 Mar 2018 (10:00am to 12:00pm) Better Utilisation of the HPC 16

17 FAQ Commonly Asked Questions How to apply for a new / removal of user account? Can I share my account with others? What are CGS-HPCF charges? How to apply for addition / reduction on disk quota? What to do when the usual queue / resource doesn t satisfy your analysis job? What is co-location service? 17

18 Requested CPU Number of jobs Good Citizen / Fair Use of resource Most people requested MORE than what they need Used CPU 18

19 Example Usage 8am) jobid username queue jobname ES End Time cpu% mem% wtime% 1577.omics cgs small canu_cre /14/ : omics cgs medium meryl_cre /14/2018 9: statgenpro cgs cgs pbalign 0 11/8/2017 8: statgenpro cgs cgs pbalign_contig 0 11/8/ : statgenpro cgs cgs pbalign_unitig 0 11/8/ : statgenpro cgs cgs pbalign_contig 0 10/31/2017 1: statgenpro cgs cgs pbalign_fl 0 10/31/ : statgenpro cgs cgs pbalign_unitig 0 10/31/2017 1: statgenpro cgs large STARlong 0 9/28/ : statgenpro cgs large STARlong 271 9/28/ : statgenpro cgs large STARlong 0 9/28/ : statgenpro cgs large STARlong 0 9/28/ : statgenpro cgs cgs insertsize 0 4/22/ : statgenpro cgs cgs insertsize 0 4/22/ : statgenpro cgs cgs insertsize -11 4/22/ : statgenpro cgs cgs insertsize 0 4/22/2017 8: statgenpro cgs cgs insertsize 0 4/22/ : statgenpro cgs cgs insertsize -11 4/22/ : statgenpro cgs cgs insertsize 0 4/22/ : statgenpro cgs cgs insertsize 0 4/22/ : statgenpro cgs cgs insertsize 0 4/22/ :

20 Scheduled Maintenance Periodically, usually quarterly, maintenance of the HPCF will be arranged for the following purposes with some downtime OS update / upgrade Patches installation Fine-tuning Health check Housekeeping Hardware upgrade Firmware update Configuration changes 20

21 The day is today hpcf2.cgs.hku.hk ************************ * Welcome to Omics * ************************

22 Questions / Suggestions Website: Wiki: itsupport.cgs@hku.hk

BioHPC Lab at Cornell

BioHPC Lab at Cornell Robert Bukowski (formerly: Computational Biology Service Unit) http://cbsu.tc.cornell.edu/lab/doc/biohpclabintro20130916.pdf (CBSU) Cornell Core Facility providing services for a