CrocoBLAST: Running BLAST Efficiently in the Age of Next-Generation Sequencing

Transcription

1 CrocoBLAST: Running BLAST Efficiently in the Age of Next-Generation Sequencing Ravi José Tristão Ramos, Allan Cézar de Azevedo Martins, Gabriele da Silva Delgado, Crina- Maria Ionescu, Turán Peter Ürményi, Rosane Silva and Jaroslav Koča

2 Fig. S1. Parallelization strategy for improving the performance of BLAST+ The parallelization strategy involves three main processes (white elements): fragmenting the input file into temporary FASTA files containing a smaller number of sequences (fragments); aligning each fragment against the target database using one BLAST+ process; and composing the alignment results for each fragment into the overall output file. Portions of each process are assigned in parallel (gray elements) to the available threads. Thread assignment is periodically evaluated and optimized (black arrows) to ensure both computational efficiency and easy accessibility to partial output.

3 Table S1. Summary of data files and machines used in the benchmark Designation Description Input sequences Proteins_E_coli_k12 E. coli strain K12 proteome, obtained on April 5, 2016 by searching NCBI for proteins, using the search term "Escherichia coli"[porgn: txid562] k12 S_cerevisiae_Genome File SRR containing reads from whole-genome shotgun sequencing of Saccharomyces cerevisiae W303; available from NCBI. Human_microbiome_metagenome File SRS from the Human Microbiome Project, containing reads from shotgun sequencing of a human stool microbiome sample; available from Human_microbiome_16S_RNA File SRR from the Human Microbiome Project containing reads from 16S sequencing of multiple samples of the human microbiome; available from Databases E_coli_P E. coli strain K12 proteome obtained in database format by conversion from Proteins_E_coli_k12 E_coli_G E. coli strain K12 genome (NCBI reference sequence: NC_ ), obtained on April 5, 2016 by searching NCBI genomic databases using the search term E coli. 16S_Micro NCBI database containing bacterial and archaeal 16S ribosomal RNA sequences; available from Machines Desktop Intel Core2 Quad Q9650 (4 cores, 4 threads); 4 GB RAM (DDR 2); ext4 file system on HDD; Ubuntu 16.4 Workstation Intel Core i7-5820k CPU (6 cores, 12 threads); 32.0 GB RAM (DDR 4); ext4 file system on SSD; Ubuntu 16.4 Server 4 x AMD Opteron 6174 (12 cores, 12 threads; total: 48 cores, 48 threads); 128 GB RAM (DDR3); ext4 on HDD; Fedora 20 Cluster node 4 x AMD Opteron 6274 (16 cores, 16 threads; total: 64 cores, 64 threads); 256 GB RAM (DDR 3); nfs4; Ubuntu The input and database files were obtained from data files available on the NCBI ( or Human Microbiome Project pages ( Abbreviations: DDR, double data rate; CPU, central processing unit; HDD, hard disk drive; ext4, fourth extended file system; NCBI, National Center for Biotechnology Information; nfs4, network file system 4; RAM, random access memory; SSD, solid-state drive

4 Fig. S2. Benchmarking strategy for assessing the performance of CrocoBLAST against the performance of BLAST+ The five most common BLAST programs were tested, and publicly available datasets (see also Table S1) were used as input sequences or reference databases, which resulted in 11 case studies. The performance of CrocoBLAST was compared against those of single- and multi-threaded BLAST+ (with the -num_threads option set to the maximum threads achievable for each machine). Each test case was run on four machines. For each case study, the tests were run in the following order: CrocoBLAST, multi-threaded BLAST+, and single-threaded BLAST+. Each test was run in triplicate, with a few minutes of break between jobs, to allow for normalization of CPU temperature and avoid other hardware interference. All benchmark tests were run using BLAST+ version 2.4.0, released by NCBI on June 2 nd, CrocoBLAST always runs BLAST+ in single-thread configuration, but assigns several such calculations to the CPU. For the benchmark tests, CrocoBLAST used a fragment size of 20 KB.

5 Fig. S3. Average runtimes on various machines The columns indicate the average runtime in minutes, with shorter runtime indicating better use of available resources. Results are compared among single-threaded BLAST+ (light gray), multi-threaded BLAST+ with the -num_threads parameter set to the maximum number of threads achievable on each machine (dark gray), and CrocoBLAST+ with the default settings (black). CrocoBLAST always runs BLAST+ in single-thread configuration, but assigns several such calculations to the CPU. An overview of the benchmarking strategy is provided in Fig. S2, while detailed machine specifications and full description of the case studies are available in Table S1. Each test was run in triplicate, and a maximum standard deviation of 1.8% from the mean was noted. Only average values are plotted. * When attempting to run case 2 (a tblastn alignment of the Escherichia coli proteome against the translated E. coli genome), multi-threaded BLAST+ crashed on all machines, whereas CrocoBLAST and singlethreaded BLAST+ ran successfully.

6 Fig. S4. Improvement of CrocoBLAST relative to multi-threaded BLAST+ on various machines Columns indicate speedup provided by CrocoBLAST with default set-tings over multi-threaded BLAST+ (dark gray) with the -num_threads parameter set to the maximum number of threads achievable on each machine (dotted lines). Each test was run in triplicate, and a maximum standard deviation of 1.8% from the mean was noted. Speedup was calculated based on average values. CrocoBLAST always runs BLAST+ in single-thread configuration, but assigns several such calculations to the CPU. An overview of the benchmarking strategy is provided in Fig. S2, while detailed machine specifications and full description of the case studies are available in Table S1. The results regarding case study 2 (a tblastn alignment of the E. coli proteome against the translated E. coli genome) were not included in this assessment because multi-threaded BLAST+ crashed on all machines when attempting to run this case study.

7 Fig. S5. Average CPU usage on various machines The columns indicate CPU usage as the ratio between the CPU capacity assigned to a given calculation and the total CPU capacity available for that machine, averaged over the entire duration of the calculation and expressed as a percentage, with higher CPU usage indicating reduced idle time and better use of available resources. Results are compared among single-threaded BLAST+ (light gray), multi-threaded BLAST+ with the -num_threads parameter set to the maximum number of threads achievable on each machine (dark gray), and CrocoBLAST+ with the default settings (black). CrocoBLAST always runs BLAST+ in single-thread configuration, but assigns several such calculations to the CPU. CPU usage was calculated based on data extracted using a script to parse and log the output of the UNIX command top in order to record the CPU utilization for each second. CPU usage was then calculated with the following formula: Where: 1s_CPU_utilization means the CPU utilization given by the UNIX top command for each

8 second (%CPU column). For CrocoBLAST, process goes through all processes associated with CrocoBLAST, namely crocoblast (input file fragmenter, thread manager, and assembler of final output), CCblast (wrapper for each single-threaded BLAST+ process), CCblast_asbly (assembler of partial output during the alignment stage), and the corresponding BLAST+ processes for that test case (e.g. blastn ). For BLAST+, process indicates the corresponding BLAST+ process for that test case (e.g. blastn ). The time was measured in seconds, from the beginning to the end of each run, registered by the script used to parse the log output. The number of CPUs accounted for both physical and virtual CPUs. An overview of the benchmarking strategy is provided in Fig. S2, while detailed machine specifications and full description of the case studies are available in Table S1. * When attempting to run case 2 (a tblastn alignment of the Escherichia coli proteome against the translated E. coli genome), multi-threaded BLAST+ crashed on all machines, whereas CrocoBLAST and single-threaded BLAST+ ran successfully.

9 Fig. S6. Peak memory usage on various machines The columns indicate the highest memory requirement over the course of each calculation, expressed as a percentage of the total memory of each machine, with lower peak memory indicating reduced requirements and thus the possibility to run said calculation on a less expensive machine. Results are compared among single-threaded BLAST+ (light gray), multi-threaded BLAST+ with the - num_threads parameter set to the maximum number of threads achievable on each machine (dark gray), and CrocoBLAST+ with the default settings (black). CrocoBLAST always runs BLAST+ in singlethread configuration, but assigns several such calculations to the CPU. An overview of the benchmarking strategy is provided in Fig. S2, while detailed machine specifications and full description of the case studies are available in Table S1. In most cases, CrocoBLAST increased the peak memory used during the calculation, but this increase was often negligible (typically, <10% of the total memory for each machine) and did not detrimentally affect performance even at its worst (case 9 on the server, where CrocoBLAST used up to 28% of the total memory); in some cases, CrocoBLAST reduced the peak memory used (case 9 on the desktop and workstation). * When attempting to run case 2 (a tblastn alignment of the E. coli proteome against the translated E. coli genome), multi-threaded BLAST+ crashed on all machines, whereas CrocoBLAST and single-threaded BLAST+ ran successfully.

10 Table S2. I/O statistics when running CrocoBLAST or BLAST+ on each machine Read Write Workstation CrocoBlast Multi-threaded BLAST Single-threaded BLAST CrocoBLAST Multi-threaded BLAST Single-threaded BLAST Read + Write CrocoBLAST Multi-threaded BLAST Single-threaded BLAST Read Write Desktop CrocoBlast Multi-threaded BLAST Single-threaded BLAST CrocoBLAST Multi-threaded BLAST Single-threaded BLAST Read + Write CrocoBLAST Multi-threaded BLAST Single-threaded BLAST Average I/O usage is computed based on the entire run. Each calculation was run in triplicate, and only mean values are shown. Only case studies 1, 7, 9, and 11 were chosen to illustrate I/O statistics, as they were expected to provide a good overview of the typical use of CrocoBLAST regarding database coverage of the query sequences and number of results for each query sequence with a hit. Detailed machine specifications and full description of the case studies are available in Table S1 and Figure S2.

11 Fig. S7. Graphical user interface of CrocoBLAST The work space is organized into three main tabs focused on queue management, job creation, and database management. Full information regarding the running status, queue position, and setup is available for all jobs, along with information regarding the progress of the currently running job (estimated time remaining, percentage completed).