Decrypting your genome data privately in the cloud
|
|
|
- Merry Parsons
- 6 years ago
- Views:
Transcription
1 Decrypting your genome data privately in the cloud Marc Sitges Data of
2 The Human Genome M (x2) Base pairs (bp) ~ genes (~30%) (Exons ~1%)
3 The Human Genome Project ( )
4 Sequencing cost decrease
5 NEXT GENERATION SEQUENCING (1st) 800bp / poor paralelization (2nd) 32 to 400bp / Extremely paralelized [
6 The Pipeline Big data files generated (not BigData though) SEQUENCING A T T C G T A T C C G T C C A T G T A G High computing requirements MAPPING Results variability VARIANT CALLING
7 [ ]
8 Per individual Per sample Per lane Analysis steps Natively parallel quality control trimming + adaptor removal mapping mapping paired unpaired merge sort by reference remove duplicates indel realignment base recalibration merge remove duplicates indel realignment snp calling snp annotation snp filtering
9 Analysis steps Step Type of computation Memory Usage I/O Disk Usage Quality control Sequence filtering Mapping (lane) Alignment reordering (lane) Remove duplicates (lane) Realignment (lane) Recalibration (lane) SNP calling + basic annotation SNP hard filtering online processing, by input file online processing by paired files, inherently parallel by read exponential problem solved approximately, inherently parallel by read sorting processing by similar chunks, unbalanced trivial parallelization by chromosome cost dependant on the target region, unbalanced trivial parallelization by chromosome cost dependant on the target region, unbalanced trivial parallelization by chromosome processing by regions, statistical analysis online processing Example: exome 30x, input fastq 2 x 1,5 GB (ascii compressed), sam 10 GB (ascii format), bam 2 GB (binary format), reference=3gb (ascii format), vcf=20 MB (ascii format)
10 Current implementation Data Staging Profiling Slurm DB shiftc Aspera Cluster File System Bash wrapper Storage In-house JSON defined workflows Computing nodes Login node
11 Current implementation pitfalls Adhoc in-house workflow design: no portability Different steps have different computing requirements Difficult to install/upgrade or switch between software versions and setups Even if limited pipeline configurability, different libraries already required Not easy restart in case of failure We need a flexible a scalable environment!!
12 MoG model 1. User sequence data (genome, exome). 2. We protect your data indefinitely. 6. The user receives personalized medical care while contributing to society. Pharma & Industry Expert doctors Research 3.User can transfer his or her data for clinical trials. Hybrid Cloud LOPD compliance Informed consent 4. User chooses to purchase an analysis + visit or a medical visit. Researche rs 3rd-party solutions Analytics / Products / Reports Healthcare providers User (genome owner) 5. The health professional can ask for and receive a complementary diagnostic test.
13 First step: workflow definition Main goal: standardize to way we describe analysis tools and workflows to enable portability and reproducibility of data-intensive jobs. Implementations: Common Workflow Language (CWL) Workflow Description Language (WDL) Nextflow s model
14 Second step: task encapsulation in containers A container ships an application with all the environment (including libraries and other dependencies) that the applications needs to run, but shares kernel and more general bins/libraries with the system. Provide close to native performance (IBM Reserach Report, RC25482)
15 User-defined images Software framework which allows to include portable and customized OS environments that accompany the application. Isolate HPC platform from dependencies Easier development cycle and software maintenance Increased flexibility and portability Better version control and compatibility Facilitate reproducible results, pipeline sharing and collaboration Image sharing in public repositories (DockerHub, Quay, Dockstore)
16 Dockerized pipeline 3rd-party solutions Analytics / Products / Reports quality control trimming adaptor removal fastqc-0.11 fastqc-0.15 trimmommatic-0.33 cutadapat mapping bwa-mem samtools-1.2 gem samtools-1.2 bowtie samtools-1.2 merge samtools-1.2 merge sort by reference samtools-1.2 sort remove duplicates picard mark_duplicates indel realignment gatk-1.6 RealignerTargetCreator gatk-1.6 IndelRealigner gatk-3.5 RealignerTargetCreator gatk-3.5 IndelRealigner base recalibration gatk-1.6 CountCovariates gatk-1.6 TableRecalibration gatk-3.5 BaseRecalibrator gatk-3.5 PrintReads snp calling samtools-1.2 mpileup bcftools-1.2 call gatk-1.6 UnifiedGenotyper gatk-3.5 UnifiedGenotyper gatk-3.5 HaplotypeCaller snp annotation SnpSift-4.1 snp filtering bcftools-1.2 filter gatk-1.6 SelectVariants gatk-3.5 VariantRecalibrator
17 Dockerized pipeline 3rd-party solutions Analytics / Products / Reports quality control trimming adaptor removal fastqc-0.11 fastqc-0.15 trimmommatic-0.33 cutadapat mapping bwa-mem samtools-1.2 gem samtools-1.2 bowtie samtools-1.2 merge samtools-1.2 merge sort by reference samtools-1.2 sort remove duplicates picard mark_duplicates indel realignment gatk-1.6 RealignerTargetCreator gatk-1.6 IndelRealigner gatk-3.5 RealignerTargetCreator gatk-3.5 IndelRealigner base recalibration gatk-1.6 CountCovariates gatk-1.6 TableRecalibration gatk-3.5 BaseRecalibrator gatk-3.5 PrintReads snp calling samtools-1.2 mpileup bcftools-1.2 call gatk-1.6 UnifiedGenotyper gatk-3.5 UnifiedGenotyper gatk-3.5 HaplotypeCaller snp annotation SnpSift-4.1 snp filtering bcftools-1.2 filter gatk-1.6 SelectVariants gatk-3.5 VariantRecalibrator
18 Dockerized pipeline 3rd-party solutions Analytics / Products / Reports quality control trimming adaptor removal fastqc-0.11 fastqc-0.15 trimmommatic-0.33 cutadapat mapping bwa-mem samtools-1.2 gem samtools-1.2 bowtie samtools-1.2 merge samtools-1.2 merge sort by reference samtools-1.2 sort remove duplicates picard mark_duplicates indel realignment gatk-1.6 RealignerTargetCreator gatk-1.6 IndelRealigner gatk-3.5 RealignerTargetCreator gatk-3.5 IndelRealigner base recalibration gatk-1.6 CountCovariates gatk-1.6 TableRecalibration gatk-3.5 BaseRecalibrator gatk-3.5 PrintReads snp calling samtools-1.2 mpileup bcftools-1.2 call gatk-1.6 UnifiedGenotyper gatk-3.5 UnifiedGenotyper gatk-3.5 HaplotypeCaller snp annotation SnpSift-4.1 snp filtering bcftools-1.2 filter gatk-1.6 SelectVariants gatk-3.5 VariantRecalibrator
19 Dockerized pipeline 3rd-party solutions Analytics / Products / Reports quality control trimming adaptor removal fastqc-0.11 fastqc-0.15 trimmommatic-0.33 cutadapat mapping bwa-mem samtools-1.2 gem samtools-1.2 bowtie samtools-1.2 merge samtools-1.2 merge sort by reference samtools-1.2 sort remove duplicates picard mark_duplicates indel realignment gatk-1.6 RealignerTargetCreator gatk-1.6 IndelRealigner gatk-3.5 RealignerTargetCreator gatk-3.5 IndelRealigner base recalibration gatk-1.6 CountCovariates gatk-1.6 TableRecalibration gatk-3.5 BaseRecalibrator gatk-3.5 PrintReads snp calling samtools-1.2 mpileup bcftools-1.2 call gatk-1.6 UnifiedGenotyper gatk-3.5 UnifiedGenotyper gatk-3.5 HaplotypeCaller snp annotation SnpSift-4.1 snp filtering bcftools-1.2 filter gatk-1.6 SelectVariants gatk-3.5 VariantRecalibrator
20 Dockerized pipeline 3rd-party solutions Analytics / Products / Reports quality control trimming adaptor removal fastqc-0.11 fastqc-0.15 trimmommatic-0.33 cutadapat mapping bwa-mem samtools-1.2 gem samtools-1.2 bowtie samtools-1.2 merge samtools-1.2 merge sort by reference samtools-1.2 sort remove duplicates picard mark_duplicates indel realignment gatk-1.6 RealignerTargetCreator gatk-1.6 IndelRealigner gatk-3.5 RealignerTargetCreator gatk-3.5 IndelRealigner base recalibration gatk-1.6 CountCovariates gatk-1.6 TableRecalibration gatk-3.5 BaseRecalibrator gatk-3.5 PrintReads snp calling samtools-1.2 mpileup bcftools-1.2 call gatk-1.6 UnifiedGenotyper gatk-3.5 UnifiedGenotyper gatk-3.5 HaplotypeCaller snp annotation SnpSift-4.1 snp filtering bcftools-1.2 filter gatk-1.6 SelectVariants gatk-3.5 VariantRecalibrator
21 Dockerized pipeline 3rd-party solutions Analytics / Products / Reports quality control trimming adaptor removal fastqc-0.11 fastqc-0.15 trimmommatic-0.33 cutadapat mapping bwa-mem samtools-1.2 gem samtools-1.2 bowtie samtools-1.2 merge samtools-1.2 merge sort by reference samtools-1.2 sort remove duplicates picard mark_duplicates indel realignment gatk-1.6 RealignerTargetCreator gatk-1.6 IndelRealigner gatk-3.5 RealignerTargetCreator gatk-3.5 IndelRealigner base recalibration gatk-1.6 CountCovariates gatk-1.6 TableRecalibration gatk-3.5 BaseRecalibrator gatk-3.5 PrintReads snp calling snp annotation samtools-1.2 mpileup bcftools-1.2 call SnpSift-4.1 gatk-1.6 UnifiedGenotyper gatk-3.5 UnifiedGenotyper gatk-3.5 HaplotypeCaller RUN! snp filtering bcftools-1.2 filter gatk-1.6 SelectVariants gatk-3.5 VariantRecalibrator
22 Managing containers in an HPC environment SGE,LSF, others Provide an abstraction layer between the pipeline process and the cluster resource manager.
23 Cloud computing Resources limitation or underuse On demand resources. Pay per use
24 Open vs Personal
25 Open vs Personal
26 Open vs Personal Gender (optional) Phenotype (optional) Disease_site (optional) Sample_Type (optional)
27 Open vs Personal Gender (optional) Phenotype (optional) Disease_site (optional) Sample_Type (optional)
28 Open vs Personal Gender (optional) Phenotype (optional) Disease_site (optional) Sample_Type (optional)
29 Open vs Personal Gender (optional) Phenotype (optional) Disease_site (optional) Sample_Type (optional)
30 Open vs Personal Open data is publicly accessible, but anonymized Phenotypes and metadata is registered on submission Omics datasets and Metadata are (usually) stored in different formats/servers (ie, BAMs on EGA but phenotype in Excel as Supplementary file in journal) Lack of evolution tracking on patients Personal data allows identification of patients Phenotypical data can be collected on-demand Evolution of patients can be followed
31 Sharing vs Compliance
32 Sharing vs Compliance
33 Sharing vs Compliance All in all, the problem is the ownership of the data NO, you cannot do whatever you want with the patient s data YES, the patient can do whatever they want with it According Patient s Autonomy Law, every single clinical record generated in an hospital belongs to the patient. The problem is that 3 rd parties should not decide what to do with recollected data, because, at the end this data belongs to a patient.
34 Sharing vs Compliance Clinical personal data cannot be stored in the cloud But the cloud was convenient and cheap! You should have your own servers in a physically controlled location Data cannot be handed over without acceptance of informed consent But we want other people to collaborate with us!
35 Hybrid cloud
36 Hybrid cloud De-identified copies of the raw and processed genomic data are stored in the public cloud instance Clinical data and personal identifiers are stored in encrypted databases in the private cloud. Raw and processed genomic data is processed and stored in the private storage.
37 Hybrid Cloud SECURITY Data is encrypted in transit and at rest using AES Databases containing PHI and identifiers are also encrypted. Analysts access encrypted data using a host-prof method. Access control mechanism in private cloud Aligned reads data Processed data GB MB De-identied and encrypted genomic data Raw data QC Trimming Mapping Aligned reads Recalibration Remove duplicates Variant calling Processed variants Private Health Information Customer identifiers Aligned reads Processed variants DATA WAREHOUSE Processed variants
38 Hybrid Cloud WebApp Genomic data storage API Computing nodes Genomic data upload service SLURM workload management system Database DMZ Databases with critical information Genomic data storage Computing nodes On demand
39 Cloud bursting VM images repository SGE,LSF, others Could Provider API
40 Cloud bursting VM images repository SGE,LSF, others Could Provider API
41 Cloud bursting VM images repository SGE,LSF, others Could Provider API
42 Cloud bursting! VM images repository SGE,LSF, others Could Provider API
43 Cloud bursting VM images repository SGE,LSF, others Could Provider API
44
45
Practical exercises Day 2. Variant Calling
Practical exercises Day 2 Variant Calling Samtools mpileup Variant calling with samtools mpileup + bcftools Variant calling with HaplotypeCaller (GATK Best Practices) Genotype GVCFs Hard Filtering Variant
Next Generation Sequence Alignment on the BRC Cluster. Steve Newhouse 22 July 2010
Next Generation Sequence Alignment on the BRC Cluster Steve Newhouse 22 July 2010 Overview Practical guide to processing next generation sequencing data on the cluster No details on the inner workings
Reads Alignment and Variant Calling
Reads Alignment and Variant Calling CB2-201 Computational Biology and Bioinformatics February 22, 2016 Emidio Capriotti http://biofold.org/ Institute for Mathematical Modeling of Biological Systems Department
SNP Calling. Tuesday 4/21/15
SNP Calling Tuesday 4/21/15 Why Call SNPs? map mutations, ex: EMS, natural variation, introgressions associate with changes in expression develop markers for whole genome QTL analysis/ GWAS access diversity
Falcon Accelerated Genomics Data Analysis Solutions. User Guide
Falcon Accelerated Genomics Data Analysis Solutions User Guide Falcon Computing Solutions, Inc. Version 1.0 3/30/2018 Table of Contents Introduction... 3 System Requirements and Installation... 4 Software
RPGC Manual. You will also need python 2.7 or above to run our home-brew python scripts.
Introduction Here we present a new approach for producing de novo whole genome sequences--recombinant population genome construction (RPGC)--that solves many of the problems encountered in standard genome
NA12878 Platinum Genome GENALICE MAP Analysis Report
NA12878 Platinum Genome GENALICE MAP Analysis Report Bas Tolhuis, PhD Jan-Jaap Wesselink, PhD GENALICE B.V. INDEX EXECUTIVE SUMMARY...4 1. MATERIALS & METHODS...5 1.1 SEQUENCE DATA...5 1.2 WORKFLOWS......5
REPORT. NA12878 Platinum Genome. GENALICE MAP Analysis Report. Bas Tolhuis, PhD GENALICE B.V.
REPORT NA12878 Platinum Genome GENALICE MAP Analysis Report Bas Tolhuis, PhD GENALICE B.V. INDEX EXECUTIVE SUMMARY...4 1. MATERIALS & METHODS...5 1.1 SEQUENCE DATA...5 1.2 WORKFLOWS......5 1.3 ACCURACY
Preparation of alignments for variant calling with GATK: exercise instructions for BioHPC Lab computers
Preparation of alignments for variant calling with GATK: exercise instructions for BioHPC Lab computers Data used in the exercise We will use D. melanogaster WGS paired-end Illumina data with NCBI accessions
Mapping NGS reads for genomics studies
Mapping NGS reads for genomics studies Valencia, 28-30 Sep 2015 BIER Alejandro Alemán aaleman@cipf.es Genomics Data Analysis CIBERER Where are we? Fastq Sequence preprocessing Fastq Alignment BAM Visualization
Galaxy Platform For NGS Data Analyses
Galaxy Platform For NGS Data Analyses Weihong Yan wyan@chem.ucla.edu Collaboratory Web Site http://qcb.ucla.edu/collaboratory Collaboratory Workshops Workshop Outline ü Day 1 UCLA galaxy and user account
Halvade: scalable sequence analysis with MapReduce
Bioinformatics Advance Access published March 26, 2015 Halvade: scalable sequence analysis with MapReduce Dries Decap 1,5, Joke Reumers 2,5, Charlotte Herzeel 3,5, Pascal Costanza, 4,5 and Jan Fostier
Super-Fast Genome BWA-Bam-Sort on GLAD
1 Hututa Technologies Limited Super-Fast Genome BWA-Bam-Sort on GLAD Zhiqiang Ma, Wangjun Lv and Lin Gu May 2016 1 2 Executive Summary Aligning the sequenced reads in FASTQ files and converting the resulted
PRACTICAL SESSION 5 GOTCLOUD ALIGNMENT WITH BWA JAN 7 TH, 2014 STOM 2014 WORKSHOP HYUN MIN KANG UNIVERSITY OF MICHIGAN, ANN ARBOR
PRACTICAL SESSION 5 GOTCLOUD ALIGNMENT WITH BWA JAN 7 TH, 2014 STOM 2014 WORKSHOP HYUN MIN KANG UNIVERSITY OF MICHIGAN, ANN ARBOR GOAL OF THIS SESSION Assuming that The audiences know how to perform GWAS
Exome sequencing. Jong Kyoung Kim
Exome sequencing Jong Kyoung Kim Genome Analysis Toolkit The GATK is the industry standard for identifying SNPs and indels in germline DNA and RNAseq data. Its scope is now expanding to include somatic
Analysing re-sequencing samples. Malin Larsson WABI / SciLifeLab
Analysing re-sequencing samples Malin Larsson Malin.larsson@scilifelab.se WABI / SciLifeLab Re-sequencing Reference genome assembly...gtgcgtagactgctagatcgaaga...! Re-sequencing IND 1! GTAGACT! AGATCGG!
Analysing re-sequencing samples. Anna Johansson WABI / SciLifeLab
Analysing re-sequencing samples Anna Johansson Anna.johansson@scilifelab.se WABI / SciLifeLab Re-sequencing Reference genome assembly...gtgcgtagactgctagatcgaaga... Re-sequencing IND 1 GTAGACT AGATCGG GCGTAGT
WM2 Bioinformatics. ExomeSeq data analysis part 1. Dietmar Rieder
WM2 Bioinformatics ExomeSeq data analysis part 1 Dietmar Rieder RAW data Use putty to logon to cluster.i med.ac.at In your home directory make directory to store raw data $ mkdir 00_RAW Copy raw fastq
Supplementary Information. Detecting and annotating genetic variations using the HugeSeq pipeline
Supplementary Information Detecting and annotating genetic variations using the HugeSeq pipeline Hugo Y. K. Lam 1,#, Cuiping Pan 1, Michael J. Clark 1, Phil Lacroute 1, Rui Chen 1, Rajini Haraksingh 1,
Hinri Kerstens. NGS pipeline using Broad's Cromwell
Hinri Kerstens NGS pipeline using Broad's Cromwell Introduction Princess Máxima Center is a organization fully specialized in pediatric oncology. By combining the best possible research and care, we will
3. Installation Download Cpipe and Run Install Script Create an Analysis Profile Create a Batch... 7
Cpipe User Guide 1. Introduction - What is Cpipe?... 3 2. Design Background... 3 2.1. Analysis Pipeline Implementation (Cpipe)... 4 2.2. Use of a Bioinformatics Pipeline Toolkit (Bpipe)... 4 2.3. Individual
NGS Analysis Using Galaxy
NGS Analysis Using Galaxy Sequences and Alignment Format Galaxy overview and Interface Get;ng Data in Galaxy Analyzing Data in Galaxy Quality Control Mapping Data History and workflow Galaxy Exercises
Calling variants in diploid or multiploid genomes
Calling variants in diploid or multiploid genomes Diploid genomes The initial steps in calling variants for diploid or multi-ploid organisms with NGS data are the same as what we've already seen: 1. 2.
Galaxy workshop at the Winter School Igor Makunin
Galaxy workshop at the Winter School 2016 Igor Makunin i.makunin@uq.edu.au Winter school, UQ, July 6, 2016 Plan Overview of the Genomics Virtual Lab Introduce Galaxy, a web based platform for analysis
Sequence Mapping and Assembly
Practical Introduction Sequence Mapping and Assembly December 8, 2014 Mary Kate Wing University of Michigan Center for Statistical Genetics Goals of This Session Learn basics of sequence data file formats
Genomic Data Analysis Services Available for PL-Grid Users
Domain-oriented services and resources of Polish Infrastructure for Supporting Computational Science in the European Research Space PLGrid Plus Domain-oriented services and resources of Polish Infrastructure
Package HTSeqGenie. April 16, 2019
Package HTSeqGenie April 16, 2019 Imports BiocGenerics (>= 0.2.0), S4Vectors (>= 0.9.25), IRanges (>= 1.21.39), GenomicRanges (>= 1.23.21), Rsamtools (>= 1.8.5), Biostrings (>= 2.24.1), chipseq (>= 1.6.1),
CORE Year 1 Whole Genome Sequencing Final Data Format Requirements
CORE Year 1 Whole Genome Sequencing Final Data Format Requirements To all incumbent contractors of CORE year 1 WGS contracts, the following acts as the agreed to sample parameters issued by NHLBI for data
Genome 373: Mapping Short Sequence Reads III. Doug Fowler
Genome 373: Mapping Short Sequence Reads III Doug Fowler What is Galaxy? Galaxy is a free, open source web platform for running all sorts of computational analyses including pretty much all of the sequencing-related
Welcome to MAPHiTS (Mapping Analysis Pipeline for High-Throughput Sequences) tutorial page.
Welcome to MAPHiTS (Mapping Analysis Pipeline for High-Throughput Sequences) tutorial page. In this page you will learn to use the tools of the MAPHiTS suite. A little advice before starting : rename your
Variation among genomes
Variation among genomes Comparing genomes The reference genome http://www.ncbi.nlm.nih.gov/nuccore/26556996 Arabidopsis thaliana, a model plant Col-0 variety is from Landsberg, Germany Ler is a mutant
Ensembl RNASeq Practical. Overview
Ensembl RNASeq Practical The aim of this practical session is to use BWA to align 2 lanes of Zebrafish paired end Illumina RNASeq reads to chromosome 12 of the zebrafish ZV9 assembly. We have restricted
Aeromancer: A Workflow Manager for Large- Scale MapReduce-Based Scientific Workflows
Aeromancer: A Workflow Manager for Large- Scale MapReduce-Based Scientific Workflows Presented by Sarunya Pumma Supervisors: Dr. Wu-chun Feng, Dr. Mark Gardner, and Dr. Hao Wang synergy.cs.vt.edu Outline
Kelly et al. Genome Biology (2015) 16:6 DOI /s x. * Correspondence:
Kelly et al. Genome Biology (215) 16:6 DOI 1.1186/s1359-14-577-x METHOD Open Access Churchill: an ultra-fast, deterministic, highly scalable and balanced parallelization strategy for the discovery of human
DRAGEN Bio-IT Platform Enabling the Global Genomic Infrastructure
TM DRAGEN Bio-IT Platform Enabling the Global Genomic Infrastructure About DRAGEN Edico Genome s DRAGEN TM (Dynamic Read Analysis for GENomics) Bio-IT Platform provides ultra-rapid secondary analysis of
NGS Data Analysis. Roberto Preste
NGS Data Analysis Roberto Preste 1 Useful info http://bit.ly/2r1y2dr Contacts: roberto.preste@gmail.com Slides: http://bit.ly/ngs-data 2 NGS data analysis Overview 3 NGS Data Analysis: the basic idea http://bit.ly/2r1y2dr
Helpful Galaxy screencasts are available at:
This user guide serves as a simplified, graphic version of the CloudMap paper for applicationoriented end-users. For more details, please see the CloudMap paper. Video versions of these user guides and
Configuring the Pipeline Docker Container
WES / WGS Pipeline Documentation This documentation is designed to allow you to set up and run the WES/WGS pipeline either on your own computer (instructions assume a Linux host) or on a Google Compute
AgroMarker Finder manual (1.1)
AgroMarker Finder manual (1.1) 1. Introduction 2. Installation 3. How to run? 4. How to use? 5. Java program for calculating of restriction enzyme sites (TaqαI). 1. Introduction AgroMarker Finder (AMF)is
Genomics on Cisco Metacloud + SwiftStack
Genomics on Cisco Metacloud + SwiftStack Technology is a large component of driving discovery in both research and providing timely answers for clinical treatments. Advances in genomic sequencing have
The software comes with 2 installers: (1) SureCall installer (2) GenAligners (contains BWA, BWA- MEM).
Release Notes Agilent SureCall 4.0 Product Number G4980AA SureCall Client 6-month named license supports installation of one client and server (to host the SureCall database) on one machine. For additional
Analyzing Variant Call results using EuPathDB Galaxy, Part II
Analyzing Variant Call results using EuPathDB Galaxy, Part II In this exercise, we will work in groups to examine the results from the SNP analysis workflow that we started yesterday. The first step is
ELPREP PERFORMANCE ACROSS PROGRAMMING LANGUAGES PASCAL COSTANZA CHARLOTTE HERZEEL FOSDEM, BRUSSELS, BELGIUM, FEBRUARY 3, 2018
ELPREP PERFORMANCE ACROSS PROGRAMMING LANGUAGES PASCAL COSTANZA CHARLOTTE HERZEEL FOSDEM, BRUSSELS, BELGIUM, FEBRUARY 3, 2018 USA SAN FRANCISCO USA ORLANDO BELGIUM - HQ LEUVEN THE NETHERLANDS EINDHOVEN
Sequence Analysis Pipeline
Sequence Analysis Pipeline Transcript fragments 1. PREPROCESSING 2. ASSEMBLY (today) Removal of contaminants, vector, adaptors, etc Put overlapping sequence together and calculate bigger sequences 3. Analysis/Annotation
CBSU/3CPG/CVG Joint Workshop Series Reference genome based sequence variation detection
CBSU/3CPG/CVG Joint Workshop Series Reference genome based sequence variation detection Computational Biology Service Unit (CBSU) Cornell Center for Comparative and Population Genomics (3CPG) Center for
NGS Sequence data. Jason Stajich. UC Riverside. jason.stajich[at]ucr.edu. twitter:hyphaltip stajichlab
![NGS Sequence data. Jason Stajich. UC Riverside. jason.stajich[at]ucr.edu. twitter:hyphaltip stajichlab](/thumbs/74/70633133.jpg "NGS Sequence data. Jason Stajich. UC Riverside. jason.stajich[at]ucr.edu. twitter:hyphaltip stajichlab")
NGS Sequence data Jason Stajich UC Riverside jason.stajich[at]ucr.edu twitter:hyphaltip stajichlab Lecture available at http://github.com/hyphaltip/cshl_2012_ngs 1/58 NGS sequence data Quality control
Handling sam and vcf data, quality control
Handling sam and vcf data, quality control We continue with the earlier analyses and get some new data: cd ~/session_3 wget http://wasabiapp.org/vbox/data/session_4/file3.tgz tar xzf file3.tgz wget http://wasabiapp.org/vbox/data/session_4/file4.tgz
Genomes On The Cloud GotCloud. University of Michigan Center for Statistical Genetics Mary Kate Wing Goo Jun
Genomes On The Cloud GotCloud University of Michigan Center for Statistical Genetics Mary Kate Wing Goo Jun Friday, March 8, 2013 Why GotCloud? Connects sequence analysis tools together Alignment, quality
RNAseq analysis: SNP calling. BTI bioinformatics course, spring 2013
RNAseq analysis: SNP calling BTI bioinformatics course, spring 2013 RNAseq overview RNAseq overview Choose technology 454 Illumina SOLiD 3 rd generation (Ion Torrent, PacBio) Library types Single reads
User Guide. SLAMseq Data Analysis Pipeline SLAMdunk on Bluebee Platform
SLAMseq Data Analysis Pipeline SLAMdunk on Bluebee Platform User Guide Catalog Numbers: 061, 062 (SLAMseq Kinetics Kits) 015 (QuantSeq 3 mrna-seq Library Prep Kits) 063UG147V0100 FOR RESEARCH USE ONLY.
Galaxy. Data intensive biology for everyone. / #usegalaxy
Galaxy Data intensive biology for everyone. www.galaxyproject.org @jxtx / #usegalaxy Engineering Dannon Baker Dan Blankenberg Dave Bouvier Nate Coraor Carl Eberhard Jeremy Goecks Sam Guerler Greg von Kuster
Variant calling using SAMtools
Variant calling using SAMtools Calling variants - a trivial use of an Interactive Session We are going to conduct the variant calling exercises in an interactive idev session just so you can get a feel
Dr. Gabriela Salinas Dr. Orr Shomroni Kaamini Rhaithata
Analysis of RNA sequencing data sets using the Galaxy environment Dr. Gabriela Salinas Dr. Orr Shomroni Kaamini Rhaithata Microarray and Deep-sequencing core facility 30.10.2017 RNA-seq workflow I Hypothesis
SweGen: A whole-genome data resource of genetic variability in a cross-section of the Swedish population
Supplementary Material and Methods SweGen: A whole-genome data resource of genetic variability in a cross-section of the Swedish population Adam Ameur, Johan Dahlberg, Pall Olason, Francesco Vezzi, Robert
Accelrys Pipeline Pilot and HP ProLiant servers
Accelrys Pipeline Pilot and HP ProLiant servers A performance overview Technical white paper Table of contents Introduction... 2 Accelrys Pipeline Pilot benchmarks on HP ProLiant servers... 2 NGS Collection
NGS Data Visualization and Exploration Using IGV
1 What is Galaxy Galaxy for Bioinformaticians Galaxy for Experimental Biologists Using Galaxy for NGS Analysis NGS Data Visualization and Exploration Using IGV 2 What is Galaxy Galaxy for Bioinformaticians
NGI-RNAseq. Processing RNA-seq data at the National Genomics Infrastructure. NGI stockholm
NGI-RNAseq Processing RNA-seq data at the National Genomics Infrastructure Phil Ewels phil.ewels@scilifelab.se NBIS RNA-seq tutorial 2017-11-09 SciLifeLab NGI Our mission is to offer a state-of-the-art
Using Galaxy to provide a NGS Analysis Platform
11/15/11 Using Galaxy to provide a NGS Analysis Platform Friedrich Miescher Institute - part of the Novartis Research Foundation - affiliated institute of Basel University - member of Swiss Institute of
ADNI Sequencing Working Group. Robert C. Green, MD, MPH Andrew J. Saykin, PsyD Arthur Toga, PhD
ADNI Sequencing Working Group Robert C. Green, MD, MPH Andrew J. Saykin, PsyD Arthur Toga, PhD Why sequencing? V V V V V V V V V V V V V A fortuitous relationship TIME s Best Invention of 2008 The initial
Analyzing massive genomics datasets using Databricks Frank Austin Nothaft,
Analyzing massive genomics datasets using Databricks Frank Austin Nothaft, PhD frank.nothaft@databricks.com @fnothaft VISION Accelerate innovation by unifying data science, engineering and business PRODUCT
DNA Sequencing analysis on Artemis
DNA Sequencing analysis on Artemis Mapping and Variant Calling Tracy Chew Senior Research Bioinformatics Technical Officer Rosemarie Sadsad Informatics Services Lead Hayim Dar Informatics Technical Officer
Security Overview of the BGI Online Platform
WHITEPAPER 2015 BGI Online All rights reserved Version: Draft v3, April 2015 Security Overview of the BGI Online Platform Data security is, in general, a very important aspect in computing. We put extra
Massively Parallel Processing of Whole Genome Sequence Data: An In-Depth Performance Study
Massively Parallel Processing of Whole Genome Sequence Data: An In-Depth Performance Study Abhishek Roy, Yanlei Diao, Uday Evani, Avinash Abhyankar Clinton Howarth, Rémi Le Priol, Toby Bloom University
Tumor-Specific NeoAntigen Detector (TSNAD) v2.0 User s Manual
Tumor-Specific NeoAntigen Detector (TSNAD) v2.0 User s Manual Zhan Zhou, Xingzheng Lyu and Jingcheng Wu Zhejiang University, CHINA March, 2016 USER'S MANUAL TABLE OF CONTENTS 1 GETTING STARTED... 1 1.1
elprep: a high- performance tool for preparing SAM/BAM files for variant calling Charlo<e Herzeel (Imec) Pascal Costanza (Intel) July 2014
elprep: a high- performance tool for preparing SAM/BAM files for variant calling Charlo
Powering Knowledge Discovery. Insights from big data with Linguamatics I2E
Powering Knowledge Discovery Insights from big data with Linguamatics I2E Gain actionable insights from unstructured data The world now generates an overwhelming amount of data, most of it written in natural
Introduction to NGS analysis on a Raspberry Pi. Beta version 1.1 (04 June 2013)
Introduction to NGS analysis on a Raspberry Pi Beta version 1.1 (04 June 2013)!! Contents Overview Contents... 3! Overview... 4! Download some simulated reads... 5! Quality Control... 7! Map reads using
Intro to NGS Tutorial
Intro to NGS Tutorial Release 8.6.0 Golden Helix, Inc. October 31, 2016 Contents 1. Overview 2 2. Import Variants and Quality Fields 3 3. Quality Filters 10 Generate Alternate Read Ratio.........................................
Evaluate NimbleGen SeqCap RNA Target Enrichment Data
Roche Sequencing Technical Note November 2014 How To Evaluate NimbleGen SeqCap RNA Target Enrichment Data 1. OVERVIEW Analysis of NimbleGen SeqCap RNA target enrichment data generated using an Illumina
ITMO Ecole de Bioinformatique Hands-on session: smallrna-seq N. Servant 21 rd November 2013
ITMO Ecole de Bioinformatique Hands-on session: smallrna-seq N. Servant 21 rd November 2013 1. Data and objectives We will use the data from GEO (GSE35368, Toedling, Servant et al. 2011). Two samples were
How to store and visualize RNA-seq data
How to store and visualize RNA-seq data Gabriella Rustici Functional Genomics Group gabry@ebi.ac.uk EBI is an Outstation of the European Molecular Biology Laboratory. Talk summary How do we archive RNA-seq
Sentieon DNA Pipeline for Variant Detection Software-only solution, over 20 faster than GATK 3.3 with identical results
Sentieon DNA Pipeline for Variant Detection Software-only solution, over 20 faster than GATK 3.3 with identical results Jessica A. Weber 1, Rafael Aldana 5, Brendan D. Gallagher 5, Jeremy S. Edwards 2,3,4
Sentieon Documentation
Sentieon Documentation Release 201808.03 Sentieon, Inc Dec 21, 2018 Sentieon Manual 1 Introduction 1 1.1 Description.............................................. 1 1.2 Benefits and Value..........................................
Financed by the European Commission 7 th Framework Programme. biobankcloud.eu. Jim Dowling, PhD Assoc. Prof, KTH Project Coordinator
Financed by the European Commission 7 th Framework Programme. biobankcloud.eu Jim Dowling, PhD Assoc. Prof, KTH Project Coordinator The Biobank Bottleneck We will soon be generating massive amounts of
Contact: Raymond Hovey Genomics Center - SFS
Bioinformatics Lunch Seminar (Summer 2014) Every other Friday at noon. 20-30 minutes plus discussion Informal, ask questions anytime, start discussions Content will be based on feedback Targeted at broad
Sentieon DNA pipeline for variant detection - Software-only solution, over 20 faster than GATK 3.3 with identical results
Sentieon DNA pipeline for variant detection - Software-only solution, over 0 faster than GATK. with identical results Sentieon DNAseq Software is a suite of tools for running DNA sequencing secondary analyses.
Modern Data Warehouse The New Approach to Azure BI
Modern Data Warehouse The New Approach to Azure BI History On-Premise SQL Server Big Data Solutions Technical Barriers Modern Analytics Platform On-Premise SQL Server Big Data Solutions Modern Analytics
CloudExpo November 2017 Tomer Levi
CloudExpo November 2017 Tomer Levi About me Full Stack Engineer @ Intel s Advanced Analytics group. Artificial Intelligence unit at Intel. Responsible for (1) Radical improvement of critical processes
Illumina Next Generation Sequencing Data analysis
Illumina Next Generation Sequencing Data analysis Chiara Dal Fiume Sr Field Application Scientist Italy 2010 Illumina, Inc. All rights reserved. Illumina, illuminadx, Solexa, Making Sense Out of Life,
RNA-seq. Manpreet S. Katari
RNA-seq Manpreet S. Katari Evolution of Sequence Technology Normalizing the Data RPKM (Reads per Kilobase of exons per million reads) Score = R NT R = # of unique reads for the gene N = Size of the gene
In-Memory Technology in Life Sciences
in Life Sciences Dr. Matthieu-P. Schapranow In-Memory Database Applications in Healthcare 2016 Apr Intelligent Healthcare Networks in the 21 st Century? Hospital Research Center Laboratory Researcher Clinician
Bioinformatics in next generation sequencing projects
Bioinformatics in next generation sequencing projects Rickard Sandberg Assistant Professor Department of Cell and Molecular Biology Karolinska Institutet March 2011 Once sequenced the problem becomes computational
Read Mapping and Variant Calling
Read Mapping and Variant Calling Whole Genome Resequencing Sequencing mul:ple individuals from the same species Reference genome is already available Discover varia:ons in the genomes between and within
Under the Hood of Alignment Algorithms for NGS Researchers
Under the Hood of Alignment Algorithms for NGS Researchers April 16, 2014 Gabe Rudy VP of Product Development Golden Helix Questions during the presentation Use the Questions pane in your GoToWebinar window
CIS : Computational Reproducibility
CIS 602-01: Computational Reproducibility Containers Dr. David Koop Virtual Machines Software Abstraction - Behaves like hardware - Encapsulates all OS and application state Virtualization Layer - Extra
Faculté Polytechnique
Faculté Polytechnique INFORMATIQUE PARALLÈLE ET DISTRIBUÉE CHAPTER 7 : CLOUD COMPUTING Sidi Ahmed Mahmoudi sidi.mahmoudi@umons.ac.be 13 December 2017 PLAN Introduction I. History of Cloud Computing and
Applying In-Memory Technology to Genome Data Analysis
Applying In-Memory Technology to Genome Data Analysis Cindy Fähnrich Hasso Plattner Institute GLOBAL HEALTH 14 Tutorial Hasso Plattner Institute Key Facts Founded as a public-private partnership in 1998
Bioinformatics Framework
Persona: A High-Performance Bioinformatics Framework Stuart Byma 1, Sam Whitlock 1, Laura Flueratoru 2, Ethan Tseng 3, Christos Kozyrakis 4, Edouard Bugnion 1, James Larus 1 EPFL 1, U. Polytehnica of Bucharest
Galaxy. Daniel Blankenberg The Galaxy Team
Galaxy Daniel Blankenberg The Galaxy Team http://galaxyproject.org Overview What is Galaxy? What you can do in Galaxy analysis interface, tools and datasources data libraries workflows visualization sharing
Resequencing Analysis. (Pseudomonas aeruginosa MAPO1 ) Sample to Insight
Resequencing Analysis (Pseudomonas aeruginosa MAPO1 ) 1 Workflow Import NGS raw data Trim reads Import Reference Sequence Reference Mapping QC on reads Variant detection Case Study Pseudomonas aeruginosa
Pre-processing and quality control of sequence data. Barbera van Schaik KEBB - Bioinformatics Laboratory
Pre-processing and quality control of sequence data Barbera van Schaik KEBB - Bioinformatics Laboratory b.d.vanschaik@amc.uva.nl Topic: quality control and prepare data for the interesting stuf Keep Throw
SolexaLIMS: A Laboratory Information Management System for the Solexa Sequencing Platform
SolexaLIMS: A Laboratory Information Management System for the Solexa Sequencing Platform Brian D. O Connor, 1, Jordan Mendler, 1, Ben Berman, 2, Stanley F. Nelson 1 1 Department of Human Genetics, David
INTRODUCTION TO NEXTFLOW
INTRODUCTION TO NEXTFLOW Paolo Di Tommaso, CRG NETTAB workshop - Roma October 25th, 2016 @PaoloDiTommaso Research software engineer Comparative Bioinformatics, Notredame Lab Center for Genomic Regulation
PRACTICAL SESSION 8 SEQUENCE-BASED ASSOCIATION, INTERPRETATION, VISUALIZATION USING EPACTS JAN 7 TH, 2014 STOM 2014 WORKSHOP
PRACTICAL SESSION 8 SEQUENCE-BASED ASSOCIATION, INTERPRETATION, VISUALIZATION USING EPACTS JAN 7 TH, 2014 STOM 2014 WORKSHOP HYUN MIN KANG UNIVERSITY OF MICHIGAN, ANN ARBOR EPACTS ASSOCIATION ANALYSIS
Maximizing Public Data Sources for Sequencing and GWAS
Maximizing Public Data Sources for Sequencing and GWAS February 4, 2014 G Bryce Christensen Director of Services Questions during the presentation Use the Questions pane in your GoToWebinar window Agenda
Lecture 12. Short read aligners
Lecture 12 Short read aligners Ebola reference genome We will align ebola sequencing data against the 1976 Mayinga reference genome. We will hold the reference gnome and all indices: mkdir -p ~/reference/ebola
Content-based Comparison for Collections Identification
Content-based Comparison for Collections Identification Weijia Xu1, Ruizhu Huang1, Maria Esteva1, Jawon Song1, Ramona Walls2, 1 Texas Advanced Computing Center, University of Texas at Austin 2 Cyverse.org
RNA-Seq in Galaxy: Tuxedo protocol. Igor Makunin, UQ RCC, QCIF
RNA-Seq in Galaxy: Tuxedo protocol Igor Makunin, UQ RCC, QCIF Acknowledgments Genomics Virtual Lab: gvl.org.au Galaxy for tutorials: galaxy-tut.genome.edu.au Galaxy Australia: galaxy-aust.genome.edu.au
New High Performance Computing Cluster For Large Scale Multi-omics Data Analysis. 28 February 2018 (Wed) 2:30pm 3:30pm Seminar Room 1A, G/F
New High Performance Computing Cluster For Large Scale Multi-omics Data Analysis 28 February 2018 (Wed) 2:30pm 3:30pm Seminar Room 1A, G/F The Team (Bioinformatics & Information Technology) Eunice Kelvin
Feed the Future Innovation Lab for Peanut (Peanut Innovation Lab) Data Management Plan Version:
Feed the Future Innovation Lab for Peanut (Peanut Innovation Lab) Data Management Plan Version: 20180316 Peanut Innovation Lab Management Entity The University of Georgia, Athens, Georgia Feed the Future