Data Processing and Analysis in Systems Medicine. Milena Kraus Data Management for Digital Health Summer 2017
|
|
- Clifton Davis
- 5 years ago
- Views:
Transcription
1 Milena Kraus Digital Health Summer
2 Agenda Real-world Use Cases Oncology Nephrology Heart Insufficiency Additional Topics Data Management & Foundations Biology Recap Data Sources Data Formats Business Processes Processing and Analysis 2
3 Where are we? Real-world Use Cases Oncology Nephrology Heart Insufficiency Additional Topics Data Management & Foundations Biology Recap Data Sources Data Formats Business Processes Processing and Analysis 3
4 In this lecture you will learn In addition to e.g., genomic data, we will take a deep dive into the Processing and analysis of RNA sequencing data and Differential gene expression analysis. For a better understanding, we will repeat some methods of unsupervised learning: Clustering strategies (Hierarchical, K-Means) and Dimensionality reduction (PCA, MFA). 4
5 Systems Information Levels Clinical information, e.g., imaging data, hemodynamics, lab reports, diagnosis etc. Patient information, e.g., age, gender, environment, history etc. Omics Genome à modeled as categorical values Transcriptome (Gene Expression) Proteome à modeled as numerical values Metabolome 5
6 Introduction to RNA Sequence Analysis Milena Kraus Digital Health Summer
7 Agenda What is RNA sequencing used for? 1. Biological background 2. From wet lab sample to transcriptome a. Experimental procedure b. Raw data c. Processing pipeline(s) d. Downstream analysis 3. Differential gene expression analysis
8 How is a muscle cell different from a liver cell? You ve learned so far: Every cell in your body contains the same DNA as every other cell The DNA codes for every process in the cell How can one cell be different from another if they use the same genetic information? Chart 8
9 What is the difference between a healthy heart and a sick heart? How do cells taken from a healthy and a diseased heart differ? And are those differences treatable, e.g, through drugs? Chart 9
10 Information Content in RNA Information retrieved from RNA: Quantity (primary, How many RNAs are transcribed from a specific gene?) Sequence (secondary, as sequence information is also in DNA) Image source: Chart 10
11 Challenge in RNAseq Alternative Splicing Image Source: National Human Genome Research Institute - Gemeinfrei, Chart 11 commons.wikimedia.org/w/index.php?curid=
12 Experimental Procedure Generally similar to DNA sequencing Over single stranded RNAs in variable abundance (1-k times) of nt Fragmented into nt Differences: RNA is single stranded and needs to be revers-transcribed to DNA for sequencing Coverage is dependent on expression value of gene Chart 12
13 Sequencing Signal Chart 13
14 Differences and Similarities of RNA- and DNAseq Wetlab and Sequencing Procedure DNAseq RNAseq Input molecule All DNA molecules extracted from any collection of cells All RNA molecules of one cell type Preparation Extraction Fragmentation Adapter Ligation Sequencing Output File FASTQ Extraction Fragmentation Reverse Transcription- Adapter Ligation FASTQ 14
15 Raw Data FASTQ HWI-ST155_0742:7:1101:1284:1981/1 NGAGATGAAGCACTGTAGCTTGGAATTCTCGGGTGCCAAGGAACTCCAGT + HWI-ST155_0742:7:1101:2777:1998/1 NGAGATGAAGCACTGTAGCTCTTTGGAATTCTCGGGTGCCAAGGAACTCC + %1=DFFFFHHHHHIIIIIIIIIIIIIIIIIIIIGIIIIIIIIIIIIIIIG Quality score (increasing from worst to best):!"#$%&'()*+,-./ :;<=>?@abcdefghijklmnopqrstuvwxyz[\]^_`abcdefghijklmnopqrstuvwxyz{ Experimental Setup Chart 15
16 Raw Data Reference Genome or Reference Transcriptome FASTA-file >Sequence 1 ;comment A ACAAGATGCCATTGTCCCCCGGCCTCCTGCTGCTGCTGCTCTCCGGGGCCACGGCCACCGCTGCCC TGCCCCTGGAGGGTGGCCCCACCGGCCGAGACAGCGAGCATATGCAGGAAGCGGCAGGAATAAG GAAAAGCAGCCTCCTGACTTTCCTCGCTTGGTGGTTTGAGTGGACCTCCCAGGCCAGTGCCGGGC CCCTCATAGGAGAGGAAGCTCGGGAGGTGGCCAGGCGGCAGGAAGGCGCACCCCCCCAGCAATCC GCGCGCCGGGACAGAATGCCCTGCAGGAACTTCTTCTGGAAGACCTTCTCCTCCTGCAAATAAAA CCTCACCCATGAATGCTCACGCAAGTTTAATTACAGACCTGAA Chart 16
17 Raw Data Gene library 20k-25k protein coding genes representing small part of the genome Using the annotation to speed up processing If the discovery of new genes in a sample is expected, a custom annotation can be calculated from the reads Chart 17
18 Tophat/Cufflinks workflow Traditional Approach Chart 18
19 Selected (Splice-aware) Alignment Tools TopHat: Aligns RNA-Seq reads to mammalian-sized genomes using the ultra highthroughput short read aligner BOWTIE, and then analyzes the mapping results to identify splice junctions between exons. HISAT2: Bowtie + multiple small Graph FM-Indices, parallel threads possible STAR: works by indexing the reference genome first, followed by producing a suffix array index to accelerate the alignment step in further processing. Being capable of running parallel threads on multi-core systems, STAR is faster in comparison with other tools Chart 19
20 Differences and Similarities of RNA- and DNAseq Processing and Output DNAseq RNAseq (traditional) Input Raw Reads, Reference Genome Raw Reads, Reference Genome/ Transcriptome, Gene Library Processing Alignment and variant calling Splice aware alignment and abundance estimation/counting Output Size VCF containing all variants kb to MB size, approx. 30 m variants/human.csv table containing a count/ abundance estimation for all genes/ transcripts/exons kb to MB size, approx k genes/human 20
21 New Fast Algorithms for RNAseq Quantification Sailfish: Facilitates the quantification of RNA-isoform abundance by totally avoiding the time-consuming mapping step. Instead of mapping, it inspects k-mers in reads to observe transcript coverage that results in a fast processing of reads. Kallisto: Same lightweight algorithm approach as Sailfish to quantify transcript abundance but improves it with a pseudoalignment process 21
22 Differential Gene Expression (DGE) Analysis Goal: Identify genes that change in abundance between conditions, i.e., they differ in counts in different conditions. Input: Count table, e.g., GenesxSamples, and design formula Processing: Cleansing, normalization, log-transformation Clustering and PCA Estimates variance-mean dependence in count data Calculates differences in expression values for groups of samples Output: Table containing fold changes 22
23 Differential Gene Expression Analysis Selected Algorithms/Packages In order to perform a statistical test on the differences, the distribution of data needs to be known Statistical methods needed to calculate significant differences Poisson distribution variance and mean of expression values are equal over samples PoissonSeq à only for large data sets Negative binomial distribution - current gold standard as in DESeq and edger as they are applicable for small data sets Abundance estimation via bootstrapping trend as in kallisto/sleuth Chart 23
24 Results of DGE Analysis Common analysis: Principal Component Analysis Volcano and MA plots Clustered Heatmaps Legend: NF Non Failing Heart, (N)ICM (Non-)Ischemic Cardiomyopathy, LVAD Left Ventricular Assisting Device Chart 24
25 Results of DGE Analysis List of differentially expressed genes and a p-value 25
26 Results of DGE Analysis Tissue specificity Chart 26
27 Recap Clustering Milena Kraus Digital Health Summer
28 Introduction/Recap Clustering The goal is to group data points that are similar to each other and identify such groupings in an unsupervised manner For 40% of sequenced genes, functionality cannot be ascertained by comparing to sequences of other known genes But, if genes A and B are grouped in the same microarray cluster, then we hypothesize that proteins A and B might interact with each other and conduct experiments to confirm x x x x x x x x x However, co-expressed genes often don t always imply similar function! Chart 28
29 Example: Gene Expression Data Gene expression data are usually transformed into an intensity matrix (below) The intensity matrix allows biologists to make correlations between different genes (even if they are dissimilar) Make a distance matrix for the distance between every two gene points Genes with a small distance share the same expression characteristics and might be functionally related or similar. Intensity (expression level) of gene at measured time Chart 29
30 Separation Principles There are two key grouping principles to keep in mind Homogeneity: Elements within a cluster are close to each other Separation: Elements in different clusters are further apart from each other Chart 30
31 Summary of Similarity Measures Using different measures for clustering can yield different clusters Common choices for gene expression data: Euclidean distance and correlation distance Euclidean vs Correlation Example g1 = (1,2,3,4,5) g2 = (100,200,300,400,500) g3 = (5,4,3,2,1) Which genes are similar according to the two different measures? 31
32 Hierarchical Clustering Given input set S, the goal is to produce a hierarchy (dendrogram) in which nodes represent subsets of S. The root is the whole input set S. The leaves are the individual elements of S. The internal nodes are defined as the union of their children. Chart 32
33 Hierarchical Clustering Approaches Agglomerative (bottom-up): Beginning with singletons (sets with 1 element) Merging them until S is achieved as the root. Most common approach. Divisive (top-down): Recursively partitioning S until singleton sets are reached. 33
34 Hierarchical Clustering Algorithm Input: a pairwise matrix involved all instances in S 1. Place each instance of S in its own cluster (singleton), creating the list of clusters L (initially, the leaves of T): L= S 1, S 2, S 3,..., S n-1, S n. 2. Compute a merging cost function between every pair of elements in L to find the two closest clusters {S i, S j } which will be the cheapest couple to merge. 3. Remove S i and S j from L. 4. Merge S i and S j to create a new internal node S ij in T which will be the parent of S i and S j in the resulting tree. 5. Go to Step 2 until there is only one set remaining. Chart 34
35 Similarity measures How to determine similarity between data observations? Let x = (x 1,,x n ) and y = (y 1, y n ) be n-dimensional vectors of data points of objects g 1 and g 2 g 1, g 2 can be two different genes in gene expression data n can be the number of samples 35
36 Distance measure l l Euclidean distance Manhattan distance d( g n 2 1, g2) = ( x i y i ) i= 1 l Minkowski distance d( g d n 1, g2) = ( x i y i ) i= 1 n m ( g1, g2) = m ( x i y i ) i= 1 36
37 Correlation distance Correlation distance r xy = Cov( X, Y ) ( Var( X ) Var( Y ) Cov(X,Y) stands for covariance of X and Y degree to which two different variables are related Var(X) stands for variance of X measurement of how a sample differs from their mean 37
38 Correlation distance Correlation r xy = Cov( X, Y ) ( Var( X ) Var( Y ) r = 1 (greens and blues variation is equal) maximum value of 1 if X and Y are perfectly correlated minimum value of -1 if X and Y are exactly opposite distance(x,y) = 1 - r xy 38
39 Hierarchical Clustering Algorithm Input: a pairwise matrix involved all instances in S 1. Place each instance of S in its own cluster (singleton), creating the list of clusters L (initially, the leaves of T): L= S 1, S 2, S 3,..., S n-1, S n. 2. Compute a merging cost function between every pair of elements in L to find the two closest clusters {S i, S j } which will be the cheapest couple to merge. 3. Remove S i and S j from L. 4. Merge S i and S j to create a new internal node S ij in T which will be the parent of S i and S j in the resulting tree. 5. Go to Step 2 until there is only one set remaining. Chart 39
40 Hierarchical Clustering Linkage Strategy Average linkage: The distance between two clusters is the average of the distances between all the points in those clusters. Complete linkage: The distance between two clusters is the distance between the furthest points in those clusters. Single linkage: The distance between two clusters is the distance between the nearest neighbors in those clusters. Chart 40
41 Hierarchical Clustering Dendrograms The algorithm computes a dendrogram which can then be visualized graphically The tree can be pruned to the needed/expected amount of clusters Chart 41
42 Hierarchical Clustering Advantages Dendograms are great for visualization and pruning Provides hierarchical relations between clusters Shown to be able to capture concentric clusters Disadvantages Not easy to define levels for clusters Experiments showed that other clustering techniques outperform hierarchical clustering Chart 42
43 K-Means Clustering Algorithm Input: a pairwise matrix involved all instances in S 1. Randomly place K points into the space represented by the objects that are being clustered. These points represent initial group centroids. 2. Assign each object to the group that has the closest centroid. 3. When all objects have been assigned, recalculate the positions of the K centroids. 4. Repeat Steps 2 and 3 until the stopping criteria is met. Stopping Criteria Convergence (No further changes.) Maximum number of iterations. Or when the squared error is less than some small threshold value α Chart 43
44 K-Means Clustering expression in condition Iteration 1 Iteration 2 Iteration 3 x 1 x 2 x expression in condition expression in condition 1 expression in condition 1 expression in condition 1 x 1 expression in condition 2 x 2 x x 2 x 3 x 1 Chart 44
45 K-Means Clustering Pros: Low complexity Fast Has shown to be efficient in biomedical use cases Disadvantages: Necessity of specifying k Sensitive to noise and outlier data points A small number of skewed data can influence the mean value Clusters are sensitive to initial assignment of centroids K-means is not a deterministic algorithm Clusters can be inconsistent from one run to another Chart 45
46 Evaluation of clusters Why validity of clusters? Given some data, any clustering algorithm generates clusters So we need to make sure the clustering results are valid and meaningful Measuring the validity of clustering results usually involve Optimality of clusters Verification of biological meaning of clusters 46
47 Optimality of clusters Optimal clusters should minimize distance within clusters (intracluster) maximize distance between clusters (intercluster) Example of intracluster measure Squared error se se = i= 1 p c p m where m i is the mean of all instances in cluster c i k i i 2 47
48 Results of DGE Analysis Common analysis: Principal Component Analysis Volcano and MA plots Clustered Heatmaps Legend: NF Non Failing Heart, (N)ICM (Non-)Ischemic Cardiomyopathy, LVAD Left Ventricular Assisting Device Chart 48
49 Recap Dimensionality Reduction Milena Kraus Digital Health Summer
50 Dimensionality Reduction PCA/MCA/MFA The main objective is to sum up and to simplify the data by reducing the dimensionality of the data set. Those methods are used depending on the type of data at hand whether variables are quantitative (numerous) or qualitative (categorical or nominal): Principal component analysis (PCA) when observations are described by quantitative variables Multiple correspondence analysis (MCA) when observations are described by categorical variables Multiple factor analysis (MFA) when observations are described by both numerical and categorical variables 50
51 Principle Component Analysis (PCA) Input from RNAseq analysis Input: Typical data set derived from an RNAseq experiment Gene vs. Sample table (e.g. 20 k genes vs. 20 samples), i.e. high dimensional set of numerical values Gene Sample 1 a 10 8 b 0 2 c d e Sample 2 Output: One (or many) two-dimensional representation of all samples Very good and easy video on PCA on RNAseq data: 51
52 Principle Components Analysis Consider the expression values of 2 samples and various genes. Sample 2 Correlated expression Sample 2 Uncorrelated expression Genes Sample 1 Sample 1 Chart 52
53 Principle Components Analysis Consider the mean of all points m, and a vector B going through the mean The vector B (PC1) is stretched along the path of most variation Sample 2 B Sample 1 Chart 53
54 Principle Components Analysis Consider the mean of all points m, and a vector B going through the mean The vector B (PC1) is stretched along the path of most variation Sample 2 B 2 B 1 Vector B 2 (PC2) is stretched along the path of second most variation and orthogonal to B 1 Length and orientation of the B vectors are most influenced by the outer points Sample 1 Chart 54
55 Principle Components Analysis Length and orientation of the B vectors are most influenced by the outer points Sample 2 d Gene Influence on PC1 Influence on PC2 a b B 2 c e B 1 a High (10) Low (1) b Low (2) Medium (3) c Low (0.5) High (-5) d high (-9) High (5) e High (-10) Low (1) Sample 1 Chart 55
56 Principle Component Analysis Principal component scores for each sample Gene Sample 1 Sample 2 a 10 8 b 0 2 c d e Gene Influence on PC1 a 10 1 b 2 3 c d -9 5 e Influence on PC2 PC score(sample) = (read count (gene a) * PC1_influence (gene a)) + (read count (gene b) * PC1_influence(gene b)) + ) Chart 56
57 Principle Components Analysis Principal component scores for each sample Gene Sample 1 Sample 2 a 10 8 b 0 2 c d e Gene Influence on PC1 a 10 1 b 2 3 c d -9 5 e Influence on PC2 PC score(sample1) =(10* 10) + (0 * 2) + (14 * 0.5) + (33*(-9)) +(50*(-10)) = -690 Chart 57
58 Principle Component Analysis Individuals Factor Map Samples are plotted onto the principal components dimensions Clusters may form to discriminate different groups of samples PC 2 Sample 2 Sample 1 PC 1 PC1 PC2 Sample Sample
59 Principle Component Analysis Variables Factor Map The variables factor map visualizes the influences (loadings) of each gene on the principal components Genes with high influences are good candidates to conduct further analysis on PC 2 Gene e Gene d Gene b Gene c Gene a PC 1 59
60 Principal Component Analysis Diagnostics Scree plot Distribution of variance for principal components The more variance is explained by the first few components, the better the PCA Tail of right-most PC s is mostly noise Cumulated variances add up to 100 % May be used to reduce/prune for further clustering and cleansing of data 60
61 Principal Component Analysis Summary Advantages: Low noise sensitivity Decreased requirements for capacity and memory Large variance = low covariance = high importance à everything else is supposed to be noise and may be removed Disadvantages: Relies on linear assumptions à If the correlation between, e.g., two genes is not linearly correlated, PCA fails Relies on orthogonal transformations à PC s are supposed to be orthogonal to each other, limiting the possibility to find others with higher variance than the orthogonals Scale variant 61
62 Thank you! 62
Statistical Analysis of Metabolomics Data. Xiuxia Du Department of Bioinformatics & Genomics University of North Carolina at Charlotte
Statistical Analysis of Metabolomics Data Xiuxia Du Department of Bioinformatics & Genomics University of North Carolina at Charlotte Outline Introduction Data pre-treatment 1. Normalization 2. Centering,
Hierarchical clustering
Hierarchical clustering Rebecca C. Steorts, Duke University STA 325, Chapter 10 ISL 1 / 63 Agenda K-means versus Hierarchical clustering Agglomerative vs divisive clustering Dendogram (tree) Hierarchical
Gene Clustering & Classification
BINF, Introduction to Computational Biology Gene Clustering & Classification Young-Rae Cho Associate Professor Department of Computer Science Baylor University Overview Introduction to Gene Clustering
Unsupervised Learning. Presenter: Anil Sharma, PhD Scholar, IIIT-Delhi
Unsupervised Learning Presenter: Anil Sharma, PhD Scholar, IIIT-Delhi Content Motivation Introduction Applications Types of clustering Clustering criterion functions Distance functions Normalization Which
Cluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1
Cluster Analysis Mu-Chun Su Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Introduction Cluster analysis is the formal study of algorithms and methods
Differential gene expression analysis using RNA-seq
https://abc.med.cornell.edu/ Differential gene expression analysis using RNA-seq Applied Bioinformatics Core, September/October 2018 Friederike Dündar with Luce Skrabanek & Paul Zumbo Day 3: Counting reads
Exploratory data analysis for microarrays
Exploratory data analysis for microarrays Jörg Rahnenführer Computational Biology and Applied Algorithmics Max Planck Institute for Informatics D-66123 Saarbrücken Germany NGFN - Courses in Practical DNA
CLUSTERING IN BIOINFORMATICS
CLUSTERING IN BIOINFORMATICS CSE/BIMM/BENG 8 MAY 4, 0 OVERVIEW Define the clustering problem Motivation: gene expression and microarrays Types of clustering Clustering algorithms Other applications of
Cluster Analysis for Microarray Data
Cluster Analysis for Microarray Data Seventh International Long Oligonucleotide Microarray Workshop Tucson, Arizona January 7-12, 2007 Dan Nettleton IOWA STATE UNIVERSITY 1 Clustering Group objects that
Clustering CS 550: Machine Learning
Clustering CS 550: Machine Learning This slide set mainly uses the slides given in the following links: http://www-users.cs.umn.edu/~kumar/dmbook/ch8.pdf http://www-users.cs.umn.edu/~kumar/dmbook/dmslides/chap8_basic_cluster_analysis.pdf
CS 2750 Machine Learning. Lecture 19. Clustering. CS 2750 Machine Learning. Clustering. Groups together similar instances in the data sample
Lecture 9 Clustering Milos Hauskrecht milos@cs.pitt.edu 539 Sennott Square Clustering Groups together similar instances in the data sample Basic clustering problem: distribute data into k different groups
Dimension reduction : PCA and Clustering
Dimension reduction : PCA and Clustering By Hanne Jarmer Slides by Christopher Workman Center for Biological Sequence Analysis DTU The DNA Array Analysis Pipeline Array design Probe design Question Experimental
How do microarrays work
Lecture 3 (continued) Alvis Brazma European Bioinformatics Institute How do microarrays work condition mrna cdna hybridise to microarray condition Sample RNA extract labelled acid acid acid nucleic acid
ECLT 5810 Clustering
ECLT 5810 Clustering What is Cluster Analysis? Cluster: a collection of data objects Similar to one another within the same cluster Dissimilar to the objects in other clusters Cluster analysis Grouping
Advanced RNA-Seq 1.5. User manual for. Windows, Mac OS X and Linux. November 2, 2016 This software is for research purposes only.
User manual for Advanced RNA-Seq 1.5 Windows, Mac OS X and Linux November 2, 2016 This software is for research purposes only. QIAGEN Aarhus Silkeborgvej 2 Prismet DK-8000 Aarhus C Denmark Contents 1 Introduction
9/29/13. Outline Data mining tasks. Clustering algorithms. Applications of clustering in biology
9/9/ I9 Introduction to Bioinformatics, Clustering algorithms Yuzhen Ye (yye@indiana.edu) School of Informatics & Computing, IUB Outline Data mining tasks Predictive tasks vs descriptive tasks Example
Cluster analysis. Agnieszka Nowak - Brzezinska
Cluster analysis Agnieszka Nowak - Brzezinska Outline of lecture What is cluster analysis? Clustering algorithms Measures of Cluster Validity What is Cluster Analysis? Finding groups of objects such that
Unsupervised Learning
Outline Unsupervised Learning Basic concepts K-means algorithm Representation of clusters Hierarchical clustering Distance functions Which clustering algorithm to use? NN Supervised learning vs. unsupervised
CS 1675 Introduction to Machine Learning Lecture 18. Clustering. Clustering. Groups together similar instances in the data sample
CS 1675 Introduction to Machine Learning Lecture 18 Clustering Milos Hauskrecht milos@cs.pitt.edu 539 Sennott Square Clustering Groups together similar instances in the data sample Basic clustering problem:
ECS 234: Data Analysis: Clustering ECS 234
: Data Analysis: Clustering What is Clustering? Given n objects, assign them to groups (clusters) based on their similarity Unsupervised Machine Learning Class Discovery Difficult, and maybe ill-posed
CLC Server. End User USER MANUAL
CLC Server End User USER MANUAL Manual for CLC Server 10.0.1 Windows, macos and Linux March 8, 2018 This software is for research purposes only. QIAGEN Aarhus Silkeborgvej 2 Prismet DK-8000 Aarhus C Denmark
ECLT 5810 Clustering
ECLT 5810 Clustering What is Cluster Analysis? Cluster: a collection of data objects Similar to one another within the same cluster Dissimilar to the objects in other clusters Cluster analysis Grouping
Our typical RNA quantification pipeline
RNA-Seq primer Our typical RNA quantification pipeline Upload your sequence data (fastq) Align to the ribosome (Bow>e) Align remaining reads to genome (TopHat) or transcriptome (RSEM) Make report of quality
Cluster Analysis: Agglomerate Hierarchical Clustering
Cluster Analysis: Agglomerate Hierarchical Clustering Yonghee Lee Department of Statistics, The University of Seoul Oct 29, 2015 Contents 1 Cluster Analysis Introduction Distance matrix Agglomerative Hierarchical
An Unsupervised Technique for Statistical Data Analysis Using Data Mining
International Journal of Information Sciences and Application. ISSN 0974-2255 Volume 5, Number 1 (2013), pp. 11-20 International Research Publication House http://www.irphouse.com An Unsupervised Technique
University of Florida CISE department Gator Engineering. Clustering Part 2
Clustering Part 2 Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville Partitional Clustering Original Points A Partitional Clustering Hierarchical
Clustering Techniques
Clustering Techniques Bioinformatics: Issues and Algorithms CSE 308-408 Fall 2007 Lecture 16 Lopresti Fall 2007 Lecture 16-1 - Administrative notes Your final project / paper proposal is due on Friday,
CHAPTER 4: CLUSTER ANALYSIS
CHAPTER 4: CLUSTER ANALYSIS WHAT IS CLUSTER ANALYSIS? A cluster is a collection of data-objects similar to one another within the same group & dissimilar to the objects in other groups. Cluster analysis
Clustering. Lecture 6, 1/24/03 ECS289A
Clustering Lecture 6, 1/24/03 What is Clustering? Given n objects, assign them to groups (clusters) based on their similarity Unsupervised Machine Learning Class Discovery Difficult, and maybe ill-posed
Clustering. Robert M. Haralick. Computer Science, Graduate Center City University of New York
Clustering Robert M. Haralick Computer Science, Graduate Center City University of New York Outline K-means 1 K-means 2 3 4 5 Clustering K-means The purpose of clustering is to determine the similarity
INF4820. Clustering. Erik Velldal. Nov. 17, University of Oslo. Erik Velldal INF / 22
INF4820 Clustering Erik Velldal University of Oslo Nov. 17, 2009 Erik Velldal INF4820 1 / 22 Topics for Today More on unsupervised machine learning for data-driven categorization: clustering. The task
Introduction to Cancer Genomics
Introduction to Cancer Genomics Gene expression data analysis part I David Gfeller Computational Cancer Biology Ludwig Center for Cancer research david.gfeller@unil.ch 1 Overview 1. Basic understanding
Hierarchical Clustering
What is clustering Partitioning of a data set into subsets. A cluster is a group of relatively homogeneous cases or observations Hierarchical Clustering Mikhail Dozmorov Fall 2016 2/61 What is clustering
ROTS: Reproducibility Optimized Test Statistic
ROTS: Reproducibility Optimized Test Statistic Fatemeh Seyednasrollah, Tomi Suomi, Laura L. Elo fatsey (at) utu.fi March 3, 2016 Contents 1 Introduction 2 2 Algorithm overview 3 3 Input data 3 4 Preprocessing
Hard clustering. Each object is assigned to one and only one cluster. Hierarchical clustering is usually hard. Soft (fuzzy) clustering
An unsupervised machine learning problem Grouping a set of objects in such a way that objects in the same group (a cluster) are more similar (in some sense or another) to each other than to those in other
Automated Bioinformatics Analysis System on Chip ABASOC. version 1.1
Automated Bioinformatics Analysis System on Chip ABASOC version 1.1 Phillip Winston Miller, Priyam Patel, Daniel L. Johnson, PhD. University of Tennessee Health Science Center Office of Research Molecular
INF4820 Algorithms for AI and NLP. Evaluating Classifiers Clustering
INF4820 Algorithms for AI and NLP Evaluating Classifiers Clustering Erik Velldal & Stephan Oepen Language Technology Group (LTG) September 23, 2015 Agenda Last week Supervised vs unsupervised learning.
High throughput Data Analysis 2. Cluster Analysis
High throughput Data Analysis 2 Cluster Analysis Overview Why clustering? Hierarchical clustering K means clustering Issues with above two Other methods Quality of clustering results Introduction WHY DO
Gene expression & Clustering (Chapter 10)
Gene expression & Clustering (Chapter 10) Determining gene function Sequence comparison tells us if a gene is similar to another gene, e.g., in a new species Dynamic programming Approximate pattern matching
Gene Expression Data Analysis. Qin Ma, Ph.D. December 10, 2017
1 Gene Expression Data Analysis Qin Ma, Ph.D. December 10, 2017 2 Bioinformatics Systems biology This interdisciplinary science is about providing computational support to studies on linking the behavior
BBS654 Data Mining. Pinar Duygulu. Slides are adapted from Nazli Ikizler
BBS654 Data Mining Pinar Duygulu Slides are adapted from Nazli Ikizler 1 Classification Classification systems: Supervised learning Make a rational prediction given evidence There are several methods for
Dr. Gabriela Salinas Dr. Orr Shomroni Kaamini Rhaithata
Analysis of RNA sequencing data sets using the Galaxy environment Dr. Gabriela Salinas Dr. Orr Shomroni Kaamini Rhaithata Microarray and Deep-sequencing core facility 30.10.2017 RNA-seq workflow I Hypothesis
Cluster Analysis. Ying Shen, SSE, Tongji University
Cluster Analysis Ying Shen, SSE, Tongji University Cluster analysis Cluster analysis groups data objects based only on the attributes in the data. The main objective is that The objects within a group
Network Traffic Measurements and Analysis
DEIB - Politecnico di Milano Fall, 2017 Introduction Often, we have only a set of features x = x 1, x 2,, x n, but no associated response y. Therefore we are not interested in prediction nor classification,
Hierarchical Clustering 4/5/17
Hierarchical Clustering 4/5/17 Hypothesis Space Continuous inputs Output is a binary tree with data points as leaves. Useful for explaining the training data. Not useful for making new predictions. Direction
Chapter 6: Cluster Analysis
Chapter 6: Cluster Analysis The major goal of cluster analysis is to separate individual observations, or items, into groups, or clusters, on the basis of the values for the q variables measured on each
CHAPTER-6 WEB USAGE MINING USING CLUSTERING
CHAPTER-6 WEB USAGE MINING USING CLUSTERING 6.1 Related work in Clustering Technique 6.2 Quantifiable Analysis of Distance Measurement Techniques 6.3 Approaches to Formation of Clusters 6.4 Conclusion
EECS730: Introduction to Bioinformatics
EECS730: Introduction to Bioinformatics Lecture 15: Microarray clustering http://compbio.pbworks.com/f/wood2.gif Some slides were adapted from Dr. Shaojie Zhang (University of Central Florida) Microarray
/ Computational Genomics. Normalization
10-810 /02-710 Computational Genomics Normalization Genes and Gene Expression Technology Display of Expression Information Yeast cell cycle expression Experiments (over time) baseline expression program
Cluster Analysis. Angela Montanari and Laura Anderlucci
Cluster Analysis Angela Montanari and Laura Anderlucci 1 Introduction Clustering a set of n objects into k groups is usually moved by the aim of identifying internally homogenous groups according to a
Classification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University
Classification Vladimir Curic Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University Outline An overview on classification Basics of classification How to choose appropriate
Olmo S. Zavala Romero. Clustering Hierarchical Distance Group Dist. K-means. Center of Atmospheric Sciences, UNAM.
Center of Atmospheric Sciences, UNAM November 16, 2016 Cluster Analisis Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster)
Clustering. CS294 Practical Machine Learning Junming Yin 10/09/06
Clustering CS294 Practical Machine Learning Junming Yin 10/09/06 Outline Introduction Unsupervised learning What is clustering? Application Dissimilarity (similarity) of objects Clustering algorithm K-means,
SYDE Winter 2011 Introduction to Pattern Recognition. Clustering
SYDE 372 - Winter 2011 Introduction to Pattern Recognition Clustering Alexander Wong Department of Systems Design Engineering University of Waterloo Outline 1 2 3 4 5 All the approaches we have learned
Statistics 202: Data Mining. c Jonathan Taylor. Week 8 Based in part on slides from textbook, slides of Susan Holmes. December 2, / 1
Week 8 Based in part on slides from textbook, slides of Susan Holmes December 2, 2012 1 / 1 Part I Clustering 2 / 1 Clustering Clustering Goal: Finding groups of objects such that the objects in a group
CSE 5243 INTRO. TO DATA MINING
CSE 5243 INTRO. TO DATA MINING Cluster Analysis: Basic Concepts and Methods Huan Sun, CSE@The Ohio State University 09/25/2017 Slides adapted from UIUC CS412, Fall 2017, by Prof. Jiawei Han 2 Chapter 10.
Services Performed. The following checklist confirms the steps of the RNA-Seq Service that were performed on your samples.
Services Performed The following checklist confirms the steps of the RNA-Seq Service that were performed on your samples. SERVICE Sample Received Sample Quality Evaluated Sample Prepared for Sequencing
COMP 551 Applied Machine Learning Lecture 13: Unsupervised learning
COMP 551 Applied Machine Learning Lecture 13: Unsupervised learning Associate Instructor: Herke van Hoof (herke.vanhoof@mail.mcgill.ca) Slides mostly by: (jpineau@cs.mcgill.ca) Class web page: www.cs.mcgill.ca/~jpineau/comp551
INF4820, Algorithms for AI and NLP: Hierarchical Clustering
INF4820, Algorithms for AI and NLP: Hierarchical Clustering Erik Velldal University of Oslo Sept. 25, 2012 Agenda Topics we covered last week Evaluating classifiers Accuracy, precision, recall and F-score
Hierarchical Clustering
Hierarchical Clustering Hierarchical Clustering Produces a set of nested clusters organized as a hierarchical tree Can be visualized as a dendrogram A tree-like diagram that records the sequences of merges
Aligners. J Fass 21 June 2017
Aligners J Fass 21 June 2017 Definitions Assembly: I ve found the shredded remains of an important document; put it back together! UC Davis Genome Center Bioinformatics Core J Fass Aligners 2017-06-21
Comparisons and validation of statistical clustering techniques for microarray gene expression data. Outline. Microarrays.
Comparisons and validation of statistical clustering techniques for microarray gene expression data Susmita Datta and Somnath Datta Presented by: Jenni Dietrich Assisted by: Jeffrey Kidd and Kristin Wheeler
Lecture 25: Review I
Lecture 25: Review I Reading: Up to chapter 5 in ISLR. STATS 202: Data mining and analysis Jonathan Taylor 1 / 18 Unsupervised learning In unsupervised learning, all the variables are on equal standing,
Introduction to Computer Science
DM534 Introduction to Computer Science Clustering and Feature Spaces Richard Roettger: About Me Computer Science (Technical University of Munich and thesis at the ICSI at the University of California at
CSE 5243 INTRO. TO DATA MINING
CSE 5243 INTRO. TO DATA MINING Cluster Analysis: Basic Concepts and Methods Huan Sun, CSE@The Ohio State University Slides adapted from UIUC CS412, Fall 2017, by Prof. Jiawei Han 2 Chapter 10. Cluster
Contents. ! Data sets. ! Distance and similarity metrics. ! K-means clustering. ! Hierarchical clustering. ! Evaluation of clustering results
Statistical Analysis of Microarray Data Contents Data sets Distance and similarity metrics K-means clustering Hierarchical clustering Evaluation of clustering results Clustering Jacques van Helden Jacques.van.Helden@ulb.ac.be
Multivariate Analysis
Multivariate Analysis Cluster Analysis Prof. Dr. Anselmo E de Oliveira anselmo.quimica.ufg.br anselmo.disciplinas@gmail.com Unsupervised Learning Cluster Analysis Natural grouping Patterns in the data
Clustering. CE-717: Machine Learning Sharif University of Technology Spring Soleymani
Clustering CE-717: Machine Learning Sharif University of Technology Spring 2016 Soleymani Outline Clustering Definition Clustering main approaches Partitional (flat) Hierarchical Clustering validation
10701 Machine Learning. Clustering
171 Machine Learning Clustering What is Clustering? Organizing data into clusters such that there is high intra-cluster similarity low inter-cluster similarity Informally, finding natural groupings among
Clustering Jacques van Helden
Statistical Analysis of Microarray Data Clustering Jacques van Helden Jacques.van.Helden@ulb.ac.be Contents Data sets Distance and similarity metrics K-means clustering Hierarchical clustering Evaluation
What to come. There will be a few more topics we will cover on supervised learning
Summary so far Supervised learning learn to predict Continuous target regression; Categorical target classification Linear Regression Classification Discriminative models Perceptron (linear) Logistic regression
11/8/2017 Trinity De novo Transcriptome Assembly Workshop trinityrnaseq/rnaseq_trinity_tuxedo_workshop Wiki GitHub
trinityrnaseq / RNASeq_Trinity_Tuxedo_Workshop Trinity De novo Transcriptome Assembly Workshop Brian Haas edited this page on Oct 17, 2015 14 revisions De novo RNA-Seq Assembly and Analysis Using Trinity
Unsupervised Learning and Clustering
Unsupervised Learning and Clustering Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2009 CS 551, Spring 2009 c 2009, Selim Aksoy (Bilkent University)
INF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering
INF4820, Algorithms for AI and NLP: Evaluating Classifiers Clustering Erik Velldal University of Oslo Sept. 18, 2012 Topics for today 2 Classification Recap Evaluating classifiers Accuracy, precision,
Uninformed Search Methods. Informed Search Methods. Midterm Exam 3/13/18. Thursday, March 15, 7:30 9:30 p.m. room 125 Ag Hall
Midterm Exam Thursday, March 15, 7:30 9:30 p.m. room 125 Ag Hall Covers topics through Decision Trees and Random Forests (does not include constraint satisfaction) Closed book 8.5 x 11 sheet with notes
Clustering analysis of gene expression data
Clustering analysis of gene expression data Chapter 11 in Jonathan Pevsner, Bioinformatics and Functional Genomics, 3 rd edition (Chapter 9 in 2 nd edition) Human T cell expression data The matrix contains
Working with Unlabeled Data Clustering Analysis. Hsiao-Lung Chan Dept Electrical Engineering Chang Gung University, Taiwan
Working with Unlabeled Data Clustering Analysis Hsiao-Lung Chan Dept Electrical Engineering Chang Gung University, Taiwan chanhl@mail.cgu.edu.tw Unsupervised learning Finding centers of similarity using
Workload Characterization Techniques
Workload Characterization Techniques Raj Jain Washington University in Saint Louis Saint Louis, MO 63130 Jain@cse.wustl.edu These slides are available on-line at: http://www.cse.wustl.edu/~jain/cse567-08/
Metabolomic Data Analysis with MetaboAnalyst
Metabolomic Data Analysis with MetaboAnalyst User ID: guest6522519400069885256 April 14, 2009 1 Data Processing and Normalization 1.1 Reading and Processing the Raw Data MetaboAnalyst accepts a variety
BGGN-213: FOUNDATIONS OF BIOINFORMATICS (Lecture 14)
BGGN-213: FOUNDATIONS OF BIOINFORMATICS (Lecture 14) Genome Informatics (Part 1) https://bioboot.github.io/bggn213_f17/lectures/#14 Dr. Barry Grant Nov 2017 Overview: The purpose of this lab session is
Unsupervised Learning and Clustering
Unsupervised Learning and Clustering Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2008 CS 551, Spring 2008 c 2008, Selim Aksoy (Bilkent University)
INF 4300 Classification III Anne Solberg The agenda today:
INF 4300 Classification III Anne Solberg 28.10.15 The agenda today: More on estimating classifier accuracy Curse of dimensionality and simple feature selection knn-classification K-means clustering 28.10.15
Clustering. RNA-seq: What is it good for? Finding Similarly Expressed Genes. Data... And Lots of It!
RNA-seq: What is it good for? Clustering High-throughput RNA sequencing experiments (RNA-seq) offer the ability to measure simultaneously the expression level of thousands of genes in a single experiment!
Statistical Methods for Data Mining
Statistical Methods for Data Mining Kuangnan Fang Xiamen University Email: xmufkn@xmu.edu.cn Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on supervised learning
Clustering and Dimensionality Reduction
Clustering and Dimensionality Reduction Some material on these is slides borrowed from Andrew Moore's excellent machine learning tutorials located at: Data Mining Automatically extracting meaning from
Lesson 3. Prof. Enza Messina
Lesson 3 Prof. Enza Messina Clustering techniques are generally classified into these classes: PARTITIONING ALGORITHMS Directly divides data points into some prespecified number of clusters without a hierarchical
Clustering in Data Mining
Clustering in Data Mining Classification Vs Clustering When the distribution is based on a single parameter and that parameter is known for each object, it is called classification. E.g. Children, young,
Clustering: Overview and K-means algorithm
Clustering: Overview and K-means algorithm Informal goal Given set of objects and measure of similarity between them, group similar objects together K-Means illustrations thanks to 2006 student Martin
Machine Learning and Data Mining. Clustering (1): Basics. Kalev Kask
Machine Learning and Data Mining Clustering (1): Basics Kalev Kask Unsupervised learning Supervised learning Predict target value ( y ) given features ( x ) Unsupervised learning Understand patterns of
Data Mining. Dr. Raed Ibraheem Hamed. University of Human Development, College of Science and Technology Department of Computer Science
Data Mining Dr. Raed Ibraheem Hamed University of Human Development, College of Science and Technology Department of Computer Science 06 07 Department of CS - DM - UHD Road map Cluster Analysis: Basic
What is Clustering? Clustering. Characterizing Cluster Methods. Clusters. Cluster Validity. Basic Clustering Methodology
Clustering Unsupervised learning Generating classes Distance/similarity measures Agglomerative methods Divisive methods Data Clustering 1 What is Clustering? Form o unsupervised learning - no inormation
Exercise 1 Review. --outfiltermismatchnmax : max number of mismatch (Default 10) --outreadsunmapped fastx: output unmapped reads
Exercise 1 Review Setting parameters STAR --quantmode GeneCounts --genomedir genomedb -- runthreadn 2 --outfiltermismatchnmax 2 --readfilesin WTa.fastq.gz --readfilescommand zcat --outfilenameprefix WTa
Data Mining Chapter 9: Descriptive Modeling Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University
Data Mining Chapter 9: Descriptive Modeling Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University Descriptive model A descriptive model presents the main features of the data
Long Read RNA-seq Mapper
UNIVERSITY OF ZAGREB FACULTY OF ELECTRICAL ENGENEERING AND COMPUTING MASTER THESIS no. 1005 Long Read RNA-seq Mapper Josip Marić Zagreb, February 2015. Table of Contents 1. Introduction... 1 2. RNA Sequencing...
Hierarchical Clustering Lecture 9
Hierarchical Clustering Lecture 9 Marina Santini Acknowledgements Slides borrowed and adapted from: Data Mining by I. H. Witten, E. Frank and M. A. Hall 1 Lecture 9: Required Reading Witten et al. (2011:
Anaquin - Vignette Ted Wong January 05, 2019
Anaquin - Vignette Ted Wong (t.wong@garvan.org.au) January 5, 219 Citation [1] Representing genetic variation with synthetic DNA standards. Nature Methods, 217 [2] Spliced synthetic genes as internal controls
Chapter DM:II. II. Cluster Analysis
Chapter DM:II II. Cluster Analysis Cluster Analysis Basics Hierarchical Cluster Analysis Iterative Cluster Analysis Density-Based Cluster Analysis Cluster Evaluation Constrained Cluster Analysis DM:II-1
Introduction to GE Microarray data analysis Practical Course MolBio 2012
Introduction to GE Microarray data analysis Practical Course MolBio 2012 Claudia Pommerenke Nov-2012 Transkriptomanalyselabor TAL Microarray and Deep Sequencing Core Facility Göttingen University Medical
Expander 7.2 Online Documentation
Expander 7.2 Online Documentation Introduction... 2 Starting EXPANDER... 2 Input Data... 3 Tabular Data File... 4 CEL Files... 6 Working on similarity data no associated expression data... 9 Working on
Analyzing ICAT Data. Analyzing ICAT Data
Analyzing ICAT Data Gary Van Domselaar University of Alberta Analyzing ICAT Data ICAT: Isotope Coded Affinity Tag Introduced in 1999 by Ruedi Aebersold as a method for quantitative analysis of complex