Unsupervised Learning. Unsupervised Learning. What is Clustering? Unsupervised Learning I Clustering 9/7/2017. Clustering

Transcription

1 Unsupervised Learning Clustering Centroid models (K-mean) Connectivity models (hierarchical clustering) Density models (DBSCAN) Graph-based models Subspace models (Biclustering) Feature extraction techniques Unsupervised Learning Feature Extraction Techniques Principal component analysis Independent component analysis Singular value decomposition Feature Selection Techniques What is Clustering? Unsupervised Learning I Clustering Clustering can be considered the most important unsupervised learning problem; so, as every other problem of this kind, it deals with finding a structure in a collection of unlabeled data. A loose definition of clustering could be the process of organizing objects into groups whose members are similar in some way. A cluster is therefore a collection of objects which are similar between them and are dissimilar to the objects belonging to other clusters. 1

2 Cluster analysis deals with separating data into groups whose identities are not known in advance. In general, even the correct number of groups into which the data should be sorted is not known in advance. Daniel S. Wilks s of use of cluster analysis in weather and climate literature: Market Segmentation othe market is divided into smaller groups that are more homogeneous needs. Taxonomy Clustering odifferentiation between different species of animals or plants according to their physical similarity What is Cluster Analysis? Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups Intra-cluster distances are minimized Inter-cluster distances are maximized Microarray Clustering s Centroid Models Given a K, find a partition of K clusters to optimize the chosen partitioning criterion. Global optimal: exhaustively enumerate all partitions Heuristic method: K-means algorithm (MacQueen 67): each cluster is represented by the centre of the cluster and the algorithm converges to stable centers of clusters

3 K-mean Algorithm Given the cluster number K, the K-means algorithm is carried out in three steps: Initialisation: set seed points Assign each object to the cluster with the nearest seed point Compute seed points as the centroids of the clusters of the current partition (the centroid is the centre, i.e., mean point, of the cluster) Go back to Step 1), stop when no more new assignment Within and Between Cluster Criteria Let s consider total point scatter for a set of N data points: T can be re-written as: Where, Within cluster scatter N N 1 T d( x i, x j ) i1 j1 K 1 T ( d( xi, x j ) d( xi, x j )) k1 C( i) k C( j) k C( j) k W ( C) B( C) K 1 W ( C) d( x i, x j ) k1 C( i) k C( j) k K 1 B( C) d( x i, x j ) k1 C( i) k C( j) k Between cluster scatter Distance between two points If d is square Euclidean distance, then and K W ( C) Nk xi mk k1 C( i) k K B( C) Nk mk m k1 Minimizing W(C) is equivalent to maximizing B(C) Grand mean webdocs.cs.ualberta.ca/~nray1/cmput466_551/clustering.ppt Problem Suppose we have 4 types of medicines and each has two attributes (ph and weight index). Our goal is to group these objects into K= group of medicine. Medicine Weight ph- Index A 1 1 B 1 C 4 3 D 5 4 A B C D Step 1: Use initial seed points for partitioning A B C D d( D, c ) 1 d( D, c ) c A, c 1 B Euclidean distance (5 1) (4 1) 5 (5 ) (4 1) 4.4 Assign each object to the cluster with the nearest seed point webdocs.cs.ualberta.ca/~nray1/cmput466_551/clustering.ppt 1 3

4 Step : Compute new centroids of the current partition c (1, 1) c, 3 3 (11/ 3, 8 / 3) (3.67,.67) Step 3: Repeat the first two steps until its convergence Knowing the members of each cluster, now we compute the new centroid of each group based on these new memberships c1, (1, 1) c, (4, 3 ) Step 3: Repeat the first two steps until its convergence Compute the distance of all objects to the new centroids Exercise For the medicine data set, use K-means with the distance metric for clustering analysis by setting K= and initializing seeds as C1 = A and C = C. Answer three questions as follows: 1. How many steps are needed for convergence?. What are memberships of two clusters? 3. What are centroids of two clusters after convergence? Medicine Weight ph- Index A 1 1 C D Stop due to no new assignment B 1 A B C 4 3 D

5 How K-mean partitions? K-means Demo When K centroids are set/fixed, they partition the whole space into K subspaces constituting a partitioning. 1. User set up the number of clusters they d like. (e.g. k=5) Changing positions of centroids leads to a new partitioning. A partitioning amounts to a Voronoi Diagram K-means Demo 1. User set up the number of clusters they d like. (e.g. K=5). Randomly guess K cluster Center locations K-means Demo 1. User set up the number of clusters they d like. (e.g. K=5). Randomly guess K cluster Center locations 3. Each data point finds out which Center it s closest to. (Thus each Center owns a set of data points) Voronoi Diagram

6 K-means Demo K-means Demo 1. User set up the number of clusters they d like. (e.g. K=5). Randomly guess K cluster centre locations 3. Each data point finds out which centre it s closest to. (Thus each Center owns a set of data points) 4. Each centre finds the centroid of the points it owns 1. User set up the number of clusters they d like. (e.g. K=5). Randomly guess K cluster centre locations 3. Each data point finds out which centre it s closest to. (Thus each centre owns a set of data points) 4. Each centre finds the centroid of the points it owns 5. and jumps there 1 K-means Demo 1. User set up the number of clusters they d like. (e.g. K=5). Randomly guess K cluster centre locations 3. Each data point finds out which centre it s closest to. (Thus each centre owns a set of data points) 4. Each centre finds the centroid of the points it owns 5. and jumps there 6. Repeat until terminated! Relevant Issues Efficient in computation O(tKn), where n is number of objects, K is number of clusters, and t is number of iterations. Normally, K, t << n. Local optimum sensitive to initial seed points converge to a local optimum that may be unwanted solution Other problems Need to specify K, the number of clusters, in advance Unable to handle noisy data and outliers (K-Medoids algorithm) 3 4 6

7 Relevant Issues Cluster Validity With different initial conditions, the K-means algorithm may result in different partitions for a given data set. Which partition is the best one for the given data set? In theory, no answer to this question as there is no ground-truth available in unsupervised learning (ill-posed problem!) Nevertheless, there are several cluster validity criteria to assess the quality of clustering analysis from different perspectives A common cluster validity criterion is the ratio of the total between-cluster to the total within-cluster distances Between-cluster distance (BCD): the distance between means of two clusters Within-cluster distance (WCD): sum of all distance between data points and the mean in a specific cluster A large ratio of BCD/WCD suggests good compactness inside clusters and good separability among different clusters! 5 K-means Limitations of K-means: Non-globular Shapes Original Points K-means ( Clusters) Conclusion K-means algorithm is a simple yet popular method for clustering analysis Its performance is determined by initialisation and appropriate distance measure There are several variants of K-means to overcome its weaknesses K-Medoids: resistance to noise and/or outliers K-Modes: extension to categorical data clustering analysis CLARA: dealing with large data sets Mixture models (EM algorithm): handling uncertainty of clusters 8 7

8 Hierarchical Clustering Hierarchical Clustering Approach A typical clustering analysis approach via partitioning data set sequentially Construct nested partitions layer by layer via grouping objects into a tree of clusters (without the need to know the number of clusters in advance) Uses distance matrix as clustering criteria Agglomerative vs. Divisive Two sequential clustering strategies for constructing a tree of clusters Agglomerative: a bottom-up strategy Initially each data object is in its own (atomic) cluster Then merge these clusters into larger and larger clusters Divisive: a top-down strategy Initially all objects are in one single cluster Then the cluster is subdivided into smaller and smaller clusters 9 Phylogenetic tree: A tree represents graphical relation between organisms, species, or genomic sequence In Bioinformatics, it s based on genomic sequence Phylogenetic Tree An Hierarchical Clustering Root: origin of evolution Leaves: current organisms, species, or genomic sequence (MSA) Branches: relationship between organisms, species, or genomic sequence Branch length: evolutionary time 8

9 Illustrative Introduction Agglomerative and divisive clustering on the data set {a, b, c, d,e } Step 0 Step 1 Step Step 3 Step 4 a b c d e a b d e c d e a b c d e Step 4 Step 3 Step Step 1 Step 0 Agglomerative Cluster distance Termination condition Divisive Cluster Distance Measures Single link: smallest distance between an element in one cluster and an element in the other, i.e., d(c i, C j ) = min{d(x ip, x jq )} Complete link: largest distance between an element in one cluster and an element in the other, i.e., d(c i, C j ) = max{d(x ip, x jq )} Average (UPGMA: Unweighted Pair Group Method with Arithmetic Mean): Avg distance between elements in one cluster and elements in the other, i.e., d(c i, C j ) = avg{d(x ip, x jq )} single link (min) complete link (max) average Cluster Distance Measures : Given a data set of five objects characterised by a single feature, assume that there are two clusters: C1: {a, b} and C: {c, d, e}. a b c d e Feature Calculate the distance matrix.. Calculate three cluster distances between C1 and C. Single link a b c d e dist(c 1,C) min{ d(a,c), d(a,d), d(a,e), d(b,c), d(b,d), d(b,e)} a min{3, 4, 5,, 3, 4} b c d e Complete link dist(c1,c) max{ d(a,c), d(a,d), d(a,e), d(b,c), d(b,d), d(b,e)} max{3,4, 5,, 3, 4} 5 Average d(a,c) d(a,d) d(a,e) d(b,c) d(b,d) d(b,e) dist(c1,c) Agglomerative Algorithm The Agglomerative algorithm is carried out in three steps: 1) Convert object attributes to distance matrix ) Set each object as a cluster (thus if we have N objects, we will have N clusters at the beginning) 3) Repeat until number of cluster is one (or known # of clusters) Merge two closest clusters Update distance matrix 36 9

10 Problem: clustering analysis with agglomerative algorithm Merge two closest clusters (iteration 1) data matrix Euclidean distance distance matrix Update distance matrix (iteration 1) Merge two closest clusters (iteration )

11 Update distance matrix (iteration ) Merge two closest clusters/update distance matrix (iteration 3) 41 4 Merge two closest clusters/update distance matrix (iteration 4) Final result (meeting termination condition)

12 lifetime Dendrogram tree representation object 3 1. In the beginning we have 6 clusters: A, B, C, D, E and F. We merge clusters D and F into cluster (D, F) at distance We merge cluster A and cluster B into (A, B) at distance We merge clusters E and (D, F) into ((D, F), E) at distance We merge clusters ((D, F), E) and C into (((D, F), E), C) at distance We merge clusters (((D, F), E), C) and (A, B) into ((((D, F), E), C), (A, B)) at distance The last cluster contain all the objects, thus conclude the computation Different Dendograms 45 Phylogenetic Trees can be: Rooted: A single node is designated as root and it represents a common ancestor with a unique path leading from it through evolutionary time to any other node Unrooted tree: specifies only the nodes interrelations but says nothing about the direction in which evolution occurred. Rooted trees Roots can be artificially assigned to unrooted trees by means of an outgroup. An outgroup is a species that have unambiguously separated early from the other species being considered Unrooted trees 1

13 Phylogenetic Tree Calculate all the distance between leaves (taxa & MSA) Based on the distance, construct a tree Good for continuous characters Not very accurate Fastest method UPGMA (average linkage) Neighbor-joining 1 n 1, and 3 are known species. We know d 1, d 13, and d 3. Neighbor-joining Want to add an internal node n to the tree such that distances between species are equal to the sum of the lengths of the branches along the path between the species. d 1n + d n = d 1 ; d 1n + d 3n = d 13 ; d n + d 3n = d 3 ; Solve simultaneous equations 3 d 1n = (d 1 + d 13 d 3 )/; d n = (d 1 + d 3 d 13 )/; d 3n = (d 13 + d 3 d 1 )/; How to construct a tree with Neighbor-joining method? Step 1: Calculate sum all distance from x and divide by (leaves ) Sx = (sum all Dx) / (leaves - ) Step : Calculate pair with smallest M Mij = Distance ij Si Sj Step 3: Create a node U that joins pair with lowest Mij SiU = (Dij / ) + (Si Sj) / How to construct a tree with Neighbor-joining method? Step 4: Join i and j according to S and make all other taxa in form of a star Step 5: Recalculate new distance matrix of all other taxa to U with: DxU = (Dix + Djx Dij)/

14 53 of Neighbor-joining B 5 A B C D E C 4 7 D E F Step 1: S calculation : Sx = (sum all Dx) / (leaves - ) S(A) = ( ) / 4 = 7.5 S(B) = ( ) / 4 = 10.5 S(C) = ( ) / 4 = 8 S(D) = ( ) / 4 = 9.5 S(E) = ( ) / 4 = 8.5 S(F) = ( ) / 4 = 11 of Neighbor-joining cont 1 Step : Calculate pair with smallest M Mij = Distance ij Si Sj Smallest are M(AB) = d(ab) S(A) S(B) = = -13 M(DE) = = B -13 A B C D E C D E F of Neighbor-joining cont Step 3: Create a node U S1U = (Dij / ) + (Si Sj) / U1 joins A and B: S(AU1) = d(ab) / + (S(A) S(B)) / = 5 / + ( ) / = 1 S(BU1) = d(ab) / + (S(B) S(A)) / = 5 / + ( ) / = 4 of Neighbor-joining cont 3 Step 4: Join A and B according to S, and make all other taxa in form of a star. Branches in black are unknown length and Branches in red are known length

15 of Neighbor-joining cont 4 57 Step5: Calculate new distance matrix Dxu = (Dix + Djx Dij) / d(cu) = (d(ac) + d(bc) - d(ab)) / = ( ) / =3 d(du) = d(ad) + d(bd) - d(ab) / = 6 Same as EU and FU Then we get the new distance matrix C 3 U1 C D E D 6 7 E F of Neighbor-joining cont 5 Repeat 1 to 5 until all branches are done In this example, we will get this at the end 1 58 F SakiMonkey NJ method produces an Unrooted, Additive Tree Additive means distance between species = distance summed along internal branches Marmoset Orangutan Gorilla PygmyChimp Gibbon Chimp Human Baboon The tree has been rooted using the Mouse as outgroup Mouse Hierarchical Clustering on Microarray Data 0.1 Mouse 47 Lemur Tarsier Tarsier Lemur SakiMonkey Marmoset Baboon Generates only one possible tree Generates only unrooted tree Gibbon Orang Gorilla 100 Human 84 PygmyChimp 100 Chimp 15

16 Community Structure In the study of complex networks, a network is said to have community structure if the nodes of the network can be easily grouped into (potentially overlapping) sets of nodes such that each set of nodes is densely connected internally. In the particular case of non-overlapping community finding, this implies that the network divides naturally into groups of nodes with dense connections internally and sparser connections between groups. But overlapping communities are also allowed. The more general definition is based on the principle that pairs of nodes are more likely to be connected if they are both members of the same community, and less likely to be connected if they do not share communities. Community Detection Groups of vertices within which connections are dense but between which they are sparser. Within-group( intra-group) edges. High density Between-group( inter-group) edges. Low density. Real Word Networks Internet Citation Networks. Transportation Network. Social Networks. Biochemical Networks. Community Structure 16

17 Finding Community Structures Divide the network into non-empty groups( communities) in such a way that every vertex belongs to one of the communities. Many possible divisions could be done. We need a good division. Measurement of good division. Finding Community Structure in very large networks Consider edges that fall within a community or between a community and the rest of the network Define modularity: Q 1 m vw A vw kvk w ( cv, c m probability of an edge between adjacency matrix two vertices is proportional to their degrees. The expected number of edges between two nodes For a random network, Q = 0 the number of edges within a community is no different from what you would expect 0.3<Q<0.7?significant community structure. Greedy approach to maximize Q. w if vertices are in the same community ) Finding Community Structure in very large networks Algorithm start with all vertices as isolates follow a greedy strategy: successively join clusters with the greatest increase DQ in modularity stop when the maximum possible DQ <= 0 from joining any two successfully used to find community structure in a graph with > 400,000 nodes with > million edges Amazon s people who bought this also bought that Newman Fast Algorithm Modularity Measure Q ( e ii a i ) i Fraction of edges that join vertices in community i to vertices in community j Fraction of edges that are attached to vertices in community i 1 Q m kvk w Avw ( cv, cw) vw m 17

18 Newman Fast Algorithm 1. Separate each vertex solely into n community.. Calculate Q for all possible community pairs. 3. Merge the pair of the largest increase in Q. 4. Repeat & 3 until all communities merged in one community. 5. Cross cut the dendogram where Q is maximum Notes: Q ( e ii a i ) Only consider merge two community i and j. Q=(e ij +e ii +e jj +e ji (a i + a j ) (a i + a j ))-(e ii +e jj - a i a i - a j a j ) = e ij +e ji a i a j i Extensions to weighted networks Q Modularity (Analysis of weighted networks, M. E. J. Newman) 1 m vw weighted edge kvk w Avw ( cv, cw) m k i A ij j Conclusions Hierarchical algorithm is a sequential clustering algorithm With distance matrix to construct a tree of clusters (dendrogram) Hierarchical representation without the need of knowing # of clusters (can set termination condition with known # of clusters) Major weakness of agglomerative clustering methods Can never undo what was done previously Sensitive to cluster distance measures and noise/outliers Less efficient: O (n ), where n is the number of total objects There are several variants to overcome its weaknesses BIRCH: uses clustering feature tree and incrementally adjusts the quality of sub-clusters, which scales well for a large data set CHAMELEON: hierarchical clustering using dynamic modeling, which integrates hierarchical method with other clustering methods Density models DBSCAN and OPTICS defines clusters as connected dense regions in the data space Major features: o Discover clusters of arbitrary shape o Handle noise o One scan o Need density parameters as termination condition 71 18

19 Density-Based Clustering Density Concepts Two global parameters: Eps: Maximum radius of the neighbourhood MinPts: Minimum number of points in an Eps-neighbourhood of that point Core Object: object with at least MinPts objects within a Clustering based on density (local cluster criterion), such as density-connected points radius Eps-neighborhood Border Object: object that on the border of a cluster Each cluster has a considerable higher density of points than outside of the cluster q p MinPts = 5 Eps = 1 cm Density-Based Clustering: Background Density-Based Clustering: Background (II) N Eps (p):{q belongs to D dist(p,q) <= Eps} Directly density-reachable: A point p is directly density-reachable from a point q wrt. Eps, MinPts if Density-reachable: A point p is density-reachable from a point q wrt. Eps, MinPts if there is a chain of points p 1,, p n, p 1 = q, p n = p such that p i+1 is directly densityreachable from p i q p 1 p 1) p belongs to N Eps (q) ) N Eps (q) >= MinPts (core point condition) q p MinPts = 5 Eps = 1 cm Density-connected A point p is density-connected to a point q wrt. Eps, MinPts if there is a point o such that both, p and q are density-reachable from o wrt. Eps and MinPts. p o q 19

20 DBSCAN: Density Based Spatial Clustering of Applications with Noise Relies on a density-based notion of cluster: A cluster is defined as a maximal set of density-connected points Discovers clusters of arbitrary shape in spatial databases with noise Border Core Outlier Eps = 1cm MinPts = 5 Arbitrary select a point p Algorithm Retrieve all points density-reachable from p w.r.t. Eps and MinPts. If p is a core point, a cluster is formed. If p is a border point, no points are density-reachable from p and DBSCAN visits the next point of the database. Continue the process until all of the points have been processed. Algorithm Advantages DBSCAN does not require one to specify the number of clusters. DBSCAN can find arbitrarily shaped clusters. It can even find a cluster completely surrounded by (but not connected to) a different cluster. DBSCAN requires just two parameters and is mostly insensitive to the ordering of the points in the database. 0

21 Disadvantages Sensitive to Parameters DBSCAN cannot cluster data sets well with large differences in densities Graph-based models Clique i.e., a subset of nodes in a graph such that every two nodes in the subset are connected by an edge can be considered as a prototypical form of cluster. Corrupted Cliques Problem Distance Graphs Input: A graph G Output: The smallest number of additions and removals of edges that will transform G into a clique graph Turn the distance matrix into a distance graph Genes are represented as vertices in the graph Choose a distance threshold θ If the distance between two vertices is below θ, draw an edge between them The resulting graph may contain cliques These cliques represent clusters of closely located data points! 1

22 Transforming Distance Graph into Clique Graph Overlapping Clustering The distance graph (threshold θ=7) is transformed into a clique graph after removing the two highlighted edges After transforming the distance graph into the clique graph, the dataset is partitioned into three clusters Also known as soft clustering, fuzzy clustering A data object can be assigned to more than one cluster Motivation is that many real world data sets have inherently overlapping clusters A gene can be a part of multiple functional modules (clusters) Subspace Models Biclustering (also known as Co-clustering or two-mode-clustering), clusters are modeled with both cluster members and relevant attributes. Biclustering Microarray data can be viewed as an NM matrix: Each of the N rows represents a gene (or a clone, ORF, etc.). Each of the M columns represents a condition (a sample, a time point, etc.). Each entry represents the expression level of a gene under a condition. It can either be an absolute value (e.g. Affymetrix GeneChip) or a relative expression ratio (e.g. cdna microarrays). A row/column is sometimes referred to as the expression profile of the gene/condition.

23 Biclustering Biclustering It is common to visualize a gene expression datasets by a color plot: Red spots: high expression values (the genes have produced many copies of the mrna). Green spots: low expression values. Gray spots: missing values. N genes M conditions If two genes are related (have similar functions or are coregulated), their expression profiles should be similar (e.g. low Euclidean distance or high correlation). However, they can have similar expression patterns only under some conditions (e.g. they have similar response to a certain external stimulus, but each of them has some distinct functions at other time). Similarly, for two related conditions, some genes may exhibit different expression patterns (e.g. two tumor samples of different sub-types) Biclustering As a result, each cluster may involve only a subset of genes and a subset of conditions, which form a checkerboard structure: Biclustering Different biclustering algorithms have different definitions of bicluster In reality, each gene/condition may participate in multiple clusters. 9 3

24 Biclustering Types of Biclusters a a a a a a a a a a a a a a a a Constant values a a a a a+i a+i a+i a+i a+j a+j a+j a+j a+k a+k a+k a+k Constant values on rows a a+i a+j a+k a a+i a+j a+k a a+i a+j a+k a a+i a+j a+k Constant values on cols a b c d a+i b+i c+i d+i a+j b+j c+j d+j a+k b+k c+k d+k Coherent evolutions (additive) a b c d a x i b x i c x i d x i a x j b x j c x j d x j a x k b xk c x k d x k Coherent evolutions (multiplicative) Cheng and Church Model: A bicluster is represented the submatrix A of the whole expression matrix (the involved rows and columns need not be contiguous in the original matrix). Each entry A ij in the bicluster is the superposition (summation) of: 1. The background level. The row (gene) effect 3. The column (condition) effect A dataset contains a number of biclusters, which are not necessarily disjoint. Cheng and Church For any submatrix C IJ where I and J are a subsets of genes and conditions, the mean squared residude score is 96 A bicluster is a submatrix C IJ that has a low mean squared residue score. 4

25 Cheng and Church Goal: to find biclusters with minimum squared residue: 1 H( I, J) ( aij aij aij aij ) I J ii, jj For an ideal bicluster, H(I, J) = 0. Biclusters (a) and (b) fits the definition of MSR Cheng and Church Constraints: 1M and N1 matrixes always give zero residue. => Find biclusters with maximum sizes, with residues not more than a threshold (largest - biclusters). Constant matrixes always give zero residue Cheng and Church Finding the largest -bicluster: The problem of finding the largest square - bicluster ( I = J ) is NP-hard. Objective function for heuristic methods (to minimize): 1 H( I, J) ( aij aij aij aij ) I J ii, jj => sum of the components from each row and column, which suggests simple greedy algorithms to evaluate each row and column independently. Cheng and Church Greedy methods: Algorithm 1: Single node deletion Parameter(s): (maximum squared residue). Initialization: the bicluster contains all rows and columns. Iteration: 1. Compute all a Ij, a ij, a IJ and H(I, J) for reuse.. Remove a row or column that gives the maximum decrease of H. Termination: when no action will decrease H or H <=. Time complexity: O(MN)

26 Issues with Biclustering Methods Computational Complexity Most methods use some heuristics to limit the search space Although this helps finds most of the TMs in a reasonable amount of time, the greedy nature of most biclustering algorithms makes them suffer from the local minima problem Difficult to Compare Different biclustering methods use different definitions of the biclusters and use different optimization techniques and implementation to solve them (See Madeira et al 004 and Prelic et al 006 for details) Difficult to choose appropriate biclustering algorithm Different algorithms exhibit significant variations in terms of their robustness and sensitivity to noise in the gene expression data (Prelic et al 006) 6