Clustering Lecture 5: Mixture Model

Size: px

Start display at page:

Download "Clustering Lecture 5: Mixture Model"

Alisha Burke
5 years ago
Views:

1 Clustering Lecture 5: Mixture Model Jing Gao SUNY Buffalo 1

2 Outline Basics Motivation, definition, evaluation Methods Partitional Hierarchical Density-based Mixture model Spectral methods Advanced topics Clustering ensemble Clustering in MapReduce Semi-supervised clustering, subspace clustering, co-clustering, etc. 2

3 Using Probabilistic Models for Clustering Hard vs. soft clustering Hard clustering: Every point belongs to exactly one cluster Soft clustering: Every point belongs to several clusters with certain degrees Probabilistic clustering Each cluster is mathematically represented by a parametric distribution The entire data set is modeled by a mixture of these distributions 3

4 Gaussian Distribution f(x) Changing μ shifts the distribution left or right σ Changing σ increases or decreases the spread μ x Probability density function f(x) is a function of x given μ and σ x N( x, ) exp( ( ) ) 4

5 Likelihood f(x) Which Gaussian distribution is more likely to generate the data? x Define likelihood as a function of μ and σ given x 1, x 2,, x n n N(, 2 ) i1 x i 5

6 6 Gaussian Distribution Multivariate Gaussian Log likelihood mean covariance ) ln )) ( ) ( 2 1 ( ), ( ln ), ( i T i n i n i i x x x N L

7 Maximum Likelihood Estimate MLE Find model parameters, that maximize log likelihood L(, ) MLE for Gaussian 7

8 Gaussian Mixture Linear combination of Gaussians where parameters to be estimated 8

9 Gaussian Mixture To generate a data point: first pick one of the clusters with probability then draw a sample from that cluster distribution 9

10 Gaussian Mixture Maximize log likelihood Each data point is generated by one of K clusters, a latent variable is associated with each Regard the values of latent variables as missing 10

11 Expectation-Maximization (EM) Algorithm E-step: for given parameter values we can compute the expected values of the latent variables Note that instead of but we still have 11

12 Expectation-Maximization (EM) Algorithm M-step: maximize the expected complete log likelihood Parameter update: 12

13 EM Algorithm Iterate E-step and M-step until the log likelihood of data does not increase any more. Converge to local optimal Need to restart algorithm with different initial guess of parameters (as in K-means) Relation to K-means Consider GMM with common covariance As, two methods coincide 13

14 14 14

15 15 15

16 16 16

17 17 17

18 18 18

19 19 19

20 K-means vs GMM Objective function Minimize sum of squared error Can be optimized by an EM algorithm E-step: assign points to clusters M-step: optimize cluster centers Performs hard assignment during E-step Assumes spherical clusters with equal probability of a cluster Objective function Maximize log-likelihood EM algorithm E-step: Compute posterior probability of membership M-step: Optimize parameters Perform soft assignment during E-step Can be used for non-spherical clusters Can generate clusters with different probabilities 20

21 Mixture Model Strengths Give probabilistic cluster assignments Have probabilistic interpretation Can handle clusters with varying sizes, variance etc. Weakness Initialization matters Choose appropriate distributions Overfitting issues 21

22 Probabilistic clustering Take-away Message Maximum likelihood estimate Gaussian mixture model for clustering EM algorithm that assigns points to clusters and estimates model parameters alternatively Strengths and weakness 22

Clustering Lecture 3: Hierarchical Methods

Clustering Lecture 3: Hierarchical Methods Jing Gao SUNY Buffalo 1 Outline Basics Motivation, definition, evaluation Methods Partitional Hierarchical Density-based Mixture model Spectral methods Advanced