Flexible Mixture Modeling and Model-Based Clustering in R

Transcription

1 Flexible Mixture Modeling and Model-Based Clustering in R Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R 0 / 170

2 Outline Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Outline 1 / 170

3 Program Day 1: Afternoon 13:30 14:00 Registration 14:00 15:40 Lecture Introduction Estimation and inference 15:40 16:00 COFFEE BREAK 16:00 17:00 Practicals Day 2: Morning 09:00 10:40 Lecture 10:40 11:00 COFFEE BREAK 11:00 12:00 Practicals Mixtures of normal distributions Package mclust Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Outline 2 / 170

4 Program / 2 Day 2: Afternoon 14:00 15:40 Lecture 15:40 16:00 COFFEE BREAK 16:00 17:00 Practicals Day 3: Morning 09:00 10:40 Lecture Latent class analysis Package flexmix Mixtures of regression models Extensions and variants Extending flexmix 10:40 11:00 COFFEE BREAK 11:00 12:00 Practicals Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Outline 3 / 170

5 Material Website: Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Outline 4 / 170

6 Introduction Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 5 / 170

7 Finite mixture models Flexible model class with different special cases depending on data types and application. Types of application: Semi-parametric tool to approximate general distribution functions. Modeling of unobserved heterogeneity / detecting groups in data Model-based clustering. Areas of application: Astronomy Biology Economy... Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 6 / 170

8 Finite mixture models / 2 Histogram of faithful$waiting Density faithful$waiting Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 7 / 170

9 Finite mixture models / 3 Histogram of faithful$waiting Density faithful$waiting Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 8 / 170

10 Finite mixture models / Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 9 / 170

11 Finite mixture models / Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 10 / 170

12 Finite mixture models / x yn Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 11 / 170

13 Finite mixture models / x yn Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 12 / 170

14 Model definition A finite mixture model is given by K H(y x, Θ) = π k F k (y x, ϑ k ), k=1 where K π k = 1 and π k > 0 k. k=1 Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 13 / 170

15 Model definition / 2 In the following we assume that the component specific density functions f k () exist and allow to define the density of the mixture distribution h(). h(y x, Θ) = K π k f k (y x, ϑ k ). k=1 Arbitrary and also different densities f k () can be used for each component. In most applications one assumes that f k () is from the same parametric distribution family or model for all k = 1,..., K. The component specific densities only differ in the values of the parameter vector ϑ k. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Introduction 14 / 170

16 Estimation and inference Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 15 / 170

17 Estimation The log-likelihood for N independent observations is given by [ N K ] log L(Θ) = l(θ) = log π k f k (y n x n, ϑ k ). n=1 k=1 Maximum-likelihood (ML) estimation: Direct optimization of likelihood (mostly in simpler cases). Expectation-maximization (EM) algorithm for more complicated models (Dempster, Laird and Rubin, 1977). EM followed by direct optimization for estimate of Hessian.... Bayesian estimation: Gibbs sampling (Diebolt and Robert, 1994) Markov chain Monte Carlo algorithm Applicable when the joint posterior distribution is not known explicitly, but the conditional posterior distributions of each variable/subsets of variables are known. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 16 / 170

18 EM algorithm General method for ML estimation in models with unobserved latent variables. The complete-data log-likelihood contains the observed and the unobserved / missing data. This is easier to estimate. Iterates between E-step, which computes the expectation of the complete-data log-likelihood, and M-step, where the expected complete-data log-likelihood is maximized. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 17 / 170

19 Missing data The component label vectors z n = (z nk ) k=1,...,k are treated as missing data. It holds that z nk {0, 1} and K k=1 z nk = 1 for all n = 1,..., N. The complete-data log-likelihood is given by K N log L c (Θ) = z nk [log π k + log f k (y n x n, ϑ k )]. k=1 n=1 Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 18 / 170

20 EM algorithm: E-step Given the current parameter estimates Θ (i) replace the missing data z nk by the estimated a-posteriori probabilities ẑ (i) nk = P(k y n, x n, Θ (i) ) = π(i) k K u=1 f k(y n x n, ϑ (i) k ) π (i) u f k (y n x n, ϑ (i) u ) The conditional expectation of log L c (Θ) at the ith step is given by Q(Θ; Θ (i) ) = E Θ (i) [log L c (Θ) y, x] K N = ẑ (i) nk [log π k + log f k (y n x n, ϑ k )]. k=1 n=1. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 19 / 170

21 EM algorithm: M-step The next parameter estimate is given by: Θ (i+1) = arg max Q(Θ; Θ (i) ). Θ The estimates for the component sizes are given by: π (i+1) k = 1 N N n=1 ẑ (i) nk. The component specific parameter estimates are determined by: ϑ (i+1) k = arg max ϑ k N n=1 ẑ (i) nk log(f k(y n x n, ϑ k )). Weighted ML estimation of the component specific model. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 20 / 170

22 Mixtures of two univ. normal distributions: Example Finite mixture of two univariate normal distributions: h(y Θ) = πf N (y µ 1, σ 2 1) + (1 π)f N (y µ 2, σ 2 2), where f N ( µ, σ 2 ) is the density of the univariate normal distribution with mean µ and variance σ 2. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 21 / 170

23 Mixtures of two univ. normal distributions: Example / 2 1 Initialize the parameters µ (0) 1, σ(0) 1, µ(0) 2, σ(0) 2 and π (0). Set i = 0. 2 E-step: Determine the a-posteriori probabilities. The contributions to the likelihood for each observation and component are: z (i) n1 = π(i) f N (y n µ (i) 1, (σ2 1) (i) ), z (i) n2 = (1 π(i) )f N (y n µ (i) 2, (σ2 2) (i) ). The a-posteriori probability is given by: z (i) n1 = z (i) n1 z (i) n1 +. z(i) n2 Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 22 / 170

24 Mixtures of two univ. normal distributions: Example / 3 3 M-step: π (i+1) = 1 N N n=1 µ (i+1) 1 = 1 Nπ (i) (σ 2 1) (i+1) = 1 Nπ (i) z (i) n1 N n=1 N n=1 (σ2) 2 (i+1) 1 = N(1 π (i) ) Increase i by 1. z (i) n1 y n, µ (i+1) 2 = z (i) n1 (y n µ (i) 1 )2 1 N(1 π (i) ) N (1 z (i) n1 )(y n µ (i) n=1 2 )2 4 Iterate between E- and M-step until convergence. N (1 z (i) n=1 n1 )y n Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 23 / 170

25 EM algorithm Advantages: The likelihood is increased in each step EM algorithm converges for bounded likelihoods. Relatively easy to implement: Disadvantages: Different mixture models require only different M-steps. Weighted ML estimation of the component specific model is sometimes already available in standard software. Standard errors have to be determined separately as the information matrix is not required during the algorithm. Convergence only to a local optimum. Slow convergence. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 24 / 170

26 Why does EM work? Given the definition of conditional probabilities it holds This can be transformed to P(z y, Θ ) = P(z, y Θ ) P(y Θ ). P(y Θ ) = P(z, y Θ ) P(z y, Θ ). For the log-likelihoods it thus holds l(θ ; y) = l c (Θ ; y, z) l 1 (Θ ; z y). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 25 / 170

27 Why does EM work? / 2 Taking expectation with respect to the conditional density of y, z y, Θ gives l(θ ; y) = E[l c (Θ ; y, z) y, Θ] E[l 1 (Θ ; z y) y, Θ] = Q(Θ, Θ) R(Θ, Θ). R(Θ, Θ) is the expected value of the log-likelihood of a density with parameter Θ taken with respect to the same density with parameter value Θ. Based on the Jensen s inequality it follows that R(Θ, Θ) is maximum for Θ = Θ. If Θ maximizes Q(Θ, Θ), it holds l(θ ; y) l(θ; y) = (Q(Θ, Θ) Q(Θ, Θ)) (R(Θ, Θ) R(Θ, Θ)) 0. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 26 / 170

28 EM as a Max-Max method The EM algorithm can also be interpreted as a joint maximization method. Consider the following function: F(Θ, P) = E P[l c (Θ ; y, z)] E P[log P(z)]. For fixed Θ the function F with respect to P(z) is maximized by P(z) = P[z y, Θ ]. In the M-step F() is maximized with respect Θ keeping P fixed. The observed log-likelihood and F(Θ, P) have the same value for P(z) = P[z y, Θ ]. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 27 / 170

29 EM as a Max-Max method / Model parameter Latent data parameter Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 28 / 170

30 EM algorithm: Variants Classification EM (CEM): assigns each observation to the component with the maximum a-posteriori probability. In general faster convergence than EM. Convergence to the classification likelihood. Stochastic EM (SEM): assigns each observation to one component by drawing from the multinomial distribution induced by the a-posteriori probabilities. Does not converge to the closest local optimum given the initial values. If started with too many components, some components will eventually become empty and will be eliminated. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 29 / 170

31 Bayesian estimation Determine the posterior density using Bayes theorem p(θ Y, X) h(y X, Θ)p(Θ), where p(θ) is the prior and Y = (y n ) n and X = (x n ) n. Standard prior distributions: Proper priors: Improper priors give improper posteriors. Independent priors for the component weights and the component specific parameters. Conjugate priors for the complete likelihood. Dirichlet distribution D(e 0,1,..., e 0,K ) for the component weights which is the conjugate prior for the multinomial distribution. Priors on the component specific parameters depend on the underlying distribution family. Invariant priors, e.g. the parameter for the Dirichlet prior is constant over all components: e 0,k e 0. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 30 / 170

32 Estimation: Gibbs sampling Starting with Z 0 = (z 0 n) n=1,...,n repeat the following steps for i = 1,..., I 0,..., I + I 0. 1 Parameter simulation conditional on the classification Z (i 1) : 1 Sample π 1,..., π K from D(( N n=1 z(i 1) nk + e 0,k ) k=1,...,k ). 2 Sample component specific parameters from the complete-data posterior p(ϑ 1,..., ϑ K Z (i 1), Y ) Store the actual values of all parameters Θ (i) = (π (i) k, ϑ(i) k ) k=1,...,k. 2 Classification of each observation (y n, x n ) conditional on knowing Θ (i) : Sample z (i) n from the multinomial distribution with parameter equal to the posterior probabilities. After discarding the burn-in draws the draws I 0 + 1,..., I + I 0 can be used to approximate all quantities of interest (after resolving label switching). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 31 / 170

33 Example: Gaussian distribution Assume an independence prior p(µ k, Σ 1 k ) f N (µ k ; b 0, B 0 )f W (Σ 1 k ; c 0, C 0 ). 1 Parameter simulation conditional on the classification Z (i 1) : 1 Sample π (i) 1,..., π(i) K from D(( N n=1 z(i 1) nk + e 0,k ) k=1,...,k ). 2 Sample (Σ 1 k ) (i) in each group k from a Wishart W(c k (Z (i 1) ), C k (Z (i 1) )) distribution. 3 Sample µ (i) k in each group k from a N (b k (Z (i 1) ), B k (Z (i 1) )) distribution. 2 Classification of each observation y n conditional on knowing Θ (i) : P(z (i) nk = 1 y n, Θ (i) ) π k f N (y n ; µ k, Σ k ) Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 32 / 170

34 Estimation: Gibbs sampling Advantages: Relatively easy to implement. Disadvantages: Different mixture models differ only in the parameter simulation step. Parameter simulation conditional on the classification is sometimes already available. Might fail to escape the attraction area of one mode Not all posterior modes are visited. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 33 / 170

35 Selecting the number of components A-priori known Information criteria: e.g. AIC, BIC, ICL: AIC = 2 log(l) + 2p, BIC = 2 log(l) + log(n)p, ICL = 2 log(l) + log(n)p + 2ent, where p is the number of estimated parameters and ent denotes the mean entropy of the a-posteriori probabilities. Likelihood ratio test statistic: in a ML framework. Comparison of nested models Regularity conditions are not fulfilled. Bayes factors in a Bayesian framework. Sampling schemes with a varying number of components in a Bayesian framework. Reversible-jump MCMC. Inclusion of birth-and-death processes. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 34 / 170

36 Initialization Construct a suitable parameter vector Θ (0). Random. Other estimation methods: e.g. moment estimators. Classify observations/assign a-posteriori probabilities to each observation. Random. Cluster analysis results: e.g. hierarchical clustering, k-means. Use short runs of EM, CEM or SEM with different initializations (Biernacki et al., 2003). Use different subsets of the complete data (Wehrens et al., 2004). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 35 / 170

37 Resampling for model diagnostics Testing for the number of components: For example, the likelihood-ratio test using the parametric bootstrap (McLachlan, 1987). Standard errors for the parameter estimates: For example using the parametric bootstrap with initialization in the solution (Basford et al., 1997). Identifiability: For example by testing for unimodality of the component specific parameter estimates using either empirical or parametric bootstrap with random initialization. Stability of induced partitions: For example by comparing the results based on the Rand index correct by chance (Hubert & Arabie, 1985) using either empirical or parametric bootstrap with random initialization. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 36 / 170

38 Identifiability Three kinds of identifiability issues: Label switching Overfitting: leads to empty components or components with equal parameters. Generic unidentifiability Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 37 / 170

39 Label switching The posterior distribution is invariant under a permutation of the components with the same component-specific model. Determine a unique labeling for component-specific inference: Impose a suitable ordering constraint, e.g. π s < π t s, t {1,..., S} with s < t. Minimize the distance to the Maximum-A-Posteriori (MAP) estimate. Fix the component membership for some observations. Relabeling algorithms. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Estimation and inference 38 / 170

40 Mixtures of distributions Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 39 / 170

41 Model definition A finite mixture of distributions is given by where H(y Θ) = K π k F k (y ϑ k ), k=1 K π k = 1 and π k > 0 k. k=1 In the case of mixtures of distributions sometimes one of the components is assumed to be from a different parametric family, e.g., to model noise. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 40 / 170

42 Distributions in the components We assume that we have p-dimensional data with Metric variables. Mixtures of multivariate normal distributions Binary / categorical variables. Latent class analysis: Assumes that variables within each component are independent. Both variable types. Mixed-mode data: Assuming again that variable types are independent within each component allows to separate the two cases (Hunt and Jorgensen, 1988). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 41 / 170

43 Generic identifiability Identifiable: (multivariate) normal-, Gamma-, exponential-, Cauchy- and Poisson distribution in the components. Not identifiable: continuous or discrete uniform distribution in the components. Identifiable under certain conditions: Mixture of binomial and multinomial distributions are identifiable if N 2K 1, where N is the parameter representing the number of trials. See for example Everitt & Hand (1981), Titterington et al. (1985), McLachlan & Peel (2000). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 42 / 170

44 Mixtures of normal distributions Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 43 / 170

45 Univariate normal distributions The density of a finite mixture of univariate normal distributions is given by h(y Θ) = K π k f N (y µ k, σk). 2 k=1 The likelihood is unbounded for degenerate solutions with σ 2 k = 0. Add a constraint that 1 σ 2 k > ɛ for all k or 2 π k > ɛ for all k (to achieve in general a similar effect). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 44 / 170

46 Univariate normal distributions / 2 Histogram of faithful$waiting Density faithful$waiting Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 45 / 170

47 Univariate normal distributions / 3 Histogram of faithful$waiting Density faithful$waiting Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 46 / 170

48 Univariate normal distributions / 4 The estimated model has the following parameters: Comp.1 Comp.2 pi mu sigma Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 47 / 170

49 Multivariate normal distributions For p-dimensional data the number of parameters to estimate is ( ) p(p + 1) K 1 + K p + 2 for unrestricted variance-covariance matrices. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 48 / 170

50 Multivariate normal distributions / 2 More parsimonious model specifications are: Restrict the variance-covariance matrices to be the same over components. Specify the structure of the variance-covariance matrix using its decomposition into volume, shape and orientation (Fraley & Raftery, 2002): Σ k = λ k D k diag(a k)d k. For these three characteristics restrictions can be imposed separately to be the same over components. In addition the orientation can be assumed to be equal to the identity matrix and the shape to be spherical. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 49 / 170

51 Multivariate normal distributions / 3 Specify a factor model for the variance-covariance matrix. Mixtures of factor analyzers (McLachlan & Peel, 2000) Analogous restrictions possible to obtain parsimonious Gaussian mixture models (McNicholas & Murphy, 2008). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 50 / 170

52 Multivariate normal distributions / Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 51 / 170

53 Multivariate normal distributions / Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 52 / 170

54 Multivariate normal distributions / Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 53 / 170

55 Mixtures of normal distributions: Example Example from Fraley & Raftery (2002), data from package spatstat. Estimation of a density for the occurrence of maple trees. 2-dimensional coordinates for 514 trees observed. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 54 / 170

56 Mixtures of normal distributions: Example / 2 y x Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 55 / 170

57 Mixtures of normal distributions: Example / 3 Mixtures with 1 to 9 components are fitted with all different restrictions on the variance-covariance matrices. The best model has 7 components and spherical variance-covariance matrices with different volume across components. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 56 / 170

58 Mixtures of normal distributions: Example / 4 y y x x Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 57 / 170

59 Package mclust Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 58 / 170

60 Package mclust Written by Chris Fraley, Adrian Raftery, Luca Scrucca, Thomas Brendan Murphy. In particular suitable for mixtures of uni- and multivariate normal distributions. Uses model-based hierarchical clustering for initialization. Variance-covariance matrices can be specified via volume, shape and orientation. Function Mclust() returns the best model according to the BIC. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 59 / 170

61 Package mclust: Model specification univariate: "E": identical variance "V": different variances multivariate: three letter abbreviation for volume: equal (E) / variable (V) shape: identity (I) / equal (E) / variable (V) orientation: identity (I) / equal (E) / variable (V) "EII": spherical, same volume "VII": spherical, different volume "EEI": diagonal matrix, same volume & shape "VEI": diagonal matrix, different volume, same shape "EVI": diagonal matrix, same volume, different shape "VVI": diagonal matrix, different volume & shape Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 60 / 170

62 Package mclust: Model specification / 2 "EEE": ellipsoidal, same volume, shape & orientation "EVE": ellipsoidal, same volume & orientation "VEE": ellipsoidal, same shape & orientation "VVE": ellipsoidal, same orientation "EEV": ellipsoidal, same volume & shape "VEV": ellipsoidal, same shape "EVV": ellipsoidal, same volume "VVV": ellipsoidal, different volume, shape and orientation Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 61 / 170

63 Package mclust: Model specification / 3 EII VII EEI VEI EVI VVI Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 62 / 170

64 Package mclust: Model specification / 4 EEE EVE VEE VVE EEV VEV EVV VVV Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 63 / 170

65 Package mclust: Function Mclust() Mclust() has the following arguments: data: A numeric vector, a numeric matrix or a data frame of observations. Categorical variables are not allowed. If a matrix or a data frame is supplied, the rows correspond to observations and columns to variables. G: An integer vector indicating the number of components for which the model should be fitted. The default is G = 1:9. modelnames: A character vector specifying the models (with different restrictions on the variance-covariance matrices) to fit. prior: The default is to use no prior. This argument allows to specify conditionally conjugate priors on the mean and the variance-covariance matrices of the components using priorcontrol(). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 64 / 170

66 Package mclust: Function Mclust() / 2 control: A list of control parameters for the EM algorithm. The defaults are obtained by calling emcontrol(). initialization: A list containing one or several elements of: (1) "hcpairs", providing the results of the hierarchical clustering, and (2) "subset", a logical or numeric vector to specify a subset. warn: A logical value to indicate if warnings (e.g., with respect to singularities) are issued. The default is to suppress warnings....: Catches unused arguments. Returns an object of class Mclust. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 65 / 170

67 Package mclust: Further methods The fitted object with class Mclust can be plotted using the associated plot method. The following plots can be generated: Comparison of the BIC values for all estimated models. Scatter plot indicating the classification for 2-dimensional data. 2-dimensional projection of the data with classification. 2-dimensional projection of the data with uncertainty of classification. Estimated density for 1- and 2-dimensional data. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 66 / 170

68 Package mclust: Maple trees > data("lansing", package = "spatstat") > maples <- subset(spatstat::as.data.frame.ppp(lansing), + marks == "maple", c("x", "y")) > library("mclust") > maplesmix <- Mclust(maples) > maplesmix Mclust model object: best model: spherical, varying volume (VII) with 7 compone > par(mfrow = c(2, 2)) > plot(maplesmix, what = "BIC") > plot(maplesmix, what = "classification") > plot(maplesmix, what = "uncertainty") > plot(maplesmix, what = "density") Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 67 / 170

69 Package mclust: Maple trees / 2 Classification BIC EII VII EEI VEI EVI VVI EEE EVE VEE VVE EEV VEV EVV VVV y Number of components x Uncertainty log Density Contour Plot y y x x Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 68 / 170

70 Latent class analysis Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 69 / 170

71 Latent class analysis Used for multivariate binary and categorical data. Often only local identifiability ensured. Observed variables are correlated because of the unobserved group structure. Within each component variables are assumed to be independent and the multivariate density is given by the product of the univariate densities. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 70 / 170

72 Latent class analysis / 2 In general the model can be written as K p h(y Θ) = f kj (y j ϑ kj ). k=1 π k In latent class analysis the densities f kj () for all k and j are from the same univariate parametric distribution family. j=1 For binary variables the Bernoulli distribution is used: f (y θ) = θ y (1 θ) 1 y. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 71 / 170

73 Latent class analysis: Example Example from Dayton (1999), data from package polca. Data from a survey among students about cheating. 319 students were questioned. The students were asked about their behavior during their bachelor studies. The following four questions are used in the following which are about 1 lying to avoid taking an exam, 2 lying to avoid handing a term paper in on time, 3 purchasing a term paper to hand in as their own or obtaining a copy of an exam prior to taking the exam, 4 copying answers during an exam from someone sitting near to them. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 72 / 170

74 Latent class analysis: Example / 2 The relative frequencies of agreement with the four questions are: 0.11, 0.12, 0.07, First it is assessed if the answers to the different questions are independent given the marginal distributions. The density is given by h(y Θ) = 4 j=1 θ y j j (1 θ j ) 1 y j, where θ j is the probability that question j is agreed to. A χ 2 -goodness-of-fit test is conducted: χ 2 = 136.3, p-value < 2e 16. This simple model does not fit the data well. Extension to mixture models. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 73 / 170

75 Latent class analysis: Example / 3 The finite mixture model is given by K 4 h(y Θ) = π k θ y j kj (1 θ kj) 1 y j, k=1 j=1 where θ kj is the probability to agree to question j, if the respondent is from component k. This model is fit with 2 components. The χ 2 -goodness-of-fit test gives χ 2 = 8.3, p-value = The null hypothesis that the data is from this model cannot be rejected. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 74 / 170

76 Latent class analysis: Example / 4 The size of the first component is The estimated parameters for both components are Comp.1 Comp.2 center.lieexam center.liepaper center.fraud center.copyexam Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 75 / 170

77 Package flexmix Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 76 / 170

78 Software: flexmix (Leisch, 2004; Grün & Leisch, 2008a) The function flexmix() provides the E-step and all data handling. The M-step is supplied by the user similar to glm() families. Multiple independent responses from different families Currently bindings to several GLM families exist (Gaussian, Poisson, Gamma, Binomial) Weighted, hard (CEM) and random (SEM) classification Components with prior probability below a user-specified threshold are automatically removed during iterations. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 77 / 170

79 flexmix design Primary goal is extensibility: ideal for trying out new mixture models. No replacement of specialized mixtures like Mclust(), but complement. Usage of S4 classes and methods. Formula-based interface. Multivariate responses: combination of univariate families: assumption of independence (given x), each response may have its own model formula, i.e., a different set of regressors. multivariate families: if family handles multivariate response directly, then arbitrary multivariate response distributions are possible. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 78 / 170

80 Function flexmix() flexmix() takes the following arguments: formula: A symbolic description of the model, which is fitted. The general form is y~x g, where y is the dependent variable, x the independent variables and g an optional categorical grouping variable for repeated observations. data: An optional data frame used for evaluating the formula. k: Number of components (not necessary if cluster is specified). cluster: Either a matrix with k columns which contains the initial a-posteriori probabilities; or a categorical variable or integer vector indicating the component memberships of observations. model: Object of class FLXM or a list of these objects. concomitant: Object of class FLXP. control: Object of class FLXcontrol or a named list. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 79 / 170

81 Function flexmix() / 2 Returns an object of class flexmix. Repeated calls of flexmix() with stepflexmix(): using random initialization. initflexmix(): combining short with long runs of EM. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 80 / 170

82 Controlling the EM algorithm FLXcontrol for setting the specifications for the EM algorithm: iter.max: maximum number of iterations. minprior: minimum component sizes. verbose: If larger than 0, then flexmix() provides status information each verbose iteration of the EM algorithm. classify: One of auto, weighted, CEM (or hard ), SEM (or random ). For convenience flexmix() also allows specification of the control argument through a named list, where the names are partially matched, e.g., flexmix(..., control = list(class = "r")) Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 81 / 170

83 Variants of mixture models Component specific models: FLXMxxx() Model-based clustering: FLXMCxxx() FLXMCmvnorm() FLXMCmvbinary() FLXMCmvpois()... Clusterwise regression: FLXMRxxx() FLXMRglm() FLXMRglmfix() FLXMRziglm()... Concomitant variable models: FLXPxxx() FLXPconstant() FLXPmultinom() Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 82 / 170

84 Methods for flexmix objects show(), summary(): some information on the fitted model. plot(): rootogram of posterior probabilities. refit(): refits an estimated mixture model to obtain the variance-covariance matrix. loglik(), BIC(),... : obtain log-likelihood and model fit criteria. parameters(), prior(): obtain component specific model parameters and prior class probabilities / component weights. posterior(), clusters(): obtain a-posteriori probabilities and assignments to the maximum a-posteriori probability. fitted(), predict(): fitted and predicted (component-specific) values. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 83 / 170

85 Package flexmix: Cheating Data preparation: > data("cheating", package = "polca") > X <- as.data.frame(table(cheating[, 1:4])) > X <- subset(x, Freq > 0) > X[, 1:4] <- sapply(x[,1:4], as.integer) - 1 > head(x) LIEEXAM LIEPAPER FRAUD COPYEXAM Freq Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 84 / 170

86 Package flexmix: Cheating / 2 Fit model with flexmix: > set.seed(201107) > FORMULA <- + cbind(lieexam, LIEPAPER, FRAUD, COPYEXAM) ~ 1 > CONTROL <- + list(tolerance = 10^-12, iter.max = 500) > cheatmix <- + stepflexmix(formula, k = 2, weights = ~ Freq, + model = FLXMCmvbinary(), data = X, + control = CONTROL, nrep = 3) 2 : * * * Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 85 / 170

87 Package flexmix: Cheating / 3 > cheatmix Call: stepflexmix(formula, weights = ~Freq, model = FLXMCmvbinary data = X, control = CONTROL, k = 2, nrep = 3) Cluster sizes: convergence after 314 iterations Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 86 / 170

88 Package flexmix: Cheating / 4 > summary(cheatmix) Call: stepflexmix(formula, weights = ~Freq, model = FLXMCmvbinary data = X, control = CONTROL, k = 2, nrep = 3) prior size post>0 ratio Comp Comp log Lik (df=9) AIC: BIC: > plot(cheatmix) Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 87 / 170

89 Package flexmix: Cheating / 5 Rootogram of posterior probabilities > 1e Comp. 1 Comp Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of distributions 88 / 170

90 Mixtures of regression models Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 89 / 170

91 Mixtures of regression models aka clusterwise regression Regression models are fitted for each component. Weighted ML estimation of linear and generalized linear models required. Heterogeneity between observations with respect to their regression parameters. Random effects can be estimated in a semi-parametric way. Possible models: Mixtures of linear regression models Mixtures of generalized linear regression models Mixtures of linear mixed regression models... Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 90 / 170

92 Mixtures of regression models / 2 The density of a mixture of regression models is given by K h(y x, w, Θ) = π k f k (y x, ϑ k ), k=1 where K π k = 1 and π k > 0 k. k=1 In the EM algorithm the weighted log-likelihoods of the regression models need to be maximized in the M-step. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 91 / 170

93 Generic identifiability Influencing factors: Component distribution (see mixtures of distributions). Model matrix. Repeated observations for each individual / partly labeled observations. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 92 / 170

94 Identifiability: Model matrix If only one observation per individual is available, the full rank of the model matrix is not sufficient for identifiability. Linear models: Mixtures of linear models with normally distributed errors are identifiable, if the number of components K is smaller than the minimum number of (feasible) hyper planes which are necessary to cover all covariate points (Hennig, 2000). Generalized linear models: Analogous condition for the linear predictor, additional constraints dependent on the distribution of the dependent variable (in particular for binomial and multinomial logit-models, see Grün & Leisch, 2008b). Coverage condition. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 93 / 170

95 Identifiability: Repetitions / labels At least one of the hyper planes in the coverage condition needs to cover all repeated / labeled observations where component membership is fixed. Violation of the coverage condition leads to: Intra-component label switching: If the labels are fixed in one covariate point according to some ordering constraint, then labels may switch in other covariate points for different parameterizations of the model. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 94 / 170

96 Illustration: Binomial logit models We consider a mixture density h(y x, Θ) = π 1 f B (y N, (1, x)β 1 ) + (1 π 1 )f B (y N, (1, x)β 2 ) with binomial logit models in the components and parameters π 1 = 0.5, β 1 = ( 2, 4), β 2 = (2, 4). Even if N 3 the mixture is not identifiable if there are only 2 covariate points available. The second solution is then given by π (2) 1 = 0.5, β (2) 1 = ( 2, 0), β (2) 2 = (2, 0). Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 95 / 170

97 Illustration: Binomial logit models / 2 Simulation design: Number of repetitions N {1, 10} in the same covariate point. Number of covariate points: #x {2, 5}, equidistantly spread across [0, 1]. 100 samples with 100 observations are drawn from the model for each combination of N and #x. Finite mixtures with 2 components are fitted to each sample. Solutions after imposing an ordering constraint on the intercept are reported. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 96 / 170

98 Illustration: Binomial logit models / 3 Relative frequency #x = 5 N = 1 #x = 2 N = 1 #x = 5 N = 10 #x = 2 N = x Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 97 / 170

99 Illustration: Binomial logit models / 4 5% 95% π #x = 5 N = 1 #x = 5 N = 10 x (Intercept) π #x = 2 N = 1 #x = 2 N = 10 x (Intercept) 5% 95% Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 98 / 170

100 Mixtures of linear regressions: Example GNP CO2 Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 99 / 170

101 Mixtures of linear regressions: Example / 2 Example from Hurn, Justel and Robert (2003), data from package mixtools. 28 observations of countries on their gross national product per capita (GNP) and the estimated carbon dioxide emission per capita (CO2) in The relationship between the two variables is analyzed using a linear model. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 100 / 170

102 Mixtures of linear regressions: Example / GNP CO2 Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 101 / 170

103 Mixtures of linear regressions: Example / 4 Call: lm(formula = CO2 ~ GNP, data = CO2data) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-05 GNP Residual standard error: 4.06 on 26 degrees of freedom Multiple R-squared: 0.047, Adjusted R-squared: 0.0 F-statistic: 1.28 on 1 and 26 DF, p-value: Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 102 / 170

104 Mixtures of linear regressions: Example / 5 It is assumed that the relationship between the two variables is not the same for all countries. A mixture of linear regression models is thus estimated. The model is given by h(co2 GNP, Θ) = K π k f N (CO2 α 0k + GNPα 1k, σk), 2 k=1 where α 0k is the component specific intercept, α 1k the component specific regression coefficient for GNP and σk 2 the component specific residual variance. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 103 / 170

105 Mixtures of linear regressions: Example / 6 Mixtures of linear regression models are fitted for 1 to 5 components. > data("co2data", package = "mixtools") > co2mix_step <- + stepflexmix(co2 ~ GNP, data = CO2data, k = 1:5) 1 : * * * 2 : * * * 3 : * * * 4 : * * * 5 : * * * Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 104 / 170

106 Mixtures of linear regressions: Example / 7 > co2mix_step Call: stepflexmix(co2 ~ GNP, data = CO2data, k = 1:5) iter converged k k0 loglik AIC BIC ICL 1 2 TRUE TRUE TRUE TRUE TRUE Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 105 / 170

107 Mixtures of linear regressions: Example / 8 The best model according to the BIC is selected. This has 2 components. > co2mix <- getmodel(co2mix_step) > co2mix Call: stepflexmix(co2 ~ GNP, data = CO2data, k = 2) Cluster sizes: convergence after 22 iterations Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 106 / 170

108 Mixtures of linear regressions: Example / 9 CO GNP Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 107 / 170

109 Mixtures of linear regressions: Example / 10 The estimated regression coefficients for each component together with the estimated standard errors are given by $Comp.1 Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 GNP $Comp.2 Estimate Std. Error z value Pr(> z ) (Intercept) GNP <2e-16 Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 108 / 170

110 Mixtures of linear regressions: Example / 11 The countries assigned to the smaller component are: GNP CO2 country TUR MEX CAN AUS NOR USA Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 109 / 170

111 Mixtures of binomial logit models: Example Proportion of infected plants Number of aphids Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 110 / 170

112 Mixtures of binomial logit models: Example / 2 Data from package mixreg; taken from Boiteau et al. (1998) and Turner (2000). Observations from 51 independent experiments. The number of aphids which were released in a room with 12 infected and 69 healthy tobacco plants was varied. After 24 hours the previously healthy plants are checked for infection. For each experiment the number of aphids released (n.aphids) and the number of newly infected plants (n.inf) were recorded. It is assumed that the data comes from a mixture of 2 binomial logit-models. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 111 / 170

113 Mixtures of binomial logit models: Example / 3 The model is given by h(n.inf n.aphids, 69, Θ) = K π k f B (n.inf 69, θ k (n.aphids)), k=1 where f B ( N, θ) is the binomial distribution with number of trials parameter N and success probability θ. Furthermore logit(θ k ) = α 0k + α 1k n.aphids α 0k is the component specific intercept, α 1k the component specific regression coefficient for number of aphids. Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 112 / 170

114 Mixtures of binomial logit models: Example / 4 > aphidmix <- + stepflexmix(cbind(n.inf, 69 - n.inf) ~ n.aphids, + data = aphids, k = 2, nrep = 5, + model = FLXMRglm(family = "binomial")) 2 : * * * * * Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 113 / 170

115 Mixtures of binomial logit models: Example / 5 > aphidmix Call: stepflexmix(cbind(n.inf, 69 - n.inf) ~ n.aphids, data = aphids, model = FLXMRglm(family = "binomial"), k = 2, nrep = 5) Cluster sizes: convergence after 6 iterations Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 114 / 170

116 Mixtures of binomial logit models: Example / 6 > summary(aphidmix) Call: stepflexmix(cbind(n.inf, 69 - n.inf) ~ n.aphids, data = aphids, model = FLXMRglm(family = "binomial"), k = 2, nrep = 5) prior size post>0 ratio Comp Comp log Lik (df=5) AIC: BIC: Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 115 / 170

117 Mixtures of binomial logit models: Example / Proportion of infected plants Number of released aphids Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 116 / 170

118 Mixtures of binomial logit models: Example / 8 > aphidmix_ref <- refit(aphidmix) > summary(aphidmix_ref) $Comp.1 Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 n.aphids $Comp.2 Estimate Std. Error z value Pr(> z ) (Intercept) < 2e-16 n.aphids e-14 Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R Mixtures of,regression models 117 / 170