Tools and methods for model-based clustering in R
|
|
|
- Vernon Greer
- 5 years ago
- Views:
Transcription
1 Tools and methods for model-based clustering in R Bettina Grün Rennes 08
2 Cluster analysis The task of grouping a set of objects such that Objects in the same group are as similar as possible and Objects in different groups are as dissimilar as possible. The aim is to determine a partition of the given set of objects, e.g., to determine which objects belong to the same group and which to different groups. Statistical methods: Heuristic methods: hierarchical clustering, partitioning methods (e.g., k-means). Model-based methods: finite mixture models.
3 Specifying the cluster problem The cluster problem is in general perceived as ill defined. Different notions of what defines a cluster exist: Compactness. Density-based levels. Connectedness. Functional similarity. Several cluster solutions might exist for a given data set depending on which notion is used. The application context is important to define which clusters should be targeted and to assess the usefulness of a clustering solution.
4 Specifying the cluster problem /
5 Model-based clustering methods Model-based clustering embeds the clustering problem in a probabilistic framework. This implies: Statistical inference tools can be used. Different cluster distributions can be used depending on the cluster notion. More explicit specification of what defines a cluster required than for heuristic methods.
6 Finite mixture models Generative model for observations (y i, x i ), i =,..., n: Draw a cluster membership indicator S i from a multinomial distribution with parameters η = (η,..., η K ). Draw y i given x i and S i from the cluster distribution: y i x i f Si (y i x i ). The distribution of (y i, x i ) is then given by y i x i K η k f k (y i x i ), k= where η k 0 for all k and K k= η k =. f k () represents the cluster distribution.
7 Finite mixture models / Methods differ with respect to: Clustering kernel: Specification of cluster distributions. Use of additional variables x i, e.g., for regression. Estimation framework: Maximum likelihood estimation. Bayesian estimation.
8 Finite mixture models / Cluster membership indicators can be inferred using the a-posteriori probabilities: P(S i = k y i, x i ) η k f k (y i x i ). A hard assignment can be obtained by Assigning to the cluster where this probability is maximum. Drawing from this probability distribution.
9 Finite mixture models / 4
10 Estimation of finite mixtures with fixed K Maximum likelihood estimation: EM algorithm. General purpose optimizers. Hybrid approaches. Bayesian estimation: MCMC sampling with data augmentation by adding S i, i =,..., n. General purpose Gibbs samplers can be used, e.g., JAGS available in R through package rjags (Plummer, 06).
11 EM algorithm Standard maximum likelihood estimation method in a missing data context. Guaranteed to converge for bounded likelihoods. Only convergence to a local optimum. In general slow convergence behavior. Consists of E- and M-step: E-step requires determining the a-posteriori probabilities. M-step requires weighted maximum likelihood estimation of the cluster distributions.
12 MCMC sampling Determination of the a-posteriori probabilities required to draw S i, i =,..., n from a multinomial distribution. Conditional on S i, i =,..., n drawing from the posterior of the cluster-specific parameters is the same as if the cluster-specific distribution is used for the complete data set. Often poor mixing observed. For symmetric priors the posterior is also symmetric and thus multimodal.
13 Determining the number of clusters No generally accepted solution available. Suggested methods include: Information criteria: AIC, BIC, ICL. Likelihood ratio test with distribution under the null determined using sampling methods. Marginal likelihoods in Bayesian estimation.
14 Clustering kernel Components corresponding to clusters: In general using parametric distributions for the components and thus also for the clusters. Multivariate continuous data. Multivariate categorical data. Multivariate mixed data. Multivariate data with regression structure. Combining components to clusters: I.e., the cluster distributions are mixture distributions. Two-step procedures. Simultaneous estimation using constraints or informative priors. In the following these variants are investigated for maximum likelihood estimation.
15 Multivariate continuous data The standard model is a mixture of multivariate Gaussians. The model-based clustering model is given by y i K η k φ(y i µ k, Σ k ). k= For K clusters and d-dimensional observations y i the number of estimated parameters corresponds to K (d + d(d + )/) + K.
16 Multivariate continuous data / Parsimonity is achieved based on the decomposition of the variance-covariance matrix into given by Volume λ. Orientation D. Shape A. Σ k = λ k D k A k D k. 4 different models emerge by imposing different constraints on the variance-covariance matrices within or across clusters. Available packages in R, e.g.: mclust (Scrucca et al., 06), mixture (Browne et al., 08), Rmixmod (Lebret et al., 05).
17 Multivariate continuous data /
18 Multivariate continuous data / 4 Alternative approaches to achieve parsimonity are mixtures of factor analyzers. E.g., package pgmm (McNicholas et al., 08) in R. If the cluster shapes are not symmetric and light tailed, alternative cluster kernels are: t-distributions (e.g., package teigen; Andrews et al. 08). Skewed and / or heavy tailed distributions: e.g., EMMIXcskew (Lee and McLachlan, 08), MixSAL (Franczak et al., 08), mixsmsn (Prates et al., 0).
19 Multivariate categorical data Often also referred to as latent class analysis. Clusters induce a dependency between variables, while variables are independent within clusters. Local independency assumption. The model-based clustering model is given by K d y i Multinomial(y ij π j k ) k= η k j= for d-dimensional observations. Available packages in R: e.g., polca (Linzer and Lewis, 0) Rmixmod (Lebret et al., 05)
20 Multivariate data with regression structure Often also referred to as clusterwise regression. The model-based clustering model is given by y i x i K η k f (y i µ k (x i ), φ k ). k= Different regression models possible: Generalized linear models. Generalized linear mixed-effects models. Available packages in R: e.g., flexmix (Leisch, 004; Grün and Leisch, 008) mixtools (Benaglia et al., 009)
21 Multivariate data with regression structure /
22 Combining components to clusters Two-step procedures: Fit a mixture model as semi-parametric tool for density estimation. Combine components of the mixture model to form clusters based on some criterion. Available packages in R, e.g.: mclust uses entropy as criterion (Baudry et al., 00). fpc (Hennig, 08) provides several variants as proposed in Hennig (00). Simultaneous estimation using informative priors in Bayesian estimation can be used in combination with standard estimation methods.
23 Post-processing tools Inference on partitions. Inference on cluster-specific parameters: In particular for Bayesian estimation the label switching problem needs to be resolved. Assigning new observations to clusters: Cluster predictions possible. Assessing cluster quality.
24 Assessing cluster quality Agreement measures between cluster assignments and true classes available as in a supervised setting: Label-invariant measures: Rand index (corrected for agreement by chance). Jaccard index. Purity. Label-specific measures: Misclassification rate. Available packages in R: Package clue (Hornik, 005) provides general infrastructure to assess cluster solutions: Function cl_agreement provides several methods to assess cluster agreement.
25 Comparing packages Use a classification data set with known classes and three continuous predictors. Compare the three packages for fitting mixtures of Gaussians with variance-covariance matrix decomposition. Fit all models for K =,..., 9 and select the best according to the BIC. Extract the cluster assignments and assess cluster quality in comparison to the true classification.
26 Comparing packages: data > data("diabetes", package = "mclust") > library("clue") > class <- as.cl_partition(diabetes$class) > X <- diabetes[, -] > pairs(x, col = diabetes$class, pch = 9)
27 Comparing packages: data / glucose insulin sspg
28 Comparing packages: mclust > library("mclust") > mclust.sol <- Mclust(X, G = :9) > class(mclust.sol) [] "Mclust" > mclust.sol[c("g", "modelname")] $G [] $modelname [] "VVV" > c(loglik = loglik(mclust.sol), BIC = BIC(mclust.sol)) loglik BIC
29 Comparing packages: mclust / Standard visualization: > plot(mclust.sol, "BIC") > plot(mclust.sol, "classification") > plot(mclust.sol, "uncertainty") > plot(mclust.sol, "density")
30 Comparing packages: mclust / BIC EII VII EEI VEI EVI VVI EEE EVE VEE VVE EEV VEV EVV VVV glucose insulin sspg Number of components glucose glucose glucose glucose glucose e 05 glucose e insulin insulin insulin insulin sspg insulin e 05 insulin insulin insulin sspg sspg glucose sspg sspg sspg sspg e 05 glucose sspg sspg sspg glucose insulin glucose insulin
31 Comparing packages: mclust / 4 Class agreement: > cl_agreement(as.cl_hard_partition(mclust.sol), class, + method = "crand") Cross-agreements using corrected Rand index: [,] [,]
32 Comparing packages: mixture > set.seed(4) > library("mixture") > mixture.sol <- gpcm(as.matrix(x), G = :9) > class(mixture.sol) [] "gpcm" > (best <- mixture.sol$bicmodel[c("g", "covtype")]) $G [] $covtype [] "VVV" > mixture.sol$bic[best$g, best$covtype, c("loglik", "BIC")] loglik BIC
33 Comparing packages: mixture / Standard visualization: > plot(mixture.sol) BIC EII VII EEI VEI EVI VVI EEE EEV VEV VVV EVE VVE VEE EVV # components
34 Comparing packages: mixture / Class agreement: > mixture.preds <- mixture.sol$map > cl_agreement(as.cl_partition(mixture.preds), class, + method = "crand") Cross-agreements using corrected Rand index: [,] [,]
35 Comparing packages: Rmixmod > library("rmixmod") > mixmod.sol. <- mixmodcluster( + data = X, nbcluster =, + models = mixmodgaussianmodel(listmodels = + c("gaussian_pk_lk_ck", "Gaussian_pk_Lk_Bk", + "Gaussian_pk_Lk_I"))) > mixmod.sol.["bestresult"]@criterionvalue [] > mixmod.sol <- mixmodcluster( + data = X, nbcluster = :9, + models = mixmodgaussianmodel(equal.proportions = FALSE), + strategy = mixmodstrategy(nbtryininit = 0), seed = 0) > class(mixmod.sol) [] "MixmodCluster" attr(,"package") [] "Rmixmod"
36 Comparing packages: Rmixmod / > mixmod.sol["bestresult"]@nbcluster [] > mixmod.sol["bestresult"]@model [] "Gaussian_pk_Lk_Ck" > c(loglik = mixmod.sol["bestresult"]@likelihood, + BIC = mixmod.sol["bestresult"]@criterionvalue) loglik BIC Standard visualization: > plot(mixmod.sol)
37 Comparing packages: Rmixmod / Histogram of glucose Histogram of insulin Histogram of sspg
38 Comparing packages: Rmixmod / 4 Some coding also allows to obtain the BIC values for all fitted models... BIC number of clusters
39 Comparing packages: Rmixmod / 5 Class agreement: > mixmod.preds <- mixmod.sol["bestresult"]@partition > cl_agreement(as.cl_partition(mixmod.preds), class, + method = "crand") Cross-agreements using corrected Rand index: [,] [,]
40 Comparing packages All packages implement the EM algorithm and allow to specify the number of clusters and which variance-covariance structures should be fitted. By default all packages use the BIC to select the best fitting model. For this simple example all packages arrive at the same solution. However, the packages use different initialization strategies. The return objects differ as well as the associated methods available to inspect the obtained solutions. Class assignments can be extracted and used in combination with package clue to assess cluster quality.
41 Summary Model-based clustering is a versatile method for clustering. Different variants exist depending on Clustering kernel. Estimation methods. A large number of R packages are available where each covers a certain set of mixture models. The same functionality is often covered by several packages. Packages vary in the range of models covered and what additional functionality is provided for fitted models. Common standards and infrastructure are mostly lacking. For more information see the CRAN Task View: Cluster Analysis & Finite Mixture Models:
42 References J. L. Andrews, J. R. Wickins, N. M. Boers, and P. D. McNicholas. teigen: An R package for model-based clustering and classification via the multivariate t distribution. Journal of Statistical Software, 8(7):, 08. doi: 0.867/jss.v08.i07. J.-P. Baudry, A. Raftery, G. Celeux, K. Lo, and R. Gottardo. Combining mixture components for clustering. Journal of Computational and Graphical Statistics, (9): 5, 00. doi: 0.98/jcgs T. Benaglia, D. Chauveau, D. R. Hunter, and D. Young. mixtools: An R package for analyzing finite mixture models. Journal of Statistical Software, (6): 9, 009. doi: 0.867/jss.v0.i06. R. P. Browne, A. ElSherbiny, and P. D. McNicholas. Mixture: Mixture Models for Clustering and Classification, 08. URL R package version.5.
43 References / B. C. Franczak, R. P. Browne, P. D. McNicholas, and K. L. Burak. MixSAL: Mixtures of Multivariate Shifted Asymmetric Laplace (SAL) Distributions, 08. URL R package version.0. B. Grün and F. Leisch. FlexMix version : Finite mixtures with concomitant variables and varying and constant parameters. Journal of Statistical Software, 8(4): 5, 008. doi: 0.867/jss.v08.i04. C. Hennig. Methods for merging Gaussian mixture components. Advances in Data Analysis and Classification, 4(): 4, 00. doi: 0.007/s C. Hennig. fpc: Flexible Procedures for Clustering, 08. URL R package version.-. K. Hornik. A CLUE for CLUster Ensembles. Journal of Statistical Software, 4(): 5, 005. doi: 0.867/jss.v04.i.
44 References / R. Lebret, S. Iovleff, F. Langrognet, C. Biernacki, G. Celeux, and G. Govaert. Rmixmod: The R package of the model-based unsupervised, supervised, and semi-supervised classification Mixmod library. Journal of Statistical Software, 67(6): 9, 05. doi: 0.867/jss.v067.i06. S. Lee and G. McLachlan. EMMIXcskew: An R package for the fitting of a mixture of canonical fundamental skew t-distributions. Journal of Statistical Software, 8():, 08. doi: 0.867/jss.v08.i0. F. Leisch. FlexMix: A general framework for finite mixture models and latent class regression in R. Journal of Statistical Software, (8): 8, 004. doi: 0.867/jss.v0.i08. D. A. Linzer and J. B. Lewis. polca: An R package for polytomous variable latent class analysis. Journal of Statistical Software, 4(0): 9, 0. doi: 0.867/jss.v04.i0. P. D. McNicholas, A. ElSherbiny, A. F. McDaid, and T. B. Murphy. pgmm: Parsimonious Gaussian Mixture Models, 08. URL R package version...
45 References / 4 M. Plummer. rjags: Bayesian Graphical Models Using MCMC, 06. URL R package version 4-6. M. O. Prates, C. R. B. Cabral, and V. H. Lachos. mixsmsn: Fitting finite mixture of scale mixture of skew-normal distributions. Journal of Statistical Software, 54(): 0, 0. doi: 0.867/jss.v054.i. L. Scrucca, M. Fop, T. B. Murphy, and A. E. Raftery. mclust 5: Clustering, classification and density estimation using Gaussian finite mixture models. The R Journal, 8():05, 06. doi: 0.6/ada45979.
Flexible Mixture Modeling and Model-Based Clustering in R
Flexible Mixture Modeling and Model-Based Clustering in R Bettina Grün c September 2017 Flexible Mixture Modeling and Model-Based Clustering in R 0 / 170 Outline Bettina Grün c September 2017 Flexible
were generated by a model and tries to model that we recover from the data then to clusters.
Model Based Clustering Model based clustering Model based clustering assumes that thedata were generated by a model and tries to recover the original model from the data. The model that we recover from
Clustering Lecture 5: Mixture Model
Clustering Lecture 5: Mixture Model Jing Gao SUNY Buffalo 1 Outline Basics Motivation, definition, evaluation Methods Partitional Hierarchical Density-based Mixture model Spectral methods Advanced topics
Package ContaminatedMixt
Type Package Package ContaminatedMixt November 5, 2017 Title Model-Based Clustering and Classification with the Multivariate Contaminated Normal Distribution Version 1.2 Date 2017-10-26 Author Antonio
MIXMOD : a software for model-based classification with continuous and categorical data
MIXMOD : a software for model-based classification with continuous and categorical data Christophe Biernacki, Gilles Celeux, Gérard Govaert, Florent Langrognet To cite this version: Christophe Biernacki,
Package flexcwm. May 20, 2018
Type Package Title Flexible Cluster-Weighted Modeling Version 1.8 Date 2018-05-20 Author Mazza A., Punzo A., Ingrassia S. Maintainer Angelo Mazza Package flexcwm May 20, 2018 Description
Unsupervised: no target value to predict
Clustering Unsupervised: no target value to predict Differences between models/algorithms: Exclusive vs. overlapping Deterministic vs. probabilistic Hierarchical vs. flat Incremental vs. batch learning
Mixture Models and the EM Algorithm
Mixture Models and the EM Algorithm Padhraic Smyth, Department of Computer Science University of California, Irvine c 2017 1 Finite Mixture Models Say we have a data set D = {x 1,..., x N } where x i is
Package sensory. R topics documented: February 23, Type Package
Package sensory February 23, 2016 Type Package Title Simultaneous Model-Based Clustering and Imputation via a Progressive Expectation-Maximization Algorithm Version 1.1 Date 2016-02-23 Author Brian C.
Hybrid Fuzzy C-Means Clustering Technique for Gene Expression Data
Hybrid Fuzzy C-Means Clustering Technique for Gene Expression Data 1 P. Valarmathie, 2 Dr MV Srinath, 3 Dr T. Ravichandran, 4 K. Dinakaran 1 Dept. of Computer Science and Engineering, Dr. MGR University,
Note Set 4: Finite Mixture Models and the EM Algorithm
Note Set 4: Finite Mixture Models and the EM Algorithm Padhraic Smyth, Department of Computer Science University of California, Irvine Finite Mixture Models A finite mixture model with K components, for
Machine Learning. Unsupervised Learning. Manfred Huber
Machine Learning Unsupervised Learning Manfred Huber 2015 1 Unsupervised Learning In supervised learning the training data provides desired target output for learning In unsupervised learning the training
Cluster Analysis. Summer School on Geocomputation. 27 June July 2011 Vysoké Pole
Cluster Analysis Summer School on Geocomputation 27 June 2011 2 July 2011 Vysoké Pole Lecture delivered by: doc. Mgr. Radoslav Harman, PhD. Faculty of Mathematics, Physics and Informatics Comenius University,
CS 229 Midterm Review
CS 229 Midterm Review Course Staff Fall 2018 11/2/2018 Outline Today: SVMs Kernels Tree Ensembles EM Algorithm / Mixture Models [ Focus on building intuition, less so on solving specific problems. Ask
Clustering: Classic Methods and Modern Views
Clustering: Classic Methods and Modern Views Marina Meilă University of Washington mmp@stat.washington.edu June 22, 2015 Lorentz Center Workshop on Clusters, Games and Axioms Outline Paradigms for clustering
Machine Learning (BSMC-GA 4439) Wenke Liu
Machine Learning (BSMC-GA 4439) Wenke Liu 01-31-017 Outline Background Defining proximity Clustering methods Determining number of clusters Comparing two solutions Cluster analysis as unsupervised Learning
Unsupervised Learning
Networks for Pattern Recognition, 2014 Networks for Single Linkage K-Means Soft DBSCAN PCA Networks for Kohonen Maps Linear Vector Quantization Networks for Problems/Approaches in Machine Learning Supervised
Package glassomix. May 30, 2013
Package glassomix May 30, 2013 Type Package Title High dimensional Mixture Graph Models selection Version 1.1 Date 2013-05-22 Author Anani Lotsi and Ernst Wit Maintainer Anani Lotsi Depends
SGN (4 cr) Chapter 11
SGN-41006 (4 cr) Chapter 11 Clustering Jussi Tohka & Jari Niemi Department of Signal Processing Tampere University of Technology February 25, 2014 J. Tohka & J. Niemi (TUT-SGN) SGN-41006 (4 cr) Chapter
Expectation Maximization (EM) and Gaussian Mixture Models
Expectation Maximization (EM) and Gaussian Mixture Models Reference: The Elements of Statistical Learning, by T. Hastie, R. Tibshirani, J. Friedman, Springer 1 2 3 4 5 6 7 8 Unsupervised Learning Motivation
Package FMsmsnReg. March 30, 2016
Package FMsmsnReg March 30, 2016 Type Package Title Regression Models with Finite Mixtures of Skew Heavy-Tailed Errors Version 1.0 Date 2016-03-29 Author Luis Benites Sanchez and Rocio Paola Maehara and
Generative and discriminative classification techniques
Generative and discriminative classification techniques Machine Learning and Category Representation 2014-2015 Jakob Verbeek, November 28, 2014 Course website: http://lear.inrialpes.fr/~verbeek/mlcr.14.15
Clustering Relational Data using the Infinite Relational Model
Clustering Relational Data using the Infinite Relational Model Ana Daglis Supervised by: Matthew Ludkin September 4, 2015 Ana Daglis Clustering Data using the Infinite Relational Model September 4, 2015
Part I. Hierarchical clustering. Hierarchical Clustering. Hierarchical clustering. Produces a set of nested clusters organized as a
Week 9 Based in part on slides from textbook, slides of Susan Holmes Part I December 2, 2012 Hierarchical Clustering 1 / 1 Produces a set of nested clusters organized as a Hierarchical hierarchical clustering
Introduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 Clustering and EM Barnabás Póczos & Aarti Singh Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K-
Machine Learning and Data Mining. Clustering (1): Basics. Kalev Kask
Machine Learning and Data Mining Clustering (1): Basics Kalev Kask Unsupervised learning Supervised learning Predict target value ( y ) given features ( x ) Unsupervised learning Understand patterns of
Package FisherEM. February 19, 2015
Type Package Title The Fisher-EM algorithm Version 1.4 Date 2013-06-21 Author Charles Bouveyron and Camille Brunet Package FisherEM February 19, 2015 Maintainer Camille Brunet
An Efficient Model Selection for Gaussian Mixture Model in a Bayesian Framework
IEEE SIGNAL PROCESSING LETTERS, VOL. XX, NO. XX, XXX 23 An Efficient Model Selection for Gaussian Mixture Model in a Bayesian Framework Ji Won Yoon arxiv:37.99v [cs.lg] 3 Jul 23 Abstract In order to cluster
MCMC Methods for data modeling
MCMC Methods for data modeling Kenneth Scerri Department of Automatic Control and Systems Engineering Introduction 1. Symposium on Data Modelling 2. Outline: a. Definition and uses of MCMC b. MCMC algorithms
Machine Learning for OR & FE
Machine Learning for OR & FE Unsupervised Learning: Clustering Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com (Some material
Growth Mixture Modeling with Measurement Selection
Growth Mixture Modeling with Measurement Selection Abby Flynt Nema Dean arxiv:1710.06930v1 [stat.me] 18 Oct 2017 Assistant Professor Lecturer Department of Mathematics School of Mathematics and Statistics
Latent Class Modeling as a Probabilistic Extension of K-Means Clustering
Latent Class Modeling as a Probabilistic Extension of K-Means Clustering Latent Class Cluster Models According to Kaufman and Rousseeuw (1990), cluster analysis is "the classification of similar objects
10. MLSP intro. (Clustering: K-means, EM, GMM, etc.)
10. MLSP intro. (Clustering: K-means, EM, GMM, etc.) Rahil Mahdian 01.04.2016 LSV Lab, Saarland University, Germany What is clustering? Clustering is the classification of objects into different groups,
Applied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs Tutorial at CVPR 2012 Erik Sudderth Brown University
Applied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs Tutorial at CVPR 2012 Erik Sudderth Brown University NIPS 2008: E. Sudderth & M. Jordan, Shared Segmentation of Natural
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Unsupervised learning Daniel Hennes 29.01.2018 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Supervised learning Regression (linear
Beta Regression: Shaken, Stirred, Mixed, and Partitioned. Achim Zeileis, Francisco Cribari-Neto, Bettina Grün
Beta Regression: Shaken, Stirred, Mixed, and Partitioned Achim Zeileis, Francisco Cribari-Neto, Bettina Grün Overview Motivation Shaken or stirred: Single or double index beta regression for mean and/or
What is machine learning?
Machine learning, pattern recognition and statistical data modelling Lecture 12. The last lecture Coryn Bailer-Jones 1 What is machine learning? Data description and interpretation finding simpler relationship
Supplementary Material: The Emergence of. Organizing Structure in Conceptual Representation
Supplementary Material: The Emergence of Organizing Structure in Conceptual Representation Brenden M. Lake, 1,2 Neil D. Lawrence, 3 Joshua B. Tenenbaum, 4,5 1 Center for Data Science, New York University
Clustering and The Expectation-Maximization Algorithm
Clustering and The Expectation-Maximization Algorithm Unsupervised Learning Marek Petrik 3/7 Some of the figures in this presentation are taken from An Introduction to Statistical Learning, with applications
Clustering. Shishir K. Shah
Clustering Shishir K. Shah Acknowledgement: Notes by Profs. M. Pollefeys, R. Jin, B. Liu, Y. Ukrainitz, B. Sarel, D. Forsyth, M. Shah, K. Grauman, and S. K. Shah Clustering l Clustering is a technique
Overview. Monte Carlo Methods. Statistics & Bayesian Inference Lecture 3. Situation At End Of Last Week
Statistics & Bayesian Inference Lecture 3 Joe Zuntz Overview Overview & Motivation Metropolis Hastings Monte Carlo Methods Importance sampling Direct sampling Gibbs sampling Monte-Carlo Markov Chains Emcee
Machine Learning (BSMC-GA 4439) Wenke Liu
Machine Learning (BSMC-GA 4439) Wenke Liu 01-25-2018 Outline Background Defining proximity Clustering methods Determining number of clusters Other approaches Cluster analysis as unsupervised Learning Unsupervised
An Introduction to Cluster Analysis. Zhaoxia Yu Department of Statistics Vice Chair of Undergraduate Affairs
An Introduction to Cluster Analysis Zhaoxia Yu Department of Statistics Vice Chair of Undergraduate Affairs zhaoxia@ics.uci.edu 1 What can you say about the figure? signal C 0.0 0.5 1.0 1500 subjects Two
08 An Introduction to Dense Continuous Robotic Mapping
NAVARCH/EECS 568, ROB 530 - Winter 2018 08 An Introduction to Dense Continuous Robotic Mapping Maani Ghaffari March 14, 2018 Previously: Occupancy Grid Maps Pose SLAM graph and its associated dense occupancy
Supervised vs. Unsupervised Learning
Clustering Supervised vs. Unsupervised Learning So far we have assumed that the training samples used to design the classifier were labeled by their class membership (supervised learning) We assume now
SWIFT: SCALABLE WEIGHTED ITERATIVE FLOW-CLUSTERING TECHNIQUE
SWIFT: SCALABLE WEIGHTED ITERATIVE FLOW-CLUSTERING TECHNIQUE Iftekhar Naim, Gaurav Sharma, Suprakash Datta, James S. Cavenaugh, Jyh-Chiang E. Wang, Jonathan A. Rebhahn, Sally A. Quataert, and Tim R. Mosmann
An Evolutionary Algorithm with Crossover and Mutation for Model-Based Clustering
An Evolutionary Algorithm with Crossover and Mutation for Model-Based Clustering arxiv:1811.00097v1 [stat.co] 31 Oct 2018 Sharon M. McNicholas, Paul D. McNicholas and Daniel A. Ashlock Department of Mathematics
Pattern Recognition. Kjell Elenius. Speech, Music and Hearing KTH. March 29, 2007 Speech recognition
Pattern Recognition Kjell Elenius Speech, Music and Hearing KTH March 29, 2007 Speech recognition 2007 1 Ch 4. Pattern Recognition 1(3) Bayes Decision Theory Minimum-Error-Rate Decision Rules Discriminant
IMAGE ANALYSIS, CLASSIFICATION, and CHANGE DETECTION in REMOTE SENSING
SECOND EDITION IMAGE ANALYSIS, CLASSIFICATION, and CHANGE DETECTION in REMOTE SENSING ith Algorithms for ENVI/IDL Morton J. Canty с*' Q\ CRC Press Taylor &. Francis Group Boca Raton London New York CRC
Monte Carlo for Spatial Models
Monte Carlo for Spatial Models Murali Haran Department of Statistics Penn State University Penn State Computational Science Lectures April 2007 Spatial Models Lots of scientific questions involve analyzing
Clustering algorithms
Clustering algorithms Machine Learning Hamid Beigy Sharif University of Technology Fall 1393 Hamid Beigy (Sharif University of Technology) Clustering algorithms Fall 1393 1 / 22 Table of contents 1 Supervised
Unsupervised Learning: Clustering
Unsupervised Learning: Clustering Vibhav Gogate The University of Texas at Dallas Slides adapted from Carlos Guestrin, Dan Klein & Luke Zettlemoyer Machine Learning Supervised Learning Unsupervised Learning
CS 1675 Introduction to Machine Learning Lecture 18. Clustering. Clustering. Groups together similar instances in the data sample
CS 1675 Introduction to Machine Learning Lecture 18 Clustering Milos Hauskrecht milos@cs.pitt.edu 539 Sennott Square Clustering Groups together similar instances in the data sample Basic clustering problem:
Mixture Models and EM
Table of Content Chapter 9 Mixture Models and EM -means Clustering Gaussian Mixture Models (GMM) Expectation Maximiation (EM) for Mixture Parameter Estimation Introduction Mixture models allows Complex
Analysis of Incomplete Multivariate Data
Analysis of Incomplete Multivariate Data J. L. Schafer Department of Statistics The Pennsylvania State University USA CHAPMAN & HALL/CRC A CR.C Press Company Boca Raton London New York Washington, D.C.
Predictive Analytics: Demystifying Current and Emerging Methodologies. Tom Kolde, FCAS, MAAA Linda Brobeck, FCAS, MAAA
Predictive Analytics: Demystifying Current and Emerging Methodologies Tom Kolde, FCAS, MAAA Linda Brobeck, FCAS, MAAA May 18, 2017 About the Presenters Tom Kolde, FCAS, MAAA Consulting Actuary Chicago,
k-means demo Administrative Machine learning: Unsupervised learning" Assignment 5 out
Machine learning: Unsupervised learning" David Kauchak cs Spring 0 adapted from: http://www.stanford.edu/class/cs76/handouts/lecture7-clustering.ppt http://www.youtube.com/watch?v=or_-y-eilqo Administrative
Package clustvarsel. April 9, 2018
Version 2.3.2 Date 2018-04-09 Package clustvarsel April 9, 2018 Title Variable Selection for Gaussian Model-Based Clustering Description Variable selection for Gaussian model-based clustering as implemented
ECE 5424: Introduction to Machine Learning
ECE 5424: Introduction to Machine Learning Topics: Unsupervised Learning: Kmeans, GMM, EM Readings: Barber 20.1-20.3 Stefan Lee Virginia Tech Tasks Supervised Learning x Classification y Discrete x Regression
Classification. Vladimir Curic. Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University
Classification Vladimir Curic Centre for Image Analysis Swedish University of Agricultural Sciences Uppsala University Outline An overview on classification Basics of classification How to choose appropriate
Chapter 3: Supervised Learning
Chapter 3: Supervised Learning Road Map Basic concepts Evaluation of classifiers Classification using association rules Naïve Bayesian classification Naïve Bayes for text classification Summary 2 An example
CLASSIFICATION AND CHANGE DETECTION
IMAGE ANALYSIS, CLASSIFICATION AND CHANGE DETECTION IN REMOTE SENSING With Algorithms for ENVI/IDL and Python THIRD EDITION Morton J. Canty CRC Press Taylor & Francis Group Boca Raton London NewYork CRC
Chapter 6 Continued: Partitioning Methods
Chapter 6 Continued: Partitioning Methods Partitioning methods fix the number of clusters k and seek the best possible partition for that k. The goal is to choose the partition which gives the optimal
Package clusternomics
Type Package Package clusternomics March 14, 2017 Title Integrative Clustering for Heterogeneous Biomedical Datasets Version 0.1.1 Author Evelina Gabasova Maintainer Evelina Gabasova
Machine Learning. B. Unsupervised Learning B.1 Cluster Analysis. Lars Schmidt-Thieme
Machine Learning B. Unsupervised Learning B.1 Cluster Analysis Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute for Computer Science University of Hildesheim, Germany
Image Analysis, Classification and Change Detection in Remote Sensing
Image Analysis, Classification and Change Detection in Remote Sensing WITH ALGORITHMS FOR ENVI/IDL Morton J. Canty Taylor &. Francis Taylor & Francis Group Boca Raton London New York CRC is an imprint
http://www.xkcd.com/233/ Text Clustering David Kauchak cs160 Fall 2009 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture17-clustering.ppt Administrative 2 nd status reports Paper review
Clustering CS 550: Machine Learning
Clustering CS 550: Machine Learning This slide set mainly uses the slides given in the following links: http://www-users.cs.umn.edu/~kumar/dmbook/ch8.pdf http://www-users.cs.umn.edu/~kumar/dmbook/dmslides/chap8_basic_cluster_analysis.pdf
Package longclust. March 18, 2018
Package longclust March 18, 2018 Type Package Title Model-Based Clustering and Classification for Longitudinal Data Version 1.2.2 Date 2018-03-18 Author Paul D. McNicholas [aut, cre], K. Raju Jampani [aut]
On Clustering: Mixture Model Averaging with the. Generalized Hyperbolic Distribution
On Clustering: Mixture Model Averaging with the Generalized Hyperbolic Distribution ON CLUSTERING: MIXTURE MODEL AVERAGING WITH THE GENERALIZED HYPERBOLIC DISTRIBUTION BY SARAH RICCIUTI, B.Sc. a thesis
Data Mining. Clustering. Hamid Beigy. Sharif University of Technology. Fall 1394
Data Mining Clustering Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Data Mining Fall 1394 1 / 31 Table of contents 1 Introduction 2 Data matrix and
MCLUST Version 3 for R: Normal Mixture Modeling and Model-Based Clustering
MCLUST Version 3 for R: Normal Mixture Modeling and Model-Based Clustering Chris Fraley and Adrian E. Raftery Technical Report No. 504 Department of Statistics University of Washington Box 354322 Seattle,
Package clustmd. May 8, 2017
Title Model Based Clustering for Mixed Data Version 1.2.1 Package May 8, 2017 Model-based clustering of mixed data (i.e. data which consist of continuous, binary, ordinal or nominal variables) using a
Unsupervised Learning and Clustering
Unsupervised Learning and Clustering Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2009 CS 551, Spring 2009 c 2009, Selim Aksoy (Bilkent University)
Machine Learning. Supervised Learning. Manfred Huber
Machine Learning Supervised Learning Manfred Huber 2015 1 Supervised Learning Supervised learning is learning where the training data contains the target output of the learning system. Training data D
Cluster Analysis: Agglomerate Hierarchical Clustering
Cluster Analysis: Agglomerate Hierarchical Clustering Yonghee Lee Department of Statistics, The University of Seoul Oct 29, 2015 Contents 1 Cluster Analysis Introduction Distance matrix Agglomerative Hierarchical
Introduction to Pattern Recognition Part II. Selim Aksoy Bilkent University Department of Computer Engineering
Introduction to Pattern Recognition Part II Selim Aksoy Bilkent University Department of Computer Engineering saksoy@cs.bilkent.edu.tr RETINA Pattern Recognition Tutorial, Summer 2005 Overview Statistical
COMS 4771 Clustering. Nakul Verma
COMS 4771 Clustering Nakul Verma Supervised Learning Data: Supervised learning Assumption: there is a (relatively simple) function such that for most i Learning task: given n examples from the data, find
Clustering. Supervised vs. Unsupervised Learning
Clustering Supervised vs. Unsupervised Learning So far we have assumed that the training samples used to design the classifier were labeled by their class membership (supervised learning) We assume now
Learning Generative Graph Prototypes using Simplified Von Neumann Entropy
Learning Generative Graph Prototypes using Simplified Von Neumann Entropy Lin Han, Edwin R. Hancock and Richard C. Wilson Department of Computer Science The University of York YO10 5DD, UK Graph representations
Model-Based Cluster and Discriminant
Model-Based Cluster and Discriminant Analysis with the mixmod Software Christophe Biernacki a, Gilles Celeux b Gérard Govaert c Florent Langrognet d a UMR 8524, CNRS & Université de Lille 1, 59655 Villeneuve
CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS
CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CHAPTER 4 CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS 4.1 Introduction Optical character recognition is one of
Data Clustering. Danushka Bollegala
Data Clustering Danushka Bollegala Outline Why cluster data? Clustering as unsupervised learning Clustering algorithms k-means, k-medoids agglomerative clustering Brown s clustering Spectral clustering
Quantitative Biology II!
Quantitative Biology II! Lecture 3: Markov Chain Monte Carlo! March 9, 2015! 2! Plan for Today!! Introduction to Sampling!! Introduction to MCMC!! Metropolis Algorithm!! Metropolis-Hastings Algorithm!!
Clustering in R d. Clustering. Widely-used clustering methods. The k-means optimization problem CSE 250B
Clustering in R d Clustering CSE 250B Two common uses of clustering: Vector quantization Find a finite set of representatives that provides good coverage of a complex, possibly infinite, high-dimensional
Warped Mixture Models
Warped Mixture Models Tomoharu Iwata, David Duvenaud, Zoubin Ghahramani Cambridge University Computational and Biological Learning Lab March 11, 2013 OUTLINE Motivation Gaussian Process Latent Variable
Preface to the Second Edition. Preface to the First Edition. 1 Introduction 1
Preface to the Second Edition Preface to the First Edition vii xi 1 Introduction 1 2 Overview of Supervised Learning 9 2.1 Introduction... 9 2.2 Variable Types and Terminology... 9 2.3 Two Simple Approaches
Supplementary Material sppmix: Poisson point process modeling using normal mixture models
Supplementary Material sppmix: Poisson point process modeling using normal mixture models Athanasios C. Micheas and Jiaxun Chen Department of Statistics University of Missouri April 19, 2018 1 The sppmix
Contents. Preface to the Second Edition
Preface to the Second Edition v 1 Introduction 1 1.1 What Is Data Mining?....................... 4 1.2 Motivating Challenges....................... 5 1.3 The Origins of Data Mining....................
Overview Citation. ML Introduction. Overview Schedule. ML Intro Dataset. Introduction to Semi-Supervised Learning Review 10/4/2010
INFORMATICS SEMINAR SEPT. 27 & OCT. 4, 2010 Introduction to Semi-Supervised Learning Review 2 Overview Citation X. Zhu and A.B. Goldberg, Introduction to Semi- Supervised Learning, Morgan & Claypool Publishers,
Clustering. CS294 Practical Machine Learning Junming Yin 10/09/06
Clustering CS294 Practical Machine Learning Junming Yin 10/09/06 Outline Introduction Unsupervised learning What is clustering? Application Dissimilarity (similarity) of objects Clustering algorithm K-means,
Package tidylpa. March 28, 2018
Type Package Title Easily Carry Out Latent Profile Analysis Version 0.1.3 Package tidylpa March 28, 2018 An interface to the 'mclust' package to easily carry out latent profile analysis (``LPA''). Provides
Hierarchical Mixture Models for Nested Data Structures
Hierarchical Mixture Models for Nested Data Structures Jeroen K. Vermunt 1 and Jay Magidson 2 1 Department of Methodology and Statistics, Tilburg University, PO Box 90153, 5000 LE Tilburg, Netherlands
Clustering and Dissimilarity Measures. Clustering. Dissimilarity Measures. Cluster Analysis. Perceptually-Inspired Measures
Clustering and Dissimilarity Measures Clustering APR Course, Delft, The Netherlands Marco Loog May 19, 2008 1 What salient structures exist in the data? How many clusters? May 19, 2008 2 Cluster Analysis
K-Means Clustering. Sargur Srihari
K-Means Clustering Sargur srihari@cedar.buffalo.edu 1 Topics in Mixture Models and EM Mixture models K-means Clustering Mixtures of Gaussians Maximum Likelihood EM for Gaussian mistures EM Algorithm Gaussian
CS 2750 Machine Learning. Lecture 19. Clustering. CS 2750 Machine Learning. Clustering. Groups together similar instances in the data sample
Lecture 9 Clustering Milos Hauskrecht milos@cs.pitt.edu 539 Sennott Square Clustering Groups together similar instances in the data sample Basic clustering problem: distribute data into k different groups
mixmod User s Guide MIXture MODelling Software
User s Guide : mixmod V.2.1.1 mixmod User s Guide MIXture MODelling Software High Performance Model-Based Cluster and Discriminant Analysis http://www-math.univ-fcomte.fr/mixmod/index.php Christophe Biernacki
University of Florida CISE department Gator Engineering. Clustering Part 5
Clustering Part 5 Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville SNN Approach to Clustering Ordinary distance measures have problems Euclidean
IBL and clustering. Relationship of IBL with CBR
IBL and clustering Distance based methods IBL and knn Clustering Distance based and hierarchical Probability-based Expectation Maximization (EM) Relationship of IBL with CBR + uses previously processed
A Handbook of Statistical Analyses Using R. Brian S. Everitt and Torsten Hothorn
A Handbook of Statistical Analyses Using R Brian S. Everitt and Torsten Hothorn CHAPTER 7 Density Estimation: Erupting Geysers and Star Clusters 7.1 Introduction 7.2 Density Estimation The three kernel