Unsupervised Learning and Clustering

Transcription

1 Unsupervsed Learnng and Clusterng

2 Why consder unlabeled samples?. Collectng and labelng large set of samples s costly Gettng recorded speech s free, labelng s tme consumng 2. Classfer could be desgned on small set of labeled samples and tuned on a large unlabeled set 3. Tran on large unlabeled set and use supervson on groupngs found 4. Characterstcs of patterns may change wth tme 5. Unsupervsed methods can be used to fnd useful features 6. Exploratory data analyss may dscover presence of sgnfcant subclasses affectng desgn

3 Mxture Denstes and Identfablty Samples come from c classes Prors are nown Pω ) Forms of the class-condtonal are nown Values for ther parameters are unnown Probablty densty functon of samples s: px, θ ) c px ω, θ ) P ω )

4 Gradent Ascent for Mxtures Mxture densty: Lelhood of observed samples: Log-lelhood: Gradent w.r.t. θ : MLE must satsfy: ) ), x ) x, c P p p ω θ ω θ n x p D p ) ) θ θ n x p l ) ln θ c n P x p x p l ) ), ) ω θ ω θ θ θ 0 ) ˆ, ln ˆ), n x p x P θ ω θ ω θ

5 Gaussan Mxture Unnown mean vectors, yelds Leadng to an teratve scheme for mprovng estmates t n n x P x x P ) ˆ,.. ˆ ˆ where ˆ), ˆ), ˆ c µ µ µ µ ω µ ω µ )) ˆ, ) ˆ + n x x P µ ω µ )) ˆ, n x P µ ω

6 -means clusterng Gaussan case wth all parameters unnown leads to a formulaton: begn ntalze n, c, µ,µ 2,..,µ c do classfy n samples accordng to nearest µ recompute µ untl no change n µ end

7 -means clusterng wth one feature One-dmensonal example Sx startng ponts lead local maxma whereas two for both of whch µ 0) µ 2 0) lead to a saddle pont

8 -means clusterng wth two features Two-dmensonal example There are three means and there are three steps n the teraton. Vorono tesselatons based on means are shown

9 Data Descrpton and Clusterng

10 Data Descrpton Learnng the structure of multdmensonal patterns from a set of unlabelled samples Form clouds of ponts n d-dmensonal space If data were from a sngle normal dstrbuton, mean and covarance metrc would suffce as a descrpton

11 Data sets havng dentcal statstcs upto second order,.e., same µ andσ

12 Mxture of c normal dstrbutons approach Estmatng parameters s non-trval Assumpton of partcular parametrc forms can lead to poor or meanngless results Alternatvely use nonparametrc approach: peas or modes can ndcate clusters If goal s to fnd sub-classes use clusterng procedures

13 Smlarty Measures Two Issues. How to measure smlarty between samples? 2. How to evaluate parttonng? If dstance s a good measure of dssmlarty dstance between samples n same cluster must be smaller than dstance between samples n dfferent clusters Two samples belong to the same cluster f dstance between them s less than a threshold d 0 Dstance threshold affects number and sze of clusters

14 Smlarty Measures for Clusterng Mnows Metrc d ' q dx, x' ) x x Metrc based on data tself: Mahanalobs dstance Angle between vectors as smlarty /q q 2 s Eucldean, q s Manhattan or cty bloc metrc s x, x') t x x x x' ' Cosne of angle between vectors s nvarant to rotaton and dlaton but not translaton and general lnear transformatons

15 Bnary Feature Smlarty Measures s x, x') ' x x t x x' Numerator no of attrbutes possessed by both x and x Denomnator x t xx t x ) /2 s geometrc mean of no of attrbutes possessed by x and x xt x ' s x, x') Fracton of attrbutes shared d s x, x') t x x + t ' x x x' t x ' t x x ' Tanmoto coeffcent: Rato of number of shared attrbutes to number possessed by x or x

16 Issues n Choce of Smlarty Functon Tanmoto coeffcent used n Informaton Retreval and Taxonomy Fundamental ssues n Measurement Theory Combnng features s trcy: nches versus meters Nomnal, ordnal, nterval and rato scales

17 Crteron Functons for Clusterng Sum of squared errors crteron Mean of samples n D m 2 x x D J e c x D x m 2 Crteron s not best when two clusters are of unequal sze Sutable when they are compact clouds

18 Related Mnmum Varance Crtera J e 2 where c s n s n 2 x D x' D x x' 2 Can be replaced by other smlarty functon sx,x ) Optmal partton extremzes the crteron functon

19 Scatter Crtera Derved from Scatter Matrces Trace crteron Determnant Crteron Invarant Crtera

20 Herarchcal Clusterng

21 Dendrogram

22

23 Agglomeratve Algorthm

24 Nearest Neghbor Algorthm

25 Farthest Neghbor Algorthm

26 How to determne nearest clusters