CSCI 5090/7090- Machine Learning. Spring Mehdi Allahyari Georgia Southern University

Transcription

1 CSCI 5090/7090- Machie Learig Sprig 018 Mehdi Allahyari Georgia Souther Uiversity Clusterig (slides borrowed from Tom Mitchell, Maria Floria Balca, Ali Borji, Ke Che) 1

2 Clusterig, Iformal Goals Goal: Automatically partitio ulabeled data ito groups of similar datapoits. Questio: Whe ad why would we wat to do this? Useful for: Automatically orgaizig data. Uderstadig hidde structure i data. Preprocessig for further aalysis. Represetig high-dimesioal data i a low-dimesioal space (e.g., for visualizatio purposes).

3 Clusterig, Iformal Goals Goal: Automatically partitio ulabeled data ito groups of similar datapoits. Questio: Whe ad why would we wat to do this? Useful for: Automatically orgaizig data. Uderstadig hidde structure i data. Preprocessig for further aalysis. Represetig high-dimesioal data i a low-dimesioal space (e.g., for visualizatio purposes). 3

4 Applicatios Cluster ews articles or web pages or search results by topic. Cluster protei sequeces by fuctio or gees accordig to expressio profile. Cluster users of social etworks by iterest (commuity detectio). Facebook etwork Twitter Network 4

5 Applicatios Cluster ews articles or web pages or search results by topic. Cluster protei sequeces by fuctio or gees accordig to expressio profile. Cluster users of social etworks by iterest (commuity detectio). Facebook etwork Twitter Network 5

6 Clusterig Groups together similar istaces i the data sample Basic clusterig problem: distribute data ito k differet groups such that data poits similar to each other are i the same group Similarity betwee data poits is defied i terms of some distace metric (ca be chose) Clusterig is useful for: Similarity/Dissimilarity aalysis Aalyze what data poits i the sample are close to each other Dimesioality reductio High dimesioal data replaced with a group (cluster) label 6

7 Example We see data poits ad wat to partitio them ito groups Which data poits belog together?

8 Example We see data poits ad wat to partitio them ito the groups Which data poits belog together?

9 Example We see data poits ad wat to partitio them ito the groups Requires a distace metric to tell us what poits are close to each other ad are i the same group 3 Euclidea distace

10 Example A set of patiet cases We wat to partitio them ito groups based o similarities Patiet # Age Sex Heart Rate Blood pressure Patiet 1 55 M 85 15/80 Patiet 6 M /85 Patiet 3 67 F 80 16/86 Patiet 4 65 F /90 Patiet 5 70 M /85 10

11 Example A set of patiet cases We wat to partitio them ito the groups based o similarities Patiet # Age Sex Heart Rate Blood pressure Patiet 1 55 M 85 15/80 Patiet 6 M /85 Patiet 3 67 F 80 16/86 Patiet 4 65 F /90 Patiet 5 70 M /85 How to desig the distace metric to quatify similarities? 11

12 Clusterig Example. Distace Measures I geeral, oe ca choose a arbitrary distace measure. Properties of distace metrics: Assume data etries a, b Positiveess: d( a, b) 0 Symmetry: d( a, b) d( b, a) Idetity: d( a, a) 0 Triagle iequality: d( a, c) d( a, b) d( b, c) 1

13 Distace Measures Assume pure real-valued data-poits: What distace metric to use? 13

14 Distace Measures Assume pure real-valued data-poits: What distace metric to use? Euclidia: works for a arbitrary k-dimesioal space d( a, b) k i 1 ( a i b i ) 14

15 Distace Measures Assume pure real-valued data-poits: What distace metric to use? Squared Euclidia: works for a arbitrary k-dimesioal space d ( a, b) k i 1 ( a i b i ) 15

16 Distace Measures Assume pure real-valued data-poits: Mahatta distace: works for a arbitrary k-dimesioal space d( a, b) Etc... k i 1 a i b i 16

17 Clusterig Algorithms K-meas algorithm suitable oly whe data poits have cotiuous values; groups are defied i terms of cluster ceters (also called meas). Refiemet of the method to categorical values: K-medoids Probabilistic methods (with EM) Latet variable models: class (cluster) is represeted by a latet (hidde) variable value Every poit goes to the class with the highest posterior Examples: mixture of Gaussias, Naïve Bayes with a hidde class Hierarchical methods Agglomerative Divisive 17

18 Itroductio Partitioig Clusterig Approach a typical clusterig aalysis approach via iteratively partitioig traiig data set to lear a partitio of the give data space learig a partitio o a data set to produce several oempty clusters (usually, the umber of clusters give i advace) i priciple, optimal partitio achieved via miimisig the sum of squared distace to its represetative object i each cluster K E = S d ( x, m k k= 1S xî C k ) e.g., Euclidea distace N ( x, mk ) = å( x mk = 1 d - ) 18

19 Itroductio Give a K, fid a partitio of K clusters to optimize the chose partitioig criterio (cost fuctio) o global optimum: exhaustively search all partitios The K-meas algorithm: a heuristic method o o K-meas algorithm (MacQuee 67): each cluster is represeted by the ceter of the cluster ad the algorithm coverges to stable cetriods of clusters. K-meas algorithm is the simplest partitioig method for clusterig aalysis ad widely used i data miig applicatios. 19

20 K-meas Algorithm Give the cluster umber K, the K-meas algorithm is carried out i three steps after iitializatio: Iitialisatio: set seed poits (radomly) Assig each object to the cluster of the earest seed poit measured with a specific distace metric Compute ew seed poits as the cetroids of the clusters of the curret partitio (the cetroid is the ceter, i.e., mea poit, of the cluster) Go back to Step 1), stop whe o more ew assigmet (i.e., membership i each cluster o loger chages) 0

21 K-meas Clusterig Choose a umber of clusters k Iitialize cluster ceters µ 1, µ k Could pick k data poits ad set cluster ceters to these poits Or could radomly assig poits to clusters ad take meas of clusters For each data poit, compute the cluster ceter it is closest to (usig some distace measure) ad assig the data poit to this cluster Re-compute cluster ceters (mea of data poits i cluster) Stop whe there are o ew re-assigmets

22 Example Problem Suppose we have 4 types of medicies ad each has two attributes (ph ad weight idex). Our goal is to group these objects ito K= group of medicie. Medicie Weight ph- Idex A 1 1 C D B 1 C 4 3 D 5 4 A B

23 Example Step 1: Use iitial seed poits for partitioig c = A, c 1 = B D C Euclidea distace A B d( D, c d( D, c 1 ) = ) = (5-1) (5 - ) + (4-1) + (4-1) = 5 = 4.4 Assig each object to the cluster with the earest seed poit 3

24 Example Step : Compute ew cetroids of the curret partitio Kowig the members of each cluster, ow we compute the ew cetroid of each group based o these ew memberships. c 1 = (1, 1) c + = ç è 11 = ( 3 æ , 3, ) 3 4 ö ø 4

25 Example Step : Reew membership based o ew cetroids Compute the distace of all objects to the ew cetroids Assig the membership to objects 5

26 Example Step 3: Repeat the first two steps util its covergece Kowig the members of each cluster, ow we compute the ew cetroid of each group based o these ew memberships. c c 1 æ ö = ç, = è ø æ ö = ç, = è ø 1 (1, 1 (4, 1) 1 3 ) 6

27 Example Step 3: Repeat the first two steps util its covergece Compute the distace of all objects to the ew cetroids Stop due to o ew assigmet Membership i each cluster o loger chage 7

28 Exercise For the medicie data set, use K-meas with the Mahatta distace metric for clusterig aalysis by settig K= ad iitialisig seeds as C1 = A ad C = C. Aswer three questios as follows: 1. How may steps are required for covergece?. What are memberships of two clusters after covergece? 3. What are cetroids of two clusters after covergece? Medicie Weight ph- Idex A 1 1 C D B 1 C 4 3 D 5 4 A B 8

29 Euclidea k-meas Clusterig Iput: A set of datapoits x 1, x,, x i R d target #clusters k Output: k represetatives c 1, c,, c k R d Objective: choose c 1, c,, c k R d to miimize i=1 mi j 1,,k x i c j 9

30 Euclidea k-meas Clusterig Iput: A set of datapoits x 1, x,, x i R d target #clusters k Output: k represetatives c 1, c,, c k R d Objective: choose c 1, c,, c k R d to miimize i=1 mi j 1,,k x i c j Natural assigmet: each poit assiged to its closest ceter, leads to a Vorooi partitio. 30

31 Euclidea k-meas Clusterig Iput: A set of datapoits x 1, x,, x i R d target #clusters k Output: k represetatives c 1, c,, c k R d Objective: choose c 1, c,, c k R d to miimize i=1 mi j 1,,k x i c j Computatioal complexity: NP hard: eve for k = [Dagupta 08] or d = [Mahaja-Nimbhorkar-Varadaraja09] There are a couple of easy cases 31

32 A Easy Case for k-meas: k=1 Iput: A set of datapoits x 1, x,, x i R d Output: c R d to miimize i=1 x i c 1 Solutio: The optimal choice is μ = i=1 x i Idea: bias/variace like decompositio So, the optimal choice for c is μ. 3

33 k-meas Clusterig Issues Computatioal complexity O(tK), where is umber of objects, K is umber of clusters, ad t is umber of iteratios. Normally, K, t <<. Local optimum sesitive to iitial seed poits coverge to a local optimum: maybe a uwated solutio Other problems Need to specify K, the umber of clusters, i advace Uable to hadle oisy data ad outliers (K-Medoids algorithm) Not suitable for discoverig clusters with o-covex shapes Applicable oly whe mea is defied, the what about categorical data? (K-mode algorithm) how to evaluate the K-mea performace? 33

34 Hierarchical Clusterig Hierarchical Clusterig All topics sports fashio soccer teis Gucci Lacoste A hierarchy might be more atural. Differet users might care about differet levels of graularity or eve pruigs. 34

35 Hierarchical Clusterig Top-dow (divisive) Partitio data ito -groups (e.g., -meas) Recursively cluster each group. Bottom-Up (agglomerative) Start with every poit i its ow cluster. All topics Repeatedly merge the closest two clusters. Differet defs of closest give differet algorithms. sports fashio soccer teis Gucci Lacoste 35

36 Bottom-Up (agglomerative) Bottom-Up (agglomerative) Have a distace measure o pairs of objects. All topics d(x,y) distace betwee x ad y sports fashio E.g., # keywords i commo, edit distace, etc soccer teis Gucci Lacoste Sigle likage: dist A, B = mi x A,x B dist(x, x ) Complete likage: dist A, B = max dist(x, x A,x B x ) Average likage: dist A, B = avg dist(x, x ) x A,x B Wards method 36

37 Sigle Likage Bottom-up (agglomerative) Sigle Likage Start with every poit i its ow cluster. Repeatedly merge the closest two clusters. Sigle likage: dist A, B = mi dist(x, x A,x B x ) Dedogram 4 A B C D E 5 A B C D E F 3 A B C 1 A B D E A B C D E F 37

38 Sigle Likage Bottom-up (agglomerative) Sigle Likage Start with every poit i its ow cluster. Repeatedly merge the closest two clusters. Sigle likage: dist A, B = mi dist(x, x A,x B x ) Oe way to thik of it: at ay momet, we see coected compoets of the graph where coect ay two pts of distace < r. Watch as r grows (oly -1 relevat values because we oly we merge at value of r correspodig to values of r i differet clusters) A B C D E F 38

39 Complete Likage Bottom-up (agglomerative) Complete Likage Start with every poit i its ow cluster. Repeatedly merge the closest two clusters. Complete likage: dist A, B = max dist(x, x A,x B x ) Oe way to thik of it: keep max diameter as small as possible at ay level. 5 A B C D E F 3 A B C 4 DEF 1 A B D E A B C D E F 39

40 Complete Likage Bottom-up (agglomerative) Complete Likage Start with every poit i its ow cluster. Repeatedly merge the closest two clusters. Complete likage: dist A, B = max dist(x, x A,x B x ) Oe way to thik of it: keep max diameter as small as possible A B C D E F 40

41 Other Clusterig Algorithms Spectral clusterig Uses similarity matrix ad its spectral decompositio (eigevalues ad eigevectors) Multidimesioal scalig techiques ofte used i data visualizatio for explorig similarities or dissimilarities i data. 41