Machine Learning: Algorithms and Applications

Transcription

1 14/05/1 Machne Learnng: Algorthms and Applcatons Florano Zn Free Unversty of Bozen-Bolzano Faculty of Computer Scence Academc Year Lecture 10: 14 May 01 Unsupervsed Learnng cont Sldes courtesy of Bng Lu: 1

2 14/05/1 Road map Basc concepts K-means algorthm Representaton of clusters Herarchcal clusterng Dstance functons Data standardzaton Handlng med attrbutes Whch clusterng algorthm to use? Cluster evaluaton Summary Herarchcal Clusterng Produce a nested sequence of clusters, a tree, also called dendrogram Sngleton clusters are at the bottom of the three One root clusters covers all the data ponts Sblngs clusters partton the data ponts of the common parent

3 14/05/1 Types of herarchcal clusterng Agglomeratve bottom up clusterng: t bulds the dendrogram tree from the bottom level, and q merges the most smlar or nearest par of clusters q stops when all the data ponts are merged nto a sngle cluster.e., the root cluster Dvsve top down clusterng: t starts wth all data ponts n one cluster, the root q splts the root nto a set of chld clusters q q each chld cluster s recursvely dvded further stops when only sngleton clusters of ndvdual data ponts reman Agglomeratve clusterng It s more popular then dvsve methods At the begnnng, each data pont forms a cluster also called a node Merge nodes/clusters that have the least dstance Go on mergng Eventually all nodes belong to one cluster 3

4 14/05/1 Agglomeratve clusterng algorthm An eample: workng of the algorthm 4

5 14/05/1 Measurng the dstance of two clusters A few ways to measure dstances of two clusters q k-means uses only the dstances between centrods Dfferent varatons of the algorthm q Sngle lnk q Complete lnk q Average lnk q Centrods q Sngle lnk method The dstance between two clusters s the dstance between two closest data ponts n the two clusters q one data pont from each cluster It can fnd arbtrarly shaped clusters, but q It may cause the undesrable chan effect by nosy ponts n black The two natural clusters n red are not found 5

6 14/05/1 Complete lnk method The dstance between two clusters s the dstance of two furthest data ponts n the two clusters It s senstve to outlers n black because they are far away It usually produces better clusters than the sngle-lnk method Average lnk and centrod methods Average lnk method A compromse between q the senstvty of complete-lnk clusterng to outlers and q the tendency of sngle-lnk clusterng to form long chans that do not correspond to the ntutve noton of clusters as compact, sphercal obects The dstance between two clusters s the average dstance of all par-wse dstances between the data ponts n two clusters Centrod method the dstance between two clusters s the dstance between ther centrods 6

7 14/05/1 The complety All the herarchcal algorthms are at least On q n s the number of data ponts Sngle lnk can be done n On Complete and average lnks can be done n On log n Due the complety, herarchcal algorthms are hard to use for large data sets q q Perform herarchcal clusterng on a sample of data ponts and then assgn the others by dstance or by supervsed learnng see lecture 9 Use scale-up methods e.g., BIRCH that fnd many small clusters usng an effcent algorthm use these clusters as the startng nodes for the herarchcal clusterng Road map Basc concepts K-means algorthm Representaton of clusters Herarchcal clusterng Dstance functons Data standardzaton Handlng med attrbutes Whch clusterng algorthm to use? Cluster evaluaton Summary 7

8 14/05/1 Dstance functons Key to clusterng q smlarty and dssmlarty are other commonly used terms There are numerous dstance functons for q Dfferent types of data Numerc data Nomnal data q Dfferent specfc applcatons Dstance functons for numerc attrbutes We denote dstance wth dst,, where and are data ponts vectors Most commonly used functons are q Eucldean dstance and q Manhattan cty block dstance They are specal cases of Mnkowsk dstance 1 h dst, 1! 1 h! h... r! r h h s postve nteger, r s the number of attrbutes 8

9 14/05/1 9 Eucldean dstance and Manhattan dstance If h =, t s the Eucldean dstance If h = 1, t s the Manhattan dstance Weghted Eucldean dstance , r r dst =..., 1 1 r r dst = , r r r w w w dst = Squared dstance and Chebychev dstance Squared Eucldean dstance: to place progressvely greater weght on data ponts that are further apart Chebychev dstance: one wants to defne two data ponts as dfferent f they are dfferent on any one of the attrbutes , r r dst = dst, ma 1! 1,!,, r! r

10 14/05/1 Dstance functons for bnary and nomnal attrbutes Bnary attrbute: has two values or states but no orderng relatonshps, q E.g., Gender: female and male q The values are conventonally represented by 1 and 0 We use a confuson matr to ntroduce the dstance functons/measures Let the th and th data ponts be and vectors Confuson matr 10

11 14/05/1 Symmetrc bnary attrbutes A bnary attrbute s symmetrc f both of ts states 0 and 1 have equal mportance, e.g., female and male of the attrbute Gender Dstance functon: Smple Matchng Dstance, proporton of msmatches of ther values dst, b c a b c d 1 There are varatons, addng weghts dst, To msmatches b c a d b c To matches dst, b c a d b c Symmetrc bnary attrbutes: eample 1 and are two data ponts Each of the 7 attrbutes s symmetrc bnary The smple matchng dstance s b c dst 1, a b c d = 1 1 = 3 7 = 0.49 If there s a weght on msmatches b c dst 1, a b c d = 1 1 = 6 10 =

12 14/05/1 Asymmetrc bnary attrbutes Asymmetrc: f one of the states s more mportant or valuable than the other q By conventon, state 1 represents the more mportant state, whch s typcally the rare or nfrequent state q Jaccard dstance s a popular measure dst, q There are varatons, addng weghts b c a b c To msmatches dst, b c a b c To matches of the mportant state dst, b c a b c Asymmetrc bnary attrbutes: eample 1 and are two data ponts Each of the 7 attrbutes s asymmetrc bnary The Jaccard dstance s dst 1, b c a b c = 1 1 = 3 5 = 0.6 If there s a weght on matches of the mportant state b c dst 1, a b c = 1 * 1 = 3 7 =

13 14/05/1 Nomnal attrbutes Nomnal attrbutes: wth more than two states or values q the commonly used dstance measure s also based on the smple matchng method q Gven two data ponts and, let the number of attrbutes be r, and the number of values that match n and be q dst, r q r 3 Road map Basc concepts K-means algorthm Representaton of clusters Herarchcal clusterng Dstance functons Data standardzaton Handlng med attrbutes Whch clusterng algorthm to use? Cluster evaluaton Summary 13

14 14/05/1 Data standardzaton In the Eucldean space, standardzaton of attrbutes s recommended so that all attrbutes can have equal mpact on the computaton of dstances Consder the followng par of data ponts q : 0.1, 0 and : 0.9, 70 dst, 0.9! ! 0 = The dstance s almost completely domnated by Standardze attrbutes: to force the attrbutes to have a common value range Interval-scaled attrbutes Ther values are real numbers followng a lnear scale q E.g., the dfference n Age between 10 and 0 s the same as that between 40 and 50 q The key dea s that ntervals keep the same mportance through out the scale Two man approaches to standardze nterval scaled attrbutes, range and z-score 14

15 14/05/1 Interval-scaled attrbutes cont Range: transform the values of an attrbute f so that they are between 0 and 1 rg f f! mn f ma f! mn f Z-score: transform the values of an attrbute f based on the mean and standard devaton of the attrbute q Indcates how far and n what drecton the value devates from the mean q The devaton s epressed n unts of the standard devaton of the attrbute µ f = 1 n n! =1 f! f = " n =1 f! µ f n!1 Z-score: z f f! µ f! f Rato-scaled attrbutes Numerc attrbutes, but unlke nterval-scaled attrbutes, ther scales are eponental For eample, the total amount of mcroorgansms that evolve n a tme t s appromately gven by Ae Bt q where A and B are postve constants Approach ' 1. Do log transform f = log f. Then treat f as an nterval-scaled attrbute 15

16 14/05/1 Nomnal unordered categorcal attrbutes Sometme, we need to transform nomnal attrbutes to numerc attrbutes Transform nomnal attrbutes to bnary attrbutes q The number of values of a nomnal attrbute s v q Create v bnary attrbutes to represent the values q If a data nstance for the nomnal attrbute takes a partcular value, the value of ts bnary attrbute s set to 1, otherwse t s set to 0 The resultng bnary attrbutes can be used as numerc attrbutes, wth two values, 0 and 1 Nomnal attrbutes: an eample Nomnal attrbute frut: has three values q Apple, Orange, and Pear We create three bnary attrbutes called, Apple, Orange, and Pear n the new data If a partcular data nstance n the orgnal data has Apple as the value for frut q then n the transformed data, we set the value of the attrbute Apple to 1, and q the values of attrbutes Orange and Pear to 0 16

17 14/05/1 Ordnal ordered categorcal attrbutes Ordnal attrbute: t s lke a nomnal attrbute, but ts values have a numercal orderng E.g., q Age attrbute wth ordered values: Young, MddleAge, and Old q Common approach to standardzaton: treat s as an nterval-scaled attrbute E.g., Young à 0, MddleAge à 1, Old à 17