Clustering Algorithms

Transcription

1 Clustring Algoritms Hirarcical Clustring k -Mans Algoritms CURE Algoritm 1

2 Mtods of Clustring Hirarcical (Agglomrativ): Initially, ac point in clustr by itslf. Rpatdly combin t two narst clustrs into on. Point Assignmnt: Maintain a st of clustrs. Plac points into tir narst clustr. 2

3 Hirarcical Clustring Two important qustions: 1. How do you dtrmin t narnss of clustrs? 2. How do you rprsnt a clustr of mor tan on point? 3

4 Hirarcical Clustring --- (2) Ky problm: as you build clustrs, ow do you rprsnt t location of ac clustr, to tll wic pair of clustrs is closst? Euclidan cas: ac clustr as a cntroid = avrag of its points. Masur intrclustr distancs by distancs of cntroids. 4

5 Exampl (5,3) o (1,2) o x (1.5,1.5) x (4.7,1.3) x (1,1) o (2,1) o (4,1) x (4.5,0.5) o (0,0) o (5,0) 5

6 And in t Non-Euclidan Cas? T only locations w can talk about ar t points tmslvs. I.., tr is no avrag of two points. Approac 1: clustroid = point closst to otr points. Trat clustroid as if it wr cntroid, wn computing intrclustr distancs. 6

7 Closst Point? Possibl manings: 1. Smallst maximum distanc to t otr points. 2. Smallst avrag distanc to otr points. 3. Smallst sum of squars of distancs to otr points. 4. Etc., tc. 7

8 Exampl clustroid clustroid intrclustr distanc 8

9 Otr Approacs to Dfining Narnss of Clustrs Approac 2: intrclustr distanc = minimum of t distancs btwn any two points, on from ac clustr. Approac 3: Pick a notion of cosion of clustrs,.g., maximum distanc from t clustroid. Mrg clustrs wos union is most cosiv. 9

10 Rturn to Euclidan Cas Approacs 2 and 3 ar also usd somtims in Euclidan clustring. Many otr approacs as wll, for bot Euclidan and non. 10

11 k Mans Algoritm(s) Assums Euclidan spac. Start by picking k, t numbr of clustrs. Initializ clustrs by picking on point pr clustr. For instanc, pick on point at random, tn k -1 otr points, ac as far away as possibl from t prvious points. 11

12 Populating Clustrs 1. For ac point, plac it in t clustr wos currnt cntroid it is narst. 2. Aftr all points ar assignd, fix t cntroids of t k clustrs. 3. Optional: rassign all points to tir closst cntroid. Somtims movs points btwn clustrs. 12

13 Exampl Rassignd points 6 x x Clustrs aftr first round 13

14 Gtting k Rigt Try diffrnt k, looking at t cang in t avrag distanc to cntroid, as k incrass. Avrag falls rapidly until rigt k, tn cangs littl. Avrag distanc to cntroid k Bst valu of k 14

15 Exampl Too fw; many long distancs to cntroid. x x x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x x x x x x 15

16 Exampl Just rigt; distancs ratr sort. x x x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x x x x x x 16

17 Exampl Too many; littl improvmnt in avrag x x distanc. x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x x x x x x 17

18 BFR Algoritm BFR (Bradly-Fayyad-Rina) is a variant of k -mans dsignd to andl vry larg (disk-rsidnt) data sts. It assums tat clustrs ar normally distributd around a cntroid in a Euclidan spac. Standard dviations in diffrnt dimnsions may vary. 18

19 BFR --- (2) Points ar rad on main-mmory-full at a tim. Most points from prvious mmory loads ar summarizd by simpl statistics. To bgin, from t initial load w slct t initial k cntroids by som snsibl approac. 19

20 Initialization: k -Mans Possibilitis includ: 1. Tak a small random sampl and clustr optimally. 2. Tak a sampl; pick a random point, and tn k 1 mor points, ac as far from t prviously slctd points as possibl. 20

21 Tr Classs of Points 1. T discard st : points clos noug to a cntroid to b rprsntd statistically. 2. T comprssion st : groups of points tat ar clos togtr but not clos to any cntroid. Ty ar rprsntd statistically, but not assignd to a clustr. 3. T rtaind st : isolatd points. 21

22 Rprsnting Sts of Points For ac clustr, t discard st is rprsntd by: 1. T numbr of points, N. 2. T vctor SUM, wos i t componnt is t sum of t coordinats of t points in t i t dimnsion. 3. T vctor SUMSQ: i t componnt = sum of squars of coordinats in i t dimnsion. 22

23 Commnts 2d + 1 valus rprsnt any numbr of points. d = numbr of dimnsions. Avrags in ac dimnsion (cntroid coordinats) can b calculatd asily as SUM i /N. SUM i = i t componnt of SUM. 23

24 Commnts --- (2) Varianc of a clustr s discard st in dimnsion i can b computd by: (SUMSQ i /N ) (SUM i /N ) 2 And t standard dviation is t squar root of tat. T sam statistics can rprsnt any comprssion st. 24

25 Galaxis Pictur Points in t RS Comprssd sts. Tir points ar in t CS. A clustr. Its points ar in t DS. T cntroid 25

26 Procssing a Mmory-Load of Points 1. Find tos points tat ar sufficintly clos to a clustr cntroid; add tos points to tat clustr and t DS. 2. Us any main-mmory clustring algoritm to clustr t rmaining points and t old RS. Clustrs go to t CS; outlying points to t RS. 26

27 Procssing --- (2) 3. Adjust statistics of t clustrs to account for t nw points. 4. Considr mrging comprssd sts in t CS. 5. If tis is t last round, mrg all comprssd sts in t CS and all RS points into tir narst clustr. 27

28 A Fw Dtails... How do w dcid if a point is clos noug to a clustr tat w will add t point to tat clustr? How do w dcid wtr two comprssd sts dsrv to b combind into on? 28

29 How Clos is Clos Enoug? W nd a way to dcid wtr to put a nw point into a clustr. BFR suggst two ways: 1. T Maalanobis distanc is lss tan a trsold. 2. Low likliood of t currntly narst cntroid canging. 29

30 Maalanobis Distanc Normalizd Euclidan distanc. For point (x 1,,x k ) and cntroid (c 1,,c k ): 1. Normaliz in ac dimnsion: y i = (x i -c i )/σ i 2. Tak sum of t squars of t y i s. 3. Tak t squar root. 30

31 Maalanobis Distanc --- (2) If clustrs ar normally distributd in d dimnsions, tn aftr transformation, on standard dviation = d. I.., 70% of t points of t clustr will av a Maalanobis distanc < d. Accpt a point for a clustr if its M.D. is < som trsold,.g. 4 standard dviations. 31

32 Pictur: Equal M.D. Rgions 2σ σ 32

33 Sould Two CS Subclustrs B Combind? Comput t varianc of t combind subclustr. N, SUM, and SUMSQ allow us to mak tat calculation. Combin if t varianc is blow som trsold. 33

34 T CURE Algoritm Problm wit BFR/k -mans: Assums clustrs ar normally distributd in ac dimnsion. And axs ar fixd --- llipss at an angl ar not OK. CURE: Assums a Euclidan distanc. Allows clustrs to assum any sap. 34

35 Exampl: Stanford Faculty Salaris salary ag 35

36 Starting CURE 1. Pick a random sampl of points tat fit in main mmory. 2. Clustr ts points irarcically --- group narst points/clustrs. 3. For ac clustr, pick a sampl of points, as disprsd as possibl. 4. From t sampl, pick rprsntativs by moving tm (say) 20% toward t cntroid of t clustr. 36

37 Exampl: Initial Clustrs salary ag 37

38 Exampl: Pick Disprsd Points salary Pick (say) 4 rmot points for ac clustr. ag 38

39 Exampl: Pick Disprsd Points salary Mov points (say) 20% toward t cntroid. ag 39

40 Finising CURE Now, visit ac point p in t data st. Plac it in t closst clustr. Normal dfinition of closst : tat clustr wit t closst (to p ) among all t sampl points of all t clustrs. 40