Clustering Algorithms

Transcription

1 Clustring Algoritms Applications Hirarcical Clustring k -Mans Algoritms CURE Algoritm 1

2 T Problm of Clustring Givn a st of points, wit a notion of distanc btwn points, group t points into som numbr of clustrs, so tat mmbrs of a clustr ar in som sns as clos to ac otr as possibl. 2

3 Exampl x x x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x x x x x x 3

4 Problms Wit Clustring Clustring in two dimnsions looks asy. Clustring small amounts of data looks asy. And in most cass, looks ar not dciving. 4

5 T Curs of Dimnsionality Many applications involv not 2, but 10 or 10,000 dimnsions. Hig-dimnsional spacs look diffrnt: almost all pairs of points ar at about t sam distanc. 5

6 Exampl: Curs of Dimnsionality Assum random points witin a bounding box,.g., valus btwn 0 and 1 in ac dimnsion. In 2 dimnsions: a varity of distancs btwn 0 and In 10,000 dimnsions, t diffrnc in any on dimnsion is distributd as a triangl. 6

7 Exampl Continud T law of larg numbrs applis. Actual distanc btwn two random points is t sqrt of t sum of squars of ssntially t sam st of diffrncs. 7

8 Exampl Hig-Dimnsion Application: SkyCat A catalog of 2 billion sky objcts rprsnts objcts by tir radiation in 7 dimnsions (frquncy bands). Problm: clustr into similar objcts,.g., galaxis, narby stars, quasars, tc. Sloan Sky Survy is a nwr, bttr vrsion. 8

9 Exampl: Clustring CD s (Collaborativ Filtring) Intuitivly: music divids into catgoris, and customrs prfr a fw catgoris. But wat ar catgoris rally? Rprsnt a CD by t customrs wo bougt it. Similar CD s av similar sts of customrs, and vic-vrsa. 9

10 T Spac of CD s Tink of a spac wit on dimnsion for ac customr. Valus in a dimnsion may b 0 or 1 only. A CD s point in tis spac is (x 1, x 2,, x k ), wr x i = 1 iff t i t customr bougt t CD. Compar wit boolan matrix: rows = customrs; cols. = CD s. 10

11 Spac of CD s (2) For Amazon, t dimnsion count is tns of millions. An altrnativ: us minasing/lsh to gt Jaccard similarity btwn clos CD s. 1 minus Jaccard similarity can srv as a (non-euclidan) distanc. 11

12 Exampl: Clustring Documnts Rprsnt a documnt by a vctor (x 1, x 2,, x k ), wr x i = 1 iff t i t word (in som ordr) appars in t documnt. It actually dosn t mattr if k is infinit; i.., w don t limit t st of words. Documnts wit similar sts of words may b about t sam topic. 12

13 Asid: Cosin, Jaccard, and Euclidan Distancs As wit CD s w av a coic wn w tink of documnts as sts of words or singls: 1. Sts as vctors: masur similarity by t cosin distanc. 2. Sts as sts: masur similarity by t Jaccard distanc. 3. Sts as points: masur similarity by Euclidan distanc. 13

14 Exampl: DNA Squncs Objcts ar squncs of {C,A,T,G}. Distanc btwn squncs is dit distanc, t minimum numbr of insrts and dlts ndd to turn on into t otr. Not tr is a distanc, but no convnint spac in wic points liv. 14

15 Mtods of Clustring Hirarcical (Agglomrativ): Initially, ac point in clustr by itslf. Rpatdly combin t two narst clustrs into on. Point Assignmnt: Maintain a st of clustrs. Plac points into tir narst clustr. 15

16 Hirarcical Clustring Two important qustions: 1. How do you dtrmin t narnss of clustrs? 2. How do you rprsnt a clustr of mor tan on point? 16

17 Hirarcical Clustring (2) Ky problm: as you build clustrs, ow do you rprsnt t location of ac clustr, to tll wic pair of clustrs is closst? Euclidan cas: ac clustr as a cntroid = avrag of its points. Masur intrclustr distancs by distancs of cntroids. 17

18 Exampl (5,3) o (1,2) o x (1.5,1.5) x (4.7,1.3) x (1,1) o (2,1) o (4,1) x (4.5,0.5) o (0,0) o (5,0) 18

19 And in t Non-Euclidan Cas? T only locations w can talk about ar t points tmslvs. I.., tr is no avrag of two points. Approac 1: clustroid = point closst to otr points. Trat clustroid as if it wr cntroid, wn computing intrclustr distancs. 19

20 Closst Point? Possibl manings: 1. Smallst maximum distanc to t otr points. 2. Smallst avrag distanc to otr points. 3. Smallst sum of squars of distancs to otr points. 4. Etc., tc. 20

21 Exampl clustroid clustroid intrclustr distanc 21

22 Otr Approacs to Dfining Narnss of Clustrs Approac 2: intrclustr distanc = minimum of t distancs btwn any two points, on from ac clustr. Approac 3: Pick a notion of cosion of clustrs,.g., maximum distanc from t clustroid. Mrg clustrs wos union is most cosiv. 22

23 Cosion Approac 1: Us t diamtr of t mrgd clustr = maximum distanc btwn points in t clustr. Approac 2: Us t avrag distanc btwn points in t clustr. 23

24 Cosion (2) Approac 3: Us a dnsity-basd approac: tak t diamtr or avrag distanc,.g., and divid by t numbr of points in t clustr. Praps rais t numbr of points to a powr first,.g., squar-root. 24

25 k Mans Algoritm(s) Assums Euclidan spac. Start by picking k, t numbr of clustrs. Initializ clustrs by picking on point pr clustr. Exampl: pick on point at random, tn k -1 otr points, ac as far away as possibl from t prvious points. 25

26 Populating Clustrs 1. For ac point, plac it in t clustr wos currnt cntroid it is narst. 2. Aftr all points ar assignd, fix t cntroids of t k clustrs. 3. Optional: rassign all points to tir closst cntroid. Somtims movs points btwn clustrs. 26

27 Exampl: Assigning Clustrs Rassignd points 6 x x Clustrs aftr first round 27

28 Gtting k Rigt Try diffrnt k, looking at t cang in t avrag distanc to cntroid, as k incrass. Avrag falls rapidly until rigt k, tn cangs littl. Avrag distanc to cntroid k Bst valu of k 28

29 Exampl: Picking k Too fw; many long distancs to cntroid. x x x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x x x x x x 29

30 Exampl: Picking k Just rigt; distancs ratr sort. x x x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x x x x x x 30

31 Exampl: Picking k Too many; littl improvmnt in avrag x x distanc. x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x x x x x x 31

32 BFR Algoritm BFR (Bradly-Fayyad-Rina) is a variant of k -mans dsignd to andl vry larg (disk-rsidnt) data sts. It assums tat clustrs ar normally distributd around a cntroid in a Euclidan spac. Standard dviations in diffrnt dimnsions may vary. 32

33 BFR (2) Points ar rad on main-mmory-full at a tim. Most points from prvious mmory loads ar summarizd by simpl statistics. To bgin, from t initial load w slct t initial k cntroids by som snsibl approac. 33

34 Initialization: k -Mans Possibilitis includ: 1. Tak a small random sampl and clustr optimally. 2. Tak a sampl; pick a random point, and tn k 1 mor points, ac as far from t prviously slctd points as possibl. 34

35 Tr Classs of Points 1. T discard st : points clos noug to a cntroid to b summarizd. 2. T comprssion st : groups of points tat ar clos togtr but not clos to any cntroid. Ty ar summarizd, but not assignd to a clustr. 3. T rtaind st : isolatd points. 35

36 Summarizing Sts of Points For ac clustr, t discard st is summarizd by: 1. T numbr of points, N. 2. T vctor SUM, wos i t componnt is t sum of t coordinats of t points in t i t dimnsion. 3. T vctor SUMSQ: i t componnt = sum of squars of coordinats in i t dimnsion. 36

37 Commnts 2d + 1 valus rprsnt any numbr of points. d = numbr of dimnsions. Avrags in ac dimnsion (cntroid coordinats) can b calculatd asily as SUM i /N. SUM i = i t componnt of SUM. 37

38 Commnts (2) Varianc of a clustr s discard st in dimnsion i can b computd by: (SUMSQ i /N ) (SUM i /N ) 2 And t standard dviation is t squar root of tat. T sam statistics can rprsnt any comprssion st. 38

39 Galaxis Pictur Points in t RS Comprssd sts. Tir points ar in t CS. A clustr. Its points ar in t DS. T cntroid 39

40 Procssing a Mmory-Load of Points 1. Find tos points tat ar sufficintly clos to a clustr cntroid; add tos points to tat clustr and t DS. 2. Us any main-mmory clustring algoritm to clustr t rmaining points and t old RS. Clustrs go to t CS; outlying points to t RS. 40

41 Procssing (2) 3. Adjust statistics of t clustrs to account for t nw points. Add N s, SUM s, SUMSQ s. 4. Considr mrging comprssd sts in t CS. 5. If tis is t last round, mrg all comprssd sts in t CS and all RS points into tir narst clustr. 41

42 A Fw Dtails... How do w dcid if a point is clos noug to a clustr tat w will add t point to tat clustr? How do w dcid wtr two comprssd sts dsrv to b combind into on? 42

43 How Clos is Clos Enoug? W nd a way to dcid wtr to put a nw point into a clustr. BFR suggst two ways: 1. T Maalanobis distanc is lss tan a trsold. 2. Low likliood of t currntly narst cntroid canging. 43

44 Maalanobis Distanc Normalizd Euclidan distanc from cntroid. For point (x 1,,x k ) and cntroid (c 1,,c k ): 1. Normaliz in ac dimnsion: y i = (x i -c i )/σ i 2. Tak sum of t squars of t y i s. 3. Tak t squar root. 44

45 Maalanobis Distanc (2) If clustrs ar normally distributd in d dimnsions, tn aftr transformation, on standard dviation = d. I.., 70% of t points of t clustr will av a Maalanobis distanc < d. Accpt a point for a clustr if its M.D. is < som trsold,.g. 4 standard dviations. 45

46 Pictur: Equal M.D. Rgions 2σ σ 46

47 Sould Two CS Subclustrs B Combind? Comput t varianc of t combind subclustr. N, SUM, and SUMSQ allow us to mak tat calculation quickly. Combin if t varianc is blow som trsold. Many altrnativs: trat dimnsions diffrntly, considr dnsity. 47

48 T CURE Algoritm Problm wit BFR/k -mans: Assums clustrs ar normally distributd in ac dimnsion. And axs ar fixd llipss at an angl ar not OK. CURE: Assums a Euclidan distanc. Allows clustrs to assum any sap. 48

49 Exampl: Stanford Faculty Salaris salary ag 49

50 Starting CURE 1. Pick a random sampl of points tat fit in main mmory. 2. Clustr ts points irarcically group narst points/clustrs. 3. For ac clustr, pick a sampl of points, as disprsd as possibl. 4. From t sampl, pick rprsntativs by moving tm (say) 20% toward t cntroid of t clustr. 50

51 Exampl: Initial Clustrs salary ag 51

52 Exampl: Pick Disprsd Points salary Pick (say) 4 rmot points for ac clustr. ag 52

53 Exampl: Pick Disprsd Points salary Mov points (say) 20% toward t cntroid. ag 53

54 Finising CURE Now, visit ac point p in t data st. Plac it in t closst clustr. Normal dfinition of closst : tat clustr wit t closst (to p ) among all t sampl points of all t clustrs. 54