Introduction to Data Mining

Transcription

1 Introduction to Data Mining Lctur #15: Clustring-2 Soul National Univrsity 1

2 In Tis Lctur Larn t motivation and advantag of BFR, an xtnsion of K-mans to vry larg data Larn t motivation and advantag of CURE, an xtnsion of K-mans to clustrs of arbitrary saps 2

3 Outlin BFR Algoritm CURE Algoritm BFR: Extnsion of k-mans to larg data 3

4 BFR Algoritm BFR [Bradly-Fayyad-Rina] is a variant of k-mans dsignd to andl vry larg (disk-rsidnt) data sts Assums tat clustrs ar normally distributd around a cntroid in a Euclidan spac Standard dviations in diffrnt dimnsions may vary Clustrs ar axis-alignd llipss Efficint way to summariz clustrs (want mmory rquird O(clustrs) and not O(data)) 4

5 BFR Algoritm Points ar rad from disk on main-mmory-full at a tim Most points from prvious mmory loads ar summarizd by simpl statistics To bgin, from t initial load w slct t initial k cntroids by som snsibl approac: Tak k random points Tak a small random sampl and clustr optimally Tak a sampl; pick a random point, and tn k 1 mor points, ac as far from t prviously slctd points as possibl 5

6 Tr Classs of Points 3 sts of points wic w kp track of: Discard st (DS): Points clos noug to a cntroid to b summarizd Comprssion st (CS): Groups of points tat ar clos togtr but not clos to any xisting cntroid Ts points ar summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points waiting to b assignd to a comprssion st 6

7 BFR: Galaxis Pictur Points in t RS Comprssd sts. Tir points ar in t CS. A clustr. Its points ar in t DS. T cntroid Discard st (DS): Clos noug to a cntroid to b summarizd Comprssion st (CS): Summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points 7

8 Summarizing Sts of Points For ac clustr, t discard st (DS) is summarizd by: T numbr of points, N T vctor SUM, wos i t componnt is t sum of t coordinats of t points in t i t dimnsion T vctor SUMSQ: i t componnt = sum of squars of coordinats in i t dimnsion A clustr. All its points ar in t DS. T cntroid 8

9 Summarizing Points: Commnts 2d + 1 valus rprsnt any siz clustr d = numbr of dimnsions Avrag in ac dimnsion (t cntroid) can b calculatd as SUM i / N SUM i = i t componnt of SUM Varianc of a clustr s discard st in dimnsion i is: (SUMSQ i / N) (SUM i / N) 2 And standard dviation is t squar root of tat Nxt stp: Actual clustring Not: Rmoving t axis-alignd clustrs assumption would rquir storing full covarianc matrix to summariz t clustr. So, instad of SUMSQ bing a d-dim vctor, it would b a d x d matrix, wic is too big! 9

10 T Mmory-Load of Points Procssing t Mmory-Load of points (1): 1) Find tos points tat ar sufficintly clos to a clustr cntroid and add tos points to tat clustr and t DS Ts points ar so clos to t cntroid tat ty can b summarizd and tn discardd 2) Us any main-mmory clustring algoritm to clustr t rmaining points and t old RS Clustrs go to t CS; outlying points to t RS Discard st (DS): Clos noug to a cntroid to b summarizd. Comprssion st (CS): Summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points 10

11 T Mmory-Load of Points Procssing t Mmory-Load of points (2): 3) DS st: Adjust statistics of t clustrs to account for t nw points Updat Ns, SUMs, SUMSQs 4) Considr mrging comprssd sts in t CS 5) If tis is t last round, mrg all comprssd sts in t CS and all RS points into tir narst clustr Discard st (DS): Clos noug to a cntroid to b summarizd. Comprssion st (CS): Summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points 11

12 A Fw Dtails Q1) How do w dcid if a point is clos noug to a clustr tat w will add t point to tat clustr? Q2) How do w dcid wtr two comprssd sts (CS) dsrv to b combind into on? 12

13 How Clos is Clos Enoug? Q1) W nd a way to dcid wtr to put a nw point into a clustr (and discard) BFR suggsts two ways: Hig likliood of t point blonging to currntly narst cntroid (and, t point far from all otr cntroids) T Maalanobis distanc is small (< t) 13

14 Maalanobis Distanc Normalizd Euclidan distanc from cntroid For point (x 1,, x d ) and cntroid (c 1,, c d ) 1. Normaliz in ac dimnsion: y i = (x i - c i ) / σ i 2. Tak sum of t squars of t y i 3. Tak t squar root dd xx, cc = dd ii=1 xx ii cc ii σσ ii 2 σ i standard dviation of points in t clustr in t i t dimnsion 14

15 Maalanobis Distanc If clustrs ar normally distributd in d dimnsions, tn aftr transformation, on standard dviation = dd Accpt a point for a clustr if its M.D. is < t (a paramtr) 15

16 Pictur: Equal M.D. Rgions Euclidan vs. Maalanobis distanc Contours of quidistant points from t origin Uniformly distributd points, Euclidan distanc Normally distributd points, Euclidan distanc Normally distributd points, Maalanobis distanc 16

17 Sould 2 CS clustrs b combind? Q2) Sould 2 CS subclustrs b combind? Comput t varianc of t combind subclustr N, SUM, and SUMSQ allow us to mak tat calculation quickly Combin if t combind varianc is small (< s) 17

18 Outlin BFR Algoritm CURE Algoritm CURE: Extnsion of k-mans to clustrs of arbitrary saps 18

19 T CURE Algoritm Problm wit BFR/k-mans: Assums clustrs ar normally distributd in ac dimnsion And axs ar fixd llipss at an angl ar not OK Vs. CURE (Clustring Using REprsntativs): Assums a Euclidan distanc Allows clustrs to assum any sap Uss a collction of rprsntativ points to rprsnt clustrs 19

20 Starting CURE 2 Pass algoritm. Pass 1: 1) Pick a random sampl of points tat fit in main mmory 2) Initial clustrs: Clustr ts points irarcically group narst points/clustrs 3) Pick rprsntativ points: For ac clustr, pick a sampl of points, as disprsd as possibl From t sampl, pick rprsntativs by moving tm (say) 20% toward t cntroid of t clustr 20

21 Starting CURE 2 Pass algoritm. Pass 1: 4) Mrg clustrs Mrg two clustrs tat ar sufficintly clos (<t) Clustr distanc: minimum distanc of rprsntativ points Rpat, until tr ar no mor sufficintly clos clustrs 21

22 Srinking Rprsntativs Wy srink rprsntativs by 20%? Rduc outlirs ability to caus t wrong clustrs to b mrgd C1 cntroid rprsntativs C3 C2 22

23 Exampl: Initial Clustrs salary ag 23

24 Exampl: Pick Disprsd Points salary Pick (say) 4 rmot points for ac clustr. ag 24

25 Exampl: Pick Disprsd Points salary Mov points (say) 20% toward t cntroid. ag 25

26 Finising CURE Pass 2: Now, rscan t wol datast and visit ac point p in t data st Plac it in t closst clustr Normal dfinition of closst : Find t closst rprsntativ to p and assign it to rprsntativ s clustr p 26

27 Wat You Nd to Know Clustring: Givn a st of points, wit a notion of distanc btwn points, group t points into som numbr of clustrs Algoritms: Agglomrativ irarcical clustring: Cntroid and clustroid k-mans: Initialization, picking k BFR CURE 27

28 Qustions? 28