CS246: Mining Massive Datasets Jure Leskovec, Stanford University

Transcription

1 CS246: Mining Massiv Datasts Jur Lskovc, Stanford Univrsity ttp://cs246.stanford.du

2 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 2 Hig dim. data Grap data Infinit data Macin larning Apps Locality snsitiv asing PagRank, SimRank Filtring data strams SVM Rcommn dr systms Clustring Community Dtction Wb advrtising Dcision Trs Association Ruls Dimnsional ity rduction Spam Dtction Quris on strams Prcptron, knn Duplicat documnt dtction

3 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 3 Givn a st of points, wit a notion of distanc btwn points, group t points into som numbr of clustrs, so tat Mmbrs of a clustr ar clos/similar to ac otr Mmbrs of diffrnt clustrs ar dissimilar Usually: Points ar in a ig-dimnsional spac Similarity is dfind using a distanc masur Euclidan, Cosin, Jaccard, dit distanc,

4 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 4 Outlir Clustr

5 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 5

6 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 6 Clustring in two dimnsions looks asy Clustring small amounts of data looks asy And in most cass, looks ar not dciving Many applications involv not 2, but 10 or 10,000 dimnsions Hig-dimnsional spacs look diffrnt: Almost all pairs of points ar at about t sam distanc --> T Curs of Dimnsionality

7 A catalog of 2 billion sky objcts rprsnts objcts by tir radiation in 7 dimnsions (frquncy bands) Problm: Clustr into similar objcts,.g., galais, narby stars, quasars, tc. Sloan Digital Sky Survy 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 7

8 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 8 Intuitivly: Music divids into catgoris, and customrs prfr a fw catgoris But wat ar catgoris rally? Rprsnt a CD by a st of customrs wo bougt it Similar CDs av similar sts of customrs, and vic-vrsa

9 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 9 Spac of all CDs: Tink of a spac wit on dim. for ac customr Valus in a dimnsion may b 0 or 1 only A CD is a point in tis spac ( 1, 2,, k ), wr i = 1 iff t i t customr bougt t CD For Amazon, t dimnsion is tns of millions Task: Find clustrs of similar CDs

10 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 10 Finding topics: Rprsnt a documnt by a vctor ( 1, 2,, k ), wr i = 1 iff t i t word (in som ordr) appars in t documnt It actually dosn t mattr if k is infinit; i.., w don t limit t st of words Documnts wit similar sts of words may b about t sam topic

11 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 11 As wit CDs w av a coic wn w tink of documnts as sts of words or singls: Sts as vctors: Masur similarity by t cosin distanc Sts as sts: Masur similarity by t Jaccard distanc Sts as points: Masur similarity by Euclidan distanc

12 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 12 Hirarcical: Agglomrativ (bottom up): Initially, ac point is a clustr Rpatdly combin t two narst clustrs into on Divisiv (top down): Start wit on clustr and rcursivly split it Point assignmnt: Maintain a st of clustrs Points blong to narst clustr

13 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 13 Ky opration: Rpatdly combin two narst clustrs Tr important qustions: 1) How do you rprsnt a clustr of mor tan on point? 2) How do you dtrmin t narnss of clustrs? 3) Wn to stop combining clustrs?

14 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 14 Point assignmnt good wn clustrs ar nic, conv saps: Hirarcical can win wn saps ar wird: Not bot clustrs av ssntially t sam cntroid. Asid: if you ralizd you ad concntric clustrs, you could map points basd on distanc from cntr, and turn t problm into a simpl, on-dimnsional cas.

15 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 15 Ky opration: Rpatdly combin two narst clustrs (1) How to rprsnt a clustr of many points? Ky problm: As you mrg clustrs, ow do you rprsnt t location of ac clustr, to tll wic pair of clustrs is closst? Euclidan cas: ac clustr as a cntroid = avrag of its (data)points (2) How to dtrmin narnss of clustrs? Masur clustr distancs by distancs of cntroids

16 (5,3) o (1,2) o (1.5,1.5) (4.7,1.3) (1,1) o (2,1) o (4,1) (4.5,0.5) o (0,0) o (5,0) Data: o data point cntroid 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts Dndrogram 16

17 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 17 Wat about t Non-Euclidan cas? T only locations w can talk about ar t points tmslvs i.., tr is no avrag of two points Approac 1: (1.1) How to rprsnt a clustr of many points? clustroid = (data)point closst to otr points (1.2) How do you dtrmin t narnss of clustrs? Trat clustroid as if it wr cntroid, wn computing intr-clustr distancs

18 (1.1) How to rprsnt a clustr of many points? clustroid = point closst to otr points Possibl manings of closst : Datapoint Smallst maimum distanc to otr points Smallst avrag distanc to otr points Smallst sum of squars of distancs to otr points For distanc mtric d clustroid c of clustr C is: X Clustr on 3 datapoints Cntroid Clustroid min 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 18 c å ÎC d(, c) Cntroid is t avg. of all (data)points in t clustr. Tis mans cntroid is an artificial point. Clustroid is an isting (data)point tat is closst to all otr points in t clustr. 2

19 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 19 (1.2) How do you dtrmin t narnss of clustrs? Trat clustroid as if it wr cntroid, wn computing intrclustr distancs. Approac 2: No cntroid, just dfin distanc Intrclustr distanc = minimum of t distancs btwn any two points, on from ac clustr

20 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 20 Approac 3: Pick a notion of cosion of clustrs Mrg clustrs wos union is most cosiv Approac 3.1: Us t diamtr of t mrgd clustr = maimum distanc btwn points in t clustr Approac 3.2: Us t avrag distanc btwn points in t clustr Approac 3.3: Us a dnsity-basd approac Tak t diamtr or avg. distanc,.g., and divid by t numbr of points in t clustr

21 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 21 It rally dpnds on t sap of clustrs. Wic you may not know in advanc. Eampl: w ll compar two approacs: 1. Mrg clustrs wit smallst distanc btwn cntroids (or clustroids for non-euclidan) 2. Mrg clustrs wit t smallst distanc btwn two points, on from ac clustr

22 Cntroid-basd mrging works wll. But mrgr basd on closst mmbrs migt accidntally mrg incorrctly. B A C A and B av closr cntroids tan A and C, but closst points ar from A and C. 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 22

23 Linking basd on closst mmbrs works wll But Cntroid-basd linking migt caus rrors 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 23

24

25 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 25 Assums Euclidan spac/distanc Start by picking k, t numbr of clustrs Initializ clustrs by picking on point pr clustr Eampl: Pick on point at random, tn k-1 otr points, ac as far away as possibl from t prvious points OK, as long as tr ar no outlirs (points tat ar far from any rasonabl clustr)

26 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 26 Basic ida: Pick a small sampl of points, clustr tm by any algoritm, and us t cntroids as a sd In k-mans++, sampl siz = k tims a factor tat is logaritmic in t total numbr of points How to pick sampl points: Visit points in random ordr, but t probability of adding a point p to t sampl is proportional to D(p) 2. D(p) = distanc btwn p and t narst pickd point.

27 k-mans++, lik otr sd mtods, is squntial You nd to updat D(p) for ac unpickd p du to nw point Paralll approac: Comput nods can ac andl a small st of points Eac picks a fw nw sampl points using sam D(p). Rally important and common trick: Don t updat aftr vry slction; ratr mak many slctions at on round Suboptimal picks don t rally mattr 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 27

28 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 28 1) For ac point, plac it in t clustr wos currnt cntroid it is narst 2) Aftr all points ar assignd, updat t locations of cntroids of t k clustrs 3) Rassign all points to tir closst cntroid Somtims movs points btwn clustrs Rpat 2 and 3 until convrgnc Convrgnc: Points don t mov btwn clustrs and cntroids stabiliz

29 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 29 data point cntroid Clustrs aftr round 1

30 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 30 data point cntroid Clustrs aftr round 2

31 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 31 data point cntroid Clustrs at t nd

32 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 32 How to slct k? Try diffrnt k, looking at t cang in t avrag distanc to cntroid as k incrass Avrag falls rapidly until rigt k, tn cangs littl Avrag distanc to cntroid k Bst valu of k

33 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 33 Too fw; many long distancs to cntroid

34 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 34 Just rigt; distancs ratr sort

35 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 35 Too many; littl improvmnt in avrag distanc

36 Etnsion of k-mans to larg data

37 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 37 BFR [Bradly-Fayyad-Rina] is a variant of k-mans dsignd to andl vry larg (disk-rsidnt) data sts Assums tat clustrs ar normally distributd around a cntroid in a Euclidan spac Standard dviations in diffrnt dimnsions may vary Clustrs ar ais-alignd llipss Goal is to find clustr cntroids; point assignmnt can b don in a scond pass troug t data.

38 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 38 Efficint way to summariz clustrs: Want mmory rquird O(clustrs) and not O(data) IDEA: Ratr tan kping points BFR kps summary statistics of groups of points 3 sts: Clustr summaris, Outlirs, Points to b clustrd Ovrviw of t algoritm: 1. Initializ K clustrs/cntroids 2. Load in a bag points from disk 3. Assign nw points to on of t K original clustrs, if ty ar witin som distanc trsold of t clustr 4. Clustr t rmaining points, and crat nw clustrs 5. Try to mrg nw clustrs from stp 4 wit any of t isting clustrs 6. Rpat stps 2-5 until all points ar amind

39 Points ar rad from disk on main-mmoryfull at a tim Most points from prvious mmory loads ar summarizd by simpl statistics Stp 1) From t initial load w slct t initial k cntroids by som snsibl approac: Tak k random points Tak a small random sampl and clustr optimally Tak a sampl; pick a random point, and tn k 1 mor points, ac as far from t prviously slctd points as possibl 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 39

40 3 sts of points wic w kp track of: Discard st (DS): Points clos noug to a cntroid to b summarizd Comprssion st (CS): Groups of points tat ar clos togtr but not clos to any isting cntroid Ts points ar summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points waiting to b assignd to a comprssion st 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 40

41 Points in t RS Comprssd sts. Tir points ar in t CS. A clustr. Its points ar in t DS. T cntroid Discard st (DS): Clos noug to a cntroid to b summarizd Comprssion st (CS): Summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 41

42 For ac clustr, t discard st (DS) is summarizd by: T numbr of points, N T vctor SUM, wos i t componnt is t sum of t coordinats of t points in t i t dimnsion T vctor SUMSQ: i t componnt = sum of squars of coordinats in i t dimnsion A clustr. All its points ar in t DS. T cntroid 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 42

43 2d + 1 valus rprsnt any siz clustr d = numbr of dimnsions Avrag in ac dimnsion (t cntroid) can b calculatd as SUM i / N SUM i = i t componnt of SUM Varianc of a clustr s discard st in dimnsion i is: (SUMSQ i / N) (SUM i / N) 2 And standard dviation is t squar root of tat Nt stp: Actual clustring Not: Dropping t ais-alignd clustrs assumption would rquir storing full covarianc matri to summariz t clustr. So, instad of SUMSQ bing a d-dim vctor, it would b a d d matri, wic is too big! 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 43

44 Stps 2-5) Procssing Mmory-Load of points: Stp 3) Find tos points tat ar sufficintly clos to a clustr cntroid and add tos points to tat clustr and t DS Ts points ar so clos to t cntroid tat ty can b summarizd and tn discardd Stp 4) Us any in-mmory clustring algoritm to clustr t rmaining points and t old RS Clustrs go to t CS; outlying points to t RS Discard st (DS): Clos noug to a cntroid to b summarizd. Comprssion st (CS): Summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 44

45 Stps 2-5) Procssing Mmory-Load of points: Stp 5) DS st: Adjust statistics of t clustrs to account for t nw points Add Ns, SUMs, SUMSQs Considr mrging comprssd sts in t CS If tis is t last round, mrg all comprssd sts in t CS and all RS points into tir narst clustr Discard st (DS): Clos noug to a cntroid to b summarizd. Comprssion st (CS): Summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 45

46 Points in t RS Comprssd sts. Tir points ar in t CS. A clustr. Its points ar in t DS. T cntroid Discard st (DS): Clos noug to a cntroid to b summarizd Comprssion st (CS): Summarizd, but not assignd to a clustr Rtaind st (RS): Isolatd points 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 46

47 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 47 Q1) How do w dcid if a point is clos noug to a clustr tat w will add t point to tat clustr? Q2) How do w dcid wtr two comprssd sts (CS) dsrv to b combind into on?

48 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 48 Q1) W nd a way to dcid wtr to put a nw point into a clustr (and discard) BFR suggsts two ways: T Maalanobis distanc is lss tan a trsold Hig likliood of t point blonging to currntly narst cntroid

49 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 49 Normalizd Euclidan distanc from cntroid For point ( 1,, d ) and cntroid (c 1,, c d ) 1. Normaliz in ac dimnsion: y i = ( i - c i ) / s i 2. Tak sum of t squars of t y i 3. Tak t squar root! ", $ = & " ' $ ' * '+, ) ' - σ i standard dviation of points in t clustr in t i t dimnsion

50 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 50 If clustrs ar normally distributd in d dimnsions, tn aftr transformation, on standard dviation =! i.., 68% of t points of t clustr will av a Maalanobis distanc <! Accpt a point for a clustr if its M.D. is < som trsold,.g. 2 standard dviations

51 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 51 Euclidan vs. Maalanobis distanc Contours of quidistant points from t origin Uniformly distributd points, Euclidan distanc Normally distributd points, Euclidan distanc Normally distributd points, Maalanobis distanc

52 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 52 Q2) Sould 2 CS subclustrs b combind? Comput t varianc of t combind subclustr N, SUM, and SUMSQ allow us to mak tat calculation quickly Combin if t combind varianc is blow som trsold Many altrnativs: Trat dimnsions diffrntly, considr dnsity

53 Etnsion of k-mans to clustrs of arbitrary saps

54 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 54 Problm wit BFR/k-mans: Assums clustrs ar normally distributd in ac dimnsion And as ar fid llipss at an angl ar not OK Vs. CURE (Clustring Using REprsntativs): Assums a Euclidan distanc Allows clustrs to assum any sap Uss a collction of rprsntativ points to rprsnt clustrs

55 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 55 salary ag

56 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 56 2 Pass algoritm. Pass 1: 0) Pick a random sampl of points tat fit in main mmory 1) Initial clustrs: Clustr ts points irarcically group narst points/clustrs 2) Pick rprsntativ points: For ac clustr, pick a sampl of points, as disprsd as possibl From t sampl, pick rprsntativs by moving tm (say) 20% toward t cntroid of t clustr

57 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 57 salary ag

58 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 58 salary Pick (say) 4 rmot points for ac clustr. ag

59 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 59 salary Mov points (say) 20% toward t cntroid. ag

60 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 60 Pass 2: Now, rscan t wol datast and visit ac point p in t data st Plac it in t closst clustr Normal dfinition of closst : Find t closst rprsntativ to p and assign it to rprsntativ s clustr p

61 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 61 Intuition: A larg, disprsd clustr will av larg movs from its boundary A small, dns clustr will av littl mov. Favors a small, dns clustr tat is nar a largr disprsd clustr

62 1/22/18 Jur Lskovc, Stanford CS246: Mining Massiv Datasts 62 Clustring: Givn a st of points, wit a notion of distanc btwn points, group t points into som numbr of clustrs Algoritms: Agglomrativ irarcical clustring: Cntroid and clustroid k-mans: Initialization, picking k BFR CURE