Administrative UNSUPERVISED LEARNING. Unsupervised learning. Supervised learning 11/25/13. Final project. No office hours today

Size: px

Start display at page:

Download "Administrative UNSUPERVISED LEARNING. Unsupervised learning. Supervised learning 11/25/13. Final project. No office hours today"

Toby Mills
5 years ago
Views:

Admiistrative Fial project No office hours today UNSUPERVISED LEARNING David Kauchak CS 451 Fall 2013 Supervised learig Usupervised learig label

1 Admiistrative Fial project No office hours today UNSUPERVISED LEARNING David Kauchak CS 451 Fall 2013 Supervised learig Usupervised learig label label 1 label 3 model/ predictor label 4 label 5 Supervised learig: give labeled examples Uupervised learig: give data, i.e. examples, but o labels 1

Usupervised learig Usupervised learig applicatios lear clusters/groups without ay label customer segmetatio (i.e. groupig) image compressio bioiformatics: lear motifs fid importat features Give some example without labels, do somethig!

2 Usupervised learig Usupervised learig applicatios lear clusters/groups without ay label customer segmetatio (i.e. groupig) image compressio bioiformatics: lear motifs fid importat features Give some example without labels, do somethig! Usupervised learig: clusterig Usupervised learig: modelig Raw data features f 1, f 2, f 3,, f Most frequetly, whe people thik of usupervised learig they thik clusterig extract features f 1, f 2, f 3,, f f 1, f 2, f 3,, f f 1, f 2, f 3,, f f 1, f 2, f 3,, f group ito classes/ clusters Clusters Aother category: learig probabilities/parameters for models without supervisio Lear a traslatio dictioary Lear a grammar for a laguage Lear the social graph No supervisio, we re oly give data ad wat to fid atural groupigs 2

3 11/25/13 Gee expressio data Clusterig Data from Garber et al. PNAS (98), Clusterig: the process of groupig a set of objects ito classes of similar objects Applicatios? Face Clusterig Face clusterig 3

users Targeted advertisig Exploratory aalysis Advertiser ~10M odes Bidded Keyword

4 Search result clusterig Google News Clusterig i search advertisig Clusterig applicatios bids Fid clusters of advertisers ad keywords Keyword suggestio Performace estimatio Fid clusters of users Targeted advertisig Exploratory aalysis Advertiser ~10M odes Bidded Keyword Who-messages-who IM/text/twitter graph ~100M odes Clusters of the Web Graph Distributed pagerak computatio 4

Data visualizatio A data set with clear cluster structure Wise et al, Visualizig the o-visual PNNL ThemeScapes, Cartia [Moutai height = cluster size] What are some of

Similarity/distace betwee examples Flat clusterig or hierarchical Number of clusters Fixed a priori Data drive?

5 Data visualizatio A data set with clear cluster structure Wise et al, Visualizig the o-visual PNNL ThemeScapes, Cartia [Moutai height = cluster size] What are some of the issues for clusterig? What clusterig algorithms have you see/used? Issues for clusterig Represetatio for clusterig How do we represet a example features, etc. Similarity/distace betwee examples Flat clusterig or hierarchical Number of clusters Fixed a priori Data drive? Clusterig Algorithms Flat algorithms Usually start with a radom (partial) partitioig Refie it iteratively K meas clusterig Model based clusterig Spectral clusterig Hierarchical algorithms Bottom-up, agglomerative Top-dow, divisive 5

6 Hard vs. soft clusterig K-meas Hard clusterig: Each example belogs to exactly oe cluster Most well-kow ad popular clusterig algorithm: Soft clusterig: A example ca belog to more tha oe cluster (probabilistic) Makes more sese for applicatios like creatig browsable hierarchies You may wat to put a pair of seakers i two clusters: (i) sports apparel ad (ii) shoes Start with some iitial cluster ceters Assig/cluster each example to closest ceter Recalculate ceters as the mea of the poits i a cluster K-meas: a example K-meas: Iitialize ceters radomly 6

7 K-meas: assig poits to earest ceter K-meas: readjust ceters K-meas: assig poits to earest ceter K-meas: readjust ceters 7

8 K-meas: assig poits to earest ceter K-meas: readjust ceters K-meas: assig poits to earest ceter K-meas Assig/cluster each example to closest ceter Recalculate ceters as the mea of the poits i a cluster No chages: Doe How do we do this? 8

9 K-meas K-meas Assig/cluster each example to closest ceter iterate over each poit: - get distace to each cluster ceter - assig to closest ceter (hard cluster) Recalculate ceters as the mea of the poits i a cluster Assig/cluster each example to closest ceter iterate over each poit: - get distace to each cluster ceter - assig to closest ceter (hard cluster) Recalculate ceters as the mea of the poits i a cluster What distace measure should we use? Distace measures Clusterig documets (e.g. wie data) Euclidea: d(x, y) = (x i y i ) 2 Oe feature for each word. The value is the umber of times that word occurs. Documets are poits or vectors i this space good for spatial data 9

10 Whe Euclidea distace does t work Issues with Euclidia distace the Euclidea distace betwee q ad d 2 is large Which documet is closest to q usig Euclidia distace? Which do you thik should be closer? but, the distributio of terms i the query q ad the distributio of terms i the documet d 2 are very similar This is ot what we wat! cosie similarity cosie distace sim(x, y) = x y x y = x x y y = 2 x i x i y i 2 y i correlated with the agle betwee two vectors cosie similarity is a similarity betwee 0 ad 1, with thigs that are similar 1 ad ot 0 We wat a distace measure, cosie distace: d(x, y) =1 sim(x, y) - good for text data ad may other real world data sets - is computatioally friedly sice we oly eed to cosider features that have o-zero values both examples 10

11 K-meas Assig/cluster each example to closest ceter Recalculate ceters as the mea of the poits i a cluster K-meas Assig/cluster each example to closest ceter Recalculate ceters as the mea of the poits i a cluster Where are the cluster ceters? How do we calculate these? K-meas Assig/cluster each example to closest ceter Recalculate ceters as the mea of the poits i a cluster Mea of the poits i the cluster: where: µ(c) = 1 C x + y = x C x x i + y i x C = x i C K-meas loss fuctio K-meas tries to miimize what is called the k-meas loss fuctio: loss = d(x i,µ k ) 2 where µ k is cluster ceter for x i that is, the sum of the squared distaces from each poit to the associated cluster ceter 11

12 Miimizig k-meas loss Miimizig k-meas loss 1. Assig/cluster each example to closest ceter 2. Recalculate ceters as the mea of the poits i a cluster 1. Assig/cluster each example to closest ceter 2. Recalculate ceters as the mea of the poits i a cluster loss = d(x i,µ k ) 2 where µ k is cluster ceter for x i Does each step of k-meas move towards reducig this loss fuctio (or at least ot icreasig)? loss = d(x i,µ k ) 2 where µ k is cluster ceter for x i This is t quite a complete proof/argumet, but: 1. Ay other assigmet would ed up i a larger loss 2. The mea of a set of values miimizes the squared error Miimizig k-meas loss Miimizig k-meas loss 1. Assig/cluster each example to closest ceter 2. Recalculate ceters as the mea of the poits i a cluster 1. Assig/cluster each example to closest ceter 2. Recalculate ceters as the mea of the poits i a cluster loss = d(x i,µ k ) 2 where µ k is cluster ceter for x i Does this mea that k-meas will always fid the miimum loss/clusterig? loss = d(x i,µ k ) 2 where µ k is cluster ceter for x i NO! It will fid a miimum. Ufortuately, the k-meas loss fuctio is geerally ot covex ad for most problems has may, may miima We re oly guarateed to fid oe of them 12

13 K-meas variatios/parameters Start with some iitial cluster ceters Assig/cluster each example to closest ceter Recalculate ceters as the mea of the poits i a cluster K-meas variatios/parameters Iitial (seed) cluster ceters Covergece A fixed umber of iteratios partitios uchaged Cluster ceters do t chage What are some other variatios/ parameters we have t specified? K! K-meas: Iitialize ceters radomly Seed choice Results ca vary drastically based o radom seed selectio Some seeds ca result i poor covergece rate, or covergece to sub-optimal clusterigs What would happe here? Commo heuristics Radom ceters i the space Radomly pick examples Poits least similar to ay existig ceter (furthest ceters heuristic) Try out multiple startig poits Iitialize with the results of aother clusterig method Seed selectio ideas? 13

14 Furthest ceters heuristic K-meas: Iitialize furthest from ceters μ 1 = pick radom poit for i = 2 to K: μ i = poit that is furthest from ay previous ceters µ i = argmax x mi µ j :1< j < i d(x,µ j ) poit with the largest distace to ay previous ceter smallest distace from x to ay previous ceter Pick a radom poit for the first ceter K-meas: Iitialize furthest from ceters K-meas: Iitialize furthest from ceters What poit will be chose ext? Furthest poit from ceter What poit will be chose ext? 14

15 K-meas: Iitialize furthest from ceters K-meas: Iitialize furthest from ceters Furthest poit from ceter What poit will be chose ext? Furthest poit from ceter Ay issues/cocers with this approach? Furthest poits cocers Furthest poits cocers If k = 4, which poits will get chose? If we do a umber of trials, will we get differet ceters? 15

16 Furthest poits cocers K-meas++ μ 1 = pick radom poit for k = 2 to K: for i = 1 to N: s i = mi d(x i, μ 1 k-1 ) // smallest distace to ay ceter μ k = radomly pick poit proportioate to s Does t deal well with outliers How does this help? K-meas++ μ 1 = pick radom poit for k = 2 to K: for i = 1 to N: s i = mi d(x i, μ 1 k-1 ) // smallest distace to ay ceter μ k = radomly pick poit proportioate to s - Makes it possible to select other poits - if #poits >> #outliers, we will pick good poits - Makes it o-determiistic, which will help with radom rus - Nice theoretical guaratees! 16

Administrative. Machine learning code. Supervised learning (e.g. classification) Machine learning: Unsupervised learning" BANANAS APPLES

Administrative. Machine learning code. Supervised learning (e.g. classification) Machine learning: Unsupervised learning BANANAS APPLES Administrative Machine learning: Unsupervised learning" Assignment 5 out soon David Kauchak cs311 Spring 2013 adapted from: http://www.stanford.edu/class/cs276/handouts/lecture17-clustering.ppt Machine