Outline. Type of Machine Learning. Examples of Application. Unsupervised Learning

Transcription

1 Outlne Artfcal Intellgence and ts applcatons Lecture 8 Unsupervsed Learnng Professor Danel Yeung danyeung@eee.org Dr. Patrck Chan patrckchan@eee.org South Chna Unversty of Technology, Chna Introducton Parametrc VS Non-Parametrc Approach Mxture of Denstes Maxmum-Lkelhood Estmates Clusterng Smlarty (Dssmlarty) Measure Clusterng Algorthm Type of Machne Learnng Examples of Applcaton Supervsed Learnng Label s gven Renforcement learnng No Label but havng feedback Unsupervsed Learnng No Labels gven Much harder than supervsed learnng Also named Learnng wthout a teacher You never know the true, correct answer How to evaluate the result? New medcal cases New stars New speces Chess game playng Performance evaluaton (may be satsfactory today, but unsatsfactory by next year s standard) 3

2 Unsupervsed Learnng Why Unsupervsed Learnng? Dfferent from supervsed or renforcement Learnng, no feedback s gven from the envronment n unsupervsed learnng How to evaluate the result? External: Expert comments Expert may be wrong Internal: Objectve functons E.g. Dstance between samples and centers Very ntutve Evaluaton method s subjectve Label s expensve Especally for huge dataset E.g. Medcal applcaton No deaon the number of classes Data Mnng Gan some nsght about data structure before desgnng classfers E.g. Feature selecton 5 6 Types of Unsupervsed Learnng Parametrc Approach Parametrc Approach Assume structure of dstrbuton s known Only need to estmate parametersof the dstrbuton E.g. Maxmum-Lkelhood Estmate Non-Parametrc Approach No assumpton on the dstrbuton Group data nto clusters Samples n the same group share somethng n common E.g. Clusterng Method Mxture of Denstes, whch assumes Samples come from known classes ω,, ω c The pror probablty (P(ω j )) s known The forms of class-condtonal probablty densty ( P(x ω j, θ j ) ) are known, but ther parameters θ j s are unknown Category labels unknown Dfferent denstes for each class ω j wth parameters θ j 7 8

3 Parametrc Approach Maxmum-Lkelhood Estmates A method to fnd the soluton of unknown parameter θ Parametrc Approach Maxmum-Lkelhood Estmates lbe the logarthm of the lkelhood, The maxmum-lkelhood estmate θ s that value of θ that maxmzes p(d θ) Take gradent of lwth respect to θ : Maxmum lkelhood estmate θ must satsfy the condtons p(d θ) called the lkelhood of θ w.r.t. the set of samples x, x,, x n 9 0 Parametrc Approach Maxmum-Lkelhood Estmates Parametrc Approach: Normal Dstrbuton Unknown Mean (Varance known) Assume the componentdensty s multvarate normal Dstrbuton ( x w, θ) ~ N( µ, p ) A few of the dfferent casesthat can arse dependng upon whch parameters are known ( ) and whch ones are unknown (?): Case s the smplest, and wll be consdered n detal The lkelhood functon t p ( x w, µ ) exp( ( / ) ( x µ ) ( x µ ) ) (π) d/ / Log of the lkelhood / d/ t lnp ( x w, µ ) ln(π) ( x µ ) ( x µ ) Case s more realstc, though somewhat more nvolved Case 3represents the problem we face encounterng a completely unknown set of data Its dervatve s µ lnp ( x w, µ ) ( x µ )

4 Parametrc Approach: Normal Dstrbuton Unknown Mean (Varance known) The maxmum-lkelhood estmate µ must satsfy Parametrc Approach: Normal Dstrbuton Unknown Mean (Varance known) Iteratve method s appled After the terms are rearranged: where: µ (0) s set to a startng pont n ntalzaton µ cannot be calculated explctly 3 Parametrc Approach: Normal Dstrbuton Unknown Mean (Varance known) Parametrc Approach: Normal Dstrbuton Unknown Mean (Varance known) Examples: 5 samples µ - and µ Ths nformaton wll not be gven P(ω ) 0.33 and P(ω ) 0.67 (don t know whch sample belongng to whch class, but do know /3 belong to ω and /3 belong to ω ) Varance 9 By usng the teratve scheme (gradent descent) Dfferent startng ponts yeld dfferent solutons Two local mnmum solutons obtaned by usng teratve method µ a µ b : : ^ µ.30 µ. 668 ^ µ µ. ^ ^ µ a and µ b are local optmal estmates for µ - and µ 5 6

5 Parametrc Approach: Normal Dstrbuton Unknown Mean (Varance known) The soluton s not unque due to dfferent pror probabltes (P(ω )s not equal top(ω )) Thus, when the mxture densty s not dentfable, the maxmum-lkelhood soluton s not unque. (note: P(xIµ a ) s better estmate to the source densty than P(xIµ b ) ) Parametrc Approach Drawback Very msleadngf the sample dstrbuton s dfferent from the assumpton E.g. Normal Dstrbuton s assumed These datasets consdered to be the same for parametrc approach snce they have same mean and varance Non-parametrc approach can solve ths problem µ - µ 7 ω ω 7 8 Non-Parametrc Approach Non-parametrc Approach Three Important Factors Objectve: seek the natural clusters n the data Internal External Internal What s a good clusterng result? Internal(wthn a cluster): Dstance should be small External(ntra-cluster): Dstance should be large 9 Smlarty (Dssmlarty) Measure How smlar between two samples? What knd of clusterng result s expected? Clusterng Algorthm E.g. optmze the crteron functon 0

6 Non-parametrc Approach Smlarty Measure No best measurefor all cases Applcaton dependent Examples: Face Recognton, should have rotaton nvarance Should be smlar Non-parametrc Approach Smlarty Measure The scale of features may be very dfferent Dfferent Ranges Weght: , wast wdth: 8 5 Dfferent Unts Km VS mle, cm VS meter Should features be normalzed? f the spread s due to the presence of clusters, normalzaton reduces cluster effect (rght dagram) For character recognton, NO rotaton nvarance Should be dfferent Non-parametrc Approach Smlarty Measure Non-parametrc Approach Smlarty Measure Eucldean Dstance d( x, x) ( x x k) k k Manhattan Dstance d( x Cosne Smlarty 3 s n n, x) x x k k k t, xx x x ( x x ) ( 5) + ( 3 ) [ 3] Other examples: Mahalanobs Dstance d( x, x) n ( x x k) k k σ Chebyshev Dstance k ( x x ) d( x, x) max k k k n

7 Non-parametrc Approach Smplest Clusterng Algorthm Non-parametrc Approach Smplest Clusterng Algorthm A smple clusterng algorthmcan be developed after a smlarty measure s defned For each sample pars, groupthe samples n the same clusterf the dstancebetween them s less than a partcular threshold d 0 Examples: large d 0 One Cluster Medum d 0 Reasonable Result Small d 0 Every sample s a cluster Advantage: Easy to understandand smple to mplement Dsadvantage: Hghly dependent on the threshold(d 0 ) 5 6 Non-parametrc Approach Non-parametrc Approach Sum-of-Squared-Error (SSE) Crteron Mean of samples n cluster D s: m n x D x n s the number of samples n D Example: [5 ] + [5 5] m + [6 ] + [6 5] [ ] + [ ] m + [ ] + [ 3] 5 [3 ] + [5.5 [.8.5].6] SSE crteron s: J e c x D x m c s the number of cluster J e ( 5 5.5) + (.5) + ( 5 5.5) + ( 5.5) ( 6 5.5) + (.5) + ( 6 5.5) + ( 5.5) (.8) + (.6) + (.8) + (.6) (.8) + (.6) + (.8) + ( 3.6) ( ) ( )

8 Non-parametrc Approach Smaller SSE s preferred e e J J Non-parametrc Approach Is SSE a good crteron for all stuatons? No! Approprate when: The clusters form compact groups Equally szed clusters Not Approprate When natural groupngs have very dfferent szes More reasonable result! 9 30 Non-parametrc Approach Non-parametrc Approach For example Result s more reasonable However, t has a larger value of SSEdue to the large cluster Result Result SSE can be rewrtten as: J e c x D x m n x D ' x D x x S s the average squared dstance between ponts n the thcluster andj e can be replaced by other crteron functons c ' ns m n x D x Wthn-cluster scatter matrx Between-cluster scatter matrx Large J e Small J e Total scatter matrx 3 3

9 Non-parametrc Approach Clusterng Algorthm Fnd the optmal clusterng result Exhaustve search s mpossble C n /c! possble parttons Methods: Iteratve Optmzaton Algorthm K-means Herarchcal Clusterng Bottom Up Approach Top Down Approach Non-parametrc Approach Iteratve Optmzaton Algorthm Algorthm (smlar to gradent descent):. Fnd a reasonable ntal partton. Move sample(s) from one cluster to another such that the objectve functon s mproved 3. Repeat step untl stable Move [ 3] from cluster x to cluster o J e Non-parametrc Approach: Iteratve Optmzaton Algorthm K-means Non-parametrc Approach: Iteratve Optmzaton Algorthm K-means K-means s a popular technque : J e c x D x m Assume there are c (k) classes We use k3 n the followng example. Intalzaton Randomly assgn the center of each cluster. Assgn Samples Assgn samples to closest center 3. Re-calculate the mean Compute the new means usng new samples Repeat untl stable (no sample moves agan) 35 36

10 st step Non-parametrc Approach: Iteratve Optmzaton Algorthm K-means nd step Centers are changed It s a smart waywhch decreases the objectve functon Fast, effcent, popular Algorthm convergesafter a fnte number of teratons of steps and 3 However, Smlar to gradent descent, K-means method may be trapped at local mnmum 3rd step Centers are changed Centers are not changed K-means Stops Trapped at Local Mnmum Global Mnmum Non-parametrc Approach: Herarchcal Clusterng Non-parametrc Approach: Herarchcal Clusterng Dendogram So far only dsjont clusters have been dscussed. In some stuatons, clusters may have subclustersand so on. Herarchcal cluster (taxonomy) s an example. Sutable way to represent a herarchcal clusterng s a Dendogram Example: Use Bnary Tree wth bottom-up parwse dstance smlarty measure Mean of x 3 & x Sample ponts Dendogram 39 0

11 Non-parametrc Approach: Herarchcal Clusterng Venn Dagram Venn dagram can also show herarchcal clusterng (refer to last slde on Dendogram) However, no quanttatve nformaton (parwse dstance) can be conveyed n a Venn Dagram Non-parametrc Approach: Herarchcal Clusterng Two types: Dvsve (top down) Approach Start wth cluster One cluster contans all samples Form herarchy by splttng the most dssmlar clusters Agglomeratve (bottom up) Approach Start wth n clusters Each cluster contans one sample Form herarchy by mergng the most smlar clusters Not effcent f a large number of samples but a number of clusters s needed Non-parametrc Approach: Herarchcal Clusterng Top Down Approach Non-parametrc Approach: Herarchcal Clusterng Bottom Up Approach Any Iteratve Optmzaton Algorthm can be appled by settng c Algorthm Do Whle more than one cluster Calculate dstance between two clusters for all cluster pars Merge the nearest two clusters Four common dstance measures Mnmum Dstance: Maxmum Dstance: Average Dstance: Mean Dstance: 3

12 Non-parametrc Approach: Herarchcal Clusterng Bottom Up Approach Sngle Lnkage(Nearest-Neghbor) Intally every pont forms a cluster Mnmum Dstancebetween clusters s used n mergng two nearest neghbors Encourage growth of elongated clusters Dsadvantage: Senstve to nose cluster6 Cluster 6 cluster3 3 Cluster Mn s shortest Dstance between Two clusters 5 cluster 5 cluster5 Ideal case Nose data Non-parametrc Approach: Herarchcal Clusterng Bottom Up Approach Mnmum and maxmum dstance are extremely senstve to nose sample snce ther measurement nvolves mnma or maxma The result s more robustto outler when the average or the mean dstance are used Non-parametrc Approach: Herarchcal Clusterng Bottom Up Approach Complete Lnkage (Farthest Neghbor) cluster Intally every pont forms a cluster Maxmum Dstances used between clusters n mergng two nearest neghbors Encourages compact clusters Does not work well f elongated clusters present D D cluster6 6 cluster cluster Max s greatest Dstance between Two clusters cluster3 cluster5 Ideally, D and D3 should be merged d Number of Clusters d D3 Snce d < d, D and D wll be merged How to decde the number of clusters? Possble soluton: Try a range of cand see whch one has the lowest crteron value Mean dstances less tme consumedthan Average dstance 7 8