Lecture 36 of 42. Expectation Maximization (EM), Unsupervised Learning and Clustering

Transcription

1 Lecture 36 of 42 Expectaton Maxmzaton (EM), Unsupervsed Learnng and Clusterng Wednesday, 18 Aprl 2007 Wllam H. Hsu, KSU Readngs: Secton 6.12, Mtchell Secton 3.2.4, Shavlk and Detterch (Rumelhart and Zpser) Secton 3.2.5, Shavlk and Detterch (Kohonen) Lecture Outlne Readngs: 6.12, Mtchell; Rumelhart and Zpser Suggested Readng: Kohonen Ths Week s Revew: Paper 9 of 13 Unsupervsed Learnng and Clusterng Defntons and framework Constructve nducton Feature constructon Cluster defnton EM, AutoClass, Prncpal Components Analyss, Self-Organzng Maps Expectaton-Maxmzaton (EM) Algorthm More on EM and Bayesan Learnng EM and unsupervsed learnng Next Lecture: Tme Seres Learnng Intro to tme seres learnng, characterzaton; stochastc processes Read Chapter 16, Russell and Norvg (decsons and utlty) 1

2 Unsupervsed Learnng: Objectves Unsupervsed Learnng Supervsed Unsupervsed x f(x) x Learnng Learnng Gven: data set D Vectors of attrbute values (x 1, x 2,, x n ) No dstncton between nput attrbutes and output attrbutes (class label) Return: (synthetc) descrptor y of each x Clusterng: groupng ponts (x) nto nherent regons of mutual smlarty Vector quantzaton: dscretzng contnuous space wth best labels Dmensonalty reducton: projectng many attrbutes down to a few Feature extracton: constructng (few) new attrbutes from (many) old ones Intutve Idea fˆ ( x ) Want to map ndependent varables (x) to dependent varables (y = f(x)) Don t always know what dependent varables (y) are Need to dscover y based on numercal crteron (e.g., dstance metrc) y Clusterng A Mode of Unsupervsed Learnng Gven: a collecton of data ponts Goal: dscover structure n the data Organze data nto sensble groups (how many here?) Crtera: convenent and vald organzaton of the data NB: not necessarly rules for classfyng future data ponts Cluster analyss: study of algorthms, methods for dscoverng ths structure Representng structure: organzng data nto clusters (cluster formaton) Descrbng structure: cluster boundares, centers (cluster segmentaton) Defnng structure: assgnng meanngful names to clusters (cluster labelng) Cluster: Informal and Formal Defntons Set whose enttes are alke and are dfferent from enttes n other clusters Aggregaton of ponts n the nstance space such that dstance between any two ponts n the cluster s less than the dstance between any pont n the cluster and any pont not n t 2

3 Quck Revew: Bayesan Learnng and EM Problem Defnton Gven: data (n-tuples) wth mssng values, aka partally observable (PO) data Want to fll n? wth expected value Soluton Approaches Expected = dstrbuton over possble values Use best guess Bayesan model (e.g., BBN) to estmate dstrbuton Expectaton-Maxmzaton (EM) algorthm can be used here Intutve Idea Want to fnd h ML n PO case (D unobserved varables observed varables) Estmaton step: calculate E[unobserved varables h], assumng current h Maxmzaton step: update w jk to maxmze E[lg P(D h)], D all varables r ( ) = = = e r r I r r r r # data cases wth n, X j N n,e e j hml arg max r = arg max r h H # data cases wth e h H I r r X j E = e ( ) j Experment EM Algorthm: Example [1] Two cons: P(Head on Con 1) = p, P(Head on Con 2) = q Con Expermenter frst selects a con: P(Con = 1) = α P(Con = 1) = α Chosen con tossed 3 tmes (per expermental run) Observe: D = {(1 H H T), (1 H T T), (2 T H T)} Want to predct: α, p, q Flp 1 Flp 2 Flp 3 How to model the problem? P(Flp = 1 Con = 1) = p P(Flp = 1 Con = 2) = q Smple Bayesan network Now, can fnd most lkely values of parameters α, p, q gven data D Parameter Estmaton Fully observable case: easy to estmate p, q, and α Suppose k heads are observed out of n con flps Maxmum lkelhood estmate v ML for Flp : p = k/n Partally observable case Don t know whch con the expermenter chose Observe: D = {(H H T), (H T T), (T H T)} {(? H H T), (? H T T), (? T H T)} 3

4 EM Algorthm: Example [2] Problem When we knew Con = 1 or Con = 2, there was no problem No known analytcal soluton to the partally observable problem.e., not known how to compute estmates of p, q, and α to get v ML Moreover, not known what the computatonal complexty s Soluton Approach: Iteratve Parameter Estmaton Gven: a guess of P(Con = 1 x), P(Con = 2 x) Generate fctonal data ponts, weghted accordng to ths probablty P(Con = 1 x) = P(x Con = 1) P(Con = 1) / P(x) based on our guess of α, p, q Expectaton step (the E n EM) Now, can fnd most lkely values of parameters α, p, q gven fctonal data Use gradent descent to update our guess of α, p, q Maxmzaton step (the M n EM) Repeat untl termnaton condton met (e.g., stoppng crteron on valdaton set) EM Converges to Local Maxma of the Lkelhood Functon P(D Θ) Expectaton Step EM Algorthm: Example [3] Suppose we observed m actual experments, each n con flps long Each experment corresponds to one choce of con (~α) Let h denote the number of heads n experment x (a sngle data pont) Q: How dd we smulate the fctonal data ponts, E[ log P(x αˆ, p, ˆ qˆ )]? A: By estmatng (for 1 m,.e., the real data ponts) r r P ( ) ( x Con = 1) P( Con = 1) P Con = 1 x = r P x Maxmzaton Step αˆ pˆ Q: What are we updatng? What objectve functon are we maxmzng? m E E E r A: We are updatng αˆ, p, ˆ qˆ to maxmze,, where E = E αˆ pˆ qˆ log P x α ˆ, p, ˆ qˆ = 1 r h ( = ) ( = r h r P Con 1 x P Con 1 x ) [ 1- P( Con = 1 x )] αˆ =, pˆ = n r, qˆ = n m P Con = x P Con = r x h αˆ pˆ ( ) h ( 1- pˆ ) n-h n-h h ( 1- pˆ ) + ( 1-αˆ ) qˆ ( 1- qˆ ) n-h ( ) ( ) [ ( )] 4

5 EM for Unsupervsed Learnng Unsupervsed Learnng Problem Objectve: estmate a probablty dstrbuton wth unobserved varables Use EM to estmate mxture polcy (more on ths later; see 6.12, Mtchell) Pattern Recognton Examples Human-computer ntellgent nteracton (HCII) Detectng facal features n emoton recognton Gesture recognton n vrtual envronments Computatonal medcne [Frey, 1998] Determnng morphology (shapes) of bactera, vruses n mcroscopy Identfyng cell structures (e.g., nucleus) and shapes n mcroscopy Other mage processng Many other examples (audo, speech, sgnal processng; motor control; etc.) Inference Examples Plan recognton: mappng from (observed) actons to agent s (hdden) plans Hdden changes n context: e.g., avaton; computer securty; MUDs Unsupervsed Learnng: AutoClass [1] Bayesan Unsupervsed Learnng Gven: D = {(x 1, x 2,, x n )} (vectors of ndstngushed attrbute values) Return: set of class labels that has maxmum a posteror (MAP) probablty Intutve Idea Bayesan learnng: h = arg max P( h D) = arg max P( D h) P( h) MAP h H MDL/BIC (Occam s Razor): prors P(h) express cost of codng each model h AutoClass Defne mutually exclusve, exhaustve clusters (class labels) y 1, y 2,, y J Suppose: each y j (1 j J) contrbutes to x Suppose also: y j s contrbuton ~ known pdf, e.g., Mxture of Gaussans (MoG) Conjugate prors: prors on y of same form as prors on x When to Use for Clusterng Beleve (or can assume): clusters generated by known pdf Beleve (or can assume): clusters combned usng fnte mxture (later) h H 5

6 Unsupervsed Learnng: AutoClass [2] AutoClass Algorthm [Cheeseman et al, 1988] Based on maxmzng P(x Θ j, y j, J) Θ j : class (cluster) parameters (e.g., mean and varance) y j : synthetc classes (can estmate margnal P(y j ) any tme) Apply Bayes s Theorem, use numercal BOC estmaton technques (cf. Gbbs) Search objectves Fnd best J (deally: ntegrate out Θ j, y j ; really: start wth bg J, decrease) Fnd Θ j, y j : use MAP estmaton, then ntegrate n the neghborhood of y MAP EM: Fnd MAP Estmate for P(x Θ j, y j, J) by Iteratve Refnement Advantages over Symbolc (Non-Numercal) Methods Returns probablty dstrbuton over class membershp More robust than best y j Compare: fuzzy set membershp (smlar but probablstcally motvated) Can deal wth contnuous as well as dscrete data Unsupervsed Learnng: AutoClass [3] AutoClass Resources Begnnng tutoral (AutoClass II): Cheeseman et al, Buchanan and Wlkns Project page: Applcatons Knowledge dscovery n databases (KDD) and data mnng Infrared astronomcal satellte (IRAS): spectral atlas (sky survey) Molecular bology: pre-clusterng DNA acceptor, donor stes (mouse, human) LandSat data from Kansas (30 km 2 regon, 1024 x 1024 pxels, 7 channels) Postve fndngs: see book chapter by Cheeseman and Stutz, onlne Other typcal applcatons: see KD Nuggets ( Implementatons Obtanng source code from project page AutoClass III: Lsp mplementaton [Cheeseman, Stutz, Taylor, 1992] AutoClass C: C mplementaton [Cheeseman, Stutz, Taylor, 1998] These and others at: 6

7 Unsupervsed Learnng: Compettve Learnng for Feature Dscovery Intutve Idea: Compettve Mechansms for Unsupervsed Learnng Global organzaton from local, compettve weght update Basc prncple expressed by Von der Malsburg Gudng examples from (neuro)bology: lateral nhbton Prevous work: Hebb, 1949; Rosenblatt, 1959; Von der Malsburg, 1973; Fukushma, 1975; Grossberg, 1976; Kohonen, 1982 A Procedural Framework for Unsupervsed Connectonst Learnng Start wth dentcal ( neural ) processng unts, wth random ntal parameters Set lmt on actvaton strength of each unt Allow unts to compete for rght to respond to a set of nputs Feature Dscovery Identfyng (or constructng) new features relevant to supervsed learnng Examples: fndng dstngushable letter characterstcs n handwrten character recognton (HCR), optcal character recognton (OCR) Compettve learnng: transform X nto X ; tran unts n X closest to x Unsupervsed Learnng: Kohonen s Self-Organzng Map (SOM) [1] Another Clusterng Algorthm aka Self-Organzng Feature Map (SOFM) Gven: vectors of attrbute values (x 1, x 2,, x n ) Returns: vectors of attrbute values (x 1, x 2,, x k ) Typcally, n >> k (n s hgh, k = 1, 2, or 3; hence dmensonalty reducng ) Output: vectors x, the projectons of nput ponts x; alsoget P(x j x ) Mappng from x to x s topology preservng Topology Preservng Networks Intutve dea: smlar nput vectors wll map to smlar clusters Recall: nformal defnton of cluster (solated set of mutually smlar enttes) Restatement: clusters of X (hgh-d) wll stll be clusters of X (low-d) Representaton of Node Clusters Group of neghborng artfcal neural network unts (neghborhood of nodes) SOMs: combne deas of topology-preservng networks, unsupervsed learnng Implementaton: and MATLAB NN Toolkt 7

8 Unsupervsed Learnng: Kohonen s Self-Organzng Map (SOM) [2] Kohonen Network (SOM) for Clusterng Tranng algorthm: unnormalzed compettve learnng Map s organzed as a grd (shown here n 2D) Each node (grd element) has a weght vector w j Update Rule Same as compettve learnng algorthm, wth one modfcaton Neghborhood functon assocated wth j* spreads the w j around r r r r w j () t + r ( t ) h j, j* ( x w j ( t )) f j Neghborhood ( j * ) w j ( t + 1) = r w j () t otherwse x : vector n n-space x : vector Dmenson of w j s n (same as nput vector) n 2-space Number of tranable parameters (weghts): m m n for an m-by-m SOM 1999 state-of-the-art: typcal small SOMs 5-20, ndustral strength > 20 Output found by selectng j* whose w j has mnmum Eucldean dstance from x Only one actve node, aka Wnner-Take-All (WTA): wnnng node j*.e., j* = arg mn j w j - x 2 Unsupervsed Learnng: Kohonen s Self-Organzng Map (SOM) [3] Tradtonal Compettve Learnng Only tran j* Corresponds to neghborhood of 0 Neghborhood Functon h j, j* For 2D Kohonen SOMs, h s typcally a square or hexagonal regon j*, the wnner, s at the center of Neghborhood (j*) h j*, j* 1 Nodes n Neghborhood (j) updated whenever j wns,.e., j* = j Strength of nformaton fed back to w j s nversely proportonal to ts dstance from the j* for each x Often use exponental or Gaussan (normal) dstrbuton on neghborhood to decay weght delta as dstance from j* ncreases Annealng of Tranng Parameters Neghborhood must shrnk to 0 to acheve convergence r (learnng rate) must also decrease monotoncally j* Neghborhood of 1 8

9 Unsupervsed Learnng: SOM and Other Projectons for Clusterng Dmensonalty- Reducng Projecton (x ) Clusters of Smlar Records Delaunay Trangulaton Vorono (Nearest Neghbor) Dagram (y) Cluster Formaton and Segmentaton Algorthm (Sketch) Unsupervsed Learnng: Other Algorthms (PCA, Factor Analyss) Intutve Idea Q: Why are dmensonalty-reducng transforms good for supervsed learnng? A: There may be many attrbutes wth undesrable propertes, e.g., Irrelevance: x has lttle dscrmnatory power over c(x) = y Sparseness of nformaton: feature of nterest spread out over many x s (e.g., text document categorzaton, where x s a word poston) We want to ncrease the nformaton densty by squeezng X down Prncpal Components Analyss (PCA) Combnng redundant varables nto a sngle varable (aka component, or factor) Example: ratngs (e.g., Nelsen) and polls (e.g., Gallup); responses to certan questons may be correlated (e.g., lke fshng? tme spent boatng ) Factor Analyss (FA) General term for a class of algorthms that ncludes PCA Tutoral: 9

10 Intuton Clusterng Methods: Desgn Choces Functonal (declaratve) defnton: easy ( We recognze a cluster when we see t ) Operatonal (procedural, constructve) defnton: much harder to gve Possble reason: clusterng of objects nto groups has taxonomc semantcs (e.g., shape, sze, tme, resoluton, etc.) Possble Assumptons Data generated by a partcular probablstc model No statstcal assumptons Desgn Choces Dstance (smlarty) measure: standard metrcs, transformaton-nvarant metrcs 2 L 1 (Manhattan): x - y, L 2 (Eucldean): ( x y ), L (Sup): max x - y Symmetry: Mahalanobs dstance Shft, scale nvarance: covarance matrx Transformatons (e.g., covarance dagonalzaton: rotate axes to get rotatonal nvarance, cf. PCA, FA) Clusterng: Applcatons Data from T. Mtchell s web ste: NCSA D2K Transactonal Database Mnng 6500 news stores from the WWW n FaceFeatureFndng.html Facal Feature Extracton Confdental and propretary to Caterpllar; may only be used wth pror wrtten consent from Caterpllar. Informaton Retreval: Text Document Categorzaton ThemeScapes - NCSA D2K

11 Unsupervsed Learnng and Constructve Inducton Unsupervsed Learnng n Support of Supervsed Learnng Gven: D labeled vectors (x, y) Return: D transformed tranng examples (x, y ) Soluton approach: constructve nducton Feature constructon : generc term Cluster defnton Feature Constructon: Front End Syntheszng new attrbutes Cluster Defnton Logcal: x 1 x 2, arthmetc: x 1 + x 5 / x 2 Other synthetc attrbutes: f(x 1, x 2,, x n ), etc. Dmensonalty-reducng projecton, feature extracton (x, y ) or ((x 1, y 1 ),, (x p, y p )) Subset selecton: fndng relevant attrbutes for a gven target y Parttonng: fndng relevant attrbutes for gven targets y 1, y 2,, y p Cluster Defnton: Back End Constructve Inducton (x, y) Feature (Attrbute) Constructon and Parttonng Form, segment, and label clusters to get ntermedate targets y Change of representaton: fnd an (x, y ) that s good for learnng target y x / (x 1,, x p ) Clusterng: Relaton to Constructve Inducton Clusterng versus Cluster Defnton Clusterng: 3-step process Cluster defnton: back end for feature constructon Clusterng: 3-Step Process Form (x 1,, x k ) n terms of (x 1,, x n ) NB: typcally part of constructon step, sometmes ntegrates both Segment (y 1,, y J ) n terms of (x 1,, x k ) NB: number of clusters J not necessarly same as number of dmensons k Label Assgn names (dscrete/symbolc labels (v 1,, v J )) to (y 1,, y J ) Important n document categorzaton (e.g., clusterng text for nfo retreval) Herarchcal Clusterng: Applyng Clusterng Recursvely 11

12 Termnology Expectaton-Maxmzaton (EM) Algorthm Iteratve refnement: repeat untl convergence to a locally optmal label Expectaton step: estmate parameters wth whch to smulate data Maxmzaton step: use smulated ( fcttous ) data to update parameters Unsupervsed Learnng and Clusterng Constructve nducton: usng unsupervsed learnng for supervsed learnng Feature constructon: front end - construct new x values Cluster defnton: back end - use these to reformulate y Clusterng problems: formaton, segmentaton, labelng Key crteron: dstance metrc (ponts closer ntra-cluster than nter-cluster) Algorthms AutoClass: Bayesan clusterng Prncpal Components Analyss (PCA), factor analyss (FA) Self-Organzng Maps (SOM): topology preservng transform (dmensonalty reducton) for compettve unsupervsed learnng Summary Ponts Expectaton-Maxmzaton (EM) Algorthm Unsupervsed Learnng and Clusterng Types of unsupervsed learnng Clusterng, vector quantzaton Feature extracton (typcally, dmensonalty reducton) Constructve nducton: unsupervsed learnng n support of supervsed learnng Feature constructon (aka feature extracton) Cluster defnton Algorthms EM: mxture parameter estmaton (e.g., for AutoClass) AutoClass: Bayesan clusterng Prncpal Components Analyss (PCA), factor analyss (FA) Self-Organzng Maps (SOM): projecton of data; compettve algorthm Clusterng problems: formaton, segmentaton, labelng Next Lecture: Tme Seres Learnng and Characterzaton 12