Dimension Reduction and Manifold Learning. Xin Zhang

Transcription

1 Dimesio Reductio ad Maifold Learig Xi Zhag

2 Cotet Motivatio of maifold learig Pricipal compoet aalysis ad its etesio Maifold learig Global oliear maifold learig (IsoMap) Local oliear maifold learig (LLE) Eample of applicatios

3 Motivatios Need to aalyze large amouts multivariate high dimesioal data. Huma faces, shape ad motio. Speech Waveforms. Global Climate patters. Gee Distributios. Difficult to visualize data i dimesios just greater tha three Curse of Dimesioality

4 Motivatio, Assumptio ad Goal Motivatio Large volumes of high-dimesioal data Curse of Dimesioality Assumptio Goal We assume the data lies o a embedded maifold (Euclidea) withi the highdimesioal space. To fid meaigful low-dimesioal structures hidde i their highdimesioal observatios

5 Motivatio (cot d) May data sets have the property that the data poits all lie close to a maifold of much lower dimesioality tha that of the origial data space. Dimesio Reductio (cotiuous latet variable). To reduce the degree of freedom (DoF) of the data set. To reduce the umber of radom variable uder cosideratio. To fid a sub-space that preserves the major property of data. 5

6 Major Maifold Learig Algorithms Liear No-liear Tesor Graph Embeddig Kerel Priciple Compoet Aalysis Liear Discrimiative Aalysis Local Liear Embeddig ISOMAP Laplacia Eigemap Coceptual Maifold Shuicheg Ya, Dog Xu, Beyu Zhag, Hog-Jiag Zhag. Graph Embeddig ad Etesios: A Geeral Framework for Dimesioality Reductio, IEEE Trasactios o PAMI, Vol.29, No., pp. 40-5, 2007,

7 Liear Dimesio Reductio: Pricipal Compoet Aalysis (PCA) PCA is defied as the orthogoal projectio of the data oto a lower-dimesioal liear space, such that the variace of the projected data is maimized. 7

8 No-liear Dimesioal Reductio (Maifold Learig) Maifold learig is the process of eplorig a low-dimesioal o-liear embeddig uderlyig a set of high-dimesioal data. 8

9 Cotet Motivatio of maifold learig Pricipal compoet aalysis ad its etesio Maifold learig Global oliear maifold learig (IsoMap) Local oliear maifold learig (LLE) Eample of applicatios

10 PCA Formulatios Maimum variace formulatio To fid a sub-space where the variace of the projected data is maimized. Miimum-error formulatio To fid a sub-space where data ca be recostructed with least square error. 0

11 Maimum Variace Formulatio Give a D-dimesioal data set D,..., N ad R We project the data o to a M-dimesioal (M<<D) subspace with the maimum variace. Let s cosider a -D space determied by T u with u u. We compute the variace of the projected data by N T u ( ) N 2 where N N

12 Variace Computatio 2 N N 2 T ) ( u N N T T ) ( ) ( u u T T ) ( ) ( u u N N N N T T ) ( ) ( u u T Su u N N T ) ( ) ( S

13 Variace Maimizatio We ow wat to maimize the projected variace with T respect to u with u u. u * u T T arg ma u Su ( u d du T T u Su u u ) 0 T A 2A T ( A A ) ( T 2Su 2u 2 0 ) The variace will be a maimum if u equal to the eige-vector havig the largest eige-value. 3 Su u u T Su

14 M-Dimesioal Subspace For a M-dimesioal projectio space, the optimal liear projectio is defied by the M eigevectors associated with the M largest eigevalues. p p i N i 2 i 2 i p p

15 PCA Applicatios: Data Compressio () 5

16 PCA Applicatios: Data Compressio (2) 6

17 PCA Applicatios: Data Pre-processig 7 N N T y y, 2, 2 y y y, 2 z z z i i i i v m z N i i N m N i i i m N v 2 ) ( ) ( T / 2 U L y L u u 2 U N N I Normalizatio whiteig

18 PCA Applicatios: Face Recogitio Origial face PCA- PCA-2 PCA-3 PCA-4 PCA-5 PCA-6 PCA-7 PCA-8 PCA-9 8

19 How about this? 9

20 Kerel PCA We itroduce a o-liear trasformatio that coverts each data poits ito a M-dimesioal feature space, where we ca perform stadard PCA. 20

21 Cotet Motivatio of maifold learig Pricipal compoet aalysis ad its etesio Maifold learig Global oliear maifold learig (IsoMap) Local oliear maifold learig (LLE) Eample of applicatios

22 Cocepts of Maifold A maifold is a topological space which is locally Euclidea. I geeral, ay object which is early "flat" o small scales is a maifold. Euclidea space is a simplest eample of a maifold. Cocept of submaifold. Maifolds arise aturally wheever there is a smooth variatio of parameters [like pose of the face i previous eample] The dimesio of a maifold is the miimum iteger umber of co-ordiates ecessary to idetify each poit i that maifold. Embed data i a higher dimesioal space to a lower dimesioal maifold Cocept of Dimesioality Reductio:

23 Maifold of Perceptio..Huma Visual System You ever see the same face twice. Preceive costacy whe raw sesory iputs are i flu..

24 Iterpolatio alog Maifold? Maifold Liear Iterpolatio Time

25 Major Maifold Learig Algorithms Liear No-liear Tesor Graph Embeddig Kerel Priciple Compoet Aalysis Liear Discrimiative Aalysis Local Liear Embeddig ISOMAP Laplacia Eigemap Coceptual Maifold Shuicheg Ya, Dog Xu, Beyu Zhag, Hog-Jiag Zhag. Graph Embeddig ad Etesios: A Geeral Framework for Dimesioality Reductio, IEEE Trasactios o PAMI, Vol.29, No., pp. 40-5, 2007,

26 Liear Dimesioality Reductio Methods Priciple compoet aalysis (PCA): It seeks projectio directios with maimal variaces. I other words, it removes directios with miimal variaces. It ca be eteded as the o-liear method, like kerel PCA Liear discrimiative aalysis (LDA): It searches for the projectio directios that are most effective for discrimiatio by miimizig the ratio betwee the itra-class ad iterclass scatters. Etra iformatio icorporatio

27 Graph Embeddig Maifold Learig Goal: Itroduce geometric property or prior topology kowledge ito similarity measuremet amog data. Assumptio: The global o-liear structure has local liear smoothess. The local similarity ca be measured as Euclidea distace. Approach: Represet each verte of a graph as a low-dimesioal vector that preserves similarities betwee verte pairs i the highdimesioal space.

28 Local Liear Embeddig (LLE) LLE maps the iput data to a lower dimesioal space i a maer that preserves the relatioship betwee the eighborig poits. Fit locally, thik globally! Discover the global iteral coordiates of the maifold The color codig illustrates the eighbor-hood preservatio S. Roweis ad L. Saul, Noliear Dimesioality Reductio by Locally Liear Embeddig, Sciece 2000

29 Fit Locally We epect each data poit ad its eighbours to lie o or close to a locally liear patch of the maifold. Each poit ca be writte as a liear combiatio of its eighbors. Weights are chose to miimize the recostructio Error. Derivatio o board

30 Importat property... The weights that miimize the recostructio errors are ivariat to rotatio, rescalig ad traslatio of the data poits. Ivariace to traslatio is eforced by addig the costrait that the weights sum to oe. The same weights that recostruct the data poits i D dimesios should recostruct it i the maifold i d dimesios. The weights characterize the itrisic geometric properties of each eighborhood.

31 Thik Globally

32 Local Liear Embeddig (LLE) Assig eighbors to each data poit. Fid liear weights by miimizig that ca be solved as a leastsquare problem with weights sum-to-oe costrait. Compute the low dimesioal embeddig vector by miimizig recostructio error with fied weights

33 Local Liear Embeddig (LLE) Assig eighbors to each data poit. Fid liear weights by miimizig that ca be solved as a leastsquare problem with weights sum-to-oe costrait. Compute the low dimesioal embeddig vector by miimizig recostructio error with fied weights

34 LLE Detailed Algorithm Iput X: D by N matri cosistig of N data items i D dimesios. Output Y: d by N matri cosistig of d < D dimesioal embeddig coordiates for the iput poits.. Fid eighbors i X space(e.g. usig KNN) for i=:n compute the distace from Xi to every other poit Xj fid the K smallest distaces assig the correspodig poits to be eighbors of Xi Ed 2. Solve for recostructio weights W. for i=:n create matri Z cosistig of all eighbors of Xi (i colums) subtract Xi from every colum of Z compute the local covariace C=Z'*Z solve liear system C*w = for w ( deotes colum vector of all oes) Set elemets i the i-th row of w equal to w/sum(w); ed 3. Compute embeddig coordiates Y usig weights W. create sparse matri M = (I-W)'*(I-W) fid bottom d+ eigevectors of M (correspodig to the d+ smallest eigevalues) set the qthrow of Y to be the q+ smallest eigevector (discard the bottom eigevector [,,,...] with eigevaluezero)

35

36

37

38 Grolliers Ecyclopedia

39 LLE Result ad Discussio PCA (top) ad LLE (bottom) compariso of the maifold learig Applied two methods to a set of image geerated by a sigle face traslated across a 2-D oisy backgroud. LLE maps the images with corer faces to the corers of its 2-D embeddig while PCA fails to preserve the eighborhood structure of earby images. Pros ad Cos of LLE Icremetal, fast, oe free parameter Simple liear algebra operatios Might distort global structure No mappig relatioship

40 ISOMAP ISOMAP fids the low-dimesioal represetatios for a data set by approimately preservig the geodesic distaces of the data pairs. Assumptio: oly geodesic distaces reflect the true geometry of the maifold ad preserve the itrisic geometry of the data. J. Teebaum, V. Silva ad J. Lagford, A Global Geometric Framework for Noliear Dimesioality Reductio, Sciece 2000

41 ISOMAP Algorithm. Costruct eighborhood graph Build a sparse graph with K-earest eighbors for every poit II. Compute shortest path Ifer other iter-poit distace by fidig shortest path o the graph III. Costruct d-dimesioal embeddig A d-dimesioal space to preserve iter-poit distaces by usig the top eigevectors scaled by their eigevalues.

42 ISOMAP Result

43 ISOMAP Discussios Pros: Cos: It preserves global structure Oe parameter for eighborhood determiatio Sesitive to oise, oise edges Computatioally epesive

44 Compariso

45 Brief Summary of Graph Embeddig The low-dimesioal vector represetatios relatioship best characterize the similar graphic or geometric relatioship betwee the high dimesioal data pairs. Etesios Prior kowledge guided maifold learig Kerel maifold learig Icremetal maifold learig Out-of-sample maifold learig Multiple output maifold learig

46 Coceptual Maifold Rather tha learig a maifold from data, a coceptual maifold is proposed to ideally characterize the itrisic geometry of data. Torus maifold ca simultaeous iferece the view ad body pose based. u: view v: pose A. Elgammal ad C. Lee, Trackig People o a Torus, IEEE Tras. o PAMI, 2009

47 Cotet Motivatio of maifold learig Pricipal compoet aalysis ad its etesio Maifold learig Global oliear maifold learig (IsoMap) Local oliear maifold learig (LLE) Eample of applicatios

48 Maifold Learig Applicatios Maifold learig has produced successful results o Image deoise Face recogitio Face epressio recogitio ad iterpolatio Character recogitio Gesture recogitio Activity recogitio Pose estimatio Dyamic appearace modelig for trackig ad so o

49 Face Iterpolatio LPP LLE He XF et al., "Face recogitio usig Laplaciafaces," IEEE Tras. Patt. Aal. Mach. It. 27(3): , March S. Roweis ad L. Saul, Noliear Dimesioality Reductio by Locally Liear Embeddig, Sciece 2000

50 Face Epressio Recogitio Y. Chag, et al, Maifold based aalysis of facial epressio, Image ad Visio Computig, 2006 Elgammal ad Lee, The role of maifold learig i huma motio aalysis, Huma Motio Aalysis, Spriger, 2008

51 Pose Estimatio Appearace maifold Sigle view appearace maifold Multiple-view appearace maifold A. Elgammal ad C. Lee, Trackig People o a Torus, IEEE Tras. o PAMI, 2009

52 People Recogitio ad Image De-oise A. Elgammal, ad C. Lee, The role of maifold learig i huma motio aalysis, Huma Motio Aalysis, Spriger, 2008

53 Pose estimatio usig Torus

54 Dyamic Appearace Modelig ad Trackig H. Lim et. al, Dyamic Appearace Modelig for Huma Trackig, CVPR 2006

55 Coclusio Maifold learig is a efficiet tool to discover the embeddig space with the itrisic structure. The graphical, geometric, topological iformatio is importat prior kowledge. There are still some issues, like the eighborhood selectio, the parameter optimizatio, the oisy ad sparse data set, the mappig relatioship betwee low-dimesioal represetatio ad the highdimesioal data. READ A itroductio to Mote Carlo Samplig by McKay Chapter of Patter Recogitio ad Machie Itelligece by Bishop