A Geometric Perspective on Machine Learning

Transcription

1 A Geometric Perspective on Machine Learning Partha Niyogi The University of Chicago Collaborators: M. Belkin, V. Sindhwani, X. He, S. Smale, S. Weinberger A Geometric Perspectiveon Machine Learning p.1

2 Manifold Learning Learning when data Clustering: connected components, min cut Classi cation: on Dimensionality Reduction: unknown: what can you learn about e.g. dimensionality, connected components holes, handles, homology curvature, geodesics from data? A Geometric Perspectiveon Machine Learning p.2

3 An Acoustic Example A Geometric Perspectiveon Machine Learning p.3

4 An Acoustic Example One Dimensional Air Flow (i) (ii) volume velocity pressure A Geometric Perspectiveon Machine Learning p.3

5 Solutions beta! beta! beta! A Geometric Perspectiveon Machine Learning p.4

6 Acoustic Phonetics A 1 A 2 l 1 Vocal Tract modeled as a sequence of tubes. (e.g. Stevens, 1998) l 2 Jansen and Niyogi (in prep.) A Geometric Perspectiveon Machine Learning p.5

7 Pattern Recognition on labeled examples nd Ill Posed A Geometric Perspectiveon Machine Learning p.6

8 Simplicity A Geometric Perspectiveon Machine Learning p.7

9 Simplicity A Geometric Perspectiveon Machine Learning p.7

10 Regularization Principle Splines Ridge Regression SVM e.g. etc. is a p.d. kernel is a corresponding RKHS e.g., certain Sobolev spaces, polynomial families, etc. A Geometric Perspectiveon Machine Learning p.8

11 Simplicity is Relative A Geometric Perspectiveon Machine Learning p.9

12 Simplicity is Relative A Geometric Perspectiveon Machine Learning p.9

13 Intuitions supp has manifold structure geodesic distance v/s ambient distance geometric structure of data should be incorporated versus A Geometric Perspectiveon Machine Learning p.10

14 Geometry and similarity A Geometric Perspectiveon Machine Learning p.11

15 Geometry and similarity A Geometric Perspectiveon Machine Learning p.11

16 Geometry and similarity Geometry is important. A Geometric Perspectiveon Machine Learning p.11

17 Geometry and similarity Geometry is important. Unlabeled data to estimate geometry. A Geometric Perspectiveon Machine Learning p.11

18 Manifold Regularization Laplacian Iterated Laplacian Heat kernel Differential Operator e Representer Theorem: Belkin, Niyogi, Sindhwani (2004) A Geometric Perspectiveon Machine Learning p.12

19 Approximating is unknown but A Geometric Perspectiveon Machine Learning p.13

20 Manifolds and Graphs if A Geometric Perspectiveon Machine Learning p.14

21 Manifold Regularization Representer Theorem: : Least squares : Hinge loss (Support Vector Machines) A Geometric Perspectiveon Machine Learning p.15

22 Ambient and Intrinsic Regularization SVM Laplacian SVM Laplacian SVM !1!1!1! ! A = ! I = 0! ! A = ! I = 0.01! ! A = ! I = 1 A Geometric Perspectiveon Machine Learning p.16

23 Experimental Results: USPS Error Rates RLS vs LapRLS RLS LapRLS Error Rates SVM vs LapSVM SVM LapSVM Error Rates TSVM vs LapSVM TSVM LapSVM Classification Problems Classification Problems Classification Problems 15 Out!of!Sample Extension 15 Out!of!Sample Extension Std Deviation of Error Rates 15 LapRLS (Test) 10 5 LapSVM (Test) 10 5 SVM (o), TSVM (x) Std Dev LapRLS (Unlabeled) LapSVM (Unlabeled) LapSVM Std Dev A Geometric Perspectiveon Machine Learning p.17

24 Experimental Results: Isolet Error Rate (unlabeled set) Error Rates (test set) RLS vs LapRLS RLS LapRLS Labeled Speaker # RLS vs LapRLS RLS LapRLS Labeled Speaker # Error Rates (unlabeled set) Error Rates (test set) SVM vs TSVM vs LapSVM SVM TSVM LapSVM Labeled Speaker # SVM vs TSVM vs LapSVM SVM TSVM LapSVM Labeled Speaker # A Geometric Perspectiveon Machine Learning p.18

25 Experimental comparisons Dataset g50c Coil20 Uspst mac-win WebKB WebKB WebKB Algorithm (link) (page) (page+link) SVM (full labels) RLS (full labels) SVM (l labels) RLS (l labels) Graph-Reg TSVM Graph-density TSVM LDS LapSVM LapRLS LapSVM LapRLS A Geometric Perspectiveon Machine Learning p.19

26 Convergence E = E = E = E = A Geometric Perspectiveon Machine Learning p.20

27 Convergence Theorem with prob. A Geometric Perspectiveon Machine Learning p.21 Belkin and Niyogi (in prep.)

28 Graph and Manifold Laplacian Fix. Fix Put, where with prob. 1, Belkin (2003), Belkin and Niyogi (in preparation) also Lafon (2004), Coifman et a A Geometric Perspectiveon Machine Learning p.22

29 Well Conditioned Submanifolds " # Tubular Neighborhood Condition No. A Geometric Perspectiveon Machine Learning p.23

30 Euclidean and Geodesic distance For all, condition where. In fact, Second Fundamental Form Bounded by A Geometric Perspectiveon Machine Learning p.24

31 Learning Homology Can you learn qualitative features of? Can you tell a torus from a sphere? Can you tell how many connected components? Can you tell the dimension of? (e.g. Carlsson, de Silva et al) A Geometric Perspectiveon Machine Learning p.25

32 Homology If well chosen, then deformation retracts to. Homology of is constructed using the nerve of and agrees with the homology of. A Geometric Perspectiveon Machine Learning p.26

33 Theorem with cond. no. uniformly sampled i.i.d. Let with prob., homology of equals the homology of (Niyogi, Smale, Weinberger, 2004) A Geometric Perspectiveon Machine Learning p.27

34 Spectral Clustering Isoperimetric inequalities. Cheeger constant. M $M 1!M 1 M 1 e 1 &e 1 =% 1 e 1 cut to cluster [Cheeger] A Geometric Perspectiveon Machine Learning p.28

35 Clustering Digits A Geometric Perspectiveon Machine Learning p.29

36 Clustering Speech A Geometric Perspectiveon Machine Learning p.30

37 Computer Vision: Laplacian Eigenmaps Machine vision: inferring joint angles. Corazza, Andriacchi, Stanford Biomotion Lab, 05, Partiview, Surendran Isometrically invariant representation. A Geometric Perspectiveon Machine Learning p.31

38 Important Issues How to handle noise theoretically and practically? How to choose the graph neighborhood correctly? How often do manifolds arise in natural data? What is the right metric on these manifolds? What are other ways in which one might utilize the geometry of natural distributions? Identify real problems where this approach can make a difference. Complexity estimates and provably correct algorithms rather than heuristics. A Geometric Perspectiveon Machine Learning p.32