A Geometric Perspective on Machine Learning

Transcription

1 A Geometric Perspective on Machine Learning Partha Niyogi The University of Chicago Thanks: M. Belkin, A. Caponnetto, X. He, I. Matveeva, H. Narayanan, V. Sindhwani, S. Smale, S. Weinberger A Geometric Perspective onmachine Learning p.1

2 High Dimensional Data When can we avoid the curse of dimensionality? Smoothness rate (1/n) d s splines,kernel methods, L 2 regularization... Sparsity wavelets, L 1 regularization, LASSO, compressed sensing.. Geometry graphs, simplicial complexes, laplacians, diffusions A Geometric Perspective onmachine Learning p.2

3 Geometry and Data: The Central Dogma Distribution of natural data is non-uniform and concentrates around low-dimensional structures. The shape (geometry) of the distribution can be exploited for efficient learning. A Geometric Perspective onmachine Learning p.3

4 Manifold Learning Learning when data M R N Clustering: M {1,...,k} connected components, min cut Classification: M { 1, +1} P on M { 1, +1} Dimensionality Reduction: f : M R n n << N M unknown: what can you learn about M from data? e.g. dimensionality, connected components holes, handles, homology curvature, geodesics A Geometric Perspective onmachine Learning p.4

5 An Acoustic Example u(t) l s(t) A Geometric Perspective onmachine Learning p.5

6 An Acoustic Example u(t) l s(t) One Dimensional Air Flow (i) V x = A P ρc 2 t (ii) P x = ρ A V t V (x, t) = volume velocity P(x, t) = pressure A Geometric Perspective onmachine Learning p.5

7 Solutions beta u(t) = P n=1 α n sin(nω 0 t) l beta beta s(t) = P n=1 β n sin(nω 0 t) l 2 A Geometric Perspective onmachine Learning p.6

8 Formal Justification Speech speech l 2 generated by vocal tract Jansen and Niyogi (2005) Vision group actions on object leading to different images Donoho and Grimes (2004) Robotics configuration spaces in joint movements Graphics Manifold + Noise may be generic model in high dimensions. A Geometric Perspective onmachine Learning p.7

9 Take Home Message Geometrically motivated approach to learning nonlinear, nonparametric, high dimensions Emphasize the role of the Laplacian and Heat Kernel Semi-supervised regression and classification Clustering and Homology Randomized Algorithms and Numerical Analysis A Geometric Perspective onmachine Learning p.8

10 Pattern Recognition P on X Y X = R N ;Y = {0, 1}, R (x i, y i ) labeled examples find f : X Y Ill Posed A Geometric Perspective onmachine Learning p.9

11 Simplicity A Geometric Perspective onmachine Learning p.10

12 Simplicity A Geometric Perspective onmachine Learning p.10

13 Regularization Principle 1 f = arg min f H K n n (y i f(x i )) 2 + γ f 2 K i=1 Splines Ridge Regression SVM K : X X R is a p.d. kernel e.g. e x y 2 σ 2, (1 + x y) d, etc. H K is a corresponding RKHS e.g., certain Sobolev spaces, polynomial families, etc. A Geometric Perspective onmachine Learning p.11

14 Simplicity is Relative A Geometric Perspective onmachine Learning p.12

15 Simplicity is Relative A Geometric Perspective onmachine Learning p.12

16 Intuitions supp P X has manifold structure geodesic distance v/s ambient distance geometric structure of data should be incorporated f versus f M A Geometric Perspective onmachine Learning p.13

17 Manifold Regularization 1 min f H K n n (y i f(x i )) 2 + γ A f 2 K + γ I f 2 I i=1 f 2 I = Laplacian Iterated Laplacian Heat kernel Differential Operator R gradm f, grad M f = R f M f R f M i f e Mt R f(df) Representer Theorem: f = P n i=1 α ik(x, x i ) + R M α(y)k(x, y) Belkin, Niyogi, Sindhwani (2004) A Geometric Perspective onmachine Learning p.14

18 Approximating f 2 I M is unknown but x 1...x M M f 2 I = M M f, M f i j W ij (f(x i ) f(x j )) 2 A Geometric Perspective onmachine Learning p.15

19 Manifolds and Graphs M G = (V,E) e ij E if x i x j < ǫ W ij = e x i x j 2 t M L = D W gradf, gradf i,j W ij(f(x i ) f(x j )) 2 f( f) f T Lf A Geometric Perspective onmachine Learning p.16

20 Manifold Regularization 1 n n i=1 V (f(x i ),y i ) + γ A f 2 K + γ I i j W ij (f(x i ) f(x j )) 2 Representer Theorem: f opt = n+m i=1 α ik(x,x i ) V (f(x), y) = (f(x) y) 2 : Least squares V (f(x), y) = (1 yf(x)) + : Hinge loss (Support Vector Machines) A Geometric Perspective onmachine Learning p.17

21 Ambient and Intrinsic Regularization SVM Laplacian SVM Laplacian SVM γ A = γ I = γ A = γ I = γ A = γ I = 1 A Geometric Perspective onmachine Learning p.18

22 Experimental comparisons Dataset g50c Coil20 Uspst mac-win WebKB WebKB WebKB Algorithm (link) (page) (page+link) SVM (full labels) RLS (full labels) SVM (l labels) RLS (l labels) Graph-Reg TSVM Graph-density TSVM LDS LapSVM LapRLS LapSVM joint LapRLS joint A Geometric Perspective onmachine Learning p.19

23 Real World A Geometric Perspective onmachine Learning p.20

24 Graph and Manifold Laplacian Fix f : X R. Fix x M [L n f](x) = 1 nt n (4πt n ) d/2 Put t n = n 1/(d+2+α), where α > 0 j (f(x) f(x j))e x xj 2 4tn with prob. 1, lim n (L nf) x = M f x Belkin (2003), Belkin and Niyogi (2004,2005) also Lafon (2004), Coifman et al,hein, Gine and Koltchinski A Geometric Perspective onmachine Learning p.21

25 Random Graphs and Matrices Given x 1,...,x n M R N W ij = 1 t(4πt) d/2e x i x j 2 t Eig[D W] = Eig[L t n n ] Eig[ M ] O( 1 n 1/(d+3)) Belkin Niyogi 06,08 Allows us to reconstruct spaces of functions on the manifold. (Patodi, Dodziuk: triangulated manifolds) A Geometric Perspective onmachine Learning p.22

26 Manifold + Noise Flexible, non-parametric, geometric probability model A Geometric Perspective onmachine Learning p.23

27 Remarks on Noise 1. Arbitrary probability distribution on the manifold: convergence to weighted Laplacian. 2. Noise off the manifold: µ = µ M + µ R N 3. Noise off the manifold: z = x + η ( N(0,σ 2 I)) We have lim lim t 0 σ 0 Lt,σ f(x) = f(x) A Geometric Perspective onmachine Learning p.24

28 Local and Global Analysis X = documents, signals, financial time series, sequences d(x,x ) makes sense locally What is good global distance? What is global geometry/topology of X? What is good space of functions on X that is adapted to geometry of X? A Geometric Perspective onmachine Learning p.25

29 Similarity Metrics Sim t (x,x ) = K t (x,x ) = α t e d2 (x,x ) t L t f(x) = X K t (x,y)(f(x) f(y))dρ(y) 1 n y X K t (x,y)(f(x) f(y) Choose t small λ i,φ i Choose T large H T (x,x ) = e λ it φ i (x)φ i (x ) f = i α i φ i ; i α 2 i g(λ i ) A Geometric Perspective onmachine Learning p.26

30 Intrinsic Spectrograms extrinsic p intrinsic p > φ (p) A Geometric Perspective onmachine Learning p.27

31 Speech and Intrinsic Eigenfunctions 0.1 f 1 : Background Noise 0.1 f 3 : Vowels Time (s) Time (s) 0.1 f 4 : Fricatives 0.1 f 10 : Stops, Nasals, and Affricates Time (s) Time (s) A Geometric Perspective onmachine Learning p.28

32 LaplacianFaces: Appearance Manifolds X. He et al. A Geometric Perspective onmachine Learning p.29

33 Visualizing Digits A Geometric Perspective onmachine Learning p.30

34 Vision Example f : R 2 [0, 1] F = {f f(x,y) = v(x t,y r)} A Geometric Perspective onmachine Learning p.31

35 PCA versus Laplacian Eigenmaps 8 x nz = x A Geometric Perspective onmachine Learning p.32

36 Computer Vision: Laplacian Eigenmaps Machine vision: inferring joint angles. Corazza, Andriacchi, Stanford Biomotion Lab, 05, Partiview, Surendran Isometrically invariant representation. A Geometric Perspective onmachine Learning p.33

37 Connections and Implications Clustering and Topology sparse cuts, combinatorial Laplacians, complexes (Niyogi, Smale, Weinberger, 2006,2008; Narayanan, Belkin, Niyogi, 2006) Numerical Analysis heat flow based algorithms, sampling, PDEs (Belkin, Narayanan, Niyogi, 2006; Narayanan and Niyogi, 2008) Random Matrices and Graphs results on spectra Belkin and Niyogi, 2008 Speech, Text, Vision Intrinsic versus Extrinsic He et al. 2005, Jansen and Niyogi, 2006 A Geometric Perspective onmachine Learning p.34

38 Learning Homology x 1,...,x n M R N Can you learn qualitative features of M? Can you tell a torus from a sphere? Can you tell how many connected components? Can you tell the dimension of M? (e.g. Carlsson, Zamorodian, Edelsbrunner, Guibas, Oudot, Lieutier, Chazal, Dey, Amenta,Choi, Cohen-Steiner, de Silva etc.) A Geometric Perspective onmachine Learning p.35

39 Well Conditioned Submanifolds τ Μ Tubular Neighborhood Condition No. 1 τ Min. distance to medial axis A Geometric Perspective onmachine Learning p.36

40 Euclidean and Geodesic distance M R N condition τ p,q M where p q R N = d. For all d τ 2, d M (p,q) τ τ 1 2d τ In fact, Second Fundamental Form Bounded by 1 τ A Geometric Perspective onmachine Learning p.37

41 Homology x 1,...,x n M R N U = n i=1b ǫ (x i ) If ǫ well chosen, then U deformation retracts to M. Homology of U is constructed using the nerve of U and agrees with the homology of M. A Geometric Perspective onmachine Learning p.38

42 Theorem M R N with cond. no. τ x = {x 1,...,x n } uniformly sampled i.i.d. 0 < ǫ < τ 2 β = Let U = x x B ǫ (x) vol(m) (sin 1 (ǫ/2τ)) d vol(b ǫ/8 ) n > β(log(β) + log( 1 δ )) with prob. > 1 δ, homology of U equals the homology of M (Niyogi, Smale, Weinberger, 2004) A Geometric Perspective onmachine Learning p.39

43 A Data-derived complex x 1,...,x n R N Pick ǫ > 0 and balls B ǫ (x i ) Put j-face for every (i 0,...,i j ) such that j m=0 B ǫ(x im ) φ A Geometric Perspective onmachine Learning p.40

44 Chains and the Combinatorial Laplacian j chain is a formal sum σ α σσ v C j is the vector space of j-chains e σ f j : C j C j 1 u g w j : C j 1 C j j = j j + j+1 j+1 A Geometric Perspective onmachine Learning p.41

45 Noise such that P on R N P(x,y) = P(x)P(y x) where x M,y N x a P(x) P(y x) = σ 2 I N d A Geometric Perspective onmachine Learning p.42

46 Small Noise N dσ cτ [Theorem] There exists an algorithm that recovers homology that is polynomial in D. Niyogi, Smale, Weinberger; 2008 A Geometric Perspective onmachine Learning p.43

47 Future Directions Machine Learning Scaling Up Multi-scale Geometry of Natural Data Geometry of Structured Data Algorithmic Nash embedding Random Hodge Theory Partial Differential Equations Graphics Algorithms A Geometric Perspective onmachine Learning p.44