A Geometric Perspective on Data Analysis

Transcription

1 A Geometric Perspective on Data Analysis Partha Niyogi The University of Chicago Collaborators: M. Belkin, X. He, A. Jansen, H. Narayanan, V. Sindhwani, S. Smale, S. Weinberger A Geometric Perspectiveon Data Analysis p.1

2 Manifold Learning Learning when data M R N Clustering: M {1,...,k} connected components, min cut Classification: M { 1, +1} P on M { 1, +1} Dimensionality Reduction: f : M R n n << N M unknown: what can you learn about M from data? e.g. dimensionality, connected components holes, handles, homology curvature, geodesics A Geometric Perspectiveon Data Analysis p.2

3 An Acoustic Example u(t) l s(t) A Geometric Perspectiveon Data Analysis p.3

4 An Acoustic Example u(t) l s(t) One Dimensional Air Flow (i) V x = A P ρc 2 t (ii) P x = ρ A V t V (x, t) = volume velocity P(x, t) = pressure A Geometric Perspectiveon Data Analysis p.3

5 Solutions beta u(t) = P n=1 α n sin(nω 0 t) l beta beta s(t) = P n=1 β n sin(nω 0 t) l 2 A Geometric Perspectiveon Data Analysis p.4

6 Acoustic Phonetics A 1 A 2 l 1 Vocal Tract modeled as a sequence of tubes. (e.g. Stevens, 1998) l 2 Jansen and Niyogi (2005,2006) A Geometric Perspectiveon Data Analysis p.5

7 Pattern Recognition P on X Y X = R N ;Y = {0, 1}, R (x i,y i ) labeled examples find f : X Y Ill Posed A Geometric Perspectiveon Data Analysis p.6

8 Simplicity A Geometric Perspectiveon Data Analysis p.7

9 Simplicity A Geometric Perspectiveon Data Analysis p.7

10 Regularization Principle 1 f = arg min f H K n n (y i f(x i )) 2 + γ f 2 K i=1 Splines Ridge Regression SVM K : X X R is a p.d. kernel e.g. e x y 2 σ 2, (1 + x y) d, etc. H K is a corresponding RKHS e.g., certain Sobolev spaces, polynomial families, etc. A Geometric Perspectiveon Data Analysis p.8

11 Simplicity is Relative A Geometric Perspectiveon Data Analysis p.9

12 Simplicity is Relative A Geometric Perspectiveon Data Analysis p.9

13 Intuitions supp P X has manifold structure geodesic distance v/s ambient distance geometric structure of data should be incorporated f versus f M A Geometric Perspectiveon Data Analysis p.10

14 Manifold Regularization 1 min f H K n n (y i f(x i )) 2 + γ A f 2 K + γ I f 2 I i=1 f 2 I = Laplacian Iterated Laplacian Heat kernel Differential Operator R gradm f, grad M f = R f M f R f M i f e Mt R f(df) Representer Theorem: f = P n i=1 α ik(x, x i ) + R M α(y)k(x, y) Belkin, Niyogi, Sindhwani (2004) A Geometric Perspectiveon Data Analysis p.11

15 Approximating f 2 I M is unknown but x 1...x M M f 2 I = M M f, M f i j W ij (f(x i ) f(x j )) 2 A Geometric Perspectiveon Data Analysis p.12

16 Manifolds and Graphs M G = (V,E) e ij E if x i x j < ɛ W ij = e x i x j 2 t M L = D W gradf, gradf i,j W ij(f(x i ) f(x j )) 2 f( f) f T Lf A Geometric Perspectiveon Data Analysis p.13

17 Manifold Regularization 1 n n i=1 V (f(x i ),y i ) + γ A f 2 K + γ I i j W ij (f(x i ) f(x j )) 2 Representer Theorem: f opt = n+m i=1 α ik(x,x i ) V (f(x), y) = (f(x) y) 2 : Least squares V (f(x), y) = (1 yf(x)) + : Hinge loss (Support Vector Machines) A Geometric Perspectiveon Data Analysis p.14

18 Ambient and Intrinsic Regularization SVM Laplacian SVM Laplacian SVM γ A = γ I = γ A = γ I = γ A = γ I = 1 A Geometric Perspectiveon Data Analysis p.15

19 Experimental Results: USPS Error Rates RLS vs LapRLS RLS LapRLS Error Rates SVM vs LapSVM SVM LapSVM Error Rates TSVM vs LapSVM TSVM LapSVM Classification Problems Classification Problems Classification Problems 15 Out of Sample Extension 15 Out of Sample Extension Std Deviation of Error Rates 15 LapRLS (Test) 10 5 LapSVM (Test) 10 5 SVM (o), TSVM (x) Std Dev LapRLS (Unlabeled) LapSVM (Unlabeled) LapSVM Std Dev A Geometric Perspectiveon Data Analysis p.16

20 Experimental Results: Isolet Error Rate (unlabeled set) Error Rates (test set) RLS vs LapRLS RLS LapRLS Labeled Speaker # RLS vs LapRLS RLS LapRLS Labeled Speaker # Error Rates (unlabeled set) Error Rates (test set) SVM vs TSVM vs LapSVM SVM TSVM LapSVM Labeled Speaker # SVM vs TSVM vs LapSVM SVM TSVM LapSVM Labeled Speaker # A Geometric Perspectiveon Data Analysis p.17

21 Experimental comparisons Dataset g50c Coil20 Uspst mac-win WebKB WebKB WebKB Algorithm (link) (page) (page+link) SVM (full labels) RLS (full labels) SVM (l labels) RLS (l labels) Graph-Reg TSVM Graph-density TSVM LDS LapSVM LapRLS LapSVM joint LapRLS joint A Geometric Perspectiveon Data Analysis p.18

22 Graph and Manifold Laplacian Fix f : X R. Fix x M (L n f) = j (f(x) f(x j))e x x j 2 4tn Put t n = n k 2 α, where α > 0 with prob. 1, lim n (4πt n ) k+1 2 n (L n f) x = M f x Belkin (2003), Belkin and Niyogi (2004,2005) also Lafon (2004), Coifman et al A Geometric Perspectiveon Data Analysis p.19

23 Learning Homology x 1,...,x n M R N Can you learn qualitative features of M? Can you tell a torus from a sphere? Can you tell how many connected components? Can you tell the dimension of M? (e.g. Carlsson, Zamorodian, Edelsbrunner, de Silva et al) A Geometric Perspectiveon Data Analysis p.20

24 Homology x 1,...,x n M R N U = n i=1b ɛ (x i ) If ɛ well chosen, then U deformation retracts to M. Homology of U is constructed using the nerve of U and agrees with the homology of M. A Geometric Perspectiveon Data Analysis p.21

25 Theorem M R d with cond. no. τ x = {x 1,...,x n } uniformly sampled i.i.d. 0 < ɛ < τ 2 β = Let U = x x B ɛ (x) vol(m) (sin 1 (ɛ/2τ)) k vol(b ɛ/8 ) n > β(log(β) + log( 1 δ )) with prob. > 1 δ, homology of U equals the homology of M (Niyogi, Smale, Weinberger, 2004) A Geometric Perspectiveon Data Analysis p.22

26 Spectral Clustering Isoperimetric inequalities. Cheeger constant. M δm 1 M1 M 1 e 1 e 1 = λ 1 e 1 cut to cluster h = inf vol n 1 (δm 1 ) min (vol n (M 1 ), vol n (M M 1 )) h λ1 2 [Cheeger] A Geometric Perspectiveon Data Analysis p.23

27 Clustering Digits A Geometric Perspectiveon Data Analysis p.24

28 Clustering Speech A Geometric Perspectiveon Data Analysis p.25

29 Volume Computation Volume of a convex body is deterministically hard to compute. (Báráany and Füredi, 1987). Polynomial by randomized algorithm. (Dyer, Kannan, Frieze, 1991). Current best O (d 4 ) by Lovász and Vempala. A Geometric Perspectiveon Data Analysis p.26

30 Volume Computation Volume of a convex body is deterministically hard to compute. (Báráany and Füredi, 1987). Polynomial by randomized algorithm. (Dyer, Kannan, Frieze, 1991). Current best O (d 4 ) by Lovász and Vempala. Surface volume at least as hard as volume. Grötschel, Lovász and Schrijver (1987) list as open problem. Dyer, Gritzmann and Hufnagel (1998) compute surface volume in randomized polynomial time. Complexity seems high. A Geometric Perspectiveon Data Analysis p.26

31 Surface Volume δμ Μ A Geometric Perspectiveon Data Analysis p.27

32 Surface Volume δμ Μ A Geometric Perspectiveon Data Analysis p.27

33 Surface Volume δμ Μ Heat Equation u = u t with u(x, 0) = f(x) Solution u(x, t) = f(x) K t (x, y) r π Z F t (M) = t R d \M u(x, t)dx A Geometric Perspectiveon Data Analysis p.27

34 Theorems [Theorem 1] lim t 0 F t (M) = M [Theorem 2] Let M be a convex body in R d such that (i) B r M B R (ii) M has condition 1/τ Then, it is possible to find the surface area of M in time O ( d 4 with error of ɛ with prob. > 3/4. ɛ 2 + d3.5 R 3 r 2 τɛ 3 ), Belkin, Narayanan, Niyogi (2005) A Geometric Perspectiveon Data Analysis p.28

35 Important Issues How to handle noise theoretically and practically? How to choose the graph neighborhood correctly? How often do manifolds arise in natural data? What is the right metric on these manifolds? What are other ways in which one might utilize the geometry of natural distributions? Identify real problems where this approach can make a difference. Complexity estimates and provably correct algorithms rather than heuristics. A Geometric Perspectiveon Data Analysis p.29